sched, steal_governor: Introduce preferred CPUs and steal-driven vCPU backoff
Subsystems (10)
Shrikanth Hegde
maintainerv14 cover sched, steal_governor: Introduce preferred CPUs and steal-driven vCPU backoff
tl;dr
This patch series represents the result of multiple iterations, redesigns and community feedback. What started as an arch-specific RFC has evolved into a scheduler mechanism paired with a virtualization driver.
This feature is intended for workloads where vCPU preemption has costs beyond the lost CPU time, such as lock-holder preemption, critical sections and TLB/cache misses. Database workloads, for example, contain a mixture of OLTP/OLAP activity and are prone to these costs. Reducing vCPU preemption in these cases can help improve overall system throughput. It may regress slightly for pure CPU-time workloads.
I believe the series has now converged and can be considered for merging. The implementation has been deliberately kept simple. It performs well for real-life workloads on PowerPC, and earlier KVM testing also showed improvements on s390 and x86.
The following work is planned post the merge:
- Design for arch specific interface which could help to bring out additional performance gains.
- A test framework for the feature
Feature has been kept as generic solution for all arch which support paravirtualization and steal time accounting. If a need arises, arch specific gating could be introduced.
Special thanks to Yury Norov for the rigorous reviews that greatly improved the series and to everyone who have provided their review comments so far. (including sashiko) Really appreciated! _/\_
Rest of cover-letter is very similar since v8. Version history has been updated.
Detailed Background and Problem Statement
=========================================
As hardware scales, the density of physical CPUs (pCPUs) per server is increasing across many architectures. On these massive systems, deploying a single bare-metal OS for general workloads becomes increasingly difficult to manage if not impossible. The natural shift is to deploy Virtual Machines(VMs). For example, on IBM PowerPC architecture customers frequently deploy Shared Processor LPARs (SPLPARs) to maximize hardware ROI.
Typical enterprise workloads are combination of bursty and long running; their average CPU utilization is low, but they require high core counts during peak transactions. To accommodate this, customers often use CPU overcommit strategies i.e. configuring VMs with a large number of virtual CPUs (vCPUs) while backing them with a smaller, shared pool of physical CPUs (pCPUs). This achieves a high server consolidation and excellent cost efficiency.
However, when multiple such VMs have high utilization simultaneously, the shared pCPU pool becomes contended. The hypervisor is forced to preempt one vCPU to run another to maintain fairness. It maybe schedule vCPU of same VM or different VM. If a vCPU is preempted while holding a lock or irq disabled section etc, overall forward progress collapses. There are some mitigation strategies such as yielding the vCPU to lock-holder, but they don't cover all the cases. In addition there are hidden costs such as cache, tlb misses, cost of vCPU preemption, host scheduling overheads etc.
Under heavy contention, an effective mitigation strategy for workloads sensitive to vCPU preemption is for guests/VMs to voluntarily fold their workloads onto a smaller subset of vCPUs. By demanding fewer pCPUs, the VMs reduce overall host contention, which decreases vCPU preemption and improves total throughput for the system.
Limitations of Existing Approaches
==================================
CPU Hotplug, Isolated cpusets, cpuset:
- This is a heavy and administrative operation that requires topology rebuild. Crucially, it breaks userspace CPU affinities.
Explicit task affinity:
- Very difficult to manage for the users, if not impossible.
We need a fast, co-operative backoff mechanism inside the kernel that can dynamically react to contention without violating user/task affinity contracts. Since reacting to the contention is agnostic to the user it cannot violate user affinity contracts.
When there is high contention, fold the workload and use limited vCPUs and when there is no contention, use all the vCPUs again. This natural expansion/contraction gives the better performance to the users based on the underlying contention.
Proposed Architecture
=====================
Current design is built on basis that contention is effectively quantified by steal time as seen in guest kernel.
Steal time is already a well established construct today in para-virtualization world. All major archs support this feature. It is indication of the contention of physical CPU. It scales according to the amount of contention. Today it is used by administrative users for changing the VM configurations. During high contention the steal time shows up in each guest based on its configuration. The proposed solution works well when all VMs honor the hint and work in co-operative manner. Note there is still no inter-guest communication to achieve this co-operation. Read the section on best practises on how to get the best out of this solution.
This series introduces a dynamic vCPU backoff mechanism. It is separated into a core scheduler mechanism and a loadable virtualization policy module.
Layer A: The Scheduler Mechanism (preferred CPUs)
=================================================
Series introduces a new CPU state called preferred. It indicates that vCPU can be safely used and using that vCPU won't increase contention for underlying physical CPUs. This state info is made available via cpu_preferred_mask, which is strictly maintained as a subset of cpu_active_mask.
The scheduler uses this mask as a hint to fold workloads onto preferred CPUs using a few mechanisms.
1. Wakeup: is_cpu_allowed() checks if CPU is preferred. If not calls select_fallback_rq, which selects a preferred CPU if tasks's affinity permits.
2. The Tick (Push): During sched_tick(), if the current CPU is non-preferred, the scheduler actively pushes the running task onto a preferred CPU using a stopper thread.
3. Load Balance: sched_balance_rq restricts its domain span to cpu_preferred_mask, preventing tasks from being pulled toward non-preferred CPUs.
Design Constraint: The scheduler strictly respects user affinities. If a task is pinned exclusively to non-preferred CPUs, it will remain there. The kernel will not break user/task affinity contracts.
Layer B: The Policy Engine (virt/steal_governor)
================================================
The core scheduler should not dictate virtualization policy. Therefore, the policy is isolated into a new driver: steal_governor. (Can be selected by CONFIG_STEAL_GOVERNOR)
This module latches onto that concept that contention is quantified by steal time. It periodically samples the steal time values across the system and depending on high/low steal values, takes appropriate action.
When it sees high steal times, i.e. steal time exceeds high_threshold (default 5%), driver reduces the preferred CPUs by 1 core. When it sees Low Steal Times, i.e. steal time drops below low_threshold (default 2%), driver increases the preferred CPUs by 1 core.
This creates a dynamic, self-maintained stepwise loop. The guest automatically shrinks its pCPU footprint when the host is saturated, and expands it when the noise clears while requiring zero cross-VM communication.
Policy Design Constraints:
- Ensure at least one core is kept as preferred.
- Ensure preferred is always subset of active.
Best Practises
==============
1. Ensure all the VM run kernel which has the patches.
2. Keep CONFIG_STEAL_GOVERNOR=m. Build it as module, but don't load it by default. When the administrative user enables it in one VM, he/she will likely enable it in all VMs. Also module parameters can only be changed at module load. Having it as module also allows one to disable it to remove additional overhead it brings.
3. Keep the interval_ms=500 to 5000. I.e. between 500ms to 5 second. Though parameters allows slightly higher range.
4. Fine tune low and high threshold depending on your platform for best results. Even where is no contention, very small steal values might show up. So it might be better to keep low threshold higher than 0.
Baseline and Revision History
==============================
tip/sched/core at commit:
'627ea30aca3b ("sched/wait: Clarify WF_SYNC wakeup semantics")'
For the talk on the problem and discussion on this issue, one can also refer to the OSPM26 talk[1].
[1]: https://youtu.be/adxUKFPlOp0
[2]: https://www.ibm.com/support/pages/ibm-power-virtualization-best-practices-guide
[3]: https://www.ibm.com/docs/en/linux-on-systems?topic=bad-daytrader
v13->v14:
- Make sched_non_preferred_cpu_push_stop look similar to __balance_push_cpu_stop. (Peter Zijlstra)
- Rename task_can_sched_on_preferred to task_can_migrate_to_preferred to make it read more natural. (Peter Zijlstra)
- Clarify that the new migration statistic is per-task and does not require a schedstat version change.
- Updated changelogs and Collected tags.
v12->v13:
- Rename push_task_work_done to npc_push_work_pending. (Yury Norov)
- Move it around steal_* fields in rq. (Yury Norov)
It doesn't increase the struct rq's number of cachelines for both 128 byte and 64 byte cachelines.
- Put PREFERRED_CPU guard for npc_push_work_pending (Yury Norov)
- Update comments around rq lock's acquisition post select_fallback_rq (Yury Norov)
v11->v12:
- Simplify the ARM64's 32 bit task case where p->cpus_ptr may contain architecturally impossible CPUs. (Dietmar Eggemann, Vincent Guittot)
- Introduce cpumask_intersects_and for that simplification (Yury Norov)
- Keep the feature without any arch specific kconfig gating for now. (Vincent Guittot)
If a need arises, it can be brought in
v10->v11:
- Return active CPUs instead of possible CPUs for steal ratio denominator. This helps to avoid dilution of thresholds.
(Yury Norov, Ionut Nechita)
- Add XEN dom0 blocking check. (Ionut Nechita)
- Update documentation/changelogs.
- Multi arch framework has been deferred. Will be designed after this series gets merged upstream. It would depend on enhancement requests.
- Collected tags for patch 1
- Fix possible crash for 32-bit tasks on arm64 (sashiko)
- Fix possible race for task affinity due to abort in __migrate_task (sashiko)
- Fix unused variable for patch 1 (sashiko)
- Disable interrupt when taking rq lock in a small window (sashiko)
- Queue stopper iff migration is enabled. (sashiko)
- Document workload characteristics and the possible regression for pure CPU-time workloads.
v9->v10:
- Introduce kcpustat_field_total helper. (Yury Norov)
- Always do the design checks. This helps to avoid placing design constraints in core hotplug code.
- Remove cpu_preferred check in idle balancing. This helps to naturally take care update of nohz.next_balance.
- find_new_ilb changes are deferred as it isn't applicable for most common use cases.
- Move scheduler documentation to sched-paravirt.rst. (Yury Norov)
- Add details of limitation of default values in documentation. (Yury Norov)
- Remove task_can_sched_on_preferred out of sched.h (Mete Durlu)
- Updated suggested-by tags for few patches. (I know i should have done it earlier, sorry about that)
- Minor polish of all changelogs.
v8->v9:
- Move to simpler layout. Everything in drivers/virt/steal_governor.c (Yury Norov)
- Move design checks into a helper function (Yury Norov)
- Comments update (Yury Norov)
- Requeue work without further checks when steal ratio is within the low/high threshold window. (Yury Norov)
- Renamed sg_core_ctx to sg_ctx.
v7->v8:
- Rename to STEAL_GOVERNOR from STEAL_MONITOR.
- Remove additional defaults.c and move it to core.c (Yury Norov)
- Remove SM_DIR gating for direction control. (Yury Norov)
- Enforce design constraint and restore the state if not met (Yury Norov)
- Drop nohz_full tick enable patch.
- Move Kconfig patch as the last patch for enablement. (Yury Norov)
- Use disable_delayed_work_sync to avoid race condition during module unload. (Yury Norov)
- Add same kconfig dependency and fail to compile the driver (Yury Norov)
- Make low < high comparison during module init instead as they are dependent parameters (Sashiko)
- Update sysfs file helper section (Yury Norov)
- Make preferred sysfs file available only with CONFIG_PREFERRED_CPU=y (Yury Norov)
- A few documentation and comments fixes. (Randy Dunlap)
- Fix possible race in sched_push_current_non_preferred_cpu (Yury Norov)
- Move is_migration_disabled check just before actual migration.
- Make 100ms as minimal interval_ms from 10ms.
- Make helper functions static and remove from header file as there are no other callers.
- Collapse helper functions and periodic work into one patch.
Short summary on previous versions:v6->v7:
- Consolidate new driver code to 4-5 patches.
- defer the arch specific interface.
- Use possible CPUs instead of active for steal value calculations.
- Simplify is_cpu_allowed.
- Make module parameters fixed at module load
- Define CONFIG_STEAL_MONITOR and Make it select CONFIG_PREFERRED_CPU
v5->v6:
- Drop the optimization of caching the preferred state in select_fallback_rq
- Drop wakeup patch
v4->v5:
- Move the computation of steal time and decide on preferred CPU state to a driver. i.e new driver called STEAL_MONITOR
v3->v4:
- Make preferred subset of active instead of online.
- Dropped RT patch and Defer sched_ext. Support only FAIR class.
v2->v3:
- Introduce a new config CONFIG_PREFERRED_CPU
v1->v2:
- A new name - Preferred CPUs and cpu_preferred_mask
- Arch independent code. Everything happens in scheduler.
- Steal time computation is gated with sched feature STEAL_MONITOR
RFC v3-> RFC v4:
- Introduced computation of steal time in arch/powerpc.
RFC PATCH v1:
- push task mechanism.
- No steal time computation. Manual sysfs hint for preferred CPUs
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/#t
v3: https://lore.kernel.org/all/[email protected]/#r
v4: https://lore.kernel.org/all/[email protected]/#t
v5: https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/all/[email protected]/#t
v7: https://lore.kernel.org/all/[email protected]/
v8: https://lore.kernel.org/all/[email protected]/
v9: https://lore.kernel.org/all/[email protected]/
v10: https://lore.kernel.org/all/[email protected]/
v11: https://lore.kernel.org/all/[email protected]/
v12: https://lore.kernel.org/all/[email protected]/
v13: https://lore.kernel.org/all/[email protected]/
Even earlier version:
https://lore.kernel.org/all/[email protected]/
========================================
Performance Numbers (powerpc, x86, s390)
========================================
PowerPC:
===================
VM1: 60VP/30EC and VM2: 30VP/20EC
Shared physical CPU pool size: 50 Cores. Each core is SMT8.
(VP - Virtual Core, EC - Entitles Core) - PowerVM terminologies of SPLPAR[2]
Default parameter values: 1000ms, 200 low threshold, 500 high threshold Both the VMs are running the same workload. Total throughput/time of VM1+VM2 is being mentioned in all cases.
Hackbench
baseline steal_governor steal_governor
disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10 groups 5.20 | 5.40 (-3.85%) | 4.65 (+10.58%) 20 groups 11.39 | 12.01 (-5.44%) | 7.09 (+37.75%) 40 groups 20.32 | 19.80 (+2.56%) | 11.31 (+44.34%) 10 groups(-p) 2.37 | 2.26 (+4.64%) | 2.06 (+13.08%) 20 groups(-p) 3.34 | 3.28 (+1.80%) | 3.20 (+4.19%) 40 groups(-p) 4.46 | 4.83 (-8.30%) | 4.26 (+4.48%)
Remarks: Net improvement with steal_governor specially high load points.
schbench ( -L -n 0 -r 30 -s 0)
baseline steal_governor steal_governor
disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-m 1 -t 128 2475162 | 2621246 (+5.90%) | 2527299 (+2.11%) -m 1 -t 256 1467350 | 1470032 (+0.18%) | 1492372 (+1.71%) -m 1 -t 512 1408813 | 1454687 (+3.26%) | 1437605 (+2.04%)
Remarks: Effectively means no-improvements or regressions
kernbench baseline steal_governor steal_governor
(elapsed time) disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-j nr_cpus 231 | 235 (-1.7%) | 199 (+14%)
Remarks: Net improvement in elapsed time.
Daytrader - A real life work which is a proxy for trading based on db2[3]
baseline steal_governor steal_governor
disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Load@30% 1x | 0.96x | 1.53x Load@60% 1x | 0.94x | 1.41x
Remarks: Very good improvement seen at different load points.
phoronix/memcached:
Cloned from github.com/phoronix-test-suite/phoronix-test-suite.git GET 1:1000 means 1 writer and 1000 readers.
baseline steal_governor steal_governor
disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
GET 1:1000 1x 0.99x 1.2x
Remarks: 20% improvement with read heavy variant.
Other variants such as GET 1:1 showed minor regressions. While inspecting it, it was found that steal time reported by PowerVM was not as expected even when only 1 VM was running. This is being looked into by PowerVM hypervisor team. So added a section in limitation that, this feature expects steal time values being accurate for right policy decisions.
When there is no steal time (such as dedicated LPAR) throughput was same with steal_governor enabled/disabled which indicates minimal overhead of steal_governor.
=======================================================================
Data from x86,s390 KVM which Ilya Leoshkevich carried out during OSPM26 time. *This was based on v2*. The core idea remains the same, although the implementation has changed and some overhead has been removed since then.
Note: Other variations of the benchmark shows no observable difference.
x86:
====
cascade-lake: 32 threads = 16 cores
Benchmark #VMs #CPUs/VM ΔRPS (%std)
===============================================
hackbench 8 16 90.73% ± 9.97% hackbench 4 24 52.67% ± 7.43% hackbench 4 16 37.96% ± 11.19% hackbench 4 32 37.82% ± 4.38% hackbench 12 8 36.90% ± 4.74% hackbench 8 8 35.30% ± 3.61% pgbench 16 4 31.77% ± 2.44% hackbench 2 24 25.85% ± 8.63% hackbench 16 8 24.87% ± 3.46% pgbench 16 8 21.83% ± 2.20% pgbench 12 8 21.35% ± 2.15% pgbench 8 8 18.46% ± 1.01% hackbench 2 32 15.56% ± 4.53% pgbench 12 4 14.28% ± 2.04% hackbench 16 4 14.07% ± 2.90% hackbench 12 4 9.60% ± 3.49% [...] pgbench 4 8 -1.16% ± 3.60% hackbench 4 4 -1.80% ± 9.55% sysbench 12 4 -2.19% ± 0.78% pgbench 4 24 -2.43% ± 4.38% pgbench 4 32 -3.21% ± 0.79% sysbench 16 4 -3.22% ± 1.09%
S390:
=====
z16: 16 threads = 8 cores (SMT-2)
Benchmark #VMs #CPUs/VM ΔRPS (std%)
===============================================
pgbench 2 8 73.50% ± 35.91% pgbench 16 4 61.30% ± 4.09% hackbench 16 4 54.11% ± 4.38% hackbench 12 4 36.34% ± 4.63% pgbench 12 4 34.83% ± 2.57% hackbench 8 4 29.75% ± 5.86% hackbench 8 8 25.98% ± 5.09% pgbench 2 4 23.31% ± 33.44% pgbench 2 16 19.95% ± 17.12% hackbench 4 8 19.43% ± 9.33% pgbench 8 4 19.32% ± 4.50% [...] schbench 8 8 -0.79% ± 0.33% sysbench 8 8 -0.81% ± 0.39% hackbench 4 16 -1.11% ± 5.82% sysbench 8 4 -1.62% ± 0.49% sysbench 16 4 -2.70% ± 0.58% schbench 16 4 -2.73% ± 0.91% sysbench 12 4 -2.91% ± 0.61% hackbench 2 24 -4.99% ± 3.31%
Summary:
- Many improvement across archs specially with real life workloads.
- Overhead of steal_governor looks minimal when there is no steal time.
- Overhead when STEAL_GOVERNOR=n is negligible.
- Pure CPU-time workloads may see a small regression.
Testing and Validation
======================
Apart from performance, To ensure the robustness of the preferred CPU masking and push mechanisms, the following scenarios were tested:
- CPU Hotplug: bringing CPUs up/down change the preferred mask accordingly under no-contention and contention.
- Housekeeping cores: Verified with different combinations of nohz_full=<beginning, middle, end set of CPUs> to ensure that policy engine restricts to first housekeeping core in extreme cases.
- User Affinity: Confirmed that tasks explicitly pinned to non-preferred CPUs via taskset remain on their assigned CPUs.
- Affine Move: Confirmed the affinity move using "taskset -cp" happens on all combinations of non-preferred, preferred under contention.
- Affinity and hotplug: It works as expected. I.e affinity gets reset if all the CPUs of p->cpus_ptr go offline even if they are non-preferred CPUs.
- Extreme load and running threads: for example 4800 stress-ng threads on 480 CPU system and it still packs to preferred CPUs.
Known Limitations
=================
- The steal governor relies on accurate steal-time accounting. Incorrect steal-time values from the underlying architecture may lead to suboptimal policy decisions or performance regressions.
- If the workload is a pure CPU time based such as stress-ng --cpu=<> which doesn't suffer from vCPU preemption, it may not benefit from the steal governor.
Future work
===========
To keep this initial implementation clean and minimal, a few optimizations have been deferred:
- Design a selftest like framework to validate steal governor, at least for KVM setup where there is infra available to spin off guests.
- Multi arch interface where arch specific hints could be provided into the driver's helper functions.
- Push all tasks on rq: Currently, the stopper thread only pushes the current running task off a non-preferred CPU. Future optimizations may look into migrating all queued tasks on that runqueue.
- Sched Classes: This feature currently only works for the FAIR class. Real-time (RT) and sched_ext classes are deferred for now, as there is no need for it.
- NUMA Splicing: The steal_governor currently removes last active core based on CPU number. It does not yet do complex NUMA-aware splicing, expecting that CPUs are spread out uniformly across nodes in most cases.
Shrikanth Hegde (13):
sched/cputime: Add kcpustat_field_total helper
cpumask: Introduce cpumask_intersects_and
sched/docs: Document cpu_preferred_mask and Preferred CPU concept
cpumask: Introduce cpu_preferred_mask
sysfs: Add preferred CPU file
sched/core: Try to use a preferred CPU in is_cpu_allowed sched/fair: Load balance only among preferred CPUs sched/core: Push current task from non preferred CPU sched/debug: Add migration stats due to non preferred CPUs
virt: Introduce steal governor driver virt/steal_governor: Add control knobs for handling steal values virt/steal_governor: Implement steal_governor policy loop virt/steal_governor: Enable the driver
.../ABI/testing/sysfs-devices-system-cpu | 14 + Documentation/driver-api/index.rst | 1 + Documentation/driver-api/steal-governor.rst | 151 +++++++++ Documentation/scheduler/index.rst | 1 + Documentation/scheduler/sched-paravirt.rst | 67 ++++ MAINTAINERS | 9 + arch/s390/kernel/hiperdispatch.c | 10 +- drivers/base/cpu.c | 12 + drivers/virt/Kconfig | 17 + drivers/virt/Makefile | 1 + drivers/virt/steal_governor.c | 296 ++++++++++++++++++ fs/proc/uptime.c | 6 +- include/linux/bitmap.h | 14 + include/linux/cpumask.h | 42 +++ include/linux/kernel_stat.h | 11 + include/linux/sched.h | 1 + kernel/Kconfig.preempt | 4 + kernel/cpu.c | 6 + kernel/sched/core.c | 123 +++++++- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 8 +- kernel/sched/sched.h | 9 + lib/bitmap.c | 17 + 23 files changed, 802 insertions(+), 19 deletions(-) create mode 100644 Documentation/driver-api/steal-governor.rst create mode 100644 Documentation/scheduler/sched-paravirt.rst create mode 100644 drivers/virt/steal_governor.c
Signature · 1 line
2.52.0
Shrikanth Hegde
maintainerPATCH v14 1/13 sched/cputime: Add kcpustat_field_total helper
Provide a new helper function which sums up a given type of cpustat over a specified cpumask.
This allows the caller's code to be simpler and avoids duplication. For example, subsequent patch in the steal governor use this exact same pattern when calculating steal time.
Number of cpus can be derived from cpumask_weight() where necessary.
--- arch/s390/kernel/hiperdispatch.c | 10 +++------- fs/proc/uptime.c | 6 +----- include/linux/kernel_stat.h | 11 +++++++++++ 3 files changed, 15 insertions(+), 12 deletions(-)
arch/s390/kernel/hiperdispatch.c +3 -7
@@ -207,16 +207,12 @@ static unsigned long hd_calculate_steal_percentage(void) {unsigned long time_delta, steal_delta, steal, percentage;static ktime_t prev; -int cpus, cpu; +int cpus;ktime_t now; - cpus =0 ; - steal =0 ; percentage =0 ; -for_each_cpu (cpu, &hd_vmvl_cpumask) { - steal +=kcpustat_cpu (cpu).cpustat[CPUTIME_STEAL ]; - cpus++; - } + steal =kcpustat_field_total (CPUTIME_STEAL , &hd_vmvl_cpumask); + cpus =cpumask_weight (&hd_vmvl_cpumask);/* * If there is no vertical medium and low CPUs steal time * is 0 as vertical high CPUs shouldn't experience steal time.
fs/proc/uptime.c +1 -5
@@ -15,12 +15,8 @@ static int uptime_proc_show(struct seq_file *m, void *v)struct timespec64 idle;u64 idle_nsec;u32 rem; -int i; - - idle_nsec =0 ; -for_each_possible_cpu (i) - idle_nsec +=kcpustat_field (CPUTIME_IDLE , i); + idle_nsec =kcpustat_field_total (CPUTIME_IDLE , cpu_possible_mask);ktime_get_boottime_ts64 (&uptime);timens_add_boottime (&uptime);
include/linux/kernel_stat.h +11 -0
@@ -196,6 +196,17 @@ static inline void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu) }#endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */ +static inline u64 kcpustat_field_total (enum cpu_usage_stat usage,const struct cpumask *cpus) +{ +u64 total =0 ; +int cpu; + +for_each_cpu (cpu, cpus) + total +=kcpustat_field (usage, cpu); + +return total; +} +extern void account_user_time (struct task_struct *,u64 );extern void account_guest_time (struct task_struct *,u64 );extern void account_system_time (struct task_struct *,int ,u64 );
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 01/13] sched/cputime: Add kcpustat_field_total helper
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] s390/hiperdispatch: TOCTOU race in hd_calculate_steal_percentage() causes artificial steal…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] s390/hiperdispatch: TOCTOU race in hd_calculate_steal_percentage() causes artificial steal percentage spikes --
commit 543d41ec158ad38d026cd1a0e58754cc9393d347
Author: Shrikanth Hegde <[email protected]>
sched/cputime: Add kcpustat_field_total helper
This patch introduces a new helper function to sum up a given cpustat type over a specified cpumask. It replaces the open-coded steal calculation in s390 hiperdispatch and idle time calculation in proc uptime to simplify the callers.
Quoted 4 lines
arch/s390/kernel/hiperdispatch.c +0
-0
[ ... ]
Quoted 7 lines
arch/s390/kernel/hiperdispatch.c +1
-1@@ -207,16 +207,12 @@ static unsigned long hd_calculate_steal_percentage(void) {unsigned long time_delta, steal_delta, steal, percentage;static ktime_t prev; -int cpus, cpu; +int cpus;ktime_t now;
arch/s390/kernel/hiperdispatch.c static unsigned long hd_calculate_steal_percentage(void) · line 215
Quoted 10 lines
arch/s390/kernel/hiperdispatch.c +2
-6- cpus =0 ; - steal =0 ; percentage =0 ; -for_each_cpu (cpu, &hd_vmvl_cpumask) { - steal +=kcpustat_cpu (cpu).cpustat[CPUTIME_STEAL ]; - cpus++; - } + steal =kcpustat_field_total (CPUTIME_STEAL , &hd_vmvl_cpumask); + cpus =cpumask_weight (&hd_vmvl_cpumask);
[Severity: High]
Could separating the summation from the divisor count open a race window against concurrent mask updates?
In the previous single-loop calculation, the total steal time and CPU count were evaluated together on a stable view of the mask. With these changes, if a concurrent CPU hotplug event calls update_cpu_masks() while hd_calculate_steal_percentage() is running, hd_vmvl_cpumask could be cleared and repopulated asynchronously.
If the mask changes from multiple CPUs to 1 CPU between the kcpustat_field_total calculation and the cpumask_weight call, steal will reflect the delta of the larger number of CPUs, but cpus will evaluate to 1.
Because the percentage is computed as steal_delta / cpus, dividing the larger steal_delta by 1 would artificially inflate the steal percentage. Can this erroneous spike trip the hd_steal_threshold, causing the scheduler topology to wrongly disable Hiperdispatch and downgrade high-capacity cores?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[tip: sched/core] sched/cputime: Add kcpustat_field_total helper
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: a8d0854a76a876ea474e17322acb680716a09da2 Gitweb: https://git.kernel.org/tip/a8d0854a76a876ea474e17322acb680716a09da2 Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:16 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:40 +02:00
sched/cputime: Add kcpustat_field_total helper
Provide a new helper function which sums up a given type of cpustat over a specified cpumask.
This allows the caller's code to be simpler and avoids duplication. For example, subsequent patch in the steal governor use this exact same pattern when calculating steal time.
Number of cpus can be derived from cpumask_weight() where necessary.
--- arch/s390/kernel/hiperdispatch.c | 10 +++------- fs/proc/uptime.c | 6 +----- include/linux/kernel_stat.h | 11 +++++++++++ 3 files changed, 15 insertions(+), 12 deletions(-)
arch/s390/kernel/hiperdispatch.c +3 -7
@@ -207,16 +207,12 @@ static unsigned long hd_calculate_steal_percentage(void) {unsigned long time_delta, steal_delta, steal, percentage;static ktime_t prev; -int cpus, cpu; +int cpus;ktime_t now; - cpus =0 ; - steal =0 ; percentage =0 ; -for_each_cpu (cpu, &hd_vmvl_cpumask) { - steal +=kcpustat_cpu (cpu).cpustat[CPUTIME_STEAL ]; - cpus++; - } + steal =kcpustat_field_total (CPUTIME_STEAL , &hd_vmvl_cpumask); + cpus =cpumask_weight (&hd_vmvl_cpumask);/* * If there is no vertical medium and low CPUs steal time * is 0 as vertical high CPUs shouldn't experience steal time.
fs/proc/uptime.c +1 -5
@@ -15,12 +15,8 @@ static int uptime_proc_show(struct seq_file *m, void *v)struct timespec64 idle;u64 idle_nsec;u32 rem; -int i; - - idle_nsec =0 ; -for_each_possible_cpu (i) - idle_nsec +=kcpustat_field (CPUTIME_IDLE , i); + idle_nsec =kcpustat_field_total (CPUTIME_IDLE , cpu_possible_mask);ktime_get_boottime_ts64 (&uptime);timens_add_boottime (&uptime);
include/linux/kernel_stat.h +11 -0
@@ -196,6 +196,17 @@ static inline void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu) }#endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */ +static inline u64 kcpustat_field_total (enum cpu_usage_stat usage,const struct cpumask *cpus) +{ +u64 total =0 ; +int cpu; + +for_each_cpu (cpu, cpus) + total +=kcpustat_field (usage, cpu); + +return total; +} +extern void account_user_time (struct task_struct *,u64 );extern void account_guest_time (struct task_struct *,u64 );extern void account_system_time (struct task_struct *,int ,u64 );
Shrikanth Hegde
maintainerPATCH v14 2/13 cpumask: Introduce cpumask_intersects_and
In reply to Shrikanth HegdeIntroduce bitmap_intersects_and() to determine whether the intersection of three bitmaps is non-empty. Unlike cpumask_first_and_and(), this returns immediately when an intersecting word is found and does not calculate the first matching bit.
Add cpumask_intersects_and() as the corresponding cpumask wrapper.
A subsequent patch uses the helper to determine whether a task has a CPU that is present in its affinity mask, the preferred CPU mask, and task possible CPU mask.
--- include/linux/bitmap.h | 14 ++++++++++++++ include/linux/cpumask.h | 18 ++++++++++++++++++ lib/bitmap.c | 17 +++++++++++++++++ 3 files changed, 49 insertions(+)
include/linux/bitmap.h +14 -0
@@ -52,6 +52,7 @@ struct device;* bitmap_complement(dst, src, nbits) *dst = ~(*src) * bitmap_equal(src1, src2, nbits) Are *src1 and *src2 equal? * bitmap_intersects(src1, src2, nbits) Do *src1 and *src2 overlap? +* bitmap_intersects_and(src1, src2, src3, nbits) Do *src1, *src2 and *src3 overlap? * bitmap_subset(src1, src2, nbits) Is *src1 a subset of *src2? * bitmap_empty(src, nbits) Are all bits zero in *src? * bitmap_full(src, nbits) Are all bits set in *src?
@@ -181,6 +182,9 @@ void __bitmap_replace(unsigned long *dst, const unsigned long *mask, unsigned int nbits);bool __bitmap_intersects (const unsigned long *bitmap1,const unsigned long *bitmap2,unsigned int nbits); +bool __bitmap_intersects_and (const unsigned long *bitmap1, +const unsigned long *bitmap2, +const unsigned long *bitmap3,unsigned int nbits);bool __bitmap_subset (const unsigned long *bitmap1,const unsigned long *bitmap2,unsigned int nbits);unsigned int __bitmap_weight (const unsigned long *bitmap,unsigned int nbits);
@@ -445,6 +449,16 @@ bool bitmap_intersects(const unsigned long *src1, const unsigned long *src2, unsreturn __bitmap_intersects (src1, src2, nbits); } +static __always_inline +bool bitmap_intersects_and(const unsigned long *src1, const unsigned long *src2, +const unsigned long *src3,unsigned int nbits) +{ +if (small_const_nbits (nbits)) +return ((*src1 & *src2 & *src3) &BITMAP_LAST_WORD_MASK (nbits)) !=0 ; +else +return __bitmap_intersects_and (src1, src2, src3, nbits); +} +static __always_inline bool bitmap_subset(const unsigned long *src1, const unsigned long *src2, unsigned int nbits) {
include/linux/cpumask.h +18 -0
@@ -824,6 +824,24 @@ bool cpumask_intersects(const struct cpumask *src1p, const struct cpumask *src2p small_cpumask_bits); } +/** +* cpumask_intersects_and - (*src1p & *src2p & *src3p) != 0 +* @src1p: the first input +* @src2p: the second input +* @src3p: the third input +* +* Return: true if AND of the three cpumasks is non-empty, +* otherwise false +*/ +static __always_inline +bool cpumask_intersects_and(const struct cpumask *src1p, +const struct cpumask *src2p, +const struct cpumask *src3p) +{ +return bitmap_intersects_and (cpumask_bits (src1p),cpumask_bits (src2p), +cpumask_bits (src3p), small_cpumask_bits); +} +/** * cpumask_subset - (*src1p & ~*src2p) == 0 * @src1p: the first input
lib/bitmap.c +17 -0
@@ -308,6 +308,23 @@ bool __bitmap_intersects(const unsigned long *bitmap1, }EXPORT_SYMBOL (__bitmap_intersects); +bool __bitmap_intersects_and (const unsigned long *bitmap1, +const unsigned long *bitmap2, +const unsigned long *bitmap3,unsigned int bits) +{ +unsigned int k, lim = bits /BITS_PER_LONG ; + +for (k =0 ; k < lim; ++k) +if (bitmap1[k] & bitmap2[k] & bitmap3[k]) +return true; + +if (bits %BITS_PER_LONG ) +if ((bitmap1[k] & bitmap2[k] & bitmap3[k]) &BITMAP_LAST_WORD_MASK (bits)) +return true; +return false; +} +EXPORT_SYMBOL (__bitmap_intersects_and); +bool __bitmap_subset (const unsigned long *bitmap1,const unsigned long *bitmap2,unsigned int bits) {
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 02/13] cpumask: Introduce cpumask_intersects_and
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines
Introduce
bitmap_intersects_and()to determine whether the intersection of three bitmaps is non-empty. Unlikecpumask_first_and_and(), this returns immediately when an intersecting word is found and does not calculate the first matching bit.Add
cpumask_intersects_and()as the corresponding cpumask wrapper.
Quoted 7 lines
A subsequent patch uses the helper to determine whether a task has a CPU that is present in its affinity mask, the preferred CPU mask, and task possible CPU mask.
Suggested-by Yury Norov <[email protected]>Reviewed-by Yury Norov <[email protected]>Signed-off-by Shrikanth Hegde <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[tip: sched/core] cpumask: Introduce cpumask_intersects_and
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: cfb463b7172dca2f29b2221b71b54890f268e3fb Gitweb: https://git.kernel.org/tip/cfb463b7172dca2f29b2221b71b54890f268e3fb Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:17 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:40 +02:00
cpumask: Introduce cpumask_intersects_and
Introduce bitmap_intersects_and() to determine whether the intersection of three bitmaps is non-empty. Unlike cpumask_first_and_and(), this returns immediately when an intersecting word is found and does not calculate the first matching bit.
Add cpumask_intersects_and() as the corresponding cpumask wrapper.
A subsequent patch uses the helper to determine whether a task has a CPU that is present in its affinity mask, the preferred CPU mask, and task possible CPU mask.
--- include/linux/bitmap.h | 14 ++++++++++++++ include/linux/cpumask.h | 18 ++++++++++++++++++ lib/bitmap.c | 17 +++++++++++++++++ 3 files changed, 49 insertions(+)
include/linux/bitmap.h +14 -0
@@ -52,6 +52,7 @@ struct device;* bitmap_complement(dst, src, nbits) *dst = ~(*src) * bitmap_equal(src1, src2, nbits) Are *src1 and *src2 equal? * bitmap_intersects(src1, src2, nbits) Do *src1 and *src2 overlap? +* bitmap_intersects_and(src1, src2, src3, nbits) Do *src1, *src2 and *src3 overlap? * bitmap_subset(src1, src2, nbits) Is *src1 a subset of *src2? * bitmap_empty(src, nbits) Are all bits zero in *src? * bitmap_full(src, nbits) Are all bits set in *src?
@@ -181,6 +182,9 @@ void __bitmap_replace(unsigned long *dst, const unsigned long *mask, unsigned int nbits);bool __bitmap_intersects (const unsigned long *bitmap1,const unsigned long *bitmap2,unsigned int nbits); +bool __bitmap_intersects_and (const unsigned long *bitmap1, +const unsigned long *bitmap2, +const unsigned long *bitmap3,unsigned int nbits);bool __bitmap_subset (const unsigned long *bitmap1,const unsigned long *bitmap2,unsigned int nbits);unsigned int __bitmap_weight (const unsigned long *bitmap,unsigned int nbits);
@@ -446,6 +450,16 @@ bool bitmap_intersects(const unsigned long *src1, const unsigned long *src2, uns }static __always_inline +bool bitmap_intersects_and(const unsigned long *src1, const unsigned long *src2, +const unsigned long *src3,unsigned int nbits) +{ +if (small_const_nbits (nbits)) +return ((*src1 & *src2 & *src3) &BITMAP_LAST_WORD_MASK (nbits)) !=0 ; +else +return __bitmap_intersects_and (src1, src2, src3, nbits); +} + +static __always_inline bool bitmap_subset(const unsigned long *src1, const unsigned long *src2, unsigned int nbits) {if (small_const_nbits (nbits))
include/linux/cpumask.h +18 -0
@@ -825,6 +825,24 @@ bool cpumask_intersects(const struct cpumask *src1p, const struct cpumask *src2p }/** +* cpumask_intersects_and - (*src1p & *src2p & *src3p) != 0 +* @src1p: the first input +* @src2p: the second input +* @src3p: the third input +* +* Return: true if AND of the three cpumasks is non-empty, +* otherwise false +*/ +static __always_inline +bool cpumask_intersects_and(const struct cpumask *src1p, +const struct cpumask *src2p, +const struct cpumask *src3p) +{ +return bitmap_intersects_and (cpumask_bits (src1p),cpumask_bits (src2p), +cpumask_bits (src3p), small_cpumask_bits); +} + +/** * cpumask_subset - (*src1p & ~*src2p) == 0 * @src1p: the first input * @src2p: the second input
lib/bitmap.c +17 -0
@@ -308,6 +308,23 @@ bool __bitmap_intersects(const unsigned long *bitmap1, }EXPORT_SYMBOL (__bitmap_intersects); +bool __bitmap_intersects_and (const unsigned long *bitmap1, +const unsigned long *bitmap2, +const unsigned long *bitmap3,unsigned int bits) +{ +unsigned int k, lim = bits /BITS_PER_LONG ; + +for (k =0 ; k < lim; ++k) +if (bitmap1[k] & bitmap2[k] & bitmap3[k]) +return true; + +if (bits %BITS_PER_LONG ) +if ((bitmap1[k] & bitmap2[k] & bitmap3[k]) &BITMAP_LAST_WORD_MASK (bits)) +return true; +return false; +} +EXPORT_SYMBOL (__bitmap_intersects_and); +bool __bitmap_subset (const unsigned long *bitmap1,const unsigned long *bitmap2,unsigned int bits) {
Shrikanth Hegde
maintainerPATCH v14 3/13 sched/docs: Document cpu_preferred_mask and Preferred CPU concept
In reply to Shrikanth HegdeAdd documentation for new CPU state called preferred CPU state and corresponding cpumask called cpu_preferred_mask.
This could help users in understanding what it is and how to use it.
Document the role of scheduler and driver for this feature to work. Details regarding the driver documentation will be added in later patches under Documentation/driver-api/steal-governor.rst.
Newly added file could be used for other paravirt usecase documentation.
--- Documentation/scheduler/index.rst | 1 + Documentation/scheduler/sched-paravirt.rst | 67 ++++++++++++++++++++++ 2 files changed, 68 insertions(+) create mode 100644 Documentation/scheduler/sched-paravirt.rst
Documentation/scheduler/index.rst +1 -0
@@ -23,5 +23,6 @@ Scheduler sched-stats sched-ext sched-debug + sched-paravirt text_files
Documentation/scheduler/sched-paravirt.rst new file +67 -0
67 lines not on this page · raw message
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 03/13] sched/docs: Document cpu_preferred_mask and Preferred CPU concept
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Add documentation for new CPU state called preferred CPU state and corresponding cpumask called
cpu_preferred_mask.
This could help users in understanding what it is and how to use it.Document the role of scheduler and driver for this feature to work. Details regarding the driver documentation will be added in later patches under
Documentation/driver-api/steal-governor.rst.Newly added file could be used for other paravirt usecase documentation.
Signed-off-by Shrikanth Hegde <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[tip: sched/core] sched/docs: Document cpu_preferred_mask and Preferred CPU concept
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 06a49ef784acf91cee78d50c5307d400f86ccafb Gitweb: https://git.kernel.org/tip/06a49ef784acf91cee78d50c5307d400f86ccafb Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:18 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:40 +02:00
sched/docs: Document cpu_preferred_mask and Preferred CPU concept
Add documentation for new CPU state called preferred CPU state and corresponding cpumask called cpu_preferred_mask.
This could help users in understanding what it is and how to use it.
Document the role of scheduler and driver for this feature to work. Details regarding the driver documentation will be added in later patches under Documentation/driver-api/steal-governor.rst.
Newly added file could be used for other paravirt usecase documentation.
--- Documentation/scheduler/index.rst | 1 +- Documentation/scheduler/sched-paravirt.rst | 67 +++++++++++++++++++++- 2 files changed, 68 insertions(+) create mode 100644 Documentation/scheduler/sched-paravirt.rst
Documentation/scheduler/index.rst +1 -0
@@ -23,5 +23,6 @@ Scheduler sched-stats sched-ext sched-debug + sched-paravirt text_files
Documentation/scheduler/sched-paravirt.rst new file +67 -0
67 lines not on this page · raw message
Shrikanth Hegde
maintainerPATCH v14 4/13 cpumask: Introduce cpu_preferred_mask
In reply to Shrikanth HegdeProvide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.
CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.
A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.
Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.
--- include/linux/cpumask.h | 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt | 4 ++++ kernel/cpu.c | 6 ++++++ kernel/sched/core.c | 5 +++++ 4 files changed, 39 insertions(+)
include/linux/cpumask.h +24 -0
@@ -121,12 +121,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; + +#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif +#define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)#define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask)#define cpu_enabled_mask ((const struct cpumask *)&__cpu_enabled_mask)#define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask)#define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask)#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) +#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)extern atomic_t __num_online_cpus;extern unsigned int __num_possible_cpus;
@@ -1181,6 +1189,12 @@ void init_cpu_possible(const struct cpumask *src);#define set_cpu_active (cpu, active) assign_cpu((cpu), &__cpu_active_mask, (active))#define set_cpu_dying (cpu, dying) assign_cpu((cpu), &__cpu_dying_mask, (dying)) +#ifdef CONFIG_PREFERRED_CPU +#define set_cpu_preferred (cpu, preferred) assign_cpu((cpu), &__cpu_preferred_mask, (preferred)) +#else +#define set_cpu_preferred (cpu, preferred) do { } while (0) +#endif +void set_cpu_online (unsigned int cpu,bool online);void set_cpu_possible (unsigned int cpu,bool possible);
@@ -1275,6 +1289,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)return cpumask_test_cpu (cpu, cpu_dying_mask); } +static __always_inline bool cpu_preferred(unsigned int cpu) +{ +return cpumask_test_cpu (cpu, cpu_preferred_mask); +} +#else #define num_online_cpus () 1U
@@ -1313,6 +1332,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)return false; } +static __always_inline bool cpu_preferred(unsigned int cpu) +{ +return cpu ==0 ; +} +#endif /* NR_CPUS > 1 */ #define cpu_is_offline (cpu) unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +4 -0
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT For more information: Documentation/scheduler/sched-ext.rst https://github.com/sched-ext/scx + +config PREFERRED_CPU + bool + depends on SMP && PARAVIRT
kernel/cpu.c +6 -0
@@ -3103,6 +3103,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);atomic_t __num_online_cpus__read_mostly ;EXPORT_SYMBOL (__num_online_cpus); +#ifdef CONFIG_PREFERRED_CPU +struct cpumask __cpu_preferred_mask__read_mostly ; +EXPORT_SYMBOL_GPL (__cpu_preferred_mask); +#endif +void init_cpu_present (const struct cpumask *src) {cpumask_copy (&__cpu_present_mask, src);
@@ -3160,6 +3165,7 @@ void __init boot_cpu_init(void)/* Mark the boot cpu "present", "online" etc for SMP and UP case */ set_cpu_online (cpu, true);set_cpu_active (cpu, true); +set_cpu_preferred (cpu, true);set_cpu_present (cpu, true);set_cpu_possible (cpu, true);
kernel/sched/core.c +5 -0
@@ -8580,6 +8580,9 @@ int sched_cpu_activate(unsigned int cpu)*/ sched_set_rq_online (rq, cpu); +/* preferred is subset of active and follows its state */ +set_cpu_preferred (cpu, true); +return 0 ; }
@@ -8593,6 +8596,8 @@ int sched_cpu_deactivate(unsigned int cpu)if (ret)return ret; +set_cpu_preferred (cpu, false); +/* * Remove CPU from nohz.idle_cpus_mask to prevent participating in * load balancing when not active
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 12 lines
Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.
CONFIG_PREFERRED_CPUwill be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be calledsteal_governorand it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the
steal_governor.
Quoted 7 lines
Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
With
CONFIG_PREFERRED_CPU=n, ensureset_cpu_preferredis a nop and get method returns the active state in that case.Signed-off-by Shrikanth Hegde <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[tip: sched/core] cpumask: Introduce cpu_preferred_mask
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 518b32bd5bb373aa334e4b73d359689071d50b5c Gitweb: https://git.kernel.org/tip/518b32bd5bb373aa334e4b73d359689071d50b5c Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:19 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:41 +02:00
cpumask: Introduce cpu_preferred_mask
Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.
CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.
A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.
Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.
--- include/linux/cpumask.h | 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt | 4 ++++ kernel/cpu.c | 6 ++++++ kernel/sched/core.c | 5 +++++ 4 files changed, 39 insertions(+)
include/linux/cpumask.h +24 -0
@@ -121,12 +121,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; + +#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif +#define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)#define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask)#define cpu_enabled_mask ((const struct cpumask *)&__cpu_enabled_mask)#define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask)#define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask)#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) +#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)extern atomic_t __num_online_cpus;extern unsigned int __num_possible_cpus;
@@ -1181,6 +1189,12 @@ void init_cpu_possible(const struct cpumask *src);#define set_cpu_active (cpu, active) assign_cpu((cpu), &__cpu_active_mask, (active))#define set_cpu_dying (cpu, dying) assign_cpu((cpu), &__cpu_dying_mask, (dying)) +#ifdef CONFIG_PREFERRED_CPU +#define set_cpu_preferred (cpu, preferred) assign_cpu((cpu), &__cpu_preferred_mask, (preferred)) +#else +#define set_cpu_preferred (cpu, preferred) do { } while (0) +#endif +void set_cpu_online (unsigned int cpu,bool online);void set_cpu_possible (unsigned int cpu,bool possible);
@@ -1275,6 +1289,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)return cpumask_test_cpu (cpu, cpu_dying_mask); } +static __always_inline bool cpu_preferred(unsigned int cpu) +{ +return cpumask_test_cpu (cpu, cpu_preferred_mask); +} +#else #define num_online_cpus () 1U
@@ -1313,6 +1332,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)return false; } +static __always_inline bool cpu_preferred(unsigned int cpu) +{ +return cpu ==0 ; +} +#endif /* NR_CPUS > 1 */ #define cpu_is_offline (cpu) unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +4 -0
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT For more information: Documentation/scheduler/sched-ext.rst https://github.com/sched-ext/scx + +config PREFERRED_CPU + bool + depends on SMP && PARAVIRT
kernel/cpu.c +6 -0
@@ -3103,6 +3103,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);atomic_t __num_online_cpus__read_mostly ;EXPORT_SYMBOL (__num_online_cpus); +#ifdef CONFIG_PREFERRED_CPU +struct cpumask __cpu_preferred_mask__read_mostly ; +EXPORT_SYMBOL_GPL (__cpu_preferred_mask); +#endif +void init_cpu_present (const struct cpumask *src) {cpumask_copy (&__cpu_present_mask, src);
@@ -3160,6 +3165,7 @@ void __init boot_cpu_init(void)/* Mark the boot cpu "present", "online" etc for SMP and UP case */ set_cpu_online (cpu, true);set_cpu_active (cpu, true); +set_cpu_preferred (cpu, true);set_cpu_present (cpu, true);set_cpu_possible (cpu, true);
kernel/sched/core.c +5 -0
@@ -8632,6 +8632,9 @@ int sched_cpu_activate(unsigned int cpu)*/ sched_set_rq_online (rq, cpu); +/* preferred is subset of active and follows its state */ +set_cpu_preferred (cpu, true); +return 0 ; }
@@ -8645,6 +8648,8 @@ int sched_cpu_deactivate(unsigned int cpu)if (ret)return ret; +set_cpu_preferred (cpu, false); +/* * Remove CPU from nohz.idle_cpus_mask to prevent participating in * load balancing when not active
Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask
In reply to Shrikanth HegdeQuoted 25 lines · Shrikanth Hegde
Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.
CONFIG_PREFERRED_CPUwill be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be calledsteal_governorand it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the
steal_governor.Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
With
CONFIG_PREFERRED_CPU=n, ensureset_cpu_preferredis a nop and get method returns the active state in that case.Signed-off-by Shrikanth Hegde <[email protected]>--- include/linux/cpumask.h | 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt | 4 ++++ kernel/cpu.c | 6 ++++++ kernel/sched/core.c | 5 +++++ 4 files changed, 39 insertions(+)
[..snip..]
Quoted 7 lines
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))kernel/Kconfig.preempt +0
-0
kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200
Quoted 7 lines
kernel/Kconfig.preempt +4
-0@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT For more information: Documentation/scheduler/sched-ext.rst https://github.com/sched-ext/scx + +config PREFERRED_CPU + bool + depends on SMP && PARAVIRT
FWICT, PREFERRED_CPU mechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummy paravirt_steal_clock() just for that.
Shrikanth, would it be possible to drop that dependency?
Thanks!
-Mete
Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask
Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde
Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.
CONFIG_PREFERRED_CPUwill be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be calledsteal_governorand it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the
steal_governor.Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
With
CONFIG_PREFERRED_CPU=n, ensureset_cpu_preferredis a nop and get method returns the active state in that case.Signed-off-by Shrikanth Hegde <[email protected]>--- include/linux/cpumask.h | 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt | 4 ++++ kernel/cpu.c | 6 ++++++ kernel/sched/core.c | 5 +++++ 4 files changed, 39 insertions(+)[..snip..]
kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200
Quoted 13 lines
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))kernel/Kconfig.preempt +4
-0@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT For more information: Documentation/scheduler/sched-ext.rst https://github.com/sched-ext/scx + +config PREFERRED_CPU + bool + depends on SMP && PARAVIRTFWICT,
PREFERRED_CPUmechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummyparavirt_steal_clock()just for that.
Quoted 1 line
Shrikanth, would it be possible to drop that dependency?
Do you have a non-paravirt case in mind?
I'm actually about to submit an RFC that drops the PARAVIRT dependency and provides write access to the preferred mask. But I want to make sure that we need it, and we've got no reasonable mechanism to put unneeded CPUs on idle already.
Simply relaxing the PARAVIRT dependency without a solid use case would bloat configuration space for nothing. If you've got such a case in mind, the rest is trivial.
Thanks,
Yury
Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask
Quoted 47 lines · Yury Norov
Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde
Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.
CONFIG_PREFERRED_CPUwill be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be calledsteal_governorand it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the
steal_governor.Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
With
CONFIG_PREFERRED_CPU=n, ensureset_cpu_preferredis a nop and get method returns the active state in that case.Signed-off-by Shrikanth Hegde <[email protected]>--- include/linux/cpumask.h | 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt | 4 ++++ kernel/cpu.c | 6 ++++++ kernel/sched/core.c | 5 +++++ 4 files changed, 39 insertions(+)[..snip..]
kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200
Quoted 13 lines
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))kernel/Kconfig.preempt +4
-0@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT For more information: Documentation/scheduler/sched-ext.rst https://github.com/sched-ext/scx + +config PREFERRED_CPU + bool + depends on SMP && PARAVIRTFWICT,
PREFERRED_CPUmechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummyparavirt_steal_clock()just for that.
Quoted 3 lines
Quoted 1 line
Shrikanth, would it be possible to drop that dependency?
Do you have a non-paravirt case in mind?
s390 LPARs are exactly that. s390 does not select PARAVIRT by default but enabling PREFERRED_CPU is strongly considered.
I removed the PARAVIRT dependency and gave the whole thing a quick spin. Builds and functions w/o any issues on s390!
Quoted 4 lines
I'm actually about to submit an RFC that drops the PARAVIRT dependency and provides write access to the preferred mask. But I want to make sure that we need it, and we've got no reasonable mechanism to put unneeded CPUs on idle already.
You can mention s390 LPARs as valid case on your RFC if need be. Thank you!
Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask
Quoted 55 lines · Mete Durlu
Quoted 47 lines · Yury Norov
Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde
Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.
CONFIG_PREFERRED_CPUwill be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be calledsteal_governorand it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the
steal_governor.Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
With
CONFIG_PREFERRED_CPU=n, ensureset_cpu_preferredis a nop and get method returns the active state in that case.Signed-off-by Shrikanth Hegde <[email protected]>--- include/linux/cpumask.h | 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt | 4 ++++ kernel/cpu.c | 6 ++++++ kernel/sched/core.c | 5 +++++ 4 files changed, 39 insertions(+)[..snip..]
kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200
Quoted 13 lines
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))kernel/Kconfig.preempt +4
-0@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT For more information: Documentation/scheduler/sched-ext.rst https://github.com/sched-ext/scx + +config PREFERRED_CPU + bool + depends on SMP && PARAVIRTFWICT,
PREFERRED_CPUmechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummyparavirt_steal_clock()just for that.Quoted 3 lines
Quoted 1 line
Shrikanth, would it be possible to drop that dependency?
Do you have a non-paravirt case in mind?
s390 LPARs are exactly that. s390 does not select PARAVIRT by default but enabling
PREFERRED_CPUis strongly considered.
Quoted 8 lines
I removed the PARAVIRT dependency and gave the whole thing a quick spin. Builds and functions w/o any issues on s390!
Quoted 4 lines
I'm actually about to submit an RFC that drops the PARAVIRT dependency and provides write access to the preferred mask. But I want to make sure that we need it, and we've got no reasonable mechanism to put unneeded CPUs on idle already.
You can mention s390 LPARs as valid case on your RFC if need be.
Sure, will do. I'm about submitting this before the end of week; will add you in CC.
Thanks,
Yury
Shrikanth Hegde
maintainerRe: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask
In reply to Mete DurluHi Mete, Yury,
Quoted 48 lines · Mete Durlu
Quoted 47 lines · Yury Norov
Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde
Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.
CONFIG_PREFERRED_CPUwill be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be calledsteal_governorand it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the
steal_governor.Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
With
CONFIG_PREFERRED_CPU=n, ensureset_cpu_preferredis a nop and get method returns the active state in that case.Signed-off-by Shrikanth Hegde <[email protected]>---
include/linux/cpumask.h| 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt | 4 ++++ kernel/cpu.c | 6 ++++++ kernel/sched/core.c | 5 +++++4 files changed, 39 insertions(+)[..snip..]
kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200
Quoted 13 lines
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +0
-0@@ -194,3 +194,7 @@ config SCHED_CLASS_EXTFor more information:
Documentation/scheduler/sched-ext.rst
https://github.com/sched-ext/scxkernel/Kconfig.preempt +4
-0+ +config PREFERRED_CPU + bool + depends on SMP && PARAVIRTFWICT,
PREFERRED_CPUmechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummyparavirt_steal_clock()just for that.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Shrikanth, would it be possible to drop that dependency?
Do you have a non-paravirt case in mind?
s390 LPARs are exactly that. s390 does not select PARAVIRT by default but enabling
PREFERRED_CPUis strongly considered.
s390 may not have PARAVIRT=y, but this use case is still a virtualizated system no? i.e you are observing steal time. By definition of steal time, it is a para- virtualized system. Though you may be deriving the steal time not using the paravirt_steal_clock.
Or my understanding here is wrong?
Quoted 8 lines
I removed the PARAVIRT dependency and gave the whole thing a quick spin. Builds and functions w/o any issues on s390!
Quoted 4 lines
I'm actually about to submit an RFC that drops the PARAVIRT dependency and provides write access to the preferred mask. But I want to make sure that we need it, and we've got no reasonable mechanism to put unneeded CPUs on idle already.
We might run into few corner cases quickly with write allowed. But yes, this is a simpler mechanism not to run anything on few CPUs in a dynamic way.
Quoted 3 lines
You can mention s390 LPARs as valid case on your RFC if need be. Thank you!
Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask
Quoted 51 lines · Shrikanth Hegde
Hi Mete, Yury,
Quoted 48 lines · Mete Durlu
Quoted 47 lines · Yury Norov
Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde
Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.
CONFIG_PREFERRED_CPUwill be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be calledsteal_governorand it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the
steal_governor.Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
With
CONFIG_PREFERRED_CPU=n, ensureset_cpu_preferredis a nop and get method returns the active state in that case.Signed-off-by Shrikanth Hegde <[email protected]>---
include/linux/cpumask.h| 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt | 4 ++++ kernel/cpu.c | 6 ++++++ kernel/sched/core.c | 5 +++++4 files changed, 39 insertions(+)[..snip..]
kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200
Quoted 13 lines
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +0
-0@@ -194,3 +194,7 @@ config SCHED_CLASS_EXTFor more information:
Documentation/scheduler/sched-ext.rst
https://github.com/sched-ext/scxkernel/Kconfig.preempt +4
-0+ +config PREFERRED_CPU + bool + depends on SMP && PARAVIRTFWICT,
PREFERRED_CPUmechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummyparavirt_steal_clock()just for that.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Shrikanth, would it be possible to drop that dependency?
Do you have a non-paravirt case in mind?
s390 LPARs are exactly that. s390 does not select PARAVIRT by default but enabling
PREFERRED_CPUis strongly considered.s390 may not have PARAVIRT=y, but this use case is still a virtualizated system no?
Yes, it is virtualized.
Quoted 6 lines
i.e you are observing steal time. By definition of steal time, it is a para- virtualized system. Though you may be deriving the steal time not using the
paravirt_steal_clock.Or my understanding here is wrong?
So you propose s390 to enable CONFIG_PARAVIRT *just because* s390 fits the "paravirtualized system" definition, even if s390 would not utilize any functionality comes with the config option? (even worse it comes with deadcode...)
That sounds a bit wrong to me, I would really like to avoid enabling a config option just because it's name suggests that it is related to our platform.
Yury already sent the patch series he mentioned. I suppose we should keep discussing this there.
Thank you!
-Mete
Shrikanth Hegde
maintainerPATCH v14 5/13 sysfs: Add preferred CPU file
In reply to Shrikanth HegdeAdd a "preferred" file in /sys/devices/system/cpu/ when kernel is built with CONFIG_PREFERRED_CPU=y.
This would help
- Users to quickly check which CPUs are marked as preferred.
- Userspace daemons such as irqbalance to use this mask to route irqs into preferred CPUs.
For example:
cat /sys/devices/system/cpu/online
0-719
cat /sys/devices/system/cpu/preferred
0-599 <<< Implies 0-599 are preferred for workloads and 600-719
should be avoided at this moment.
cat /sys/devices/system/cpu/preferred
0-719 <<< All CPUs are usable. There is no preference.
--- Documentation/ABI/testing/sysfs-devices-system-cpu | 14 ++++++++++++++ drivers/base/cpu.c | 12 ++++++++++++ 2 files changed, 26 insertions(+)
Documentation/ABI/testing/sysfs-devices-system-cpu +14 -0
@@ -806,3 +806,17 @@ Date: Nov 2022 Contact: Linux kernel mailing list <[email protected]> Description: (RO) the list of CPUs that can be brought online. + +What: /sys/devices/system/cpu/preferred +Date: Sep 2026 +Contact: Linux kernel mailing list <[email protected]> +Description: + (RO) the list of preferred CPUs applicable in + paravirtualized environments. + + The steal governor driver dynamically adjusts this mask + based on observed steal time. Scheduling tasks on + CPUs outside of this list may lead to performance + degradations due to underlying physical CPU contention. + + See Documentation/scheduler/sched-paravirt.rst for more details.
drivers/base/cpu.c +12 -0
@@ -391,6 +391,15 @@ static int cpu_uevent(const struct device *dev, struct kobj_uevent_env *env) }#endif +#ifdef CONFIG_PREFERRED_CPU +static ssize_t preferred_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +return sysfs_emit (buf,"%*pbl\n" ,cpumask_pr_args (cpu_preferred_mask)); +} +static DEVICE_ATTR_RO (preferred); +#endif +const struct bus_type cpu_subsys = { .name ="cpu" , .dev_name ="cpu" ,
@@ -531,6 +540,9 @@ static struct attribute *cpu_root_attrs[] = {#endif #ifdef CONFIG_GENERIC_CPU_AUTOPROBE &dev_attr_modalias.attr, +#endif +#ifdef CONFIG_PREFERRED_CPU + &dev_attr_preferred.attr, #endifNULL };
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 05/13] sysfs: Add preferred CPU file
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines
Add a "preferred" file in
/sys/devices/system/cpu/when kernel is built withCONFIG_PREFERRED_CPU=y.This would help
- Users to quickly check which CPUs are marked as preferred.
- Userspace daemons such as irqbalance to use this mask to route irqs into preferred CPUs.For example:
cat/sys/devices/system/cpu/online
0-719
cat/sys/devices/system/cpu/preferred
0-599 <<< Implies 0-599 are preferred for workloads and 600-719
should be avoided at this moment.
Quoted 4 lines
cat
/sys/devices/system/cpu/preferred
0-719 <<< All CPUs are usable. There is no preference.Signed-off-by Shrikanth Hegde <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
[tip: sched/core] sysfs: Add preferred CPU file
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 62082451655747dae25496d126e58f985df29f50 Gitweb: https://git.kernel.org/tip/62082451655747dae25496d126e58f985df29f50 Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:20 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:41 +02:00
sysfs: Add preferred CPU file
Add a "preferred" file in /sys/devices/system/cpu/ when kernel is built with CONFIG_PREFERRED_CPU=y.
This would help
- Users to quickly check which CPUs are marked as preferred.
- Userspace daemons such as irqbalance to use this mask to route irqs into preferred CPUs.
For example:
cat /sys/devices/system/cpu/online
0-719
cat /sys/devices/system/cpu/preferred
0-599 <<< Implies 0-599 are preferred for workloads and 600-719
should be avoided at this moment.
cat /sys/devices/system/cpu/preferred
0-719 <<< All CPUs are usable. There is no preference.
--- Documentation/ABI/testing/sysfs-devices-system-cpu | 14 +++++++++++++- drivers/base/cpu.c | 12 +++++++++++- 2 files changed, 26 insertions(+)
Documentation/ABI/testing/sysfs-devices-system-cpu +14 -0
@@ -806,3 +806,17 @@ Date: Nov 2022 Contact: Linux kernel mailing list <[email protected]> Description: (RO) the list of CPUs that can be brought online. + +What: /sys/devices/system/cpu/preferred +Date: Sep 2026 +Contact: Linux kernel mailing list <[email protected]> +Description: + (RO) the list of preferred CPUs applicable in + paravirtualized environments. + + The steal governor driver dynamically adjusts this mask + based on observed steal time. Scheduling tasks on + CPUs outside of this list may lead to performance + degradations due to underlying physical CPU contention. + + See Documentation/scheduler/sched-paravirt.rst for more details.
drivers/base/cpu.c +12 -0
@@ -391,6 +391,15 @@ static int cpu_uevent(const struct device *dev, struct kobj_uevent_env *env) }#endif +#ifdef CONFIG_PREFERRED_CPU +static ssize_t preferred_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +return sysfs_emit (buf,"%*pbl\n" ,cpumask_pr_args (cpu_preferred_mask)); +} +static DEVICE_ATTR_RO (preferred); +#endif +const struct bus_type cpu_subsys = { .name ="cpu" , .dev_name ="cpu" ,
@@ -532,6 +541,9 @@ static struct attribute *cpu_root_attrs[] = {#ifdef CONFIG_GENERIC_CPU_AUTOPROBE &dev_attr_modalias.attr,#endif +#ifdef CONFIG_PREFERRED_CPU + &dev_attr_preferred.attr, +#endifNULL };
Shrikanth Hegde
maintainerPATCH v14 6/13 sched/core: Try to use a preferred CPU in is_cpu_allowed
In reply to Shrikanth HegdeWhen possible, try to choose a preferred CPU.
This is essential to maintain user affinities when preferred CPUs change. A task pinned on a non-preferred CPU should continue to run there, since this is a non-user triggered event.
If a CPU is non-preferred and the task can run on other CPUs which are currently preferred, then choose a preferred CPU instead. This is decided by checking if cpus_ptr, cpu_preferred_mask and task possible CPUs intersect or not. If yes, then the task has other preferred CPUs. This takes care of tasks with architecture-specific CPU masks (e.g., 32-bit tasks on arm64).
The push task mechanism uses a stopper thread which calls select_fallback_rq() and uses this mechanism to pick a preferred CPU.
This takes care of the wakeup path for FAIR tasks too. is_cpu_allowed() is called to ensure wakeups happen on preferred CPUs. With that, additional checks in available_idle_cpu() are not necessary.
Ignore the preferred CPU state if a task's affinity is changing and its new mask no longer includes the CPU it is currently running on. This ensures migration_cpu_stop() does not abort, preventing the task from being stranded outside its allowed affinity.
For the majority of cases, this would still keep select_fallback_rq() as O(N). cpumask_intersects_and(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomes O(N**2) only for tasks pinned solely to non-preferred CPUs. That is a rare case.
Overhead is minimal when the CPU is preferred.
--- kernel/sched/core.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
kernel/sched/core.c +28 -2
@@ -2504,6 +2504,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)return rq->nr_pinned; } +static inline bool task_can_migrate_to_preferred (struct task_struct *p,int cpu) +{ +/* No need to migrate from a preferred CPU */ +if (cpu_preferred (cpu)) +return false; + +/* Only FAIR tasks honor preferred CPU state */ +if (unlikely (p->sched_class != &fair_sched_class)) +return false; + +/* Ignore preferred state if task affinity is changing */ +if (unlikely (!cpumask_test_cpu (task_cpu (p), p->cpus_ptr))) +return false; + +return cpumask_intersects_and (p->cpus_ptr, cpu_preferred_mask, +task_cpu_possible_mask (p)); +} +/* * Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq().
@@ -2519,8 +2537,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)return cpu_online (cpu);/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +/* Try to use preferred CPU if task's affinity allows */ +if (task_can_migrate_to_preferred (p, cpu)) +return false;return cpu_active (cpu); + }/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))
@@ -2530,7 +2552,11 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (cpu_dying (cpu))return false; -/* But are allowed during online. */ +/* Try to keep unbound kthreads on a preferred CPU if possible. */ +if (task_can_migrate_to_preferred (p, cpu)) +return false; + +/* Otherwise, they are allowed to run on online CPU. */ return cpu_online (cpu); }
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 06/13] sched/core: Try to use a preferred CPU in is_cpu_allowed
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] sched/core: O(N^2) wakeup latency and task clumping in fallback path - [High] sched/fair: Active…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] sched/core: O(N^2) wakeup latency and task clumping in fallback path
- [High] sched/fair: Active load balancing to non-preferred CPUs silently fails
- [High] sched/core: TOCTOU race with CPU hotplug breaks task affinity
--
commit dfa0f159ec643e02807bc453bbfcc49c9f1e7915
Author: Shrikanth Hegde <[email protected]>
sched/core: Try to use a preferred CPU in is_cpu_allowed
This commit modifies the scheduler to preferentially select preferred CPUs when checking if a task is allowed to run on a given CPU. It introduces a helper function to check if a task can migrate to a preferred CPU based on its affinity mask.
Quoted 5 lines
For the majority of cases, this would still keep
select_fallback_rq()as O(N).cpumask_intersects_and(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomesO(N**2)only for tasks pinned solely to non-preferred CPUs. That is a rare case.
Is this description accurate?
Because the fallback loop iterates over the affinity mask sequentially, if the allowed preferred CPUs happen to have higher indices than the non-preferred CPUs, the loop will perform the intersection operation for every non-preferred CPU it checks. This means any task with preferred CPUs at higher indices could experience the quadratic latency overhead, rather than only tasks pinned solely to non-preferred CPUs.
Quoted 4 lines
kernel/sched/core.c +0
-0
[ ... ]
Quoted 11 lines
kernel/sched/core.c +7
-0@@ -2504,6 +2504,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)return rq->nr_pinned; } +static inline bool task_can_migrate_to_preferred (struct task_struct *p,int cpu) +{ +/* No need to migrate from a preferred CPU */ +if (cpu_preferred (cpu)) +return false; + +/* Only FAIR tasks honor preferred CPU state */
kernel/sched/core.c static inline bool rq_has_pinned_tasks(struct rq *rq) · line 2523
Quoted 10 lines
kernel/sched/core.c +10
-0+if (unlikely (p->sched_class != &fair_sched_class)) +return false; + +/* Ignore preferred state if task affinity is changing */ +if (unlikely (!cpumask_test_cpu (task_cpu (p), p->cpus_ptr))) +return false; + +return cpumask_intersects_and (p->cpus_ptr, cpu_preferred_mask, +task_cpu_possible_mask (p)); +}
[Severity: High]
Does this intersection operation lead to quadratic latency during wakeups when combined with the fallback loop?
When select_task_rq() in kernel/sched/core.c picks a non-preferred CPU because it is idle, is_cpu_allowed() will reject it, forcing the task into select_fallback_rq(). The fallback loop iterates sequentially over all CPUs:
kernel/sched/core.c:select_fallback_rq() {
...
for_each_cpu(dest_cpu, p->cpus_ptr) {
if (!is_cpu_allowed(p, dest_cpu))
continue;
...
goto out;
}
...
}If the allowed preferred CPUs are at the end of the cpumask, wouldn't this perform an intersection for every non-preferred CPU it checks, leading to severe scheduling latency? Additionally, wouldn't this deterministic loop always dump the task on the first preferred CPU it finds, destroying load distribution by clumping tasks on the lowest-ID preferred CPU?
This performance degradation from wakeups on non-preferred CPUs does not appear to be resolved in the final state of the patch series.
[ ... ]
kernel/sched/core.c static inline bool is_cpu_allowed(struct task_struct *p, int cpu) · line 2543
Quoted 9 lines
kernel/sched/core.c +4
-1@@ -2519,8 +2537,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)return cpu_online (cpu);/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +/* Try to use preferred CPU if task's affinity allows */ +if (task_can_migrate_to_preferred (p, cpu)) +return false;
[Severity: High]
Could returning false for non-preferred CPUs here cause active load balancing to silently fail?
During routine background load balancing, if the system attempts to pull load from a preferred CPU to an idle non-preferred CPU, the stopper thread will run __migrate_task().
kernel/sched/core.c:__migrate_task() {
...
if (!is_cpu_allowed(p, dest_cpu))
return rq;
...
}Since the destination is non-preferred, won't is_cpu_allowed() reject it, causing the stopper thread to silently abort the migration and break load distribution?
While follow-up patches in the series restrict the busiest CPU to preferred CPUs, they do not appear to prevent the destination from being non-preferred, leaving this active load balancing path broken.
[Severity: High]
Can this rejection introduce a race condition with CPU hotplug that breaks a task's user-configured affinity?
In select_fallback_rq() in kernel/sched/core.c, the loop might skip an active, non-preferred CPU because task_can_migrate_to_preferred() indicates another preferred CPU is available in the affinity mask. If a concurrent CPU hotplug operation removes that preferred CPU from the active mask before the loop reaches it, the loop will skip it too:
kernel/sched/core.c:select_fallback_rq() {
...
for_each_cpu(dest_cpu, p->cpus_ptr) {
if (!is_cpu_allowed(p, dest_cpu))
continue;
...
}Could the loop then terminate without finding any allowed CPU, permanently forcing the task outside of its allowed affinity?
Quoted 2 lines
kernel/sched/core.c +1
-0return cpu_active (cpu); + }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Shrikanth Hegde
maintainerRe: [PATCH v14 06/13] sched/core: Try to use a preferred CPU in is_cpu_allowed
I believe most of the comments have been addressed in the previous rounds.
Quoted 77 lines · On 9/28/26 11:30 AM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] sched/core: O(N^2) wakeup latency and task clumping in fallback path
- [High] sched/fair: Active load balancing to non-preferred CPUs silently fails
- [High] sched/core: TOCTOU race with CPU hotplug breaks task affinity
--commit
dfa0f159ec643e02807bc453bbfcc49c9f1e7915
Author: Shrikanth Hegde <[email protected]>sched/core: Try to use a preferred CPU in
is_cpu_allowedThis commit modifies the scheduler to preferentially select preferred CPUs when checking if a task is allowed to run on a given CPU. It introduces a helper function to check if a task can migrate to a preferred CPU based on its affinity mask.
Quoted 5 lines
For the majority of cases, this would still keep
select_fallback_rq()as O(N).cpumask_intersects_and(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomesO(N**2)only for tasks pinned solely to non-preferred CPUs. That is a rare case.Is this description accurate?
Because the fallback loop iterates over the affinity mask sequentially, if the allowed preferred CPUs happen to have higher indices than the non-preferred CPUs, the loop will perform the intersection operation for every non-preferred CPU it checks. This means any task with preferred CPUs at higher indices could experience the quadratic latency overhead, rather than only tasks pinned solely to non-preferred CPUs.
Quoted 4 lines
kernel/sched/core.c +0
-0[ ... ]
Quoted 11 lines
kernel/sched/core.c +7
-0@@ -2504,6 +2504,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)return rq->nr_pinned; } +static inline bool task_can_migrate_to_preferred (struct task_struct *p,int cpu) +{ +/* No need to migrate from a preferred CPU */ +if (cpu_preferred (cpu)) +return false; + +/* Only FAIR tasks honor preferred CPU state */ kernel/sched/core.c static inline bool rq_has_pinned_tasks(struct rq *rq) · line 2523
Quoted 10 lines
kernel/sched/core.c +10
-0+if (unlikely (p->sched_class != &fair_sched_class)) +return false; + +/* Ignore preferred state if task affinity is changing */ +if (unlikely (!cpumask_test_cpu (task_cpu (p), p->cpus_ptr))) +return false; + +return cpumask_intersects_and (p->cpus_ptr, cpu_preferred_mask, +task_cpu_possible_mask (p)); +}[Severity: High]
Does this intersection operation lead to quadratic latency during wakeups when combined with the fallback loop?When
select_task_rq()inkernel/sched/core.cpicks a non-preferred CPU because it is idle,is_cpu_allowed()will reject it, forcing the task intoselect_fallback_rq(). The fallback loop iterates sequentially over all CPUs:kernel/sched/core.c:select_fallback_rq() { ... for_each_cpu(dest_cpu, p->cpus_ptr) { if (!is_cpu_allowed(p, dest_cpu)) continue; ... goto out; } ... }
Quoted 8 lines
If the allowed preferred CPUs are at the end of the cpumask, wouldn't this perform an intersection for every non-preferred CPU it checks, leading to severe scheduling latency? Additionally, wouldn't this deterministic loop always dump the task on the first preferred CPU it finds, destroying load distribution by clumping tasks on the lowest-ID preferred CPU?
This performance degradation from wakeups on non-preferred CPUs does not appear to be resolved in the final state of the patch series.
It is mentioned in the changelog and this is true only when wakeup happens on a non-preferred CPUs. Even if it runs there is meant to pushed out. So take a bit more cycles and do the right thing.
Quoted 5 lines
[ ... ]
Quoted 3 lines
kernel/sched/core.c +0
-0@@ -2519,8 +2537,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)return cpu_online (cpu);
Quoted 10 lines
kernel/sched/core.c static inline bool is_cpu_allowed(struct task_struct *p, int cpu) · line 2543
Quoted 6 lines
kernel/sched/core.c +4
-1/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +/* Try to use preferred CPU if task's affinity allows */ +if (task_can_migrate_to_preferred (p, cpu)) +return false;[Severity: High]
Could returning false for non-preferred CPUs here cause active load balancing to silently fail?
Spreading load to non-preferred CPUs is by design and expected.
Quoted 24 lines
During routine background load balancing, if the system attempts to pull load from a preferred CPU to an idle non-preferred CPU, the stopper thread will run
__migrate_task().kernel/sched/core.c:__migrate_task() { ... if (!is_cpu_allowed(p, dest_cpu)) return rq; ... }Since the destination is non-preferred, won't
is_cpu_allowed()reject it, causing the stopper thread to silently abort the migration and break load distribution?While follow-up patches in the series restrict the busiest CPU to preferred CPUs, they do not appear to prevent the destination from being non-preferred, leaving this active load balancing path broken.
[Severity: High]
Can this rejection introduce a race condition with CPU hotplug that breaks a task's user-configured affinity?
Quoted 5 lines
In
select_fallback_rq()inkernel/sched/core.c, the loop might skip an active, non-preferred CPU becausetask_can_migrate_to_preferred()indicates another preferred CPU is available in the affinity mask. If a concurrent CPU hotplug operation removes that preferred CPU from the active mask before the loop reaches it, the loop will skip it too:
There is two iterations and steal governor's minimal interval is 100ms. So it is likely safe.
Quoted 15 lines
kernel/sched/core.c:select_fallback_rq() { ... for_each_cpu(dest_cpu, p->cpus_ptr) { if (!is_cpu_allowed(p, dest_cpu)) continue; ... }Could the loop then terminate without finding any allowed CPU, permanently forcing the task outside of its allowed affinity?
Quoted 2 lines
kernel/sched/core.c +1
-0return cpu_active (cpu); + }
[tip: sched/core] sched/core: Try to use a preferred CPU in is_cpu_allowed
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: d8a3da0de8433d494db1ff5ec541559c764da4ab Gitweb: https://git.kernel.org/tip/d8a3da0de8433d494db1ff5ec541559c764da4ab Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:21 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:41 +02:00
sched/core: Try to use a preferred CPU in is_cpu_allowed
When possible, try to choose a preferred CPU.
This is essential to maintain user affinities when preferred CPUs change. A task pinned on a non-preferred CPU should continue to run there, since this is a non-user triggered event.
If a CPU is non-preferred and the task can run on other CPUs which are currently preferred, then choose a preferred CPU instead. This is decided by checking if cpus_ptr, cpu_preferred_mask and task possible CPUs intersect or not. If yes, then the task has other preferred CPUs. This takes care of tasks with architecture-specific CPU masks (e.g., 32-bit tasks on arm64).
The push task mechanism uses a stopper thread which calls select_fallback_rq() and uses this mechanism to pick a preferred CPU.
This takes care of the wakeup path for FAIR tasks too. is_cpu_allowed() is called to ensure wakeups happen on preferred CPUs. With that, additional checks in available_idle_cpu() are not necessary.
Ignore the preferred CPU state if a task's affinity is changing and its new mask no longer includes the CPU it is currently running on. This ensures migration_cpu_stop() does not abort, preventing the task from being stranded outside its allowed affinity.
For the majority of cases, this would still keep select_fallback_rq() as O(N). cpumask_intersects_and(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomes O(N**2) only for tasks pinned solely to non-preferred CPUs. That is a rare case.
Overhead is minimal when the CPU is preferred.
--- kernel/sched/core.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
kernel/sched/core.c +28 -2
@@ -2511,6 +2511,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)return rq->nr_pinned; } +static inline bool task_can_migrate_to_preferred (struct task_struct *p,int cpu) +{ +/* No need to migrate from a preferred CPU */ +if (cpu_preferred (cpu)) +return false; + +/* Only FAIR tasks honor preferred CPU state */ +if (unlikely (p->sched_class != &fair_sched_class)) +return false; + +/* Ignore preferred state if task affinity is changing */ +if (unlikely (!cpumask_test_cpu (task_cpu (p), p->cpus_ptr))) +return false; + +return cpumask_intersects_and (p->cpus_ptr, cpu_preferred_mask, +task_cpu_possible_mask (p)); +} +/* * Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq().
@@ -2526,8 +2544,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)return cpu_online (cpu);/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +/* Try to use preferred CPU if task's affinity allows */ +if (task_can_migrate_to_preferred (p, cpu)) +return false;return cpu_active (cpu); + }/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))
@@ -2537,7 +2559,11 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (cpu_dying (cpu))return false; -/* But are allowed during online. */ +/* Try to keep unbound kthreads on a preferred CPU if possible. */ +if (task_can_migrate_to_preferred (p, cpu)) +return false; + +/* Otherwise, they are allowed to run on online CPU. */ return cpu_online (cpu); }
Shrikanth Hegde
maintainerPATCH v14 7/13 sched/fair: Load balance only among preferred CPUs
In reply to Shrikanth HegdeWhen a CPU is marked as non-preferred, any load pulled towards it is pointless since the task will be pushed out again in the next tick. So, consider only preferred CPUs for load balancing.
This ensures load balancing does not fight against the push task mechanism which happens at the tick. Also, this stops active balancing from happening on a non-preferred CPU pulling the load.
This also means there is no load balancing if a task is pinned only to non-preferred CPUs. They will continue to run where they were previously running before the CPUs were marked as non-preferred.
Bail out early for NEWIDLE balancing, as load balancing is done only on preferred CPUs. Note that idle balancing is allowed to go through, since that naturally updates nohz.next_balance when all the idle CPUs are non-preferred.
Also, optimization in find_new_ilb() is skipped. The steal governor driver, which is introduced in later patches, updates the preferred CPUs state in descending order. find_new_ilb() checks for idle CPUs in ascending order. Hence, in most common scenarios, the idle CPU found by find_new_ilb() will already be a preferred CPU. When all idle CPUs are non-preferred, the first idle CPU has to be chosen anyway. All of this is naturally handled in find_new_ilb() currently. Adding additional complexity to it for rare edge cases is not necessary.
--- kernel/sched/fair.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
kernel/sched/fair.c +3 -5
@@ -13467,7 +13467,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq, };bool need_unlock = false; -cpumask_and (cpus,sched_domain_span (sd), cpu_active_mask); +cpumask_and (cpus,sched_domain_span (sd), cpu_preferred_mask);schedstat_inc (sd->lb_count[idle]);
@@ -14582,10 +14582,8 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)*/ this_rq->idle_stamp =rq_clock (this_rq); -/* -* Do not pull tasks towards !active CPUs... -*/ -if (!cpu_active (this_cpu)) +/* Do not pull tasks towards !preferred CPUs */ +if (!cpu_preferred (this_cpu))return 0 ;/*
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 07/13] sched/fair: Load balance only among preferred CPUs
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] sched: capacity mismatch in load balance due to restricted env->cpus --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] sched: capacity mismatch in load balance due to restricted env->cpus --
commit 614fa1366931a4c3a90955d4b7c85dd2e0e3bb49
Author: Shrikanth Hegde <[email protected]>
Subject: sched/fair: Load balance only among preferred CPUs
This commit restricts load balancing to preferred CPUs to avoid conflicting with the tick's push task mechanism. It replaces cpu_active_mask with cpu_preferred_mask during load balance CPU mask initialization and bails out early in newidle balancing if the CPU is not preferred.
kernel/sched/fair.c static int sched_balance_rq(int this_cpu, struct rq *this_rq, · line 13470
Quoted 10 lines
kernel/sched/fair.c +1
-1@@ -13467,7 +13467,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq, };bool need_unlock = false; -cpumask_and (cpus,sched_domain_span (sd), cpu_active_mask); +cpumask_and (cpus,sched_domain_span (sd), cpu_preferred_mask);
[Severity: High]
Will this change create a capacity mismatch in the load balancer?
When env->cpus is restricted to cpu_preferred_mask here, update_sg_lb_stats() will only aggregate group load for those preferred CPUs:
kernel/sched/fair.c:update_sg_lb_stats() {
...
for_each_cpu_and(i, sched_group_span(group), env->cpus) {
struct rq *rq = cpu_rq(i);
unsigned long load = cpu_load(rq);
sgs->group_load += load;
...
}However, the group's capacity is set using the full capacity of all active CPUs:
kernel/sched/fair.c:update_sg_lb_stats() {
...
sgs->group_capacity = group->sgc->capacity;
...
}Could the mismatch between the deflated group_load and the full group_capacity artificially lower sgs->avg_load?
This phantom spare capacity might cause the load balancer to aggressively pull tasks from other groups into the preferred CPUs of the destination group, overloading them.
Can this also prevent tasks on non-preferred CPUs from ever being load balanced?
kernel/sched/fair.c:sched_balance_find_src_rq() {
...
for_each_cpu_and(i, sched_group_span(group), env->cpus) {
...
}Because this loop uses env->cpus, the load balancer ignores non-preferred CPUs entirely, even if they are overloaded.
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
[tip: sched/core] sched/fair: Load balance only among preferred CPUs
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4ee29b029058a8f06fbb91425e1d9351e8015b53 Gitweb: https://git.kernel.org/tip/4ee29b029058a8f06fbb91425e1d9351e8015b53 Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:22 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:42 +02:00
sched/fair: Load balance only among preferred CPUs
When a CPU is marked as non-preferred, any load pulled towards it is pointless since the task will be pushed out again in the next tick. So, consider only preferred CPUs for load balancing.
This ensures load balancing does not fight against the push task mechanism which happens at the tick. Also, this stops active balancing from happening on a non-preferred CPU pulling the load.
This also means there is no load balancing if a task is pinned only to non-preferred CPUs. They will continue to run where they were previously running before the CPUs were marked as non-preferred.
Bail out early for NEWIDLE balancing, as load balancing is done only on preferred CPUs. Note that idle balancing is allowed to go through, since that naturally updates nohz.next_balance when all the idle CPUs are non-preferred.
Also, optimization in find_new_ilb() is skipped. The steal governor driver, which is introduced in later patches, updates the preferred CPUs state in descending order. find_new_ilb() checks for idle CPUs in ascending order. Hence, in most common scenarios, the idle CPU found by find_new_ilb() will already be a preferred CPU. When all idle CPUs are non-preferred, the first idle CPU has to be chosen anyway. All of this is naturally handled in find_new_ilb() currently. Adding additional complexity to it for rare edge cases is not necessary.
--- kernel/sched/fair.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
kernel/sched/fair.c +3 -5
@@ -13467,7 +13467,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq, };bool need_unlock = false; -cpumask_and (cpus,sched_domain_span (sd), cpu_active_mask); +cpumask_and (cpus,sched_domain_span (sd), cpu_preferred_mask);schedstat_inc (sd->lb_count[idle]);
@@ -14582,10 +14582,8 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)*/ this_rq->idle_stamp =rq_clock (this_rq); -/* -* Do not pull tasks towards !active CPUs... -*/ -if (!cpu_active (this_cpu)) +/* Do not pull tasks towards !preferred CPUs */ +if (!cpu_preferred (this_cpu))return 0 ;/*
Shrikanth Hegde
maintainerPATCH v14 8/13 sched/core: Push current task from non preferred CPU
In reply to Shrikanth HegdeActively push out the current running task on a non-preferred CPU (NPC). Since the task is currently running, a stopper thread must be queued to push the task out. However, if the task is pinned only to non-preferred CPUs, it will continue running there.
This helps to maintain userspace affinities, unlike CPU hotplug or isolated cpusets.
The implementation follows the structure of __balance_push_cpu_stop(), but is kept separate to handle the preferred-CPU specific conditions and pending-work state under CONFIG_PREFERRED_CPU.
Add the npc_push_work_pending flag to protect the work buffer.
For now, only the currently running task is pushed out. This keeps the code simpler. In the future, an optimization may be added to move all queued tasks on the runqueue.
This works only for the FAIR scheduling class.
--- kernel/sched/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 9 +++++ 2 files changed, 92 insertions(+)
kernel/sched/core.c +83 -0
@@ -5809,6 +5809,9 @@ void sched_tick(void)unsigned long hw_pressure;u64 resched_latency; +if (!cpu_preferred (cpu)) +sched_push_current_non_preferred_cpu (rq); +if (housekeeping_cpu (cpu,HK_TYPE_KERNEL_NOISE ))arch_scale_freq_tick ();
@@ -11201,3 +11204,83 @@ void sched_change_end(struct sched_change_ctx *ctx) p->sched_class->prio_changed (rq, p, ctx->prio); } } + +#ifdef CONFIG_PREFERRED_CPU +static DEFINE_PER_CPU(struct cpu_stop_work, npc_push_task_work); + +static int sched_non_preferred_cpu_push_stop (void *arg) +{ +struct task_struct *p = arg; +struct rq *rq =this_rq (); +struct rq_flags rf; +int cpu; + +if (cpu_preferred (rq->cpu)) { +scoped_guard (rq_lock_irqsave, rq) + rq->npc_push_work_pending = false; +put_task_struct (p); +return 0 ; + } + + scoped_guard (raw_spinlock_irq, &p->pi_lock) { +/* +* select_fallback_rq() may acquire the rq lock in case of +* fallback. So call it before grabbing rq lock. If the task +* migrates to another CPU before the rq lock is acquired, +* subsequent validation of task's current rq will help to +* safely bail out. +*/ + cpu =select_fallback_rq (rq->cpu, p); +rq_lock (rq, &rf); + rq->npc_push_work_pending = false; +update_rq_clock (rq); +context_unsafe_alias (rq); + +if (task_rq (p) == rq &&task_on_rq_queued (p)) + rq =__migrate_task (rq, &rf, p, cpu); +rq_unlock (rq, &rf); + } + +put_task_struct (p); +return 0 ; +} + +/* +* Push the current task running on non-preferred CPU(npc). +* Using this non preferred CPU will lead to more contention +* in the host. So it is better not to use this CPU. +* +* Since task is running, call a stopper to push the task out. This is +* similar to how task moves during hotplug. In select_fallback_rq() a +* preferred CPU will be chosen and henceforth task shouldn't come back to +* this CPU again. +* +* Works for FAIR class only. +* +* If task is affined only on non-preferred CPUs, no point in moving it out. +*/ +void sched_push_current_non_preferred_cpu (struct rq *rq) +{ +struct task_struct *push_task = rq->curr; + + scoped_guard(rq_lock, rq) { +/* Push the task if its explicit affinity allows */ +if (!task_can_migrate_to_preferred (push_task, rq->cpu)) +return ; + +/* There is already a stopper thread. Don't race with it. */ +if (rq->npc_push_work_pending) +return ; + +if (is_migration_disabled (push_task)) +return ; + + rq->npc_push_work_pending = true; + } + +/* sched_tick runs with interrupts disabled. */ +get_task_struct (push_task); +stop_one_cpu_nowait (rq->cpu, sched_non_preferred_cpu_push_stop, + push_task,this_cpu_ptr (&npc_push_task_work)); +} +#endif
kernel/sched/sched.h +9 -0
@@ -1326,6 +1326,9 @@ struct rq {#ifdef CONFIG_PARAVIRT_TIME_ACCOUNTING u64 prev_steal_time_rq;#endif +#ifdef CONFIG_PREFERRED_CPU +bool npc_push_work_pending; +#endif /* calc_load related fields */ unsigned long calc_load_update;
@@ -4292,4 +4295,10 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)#include "ext/ext.h" +#ifdef CONFIG_PREFERRED_CPU +void sched_push_current_non_preferred_cpu (struct rq *rq); +#else /* !CONFIG_PREFERRED_CPU */ +static inline void sched_push_current_non_preferred_cpu (struct rq *rq) { } +#endif +#endif /* _KERNEL_SCHED_SCHED_H */
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 08/13] sched/core: Push current task from non preferred CPU
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] sched/core: FAIR kthreads cause stopper thread livelock on non-preferred CPUs --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] sched/core: FAIR kthreads cause stopper thread livelock on non-preferred CPUs --
commit a6718dc77694942ae8b63a9eb25ace9c01b3c5ef
Author: Shrikanth Hegde <[email protected]>
sched/core: Push current task from non preferred CPU
This patch actively pushes the currently running task off a non-preferred CPU using a stopper thread, specifically targeting tasks in the FAIR scheduling class. It adds the npc_push_work_pending flag to avoid queueing multiple stopper threads.
Quoted 3 lines
kernel/sched/core.c +0
-0
[ ... ]
Quoted 23 lines
kernel/sched/core.c +23
-0+static int sched_non_preferred_cpu_push_stop (void *arg) +{ +struct task_struct *p = arg; +struct rq *rq =this_rq (); +struct rq_flags rf; +int cpu; + +if (cpu_preferred (rq->cpu)) { +scoped_guard (rq_lock_irqsave, rq) + rq->npc_push_work_pending = false; +put_task_struct (p); +return 0 ; + } + + scoped_guard (raw_spinlock_irq, &p->pi_lock) { +/* +* select_fallback_rq() may acquire the rq lock in case of +* fallback. So call it before grabbing rq lock. If the task +* migrates to another CPU before the rq lock is acquired, +* subsequent validation of task's current rq will help to +* safely bail out. +*/ + cpu =select_fallback_rq (rq->cpu, p);
[ ... ]
kernel/sched/core.c void sched_change_end(struct sched_change_ctx *ctx) · line 11269
Quoted 8 lines
kernel/sched/core.c +8
-0+void sched_push_current_non_preferred_cpu (struct rq *rq) +{ +struct task_struct *push_task = rq->curr; + + scoped_guard(rq_lock, rq) { +/* Push the task if its explicit affinity allows */ +if (!task_can_migrate_to_preferred (push_task, rq->cpu)) +return ;
[Severity: High]
Can this result in a livelock with endless stopper thread preemptions when the task being pushed is a FAIR kthread (such as an unbound kworker)?
When task_can_migrate_to_preferred() returns true for a FAIR kthread, this function successfully queues a stopper thread. However, when the stopper thread runs sched_non_preferred_cpu_push_stop() and calls select_fallback_rq(), it ultimately evaluates is_cpu_allowed(), which skips the preferred-CPU check for kthreads:
kernel/sched/core.c:is_cpu_allowed() {
...
/* Non kernel threads are not allowed during either online or offline. */
if (!(p->flags & PF_KTHREAD)) {
/* Try to use preferred CPU if task's affinity allows */
if (task_can_migrate_to_preferred(p, cpu))
return false;
return cpu_active(cpu);
}
...
}Because the kthread bypasses the preferred-CPU enforcement, select_fallback_rq() can pick the first online CPU in the local node, which might be the exact same non-preferred CPU we are trying to push it away from.
If that happens, the migration becomes a no-op, npc_push_work_pending gets cleared, and sched_tick() will just restart the same sequence on the next tick, wasting CPU cycles in an endless loop.
Quoted 4 lines
kernel/sched/core.c +4
-0+ +/* There is already a stopper thread. Don't race with it. */ +if (rq->npc_push_work_pending) +return ;
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
[tip: sched/core] sched/core: Push current task from non preferred CPU
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 74699f56ebcfe0f401ee6ce4c5a72822101bb222 Gitweb: https://git.kernel.org/tip/74699f56ebcfe0f401ee6ce4c5a72822101bb222 Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:23 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:42 +02:00
sched/core: Push current task from non preferred CPU
Actively push out the current running task on a non-preferred CPU (NPC). Since the task is currently running, a stopper thread must be queued to push the task out. However, if the task is pinned only to non-preferred CPUs, it will continue running there.
This helps to maintain userspace affinities, unlike CPU hotplug or isolated cpusets.
The implementation follows the structure of __balance_push_cpu_stop(), but is kept separate to handle the preferred-CPU specific conditions and pending-work state under CONFIG_PREFERRED_CPU.
Add the npc_push_work_pending flag to protect the work buffer.
For now, only the currently running task is pushed out. This keeps the code simpler. In the future, an optimization may be added to move all queued tasks on the runqueue.
This works only for the FAIR scheduling class.
--- kernel/sched/core.c | 83 +++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 9 +++++- 2 files changed, 92 insertions(+)
kernel/sched/core.c +83 -0
@@ -5829,6 +5829,9 @@ void sched_tick(void)unsigned long hw_pressure;u64 resched_latency; +if (!cpu_preferred (cpu)) +sched_push_current_non_preferred_cpu (rq); +if (housekeeping_cpu (cpu,HK_TYPE_KERNEL_NOISE ))arch_scale_freq_tick ();
@@ -11253,3 +11256,83 @@ void sched_change_end(struct sched_change_ctx *ctx) p->sched_class->prio_changed (rq, p, ctx->prio); } } + +#ifdef CONFIG_PREFERRED_CPU +static DEFINE_PER_CPU(struct cpu_stop_work, npc_push_task_work); + +static int sched_non_preferred_cpu_push_stop (void *arg) +{ +struct task_struct *p = arg; +struct rq *rq =this_rq (); +struct rq_flags rf; +int cpu; + +if (cpu_preferred (rq->cpu)) { +scoped_guard (rq_lock_irqsave, rq) + rq->npc_push_work_pending = false; +put_task_struct (p); +return 0 ; + } + + scoped_guard (raw_spinlock_irq, &p->pi_lock) { +/* +* select_fallback_rq() may acquire the rq lock in case of +* fallback. So call it before grabbing rq lock. If the task +* migrates to another CPU before the rq lock is acquired, +* subsequent validation of task's current rq will help to +* safely bail out. +*/ + cpu =select_fallback_rq (rq->cpu, p); +rq_lock (rq, &rf); + rq->npc_push_work_pending = false; +update_rq_clock (rq); +context_unsafe_alias (rq); + +if (task_rq (p) == rq &&task_on_rq_queued (p)) + rq =__migrate_task (rq, &rf, p, cpu); +rq_unlock (rq, &rf); + } + +put_task_struct (p); +return 0 ; +} + +/* +* Push the current task running on non-preferred CPU(npc). +* Using this non preferred CPU will lead to more contention +* in the host. So it is better not to use this CPU. +* +* Since task is running, call a stopper to push the task out. This is +* similar to how task moves during hotplug. In select_fallback_rq() a +* preferred CPU will be chosen and henceforth task shouldn't come back to +* this CPU again. +* +* Works for FAIR class only. +* +* If task is affined only on non-preferred CPUs, no point in moving it out. +*/ +void sched_push_current_non_preferred_cpu (struct rq *rq) +{ +struct task_struct *push_task = rq->curr; + + scoped_guard(rq_lock, rq) { +/* Push the task if its explicit affinity allows */ +if (!task_can_migrate_to_preferred (push_task, rq->cpu)) +return ; + +/* There is already a stopper thread. Don't race with it. */ +if (rq->npc_push_work_pending) +return ; + +if (is_migration_disabled (push_task)) +return ; + + rq->npc_push_work_pending = true; + } + +/* sched_tick runs with interrupts disabled. */ +get_task_struct (push_task); +stop_one_cpu_nowait (rq->cpu, sched_non_preferred_cpu_push_stop, + push_task,this_cpu_ptr (&npc_push_task_work)); +} +#endif
kernel/sched/sched.h +9 -0
@@ -1326,6 +1326,9 @@ struct rq {#ifdef CONFIG_PARAVIRT_TIME_ACCOUNTING u64 prev_steal_time_rq;#endif +#ifdef CONFIG_PREFERRED_CPU +bool npc_push_work_pending; +#endif /* calc_load related fields */ unsigned long calc_load_update;
@@ -4299,4 +4302,10 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)#include "ext/ext.h" +#ifdef CONFIG_PREFERRED_CPU +void sched_push_current_non_preferred_cpu (struct rq *rq); +#else /* !CONFIG_PREFERRED_CPU */ +static inline void sched_push_current_non_preferred_cpu (struct rq *rq) { } +#endif +#endif /* _KERNEL_SCHED_SCHED_H */
Shrikanth Hegde
maintainerPATCH v14 9/13 sched/debug: Add migration stats due to non preferred CPUs
In reply to Shrikanth HegdeAdd a new per-task stat,
- nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.
Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update /proc/schedstat. Hence increasing the schedstat version is not necessary.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 9 +++++++-- kernel/sched/debug.c | 1 + 3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
@@ -554,6 +554,7 @@ struct sched_statistics {u64 nr_failed_migrations_running;u64 nr_failed_migrations_hot;u64 nr_forced_migrations; +u64 nr_migrations_cpu_non_preferred;u64 nr_wakeups;u64 nr_wakeups_sync;
kernel/sched/core.c +7 -2
@@ -11236,8 +11236,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg)update_rq_clock (rq);context_unsafe_alias (rq); -if (task_rq (p) == rq &&task_on_rq_queued (p)) - rq =__migrate_task (rq, &rf, p, cpu); +if (task_rq (p) == rq &&task_on_rq_queued (p)) { +struct rq *dest_rq =__migrate_task (rq, &rf, p, cpu); + +if (rq != dest_rq) +schedstat_inc (p->stats.nr_migrations_cpu_non_preferred); + rq = dest_rq; + }rq_unlock (rq, &rf); }
kernel/sched/debug.c +1 -0
@@ -1440,6 +1440,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P_SCHEDSTAT (nr_failed_migrations_running);P_SCHEDSTAT (nr_failed_migrations_hot);P_SCHEDSTAT (nr_forced_migrations); +P_SCHEDSTAT (nr_migrations_cpu_non_preferred);P_SCHEDSTAT (nr_wakeups);P_SCHEDSTAT (nr_wakeups_sync);P_SCHEDSTAT (nr_wakeups_migrate);
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 09/13] sched/debug: Add migration stats due to non preferred CPUs
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Add a new per-task stat,
-nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update
/proc/schedstat. Hence increasing the schedstat version is not necessary.Signed-off-by Shrikanth Hegde <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
[tip: sched/core] sched/debug: Add migration stats due to non preferred CPUs
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 68957caaa9c0e127bc87fee64e45be843f6c5c91 Gitweb: https://git.kernel.org/tip/68957caaa9c0e127bc87fee64e45be843f6c5c91 Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:24 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:42 +02:00
sched/debug: Add migration stats due to non preferred CPUs
Add a new per-task stat,
- nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.
Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update /proc/schedstat. Hence increasing the schedstat version is not necessary.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 9 +++++++-- kernel/sched/debug.c | 1 + 3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
@@ -554,6 +554,7 @@ struct sched_statistics {u64 nr_failed_migrations_running;u64 nr_failed_migrations_hot;u64 nr_forced_migrations; +u64 nr_migrations_cpu_non_preferred;u64 nr_wakeups;u64 nr_wakeups_sync;
kernel/sched/core.c +7 -2
@@ -11288,8 +11288,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg)update_rq_clock (rq);context_unsafe_alias (rq); -if (task_rq (p) == rq &&task_on_rq_queued (p)) - rq =__migrate_task (rq, &rf, p, cpu); +if (task_rq (p) == rq &&task_on_rq_queued (p)) { +struct rq *dest_rq =__migrate_task (rq, &rf, p, cpu); + +if (rq != dest_rq) +schedstat_inc (p->stats.nr_migrations_cpu_non_preferred); + rq = dest_rq; + }rq_unlock (rq, &rf); }
kernel/sched/debug.c +1 -0
@@ -1440,6 +1440,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P_SCHEDSTAT (nr_failed_migrations_running);P_SCHEDSTAT (nr_failed_migrations_hot);P_SCHEDSTAT (nr_forced_migrations); +P_SCHEDSTAT (nr_migrations_cpu_non_preferred);P_SCHEDSTAT (nr_wakeups);P_SCHEDSTAT (nr_wakeups_sync);P_SCHEDSTAT (nr_wakeups_migrate);
Re: [PATCH v14 09/13] sched/debug: Add migration stats due to non preferred CPUs
In reply to Shrikanth HegdeQuoted 9 lines · Shrikanth Hegde
Add a new per-task stat,
-nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update
/proc/schedstat. Hence increasing the schedstat version is not necessary.Signed-off-by Shrikanth Hegde <[email protected]>
...
Quoted 9 lines
kernel/sched/core.c +0
-1@@ -11236,8 +11236,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg)update_rq_clock (rq);context_unsafe_alias (rq); -if (task_rq (p) == rq &&task_on_rq_queued (p))
kernel/sched/core.c static int sched_non_preferred_cpu_push_stop(void *arg) · line 11247
Quoted 10 lines
kernel/sched/core.c +7
-1- rq =__migrate_task (rq, &rf, p, cpu); +if (task_rq (p) == rq &&task_on_rq_queued (p)) { +struct rq *dest_rq =__migrate_task (rq, &rf, p, cpu); + +if (rq != dest_rq) +schedstat_inc (p->stats.nr_migrations_cpu_non_preferred); + rq = dest_rq; + }rq_unlock (rq, &rf); }
This breaks the build for me with clang-23+ (which have context analysis enabled by default), although it bisects to the final patch of the series since this is under CONFIG_PREFERRED_CPU and it is not selected until then.
kernel/sched/core.c:11292:25: error: calling function '__migrate_task' requires holding raw_spinlock 'rq_lockp(rq)' exclusively [-Werror,-Wthread-safety-analysis]
11292 | struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
| ^
kernel/sched/core.c:11298:3: error: releasing raw_spinlock 'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis]
11298 | rq_unlock(rq, &rf);
| ^
kernel/sched/core.c:11303:1: error: raw_spinlock 'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis]
11303 | }
| ^
kernel/sched/core.c:11286:3: note: raw_spinlock acquired here
11286 | rq_lock(rq, &rf);
| ^
3 errors generated.Not sure what the proper fix for this is, maybe another context_unsafe_alias()? cc Marco just in case
Signature · 2 lines
Cheers, Nathan
Shrikanth Hegde
maintainerRe: [PATCH v14 09/13] sched/debug: Add migration stats due to non preferred CPUs
Hi Nathan. Thanks for report.
Quoted 50 lines · Nathan Chancellor
Quoted 9 lines · Shrikanth Hegde
Add a new per-task stat,
-nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update
/proc/schedstat. Hence increasing the schedstat version is not necessary.Signed-off-by Shrikanth Hegde <[email protected]>...
Quoted 9 lines
kernel/sched/core.c +0
-1@@ -11236,8 +11236,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg)update_rq_clock (rq);context_unsafe_alias (rq); -if (task_rq (p) == rq &&task_on_rq_queued (p))kernel/sched/core.c static int sched_non_preferred_cpu_push_stop(void *arg) · line 11247
Quoted 10 lines
kernel/sched/core.c +7
-1- rq =__migrate_task (rq, &rf, p, cpu); +if (task_rq (p) == rq &&task_on_rq_queued (p)) { +struct rq *dest_rq =__migrate_task (rq, &rf, p, cpu); + +if (rq != dest_rq) +schedstat_inc (p->stats.nr_migrations_cpu_non_preferred); + rq = dest_rq; + }rq_unlock (rq, &rf); }This breaks the build for me with clang-23+ (which have context analysis enabled by default), although it bisects to the final patch of the series since this is under
CONFIG_PREFERRED_CPUand it is not selected until then.kernel/sched/core.c:11292:25: error: calling function '__migrate_task' requires holding raw_spinlock 'rq_lockp(rq)' exclusively [-Werror,-Wthread-safety-analysis] 11292 | struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu); | ^ kernel/sched/core.c:11298:3: error: releasing raw_spinlock 'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis] 11298 | rq_unlock(rq, &rf); | ^ kernel/sched/core.c:11303:1: error: raw_spinlock 'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis] 11303 | } | ^ kernel/sched/core.c:11286:3: note: raw_spinlock acquired here 11286 | rq_lock(rq, &rf); | ^ 3 errors generated.
Quoted 3 lines
Not sure what the proper fix for this is, maybe another
context_unsafe_alias()? cc Marco just in case
I suspect it is due to using of dest_rq = rq and rq is changing context.
let me try locally and see the fix.
One fix is use rq->cpu instead of rq comparison to see if migration happened.
Shrikanth Hegde
maintainerRe: [PATCH v14 09/13] sched/debug: Add migration stats due to non preferred CPUs
Hi Nathan.
Quoted 52 lines · Shrikanth Hegde
Hi Nathan. Thanks for report.
Quoted 49 lines · Nathan Chancellor
Quoted 9 lines · Shrikanth Hegde
Add a new per-task stat,
-nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update
/proc/schedstat. Hence increasing the schedstat version is not necessary.Signed-off-by Shrikanth Hegde <[email protected]>...
Quoted 8 lines
kernel/sched/core.c +0
-1@@ -11236,8 +11236,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg) update_rq_clock(rq); context_unsafe_alias(rq); - if (task_rq(p) == rq && task_on_rq_queued(p))kernel/sched/core.c static int sched_non_preferred_cpu_push_stop(void *arg) · line 11246
Quoted 10 lines
kernel/sched/core.c +7
-1- rq = __migrate_task(rq, &rf, p, cpu); + if (task_rq(p) == rq && task_on_rq_queued(p)) { + struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu); + + if (rq != dest_rq) + schedstat_inc(p->stats.nr_migrations_cpu_non_preferred); + rq = dest_rq; + } rq_unlock(rq, &rf); }This breaks the build for me with clang-23+ (which have context analysis enabled by default), although it bisects to the final patch of the series since this is under
CONFIG_PREFERRED_CPUand it is not selected until then.
kernel/sched/core.c:11292:25: error: calling function '__migrate_task' requires holdingraw_spinlock'rq_lockp(rq)' exclusively [-Werror,-Wthread-safety-analysis] 11292 | struct rq*dest_rq=__migrate_task(rq, &rf, p, cpu);
| ^
kernel/sched/core.c:11298:3: error: releasingraw_spinlock'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis]
11298 |rq_unlock(rq, &rf);
| ^
kernel/sched/core.c:11303:1: error:raw_spinlock'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis] 11303 | }
| ^
kernel/sched/core.c:11286:3: note:raw_spinlockacquired here
11286 |rq_lock(rq, &rf);
| ^
3 errors generated.
Quoted 8 lines
Quoted 3 lines
Not sure what the proper fix for this is, maybe another
context_unsafe_alias()? cc Marco just in caseI suspect it is due to using of
dest_rq= rq and rq is changing context.let me try locally and see the fix.
One fix is userq->cpuinstead of rq comparison to see if migration happened.
Well, it was rather due to Patch 8/13 which had rq marked as context_unsafe_alias after acquiring it.
I have tried below and that helps to fix the warnings. I will write a changelog and send it across soon.
Let me know if it works for you.
---
kernel/sched/core.c +1 -1
@@ -11283,10 +11283,10 @@ static int sched_non_preferred_cpu_push_stop(void *arg)* safely bail out. */ cpu =select_fallback_rq (rq->cpu, p); +context_unsafe_alias (rq);rq_lock (rq, &rf); rq->npc_push_work_pending = false;update_rq_clock (rq); -context_unsafe_alias (rq);if (task_rq (p) == rq &&task_on_rq_queued (p)) {struct rq *dest_rq =__migrate_task (rq, &rf, p, cpu);
Shrikanth Hegde
maintainerPATCH v14 10/13 virt: Introduce steal governor driver
In reply to Shrikanth HegdeIntroduce a new driver in virt named steal_governor. This driver will compute the steal time and drive the policy decisions regarding the preferred CPU state.
Note that this driver is strictly intended for actual guests. Hence block it on Xen dom0.
More details can be found in Documentation/driver-api/steal-governor.rst.
A new kconfig called STEAL_GOVERNOR is introduced in subsequent patches, which enables this driver. This driver will select CONFIG_PREFERRED_CPU. This makes configs driven by user preference/configuration. When the driver is disabled, preferred CPUs remain the same as active CPUs.
The file layout of the driver is kept simple for now. The code is in drivers/virt/steal_governor.c, and the configs are part of drivers/virt/Kconfig.
The main structure of the steal governor contains:
- work, delay: Deferred periodic work function variables.
- steal, time: Used to calculate deltas during periodic work.
- interval_ms, high_threshold, low_threshold: Tuning knobs for the steal governor.
While there, add MAINTAINERS entry for this new driver.
--- Documentation/driver-api/index.rst | 1 + Documentation/driver-api/steal-governor.rst | 151 ++++++++++++++++++++ MAINTAINERS | 9 ++ drivers/virt/steal_governor.c | 78 ++++++++++ 4 files changed, 239 insertions(+) create mode 100644 Documentation/driver-api/steal-governor.rst create mode 100644 drivers/virt/steal_governor.c
Documentation/driver-api/index.rst +1 -0
@@ -139,6 +139,7 @@ Subsystem-specific APIs sm501 soundwire/index spi + steal-governor surface_aggregator/index switchtec sync_file
Documentation/driver-api/steal-governor.rst new file +151 -0
151 lines not on this page · raw message
MAINTAINERS +9 -0
@@ -26223,6 +26223,15 @@ F: rust/helpers/jump_label.c F: rust/kernel/generated_arch_static_branch_asm.rs.S F: rust/kernel/jump_label.rs +STEAL GOVERNOR DRIVER +M: Shrikanth Hegde <[email protected]> +R: Yury Norov <[email protected]> +L: [email protected] +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core +F: Documentation/driver-api/steal-governor.rst +F: drivers/virt/steal_governor.c + STI AUDIO (ASoC) DRIVERS M: Arnaud Pouliquen <[email protected]> L: [email protected]
drivers/virt/steal_governor.c new file +78 -0
78 lines not on this page · raw message
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 10/13] virt: Introduce steal governor driver
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 23 lines
Introduce a new driver in virt named
steal_governor. This driver will compute the steal time and drive the policy decisions regarding the preferred CPU state.Note that this driver is strictly intended for actual guests. Hence block it on Xen dom0.
More details can be found inDocumentation/driver-api/steal-governor.rst.A new kconfig called
STEAL_GOVERNORis introduced in subsequent patches, which enables this driver. This driver will selectCONFIG_PREFERRED_CPU. This makes configs driven by user preference/configuration. When the driver is disabled, preferred CPUs remain the same as active CPUs.The file layout of the driver is kept simple for now. The code is in
drivers/virt/steal_governor.c, and the configs are part ofdrivers/virt/Kconfig.The main structure of the steal governor contains:
- work, delay: Deferred periodic work function variables.
- steal, time: Used to calculate deltas during periodic work.
-interval_ms,high_threshold,low_threshold: Tuning knobs for the steal governor.
Quoted 5 lines
While there, add
MAINTAINERSentry for this new driver.Suggested-by Yury Norov <[email protected]>Suggested-by K Prateek Nayak <[email protected]>Signed-off-by Shrikanth Hegde <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
[tip: sched/core] virt: Introduce steal governor driver
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 9a8e740ee9f69ec857ffa0c76cf1a01d1cb7360f Gitweb: https://git.kernel.org/tip/9a8e740ee9f69ec857ffa0c76cf1a01d1cb7360f Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:25 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:43 +02:00
virt: Introduce steal governor driver
Introduce a new driver in virt named steal_governor. This driver will compute the steal time and drive the policy decisions regarding the preferred CPU state.
Note that this driver is strictly intended for actual guests. Hence block it on Xen dom0.
More details can be found in Documentation/driver-api/steal-governor.rst.
A new kconfig called STEAL_GOVERNOR is introduced in subsequent patches, which enables this driver. This driver will select CONFIG_PREFERRED_CPU. This makes configs driven by user preference/configuration. When the driver is disabled, preferred CPUs remain the same as active CPUs.
The file layout of the driver is kept simple for now. The code is in drivers/virt/steal_governor.c, and the configs are part of drivers/virt/Kconfig.
The main structure of the steal governor contains:
- work, delay: Deferred periodic work function variables.
- steal, time: Used to calculate deltas during periodic work.
- interval_ms, high_threshold, low_threshold: Tuning knobs for the steal governor.
While there, add MAINTAINERS entry for this new driver.
--- Documentation/driver-api/index.rst | 1 +- Documentation/driver-api/steal-governor.rst | 151 +++++++++++++++++++- MAINTAINERS | 9 +- drivers/virt/steal_governor.c | 78 ++++++++++- 4 files changed, 239 insertions(+) create mode 100644 Documentation/driver-api/steal-governor.rst create mode 100644 drivers/virt/steal_governor.c
Documentation/driver-api/index.rst +1 -0
@@ -139,6 +139,7 @@ Subsystem-specific APIs sm501 soundwire/index spi + steal-governor surface_aggregator/index switchtec sync_file
Documentation/driver-api/steal-governor.rst new file +151 -0
151 lines not on this page · raw message
MAINTAINERS +9 -0
@@ -26223,6 +26223,15 @@ F: rust/helpers/jump_label.c F: rust/kernel/generated_arch_static_branch_asm.rs.S F: rust/kernel/jump_label.rs +STEAL GOVERNOR DRIVER +M: Shrikanth Hegde <[email protected]> +R: Yury Norov <[email protected]> +L: [email protected] +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core +F: Documentation/driver-api/steal-governor.rst +F: drivers/virt/steal_governor.c + STI AUDIO (ASoC) DRIVERS M: Arnaud Pouliquen <[email protected]> L: [email protected]
drivers/virt/steal_governor.c new file +78 -0
78 lines not on this page · raw message
Shrikanth Hegde
maintainerPATCH v14 11/13 virt/steal_governor: Add control knobs for handling steal values
In reply to Shrikanth HegdeThese are the knobs to control the steal_governor.
interval_ms:
How often steal governor checks for steal time. (Default: 1000 i.e 1 second)
This controls how fast steal governor driver reacts to changes to the contention of physical CPUs.
Can be set between 100 to 100000. i.e. 100ms to 100seconds. 100ms is kept as minimum to ensure few meaningful steal values accumulate even with HZ=100.
low_threshold:
lower threshold value in percentage * 100.
(Default: 200, i.e 2% steal is considered as low threshold) This determines what values should be considered as nil/no steal values. When steal governor see steal ratio is below or equal to this value, it will increase the preferred CPUs by 1 core. Having value as zero might cause oscillations
high_threshold:
higher threshold value in percentage * 100
(Default: 500, i.e 5% steal is considered as high threshold) This determines what values should be considered as high steal values. When steal governor sees steal ratio is higher than this value, it will reduce the preferred CPUs by 1 core.
module_param_cb methods are used to do the validation checks. This helps to ensure one configures sane values.
Since low and high are dependent, that check is done at module init.
Notes:
- Parameters values can't be changed at runtime. One has to unload the module and change it. Hence recommended to build it as module.
- Default values may not work well for all configurations. Tune it according to the system under test.
Documentation is available at: Documentation/driver-api/steal-governor.rst
--- drivers/virt/steal_governor.c | 73 ++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 2 deletions(-)
drivers/virt/steal_governor.c +71 -2
@@ -40,7 +40,11 @@ struct steal_governor {struct delayed_work work; }; -static struct steal_governor sg_ctx; +static struct steal_governor sg_ctx = { + .interval_ms =1000 ,/* 1 second */ + .high_threshold =500 ,/* 5% */ + .low_threshold =200 ,/* 2% */ +};static void restore_preferred_to_active (void ) {
@@ -51,6 +55,62 @@ static void restore_preferred_to_active(void)set_cpu_preferred (cpu, true); } +static int param_set_interval_ms (const char *val,const struct kernel_param *kp) +{ +unsigned int interval; +int ret; + + ret =kstrtouint (val,0 , &interval); +if (ret) +return ret; + +if (interval <100 || interval >100000 ) { +pr_err ("interval_ms must be between 100 and 100000\n" ); +return -EINVAL ; + } + +return param_set_uint (val, kp); +} + +static const struct kernel_param_ops interval_ms_ops = { + .set = param_set_interval_ms, + .get = param_get_uint, +}; + +module_param_cb (interval_ms, &interval_ms_ops, &sg_ctx.interval_ms,0444 ); +MODULE_PARM_DESC (interval_ms, +"Sampling frequency in milliseconds. default: 1000" ); + +static int param_set_high_threshold (const char *val,const struct kernel_param *kp) +{ +unsigned int threshold; +int ret; + + ret =kstrtouint (val,0 , &threshold); +if (ret) +return ret; + +if (threshold >=100 *100 ) { +pr_err ("high_threshold (%u) can't be more than 99.99%%\n" , threshold); +return -EINVAL ; + } + +return param_set_uint (val, kp); +} + +static const struct kernel_param_ops high_threshold_ops = { + .set = param_set_high_threshold, + .get = param_get_uint, +}; + +module_param_cb (high_threshold, &high_threshold_ops, &sg_ctx.high_threshold,0444 ); +MODULE_PARM_DESC (high_threshold, +"High steal threshold. default: 500 i.e 5%. Must be > low_threshold" ); + +module_param_named (low_threshold, sg_ctx.low_threshold, uint,0444 ); +MODULE_PARM_DESC (low_threshold, +"Low steal threshold. default: 200 i.e 2%. Must be < high_threshold" ); +static int __init steal_governor_init (void ) {#ifdef CONFIG_XEN
@@ -60,7 +120,16 @@ static int __init steal_governor_init(void) }#endif -pr_info ("enabled\n" ); +if (sg_ctx.low_threshold >= sg_ctx.high_threshold) { +pr_err ("low_threshold (%u) must be less than high_threshold (%u)\n" , + sg_ctx.low_threshold, sg_ctx.high_threshold); +return -EINVAL ; + } + + sg_ctx.delay =msecs_to_jiffies (sg_ctx.interval_ms); +pr_info ("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n" , + sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold); +return 0 ; }
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 11/13] virt/steal_governor: Add control knobs for handling steal values
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines
These are the knobs to control the
steal_governor.
interval_ms:
How often steal governor checks for steal time. (Default: 1000 i.e 1 second)
This controls how fast steal governor driver reacts to changes to the contention of physical CPUs.
Can be set between 100 to 100000. i.e. 100ms to 100seconds. 100ms is kept as minimum to ensure few meaningful steal values accumulate even with HZ=100.
low_threshold:
lower threshold value in percentage * 100.
(Default: 200, i.e 2% steal is considered as low threshold) This determines what values should be considered as nil/no steal values. [ ... ]
Quoted 4 lines
Documentation is available at:
Documentation/driver-api/steal-governor.rstSuggested-by Yury Norov <[email protected]>Signed-off-by Shrikanth Hegde <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
[tip: sched/core] virt/steal_governor: Add control knobs for handling steal values
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4b9302d494fffee7318bd9cdd2021198b6a7badd Gitweb: https://git.kernel.org/tip/4b9302d494fffee7318bd9cdd2021198b6a7badd Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:26 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:43 +02:00
virt/steal_governor: Add control knobs for handling steal values
These are the knobs to control the steal_governor.
interval_ms:
How often steal governor checks for steal time. (Default: 1000 i.e 1 second)
This controls how fast steal governor driver reacts to changes to the contention of physical CPUs.
Can be set between 100 to 100000. i.e. 100ms to 100seconds. 100ms is kept as minimum to ensure few meaningful steal values accumulate even with HZ=100.
low_threshold:
lower threshold value in percentage * 100.
(Default: 200, i.e 2% steal is considered as low threshold) This determines what values should be considered as nil/no steal values. When steal governor see steal ratio is below or equal to this value, it will increase the preferred CPUs by 1 core. Having value as zero might cause oscillations
high_threshold:
higher threshold value in percentage * 100
(Default: 500, i.e 5% steal is considered as high threshold) This determines what values should be considered as high steal values. When steal governor sees steal ratio is higher than this value, it will reduce the preferred CPUs by 1 core.
module_param_cb methods are used to do the validation checks. This helps to ensure one configures sane values.
Since low and high are dependent, that check is done at module init.
Notes:
- Parameters values can't be changed at runtime. One has to unload the module and change it. Hence recommended to build it as module.
- Default values may not work well for all configurations. Tune it according to the system under test.
Documentation is available at: Documentation/driver-api/steal-governor.rst
--- drivers/virt/steal_governor.c | 73 +++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 2 deletions(-)
drivers/virt/steal_governor.c +71 -2
@@ -40,7 +40,11 @@ struct steal_governor {struct delayed_work work; }; -static struct steal_governor sg_ctx; +static struct steal_governor sg_ctx = { + .interval_ms =1000 ,/* 1 second */ + .high_threshold =500 ,/* 5% */ + .low_threshold =200 ,/* 2% */ +};static void restore_preferred_to_active (void ) {
@@ -51,6 +55,62 @@ static void restore_preferred_to_active(void)set_cpu_preferred (cpu, true); } +static int param_set_interval_ms (const char *val,const struct kernel_param *kp) +{ +unsigned int interval; +int ret; + + ret =kstrtouint (val,0 , &interval); +if (ret) +return ret; + +if (interval <100 || interval >100000 ) { +pr_err ("interval_ms must be between 100 and 100000\n" ); +return -EINVAL ; + } + +return param_set_uint (val, kp); +} + +static const struct kernel_param_ops interval_ms_ops = { + .set = param_set_interval_ms, + .get = param_get_uint, +}; + +module_param_cb (interval_ms, &interval_ms_ops, &sg_ctx.interval_ms,0444 ); +MODULE_PARM_DESC (interval_ms, +"Sampling frequency in milliseconds. default: 1000" ); + +static int param_set_high_threshold (const char *val,const struct kernel_param *kp) +{ +unsigned int threshold; +int ret; + + ret =kstrtouint (val,0 , &threshold); +if (ret) +return ret; + +if (threshold >=100 *100 ) { +pr_err ("high_threshold (%u) can't be more than 99.99%%\n" , threshold); +return -EINVAL ; + } + +return param_set_uint (val, kp); +} + +static const struct kernel_param_ops high_threshold_ops = { + .set = param_set_high_threshold, + .get = param_get_uint, +}; + +module_param_cb (high_threshold, &high_threshold_ops, &sg_ctx.high_threshold,0444 ); +MODULE_PARM_DESC (high_threshold, +"High steal threshold. default: 500 i.e 5%. Must be > low_threshold" ); + +module_param_named (low_threshold, sg_ctx.low_threshold, uint,0444 ); +MODULE_PARM_DESC (low_threshold, +"Low steal threshold. default: 200 i.e 2%. Must be < high_threshold" ); +static int __init steal_governor_init (void ) {#ifdef CONFIG_XEN
@@ -60,7 +120,16 @@ static int __init steal_governor_init(void) }#endif -pr_info ("enabled\n" ); +if (sg_ctx.low_threshold >= sg_ctx.high_threshold) { +pr_err ("low_threshold (%u) must be less than high_threshold (%u)\n" , + sg_ctx.low_threshold, sg_ctx.high_threshold); +return -EINVAL ; + } + + sg_ctx.delay =msecs_to_jiffies (sg_ctx.interval_ms); +pr_info ("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n" , + sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold); +return 0 ; }
Shrikanth Hegde
maintainerPATCH v14 12/13 virt/steal_governor: Implement steal_governor policy loop
In reply to Shrikanth HegdeSchedule work at regular intervals to implement the steal_governor policy loop, which monitors steal time and takes action on the state of preferred CPUs. The interval is determined by the interval_ms parameter. schedule_delayed_work() is used since interval_ms is on the order of milliseconds and the work does not need to happen instantly.
Periodic policy loop essentially does:
- Gets the total/delta steal values and cpus to use steal_ratio.
- Calculate the steal_ratio as below.
steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())
It is calculated this way to consider the fractional values of steal time. I.e 10 means 0.1% steal time. A few tricks such as divide by 10,000 are used to avoid possible overflow.
- If steal ratio is higher than high threshold, call the method to reduce the preferred CPUs.
- If steal ratio is lower or equal to low threshold, call the method to increase the preferred CPUs.
- If the steal ratio falls in between, no action is taken.
- Ensures design constraints always met.
1. At least one core/CPU must be there in preferred mask.
2. preferred CPUs is subset of active CPUs.
If not met, then restore preferred CPUs to active and stop requeue of the work. Driver is effectively non-functional after that.
Note that design checks are always performed. This helps avoid placing driver-specific design constraints inside the core CPU hotplug mechanism. User may offline specific set of CPUs that could leave the preferred mask as empty. With the design check performed always, driver gracefully shuts down upon detecting that edge case.
In order to help the above loop, a few helper functions have been added.
1. get_system_steal_time()
- steal governor takes global view of steal time instead of individual vCPU. Collect the steal values across the vCPUs of interest.
- Sum up steal time values across possible CPUs. This helps to keep it a monotonically increasing number and avoids spikes due to CPU hotplug.
2. decrease_preferred_cpus()
- Called when there is high steal time. It needs to decide which CPUs to mark as non-preferred.
- Get first housekeeping CPU and its core mask. Mark it as protected core. This helps to keep at least one core as preferred. (kernel ensures at least one housekeeping CPU stays active.)
- Find the last CPU outside of this protected core mask. i.e target CPU
- Based on that target CPU, get its sibling and mark them as non-preferred.
3. increase_preferred_cpus()
- Called when there is low steal time. It needs to decide which CPUs to mark as preferred and set that state.
- Get the first active non-preferred CPUs. This likely is the last set of CPUs being marked as non-preferred.
- get the siblings of that CPU and mark them as preferred.
4. get_system_cpus()
- informs how many CPUs needs to be considered for steal_ratio calculations.
- Since only active CPUs effectively contribute to steal time delta, returns number of active CPUs. This also helps to avoid dilution of thresholds in sparsely populated systems.
Notes:
1. Using core instead of individual CPUs performs better as SMT is quite common and some hypervisor such as powerVM does core scheduling.
2. This doesn't do any NUMA splicing to keep the code simpler and minimal overhead. Current code expects CPUs spread uniformly across NUMA nodes.
--- drivers/virt/steal_governor.c | 149 ++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+)
drivers/virt/steal_governor.c +149 -0
@@ -13,13 +13,18 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/cleanup.h> #include <linux/cpuhplock.h> #include <linux/cpumask.h> #include <linux/init.h> #include <linux/kernel.h> +#include <linux/kernel_stat.h> #include <linux/kconfig.h> #include <linux/ktime.h> +#include <linux/math64.h> #include <linux/module.h> +#include <linux/sched/isolation.h> +#include <linux/topology.h> #include <linux/types.h> #include <linux/workqueue.h> #ifdef CONFIG_XEN
@@ -111,6 +116,145 @@ module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444);MODULE_PARM_DESC (low_threshold,"Low steal threshold. default: 200 i.e 2%. Must be < high_threshold" ); +/* Return collective steal time across system. */ +static u64 get_system_steal_time (void ) +{ +return kcpustat_field_total (CPUTIME_STEAL , cpu_possible_mask); +} + +/* Return number of CPUs to consider for steal ratio. */ +static unsigned int get_system_cpus (void ) +{ +return num_active_cpus (); +} + +/* +* Called when the steal governor detects high physical CPU contention. +* It finds the last active core in the preferred mask and mark those +* CPUs as non-preferred. +* +* Must ensure: +* - at least one core is always kept as preferred +* - preferred is always subset of active. +*/ +static void decrease_preferred_cpus (void ) +{ +const struct cpumask *first_hk_core; +int target_cpu = nr_cpu_ids; +int cpu; + +guard (cpus_read_lock)(); + cpu =cpumask_first_and (housekeeping_cpumask (HK_TYPE_KERNEL_NOISE ), + cpu_preferred_mask); +if (cpu >= nr_cpu_ids) +return ; + +/* Always leave first housekeeping core as preferred. */ + first_hk_core =topology_sibling_cpumask (cpu); + cpu =cpumask_last (cpu_preferred_mask); +if (cpu >= nr_cpu_ids) +return ; + +/* Find the last CPU which doesn't belong to that first hk_core. */ +if (!cpumask_test_cpu (cpu, first_hk_core)) { + target_cpu = cpu; + }else { +for_each_cpu_andnot (cpu, cpu_preferred_mask, first_hk_core) + target_cpu = cpu; + } + +/* Only the first housekeeping core remains */ +if (target_cpu >= nr_cpu_ids) +return ; + +for_each_cpu_and (cpu,topology_sibling_cpumask (target_cpu), + cpu_preferred_mask) +set_cpu_preferred (cpu, false); +} + +/* +* Called when the steal governor detects no/low physical CPU contention. +* It finds the first active core outside of preferred mask and mark +* those CPUs as preferred. +* +* Must ensure preferred is subset of active. +*/ +static void increase_preferred_cpus (void ) +{ +int first_cpu, cpu; + +guard (cpus_read_lock)(); + first_cpu =cpumask_first_andnot (cpu_active_mask, cpu_preferred_mask); + +/* All CPUs are preferred. Nothing to increase further */ +if (first_cpu >= nr_cpu_ids) +return ; + +for_each_cpu_and (cpu,topology_sibling_cpumask (first_cpu), + cpu_active_mask) +set_cpu_preferred (cpu, true); +} + +static bool preferred_cpus_valid (void ) +{ +if (cpumask_empty (cpu_preferred_mask)) { +pr_err ("empty preferred mask. stopping\n" ); +return false; + } + +if (!cpumask_subset (cpu_preferred_mask, cpu_active_mask)) { +pr_err ("preferred: %*pbl is not subset of active: %*pbl, stopping\n" , +cpumask_pr_args (cpu_preferred_mask), +cpumask_pr_args (cpu_active_mask)); +return false; + } + +return true; +} + +static void steal_governor_loop (struct work_struct *work) +{ +u64 curr_steal, delta_steal, delta_ns, steal_ratio; +ktime_t now; + + now =ktime_get (); + delta_ns =ktime_to_ns (ktime_sub (now, sg_ctx.time)); + +if (unlikely (delta_ns <NSEC_PER_MSEC )) { +pr_err_ratelimited ("work scheduled too soon delta_ns: %llu\n" , delta_ns); + goto requeue_work; + } + + curr_steal =get_system_steal_time (); + delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal :0 ; + sg_ctx.steal = curr_steal; + sg_ctx.time = now; + +/* +* steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus()) +* To avoid possible overflow, divide the denominator early. +* Note minimum interval is 100ms. +*/ + delta_ns =max_t (u64,div_u64 (delta_ns *get_system_cpus (),10000 ),1 ); + steal_ratio =div64_u64 (delta_steal, delta_ns); + +if (steal_ratio > sg_ctx.high_threshold) +decrease_preferred_cpus (); +else if (steal_ratio <= sg_ctx.low_threshold) +increase_preferred_cpus (); +/* +* else: steal ratio is within bounds. Still do design checks so that +* module restores to active if CPU hotplug breaks those assumptions. +*/ +if (!preferred_cpus_valid ()) { +restore_preferred_to_active (); +return ; + } + +requeue_work: +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay); +} +static int __init steal_governor_init (void ) {#ifdef CONFIG_XEN
@@ -127,6 +271,10 @@ static int __init steal_governor_init(void) } sg_ctx.delay =msecs_to_jiffies (sg_ctx.interval_ms); +INIT_DELAYED_WORK (&sg_ctx.work, steal_governor_loop); + sg_ctx.steal =get_system_steal_time (); + sg_ctx.time =ktime_get (); +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);pr_info ("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n" , sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
@@ -135,6 +283,7 @@ static int __init steal_governor_init(void) static void __exit steal_governor_exit(void) { +disable_delayed_work_sync (&sg_ctx.work);restore_preferred_to_active ();pr_info ("disabled\n" ); }
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 12/13] virt/steal_governor: Implement steal_governor policy loop
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Missing bounds check on interval_ms allows infinite workqueue spinloop - [High] TOCTOU data race…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Missing bounds check on interval_ms allows infinite workqueue spinloop
- [High] TOCTOU data race in preferred_cpus_valid() causes erroneous driver shutdown
- [Medium] Integer overflow in steal_ratio denominator when VM is paused
--
commit 5f38399332407b64e0ce2da43681c27f1e1287d5
Author: Shrikanth Hegde <[email protected]>
virt/steal_governor: Implement steal_governor policy loop
Schedule work at regular intervals to implement the steal_governor policy loop, which monitors steal time and takes action on the state of preferred CPUs. The interval is determined by the interval_ms parameter.
Quoted 4 lines
drivers/virt/steal_governor.c +0
-0
[ ... ]
drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 205
Quoted 8 lines
drivers/virt/steal_governor.c +8
-0+static bool preferred_cpus_valid (void ) +{ +if (cpumask_empty (cpu_preferred_mask)) { +pr_err ("empty preferred mask. stopping\n" ); +return false; + } + +if (!cpumask_subset (cpu_preferred_mask, cpu_active_mask)) {
[Severity: High]
Does this lockless read of cpu_preferred_mask and cpu_active_mask race with CPU hotplug?
Since CPU hotplug clears the preferred mask bit before the active mask bit, it seems cpumask_subset() can read a torn state where a word is read from the preferred mask before the bit is cleared, and the corresponding word is read from the active mask after the bit is cleared, returning false.
Quoted 10 lines
drivers/virt/steal_governor.c +10
-0+pr_err ("preferred: %*pbl is not subset of active: %*pbl, stopping\n" , +cpumask_pr_args (cpu_preferred_mask), +cpumask_pr_args (cpu_active_mask)); +return false; + } + +return true; +} + +static void steal_governor_loop (struct work_struct *work)
drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 225
Quoted 10 lines
drivers/virt/steal_governor.c +10
-0+{ +u64 curr_steal, delta_steal, delta_ns, steal_ratio; +ktime_t now; + + now =ktime_get (); + delta_ns =ktime_to_ns (ktime_sub (now, sg_ctx.time)); + +if (unlikely (delta_ns <NSEC_PER_MSEC )) { +pr_err_ratelimited ("work scheduled too soon delta_ns: %llu\n" , delta_ns); + goto requeue_work;
[Severity: High]
Can this create an infinite spinloop if interval_ms is set to 0?
If interval_ms is 0, the delay is 0 jiffies. When the work executes, it will take this branch because delta_ns < 1ms. It jumps directly to requeue_work without updating sg_ctx.time, which schedules it again with 0 delay. This will continuously spin and consume an entire CPU core.
drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 238
Quoted 13 lines
drivers/virt/steal_governor.c +13
-0+ } + + curr_steal =get_system_steal_time (); + delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal :0 ; + sg_ctx.steal = curr_steal; + sg_ctx.time = now; + +/* +* steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus()) +* To avoid possible overflow, divide the denominator early. +* Note minimum interval is 100ms. +*/ + delta_ns =max_t (u64,div_u64 (delta_ns *get_system_cpus (),10000 ),1 );
[Severity: Medium]
Will this silently overflow the 64-bit multiplication if the VM is paused for an extended period?
Since the multiplication of delta_ns * get_system_cpus() happens before the division, a large time delta (e.g., a large VM suspended for a long time) can overflow U64_MAX, leading to a wildly incorrect steal_ratio calculation.
drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 249
Quoted 11 lines
drivers/virt/steal_governor.c +11
-0+ steal_ratio =div64_u64 (delta_steal, delta_ns); + +if (steal_ratio > sg_ctx.high_threshold) +decrease_preferred_cpus (); +else if (steal_ratio <= sg_ctx.low_threshold) +increase_preferred_cpus (); +/* +* else: steal ratio is within bounds. Still do design checks so that +* module restores to active if CPU hotplug breaks those assumptions. +*/ +if (!preferred_cpus_valid ()) {
[Severity: High]
Does calling preferred_cpus_valid() here without cpus_read_lock() lead to a false teardown?
If a hotplug operation is happening concurrently, the torn state read in preferred_cpus_valid() will cause the governor to erroneously shut down the policy loop permanently.
drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 255
Quoted 6 lines
drivers/virt/steal_governor.c +6
-0+restore_preferred_to_active (); +return ; + } + +requeue_work: +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);
[Severity: High]
Does this immediately spin if sg_ctx.delay is 0, as mentioned above?
Quoted 4 lines
drivers/virt/steal_governor.c +2
-0+} +static int __init steal_governor_init (void ) {
[ ... ]
Quoted 1 line
sg_ctx.delay =
msecs_to_jiffies(sg_ctx.interval_ms);
[Severity: High]
Is it possible for sg_ctx.interval_ms to be 0 here?
The module parameter interval_ms doesn't appear to be bounds-checked before this point, meaning it could be set to 0 and trigger the infinite workqueue spinloop.
Quoted 7 lines
drivers/virt/steal_governor.c +4
-0+INIT_DELAYED_WORK (&sg_ctx.work, steal_governor_loop); + sg_ctx.steal =get_system_steal_time (); + sg_ctx.time =ktime_get (); +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);pr_info ("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n" , sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Shrikanth Hegde
maintainerRe: [PATCH v14 12/13] virt/steal_governor: Implement steal_governor policy loop
I don't see any real concerns so far. Some optimizations yes, but they can be deferred post the merge if they are necessary.
Quoted 33 lines · On 9/28/26 11:21 AM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Missing bounds check oninterval_msallows infinite workqueue spinloop
- [High] TOCTOU data race inpreferred_cpus_valid()causes erroneous driver shutdown
- [Medium] Integer overflow insteal_ratiodenominator when VM is paused
--commit
5f38399332407b64e0ce2da43681c27f1e1287d5
Author: Shrikanth Hegde <[email protected]>
virt/steal_governor: Implementsteal_governorpolicy loopSchedule work at regular intervals to implement the
steal_governorpolicy loop, which monitors steal time and takes action on the state of preferred CPUs. The interval is determined by theinterval_msparameter.Quoted 4 lines
drivers/virt/steal_governor.c +0
-0[ ... ]
drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 205
Quoted 8 lines
drivers/virt/steal_governor.c +8
-0+static bool preferred_cpus_valid (void ) +{ +if (cpumask_empty (cpu_preferred_mask)) { +pr_err ("empty preferred mask. stopping\n" ); +return false; + } + +if (!cpumask_subset (cpu_preferred_mask, cpu_active_mask)) {[Severity: High]
Does this lockless read ofcpu_preferred_maskandcpu_active_maskrace with CPU hotplug?
Quoted 4 lines
Since CPU hotplug clears the preferred mask bit before the active mask bit, it seems
cpumask_subset()can read a torn state where a word is read from the preferred mask before the bit is cleared, and the corresponding word is read from the active mask after the bit is cleared, returning false.
Update ordering is taken care. Not an issue is clear/set bit involves atomics.
Quoted 25 lines
Quoted 11 lines
drivers/virt/steal_governor.c +10
-0+pr_err ("preferred: %*pbl is not subset of active: %*pbl, stopping\n" , +cpumask_pr_args (cpu_preferred_mask), +cpumask_pr_args (cpu_active_mask)); +return false; + } + +return true; +} + +static void steal_governor_loop (struct work_struct *work)drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 225
Quoted 10 lines
drivers/virt/steal_governor.c +10
-0+{ +u64 curr_steal, delta_steal, delta_ns, steal_ratio; +ktime_t now; + + now =ktime_get (); + delta_ns =ktime_to_ns (ktime_sub (now, sg_ctx.time)); + +if (unlikely (delta_ns <NSEC_PER_MSEC )) { +pr_err_ratelimited ("work scheduled too soon delta_ns: %llu\n" , delta_ns); + goto requeue_work;[Severity: High]
Can this create an infinite spinloop ifinterval_msis set to 0?
Quoted 5 lines
If
interval_msis 0, the delay is 0 jiffies. When the work executes, it will take this branch becausedelta_ns< 1ms. It jumps directly torequeue_workwithout updating sg_ctx.time, which schedules it again with 0 delay. This will continuously spin and consume an entire CPU core.
No. Interval can't be set to 0. Minimum is 100ms.
Quoted 18 lines
drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 238
Quoted 13 lines
drivers/virt/steal_governor.c +13
-0+ } + + curr_steal =get_system_steal_time (); + delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal :0 ; + sg_ctx.steal = curr_steal; + sg_ctx.time = now; + +/* +* steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus()) +* To avoid possible overflow, divide the denominator early. +* Note minimum interval is 100ms. +*/ + delta_ns =max_t (u64,div_u64 (delta_ns *get_system_cpus (),10000 ),1 );[Severity: Medium]
Will this silently overflow the 64-bit multiplication if the VM is paused for an extended period?
Quoted 4 lines
Since the multiplication of
delta_ns*get_system_cpus()happens before the division, a large time delta (e.g., a large VM suspended for a long time) can overflowU64_MAX, leading to a wildly incorrectsteal_ratiocalculation.
Not a reason to fret about. It is way too long duration.
Quoted 20 lines
drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 249
Quoted 11 lines
drivers/virt/steal_governor.c +11
-0+ steal_ratio =div64_u64 (delta_steal, delta_ns); + +if (steal_ratio > sg_ctx.high_threshold) +decrease_preferred_cpus (); +else if (steal_ratio <= sg_ctx.low_threshold) +increase_preferred_cpus (); +/* +* else: steal ratio is within bounds. Still do design checks so that +* module restores to active if CPU hotplug breaks those assumptions. +*/ +if (!preferred_cpus_valid ()) {[Severity: High]
Does callingpreferred_cpus_valid()here withoutcpus_read_lock()lead to a false teardown?If a hotplug operation is happening concurrently, the torn state read in
preferred_cpus_valid()will cause the governor to erroneously shut down the policy loop permanently.
Quoted 9 lines
drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 255
Quoted 6 lines
drivers/virt/steal_governor.c +6
-0+restore_preferred_to_active (); +return ; + } + +requeue_work: +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);[Severity: High]
Does this immediately spin if sg_ctx.delay is 0, as mentioned above?
As said above, interval cannot be less than 100ms.
Quoted 6 lines
Quoted 5 lines
drivers/virt/steal_governor.c +2
-0+} +static int __init steal_governor_init (void ) {[ ... ]
Quoted 8 lines
Quoted 1 line
sg_ctx.delay = msecs_to_jiffies(sg_ctx.interval_ms);[Severity: High]
Is it possible for sg_ctx.interval_ms to be 0 here?The module parameter
interval_msdoesn't appear to be bounds-checked before this point, meaning it could be set to 0 and trigger the infinite workqueue spinloop.
As said above, interval cannot be less than 100ms.
Quoted 9 lines
Quoted 8 lines
drivers/virt/steal_governor.c +4
-0+INIT_DELAYED_WORK (&sg_ctx.work, steal_governor_loop); + sg_ctx.steal =get_system_steal_time (); + sg_ctx.time =ktime_get (); +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);pr_info ("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n" , sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
[tip: sched/core] virt/steal_governor: Implement steal_governor policy loop
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 27d47ebce4d6490ce093031a48cc2d81bf9997a3 Gitweb: https://git.kernel.org/tip/27d47ebce4d6490ce093031a48cc2d81bf9997a3 Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:27 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:43 +02:00
virt/steal_governor: Implement steal_governor policy loop
Schedule work at regular intervals to implement the steal_governor policy loop, which monitors steal time and takes action on the state of preferred CPUs. The interval is determined by the interval_ms parameter. schedule_delayed_work() is used since interval_ms is on the order of milliseconds and the work does not need to happen instantly.
Periodic policy loop essentially does:
- Gets the total/delta steal values and cpus to use steal_ratio.
- Calculate the steal_ratio as below.
steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())
It is calculated this way to consider the fractional values of steal time. I.e 10 means 0.1% steal time. A few tricks such as divide by 10,000 are used to avoid possible overflow.
- If steal ratio is higher than high threshold, call the method to reduce the preferred CPUs.
- If steal ratio is lower or equal to low threshold, call the method to increase the preferred CPUs.
- If the steal ratio falls in between, no action is taken.
- Ensures design constraints always met.
1. At least one core/CPU must be there in preferred mask.
2. preferred CPUs is subset of active CPUs.
If not met, then restore preferred CPUs to active and stop requeue of the work. Driver is effectively non-functional after that.
Note that design checks are always performed. This helps avoid placing driver-specific design constraints inside the core CPU hotplug mechanism. User may offline specific set of CPUs that could leave the preferred mask as empty. With the design check performed always, driver gracefully shuts down upon detecting that edge case.
In order to help the above loop, a few helper functions have been added.
1. get_system_steal_time()
- steal governor takes global view of steal time instead of individual vCPU. Collect the steal values across the vCPUs of interest.
- Sum up steal time values across possible CPUs. This helps to keep it a monotonically increasing number and avoids spikes due to CPU hotplug.
2. decrease_preferred_cpus()
- Called when there is high steal time. It needs to decide which CPUs to mark as non-preferred.
- Get first housekeeping CPU and its core mask. Mark it as protected core. This helps to keep at least one core as preferred. (kernel ensures at least one housekeeping CPU stays active.)
- Find the last CPU outside of this protected core mask. i.e target CPU
- Based on that target CPU, get its sibling and mark them as non-preferred.
3. increase_preferred_cpus()
- Called when there is low steal time. It needs to decide which CPUs to mark as preferred and set that state.
- Get the first active non-preferred CPUs. This likely is the last set of CPUs being marked as non-preferred.
- get the siblings of that CPU and mark them as preferred.
4. get_system_cpus()
- informs how many CPUs needs to be considered for steal_ratio calculations.
- Since only active CPUs effectively contribute to steal time delta, returns number of active CPUs. This also helps to avoid dilution of thresholds in sparsely populated systems.
Notes:
1. Using core instead of individual CPUs performs better as SMT is quite common and some hypervisor such as powerVM does core scheduling.
2. This doesn't do any NUMA splicing to keep the code simpler and minimal overhead. Current code expects CPUs spread uniformly across NUMA nodes.
--- drivers/virt/steal_governor.c | 149 +++++++++++++++++++++++++++++++++- 1 file changed, 149 insertions(+)
drivers/virt/steal_governor.c +149 -0
@@ -13,13 +13,18 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/cleanup.h> #include <linux/cpuhplock.h> #include <linux/cpumask.h> #include <linux/init.h> #include <linux/kernel.h> +#include <linux/kernel_stat.h> #include <linux/kconfig.h> #include <linux/ktime.h> +#include <linux/math64.h> #include <linux/module.h> +#include <linux/sched/isolation.h> +#include <linux/topology.h> #include <linux/types.h> #include <linux/workqueue.h> #ifdef CONFIG_XEN
@@ -111,6 +116,145 @@ module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444);MODULE_PARM_DESC (low_threshold,"Low steal threshold. default: 200 i.e 2%. Must be < high_threshold" ); +/* Return collective steal time across system. */ +static u64 get_system_steal_time (void ) +{ +return kcpustat_field_total (CPUTIME_STEAL , cpu_possible_mask); +} + +/* Return number of CPUs to consider for steal ratio. */ +static unsigned int get_system_cpus (void ) +{ +return num_active_cpus (); +} + +/* +* Called when the steal governor detects high physical CPU contention. +* It finds the last active core in the preferred mask and mark those +* CPUs as non-preferred. +* +* Must ensure: +* - at least one core is always kept as preferred +* - preferred is always subset of active. +*/ +static void decrease_preferred_cpus (void ) +{ +const struct cpumask *first_hk_core; +int target_cpu = nr_cpu_ids; +int cpu; + +guard (cpus_read_lock)(); + cpu =cpumask_first_and (housekeeping_cpumask (HK_TYPE_KERNEL_NOISE ), + cpu_preferred_mask); +if (cpu >= nr_cpu_ids) +return ; + +/* Always leave first housekeeping core as preferred. */ + first_hk_core =topology_sibling_cpumask (cpu); + cpu =cpumask_last (cpu_preferred_mask); +if (cpu >= nr_cpu_ids) +return ; + +/* Find the last CPU which doesn't belong to that first hk_core. */ +if (!cpumask_test_cpu (cpu, first_hk_core)) { + target_cpu = cpu; + }else { +for_each_cpu_andnot (cpu, cpu_preferred_mask, first_hk_core) + target_cpu = cpu; + } + +/* Only the first housekeeping core remains */ +if (target_cpu >= nr_cpu_ids) +return ; + +for_each_cpu_and (cpu,topology_sibling_cpumask (target_cpu), + cpu_preferred_mask) +set_cpu_preferred (cpu, false); +} + +/* +* Called when the steal governor detects no/low physical CPU contention. +* It finds the first active core outside of preferred mask and mark +* those CPUs as preferred. +* +* Must ensure preferred is subset of active. +*/ +static void increase_preferred_cpus (void ) +{ +int first_cpu, cpu; + +guard (cpus_read_lock)(); + first_cpu =cpumask_first_andnot (cpu_active_mask, cpu_preferred_mask); + +/* All CPUs are preferred. Nothing to increase further */ +if (first_cpu >= nr_cpu_ids) +return ; + +for_each_cpu_and (cpu,topology_sibling_cpumask (first_cpu), + cpu_active_mask) +set_cpu_preferred (cpu, true); +} + +static bool preferred_cpus_valid (void ) +{ +if (cpumask_empty (cpu_preferred_mask)) { +pr_err ("empty preferred mask. stopping\n" ); +return false; + } + +if (!cpumask_subset (cpu_preferred_mask, cpu_active_mask)) { +pr_err ("preferred: %*pbl is not subset of active: %*pbl, stopping\n" , +cpumask_pr_args (cpu_preferred_mask), +cpumask_pr_args (cpu_active_mask)); +return false; + } + +return true; +} + +static void steal_governor_loop (struct work_struct *work) +{ +u64 curr_steal, delta_steal, delta_ns, steal_ratio; +ktime_t now; + + now =ktime_get (); + delta_ns =ktime_to_ns (ktime_sub (now, sg_ctx.time)); + +if (unlikely (delta_ns <NSEC_PER_MSEC )) { +pr_err_ratelimited ("work scheduled too soon delta_ns: %llu\n" , delta_ns); + goto requeue_work; + } + + curr_steal =get_system_steal_time (); + delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal :0 ; + sg_ctx.steal = curr_steal; + sg_ctx.time = now; + +/* +* steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus()) +* To avoid possible overflow, divide the denominator early. +* Note minimum interval is 100ms. +*/ + delta_ns =max_t (u64,div_u64 (delta_ns *get_system_cpus (),10000 ),1 ); + steal_ratio =div64_u64 (delta_steal, delta_ns); + +if (steal_ratio > sg_ctx.high_threshold) +decrease_preferred_cpus (); +else if (steal_ratio <= sg_ctx.low_threshold) +increase_preferred_cpus (); +/* +* else: steal ratio is within bounds. Still do design checks so that +* module restores to active if CPU hotplug breaks those assumptions. +*/ +if (!preferred_cpus_valid ()) { +restore_preferred_to_active (); +return ; + } + +requeue_work: +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay); +} +static int __init steal_governor_init (void ) {#ifdef CONFIG_XEN
@@ -127,6 +271,10 @@ static int __init steal_governor_init(void) } sg_ctx.delay =msecs_to_jiffies (sg_ctx.interval_ms); +INIT_DELAYED_WORK (&sg_ctx.work, steal_governor_loop); + sg_ctx.steal =get_system_steal_time (); + sg_ctx.time =ktime_get (); +schedule_delayed_work (&sg_ctx.work, sg_ctx.delay);pr_info ("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n" , sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
@@ -135,6 +283,7 @@ static int __init steal_governor_init(void) static void __exit steal_governor_exit(void) { +disable_delayed_work_sync (&sg_ctx.work);restore_preferred_to_active ();pr_info ("disabled\n" ); }
Shrikanth Hegde
maintainerPATCH v14 13/13 virt/steal_governor: Enable the driver
In reply to Shrikanth HegdeProvide a config option to enable the steal_governor driver. Since the feature targets paravirtualized environments and requires SMP, enforce those dependencies. The driver selects CONFIG_PREFERRED_CPU for the core scheduler mechanisms to work.
It is recommended to build the driver as a module (m) instead of built-in (y) due to the following reasons:
- Module parameters are read-only after initialization. Building as a module allows updating these parameters by simply reloading the module. Default module parameters cannot work in all configurations.
- The driver can be completely disabled by unloading the module.
- This feature works best when all VMs operate in a cooperative manner. Requiring an explicit module load ensures intentional deployment across all VMs by the system administrator.
--- drivers/virt/Kconfig | 17 +++++++++++++++++ drivers/virt/Makefile | 1 + 2 files changed, 18 insertions(+)
drivers/virt/Kconfig +17 -0
@@ -41,6 +41,23 @@ config FSL_HV_MANAGER 4) A kernel interface for receiving callbacks when a managed partition shuts down. +config STEAL_GOVERNOR + tristate "Dynamic vCPU management based on steal time" + depends on PARAVIRT && SMP + select PREFERRED_CPU + default m + help + This driver helps to reduce the steal time in paravirtualized + environments, thereby reducing vCPU preemption costs. + + By default preferred CPUs will be same as active CPUs. Depending + on the steal time when steal_governor driver is enabled, + preferred CPUs could become subset of active CPUs. + More details are at: Documentation/driver-api/steal-governor.rst + + It is recommended to build it as module and load the module + to enable it. + source "drivers/virt/vboxguest/Kconfig" source "drivers/virt/nitro_enclaves/Kconfig"
drivers/virt/Makefile +1 -0
@@ -5,6 +5,7 @@ obj-$(CONFIG_FSL_HV_MANAGER) += fsl_hypervisor.o obj-$(CONFIG_VMGENID) += vmgenid.o +obj-$(CONFIG_STEAL_GOVERNOR) += steal_governor.o obj-y += vboxguest/ obj-$(CONFIG_NITRO_ENCLAVES) += nitro_enclaves/
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH v14 13/13] virt/steal_governor: Enable the driver
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 12 lines
Provide a config option to enable the
steal_governordriver. Since the feature targets paravirtualized environments and requires SMP, enforce those dependencies. The driver selectsCONFIG_PREFERRED_CPUfor the core scheduler mechanisms to work.It is recommended to build the driver as a module (m) instead of built-in (y) due to the following reasons:
- Module parameters are read-only after initialization. Building as a module allows updating these parameters by simply reloading the module. Default module parameters cannot work in all configurations.
Quoted 8 lines
- The driver can be completely disabled by unloading the module.
- This feature works best when all VMs operate in a cooperative manner. Requiring an explicit module load ensures intentional deployment across all VMs by the system administrator.
Suggested-by Yury Norov <[email protected]>Signed-off-by Shrikanth Hegde <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
[tip: sched/core] virt/steal_governor: Enable the driver
In reply to Shrikanth HegdeThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 1fb28c664a19df8d45a6afa04d28d102b04ea680 Gitweb: https://git.kernel.org/tip/1fb28c664a19df8d45a6afa04d28d102b04ea680 Author: Shrikanth Hegde <[email protected]> AuthorDate: Mon, 28 Sep 2026 11:07:28 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 28 Sep 2026 08:44:44 +02:00
virt/steal_governor: Enable the driver
Provide a config option to enable the steal_governor driver. Since the feature targets paravirtualized environments and requires SMP, enforce those dependencies. The driver selects CONFIG_PREFERRED_CPU for the core scheduler mechanisms to work.
It is recommended to build the driver as a module (m) instead of built-in (y) due to the following reasons:
- Module parameters are read-only after initialization. Building as a module allows updating these parameters by simply reloading the module. Default module parameters cannot work in all configurations.
- The driver can be completely disabled by unloading the module.
- This feature works best when all VMs operate in a cooperative manner. Requiring an explicit module load ensures intentional deployment across all VMs by the system administrator.
--- drivers/virt/Kconfig | 17 +++++++++++++++++ drivers/virt/Makefile | 1 + 2 files changed, 18 insertions(+)
drivers/virt/Kconfig +17 -0
@@ -41,6 +41,23 @@ config FSL_HV_MANAGER 4) A kernel interface for receiving callbacks when a managed partition shuts down. +config STEAL_GOVERNOR + tristate "Dynamic vCPU management based on steal time" + depends on PARAVIRT && SMP + select PREFERRED_CPU + default m + help + This driver helps to reduce the steal time in paravirtualized + environments, thereby reducing vCPU preemption costs. + + By default preferred CPUs will be same as active CPUs. Depending + on the steal time when steal_governor driver is enabled, + preferred CPUs could become subset of active CPUs. + More details are at: Documentation/driver-api/steal-governor.rst + + It is recommended to build it as module and load the module + to enable it. + source "drivers/virt/vboxguest/Kconfig" source "drivers/virt/nitro_enclaves/Kconfig"
drivers/virt/Makefile +1 -0
@@ -5,6 +5,7 @@ obj-$(CONFIG_FSL_HV_MANAGER) += fsl_hypervisor.o obj-$(CONFIG_VMGENID) += vmgenid.o +obj-$(CONFIG_STEAL_GOVERNOR) += steal_governor.o obj-y += vboxguest/ obj-$(CONFIG_NITRO_ENCLAVES) += nitro_enclaves/