Skip to contentSkip to conversation

sched, steal_governor: Introduce preferred CPUs and steal-driven vCPU backoff

PATCH Applied Shrikanth Hegde 6 participants 51 messages Sep 28 – Nov 2

Subsystems (10)
lore
Share & tools
lore

Shrikanth Hegde

author maintainer
#0 lore raw

v14 cover sched, steal_governor: Introduce preferred CPUs and steal-driven vCPU backoff

tl;dr

This patch series represents the result of multiple iterations, redesigns and community feedback. What started as an arch-specific RFC has evolved into a scheduler mechanism paired with a virtualization driver.

This feature is intended for workloads where vCPU preemption has costs beyond the lost CPU time, such as lock-holder preemption, critical sections and TLB/cache misses. Database workloads, for example, contain a mixture of OLTP/OLAP activity and are prone to these costs. Reducing vCPU preemption in these cases can help improve overall system throughput. It may regress slightly for pure CPU-time workloads.

I believe the series has now converged and can be considered for merging. The implementation has been deliberately kept simple. It performs well for real-life workloads on PowerPC, and earlier KVM testing also showed improvements on s390 and x86.

The following work is planned post the merge:
- Design for arch specific interface which could help to bring out additional performance gains.
- A test framework for the feature

Feature has been kept as generic solution for all arch which support paravirtualization and steal time accounting. If a need arises, arch specific gating could be introduced.

Special thanks to Yury Norov for the rigorous reviews that greatly improved the series and to everyone who have provided their review comments so far. (including sashiko) Really appreciated! _/\_

Rest of cover-letter is very similar since v8. Version history has been updated.

Detailed Background and Problem Statement
=========================================

As hardware scales, the density of physical CPUs (pCPUs) per server is increasing across many architectures. On these massive systems, deploying a single bare-metal OS for general workloads becomes increasingly difficult to manage if not impossible. The natural shift is to deploy Virtual Machines(VMs). For example, on IBM PowerPC architecture customers frequently deploy Shared Processor LPARs (SPLPARs) to maximize hardware ROI.

Typical enterprise workloads are combination of bursty and long running; their average CPU utilization is low, but they require high core counts during peak transactions. To accommodate this, customers often use CPU overcommit strategies i.e. configuring VMs with a large number of virtual CPUs (vCPUs) while backing them with a smaller, shared pool of physical CPUs (pCPUs). This achieves a high server consolidation and excellent cost efficiency.

However, when multiple such VMs have high utilization simultaneously, the shared pCPU pool becomes contended. The hypervisor is forced to preempt one vCPU to run another to maintain fairness. It maybe schedule vCPU of same VM or different VM. If a vCPU is preempted while holding a lock or irq disabled section etc, overall forward progress collapses. There are some mitigation strategies such as yielding the vCPU to lock-holder, but they don't cover all the cases. In addition there are hidden costs such as cache, tlb misses, cost of vCPU preemption, host scheduling overheads etc.

Under heavy contention, an effective mitigation strategy for workloads sensitive to vCPU preemption is for guests/VMs to voluntarily fold their workloads onto a smaller subset of vCPUs. By demanding fewer pCPUs, the VMs reduce overall host contention, which decreases vCPU preemption and improves total throughput for the system.

Limitations of Existing Approaches
==================================

CPU Hotplug, Isolated cpusets, cpuset:
- This is a heavy and administrative operation that requires topology rebuild. Crucially, it breaks userspace CPU affinities.

Explicit task affinity:
- Very difficult to manage for the users, if not impossible.

We need a fast, co-operative backoff mechanism inside the kernel that can dynamically react to contention without violating user/task affinity contracts. Since reacting to the contention is agnostic to the user it cannot violate user affinity contracts.

When there is high contention, fold the workload and use limited vCPUs and when there is no contention, use all the vCPUs again. This natural expansion/contraction gives the better performance to the users based on the underlying contention.

Proposed Architecture
=====================

Current design is built on basis that contention is effectively quantified by steal time as seen in guest kernel.
Steal time is already a well established construct today in para-virtualization world. All major archs support this feature. It is indication of the contention of physical CPU. It scales according to the amount of contention. Today it is used by administrative users for changing the VM configurations. During high contention the steal time shows up in each guest based on its configuration. The proposed solution works well when all VMs honor the hint and work in co-operative manner. Note there is still no inter-guest communication to achieve this co-operation. Read the section on best practises on how to get the best out of this solution.

This series introduces a dynamic vCPU backoff mechanism. It is separated into a core scheduler mechanism and a loadable virtualization policy module.

Layer A: The Scheduler Mechanism (preferred CPUs)
=================================================

Series introduces a new CPU state called preferred. It indicates that vCPU can be safely used and using that vCPU won't increase contention for underlying physical CPUs. This state info is made available via cpu_preferred_mask, which is strictly maintained as a subset of cpu_active_mask.

The scheduler uses this mask as a hint to fold workloads onto preferred CPUs using a few mechanisms.

1. Wakeup: is_cpu_allowed() checks if CPU is preferred. If not calls select_fallback_rq, which selects a preferred CPU if tasks's affinity permits.

2. The Tick (Push): During sched_tick(), if the current CPU is non-preferred, the scheduler actively pushes the running task onto a preferred CPU using a stopper thread.

3. Load Balance: sched_balance_rq restricts its domain span to cpu_preferred_mask, preventing tasks from being pulled toward non-preferred CPUs.

Design Constraint: The scheduler strictly respects user affinities. If a task is pinned exclusively to non-preferred CPUs, it will remain there. The kernel will not break user/task affinity contracts.

Layer B: The Policy Engine (virt/steal_governor)
================================================

The core scheduler should not dictate virtualization policy. Therefore, the policy is isolated into a new driver: steal_governor. (Can be selected by CONFIG_STEAL_GOVERNOR)
This module latches onto that concept that contention is quantified by steal time. It periodically samples the steal time values across the system and depending on high/low steal values, takes appropriate action.

When it sees high steal times, i.e. steal time exceeds high_threshold (default 5%), driver reduces the preferred CPUs by 1 core. When it sees Low Steal Times, i.e. steal time drops below low_threshold (default 2%), driver increases the preferred CPUs by 1 core.

This creates a dynamic, self-maintained stepwise loop. The guest automatically shrinks its pCPU footprint when the host is saturated, and expands it when the noise clears while requiring zero cross-VM communication.

Policy Design Constraints:
- Ensure at least one core is kept as preferred.
- Ensure preferred is always subset of active.

Best Practises
==============
1. Ensure all the VM run kernel which has the patches.

2. Keep CONFIG_STEAL_GOVERNOR=m. Build it as module, but don't load it by default. When the administrative user enables it in one VM, he/she will likely enable it in all VMs. Also module parameters can only be changed at module load. Having it as module also allows one to disable it to remove additional overhead it brings.

3. Keep the interval_ms=500 to 5000. I.e. between 500ms to 5 second. Though parameters allows slightly higher range.

4. Fine tune low and high threshold depending on your platform for best results. Even where is no contention, very small steal values might show up. So it might be better to keep low threshold higher than 0.

Baseline and Revision History
==============================

tip/sched/core at commit:
'627ea30aca3b ("sched/wait: Clarify WF_SYNC wakeup semantics")'

For the talk on the problem and discussion on this issue, one can also refer to the OSPM26 talk[1].

[1]: https://youtu.be/adxUKFPlOp0
[2]: https://www.ibm.com/support/pages/ibm-power-virtualization-best-practices-guide
[3]: https://www.ibm.com/docs/en/linux-on-systems?topic=bad-daytrader

v13->v14:
- Make sched_non_preferred_cpu_push_stop look similar to __balance_push_cpu_stop. (Peter Zijlstra)
- Rename task_can_sched_on_preferred to task_can_migrate_to_preferred to make it read more natural. (Peter Zijlstra)
- Clarify that the new migration statistic is per-task and does not require a schedstat version change.
- Updated changelogs and Collected tags.

v12->v13:
- Rename push_task_work_done to npc_push_work_pending. (Yury Norov)
- Move it around steal_* fields in rq. (Yury Norov)
It doesn't increase the struct rq's number of cachelines for both 128 byte and 64 byte cachelines.
- Put PREFERRED_CPU guard for npc_push_work_pending (Yury Norov)
- Update comments around rq lock's acquisition post select_fallback_rq (Yury Norov)

v11->v12:
- Simplify the ARM64's 32 bit task case where p->cpus_ptr may contain architecturally impossible CPUs. (Dietmar Eggemann, Vincent Guittot)
- Introduce cpumask_intersects_and for that simplification (Yury Norov)
- Keep the feature without any arch specific kconfig gating for now. (Vincent Guittot)
If a need arises, it can be brought in

v10->v11:
- Return active CPUs instead of possible CPUs for steal ratio denominator. This helps to avoid dilution of thresholds.
(Yury Norov, Ionut Nechita)
- Add XEN dom0 blocking check. (Ionut Nechita)
- Update documentation/changelogs.
- Multi arch framework has been deferred. Will be designed after this series gets merged upstream. It would depend on enhancement requests.
- Collected tags for patch 1
- Fix possible crash for 32-bit tasks on arm64 (sashiko)
- Fix possible race for task affinity due to abort in __migrate_task (sashiko)
- Fix unused variable for patch 1 (sashiko)
- Disable interrupt when taking rq lock in a small window (sashiko)
- Queue stopper iff migration is enabled. (sashiko)
- Document workload characteristics and the possible regression for pure CPU-time workloads.

v9->v10:
- Introduce kcpustat_field_total helper. (Yury Norov)
- Always do the design checks. This helps to avoid placing design constraints in core hotplug code.
- Remove cpu_preferred check in idle balancing. This helps to naturally take care update of nohz.next_balance.
- find_new_ilb changes are deferred as it isn't applicable for most common use cases.
- Move scheduler documentation to sched-paravirt.rst. (Yury Norov)
- Add details of limitation of default values in documentation. (Yury Norov)
- Remove task_can_sched_on_preferred out of sched.h (Mete Durlu)
- Updated suggested-by tags for few patches. (I know i should have done it earlier, sorry about that)
- Minor polish of all changelogs.

v8->v9:
- Move to simpler layout. Everything in drivers/virt/steal_governor.c (Yury Norov)
- Move design checks into a helper function (Yury Norov)
- Comments update (Yury Norov)
- Requeue work without further checks when steal ratio is within the low/high threshold window. (Yury Norov)
- Renamed sg_core_ctx to sg_ctx.

v7->v8:
- Rename to STEAL_GOVERNOR from STEAL_MONITOR.
- Remove additional defaults.c and move it to core.c (Yury Norov)
- Remove SM_DIR gating for direction control. (Yury Norov)
- Enforce design constraint and restore the state if not met (Yury Norov)
- Drop nohz_full tick enable patch.
- Move Kconfig patch as the last patch for enablement. (Yury Norov)
- Use disable_delayed_work_sync to avoid race condition during module unload. (Yury Norov)
- Add same kconfig dependency and fail to compile the driver (Yury Norov)
- Make low < high comparison during module init instead as they are dependent parameters (Sashiko)
- Update sysfs file helper section (Yury Norov)
- Make preferred sysfs file available only with CONFIG_PREFERRED_CPU=y (Yury Norov)
- A few documentation and comments fixes. (Randy Dunlap)
- Fix possible race in sched_push_current_non_preferred_cpu (Yury Norov)
- Move is_migration_disabled check just before actual migration.
- Make 100ms as minimal interval_ms from 10ms.
- Make helper functions static and remove from header file as there are no other callers.
- Collapse helper functions and periodic work into one patch.

Short summary on previous versions:
v6->v7:
- Consolidate new driver code to 4-5 patches.
- defer the arch specific interface.
- Use possible CPUs instead of active for steal value calculations.
- Simplify is_cpu_allowed.
- Make module parameters fixed at module load
- Define CONFIG_STEAL_MONITOR and Make it select CONFIG_PREFERRED_CPU

v5->v6:
- Drop the optimization of caching the preferred state in select_fallback_rq
- Drop wakeup patch

v4->v5:
- Move the computation of steal time and decide on preferred CPU state to a driver. i.e new driver called STEAL_MONITOR

v3->v4:
- Make preferred subset of active instead of online.
- Dropped RT patch and Defer sched_ext. Support only FAIR class.

v2->v3:
- Introduce a new config CONFIG_PREFERRED_CPU

v1->v2:
- A new name - Preferred CPUs and cpu_preferred_mask
- Arch independent code. Everything happens in scheduler.
- Steal time computation is gated with sched feature STEAL_MONITOR

RFC v3-> RFC v4:
- Introduced computation of steal time in arch/powerpc.

RFC PATCH v1:
- push task mechanism.
- No steal time computation. Manual sysfs hint for preferred CPUs

v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/#t
v3: https://lore.kernel.org/all/[email protected]/#r
v4: https://lore.kernel.org/all/[email protected]/#t
v5: https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/all/[email protected]/#t
v7: https://lore.kernel.org/all/[email protected]/
v8: https://lore.kernel.org/all/[email protected]/
v9: https://lore.kernel.org/all/[email protected]/
v10: https://lore.kernel.org/all/[email protected]/
v11: https://lore.kernel.org/all/[email protected]/
v12: https://lore.kernel.org/all/[email protected]/
v13: https://lore.kernel.org/all/[email protected]/
Even earlier version:
https://lore.kernel.org/all/[email protected]/

========================================
Performance Numbers (powerpc, x86, s390)
========================================

PowerPC:
===================
VM1: 60VP/30EC and VM2: 30VP/20EC
Shared physical CPU pool size: 50 Cores. Each core is SMT8.
(VP - Virtual Core, EC - Entitles Core) - PowerVM terminologies of SPLPAR[2]

Default parameter values: 1000ms, 200 low threshold, 500 high threshold Both the VMs are running the same workload. Total throughput/time of VM1+VM2 is being mentioned in all cases.

Hackbench
baseline steal_governor steal_governor
disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

10 groups        5.20   |    5.40 (-3.85%)  |     4.65 (+10.58%)
20 groups       11.39   |   12.01 (-5.44%)  |     7.09 (+37.75%)
40 groups       20.32   |   19.80 (+2.56%)  |    11.31 (+44.34%)
10 groups(-p)    2.37   |    2.26 (+4.64%)  |     2.06 (+13.08%)
20 groups(-p)    3.34   |    3.28 (+1.80%)  |     3.20 (+4.19%)
40 groups(-p)    4.46   |    4.83 (-8.30%)  |     4.26 (+4.48%)

Remarks: Net improvement with steal_governor specially high load points.

schbench ( -L -n 0 -r 30 -s 0)
baseline steal_governor steal_governor
disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

-m 1 -t 128     2475162 |    2621246 (+5.90%)  |      2527299 (+2.11%)
-m 1 -t 256     1467350 |    1470032 (+0.18%)  |      1492372 (+1.71%)
-m 1 -t 512     1408813 |    1454687 (+3.26%)  |      1437605 (+2.04%)

Remarks: Effectively means no-improvements or regressions

kernbench baseline steal_governor steal_governor
(elapsed time) disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-j nr_cpus 231 | 235 (-1.7%) | 199 (+14%)
Remarks: Net improvement in elapsed time.

Daytrader - A real life work which is a proxy for trading based on db2[3]
baseline steal_governor steal_governor
disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Load@30%	1x	|	0.96x	|	 1.53x			
Load@60%	1x	|	0.94x	|	 1.41x

Remarks: Very good improvement seen at different load points.

phoronix/memcached:
Cloned from github.com/phoronix-test-suite/phoronix-test-suite.git GET 1:1000 means 1 writer and 1000 readers.
baseline steal_governor steal_governor
disabled enabled
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
GET 1:1000 1x 0.99x 1.2x
Remarks: 20% improvement with read heavy variant.
Other variants such as GET 1:1 showed minor regressions. While inspecting it, it was found that steal time reported by PowerVM was not as expected even when only 1 VM was running. This is being looked into by PowerVM hypervisor team. So added a section in limitation that, this feature expects steal time values being accurate for right policy decisions.

When there is no steal time (such as dedicated LPAR) throughput was same with steal_governor enabled/disabled which indicates minimal overhead of steal_governor.

=======================================================================

Data from x86,s390 KVM which Ilya Leoshkevich carried out during OSPM26 time. *This was based on v2*. The core idea remains the same, although the implementation has changed and some overhead has been removed since then.
Note: Other variations of the benchmark shows no observable difference.

x86:
====
cascade-lake: 32 threads = 16 cores
Benchmark #VMs #CPUs/VM ΔRPS (%std)
===============================================

hackbench         8          16  90.73% ± 9.97%
hackbench         4          24  52.67% ± 7.43%
hackbench         4          16  37.96% ± 11.19%
hackbench         4          32  37.82% ± 4.38%
hackbench        12           8  36.90% ± 4.74%
hackbench         8           8  35.30% ± 3.61%
pgbench          16           4  31.77% ± 2.44%
hackbench         2          24  25.85% ± 8.63%
hackbench        16           8  24.87% ± 3.46%
pgbench          16           8  21.83% ± 2.20%
pgbench          12           8  21.35% ± 2.15%
pgbench           8           8  18.46% ± 1.01%
hackbench         2          32  15.56% ± 4.53%
pgbench          12           4  14.28% ± 2.04%
hackbench        16           4  14.07% ± 2.90%
hackbench        12           4  9.60% ± 3.49%
[...]
pgbench           4           8  -1.16% ± 3.60%
hackbench         4           4  -1.80% ± 9.55%
sysbench         12           4  -2.19% ± 0.78%
pgbench           4          24  -2.43% ± 4.38%
pgbench           4          32  -3.21% ± 0.79%
sysbench         16           4  -3.22% ± 1.09%

S390:
=====
z16: 16 threads = 8 cores (SMT-2)
Benchmark #VMs #CPUs/VM ΔRPS (std%)
===============================================

pgbench           2           8  73.50% ± 35.91%
pgbench          16           4  61.30% ± 4.09%
hackbench        16           4  54.11% ± 4.38%
hackbench        12           4  36.34% ± 4.63%
pgbench          12           4  34.83% ± 2.57%
hackbench         8           4  29.75% ± 5.86%
hackbench         8           8  25.98% ± 5.09%
pgbench           2           4  23.31% ± 33.44%
pgbench           2          16  19.95% ± 17.12%
hackbench         4           8  19.43% ± 9.33%
pgbench           8           4  19.32% ± 4.50%
[...]
schbench          8           8  -0.79% ± 0.33%
sysbench          8           8  -0.81% ± 0.39%
hackbench         4          16  -1.11% ± 5.82%
sysbench          8           4  -1.62% ± 0.49%
sysbench         16           4  -2.70% ± 0.58%
schbench         16           4  -2.73% ± 0.91%
sysbench         12           4  -2.91% ± 0.61%
hackbench         2          24  -4.99% ± 3.31%

Summary:
- Many improvement across archs specially with real life workloads.
- Overhead of steal_governor looks minimal when there is no steal time.
- Overhead when STEAL_GOVERNOR=n is negligible.
- Pure CPU-time workloads may see a small regression.

Testing and Validation
======================

Apart from performance, To ensure the robustness of the preferred CPU masking and push mechanisms, the following scenarios were tested:
- CPU Hotplug: bringing CPUs up/down change the preferred mask accordingly under no-contention and contention.
- Housekeeping cores: Verified with different combinations of nohz_full=<beginning, middle, end set of CPUs> to ensure that policy engine restricts to first housekeeping core in extreme cases.
- User Affinity: Confirmed that tasks explicitly pinned to non-preferred CPUs via taskset remain on their assigned CPUs.
- Affine Move: Confirmed the affinity move using "taskset -cp" happens on all combinations of non-preferred, preferred under contention.
- Affinity and hotplug: It works as expected. I.e affinity gets reset if all the CPUs of p->cpus_ptr go offline even if they are non-preferred CPUs.
- Extreme load and running threads: for example 4800 stress-ng threads on 480 CPU system and it still packs to preferred CPUs.

Known Limitations
=================

- The steal governor relies on accurate steal-time accounting. Incorrect steal-time values from the underlying architecture may lead to suboptimal policy decisions or performance regressions.

- If the workload is a pure CPU time based such as stress-ng --cpu=<> which doesn't suffer from vCPU preemption, it may not benefit from the steal governor.

Future work
===========
To keep this initial implementation clean and minimal, a few optimizations have been deferred:

- Design a selftest like framework to validate steal governor, at least for KVM setup where there is infra available to spin off guests.

- Multi arch interface where arch specific hints could be provided into the driver's helper functions.

- Push all tasks on rq: Currently, the stopper thread only pushes the current running task off a non-preferred CPU. Future optimizations may look into migrating all queued tasks on that runqueue.

- Sched Classes: This feature currently only works for the FAIR class. Real-time (RT) and sched_ext classes are deferred for now, as there is no need for it.

- NUMA Splicing: The steal_governor currently removes last active core based on CPU number. It does not yet do complex NUMA-aware splicing, expecting that CPUs are spread out uniformly across nodes in most cases.

Shrikanth Hegde (13):
sched/cputime: Add kcpustat_field_total helper
cpumask: Introduce cpumask_intersects_and
sched/docs: Document cpu_preferred_mask and Preferred CPU concept
cpumask: Introduce cpu_preferred_mask
sysfs: Add preferred CPU file
sched/core: Try to use a preferred CPU in is_cpu_allowed sched/fair: Load balance only among preferred CPUs sched/core: Push current task from non preferred CPU sched/debug: Add migration stats due to non preferred CPUs
virt: Introduce steal governor driver virt/steal_governor: Add control knobs for handling steal values virt/steal_governor: Implement steal_governor policy loop virt/steal_governor: Enable the driver

 .../ABI/testing/sysfs-devices-system-cpu      |  14 +
 Documentation/driver-api/index.rst            |   1 +
 Documentation/driver-api/steal-governor.rst   | 151 +++++++++
 Documentation/scheduler/index.rst             |   1 +
 Documentation/scheduler/sched-paravirt.rst    |  67 ++++
 MAINTAINERS                                   |   9 +
 arch/s390/kernel/hiperdispatch.c              |  10 +-
 drivers/base/cpu.c                            |  12 +
 drivers/virt/Kconfig                          |  17 +
 drivers/virt/Makefile                         |   1 +
 drivers/virt/steal_governor.c                 | 296 ++++++++++++++++++
 fs/proc/uptime.c                              |   6 +-
 include/linux/bitmap.h                        |  14 +
 include/linux/cpumask.h                       |  42 +++
 include/linux/kernel_stat.h                   |  11 +
 include/linux/sched.h                         |   1 +
 kernel/Kconfig.preempt                        |   4 +
 kernel/cpu.c                                  |   6 +
 kernel/sched/core.c                           | 123 +++++++-
 kernel/sched/debug.c                          |   1 +
 kernel/sched/fair.c                           |   8 +-
 kernel/sched/sched.h                          |   9 +
 lib/bitmap.c                                  |  17 +
 23 files changed, 802 insertions(+), 19 deletions(-)
 create mode 100644 Documentation/driver-api/steal-governor.rst
 create mode 100644 Documentation/scheduler/sched-paravirt.rst
 create mode 100644 drivers/virt/steal_governor.c
Signature · 1 line
2.52.0
author maintainer
#1 lore raw

PATCH v14 1/13 sched/cputime: Add kcpustat_field_total helper

Provide a new helper function which sums up a given type of cpustat over a specified cpumask.

This allows the caller's code to be simpler and avoids duplication. For example, subsequent patch in the steal governor use this exact same pattern when calculating steal time.

Number of cpus can be derived from cpumask_weight() where necessary.

---
 arch/s390/kernel/hiperdispatch.c | 10 +++-------
 fs/proc/uptime.c                 |  6 +-----
 include/linux/kernel_stat.h      | 11 +++++++++++
 3 files changed, 15 insertions(+), 12 deletions(-)
arch/s390/kernel/hiperdispatch.c +3 -7
index 217206522266..c21496f0a141 100644
@@ -207,16 +207,12 @@ static unsigned long hd_calculate_steal_percentage(void)
 {
 	unsigned long time_delta, steal_delta, steal, percentage;
 	static ktime_t prev;
-	int cpus, cpu;
+	int cpus;
 	ktime_t now;
 
-	cpus = 0;
-	steal = 0;
 	percentage = 0;
-	for_each_cpu(cpu, &hd_vmvl_cpumask) {
-		steal += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL];
-		cpus++;
-	}
+	steal = kcpustat_field_total(CPUTIME_STEAL, &hd_vmvl_cpumask);
+	cpus = cpumask_weight(&hd_vmvl_cpumask);
 	/*
 	 * If there is no vertical medium and low CPUs steal time
 	 * is 0 as vertical high CPUs shouldn't experience steal time.
fs/proc/uptime.c +1 -5
index 433aa947cd57..53143c66cbe1 100644
@@ -15,12 +15,8 @@ static int uptime_proc_show(struct seq_file *m, void *v)
 	struct timespec64 idle;
 	u64 idle_nsec;
 	u32 rem;
-	int i;
-
-	idle_nsec = 0;
-	for_each_possible_cpu(i)
-		idle_nsec += kcpustat_field(CPUTIME_IDLE, i);
 
+	idle_nsec = kcpustat_field_total(CPUTIME_IDLE, cpu_possible_mask);
 	ktime_get_boottime_ts64(&uptime);
 	timens_add_boottime(&uptime);
 
include/linux/kernel_stat.h +11 -0
index 9ca6c2259dfe..c1e85550bf12 100644
@@ -196,6 +196,17 @@ static inline void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu)
 }
 #endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */
 
+static inline u64 kcpustat_field_total(enum cpu_usage_stat usage, const struct cpumask *cpus)
+{
+	u64 total = 0;
+	int cpu;
+
+	for_each_cpu(cpu, cpus)
+		total += kcpustat_field(usage, cpu);
+
+	return total;
+}
+
 extern void account_user_time(struct task_struct *, u64);
 extern void account_guest_time(struct task_struct *, u64);
 extern void account_system_time(struct task_struct *, int, u64);
Signature · 1 line
2.52.0
#2 lore raw

Re: [PATCH v14 01/13] sched/cputime: Add kcpustat_field_total helper

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] s390/hiperdispatch: TOCTOU race in hd_calculate_steal_percentage() causes artificial steal…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] s390/hiperdispatch: TOCTOU race in hd_calculate_steal_percentage() causes artificial steal percentage spikes --

commit 543d41ec158ad38d026cd1a0e58754cc9393d347
Author: Shrikanth Hegde <[email protected]>

sched/cputime: Add kcpustat_field_total helper

This patch introduces a new helper function to sum up a given cpustat type over a specified cpumask. It replaces the open-coded steal calculation in s390 hiperdispatch and idle time calculation in proc uptime to simplify the callers.

Quoted 4 lines
arch/s390/kernel/hiperdispatch.c +0 -0
index 2172065222661..c21496f0a141a 100644

[ ... ]

Quoted 7 lines
arch/s390/kernel/hiperdispatch.c +1 -1
@@ -207,16 +207,12 @@ static unsigned long hd_calculate_steal_percentage(void)
 {
 	unsigned long time_delta, steal_delta, steal, percentage;
 	static ktime_t prev;
-	int cpus, cpu;
+	int cpus;
 	ktime_t now;

arch/s390/kernel/hiperdispatch.c static unsigned long hd_calculate_steal_percentage(void) · line 215

Quoted 10 lines
arch/s390/kernel/hiperdispatch.c +2 -6
 
-	cpus = 0;
-	steal = 0;
 	percentage = 0;
-	for_each_cpu(cpu, &hd_vmvl_cpumask) {
-		steal += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL];
-		cpus++;
-	}
+	steal = kcpustat_field_total(CPUTIME_STEAL, &hd_vmvl_cpumask);
+	cpus = cpumask_weight(&hd_vmvl_cpumask);

[Severity: High]
Could separating the summation from the divisor count open a race window against concurrent mask updates?

In the previous single-loop calculation, the total steal time and CPU count were evaluated together on a stable view of the mask. With these changes, if a concurrent CPU hotplug event calls update_cpu_masks() while hd_calculate_steal_percentage() is running, hd_vmvl_cpumask could be cleared and repopulated asynchronously.

If the mask changes from multiple CPUs to 1 CPU between the kcpustat_field_total calculation and the cpumask_weight call, steal will reflect the delta of the larger number of CPUs, but cpus will evaluate to 1.

Because the percentage is computed as steal_delta / cpus, dividing the larger steal_delta by 1 would artificially inflate the steal percentage. Can this erroneous spike trip the hd_steal_threshold, causing the scheduler topology to wrongly disable Hiperdispatch and downgrade high-capacity cores?

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

[tip: sched/core] sched/cputime: Add kcpustat_field_total helper

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     a8d0854a76a876ea474e17322acb680716a09da2
Gitweb:        https://git.kernel.org/tip/a8d0854a76a876ea474e17322acb680716a09da2
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:16 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:40 +02:00

sched/cputime: Add kcpustat_field_total helper

Provide a new helper function which sums up a given type of cpustat over a specified cpumask.

This allows the caller's code to be simpler and avoids duplication. For example, subsequent patch in the steal governor use this exact same pattern when calculating steal time.

Number of cpus can be derived from cpumask_weight() where necessary.

---
 arch/s390/kernel/hiperdispatch.c | 10 +++-------
 fs/proc/uptime.c                 |  6 +-----
 include/linux/kernel_stat.h      | 11 +++++++++++
 3 files changed, 15 insertions(+), 12 deletions(-)
arch/s390/kernel/hiperdispatch.c +3 -7
index 2172065..c21496f 100644
@@ -207,16 +207,12 @@ static unsigned long hd_calculate_steal_percentage(void)
 {
 	unsigned long time_delta, steal_delta, steal, percentage;
 	static ktime_t prev;
-	int cpus, cpu;
+	int cpus;
 	ktime_t now;
 
-	cpus = 0;
-	steal = 0;
 	percentage = 0;
-	for_each_cpu(cpu, &hd_vmvl_cpumask) {
-		steal += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL];
-		cpus++;
-	}
+	steal = kcpustat_field_total(CPUTIME_STEAL, &hd_vmvl_cpumask);
+	cpus = cpumask_weight(&hd_vmvl_cpumask);
 	/*
 	 * If there is no vertical medium and low CPUs steal time
 	 * is 0 as vertical high CPUs shouldn't experience steal time.
fs/proc/uptime.c +1 -5
index 433aa94..53143c6 100644
@@ -15,12 +15,8 @@ static int uptime_proc_show(struct seq_file *m, void *v)
 	struct timespec64 idle;
 	u64 idle_nsec;
 	u32 rem;
-	int i;
-
-	idle_nsec = 0;
-	for_each_possible_cpu(i)
-		idle_nsec += kcpustat_field(CPUTIME_IDLE, i);
 
+	idle_nsec = kcpustat_field_total(CPUTIME_IDLE, cpu_possible_mask);
 	ktime_get_boottime_ts64(&uptime);
 	timens_add_boottime(&uptime);
 
include/linux/kernel_stat.h +11 -0
index 9ca6c22..c1e8555 100644
@@ -196,6 +196,17 @@ static inline void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu)
 }
 #endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */
 
+static inline u64 kcpustat_field_total(enum cpu_usage_stat usage, const struct cpumask *cpus)
+{
+	u64 total = 0;
+	int cpu;
+
+	for_each_cpu(cpu, cpus)
+		total += kcpustat_field(usage, cpu);
+
+	return total;
+}
+
 extern void account_user_time(struct task_struct *, u64);
 extern void account_guest_time(struct task_struct *, u64);
 extern void account_system_time(struct task_struct *, int, u64);
author maintainer
#4 lore raw

PATCH v14 2/13 cpumask: Introduce cpumask_intersects_and

In reply to Shrikanth Hegde

Introduce bitmap_intersects_and() to determine whether the intersection of three bitmaps is non-empty. Unlike cpumask_first_and_and(), this returns immediately when an intersecting word is found and does not calculate the first matching bit.

Add cpumask_intersects_and() as the corresponding cpumask wrapper.

A subsequent patch uses the helper to determine whether a task has a CPU that is present in its affinity mask, the preferred CPU mask, and task possible CPU mask.

---
 include/linux/bitmap.h  | 14 ++++++++++++++
 include/linux/cpumask.h | 18 ++++++++++++++++++
 lib/bitmap.c            | 17 +++++++++++++++++
 3 files changed, 49 insertions(+)
include/linux/bitmap.h +14 -0
index 7df1573a409c..adafbcf2016b 100644
@@ -52,6 +52,7 @@ struct device;
  *  bitmap_complement(dst, src, nbits)          *dst = ~(*src)
  *  bitmap_equal(src1, src2, nbits)             Are *src1 and *src2 equal?
  *  bitmap_intersects(src1, src2, nbits)        Do *src1 and *src2 overlap?
+ *  bitmap_intersects_and(src1, src2, src3, nbits) Do *src1, *src2 and *src3 overlap?
  *  bitmap_subset(src1, src2, nbits)            Is *src1 a subset of *src2?
  *  bitmap_empty(src, nbits)                    Are all bits zero in *src?
  *  bitmap_full(src, nbits)                     Are all bits set in *src?
@@ -181,6 +182,9 @@ void __bitmap_replace(unsigned long *dst,
 		      const unsigned long *mask, unsigned int nbits);
 bool __bitmap_intersects(const unsigned long *bitmap1,
 			 const unsigned long *bitmap2, unsigned int nbits);
+bool __bitmap_intersects_and(const unsigned long *bitmap1,
+			     const unsigned long *bitmap2,
+			     const unsigned long *bitmap3, unsigned int nbits);
 bool __bitmap_subset(const unsigned long *bitmap1,
 		     const unsigned long *bitmap2, unsigned int nbits);
 unsigned int __bitmap_weight(const unsigned long *bitmap, unsigned int nbits);
@@ -445,6 +449,16 @@ bool bitmap_intersects(const unsigned long *src1, const unsigned long *src2, uns
 		return __bitmap_intersects(src1, src2, nbits);
 }
 
+static __always_inline
+bool bitmap_intersects_and(const unsigned long *src1, const unsigned long *src2,
+			   const unsigned long *src3, unsigned int nbits)
+{
+	if (small_const_nbits(nbits))
+		return ((*src1 & *src2 & *src3) & BITMAP_LAST_WORD_MASK(nbits)) != 0;
+	else
+		return __bitmap_intersects_and(src1, src2, src3, nbits);
+}
+
 static __always_inline
 bool bitmap_subset(const unsigned long *src1, const unsigned long *src2, unsigned int nbits)
 {
include/linux/cpumask.h +18 -0
index 4c8bb6953107..7c8f16797f94 100644
@@ -824,6 +824,24 @@ bool cpumask_intersects(const struct cpumask *src1p, const struct cpumask *src2p
 						      small_cpumask_bits);
 }
 
+/**
+ * cpumask_intersects_and - (*src1p & *src2p & *src3p) != 0
+ * @src1p: the first input
+ * @src2p: the second input
+ * @src3p: the third input
+ *
+ * Return: true if AND of the three cpumasks is non-empty,
+ *	   otherwise false
+ */
+static __always_inline
+bool cpumask_intersects_and(const struct cpumask *src1p,
+			    const struct cpumask *src2p,
+			    const struct cpumask *src3p)
+{
+	return bitmap_intersects_and(cpumask_bits(src1p), cpumask_bits(src2p),
+				 cpumask_bits(src3p), small_cpumask_bits);
+}
+
 /**
  * cpumask_subset - (*src1p & ~*src2p) == 0
  * @src1p: the first input
lib/bitmap.c +17 -0
index ed685127a107..d1cb8a507c60 100644
@@ -308,6 +308,23 @@ bool __bitmap_intersects(const unsigned long *bitmap1,
 }
 EXPORT_SYMBOL(__bitmap_intersects);
 
+bool __bitmap_intersects_and(const unsigned long *bitmap1,
+			     const unsigned long *bitmap2,
+			     const unsigned long *bitmap3, unsigned int bits)
+{
+	unsigned int k, lim = bits / BITS_PER_LONG;
+
+	for (k = 0; k < lim; ++k)
+		if (bitmap1[k] & bitmap2[k] & bitmap3[k])
+			return true;
+
+	if (bits % BITS_PER_LONG)
+		if ((bitmap1[k] & bitmap2[k] & bitmap3[k]) & BITMAP_LAST_WORD_MASK(bits))
+			return true;
+	return false;
+}
+EXPORT_SYMBOL(__bitmap_intersects_and);
+
 bool __bitmap_subset(const unsigned long *bitmap1,
 		     const unsigned long *bitmap2, unsigned int bits)
 {
Signature · 1 line
2.52.0
#5 lore raw

Re: [PATCH v14 02/13] cpumask: Introduce cpumask_intersects_and

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines

Introduce bitmap_intersects_and() to determine whether the intersection of three bitmaps is non-empty. Unlike cpumask_first_and_and(), this returns immediately when an intersecting word is found and does not calculate the first matching bit.

Add cpumask_intersects_and() as the corresponding cpumask wrapper.

Quoted 7 lines

A subsequent patch uses the helper to determine whether a task has a CPU that is present in its affinity mask, the preferred CPU mask, and task possible CPU mask.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#6 lore raw

[tip: sched/core] cpumask: Introduce cpumask_intersects_and

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     cfb463b7172dca2f29b2221b71b54890f268e3fb
Gitweb:        https://git.kernel.org/tip/cfb463b7172dca2f29b2221b71b54890f268e3fb
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:17 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:40 +02:00

cpumask: Introduce cpumask_intersects_and

Introduce bitmap_intersects_and() to determine whether the intersection of three bitmaps is non-empty. Unlike cpumask_first_and_and(), this returns immediately when an intersecting word is found and does not calculate the first matching bit.

Add cpumask_intersects_and() as the corresponding cpumask wrapper.

A subsequent patch uses the helper to determine whether a task has a CPU that is present in its affinity mask, the preferred CPU mask, and task possible CPU mask.

---
 include/linux/bitmap.h  | 14 ++++++++++++++
 include/linux/cpumask.h | 18 ++++++++++++++++++
 lib/bitmap.c            | 17 +++++++++++++++++
 3 files changed, 49 insertions(+)
include/linux/bitmap.h +14 -0
index 7df1573..adafbcf 100644
@@ -52,6 +52,7 @@ struct device;
  *  bitmap_complement(dst, src, nbits)          *dst = ~(*src)
  *  bitmap_equal(src1, src2, nbits)             Are *src1 and *src2 equal?
  *  bitmap_intersects(src1, src2, nbits)        Do *src1 and *src2 overlap?
+ *  bitmap_intersects_and(src1, src2, src3, nbits) Do *src1, *src2 and *src3 overlap?
  *  bitmap_subset(src1, src2, nbits)            Is *src1 a subset of *src2?
  *  bitmap_empty(src, nbits)                    Are all bits zero in *src?
  *  bitmap_full(src, nbits)                     Are all bits set in *src?
@@ -181,6 +182,9 @@ void __bitmap_replace(unsigned long *dst,
 		      const unsigned long *mask, unsigned int nbits);
 bool __bitmap_intersects(const unsigned long *bitmap1,
 			 const unsigned long *bitmap2, unsigned int nbits);
+bool __bitmap_intersects_and(const unsigned long *bitmap1,
+			     const unsigned long *bitmap2,
+			     const unsigned long *bitmap3, unsigned int nbits);
 bool __bitmap_subset(const unsigned long *bitmap1,
 		     const unsigned long *bitmap2, unsigned int nbits);
 unsigned int __bitmap_weight(const unsigned long *bitmap, unsigned int nbits);
@@ -446,6 +450,16 @@ bool bitmap_intersects(const unsigned long *src1, const unsigned long *src2, uns
 }
 
 static __always_inline
+bool bitmap_intersects_and(const unsigned long *src1, const unsigned long *src2,
+			   const unsigned long *src3, unsigned int nbits)
+{
+	if (small_const_nbits(nbits))
+		return ((*src1 & *src2 & *src3) & BITMAP_LAST_WORD_MASK(nbits)) != 0;
+	else
+		return __bitmap_intersects_and(src1, src2, src3, nbits);
+}
+
+static __always_inline
 bool bitmap_subset(const unsigned long *src1, const unsigned long *src2, unsigned int nbits)
 {
 	if (small_const_nbits(nbits))
include/linux/cpumask.h +18 -0
index 4c8bb69..7c8f167 100644
@@ -825,6 +825,24 @@ bool cpumask_intersects(const struct cpumask *src1p, const struct cpumask *src2p
 }
 
 /**
+ * cpumask_intersects_and - (*src1p & *src2p & *src3p) != 0
+ * @src1p: the first input
+ * @src2p: the second input
+ * @src3p: the third input
+ *
+ * Return: true if AND of the three cpumasks is non-empty,
+ *	   otherwise false
+ */
+static __always_inline
+bool cpumask_intersects_and(const struct cpumask *src1p,
+			    const struct cpumask *src2p,
+			    const struct cpumask *src3p)
+{
+	return bitmap_intersects_and(cpumask_bits(src1p), cpumask_bits(src2p),
+				 cpumask_bits(src3p), small_cpumask_bits);
+}
+
+/**
  * cpumask_subset - (*src1p & ~*src2p) == 0
  * @src1p: the first input
  * @src2p: the second input
lib/bitmap.c +17 -0
index ed68512..d1cb8a5 100644
@@ -308,6 +308,23 @@ bool __bitmap_intersects(const unsigned long *bitmap1,
 }
 EXPORT_SYMBOL(__bitmap_intersects);
 
+bool __bitmap_intersects_and(const unsigned long *bitmap1,
+			     const unsigned long *bitmap2,
+			     const unsigned long *bitmap3, unsigned int bits)
+{
+	unsigned int k, lim = bits / BITS_PER_LONG;
+
+	for (k = 0; k < lim; ++k)
+		if (bitmap1[k] & bitmap2[k] & bitmap3[k])
+			return true;
+
+	if (bits % BITS_PER_LONG)
+		if ((bitmap1[k] & bitmap2[k] & bitmap3[k]) & BITMAP_LAST_WORD_MASK(bits))
+			return true;
+	return false;
+}
+EXPORT_SYMBOL(__bitmap_intersects_and);
+
 bool __bitmap_subset(const unsigned long *bitmap1,
 		     const unsigned long *bitmap2, unsigned int bits)
 {
author maintainer
#7 lore raw

PATCH v14 3/13 sched/docs: Document cpu_preferred_mask and Preferred CPU concept

In reply to Shrikanth Hegde

Add documentation for new CPU state called preferred CPU state and corresponding cpumask called cpu_preferred_mask.
This could help users in understanding what it is and how to use it.

Document the role of scheduler and driver for this feature to work. Details regarding the driver documentation will be added in later patches under Documentation/driver-api/steal-governor.rst.

Newly added file could be used for other paravirt usecase documentation.

---
 Documentation/scheduler/index.rst          |  1 +
 Documentation/scheduler/sched-paravirt.rst | 67 ++++++++++++++++++++++
 2 files changed, 68 insertions(+)
 create mode 100644 Documentation/scheduler/sched-paravirt.rst
Documentation/scheduler/index.rst +1 -0
index 17ce8d76befc..a43647b9706d 100644
@@ -23,5 +23,6 @@ Scheduler
     sched-stats
     sched-ext
     sched-debug
+    sched-paravirt
 
     text_files
Documentation/scheduler/sched-paravirt.rst new file +67 -0

67 lines not on this page · raw message

Signature · 1 line
2.52.0
#8 lore raw

Re: [PATCH v14 03/13] sched/docs: Document cpu_preferred_mask and Preferred CPU concept

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines

Add documentation for new CPU state called preferred CPU state and corresponding cpumask called cpu_preferred_mask.
This could help users in understanding what it is and how to use it.

Document the role of scheduler and driver for this feature to work. Details regarding the driver documentation will be added in later patches under Documentation/driver-api/steal-governor.rst.

Newly added file could be used for other paravirt usecase documentation.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#9 lore raw

[tip: sched/core] sched/docs: Document cpu_preferred_mask and Preferred CPU concept

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     06a49ef784acf91cee78d50c5307d400f86ccafb
Gitweb:        https://git.kernel.org/tip/06a49ef784acf91cee78d50c5307d400f86ccafb
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:18 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:40 +02:00

sched/docs: Document cpu_preferred_mask and Preferred CPU concept

Add documentation for new CPU state called preferred CPU state and corresponding cpumask called cpu_preferred_mask.
This could help users in understanding what it is and how to use it.

Document the role of scheduler and driver for this feature to work. Details regarding the driver documentation will be added in later patches under Documentation/driver-api/steal-governor.rst.

Newly added file could be used for other paravirt usecase documentation.

---
 Documentation/scheduler/index.rst          |  1 +-
 Documentation/scheduler/sched-paravirt.rst | 67 +++++++++++++++++++++-
 2 files changed, 68 insertions(+)
 create mode 100644 Documentation/scheduler/sched-paravirt.rst
Documentation/scheduler/index.rst +1 -0
index 17ce8d7..a43647b 100644
@@ -23,5 +23,6 @@ Scheduler
     sched-stats
     sched-ext
     sched-debug
+    sched-paravirt
 
     text_files
Documentation/scheduler/sched-paravirt.rst new file +67 -0

67 lines not on this page · raw message

author maintainer
#10 lore raw

PATCH v14 4/13 cpumask: Introduce cpu_preferred_mask

In reply to Shrikanth Hegde

Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.

CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.

A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.

Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible

With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.

---
 include/linux/cpumask.h | 24 ++++++++++++++++++++++++
 kernel/Kconfig.preempt  |  4 ++++
 kernel/cpu.c            |  6 ++++++
 kernel/sched/core.c     |  5 +++++
 4 files changed, 39 insertions(+)
include/linux/cpumask.h +24 -0
index 7c8f16797f94..bf89bb3f30f6 100644
@@ -121,12 +121,20 @@ extern struct cpumask __cpu_enabled_mask;
 extern struct cpumask __cpu_present_mask;
 extern struct cpumask __cpu_active_mask;
 extern struct cpumask __cpu_dying_mask;
+
+#ifdef CONFIG_PREFERRED_CPU
+extern struct cpumask __cpu_preferred_mask;
+#else
+#define __cpu_preferred_mask __cpu_active_mask
+#endif
+
 #define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)
 #define cpu_online_mask   ((const struct cpumask *)&__cpu_online_mask)
 #define cpu_enabled_mask   ((const struct cpumask *)&__cpu_enabled_mask)
 #define cpu_present_mask  ((const struct cpumask *)&__cpu_present_mask)
 #define cpu_active_mask   ((const struct cpumask *)&__cpu_active_mask)
 #define cpu_dying_mask    ((const struct cpumask *)&__cpu_dying_mask)
+#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)
 
 extern atomic_t __num_online_cpus;
 extern unsigned int __num_possible_cpus;
@@ -1181,6 +1189,12 @@ void init_cpu_possible(const struct cpumask *src);
 #define set_cpu_active(cpu, active)	assign_cpu((cpu), &__cpu_active_mask, (active))
 #define set_cpu_dying(cpu, dying)	assign_cpu((cpu), &__cpu_dying_mask, (dying))
 
+#ifdef CONFIG_PREFERRED_CPU
+#define set_cpu_preferred(cpu, preferred) assign_cpu((cpu), &__cpu_preferred_mask, (preferred))
+#else
+#define set_cpu_preferred(cpu, preferred) do { } while (0)
+#endif
+
 void set_cpu_online(unsigned int cpu, bool online);
 void set_cpu_possible(unsigned int cpu, bool possible);
 
@@ -1275,6 +1289,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)
 	return cpumask_test_cpu(cpu, cpu_dying_mask);
 }
 
+static __always_inline bool cpu_preferred(unsigned int cpu)
+{
+	return cpumask_test_cpu(cpu, cpu_preferred_mask);
+}
+
 #else
 
 #define num_online_cpus()	1U
@@ -1313,6 +1332,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)
 	return false;
 }
 
+static __always_inline bool cpu_preferred(unsigned int cpu)
+{
+	return cpu == 0;
+}
+
 #endif /* NR_CPUS > 1 */
 
 #define cpu_is_offline(cpu)	unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +4 -0
index 985aea617cfe..edc067a0c422 100644
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT
 	  For more information:
 	    Documentation/scheduler/sched-ext.rst
 	    https://github.com/sched-ext/scx
+
+config PREFERRED_CPU
+	bool
+	depends on SMP && PARAVIRT
kernel/cpu.c +6 -0
index b3c8553d7bd6..376d297a6292 100644
@@ -3103,6 +3103,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);
 atomic_t __num_online_cpus __read_mostly;
 EXPORT_SYMBOL(__num_online_cpus);
 
+#ifdef CONFIG_PREFERRED_CPU
+struct cpumask __cpu_preferred_mask __read_mostly;
+EXPORT_SYMBOL_GPL(__cpu_preferred_mask);
+#endif
+
 void init_cpu_present(const struct cpumask *src)
 {
 	cpumask_copy(&__cpu_present_mask, src);
@@ -3160,6 +3165,7 @@ void __init boot_cpu_init(void)
 	/* Mark the boot cpu "present", "online" etc for SMP and UP case */
 	set_cpu_online(cpu, true);
 	set_cpu_active(cpu, true);
+	set_cpu_preferred(cpu, true);
 	set_cpu_present(cpu, true);
 	set_cpu_possible(cpu, true);
 
kernel/sched/core.c +5 -0
index ee9b443f760d..0183c75d7d51 100644
@@ -8580,6 +8580,9 @@ int sched_cpu_activate(unsigned int cpu)
 	 */
 	sched_set_rq_online(rq, cpu);
 
+	/* preferred is subset of active and follows its state */
+	set_cpu_preferred(cpu, true);
+
 	return 0;
 }
 
@@ -8593,6 +8596,8 @@ int sched_cpu_deactivate(unsigned int cpu)
 	if (ret)
 		return ret;
 
+	set_cpu_preferred(cpu, false);
+
 	/*
 	 * Remove CPU from nohz.idle_cpus_mask to prevent participating in
 	 * load balancing when not active
Signature · 1 line
2.52.0
#11 lore raw

Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 12 lines

Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.

CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.

A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.

Quoted 7 lines

Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible

With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#12 lore raw

[tip: sched/core] cpumask: Introduce cpu_preferred_mask

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     518b32bd5bb373aa334e4b73d359689071d50b5c
Gitweb:        https://git.kernel.org/tip/518b32bd5bb373aa334e4b73d359689071d50b5c
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:19 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:41 +02:00

cpumask: Introduce cpu_preferred_mask

Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.

CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.

A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.

Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible

With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.

---
 include/linux/cpumask.h | 24 ++++++++++++++++++++++++
 kernel/Kconfig.preempt  |  4 ++++
 kernel/cpu.c            |  6 ++++++
 kernel/sched/core.c     |  5 +++++
 4 files changed, 39 insertions(+)
include/linux/cpumask.h +24 -0
index 7c8f167..bf89bb3 100644
@@ -121,12 +121,20 @@ extern struct cpumask __cpu_enabled_mask;
 extern struct cpumask __cpu_present_mask;
 extern struct cpumask __cpu_active_mask;
 extern struct cpumask __cpu_dying_mask;
+
+#ifdef CONFIG_PREFERRED_CPU
+extern struct cpumask __cpu_preferred_mask;
+#else
+#define __cpu_preferred_mask __cpu_active_mask
+#endif
+
 #define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)
 #define cpu_online_mask   ((const struct cpumask *)&__cpu_online_mask)
 #define cpu_enabled_mask   ((const struct cpumask *)&__cpu_enabled_mask)
 #define cpu_present_mask  ((const struct cpumask *)&__cpu_present_mask)
 #define cpu_active_mask   ((const struct cpumask *)&__cpu_active_mask)
 #define cpu_dying_mask    ((const struct cpumask *)&__cpu_dying_mask)
+#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)
 
 extern atomic_t __num_online_cpus;
 extern unsigned int __num_possible_cpus;
@@ -1181,6 +1189,12 @@ void init_cpu_possible(const struct cpumask *src);
 #define set_cpu_active(cpu, active)	assign_cpu((cpu), &__cpu_active_mask, (active))
 #define set_cpu_dying(cpu, dying)	assign_cpu((cpu), &__cpu_dying_mask, (dying))
 
+#ifdef CONFIG_PREFERRED_CPU
+#define set_cpu_preferred(cpu, preferred) assign_cpu((cpu), &__cpu_preferred_mask, (preferred))
+#else
+#define set_cpu_preferred(cpu, preferred) do { } while (0)
+#endif
+
 void set_cpu_online(unsigned int cpu, bool online);
 void set_cpu_possible(unsigned int cpu, bool possible);
 
@@ -1275,6 +1289,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)
 	return cpumask_test_cpu(cpu, cpu_dying_mask);
 }
 
+static __always_inline bool cpu_preferred(unsigned int cpu)
+{
+	return cpumask_test_cpu(cpu, cpu_preferred_mask);
+}
+
 #else
 
 #define num_online_cpus()	1U
@@ -1313,6 +1332,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)
 	return false;
 }
 
+static __always_inline bool cpu_preferred(unsigned int cpu)
+{
+	return cpu == 0;
+}
+
 #endif /* NR_CPUS > 1 */
 
 #define cpu_is_offline(cpu)	unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +4 -0
index 985aea6..edc067a 100644
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT
 	  For more information:
 	    Documentation/scheduler/sched-ext.rst
 	    https://github.com/sched-ext/scx
+
+config PREFERRED_CPU
+	bool
+	depends on SMP && PARAVIRT
kernel/cpu.c +6 -0
index b3c8553..376d297 100644
@@ -3103,6 +3103,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);
 atomic_t __num_online_cpus __read_mostly;
 EXPORT_SYMBOL(__num_online_cpus);
 
+#ifdef CONFIG_PREFERRED_CPU
+struct cpumask __cpu_preferred_mask __read_mostly;
+EXPORT_SYMBOL_GPL(__cpu_preferred_mask);
+#endif
+
 void init_cpu_present(const struct cpumask *src)
 {
 	cpumask_copy(&__cpu_present_mask, src);
@@ -3160,6 +3165,7 @@ void __init boot_cpu_init(void)
 	/* Mark the boot cpu "present", "online" etc for SMP and UP case */
 	set_cpu_online(cpu, true);
 	set_cpu_active(cpu, true);
+	set_cpu_preferred(cpu, true);
 	set_cpu_present(cpu, true);
 	set_cpu_possible(cpu, true);
 
kernel/sched/core.c +5 -0
index acc8931..d4edee7 100644
@@ -8632,6 +8632,9 @@ int sched_cpu_activate(unsigned int cpu)
 	 */
 	sched_set_rq_online(rq, cpu);
 
+	/* preferred is subset of active and follows its state */
+	set_cpu_preferred(cpu, true);
+
 	return 0;
 }
 
@@ -8645,6 +8648,8 @@ int sched_cpu_deactivate(unsigned int cpu)
 	if (ret)
 		return ret;
 
+	set_cpu_preferred(cpu, false);
+
 	/*
 	 * Remove CPU from nohz.idle_cpus_mask to prevent participating in
 	 * load balancing when not active
#13 lore raw

Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask

In reply to Shrikanth Hegde
Quoted 25 lines · Shrikanth Hegde

Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.

CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.

A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.

Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible

With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.

---
  include/linux/cpumask.h | 24 ++++++++++++++++++++++++
  kernel/Kconfig.preempt  |  4 ++++
  kernel/cpu.c            |  6 ++++++
  kernel/sched/core.c     |  5 +++++
  4 files changed, 39 insertions(+)

[..snip..]

Quoted 7 lines
  
  #define cpu_is_offline(cpu)	unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +0 -0
index 985aea617cfe..edc067a0c422 100644

kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200

Quoted 7 lines
kernel/Kconfig.preempt +4 -0
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT
  	  For more information:
  	    Documentation/scheduler/sched-ext.rst
  	    https://github.com/sched-ext/scx
+
+config PREFERRED_CPU
+	bool
+	depends on SMP && PARAVIRT

FWICT, PREFERRED_CPU mechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummy paravirt_steal_clock() just for that.

Shrikanth, would it be possible to drop that dependency?

Thanks!
-Mete

#14 lore raw

Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask

Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde

Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.

CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.

A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.

Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible

With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.

---
  include/linux/cpumask.h | 24 ++++++++++++++++++++++++
  kernel/Kconfig.preempt  |  4 ++++
  kernel/cpu.c            |  6 ++++++
  kernel/sched/core.c     |  5 +++++
  4 files changed, 39 insertions(+)

[..snip..]

kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200

Quoted 13 lines
  #define cpu_is_offline(cpu)	unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +4 -0
index 985aea617cfe..edc067a0c422 100644
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT
  	  For more information:
  	    Documentation/scheduler/sched-ext.rst
  	    https://github.com/sched-ext/scx
+
+config PREFERRED_CPU
+	bool
+	depends on SMP && PARAVIRT

FWICT, PREFERRED_CPU mechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummy paravirt_steal_clock() just for that.

Quoted 1 line

Shrikanth, would it be possible to drop that dependency?

Do you have a non-paravirt case in mind?

I'm actually about to submit an RFC that drops the PARAVIRT dependency and provides write access to the preferred mask. But I want to make sure that we need it, and we've got no reasonable mechanism to put unneeded CPUs on idle already.

Simply relaxing the PARAVIRT dependency without a solid use case would bloat configuration space for nothing. If you've got such a case in mind, the rest is trivial.

Thanks,
Yury

#15 lore raw

Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask

Quoted 47 lines · Yury Norov
Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde

Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.

CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.

A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.

Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible

With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.

---
   include/linux/cpumask.h | 24 ++++++++++++++++++++++++
   kernel/Kconfig.preempt  |  4 ++++
   kernel/cpu.c            |  6 ++++++
   kernel/sched/core.c     |  5 +++++
   4 files changed, 39 insertions(+)

[..snip..]

kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200

Quoted 13 lines
   #define cpu_is_offline(cpu)	unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +4 -0
index 985aea617cfe..edc067a0c422 100644
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT
   	  For more information:
   	    Documentation/scheduler/sched-ext.rst
   	    https://github.com/sched-ext/scx
+
+config PREFERRED_CPU
+	bool
+	depends on SMP && PARAVIRT

FWICT, PREFERRED_CPU mechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummy paravirt_steal_clock() just for that.

Quoted 3 lines
Quoted 1 line

Shrikanth, would it be possible to drop that dependency?

Do you have a non-paravirt case in mind?

s390 LPARs are exactly that. s390 does not select PARAVIRT by default but enabling PREFERRED_CPU is strongly considered.

I removed the PARAVIRT dependency and gave the whole thing a quick spin. Builds and functions w/o any issues on s390!

Quoted 4 lines

I'm actually about to submit an RFC that drops the PARAVIRT dependency and provides write access to the preferred mask. But I want to make sure that we need it, and we've got no reasonable mechanism to put unneeded CPUs on idle already.

You can mention s390 LPARs as valid case on your RFC if need be. Thank you!

#16 lore raw

Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask

Quoted 55 lines · Mete Durlu
Quoted 47 lines · Yury Norov
Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde

Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.

CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.

A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.

Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible

With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.

---
   include/linux/cpumask.h | 24 ++++++++++++++++++++++++
   kernel/Kconfig.preempt  |  4 ++++
   kernel/cpu.c            |  6 ++++++
   kernel/sched/core.c     |  5 +++++
   4 files changed, 39 insertions(+)

[..snip..]

kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200

Quoted 13 lines
   #define cpu_is_offline(cpu)	unlikely(!cpu_online(cpu))
kernel/Kconfig.preempt +4 -0
index 985aea617cfe..edc067a0c422 100644
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT
   	  For more information:
   	    Documentation/scheduler/sched-ext.rst
   	    https://github.com/sched-ext/scx
+
+config PREFERRED_CPU
+	bool
+	depends on SMP && PARAVIRT

FWICT, PREFERRED_CPU mechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummy paravirt_steal_clock() just for that.

Quoted 3 lines
Quoted 1 line

Shrikanth, would it be possible to drop that dependency?

Do you have a non-paravirt case in mind?

s390 LPARs are exactly that. s390 does not select PARAVIRT by default but enabling PREFERRED_CPU is strongly considered.

Quoted 8 lines

I removed the PARAVIRT dependency and gave the whole thing a quick spin. Builds and functions w/o any issues on s390!

Quoted 4 lines

I'm actually about to submit an RFC that drops the PARAVIRT dependency and provides write access to the preferred mask. But I want to make sure that we need it, and we've got no reasonable mechanism to put unneeded CPUs on idle already.

You can mention s390 LPARs as valid case on your RFC if need be.

Sure, will do. I'm about submitting this before the end of week; will add you in CC.

Thanks,
Yury

author maintainer
#17 lore raw

Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask

In reply to Mete Durlu

Hi Mete, Yury,

Quoted 48 lines · Mete Durlu
Quoted 47 lines · Yury Norov
Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde

Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.

CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.

A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.

Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible

With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.

---
   include/linux/cpumask.h | 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt  |  4 ++++ kernel/cpu.c            |  6 ++++++ kernel/sched/core.c     |  5 +++++

   4 files changed, 39 insertions(+)

[..snip..]

kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200

Quoted 13 lines

   #define cpu_is_offline(cpu)    unlikely(!cpu_online(cpu))

kernel/Kconfig.preempt +0 -0
index 985aea617cfe..edc067a0c422 100644
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT

         For more information:
           Documentation/scheduler/sched-ext.rst
           https://github.com/sched-ext/scx

kernel/Kconfig.preempt +4 -0
+
+config PREFERRED_CPU
+    bool
+    depends on SMP && PARAVIRT

FWICT, PREFERRED_CPU mechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummy paravirt_steal_clock() just for that.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Shrikanth, would it be possible to drop that dependency?

Do you have a non-paravirt case in mind?

s390 LPARs are exactly that. s390 does not select PARAVIRT by default but enabling PREFERRED_CPU is strongly considered.

s390 may not have PARAVIRT=y, but this use case is still a virtualizated system no? i.e you are observing steal time. By definition of steal time, it is a para- virtualized system. Though you may be deriving the steal time not using the paravirt_steal_clock.

Or my understanding here is wrong?

Quoted 8 lines

I removed the PARAVIRT dependency and gave the whole thing a quick spin. Builds and functions w/o any issues on s390!

Quoted 4 lines

I'm actually about to submit an RFC that drops the PARAVIRT dependency and provides write access to the preferred mask. But I want to make sure that we need it, and we've got no reasonable mechanism to put unneeded CPUs on idle already.

We might run into few corner cases quickly with write allowed. But yes, this is a simpler mechanism not to run anything on few CPUs in a dynamic way.

Quoted 3 lines

You can mention s390 LPARs as valid case on your RFC if need be. Thank you!

#18 lore raw

Re: [PATCH v14 04/13] cpumask: Introduce cpu_preferred_mask

Quoted 51 lines · Shrikanth Hegde

Hi Mete, Yury,

Quoted 48 lines · Mete Durlu
Quoted 47 lines · Yury Norov
Quoted 46 lines · Mete Durlu
Quoted 25 lines · Shrikanth Hegde

Provide the preferred CPU infrastructure. Define get/set macros which could be used to get/set CPU state as preferred.

CONFIG_PREFERRED_CPU will be selected by the driver which handles steal time values. It is going to set/clear preferred CPU state. This driver will be called steal_governor and it is introduced in subsequent patches. It periodically computes the steal ratio and decides on preferred CPU state.

A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal ratio by the steal_governor.

Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible

With CONFIG_PREFERRED_CPU=n, ensure set_cpu_preferred is a nop and get method returns the active state in that case.

---
   include/linux/cpumask.h | 24 ++++++++++++++++++++++++ kernel/Kconfig.preempt  |  4 ++++ kernel/cpu.c            |  6 ++++++ kernel/sched/core.c     |  5 +++++

   4 files changed, 39 insertions(+)

[..snip..]

kernel/Kconfig.preempt config SCHED_CLASS_EXT · line 200

Quoted 13 lines

   #define cpu_is_offline(cpu)    unlikely(!cpu_online(cpu))

kernel/Kconfig.preempt +0 -0
index 985aea617cfe..edc067a0c422 100644
@@ -194,3 +194,7 @@ config SCHED_CLASS_EXT

         For more information:
           Documentation/scheduler/sched-ext.rst
           https://github.com/sched-ext/scx

kernel/Kconfig.preempt +4 -0
+
+config PREFERRED_CPU
+    bool
+    depends on SMP && PARAVIRT

FWICT, PREFERRED_CPU mechanism can work perfectly fine without PARAVIRT selected. It would be annoying to enable it and implement a dummy paravirt_steal_clock() just for that.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Shrikanth, would it be possible to drop that dependency?

Do you have a non-paravirt case in mind?

s390 LPARs are exactly that. s390 does not select PARAVIRT by default but enabling PREFERRED_CPU is strongly considered.

s390 may not have PARAVIRT=y, but this use case is still a virtualizated system no?

Yes, it is virtualized.

Quoted 6 lines

i.e you are observing steal time. By definition of steal time, it is a para- virtualized system. Though you may be deriving the steal time not using the paravirt_steal_clock.

Or my understanding here is wrong?

So you propose s390 to enable CONFIG_PARAVIRT *just because* s390 fits the "paravirtualized system" definition, even if s390 would not utilize any functionality comes with the config option? (even worse it comes with deadcode...)

That sounds a bit wrong to me, I would really like to avoid enabling a config option just because it's name suggests that it is related to our platform.

Yury already sent the patch series he mentioned. I suppose we should keep discussing this there.

Thank you!
-Mete

author maintainer
#19 lore raw

PATCH v14 5/13 sysfs: Add preferred CPU file

In reply to Shrikanth Hegde

Add a "preferred" file in /sys/devices/system/cpu/ when kernel is built with CONFIG_PREFERRED_CPU=y.

This would help
- Users to quickly check which CPUs are marked as preferred.
- Userspace daemons such as irqbalance to use this mask to route irqs into preferred CPUs.

For example:
cat /sys/devices/system/cpu/online
0-719
cat /sys/devices/system/cpu/preferred
0-599 <<< Implies 0-599 are preferred for workloads and 600-719
should be avoided at this moment.

cat /sys/devices/system/cpu/preferred
0-719 <<< All CPUs are usable. There is no preference.

---
 Documentation/ABI/testing/sysfs-devices-system-cpu | 14 ++++++++++++++
 drivers/base/cpu.c                                 | 12 ++++++++++++
 2 files changed, 26 insertions(+)
Documentation/ABI/testing/sysfs-devices-system-cpu +14 -0
index 82d10d556cc8..8f250b8693b8 100644
@@ -806,3 +806,17 @@ Date:		Nov 2022
 Contact:	Linux kernel mailing list <[email protected]>
 Description:
 		(RO) the list of CPUs that can be brought online.
+
+What:		/sys/devices/system/cpu/preferred
+Date:		Sep 2026
+Contact:	Linux kernel mailing list <[email protected]>
+Description:
+		(RO) the list of preferred CPUs applicable in
+		paravirtualized environments.
+
+		The steal governor driver dynamically adjusts this mask
+		based on observed steal time. Scheduling tasks on
+		CPUs outside of this list may lead to performance
+		degradations due to underlying physical CPU contention.
+
+		See Documentation/scheduler/sched-paravirt.rst for more details.
drivers/base/cpu.c +12 -0
index 69e52fed4241..747915ff974f 100644
@@ -391,6 +391,15 @@ static int cpu_uevent(const struct device *dev, struct kobj_uevent_env *env)
 }
 #endif
 
+#ifdef CONFIG_PREFERRED_CPU
+static ssize_t preferred_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, "%*pbl\n", cpumask_pr_args(cpu_preferred_mask));
+}
+static DEVICE_ATTR_RO(preferred);
+#endif
+
 const struct bus_type cpu_subsys = {
 	.name = "cpu",
 	.dev_name = "cpu",
@@ -531,6 +540,9 @@ static struct attribute *cpu_root_attrs[] = {
 #endif
 #ifdef CONFIG_GENERIC_CPU_AUTOPROBE
 	&dev_attr_modalias.attr,
+#endif
+#ifdef CONFIG_PREFERRED_CPU
+	&dev_attr_preferred.attr,
 #endif
 	NULL
 };
Signature · 1 line
2.52.0
#20 lore raw

Re: [PATCH v14 05/13] sysfs: Add preferred CPU file

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines

Add a "preferred" file in /sys/devices/system/cpu/ when kernel is built with CONFIG_PREFERRED_CPU=y.

This would help
- Users to quickly check which CPUs are marked as preferred.
- Userspace daemons such as irqbalance to use this mask to route irqs into preferred CPUs.

For example:
cat /sys/devices/system/cpu/online
0-719
cat /sys/devices/system/cpu/preferred
0-599 <<< Implies 0-599 are preferred for workloads and 600-719
should be avoided at this moment.

Quoted 4 lines

cat /sys/devices/system/cpu/preferred
0-719 <<< All CPUs are usable. There is no preference.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#21 lore raw

[tip: sched/core] sysfs: Add preferred CPU file

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     62082451655747dae25496d126e58f985df29f50
Gitweb:        https://git.kernel.org/tip/62082451655747dae25496d126e58f985df29f50
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:20 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:41 +02:00

sysfs: Add preferred CPU file

Add a "preferred" file in /sys/devices/system/cpu/ when kernel is built with CONFIG_PREFERRED_CPU=y.

This would help
- Users to quickly check which CPUs are marked as preferred.
- Userspace daemons such as irqbalance to use this mask to route irqs into preferred CPUs.

For example:
cat /sys/devices/system/cpu/online
0-719
cat /sys/devices/system/cpu/preferred
0-599 <<< Implies 0-599 are preferred for workloads and 600-719
should be avoided at this moment.

cat /sys/devices/system/cpu/preferred
0-719 <<< All CPUs are usable. There is no preference.

---
 Documentation/ABI/testing/sysfs-devices-system-cpu | 14 +++++++++++++-
 drivers/base/cpu.c                                 | 12 +++++++++++-
 2 files changed, 26 insertions(+)
Documentation/ABI/testing/sysfs-devices-system-cpu +14 -0
index 82d10d5..8f250b8 100644
@@ -806,3 +806,17 @@ Date:		Nov 2022
 Contact:	Linux kernel mailing list <[email protected]>
 Description:
 		(RO) the list of CPUs that can be brought online.
+
+What:		/sys/devices/system/cpu/preferred
+Date:		Sep 2026
+Contact:	Linux kernel mailing list <[email protected]>
+Description:
+		(RO) the list of preferred CPUs applicable in
+		paravirtualized environments.
+
+		The steal governor driver dynamically adjusts this mask
+		based on observed steal time. Scheduling tasks on
+		CPUs outside of this list may lead to performance
+		degradations due to underlying physical CPU contention.
+
+		See Documentation/scheduler/sched-paravirt.rst for more details.
drivers/base/cpu.c +12 -0
index 69e52fe..747915f 100644
@@ -391,6 +391,15 @@ static int cpu_uevent(const struct device *dev, struct kobj_uevent_env *env)
 }
 #endif
 
+#ifdef CONFIG_PREFERRED_CPU
+static ssize_t preferred_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, "%*pbl\n", cpumask_pr_args(cpu_preferred_mask));
+}
+static DEVICE_ATTR_RO(preferred);
+#endif
+
 const struct bus_type cpu_subsys = {
 	.name = "cpu",
 	.dev_name = "cpu",
@@ -532,6 +541,9 @@ static struct attribute *cpu_root_attrs[] = {
 #ifdef CONFIG_GENERIC_CPU_AUTOPROBE
 	&dev_attr_modalias.attr,
 #endif
+#ifdef CONFIG_PREFERRED_CPU
+	&dev_attr_preferred.attr,
+#endif
 	NULL
 };
 
author maintainer
#22 lore raw

PATCH v14 6/13 sched/core: Try to use a preferred CPU in is_cpu_allowed

In reply to Shrikanth Hegde

When possible, try to choose a preferred CPU.

This is essential to maintain user affinities when preferred CPUs change. A task pinned on a non-preferred CPU should continue to run there, since this is a non-user triggered event.

If a CPU is non-preferred and the task can run on other CPUs which are currently preferred, then choose a preferred CPU instead. This is decided by checking if cpus_ptr, cpu_preferred_mask and task possible CPUs intersect or not. If yes, then the task has other preferred CPUs. This takes care of tasks with architecture-specific CPU masks (e.g., 32-bit tasks on arm64).

The push task mechanism uses a stopper thread which calls select_fallback_rq() and uses this mechanism to pick a preferred CPU.

This takes care of the wakeup path for FAIR tasks too. is_cpu_allowed() is called to ensure wakeups happen on preferred CPUs. With that, additional checks in available_idle_cpu() are not necessary.

Ignore the preferred CPU state if a task's affinity is changing and its new mask no longer includes the CPU it is currently running on. This ensures migration_cpu_stop() does not abort, preventing the task from being stranded outside its allowed affinity.

For the majority of cases, this would still keep select_fallback_rq() as O(N). cpumask_intersects_and(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomes O(N**2) only for tasks pinned solely to non-preferred CPUs. That is a rare case.

Overhead is minimal when the CPU is preferred.

---
 kernel/sched/core.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
kernel/sched/core.c +28 -2
index 0183c75d7d51..04400f934cc7 100644
@@ -2504,6 +2504,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)
 	return rq->nr_pinned;
 }
 
+static inline bool task_can_migrate_to_preferred(struct task_struct *p, int cpu)
+{
+	/* No need to migrate from a preferred CPU */
+	if (cpu_preferred(cpu))
+		return false;
+
+	/* Only FAIR tasks honor preferred CPU state */
+	if (unlikely(p->sched_class != &fair_sched_class))
+		return false;
+
+	/* Ignore preferred state if task affinity is changing */
+	if (unlikely(!cpumask_test_cpu(task_cpu(p), p->cpus_ptr)))
+		return false;
+
+	return cpumask_intersects_and(p->cpus_ptr, cpu_preferred_mask,
+				      task_cpu_possible_mask(p));
+}
+
 /*
  * Per-CPU kthreads are allowed to run on !active && online CPUs, see
  * __set_cpus_allowed_ptr() and select_fallback_rq().
@@ -2519,8 +2537,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
 		return cpu_online(cpu);
 
 	/* Non kernel threads are not allowed during either online or offline. */
-	if (!(p->flags & PF_KTHREAD))
+	if (!(p->flags & PF_KTHREAD)) {
+		/* Try to use preferred CPU if task's affinity allows */
+		if (task_can_migrate_to_preferred(p, cpu))
+			return false;
 		return cpu_active(cpu);
+	}
 
 	/* KTHREAD_IS_PER_CPU is always allowed. */
 	if (kthread_is_per_cpu(p))
@@ -2530,7 +2552,11 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
 	if (cpu_dying(cpu))
 		return false;
 
-	/* But are allowed during online. */
+	/* Try to keep unbound kthreads on a preferred CPU if possible. */
+	if (task_can_migrate_to_preferred(p, cpu))
+		return false;
+
+	/* Otherwise, they are allowed to run on online CPU. */
 	return cpu_online(cpu);
 }
 
Signature · 1 line
2.52.0
#23 lore raw

Re: [PATCH v14 06/13] sched/core: Try to use a preferred CPU in is_cpu_allowed

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] sched/core: O(N^2) wakeup latency and task clumping in fallback path - [High] sched/fair: Active…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] sched/core: O(N^2) wakeup latency and task clumping in fallback path
- [High] sched/fair: Active load balancing to non-preferred CPUs silently fails
- [High] sched/core: TOCTOU race with CPU hotplug breaks task affinity
--

commit dfa0f159ec643e02807bc453bbfcc49c9f1e7915
Author: Shrikanth Hegde <[email protected]>

sched/core: Try to use a preferred CPU in is_cpu_allowed

This commit modifies the scheduler to preferentially select preferred CPUs when checking if a task is allowed to run on a given CPU. It introduces a helper function to check if a task can migrate to a preferred CPU based on its affinity mask.

Quoted 5 lines

For the majority of cases, this would still keep select_fallback_rq() as O(N). cpumask_intersects_and(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomes O(N**2) only for tasks pinned solely to non-preferred CPUs. That is a rare case.

Is this description accurate?

Because the fallback loop iterates over the affinity mask sequentially, if the allowed preferred CPUs happen to have higher indices than the non-preferred CPUs, the loop will perform the intersection operation for every non-preferred CPU it checks. This means any task with preferred CPUs at higher indices could experience the quadratic latency overhead, rather than only tasks pinned solely to non-preferred CPUs.

Quoted 4 lines
kernel/sched/core.c +0 -0
index 0183c75d7d518..04400f934cc72 100644

[ ... ]

Quoted 11 lines
kernel/sched/core.c +7 -0
@@ -2504,6 +2504,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)
 	return rq->nr_pinned;
 }
 
+static inline bool task_can_migrate_to_preferred(struct task_struct *p, int cpu)
+{
+	/* No need to migrate from a preferred CPU */
+	if (cpu_preferred(cpu))
+		return false;
+
+	/* Only FAIR tasks honor preferred CPU state */

kernel/sched/core.c static inline bool rq_has_pinned_tasks(struct rq *rq) · line 2523

Quoted 10 lines
kernel/sched/core.c +10 -0
+	if (unlikely(p->sched_class != &fair_sched_class))
+		return false;
+
+	/* Ignore preferred state if task affinity is changing */
+	if (unlikely(!cpumask_test_cpu(task_cpu(p), p->cpus_ptr)))
+		return false;
+
+	return cpumask_intersects_and(p->cpus_ptr, cpu_preferred_mask,
+				      task_cpu_possible_mask(p));
+}

[Severity: High]
Does this intersection operation lead to quadratic latency during wakeups when combined with the fallback loop?

When select_task_rq() in kernel/sched/core.c picks a non-preferred CPU because it is idle, is_cpu_allowed() will reject it, forcing the task into select_fallback_rq(). The fallback loop iterates sequentially over all CPUs:

kernel/sched/core.c:select_fallback_rq() {
    ...
    for_each_cpu(dest_cpu, p->cpus_ptr) {
        if (!is_cpu_allowed(p, dest_cpu))
            continue;
        ...
        goto out;
    }
    ...
}

If the allowed preferred CPUs are at the end of the cpumask, wouldn't this perform an intersection for every non-preferred CPU it checks, leading to severe scheduling latency? Additionally, wouldn't this deterministic loop always dump the task on the first preferred CPU it finds, destroying load distribution by clumping tasks on the lowest-ID preferred CPU?

This performance degradation from wakeups on non-preferred CPUs does not appear to be resolved in the final state of the patch series.

[ ... ]

kernel/sched/core.c static inline bool is_cpu_allowed(struct task_struct *p, int cpu) · line 2543

Quoted 9 lines
kernel/sched/core.c +4 -1
@@ -2519,8 +2537,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
 		return cpu_online(cpu);
 
 	/* Non kernel threads are not allowed during either online or offline. */
-	if (!(p->flags & PF_KTHREAD))
+	if (!(p->flags & PF_KTHREAD)) {
+		/* Try to use preferred CPU if task's affinity allows */
+		if (task_can_migrate_to_preferred(p, cpu))
+			return false;

[Severity: High]
Could returning false for non-preferred CPUs here cause active load balancing to silently fail?

During routine background load balancing, if the system attempts to pull load from a preferred CPU to an idle non-preferred CPU, the stopper thread will run __migrate_task().

kernel/sched/core.c:__migrate_task() {
    ...
    if (!is_cpu_allowed(p, dest_cpu))
        return rq;
    ...
}

Since the destination is non-preferred, won't is_cpu_allowed() reject it, causing the stopper thread to silently abort the migration and break load distribution?

While follow-up patches in the series restrict the busiest CPU to preferred CPUs, they do not appear to prevent the destination from being non-preferred, leaving this active load balancing path broken.

[Severity: High]
Can this rejection introduce a race condition with CPU hotplug that breaks a task's user-configured affinity?

In select_fallback_rq() in kernel/sched/core.c, the loop might skip an active, non-preferred CPU because task_can_migrate_to_preferred() indicates another preferred CPU is available in the affinity mask. If a concurrent CPU hotplug operation removes that preferred CPU from the active mask before the loop reaches it, the loop will skip it too:

kernel/sched/core.c:select_fallback_rq() {
    ...
    for_each_cpu(dest_cpu, p->cpus_ptr) {
        if (!is_cpu_allowed(p, dest_cpu))
            continue;
    ...
}

Could the loop then terminate without finding any allowed CPU, permanently forcing the task outside of its allowed affinity?

Quoted 2 lines
kernel/sched/core.c +1 -0
 		return cpu_active(cpu);
+	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author maintainer
#24 lore raw

Re: [PATCH v14 06/13] sched/core: Try to use a preferred CPU in is_cpu_allowed

I believe most of the comments have been addressed in the previous rounds.

Quoted 77 lines · On 9/28/26 11:30 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] sched/core: O(N^2) wakeup latency and task clumping in fallback path
- [High] sched/fair: Active load balancing to non-preferred CPUs silently fails
- [High] sched/core: TOCTOU race with CPU hotplug breaks task affinity
--

commit dfa0f159ec643e02807bc453bbfcc49c9f1e7915
Author: Shrikanth Hegde <[email protected]>

sched/core: Try to use a preferred CPU in is_cpu_allowed

This commit modifies the scheduler to preferentially select preferred CPUs when checking if a task is allowed to run on a given CPU. It introduces a helper function to check if a task can migrate to a preferred CPU based on its affinity mask.

Quoted 5 lines

For the majority of cases, this would still keep select_fallback_rq() as O(N). cpumask_intersects_and(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomes O(N**2) only for tasks pinned solely to non-preferred CPUs. That is a rare case.

Is this description accurate?

Because the fallback loop iterates over the affinity mask sequentially, if the allowed preferred CPUs happen to have higher indices than the non-preferred CPUs, the loop will perform the intersection operation for every non-preferred CPU it checks. This means any task with preferred CPUs at higher indices could experience the quadratic latency overhead, rather than only tasks pinned solely to non-preferred CPUs.

Quoted 4 lines
kernel/sched/core.c +0 -0
index 0183c75d7d518..04400f934cc72 100644

[ ... ]

Quoted 11 lines
kernel/sched/core.c +7 -0
@@ -2504,6 +2504,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)
  	return rq->nr_pinned;
  }
 
+static inline bool task_can_migrate_to_preferred(struct task_struct *p, int cpu)
+{
+	/* No need to migrate from a preferred CPU */
+	if (cpu_preferred(cpu))
+		return false;
+
+	/* Only FAIR tasks honor preferred CPU state */

kernel/sched/core.c static inline bool rq_has_pinned_tasks(struct rq *rq) · line 2523

Quoted 10 lines
kernel/sched/core.c +10 -0
+	if (unlikely(p->sched_class != &fair_sched_class))
+		return false;
+
+	/* Ignore preferred state if task affinity is changing */
+	if (unlikely(!cpumask_test_cpu(task_cpu(p), p->cpus_ptr)))
+		return false;
+
+	return cpumask_intersects_and(p->cpus_ptr, cpu_preferred_mask,
+				      task_cpu_possible_mask(p));
+}

[Severity: High]
Does this intersection operation lead to quadratic latency during wakeups when combined with the fallback loop?

When select_task_rq() in kernel/sched/core.c picks a non-preferred CPU because it is idle, is_cpu_allowed() will reject it, forcing the task into select_fallback_rq(). The fallback loop iterates sequentially over all CPUs:

kernel/sched/core.c:select_fallback_rq() {
     ...
     for_each_cpu(dest_cpu, p->cpus_ptr) {
         if (!is_cpu_allowed(p, dest_cpu))
             continue;
         ...
         goto out;
     }
     ...
}
Quoted 8 lines

If the allowed preferred CPUs are at the end of the cpumask, wouldn't this perform an intersection for every non-preferred CPU it checks, leading to severe scheduling latency? Additionally, wouldn't this deterministic loop always dump the task on the first preferred CPU it finds, destroying load distribution by clumping tasks on the lowest-ID preferred CPU?

This performance degradation from wakeups on non-preferred CPUs does not appear to be resolved in the final state of the patch series.

It is mentioned in the changelog and this is true only when wakeup happens on a non-preferred CPUs. Even if it runs there is meant to pushed out. So take a bit more cycles and do the right thing.

Quoted 5 lines

[ ... ]

Quoted 3 lines
kernel/sched/core.c +0 -0
@@ -2519,8 +2537,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
  		return cpu_online(cpu);
 
Quoted 10 lines

kernel/sched/core.c static inline bool is_cpu_allowed(struct task_struct *p, int cpu) · line 2543

Quoted 6 lines
kernel/sched/core.c +4 -1
  	/* Non kernel threads are not allowed during either online or offline. */
-	if (!(p->flags & PF_KTHREAD))
+	if (!(p->flags & PF_KTHREAD)) {
+		/* Try to use preferred CPU if task's affinity allows */
+		if (task_can_migrate_to_preferred(p, cpu))
+			return false;

[Severity: High]
Could returning false for non-preferred CPUs here cause active load balancing to silently fail?

Spreading load to non-preferred CPUs is by design and expected.

Quoted 24 lines

During routine background load balancing, if the system attempts to pull load from a preferred CPU to an idle non-preferred CPU, the stopper thread will run __migrate_task().

kernel/sched/core.c:__migrate_task() {
     ...
     if (!is_cpu_allowed(p, dest_cpu))
         return rq;
     ...
}

Since the destination is non-preferred, won't is_cpu_allowed() reject it, causing the stopper thread to silently abort the migration and break load distribution?

While follow-up patches in the series restrict the busiest CPU to preferred CPUs, they do not appear to prevent the destination from being non-preferred, leaving this active load balancing path broken.

[Severity: High]
Can this rejection introduce a race condition with CPU hotplug that breaks a task's user-configured affinity?

Quoted 5 lines

In select_fallback_rq() in kernel/sched/core.c, the loop might skip an active, non-preferred CPU because task_can_migrate_to_preferred() indicates another preferred CPU is available in the affinity mask. If a concurrent CPU hotplug operation removes that preferred CPU from the active mask before the loop reaches it, the loop will skip it too:

There is two iterations and steal governor's minimal interval is 100ms. So it is likely safe.

Quoted 15 lines
kernel/sched/core.c:select_fallback_rq() {
     ...
     for_each_cpu(dest_cpu, p->cpus_ptr) {
         if (!is_cpu_allowed(p, dest_cpu))
             continue;
     ...
}

Could the loop then terminate without finding any allowed CPU, permanently forcing the task outside of its allowed affinity?

Quoted 2 lines
kernel/sched/core.c +1 -0
  		return cpu_active(cpu);
+	}
#25 lore raw

[tip: sched/core] sched/core: Try to use a preferred CPU in is_cpu_allowed

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     d8a3da0de8433d494db1ff5ec541559c764da4ab
Gitweb:        https://git.kernel.org/tip/d8a3da0de8433d494db1ff5ec541559c764da4ab
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:21 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:41 +02:00

sched/core: Try to use a preferred CPU in is_cpu_allowed

When possible, try to choose a preferred CPU.

This is essential to maintain user affinities when preferred CPUs change. A task pinned on a non-preferred CPU should continue to run there, since this is a non-user triggered event.

If a CPU is non-preferred and the task can run on other CPUs which are currently preferred, then choose a preferred CPU instead. This is decided by checking if cpus_ptr, cpu_preferred_mask and task possible CPUs intersect or not. If yes, then the task has other preferred CPUs. This takes care of tasks with architecture-specific CPU masks (e.g., 32-bit tasks on arm64).

The push task mechanism uses a stopper thread which calls select_fallback_rq() and uses this mechanism to pick a preferred CPU.

This takes care of the wakeup path for FAIR tasks too. is_cpu_allowed() is called to ensure wakeups happen on preferred CPUs. With that, additional checks in available_idle_cpu() are not necessary.

Ignore the preferred CPU state if a task's affinity is changing and its new mask no longer includes the CPU it is currently running on. This ensures migration_cpu_stop() does not abort, preventing the task from being stranded outside its allowed affinity.

For the majority of cases, this would still keep select_fallback_rq() as O(N). cpumask_intersects_and(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomes O(N**2) only for tasks pinned solely to non-preferred CPUs. That is a rare case.

Overhead is minimal when the CPU is preferred.

---
 kernel/sched/core.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
kernel/sched/core.c +28 -2
index d4edee7..d095fb1 100644
@@ -2511,6 +2511,24 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)
 	return rq->nr_pinned;
 }
 
+static inline bool task_can_migrate_to_preferred(struct task_struct *p, int cpu)
+{
+	/* No need to migrate from a preferred CPU */
+	if (cpu_preferred(cpu))
+		return false;
+
+	/* Only FAIR tasks honor preferred CPU state */
+	if (unlikely(p->sched_class != &fair_sched_class))
+		return false;
+
+	/* Ignore preferred state if task affinity is changing */
+	if (unlikely(!cpumask_test_cpu(task_cpu(p), p->cpus_ptr)))
+		return false;
+
+	return cpumask_intersects_and(p->cpus_ptr, cpu_preferred_mask,
+				      task_cpu_possible_mask(p));
+}
+
 /*
  * Per-CPU kthreads are allowed to run on !active && online CPUs, see
  * __set_cpus_allowed_ptr() and select_fallback_rq().
@@ -2526,8 +2544,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
 		return cpu_online(cpu);
 
 	/* Non kernel threads are not allowed during either online or offline. */
-	if (!(p->flags & PF_KTHREAD))
+	if (!(p->flags & PF_KTHREAD)) {
+		/* Try to use preferred CPU if task's affinity allows */
+		if (task_can_migrate_to_preferred(p, cpu))
+			return false;
 		return cpu_active(cpu);
+	}
 
 	/* KTHREAD_IS_PER_CPU is always allowed. */
 	if (kthread_is_per_cpu(p))
@@ -2537,7 +2559,11 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
 	if (cpu_dying(cpu))
 		return false;
 
-	/* But are allowed during online. */
+	/* Try to keep unbound kthreads on a preferred CPU if possible. */
+	if (task_can_migrate_to_preferred(p, cpu))
+		return false;
+
+	/* Otherwise, they are allowed to run on online CPU. */
 	return cpu_online(cpu);
 }
 
author maintainer
#26 lore raw

PATCH v14 7/13 sched/fair: Load balance only among preferred CPUs

In reply to Shrikanth Hegde

When a CPU is marked as non-preferred, any load pulled towards it is pointless since the task will be pushed out again in the next tick. So, consider only preferred CPUs for load balancing.

This ensures load balancing does not fight against the push task mechanism which happens at the tick. Also, this stops active balancing from happening on a non-preferred CPU pulling the load.

This also means there is no load balancing if a task is pinned only to non-preferred CPUs. They will continue to run where they were previously running before the CPUs were marked as non-preferred.

Bail out early for NEWIDLE balancing, as load balancing is done only on preferred CPUs. Note that idle balancing is allowed to go through, since that naturally updates nohz.next_balance when all the idle CPUs are non-preferred.

Also, optimization in find_new_ilb() is skipped. The steal governor driver, which is introduced in later patches, updates the preferred CPUs state in descending order. find_new_ilb() checks for idle CPUs in ascending order. Hence, in most common scenarios, the idle CPU found by find_new_ilb() will already be a preferred CPU. When all idle CPUs are non-preferred, the first idle CPU has to be chosen anyway. All of this is naturally handled in find_new_ilb() currently. Adding additional complexity to it for rare edge cases is not necessary.

---
 kernel/sched/fair.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
kernel/sched/fair.c +3 -5
index 03206e15e6fe..1c687c3c70f1 100644
@@ -13467,7 +13467,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 	};
 	bool need_unlock = false;
 
-	cpumask_and(cpus, sched_domain_span(sd), cpu_active_mask);
+	cpumask_and(cpus, sched_domain_span(sd), cpu_preferred_mask);
 
 	schedstat_inc(sd->lb_count[idle]);
 
@@ -14582,10 +14582,8 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)
 	 */
 	this_rq->idle_stamp = rq_clock(this_rq);
 
-	/*
-	 * Do not pull tasks towards !active CPUs...
-	 */
-	if (!cpu_active(this_cpu))
+	/* Do not pull tasks towards !preferred CPUs */
+	if (!cpu_preferred(this_cpu))
 		return 0;
 
 	/*
Signature · 1 line
2.52.0
#27 lore raw

Re: [PATCH v14 07/13] sched/fair: Load balance only among preferred CPUs

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] sched: capacity mismatch in load balance due to restricted env->cpus --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] sched: capacity mismatch in load balance due to restricted env->cpus --

commit 614fa1366931a4c3a90955d4b7c85dd2e0e3bb49
Author: Shrikanth Hegde <[email protected]>
Subject: sched/fair: Load balance only among preferred CPUs

This commit restricts load balancing to preferred CPUs to avoid conflicting with the tick's push task mechanism. It replaces cpu_active_mask with cpu_preferred_mask during load balance CPU mask initialization and bails out early in newidle balancing if the CPU is not preferred.

kernel/sched/fair.c static int sched_balance_rq(int this_cpu, struct rq *this_rq, · line 13470

Quoted 10 lines
kernel/sched/fair.c +1 -1
index 03206e15e6fe4..1c687c3c70f11 100644
@@ -13467,7 +13467,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 	};
 	bool need_unlock = false;
 
-	cpumask_and(cpus, sched_domain_span(sd), cpu_active_mask);
+	cpumask_and(cpus, sched_domain_span(sd), cpu_preferred_mask);

[Severity: High]
Will this change create a capacity mismatch in the load balancer?

When env->cpus is restricted to cpu_preferred_mask here, update_sg_lb_stats() will only aggregate group load for those preferred CPUs:

kernel/sched/fair.c:update_sg_lb_stats() {
    ...
    for_each_cpu_and(i, sched_group_span(group), env->cpus) {
        struct rq *rq = cpu_rq(i);
        unsigned long load = cpu_load(rq);

        sgs->group_load += load;
    ...
}

However, the group's capacity is set using the full capacity of all active CPUs:

kernel/sched/fair.c:update_sg_lb_stats() {
    ...
    sgs->group_capacity = group->sgc->capacity;
    ...
}

Could the mismatch between the deflated group_load and the full group_capacity artificially lower sgs->avg_load?

This phantom spare capacity might cause the load balancer to aggressively pull tasks from other groups into the preferred CPUs of the destination group, overloading them.

Can this also prevent tasks on non-preferred CPUs from ever being load balanced?

kernel/sched/fair.c:sched_balance_find_src_rq() {
    ...
    for_each_cpu_and(i, sched_group_span(group), env->cpus) {
    ...
}

Because this loop uses env->cpus, the load balancer ignores non-preferred CPUs entirely, even if they are overloaded.

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
#28 lore raw

[tip: sched/core] sched/fair: Load balance only among preferred CPUs

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     4ee29b029058a8f06fbb91425e1d9351e8015b53
Gitweb:        https://git.kernel.org/tip/4ee29b029058a8f06fbb91425e1d9351e8015b53
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:22 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:42 +02:00

sched/fair: Load balance only among preferred CPUs

When a CPU is marked as non-preferred, any load pulled towards it is pointless since the task will be pushed out again in the next tick. So, consider only preferred CPUs for load balancing.

This ensures load balancing does not fight against the push task mechanism which happens at the tick. Also, this stops active balancing from happening on a non-preferred CPU pulling the load.

This also means there is no load balancing if a task is pinned only to non-preferred CPUs. They will continue to run where they were previously running before the CPUs were marked as non-preferred.

Bail out early for NEWIDLE balancing, as load balancing is done only on preferred CPUs. Note that idle balancing is allowed to go through, since that naturally updates nohz.next_balance when all the idle CPUs are non-preferred.

Also, optimization in find_new_ilb() is skipped. The steal governor driver, which is introduced in later patches, updates the preferred CPUs state in descending order. find_new_ilb() checks for idle CPUs in ascending order. Hence, in most common scenarios, the idle CPU found by find_new_ilb() will already be a preferred CPU. When all idle CPUs are non-preferred, the first idle CPU has to be chosen anyway. All of this is naturally handled in find_new_ilb() currently. Adding additional complexity to it for rare edge cases is not necessary.

---
 kernel/sched/fair.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
kernel/sched/fair.c +3 -5
index 03206e1..1c687c3 100644
@@ -13467,7 +13467,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 	};
 	bool need_unlock = false;
 
-	cpumask_and(cpus, sched_domain_span(sd), cpu_active_mask);
+	cpumask_and(cpus, sched_domain_span(sd), cpu_preferred_mask);
 
 	schedstat_inc(sd->lb_count[idle]);
 
@@ -14582,10 +14582,8 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)
 	 */
 	this_rq->idle_stamp = rq_clock(this_rq);
 
-	/*
-	 * Do not pull tasks towards !active CPUs...
-	 */
-	if (!cpu_active(this_cpu))
+	/* Do not pull tasks towards !preferred CPUs */
+	if (!cpu_preferred(this_cpu))
 		return 0;
 
 	/*
author maintainer
#29 lore raw

PATCH v14 8/13 sched/core: Push current task from non preferred CPU

In reply to Shrikanth Hegde

Actively push out the current running task on a non-preferred CPU (NPC). Since the task is currently running, a stopper thread must be queued to push the task out. However, if the task is pinned only to non-preferred CPUs, it will continue running there.
This helps to maintain userspace affinities, unlike CPU hotplug or isolated cpusets.

The implementation follows the structure of __balance_push_cpu_stop(), but is kept separate to handle the preferred-CPU specific conditions and pending-work state under CONFIG_PREFERRED_CPU.

Add the npc_push_work_pending flag to protect the work buffer.

For now, only the currently running task is pushed out. This keeps the code simpler. In the future, an optimization may be added to move all queued tasks on the runqueue.

This works only for the FAIR scheduling class.

---
 kernel/sched/core.c  | 83 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h |  9 +++++
 2 files changed, 92 insertions(+)
kernel/sched/core.c +83 -0
index 04400f934cc7..5049eff58fb7 100644
@@ -5809,6 +5809,9 @@ void sched_tick(void)
 	unsigned long hw_pressure;
 	u64 resched_latency;
 
+	if (!cpu_preferred(cpu))
+		sched_push_current_non_preferred_cpu(rq);
+
 	if (housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE))
 		arch_scale_freq_tick();
 
@@ -11201,3 +11204,83 @@ void sched_change_end(struct sched_change_ctx *ctx)
 		p->sched_class->prio_changed(rq, p, ctx->prio);
 	}
 }
+
+#ifdef CONFIG_PREFERRED_CPU
+static DEFINE_PER_CPU(struct cpu_stop_work, npc_push_task_work);
+
+static int sched_non_preferred_cpu_push_stop(void *arg)
+{
+	struct task_struct *p = arg;
+	struct rq *rq = this_rq();
+	struct rq_flags rf;
+	int cpu;
+
+	if (cpu_preferred(rq->cpu)) {
+		scoped_guard(rq_lock_irqsave, rq)
+			rq->npc_push_work_pending = false;
+		put_task_struct(p);
+		return 0;
+	}
+
+	scoped_guard (raw_spinlock_irq, &p->pi_lock) {
+		/*
+		 * select_fallback_rq() may acquire the rq lock in case of
+		 * fallback. So call it before grabbing rq lock. If the task
+		 * migrates to another CPU before the rq lock is acquired,
+		 * subsequent validation of task's current rq will help to
+		 * safely bail out.
+		 */
+		cpu = select_fallback_rq(rq->cpu, p);
+		rq_lock(rq, &rf);
+		rq->npc_push_work_pending = false;
+		update_rq_clock(rq);
+		context_unsafe_alias(rq);
+
+		if (task_rq(p) == rq && task_on_rq_queued(p))
+			rq = __migrate_task(rq, &rf, p, cpu);
+		rq_unlock(rq, &rf);
+	}
+
+	put_task_struct(p);
+	return 0;
+}
+
+/*
+ * Push the current task running on non-preferred CPU(npc).
+ * Using this non preferred CPU will lead to more contention
+ * in the host. So it is better not to use this CPU.
+ *
+ * Since task is running, call a stopper to push the task out. This is
+ * similar to how task moves during hotplug. In select_fallback_rq() a
+ * preferred CPU will be chosen and henceforth task shouldn't come back to
+ * this CPU again.
+ *
+ * Works for FAIR class only.
+ *
+ * If task is affined only on non-preferred CPUs, no point in moving it out.
+ */
+void sched_push_current_non_preferred_cpu(struct rq *rq)
+{
+	struct task_struct *push_task = rq->curr;
+
+	scoped_guard(rq_lock, rq) {
+		/* Push the task if its explicit affinity allows */
+		if (!task_can_migrate_to_preferred(push_task, rq->cpu))
+			return;
+
+		/* There is already a stopper thread. Don't race with it. */
+		if (rq->npc_push_work_pending)
+			return;
+
+		if (is_migration_disabled(push_task))
+			return;
+
+		rq->npc_push_work_pending = true;
+	}
+
+	/* sched_tick runs with interrupts disabled. */
+	get_task_struct(push_task);
+	stop_one_cpu_nowait(rq->cpu, sched_non_preferred_cpu_push_stop,
+			    push_task, this_cpu_ptr(&npc_push_task_work));
+}
+#endif
kernel/sched/sched.h +9 -0
index b98084e1f5b0..ee482bb12a66 100644
@@ -1326,6 +1326,9 @@ struct rq {
 #ifdef CONFIG_PARAVIRT_TIME_ACCOUNTING
 	u64			prev_steal_time_rq;
 #endif
+#ifdef CONFIG_PREFERRED_CPU
+	bool			npc_push_work_pending;
+#endif
 
 	/* calc_load related fields */
 	unsigned long		calc_load_update;
@@ -4292,4 +4295,10 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)
 
 #include "ext/ext.h"
 
+#ifdef CONFIG_PREFERRED_CPU
+void sched_push_current_non_preferred_cpu(struct rq *rq);
+#else	/* !CONFIG_PREFERRED_CPU */
+static inline void sched_push_current_non_preferred_cpu(struct rq *rq) { }
+#endif
+
 #endif /* _KERNEL_SCHED_SCHED_H */
Signature · 1 line
2.52.0
#30 lore raw

Re: [PATCH v14 08/13] sched/core: Push current task from non preferred CPU

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] sched/core: FAIR kthreads cause stopper thread livelock on non-preferred CPUs --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] sched/core: FAIR kthreads cause stopper thread livelock on non-preferred CPUs --

commit a6718dc77694942ae8b63a9eb25ace9c01b3c5ef
Author: Shrikanth Hegde <[email protected]>

sched/core: Push current task from non preferred CPU

This patch actively pushes the currently running task off a non-preferred CPU using a stopper thread, specifically targeting tasks in the FAIR scheduling class. It adds the npc_push_work_pending flag to avoid queueing multiple stopper threads.

Quoted 3 lines
kernel/sched/core.c +0 -0

[ ... ]

Quoted 23 lines
kernel/sched/core.c +23 -0
+static int sched_non_preferred_cpu_push_stop(void *arg)
+{
+	struct task_struct *p = arg;
+	struct rq *rq = this_rq();
+	struct rq_flags rf;
+	int cpu;
+
+	if (cpu_preferred(rq->cpu)) {
+		scoped_guard(rq_lock_irqsave, rq)
+			rq->npc_push_work_pending = false;
+		put_task_struct(p);
+		return 0;
+	}
+
+	scoped_guard (raw_spinlock_irq, &p->pi_lock) {
+		/*
+		 * select_fallback_rq() may acquire the rq lock in case of
+		 * fallback. So call it before grabbing rq lock. If the task
+		 * migrates to another CPU before the rq lock is acquired,
+		 * subsequent validation of task's current rq will help to
+		 * safely bail out.
+		 */
+		cpu = select_fallback_rq(rq->cpu, p);

[ ... ]

kernel/sched/core.c void sched_change_end(struct sched_change_ctx *ctx) · line 11269

Quoted 8 lines
kernel/sched/core.c +8 -0
+void sched_push_current_non_preferred_cpu(struct rq *rq)
+{
+	struct task_struct *push_task = rq->curr;
+
+	scoped_guard(rq_lock, rq) {
+		/* Push the task if its explicit affinity allows */
+		if (!task_can_migrate_to_preferred(push_task, rq->cpu))
+			return;

[Severity: High]
Can this result in a livelock with endless stopper thread preemptions when the task being pushed is a FAIR kthread (such as an unbound kworker)?

When task_can_migrate_to_preferred() returns true for a FAIR kthread, this function successfully queues a stopper thread. However, when the stopper thread runs sched_non_preferred_cpu_push_stop() and calls select_fallback_rq(), it ultimately evaluates is_cpu_allowed(), which skips the preferred-CPU check for kthreads:

kernel/sched/core.c:is_cpu_allowed() {
    ...
    /* Non kernel threads are not allowed during either online or offline. */
    if (!(p->flags & PF_KTHREAD)) {
        /* Try to use preferred CPU if task's affinity allows */
        if (task_can_migrate_to_preferred(p, cpu))
            return false;
        return cpu_active(cpu);
    }
    ...
}

Because the kthread bypasses the preferred-CPU enforcement, select_fallback_rq() can pick the first online CPU in the local node, which might be the exact same non-preferred CPU we are trying to push it away from.

If that happens, the migration becomes a no-op, npc_push_work_pending gets cleared, and sched_tick() will just restart the same sequence on the next tick, wasting CPU cycles in an endless loop.

Quoted 4 lines
kernel/sched/core.c +4 -0
+
+		/* There is already a stopper thread. Don't race with it. */
+		if (rq->npc_push_work_pending)
+			return;

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#31 lore raw

[tip: sched/core] sched/core: Push current task from non preferred CPU

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     74699f56ebcfe0f401ee6ce4c5a72822101bb222
Gitweb:        https://git.kernel.org/tip/74699f56ebcfe0f401ee6ce4c5a72822101bb222
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:23 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:42 +02:00

sched/core: Push current task from non preferred CPU

Actively push out the current running task on a non-preferred CPU (NPC). Since the task is currently running, a stopper thread must be queued to push the task out. However, if the task is pinned only to non-preferred CPUs, it will continue running there.
This helps to maintain userspace affinities, unlike CPU hotplug or isolated cpusets.

The implementation follows the structure of __balance_push_cpu_stop(), but is kept separate to handle the preferred-CPU specific conditions and pending-work state under CONFIG_PREFERRED_CPU.

Add the npc_push_work_pending flag to protect the work buffer.

For now, only the currently running task is pushed out. This keeps the code simpler. In the future, an optimization may be added to move all queued tasks on the runqueue.

This works only for the FAIR scheduling class.

---
 kernel/sched/core.c  | 83 +++++++++++++++++++++++++++++++++++++++++++-
 kernel/sched/sched.h |  9 +++++-
 2 files changed, 92 insertions(+)
kernel/sched/core.c +83 -0
index d095fb1..0e40055 100644
@@ -5829,6 +5829,9 @@ void sched_tick(void)
 	unsigned long hw_pressure;
 	u64 resched_latency;
 
+	if (!cpu_preferred(cpu))
+		sched_push_current_non_preferred_cpu(rq);
+
 	if (housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE))
 		arch_scale_freq_tick();
 
@@ -11253,3 +11256,83 @@ void sched_change_end(struct sched_change_ctx *ctx)
 		p->sched_class->prio_changed(rq, p, ctx->prio);
 	}
 }
+
+#ifdef CONFIG_PREFERRED_CPU
+static DEFINE_PER_CPU(struct cpu_stop_work, npc_push_task_work);
+
+static int sched_non_preferred_cpu_push_stop(void *arg)
+{
+	struct task_struct *p = arg;
+	struct rq *rq = this_rq();
+	struct rq_flags rf;
+	int cpu;
+
+	if (cpu_preferred(rq->cpu)) {
+		scoped_guard(rq_lock_irqsave, rq)
+			rq->npc_push_work_pending = false;
+		put_task_struct(p);
+		return 0;
+	}
+
+	scoped_guard (raw_spinlock_irq, &p->pi_lock) {
+		/*
+		 * select_fallback_rq() may acquire the rq lock in case of
+		 * fallback. So call it before grabbing rq lock. If the task
+		 * migrates to another CPU before the rq lock is acquired,
+		 * subsequent validation of task's current rq will help to
+		 * safely bail out.
+		 */
+		cpu = select_fallback_rq(rq->cpu, p);
+		rq_lock(rq, &rf);
+		rq->npc_push_work_pending = false;
+		update_rq_clock(rq);
+		context_unsafe_alias(rq);
+
+		if (task_rq(p) == rq && task_on_rq_queued(p))
+			rq = __migrate_task(rq, &rf, p, cpu);
+		rq_unlock(rq, &rf);
+	}
+
+	put_task_struct(p);
+	return 0;
+}
+
+/*
+ * Push the current task running on non-preferred CPU(npc).
+ * Using this non preferred CPU will lead to more contention
+ * in the host. So it is better not to use this CPU.
+ *
+ * Since task is running, call a stopper to push the task out. This is
+ * similar to how task moves during hotplug. In select_fallback_rq() a
+ * preferred CPU will be chosen and henceforth task shouldn't come back to
+ * this CPU again.
+ *
+ * Works for FAIR class only.
+ *
+ * If task is affined only on non-preferred CPUs, no point in moving it out.
+ */
+void sched_push_current_non_preferred_cpu(struct rq *rq)
+{
+	struct task_struct *push_task = rq->curr;
+
+	scoped_guard(rq_lock, rq) {
+		/* Push the task if its explicit affinity allows */
+		if (!task_can_migrate_to_preferred(push_task, rq->cpu))
+			return;
+
+		/* There is already a stopper thread. Don't race with it. */
+		if (rq->npc_push_work_pending)
+			return;
+
+		if (is_migration_disabled(push_task))
+			return;
+
+		rq->npc_push_work_pending = true;
+	}
+
+	/* sched_tick runs with interrupts disabled. */
+	get_task_struct(push_task);
+	stop_one_cpu_nowait(rq->cpu, sched_non_preferred_cpu_push_stop,
+			    push_task, this_cpu_ptr(&npc_push_task_work));
+}
+#endif
kernel/sched/sched.h +9 -0
index d41d23a..7d2ec52 100644
@@ -1326,6 +1326,9 @@ struct rq {
 #ifdef CONFIG_PARAVIRT_TIME_ACCOUNTING
 	u64			prev_steal_time_rq;
 #endif
+#ifdef CONFIG_PREFERRED_CPU
+	bool			npc_push_work_pending;
+#endif
 
 	/* calc_load related fields */
 	unsigned long		calc_load_update;
@@ -4299,4 +4302,10 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)
 
 #include "ext/ext.h"
 
+#ifdef CONFIG_PREFERRED_CPU
+void sched_push_current_non_preferred_cpu(struct rq *rq);
+#else	/* !CONFIG_PREFERRED_CPU */
+static inline void sched_push_current_non_preferred_cpu(struct rq *rq) { }
+#endif
+
 #endif /* _KERNEL_SCHED_SCHED_H */
author maintainer
#32 lore raw

PATCH v14 9/13 sched/debug: Add migration stats due to non preferred CPUs

In reply to Shrikanth Hegde

Add a new per-task stat,
- nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.

Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update /proc/schedstat. Hence increasing the schedstat version is not necessary.

---
 include/linux/sched.h | 1 +
 kernel/sched/core.c   | 9 +++++++--
 kernel/sched/debug.c  | 1 +
 3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
index d7cc77181ef9..48639b0c75d0 100644
@@ -554,6 +554,7 @@ struct sched_statistics {
 	u64				nr_failed_migrations_running;
 	u64				nr_failed_migrations_hot;
 	u64				nr_forced_migrations;
+	u64				nr_migrations_cpu_non_preferred;
 
 	u64				nr_wakeups;
 	u64				nr_wakeups_sync;
kernel/sched/core.c +7 -2
index 5049eff58fb7..7465c983e6f6 100644
@@ -11236,8 +11236,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg)
 		update_rq_clock(rq);
 		context_unsafe_alias(rq);
 
-		if (task_rq(p) == rq && task_on_rq_queued(p))
-			rq = __migrate_task(rq, &rf, p, cpu);
+		if (task_rq(p) == rq && task_on_rq_queued(p)) {
+			struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
+
+			if (rq != dest_rq)
+				schedstat_inc(p->stats.nr_migrations_cpu_non_preferred);
+			rq = dest_rq;
+		}
 		rq_unlock(rq, &rf);
 	}
 
kernel/sched/debug.c +1 -0
index fda10b3fa249..e6a3b516c703 100644
@@ -1440,6 +1440,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 		P_SCHEDSTAT(nr_failed_migrations_running);
 		P_SCHEDSTAT(nr_failed_migrations_hot);
 		P_SCHEDSTAT(nr_forced_migrations);
+		P_SCHEDSTAT(nr_migrations_cpu_non_preferred);
 		P_SCHEDSTAT(nr_wakeups);
 		P_SCHEDSTAT(nr_wakeups_sync);
 		P_SCHEDSTAT(nr_wakeups_migrate);
Signature · 1 line
2.52.0
#33 lore raw

Re: [PATCH v14 09/13] sched/debug: Add migration stats due to non preferred CPUs

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines

Add a new per-task stat,
- nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.

Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update /proc/schedstat. Hence increasing the schedstat version is not necessary.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
#34 lore raw

[tip: sched/core] sched/debug: Add migration stats due to non preferred CPUs

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     68957caaa9c0e127bc87fee64e45be843f6c5c91
Gitweb:        https://git.kernel.org/tip/68957caaa9c0e127bc87fee64e45be843f6c5c91
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:24 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:42 +02:00

sched/debug: Add migration stats due to non preferred CPUs

Add a new per-task stat,
- nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.

Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update /proc/schedstat. Hence increasing the schedstat version is not necessary.

---
 include/linux/sched.h |  1 +
 kernel/sched/core.c   |  9 +++++++--
 kernel/sched/debug.c  |  1 +
 3 files changed, 9 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
index d7cc771..48639b0 100644
@@ -554,6 +554,7 @@ struct sched_statistics {
 	u64				nr_failed_migrations_running;
 	u64				nr_failed_migrations_hot;
 	u64				nr_forced_migrations;
+	u64				nr_migrations_cpu_non_preferred;
 
 	u64				nr_wakeups;
 	u64				nr_wakeups_sync;
kernel/sched/core.c +7 -2
index 0e40055..0bb86a4 100644
@@ -11288,8 +11288,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg)
 		update_rq_clock(rq);
 		context_unsafe_alias(rq);
 
-		if (task_rq(p) == rq && task_on_rq_queued(p))
-			rq = __migrate_task(rq, &rf, p, cpu);
+		if (task_rq(p) == rq && task_on_rq_queued(p)) {
+			struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
+
+			if (rq != dest_rq)
+				schedstat_inc(p->stats.nr_migrations_cpu_non_preferred);
+			rq = dest_rq;
+		}
 		rq_unlock(rq, &rf);
 	}
 
kernel/sched/debug.c +1 -0
index fda10b3..e6a3b51 100644
@@ -1440,6 +1440,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 		P_SCHEDSTAT(nr_failed_migrations_running);
 		P_SCHEDSTAT(nr_failed_migrations_hot);
 		P_SCHEDSTAT(nr_forced_migrations);
+		P_SCHEDSTAT(nr_migrations_cpu_non_preferred);
 		P_SCHEDSTAT(nr_wakeups);
 		P_SCHEDSTAT(nr_wakeups_sync);
 		P_SCHEDSTAT(nr_wakeups_migrate);
#35 lore raw

Re: [PATCH v14 09/13] sched/debug: Add migration stats due to non preferred CPUs

In reply to Shrikanth Hegde
Quoted 9 lines · Shrikanth Hegde

Add a new per-task stat,
- nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.

Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update /proc/schedstat. Hence increasing the schedstat version is not necessary.

...

Quoted 9 lines
kernel/sched/core.c +0 -1
index 5049eff58fb7..7465c983e6f6 100644
@@ -11236,8 +11236,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg)
 		update_rq_clock(rq);
 		context_unsafe_alias(rq);
 
-		if (task_rq(p) == rq && task_on_rq_queued(p))

kernel/sched/core.c static int sched_non_preferred_cpu_push_stop(void *arg) · line 11247

Quoted 10 lines
kernel/sched/core.c +7 -1
-			rq = __migrate_task(rq, &rf, p, cpu);
+		if (task_rq(p) == rq && task_on_rq_queued(p)) {
+			struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
+
+			if (rq != dest_rq)
+				schedstat_inc(p->stats.nr_migrations_cpu_non_preferred);
+			rq = dest_rq;
+		}
 		rq_unlock(rq, &rf);
 	}

This breaks the build for me with clang-23+ (which have context analysis enabled by default), although it bisects to the final patch of the series since this is under CONFIG_PREFERRED_CPU and it is not selected until then.

  kernel/sched/core.c:11292:25: error: calling function '__migrate_task' requires holding raw_spinlock 'rq_lockp(rq)' exclusively [-Werror,-Wthread-safety-analysis]
   11292 |                         struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
         |                                              ^
  kernel/sched/core.c:11298:3: error: releasing raw_spinlock 'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis]
   11298 |                 rq_unlock(rq, &rf);
         |                 ^
  kernel/sched/core.c:11303:1: error: raw_spinlock 'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis]
   11303 | }
         | ^
  kernel/sched/core.c:11286:3: note: raw_spinlock acquired here
   11286 |                 rq_lock(rq, &rf);
         |                 ^
  3 errors generated.

Not sure what the proper fix for this is, maybe another context_unsafe_alias()? cc Marco just in case

Signature · 2 lines
Cheers,
Nathan
author maintainer
#36 lore raw

Re: [PATCH v14 09/13] sched/debug: Add migration stats due to non preferred CPUs

Hi Nathan. Thanks for report.

Quoted 50 lines · Nathan Chancellor
Quoted 9 lines · Shrikanth Hegde

Add a new per-task stat,
- nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.

Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update /proc/schedstat. Hence increasing the schedstat version is not necessary.

...

Quoted 9 lines
kernel/sched/core.c +0 -1
index 5049eff58fb7..7465c983e6f6 100644
@@ -11236,8 +11236,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg)
  		update_rq_clock(rq);
  		context_unsafe_alias(rq);
 
-		if (task_rq(p) == rq && task_on_rq_queued(p))

kernel/sched/core.c static int sched_non_preferred_cpu_push_stop(void *arg) · line 11247

Quoted 10 lines
kernel/sched/core.c +7 -1
-			rq = __migrate_task(rq, &rf, p, cpu);
+		if (task_rq(p) == rq && task_on_rq_queued(p)) {
+			struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
+
+			if (rq != dest_rq)
+				schedstat_inc(p->stats.nr_migrations_cpu_non_preferred);
+			rq = dest_rq;
+		}
  		rq_unlock(rq, &rf);
  	}

This breaks the build for me with clang-23+ (which have context analysis enabled by default), although it bisects to the final patch of the series since this is under CONFIG_PREFERRED_CPU and it is not selected until then.

   kernel/sched/core.c:11292:25: error: calling function '__migrate_task' requires holding raw_spinlock 'rq_lockp(rq)' exclusively [-Werror,-Wthread-safety-analysis]
    11292 |                         struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
          |                                              ^
   kernel/sched/core.c:11298:3: error: releasing raw_spinlock 'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis]
    11298 |                 rq_unlock(rq, &rf);
          |                 ^
   kernel/sched/core.c:11303:1: error: raw_spinlock 'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis]
    11303 | }
          | ^
   kernel/sched/core.c:11286:3: note: raw_spinlock acquired here
    11286 |                 rq_lock(rq, &rf);
          |                 ^
   3 errors generated.
Quoted 3 lines

Not sure what the proper fix for this is, maybe another context_unsafe_alias()? cc Marco just in case

I suspect it is due to using of dest_rq = rq and rq is changing context.

let me try locally and see the fix.
One fix is use rq->cpu instead of rq comparison to see if migration happened.

author maintainer
#37 lore raw

Re: [PATCH v14 09/13] sched/debug: Add migration stats due to non preferred CPUs

Hi Nathan.

Quoted 52 lines · Shrikanth Hegde

Hi Nathan. Thanks for report.

Quoted 49 lines · Nathan Chancellor
Quoted 9 lines · Shrikanth Hegde

Add a new per-task stat,
- nr_migrations_cpu_non_preferred: number of push migrations while the CPU is non-preferred.

Since this new stat is per-task, it changes only /proc/<pid>/sched. It doesn't update /proc/schedstat. Hence increasing the schedstat version is not necessary.

...

Quoted 8 lines
kernel/sched/core.c +0 -1
index 5049eff58fb7..7465c983e6f6 100644
@@ -11236,8 +11236,13 @@ static int sched_non_preferred_cpu_push_stop(void *arg)
           update_rq_clock(rq);
           context_unsafe_alias(rq);
-        if (task_rq(p) == rq && task_on_rq_queued(p))

kernel/sched/core.c static int sched_non_preferred_cpu_push_stop(void *arg) · line 11246

Quoted 10 lines
kernel/sched/core.c +7 -1
-            rq = __migrate_task(rq, &rf, p, cpu);
+        if (task_rq(p) == rq && task_on_rq_queued(p)) {
+            struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
+
+            if (rq != dest_rq)
+                schedstat_inc(p->stats.nr_migrations_cpu_non_preferred);
+            rq = dest_rq;
+        }
           rq_unlock(rq, &rf);
       }

This breaks the build for me with clang-23+ (which have context analysis enabled by default), although it bisects to the final patch of the series since this is under CONFIG_PREFERRED_CPU and it is not selected until then.

   kernel/sched/core.c:11292:25: error: calling function '__migrate_task' requires holding raw_spinlock 'rq_lockp(rq)' exclusively [-Werror,-Wthread-safety-analysis] 11292 |                         struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
          |                                              ^
   kernel/sched/core.c:11298:3: error: releasing raw_spinlock 'rq_lockp(rq)' that was not held [-Werror,-Wthread-safety-analysis]
    11298 |                 rq_unlock(rq, &rf);
          |                 ^
   kernel/sched/core.c:11303:1: error: raw_spinlock 'rq_lockp(__this_rq())' is not held on every path through here [-Werror,-Wthread-safety-analysis] 11303 | }
          | ^
   kernel/sched/core.c:11286:3: note: raw_spinlock acquired here
    11286 |                 rq_lock(rq, &rf);
          |                 ^
   3 errors generated.

Quoted 8 lines
Quoted 3 lines

Not sure what the proper fix for this is, maybe another context_unsafe_alias()? cc Marco just in case

I suspect it is due to using of dest_rq = rq and rq is changing context.

let me try locally and see the fix.
One fix is use rq->cpu instead of rq comparison to see if migration happened.

Well, it was rather due to Patch 8/13 which had rq marked as context_unsafe_alias after acquiring it.

I have tried below and that helps to fix the warnings. I will write a changelog and send it across soon.

Let me know if it works for you.

---

kernel/sched/core.c +1 -1
index 0bb86a43a592..23677d76f9d2 100644
@@ -11283,10 +11283,10 @@ static int sched_non_preferred_cpu_push_stop(void *arg)
 		 * safely bail out.
 		 */
 		cpu = select_fallback_rq(rq->cpu, p);
+		context_unsafe_alias(rq);
 		rq_lock(rq, &rf);
 		rq->npc_push_work_pending = false;
 		update_rq_clock(rq);
-		context_unsafe_alias(rq);
 
 		if (task_rq(p) == rq && task_on_rq_queued(p)) {
 			struct rq *dest_rq = __migrate_task(rq, &rf, p, cpu);
author maintainer
#38 lore raw

PATCH v14 10/13 virt: Introduce steal governor driver

In reply to Shrikanth Hegde

Introduce a new driver in virt named steal_governor. This driver will compute the steal time and drive the policy decisions regarding the preferred CPU state.

Note that this driver is strictly intended for actual guests. Hence block it on Xen dom0.
More details can be found in Documentation/driver-api/steal-governor.rst.

A new kconfig called STEAL_GOVERNOR is introduced in subsequent patches, which enables this driver. This driver will select CONFIG_PREFERRED_CPU. This makes configs driven by user preference/configuration. When the driver is disabled, preferred CPUs remain the same as active CPUs.

The file layout of the driver is kept simple for now. The code is in drivers/virt/steal_governor.c, and the configs are part of drivers/virt/Kconfig.

The main structure of the steal governor contains:
- work, delay: Deferred periodic work function variables.
- steal, time: Used to calculate deltas during periodic work.
- interval_ms, high_threshold, low_threshold: Tuning knobs for the steal governor.

While there, add MAINTAINERS entry for this new driver.

---
 Documentation/driver-api/index.rst          |   1 +
 Documentation/driver-api/steal-governor.rst | 151 ++++++++++++++++++++
 MAINTAINERS                                 |   9 ++
 drivers/virt/steal_governor.c               |  78 ++++++++++
 4 files changed, 239 insertions(+)
 create mode 100644 Documentation/driver-api/steal-governor.rst
 create mode 100644 drivers/virt/steal_governor.c
Documentation/driver-api/index.rst +1 -0
index 6601a258690f..26b7638a327d 100644
@@ -139,6 +139,7 @@ Subsystem-specific APIs
    sm501
    soundwire/index
    spi
+   steal-governor
    surface_aggregator/index
    switchtec
    sync_file
Documentation/driver-api/steal-governor.rst new file +151 -0

151 lines not on this page · raw message

MAINTAINERS +9 -0
index 3a19da74d00c..28003d36b3f7 100644
@@ -26223,6 +26223,15 @@ F:	rust/helpers/jump_label.c
 F:	rust/kernel/generated_arch_static_branch_asm.rs.S
 F:	rust/kernel/jump_label.rs
 
+STEAL GOVERNOR DRIVER
+M:	Shrikanth Hegde <[email protected]>
+R:	Yury Norov <[email protected]>
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core
+F:	Documentation/driver-api/steal-governor.rst
+F:	drivers/virt/steal_governor.c
+
 STI AUDIO (ASoC) DRIVERS
 M:	Arnaud Pouliquen <[email protected]>
 L:	[email protected]
drivers/virt/steal_governor.c new file +78 -0

78 lines not on this page · raw message

Signature · 1 line
2.52.0
#39 lore raw

Re: [PATCH v14 10/13] virt: Introduce steal governor driver

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 23 lines

Introduce a new driver in virt named steal_governor. This driver will compute the steal time and drive the policy decisions regarding the preferred CPU state.

Note that this driver is strictly intended for actual guests. Hence block it on Xen dom0.
More details can be found in Documentation/driver-api/steal-governor.rst.

A new kconfig called STEAL_GOVERNOR is introduced in subsequent patches, which enables this driver. This driver will select CONFIG_PREFERRED_CPU. This makes configs driven by user preference/configuration. When the driver is disabled, preferred CPUs remain the same as active CPUs.

The file layout of the driver is kept simple for now. The code is in drivers/virt/steal_governor.c, and the configs are part of drivers/virt/Kconfig.

The main structure of the steal governor contains:
- work, delay: Deferred periodic work function variables.
- steal, time: Used to calculate deltas during periodic work.
- interval_ms, high_threshold, low_threshold: Tuning knobs for the steal governor.

Quoted 5 lines

While there, add MAINTAINERS entry for this new driver.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
#40 lore raw

[tip: sched/core] virt: Introduce steal governor driver

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     9a8e740ee9f69ec857ffa0c76cf1a01d1cb7360f
Gitweb:        https://git.kernel.org/tip/9a8e740ee9f69ec857ffa0c76cf1a01d1cb7360f
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:25 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:43 +02:00

virt: Introduce steal governor driver

Introduce a new driver in virt named steal_governor. This driver will compute the steal time and drive the policy decisions regarding the preferred CPU state.

Note that this driver is strictly intended for actual guests. Hence block it on Xen dom0.
More details can be found in Documentation/driver-api/steal-governor.rst.

A new kconfig called STEAL_GOVERNOR is introduced in subsequent patches, which enables this driver. This driver will select CONFIG_PREFERRED_CPU. This makes configs driven by user preference/configuration. When the driver is disabled, preferred CPUs remain the same as active CPUs.

The file layout of the driver is kept simple for now. The code is in drivers/virt/steal_governor.c, and the configs are part of drivers/virt/Kconfig.

The main structure of the steal governor contains:
- work, delay: Deferred periodic work function variables.
- steal, time: Used to calculate deltas during periodic work.
- interval_ms, high_threshold, low_threshold: Tuning knobs for the steal governor.

While there, add MAINTAINERS entry for this new driver.

---
 Documentation/driver-api/index.rst          |   1 +-
 Documentation/driver-api/steal-governor.rst | 151 +++++++++++++++++++-
 MAINTAINERS                                 |   9 +-
 drivers/virt/steal_governor.c               |  78 ++++++++++-
 4 files changed, 239 insertions(+)
 create mode 100644 Documentation/driver-api/steal-governor.rst
 create mode 100644 drivers/virt/steal_governor.c
Documentation/driver-api/index.rst +1 -0
index 6601a25..26b7638 100644
@@ -139,6 +139,7 @@ Subsystem-specific APIs
    sm501
    soundwire/index
    spi
+   steal-governor
    surface_aggregator/index
    switchtec
    sync_file
Documentation/driver-api/steal-governor.rst new file +151 -0

151 lines not on this page · raw message

MAINTAINERS +9 -0
index 3a19da7..28003d3 100644
@@ -26223,6 +26223,15 @@ F:	rust/helpers/jump_label.c
 F:	rust/kernel/generated_arch_static_branch_asm.rs.S
 F:	rust/kernel/jump_label.rs
 
+STEAL GOVERNOR DRIVER
+M:	Shrikanth Hegde <[email protected]>
+R:	Yury Norov <[email protected]>
+L:	[email protected]
+S:	Maintained
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core
+F:	Documentation/driver-api/steal-governor.rst
+F:	drivers/virt/steal_governor.c
+
 STI AUDIO (ASoC) DRIVERS
 M:	Arnaud Pouliquen <[email protected]>
 L:	[email protected]
drivers/virt/steal_governor.c new file +78 -0

78 lines not on this page · raw message

author maintainer
#41 lore raw

PATCH v14 11/13 virt/steal_governor: Add control knobs for handling steal values

In reply to Shrikanth Hegde

These are the knobs to control the steal_governor.

interval_ms:
How often steal governor checks for steal time. (Default: 1000 i.e 1 second)
This controls how fast steal governor driver reacts to changes to the contention of physical CPUs.
Can be set between 100 to 100000. i.e. 100ms to 100seconds. 100ms is kept as minimum to ensure few meaningful steal values accumulate even with HZ=100.

low_threshold:
lower threshold value in percentage * 100.
(Default: 200, i.e 2% steal is considered as low threshold) This determines what values should be considered as nil/no steal values. When steal governor see steal ratio is below or equal to this value, it will increase the preferred CPUs by 1 core. Having value as zero might cause oscillations

high_threshold:
higher threshold value in percentage * 100
(Default: 500, i.e 5% steal is considered as high threshold) This determines what values should be considered as high steal values. When steal governor sees steal ratio is higher than this value, it will reduce the preferred CPUs by 1 core.

module_param_cb methods are used to do the validation checks. This helps to ensure one configures sane values.
Since low and high are dependent, that check is done at module init.

Notes:
- Parameters values can't be changed at runtime. One has to unload the module and change it. Hence recommended to build it as module.
- Default values may not work well for all configurations. Tune it according to the system under test.

Documentation is available at: Documentation/driver-api/steal-governor.rst

---
 drivers/virt/steal_governor.c | 73 ++++++++++++++++++++++++++++++++++-
 1 file changed, 71 insertions(+), 2 deletions(-)
drivers/virt/steal_governor.c +71 -2
index 2320cbfa4b47..27f53ea16498 100644
@@ -40,7 +40,11 @@ struct steal_governor {
 	struct delayed_work	work;
 };
 
-static struct steal_governor sg_ctx;
+static struct steal_governor sg_ctx = {
+	.interval_ms	=	1000,	/* 1 second */
+	.high_threshold =	500,	/* 5% */
+	.low_threshold	=	200,	/* 2% */
+};
 
 static void restore_preferred_to_active(void)
 {
@@ -51,6 +55,62 @@ static void restore_preferred_to_active(void)
 		set_cpu_preferred(cpu, true);
 }
 
+static int param_set_interval_ms(const char *val, const struct kernel_param *kp)
+{
+	unsigned int interval;
+	int ret;
+
+	ret = kstrtouint(val, 0, &interval);
+	if (ret)
+		return ret;
+
+	if (interval < 100 || interval > 100000) {
+		pr_err("interval_ms must be between 100 and 100000\n");
+		return -EINVAL;
+	}
+
+	return param_set_uint(val, kp);
+}
+
+static const struct kernel_param_ops interval_ms_ops = {
+	.set = param_set_interval_ms,
+	.get = param_get_uint,
+};
+
+module_param_cb(interval_ms, &interval_ms_ops, &sg_ctx.interval_ms, 0444);
+MODULE_PARM_DESC(interval_ms,
+		 "Sampling frequency in milliseconds. default: 1000");
+
+static int param_set_high_threshold(const char *val, const struct kernel_param *kp)
+{
+	unsigned int threshold;
+	int ret;
+
+	ret = kstrtouint(val, 0, &threshold);
+	if (ret)
+		return ret;
+
+	if (threshold >= 100 * 100) {
+		pr_err("high_threshold (%u) can't be more than 99.99%%\n", threshold);
+		return -EINVAL;
+	}
+
+	return param_set_uint(val, kp);
+}
+
+static const struct kernel_param_ops high_threshold_ops = {
+	.set = param_set_high_threshold,
+	.get = param_get_uint,
+};
+
+module_param_cb(high_threshold, &high_threshold_ops, &sg_ctx.high_threshold, 0444);
+MODULE_PARM_DESC(high_threshold,
+		 "High steal threshold. default: 500 i.e 5%. Must be > low_threshold");
+
+module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444);
+MODULE_PARM_DESC(low_threshold,
+		 "Low steal threshold. default: 200 i.e 2%. Must be < high_threshold");
+
 static int __init steal_governor_init(void)
 {
 #ifdef CONFIG_XEN
@@ -60,7 +120,16 @@ static int __init steal_governor_init(void)
 	}
 #endif
 
-	pr_info("enabled\n");
+	if (sg_ctx.low_threshold >= sg_ctx.high_threshold) {
+		pr_err("low_threshold (%u) must be less than high_threshold (%u)\n",
+		       sg_ctx.low_threshold, sg_ctx.high_threshold);
+		return -EINVAL;
+	}
+
+	sg_ctx.delay = msecs_to_jiffies(sg_ctx.interval_ms);
+	pr_info("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n",
+		sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
+
 	return 0;
 }
 
Signature · 1 line
2.52.0
#42 lore raw

Re: [PATCH v14 11/13] virt/steal_governor: Add control knobs for handling steal values

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines

These are the knobs to control the steal_governor.

interval_ms:
How often steal governor checks for steal time. (Default: 1000 i.e 1 second)
This controls how fast steal governor driver reacts to changes to the contention of physical CPUs.
Can be set between 100 to 100000. i.e. 100ms to 100seconds. 100ms is kept as minimum to ensure few meaningful steal values accumulate even with HZ=100.

low_threshold:
lower threshold value in percentage * 100.
(Default: 200, i.e 2% steal is considered as low threshold) This determines what values should be considered as nil/no steal values. [ ... ]

Quoted 4 lines

Documentation is available at: Documentation/driver-api/steal-governor.rst

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
#43 lore raw

[tip: sched/core] virt/steal_governor: Add control knobs for handling steal values

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     4b9302d494fffee7318bd9cdd2021198b6a7badd
Gitweb:        https://git.kernel.org/tip/4b9302d494fffee7318bd9cdd2021198b6a7badd
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:26 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:43 +02:00

virt/steal_governor: Add control knobs for handling steal values

These are the knobs to control the steal_governor.

interval_ms:
How often steal governor checks for steal time. (Default: 1000 i.e 1 second)
This controls how fast steal governor driver reacts to changes to the contention of physical CPUs.
Can be set between 100 to 100000. i.e. 100ms to 100seconds. 100ms is kept as minimum to ensure few meaningful steal values accumulate even with HZ=100.

low_threshold:
lower threshold value in percentage * 100.
(Default: 200, i.e 2% steal is considered as low threshold) This determines what values should be considered as nil/no steal values. When steal governor see steal ratio is below or equal to this value, it will increase the preferred CPUs by 1 core. Having value as zero might cause oscillations

high_threshold:
higher threshold value in percentage * 100
(Default: 500, i.e 5% steal is considered as high threshold) This determines what values should be considered as high steal values. When steal governor sees steal ratio is higher than this value, it will reduce the preferred CPUs by 1 core.

module_param_cb methods are used to do the validation checks. This helps to ensure one configures sane values.
Since low and high are dependent, that check is done at module init.

Notes:
- Parameters values can't be changed at runtime. One has to unload the module and change it. Hence recommended to build it as module.
- Default values may not work well for all configurations. Tune it according to the system under test.

Documentation is available at: Documentation/driver-api/steal-governor.rst

---
 drivers/virt/steal_governor.c | 73 +++++++++++++++++++++++++++++++++-
 1 file changed, 71 insertions(+), 2 deletions(-)
drivers/virt/steal_governor.c +71 -2
index 2320cbf..27f53ea 100644
@@ -40,7 +40,11 @@ struct steal_governor {
 	struct delayed_work	work;
 };
 
-static struct steal_governor sg_ctx;
+static struct steal_governor sg_ctx = {
+	.interval_ms	=	1000,	/* 1 second */
+	.high_threshold =	500,	/* 5% */
+	.low_threshold	=	200,	/* 2% */
+};
 
 static void restore_preferred_to_active(void)
 {
@@ -51,6 +55,62 @@ static void restore_preferred_to_active(void)
 		set_cpu_preferred(cpu, true);
 }
 
+static int param_set_interval_ms(const char *val, const struct kernel_param *kp)
+{
+	unsigned int interval;
+	int ret;
+
+	ret = kstrtouint(val, 0, &interval);
+	if (ret)
+		return ret;
+
+	if (interval < 100 || interval > 100000) {
+		pr_err("interval_ms must be between 100 and 100000\n");
+		return -EINVAL;
+	}
+
+	return param_set_uint(val, kp);
+}
+
+static const struct kernel_param_ops interval_ms_ops = {
+	.set = param_set_interval_ms,
+	.get = param_get_uint,
+};
+
+module_param_cb(interval_ms, &interval_ms_ops, &sg_ctx.interval_ms, 0444);
+MODULE_PARM_DESC(interval_ms,
+		 "Sampling frequency in milliseconds. default: 1000");
+
+static int param_set_high_threshold(const char *val, const struct kernel_param *kp)
+{
+	unsigned int threshold;
+	int ret;
+
+	ret = kstrtouint(val, 0, &threshold);
+	if (ret)
+		return ret;
+
+	if (threshold >= 100 * 100) {
+		pr_err("high_threshold (%u) can't be more than 99.99%%\n", threshold);
+		return -EINVAL;
+	}
+
+	return param_set_uint(val, kp);
+}
+
+static const struct kernel_param_ops high_threshold_ops = {
+	.set = param_set_high_threshold,
+	.get = param_get_uint,
+};
+
+module_param_cb(high_threshold, &high_threshold_ops, &sg_ctx.high_threshold, 0444);
+MODULE_PARM_DESC(high_threshold,
+		 "High steal threshold. default: 500 i.e 5%. Must be > low_threshold");
+
+module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444);
+MODULE_PARM_DESC(low_threshold,
+		 "Low steal threshold. default: 200 i.e 2%. Must be < high_threshold");
+
 static int __init steal_governor_init(void)
 {
 #ifdef CONFIG_XEN
@@ -60,7 +120,16 @@ static int __init steal_governor_init(void)
 	}
 #endif
 
-	pr_info("enabled\n");
+	if (sg_ctx.low_threshold >= sg_ctx.high_threshold) {
+		pr_err("low_threshold (%u) must be less than high_threshold (%u)\n",
+		       sg_ctx.low_threshold, sg_ctx.high_threshold);
+		return -EINVAL;
+	}
+
+	sg_ctx.delay = msecs_to_jiffies(sg_ctx.interval_ms);
+	pr_info("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n",
+		sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
+
 	return 0;
 }
 
author maintainer
#44 lore raw

PATCH v14 12/13 virt/steal_governor: Implement steal_governor policy loop

In reply to Shrikanth Hegde

Schedule work at regular intervals to implement the steal_governor policy loop, which monitors steal time and takes action on the state of preferred CPUs. The interval is determined by the interval_ms parameter. schedule_delayed_work() is used since interval_ms is on the order of milliseconds and the work does not need to happen instantly.

Periodic policy loop essentially does:

- Gets the total/delta steal values and cpus to use steal_ratio.
- Calculate the steal_ratio as below.

       steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())

It is calculated this way to consider the fractional values of steal time. I.e 10 means 0.1% steal time. A few tricks such as divide by 10,000 are used to avoid possible overflow.
- If steal ratio is higher than high threshold, call the method to reduce the preferred CPUs.
- If steal ratio is lower or equal to low threshold, call the method to increase the preferred CPUs.
- If the steal ratio falls in between, no action is taken.
- Ensures design constraints always met.
1. At least one core/CPU must be there in preferred mask.
2. preferred CPUs is subset of active CPUs.
If not met, then restore preferred CPUs to active and stop requeue of the work. Driver is effectively non-functional after that.

Note that design checks are always performed. This helps avoid placing driver-specific design constraints inside the core CPU hotplug mechanism. User may offline specific set of CPUs that could leave the preferred mask as empty. With the design check performed always, driver gracefully shuts down upon detecting that edge case.

In order to help the above loop, a few helper functions have been added.
1. get_system_steal_time()
- steal governor takes global view of steal time instead of individual vCPU. Collect the steal values across the vCPUs of interest.
- Sum up steal time values across possible CPUs. This helps to keep it a monotonically increasing number and avoids spikes due to CPU hotplug.

2. decrease_preferred_cpus()
- Called when there is high steal time. It needs to decide which CPUs to mark as non-preferred.
- Get first housekeeping CPU and its core mask. Mark it as protected core. This helps to keep at least one core as preferred. (kernel ensures at least one housekeeping CPU stays active.)
- Find the last CPU outside of this protected core mask. i.e target CPU
- Based on that target CPU, get its sibling and mark them as non-preferred.

3. increase_preferred_cpus()
- Called when there is low steal time. It needs to decide which CPUs to mark as preferred and set that state.
- Get the first active non-preferred CPUs. This likely is the last set of CPUs being marked as non-preferred.
- get the siblings of that CPU and mark them as preferred.

4. get_system_cpus()
- informs how many CPUs needs to be considered for steal_ratio calculations.
- Since only active CPUs effectively contribute to steal time delta, returns number of active CPUs. This also helps to avoid dilution of thresholds in sparsely populated systems.

Notes:
1. Using core instead of individual CPUs performs better as SMT is quite common and some hypervisor such as powerVM does core scheduling.

2. This doesn't do any NUMA splicing to keep the code simpler and minimal overhead. Current code expects CPUs spread uniformly across NUMA nodes.

---
 drivers/virt/steal_governor.c | 149 ++++++++++++++++++++++++++++++++++
 1 file changed, 149 insertions(+)
drivers/virt/steal_governor.c +149 -0
index 27f53ea16498..6e31f9923dea 100644
@@ -13,13 +13,18 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/cleanup.h>
 #include <linux/cpuhplock.h>
 #include <linux/cpumask.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
+#include <linux/kernel_stat.h>
 #include <linux/kconfig.h>
 #include <linux/ktime.h>
+#include <linux/math64.h>
 #include <linux/module.h>
+#include <linux/sched/isolation.h>
+#include <linux/topology.h>
 #include <linux/types.h>
 #include <linux/workqueue.h>
 #ifdef CONFIG_XEN
@@ -111,6 +116,145 @@ module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444);
 MODULE_PARM_DESC(low_threshold,
 		 "Low steal threshold. default: 200 i.e 2%. Must be < high_threshold");
 
+/* Return collective steal time across system. */
+static u64 get_system_steal_time(void)
+{
+	return kcpustat_field_total(CPUTIME_STEAL, cpu_possible_mask);
+}
+
+/* Return number of CPUs to consider for steal ratio. */
+static unsigned int get_system_cpus(void)
+{
+	return num_active_cpus();
+}
+
+/*
+ * Called when the steal governor detects high physical CPU contention.
+ * It finds the last active core in the preferred mask and mark those
+ * CPUs as non-preferred.
+ *
+ * Must ensure:
+ * - at least one core is always kept as preferred
+ * - preferred is always subset of active.
+ */
+static void decrease_preferred_cpus(void)
+{
+	const struct cpumask *first_hk_core;
+	int target_cpu = nr_cpu_ids;
+	int cpu;
+
+	guard(cpus_read_lock)();
+	cpu = cpumask_first_and(housekeeping_cpumask(HK_TYPE_KERNEL_NOISE),
+				cpu_preferred_mask);
+	if (cpu >= nr_cpu_ids)
+		return;
+
+	/* Always leave first housekeeping core as preferred. */
+	first_hk_core = topology_sibling_cpumask(cpu);
+	cpu = cpumask_last(cpu_preferred_mask);
+	if (cpu >= nr_cpu_ids)
+		return;
+
+	/* Find the last CPU which doesn't belong to that first hk_core. */
+	if (!cpumask_test_cpu(cpu, first_hk_core)) {
+		target_cpu = cpu;
+	} else {
+		for_each_cpu_andnot(cpu, cpu_preferred_mask, first_hk_core)
+			target_cpu = cpu;
+	}
+
+	/* Only the first housekeeping core remains */
+	if (target_cpu >= nr_cpu_ids)
+		return;
+
+	for_each_cpu_and(cpu, topology_sibling_cpumask(target_cpu),
+			 cpu_preferred_mask)
+		set_cpu_preferred(cpu, false);
+}
+
+/*
+ * Called when the steal governor detects no/low physical CPU contention.
+ * It finds the first active core outside of preferred mask and mark
+ * those CPUs as preferred.
+ *
+ * Must ensure preferred is subset of active.
+ */
+static void increase_preferred_cpus(void)
+{
+	int first_cpu, cpu;
+
+	guard(cpus_read_lock)();
+	first_cpu = cpumask_first_andnot(cpu_active_mask, cpu_preferred_mask);
+
+	/* All CPUs are preferred. Nothing to increase further */
+	if (first_cpu >= nr_cpu_ids)
+		return;
+
+	for_each_cpu_and(cpu, topology_sibling_cpumask(first_cpu),
+			 cpu_active_mask)
+		set_cpu_preferred(cpu, true);
+}
+
+static bool preferred_cpus_valid(void)
+{
+	if (cpumask_empty(cpu_preferred_mask)) {
+		pr_err("empty preferred mask. stopping\n");
+		return false;
+	}
+
+	if (!cpumask_subset(cpu_preferred_mask, cpu_active_mask)) {
+		pr_err("preferred: %*pbl is not subset of active: %*pbl, stopping\n",
+		       cpumask_pr_args(cpu_preferred_mask),
+		       cpumask_pr_args(cpu_active_mask));
+		return false;
+	}
+
+	return true;
+}
+
+static void steal_governor_loop(struct work_struct *work)
+{
+	u64 curr_steal, delta_steal, delta_ns, steal_ratio;
+	ktime_t now;
+
+	now = ktime_get();
+	delta_ns = ktime_to_ns(ktime_sub(now, sg_ctx.time));
+
+	if (unlikely(delta_ns < NSEC_PER_MSEC)) {
+		pr_err_ratelimited("work scheduled too soon delta_ns: %llu\n", delta_ns);
+		goto requeue_work;
+	}
+
+	curr_steal = get_system_steal_time();
+	delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal : 0;
+	sg_ctx.steal = curr_steal;
+	sg_ctx.time = now;
+
+	/*
+	 * steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())
+	 * To avoid possible overflow, divide the denominator early.
+	 * Note minimum interval is 100ms.
+	 */
+	delta_ns = max_t(u64, div_u64(delta_ns * get_system_cpus(), 10000), 1);
+	steal_ratio = div64_u64(delta_steal, delta_ns);
+
+	if (steal_ratio > sg_ctx.high_threshold)
+		decrease_preferred_cpus();
+	else if (steal_ratio <= sg_ctx.low_threshold)
+		increase_preferred_cpus();
+	/*
+	 * else: steal ratio is within bounds. Still do design checks so that
+	 * module restores to active if CPU hotplug breaks those assumptions.
+	 */
+	if (!preferred_cpus_valid()) {
+		restore_preferred_to_active();
+		return;
+	}
+
+requeue_work:
+	schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
+}
+
 static int __init steal_governor_init(void)
 {
 #ifdef CONFIG_XEN
@@ -127,6 +271,10 @@ static int __init steal_governor_init(void)
 	}
 
 	sg_ctx.delay = msecs_to_jiffies(sg_ctx.interval_ms);
+	INIT_DELAYED_WORK(&sg_ctx.work, steal_governor_loop);
+	sg_ctx.steal = get_system_steal_time();
+	sg_ctx.time = ktime_get();
+	schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
 	pr_info("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n",
 		sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
 
@@ -135,6 +283,7 @@ static int __init steal_governor_init(void)
 
 static void __exit steal_governor_exit(void)
 {
+	disable_delayed_work_sync(&sg_ctx.work);
 	restore_preferred_to_active();
 	pr_info("disabled\n");
 }
Signature · 1 line
2.52.0
#45 lore raw

Re: [PATCH v14 12/13] virt/steal_governor: Implement steal_governor policy loop

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Missing bounds check on interval_ms allows infinite workqueue spinloop - [High] TOCTOU data race…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Missing bounds check on interval_ms allows infinite workqueue spinloop
- [High] TOCTOU data race in preferred_cpus_valid() causes erroneous driver shutdown
- [Medium] Integer overflow in steal_ratio denominator when VM is paused
--

commit 5f38399332407b64e0ce2da43681c27f1e1287d5
Author: Shrikanth Hegde <[email protected]>

virt/steal_governor: Implement steal_governor policy loop

Schedule work at regular intervals to implement the steal_governor policy loop, which monitors steal time and takes action on the state of preferred CPUs. The interval is determined by the interval_ms parameter.

Quoted 4 lines
drivers/virt/steal_governor.c +0 -0
index 27f53ea16498b..6e31f9923dea4 100644

[ ... ]

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 205

Quoted 8 lines
drivers/virt/steal_governor.c +8 -0
+static bool preferred_cpus_valid(void)
+{
+	if (cpumask_empty(cpu_preferred_mask)) {
+		pr_err("empty preferred mask. stopping\n");
+		return false;
+	}
+
+	if (!cpumask_subset(cpu_preferred_mask, cpu_active_mask)) {

[Severity: High]
Does this lockless read of cpu_preferred_mask and cpu_active_mask race with CPU hotplug?

Since CPU hotplug clears the preferred mask bit before the active mask bit, it seems cpumask_subset() can read a torn state where a word is read from the preferred mask before the bit is cleared, and the corresponding word is read from the active mask after the bit is cleared, returning false.

Quoted 10 lines
drivers/virt/steal_governor.c +10 -0
+		pr_err("preferred: %*pbl is not subset of active: %*pbl, stopping\n",
+		       cpumask_pr_args(cpu_preferred_mask),
+		       cpumask_pr_args(cpu_active_mask));
+		return false;
+	}
+
+	return true;
+}
+
+static void steal_governor_loop(struct work_struct *work)

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 225

Quoted 10 lines
drivers/virt/steal_governor.c +10 -0
+{
+	u64 curr_steal, delta_steal, delta_ns, steal_ratio;
+	ktime_t now;
+
+	now = ktime_get();
+	delta_ns = ktime_to_ns(ktime_sub(now, sg_ctx.time));
+
+	if (unlikely(delta_ns < NSEC_PER_MSEC)) {
+		pr_err_ratelimited("work scheduled too soon delta_ns: %llu\n", delta_ns);
+		goto requeue_work;

[Severity: High]
Can this create an infinite spinloop if interval_ms is set to 0?

If interval_ms is 0, the delay is 0 jiffies. When the work executes, it will take this branch because delta_ns < 1ms. It jumps directly to requeue_work without updating sg_ctx.time, which schedules it again with 0 delay. This will continuously spin and consume an entire CPU core.

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 238

Quoted 13 lines
drivers/virt/steal_governor.c +13 -0
+	}
+
+	curr_steal = get_system_steal_time();
+	delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal : 0;
+	sg_ctx.steal = curr_steal;
+	sg_ctx.time = now;
+
+	/*
+	 * steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())
+	 * To avoid possible overflow, divide the denominator early.
+	 * Note minimum interval is 100ms.
+	 */
+	delta_ns = max_t(u64, div_u64(delta_ns * get_system_cpus(), 10000), 1);

[Severity: Medium]
Will this silently overflow the 64-bit multiplication if the VM is paused for an extended period?

Since the multiplication of delta_ns * get_system_cpus() happens before the division, a large time delta (e.g., a large VM suspended for a long time) can overflow U64_MAX, leading to a wildly incorrect steal_ratio calculation.

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 249

Quoted 11 lines
drivers/virt/steal_governor.c +11 -0
+	steal_ratio = div64_u64(delta_steal, delta_ns);
+
+	if (steal_ratio > sg_ctx.high_threshold)
+		decrease_preferred_cpus();
+	else if (steal_ratio <= sg_ctx.low_threshold)
+		increase_preferred_cpus();
+	/*
+	 * else: steal ratio is within bounds. Still do design checks so that
+	 * module restores to active if CPU hotplug breaks those assumptions.
+	 */
+	if (!preferred_cpus_valid()) {

[Severity: High]
Does calling preferred_cpus_valid() here without cpus_read_lock() lead to a false teardown?

If a hotplug operation is happening concurrently, the torn state read in preferred_cpus_valid() will cause the governor to erroneously shut down the policy loop permanently.

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 255

Quoted 6 lines
drivers/virt/steal_governor.c +6 -0
+		restore_preferred_to_active();
+		return;
+	}
+
+requeue_work:
+	schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);

[Severity: High]
Does this immediately spin if sg_ctx.delay is 0, as mentioned above?

Quoted 4 lines
drivers/virt/steal_governor.c +2 -0
+}
+
 static int __init steal_governor_init(void)
 {

[ ... ]

Quoted 1 line

sg_ctx.delay = msecs_to_jiffies(sg_ctx.interval_ms);

[Severity: High]
Is it possible for sg_ctx.interval_ms to be 0 here?

The module parameter interval_ms doesn't appear to be bounds-checked before this point, meaning it could be set to 0 and trigger the infinite workqueue spinloop.

Quoted 7 lines
drivers/virt/steal_governor.c +4 -0
+	INIT_DELAYED_WORK(&sg_ctx.work, steal_governor_loop);
+	sg_ctx.steal = get_system_steal_time();
+	sg_ctx.time = ktime_get();
+	schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
 	pr_info("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n",
 		sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
author maintainer
#46 lore raw

Re: [PATCH v14 12/13] virt/steal_governor: Implement steal_governor policy loop

I don't see any real concerns so far. Some optimizations yes, but they can be deferred post the merge if they are necessary.

Quoted 33 lines · On 9/28/26 11:21 AM, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Missing bounds check on interval_ms allows infinite workqueue spinloop
- [High] TOCTOU data race in preferred_cpus_valid() causes erroneous driver shutdown
- [Medium] Integer overflow in steal_ratio denominator when VM is paused
--

commit 5f38399332407b64e0ce2da43681c27f1e1287d5
Author: Shrikanth Hegde <[email protected]>

virt/steal_governor: Implement steal_governor policy loop

Schedule work at regular intervals to implement the steal_governor policy loop, which monitors steal time and takes action on the state of preferred CPUs. The interval is determined by the interval_ms parameter.

Quoted 4 lines
drivers/virt/steal_governor.c +0 -0
index 27f53ea16498b..6e31f9923dea4 100644

[ ... ]

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 205

Quoted 8 lines
drivers/virt/steal_governor.c +8 -0
+static bool preferred_cpus_valid(void)
+{
+	if (cpumask_empty(cpu_preferred_mask)) {
+		pr_err("empty preferred mask. stopping\n");
+		return false;
+	}
+
+	if (!cpumask_subset(cpu_preferred_mask, cpu_active_mask)) {

[Severity: High]
Does this lockless read of cpu_preferred_mask and cpu_active_mask race with CPU hotplug?

Quoted 4 lines

Since CPU hotplug clears the preferred mask bit before the active mask bit, it seems cpumask_subset() can read a torn state where a word is read from the preferred mask before the bit is cleared, and the corresponding word is read from the active mask after the bit is cleared, returning false.

Update ordering is taken care. Not an issue is clear/set bit involves atomics.

Quoted 25 lines
Quoted 11 lines
drivers/virt/steal_governor.c +10 -0
+		pr_err("preferred: %*pbl is not subset of active: %*pbl, stopping\n",
+		       cpumask_pr_args(cpu_preferred_mask),
+		       cpumask_pr_args(cpu_active_mask));
+		return false;
+	}
+
+	return true;
+}
+
+static void steal_governor_loop(struct work_struct *work)

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 225

Quoted 10 lines
drivers/virt/steal_governor.c +10 -0
+{
+	u64 curr_steal, delta_steal, delta_ns, steal_ratio;
+	ktime_t now;
+
+	now = ktime_get();
+	delta_ns = ktime_to_ns(ktime_sub(now, sg_ctx.time));
+
+	if (unlikely(delta_ns < NSEC_PER_MSEC)) {
+		pr_err_ratelimited("work scheduled too soon delta_ns: %llu\n", delta_ns);
+		goto requeue_work;

[Severity: High]
Can this create an infinite spinloop if interval_ms is set to 0?

Quoted 5 lines

If interval_ms is 0, the delay is 0 jiffies. When the work executes, it will take this branch because delta_ns < 1ms. It jumps directly to requeue_work without updating sg_ctx.time, which schedules it again with 0 delay. This will continuously spin and consume an entire CPU core.

No. Interval can't be set to 0. Minimum is 100ms.

Quoted 18 lines

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 238

Quoted 13 lines
drivers/virt/steal_governor.c +13 -0
+	}
+
+	curr_steal = get_system_steal_time();
+	delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal : 0;
+	sg_ctx.steal = curr_steal;
+	sg_ctx.time = now;
+
+	/*
+	 * steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())
+	 * To avoid possible overflow, divide the denominator early.
+	 * Note minimum interval is 100ms.
+	 */
+	delta_ns = max_t(u64, div_u64(delta_ns * get_system_cpus(), 10000), 1);

[Severity: Medium]
Will this silently overflow the 64-bit multiplication if the VM is paused for an extended period?

Quoted 4 lines

Since the multiplication of delta_ns * get_system_cpus() happens before the division, a large time delta (e.g., a large VM suspended for a long time) can overflow U64_MAX, leading to a wildly incorrect steal_ratio calculation.

Not a reason to fret about. It is way too long duration.

Quoted 20 lines

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 249

Quoted 11 lines
drivers/virt/steal_governor.c +11 -0
+	steal_ratio = div64_u64(delta_steal, delta_ns);
+
+	if (steal_ratio > sg_ctx.high_threshold)
+		decrease_preferred_cpus();
+	else if (steal_ratio <= sg_ctx.low_threshold)
+		increase_preferred_cpus();
+	/*
+	 * else: steal ratio is within bounds. Still do design checks so that
+	 * module restores to active if CPU hotplug breaks those assumptions.
+	 */
+	if (!preferred_cpus_valid()) {

[Severity: High]
Does calling preferred_cpus_valid() here without cpus_read_lock() lead to a false teardown?

If a hotplug operation is happening concurrently, the torn state read in preferred_cpus_valid() will cause the governor to erroneously shut down the policy loop permanently.

Quoted 9 lines

drivers/virt/steal_governor.c module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444); · line 255

Quoted 6 lines
drivers/virt/steal_governor.c +6 -0
+		restore_preferred_to_active();
+		return;
+	}
+
+requeue_work:
+	schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);

[Severity: High]
Does this immediately spin if sg_ctx.delay is 0, as mentioned above?

As said above, interval cannot be less than 100ms.

Quoted 6 lines
Quoted 5 lines
drivers/virt/steal_governor.c +2 -0
+}
+
  static int __init steal_governor_init(void)
  {

[ ... ]

Quoted 8 lines
Quoted 1 line
  	sg_ctx.delay = msecs_to_jiffies(sg_ctx.interval_ms);

[Severity: High]
Is it possible for sg_ctx.interval_ms to be 0 here?

The module parameter interval_ms doesn't appear to be bounds-checked before this point, meaning it could be set to 0 and trigger the infinite workqueue spinloop.

As said above, interval cannot be less than 100ms.

Quoted 9 lines
Quoted 8 lines
drivers/virt/steal_governor.c +4 -0
+	INIT_DELAYED_WORK(&sg_ctx.work, steal_governor_loop);
+	sg_ctx.steal = get_system_steal_time();
+	sg_ctx.time = ktime_get();
+	schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
  	pr_info("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n",
  		sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
#47 lore raw

[tip: sched/core] virt/steal_governor: Implement steal_governor policy loop

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     27d47ebce4d6490ce093031a48cc2d81bf9997a3
Gitweb:        https://git.kernel.org/tip/27d47ebce4d6490ce093031a48cc2d81bf9997a3
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:27 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:43 +02:00

virt/steal_governor: Implement steal_governor policy loop

Schedule work at regular intervals to implement the steal_governor policy loop, which monitors steal time and takes action on the state of preferred CPUs. The interval is determined by the interval_ms parameter. schedule_delayed_work() is used since interval_ms is on the order of milliseconds and the work does not need to happen instantly.

Periodic policy loop essentially does:

- Gets the total/delta steal values and cpus to use steal_ratio.
- Calculate the steal_ratio as below.

       steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())

It is calculated this way to consider the fractional values of steal time. I.e 10 means 0.1% steal time. A few tricks such as divide by 10,000 are used to avoid possible overflow.
- If steal ratio is higher than high threshold, call the method to reduce the preferred CPUs.
- If steal ratio is lower or equal to low threshold, call the method to increase the preferred CPUs.
- If the steal ratio falls in between, no action is taken.
- Ensures design constraints always met.
1. At least one core/CPU must be there in preferred mask.
2. preferred CPUs is subset of active CPUs.
If not met, then restore preferred CPUs to active and stop requeue of the work. Driver is effectively non-functional after that.

Note that design checks are always performed. This helps avoid placing driver-specific design constraints inside the core CPU hotplug mechanism. User may offline specific set of CPUs that could leave the preferred mask as empty. With the design check performed always, driver gracefully shuts down upon detecting that edge case.

In order to help the above loop, a few helper functions have been added.
1. get_system_steal_time()
- steal governor takes global view of steal time instead of individual vCPU. Collect the steal values across the vCPUs of interest.
- Sum up steal time values across possible CPUs. This helps to keep it a monotonically increasing number and avoids spikes due to CPU hotplug.

2. decrease_preferred_cpus()
- Called when there is high steal time. It needs to decide which CPUs to mark as non-preferred.
- Get first housekeeping CPU and its core mask. Mark it as protected core. This helps to keep at least one core as preferred. (kernel ensures at least one housekeeping CPU stays active.)
- Find the last CPU outside of this protected core mask. i.e target CPU
- Based on that target CPU, get its sibling and mark them as non-preferred.

3. increase_preferred_cpus()
- Called when there is low steal time. It needs to decide which CPUs to mark as preferred and set that state.
- Get the first active non-preferred CPUs. This likely is the last set of CPUs being marked as non-preferred.
- get the siblings of that CPU and mark them as preferred.

4. get_system_cpus()
- informs how many CPUs needs to be considered for steal_ratio calculations.
- Since only active CPUs effectively contribute to steal time delta, returns number of active CPUs. This also helps to avoid dilution of thresholds in sparsely populated systems.

Notes:
1. Using core instead of individual CPUs performs better as SMT is quite common and some hypervisor such as powerVM does core scheduling.

2. This doesn't do any NUMA splicing to keep the code simpler and minimal overhead. Current code expects CPUs spread uniformly across NUMA nodes.

---
 drivers/virt/steal_governor.c | 149 +++++++++++++++++++++++++++++++++-
 1 file changed, 149 insertions(+)
drivers/virt/steal_governor.c +149 -0
index 27f53ea..6e31f99 100644
@@ -13,13 +13,18 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include <linux/cleanup.h>
 #include <linux/cpuhplock.h>
 #include <linux/cpumask.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
+#include <linux/kernel_stat.h>
 #include <linux/kconfig.h>
 #include <linux/ktime.h>
+#include <linux/math64.h>
 #include <linux/module.h>
+#include <linux/sched/isolation.h>
+#include <linux/topology.h>
 #include <linux/types.h>
 #include <linux/workqueue.h>
 #ifdef CONFIG_XEN
@@ -111,6 +116,145 @@ module_param_named(low_threshold, sg_ctx.low_threshold, uint, 0444);
 MODULE_PARM_DESC(low_threshold,
 		 "Low steal threshold. default: 200 i.e 2%. Must be < high_threshold");
 
+/* Return collective steal time across system. */
+static u64 get_system_steal_time(void)
+{
+	return kcpustat_field_total(CPUTIME_STEAL, cpu_possible_mask);
+}
+
+/* Return number of CPUs to consider for steal ratio. */
+static unsigned int get_system_cpus(void)
+{
+	return num_active_cpus();
+}
+
+/*
+ * Called when the steal governor detects high physical CPU contention.
+ * It finds the last active core in the preferred mask and mark those
+ * CPUs as non-preferred.
+ *
+ * Must ensure:
+ * - at least one core is always kept as preferred
+ * - preferred is always subset of active.
+ */
+static void decrease_preferred_cpus(void)
+{
+	const struct cpumask *first_hk_core;
+	int target_cpu = nr_cpu_ids;
+	int cpu;
+
+	guard(cpus_read_lock)();
+	cpu = cpumask_first_and(housekeeping_cpumask(HK_TYPE_KERNEL_NOISE),
+				cpu_preferred_mask);
+	if (cpu >= nr_cpu_ids)
+		return;
+
+	/* Always leave first housekeeping core as preferred. */
+	first_hk_core = topology_sibling_cpumask(cpu);
+	cpu = cpumask_last(cpu_preferred_mask);
+	if (cpu >= nr_cpu_ids)
+		return;
+
+	/* Find the last CPU which doesn't belong to that first hk_core. */
+	if (!cpumask_test_cpu(cpu, first_hk_core)) {
+		target_cpu = cpu;
+	} else {
+		for_each_cpu_andnot(cpu, cpu_preferred_mask, first_hk_core)
+			target_cpu = cpu;
+	}
+
+	/* Only the first housekeeping core remains */
+	if (target_cpu >= nr_cpu_ids)
+		return;
+
+	for_each_cpu_and(cpu, topology_sibling_cpumask(target_cpu),
+			 cpu_preferred_mask)
+		set_cpu_preferred(cpu, false);
+}
+
+/*
+ * Called when the steal governor detects no/low physical CPU contention.
+ * It finds the first active core outside of preferred mask and mark
+ * those CPUs as preferred.
+ *
+ * Must ensure preferred is subset of active.
+ */
+static void increase_preferred_cpus(void)
+{
+	int first_cpu, cpu;
+
+	guard(cpus_read_lock)();
+	first_cpu = cpumask_first_andnot(cpu_active_mask, cpu_preferred_mask);
+
+	/* All CPUs are preferred. Nothing to increase further */
+	if (first_cpu >= nr_cpu_ids)
+		return;
+
+	for_each_cpu_and(cpu, topology_sibling_cpumask(first_cpu),
+			 cpu_active_mask)
+		set_cpu_preferred(cpu, true);
+}
+
+static bool preferred_cpus_valid(void)
+{
+	if (cpumask_empty(cpu_preferred_mask)) {
+		pr_err("empty preferred mask. stopping\n");
+		return false;
+	}
+
+	if (!cpumask_subset(cpu_preferred_mask, cpu_active_mask)) {
+		pr_err("preferred: %*pbl is not subset of active: %*pbl, stopping\n",
+		       cpumask_pr_args(cpu_preferred_mask),
+		       cpumask_pr_args(cpu_active_mask));
+		return false;
+	}
+
+	return true;
+}
+
+static void steal_governor_loop(struct work_struct *work)
+{
+	u64 curr_steal, delta_steal, delta_ns, steal_ratio;
+	ktime_t now;
+
+	now = ktime_get();
+	delta_ns = ktime_to_ns(ktime_sub(now, sg_ctx.time));
+
+	if (unlikely(delta_ns < NSEC_PER_MSEC)) {
+		pr_err_ratelimited("work scheduled too soon delta_ns: %llu\n", delta_ns);
+		goto requeue_work;
+	}
+
+	curr_steal = get_system_steal_time();
+	delta_steal = curr_steal > sg_ctx.steal ? curr_steal - sg_ctx.steal : 0;
+	sg_ctx.steal = curr_steal;
+	sg_ctx.time = now;
+
+	/*
+	 * steal_ratio = (delta_steal * 100*100)/(delta_ns * num_cpus())
+	 * To avoid possible overflow, divide the denominator early.
+	 * Note minimum interval is 100ms.
+	 */
+	delta_ns = max_t(u64, div_u64(delta_ns * get_system_cpus(), 10000), 1);
+	steal_ratio = div64_u64(delta_steal, delta_ns);
+
+	if (steal_ratio > sg_ctx.high_threshold)
+		decrease_preferred_cpus();
+	else if (steal_ratio <= sg_ctx.low_threshold)
+		increase_preferred_cpus();
+	/*
+	 * else: steal ratio is within bounds. Still do design checks so that
+	 * module restores to active if CPU hotplug breaks those assumptions.
+	 */
+	if (!preferred_cpus_valid()) {
+		restore_preferred_to_active();
+		return;
+	}
+
+requeue_work:
+	schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
+}
+
 static int __init steal_governor_init(void)
 {
 #ifdef CONFIG_XEN
@@ -127,6 +271,10 @@ static int __init steal_governor_init(void)
 	}
 
 	sg_ctx.delay = msecs_to_jiffies(sg_ctx.interval_ms);
+	INIT_DELAYED_WORK(&sg_ctx.work, steal_governor_loop);
+	sg_ctx.steal = get_system_steal_time();
+	sg_ctx.time = ktime_get();
+	schedule_delayed_work(&sg_ctx.work, sg_ctx.delay);
 	pr_info("enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n",
 		sg_ctx.interval_ms, sg_ctx.high_threshold, sg_ctx.low_threshold);
 
@@ -135,6 +283,7 @@ static int __init steal_governor_init(void)
 
 static void __exit steal_governor_exit(void)
 {
+	disable_delayed_work_sync(&sg_ctx.work);
 	restore_preferred_to_active();
 	pr_info("disabled\n");
 }
author maintainer
#48 lore raw

PATCH v14 13/13 virt/steal_governor: Enable the driver

In reply to Shrikanth Hegde

Provide a config option to enable the steal_governor driver. Since the feature targets paravirtualized environments and requires SMP, enforce those dependencies. The driver selects CONFIG_PREFERRED_CPU for the core scheduler mechanisms to work.

It is recommended to build the driver as a module (m) instead of built-in (y) due to the following reasons:

- Module parameters are read-only after initialization. Building as a module allows updating these parameters by simply reloading the module. Default module parameters cannot work in all configurations.

- The driver can be completely disabled by unloading the module.

- This feature works best when all VMs operate in a cooperative manner. Requiring an explicit module load ensures intentional deployment across all VMs by the system administrator.

---
 drivers/virt/Kconfig  | 17 +++++++++++++++++
 drivers/virt/Makefile |  1 +
 2 files changed, 18 insertions(+)
drivers/virt/Kconfig +17 -0
index 52eb7e4ba71f..eeb84e578ddf 100644
@@ -41,6 +41,23 @@ config FSL_HV_MANAGER
           4) A kernel interface for receiving callbacks when a managed
 	     partition shuts down.
 
+config STEAL_GOVERNOR
+	tristate "Dynamic vCPU management based on steal time"
+	depends on PARAVIRT && SMP
+	select PREFERRED_CPU
+	default m
+	help
+	  This driver helps to reduce the steal time in paravirtualized
+	  environments, thereby reducing vCPU preemption costs.
+
+	  By default preferred CPUs will be same as active CPUs. Depending
+	  on the steal time when steal_governor driver is enabled,
+	  preferred CPUs could become subset of active CPUs.
+	  More details are at: Documentation/driver-api/steal-governor.rst
+
+	  It is recommended to build it as module and load the module
+	  to enable it.
+
 source "drivers/virt/vboxguest/Kconfig"
 
 source "drivers/virt/nitro_enclaves/Kconfig"
drivers/virt/Makefile +1 -0
index f29901bd7820..05fb075ef5b8 100644
@@ -5,6 +5,7 @@
 
 obj-$(CONFIG_FSL_HV_MANAGER)	+= fsl_hypervisor.o
 obj-$(CONFIG_VMGENID)		+= vmgenid.o
+obj-$(CONFIG_STEAL_GOVERNOR)	+= steal_governor.o
 obj-y				+= vboxguest/
 
 obj-$(CONFIG_NITRO_ENCLAVES)	+= nitro_enclaves/
Signature · 1 line
2.52.0
#49 lore raw

Re: [PATCH v14 13/13] virt/steal_governor: Enable the driver

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 12 lines

Provide a config option to enable the steal_governor driver. Since the feature targets paravirtualized environments and requires SMP, enforce those dependencies. The driver selects CONFIG_PREFERRED_CPU for the core scheduler mechanisms to work.

It is recommended to build the driver as a module (m) instead of built-in (y) due to the following reasons:

- Module parameters are read-only after initialization. Building as a module allows updating these parameters by simply reloading the module. Default module parameters cannot work in all configurations.

Quoted 8 lines

- The driver can be completely disabled by unloading the module.

- This feature works best when all VMs operate in a cooperative manner. Requiring an explicit module load ensures intentional deployment across all VMs by the system administrator.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
#50 lore raw

[tip: sched/core] virt/steal_governor: Enable the driver

In reply to Shrikanth Hegde
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     1fb28c664a19df8d45a6afa04d28d102b04ea680
Gitweb:        https://git.kernel.org/tip/1fb28c664a19df8d45a6afa04d28d102b04ea680
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Mon, 28 Sep 2026 11:07:28 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 28 Sep 2026 08:44:44 +02:00

virt/steal_governor: Enable the driver

Provide a config option to enable the steal_governor driver. Since the feature targets paravirtualized environments and requires SMP, enforce those dependencies. The driver selects CONFIG_PREFERRED_CPU for the core scheduler mechanisms to work.

It is recommended to build the driver as a module (m) instead of built-in (y) due to the following reasons:

- Module parameters are read-only after initialization. Building as a module allows updating these parameters by simply reloading the module. Default module parameters cannot work in all configurations.

- The driver can be completely disabled by unloading the module.

- This feature works best when all VMs operate in a cooperative manner. Requiring an explicit module load ensures intentional deployment across all VMs by the system administrator.

---
 drivers/virt/Kconfig  | 17 +++++++++++++++++
 drivers/virt/Makefile |  1 +
 2 files changed, 18 insertions(+)
drivers/virt/Kconfig +17 -0
index 52eb7e4..eeb84e5 100644
@@ -41,6 +41,23 @@ config FSL_HV_MANAGER
           4) A kernel interface for receiving callbacks when a managed
 	     partition shuts down.
 
+config STEAL_GOVERNOR
+	tristate "Dynamic vCPU management based on steal time"
+	depends on PARAVIRT && SMP
+	select PREFERRED_CPU
+	default m
+	help
+	  This driver helps to reduce the steal time in paravirtualized
+	  environments, thereby reducing vCPU preemption costs.
+
+	  By default preferred CPUs will be same as active CPUs. Depending
+	  on the steal time when steal_governor driver is enabled,
+	  preferred CPUs could become subset of active CPUs.
+	  More details are at: Documentation/driver-api/steal-governor.rst
+
+	  It is recommended to build it as module and load the module
+	  to enable it.
+
 source "drivers/virt/vboxguest/Kconfig"
 
 source "drivers/virt/nitro_enclaves/Kconfig"
drivers/virt/Makefile +1 -0
index f29901b..05fb075 100644
@@ -5,6 +5,7 @@
 
 obj-$(CONFIG_FSL_HV_MANAGER)	+= fsl_hypervisor.o
 obj-$(CONFIG_VMGENID)		+= vmgenid.o
+obj-$(CONFIG_STEAL_GOVERNOR)	+= steal_governor.o
 obj-y				+= vboxguest/
 
 obj-$(CONFIG_NITRO_ENCLAVES)	+= nitro_enclaves/