Expose writable interface for preferred CPUs
Subsystems (2)
cover Expose writable interface for preferred CPUs
On top of tip sched/core + [1].
The preferred CPU mask currently lets the steal governor reduce vCPU demand when physical CPUs are contended.
This series adds a writable sysfs interface, allowing userspace governors such as thermald to encourage CPU idling in response to external pressure.
Patches #1 and #2 are partial resends of [2]. Patch #3 adds a thermald example.
The complete series is available at [3].
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://github.com/norov/linux/tree/sysfs-preferred-cpumask
Request feedback on preferred CPUs as a thermal-control mechanism and its relationship to existing cooling methods.
Yury Norov (3):
cpu: add sysfs control for preferred CPUS
sched: decouple preferred CPU policies from steal-time accounting
docs: sched: describe thermal control with preferred CPUs
.../ABI/testing/sysfs-devices-system-cpu | 14 ++- .../sysfs-devices-system-cpu-preferred | 42 +++++++++ Documentation/scheduler/index.rst | 1 + .../scheduler/preferred-cpu-thermal.rst | 94 +++++++++++++++++++ Documentation/scheduler/sched-paravirt.rst | 20 +++- drivers/base/cpu.c | 76 +++++++++++++++ kernel/Kconfig.preempt | 18 +++- 7 files changed, 251 insertions(+), 14 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-devices-system-cpu-preferred create mode 100644 Documentation/scheduler/preferred-cpu-thermal.rst
Signature · 1 line
2.53.0
RFC PATCH 1/3 cpu: add sysfs control for preferred CPUS
Preferred cpumask is a powerful instrument to hint the fair scheduler on tasks placement under certain conditions like vCPU overcommitment. It may be even more powerful if the final decision on which vCPUs are preferred is delegated to userspace.
Considering two VMs running training and inference AI workloads, the inference is more latency critical. In case of contention, the inference VM should keep more preferred vCPUs for better UX.
The in-kernel steal governor is the reasonable default approach that works good for cooperative VMs running similar workloads. In case of non-symmetrical VMs, optimal vCPUs throttling may require sharing more information between VMs, or at least knowing their role. This may be naturally addressed on userspace level.
Preferred cpumask has no write interface other than via a loadable module. Modules are disabled under certain configurations or can't be loaded due to system policies; or impractical if the preferred CPU state is supposed to be controlled by an existing user level software.
Add the write access via sysfs interface, similar to CPU hotplug. The write is not available if the preferred cpumask is claimed by a kernel module. Kernel ownership does not prevent reads.
Sysfs interface shares the kernel modules limitations: making non-active CPUs preferred and making all active CPUs non-preferred is not allowed.
Write access to the preferred cpumask may also be useful for the system debugging, benchmarking or CPU isolation without changing tasks affinity.
--- .../ABI/testing/sysfs-devices-system-cpu | 3 +- .../sysfs-devices-system-cpu-preferred | 42 ++++++++++ Documentation/scheduler/sched-paravirt.rst | 20 ++++- drivers/base/cpu.c | 76 +++++++++++++++++++ kernel/Kconfig.preempt | 16 +++- 5 files changed, 151 insertions(+), 6 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-devices-system-cpu-preferred
Documentation/ABI/testing/sysfs-devices-system-cpu +2 -1
@@ -829,4 +829,5 @@ Description: CPUs outside of this list may lead to performance degradations due to underlying physical CPU contention. - See Documentation/scheduler/sched-paravirt.rst for more details. + See Documentation/ABI/testing/sysfs-devices-system-cpu-preferred + and Documentation/scheduler/sched-paravirt.rst for more details.
Documentation/ABI/testing/sysfs-devices-system-cpu-preferred new file +42 -0
42 lines not on this page · raw message
Documentation/scheduler/sched-paravirt.rst +16 -4
@@ -42,13 +42,25 @@ This is achieved by: ``/sys/devices/system/cpu/preferred`` prints the current ``cpu_preferred_mask`` in cpulist format. +Userspace governors may control preferred cpumask through the sysfs +interface similar to CPU hotplug:: + + echo 0 > /sys/devices/system/cpu/cpu1/preferred + echo 1 > /sys/devices/system/cpu/cpu1/preferred + +If ``cpu_preferred_mask`` is claimed by a kernel module, an attempt to +write would return EBUSY. Disabling the last preferred CPU or enabling +non-active CPU in the preferred mask is not allowed, and EINVAL would +be returned. + Notes: 1. This feature is available under ``CONFIG_PREFERRED_CPU``. Driver which makes decisions should enable it. For example, steal_governor driver (``CONFIG_STEAL_GOVERNOR``). On enabling the driver, CPU preferred state - can change based on steal time. Without the driver, preferred CPUs is - same as active CPUs. Only one governor may be enabled at a time. + can change based on steal time. Preferred CPUs mirror active cpumask unless + modified by governor or via sysfs. Only one governor may be enabled at a + time. 2. This feature works for the FAIR class only.
@@ -56,8 +68,8 @@ Notes: to run based on its affinity. But no load balancing happens if it is affined only on non-preferred CPUs. -4. Decision to change the preferred CPU state is driven by the kernel. - Hence it shouldn't break user affinities. One of the main reasons why +4. Decision to change the preferred CPU state is driven by the governor + and does not break user affinities. One of the main reasons why CPU hotplug or Isolated cpuset partitions was not a solution. 5. This feature works best only when all the Guest VMs enable the feature as
drivers/base/cpu.c +76 -0
100 lines not on this page · raw message
kernel/Kconfig.preempt +15 -1
@@ -196,5 +196,19 @@ config SCHED_CLASS_EXT https://github.com/sched-ext/scx config PREFERRED_CPU - bool + bool "Preferred CPU scheduling" depends on SMP && HAVE_CPUTIME_STEAL + help + Hint the fair scheduler to favor a subset of active CPUs. + Non-preferred CPUs remain online and can still run tasks + if their affinity requires it. + + Preferred CPUs can be selected through the per-CPU preferred + sysfs attribute or managed by a kernel module, like steal + governor, to reduce CPU contention in virtual machines. + + See Documentation/scheduler/sched-paravirt.rst and + Documentation/ABI/testing/sysfs-devices-system-cpu-preferred + for details. + + If unsure, say N.
Signature · 1 line
2.53.0
RFC PATCH 2/3 sched: decouple preferred CPU policies from steal-time accounting
In reply to Yury NorovScheduling on preferred subset of active CPUs is useful beyond the paravirtualization scope.
For example, thermald can trim preferred list of CPUs as temperature rises and restore them on cooling, allowing fair tasks to concentrate on fewer CPUs without changing their affinity or taking CPUs offline.
--- Documentation/ABI/testing/sysfs-devices-system-cpu | 11 ++++------- kernel/Kconfig.preempt | 2 +- 2 files changed, 5 insertions(+), 8 deletions(-)
Documentation/ABI/testing/sysfs-devices-system-cpu +4 -7
@@ -821,13 +821,10 @@ What: /sys/devices/system/cpu/preferred Date: Sep 2026 Contact: Linux kernel mailing list <[email protected]> Description: - (RO) the list of preferred CPUs applicable in - paravirtualized environments. - - The steal governor driver dynamically adjusts this mask - based on observed steal time. Scheduling tasks on - CPUs outside of this list may lead to performance - degradations due to underlying physical CPU contention. + (RO) the list of CPUs preferred for fair tasks placement. + Fair tasks whose allowed CPUs contain no preferred CPU can + still run on active non-preferred CPUs. Other scheduling + classes are unaffected. See Documentation/ABI/testing/sysfs-devices-system-cpu-preferred and Documentation/scheduler/sched-paravirt.rst for more details.
kernel/Kconfig.preempt +1 -1
@@ -197,7 +197,7 @@ config SCHED_CLASS_EXT config PREFERRED_CPU bool "Preferred CPU scheduling" - depends on SMP && HAVE_CPUTIME_STEAL + depends on SMP help Hint the fair scheduler to favor a subset of active CPUs. Non-preferred CPUs remain online and can still run tasks
Signature · 1 line
2.53.0
RFC PATCH 3/3 docs: sched: describe thermal control with preferred CPUs
In reply to Yury NorovDocument a thermald policy that keeps CPU0 preferred, marks CPU1 non-preferred at 80 C, and restores its preferred status below 75 C.
Some Qualcomm-based Android devices, including the Nexus 6P, have used temperature-driven CPU offlining. Preferred CPU selection offers a weaker alternative that preserves task affinities and avoids CPU hotplug transitions. If pinned tasks or tasks outside the fair scheduling class prevent a CPU from becoming idle, a userspace thermal policy can use stronger controls, such as CPU offlining.
Tested with thermald 2.5.11 in an eight-vCPU virtme/KVM guest on an Intel Core Ultra 7 268V. Each vCPU is pinned to a separate host CPU. Thermald uses the relayed host package temperature to control preferred CPUs, with staggered trips at 46-58 C and 2 C hysteresis. CPU0 remains preferred throughout.
No host thermal throttling occurred during the test. Guest events, with sampled package temperatures and preferred CPU masks are logged as follows.
Elapsed Event Package temp Preferred CPUs
0s Start eight busy loops 42 C 0-7
1s CPU1-6 non-preferred 57 C 0,7
2s CPU6 preferred 52 C 0,6-7
13s CPU6 non-preferred 56 C 0,7
30s CPU7 non-preferred 58 C 0
90s Stop workload 56 C 0
90s CPU4-7 preferred 49 C 0,4-7
93s CPU3 preferred 47 C 0,3-7
102s CPU2 preferred 45 C 0,2-7
106s CPU2-3 non-preferred 50 C 0,4-7
107s CPU2-3 preferred 45 C 0,2-7--- Documentation/scheduler/index.rst | 1 + .../scheduler/preferred-cpu-thermal.rst | 94 +++++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 Documentation/scheduler/preferred-cpu-thermal.rst
Documentation/scheduler/index.rst +1 -0
@@ -24,6 +24,7 @@ Scheduler sched-ext sched-debug sched-paravirt + preferred-cpu-thermal sched-preemption text_files
Documentation/scheduler/preferred-cpu-thermal.rst new file +94 -0
94 lines not on this page · raw message
Signature · 1 line
2.53.0