Skip to contentSkip to conversation

Expose writable interface for preferred CPUs

RFC PATCH New Yury Norov 1 participant 4 messages Nov 17

Subsystems (2)
lore
Share & tools
lore

Yury Norov

author
#0 lore raw

cover Expose writable interface for preferred CPUs

On top of tip sched/core + [1].

The preferred CPU mask currently lets the steal governor reduce vCPU demand when physical CPUs are contended.

This series adds a writable sysfs interface, allowing userspace governors such as thermald to encourage CPU idling in response to external pressure.

Patches #1 and #2 are partial resends of [2]. Patch #3 adds a thermald example.

The complete series is available at [3].

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://github.com/norov/linux/tree/sysfs-preferred-cpumask

Request feedback on preferred CPUs as a thermal-control mechanism and its relationship to existing cooling methods.

Yury Norov (3):
cpu: add sysfs control for preferred CPUS
sched: decouple preferred CPU policies from steal-time accounting
docs: sched: describe thermal control with preferred CPUs

 .../ABI/testing/sysfs-devices-system-cpu      | 14 ++-
 .../sysfs-devices-system-cpu-preferred        | 42 +++++++++
 Documentation/scheduler/index.rst             |  1 +
 .../scheduler/preferred-cpu-thermal.rst       | 94 +++++++++++++++++++
 Documentation/scheduler/sched-paravirt.rst    | 20 +++-
 drivers/base/cpu.c                            | 76 +++++++++++++++
 kernel/Kconfig.preempt                        | 18 +++-
 7 files changed, 251 insertions(+), 14 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-system-cpu-preferred
 create mode 100644 Documentation/scheduler/preferred-cpu-thermal.rst
Signature · 1 line
2.53.0
author
#1 lore raw

RFC PATCH 1/3 cpu: add sysfs control for preferred CPUS

Preferred cpumask is a powerful instrument to hint the fair scheduler on tasks placement under certain conditions like vCPU overcommitment. It may be even more powerful if the final decision on which vCPUs are preferred is delegated to userspace.

Considering two VMs running training and inference AI workloads, the inference is more latency critical. In case of contention, the inference VM should keep more preferred vCPUs for better UX.

The in-kernel steal governor is the reasonable default approach that works good for cooperative VMs running similar workloads. In case of non-symmetrical VMs, optimal vCPUs throttling may require sharing more information between VMs, or at least knowing their role. This may be naturally addressed on userspace level.

Preferred cpumask has no write interface other than via a loadable module. Modules are disabled under certain configurations or can't be loaded due to system policies; or impractical if the preferred CPU state is supposed to be controlled by an existing user level software.

Add the write access via sysfs interface, similar to CPU hotplug. The write is not available if the preferred cpumask is claimed by a kernel module. Kernel ownership does not prevent reads.

Sysfs interface shares the kernel modules limitations: making non-active CPUs preferred and making all active CPUs non-preferred is not allowed.

Write access to the preferred cpumask may also be useful for the system debugging, benchmarking or CPU isolation without changing tasks affinity.

---
 .../ABI/testing/sysfs-devices-system-cpu      |  3 +-
 .../sysfs-devices-system-cpu-preferred        | 42 ++++++++++
 Documentation/scheduler/sched-paravirt.rst    | 20 ++++-
 drivers/base/cpu.c                            | 76 +++++++++++++++++++
 kernel/Kconfig.preempt                        | 16 +++-
 5 files changed, 151 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-system-cpu-preferred
Documentation/ABI/testing/sysfs-devices-system-cpu +2 -1
index 545687ebadcc..3bf89c7c4b91 100644
@@ -829,4 +829,5 @@ Description:
 		CPUs outside of this list may lead to performance
 		degradations due to underlying physical CPU contention.
 
-		See Documentation/scheduler/sched-paravirt.rst for more details.
+		See Documentation/ABI/testing/sysfs-devices-system-cpu-preferred
+		and  Documentation/scheduler/sched-paravirt.rst for more details.
Documentation/ABI/testing/sysfs-devices-system-cpu-preferred new file +42 -0

42 lines not on this page · raw message

Documentation/scheduler/sched-paravirt.rst +16 -4
index d8f446c3ab6e..4cea2bf7c4f4 100644
@@ -42,13 +42,25 @@ This is achieved by:
 ``/sys/devices/system/cpu/preferred`` prints the current ``cpu_preferred_mask``
 in cpulist format.
 
+Userspace governors may control preferred cpumask through the sysfs
+interface similar to CPU hotplug::
+
+    echo 0 > /sys/devices/system/cpu/cpu1/preferred
+    echo 1 > /sys/devices/system/cpu/cpu1/preferred
+
+If ``cpu_preferred_mask`` is claimed by a kernel module, an attempt to
+write would return EBUSY. Disabling the last preferred CPU or enabling
+non-active CPU in the preferred mask is not allowed, and EINVAL would
+be returned.
+
 Notes:
 
 1. This feature is available under ``CONFIG_PREFERRED_CPU``. Driver which
    makes decisions should enable it. For example, steal_governor driver
    (``CONFIG_STEAL_GOVERNOR``). On enabling the driver, CPU preferred state
-   can change based on steal time. Without the driver, preferred CPUs is
-   same as active CPUs. Only one governor may be enabled at a time.
+   can change based on steal time. Preferred CPUs mirror active cpumask unless
+   modified by governor or via sysfs. Only one governor may be enabled at a
+   time.
 
 2. This feature works for the FAIR class only.
 
@@ -56,8 +68,8 @@ Notes:
    to run based on its affinity. But no load balancing happens if it is affined
    only on non-preferred CPUs.
 
-4. Decision to change the preferred CPU state is driven by the kernel.
-   Hence it shouldn't break user affinities. One of the main reasons why
+4. Decision to change the preferred CPU state is driven by the governor
+   and does not break user affinities. One of the main reasons why
    CPU hotplug or Isolated cpuset partitions was not a solution.
 
 5. This feature works best only when all the Guest VMs enable the feature as
drivers/base/cpu.c +76 -0

100 lines not on this page · raw message

kernel/Kconfig.preempt +15 -1
index ffd2cb2a4d69..0cbc697202dd 100644
@@ -196,5 +196,19 @@ config SCHED_CLASS_EXT
 	    https://github.com/sched-ext/scx
 
 config PREFERRED_CPU
-	bool
+	bool "Preferred CPU scheduling"
 	depends on SMP && HAVE_CPUTIME_STEAL
+	help
+	  Hint the fair scheduler to favor a subset of active CPUs.
+	  Non-preferred CPUs remain online and can still run tasks
+	  if their affinity requires it.
+
+	  Preferred CPUs can be selected through the per-CPU preferred
+	  sysfs attribute or managed by a kernel module, like steal
+	  governor, to reduce CPU contention in virtual machines.
+
+	  See Documentation/scheduler/sched-paravirt.rst and
+	  Documentation/ABI/testing/sysfs-devices-system-cpu-preferred
+	  for details.
+
+	  If unsure, say N.
Signature · 1 line
2.53.0
author
#2 lore raw

RFC PATCH 2/3 sched: decouple preferred CPU policies from steal-time accounting

In reply to Yury Norov

Scheduling on preferred subset of active CPUs is useful beyond the paravirtualization scope.

For example, thermald can trim preferred list of CPUs as temperature rises and restore them on cooling, allowing fair tasks to concentrate on fewer CPUs without changing their affinity or taking CPUs offline.

---
 Documentation/ABI/testing/sysfs-devices-system-cpu | 11 ++++-------
 kernel/Kconfig.preempt                             |  2 +-
 2 files changed, 5 insertions(+), 8 deletions(-)
Documentation/ABI/testing/sysfs-devices-system-cpu +4 -7
index 3bf89c7c4b91..828c5173c3b4 100644
@@ -821,13 +821,10 @@ What:		/sys/devices/system/cpu/preferred
 Date:		Sep 2026
 Contact:	Linux kernel mailing list <[email protected]>
 Description:
-		(RO) the list of preferred CPUs applicable in
-		paravirtualized environments.
-
-		The steal governor driver dynamically adjusts this mask
-		based on observed steal time. Scheduling tasks on
-		CPUs outside of this list may lead to performance
-		degradations due to underlying physical CPU contention.
+		(RO) the list of CPUs preferred for fair tasks placement.
+		Fair tasks whose allowed CPUs contain no preferred CPU can
+		still run on active non-preferred CPUs. Other scheduling
+		classes are unaffected.
 
 		See Documentation/ABI/testing/sysfs-devices-system-cpu-preferred
 		and  Documentation/scheduler/sched-paravirt.rst for more details.
kernel/Kconfig.preempt +1 -1
index 0cbc697202dd..9736c6e291d8 100644
@@ -197,7 +197,7 @@ config SCHED_CLASS_EXT
 
 config PREFERRED_CPU
 	bool "Preferred CPU scheduling"
-	depends on SMP && HAVE_CPUTIME_STEAL
+	depends on SMP
 	help
 	  Hint the fair scheduler to favor a subset of active CPUs.
 	  Non-preferred CPUs remain online and can still run tasks
Signature · 1 line
2.53.0
author
#3 lore raw

RFC PATCH 3/3 docs: sched: describe thermal control with preferred CPUs

In reply to Yury Norov

Document a thermald policy that keeps CPU0 preferred, marks CPU1 non-preferred at 80 C, and restores its preferred status below 75 C.

Some Qualcomm-based Android devices, including the Nexus 6P, have used temperature-driven CPU offlining. Preferred CPU selection offers a weaker alternative that preserves task affinities and avoids CPU hotplug transitions. If pinned tasks or tasks outside the fair scheduling class prevent a CPU from becoming idle, a userspace thermal policy can use stronger controls, such as CPU offlining.

Tested with thermald 2.5.11 in an eight-vCPU virtme/KVM guest on an Intel Core Ultra 7 268V. Each vCPU is pinned to a separate host CPU. Thermald uses the relayed host package temperature to control preferred CPUs, with staggered trips at 46-58 C and 2 C hysteresis. CPU0 remains preferred throughout.

No host thermal throttling occurred during the test. Guest events, with sampled package temperatures and preferred CPU masks are logged as follows.

Elapsed  Event                      Package temp  Preferred CPUs
     0s  Start eight busy loops             42 C  0-7
     1s  CPU1-6 non-preferred               57 C  0,7
     2s  CPU6 preferred                     52 C  0,6-7
    13s  CPU6 non-preferred                 56 C  0,7
    30s  CPU7 non-preferred                 58 C  0
    90s  Stop workload                      56 C  0
    90s  CPU4-7 preferred                   49 C  0,4-7
    93s  CPU3 preferred                     47 C  0,3-7
   102s  CPU2 preferred                     45 C  0,2-7
   106s  CPU2-3 non-preferred               50 C  0,4-7
   107s  CPU2-3 preferred                   45 C  0,2-7
---
 Documentation/scheduler/index.rst             |  1 +
 .../scheduler/preferred-cpu-thermal.rst       | 94 +++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100644 Documentation/scheduler/preferred-cpu-thermal.rst
Documentation/scheduler/index.rst +1 -0
index 791178ac8bec..25d1590d8217 100644
@@ -24,6 +24,7 @@ Scheduler
     sched-ext
     sched-debug
     sched-paravirt
+    preferred-cpu-thermal
     sched-preemption
 
     text_files
Documentation/scheduler/preferred-cpu-thermal.rst new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
2.53.0