sched: Introduce cpu_preferred_mask and steal-driven vCPU backoff
Subsystems (5)
v5 cover sched: Introduce cpu_preferred_mask and steal-driven vCPU backoff
Very briefly,
- Maintain set of CPUs which can be used by workload. It is denoted as cpu_preferred_mask
- Periodically compute the steal time. If steal time is high/low based on the thresholds, either reduce/increase the preferred CPUs. This is handled in a new driver called steal_monitor
- If a CPU is marked as non-preferred, push the task running on it if possible.
- Use this CPU state in wakeup and load balance to ensure tasks run within preferred CPUs.
For more details on idea, problem statement and performance numbers, please refer to cover-letter of v2[2] and OSPM talk[1].
*** Please review and provide your feedback!! ***
[1]:https://youtu.be/adxUKFPlOp0
[2] v2: https://lore.kernel.org/all/[email protected]/#t
[3] v4: https://lore.kernel.org/all/[email protected]/#t
Thank you very much for feedback so far. This has helped the code to evolve towards a clear abstraction layers and get simplified. (Hopefully). Apologies in advance if I have missed any comment.
base commit:
tip/sched/core at c095741713d1 ("sched/fair: Fix newidle vs core-sched")
v4->v5:
- Move the computation of steal time and decide on preferred CPU state to a driver. Drop those changes in core scheduler. (Yury Norov, K Prateek Nayak)
- A new driver called steal_monitor is added in drivers/virt/ (K Prateek Nayak) (Please let me know if there is a better place for it. I can move it there)
- New driver does periodic computation of steal time and increments/decrements the preferred CPUs.
- Debug knobs can be changed via module parameters. (Yury Norov)
- Default implementation are weak symbols. Archs may override by providing strong symbols in new respective arch specific file.
- Everything is centered around CONFIG_PREFERRED_CPU. No new config for new driver. Driver gets added to kernel, but not loaded by default.
- Load the driver to enable steal_monitor functionality. Unload to remove the same.
- Make CONFIG_PREFERRED_CPU depend on PARAVIRT && SMP (Yury Norov)
- move set_cpu_preferred to a macro. (Yury Norov)
on CONFIG_PREFERRED_CPU=n it will just act on active CPUs in that case. It shouldn't alter any functionality.
- Do a simple encoding for has_preferred_cpu_state, which aims to avoid repeated cpumask_interest in is_cpu_allowed.
(Please let me know if new variable based approach to is_cpu_allowed should be done instead).
- Move select_fallback_rq above the rq_lock. (sashiko)
- Few documentation nitpicks (Randy Dunlap, sashiko)
- Avoid any decision for is_cpu_allowed for other classes (sashiko)
- Don't pull the load towards a non-preferred CPUs in idle and new idle balanced. (Inferred when seeing sashiko comments)
- Fix leaking of task_struct in push_work_done (K Prateek Nayak)
- Module parameters aren't checked for sane values. One should know what they are writing to it. If one writes 0 for interval_ms,
then it gets set to default value again to avoid workqueue lockup.
- Added a few design construct related checks in the periodic work
to ensure any future arch specific implementations follow it.
1. preferred is subset of active.
2. preferred cannot be empty.
- Added Documentation of steal_monitor in Documentation/driver-api/ (Let me know if there is better place for it)
performance numbers are expected to be same or slightly better than v2. With driver, one major overhead in sched_tick has been removed. i.e finding the first housekeeping CPU which was O(N).
Apologies in advance if there is any critical information is missing regarding new driver such as policy, documentation or missing implementation. Please let me know, and I can make those changes. I have ensured checkpatch --strict is happy.
Also, I think there should be a MAINTAINERS file entry for new driver. I don't see a drivers/virt/* entry.
Either as a new entry for driver or a few file in SCHEDULER entry. Let me know if/what I should add it. I am bit cautious about such change. I am willing to maintain this driver, other than that I don't know what else i going to be necessary for it. I don't have any maintainer experience either :)
PS: Sorry for the long CC list. Please unicast it to me if you want to be dropped for the CC list.
Shrikanth Hegde (24):
sched/debug: Remove unused schedstats
sched/docs: Document cpu_preferred_mask and Preferred CPU concept
kconfig: Provide PREFERRED_CPU option
cpumask: Introduce cpu_preferred_mask
sysfs: Add preferred CPU file
sched/core: allow only preferred CPUs in is_cpu_allowed sched/fair: Select preferred CPU at wakeup when possible sched/fair: load balance only among preferred CPUs sched/fair: Pull the load on preferred CPU
sched/core: Keep tick on non-preferred CPUs until tasks are out sched/core: Push current task from non preferred CPU sched/debug: Add migration stats due to non preferred CPUs virt/steal_monitor: Add documentation
virt: Introduce steal monitor driver
virt/steal_monitor: Restore to active on module disable virt/steal_monitor: Define steal_monitor structure virt/steal_monitor: Add control knobs for handling steal values virt/steal_monitor: Compute work at regular intervals virt/steal_monitor: Provide default method to get systemwide steal
time
virt/steal_monitor: Provide default method to inc/dec preferred CPUs virt/steal_monitor: Provide default method to get num of CPUs for
steal ratio
virt/steal_monitor: Act on steal values at regular intervals virt/steal_monitor: Add direction control virt/steal_monitor: Add design check of preferred subset of active
.../ABI/testing/sysfs-devices-system-cpu | 11 ++ Documentation/driver-api/index.rst | 1 + Documentation/driver-api/steal-monitor.rst | 93 ++++++++++++ Documentation/scheduler/sched-arch.rst | 50 +++++++ drivers/base/cpu.c | 8 ++ drivers/virt/Makefile | 1 + drivers/virt/steal_monitor/Makefile | 14 ++ drivers/virt/steal_monitor/defaults.c | 105 ++++++++++++++ drivers/virt/steal_monitor/sm_core.c | 124 ++++++++++++++++ drivers/virt/steal_monitor/sm_core.h | 32 +++++ include/linux/cpumask.h | 21 ++- include/linux/sched.h | 5 +- kernel/Kconfig.preempt | 14 ++ kernel/cpu.c | 6 + kernel/sched/core.c | 133 +++++++++++++++++- kernel/sched/debug.c | 4 +- kernel/sched/fair.c | 11 +- kernel/sched/sched.h | 36 +++++ 18 files changed, 659 insertions(+), 10 deletions(-) create mode 100644 Documentation/driver-api/steal-monitor.rst create mode 100644 drivers/virt/steal_monitor/Makefile create mode 100644 drivers/virt/steal_monitor/defaults.c create mode 100644 drivers/virt/steal_monitor/sm_core.c create mode 100644 drivers/virt/steal_monitor/sm_core.h
Signature · 1 line
2.47.3
PATCH v5 1/24 sched/debug: Remove unused schedstats
nr_migrations_cold, nr_wakeups_passive and nr_wakeups_idle are not being updated anywhere. So remove them.
These are per process stats. So updating sched stats version isn't necessary.
--- include/linux/sched.h | 3 --- kernel/sched/debug.c | 3 --- 2 files changed, 6 deletions(-)
include/linux/sched.h +0 -3
@@ -550,7 +550,6 @@ struct sched_statistics {s64 exec_max;u64 slice_max; -u64 nr_migrations_cold;u64 nr_failed_migrations_affine;u64 nr_failed_migrations_running;u64 nr_failed_migrations_hot;
@@ -563,8 +562,6 @@ struct sched_statistics {u64 nr_wakeups_remote;u64 nr_wakeups_affine;u64 nr_wakeups_affine_attempts; -u64 nr_wakeups_passive; -u64 nr_wakeups_idle;#ifdef CONFIG_SCHED_CORE u64 core_forceidle_sum;
kernel/sched/debug.c +0 -3
@@ -1359,7 +1359,6 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P_SCHEDSTAT (wait_count);PN_SCHEDSTAT (iowait_sum);P_SCHEDSTAT (iowait_count); -P_SCHEDSTAT (nr_migrations_cold);P_SCHEDSTAT (nr_failed_migrations_affine);P_SCHEDSTAT (nr_failed_migrations_running);P_SCHEDSTAT (nr_failed_migrations_hot);
@@ -1371,8 +1370,6 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P_SCHEDSTAT (nr_wakeups_remote);P_SCHEDSTAT (nr_wakeups_affine);P_SCHEDSTAT (nr_wakeups_affine_attempts); -P_SCHEDSTAT (nr_wakeups_passive); -P_SCHEDSTAT (nr_wakeups_idle); avg_atom = p->se.sum_exec_runtime;if (nr_switches)
Signature · 1 line
2.47.3
[tip: sched/core] sched/debug: Remove unused schedstats
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b2463ebf2674ddec62f0f0e63061670bc2c75346 Gitweb: https://git.kernel.org/tip/b2463ebf2674ddec62f0f0e63061670bc2c75346 Author: Shrikanth Hegde <[email protected]> AuthorDate: Thu, 25 Jun 2026 18:16:25 +05:30 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:57 +02:00
sched/debug: Remove unused schedstats
nr_migrations_cold, nr_wakeups_passive and nr_wakeups_idle are not being updated anywhere. So remove them.
These are per process stats. So updating sched stats version isn't necessary.
--- include/linux/sched.h | 3 --- kernel/sched/debug.c | 3 --- 2 files changed, 6 deletions(-)
include/linux/sched.h +0 -3
@@ -550,7 +550,6 @@ struct sched_statistics {s64 exec_max;u64 slice_max; -u64 nr_migrations_cold;u64 nr_failed_migrations_affine;u64 nr_failed_migrations_running;u64 nr_failed_migrations_hot;
@@ -563,8 +562,6 @@ struct sched_statistics {u64 nr_wakeups_remote;u64 nr_wakeups_affine;u64 nr_wakeups_affine_attempts; -u64 nr_wakeups_passive; -u64 nr_wakeups_idle;#ifdef CONFIG_SCHED_CORE u64 core_forceidle_sum;
kernel/sched/debug.c +0 -3
@@ -1442,7 +1442,6 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P_SCHEDSTAT (wait_count);PN_SCHEDSTAT (iowait_sum);P_SCHEDSTAT (iowait_count); -P_SCHEDSTAT (nr_migrations_cold);P_SCHEDSTAT (nr_failed_migrations_affine);P_SCHEDSTAT (nr_failed_migrations_running);P_SCHEDSTAT (nr_failed_migrations_hot);
@@ -1454,8 +1453,6 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P_SCHEDSTAT (nr_wakeups_remote);P_SCHEDSTAT (nr_wakeups_affine);P_SCHEDSTAT (nr_wakeups_affine_attempts); -P_SCHEDSTAT (nr_wakeups_passive); -P_SCHEDSTAT (nr_wakeups_idle); avg_atom = p->se.sum_exec_runtime;if (nr_switches)
PATCH v5 2/24 sched/docs: Document cpu_preferred_mask and Preferred CPU concept
In reply to Shrikanth HegdeAdd documentation for new cpumask called cpu_preferred_mask. This could help users in understanding what this mask is and the concept behind it.
Document how to enable it and implementation aspects of it.
---v4->v5:
- Change text to reflect new driver info.
- Changes suggested by Randy Dunlap.
- Sashiko nitpicks
Documentation/scheduler/sched-arch.rst | 50 ++++++++++++++++++++++++++ 1 file changed, 50 insertions(+)
Documentation/scheduler/sched-arch.rst +50 -0
@@ -62,6 +62,56 @@ Your cpu_idle routines need to obey the following rules: arch/x86/kernel/process.c has examples of both polling and sleeping idle functions. +Preferred CPUs +============== + +In virtualised environments it is possible to overcommit CPU resources. +i.e sum of virtual CPU(vCPU) of all VMs is greater than number of physical +CPUs(pCPU). Under such conditions when all or many VMs have high utilization, +hypervisor won't be able to satisfy the CPU requirement and has to context +switch within or across VMs. i.e hypervisor needs to preempt one vCPU to run +another. This is called vCPU preemption. This is more expensive compared to +task context switch within a vCPU. + +In such cases it is better that combined vCPU ask from all VMs is reduced +by not using some of the vCPUs in each VM. vCPUs where workload can be safely +scheduled which won't increase any contention for pCPU are called as +"Preferred CPUs". + +Main design construct is preferred CPUs is always subset of active CPUs. +In most cases preferred CPUs will be same as active CPUs, when there is pCPU +contention, Preferred CPUs will reduce based on the amount of steal time. +When the pCPU contention goes away as indicated by steal time, Preferred CPUs +will become same as active CPUs again. This is done by loading the +steal_monitor driver available at drivers/virt/steal_monitor. + +For scheduling decisions such as wakeup, pushing the task etc, needs this +CPU state info. This is maintained in cpu_preferred_mask. +vCPUs which are not in cpu_preferred_mask should be treated as vCPUs which +should not be used at this moment provided it doesn't break user affinity. + +This is achieved by +1. Selecting a preferred CPU at wakeup. +2. Push the task away from non-preferred CPU at tick. +3. Only select preferred CPUs for load balance. + +/sys/devices/system/cpu/preferred prints the current cpu_preferred_mask in +cpulist format. + +Notes: +1. This feature is available under CONFIG_PREFERRED_CPU. This enables + steal_monitor driver. On enabling the driver, CPU preferred state + can change based on steal time. With CONFIG_PREFERRED_CPU=n, + preferred CPUs is same as active CPUs. + +2. This feature works for FAIR class only. + +3. A task pinned, which can't be moved to preferred CPUs will continue + to run based on its affinity. But no load balancing happens. + +4. Decision to use/not use is driven by kernel. Hence it shouldn't + break user affinities. One of the main reasons why CPU hotplug + or Isolated cpuset partitions was not a solution. Possible arch/ problems =======================
Signature · 1 line
2.47.3
PATCH v5 3/24 kconfig: Provide PREFERRED_CPU option
In reply to Shrikanth HegdeIntroduce a new config named PREFERRED_CPU.
This helps to:
- Avoid the code bloat when PREFERRED_CPU=n. In that cases preferred is same as active.
- Avoid the ifdeffery around PREFERRED_CPU in many files.
Since paravirtualized use case is the main driving force of this feature, make it default for kernels with PARAVIRT=y
---v4->v5:
- Make it depend on instead. (Yury Norov)
- Fix helper indentation (sashiko)
kernel/Kconfig.preempt | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
kernel/Kconfig.preempt +14 -0
@@ -192,3 +192,17 @@ config SCHED_CLASS_EXT For more information: Documentation/scheduler/sched-ext.rst https://github.com/sched-ext/scx + +config PREFERRED_CPU + bool "Dynamic vCPU management based on steal time" + depends on PARAVIRT && SMP + default y + help + This feature helps to reduce the steal time in paravirtualised + environment, there by reducing vCPU preemption. Reducing vCPU + preemption provides improved lock holder preemption and reduces + cost of vCPU preemption in the host. + + By default preferred CPUs will be same as active CPUs. Depending + on the steal time when steal_monitor driver is enabled, + preferred CPUs could become subset of active CPUs.
Signature · 1 line
2.47.3
PATCH v5 4/24 cpumask: Introduce cpu_preferred_mask
In reply to Shrikanth HegdeThis patch does
- Declare and Define cpu_preferred_mask.
- Get/Set helpers for it.
Values are set/clear by the scheduler by detecting the steal time values.
A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal time values with steal monitor being enabled.
Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
---v4->v5:
- Make it macro instead (Yury Norov)
include/linux/cpumask.h | 21 ++++++++++++++++++++- kernel/cpu.c | 6 ++++++ kernel/sched/core.c | 5 +++++ 3 files changed, 31 insertions(+), 1 deletion(-)
include/linux/cpumask.h +20 -1
@@ -120,12 +120,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; + +#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif +#define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)#define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask)#define cpu_enabled_mask ((const struct cpumask *)&__cpu_enabled_mask)#define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask)#define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask)#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) +#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)extern atomic_t __num_online_cpus;extern unsigned int __num_possible_cpus;
@@ -1161,6 +1169,7 @@ void init_cpu_possible(const struct cpumask *src);#define set_cpu_present (cpu, present) assign_cpu((cpu), &__cpu_present_mask, (present))#define set_cpu_active (cpu, active) assign_cpu((cpu), &__cpu_active_mask, (active))#define set_cpu_dying (cpu, dying) assign_cpu((cpu), &__cpu_dying_mask, (dying)) +#define set_cpu_preferred (cpu, preferred) assign_cpu((cpu), &__cpu_preferred_mask, (preferred))void set_cpu_online (unsigned int cpu,bool online);void set_cpu_possible (unsigned int cpu,bool possible);
@@ -1256,7 +1265,12 @@ static __always_inline bool cpu_dying(unsigned int cpu)return cpumask_test_cpu (cpu, cpu_dying_mask); } -#else +static __always_inline bool cpu_preferred(unsigned int cpu) +{ +return cpumask_test_cpu (cpu, cpu_preferred_mask); +} + +#else /* NR_CPUS <= 1 */ #define num_online_cpus () 1U#define num_possible_cpus () 1U
@@ -1294,6 +1308,11 @@ static __always_inline bool cpu_dying(unsigned int cpu)return false; } +static __always_inline bool cpu_preferred(unsigned int cpu) +{ +return cpu ==0 ; +} +#endif /* NR_CPUS > 1 */ #define cpu_is_offline (cpu) unlikely(!cpu_online(cpu))
kernel/cpu.c +6 -0
@@ -3107,6 +3107,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);atomic_t __num_online_cpus__read_mostly ;EXPORT_SYMBOL (__num_online_cpus); +#ifdef CONFIG_PREFERRED_CPU +struct cpumask __cpu_preferred_mask__read_mostly ; +EXPORT_SYMBOL (__cpu_preferred_mask); +#endif +void init_cpu_present (const struct cpumask *src) {cpumask_copy (&__cpu_present_mask, src);
@@ -3164,6 +3169,7 @@ void __init boot_cpu_init(void)/* Mark the boot cpu "present", "online" etc for SMP and UP case */ set_cpu_online (cpu, true);set_cpu_active (cpu, true); +set_cpu_preferred (cpu, true);set_cpu_present (cpu, true);set_cpu_possible (cpu, true);
kernel/sched/core.c +5 -0
@@ -8685,6 +8685,9 @@ int sched_cpu_activate(unsigned int cpu)*/ sched_set_rq_online (rq, cpu); +/* preferred is subset of active and follows its state */ +set_cpu_preferred (cpu, true); +return 0 ; }
@@ -8698,6 +8701,8 @@ int sched_cpu_deactivate(unsigned int cpu)if (ret)return ret; +set_cpu_preferred (cpu, false); +/* * Remove CPU from nohz.idle_cpus_mask to prevent participating in * load balancing when not active
Signature · 1 line
2.47.3
Peter Zijlstra
maintainerRe: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
Quoted 3 lines · Shrikanth Hegde
This patch does
- Declare and Definecpu_preferred_mask.
- Get/Set helpers for it.
There is a blub in submitting-patches.rst about how 'this patch' is basically a red-flag for a changelog.
The changelog is per-definition pertaining to 'this patch', therefore stating this is a tautology. Further, it is often fairly clear what the patch does, but less clear as to why.
So the suggestion is to phrase this like:
Provide cpu_preferred_mask infrastructure (definitions, declarations and helper methods) to facilitate ....
Quoted 10 lines
Values are set/clear by the scheduler by detecting the steal time values.
A CPU is set to preferred when it becomes active. Later it may be marked as non-preferred depending on steal time values with steal monitor being enabled.
Always maintain design construct of preferred is subset of active. i.e. preferred ⊆ active ⊆ online ⊆ present ⊆ possible
Signed-off-by Shrikanth Hegde <[email protected]>
Re: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Shrikanth Hegde
This patch does
- Declare and Definecpu_preferred_mask.
- Get/Set helpers for it.There is a blub in
submitting-patches.rstabout how 'this patch' is basically a red-flag for a changelog.
Quoted 10 lines
The changelog is per-definition pertaining to 'this patch', therefore stating this is a tautology. Further, it is often fairly clear what the patch does, but less clear as to why.
So the suggestion is to phrase this like:
Provide
cpu_preferred_maskinfrastructure (definitions, declarations and helper methods) to facilitate ....
Ok. My bad, i will update the changelog. thanks for catching it.
Peter Zijlstra
maintainerRe: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
In reply to Shrikanth HegdeQuoted 5 lines · Shrikanth Hegde
include/linux/cpumask.h +0
-0
include/linux/cpumask.h extern struct cpumask __cpu_enabled_mask; · line 128
Quoted 9 lines
include/linux/cpumask.h +6
-0@@ -120,12 +120,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; + +#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif
This is cure, but does it not result in set_cpu_preferred() changing active mask, and it that not somewhat unexpected behaviour?
Quoted 16 lines
include/linux/cpumask.h +1
-0#define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)#define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask)#define cpu_enabled_mask ((const struct cpumask *)&__cpu_enabled_mask)#define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask)#define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask)#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) +#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)extern atomic_t __num_online_cpus;extern unsigned int __num_possible_cpus;kernel/cpu.c +0
-0
kernel/cpu.c EXPORT_SYMBOL(__cpu_dying_mask); · line 3113
Quoted 7 lines
kernel/cpu.c +4
-0@@ -3107,6 +3107,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);atomic_t __num_online_cpus__read_mostly ;EXPORT_SYMBOL (__num_online_cpus); +#ifdef CONFIG_PREFERRED_CPU +struct cpumask __cpu_preferred_mask__read_mostly ; +EXPORT_SYMBOL (__cpu_preferred_mask); +#endif
Precedent is definitely towards !GPL exports for this, but could we get away with making this one GPL?
kernel/cpu.c void __init boot_cpu_init(void) · line 3172
Quoted 5 lines
kernel/cpu.c +1
-0@@ -3164,6 +3169,7 @@ void __init boot_cpu_init(void)/* Mark the boot cpu "present", "online" etc for SMP and UP case */ set_cpu_online (cpu, true);set_cpu_active (cpu, true); +set_cpu_preferred (cpu, true);
This sets active twice, which is harmless, but wasteful...
Quoted 18 lines
kernel/cpu.c +0
-0set_cpu_present (cpu, true);set_cpu_possible (cpu, true);kernel/sched/core.c +3
-0@@ -8685,6 +8685,9 @@ int sched_cpu_activate(unsigned int cpu)*/ sched_set_rq_online (rq, cpu); +/* preferred is subset of active and follows its state */ +set_cpu_preferred (cpu, true); +return 0 ; }
kernel/sched/core.c int sched_cpu_deactivate(unsigned int cpu) · line 8708
Quoted 8 lines
kernel/sched/core.c +2
-0@@ -8698,6 +8701,8 @@ int sched_cpu_deactivate(unsigned int cpu)if (ret)return ret; +set_cpu_preferred (cpu, false); +/* * Remove CPU from nohz.idle_cpus_mask to prevent participating in * load balancing when not active
But this one clears active earlier, is that not a problem?
Perhaps it is best if the modifier is a no-op when preferred mask does not exist?
Peter Zijlstra
maintainerRe: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
Quoted 18 lines · Peter Zijlstra
Quoted 5 lines · Shrikanth Hegde
include/linux/cpumask.h +0
-0include/linux/cpumask.h extern struct cpumask __cpu_enabled_mask; · line 128
Quoted 9 lines
include/linux/cpumask.h +6
-0@@ -120,12 +120,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; + +#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif This is cure, but does it not result in
set_cpu_preferred()changing
s/cure/cute/
Quoted 1 line
active mask, and it that not somewhat unexpected behaviour?
s/it/is/
Typing hard, clearly. Also hitting 30C before noon :-(
Re: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
Hi Peter, Yury.
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 5 lines · Shrikanth Hegde
include/linux/cpumask.h +0
-0include/linux/cpumask.h extern struct cpumask __cpu_enabled_mask; · line 128
Quoted 9 lines
include/linux/cpumask.h +6
-0@@ -120,12 +120,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; + +#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif This is cure, but does it not result in
set_cpu_preferred()changing
Quoted 4 lines
s/cure/cute/
Quoted 1 line
active mask, and it that not somewhat unexpected behaviour?
s/it/is/
Yes. I thought about this, but i didn't see anything bad happening apart from setting it twice. But I do agree, it is an eyesore when CONFIG_PREFERRED_CPU=n.
Quoted 2 lines
Typing hard, clearly. Also hitting 30C before noon :-(
Take care. Even we should have had monsoon by now. But its bright sunshine :(
Quoted 1 line
For this reason, i had it as a function instead of macro in v4. Do you think we can still fallback to it?
only caveat is it won't be a macro. But since it is still compile time optimized due to IS_ENABLED, it should be relatively ok right?
+void set_cpu_preferred(unsigned int cpu, bool preferred) +{
+ if (!IS_ENABLED(CONFIG_PREFERRED_CPU))
+ return;
+
+ assign_cpu((cpu), &__cpu_preferred_mask, (preferred));
+}
Yury Norov
maintainerRe: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
Quoted 44 lines · Shrikanth Hegde
Hi Peter, Yury.
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 5 lines · Shrikanth Hegde
include/linux/cpumask.h +0
-0include/linux/cpumask.h extern struct cpumask __cpu_enabled_mask; · line 128
Quoted 9 lines
include/linux/cpumask.h +6
-0@@ -120,12 +120,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; + +#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif This is cure, but does it not result in
set_cpu_preferred()changingQuoted 4 lines
s/cure/cute/
Quoted 1 line
active mask, and it that not somewhat unexpected behaviour?
s/it/is/
Yes. I thought about this, but i didn't see anything bad happening apart from setting it twice. But I do agree, it is an eyesore when
CONFIG_PREFERRED_CPU=n.Quoted 2 lines
Typing hard, clearly. Also hitting 30C before noon :-(
Take care. Even we should have had monsoon by now. But its bright sunshine :(
Quoted 1 line
For this reason, i had it as a function instead of macro in v4. Do you think we can still fallback to it?
only caveat is it won't be a macro. But since it is still compile time optimized due to
IS_ENABLED, it should be relatively ok right?
Review
Quoted 7 lines
(quoted diff) +7
-0+void set_cpu_preferred(unsigned int cpu, bool preferred) +{ + if (!IS_ENABLED(CONFIG_PREFERRED_CPU)) + return; + + assign_cpu((cpu), &__cpu_preferred_mask, (preferred)); +}
#ifdef CONFIG_PREFERRED_CPU
#define set_cpu_preferred(cpu, preferred) assign_cpu((cpu), &__cpu_preferred_mask, (preferred)) #else
#define set_cpu_preferred(cpu, preferred) {}
#endif
Re: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
Quoted 45 lines · Yury Norov
Quoted 44 lines · Shrikanth Hegde
Hi Peter, Yury.
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 5 lines · Shrikanth Hegde
include/linux/cpumask.h +0
-0include/linux/cpumask.h extern struct cpumask __cpu_enabled_mask; · line 128
Quoted 9 lines
include/linux/cpumask.h +6
-0@@ -120,12 +120,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; + +#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif This is cure, but does it not result in
set_cpu_preferred()changingQuoted 4 lines
s/cure/cute/
Quoted 1 line
active mask, and it that not somewhat unexpected behaviour?
s/it/is/
Yes. I thought about this, but i didn't see anything bad happening apart from setting it twice. But I do agree, it is an eyesore when
CONFIG_PREFERRED_CPU=n.Quoted 2 lines
Typing hard, clearly. Also hitting 30C before noon :-(
Take care. Even we should have had monsoon by now. But its bright sunshine :(
Quoted 1 line
For this reason, i had it as a function instead of macro in v4. Do you think we can still fallback to it?
only caveat is it won't be a macro. But since it is still compile time optimized due to
IS_ENABLED, it should be relatively ok right?
Quoted 14 lines
Review
Quoted 7 lines
(quoted diff) +7
-0+void set_cpu_preferred(unsigned int cpu, bool preferred) +{ + if (!IS_ENABLED(CONFIG_PREFERRED_CPU)) + return; + + assign_cpu((cpu), &__cpu_preferred_mask, (preferred)); +}#ifdef CONFIG_PREFERRED_CPU #define set_cpu_preferred(cpu, preferred) assign_cpu((cpu), &__cpu_preferred_mask, (preferred)) #else #define set_cpu_preferred(cpu, preferred) {} #endif
Ah! thanks.
Yury Norov
maintainerRe: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
In reply to Peter ZijlstraQuoted 11 lines · Peter Zijlstra
Quoted 9 lines · Shrikanth Hegde
include/linux/cpumask.h +1
-0@@ -120,12 +120,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; +
Quoted 8 lines
include/linux/cpumask.h extern struct cpumask __cpu_enabled_mask; · line 128
Quoted 5 lines
include/linux/cpumask.h +5
-0+#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif This is cure, but does it not result in
set_cpu_preferred()changing active mask, and it that not somewhat unexpected behaviour?
I agree, and I think I already commented on it on previous round. set_cpu_preferred() should be protected the same way as the corresponding mask, and should be a NOP when CONFIG_PREFERRED_CPU is disabled.
Quoted 28 lines
Quoted 16 lines
include/linux/cpumask.h +1
-0#define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)#define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask)#define cpu_enabled_mask ((const struct cpumask *)&__cpu_enabled_mask)#define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask)#define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask)#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) +#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)extern atomic_t __num_online_cpus;extern unsigned int __num_possible_cpus;kernel/cpu.c +0
-0kernel/cpu.c EXPORT_SYMBOL(__cpu_dying_mask); · line 3113
Quoted 7 lines
kernel/cpu.c +4
-0@@ -3107,6 +3107,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);atomic_t __num_online_cpus__read_mostly ;EXPORT_SYMBOL (__num_online_cpus); +#ifdef CONFIG_PREFERRED_CPU +struct cpumask __cpu_preferred_mask__read_mostly ; +EXPORT_SYMBOL (__cpu_preferred_mask); +#endif Precedent is definitely towards !GPL exports for this, but could we get away with making this one GPL?
Quoted 7 lines
kernel/cpu.c void __init boot_cpu_init(void) · line 3172
Quoted 5 lines
kernel/cpu.c +1
-0@@ -3164,6 +3169,7 @@ void __init boot_cpu_init(void)/* Mark the boot cpu "present", "online" etc for SMP and UP case */ set_cpu_online (cpu, true);set_cpu_active (cpu, true); +set_cpu_preferred (cpu, true);This sets active twice, which is harmless, but wasteful...
I think, the good criteria for correctness of this series would be the identical binaries before the series, and when CONFIG_PREFERRED_CPU is off. At least, as a mental model. This double-set chunk breaks that model.
Thanks,
Yury
Quoted 31 lines
Quoted 18 lines
kernel/cpu.c +0
-0set_cpu_present (cpu, true);set_cpu_possible (cpu, true);kernel/sched/core.c +3
-0@@ -8685,6 +8685,9 @@ int sched_cpu_activate(unsigned int cpu)*/ sched_set_rq_online (rq, cpu); +/* preferred is subset of active and follows its state */ +set_cpu_preferred (cpu, true); +return 0 ; }kernel/sched/core.c int sched_cpu_deactivate(unsigned int cpu) · line 8708
Quoted 8 lines
kernel/sched/core.c +2
-0@@ -8698,6 +8701,8 @@ int sched_cpu_deactivate(unsigned int cpu)if (ret)return ret; +set_cpu_preferred (cpu, false); +/* * Remove CPU from nohz.idle_cpus_mask to prevent participating in * load balancing when not active But this one clears active earlier, is that not a problem?
Perhaps it is best if the modifier is a no-op when preferred mask does not exist?
Re: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
Quoted 62 lines · Yury Norov
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Shrikanth Hegde
include/linux/cpumask.h +1
-0@@ -120,12 +120,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; +Quoted 8 lines
include/linux/cpumask.h extern struct cpumask __cpu_enabled_mask; · line 128
Quoted 5 lines
include/linux/cpumask.h +5
-0+#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif This is cure, but does it not result in
set_cpu_preferred()changing active mask, and it that not somewhat unexpected behaviour?I agree, and I think I already commented on it on previous round.
set_cpu_preferred()should be protected the same way as the corresponding mask, and should be a NOP whenCONFIG_PREFERRED_CPUis disabled.Quoted 28 lines
Quoted 16 lines
include/linux/cpumask.h +1
-0#define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)#define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask)#define cpu_enabled_mask ((const struct cpumask *)&__cpu_enabled_mask)#define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask)#define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask)#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) +#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)extern atomic_t __num_online_cpus;extern unsigned int __num_possible_cpus;kernel/cpu.c +0
-0kernel/cpu.c EXPORT_SYMBOL(__cpu_dying_mask); · line 3113
Quoted 7 lines
kernel/cpu.c +4
-0@@ -3107,6 +3107,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);atomic_t __num_online_cpus__read_mostly ;EXPORT_SYMBOL (__num_online_cpus); +#ifdef CONFIG_PREFERRED_CPU +struct cpumask __cpu_preferred_mask__read_mostly ; +EXPORT_SYMBOL (__cpu_preferred_mask); +#endif Precedent is definitely towards !GPL exports for this, but could we get away with making this one GPL?
Quoted 7 lines
kernel/cpu.c void __init boot_cpu_init(void) · line 3172
Quoted 5 lines
kernel/cpu.c +1
-0@@ -3164,6 +3169,7 @@ void __init boot_cpu_init(void)/* Mark the boot cpu "present", "online" etc for SMP and UP case */ set_cpu_online (cpu, true);set_cpu_active (cpu, true); +set_cpu_preferred (cpu, true);This sets active twice, which is harmless, but wasteful...
Quoted 5 lines
I think, the good criteria for correctness of this series would be the identical binaries before the series, and when
CONFIG_PREFERRED_CPUis off. At least, as a mental model. This double-set chunk breaks that model.
Sorry, i didn't get how comparison could be,
You mean bloat-o-meter or kernel/cpu.o size or vmlinux size file?
That would mean everything should be under ifdef CONFIG_PREFERRED_CPU.
No? That was the case in few versions earlier, and it was not looking good since due to many ifdefs.
If we fix set_cpu_preferred to be a NOP when CONFIG_PREFERRED_CPU=n and driver depends on it, i think we should be good.
What do you think?
Re: [PATCH v5 04/24] cpumask: Introduce cpu_preferred_mask
In reply to Peter ZijlstraHi Peter,
Quoted 39 lines · Peter Zijlstra
Quoted 5 lines · Shrikanth Hegde
include/linux/cpumask.h +0
-0include/linux/cpumask.h extern struct cpumask __cpu_enabled_mask; · line 128
Quoted 9 lines
include/linux/cpumask.h +6
-0@@ -120,12 +120,20 @@ extern struct cpumask __cpu_enabled_mask;extern struct cpumask __cpu_present_mask;extern struct cpumask __cpu_active_mask;extern struct cpumask __cpu_dying_mask; + +#ifdef CONFIG_PREFERRED_CPU +extern struct cpumask __cpu_preferred_mask; +#else +#define __cpu_preferred_mask __cpu_active_mask +#endif This is cure, but does it not result in
set_cpu_preferred()changing active mask, and it that not somewhat unexpected behaviour?Quoted 19 lines
include/linux/cpumask.h +1
-0#define cpu_possible_mask ((const struct cpumask *)&__cpu_possible_mask)#define cpu_online_mask ((const struct cpumask *)&__cpu_online_mask)#define cpu_enabled_mask ((const struct cpumask *)&__cpu_enabled_mask)#define cpu_present_mask ((const struct cpumask *)&__cpu_present_mask)#define cpu_active_mask ((const struct cpumask *)&__cpu_active_mask)#define cpu_dying_mask ((const struct cpumask *)&__cpu_dying_mask) +#define cpu_preferred_mask ((const struct cpumask *)&__cpu_preferred_mask)extern atomic_t __num_online_cpus;extern unsigned int __num_possible_cpus;kernel/cpu.c +0
-0@@ -3107,6 +3107,11 @@ EXPORT_SYMBOL(__cpu_dying_mask);atomic_t __num_online_cpus__read_mostly ;EXPORT_SYMBOL (__num_online_cpus);
Quoted 8 lines
kernel/cpu.c EXPORT_SYMBOL(__cpu_dying_mask); · line 3113
Quoted 4 lines
kernel/cpu.c +4
-0+#ifdef CONFIG_PREFERRED_CPU +struct cpumask __cpu_preferred_mask__read_mostly ; +EXPORT_SYMBOL (__cpu_preferred_mask); +#endif Precedent is definitely towards !GPL exports for this, but could we get away with making this one GPL?
I think it is better to put EXPORT_SYMBOL_GPL. So that any module tries to alter it must be a GPL one. Would be less headache for us i guess.
Quoted 38 lines
kernel/cpu.c void __init boot_cpu_init(void) · line 3172
Quoted 6 lines
kernel/cpu.c +1
-0@@ -3164,6 +3169,7 @@ void __init boot_cpu_init(void)/* Mark the boot cpu "present", "online" etc for SMP and UP case */ set_cpu_online (cpu, true);set_cpu_active (cpu, true); +set_cpu_preferred (cpu, true);This sets active twice, which is harmless, but wasteful...
Quoted 18 lines
kernel/cpu.c +0
-0set_cpu_present (cpu, true);set_cpu_possible (cpu, true);kernel/sched/core.c +3
-0@@ -8685,6 +8685,9 @@ int sched_cpu_activate(unsigned int cpu)*/ sched_set_rq_online (rq, cpu); +/* preferred is subset of active and follows its state */ +set_cpu_preferred (cpu, true); +return 0 ; }kernel/sched/core.c int sched_cpu_deactivate(unsigned int cpu) · line 8708
Quoted 8 lines
kernel/sched/core.c +2
-0@@ -8698,6 +8701,8 @@ int sched_cpu_deactivate(unsigned int cpu)if (ret)return ret; +set_cpu_preferred (cpu, false); +/* * Remove CPU from nohz.idle_cpus_mask to prevent participating in * load balancing when not active But this one clears active earlier, is that not a problem?
Quoted 2 lines
Perhaps it is best if the modifier is a no-op when preferred mask does not exist?
Will make it as yury suggested in other thread with ifdefs.
PATCH v5 5/24 sysfs: Add preferred CPU file
In reply to Shrikanth HegdeAdd "preferred" file in /sys/devices/system/cpu
This offers
- User can quickly check which CPUs are marked as preferred at this moment.
- Userspace algorithms irqbalance could use this mask to send irq into preferred CPUs.
For example:
cat /sys/devices/system/cpu/online
0-719
cat /sys/devices/system/cpu/preferred
0-599 <<< Implies 0-599 are preferred for workloads and 600-719
should be avoided at this moment.
cat /sys/devices/system/cpu/preferred
0-719 <<< All CPUs are usable. There is no preferrence.
--- Documentation/ABI/testing/sysfs-devices-system-cpu | 11 +++++++++++ drivers/base/cpu.c | 8 ++++++++ 2 files changed, 19 insertions(+)
Documentation/ABI/testing/sysfs-devices-system-cpu +11 -0
@@ -806,3 +806,14 @@ Date: Nov 2022 Contact: Linux kernel mailing list <[email protected]> Description: (RO) the list of CPUs that can be brought online. + +What: /sys/devices/system/cpu/preferred +Date: Jun 2026 +Contact: Linux kernel mailing list <[email protected]> +Description: + (RO) the list of preferred CPUs at this moment. + These are the only CPUs meant to be used at the moment. + Using CPU outside of the list could lead to more + contention of underlying physical CPU resource. Dynamically + changes based on steal time. With CONFIG_PREFERRED_CPU=n it + is same as active CPUs. See sched-arch.rst for more details.
drivers/base/cpu.c +8 -0
@@ -391,6 +391,13 @@ static int cpu_uevent(const struct device *dev, struct kobj_uevent_env *env) }#endif +static ssize_t preferred_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +return sysfs_emit (buf,"%*pbl\n" ,cpumask_pr_args (cpu_preferred_mask)); +} +static DEVICE_ATTR_RO (preferred); +const struct bus_type cpu_subsys = { .name ="cpu" , .dev_name ="cpu" ,
@@ -532,6 +539,7 @@ static struct attribute *cpu_root_attrs[] = {#ifdef CONFIG_GENERIC_CPU_AUTOPROBE &dev_attr_modalias.attr, #endif + &dev_attr_preferred.attr,NULL };
Signature · 1 line
2.47.3
PATCH v5 6/24 sched/core: allow only preferred CPUs in is_cpu_allowed
In reply to Shrikanth HegdeWhen possible, choose a preferred CPUs to pick.
Push task mechanism uses stopper thread which going to call select_fallback_rq and use this mechanism to pick only a preferred CPU.
When task is affined only to non-preferred CPUs it should continue to run there. Detect that by checking if cpus_ptr and cpu_preferred_mask intersect or not.
Since is_cpu_allowed can be called directly or repeatedly in select_fallback_rq, encode the info in task_struct->has_preferred_cpu_state if the path is via select_fallback_rq or not.
This helps to avoid N**2 complexity for the rare cases.
Additional overhead of O(N) comes to is_cpu_allowed only when cpu is not preferred. So in normal scenarios overhead is only a bit check.
---v4->v5:
- Do simple encoding of -1,0,1 instead (K Prateek Nayak)
- Make it s8 (K Prateek Nayak)
- Update changelog to address sashiko concerns of overhead.
include/linux/sched.h | 1 + kernel/sched/core.c | 35 +++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 25 +++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 2 deletions(-)
include/linux/sched.h +1 -0
@@ -1657,6 +1657,7 @@ struct task_struct {#ifdef CONFIG_UNWIND_USER struct unwind_task_info unwind_info;#endif +s8 has_preferred_cpu_state;/* CPU-specific state of this task: */ struct thread_struct thread;
kernel/sched/core.c +33 -2
@@ -2500,6 +2500,8 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)*/ static inline bool is_cpu_allowed (struct task_struct *p,int cpu) { +bool task_check_preferred_cpu; +/* When not in the task's cpumask, no point in looking further. */ if (!task_allowed_on_cpu (p, cpu))return false;
@@ -2508,9 +2510,23 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (is_migration_disabled (p))return cpu_online (cpu); +/* +* This is essential to maintain user affinities when preferred +* CPUs change. A task pinned on non-preferred CPU should continue +* to run there, since this is non-user triggered. +* +* If CPU is non-preferred and task can run on other CPUs which are +* currently preferred, then choose those other CPUs instead. +* Overhead is minimal when CPU is preferred. +*/ + task_check_preferred_cpu = !cpu_preferred (cpu) &&task_has_preferred_cpus (p); +/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +if (task_check_preferred_cpu) +return false;return cpu_active (cpu); + }/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))
@@ -2520,6 +2536,10 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (cpu_dying (cpu))return false; +/* Try on preferred CPU first if possible*/ +if (task_check_preferred_cpu) +return false; +/* But are allowed during online. */ return cpu_online (cpu); }
@@ -3549,6 +3569,14 @@ static int select_fallback_rq(int cpu, struct task_struct *p)enum { cpuset, possible, fail } state = cpuset;int dest_cpu; +/* +* Cache the value whether task's affinity spans preferred CPUs. +* This helps to avoid repeating the same for each CPU +* later in the loop. Encode call to is_cpu_allowed coming +* via select_fallback_rq. +*/ + p->has_preferred_cpu_state =task_has_preferred_cpus (p) ?1 :-1 ; +/* * If the node that the CPU is on has been offlined, cpu_to_node() * will return -1. There is no CPU on the node, and we should
@@ -3560,7 +3588,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* Look for allowed, online CPU in same node. */ for_each_cpu (dest_cpu, nodemask) {if (is_cpu_allowed (p, dest_cpu)) -return dest_cpu; + goto clear_and_return; } }
@@ -3604,6 +3632,8 @@ static int select_fallback_rq(int cpu, struct task_struct *p) } } +clear_and_return: + p->has_preferred_cpu_state =0 ;return dest_cpu; }
@@ -4612,6 +4642,7 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p)init_numa_balancing (clone_flags, p); p->wake_entry.u_flags =CSD_TYPE_TTWU ; p->migration_pending =NULL ; + p->has_preferred_cpu_state =0 ;init_sched_mm (p); }
kernel/sched/sched.h +25 -0
@@ -4213,4 +4213,29 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)#include "ext.h" +/* +* has_preferred_cpu_state could have the value cached from +* select_fallback_rq. It is set/cleared while holding pi_lock +* and irq disabled. +* +* 1: Cached and preferred CPUs exists in task's affinity. +* 0: Not cached and need to evaluate. +* -1: Cached and preferred CPU doesn't exits task's affinity +* +* Only affects FAIR task. +*/ +static inline bool task_has_preferred_cpus (struct task_struct *p) +{ +int cached; + +/* Only FAIR tasks honor preferred CPU state */ +if (unlikely (p->sched_class != &fair_sched_class)) +return false; + + cached =READ_ONCE (p->has_preferred_cpu_state); +if (cached) +return cached >0 ; +else +return cpumask_intersects (p->cpus_ptr, cpu_preferred_mask); +}#endif /* _KERNEL_SCHED_SCHED_H */
Signature · 1 line
2.47.3
Yury Norov
maintainerRe: [PATCH v5 06/24] sched/core: allow only preferred CPUs in is_cpu_allowed
Quoted 34 lines · Shrikanth Hegde
When possible, choose a preferred CPUs to pick.
Push task mechanism uses stopper thread which going to call
select_fallback_rqand use this mechanism to pick only a preferred CPU.When task is affined only to non-preferred CPUs it should continue to run there. Detect that by checking if
cpus_ptrandcpu_preferred_maskintersect or not.Since
is_cpu_allowedcan be called directly or repeatedly inselect_fallback_rq, encode the info intask_struct->has_preferred_cpu_stateif the path is viaselect_fallback_rqor not.
This helps to avoid N**2 complexity for the rare cases.Additional overhead of O(N) comes to
is_cpu_allowedonly when cpu is not preferred. So in normal scenarios overhead is only a bit check.Signed-off-by Shrikanth Hegde <[email protected]>---
v4->v5:
- Do simple encoding of -1,0,1 instead (K Prateek Nayak)
- Make it s8 (K Prateek Nayak)
- Update changelog to address sashiko concerns of overhead.include/linux/sched.h | 1 + kernel/sched/core.c | 35 +++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 25 +++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 2 deletions(-)include/linux/sched.h +0
-0
include/linux/sched.h struct task_struct { · line 1660
Quoted 4 lines
include/linux/sched.h +1
-0@@ -1657,6 +1657,7 @@ struct task_struct {#ifdef CONFIG_UNWIND_USER struct unwind_task_info unwind_info;#endif +s8 has_preferred_cpu_state;
Why not protected with the config?
It looks like you didn't ever ran pahole on it. Maybe it's worth to try now?
Quoted 76 lines
include/linux/sched.h +0
-0/* CPU-specific state of this task: */ struct thread_struct thread;kernel/sched/core.c +30
-2@@ -2500,6 +2500,8 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)*/ static inline bool is_cpu_allowed (struct task_struct *p,int cpu) { +bool task_check_preferred_cpu; +/* When not in the task's cpumask, no point in looking further. */ if (!task_allowed_on_cpu (p, cpu))return false;@@ -2508,9 +2510,23 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (is_migration_disabled (p))return cpu_online (cpu); +/* +* This is essential to maintain user affinities when preferred +* CPUs change. A task pinned on non-preferred CPU should continue +* to run there, since this is non-user triggered. +* +* If CPU is non-preferred and task can run on other CPUs which are +* currently preferred, then choose those other CPUs instead. +* Overhead is minimal when CPU is preferred. +*/ + task_check_preferred_cpu = !cpu_preferred (cpu) &&task_has_preferred_cpus (p); +/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +if (task_check_preferred_cpu) +return false;return cpu_active (cpu); + }/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))@@ -2520,6 +2536,10 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (cpu_dying (cpu))return false; +/* Try on preferred CPU first if possible*/ +if (task_check_preferred_cpu) +return false; +/* But are allowed during online. */ return cpu_online (cpu); }@@ -3549,6 +3569,14 @@ static int select_fallback_rq(int cpu, struct task_struct *p)enum { cpuset, possible, fail } state = cpuset;int dest_cpu; +/* +* Cache the value whether task's affinity spans preferred CPUs. +* This helps to avoid repeating the same for each CPU +* later in the loop. Encode call to is_cpu_allowed coming +* via select_fallback_rq. +*/ + p->has_preferred_cpu_state =task_has_preferred_cpus (p) ?1 :-1 ; +/* * If the node that the CPU is on has been offlined, cpu_to_node() * will return -1. There is no CPU on the node, and we should @@ -3560,7 +3588,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* Look for allowed, online CPU in same node. */ for_each_cpu (dest_cpu, nodemask) {if (is_cpu_allowed (p, dest_cpu)) -return dest_cpu; + goto clear_and_return; } }
kernel/sched/core.c static int select_fallback_rq(int cpu, struct task_struct *p) · line 3636
Quoted 5 lines
kernel/sched/core.c +2
-0@@ -3604,6 +3632,8 @@ static int select_fallback_rq(int cpu, struct task_struct *p) } } +clear_and_return: + p->has_preferred_cpu_state =0 ;
Sadly, you've ignored my comments from the previous round. Let me repeat it once again:
This ->has_preferred_cpu_state is always zero out of the scope of the function. It means, it's a local variable, and should not belong to the task_struct.
Quoted 17 lines
kernel/sched/core.c +1
-0return dest_cpu; }@@ -4612,6 +4642,7 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p)init_numa_balancing (clone_flags, p); p->wake_entry.u_flags =CSD_TYPE_TTWU ; p->migration_pending =NULL ; + p->has_preferred_cpu_state =0 ;init_sched_mm (p); }kernel/sched/sched.h +0
-0@@ -4213,4 +4213,29 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)
kernel/sched/sched.h DEFINE_CLASS_IS_UNCONDITIONAL(sched_change) · line 4223
Quoted 10 lines
kernel/sched/sched.h +8
-0#include "ext.h" +/* +* has_preferred_cpu_state could have the value cached from +* select_fallback_rq. It is set/cleared while holding pi_lock +* and irq disabled. +* +* 1: Cached and preferred CPUs exists in task's affinity. +* 0: Not cached and need to evaluate. +* -1: Cached and preferred CPU doesn't exits task's affinity
So, you've got 3 options to declare the status: self-explaining enum, self-explaining #defines, and this random numbers explained in comment. The latter option is the worst to me.
And you didn't provide any benchmark advocating this caching optimization.
Sorry, but NAK.
Quoted 20 lines
kernel/sched/sched.h +17
-0+* +* Only affects FAIR task. +*/ +static inline bool task_has_preferred_cpus (struct task_struct *p) +{ +int cached; + +/* Only FAIR tasks honor preferred CPU state */ +if (unlikely (p->sched_class != &fair_sched_class)) +return false; + + cached =READ_ONCE (p->has_preferred_cpu_state); +if (cached) +return cached >0 ; +else +return cpumask_intersects (p->cpus_ptr, cpu_preferred_mask); +}#endif /* _KERNEL_SCHED_SCHED_H */ Signature · 1 line
2.47.3
Re: [PATCH v5 06/24] sched/core: allow only preferred CPUs in is_cpu_allowed
Hi Yury. Thanks for going through the patches.
Quoted 33 lines · Yury Norov
Quoted 31 lines · Shrikanth Hegde
When possible, choose a preferred CPUs to pick.
Push task mechanism uses stopper thread which going to call
select_fallback_rqand use this mechanism to pick only a preferred CPU.When task is affined only to non-preferred CPUs it should continue to run there. Detect that by checking if
cpus_ptrandcpu_preferred_maskintersect or not.Since
is_cpu_allowedcan be called directly or repeatedly inselect_fallback_rq, encode the info intask_struct->has_preferred_cpu_stateif the path is viaselect_fallback_rqor not.
This helps to avoid N**2 complexity for the rare cases.Additional overhead of O(N) comes to
is_cpu_allowedonly when cpu is not preferred. So in normal scenarios overhead is only a bit check.Signed-off-by Shrikanth Hegde <[email protected]>---
v4->v5:
- Do simple encoding of -1,0,1 instead (K Prateek Nayak)
- Make it s8 (K Prateek Nayak)
- Update changelog to address sashiko concerns of overhead.include/linux/sched.h | 1 + kernel/sched/core.c | 35 +++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 25 +++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 2 deletions(-)include/linux/sched.h +0
-0
Quoted 9 lines
include/linux/sched.h struct task_struct { · line 1660
Quoted 7 lines
include/linux/sched.h +1
-0@@ -1657,6 +1657,7 @@ struct task_struct {#ifdef CONFIG_UNWIND_USER struct unwind_task_info unwind_info;#endif +s8 has_preferred_cpu_state;Why not protected with the config?
Ok. I will add, i thought it would mean too many ifdefs due to usage in the below function.
Quoted 3 lines
It looks like you didn't ever ran pahole on it. Maybe it's worth to try now?
I did, This is what i saw in powerpc. It did fit in the available cacheline.
struct bpf_net_context * bpf_net_context; /* 4736 8 */ struct llist_head kretprobe_instances; /* 4744 8 */ struct llist_head rethooks; /* 4752 8 */ union rv_task_monitor rv[2]; /* 4760 16 */ s8 has_preferred_cpu_state; /* 4776 1 */
/* XXX 7 bytes hole, try to pack */ struct thread_struct thread; /* 4784 2864 */
Quoted 86 lines
Quoted 77 lines
include/linux/sched.h +0
-0/* CPU-specific state of this task: */ struct thread_struct thread;kernel/sched/core.c +30
-2@@ -2500,6 +2500,8 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)*/ static inline bool is_cpu_allowed (struct task_struct *p,int cpu) { +bool task_check_preferred_cpu; +/* When not in the task's cpumask, no point in looking further. */ if (!task_allowed_on_cpu (p, cpu))return false;@@ -2508,9 +2510,23 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (is_migration_disabled (p))return cpu_online (cpu); +/* +* This is essential to maintain user affinities when preferred +* CPUs change. A task pinned on non-preferred CPU should continue +* to run there, since this is non-user triggered. +* +* If CPU is non-preferred and task can run on other CPUs which are +* currently preferred, then choose those other CPUs instead. +* Overhead is minimal when CPU is preferred. +*/ + task_check_preferred_cpu = !cpu_preferred (cpu) &&task_has_preferred_cpus (p); +/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +if (task_check_preferred_cpu) +return false;return cpu_active (cpu); + }/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))@@ -2520,6 +2536,10 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (cpu_dying (cpu))return false; +/* Try on preferred CPU first if possible*/ +if (task_check_preferred_cpu) +return false; +/* But are allowed during online. */ return cpu_online (cpu); }@@ -3549,6 +3569,14 @@ static int select_fallback_rq(int cpu, struct task_struct *p)enum { cpuset, possible, fail } state = cpuset;int dest_cpu; +/* +* Cache the value whether task's affinity spans preferred CPUs. +* This helps to avoid repeating the same for each CPU +* later in the loop. Encode call to is_cpu_allowed coming +* via select_fallback_rq. +*/ + p->has_preferred_cpu_state =task_has_preferred_cpus (p) ?1 :-1 ; +/* * If the node that the CPU is on has been offlined, cpu_to_node() * will return -1. There is no CPU on the node, and we should @@ -3560,7 +3588,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* Look for allowed, online CPU in same node. */ for_each_cpu (dest_cpu, nodemask) {if (is_cpu_allowed (p, dest_cpu)) -return dest_cpu; + goto clear_and_return; } }kernel/sched/core.c static int select_fallback_rq(int cpu, struct task_struct *p) · line 3636
Quoted 5 lines
kernel/sched/core.c +2
-0@@ -3604,6 +3632,8 @@ static int select_fallback_rq(int cpu, struct task_struct *p) } } +clear_and_return: + p->has_preferred_cpu_state =0 ;Sadly, you've ignored my comments from the previous round. Let me repeat it once again:
Quoted 3 lines
This ->has_preferred_cpu_state is always zero out of the scope of the function. It means, it's a local variable, and should not belong to the
task_struct.
Ok. Making it as another variable is better. I will make change accordingly.
Quoted 20 lines
Quoted 20 lines
kernel/sched/core.c +1
-0return dest_cpu; }@@ -4612,6 +4642,7 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p)init_numa_balancing (clone_flags, p); p->wake_entry.u_flags =CSD_TYPE_TTWU ; p->migration_pending =NULL ; + p->has_preferred_cpu_state =0 ;init_sched_mm (p); }kernel/sched/sched.h +0
-0@@ -4213,4 +4213,29 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)#include "ext.h"
Quoted 12 lines
kernel/sched/sched.h DEFINE_CLASS_IS_UNCONDITIONAL(sched_change) · line 4223
Quoted 8 lines
kernel/sched/sched.h +8
-0+/* +* has_preferred_cpu_state could have the value cached from +* select_fallback_rq. It is set/cleared while holding pi_lock +* and irq disabled. +* +* 1: Cached and preferred CPUs exists in task's affinity. +* 0: Not cached and need to evaluate. +* -1: Cached and preferred CPU doesn't exits task's affinity So, you've got 3 options to declare the status: self-explaining enum, self-explaining #defines, and this random numbers explained in comment. The latter option is the worst to me.
ok. I will define the enums.
Quoted 6 lines
And you didn't provide any benchmark advocating this caching optimization.
Sorry, but NAK.
If we move to local variable then this won;t be necessary, just enum's would be enough (I think). Let me go stare at it.
Re: [PATCH v5 06/24] sched/core: allow only preferred CPUs in is_cpu_allowed
Hi Yury.
Quoted 2 lines · Shrikanth Hegde
Hi Yury. Thanks for going through the patches.
[...]
Quoted 9 lines
Quoted 3 lines
So, you've got 3 options to declare the status: self-explaining enum, self-explaining #defines, and this random numbers explained in comment. The latter option is the worst to me.
ok. I will define the enums.
Quoted 3 lines
And you didn't provide any benchmark advocating this caching optimization.
I did below to see. Made interval as 100ms.
Ran ./hackbench 30 process 30000 loops in both the VM at the same time. Values are average of 5 runs.
With optimization:
13.6 seconds
Without optimization:
13.8 seconds
Quoted 6 lines
Quoted 3 lines
Sorry, but NAK.
If we move to local variable then this won;t be necessary, just enum's would be enough (I think). Let me go stare at it.
I have made it use the local variable instead. There maybe better names for variable, put something quickly to check the idea. Effectively this PATCH 6 becomes:
Does this seems better?
Please let me know your comments.
---
kernel/sched/core.c +37 -8
@@ -2498,8 +2498,10 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)* Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq(). */ -static inline bool is_cpu_allowed (struct task_struct *p,int cpu) +static inline bool is_cpu_allowed (struct task_struct *p,int cpu,int cached) { +bool task_check_preferred_cpu; +/* When not in the task's cpumask, no point in looking further. */ if (!task_allowed_on_cpu (p, cpu))return false;
@@ -2508,9 +2510,24 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (is_migration_disabled (p))return cpu_online (cpu); +/* +* This is essential to maintain user affinities when preferred +* CPUs change. A task pinned on non-preferred CPU should continue +* to run there, since this is non-user triggered. +* +* If CPU is non-preferred and task can run on other CPUs which are +* currently preferred, then choose those other CPUs instead. +* Overhead is minimal when CPU is preferred. +*/ + task_check_preferred_cpu = !cpu_preferred (cpu) && +task_has_preferred_cpus (p, cached); +/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +if (task_check_preferred_cpu) +return false;return cpu_active (cpu); + }/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))
@@ -2520,6 +2537,10 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (cpu_dying (cpu))return false; +/* Try on preferred CPU first if possible*/ +if (task_check_preferred_cpu) +return false; +/* But are allowed during online. */ return cpu_online (cpu); }
@@ -2595,7 +2616,7 @@ static struct rq *__migrate_task(struct rq *rq, struct rq_flags *rf,__must_hold (__rq_lockp (rq )) {/* Affinity changed (again). */ -if (!is_cpu_allowed (p, dest_cpu)) +if (!is_cpu_allowed (p, dest_cpu,NO_CACHED_VAL ))return rq; rq =move_queued_task (rq, rf, p, dest_cpu);
@@ -3547,7 +3568,15 @@ static int select_fallback_rq(int cpu, struct task_struct *p)int nid =cpu_to_node (cpu);const struct cpumask *nodemask =NULL ;enum { cpuset, possible, fail } state = cpuset; -int dest_cpu; +int dest_cpu, has_preferred_cpu; + +/* +* Cache the value whether task's affinity spans preferred CPUs. +* This helps to avoid repeating the same for each CPU +* later in the loop. +*/ + has_preferred_cpu =task_has_preferred_cpus (p,NO_CACHED_VAL ) ? +TASK_HAS_PREFERRED_CPUS :TASK_NO_PREFERRED_CPUS ;/* * If the node that the CPU is on has been offlined, cpu_to_node()
@@ -3559,7 +3588,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* Look for allowed, online CPU in same node. */ for_each_cpu (dest_cpu, nodemask) { -if (is_cpu_allowed (p, dest_cpu)) +if (is_cpu_allowed (p, dest_cpu, has_preferred_cpu))return dest_cpu; } }
@@ -3567,7 +3596,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)for (;;) {/* Any allowed, online CPU? */ for_each_cpu (dest_cpu, p->cpus_ptr) { -if (!is_cpu_allowed (p, dest_cpu)) +if (!is_cpu_allowed (p, dest_cpu, has_preferred_cpu))continue ; goto out;
@@ -3632,7 +3661,7 @@ int select_task_rq(struct task_struct *p, int cpu, int *wake_flags)* [ this allows ->select_task() to simply return task_cpu(p) and * not worry about this generic constraint ] */ -if (unlikely (!is_cpu_allowed (p, cpu))) +if (unlikely (!is_cpu_allowed (p, cpu,NO_CACHED_VAL ))) cpu =select_fallback_rq (task_cpu (p), p);return cpu;
@@ -6467,7 +6496,7 @@ static bool try_steal_cookie(int this, int that)if (p == src->core_pick || p == src->curr) goto next; -if (!is_cpu_allowed (p, this)) +if (!is_cpu_allowed (p, this,NO_CACHED_VAL )) goto next;if (p->core_occupation > dst->idle->core_occupation)
kernel/sched/sched.h +28 -0
@@ -4213,4 +4213,32 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)#include "ext.h" +enum task_preferred_cached { +TASK_NO_PREFERRED_CPUS =-1 , +NO_CACHED_VAL , +TASK_HAS_PREFERRED_CPUS , +}; + +/* +* Value is cached when called via select_fallback_rq(). +* +* TASK_NO_PREFERRED_CPUS : Cached and preferred CPUs exists in task's +* affinity. +* NO_CACHED_VAL: Not cached and need to evaluate. +* TASK_HAS_PREFERRED_CPUS: Cached and preferred CPU doesn't exits +* task's affinity +* +* Only affects FAIR task. +*/ +static inline bool task_has_preferred_cpus (struct task_struct *p,int cached) +{ +/* Only FAIR tasks honor preferred CPU state */ +if (unlikely (p->sched_class != &fair_sched_class)) +return false; + +if (cached) +return cached >0 ; +else +return cpumask_intersects (p->cpus_ptr, cpu_preferred_mask); +}#endif /* _KERNEL_SCHED_SCHED_H */
Yury Norov
maintainerRe: [PATCH v5 06/24] sched/core: allow only preferred CPUs in is_cpu_allowed
Quoted 23 lines · Shrikanth Hegde
Hi Yury.
Quoted 2 lines · Shrikanth Hegde
Hi Yury. Thanks for going through the patches.
[...]
Quoted 9 lines
Quoted 3 lines
So, you've got 3 options to declare the status: self-explaining enum, self-explaining #defines, and this random numbers explained in comment. The latter option is the worst to me.
ok. I will define the enums.
Quoted 3 lines
And you didn't provide any benchmark advocating this caching optimization.
I did below to see. Made interval as 100ms.
Ran ./hackbench 30 process 30000 loops in both the VM at the same time. Values are average of 5 runs.
Quoted 5 lines
With optimization:
13.6 secondsWithout optimization:
13.8 seconds
And what's the p-value for them?
...
Quoted 9 lines
Quoted 2 lines
If we move to local variable then this won;t be necessary, just enum's would be enough (I think). Let me go stare at it.
I have made it use the local variable instead. There maybe better names for variable, put something quickly to check the idea. Effectively this PATCH 6 becomes:
Does this seems better?
Please let me know your comments.
I think, the below is too massive change for optimization of an optimization for a particular, not too common config. And the whole improvement is ~1%, assuming it's statistically important...
Just as said on previous round. Please order your series such that the core logic goes first, and all sorts of complications, like this optimization, are appended at the end.
Quoted 53 lines
---
kernel/sched/core.c +23
-2@@ -2498,8 +2498,10 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)* Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq(). */ -static inline bool is_cpu_allowed (struct task_struct *p,int cpu) +static inline bool is_cpu_allowed (struct task_struct *p,int cpu,int cached) { +bool task_check_preferred_cpu; +/* When not in the task's cpumask, no point in looking further. */ if (!task_allowed_on_cpu (p, cpu))return false;@@ -2508,9 +2510,24 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (is_migration_disabled (p))return cpu_online (cpu); +/* +* This is essential to maintain user affinities when preferred +* CPUs change. A task pinned on non-preferred CPU should continue +* to run there, since this is non-user triggered. +* +* If CPU is non-preferred and task can run on other CPUs which are +* currently preferred, then choose those other CPUs instead. +* Overhead is minimal when CPU is preferred. +*/ + task_check_preferred_cpu = !cpu_preferred (cpu) && +task_has_preferred_cpus (p, cached); +/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +if (task_check_preferred_cpu) +return false;return cpu_active (cpu); + }/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))@@ -2520,6 +2537,10 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (cpu_dying (cpu))return false; +/* Try on preferred CPU first if possible*/ +if (task_check_preferred_cpu) +return false; +/* But are allowed during online. */ return cpu_online (cpu); }
kernel/sched/core.c static struct rq *__migrate_task(struct rq *rq, struct rq_flags *rf, · line 2620
Quoted 6 lines
kernel/sched/core.c +1
-1@@ -2595,7 +2616,7 @@ static struct rq *__migrate_task(struct rq *rq, struct rq_flags *rf,__must_hold (__rq_lockp (rq )) {/* Affinity changed (again). */ -if (!is_cpu_allowed (p, dest_cpu)) +if (!is_cpu_allowed (p, dest_cpu,NO_CACHED_VAL ))return rq;
This thing I really dislike. The unrelated code should not be affected. You can make it less visually invasive with:
#define is_cpu_allowed(p, cpu) __is_cpu_allowed(p, cpu, NO_CACHED_VAL)
Please reconsider your code to have the changes better localized.
Thanks,
Yury
Quoted 83 lines
kernel/sched/core.c +13
-5rq =move_queued_task (rq, rf, p, dest_cpu);@@ -3547,7 +3568,15 @@ static int select_fallback_rq(int cpu, struct task_struct *p)int nid =cpu_to_node (cpu);const struct cpumask *nodemask =NULL ;enum { cpuset, possible, fail } state = cpuset; -int dest_cpu; +int dest_cpu, has_preferred_cpu; + +/* +* Cache the value whether task's affinity spans preferred CPUs. +* This helps to avoid repeating the same for each CPU +* later in the loop. +*/ + has_preferred_cpu =task_has_preferred_cpus (p,NO_CACHED_VAL ) ? +TASK_HAS_PREFERRED_CPUS :TASK_NO_PREFERRED_CPUS ;/* * If the node that the CPU is on has been offlined, cpu_to_node() @@ -3559,7 +3588,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* Look for allowed, online CPU in same node. */ for_each_cpu (dest_cpu, nodemask) { -if (is_cpu_allowed (p, dest_cpu)) +if (is_cpu_allowed (p, dest_cpu, has_preferred_cpu))return dest_cpu; } }@@ -3567,7 +3596,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)for (;;) {/* Any allowed, online CPU? */ for_each_cpu (dest_cpu, p->cpus_ptr) { -if (!is_cpu_allowed (p, dest_cpu)) +if (!is_cpu_allowed (p, dest_cpu, has_preferred_cpu))continue ; goto out;@@ -3632,7 +3661,7 @@ int select_task_rq(struct task_struct *p, int cpu, int *wake_flags)* [ this allows ->select_task() to simply return task_cpu(p) and * not worry about this generic constraint ] */ -if (unlikely (!is_cpu_allowed (p, cpu))) +if (unlikely (!is_cpu_allowed (p, cpu,NO_CACHED_VAL ))) cpu =select_fallback_rq (task_cpu (p), p);return cpu;@@ -6467,7 +6496,7 @@ static bool try_steal_cookie(int this, int that)if (p == src->core_pick || p == src->curr) goto next; -if (!is_cpu_allowed (p, this)) +if (!is_cpu_allowed (p, this,NO_CACHED_VAL )) goto next;if (p->core_occupation > dst->idle->core_occupation)kernel/sched/sched.h +28
-0@@ -4213,4 +4213,32 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)#include "ext.h" +enum task_preferred_cached { +TASK_NO_PREFERRED_CPUS =-1 , +NO_CACHED_VAL , +TASK_HAS_PREFERRED_CPUS , +}; + +/* +* Value is cached when called via select_fallback_rq(). +* +* TASK_NO_PREFERRED_CPUS : Cached and preferred CPUs exists in task's +* affinity. +* NO_CACHED_VAL: Not cached and need to evaluate. +* TASK_HAS_PREFERRED_CPUS: Cached and preferred CPU doesn't exits +* task's affinity +* +* Only affects FAIR task. +*/ +static inline bool task_has_preferred_cpus (struct task_struct *p,int cached) +{ +/* Only FAIR tasks honor preferred CPU state */ +if (unlikely (p->sched_class != &fair_sched_class)) +return false; + +if (cached) +return cached >0 ; +else +return cpumask_intersects (p->cpus_ptr, cpu_preferred_mask); +}#endif /* _KERNEL_SCHED_SCHED_H */
Re: [PATCH v5 06/24] sched/core: allow only preferred CPUs in is_cpu_allowed
Hi Yury.
Quoted 44 lines · Yury Norov
Quoted 23 lines · Shrikanth Hegde
Hi Yury.
Quoted 2 lines · Shrikanth Hegde
Hi Yury. Thanks for going through the patches.
[...]
Quoted 9 lines
Quoted 3 lines
So, you've got 3 options to declare the status: self-explaining enum, self-explaining #defines, and this random numbers explained in comment. The latter option is the worst to me.
ok. I will define the enums.
Quoted 3 lines
And you didn't provide any benchmark advocating this caching optimization.
I did below to see. Made interval as 100ms.
Ran ./hackbench 30 process 30000 loops in both the VM at the same time. Values are average of 5 runs.Quoted 5 lines
With optimization:
13.6 secondsWithout optimization:
13.8 secondsAnd what's the p-value for them?
...
Quoted 9 lines
Quoted 2 lines
If we move to local variable then this won;t be necessary, just enum's would be enough (I think). Let me go stare at it.
I have made it use the local variable instead. There maybe better names for variable, put something quickly to check the idea. Effectively this PATCH 6 becomes:
Does this seems better?
Please let me know your comments.
Quoted 7 lines
I think, the below is too massive change for optimization of an optimization for a particular, not too common config. And the whole improvement is ~1%, assuming it's statistically important...
Just as said on previous round. Please order your series such that the core logic goes first, and all sorts of complications, like this optimization, are appended at the end.
Ok. I will split it up into two patches.
One without any optimization but with comment explaining the rare case of N**2. Second one at the end of the series, with a patch do the optimization.
Quoted 64 lines
Quoted 54 lines
---
kernel/sched/core.c +23
-2@@ -2498,8 +2498,10 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)* Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq(). */ -static inline bool is_cpu_allowed (struct task_struct *p,int cpu) +static inline bool is_cpu_allowed (struct task_struct *p,int cpu,int cached) { +bool task_check_preferred_cpu; +/* When not in the task's cpumask, no point in looking further. */ if (!task_allowed_on_cpu (p, cpu))return false;@@ -2508,9 +2510,24 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (is_migration_disabled (p))return cpu_online (cpu); +/* +* This is essential to maintain user affinities when preferred +* CPUs change. A task pinned on non-preferred CPU should continue +* to run there, since this is non-user triggered. +* +* If CPU is non-preferred and task can run on other CPUs which are +* currently preferred, then choose those other CPUs instead. +* Overhead is minimal when CPU is preferred. +*/ + task_check_preferred_cpu = !cpu_preferred (cpu) && +task_has_preferred_cpus (p, cached); +/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +if (task_check_preferred_cpu) +return false;return cpu_active (cpu); + }/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))@@ -2520,6 +2537,10 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (cpu_dying (cpu))return false; +/* Try on preferred CPU first if possible*/ +if (task_check_preferred_cpu) +return false; +/* But are allowed during online. */ return cpu_online (cpu); }kernel/sched/core.c static struct rq *__migrate_task(struct rq *rq, struct rq_flags *rf, · line 2620
Quoted 6 lines
kernel/sched/core.c +1
-1@@ -2595,7 +2616,7 @@ static struct rq *__migrate_task(struct rq *rq, struct rq_flags *rf,__must_hold (__rq_lockp (rq )) {/* Affinity changed (again). */ -if (!is_cpu_allowed (p, dest_cpu)) +if (!is_cpu_allowed (p, dest_cpu,NO_CACHED_VAL ))return rq;This thing I really dislike. The unrelated code should not be affected. You can make it less visually invasive with:
Quoted 7 lines
#define is_cpu_allowed(p, cpu) __is_cpu_allowed(p, cpu, NO_CACHED_VAL)Please reconsider your code to have the changes better localized.
Thanks,
Yury
That was typed out too fast. I did refactor something like that later. But i will split this into twp patches as said above.
kernel/sched/core.c +40 -5
@@ -2498,8 +2498,11 @@ static inline bool rq_has_pinned_tasks(struct rq *rq)* Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq(). */ -static inline bool is_cpu_allowed (struct task_struct *p,int cpu) +static inline bool __is_cpu_allowed (struct task_struct *p,int cpu, +int pref_state) { +bool task_check_preferred_cpu; +/* When not in the task's cpumask, no point in looking further. */ if (!task_allowed_on_cpu (p, cpu))return false;
@@ -2508,9 +2511,24 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (is_migration_disabled (p))return cpu_online (cpu); +/* +* This is essential to maintain user affinities when preferred +* CPUs change. A task pinned on non-preferred CPU should continue +* to run there, since this is non-user triggered. +* +* If CPU is non-preferred and task can run on other CPUs which are +* currently preferred, then choose those other CPUs instead. +* Overhead is minimal when CPU is preferred. +*/ + task_check_preferred_cpu = !cpu_preferred (cpu) && +task_has_preferred_cpus (p, pref_state); +/* Non kernel threads are not allowed during either online or offline. */ -if (!(p->flags &PF_KTHREAD )) +if (!(p->flags &PF_KTHREAD )) { +if (task_check_preferred_cpu) +return false;return cpu_active (cpu); + }/* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu (p))
@@ -2520,10 +2538,19 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)if (cpu_dying (cpu))return false; +/* Try on preferred CPU first if possible*/ +if (task_check_preferred_cpu) +return false; +/* But are allowed during online. */ return cpu_online (cpu); } +static inline bool is_cpu_allowed (struct task_struct *p,int cpu) +{ +return __is_cpu_allowed (p, cpu,PREFERRED_CPU_UNKNOWN ); +} +/* * This is how migration works: *
@@ -3547,7 +3574,15 @@ static int select_fallback_rq(int cpu, struct task_struct *p)int nid =cpu_to_node (cpu);const struct cpumask *nodemask =NULL ;enum { cpuset, possible, fail } state = cpuset; -int dest_cpu; +int dest_cpu, pref_state; + +/* +* Cache the value whether task's affinity spans preferred CPUs. +* This helps to avoid repeating the same for each CPU +* later in the loop. +*/ + pref_state =task_has_preferred_cpus (p,PREFERRED_CPU_UNKNOWN ) ? +PREFERRED_CPU_EXISTS :PREFERRED_CPU_NONE ;/* * If the node that the CPU is on has been offlined, cpu_to_node()
@@ -3559,7 +3594,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)/* Look for allowed, online CPU in same node. */ for_each_cpu (dest_cpu, nodemask) { -if (is_cpu_allowed (p, dest_cpu)) +if (__is_cpu_allowed (p, dest_cpu, pref_state))return dest_cpu; } }
@@ -3567,7 +3602,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p)for (;;) {/* Any allowed, online CPU? */ for_each_cpu (dest_cpu, p->cpus_ptr) { -if (!is_cpu_allowed (p, dest_cpu)) +if (!__is_cpu_allowed (p, dest_cpu, pref_state))continue ; goto out;
kernel/sched/sched.h +29 -0
@@ -4213,4 +4213,33 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)#include "ext.h" +/* +* PREFERRED_CPU_UNKNOWN: Unknown and need to evaluate. +* PREFERRED_CPU_NONE : Known and No preferred CPUs exists in task's affinity. +* PREFERRED_CPU_EXISTS: Known and preferred CPU exists in task's affinity. +*/ + +enum task_preferred_cached { +PREFERRED_CPU_UNKNOWN , +PREFERRED_CPU_NONE , +PREFERRED_CPU_EXISTS , +}; + +/* +* Value is known when called via select_fallback_rq(). This helps to +* avoid calling cpumask_intersects repeatedly in the loop. +* +* Only affects FAIR task. +*/ +static inline bool task_has_preferred_cpus (struct task_struct *p,int pref_state) +{ +/* Only FAIR tasks honor preferred CPU state */ +if (unlikely (p->sched_class != &fair_sched_class)) +return false; + +if (pref_state !=PREFERRED_CPU_UNKNOWN ) +return pref_state ==PREFERRED_CPU_EXISTS ; + +return cpumask_intersects (p->cpus_ptr, cpu_preferred_mask); +}#endif /* _KERNEL_SCHED_SCHED_H */
Re: [PATCH v5 06/24] sched/core: allow only preferred CPUs in is_cpu_allowed
Hi Yury, Prateek,
Quoted 231 lines · Shrikanth Hegde
Hi Yury.
Quoted 3 lines
Just as said on previous round. Please order your series such that the core logic goes first, and all sorts of complications, like this optimization, are appended at the end.
Ok. I will split it up into two patches.
One without any optimization but with comment explaining the rare case of N**2. Second one at the end of the series, with a patch do the optimization.
Quoted 72 lines
Quoted 60 lines
---
kernel/sched/core.c +3
-1@@ -2498,8 +2498,10 @@ static inline bool rq_has_pinned_tasks(structrq *rq) * Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq(). */ -static inline bool is_cpu_allowed (struct task_struct *p,int cpu) +static inline bool is_cpu_allowed (struct task_struct *p,int cpu,int cached) { + bool task_check_preferred_cpu; + /* When not in the task's cpumask, no point in looking further. */if (!task_allowed_on_cpu(p, cpu))
return false;kernel/sched/core.c +3
-0@@ -2508,9 +2510,24 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu) if (is_migration_disabled(p)) return cpu_online(cpu); + /* +* This is essential to maintain user affinities when preferred +* CPUs change. A task pinned on non-preferred CPU should continue
+ * to run there, since this is non-user triggered.
+ *
+ * If CPU is non-preferred and task can run on other CPUs which are
+ * currently preferred, then choose those other CPUs instead. + * Overhead is minimal when CPU is preferred.
+ */
+task_check_preferred_cpu= !cpu_preferred(cpu) &&
+task_has_preferred_cpus(p, cached); +
/* Non kernel threads are not allowed during either online or offline. */ - if (!(p->flags &PF_KTHREAD))
+ if (!(p->flags &PF_KTHREAD)) {
+ if (task_check_preferred_cpu)
+ return false;
returncpu_active(cpu);
+ }
/*KTHREAD_IS_PER_CPUis always allowed. */
if (kthread_is_per_cpu(p))kernel/sched/core.c +4
-0@@ -2520,6 +2537,10 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu) if (cpu_dying(cpu)) return false; + /* Try on preferred CPU first if possible*/ + if (task_check_preferred_cpu) + return false; + /* But are allowed during online. */ return cpu_online(cpu); }kernel/sched/core.c static struct rq *__migrate_task(struct rq *rq, · line 2621
Quoted 7 lines
kernel/sched/core.c +1
-1@@ -2595,7 +2616,7 @@ static struct rq *__migrate_task(struct rq *rq, struct rq_flags *rf, __must_hold(__rq_lockp(rq)) { /* Affinity changed (again). */ - if (!is_cpu_allowed(p, dest_cpu)) + if (!is_cpu_allowed(p, dest_cpu, NO_CACHED_VAL)) return rq;This thing I really dislike. The unrelated code should not be affected. You can make it less visually invasive with:
#defineis_cpu_allowed(p, cpu)__is_cpu_allowed(p, cpu, NO_CACHED_VAL)Quoted 5 lines
Please reconsider your code to have the changes better localized.
Thanks,
YuryThat was typed out too fast. I did refactor something like that later. But i will split this into twp patches as said above.
kernel/sched/core.c +4
-1@@ -2498,8 +2498,11 @@ static inline bool rq_has_pinned_tasks(struct rq*rq) * Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq(). */ -static inline bool is_cpu_allowed (struct task_struct *p,int cpu) +static inline bool __is_cpu_allowed (struct task_struct *p,int cpu, + int pref_state) { + bool task_check_preferred_cpu; + /* When not in the task's cpumask, no point in looking further. */ if (!task_allowed_on_cpu(p, cpu))return false;
kernel/sched/core.c +12
-0@@ -2508,9 +2511,24 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu) if (is_migration_disabled(p)) return cpu_online(cpu); + /* + * This is essential to maintain user affinities when preferred + * CPUs change. A task pinned on non-preferred CPU should continue + * to run there, since this is non-user triggered. + * + * If CPU is non-preferred and task can run on other CPUs which are + * currently preferred, then choose those other CPUs instead. + * Overhead is minimal when CPU is preferred. + */ + task_check_preferred_cpu = !cpu_preferred(cpu) && + task_has_preferred_cpus(p, pref_state); + /* Non kernel threads are not allowed during either online oroffline. */
- if (!(p->flags &PF_KTHREAD))
+ if (!(p->flags &PF_KTHREAD)) {
+ if (task_check_preferred_cpu)
+ return false;
returncpu_active(cpu);
+ }/*
KTHREAD_IS_PER_CPUis always allowed. */ if (kthread_is_per_cpu(p))kernel/sched/core.c +9
-0@@ -2520,10 +2538,19 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu) if (cpu_dying(cpu)) return false; + /* Try on preferred CPU first if possible*/ + if (task_check_preferred_cpu) + return false; + /* But are allowed during online. */ return cpu_online(cpu); } +static inline bool is_cpu_allowed (struct task_struct *p,int cpu) +{ + return __is_cpu_allowed(p, cpu, PREFERRED_CPU_UNKNOWN); +} + /** This is how migration works: *
kernel/sched/core.c +9
-1@@ -3547,7 +3574,15 @@ static int select_fallback_rq(int cpu, struct task_struct *p) int nid = cpu_to_node(cpu); const struct cpumask *nodemask = NULL; enum { cpuset, possible, fail } state = cpuset; - int dest_cpu; + int dest_cpu, pref_state; + + /* + * Cache the value whether task's affinity spans preferred CPUs. + * This helps to avoid repeating the same for each CPU + * later in the loop. + */ + pref_state = task_has_preferred_cpus(p, PREFERRED_CPU_UNKNOWN) ? + PREFERRED_CPU_EXISTS : PREFERRED_CPU_NONE; /** If the node that the CPU is on has been offlined,
cpu_to_node()kernel/sched/core.c +1
-1@@ -3559,7 +3594,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p) /* Look for allowed, online CPU in same node. */ for_each_cpu(dest_cpu, nodemask) { - if (is_cpu_allowed(p, dest_cpu)) + if (__is_cpu_allowed(p, dest_cpu, pref_state)) return dest_cpu; }}
kernel/sched/core.c +1
-1@@ -3567,7 +3602,7 @@ static int select_fallback_rq(int cpu, struct task_struct *p) for (;;) { /* Any allowed, online CPU? */ for_each_cpu(dest_cpu, p->cpus_ptr) { - if (!is_cpu_allowed(p, dest_cpu)) + if (!__is_cpu_allowed(p, dest_cpu, pref_state)) continue;goto out;
kernel/sched/sched.h +3
-0@@ -4213,4 +4213,33 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change) #include "ext.h" +/* +* PREFERRED_CPU_UNKNOWN: Unknown and need to evaluate. +* PREFERRED_CPU_NONE : Known and No preferred CPUs exists in task's affinity.
+ *PREFERRED_CPU_EXISTS: Known and preferred CPU exists in task's affinity.
+ */kernel/sched/sched.h +14
-0+ +enum task_preferred_cached { + PREFERRED_CPU_UNKNOWN, + PREFERRED_CPU_NONE, + PREFERRED_CPU_EXISTS, +}; + +/* +* Value is known when called via select_fallback_rq(). This helps to +* avoid calling cpumask_intersects repeatedly in the loop. +* +* Only affects FAIR task. +*/ +static inline bool task_has_preferred_cpus(struct task_struct *p, int
pref_state)kernel/sched/sched.h +1
-0+{
kernel/sched/sched.h DEFINE_CLASS_IS_UNCONDITIONAL(sched_change) · line 4245
Quoted 10 lines
kernel/sched/sched.h +9
-0+ /* Only FAIR tasks honor preferred CPU state */ + if (unlikely(p->sched_class != &fair_sched_class)) + return false; + + if (pref_state != PREFERRED_CPU_UNKNOWN) + return pref_state == PREFERRED_CPU_EXISTS; + + return cpumask_intersects(p->cpus_ptr, cpu_preferred_mask); +} #endif /* _KERNEL_SCHED_SCHED_H */
I was thinking caching the value can cause affinity to be reset, because there is no protection of mask change within the loop of select_fallback_rq. So i did some testing, and
I hit a case in practice with 10ms as interval with explicit affinities on overlapping preferred and non-preferred CPUs and hotplugging specific CPUs. caching exposes this race
Lets say task affined to 464-479 and preferred mask is 0-471 (472-479 is non preferred)
CPU468 CPU0select_fallback_rq
- pref_state = PREFERRED_CPU_EXISTS
(Now preferred_mask becomes 0-463, but (Changes preferred to 0-463)
before any further call to is_cpu_allowed is made)
- is_cpu_allowed tries to find a preferred CPU since cached state says one exists.
- is_cpu_allowed is called twice (once on nodemask,
and once on p->cpus_ptr) but cached state remains the
same.- no CPU found, fallback to reset to possible CPUs.
Without cached state, there is evaluation on each !preferred CPU and such race isn;t possible between two calls. So chance of race is extremely rare if non-existent.
I couldn't hit the same race in any permutations I tried.
Even if we take task was affined to only one CPU and mask changed between &&. !cpu_preferred(cpu) && task_has_preferred_cpus(p);
Two cases.
Case 1: cpu was marked as preferred and after && it got removed from preferred_mask. In that case task may end up on non-preferred CPU and it gets pushed out if possible. No reset of its affinity.
Case 2: cpu was non-preferred and now it became preferred after && Now cpumask_intersects will be true true and task_has_preferred_cpus is true as well, but this CPU will be skipped. But second call in select_fallback_rq will ensure it returns the since cpu_preferred check will succeed. Since select_fallback_rq is called on task_cpu(p) where it previously ran, first check may fail due to race, but second one can't since evaluation between the two can't be more than 1ms.
* So I will drop this optimization of caching the state for now *
PATCH v5 7/24 sched/fair: Select preferred CPU at wakeup when possible
In reply to Shrikanth HegdeUpdate available_idle_cpu to consider preferred CPUs. This takes care of lot of decisions at wakeup to use only preferred CPUs. There is no need to put those explicit checks everywhere.
--- kernel/sched/sched.h | 3 +++ 1 file changed, 3 insertions(+)
kernel/sched/sched.h +3 -0
@@ -1434,6 +1434,9 @@ static inline bool available_idle_cpu(int cpu)if (!idle_rq (cpu_rq (cpu)))return 0 ; +if (!cpu_preferred (cpu)) +return 0 ; +if (vcpu_is_preempted (cpu))return 0 ;
Signature · 1 line
2.47.3
Peter Zijlstra
maintainerRe: [PATCH v5 07/24] sched/fair: Select preferred CPU at wakeup when possible
Quoted 14 lines · Shrikanth Hegde
Update
available_idle_cputo consider preferred CPUs. This takes care of lot of decisions at wakeup to use only preferred CPUs. There is no need to put those explicit checks everywhere.Signed-off-by Shrikanth Hegde <[email protected]>--- kernel/sched/sched.h | 3 +++ 1 file changed, 3 insertions(+)kernel/sched/sched.h +0
-0
kernel/sched/sched.h static inline bool available_idle_cpu(int cpu) · line 1442
Quoted 9 lines
kernel/sched/sched.h +3
-0@@ -1434,6 +1434,9 @@ static inline bool available_idle_cpu(int cpu)if (!idle_rq (cpu_rq (cpu)))return 0 ; +if (!cpu_preferred (cpu)) +return 0 ; +if (vcpu_is_preempted (cpu))return 0 ;
This one might hurt, it is a whole extra cacheline in otherwise already sensitive (wakeup) paths.
Re: [PATCH v5 07/24] sched/fair: Select preferred CPU at wakeup when possible
Hi Peter, Thank you very much for going through the patches.
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Shrikanth Hegde
Update
available_idle_cputo consider preferred CPUs. This takes care of lot of decisions at wakeup to use only preferred CPUs. There is no need to put those explicit checks everywhere.Signed-off-by Shrikanth Hegde <[email protected]>--- kernel/sched/sched.h | 3 +++ 1 file changed, 3 insertions(+)kernel/sched/sched.h +0
-0@@ -1434,6 +1434,9 @@ static inline bool available_idle_cpu(int cpu)if (!idle_rq (cpu_rq (cpu)))return 0 ;
Quoted 10 lines
kernel/sched/sched.h static inline bool available_idle_cpu(int cpu) · line 1442
Quoted 6 lines
kernel/sched/sched.h +3
-0+if (!cpu_preferred (cpu)) +return 0 ; +if (vcpu_is_preempted (cpu))return 0 ;This one might hurt, it is a whole extra cacheline in otherwise already sensitive (wakeup) paths.
Yes, this could be costly. If wakeup returns a non-preferred CPU, is_cpu_allowed would catch it.
So, i think we can avoid repeated computation of it in available_idle_cpu.
Let me see if removing it still achieves the functionality of moving out fast enough and numbers are close enough to with it.
PATCH v5 8/24 sched/fair: load balance only among preferred CPUs
In reply to Shrikanth HegdeConsider only preferred CPUs for load balance.
With this, load balance will end up choosing a preferred CPUs for pull. This makes it not fight against the push task mechanism which happens at tick. Also, this stops active balance to happen on non-preferred CPU pulling the load.
This means there is no load balancing if the task is pinned only to non-preferred CPUs. They will continue to run where they were previously running before the CPUs was marked as non-preferred.
---v4->v5:
- Remove previous cpumask_and (K Prateek Nayak)
kernel/sched/fair.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/sched/fair.c +2 -1
@@ -13289,7 +13289,8 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq, };bool need_unlock = false; -cpumask_and (cpus,sched_domain_span (sd), cpu_active_mask); +/* Spread load among preferred CPUs */ +cpumask_and (cpus,sched_domain_span (sd), cpu_preferred_mask);schedstat_inc (sd->lb_count[idle]);
Signature · 1 line
2.47.3
PATCH v5 9/24 sched/fair: Pull the load on preferred CPU
In reply to Shrikanth HegdeWhen cpu is marked as non preferred, any load pulled towards it is pointless since in the next tick task will be pushed out again.
Since load balancing only happens among preferred CPUs, should_we_balance will bail out. But for NEWIDLE and IDLE balance, this bailout can happen even earlier.
---v4->v5:
- new patch
kernel/sched/fair.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/sched/fair.c +8 -0
@@ -14196,6 +14196,10 @@ static void _nohz_idle_balance(struct rq *this_rq, unsigned int flags)if (!idle_cpu (balance_cpu))continue ; +/* There is no point in pulling the load, just to push it out next */ +if (!cpu_preferred (balance_cpu)) +continue ; +/* * If this CPU gets work to do, stop the load balancing * work being done for other CPUs. Next load
@@ -14375,6 +14379,10 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)if (!cpu_active (this_cpu))return 0 ; +/* Do not pull to a !preferred CPU just to push it out next */ +if (!cpu_preferred (this_cpu)) +return 0 ; +/* * This is OK, because current is on_cpu, which avoids it being picked * for load-balance and preemption/IRQs are still disabled avoiding
Signature · 1 line
2.47.3
Peter Zijlstra
maintainerRe: [PATCH v5 09/24] sched/fair: Pull the load on preferred CPU
kernel/sched/fair.c static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf) · line 14388
Quoted 11 lines · Shrikanth Hegde
kernel/sched/fair.c +4
-0@@ -14375,6 +14379,10 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)if (!cpu_active (this_cpu))return 0 ; +/* Do not pull to a !preferred CPU just to push it out next */ +if (!cpu_preferred (this_cpu)) +return 0 ; +/* * This is OK, because current is on_cpu, which avoids it being picked * for load-balance and preemption/IRQs are still disabled avoiding
Why not just replace the cpu_active() check above?
Re: [PATCH v5 09/24] sched/fair: Pull the load on preferred CPU
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Shrikanth Hegde
kernel/sched/fair.c +0
-0@@ -14375,6 +14379,10 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)if (!cpu_active (this_cpu))return 0 ;
Quoted 9 lines
kernel/sched/fair.c static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf) · line 14388
Quoted 7 lines
kernel/sched/fair.c +4
-0+/* Do not pull to a !preferred CPU just to push it out next */ +if (!cpu_preferred (this_cpu)) +return 0 ; +/* * This is OK, because current is on_cpu, which avoids it being picked * for load-balance and preemption/IRQs are still disabled avoiding Why not just replace the
cpu_active()check above?
Ok. that should be fine. i will add a comment there.
PATCH v5 10/24 sched/core: Keep tick on non-preferred CPUs until tasks are out
In reply to Shrikanth HegdeEnable tick on nohz full CPU when it is marked as non-preferred. If there in no CFS running there, disable the tick to save the power.
Steal time handling code will call tick_nohz_dep_set_cpu with TICK_DEP_BIT_SCHED for moving the task out of nohz_full CPU fast.
---v4->v5:
- Move it below rt checks. (Sashiko)
kernel/sched/core.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/sched/core.c +4 -0
@@ -1473,6 +1473,10 @@ bool sched_can_stop_tick(struct rq *rq) return false; } +/* Keep the tick running until CFS tasks are pushed out*/ +if (!cpu_preferred (rq->cpu) && rq->cfs.h_nr_queued) +return false; +return true; }#endif /* CONFIG_NO_HZ_FULL */
Signature · 1 line
2.47.3
PATCH v5 11/24 sched/core: Push current task from non preferred CPU
In reply to Shrikanth HegdeActively push out task running on a non-preferred CPU. Since the task is running on the CPU, need to stop the cpu and push the task out. However, if the task in pinned only to non-preferred CPUs, it will continue running there. This will help in maintaining the userspace affinities unlike CPU hotplug or isolated cpusets.
Though code is almost same as __balance_push_cpu_stop and quite close to push_cpu_stop, it is being kept separate as it provides a cleaner implementation w.r.t CONFIG_HOTPLUG_CPU.
Add push_task_work_done flag to protect work buffer. Works only with FAIR class.
---v4->v5:
- Move select_fallback_rq outside of rq_lock (Sashiko)
- Add context_unsafe_alias (K Prateek Nayak)
- Cleanup properly on early exit.
kernel/sched/core.c | 87 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 8 ++++ 2 files changed, 95 insertions(+)
kernel/sched/core.c +87 -0
@@ -5794,6 +5794,9 @@ void sched_tick(void)unsigned long hw_pressure;u64 resched_latency; +if (!cpu_preferred (cpu)) +sched_push_current_non_preferred_cpu (rq); +if (housekeeping_cpu (cpu,HK_TYPE_KERNEL_NOISE ))arch_scale_freq_tick ();
@@ -11303,3 +11306,87 @@ void sched_change_end(struct sched_change_ctx *ctx) p->sched_class->prio_changed (rq, p, ctx->prio); } } + +#ifdef CONFIG_PREFERRED_CPU +/* npc - non preferred CPU */ +static DEFINE_PER_CPU(struct cpu_stop_work, npc_push_task_work); + +static int sched_non_preferred_cpu_push_stop (void *arg) +{ +struct task_struct *p = arg; +struct rq *rq =this_rq (); +struct rq_flags rf; +int cpu; + +/* sanity check and clear */ +if (cpu_preferred (rq->cpu)) { +scoped_guard (rq_lock, rq) + rq->push_task_work_done = 0; +put_task_struct (p); +return 0 ; + } + +raw_spin_lock_irq (&p->pi_lock); + +/* This could take rq lock. So call it before rq lock is taken */ + cpu =select_fallback_rq (rq->cpu, p); +rq_lock (rq, &rf); + rq->push_task_work_done =0 ; +update_rq_clock (rq); + +context_unsafe_alias (rq); + +if (task_rq (p) == rq &&task_on_rq_queued (p)) + rq =__migrate_task (rq, &rf, p, cpu); + +rq_unlock (rq, &rf); +raw_spin_unlock_irq (&p->pi_lock); +put_task_struct (p); + +return 0 ; +} + +/* +* Push the current task running on non-preferred CPU. +* Using this non preferred CPU will lead to more vCPU preemptions +* in the host. So it is better not to use this CPU. +* +* Since task is running, call a stopper to push the task out. This is +* similar to how task moves during hotplug. In select_fallback_rq a +* preferred CPU will be chosen and henceforth task shouldn't come back to +* this CPU again. +* +* Works for FAIR class only +* +* If task is affined only non-preferred CPUs, it can't be moved out +*/ +void sched_push_current_non_preferred_cpu (struct rq *rq) +{ +struct task_struct *push_task = rq->curr; + +/* Push only if it is FAIR class */ +if (push_task->sched_class != &fair_sched_class) +return ; + +if (kthread_is_per_cpu (push_task) || +is_migration_disabled (push_task)) +return ; + +/* Is there any preferred CPU in the affinity list */ +if (!task_has_preferred_cpus (push_task)) +return ; + +/* There is already a stopper thread for this. Dont race with it */ +if (rq->push_task_work_done ==1 ) +return ; + +/* sched_tick runs with interrupts disabled. Don't disable again */ +get_task_struct (push_task); + +scoped_guard (rq_lock, rq) + rq->push_task_work_done = 1; + +stop_one_cpu_nowait (rq->cpu, sched_non_preferred_cpu_push_stop, + push_task,this_cpu_ptr (&npc_push_task_work)); +} +#endif
kernel/sched/sched.h +8 -0
@@ -1274,6 +1274,8 @@ struct rq {struct list_head cfs_tasks; +bool push_task_work_done; +struct sched_avg avg_rt;struct sched_avg avg_dl;#ifdef CONFIG_HAVE_SCHED_AVG_IRQ
@@ -4241,4 +4243,10 @@ static inline bool task_has_preferred_cpus(struct task_struct *p) else return cpumask_intersects(p->cpus_ptr, cpu_preferred_mask); } + +#ifdef CONFIG_PREFERRED_CPU +void sched_push_current_non_preferred_cpu (struct rq *rq); +#else /* !CONFIG_PREFERRED_CPU */ +static inline void sched_push_current_non_preferred_cpu (struct rq *rq) { } +#endif #endif /* _KERNEL_SCHED_SCHED_H */
Signature · 1 line
2.47.3
PATCH v5 12/24 sched/debug: Add migration stats due to non preferred CPUs
In reply to Shrikanth HegdeAdd a new stat.
- nr_migrations_cpu_non_preferred: number of migrations happened since a CPU was marked as non preferred due to high steal time.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 4 +++- kernel/sched/debug.c | 1 + 3 files changed, 5 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -554,6 +554,7 @@ struct sched_statistics {u64 nr_failed_migrations_running;u64 nr_failed_migrations_hot;u64 nr_forced_migrations; +u64 nr_migrations_cpu_non_preferred;u64 nr_wakeups;u64 nr_wakeups_sync;
kernel/sched/core.c +3 -1
@@ -11336,8 +11336,10 @@ static int sched_non_preferred_cpu_push_stop(void *arg)context_unsafe_alias (rq); -if (task_rq (p) == rq &&task_on_rq_queued (p)) +if (task_rq (p) == rq &&task_on_rq_queued (p)) { rq =__migrate_task (rq, &rf, p, cpu); +schedstat_inc (p->stats.nr_migrations_cpu_non_preferred); + }rq_unlock (rq, &rf);raw_spin_unlock_irq (&p->pi_lock);
kernel/sched/debug.c +1 -0
@@ -1363,6 +1363,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P_SCHEDSTAT (nr_failed_migrations_running);P_SCHEDSTAT (nr_failed_migrations_hot);P_SCHEDSTAT (nr_forced_migrations); +P_SCHEDSTAT (nr_migrations_cpu_non_preferred);P_SCHEDSTAT (nr_wakeups);P_SCHEDSTAT (nr_wakeups_sync);P_SCHEDSTAT (nr_wakeups_migrate);
Signature · 1 line
2.47.3
PATCH v5 13/24 virt/steal_monitor: Add documentation
In reply to Shrikanth HegdeDocument this module named steal_monitor and its parameters.
---
v4-v5:
- new patch
Please let me know if the placing is not right.
Documentation/driver-api/index.rst | 1 + Documentation/driver-api/steal-monitor.rst | 93 ++++++++++++++++++++++ 2 files changed, 94 insertions(+) create mode 100644 Documentation/driver-api/steal-monitor.rst
Documentation/driver-api/index.rst +1 -0
@@ -138,6 +138,7 @@ Subsystem-specific APIs sm501 soundwire/index spi + steal-monitor surface_aggregator/index switchtec sync_file
Documentation/driver-api/steal-monitor.rst new file +93 -0
93 lines not on this page · raw message
Signature · 1 line
2.47.3
Randy Dunlap
reviewerRe: [PATCH v5 13/24] virt/steal_monitor: Add documentation
Hi,
Quoted 21 lines · Shrikanth Hegde
Document this module named
steal_monitorand its parameters.Signed-off-by Shrikanth Hegde <[email protected]>---
v4-v5:
- new patchPlease let me know if the placing is not right.
Documentation/driver-api/index.rst | 1 + Documentation/driver-api/steal-monitor.rst | 93 ++++++++++++++++++++++ 2 files changed, 94 insertions(+) create mode 100644 Documentation/driver-api/steal-monitor.rstDocumentation/driver-api/steal-monitor.rst new file +0
-0
Documentation/driver-api/steal-monitor.rst line 9
Quoted 9 lines
Documentation/driver-api/steal-monitor.rst new file +9
-0@@ -0,0 +1,93 @@ +.. SPDX-License-Identifier: GPL-2.0 +============= +Steal Monitor +============= + +:Author: Shrikanth Hegde + +Introduction: +=============
Nit:
Kernel heading adornment style does not include an ending ':' character (4 places).
Quoted 5 lines
Documentation/driver-api/steal-monitor.rst new file +5
-0+ +Steal monitor is a driver aimed at solving the Noisy Neighbour problem +in virtualized environments. I.e performance of workload +running in one VM gets affected significantly due to other VMs and +combined they make slower forward progress.
Signature · 1 line
~Randy
Re: [PATCH v5 13/24] virt/steal_monitor: Add documentation
Hi Randy, Thanks for checking the patches.
Quoted 29 lines · Randy Dunlap
Hi,
Quoted 26 lines · Shrikanth Hegde
Document this module named
steal_monitorand its parameters.Signed-off-by Shrikanth Hegde <[email protected]>---
v4-v5:
- new patchPlease let me know if the placing is not right.
Documentation/driver-api/index.rst | 1 + Documentation/driver-api/steal-monitor.rst | 93 ++++++++++++++++++++++ 2 files changed, 94 insertions(+)create mode 100644
Documentation/driver-api/steal-monitor.rstDocumentation/driver-api/steal-monitor.rst new file +5
-0@@ -0,0 +1,93 @@ +.. SPDX-License-Identifier: GPL-2.0 +============= +Steal Monitor +============= +
Quoted 9 lines
Documentation/driver-api/steal-monitor.rst line 9
Quoted 4 lines
Documentation/driver-api/steal-monitor.rst new file +4
-0+:Author: Shrikanth Hegde + +Introduction: +=============Nit:
Kernel heading adornment style does not include an ending ':' character (4 places).
Ok. I will fix it next version.
Quoted 7 lines
Quoted 5 lines
Documentation/driver-api/steal-monitor.rst new file +5
-0+ +Steal monitor is a driver aimed at solving the Noisy Neighbour problem +in virtualized environments. I.e performance of workload +running in one VM gets affected significantly due to other VMs and +combined they make slower forward progress.
Peter Zijlstra
maintainerRe: [PATCH v5 13/24] virt/steal_monitor: Add documentation
In reply to Shrikanth HegdeQuoted 14 lines · Shrikanth Hegde
Documentation/driver-api/steal-monitor.rst new file +14
-0+Core idea: +========== +steal time is an indication available today in Guest which shows contention +for underlying physical CPU. Use it as a hint in the guest to fold the +workload to a reduced set of vCPUs. When there is contention, steal time +will show up in all the guests. When each guest honors the hint and folds +the workload to a smaller set of vCPUs(Preferred CPUs), it reduces the +contention and thereby reduces vCPU preemption. +This is achieved without any cross-guest communication. + +Steal monitor driver effectively does: + +1. Periodically computes steal time across the system. +
Documentation/driver-api/steal-monitor.rst line 57
Quoted 10 lines
Documentation/driver-api/steal-monitor.rst new file +10
-0+2. If steal time is greater than high threshold, reduce the number of + preferred CPUs by 1 core. Ensure at least one core is left always. + This avoids running into extreme cases. + +3. If steal time is lower or equal to low threshold, increase the + number of preferred CPUs by 1 core. If preferred is same as active, + nothing to be done. + +4. Ensure preferred CPUs is always subset of active CPUs. + On feature disable it is same as active CPUs.
So this is very much a co-operative scheme. Perhaps add a few words to describe the effect of a non cooperative guest. IIRC the result is not worse than the status quo. That is, if one (or more) guests refuse to co-operate it will not make things worse, it will just not result in improvements, right?
Re: [PATCH v5 13/24] virt/steal_monitor: Add documentation
Quoted 25 lines · Peter Zijlstra
Quoted 23 lines · Shrikanth Hegde
Documentation/driver-api/steal-monitor.rst new file +23
-0+Core idea: +========== +steal time is an indication available today in Guest which shows contention +for underlying physical CPU. Use it as a hint in the guest to fold the +workload to a reduced set of vCPUs. When there is contention, steal time +will show up in all the guests. When each guest honors the hint and folds +the workload to a smaller set of vCPUs(Preferred CPUs), it reduces the +contention and thereby reduces vCPU preemption. +This is achieved without any cross-guest communication. + +Steal monitor driver effectively does: + +1. Periodically computes steal time across the system. + +2. If steal time is greater than high threshold, reduce the number of + preferred CPUs by 1 core. Ensure at least one core is left always. + This avoids running into extreme cases. + +3. If steal time is lower or equal to low threshold, increase the + number of preferred CPUs by 1 core. If preferred is same as active, + nothing to be done. + +4. Ensure preferred CPUs is always subset of active CPUs.
Quoted 8 lines
Quoted 1 line
+ On feature disable it is same as active CPUs.
So this is very much a co-operative scheme. Perhaps add a few words to describe the effect of a non cooperative guest. IIRC the result is not worse than the status quo. That is, if one (or more) guests refuse to co-operate it will not make things worse, it will just not result in improvements, right?
Yes, for the benefits all the guests should enable the feature. If not, one guest may use more. but if we see overall combined performance, it should be better than status quo.
I will add a paragraph about it.
PATCH v5 14/24 virt: Introduce steal monitor driver
In reply to Shrikanth HegdeIntroduce a new driver in virt named steal_monitor. This driver will compute the steal time and drive the policy decisions of preferred CPU state.
More on it can be found in the Documentation/driver-api/steal-monitor.rst This patch introduces the skeleton code.
There is no new kconfig. It depends on CONFIG_PREFERRED_CPU.
- If CONFIG_PREFERRED_CPU=y, it gets compiled as a module. It is not loaded by default.
- If CONFIG_PREFERRED_CPU=n, module isn't compiled.
File layout of the driver is designed with having arch specific files in the future.
- sm_core.c - contains main driver code. This includes the periodic work function and take action on steal time.
- defaults.c - contains the default implementation defined with __weak symbols.
- sm_core.h - header file which includes data structure.
---v4->v5:
- new patch
Please let me know if the placing is not right.
drivers/virt/Makefile | 1 + drivers/virt/steal_monitor/Makefile | 14 ++++++++++++ drivers/virt/steal_monitor/sm_core.c | 33 ++++++++++++++++++++++++++++ drivers/virt/steal_monitor/sm_core.h | 11 ++++++++++ 4 files changed, 59 insertions(+) create mode 100644 drivers/virt/steal_monitor/Makefile create mode 100644 drivers/virt/steal_monitor/sm_core.c create mode 100644 drivers/virt/steal_monitor/sm_core.h
drivers/virt/Makefile +1 -0
@@ -9,4 +9,5 @@ obj-y += vboxguest/ obj-$(CONFIG_NITRO_ENCLAVES) += nitro_enclaves/ obj-$(CONFIG_ACRN_HSM) += acrn/ +obj-$(CONFIG_PREFERRED_CPU) += steal_monitor/ obj-y += coco/
drivers/virt/steal_monitor/Makefile new file +14 -0
@@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Steal time monitor to alter preferred CPU state. +# +# Arch can implement strong function definitions and override the +# default by adding them in arch specific file. It must ensure +# that preferred is always subset of active. +# +# It is always compiled as module if CONFIG_PREFERRED_CPU=y +# One has to enable the module. +# +obj-$(subst y,m,$(CONFIG_PREFERRED_CPU)) += steal_monitor.o + +steal_monitor-y := sm_core.o
drivers/virt/steal_monitor/sm_core.c new file +33 -0
@@ -0,0 +1,33 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Steal time Monitor. +* +* Periodically compute steal time. Based on the thresholds either +* reduce/increase the preferred CPUs which can be made use +* by the workload to avoid vCPU preemption to an extent possible. +* +* Available as module with CONFIG_PREFERRED_CPU=y +* +* Copyright (C) 2026 IBM +* Author: Shrikanth Hegde <[email protected]> +*/ + +#include "sm_core.h" + +static int __init steal_monitor_init (void ) +{ +pr_info ("steal_monitor is enabled\n" ); +return 0 ; +} + +static void __exit steal_monitor_exit(void) +{ +pr_info ("steal_monitor is disabled\n" ); +} + +module_init (steal_monitor_init); +module_exit (steal_monitor_exit); + +MODULE_LICENSE ("GPL" ); +MODULE_AUTHOR ("IBM Corporation" ); +MODULE_DESCRIPTION ("Virtualization Steal Time Monitor" );
drivers/virt/steal_monitor/sm_core.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __VIRT_STEAL_CORE_H +#define __VIRT_STEAL_CORE_H + +#include <linux/types.h> + +#include <linux/module.h> +#include <linux/kernel.h> +#include <linux/init.h> + +#endif /* __VIRT_STEAL_CORE_H */
Signature · 1 line
2.47.3
PATCH v5 15/24 virt/steal_monitor: Restore to active on module disable
In reply to Shrikanth HegdeWhen the module is not in use, preferred CPUs must be same as active CPUs.
Even if one disables the module during high steal time, it still restores the preferred CPUs to be same as active CPUs to keep disable path simple.
---v4->v5:
- Modified for steal_monitor
drivers/virt/steal_monitor/sm_core.c | 1 + 1 file changed, 1 insertion(+)
drivers/virt/steal_monitor/sm_core.c +1 -0
@@ -23,6 +23,7 @@ static int __init steal_monitor_init(void) static void __exit steal_monitor_exit(void) {pr_info ("steal_monitor is disabled\n" ); +cpumask_copy (&__cpu_preferred_mask, cpu_active_mask); }module_init (steal_monitor_init);
Signature · 1 line
2.47.3
PATCH v5 16/24 virt/steal_monitor: Define steal_monitor structure
In reply to Shrikanth HegdeMain structure of steal monitor. It has
- work: deferred periodic work function
- prev_steal, prev_time - To calculate the delta in periodic work.
- interval_ms, high_threshold, low_threshold - debug knobs of steal_monitor.
sm_core_ctx - instance used by the core code.
---v4->v5:
- Modified for steal_monitor
drivers/virt/steal_monitor/sm_core.c | 2 ++ drivers/virt/steal_monitor/sm_core.h | 12 ++++++++++++ 2 files changed, 14 insertions(+)
drivers/virt/steal_monitor/sm_core.c +2 -0
@@ -14,6 +14,8 @@#include "sm_core.h" +struct steal_monitor sm_core_ctx; +static int __init steal_monitor_init (void ) {pr_info ("steal_monitor is enabled\n" );
drivers/virt/steal_monitor/sm_core.h +12 -0
@@ -8,4 +8,16 @@#include <linux/kernel.h> #include <linux/init.h> +struct steal_monitor { +struct delayed_work work; +u64 prev_steal; +int prev_direction; +unsigned int interval_ms; +unsigned int high_threshold; +unsigned int low_threshold; +ktime_t prev_time; +}; + +extern struct steal_monitor sm_core_ctx; +#endif /* __VIRT_STEAL_CORE_H */
Signature · 1 line
2.47.3
PATCH v5 17/24 virt/steal_monitor: Add control knobs for handling steal values
In reply to Shrikanth HegdeThese are the knobs to control the steal_monitor.
interval_ms:
How often steal monitor checks for steal time.
(Default: 1000 i.e 1 second)
This controls how fast steal monitor driver reacts to changes to the contention of physical CPUs. Since it does fair amount of work, setting too low will have overheads. If set to 0, on next work it will be set to default.
low_threshold:
lower threshold value in percentage * 100. (Default: 200, i.e 2% steal is considered as low threshold)
This determines what values should be considered as nil/no steal values. When steal monitor see steal time is below or equal to this value, it will increase the preferred CPUs by 1 core. Having value as zero might cause too much oscillations.
high_threshold:
higher threshold value in percentage * 100 (Default: 500, i.e 5% steal is considered as high threshold)
This determines what values should be considered as high steal values. When steal monitor sees steal time is higher than this value, it will reduce the preferred CPUs by 1 core.
Also available at: Documentation/driver-api/steal-monitor.rst
---v4->v5:
- Modified for steal_monitor
drivers/virt/steal_monitor/sm_core.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/virt/steal_monitor/sm_core.c +17 -1
@@ -14,7 +14,23 @@#include "sm_core.h" -struct steal_monitor sm_core_ctx; +struct steal_monitor sm_core_ctx = { + .interval_ms =1000 ,/* 1 second */ + .high_threshold =500 ,/* 5% */ + .low_threshold =200 ,/* 2% */ +}; + +module_param_named (interval_ms, sm_core_ctx.interval_ms, uint,0644 ); +MODULE_PARM_DESC (interval_ms, +"Sampling frequency for steal values in milliseconds (default: 1000)" ); + +module_param_named (high_threshold, sm_core_ctx.high_threshold, uint,0644 ); +MODULE_PARM_DESC (high_threshold, +"High steal threshold (default: 500 i.e 5%)" ); + +module_param_named (low_threshold, sm_core_ctx.low_threshold, uint,0644 ); +MODULE_PARM_DESC (low_threshold, +"Low steal threshold (default: 200 i.e 2%)" );static int __init steal_monitor_init (void ) {
Signature · 1 line
2.47.3
PATCH v5 18/24 virt/steal_monitor: Compute work at regular intervals
In reply to Shrikanth HegdeTrigger periodic work at intervals specified in interval_ms. schedule_delayed_work is chosen since this work need not happen at this instant and this variant is safe w.r.t to CPU hotplug.
Reset the interval_ms to default if one sets it to 0 to avoid workqueue stalls.
---v4->v5:
- Modified for steal_monitor
drivers/virt/steal_monitor/sm_core.c | 26 +++++++++++++++++++++++++- drivers/virt/steal_monitor/sm_core.h | 2 ++ 2 files changed, 27 insertions(+), 1 deletion(-)
drivers/virt/steal_monitor/sm_core.c +25 -1
@@ -32,15 +32,39 @@ module_param_named(low_threshold, sm_core_ctx.low_threshold, uint, 0644);MODULE_PARM_DESC (low_threshold,"Low steal threshold (default: 200 i.e 2%)" ); +static void compute_preferred_cpus_work (struct work_struct *work) +{ +/* At least one core is kept as preferred */ +WARN_ON (cpumask_empty (cpu_preferred_mask)); + +/* Warn if interval_ms is set to 0, that might cause lockup. */ +if (unlikely (sm_core_ctx.interval_ms ==0 )) { +WARN_ON (1 ); + sm_core_ctx.interval_ms =1000 ;/* Fallback to default */ + } + +/* Trigger for next sampling */ +schedule_delayed_work (&sm_core_ctx.work, +msecs_to_jiffies (sm_core_ctx.interval_ms)); +} +static int __init steal_monitor_init (void ) { -pr_info ("steal_monitor is enabled\n" ); +pr_info ("steal_monitor is enabled. interval: %ums, high_threshold: %u, low_threshold: %u\n" , + sm_core_ctx.interval_ms, sm_core_ctx.high_threshold, sm_core_ctx.low_threshold); + +INIT_DELAYED_WORK (&sm_core_ctx.work, compute_preferred_cpus_work); + +schedule_delayed_work (&sm_core_ctx.work, +msecs_to_jiffies (sm_core_ctx.interval_ms)); +return 0 ; } static void __exit steal_monitor_exit(void) {pr_info ("steal_monitor is disabled\n" ); +cancel_delayed_work_sync (&sm_core_ctx.work);cpumask_copy (&__cpu_preferred_mask, cpu_active_mask); }
drivers/virt/steal_monitor/sm_core.h +2 -0
@@ -7,6 +7,8 @@#include <linux/module.h> #include <linux/kernel.h> #include <linux/init.h> +#include <linux/cpumask.h> +#include <linux/workqueue.h> struct steal_monitor {struct delayed_work work;
Signature · 1 line
2.47.3
PATCH v5 19/24 virt/steal_monitor: Provide default method to get systemwide steal time
In reply to Shrikanth Hegdesteal monitor takes global view of steal time instead of individual vCPU. For this collect overall steal values across all the vCPUs or vCPUs of interest.
Default implementation chooses steal time across all active CPUs.
---v4->v5:
- new patch
drivers/virt/steal_monitor/Makefile | 2 +- drivers/virt/steal_monitor/defaults.c | 27 +++++++++++++++++++++++++++ drivers/virt/steal_monitor/sm_core.h | 2 ++ 3 files changed, 30 insertions(+), 1 deletion(-) create mode 100644 drivers/virt/steal_monitor/defaults.c
drivers/virt/steal_monitor/Makefile +1 -1
@@ -11,4 +11,4 @@ # obj-$(subst y,m,$(CONFIG_PREFERRED_CPU)) += steal_monitor.o -steal_monitor-y := sm_core.o +steal_monitor-y := sm_core.o defaults.o
drivers/virt/steal_monitor/defaults.c new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Base file contains the default implementations. +* These are defined as __weak so that arch may define +* strong symbols to override. +* +* Copyright (C) 2026 IBM +* Author: Shrikanth Hegde <[email protected]> +*/ +#include "sm_core.h" + +/* +* Compute steal time of the full system. +* +* Default implementation returns steal time across all active CPUs +*/ + +u64 __weak get_system_steal_time (void ) +{ +int tmp_cpu; +u64 total_steal =0 ; + +for_each_cpu (tmp_cpu, cpu_active_mask) + total_steal +=kcpustat_cpu (tmp_cpu).cpustat[CPUTIME_STEAL ]; + +return total_steal; +}
drivers/virt/steal_monitor/sm_core.h +2 -0
@@ -9,6 +9,7 @@#include <linux/init.h> #include <linux/cpumask.h> #include <linux/workqueue.h> +#include <linux/kernel_stat.h> struct steal_monitor {struct delayed_work work;
@@ -22,4 +23,5 @@ struct steal_monitor {extern struct steal_monitor sm_core_ctx; +u64 get_system_steal_time (void );#endif /* __VIRT_STEAL_CORE_H */
Signature · 1 line
2.47.3
PATCH v5 20/24 virt/steal_monitor: Provide default method to inc/dec preferred CPUs
In reply to Shrikanth HegdeThese methods will be used by the steal_monitor core in subsequent patches. Default implementation are likely good enough for most archs.
decrease_preferred_cpus() - Called when there is high steal time. It needs to decide which CPUs to mark as non-preferred and set that state. increase_preferred_cpus() - Called when there is low steal time. It needs to decide which CPUs to mark as preferred and set that state.
Default Implementations:decrease_preferred_cpus()
- Get the last CPU in cpu_preferred_mask.
- Check if that last CPU belong to first housekeeping core. If so there is nothing to do. This helps to keep at least one core as preferred. This is to be safe under non-normal cases.
- If it is not first housekeeping core, get its sibling and mark them as non-preferred. If they are nohz_full, enable the tick. push mechanism relies on sched_tick.
increase_preferred_cpus()
- Get the first active non-preferred CPUs. This likely is the last set of CPUs being marked as non-preferred.
- If there is no such CPU, i.e preferred is same as active. Nothing todo further.
- If not, get the siblings of that core and mark them as preferred. Note that clearing the tick isn't needed as that would be handled via sched_can_stop_tick.
Using core instead of individual CPUs give better numbers as SMT is quite common and some hypervisor such as powerVM does core scheduling.
---v4->v5:
- Modified for steal_monitor
drivers/virt/steal_monitor/defaults.c | 68 +++++++++++++++++++++++++++ drivers/virt/steal_monitor/sm_core.h | 4 ++ 2 files changed, 72 insertions(+)
drivers/virt/steal_monitor/defaults.c +68 -0
@@ -25,3 +25,71 @@ u64 __weak get_system_steal_time(void)return total_steal; } + +/* +* Default implementation of decrementing the preferred CPUs based on steal +* time. This is simple logic and decrease the preferred CPUs by 1 core. +* It takes out the last core in the active & preferred. +* +* Ensure at least one housekeeping core is always kept as preferred +* +* Could be overwritten by arch specific handling. Arch must ensure +* preferred is always subset of active. +*/ + +#define get_core_mask (cpu) topology_sibling_cpumask(cpu) + +void __weak decrease_preferred_cpus (struct steal_monitor *ctx) +{ +int last_cpu, tmp_cpu; +int first_hk_cpu; + +guard (cpus_read_lock)(); + + last_cpu =cpumask_last (cpu_preferred_mask); + first_hk_cpu =cpumask_first_and (housekeeping_cpumask (HK_TYPE_KERNEL_NOISE ), + cpu_active_mask); +/* +* If the core belongs to the first housekeeping CPUs, no action is +* taken. This leaves at least one core preferred always. +* This ensures at least some CPUs are available to run. +*/ +if (cpumask_equal (get_core_mask (last_cpu),get_core_mask (first_hk_cpu))) +return ; + +/* +* set tick bit for nohz_full CPU to push the task out. Once the tasks +* are pushed out, bit will be cleared if there are no tasks. +*/ + +for_each_cpu_and (tmp_cpu,get_core_mask (last_cpu), cpu_active_mask) { +set_cpu_preferred (tmp_cpu, false); +if (tick_nohz_full_cpu (tmp_cpu)) +tick_nohz_dep_set_cpu (tmp_cpu,TICK_DEP_BIT_SCHED ); + } +} + +/* +* Default implementation of incrementing preferred CPUs based on steal +* time. This is simple logic and increases the preferred CPUs by 1 core. +* It adds the first core in active & !preferred +* +* Nothing to do if active == preferred +* +* Could be overwritten by arch specific handling. Arch must ensure +* preferred is subset of active. +*/ +void __weak increase_preferred_cpus (struct steal_monitor *ctx) +{ +int first_cpu, tmp_cpu; + +guard (cpus_read_lock)(); + + first_cpu =cpumask_first_andnot (cpu_active_mask, cpu_preferred_mask); +/* All CPUs are preferred. Nothing to increase further */ +if (first_cpu >= nr_cpu_ids) +return ; + +for_each_cpu_and (tmp_cpu,get_core_mask (first_cpu), cpu_active_mask) +set_cpu_preferred (tmp_cpu, true); +}
drivers/virt/steal_monitor/sm_core.h +4 -0
@@ -10,6 +10,8 @@#include <linux/cpumask.h> #include <linux/workqueue.h> #include <linux/kernel_stat.h> +#include <linux/tick.h> +#include <linux/sched/isolation.h> struct steal_monitor {struct delayed_work work;
@@ -24,4 +26,6 @@ struct steal_monitor {extern struct steal_monitor sm_core_ctx;u64 get_system_steal_time (void ); +void increase_preferred_cpus (struct steal_monitor *ctx); +void decrease_preferred_cpus (struct steal_monitor *ctx);#endif /* __VIRT_STEAL_CORE_H */
Signature · 1 line
2.47.3
PATCH v5 21/24 virt/steal_monitor: Provide default method to get num of CPUs for steal ratio
In reply to Shrikanth HegdeThis method informs the steal_monitor core, how many CPUs it needs to consider for steal ratio calculations.steal_ratio = (delta_steal * 100 * 100) / (delta_ns * number_of_cpus);
Default method returns number of Active CPUs since it calculates steal time across active CPUs.
---v4->v5:
- new patch
drivers/virt/steal_monitor/defaults.c | 10 ++++++++++ drivers/virt/steal_monitor/sm_core.h | 1 + 2 files changed, 11 insertions(+)
drivers/virt/steal_monitor/defaults.c +10 -0
@@ -26,6 +26,16 @@ u64 __weak get_system_steal_time(void)return total_steal; } +/* +* Return number of CPUs to consider for steal ratio calculation +* +* Default returns number of active CPUs. +*/ +unsigned int __weak get_num_cpus_steal_ratio (void ) +{ +return num_active_cpus (); +} +/* * Default implementation of decrementing the preferred CPUs based on steal * time. This is simple logic and decrease the preferred CPUs by 1 core.
drivers/virt/steal_monitor/sm_core.h +1 -0
@@ -26,6 +26,7 @@ struct steal_monitor {extern struct steal_monitor sm_core_ctx;u64 get_system_steal_time (void ); +unsigned int get_num_cpus_steal_ratio (void );void increase_preferred_cpus (struct steal_monitor *ctx);void decrease_preferred_cpus (struct steal_monitor *ctx);#endif /* __VIRT_STEAL_CORE_H */
Signature · 1 line
2.47.3
PATCH v5 22/24 virt/steal_monitor: Act on steal values at regular intervals
In reply to Shrikanth HegdeThis is the steal_monitor core functionality done in periodic work
- Calculate the steal_ratio. It is multiplied by 100 to consider the fractional values of steal time. I.e 10 means 0.1% steal time.
- If steal value is higher than high threshold, call the method to reduce the preferred CPUs.
- If steal value is lower or equal to low threshold, call the method to increase the preferred CPUs.
- If the steal value is in between, no action is taken.
- Save the values for next delta calculations.
---v4->v5:
- Modified for steal_monitor
drivers/virt/steal_monitor/sm_core.c | 29 ++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
drivers/virt/steal_monitor/sm_core.c +29 -0
@@ -34,6 +34,33 @@ MODULE_PARM_DESC(low_threshold,static void compute_preferred_cpus_work (struct work_struct *work) { +u64 curr_steal, delta_steal, delta_ns, steal_ratio; +ktime_t now; + + curr_steal =get_system_steal_time (); + now =ktime_get (); + +/* get the deltas */ + delta_steal = curr_steal > sm_core_ctx.prev_steal ? + curr_steal - sm_core_ctx.prev_steal :0 ; + delta_ns =max_t (u64,ktime_to_ns (ktime_sub (now, sm_core_ctx.prev_time)),1 ); + +/* Update for next calculation */ + sm_core_ctx.prev_steal = curr_steal; + sm_core_ctx.prev_time = now; + +/* Multiply by 100 to consider the fractional values of steal time */ + steal_ratio = (delta_steal *100 *100 ) / + (delta_ns *get_num_cpus_steal_ratio ()); + +/* If the steal time values are high, reduce preferred CPUs */ +if (steal_ratio > sm_core_ctx.high_threshold) +decrease_preferred_cpus (&sm_core_ctx); + +/* If the steal time values are low, increase preferred CPUs */ +if (steal_ratio <= sm_core_ctx.low_threshold) +increase_preferred_cpus (&sm_core_ctx); +/* At least one core is kept as preferred */ WARN_ON (cpumask_empty (cpu_preferred_mask));
@@ -54,6 +81,8 @@ static int __init steal_monitor_init(void) sm_core_ctx.interval_ms, sm_core_ctx.high_threshold, sm_core_ctx.low_threshold);INIT_DELAYED_WORK (&sm_core_ctx.work, compute_preferred_cpus_work); + sm_core_ctx.prev_steal =get_system_steal_time (); + sm_core_ctx.prev_time =ktime_get ();schedule_delayed_work (&sm_core_ctx.work,msecs_to_jiffies (sm_core_ctx.interval_ms));
Signature · 1 line
2.47.3
PATCH v5 23/24 virt/steal_monitor: Add direction control
In reply to Shrikanth HegdeCache the previous direction on steal time. So two consecutive values of high values or low values are taken for decrease/increase of preferred CPUs. This helps to avoid oscillations.
---v4->v5:
- Modified for steal_monitor
drivers/virt/steal_monitor/sm_core.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
drivers/virt/steal_monitor/sm_core.c +18 -2
@@ -20,6 +20,12 @@ struct steal_monitor sm_core_ctx = { .low_threshold = 200, /* 2% */ }; +enum sm_direction { +SM_DIR_INCREASE =-1 , +SM_DIR_NONE =0 , +SM_DIR_DECREASE =1 , +}; +module_param_named (interval_ms, sm_core_ctx.interval_ms, uint,0644 );MODULE_PARM_DESC (interval_ms,"Sampling frequency for steal values in milliseconds (default: 1000)" );
@@ -54,13 +60,23 @@ static void compute_preferred_cpus_work(struct work_struct *work) (delta_ns *get_num_cpus_steal_ratio ());/* If the steal time values are high, reduce preferred CPUs */ -if (steal_ratio > sm_core_ctx.high_threshold) +if (sm_core_ctx.prev_direction ==SM_DIR_DECREASE && + steal_ratio > sm_core_ctx.high_threshold)decrease_preferred_cpus (&sm_core_ctx);/* If the steal time values are low, increase preferred CPUs */ -if (steal_ratio <= sm_core_ctx.low_threshold) +if (sm_core_ctx.prev_direction ==SM_DIR_INCREASE && + steal_ratio <= sm_core_ctx.low_threshold)increase_preferred_cpus (&sm_core_ctx); +/* mark the direction. This helps to avoid ping-pongs */ +if (steal_ratio > sm_core_ctx.high_threshold) + sm_core_ctx.prev_direction =SM_DIR_DECREASE ; +else if (steal_ratio <= sm_core_ctx.low_threshold) + sm_core_ctx.prev_direction =SM_DIR_INCREASE ; +else + sm_core_ctx.prev_direction =SM_DIR_NONE ; +/* At least one core is kept as preferred */ WARN_ON (cpumask_empty (cpu_preferred_mask));
Signature · 1 line
2.47.3
PATCH v5 24/24 virt/steal_monitor: Add design check of preferred subset of active
In reply to Shrikanth HegdeOne of the main design construct that CONFIG_PREFERRED_CPU maintains is that preferred is always subset of active. Force that in any future arch specific implementations.
---v4->v5:
- new patch
drivers/virt/steal_monitor/sm_core.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/virt/steal_monitor/sm_core.c +3 -0
@@ -80,6 +80,9 @@ static void compute_preferred_cpus_work(struct work_struct *work)/* At least one core is kept as preferred */ WARN_ON (cpumask_empty (cpu_preferred_mask)); +/* Maintain design construct */ +WARN_ON (!cpumask_subset (cpu_preferred_mask, cpu_active_mask)); +/* Warn if interval_ms is set to 0, that might cause lockup. */ if (unlikely (sm_core_ctx.interval_ms ==0 )) {WARN_ON (1 );
Signature · 1 line
2.47.3