sched: Flatten the pick
Subsystems (4)
Peter Zijlstra
maintainerv3 cover sched: Flatten the pick
Hi!
New version, same story [1]. TL;DR:
- Adds new cgroup_mode knob and implements new policies to address the hierarchy level weight mismatch.
- Builds upon that base to create a flat / single runqueue scheduler where the cgroup hierarchy is expressed through dynamic weight management.
I'm hoping to be able to merge these patches early in the next cycle (after 7.2-rc1).
Random benchmark:
Game vs 'for ((i=0; i<8; i++)) do nice ./spin.sh; done':
Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper) Intel Core i7-2600K
AMD Radeon RX 580
Shadows Awakening (GOG)
default slice(*)
FPS min 4.0 29.0
avg 47.5 59.2
max 83.7 83.7 FT min 9.3 10.2
avg 34.0 17.0
max 121.2 30.0 FPS (Frames Per Second)
FT (FrameTime)
[*] Command prefix: 'chrt -o --sched-runtime 100000 0'
Changes since v2:
- merged debug and prep patches
- fixed update_entity_lag() on dequeue (Vincent)
- fixed throttle vs tick (Prateek)
- fixed wakeup_preempt_fair()
- rebased on tip/sched/core
- rewritten cgroup_mode changelogs
- reworked cgroup_mode concur
- added cgroup_mode tasks
- changed default cgroup_mode
[1] - https://lore.kernel.org/r/[email protected]
Can also be had:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat
include/linux/cpuset.h | 6 include/linux/sched.h | 1 kernel/cgroup/cpuset.c | 15 kernel/sched/core.c | 5 kernel/sched/debug.c | 89 ++++ kernel/sched/fair.c | 943 ++++++++++++++++++++++++------------------------- kernel/sched/pelt.c | 6 kernel/sched/sched.h | 30 - 8 files changed, 607 insertions(+), 488 deletions(-)
Peter Zijlstra
maintainerPATCH v3 1/7 sched/fair: Add cgroup_mode switch
The effective task weight (W_t') for a task in cgroup g on CPU n is given by:
W_t
W_t' = W_g * F_g_n * ----------
\Sum W_t_nWhere W_g is the group's weight (cpu.weight), F_g_n is the fraction of the group weight for CPU n and W_t/W is the relative weight of this task against all other tasks in the same group on the same CPU.
Furthermore, this makes:
\Sum W_t_n F_g_n = ---------- \Sum W_t
The fraction of weight inside the group of CPU n against the whole group.
The problem is with F_g_n, the primary goal of this fraction is to make sure that the relative weight of tasks, when distributed over CPUs is maintained. For example, consider 4 (equal weight) tasks and 2 CPUs with a 1:3 distribution, then if F_g_n would simply be 1 (no weight re-distribution) the effective relative weights (W_t') of the tasks in our group would be:
CPU0 CPU1
W_g W_g/3
W_g/3
W_g/3IOW, the lucky task on CPU0 would get an equal amount of weight as all 3 tasks on CPU1 combined. However, with the weight redistribution, this becomes:
CPU0 CPU1
W_g/4 W_g/4
W_g/4
W_g/4All tasks are equal weight (as intended). However, as is already evident from this example, the more CPUs you add, the smaller F_g_n becomes, which creates a disparity against tasks not in our group.
Specifically:
avg(F_g_n) ~ 1/N
This leads to a weight mismatch in the hierarchy. IOW tasks cannot compete fairly across hierarchy levels.
*Notably*, what is meant by avg(F_g_n) being proportional to 1/N is that when there are at least N runnable tasks, the average of this fraction tends to 1/N.
For a hierarchy of depth d, this gets even worse, since that gets terms on the order of:
avg(F_g_n)^d ~ 1/(N^d)
Given fixed point arithmetic, this also leads to numerical trouble.
However, the meaning of "cpu.weight" is simple and intiutive: the total weight of the cgroup. But as explored above, there is deception in this simplicity.
Prepare to add a few alternative methods for distributing weight.
--- kernel/sched/debug.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+)
kernel/sched/debug.c +74 -0
@@ -633,6 +633,76 @@ static void debugfs_fair_server_init(voi } } +#ifdef CONFIG_FAIR_GROUP_SCHED +static int cgroup_mode =0 ; + +static const char *cgroup_mode_str[] = { +"smp" , +}; + +static int sched_cgroup_mode (const char *str) +{ +for (int i =0 ; i <ARRAY_SIZE (cgroup_mode_str); i++) { +if (!strcmp (str, cgroup_mode_str[i])) +return i; + } +return -EINVAL ; +} + +static ssize_t sched_cgroup_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) +{ +char buf[16 ]; +int mode; + +if (cnt >15 ) + cnt =15 ; + +if (copy_from_user (buf, ubuf, cnt)) +return -EFAULT ; + + buf[cnt] =0 ; + mode =sched_cgroup_mode (strstrip (buf)); +if (mode <0 ) +return mode; + +WRITE_ONCE (cgroup_mode, mode); + + *ppos += cnt; +return cnt; +} + +static int sched_cgroup_show (struct seq_file *m,void *v) +{ +int mode =READ_ONCE (cgroup_mode); + +for (int i =0 ; i <ARRAY_SIZE (cgroup_mode_str); i++) { +if (mode == i) +seq_puts (m,"(" ); +seq_puts (m, cgroup_mode_str[i]); +if (mode == i) +seq_puts (m,")" ); + +seq_puts (m," " ); + } +seq_puts (m,"\n" ); +return 0 ; +} + +static int sched_cgroup_open (struct inode *inode,struct file *filp) +{ +return single_open (filp, sched_cgroup_show,NULL ); +} + +static const struct file_operations sched_cgroup_fops = { + .open = sched_cgroup_open, + .write = sched_cgroup_write, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +#endif +static __init int sched_init_debug (void ) {struct dentry __maybe_unused *numa, *llc;
@@ -686,6 +756,10 @@ static __init int sched_init_debug(void)debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops); +#ifdef CONFIG_FAIR_GROUP_SCHED +debugfs_create_file ("cgroup_mode" ,0644 , debugfs_sched,NULL , &sched_cgroup_fops); +#endif +debugfs_fair_server_init ();#ifdef CONFIG_SCHED_CLASS_EXT debugfs_ext_server_init ();
[tip: sched/core] sched/fair: Add cgroup_mode switch
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4161cb2d867b2210b447afc4a941ab4b96e1bd13 Gitweb: https://git.kernel.org/tip/4161cb2d867b2210b447afc4a941ab4b96e1bd13 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Mar 2026 14:48:44 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:51 +02:00
sched/fair: Add cgroup_mode switch
The effective task weight (W_t') for a task in cgroup g on CPU n is given by:
W_t
W_t' = W_g * F_g_n * ----------
\Sum W_t_nWhere W_g is the group's weight (cpu.weight), F_g_n is the fraction of the group weight for CPU n and W_t/W is the relative weight of this task against all other tasks in the same group on the same CPU.
Furthermore, this makes:
\Sum W_t_n F_g_n = ---------- \Sum W_t
The fraction of weight inside the group of CPU n against the whole group.
The problem is with F_g_n, the primary goal of this fraction is to make sure that the relative weight of tasks, when distributed over CPUs is maintained. For example, consider 4 (equal weight) tasks and 2 CPUs with a 1:3 distribution, then if F_g_n would simply be 1 (no weight re-distribution) the effective relative weights (W_t') of the tasks in our group would be:
CPU0 CPU1
W_g W_g/3
W_g/3
W_g/3IOW, the lucky task on CPU0 would get an equal amount of weight as all 3 tasks on CPU1 combined. However, with the weight redistribution, this becomes:
CPU0 CPU1
W_g/4 W_g/4
W_g/4
W_g/4All tasks are equal weight (as intended). However, as is already evident from this example, the more CPUs you add, the smaller F_g_n becomes, which creates a disparity against tasks not in our group.
Specifically:
avg(F_g_n) ~ 1/N
This leads to a weight mismatch in the hierarchy. IOW tasks cannot compete fairly across hierarchy levels.
*Notably*, what is meant by avg(F_g_n) being proportional to 1/N is that when there are at least N runnable tasks, the average of this fraction tends to 1/N.
For a hierarchy of depth d, this gets even worse, since that gets terms on the order of:
avg(F_g_n)^d ~ 1/(N^d)
Given fixed point arithmetic, this also leads to numerical trouble.
However, the meaning of "cpu.weight" is simple and intiutive: the total weight of the cgroup. But as explored above, there is deception in this simplicity.
Prepare to add a few alternative methods for distributing weight.
--- kernel/sched/debug.c | 74 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 74 insertions(+)
kernel/sched/debug.c +74 -0
@@ -633,6 +633,76 @@ static void debugfs_fair_server_init(void) } } +#ifdef CONFIG_FAIR_GROUP_SCHED +static int cgroup_mode =0 ; + +static const char *cgroup_mode_str[] = { +"smp" , +}; + +static int sched_cgroup_mode (const char *str) +{ +for (int i =0 ; i <ARRAY_SIZE (cgroup_mode_str); i++) { +if (!strcmp (str, cgroup_mode_str[i])) +return i; + } +return -EINVAL ; +} + +static ssize_t sched_cgroup_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) +{ +char buf[16 ]; +int mode; + +if (cnt >15 ) + cnt =15 ; + +if (copy_from_user (buf, ubuf, cnt)) +return -EFAULT ; + + buf[cnt] =0 ; + mode =sched_cgroup_mode (strstrip (buf)); +if (mode <0 ) +return mode; + +WRITE_ONCE (cgroup_mode, mode); + + *ppos += cnt; +return cnt; +} + +static int sched_cgroup_show (struct seq_file *m,void *v) +{ +int mode =READ_ONCE (cgroup_mode); + +for (int i =0 ; i <ARRAY_SIZE (cgroup_mode_str); i++) { +if (mode == i) +seq_puts (m,"(" ); +seq_puts (m, cgroup_mode_str[i]); +if (mode == i) +seq_puts (m,")" ); + +seq_puts (m," " ); + } +seq_puts (m,"\n" ); +return 0 ; +} + +static int sched_cgroup_open (struct inode *inode,struct file *filp) +{ +return single_open (filp, sched_cgroup_show,NULL ); +} + +static const struct file_operations sched_cgroup_fops = { + .open = sched_cgroup_open, + .write = sched_cgroup_write, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +#endif +static __init int sched_init_debug (void ) {struct dentry __maybe_unused *numa, *llc;
@@ -686,6 +756,10 @@ static __init int sched_init_debug(void)debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops); +#ifdef CONFIG_FAIR_GROUP_SCHED +debugfs_create_file ("cgroup_mode" ,0644 , debugfs_sched,NULL , &sched_cgroup_fops); +#endif +debugfs_fair_server_init ();#ifdef CONFIG_SCHED_CLASS_EXT debugfs_ext_server_init ();
Peter Zijlstra
maintainerPATCH v3 2/7 sched/fair: Add cgroup_mode: up
In reply to Peter ZijlstraInstead of calculating the proportional fraction of the group weight for each CPU, just give each CPU the full measure, ignoring these pesky SMP problems.
This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single task in a cgroup competes on equal footing to a task in a level above.
However, as already explored, this is not a very good policy because it gets the SMP weight distribution wrong. Included for completeness.
--- kernel/sched/debug.c | 5 ++++- kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 3 files changed, 34 insertions(+), 3 deletions(-)
kernel/sched/debug.c +4 -1
@@ -271,6 +271,7 @@ static ssize_t sched_dynamic_write(strucif (mode <0 )return mode; +__sched_cgroup_mode_update (mode);sched_dynamic_update (mode); *ppos += cnt;
@@ -634,9 +635,11 @@ static void debugfs_fair_server_init(voi }#ifdef CONFIG_FAIR_GROUP_SCHED -static int cgroup_mode =0 ; +static int cgroup_mode =1 ; +/* See __sched_cgroup_mode_update(). */ static const char *cgroup_mode_str[] = { +"up" ,"smp" , };
kernel/sched/fair.c +29 -2
@@ -38,6 +38,7 @@#include <linux/sched/isolation.h> #include <linux/sched/nohz.h> #include <linux/sched/prio.h> +#include <linux/static_call.h> #include <linux/cpuidle.h> #include <linux/interrupt.h>
@@ -4800,7 +4801,7 @@ static inline int throttled_hierarchy(st* * hence icky! */ -static long calc_group_shares (struct cfs_rq *cfs_rq) +static long calc_smp (struct cfs_rq *cfs_rq) {long tg_weight, tg_shares, load, shares;struct task_group *tg = cfs_rq->tg;
@@ -4835,6 +4836,32 @@ static long calc_group_shares(struct cfs }/* +* Ignore this pesky SMP stuff, use (4). +*/ +static long calc_up_shares (struct cfs_rq *cfs_rq) +{ +struct task_group *tg = cfs_rq->tg; +return READ_ONCE (tg->shares); +} + +DEFINE_STATIC_CALL (calc_group_shares, calc_smp_shares); + +void __sched_cgroup_mode_update (int mode) +{ +long (*func)(struct cfs_rq *); +switch (mode) { +case 0 : + func = &calc_up_shares; +break ; +case 1 : +default : + func = &calc_smp_shares; +break ; + } +static_call_update (calc_group_shares, func); +} + +/* * Recomputes the group entity based on the current state of its group * runqueue. */
@@ -4850,7 +4877,7 @@ static void update_cfs_group(struct scheif (!gcfs_rq || !gcfs_rq->load.weight)return ; - shares =calc_group_shares (gcfs_rq); + shares =static_call (calc_group_shares)(gcfs_rq);if (unlikely (se->load.weight != shares))reweight_entity (cfs_rq_of (se), se, shares); }
kernel/sched/sched.h +1 -0
@@ -571,6 +571,7 @@ extern void free_fair_sched_group(structextern int alloc_fair_sched_group (struct task_group *tg,struct task_group *parent);extern void online_fair_sched_group (struct task_group *tg);extern void unregister_fair_sched_group (struct task_group *tg); +extern void __sched_cgroup_mode_update (int mode);#else /* !CONFIG_FAIR_GROUP_SCHED: */ static inline void free_fair_sched_group (struct task_group *tg) { }static inline int alloc_fair_sched_group (struct task_group *tg,struct task_group *parent)
Peter Zijlstra
maintainerRe: [PATCH v3 2/7] sched/fair: Add cgroup_mode: up
Quoted 19 lines · Peter Zijlstra
Instead of calculating the proportional fraction of the group weight for each CPU, just give each CPU the full measure, ignoring these pesky SMP problems.
This makes the SMP cgroup fraction (
F_g_n) equal to 1, and ensures a single task in a cgroup competes on equal footing to a task in a level above.However, as already explored, this is not a very good policy because it gets the SMP weight distribution wrong. Included for completeness.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/debug.c | 5 ++++- kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 3 files changed, 34 insertions(+), 3 deletions(-)
kernel/sched/debug.c static ssize_t sched_dynamic_write(struc · line 277
Quoted 7 lines
kernel/sched/debug.c +1
-0@@ -271,6 +271,7 @@ static ssize_t sched_dynamic_write(strucif (mode <0 )return mode; +__sched_cgroup_mode_update (mode);sched_dynamic_update (mode); *ppos += cnt;
Yeez, I'm not sute WTF happened here. Let me go fix that up.
[tip: sched/core] sched/fair: Add cgroup_mode: up
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 80ad6d3338ebe136b0c43c5caa962304a8454e3e Gitweb: https://git.kernel.org/tip/80ad6d3338ebe136b0c43c5caa962304a8454e3e Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Mar 2026 14:59:06 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:52 +02:00
sched/fair: Add cgroup_mode: up
Instead of calculating the proportional fraction of the group weight for each CPU, just give each CPU the full measure, ignoring these pesky SMP problems.
This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single task in a cgroup competes on equal footing to a task in a level above.
However, as already explored, this is not a very good policy because it gets the SMP weight distribution wrong. Included for completeness.
--- kernel/sched/debug.c | 5 ++++- kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++-- kernel/sched/sched.h | 1 + 3 files changed, 34 insertions(+), 3 deletions(-)
kernel/sched/debug.c +4 -1
@@ -634,9 +634,11 @@ static void debugfs_fair_server_init(void) }#ifdef CONFIG_FAIR_GROUP_SCHED -static int cgroup_mode =0 ; +static int cgroup_mode =1 ; +/* See __sched_cgroup_mode_update(). */ static const char *cgroup_mode_str[] = { +"up" ,"smp" , };
@@ -666,6 +668,7 @@ static ssize_t sched_cgroup_write(struct file *filp, const char __user *ubuf,if (mode <0 )return mode; +__sched_cgroup_mode_update (mode);WRITE_ONCE (cgroup_mode, mode); *ppos += cnt;
kernel/sched/fair.c +29 -2
@@ -38,6 +38,7 @@#include <linux/sched/isolation.h> #include <linux/sched/nohz.h> #include <linux/sched/prio.h> +#include <linux/static_call.h> #include <linux/cpuidle.h> #include <linux/interrupt.h>
@@ -4800,7 +4801,7 @@ static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);* * hence icky! */ -static long calc_group_shares (struct cfs_rq *cfs_rq) +static long calc_smp_shares (struct cfs_rq *cfs_rq) {long tg_weight, tg_shares, load, shares;struct task_group *tg = cfs_rq->tg;
@@ -4835,6 +4836,32 @@ static long calc_group_shares(struct cfs_rq *cfs_rq) }/* +* Ignore this pesky SMP stuff, use (4). +*/ +static long calc_up_shares (struct cfs_rq *cfs_rq) +{ +struct task_group *tg = cfs_rq->tg; +return READ_ONCE (tg->shares); +} + +DEFINE_STATIC_CALL (calc_group_shares, calc_smp_shares); + +void __sched_cgroup_mode_update (int mode) +{ +long (*func)(struct cfs_rq *); +switch (mode) { +case 0 : + func = &calc_up_shares; +break ; +case 1 : +default : + func = &calc_smp_shares; +break ; + } +static_call_update (calc_group_shares, func); +} + +/* * Recomputes the group entity based on the current state of its group * runqueue. */
@@ -4850,7 +4877,7 @@ static void update_cfs_group(struct sched_entity *se)if (!gcfs_rq || !gcfs_rq->load.weight)return ; - shares =calc_group_shares (gcfs_rq); + shares =static_call (calc_group_shares)(gcfs_rq);if (unlikely (se->load.weight != shares))reweight_entity (cfs_rq_of (se), se, shares); }
kernel/sched/sched.h +1 -0
@@ -571,6 +571,7 @@ extern void free_fair_sched_group(struct task_group *tg);extern int alloc_fair_sched_group (struct task_group *tg,struct task_group *parent);extern void online_fair_sched_group (struct task_group *tg);extern void unregister_fair_sched_group (struct task_group *tg); +extern void __sched_cgroup_mode_update (int mode);#else /* !CONFIG_FAIR_GROUP_SCHED: */ static inline void free_fair_sched_group (struct task_group *tg) { }static inline int alloc_fair_sched_group (struct task_group *tg,struct task_group *parent)
Peter Zijlstra
maintainerPATCH v3 3/7 sched/fair: Add cgroup_mode: max
In reply to Peter ZijlstraIn order to avoid the average CPU fraction avg(F_g_n) becoming tiny '1/N', assume each cgroup is maximally concurrent and distrubute 'N*weight', such that:
F_g_n' = N * F_g_n
Giving:
avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1
And while this sounds like it solves things, remember what that ~ meant. There is the corner case when a cgroup is minimally loaded, eg a single runnable task, therefore limit the CPU fraction to that of a nice -20 task to avoid getting too much load.
This last bit is what makes it different from a previous proposal to allow raising cpu.weight to '100 * N', that would not limit the mininal concurrency case and results in a very large F_g_n. And just like F_g_n << 1 is problematic, so is F_g_n >> 1 for the exact same reasons (it would drown the kthreads, but it also risks overflowing the load values).
So while this might appear to be a better scheme than the current default scheme, it doesn't really handle less than maximal concurrency nicely -- it clips and introduces artificially large weights. So where the traditional SMP mode works well when nr_tasks << nr_cpus, MAX doesn't work well in that regime and vice-versa.
The meaning of "cpu.weight" would be: weight per allowed CPU.
Included for completeness (and infrastructure).
--- include/linux/cpuset.h | 6 +++++ kernel/cgroup/cpuset.c | 15 ++++++++++++++ kernel/sched/debug.c | 1 kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++++++++----- 4 files changed, 69 insertions(+), 5 deletions(-)
include/linux/cpuset.h +6 -0
@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_hextern void cpuset_cpus_allowed_locked (struct task_struct *p,struct cpumask *mask);extern void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask);extern bool cpuset_cpus_allowed_fallback (struct task_struct *p); +extern int cpuset_num_cpus (struct cgroup *cgroup);extern nodemask_t cpuset_mems_allowed (struct task_struct *p);#define cpuset_current_mems_allowed (current->mems_allowed)void cpuset_init_current_mems_allowed (void );
@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_freturn false; } +static inline int cpuset_num_cpus (struct cgroup *cgroup) +{ +return num_online_cpus (); +} +static inline nodemask_t cpuset_mems_allowed (struct task_struct *p) {return node_possible_map;
kernel/cgroup/cpuset.c +15 -0
@@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(structreturn changed; } +int cpuset_num_cpus (struct cgroup *cgrp) +{ +int nr =num_online_cpus (); +struct cpuset *cs; + +if (is_in_v2_mode ()) { +guard (rcu)(); + cs =css_cs (cgroup_e_css (cgrp, &cpuset_cgrp_subsys)); +if (cs) + nr =cpumask_weight (cs->effective_cpus); + } + +return nr; +} +void __init cpuset_init_current_mems_allowed (void ) {nodes_setall (current->mems_allowed);
kernel/sched/debug.c +1 -0
@@ -641,6 +641,7 @@ static int cgroup_mode = 1;static const char *cgroup_mode_str[] = {"up" ,"smp" , +"max" , };static int sched_cgroup_mode (const char *str)
kernel/sched/fair.c +47 -5
72 lines not on this page · raw message
Waiman Long
maintainerRe: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max
Quoted 71 lines · Peter Zijlstra
In order to avoid the average CPU fraction
avg(F_g_n)becoming tiny '1/N', assume each cgroup is maximally concurrent and distrubute 'N*weight', such that:F_g_n' = N * F_g_nGiving:
avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1And while this sounds like it solves things, remember what that ~ meant. There is the corner case when a cgroup is minimally loaded, eg a single runnable task, therefore limit the CPU fraction to that of a nice -20 task to avoid getting too much load.
This last bit is what makes it different from a previous proposal to allow raising cpu.weight to '100 * N', that would not limit the mininal concurrency case and results in a very large
F_g_n. And just likeF_g_n<< 1 is problematic, so isF_g_n>> 1 for the exact same reasons (it would drown the kthreads, but it also risks overflowing the load values).So while this might appear to be a better scheme than the current default scheme, it doesn't really handle less than maximal concurrency nicely -- it clips and introduces artificially large weights. So where the traditional SMP mode works well when
nr_tasks<<nr_cpus, MAX doesn't work well in that regime and vice-versa.The meaning of "cpu.weight" would be: weight per allowed CPU.
Included for completeness (and infrastructure).
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- include/linux/cpuset.h | 6 +++++ kernel/cgroup/cpuset.c | 15 ++++++++++++++ kernel/sched/debug.c | 1 kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++++++++----- 4 files changed, 69 insertions(+), 5 deletions(-)include/linux/cpuset.h +6
-0@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_hextern void cpuset_cpus_allowed_locked (struct task_struct *p,struct cpumask *mask);extern void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask);extern bool cpuset_cpus_allowed_fallback (struct task_struct *p); +extern int cpuset_num_cpus (struct cgroup *cgroup);extern nodemask_t cpuset_mems_allowed (struct task_struct *p);#define cpuset_current_mems_allowed (current->mems_allowed)void cpuset_init_current_mems_allowed (void );@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_freturn false; } +static inline int cpuset_num_cpus (struct cgroup *cgroup) +{ +return num_online_cpus (); +} +static inline nodemask_t cpuset_mems_allowed (struct task_struct *p) {return node_possible_map;kernel/cgroup/cpuset.c +4
-0@@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(structreturn changed; } +int cpuset_num_cpus (struct cgroup *cgrp) +{ +int nr =num_online_cpus (); +struct cpuset *cs;
kernel/cgroup/cpuset.c bool cpuset_cpus_allowed_fallback(struct · line 4132
Quoted 10 lines
kernel/cgroup/cpuset.c +10
-0+ +if (is_in_v2_mode ()) { +guard (rcu)(); + cs =css_cs (cgroup_e_css (cgrp, &cpuset_cgrp_subsys)); +if (cs) + nr =cpumask_weight (cs->effective_cpus); + } + +return nr; +}
I just have a question about cgroup v1 support. I am assuming that cgroup v1 without the cpuset_v2_mode mount option is not supported. To fully support cgroup v1, you may have to use guarantee_active_cpus() to return the actual set of CPUs that the task can run on. Also there is a caveat about the arm64 specific task_cpu_possible_mask() for certain arm64 CPUs. That is for 32-bit binary running on 64-bit core which are allowed only on a selected subset of cores within the CPU.
This is probably not what you want to focus on right now, but it will be good to have a comment to list items that are not fully supported here.
Cheers,
Longman
Waiman Long
maintainerRe: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max
Quoted 85 lines · Waiman Long
Quoted 71 lines · Peter Zijlstra
In order to avoid the average CPU fraction
avg(F_g_n)becoming tiny '1/N', assume each cgroup is maximally concurrent and distrubute 'N*weight', such that:F_g_n' = N *
F_g_nGiving:
avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1
And while this sounds like it solves things, remember what that ~ meant. There is the corner case when a cgroup is minimally loaded, eg a single runnable task, therefore limit the CPU fraction to that of a nice -20 task to avoid getting too much load.
This last bit is what makes it different from a previous proposal to allow raising cpu.weight to '100 * N', that would not limit the mininal concurrency case and results in a very large
F_g_n. And just likeF_g_n<< 1 is problematic, so isF_g_n>> 1 for the exact same reasons (it would drown the kthreads, but it also risks overflowing the load values).So while this might appear to be a better scheme than the current default scheme, it doesn't really handle less than maximal concurrency nicely -- it
clips and introduces artificially large weights. So where the traditional SMP mode works well whennr_tasks<<nr_cpus, MAX doesn't work well in that regime and vice-versa.The meaning of "cpu.weight" would be: weight per allowed CPU.
Included for completeness (and infrastructure).
Signed-off-by Peter Zijlstra (Intel) <[email protected]>---
include/linux/cpuset.h| 6 +++++
kernel/cgroup/cpuset.c| 15 ++++++++++++++
kernel/sched/debug.c | 1
kernel/sched/fair.c | 52
++++++++++++++++++++++++++++++++++++++++++++-----4 files changed, 69 insertions(+), 5 deletions(-)include/linux/cpuset.h +5
-0@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_h extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask); extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask); extern bool cpuset_cpus_allowed_fallback(struct task_struct *p); +extern int cpuset_num_cpus (struct cgroup *cgroup); extern nodemask_t cpuset_mems_allowed(struct task_struct *p); #define cpuset_current_mems_allowed (current->mems_allowed) void cpuset_init_current_mems_allowed(void);@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_f return false; } +static inline int cpuset_num_cpus(struct cgroup *cgroup) +{ + return num_online_cpus(); +} + static inline nodemask_t cpuset_mems_allowed(struct task_struct *p) { return node_possible_map;kernel/cgroup/cpuset.c +3
-0@@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct return changed; } +int cpuset_num_cpus(struct cgroup *cgrp) +{ + int nr = num_online_cpus(); + struct cpuset *cs;kernel/cgroup/cpuset.c bool cpuset_cpus_allowed_fallback(struct · line 4131
Quoted 10 lines
kernel/cgroup/cpuset.c +10
-0+ + if (is_in_v2_mode()) { + guard(rcu)(); + cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys)); + if (cs) + nr = cpumask_weight(cs->effective_cpus); + } + + return nr; +}I just have a question about cgroup v1 support. I am assuming that cgroup v1 without the
cpuset_v2_modemount option is not supported. To fully support cgroup v1, you may have to useguarantee_active_cpus()to return the actual set of CPUs that the task can run on. Also there is a caveat about the arm64 specifictask_cpu_possible_mask()for certain arm64 CPUs. That is for 32-bit binary running on 64-bit core which are allowed only on a selected subset of cores within the CPU.
Quoted 1 line
This is probably not what you want to focus on right now, but it will be good to have a comment to list items that are not fully supported here.
FYI, you may have to take the callback_lock to ensure the stability of the effective_cpus mask.
Cheers,
Longman
Peter Zijlstra
maintainerRe: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max
Quoted 14 lines · Waiman Long
Quoted 14 lines
Quoted 14 lines
kernel/cgroup/cpuset.c +8
-0@@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct return changed; } +int cpuset_num_cpus(struct cgroup *cgrp) +{ + int nr = num_online_cpus(); + struct cpuset *cs; + + if (is_in_v2_mode()) { + guard(rcu)(); + cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys)); + if (cs)
Quoted 8 lines
Quoted 5 lines
kernel/cgroup/cpuset.c bool cpuset_cpus_allowed_fallback(struct · line 4132
Quoted 5 lines
kernel/cgroup/cpuset.c +5
-0+ nr = cpumask_weight(cs->effective_cpus); + } + + return nr; +}FYI, you may have to take the
callback_lockto ensure the stability of theeffective_cpusmask.
That seems pointless, the moment we drop that lock, its changeable again. Either way around nr is but a snapshot.
Peter Zijlstra
maintainerRe: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max
In reply to Waiman LongQuoted 14 lines · Waiman Long
Quoted 14 lines
kernel/cgroup/cpuset.c +9
-0@@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(structreturn changed; } +int cpuset_num_cpus (struct cgroup *cgrp) +{ +int nr =num_online_cpus (); +struct cpuset *cs; + +if (is_in_v2_mode ()) { +guard (rcu)(); + cs =css_cs (cgroup_e_css (cgrp, &cpuset_cgrp_subsys)); +if (cs)
Quoted 8 lines
kernel/cgroup/cpuset.c bool cpuset_cpus_allowed_fallback(struct · line 4132
Quoted 5 lines
kernel/cgroup/cpuset.c +5
-0+ nr =cpumask_weight (cs->effective_cpus); + } + +return nr; +}I just have a question about cgroup v1 support. I am assuming that cgroup v1 without the
cpuset_v2_modemount option is not supported.
Correct.
Quoted 3 lines
To fully support
cgroup v1, you may have to useguarantee_active_cpus()to return the actual set of CPUs that the task can run on.
Except this is group based, we'd need an iteration of all tasks in the group and compute a union of guarantee_active_cpus(). Which all seems far too expensive and not worth the effort.
Quoted 7 lines
Also there is a caveat about the arm64 specific
task_cpu_possible_mask()for certain arm64 CPUs. That is for 32-bit binary running on 64-bit core which are allowed only on a selected subset of cores within the CPU.This is probably not what you want to focus on right now, but it will be good to have a comment to list items that are not fully supported here.
Will add a comment!
Waiman Long
maintainerRe: [PATCH v3 3/7] sched/fair: Add cgroup_mode: max
Quoted 23 lines · Peter Zijlstra
Quoted 13 lines · Waiman Long
Quoted 13 lines
kernel/cgroup/cpuset.c +8
-0@@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(structreturn changed; } +int cpuset_num_cpus (struct cgroup *cgrp) +{ +int nr =num_online_cpus (); +struct cpuset *cs; + +if (is_in_v2_mode ()) { +guard (rcu)(); + cs =css_cs (cgroup_e_css (cgrp, &cpuset_cgrp_subsys));Quoted 8 lines
kernel/cgroup/cpuset.c bool cpuset_cpus_allowed_fallback(struct · line 4132
Quoted 6 lines
kernel/cgroup/cpuset.c +6
-0+if (cs) + nr =cpumask_weight (cs->effective_cpus); + } + +return nr; +}I just have a question about cgroup v1 support. I am assuming that cgroup v1 without the
cpuset_v2_modemount option is not supported.
Quoted 8 lines
Correct.
Quoted 3 lines
To fully support
cgroup v1, you may have to useguarantee_active_cpus()to return the actual set of CPUs that the task can run on.Except this is group based, we'd need an iteration of all tasks in the group and compute a union of
guarantee_active_cpus(). Which all seems far too expensive and not worth the effort.
I thought so.
Quoted 9 lines
Quoted 8 lines
Also there is a caveat about the arm64 specific
task_cpu_possible_mask()for certain arm64 CPUs. That is for 32-bit binary running on 64-bit core which are allowed only on a selected subset of cores within the CPU.This is probably not what you want to focus on right now, but it will be good to have a comment to list items that are not fully supported here.
Will add a comment!
Thanks,
Longman
[tip: sched/core] sched/fair: Add cgroup_mode: max
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 90ac22ffef48dbc6e7561434b6e01753a859bb51 Gitweb: https://git.kernel.org/tip/90ac22ffef48dbc6e7561434b6e01753a859bb51 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Mar 2026 15:42:35 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:52 +02:00
sched/fair: Add cgroup_mode: max
In order to avoid the average CPU fraction avg(F_g_n) becoming tiny '1/N', assume each cgroup is maximally concurrent and distrubute 'N*weight', such that:
F_g_n' = N * F_g_n
Giving:
avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1
And while this sounds like it solves things, remember what that ~ meant. There is the corner case when a cgroup is minimally loaded, eg a single runnable task, therefore limit the CPU fraction to that of a nice -20 task to avoid getting too much load.
This last bit is what makes it different from a previous proposal to allow raising cpu.weight to '100 * N', that would not limit the mininal concurrency case and results in a very large F_g_n. And just like F_g_n << 1 is problematic, so is F_g_n >> 1 for the exact same reasons (it would drown the kthreads, but it also risks overflowing the load values).
So while this might appear to be a better scheme than the current default scheme, it doesn't really handle less than maximal concurrency nicely -- it clips and introduces artificially large weights. So where the traditional SMP mode works well when nr_tasks << nr_cpus, MAX doesn't work well in that regime and vice-versa.
The meaning of "cpu.weight" would be: weight per allowed CPU.
Included for completeness (and infrastructure).
--- include/linux/cpuset.h | 6 +++++- kernel/cgroup/cpuset.c | 22 +++++++++++++++++- kernel/sched/debug.c | 1 +- kernel/sched/fair.c | 52 +++++++++++++++++++++++++++++++++++++---- 4 files changed, 76 insertions(+), 5 deletions(-)
include/linux/cpuset.h +6 -0
@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_held(void);extern void cpuset_cpus_allowed_locked (struct task_struct *p,struct cpumask *mask);extern void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask);extern bool cpuset_cpus_allowed_fallback (struct task_struct *p); +extern int cpuset_num_cpus (struct cgroup *cgroup);extern nodemask_t cpuset_mems_allowed (struct task_struct *p);#define cpuset_current_mems_allowed (current->mems_allowed)void cpuset_init_current_mems_allowed (void );
@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_fallback(struct task_struct *p)return false; } +static inline int cpuset_num_cpus (struct cgroup *cgroup) +{ +return num_online_cpus (); +} +static inline nodemask_t cpuset_mems_allowed (struct task_struct *p) {return node_possible_map;
kernel/cgroup/cpuset.c +22 -0
@@ -4124,6 +4124,28 @@ bool cpuset_cpus_allowed_fallback(struct task_struct *tsk)return changed; } +/* +* Returns the number of CPUs available for this cgroup. +* +* This only really works for cgroup-v2 where all the controllers are mounted +* in the same hierarchy. If not cgroup-v2 or no cpuset controller is +* configured it reverts to num_online_cpus(). +*/ +int cpuset_num_cpus (struct cgroup *cgrp) +{ +int nr =num_online_cpus (); +struct cpuset *cs; + +if (is_in_v2_mode ()) { +guard (rcu)(); + cs =css_cs (cgroup_e_css (cgrp, &cpuset_cgrp_subsys)); +if (cs) + nr =cpumask_weight (cs->effective_cpus); + } + +return nr; +} +void __init cpuset_init_current_mems_allowed (void ) {nodes_setall (current->mems_allowed);
kernel/sched/debug.c +1 -0
@@ -640,6 +640,7 @@ static int cgroup_mode = 1;static const char *cgroup_mode_str[] = {"up" ,"smp" , +"max" , };static int sched_cgroup_mode (const char *str)
kernel/sched/fair.c +47 -5
72 lines not on this page · raw message
Peter Zijlstra
maintainerPATCH v3 4/7 sched/fair: Add cgroup_mode: concur
In reply to Peter ZijlstraImprove upon the previous scheme ("max") by no longer assuming maximal concurrency. Instead scale by: 'min(nr_tasks, nr_cpus)'. This handles the low concurrency cases more gracefully:
F_g_n' = min(M, N) * F_g_n
Notably this is the first mode where:
avg(F_g_n) = 1
In the single task case it reduces to ("smp") and then it nicely scales up until it hits N, where it behaves like ("max").
This is no longer clipped at nice -20. Strictly speaking it isn't different from the normal SMP scenario where all tasks are extremely unbalanced. There are no unnatural inflations in this scheme.
The meaning of "cpu.weight" would be: weight per active CPU.
NOTE: Compute the group wide number of tasks by extending the tg->load_avg computation with tg->runnable_avg, since cfs_rq->runnable_avg is based on cfs_rq->h_nr_running.
--- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 43 ++++++++++++++++++++++++++++++++++++------- kernel/sched/sched.h | 3 +++ 3 files changed, 40 insertions(+), 7 deletions(-)
kernel/sched/debug.c +1 -0
@@ -641,6 +641,7 @@ static int cgroup_mode = 1;static const char *cgroup_mode_str[] = {"up" ,"smp" , +"concur" ,"max" , };
kernel/sched/fair.c +36 -7
87 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -493,6 +493,8 @@ struct task_group {* will also be accessed at each tick. */ atomic_long_t load_avg____cacheline_aligned ; +atomic_long_t runnable_avg; +#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED
@@ -722,6 +724,7 @@ struct cfs_rq {#ifdef CONFIG_FAIR_GROUP_SCHED u64 last_update_tg_load_avg;unsigned long tg_load_avg_contrib; +unsigned long tg_runnable_avg_contrib;long propagate;long prop_runnable_sum;
[tip: sched/core] sched/fair: Add cgroup_mode: concur
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5c0b58bd1c9f09c4e9363665702c82694c6ee7a0 Gitweb: https://git.kernel.org/tip/5c0b58bd1c9f09c4e9363665702c82694c6ee7a0 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Mar 2026 16:04:18 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:52 +02:00
sched/fair: Add cgroup_mode: concur
Improve upon the previous scheme ("max") by no longer assuming maximal concurrency. Instead scale by: 'min(nr_tasks, nr_cpus)'. This handles the low concurrency cases more gracefully:
F_g_n' = min(M, N) * F_g_n
Notably this is the first mode where:
avg(F_g_n) = 1
In the single task case it reduces to ("smp") and then it nicely scales up until it hits N, where it behaves like ("max").
This is no longer clipped at nice -20. Strictly speaking it isn't different from the normal SMP scenario where all tasks are extremely unbalanced. There are no unnatural inflations in this scheme.
The meaning of "cpu.weight" would be: weight per active CPU.
NOTE: Compute the group wide number of tasks by extending the tg->load_avg computation with tg->runnable_avg, since cfs_rq->runnable_avg is based on cfs_rq->h_nr_running.
--- kernel/sched/debug.c | 1 +- kernel/sched/fair.c | 43 ++++++++++++++++++++++++++++++++++++------- kernel/sched/sched.h | 3 +++- 3 files changed, 40 insertions(+), 7 deletions(-)
kernel/sched/debug.c +1 -0
@@ -640,6 +640,7 @@ static int cgroup_mode = 1;static const char *cgroup_mode_str[] = {"up" ,"smp" , +"concur" ,"max" , };
kernel/sched/fair.c +36 -7
87 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -493,6 +493,8 @@ struct task_group {* will also be accessed at each tick. */ atomic_long_t load_avg____cacheline_aligned ; +atomic_long_t runnable_avg; +#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED
@@ -722,6 +724,7 @@ struct cfs_rq {#ifdef CONFIG_FAIR_GROUP_SCHED u64 last_update_tg_load_avg;unsigned long tg_load_avg_contrib; +unsigned long tg_runnable_avg_contrib;long propagate;long prop_runnable_sum;
Peter Zijlstra
maintainerPATCH v3 5/7 sched/fair: Add cgroup_mode: tasks
In reply to Peter ZijlstraSince we are exploring this space; include a scheme that scales by total number of runnable tasks. This results in:
F_g_n' = M * F_g_n
This will obviously have: avg(F_g_n') > 1, (it will be ~M/N in fact).
And while that sounds odd, it actually has a fairly straight foward meaning for "cpu.weight": average weight per member task.
This is an entirely valid and workable option, it is however wildly different from the traditional meaning.
Included for completeness (and curiosity).
--- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 16 ++++++++++++++++ 2 files changed, 17 insertions(+)
kernel/sched/debug.c +1 -0
@@ -643,6 +643,7 @@ static const char *cgroup_mode_str[] = {"smp" ,"concur" ,"max" , + "tasks", };static int sched_cgroup_mode (const char *str)
kernel/sched/fair.c +16 -0
@@ -4852,6 +4852,19 @@ static inline int tg_tasks(struct task_g }/* +* Func: fraction(nr_tasks * tg->shares) +* +* Scale tg->shares by the number of tasks. +*/ +static long calc_tasks_shares (struct cfs_rq *cfs_rq) +{ +struct task_group *tg = cfs_rq->tg; +int nr =tg_tasks (tg); +long tg_shares =READ_ONCE (tg->shares); +return __calc_smp_shares (cfs_rq, nr * tg_shares, nr * tg_shares); +} + +/* * Func: min(fraction(nr_cpus * tg->shares), nice -20) * * Scale tg->shares by the maximal number of CPUs; but clip the max shares at
@@ -4921,6 +4934,9 @@ void __sched_cgroup_mode_update(int modecase 3 : func = &calc_max_shares;break ; +case 4 : + func = &calc_tasks_shares; +break ; }static_call_update (calc_group_shares, func); }
[tip: sched/core] sched/fair: Add cgroup_mode: tasks
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 507f910a4e1f140a245b693441b42bcbc8128938 Gitweb: https://git.kernel.org/tip/507f910a4e1f140a245b693441b42bcbc8128938 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 05 Jun 2026 10:35:01 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:53 +02:00
sched/fair: Add cgroup_mode: tasks
Since we are exploring this space; include a scheme that scales by total number of runnable tasks. This results in:
F_g_n' = M * F_g_n
This will obviously have: avg(F_g_n') > 1, (it will be ~M/N in fact).
And while that sounds odd, it actually has a fairly straight foward meaning for "cpu.weight": average weight per member task.
This is an entirely valid and workable option, it is however wildly different from the traditional meaning.
Included for completeness (and curiosity).
--- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 16 ++++++++++++++++ 2 files changed, 17 insertions(+)
kernel/sched/debug.c +1 -0
@@ -642,6 +642,7 @@ static const char *cgroup_mode_str[] = {"smp" ,"concur" ,"max" , + "tasks", };static int sched_cgroup_mode (const char *str)
kernel/sched/fair.c +16 -0
@@ -4852,6 +4852,19 @@ static inline int tg_tasks(struct task_group *tg) }/* +* Func: fraction(nr_tasks * tg->shares) +* +* Scale tg->shares by the number of tasks. +*/ +static long calc_tasks_shares (struct cfs_rq *cfs_rq) +{ +struct task_group *tg = cfs_rq->tg; +int nr =tg_tasks (tg); +long tg_shares =READ_ONCE (tg->shares); +return __calc_smp_shares (cfs_rq, nr * tg_shares, nr * tg_shares); +} + +/* * Func: min(fraction(nr_cpus * tg->shares), nice -20) * * Scale tg->shares by the maximal number of CPUs; but clip the max shares at
@@ -4921,6 +4934,9 @@ void __sched_cgroup_mode_update(int mode)case 3 : func = &calc_max_shares;break ; +case 4 : + func = &calc_tasks_shares; +break ; }static_call_update (calc_group_shares, func); }
Peter Zijlstra
maintainerPATCH v3 6/7 sched/fair: Change the default cgroup_mode to concur
In reply to Peter ZijlstraFor all the reasons described in the preceding patches, the way cgroup weight is computed is problematic. However, changing it is bound to also lead to trouble. Esp. since people might have taken to inflating the weight value where they can.
Since things are configurable, change the default and hope this serves more people than it hurts, esp. in the longer run.
Specifically, this prepares for a flattened runqueue, where the hierarchical weight becomes far more important (F_g^d terms), so getting rid of small F_g is imperative.
--- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
kernel/sched/debug.c +1 -1
@@ -635,7 +635,7 @@ static void debugfs_fair_server_init(voi }#ifdef CONFIG_FAIR_GROUP_SCHED -static int cgroup_mode =1 ; +static int cgroup_mode =2 ;/* See __sched_cgroup_mode_update(). */ static const char *cgroup_mode_str[] = {
kernel/sched/fair.c +2 -2
@@ -4921,7 +4921,7 @@ static long calc_up_shares(struct cfs_rqreturn READ_ONCE (tg->shares); } -DEFINE_STATIC_CALL (calc_group_shares, calc_smp_shares); +DEFINE_STATIC_CALL (calc_group_shares, calc_concur_shares);void __sched_cgroup_mode_update (int mode) {
@@ -4931,10 +4931,10 @@ void __sched_cgroup_mode_update(int mode func = &calc_up_shares;break ;case 1 : -default : func = &calc_smp_shares;break ;case 2 : +default : func = &calc_concur_shares;break ;case 3 :
[tip: sched/core] sched/fair: Change the default cgroup_mode to concur
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: fb1050ac8e40aaa014291b5b53612d27890c2b09 Gitweb: https://git.kernel.org/tip/fb1050ac8e40aaa014291b5b53612d27890c2b09 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 04 Jun 2026 15:15:51 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:53 +02:00
sched/fair: Change the default cgroup_mode to concur
For all the reasons described in the preceding patches, the way cgroup weight is computed is problematic. However, changing it is bound to also lead to trouble. Esp. since people might have taken to inflating the weight value where they can.
Since things are configurable, change the default and hope this serves more people than it hurts, esp. in the longer run.
Specifically, this prepares for a flattened runqueue, where the hierarchical weight becomes far more important (F_g^d terms), so getting rid of small F_g is imperative.
--- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
kernel/sched/debug.c +1 -1
@@ -634,7 +634,7 @@ static void debugfs_fair_server_init(void) }#ifdef CONFIG_FAIR_GROUP_SCHED -static int cgroup_mode =1 ; +static int cgroup_mode =2 ;/* See __sched_cgroup_mode_update(). */ static const char *cgroup_mode_str[] = {
kernel/sched/fair.c +2 -2
@@ -4915,7 +4915,7 @@ static long calc_up_shares(struct cfs_rq *cfs_rq)return READ_ONCE (tg->shares); } -DEFINE_STATIC_CALL (calc_group_shares, calc_smp_shares); +DEFINE_STATIC_CALL (calc_group_shares, calc_concur_shares);void __sched_cgroup_mode_update (int mode) {
@@ -4925,10 +4925,10 @@ void __sched_cgroup_mode_update(int mode) func = &calc_up_shares;break ;case 1 : -default : func = &calc_smp_shares;break ;case 2 : +default : func = &calc_concur_shares;break ;case 3 :
Peter Zijlstra
maintainerPATCH v3 7/7 sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraChange fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the shares_weight approximation are fully preserved.
That is, given a hierarchy like:
R
|
se--G1
/ \
G2--se se--G3
/ \ |T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:
_R_
/ | \T1 T2 T3
Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (se->h_load) is computed from the hierarchical fraction: se->load / cfs_rq->load.
Since EEVDF is now exclusively operating on rq->cfs, it needs to consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.
One place where additional information was required was set_next_task() / put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).
As a result of only having a single level to pick from, much of the complications in pick_next_task() and preemption go away.
Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.
TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr
--- include/linux/sched.h | 1 kernel/sched/core.c | 5 kernel/sched/debug.c | 9 kernel/sched/fair.c | 803 +++++++++++++++++++++----------------------------- kernel/sched/pelt.c | 6 kernel/sched/sched.h | 26 - 6 files changed, 375 insertions(+), 475 deletions(-)
include/linux/sched.h +1 -0
@@ -574,6 +574,7 @@ struct sched_statistics {struct sched_entity {/* For load-balancing: */ struct load_weight load; +struct load_weight h_load;struct rb_node run_node;u64 deadline;u64 min_vruntime;
kernel/sched/core.c +1 -4
@@ -5652,11 +5652,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);*/ static inline void prefetch_curr_exec_start (struct task_struct *p) { -#ifdef CONFIG_FAIR_GROUP_SCHED -struct sched_entity *curr = p->se.cfs_rq->curr; -#else struct sched_entity *curr =task_rq (p)->cfs.curr; -#endif +prefetch (curr);prefetch (&curr->exec_start); }
kernel/sched/debug.c +6 -3
@@ -975,10 +975,11 @@ print_task(struct seq_file *m, struct rqelse SEQ_printf (m," %c" ,task_state_to_char (p)); - SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", + SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), + p->se.h_load.weight, SPLIT_NS(p->se.vruntime), - entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', + entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice),
@@ -1007,7 +1008,7 @@ static void print_rq(struct seq_file *m,SEQ_printf (m,"\n" );SEQ_printf (m,"runnable tasks:\n" ); -SEQ_printf (m," S task PID vruntime eligible " +SEQ_printf (m," S task PID weight vruntime eligible " "deadline slice sum-exec switches " "prio wait-time sum-sleep sum-block" #ifdef CONFIG_NUMA_BALANCING
@@ -1115,6 +1116,8 @@ void print_cfs_rq(struct seq_file *m, in cfs_rq->tg_load_avg_contrib);SEQ_printf (m," .%-30s: %ld\n" ,"tg_load_avg" ,atomic_long_read (&cfs_rq->tg->load_avg)); +SEQ_printf (m," .%-30s: %lu\n" ,"h_load" , + cfs_rq->h_load);#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_CFS_BANDWIDTH SEQ_printf(m, " .%-30s: %d\n", "throttled",
kernel/sched/fair.c +351 -452
1,408 lines not on this page · raw message
kernel/sched/pelt.c +3 -3
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched/* * running is a subset of runnable (weight) so running can't be set if * runnable is clear. But there are some corner cases where the current -* se has been already dequeued but cfs_rq->curr still points to it. +* se has been already dequeued but cfs_rq->h_curr still points to it. * This means that weight will be 0 but not running for a sched_entity * but also for a cfs_rq if the latter becomes idle. As an example, * this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 nowint __update_load_avg_se (u64 now,struct cfs_rq *cfs_rq,struct sched_entity *se) {if (___update_load_sum (now, &se->avg, !!se->on_rq,se_runnable (se), - cfs_rq->curr == se)) { + cfs_rq->h_curr == se)) {___update_load_avg (&se->avg,se_weight (se));cfs_se_util_change (&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, stif (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), cfs_rq->h_nr_runnable, - cfs_rq->curr !=NULL )) { + cfs_rq->h_curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );trace_pelt_cfs_tp (cfs_rq);
kernel/sched/sched.h +13 -13
@@ -530,21 +530,8 @@ struct task_group { }; -#ifdef CONFIG_GROUP_SCHED_WEIGHT #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD -/* -* A weight of 0 or 1 can cause arithmetics problems. -* A weight of a cfs_rq is the sum of weights of which entities -* are queued on this cfs_rq, so a weight of a entity should not be -* too large, so as the shares value of a task group. -* (The default weight is 1024 - so there's no practical -* limitation from this.) -*/ -#define MIN_SHARES (1UL << 1) -#define MAX_SHARES (1UL << 18) -#endif -typedef int (*tg_visitor )(struct task_group *,void *); extern int walk_tg_tree_from(struct task_group *from,
@@ -631,6 +618,17 @@ static inline bool cfs_task_bw_constrain#endif /* !CONFIG_CGROUP_SCHED */ +/* +* A weight of 0 or 1 can cause arithmetics problems. +* A weight of a cfs_rq is the sum of weights of which entities +* are queued on this cfs_rq, so a weight of a entity should not be +* too large, so as the shares value of a task group. +* (The default weight is 1024 - so there's no practical +* limitation from this.) +*/ +#define MIN_SHARES (1UL << 1) +#define MAX_SHARES (1UL << 18) +extern void unregister_rt_sched_group (struct task_group *tg);extern void free_rt_sched_group (struct task_group *tg);extern int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent);
@@ -709,6 +707,7 @@ struct cfs_rq {/* * CFS load tracking */ +struct sched_entity *h_curr;struct sched_avg avg;#ifndef CONFIG_64BIT u64 last_update_time_copy;
@@ -2575,6 +2574,7 @@ extern const u32 sched_prio_to_wmult[40#define ENQUEUE_MIGRATED 0x00040000#define ENQUEUE_INITIAL 0x00080000#define ENQUEUE_RQ_SELECTED 0x00100000 +#define ENQUEUE_QUEUED 0x00200000#define RETRY_TASK ((void *)-1UL)
Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 48 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the
shares_weightapproximation are fully preserved.That is, given a hierarchy like:
R | se--G1 / \ G2--se se--G3 / \ |T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:
_R_ / | \ T1 T2 T3Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (
se->h_load) is computed from the hierarchical fraction:se->load/cfs_rq->load.Since EEVDF is now exclusively operating on
rq->cfs, it needs to considercfs_rq->h_nr_queuedrather thancfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.One place where additional information was required was
set_next_task()/put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).As a result of only having a single level to pick from, much of the complications in
pick_next_task()and preemption go away.
Quoted 8 lines
Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.
TODO: split struct
cfs_rq/ structsched_entity
TODO: try and get rid ofh_currSigned-off-by Peter Zijlstra (Intel) <[email protected]>
A divide-by-zero crash is observed when running hackbench:
[14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not tainted 7.1.0-rc2+ [14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0 [14697.506799] <TASK> [14697.507411] __dequeue_task+0x2b4/0xc70 [14697.508677] dequeue_task_fair+0x36/0x370 [14697.509047] dequeue_task+0x101/0x2f0 [14697.509426] __schedule+0x1b1/0x1a00 [14697.510868] anon_pipe_read+0x3da/0x450 [14697.511400] vfs_read+0x361/0x390 [14697.512053] __x64_sys_read+0x19/0x30
The divide-by-zero happens here:
if (scale_load_down(gcfs_rq->load.weight)) {
load_sum = div_u64(gcfs_rq->avg.load_sum,
scale_load_down(gcfs_rq->load.weight));
}gcfs_rq->load.weight is an insane large value and is truncated to the lower 32 bits by div_u64, which happen to be 0.
Using AI for investigation, the cause is a u32 overflow in update_tg_cfs_runnable(), and flat pickup became a victim when using tg_tasks():
u32 new_sum, divider;
...
new_sum = se->avg.runnable_avg * divider; <-- boom
The following sequence shows how this triggers the crash:
propagate_entity_load_avg()
update_tg_cfs_runnable() # u32 overflow corrupts runnable_sum
__update_load_avg_cfs_rq()
___update_load_avg() # computes insane runnable_avg
update_tg_load_avg() # propagates to tg->runnable_avg
update_cfs_group()
calc_concur_shares()
tg_tasks() # long-to-int truncation, negative nr
reweight_entity() # corrupted se->load.weight
update_load_add() # corrupted cfs_rq->load.weight propagate_entity_load_avg()
update_tg_cfs_load()
div_u64() # divide-by-zero
Fix by widening new_sum from u32 to u64(no need to force tg_tasks() to return unsigned long after this fix)
--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/sched/fair.c +0 -0
@@ -5305,7 +5305,8 @@ static inline void update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se,
struct cfs_rq *gcfs_rq)
{
long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg -se->avg.runnable_avg;
- u32 new_sum, divider;
+ u64 new_sum;
+ u32 divider;
/* Nothing to update */
if (!delta_avg)
@@ -5319,7 +5320,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf
/* Set new sched_entity's runnable */ se->avg.runnable_avg = gcfs_rq->avg.runnable_avg;
- new_sum = se->avg.runnable_avg * divider;
+ new_sum = (u64)se->avg.runnable_avg * divider;
delta_sum = (long)new_sum - (long)se->avg.runnable_sum; se->avg.runnable_sum = new_sum;
Signature · 1 line
2.45.2
Peter Zijlstra
maintainerRe: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 74 lines · Yu C
A divide-by-zero crash is observed when running hackbench:
[14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not tainted 7.1.0-rc2+ [14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0 [14697.506799] <TASK> [14697.507411] __dequeue_task+0x2b4/0xc70 [14697.508677] dequeue_task_fair+0x36/0x370 [14697.509047] dequeue_task+0x101/0x2f0 [14697.509426] __schedule+0x1b1/0x1a00 [14697.510868] anon_pipe_read+0x3da/0x450 [14697.511400] vfs_read+0x361/0x390 [14697.512053] __x64_sys_read+0x19/0x30The divide-by-zero happens here:
if (scale_load_down(gcfs_rq->load.weight)) { load_sum = div_u64(gcfs_rq->avg.load_sum, scale_load_down(gcfs_rq->load.weight)); }
gcfs_rq->load.weightis an insane large value and is truncated to the lower 32 bits bydiv_u64, which happen to be 0.Using AI for investigation, the cause is a u32 overflow in
update_tg_cfs_runnable(), and flat pickup became a victim when usingtg_tasks():u32
new_sum, divider;
...
new_sum=se->avg.runnable_avg* divider; <-- boomThe following sequence shows how this triggers the crash:
propagate_entity_load_avg()
update_tg_cfs_runnable()# u32 overflow corruptsrunnable_sum
__update_load_avg_cfs_rq()
___update_load_avg()# computes insanerunnable_avg
update_tg_load_avg()# propagates totg->runnable_avg
update_cfs_group()calc_concur_shares() tg_tasks() # long-to-int truncation, negative nr reweight_entity() # corrupted se->load.weight update_load_add() # corrupted cfs_rq->load.weight
propagate_entity_load_avg()
update_tg_cfs_load()
div_u64()# divide-by-zeroFix by widening
new_sumfrom u32 tou64(no need to force tg_tasks() to return unsigned long after this fix)Assisted-by Claude:claude-opus-4.6Signed-off-by Chen Yu <[email protected]>--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/sched/fair.c +2
-1@@ -5305,7 +5305,8 @@ static inline void update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cfs_rq *gcfs_rq) { long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg - se->avg.runnable_avg; -u32 new_sum, divider; +u64 new_sum; +u32 divider;/* Nothing to update */ if (!delta_avg)
kernel/sched/fair.c update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf · line 5325
Quoted 7 lines
kernel/sched/fair.c +1
-1@@ -5319,7 +5320,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf/* Set new sched_entity's runnable */ se->avg.runnable_avg = gcfs_rq->avg.runnable_avg; - new_sum = se->avg.runnable_avg * divider; + new_sum = (u64 )se->avg.runnable_avg * divider; delta_sum = (long )new_sum - (long )se->avg.runnable_sum; se->avg.runnable_sum = new_sum;
Hmm, nice one. This makes sense because sched_avg::runnable_sum is a u64 itself, so having the delta be one is only sensible.
I do wonder though, this doesn't actually look to be specific to flat, it just managed to trip it somehow.
I'll stick this on as a separate fix.
Vincent Guittot
maintainerRe: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 88 lines · Peter Zijlstra
Quoted 74 lines · Yu C
A divide-by-zero crash is observed when running hackbench:
[14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not tainted 7.1.0-rc2+ [14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0 [14697.506799] <TASK> [14697.507411] __dequeue_task+0x2b4/0xc70 [14697.508677] dequeue_task_fair+0x36/0x370 [14697.509047] dequeue_task+0x101/0x2f0 [14697.509426] __schedule+0x1b1/0x1a00 [14697.510868] anon_pipe_read+0x3da/0x450 [14697.511400] vfs_read+0x361/0x390 [14697.512053] __x64_sys_read+0x19/0x30The divide-by-zero happens here:
if (scale_load_down(gcfs_rq->load.weight)) { load_sum = div_u64(gcfs_rq->avg.load_sum, scale_load_down(gcfs_rq->load.weight)); }
gcfs_rq->load.weightis an insane large value and is truncated to the lower 32 bits bydiv_u64, which happen to be 0.Using AI for investigation, the cause is a u32 overflow in
update_tg_cfs_runnable(), and flat pickup became a victim when usingtg_tasks():u32
new_sum, divider;
...
new_sum=se->avg.runnable_avg* divider; <-- boomThe following sequence shows how this triggers the crash:
propagate_entity_load_avg()
update_tg_cfs_runnable()# u32 overflow corruptsrunnable_sum
__update_load_avg_cfs_rq()
___update_load_avg()# computes insanerunnable_avg
update_tg_load_avg()# propagates totg->runnable_avg
update_cfs_group()calc_concur_shares() tg_tasks() # long-to-int truncation, negative nr reweight_entity() # corrupted se->load.weight update_load_add() # corrupted cfs_rq->load.weight
propagate_entity_load_avg()
update_tg_cfs_load()
div_u64()# divide-by-zeroFix by widening
new_sumfrom u32 tou64(no need to force tg_tasks() to return unsigned long after this fix)Assisted-by Claude:claude-opus-4.6Signed-off-by Chen Yu <[email protected]>--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/sched/fair.c +2
-1@@ -5305,7 +5305,8 @@ static inline void update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cfs_rq *gcfs_rq) { long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg - se->avg.runnable_avg; -u32 new_sum, divider; +u64 new_sum; +u32 divider;/* Nothing to update */ if (!delta_avg)kernel/sched/fair.c update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf · line 5325
Quoted 7 lines
kernel/sched/fair.c +1
-1@@ -5319,7 +5320,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf/* Set new sched_entity's runnable */ se->avg.runnable_avg = gcfs_rq->avg.runnable_avg; - new_sum = se->avg.runnable_avg * divider; + new_sum = (u64 )se->avg.runnable_avg * divider; delta_sum = (long )new_sum - (long )se->avg.runnable_sum; se->avg.runnable_sum = new_sum;Hmm, nice one. This makes sense because
sched_avg::runnable_sumis a u64 itself, so having the delta be one is only sensible.
Quoted 2 lines
I do wonder though, this doesn't actually look to be specific to flat, it just managed to trip it somehow.
I think, it's specific to the new share policies introduced with flat. Before flat we were using:
return clamp_t(long, shares, MIN_SHARES, tg_shares); where tg_shares was clamp(shares, scale_load(MIN_SHARES), scale_load(MAX_SHARES));
With flat, we are using:
return clamp_t(long, shares, MIN_SHARES, shares_max); where shares_max can be:
max(1, atomic_long_read(&tg->runnable_avg) >>
SCHED_CAPACITY_SHIFT) * clamp(shares, scale_load(MIN_SHARES), scale_load(MAX_SHARES))
Quoted 2 lines
I'll stick this on as a separate fix.
[tip: sched/core] sched/fair: Fix overflow in update_tg_cfs_runnable()
In reply to Chen, Yu CThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4f166adb5cb0525d9e32d45729fd8f28c80acbee Gitweb: https://git.kernel.org/tip/4f166adb5cb0525d9e32d45729fd8f28c80acbee Author: Chen, Yu C <[email protected]> AuthorDate: Sat, 20 Jun 2026 11:54:22 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:51 +02:00
sched/fair: Fix overflow in update_tg_cfs_runnable()
A divide-by-zero crash is observed when running hackbench:
[14697.488452] CPU: 112 UID: 0 PID: 124791 Comm: hackbench Not tainted 7.1.0-rc2+ [14697.492627] RIP: 0010:propagate_entity_load_avg+0x35f/0x3e0 [14697.506799] <TASK> [14697.507411] __dequeue_task+0x2b4/0xc70 [14697.508677] dequeue_task_fair+0x36/0x370 [14697.509047] dequeue_task+0x101/0x2f0 [14697.509426] __schedule+0x1b1/0x1a00 [14697.510868] anon_pipe_read+0x3da/0x450 [14697.511400] vfs_read+0x361/0x390 [14697.512053] __x64_sys_read+0x19/0x30
The divide-by-zero happens here:
if (scale_load_down(gcfs_rq->load.weight)) {
load_sum = div_u64(gcfs_rq->avg.load_sum,
scale_load_down(gcfs_rq->load.weight));
}gcfs_rq->load.weight is an insane large value and is truncated to the lower 32 bits by div_u64, which happen to be 0.
Using AI for investigation, the cause is a u32 overflow in update_tg_cfs_runnable(), and flat pickup became a victim when using tg_tasks():
u32 new_sum, divider;
...
new_sum = se->avg.runnable_avg * divider; <-- boom
The following sequence shows how this triggers the crash:
propagate_entity_load_avg()
update_tg_cfs_runnable() # u32 overflow corrupts runnable_sum
__update_load_avg_cfs_rq()
___update_load_avg() # computes insane runnable_avg
update_tg_load_avg() # propagates to tg->runnable_avg
update_cfs_group()
calc_concur_shares()
tg_tasks() # long-to-int truncation, negative nr
reweight_entity() # corrupted se->load.weight
update_load_add() # corrupted cfs_rq->load.weight propagate_entity_load_avg()
update_tg_cfs_load()
div_u64() # divide-by-zero
Fix by widening new_sum from u32 to u64 (no need to force tg_tasks() to return unsigned long after this fix)
--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/sched/fair.c +3 -2
@@ -5174,7 +5174,8 @@ static inline void update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cfs_rq *gcfs_rq) { long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg - se->avg.runnable_avg; -u32 new_sum, divider; +u64 new_sum; +u32 divider;/* Nothing to update */ if (!delta_avg)
@@ -5188,7 +5189,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf/* Set new sched_entity's runnable */ se->avg.runnable_avg = gcfs_rq->avg.runnable_avg; - new_sum = se->avg.runnable_avg * divider; + new_sum = (u64 )se->avg.runnable_avg * divider; delta_sum = (long )new_sum - (long )se->avg.runnable_sum; se->avg.runnable_sum = new_sum;
[tip: sched/core] sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 85570f10a4c61372c1d437365b9d8dbad512ec6f Gitweb: https://git.kernel.org/tip/85570f10a4c61372c1d437365b9d8dbad512ec6f Author: Peter Zijlstra (Intel) <[email protected]> AuthorDate: Sat, 06 Dec 2025 10:08:58 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:53 +02:00
sched/eevdf: Move to a single runqueue
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the shares_weight approximation are fully preserved.
That is, given a hierarchy like:
R
|
se--G1
/ \
G2--se se--G3
/ \ |T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:
_R_
/ | \T1 T2 T3
Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (se->h_load) is computed from the hierarchical fraction: se->load / cfs_rq->load.
Since EEVDF is now exclusively operating on rq->cfs, it needs to consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.
One place where additional information was required was set_next_task() / put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).
As a result of only having a single level to pick from, much of the complications in pick_next_task() and preemption go away.
Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.
TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr
--- include/linux/sched.h | 1 +- kernel/sched/core.c | 5 +- kernel/sched/debug.c | 9 +- kernel/sched/fair.c | 801 +++++++++++++++++------------------------ kernel/sched/pelt.c | 6 +- kernel/sched/sched.h | 26 +- 6 files changed, 374 insertions(+), 474 deletions(-)
include/linux/sched.h +1 -0
@@ -575,6 +575,7 @@ struct sched_statistics {struct sched_entity {/* For load-balancing: */ struct load_weight load; +struct load_weight h_load;struct rb_node run_node;u64 deadline;u64 min_vruntime;
kernel/sched/core.c +1 -4
@@ -5657,11 +5657,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);*/ static inline void prefetch_curr_exec_start (struct task_struct *p) { -#ifdef CONFIG_FAIR_GROUP_SCHED -struct sched_entity *curr = p->se.cfs_rq->curr; -#else struct sched_entity *curr =task_rq (p)->cfs.curr; -#endif +prefetch (curr);prefetch (&curr->exec_start); }
kernel/sched/debug.c +6 -3
@@ -975,10 +975,11 @@ print_task(struct seq_file *m, struct rq *rq, struct task_struct *p)else SEQ_printf (m," %c" ,task_state_to_char (p)); - SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", + SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), + p->se.h_load.weight, SPLIT_NS(p->se.vruntime), - entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', + entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice),
@@ -1007,7 +1008,7 @@ static void print_rq(struct seq_file *m, struct rq *rq, int rq_cpu)SEQ_printf (m,"\n" );SEQ_printf (m,"runnable tasks:\n" ); -SEQ_printf (m," S task PID vruntime eligible " +SEQ_printf (m," S task PID weight vruntime eligible " "deadline slice sum-exec switches " "prio wait-time sum-sleep sum-block" #ifdef CONFIG_NUMA_BALANCING
@@ -1115,6 +1116,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) cfs_rq->tg_load_avg_contrib);SEQ_printf (m," .%-30s: %ld\n" ,"tg_load_avg" ,atomic_long_read (&cfs_rq->tg->load_avg)); +SEQ_printf (m," .%-30s: %lu\n" ,"h_load" , + cfs_rq->h_load);#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_CFS_BANDWIDTH SEQ_printf(m, " .%-30s: %d\n", "throttled",
kernel/sched/fair.c +350 -451
1,407 lines not on this page · raw message
kernel/sched/pelt.c +3 -3
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched_avg *sa,/* * running is a subset of runnable (weight) so running can't be set if * runnable is clear. But there are some corner cases where the current -* se has been already dequeued but cfs_rq->curr still points to it. +* se has been already dequeued but cfs_rq->h_curr still points to it. * This means that weight will be 0 but not running for a sched_entity * but also for a cfs_rq if the latter becomes idle. As an example, * this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now, struct sched_entity *se)int __update_load_avg_se (u64 now,struct cfs_rq *cfs_rq,struct sched_entity *se) {if (___update_load_sum (now, &se->avg, !!se->on_rq,se_runnable (se), - cfs_rq->curr == se)) { + cfs_rq->h_curr == se)) {___update_load_avg (&se->avg,se_weight (se));cfs_se_util_change (&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq)if (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), cfs_rq->h_nr_runnable, - cfs_rq->curr !=NULL )) { + cfs_rq->h_curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );trace_pelt_cfs_tp (cfs_rq);
kernel/sched/sched.h +13 -13
@@ -530,21 +530,8 @@ struct task_group { }; -#ifdef CONFIG_GROUP_SCHED_WEIGHT #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD -/* -* A weight of 0 or 1 can cause arithmetics problems. -* A weight of a cfs_rq is the sum of weights of which entities -* are queued on this cfs_rq, so a weight of a entity should not be -* too large, so as the shares value of a task group. -* (The default weight is 1024 - so there's no practical -* limitation from this.) -*/ -#define MIN_SHARES (1UL << 1) -#define MAX_SHARES (1UL << 18) -#endif -typedef int (*tg_visitor )(struct task_group *,void *); extern int walk_tg_tree_from(struct task_group *from,
@@ -631,6 +618,17 @@ static inline bool cfs_task_bw_constrained(struct task_struct *p) { return false#endif /* !CONFIG_CGROUP_SCHED */ +/* +* A weight of 0 or 1 can cause arithmetics problems. +* A weight of a cfs_rq is the sum of weights of which entities +* are queued on this cfs_rq, so a weight of a entity should not be +* too large, so as the shares value of a task group. +* (The default weight is 1024 - so there's no practical +* limitation from this.) +*/ +#define MIN_SHARES (1UL << 1) +#define MAX_SHARES (1UL << 18) +extern void unregister_rt_sched_group (struct task_group *tg);extern void free_rt_sched_group (struct task_group *tg);extern int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent);
@@ -709,6 +707,7 @@ struct cfs_rq {/* * CFS load tracking */ +struct sched_entity *h_curr;struct sched_avg avg;#ifndef CONFIG_64BIT u64 last_update_time_copy;
@@ -2575,6 +2574,7 @@ extern const u32 sched_prio_to_wmult[40];#define ENQUEUE_MIGRATED 0x00040000#define ENQUEUE_INITIAL 0x00080000#define ENQUEUE_RQ_SELECTED 0x00100000 +#define ENQUEUE_QUEUED 0x00200000#define RETRY_TASK ((void *)-1UL)
PATCH v3 7/7 sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraHi Peter,
We have observed some schbench p99 request latency regressions while comparing v7.2 against v7.3 cycle on an AWS Graviton3 (m7g.metal) using Fastpath (Linux kernel performance benchmarking tool).
The schbench suite as a whole does not show a significant regression, and the results appear to be a mix of improvements and regressions depending on the thread configuration. However, a few configurations show fairly large p99 request latency regressions. For example, comparing v7.2 against v7.3-rc4:
message-thread:16, worker-thread:16
v7.2 = 662869 us
v7.3-rc4 = 969557 us
change% = -31.63%
message-thread:64, worker-thread:4
v7.2 = 730453 us
v7.3-rc4 = 942251 us
change% = -22.48%
message-thread:32, worker-thread:4
v7.2 = 74251 us
v7.3-rc4 = 56725 us
change% = +30.90%
So this does not appear to be a general latency degradation; some configs improve while others regress.
I also ran a Fastpath automated git bisection for the m 16, t 16 p99 request latency regression between v7.2 and v7.3-rc1, which identified this commit as the first bad commit. The original regression was ~30.9%, & the bisection reproducibility check showed ~27.8% regression with 20 repeats across 2 boot sessions.
This commit looks relevant since it changes EEVDF runqueue organization and is intended, in part, to address latency issues with hierarchical fair/ cgroup scheduling. I also noticed that there have been fixes on top of this change, including 68e37487810a ("sched/fair: Fix flat hierarchy"), which fixes vruntime handling and has a Fixes: 85570f10a4c6 tag.
Given that the overall schbench result is mostly neutral while some p99 configurations move considerably in both directions, I wanted to check whether this could be an expected trade-off from the single-runqueue change, or whether the p99 regressions are worth investigating further.
I can run additional tests or provide the full results/bisection data if useful.
Thank you.
Aishwarya Rambhadran
Quoted 1,694 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the
shares_weightapproximation are fully preserved.That is, given a hierarchy like:
R | se--G1 / \ G2--se se--G3 / \ |T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:
_R_ / | \ T1 T2 T3Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (
se->h_load) is computed from the hierarchical fraction:se->load/cfs_rq->load.Since EEVDF is now exclusively operating on
rq->cfs, it needs to considercfs_rq->h_nr_queuedrather thancfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.One place where additional information was required was
set_next_task()/put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).As a result of only having a single level to pick from, much of the complications in
pick_next_task()and preemption go away.Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.
TODO: split struct
cfs_rq/ structsched_entity
TODO: try and get rid ofh_currSigned-off-by Peter Zijlstra (Intel) <[email protected]>--- include/linux/sched.h | 1 kernel/sched/core.c | 5 kernel/sched/debug.c | 9 kernel/sched/fair.c | 803 +++++++++++++++++++++----------------------------- kernel/sched/pelt.c | 6 kernel/sched/sched.h | 26 - 6 files changed, 375 insertions(+), 475 deletions(-)include/linux/sched.h +1
-0@@ -574,6 +574,7 @@ struct sched_statistics {struct sched_entity {/* For load-balancing: */ struct load_weight load; +struct load_weight h_load;struct rb_node run_node;u64 deadline;u64 min_vruntime;kernel/sched/core.c +1
-4@@ -5652,11 +5652,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);*/ static inline void prefetch_curr_exec_start (struct task_struct *p) { -#ifdef CONFIG_FAIR_GROUP_SCHED -struct sched_entity *curr = p->se.cfs_rq->curr; -#else struct sched_entity *curr =task_rq (p)->cfs.curr; -#endif +prefetch (curr);prefetch (&curr->exec_start); }kernel/sched/debug.c +6
-3@@ -975,10 +975,11 @@ print_task(struct seq_file *m, struct rqelse SEQ_printf (m," %c" ,task_state_to_char (p)); - SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", + SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), + p->se.h_load.weight, SPLIT_NS(p->se.vruntime), - entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', + entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice),@@ -1007,7 +1008,7 @@ static void print_rq(struct seq_file *m,SEQ_printf (m,"\n" );SEQ_printf (m,"runnable tasks:\n" ); -SEQ_printf (m," S task PID vruntime eligible " +SEQ_printf (m," S task PID weight vruntime eligible " "deadline slice sum-exec switches " "prio wait-time sum-sleep sum-block" #ifdef CONFIG_NUMA_BALANCING@@ -1115,6 +1116,8 @@ void print_cfs_rq(struct seq_file *m, in cfs_rq->tg_load_avg_contrib);SEQ_printf (m," .%-30s: %ld\n" ,"tg_load_avg" ,atomic_long_read (&cfs_rq->tg->load_avg)); +SEQ_printf (m," .%-30s: %lu\n" ,"h_load" , + cfs_rq->h_load);#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_CFS_BANDWIDTH SEQ_printf(m, " .%-30s: %d\n", "throttled",kernel/sched/fair.c +351
-452@@ -297,8 +297,8 @@ static u64 __calc_delta(u64 delta_exec,*/ static inline u64 calc_delta_fair (u64 delta,struct sched_entity *se) { -if (unlikely (se->load.weight !=NICE_0_LOAD )) - delta =__calc_delta (delta,NICE_0_LOAD , &se->load); +if (se->h_load.weight !=NICE_0_LOAD ) + delta =__calc_delta (delta,NICE_0_LOAD , &se->h_load);return delta; }@@ -428,38 +428,6 @@ static inline struct sched_entity *parenreturn se->parent; } -static void -find_matching_se (struct sched_entity **se,struct sched_entity **pse) -{ -int se_depth, pse_depth; - -/* -* preemption test can be made between sibling entities who are in the -* same cfs_rq i.e who have a common parent. Walk up the hierarchy of -* both tasks until we find their ancestors who are siblings of common -* parent. -*/ - -/* First walk up until both entities are at same depth */ - se_depth = (*se)->depth; - pse_depth = (*pse)->depth; - -while (se_depth > pse_depth) { - se_depth--; - *se =parent_entity (*se); - } - -while (pse_depth > se_depth) { - pse_depth--; - *pse =parent_entity (*pse); - } - -while (!is_same_group (*se, *pse)) { - *se =parent_entity (*se); - *pse =parent_entity (*pse); - } -} -static int tg_is_idle (struct task_group *tg) {return tg->idle >0 ;@@ -503,11 +471,6 @@ static inline struct sched_entity *parenreturn NULL ; } -static inline void -find_matching_se (struct sched_entity **se,struct sched_entity **pse) -{ -} -static inline int tg_is_idle (struct task_group *tg) {return 0 ;@@ -686,7 +649,7 @@ static inline unsigned long avg_vruntimestatic inline void __sum_w_vruntime_add (struct cfs_rq *cfs_rq,struct sched_entity *se) { -unsigned long weight =avg_vruntime_weight (cfs_rq, se->load.weight); +unsigned long weight =avg_vruntime_weight (cfs_rq, se->h_load.weight);s64 w_vruntime, key =entity_key (cfs_rq, se); w_vruntime = key * weight;@@ -703,7 +666,7 @@ sum_w_vruntime_add_paranoid(struct cfs_rs64 key, tmp; again: - weight =avg_vruntime_weight (cfs_rq, se->load.weight); + weight =avg_vruntime_weight (cfs_rq, se->h_load.weight); key =entity_key (cfs_rq, se);if (check_mul_overflow (key, weight, &key))@@ -749,7 +712,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rqstatic void sum_w_vruntime_sub (struct cfs_rq *cfs_rq,struct sched_entity *se) { -unsigned long weight =avg_vruntime_weight (cfs_rq, se->load.weight); +unsigned long weight =avg_vruntime_weight (cfs_rq, se->h_load.weight);s64 key =entity_key (cfs_rq, se); cfs_rq->sum_w_vruntime -= key * weight;@@ -791,7 +754,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)s64 runtime = cfs_rq->sum_w_vruntime;if (curr) { -unsigned long w =avg_vruntime_weight (cfs_rq, curr->load.weight); +unsigned long w =avg_vruntime_weight (cfs_rq, curr->h_load.weight); runtime +=entity_key (cfs_rq, curr) * w; weight += w;@@ -862,8 +825,6 @@ bool update_entity_lag(struct cfs_rq *cfu64 avruntime =avg_vruntime (cfs_rq);s64 vlag =entity_lag (cfs_rq, se, avruntime); -WARN_ON_ONCE (!se->on_rq); -if (se->sched_delayed) {/* previous vlag < 0 otherwise se would not be delayed */ vlag =max (vlag, se->vlag);@@ -899,7 +860,7 @@ static int vruntime_eligible(struct cfs_long load = cfs_rq->sum_weight;if (curr && curr->on_rq) { -unsigned long weight =avg_vruntime_weight (cfs_rq, curr->load.weight); +unsigned long weight =avg_vruntime_weight (cfs_rq, curr->h_load.weight); avg +=entity_key (cfs_rq, curr) * weight; load += weight;@@ -1040,6 +1001,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim*/ static void __enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (&rq_of (cfs_rq)->cfs != cfs_rq); +WARN_ON_ONCE (!entity_is_task (se)); +sum_w_vruntime_add (cfs_rq, se); se->min_vruntime = se->vruntime; se->min_slice = se->slice;@@ -1049,6 +1013,9 @@ static void __enqueue_entity(struct cfs_static void __dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (&rq_of (cfs_rq)->cfs != cfs_rq); +WARN_ON_ONCE (!entity_is_task (se)); +rb_erase_augmented_cached (&se->run_node, &cfs_rq->tasks_timeline, &min_vruntime_cb);sum_w_vruntime_sub (cfs_rq, se);@@ -1145,7 +1112,7 @@ static struct sched_entity *pick_eevdf(s* We can safely skip eligibility check if there is only one entity * in this cfs_rq, saving some cycles. */ -if (cfs_rq->nr_queued ==1 ) +if (cfs_rq->h_nr_queued ==1 )return curr && curr->on_rq ? curr : se;/* @@ -1395,8 +1362,6 @@ static s64 update_se(struct rq *rq, strureturn delta_exec; } -static void set_next_buddy (struct sched_entity *se); -#ifdef CONFIG_SCHED_CACHE /* @@ -1991,7 +1956,7 @@ static void update_curr(struct cfs_rq *c* not necessarily be the actual task running * (rq->curr.se). This is easy to confuse! */ -struct sched_entity *curr = cfs_rq->curr; +struct sched_entity *curr = cfs_rq->h_curr;struct rq *rq =rq_of (cfs_rq);s64 delta_exec;bool resched;@@ -2003,26 +1968,29 @@ static void update_curr(struct cfs_rq *cif (unlikely (delta_exec <=0 ))return ; +account_cfs_rq_runtime (cfs_rq, delta_exec); + +if (!entity_is_task (curr)) +return ; + + cfs_rq = &rq->cfs; + curr->vruntime +=calc_delta_fair (delta_exec, curr); resched =update_deadline (cfs_rq, curr); -if (entity_is_task (curr)) { -/* -* If the fair_server is active, we need to account for the -* fair_server time whether or not the task is running on -* behalf of fair_server or not: -* - If the task is running on behalf of fair_server, we need -* to limit its time based on the assigned runtime. -* - Fair task that runs outside of fair_server should account -* against fair_server such that it can account for this time -* and possibly avoid running this period. -*/ -dl_server_update (&rq->fair_server, delta_exec); - } - -account_cfs_rq_runtime (cfs_rq, delta_exec); +/* +* If the fair_server is active, we need to account for the +* fair_server time whether or not the task is running on +* behalf of fair_server or not: +* - If the task is running on behalf of fair_server, we need +* to limit its time based on the assigned runtime. +* - Fair task that runs outside of fair_server should account +* against fair_server such that it can account for this time +* and possibly avoid running this period. +*/ +dl_server_update (&rq->fair_server, delta_exec); -if (cfs_rq->nr_queued ==1 ) +if (cfs_rq->h_nr_queued ==1 )return ;if (resched || !protect_slice (curr)) {@@ -2033,7 +2001,10 @@ static void update_curr(struct cfs_rq *cstatic void update_curr_fair (struct rq *rq) { -update_curr (cfs_rq_of (&rq->donor->se)); +struct sched_entity *se = &rq->donor->se; + +for_each_sched_entity (se) +update_curr (cfs_rq_of (se)); } static inline void@@ -2109,7 +2080,7 @@ update_stats_enqueue_fair(struct cfs_rq* Are we enqueueing a waiting task? (for current tasks * a dequeue/enqueue event is a NOP) */ -if (se != cfs_rq->curr) +if (se != cfs_rq->h_curr)update_stats_wait_start_fair (cfs_rq, se);if (flags &ENQUEUE_WAKEUP )@@ -2127,7 +2098,7 @@ update_stats_dequeue_fair(struct cfs_rq* Mark the end of the wait period if dequeueing a * waiting task: */ -if (se != cfs_rq->curr) +if (se != cfs_rq->h_curr)update_stats_wait_end_fair (cfs_rq, se);if ((flags &DEQUEUE_SLEEP ) &&entity_is_task (se)) {@@ -4468,6 +4439,7 @@ static inline void update_scan_period(ststatic void account_entity_enqueue (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (cfs_rq !=cfs_rq_of (se));update_load_add (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {struct task_struct *p =task_of (se);@@ -4483,6 +4455,7 @@ account_entity_enqueue(struct cfs_rq *cfstatic void account_entity_dequeue (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (cfs_rq !=cfs_rq_of (se));update_load_sub (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {struct task_struct *p =task_of (se);@@ -4564,7 +4537,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,static void rescale_entity (struct sched_entity *se,unsigned long weight,bool rel_vprot) { -unsigned long old_weight = se->load.weight; +long old_weight = se->h_load.weight;/* * VRUNTIME @@ -4664,16 +4637,17 @@ rescale_entity(struct sched_entity *se, se->vprot =div64_long (se->vprot * old_weight, weight); } -static void reweight_entity (struct cfs_rq *cfs_rq,struct sched_entity *se, -unsigned long weight) +static void reweight_eevdf (struct cfs_rq *cfs_rq,struct sched_entity *se, +unsigned long weight,bool on_rq) {bool curr = cfs_rq->curr == se;bool rel_vprot = false;u64 avruntime =0 ; -if (se->on_rq) { -/* commit outstanding execution time */ -update_curr (cfs_rq); +if (se->h_load.weight == weight) +return ; + +if (on_rq) { avruntime =avg_vruntime (cfs_rq); se->vlag =entity_lag (cfs_rq, se, avruntime); se->deadline -= avruntime;@@ -4683,46 +4657,90 @@ static void reweight_entity(struct cfs_r rel_vprot = true; } - cfs_rq->nr_queued--; + cfs_rq->h_nr_queued--;if (!curr)__dequeue_entity (cfs_rq, se); -update_load_sub (&cfs_rq->load, se->load.weight); } -dequeue_load_avg (cfs_rq, se);rescale_entity (se, weight, rel_vprot); -update_load_set (&se->load, weight); +update_load_set (&se->h_load, weight); -do { -u32 divider =get_pelt_divider (&se->avg); - se->avg.load_avg =div_u64 (se_weight (se) * se->avg.load_sum, divider); - }while (0 ); - -enqueue_load_avg (cfs_rq, se); -if (se->on_rq) { +if (on_rq) {if (rel_vprot) se->vprot += avruntime; se->deadline += avruntime; se->rel_deadline =0 ; se->vruntime = avruntime - se->vlag; -update_load_add (&cfs_rq->load, se->load.weight);if (!curr)__enqueue_entity (cfs_rq, se); - cfs_rq->nr_queued++; + cfs_rq->h_nr_queued++; } } +static void reweight_entity (struct cfs_rq *cfs_rq,struct sched_entity *se, +unsigned long weight) +{ +if (se->load.weight == weight) +return ; + +if (se->on_rq) { +WARN_ON_ONCE (cfs_rq !=cfs_rq_of (se)); +update_load_sub (&cfs_rq->load, se->load.weight); + } +dequeue_load_avg (cfs_rq, se); + +update_load_set (&se->load, weight); + +do { +u32 divider =get_pelt_divider (&se->avg); + se->avg.load_avg =div_u64 (se_weight (se) * se->avg.load_sum, divider); + }while (0 ); + +enqueue_load_avg (cfs_rq, se); + +if (se->on_rq) +update_load_add (&cfs_rq->load, se->load.weight); +} + +/* +* weight = NICE_0_LOAD; +* for_each_entity_se(se) +* weight = __calc_prop_weight(cfs_rq_of(se), se, weight); +*/ +static __always_inline +unsigned long __calc_prop_weight (struct cfs_rq *cfs_rq,struct sched_entity *se, +unsigned long weight) +{ + weight *= se->load.weight; +if (parent_entity (se)) + weight /= cfs_rq->load.weight; +else + weight /=NICE_0_LOAD ; + +return max (weight,MIN_SHARES ); +} +static void reweight_task_fair (struct rq *rq,struct task_struct *p,const struct load_weight *lw) {struct sched_entity *se = &p->se; -struct cfs_rq *cfs_rq =cfs_rq_of (se); -struct load_weight *load = &se->load; +unsigned long weight =NICE_0_LOAD ; -reweight_entity (cfs_rq, se, lw->weight); - load->inv_weight = lw->inv_weight; +if (se->on_rq) +update_curr_fair (rq); + +reweight_entity (cfs_rq_of (se), se, lw->weight); + se->load.inv_weight = lw->inv_weight; + +if (!se->on_rq) +return ; + +for_each_sched_entity (se) + weight =__calc_prop_weight (cfs_rq_of (se), se, weight); + +reweight_eevdf (&rq->cfs, &p->se, weight, p->se.on_rq); }static inline int throttled_hierarchy (struct cfs_rq *cfs_rq);@@ -4958,8 +4976,7 @@ static void update_cfs_group(struct schereturn ; shares =static_call (calc_group_shares)(gcfs_rq); -if (unlikely (se->load.weight != shares)) - reweight_entity(cfs_rq_of(se), se, shares); + reweight_entity(cfs_rq_of(se), se, shares); }#else /* !CONFIG_FAIR_GROUP_SCHED: */ @@ -5077,7 +5094,7 @@ static inline bool cfs_rq_is_decayed(str* differential update where we store the last value we propagated. This in * turn allows skipping updates if the differential is 'small'. * -* Updating tg's load_avg is necessary before update_cfs_share(). +* Updating tg's load_avg is necessary before update_cfs_group(). */ static inline void update_tg_load_avg (struct cfs_rq *cfs_rq) {@@ -5544,7 +5561,7 @@ static void migrate_se_pelt_lag(struct s* The cfs_rq avg is the direct sum of all its entities (blocked and runnable) * avg. The immediate corollary is that all (fair) tasks must be attached. * -* cfs_rq->avg is used for task_h_load() and update_cfs_share() for example. +* cfs_rq->avg is used for task_h_load() and update_cfs_group() for example. * * Return: true if the load decayed or we removed load. * @@ -6082,6 +6099,7 @@ static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { u64 vslice, vruntime = avg_vruntime(cfs_rq); +unsigned int nr_queued = cfs_rq->h_nr_queued;bool update_zero = false;s64 lag =0 ;@@ -6089,6 +6107,9 @@ place_entity(struct cfs_rq *cfs_rq, stru se->slice = sysctl_sched_base_slice; vslice =calc_delta_fair (se->slice, se); +if (flags &ENQUEUE_QUEUED ) + nr_queued -=1 ; +/* * Due to how V is constructed as the weighted average of entities, * adding tasks with positive lag, or removing tasks with negative lag @@ -6097,7 +6118,7 @@ place_entity(struct cfs_rq *cfs_rq, stru* * EEVDF: placement strategy #1 / #2 */ -if (sched_feat (PLACE_LAG ) && cfs_rq->nr_queued && se->vlag) { +if (sched_feat (PLACE_LAG ) && nr_queued && se->vlag) {struct sched_entity *curr = cfs_rq->curr;long load, weight;@@ -6157,9 +6178,9 @@ place_entity(struct cfs_rq *cfs_rq, stru*/ load = cfs_rq->sum_weight;if (curr && curr->on_rq) - load +=avg_vruntime_weight (cfs_rq, curr->load.weight); + load +=avg_vruntime_weight (cfs_rq, curr->h_load.weight); - weight =avg_vruntime_weight (cfs_rq, se->load.weight); + weight =avg_vruntime_weight (cfs_rq, se->h_load.weight); lag *= load + weight;if (WARN_ON_ONCE (!load)) load =1 ;@@ -6218,22 +6239,8 @@ static void check_enqueue_throttle(strucstatic inline int cfs_rq_throttled (struct cfs_rq *cfs_rq);static void -requeue_delayed_entity (struct sched_entity *se); - -static void enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -bool curr = cfs_rq->curr == se; - -/* -* If we're the current task, we must renormalise before calling -* update_curr(). -*/ -if (curr) -place_entity (cfs_rq, se, flags); - -update_curr (cfs_rq); -/* * When enqueuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. @@ -6252,13 +6259,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st*/ update_cfs_group (se); -/* -* XXX now that the entity has been re-weighted, and it's lag adjusted, -* we can place the entity. -*/ -if (!curr) -place_entity (cfs_rq, se, flags); -account_entity_enqueue (cfs_rq, se);/* Entity has migrated, no longer consider this task hot */ @@ -6267,8 +6267,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, stcheck_schedstat_required ();update_stats_enqueue_fair (cfs_rq, se, flags); -if (!curr) -__enqueue_entity (cfs_rq, se); se->on_rq =1 ;if (cfs_rq->nr_queued ==1 ) {@@ -6286,21 +6284,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st } } -static void __clear_buddies_next (struct sched_entity *se) +static void set_next_buddy (struct cfs_rq *cfs_rq,struct sched_entity *se) { -for_each_sched_entity (se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); -if (cfs_rq->next != se) -break ; - - cfs_rq->next =NULL ; - } +if (WARN_ON_ONCE (!se->on_rq || se->sched_delayed)) +return ; +if (se_is_idle (se)) +return ; + cfs_rq->next = se; }static void clear_buddies (struct cfs_rq *cfs_rq,struct sched_entity *se) {if (cfs_rq->next == se) -__clear_buddies_next (se); + cfs_rq->next =NULL ; } static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);@@ -6311,7 +6307,7 @@ static void set_delayed(struct sched_ent/* * Delayed se of cfs_rq have no tasks queued on them. -* Do not adjust h_nr_runnable since dequeue_entities() +* Do not adjust h_nr_runnable since __dequeue_task() * will account it for blocked tasks. */ if (!entity_is_task (se))@@ -6344,45 +6340,16 @@ static void clear_delayed(struct sched_e } } -static bool +static void dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -bool sleep = flags &DEQUEUE_SLEEP ; -int action =0 ; - -update_curr (cfs_rq); -clear_buddies (cfs_rq, se); - -if (flags &DEQUEUE_DELAYED ) { -WARN_ON_ONCE (!se->sched_delayed); - }else { -bool delay = sleep; -/* -* DELAY_DEQUEUE relies on spurious wakeups, special task -* states must not suffer spurious wakeups, excempt them. -*/ -if (flags & (DEQUEUE_SPECIAL |DEQUEUE_THROTTLE )) - delay = false; - -WARN_ON_ONCE (delay && se->sched_delayed); +int action =UPDATE_TG ; -if (sched_feat (DELAY_DEQUEUE ) && delay && - !entity_eligible (cfs_rq, se)) { -if (entity_is_task (se)) - action |=UPDATE_UTIL_EST ; -update_load_avg (cfs_rq, se, action); -update_entity_lag (cfs_rq, se); -set_delayed (se); -return false; - } - } - - action =UPDATE_TG ;if (entity_is_task (se)) {if (task_on_rq_migrating (task_of (se))) action |=DO_DETACH ; -if (sleep && !(flags &DEQUEUE_DELAYED )) +if ((flags &DEQUEUE_SLEEP ) && !(flags &DEQUEUE_DELAYED )) action |=UPDATE_UTIL_EST ; }@@ -6400,14 +6367,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_stats_dequeue_fair (cfs_rq, se, flags); -update_entity_lag (cfs_rq, se); -if (sched_feat (PLACE_REL_DEADLINE ) && !sleep) { - se->deadline -= se->vruntime; - se->rel_deadline =1 ; - } - -if (se != cfs_rq->curr) -__dequeue_entity (cfs_rq, se); se->on_rq =0 ;account_entity_dequeue (cfs_rq, se);@@ -6416,9 +6375,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_cfs_group (se); -if (flags &DEQUEUE_DELAYED ) -clear_delayed (se); -if (cfs_rq->nr_queued ==0 ) {update_idle_cfs_rq_clock_pelt (cfs_rq);#ifdef CONFIG_CFS_BANDWIDTH @@ -6431,15 +6387,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st }#endif } - -return true; }static void -set_next_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,bool first) +set_next_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { -clear_buddies (cfs_rq, se); -/* 'current' is not kept within the tree. */ if (se->on_rq) {/* @@ -6448,16 +6400,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s* runqueue. */ update_stats_wait_end_fair (cfs_rq, se); -__dequeue_entity (cfs_rq, se);update_load_avg (cfs_rq, se,UPDATE_TG ); - -if (first) -set_protect_slice (cfs_rq, se); }update_stats_curr_start (cfs_rq, se); -WARN_ON_ONCE (cfs_rq->curr); - cfs_rq->curr = se; +WARN_ON_ONCE (cfs_rq->h_curr); + cfs_rq->h_curr = se;/* * Track our maximum slice length, if the CPU's load is at @@ -6477,23 +6425,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s se->prev_sum_exec_runtime = se->sum_exec_runtime; } -static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags); +static bool __dequeue_task (struct rq *rq,struct task_struct *p,int flags); -/* -* Pick the next process, keeping these things in mind, in this order: -* 1) keep things fair between processes/task groups -* 2) pick the "next" process, since someone really wants that to run -* 3) pick the "last" process, for cache locality -* 4) do not run the "skip" process, if something else is available -*/ static struct sched_entity * -pick_next_entity (struct rq *rq,struct cfs_rq *cfs_rq,bool protect) +pick_next_entity (struct rq *rq,bool protect) { +struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; se =pick_eevdf (cfs_rq, protect);if (se->sched_delayed) { -dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +__dequeue_task (rq,task_of (se),DEQUEUE_SLEEP |DEQUEUE_DELAYED );/* * Must not reference @se again, see __block_task(). */ @@ -6513,13 +6455,11 @@ static void put_prev_entity(struct cfs_rif (prev->on_rq) {update_stats_wait_start_fair (cfs_rq, prev); -/* Put 'current' back into the tree. */ -__enqueue_entity (cfs_rq, prev);/* in !on_rq case, update occurred at dequeue */ update_load_avg (cfs_rq, prev,0 ); } -WARN_ON_ONCE (cfs_rq->curr != prev); - cfs_rq->curr =NULL ; +WARN_ON_ONCE (cfs_rq->h_curr != prev); + cfs_rq->h_curr =NULL ; } static void@@ -7074,7 +7014,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfassert_list_leaf_cfs_rq (rq);/* Determine whether we need to wake up potentially idle CPU: */ -if (rq->curr == rq->idle && rq->cfs.nr_queued) +if (rq->curr == rq->idle && rq->cfs.h_nr_queued)resched_curr (rq); }@@ -7409,7 +7349,7 @@ static void check_enqueue_throttle(strucreturn ;/* an active group must be handled by the update_curr() path */ -if (!cfs_rq->runtime_enabled || cfs_rq->curr) +if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)return ;/* ensure the group is not already throttled */ @@ -7781,7 +7721,7 @@ static void hrtick_start_fair(struct rqresched_curr (rq);return ; } - delta = (se->load.weight * vdelta) /NICE_0_LOAD ; + delta = (se->h_load.weight * vdelta) /NICE_0_LOAD ;/* * Correct for instantaneous load of other classes. @@ -7881,10 +7821,8 @@ static int choose_idle_cpu(int cpu, stru }static void -requeue_delayed_entity (struct sched_entity *se) +requeue_delayed_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); -/* * se->sched_delayed should imply: se->on_rq == 1. * Because a delayed entity is one that is still on @@ -7894,19 +7832,58 @@ requeue_delayed_entity(struct sched_entiWARN_ON_ONCE (!se->on_rq);if (update_entity_lag (cfs_rq, se)) { - cfs_rq->nr_queued--; + cfs_rq->h_nr_queued--;if (se != cfs_rq->curr)__dequeue_entity (cfs_rq, se);place_entity (cfs_rq, se,0 );if (se != cfs_rq->curr)__enqueue_entity (cfs_rq, se); - cfs_rq->nr_queued++; + cfs_rq->h_nr_queued++; }update_load_avg (cfs_rq, se,0 );clear_delayed (se); } +static unsigned long enqueue_hierarchy (struct task_struct *p,int flags) +{ +unsigned long weight =NICE_0_LOAD ; +int task_new = !(flags &ENQUEUE_WAKEUP ); +struct sched_entity *se = &p->se; +int h_nr_idle =task_has_idle_policy (p); +int h_nr_runnable =1 ; + +if (task_new && se->sched_delayed) + h_nr_runnable =0 ; + +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +update_curr (cfs_rq); + +if (!se->on_rq) { +enqueue_entity (cfs_rq, se, flags); + }else { +update_load_avg (cfs_rq, se,UPDATE_TG ); +se_update_runnable (se); +update_cfs_group (se); + } + + cfs_rq->h_nr_runnable += h_nr_runnable; + cfs_rq->h_nr_queued++; + cfs_rq->h_nr_idle += h_nr_idle; + +if (cfs_rq_is_idle (cfs_rq)) + h_nr_idle =1 ; + + weight =__calc_prop_weight (cfs_rq, se, weight); + + flags =ENQUEUE_WAKEUP ; + } + +return weight; +} +/* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -7915,13 +7892,12 @@ requeue_delayed_entity(struct sched_entistatic void enqueue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -struct cfs_rq *cfs_rq; -struct sched_entity *se = &p->se; -int h_nr_idle =task_has_idle_policy (p); -int h_nr_runnable =1 ; -int task_new = !(flags &ENQUEUE_WAKEUP );int rq_h_nr_queued = rq->cfs.h_nr_queued; -u64 slice =0 ; +int task_new = !(flags &ENQUEUE_WAKEUP ); +struct sched_entity *se = &p->se; +struct cfs_rq *cfs_rq = &rq->cfs; +unsigned long weight; +bool curr;if (task_is_throttled (p) &&enqueue_throttled_task (p))return ;@@ -7933,10 +7909,10 @@ enqueue_task_fair(struct rq *rq, struct* estimated utilization, before we update schedutil. */ if (!p->se.sched_delayed || (flags &ENQUEUE_DELAYED )) -util_est_enqueue (&rq->cfs, p); +util_est_enqueue (cfs_rq, p);if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); +requeue_delayed_entity (cfs_rq, se);return ; }@@ -7948,57 +7924,22 @@ enqueue_task_fair(struct rq *rq, structif (p->in_iowait)cpufreq_update_util (rq,SCHED_CPUFREQ_IOWAIT ); -if (task_new && se->sched_delayed) - h_nr_runnable =0 ; - -for_each_sched_entity (se) { -if (se->on_rq) { -if (se->sched_delayed) -requeue_delayed_entity (se); -break ; - } - cfs_rq =cfs_rq_of (se); - -/* -* Basically set the slice of group entries to the min_slice of -* their respective cfs_rq. This ensures the group can service -* its entities in the desired time-frame. -*/ -if (slice) { - se->slice = slice; - se->custom_slice =1 ; - } -enqueue_entity (cfs_rq, se, flags); - slice =cfs_rq_min_slice (cfs_rq); - - cfs_rq->h_nr_runnable += h_nr_runnable; - cfs_rq->h_nr_queued++; - cfs_rq->h_nr_idle += h_nr_idle; - -if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle =1 ; - - flags =ENQUEUE_WAKEUP ; - } - -for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); +/* +* XXX comment on the curr thing +*/ + curr = (cfs_rq->curr == se); +if (curr) +place_entity (cfs_rq, se, flags); -update_load_avg (cfs_rq, se,UPDATE_TG ); -se_update_runnable (se); -update_cfs_group (se); +if (se->on_rq && se->sched_delayed) +requeue_delayed_entity (cfs_rq, se); - se->slice = slice; -if (se != cfs_rq->curr) -min_vruntime_cb_propagate (&se->run_node,NULL ); - slice =cfs_rq_min_slice (cfs_rq); + weight =enqueue_hierarchy (p, flags); - cfs_rq->h_nr_runnable += h_nr_runnable; - cfs_rq->h_nr_queued++; - cfs_rq->h_nr_idle += h_nr_idle; - -if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle =1 ; +if (!curr) { +reweight_eevdf (cfs_rq, se, weight, false); +place_entity (cfs_rq, se, flags |ENQUEUE_QUEUED ); +__enqueue_entity (cfs_rq, se); }if (!rq_h_nr_queued && rq->cfs.h_nr_queued)@@ -8029,105 +7970,109 @@ enqueue_task_fair(struct rq *rq, structhrtick_update (rq); } -/* -* Basically dequeue_task_fair(), except it can deal with dequeue_entity() -* failing half-way through and resume the dequeue later. -* -* Returns: -* -1 - dequeue delayed -* 0 - dequeue throttled -* 1 - dequeue complete -*/ -static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags) +static void dequeue_hierarchy (struct task_struct *p,int flags) { -bool was_sched_idle =sched_idle_rq (rq); +struct sched_entity *se = &p->se;bool task_sleep = flags &DEQUEUE_SLEEP ;bool task_delayed = flags &DEQUEUE_DELAYED ;bool task_throttled = flags &DEQUEUE_THROTTLE ; -struct task_struct *p =NULL ; -int h_nr_idle =0 ; -int h_nr_queued =0 ;int h_nr_runnable =0 ; -struct cfs_rq *cfs_rq; -u64 slice =0 ; +int h_nr_idle =task_has_idle_policy (p); +bool dequeue = true; -if (entity_is_task (se)) { - p =task_of (se); - h_nr_queued =1 ; - h_nr_idle =task_has_idle_policy (p); -if (task_sleep || task_delayed || !se->sched_delayed) - h_nr_runnable =1 ; - } +if (task_sleep || task_delayed || !se->sched_delayed) + h_nr_runnable =1 ;for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); +struct cfs_rq *cfs_rq =cfs_rq_of (se); -if (!dequeue_entity (cfs_rq, se, flags)) { -if (p && &p->se == se) -return -1 ; +update_curr (cfs_rq); - slice =cfs_rq_min_slice (cfs_rq); -break ; +if (dequeue) { +dequeue_entity (cfs_rq, se, flags); +/* Don't dequeue parent if it has other entities besides us */ +if (cfs_rq->load.weight) + dequeue = false; + }else { +update_load_avg (cfs_rq, se,UPDATE_TG ); +se_update_runnable (se); +update_cfs_group (se); } cfs_rq->h_nr_runnable -= h_nr_runnable; - cfs_rq->h_nr_queued -= h_nr_queued; + cfs_rq->h_nr_queued--; cfs_rq->h_nr_idle -= h_nr_idle;if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle = h_nr_queued; + h_nr_idle =1 ;if (throttled_hierarchy (cfs_rq) && task_throttled)record_throttle_clock (cfs_rq); -/* Don't dequeue parent if it has other entities besides us */ -if (cfs_rq->load.weight) { - slice =cfs_rq_min_slice (cfs_rq); - -/* Avoid re-evaluating load for this entity: */ - se =parent_entity (se); -/* -* Bias pick_next to pick a task from this cfs_rq, as -* p is sleeping when it is within its sched_slice. -*/ -if (task_sleep && se) -set_next_buddy (se); -break ; - } flags |=DEQUEUE_SLEEP ; flags &= ~(DEQUEUE_DELAYED |DEQUEUE_SPECIAL ); } +} -for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); +/* +* The part of dequeue_task_fair() that is needed to dequeue delayed tasks. +* +* Returns: +* true - dequeued +* false - delayed +*/ +static bool __dequeue_task (struct rq *rq,struct task_struct *p,int flags) +{ +struct sched_entity *se = &p->se; +struct cfs_rq *cfs_rq = &rq->cfs; +bool was_sched_idle =sched_idle_rq (rq); +bool task_sleep = flags &DEQUEUE_SLEEP ; +bool task_delayed = flags &DEQUEUE_DELAYED ; -update_load_avg (cfs_rq, se,UPDATE_TG ); -se_update_runnable (se); -update_cfs_group (se); +clear_buddies (cfs_rq, se); - se->slice = slice; -if (se != cfs_rq->curr) -min_vruntime_cb_propagate (&se->run_node,NULL ); - slice =cfs_rq_min_slice (cfs_rq); +update_curr (cfs_rq_of (se)); +update_entity_lag (cfs_rq, se); - cfs_rq->h_nr_runnable -= h_nr_runnable; - cfs_rq->h_nr_queued -= h_nr_queued; - cfs_rq->h_nr_idle -= h_nr_idle; +if (flags &DEQUEUE_DELAYED ) { +WARN_ON_ONCE (!se->sched_delayed); + }else { +bool delay = task_sleep; +/* +* DELAY_DEQUEUE relies on spurious wakeups, special task +* states must not suffer spurious wakeups, excempt them. +*/ +if (flags & (DEQUEUE_SPECIAL |DEQUEUE_THROTTLE )) + delay = false; -if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle = h_nr_queued; +WARN_ON_ONCE (delay && se->sched_delayed); -if (throttled_hierarchy (cfs_rq) && task_throttled) -record_throttle_clock (cfs_rq); +if (sched_feat (DELAY_DEQUEUE ) && delay && + !entity_eligible (cfs_rq, se)) { +update_load_avg (cfs_rq_of (se), se,UPDATE_UTIL_EST ); +set_delayed (se); +return false; + } } -sub_nr_running (rq, h_nr_queued); +dequeue_hierarchy (p, flags); + +if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { + se->deadline -= se->vruntime; + se->rel_deadline =1 ; + } +if (se != cfs_rq->curr) +__dequeue_entity (cfs_rq, se); + +sub_nr_running (rq,1 );/* balance early to pull high priority tasks */ if (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; -if (p && task_delayed) { +if (task_delayed) { +clear_delayed (se); +WARN_ON_ONCE (!task_sleep);WARN_ON_ONCE (p->on_rq !=1 );@@ -8139,7 +8084,7 @@ static int dequeue_entities(struct rq *r__block_task (rq, p); } -return 1 ; +return true; }/* @@ -8157,11 +8102,11 @@ static bool dequeue_task_fair(struct rqif (!p->se.sched_delayed)util_est_dequeue (&rq->cfs, p); -if (dequeue_entities (rq, &p->se, flags) <0 ) +if (!__dequeue_task (rq, p, flags))return false;/* -* Must not reference @p after dequeue_entities(DEQUEUE_DELAYED). +* Must not reference @p after __dequeue_task(DEQUEUE_DELAYED). */ return true; }@@ -9749,19 +9694,6 @@ static void migrate_task_rq_fair(structstatic void task_dead_fair (struct task_struct *p) {struct sched_entity *se = &p->se; - -if (se->sched_delayed) { -struct rq_flags rf; -struct rq *rq; - - rq =task_rq_lock (p, &rf); -if (se->sched_delayed) { -update_rq_clock (rq); -dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); - } -task_rq_unlock (rq, p, &rf); - } -remove_entity_load_avg (se); }@@ -9795,21 +9727,10 @@ static void set_cpus_allowed_fair(structset_task_max_allowed_capacity (p); } -static void set_next_buddy (struct sched_entity *se) -{ -for_each_sched_entity (se) { -if (WARN_ON_ONCE (!se->on_rq)) -return ; -if (se_is_idle (se)) -return ; -cfs_rq_of (se)->next = se; - } -} -enum preempt_wakeup_action {PREEMPT_WAKEUP_NONE ,/* No preemption. */ PREEMPT_WAKEUP_SHORT ,/* Ignore slice protection. */ -PREEMPT_WAKEUP_PICK ,/* Let __pick_eevdf() decide. */ +PREEMPT_WAKEUP_PICK ,/* Let pick_eevdf() decide. */ PREEMPT_WAKEUP_RESCHED ,/* Force reschedule. */ };@@ -9826,7 +9747,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,if (cfs_rq->next &&entity_before (cfs_rq->next, pse))return false; -set_next_buddy (pse); +set_next_buddy (cfs_rq, pse);return true; }@@ -9879,7 +9800,7 @@ static void wakeup_preempt_fair(struct renum preempt_wakeup_action preempt_action =PREEMPT_WAKEUP_PICK ;struct task_struct *donor = rq->donor;struct sched_entity *nse, *se = &donor->se, *pse = &p->se; -struct cfs_rq *cfs_rq =task_cfs_rq (donor); +struct cfs_rq *cfs_rq = &rq->cfs;int cse_is_idle, pse_is_idle;/* @@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct rif (!sched_feat (WAKEUP_PREEMPTION ))return ; -find_matching_se (&se, &pse);WARN_ON_ONCE (!pse); cse_is_idle =se_is_idle (se);@@ -9944,8 +9864,7 @@ static void wakeup_preempt_fair(struct rif (unlikely (!normal_policy (p->policy)))return ; - cfs_rq =cfs_rq_of (se); -update_curr (cfs_rq); +update_curr_fair (rq);/* * If @p has a shorter slice than current and @p is eligible, override * current's slice protection in order to allow preemption. @@ -9989,18 +9908,15 @@ static void wakeup_preempt_fair(struct r } pick: - nse =pick_next_entity (rq, cfs_rq, preempt_action !=PREEMPT_WAKEUP_SHORT ); -/* If @p has become the most eligible task, force preemption */ -if (nse == pse) - goto preempt; - -/* -* Because p is enqueued, nse being null can only mean that we -* dequeued a delayed task. If there are still entities queued in -* cfs, check if the next one will be p. -*/ -if (!nse && cfs_rq->nr_queued) - goto pick; +if (cfs_rq->h_nr_queued) { + nse =pick_next_entity (rq, preempt_action !=PREEMPT_WAKEUP_SHORT ); +if (unlikely (!nse)) + goto pick; + +/* If @p has become the most eligible task, force preemption */ +if (nse == pse) + goto preempt; + }if (sched_feat (RUN_TO_PARITY ))update_protect_slice (cfs_rq, se);@@ -10019,33 +9935,24 @@ static void wakeup_preempt_fair(struct rstruct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf)__must_hold (__rq_lockp (rq)) { +struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; -struct cfs_rq *cfs_rq;struct task_struct *p; -bool throttled;int new_tasks; again: - cfs_rq = &rq->cfs; -if (!cfs_rq->nr_queued) +if (!cfs_rq->h_nr_queued) goto idle; - throttled = false; - -do { -/* Might not have done put_prev_entity() */ -if (cfs_rq->curr && cfs_rq->curr->on_rq) -update_curr (cfs_rq); +/* Might not have done put_prev_entity() */ +if (cfs_rq->curr && cfs_rq->curr->on_rq) +update_curr (cfs_rq); - se =pick_next_entity (rq, cfs_rq, true); -if (!se) - goto again; - cfs_rq =group_cfs_rq (se); - }while (cfs_rq); + se =pick_next_entity (rq, true); +if (!se) + goto again; p =task_of (se); -if (unlikely (throttled)) -task_throttle_setup_work (p);return p; idle:@@ -10079,7 +9986,7 @@ void fair_server_init(struct rq *rq)static void put_prev_task_fair (struct rq *rq,struct task_struct *prev,struct task_struct *next) {struct sched_entity *se = &prev->se; -struct cfs_rq *cfs_rq; +struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *nse =NULL ;#ifdef CONFIG_FAIR_GROUP_SCHED @@ -10089,7 +9996,7 @@ static void put_prev_task_fair(struct rqwhile (se) { cfs_rq =cfs_rq_of (se); -if (!nse || cfs_rq->curr) +if (!nse || cfs_rq->h_curr)put_prev_entity (cfs_rq, se);#ifdef CONFIG_FAIR_GROUP_SCHED if (nse) {@@ -10108,6 +10015,14 @@ static void put_prev_task_fair(struct rq#endif se =parent_entity (se); } + +/* Put 'current' back into the tree. */ + cfs_rq = &rq->cfs; + se = &prev->se; +WARN_ON_ONCE (cfs_rq->curr != se); + cfs_rq->curr =NULL ; +if (se->on_rq) +__enqueue_entity (cfs_rq, se); }/* @@ -10116,8 +10031,8 @@ static void put_prev_task_fair(struct rqstatic void yield_task_fair (struct rq *rq) {struct task_struct *curr = rq->donor; -struct cfs_rq *cfs_rq =task_cfs_rq (curr);struct sched_entity *se = &curr->se; +struct cfs_rq *cfs_rq = &rq->cfs;/* * Are we the only task in the tree? @@ -10158,11 +10073,11 @@ static bool yield_to_task_fair(struct rqstruct sched_entity *se = &p->se;/* !se->on_rq also covers throttled task */ -if (!se->on_rq) +if (!se->on_rq || se->sched_delayed)return false;/* Tell the scheduler that we'd really like se to run next. */ -set_next_buddy (se); +set_next_buddy (&task_rq (p)->cfs, se);yield_task_fair (rq);@@ -10501,15 +10416,10 @@ static inline long migrate_degrades_loca*/ static inline int task_is_ineligible_on_dst_cpu (struct task_struct *p,int dest_cpu) { -struct cfs_rq *dst_cfs_rq; +struct cfs_rq *dst_cfs_rq = &cpu_rq (dest_cpu)->cfs; -#ifdef CONFIG_FAIR_GROUP_SCHED - dst_cfs_rq =tg_cfs_rq (task_group (p), dest_cpu); -#else - dst_cfs_rq = &cpu_rq (dest_cpu)->cfs; -#endif -if (sched_feat (PLACE_LAG ) && dst_cfs_rq->nr_queued && - !entity_eligible (task_cfs_rq (p), &p->se)) +if (sched_feat (PLACE_LAG ) && dst_cfs_rq->h_nr_queued && + !entity_eligible (&task_rq (p)->cfs, &p->se))return 1 ;return 0 ;@@ -11292,7 +11202,7 @@ static void update_cfs_rq_h_load(structwhile ((se =READ_ONCE (cfs_rq->h_load_next)) !=NULL ) { load = cfs_rq->h_load; load =div64_ul (load * se->avg.load_avg, -cfs_rq_load_avg (cfs_rq) +1 ); +cfs_rq_load_avg (cfs_rq) +1 ); cfs_rq =group_cfs_rq (se); cfs_rq->h_load = load; cfs_rq->last_h_load_update = now;@@ -14684,7 +14594,7 @@ static inline void task_tick_core(struct* MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check * if we need to give up the CPU. */ -if (rq->core->core_forceidle_count && rq->cfs.nr_queued ==1 && +if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued ==1 &&__entity_slice_used (&curr->se,MIN_NR_TASKS_DURING_FORCEIDLE ))resched_curr (rq); }@@ -14893,30 +14803,8 @@ bool cfs_prio_less(const struct task_strWARN_ON_ONCE (task_rq (b)->core != rq->core); -#ifdef CONFIG_FAIR_GROUP_SCHED -/* -* Find an se in the hierarchy for tasks a and b, such that the se's -* are immediate siblings. -*/ -while (sea->cfs_rq->tg != seb->cfs_rq->tg) { -int sea_depth = sea->depth; -int seb_depth = seb->depth; - -if (sea_depth >= seb_depth) - sea =parent_entity (sea); -if (sea_depth <= seb_depth) - seb =parent_entity (seb); - } - -se_fi_update (sea, rq->core->core_forceidle_seq, in_fi); -se_fi_update (seb, rq->core->core_forceidle_seq, in_fi); - - cfs_rqa = sea->cfs_rq; - cfs_rqb = seb->cfs_rq; -#else /* !CONFIG_FAIR_GROUP_SCHED: */ cfs_rqa = &task_rq (a)->cfs; cfs_rqb = &task_rq (b)->cfs; -#endif /* !CONFIG_FAIR_GROUP_SCHED */ /* * Find delta after normalizing se's vruntime with its cfs_rq's @@ -14955,11 +14843,20 @@ static inline void task_tick_core(structstatic void task_tick_fair (struct rq *rq,struct task_struct *curr,int queued) {struct sched_entity *se = &curr->se; -struct cfs_rq *cfs_rq; -for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); -entity_tick (cfs_rq, se, queued); +if (se->on_rq) { +unsigned long weight =NICE_0_LOAD ; +struct cfs_rq *cfs_rq; + +for_each_sched_entity (se) { + cfs_rq =cfs_rq_of (se); +entity_tick (cfs_rq, se, queued); + + weight =__calc_prop_weight (cfs_rq, se, weight); + } + + se = &curr->se; +reweight_eevdf (cfs_rq, se, weight, se->on_rq); }if (queued)@@ -14999,7 +14896,7 @@ prio_changed_fair(struct rq *rq, structif (p->prio == oldprio)return ; -if (rq->cfs.nr_queued ==1 ) +if (rq->cfs.h_nr_queued ==1 )return ;/* @@ -15128,33 +15025,44 @@ static void switched_to_fair(struct rq * } } -/* -* Account for a task changing its policy or group. -* -* This routine is mostly called to set cfs_rq->curr field when a task -* migrates between groups/classes. -*/ static void set_next_task_fair (struct rq *rq,struct task_struct *p,bool first) {struct sched_entity *se = &p->se;bool throttled = false; +struct cfs_rq *cfs_rq = &rq->cfs; +unsigned long weight =NICE_0_LOAD ; +bool on_rq = se->on_rq; + +clear_buddies (cfs_rq, se); + +if (on_rq) +__dequeue_entity (cfs_rq, se);for_each_sched_entity (se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); + cfs_rq =cfs_rq_of (se); -if (IS_ENABLED (CONFIG_FAIR_GROUP_SCHED ) && - first && cfs_rq->curr) -break ; +if (!IS_ENABLED (CONFIG_FAIR_GROUP_SCHED ) || + !first || !cfs_rq->h_curr) +set_next_entity (cfs_rq, se); -set_next_entity (cfs_rq, se, first);/* ensure bandwidth has been allocated on our new cfs_rq */ throttled |=account_cfs_rq_runtime (cfs_rq,0 ); + +if (on_rq) + weight =__calc_prop_weight (cfs_rq, se, weight); }if (throttled)task_throttle_setup_work (p); se = &p->se; + cfs_rq->curr = se; + +if (on_rq) { +reweight_eevdf (cfs_rq, se, weight, se->on_rq); +if (first) +set_protect_slice (cfs_rq, se); + }if (task_on_rq_queued (p)) {/* @@ -15267,17 +15175,8 @@ void unregister_fair_sched_group(structstruct sched_entity *se =tg_se (tg, cpu);struct rq *rq =cpu_rq (cpu); -if (se) { -if (se->sched_delayed) { -guard (rq_lock_irqsave)(rq); -if (se->sched_delayed) { -update_rq_clock (rq); -dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); - } -list_del_leaf_cfs_rq (cfs_rq); - } +if (se)remove_entity_load_avg (se); - }/* * Only empty task groups can be destroyed; so we can speculatively kernel/sched/pelt.c +3
-3@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched/* * running is a subset of runnable (weight) so running can't be set if * runnable is clear. But there are some corner cases where the current -* se has been already dequeued but cfs_rq->curr still points to it. +* se has been already dequeued but cfs_rq->h_curr still points to it. * This means that weight will be 0 but not running for a sched_entity * but also for a cfs_rq if the latter becomes idle. As an example, * this happens during sched_balance_newidle() which calls @@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 nowint __update_load_avg_se (u64 now,struct cfs_rq *cfs_rq,struct sched_entity *se) {if (___update_load_sum (now, &se->avg, !!se->on_rq,se_runnable (se), - cfs_rq->curr == se)) { + cfs_rq->h_curr == se)) {___update_load_avg (&se->avg,se_weight (se));cfs_se_util_change (&se->avg);@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, stif (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), cfs_rq->h_nr_runnable, - cfs_rq->curr !=NULL )) { + cfs_rq->h_curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );trace_pelt_cfs_tp (cfs_rq);kernel/sched/sched.h +13
-13@@ -530,21 +530,8 @@ struct task_group { }; -#ifdef CONFIG_GROUP_SCHED_WEIGHT #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD -/* -* A weight of 0 or 1 can cause arithmetics problems. -* A weight of a cfs_rq is the sum of weights of which entities -* are queued on this cfs_rq, so a weight of a entity should not be -* too large, so as the shares value of a task group. -* (The default weight is 1024 - so there's no practical -* limitation from this.) -*/ -#define MIN_SHARES (1UL << 1) -#define MAX_SHARES (1UL << 18) -#endif -typedef int (*tg_visitor )(struct task_group *,void *); extern int walk_tg_tree_from(struct task_group *from,@@ -631,6 +618,17 @@ static inline bool cfs_task_bw_constrain#endif /* !CONFIG_CGROUP_SCHED */ +/* +* A weight of 0 or 1 can cause arithmetics problems. +* A weight of a cfs_rq is the sum of weights of which entities +* are queued on this cfs_rq, so a weight of a entity should not be +* too large, so as the shares value of a task group. +* (The default weight is 1024 - so there's no practical +* limitation from this.) +*/ +#define MIN_SHARES (1UL << 1) +#define MAX_SHARES (1UL << 18) +extern void unregister_rt_sched_group (struct task_group *tg);extern void free_rt_sched_group (struct task_group *tg);extern int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent);@@ -709,6 +707,7 @@ struct cfs_rq {/* * CFS load tracking */ +struct sched_entity *h_curr;struct sched_avg avg;#ifndef CONFIG_64BIT u64 last_update_time_copy;@@ -2575,6 +2574,7 @@ extern const u32 sched_prio_to_wmult[40#define ENQUEUE_MIGRATED 0x00040000#define ENQUEUE_INITIAL 0x00080000#define ENQUEUE_RQ_SELECTED 0x00100000 +#define ENQUEUE_QUEUED 0x00200000#define RETRY_TASK ((void *)-1UL)
Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 43 lines · Aishwarya Rambhadran
Hi Peter,
We have observed some schbench p99 request latency regressions while comparing v7.2 against v7.3 cycle on an AWS Graviton3 (m7g.metal) using Fastpath (Linux kernel performance benchmarking tool).
The schbench suite as a whole does not show a significant regression, and the results appear to be a mix of improvements and regressions depending on the thread configuration. However, a few configurations show fairly large p99 request latency regressions. For example, comparing v7.2 against v7.3-rc4:
message-thread:16, worker-thread:16
v7.2 = 662869 us
v7.3-rc4 = 969557 us
change% = -31.63%message-thread:64, worker-thread:4
v7.2 = 730453 us
v7.3-rc4 = 942251 us
change% = -22.48%message-thread:32, worker-thread:4
v7.2 = 74251 us
v7.3-rc4 = 56725 us
change% = +30.90%So this does not appear to be a general latency degradation; some configs improve while others regress.
I also ran a Fastpath automated git bisection for the m 16, t 16 p99 request latency regression between v7.2 and v7.3-rc1, which identified this commit as the first bad commit. The original regression was ~30.9%, & the bisection reproducibility check showed ~27.8% regression with 20 repeats across 2 boot sessions.
This commit looks relevant since it changes EEVDF runqueue organization and is intended, in part, to address latency issues with hierarchical fair/ cgroup scheduling. I also noticed that there have been fixes on top of this change, including
68e37487810a("sched/fair: Fix flat hierarchy"), which fixes vruntime handling and has a Fixes:85570f10a4c6tag.
Quoted 9 lines
Given that the overall schbench result is mostly neutral while some p99 configurations move considerably in both directions, I wanted to check whether this could be an expected trade-off from the single-runqueue change, or whether the p99 regressions are worth investigating further.
I can run additional tests or provide the full results/bisection data if useful.
Would disabling WA_WEIGHT help? I happened to find that this workaround works for me to restore netperf performance (though it is a throughput score rather than a latency one), it seems that task_h_load() becomes much larger(also on sched/task_h_load):
echo NO_WA_WEIGHT > /sys/kernel/debug/sched/features
https://lore.kernel.org/lkml/aoxah90s0bQ4tcUW@three-body/
thanks,
Chenyu
K Prateek Nayak
reviewerRe: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 54 lines · Chen Yu
Quoted 43 lines · Aishwarya Rambhadran
Hi Peter,
We have observed some schbench p99 request latency regressions while comparing v7.2 against v7.3 cycle on an AWS Graviton3 (m7g.metal) using Fastpath (Linux kernel performance benchmarking tool).
The schbench suite as a whole does not show a significant regression, and the results appear to be a mix of improvements and regressions depending on the thread configuration. However, a few configurations show fairly large p99 request latency regressions. For example, comparing v7.2 against v7.3-rc4:
message-thread:16, worker-thread:16
v7.2 = 662869 us
v7.3-rc4 = 969557 us
change% = -31.63%message-thread:64, worker-thread:4
v7.2 = 730453 us
v7.3-rc4 = 942251 us
change% = -22.48%message-thread:32, worker-thread:4
v7.2 = 74251 us
v7.3-rc4 = 56725 us
change% = +30.90%So this does not appear to be a general latency degradation; some configs improve while others regress.
I also ran a Fastpath automated git bisection for the m 16, t 16 p99 request latency regression between v7.2 and v7.3-rc1, which identified this commit as the first bad commit. The original regression was ~30.9%, & the bisection reproducibility check showed ~27.8% regression with 20 repeats across 2 boot sessions.
This commit looks relevant since it changes EEVDF runqueue organization and is intended, in part, to address latency issues with hierarchical fair/ cgroup scheduling. I also noticed that there have been fixes on top of this change, including
68e37487810a("sched/fair: Fix flat hierarchy"), which fixes vruntime handling and has a Fixes:85570f10a4c6tag.Quoted 9 lines
Given that the overall schbench result is mostly neutral while some p99 configurations move considerably in both directions, I wanted to check whether this could be an expected trade-off from the single-runqueue change, or whether the p99 regressions are worth investigating further.
I can run additional tests or provide the full results/bisection data if useful.
Quoted 6 lines
Would disabling
WA_WEIGHThelp? I happened to find that this workaround works for me to restore netperf performance (though it is a throughput score rather than a latency one), it seems thattask_h_load()becomes muchlarger(also on sched/task_h_load):
echoNO_WA_WEIGHT>/sys/kernel/debug/sched/features
https://lore.kernel.org/lkml/aoxah90s0bQ4tcUW@three-body/
In addition to Chenyu's suggestion, I would also suggest trying out different cgroup modes via "/sys/kernel/debug/sched/cgroup_mode" if NO_WA_WEIGHT doesn't resolve your issue.
(P.S. Peter has a branch with some fixes for the task_h_load() at
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=sched/task_h_load
in case you can try a custom kernel build on your setup)
In my testing, I did see some regression in schbench with the default "concur" mode for RPS metric but the "smp" mode which used the older formula for weight calculation seemed to be running on par with the hierarchical pick.
https://lore.kernel.org/lkml/[email protected]/
Signature · 2 lines
Thanks and Regards, Prateek
Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
In reply to Chen YuQuoted 7 lines · Chen Yu
Would disabling
WA_WEIGHThelp? I happened to find that this workaround works for me to restore netperf performance (though it is a throughput score rather than a latency one), it seems thattask_h_load()becomes muchlarger(also on sched/task_h_load):
echoNO_WA_WEIGHT>/sys/kernel/debug/sched/features
https://lore.kernel.org/lkml/aoxah90s0bQ4tcUW@three-body/
A long while back, I caught WA_WEIGHT causing firefox's event thread pool to stack on a single CPU. In that particular case, the victims (32 of 'em in 8 rq box!) did nothing but go back to sleep, so no real harm was done. However, I also measured modest real stacking impact, so gave task_h_load() a floor on GP.
Perhaps weightless burst stacking is a bigger deal for some than it was here? That firefox event thread burst showed potential.
-Mike
Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Hi Mike, Prateek,
Quoted 11 lines · Mike Galbraith
Quoted 7 lines · Chen Yu
Would disabling
WA_WEIGHThelp? I happened to find that this workaround works for me to restore netperf performance (though it is a throughput score rather than a latency one), it seems thattask_h_load()becomes muchlarger(also on sched/task_h_load):
echoNO_WA_WEIGHT>/sys/kernel/debug/sched/features
https://lore.kernel.org/lkml/aoxah90s0bQ4tcUW@three-body/A long while back, I caught
WA_WEIGHTcausing firefox's event thread pool to stack on a single CPU.
I just realized that what I suggested - disabling WA_WEIGHT - does the opposite. Actually, WA_WEIGHT does help in the netperf scenario. Disabling WA_WEIGHT makes the performance of both "smp" and "concur" modes drop to the same poor level, which makes it look like the regression disappears, but actually it does not.
According to my test last week, stacking the waker and wakee on the same CPU brings a big improvement on my machine, iff the memory bandwidth is saturated. By comparison, if the memory bandwidth is low, stacking the wakee on top of the waker causes harm.
The machine I used has 2 NUMA nodes; each node has 96 cores, and these 96 cores share L3, while every 4 cores share L2. The L2 miss rate is much lower if the waker and wakee are on the same CPU than if the wakee is put on other idle CPUs in the same L2 domain. Yes, the same L2 - which goes against intuition. How could the L2 miss rate rise even if the waker and wakee are in the same L2 domain?
I guess this is because stacking the waker and wakee on the same CPU and letting them run alternately would be the most efficient way to consume the data without evicting L2 cache lines too much, when the memory bandwidth is saturated.
Quoted 8 lines
In that particular case, the victims
(32 of 'em in 8 rq box!) did nothing but go back to sleep, so no real harm was done. However, I also measured modest real stacking impact, so gavetask_h_load()a floor on GP.Perhaps weightless burst stacking is a bigger deal for some than it was here? That firefox event thread burst showed potential.
It depends on the workload and the memory bandwidth of the system, I suppose. If the workload is a producer-consumer type, and the system is under memory bandwidth pressure, stacking the producer and consumer would bring good data locality - in terms of L2 locality, per my test.
In summary, the "regression" of flat cgroup task-pick on netperf (or, producer-consumer type workloads) seems to be the expected consequence. Maybe we need to make WA_WEIGHT consider more factors here.
I'll collect more data and provide the information later.
thanks,
Chenyu
Quoted 1 line
-Mike
Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 4 lines · Chen Yu
According to my test last week, stacking the waker and wakee on the same CPU brings a big improvement on my machine, iff the memory bandwidth is saturated. By comparison, if the memory bandwidth is low, stacking the wakee on top of the waker causes harm.
Ditto CPU wise. The impact of even modest waker/wakee concurrency is considerable. For much of netperf, the post-wakeup tail becomes a CPU service latency injection for the wakee when unnecessarily stacked. While cache misses sting, those hurt.
As box approaches saturation, stacking of somewhat synchronous stuff becomes a better bet. Aiming for having only waker's tail between wakee and CPU has decent chance of being a winner in a box full of wider obstacles.
-Mike
Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Hi all,
Thanks for the suggestions. I ran a couple of follow-up tests on the same AWS Graviton3 (m7g.metal) setup with v7.3-rc4:
1. Disable WA_WEIGHT:
echo NO_WA_WEIGHT > /sys/kernel/debug/sched/features
2. Use the "smp" cgroup mode:
echo smp > /sys/kernel/debug/sched/cgroup_mode
Both tests were run with 20 repeats across 2 boot sessions. Below are the request latency p99 results for schbench configurations discussed earlier (usec, smaller is better):
message-thread:16, worker-thread:16
v7.2 = 662869
v7.3-rc4 = 969557NO_WA_WEIGHT = 958541
smp = 723098
message-thread:64, worker-thread:4
v7.2 = 730453
v7.3-rc4 = 942251NO_WA_WEIGHT = 740454
smp = 818483
message-thread:32, worker-thread:4
v7.2 = 74251
v7.3-rc4 = 56725NO_WA_WEIGHT = 59008
smp = 82053
For the m:16 t:16 case that I originally bisected, disabling WA_WEIGHT does not materially change the regression. Switching to cgroup_mode=smp, however, recovers most of it.
The m:64 t:4 case behaves differently. Disabling WA_WEIGHT brings the p99 latency almost back to the v7.2 value, while using smp mode gives a partial recovery.
There is also an interesting result with m:32 t:4, which was one of the configurations that improved with v7.3-rc4. That improvement is mostly preserved with NO_WA_WEIGHT, but disappears with cgroup_mode=smp.
So this does not seem to point to WA_WEIGHT alone as the cause of the original regression. The results seem consistent with the workload/load- dependent behavior discussed in this thread, where weight calculation and resulting placement decisions can help some schbench configurations while hurting others.
For the original m:16 t:16 regression in particular, the larger recovery with cgroup_mode=smp also seems consistent with Prateek's observation that the older weight calculation performs closer to the hierarchical pick.
Overall, the results suggest that the latency changes are sensitive to the workload configuration and the weight calculation being used, rather than being a uniform regression from the single-runqueue change. It would be interesting to understand whether this variation across load points is an expected consequence of the new weight calculation, or whether there is still scope to avoid the larger p99 regressions.
Thanks,
Aishwarya
Quoted 15 lines · Mike Galbraith
Quoted 3 lines · Chen Yu
According to my test last week, stacking the waker and wakee on the same CPU brings a big improvement on my machine, iff the memory bandwidth is saturated. By comparison, if the memory bandwidth is low, stacking the wakee on top of the waker causes harm.
Ditto CPU wise. The impact of even modest waker/wakee concurrency is considerable. For much of netperf, the post-wakeup tail becomes a CPU service latency injection for the wakee when unnecessarily stacked. While cache misses sting, those hurt.
As box approaches saturation, stacking of somewhat synchronous stuff becomes a better bet. Aiming for having only waker's tail between wakee and CPU has decent chance of being a winner in a box full of wider obstacles.
-Mike
Re: [REGRESSION] [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 33 lines · Aishwarya Rambhadran
Hi all,
Thanks for the suggestions. I ran a couple of follow-up tests on the same AWS Graviton3 (m7g.metal) setup with v7.3-rc4:
1. Disable
WA_WEIGHT:
echoNO_WA_WEIGHT>/sys/kernel/debug/sched/features2. Use the "smp" cgroup mode:
echo smp >/sys/kernel/debug/sched/cgroup_modeBoth tests were run with 20 repeats across 2 boot sessions. Below are the request latency p99 results for schbench configurations discussed earlier (usec, smaller is better):
message-thread:16, worker-thread:16
v7.2 = 662869
v7.3-rc4 = 969557NO_WA_WEIGHT= 958541
smp = 723098message-thread:64, worker-thread:4
v7.2 = 730453
v7.3-rc4 = 942251NO_WA_WEIGHT= 740454
smp = 818483message-thread:32, worker-thread:4
v7.2 = 74251
v7.3-rc4 = 56725NO_WA_WEIGHT= 59008
smp = 82053
Quoted 3 lines
For the m:16 t:16 case that I originally bisected, disabling
WA_WEIGHTdoes not materially change the regression. Switching to cgroup_mode=smp, however, recovers most of it.
I took flat back to 7.2 (to test only that) and did a little mixed load 7.2-virgin vs 7.2+flat testing with defaults. CPU distribution delta when switching competitor load configuration from 1 group of 8 to 4 groups of 2 vs a fixed 4 client netperf load made it clear that numbers for anything cgroup vs cgroup will definitely wiggle more than usual.
-Mike
Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraHi Peter,
Quoted 7 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
[...]
kernel/sched/fair.c static void wakeup_preempt_fair(struct r · line 9842
Quoted 7 lines
kernel/sched/fair.c +0
-1@@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct rif (!sched_feat (WAKEUP_PREEMPTION ))return ; -find_matching_se (&se, &pse);WARN_ON_ONCE (!pse); cse_is_idle =se_is_idle (se);
I think this changes how cpu.idle affects wakeup preemption.
With the single runqueue, find_matching_se() is gone and se_is_idle() reduces to task_has_idle_policy(). Wouldn't this mean that SCHED_NORMAL tasks in a cpu.idle cgroup are no longer treated as idle for wakeup_preempt_fair()?
I tested my existing 7.2.9 desktop against 7.3-rc6 with the setup below. cyclictest doesn't reproduce it because it just sleeps and wakes, no CPU burn and therefore its accrued EEVDF lag keeps the latency low?
cc -O2 -o wakelat wakelat.c
systemd-run --user --collect --unit=spin -p AllowedCPUs=7 -p CPUWeight=idle sh -c 'while :; do :; done' systemd-run --user --collect --wait --pipe --same-dir -p AllowedCPUs=7 ./wakelat 1000 15 200 |
mlr --inidx --ifs space --opprint --barred --right label lat \
then stats1 -a count,p50,p90,p99,max -f lat then rename -r '^lat_,' \
then put -q 'emit mapsum({"kernel": system("uname -r")}, $*)'systemctl --user stop spin
wakelat wakes on a fixed 1 ms schedule, records how late each wakeup was, then busy-loops for 200 micros (wall clock).
+-----------+-------+-----+-----+------+------+ | kernel | count | p50 | p90 | p99 | max | +-----------+-------+-----+-----+------+------+ | 7.2.9 | 14998 | 11 | 11 | 12 | 1869 | | 7.3.0-rc6 | 13853 | 11 | 12 | 2029 | 2116 | +-----------+-------+-----+-----+------+------+
count is out of 15000 periods. rc6 had 1147 wakeups delayed by over 1 ms.
wakelat.c:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/prctl.h>
#include <time.h>
static long long now_ns(void)
{
struct timespec t;
clock_gettime(CLOCK_MONOTONIC, &t);
return t.tv_sec * 1000000000LL + t.tv_nsec;
}int main(int argc, char **argv)
{
if (argc != 4)
return 2;
long long interval = atoll(argv[1]) * 1000, dur = atoll(argv[2]) * 1000000000LL;
long long work = atoll(argv[3]) * 1000;
long max = dur / interval + 1, n = 0;
long *lat = malloc(max * sizeof(*lat));
prctl(PR_SET_TIMERSLACK, 1UL);
long long start = now_ns(), next = start + interval;
while (next - start < dur && n < max) {
struct timespec ts = { next / 1000000000LL, next % 1000000000LL };
clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &ts, NULL);
long long now = now_ns();
lat[n++] = (now - next) / 1000;
while (now_ns() - now < work)
;
do next += interval; while (next <= now);
}
for (long i = 0; i < n; i++)
printf("%ld\n", lat[i]);
return 0;
}Regards,
Jakub
Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Greetings!
Quoted 8 lines · Jakub Okoński
Hi Peter,
Quoted 4 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
[...]
Quoted 12 lines
Quoted 10 lines
kernel/sched/fair.c +0
-0@@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r if (!sched_feat(WAKEUP_PREEMPTION)) return;kernel/sched/fair.c +0
-1-find_matching_se (&se, &pse);
WARN_ON_ONCE(!pse);
cse_is_idle=se_is_idle(se);I think this changes how cpu.idle affects wakeup preemption.
My box claims this to be unrelated to the jitter you're seeing, as it is present in 7.2.11, with and without the sched-flat set applied.
Quoted 20 lines
With the single runqueue,
find_matching_se()is gone andse_is_idle()reduces totask_has_idle_policy(). Wouldn't this mean thatSCHED_NORMALtasks in a cpu.idle cgroup are no longer treated as idle forwakeup_preempt_fair()?I tested my existing 7.2.9 desktop against 7.3-rc6 with the setup below. cyclictest doesn't reproduce it because it just sleeps and wakes, no CPU burn and therefore its accrued EEVDF lag keeps the latency low?
cc -O2 -o wakelat
wakelat.c
systemd-run --user --collect --unit=spin -p AllowedCPUs=7 -p CPUWeight=idle sh -c 'while :; do :; done' systemd-run --user --collect --wait --pipe --same-dir -p AllowedCPUs=7 ./wakelat 1000 15 200 | mlr --inidx --ifs space --opprint --barred --right label lat \
then stats1 -a count,p50,p90,p99,max -f lat then rename -r '^lat_,' \
then put -q 'emitmapsum({"kernel": system("uname -r")}, $*)'
systemctl --user stop spinwakelat wakes on a fixed 1 ms schedule, records how late each wakeup was, then busy-loops for 200 micros (wall clock).
Quoted 8 lines
+-----------+-------+-----+-----+------+------+ > kernel | count | p50 | p90 | p99 | max | +-----------+-------+-----+-----+------+------+ > 7.2.9 | 14998 | 11 | 11 | 12 | 1869 | > 7.3.0-rc6 | 13853 | 11 | 12 | 2029 | 2116 | +-----------+-------+-----+-----+------+------+count is out of 15000 periods. rc6 had 1147 wakeups delayed by over 1 ms.
Thanks a bunch for the proggy. I've been looking for a clear signal with various loads, and stuffing wakelat+awk into one of my scripts provided a perfectly clear signal, which I bisected and confirmed.
Note: below is box pointing it's little silicon finger at the source of aforementioned jitter, not necessarily at breakage.
Reverting the fingered commit removed the jitter from 7.2.11, and force reverting 68e37487810a + 333238da9a19 removed it from master as well.
333238da9a193ffc58792995f3e951e4cb87bfd2 is the first 'bad' commit commit 333238da9a193ffc58792995f3e951e4cb87bfd2
Author: Vincent Guittot <[email protected]> Date: Fri Aug 14 15:52:41 2026 +0200
sched: Update time before requeueing delayed entities
In order to compute the right lag, it is required to update time to 'now'. Without this, the delayed entity might appear younger than it really is and receive less compensation for having waited.
kernel/sched/fair.c | 2 ++ 1 file changed, 2 insertions(+)
Vincent Guittot
maintainerRe: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 24 lines · Mike Galbraith
Greetings!
Quoted 8 lines · Jakub Okoński
Hi Peter,
Quoted 4 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
[...]
Quoted 12 lines
Quoted 10 lines
kernel/sched/fair.c +0
-0@@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r if (!sched_feat(WAKEUP_PREEMPTION)) return;kernel/sched/fair.c +0
-1-find_matching_se (&se, &pse);
WARN_ON_ONCE(!pse);
cse_is_idle=se_is_idle(se);I think this changes how cpu.idle affects wakeup preemption.
Quoted 2 lines
My box claims this to be unrelated to the jitter you're seeing, as it is present in 7.2.11, with and without the sched-flat set applied.
Yes, we don't take into account whether the task group is sched_idle
Jakub, Mike,
Could you try the patch below ? It solves the problem on my setup
--- kernel/sched/fair.c | 80 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 68 insertions(+), 12 deletions(-)
kernel/sched/fair.c +68 -12
@@ -433,6 +433,38 @@ static inline struct sched_entity *parent_entity(const struct sched_entity *se)return se->parent; } +static void +find_matching_se (struct sched_entity **se,struct sched_entity **pse) +{ +int se_depth, pse_depth; + +/* +* preemption test can be made between sibling entities who are in the +* same cfs_rq i.e who have a common parent. Walk up the hierarchy of +* both tasks until we find their ancestors who are siblings of common +* parent. +*/ + +/* First walk up until both entities are at same depth */ + se_depth = (*se)->depth; + pse_depth = (*pse)->depth; + +while (se_depth > pse_depth) { + se_depth--; + *se =parent_entity (*se); + } + +while (pse_depth > se_depth) { + pse_depth--; + *pse =parent_entity (*pse); + } + +while (!is_same_group (*se, *pse)) { + *se =parent_entity (*se); + *pse =parent_entity (*pse); + } +} +static int tg_is_idle (struct task_group *tg) {return tg->idle >0 ;
@@ -476,6 +508,11 @@ static inline struct sched_entity *parent_entity(struct sched_entity *se)return NULL ; } +static inline void +find_matching_se (struct sched_entity **se,struct sched_entity **pse) +{ +} +static inline int tg_is_idle (struct task_group *tg) {return 0 ;
@@ -10129,7 +10166,34 @@ preempt_sync(struct rq *rq, int wake_flags, }/* -* Preempt the current task with a newly woken task if needed: +* Preempt an idle entity in favor of a non-idle entity (and don't preempt +* in the inverse case). +*/ +static bool wakeup_preempt_sched_idle (struct sched_entity *cse,struct sched_entity *pse) +{ +int cse_is_idle, pse_is_idle; + +find_matching_se (&cse, &pse); +WARN_ON_ONCE (!pse); + + cse_is_idle =se_is_idle (cse); + pse_is_idle =se_is_idle (pse); + +/* +* Preempt an idle entity in favor of a non-idle entity (and don't preempt +* in the inverse case). +*/ +if (cse_is_idle && !pse_is_idle) +return PREEMPT_WAKEUP_RESCHED ; + +if (cse_is_idle != pse_is_idle) +return PREEMPT_WAKEUP_NONE ; + +return PREEMPT_WAKEUP_PICK ; +} + +/* +* Preempt the current task with a newly woken task if needed. */ static void wakeup_preempt_fair (struct rq *rq,struct task_struct *p,int wake_flags) {
@@ -10174,22 +10238,14 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_fif (!sched_feat (WAKEUP_PREEMPTION ))return ; -WARN_ON_ONCE (!pse); - - cse_is_idle =se_is_idle (se); - pse_is_idle =se_is_idle (pse); + preempt_action =wakeup_preempt_sched_idle (se, pse); - nse = se; -/* -* Preempt an idle entity in favor of a non-idle entity (and don't preempt -* in the inverse case). -*/ -if (cse_is_idle && !pse_is_idle) +if (preempt_action ==PREEMPT_WAKEUP_RESCHED ) goto preempt;update_curr_fair (rq); -if (cse_is_idle != pse_is_idle) +if (preempt_action ==PREEMPT_WAKEUP_NONE ) goto update;/*
Signature · 1 line
2.53.0
Quoted 58 lines
Quoted 22 lines
With the single runqueue,
find_matching_se()is gone andse_is_idle()reduces totask_has_idle_policy(). Wouldn't this mean thatSCHED_NORMALtasks in a cpu.idle cgroup are no longer treated as idle forwakeup_preempt_fair()?I tested my existing 7.2.9 desktop against 7.3-rc6 with the setup below. cyclictest doesn't reproduce it because it just sleeps and wakes, no CPU burn and therefore its accrued EEVDF lag keeps the latency low?
cc -O2 -o wakelat
wakelat.c
systemd-run --user --collect --unit=spin -p AllowedCPUs=7 -p CPUWeight=idle sh -c 'while :; do :; done' systemd-run --user --collect --wait --pipe --same-dir -p AllowedCPUs=7 ./wakelat 1000 15 200 | mlr --inidx --ifs space --opprint --barred --right label lat \
then stats1 -a count,p50,p90,p99,max -f lat then rename -r '^lat_,' \
then put -q 'emitmapsum({"kernel": system("uname -r")}, $*)'
systemctl --user stop spinwakelat wakes on a fixed 1 ms schedule, records how late each wakeup was, then busy-loops for 200 micros (wall clock).
Quoted 8 lines
+-----------+-------+-----+-----+------+------+ > kernel | count | p50 | p90 | p99 | max | +-----------+-------+-----+-----+------+------+ > 7.2.9 | 14998 | 11 | 11 | 12 | 1869 | > 7.3.0-rc6 | 13853 | 11 | 12 | 2029 | 2116 | +-----------+-------+-----+-----+------+------+count is out of 15000 periods. rc6 had 1147 wakeups delayed by over 1 ms.
Thanks a bunch for the proggy. I've been looking for a clear signal with various loads, and stuffing wakelat+awk into one of my scripts provided a perfectly clear signal, which I bisected and confirmed.
Note: below is box pointing it's little silicon finger at the source of aforementioned jitter, not necessarily at breakage.
Reverting the fingered commit removed the jitter from 7.2.11, and force reverting
68e37487810a+333238da9a19removed it from master as well.
333238da9a193ffc58792995f3e951e4cb87bfd2is the first 'bad' commit commit333238da9a193ffc58792995f3e951e4cb87bfd2Author: Vincent Guittot <[email protected]> Date: Fri Aug 14 15:52:41 2026 +0200sched: Update time before requeueing delayed entities
In order to compute the right lag, it is required to update time to 'now'. Without this, the delayed entity might appear younger than it really is and receive less compensation for having waited.
Signed-off-by Vincent Guittot <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>kernel/sched/fair.c | 2 ++ 1 file changed, 2 insertions(+)
Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Hi Vincent,
Your patch fixes my cpu.idle test case.
Quoted 6 lines · Vincent Guittot
/*
- * Preempt the current task with a newly woken task if needed:
+ * Preempt an idle entity in favor of a non-idle entity (and don't preempt
+ * in the inverse case).
+ */
+static boolwakeup_preempt_sched_idle(struct sched_entity *cse, struct sched_entity *pse)
This should return enum preempt_wakeup_action. But with that change, on top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19 spinner in the same cgroup. I think it's incomplete.
Vincent Guittot
maintainerRe: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 6 lines · Jakub Okoński
Hi Vincent,
Your patch fixes my cpu.idle test case.
On 10/19/26 19:01, Vincent Guittot wrote:
Quoted 8 lines
Quoted 6 lines
/*
- * Preempt the current task with a newly woken task if needed:
+ * Preempt an idle entity in favor of a non-idle entity (and don't preempt
+ * in the inverse case).
+ */
+static boolwakeup_preempt_sched_idle(struct sched_entity *cse, struct sched_entity *pse)This should return enum
preempt_wakeup_action. But with that change, on
Argh! Yes I moved from a binary to a ternary value but forgot to update the prototype
Quoted 2 lines
top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19 spinner in the same cgroup. I think it's incomplete.
This patch only aims to fix the sched_idle task group case. I will have a closer look at the nice 19 case. Have you checked the different cgroup_mode with nice 19 ?
Vincent
Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 10 lines · Vincent Guittot
Quoted 3 lines · Jakub Okoński
top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19 spinner in the same cgroup. I think it's incomplete.
This patch only aims to fix the
sched_idletask group case. I will have a closer look at the nice 19 case. Have you checked the differentcgroup_modewith nice 19 ?Vincent
Same result in my nice 19 test for all of up/smp/concur/max/tasks.
Vincent Guittot
maintainerRe: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 7 lines · Jakub Okoński
Quoted 5 lines · Vincent Guittot
Quoted 3 lines · Jakub Okoński
top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19 spinner in the same cgroup. I think it's incomplete.
Quoted 7 lines
Quoted 5 lines
This patch only aims to fix the
sched_idletask group case. I will have a closer look at the nice 19 case. Have you checked the differentcgroup_modewith nice 19 ?Vincent
Same result in my nice 19 test for all of up/smp/concur/max/tasks.
Okay
Thanks for the tests results
Vincent Guittot
maintainerRe: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 13 lines · Vincent Guittot
Quoted 11 lines · Jakub Okoński
Quoted 9 lines · Vincent Guittot
Quoted 3 lines · Jakub Okoński
top of 7.3.0, it still regresses from 7.2.9 if you test with a nice 19 spinner in the same cgroup. I think it's incomplete.
This patch only aims to fix the
sched_idletask group case. I will have a closer look at the nice 19 case. Have you checked the differentcgroup_modewith nice 19 ?
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Vincent
Same result in my nice 19 test for all of up/smp/concur/max/tasks.
Okay
Thanks for the tests results
I ran more tests on 7.2.9 and 7.3. I confirm that there is a difference between 7.2.9 and 7.3. With flat hierarchy, we change the weight used to compute vruntime, deadline and vlag. The positive vlag elapses differently with the flat hierarchy and slice protection becomes more active. I used rt-app for the short task that wakes up periodically in my test
- v7.2.9
Metric | Value (us)
-----------------------------------
Total Samples | 16983 Total outlier | 17 Average | 54 Median (P50) | 53 90th Percentile | 53 99th Percentile | 54 99.9th Percentile | 1582 Maximum | 2248
-----------------------------------
- v7.3
Metric | Value (us)
-----------------------------------
Total Samples | 16368 Total outlier | 632 Average | 145 Median (P50) | 53 90th Percentile | 53 99th Percentile | 2039 99.9th Percentile | 2132 Maximum | 3367
-----------------------------------
That being said, there is a trend to make the lag to elapse quicker and an app can't expect to maintain its lag forever while sleeping. Instead, the app should set its slice to notify the scheduler that it would like to run 1st as long as fairness is not broken. If I set a slice of 1ms for the short task I have :
- v7.2.9
Metric | Value (us)
-----------------------------------
Total Samples | 17000 Total outlier | 16 Average | 52 Median (P50) | 53 90th Percentile | 53 99th Percentile | 55 99.9th Percentile | 55 Maximum | 56
-----------------------------------
- v7.3
Metric | Value (us)
-----------------------------------
Total Samples | 17000 Total outlier | 11 Average | 52 Median (P50) | 53 90th Percentile | 53 99th Percentile | 54 99.9th Percentile | 55 Maximum | 57
-----------------------------------
Which is the correct way to ask scheduler to make a task to run 1st
Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
Quoted 1 line · Vincent Guittot
Which is the correct way to ask scheduler to make a task to run 1st
I totally get that and will try to make the changes in my workloads, starting with https://invent.kde.org/plasma/kwin/-/merge_requests/10192
That said, I hope both the cpu.idle and nice 19 issues get resolved and I'll be watching those conversations. It's too disruptive otherwise!
Regards,
Jakub
Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
In reply to Mike GalbraithHi Vincent, Mike, Peter,
Sorry, my earlier report was sent without the reply headers. It contains additional will-it-scale pipe-handoff results for 333238da9a19:
https://lists.openwall.net/linux-kernel/2026/10/30/1506
I'm linking it here to keep it connected to Mike's report in this thread. The measurements and reproduction instructions are unchanged.
Thanks,
Chengfeng
Re: [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
In reply to Jakub OkońskiQuoted 21 lines · Jakub Okoński
Hi Peter,
Quoted 11 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
[...]
kernel/sched/fair.c +0
-0@@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r if (!sched_feat(WAKEUP_PREEMPTION)) return;kernel/sched/fair.c +0
-1-find_matching_se (&se, &pse);Quoted 3 lines
WARN_ON_ONCE(!pse);
cse_is_idle=se_is_idle(se);I think this changes how cpu.idle affects wakeup preemption.
Quoted 4 lines
With the single runqueue,
find_matching_se()is gone andse_is_idle()reduces totask_has_idle_policy(). Wouldn't this mean thatSCHED_NORMALtasks in a cpu.idle cgroup are no longer treated as idle forwakeup_preempt_fair()?
Am I perhaps testing wrong?
What I do to produce the perf sched data below is stuff pinned hogs in one cgroup, pinned wakelat in another, diddle the hog populated cgroup cpu.idle... and watch wakelat blast straight through the hog between it and the CPU just fine, regardless of cpu.idle state.
WRT the data, ignore perf's max below, take awk processed wakelat output as gospel. I can make the perf numbers match nearly perfects by running all players in the background and terminating via a time event but didn't bother here. The important bits are accumulated runtime, ctx switch count, and avg delay for the bottom two lines of each report. Ignore report format, that's a local hacklet, and also ignore the wakelat runtime delta between tested kernels when cpu.idle=0 in the hog group, that's a DELAY_DEQUEUE issue I've verified to be the source of the jitter I can reproduce.
BEGIN 7.3.0.g14c89774-master
7.3.0.g14c89774-master cgroupA-hogs-(cpu.idle=0)-vs-groupB-wakelat wakeup latency >1ms: 4568 max: 2.4ms stddev: 554.882
----------------------------------------------------------------------------------------------------------
Task | Runtime | Count | Avg delay | Max delay | Sum delay |
----------------------------------------------------------------------------------------------------------
cpuhog:1267 | 60.048 s | 40 | 6.186 us | 43.523 us | 247.445 us | cpuhog:1269 | 60.047 s | 102 | 8.867 us | 144.492 us | 904.512 us | cpuhog:1268 | 60.047 s | 126 | 10.899 us | 774.840 us | 1.373 ms | cpuhog:1266 | 60.045 s | 58 | 19.230 us | 531.945 us | 1.115 ms | cpuhog:1270 | 60.044 s | 35 | 4.708 us | 32.297 us | 164.798 us | cpuhog:1264 | 60.033 s | 492 | 27.200 us | 2.383 ms | 13.383 ms | cpuhog:1265 | 60.030 s | 783 | 23.775 us | 83.871 us | 18.617 ms | cpuhog:1271 | 49.519 s | 51947 | 202.691 us | 5.607 ms | 10.529 s | wakelat:1277 | 10.512 s | 52011 | 171.849 us | 10.416 ms | 8.938 s |
----------------------------------------------------------------------------------------------------------
TOTAL: | 480.376 s | 107581 | | 10.416 ms | 19.627 s |
----------------------------------------------------------------------------------------------------------
7.3.0.g14c89774-master cgroupA-hogs-(cpu.idle=1)-vs-groupB-wakelat wakeup latency >1ms: 0 max: 0.0ms stddev: 0.500
----------------------------------------------------------------------------------------------------------
Task | Runtime | Count | Avg delay | Max delay | Sum delay |
----------------------------------------------------------------------------------------------------------
cpuhog:1297 | 60.026 s | 43 | 9.577 us | 83.461 us | 411.831 us | cpuhog:1298 | 60.024 s | 27 | 4.094 us | 31.306 us | 110.547 us | cpuhog:1294 | 60.023 s | 60 | 10.620 us | 52.234 us | 637.217 us | cpuhog:1296 | 60.022 s | 131 | 15.621 us | 908.988 us | 2.046 ms | cpuhog:1293 | 60.019 s | 224 | 6.861 us | 593.440 us | 1.537 ms | cpuhog:1292 | 60.006 s | 690 | 28.075 us | 286.517 us | 19.372 ms | cpuhog:1295 | 60.005 s | 26 | 4.119 us | 32.429 us | 107.111 us | cpuhog:1299 | 47.903 s | 60216 | 201.386 us | 5.270 ms | 12.127 s | wakelat:1305 | 12.124 s | 60023 | 1.963 us | 2.156 ms | 117.862 ms |
----------------------------------------------------------------------------------------------------------
TOTAL: | 480.196 s | 122918 | | 5.270 ms | 12.394 s |
----------------------------------------------------------------------------------------------------------
BEGIN 7.2.11-ag-flat+hack
Note: this tree has virgin flat applied, and the bisected commit reverted.
7.2.11-ag-flat+hack cgroupA-hogs-(cpu.idle=0)-vs-groupB-wakelat wakeup latency >1ms: 0 max: 0.0ms stddev: 0.438
----------------------------------------------------------------------------------------------------------
Task | Runtime | Count | Avg delay | Max delay | Sum delay |
----------------------------------------------------------------------------------------------------------
cpuhog:1272 | 60.040 s | 44 | 6.139 us | 40.163 us | 270.136 us | cpuhog:1275 | 60.039 s | 139 | 3.768 us | 185.461 us | 523.864 us | cpuhog:1276 | 60.039 s | 140 | 5.432 us | 28.697 us | 760.493 us | cpuhog:1274 | 60.039 s | 104 | 12.441 us | 814.146 us | 1.294 ms | cpuhog:1270 | 60.037 s | 271 | 13.571 us | 637.071 us | 3.678 ms | cpuhog:1273 | 60.036 s | 51 | 17.658 us | 439.987 us | 900.594 us | cpuhog:1271 | 60.018 s | 636 | 28.960 us | 362.859 us | 18.419 ms | cpuhog:1277 | 47.902 s | 60081 | 202.120 us | 9.812 ms | 12.144 s | wakelat:1283 | 12.122 s | 60016 | 1.905 us | 2.801 ms | 114.368 ms |
----------------------------------------------------------------------------------------------------------
TOTAL: | 480.316 s | 123011 | | 9.812 ms | 12.352 s |
----------------------------------------------------------------------------------------------------------
7.2.11-ag-flat+hack cgroupA-hogs-(cpu.idle=1)-vs-groupB-wakelat wakeup latency >1ms: 0 max: 0.0ms stddev: 0.455
----------------------------------------------------------------------------------------------------------
Task | Runtime | Count | Avg delay | Max delay | Sum delay |
----------------------------------------------------------------------------------------------------------
cpuhog:1298 | 60.021 s | 307 | 8.860 us | 272.423 us | 2.720 ms | cpuhog:1301 | 60.020 s | 20 | 3.661 us | 30.994 us | 73.236 us | cpuhog:1303 | 60.020 s | 135 | 2.070 us | 28.715 us | 279.571 us | cpuhog:1302 | 60.019 s | 80 | 14.390 us | 782.611 us | 1.151 ms | cpuhog:1300 | 60.017 s | 46 | 42.518 us | 1.656 ms | 1.956 ms | cpuhog:1299 | 60.005 s | 22 | 6.165 us | 62.913 us | 135.646 us | cpuhog:1304 | 60.001 s | 666 | 27.965 us | 92.954 us | 18.625 ms | cpuhog:1305 | 47.900 s | 60084 | 201.849 us | 4.224 ms | 12.128 s | wakelat:1311 | 12.124 s | 60009 | 1.977 us | 1.440 ms | 118.685 ms |
----------------------------------------------------------------------------------------------------------
TOTAL: | 480.165 s | 122809 | | 4.224 ms | 12.364 s |
----------------------------------------------------------------------------------------------------------
[REGRESSION] core scheduling latency with idle load
In reply to Peter ZijlstraHi Peter,
I was running some experiments that co-locate a latency-sensitive service with SCHED_IDLE batch work in a cpu.idle=1 cgroup, with the service in its own core-scheduling cookie. After updating to 7.3 its tail latency got much worse. With the help of an LLM I bisected it to 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") and reduced it to the script below. It is still present in v7.4-rc3.
Reproducer (root, SMT machine, CONFIG_SCHED_CORE, cyclictest):
#!/bin/sh
set -e
mkdir -p /sys/fs/cgroup/be
echo 1 > /sys/fs/cgroup/be/cpu.idle
trap 'echo 1 > /sys/fs/cgroup/be/cgroup.kill; sleep 1; rmdir /sys/fs/cgroup/be' EXIT
# one SCHED_IDLE spinner per CPU in the cpu.idle=1 cgroup
chrt -i 0 sh -c 'echo $$ > /sys/fs/cgroup/be/cgroup.procs; for i in $(seq $(nproc)); do while :; do :; done & done'
sleep 1
# p99 from cyclictest's 1 us histogram, overflows (> 1 s) included
p99() { awk '/^[0-9]/ { h[$1+0] = $2; n += $2 } /Overflows/ { o = $4 } END { n += o; for (i = 0; c < n * 0.99; i++) c += h[i]; print "p99", i - 1, "us" }'; }
printf 'no cookie: '; cyclictest -q -m -i 1000 -D 10 -h 1000000 | p99
printf 'cookie: '; coresched new -- cyclictest -q -m -i 1000 -D 10 -h 1000000 | p99i7-1185G7 (4C/8T), vanilla kernels built with the same config (Gentoo's distribution kernel config, HZ=300, PREEMPT_LAZY). p99 in us:
kernel no cookie cookie 85570f10a4c6^ 54 59 85570f10a4c6 53 3695 v7.3.4 54 3276 v7.4-rc3 53 3290
I also tried the two cpu.idle preemption fixes under discussion [1][2] on v7.3.4; the cookie p99 stayed around 3.1 ms with both.
Changing cgroup_mode (up, smp, concur, max, tasks) on v7.4-rc3 did not fix it.
Quoted 1 line · Peter Zijlstra
(quoted diff) +0
-0@@ -14893,30 +14803,8 @@ bool cfs_prio_less(const struct task_str
[...]
kernel/sched/fair.c bool cfs_prio_less(const struct task_str · line 14912
Quoted 2 lines
kernel/sched/fair.c +0
-2-se_fi_update (sea, rq->core->core_forceidle_seq, in_fi); -se_fi_update (seb, rq->core->core_forceidle_seq, in_fi);
In case it helps: adding these two calls back on v7.3.4 brought the cookie p99 back to 55 us. I don't know whether that is the right thing to do, only that it changes this result.
Thanks,
Mihail
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
#regzbot introduced: 85570f10a4c6
K Prateek Nayak
reviewerRe: [REGRESSION] core scheduling latency with idle load
Hello Mihail,
Quoted 8 lines · Mihail Mihov
Quoted 1 line · Peter Zijlstra
(quoted diff) +0
-0@@ -14893,30 +14803,8 @@ bool cfs_prio_less(const struct task_str[...]
Review
Quoted 2 lines
(quoted diff) +0
-2- se_fi_update(sea, rq->core->core_forceidle_seq, in_fi); - se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);In case it helps: adding these two calls back on v7.3.4 brought the cookie p99 back to 55 us.
I can confirm I see skipped cfs_rq->zero_vruntime_fi updates for root from cfs_prio_less() which would have previously bumped up the zero_vruntime_fi at common ancestor.
Quoted 2 lines
I don't know whether that is the right thing
to do, only that it changes this result.
It might be since cfs_prio_less() would have compared the zero_vruntime_fi at common ancestor but now, everything is driven at root cfs_rq.
That said, perhaps se_fi_update() could itself be optimized now with flattened pick. Could you try below changes a try:
(Prepared on top of v7.4-rc3; Lightly tested)
kernel/sched/fair.c +9 -15
@@ -15278,30 +15278,21 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr)/* * se_fi_update - Update the cfs_rq->zero_vruntime_fi in a CFS hierarchy if needed. */ -static void se_fi_update (const struct sched_entity *se,unsigned int fi_seq, -bool forceidle) +static void cfs_rq_fi_update (struct cfs_rq *cfs_rq,unsigned int fi_seq,bool forceidle) { -for_each_sched_entity (se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); - -if (forceidle) { -if (cfs_rq->forceidle_seq == fi_seq) -break ; - cfs_rq->forceidle_seq = fi_seq; - } +if (forceidle && cfs_rq->forceidle_seq == fi_seq) +return ; - cfs_rq->zero_vruntime_fi = cfs_rq->zero_vruntime; - } + cfs_rq->forceidle_seq = fi_seq; + cfs_rq->zero_vruntime_fi = cfs_rq->zero_vruntime; }void task_vruntime_update (struct rq *rq,struct task_struct *p,bool in_fi) { -struct sched_entity *se = &p->se; -if (p->sched_class != &fair_sched_class)return ; -se_fi_update (se, rq->core->core_forceidle_seq, in_fi); +cfs_rq_fi_update (&rq->cfs, rq->core->core_forceidle_seq, in_fi); } bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b,
@@ -15319,6 +15310,9 @@ bool cfs_prio_less(const struct task_struct *a, const struct task_struct *b, cfs_rqa = &task_rq (a)->cfs; cfs_rqb = &task_rq (b)->cfs; +cfs_rq_fi_update (cfs_rqa, rq->core->core_forceidle_seq, in_fi); +cfs_rq_fi_update (cfs_rqb, rq->core->core_forceidle_seq, in_fi); +/* * Find delta after normalizing se's vruntime with its cfs_rq's * zero_vruntime_fi, which would have been updated in prior calls
Signature · 2 lines
Thanks and Regards, Prateek
Mihail Mihov
Tested-byRe: [REGRESSION] core scheduling latency with idle load
Hi Prateek,
Quoted 2 lines · K Prateek Nayak
That said, perhaps
se_fi_update()could itself be optimized now with flattened pick. Could you try below changes a try:
With this on top of v7.4-rc3 the reproducer gives:
kernel no cookie cookie v7.4-rc3 53 3290 v7.4-rc3 + patch 53 54
(p99 in us, same machine and config as in the report.)
Thanks,
Mihail
K Prateek Nayak
reviewerHello Peter,
Quoted 3 lines · Peter Zijlstra
Can also be had:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.gitsched/flat
Here are early performance numbers at commit 7eb85c33dd20 ("sched/eevdf:
Move to a single runqueue").
tl;dr
Apart form a regression in the tbench and schbench at super high utilization, all numbers for concur mode looks good so 7.2-rc1 target should be good.
P.S. I won't be able to look into this until Thursday; Sorry in advance.
Note: I haven't gotten around to analyzing anything in depth; These are just early numbers on microbenchmarks and DeathStarBench. Please take them with a grain of salt.
tip was at commit f666241e6bd5 ("sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime()") and each label are the individual cgroup mode togged on at queue:sched/flat:
Machine:
o 4th Generation EPYC system (Zen4c)
o 2 x 128C/256T (32LLCs)
o Boost enabled
o C2 disabled; MWAIT based C1 and POLL remained enabled
Benchmark numbers:
==================================================================
Test : hackbench
Units : Normalized time in seconds
Interpretation: Lower is better
Statistic : AMean
==================================================================
Case: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1-groups 1.00 [ -0.00](11.18) 1.01 [ -0.60](12.84) 1.12 [-12.08](12.50) 1.16 [-15.71]( 9.46) 1.07 [ -7.25](11.46) 0.99 [ 1.21](15.44) 2-groups 1.00 [ -0.00]( 4.73) 1.01 [ -1.00]( 9.29) 2.88 [-188.25](63.11) 0.97 [ 2.75](11.62) 0.95 [ 5.00](12.47) 0.96 [ 4.00](11.72) 4-groups 1.00 [ -0.00]( 2.90) 0.99 [ 0.96]( 2.48) 0.99 [ 0.72]( 2.38) 1.00 [ 0.48]( 1.78) 0.99 [ 0.96]( 6.61) 1.00 [ -0.00]( 5.23) 8-groups 1.00 [ -0.00]( 1.82) 1.03 [ -2.51]( 2.81) 0.99 [ 0.91]( 3.33) 0.99 [ 0.91]( 2.40) 1.01 [ -0.68]( 2.44) 1.02 [ -1.60]( 2.45) 16-groups 1.00 [ -0.00]( 3.05) 1.03 [ -2.96]( 1.97) 1.23 [-22.82](22.67) 1.01 [ -1.31]( 2.03) 0.99 [ 0.99]( 2.48) 1.01 [ -0.66]( 2.76)
Note: For smp variant runs, I think there was some system noise form an unrelated job that stated by mistake. I have to go back and rerun to confirm if the regression holds.
==================================================================
Test : tbench
Units : Normalized throughput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ 0.00]( 0.23) 1.02 [ 1.53]( 0.10) 1.02 [ 1.80]( 0.96) 1.00 [ 0.19]( 0.17) 0.99 [ -0.58]( 0.18) 1.01 [ 1.13]( 0.18)
2 1.00 [ 0.00]( 0.11) 1.01 [ 1.13]( 0.04) 1.01 [ 1.26]( 0.08) 1.00 [ -0.12]( 0.21) 0.99 [ -1.00]( 0.29) 1.01 [ 0.51]( 0.02)
4 1.00 [ 0.00]( 0.11) 1.01 [ 1.30]( 0.16) 1.02 [ 2.26]( 0.48) 1.00 [ 0.06]( 0.37) 0.99 [ -0.74]( 0.52) 1.01 [ 1.35]( 0.35)
8 1.00 [ 0.00]( 0.24) 1.01 [ 1.19]( 0.81) 1.02 [ 2.17]( 0.52) 1.00 [ 0.45]( 0.36) 0.99 [ -0.80]( 0.05) 1.01 [ 0.92]( 0.39)
16 1.00 [ 0.00]( 0.15) 1.01 [ 0.87]( 0.70) 1.03 [ 2.56]( 0.18) 1.00 [ -0.41]( 0.46) 0.99 [ -0.99]( 0.36) 1.01 [ 0.55]( 1.02)
32 1.00 [ 0.00]( 1.02) 1.02 [ 2.42]( 0.36) 1.04 [ 3.76]( 0.94) 1.01 [ 1.20]( 0.19) 1.00 [ -0.10]( 0.44) 1.02 [ 1.61]( 0.27)
64 1.00 [ 0.00]( 0.36) 1.02 [ 1.92]( 1.71) 1.03 [ 2.59]( 1.15) 0.99 [ -0.51]( 0.88) 1.01 [ 1.19]( 0.29) 1.02 [ 2.42]( 0.57)
128 1.00 [ 0.00]( 0.45) 1.01 [ 1.11]( 1.37) 1.05 [ 4.64]( 1.05) 1.01 [ 0.98]( 2.47) 1.01 [ 0.84]( 1.97) 1.03 [ 2.56]( 1.22)
256 1.00 [ 0.00]( 0.06) 1.02 [ 2.23]( 1.11) 1.02 [ 2.17]( 0.69) 1.03 [ 2.87]( 0.46) 1.03 [ 2.57]( 0.41) 1.03 [ 2.99]( 0.84)
512 1.00 [ 0.00]( 1.50) 0.92 [ -7.62]( 6.42) 1.02 [ 1.94]( 2.12) 1.01 [ 0.74]( 6.70) 0.94 [ -6.09]( 5.19) 0.98 [ -2.40]( 3.00)
1024 1.00 [ 0.00]( 0.07) 0.98 [ -1.51]( 0.30) 1.02 [ 1.66]( 0.13) 0.97 [ -2.97]( 0.76) 0.94 [ -6.33]( 0.26) 0.95 [ -4.63]( 0.47)
2048 1.00 [ 0.00]( 0.25) 0.98 [ -1.57]( 0.59) 1.02 [ 1.81]( 0.20) 0.98 [ -1.94]( 0.38) 0.94 [ -5.54]( 0.17) 0.95 [ -4.56]( 0.77) ==================================================================
Test : stream-10
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
Copy 1.00 [ 0.00]( 0.51) 0.92 [ -7.61](12.64) 0.99 [ -0.79]( 0.30) 0.85 [-15.33](21.73) 0.99 [ -0.61]( 0.35) 0.99 [ -0.90]( 0.38)
Scale 1.00 [ 0.00]( 0.35) 0.90 [ -9.96](14.86) 0.99 [ -1.45]( 0.76) 0.85 [-14.73](21.14) 0.99 [ -1.12]( 0.86) 0.99 [ -1.13]( 0.71)
Add 1.00 [ 0.00]( 0.21) 0.93 [ -6.95](10.26) 0.98 [ -1.51]( 0.80) 0.95 [ -4.91]( 9.84) 0.99 [ -0.98]( 0.69) 0.99 [ -1.37]( 0.67)
Triad 1.00 [ 0.00]( 0.24) 0.89 [-10.61](15.55) 0.99 [ -1.46]( 0.72) 0.95 [ -4.87]( 9.50) 0.99 [ -0.90]( 0.62) 0.99 [ -1.44]( 0.61) ==================================================================
Test : stream-100
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
Copy 1.00 [ 0.00]( 1.52) 1.00 [ -0.16]( 0.56) 0.97 [ -2.54]( 2.30) 1.00 [ 0.09]( 0.18) 1.00 [ 0.17]( 0.44) 0.99 [ -0.74]( 1.49)
Scale 1.00 [ 0.00]( 1.43) 0.99 [ -0.63]( 0.55) 0.97 [ -2.72]( 2.44) 1.00 [ 0.20]( 0.16) 0.99 [ -0.67]( 0.43) 0.99 [ -1.39]( 1.45)
Add 1.00 [ 0.00]( 1.06) 0.99 [ -1.27]( 0.42) 0.97 [ -3.08]( 1.95) 1.00 [ -0.13]( 0.19) 0.99 [ -1.19]( 0.34) 0.98 [ -1.76]( 1.07)
Triad 1.00 [ 0.00]( 1.10) 0.98 [ -1.55]( 0.41) 0.97 [ -3.40]( 1.95) 1.00 [ -0.42]( 0.16) 0.99 [ -1.49]( 0.35) 0.98 [ -2.08]( 1.11) ==================================================================
Test : netperf
Units : Normalized Througput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1-clients 1.00 [ 0.00]( 0.15) 1.02 [ 2.24]( 0.12) 1.02 [ 1.50]( 0.22) 1.07 [ 6.86]( 2.15) 1.00 [ 0.27]( 0.45) 1.02 [ 1.86]( 0.15)
2-clients 1.00 [ 0.00]( 0.37) 1.03 [ 2.50]( 0.80) 1.03 [ 3.11]( 0.78) 1.07 [ 7.15]( 0.96) 1.00 [ -0.43]( 0.54) 1.02 [ 1.67]( 0.69)
4-clients 1.00 [ 0.00]( 0.26) 1.02 [ 2.26]( 0.33) 1.02 [ 2.02]( 0.37) 1.07 [ 7.37]( 0.78) 1.00 [ -0.24]( 0.42) 1.02 [ 1.84]( 0.22)
8-clients 1.00 [ 0.00]( 0.21) 1.02 [ 2.47]( 0.55) 1.02 [ 2.13]( 0.48) 1.08 [ 7.60]( 0.57) 1.00 [ -0.15]( 0.37) 1.02 [ 1.87]( 0.29)
16-clients 1.00 [ 0.00]( 0.23) 1.02 [ 2.07]( 0.69) 1.02 [ 1.87]( 0.42) 1.07 [ 7.38]( 0.50) 0.99 [ -0.55]( 0.48) 1.02 [ 1.78]( 0.25)
32-clients 1.00 [ 0.00]( 0.47) 1.02 [ 2.14]( 0.63) 1.02 [ 1.81]( 0.75) 1.07 [ 7.44]( 0.94) 1.00 [ -0.38]( 0.53) 1.02 [ 1.76]( 0.43)
64-clients 1.00 [ 0.00]( 0.91) 1.02 [ 2.00]( 0.81) 1.02 [ 1.74]( 0.96) 1.07 [ 7.12]( 1.06) 1.00 [ -0.20]( 0.68) 1.02 [ 1.63]( 0.76)
128-clients 1.00 [ 0.00]( 1.19) 1.01 [ 1.36]( 1.28) 1.01 [ 1.34]( 1.18) 1.06 [ 6.37]( 1.47) 1.00 [ -0.49]( 1.15) 1.01 [ 1.06]( 1.09)
256-clients 1.00 [ 0.00]( 1.00) 1.02 [ 1.70]( 1.15) 1.02 [ 1.64]( 1.18) 1.07 [ 7.17]( 1.87) 1.00 [ -0.31]( 1.19) 1.01 [ 1.29]( 1.12)
512-clients 1.00 [ 0.00]( 5.16) 1.00 [ 0.02]( 6.48) 0.99 [ -0.60]( 4.31) 1.04 [ 4.08]( 3.83) 1.00 [ 0.02]( 2.52) 1.00 [ 0.48]( 2.86)
768-clients 1.00 [ 0.00](34.61) 1.03 [ 2.84](62.48) 1.00 [ 0.26](30.91) 0.98 [ -2.12](13.41) 0.94 [ -6.34](11.15) 0.95 [ -4.82](10.61)
1024-clients 1.00 [ 0.00](41.78) 1.04 [ 3.95](76.45) 1.01 [ 0.99](40.23) 0.97 [ -2.58](11.79) 0.95 [ -5.36](11.21) 0.96 [ -4.01](12.98) ==================================================================
Test : schbench
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ -0.00]( 5.88) 0.94 [ 5.88](19.25) 0.88 [ 11.76]( 7.37) 0.88 [ 11.76](34.02) 1.06 [ -5.88](11.11) 0.94 [ 5.88]( 3.69)
2 1.00 [ -0.00](36.56) 0.97 [ 3.03]( 9.52) 0.97 [ 3.03]( 4.82) 1.00 [ -0.00](26.93) 0.97 [ 3.03](14.53) 0.82 [ 18.18](13.87)
4 1.00 [ -0.00]( 9.35) 0.97 [ 3.12](18.25) 0.94 [ 6.25]( 9.12) 1.00 [ -0.00](20.82) 1.00 [ -0.00]( 4.82) 0.88 [ 12.50](15.93)
8 1.00 [ -0.00](23.38) 0.94 [ 6.45](23.26) 1.26 [-25.81](14.43) 1.26 [-25.81](26.13) 1.06 [ -6.45]( 4.68) 0.97 [ 3.23](10.00)
16 1.00 [ -0.00]( 2.71) 1.04 [ -3.57]( 2.62) 1.00 [ -0.00]( 2.04) 0.98 [ 1.79]( 2.76) 1.04 [ -3.57]( 0.00) 1.02 [ -1.79]( 2.70)
32 1.00 [ -0.00]( 0.72) 1.00 [ -0.00]( 0.72) 0.96 [ 3.75]( 1.30) 1.01 [ -1.25]( 0.71) 1.01 [ -1.25]( 1.88) 1.02 [ -2.50]( 1.86)
64 1.00 [ -0.00]( 1.52) 0.96 [ 4.41]( 1.18) 0.97 [ 2.94]( 4.49) 0.94 [ 5.88]( 1.19) 0.95 [ 5.15]( 0.45) 0.96 [ 4.41]( 0.45)
128 1.00 [ -0.00]( 3.24) 0.96 [ 3.80]( 0.91) 0.96 [ 3.80]( 1.53) 0.95 [ 4.64]( 0.26) 0.96 [ 3.80]( 0.44) 0.97 [ 2.53]( 1.32)
256 1.00 [ -0.00]( 0.90) 0.92 [ 7.60]( 0.75) 0.95 [ 5.40]( 0.61) 0.92 [ 7.80]( 0.55) 0.93 [ 6.60]( 6.23) 1.00 [ -0.00]( 0.61)
512 1.00 [ -0.00]( 1.76) 1.10 [-10.36]( 0.48) 1.04 [ -4.15]( 1.46) 0.97 [ 3.45]( 7.07) 1.07 [ -6.56]( 2.54) 0.93 [ 6.56]( 2.45)
768 1.00 [ -0.00]( 2.79) 0.95 [ 5.10]( 6.65) 1.71 [-70.87]( 4.67) 0.80 [ 20.13]( 1.81) 0.78 [ 21.80]( 1.19) 0.78 [ 22.36]( 1.46)
1024 1.00 [ -0.00]( 0.86) 0.35 [ 64.79](18.07) 1.16 [-15.56]( 1.17) 1.00 [ 0.32]( 3.65) 0.95 [ 4.54]( 3.10) 0.90 [ 9.72]( 2.33) ==================================================================
Test : new-schbench-requests-per-second
Units : Normalized Requests per second
Interpretation: Higher is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ 0.00]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ -0.30]( 0.00) 1.00 [ -0.30]( 0.31) 0.99 [ -0.59]( 0.15) 1.00 [ 0.00]( 0.00)
2 1.00 [ 0.00]( 0.00) 1.00 [ 0.30]( 0.00) 1.00 [ 0.00]( 0.61) 1.00 [ 0.30]( 0.15) 1.00 [ -0.30]( 0.70) 1.00 [ 0.00]( 0.61)
4 1.00 [ 0.00]( 0.00) 1.00 [ 0.29]( 0.30) 1.00 [ 0.00]( 0.15) 1.00 [ 0.29]( 0.00) 1.00 [ -0.29]( 0.15) 0.99 [ -0.59]( 0.31)
8 1.00 [ 0.00]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ -0.29]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ -0.29]( 0.00) 1.00 [ -0.29]( 0.15)
16 1.00 [ 0.00]( 0.00) 1.00 [ 0.29]( 0.15) 1.00 [ 0.00]( 0.15) 1.00 [ 0.00]( 0.15) 1.00 [ -0.29]( 0.00) 1.00 [ -0.29]( 0.15)
32 1.00 [ 0.00]( 0.15) 1.00 [ 0.29]( 0.15) 1.00 [ 0.29]( 0.15) 1.00 [ 0.29]( 0.00) 1.00 [ 0.00]( 0.15) 1.00 [ 0.29]( 0.15)
64 1.00 [ 0.00]( 0.15) 1.00 [ 0.29]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ 0.29]( 0.00) 1.00 [ 0.00]( 0.00) 1.00 [ 0.29]( 0.15)
128 1.00 [ 0.00](17.05) 1.01 [ 0.59](14.45) 1.00 [ 0.30]( 0.00) 1.01 [ 0.59](10.49) 1.00 [ 0.00]( 9.79) 1.00 [ 0.30]( 0.31)
256 1.00 [ 0.00]( 0.59) 1.00 [ 0.28]( 0.59) 1.00 [ 0.28]( 0.44) 1.00 [ -0.28]( 0.82) 0.99 [ -0.57]( 0.39) 1.00 [ 0.00]( 0.67)
512 1.00 [ 0.00]( 0.69) 1.01 [ 1.50]( 1.66) 1.00 [ 0.00]( 0.33) 0.99 [ -1.50]( 0.39) 0.99 [ -1.12]( 0.58) 1.00 [ 0.00]( 0.78)
768 1.00 [ 0.00]( 1.13) 1.12 [ 11.51]( 9.06) 0.99 [ -1.15]( 0.21) 0.91 [ -8.52]( 0.47) 0.91 [ -8.98]( 0.47) 0.93 [ -6.67]( 0.84)
1024 1.00 [ 0.00]( 1.23) 1.12 [ 12.41]( 4.32) 1.01 [ 1.10]( 0.71) 0.87 [-13.24]( 0.59) 0.87 [-12.97]( 0.49) 0.86 [-14.07]( 0.00) ==================================================================
Test : new-schbench-wakeup-latency
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ -0.00](21.48) 2.75 [-175.00](14.06) 3.00 [-200.00](12.91) 3.62 [-262.50](29.64) 3.62 [-262.50](21.84) 2.25 [-125.00](12.45)
2 1.00 [ -0.00]( 5.96) 1.33 [-33.33](24.78) 1.33 [-33.33](18.25) 1.11 [-11.11](25.37) 1.11 [-11.11]( 9.68) 1.11 [-11.11]( 5.00)
4 1.00 [ -0.00]( 6.20) 1.38 [-37.50]( 4.84) 1.12 [-12.50]( 0.00) 1.25 [-25.00](21.51) 1.12 [-12.50](19.99) 1.38 [-37.50](12.81)
8 1.00 [ -0.00]( 5.34) 1.20 [-20.00]( 0.00) 1.00 [ -0.00]( 5.00) 1.20 [-20.00]( 7.45) 1.20 [-20.00]( 0.00) 1.20 [-20.00]( 4.19)
16 1.00 [ -0.00]( 5.53) 1.22 [-22.22]( 0.00) 1.11 [-11.11]( 0.00) 1.11 [-11.11]( 9.68) 1.33 [-33.33]( 0.00) 1.33 [-33.33]( 4.43)
32 1.00 [ -0.00]( 5.53) 1.11 [-11.11]( 5.00) 1.00 [ -0.00]( 0.00) 1.11 [-11.11]( 5.00) 1.11 [-11.11]( 0.00) 1.11 [-11.11]( 0.00)
64 1.00 [ -0.00](12.81) 1.00 [ -0.00]( 0.00) 0.91 [ 9.09]( 5.34) 0.91 [ 9.09]( 5.00) 0.91 [ 9.09]( 5.00) 0.82 [ 18.18](10.68)
128 1.00 [ -0.00](12.14) 1.12 [-12.50]( 4.97) 0.81 [ 18.75](13.62) 1.06 [ -6.25]( 5.26) 0.94 [ 6.25](14.68) 0.81 [ 18.75]( 6.88)
256 1.00 [ -0.00]( 4.83) 0.98 [ 1.90]( 7.97) 0.98 [ 2.37]( 2.41) 0.95 [ 4.74]( 5.31) 0.95 [ 5.21]( 3.25) 0.97 [ 2.84]( 2.84)
512 1.00 [ -0.00]( 0.00) 0.98 [ 2.00]( 5.86) 0.96 [ 3.99]( 1.61) 0.83 [ 16.83]( 1.08) 0.85 [ 14.84]( 7.53) 0.97 [ 3.42]( 7.91)
768 1.00 [ -0.00]( 0.71) 0.96 [ 3.63](14.27) 0.96 [ 3.73]( 1.19) 0.67 [ 32.98]( 0.00) 0.67 [ 32.98]( 0.00) 0.67 [ 32.98]( 0.15)
1024 1.00 [ -0.00]( 2.55) 0.85 [ 14.99](14.23) 0.98 [ 1.80]( 0.82) 0.79 [ 21.29]( 0.00) 0.79 [ 21.29]( 0.00) 0.79 [ 21.29]( 0.00) ==================================================================
Test : new-schbench-request-latency
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) up[pct imp](CV) smp[pct imp](CV) max[pct imp](CV) concur[pct imp](CV) tasks[pct imp](CV)
1 1.00 [ -0.00]( 0.14) 1.00 [ -0.00]( 0.00) 1.00 [ -0.00]( 0.14) 1.00 [ -0.26]( 0.27) 1.01 [ -0.53]( 0.27) 1.00 [ -0.00]( 0.14)
2 1.00 [ -0.00]( 0.00) 1.00 [ 0.26]( 0.14) 1.02 [ -2.36]( 1.51) 0.99 [ 0.52]( 0.14) 1.00 [ -0.00]( 2.00) 1.00 [ -0.00]( 1.74)
4 1.00 [ -0.00]( 0.00) 1.00 [ 0.26]( 1.82) 1.03 [ -2.63]( 0.23) 0.99 [ 0.53]( 0.14) 1.00 [ -0.26]( 1.61) 1.03 [ -3.42]( 1.79)
8 1.00 [ -0.00]( 0.14) 0.99 [ 0.79]( 0.00) 1.00 [ -0.00]( 0.00) 0.99 [ 1.05]( 0.14) 1.00 [ -0.00]( 0.00) 1.00 [ -0.00]( 0.00)
16 1.00 [ -0.00]( 0.14) 1.00 [ -0.00]( 1.16) 1.00 [ -0.26]( 1.28) 0.99 [ 0.79]( 1.97) 1.00 [ -0.26]( 0.00) 1.02 [ -2.38]( 1.15)
32 1.00 [ -0.00]( 0.23) 0.99 [ 1.04]( 0.24) 1.00 [ -0.00]( 0.67) 0.99 [ 1.04]( 0.14) 0.99 [ 0.52]( 0.49) 0.99 [ 1.30]( 0.95)
64 1.00 [ -0.00](12.39) 1.72 [-71.69](26.18) 1.01 [ -1.29](28.90) 0.97 [ 3.09]( 0.27) 0.98 [ 2.06]( 0.41) 0.98 [ 2.32]( 1.22)
128 1.00 [ -0.00]( 4.75) 0.98 [ 2.07](12.01) 0.99 [ 0.59]( 0.41) 1.00 [ 0.30]( 3.02) 1.00 [ -0.00]( 2.72) 0.99 [ 0.59]( 0.77)
256 1.00 [ -0.00]( 0.13) 0.99 [ 0.76]( 0.13) 1.00 [ -0.00]( 0.13) 0.99 [ 0.76]( 0.00) 0.99 [ 0.51]( 0.13) 0.99 [ 0.51]( 0.13)
512 1.00 [ -0.00]( 7.49) 1.04 [ -4.36](28.15) 0.89 [ 10.50]( 4.15) 0.42 [ 57.93]( 9.03) 0.48 [ 52.23](25.18) 0.71 [ 28.83](25.82)
768 1.00 [ -0.00]( 3.13) 1.31 [-31.19](10.91) 1.10 [-10.13]( 1.12) 0.71 [ 28.69]( 3.69) 0.70 [ 29.59]( 3.58) 0.84 [ 15.65]( 4.74)
1024 1.00 [ -0.00]( 2.28) 1.44 [-44.48](16.05) 1.13 [-13.09]( 0.88) 0.85 [ 15.39]( 6.64) 0.85 [ 15.03]( 0.39) 0.81 [ 19.39]( 1.36)---
DeathStarBench was run on a 3rd Generation EPYC system (2 x 64C/128T) with boost enabled and C2 disabled:
==================================================================
Test : DeathStarBench
Units : %diff compared to tip
Interpretation: Higher is better
Statistic : Average throughput
==================================================================
Scaling\cg_mode up smp max concur tasks
2x 1.63% -0.32% -0.03% -0.65% 0.62%
4x -4.94% 1.38% -6.59% 1.09% -6.11%
6x 1.83% 0.16% -0.87% 2.24% -0.23%---
I'll get to look any deeper into any of the regressions until Thursday but overall the concur mode seems good in my testing for most part.
Signature · 2 lines
Thanks and Regards, Prateek
Hello Peter,
I applied the sched/flat patchset from your tree on top of the tip/sched/core base commit (9ebe5c3c29f62)/(7.1-rc2)
The evaluation was performed on an 80-core, Ampere Altra system running Fedora Linux 41.
Benchmark Runs:-
1. Hackbench (Execution time in seconds: lower is better) The data reveals a clear architectural pivot point at 4 tasks:
- Low Concurrency (< 4 tasks): Regresses by +1.8% to +4.0%. Removing cgroup isolation boundaries expands the idle CPU search adding slight overhead to the wake-up path.
* 1 Thread: (+1.8%)
* 2 Threads: (+4.0%)
* 2 Procs: (+3.3%)
- Tipping Point (4 tasks): Performance is completely flat.
* 4 Threads: (+0.03%)
* 4 Procs: (+0.1%)
- High Concurrency (>= 8 tasks): Improves by -0.7% to -2.3%. Collapsing the tree structure down to a flat layout removes multi-layer load tracking updates (update_load_avg), saving cycles under load.
* 8 Threads: (-0.7%)
* 16 Threads: (-1.8%)
* 8 Procs: (-1.2%)
* 16 Procs: (-2.3%)
* 32 Procs: (-1.6%)
2. Schbench (Wakeup Tail Latency)
- 16 Threads (128kb footprint): 99.9th percentile tail latency drops significantly by -12.21% (us). Operating on a unified runqueue layer prevents induced group-level throttling.
- 32 Threads (128kb footprint): 99.9th percentile tail latency regresses by +5.50% (us). Eliminating nested queues increases lock contention during heavy simultaneous wakeups.
3. Sysbench
- Sysbench RAM: Throughput increases by +1.55% (MiB/sec). Fewer tree traversals reduce cache-line bouncing, freeing up cycles.
The patchset trades minor low-load performance for better scaling and tighter tail latencies under distributed load. However, the majority of these deltas remain small and sit near the measurement noise floor (<= 4%).
Regards,
Shubhang Kaushik
Quoted 70 lines · Peter Zijlstra
Hi!
New version, same story [1]. TL;DR:
- Adds new
cgroup_modeknob and implements new policies to address the hierarchy level weight mismatch.- Builds upon that base to create a flat / single runqueue scheduler where the cgroup hierarchy is expressed through dynamic weight management.
I'm hoping to be able to merge these patches early in the next cycle (after 7.2-rc1).
Random benchmark:
Game vs 'for ((i=0; i<8; i++)) do nice ./spin.sh; done':
Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper) Intel Core i7-2600K
AMD Radeon RX 580Shadows Awakening (GOG)
default
slice(*)FPS min 4.0 29.0 avg 47.5 59.2 max 83.7 83.7FT min 9.3 10.2 avg 34.0 17.0 max 121.2 30.0FPS (Frames Per Second)
FT (FrameTime)[*] Command prefix: 'chrt -o --sched-runtime 100000 0'
Changes since v2:
- merged debug and prep patches
- fixedupdate_entity_lag()on dequeue (Vincent)
- fixed throttle vs tick (Prateek)
- fixedwakeup_preempt_fair()
- rebased on tip/sched/core
- rewrittencgroup_modechangelogs
- reworkedcgroup_modeconcur
- addedcgroup_modetasks
- changed defaultcgroup_mode[1] - https://lore.kernel.org/r/[email protected]
Can also be had:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.gitsched/flatinclude/linux/cpuset.h | 6 include/linux/sched.h | 1 kernel/cgroup/cpuset.c | 15 kernel/sched/core.c | 5 kernel/sched/debug.c | 89 ++++ kernel/sched/fair.c | 943 ++++++++++++++++++++++++------------------------- kernel/sched/pelt.c | 6 kernel/sched/sched.h | 30 - 8 files changed, 607 insertions(+), 488 deletions(-)
Hello Peter,
We ran the same benchmarks (schbench, sysbench, hackbench) as Shubhang has on s390 on an LPAR running fedora 43 with 32 vCPUs.
We ran the benchmarks for each of the cgroup modes, and for the baseline, we chose the commit prior to the patches (f666241e6bd5 - sched/fair: Unify cfs_rq throttling via account_cfs_rq_runtime() ).
We have also tried running stress-ng in parallel with the benchmarks (set to generate 50% or 90% utilization for each vCPU).
Compared to simply running the benchmarks on their own, this has revealed some performance trade-offs that the move to a single runqueue can introduce.
=======================================================================
HACKBENCH (via phoronix-test-suite pts/hackbench)
=======================================================================
The hackbench results were quite consistent, CV was under 2.5% in most cases, a '*' marks the cases where they weren't.
What we've found is that if there is no other workload running, the results were generally favorable, especially for higher number of threads/processes.
However, with stress-ng also running in parallel, while the high thread/process count cases showed even greater improvement, the lower- count cases actually started to regress, which got worse at higher CPU utilization.
To compound this problem, the stress-ng results also showed regression (the stress-ng figures were recorded over the whole hackbench stress-ng run for a specific mode, so currently there is no higher granularity data for the 32 process case for example).
(lower = better)
Hackbench % diff from baseline by mode:
[s-00] Hackbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+ | Hackbench arg | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | 1 thread | -0.40% | +0.81% | +0.40% | +0.37% | -2.82% | | 2 thread | -0.50% | -0.96% | -0.65% | -1.68% | -1.55% | | 4 thread | -3.67% | -3.95% | -4.27% | -3.67% | -5.38% | | 8 thread | -6.21% | -6.62% | -6.86% | -6.30% | -9.55% | | 16 thread | -8.90% | -9.09% | -9.21% | -8.58% | -8.39% | | 32 thread | -6.67% | -6.57% | -6.84% | -6.25% | -5.64% | | 1 process | +1.41% | +0.84% | +0.19% | +2.36% | -3.50% | | 2 process | +0.23% | +0.91% | +0.00% | +1.49% | -0.75% | | 4 process | -3.21% | -2.98% | -3.54% | -3.31% | -5.57% | | 8 process | -5.55% | -5.40% | -5.53% | -5.56% | -8.47% | | 16 process | -7.77% | -7.75% | -8.33% | -7.47% | -7.89% | | 32 process | -7.44% | -6.92% | -7.19% | -4.76% | -6.89% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Hackbench — % diff from baseline by mode
+------------------+---------+---------+---------+---------+----------+ | Hackbench arg | concur | max | smp | tasks | up | +------------------+---------+---------+---------+---------+----------+ | 1 thread | +5.73% | +0.98% | +0.65% | +0.77% | -0.46% | | 2 thread | +3.24% | -1.82% | -2.04% | -2.11% | -0.65% | | 4 thread | -2.51% | -4.50% | -4.49% | -4.18% | -8.75% | | 8 thread | -9.29% | -18.48% | -18.05% | -18.33% | -21.06% | | 16 thread | -14.77% | -35.28% | -34.42% | -35.24% | -28.20%* | | 32 thread | -5.79% | -39.03% | -38.58% | -39.16% | -27.75% | | 1 process | +6.35% | +1.20% | +1.01% | +0.87% | +0.34% | | 2 process | +3.91% | -1.87% | -1.90% | -2.26% | -0.54% | | 4 process | -1.71% | -3.54% | -3.95% | -3.69% | -7.42% | | 8 process | -8.96% | -17.83% | -17.35% | -17.82% | -21.43% | | 16 process | -14.86% | -34.80% | -33.98% | -35.37% | -30.54% | | 32 process | -6.72% | -38.85% | -38.81% | -39.50% | -33.24% | +------------------+---------+---------+---------+---------+----------+
[s-90] Hackbench — % diff from baseline by mode
+-----------------+----------+---------+---------+---------+----------+ | Hackbench arg | concur | max | smp | tasks | up | +-----------------+----------+---------+---------+---------+----------+ | 1 thread | +17.81% | +18.24% | +17.29% | +17.40% | +31.11% | | 2 thread | +5.81% | +5.43% | +6.34% | +5.59% | +11.62% | | 4 thread | -8.69% | -8.65% | -7.40% | -8.80% | -5.85% | | 8 thread | -22.75% | -22.69% | -21.47% | -22.88% | -23.04% | | 16 thread | -37.81% | -37.71% | -36.49% | -37.84% | -28.13% | | 32 thread | -39.75% | -39.97% | -39.45% | -40.06% | -32.54% | | 1 process | +23.63%* | +22.55% | +22.17% | +21.17% | +34.84% | | 2 process | +7.54% | +7.44% | +7.34% | +7.47% | +13.74% | | 4 process | -6.61% | -6.87% | -5.53% | -6.45% | -4.03% | | 8 process | -21.92% | -21.70% | -20.40% | -21.79% | -21.90% | | 16 process | -37.17% | -36.92% | -35.98% | -37.31% | -30.01%* | | 32 process | -39.20% | -38.32% | -39.03% | -39.19% | -25.16%* | +-----------------+----------+---------+---------+---------+----------+
stress-ng bogo-ops/s statistics by stress level (Hackbench):
(higher = better)
[s-50] Hackbench — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | concur | -3.35% | ±0.15% | 0.15% | | max | -29.48% | ±0.75% | 0.75% | | smp | -29.57% | ±0.68% | 0.68% | | tasks | -30.01% | ±0.42% | 0.42% | | up | -24.68% | ±2.27% | 2.26% | +----------+------------+---------+-------+
[s-90] Hackbench — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | concur | -13.55% | ±0.23% | 0.23% | | max | -20.19% | ±0.44% | 0.44% | | smp | -19.72% | ±0.57% | 0.57% | | tasks | -20.61% | ±0.33% | 0.33% | | up | -25.52%* | ±5.86% | 5.86% | +----------+------------+---------+-------+
=======================================================================
SYSBENCH (via phoronix-test-suite ciunas/sysbench) - throughput
=======================================================================
The sysbench results showed small improvements, but more interesting were the stress-ng results, showing regressions across all modes at 50% CPU utilization, but improvements at 90%.
RAM/Memory testcase
(higher = better)
Sysbench % diff from baseline by mode:
[s-00] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+ | | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | memory run | -3.91% | +0.02% | -4.05% | +0.44% | -4.11% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+ | | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | memory run | +2.84% | +1.09% | +0.73% | +2.62% | +1.91% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+ | | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | memory run | +1.83% | +4.03% | +2.64% | +1.16% | -12.52% | +-------------------+---------+---------+---------+---------+---------+
stress-ng bogo-ops/s statistics by stress level (Sysbench):
(higher = better)
[s-50] Sysbench — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 588.34 | ±1.72% | 1.72% | +----------+------------+---------+-------+ | concur | -10.77% | ±2.52% | 2.52% | | max | -6.24% | ±2.21% | 2.21% | | smp | -6.16% | ±1.93% | 1.93% | | tasks | -8.51% | ±3.21% | 3.21% | | up | -10.88% | ±3.25% | 3.25% | +----------+------------+---------+-------+
[s-90] Sysbench — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 726.96 | ±2.00% | 2.00% | +----------+------------+---------+-------+ | concur | +4.22% | ±1.24% | 1.24% | | max | +4.00% | ±0.90% | 0.90% | | smp | +3.10% | ±1.73% | 1.73% | | tasks | +4.28% | ±1.83% | 1.83% | | up | +15.61% | ±5.10% | 5.10% | +----------+------------+---------+-------+
=======================================================================
SCHBENCH
=======================================================================
All testruns were done with footprint set to 128kb.
There is some variability between the different test environments (especially between the 32 threads-locking-yes case and the others), but there is a general pattern.
Without stress-ng running in parallel, over the different test cases, we did generally see a reduction in tail latency while other metrics remained largely unchanged.
At 50% cpu utilization, tail latency is still improved (or close to the noise floor), but average latency shows a more significant regression.
Then, at 90% cpu utilization, all latency components show regression, tail latency the most of all.
RPS is not impacted as strongly.
Throughout this the stress-ng benchmark results show improvement at 90% cpu utilization, while at 50% the changes are quite close to the noise floor.
=======================================================================
SCHBENCH 16T -- LOCKING: NO
=======================================================================
Schbench % diff from baseline by mode:
(Request latency percentiles: lower = better; RPS: higher = better)
[s-00] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +0.00% | +0.00% | +0.00% | +0.00% | +0.00% | | Req p90 (us) | -0.23% | +0.00% | +0.00% | +0.00% | -0.23% | | Req p99.9(us) | -7.23% | -5.16% | -3.10% | -5.85% | -5.51% | | RPS p50 (req) | +0.00% | +0.00% | +0.00% | +0.00% | +0.35% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +13.51% | +15.57% | +15.27% | +13.80% | +14.68% | | Req p90 (us) | -1.48% | +2.32% | +2.53% | -2.11% | +1.48% | | Req p99.9(us) | -8.99% | -7.53% | -5.35% | -8.75% | +4.86% | | RPS p50 (req) | +3.65% | +1.00% | +1.00% | +3.65% | +0.66% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +7.72% | +7.72% | +7.46% | +7.72% | +8.49% | | Req p90 (us) | +4.76% | +4.36% | +4.76% | +4.76% | +19.35% | | Req p99.9(us) | +34.35% | +34.35% | +34.35% | +34.35% | +38.10% | | RPS p50 (req) | -1.09% | +0.00% | +0.00% | -1.09% | -7.56% | +-------------------+---------+---------+---------+---------+---------+
stress-ng bogo-ops/s statistics by stress level (schbench):
[s-50] Schbench 16t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | concur | -2.17% | ±1.50% | 1.50% | | max | +1.08% | ±1.61% | 1.61% | | smp | +0.37% | ±1.63% | 1.63% | | tasks | -2.20% | ±1.00% | 1.00% | | up | +2.27% | ±1.56% | 1.56% | +----------+------------+---------+-------+
[s-90] Schbench 16t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | concur | +3.23% | ±1.05% | 1.05% | | max | -0.25% | ±1.12% | 1.12% | | smp | +0.93% | ±2.70% | 2.70% | | tasks | +1.33% | ±1.62% | 1.62% | | up | +8.09% | ±5.19% | 5.18% | +----------+------------+---------+-------+
=======================================================================
SCHBENCH 16T -- LOCKING: YES
=======================================================================
Schbench % diff from baseline by mode:
(Request latency percentiles: lower = better; RPS: higher = better)
[s-00] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +0.00% | +0.00% | +0.00% | +0.00% | +0.00% | | Req p90 (us) | +0.23% | +0.23% | +0.23% | +0.23% | +0.00% | | Req p99.9(us) | -12.86% | -15.25% | -2.39% | -11.66% | -8.67% | | RPS p50 (req) | +0.00% | +0.00% | +0.00% | +0.00% | +0.35% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +10.16% | +11.85% | +11.85% | +10.16% | +11.00% | | Req p90 (us) | -0.36% | +4.63% | +5.35% | -0.36% | +2.50% | | Req p99.9(us) | +0.58% | +0.87% | +4.05% | -1.45% | +2.03% | | RPS p50 (req) | +3.21% | -0.36% | +0.00% | +2.85% | +0.71% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +7.70% | +7.45% | +7.45% | +7.70% | +9.24% | | Req p90 (us) | +4.13% | +1.50% | +3.38% | +3.75% | +102.2… | | Req p99.9(us) | +18.21% | +13.94% | +16.22% | +13.66% | +306.8… | | RPS p50 (req) | -1.52% | -0.38% | -0.38% | -1.52% | -26.95% | +-------------------+---------+---------+---------+---------+---------+
stress-ng bogo-ops/s statistics by stress level (schbench):
[s-50] Schbench 16t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | concur | -1.29% | ±1.16% | 1.16% | | max | +1.26% | ±1.18% | 1.17% | | smp | +1.18% | ±1.15% | 1.15% | | tasks | +0.02% | ±2.05% | 2.04% | | up | +2.31% | ±2.56% | 2.56% | +----------+------------+---------+-------+
[s-90] Schbench 16t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | concur | +6.16% | ±1.47% | 1.47% | | max | +3.10% | ±1.89% | 1.89% | | smp | +3.91% | ±1.83% | 1.83% | | tasks | +5.68% | ±1.87% | 1.87% | | up | +16.60% | ±8.60% | 8.60% | +----------+------------+---------+-------+
=======================================================================
SCHBENCH 32T -- LOCKING: NO
=======================================================================
Schbench % diff from baseline by mode:
(Request latency percentiles: lower = better; RPS: higher = better)
[s-00] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +0.00% | +0.00% | +0.00% | +0.00% | +0.00% | | Req p90 (us) | +0.00% | +0.00% | -0.23% | +0.00% | -0.23% | | Req p99.9(us) | -4.80% | -3.77% | -6.17% | -7.55% | -10.98% | | RPS p50 (req) | +0.35% | +0.35% | +0.35% | +0.35% | +0.35% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +11.76% | +13.77% | +13.49% | +11.76% | +12.63% | | Req p90 (us) | -2.52% | +2.31% | +2.31% | -2.73% | -0.63% | | Req p99.9(us) | -8.99% | -6.80% | -4.13% | -9.96% | +2.43% | | RPS p50 (req) | +2.61% | -0.65% | -0.33% | +2.61% | +0.65% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +5.55% | +5.55% | +5.55% | +5.55% | +7.06% | | Req p90 (us) | +2.62% | +2.62% | +2.62% | +2.62% | +32.73% | | Req p99.9(us) | +31.57% | +31.11% | +31.57% | +31.57% | +34.33% | | RPS p50 (req) | +1.46% | +2.55% | +2.55% | +1.46% | -10.66% | +-------------------+---------+---------+---------+---------+---------+
stress-ng bogo-ops/s statistics by stress level (schbench):
[s-50] Schbench 32t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 478.27 | ±1.98% | 1.99% | +----------+------------+---------+-------+ | concur | -2.85% | ±1.45% | 1.45% | | max | +0.12% | ±1.95% | 1.95% | | smp | -0.23% | ±1.22% | 1.22% | | tasks | -2.39% | ±1.96% | 1.96% | | up | -0.27% | ±1.59% | 1.59% | +----------+------------+---------+-------+
[s-90] Schbench 32t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 577.45 | ±0.69% | 0.70% | +----------+------------+---------+-------+ | concur | +2.17% | ±1.54% | 1.54% | | max | +0.25% | ±1.28% | 1.28% | | smp | -0.33% | ±1.42% | 1.42% | | tasks | +1.07% | ±1.17% | 1.17% | | up | +10.70% | ±5.47% | 5.47% | +----------+------------+---------+-------+
=======================================================================
SCHBENCH 32T -- LOCKING: YES
=======================================================================
Schbench % diff from baseline by mode:
(Request latency percentiles: lower = better; RPS: higher = better)
[s-00] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +0.00% | +0.00% | +0.00% | +0.00% | +0.00% | | Req p90 (us) | +0.00% | +0.00% | +0.00% | +0.00% | -0.23% | | Req p99.9(us) | +7.31% | +10.05% | +13.70% | +10.35% | -16.74% | | RPS p50 (req) | +0.00% | +0.00% | +0.00% | +0.35% | +0.35% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +8.30% | +9.96% | +9.96% | +8.30% | +9.41% | | Req p90 (us) | -0.71% | +3.54% | +4.60% | -0.71% | +0.00% | | Req p99.9(us) | -4.05% | +3.76% | +3.47% | -2.89% | -4.92% | | RPS p50 (req) | +2.11% | -1.05% | -1.41% | +2.11% | +1.05% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +5.55% | +5.55% | +5.55% | +5.55% | +6.81% | | Req p90 (us) | -0.77% | -1.16% | -0.77% | -0.77% | +55.49% | | Req p99.9(us) | +11.00% | +11.28% | +14.67% | +19.18% | +92.67% | | RPS p50 (req) | +1.51% | +2.64% | +2.26% | +1.51% | -15.16% | +-------------------+---------+---------+---------+---------+---------+
stress-ng bogo-ops/s statistics by stress level (schbench):
[s-50] Schbench 32t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 506.91 | ±1.74% | 1.74% | +----------+------------+---------+-------+ | concur | -6.03% | ±1.28% | 1.28% | | max | -3.98% | ±1.68% | 1.68% | | smp | -4.23% | ±1.31% | 1.31% | | tasks | -5.75% | ±1.47% | 1.47% | | up | -3.84% | ±2.10% | 2.10% | +----------+------------+---------+-------+
[s-90] Schbench 32t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 555.39 | ±1.03% | 1.03% | +----------+------------+---------+-------+ | concur | +6.59% | ±1.38% | 1.38% | | max | +5.05% | ±1.25% | 1.25% | | smp | +7.28% | ±1.90% | 1.90% | | tasks | +7.07% | ±1.46% | 1.46% | | up | +18.88% | ±7.68% | 7.68% | +----------+------------+---------+-------+
Regards,
Signature · 13 lines
Szabina Korbai Linux on Z development Software Labs Campus Unlimited Company 25 North Wall Quay, Dublin 1, D01 H104, Ireland [email protected] IBM
K Prateek Nayak
reviewerHello Szabina,
Quoted 9 lines · Szabina Korbai
Hello Peter,
We ran the same benchmarks (schbench, sysbench, hackbench) as Shubhang has on s390 on an LPAR running fedora 43 with 32 vCPUs.
We ran the benchmarks for each of the cgroup modes, and for the baseline, we chose the commit prior to the patches (
f666241e6bd5- sched/fair: Unifycfs_rqthrottling viaaccount_cfs_rq_runtime()).
Quoted 6 lines
We have also tried running stress-ng in parallel with the benchmarks (set to generate 50% or 90% utilization for each vCPU).
Compared to simply running the benchmarks on their own, this has revealed some performance trade-offs that the move to a single runqueue can introduce.
Are you using tip:sched/core at commit 68e3748781 ("sched/fair: Fix flat hierarchy") for the flat_cg numbers or did you checkout at 85570f10a4c6 ("sched/eevdf: Move to a single runqueue")?
There are a couple fixes for vruntime update and Vincent's optimizations for preemption bits which might make a difference to the overall results.
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 28 lines · K Prateek Nayak
Hello Szabina,
Quoted 10 lines · Szabina Korbai
Hello Peter,
We ran the same benchmarks (schbench, sysbench, hackbench) as Shubhang
has on s390 on an LPAR running fedora 43 with 32 vCPUs.We ran the benchmarks for each of the cgroup modes, and for the baseline, we chose the commit prior to the patches (
f666241e6bd5- sched/fair: Unifycfs_rqthrottling viaaccount_cfs_rq_runtime()).Quoted 8 lines
We have also tried running stress-ng in parallel with the benchmarks
(set to generate 50% or 90% utilization for each vCPU).Compared to simply running the benchmarks on their own, this has revealed some performance trade-offs that the move to a single runqueue
can introduce.Are you using tip:sched/core at commit
68e3748781("sched/fair: Fix flat
hierarchy") for theflat_cgnumbers or did you checkout at85570f10a4c6
("sched/eevdf: Move to a single runqueue")?
Quoted 4 lines
There are a couple fixes for vruntime update and Vincent's optimizations
for preemption bits which might make a difference to the overall results.
Hi Prateek,
thank you, that's a good call. I did checkout at "Move to a single runqueue". Let me try it with the fix included, see how the results are affected.
Regards,
Signature · 13 lines
Szabina Korbai Linux on Z development Software Labs Campus Unlimited Company 25 North Wall Quay, Dublin 1, D01 H104, Ireland [email protected] IBM
Peter Zijlstra
maintainerQuoted 35 lines · Szabina Korbai
Quoted 28 lines · K Prateek Nayak
Hello Szabina,
Quoted 10 lines · Szabina Korbai
Hello Peter,
We ran the same benchmarks (schbench, sysbench, hackbench) as Shubhang
has on s390 on an LPAR running fedora 43 with 32 vCPUs.We ran the benchmarks for each of the cgroup modes, and for the baseline, we chose the commit prior to the patches (
f666241e6bd5- sched/fair: Unifycfs_rqthrottling viaaccount_cfs_rq_runtime()).Quoted 8 lines
We have also tried running stress-ng in parallel with the benchmarks
(set to generate 50% or 90% utilization for each vCPU).Compared to simply running the benchmarks on their own, this has revealed some performance trade-offs that the move to a single runqueue
can introduce.Are you using tip:sched/core at commit
68e3748781("sched/fair: Fix flat
hierarchy") for theflat_cgnumbers or did you checkout at85570f10a4c6
("sched/eevdf: Move to a single runqueue")?Quoted 4 lines
There are a couple fixes for vruntime update and Vincent's optimizations
for preemption bits which might make a difference to the overall results.
Quoted 5 lines
Hi Prateek,
thank you, that's a good call. I did checkout at "Move to a single runqueue". Let me try it with the fix included, see how the results are affected.
I've not yet managed to digest your various benchmark results, but also double check that patch 6/7 from this series is not to 'blame' for the some of the changes.
The 0day robot fingered that patch for at least one issue.
In that case the benchmark threads ended up 'heavier' than before, which resulted in less preemptions. Probably ksoftirqd getting ran less and causing a regression in network throughput for that thing.
I did suggest trying to change the slice of ksoftirqd down, such that it might be ran more readily, but I'm not sure that ever got tried.
Quoted 55 lines · Peter Zijlstra
Quoted 39 lines · Szabina Korbai
Quoted 32 lines · K Prateek Nayak
Hello Szabina,
Quoted 16 lines · Szabina Korbai
Hello Peter,
We ran the same benchmarks (schbench, sysbench, hackbench) as Shubhang
has on s390 on an LPAR running fedora 43 with 32 vCPUs.We ran the benchmarks for each of the cgroup modes, and for the baseline, we chose the commit prior to the patches (
f666241e6bd5-
sched/fair: Unifycfs_rqthrottling viaaccount_cfs_rq_runtime()).We have also tried running stress-ng in parallel with the benchmarks
(set to generate 50% or 90% utilization for each vCPU).Quoted 5 lines
Compared to simply running the benchmarks on their own, this has
revealed some performance trade-offs that the move to a single runqueue
can introduce.Are you using tip:sched/core at commit
68e3748781("sched/fair:
Fix
flat
hierarchy") for theflat_cgnumbers or did you checkout at85570f10a4c6
("sched/eevdf: Move to a single runqueue")?Quoted 4 lines
There are a couple fixes for vruntime update and Vincent's optimizations
for preemption bits which might make a difference to the overall results.Quoted 6 lines
Hi Prateek,
thank you, that's a good call. I did checkout at "Move to a single runqueue". Let me try it with the fix included, see how the results are
affected.I've not yet managed to digest your various benchmark results, but also
double check that patch 6/7 from this series is not to 'blame' for the
some of the changes.The 0day robot fingered that patch for at least one issue.
Quoted 8 lines
In that case the benchmark threads ended up 'heavier' than before, which
resulted in less preemptions. Probably ksoftirqd getting ran less and causing a regression in network throughput for that thing.I did suggest trying to change the slice of ksoftirqd down, such that it
might be ran more readily, but I'm not sure that ever got tried.
Hello Peter,
I've repeated the benchmark runs by checking out at tip:sched/core 85570f10a4c6 - "sched/eevdf: Move to a single runqueue" and cherry- picking the fix (tip:sched/core at commit 68e3748781 - "sched/fair: Fix flat hierarchy") on top of it. The baseline was 4f166adb5cb0 - sched/fair: Fix overflow in update_tg_cfs_runnable().
It does look like most of the regressions/inconsistencies I saw earlier were addressed by applying Vincent's fix.
Overall the effect of the patch series on system performance is now more positive than negative, though there are still some regressions in some cases. (Up mode remains something of an outlier in this regard.)
At this time I haven't yet applied Jake Steinman's patch (sched/fair:
floor tg_cpus() at 1), but neither did the system experience a kernel panic.
Observations by benchmark:
HACKBENCH
More parallel threads/processes benefit more from the patch, while fewer ones show regression - without an additional stress-ng workload. This is the same trend that Shubhang's benchmark results showed.
With an additional stress-ng workload running in parallel, there is regression in the hackbench benchmark, but the bogo-ops/second metric of stress-ng does improve to a much greater degree (for example 1.30% regression with 32 threads for concur mode - and a 66.22% improvement to the stress-ng workload).
There was quite a high variance to the stress-ng results though, especially to the baseline: worst was the 32 processes case where the CV was 15.83%.
SCHBENCH
The improvement in p99.9 tail latency is now more consistent even at 90% CPU utilization. One exception is the 32 threads with no locking case, which shows a ~5% regression in concur mode without the stress-ng workload.
Additionally, with the stress-ng load set to 90% the p50 and p90 latency of the hackbench workload still does show regression under some conditions.
Aside from up mode, schbench RPS and stress-ng bogo-ops/s values showed more minor changes (<2% difference) individually, and usually balanced each other out.
SYSBENCH
The two (sysbench and stress-ng) workloads together showed overall performance improvement (~2%), but not sysbench on its own.
Numbers below:
=======================================================================
HACKBENCH RESULTS
=======================================================================
* - CV over 5%
(lower = better)
Hackbench % diff from baseline by mode:
[s-00] Hackbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+ | Argument | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | 1 thread | +2.07% | +2.55% | +2.14% | +3.11% | -5.10% | | 2 thread | +2.65% | +2.84% | +2.16% | +1.34% | -1.37% | | 4 thread | -1.98% | -1.63% | -1.62% | -2.55% | -5.29% | | 8 thread | -5.71% | -5.09% | -5.19% | -6.46% | -8.83% | | 16 thread | -9.62% | -8.66% | -9.09% | -10.38% | -10.75% | | 32 thread | -7.47% | -6.08% | -6.11% | -7.94% | -1.15%* | | 1 process | +3.00% | +3.28% | +3.39% | +2.61% | -1.95% | | 2 process | +2.22% | +2.46% | +2.70% | +2.49% | -0.51% | | 4 process | -1.65% | -0.90% | -1.65% | -2.28% | -3.50% | | 8 process | -4.45% | -3.76% | -4.66% | -5.49% | -8.01% | | 16 process | -8.58% | -7.75% | -8.81% | -9.53% | -9.59% | | 32 process | -6.56% | -5.26% | -5.86% | -7.53% | -0.57% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Hackbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+ | Argument | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | 1 thread | +2.88% | +3.44% | +3.23% | +3.25% | +2.29% | | 2 thread | -0.57% | +0.16% | -0.26% | -0.88% | +1.35% | | 4 thread | +1.38% | +1.77% | +1.57% | +1.08% | -2.44% | | 8 thread | +7.57% | +8.20% | +8.69% | +7.22% | +2.94% | | 16 thread | -0.32% | +0.36% | +1.54% | -1.13% | +8.17% | | 32 thread | -1.65% | -0.61% | -0.70% | -2.50% | +12.28% | | 1 process | +3.27% | +2.87% | +2.83% | +3.11% | +2.24% | | 2 process | +0.09% | +0.46% | -0.10% | -0.40% | +1.80% | | 4 process | +1.03% | +1.66% | +1.24% | +0.79% | -2.38% | | 8 process | +7.82% | +8.33% | +9.05% | +7.89% | +4.85%* | | 16 process | +0.80% | +1.55% | +3.47% | +0.23% | +6.80%* | | 32 process | -1.09% | +0.06% | -0.12% | -1.86% | +17.00% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Hackbench — % diff from baseline by mode
+------------------+---------+---------+---------+---------+----------+ | Argument | concur | max | smp | tasks | up | +------------------+---------+---------+---------+---------+----------+ | 1 thread | +7.32% | +7.55% | +7.03% | +5.90% | +15.32% | | 2 thread | +9.16% | +9.38% | +8.95% | +8.19% | +13.79% | | 4 thread | +18.34% | +18.44% | +18.34% | +17.76% | +21.62% | | 8 thread | +17.14% | +17.64% | +19.50% | +16.86% | +21.07%* | | 16 thread | +7.39% | +8.01% | +10.73% | +6.20% | +26.18% | | 32 thread | +0.14% | +0.58% | +0.87% | -0.55% | +22.58%* | | 1 process | +6.12% | +7.08% | +6.46% | +5.84% | +19.06% | | 2 process | +8.79% | +8.31% | +8.60% | +8.09% | +14.79% | | 4 process | +18.76% | +19.15% | +20.63% | +18.46% | +21.42% | | 8 process | +17.55% | +17.99% | +19.75% | +16.96% | +19.02%* | | 16 process | +7.49% | +8.22% | +10.65% | +6.81% | +25.28% | | 32 process | +1.30% | +2.95% | +1.77% | +0.50% | +23.78%* | +------------------+---------+---------+---------+---------+----------+
stress-ng (int64 stressor) mean % diff from baseline by stress level (Hackbench):
(higher = better)
[s-50] Hackbench — stress-ng mean % diff from baseline
+----------------+---------+----------+----------+---------+----------+ | Argument | concur | max | smp | tasks | up | +----------------+---------+----------+----------+---------+----------+ | 1 thread | -3.16% | -2.78% | -3.38% | -2.90% | -2.76% | | 2 thread | -0.32% | -0.48% | -0.64% | -0.23% | -0.61% | | 4 thread | +14.47% | +14.50% | +14.76% | +14.82% | +14.61% | | 8 thread | +38.73% | +38.45% | +39.97% | +39.00% | +41.90% | | 16 thread | +42.93% | +44.01% | +46.30%* | +42.41% | +71.38%* | | 32 thread | +47.79% | +50.61%* | +50.44%* | +49.75% | +129.7…* | | 1 process | -2.19% | -2.80% | -2.94% | -2.56% | -2.72% | | 2 process | -0.30% | -0.58% | -0.76% | -0.61% | -0.49% | | 4 process | +12.84% | +12.70% | +12.97% | +12.98% | +12.90% | | 8 process | +37.22% | +36.93% | +38.34% | +37.60% | +40.19% | | 16 process | +44.33% | +44.00% | +48.00%* | +44.04% | +61.75%* | | 32 process | +51.70% | +51.24%* | +50.52%* | +51.62% | +142.8…* | +----------------+---------+----------+----------+---------+----------+
[s-90] Hackbench — stress-ng mean % diff from baseline
+--------------+----------+----------+----------+----------+----------+ | Argument | concur | max | smp | tasks | up | +----- --------+----------+----------+----------+----------+----------+ | 1 thread | +2.68% | +2.61% | +2.63% | +2.48% | +5.19% | | 2 thread | +5.85% | +6.20%* | +6.06% | +5.92% | +10.09% | | 4 thread | +23.55%* | +23.53%* | +23.64%* | +23.79%* | +34.50%* | | 8 thread | +47.11% | +47.47%* | +49.44% | +47.59% | +61.43% | | 16 thread | +56.40% | +58.08%* | +63.22%* | +55.92% | +102.2…* | | 32 thread | +58.54% | +59.72%* | +60.52%* | +58.75%* | +164.9…* | | 1 process | +2.16% | +2.50% | +2.44% | +2.46% | +5.99% | | 2 process | +5.48% | +5.37% | +5.58% | +5.70% | +10.38%* | | 4 process | +23.87%* | +23.74%* | +25.94%* | +24.97%* | +33.32%* | | 8 process | +45.77% | +46.32% | +48.47% | +46.12% | +57.70% | | 16 process | +56.41% | +57.84% | +62.58%* | +56.79% | +100.9…* | | 32 process | +66.22% | +72.01%* | +65.52%* | +68.43%* | +178.1…* | +--------------+----------+----------+----------+----------+----------+
=======================================================================
SCHBENCH 16T -- LOCKING: NO
=======================================================================
Schbench % diff from baseline by mode:
(Latency: lower = better, RPS: higher = better)
[s-00] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +0.00% | +0.00% | +0.00% | +0.00% | +0.00% | | Req p90 (us) | +0.23% | +0.23% | +0.23% | +0.23% | -0.45% | | Req p99.9(us) | -9.57% | -6.84% | -4.79% | -6.84% | -7.18% | | RPS p50 (req) | +0.00% | +0.00% | +0.00% | +0.00% | +0.35% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | -3.54% | -3.81% | -2.99% | -3.27% | -3.54% | | Req p90 (us) | -2.19% | -1.97% | -1.75% | -2.41% | -3.29% | | Req p99.9(us) | -3.11% | -2.59% | -0.52% | -3.63% | +1.82% | | RPS p50 (req) | +0.96% | +0.96% | +1.28% | +0.96% | +0.64% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Schbench 16t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +3.52% | +3.52% | +3.52% | +3.52% | +3.52% | | Req p90 (us) | +6.04% | +6.24% | +6.24% | +6.04% | +25.91% | | Req p99.9(us) | -16.82% | -14.59% | -14.18% | -17.63% | -3.24% | | RPS p50 (req) | +0.37% | +0.37% | +0.37% | +0.37% | -4.08% | +-------------------+---------+---------+---------+---------+---------+
stress-ng (int64 stressor) bogo-ops/s statistics by stress level (schbench):
(higher = better)
[s-50] Schbench 16t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 442.34 | ±0.48% | 0.48% | +----------+------------+---------+-------+ | concur | -1.32% | ±0.76% | 0.76% | | max | -1.28% | ±0.86% | 0.86% | | smp | -1.37% | ±0.77% | 0.77% | | tasks | -1.23% | ±0.75% | 0.75% | | up | -1.07% | ±0.45% | 0.45% | +----------+------------+---------+-------+
[s-90] Schbench 16t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+--------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+--------+ | baseline | 524.18 | ±1.51% | 1.50% | +----------+------------+---------+--------+ | concur | +0.03% | ±1.19% | 1.19% | | max | +0.17% | ±1.57% | 1.57% | | smp | +0.18% | ±1.16% | 1.16% | | tasks | +0.14% | ±1.25% | 1.25% | | up | +4.94% | ±13.42% | 13.42% | +----------+------------+---------+--------+
=======================================================================
SCHBENCH 16T -- LOCKING: YES
=======================================================================
Schbench % diff from baseline by mode:
(Latency: lower = better, RPS: higher = better)
[s-00] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +0.00% | +0.00% | +0.00% | +0.00% | +0.00% | | Req p90 (us) | +0.23% | +0.23% | +0.00% | +0.23% | -0.23% | | Req p99.9(us) | -10.29% | -13.01% | -10.59% | -13.92% | -17.55% | | RPS p50 (req) | +0.00% | +0.00% | +0.00% | +0.00% | +0.35% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | -3.21% | -2.95% | -2.68% | -2.95% | -5.35% | | Req p90 (us) | -3.33% | -3.33% | -2.22% | -3.70% | -6.56% | | Req p99.9(us) | -3.15% | -5.15% | -5.72% | -5.72% | -9.44% | | RPS p50 (req) | +1.37% | +1.37% | +1.03% | +1.37% | +2.39% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Schbench 16t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +3.51% | +3.51% | +3.26% | +3.51% | +2.01% | | Req p90 (us) | -0.76% | -0.38% | -0.38% | -0.76% | +9.56% | | Req p99.9(us) | -7.08% | -2.10% | -0.26% | -6.82% | +32.77% | | RPS p50 (req) | +0.77% | +0.77% | +0.77% | +0.77% | -3.56% | +-------------------+---------+---------+---------+---------+---------+
stress-ng (int64 stressor) bogo-ops/s statistics by stress level (schbench):
(higher = better)
[s-50] Schbench 16t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 453.53 | ±0.38% | 0.38% | +----------+------------+---------+-------+ | concur | -1.54% | ±0.53% | 0.53% | | max | -1.57% | ±0.60% | 0.60% | | smp | -1.59% | ±0.68% | 0.68% | | tasks | -1.59% | ±0.55% | 0.55% | | up | -1.92% | ±0.58% | 0.59% | +----------+------------+---------+-------+
[s-90] Schbench 16t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+--------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+--------+ | baseline | 531.40 | ±1.73% | 1.73% | +----------+------------+---------+--------+ | concur | -0.26% | ±1.14% | 1.14% | | max | -0.16% | ±1.71% | 1.71% | | smp | -0.13% | ±1.49% | 1.49% | | tasks | -0.32% | ±1.19% | 1.19% | | up | +1.91% | ±10.60% | 10.60% | +----------+------------+---------+--------+
=======================================================================
SCHBENCH 32T -- LOCKING: NO
=======================================================================
Schbench % diff from baseline by mode:
(Latency: lower = better, RPS: higher = better)
[s-00] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +0.00% | +0.00% | +0.00% | +0.00% | +0.00% | | Req p90 (us) | -0.23% | +0.00% | +0.00% | -0.45% | -0.23% | | Req p99.9(us) | +5.14% | +1.47% | +6.97% | +3.30% | -1.10% | | RPS p50 (req) | +0.00% | +0.00% | -0.35% | +0.00% | +0.00% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | -2.70% | -2.70% | -2.43% | -2.16% | -1.62% | | Req p90 (us) | -2.20% | -1.98% | -2.20% | -2.20% | -3.51% | | Req p99.9(us) | -3.40% | -1.05% | -1.57% | -2.61% | +0.00% | | RPS p50 (req) | +1.27% | +1.27% | +1.27% | +1.27% | +1.58% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Schbench 32t -- Locking: No
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +3.27% | +3.27% | +3.02% | +3.27% | +3.27% | | Req p90 (us) | +12.35% | +12.35% | +12.13% | +12.13% | +32.08% | | Req p99.9(us) | -16.45% | -16.06% | -17.04% | -16.06% | -6.07% | | RPS p50 (req) | +1.11% | +1.11% | +1.11% | +1.11% | -3.34% | +-------------------+---------+---------+---------+---------+---------+
stress-ng (int64 stressor) bogo-ops/s statistics by stress level (schbench):
(higher = better)
[s-50] Schbench 32t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 438.90 | ±0.49% | 0.48% | +----------+------------+---------+-------+ | concur | -1.55% | ±0.63% | 0.63% | | max | -1.59% | ±0.71% | 0.71% | | smp | -1.50% | ±0.79% | 0.79% | | tasks | -1.49% | ±0.63% | 0.63% | | up | -1.86% | ±0.62% | 0.62% | +----------+------------+---------+-------+
[s-90] Schbench 32t -- Locking: No — stress-ng vs baseline
+----------+------------+---------+--------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+--------+ | baseline | 523.09 | ±1.70% | 1.70% | +----------+------------+---------+--------+ | concur | -0.33% | ±1.21% | 1.21% | | max | -0.29% | ±1.36% | 1.36% | | smp | -0.40% | ±1.24% | 1.24% | | tasks | -0.27% | ±1.31% | 1.31% | | up | +3.62% | ±11.49% | 11.49% | +----------+------------+---------+--------+
=======================================================================
SCHBENCH 32T -- LOCKING: YES
=======================================================================
Schbench % diff from baseline by mode:
(Latency: lower = better, RPS: higher = better)
[s-00] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +0.00% | +0.00% | +0.00% | +0.00% | +0.00% | | Req p90 (us) | +0.00% | +0.23% | +0.23% | +0.23% | -0.45% | | Req p99.9(us) | -19.66% | -11.98% | -18.98% | -24.41% | -36.38% | | RPS p50 (req) | +0.00% | +0.00% | +0.00% | +0.00% | +0.35% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | -3.18% | -3.18% | -3.44% | -3.18% | -3.18% | | Req p90 (us) | -2.59% | -2.22% | -1.85% | -2.96% | -5.45% | | Req p99.9(us) | -2.97% | -1.49% | -2.38% | -3.27% | -6.54% | | RPS p50 (req) | +1.70% | +1.36% | +1.36% | +1.36% | +3.06% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Schbench 32t -- Locking: Yes
+-------------------+---------+---------+---------+---------+---------+ | Metric | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | Req p50 (us) | +3.27% | +3.27% | +3.02% | +3.27% | +1.76% | | Req p90 (us) | +2.69% | +2.69% | +2.69% | +2.69% | +47.36% | | Req p99.9(us) | -8.60% | -8.60% | -2.28% | -8.60% | +41.34% | | RPS p50 (req) | +1.54% | +1.54% | +1.54% | +1.54% | -7.39% | +-------------------+---------+---------+---------+---------+---------+
stress-ng (int64 stressor) bogo-ops/s statistics by stress level (schbench):
(higher = better)
[s-50] Schbench 32t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+-------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+-------+ | baseline | 451.63 | ±0.40% | 0.40% | +----------+------------+---------+-------+ | concur | -1.82% | ±0.45% | 0.45% | | max | -1.80% | ±0.45% | 0.45% | | smp | -1.72% | ±0.55% | 0.55% | | tasks | -1.76% | ±0.46% | 0.47% | | up | -2.61% | ±0.87% | 0.87% | +----------+------------+---------+-------+
[s-90] Schbench 32t -- Locking: Yes — stress-ng vs baseline
+----------+------------+---------+--------+ | Variant | Mean %diff | StdDev± | CV | +----------+------------+---------+--------+ | baseline | 529.80 | ±1.78% | 1.78% | +----------+------------+---------+--------+ | concur | -0.75% | ±1.12% | 1.12% | | max | -0.72% | ±1.14% | 1.14% | | smp | -0.66% | ±0.96% | 0.96% | | tasks | -0.67% | ±1.19% | 1.19% | | up | +5.03% | ±15.81% | 15.81% | +----------+------------+---------+--------+
=======================================================================
SYSBENCH RESULTS
=======================================================================
Sysbench throughput % diff from baseline by mode:
(higher = better)
[s-00] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+ | Argument | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | memory | -0.18% | +0.12% | -0.36% | -0.18% | +0.00% | +-------------------+---------+---------+---------+---------+---------+
[s-50] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+ | Argument | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | memory | -1.71% | -0.43% | -0.71% | -1.07% | -2.21% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Sysbench — % diff from baseline by mode
+-------------------+---------+---------+---------+---------+---------+ | Argument | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | memory | +2.94% | +3.97% | +3.46% | +4.34% | -6.76% | +-------------------+---------+---------+---------+---------+---------+
stress-ng (int64 stressor) mean % diff from baseline by stress level (Sysbench):
(higher = better)
[s-50] Sysbench — stress-ng mean % diff from baseline
+-------------------+---------+---------+---------+---------+---------+ | Argument | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | memory | +3.59% | +2.78% | +4.36% | +3.60% | +1.46% | +-------------------+---------+---------+---------+---------+---------+
[s-90] Sysbench — stress-ng mean % diff from baseline
+-------------------+---------+---------+---------+---------+---------+ | Argument | concur | max | smp | tasks | up | +-------------------+---------+---------+---------+---------+---------+ | memory | +0.85% | -1.35% | -0.00% | -1.79% | +18.47% | +-------------------+---------+---------+---------+---------+---------+
Regards,
Signature · 13 lines
Szabina Korbai Linux on Z development Software Labs Campus Unlimited Company 25 North Wall Quay, Dublin 1, D01 H104, Ireland [email protected] IBM
Quoted 23 lines · Peter Zijlstra
Quoted 13 lines
Quoted 11 lines
Are you using tip:sched/core at commit
68e3748781("sched/fair: Fix flat
hierarchy") for theflat_cgnumbers or did you checkout at85570f10a4c6
("sched/eevdf: Move to a single runqueue")?There are a couple fixes for vruntime update and Vincent's optimizations
for preemption bits which might make a difference to the overall results.Quoted 5 lines
Hi Prateek,
thank you, that's a good call. I did checkout at "Move to a single runqueue". Let me try it with the fix included, see how the results are affected.
I've not yet managed to digest your various benchmark results, but also double check that patch 6/7 from this series is not to 'blame' for the some of the changes.
Quoted 6 lines
The 0day robot fingered that patch for at least one issue.
In that case the benchmark threads ended up 'heavier' than before, which resulted in less preemptions. Probably ksoftirqd getting ran less and causing a regression in network throughput for that thing.
The 0day' regression was triggered by running netperf/netserver using loopback, so ksoftirqd was not really running very frequently IMO.
Quoted 2 lines
I did suggest trying to change the slice of ksoftirqd down, such that it might be ran more readily, but I'm not sure that ever got tried.
The 0day regression was triggered by running netperf/netserver over loopback, so ksoftirqd was not running very frequently IIUC.
I borrowed a similar machine(no-SMT, every 4 Cores share L2, 192 Cores) as 0day used. I asked AI to generate a simple test script based on 0day's reproducer (attached at the end of this email), and successfully reproduced the regression.
TL;DR,
The decision made by wake_affine_weight() seems to be affected by the increased weight of the wakee under concur mode. As a result, wake_affine_weight() became less likely to select this_cpu. This reduced preference for this_cpu leads to a higher L2 miss rate on this platform, and consequently lower throughput.
Test summary:
| Config | Throughput (Mbps) | **L2 miss%** | **LLC miss%** | | smp + WA_WEIGHT | **470249** | **19.37%** | 60.08% | | smp + NO_WA_WEIGHT | 305360 | 32.67% | 60.46% | | concur + WA_WEIGHT | 338770 | 31.45% | 62.85% | | concur + NO_WA_WEIGHT | 259358 | 36.24% | 61.95% |
We can see if WA_WEIGHT is disabled in smp mode, the performance drops to concur. One possible reason is that task_h_load(p) returns a very small value in smp mode(just the issue that concur wants to fix), whereas it returns a "normal" value in concur mode.
In smp mode, the extremly small value returned by task_h_load(p) can be effectively ignored in the comparison in wake_affine_weight(). As a result, the condition for returning this_cpu becomes:
cpu_load(cpu_rq(this_cpu)) * 100 < cpu_load(cpu_rq(prev_cpu) * 108
As a result, this_cpu has a higher chance of being selected in smp mode. In concur mode, task_h_load(p) is on par with cpu_load(), so it cannot be simply ignored and provides a fairer basis for choosing between this_cpu and prev_cpu. Since netperf/netserver is a sync wakeup, it prefers to be co-located on the same CPU/L2 domain for better L2 cache locality.
I'm not sure if this is the expected behavior before I further digging into the details, a hack patch could restore part of the performance, because it increases the ratio to return this_cpu:
thanks,
Chenyu
kernel/sched/fair.c +5 -2
@@ -8385,6 +8385,7 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,unsigned long task_load; this_eff_load =cpu_load (cpu_rq (this_cpu)); + task_load =task_h_load (p);if (sync) {unsigned long current_load =task_h_load (current);
@@ -8392,11 +8393,13 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,if (current_load > this_eff_load)return this_cpu; +/* replace current load with wakee load does no harm */ +if (sched_feat (WA_SYNC_WEIGHT ) && current_load >= task_load) +return this_cpu; + this_eff_load -= current_load; } - task_load =task_h_load (p); - this_eff_load += task_load;if (sched_feat (WA_BIAS )) this_eff_load *=100 ;
kernel/sched/features.h +1 -0
@@ -129,6 +129,7 @@ SCHED_FEAT(ATTACH_AGE_LOAD, true) SCHED_FEAT(WA_IDLE, true) SCHED_FEAT(WA_WEIGHT, true) SCHED_FEAT(WA_BIAS, true) +SCHED_FEAT (WA_SYNC_WEIGHT , true)
test command:
NR=$(( $(nproc) * 2 ))run() {
local mode=$1 waw=$2 tag=$3
echo $mode | sudo -n tee /sys/kernel/debug/sched/cgroup_mode >/dev/null
echo $waw | sudo -n tee /sys/kernel/debug/sched/features >/dev/null
sudo -n pkill -x netserver; sleep 1
netserver -4 >/dev/null 2>&1; sleep 1
rm -f /tmp/m4.$tag
for i in $(seq $NR); do
netperf -4 -H 127.0.0.1 -t TCP_MAERTS -l 70 -P 0 >> /tmp/m4.$tag 2>/dev/null &
done
sleep 15
sudo -n ./tools/perf/perf stat -a \
-e l2_request.hit,l2_request.miss,longest_lat_cache.reference,longest_lat_cache.miss \
-x, -o /tmp/p4.$tag -- sleep 20
wait
} >/dev/null 2>&1
run smp WA_WEIGHT smp_waw
run smp NO_WA_WEIGHT smp_nowaw
run concur WA_WEIGHT con_waw
run concur NO_WA_WEIGHT con_nowaw
echo WA_WEIGHT | sudo -n tee /sys/kernel/debug/sched/features >/dev/null
printf "%-12s %10s %8s %10s %10s\n" CONFIG Mbps streams L2miss% LLCmiss% for t in smp_waw smp_nowaw con_waw con_nowaw; do
read TP NS < <(awk '{s+=$NF;n++} END{print s, n}' /tmp/m4.$t) awk -F, -v t=$t -v tp=$TP -v ns=$NS '
$3=="l2_request.hit"{h=$1} $3=="l2_request.miss"{m=$1}
$3=="longest_lat_cache.reference"{r=$1}
$3=="longest_lat_cache.miss"{lm=$1}
END{printf "%-12s %10.0f %8d %9.2f%% %9.2f%%\n", t, tp, ns, 100*m/(h+m), 100*lm/r}' /tmp/p4.$t
done
Peter Zijlstra
maintainerQuoted 6 lines · Chen Yu
TL;DR,
The decision made bywake_affine_weight()seems to be affected by the increased weight of the wakee under concur mode. As a result,wake_affine_weight()became less likely to selectthis_cpu. This reduced preference forthis_cpuleads to a higher L2 miss rate on this platform, and consequently lower throughput.
Quoted 7 lines
Test summary:
| Config | Throughput (Mbps) | **L2 miss%** | **LLC miss%** | | smp + WA_WEIGHT | **470249** | **19.37%** | 60.08% | | smp + NO_WA_WEIGHT | 305360 | 32.67% | 60.46% | | concur + WA_WEIGHT | 338770 | 31.45% | 62.85% | | concur + NO_WA_WEIGHT | 259358 | 36.24% | 61.95% |
(I'm thinking it would've been easier to read if all those '|' were properly aligned?)
Quoted 8 lines
We can see if
WA_WEIGHTis disabled in smp mode, the performance drops to concur. One possible reason is thattask_h_load(p)returns a very small value in smp mode(just the issue that concur wants to fix), whereas it returns a "normal" value in concur mode.In smp mode, the extremly small value returned by
task_h_load(p)can be effectively ignored in the comparison inwake_affine_weight(). As a result, the condition for returningthis_cpubecomes:
Quoted 6 lines
cpu_load(cpu_rq(this_cpu)) * 100 < cpu_load(cpu_rq(prev_cpu) * 108
As a result,
this_cpuhas a higher chance of being selected in smp mode. In concur mode,task_h_load(p)is on par withcpu_load(), so it cannot be simply ignored and provides a fairer basis for choosing betweenthis_cpuandprev_cpu. Since netperf/netserver is a sync wakeup, it prefers to be co-located on the same CPU/L2 domain for better L2 cache locality.
There is something wonky with task_h_load(), it isn't behaving properly. I've not managed to put a finger on it yet. Let me continue poking at it.