sched: Flatten the pick
Subsystems (3)
Peter Zijlstra
maintainerv2 cover sched: Flatten the pick
Hi!
So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.
The problems with weight distribution are related to that infernal global fraction:
tg->w * grq_i->w
ge_i->w = ----------------
\Sum_j grq_j->wwhich we've approximated reasonably well by now. However, the immediate consequence of this fraction is that the total group weight (tg->w) gets fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine with the fact that 256 CPU systems are relatively common these days, this becomes painful.
The common 'solution' is to inflate the group weight by 'nr_cpus'; the immediate problem with that is that when all load of a group gets concentrated on a single CPU, the per-cpu cgroup weight becomes insanely large, easily exceeding nice -20.
Additionally there are numerical limits on the max weight you can have before the math starts suffering overflows. As such there is a definite limit on the total group weight. Which has annoyed people ;-)
The first few patches add a knob /debug/sched/cgroup_mode and a few different options on how to deal with this. My favourite is 'concur', but obviously that is also the most expensive one :-/ It adds a tg->tasks counter which makes the update_tg_load_avg() thing more expensive.
I have some ideas but I figured I ought to share these things before sinking more time into it.
On to the hierarchical pick; this has been causing trouble for a very long time. So once again an attempt at flatting it. The basic idea is to keep the full hierarchical load tracking as-is, but keep all the runnable entities in a single level. The immediate concequence of all this is ofcourse that we need to constantly re-compute the effective weight of each entity as things progress.
Reweight is done on:
- enqueue
- pick -- or rather set_next_entity(.first=true)
- tick
So while the {en,de}queue operations are still O(depth) due to the full accounting mess, the pick is now a single level. Removing the intermediate levels that obscure runnability etc.
For testing, I've done a little experiment, I dug out what is colloqually known as a potato. A trusty old Sandybridge 12600k with a RX 580, and ran a game on it. From GOG, I had available 'Shadows: Awakens', a fun title that normally runs really well on this machine (provided you stick to 1080p).
To make it interesting, I added 8 (one for each logical CPU) copies of: 'nice spin.sh'; this results in the game becoming almost unplayable, as in proper terrible.
I used MangoHUD to record a few minutes of playtime for statistics, and then quit the came and re-started it with a shorter slice set (base/10). This results in the game being entirely playable -- not great, but definiltey playable.
Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper) Intel Core i7-2600K
AMD Radeon RX 580
Shadows Awakening (GOG)
default slice(*)
FPS min 3.8 20.6
avg 48.0 57.2
mag 87.4 80.3FT min 9.4 8.4
avg 34.5 19.5
max 107.4 37.2 FPS (Frames Per Second)
FT (FrameTime)
[*] Command prefix: 'chrt -o --sched-runtime 280000 0'
effectively setting 'base_slice_ns/10'I have not compared to a kernel without flat on, just wanted to run non trivial workloads and play with slice to make sure everything 'works'.
Can also be had:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat
include/linux/cpuset.h | 6 include/linux/sched.h | 1 kernel/cgroup/cpuset.c | 15 kernel/sched/core.c | 47 -- kernel/sched/debug.c | 171 +++++--- kernel/sched/fair.c | 1038 ++++++++++++++++++++++--------------------------- kernel/sched/pelt.c | 6 kernel/sched/sched.h | 44 -- 8 files changed, 672 insertions(+), 656 deletions(-)
---
Change since v1 ( https://patch.msgid.link/[email protected] ):
- various Sashiko thingies
- rebase atop curren -tip
Peter Zijlstra
maintainerPATCH v2 1/10 sched/debug: Use char * instead of char (*)[]
Some of the fancy AI robots are getting 'upset'.
--- kernel/sched/debug.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/debug.c +2 -2
@@ -136,7 +136,7 @@ sched_feat_write(struct file *filp, consif (cnt >63 ) cnt =63 ; -if (copy_from_user (&buf, ubuf, cnt)) +if (copy_from_user (buf, ubuf, cnt))return -EFAULT ; buf[cnt] =0 ;
@@ -221,7 +221,7 @@ static ssize_t sched_dynamic_write(strucif (cnt >15 ) cnt =15 ; -if (copy_from_user (&buf, ubuf, cnt)) +if (copy_from_user (buf, ubuf, cnt))return -EFAULT ; buf[cnt] =0 ;
[tip: sched/core] sched/debug: Use char * instead of char (*)[]
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 333f6f0e11acc20d036f94f94709874f76d0b430 Gitweb: https://git.kernel.org/tip/333f6f0e11acc20d036f94f94709874f76d0b430 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 11 May 2026 13:31:05 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 26 May 2026 13:53:12 +02:00
sched/debug: Use char * instead of char (*)[]
Some of the fancy AI robots are getting 'upset'.
--- kernel/sched/debug.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/debug.c +2 -2
@@ -136,7 +136,7 @@ sched_feat_write(struct file *filp, const char __user *ubuf,if (cnt >63 ) cnt =63 ; -if (copy_from_user (&buf, ubuf, cnt)) +if (copy_from_user (buf, ubuf, cnt))return -EFAULT ; buf[cnt] =0 ;
@@ -263,7 +263,7 @@ static ssize_t sched_dynamic_write(struct file *filp, const char __user *ubuf,if (cnt >15 ) cnt =15 ; -if (copy_from_user (&buf, ubuf, cnt)) +if (copy_from_user (buf, ubuf, cnt))return -EFAULT ; buf[cnt] =0 ;
Peter Zijlstra
maintainerPATCH v2 2/10 sched: Use {READ,WRITE}_ONCE() for preempt_dynamic_mode
In reply to Peter ZijlstraRobots figured out you can read and write this concurrently and got 'upset'. Gemini even noted sched_dynamic_show() can generate 'confusing' output if it observed different values during the printing.
--- kernel/sched/core.c | 15 ++++++++------- kernel/sched/debug.c | 5 +++-- 2 files changed, 11 insertions(+), 9 deletions(-)
kernel/sched/core.c +8 -7
@@ -7743,7 +7743,7 @@ static void __sched_dynamic_update(int mbreak ; } - preempt_dynamic_mode = mode; +WRITE_ONCE (preempt_dynamic_mode, mode); }void sched_dynamic_update (int mode)
@@ -7784,12 +7784,13 @@ static void __init preempt_dynamic_init( } } -# define PREEMPT_MODEL_ACCESSOR (mode) \ - bool preempt_model_##mode(void) \ - { \ - WARN_ON_ONCE(preempt_dynamic_mode == preempt_dynamic_undefined); \ - return preempt_dynamic_mode == preempt_dynamic_##mode; \ - } \ +# define PREEMPT_MODEL_ACCESSOR (mode) \ + bool preempt_model_##mode(void) \ + { \ + int mode = READ_ONCE(preempt_dynamic_mode); \ + WARN_ON_ONCE(mode == preempt_dynamic_undefined); \ + return mode == preempt_dynamic_##mode; \ + } \ EXPORT_SYMBOL_GPL(preempt_model_##mode)PREEMPT_MODEL_ACCESSOR (none);
kernel/sched/debug.c +3 -2
@@ -239,6 +239,7 @@ static ssize_t sched_dynamic_write(strucstatic int sched_dynamic_show (struct seq_file *m,void *v) {int i = (IS_ENABLED (CONFIG_PREEMPT_RT ) ||IS_ENABLED (CONFIG_ARCH_HAS_PREEMPT_LAZY )) *2 ; +int mode =READ_ONCE (preempt_dynamic_mode);int j;/* Count entries in NULL terminated preempt_modes */
@@ -247,10 +248,10 @@ static int sched_dynamic_show(struct seq j -= !IS_ENABLED (CONFIG_ARCH_HAS_PREEMPT_LAZY );for (; i < j; i++) { -if (preempt_dynamic_mode == i) +if (mode == i)seq_puts (m,"(" );seq_puts (m, preempt_modes[i]); -if (preempt_dynamic_mode == i) +if (mode == i)seq_puts (m,")" );seq_puts (m," " );
[tip: sched/core] sched: Use {READ,WRITE}_ONCE() for preempt_dynamic_mode
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 77557002234546a4fb46ebf517d6cb1d515535a9 Gitweb: https://git.kernel.org/tip/77557002234546a4fb46ebf517d6cb1d515535a9 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 11 May 2026 13:31:06 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 26 May 2026 13:53:13 +02:00
sched: Use {READ,WRITE}_ONCE() for preempt_dynamic_mode
Robots figured out you can read and write this concurrently and got 'upset'. Gemini even noted sched_dynamic_show() can generate 'confusing' output if it observed different values during the printing.
--- kernel/sched/core.c | 15 ++++++++------- kernel/sched/debug.c | 5 +++-- 2 files changed, 11 insertions(+), 9 deletions(-)
kernel/sched/core.c +8 -7
@@ -7979,7 +7979,7 @@ static void __sched_dynamic_update(int mode)break ; } - preempt_dynamic_mode = mode; +WRITE_ONCE (preempt_dynamic_mode, mode); }void sched_dynamic_update (int mode)
@@ -8020,12 +8020,13 @@ static void __init preempt_dynamic_init(void) } } -# define PREEMPT_MODEL_ACCESSOR (mode) \ - bool preempt_model_##mode(void) \ - { \ - WARN_ON_ONCE(preempt_dynamic_mode == preempt_dynamic_undefined); \ - return preempt_dynamic_mode == preempt_dynamic_##mode; \ - } \ +# define PREEMPT_MODEL_ACCESSOR (mode) \ + bool preempt_model_##mode(void) \ + { \ + int mode = READ_ONCE(preempt_dynamic_mode); \ + WARN_ON_ONCE(mode == preempt_dynamic_undefined); \ + return mode == preempt_dynamic_##mode; \ + } \ EXPORT_SYMBOL_GPL(preempt_model_##mode)PREEMPT_MODEL_ACCESSOR (none);
kernel/sched/debug.c +3 -2
@@ -281,6 +281,7 @@ static ssize_t sched_dynamic_write(struct file *filp, const char __user *ubuf,static int sched_dynamic_show (struct seq_file *m,void *v) {int i = (IS_ENABLED (CONFIG_PREEMPT_RT ) ||IS_ENABLED (CONFIG_ARCH_HAS_PREEMPT_LAZY )) *2 ; +int mode =READ_ONCE (preempt_dynamic_mode);int j;/* Count entries in NULL terminated preempt_modes */
@@ -289,10 +290,10 @@ static int sched_dynamic_show(struct seq_file *m, void *v) j -= !IS_ENABLED (CONFIG_ARCH_HAS_PREEMPT_LAZY );for (; i < j; i++) { -if (preempt_dynamic_mode == i) +if (mode == i)seq_puts (m,"(" );seq_puts (m, preempt_modes[i]); -if (preempt_dynamic_mode == i) +if (mode == i)seq_puts (m,")" );seq_puts (m," " );
Peter Zijlstra
maintainerPATCH v2 3/10 sched/debug: Collapse subsequent CONFIG_SCHED_CLASS_EXT sections
In reply to Peter Zijlstra--- kernel/sched/debug.c | 92 ++++++++++++++++++++++++--------------------------- 1 file changed, 44 insertions(+), 48 deletions(-)
kernel/sched/debug.c +44 -48
@@ -446,6 +446,8 @@ static const struct file_operations fair .release = single_release, }; +static struct dentry *debugfs_sched; +#ifdef CONFIG_SCHED_CLASS_EXT static ssize_t sched_ext_server_runtime_write(struct file *filp, const char __user *ubuf,
@@ -478,75 +480,92 @@ static const struct file_operations ext_ .llseek = seq_lseek, .release = single_release, }; -#endif /* CONFIG_SCHED_CLASS_EXT */ static ssize_t -sched_fair_server_period_write (struct file *filp,const char __user *ubuf, -size_t cnt,loff_t *ppos) +sched_ext_server_period_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) {long cpu = (long ) ((struct seq_file *) filp->private_data)->private;struct rq *rq =cpu_rq (cpu);return sched_server_write_common (filp, ubuf, cnt, ppos,DL_PERIOD , - &rq->fair_server); + &rq->ext_server); } -static int sched_fair_server_period_show (struct seq_file *m,void *v) +static int sched_ext_server_period_show (struct seq_file *m,void *v) {unsigned long cpu = (unsigned long ) m->private;struct rq *rq =cpu_rq (cpu); -return sched_server_show_common (m, v,DL_PERIOD , &rq->fair_server); +return sched_server_show_common (m, v,DL_PERIOD , &rq->ext_server); } -static int sched_fair_server_period_open (struct inode *inode,struct file *filp) +static int sched_ext_server_period_open (struct inode *inode,struct file *filp) { -return single_open (filp, sched_fair_server_period_show, inode->i_private); +return single_open (filp, sched_ext_server_period_show, inode->i_private); } -static const struct file_operations fair_server_period_fops = { - .open = sched_fair_server_period_open, - .write = sched_fair_server_period_write, +static const struct file_operations ext_server_period_fops = { + .open = sched_ext_server_period_open, + .write = sched_ext_server_period_write, .read = seq_read, .llseek = seq_lseek, .release = single_release, }; -#ifdef CONFIG_SCHED_CLASS_EXT +static void debugfs_ext_server_init (void ) +{ +struct dentry *d_ext; +unsigned long cpu; + + d_ext =debugfs_create_dir ("ext_server" , debugfs_sched); +if (!d_ext) +return ; + +for_each_possible_cpu (cpu) { +struct dentry *d_cpu; +char buf[32 ]; + +snprintf (buf,sizeof (buf),"cpu%lu" , cpu); + d_cpu =debugfs_create_dir (buf, d_ext); + +debugfs_create_file ("runtime" ,0644 , d_cpu, (void *) cpu, &ext_server_runtime_fops); +debugfs_create_file ("period" ,0644 , d_cpu, (void *) cpu, &ext_server_period_fops); + } +} +#endif /* CONFIG_SCHED_CLASS_EXT */ +static ssize_t -sched_ext_server_period_write (struct file *filp,const char __user *ubuf, -size_t cnt,loff_t *ppos) +sched_fair_server_period_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) {long cpu = (long ) ((struct seq_file *) filp->private_data)->private;struct rq *rq =cpu_rq (cpu);return sched_server_write_common (filp, ubuf, cnt, ppos,DL_PERIOD , - &rq->ext_server); + &rq->fair_server); } -static int sched_ext_server_period_show (struct seq_file *m,void *v) +static int sched_fair_server_period_show (struct seq_file *m,void *v) {unsigned long cpu = (unsigned long ) m->private;struct rq *rq =cpu_rq (cpu); -return sched_server_show_common (m, v,DL_PERIOD , &rq->ext_server); +return sched_server_show_common (m, v,DL_PERIOD , &rq->fair_server); } -static int sched_ext_server_period_open (struct inode *inode,struct file *filp) +static int sched_fair_server_period_open (struct inode *inode,struct file *filp) { -return single_open (filp, sched_ext_server_period_show, inode->i_private); +return single_open (filp, sched_fair_server_period_show, inode->i_private); } -static const struct file_operations ext_server_period_fops = { - .open = sched_ext_server_period_open, - .write = sched_ext_server_period_write, +static const struct file_operations fair_server_period_fops = { + .open = sched_fair_server_period_open, + .write = sched_fair_server_period_write, .read = seq_read, .llseek = seq_lseek, .release = single_release, }; -#endif /* CONFIG_SCHED_CLASS_EXT */ - -static struct dentry *debugfs_sched;static void debugfs_fair_server_init (void ) {
@@ -569,29 +588,6 @@ static void debugfs_fair_server_init(voi } } -#ifdef CONFIG_SCHED_CLASS_EXT -static void debugfs_ext_server_init (void ) -{ -struct dentry *d_ext; -unsigned long cpu; - - d_ext =debugfs_create_dir ("ext_server" , debugfs_sched); -if (!d_ext) -return ; - -for_each_possible_cpu (cpu) { -struct dentry *d_cpu; -char buf[32 ]; - -snprintf (buf,sizeof (buf),"cpu%lu" , cpu); - d_cpu =debugfs_create_dir (buf, d_ext); - -debugfs_create_file ("runtime" ,0644 , d_cpu, (void *) cpu, &ext_server_runtime_fops); -debugfs_create_file ("period" ,0644 , d_cpu, (void *) cpu, &ext_server_period_fops); - } -} -#endif /* CONFIG_SCHED_CLASS_EXT */ -static __init int sched_init_debug (void ) {struct dentry __maybe_unused *numa;
[tip: sched/core] sched/debug: Collapse subsequent CONFIG_SCHED_CLASS_EXT sections
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: e05777c44e53df8ab41d930510a384d65a34aafa Gitweb: https://git.kernel.org/tip/e05777c44e53df8ab41d930510a384d65a34aafa Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 11 May 2026 13:31:07 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 26 May 2026 13:53:13 +02:00
sched/debug: Collapse subsequent CONFIG_SCHED_CLASS_EXT sections
--- kernel/sched/debug.c | 92 ++++++++++++++++++++----------------------- 1 file changed, 44 insertions(+), 48 deletions(-)
kernel/sched/debug.c +44 -48
@@ -488,6 +488,8 @@ static const struct file_operations fair_server_runtime_fops = { .release = single_release, }; +static struct dentry *debugfs_sched; +#ifdef CONFIG_SCHED_CLASS_EXT static ssize_t sched_ext_server_runtime_write(struct file *filp, const char __user *ubuf,
@@ -520,75 +522,92 @@ static const struct file_operations ext_server_runtime_fops = { .llseek = seq_lseek, .release = single_release, }; -#endif /* CONFIG_SCHED_CLASS_EXT */ static ssize_t -sched_fair_server_period_write (struct file *filp,const char __user *ubuf, -size_t cnt,loff_t *ppos) +sched_ext_server_period_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) {long cpu = (long ) ((struct seq_file *) filp->private_data)->private;struct rq *rq =cpu_rq (cpu);return sched_server_write_common (filp, ubuf, cnt, ppos,DL_PERIOD , - &rq->fair_server); + &rq->ext_server); } -static int sched_fair_server_period_show (struct seq_file *m,void *v) +static int sched_ext_server_period_show (struct seq_file *m,void *v) {unsigned long cpu = (unsigned long ) m->private;struct rq *rq =cpu_rq (cpu); -return sched_server_show_common (m, v,DL_PERIOD , &rq->fair_server); +return sched_server_show_common (m, v,DL_PERIOD , &rq->ext_server); } -static int sched_fair_server_period_open (struct inode *inode,struct file *filp) +static int sched_ext_server_period_open (struct inode *inode,struct file *filp) { -return single_open (filp, sched_fair_server_period_show, inode->i_private); +return single_open (filp, sched_ext_server_period_show, inode->i_private); } -static const struct file_operations fair_server_period_fops = { - .open = sched_fair_server_period_open, - .write = sched_fair_server_period_write, +static const struct file_operations ext_server_period_fops = { + .open = sched_ext_server_period_open, + .write = sched_ext_server_period_write, .read = seq_read, .llseek = seq_lseek, .release = single_release, }; -#ifdef CONFIG_SCHED_CLASS_EXT +static void debugfs_ext_server_init (void ) +{ +struct dentry *d_ext; +unsigned long cpu; + + d_ext =debugfs_create_dir ("ext_server" , debugfs_sched); +if (!d_ext) +return ; + +for_each_possible_cpu (cpu) { +struct dentry *d_cpu; +char buf[32 ]; + +snprintf (buf,sizeof (buf),"cpu%lu" , cpu); + d_cpu =debugfs_create_dir (buf, d_ext); + +debugfs_create_file ("runtime" ,0644 , d_cpu, (void *) cpu, &ext_server_runtime_fops); +debugfs_create_file ("period" ,0644 , d_cpu, (void *) cpu, &ext_server_period_fops); + } +} +#endif /* CONFIG_SCHED_CLASS_EXT */ +static ssize_t -sched_ext_server_period_write (struct file *filp,const char __user *ubuf, -size_t cnt,loff_t *ppos) +sched_fair_server_period_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) {long cpu = (long ) ((struct seq_file *) filp->private_data)->private;struct rq *rq =cpu_rq (cpu);return sched_server_write_common (filp, ubuf, cnt, ppos,DL_PERIOD , - &rq->ext_server); + &rq->fair_server); } -static int sched_ext_server_period_show (struct seq_file *m,void *v) +static int sched_fair_server_period_show (struct seq_file *m,void *v) {unsigned long cpu = (unsigned long ) m->private;struct rq *rq =cpu_rq (cpu); -return sched_server_show_common (m, v,DL_PERIOD , &rq->ext_server); +return sched_server_show_common (m, v,DL_PERIOD , &rq->fair_server); } -static int sched_ext_server_period_open (struct inode *inode,struct file *filp) +static int sched_fair_server_period_open (struct inode *inode,struct file *filp) { -return single_open (filp, sched_ext_server_period_show, inode->i_private); +return single_open (filp, sched_fair_server_period_show, inode->i_private); } -static const struct file_operations ext_server_period_fops = { - .open = sched_ext_server_period_open, - .write = sched_ext_server_period_write, +static const struct file_operations fair_server_period_fops = { + .open = sched_fair_server_period_open, + .write = sched_fair_server_period_write, .read = seq_read, .llseek = seq_lseek, .release = single_release, }; -#endif /* CONFIG_SCHED_CLASS_EXT */ - -static struct dentry *debugfs_sched;static void debugfs_fair_server_init (void ) {
@@ -611,29 +630,6 @@ static void debugfs_fair_server_init(void) } } -#ifdef CONFIG_SCHED_CLASS_EXT -static void debugfs_ext_server_init (void ) -{ -struct dentry *d_ext; -unsigned long cpu; - - d_ext =debugfs_create_dir ("ext_server" , debugfs_sched); -if (!d_ext) -return ; - -for_each_possible_cpu (cpu) { -struct dentry *d_cpu; -char buf[32 ]; - -snprintf (buf,sizeof (buf),"cpu%lu" , cpu); - d_cpu =debugfs_create_dir (buf, d_ext); - -debugfs_create_file ("runtime" ,0644 , d_cpu, (void *) cpu, &ext_server_runtime_fops); -debugfs_create_file ("period" ,0644 , d_cpu, (void *) cpu, &ext_server_period_fops); - } -} -#endif /* CONFIG_SCHED_CLASS_EXT */ -static __init int sched_init_debug (void ) {struct dentry __maybe_unused *numa, *llc;
Peter Zijlstra
maintainerPATCH v2 4/10 sched/fair: Add cgroup_mode switch
In reply to Peter ZijlstraSince calc_group_shares() has issues with 'many' CPUs, specifically the computed shares value gets to be roughly 1/nr_cpus, prepare to add a few alternative methods.
--- kernel/sched/debug.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 2 files changed, 75 insertions(+)
kernel/sched/debug.c +74 -0
@@ -588,6 +588,76 @@ static void debugfs_fair_server_init(voi } } +#ifdef CONFIG_FAIR_GROUP_SCHED +int cgroup_mode =0 ; + +static const char *cgroup_mode_str[] = { +"smp" , +}; + +static int sched_cgroup_mode (const char *str) +{ +for (int i =0 ; i <ARRAY_SIZE (cgroup_mode_str); i++) { +if (!strcmp (str, cgroup_mode_str[i])) +return i; + } +return -EINVAL ; +} + +static ssize_t sched_cgroup_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) +{ +char buf[16 ]; +int mode; + +if (cnt >15 ) + cnt =15 ; + +if (copy_from_user (buf, ubuf, cnt)) +return -EFAULT ; + + buf[cnt] =0 ; + mode =sched_cgroup_mode (strstrip (buf)); +if (mode <0 ) +return mode; + +WRITE_ONCE (cgroup_mode, mode); + + *ppos += cnt; +return cnt; +} + +static int sched_cgroup_show (struct seq_file *m,void *v) +{ +int mode =READ_ONCE (cgroup_mode); + +for (int i =0 ; i <ARRAY_SIZE (cgroup_mode_str); i++) { +if (mode == i) +seq_puts (m,"(" ); +seq_puts (m, cgroup_mode_str[i]); +if (mode == i) +seq_puts (m,")" ); + +seq_puts (m," " ); + } +seq_puts (m,"\n" ); +return 0 ; +} + +static int sched_cgroup_open (struct inode *inode,struct file *filp) +{ +return single_open (filp, sched_cgroup_show,NULL ); +} + +static const struct file_operations sched_cgroup_fops = { + .open = sched_cgroup_open, + .write = sched_cgroup_write, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +#endif +static __init int sched_init_debug (void ) {struct dentry __maybe_unused *numa;
@@ -625,6 +695,10 @@ static __init int sched_init_debug(void)debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops); +#ifdef CONFIG_FAIR_GROUP_SCHED +debugfs_create_file ("cgroup_mode" ,0644 , debugfs_sched,NULL , &sched_cgroup_fops); +#endif +debugfs_fair_server_init ();#ifdef CONFIG_SCHED_CLASS_EXT debugfs_ext_server_init ();
kernel/sched/sched.h +1 -0
@@ -565,6 +565,7 @@ static inline struct task_group *css_tg(extern int tg_nop (struct task_group *tg,void *data);#ifdef CONFIG_FAIR_GROUP_SCHED +extern int cgroup_mode;extern void free_fair_sched_group (struct task_group *tg);extern int alloc_fair_sched_group (struct task_group *tg,struct task_group *parent);extern void online_fair_sched_group (struct task_group *tg);
Peter Zijlstra
maintainerPATCH v2 5/10 sched/fair: Add cgroup_mode: UP
In reply to Peter ZijlstraInstead of calculating the proportional fraction of tg->shares for each CPU, just give each CPU the full measure, ignoring these pesky SMP problems.
--- kernel/sched/debug.c | 3 ++- kernel/sched/fair.c | 21 ++++++++++++++++++++- 2 files changed, 22 insertions(+), 2 deletions(-)
kernel/sched/debug.c +2 -1
@@ -589,9 +589,10 @@ static void debugfs_fair_server_init(voi }#ifdef CONFIG_FAIR_GROUP_SCHED -int cgroup_mode =0 ; +int cgroup_mode =1 ;static const char *cgroup_mode_str[] = { +"up" ,"smp" , };
kernel/sched/fair.c +20 -1
@@ -4150,7 +4150,7 @@ static inline int throttled_hierarchy(st* * hence icky! */ -static long calc_group_shares (struct cfs_rq *cfs_rq) +static long calc_smp_shares (struct cfs_rq *cfs_rq) {long tg_weight, tg_shares, load, shares;struct task_group *tg = cfs_rq->tg;
@@ -4185,6 +4185,25 @@ static long calc_group_shares(struct cfs }/* +* Ignore this pesky SMP stuff, use (4). +*/ +static long calc_up_shares (struct cfs_rq *cfs_rq) +{ +struct task_group *tg = cfs_rq->tg; +return READ_ONCE (tg->shares); +} + +static long calc_group_shares (struct cfs_rq *cfs_rq) +{ +int mode =READ_ONCE (cgroup_mode); + +if (mode ==0 ) +return calc_up_shares (cfs_rq); + +return calc_smp_shares (cfs_rq); +} + +/* * Recomputes the group entity based on the current state of its group * runqueue. */
Peter Zijlstra
maintainerPATCH v2 6/10 sched/fair: Add cgroup_mode: MAX
In reply to Peter ZijlstraIn order to avoid the CPU shares becoming tiny '1 / nr_cpus', assume each cgroup is maximally concurrent and distrubute 'nr_cpus * tg->shares', such that each CPU ends up with a 'tg->shares' sized fraction (on average).
There is the corner case, when a cgroup is minimally loaded, eg a single spinner, therefore limit the CPU shares to that of a nice -20 task to avoid getting too much load.
It was previously suggested to allow raising cpu.weight to '100 * nr_cpus' to combat this same problem, but the problem there is the above corner case, allowing multiple cgroups with such immense weight to the runqueue has significant problems.
It would drown the kthreads, but it also risks overflowing the load values.
--- include/linux/cpuset.h | 6 +++++ kernel/cgroup/cpuset.c | 15 ++++++++++++++ kernel/sched/debug.c | 1 kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++++++++----- 4 files changed, 69 insertions(+), 5 deletions(-)
include/linux/cpuset.h +6 -0
@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_hextern void cpuset_cpus_allowed_locked (struct task_struct *p,struct cpumask *mask);extern void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask);extern bool cpuset_cpus_allowed_fallback (struct task_struct *p); +extern int cpuset_num_cpus (struct cgroup *cgroup);extern nodemask_t cpuset_mems_allowed (struct task_struct *p);#define cpuset_current_mems_allowed (current->mems_allowed)void cpuset_init_current_mems_allowed (void );
@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_freturn false; } +static inline int cpuset_num_cpus (struct cgroup *cgroup) +{ +return num_online_cpus (); +} +static inline nodemask_t cpuset_mems_allowed (struct task_struct *p) {return node_possible_map;
kernel/cgroup/cpuset.c +15 -0
@@ -4100,6 +4100,21 @@ bool cpuset_cpus_allowed_fallback(structreturn changed; } +int cpuset_num_cpus (struct cgroup *cgrp) +{ +int nr =num_online_cpus (); +struct cpuset *cs; + +if (is_in_v2_mode ()) { +guard (rcu)(); + cs =css_cs (cgroup_e_css (cgrp, &cpuset_cgrp_subsys)); +if (cs) + nr =cpumask_weight (cs->effective_cpus); + } + +return nr; +} +void __init cpuset_init_current_mems_allowed (void ) {nodes_setall (current->mems_allowed);
kernel/sched/debug.c +1 -0
@@ -594,6 +594,7 @@ int cgroup_mode = 1;static const char *cgroup_mode_str[] = {"up" ,"smp" , +"max" , };static int sched_cgroup_mode (const char *str)
kernel/sched/fair.c +47 -5
72 lines not on this page · raw message
Peter Zijlstra
maintainerPATCH v2 7/10 sched/fair: Add cgroup_mode: CONCUR
In reply to Peter ZijlstraA variation of MAX; where instead of assuming maximal concurrent, this scales with 'min(nr_tasks, nr_cpus)'. This handles the low concurrency cases more gracefully, with the exception of CPU affnity.
Note: the tracking of tg->tasks is somewhat expensive :-/
--- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 39 ++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 3 +++ 3 files changed, 40 insertions(+), 3 deletions(-)
kernel/sched/debug.c +1 -0
@@ -594,6 +594,7 @@ int cgroup_mode = 1;static const char *cgroup_mode_str[] = {"up" ,"smp" , +"concur" ,"max" , };
kernel/sched/fair.c +36 -3
@@ -4211,6 +4211,30 @@ static long calc_max_shares(struct cfs_rreturn __calc_smp_shares (cfs_rq, tg_shares * nr, max_shares); } +static inline int tg_tasks (struct task_group *tg) +{ +return max (1 ,atomic_long_read (&tg->tasks)); +} + +/* +* Func: min(fraction(num * tg->shares), nice -20); where +* num = min(nr_tasks, nr_cpus) +* +* Similar to max, except scale with min(nr_tasks, nr_cpus), which gives +* a far more natural distrubution. Can still create edge case using CPU +* affinity. +*/ +static long calc_concur_shares (struct cfs_rq *cfs_rq) +{ +struct task_group *tg = cfs_rq->tg; +int nr_cpus =tg_cpus (tg); +int nr_tasks =tg_tasks (tg); +int nr =min (nr_tasks, nr_cpus); +long tg_shares =READ_ONCE (tg->shares); +long max_shares =scale_load (sched_prio_to_weight[0 ]); +return __calc_smp_shares (cfs_rq, nr * tg_shares, max_shares); +} +/* * Func: fraction(tg->shares) *
@@ -4240,6 +4264,9 @@ static long calc_group_shares(struct cfsreturn calc_up_shares (cfs_rq);if (mode ==2 ) +return calc_concur_shares (cfs_rq); + +if (mode ==3 )return calc_max_shares (cfs_rq);return calc_smp_shares (cfs_rq);
@@ -4385,7 +4412,7 @@ static inline bool cfs_rq_is_decayed(str*/ static inline void update_tg_load_avg (struct cfs_rq *cfs_rq) { -long delta; +long delta, dt;u64 now;/*
@@ -4407,16 +4434,19 @@ static inline void update_tg_load_avg(streturn ; delta = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib; -if (abs (delta) > cfs_rq->tg_load_avg_contrib /64 ) { + dt = cfs_rq->h_nr_queued - cfs_rq->tg_tasks_contrib; +if (dt ||abs (delta) > cfs_rq->tg_load_avg_contrib /64 ) {atomic_long_add (delta, &cfs_rq->tg->load_avg); +atomic_long_add (dt, &cfs_rq->tg->tasks); cfs_rq->tg_load_avg_contrib = cfs_rq->avg.load_avg; + cfs_rq->tg_tasks_contrib = cfs_rq->h_nr_queued; cfs_rq->last_update_tg_load_avg = now; } }static inline void clear_tg_load_avg (struct cfs_rq *cfs_rq) { -long delta; +long delta, dt;u64 now;/*
@@ -4427,8 +4457,11 @@ static inline void clear_tg_load_avg(str now =sched_clock_cpu (cpu_of (rq_of (cfs_rq))); delta =0 - cfs_rq->tg_load_avg_contrib; + dt =0 - cfs_rq->tg_tasks_contrib;atomic_long_add (delta, &cfs_rq->tg->load_avg); +atomic_long_add (dt, &cfs_rq->tg->tasks); cfs_rq->tg_load_avg_contrib =0 ; + cfs_rq->tg_tasks_contrib =0 ; cfs_rq->last_update_tg_load_avg = now; }
kernel/sched/sched.h +3 -0
@@ -491,6 +491,8 @@ struct task_group {* will also be accessed at each tick. */ atomic_long_t load_avg____cacheline_aligned ; +atomic_long_t tasks; +#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED
@@ -720,6 +722,7 @@ struct cfs_rq {#ifdef CONFIG_FAIR_GROUP_SCHED u64 last_update_tg_load_avg;unsigned long tg_load_avg_contrib; +unsigned long tg_tasks_contrib;long propagate;long prop_runnable_sum;
Peter Zijlstra
maintainerPATCH v2 8/10 sched/fair: Add newidle balance to pick_task_fair()
In reply to Peter ZijlstraWith commit 50653216e4ff ("sched: Add support to pick functions to take rf") removing the balance callback, the pick_task() callback is in charge of newidle balancing.
This means pick_task_fair() should do so too. This hasn't been a problem in practise because pick_next_task_fair() is used. However, since we'll be removing that one shortly, make sure pick_next_task() is up to scratch.
--- kernel/sched/fair.c | 38 +++++++++++++++----------------------- 1 file changed, 15 insertions(+), 23 deletions(-)
kernel/sched/fair.c +15 -23
@@ -9215,16 +9215,18 @@ static void wakeup_preempt_fair(struct r }static struct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf) +__must_hold (__rq_lockp (rq)) {struct sched_entity *se;struct cfs_rq *cfs_rq;struct task_struct *p;bool throttled; +int new_tasks; again: cfs_rq = &rq->cfs;if (!cfs_rq->nr_queued) -return NULL ; + goto idle; throttled = false;
@@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_faiif (unlikely (throttled))task_throttle_setup_work (p);return p; + +idle: + new_tasks =sched_balance_newidle (rq, rf); +if (new_tasks <0 ) +return RETRY_TASK ; +if (new_tasks >0 ) + goto again; +return NULL ; }static void __set_next_task_fair (struct rq *rq,struct task_struct *p,bool first);
@@ -9256,12 +9266,12 @@ pick_next_task_fair(struct rq *rq, struc {struct sched_entity *se;struct task_struct *p; -int new_tasks; -again: p =pick_task_fair (rq, rf); +if (unlikely (p ==RETRY_TASK )) +return p;if (!p) - goto idle; +return p; se = &p->se;#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9311,29 +9321,11 @@ pick_next_task_fair(struct rq *rq, struc#endif /* CONFIG_FAIR_GROUP_SCHED */ put_prev_set_next_task (rq, prev, p);return p; - -idle: -if (rf) { - new_tasks =sched_balance_newidle (rq, rf); - -/* -* Because sched_balance_newidle() releases (and re-acquires) -* rq->lock, it is possible for any higher priority task to -* appear. In that case we must re-start the pick_next_entity() -* loop. -*/ -if (new_tasks <0 ) -return RETRY_TASK ; - -if (new_tasks >0 ) - goto again; - } - -return NULL ; }static struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se,struct rq_flags *rf) + __must_hold(__rq_lockp(dl_se->rq)) {return pick_task_fair (dl_se->rq, rf); }
K Prateek Nayak
reviewerRe: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
Hello Peter,
kernel/sched/fair.c static struct task_struct *pick_task_fai · line 9258
Quoted 13 lines · Peter Zijlstra
kernel/sched/fair.c +8
-0@@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_faiif (unlikely (throttled))task_throttle_setup_work (p);return p; + +idle: + new_tasks =sched_balance_newidle (rq, rf); +if (new_tasks <0 ) +return RETRY_TASK ; +if (new_tasks >0 ) + goto again; +return NULL ; }
For core scheduling will now trigger a newidle balance during the pick when core_cookie is reset to 0 which can cause tasks to migrate only for them to find they cannot run on the CPU since core-wide selection leads to a cookie mismatch and it is kept hanging there.
Can we return early if sched_core_enabled() here or are the additional newidle balance okay?
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerRe: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
Quoted 22 lines · K Prateek Nayak
Hello Peter,
kernel/sched/fair.c static struct task_struct *pick_task_fai · line 9258
Quoted 13 lines · Peter Zijlstra
kernel/sched/fair.c +8
-0@@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_faiif (unlikely (throttled))task_throttle_setup_work (p);return p; + +idle: + new_tasks =sched_balance_newidle (rq, rf); +if (new_tasks <0 ) +return RETRY_TASK ; +if (new_tasks >0 ) + goto again; +return NULL ; }For core scheduling will now trigger a newidle balance during the pick when
core_cookieis reset to 0 which can cause tasks to migrate only for them to find they cannot run on the CPU since core-wide selection leads to a cookie mismatch and it is kept hanging there.
Quoted 2 lines
Can we return early if
sched_core_enabled()here or are the additional newidle balance okay?
This basically makes fair behave like every other class, so in that sense this is probably okay. That said, fair is the most common case, so perhaps.
Lets see if the people actually using this notice first though ;-)
Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
In reply to Peter ZijlstraQuoted 11 lines · Peter Zijlstra
With commit
50653216e4ff("sched: Add support to pick functions to take rf") removing the balance callback, thepick_task()callback is in charge of newidle balancing.This means
pick_task_fair()should do so too. This hasn't been a problem in practise becausepick_next_task_fair()is used. However, since we'll be removing that one shortly, make surepick_next_task()is up to scratch.Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Quoted 90 lines
--- kernel/sched/fair.c | 38 +++++++++++++++----------------------- 1 file changed, 15 insertions(+), 23 deletions(-)kernel/sched/fair.c +15
-23@@ -9215,16 +9215,18 @@ static void wakeup_preempt_fair(struct r }static struct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf) +__must_hold (__rq_lockp (rq)) {struct sched_entity *se;struct cfs_rq *cfs_rq;struct task_struct *p;bool throttled; +int new_tasks; again: cfs_rq = &rq->cfs;if (!cfs_rq->nr_queued) -return NULL ; + goto idle; throttled = false;@@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_faiif (unlikely (throttled))task_throttle_setup_work (p);return p; + +idle: + new_tasks =sched_balance_newidle (rq, rf); +if (new_tasks <0 ) +return RETRY_TASK ; +if (new_tasks >0 ) + goto again; +return NULL ; }static void __set_next_task_fair (struct rq *rq,struct task_struct *p,bool first);@@ -9256,12 +9266,12 @@ pick_next_task_fair(struct rq *rq, struc {struct sched_entity *se;struct task_struct *p; -int new_tasks; -again: p =pick_task_fair (rq, rf); +if (unlikely (p ==RETRY_TASK )) +return p;if (!p) - goto idle; +return p; se = &p->se;#ifdef CONFIG_FAIR_GROUP_SCHED @@ -9311,29 +9321,11 @@ pick_next_task_fair(struct rq *rq, struc#endif /* CONFIG_FAIR_GROUP_SCHED */ put_prev_set_next_task (rq, prev, p);return p; - -idle: -if (rf) { - new_tasks =sched_balance_newidle (rq, rf); - -/* -* Because sched_balance_newidle() releases (and re-acquires) -* rq->lock, it is possible for any higher priority task to -* appear. In that case we must re-start the pick_next_entity() -* loop. -*/ -if (new_tasks <0 ) -return RETRY_TASK ; - -if (new_tasks >0 ) - goto again; - } - -return NULL ; }static struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se,struct rq_flags *rf) + __must_hold(__rq_lockp(dl_se->rq)) {return pick_task_fair (dl_se->rq, rf); }
[tip: sched/core] sched/fair: Add newidle balance to pick_task_fair()
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b3a2dfa8b42e5b97dd144aa59374f4e045725cac Gitweb: https://git.kernel.org/tip/b3a2dfa8b42e5b97dd144aa59374f4e045725cac Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 11 May 2026 13:31:12 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 26 May 2026 13:53:13 +02:00
sched/fair: Add newidle balance to pick_task_fair()
With commit 50653216e4ff ("sched: Add support to pick functions to take rf") removing the balance callback, the pick_task() callback is in charge of newidle balancing.
This means pick_task_fair() should do so too. This hasn't been a problem in practise because pick_next_task_fair() is used. However, since we'll be removing that one shortly, make sure pick_next_task() is up to scratch.
--- kernel/sched/fair.c | 38 +++++++++++++++----------------------- 1 file changed, 15 insertions(+), 23 deletions(-)
kernel/sched/fair.c +15 -23
@@ -9863,16 +9863,18 @@ preempt: }static struct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf) +__must_hold (__rq_lockp (rq)) {struct sched_entity *se;struct cfs_rq *cfs_rq;struct task_struct *p;bool throttled; +int new_tasks; again: cfs_rq = &rq->cfs;if (!cfs_rq->nr_queued) -return NULL ; + goto idle; throttled = false;
@@ -9893,6 +9895,14 @@ again:if (unlikely (throttled))task_throttle_setup_work (p);return p; + +idle: + new_tasks =sched_balance_newidle (rq, rf); +if (new_tasks <0 ) +return RETRY_TASK ; +if (new_tasks >0 ) + goto again; +return NULL ; }static void __set_next_task_fair (struct rq *rq,struct task_struct *p,bool first);
@@ -9904,12 +9914,12 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf {struct sched_entity *se;struct task_struct *p; -int new_tasks; -again: p =pick_task_fair (rq, rf); +if (unlikely (p ==RETRY_TASK )) +return p;if (!p) - goto idle; +return p; se = &p->se;#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9959,29 +9969,11 @@ simple:#endif /* CONFIG_FAIR_GROUP_SCHED */ put_prev_set_next_task (rq, prev, p);return p; - -idle: -if (rf) { - new_tasks =sched_balance_newidle (rq, rf); - -/* -* Because sched_balance_newidle() releases (and re-acquires) -* rq->lock, it is possible for any higher priority task to -* appear. In that case we must re-start the pick_next_entity() -* loop. -*/ -if (new_tasks <0 ) -return RETRY_TASK ; - -if (new_tasks >0 ) - goto again; - } - -return NULL ; }static struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se,struct rq_flags *rf) + __must_hold(__rq_lockp(dl_se->rq)) {return pick_task_fair (dl_se->rq, rf); }
Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
In reply to Peter ZijlstraHi Peter,
Quoted 8 lines · Peter Zijlstra
With commit
50653216e4ff("sched: Add support to pick functions to take rf") removing the balance callback, thepick_task()callback is in charge of newidle balancing.This means
pick_task_fair()should do so too. This hasn't been a problem in practise becausepick_next_task_fair()is used. However, since we'll be removing that one shortly, make surepick_next_task()is up to scratch.
While testing Prateek's throttle series, I noticed a panic issue when coresched is enabled and bisected to this patch.
I fed the panic log and this patch to an agent and its analysis looks correct to me(cpu56 and cpu57 are siblings in a VM):
cpu57 (holds core-wide lock)
pick_next_task() [core scheduling]
for_each_cpu_wrap(i, smt_mask, 57):
i=57: pick_task(rq_57)
pick_task_fair(rq_57)
-> picks task A
rq_57->core_pick = task A
// task_rq(A) == rq_57
i=56: pick_task(rq_56)
pick_task_fair(rq_56)
cfs_rq->nr_queued == 0
goto idle
sched_balance_newidle(rq_56)
raw_spin_rq_unlock(rq_56)
// core-wide lock released
newidle_balance() pulls
task A: rq_57 -> rq_56
// task_rq(A) == rq_56 now
raw_spin_rq_lock(rq_56)
// core-wide lock re-acquired
return > 0
goto again
pick_task_fair(rq_56)
-> picks task A
rq_56->core_pick = task A
// first loop done
// rq_57->core_pick is still task A (set before lock release)
// but task_rq(A) == rq_56 now
next = rq_57->core_pick // = task A
put_prev_set_next_task(rq_57, prev, task A)
__set_next_task_fair(rq_57, task A)
hrtick_start_fair(rq_57, task A)
WARN_ON_ONCE(task_rq(task A) != rq_57)
// task_rq(A) == rq_56I applied below diff and the problem is gone:
kernel/sched/fair.c +3 -0
@@ -9897,6 +9897,9 @@ static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)return p; idle: +if (sched_core_enabled (rq)) +return NULL ; + new_tasks =sched_balance_newidle (rq, rf);if (new_tasks <0 )return RETRY_TASK ;
Full dmesg of the panic is attached for your reference.
Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
Aaron,
Sorry I failed to notice this email earlier.
Quoted 8 lines · Aaron Lu
I applied below diff and the problem is gone:
kernel/sched/fair.c +0
-0
kernel/sched/fair.c static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) · line 9905
Quoted 9 lines
kernel/sched/fair.c +3
-0@@ -9897,6 +9897,9 @@ static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)return p; idle: +if (sched_core_enabled (rq)) +return NULL ; + new_tasks =sched_balance_newidle (rq, rf);if (new_tasks <0 )return RETRY_TASK ;
Right, this is the safe patch and restores pick_task_fair() to its previous status (for core-sched).
Since people are hitting this problem, I'm going to merge it as below. I've presumed your SoB, please let me know if that's a problem.
I think I'm going to try and move newidle into sched_class::balance / balance_fair(), but I'll do that next cycle.
Thanks!
---
Subject: sched/fair: Fix newidle vs core-sched
From: "Aaron Lu" <[email protected]>
Date: Wed, 3 Jun 2026 17:51:08 +0800
From: "Aaron Lu" <[email protected]>
While testing Prateek's throttle series, I noticed a panic issue when coresched is enabled and bisected to this patch.
I fed the panic log and this patch to an agent and its analysis looks correct to me(cpu56 and cpu57 are siblings in a VM):
cpu57 (holds core-wide lock)
pick_next_task() [core scheduling]
for_each_cpu_wrap(i, smt_mask, 57):
i=57: pick_task(rq_57)
pick_task_fair(rq_57)
-> picks task A
rq_57->core_pick = task A
// task_rq(A) == rq_57
i=56: pick_task(rq_56)
pick_task_fair(rq_56)
cfs_rq->nr_queued == 0
goto idle
sched_balance_newidle(rq_56)
raw_spin_rq_unlock(rq_56)
// core-wide lock released
newidle_balance() pulls
task A: rq_57 -> rq_56
// task_rq(A) == rq_56 now
raw_spin_rq_lock(rq_56)
// core-wide lock re-acquired
return > 0
goto again
pick_task_fair(rq_56)
-> picks task A
rq_56->core_pick = task A
// first loop done
// rq_57->core_pick is still task A (set before lock release)
// but task_rq(A) == rq_56 now
next = rq_57->core_pick // = task A
put_prev_set_next_task(rq_57, prev, task A)
__set_next_task_fair(rq_57, task A)
hrtick_start_fair(rq_57, task A)
WARN_ON_ONCE(task_rq(task A) != rq_57)
// task_rq(A) == rq_56IOW: by allowing pick_task_fair() to do newidle_balance and not returning RETRY_TASK, it can end up selecting the same task on two CPUs. Restore the previous state by never doing newidle when core scheduling is enabled.
--- kernel/sched/fair.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/sched/fair.c +3 -0
@@ -9942,6 +9942,9 @@ struct task_struct *pick_task_fair(strucreturn p; idle: +if (sched_core_enabled (rq)) +return NULL ; + new_tasks =sched_balance_newidle (rq, rf);if (new_tasks <0 )return RETRY_TASK ;
Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
Hi Peter,
Quoted 5 lines · Peter Zijlstra
Aaron,
Sorry I failed to notice this email earlier.
Never mind.
Quoted 23 lines · Aaron Lu
Quoted 9 lines
I applied below diff and the problem is gone:
kernel/sched/fair.c +0
-0kernel/sched/fair.c static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) · line 9905
Quoted 9 lines
kernel/sched/fair.c +3
-0@@ -9897,6 +9897,9 @@ static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)return p; idle: +if (sched_core_enabled (rq)) +return NULL ; + new_tasks =sched_balance_newidle (rq, rf);if (new_tasks <0 )return RETRY_TASK ;Right, this is the safe patch and restores
pick_task_fair()to its previous status (for core-sched).
Quoted 2 lines
Since people are hitting this problem, I'm going to merge it as below. I've presumed your SoB, please let me know if that's a problem.
No problem.
Quoted 3 lines
I think I'm going to try and move newidle into
sched_class::balance/balance_fair(), but I'll do that next cycle.
I'll surely test it then.
Best regards,
Aaron
[tip: sched/core] sched/fair: Fix newidle vs core-sched
In reply to Aaron LuThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: c095741713d1bc317b53e2da2b222e7448b6021f Gitweb: https://git.kernel.org/tip/c095741713d1bc317b53e2da2b222e7448b6021f Author: Aaron Lu <[email protected]> AuthorDate: Wed, 03 Jun 2026 17:51:08 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 11 Jun 2026 13:41:23 +02:00
sched/fair: Fix newidle vs core-sched
While testing Prateek's throttle series, I noticed a panic issue when coresched is enabled and bisected to this patch.
I fed the panic log and this patch to an agent and its analysis looks correct to me(cpu56 and cpu57 are siblings in a VM):
cpu57 (holds core-wide lock)
pick_next_task() [core scheduling]
for_each_cpu_wrap(i, smt_mask, 57):
i=57: pick_task(rq_57)
pick_task_fair(rq_57)
-> picks task A
rq_57->core_pick = task A
// task_rq(A) == rq_57
i=56: pick_task(rq_56)
pick_task_fair(rq_56)
cfs_rq->nr_queued == 0
goto idle
sched_balance_newidle(rq_56)
raw_spin_rq_unlock(rq_56)
// core-wide lock released
newidle_balance() pulls
task A: rq_57 -> rq_56
// task_rq(A) == rq_56 now
raw_spin_rq_lock(rq_56)
// core-wide lock re-acquired
return > 0
goto again
pick_task_fair(rq_56)
-> picks task A
rq_56->core_pick = task A
// first loop done
// rq_57->core_pick is still task A (set before lock release)
// but task_rq(A) == rq_56 now
next = rq_57->core_pick // = task A
put_prev_set_next_task(rq_57, prev, task A)
__set_next_task_fair(rq_57, task A)
hrtick_start_fair(rq_57, task A)
WARN_ON_ONCE(task_rq(task A) != rq_57)
// task_rq(A) == rq_56IOW: by allowing pick_task_fair() to do newidle_balance and not returning RETRY_TASK, it can end up selecting the same task on two CPUs. Restore the previous state by never doing newidle when core scheduling is enabled.
--- kernel/sched/fair.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/sched/fair.c +3 -0
@@ -9942,6 +9942,9 @@ again:return p; idle: +if (sched_core_enabled (rq)) +return NULL ; + new_tasks =sched_balance_newidle (rq, rf);if (new_tasks <0 )return RETRY_TASK ;
Peter Zijlstra
maintainerPATCH v2 9/10 sched: Remove sched_class::pick_next_task()
In reply to Peter ZijlstraThe reason for pick_next_task_fair() is the put/set optimization that avoids touching the common ancestors. However, it is possible to implement this in the put_prev_task() and set_next_task() calls as used in put_prev_set_next_task().
Notably, put_prev_set_next_task() is the only site that:
- calls put_prev_task() with a .next argument;
- calls set_next_task() with .first = true.
This means that put_prev_task() can determine the common hierarchy and stop there, and then set_next_task() can terminate where put_prev_task stopped.
--- kernel/sched/core.c | 27 +++------ kernel/sched/fair.c | 139 +++++++++++++++++---------------------------------- kernel/sched/sched.h | 14 ----- 3 files changed, 57 insertions(+), 123 deletions(-)
kernel/sched/core.c +9 -18
@@ -5980,16 +5980,15 @@ __pick_next_task(struct rq *rq, struct tif (likely (!sched_class_above (prev->sched_class, &fair_sched_class) && rq->nr_running == rq->cfs.h_nr_queued)) { - p =pick_next_task_fair (rq, prev, rf); + p =pick_task_fair (rq, rf);if (unlikely (p ==RETRY_TASK )) goto restart;/* Assume the next prioritized class is idle_sched_class */ -if (!p) { +if (!p) p =pick_task_idle (rq, rf); -put_prev_set_next_task (rq, prev, p); - } +put_prev_set_next_task (rq, prev, p);return p; }
@@ -5997,20 +5996,12 @@ __pick_next_task(struct rq *rq, struct tprev_balance (rq, prev, rf);for_each_active_class (class) { -if (class->pick_next_task) { - p = class->pick_next_task (rq, prev, rf); -if (unlikely (p ==RETRY_TASK )) - goto restart; -if (p) -return p; - }else { - p = class->pick_task (rq, rf); -if (unlikely (p ==RETRY_TASK )) - goto restart; -if (p) { -put_prev_set_next_task (rq, prev, p); -return p; - } + p = class->pick_task (rq, rf); +if (unlikely (p ==RETRY_TASK )) + goto restart; +if (p) { +put_prev_set_next_task (rq, prev, p); +return p; } }
kernel/sched/fair.c +47 -92
180 lines not on this page · raw message
kernel/sched/sched.h +1 -13
@@ -2555,17 +2555,6 @@ struct sched_class {* schedule/pick_next_task: rq->lock */ struct task_struct *(*pick_task)(struct rq *rq,struct rq_flags *rf); -/* -* Optional! When implemented pick_next_task() should be equivalent to: -* -* next = pick_task(); -* if (next) { -* put_prev_task(prev); -* set_next_task_first(next); -* } -*/ -struct task_struct *(*pick_next_task)(struct rq *rq,struct task_struct *prev, -struct rq_flags *rf);/* * sched_change:
@@ -2789,8 +2778,7 @@ static inline bool sched_fair_runnable(sreturn rq->cfs.nr_queued >0 ; } -extern struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev, -struct rq_flags *rf); +extern struct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf);extern struct task_struct *pick_task_idle (struct rq *rq,struct rq_flags *rf);#define SCA_CHECK 0x01
Re: [PATCH v2 09/10] sched: Remove sched_class::pick_next_task()
Quoted 8 lines · Peter Zijlstra
The reason for
pick_next_task_fair()is the put/set optimization that avoids touching the common ancestors. However, it is possible to implement this in theput_prev_task()andset_next_task()calls as used input_prev_set_next_task().Notably,
put_prev_set_next_task()is the only site that:
Quoted 8 lines
- calls
put_prev_task()with a .next argument;
- callsset_next_task()with .first = true.This means that
put_prev_task()can determine the common hierarchy and stop there, and thenset_next_task()can terminate whereput_prev_taskstopped.Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Hackbench results on my Arm64 dev machine stay similars with patch 8 and 9 (unlike patch 10)
Quoted 275 lines
--- kernel/sched/core.c | 27 +++------ kernel/sched/fair.c | 139 +++++++++++++++++---------------------------------- kernel/sched/sched.h | 14 ----- 3 files changed, 57 insertions(+), 123 deletions(-)kernel/sched/core.c +9
-18@@ -5980,16 +5980,15 @@ __pick_next_task(struct rq *rq, struct tif (likely (!sched_class_above (prev->sched_class, &fair_sched_class) && rq->nr_running == rq->cfs.h_nr_queued)) { - p =pick_next_task_fair (rq, prev, rf); + p =pick_task_fair (rq, rf);if (unlikely (p ==RETRY_TASK )) goto restart;/* Assume the next prioritized class is idle_sched_class */ -if (!p) { +if (!p) p =pick_task_idle (rq, rf); -put_prev_set_next_task (rq, prev, p); - } +put_prev_set_next_task (rq, prev, p);return p; }@@ -5997,20 +5996,12 @@ __pick_next_task(struct rq *rq, struct tprev_balance (rq, prev, rf);for_each_active_class (class) { -if (class->pick_next_task) { - p = class->pick_next_task (rq, prev, rf); -if (unlikely (p ==RETRY_TASK )) - goto restart; -if (p) -return p; - }else { - p = class->pick_task (rq, rf); -if (unlikely (p ==RETRY_TASK )) - goto restart; -if (p) { -put_prev_set_next_task (rq, prev, p); -return p; - } + p = class->pick_task (rq, rf); +if (unlikely (p ==RETRY_TASK )) + goto restart; +if (p) { +put_prev_set_next_task (rq, prev, p); +return p; } }kernel/sched/fair.c +47
-92@@ -9214,7 +9214,7 @@ static void wakeup_preempt_fair(struct rresched_curr_lazy (rq); } -static struct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf) +struct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf)__must_hold (__rq_lockp (rq)) {struct sched_entity *se;@@ -9257,72 +9257,6 @@ static struct task_struct *pick_task_faireturn NULL ; } -static void __set_next_task_fair (struct rq *rq,struct task_struct *p,bool first); -static void set_next_task_fair (struct rq *rq,struct task_struct *p,bool first); - -struct task_struct * -pick_next_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) -__must_hold (__rq_lockp (rq)) -{ -struct sched_entity *se; -struct task_struct *p; - - p =pick_task_fair (rq, rf); -if (unlikely (p ==RETRY_TASK )) -return p; -if (!p) -return p; - se = &p->se; - -#ifdef CONFIG_FAIR_GROUP_SCHED -if (prev->sched_class != &fair_sched_class) - goto simple; - -__put_prev_set_next_dl_server (rq, prev, p); - -/* -* Because of the set_next_buddy() in dequeue_task_fair() it is rather -* likely that a next task is from the same cgroup as the current. -* -* Therefore attempt to avoid putting and setting the entire cgroup -* hierarchy, only change the part that actually changes. -* -* Since we haven't yet done put_prev_entity and if the selected task -* is a different task than we started out with, try and touch the -* least amount of cfs_rqs. -*/ -if (prev != p) { -struct sched_entity *pse = &prev->se; -struct cfs_rq *cfs_rq; - -while (!(cfs_rq =is_same_group (se, pse))) { -int se_depth = se->depth; -int pse_depth = pse->depth; - -if (se_depth <= pse_depth) { -put_prev_entity (cfs_rq_of (pse), pse); - pse =parent_entity (pse); - } -if (se_depth >= pse_depth) { -set_next_entity (cfs_rq_of (se), se, true); - se =parent_entity (se); - } - } - -put_prev_entity (cfs_rq, pse); -set_next_entity (cfs_rq, se, true); - -__set_next_task_fair (rq, p, true); - } - -return p; - -simple: -#endif /* CONFIG_FAIR_GROUP_SCHED */ -put_prev_set_next_task (rq, prev, p); -return p; -} -static struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se,struct rq_flags *rf) __must_hold(__rq_lockp(dl_se->rq))@@ -9346,10 +9280,33 @@ static void put_prev_task_fair(struct rq {struct sched_entity *se = &prev->se;struct cfs_rq *cfs_rq; +struct sched_entity *nse =NULL ; -for_each_sched_entity (se) { +#ifdef CONFIG_FAIR_GROUP_SCHED +if (next && next->sched_class == &fair_sched_class) + nse = &next->se; +#endif + +while (se) { cfs_rq =cfs_rq_of (se); -put_prev_entity (cfs_rq, se); +if (!nse || cfs_rq->curr) +put_prev_entity (cfs_rq, se); +#ifdef CONFIG_FAIR_GROUP_SCHED +if (nse) { +if (is_same_group (se, nse)) +break ; + +int d = nse->depth - se->depth; +if (d >=0 ) { +/* nse has equal or greater depth, ascend */ + nse =parent_entity (nse); +/* if nse is the deeper, do not ascend se */ +if (d >0 ) +continue ; + } + } +#endif + se =parent_entity (se); } }@@ -13896,10 +13853,30 @@ static void switched_to_fair(struct rq * } } -static void __set_next_task_fair (struct rq *rq,struct task_struct *p,bool first) +/* +* Account for a task changing its policy or group. +* +* This routine is mostly called to set cfs_rq->curr field when a task +* migrates between groups/classes. +*/ +static void set_next_task_fair (struct rq *rq,struct task_struct *p,bool first) {struct sched_entity *se = &p->se; +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +if (IS_ENABLED (CONFIG_FAIR_GROUP_SCHED ) && + first && cfs_rq->curr) +break ; + +set_next_entity (cfs_rq, se, first); +/* ensure bandwidth has been allocated on our new cfs_rq */ +account_cfs_rq_runtime (cfs_rq,0 ); + } + + se = &p->se; +if (task_on_rq_queued (p)) {/* * Move the next running task to the front of the list, so our @@ -13919,27 +13896,6 @@ static void __set_next_task_fair(structsched_fair_update_stop_tick (rq, p); } -/* -* Account for a task changing its policy or group. -* -* This routine is mostly called to set cfs_rq->curr field when a task -* migrates between groups/classes. -*/ -static void set_next_task_fair (struct rq *rq,struct task_struct *p,bool first) -{ -struct sched_entity *se = &p->se; - -for_each_sched_entity (se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); - -set_next_entity (cfs_rq, se, first); -/* ensure bandwidth has been allocated on our new cfs_rq */ -account_cfs_rq_runtime (cfs_rq,0 ); - } - -__set_next_task_fair (rq, p, first); -} -void init_cfs_rq (struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline =RB_ROOT_CACHED ;@@ -14251,7 +14207,6 @@ DEFINE_SCHED_CLASS(fair) = { .wakeup_preempt = wakeup_preempt_fair, .pick_task = pick_task_fair, - .pick_next_task = pick_next_task_fair, .put_prev_task = put_prev_task_fair, .set_next_task = set_next_task_fair,kernel/sched/sched.h +1
-13@@ -2555,17 +2555,6 @@ struct sched_class {* schedule/pick_next_task: rq->lock */ struct task_struct *(*pick_task)(struct rq *rq,struct rq_flags *rf); -/* -* Optional! When implemented pick_next_task() should be equivalent to: -* -* next = pick_task(); -* if (next) { -* put_prev_task(prev); -* set_next_task_first(next); -* } -*/ -struct task_struct *(*pick_next_task)(struct rq *rq,struct task_struct *prev, -struct rq_flags *rf);/* * sched_change: @@ -2789,8 +2778,7 @@ static inline bool sched_fair_runnable(sreturn rq->cfs.nr_queued >0 ; } -extern struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev, -struct rq_flags *rf); +extern struct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf);extern struct task_struct *pick_task_idle (struct rq *rq,struct rq_flags *rf);#define SCA_CHECK 0x01
[tip: sched/core] sched: Remove sched_class::pick_next_task()
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5ad278dd20bdf59714443894d7b3044471af97d0 Gitweb: https://git.kernel.org/tip/5ad278dd20bdf59714443894d7b3044471af97d0 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 11 May 2026 13:31:13 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 26 May 2026 13:53:14 +02:00
sched: Remove sched_class::pick_next_task()
The reason for pick_next_task_fair() is the put/set optimization that avoids touching the common ancestors. However, it is possible to implement this in the put_prev_task() and set_next_task() calls as used in put_prev_set_next_task().
Notably, put_prev_set_next_task() is the only site that:
- calls put_prev_task() with a .next argument;
- calls set_next_task() with .first = true.
This means that put_prev_task() can determine the common hierarchy and stop there, and then set_next_task() can terminate where put_prev_task stopped.
--- kernel/sched/core.c | 27 ++------ kernel/sched/fair.c | 139 ++++++++++++++---------------------------- kernel/sched/sched.h | 14 +---- 3 files changed, 57 insertions(+), 123 deletions(-)
kernel/sched/core.c +9 -18
@@ -6030,16 +6030,15 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)if (likely (!sched_class_above (prev->sched_class, &fair_sched_class) && rq->nr_running == rq->cfs.h_nr_queued)) { - p =pick_next_task_fair (rq, prev, rf); + p =pick_task_fair (rq, rf);if (unlikely (p ==RETRY_TASK )) goto restart;/* Assume the next prioritized class is idle_sched_class */ -if (!p) { +if (!p) p =pick_task_idle (rq, rf); -put_prev_set_next_task (rq, prev, p); - } +put_prev_set_next_task (rq, prev, p);return p; }
@@ -6047,20 +6046,12 @@ restart:prev_balance (rq, prev, rf);for_each_active_class (class) { -if (class->pick_next_task) { - p = class->pick_next_task (rq, prev, rf); -if (unlikely (p ==RETRY_TASK )) - goto restart; -if (p) -return p; - }else { - p = class->pick_task (rq, rf); -if (unlikely (p ==RETRY_TASK )) - goto restart; -if (p) { -put_prev_set_next_task (rq, prev, p); -return p; - } + p = class->pick_task (rq, rf); +if (unlikely (p ==RETRY_TASK )) + goto restart; +if (p) { +put_prev_set_next_task (rq, prev, p); +return p; } }
kernel/sched/fair.c +47 -92
180 lines not on this page · raw message
kernel/sched/sched.h +1 -13
@@ -2589,17 +2589,6 @@ struct sched_class {* schedule/pick_next_task: rq->lock */ struct task_struct *(*pick_task)(struct rq *rq,struct rq_flags *rf); -/* -* Optional! When implemented pick_next_task() should be equivalent to: -* -* next = pick_task(); -* if (next) { -* put_prev_task(prev); -* set_next_task_first(next); -* } -*/ -struct task_struct *(*pick_next_task)(struct rq *rq,struct task_struct *prev, -struct rq_flags *rf);/* * sched_change:
@@ -2823,8 +2812,7 @@ static inline bool sched_fair_runnable(struct rq *rq)return rq->cfs.nr_queued >0 ; } -extern struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev, -struct rq_flags *rf); +extern struct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf);extern struct task_struct *pick_task_idle (struct rq *rq,struct rq_flags *rf);#define SCA_CHECK 0x01
Peter Zijlstra
maintainerPATCH v2 10/10 sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraChange fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the shares_weight approximation are fully preserved.
That is, given a hierarchy like:
R
|
se--G1
/ \
G2--se se--G3
/ \ |T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:
_R_
/ | \T1 T2 T3
Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (se->h_load) is computed from the hierarchical fraction: se->load / cfs_rq->load.
Since EEVDF is now exclusive operating on rq->cfs, it needs to consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.
One place where additional information was required was set_next_task() / put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).
As a result of only having a single level to pick from, much of the complications in pick_next_task() and preemption go away.
Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.
TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr
--- include/linux/sched.h | 1 kernel/sched/core.c | 5 kernel/sched/debug.c | 9 kernel/sched/fair.c | 789 +++++++++++++++++++++----------------------------- kernel/sched/pelt.c | 6 kernel/sched/sched.h | 26 - 6 files changed, 366 insertions(+), 470 deletions(-)
include/linux/sched.h +1 -0
@@ -575,6 +575,7 @@ struct sched_statistics {struct sched_entity {/* For load-balancing: */ struct load_weight load; +struct load_weight h_load;struct rb_node run_node;u64 deadline;u64 min_vruntime;
kernel/sched/core.c +1 -4
@@ -5539,11 +5539,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);*/ static inline void prefetch_curr_exec_start (struct task_struct *p) { -#ifdef CONFIG_FAIR_GROUP_SCHED -struct sched_entity *curr = p->se.cfs_rq->curr; -#else struct sched_entity *curr =task_rq (p)->cfs.curr; -#endif +prefetch (curr);prefetch (&curr->exec_start); }
kernel/sched/debug.c +6 -3
@@ -911,10 +911,11 @@ print_task(struct seq_file *m, struct rqelse SEQ_printf (m," %c" ,task_state_to_char (p)); - SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", + SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), + p->se.h_load.weight, SPLIT_NS(p->se.vruntime), - entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', + entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice),
@@ -943,7 +944,7 @@ static void print_rq(struct seq_file *m,SEQ_printf (m,"\n" );SEQ_printf (m,"runnable tasks:\n" ); -SEQ_printf (m," S task PID vruntime eligible " +SEQ_printf (m," S task PID weight vruntime eligible " "deadline slice sum-exec switches " "prio wait-time sum-sleep sum-block" #ifdef CONFIG_NUMA_BALANCING
@@ -1051,6 +1052,8 @@ void print_cfs_rq(struct seq_file *m, in cfs_rq->tg_load_avg_contrib);SEQ_printf (m," .%-30s: %ld\n" ,"tg_load_avg" ,atomic_long_read (&cfs_rq->tg->load_avg)); +SEQ_printf (m," .%-30s: %lu\n" ,"h_load" , + cfs_rq->h_load);#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_CFS_BANDWIDTH SEQ_printf(m, " .%-30s: %d\n", "throttled",
kernel/sched/fair.c +342 -447
1,399 lines not on this page · raw message
kernel/sched/pelt.c +3 -3
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched/* * running is a subset of runnable (weight) so running can't be set if * runnable is clear. But there are some corner cases where the current -* se has been already dequeued but cfs_rq->curr still points to it. +* se has been already dequeued but cfs_rq->h_curr still points to it. * This means that weight will be 0 but not running for a sched_entity * but also for a cfs_rq if the latter becomes idle. As an example, * this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 nowint __update_load_avg_se (u64 now,struct cfs_rq *cfs_rq,struct sched_entity *se) {if (___update_load_sum (now, &se->avg, !!se->on_rq,se_runnable (se), - cfs_rq->curr == se)) { + cfs_rq->h_curr == se)) {___update_load_avg (&se->avg,se_weight (se));cfs_se_util_change (&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, stif (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), cfs_rq->h_nr_runnable, - cfs_rq->curr !=NULL )) { + cfs_rq->h_curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );trace_pelt_cfs_tp (cfs_rq);
kernel/sched/sched.h +13 -13
@@ -528,21 +528,8 @@ struct task_group { }; -#ifdef CONFIG_GROUP_SCHED_WEIGHT #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD -/* -* A weight of 0 or 1 can cause arithmetics problems. -* A weight of a cfs_rq is the sum of weights of which entities -* are queued on this cfs_rq, so a weight of a entity should not be -* too large, so as the shares value of a task group. -* (The default weight is 1024 - so there's no practical -* limitation from this.) -*/ -#define MIN_SHARES (1UL << 1) -#define MAX_SHARES (1UL << 18) -#endif -typedef int (*tg_visitor )(struct task_group *,void *); extern int walk_tg_tree_from(struct task_group *from,
@@ -629,6 +616,17 @@ static inline bool cfs_task_bw_constrain#endif /* !CONFIG_CGROUP_SCHED */ +/* +* A weight of 0 or 1 can cause arithmetics problems. +* A weight of a cfs_rq is the sum of weights of which entities +* are queued on this cfs_rq, so a weight of a entity should not be +* too large, so as the shares value of a task group. +* (The default weight is 1024 - so there's no practical +* limitation from this.) +*/ +#define MIN_SHARES (1UL << 1) +#define MAX_SHARES (1UL << 18) +extern void unregister_rt_sched_group (struct task_group *tg);extern void free_rt_sched_group (struct task_group *tg);extern int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent);
@@ -707,6 +705,7 @@ struct cfs_rq {/* * CFS load tracking */ +struct sched_entity *h_curr;struct sched_avg avg;#ifndef CONFIG_64BIT u64 last_update_time_copy;
@@ -2509,6 +2508,7 @@ extern const u32 sched_prio_to_wmult[40#define ENQUEUE_MIGRATED 0x00040000#define ENQUEUE_INITIAL 0x00080000#define ENQUEUE_RQ_SELECTED 0x00100000 +#define ENQUEUE_QUEUED 0x00200000#define RETRY_TASK ((void *)-1UL)
K Prateek Nayak
reviewerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Hello Peter,
Review static void wakeup_preempt_fair(struct r · line 9212
Quoted 9 lines · Peter Zijlstra
(quoted diff) +4
-1@@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r + se = pick_next_entity(rq, true); + if (!se) + goto again; p = task_of(se); - if (unlikely(throttled)) + if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se)))) task_throttle_setup_work(p);
I think this bit should also be replicated in set_next_task() after account_cfs_rq_runtime() since any part of the hierarchy may get throttled as a result of failing to grab runtime.
Also check_cfs_rq_runtime() only sees if the cfs_rq is throttled but the task can fail to run if it is on a throttled_hierarchy() too so that should be the correct check here.
Something like below (only build tested on queue/sched/flat):
kernel/sched/fair.c +19 -1
@@ -9224,7 +9224,19 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) goto again; p =task_of (se); -if (unlikely (check_cfs_rq_runtime (cfs_rq_of (se)))) +/* +* For cases where prev is picked again after +* being throttled, entity_tick() would have +* already marked its hierarchy as throttled. +* +* Add throttle work here since +* put_prev_set_next_task() is skipped on +* same task's selection. +* +* For other case, set_next_task_fair() will +* handle adding the throttle work. +*/ +if (throttled_hierarchy (cfs_rq_of (se)))task_throttle_setup_work (p);return p;
@@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)if (on_rq) weight =__calc_prop_weight (cfs_rq, se, weight); } +/* +* Add throttle work if the bandwidth allocation above failed +* to grab any runtime and throttled the task's hierarchy. +*/ +if (throttled_hierarchy (task_cfs_rq (p))) +task_throttle_setup_work (p); se = &p->se; cfs_rq->curr = se;
---
Quoted 2 lines
(quoted diff) +0
-0return p;
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 36 lines · K Prateek Nayak
Hello Peter,
Review static void wakeup_preempt_fair(struct r · line 9212
Quoted 9 lines · Peter Zijlstra
(quoted diff) +4
-1@@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r + se = pick_next_entity(rq, true); + if (!se) + goto again; p = task_of(se); - if (unlikely(throttled)) + if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se)))) task_throttle_setup_work(p);I think this bit should also be replicated in
set_next_task()afteraccount_cfs_rq_runtime()since any part of the hierarchy may get throttled as a result of failing to grab runtime.Also
check_cfs_rq_runtime()only sees if thecfs_rqis throttled but the task can fail to run if it is on athrottled_hierarchy()too so that should be the correct check here.Something like below (only build tested on queue/sched/flat):
kernel/sched/fair.c +4
-1@@ -9224,7 +9224,19 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) goto again; p =task_of (se); -if (unlikely (check_cfs_rq_runtime (cfs_rq_of (se)))) +/* +* For cases where prev is picked again after +* being throttled, entity_tick() would have +* already marked its hierarchy as throttled.
kernel/sched/fair.c struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) · line 9240
Quoted 10 lines
kernel/sched/fair.c +9
-0+* +* Add throttle work here since +* put_prev_set_next_task() is skipped on +* same task's selection. +* +* For other case, set_next_task_fair() will +* handle adding the throttle work. +*/ +if (throttled_hierarchy (cfs_rq_of (se)))task_throttle_setup_work (p);
Ah, right, because we've not accumulated runtime, it doesn't make sense to use check_cfs_rq_runtime() at pick time, all we need to do is check if the task should be throttled.
However, since set_next_task_fair() will walk the entire hierarchy anyway, we can remove it here entirely and fully rely on that.
kernel/sched/fair.c static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) · line 13839
Quoted 12 lines
kernel/sched/fair.c +6
-0return p;@@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)if (on_rq) weight =__calc_prop_weight (cfs_rq, se, weight); } +/* +* Add throttle work if the bandwidth allocation above failed +* to grab any runtime and throttled the task's hierarchy. +*/ +if (throttled_hierarchy (task_cfs_rq (p))) +task_throttle_setup_work (p);
We already call into account_cfs_rq_runtime(); which basically does all we need.
I think the distinction between account_cfs_rq_runtime() and check_cfs_rq_runtime() no longer makes sense. We can throttle a cfs_rq at any point now, since we no longer remove the cfs_rq, but rather we make the tasks suspend themselves until the cfs_rq naturally dequeues for being empty.
Something like so perhaps?
---
kernel/sched/fair.c +38 -65
@@ -488,7 +488,7 @@ static int se_is_idle(struct sched_entit#endif /* !CONFIG_FAIR_GROUP_SCHED */ static __always_inline -void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec); +bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);/************************************************************** * Scheduling class tree data structure manipulation methods:
@@ -1420,12 +1420,22 @@ static void update_curr(struct cfs_rq *c } } +static inline int cfs_rq_throttled (struct cfs_rq *cfs_rq); +static inline void task_throttle_setup_work (struct task_struct *p); +static void update_curr_fair (struct rq *rq) {struct sched_entity *se = &rq->donor->se; +bool throttled = false; -for_each_sched_entity (se) -update_curr (cfs_rq_of (se)); +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); +update_curr (cfs_rq); + throttled |=cfs_rq_throttled (cfs_rq); + } + +if (throttled) +task_throttle_setup_work (rq->donor); } static inline void
@@ -5627,7 +5637,6 @@ place_entity(struct cfs_rq *cfs_rq, stru }static void check_enqueue_throttle (struct cfs_rq *cfs_rq); -static inline int cfs_rq_throttled (struct cfs_rq *cfs_rq);static void enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags)
@@ -5830,8 +5839,6 @@ pick_next_entity(struct rq *rq, bool proreturn se; } -static bool check_cfs_rq_runtime (struct cfs_rq *cfs_rq); -static void put_prev_entity (struct cfs_rq *cfs_rq,struct sched_entity *prev) {/*
@@ -5841,9 +5848,6 @@ static void put_prev_entity(struct cfs_rif (prev->on_rq)update_curr (cfs_rq); -/* throttle cfs_rqs exceeding runtime */ -check_cfs_rq_runtime (cfs_rq); -if (prev->on_rq) {update_stats_wait_start_fair (cfs_rq, prev);/* in !on_rq case, update occurred at dequeue */
@@ -5976,44 +5980,29 @@ static int __assign_cfs_rq_runtime(strucreturn cfs_rq->runtime_remaining >0 ; } -/* returns 0 on failure to allocate runtime */ -static int assign_cfs_rq_runtime (struct cfs_rq *cfs_rq) -{ -struct cfs_bandwidth *cfs_b =tg_cfs_bandwidth (cfs_rq->tg); -int ret; - -raw_spin_lock (&cfs_b->lock); - ret =__assign_cfs_rq_runtime (cfs_b, cfs_rq,sched_cfs_bandwidth_slice ()); -raw_spin_unlock (&cfs_b->lock); - -return ret; -} +static bool throttle_cfs_rq (struct cfs_rq *cfs_rq); -static void __account_cfs_rq_runtime (struct cfs_rq *cfs_rq,u64 delta_exec) +static bool __account_cfs_rq_runtime (struct cfs_rq *cfs_rq,u64 delta_exec) {/* dock delta_exec before expiring quota (as it could span periods) */ cfs_rq->runtime_remaining -= delta_exec;if (likely (cfs_rq->runtime_remaining >0 )) -return ; +return false;if (cfs_rq->throttled) -return ; -/* -* if we're unable to extend our runtime we resched so that the active -* hierarchy can be throttled -*/ -if (!assign_cfs_rq_runtime (cfs_rq) &&likely (cfs_rq->h_curr)) -resched_curr (rq_of (cfs_rq)); +return true; + +return throttle_cfs_rq (cfs_rq); }static __always_inline -void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) +bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) {if (!cfs_bandwidth_used () || !cfs_rq->runtime_enabled) -return ; +return false; -__account_cfs_rq_runtime (cfs_rq, delta_exec); +return __account_cfs_rq_runtime (cfs_rq, delta_exec); }static inline int cfs_rq_throttled (struct cfs_rq *cfs_rq)
@@ -6284,9 +6273,9 @@ static bool throttle_cfs_rq(struct cfs_r* We have raced with bandwidth becoming available, and if we * actually throttled the timer might not unthrottle us for an * entire period. We additionally needed to make sure that any -* subsequent check_cfs_rq_runtime calls agree not to throttle -* us, as we may commit to do cfs put_prev+pick_next, so we ask -* for 1ns of runtime rather than just check cfs_b. +* subsequent account_cfs_rq_runtime() calls agree not to +* throttle us, as we may commit to do cfs put_prev+pick_next, +* so we ask for 1ns of runtime rather than just check cfs_b. */ dequeue =0 ; }else {
@@ -6711,8 +6700,6 @@ static void check_enqueue_throttle(struc/* update runtime allocation */ account_cfs_rq_runtime (cfs_rq,0 ); -if (cfs_rq->runtime_remaining <=0 ) -throttle_cfs_rq (cfs_rq); }static void sync_throttle (struct task_group *tg,int cpu)
@@ -6742,25 +6729,6 @@ static void sync_throttle(struct task_gr cfs_rq->pelt_clock_throttled =1 ; } -/* conditionally throttle active cfs_rq's from put_prev_entity() */ -static bool check_cfs_rq_runtime (struct cfs_rq *cfs_rq) -{ -if (!cfs_bandwidth_used ()) -return false; - -if (likely (!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining >0 )) -return false; - -/* -* it's possible for a throttled entity to be forced into a running -* state (e.g. set_curr_task), in this case we're finished. -*/ -if (cfs_rq_throttled (cfs_rq)) -return true; - -return throttle_cfs_rq (cfs_rq); -} -static enum hrtimer_restart sched_cfs_slack_timer (struct hrtimer *timer) { struct cfs_bandwidth *cfs_b =
@@ -7015,8 +6983,7 @@ static void sched_fair_update_stop_tick(#else /* !CONFIG_CFS_BANDWIDTH: */ -static void account_cfs_rq_runtime (struct cfs_rq *cfs_rq,u64 delta_exec) {} -static bool check_cfs_rq_runtime (struct cfs_rq *cfs_rq) {return false; } +static bool account_cfs_rq_runtime (struct cfs_rq *cfs_rq,u64 delta_exec) {return false; }static void check_enqueue_throttle (struct cfs_rq *cfs_rq) {}static inline void sync_throttle (struct task_group *tg,int cpu) {} static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) {}
@@ -9208,7 +9175,6 @@ struct task_struct *pick_task_fair(struc {struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; -struct task_struct *p;int new_tasks; again:
@@ -9223,10 +9189,7 @@ struct task_struct *pick_task_fair(strucif (!se) goto again; - p =task_of (se); -if (unlikely (check_cfs_rq_runtime (cfs_rq_of (se)))) -task_throttle_setup_work (p); -return p; +return task_of (se); idle: new_tasks =sched_balance_newidle (rq, rf);
@@ -13618,6 +13581,7 @@ static void task_tick_fair(struct rq *rq {struct sched_entity *se = &curr->se;unsigned long weight =NICE_0_LOAD ; +bool throttled = false;struct cfs_rq *cfs_rq;for_each_sched_entity (se) {
@@ -13625,8 +13589,13 @@ static void task_tick_fair(struct rq *rqentity_tick (cfs_rq, se, queued); weight =__calc_prop_weight (cfs_rq, se, weight); + + throttled |=cfs_rq_throttled (cfs_rq); } +if (throttled) +task_throttle_setup_work (curr); + se = &curr->se;reweight_eevdf (cfs_rq, se, weight, se->on_rq);
@@ -13800,6 +13769,7 @@ static void set_next_task_fair(struct rqstruct cfs_rq *cfs_rq = &rq->cfs;unsigned long weight =NICE_0_LOAD ;bool on_rq = se->on_rq; +bool throttled = false;clear_buddies (cfs_rq, se);
@@ -13814,12 +13784,15 @@ static void set_next_task_fair(struct rqset_next_entity (cfs_rq, se);/* ensure bandwidth has been allocated on our new cfs_rq */ -account_cfs_rq_runtime (cfs_rq,0 ); + throttled |=account_cfs_rq_runtime (cfs_rq,0 );if (on_rq) weight =__calc_prop_weight (cfs_rq, se, weight); } +if (throttled) +task_throttle_setup_work (p); + se = &p->se; cfs_rq->curr = se;
K Prateek Nayak
reviewerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Hello Peter,
Quoted 14 lines · Peter Zijlstra
Review static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) · line 13839
Quoted 10 lines
(quoted diff) +6
-0@@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) if (on_rq) weight = __calc_prop_weight(cfs_rq, se, weight); } + /* + * Add throttle work if the bandwidth allocation above failed + * to grab any runtime and throttled the task's hierarchy. + */ + if (throttled_hierarchy(task_cfs_rq(p))) + task_throttle_setup_work(p);We already call into
account_cfs_rq_runtime(); which basically does all we need.
Quoted 7 lines
I think the distinction between
account_cfs_rq_runtime()andcheck_cfs_rq_runtime()no longer makes sense. We can throttle acfs_rqat any point now, since we no longer remove thecfs_rq, but rather we make the tasks suspend themselves until thecfs_rqnaturally dequeues for being empty.Something like so perhaps?
That makes sense! The task should naturally execute the task work when exiting out of the kernel / IRQ handler into the userspace so we should be good.
I'll rebase the below diff on tip, test it a bit, add a commit log, and send it your way if you don't mind or would you like to keep it with the flat_cg bits?
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 25 lines · K Prateek Nayak
Hello Peter,
Quoted 14 lines · Peter Zijlstra
Review static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) · line 13839
Quoted 10 lines
(quoted diff) +6
-0@@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) if (on_rq) weight = __calc_prop_weight(cfs_rq, se, weight); } + /* + * Add throttle work if the bandwidth allocation above failed + * to grab any runtime and throttled the task's hierarchy. + */ + if (throttled_hierarchy(task_cfs_rq(p))) + task_throttle_setup_work(p);We already call into
account_cfs_rq_runtime(); which basically does all we need.Quoted 7 lines
I think the distinction between
account_cfs_rq_runtime()andcheck_cfs_rq_runtime()no longer makes sense. We can throttle acfs_rqat any point now, since we no longer remove thecfs_rq, but rather we make the tasks suspend themselves until thecfs_rqnaturally dequeues for being empty.Something like so perhaps?
Quoted 7 lines
That makes sense! The task should naturally execute the task work when exiting out of the kernel / IRQ handler into the userspace so we should be good.
I'll rebase the below diff on tip, test it a bit, add a commit log, and send it your way if you don't mind or would you like to keep it with the
flat_cgbits?
Nah, this seems like something that can be done independent. And thus is should be. That flat patch is big enough as is.
Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraQuoted 49 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the
shares_weightapproximation are fully preserved.That is, given a hierarchy like:
R | se--G1 / \ G2--se se--G3 / \ |T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:
_R_ / | \T1 T2 T3
Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (
se->h_load) is computed from the hierarchical fraction:se->load/cfs_rq->load.Since EEVDF is now exclusive operating on
rq->cfs, it needs to considercfs_rq->h_nr_queuedrather thancfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.One place where additional information was required was
set_next_task()/put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).As a result of only having a single level to pick from, much of the complications in
pick_next_task()and preemption go away.
Quoted 8 lines
Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.
TODO: split struct
cfs_rq/ structsched_entity
TODO: try and get rid ofh_currSigned-off-by Peter Zijlstra (Intel) <[email protected]>
I know Vincent was having some perf troubles with this patch, but booting on a 64 vCPU qemu environment, I'm seeing:
[ 5.688490] Oops: divide error: 0000 [#1] SMP NOPTI [ 5.689457] CPU: 47 UID: 0 PID: 0 Comm: swapper/47 Not tainted 7.1.0-rc2-00026-g82a8ec6fb3f9 #38 PREEMPT(full) [ 5.689457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 [ 5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430 [ 5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9 90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0 14 31 9 [ 5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046 [ 5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80 [ 5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00 [ 5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 [ 5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00 [ 5.689457] FS: 0000000000000000(0000) GS:ffff888235c2e000(0000) knlGS:0000000000000000 [ 5.689457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0 [ 5.689457] Call Trace: [ 5.689457] <TASK> [ 5.689457] wakeup_preempt+0xa8/0xd0 [ 5.689457] attach_one_task+0xec/0x150 [ 5.689457] __schedule+0x1ad8/0x21c0 [ 5.689457] schedule_idle+0x22/0x40 [ 5.689457] cpu_startup_entry+0x29/0x30 [ 5.689457] start_secondary+0xf7/0x100 [ 5.689457] common_startup_64+0x13e/0x148 [ 5.689457] </TASK> [ 5.689457] Dumping ftrace buffer: [ 5.689457] (ftrace buffer empty) [ 5.689457] ---[ end trace 0000000000000000 ]--- [ 5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430 [ 5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9 90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0 14 31 9 [ 5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046 [ 5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80 [ 5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00 [ 5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 [ 5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00 [ 5.689457] FS: 0000000000000000(0000) GS:ffff888235c2e000(0000) knlGS:0000000000000000 [ 5.689457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0 [ 5.689457] Kernel panic - not syncing: Fatal exception
Which I bisected down to this last patch in the series.
faddr2line gave me:__calc_delta at kernel/sched/fair.c:290
(inlined by) calc_delta_fair at kernel/sched/fair.c:300 (inlined by) update_protect_slice at kernel/sched/fair.c:1070 (inlined by) wakeup_preempt_fair at kernel/sched/fair.c:9193
This usually trips as the ww_mutex selftest starts at bootup.
Unfortunately I still see it with the add-on changes you proposed to K Prateek's feedback here.
I'll try to narrow it down further tomorrow.
thanks
-john
Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 5 lines · John Stultz
Quoted 3 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
...
Quoted 60 lines
I know Vincent was having some perf troubles with this patch, but booting on a 64 vCPU qemu environment, I'm seeing:
[ 5.688490] Oops: divide error: 0000 [#1] SMP NOPTI [ 5.689457] CPU: 47 UID: 0 PID: 0 Comm: swapper/47 Not tainted 7.1.0-rc2-00026-g82a8ec6fb3f9 #38 PREEMPT(full) [ 5.689457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 [ 5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430 [ 5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9 90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0 14 31 9 [ 5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046 [ 5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80 [ 5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00 [ 5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 [ 5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00 [ 5.689457] FS: 0000000000000000(0000) GS:ffff888235c2e000(0000) knlGS:0000000000000000 [ 5.689457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0 [ 5.689457] Call Trace: [ 5.689457] <TASK> [ 5.689457] wakeup_preempt+0xa8/0xd0 [ 5.689457] attach_one_task+0xec/0x150 [ 5.689457] __schedule+0x1ad8/0x21c0 [ 5.689457] schedule_idle+0x22/0x40 [ 5.689457] cpu_startup_entry+0x29/0x30 [ 5.689457] start_secondary+0xf7/0x100 [ 5.689457] common_startup_64+0x13e/0x148 [ 5.689457] </TASK> [ 5.689457] Dumping ftrace buffer: [ 5.689457] (ftrace buffer empty) [ 5.689457] ---[ end trace 0000000000000000 ]--- [ 5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430 [ 5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9 90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0 14 31 9 [ 5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046 [ 5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80 [ 5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00 [ 5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 [ 5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00 [ 5.689457] FS: 0000000000000000(0000) GS:ffff888235c2e000(0000) knlGS:0000000000000000 [ 5.689457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0 [ 5.689457] Kernel panic - not syncing: Fatal exceptionWhich I bisected down to this last patch in the series.
faddr2line gave me:
__calc_deltaatkernel/sched/fair.c:290
(inlined by)calc_delta_fairatkernel/sched/fair.c:300(inlined by)update_protect_sliceatkernel/sched/fair.c:1070(inlined by)wakeup_preempt_fairatkernel/sched/fair.c:9193
Quoted 6 lines
This usually trips as the
ww_mutexselftest starts at bootup.Unfortunately I still see it with the add-on changes you proposed to K Prateek's feedback here.
I'll try to narrow it down further tomorrow.
As karma would have it, this does seem to depend on CONFIG_SCHED_PROXY_EXEC. :) I'm guessing the switch in calc_delta_fair() to use se->h_load is uncovering something proxy isn't handling properly with that value.
But I'll have more tomorrow.
thanks
-john
Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 69 lines · John Stultz
Quoted 5 lines · John Stultz
Quoted 3 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
...
Quoted 62 lines
I know Vincent was having some perf troubles with this patch, but booting on a 64 vCPU qemu environment, I'm seeing:
[ 5.688490] Oops: divide error: 0000 [#1] SMP NOPTI [ 5.689457] CPU: 47 UID: 0 PID: 0 Comm: swapper/47 Not tainted 7.1.0-rc2-00026-g82a8ec6fb3f9 #38 PREEMPT(full) [ 5.689457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 [ 5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430 [ 5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9 90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0 14 31 9 [ 5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046 [ 5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80 [ 5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00 [ 5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 [ 5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00 [ 5.689457] FS: 0000000000000000(0000) GS:ffff888235c2e000(0000) knlGS:0000000000000000 [ 5.689457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0 [ 5.689457] Call Trace: [ 5.689457] <TASK> [ 5.689457] wakeup_preempt+0xa8/0xd0 [ 5.689457] attach_one_task+0xec/0x150 [ 5.689457] __schedule+0x1ad8/0x21c0 [ 5.689457] schedule_idle+0x22/0x40 [ 5.689457] cpu_startup_entry+0x29/0x30 [ 5.689457] start_secondary+0xf7/0x100 [ 5.689457] common_startup_64+0x13e/0x148 [ 5.689457] </TASK> [ 5.689457] Dumping ftrace buffer: [ 5.689457] (ftrace buffer empty) [ 5.689457] ---[ end trace 0000000000000000 ]--- [ 5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430 [ 5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9 90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0 14 31 9 [ 5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046 [ 5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80 [ 5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00 [ 5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 [ 5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00 [ 5.689457] FS: 0000000000000000(0000) GS:ffff888235c2e000(0000) knlGS:0000000000000000 [ 5.689457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0 [ 5.689457] Kernel panic - not syncing: Fatal exceptionWhich I bisected down to this last patch in the series.
faddr2line gave me:
__calc_deltaatkernel/sched/fair.c:290
(inlined by)calc_delta_fairatkernel/sched/fair.c:300(inlined by)update_protect_sliceatkernel/sched/fair.c:1070(inlined by)wakeup_preempt_fairatkernel/sched/fair.c:9193This usually trips as the
ww_mutexselftest starts at bootup.
Quoted 9 lines
Quoted 4 lines
Unfortunately I still see it with the add-on changes you proposed to K Prateek's feedback here.
I'll try to narrow it down further tomorrow.
As karma would have it, this does seem to depend on
CONFIG_SCHED_PROXY_EXEC. :) I'm guessing the switch incalc_delta_fair()to usese->h_loadis uncovering something proxy isn't handling properly with that value.
So looking at the callstack when I see the failure:proxy_find_task()
proxy_force_return()
proxy_resched_idle() <- sets rq->donor to idle attach_one_task()
wakeup_preempt()
wakeup_preempt_fair()
update_protect_slice() <- called with the donor's se
calc_delta_fair()
__calc_delta() <- div by zero
Basically we end up in wakeup_preempt_fair() with rq->donor == rq->idle because we earlier called proxy_resched_idle().
Without proxy, if we call wakeup_preempt_fair() when rq->donor (and rq->curr) is rq->idle, we usually end up taking the if (test_tsk_need_resched(rq->curr)) early exit and we don't hit this.
But with proxy, rq->curr isn't idle at this point. So we end up continuing on. Despite the se_is_idle(se) checks (where se is the &donor->se), those don't catch because rq->idle (maybe unintuitvely) has a SCHED_NORMAL policy.
So we end up getting down to update_protect_slice() with rq->idle as the se and the idle h_load.weight is zero.
Not sure what the best approach might be, but adding:
if (donor == rq->idle) {
/* don't give rq->idle slice protection */
preempt_action = PREEMPT_WAKEUP_SHORT;
goto preempt;}
similar to the if (cse_is_idle && !pse_is_idle) check seems to resolve this.
Anyway, if you have thoughts on better approach, I'd be happy to work up a patch to add on top of this one.
thanks
-john
K Prateek Nayak
reviewerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Hello John,
Quoted 7 lines · John Stultz
So looking at the callstack when I see the failure:
proxy_find_task()
proxy_force_return()proxy_resched_idle() <- sets rq->donor to idle attach_one_task() wakeup_preempt() wakeup_preempt_fair()
After this point, I would have expected we called idle class's wakeup_preempt() since that is the donor context ...
Quoted 6 lines
update_protect_slice()<- called with the donor's se
calc_delta_fair()
__calc_delta()<- div by zeroBasically we end up in
wakeup_preempt_fair()withrq->donor==rq->idlebecause we earlier calledproxy_resched_idle().
Could you check if following makes things better:
(Only build tested)
kernel/sched/core.c +1 -0
@@ -6653,6 +6653,7 @@ static inline void proxy_set_task_cpu(struct task_struct *p, int cpu)static inline struct task_struct *proxy_resched_idle (struct rq *rq) {put_prev_set_next_task (rq, rq->donor, rq->idle); + rq->next_class = &idle_sched_class;rq_set_donor (rq, rq->idle);set_tsk_need_resched (rq->idle);return rq->idle;
---
I'm just getting started for the day so it'll be a while before I actually get to test this on top of flat cgroup bits which I haven't yet run with SCHED_PROXY_EXEC enabled.
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 10 lines · K Prateek Nayak
Quoted 7 lines · John Stultz
So looking at the callstack when I see the failure:
proxy_find_task()
proxy_force_return()proxy_resched_idle() <- sets rq->donor to idle attach_one_task() wakeup_preempt() wakeup_preempt_fair()
Quoted 2 lines
After this point, I would have expected we called idle class's
wakeup_preempt()since that is the donor context ...
Ah, that's a good point! (I was getting muddied by the rq->idle having SCHED_NORMAL policy value and assuming that was why we were in the fair code).
Quoted 16 lines
Quoted 6 lines
update_protect_slice()<- called with the donor's se
calc_delta_fair()
__calc_delta()<- div by zeroBasically we end up in
wakeup_preempt_fair()withrq->donor==rq->idlebecause we earlier calledproxy_resched_idle().Could you check if following makes things better:
(Only build tested)
kernel/sched/core.c +0
-0
kernel/sched/core.c static inline void proxy_set_task_cpu(struct task_struct *p, int cpu) · line 6659
Quoted 7 lines
kernel/sched/core.c +1
-0@@ -6653,6 +6653,7 @@ static inline void proxy_set_task_cpu(struct task_struct *p, int cpu)static inline struct task_struct *proxy_resched_idle (struct rq *rq) {put_prev_set_next_task (rq, rq->donor, rq->idle); + rq->next_class = &idle_sched_class;rq_set_donor (rq, rq->idle);set_tsk_need_resched (rq->idle);return rq->idle;
Yeah, that looks to avoid the problem and is a fair bit cleaner. I missed the introduction of the rq->next_class detail! Thanks for pointing this out!
I'll do some testing against the full series and get a patch sent out here soon. -john
Vincent Guittot
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraQuoted 1,168 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the
shares_weightapproximation are fully preserved.That is, given a hierarchy like:
R | se--G1 / \ G2--se se--G3 / \ |T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:
_R_ / | \T1 T2 T3
Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (
se->h_load) is computed from the hierarchical fraction:se->load/cfs_rq->load.Since EEVDF is now exclusive operating on
rq->cfs, it needs to considercfs_rq->h_nr_queuedrather thancfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.One place where additional information was required was
set_next_task()/put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).As a result of only having a single level to pick from, much of the complications in
pick_next_task()and preemption go away.Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.
TODO: split struct
cfs_rq/ structsched_entity
TODO: try and get rid ofh_currSigned-off-by Peter Zijlstra (Intel) <[email protected]>--- include/linux/sched.h | 1 kernel/sched/core.c | 5 kernel/sched/debug.c | 9 kernel/sched/fair.c | 789 +++++++++++++++++++++----------------------------- kernel/sched/pelt.c | 6 kernel/sched/sched.h | 26 - 6 files changed, 366 insertions(+), 470 deletions(-)include/linux/sched.h +1
-0@@ -575,6 +575,7 @@ struct sched_statistics {struct sched_entity {/* For load-balancing: */ struct load_weight load; +struct load_weight h_load;struct rb_node run_node;u64 deadline;u64 min_vruntime;kernel/sched/core.c +1
-4@@ -5539,11 +5539,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);*/ static inline void prefetch_curr_exec_start (struct task_struct *p) { -#ifdef CONFIG_FAIR_GROUP_SCHED -struct sched_entity *curr = p->se.cfs_rq->curr; -#else struct sched_entity *curr =task_rq (p)->cfs.curr; -#endif +prefetch (curr);prefetch (&curr->exec_start); }kernel/sched/debug.c +6
-3@@ -911,10 +911,11 @@ print_task(struct seq_file *m, struct rqelse SEQ_printf (m," %c" ,task_state_to_char (p)); - SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", + SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), + p->se.h_load.weight, SPLIT_NS(p->se.vruntime), - entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', + entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice),@@ -943,7 +944,7 @@ static void print_rq(struct seq_file *m,SEQ_printf (m,"\n" );SEQ_printf (m,"runnable tasks:\n" ); -SEQ_printf (m," S task PID vruntime eligible " +SEQ_printf (m," S task PID weight vruntime eligible " "deadline slice sum-exec switches " "prio wait-time sum-sleep sum-block" #ifdef CONFIG_NUMA_BALANCING@@ -1051,6 +1052,8 @@ void print_cfs_rq(struct seq_file *m, in cfs_rq->tg_load_avg_contrib);SEQ_printf (m," .%-30s: %ld\n" ,"tg_load_avg" ,atomic_long_read (&cfs_rq->tg->load_avg)); +SEQ_printf (m," .%-30s: %lu\n" ,"h_load" , + cfs_rq->h_load);#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_CFS_BANDWIDTH SEQ_printf(m, " .%-30s: %d\n", "throttled",kernel/sched/fair.c +249
-316@@ -296,8 +296,8 @@ static u64 __calc_delta(u64 delta_exec,*/ static inline u64 calc_delta_fair (u64 delta,struct sched_entity *se) { -if (unlikely (se->load.weight !=NICE_0_LOAD )) - delta =__calc_delta (delta,NICE_0_LOAD , &se->load); +if (se->h_load.weight !=NICE_0_LOAD ) + delta =__calc_delta (delta,NICE_0_LOAD , &se->h_load);return delta; }@@ -427,38 +427,6 @@ static inline struct sched_entity *parenreturn se->parent; } -static void -find_matching_se (struct sched_entity **se,struct sched_entity **pse) -{ -int se_depth, pse_depth; - -/* -* preemption test can be made between sibling entities who are in the -* same cfs_rq i.e who have a common parent. Walk up the hierarchy of -* both tasks until we find their ancestors who are siblings of common -* parent. -*/ - -/* First walk up until both entities are at same depth */ - se_depth = (*se)->depth; - pse_depth = (*pse)->depth; - -while (se_depth > pse_depth) { - se_depth--; - *se =parent_entity (*se); - } - -while (pse_depth > se_depth) { - pse_depth--; - *pse =parent_entity (*pse); - } - -while (!is_same_group (*se, *pse)) { - *se =parent_entity (*se); - *pse =parent_entity (*pse); - } -} -static int tg_is_idle (struct task_group *tg) {return tg->idle >0 ;@@ -502,11 +470,6 @@ static inline struct sched_entity *parenreturn NULL ; } -static inline void -find_matching_se (struct sched_entity **se,struct sched_entity **pse) -{ -} -static inline int tg_is_idle (struct task_group *tg) {return 0 ;@@ -685,7 +648,7 @@ static inline unsigned long avg_vruntimestatic inline void __sum_w_vruntime_add (struct cfs_rq *cfs_rq,struct sched_entity *se) { -unsigned long weight =avg_vruntime_weight (cfs_rq, se->load.weight); +unsigned long weight =avg_vruntime_weight (cfs_rq, se->h_load.weight);s64 w_vruntime, key =entity_key (cfs_rq, se); w_vruntime = key * weight;@@ -702,7 +665,7 @@ sum_w_vruntime_add_paranoid(struct cfs_rs64 key, tmp; again: - weight =avg_vruntime_weight (cfs_rq, se->load.weight); + weight =avg_vruntime_weight (cfs_rq, se->h_load.weight); key =entity_key (cfs_rq, se);if (check_mul_overflow (key, weight, &key))@@ -748,7 +711,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rqstatic void sum_w_vruntime_sub (struct cfs_rq *cfs_rq,struct sched_entity *se) { -unsigned long weight =avg_vruntime_weight (cfs_rq, se->load.weight); +unsigned long weight =avg_vruntime_weight (cfs_rq, se->h_load.weight);s64 key =entity_key (cfs_rq, se); cfs_rq->sum_w_vruntime -= key * weight;@@ -790,7 +753,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)s64 runtime = cfs_rq->sum_w_vruntime;if (curr) { -unsigned long w =avg_vruntime_weight (cfs_rq, curr->load.weight); +unsigned long w =avg_vruntime_weight (cfs_rq, curr->h_load.weight); runtime +=entity_key (cfs_rq, curr) * w; weight += w;@@ -861,8 +824,6 @@ bool update_entity_lag(struct cfs_rq *cfu64 avruntime =avg_vruntime (cfs_rq);s64 vlag =entity_lag (cfs_rq, se, avruntime); -WARN_ON_ONCE (!se->on_rq); -if (se->sched_delayed) {/* previous vlag < 0 otherwise se would not be delayed */ vlag =max (vlag, se->vlag);@@ -898,7 +859,7 @@ static int vruntime_eligible(struct cfs_long load = cfs_rq->sum_weight;if (curr && curr->on_rq) { -unsigned long weight =avg_vruntime_weight (cfs_rq, curr->load.weight); +unsigned long weight =avg_vruntime_weight (cfs_rq, curr->h_load.weight); avg +=entity_key (cfs_rq, curr) * weight; load += weight;@@ -1039,6 +1000,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim*/ static void __enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (&rq_of (cfs_rq)->cfs != cfs_rq); +WARN_ON_ONCE (!entity_is_task (se)); +sum_w_vruntime_add (cfs_rq, se); se->min_vruntime = se->vruntime; se->min_slice = se->slice;@@ -1048,6 +1012,9 @@ static void __enqueue_entity(struct cfs_static void __dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (&rq_of (cfs_rq)->cfs != cfs_rq); +WARN_ON_ONCE (!entity_is_task (se)); +rb_erase_augmented_cached (&se->run_node, &cfs_rq->tasks_timeline, &min_vruntime_cb);sum_w_vruntime_sub (cfs_rq, se);@@ -1144,7 +1111,7 @@ static struct sched_entity *pick_eevdf(s* We can safely skip eligibility check if there is only one entity * in this cfs_rq, saving some cycles. */ -if (cfs_rq->nr_queued ==1 ) +if (cfs_rq->h_nr_queued ==1 )return curr && curr->on_rq ? curr : se;/* @@ -1391,8 +1358,6 @@ static s64 update_se(struct rq *rq, strureturn delta_exec; } -static void set_next_buddy (struct sched_entity *se); -/* * Used by other classes to account runtime. */ @@ -1412,7 +1377,7 @@ static void update_curr(struct cfs_rq *c* not necessarily be the actual task running * (rq->curr.se). This is easy to confuse! */ -struct sched_entity *curr = cfs_rq->curr; +struct sched_entity *curr = cfs_rq->h_curr;struct rq *rq =rq_of (cfs_rq);s64 delta_exec;bool resched;@@ -1424,26 +1389,29 @@ static void update_curr(struct cfs_rq *cif (unlikely (delta_exec <=0 ))return ; +account_cfs_rq_runtime (cfs_rq, delta_exec); + +if (!entity_is_task (curr)) +return ; + + cfs_rq = &rq->cfs; + curr->vruntime +=calc_delta_fair (delta_exec, curr); resched =update_deadline (cfs_rq, curr); -if (entity_is_task (curr)) { -/* -* If the fair_server is active, we need to account for the -* fair_server time whether or not the task is running on -* behalf of fair_server or not: -* - If the task is running on behalf of fair_server, we need -* to limit its time based on the assigned runtime. -* - Fair task that runs outside of fair_server should account -* against fair_server such that it can account for this time -* and possibly avoid running this period. -*/ -dl_server_update (&rq->fair_server, delta_exec); - } - -account_cfs_rq_runtime (cfs_rq, delta_exec); +/* +* If the fair_server is active, we need to account for the +* fair_server time whether or not the task is running on +* behalf of fair_server or not: +* - If the task is running on behalf of fair_server, we need +* to limit its time based on the assigned runtime. +* - Fair task that runs outside of fair_server should account +* against fair_server such that it can account for this time +* and possibly avoid running this period. +*/ +dl_server_update (&rq->fair_server, delta_exec); -if (cfs_rq->nr_queued ==1 ) +if (cfs_rq->h_nr_queued ==1 )return ;if (resched || !protect_slice (curr)) {@@ -1454,7 +1422,10 @@ static void update_curr(struct cfs_rq *cstatic void update_curr_fair (struct rq *rq) { -update_curr (cfs_rq_of (&rq->donor->se)); +struct sched_entity *se = &rq->donor->se; + +for_each_sched_entity (se) +update_curr (cfs_rq_of (se)); } static inline void@@ -1530,7 +1501,7 @@ update_stats_enqueue_fair(struct cfs_rq* Are we enqueueing a waiting task? (for current tasks * a dequeue/enqueue event is a NOP) */ -if (se != cfs_rq->curr) +if (se != cfs_rq->h_curr)update_stats_wait_start_fair (cfs_rq, se);if (flags &ENQUEUE_WAKEUP )@@ -1548,7 +1519,7 @@ update_stats_dequeue_fair(struct cfs_rq* Mark the end of the wait period if dequeueing a * waiting task: */ -if (se != cfs_rq->curr) +if (se != cfs_rq->h_curr)update_stats_wait_end_fair (cfs_rq, se);if ((flags &DEQUEUE_SLEEP ) &&entity_is_task (se)) {@@ -3875,6 +3846,7 @@ static inline void update_scan_period(ststatic void account_entity_enqueue (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (cfs_rq !=cfs_rq_of (se));update_load_add (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {struct rq *rq =rq_of (cfs_rq);@@ -3888,6 +3860,7 @@ account_entity_enqueue(struct cfs_rq *cfstatic void account_entity_dequeue (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (cfs_rq !=cfs_rq_of (se));update_load_sub (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se));@@ -3965,7 +3938,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,static void rescale_entity (struct sched_entity *se,unsigned long weight,bool rel_vprot) { -unsigned long old_weight = se->load.weight; +long old_weight = se->h_load.weight;/* * VRUNTIME @@ -4065,16 +4038,17 @@ rescale_entity(struct sched_entity *se, se->vprot =div64_long (se->vprot * old_weight, weight); } -static void reweight_entity (struct cfs_rq *cfs_rq,struct sched_entity *se, -unsigned long weight) +static void reweight_eevdf (struct cfs_rq *cfs_rq,struct sched_entity *se, +unsigned long weight,bool on_rq) {bool curr = cfs_rq->curr == se;bool rel_vprot = false;u64 avruntime =0 ; -if (se->on_rq) { -/* commit outstanding execution time */ -update_curr (cfs_rq); +if (se->h_load.weight == weight) +return ; + +if (on_rq) { avruntime =avg_vruntime (cfs_rq); se->vlag =entity_lag (cfs_rq, se, avruntime); se->deadline -= avruntime;@@ -4084,46 +4058,90 @@ static void reweight_entity(struct cfs_r rel_vprot = true; } - cfs_rq->nr_queued--; + cfs_rq->h_nr_queued--;if (!curr)__dequeue_entity (cfs_rq, se); -update_load_sub (&cfs_rq->load, se->load.weight); } -dequeue_load_avg (cfs_rq, se);rescale_entity (se, weight, rel_vprot); -update_load_set (&se->load, weight); +update_load_set (&se->h_load, weight); -do { -u32 divider =get_pelt_divider (&se->avg); - se->avg.load_avg =div_u64 (se_weight (se) * se->avg.load_sum, divider); - }while (0 ); - -enqueue_load_avg (cfs_rq, se); -if (se->on_rq) { +if (on_rq) {if (rel_vprot) se->vprot += avruntime; se->deadline += avruntime; se->rel_deadline =0 ; se->vruntime = avruntime - se->vlag; -update_load_add (&cfs_rq->load, se->load.weight);if (!curr)__enqueue_entity (cfs_rq, se); - cfs_rq->nr_queued++; + cfs_rq->h_nr_queued++; } } +static void reweight_entity (struct cfs_rq *cfs_rq,struct sched_entity *se, +unsigned long weight) +{ +if (se->load.weight == weight) +return ; + +if (se->on_rq) { +WARN_ON_ONCE (cfs_rq !=cfs_rq_of (se)); +update_load_sub (&cfs_rq->load, se->load.weight); + } +dequeue_load_avg (cfs_rq, se); + +update_load_set (&se->load, weight); + +do { +u32 divider =get_pelt_divider (&se->avg); + se->avg.load_avg =div_u64 (se_weight (se) * se->avg.load_sum, divider); + }while (0 ); + +enqueue_load_avg (cfs_rq, se); + +if (se->on_rq) +update_load_add (&cfs_rq->load, se->load.weight); +} + +/* +* weight = NICE_0_LOAD; +* for_each_entity_se(se) +* weight = __calc_prop_weight(cfs_rq_of(se), se, weight); +*/ +static __always_inline +unsigned long __calc_prop_weight (struct cfs_rq *cfs_rq,struct sched_entity *se, +unsigned long weight) +{ + weight *= se->load.weight; +if (parent_entity (se)) + weight /= cfs_rq->load.weight; +else + weight /=NICE_0_LOAD ; + +return max (weight,MIN_SHARES ); +} +static void reweight_task_fair (struct rq *rq,struct task_struct *p,const struct load_weight *lw) {struct sched_entity *se = &p->se; -struct cfs_rq *cfs_rq =cfs_rq_of (se); -struct load_weight *load = &se->load; +unsigned long weight =NICE_0_LOAD ; + +if (se->on_rq) +update_curr_fair (rq); + +reweight_entity (cfs_rq_of (se), se, lw->weight); + se->load.inv_weight = lw->inv_weight; + +if (!se->on_rq) +return ; + +for_each_sched_entity (se) + weight =__calc_prop_weight (cfs_rq_of (se), se, weight); -reweight_entity (cfs_rq, se, lw->weight); - load->inv_weight = lw->inv_weight; +reweight_eevdf (&rq->cfs, &p->se, weight, p->se.on_rq); }static inline int throttled_hierarchy (struct cfs_rq *cfs_rq);@@ -4331,7 +4349,6 @@ static long calc_group_shares(struct cfsstatic void update_cfs_group (struct sched_entity *se) {struct cfs_rq *gcfs_rq =group_cfs_rq (se); -long shares;/* * When a group becomes empty, preserve its weight. This matters for @@ -4340,9 +4357,7 @@ static void update_cfs_group(struct scheif (!gcfs_rq || !gcfs_rq->load.weight)return ; - shares =calc_group_shares (gcfs_rq); -if (unlikely (se->load.weight != shares)) - reweight_entity(cfs_rq_of(se), se, shares); + reweight_entity(cfs_rq_of(se), se, calc_group_shares(gcfs_rq)); }#else /* !CONFIG_FAIR_GROUP_SCHED: */ @@ -4460,7 +4475,7 @@ static inline bool cfs_rq_is_decayed(str* differential update where we store the last value we propagated. This in * turn allows skipping updates if the differential is 'small'. * -* Updating tg's load_avg is necessary before update_cfs_share(). +* Updating tg's load_avg is necessary before update_cfs_group(). */ static inline void update_tg_load_avg (struct cfs_rq *cfs_rq) {@@ -4926,7 +4941,7 @@ static void migrate_se_pelt_lag(struct s* The cfs_rq avg is the direct sum of all its entities (blocked and runnable) * avg. The immediate corollary is that all (fair) tasks must be attached. * -* cfs_rq->avg is used for task_h_load() and update_cfs_share() for example. +* cfs_rq->avg is used for task_h_load() and update_cfs_group() for example. * * Return: true if the load decayed or we removed load. * @@ -5475,6 +5490,7 @@ static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { u64 vslice, vruntime = avg_vruntime(cfs_rq); +unsigned int nr_queued = cfs_rq->h_nr_queued;bool update_zero = false;s64 lag =0 ;@@ -5482,6 +5498,9 @@ place_entity(struct cfs_rq *cfs_rq, stru se->slice = sysctl_sched_base_slice; vslice =calc_delta_fair (se->slice, se); +if (flags &ENQUEUE_QUEUED ) + nr_queued -=1 ; +/* * Due to how V is constructed as the weighted average of entities, * adding tasks with positive lag, or removing tasks with negative lag @@ -5490,7 +5509,7 @@ place_entity(struct cfs_rq *cfs_rq, stru* * EEVDF: placement strategy #1 / #2 */ -if (sched_feat (PLACE_LAG ) && cfs_rq->nr_queued && se->vlag) { +if (sched_feat (PLACE_LAG ) && nr_queued && se->vlag) {struct sched_entity *curr = cfs_rq->curr;long load, weight;@@ -5550,9 +5569,9 @@ place_entity(struct cfs_rq *cfs_rq, stru*/ load = cfs_rq->sum_weight;if (curr && curr->on_rq) - load +=avg_vruntime_weight (cfs_rq, curr->load.weight); + load +=avg_vruntime_weight (cfs_rq, curr->h_load.weight); - weight =avg_vruntime_weight (cfs_rq, se->load.weight); + weight =avg_vruntime_weight (cfs_rq, se->h_load.weight); lag *= load + weight;if (WARN_ON_ONCE (!load)) load =1 ;@@ -5611,22 +5630,8 @@ static void check_enqueue_throttle(strucstatic inline int cfs_rq_throttled (struct cfs_rq *cfs_rq);static void -requeue_delayed_entity (struct sched_entity *se); - -static void enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -bool curr = cfs_rq->curr == se; - -/* -* If we're the current task, we must renormalise before calling -* update_curr(). -*/ -if (curr) -place_entity (cfs_rq, se, flags); - -update_curr (cfs_rq); -/* * When enqueuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. @@ -5645,13 +5650,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st*/ update_cfs_group (se); -/* -* XXX now that the entity has been re-weighted, and it's lag adjusted, -* we can place the entity. -*/ -if (!curr) -place_entity (cfs_rq, se, flags); -account_entity_enqueue (cfs_rq, se);/* Entity has migrated, no longer consider this task hot */ @@ -5660,8 +5658,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, stcheck_schedstat_required ();update_stats_enqueue_fair (cfs_rq, se, flags); -if (!curr) -__enqueue_entity (cfs_rq, se); se->on_rq =1 ;if (cfs_rq->nr_queued ==1 ) {@@ -5679,21 +5675,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st } } -static void __clear_buddies_next (struct sched_entity *se) +static void set_next_buddy (struct cfs_rq *cfs_rq,struct sched_entity *se) { -for_each_sched_entity (se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); -if (cfs_rq->next != se) -break ; - - cfs_rq->next =NULL ; - } +if (WARN_ON_ONCE (!se->on_rq || se->sched_delayed)) +return ; +if (se_is_idle (se)) +return ; + cfs_rq->next = se; }static void clear_buddies (struct cfs_rq *cfs_rq,struct sched_entity *se) {if (cfs_rq->next == se) -__clear_buddies_next (se); + cfs_rq->next =NULL ; } static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);@@ -5704,7 +5698,7 @@ static void set_delayed(struct sched_ent/* * Delayed se of cfs_rq have no tasks queued on them. -* Do not adjust h_nr_runnable since dequeue_entities() +* Do not adjust h_nr_runnable since __dequeue_task() * will account it for blocked tasks. */ if (!entity_is_task (se))@@ -5737,37 +5731,11 @@ static void clear_delayed(struct sched_e } } -static bool +static void dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -bool sleep = flags &DEQUEUE_SLEEP ;int action =UPDATE_TG ; -update_curr (cfs_rq); -clear_buddies (cfs_rq, se); - -if (flags &DEQUEUE_DELAYED ) { -WARN_ON_ONCE (!se->sched_delayed); - }else { -bool delay = sleep; -/* -* DELAY_DEQUEUE relies on spurious wakeups, special task -* states must not suffer spurious wakeups, excempt them. -*/ -if (flags & (DEQUEUE_SPECIAL |DEQUEUE_THROTTLE )) - delay = false; - -WARN_ON_ONCE (delay && se->sched_delayed); - -if (sched_feat (DELAY_DEQUEUE ) && delay && - !entity_eligible (cfs_rq, se)) { -update_load_avg (cfs_rq, se,0 ); -update_entity_lag (cfs_rq, se); -set_delayed (se); -return false; - } - } -if (entity_is_task (se) &&task_on_rq_migrating (task_of (se))) action |=DO_DETACH ;@@ -5785,14 +5753,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_stats_dequeue_fair (cfs_rq, se, flags); -update_entity_lag (cfs_rq, se); -if (sched_feat (PLACE_REL_DEADLINE ) && !sleep) { - se->deadline -= se->vruntime; - se->rel_deadline =1 ; - } - -if (se != cfs_rq->curr) -__dequeue_entity (cfs_rq, se); se->on_rq =0 ;account_entity_dequeue (cfs_rq, se);@@ -5801,9 +5761,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_cfs_group (se); -if (flags &DEQUEUE_DELAYED ) -clear_delayed (se); -if (cfs_rq->nr_queued ==0 ) {update_idle_cfs_rq_clock_pelt (cfs_rq);#ifdef CONFIG_CFS_BANDWIDTH @@ -5816,15 +5773,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st }#endif } - -return true; }static void -set_next_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,bool first) +set_next_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { -clear_buddies (cfs_rq, se); -/* 'current' is not kept within the tree. */ if (se->on_rq) {/* @@ -5833,16 +5786,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s* runqueue. */ update_stats_wait_end_fair (cfs_rq, se); -__dequeue_entity (cfs_rq, se);update_load_avg (cfs_rq, se,UPDATE_TG ); - -if (first) -set_protect_slice (cfs_rq, se); }update_stats_curr_start (cfs_rq, se); -WARN_ON_ONCE (cfs_rq->curr); - cfs_rq->curr = se; +WARN_ON_ONCE (cfs_rq->h_curr); + cfs_rq->h_curr = se;/* * Track our maximum slice length, if the CPU's load is at @@ -5862,23 +5811,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s se->prev_sum_exec_runtime = se->sum_exec_runtime; } -static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags); +static bool __dequeue_task (struct rq *rq,struct task_struct *p,int flags); -/* -* Pick the next process, keeping these things in mind, in this order: -* 1) keep things fair between processes/task groups -* 2) pick the "next" process, since someone really wants that to run -* 3) pick the "last" process, for cache locality -* 4) do not run the "skip" process, if something else is available -*/ static struct sched_entity * -pick_next_entity (struct rq *rq,struct cfs_rq *cfs_rq,bool protect) +pick_next_entity (struct rq *rq,bool protect) { +struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; se =pick_eevdf (cfs_rq, protect);if (se->sched_delayed) { -dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +__dequeue_task (rq,task_of (se),DEQUEUE_SLEEP |DEQUEUE_DELAYED );/* * Must not reference @se again, see __block_task(). */ @@ -5903,13 +5846,11 @@ static void put_prev_entity(struct cfs_rif (prev->on_rq) {update_stats_wait_start_fair (cfs_rq, prev); -/* Put 'current' back into the tree. */ -__enqueue_entity (cfs_rq, prev);/* in !on_rq case, update occurred at dequeue */ update_load_avg (cfs_rq, prev,0 ); } -WARN_ON_ONCE (cfs_rq->curr != prev); - cfs_rq->curr =NULL ; +WARN_ON_ONCE (cfs_rq->h_curr != prev); + cfs_rq->h_curr =NULL ; } static void@@ -6062,7 +6003,7 @@ static void __account_cfs_rq_runtime(str* if we're unable to extend our runtime we resched so that the active * hierarchy can be throttled */ -if (!assign_cfs_rq_runtime (cfs_rq) &&likely (cfs_rq->curr)) +if (!assign_cfs_rq_runtime (cfs_rq) &&likely (cfs_rq->h_curr))resched_curr (rq_of (cfs_rq)); }@@ -6420,7 +6361,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfassert_list_leaf_cfs_rq (rq);/* Determine whether we need to wake up potentially idle CPU: */ -if (rq->curr == rq->idle && rq->cfs.nr_queued) +if (rq->curr == rq->idle && rq->cfs.h_nr_queued)resched_curr (rq); }@@ -6761,7 +6702,7 @@ static void check_enqueue_throttle(strucreturn ;/* an active group must be handled by the update_curr()->put() path */ -if (!cfs_rq->runtime_enabled || cfs_rq->curr) +if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)return ;/* ensure the group is not already throttled */ @@ -7156,7 +7097,7 @@ static void hrtick_start_fair(struct rqresched_curr (rq);return ; } - delta = (se->load.weight * vdelta) /NICE_0_LOAD ; + delta = (se->h_load.weight * vdelta) /NICE_0_LOAD ;/* * Correct for instantaneous load of other classes. @@ -7256,10 +7197,8 @@ static int choose_idle_cpu(int cpu, stru }static void -requeue_delayed_entity (struct sched_entity *se) +requeue_delayed_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); -/* * se->sched_delayed should imply: se->on_rq == 1. * Because a delayed entity is one that is still on @@ -7269,19 +7208,58 @@ requeue_delayed_entity(struct sched_entiWARN_ON_ONCE (!se->on_rq);if (update_entity_lag (cfs_rq, se)) { - cfs_rq->nr_queued--; + cfs_rq->h_nr_queued--;if (se != cfs_rq->curr)__dequeue_entity (cfs_rq, se);place_entity (cfs_rq, se,0 );if (se != cfs_rq->curr)__enqueue_entity (cfs_rq, se); - cfs_rq->nr_queued++; + cfs_rq->h_nr_queued++; }update_load_avg (cfs_rq, se,0 );clear_delayed (se); } +static unsigned long enqueue_hierarchy (struct task_struct *p,int flags) +{ +unsigned long weight =NICE_0_LOAD ; +int task_new = !(flags &ENQUEUE_WAKEUP ); +struct sched_entity *se = &p->se; +int h_nr_idle =task_has_idle_policy (p); +int h_nr_runnable =1 ; + +if (task_new && se->sched_delayed) + h_nr_runnable =0 ; + +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +update_curr (cfs_rq); + +if (!se->on_rq) { +enqueue_entity (cfs_rq, se, flags); + }else { +update_load_avg (cfs_rq, se,UPDATE_TG ); +se_update_runnable (se); +update_cfs_group (se); + } + + cfs_rq->h_nr_runnable += h_nr_runnable; + cfs_rq->h_nr_queued++; + cfs_rq->h_nr_idle += h_nr_idle; + +if (cfs_rq_is_idle (cfs_rq)) + h_nr_idle =1 ; + + weight =__calc_prop_weight (cfs_rq, se, weight); + + flags =ENQUEUE_WAKEUP ; + } + +return weight; +} +/* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -7290,13 +7268,12 @@ requeue_delayed_entity(struct sched_entistatic void enqueue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -struct cfs_rq *cfs_rq; -struct sched_entity *se = &p->se; -int h_nr_idle =task_has_idle_policy (p); -int h_nr_runnable =1 ; -int task_new = !(flags &ENQUEUE_WAKEUP );int rq_h_nr_queued = rq->cfs.h_nr_queued; -u64 slice =0 ; +int task_new = !(flags &ENQUEUE_WAKEUP ); +struct sched_entity *se = &p->se; +struct cfs_rq *cfs_rq = &rq->cfs; +unsigned long weight; +bool curr;if (task_is_throttled (p) &&enqueue_throttled_task (p))return ;@@ -7308,10 +7285,10 @@ enqueue_task_fair(struct rq *rq, struct* estimated utilization, before we update schedutil. */ if (!p->se.sched_delayed || (flags &ENQUEUE_DELAYED )) -util_est_enqueue (&rq->cfs, p); +util_est_enqueue (cfs_rq, p);if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); +requeue_delayed_entity (cfs_rq, se);return ; }@@ -7323,57 +7300,22 @@ enqueue_task_fair(struct rq *rq, structif (p->in_iowait)cpufreq_update_util (rq,SCHED_CPUFREQ_IOWAIT ); -if (task_new && se->sched_delayed) - h_nr_runnable =0 ; - -for_each_sched_entity (se) { -if (se->on_rq) { -if (se->sched_delayed) -requeue_delayed_entity (se); -break ; - } - cfs_rq =cfs_rq_of (se); - -/* -* Basically set the slice of group entries to the min_slice of -* their respective cfs_rq. This ensures the group can service -* its entities in the desired time-frame. -*/ -if (slice) { - se->slice = slice; - se->custom_slice =1 ; - } -enqueue_entity (cfs_rq, se, flags); - slice =cfs_rq_min_slice (cfs_rq); - - cfs_rq->h_nr_runnable += h_nr_runnable; - cfs_rq->h_nr_queued++; - cfs_rq->h_nr_idle += h_nr_idle; - -if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle =1 ; - - flags =ENQUEUE_WAKEUP ; - } - -for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); - -update_load_avg (cfs_rq, se,UPDATE_TG ); -se_update_runnable (se); -update_cfs_group (se); +/* +* XXX comment on the curr thing +*/ + curr = (cfs_rq->curr == se); +if (curr) +place_entity (cfs_rq, se, flags); - se->slice = slice; -if (se != cfs_rq->curr) -min_vruntime_cb_propagate (&se->run_node,NULL ); - slice =cfs_rq_min_slice (cfs_rq); +if (se->on_rq && se->sched_delayed) +requeue_delayed_entity (cfs_rq, se); - cfs_rq->h_nr_runnable += h_nr_runnable; - cfs_rq->h_nr_queued++; - cfs_rq->h_nr_idle += h_nr_idle; + weight =enqueue_hierarchy (p, flags); -if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle =1 ; +if (!curr) { +reweight_eevdf (cfs_rq, se, weight, false); +place_entity (cfs_rq, se, flags |ENQUEUE_QUEUED ); +__enqueue_entity (cfs_rq, se); }if (!rq_h_nr_queued && rq->cfs.h_nr_queued)@@ -7404,105 +7346,107 @@ enqueue_task_fair(struct rq *rq, structhrtick_update (rq); } -/* -* Basically dequeue_task_fair(), except it can deal with dequeue_entity() -* failing half-way through and resume the dequeue later. -* -* Returns: -* -1 - dequeue delayed -* 0 - dequeue throttled -* 1 - dequeue complete -*/ -static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags) +static void dequeue_hierarchy (struct task_struct *p,int flags) { -bool was_sched_idle =sched_idle_rq (rq); +struct sched_entity *se = &p->se;bool task_sleep = flags &DEQUEUE_SLEEP ;bool task_delayed = flags &DEQUEUE_DELAYED ;bool task_throttled = flags &DEQUEUE_THROTTLE ; -struct task_struct *p =NULL ; -int h_nr_idle =0 ; -int h_nr_queued =0 ;int h_nr_runnable =0 ; -struct cfs_rq *cfs_rq; -u64 slice =0 ; +int h_nr_idle =task_has_idle_policy (p); +bool dequeue = true; -if (entity_is_task (se)) { - p =task_of (se); - h_nr_queued =1 ; - h_nr_idle =task_has_idle_policy (p); -if (task_sleep || task_delayed || !se->sched_delayed) - h_nr_runnable =1 ; - } +if (task_sleep || task_delayed || !se->sched_delayed) + h_nr_runnable =1 ;for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); +struct cfs_rq *cfs_rq =cfs_rq_of (se); -if (!dequeue_entity (cfs_rq, se, flags)) { -if (p && &p->se == se) -return -1 ; +update_curr (cfs_rq); - slice =cfs_rq_min_slice (cfs_rq); -break ; +if (dequeue) { +dequeue_entity (cfs_rq, se, flags); +/* Don't dequeue parent if it has other entities besides us */ +if (cfs_rq->load.weight) + dequeue = false; + }else { +update_load_avg (cfs_rq, se,UPDATE_TG ); +se_update_runnable (se); +update_cfs_group (se); } cfs_rq->h_nr_runnable -= h_nr_runnable; - cfs_rq->h_nr_queued -= h_nr_queued; + cfs_rq->h_nr_queued--; cfs_rq->h_nr_idle -= h_nr_idle;if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle = h_nr_queued; + h_nr_idle =1 ;if (throttled_hierarchy (cfs_rq) && task_throttled)record_throttle_clock (cfs_rq); -/* Don't dequeue parent if it has other entities besides us */ -if (cfs_rq->load.weight) { - slice =cfs_rq_min_slice (cfs_rq); - -/* Avoid re-evaluating load for this entity: */ - se =parent_entity (se); -/* -* Bias pick_next to pick a task from this cfs_rq, as -* p is sleeping when it is within its sched_slice. -*/ -if (task_sleep && se) -set_next_buddy (se); -break ; - } flags |=DEQUEUE_SLEEP ; flags &= ~(DEQUEUE_DELAYED |DEQUEUE_SPECIAL ); } +} -for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); +/* +* The part of dequeue_task_fair() that is needed to dequeue delayed tasks. +* +* Returns: +* true - dequeued +* false - delayed +*/ +static bool __dequeue_task (struct rq *rq,struct task_struct *p,int flags) +{ +struct sched_entity *se = &p->se; +struct cfs_rq *cfs_rq = &rq->cfs; +bool was_sched_idle =sched_idle_rq (rq); +bool task_sleep = flags &DEQUEUE_SLEEP ; +bool task_delayed = flags &DEQUEUE_DELAYED ; -update_load_avg (cfs_rq, se,UPDATE_TG ); -se_update_runnable (se); -update_cfs_group (se); +clear_buddies (cfs_rq, se); - se->slice = slice; -if (se != cfs_rq->curr) -min_vruntime_cb_propagate (&se->run_node,NULL ); - slice =cfs_rq_min_slice (cfs_rq); +if (flags &DEQUEUE_DELAYED ) { +WARN_ON_ONCE (!se->sched_delayed); + }else { +bool delay = task_sleep; +/* +* DELAY_DEQUEUE relies on spurious wakeups, special task +* states must not suffer spurious wakeups, excempt them. +*/ +if (flags & (DEQUEUE_SPECIAL |DEQUEUE_THROTTLE )) + delay = false;
kernel/sched/fair.c enqueue_task_fair(struct rq *rq, struct · line 7425
Quoted 10 lines
kernel/sched/fair.c +4
-5- cfs_rq->h_nr_runnable -= h_nr_runnable; - cfs_rq->h_nr_queued -= h_nr_queued; - cfs_rq->h_nr_idle -= h_nr_idle; +WARN_ON_ONCE (delay && se->sched_delayed); -if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle = h_nr_queued; +if (sched_feat (DELAY_DEQUEUE ) && delay && + !entity_eligible (cfs_rq, se)) { +update_load_avg (cfs_rq_of (se), se,0 );
update_entity_lag(cfs_rq, se); is missing here. Unfortunately this doesn't fix my regression
Quoted 512 lines
kernel/sched/fair.c +89
-126+set_delayed (se); +return false; + } + } -if (throttled_hierarchy (cfs_rq) && task_throttled) -record_throttle_clock (cfs_rq); +dequeue_hierarchy (p, flags); + +update_entity_lag (cfs_rq, se); +if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { + se->deadline -= se->vruntime; + se->rel_deadline =1 ; } +if (se != cfs_rq->curr) +__dequeue_entity (cfs_rq, se); -sub_nr_running (rq, h_nr_queued); +sub_nr_running (rq,1 );/* balance early to pull high priority tasks */ if (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; -if (p && task_delayed) { +if (task_delayed) { +clear_delayed (se); +WARN_ON_ONCE (!task_sleep);WARN_ON_ONCE (p->on_rq !=1 );@@ -7514,7 +7458,7 @@ static int dequeue_entities(struct rq *r__block_task (rq, p); } -return 1 ; +return true; }/* @@ -7533,11 +7477,11 @@ static bool dequeue_task_fair(struct rqutil_est_dequeue (&rq->cfs, p);util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP ); -if (dequeue_entities (rq, &p->se, flags) <0 ) +if (!__dequeue_task (rq, p, flags))return false;/* -* Must not reference @p after dequeue_entities(DEQUEUE_DELAYED). +* Must not reference @p after __dequeue_task(DEQUEUE_DELAYED). */ return true; }@@ -9021,19 +8965,6 @@ static void migrate_task_rq_fair(structstatic void task_dead_fair (struct task_struct *p) {struct sched_entity *se = &p->se; - -if (se->sched_delayed) { -struct rq_flags rf; -struct rq *rq; - - rq =task_rq_lock (p, &rf); -if (se->sched_delayed) { -update_rq_clock (rq); -dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); - } -task_rq_unlock (rq, p, &rf); - } -remove_entity_load_avg (se); }@@ -9067,21 +8998,10 @@ static void set_cpus_allowed_fair(structset_task_max_allowed_capacity (p); } -static void set_next_buddy (struct sched_entity *se) -{ -for_each_sched_entity (se) { -if (WARN_ON_ONCE (!se->on_rq)) -return ; -if (se_is_idle (se)) -return ; -cfs_rq_of (se)->next = se; - } -} -enum preempt_wakeup_action {PREEMPT_WAKEUP_NONE ,/* No preemption. */ PREEMPT_WAKEUP_SHORT ,/* Ignore slice protection. */ -PREEMPT_WAKEUP_PICK ,/* Let __pick_eevdf() decide. */ +PREEMPT_WAKEUP_PICK ,/* Let pick_eevdf() decide. */ PREEMPT_WAKEUP_RESCHED ,/* Force reschedule. */ };@@ -9098,7 +9018,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,if (cfs_rq->next &&entity_before (cfs_rq->next, pse))return false; -set_next_buddy (pse); +set_next_buddy (cfs_rq, pse);return true; }@@ -9188,7 +9108,6 @@ static void wakeup_preempt_fair(struct rif (!sched_feat (WAKEUP_PREEMPTION ))return ; -find_matching_se (&se, &pse);WARN_ON_ONCE (!pse); cse_is_idle =se_is_idle (se);@@ -9216,8 +9135,7 @@ static void wakeup_preempt_fair(struct rif (unlikely (!normal_policy (p->policy)))return ; - cfs_rq =cfs_rq_of (se); -update_curr (cfs_rq); +update_curr_fair (rq);/* * If @p has a shorter slice than current and @p is eligible, override * current's slice protection in order to allow preemption. @@ -9261,18 +9179,15 @@ static void wakeup_preempt_fair(struct r } pick: - nse =pick_next_entity (rq, cfs_rq, preempt_action !=PREEMPT_WAKEUP_SHORT ); -/* If @p has become the most eligible task, force preemption */ -if (nse == pse) - goto preempt; - -/* -* Because p is enqueued, nse being null can only mean that we -* dequeued a delayed task. If there are still entities queued in -* cfs, check if the next one will be p. -*/ -if (!nse && cfs_rq->nr_queued) - goto pick; +if (cfs_rq->h_nr_queued) { + nse =pick_next_entity (rq, preempt_action !=PREEMPT_WAKEUP_SHORT ); +if (unlikely (!nse)) + goto pick; + +/* If @p has become the most eligible task, force preemption */ +if (nse == pse) + goto preempt; + }if (sched_feat (RUN_TO_PARITY ))update_protect_slice (cfs_rq, se);@@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct rstruct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf)__must_hold (__rq_lockp (rq)) { +struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; -struct cfs_rq *cfs_rq;struct task_struct *p; -bool throttled;int new_tasks; again: - cfs_rq = &rq->cfs; -if (!cfs_rq->nr_queued) +if (!cfs_rq->h_nr_queued) goto idle; - throttled = false; - -do { -/* Might not have done put_prev_entity() */ -if (cfs_rq->curr && cfs_rq->curr->on_rq) -update_curr (cfs_rq); - - throttled |=check_cfs_rq_runtime (cfs_rq); +/* Might not have done put_prev_entity() */ +if (cfs_rq->curr && cfs_rq->curr->on_rq) +update_curr (cfs_rq); - se =pick_next_entity (rq, cfs_rq, true); -if (!se) - goto again; - cfs_rq =group_cfs_rq (se); - }while (cfs_rq); + se =pick_next_entity (rq, true); +if (!se) + goto again; p =task_of (se); -if (unlikely (throttled)) +if (unlikely (check_cfs_rq_runtime (cfs_rq_of (se))))task_throttle_setup_work (p);return p;@@ -9353,7 +9259,7 @@ void fair_server_init(struct rq *rq)static void put_prev_task_fair (struct rq *rq,struct task_struct *prev,struct task_struct *next) {struct sched_entity *se = &prev->se; -struct cfs_rq *cfs_rq; +struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *nse =NULL ;#ifdef CONFIG_FAIR_GROUP_SCHED @@ -9363,7 +9269,7 @@ static void put_prev_task_fair(struct rqwhile (se) { cfs_rq =cfs_rq_of (se); -if (!nse || cfs_rq->curr) +if (!nse || cfs_rq->h_curr)put_prev_entity (cfs_rq, se);#ifdef CONFIG_FAIR_GROUP_SCHED if (nse) {@@ -9382,6 +9288,14 @@ static void put_prev_task_fair(struct rq#endif se =parent_entity (se); } + +/* Put 'current' back into the tree. */ + cfs_rq = &rq->cfs; + se = &prev->se; +WARN_ON_ONCE (cfs_rq->curr != se); + cfs_rq->curr =NULL ; +if (se->on_rq) +__enqueue_entity (cfs_rq, se); }/* @@ -9390,8 +9304,8 @@ static void put_prev_task_fair(struct rqstatic void yield_task_fair (struct rq *rq) {struct task_struct *curr = rq->donor; -struct cfs_rq *cfs_rq =task_cfs_rq (curr);struct sched_entity *se = &curr->se; +struct cfs_rq *cfs_rq = &rq->cfs;/* * Are we the only task in the tree? @@ -9432,11 +9346,11 @@ static bool yield_to_task_fair(struct rqstruct sched_entity *se = &p->se;/* !se->on_rq also covers throttled task */ -if (!se->on_rq) +if (!se->on_rq || se->sched_delayed)return false;/* Tell the scheduler that we'd really like se to run next. */ -set_next_buddy (se); +set_next_buddy (&task_rq (p)->cfs, se);yield_task_fair (rq);@@ -9762,15 +9676,10 @@ static inline long migrate_degrades_loca*/ static inline int task_is_ineligible_on_dst_cpu (struct task_struct *p,int dest_cpu) { -struct cfs_rq *dst_cfs_rq; +struct cfs_rq *dst_cfs_rq = &cpu_rq (dest_cpu)->cfs; -#ifdef CONFIG_FAIR_GROUP_SCHED - dst_cfs_rq =task_group (p)->cfs_rq[dest_cpu]; -#else - dst_cfs_rq = &cpu_rq (dest_cpu)->cfs; -#endif -if (sched_feat (PLACE_LAG ) && dst_cfs_rq->nr_queued && - !entity_eligible (task_cfs_rq (p), &p->se)) +if (sched_feat (PLACE_LAG ) && dst_cfs_rq->h_nr_queued && + !entity_eligible (&task_rq (p)->cfs, &p->se))return 1 ;return 0 ;@@ -10240,7 +10149,7 @@ static void update_cfs_rq_h_load(structwhile ((se =READ_ONCE (cfs_rq->h_load_next)) !=NULL ) { load = cfs_rq->h_load; load =div64_ul (load * se->avg.load_avg, -cfs_rq_load_avg (cfs_rq) +1 ); +cfs_rq_load_avg (cfs_rq) +1 ); cfs_rq =group_cfs_rq (se); cfs_rq->h_load = load; cfs_rq->last_h_load_update = now;@@ -13459,7 +13368,7 @@ static inline void task_tick_core(struct* MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check * if we need to give up the CPU. */ -if (rq->core->core_forceidle_count && rq->cfs.nr_queued ==1 && +if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued ==1 &&__entity_slice_used (&curr->se,MIN_NR_TASKS_DURING_FORCEIDLE ))resched_curr (rq); }@@ -13668,30 +13577,8 @@ bool cfs_prio_less(const struct task_strWARN_ON_ONCE (task_rq (b)->core != rq->core); -#ifdef CONFIG_FAIR_GROUP_SCHED -/* -* Find an se in the hierarchy for tasks a and b, such that the se's -* are immediate siblings. -*/ -while (sea->cfs_rq->tg != seb->cfs_rq->tg) { -int sea_depth = sea->depth; -int seb_depth = seb->depth; - -if (sea_depth >= seb_depth) - sea =parent_entity (sea); -if (sea_depth <= seb_depth) - seb =parent_entity (seb); - } - -se_fi_update (sea, rq->core->core_forceidle_seq, in_fi); -se_fi_update (seb, rq->core->core_forceidle_seq, in_fi); - - cfs_rqa = sea->cfs_rq; - cfs_rqb = seb->cfs_rq; -#else /* !CONFIG_FAIR_GROUP_SCHED: */ cfs_rqa = &task_rq (a)->cfs; cfs_rqb = &task_rq (b)->cfs; -#endif /* !CONFIG_FAIR_GROUP_SCHED */ /* * Find delta after normalizing se's vruntime with its cfs_rq's @@ -13729,14 +13616,20 @@ static inline void task_tick_core(struct*/ static void task_tick_fair (struct rq *rq,struct task_struct *curr,int queued) { -struct cfs_rq *cfs_rq;struct sched_entity *se = &curr->se; +unsigned long weight =NICE_0_LOAD ; +struct cfs_rq *cfs_rq;for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se);entity_tick (cfs_rq, se, queued); + + weight =__calc_prop_weight (cfs_rq, se, weight); } + se = &curr->se; +reweight_eevdf (cfs_rq, se, weight, se->on_rq); +if (queued)return ;@@ -13772,7 +13665,7 @@ prio_changed_fair(struct rq *rq, structif (p->prio == oldprio)return ; -if (rq->cfs.nr_queued ==1 ) +if (rq->cfs.h_nr_queued ==1 )return ;/* @@ -13901,29 +13794,40 @@ static void switched_to_fair(struct rq * } } -/* -* Account for a task changing its policy or group. -* -* This routine is mostly called to set cfs_rq->curr field when a task -* migrates between groups/classes. -*/ static void set_next_task_fair (struct rq *rq,struct task_struct *p,bool first) {struct sched_entity *se = &p->se; +struct cfs_rq *cfs_rq = &rq->cfs; +unsigned long weight =NICE_0_LOAD ; +bool on_rq = se->on_rq; + +clear_buddies (cfs_rq, se); + +if (on_rq) +__dequeue_entity (cfs_rq, se);for_each_sched_entity (se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); + cfs_rq =cfs_rq_of (se); -if (IS_ENABLED (CONFIG_FAIR_GROUP_SCHED ) && - first && cfs_rq->curr) -break ; +if (!IS_ENABLED (CONFIG_FAIR_GROUP_SCHED ) || + !first || !cfs_rq->h_curr) +set_next_entity (cfs_rq, se); -set_next_entity (cfs_rq, se, first);/* ensure bandwidth has been allocated on our new cfs_rq */ account_cfs_rq_runtime (cfs_rq,0 ); + +if (on_rq) + weight =__calc_prop_weight (cfs_rq, se, weight); } se = &p->se; + cfs_rq->curr = se; + +if (on_rq) { +reweight_eevdf (cfs_rq, se, weight, se->on_rq); +if (first) +set_protect_slice (cfs_rq, se); + }if (task_on_rq_queued (p)) {/* @@ -14054,17 +13958,8 @@ void unregister_fair_sched_group(structstruct sched_entity *se = tg->se[cpu];struct rq *rq =cpu_rq (cpu); -if (se) { -if (se->sched_delayed) { -guard (rq_lock_irqsave)(rq); -if (se->sched_delayed) { -update_rq_clock (rq); -dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); - } -list_del_leaf_cfs_rq (cfs_rq); - } +if (se)remove_entity_load_avg (se); - }/* * Only empty task groups can be destroyed; so we can speculatively kernel/sched/pelt.c +3
-3@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched/* * running is a subset of runnable (weight) so running can't be set if * runnable is clear. But there are some corner cases where the current -* se has been already dequeued but cfs_rq->curr still points to it. +* se has been already dequeued but cfs_rq->h_curr still points to it. * This means that weight will be 0 but not running for a sched_entity * but also for a cfs_rq if the latter becomes idle. As an example, * this happens during sched_balance_newidle() which calls @@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 nowint __update_load_avg_se (u64 now,struct cfs_rq *cfs_rq,struct sched_entity *se) {if (___update_load_sum (now, &se->avg, !!se->on_rq,se_runnable (se), - cfs_rq->curr == se)) { + cfs_rq->h_curr == se)) {___update_load_avg (&se->avg,se_weight (se));cfs_se_util_change (&se->avg);@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, stif (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), cfs_rq->h_nr_runnable, - cfs_rq->curr !=NULL )) { + cfs_rq->h_curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );trace_pelt_cfs_tp (cfs_rq);kernel/sched/sched.h +13
-13@@ -528,21 +528,8 @@ struct task_group { }; -#ifdef CONFIG_GROUP_SCHED_WEIGHT #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD -/* -* A weight of 0 or 1 can cause arithmetics problems. -* A weight of a cfs_rq is the sum of weights of which entities -* are queued on this cfs_rq, so a weight of a entity should not be -* too large, so as the shares value of a task group. -* (The default weight is 1024 - so there's no practical -* limitation from this.) -*/ -#define MIN_SHARES (1UL << 1) -#define MAX_SHARES (1UL << 18) -#endif -typedef int (*tg_visitor )(struct task_group *,void *); extern int walk_tg_tree_from(struct task_group *from,@@ -629,6 +616,17 @@ static inline bool cfs_task_bw_constrain#endif /* !CONFIG_CGROUP_SCHED */ +/* +* A weight of 0 or 1 can cause arithmetics problems. +* A weight of a cfs_rq is the sum of weights of which entities +* are queued on this cfs_rq, so a weight of a entity should not be +* too large, so as the shares value of a task group. +* (The default weight is 1024 - so there's no practical +* limitation from this.) +*/ +#define MIN_SHARES (1UL << 1) +#define MAX_SHARES (1UL << 18) +extern void unregister_rt_sched_group (struct task_group *tg);extern void free_rt_sched_group (struct task_group *tg);extern int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent);@@ -707,6 +705,7 @@ struct cfs_rq {/* * CFS load tracking */ +struct sched_entity *h_curr;struct sched_avg avg;#ifndef CONFIG_64BIT u64 last_update_time_copy;@@ -2509,6 +2508,7 @@ extern const u32 sched_prio_to_wmult[40#define ENQUEUE_MIGRATED 0x00040000#define ENQUEUE_INITIAL 0x00080000#define ENQUEUE_RQ_SELECTED 0x00100000 +#define ENQUEUE_QUEUED 0x00200000#define RETRY_TASK ((void *)-1UL)
Vincent Guittot
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 50 lines · Vincent Guittot
Quoted 49 lines · Peter Zijlstra
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the
shares_weightapproximation are fully preserved.That is, given a hierarchy like:
R | se--G1 / \ G2--se se--G3 / \ |T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:
_R_ / | \T1 T2 T3
Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (
se->h_load) is computed from the hierarchical fraction:se->load/cfs_rq->load.Since EEVDF is now exclusive operating on
rq->cfs, it needs to considercfs_rq->h_nr_queuedrather thancfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.One place where additional information was required was
set_next_task()/put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).As a result of only having a single level to pick from, much of the complications in
pick_next_task()and preemption go away.
Quoted 6 lines
Quoted 6 lines
Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.
TODO: split struct
cfs_rq/ structsched_entity
TODO: try and get rid ofh_curr
I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed update_entity_lag():
update_entity_lag must be called after updating curr and cfs_rd and before clearing on_rq
With the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet
--- kernel/sched/fair.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/sched/fair.c +4 -1
@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)update_stats_dequeue_fair (cfs_rq, se, flags); +if (entity_is_task (se)) +update_entity_lag (&rq_of (cfs_rq)->cfs, se); + se->on_rq =0 ;account_entity_dequeue (cfs_rq, se);
@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (sched_feat (DELAY_DEQUEUE ) && delay && !entity_eligible (cfs_rq, se)) {update_load_avg (cfs_rq_of (se), se,0 ); +update_entity_lag (cfs_rq, se);set_delayed (se);return false; }
@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)dequeue_hierarchy (p, flags); -update_entity_lag (cfs_rq, se);if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;
Signature · 1 line
2.43.0
Quoted 1,639 lines
Quoted 1,113 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- include/linux/sched.h | 1 kernel/sched/core.c | 5 kernel/sched/debug.c | 9 kernel/sched/fair.c | 789 +++++++++++++++++++++----------------------------- kernel/sched/pelt.c | 6 kernel/sched/sched.h | 26 - 6 files changed, 366 insertions(+), 470 deletions(-)include/linux/sched.h +1
-0@@ -575,6 +575,7 @@ struct sched_statistics {struct sched_entity {/* For load-balancing: */ struct load_weight load; +struct load_weight h_load;struct rb_node run_node;u64 deadline;u64 min_vruntime;kernel/sched/core.c +1
-4@@ -5539,11 +5539,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);*/ static inline void prefetch_curr_exec_start (struct task_struct *p) { -#ifdef CONFIG_FAIR_GROUP_SCHED -struct sched_entity *curr = p->se.cfs_rq->curr; -#else struct sched_entity *curr =task_rq (p)->cfs.curr; -#endif +prefetch (curr);prefetch (&curr->exec_start); }kernel/sched/debug.c +6
-3@@ -911,10 +911,11 @@ print_task(struct seq_file *m, struct rqelse SEQ_printf (m," %c" ,task_state_to_char (p)); - SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", + SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ", p->comm, task_pid_nr(p), + p->se.h_load.weight, SPLIT_NS(p->se.vruntime), - entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N', + entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N', SPLIT_NS(p->se.deadline), p->se.custom_slice ? 'S' : ' ', SPLIT_NS(p->se.slice),@@ -943,7 +944,7 @@ static void print_rq(struct seq_file *m,SEQ_printf (m,"\n" );SEQ_printf (m,"runnable tasks:\n" ); -SEQ_printf (m," S task PID vruntime eligible " +SEQ_printf (m," S task PID weight vruntime eligible " "deadline slice sum-exec switches " "prio wait-time sum-sleep sum-block" #ifdef CONFIG_NUMA_BALANCING@@ -1051,6 +1052,8 @@ void print_cfs_rq(struct seq_file *m, in cfs_rq->tg_load_avg_contrib);SEQ_printf (m," .%-30s: %ld\n" ,"tg_load_avg" ,atomic_long_read (&cfs_rq->tg->load_avg)); +SEQ_printf (m," .%-30s: %lu\n" ,"h_load" , + cfs_rq->h_load);#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_CFS_BANDWIDTH SEQ_printf(m, " .%-30s: %d\n", "throttled",kernel/sched/fair.c +249
-316@@ -296,8 +296,8 @@ static u64 __calc_delta(u64 delta_exec,*/ static inline u64 calc_delta_fair (u64 delta,struct sched_entity *se) { -if (unlikely (se->load.weight !=NICE_0_LOAD )) - delta =__calc_delta (delta,NICE_0_LOAD , &se->load); +if (se->h_load.weight !=NICE_0_LOAD ) + delta =__calc_delta (delta,NICE_0_LOAD , &se->h_load);return delta; }@@ -427,38 +427,6 @@ static inline struct sched_entity *parenreturn se->parent; } -static void -find_matching_se (struct sched_entity **se,struct sched_entity **pse) -{ -int se_depth, pse_depth; - -/* -* preemption test can be made between sibling entities who are in the -* same cfs_rq i.e who have a common parent. Walk up the hierarchy of -* both tasks until we find their ancestors who are siblings of common -* parent. -*/ - -/* First walk up until both entities are at same depth */ - se_depth = (*se)->depth; - pse_depth = (*pse)->depth; - -while (se_depth > pse_depth) { - se_depth--; - *se =parent_entity (*se); - } - -while (pse_depth > se_depth) { - pse_depth--; - *pse =parent_entity (*pse); - } - -while (!is_same_group (*se, *pse)) { - *se =parent_entity (*se); - *pse =parent_entity (*pse); - } -} -static int tg_is_idle (struct task_group *tg) {return tg->idle >0 ;@@ -502,11 +470,6 @@ static inline struct sched_entity *parenreturn NULL ; } -static inline void -find_matching_se (struct sched_entity **se,struct sched_entity **pse) -{ -} -static inline int tg_is_idle (struct task_group *tg) {return 0 ;@@ -685,7 +648,7 @@ static inline unsigned long avg_vruntimestatic inline void __sum_w_vruntime_add (struct cfs_rq *cfs_rq,struct sched_entity *se) { -unsigned long weight =avg_vruntime_weight (cfs_rq, se->load.weight); +unsigned long weight =avg_vruntime_weight (cfs_rq, se->h_load.weight);s64 w_vruntime, key =entity_key (cfs_rq, se); w_vruntime = key * weight;@@ -702,7 +665,7 @@ sum_w_vruntime_add_paranoid(struct cfs_rs64 key, tmp; again: - weight =avg_vruntime_weight (cfs_rq, se->load.weight); + weight =avg_vruntime_weight (cfs_rq, se->h_load.weight); key =entity_key (cfs_rq, se);if (check_mul_overflow (key, weight, &key))@@ -748,7 +711,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rqstatic void sum_w_vruntime_sub (struct cfs_rq *cfs_rq,struct sched_entity *se) { -unsigned long weight =avg_vruntime_weight (cfs_rq, se->load.weight); +unsigned long weight =avg_vruntime_weight (cfs_rq, se->h_load.weight);s64 key =entity_key (cfs_rq, se); cfs_rq->sum_w_vruntime -= key * weight;@@ -790,7 +753,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)s64 runtime = cfs_rq->sum_w_vruntime;if (curr) { -unsigned long w =avg_vruntime_weight (cfs_rq, curr->load.weight); +unsigned long w =avg_vruntime_weight (cfs_rq, curr->h_load.weight); runtime +=entity_key (cfs_rq, curr) * w; weight += w;@@ -861,8 +824,6 @@ bool update_entity_lag(struct cfs_rq *cfu64 avruntime =avg_vruntime (cfs_rq);s64 vlag =entity_lag (cfs_rq, se, avruntime); -WARN_ON_ONCE (!se->on_rq); -if (se->sched_delayed) {/* previous vlag < 0 otherwise se would not be delayed */ vlag =max (vlag, se->vlag);@@ -898,7 +859,7 @@ static int vruntime_eligible(struct cfs_long load = cfs_rq->sum_weight;if (curr && curr->on_rq) { -unsigned long weight =avg_vruntime_weight (cfs_rq, curr->load.weight); +unsigned long weight =avg_vruntime_weight (cfs_rq, curr->h_load.weight); avg +=entity_key (cfs_rq, curr) * weight; load += weight;@@ -1039,6 +1000,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim*/ static void __enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (&rq_of (cfs_rq)->cfs != cfs_rq); +WARN_ON_ONCE (!entity_is_task (se)); +sum_w_vruntime_add (cfs_rq, se); se->min_vruntime = se->vruntime; se->min_slice = se->slice;@@ -1048,6 +1012,9 @@ static void __enqueue_entity(struct cfs_static void __dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (&rq_of (cfs_rq)->cfs != cfs_rq); +WARN_ON_ONCE (!entity_is_task (se)); +rb_erase_augmented_cached (&se->run_node, &cfs_rq->tasks_timeline, &min_vruntime_cb);sum_w_vruntime_sub (cfs_rq, se);@@ -1144,7 +1111,7 @@ static struct sched_entity *pick_eevdf(s* We can safely skip eligibility check if there is only one entity * in this cfs_rq, saving some cycles. */ -if (cfs_rq->nr_queued ==1 ) +if (cfs_rq->h_nr_queued ==1 )return curr && curr->on_rq ? curr : se;/* @@ -1391,8 +1358,6 @@ static s64 update_se(struct rq *rq, strureturn delta_exec; } -static void set_next_buddy (struct sched_entity *se); -/* * Used by other classes to account runtime. */ @@ -1412,7 +1377,7 @@ static void update_curr(struct cfs_rq *c* not necessarily be the actual task running * (rq->curr.se). This is easy to confuse! */ -struct sched_entity *curr = cfs_rq->curr; +struct sched_entity *curr = cfs_rq->h_curr;struct rq *rq =rq_of (cfs_rq);s64 delta_exec;bool resched;@@ -1424,26 +1389,29 @@ static void update_curr(struct cfs_rq *cif (unlikely (delta_exec <=0 ))return ; +account_cfs_rq_runtime (cfs_rq, delta_exec); + +if (!entity_is_task (curr)) +return ; + + cfs_rq = &rq->cfs; + curr->vruntime +=calc_delta_fair (delta_exec, curr); resched =update_deadline (cfs_rq, curr); -if (entity_is_task (curr)) { -/* -* If the fair_server is active, we need to account for the -* fair_server time whether or not the task is running on -* behalf of fair_server or not: -* - If the task is running on behalf of fair_server, we need -* to limit its time based on the assigned runtime. -* - Fair task that runs outside of fair_server should account -* against fair_server such that it can account for this time -* and possibly avoid running this period. -*/ -dl_server_update (&rq->fair_server, delta_exec); - } - -account_cfs_rq_runtime (cfs_rq, delta_exec); +/* +* If the fair_server is active, we need to account for the +* fair_server time whether or not the task is running on +* behalf of fair_server or not: +* - If the task is running on behalf of fair_server, we need +* to limit its time based on the assigned runtime. +* - Fair task that runs outside of fair_server should account +* against fair_server such that it can account for this time +* and possibly avoid running this period. +*/ +dl_server_update (&rq->fair_server, delta_exec); -if (cfs_rq->nr_queued ==1 ) +if (cfs_rq->h_nr_queued ==1 )return ;if (resched || !protect_slice (curr)) {@@ -1454,7 +1422,10 @@ static void update_curr(struct cfs_rq *cstatic void update_curr_fair (struct rq *rq) { -update_curr (cfs_rq_of (&rq->donor->se)); +struct sched_entity *se = &rq->donor->se; + +for_each_sched_entity (se) +update_curr (cfs_rq_of (se)); } static inline void@@ -1530,7 +1501,7 @@ update_stats_enqueue_fair(struct cfs_rq* Are we enqueueing a waiting task? (for current tasks * a dequeue/enqueue event is a NOP) */ -if (se != cfs_rq->curr) +if (se != cfs_rq->h_curr)update_stats_wait_start_fair (cfs_rq, se);if (flags &ENQUEUE_WAKEUP )@@ -1548,7 +1519,7 @@ update_stats_dequeue_fair(struct cfs_rq* Mark the end of the wait period if dequeueing a * waiting task: */ -if (se != cfs_rq->curr) +if (se != cfs_rq->h_curr)update_stats_wait_end_fair (cfs_rq, se);if ((flags &DEQUEUE_SLEEP ) &&entity_is_task (se)) {@@ -3875,6 +3846,7 @@ static inline void update_scan_period(ststatic void account_entity_enqueue (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (cfs_rq !=cfs_rq_of (se));update_load_add (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {struct rq *rq =rq_of (cfs_rq);@@ -3888,6 +3860,7 @@ account_entity_enqueue(struct cfs_rq *cfstatic void account_entity_dequeue (struct cfs_rq *cfs_rq,struct sched_entity *se) { +WARN_ON_ONCE (cfs_rq !=cfs_rq_of (se));update_load_sub (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se));@@ -3965,7 +3938,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,static void rescale_entity (struct sched_entity *se,unsigned long weight,bool rel_vprot) { -unsigned long old_weight = se->load.weight; +long old_weight = se->h_load.weight;/* * VRUNTIME @@ -4065,16 +4038,17 @@ rescale_entity(struct sched_entity *se, se->vprot =div64_long (se->vprot * old_weight, weight); } -static void reweight_entity (struct cfs_rq *cfs_rq,struct sched_entity *se, -unsigned long weight) +static void reweight_eevdf (struct cfs_rq *cfs_rq,struct sched_entity *se, +unsigned long weight,bool on_rq) {bool curr = cfs_rq->curr == se;bool rel_vprot = false;u64 avruntime =0 ; -if (se->on_rq) { -/* commit outstanding execution time */ -update_curr (cfs_rq); +if (se->h_load.weight == weight) +return ; + +if (on_rq) { avruntime =avg_vruntime (cfs_rq); se->vlag =entity_lag (cfs_rq, se, avruntime); se->deadline -= avruntime;@@ -4084,46 +4058,90 @@ static void reweight_entity(struct cfs_r rel_vprot = true; } - cfs_rq->nr_queued--; + cfs_rq->h_nr_queued--;if (!curr)__dequeue_entity (cfs_rq, se); -update_load_sub (&cfs_rq->load, se->load.weight); } -dequeue_load_avg (cfs_rq, se);rescale_entity (se, weight, rel_vprot); -update_load_set (&se->load, weight); +update_load_set (&se->h_load, weight); -do { -u32 divider =get_pelt_divider (&se->avg); - se->avg.load_avg =div_u64 (se_weight (se) * se->avg.load_sum, divider); - }while (0 ); - -enqueue_load_avg (cfs_rq, se); -if (se->on_rq) { +if (on_rq) {if (rel_vprot) se->vprot += avruntime; se->deadline += avruntime; se->rel_deadline =0 ; se->vruntime = avruntime - se->vlag; -update_load_add (&cfs_rq->load, se->load.weight);if (!curr)__enqueue_entity (cfs_rq, se); - cfs_rq->nr_queued++; + cfs_rq->h_nr_queued++; } } +static void reweight_entity (struct cfs_rq *cfs_rq,struct sched_entity *se, +unsigned long weight) +{ +if (se->load.weight == weight) +return ; + +if (se->on_rq) { +WARN_ON_ONCE (cfs_rq !=cfs_rq_of (se)); +update_load_sub (&cfs_rq->load, se->load.weight); + } +dequeue_load_avg (cfs_rq, se); + +update_load_set (&se->load, weight); + +do { +u32 divider =get_pelt_divider (&se->avg); + se->avg.load_avg =div_u64 (se_weight (se) * se->avg.load_sum, divider); + }while (0 ); + +enqueue_load_avg (cfs_rq, se); + +if (se->on_rq) +update_load_add (&cfs_rq->load, se->load.weight); +} + +/* +* weight = NICE_0_LOAD; +* for_each_entity_se(se) +* weight = __calc_prop_weight(cfs_rq_of(se), se, weight); +*/ +static __always_inline +unsigned long __calc_prop_weight (struct cfs_rq *cfs_rq,struct sched_entity *se, +unsigned long weight) +{ + weight *= se->load.weight; +if (parent_entity (se)) + weight /= cfs_rq->load.weight; +else + weight /=NICE_0_LOAD ; + +return max (weight,MIN_SHARES ); +} +static void reweight_task_fair (struct rq *rq,struct task_struct *p,const struct load_weight *lw) {struct sched_entity *se = &p->se; -struct cfs_rq *cfs_rq =cfs_rq_of (se); -struct load_weight *load = &se->load; +unsigned long weight =NICE_0_LOAD ; + +if (se->on_rq) +update_curr_fair (rq); + +reweight_entity (cfs_rq_of (se), se, lw->weight); + se->load.inv_weight = lw->inv_weight; + +if (!se->on_rq) +return ; + +for_each_sched_entity (se) + weight =__calc_prop_weight (cfs_rq_of (se), se, weight); -reweight_entity (cfs_rq, se, lw->weight); - load->inv_weight = lw->inv_weight; +reweight_eevdf (&rq->cfs, &p->se, weight, p->se.on_rq); }static inline int throttled_hierarchy (struct cfs_rq *cfs_rq);@@ -4331,7 +4349,6 @@ static long calc_group_shares(struct cfsstatic void update_cfs_group (struct sched_entity *se) {struct cfs_rq *gcfs_rq =group_cfs_rq (se); -long shares;/* * When a group becomes empty, preserve its weight. This matters for @@ -4340,9 +4357,7 @@ static void update_cfs_group(struct scheif (!gcfs_rq || !gcfs_rq->load.weight)return ; - shares =calc_group_shares (gcfs_rq); -if (unlikely (se->load.weight != shares)) - reweight_entity(cfs_rq_of(se), se, shares); + reweight_entity(cfs_rq_of(se), se, calc_group_shares(gcfs_rq)); }#else /* !CONFIG_FAIR_GROUP_SCHED: */ @@ -4460,7 +4475,7 @@ static inline bool cfs_rq_is_decayed(str* differential update where we store the last value we propagated. This in * turn allows skipping updates if the differential is 'small'. * -* Updating tg's load_avg is necessary before update_cfs_share(). +* Updating tg's load_avg is necessary before update_cfs_group(). */ static inline void update_tg_load_avg (struct cfs_rq *cfs_rq) {@@ -4926,7 +4941,7 @@ static void migrate_se_pelt_lag(struct s* The cfs_rq avg is the direct sum of all its entities (blocked and runnable) * avg. The immediate corollary is that all (fair) tasks must be attached. * -* cfs_rq->avg is used for task_h_load() and update_cfs_share() for example. +* cfs_rq->avg is used for task_h_load() and update_cfs_group() for example. * * Return: true if the load decayed or we removed load. * @@ -5475,6 +5490,7 @@ static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { u64 vslice, vruntime = avg_vruntime(cfs_rq); +unsigned int nr_queued = cfs_rq->h_nr_queued;bool update_zero = false;s64 lag =0 ;@@ -5482,6 +5498,9 @@ place_entity(struct cfs_rq *cfs_rq, stru se->slice = sysctl_sched_base_slice; vslice =calc_delta_fair (se->slice, se); +if (flags &ENQUEUE_QUEUED ) + nr_queued -=1 ; +/* * Due to how V is constructed as the weighted average of entities, * adding tasks with positive lag, or removing tasks with negative lag @@ -5490,7 +5509,7 @@ place_entity(struct cfs_rq *cfs_rq, stru* * EEVDF: placement strategy #1 / #2 */ -if (sched_feat (PLACE_LAG ) && cfs_rq->nr_queued && se->vlag) { +if (sched_feat (PLACE_LAG ) && nr_queued && se->vlag) {struct sched_entity *curr = cfs_rq->curr;long load, weight;@@ -5550,9 +5569,9 @@ place_entity(struct cfs_rq *cfs_rq, stru*/ load = cfs_rq->sum_weight;if (curr && curr->on_rq) - load +=avg_vruntime_weight (cfs_rq, curr->load.weight); + load +=avg_vruntime_weight (cfs_rq, curr->h_load.weight); - weight =avg_vruntime_weight (cfs_rq, se->load.weight); + weight =avg_vruntime_weight (cfs_rq, se->h_load.weight); lag *= load + weight;if (WARN_ON_ONCE (!load)) load =1 ;@@ -5611,22 +5630,8 @@ static void check_enqueue_throttle(strucstatic inline int cfs_rq_throttled (struct cfs_rq *cfs_rq);static void -requeue_delayed_entity (struct sched_entity *se); - -static void enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -bool curr = cfs_rq->curr == se; - -/* -* If we're the current task, we must renormalise before calling -* update_curr(). -*/ -if (curr) -place_entity (cfs_rq, se, flags); - -update_curr (cfs_rq); -/* * When enqueuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. @@ -5645,13 +5650,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st*/ update_cfs_group (se); -/* -* XXX now that the entity has been re-weighted, and it's lag adjusted, -* we can place the entity. -*/ -if (!curr) -place_entity (cfs_rq, se, flags); -account_entity_enqueue (cfs_rq, se);/* Entity has migrated, no longer consider this task hot */ @@ -5660,8 +5658,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, stcheck_schedstat_required ();update_stats_enqueue_fair (cfs_rq, se, flags); -if (!curr) -__enqueue_entity (cfs_rq, se); se->on_rq =1 ;if (cfs_rq->nr_queued ==1 ) {@@ -5679,21 +5675,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st } } -static void __clear_buddies_next (struct sched_entity *se) +static void set_next_buddy (struct cfs_rq *cfs_rq,struct sched_entity *se) { -for_each_sched_entity (se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); -if (cfs_rq->next != se) -break ; - - cfs_rq->next =NULL ; - } +if (WARN_ON_ONCE (!se->on_rq || se->sched_delayed)) +return ; +if (se_is_idle (se)) +return ; + cfs_rq->next = se; }static void clear_buddies (struct cfs_rq *cfs_rq,struct sched_entity *se) {if (cfs_rq->next == se) -__clear_buddies_next (se); + cfs_rq->next =NULL ; } static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);@@ -5704,7 +5698,7 @@ static void set_delayed(struct sched_ent/* * Delayed se of cfs_rq have no tasks queued on them. -* Do not adjust h_nr_runnable since dequeue_entities() +* Do not adjust h_nr_runnable since __dequeue_task() * will account it for blocked tasks. */ if (!entity_is_task (se))@@ -5737,37 +5731,11 @@ static void clear_delayed(struct sched_e } } -static bool +static void dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -bool sleep = flags &DEQUEUE_SLEEP ;int action =UPDATE_TG ; -update_curr (cfs_rq); -clear_buddies (cfs_rq, se); - -if (flags &DEQUEUE_DELAYED ) { -WARN_ON_ONCE (!se->sched_delayed); - }else { -bool delay = sleep; -/* -* DELAY_DEQUEUE relies on spurious wakeups, special task -* states must not suffer spurious wakeups, excempt them. -*/ -if (flags & (DEQUEUE_SPECIAL |DEQUEUE_THROTTLE )) - delay = false; - -WARN_ON_ONCE (delay && se->sched_delayed); - -if (sched_feat (DELAY_DEQUEUE ) && delay && - !entity_eligible (cfs_rq, se)) { -update_load_avg (cfs_rq, se,0 ); -update_entity_lag (cfs_rq, se); -set_delayed (se); -return false; - } - } -if (entity_is_task (se) &&task_on_rq_migrating (task_of (se))) action |=DO_DETACH ;@@ -5785,14 +5753,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_stats_dequeue_fair (cfs_rq, se, flags); -update_entity_lag (cfs_rq, se); -if (sched_feat (PLACE_REL_DEADLINE ) && !sleep) { - se->deadline -= se->vruntime; - se->rel_deadline =1 ; - } - -if (se != cfs_rq->curr) -__dequeue_entity (cfs_rq, se); se->on_rq =0 ;account_entity_dequeue (cfs_rq, se);@@ -5801,9 +5761,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, stupdate_cfs_group (se); -if (flags &DEQUEUE_DELAYED ) -clear_delayed (se); -if (cfs_rq->nr_queued ==0 ) {update_idle_cfs_rq_clock_pelt (cfs_rq);#ifdef CONFIG_CFS_BANDWIDTH @@ -5816,15 +5773,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st }#endif } - -return true; }static void -set_next_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,bool first) +set_next_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { -clear_buddies (cfs_rq, se); -/* 'current' is not kept within the tree. */ if (se->on_rq) {/* @@ -5833,16 +5786,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s* runqueue. */ update_stats_wait_end_fair (cfs_rq, se); -__dequeue_entity (cfs_rq, se);update_load_avg (cfs_rq, se,UPDATE_TG ); - -if (first) -set_protect_slice (cfs_rq, se); }update_stats_curr_start (cfs_rq, se); -WARN_ON_ONCE (cfs_rq->curr); - cfs_rq->curr = se; +WARN_ON_ONCE (cfs_rq->h_curr); + cfs_rq->h_curr = se;/* * Track our maximum slice length, if the CPU's load is at @@ -5862,23 +5811,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s se->prev_sum_exec_runtime = se->sum_exec_runtime; } -static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags); +static bool __dequeue_task (struct rq *rq,struct task_struct *p,int flags); -/* -* Pick the next process, keeping these things in mind, in this order: -* 1) keep things fair between processes/task groups -* 2) pick the "next" process, since someone really wants that to run -* 3) pick the "last" process, for cache locality -* 4) do not run the "skip" process, if something else is available -*/ static struct sched_entity * -pick_next_entity (struct rq *rq,struct cfs_rq *cfs_rq,bool protect) +pick_next_entity (struct rq *rq,bool protect) { +struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; se =pick_eevdf (cfs_rq, protect);if (se->sched_delayed) { -dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +__dequeue_task (rq,task_of (se),DEQUEUE_SLEEP |DEQUEUE_DELAYED );/* * Must not reference @se again, see __block_task(). */ @@ -5903,13 +5846,11 @@ static void put_prev_entity(struct cfs_rif (prev->on_rq) {update_stats_wait_start_fair (cfs_rq, prev); -/* Put 'current' back into the tree. */ -__enqueue_entity (cfs_rq, prev);/* in !on_rq case, update occurred at dequeue */ update_load_avg (cfs_rq, prev,0 ); } -WARN_ON_ONCE (cfs_rq->curr != prev); - cfs_rq->curr =NULL ; +WARN_ON_ONCE (cfs_rq->h_curr != prev); + cfs_rq->h_curr =NULL ; } static void@@ -6062,7 +6003,7 @@ static void __account_cfs_rq_runtime(str* if we're unable to extend our runtime we resched so that the active * hierarchy can be throttled */ -if (!assign_cfs_rq_runtime (cfs_rq) &&likely (cfs_rq->curr)) +if (!assign_cfs_rq_runtime (cfs_rq) &&likely (cfs_rq->h_curr))resched_curr (rq_of (cfs_rq)); }@@ -6420,7 +6361,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfassert_list_leaf_cfs_rq (rq);/* Determine whether we need to wake up potentially idle CPU: */ -if (rq->curr == rq->idle && rq->cfs.nr_queued) +if (rq->curr == rq->idle && rq->cfs.h_nr_queued)resched_curr (rq); }@@ -6761,7 +6702,7 @@ static void check_enqueue_throttle(strucreturn ;/* an active group must be handled by the update_curr()->put() path */ -if (!cfs_rq->runtime_enabled || cfs_rq->curr) +if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)return ;/* ensure the group is not already throttled */ @@ -7156,7 +7097,7 @@ static void hrtick_start_fair(struct rqresched_curr (rq);return ; } - delta = (se->load.weight * vdelta) /NICE_0_LOAD ; + delta = (se->h_load.weight * vdelta) /NICE_0_LOAD ;/* * Correct for instantaneous load of other classes. @@ -7256,10 +7197,8 @@ static int choose_idle_cpu(int cpu, stru }static void -requeue_delayed_entity (struct sched_entity *se) +requeue_delayed_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); -/* * se->sched_delayed should imply: se->on_rq == 1. * Because a delayed entity is one that is still on @@ -7269,19 +7208,58 @@ requeue_delayed_entity(struct sched_entiWARN_ON_ONCE (!se->on_rq);if (update_entity_lag (cfs_rq, se)) { - cfs_rq->nr_queued--; + cfs_rq->h_nr_queued--;if (se != cfs_rq->curr)__dequeue_entity (cfs_rq, se);place_entity (cfs_rq, se,0 );if (se != cfs_rq->curr)__enqueue_entity (cfs_rq, se); - cfs_rq->nr_queued++; + cfs_rq->h_nr_queued++; }update_load_avg (cfs_rq, se,0 );clear_delayed (se); } +static unsigned long enqueue_hierarchy (struct task_struct *p,int flags) +{ +unsigned long weight =NICE_0_LOAD ; +int task_new = !(flags &ENQUEUE_WAKEUP ); +struct sched_entity *se = &p->se; +int h_nr_idle =task_has_idle_policy (p); +int h_nr_runnable =1 ; + +if (task_new && se->sched_delayed) + h_nr_runnable =0 ; + +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +update_curr (cfs_rq); + +if (!se->on_rq) { +enqueue_entity (cfs_rq, se, flags); + }else { +update_load_avg (cfs_rq, se,UPDATE_TG ); +se_update_runnable (se); +update_cfs_group (se); + } + + cfs_rq->h_nr_runnable += h_nr_runnable; + cfs_rq->h_nr_queued++; + cfs_rq->h_nr_idle += h_nr_idle; + +if (cfs_rq_is_idle (cfs_rq)) + h_nr_idle =1 ; + + weight =__calc_prop_weight (cfs_rq, se, weight); + + flags =ENQUEUE_WAKEUP ; + } + +return weight; +} +/* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -7290,13 +7268,12 @@ requeue_delayed_entity(struct sched_entistatic void enqueue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -struct cfs_rq *cfs_rq; -struct sched_entity *se = &p->se; -int h_nr_idle =task_has_idle_policy (p); -int h_nr_runnable =1 ; -int task_new = !(flags &ENQUEUE_WAKEUP );int rq_h_nr_queued = rq->cfs.h_nr_queued; -u64 slice =0 ; +int task_new = !(flags &ENQUEUE_WAKEUP ); +struct sched_entity *se = &p->se; +struct cfs_rq *cfs_rq = &rq->cfs; +unsigned long weight; +bool curr;if (task_is_throttled (p) &&enqueue_throttled_task (p))return ;@@ -7308,10 +7285,10 @@ enqueue_task_fair(struct rq *rq, struct* estimated utilization, before we update schedutil. */ if (!p->se.sched_delayed || (flags &ENQUEUE_DELAYED )) -util_est_enqueue (&rq->cfs, p); +util_est_enqueue (cfs_rq, p);if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); +requeue_delayed_entity (cfs_rq, se);return ; }@@ -7323,57 +7300,22 @@ enqueue_task_fair(struct rq *rq, structif (p->in_iowait)cpufreq_update_util (rq,SCHED_CPUFREQ_IOWAIT ); -if (task_new && se->sched_delayed) - h_nr_runnable =0 ; - -for_each_sched_entity (se) { -if (se->on_rq) { -if (se->sched_delayed) -requeue_delayed_entity (se); -break ; - } - cfs_rq =cfs_rq_of (se); - -/* -* Basically set the slice of group entries to the min_slice of -* their respective cfs_rq. This ensures the group can service -* its entities in the desired time-frame. -*/ -if (slice) { - se->slice = slice; - se->custom_slice =1 ; - } -enqueue_entity (cfs_rq, se, flags); - slice =cfs_rq_min_slice (cfs_rq); - - cfs_rq->h_nr_runnable += h_nr_runnable; - cfs_rq->h_nr_queued++; - cfs_rq->h_nr_idle += h_nr_idle; - -if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle =1 ; - - flags =ENQUEUE_WAKEUP ; - } - -for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); - -update_load_avg (cfs_rq, se,UPDATE_TG ); -se_update_runnable (se); -update_cfs_group (se); +/* +* XXX comment on the curr thing +*/ + curr = (cfs_rq->curr == se); +if (curr) +place_entity (cfs_rq, se, flags); - se->slice = slice; -if (se != cfs_rq->curr) -min_vruntime_cb_propagate (&se->run_node,NULL ); - slice =cfs_rq_min_slice (cfs_rq); +if (se->on_rq && se->sched_delayed) +requeue_delayed_entity (cfs_rq, se); - cfs_rq->h_nr_runnable += h_nr_runnable; - cfs_rq->h_nr_queued++; - cfs_rq->h_nr_idle += h_nr_idle; + weight =enqueue_hierarchy (p, flags); -if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle =1 ; +if (!curr) { +reweight_eevdf (cfs_rq, se, weight, false); +place_entity (cfs_rq, se, flags |ENQUEUE_QUEUED ); +__enqueue_entity (cfs_rq, se); }if (!rq_h_nr_queued && rq->cfs.h_nr_queued)@@ -7404,105 +7346,107 @@ enqueue_task_fair(struct rq *rq, structhrtick_update (rq); } -/* -* Basically dequeue_task_fair(), except it can deal with dequeue_entity() -* failing half-way through and resume the dequeue later. -* -* Returns: -* -1 - dequeue delayed -* 0 - dequeue throttled -* 1 - dequeue complete -*/ -static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags) +static void dequeue_hierarchy (struct task_struct *p,int flags) { -bool was_sched_idle =sched_idle_rq (rq); +struct sched_entity *se = &p->se;bool task_sleep = flags &DEQUEUE_SLEEP ;bool task_delayed = flags &DEQUEUE_DELAYED ;bool task_throttled = flags &DEQUEUE_THROTTLE ; -struct task_struct *p =NULL ; -int h_nr_idle =0 ; -int h_nr_queued =0 ;int h_nr_runnable =0 ; -struct cfs_rq *cfs_rq; -u64 slice =0 ; +int h_nr_idle =task_has_idle_policy (p); +bool dequeue = true; -if (entity_is_task (se)) { - p =task_of (se); - h_nr_queued =1 ; - h_nr_idle =task_has_idle_policy (p); -if (task_sleep || task_delayed || !se->sched_delayed) - h_nr_runnable =1 ; - } +if (task_sleep || task_delayed || !se->sched_delayed) + h_nr_runnable =1 ;for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); +struct cfs_rq *cfs_rq =cfs_rq_of (se); -if (!dequeue_entity (cfs_rq, se, flags)) { -if (p && &p->se == se) -return -1 ; +update_curr (cfs_rq); - slice =cfs_rq_min_slice (cfs_rq); -break ; +if (dequeue) { +dequeue_entity (cfs_rq, se, flags); +/* Don't dequeue parent if it has other entities besides us */ +if (cfs_rq->load.weight) + dequeue = false; + }else { +update_load_avg (cfs_rq, se,UPDATE_TG ); +se_update_runnable (se); +update_cfs_group (se); } cfs_rq->h_nr_runnable -= h_nr_runnable; - cfs_rq->h_nr_queued -= h_nr_queued; + cfs_rq->h_nr_queued--; cfs_rq->h_nr_idle -= h_nr_idle;if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle = h_nr_queued; + h_nr_idle =1 ;if (throttled_hierarchy (cfs_rq) && task_throttled)record_throttle_clock (cfs_rq); -/* Don't dequeue parent if it has other entities besides us */ -if (cfs_rq->load.weight) { - slice =cfs_rq_min_slice (cfs_rq); - -/* Avoid re-evaluating load for this entity: */ - se =parent_entity (se); -/* -* Bias pick_next to pick a task from this cfs_rq, as -* p is sleeping when it is within its sched_slice. -*/ -if (task_sleep && se) -set_next_buddy (se); -break ; - } flags |=DEQUEUE_SLEEP ; flags &= ~(DEQUEUE_DELAYED |DEQUEUE_SPECIAL ); } +} -for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); +/* +* The part of dequeue_task_fair() that is needed to dequeue delayed tasks. +* +* Returns: +* true - dequeued +* false - delayed +*/ +static bool __dequeue_task (struct rq *rq,struct task_struct *p,int flags) +{ +struct sched_entity *se = &p->se; +struct cfs_rq *cfs_rq = &rq->cfs; +bool was_sched_idle =sched_idle_rq (rq); +bool task_sleep = flags &DEQUEUE_SLEEP ; +bool task_delayed = flags &DEQUEUE_DELAYED ; -update_load_avg (cfs_rq, se,UPDATE_TG ); -se_update_runnable (se); -update_cfs_group (se); +clear_buddies (cfs_rq, se); - se->slice = slice; -if (se != cfs_rq->curr) -min_vruntime_cb_propagate (&se->run_node,NULL ); - slice =cfs_rq_min_slice (cfs_rq); +if (flags &DEQUEUE_DELAYED ) { +WARN_ON_ONCE (!se->sched_delayed); + }else { +bool delay = task_sleep; +/* +* DELAY_DEQUEUE relies on spurious wakeups, special task +* states must not suffer spurious wakeups, excempt them. +*/ +if (flags & (DEQUEUE_SPECIAL |DEQUEUE_THROTTLE )) + delay = false;kernel/sched/fair.c enqueue_task_fair(struct rq *rq, struct · line 7425
Quoted 10 lines
kernel/sched/fair.c +4
-5- cfs_rq->h_nr_runnable -= h_nr_runnable; - cfs_rq->h_nr_queued -= h_nr_queued; - cfs_rq->h_nr_idle -= h_nr_idle; +WARN_ON_ONCE (delay && se->sched_delayed); -if (cfs_rq_is_idle (cfs_rq)) - h_nr_idle = h_nr_queued; +if (sched_feat (DELAY_DEQUEUE ) && delay && + !entity_eligible (cfs_rq, se)) { +update_load_avg (cfs_rq_of (se), se,0 );
update_entity_lag(cfs_rq, se); is missing here. Unfortunately this doesn't fix my regressionQuoted 512 lines
kernel/sched/fair.c +89
-126+set_delayed (se); +return false; + } + } -if (throttled_hierarchy (cfs_rq) && task_throttled) -record_throttle_clock (cfs_rq); +dequeue_hierarchy (p, flags); + +update_entity_lag (cfs_rq, se); +if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { + se->deadline -= se->vruntime; + se->rel_deadline =1 ; } +if (se != cfs_rq->curr) +__dequeue_entity (cfs_rq, se); -sub_nr_running (rq, h_nr_queued); +sub_nr_running (rq,1 );/* balance early to pull high priority tasks */ if (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; -if (p && task_delayed) { +if (task_delayed) { +clear_delayed (se); +WARN_ON_ONCE (!task_sleep);WARN_ON_ONCE (p->on_rq !=1 );@@ -7514,7 +7458,7 @@ static int dequeue_entities(struct rq *r__block_task (rq, p); } -return 1 ; +return true; }/* @@ -7533,11 +7477,11 @@ static bool dequeue_task_fair(struct rqutil_est_dequeue (&rq->cfs, p);util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP ); -if (dequeue_entities (rq, &p->se, flags) <0 ) +if (!__dequeue_task (rq, p, flags))return false;/* -* Must not reference @p after dequeue_entities(DEQUEUE_DELAYED). +* Must not reference @p after __dequeue_task(DEQUEUE_DELAYED). */ return true; }@@ -9021,19 +8965,6 @@ static void migrate_task_rq_fair(structstatic void task_dead_fair (struct task_struct *p) {struct sched_entity *se = &p->se; - -if (se->sched_delayed) { -struct rq_flags rf; -struct rq *rq; - - rq =task_rq_lock (p, &rf); -if (se->sched_delayed) { -update_rq_clock (rq); -dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); - } -task_rq_unlock (rq, p, &rf); - } -remove_entity_load_avg (se); }@@ -9067,21 +8998,10 @@ static void set_cpus_allowed_fair(structset_task_max_allowed_capacity (p); } -static void set_next_buddy (struct sched_entity *se) -{ -for_each_sched_entity (se) { -if (WARN_ON_ONCE (!se->on_rq)) -return ; -if (se_is_idle (se)) -return ; -cfs_rq_of (se)->next = se; - } -} -enum preempt_wakeup_action {PREEMPT_WAKEUP_NONE ,/* No preemption. */ PREEMPT_WAKEUP_SHORT ,/* Ignore slice protection. */ -PREEMPT_WAKEUP_PICK ,/* Let __pick_eevdf() decide. */ +PREEMPT_WAKEUP_PICK ,/* Let pick_eevdf() decide. */ PREEMPT_WAKEUP_RESCHED ,/* Force reschedule. */ };@@ -9098,7 +9018,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,if (cfs_rq->next &&entity_before (cfs_rq->next, pse))return false; -set_next_buddy (pse); +set_next_buddy (cfs_rq, pse);return true; }@@ -9188,7 +9108,6 @@ static void wakeup_preempt_fair(struct rif (!sched_feat (WAKEUP_PREEMPTION ))return ; -find_matching_se (&se, &pse);WARN_ON_ONCE (!pse); cse_is_idle =se_is_idle (se);@@ -9216,8 +9135,7 @@ static void wakeup_preempt_fair(struct rif (unlikely (!normal_policy (p->policy)))return ; - cfs_rq =cfs_rq_of (se); -update_curr (cfs_rq); +update_curr_fair (rq);/* * If @p has a shorter slice than current and @p is eligible, override * current's slice protection in order to allow preemption. @@ -9261,18 +9179,15 @@ static void wakeup_preempt_fair(struct r } pick: - nse =pick_next_entity (rq, cfs_rq, preempt_action !=PREEMPT_WAKEUP_SHORT ); -/* If @p has become the most eligible task, force preemption */ -if (nse == pse) - goto preempt; - -/* -* Because p is enqueued, nse being null can only mean that we -* dequeued a delayed task. If there are still entities queued in -* cfs, check if the next one will be p. -*/ -if (!nse && cfs_rq->nr_queued) - goto pick; +if (cfs_rq->h_nr_queued) { + nse =pick_next_entity (rq, preempt_action !=PREEMPT_WAKEUP_SHORT ); +if (unlikely (!nse)) + goto pick; + +/* If @p has become the most eligible task, force preemption */ +if (nse == pse) + goto preempt; + }if (sched_feat (RUN_TO_PARITY ))update_protect_slice (cfs_rq, se);@@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct rstruct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf)__must_hold (__rq_lockp (rq)) { +struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; -struct cfs_rq *cfs_rq;struct task_struct *p; -bool throttled;int new_tasks; again: - cfs_rq = &rq->cfs; -if (!cfs_rq->nr_queued) +if (!cfs_rq->h_nr_queued) goto idle; - throttled = false; - -do { -/* Might not have done put_prev_entity() */ -if (cfs_rq->curr && cfs_rq->curr->on_rq) -update_curr (cfs_rq); - - throttled |=check_cfs_rq_runtime (cfs_rq); +/* Might not have done put_prev_entity() */ +if (cfs_rq->curr && cfs_rq->curr->on_rq) +update_curr (cfs_rq); - se =pick_next_entity (rq, cfs_rq, true); -if (!se) - goto again; - cfs_rq =group_cfs_rq (se); - }while (cfs_rq); + se =pick_next_entity (rq, true); +if (!se) + goto again; p =task_of (se); -if (unlikely (throttled)) +if (unlikely (check_cfs_rq_runtime (cfs_rq_of (se))))task_throttle_setup_work (p);return p;@@ -9353,7 +9259,7 @@ void fair_server_init(struct rq *rq)static void put_prev_task_fair (struct rq *rq,struct task_struct *prev,struct task_struct *next) {struct sched_entity *se = &prev->se; -struct cfs_rq *cfs_rq; +struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *nse =NULL ;#ifdef CONFIG_FAIR_GROUP_SCHED @@ -9363,7 +9269,7 @@ static void put_prev_task_fair(struct rqwhile (se) { cfs_rq =cfs_rq_of (se); -if (!nse || cfs_rq->curr) +if (!nse || cfs_rq->h_curr)put_prev_entity (cfs_rq, se);#ifdef CONFIG_FAIR_GROUP_SCHED if (nse) {@@ -9382,6 +9288,14 @@ static void put_prev_task_fair(struct rq#endif se =parent_entity (se); } + +/* Put 'current' back into the tree. */ + cfs_rq = &rq->cfs; + se = &prev->se; +WARN_ON_ONCE (cfs_rq->curr != se); + cfs_rq->curr =NULL ; +if (se->on_rq) +__enqueue_entity (cfs_rq, se); }/* @@ -9390,8 +9304,8 @@ static void put_prev_task_fair(struct rqstatic void yield_task_fair (struct rq *rq) {struct task_struct *curr = rq->donor; -struct cfs_rq *cfs_rq =task_cfs_rq (curr);struct sched_entity *se = &curr->se; +struct cfs_rq *cfs_rq = &rq->cfs;/* * Are we the only task in the tree? @@ -9432,11 +9346,11 @@ static bool yield_to_task_fair(struct rqstruct sched_entity *se = &p->se;/* !se->on_rq also covers throttled task */ -if (!se->on_rq) +if (!se->on_rq || se->sched_delayed)return false;/* Tell the scheduler that we'd really like se to run next. */ -set_next_buddy (se); +set_next_buddy (&task_rq (p)->cfs, se);yield_task_fair (rq);@@ -9762,15 +9676,10 @@ static inline long migrate_degrades_loca*/ static inline int task_is_ineligible_on_dst_cpu (struct task_struct *p,int dest_cpu) { -struct cfs_rq *dst_cfs_rq; +struct cfs_rq *dst_cfs_rq = &cpu_rq (dest_cpu)->cfs; -#ifdef CONFIG_FAIR_GROUP_SCHED - dst_cfs_rq =task_group (p)->cfs_rq[dest_cpu]; -#else - dst_cfs_rq = &cpu_rq (dest_cpu)->cfs; -#endif -if (sched_feat (PLACE_LAG ) && dst_cfs_rq->nr_queued && - !entity_eligible (task_cfs_rq (p), &p->se)) +if (sched_feat (PLACE_LAG ) && dst_cfs_rq->h_nr_queued && + !entity_eligible (&task_rq (p)->cfs, &p->se))return 1 ;return 0 ;@@ -10240,7 +10149,7 @@ static void update_cfs_rq_h_load(structwhile ((se =READ_ONCE (cfs_rq->h_load_next)) !=NULL ) { load = cfs_rq->h_load; load =div64_ul (load * se->avg.load_avg, -cfs_rq_load_avg (cfs_rq) +1 ); +cfs_rq_load_avg (cfs_rq) +1 ); cfs_rq =group_cfs_rq (se); cfs_rq->h_load = load; cfs_rq->last_h_load_update = now;@@ -13459,7 +13368,7 @@ static inline void task_tick_core(struct* MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check * if we need to give up the CPU. */ -if (rq->core->core_forceidle_count && rq->cfs.nr_queued ==1 && +if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued ==1 &&__entity_slice_used (&curr->se,MIN_NR_TASKS_DURING_FORCEIDLE ))resched_curr (rq); }@@ -13668,30 +13577,8 @@ bool cfs_prio_less(const struct task_strWARN_ON_ONCE (task_rq (b)->core != rq->core); -#ifdef CONFIG_FAIR_GROUP_SCHED -/* -* Find an se in the hierarchy for tasks a and b, such that the se's -* are immediate siblings. -*/ -while (sea->cfs_rq->tg != seb->cfs_rq->tg) { -int sea_depth = sea->depth; -int seb_depth = seb->depth; - -if (sea_depth >= seb_depth) - sea =parent_entity (sea); -if (sea_depth <= seb_depth) - seb =parent_entity (seb); - } - -se_fi_update (sea, rq->core->core_forceidle_seq, in_fi); -se_fi_update (seb, rq->core->core_forceidle_seq, in_fi); - - cfs_rqa = sea->cfs_rq; - cfs_rqb = seb->cfs_rq; -#else /* !CONFIG_FAIR_GROUP_SCHED: */ cfs_rqa = &task_rq (a)->cfs; cfs_rqb = &task_rq (b)->cfs; -#endif /* !CONFIG_FAIR_GROUP_SCHED */ /* * Find delta after normalizing se's vruntime with its cfs_rq's @@ -13729,14 +13616,20 @@ static inline void task_tick_core(struct*/ static void task_tick_fair (struct rq *rq,struct task_struct *curr,int queued) { -struct cfs_rq *cfs_rq;struct sched_entity *se = &curr->se; +unsigned long weight =NICE_0_LOAD ; +struct cfs_rq *cfs_rq;for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se);entity_tick (cfs_rq, se, queued); + + weight =__calc_prop_weight (cfs_rq, se, weight); } + se = &curr->se; +reweight_eevdf (cfs_rq, se, weight, se->on_rq); +if (queued)return ;@@ -13772,7 +13665,7 @@ prio_changed_fair(struct rq *rq, structif (p->prio == oldprio)return ; -if (rq->cfs.nr_queued ==1 ) +if (rq->cfs.h_nr_queued ==1 )return ;/* @@ -13901,29 +13794,40 @@ static void switched_to_fair(struct rq * } } -/* -* Account for a task changing its policy or group. -* -* This routine is mostly called to set cfs_rq->curr field when a task -* migrates between groups/classes. -*/ static void set_next_task_fair (struct rq *rq,struct task_struct *p,bool first) {struct sched_entity *se = &p->se; +struct cfs_rq *cfs_rq = &rq->cfs; +unsigned long weight =NICE_0_LOAD ; +bool on_rq = se->on_rq; + +clear_buddies (cfs_rq, se); + +if (on_rq) +__dequeue_entity (cfs_rq, se);for_each_sched_entity (se) { -struct cfs_rq *cfs_rq =cfs_rq_of (se); + cfs_rq =cfs_rq_of (se); -if (IS_ENABLED (CONFIG_FAIR_GROUP_SCHED ) && - first && cfs_rq->curr) -break ; +if (!IS_ENABLED (CONFIG_FAIR_GROUP_SCHED ) || + !first || !cfs_rq->h_curr) +set_next_entity (cfs_rq, se); -set_next_entity (cfs_rq, se, first);/* ensure bandwidth has been allocated on our new cfs_rq */ account_cfs_rq_runtime (cfs_rq,0 ); + +if (on_rq) + weight =__calc_prop_weight (cfs_rq, se, weight); } se = &p->se; + cfs_rq->curr = se; + +if (on_rq) { +reweight_eevdf (cfs_rq, se, weight, se->on_rq); +if (first) +set_protect_slice (cfs_rq, se); + }if (task_on_rq_queued (p)) {/* @@ -14054,17 +13958,8 @@ void unregister_fair_sched_group(structstruct sched_entity *se = tg->se[cpu];struct rq *rq =cpu_rq (cpu); -if (se) { -if (se->sched_delayed) { -guard (rq_lock_irqsave)(rq); -if (se->sched_delayed) { -update_rq_clock (rq); -dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); - } -list_del_leaf_cfs_rq (cfs_rq); - } +if (se)remove_entity_load_avg (se); - }/* * Only empty task groups can be destroyed; so we can speculatively kernel/sched/pelt.c +3
-3@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched/* * running is a subset of runnable (weight) so running can't be set if * runnable is clear. But there are some corner cases where the current -* se has been already dequeued but cfs_rq->curr still points to it. +* se has been already dequeued but cfs_rq->h_curr still points to it. * This means that weight will be 0 but not running for a sched_entity * but also for a cfs_rq if the latter becomes idle. As an example, * this happens during sched_balance_newidle() which calls @@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 nowint __update_load_avg_se (u64 now,struct cfs_rq *cfs_rq,struct sched_entity *se) {if (___update_load_sum (now, &se->avg, !!se->on_rq,se_runnable (se), - cfs_rq->curr == se)) { + cfs_rq->h_curr == se)) {___update_load_avg (&se->avg,se_weight (se));cfs_se_util_change (&se->avg);@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, stif (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), cfs_rq->h_nr_runnable, - cfs_rq->curr !=NULL )) { + cfs_rq->h_curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );trace_pelt_cfs_tp (cfs_rq);kernel/sched/sched.h +13
-13@@ -528,21 +528,8 @@ struct task_group { }; -#ifdef CONFIG_GROUP_SCHED_WEIGHT #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD -/* -* A weight of 0 or 1 can cause arithmetics problems. -* A weight of a cfs_rq is the sum of weights of which entities -* are queued on this cfs_rq, so a weight of a entity should not be -* too large, so as the shares value of a task group. -* (The default weight is 1024 - so there's no practical -* limitation from this.) -*/ -#define MIN_SHARES (1UL << 1) -#define MAX_SHARES (1UL << 18) -#endif -typedef int (*tg_visitor )(struct task_group *,void *); extern int walk_tg_tree_from(struct task_group *from,@@ -629,6 +616,17 @@ static inline bool cfs_task_bw_constrain#endif /* !CONFIG_CGROUP_SCHED */ +/* +* A weight of 0 or 1 can cause arithmetics problems. +* A weight of a cfs_rq is the sum of weights of which entities +* are queued on this cfs_rq, so a weight of a entity should not be +* too large, so as the shares value of a task group. +* (The default weight is 1024 - so there's no practical +* limitation from this.) +*/ +#define MIN_SHARES (1UL << 1) +#define MAX_SHARES (1UL << 18) +extern void unregister_rt_sched_group (struct task_group *tg);extern void free_rt_sched_group (struct task_group *tg);extern int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent);@@ -707,6 +705,7 @@ struct cfs_rq {/* * CFS load tracking */ +struct sched_entity *h_curr;struct sched_avg avg;#ifndef CONFIG_64BIT u64 last_update_time_copy;@@ -2509,6 +2508,7 @@ extern const u32 sched_prio_to_wmult[40#define ENQUEUE_MIGRATED 0x00040000#define ENQUEUE_INITIAL 0x00080000#define ENQUEUE_RQ_SELECTED 0x00100000 +#define ENQUEUE_QUEUED 0x00200000#define RETRY_TASK ((void *)-1UL)
K Prateek Nayak
reviewerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Hello Vincent,
Quoted 2 lines · Vincent Guittot
I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed
update_entity_lag():
Great catch!
Quoted 16 lines
update_entity_lagmust be called after updating curr andcfs_rdand before clearingon_rqWith the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet
--- kernel/sched/fair.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)kernel/sched/fair.c +0
-0
kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 5759
Quoted 7 lines
kernel/sched/fair.c +3
-0@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)update_stats_dequeue_fair (cfs_rq, se, flags); +if (entity_is_task (se)) +update_entity_lag (&rq_of (cfs_rq)->cfs, se); + se->on_rq =0 ;
Ah! The curr->on_rq indicator changes here and we'll start ignoring it for avg_vruntime() calculation afterwards! Makes sense.
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7427
Quoted 5 lines
kernel/sched/fair.c +0
-0account_entity_dequeue (cfs_rq, se);@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (sched_feat (DELAY_DEQUEUE ) && delay && !entity_eligible (cfs_rq, se)) {
Does this need a update_curr() before checking entity_eligible()?
Currently these bits reside in dequeue_entity() and is always done after a update_curr(cfs_rq) but here we may need a:
update_curr(task_cfs_rq(p)); /* to catch up h_curr's vruntime */
Just doing it for task_cfs_rq(p) should be fine since we only have to catch up curr's vruntime - sum_w_vruntime and sum_weight at root cfs_rq should be stable for all the tasks on rb-tree.
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7433
Quoted 10 lines
kernel/sched/fair.c +1
-1update_load_avg (cfs_rq_of (se), se,0 ); +update_entity_lag (cfs_rq, se);set_delayed (se);return false; }@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)dequeue_hierarchy (p, flags); -update_entity_lag (cfs_rq, se);
If we decide to do a update_curr(task_cfs_rq(p)) at the beginning of __dequeue_task(), we can just move this to above dequeue_hierarchy() before se->on_rq indicators are modified.
Thoughts?
Quoted 3 lines
kernel/sched/fair.c +0
-0if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;
Signature · 2 lines
Thanks and Regards, Prateek
Vincent Guittot
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 36 lines · K Prateek Nayak
Hello Vincent,
Quoted 2 lines · Vincent Guittot
I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed
update_entity_lag():Great catch!
Quoted 16 lines
update_entity_lagmust be called after updating curr andcfs_rdand before clearingon_rqWith the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet
--- kernel/sched/fair.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)kernel/sched/fair.c +0
-0kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 5759
Quoted 7 lines
kernel/sched/fair.c +3
-0@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)update_stats_dequeue_fair (cfs_rq, se, flags); +if (entity_is_task (se)) +update_entity_lag (&rq_of (cfs_rq)->cfs, se); + se->on_rq =0 ;Ah! The
curr->on_rqindicator changes here and we'll start ignoring it foravg_vruntime()calculation afterwards! Makes sense.
Quoted 7 lines
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7427
Quoted 5 lines
kernel/sched/fair.c +0
-0account_entity_dequeue (cfs_rq, se);@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (sched_feat (DELAY_DEQUEUE ) && delay && !entity_eligible (cfs_rq, se)) {Does this need a
update_curr()before checkingentity_eligible()?
Yes we need to update curr first
Quoted 19 lines
Currently these bits reside in
dequeue_entity()and is always done after aupdate_curr(cfs_rq)but here we may need a:update_curr(task_cfs_rq(p)); /* to catch up h_curr's vruntime */
Just doing it for
task_cfs_rq(p)should be fine since we only have to catch up curr's vruntime -sum_w_vruntimeandsum_weightat rootcfs_rqshould be stable for all the tasks on rb-tree.Quoted 9 lines
kernel/sched/fair.c +1
-0update_load_avg (cfs_rq_of (se), se,0 ); +update_entity_lag (cfs_rq, se);set_delayed (se);return false; }@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)dequeue_hierarchy (p, flags);
Quoted 7 lines
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7433
Quoted 1 line
kernel/sched/fair.c +0
-1-update_entity_lag (cfs_rq, se);If we decide to do a update_curr(task_cfs_rq(p)) at the beginning of
__dequeue_task(), we can just move this to abovedequeue_hierarchy()beforese->on_rqindicators are modified.Thoughts?
yes it's doable, we will have a spurious update_curr in dequeue_hierarchy but that will be a nop because of a null delta_exec
With flat hierarchy, vruntime and deadline are no longer linked to the cfs hierarchy. A possibility could be to move the update of vruntime and deadline outside but this is more complex because of delta_exec
The same apply for dl_server
Quoted 9 lines
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7439
Quoted 4 lines
kernel/sched/fair.c +0
-0if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;--
Thanks and Regards,
Prateek
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
In reply to Vincent GuittotQuoted 36 lines · Vincent Guittot
I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed
update_entity_lag():
update_entity_lagmust be called after updating curr andcfs_rdand before clearingon_rqWith the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet
--- kernel/sched/fair.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)kernel/sched/fair.c +4
-0@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)update_stats_dequeue_fair (cfs_rq, se, flags); +if (entity_is_task (se)) +update_entity_lag (&rq_of (cfs_rq)->cfs, se); + se->on_rq =0 ;account_entity_dequeue (cfs_rq, se);@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (sched_feat (DELAY_DEQUEUE ) && delay && !entity_eligible (cfs_rq, se)) {update_load_avg (cfs_rq_of (se), se,0 ); +update_entity_lag (cfs_rq, se);set_delayed (se);return false; }
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7439
Quoted 7 lines
kernel/sched/fair.c +0
-1@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)dequeue_hierarchy (p, flags); -update_entity_lag (cfs_rq, se);if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;
Argh!!! Thank you! I've gone blind staring at all this :/
Would it not be simpler to just move the update_entity_lag() call up a bit, like so?
---
kernel/sched/fair.c +3 -1
@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rqclear_buddies (cfs_rq, se); +update_curr (cfs_rq); +update_entity_lag (cfs_rq, se); +if (flags &DEQUEUE_DELAYED ) {WARN_ON_ONCE (!se->sched_delayed); }else {
@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rqdequeue_hierarchy (p, flags); -update_entity_lag (cfs_rq, se);if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;
Vincent Guittot
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 56 lines · Peter Zijlstra
Quoted 36 lines · Vincent Guittot
I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed
update_entity_lag():
update_entity_lagmust be called after updating curr andcfs_rdand before clearingon_rqWith the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet
--- kernel/sched/fair.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)kernel/sched/fair.c +4
-0@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)update_stats_dequeue_fair (cfs_rq, se, flags); +if (entity_is_task (se)) +update_entity_lag (&rq_of (cfs_rq)->cfs, se); + se->on_rq =0 ;account_entity_dequeue (cfs_rq, se);@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (sched_feat (DELAY_DEQUEUE ) && delay && !entity_eligible (cfs_rq, se)) {update_load_avg (cfs_rq_of (se), se,0 ); +update_entity_lag (cfs_rq, se);set_delayed (se);return false; }kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7439
Quoted 7 lines
kernel/sched/fair.c +0
-1@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)dequeue_hierarchy (p, flags); -update_entity_lag (cfs_rq, se);if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;Argh!!! Thank you! I've gone blind staring at all this :/
Would it not be simpler to just move the
update_entity_lag()call up a bit, like so?---
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8002
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rqclear_buddies (cfs_rq, se); +update_curr (cfs_rq);
I agree it's simpler although we will call update_curr twice for one level, but the 2nd call should be nop because of delta_exec being null
Prateek proposed update_curr(task_cfs_rq(p)). Using task_cfs_rq(p) will ensure that we keep the same ordering as for_each_sched_entity
Quoted 13 lines
kernel/sched/fair.c +2
-1+update_entity_lag (cfs_rq, se); +if (flags &DEQUEUE_DELAYED ) {WARN_ON_ONCE (!se->sched_delayed); }else {@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rqdequeue_hierarchy (p, flags); -update_entity_lag (cfs_rq, se);if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 10 lines · Vincent Guittot
Quoted 10 lines
Would it not be simpler to just move the
update_entity_lag()call up a bit, like so?---
kernel/sched/fair.c +0
-0@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rqclear_buddies (cfs_rq, se);
Quoted 7 lines
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8002
Quoted 1 line
kernel/sched/fair.c +1
-0+update_curr (cfs_rq);I agree it's simpler although we will call
update_currtwice for one level, but the 2nd call should be nop because ofdelta_execbeing nullPrateek proposed update_curr(task_cfs_rq(p)). Using
task_cfs_rq(p)will ensure that we keep the same ordering asfor_each_sched_entity
Given:
R
|
G
|
tThen task_cfs_rq() will be G's cfs_rq, while cfs_rq is R's cfs_rq.
Since all the actual running happens inside R, this is what is required by update_entity_lag().
Doing update_curr(task_cfs_rq()) here doesn't make sense.
I'm not sure I see a way in which running them out of order hurts anything.
Quoted 13 lines
Quoted 13 lines
kernel/sched/fair.c +2
-1+update_entity_lag (cfs_rq, se); +if (flags &DEQUEUE_DELAYED ) {WARN_ON_ONCE (!se->sched_delayed); }else {@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rqdequeue_hierarchy (p, flags); -update_entity_lag (cfs_rq, se);if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;
Vincent Guittot
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 23 lines · Peter Zijlstra
Quoted 10 lines · Vincent Guittot
Quoted 10 lines
Would it not be simpler to just move the
update_entity_lag()call up a bit, like so?---
kernel/sched/fair.c +0
-0@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rqclear_buddies (cfs_rq, se);Quoted 7 lines
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8002
Quoted 1 line
kernel/sched/fair.c +1
-0+update_curr (cfs_rq);I agree it's simpler although we will call
update_currtwice for one level, but the 2nd call should be nop because ofdelta_execbeing nullPrateek proposed update_curr(task_cfs_rq(p)). Using
task_cfs_rq(p)will ensure that we keep the same ordering asfor_each_sched_entityGiven:
Quoted 7 lines
R | G | tThen
task_cfs_rq()will be G'scfs_rq, whilecfs_rqis R'scfs_rq.
Yes but update_curr() moves to R's cfs anyway before updating vruntime, deadline and dl_server
Quoted 3 lines
Since all the actual running happens inside R, this is what is required by
update_entity_lag().
In other places like task_tick_fair, we follow the G then R order and vruntime and deadline are updated while updating G
Quoted 5 lines
Doing update_curr(task_cfs_rq()) here doesn't make sense.
I'm not sure I see a way in which running them out of order hurts anything.
I was thinking of use cases which involves throttling but I haven't gone deeply in the analyses
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
kernel/sched/fair.c +2
-1+update_entity_lag (cfs_rq, se); +if (flags &DEQUEUE_DELAYED ) {WARN_ON_ONCE (!se->sched_delayed); }else {@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rqdequeue_hierarchy (p, flags); -update_entity_lag (cfs_rq, se);if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraQuoted 30 lines · Peter Zijlstra
Quoted 10 lines · Vincent Guittot
Quoted 10 lines
Would it not be simpler to just move the
update_entity_lag()call up a bit, like so?---
kernel/sched/fair.c +0
-0@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rqclear_buddies (cfs_rq, se);Quoted 7 lines
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8002
Quoted 1 line
kernel/sched/fair.c +1
-0+update_curr (cfs_rq);I agree it's simpler although we will call
update_currtwice for one level, but the 2nd call should be nop because ofdelta_execbeing nullPrateek proposed update_curr(task_cfs_rq(p)). Using
task_cfs_rq(p)will ensure that we keep the same ordering asfor_each_sched_entityGiven:
R | G | tThen
task_cfs_rq()will be G'scfs_rq, whilecfs_rqis R'scfs_rq.
Quoted 7 lines
Since all the actual running happens inside R, this is what is required by
update_entity_lag().Doing update_curr(task_cfs_rq()) here doesn't make sense.
I'm not sure I see a way in which running them out of order hurts anything.
Bah, I'm so full of fail. So update_curr() takes ->h_curr, which for R would be G's se, not t. So yeah, Prateek is right and I should stop trying to do more than one thing at a time :-(
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraQuoted 10 lines · Peter Zijlstra
Quoted 8 lines · Vincent Guittot
I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed
update_entity_lag():
update_entity_lagmust be called after updating curr andcfs_rdand before clearingon_rqWith the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet
I see a very slight hackbench regression on the high end, but meh. The latency-slice test seems to have slightly improved max values, but this isn't the most stable of things.
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraQuoted 17 lines · Peter Zijlstra
Would it not be simpler to just move the
update_entity_lag()call up a bit, like so?---
kernel/sched/fair.c +3
-0@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rqclear_buddies (cfs_rq, se); +update_curr (cfs_rq); +update_entity_lag (cfs_rq, se); +if (flags &DEQUEUE_DELAYED ) {WARN_ON_ONCE (!se->sched_delayed); }else {
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8030
Quoted 7 lines
kernel/sched/fair.c +0
-1@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rqdequeue_hierarchy (p, flags); -update_entity_lag (cfs_rq, se);if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;
FWIW, I pushed out a new queue:sched/flat with this on. I had to rebase because of: 6d2051403d6c ("sched/fair: Update util_est after updating util_avg during dequeue"), hopefully I didn't wreck that :/
Vincent Guittot
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 23 lines · Peter Zijlstra
Quoted 20 lines · Peter Zijlstra
Would it not be simpler to just move the
update_entity_lag()call up a bit, like so?---
kernel/sched/fair.c +3
-0@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rqclear_buddies (cfs_rq, se); +update_curr (cfs_rq); +update_entity_lag (cfs_rq, se); +if (flags &DEQUEUE_DELAYED ) {WARN_ON_ONCE (!se->sched_delayed); }else {@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rqdequeue_hierarchy (p, flags);
Quoted 8 lines
kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8030
Quoted 4 lines
kernel/sched/fair.c +0
-1-update_entity_lag (cfs_rq, se);if (sched_feat (PLACE_REL_DEADLINE ) && !task_sleep) { se->deadline -= se->vruntime; se->rel_deadline =1 ;FWIW, I pushed out a new queue:sched/flat with this on. I had to rebase because of:
6d2051403d6c("sched/fair: Updateutil_estafter updatingutil_avgduring dequeue"), hopefully I didn't wreck that :/
This looks good to me
Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
In reply to Peter ZijlstraHi Peter,
Quoted 4 lines · Peter Zijlstra
kernel/sched/fair.c +0
-0@@ -13729,14 +13616,20 @@ static inline void task_tick_core(struct*/ static void task_tick_fair (struct rq *rq,struct task_struct *curr,int queued) {
kernel/sched/fair.c static inline void task_tick_core(struct · line 13627
Quoted 10 lines
kernel/sched/fair.c +4
-1-struct cfs_rq *cfs_rq;struct sched_entity *se = &curr->se; +unsigned long weight =NICE_0_LOAD ; +struct cfs_rq *cfs_rq;for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se);entity_tick (cfs_rq, se, queued); + + weight =__calc_prop_weight (cfs_rq, se, weight);
Testing sched/flat branch on AMD EPYC 9654 (384 CPUs, 8 NUMA nodes) with a 2-level cgroup hierarchy and cfs_bandwidth quota enabled, hackbench triggers a divide-by-zero oops:
[ 142.308571] divide error: 0000 [#1] SMP NOPTI [ 142.308582] RIP: 0010:task_tick_fair+0x19e/0x410 [ 142.308601] Call Trace: [ 142.308604] <IRQ> [ 142.308607] scheduler_tick+0x6a/0x110 [ 142.308609] update_process_times+0x6b/0x90 [ 142.308611] tick_sched_handle+0x2a/0x70 [ 142.308613] tick_sched_timer+0x57/0xb0
faddr2line confirms:
task_tick_fair+0x19e/0x410: __calc_prop_weight at kernel/sched/fair.c:4085 (inlined by) task_tick_fair at kernel/sched/fair.c:13576
===========================================================
Reproduction
===========================================================
Kernel: sched/flat branch (54d493980e00 and later)
Hardware: AMD EPYC 9654, 2S 384 logical CPUs
# 2-level cgroup, quota = 50% of one period
cgcreate -g cpu:/bw/l1/l2
cgset -r cpu.cfs_quota_us=50000 /bw/l1/l2
cgset -r cpu.cfs_period_us=100000 /bw/l1/l2
# high task count amplifies the throttle→tick race window cgexec -g cpu:/bw/l1/l2 hackbench -g 48 -l 1000 -s 512 -T
Typically crashes within 30 seconds on this machine. A single-CPU kernel or a very loose quota (e.g. 90%) is unlikely to trigger it because the race window is narrow.
Thanks,
Zhang Qiao
Quoted 29 lines
kernel/sched/fair.c +4
-1} + se = &curr->se; +reweight_eevdf (cfs_rq, se, weight, se->on_rq); +if (queued)return ;@@ -13772,7 +13665,7 @@ prio_changed_fair(struct rq *rq, structif (p->prio == oldprio)return ; -if (rq->cfs.nr_queued ==1 ) +if (rq->cfs.h_nr_queued ==1 )return ;/* @@ -13901,29 +13794,40 @@ static void switched_to_fair(struct rq * } }.
K Prateek Nayak
reviewerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Hello Zhang,
Quoted 4 lines · Zhang Qiao
Testing sched/flat branch on AMD EPYC 9654 (384 CPUs, 8 NUMA nodes) with a 2-level cgroup hierarchy and
cfs_bandwidthquota enabled, hackbench triggers a divide-by-zero oops:
Quoted 8 lines
[ 142.308571] divide error: 0000 [#1] SMP NOPTI [ 142.308582] RIP: 0010:task_tick_fair+0x19e/0x410 [ 142.308601] Call Trace: [ 142.308604] <IRQ> [ 142.308607] scheduler_tick+0x6a/0x110 [ 142.308609] update_process_times+0x6b/0x90 [ 142.308611] tick_sched_handle+0x2a/0x70 [ 142.308613] tick_sched_timer+0x57/0xb0
More of this trace would have been helpful.
Quoted 6 lines
faddr2line confirms:
task_tick_fair+0x19e/0x410: __calc_prop_weight at kernel/sched/fair.c:4085 (inlined by) task_tick_fair at kernel/sched/fair.c:13576
Those line numbers don't match on the latest sched/flat but since you mention this happens with throttling, I believe it is tick hitting somewhere in between the task being dequeued by throttle_cfs_rq_work() and the CPU rescheduling and taking the task off the runqueue.
Dequeue from throttle is slightly special since it keeps the task on runqueue but the sched entity goes off the cfs_rq changing the hierarchical weights.
Can you check if this helps:
(Lightly tested with your reproducer)
kernel/sched/fair.c +12 -9
@@ -14815,18 +14815,21 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}static void task_tick_fair (struct rq *rq,struct task_struct *curr,int queued) {struct sched_entity *se = &curr->se; -unsigned long weight =NICE_0_LOAD ; -struct cfs_rq *cfs_rq; -for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); -entity_tick (cfs_rq, se, queued); +if (se->on_rq) { +unsigned long weight =NICE_0_LOAD ; +struct cfs_rq *cfs_rq; - weight =__calc_prop_weight (cfs_rq, se, weight); - } +for_each_sched_entity (se) { + cfs_rq =cfs_rq_of (se); +entity_tick (cfs_rq, se, queued); + + weight =__calc_prop_weight (cfs_rq, se, weight); + } - se = &curr->se; -reweight_eevdf (cfs_rq, se, weight, se->on_rq); + se = &curr->se; +reweight_eevdf (cfs_rq, se, weight, se->on_rq); + }if (queued)return ;
---
I don't think it makes too much sense to reweight an entity that has been dequeued. The enqueue at unthrottle will do it anyways.
Quoted 13 lines
===========================================================
Reproduction
===========================================================Kernel: sched/flat branch (
54d493980e00and later)
Hardware: AMD EPYC 9654, 2S 384 logical CPUs# 2-level cgroup, quota = 50% of one period
cgcreate -g cpu:/bw/l1/l2
cgset -r cpu.cfs_quota_us=50000 /bw/l1/l2
cgset -r cpu.cfs_period_us=100000 /bw/l1/l2
Quoted 6 lines
# high task count amplifies the throttle→tick race window cgexec -g cpu:/bw/l1/l2 hackbench -g 48 -l 1000 -s 512 -TTypically crashes within 30 seconds on this machine. A single-CPU kernel or a very loose quota (e.g. 90%) is unlikely to trigger it because the race window is narrow.
This was helpful! I see:
[ 209.935597] Oops: divide error: 0000 [#1] SMP NOPTI [ 209.941061] CPU: 329 UID: 0 PID: 8247 Comm: sched-messaging Not tainted 7.1.0-rc2-test+ #73 PREEMPT(full) [ 209.951841] Hardware name: AMD Corporation Titanite_4G/Titanite_4G, BIOS RTI100CC 03/28/2024 [ 209.961254] RIP: 0010:task_tick_fair+0x10d/0x850 [ 209.966420] Code: dc 00 00 00 4c 89 f7 e8 f1 52 ff ff 45 85 e4 0f 85 ba 00 00 00 49 8b 06 4d 8b b6 b8 00 00 00 48 0f af c3 4d 85 f6 74 19 31 d2 <49> f7 37 ba 02 00 00 00 48 89 d3 48 39 d0 48 0f 43 d8 e9 20 ff ff [ 209.987382] RSP: 0018:ff581fd71e1fce58 EFLAGS: 00010046 [ 209.993216] RAX: 0000010000000000 RBX: 0000000000100000 RCX: ff295dbfa9ad8080 [ 210.001179] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff295dbfa9ad8080 [ 210.009141] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000000063eb [ 210.017104] R10: 0000000000000000 R11: ff581fd71e1fcff8 R12: 0000000000000000 [ 210.025061] R13: ff295dbfa9ad8000 R14: ff295dc06c6eac00 R15: ff295dbfd9bc8600 [ 210.033027] FS: 00007faef8c8b640(0000) GS:ff295e7c4acca000(0000) knlGS:0000000000000000 [ 210.042060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 210.048474] CR2: 00007f9884292d30 CR3: 000000011aa26001 CR4: 0000000000f71ef0 [ 210.056430] PKRU: 55555554 [ 210.059448] Call Trace: [ 210.062177] <IRQ> [ 210.064426] sched_tick+0x94/0x250 [ 210.068229] update_process_times+0x99/0xc0 [ 210.072903] tick_nohz_handler+0x95/0x1a0 [ 210.077380] ? __pfx_tick_nohz_handler+0x10/0x10 [ 210.082534] __hrtimer_run_queues+0xfe/0x260 [ 210.087304] hrtimer_interrupt+0x122/0x1f0 [ 210.091880] __sysvec_apic_timer_interrupt+0x55/0x130 [ 210.097525] sysvec_apic_timer_interrupt+0x7a/0xb0 [ 210.102873] </IRQ> [ 210.105203] <TASK> [ 210.107542] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 210.113284] RIP: 0010:_raw_spin_unlock_irqrestore+0x1d/0x40 [ 210.119511] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 c6 07 00 0f 1f 00 f7 c6 00 02 00 00 74 06 fb 0f 1f 44 00 00 <65> ff 0d ec 20 fd 01 74 05 e9 c0 81 d4 fe e8 00 93 ec fe e9 b6 81 [ 210.140469] RSP: 0018:ff581fd74032fe88 EFLAGS: 00000206 [ 210.146308] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000004 [ 210.154271] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ff295dbfa9ad8d64 [ 210.162235] RBP: ff295dbfa9ad8000 R08: 0000000000000000 R09: 0000000000000000 [ 210.170196] R10: 0000000000000000 R11: 0000000000000000 R12: ff295dbfa9ad8d64 [ 210.178159] R13: ff581fd74032ff48 R14: ff295dbfa9ad8000 R15: 00fffffffffff000 [ 210.186139] task_work_run+0x5c/0x90 [ 210.190137] exit_to_user_mode_loop+0x16e/0x550 [ 210.195198] ? srso_alias_return_thunk+0x5/0xfbef5 [ 210.200552] ? ksys_read+0xc5/0xe0 [ 210.204352] do_syscall_64+0x26e/0x750 [ 210.208540] ? do_syscall_64+0xaa/0x750 [ 210.212823] ? srso_alias_return_thunk+0x5/0xfbef5 [ 210.218174] entry_SYSCALL_64_after_hwframe+0x76/0x7e
---
So the theory of throttle work causing this checks out.
The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)
Signature · 2 lines
Thanks and Regards, Prateek
Zhang Qiao
Tested-byRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Hi Prateek,
Quoted 21 lines · K Prateek Nayak
Hello Zhang,
Quoted 4 lines · Zhang Qiao
Testing sched/flat branch on AMD EPYC 9654 (384 CPUs, 8 NUMA nodes) with a 2-level cgroup hierarchy and
cfs_bandwidthquota enabled, hackbench triggers a divide-by-zero oops:Quoted 8 lines
[ 142.308571] divide error: 0000 [#1] SMP NOPTI [ 142.308582] RIP: 0010:task_tick_fair+0x19e/0x410 [ 142.308601] Call Trace: [ 142.308604] <IRQ> [ 142.308607] scheduler_tick+0x6a/0x110 [ 142.308609] update_process_times+0x6b/0x90 [ 142.308611] tick_sched_handle+0x2a/0x70 [ 142.308613] tick_sched_timer+0x57/0xb0More of this trace would have been helpful.
Quoted 3 lines
faddr2line confirms:
Quoted 9 lines
Quoted 3 lines
task_tick_fair+0x19e/0x410: __calc_prop_weight at kernel/sched/fair.c:4085 (inlined by) task_tick_fair at kernel/sched/fair.c:13576Those line numbers don't match on the latest sched/flat but since you mention this happens with throttling, I believe it is tick hitting somewhere in between the task being dequeued by
throttle_cfs_rq_work()and the CPU rescheduling and taking the task off the runqueue.
Sorry for the confusion on the line numbers — the mismatch was due to some local debug code I had added on top of sched/flat, not a difference in the base tree.
Quoted 30 lines
Dequeue from throttle is slightly special since it keeps the task on runqueue but the sched entity goes off the
cfs_rqchanging the hierarchical weights.Quoted 1 line
Can you check if this helps:
(Lightly tested with your reproducer)
kernel/sched/fair.c +6
-7@@ -14815,18 +14815,21 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}static void task_tick_fair (struct rq *rq,struct task_struct *curr,int queued) {struct sched_entity *se = &curr->se; -unsigned long weight =NICE_0_LOAD ; -struct cfs_rq *cfs_rq; -for_each_sched_entity (se) { - cfs_rq =cfs_rq_of (se); -entity_tick (cfs_rq, se, queued); +if (se->on_rq) { +unsigned long weight =NICE_0_LOAD ; +struct cfs_rq *cfs_rq; - weight =__calc_prop_weight (cfs_rq, se, weight); - } +for_each_sched_entity (se) { + cfs_rq =cfs_rq_of (se); +entity_tick (cfs_rq, se, queued);
kernel/sched/fair.c static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {} · line 14833
Quoted 10 lines
kernel/sched/fair.c +6
-2+ + weight =__calc_prop_weight (cfs_rq, se, weight); + } - se = &curr->se; -reweight_eevdf (cfs_rq, se, weight, se->on_rq); + se = &curr->se; +reweight_eevdf (cfs_rq, se, weight, se->on_rq); + }
throttle_cfs_rq_work() sets se->on_rq = 0 while the task is still running as rq->curr, and the subsequent tick should not attempt to reweight an already-dequeued entity. The unthrottle enqueue will handle the reweight anyway.
I've tested your suggested diff on my AMD EPYC 9654 (384 CPUs, 8 NUMA nodes) and it resolves the crash. The reproducer no longer triggers the divide error after running for several minutes.
Thanks,
Zhang Qiao
.
Quoted 83 lines
kernel/sched/fair.c +0
-1if (queued)return ; ---I don't think it makes too much sense to reweight an entity that has been dequeued. The enqueue at unthrottle will do it anyways.
Quoted 13 lines
===========================================================
Reproduction
===========================================================Kernel: sched/flat branch (
54d493980e00and later)
Hardware: AMD EPYC 9654, 2S 384 logical CPUs# 2-level cgroup, quota = 50% of one period
cgcreate -g cpu:/bw/l1/l2
cgset -r cpu.cfs_quota_us=50000 /bw/l1/l2
cgset -r cpu.cfs_period_us=100000 /bw/l1/l2Quoted 6 lines
# high task count amplifies the throttle→tick race window cgexec -g cpu:/bw/l1/l2 hackbench -g 48 -l 1000 -s 512 -TTypically crashes within 30 seconds on this machine. A single-CPU kernel or a very loose quota (e.g. 90%) is unlikely to trigger it because the race window is narrow.
This was helpful! I see:
[ 209.935597] Oops: divide error: 0000 [#1] SMP NOPTI [ 209.941061] CPU: 329 UID: 0 PID: 8247 Comm: sched-messaging Not tainted 7.1.0-rc2-test+ #73 PREEMPT(full) [ 209.951841] Hardware name: AMD Corporation Titanite_4G/Titanite_4G, BIOS RTI100CC 03/28/2024 [ 209.961254] RIP: 0010:task_tick_fair+0x10d/0x850 [ 209.966420] Code: dc 00 00 00 4c 89 f7 e8 f1 52 ff ff 45 85 e4 0f 85 ba 00 00 00 49 8b 06 4d 8b b6 b8 00 00 00 48 0f af c3 4d 85 f6 74 19 31 d2 <49> f7 37 ba 02 00 00 00 48 89 d3 48 39 d0 48 0f 43 d8 e9 20 ff ff [ 209.987382] RSP: 0018:ff581fd71e1fce58 EFLAGS: 00010046 [ 209.993216] RAX: 0000010000000000 RBX: 0000000000100000 RCX: ff295dbfa9ad8080 [ 210.001179] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff295dbfa9ad8080 [ 210.009141] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000000063eb [ 210.017104] R10: 0000000000000000 R11: ff581fd71e1fcff8 R12: 0000000000000000 [ 210.025061] R13: ff295dbfa9ad8000 R14: ff295dc06c6eac00 R15: ff295dbfd9bc8600 [ 210.033027] FS: 00007faef8c8b640(0000) GS:ff295e7c4acca000(0000) knlGS:0000000000000000 [ 210.042060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 210.048474] CR2: 00007f9884292d30 CR3: 000000011aa26001 CR4: 0000000000f71ef0 [ 210.056430] PKRU: 55555554 [ 210.059448] Call Trace: [ 210.062177] <IRQ> [ 210.064426] sched_tick+0x94/0x250 [ 210.068229] update_process_times+0x99/0xc0 [ 210.072903] tick_nohz_handler+0x95/0x1a0 [ 210.077380] ? __pfx_tick_nohz_handler+0x10/0x10 [ 210.082534] __hrtimer_run_queues+0xfe/0x260 [ 210.087304] hrtimer_interrupt+0x122/0x1f0 [ 210.091880] __sysvec_apic_timer_interrupt+0x55/0x130 [ 210.097525] sysvec_apic_timer_interrupt+0x7a/0xb0 [ 210.102873] </IRQ> [ 210.105203] <TASK> [ 210.107542] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 210.113284] RIP: 0010:_raw_spin_unlock_irqrestore+0x1d/0x40 [ 210.119511] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 c6 07 00 0f 1f 00 f7 c6 00 02 00 00 74 06 fb 0f 1f 44 00 00 <65> ff 0d ec 20 fd 01 74 05 e9 c0 81 d4 fe e8 00 93 ec fe e9 b6 81 [ 210.140469] RSP: 0018:ff581fd74032fe88 EFLAGS: 00000206 [ 210.146308] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000004 [ 210.154271] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ff295dbfa9ad8d64 [ 210.162235] RBP: ff295dbfa9ad8000 R08: 0000000000000000 R09: 0000000000000000 [ 210.170196] R10: 0000000000000000 R11: 0000000000000000 R12: ff295dbfa9ad8d64 [ 210.178159] R13: ff581fd74032ff48 R14: ff295dbfa9ad8000 R15: 00fffffffffff000 [ 210.186139] task_work_run+0x5c/0x90 [ 210.190137] exit_to_user_mode_loop+0x16e/0x550 [ 210.195198] ? srso_alias_return_thunk+0x5/0xfbef5 [ 210.200552] ? ksys_read+0xc5/0xe0 [ 210.204352] do_syscall_64+0x26e/0x750 [ 210.208540] ? do_syscall_64+0xaa/0x750 [ 210.212823] ? srso_alias_return_thunk+0x5/0xfbef5 [ 210.218174] entry_SYSCALL_64_after_hwframe+0x76/0x7e---
So the theory of throttle work causing this checks out.
The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
In reply to K Prateek NayakQuoted 3 lines · K Prateek Nayak
The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)
Is this a different issue than the one you raised before? We talked about throtte, and you were going to make a proper patch of that cleanup iirc.
K Prateek Nayak
reviewerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Hello Peter,
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · K Prateek Nayak
The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)
Is this a different issue than the one you raised before?
Yes, this is different. Essentially, this is what is happening:
throttle_cfs_rq_work()
task_rq_lock()
dequeue_task_fair(current) /* Task is dequeued on cfs side */
__dequeue_task(current)
dequeue_hierarchy(current);
current->se.on_rq = 0;
/* update_load_sub() */
resched_curr(); /* Initiates a resched */
task_rq_unlock()
local_irq_enable(); =====> sched_tick()
task_tick_fair()
__calc_prop_weight()
/*
* Oops: update_load_sub() above has
* 0ed the weight of cfs_rq.
*/<====
preempt_schedule_irq()
next = ...
put_prev_set_next_task() /* The runtime context is switched here */
Quoted 2 lines
We talked about throtte, and you were going to make a proper patch of that cleanup iirc.
I had rebased your suggestion on tip and fixed a couple of splats but once it was functional, I noticed hackbench taking twice as long to complete compared to tip and I was chasing that before I fell sick.
Let me go dig deeper to see where exactly it is all going sideways.
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 33 lines · K Prateek Nayak
Hello Peter,
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · K Prateek Nayak
The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)
Is this a different issue than the one you raised before?
Yes, this is different. Essentially, this is what is happening:
throttle_cfs_rq_work()task_rq_lock() dequeue_task_fair(current) /* Task is dequeued on cfs side */ __dequeue_task(current) dequeue_hierarchy(current); current->se.on_rq = 0; /* update_load_sub() */ resched_curr(); /* Initiates a resched */ task_rq_unlock() local_irq_enable();=====>
sched_tick()task_tick_fair() __calc_prop_weight() /* * Oops: update_load_sub() above has * 0ed the weight of cfs_rq. */
Quoted 6 lines
<====
preempt_schedule_irq()
next = ...
put_prev_set_next_task()/* The runtime context is switched here */
Ah, right. OK, I'll go have a poke once I get these proxy patches I've been spending too much time on posted.
I think I've found a 'problem' with that PROXY_WAKING ==> '->is_blocked && !->blocked_on' scheme :-(
Quoted 8 lines
Quoted 2 lines
We talked about throtte, and you were going to make a proper patch of that cleanup iirc.
I had rebased your suggestion on tip and fixed a couple of splats but once it was functional, I noticed hackbench taking twice as long to complete compared to tip and I was chasing that before I fell sick.
Let me go dig deeper to see where exactly it is all going sideways.
Sure, no worries. This happens; computers just never want to just DTRT already. I lost a day and then some trying to figure out why my seemingly 'trivial' proxy changes ended up trying to run a dead task last week...
Peter Zijlstra
maintainerRe: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
Quoted 36 lines · Peter Zijlstra
Quoted 35 lines · K Prateek Nayak
Hello Peter,
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · K Prateek Nayak
The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)
Is this a different issue than the one you raised before?
Yes, this is different. Essentially, this is what is happening:
throttle_cfs_rq_work()task_rq_lock() dequeue_task_fair(current) /* Task is dequeued on cfs side */ __dequeue_task(current) dequeue_hierarchy(current); current->se.on_rq = 0; /* update_load_sub() */ resched_curr(); /* Initiates a resched */ task_rq_unlock() local_irq_enable();=====>
sched_tick()task_tick_fair() __calc_prop_weight() /* * Oops: update_load_sub() above has * 0ed the weight of cfs_rq. */<====
Quoted 7 lines
Quoted 4 lines
preempt_schedule_irq()
next = ...
put_prev_set_next_task()/* The runtime context is switched here */Ah, right. OK, I'll go have a poke once I get these proxy patches I've been spending too much time on posted.
Yes, your solution seems reasonable. I'll fold that and push out a new version a little later today.
Tejun Heo
maintainerHello, Peter.
Quoted 22 lines · Peter Zijlstra
So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.
The problems with weight distribution are related to that infernal global fraction:
tg->w * grq_i->w ge_i->w = ---------------- \Sum_j grq_j->wwhich we've approximated reasonably well by now. However, the immediate consequence of this fraction is that the total group weight (
tg->w) gets fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine with the fact that 256 CPU systems are relatively common these days, this becomes painful.The common 'solution' is to inflate the group weight by '
nr_cpus'; the immediate problem with that is that when all load of a group gets concentrated on a single CPU, the per-cpu cgroup weight becomes insanely large, easily exceeding nice -20.
Quoted 8 lines
Additionally there are numerical limits on the max weight you can have before the math starts suffering overflows. As such there is a definite limit on the total group weight. Which has annoyed people ;-)
The first few patches add a knob /debug/sched/cgroup_mode and a few different options on how to deal with this. My favourite is 'concur', but obviously that is also the most expensive one :-/ It adds a
tg->taskscounter which makes theupdate_tg_load_avg()thing more expensive.
Ignoring fixed math accuracy problems, isn't the root problem here that every thread in the root cgroup competes as if each is its own cgroup? ie. Isn't the canonical solution here to create an enveloping group, at least for share calculation purposes, for root threads and then assign them some weight so that they compete in the same way that other cgroups do? Then, the different modes go away or rather whatever the user wants can be expressed via root's weight if that's to be made configurable.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 34 lines · Tejun Heo
Hello, Peter.
Quoted 22 lines · Peter Zijlstra
So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.
The problems with weight distribution are related to that infernal global fraction:
tg->w * grq_i->w ge_i->w = ---------------- \Sum_j grq_j->wwhich we've approximated reasonably well by now. However, the immediate consequence of this fraction is that the total group weight (
tg->w) gets fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine with the fact that 256 CPU systems are relatively common these days, this becomes painful.The common 'solution' is to inflate the group weight by '
nr_cpus'; the immediate problem with that is that when all load of a group gets concentrated on a single CPU, the per-cpu cgroup weight becomes insanely large, easily exceeding nice -20.Quoted 8 lines
Additionally there are numerical limits on the max weight you can have before the math starts suffering overflows. As such there is a definite limit on the total group weight. Which has annoyed people ;-)
The first few patches add a knob /debug/sched/cgroup_mode and a few different options on how to deal with this. My favourite is 'concur', but obviously that is also the most expensive one :-/ It adds a
tg->taskscounter which makes theupdate_tg_load_avg()thing more expensive.
Quoted 7 lines
Ignoring fixed math accuracy problems, isn't the root problem here that every thread in the root cgroup competes as if each is its own cgroup? ie. Isn't the canonical solution here to create an enveloping group, at least for share calculation purposes, for root threads and then assign them some weight so that they compete in the same way that other cgroups do? Then, the different modes go away or rather whatever the user wants can be expressed via root's weight if that's to be made configurable.
As long as the total group weight is a fraction; and it sorta has to be. You can run into trouble by stacking that fraction.
Take 256 CPUs and a group weight of 1024. Then each CPU gets a weight of 1/256 or 4. Even if we increase the internal accuracy to 20 bits (we do on 64bit) then this becomes 4096, do this for 2 more levels in the hierarchy and you're down to scraping the barrel again.
So if each level runs at a fraction f of the level above, then level n runs at f^n. Moving root into a phantom group at level 1, only solves the problem against other tasks at level 1, but then you have the same problem again at level 2 and below.
Both the numerical problems and the scale problem of the root group can be avoided if we can get the average/nominal fraction to be near 1.
The 'normal' way around this is to ensure the group weight is nr_cpus * 1024, then, when everybody is running, the per CPU weight is 1024 or 1 and the continued fraction is also 1-ish. This is why people like to increase the max group weight.
Trouble is of course that if not all CPUs are busy, with the extreme being only a single CPU carrying that weight of nr_cpus*1024, this then causes trouble because that one CPU gets overloaded.
One of the options is to simply put a max on the single CPU load; which is the crudest option to just make it 'work'. The one I favour though is the one where we scale the group weight by: 'min(cpumas, nr_tasks)'.
Anyway, this is why I've been looking at these alternative weight schemes, to get the nominal fraction near 1 and make these problems go away. It is both the numerical issues and the disparity between levels (with root being at level 0 being the most obvious).
Does that make sense?
Tejun Heo
maintainerHello, Peter.
On Tue, May 12, 2026 at 10:10:00AM +0200, Peter Zijlstra wrote:
...
Quoted 4 lines
Anyway, this is why I've been looking at these alternative weight schemes, to get the nominal fraction near 1 and make these problems go away. It is both the numerical issues and the disparity between levels (with root being at level 0 being the most obvious).
I see. I think what bothers me is that I'm unsure what the weight config would mean when the shares are scaled by the number of active cpus in that cgroup. Here's a simple example:
- There are 256 cpus.
- /cgroup-A has weight 100 and 128 active threads. No pinning.
- /cgroup-B has weight 100 and 256 active thredas. No pinning.
In the current code, assuming math holds up, cgroup-A and B would get about the same shares - ~128 CPUs each. However, if we scale the share by active CPUs in each cgroup, B's tasks would end up with the same weight as A's on CPUs that they end up competing on, which would lead to ~ 1:3 distribution. Is that the right reading of the code?
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 4 lines · Tejun Heo
Hello, Peter.
On Tue, May 12, 2026 at 10:10:00AM +0200, Peter Zijlstra wrote:
...
Quoted 8 lines
Quoted 4 lines
Anyway, this is why I've been looking at these alternative weight schemes, to get the nominal fraction near 1 and make these problems go away. It is both the numerical issues and the disparity between levels (with root being at level 0 being the most obvious).
I see. I think what bothers me is that I'm unsure what the weight config would mean when the shares are scaled by the number of active cpus in that cgroup.
Relative weight per active cpu :-), but yes, that is a somewhat more difficult concept I suppose.
Quoted 11 lines
Here's a simple example:
- There are 256 cpus.
- /cgroup-A has weight 100 and 128 active threads. No pinning.
- /cgroup-B has weight 100 and 256 active thredas. No pinning.In the current code, assuming math holds up, cgroup-A and B would get about the same shares - ~128 CPUs each. However, if we scale the share by active CPUs in each cgroup, B's tasks would end up with the same weight as A's on CPUs that they end up competing on, which would lead to ~ 1:3 distribution. Is that the right reading of the code?
Indeed. So both A and B will get ~1024 weight per (active) CPU, such that on the CPUs they contend they will get 1:1 and then B will get the full CPU on the uncontested CPUs, resulting in a total of 1:3 distribution.
This can of course be compensated by increasing the relative weight of A, if that is so desired. But the alternative view is that for those 128 CPUs they overlap, A and B will get equal parts, it is just that B consumes another 128 CPUs and will not have contention there.
So the current scheme will inflate the part of A to be double the weight (of B), giving them 2 out of 3 parts on the contended CPUs, but then B will still get complete / uncontested access to those extra 128 CPUs, resulting in a 2:4 weight distribution.
Which also isn't as straight forward as one might think.
So perhaps 'weight on the CPUs you contest on' isn't as unintuitive as it seems on first glance, its just different.
And it has tremendous advantages as outlined before; it is naturally normalized -- the disparity between nesting levels goes away, and the edge case of a single CPU active will be sane.
Eg. consider your example except now A will have 1 active thread. Then A will get the full group weight (1024) on its one CPU, while B will get (1024/256=8) on each CPU.
So for the one contended CPU A gets 256 out of 257 parts, while B gets the full CPU for the remaining 255 CPUs, for a:
256 1 257
--- : --- + 255*--- = 256:65535 ~ 1:256
257 257 257
distribution. While with the new scheme it would be:
1 1 2
- : - + 255*- = 1:511
2 2 2
Which, realistically isn't all that different, except the old scheme has this really large weight to deal with.
So from where I'm sitting, yes different, but it behaves better.
Tejun Heo
maintainerHello, Peter.
On Mon, May 18, 2026 at 09:14:56AM +0200, Peter Zijlstra wrote:
...
Quoted 6 lines
So the current scheme will inflate the part of A to be double the weight (of B), giving them 2 out of 3 parts on the contended CPUs, but then B will still get complete / uncontested access to those extra 128 CPUs, resulting in a 2:4 weight distribution.
Which also isn't as straight forward as one might think.
Right, the current behavior isn't quite what people would expect intuitively either.
...
Quoted 9 lines
So for the one contended CPU A gets 256 out of 257 parts, while B gets the full CPU for the remaining 255 CPUs, for a:
256 1 257
--- : --- + 255*--- = 256:65535 ~ 1:256
257 257 257distribution. While with the new scheme it would be:
Quoted 8 lines
1 1 2
- : - + 255*- = 1:511
2 2 2Which, realistically isn't all that different, except the old scheme has this really large weight to deal with.
So from where I'm sitting, yes different, but it behaves better.
I see. Thread cardinality and affinity problems make weight based distribution such a pain. I wonder whether this can be better solved by turning it into a two-layer allocation problem - groups to CPUs and then timeshare on CPUs as necessary. That comes with a lot of its own problems but it can, aspirationally at least, approximate global weight distribution and would have better locality properties.
Thanks.
Signature · 1 line
tejun
Peter Zijlstra
maintainerQuoted 9 lines · Tejun Heo
Hello, Peter.
On Mon, May 18, 2026 at 09:14:56AM +0200, Peter Zijlstra wrote:
...Quoted 5 lines
So the current scheme will inflate the part of A to be double the weight (of B), giving them 2 out of 3 parts on the contended CPUs, but then B will still get complete / uncontested access to those extra 128 CPUs, resulting in a 2:4 weight distribution.
Quoted 23 lines
Quoted 1 line
Which also isn't as straight forward as one might think.
Right, the current behavior isn't quite what people would expect intuitively either.
...
Quoted 17 lines
So for the one contended CPU A gets 256 out of 257 parts, while B gets the full CPU for the remaining 255 CPUs, for a:
256 1 257
--- : --- + 255*--- = 256:65535 ~ 1:256
257 257 257distribution. While with the new scheme it would be:
1 1 2
- : - + 255*- = 1:511
2 2 2Which, realistically isn't all that different, except the old scheme has this really large weight to deal with.
So from where I'm sitting, yes different, but it behaves better.
FWIW if the workload was single threads per CPU; the above is also the exact behaviour we'd have without cgroups.
Quoted 6 lines
I see. Thread cardinality and affinity problems make weight based distribution such a pain. I wonder whether this can be better solved by turning it into a two-layer allocation problem - groups to CPUs and then timeshare on CPUs as necessary. That comes with a lot of its own problems but it can, aspirationally at least, approximate global weight distribution and would have better locality properties.
If people want, they can already do this today. I don't see a reason to mandate something like that. That is, combine cpuset and cpu in a v2 hierarchy and you get this.
The main problem with doing something like that is of course that it isn't always clear how many CPUs will be needed for a particular 'job'. So assigning groups to CPUs isn't a straight forward thing.
If I remember, Meta was actually doing some of this. It was dynamically resizing cpusets based on load predictions and the like in order to separate various worloads on the same large machine, right?
Anyway, while it is somewhat tedious to change behaviour, I do think it is worth doing in this case.
Vincent Guittot
maintainerQuoted 84 lines · Peter Zijlstra
Hi!
So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.
The problems with weight distribution are related to that infernal global fraction:
tg->w * grq_i->w ge_i->w = ---------------- \Sum_j grq_j->wwhich we've approximated reasonably well by now. However, the immediate consequence of this fraction is that the total group weight (
tg->w) gets fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine with the fact that 256 CPU systems are relatively common these days, this becomes painful.The common 'solution' is to inflate the group weight by '
nr_cpus'; the immediate problem with that is that when all load of a group gets concentrated on a single CPU, the per-cpu cgroup weight becomes insanely large, easily exceeding nice -20.Additionally there are numerical limits on the max weight you can have before the math starts suffering overflows. As such there is a definite limit on the total group weight. Which has annoyed people ;-)
The first few patches add a knob /debug/sched/cgroup_mode and a few different options on how to deal with this. My favourite is 'concur', but obviously that is also the most expensive one :-/ It adds a
tg->taskscounter which makes theupdate_tg_load_avg()thing more expensive.I have some ideas but I figured I ought to share these things before sinking more time into it.
On to the hierarchical pick; this has been causing trouble for a very long time. So once again an attempt at flatting it. The basic idea is to keep the full hierarchical load tracking as-is, but keep all the runnable entities in a single level. The immediate concequence of all this is ofcourse that we need to constantly re-compute the effective weight of each entity as things progress.
Reweight is done on:
- enqueue
- pick -- or ratherset_next_entity(.first=true)
- tickSo while the {en,de}queue operations are still O(depth) due to the full accounting mess, the pick is now a single level. Removing the intermediate levels that obscure runnability etc.
For testing, I've done a little experiment, I dug out what is colloqually known as a potato. A trusty old Sandybridge 12600k with a RX 580, and ran a game on it. From GOG, I had available 'Shadows: Awakens', a fun title that normally runs really well on this machine (provided you stick to 1080p).
To make it interesting, I added 8 (one for each logical CPU) copies of: 'nice
spin.sh'; this results in the game becoming almost unplayable, as in proper terrible.I used MangoHUD to record a few minutes of playtime for statistics, and then quit the came and re-started it with a shorter slice set (base/10). This results in the game being entirely playable -- not great, but definiltey playable.
Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper) Intel Core i7-2600K
AMD Radeon RX 580Shadows Awakening (GOG)
default
slice(*)FPS min 3.8 20.6 avg 48.0 57.2 mag 87.4 80.3FT min 9.4 8.4
avg 34.5 19.5 max 107.4 37.2
Quoted 8 lines
FPS (Frames Per Second)
FT (FrameTime)[*] Command prefix: 'chrt -o --sched-runtime 280000 0' effectively setting 'base_slice_ns/10'I have not compared to a kernel without flat on, just wanted to run non trivial workloads and play with slice to make sure everything 'works'.
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16ms
dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%
radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%
Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
I will review the patchset and try to get what is going wrong
Quoted 22 lines
Can also be had:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.gitsched/flatinclude/linux/cpuset.h | 6 include/linux/sched.h | 1 kernel/cgroup/cpuset.c | 15 kernel/sched/core.c | 47 -- kernel/sched/debug.c | 171 +++++--- kernel/sched/fair.c | 1038 ++++++++++++++++++++++--------------------------- kernel/sched/pelt.c | 6 kernel/sched/sched.h | 44 -- 8 files changed, 672 insertions(+), 656 deletions(-)---
Change since v1 ( https://patch.msgid.link/[email protected] ):
- various Sashiko thingies
- rebase atop curren -tip
Peter Zijlstra
maintainerQuoted 18 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%
Quoted 5 lines
Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
I will review the patchset and try to get what is going wrong
Yeah, that is unexpected. Let me go have another look too.
Peter Zijlstra
maintainerQuoted 20 lines · Peter Zijlstra
Quoted 18 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%
Quoted 7 lines
Quoted 5 lines
Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
I will review the patchset and try to get what is going wrong
Yeah, that is unexpected. Let me go have another look too.
So I can reproduce even without the last patch applied. I suspect it is in the cgroup mode patches somewhere. My first suspect is that concur mode thing doing bad things to track the 'global' nr_running thing.
Peter Zijlstra
maintainerQuoted 25 lines · Peter Zijlstra
Quoted 24 lines · Peter Zijlstra
Quoted 22 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I will review the patchset and try to get what is going wrong
Yeah, that is unexpected. Let me go have another look too.
So I can reproduce even without the last patch applied. I suspect it is in the cgroup mode patches somewhere. My first suspect is that concur mode thing doing bad things to track the 'global'
nr_runningthing.
Argh, n/m PEBKAC. I'll try this again in the morning :/
Vincent Guittot
maintainerQuoted 29 lines · Peter Zijlstra
Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Peter Zijlstra
Quoted 23 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
I will review the patchset and try to get what is going wrong
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Yeah, that is unexpected. Let me go have another look too.
So I can reproduce even without the last patch applied. I suspect it is in the cgroup mode patches somewhere. My first suspect is that concur mode thing doing bad things to track the 'global'
nr_runningthing.Argh, n/m PEBKAC. I'll try this again in the morning :/
Reverting the last patch is enough to recover performance
Peter Zijlstra
maintainerQuoted 32 lines · Vincent Guittot
Quoted 31 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 27 lines · Peter Zijlstra
Quoted 18 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%Quoted 5 lines
Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
I will review the patchset and try to get what is going wrong
Yeah, that is unexpected. Let me go have another look too.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
So I can reproduce even without the last patch applied. I suspect it is in the cgroup mode patches somewhere. My first suspect is that concur mode thing doing bad things to track the 'global'
nr_runningthing.Argh, n/m PEBKAC. I'll try this again in the morning :/
Reverting the last patch is enough to recover performance
Yeah, I was on a fail-streak yesterday. I forgot to copy the kernel image before reboot ...
Lets see if today is better :-)
Peter Zijlstra
maintainerQuoted 13 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%
Quoted 7 lines
radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
D'oh :/
kernel/sched/fair.c +1 -1
@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_fenum preempt_wakeup_action preempt_action =PREEMPT_WAKEUP_PICK ;struct task_struct *donor = rq->donor;struct sched_entity *nse, *se = &donor->se, *pse = &p->se; -struct cfs_rq *cfs_rq =task_cfs_rq (donor); +struct cfs_rq *cfs_rq = &rq->cfs;int cse_is_idle, pse_is_idle;/*
Peter Zijlstra
maintainerQuoted 30 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%Quoted 7 lines
radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
D'oh :/
kernel/sched/fair.c +0
-0
kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9077
Quoted 8 lines
kernel/sched/fair.c +1
-1@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_fenum preempt_wakeup_action preempt_action =PREEMPT_WAKEUP_PICK ;struct task_struct *donor = rq->donor;struct sched_entity *nse, *se = &donor->se, *pse = &p->se; -struct cfs_rq *cfs_rq =task_cfs_rq (donor); +struct cfs_rq *cfs_rq = &rq->cfs;int cse_is_idle, pse_is_idle;/*
With that fixed, I now get:
vanilla slice(*)
FPS min 3.0 11.1
avg 44.7 57.3
max 88.1 96.2FT min 9.1 8.0
avg 41.4 21.0
max 157.2 53.9FPS (Frames Per Second)
FT (FrameTime)
Which I suppose shows we now preempt less. Its still significantly better with reduced slice, but not as good as it was.
Vincent Guittot
maintainerQuoted 31 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%Quoted 7 lines
radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
D'oh :/
kernel/sched/fair.c +0
-0
kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9074
Quoted 5 lines
kernel/sched/fair.c +1
-1@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_fenum preempt_wakeup_action preempt_action =PREEMPT_WAKEUP_PICK ;struct task_struct *donor = rq->donor;struct sched_entity *nse, *se = &donor->se, *pse = &p->se; -struct cfs_rq *cfs_rq =task_cfs_rq (donor); +struct cfs_rq *cfs_rq = &rq->cfs;
I tested this patch on top of the series but it doesn't fix the perf regression on rb5
hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default slice duration
Quoted 3 lines
kernel/sched/fair.c +0
-0int cse_is_idle, pse_is_idle;/*
Peter Zijlstra
maintainerQuoted 41 lines · Vincent Guittot
Quoted 31 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%Quoted 7 lines
radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
D'oh :/
kernel/sched/fair.c +0
-0kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9074
Quoted 5 lines
kernel/sched/fair.c +1
-1@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_fenum preempt_wakeup_action preempt_action =PREEMPT_WAKEUP_PICK ;struct task_struct *donor = rq->donor;struct sched_entity *nse, *se = &donor->se, *pse = &p->se; -struct cfs_rq *cfs_rq =task_cfs_rq (donor); +struct cfs_rq *cfs_rq = &rq->cfs;I tested this patch on top of the series but it doesn't fix the perf regression on rb5
Quoted 2 lines
hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default slice duration
Weird, I can't reproduce anymore with this fixed :/
I'll try more hackbench variants tomorrow I suppose.
Vincent Guittot
maintainerQuoted 43 lines · Peter Zijlstra
Quoted 41 lines · Vincent Guittot
Quoted 31 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%Quoted 7 lines
radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
D'oh :/
kernel/sched/fair.c +0
-0kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9074
Quoted 5 lines
kernel/sched/fair.c +1
-1@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_fenum preempt_wakeup_action preempt_action =PREEMPT_WAKEUP_PICK ;struct task_struct *donor = rq->donor;struct sched_entity *nse, *se = &donor->se, *pse = &p->se; -struct cfs_rq *cfs_rq =task_cfs_rq (donor); +struct cfs_rq *cfs_rq = &rq->cfs;I tested this patch on top of the series but it doesn't fix the perf regression on rb5
Quoted 6 lines
Quoted 2 lines
hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default slice duration
Weird, I can't reproduce anymore with this fixed :/
I'll try more hackbench variants tomorrow I suppose.
I tried several conf :
- HMP with EAS enabled
- HMP without EAS enabled (perf cpufreq gov)
- SMP (only the 4 little cores)
All of them show large regressions with hackbench which are almost recovered when increasing the slice from 2.8 to 16ms
Vincent Guittot
maintainerQuoted 52 lines · Vincent Guittot
Quoted 43 lines · Peter Zijlstra
Quoted 41 lines · Vincent Guittot
Quoted 31 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot
I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs
One example with hackbench 8 groups thread pipe
tip/sched/core tip/sched/core +this patchset +this patchset slice 2.8ms 16ms 2.8ms 16msdragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%Quoted 7 lines
radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case
D'oh :/
kernel/sched/fair.c +0
-0kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9074
Quoted 5 lines
kernel/sched/fair.c +1
-1@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_fenum preempt_wakeup_action preempt_action =PREEMPT_WAKEUP_PICK ;struct task_struct *donor = rq->donor;struct sched_entity *nse, *se = &donor->se, *pse = &p->se; -struct cfs_rq *cfs_rq =task_cfs_rq (donor); +struct cfs_rq *cfs_rq = &rq->cfs;I tested this patch on top of the series but it doesn't fix the perf regression on rb5
Quoted 6 lines
Quoted 2 lines
hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default slice duration
Weird, I can't reproduce anymore with this fixed :/
I'll try more hackbench variants tomorrow I suppose.
Quoted 7 lines
I tried several conf :
- HMP with EAS enabled
- HMP without EAS enabled (perf cpufreq gov)
- SMP (only the 4 little cores)All of them show large regressions with hackbench which are almost recovered when increasing the slice from 2.8 to 16ms
With patch 10 the vlag value is very often set to the max 3.8ms (the clamp value of 2.8ms slice + 1ms tick) whereas it is usually less than a 1ms without patch 10
Quoted 4 lines · Peter Zijlstra
Hi!
So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.
It does..
Not that it is useful info, but we talked briefly about it at OSPM, so thought I'll report back. I gave this a go with my test case from schedqos announcement
[1] of running schbench with kernel build as BACKGROUND noise, but the fairness imposed at group level is preserved (as expected) even if the pick is flattened.
I do actually want a total flat system, ie: disable this whole thing :-)
The problem is to create a system where you want to introduce smart tagging based on tasks, group scheduling becomes a big problem. If a task is set as background or interactive, it has to be global to be enforced otherwise it loses its meaning. And my test case stresses two long running tasks one is interactive but the other is background and group scheduling imposes fairness that breaks the task level tagging. Managing deadline via runtime doesn't help here since they are both always busy tasks; and one must use nice values to manage bandwidth. But nice values are local when autogroup/cgroups are present.
I need to find a simple way to turn this thing off at runtime and properly flatten it ;-)
/runs away
[1] https://lore.kernel.org/lkml/20260415000910.2h5misvwc45bdumu@airbuntu/