Skip to contentSkip to conversation

sched: Flatten the pick

PATCH Applied Peter Zijlstra 9 participants 71 messages May 11 – Jun 12

Subsystems (3)

v2 of 3 Applied Latest is v3 →

lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

v2 cover sched: Flatten the pick

Hi!

So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.

The problems with weight distribution are related to that infernal global fraction:

             tg->w * grq_i->w
   ge_i->w = ----------------
             \Sum_j grq_j->w

which we've approximated reasonably well by now. However, the immediate consequence of this fraction is that the total group weight (tg->w) gets fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine with the fact that 256 CPU systems are relatively common these days, this becomes painful.

The common 'solution' is to inflate the group weight by 'nr_cpus'; the immediate problem with that is that when all load of a group gets concentrated on a single CPU, the per-cpu cgroup weight becomes insanely large, easily exceeding nice -20.

Additionally there are numerical limits on the max weight you can have before the math starts suffering overflows. As such there is a definite limit on the total group weight. Which has annoyed people ;-)

The first few patches add a knob /debug/sched/cgroup_mode and a few different options on how to deal with this. My favourite is 'concur', but obviously that is also the most expensive one :-/ It adds a tg->tasks counter which makes the update_tg_load_avg() thing more expensive.

I have some ideas but I figured I ought to share these things before sinking more time into it.

On to the hierarchical pick; this has been causing trouble for a very long time. So once again an attempt at flatting it. The basic idea is to keep the full hierarchical load tracking as-is, but keep all the runnable entities in a single level. The immediate concequence of all this is ofcourse that we need to constantly re-compute the effective weight of each entity as things progress.

Reweight is done on:
- enqueue
- pick -- or rather set_next_entity(.first=true)
- tick

So while the {en,de}queue operations are still O(depth) due to the full accounting mess, the pick is now a single level. Removing the intermediate levels that obscure runnability etc.

For testing, I've done a little experiment, I dug out what is colloqually known as a potato. A trusty old Sandybridge 12600k with a RX 580, and ran a game on it. From GOG, I had available 'Shadows: Awakens', a fun title that normally runs really well on this machine (provided you stick to 1080p).

To make it interesting, I added 8 (one for each logical CPU) copies of: 'nice spin.sh'; this results in the game becoming almost unplayable, as in proper terrible.

I used MangoHUD to record a few minutes of playtime for statistics, and then quit the came and re-started it with a shorter slice set (base/10). This results in the game being entirely playable -- not great, but definiltey playable.

Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper) Intel Core i7-2600K
AMD Radeon RX 580

Shadows Awakening (GOG)

default slice(*)

  FPS min  3.8    20.6
      avg 48.0    57.2
      mag 87.4    80.3

FT min 9.4 8.4

      avg  34.5   19.5
      max 107.4   37.2

FPS (Frames Per Second)
FT (FrameTime)

  [*] Command prefix: 'chrt -o --sched-runtime 280000 0'
      effectively setting 'base_slice_ns/10'

I have not compared to a kernel without flat on, just wanted to run non trivial workloads and play with slice to make sure everything 'works'.

Can also be had:

git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat

 include/linux/cpuset.h |    6 
 include/linux/sched.h  |    1 
 kernel/cgroup/cpuset.c |   15 
 kernel/sched/core.c    |   47 --
 kernel/sched/debug.c   |  171 +++++---
 kernel/sched/fair.c    | 1038 ++++++++++++++++++++++---------------------------
 kernel/sched/pelt.c    |    6 
 kernel/sched/sched.h   |   44 --
 8 files changed, 672 insertions(+), 656 deletions(-)

---
Change since v1 ( https://patch.msgid.link/[email protected] ):
- various Sashiko thingies
- rebase atop curren -tip

author maintainer
#1 lore raw

PATCH v2 1/10 sched/debug: Use char * instead of char (*)[]

Some of the fancy AI robots are getting 'upset'.

---
 kernel/sched/debug.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/debug.c +2 -2
@@ -136,7 +136,7 @@ sched_feat_write(struct file *filp, cons
 	if (cnt > 63)
 		cnt = 63;
 
-	if (copy_from_user(&buf, ubuf, cnt))
+	if (copy_from_user(buf, ubuf, cnt))
 		return -EFAULT;
 
 	buf[cnt] = 0;
@@ -221,7 +221,7 @@ static ssize_t sched_dynamic_write(struc
 	if (cnt > 15)
 		cnt = 15;
 
-	if (copy_from_user(&buf, ubuf, cnt))
+	if (copy_from_user(buf, ubuf, cnt))
 		return -EFAULT;
 
 	buf[cnt] = 0;
merge bot Applied
#2 lore raw

[tip: sched/core] sched/debug: Use char * instead of char (*)[]

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     333f6f0e11acc20d036f94f94709874f76d0b430
Gitweb:        https://git.kernel.org/tip/333f6f0e11acc20d036f94f94709874f76d0b430
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 11 May 2026 13:31:05 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Tue, 26 May 2026 13:53:12 +02:00

sched/debug: Use char * instead of char (*)[]

Some of the fancy AI robots are getting 'upset'.

---
 kernel/sched/debug.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/debug.c +2 -2
index ed3a0d6..af13a89 100644
@@ -136,7 +136,7 @@ sched_feat_write(struct file *filp, const char __user *ubuf,
 	if (cnt > 63)
 		cnt = 63;
 
-	if (copy_from_user(&buf, ubuf, cnt))
+	if (copy_from_user(buf, ubuf, cnt))
 		return -EFAULT;
 
 	buf[cnt] = 0;
@@ -263,7 +263,7 @@ static ssize_t sched_dynamic_write(struct file *filp, const char __user *ubuf,
 	if (cnt > 15)
 		cnt = 15;
 
-	if (copy_from_user(&buf, ubuf, cnt))
+	if (copy_from_user(buf, ubuf, cnt))
 		return -EFAULT;
 
 	buf[cnt] = 0;
author maintainer
#3 lore raw

PATCH v2 2/10 sched: Use {READ,WRITE}_ONCE() for preempt_dynamic_mode

In reply to Peter Zijlstra

Robots figured out you can read and write this concurrently and got 'upset'. Gemini even noted sched_dynamic_show() can generate 'confusing' output if it observed different values during the printing.

---
 kernel/sched/core.c  |   15 ++++++++-------
 kernel/sched/debug.c |    5 +++--
 2 files changed, 11 insertions(+), 9 deletions(-)
kernel/sched/core.c +8 -7
@@ -7743,7 +7743,7 @@ static void __sched_dynamic_update(int m
 		break;
 	}
 
-	preempt_dynamic_mode = mode;
+	WRITE_ONCE(preempt_dynamic_mode, mode);
 }
 
 void sched_dynamic_update(int mode)
@@ -7784,12 +7784,13 @@ static void __init preempt_dynamic_init(
 	}
 }
 
-# define PREEMPT_MODEL_ACCESSOR(mode) \
-	bool preempt_model_##mode(void)						 \
-	{									 \
-		WARN_ON_ONCE(preempt_dynamic_mode == preempt_dynamic_undefined); \
-		return preempt_dynamic_mode == preempt_dynamic_##mode;		 \
-	}									 \
+# define PREEMPT_MODEL_ACCESSOR(mode)					\
+	bool preempt_model_##mode(void)					\
+	{								\
+		int mode = READ_ONCE(preempt_dynamic_mode);		\
+		WARN_ON_ONCE(mode == preempt_dynamic_undefined);	\
+		return mode == preempt_dynamic_##mode;			\
+	}								\
 	EXPORT_SYMBOL_GPL(preempt_model_##mode)
 
 PREEMPT_MODEL_ACCESSOR(none);
kernel/sched/debug.c +3 -2
@@ -239,6 +239,7 @@ static ssize_t sched_dynamic_write(struc
 static int sched_dynamic_show(struct seq_file *m, void *v)
 {
 	int i = (IS_ENABLED(CONFIG_PREEMPT_RT) || IS_ENABLED(CONFIG_ARCH_HAS_PREEMPT_LAZY)) * 2;
+	int mode = READ_ONCE(preempt_dynamic_mode);
 	int j;
 
 	/* Count entries in NULL terminated preempt_modes */
@@ -247,10 +248,10 @@ static int sched_dynamic_show(struct seq
 	j -= !IS_ENABLED(CONFIG_ARCH_HAS_PREEMPT_LAZY);
 
 	for (; i < j; i++) {
-		if (preempt_dynamic_mode == i)
+		if (mode == i)
 			seq_puts(m, "(");
 		seq_puts(m, preempt_modes[i]);
-		if (preempt_dynamic_mode == i)
+		if (mode == i)
 			seq_puts(m, ")");
 
 		seq_puts(m, " ");
merge bot Applied
#4 lore raw

[tip: sched/core] sched: Use {READ,WRITE}_ONCE() for preempt_dynamic_mode

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     77557002234546a4fb46ebf517d6cb1d515535a9
Gitweb:        https://git.kernel.org/tip/77557002234546a4fb46ebf517d6cb1d515535a9
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 11 May 2026 13:31:06 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Tue, 26 May 2026 13:53:13 +02:00

sched: Use {READ,WRITE}_ONCE() for preempt_dynamic_mode

Robots figured out you can read and write this concurrently and got 'upset'. Gemini even noted sched_dynamic_show() can generate 'confusing' output if it observed different values during the printing.

---
 kernel/sched/core.c  | 15 ++++++++-------
 kernel/sched/debug.c |  5 +++--
 2 files changed, 11 insertions(+), 9 deletions(-)
kernel/sched/core.c +8 -7
index b3a416b..83202f0 100644
@@ -7979,7 +7979,7 @@ static void __sched_dynamic_update(int mode)
 		break;
 	}
 
-	preempt_dynamic_mode = mode;
+	WRITE_ONCE(preempt_dynamic_mode, mode);
 }
 
 void sched_dynamic_update(int mode)
@@ -8020,12 +8020,13 @@ static void __init preempt_dynamic_init(void)
 	}
 }
 
-# define PREEMPT_MODEL_ACCESSOR(mode) \
-	bool preempt_model_##mode(void)						 \
-	{									 \
-		WARN_ON_ONCE(preempt_dynamic_mode == preempt_dynamic_undefined); \
-		return preempt_dynamic_mode == preempt_dynamic_##mode;		 \
-	}									 \
+# define PREEMPT_MODEL_ACCESSOR(mode)					\
+	bool preempt_model_##mode(void)					\
+	{								\
+		int mode = READ_ONCE(preempt_dynamic_mode);		\
+		WARN_ON_ONCE(mode == preempt_dynamic_undefined);	\
+		return mode == preempt_dynamic_##mode;			\
+	}								\
 	EXPORT_SYMBOL_GPL(preempt_model_##mode)
 
 PREEMPT_MODEL_ACCESSOR(none);
kernel/sched/debug.c +3 -2
index af13a89..4cdf2f9 100644
@@ -281,6 +281,7 @@ static ssize_t sched_dynamic_write(struct file *filp, const char __user *ubuf,
 static int sched_dynamic_show(struct seq_file *m, void *v)
 {
 	int i = (IS_ENABLED(CONFIG_PREEMPT_RT) || IS_ENABLED(CONFIG_ARCH_HAS_PREEMPT_LAZY)) * 2;
+	int mode = READ_ONCE(preempt_dynamic_mode);
 	int j;
 
 	/* Count entries in NULL terminated preempt_modes */
@@ -289,10 +290,10 @@ static int sched_dynamic_show(struct seq_file *m, void *v)
 	j -= !IS_ENABLED(CONFIG_ARCH_HAS_PREEMPT_LAZY);
 
 	for (; i < j; i++) {
-		if (preempt_dynamic_mode == i)
+		if (mode == i)
 			seq_puts(m, "(");
 		seq_puts(m, preempt_modes[i]);
-		if (preempt_dynamic_mode == i)
+		if (mode == i)
 			seq_puts(m, ")");
 
 		seq_puts(m, " ");
author maintainer
#5 lore raw

PATCH v2 3/10 sched/debug: Collapse subsequent CONFIG_SCHED_CLASS_EXT sections

In reply to Peter Zijlstra
---
 kernel/sched/debug.c |   92 ++++++++++++++++++++++++---------------------------
 1 file changed, 44 insertions(+), 48 deletions(-)
kernel/sched/debug.c +44 -48
@@ -446,6 +446,8 @@ static const struct file_operations fair
 	.release	= single_release,
 };
 
+static struct dentry *debugfs_sched;
+
 #ifdef CONFIG_SCHED_CLASS_EXT
 static ssize_t
 sched_ext_server_runtime_write(struct file *filp, const char __user *ubuf,
@@ -478,75 +480,92 @@ static const struct file_operations ext_
 	.llseek		= seq_lseek,
 	.release	= single_release,
 };
-#endif /* CONFIG_SCHED_CLASS_EXT */
 
 static ssize_t
-sched_fair_server_period_write(struct file *filp, const char __user *ubuf,
-			       size_t cnt, loff_t *ppos)
+sched_ext_server_period_write(struct file *filp, const char __user *ubuf,
+			      size_t cnt, loff_t *ppos)
 {
 	long cpu = (long) ((struct seq_file *) filp->private_data)->private;
 	struct rq *rq = cpu_rq(cpu);
 
 	return sched_server_write_common(filp, ubuf, cnt, ppos, DL_PERIOD,
-					&rq->fair_server);
+					&rq->ext_server);
 }
 
-static int sched_fair_server_period_show(struct seq_file *m, void *v)
+static int sched_ext_server_period_show(struct seq_file *m, void *v)
 {
 	unsigned long cpu = (unsigned long) m->private;
 	struct rq *rq = cpu_rq(cpu);
 
-	return sched_server_show_common(m, v, DL_PERIOD, &rq->fair_server);
+	return sched_server_show_common(m, v, DL_PERIOD, &rq->ext_server);
 }
 
-static int sched_fair_server_period_open(struct inode *inode, struct file *filp)
+static int sched_ext_server_period_open(struct inode *inode, struct file *filp)
 {
-	return single_open(filp, sched_fair_server_period_show, inode->i_private);
+	return single_open(filp, sched_ext_server_period_show, inode->i_private);
 }
 
-static const struct file_operations fair_server_period_fops = {
-	.open		= sched_fair_server_period_open,
-	.write		= sched_fair_server_period_write,
+static const struct file_operations ext_server_period_fops = {
+	.open		= sched_ext_server_period_open,
+	.write		= sched_ext_server_period_write,
 	.read		= seq_read,
 	.llseek		= seq_lseek,
 	.release	= single_release,
 };
 
-#ifdef CONFIG_SCHED_CLASS_EXT
+static void debugfs_ext_server_init(void)
+{
+	struct dentry *d_ext;
+	unsigned long cpu;
+
+	d_ext = debugfs_create_dir("ext_server", debugfs_sched);
+	if (!d_ext)
+		return;
+
+	for_each_possible_cpu(cpu) {
+		struct dentry *d_cpu;
+		char buf[32];
+
+		snprintf(buf, sizeof(buf), "cpu%lu", cpu);
+		d_cpu = debugfs_create_dir(buf, d_ext);
+
+		debugfs_create_file("runtime", 0644, d_cpu, (void *) cpu, &ext_server_runtime_fops);
+		debugfs_create_file("period", 0644, d_cpu, (void *) cpu, &ext_server_period_fops);
+	}
+}
+#endif /* CONFIG_SCHED_CLASS_EXT */
+
 static ssize_t
-sched_ext_server_period_write(struct file *filp, const char __user *ubuf,
-			      size_t cnt, loff_t *ppos)
+sched_fair_server_period_write(struct file *filp, const char __user *ubuf,
+			       size_t cnt, loff_t *ppos)
 {
 	long cpu = (long) ((struct seq_file *) filp->private_data)->private;
 	struct rq *rq = cpu_rq(cpu);
 
 	return sched_server_write_common(filp, ubuf, cnt, ppos, DL_PERIOD,
-					&rq->ext_server);
+					&rq->fair_server);
 }
 
-static int sched_ext_server_period_show(struct seq_file *m, void *v)
+static int sched_fair_server_period_show(struct seq_file *m, void *v)
 {
 	unsigned long cpu = (unsigned long) m->private;
 	struct rq *rq = cpu_rq(cpu);
 
-	return sched_server_show_common(m, v, DL_PERIOD, &rq->ext_server);
+	return sched_server_show_common(m, v, DL_PERIOD, &rq->fair_server);
 }
 
-static int sched_ext_server_period_open(struct inode *inode, struct file *filp)
+static int sched_fair_server_period_open(struct inode *inode, struct file *filp)
 {
-	return single_open(filp, sched_ext_server_period_show, inode->i_private);
+	return single_open(filp, sched_fair_server_period_show, inode->i_private);
 }
 
-static const struct file_operations ext_server_period_fops = {
-	.open		= sched_ext_server_period_open,
-	.write		= sched_ext_server_period_write,
+static const struct file_operations fair_server_period_fops = {
+	.open		= sched_fair_server_period_open,
+	.write		= sched_fair_server_period_write,
 	.read		= seq_read,
 	.llseek		= seq_lseek,
 	.release	= single_release,
 };
-#endif /* CONFIG_SCHED_CLASS_EXT */
-
-static struct dentry *debugfs_sched;
 
 static void debugfs_fair_server_init(void)
 {
@@ -569,29 +588,6 @@ static void debugfs_fair_server_init(voi
 	}
 }
 
-#ifdef CONFIG_SCHED_CLASS_EXT
-static void debugfs_ext_server_init(void)
-{
-	struct dentry *d_ext;
-	unsigned long cpu;
-
-	d_ext = debugfs_create_dir("ext_server", debugfs_sched);
-	if (!d_ext)
-		return;
-
-	for_each_possible_cpu(cpu) {
-		struct dentry *d_cpu;
-		char buf[32];
-
-		snprintf(buf, sizeof(buf), "cpu%lu", cpu);
-		d_cpu = debugfs_create_dir(buf, d_ext);
-
-		debugfs_create_file("runtime", 0644, d_cpu, (void *) cpu, &ext_server_runtime_fops);
-		debugfs_create_file("period", 0644, d_cpu, (void *) cpu, &ext_server_period_fops);
-	}
-}
-#endif /* CONFIG_SCHED_CLASS_EXT */
-
 static __init int sched_init_debug(void)
 {
 	struct dentry __maybe_unused *numa;
merge bot Applied
#6 lore raw

[tip: sched/core] sched/debug: Collapse subsequent CONFIG_SCHED_CLASS_EXT sections

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     e05777c44e53df8ab41d930510a384d65a34aafa
Gitweb:        https://git.kernel.org/tip/e05777c44e53df8ab41d930510a384d65a34aafa
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 11 May 2026 13:31:07 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Tue, 26 May 2026 13:53:13 +02:00

sched/debug: Collapse subsequent CONFIG_SCHED_CLASS_EXT sections

---
 kernel/sched/debug.c | 92 ++++++++++++++++++++-----------------------
 1 file changed, 44 insertions(+), 48 deletions(-)
kernel/sched/debug.c +44 -48
index 4cdf2f9..4809e1d 100644
@@ -488,6 +488,8 @@ static const struct file_operations fair_server_runtime_fops = {
 	.release	= single_release,
 };
 
+static struct dentry *debugfs_sched;
+
 #ifdef CONFIG_SCHED_CLASS_EXT
 static ssize_t
 sched_ext_server_runtime_write(struct file *filp, const char __user *ubuf,
@@ -520,75 +522,92 @@ static const struct file_operations ext_server_runtime_fops = {
 	.llseek		= seq_lseek,
 	.release	= single_release,
 };
-#endif /* CONFIG_SCHED_CLASS_EXT */
 
 static ssize_t
-sched_fair_server_period_write(struct file *filp, const char __user *ubuf,
-			       size_t cnt, loff_t *ppos)
+sched_ext_server_period_write(struct file *filp, const char __user *ubuf,
+			      size_t cnt, loff_t *ppos)
 {
 	long cpu = (long) ((struct seq_file *) filp->private_data)->private;
 	struct rq *rq = cpu_rq(cpu);
 
 	return sched_server_write_common(filp, ubuf, cnt, ppos, DL_PERIOD,
-					&rq->fair_server);
+					&rq->ext_server);
 }
 
-static int sched_fair_server_period_show(struct seq_file *m, void *v)
+static int sched_ext_server_period_show(struct seq_file *m, void *v)
 {
 	unsigned long cpu = (unsigned long) m->private;
 	struct rq *rq = cpu_rq(cpu);
 
-	return sched_server_show_common(m, v, DL_PERIOD, &rq->fair_server);
+	return sched_server_show_common(m, v, DL_PERIOD, &rq->ext_server);
 }
 
-static int sched_fair_server_period_open(struct inode *inode, struct file *filp)
+static int sched_ext_server_period_open(struct inode *inode, struct file *filp)
 {
-	return single_open(filp, sched_fair_server_period_show, inode->i_private);
+	return single_open(filp, sched_ext_server_period_show, inode->i_private);
 }
 
-static const struct file_operations fair_server_period_fops = {
-	.open		= sched_fair_server_period_open,
-	.write		= sched_fair_server_period_write,
+static const struct file_operations ext_server_period_fops = {
+	.open		= sched_ext_server_period_open,
+	.write		= sched_ext_server_period_write,
 	.read		= seq_read,
 	.llseek		= seq_lseek,
 	.release	= single_release,
 };
 
-#ifdef CONFIG_SCHED_CLASS_EXT
+static void debugfs_ext_server_init(void)
+{
+	struct dentry *d_ext;
+	unsigned long cpu;
+
+	d_ext = debugfs_create_dir("ext_server", debugfs_sched);
+	if (!d_ext)
+		return;
+
+	for_each_possible_cpu(cpu) {
+		struct dentry *d_cpu;
+		char buf[32];
+
+		snprintf(buf, sizeof(buf), "cpu%lu", cpu);
+		d_cpu = debugfs_create_dir(buf, d_ext);
+
+		debugfs_create_file("runtime", 0644, d_cpu, (void *) cpu, &ext_server_runtime_fops);
+		debugfs_create_file("period", 0644, d_cpu, (void *) cpu, &ext_server_period_fops);
+	}
+}
+#endif /* CONFIG_SCHED_CLASS_EXT */
+
 static ssize_t
-sched_ext_server_period_write(struct file *filp, const char __user *ubuf,
-			      size_t cnt, loff_t *ppos)
+sched_fair_server_period_write(struct file *filp, const char __user *ubuf,
+			       size_t cnt, loff_t *ppos)
 {
 	long cpu = (long) ((struct seq_file *) filp->private_data)->private;
 	struct rq *rq = cpu_rq(cpu);
 
 	return sched_server_write_common(filp, ubuf, cnt, ppos, DL_PERIOD,
-					&rq->ext_server);
+					&rq->fair_server);
 }
 
-static int sched_ext_server_period_show(struct seq_file *m, void *v)
+static int sched_fair_server_period_show(struct seq_file *m, void *v)
 {
 	unsigned long cpu = (unsigned long) m->private;
 	struct rq *rq = cpu_rq(cpu);
 
-	return sched_server_show_common(m, v, DL_PERIOD, &rq->ext_server);
+	return sched_server_show_common(m, v, DL_PERIOD, &rq->fair_server);
 }
 
-static int sched_ext_server_period_open(struct inode *inode, struct file *filp)
+static int sched_fair_server_period_open(struct inode *inode, struct file *filp)
 {
-	return single_open(filp, sched_ext_server_period_show, inode->i_private);
+	return single_open(filp, sched_fair_server_period_show, inode->i_private);
 }
 
-static const struct file_operations ext_server_period_fops = {
-	.open		= sched_ext_server_period_open,
-	.write		= sched_ext_server_period_write,
+static const struct file_operations fair_server_period_fops = {
+	.open		= sched_fair_server_period_open,
+	.write		= sched_fair_server_period_write,
 	.read		= seq_read,
 	.llseek		= seq_lseek,
 	.release	= single_release,
 };
-#endif /* CONFIG_SCHED_CLASS_EXT */
-
-static struct dentry *debugfs_sched;
 
 static void debugfs_fair_server_init(void)
 {
@@ -611,29 +630,6 @@ static void debugfs_fair_server_init(void)
 	}
 }
 
-#ifdef CONFIG_SCHED_CLASS_EXT
-static void debugfs_ext_server_init(void)
-{
-	struct dentry *d_ext;
-	unsigned long cpu;
-
-	d_ext = debugfs_create_dir("ext_server", debugfs_sched);
-	if (!d_ext)
-		return;
-
-	for_each_possible_cpu(cpu) {
-		struct dentry *d_cpu;
-		char buf[32];
-
-		snprintf(buf, sizeof(buf), "cpu%lu", cpu);
-		d_cpu = debugfs_create_dir(buf, d_ext);
-
-		debugfs_create_file("runtime", 0644, d_cpu, (void *) cpu, &ext_server_runtime_fops);
-		debugfs_create_file("period", 0644, d_cpu, (void *) cpu, &ext_server_period_fops);
-	}
-}
-#endif /* CONFIG_SCHED_CLASS_EXT */
-
 static __init int sched_init_debug(void)
 {
 	struct dentry __maybe_unused *numa, *llc;
author maintainer
#7 lore raw

PATCH v2 4/10 sched/fair: Add cgroup_mode switch

In reply to Peter Zijlstra

Since calc_group_shares() has issues with 'many' CPUs, specifically the computed shares value gets to be roughly 1/nr_cpus, prepare to add a few alternative methods.

---
 kernel/sched/debug.c |   74 +++++++++++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h |    1 
 2 files changed, 75 insertions(+)
kernel/sched/debug.c +74 -0
@@ -588,6 +588,76 @@ static void debugfs_fair_server_init(voi
 	}
 }
 
+#ifdef CONFIG_FAIR_GROUP_SCHED
+int cgroup_mode = 0;
+
+static const char *cgroup_mode_str[] = {
+	"smp",
+};
+
+static int sched_cgroup_mode(const char *str)
+{
+	for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+		if (!strcmp(str, cgroup_mode_str[i]))
+			return i;
+	}
+	return -EINVAL;
+}
+
+static ssize_t sched_cgroup_write(struct file *filp, const char __user *ubuf,
+				   size_t cnt, loff_t *ppos)
+{
+	char buf[16];
+	int mode;
+
+	if (cnt > 15)
+		cnt = 15;
+
+	if (copy_from_user(buf, ubuf, cnt))
+		return -EFAULT;
+
+	buf[cnt] = 0;
+	mode = sched_cgroup_mode(strstrip(buf));
+	if (mode < 0)
+		return mode;
+
+	WRITE_ONCE(cgroup_mode, mode);
+
+	*ppos += cnt;
+	return cnt;
+}
+
+static int sched_cgroup_show(struct seq_file *m, void *v)
+{
+	int mode = READ_ONCE(cgroup_mode);
+
+	for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+		if (mode == i)
+			seq_puts(m, "(");
+		seq_puts(m, cgroup_mode_str[i]);
+		if (mode == i)
+			seq_puts(m, ")");
+
+		seq_puts(m, " ");
+	}
+	seq_puts(m, "\n");
+	return 0;
+}
+
+static int sched_cgroup_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, sched_cgroup_show, NULL);
+}
+
+static const struct file_operations sched_cgroup_fops = {
+	.open		= sched_cgroup_open,
+	.write		= sched_cgroup_write,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+#endif
+
 static __init int sched_init_debug(void)
 {
 	struct dentry __maybe_unused *numa;
@@ -625,6 +695,10 @@ static __init int sched_init_debug(void)
 
 	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
 
+#ifdef CONFIG_FAIR_GROUP_SCHED
+	debugfs_create_file("cgroup_mode", 0644, debugfs_sched, NULL, &sched_cgroup_fops);
+#endif
+
 	debugfs_fair_server_init();
 #ifdef CONFIG_SCHED_CLASS_EXT
 	debugfs_ext_server_init();
kernel/sched/sched.h +1 -0
@@ -565,6 +565,7 @@ static inline struct task_group *css_tg(
 extern int tg_nop(struct task_group *tg, void *data);
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
+extern int cgroup_mode;
 extern void free_fair_sched_group(struct task_group *tg);
 extern int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent);
 extern void online_fair_sched_group(struct task_group *tg);
author maintainer
#8 lore raw

PATCH v2 5/10 sched/fair: Add cgroup_mode: UP

In reply to Peter Zijlstra

Instead of calculating the proportional fraction of tg->shares for each CPU, just give each CPU the full measure, ignoring these pesky SMP problems.

---
 kernel/sched/debug.c |    3 ++-
 kernel/sched/fair.c  |   21 ++++++++++++++++++++-
 2 files changed, 22 insertions(+), 2 deletions(-)
kernel/sched/debug.c +2 -1
@@ -589,9 +589,10 @@ static void debugfs_fair_server_init(voi
 }
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
-int cgroup_mode = 0;
+int cgroup_mode = 1;
 
 static const char *cgroup_mode_str[] = {
+	"up",
 	"smp",
 };
 
kernel/sched/fair.c +20 -1
@@ -4150,7 +4150,7 @@ static inline int throttled_hierarchy(st
  *
  * hence icky!
  */
-static long calc_group_shares(struct cfs_rq *cfs_rq)
+static long calc_smp_shares(struct cfs_rq *cfs_rq)
 {
 	long tg_weight, tg_shares, load, shares;
 	struct task_group *tg = cfs_rq->tg;
@@ -4185,6 +4185,25 @@ static long calc_group_shares(struct cfs
 }
 
 /*
+ * Ignore this pesky SMP stuff, use (4).
+ */
+static long calc_up_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	return READ_ONCE(tg->shares);
+}
+
+static long calc_group_shares(struct cfs_rq *cfs_rq)
+{
+	int mode = READ_ONCE(cgroup_mode);
+
+	if (mode == 0)
+		return calc_up_shares(cfs_rq);
+
+	return calc_smp_shares(cfs_rq);
+}
+
+/*
  * Recomputes the group entity based on the current state of its group
  * runqueue.
  */
author maintainer
#9 lore raw

PATCH v2 6/10 sched/fair: Add cgroup_mode: MAX

In reply to Peter Zijlstra

In order to avoid the CPU shares becoming tiny '1 / nr_cpus', assume each cgroup is maximally concurrent and distrubute 'nr_cpus * tg->shares', such that each CPU ends up with a 'tg->shares' sized fraction (on average).

There is the corner case, when a cgroup is minimally loaded, eg a single spinner, therefore limit the CPU shares to that of a nice -20 task to avoid getting too much load.

It was previously suggested to allow raising cpu.weight to '100 * nr_cpus' to combat this same problem, but the problem there is the above corner case, allowing multiple cgroups with such immense weight to the runqueue has significant problems.

It would drown the kthreads, but it also risks overflowing the load values.

---
 include/linux/cpuset.h |    6 +++++
 kernel/cgroup/cpuset.c |   15 ++++++++++++++
 kernel/sched/debug.c   |    1 
 kernel/sched/fair.c    |   52 ++++++++++++++++++++++++++++++++++++++++++++-----
 4 files changed, 69 insertions(+), 5 deletions(-)
include/linux/cpuset.h +6 -0
@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_h
 extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
 extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
 extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
+extern int cpuset_num_cpus(struct cgroup *cgroup);
 extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
 #define cpuset_current_mems_allowed (current->mems_allowed)
 void cpuset_init_current_mems_allowed(void);
@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_f
 	return false;
 }
 
+static inline int cpuset_num_cpus(struct cgroup *cgroup)
+{
+	return num_online_cpus();
+}
+
 static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
 {
 	return node_possible_map;
kernel/cgroup/cpuset.c +15 -0
@@ -4100,6 +4100,21 @@ bool cpuset_cpus_allowed_fallback(struct
 	return changed;
 }
 
+int cpuset_num_cpus(struct cgroup *cgrp)
+{
+	int nr = num_online_cpus();
+	struct cpuset *cs;
+
+	if (is_in_v2_mode()) {
+		guard(rcu)();
+		cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
+		if (cs)
+			nr = cpumask_weight(cs->effective_cpus);
+	}
+
+	return nr;
+}
+
 void __init cpuset_init_current_mems_allowed(void)
 {
 	nodes_setall(current->mems_allowed);
kernel/sched/debug.c +1 -0
@@ -594,6 +594,7 @@ int cgroup_mode = 1;
 static const char *cgroup_mode_str[] = {
 	"up",
 	"smp",
+	"max",
 };
 
 static int sched_cgroup_mode(const char *str)
kernel/sched/fair.c +47 -5

72 lines not on this page · raw message

author maintainer
#10 lore raw

PATCH v2 7/10 sched/fair: Add cgroup_mode: CONCUR

In reply to Peter Zijlstra

A variation of MAX; where instead of assuming maximal concurrent, this scales with 'min(nr_tasks, nr_cpus)'. This handles the low concurrency cases more gracefully, with the exception of CPU affnity.

Note: the tracking of tg->tasks is somewhat expensive :-/

---
 kernel/sched/debug.c |    1 +
 kernel/sched/fair.c  |   39 ++++++++++++++++++++++++++++++++++++---
 kernel/sched/sched.h |    3 +++
 3 files changed, 40 insertions(+), 3 deletions(-)
kernel/sched/debug.c +1 -0
@@ -594,6 +594,7 @@ int cgroup_mode = 1;
 static const char *cgroup_mode_str[] = {
 	"up",
 	"smp",
+	"concur",
 	"max",
 };
 
kernel/sched/fair.c +36 -3
@@ -4211,6 +4211,30 @@ static long calc_max_shares(struct cfs_r
 	return __calc_smp_shares(cfs_rq, tg_shares * nr, max_shares);
 }
 
+static inline int tg_tasks(struct task_group *tg)
+{
+	return max(1, atomic_long_read(&tg->tasks));
+}
+
+/*
+ * Func: min(fraction(num * tg->shares), nice -20); where
+ *       num = min(nr_tasks, nr_cpus)
+ *
+ * Similar to max, except scale with min(nr_tasks, nr_cpus), which gives
+ * a far more natural distrubution. Can still create edge case using CPU
+ * affinity.
+ */
+static long calc_concur_shares(struct cfs_rq *cfs_rq)
+{
+	struct task_group *tg = cfs_rq->tg;
+	int nr_cpus = tg_cpus(tg);
+	int nr_tasks = tg_tasks(tg);
+	int nr = min(nr_tasks, nr_cpus);
+	long tg_shares = READ_ONCE(tg->shares);
+	long max_shares = scale_load(sched_prio_to_weight[0]);
+	return __calc_smp_shares(cfs_rq, nr * tg_shares, max_shares);
+}
+
 /*
  * Func: fraction(tg->shares)
  *
@@ -4240,6 +4264,9 @@ static long calc_group_shares(struct cfs
 		return calc_up_shares(cfs_rq);
 
 	if (mode == 2)
+		return calc_concur_shares(cfs_rq);
+
+	if (mode == 3)
 		return calc_max_shares(cfs_rq);
 
 	return calc_smp_shares(cfs_rq);
@@ -4385,7 +4412,7 @@ static inline bool cfs_rq_is_decayed(str
  */
 static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
 {
-	long delta;
+	long delta, dt;
 	u64 now;
 
 	/*
@@ -4407,16 +4434,19 @@ static inline void update_tg_load_avg(st
 		return;
 
 	delta = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib;
-	if (abs(delta) > cfs_rq->tg_load_avg_contrib / 64) {
+	dt = cfs_rq->h_nr_queued - cfs_rq->tg_tasks_contrib;
+	if (dt || abs(delta) > cfs_rq->tg_load_avg_contrib / 64) {
 		atomic_long_add(delta, &cfs_rq->tg->load_avg);
+		atomic_long_add(dt, &cfs_rq->tg->tasks);
 		cfs_rq->tg_load_avg_contrib = cfs_rq->avg.load_avg;
+		cfs_rq->tg_tasks_contrib = cfs_rq->h_nr_queued;
 		cfs_rq->last_update_tg_load_avg = now;
 	}
 }
 
 static inline void clear_tg_load_avg(struct cfs_rq *cfs_rq)
 {
-	long delta;
+	long delta, dt;
 	u64 now;
 
 	/*
@@ -4427,8 +4457,11 @@ static inline void clear_tg_load_avg(str
 
 	now = sched_clock_cpu(cpu_of(rq_of(cfs_rq)));
 	delta = 0 - cfs_rq->tg_load_avg_contrib;
+	dt = 0 - cfs_rq->tg_tasks_contrib;
 	atomic_long_add(delta, &cfs_rq->tg->load_avg);
+	atomic_long_add(dt, &cfs_rq->tg->tasks);
 	cfs_rq->tg_load_avg_contrib = 0;
+	cfs_rq->tg_tasks_contrib = 0;
 	cfs_rq->last_update_tg_load_avg = now;
 }
 
kernel/sched/sched.h +3 -0
@@ -491,6 +491,8 @@ struct task_group {
 	 * will also be accessed at each tick.
 	 */
 	atomic_long_t		load_avg ____cacheline_aligned;
+	atomic_long_t		tasks;
+
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 #ifdef CONFIG_RT_GROUP_SCHED
@@ -720,6 +722,7 @@ struct cfs_rq {
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	u64			last_update_tg_load_avg;
 	unsigned long		tg_load_avg_contrib;
+	unsigned long		tg_tasks_contrib;
 	long			propagate;
 	long			prop_runnable_sum;
 
author maintainer
#11 lore raw

PATCH v2 8/10 sched/fair: Add newidle balance to pick_task_fair()

In reply to Peter Zijlstra

With commit 50653216e4ff ("sched: Add support to pick functions to take rf") removing the balance callback, the pick_task() callback is in charge of newidle balancing.

This means pick_task_fair() should do so too. This hasn't been a problem in practise because pick_next_task_fair() is used. However, since we'll be removing that one shortly, make sure pick_next_task() is up to scratch.

---
 kernel/sched/fair.c |   38 +++++++++++++++-----------------------
 1 file changed, 15 insertions(+), 23 deletions(-)
kernel/sched/fair.c +15 -23
@@ -9215,16 +9215,18 @@ static void wakeup_preempt_fair(struct r
 }
 
 static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
+	__must_hold(__rq_lockp(rq))
 {
 	struct sched_entity *se;
 	struct cfs_rq *cfs_rq;
 	struct task_struct *p;
 	bool throttled;
+	int new_tasks;
 
 again:
 	cfs_rq = &rq->cfs;
 	if (!cfs_rq->nr_queued)
-		return NULL;
+		goto idle;
 
 	throttled = false;
 
@@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_fai
 	if (unlikely(throttled))
 		task_throttle_setup_work(p);
 	return p;
+
+idle:
+	new_tasks = sched_balance_newidle(rq, rf);
+	if (new_tasks < 0)
+		return RETRY_TASK;
+	if (new_tasks > 0)
+		goto again;
+	return NULL;
 }
 
 static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
@@ -9256,12 +9266,12 @@ pick_next_task_fair(struct rq *rq, struc
 {
 	struct sched_entity *se;
 	struct task_struct *p;
-	int new_tasks;
 
-again:
 	p = pick_task_fair(rq, rf);
+	if (unlikely(p == RETRY_TASK))
+		return p;
 	if (!p)
-		goto idle;
+		return p;
 	se = &p->se;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9311,29 +9321,11 @@ pick_next_task_fair(struct rq *rq, struc
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 	put_prev_set_next_task(rq, prev, p);
 	return p;
-
-idle:
-	if (rf) {
-		new_tasks = sched_balance_newidle(rq, rf);
-
-		/*
-		 * Because sched_balance_newidle() releases (and re-acquires)
-		 * rq->lock, it is possible for any higher priority task to
-		 * appear. In that case we must re-start the pick_next_entity()
-		 * loop.
-		 */
-		if (new_tasks < 0)
-			return RETRY_TASK;
-
-		if (new_tasks > 0)
-			goto again;
-	}
-
-	return NULL;
 }
 
 static struct task_struct *
 fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
+	__must_hold(__rq_lockp(dl_se->rq))
 {
 	return pick_task_fair(dl_se->rq, rf);
 }
#12 lore raw

Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()

Hello Peter,

kernel/sched/fair.c static struct task_struct *pick_task_fai · line 9258

Quoted 13 lines · Peter Zijlstra
kernel/sched/fair.c +8 -0
@@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_fai
 	if (unlikely(throttled))
 		task_throttle_setup_work(p);
 	return p;
+
+idle:
+	new_tasks = sched_balance_newidle(rq, rf);
+	if (new_tasks < 0)
+		return RETRY_TASK;
+	if (new_tasks > 0)
+		goto again;
+	return NULL;
 }

For core scheduling will now trigger a newidle balance during the pick when core_cookie is reset to 0 which can cause tasks to migrate only for them to find they cannot run on the CPU since core-wide selection leads to a cookie mismatch and it is kept hanging there.

Can we return early if sched_core_enabled() here or are the additional newidle balance okay?

Signature · 2 lines
Thanks and Regards,
Prateek
author maintainer
#13 lore raw

Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()

Quoted 22 lines · K Prateek Nayak

Hello Peter,

kernel/sched/fair.c static struct task_struct *pick_task_fai · line 9258

Quoted 13 lines · Peter Zijlstra
kernel/sched/fair.c +8 -0
@@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_fai
 	if (unlikely(throttled))
 		task_throttle_setup_work(p);
 	return p;
+
+idle:
+	new_tasks = sched_balance_newidle(rq, rf);
+	if (new_tasks < 0)
+		return RETRY_TASK;
+	if (new_tasks > 0)
+		goto again;
+	return NULL;
 }

For core scheduling will now trigger a newidle balance during the pick when core_cookie is reset to 0 which can cause tasks to migrate only for them to find they cannot run on the CPU since core-wide selection leads to a cookie mismatch and it is kept hanging there.

Quoted 2 lines

Can we return early if sched_core_enabled() here or are the additional newidle balance okay?

This basically makes fair behave like every other class, so in that sense this is probably okay. That said, fair is the most common case, so perhaps.

Lets see if the people actually using this notice first though ;-)

maintainer Reviewed-by on v2 8/10
#14 lore raw

Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()

In reply to Peter Zijlstra
Quoted 11 lines · Peter Zijlstra

With commit 50653216e4ff ("sched: Add support to pick functions to take rf") removing the balance callback, the pick_task() callback is in charge of newidle balancing.

This means pick_task_fair() should do so too. This hasn't been a problem in practise because pick_next_task_fair() is used. However, since we'll be removing that one shortly, make sure pick_next_task() is up to scratch.

Quoted 90 lines
---
 kernel/sched/fair.c |   38 +++++++++++++++-----------------------
 1 file changed, 15 insertions(+), 23 deletions(-)
kernel/sched/fair.c +15 -23
@@ -9215,16 +9215,18 @@ static void wakeup_preempt_fair(struct r
 }
 
 static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
+       __must_hold(__rq_lockp(rq))
 {
        struct sched_entity *se;
        struct cfs_rq *cfs_rq;
        struct task_struct *p;
        bool throttled;
+       int new_tasks;
 
 again:
        cfs_rq = &rq->cfs;
        if (!cfs_rq->nr_queued)
-               return NULL;
+               goto idle;
 
        throttled = false;
 
@@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_fai
        if (unlikely(throttled))
                task_throttle_setup_work(p);
        return p;
+
+idle:
+       new_tasks = sched_balance_newidle(rq, rf);
+       if (new_tasks < 0)
+               return RETRY_TASK;
+       if (new_tasks > 0)
+               goto again;
+       return NULL;
 }
 
 static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
@@ -9256,12 +9266,12 @@ pick_next_task_fair(struct rq *rq, struc
 {
        struct sched_entity *se;
        struct task_struct *p;
-       int new_tasks;
 
-again:
        p = pick_task_fair(rq, rf);
+       if (unlikely(p == RETRY_TASK))
+               return p;
        if (!p)
-               goto idle;
+               return p;
        se = &p->se;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9311,29 +9321,11 @@ pick_next_task_fair(struct rq *rq, struc
 #endif /* CONFIG_FAIR_GROUP_SCHED */
        put_prev_set_next_task(rq, prev, p);
        return p;
-
-idle:
-       if (rf) {
-               new_tasks = sched_balance_newidle(rq, rf);
-
-               /*
-                * Because sched_balance_newidle() releases (and re-acquires)
-                * rq->lock, it is possible for any higher priority task to
-                * appear. In that case we must re-start the pick_next_entity()
-                * loop.
-                */
-               if (new_tasks < 0)
-                       return RETRY_TASK;
-
-               if (new_tasks > 0)
-                       goto again;
-       }
-
-       return NULL;
 }
 
 static struct task_struct *
 fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
+       __must_hold(__rq_lockp(dl_se->rq))
 {
        return pick_task_fair(dl_se->rq, rf);
 }
merge bot Applied
#15 lore raw

[tip: sched/core] sched/fair: Add newidle balance to pick_task_fair()

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     b3a2dfa8b42e5b97dd144aa59374f4e045725cac
Gitweb:        https://git.kernel.org/tip/b3a2dfa8b42e5b97dd144aa59374f4e045725cac
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 11 May 2026 13:31:12 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Tue, 26 May 2026 13:53:13 +02:00

sched/fair: Add newidle balance to pick_task_fair()

With commit 50653216e4ff ("sched: Add support to pick functions to take rf") removing the balance callback, the pick_task() callback is in charge of newidle balancing.

This means pick_task_fair() should do so too. This hasn't been a problem in practise because pick_next_task_fair() is used. However, since we'll be removing that one shortly, make sure pick_next_task() is up to scratch.

---
 kernel/sched/fair.c | 38 +++++++++++++++-----------------------
 1 file changed, 15 insertions(+), 23 deletions(-)
kernel/sched/fair.c +15 -23
index 8e858ca..5f48af7 100644
@@ -9863,16 +9863,18 @@ preempt:
 }
 
 static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
+	__must_hold(__rq_lockp(rq))
 {
 	struct sched_entity *se;
 	struct cfs_rq *cfs_rq;
 	struct task_struct *p;
 	bool throttled;
+	int new_tasks;
 
 again:
 	cfs_rq = &rq->cfs;
 	if (!cfs_rq->nr_queued)
-		return NULL;
+		goto idle;
 
 	throttled = false;
 
@@ -9893,6 +9895,14 @@ again:
 	if (unlikely(throttled))
 		task_throttle_setup_work(p);
 	return p;
+
+idle:
+	new_tasks = sched_balance_newidle(rq, rf);
+	if (new_tasks < 0)
+		return RETRY_TASK;
+	if (new_tasks > 0)
+		goto again;
+	return NULL;
 }
 
 static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
@@ -9904,12 +9914,12 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf
 {
 	struct sched_entity *se;
 	struct task_struct *p;
-	int new_tasks;
 
-again:
 	p = pick_task_fair(rq, rf);
+	if (unlikely(p == RETRY_TASK))
+		return p;
 	if (!p)
-		goto idle;
+		return p;
 	se = &p->se;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9959,29 +9969,11 @@ simple:
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 	put_prev_set_next_task(rq, prev, p);
 	return p;
-
-idle:
-	if (rf) {
-		new_tasks = sched_balance_newidle(rq, rf);
-
-		/*
-		 * Because sched_balance_newidle() releases (and re-acquires)
-		 * rq->lock, it is possible for any higher priority task to
-		 * appear. In that case we must re-start the pick_next_entity()
-		 * loop.
-		 */
-		if (new_tasks < 0)
-			return RETRY_TASK;
-
-		if (new_tasks > 0)
-			goto again;
-	}
-
-	return NULL;
 }
 
 static struct task_struct *
 fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
+	__must_hold(__rq_lockp(dl_se->rq))
 {
 	return pick_task_fair(dl_se->rq, rf);
 }
#16 lore raw

Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()

In reply to Peter Zijlstra

Hi Peter,

Quoted 8 lines · Peter Zijlstra

With commit 50653216e4ff ("sched: Add support to pick functions to take rf") removing the balance callback, the pick_task() callback is in charge of newidle balancing.

This means pick_task_fair() should do so too. This hasn't been a problem in practise because pick_next_task_fair() is used. However, since we'll be removing that one shortly, make sure pick_next_task() is up to scratch.

While testing Prateek's throttle series, I noticed a panic issue when coresched is enabled and bisected to this patch.

I fed the panic log and this patch to an agent and its analysis looks correct to me(cpu56 and cpu57 are siblings in a VM):

       cpu57 (holds core-wide lock)

     pick_next_task() [core scheduling]
     for_each_cpu_wrap(i, smt_mask, 57):
       i=57: pick_task(rq_57)
             pick_task_fair(rq_57)
             -> picks task A
       rq_57->core_pick = task A
       // task_rq(A) == rq_57

       i=56: pick_task(rq_56)
             pick_task_fair(rq_56)
             cfs_rq->nr_queued == 0
             goto idle
             sched_balance_newidle(rq_56)
             raw_spin_rq_unlock(rq_56)
             // core-wide lock released
             newidle_balance() pulls
               task A: rq_57 -> rq_56
             // task_rq(A) == rq_56 now
             raw_spin_rq_lock(rq_56)
             // core-wide lock re-acquired
             return > 0
             goto again
             pick_task_fair(rq_56)
             -> picks task A
       rq_56->core_pick = task A

     // first loop done
     // rq_57->core_pick is still task A (set before lock release)
     // but task_rq(A) == rq_56 now
     next = rq_57->core_pick  // = task A

     put_prev_set_next_task(rq_57, prev, task A)
     __set_next_task_fair(rq_57, task A)
     hrtick_start_fair(rq_57, task A)
     WARN_ON_ONCE(task_rq(task A) != rq_57)
     // task_rq(A) == rq_56

I applied below diff and the problem is gone:

kernel/sched/fair.c +3 -0
index 5f48af700fd44..942a543af3e54 100644
@@ -9897,6 +9897,9 @@ static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
 	return p;
 
 idle:
+	if (sched_core_enabled(rq))
+		return NULL;
+
 	new_tasks = sched_balance_newidle(rq, rf);
 	if (new_tasks < 0)
 		return RETRY_TASK;

Full dmesg of the panic is attached for your reference.

author maintainer Tested-by Sven Schnelle
#17 lore raw

Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()

Aaron,

Sorry I failed to notice this email earlier.

Quoted 8 lines · Aaron Lu

I applied below diff and the problem is gone:

kernel/sched/fair.c +0 -0
index 5f48af700fd44..942a543af3e54 100644

kernel/sched/fair.c static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) · line 9905

Quoted 9 lines
kernel/sched/fair.c +3 -0
@@ -9897,6 +9897,9 @@ static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
 	return p;
 
 idle:
+	if (sched_core_enabled(rq))
+		return NULL;
+
 	new_tasks = sched_balance_newidle(rq, rf);
 	if (new_tasks < 0)
 		return RETRY_TASK;

Right, this is the safe patch and restores pick_task_fair() to its previous status (for core-sched).

Since people are hitting this problem, I'm going to merge it as below. I've presumed your SoB, please let me know if that's a problem.

I think I'm going to try and move newidle into sched_class::balance / balance_fair(), but I'll do that next cycle.

Thanks!

---
Subject: sched/fair: Fix newidle vs core-sched
From: "Aaron Lu" <[email protected]>
Date: Wed, 3 Jun 2026 17:51:08 +0800

From: "Aaron Lu" <[email protected]>

While testing Prateek's throttle series, I noticed a panic issue when coresched is enabled and bisected to this patch.

I fed the panic log and this patch to an agent and its analysis looks correct to me(cpu56 and cpu57 are siblings in a VM):

       cpu57 (holds core-wide lock)

     pick_next_task() [core scheduling]
     for_each_cpu_wrap(i, smt_mask, 57):
       i=57: pick_task(rq_57)
             pick_task_fair(rq_57)
             -> picks task A
       rq_57->core_pick = task A
       // task_rq(A) == rq_57

       i=56: pick_task(rq_56)
             pick_task_fair(rq_56)
             cfs_rq->nr_queued == 0
             goto idle
             sched_balance_newidle(rq_56)
             raw_spin_rq_unlock(rq_56)
             // core-wide lock released
             newidle_balance() pulls
               task A: rq_57 -> rq_56
             // task_rq(A) == rq_56 now
             raw_spin_rq_lock(rq_56)
             // core-wide lock re-acquired
             return > 0
             goto again
             pick_task_fair(rq_56)
             -> picks task A
       rq_56->core_pick = task A

     // first loop done
     // rq_57->core_pick is still task A (set before lock release)
     // but task_rq(A) == rq_56 now
     next = rq_57->core_pick  // = task A

     put_prev_set_next_task(rq_57, prev, task A)
     __set_next_task_fair(rq_57, task A)
     hrtick_start_fair(rq_57, task A)
     WARN_ON_ONCE(task_rq(task A) != rq_57)
     // task_rq(A) == rq_56

IOW: by allowing pick_task_fair() to do newidle_balance and not returning RETRY_TASK, it can end up selecting the same task on two CPUs. Restore the previous state by never doing newidle when core scheduling is enabled.

---
 kernel/sched/fair.c |    3 +++
 1 file changed, 3 insertions(+)
kernel/sched/fair.c +3 -0
@@ -9942,6 +9942,9 @@ struct task_struct *pick_task_fair(struc
 	return p;
 
 idle:
+	if (sched_core_enabled(rq))
+		return NULL;
+
 	new_tasks = sched_balance_newidle(rq, rf);
 	if (new_tasks < 0)
 		return RETRY_TASK;
#18 lore raw

Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()

Hi Peter,

Quoted 5 lines · Peter Zijlstra

Aaron,

Sorry I failed to notice this email earlier.

Never mind.

Quoted 23 lines · Aaron Lu
Quoted 9 lines

I applied below diff and the problem is gone:

kernel/sched/fair.c +0 -0
index 5f48af700fd44..942a543af3e54 100644

kernel/sched/fair.c static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) · line 9905

Quoted 9 lines
kernel/sched/fair.c +3 -0
@@ -9897,6 +9897,9 @@ static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
 	return p;
 
 idle:
+	if (sched_core_enabled(rq))
+		return NULL;
+
 	new_tasks = sched_balance_newidle(rq, rf);
 	if (new_tasks < 0)
 		return RETRY_TASK;

Right, this is the safe patch and restores pick_task_fair() to its previous status (for core-sched).

Quoted 2 lines

Since people are hitting this problem, I'm going to merge it as below. I've presumed your SoB, please let me know if that's a problem.

No problem.

Quoted 3 lines

I think I'm going to try and move newidle into sched_class::balance / balance_fair(), but I'll do that next cycle.

I'll surely test it then.

Best regards,
Aaron

merge bot Applied
#19 lore raw

[tip: sched/core] sched/fair: Fix newidle vs core-sched

In reply to Aaron Lu
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     c095741713d1bc317b53e2da2b222e7448b6021f
Gitweb:        https://git.kernel.org/tip/c095741713d1bc317b53e2da2b222e7448b6021f
Author:        Aaron Lu <[email protected]>
AuthorDate:    Wed, 03 Jun 2026 17:51:08 +08:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 11 Jun 2026 13:41:23 +02:00

sched/fair: Fix newidle vs core-sched

While testing Prateek's throttle series, I noticed a panic issue when coresched is enabled and bisected to this patch.

I fed the panic log and this patch to an agent and its analysis looks correct to me(cpu56 and cpu57 are siblings in a VM):

       cpu57 (holds core-wide lock)

     pick_next_task() [core scheduling]
     for_each_cpu_wrap(i, smt_mask, 57):
       i=57: pick_task(rq_57)
             pick_task_fair(rq_57)
             -> picks task A
       rq_57->core_pick = task A
       // task_rq(A) == rq_57

       i=56: pick_task(rq_56)
             pick_task_fair(rq_56)
             cfs_rq->nr_queued == 0
             goto idle
             sched_balance_newidle(rq_56)
             raw_spin_rq_unlock(rq_56)
             // core-wide lock released
             newidle_balance() pulls
               task A: rq_57 -> rq_56
             // task_rq(A) == rq_56 now
             raw_spin_rq_lock(rq_56)
             // core-wide lock re-acquired
             return > 0
             goto again
             pick_task_fair(rq_56)
             -> picks task A
       rq_56->core_pick = task A

     // first loop done
     // rq_57->core_pick is still task A (set before lock release)
     // but task_rq(A) == rq_56 now
     next = rq_57->core_pick  // = task A

     put_prev_set_next_task(rq_57, prev, task A)
     __set_next_task_fair(rq_57, task A)
     hrtick_start_fair(rq_57, task A)
     WARN_ON_ONCE(task_rq(task A) != rq_57)
     // task_rq(A) == rq_56

IOW: by allowing pick_task_fair() to do newidle_balance and not returning RETRY_TASK, it can end up selecting the same task on two CPUs. Restore the previous state by never doing newidle when core scheduling is enabled.

---
 kernel/sched/fair.c | 3 +++
 1 file changed, 3 insertions(+)
kernel/sched/fair.c +3 -0
index 1b23e73..d78467e 100644
@@ -9942,6 +9942,9 @@ again:
 	return p;
 
 idle:
+	if (sched_core_enabled(rq))
+		return NULL;
+
 	new_tasks = sched_balance_newidle(rq, rf);
 	if (new_tasks < 0)
 		return RETRY_TASK;
author maintainer
#20 lore raw

PATCH v2 9/10 sched: Remove sched_class::pick_next_task()

In reply to Peter Zijlstra

The reason for pick_next_task_fair() is the put/set optimization that avoids touching the common ancestors. However, it is possible to implement this in the put_prev_task() and set_next_task() calls as used in put_prev_set_next_task().

Notably, put_prev_set_next_task() is the only site that:

- calls put_prev_task() with a .next argument;
- calls set_next_task() with .first = true.

This means that put_prev_task() can determine the common hierarchy and stop there, and then set_next_task() can terminate where put_prev_task stopped.

---
 kernel/sched/core.c  |   27 +++------
 kernel/sched/fair.c  |  139 +++++++++++++++++----------------------------------
 kernel/sched/sched.h |   14 -----
 3 files changed, 57 insertions(+), 123 deletions(-)
kernel/sched/core.c +9 -18
@@ -5980,16 +5980,15 @@ __pick_next_task(struct rq *rq, struct t
 	if (likely(!sched_class_above(prev->sched_class, &fair_sched_class) &&
 		   rq->nr_running == rq->cfs.h_nr_queued)) {
 
-		p = pick_next_task_fair(rq, prev, rf);
+		p = pick_task_fair(rq, rf);
 		if (unlikely(p == RETRY_TASK))
 			goto restart;
 
 		/* Assume the next prioritized class is idle_sched_class */
-		if (!p) {
+		if (!p)
 			p = pick_task_idle(rq, rf);
-			put_prev_set_next_task(rq, prev, p);
-		}
 
+		put_prev_set_next_task(rq, prev, p);
 		return p;
 	}
 
@@ -5997,20 +5996,12 @@ __pick_next_task(struct rq *rq, struct t
 	prev_balance(rq, prev, rf);
 
 	for_each_active_class(class) {
-		if (class->pick_next_task) {
-			p = class->pick_next_task(rq, prev, rf);
-			if (unlikely(p == RETRY_TASK))
-				goto restart;
-			if (p)
-				return p;
-		} else {
-			p = class->pick_task(rq, rf);
-			if (unlikely(p == RETRY_TASK))
-				goto restart;
-			if (p) {
-				put_prev_set_next_task(rq, prev, p);
-				return p;
-			}
+		p = class->pick_task(rq, rf);
+		if (unlikely(p == RETRY_TASK))
+			goto restart;
+		if (p) {
+			put_prev_set_next_task(rq, prev, p);
+			return p;
 		}
 	}
 
kernel/sched/fair.c +47 -92

180 lines not on this page · raw message

kernel/sched/sched.h +1 -13
@@ -2555,17 +2555,6 @@ struct sched_class {
 	 * schedule/pick_next_task: rq->lock
 	 */
 	struct task_struct *(*pick_task)(struct rq *rq, struct rq_flags *rf);
-	/*
-	 * Optional! When implemented pick_next_task() should be equivalent to:
-	 *
-	 *   next = pick_task();
-	 *   if (next) {
-	 *       put_prev_task(prev);
-	 *       set_next_task_first(next);
-	 *   }
-	 */
-	struct task_struct *(*pick_next_task)(struct rq *rq, struct task_struct *prev,
-					      struct rq_flags *rf);
 
 	/*
 	 * sched_change:
@@ -2789,8 +2778,7 @@ static inline bool sched_fair_runnable(s
 	return rq->cfs.nr_queued > 0;
 }
 
-extern struct task_struct *pick_next_task_fair(struct rq *rq, struct task_struct *prev,
-					       struct rq_flags *rf);
+extern struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf);
 extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
 
 #define SCA_CHECK		0x01
maintainer Reviewed-by on v2 9/10
#21 lore raw

Re: [PATCH v2 09/10] sched: Remove sched_class::pick_next_task()

Quoted 8 lines · Peter Zijlstra

The reason for pick_next_task_fair() is the put/set optimization that avoids touching the common ancestors. However, it is possible to implement this in the put_prev_task() and set_next_task() calls as used in put_prev_set_next_task().

Notably, put_prev_set_next_task() is the only site that:

Quoted 8 lines

- calls put_prev_task() with a .next argument;
- calls set_next_task() with .first = true.

This means that put_prev_task() can determine the common hierarchy and stop there, and then set_next_task() can terminate where put_prev_task stopped.

Hackbench results on my Arm64 dev machine stay similars with patch 8 and 9 (unlike patch 10)

Quoted 275 lines
---
 kernel/sched/core.c  |   27 +++------
 kernel/sched/fair.c  |  139 +++++++++++++++++----------------------------------
 kernel/sched/sched.h |   14 -----
 3 files changed, 57 insertions(+), 123 deletions(-)
kernel/sched/core.c +9 -18
@@ -5980,16 +5980,15 @@ __pick_next_task(struct rq *rq, struct t
        if (likely(!sched_class_above(prev->sched_class, &fair_sched_class) &&
                   rq->nr_running == rq->cfs.h_nr_queued)) {
 
-               p = pick_next_task_fair(rq, prev, rf);
+               p = pick_task_fair(rq, rf);
                if (unlikely(p == RETRY_TASK))
                        goto restart;
 
                /* Assume the next prioritized class is idle_sched_class */
-               if (!p) {
+               if (!p)
                        p = pick_task_idle(rq, rf);
-                       put_prev_set_next_task(rq, prev, p);
-               }
 
+               put_prev_set_next_task(rq, prev, p);
                return p;
        }
 
@@ -5997,20 +5996,12 @@ __pick_next_task(struct rq *rq, struct t
        prev_balance(rq, prev, rf);
 
        for_each_active_class(class) {
-               if (class->pick_next_task) {
-                       p = class->pick_next_task(rq, prev, rf);
-                       if (unlikely(p == RETRY_TASK))
-                               goto restart;
-                       if (p)
-                               return p;
-               } else {
-                       p = class->pick_task(rq, rf);
-                       if (unlikely(p == RETRY_TASK))
-                               goto restart;
-                       if (p) {
-                               put_prev_set_next_task(rq, prev, p);
-                               return p;
-                       }
+               p = class->pick_task(rq, rf);
+               if (unlikely(p == RETRY_TASK))
+                       goto restart;
+               if (p) {
+                       put_prev_set_next_task(rq, prev, p);
+                       return p;
                }
        }
 
kernel/sched/fair.c +47 -92
@@ -9214,7 +9214,7 @@ static void wakeup_preempt_fair(struct r
        resched_curr_lazy(rq);
 }
 
-static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
+struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
        __must_hold(__rq_lockp(rq))
 {
        struct sched_entity *se;
@@ -9257,72 +9257,6 @@ static struct task_struct *pick_task_fai
        return NULL;
 }
 
-static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
-static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
-
-struct task_struct *
-pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
-       __must_hold(__rq_lockp(rq))
-{
-       struct sched_entity *se;
-       struct task_struct *p;
-
-       p = pick_task_fair(rq, rf);
-       if (unlikely(p == RETRY_TASK))
-               return p;
-       if (!p)
-               return p;
-       se = &p->se;
-
-#ifdef CONFIG_FAIR_GROUP_SCHED
-       if (prev->sched_class != &fair_sched_class)
-               goto simple;
-
-       __put_prev_set_next_dl_server(rq, prev, p);
-
-       /*
-        * Because of the set_next_buddy() in dequeue_task_fair() it is rather
-        * likely that a next task is from the same cgroup as the current.
-        *
-        * Therefore attempt to avoid putting and setting the entire cgroup
-        * hierarchy, only change the part that actually changes.
-        *
-        * Since we haven't yet done put_prev_entity and if the selected task
-        * is a different task than we started out with, try and touch the
-        * least amount of cfs_rqs.
-        */
-       if (prev != p) {
-               struct sched_entity *pse = &prev->se;
-               struct cfs_rq *cfs_rq;
-
-               while (!(cfs_rq = is_same_group(se, pse))) {
-                       int se_depth = se->depth;
-                       int pse_depth = pse->depth;
-
-                       if (se_depth <= pse_depth) {
-                               put_prev_entity(cfs_rq_of(pse), pse);
-                               pse = parent_entity(pse);
-                       }
-                       if (se_depth >= pse_depth) {
-                               set_next_entity(cfs_rq_of(se), se, true);
-                               se = parent_entity(se);
-                       }
-               }
-
-               put_prev_entity(cfs_rq, pse);
-               set_next_entity(cfs_rq, se, true);
-
-               __set_next_task_fair(rq, p, true);
-       }
-
-       return p;
-
-simple:
-#endif /* CONFIG_FAIR_GROUP_SCHED */
-       put_prev_set_next_task(rq, prev, p);
-       return p;
-}
-
 static struct task_struct *
 fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
        __must_hold(__rq_lockp(dl_se->rq))
@@ -9346,10 +9280,33 @@ static void put_prev_task_fair(struct rq
 {
        struct sched_entity *se = &prev->se;
        struct cfs_rq *cfs_rq;
+       struct sched_entity *nse = NULL;
 
-       for_each_sched_entity(se) {
+#ifdef CONFIG_FAIR_GROUP_SCHED
+       if (next && next->sched_class == &fair_sched_class)
+               nse = &next->se;
+#endif
+
+       while (se) {
                cfs_rq = cfs_rq_of(se);
-               put_prev_entity(cfs_rq, se);
+               if (!nse || cfs_rq->curr)
+                       put_prev_entity(cfs_rq, se);
+#ifdef CONFIG_FAIR_GROUP_SCHED
+               if (nse) {
+                       if (is_same_group(se, nse))
+                               break;
+
+                       int d = nse->depth - se->depth;
+                       if (d >= 0) {
+                               /* nse has equal or greater depth, ascend */
+                               nse = parent_entity(nse);
+                               /* if nse is the deeper, do not ascend se */
+                               if (d > 0)
+                                       continue;
+                       }
+               }
+#endif
+               se = parent_entity(se);
        }
 }
 
@@ -13896,10 +13853,30 @@ static void switched_to_fair(struct rq *
        }
 }
 
-static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
+/*
+ * Account for a task changing its policy or group.
+ *
+ * This routine is mostly called to set cfs_rq->curr field when a task
+ * migrates between groups/classes.
+ */
+static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 {
        struct sched_entity *se = &p->se;
 
+       for_each_sched_entity(se) {
+               struct cfs_rq *cfs_rq = cfs_rq_of(se);
+
+               if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
+                   first && cfs_rq->curr)
+                       break;
+
+               set_next_entity(cfs_rq, se, first);
+               /* ensure bandwidth has been allocated on our new cfs_rq */
+               account_cfs_rq_runtime(cfs_rq, 0);
+       }
+
+       se = &p->se;
+
        if (task_on_rq_queued(p)) {
                /*
                 * Move the next running task to the front of the list, so our
@@ -13919,27 +13896,6 @@ static void __set_next_task_fair(struct
        sched_fair_update_stop_tick(rq, p);
 }
 
-/*
- * Account for a task changing its policy or group.
- *
- * This routine is mostly called to set cfs_rq->curr field when a task
- * migrates between groups/classes.
- */
-static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
-{
-       struct sched_entity *se = &p->se;
-
-       for_each_sched_entity(se) {
-               struct cfs_rq *cfs_rq = cfs_rq_of(se);
-
-               set_next_entity(cfs_rq, se, first);
-               /* ensure bandwidth has been allocated on our new cfs_rq */
-               account_cfs_rq_runtime(cfs_rq, 0);
-       }
-
-       __set_next_task_fair(rq, p, first);
-}
-
 void init_cfs_rq(struct cfs_rq *cfs_rq)
 {
        cfs_rq->tasks_timeline = RB_ROOT_CACHED;
@@ -14251,7 +14207,6 @@ DEFINE_SCHED_CLASS(fair) = {
        .wakeup_preempt         = wakeup_preempt_fair,
 
        .pick_task              = pick_task_fair,
-       .pick_next_task         = pick_next_task_fair,
        .put_prev_task          = put_prev_task_fair,
        .set_next_task          = set_next_task_fair,
 
kernel/sched/sched.h +1 -13
@@ -2555,17 +2555,6 @@ struct sched_class {
         * schedule/pick_next_task: rq->lock
         */
        struct task_struct *(*pick_task)(struct rq *rq, struct rq_flags *rf);
-       /*
-        * Optional! When implemented pick_next_task() should be equivalent to:
-        *
-        *   next = pick_task();
-        *   if (next) {
-        *       put_prev_task(prev);
-        *       set_next_task_first(next);
-        *   }
-        */
-       struct task_struct *(*pick_next_task)(struct rq *rq, struct task_struct *prev,
-                                             struct rq_flags *rf);
 
        /*
         * sched_change:
@@ -2789,8 +2778,7 @@ static inline bool sched_fair_runnable(s
        return rq->cfs.nr_queued > 0;
 }
 
-extern struct task_struct *pick_next_task_fair(struct rq *rq, struct task_struct *prev,
-                                              struct rq_flags *rf);
+extern struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf);
 extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
 
 #define SCA_CHECK              0x01
merge bot Applied
#22 lore raw

[tip: sched/core] sched: Remove sched_class::pick_next_task()

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     5ad278dd20bdf59714443894d7b3044471af97d0
Gitweb:        https://git.kernel.org/tip/5ad278dd20bdf59714443894d7b3044471af97d0
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 11 May 2026 13:31:13 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Tue, 26 May 2026 13:53:14 +02:00

sched: Remove sched_class::pick_next_task()

The reason for pick_next_task_fair() is the put/set optimization that avoids touching the common ancestors. However, it is possible to implement this in the put_prev_task() and set_next_task() calls as used in put_prev_set_next_task().

Notably, put_prev_set_next_task() is the only site that:

- calls put_prev_task() with a .next argument;
- calls set_next_task() with .first = true.

This means that put_prev_task() can determine the common hierarchy and stop there, and then set_next_task() can terminate where put_prev_task stopped.

---
 kernel/sched/core.c  |  27 ++------
 kernel/sched/fair.c  | 139 ++++++++++++++----------------------------
 kernel/sched/sched.h |  14 +----
 3 files changed, 57 insertions(+), 123 deletions(-)
kernel/sched/core.c +9 -18
index 83202f0..3c8bfd6 100644
@@ -6030,16 +6030,15 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	if (likely(!sched_class_above(prev->sched_class, &fair_sched_class) &&
 		   rq->nr_running == rq->cfs.h_nr_queued)) {
 
-		p = pick_next_task_fair(rq, prev, rf);
+		p = pick_task_fair(rq, rf);
 		if (unlikely(p == RETRY_TASK))
 			goto restart;
 
 		/* Assume the next prioritized class is idle_sched_class */
-		if (!p) {
+		if (!p)
 			p = pick_task_idle(rq, rf);
-			put_prev_set_next_task(rq, prev, p);
-		}
 
+		put_prev_set_next_task(rq, prev, p);
 		return p;
 	}
 
@@ -6047,20 +6046,12 @@ restart:
 	prev_balance(rq, prev, rf);
 
 	for_each_active_class(class) {
-		if (class->pick_next_task) {
-			p = class->pick_next_task(rq, prev, rf);
-			if (unlikely(p == RETRY_TASK))
-				goto restart;
-			if (p)
-				return p;
-		} else {
-			p = class->pick_task(rq, rf);
-			if (unlikely(p == RETRY_TASK))
-				goto restart;
-			if (p) {
-				put_prev_set_next_task(rq, prev, p);
-				return p;
-			}
+		p = class->pick_task(rq, rf);
+		if (unlikely(p == RETRY_TASK))
+			goto restart;
+		if (p) {
+			put_prev_set_next_task(rq, prev, p);
+			return p;
 		}
 	}
 
kernel/sched/fair.c +47 -92

180 lines not on this page · raw message

kernel/sched/sched.h +1 -13
index 8eb8f83..6b48bb3 100644
@@ -2589,17 +2589,6 @@ struct sched_class {
 	 * schedule/pick_next_task: rq->lock
 	 */
 	struct task_struct *(*pick_task)(struct rq *rq, struct rq_flags *rf);
-	/*
-	 * Optional! When implemented pick_next_task() should be equivalent to:
-	 *
-	 *   next = pick_task();
-	 *   if (next) {
-	 *       put_prev_task(prev);
-	 *       set_next_task_first(next);
-	 *   }
-	 */
-	struct task_struct *(*pick_next_task)(struct rq *rq, struct task_struct *prev,
-					      struct rq_flags *rf);
 
 	/*
 	 * sched_change:
@@ -2823,8 +2812,7 @@ static inline bool sched_fair_runnable(struct rq *rq)
 	return rq->cfs.nr_queued > 0;
 }
 
-extern struct task_struct *pick_next_task_fair(struct rq *rq, struct task_struct *prev,
-					       struct rq_flags *rf);
+extern struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf);
 extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
 
 #define SCA_CHECK		0x01
author maintainer
#23 lore raw

PATCH v2 10/10 sched/eevdf: Move to a single runqueue

In reply to Peter Zijlstra

Change fair/cgroup to a single runqueue.

Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.

The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the shares_weight approximation are fully preserved.

That is, given a hierarchy like:

        R
        |
        se--G1
            / \
      G2--se   se--G3
     / \           |

T1--se se--T2 se--T3

This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:

     _R_
    / | \

T1 T2 T3

Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.

Notably, the effective weight (se->h_load) is computed from the hierarchical fraction: se->load / cfs_rq->load.

Since EEVDF is now exclusive operating on rq->cfs, it needs to consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.

One place where additional information was required was set_next_task() / put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).

As a result of only having a single level to pick from, much of the complications in pick_next_task() and preemption go away.

Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.

TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr

---
 include/linux/sched.h |    1 
 kernel/sched/core.c   |    5 
 kernel/sched/debug.c  |    9 
 kernel/sched/fair.c   |  789 +++++++++++++++++++++-----------------------------
 kernel/sched/pelt.c   |    6 
 kernel/sched/sched.h  |   26 -
 6 files changed, 366 insertions(+), 470 deletions(-)
include/linux/sched.h +1 -0
@@ -575,6 +575,7 @@ struct sched_statistics {
 struct sched_entity {
 	/* For load-balancing: */
 	struct load_weight		load;
+	struct load_weight		h_load;
 	struct rb_node			run_node;
 	u64				deadline;
 	u64				min_vruntime;
kernel/sched/core.c +1 -4
@@ -5539,11 +5539,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);
  */
 static inline void prefetch_curr_exec_start(struct task_struct *p)
 {
-#ifdef CONFIG_FAIR_GROUP_SCHED
-	struct sched_entity *curr = p->se.cfs_rq->curr;
-#else
 	struct sched_entity *curr = task_rq(p)->cfs.curr;
-#endif
+
 	prefetch(curr);
 	prefetch(&curr->exec_start);
 }
kernel/sched/debug.c +6 -3
@@ -911,10 +911,11 @@ print_task(struct seq_file *m, struct rq
 	else
 		SEQ_printf(m, " %c", task_state_to_char(p));
 
-	SEQ_printf(m, " %15s %5d %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
+	SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
 		p->comm, task_pid_nr(p),
+		p->se.h_load.weight,
 		SPLIT_NS(p->se.vruntime),
-		entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
+		entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N',
 		SPLIT_NS(p->se.deadline),
 		p->se.custom_slice ? 'S' : ' ',
 		SPLIT_NS(p->se.slice),
@@ -943,7 +944,7 @@ static void print_rq(struct seq_file *m,
 
 	SEQ_printf(m, "\n");
 	SEQ_printf(m, "runnable tasks:\n");
-	SEQ_printf(m, " S            task   PID       vruntime   eligible    "
+	SEQ_printf(m, " S            task   PID     weight       vruntime   eligible    "
 		   "deadline             slice          sum-exec      switches  "
 		   "prio         wait-time        sum-sleep       sum-block"
 #ifdef CONFIG_NUMA_BALANCING
@@ -1051,6 +1052,8 @@ void print_cfs_rq(struct seq_file *m, in
 			cfs_rq->tg_load_avg_contrib);
 	SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_avg",
 			atomic_long_read(&cfs_rq->tg->load_avg));
+	SEQ_printf(m, "  .%-30s: %lu\n", "h_load",
+			cfs_rq->h_load);
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 #ifdef CONFIG_CFS_BANDWIDTH
 	SEQ_printf(m, "  .%-30s: %d\n", "throttled",
kernel/sched/fair.c +342 -447

1,399 lines not on this page · raw message

kernel/sched/pelt.c +3 -3
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched
 	/*
 	 * running is a subset of runnable (weight) so running can't be set if
 	 * runnable is clear. But there are some corner cases where the current
-	 * se has been already dequeued but cfs_rq->curr still points to it.
+	 * se has been already dequeued but cfs_rq->h_curr still points to it.
 	 * This means that weight will be 0 but not running for a sched_entity
 	 * but also for a cfs_rq if the latter becomes idle. As an example,
 	 * this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now
 int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
 	if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
-				cfs_rq->curr == se)) {
+				cfs_rq->h_curr == se)) {
 
 		___update_load_avg(&se->avg, se_weight(se));
 		cfs_se_util_change(&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, st
 	if (___update_load_sum(now, &cfs_rq->avg,
 				scale_load_down(cfs_rq->load.weight),
 				cfs_rq->h_nr_runnable,
-				cfs_rq->curr != NULL)) {
+				cfs_rq->h_curr != NULL)) {
 
 		___update_load_avg(&cfs_rq->avg, 1);
 		trace_pelt_cfs_tp(cfs_rq);
kernel/sched/sched.h +13 -13
@@ -528,21 +528,8 @@ struct task_group {
 
 };
 
-#ifdef CONFIG_GROUP_SCHED_WEIGHT
 #define ROOT_TASK_GROUP_LOAD	NICE_0_LOAD
 
-/*
- * A weight of 0 or 1 can cause arithmetics problems.
- * A weight of a cfs_rq is the sum of weights of which entities
- * are queued on this cfs_rq, so a weight of a entity should not be
- * too large, so as the shares value of a task group.
- * (The default weight is 1024 - so there's no practical
- *  limitation from this.)
- */
-#define MIN_SHARES		(1UL <<  1)
-#define MAX_SHARES		(1UL << 18)
-#endif
-
 typedef int (*tg_visitor)(struct task_group *, void *);
 
 extern int walk_tg_tree_from(struct task_group *from,
@@ -629,6 +616,17 @@ static inline bool cfs_task_bw_constrain
 
 #endif /* !CONFIG_CGROUP_SCHED */
 
+/*
+ * A weight of 0 or 1 can cause arithmetics problems.
+ * A weight of a cfs_rq is the sum of weights of which entities
+ * are queued on this cfs_rq, so a weight of a entity should not be
+ * too large, so as the shares value of a task group.
+ * (The default weight is 1024 - so there's no practical
+ *  limitation from this.)
+ */
+#define MIN_SHARES		(1UL <<  1)
+#define MAX_SHARES		(1UL << 18)
+
 extern void unregister_rt_sched_group(struct task_group *tg);
 extern void free_rt_sched_group(struct task_group *tg);
 extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
@@ -707,6 +705,7 @@ struct cfs_rq {
 	/*
 	 * CFS load tracking
 	 */
+	struct sched_entity	*h_curr;
 	struct sched_avg	avg;
 #ifndef CONFIG_64BIT
 	u64			last_update_time_copy;
@@ -2509,6 +2508,7 @@ extern const u32		sched_prio_to_wmult[40
 #define ENQUEUE_MIGRATED	0x00040000
 #define ENQUEUE_INITIAL		0x00080000
 #define ENQUEUE_RQ_SELECTED	0x00100000
+#define ENQUEUE_QUEUED		0x00200000
 
 #define RETRY_TASK		((void *)-1UL)
 
#24 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Hello Peter,

Review static void wakeup_preempt_fair(struct r · line 9212

Quoted 9 lines · Peter Zijlstra
(quoted diff) +4 -1
@@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r
+	se = pick_next_entity(rq, true);
+	if (!se)
+		goto again;
 
 	p = task_of(se);
-	if (unlikely(throttled))
+	if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
 		task_throttle_setup_work(p);

I think this bit should also be replicated in set_next_task() after account_cfs_rq_runtime() since any part of the hierarchy may get throttled as a result of failing to grab runtime.

Also check_cfs_rq_runtime() only sees if the cfs_rq is throttled but the task can fail to run if it is on a throttled_hierarchy() too so that should be the correct check here.

Something like below (only build tested on queue/sched/flat):

kernel/sched/fair.c +19 -1
index e54da4c6c945..950c072244b2 100644
@@ -9224,7 +9224,19 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
 		goto again;
 
 	p = task_of(se);
-	if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
+	/*
+	 * For cases where prev is picked again after
+	 * being throttled, entity_tick() would have
+	 * already marked its hierarchy as throttled.
+	 *
+	 * Add throttle work here since
+	 * put_prev_set_next_task() is skipped on
+	 * same task's selection.
+	 *
+	 * For other case, set_next_task_fair() will
+	 * handle adding the throttle work.
+	 */
+	if (throttled_hierarchy(cfs_rq_of(se)))
 		task_throttle_setup_work(p);
 	return p;
 
@@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 		if (on_rq)
 			weight = __calc_prop_weight(cfs_rq, se, weight);
 	}
+	/*
+	 * Add throttle work if the bandwidth allocation above failed
+	 * to grab any runtime and throttled the task's hierarchy.
+	 */
+	if (throttled_hierarchy(task_cfs_rq(p)))
+		task_throttle_setup_work(p);
 
 	se = &p->se;
 	cfs_rq->curr = se;

---

Quoted 2 lines
(quoted diff) +0 -0
 	return p;
 
Signature · 2 lines
Thanks and Regards,
Prateek
author maintainer
#25 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 36 lines · K Prateek Nayak

Hello Peter,

Review static void wakeup_preempt_fair(struct r · line 9212

Quoted 9 lines · Peter Zijlstra
(quoted diff) +4 -1
@@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r
+	se = pick_next_entity(rq, true);
+	if (!se)
+		goto again;
 
 	p = task_of(se);
-	if (unlikely(throttled))
+	if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
 		task_throttle_setup_work(p);

I think this bit should also be replicated in set_next_task() after account_cfs_rq_runtime() since any part of the hierarchy may get throttled as a result of failing to grab runtime.

Also check_cfs_rq_runtime() only sees if the cfs_rq is throttled but the task can fail to run if it is on a throttled_hierarchy() too so that should be the correct check here.

Something like below (only build tested on queue/sched/flat):

kernel/sched/fair.c +4 -1
index e54da4c6c945..950c072244b2 100644
@@ -9224,7 +9224,19 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
 		goto again;
 
 	p = task_of(se);
-	if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
+	/*
+	 * For cases where prev is picked again after
+	 * being throttled, entity_tick() would have
+	 * already marked its hierarchy as throttled.

kernel/sched/fair.c struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf) · line 9240

Quoted 10 lines
kernel/sched/fair.c +9 -0
+	 *
+	 * Add throttle work here since
+	 * put_prev_set_next_task() is skipped on
+	 * same task's selection.
+	 *
+	 * For other case, set_next_task_fair() will
+	 * handle adding the throttle work.
+	 */
+	if (throttled_hierarchy(cfs_rq_of(se)))
 		task_throttle_setup_work(p);

Ah, right, because we've not accumulated runtime, it doesn't make sense to use check_cfs_rq_runtime() at pick time, all we need to do is check if the task should be throttled.

However, since set_next_task_fair() will walk the entire hierarchy anyway, we can remove it here entirely and fully rely on that.

kernel/sched/fair.c static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) · line 13839

Quoted 12 lines
kernel/sched/fair.c +6 -0
 	return p;
 
@@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 		if (on_rq)
 			weight = __calc_prop_weight(cfs_rq, se, weight);
 	}
+	/*
+	 * Add throttle work if the bandwidth allocation above failed
+	 * to grab any runtime and throttled the task's hierarchy.
+	 */
+	if (throttled_hierarchy(task_cfs_rq(p)))
+		task_throttle_setup_work(p);

We already call into account_cfs_rq_runtime(); which basically does all we need.

I think the distinction between account_cfs_rq_runtime() and check_cfs_rq_runtime() no longer makes sense. We can throttle a cfs_rq at any point now, since we no longer remove the cfs_rq, but rather we make the tasks suspend themselves until the cfs_rq naturally dequeues for being empty.

Something like so perhaps?

---

kernel/sched/fair.c +38 -65
@@ -488,7 +488,7 @@ static int se_is_idle(struct sched_entit
 #endif /* !CONFIG_FAIR_GROUP_SCHED */
 
 static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
 
 /**************************************************************
  * Scheduling class tree data structure manipulation methods:
@@ -1420,12 +1420,22 @@ static void update_curr(struct cfs_rq *c
 	}
 }
 
+static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
+static inline void task_throttle_setup_work(struct task_struct *p);
+
 static void update_curr_fair(struct rq *rq)
 {
 	struct sched_entity *se = &rq->donor->se;
+	bool throttled = false;
 
-	for_each_sched_entity(se)
-		update_curr(cfs_rq_of(se));
+	for_each_sched_entity(se) {
+		struct cfs_rq *cfs_rq = cfs_rq_of(se);
+		update_curr(cfs_rq);
+		throttled |= cfs_rq_throttled(cfs_rq);
+	}
+
+	if (throttled)
+		task_throttle_setup_work(rq->donor);
 }
 
 static inline void
@@ -5627,7 +5637,6 @@ place_entity(struct cfs_rq *cfs_rq, stru
 }
 
 static void check_enqueue_throttle(struct cfs_rq *cfs_rq);
-static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
 
 static void
 enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
@@ -5830,8 +5839,6 @@ pick_next_entity(struct rq *rq, bool pro
 	return se;
 }
 
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq);
-
 static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
 {
 	/*
@@ -5841,9 +5848,6 @@ static void put_prev_entity(struct cfs_r
 	if (prev->on_rq)
 		update_curr(cfs_rq);
 
-	/* throttle cfs_rqs exceeding runtime */
-	check_cfs_rq_runtime(cfs_rq);
-
 	if (prev->on_rq) {
 		update_stats_wait_start_fair(cfs_rq, prev);
 		/* in !on_rq case, update occurred at dequeue */
@@ -5976,44 +5980,29 @@ static int __assign_cfs_rq_runtime(struc
 	return cfs_rq->runtime_remaining > 0;
 }
 
-/* returns 0 on failure to allocate runtime */
-static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
-	struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
-	int ret;
-
-	raw_spin_lock(&cfs_b->lock);
-	ret = __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice());
-	raw_spin_unlock(&cfs_b->lock);
-
-	return ret;
-}
+static bool throttle_cfs_rq(struct cfs_rq *cfs_rq);
 
-static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+static bool __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
 {
 	/* dock delta_exec before expiring quota (as it could span periods) */
 	cfs_rq->runtime_remaining -= delta_exec;
 
 	if (likely(cfs_rq->runtime_remaining > 0))
-		return;
+		return false;
 
 	if (cfs_rq->throttled)
-		return;
-	/*
-	 * if we're unable to extend our runtime we resched so that the active
-	 * hierarchy can be throttled
-	 */
-	if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->h_curr))
-		resched_curr(rq_of(cfs_rq));
+		return true;
+
+	return throttle_cfs_rq(cfs_rq);
 }
 
 static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
 {
 	if (!cfs_bandwidth_used() || !cfs_rq->runtime_enabled)
-		return;
+		return false;
 
-	__account_cfs_rq_runtime(cfs_rq, delta_exec);
+	return __account_cfs_rq_runtime(cfs_rq, delta_exec);
 }
 
 static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq)
@@ -6284,9 +6273,9 @@ static bool throttle_cfs_rq(struct cfs_r
 		 * We have raced with bandwidth becoming available, and if we
 		 * actually throttled the timer might not unthrottle us for an
 		 * entire period. We additionally needed to make sure that any
-		 * subsequent check_cfs_rq_runtime calls agree not to throttle
-		 * us, as we may commit to do cfs put_prev+pick_next, so we ask
-		 * for 1ns of runtime rather than just check cfs_b.
+		 * subsequent account_cfs_rq_runtime() calls agree not to
+		 * throttle us, as we may commit to do cfs put_prev+pick_next,
+		 * so we ask for 1ns of runtime rather than just check cfs_b.
 		 */
 		dequeue = 0;
 	} else {
@@ -6711,8 +6700,6 @@ static void check_enqueue_throttle(struc
 
 	/* update runtime allocation */
 	account_cfs_rq_runtime(cfs_rq, 0);
-	if (cfs_rq->runtime_remaining <= 0)
-		throttle_cfs_rq(cfs_rq);
 }
 
 static void sync_throttle(struct task_group *tg, int cpu)
@@ -6742,25 +6729,6 @@ static void sync_throttle(struct task_gr
 		cfs_rq->pelt_clock_throttled = 1;
 }
 
-/* conditionally throttle active cfs_rq's from put_prev_entity() */
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
-	if (!cfs_bandwidth_used())
-		return false;
-
-	if (likely(!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining > 0))
-		return false;
-
-	/*
-	 * it's possible for a throttled entity to be forced into a running
-	 * state (e.g. set_curr_task), in this case we're finished.
-	 */
-	if (cfs_rq_throttled(cfs_rq))
-		return true;
-
-	return throttle_cfs_rq(cfs_rq);
-}
-
 static enum hrtimer_restart sched_cfs_slack_timer(struct hrtimer *timer)
 {
 	struct cfs_bandwidth *cfs_b =
@@ -7015,8 +6983,7 @@ static void sched_fair_update_stop_tick(
 
 #else /* !CONFIG_CFS_BANDWIDTH: */
 
-static void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) {}
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq) { return false; }
+static bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { return false; }
 static void check_enqueue_throttle(struct cfs_rq *cfs_rq) {}
 static inline void sync_throttle(struct task_group *tg, int cpu) {}
 static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) {}
@@ -9208,7 +9175,6 @@ struct task_struct *pick_task_fair(struc
 {
 	struct cfs_rq *cfs_rq = &rq->cfs;
 	struct sched_entity *se;
-	struct task_struct *p;
 	int new_tasks;
 
 again:
@@ -9223,10 +9189,7 @@ struct task_struct *pick_task_fair(struc
 	if (!se)
 		goto again;
 
-	p = task_of(se);
-	if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
-		task_throttle_setup_work(p);
-	return p;
+	return task_of(se);
 
 idle:
 	new_tasks = sched_balance_newidle(rq, rf);
@@ -13618,6 +13581,7 @@ static void task_tick_fair(struct rq *rq
 {
 	struct sched_entity *se = &curr->se;
 	unsigned long weight = NICE_0_LOAD;
+	bool throttled = false;
 	struct cfs_rq *cfs_rq;
 
 	for_each_sched_entity(se) {
@@ -13625,8 +13589,13 @@ static void task_tick_fair(struct rq *rq
 		entity_tick(cfs_rq, se, queued);
 
 		weight = __calc_prop_weight(cfs_rq, se, weight);
+
+		throttled |= cfs_rq_throttled(cfs_rq);
 	}
 
+	if (throttled)
+		task_throttle_setup_work(curr);
+
 	se = &curr->se;
 	reweight_eevdf(cfs_rq, se, weight, se->on_rq);
 
@@ -13800,6 +13769,7 @@ static void set_next_task_fair(struct rq
 	struct cfs_rq *cfs_rq = &rq->cfs;
 	unsigned long weight = NICE_0_LOAD;
 	bool on_rq = se->on_rq;
+	bool throttled = false;
 
 	clear_buddies(cfs_rq, se);
 
@@ -13814,12 +13784,15 @@ static void set_next_task_fair(struct rq
 			set_next_entity(cfs_rq, se);
 
 		/* ensure bandwidth has been allocated on our new cfs_rq */
-		account_cfs_rq_runtime(cfs_rq, 0);
+		throttled |= account_cfs_rq_runtime(cfs_rq, 0);
 
 		if (on_rq)
 			weight = __calc_prop_weight(cfs_rq, se, weight);
 	}
 
+	if (throttled)
+		task_throttle_setup_work(p);
+
 	se = &p->se;
 	cfs_rq->curr = se;
 
#26 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Hello Peter,

Quoted 14 lines · Peter Zijlstra

Review static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) · line 13839

Quoted 10 lines
(quoted diff) +6 -0
@@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 		if (on_rq)
 			weight = __calc_prop_weight(cfs_rq, se, weight);
 	}
+	/*
+	 * Add throttle work if the bandwidth allocation above failed
+	 * to grab any runtime and throttled the task's hierarchy.
+	 */
+	if (throttled_hierarchy(task_cfs_rq(p)))
+		task_throttle_setup_work(p);

We already call into account_cfs_rq_runtime(); which basically does all we need.

Quoted 7 lines

I think the distinction between account_cfs_rq_runtime() and check_cfs_rq_runtime() no longer makes sense. We can throttle a cfs_rq at any point now, since we no longer remove the cfs_rq, but rather we make the tasks suspend themselves until the cfs_rq naturally dequeues for being empty.

Something like so perhaps?

That makes sense! The task should naturally execute the task work when exiting out of the kernel / IRQ handler into the userspace so we should be good.

I'll rebase the below diff on tip, test it a bit, add a commit log, and send it your way if you don't mind or would you like to keep it with the flat_cg bits?

Signature · 2 lines
Thanks and Regards,
Prateek
author maintainer
#27 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 25 lines · K Prateek Nayak

Hello Peter,

Quoted 14 lines · Peter Zijlstra

Review static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) · line 13839

Quoted 10 lines
(quoted diff) +6 -0
@@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 		if (on_rq)
 			weight = __calc_prop_weight(cfs_rq, se, weight);
 	}
+	/*
+	 * Add throttle work if the bandwidth allocation above failed
+	 * to grab any runtime and throttled the task's hierarchy.
+	 */
+	if (throttled_hierarchy(task_cfs_rq(p)))
+		task_throttle_setup_work(p);

We already call into account_cfs_rq_runtime(); which basically does all we need.

Quoted 7 lines

I think the distinction between account_cfs_rq_runtime() and check_cfs_rq_runtime() no longer makes sense. We can throttle a cfs_rq at any point now, since we no longer remove the cfs_rq, but rather we make the tasks suspend themselves until the cfs_rq naturally dequeues for being empty.

Something like so perhaps?

Quoted 7 lines

That makes sense! The task should naturally execute the task work when exiting out of the kernel / IRQ handler into the userspace so we should be good.

I'll rebase the below diff on tip, test it a bit, add a commit log, and send it your way if you don't mind or would you like to keep it with the flat_cg bits?

Nah, this seems like something that can be done independent. And thus is should be. That flat patch is big enough as is.

#28 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

In reply to Peter Zijlstra
Quoted 49 lines · Peter Zijlstra

Change fair/cgroup to a single runqueue.

Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.

The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the shares_weight approximation are fully preserved.

That is, given a hierarchy like:

        R
        |
        se--G1
            / \
      G2--se   se--G3
     / \           |

T1--se se--T2 se--T3

This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:

     _R_
    / | \

T1 T2 T3

Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.

Notably, the effective weight (se->h_load) is computed from the hierarchical fraction: se->load / cfs_rq->load.

Since EEVDF is now exclusive operating on rq->cfs, it needs to consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.

One place where additional information was required was set_next_task() / put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).

As a result of only having a single level to pick from, much of the complications in pick_next_task() and preemption go away.

Quoted 8 lines

Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.

TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr

I know Vincent was having some perf troubles with this patch, but booting on a 64 vCPU qemu environment, I'm seeing:

[    5.688490] Oops: divide error: 0000 [#1] SMP NOPTI
[    5.689457] CPU: 47 UID: 0 PID: 0 Comm: swapper/47 Not tainted
7.1.0-rc2-00026-g82a8ec6fb3f9 #38 PREEMPT(full)
[    5.689457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.17.0-debian-1.17.0-1 04/01/2014
[    5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430
[    5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9
90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0
14 31 9
[    5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046
[    5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80
[    5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00
[    5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[    5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00
[    5.689457] FS:  0000000000000000(0000) GS:ffff888235c2e000(0000)
knlGS:0000000000000000
[    5.689457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0
[    5.689457] Call Trace:
[    5.689457]  <TASK>
[    5.689457]  wakeup_preempt+0xa8/0xd0
[    5.689457]  attach_one_task+0xec/0x150
[    5.689457]  __schedule+0x1ad8/0x21c0
[    5.689457]  schedule_idle+0x22/0x40
[    5.689457]  cpu_startup_entry+0x29/0x30
[    5.689457]  start_secondary+0xf7/0x100
[    5.689457]  common_startup_64+0x13e/0x148
[    5.689457]  </TASK>
[    5.689457] Dumping ftrace buffer:
[    5.689457]    (ftrace buffer empty)
[    5.689457] ---[ end trace 0000000000000000 ]---
[    5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430
[    5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9
90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0
14 31 9
[    5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046
[    5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80
[    5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00
[    5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[    5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00
[    5.689457] FS:  0000000000000000(0000) GS:ffff888235c2e000(0000)
knlGS:0000000000000000
[    5.689457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0
[    5.689457] Kernel panic - not syncing: Fatal exception

Which I bisected down to this last patch in the series.

faddr2line gave me:
__calc_delta at kernel/sched/fair.c:290
(inlined by) calc_delta_fair at kernel/sched/fair.c:300 (inlined by) update_protect_slice at kernel/sched/fair.c:1070 (inlined by) wakeup_preempt_fair at kernel/sched/fair.c:9193

This usually trips as the ww_mutex selftest starts at bootup.

Unfortunately I still see it with the add-on changes you proposed to K Prateek's feedback here.

I'll try to narrow it down further tomorrow.

thanks
-john

#29 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 5 lines · John Stultz
Quoted 3 lines · Peter Zijlstra

Change fair/cgroup to a single runqueue.

...

Quoted 60 lines

I know Vincent was having some perf troubles with this patch, but booting on a 64 vCPU qemu environment, I'm seeing:

[    5.688490] Oops: divide error: 0000 [#1] SMP NOPTI
[    5.689457] CPU: 47 UID: 0 PID: 0 Comm: swapper/47 Not tainted
7.1.0-rc2-00026-g82a8ec6fb3f9 #38 PREEMPT(full)
[    5.689457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.17.0-debian-1.17.0-1 04/01/2014
[    5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430
[    5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9
90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0
14 31 9
[    5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046
[    5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80
[    5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00
[    5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[    5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00
[    5.689457] FS:  0000000000000000(0000) GS:ffff888235c2e000(0000)
knlGS:0000000000000000
[    5.689457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0
[    5.689457] Call Trace:
[    5.689457]  <TASK>
[    5.689457]  wakeup_preempt+0xa8/0xd0
[    5.689457]  attach_one_task+0xec/0x150
[    5.689457]  __schedule+0x1ad8/0x21c0
[    5.689457]  schedule_idle+0x22/0x40
[    5.689457]  cpu_startup_entry+0x29/0x30
[    5.689457]  start_secondary+0xf7/0x100
[    5.689457]  common_startup_64+0x13e/0x148
[    5.689457]  </TASK>
[    5.689457] Dumping ftrace buffer:
[    5.689457]    (ftrace buffer empty)
[    5.689457] ---[ end trace 0000000000000000 ]---
[    5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430
[    5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9
90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0
14 31 9
[    5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046
[    5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80
[    5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00
[    5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[    5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00
[    5.689457] FS:  0000000000000000(0000) GS:ffff888235c2e000(0000)
knlGS:0000000000000000
[    5.689457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0
[    5.689457] Kernel panic - not syncing: Fatal exception

Which I bisected down to this last patch in the series.

faddr2line gave me:
__calc_delta at kernel/sched/fair.c:290
(inlined by) calc_delta_fair at kernel/sched/fair.c:300 (inlined by) update_protect_slice at kernel/sched/fair.c:1070 (inlined by) wakeup_preempt_fair at kernel/sched/fair.c:9193

Quoted 6 lines

This usually trips as the ww_mutex selftest starts at bootup.

Unfortunately I still see it with the add-on changes you proposed to K Prateek's feedback here.

I'll try to narrow it down further tomorrow.

As karma would have it, this does seem to depend on CONFIG_SCHED_PROXY_EXEC. :) I'm guessing the switch in calc_delta_fair() to use se->h_load is uncovering something proxy isn't handling properly with that value.

But I'll have more tomorrow.

thanks
-john

#30 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 69 lines · John Stultz
Quoted 5 lines · John Stultz
Quoted 3 lines · Peter Zijlstra

Change fair/cgroup to a single runqueue.

...

Quoted 62 lines

I know Vincent was having some perf troubles with this patch, but booting on a 64 vCPU qemu environment, I'm seeing:

[    5.688490] Oops: divide error: 0000 [#1] SMP NOPTI
[    5.689457] CPU: 47 UID: 0 PID: 0 Comm: swapper/47 Not tainted
7.1.0-rc2-00026-g82a8ec6fb3f9 #38 PREEMPT(full)
[    5.689457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.17.0-debian-1.17.0-1 04/01/2014
[    5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430
[    5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9
90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0
14 31 9
[    5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046
[    5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80
[    5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00
[    5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[    5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00
[    5.689457] FS:  0000000000000000(0000) GS:ffff888235c2e000(0000)
knlGS:0000000000000000
[    5.689457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0
[    5.689457] Call Trace:
[    5.689457]  <TASK>
[    5.689457]  wakeup_preempt+0xa8/0xd0
[    5.689457]  attach_one_task+0xec/0x150
[    5.689457]  __schedule+0x1ad8/0x21c0
[    5.689457]  schedule_idle+0x22/0x40
[    5.689457]  cpu_startup_entry+0x29/0x30
[    5.689457]  start_secondary+0xf7/0x100
[    5.689457]  common_startup_64+0x13e/0x148
[    5.689457]  </TASK>
[    5.689457] Dumping ftrace buffer:
[    5.689457]    (ftrace buffer empty)
[    5.689457] ---[ end trace 0000000000000000 ]---
[    5.689457] RIP: 0010:wakeup_preempt_fair+0x1b7/0x430
[    5.689457] Code: 74 0b 48 8b 52 28 48 39 d0 48 0f 47 c2 48 8b b9
90 00 00 00 48 8b b1 08 01 00 00 48 81 ff 00 00 10 00 74 09 48 c1 e0
14 31 9
[    5.689457] RSP: 0000:ffffc9000021fd70 EFLAGS: 00010046
[    5.689457] RAX: 000002ab98000000 RBX: ffff8881b8e2db40 RCX: ffffffff83022a80
[    5.689457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    5.689457] RBP: 0000000000000001 R08: ffff88810cb14380 R09: ffffffff83022b00
[    5.689457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[    5.689457] R13: 0000000000000000 R14: ffff88810cb14300 R15: ffff8881b8e2da00
[    5.689457] FS:  0000000000000000(0000) GS:ffff888235c2e000(0000)
knlGS:0000000000000000
[    5.689457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    5.689457] CR2: 0000000000000000 CR3: 000000000304c001 CR4: 0000000000370ef0
[    5.689457] Kernel panic - not syncing: Fatal exception

Which I bisected down to this last patch in the series.

faddr2line gave me:
__calc_delta at kernel/sched/fair.c:290
(inlined by) calc_delta_fair at kernel/sched/fair.c:300 (inlined by) update_protect_slice at kernel/sched/fair.c:1070 (inlined by) wakeup_preempt_fair at kernel/sched/fair.c:9193

This usually trips as the ww_mutex selftest starts at bootup.

Quoted 9 lines
Quoted 4 lines

Unfortunately I still see it with the add-on changes you proposed to K Prateek's feedback here.

I'll try to narrow it down further tomorrow.

As karma would have it, this does seem to depend on CONFIG_SCHED_PROXY_EXEC. :) I'm guessing the switch in calc_delta_fair() to use se->h_load is uncovering something proxy isn't handling properly with that value.

So looking at the callstack when I see the failure:
proxy_find_task()
proxy_force_return()
proxy_resched_idle() <- sets rq->donor to idle attach_one_task()
wakeup_preempt()
wakeup_preempt_fair()
update_protect_slice() <- called with the donor's se
calc_delta_fair()
__calc_delta() <- div by zero

Basically we end up in wakeup_preempt_fair() with rq->donor == rq->idle because we earlier called proxy_resched_idle().

Without proxy, if we call wakeup_preempt_fair() when rq->donor (and rq->curr) is rq->idle, we usually end up taking the if (test_tsk_need_resched(rq->curr)) early exit and we don't hit this.

But with proxy, rq->curr isn't idle at this point. So we end up continuing on. Despite the se_is_idle(se) checks (where se is the &donor->se), those don't catch because rq->idle (maybe unintuitvely) has a SCHED_NORMAL policy.

So we end up getting down to update_protect_slice() with rq->idle as the se and the idle h_load.weight is zero.

Not sure what the best approach might be, but adding:

  if (donor == rq->idle) {
    /* don't give rq->idle slice protection */
    preempt_action = PREEMPT_WAKEUP_SHORT;
    goto preempt;

}

similar to the if (cse_is_idle && !pse_is_idle) check seems to resolve this.

Anyway, if you have thoughts on better approach, I'd be happy to work up a patch to add on top of this one.

thanks
-john

#31 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Hello John,

Quoted 7 lines · John Stultz

So looking at the callstack when I see the failure:
proxy_find_task()
proxy_force_return()

    proxy_resched_idle()  <- sets rq->donor to idle
    attach_one_task()
      wakeup_preempt()
        wakeup_preempt_fair()

After this point, I would have expected we called idle class's wakeup_preempt() since that is the donor context ...

Quoted 6 lines

update_protect_slice() <- called with the donor's se
calc_delta_fair()
__calc_delta() <- div by zero

Basically we end up in wakeup_preempt_fair() with rq->donor == rq->idle because we earlier called proxy_resched_idle().

Could you check if following makes things better:

(Only build tested)

kernel/sched/core.c +1 -0
index 3ae5f19c1b7e..77f4ebe8f5c7 100644
@@ -6653,6 +6653,7 @@ static inline void proxy_set_task_cpu(struct task_struct *p, int cpu)
 static inline struct task_struct *proxy_resched_idle(struct rq *rq)
 {
 	put_prev_set_next_task(rq, rq->donor, rq->idle);
+	rq->next_class = &idle_sched_class;
 	rq_set_donor(rq, rq->idle);
 	set_tsk_need_resched(rq->idle);
 	return rq->idle;

---

I'm just getting started for the day so it'll be a while before I actually get to test this on top of flat cgroup bits which I haven't yet run with SCHED_PROXY_EXEC enabled.

Signature · 2 lines
Thanks and Regards,
Prateek
#32 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 10 lines · K Prateek Nayak
Quoted 7 lines · John Stultz

So looking at the callstack when I see the failure:
proxy_find_task()
proxy_force_return()

    proxy_resched_idle()  <- sets rq->donor to idle
    attach_one_task()
      wakeup_preempt()
        wakeup_preempt_fair()
Quoted 2 lines

After this point, I would have expected we called idle class's wakeup_preempt() since that is the donor context ...

Ah, that's a good point! (I was getting muddied by the rq->idle having SCHED_NORMAL policy value and assuming that was why we were in the fair code).

Quoted 16 lines
Quoted 6 lines

update_protect_slice() <- called with the donor's se
calc_delta_fair()
__calc_delta() <- div by zero

Basically we end up in wakeup_preempt_fair() with rq->donor == rq->idle because we earlier called proxy_resched_idle().

Could you check if following makes things better:

(Only build tested)

kernel/sched/core.c +0 -0
index 3ae5f19c1b7e..77f4ebe8f5c7 100644

kernel/sched/core.c static inline void proxy_set_task_cpu(struct task_struct *p, int cpu) · line 6659

Quoted 7 lines
kernel/sched/core.c +1 -0
@@ -6653,6 +6653,7 @@ static inline void proxy_set_task_cpu(struct task_struct *p, int cpu)
 static inline struct task_struct *proxy_resched_idle(struct rq *rq)
 {
        put_prev_set_next_task(rq, rq->donor, rq->idle);
+       rq->next_class = &idle_sched_class;
        rq_set_donor(rq, rq->idle);
        set_tsk_need_resched(rq->idle);
        return rq->idle;

Yeah, that looks to avoid the problem and is a fair bit cleaner. I missed the introduction of the rq->next_class detail! Thanks for pointing this out!

I'll do some testing against the full series and get a patch sent out here soon. -john

maintainer
#33 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

In reply to Peter Zijlstra
Quoted 1,168 lines · Peter Zijlstra

Change fair/cgroup to a single runqueue.

Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.

The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the shares_weight approximation are fully preserved.

That is, given a hierarchy like:

        R
        |
        se--G1
            / \
      G2--se   se--G3
     / \           |

T1--se se--T2 se--T3

This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:

     _R_
    / | \

T1 T2 T3

Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.

Notably, the effective weight (se->h_load) is computed from the hierarchical fraction: se->load / cfs_rq->load.

Since EEVDF is now exclusive operating on rq->cfs, it needs to consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.

One place where additional information was required was set_next_task() / put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).

As a result of only having a single level to pick from, much of the complications in pick_next_task() and preemption go away.

Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.

TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr

---
 include/linux/sched.h |    1
 kernel/sched/core.c   |    5
 kernel/sched/debug.c  |    9
 kernel/sched/fair.c   |  789 +++++++++++++++++++++-----------------------------
 kernel/sched/pelt.c   |    6
 kernel/sched/sched.h  |   26 -
 6 files changed, 366 insertions(+), 470 deletions(-)
include/linux/sched.h +1 -0
@@ -575,6 +575,7 @@ struct sched_statistics {
 struct sched_entity {
        /* For load-balancing: */
        struct load_weight              load;
+       struct load_weight              h_load;
        struct rb_node                  run_node;
        u64                             deadline;
        u64                             min_vruntime;
kernel/sched/core.c +1 -4
@@ -5539,11 +5539,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);
  */
 static inline void prefetch_curr_exec_start(struct task_struct *p)
 {
-#ifdef CONFIG_FAIR_GROUP_SCHED
-       struct sched_entity *curr = p->se.cfs_rq->curr;
-#else
        struct sched_entity *curr = task_rq(p)->cfs.curr;
-#endif
+
        prefetch(curr);
        prefetch(&curr->exec_start);
 }
kernel/sched/debug.c +6 -3
@@ -911,10 +911,11 @@ print_task(struct seq_file *m, struct rq
        else
                SEQ_printf(m, " %c", task_state_to_char(p));
 
-       SEQ_printf(m, " %15s %5d %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
+       SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
                p->comm, task_pid_nr(p),
+               p->se.h_load.weight,
                SPLIT_NS(p->se.vruntime),
-               entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
+               entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N',
                SPLIT_NS(p->se.deadline),
                p->se.custom_slice ? 'S' : ' ',
                SPLIT_NS(p->se.slice),
@@ -943,7 +944,7 @@ static void print_rq(struct seq_file *m,
 
        SEQ_printf(m, "\n");
        SEQ_printf(m, "runnable tasks:\n");
-       SEQ_printf(m, " S            task   PID       vruntime   eligible    "
+       SEQ_printf(m, " S            task   PID     weight       vruntime   eligible    "
                   "deadline             slice          sum-exec      switches  "
                   "prio         wait-time        sum-sleep       sum-block"
 #ifdef CONFIG_NUMA_BALANCING
@@ -1051,6 +1052,8 @@ void print_cfs_rq(struct seq_file *m, in
                        cfs_rq->tg_load_avg_contrib);
        SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_avg",
                        atomic_long_read(&cfs_rq->tg->load_avg));
+       SEQ_printf(m, "  .%-30s: %lu\n", "h_load",
+                       cfs_rq->h_load);
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 #ifdef CONFIG_CFS_BANDWIDTH
        SEQ_printf(m, "  .%-30s: %d\n", "throttled",
kernel/sched/fair.c +249 -316
@@ -296,8 +296,8 @@ static u64 __calc_delta(u64 delta_exec,
  */
 static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)
 {
-       if (unlikely(se->load.weight != NICE_0_LOAD))
-               delta = __calc_delta(delta, NICE_0_LOAD, &se->load);
+       if (se->h_load.weight != NICE_0_LOAD)
+               delta = __calc_delta(delta, NICE_0_LOAD, &se->h_load);
 
        return delta;
 }
@@ -427,38 +427,6 @@ static inline struct sched_entity *paren
        return se->parent;
 }
 
-static void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-       int se_depth, pse_depth;
-
-       /*
-        * preemption test can be made between sibling entities who are in the
-        * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
-        * both tasks until we find their ancestors who are siblings of common
-        * parent.
-        */
-
-       /* First walk up until both entities are at same depth */
-       se_depth = (*se)->depth;
-       pse_depth = (*pse)->depth;
-
-       while (se_depth > pse_depth) {
-               se_depth--;
-               *se = parent_entity(*se);
-       }
-
-       while (pse_depth > se_depth) {
-               pse_depth--;
-               *pse = parent_entity(*pse);
-       }
-
-       while (!is_same_group(*se, *pse)) {
-               *se = parent_entity(*se);
-               *pse = parent_entity(*pse);
-       }
-}
-
 static int tg_is_idle(struct task_group *tg)
 {
        return tg->idle > 0;
@@ -502,11 +470,6 @@ static inline struct sched_entity *paren
        return NULL;
 }
 
-static inline void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-}
-
 static inline int tg_is_idle(struct task_group *tg)
 {
        return 0;
@@ -685,7 +648,7 @@ static inline unsigned long avg_vruntime
 static inline void
 __sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+       unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
        s64 w_vruntime, key = entity_key(cfs_rq, se);
 
        w_vruntime = key * weight;
@@ -702,7 +665,7 @@ sum_w_vruntime_add_paranoid(struct cfs_r
        s64 key, tmp;
 
 again:
-       weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+       weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
        key = entity_key(cfs_rq, se);
 
        if (check_mul_overflow(key, weight, &key))
@@ -748,7 +711,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq
 static void
 sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+       unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
        s64 key = entity_key(cfs_rq, se);
 
        cfs_rq->sum_w_vruntime -= key * weight;
@@ -790,7 +753,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)
                s64 runtime = cfs_rq->sum_w_vruntime;
 
                if (curr) {
-                       unsigned long w = avg_vruntime_weight(cfs_rq, curr->load.weight);
+                       unsigned long w = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
                        runtime += entity_key(cfs_rq, curr) * w;
                        weight += w;
@@ -861,8 +824,6 @@ bool update_entity_lag(struct cfs_rq *cf
        u64 avruntime = avg_vruntime(cfs_rq);
        s64 vlag = entity_lag(cfs_rq, se, avruntime);
 
-       WARN_ON_ONCE(!se->on_rq);
-
        if (se->sched_delayed) {
                /* previous vlag < 0 otherwise se would not be delayed */
                vlag = max(vlag, se->vlag);
@@ -898,7 +859,7 @@ static int vruntime_eligible(struct cfs_
        long load = cfs_rq->sum_weight;
 
        if (curr && curr->on_rq) {
-               unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight);
+               unsigned long weight = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
                avg += entity_key(cfs_rq, curr) * weight;
                load += weight;
@@ -1039,6 +1000,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim
  */
 static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+       WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+       WARN_ON_ONCE(!entity_is_task(se));
+
        sum_w_vruntime_add(cfs_rq, se);
        se->min_vruntime = se->vruntime;
        se->min_slice = se->slice;
@@ -1048,6 +1012,9 @@ static void __enqueue_entity(struct cfs_
 
 static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+       WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+       WARN_ON_ONCE(!entity_is_task(se));
+
        rb_erase_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
                                  &min_vruntime_cb);
        sum_w_vruntime_sub(cfs_rq, se);
@@ -1144,7 +1111,7 @@ static struct sched_entity *pick_eevdf(s
         * We can safely skip eligibility check if there is only one entity
         * in this cfs_rq, saving some cycles.
         */
-       if (cfs_rq->nr_queued == 1)
+       if (cfs_rq->h_nr_queued == 1)
                return curr && curr->on_rq ? curr : se;
 
        /*
@@ -1391,8 +1358,6 @@ static s64 update_se(struct rq *rq, stru
        return delta_exec;
 }
 
-static void set_next_buddy(struct sched_entity *se);
-
 /*
  * Used by other classes to account runtime.
  */
@@ -1412,7 +1377,7 @@ static void update_curr(struct cfs_rq *c
         * not necessarily be the actual task running
         * (rq->curr.se). This is easy to confuse!
         */
-       struct sched_entity *curr = cfs_rq->curr;
+       struct sched_entity *curr = cfs_rq->h_curr;
        struct rq *rq = rq_of(cfs_rq);
        s64 delta_exec;
        bool resched;
@@ -1424,26 +1389,29 @@ static void update_curr(struct cfs_rq *c
        if (unlikely(delta_exec <= 0))
                return;
 
+       account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+       if (!entity_is_task(curr))
+               return;
+
+       cfs_rq = &rq->cfs;
+
        curr->vruntime += calc_delta_fair(delta_exec, curr);
        resched = update_deadline(cfs_rq, curr);
 
-       if (entity_is_task(curr)) {
-               /*
-                * If the fair_server is active, we need to account for the
-                * fair_server time whether or not the task is running on
-                * behalf of fair_server or not:
-                *  - If the task is running on behalf of fair_server, we need
-                *    to limit its time based on the assigned runtime.
-                *  - Fair task that runs outside of fair_server should account
-                *    against fair_server such that it can account for this time
-                *    and possibly avoid running this period.
-                */
-               dl_server_update(&rq->fair_server, delta_exec);
-       }
-
-       account_cfs_rq_runtime(cfs_rq, delta_exec);
+       /*
+        * If the fair_server is active, we need to account for the
+        * fair_server time whether or not the task is running on
+        * behalf of fair_server or not:
+        *  - If the task is running on behalf of fair_server, we need
+        *    to limit its time based on the assigned runtime.
+        *  - Fair task that runs outside of fair_server should account
+        *    against fair_server such that it can account for this time
+        *    and possibly avoid running this period.
+        */
+       dl_server_update(&rq->fair_server, delta_exec);
 
-       if (cfs_rq->nr_queued == 1)
+       if (cfs_rq->h_nr_queued == 1)
                return;
 
        if (resched || !protect_slice(curr)) {
@@ -1454,7 +1422,10 @@ static void update_curr(struct cfs_rq *c
 
 static void update_curr_fair(struct rq *rq)
 {
-       update_curr(cfs_rq_of(&rq->donor->se));
+       struct sched_entity *se = &rq->donor->se;
+
+       for_each_sched_entity(se)
+               update_curr(cfs_rq_of(se));
 }
 
 static inline void
@@ -1530,7 +1501,7 @@ update_stats_enqueue_fair(struct cfs_rq
         * Are we enqueueing a waiting task? (for current tasks
         * a dequeue/enqueue event is a NOP)
         */
-       if (se != cfs_rq->curr)
+       if (se != cfs_rq->h_curr)
                update_stats_wait_start_fair(cfs_rq, se);
 
        if (flags & ENQUEUE_WAKEUP)
@@ -1548,7 +1519,7 @@ update_stats_dequeue_fair(struct cfs_rq
         * Mark the end of the wait period if dequeueing a
         * waiting task:
         */
-       if (se != cfs_rq->curr)
+       if (se != cfs_rq->h_curr)
                update_stats_wait_end_fair(cfs_rq, se);
 
        if ((flags & DEQUEUE_SLEEP) && entity_is_task(se)) {
@@ -3875,6 +3846,7 @@ static inline void update_scan_period(st
 static void
 account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+       WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
        update_load_add(&cfs_rq->load, se->load.weight);
        if (entity_is_task(se)) {
                struct rq *rq = rq_of(cfs_rq);
@@ -3888,6 +3860,7 @@ account_entity_enqueue(struct cfs_rq *cf
 static void
 account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+       WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
        update_load_sub(&cfs_rq->load, se->load.weight);
        if (entity_is_task(se)) {
                account_numa_dequeue(rq_of(cfs_rq), task_of(se));
@@ -3965,7 +3938,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,
 static void
 rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vprot)
 {
-       unsigned long old_weight = se->load.weight;
+       long old_weight = se->h_load.weight;
 
        /*
         * VRUNTIME
@@ -4065,16 +4038,17 @@ rescale_entity(struct sched_entity *se,
                se->vprot = div64_long(se->vprot * old_weight, weight);
 }
 
-static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
-                           unsigned long weight)
+static void reweight_eevdf(struct cfs_rq *cfs_rq, struct sched_entity *se,
+                          unsigned long weight, bool on_rq)
 {
        bool curr = cfs_rq->curr == se;
        bool rel_vprot = false;
        u64 avruntime = 0;
 
-       if (se->on_rq) {
-               /* commit outstanding execution time */
-               update_curr(cfs_rq);
+       if (se->h_load.weight == weight)
+               return;
+
+       if (on_rq) {
                avruntime = avg_vruntime(cfs_rq);
                se->vlag = entity_lag(cfs_rq, se, avruntime);
                se->deadline -= avruntime;
@@ -4084,46 +4058,90 @@ static void reweight_entity(struct cfs_r
                        rel_vprot = true;
                }
 
-               cfs_rq->nr_queued--;
+               cfs_rq->h_nr_queued--;
                if (!curr)
                        __dequeue_entity(cfs_rq, se);
-               update_load_sub(&cfs_rq->load, se->load.weight);
        }
-       dequeue_load_avg(cfs_rq, se);
 
        rescale_entity(se, weight, rel_vprot);
 
-       update_load_set(&se->load, weight);
+       update_load_set(&se->h_load, weight);
 
-       do {
-               u32 divider = get_pelt_divider(&se->avg);
-               se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
-       } while (0);
-
-       enqueue_load_avg(cfs_rq, se);
-       if (se->on_rq) {
+       if (on_rq) {
                if (rel_vprot)
                        se->vprot += avruntime;
                se->deadline += avruntime;
                se->rel_deadline = 0;
                se->vruntime = avruntime - se->vlag;
 
-               update_load_add(&cfs_rq->load, se->load.weight);
                if (!curr)
                        __enqueue_entity(cfs_rq, se);
-               cfs_rq->nr_queued++;
+               cfs_rq->h_nr_queued++;
        }
 }
 
+static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
+                           unsigned long weight)
+{
+       if (se->load.weight == weight)
+               return;
+
+       if (se->on_rq) {
+               WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
+               update_load_sub(&cfs_rq->load, se->load.weight);
+       }
+       dequeue_load_avg(cfs_rq, se);
+
+       update_load_set(&se->load, weight);
+
+       do {
+               u32 divider = get_pelt_divider(&se->avg);
+               se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
+       } while (0);
+
+       enqueue_load_avg(cfs_rq, se);
+
+       if (se->on_rq)
+               update_load_add(&cfs_rq->load, se->load.weight);
+}
+
+/*
+ * weight = NICE_0_LOAD;
+ * for_each_entity_se(se)
+ *   weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
+ */
+static __always_inline
+unsigned long __calc_prop_weight(struct cfs_rq *cfs_rq, struct sched_entity *se,
+                                unsigned long weight)
+{
+       weight *= se->load.weight;
+       if (parent_entity(se))
+               weight /= cfs_rq->load.weight;
+       else
+               weight /= NICE_0_LOAD;
+
+       return max(weight, MIN_SHARES);
+}
+
 static void reweight_task_fair(struct rq *rq, struct task_struct *p,
                               const struct load_weight *lw)
 {
        struct sched_entity *se = &p->se;
-       struct cfs_rq *cfs_rq = cfs_rq_of(se);
-       struct load_weight *load = &se->load;
+       unsigned long weight = NICE_0_LOAD;
+
+       if (se->on_rq)
+               update_curr_fair(rq);
+
+       reweight_entity(cfs_rq_of(se), se, lw->weight);
+       se->load.inv_weight = lw->inv_weight;
+
+       if (!se->on_rq)
+               return;
+
+       for_each_sched_entity(se)
+               weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
 
-       reweight_entity(cfs_rq, se, lw->weight);
-       load->inv_weight = lw->inv_weight;
+       reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq);
 }
 
 static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
@@ -4331,7 +4349,6 @@ static long calc_group_shares(struct cfs
 static void update_cfs_group(struct sched_entity *se)
 {
        struct cfs_rq *gcfs_rq = group_cfs_rq(se);
-       long shares;
 
        /*
         * When a group becomes empty, preserve its weight. This matters for
@@ -4340,9 +4357,7 @@ static void update_cfs_group(struct sche
        if (!gcfs_rq || !gcfs_rq->load.weight)
                return;
 
-       shares = calc_group_shares(gcfs_rq);
-       if (unlikely(se->load.weight != shares))
-               reweight_entity(cfs_rq_of(se), se, shares);
+       reweight_entity(cfs_rq_of(se), se, calc_group_shares(gcfs_rq));
 }
 
 #else /* !CONFIG_FAIR_GROUP_SCHED: */
@@ -4460,7 +4475,7 @@ static inline bool cfs_rq_is_decayed(str
  * differential update where we store the last value we propagated. This in
  * turn allows skipping updates if the differential is 'small'.
  *
- * Updating tg's load_avg is necessary before update_cfs_share().
+ * Updating tg's load_avg is necessary before update_cfs_group().
  */
 static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
 {
@@ -4926,7 +4941,7 @@ static void migrate_se_pelt_lag(struct s
  * The cfs_rq avg is the direct sum of all its entities (blocked and runnable)
  * avg. The immediate corollary is that all (fair) tasks must be attached.
  *
- * cfs_rq->avg is used for task_h_load() and update_cfs_share() for example.
+ * cfs_rq->avg is used for task_h_load() and update_cfs_group() for example.
  *
  * Return: true if the load decayed or we removed load.
  *
@@ -5475,6 +5490,7 @@ static void
 place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
        u64 vslice, vruntime = avg_vruntime(cfs_rq);
+       unsigned int nr_queued = cfs_rq->h_nr_queued;
        bool update_zero = false;
        s64 lag = 0;
 
@@ -5482,6 +5498,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
                se->slice = sysctl_sched_base_slice;
        vslice = calc_delta_fair(se->slice, se);
 
+       if (flags & ENQUEUE_QUEUED)
+               nr_queued -= 1;
+
        /*
         * Due to how V is constructed as the weighted average of entities,
         * adding tasks with positive lag, or removing tasks with negative lag
@@ -5490,7 +5509,7 @@ place_entity(struct cfs_rq *cfs_rq, stru
         *
         * EEVDF: placement strategy #1 / #2
         */
-       if (sched_feat(PLACE_LAG) && cfs_rq->nr_queued && se->vlag) {
+       if (sched_feat(PLACE_LAG) && nr_queued && se->vlag) {
                struct sched_entity *curr = cfs_rq->curr;
                long load, weight;
 
@@ -5550,9 +5569,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
                 */
                load = cfs_rq->sum_weight;
                if (curr && curr->on_rq)
-                       load += avg_vruntime_weight(cfs_rq, curr->load.weight);
+                       load += avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
-               weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+               weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
                lag *= load + weight;
                if (WARN_ON_ONCE(!load))
                        load = 1;
@@ -5611,22 +5630,8 @@ static void check_enqueue_throttle(struc
 static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
 
 static void
-requeue_delayed_entity(struct sched_entity *se);
-
-static void
 enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
-       bool curr = cfs_rq->curr == se;
-
-       /*
-        * If we're the current task, we must renormalise before calling
-        * update_curr().
-        */
-       if (curr)
-               place_entity(cfs_rq, se, flags);
-
-       update_curr(cfs_rq);
-
        /*
         * When enqueuing a sched_entity, we must:
         *   - Update loads to have both entity and cfs_rq synced with now.
@@ -5645,13 +5650,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
         */
        update_cfs_group(se);
 
-       /*
-        * XXX now that the entity has been re-weighted, and it's lag adjusted,
-        * we can place the entity.
-        */
-       if (!curr)
-               place_entity(cfs_rq, se, flags);
-
        account_entity_enqueue(cfs_rq, se);
 
        /* Entity has migrated, no longer consider this task hot */
@@ -5660,8 +5658,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
 
        check_schedstat_required();
        update_stats_enqueue_fair(cfs_rq, se, flags);
-       if (!curr)
-               __enqueue_entity(cfs_rq, se);
        se->on_rq = 1;
 
        if (cfs_rq->nr_queued == 1) {
@@ -5679,21 +5675,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
        }
 }
 
-static void __clear_buddies_next(struct sched_entity *se)
+static void set_next_buddy(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       for_each_sched_entity(se) {
-               struct cfs_rq *cfs_rq = cfs_rq_of(se);
-               if (cfs_rq->next != se)
-                       break;
-
-               cfs_rq->next = NULL;
-       }
+       if (WARN_ON_ONCE(!se->on_rq || se->sched_delayed))
+               return;
+       if (se_is_idle(se))
+               return;
+       cfs_rq->next = se;
 }
 
 static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
        if (cfs_rq->next == se)
-               __clear_buddies_next(se);
+               cfs_rq->next = NULL;
 }
 
 static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
@@ -5704,7 +5698,7 @@ static void set_delayed(struct sched_ent
 
        /*
         * Delayed se of cfs_rq have no tasks queued on them.
-        * Do not adjust h_nr_runnable since dequeue_entities()
+        * Do not adjust h_nr_runnable since __dequeue_task()
         * will account it for blocked tasks.
         */
        if (!entity_is_task(se))
@@ -5737,37 +5731,11 @@ static void clear_delayed(struct sched_e
        }
 }
 
-static bool
+static void
 dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
-       bool sleep = flags & DEQUEUE_SLEEP;
        int action = UPDATE_TG;
 
-       update_curr(cfs_rq);
-       clear_buddies(cfs_rq, se);
-
-       if (flags & DEQUEUE_DELAYED) {
-               WARN_ON_ONCE(!se->sched_delayed);
-       } else {
-               bool delay = sleep;
-               /*
-                * DELAY_DEQUEUE relies on spurious wakeups, special task
-                * states must not suffer spurious wakeups, excempt them.
-                */
-               if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
-                       delay = false;
-
-               WARN_ON_ONCE(delay && se->sched_delayed);
-
-               if (sched_feat(DELAY_DEQUEUE) && delay &&
-                   !entity_eligible(cfs_rq, se)) {
-                       update_load_avg(cfs_rq, se, 0);
-                       update_entity_lag(cfs_rq, se);
-                       set_delayed(se);
-                       return false;
-               }
-       }
-
        if (entity_is_task(se) && task_on_rq_migrating(task_of(se)))
                action |= DO_DETACH;
 
@@ -5785,14 +5753,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 
        update_stats_dequeue_fair(cfs_rq, se, flags);
 
-       update_entity_lag(cfs_rq, se);
-       if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
-               se->deadline -= se->vruntime;
-               se->rel_deadline = 1;
-       }
-
-       if (se != cfs_rq->curr)
-               __dequeue_entity(cfs_rq, se);
        se->on_rq = 0;
        account_entity_dequeue(cfs_rq, se);
 
@@ -5801,9 +5761,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 
        update_cfs_group(se);
 
-       if (flags & DEQUEUE_DELAYED)
-               clear_delayed(se);
-
        if (cfs_rq->nr_queued == 0) {
                update_idle_cfs_rq_clock_pelt(cfs_rq);
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -5816,15 +5773,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
                }
 #endif
        }
-
-       return true;
 }
 
 static void
-set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
+set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       clear_buddies(cfs_rq, se);
-
        /* 'current' is not kept within the tree. */
        if (se->on_rq) {
                /*
@@ -5833,16 +5786,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s
                 * runqueue.
                 */
                update_stats_wait_end_fair(cfs_rq, se);
-               __dequeue_entity(cfs_rq, se);
                update_load_avg(cfs_rq, se, UPDATE_TG);
-
-               if (first)
-                       set_protect_slice(cfs_rq, se);
        }
 
        update_stats_curr_start(cfs_rq, se);
-       WARN_ON_ONCE(cfs_rq->curr);
-       cfs_rq->curr = se;
+       WARN_ON_ONCE(cfs_rq->h_curr);
+       cfs_rq->h_curr = se;
 
        /*
         * Track our maximum slice length, if the CPU's load is at
@@ -5862,23 +5811,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s
        se->prev_sum_exec_runtime = se->sum_exec_runtime;
 }
 
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags);
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags);
 
-/*
- * Pick the next process, keeping these things in mind, in this order:
- * 1) keep things fair between processes/task groups
- * 2) pick the "next" process, since someone really wants that to run
- * 3) pick the "last" process, for cache locality
- * 4) do not run the "skip" process, if something else is available
- */
 static struct sched_entity *
-pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
+pick_next_entity(struct rq *rq, bool protect)
 {
+       struct cfs_rq *cfs_rq = &rq->cfs;
        struct sched_entity *se;
 
        se = pick_eevdf(cfs_rq, protect);
        if (se->sched_delayed) {
-               dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+               __dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED);
                /*
                 * Must not reference @se again, see __block_task().
                 */
@@ -5903,13 +5846,11 @@ static void put_prev_entity(struct cfs_r
 
        if (prev->on_rq) {
                update_stats_wait_start_fair(cfs_rq, prev);
-               /* Put 'current' back into the tree. */
-               __enqueue_entity(cfs_rq, prev);
                /* in !on_rq case, update occurred at dequeue */
                update_load_avg(cfs_rq, prev, 0);
        }
-       WARN_ON_ONCE(cfs_rq->curr != prev);
-       cfs_rq->curr = NULL;
+       WARN_ON_ONCE(cfs_rq->h_curr != prev);
+       cfs_rq->h_curr = NULL;
 }
 
 static void
@@ -6062,7 +6003,7 @@ static void __account_cfs_rq_runtime(str
         * if we're unable to extend our runtime we resched so that the active
         * hierarchy can be throttled
         */
-       if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->curr))
+       if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->h_curr))
                resched_curr(rq_of(cfs_rq));
 }
 
@@ -6420,7 +6361,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cf
        assert_list_leaf_cfs_rq(rq);
 
        /* Determine whether we need to wake up potentially idle CPU: */
-       if (rq->curr == rq->idle && rq->cfs.nr_queued)
+       if (rq->curr == rq->idle && rq->cfs.h_nr_queued)
                resched_curr(rq);
 }
 
@@ -6761,7 +6702,7 @@ static void check_enqueue_throttle(struc
                return;
 
        /* an active group must be handled by the update_curr()->put() path */
-       if (!cfs_rq->runtime_enabled || cfs_rq->curr)
+       if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)
                return;
 
        /* ensure the group is not already throttled */
@@ -7156,7 +7097,7 @@ static void hrtick_start_fair(struct rq
                        resched_curr(rq);
                return;
        }
-       delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+       delta = (se->h_load.weight * vdelta) / NICE_0_LOAD;
 
        /*
         * Correct for instantaneous load of other classes.
@@ -7256,10 +7197,8 @@ static int choose_idle_cpu(int cpu, stru
 }
 
 static void
-requeue_delayed_entity(struct sched_entity *se)
+requeue_delayed_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       struct cfs_rq *cfs_rq = cfs_rq_of(se);
-
        /*
         * se->sched_delayed should imply: se->on_rq == 1.
         * Because a delayed entity is one that is still on
@@ -7269,19 +7208,58 @@ requeue_delayed_entity(struct sched_enti
        WARN_ON_ONCE(!se->on_rq);
 
        if (update_entity_lag(cfs_rq, se)) {
-               cfs_rq->nr_queued--;
+               cfs_rq->h_nr_queued--;
                if (se != cfs_rq->curr)
                        __dequeue_entity(cfs_rq, se);
                place_entity(cfs_rq, se, 0);
                if (se != cfs_rq->curr)
                        __enqueue_entity(cfs_rq, se);
-               cfs_rq->nr_queued++;
+               cfs_rq->h_nr_queued++;
        }
 
        update_load_avg(cfs_rq, se, 0);
        clear_delayed(se);
 }
 
+static unsigned long enqueue_hierarchy(struct task_struct *p, int flags)
+{
+       unsigned long weight = NICE_0_LOAD;
+       int task_new = !(flags & ENQUEUE_WAKEUP);
+       struct sched_entity *se = &p->se;
+       int h_nr_idle = task_has_idle_policy(p);
+       int h_nr_runnable = 1;
+
+       if (task_new && se->sched_delayed)
+               h_nr_runnable = 0;
+
+       for_each_sched_entity(se) {
+               struct cfs_rq *cfs_rq = cfs_rq_of(se);
+
+               update_curr(cfs_rq);
+
+               if (!se->on_rq) {
+                       enqueue_entity(cfs_rq, se, flags);
+               } else {
+                       update_load_avg(cfs_rq, se, UPDATE_TG);
+                       se_update_runnable(se);
+                       update_cfs_group(se);
+               }
+
+               cfs_rq->h_nr_runnable += h_nr_runnable;
+               cfs_rq->h_nr_queued++;
+               cfs_rq->h_nr_idle += h_nr_idle;
+
+               if (cfs_rq_is_idle(cfs_rq))
+                       h_nr_idle = 1;
+
+               weight = __calc_prop_weight(cfs_rq, se, weight);
+
+               flags = ENQUEUE_WAKEUP;
+       }
+
+       return weight;
+}
+
 /*
  * The enqueue_task method is called before nr_running is
  * increased. Here we update the fair scheduling stats and
@@ -7290,13 +7268,12 @@ requeue_delayed_entity(struct sched_enti
 static void
 enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 {
-       struct cfs_rq *cfs_rq;
-       struct sched_entity *se = &p->se;
-       int h_nr_idle = task_has_idle_policy(p);
-       int h_nr_runnable = 1;
-       int task_new = !(flags & ENQUEUE_WAKEUP);
        int rq_h_nr_queued = rq->cfs.h_nr_queued;
-       u64 slice = 0;
+       int task_new = !(flags & ENQUEUE_WAKEUP);
+       struct sched_entity *se = &p->se;
+       struct cfs_rq *cfs_rq = &rq->cfs;
+       unsigned long weight;
+       bool curr;
 
        if (task_is_throttled(p) && enqueue_throttled_task(p))
                return;
@@ -7308,10 +7285,10 @@ enqueue_task_fair(struct rq *rq, struct
         * estimated utilization, before we update schedutil.
         */
        if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))
-               util_est_enqueue(&rq->cfs, p);
+               util_est_enqueue(cfs_rq, p);
 
        if (flags & ENQUEUE_DELAYED) {
-               requeue_delayed_entity(se);
+               requeue_delayed_entity(cfs_rq, se);
                return;
        }
 
@@ -7323,57 +7300,22 @@ enqueue_task_fair(struct rq *rq, struct
        if (p->in_iowait)
                cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT);
 
-       if (task_new && se->sched_delayed)
-               h_nr_runnable = 0;
-
-       for_each_sched_entity(se) {
-               if (se->on_rq) {
-                       if (se->sched_delayed)
-                               requeue_delayed_entity(se);
-                       break;
-               }
-               cfs_rq = cfs_rq_of(se);
-
-               /*
-                * Basically set the slice of group entries to the min_slice of
-                * their respective cfs_rq. This ensures the group can service
-                * its entities in the desired time-frame.
-                */
-               if (slice) {
-                       se->slice = slice;
-                       se->custom_slice = 1;
-               }
-               enqueue_entity(cfs_rq, se, flags);
-               slice = cfs_rq_min_slice(cfs_rq);
-
-               cfs_rq->h_nr_runnable += h_nr_runnable;
-               cfs_rq->h_nr_queued++;
-               cfs_rq->h_nr_idle += h_nr_idle;
-
-               if (cfs_rq_is_idle(cfs_rq))
-                       h_nr_idle = 1;
-
-               flags = ENQUEUE_WAKEUP;
-       }
-
-       for_each_sched_entity(se) {
-               cfs_rq = cfs_rq_of(se);
-
-               update_load_avg(cfs_rq, se, UPDATE_TG);
-               se_update_runnable(se);
-               update_cfs_group(se);
+       /*
+        * XXX comment on the curr thing
+        */
+       curr = (cfs_rq->curr == se);
+       if (curr)
+               place_entity(cfs_rq, se, flags);
 
-               se->slice = slice;
-               if (se != cfs_rq->curr)
-                       min_vruntime_cb_propagate(&se->run_node, NULL);
-               slice = cfs_rq_min_slice(cfs_rq);
+       if (se->on_rq && se->sched_delayed)
+               requeue_delayed_entity(cfs_rq, se);
 
-               cfs_rq->h_nr_runnable += h_nr_runnable;
-               cfs_rq->h_nr_queued++;
-               cfs_rq->h_nr_idle += h_nr_idle;
+       weight = enqueue_hierarchy(p, flags);
 
-               if (cfs_rq_is_idle(cfs_rq))
-                       h_nr_idle = 1;
+       if (!curr) {
+               reweight_eevdf(cfs_rq, se, weight, false);
+               place_entity(cfs_rq, se, flags | ENQUEUE_QUEUED);
+               __enqueue_entity(cfs_rq, se);
        }
 
        if (!rq_h_nr_queued && rq->cfs.h_nr_queued)
@@ -7404,105 +7346,107 @@ enqueue_task_fair(struct rq *rq, struct
        hrtick_update(rq);
 }
 
-/*
- * Basically dequeue_task_fair(), except it can deal with dequeue_entity()
- * failing half-way through and resume the dequeue later.
- *
- * Returns:
- * -1 - dequeue delayed
- *  0 - dequeue throttled
- *  1 - dequeue complete
- */
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
+static void dequeue_hierarchy(struct task_struct *p, int flags)
 {
-       bool was_sched_idle = sched_idle_rq(rq);
+       struct sched_entity *se = &p->se;
        bool task_sleep = flags & DEQUEUE_SLEEP;
        bool task_delayed = flags & DEQUEUE_DELAYED;
        bool task_throttled = flags & DEQUEUE_THROTTLE;
-       struct task_struct *p = NULL;
-       int h_nr_idle = 0;
-       int h_nr_queued = 0;
        int h_nr_runnable = 0;
-       struct cfs_rq *cfs_rq;
-       u64 slice = 0;
+       int h_nr_idle = task_has_idle_policy(p);
+       bool dequeue = true;
 
-       if (entity_is_task(se)) {
-               p = task_of(se);
-               h_nr_queued = 1;
-               h_nr_idle = task_has_idle_policy(p);
-               if (task_sleep || task_delayed || !se->sched_delayed)
-                       h_nr_runnable = 1;
-       }
+       if (task_sleep || task_delayed || !se->sched_delayed)
+               h_nr_runnable = 1;
 
        for_each_sched_entity(se) {
-               cfs_rq = cfs_rq_of(se);
+               struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
-               if (!dequeue_entity(cfs_rq, se, flags)) {
-                       if (p && &p->se == se)
-                               return -1;
+               update_curr(cfs_rq);
 
-                       slice = cfs_rq_min_slice(cfs_rq);
-                       break;
+               if (dequeue) {
+                       dequeue_entity(cfs_rq, se, flags);
+                       /* Don't dequeue parent if it has other entities besides us */
+                       if (cfs_rq->load.weight)
+                               dequeue = false;
+               } else {
+                       update_load_avg(cfs_rq, se, UPDATE_TG);
+                       se_update_runnable(se);
+                       update_cfs_group(se);
                }
 
                cfs_rq->h_nr_runnable -= h_nr_runnable;
-               cfs_rq->h_nr_queued -= h_nr_queued;
+               cfs_rq->h_nr_queued--;
                cfs_rq->h_nr_idle -= h_nr_idle;
 
                if (cfs_rq_is_idle(cfs_rq))
-                       h_nr_idle = h_nr_queued;
+                       h_nr_idle = 1;
 
                if (throttled_hierarchy(cfs_rq) && task_throttled)
                        record_throttle_clock(cfs_rq);
 
-               /* Don't dequeue parent if it has other entities besides us */
-               if (cfs_rq->load.weight) {
-                       slice = cfs_rq_min_slice(cfs_rq);
-
-                       /* Avoid re-evaluating load for this entity: */
-                       se = parent_entity(se);
-                       /*
-                        * Bias pick_next to pick a task from this cfs_rq, as
-                        * p is sleeping when it is within its sched_slice.
-                        */
-                       if (task_sleep && se)
-                               set_next_buddy(se);
-                       break;
-               }
                flags |= DEQUEUE_SLEEP;
                flags &= ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL);
        }
+}
 
-       for_each_sched_entity(se) {
-               cfs_rq = cfs_rq_of(se);
+/*
+ * The part of dequeue_task_fair() that is needed to dequeue delayed tasks.
+ *
+ * Returns:
+ *   true  - dequeued
+ *   false - delayed
+ */
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
+{
+       struct sched_entity *se = &p->se;
+       struct cfs_rq *cfs_rq = &rq->cfs;
+       bool was_sched_idle = sched_idle_rq(rq);
+       bool task_sleep = flags & DEQUEUE_SLEEP;
+       bool task_delayed = flags & DEQUEUE_DELAYED;
 
-               update_load_avg(cfs_rq, se, UPDATE_TG);
-               se_update_runnable(se);
-               update_cfs_group(se);
+       clear_buddies(cfs_rq, se);
 
-               se->slice = slice;
-               if (se != cfs_rq->curr)
-                       min_vruntime_cb_propagate(&se->run_node, NULL);
-               slice = cfs_rq_min_slice(cfs_rq);
+       if (flags & DEQUEUE_DELAYED) {
+               WARN_ON_ONCE(!se->sched_delayed);
+       } else {
+               bool delay = task_sleep;
+               /*
+                * DELAY_DEQUEUE relies on spurious wakeups, special task
+                * states must not suffer spurious wakeups, excempt them.
+                */
+               if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
+                       delay = false;
 

kernel/sched/fair.c enqueue_task_fair(struct rq *rq, struct · line 7425

Quoted 10 lines
kernel/sched/fair.c +4 -5
-               cfs_rq->h_nr_runnable -= h_nr_runnable;
-               cfs_rq->h_nr_queued -= h_nr_queued;
-               cfs_rq->h_nr_idle -= h_nr_idle;
+               WARN_ON_ONCE(delay && se->sched_delayed);
 
-               if (cfs_rq_is_idle(cfs_rq))
-                       h_nr_idle = h_nr_queued;
+               if (sched_feat(DELAY_DEQUEUE) && delay &&
+                   !entity_eligible(cfs_rq, se)) {
+                       update_load_avg(cfs_rq_of(se), se, 0);

update_entity_lag(cfs_rq, se); is missing here. Unfortunately this doesn't fix my regression

Quoted 512 lines
kernel/sched/fair.c +89 -126
+                       set_delayed(se);
+                       return false;
+               }
+       }
 
-               if (throttled_hierarchy(cfs_rq) && task_throttled)
-                       record_throttle_clock(cfs_rq);
+       dequeue_hierarchy(p, flags);
+
+       update_entity_lag(cfs_rq, se);
+       if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
+               se->deadline -= se->vruntime;
+               se->rel_deadline = 1;
        }
+       if (se != cfs_rq->curr)
+               __dequeue_entity(cfs_rq, se);
 
-       sub_nr_running(rq, h_nr_queued);
+       sub_nr_running(rq, 1);
 
        /* balance early to pull high priority tasks */
        if (unlikely(!was_sched_idle && sched_idle_rq(rq)))
                rq->next_balance = jiffies;
 
-       if (p && task_delayed) {
+       if (task_delayed) {
+               clear_delayed(se);
+
                WARN_ON_ONCE(!task_sleep);
                WARN_ON_ONCE(p->on_rq != 1);
 
@@ -7514,7 +7458,7 @@ static int dequeue_entities(struct rq *r
                __block_task(rq, p);
        }
 
-       return 1;
+       return true;
 }
 
 /*
@@ -7533,11 +7477,11 @@ static bool dequeue_task_fair(struct rq
                util_est_dequeue(&rq->cfs, p);
 
        util_est_update(&rq->cfs, p, flags & DEQUEUE_SLEEP);
-       if (dequeue_entities(rq, &p->se, flags) < 0)
+       if (!__dequeue_task(rq, p, flags))
                return false;
 
        /*
-        * Must not reference @p after dequeue_entities(DEQUEUE_DELAYED).
+        * Must not reference @p after __dequeue_task(DEQUEUE_DELAYED).
         */
        return true;
 }
@@ -9021,19 +8965,6 @@ static void migrate_task_rq_fair(struct
 static void task_dead_fair(struct task_struct *p)
 {
        struct sched_entity *se = &p->se;
-
-       if (se->sched_delayed) {
-               struct rq_flags rf;
-               struct rq *rq;
-
-               rq = task_rq_lock(p, &rf);
-               if (se->sched_delayed) {
-                       update_rq_clock(rq);
-                       dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-               }
-               task_rq_unlock(rq, p, &rf);
-       }
-
        remove_entity_load_avg(se);
 }
 
@@ -9067,21 +8998,10 @@ static void set_cpus_allowed_fair(struct
        set_task_max_allowed_capacity(p);
 }
 
-static void set_next_buddy(struct sched_entity *se)
-{
-       for_each_sched_entity(se) {
-               if (WARN_ON_ONCE(!se->on_rq))
-                       return;
-               if (se_is_idle(se))
-                       return;
-               cfs_rq_of(se)->next = se;
-       }
-}
-
 enum preempt_wakeup_action {
        PREEMPT_WAKEUP_NONE,    /* No preemption. */
        PREEMPT_WAKEUP_SHORT,   /* Ignore slice protection. */
-       PREEMPT_WAKEUP_PICK,    /* Let __pick_eevdf() decide. */
+       PREEMPT_WAKEUP_PICK,    /* Let pick_eevdf() decide. */
        PREEMPT_WAKEUP_RESCHED, /* Force reschedule. */
 };
 
@@ -9098,7 +9018,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,
        if (cfs_rq->next && entity_before(cfs_rq->next, pse))
                return false;
 
-       set_next_buddy(pse);
+       set_next_buddy(cfs_rq, pse);
        return true;
 }
 
@@ -9188,7 +9108,6 @@ static void wakeup_preempt_fair(struct r
        if (!sched_feat(WAKEUP_PREEMPTION))
                return;
 
-       find_matching_se(&se, &pse);
        WARN_ON_ONCE(!pse);
 
        cse_is_idle = se_is_idle(se);
@@ -9216,8 +9135,7 @@ static void wakeup_preempt_fair(struct r
        if (unlikely(!normal_policy(p->policy)))
                return;
 
-       cfs_rq = cfs_rq_of(se);
-       update_curr(cfs_rq);
+       update_curr_fair(rq);
        /*
         * If @p has a shorter slice than current and @p is eligible, override
         * current's slice protection in order to allow preemption.
@@ -9261,18 +9179,15 @@ static void wakeup_preempt_fair(struct r
        }
 
 pick:
-       nse = pick_next_entity(rq, cfs_rq, preempt_action != PREEMPT_WAKEUP_SHORT);
-       /* If @p has become the most eligible task, force preemption */
-       if (nse == pse)
-               goto preempt;
-
-       /*
-        * Because p is enqueued, nse being null can only mean that we
-        * dequeued a delayed task. If there are still entities queued in
-        * cfs, check if the next one will be p.
-        */
-       if (!nse && cfs_rq->nr_queued)
-               goto pick;
+       if (cfs_rq->h_nr_queued) {
+               nse = pick_next_entity(rq, preempt_action != PREEMPT_WAKEUP_SHORT);
+               if (unlikely(!nse))
+                       goto pick;
+
+               /* If @p has become the most eligible task, force preemption */
+               if (nse == pse)
+                       goto preempt;
+       }
 
        if (sched_feat(RUN_TO_PARITY))
                update_protect_slice(cfs_rq, se);
@@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r
 struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
        __must_hold(__rq_lockp(rq))
 {
+       struct cfs_rq *cfs_rq = &rq->cfs;
        struct sched_entity *se;
-       struct cfs_rq *cfs_rq;
        struct task_struct *p;
-       bool throttled;
        int new_tasks;
 
 again:
-       cfs_rq = &rq->cfs;
-       if (!cfs_rq->nr_queued)
+       if (!cfs_rq->h_nr_queued)
                goto idle;
 
-       throttled = false;
-
-       do {
-               /* Might not have done put_prev_entity() */
-               if (cfs_rq->curr && cfs_rq->curr->on_rq)
-                       update_curr(cfs_rq);
-
-               throttled |= check_cfs_rq_runtime(cfs_rq);
+       /* Might not have done put_prev_entity() */
+       if (cfs_rq->curr && cfs_rq->curr->on_rq)
+               update_curr(cfs_rq);
 
-               se = pick_next_entity(rq, cfs_rq, true);
-               if (!se)
-                       goto again;
-               cfs_rq = group_cfs_rq(se);
-       } while (cfs_rq);
+       se = pick_next_entity(rq, true);
+       if (!se)
+               goto again;
 
        p = task_of(se);
-       if (unlikely(throttled))
+       if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
                task_throttle_setup_work(p);
        return p;
 
@@ -9353,7 +9259,7 @@ void fair_server_init(struct rq *rq)
 static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct task_struct *next)
 {
        struct sched_entity *se = &prev->se;
-       struct cfs_rq *cfs_rq;
+       struct cfs_rq *cfs_rq = &rq->cfs;
        struct sched_entity *nse = NULL;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9363,7 +9269,7 @@ static void put_prev_task_fair(struct rq
 
        while (se) {
                cfs_rq = cfs_rq_of(se);
-               if (!nse || cfs_rq->curr)
+               if (!nse || cfs_rq->h_curr)
                        put_prev_entity(cfs_rq, se);
 #ifdef CONFIG_FAIR_GROUP_SCHED
                if (nse) {
@@ -9382,6 +9288,14 @@ static void put_prev_task_fair(struct rq
 #endif
                se = parent_entity(se);
        }
+
+       /* Put 'current' back into the tree. */
+       cfs_rq = &rq->cfs;
+       se = &prev->se;
+       WARN_ON_ONCE(cfs_rq->curr != se);
+       cfs_rq->curr = NULL;
+       if (se->on_rq)
+               __enqueue_entity(cfs_rq, se);
 }
 
 /*
@@ -9390,8 +9304,8 @@ static void put_prev_task_fair(struct rq
 static void yield_task_fair(struct rq *rq)
 {
        struct task_struct *curr = rq->donor;
-       struct cfs_rq *cfs_rq = task_cfs_rq(curr);
        struct sched_entity *se = &curr->se;
+       struct cfs_rq *cfs_rq = &rq->cfs;
 
        /*
         * Are we the only task in the tree?
@@ -9432,11 +9346,11 @@ static bool yield_to_task_fair(struct rq
        struct sched_entity *se = &p->se;
 
        /* !se->on_rq also covers throttled task */
-       if (!se->on_rq)
+       if (!se->on_rq || se->sched_delayed)
                return false;
 
        /* Tell the scheduler that we'd really like se to run next. */
-       set_next_buddy(se);
+       set_next_buddy(&task_rq(p)->cfs, se);
 
        yield_task_fair(rq);
 
@@ -9762,15 +9676,10 @@ static inline long migrate_degrades_loca
  */
 static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_cpu)
 {
-       struct cfs_rq *dst_cfs_rq;
+       struct cfs_rq *dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
-       dst_cfs_rq = task_group(p)->cfs_rq[dest_cpu];
-#else
-       dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
-#endif
-       if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_queued &&
-           !entity_eligible(task_cfs_rq(p), &p->se))
+       if (sched_feat(PLACE_LAG) && dst_cfs_rq->h_nr_queued &&
+           !entity_eligible(&task_rq(p)->cfs, &p->se))
                return 1;
 
        return 0;
@@ -10240,7 +10149,7 @@ static void update_cfs_rq_h_load(struct
        while ((se = READ_ONCE(cfs_rq->h_load_next)) != NULL) {
                load = cfs_rq->h_load;
                load = div64_ul(load * se->avg.load_avg,
-                       cfs_rq_load_avg(cfs_rq) + 1);
+                               cfs_rq_load_avg(cfs_rq) + 1);
                cfs_rq = group_cfs_rq(se);
                cfs_rq->h_load = load;
                cfs_rq->last_h_load_update = now;
@@ -13459,7 +13368,7 @@ static inline void task_tick_core(struct
         * MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check
         * if we need to give up the CPU.
         */
-       if (rq->core->core_forceidle_count && rq->cfs.nr_queued == 1 &&
+       if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued == 1 &&
            __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
                resched_curr(rq);
 }
@@ -13668,30 +13577,8 @@ bool cfs_prio_less(const struct task_str
 
        WARN_ON_ONCE(task_rq(b)->core != rq->core);
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
-       /*
-        * Find an se in the hierarchy for tasks a and b, such that the se's
-        * are immediate siblings.
-        */
-       while (sea->cfs_rq->tg != seb->cfs_rq->tg) {
-               int sea_depth = sea->depth;
-               int seb_depth = seb->depth;
-
-               if (sea_depth >= seb_depth)
-                       sea = parent_entity(sea);
-               if (sea_depth <= seb_depth)
-                       seb = parent_entity(seb);
-       }
-
-       se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
-       se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);
-
-       cfs_rqa = sea->cfs_rq;
-       cfs_rqb = seb->cfs_rq;
-#else /* !CONFIG_FAIR_GROUP_SCHED: */
        cfs_rqa = &task_rq(a)->cfs;
        cfs_rqb = &task_rq(b)->cfs;
-#endif /* !CONFIG_FAIR_GROUP_SCHED */
 
        /*
         * Find delta after normalizing se's vruntime with its cfs_rq's
@@ -13729,14 +13616,20 @@ static inline void task_tick_core(struct
  */
 static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 {
-       struct cfs_rq *cfs_rq;
        struct sched_entity *se = &curr->se;
+       unsigned long weight = NICE_0_LOAD;
+       struct cfs_rq *cfs_rq;
 
        for_each_sched_entity(se) {
                cfs_rq = cfs_rq_of(se);
                entity_tick(cfs_rq, se, queued);
+
+               weight = __calc_prop_weight(cfs_rq, se, weight);
        }
 
+       se = &curr->se;
+       reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+
        if (queued)
                return;
 
@@ -13772,7 +13665,7 @@ prio_changed_fair(struct rq *rq, struct
        if (p->prio == oldprio)
                return;
 
-       if (rq->cfs.nr_queued == 1)
+       if (rq->cfs.h_nr_queued == 1)
                return;
 
        /*
@@ -13901,29 +13794,40 @@ static void switched_to_fair(struct rq *
        }
 }
 
-/*
- * Account for a task changing its policy or group.
- *
- * This routine is mostly called to set cfs_rq->curr field when a task
- * migrates between groups/classes.
- */
 static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 {
        struct sched_entity *se = &p->se;
+       struct cfs_rq *cfs_rq = &rq->cfs;
+       unsigned long weight = NICE_0_LOAD;
+       bool on_rq = se->on_rq;
+
+       clear_buddies(cfs_rq, se);
+
+       if (on_rq)
+               __dequeue_entity(cfs_rq, se);
 
        for_each_sched_entity(se) {
-               struct cfs_rq *cfs_rq = cfs_rq_of(se);
+               cfs_rq = cfs_rq_of(se);
 
-               if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
-                   first && cfs_rq->curr)
-                       break;
+               if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) ||
+                   !first || !cfs_rq->h_curr)
+                       set_next_entity(cfs_rq, se);
 
-               set_next_entity(cfs_rq, se, first);
                /* ensure bandwidth has been allocated on our new cfs_rq */
                account_cfs_rq_runtime(cfs_rq, 0);
+
+               if (on_rq)
+                       weight = __calc_prop_weight(cfs_rq, se, weight);
        }
 
        se = &p->se;
+       cfs_rq->curr = se;
+
+       if (on_rq) {
+               reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+               if (first)
+                       set_protect_slice(cfs_rq, se);
+       }
 
        if (task_on_rq_queued(p)) {
                /*
@@ -14054,17 +13958,8 @@ void unregister_fair_sched_group(struct
                struct sched_entity *se = tg->se[cpu];
                struct rq *rq = cpu_rq(cpu);
 
-               if (se) {
-                       if (se->sched_delayed) {
-                               guard(rq_lock_irqsave)(rq);
-                               if (se->sched_delayed) {
-                                       update_rq_clock(rq);
-                                       dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-                               }
-                               list_del_leaf_cfs_rq(cfs_rq);
-                       }
+               if (se)
                        remove_entity_load_avg(se);
-               }
 
                /*
                 * Only empty task groups can be destroyed; so we can speculatively
kernel/sched/pelt.c +3 -3
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched
        /*
         * running is a subset of runnable (weight) so running can't be set if
         * runnable is clear. But there are some corner cases where the current
-        * se has been already dequeued but cfs_rq->curr still points to it.
+        * se has been already dequeued but cfs_rq->h_curr still points to it.
         * This means that weight will be 0 but not running for a sched_entity
         * but also for a cfs_rq if the latter becomes idle. As an example,
         * this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now
 int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
        if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
-                               cfs_rq->curr == se)) {
+                               cfs_rq->h_curr == se)) {
 
                ___update_load_avg(&se->avg, se_weight(se));
                cfs_se_util_change(&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, st
        if (___update_load_sum(now, &cfs_rq->avg,
                                scale_load_down(cfs_rq->load.weight),
                                cfs_rq->h_nr_runnable,
-                               cfs_rq->curr != NULL)) {
+                               cfs_rq->h_curr != NULL)) {
 
                ___update_load_avg(&cfs_rq->avg, 1);
                trace_pelt_cfs_tp(cfs_rq);
kernel/sched/sched.h +13 -13
@@ -528,21 +528,8 @@ struct task_group {
 
 };
 
-#ifdef CONFIG_GROUP_SCHED_WEIGHT
 #define ROOT_TASK_GROUP_LOAD   NICE_0_LOAD
 
-/*
- * A weight of 0 or 1 can cause arithmetics problems.
- * A weight of a cfs_rq is the sum of weights of which entities
- * are queued on this cfs_rq, so a weight of a entity should not be
- * too large, so as the shares value of a task group.
- * (The default weight is 1024 - so there's no practical
- *  limitation from this.)
- */
-#define MIN_SHARES             (1UL <<  1)
-#define MAX_SHARES             (1UL << 18)
-#endif
-
 typedef int (*tg_visitor)(struct task_group *, void *);
 
 extern int walk_tg_tree_from(struct task_group *from,
@@ -629,6 +616,17 @@ static inline bool cfs_task_bw_constrain
 
 #endif /* !CONFIG_CGROUP_SCHED */
 
+/*
+ * A weight of 0 or 1 can cause arithmetics problems.
+ * A weight of a cfs_rq is the sum of weights of which entities
+ * are queued on this cfs_rq, so a weight of a entity should not be
+ * too large, so as the shares value of a task group.
+ * (The default weight is 1024 - so there's no practical
+ *  limitation from this.)
+ */
+#define MIN_SHARES             (1UL <<  1)
+#define MAX_SHARES             (1UL << 18)
+
 extern void unregister_rt_sched_group(struct task_group *tg);
 extern void free_rt_sched_group(struct task_group *tg);
 extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
@@ -707,6 +705,7 @@ struct cfs_rq {
        /*
         * CFS load tracking
         */
+       struct sched_entity     *h_curr;
        struct sched_avg        avg;
 #ifndef CONFIG_64BIT
        u64                     last_update_time_copy;
@@ -2509,6 +2508,7 @@ extern const u32          sched_prio_to_wmult[40
 #define ENQUEUE_MIGRATED       0x00040000
 #define ENQUEUE_INITIAL                0x00080000
 #define ENQUEUE_RQ_SELECTED    0x00100000
+#define ENQUEUE_QUEUED         0x00200000
 
 #define RETRY_TASK             ((void *)-1UL)
 
maintainer
#34 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 50 lines · Vincent Guittot
Quoted 49 lines · Peter Zijlstra

Change fair/cgroup to a single runqueue.

Infamously fair/cgroup isn't working for a number of people; typically the complaint is latencies and/or overhead. The latency issue is due to the intermediate entries that represent a combination of tasks and thereby obfuscate the runnability of tasks.

The approach here is to leave the cgroup hierarchy as is; including the intermediate enqueue/dequeue but move the actual EEVDF runqueue outside. This means things like the shares_weight approximation are fully preserved.

That is, given a hierarchy like:

        R
        |
        se--G1
            / \
      G2--se   se--G3
     / \           |

T1--se se--T2 se--T3

This is fully maintained for load tracking, however the EEVDF parts of cfs_rq/se go unused for the intermediates and are instead connected like:

     _R_
    / | \

T1 T2 T3

Since the effective weight of the entities is determined by the hierarchy, this gets recomputed on enqueue,set_next_task and tick.

Notably, the effective weight (se->h_load) is computed from the hierarchical fraction: se->load / cfs_rq->load.

Since EEVDF is now exclusive operating on rq->cfs, it needs to consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly, only tasks can get delayed, simplifying some of the cgroup cleanup.

One place where additional information was required was set_next_task() / put_prev_task(), where we need to track 'current' both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense (cfs_rq->curr).

As a result of only having a single level to pick from, much of the complications in pick_next_task() and preemption go away.

Quoted 6 lines
Quoted 6 lines

Since many of the hierarchical operations are still there, this won't immediately fix the performance issues, but hopefully it will fix some of the latency issues.

TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr

I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed update_entity_lag():

update_entity_lag must be called after updating curr and cfs_rd and before clearing on_rq

With the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet

---
 kernel/sched/fair.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/sched/fair.c +4 -1
index 77d0e1937f2c..32fe57004f27 100644
@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
 	update_stats_dequeue_fair(cfs_rq, se, flags);
 
+	if (entity_is_task(se))
+		update_entity_lag(&rq_of(cfs_rq)->cfs, se);
+
 	se->on_rq = 0;
 	account_entity_dequeue(cfs_rq, se);
 
@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 		if (sched_feat(DELAY_DEQUEUE) && delay &&
 		    !entity_eligible(cfs_rq, se)) {
 			update_load_avg(cfs_rq_of(se), se, 0);
+			update_entity_lag(cfs_rq, se);
 			set_delayed(se);
 			return false;
 		}
@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
 	dequeue_hierarchy(p, flags);
 
-	update_entity_lag(cfs_rq, se);
 	if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
 		se->deadline -= se->vruntime;
 		se->rel_deadline = 1;
Signature · 1 line
2.43.0
Quoted 1,639 lines
Quoted 1,113 lines
---
 include/linux/sched.h |    1
 kernel/sched/core.c   |    5
 kernel/sched/debug.c  |    9
 kernel/sched/fair.c   |  789 +++++++++++++++++++++-----------------------------
 kernel/sched/pelt.c   |    6
 kernel/sched/sched.h  |   26 -
 6 files changed, 366 insertions(+), 470 deletions(-)
include/linux/sched.h +1 -0
@@ -575,6 +575,7 @@ struct sched_statistics {
 struct sched_entity {
        /* For load-balancing: */
        struct load_weight              load;
+       struct load_weight              h_load;
        struct rb_node                  run_node;
        u64                             deadline;
        u64                             min_vruntime;
kernel/sched/core.c +1 -4
@@ -5539,11 +5539,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);
  */
 static inline void prefetch_curr_exec_start(struct task_struct *p)
 {
-#ifdef CONFIG_FAIR_GROUP_SCHED
-       struct sched_entity *curr = p->se.cfs_rq->curr;
-#else
        struct sched_entity *curr = task_rq(p)->cfs.curr;
-#endif
+
        prefetch(curr);
        prefetch(&curr->exec_start);
 }
kernel/sched/debug.c +6 -3
@@ -911,10 +911,11 @@ print_task(struct seq_file *m, struct rq
        else
                SEQ_printf(m, " %c", task_state_to_char(p));
 
-       SEQ_printf(m, " %15s %5d %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
+       SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld   %c   %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld   %5d ",
                p->comm, task_pid_nr(p),
+               p->se.h_load.weight,
                SPLIT_NS(p->se.vruntime),
-               entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
+               entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N',
                SPLIT_NS(p->se.deadline),
                p->se.custom_slice ? 'S' : ' ',
                SPLIT_NS(p->se.slice),
@@ -943,7 +944,7 @@ static void print_rq(struct seq_file *m,
 
        SEQ_printf(m, "\n");
        SEQ_printf(m, "runnable tasks:\n");
-       SEQ_printf(m, " S            task   PID       vruntime   eligible    "
+       SEQ_printf(m, " S            task   PID     weight       vruntime   eligible    "
                   "deadline             slice          sum-exec      switches  "
                   "prio         wait-time        sum-sleep       sum-block"
 #ifdef CONFIG_NUMA_BALANCING
@@ -1051,6 +1052,8 @@ void print_cfs_rq(struct seq_file *m, in
                        cfs_rq->tg_load_avg_contrib);
        SEQ_printf(m, "  .%-30s: %ld\n", "tg_load_avg",
                        atomic_long_read(&cfs_rq->tg->load_avg));
+       SEQ_printf(m, "  .%-30s: %lu\n", "h_load",
+                       cfs_rq->h_load);
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 #ifdef CONFIG_CFS_BANDWIDTH
        SEQ_printf(m, "  .%-30s: %d\n", "throttled",
kernel/sched/fair.c +249 -316
@@ -296,8 +296,8 @@ static u64 __calc_delta(u64 delta_exec,
  */
 static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)
 {
-       if (unlikely(se->load.weight != NICE_0_LOAD))
-               delta = __calc_delta(delta, NICE_0_LOAD, &se->load);
+       if (se->h_load.weight != NICE_0_LOAD)
+               delta = __calc_delta(delta, NICE_0_LOAD, &se->h_load);
 
        return delta;
 }
@@ -427,38 +427,6 @@ static inline struct sched_entity *paren
        return se->parent;
 }
 
-static void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-       int se_depth, pse_depth;
-
-       /*
-        * preemption test can be made between sibling entities who are in the
-        * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
-        * both tasks until we find their ancestors who are siblings of common
-        * parent.
-        */
-
-       /* First walk up until both entities are at same depth */
-       se_depth = (*se)->depth;
-       pse_depth = (*pse)->depth;
-
-       while (se_depth > pse_depth) {
-               se_depth--;
-               *se = parent_entity(*se);
-       }
-
-       while (pse_depth > se_depth) {
-               pse_depth--;
-               *pse = parent_entity(*pse);
-       }
-
-       while (!is_same_group(*se, *pse)) {
-               *se = parent_entity(*se);
-               *pse = parent_entity(*pse);
-       }
-}
-
 static int tg_is_idle(struct task_group *tg)
 {
        return tg->idle > 0;
@@ -502,11 +470,6 @@ static inline struct sched_entity *paren
        return NULL;
 }
 
-static inline void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-}
-
 static inline int tg_is_idle(struct task_group *tg)
 {
        return 0;
@@ -685,7 +648,7 @@ static inline unsigned long avg_vruntime
 static inline void
 __sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+       unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
        s64 w_vruntime, key = entity_key(cfs_rq, se);
 
        w_vruntime = key * weight;
@@ -702,7 +665,7 @@ sum_w_vruntime_add_paranoid(struct cfs_r
        s64 key, tmp;
 
 again:
-       weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+       weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
        key = entity_key(cfs_rq, se);
 
        if (check_mul_overflow(key, weight, &key))
@@ -748,7 +711,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq
 static void
 sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+       unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
        s64 key = entity_key(cfs_rq, se);
 
        cfs_rq->sum_w_vruntime -= key * weight;
@@ -790,7 +753,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)
                s64 runtime = cfs_rq->sum_w_vruntime;
 
                if (curr) {
-                       unsigned long w = avg_vruntime_weight(cfs_rq, curr->load.weight);
+                       unsigned long w = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
                        runtime += entity_key(cfs_rq, curr) * w;
                        weight += w;
@@ -861,8 +824,6 @@ bool update_entity_lag(struct cfs_rq *cf
        u64 avruntime = avg_vruntime(cfs_rq);
        s64 vlag = entity_lag(cfs_rq, se, avruntime);
 
-       WARN_ON_ONCE(!se->on_rq);
-
        if (se->sched_delayed) {
                /* previous vlag < 0 otherwise se would not be delayed */
                vlag = max(vlag, se->vlag);
@@ -898,7 +859,7 @@ static int vruntime_eligible(struct cfs_
        long load = cfs_rq->sum_weight;
 
        if (curr && curr->on_rq) {
-               unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight);
+               unsigned long weight = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
                avg += entity_key(cfs_rq, curr) * weight;
                load += weight;
@@ -1039,6 +1000,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim
  */
 static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+       WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+       WARN_ON_ONCE(!entity_is_task(se));
+
        sum_w_vruntime_add(cfs_rq, se);
        se->min_vruntime = se->vruntime;
        se->min_slice = se->slice;
@@ -1048,6 +1012,9 @@ static void __enqueue_entity(struct cfs_
 
 static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+       WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+       WARN_ON_ONCE(!entity_is_task(se));
+
        rb_erase_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
                                  &min_vruntime_cb);
        sum_w_vruntime_sub(cfs_rq, se);
@@ -1144,7 +1111,7 @@ static struct sched_entity *pick_eevdf(s
         * We can safely skip eligibility check if there is only one entity
         * in this cfs_rq, saving some cycles.
         */
-       if (cfs_rq->nr_queued == 1)
+       if (cfs_rq->h_nr_queued == 1)
                return curr && curr->on_rq ? curr : se;
 
        /*
@@ -1391,8 +1358,6 @@ static s64 update_se(struct rq *rq, stru
        return delta_exec;
 }
 
-static void set_next_buddy(struct sched_entity *se);
-
 /*
  * Used by other classes to account runtime.
  */
@@ -1412,7 +1377,7 @@ static void update_curr(struct cfs_rq *c
         * not necessarily be the actual task running
         * (rq->curr.se). This is easy to confuse!
         */
-       struct sched_entity *curr = cfs_rq->curr;
+       struct sched_entity *curr = cfs_rq->h_curr;
        struct rq *rq = rq_of(cfs_rq);
        s64 delta_exec;
        bool resched;
@@ -1424,26 +1389,29 @@ static void update_curr(struct cfs_rq *c
        if (unlikely(delta_exec <= 0))
                return;
 
+       account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+       if (!entity_is_task(curr))
+               return;
+
+       cfs_rq = &rq->cfs;
+
        curr->vruntime += calc_delta_fair(delta_exec, curr);
        resched = update_deadline(cfs_rq, curr);
 
-       if (entity_is_task(curr)) {
-               /*
-                * If the fair_server is active, we need to account for the
-                * fair_server time whether or not the task is running on
-                * behalf of fair_server or not:
-                *  - If the task is running on behalf of fair_server, we need
-                *    to limit its time based on the assigned runtime.
-                *  - Fair task that runs outside of fair_server should account
-                *    against fair_server such that it can account for this time
-                *    and possibly avoid running this period.
-                */
-               dl_server_update(&rq->fair_server, delta_exec);
-       }
-
-       account_cfs_rq_runtime(cfs_rq, delta_exec);
+       /*
+        * If the fair_server is active, we need to account for the
+        * fair_server time whether or not the task is running on
+        * behalf of fair_server or not:
+        *  - If the task is running on behalf of fair_server, we need
+        *    to limit its time based on the assigned runtime.
+        *  - Fair task that runs outside of fair_server should account
+        *    against fair_server such that it can account for this time
+        *    and possibly avoid running this period.
+        */
+       dl_server_update(&rq->fair_server, delta_exec);
 
-       if (cfs_rq->nr_queued == 1)
+       if (cfs_rq->h_nr_queued == 1)
                return;
 
        if (resched || !protect_slice(curr)) {
@@ -1454,7 +1422,10 @@ static void update_curr(struct cfs_rq *c
 
 static void update_curr_fair(struct rq *rq)
 {
-       update_curr(cfs_rq_of(&rq->donor->se));
+       struct sched_entity *se = &rq->donor->se;
+
+       for_each_sched_entity(se)
+               update_curr(cfs_rq_of(se));
 }
 
 static inline void
@@ -1530,7 +1501,7 @@ update_stats_enqueue_fair(struct cfs_rq
         * Are we enqueueing a waiting task? (for current tasks
         * a dequeue/enqueue event is a NOP)
         */
-       if (se != cfs_rq->curr)
+       if (se != cfs_rq->h_curr)
                update_stats_wait_start_fair(cfs_rq, se);
 
        if (flags & ENQUEUE_WAKEUP)
@@ -1548,7 +1519,7 @@ update_stats_dequeue_fair(struct cfs_rq
         * Mark the end of the wait period if dequeueing a
         * waiting task:
         */
-       if (se != cfs_rq->curr)
+       if (se != cfs_rq->h_curr)
                update_stats_wait_end_fair(cfs_rq, se);
 
        if ((flags & DEQUEUE_SLEEP) && entity_is_task(se)) {
@@ -3875,6 +3846,7 @@ static inline void update_scan_period(st
 static void
 account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+       WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
        update_load_add(&cfs_rq->load, se->load.weight);
        if (entity_is_task(se)) {
                struct rq *rq = rq_of(cfs_rq);
@@ -3888,6 +3860,7 @@ account_entity_enqueue(struct cfs_rq *cf
 static void
 account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+       WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
        update_load_sub(&cfs_rq->load, se->load.weight);
        if (entity_is_task(se)) {
                account_numa_dequeue(rq_of(cfs_rq), task_of(se));
@@ -3965,7 +3938,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,
 static void
 rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vprot)
 {
-       unsigned long old_weight = se->load.weight;
+       long old_weight = se->h_load.weight;
 
        /*
         * VRUNTIME
@@ -4065,16 +4038,17 @@ rescale_entity(struct sched_entity *se,
                se->vprot = div64_long(se->vprot * old_weight, weight);
 }
 
-static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
-                           unsigned long weight)
+static void reweight_eevdf(struct cfs_rq *cfs_rq, struct sched_entity *se,
+                          unsigned long weight, bool on_rq)
 {
        bool curr = cfs_rq->curr == se;
        bool rel_vprot = false;
        u64 avruntime = 0;
 
-       if (se->on_rq) {
-               /* commit outstanding execution time */
-               update_curr(cfs_rq);
+       if (se->h_load.weight == weight)
+               return;
+
+       if (on_rq) {
                avruntime = avg_vruntime(cfs_rq);
                se->vlag = entity_lag(cfs_rq, se, avruntime);
                se->deadline -= avruntime;
@@ -4084,46 +4058,90 @@ static void reweight_entity(struct cfs_r
                        rel_vprot = true;
                }
 
-               cfs_rq->nr_queued--;
+               cfs_rq->h_nr_queued--;
                if (!curr)
                        __dequeue_entity(cfs_rq, se);
-               update_load_sub(&cfs_rq->load, se->load.weight);
        }
-       dequeue_load_avg(cfs_rq, se);
 
        rescale_entity(se, weight, rel_vprot);
 
-       update_load_set(&se->load, weight);
+       update_load_set(&se->h_load, weight);
 
-       do {
-               u32 divider = get_pelt_divider(&se->avg);
-               se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
-       } while (0);
-
-       enqueue_load_avg(cfs_rq, se);
-       if (se->on_rq) {
+       if (on_rq) {
                if (rel_vprot)
                        se->vprot += avruntime;
                se->deadline += avruntime;
                se->rel_deadline = 0;
                se->vruntime = avruntime - se->vlag;
 
-               update_load_add(&cfs_rq->load, se->load.weight);
                if (!curr)
                        __enqueue_entity(cfs_rq, se);
-               cfs_rq->nr_queued++;
+               cfs_rq->h_nr_queued++;
        }
 }
 
+static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
+                           unsigned long weight)
+{
+       if (se->load.weight == weight)
+               return;
+
+       if (se->on_rq) {
+               WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
+               update_load_sub(&cfs_rq->load, se->load.weight);
+       }
+       dequeue_load_avg(cfs_rq, se);
+
+       update_load_set(&se->load, weight);
+
+       do {
+               u32 divider = get_pelt_divider(&se->avg);
+               se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
+       } while (0);
+
+       enqueue_load_avg(cfs_rq, se);
+
+       if (se->on_rq)
+               update_load_add(&cfs_rq->load, se->load.weight);
+}
+
+/*
+ * weight = NICE_0_LOAD;
+ * for_each_entity_se(se)
+ *   weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
+ */
+static __always_inline
+unsigned long __calc_prop_weight(struct cfs_rq *cfs_rq, struct sched_entity *se,
+                                unsigned long weight)
+{
+       weight *= se->load.weight;
+       if (parent_entity(se))
+               weight /= cfs_rq->load.weight;
+       else
+               weight /= NICE_0_LOAD;
+
+       return max(weight, MIN_SHARES);
+}
+
 static void reweight_task_fair(struct rq *rq, struct task_struct *p,
                               const struct load_weight *lw)
 {
        struct sched_entity *se = &p->se;
-       struct cfs_rq *cfs_rq = cfs_rq_of(se);
-       struct load_weight *load = &se->load;
+       unsigned long weight = NICE_0_LOAD;
+
+       if (se->on_rq)
+               update_curr_fair(rq);
+
+       reweight_entity(cfs_rq_of(se), se, lw->weight);
+       se->load.inv_weight = lw->inv_weight;
+
+       if (!se->on_rq)
+               return;
+
+       for_each_sched_entity(se)
+               weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
 
-       reweight_entity(cfs_rq, se, lw->weight);
-       load->inv_weight = lw->inv_weight;
+       reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq);
 }
 
 static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
@@ -4331,7 +4349,6 @@ static long calc_group_shares(struct cfs
 static void update_cfs_group(struct sched_entity *se)
 {
        struct cfs_rq *gcfs_rq = group_cfs_rq(se);
-       long shares;
 
        /*
         * When a group becomes empty, preserve its weight. This matters for
@@ -4340,9 +4357,7 @@ static void update_cfs_group(struct sche
        if (!gcfs_rq || !gcfs_rq->load.weight)
                return;
 
-       shares = calc_group_shares(gcfs_rq);
-       if (unlikely(se->load.weight != shares))
-               reweight_entity(cfs_rq_of(se), se, shares);
+       reweight_entity(cfs_rq_of(se), se, calc_group_shares(gcfs_rq));
 }
 
 #else /* !CONFIG_FAIR_GROUP_SCHED: */
@@ -4460,7 +4475,7 @@ static inline bool cfs_rq_is_decayed(str
  * differential update where we store the last value we propagated. This in
  * turn allows skipping updates if the differential is 'small'.
  *
- * Updating tg's load_avg is necessary before update_cfs_share().
+ * Updating tg's load_avg is necessary before update_cfs_group().
  */
 static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
 {
@@ -4926,7 +4941,7 @@ static void migrate_se_pelt_lag(struct s
  * The cfs_rq avg is the direct sum of all its entities (blocked and runnable)
  * avg. The immediate corollary is that all (fair) tasks must be attached.
  *
- * cfs_rq->avg is used for task_h_load() and update_cfs_share() for example.
+ * cfs_rq->avg is used for task_h_load() and update_cfs_group() for example.
  *
  * Return: true if the load decayed or we removed load.
  *
@@ -5475,6 +5490,7 @@ static void
 place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
        u64 vslice, vruntime = avg_vruntime(cfs_rq);
+       unsigned int nr_queued = cfs_rq->h_nr_queued;
        bool update_zero = false;
        s64 lag = 0;
 
@@ -5482,6 +5498,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
                se->slice = sysctl_sched_base_slice;
        vslice = calc_delta_fair(se->slice, se);
 
+       if (flags & ENQUEUE_QUEUED)
+               nr_queued -= 1;
+
        /*
         * Due to how V is constructed as the weighted average of entities,
         * adding tasks with positive lag, or removing tasks with negative lag
@@ -5490,7 +5509,7 @@ place_entity(struct cfs_rq *cfs_rq, stru
         *
         * EEVDF: placement strategy #1 / #2
         */
-       if (sched_feat(PLACE_LAG) && cfs_rq->nr_queued && se->vlag) {
+       if (sched_feat(PLACE_LAG) && nr_queued && se->vlag) {
                struct sched_entity *curr = cfs_rq->curr;
                long load, weight;
 
@@ -5550,9 +5569,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
                 */
                load = cfs_rq->sum_weight;
                if (curr && curr->on_rq)
-                       load += avg_vruntime_weight(cfs_rq, curr->load.weight);
+                       load += avg_vruntime_weight(cfs_rq, curr->h_load.weight);
 
-               weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+               weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
                lag *= load + weight;
                if (WARN_ON_ONCE(!load))
                        load = 1;
@@ -5611,22 +5630,8 @@ static void check_enqueue_throttle(struc
 static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
 
 static void
-requeue_delayed_entity(struct sched_entity *se);
-
-static void
 enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
-       bool curr = cfs_rq->curr == se;
-
-       /*
-        * If we're the current task, we must renormalise before calling
-        * update_curr().
-        */
-       if (curr)
-               place_entity(cfs_rq, se, flags);
-
-       update_curr(cfs_rq);
-
        /*
         * When enqueuing a sched_entity, we must:
         *   - Update loads to have both entity and cfs_rq synced with now.
@@ -5645,13 +5650,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
         */
        update_cfs_group(se);
 
-       /*
-        * XXX now that the entity has been re-weighted, and it's lag adjusted,
-        * we can place the entity.
-        */
-       if (!curr)
-               place_entity(cfs_rq, se, flags);
-
        account_entity_enqueue(cfs_rq, se);
 
        /* Entity has migrated, no longer consider this task hot */
@@ -5660,8 +5658,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
 
        check_schedstat_required();
        update_stats_enqueue_fair(cfs_rq, se, flags);
-       if (!curr)
-               __enqueue_entity(cfs_rq, se);
        se->on_rq = 1;
 
        if (cfs_rq->nr_queued == 1) {
@@ -5679,21 +5675,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
        }
 }
 
-static void __clear_buddies_next(struct sched_entity *se)
+static void set_next_buddy(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       for_each_sched_entity(se) {
-               struct cfs_rq *cfs_rq = cfs_rq_of(se);
-               if (cfs_rq->next != se)
-                       break;
-
-               cfs_rq->next = NULL;
-       }
+       if (WARN_ON_ONCE(!se->on_rq || se->sched_delayed))
+               return;
+       if (se_is_idle(se))
+               return;
+       cfs_rq->next = se;
 }
 
 static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
        if (cfs_rq->next == se)
-               __clear_buddies_next(se);
+               cfs_rq->next = NULL;
 }
 
 static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
@@ -5704,7 +5698,7 @@ static void set_delayed(struct sched_ent
 
        /*
         * Delayed se of cfs_rq have no tasks queued on them.
-        * Do not adjust h_nr_runnable since dequeue_entities()
+        * Do not adjust h_nr_runnable since __dequeue_task()
         * will account it for blocked tasks.
         */
        if (!entity_is_task(se))
@@ -5737,37 +5731,11 @@ static void clear_delayed(struct sched_e
        }
 }
 
-static bool
+static void
 dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 {
-       bool sleep = flags & DEQUEUE_SLEEP;
        int action = UPDATE_TG;
 
-       update_curr(cfs_rq);
-       clear_buddies(cfs_rq, se);
-
-       if (flags & DEQUEUE_DELAYED) {
-               WARN_ON_ONCE(!se->sched_delayed);
-       } else {
-               bool delay = sleep;
-               /*
-                * DELAY_DEQUEUE relies on spurious wakeups, special task
-                * states must not suffer spurious wakeups, excempt them.
-                */
-               if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
-                       delay = false;
-
-               WARN_ON_ONCE(delay && se->sched_delayed);
-
-               if (sched_feat(DELAY_DEQUEUE) && delay &&
-                   !entity_eligible(cfs_rq, se)) {
-                       update_load_avg(cfs_rq, se, 0);
-                       update_entity_lag(cfs_rq, se);
-                       set_delayed(se);
-                       return false;
-               }
-       }
-
        if (entity_is_task(se) && task_on_rq_migrating(task_of(se)))
                action |= DO_DETACH;
 
@@ -5785,14 +5753,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 
        update_stats_dequeue_fair(cfs_rq, se, flags);
 
-       update_entity_lag(cfs_rq, se);
-       if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
-               se->deadline -= se->vruntime;
-               se->rel_deadline = 1;
-       }
-
-       if (se != cfs_rq->curr)
-               __dequeue_entity(cfs_rq, se);
        se->on_rq = 0;
        account_entity_dequeue(cfs_rq, se);
 
@@ -5801,9 +5761,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
 
        update_cfs_group(se);
 
-       if (flags & DEQUEUE_DELAYED)
-               clear_delayed(se);
-
        if (cfs_rq->nr_queued == 0) {
                update_idle_cfs_rq_clock_pelt(cfs_rq);
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -5816,15 +5773,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
                }
 #endif
        }
-
-       return true;
 }
 
 static void
-set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
+set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       clear_buddies(cfs_rq, se);
-
        /* 'current' is not kept within the tree. */
        if (se->on_rq) {
                /*
@@ -5833,16 +5786,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s
                 * runqueue.
                 */
                update_stats_wait_end_fair(cfs_rq, se);
-               __dequeue_entity(cfs_rq, se);
                update_load_avg(cfs_rq, se, UPDATE_TG);
-
-               if (first)
-                       set_protect_slice(cfs_rq, se);
        }
 
        update_stats_curr_start(cfs_rq, se);
-       WARN_ON_ONCE(cfs_rq->curr);
-       cfs_rq->curr = se;
+       WARN_ON_ONCE(cfs_rq->h_curr);
+       cfs_rq->h_curr = se;
 
        /*
         * Track our maximum slice length, if the CPU's load is at
@@ -5862,23 +5811,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s
        se->prev_sum_exec_runtime = se->sum_exec_runtime;
 }
 
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags);
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags);
 
-/*
- * Pick the next process, keeping these things in mind, in this order:
- * 1) keep things fair between processes/task groups
- * 2) pick the "next" process, since someone really wants that to run
- * 3) pick the "last" process, for cache locality
- * 4) do not run the "skip" process, if something else is available
- */
 static struct sched_entity *
-pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
+pick_next_entity(struct rq *rq, bool protect)
 {
+       struct cfs_rq *cfs_rq = &rq->cfs;
        struct sched_entity *se;
 
        se = pick_eevdf(cfs_rq, protect);
        if (se->sched_delayed) {
-               dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+               __dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED);
                /*
                 * Must not reference @se again, see __block_task().
                 */
@@ -5903,13 +5846,11 @@ static void put_prev_entity(struct cfs_r
 
        if (prev->on_rq) {
                update_stats_wait_start_fair(cfs_rq, prev);
-               /* Put 'current' back into the tree. */
-               __enqueue_entity(cfs_rq, prev);
                /* in !on_rq case, update occurred at dequeue */
                update_load_avg(cfs_rq, prev, 0);
        }
-       WARN_ON_ONCE(cfs_rq->curr != prev);
-       cfs_rq->curr = NULL;
+       WARN_ON_ONCE(cfs_rq->h_curr != prev);
+       cfs_rq->h_curr = NULL;
 }
 
 static void
@@ -6062,7 +6003,7 @@ static void __account_cfs_rq_runtime(str
         * if we're unable to extend our runtime we resched so that the active
         * hierarchy can be throttled
         */
-       if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->curr))
+       if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->h_curr))
                resched_curr(rq_of(cfs_rq));
 }
 
@@ -6420,7 +6361,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cf
        assert_list_leaf_cfs_rq(rq);
 
        /* Determine whether we need to wake up potentially idle CPU: */
-       if (rq->curr == rq->idle && rq->cfs.nr_queued)
+       if (rq->curr == rq->idle && rq->cfs.h_nr_queued)
                resched_curr(rq);
 }
 
@@ -6761,7 +6702,7 @@ static void check_enqueue_throttle(struc
                return;
 
        /* an active group must be handled by the update_curr()->put() path */
-       if (!cfs_rq->runtime_enabled || cfs_rq->curr)
+       if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)
                return;
 
        /* ensure the group is not already throttled */
@@ -7156,7 +7097,7 @@ static void hrtick_start_fair(struct rq
                        resched_curr(rq);
                return;
        }
-       delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+       delta = (se->h_load.weight * vdelta) / NICE_0_LOAD;
 
        /*
         * Correct for instantaneous load of other classes.
@@ -7256,10 +7197,8 @@ static int choose_idle_cpu(int cpu, stru
 }
 
 static void
-requeue_delayed_entity(struct sched_entity *se)
+requeue_delayed_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
-       struct cfs_rq *cfs_rq = cfs_rq_of(se);
-
        /*
         * se->sched_delayed should imply: se->on_rq == 1.
         * Because a delayed entity is one that is still on
@@ -7269,19 +7208,58 @@ requeue_delayed_entity(struct sched_enti
        WARN_ON_ONCE(!se->on_rq);
 
        if (update_entity_lag(cfs_rq, se)) {
-               cfs_rq->nr_queued--;
+               cfs_rq->h_nr_queued--;
                if (se != cfs_rq->curr)
                        __dequeue_entity(cfs_rq, se);
                place_entity(cfs_rq, se, 0);
                if (se != cfs_rq->curr)
                        __enqueue_entity(cfs_rq, se);
-               cfs_rq->nr_queued++;
+               cfs_rq->h_nr_queued++;
        }
 
        update_load_avg(cfs_rq, se, 0);
        clear_delayed(se);
 }
 
+static unsigned long enqueue_hierarchy(struct task_struct *p, int flags)
+{
+       unsigned long weight = NICE_0_LOAD;
+       int task_new = !(flags & ENQUEUE_WAKEUP);
+       struct sched_entity *se = &p->se;
+       int h_nr_idle = task_has_idle_policy(p);
+       int h_nr_runnable = 1;
+
+       if (task_new && se->sched_delayed)
+               h_nr_runnable = 0;
+
+       for_each_sched_entity(se) {
+               struct cfs_rq *cfs_rq = cfs_rq_of(se);
+
+               update_curr(cfs_rq);
+
+               if (!se->on_rq) {
+                       enqueue_entity(cfs_rq, se, flags);
+               } else {
+                       update_load_avg(cfs_rq, se, UPDATE_TG);
+                       se_update_runnable(se);
+                       update_cfs_group(se);
+               }
+
+               cfs_rq->h_nr_runnable += h_nr_runnable;
+               cfs_rq->h_nr_queued++;
+               cfs_rq->h_nr_idle += h_nr_idle;
+
+               if (cfs_rq_is_idle(cfs_rq))
+                       h_nr_idle = 1;
+
+               weight = __calc_prop_weight(cfs_rq, se, weight);
+
+               flags = ENQUEUE_WAKEUP;
+       }
+
+       return weight;
+}
+
 /*
  * The enqueue_task method is called before nr_running is
  * increased. Here we update the fair scheduling stats and
@@ -7290,13 +7268,12 @@ requeue_delayed_entity(struct sched_enti
 static void
 enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 {
-       struct cfs_rq *cfs_rq;
-       struct sched_entity *se = &p->se;
-       int h_nr_idle = task_has_idle_policy(p);
-       int h_nr_runnable = 1;
-       int task_new = !(flags & ENQUEUE_WAKEUP);
        int rq_h_nr_queued = rq->cfs.h_nr_queued;
-       u64 slice = 0;
+       int task_new = !(flags & ENQUEUE_WAKEUP);
+       struct sched_entity *se = &p->se;
+       struct cfs_rq *cfs_rq = &rq->cfs;
+       unsigned long weight;
+       bool curr;
 
        if (task_is_throttled(p) && enqueue_throttled_task(p))
                return;
@@ -7308,10 +7285,10 @@ enqueue_task_fair(struct rq *rq, struct
         * estimated utilization, before we update schedutil.
         */
        if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))
-               util_est_enqueue(&rq->cfs, p);
+               util_est_enqueue(cfs_rq, p);
 
        if (flags & ENQUEUE_DELAYED) {
-               requeue_delayed_entity(se);
+               requeue_delayed_entity(cfs_rq, se);
                return;
        }
 
@@ -7323,57 +7300,22 @@ enqueue_task_fair(struct rq *rq, struct
        if (p->in_iowait)
                cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT);
 
-       if (task_new && se->sched_delayed)
-               h_nr_runnable = 0;
-
-       for_each_sched_entity(se) {
-               if (se->on_rq) {
-                       if (se->sched_delayed)
-                               requeue_delayed_entity(se);
-                       break;
-               }
-               cfs_rq = cfs_rq_of(se);
-
-               /*
-                * Basically set the slice of group entries to the min_slice of
-                * their respective cfs_rq. This ensures the group can service
-                * its entities in the desired time-frame.
-                */
-               if (slice) {
-                       se->slice = slice;
-                       se->custom_slice = 1;
-               }
-               enqueue_entity(cfs_rq, se, flags);
-               slice = cfs_rq_min_slice(cfs_rq);
-
-               cfs_rq->h_nr_runnable += h_nr_runnable;
-               cfs_rq->h_nr_queued++;
-               cfs_rq->h_nr_idle += h_nr_idle;
-
-               if (cfs_rq_is_idle(cfs_rq))
-                       h_nr_idle = 1;
-
-               flags = ENQUEUE_WAKEUP;
-       }
-
-       for_each_sched_entity(se) {
-               cfs_rq = cfs_rq_of(se);
-
-               update_load_avg(cfs_rq, se, UPDATE_TG);
-               se_update_runnable(se);
-               update_cfs_group(se);
+       /*
+        * XXX comment on the curr thing
+        */
+       curr = (cfs_rq->curr == se);
+       if (curr)
+               place_entity(cfs_rq, se, flags);
 
-               se->slice = slice;
-               if (se != cfs_rq->curr)
-                       min_vruntime_cb_propagate(&se->run_node, NULL);
-               slice = cfs_rq_min_slice(cfs_rq);
+       if (se->on_rq && se->sched_delayed)
+               requeue_delayed_entity(cfs_rq, se);
 
-               cfs_rq->h_nr_runnable += h_nr_runnable;
-               cfs_rq->h_nr_queued++;
-               cfs_rq->h_nr_idle += h_nr_idle;
+       weight = enqueue_hierarchy(p, flags);
 
-               if (cfs_rq_is_idle(cfs_rq))
-                       h_nr_idle = 1;
+       if (!curr) {
+               reweight_eevdf(cfs_rq, se, weight, false);
+               place_entity(cfs_rq, se, flags | ENQUEUE_QUEUED);
+               __enqueue_entity(cfs_rq, se);
        }
 
        if (!rq_h_nr_queued && rq->cfs.h_nr_queued)
@@ -7404,105 +7346,107 @@ enqueue_task_fair(struct rq *rq, struct
        hrtick_update(rq);
 }
 
-/*
- * Basically dequeue_task_fair(), except it can deal with dequeue_entity()
- * failing half-way through and resume the dequeue later.
- *
- * Returns:
- * -1 - dequeue delayed
- *  0 - dequeue throttled
- *  1 - dequeue complete
- */
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
+static void dequeue_hierarchy(struct task_struct *p, int flags)
 {
-       bool was_sched_idle = sched_idle_rq(rq);
+       struct sched_entity *se = &p->se;
        bool task_sleep = flags & DEQUEUE_SLEEP;
        bool task_delayed = flags & DEQUEUE_DELAYED;
        bool task_throttled = flags & DEQUEUE_THROTTLE;
-       struct task_struct *p = NULL;
-       int h_nr_idle = 0;
-       int h_nr_queued = 0;
        int h_nr_runnable = 0;
-       struct cfs_rq *cfs_rq;
-       u64 slice = 0;
+       int h_nr_idle = task_has_idle_policy(p);
+       bool dequeue = true;
 
-       if (entity_is_task(se)) {
-               p = task_of(se);
-               h_nr_queued = 1;
-               h_nr_idle = task_has_idle_policy(p);
-               if (task_sleep || task_delayed || !se->sched_delayed)
-                       h_nr_runnable = 1;
-       }
+       if (task_sleep || task_delayed || !se->sched_delayed)
+               h_nr_runnable = 1;
 
        for_each_sched_entity(se) {
-               cfs_rq = cfs_rq_of(se);
+               struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
-               if (!dequeue_entity(cfs_rq, se, flags)) {
-                       if (p && &p->se == se)
-                               return -1;
+               update_curr(cfs_rq);
 
-                       slice = cfs_rq_min_slice(cfs_rq);
-                       break;
+               if (dequeue) {
+                       dequeue_entity(cfs_rq, se, flags);
+                       /* Don't dequeue parent if it has other entities besides us */
+                       if (cfs_rq->load.weight)
+                               dequeue = false;
+               } else {
+                       update_load_avg(cfs_rq, se, UPDATE_TG);
+                       se_update_runnable(se);
+                       update_cfs_group(se);
                }
 
                cfs_rq->h_nr_runnable -= h_nr_runnable;
-               cfs_rq->h_nr_queued -= h_nr_queued;
+               cfs_rq->h_nr_queued--;
                cfs_rq->h_nr_idle -= h_nr_idle;
 
                if (cfs_rq_is_idle(cfs_rq))
-                       h_nr_idle = h_nr_queued;
+                       h_nr_idle = 1;
 
                if (throttled_hierarchy(cfs_rq) && task_throttled)
                        record_throttle_clock(cfs_rq);
 
-               /* Don't dequeue parent if it has other entities besides us */
-               if (cfs_rq->load.weight) {
-                       slice = cfs_rq_min_slice(cfs_rq);
-
-                       /* Avoid re-evaluating load for this entity: */
-                       se = parent_entity(se);
-                       /*
-                        * Bias pick_next to pick a task from this cfs_rq, as
-                        * p is sleeping when it is within its sched_slice.
-                        */
-                       if (task_sleep && se)
-                               set_next_buddy(se);
-                       break;
-               }
                flags |= DEQUEUE_SLEEP;
                flags &= ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL);
        }
+}
 
-       for_each_sched_entity(se) {
-               cfs_rq = cfs_rq_of(se);
+/*
+ * The part of dequeue_task_fair() that is needed to dequeue delayed tasks.
+ *
+ * Returns:
+ *   true  - dequeued
+ *   false - delayed
+ */
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
+{
+       struct sched_entity *se = &p->se;
+       struct cfs_rq *cfs_rq = &rq->cfs;
+       bool was_sched_idle = sched_idle_rq(rq);
+       bool task_sleep = flags & DEQUEUE_SLEEP;
+       bool task_delayed = flags & DEQUEUE_DELAYED;
 
-               update_load_avg(cfs_rq, se, UPDATE_TG);
-               se_update_runnable(se);
-               update_cfs_group(se);
+       clear_buddies(cfs_rq, se);
 
-               se->slice = slice;
-               if (se != cfs_rq->curr)
-                       min_vruntime_cb_propagate(&se->run_node, NULL);
-               slice = cfs_rq_min_slice(cfs_rq);
+       if (flags & DEQUEUE_DELAYED) {
+               WARN_ON_ONCE(!se->sched_delayed);
+       } else {
+               bool delay = task_sleep;
+               /*
+                * DELAY_DEQUEUE relies on spurious wakeups, special task
+                * states must not suffer spurious wakeups, excempt them.
+                */
+               if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
+                       delay = false;
 

kernel/sched/fair.c enqueue_task_fair(struct rq *rq, struct · line 7425

Quoted 10 lines
kernel/sched/fair.c +4 -5
-               cfs_rq->h_nr_runnable -= h_nr_runnable;
-               cfs_rq->h_nr_queued -= h_nr_queued;
-               cfs_rq->h_nr_idle -= h_nr_idle;
+               WARN_ON_ONCE(delay && se->sched_delayed);
 
-               if (cfs_rq_is_idle(cfs_rq))
-                       h_nr_idle = h_nr_queued;
+               if (sched_feat(DELAY_DEQUEUE) && delay &&
+                   !entity_eligible(cfs_rq, se)) {
+                       update_load_avg(cfs_rq_of(se), se, 0);

update_entity_lag(cfs_rq, se); is missing here. Unfortunately this doesn't fix my regression

Quoted 512 lines
kernel/sched/fair.c +89 -126
+                       set_delayed(se);
+                       return false;
+               }
+       }
 
-               if (throttled_hierarchy(cfs_rq) && task_throttled)
-                       record_throttle_clock(cfs_rq);
+       dequeue_hierarchy(p, flags);
+
+       update_entity_lag(cfs_rq, se);
+       if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
+               se->deadline -= se->vruntime;
+               se->rel_deadline = 1;
        }
+       if (se != cfs_rq->curr)
+               __dequeue_entity(cfs_rq, se);
 
-       sub_nr_running(rq, h_nr_queued);
+       sub_nr_running(rq, 1);
 
        /* balance early to pull high priority tasks */
        if (unlikely(!was_sched_idle && sched_idle_rq(rq)))
                rq->next_balance = jiffies;
 
-       if (p && task_delayed) {
+       if (task_delayed) {
+               clear_delayed(se);
+
                WARN_ON_ONCE(!task_sleep);
                WARN_ON_ONCE(p->on_rq != 1);
 
@@ -7514,7 +7458,7 @@ static int dequeue_entities(struct rq *r
                __block_task(rq, p);
        }
 
-       return 1;
+       return true;
 }
 
 /*
@@ -7533,11 +7477,11 @@ static bool dequeue_task_fair(struct rq
                util_est_dequeue(&rq->cfs, p);
 
        util_est_update(&rq->cfs, p, flags & DEQUEUE_SLEEP);
-       if (dequeue_entities(rq, &p->se, flags) < 0)
+       if (!__dequeue_task(rq, p, flags))
                return false;
 
        /*
-        * Must not reference @p after dequeue_entities(DEQUEUE_DELAYED).
+        * Must not reference @p after __dequeue_task(DEQUEUE_DELAYED).
         */
        return true;
 }
@@ -9021,19 +8965,6 @@ static void migrate_task_rq_fair(struct
 static void task_dead_fair(struct task_struct *p)
 {
        struct sched_entity *se = &p->se;
-
-       if (se->sched_delayed) {
-               struct rq_flags rf;
-               struct rq *rq;
-
-               rq = task_rq_lock(p, &rf);
-               if (se->sched_delayed) {
-                       update_rq_clock(rq);
-                       dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-               }
-               task_rq_unlock(rq, p, &rf);
-       }
-
        remove_entity_load_avg(se);
 }
 
@@ -9067,21 +8998,10 @@ static void set_cpus_allowed_fair(struct
        set_task_max_allowed_capacity(p);
 }
 
-static void set_next_buddy(struct sched_entity *se)
-{
-       for_each_sched_entity(se) {
-               if (WARN_ON_ONCE(!se->on_rq))
-                       return;
-               if (se_is_idle(se))
-                       return;
-               cfs_rq_of(se)->next = se;
-       }
-}
-
 enum preempt_wakeup_action {
        PREEMPT_WAKEUP_NONE,    /* No preemption. */
        PREEMPT_WAKEUP_SHORT,   /* Ignore slice protection. */
-       PREEMPT_WAKEUP_PICK,    /* Let __pick_eevdf() decide. */
+       PREEMPT_WAKEUP_PICK,    /* Let pick_eevdf() decide. */
        PREEMPT_WAKEUP_RESCHED, /* Force reschedule. */
 };
 
@@ -9098,7 +9018,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,
        if (cfs_rq->next && entity_before(cfs_rq->next, pse))
                return false;
 
-       set_next_buddy(pse);
+       set_next_buddy(cfs_rq, pse);
        return true;
 }
 
@@ -9188,7 +9108,6 @@ static void wakeup_preempt_fair(struct r
        if (!sched_feat(WAKEUP_PREEMPTION))
                return;
 
-       find_matching_se(&se, &pse);
        WARN_ON_ONCE(!pse);
 
        cse_is_idle = se_is_idle(se);
@@ -9216,8 +9135,7 @@ static void wakeup_preempt_fair(struct r
        if (unlikely(!normal_policy(p->policy)))
                return;
 
-       cfs_rq = cfs_rq_of(se);
-       update_curr(cfs_rq);
+       update_curr_fair(rq);
        /*
         * If @p has a shorter slice than current and @p is eligible, override
         * current's slice protection in order to allow preemption.
@@ -9261,18 +9179,15 @@ static void wakeup_preempt_fair(struct r
        }
 
 pick:
-       nse = pick_next_entity(rq, cfs_rq, preempt_action != PREEMPT_WAKEUP_SHORT);
-       /* If @p has become the most eligible task, force preemption */
-       if (nse == pse)
-               goto preempt;
-
-       /*
-        * Because p is enqueued, nse being null can only mean that we
-        * dequeued a delayed task. If there are still entities queued in
-        * cfs, check if the next one will be p.
-        */
-       if (!nse && cfs_rq->nr_queued)
-               goto pick;
+       if (cfs_rq->h_nr_queued) {
+               nse = pick_next_entity(rq, preempt_action != PREEMPT_WAKEUP_SHORT);
+               if (unlikely(!nse))
+                       goto pick;
+
+               /* If @p has become the most eligible task, force preemption */
+               if (nse == pse)
+                       goto preempt;
+       }
 
        if (sched_feat(RUN_TO_PARITY))
                update_protect_slice(cfs_rq, se);
@@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r
 struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
        __must_hold(__rq_lockp(rq))
 {
+       struct cfs_rq *cfs_rq = &rq->cfs;
        struct sched_entity *se;
-       struct cfs_rq *cfs_rq;
        struct task_struct *p;
-       bool throttled;
        int new_tasks;
 
 again:
-       cfs_rq = &rq->cfs;
-       if (!cfs_rq->nr_queued)
+       if (!cfs_rq->h_nr_queued)
                goto idle;
 
-       throttled = false;
-
-       do {
-               /* Might not have done put_prev_entity() */
-               if (cfs_rq->curr && cfs_rq->curr->on_rq)
-                       update_curr(cfs_rq);
-
-               throttled |= check_cfs_rq_runtime(cfs_rq);
+       /* Might not have done put_prev_entity() */
+       if (cfs_rq->curr && cfs_rq->curr->on_rq)
+               update_curr(cfs_rq);
 
-               se = pick_next_entity(rq, cfs_rq, true);
-               if (!se)
-                       goto again;
-               cfs_rq = group_cfs_rq(se);
-       } while (cfs_rq);
+       se = pick_next_entity(rq, true);
+       if (!se)
+               goto again;
 
        p = task_of(se);
-       if (unlikely(throttled))
+       if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
                task_throttle_setup_work(p);
        return p;
 
@@ -9353,7 +9259,7 @@ void fair_server_init(struct rq *rq)
 static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct task_struct *next)
 {
        struct sched_entity *se = &prev->se;
-       struct cfs_rq *cfs_rq;
+       struct cfs_rq *cfs_rq = &rq->cfs;
        struct sched_entity *nse = NULL;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9363,7 +9269,7 @@ static void put_prev_task_fair(struct rq
 
        while (se) {
                cfs_rq = cfs_rq_of(se);
-               if (!nse || cfs_rq->curr)
+               if (!nse || cfs_rq->h_curr)
                        put_prev_entity(cfs_rq, se);
 #ifdef CONFIG_FAIR_GROUP_SCHED
                if (nse) {
@@ -9382,6 +9288,14 @@ static void put_prev_task_fair(struct rq
 #endif
                se = parent_entity(se);
        }
+
+       /* Put 'current' back into the tree. */
+       cfs_rq = &rq->cfs;
+       se = &prev->se;
+       WARN_ON_ONCE(cfs_rq->curr != se);
+       cfs_rq->curr = NULL;
+       if (se->on_rq)
+               __enqueue_entity(cfs_rq, se);
 }
 
 /*
@@ -9390,8 +9304,8 @@ static void put_prev_task_fair(struct rq
 static void yield_task_fair(struct rq *rq)
 {
        struct task_struct *curr = rq->donor;
-       struct cfs_rq *cfs_rq = task_cfs_rq(curr);
        struct sched_entity *se = &curr->se;
+       struct cfs_rq *cfs_rq = &rq->cfs;
 
        /*
         * Are we the only task in the tree?
@@ -9432,11 +9346,11 @@ static bool yield_to_task_fair(struct rq
        struct sched_entity *se = &p->se;
 
        /* !se->on_rq also covers throttled task */
-       if (!se->on_rq)
+       if (!se->on_rq || se->sched_delayed)
                return false;
 
        /* Tell the scheduler that we'd really like se to run next. */
-       set_next_buddy(se);
+       set_next_buddy(&task_rq(p)->cfs, se);
 
        yield_task_fair(rq);
 
@@ -9762,15 +9676,10 @@ static inline long migrate_degrades_loca
  */
 static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_cpu)
 {
-       struct cfs_rq *dst_cfs_rq;
+       struct cfs_rq *dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
-       dst_cfs_rq = task_group(p)->cfs_rq[dest_cpu];
-#else
-       dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
-#endif
-       if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_queued &&
-           !entity_eligible(task_cfs_rq(p), &p->se))
+       if (sched_feat(PLACE_LAG) && dst_cfs_rq->h_nr_queued &&
+           !entity_eligible(&task_rq(p)->cfs, &p->se))
                return 1;
 
        return 0;
@@ -10240,7 +10149,7 @@ static void update_cfs_rq_h_load(struct
        while ((se = READ_ONCE(cfs_rq->h_load_next)) != NULL) {
                load = cfs_rq->h_load;
                load = div64_ul(load * se->avg.load_avg,
-                       cfs_rq_load_avg(cfs_rq) + 1);
+                               cfs_rq_load_avg(cfs_rq) + 1);
                cfs_rq = group_cfs_rq(se);
                cfs_rq->h_load = load;
                cfs_rq->last_h_load_update = now;
@@ -13459,7 +13368,7 @@ static inline void task_tick_core(struct
         * MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check
         * if we need to give up the CPU.
         */
-       if (rq->core->core_forceidle_count && rq->cfs.nr_queued == 1 &&
+       if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued == 1 &&
            __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
                resched_curr(rq);
 }
@@ -13668,30 +13577,8 @@ bool cfs_prio_less(const struct task_str
 
        WARN_ON_ONCE(task_rq(b)->core != rq->core);
 
-#ifdef CONFIG_FAIR_GROUP_SCHED
-       /*
-        * Find an se in the hierarchy for tasks a and b, such that the se's
-        * are immediate siblings.
-        */
-       while (sea->cfs_rq->tg != seb->cfs_rq->tg) {
-               int sea_depth = sea->depth;
-               int seb_depth = seb->depth;
-
-               if (sea_depth >= seb_depth)
-                       sea = parent_entity(sea);
-               if (sea_depth <= seb_depth)
-                       seb = parent_entity(seb);
-       }
-
-       se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
-       se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);
-
-       cfs_rqa = sea->cfs_rq;
-       cfs_rqb = seb->cfs_rq;
-#else /* !CONFIG_FAIR_GROUP_SCHED: */
        cfs_rqa = &task_rq(a)->cfs;
        cfs_rqb = &task_rq(b)->cfs;
-#endif /* !CONFIG_FAIR_GROUP_SCHED */
 
        /*
         * Find delta after normalizing se's vruntime with its cfs_rq's
@@ -13729,14 +13616,20 @@ static inline void task_tick_core(struct
  */
 static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 {
-       struct cfs_rq *cfs_rq;
        struct sched_entity *se = &curr->se;
+       unsigned long weight = NICE_0_LOAD;
+       struct cfs_rq *cfs_rq;
 
        for_each_sched_entity(se) {
                cfs_rq = cfs_rq_of(se);
                entity_tick(cfs_rq, se, queued);
+
+               weight = __calc_prop_weight(cfs_rq, se, weight);
        }
 
+       se = &curr->se;
+       reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+
        if (queued)
                return;
 
@@ -13772,7 +13665,7 @@ prio_changed_fair(struct rq *rq, struct
        if (p->prio == oldprio)
                return;
 
-       if (rq->cfs.nr_queued == 1)
+       if (rq->cfs.h_nr_queued == 1)
                return;
 
        /*
@@ -13901,29 +13794,40 @@ static void switched_to_fair(struct rq *
        }
 }
 
-/*
- * Account for a task changing its policy or group.
- *
- * This routine is mostly called to set cfs_rq->curr field when a task
- * migrates between groups/classes.
- */
 static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
 {
        struct sched_entity *se = &p->se;
+       struct cfs_rq *cfs_rq = &rq->cfs;
+       unsigned long weight = NICE_0_LOAD;
+       bool on_rq = se->on_rq;
+
+       clear_buddies(cfs_rq, se);
+
+       if (on_rq)
+               __dequeue_entity(cfs_rq, se);
 
        for_each_sched_entity(se) {
-               struct cfs_rq *cfs_rq = cfs_rq_of(se);
+               cfs_rq = cfs_rq_of(se);
 
-               if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
-                   first && cfs_rq->curr)
-                       break;
+               if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) ||
+                   !first || !cfs_rq->h_curr)
+                       set_next_entity(cfs_rq, se);
 
-               set_next_entity(cfs_rq, se, first);
                /* ensure bandwidth has been allocated on our new cfs_rq */
                account_cfs_rq_runtime(cfs_rq, 0);
+
+               if (on_rq)
+                       weight = __calc_prop_weight(cfs_rq, se, weight);
        }
 
        se = &p->se;
+       cfs_rq->curr = se;
+
+       if (on_rq) {
+               reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+               if (first)
+                       set_protect_slice(cfs_rq, se);
+       }
 
        if (task_on_rq_queued(p)) {
                /*
@@ -14054,17 +13958,8 @@ void unregister_fair_sched_group(struct
                struct sched_entity *se = tg->se[cpu];
                struct rq *rq = cpu_rq(cpu);
 
-               if (se) {
-                       if (se->sched_delayed) {
-                               guard(rq_lock_irqsave)(rq);
-                               if (se->sched_delayed) {
-                                       update_rq_clock(rq);
-                                       dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-                               }
-                               list_del_leaf_cfs_rq(cfs_rq);
-                       }
+               if (se)
                        remove_entity_load_avg(se);
-               }
 
                /*
                 * Only empty task groups can be destroyed; so we can speculatively
kernel/sched/pelt.c +3 -3
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched
        /*
         * running is a subset of runnable (weight) so running can't be set if
         * runnable is clear. But there are some corner cases where the current
-        * se has been already dequeued but cfs_rq->curr still points to it.
+        * se has been already dequeued but cfs_rq->h_curr still points to it.
         * This means that weight will be 0 but not running for a sched_entity
         * but also for a cfs_rq if the latter becomes idle. As an example,
         * this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now
 int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
        if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
-                               cfs_rq->curr == se)) {
+                               cfs_rq->h_curr == se)) {
 
                ___update_load_avg(&se->avg, se_weight(se));
                cfs_se_util_change(&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, st
        if (___update_load_sum(now, &cfs_rq->avg,
                                scale_load_down(cfs_rq->load.weight),
                                cfs_rq->h_nr_runnable,
-                               cfs_rq->curr != NULL)) {
+                               cfs_rq->h_curr != NULL)) {
 
                ___update_load_avg(&cfs_rq->avg, 1);
                trace_pelt_cfs_tp(cfs_rq);
kernel/sched/sched.h +13 -13
@@ -528,21 +528,8 @@ struct task_group {
 
 };
 
-#ifdef CONFIG_GROUP_SCHED_WEIGHT
 #define ROOT_TASK_GROUP_LOAD   NICE_0_LOAD
 
-/*
- * A weight of 0 or 1 can cause arithmetics problems.
- * A weight of a cfs_rq is the sum of weights of which entities
- * are queued on this cfs_rq, so a weight of a entity should not be
- * too large, so as the shares value of a task group.
- * (The default weight is 1024 - so there's no practical
- *  limitation from this.)
- */
-#define MIN_SHARES             (1UL <<  1)
-#define MAX_SHARES             (1UL << 18)
-#endif
-
 typedef int (*tg_visitor)(struct task_group *, void *);
 
 extern int walk_tg_tree_from(struct task_group *from,
@@ -629,6 +616,17 @@ static inline bool cfs_task_bw_constrain
 
 #endif /* !CONFIG_CGROUP_SCHED */
 
+/*
+ * A weight of 0 or 1 can cause arithmetics problems.
+ * A weight of a cfs_rq is the sum of weights of which entities
+ * are queued on this cfs_rq, so a weight of a entity should not be
+ * too large, so as the shares value of a task group.
+ * (The default weight is 1024 - so there's no practical
+ *  limitation from this.)
+ */
+#define MIN_SHARES             (1UL <<  1)
+#define MAX_SHARES             (1UL << 18)
+
 extern void unregister_rt_sched_group(struct task_group *tg);
 extern void free_rt_sched_group(struct task_group *tg);
 extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
@@ -707,6 +705,7 @@ struct cfs_rq {
        /*
         * CFS load tracking
         */
+       struct sched_entity     *h_curr;
        struct sched_avg        avg;
 #ifndef CONFIG_64BIT
        u64                     last_update_time_copy;
@@ -2509,6 +2508,7 @@ extern const u32          sched_prio_to_wmult[40
 #define ENQUEUE_MIGRATED       0x00040000
 #define ENQUEUE_INITIAL                0x00080000
 #define ENQUEUE_RQ_SELECTED    0x00100000
+#define ENQUEUE_QUEUED         0x00200000
 
 #define RETRY_TASK             ((void *)-1UL)
 
#35 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Hello Vincent,

Quoted 2 lines · Vincent Guittot

I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed update_entity_lag():

Great catch!

Quoted 16 lines

update_entity_lag must be called after updating curr and cfs_rd and before clearing on_rq

With the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet

---
 kernel/sched/fair.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index 77d0e1937f2c..32fe57004f27 100644

kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 5759

Quoted 7 lines
kernel/sched/fair.c +3 -0
@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
 	update_stats_dequeue_fair(cfs_rq, se, flags);
 
+	if (entity_is_task(se))
+		update_entity_lag(&rq_of(cfs_rq)->cfs, se);
+
 	se->on_rq = 0;

Ah! The curr->on_rq indicator changes here and we'll start ignoring it for avg_vruntime() calculation afterwards! Makes sense.

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7427

Quoted 5 lines
kernel/sched/fair.c +0 -0
 	account_entity_dequeue(cfs_rq, se);
 
@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 		if (sched_feat(DELAY_DEQUEUE) && delay &&
 		    !entity_eligible(cfs_rq, se)) {

Does this need a update_curr() before checking entity_eligible()?

Currently these bits reside in dequeue_entity() and is always done after a update_curr(cfs_rq) but here we may need a:

update_curr(task_cfs_rq(p)); /* to catch up h_curr's vruntime */

Just doing it for task_cfs_rq(p) should be fine since we only have to catch up curr's vruntime - sum_w_vruntime and sum_weight at root cfs_rq should be stable for all the tasks on rb-tree.

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7433

Quoted 10 lines
kernel/sched/fair.c +1 -1
 			update_load_avg(cfs_rq_of(se), se, 0);
+			update_entity_lag(cfs_rq, se);
 			set_delayed(se);
 			return false;
 		}
@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
 	dequeue_hierarchy(p, flags);
 
-	update_entity_lag(cfs_rq, se);

If we decide to do a update_curr(task_cfs_rq(p)) at the beginning of __dequeue_task(), we can just move this to above dequeue_hierarchy() before se->on_rq indicators are modified.

Thoughts?

Quoted 3 lines
kernel/sched/fair.c +0 -0
 	if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
 		se->deadline -= se->vruntime;
 		se->rel_deadline = 1;
Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#36 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 36 lines · K Prateek Nayak

Hello Vincent,

Quoted 2 lines · Vincent Guittot

I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed update_entity_lag():

Great catch!

Quoted 16 lines

update_entity_lag must be called after updating curr and cfs_rd and before clearing on_rq

With the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet

---
 kernel/sched/fair.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/sched/fair.c +0 -0
index 77d0e1937f2c..32fe57004f27 100644

kernel/sched/fair.c dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) · line 5759

Quoted 7 lines
kernel/sched/fair.c +3 -0
@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
      update_stats_dequeue_fair(cfs_rq, se, flags);
 
+     if (entity_is_task(se))
+             update_entity_lag(&rq_of(cfs_rq)->cfs, se);
+
      se->on_rq = 0;

Ah! The curr->on_rq indicator changes here and we'll start ignoring it for avg_vruntime() calculation afterwards! Makes sense.

Quoted 7 lines

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7427

Quoted 5 lines
kernel/sched/fair.c +0 -0
      account_entity_dequeue(cfs_rq, se);
 
@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
              if (sched_feat(DELAY_DEQUEUE) && delay &&
                  !entity_eligible(cfs_rq, se)) {

Does this need a update_curr() before checking entity_eligible()?

Yes we need to update curr first

Quoted 19 lines

Currently these bits reside in dequeue_entity() and is always done after a update_curr(cfs_rq) but here we may need a:

update_curr(task_cfs_rq(p)); /* to catch up h_curr's vruntime */

Just doing it for task_cfs_rq(p) should be fine since we only have to catch up curr's vruntime - sum_w_vruntime and sum_weight at root cfs_rq should be stable for all the tasks on rb-tree.

Quoted 9 lines
kernel/sched/fair.c +1 -0
                      update_load_avg(cfs_rq_of(se), se, 0);
+                     update_entity_lag(cfs_rq, se);
                      set_delayed(se);
                      return false;
              }
@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
      dequeue_hierarchy(p, flags);
 
Quoted 7 lines

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7433

Quoted 1 line
kernel/sched/fair.c +0 -1
-     update_entity_lag(cfs_rq, se);

If we decide to do a update_curr(task_cfs_rq(p)) at the beginning of __dequeue_task(), we can just move this to above dequeue_hierarchy() before se->on_rq indicators are modified.

Thoughts?

yes it's doable, we will have a spurious update_curr in dequeue_hierarchy but that will be a nop because of a null delta_exec

With flat hierarchy, vruntime and deadline are no longer linked to the cfs hierarchy. A possibility could be to move the update of vruntime and deadline outside but this is more complex because of delta_exec

The same apply for dl_server

Quoted 9 lines

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7439

Quoted 4 lines
kernel/sched/fair.c +0 -0
      if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
              se->deadline -= se->vruntime;
              se->rel_deadline = 1;

--
Thanks and Regards,
Prateek

author maintainer
#37 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

In reply to Vincent Guittot
Quoted 36 lines · Vincent Guittot

I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed update_entity_lag():

update_entity_lag must be called after updating curr and cfs_rd and before clearing on_rq

With the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet

---
 kernel/sched/fair.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/sched/fair.c +4 -0
index 77d0e1937f2c..32fe57004f27 100644
@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
 	update_stats_dequeue_fair(cfs_rq, se, flags);
 
+	if (entity_is_task(se))
+		update_entity_lag(&rq_of(cfs_rq)->cfs, se);
+
 	se->on_rq = 0;
 	account_entity_dequeue(cfs_rq, se);
 
@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 		if (sched_feat(DELAY_DEQUEUE) && delay &&
 		    !entity_eligible(cfs_rq, se)) {
 			update_load_avg(cfs_rq_of(se), se, 0);
+			update_entity_lag(cfs_rq, se);
 			set_delayed(se);
 			return false;
 		}

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7439

Quoted 7 lines
kernel/sched/fair.c +0 -1
@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
 	dequeue_hierarchy(p, flags);
 
-	update_entity_lag(cfs_rq, se);
 	if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
 		se->deadline -= se->vruntime;
 		se->rel_deadline = 1;

Argh!!! Thank you! I've gone blind staring at all this :/

Would it not be simpler to just move the update_entity_lag() call up a bit, like so?

---

kernel/sched/fair.c +3 -1
@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
 
 	clear_buddies(cfs_rq, se);
 
+	update_curr(cfs_rq);
+	update_entity_lag(cfs_rq, se);
+
 	if (flags & DEQUEUE_DELAYED) {
 		WARN_ON_ONCE(!se->sched_delayed);
 	} else {
@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
 
 	dequeue_hierarchy(p, flags);
 
-	update_entity_lag(cfs_rq, se);
 	if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
 		se->deadline -= se->vruntime;
 		se->rel_deadline = 1;
maintainer
#38 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 56 lines · Peter Zijlstra
Quoted 36 lines · Vincent Guittot

I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed update_entity_lag():

update_entity_lag must be called after updating curr and cfs_rd and before clearing on_rq

With the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet

---
 kernel/sched/fair.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/sched/fair.c +4 -0
index 77d0e1937f2c..32fe57004f27 100644
@@ -5753,6 +5753,9 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 
      update_stats_dequeue_fair(cfs_rq, se, flags);
 
+     if (entity_is_task(se))
+             update_entity_lag(&rq_of(cfs_rq)->cfs, se);
+
      se->on_rq = 0;
      account_entity_dequeue(cfs_rq, se);
 
@@ -7423,6 +7426,7 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
              if (sched_feat(DELAY_DEQUEUE) && delay &&
                  !entity_eligible(cfs_rq, se)) {
                      update_load_avg(cfs_rq_of(se), se, 0);
+                     update_entity_lag(cfs_rq, se);
                      set_delayed(se);
                      return false;
              }

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 7439

Quoted 7 lines
kernel/sched/fair.c +0 -1
@@ -7430,7 +7434,6 @@ static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
      dequeue_hierarchy(p, flags);
 
-     update_entity_lag(cfs_rq, se);
      if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
              se->deadline -= se->vruntime;
              se->rel_deadline = 1;

Argh!!! Thank you! I've gone blind staring at all this :/

Would it not be simpler to just move the update_entity_lag() call up a bit, like so?

---

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8002

Quoted 4 lines
kernel/sched/fair.c +1 -0
@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
 
        clear_buddies(cfs_rq, se);
 
+       update_curr(cfs_rq);

I agree it's simpler although we will call update_curr twice for one level, but the 2nd call should be nop because of delta_exec being null

Prateek proposed update_curr(task_cfs_rq(p)). Using task_cfs_rq(p) will ensure that we keep the same ordering as for_each_sched_entity

Quoted 13 lines
kernel/sched/fair.c +2 -1
+       update_entity_lag(cfs_rq, se);
+
        if (flags & DEQUEUE_DELAYED) {
                WARN_ON_ONCE(!se->sched_delayed);
        } else {
@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
 
        dequeue_hierarchy(p, flags);
 
-       update_entity_lag(cfs_rq, se);
        if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
                se->deadline -= se->vruntime;
                se->rel_deadline = 1;
author maintainer
#39 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 10 lines · Vincent Guittot
Quoted 10 lines

Would it not be simpler to just move the update_entity_lag() call up a bit, like so?

---

kernel/sched/fair.c +0 -0
@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
 
        clear_buddies(cfs_rq, se);
 
Quoted 7 lines

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8002

Quoted 1 line
kernel/sched/fair.c +1 -0
+       update_curr(cfs_rq);

I agree it's simpler although we will call update_curr twice for one level, but the 2nd call should be nop because of delta_exec being null

Prateek proposed update_curr(task_cfs_rq(p)). Using task_cfs_rq(p) will ensure that we keep the same ordering as for_each_sched_entity

Given:

    R
    |
    G
    |
    t

Then task_cfs_rq() will be G's cfs_rq, while cfs_rq is R's cfs_rq.

Since all the actual running happens inside R, this is what is required by update_entity_lag().

Doing update_curr(task_cfs_rq()) here doesn't make sense.

I'm not sure I see a way in which running them out of order hurts anything.

Quoted 13 lines
Quoted 13 lines
kernel/sched/fair.c +2 -1
+       update_entity_lag(cfs_rq, se);
+
        if (flags & DEQUEUE_DELAYED) {
                WARN_ON_ONCE(!se->sched_delayed);
        } else {
@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
 
        dequeue_hierarchy(p, flags);
 
-       update_entity_lag(cfs_rq, se);
        if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
                se->deadline -= se->vruntime;
                se->rel_deadline = 1;
maintainer
#40 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 23 lines · Peter Zijlstra
Quoted 10 lines · Vincent Guittot
Quoted 10 lines

Would it not be simpler to just move the update_entity_lag() call up a bit, like so?

---

kernel/sched/fair.c +0 -0
@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
 
        clear_buddies(cfs_rq, se);
 
Quoted 7 lines

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8002

Quoted 1 line
kernel/sched/fair.c +1 -0
+       update_curr(cfs_rq);

I agree it's simpler although we will call update_curr twice for one level, but the 2nd call should be nop because of delta_exec being null

Prateek proposed update_curr(task_cfs_rq(p)). Using task_cfs_rq(p) will ensure that we keep the same ordering as for_each_sched_entity

Given:

Quoted 7 lines
    R
    |
    G
    |
    t

Then task_cfs_rq() will be G's cfs_rq, while cfs_rq is R's cfs_rq.

Yes but update_curr() moves to R's cfs anyway before updating vruntime, deadline and dl_server

Quoted 3 lines

Since all the actual running happens inside R, this is what is required by update_entity_lag().

In other places like task_tick_fair, we follow the G then R order and vruntime and deadline are updated while updating G

Quoted 5 lines

Doing update_curr(task_cfs_rq()) here doesn't make sense.

I'm not sure I see a way in which running them out of order hurts anything.

I was thinking of use cases which involves throttling but I haven't gone deeply in the analyses

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
kernel/sched/fair.c +2 -1
+       update_entity_lag(cfs_rq, se);
+
        if (flags & DEQUEUE_DELAYED) {
                WARN_ON_ONCE(!se->sched_delayed);
        } else {
@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
 
        dequeue_hierarchy(p, flags);
 
-       update_entity_lag(cfs_rq, se);
        if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
                se->deadline -= se->vruntime;
                se->rel_deadline = 1;
author maintainer
#41 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

In reply to Peter Zijlstra
Quoted 30 lines · Peter Zijlstra
Quoted 10 lines · Vincent Guittot
Quoted 10 lines

Would it not be simpler to just move the update_entity_lag() call up a bit, like so?

---

kernel/sched/fair.c +0 -0
@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
 
        clear_buddies(cfs_rq, se);
 
Quoted 7 lines

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8002

Quoted 1 line
kernel/sched/fair.c +1 -0
+       update_curr(cfs_rq);

I agree it's simpler although we will call update_curr twice for one level, but the 2nd call should be nop because of delta_exec being null

Prateek proposed update_curr(task_cfs_rq(p)). Using task_cfs_rq(p) will ensure that we keep the same ordering as for_each_sched_entity

Given:

    R
    |
    G
    |
    t

Then task_cfs_rq() will be G's cfs_rq, while cfs_rq is R's cfs_rq.

Quoted 7 lines

Since all the actual running happens inside R, this is what is required by update_entity_lag().

Doing update_curr(task_cfs_rq()) here doesn't make sense.

I'm not sure I see a way in which running them out of order hurts anything.

Bah, I'm so full of fail. So update_curr() takes ->h_curr, which for R would be G's se, not t. So yeah, Prateek is right and I should stop trying to do more than one thing at a time :-(

author maintainer
#42 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Vincent Guittot

I finally fount the root cause of regression: the update of entity lag happened after the task has been dequeued which screwed update_entity_lag():

update_entity_lag must be called after updating curr and cfs_rd and before clearing on_rq

With the fix below I'm back to original hackbench figures and maybe even a bit better. I haven't checked shceduling latency yet

I see a very slight hackbench regression on the high end, but meh. The latency-slice test seems to have slightly improved max values, but this isn't the most stable of things.

author maintainer
#43 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

In reply to Peter Zijlstra
Quoted 17 lines · Peter Zijlstra

Would it not be simpler to just move the update_entity_lag() call up a bit, like so?

---

kernel/sched/fair.c +3 -0
@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
 
 	clear_buddies(cfs_rq, se);
 
+	update_curr(cfs_rq);
+	update_entity_lag(cfs_rq, se);
+
 	if (flags & DEQUEUE_DELAYED) {
 		WARN_ON_ONCE(!se->sched_delayed);
 	} else {

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8030

Quoted 7 lines
kernel/sched/fair.c +0 -1
@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
 
 	dequeue_hierarchy(p, flags);
 
-	update_entity_lag(cfs_rq, se);
 	if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
 		se->deadline -= se->vruntime;
 		se->rel_deadline = 1;

FWIW, I pushed out a new queue:sched/flat with this on. I had to rebase because of: 6d2051403d6c ("sched/fair: Update util_est after updating util_avg during dequeue"), hopefully I didn't wreck that :/

maintainer
#44 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 23 lines · Peter Zijlstra
Quoted 20 lines · Peter Zijlstra

Would it not be simpler to just move the update_entity_lag() call up a bit, like so?

---

kernel/sched/fair.c +3 -0
@@ -7999,6 +7999,9 @@ static bool __dequeue_task(struct rq *rq
 
      clear_buddies(cfs_rq, se);
 
+     update_curr(cfs_rq);
+     update_entity_lag(cfs_rq, se);
+
      if (flags & DEQUEUE_DELAYED) {
              WARN_ON_ONCE(!se->sched_delayed);
      } else {
@@ -8022,7 +8025,6 @@ static bool __dequeue_task(struct rq *rq
 
      dequeue_hierarchy(p, flags);
 
Quoted 8 lines

kernel/sched/fair.c static bool __dequeue_task(struct rq *rq · line 8030

Quoted 4 lines
kernel/sched/fair.c +0 -1
-     update_entity_lag(cfs_rq, se);
      if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
              se->deadline -= se->vruntime;
              se->rel_deadline = 1;

FWIW, I pushed out a new queue:sched/flat with this on. I had to rebase because of: 6d2051403d6c ("sched/fair: Update util_est after updating util_avg during dequeue"), hopefully I didn't wreck that :/

This looks good to me

#45 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

In reply to Peter Zijlstra

Hi Peter,

Quoted 4 lines · Peter Zijlstra
kernel/sched/fair.c +0 -0
@@ -13729,14 +13616,20 @@ static inline void task_tick_core(struct
  */
 static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 {

kernel/sched/fair.c static inline void task_tick_core(struct · line 13627

Quoted 10 lines
kernel/sched/fair.c +4 -1
-	struct cfs_rq *cfs_rq;
 	struct sched_entity *se = &curr->se;
+	unsigned long weight = NICE_0_LOAD;
+	struct cfs_rq *cfs_rq;
 
 	for_each_sched_entity(se) {
 		cfs_rq = cfs_rq_of(se);
 		entity_tick(cfs_rq, se, queued);
+
+		weight = __calc_prop_weight(cfs_rq, se, weight);

Testing sched/flat branch on AMD EPYC 9654 (384 CPUs, 8 NUMA nodes) with a 2-level cgroup hierarchy and cfs_bandwidth quota enabled, hackbench triggers a divide-by-zero oops:

  [  142.308571] divide error: 0000 [#1] SMP NOPTI
  [  142.308582] RIP: 0010:task_tick_fair+0x19e/0x410
  [  142.308601] Call Trace:
  [  142.308604]  <IRQ>
  [  142.308607]  scheduler_tick+0x6a/0x110
  [  142.308609]  update_process_times+0x6b/0x90
  [  142.308611]  tick_sched_handle+0x2a/0x70
  [  142.308613]  tick_sched_timer+0x57/0xb0

faddr2line confirms:

  task_tick_fair+0x19e/0x410:
  __calc_prop_weight at kernel/sched/fair.c:4085
  (inlined by) task_tick_fair at kernel/sched/fair.c:13576

===========================================================
Reproduction
===========================================================

Kernel: sched/flat branch (54d493980e00 and later)
Hardware: AMD EPYC 9654, 2S 384 logical CPUs

# 2-level cgroup, quota = 50% of one period
cgcreate -g cpu:/bw/l1/l2
cgset -r cpu.cfs_quota_us=50000 /bw/l1/l2
cgset -r cpu.cfs_period_us=100000 /bw/l1/l2

  # high task count amplifies the throttle→tick race window
  cgexec -g cpu:/bw/l1/l2 hackbench -g 48 -l 1000 -s 512 -T

Typically crashes within 30 seconds on this machine. A single-CPU kernel or a very loose quota (e.g. 90%) is unlikely to trigger it because the race window is narrow.

Thanks,
Zhang Qiao

Quoted 29 lines
kernel/sched/fair.c +4 -1
 	}
 
+	se = &curr->se;
+	reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+
 	if (queued)
 		return;
 
@@ -13772,7 +13665,7 @@ prio_changed_fair(struct rq *rq, struct
 	if (p->prio == oldprio)
 		return;
 
-	if (rq->cfs.nr_queued == 1)
+	if (rq->cfs.h_nr_queued == 1)
 		return;
 
 	/*
@@ -13901,29 +13794,40 @@ static void switched_to_fair(struct rq *
 	}
 }
 
 
 
 
 
 
 

.

#46 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Hello Zhang,

Quoted 4 lines · Zhang Qiao

Testing sched/flat branch on AMD EPYC 9654 (384 CPUs, 8 NUMA nodes) with a 2-level cgroup hierarchy and cfs_bandwidth quota enabled, hackbench triggers a divide-by-zero oops:

Quoted 8 lines
  [  142.308571] divide error: 0000 [#1] SMP NOPTI
  [  142.308582] RIP: 0010:task_tick_fair+0x19e/0x410
  [  142.308601] Call Trace:
  [  142.308604]  <IRQ>
  [  142.308607]  scheduler_tick+0x6a/0x110
  [  142.308609]  update_process_times+0x6b/0x90
  [  142.308611]  tick_sched_handle+0x2a/0x70
  [  142.308613]  tick_sched_timer+0x57/0xb0

More of this trace would have been helpful.

Quoted 6 lines

faddr2line confirms:

  task_tick_fair+0x19e/0x410:
  __calc_prop_weight at kernel/sched/fair.c:4085
  (inlined by) task_tick_fair at kernel/sched/fair.c:13576

Those line numbers don't match on the latest sched/flat but since you mention this happens with throttling, I believe it is tick hitting somewhere in between the task being dequeued by throttle_cfs_rq_work() and the CPU rescheduling and taking the task off the runqueue.

Dequeue from throttle is slightly special since it keeps the task on runqueue but the sched entity goes off the cfs_rq changing the hierarchical weights.

Can you check if this helps:

(Lightly tested with your reproducer)

kernel/sched/fair.c +12 -9
index b8bae794f063..d96e5915fb3e 100644
@@ -14815,18 +14815,21 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}
 static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 {
 	struct sched_entity *se = &curr->se;
-	unsigned long weight = NICE_0_LOAD;
-	struct cfs_rq *cfs_rq;
 
-	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
-		entity_tick(cfs_rq, se, queued);
+	if (se->on_rq) {
+		unsigned long weight = NICE_0_LOAD;
+		struct cfs_rq *cfs_rq;
 
-		weight = __calc_prop_weight(cfs_rq, se, weight);
-	}
+		for_each_sched_entity(se) {
+			cfs_rq = cfs_rq_of(se);
+			entity_tick(cfs_rq, se, queued);
+
+			weight = __calc_prop_weight(cfs_rq, se, weight);
+		}
 
-	se = &curr->se;
-	reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+		se = &curr->se;
+		reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+	}
 
 	if (queued)
 		return;

---

I don't think it makes too much sense to reweight an entity that has been dequeued. The enqueue at unthrottle will do it anyways.

Quoted 13 lines

===========================================================
Reproduction
===========================================================

Kernel: sched/flat branch (54d493980e00 and later)
Hardware: AMD EPYC 9654, 2S 384 logical CPUs

# 2-level cgroup, quota = 50% of one period
cgcreate -g cpu:/bw/l1/l2
cgset -r cpu.cfs_quota_us=50000 /bw/l1/l2
cgset -r cpu.cfs_period_us=100000 /bw/l1/l2

Quoted 6 lines
  # high task count amplifies the throttle→tick race window
  cgexec -g cpu:/bw/l1/l2 hackbench -g 48 -l 1000 -s 512 -T

Typically crashes within 30 seconds on this machine. A single-CPU kernel or a very loose quota (e.g. 90%) is unlikely to trigger it because the race window is narrow.

This was helpful! I see:

[  209.935597] Oops: divide error: 0000 [#1] SMP NOPTI
[  209.941061] CPU: 329 UID: 0 PID: 8247 Comm: sched-messaging Not tainted 7.1.0-rc2-test+ #73 PREEMPT(full)
[  209.951841] Hardware name: AMD Corporation Titanite_4G/Titanite_4G, BIOS RTI100CC 03/28/2024
[  209.961254] RIP: 0010:task_tick_fair+0x10d/0x850
[  209.966420] Code: dc 00 00 00 4c 89 f7 e8 f1 52 ff ff 45 85 e4 0f 85 ba 00 00 00 49 8b 06 4d 8b b6 b8 00 00 00 48 0f af c3 4d 85 f6 74 19 31 d2 <49> f7 37 ba 02 00 00 00 48 89 d3 48 39 d0 48 0f 43 d8 e9 20 ff ff
[  209.987382] RSP: 0018:ff581fd71e1fce58 EFLAGS: 00010046
[  209.993216] RAX: 0000010000000000 RBX: 0000000000100000 RCX: ff295dbfa9ad8080
[  210.001179] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff295dbfa9ad8080
[  210.009141] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000000063eb
[  210.017104] R10: 0000000000000000 R11: ff581fd71e1fcff8 R12: 0000000000000000
[  210.025061] R13: ff295dbfa9ad8000 R14: ff295dc06c6eac00 R15: ff295dbfd9bc8600
[  210.033027] FS:  00007faef8c8b640(0000) GS:ff295e7c4acca000(0000) knlGS:0000000000000000
[  210.042060] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  210.048474] CR2: 00007f9884292d30 CR3: 000000011aa26001 CR4: 0000000000f71ef0
[  210.056430] PKRU: 55555554
[  210.059448] Call Trace:
[  210.062177]  <IRQ>
[  210.064426]  sched_tick+0x94/0x250
[  210.068229]  update_process_times+0x99/0xc0
[  210.072903]  tick_nohz_handler+0x95/0x1a0
[  210.077380]  ? __pfx_tick_nohz_handler+0x10/0x10
[  210.082534]  __hrtimer_run_queues+0xfe/0x260
[  210.087304]  hrtimer_interrupt+0x122/0x1f0
[  210.091880]  __sysvec_apic_timer_interrupt+0x55/0x130
[  210.097525]  sysvec_apic_timer_interrupt+0x7a/0xb0
[  210.102873]  </IRQ>
[  210.105203]  <TASK>
[  210.107542]  asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  210.113284] RIP: 0010:_raw_spin_unlock_irqrestore+0x1d/0x40
[  210.119511] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 c6 07 00 0f 1f 00 f7 c6 00 02 00 00 74 06 fb 0f 1f 44 00 00 <65> ff 0d ec 20 fd 01 74 05 e9 c0 81 d4 fe e8 00 93 ec fe e9 b6 81
[  210.140469] RSP: 0018:ff581fd74032fe88 EFLAGS: 00000206
[  210.146308] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000004
[  210.154271] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ff295dbfa9ad8d64
[  210.162235] RBP: ff295dbfa9ad8000 R08: 0000000000000000 R09: 0000000000000000
[  210.170196] R10: 0000000000000000 R11: 0000000000000000 R12: ff295dbfa9ad8d64
[  210.178159] R13: ff581fd74032ff48 R14: ff295dbfa9ad8000 R15: 00fffffffffff000
[  210.186139]  task_work_run+0x5c/0x90
[  210.190137]  exit_to_user_mode_loop+0x16e/0x550
[  210.195198]  ? srso_alias_return_thunk+0x5/0xfbef5
[  210.200552]  ? ksys_read+0xc5/0xe0
[  210.204352]  do_syscall_64+0x26e/0x750
[  210.208540]  ? do_syscall_64+0xaa/0x750
[  210.212823]  ? srso_alias_return_thunk+0x5/0xfbef5
[  210.218174]  entry_SYSCALL_64_after_hwframe+0x76/0x7e

---

So the theory of throttle work causing this checks out.

The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)

Signature · 2 lines
Thanks and Regards,
Prateek
Tested-by
#47 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Hi Prateek,

Quoted 21 lines · K Prateek Nayak

Hello Zhang,

Quoted 4 lines · Zhang Qiao

Testing sched/flat branch on AMD EPYC 9654 (384 CPUs, 8 NUMA nodes) with a 2-level cgroup hierarchy and cfs_bandwidth quota enabled, hackbench triggers a divide-by-zero oops:

Quoted 8 lines
  [  142.308571] divide error: 0000 [#1] SMP NOPTI
  [  142.308582] RIP: 0010:task_tick_fair+0x19e/0x410
  [  142.308601] Call Trace:
  [  142.308604]  <IRQ>
  [  142.308607]  scheduler_tick+0x6a/0x110
  [  142.308609]  update_process_times+0x6b/0x90
  [  142.308611]  tick_sched_handle+0x2a/0x70
  [  142.308613]  tick_sched_timer+0x57/0xb0

More of this trace would have been helpful.

Quoted 3 lines

faddr2line confirms:

Quoted 9 lines
Quoted 3 lines
  task_tick_fair+0x19e/0x410:
  __calc_prop_weight at kernel/sched/fair.c:4085
  (inlined by) task_tick_fair at kernel/sched/fair.c:13576

Those line numbers don't match on the latest sched/flat but since you mention this happens with throttling, I believe it is tick hitting somewhere in between the task being dequeued by throttle_cfs_rq_work() and the CPU rescheduling and taking the task off the runqueue.

Sorry for the confusion on the line numbers — the mismatch was due to some local debug code I had added on top of sched/flat, not a difference in the base tree.

Quoted 30 lines

Dequeue from throttle is slightly special since it keeps the task on runqueue but the sched entity goes off the cfs_rq changing the hierarchical weights.

Quoted 1 line

Can you check if this helps:

(Lightly tested with your reproducer)

kernel/sched/fair.c +6 -7
index b8bae794f063..d96e5915fb3e 100644
@@ -14815,18 +14815,21 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}
 static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 {
 	struct sched_entity *se = &curr->se;
-	unsigned long weight = NICE_0_LOAD;
-	struct cfs_rq *cfs_rq;
 
-	for_each_sched_entity(se) {
-		cfs_rq = cfs_rq_of(se);
-		entity_tick(cfs_rq, se, queued);
+	if (se->on_rq) {
+		unsigned long weight = NICE_0_LOAD;
+		struct cfs_rq *cfs_rq;
 
-		weight = __calc_prop_weight(cfs_rq, se, weight);
-	}
+		for_each_sched_entity(se) {
+			cfs_rq = cfs_rq_of(se);
+			entity_tick(cfs_rq, se, queued);

kernel/sched/fair.c static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {} · line 14833

Quoted 10 lines
kernel/sched/fair.c +6 -2
+
+			weight = __calc_prop_weight(cfs_rq, se, weight);
+		}
 
-	se = &curr->se;
-	reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+		se = &curr->se;
+		reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+	}
 

throttle_cfs_rq_work() sets se->on_rq = 0 while the task is still running as rq->curr, and the subsequent tick should not attempt to reweight an already-dequeued entity. The unthrottle enqueue will handle the reweight anyway.

I've tested your suggested diff on my AMD EPYC 9654 (384 CPUs, 8 NUMA nodes) and it resolves the crash. The reproducer no longer triggers the divide error after running for several minutes.

Thanks,
Zhang Qiao

.

Quoted 83 lines
kernel/sched/fair.c +0 -1
 	if (queued)
 		return;
---

I don't think it makes too much sense to reweight an entity that has been dequeued. The enqueue at unthrottle will do it anyways.

Quoted 13 lines

===========================================================
Reproduction
===========================================================

Kernel: sched/flat branch (54d493980e00 and later)
Hardware: AMD EPYC 9654, 2S 384 logical CPUs

# 2-level cgroup, quota = 50% of one period
cgcreate -g cpu:/bw/l1/l2
cgset -r cpu.cfs_quota_us=50000 /bw/l1/l2
cgset -r cpu.cfs_period_us=100000 /bw/l1/l2

Quoted 6 lines
  # high task count amplifies the throttle→tick race window
  cgexec -g cpu:/bw/l1/l2 hackbench -g 48 -l 1000 -s 512 -T

Typically crashes within 30 seconds on this machine. A single-CPU kernel or a very loose quota (e.g. 90%) is unlikely to trigger it because the race window is narrow.

This was helpful! I see:

[  209.935597] Oops: divide error: 0000 [#1] SMP NOPTI
[  209.941061] CPU: 329 UID: 0 PID: 8247 Comm: sched-messaging Not tainted 7.1.0-rc2-test+ #73 PREEMPT(full)
[  209.951841] Hardware name: AMD Corporation Titanite_4G/Titanite_4G, BIOS RTI100CC 03/28/2024
[  209.961254] RIP: 0010:task_tick_fair+0x10d/0x850
[  209.966420] Code: dc 00 00 00 4c 89 f7 e8 f1 52 ff ff 45 85 e4 0f 85 ba 00 00 00 49 8b 06 4d 8b b6 b8 00 00 00 48 0f af c3 4d 85 f6 74 19 31 d2 <49> f7 37 ba 02 00 00 00 48 89 d3 48 39 d0 48 0f 43 d8 e9 20 ff ff
[  209.987382] RSP: 0018:ff581fd71e1fce58 EFLAGS: 00010046
[  209.993216] RAX: 0000010000000000 RBX: 0000000000100000 RCX: ff295dbfa9ad8080
[  210.001179] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff295dbfa9ad8080
[  210.009141] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000000063eb
[  210.017104] R10: 0000000000000000 R11: ff581fd71e1fcff8 R12: 0000000000000000
[  210.025061] R13: ff295dbfa9ad8000 R14: ff295dc06c6eac00 R15: ff295dbfd9bc8600
[  210.033027] FS:  00007faef8c8b640(0000) GS:ff295e7c4acca000(0000) knlGS:0000000000000000
[  210.042060] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  210.048474] CR2: 00007f9884292d30 CR3: 000000011aa26001 CR4: 0000000000f71ef0
[  210.056430] PKRU: 55555554
[  210.059448] Call Trace:
[  210.062177]  <IRQ>
[  210.064426]  sched_tick+0x94/0x250
[  210.068229]  update_process_times+0x99/0xc0
[  210.072903]  tick_nohz_handler+0x95/0x1a0
[  210.077380]  ? __pfx_tick_nohz_handler+0x10/0x10
[  210.082534]  __hrtimer_run_queues+0xfe/0x260
[  210.087304]  hrtimer_interrupt+0x122/0x1f0
[  210.091880]  __sysvec_apic_timer_interrupt+0x55/0x130
[  210.097525]  sysvec_apic_timer_interrupt+0x7a/0xb0
[  210.102873]  </IRQ>
[  210.105203]  <TASK>
[  210.107542]  asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  210.113284] RIP: 0010:_raw_spin_unlock_irqrestore+0x1d/0x40
[  210.119511] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 c6 07 00 0f 1f 00 f7 c6 00 02 00 00 74 06 fb 0f 1f 44 00 00 <65> ff 0d ec 20 fd 01 74 05 e9 c0 81 d4 fe e8 00 93 ec fe e9 b6 81
[  210.140469] RSP: 0018:ff581fd74032fe88 EFLAGS: 00000206
[  210.146308] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000004
[  210.154271] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ff295dbfa9ad8d64
[  210.162235] RBP: ff295dbfa9ad8000 R08: 0000000000000000 R09: 0000000000000000
[  210.170196] R10: 0000000000000000 R11: 0000000000000000 R12: ff295dbfa9ad8d64
[  210.178159] R13: ff581fd74032ff48 R14: ff295dbfa9ad8000 R15: 00fffffffffff000
[  210.186139]  task_work_run+0x5c/0x90
[  210.190137]  exit_to_user_mode_loop+0x16e/0x550
[  210.195198]  ? srso_alias_return_thunk+0x5/0xfbef5
[  210.200552]  ? ksys_read+0xc5/0xe0
[  210.204352]  do_syscall_64+0x26e/0x750
[  210.208540]  ? do_syscall_64+0xaa/0x750
[  210.212823]  ? srso_alias_return_thunk+0x5/0xfbef5
[  210.218174]  entry_SYSCALL_64_after_hwframe+0x76/0x7e

---

So the theory of throttle work causing this checks out.

The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)

author maintainer
#48 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

In reply to K Prateek Nayak
Quoted 3 lines · K Prateek Nayak

The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)

Is this a different issue than the one you raised before? We talked about throtte, and you were going to make a proper patch of that cleanup iirc.

#49 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Hello Peter,

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · K Prateek Nayak

The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)

Is this a different issue than the one you raised before?

Yes, this is different. Essentially, this is what is happening:

throttle_cfs_rq_work()

    task_rq_lock()

    dequeue_task_fair(current)    /* Task is dequeued on cfs side */
      __dequeue_task(current)
        dequeue_hierarchy(current);
          current->se.on_rq = 0;
          /* update_load_sub() */
    resched_curr();               /* Initiates a resched */

    task_rq_unlock()
      local_irq_enable();

=====> sched_tick()

          task_tick_fair()
             __calc_prop_weight()
               /*
                * Oops: update_load_sub() above has
                * 0ed the weight of cfs_rq.
                */

<====

preempt_schedule_irq()
next = ...
put_prev_set_next_task() /* The runtime context is switched here */

Quoted 2 lines

We talked about throtte, and you were going to make a proper patch of that cleanup iirc.

I had rebased your suggestion on tip and fixed a couple of splats but once it was functional, I noticed hackbench taking twice as long to complete compared to tip and I was chasing that before I fell sick.

Let me go dig deeper to see where exactly it is all going sideways.

Signature · 2 lines
Thanks and Regards,
Prateek
author maintainer
#50 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 33 lines · K Prateek Nayak

Hello Peter,

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · K Prateek Nayak

The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)

Is this a different issue than the one you raised before?

Yes, this is different. Essentially, this is what is happening:

throttle_cfs_rq_work()

    task_rq_lock()

    dequeue_task_fair(current)    /* Task is dequeued on cfs side */
      __dequeue_task(current)
        dequeue_hierarchy(current);
          current->se.on_rq = 0;
          /* update_load_sub() */
    resched_curr();               /* Initiates a resched */

    task_rq_unlock()
      local_irq_enable();

=====> sched_tick()

          task_tick_fair()
             __calc_prop_weight()
               /*
                * Oops: update_load_sub() above has
                * 0ed the weight of cfs_rq.
                */
Quoted 6 lines

<====

preempt_schedule_irq()
next = ...
put_prev_set_next_task() /* The runtime context is switched here */

Ah, right. OK, I'll go have a poke once I get these proxy patches I've been spending too much time on posted.

I think I've found a 'problem' with that PROXY_WAKING ==> '->is_blocked && !->blocked_on' scheme :-(

Quoted 8 lines
Quoted 2 lines

We talked about throtte, and you were going to make a proper patch of that cleanup iirc.

I had rebased your suggestion on tip and fixed a couple of splats but once it was functional, I noticed hackbench taking twice as long to complete compared to tip and I was chasing that before I fell sick.

Let me go dig deeper to see where exactly it is all going sideways.

Sure, no worries. This happens; computers just never want to just DTRT already. I lost a day and then some trying to figure out why my seemingly 'trivial' proxy changes ended up trying to run a dead task last week...

author maintainer
#51 lore raw

Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue

Quoted 36 lines · Peter Zijlstra
Quoted 35 lines · K Prateek Nayak

Hello Peter,

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · K Prateek Nayak

The suggested diff above solves the crash in my case but your mileage may vary. Peter can comment if this is the right thing to do or not :-)

Is this a different issue than the one you raised before?

Yes, this is different. Essentially, this is what is happening:

throttle_cfs_rq_work()

    task_rq_lock()

    dequeue_task_fair(current)    /* Task is dequeued on cfs side */
      __dequeue_task(current)
        dequeue_hierarchy(current);
          current->se.on_rq = 0;
          /* update_load_sub() */
    resched_curr();               /* Initiates a resched */

    task_rq_unlock()
      local_irq_enable();

=====> sched_tick()

          task_tick_fair()
             __calc_prop_weight()
               /*
                * Oops: update_load_sub() above has
                * 0ed the weight of cfs_rq.
                */

<====

Quoted 7 lines
Quoted 4 lines

preempt_schedule_irq()
next = ...
put_prev_set_next_task() /* The runtime context is switched here */

Ah, right. OK, I'll go have a poke once I get these proxy patches I've been spending too much time on posted.

Yes, your solution seems reasonable. I'll fold that and push out a new version a little later today.

maintainer
#52 lore raw
In reply to Peter Zijlstra

Hello, Peter.

Quoted 22 lines · Peter Zijlstra

So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.

The problems with weight distribution are related to that infernal global fraction:

             tg->w * grq_i->w
   ge_i->w = ----------------
             \Sum_j grq_j->w

which we've approximated reasonably well by now. However, the immediate consequence of this fraction is that the total group weight (tg->w) gets fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine with the fact that 256 CPU systems are relatively common these days, this becomes painful.

The common 'solution' is to inflate the group weight by 'nr_cpus'; the immediate problem with that is that when all load of a group gets concentrated on a single CPU, the per-cpu cgroup weight becomes insanely large, easily exceeding nice -20.

Quoted 8 lines

Additionally there are numerical limits on the max weight you can have before the math starts suffering overflows. As such there is a definite limit on the total group weight. Which has annoyed people ;-)

The first few patches add a knob /debug/sched/cgroup_mode and a few different options on how to deal with this. My favourite is 'concur', but obviously that is also the most expensive one :-/ It adds a tg->tasks counter which makes the update_tg_load_avg() thing more expensive.

Ignoring fixed math accuracy problems, isn't the root problem here that every thread in the root cgroup competes as if each is its own cgroup? ie. Isn't the canonical solution here to create an enveloping group, at least for share calculation purposes, for root threads and then assign them some weight so that they compete in the same way that other cgroups do? Then, the different modes go away or rather whatever the user wants can be expressed via root's weight if that's to be made configurable.

Thanks.

Signature · 1 line
tejun
author maintainer
#53 lore raw
Quoted 34 lines · Tejun Heo

Hello, Peter.

Quoted 22 lines · Peter Zijlstra

So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.

The problems with weight distribution are related to that infernal global fraction:

             tg->w * grq_i->w
   ge_i->w = ----------------
             \Sum_j grq_j->w

which we've approximated reasonably well by now. However, the immediate consequence of this fraction is that the total group weight (tg->w) gets fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine with the fact that 256 CPU systems are relatively common these days, this becomes painful.

The common 'solution' is to inflate the group weight by 'nr_cpus'; the immediate problem with that is that when all load of a group gets concentrated on a single CPU, the per-cpu cgroup weight becomes insanely large, easily exceeding nice -20.

Quoted 8 lines

Additionally there are numerical limits on the max weight you can have before the math starts suffering overflows. As such there is a definite limit on the total group weight. Which has annoyed people ;-)

The first few patches add a knob /debug/sched/cgroup_mode and a few different options on how to deal with this. My favourite is 'concur', but obviously that is also the most expensive one :-/ It adds a tg->tasks counter which makes the update_tg_load_avg() thing more expensive.

Quoted 7 lines

Ignoring fixed math accuracy problems, isn't the root problem here that every thread in the root cgroup competes as if each is its own cgroup? ie. Isn't the canonical solution here to create an enveloping group, at least for share calculation purposes, for root threads and then assign them some weight so that they compete in the same way that other cgroups do? Then, the different modes go away or rather whatever the user wants can be expressed via root's weight if that's to be made configurable.

As long as the total group weight is a fraction; and it sorta has to be. You can run into trouble by stacking that fraction.

Take 256 CPUs and a group weight of 1024. Then each CPU gets a weight of 1/256 or 4. Even if we increase the internal accuracy to 20 bits (we do on 64bit) then this becomes 4096, do this for 2 more levels in the hierarchy and you're down to scraping the barrel again.

So if each level runs at a fraction f of the level above, then level n runs at f^n. Moving root into a phantom group at level 1, only solves the problem against other tasks at level 1, but then you have the same problem again at level 2 and below.

Both the numerical problems and the scale problem of the root group can be avoided if we can get the average/nominal fraction to be near 1.

The 'normal' way around this is to ensure the group weight is nr_cpus * 1024, then, when everybody is running, the per CPU weight is 1024 or 1 and the continued fraction is also 1-ish. This is why people like to increase the max group weight.

Trouble is of course that if not all CPUs are busy, with the extreme being only a single CPU carrying that weight of nr_cpus*1024, this then causes trouble because that one CPU gets overloaded.

One of the options is to simply put a max on the single CPU load; which is the crudest option to just make it 'work'. The one I favour though is the one where we scale the group weight by: 'min(cpumas, nr_tasks)'.

Anyway, this is why I've been looking at these alternative weight schemes, to get the nominal fraction near 1 and make these problems go away. It is both the numerical issues and the disparity between levels (with root being at level 0 being the most obvious).

Does that make sense?

maintainer
#54 lore raw

Hello, Peter.

On Tue, May 12, 2026 at 10:10:00AM +0200, Peter Zijlstra wrote:
...

Quoted 4 lines

Anyway, this is why I've been looking at these alternative weight schemes, to get the nominal fraction near 1 and make these problems go away. It is both the numerical issues and the disparity between levels (with root being at level 0 being the most obvious).

I see. I think what bothers me is that I'm unsure what the weight config would mean when the shares are scaled by the number of active cpus in that cgroup. Here's a simple example:

- There are 256 cpus.
- /cgroup-A has weight 100 and 128 active threads. No pinning.
- /cgroup-B has weight 100 and 256 active thredas. No pinning.

In the current code, assuming math holds up, cgroup-A and B would get about the same shares - ~128 CPUs each. However, if we scale the share by active CPUs in each cgroup, B's tasks would end up with the same weight as A's on CPUs that they end up competing on, which would lead to ~ 1:3 distribution. Is that the right reading of the code?

Thanks.

Signature · 1 line
tejun
author maintainer
#55 lore raw
Quoted 4 lines · Tejun Heo

Hello, Peter.

On Tue, May 12, 2026 at 10:10:00AM +0200, Peter Zijlstra wrote:
...

Quoted 8 lines
Quoted 4 lines

Anyway, this is why I've been looking at these alternative weight schemes, to get the nominal fraction near 1 and make these problems go away. It is both the numerical issues and the disparity between levels (with root being at level 0 being the most obvious).

I see. I think what bothers me is that I'm unsure what the weight config would mean when the shares are scaled by the number of active cpus in that cgroup.

Relative weight per active cpu :-), but yes, that is a somewhat more difficult concept I suppose.

Quoted 11 lines

Here's a simple example:

- There are 256 cpus.
- /cgroup-A has weight 100 and 128 active threads. No pinning.
- /cgroup-B has weight 100 and 256 active thredas. No pinning.

In the current code, assuming math holds up, cgroup-A and B would get about the same shares - ~128 CPUs each. However, if we scale the share by active CPUs in each cgroup, B's tasks would end up with the same weight as A's on CPUs that they end up competing on, which would lead to ~ 1:3 distribution. Is that the right reading of the code?

Indeed. So both A and B will get ~1024 weight per (active) CPU, such that on the CPUs they contend they will get 1:1 and then B will get the full CPU on the uncontested CPUs, resulting in a total of 1:3 distribution.

This can of course be compensated by increasing the relative weight of A, if that is so desired. But the alternative view is that for those 128 CPUs they overlap, A and B will get equal parts, it is just that B consumes another 128 CPUs and will not have contention there.

So the current scheme will inflate the part of A to be double the weight (of B), giving them 2 out of 3 parts on the contended CPUs, but then B will still get complete / uncontested access to those extra 128 CPUs, resulting in a 2:4 weight distribution.

Which also isn't as straight forward as one might think.

So perhaps 'weight on the CPUs you contest on' isn't as unintuitive as it seems on first glance, its just different.

And it has tremendous advantages as outlined before; it is naturally normalized -- the disparity between nesting levels goes away, and the edge case of a single CPU active will be sane.

Eg. consider your example except now A will have 1 active thread. Then A will get the full group weight (1024) on its one CPU, while B will get (1024/256=8) on each CPU.

So for the one contended CPU A gets 256 out of 257 parts, while B gets the full CPU for the remaining 255 CPUs, for a:

256 1 257
--- : --- + 255*--- = 256:65535 ~ 1:256
257 257 257

distribution. While with the new scheme it would be:

1 1 2
- : - + 255*- = 1:511
2 2 2

Which, realistically isn't all that different, except the old scheme has this really large weight to deal with.

So from where I'm sitting, yes different, but it behaves better.

maintainer
#56 lore raw

Hello, Peter.

On Mon, May 18, 2026 at 09:14:56AM +0200, Peter Zijlstra wrote:
...

Quoted 6 lines

So the current scheme will inflate the part of A to be double the weight (of B), giving them 2 out of 3 parts on the contended CPUs, but then B will still get complete / uncontested access to those extra 128 CPUs, resulting in a 2:4 weight distribution.

Which also isn't as straight forward as one might think.

Right, the current behavior isn't quite what people would expect intuitively either.

...

Quoted 9 lines

So for the one contended CPU A gets 256 out of 257 parts, while B gets the full CPU for the remaining 255 CPUs, for a:

256 1 257
--- : --- + 255*--- = 256:65535 ~ 1:256
257 257 257

distribution. While with the new scheme it would be:

Quoted 8 lines

1 1 2
- : - + 255*- = 1:511
2 2 2

Which, realistically isn't all that different, except the old scheme has this really large weight to deal with.

So from where I'm sitting, yes different, but it behaves better.

I see. Thread cardinality and affinity problems make weight based distribution such a pain. I wonder whether this can be better solved by turning it into a two-layer allocation problem - groups to CPUs and then timeshare on CPUs as necessary. That comes with a lot of its own problems but it can, aspirationally at least, approximate global weight distribution and would have better locality properties.

Thanks.

Signature · 1 line
tejun
author maintainer
#57 lore raw
Quoted 9 lines · Tejun Heo

Hello, Peter.

On Mon, May 18, 2026 at 09:14:56AM +0200, Peter Zijlstra wrote:
...

Quoted 5 lines

So the current scheme will inflate the part of A to be double the weight (of B), giving them 2 out of 3 parts on the contended CPUs, but then B will still get complete / uncontested access to those extra 128 CPUs, resulting in a 2:4 weight distribution.

Quoted 23 lines
Quoted 1 line

Which also isn't as straight forward as one might think.

Right, the current behavior isn't quite what people would expect intuitively either.

...

Quoted 17 lines

So for the one contended CPU A gets 256 out of 257 parts, while B gets the full CPU for the remaining 255 CPUs, for a:

256 1 257
--- : --- + 255*--- = 256:65535 ~ 1:256
257 257 257

distribution. While with the new scheme it would be:

1 1 2
- : - + 255*- = 1:511
2 2 2

Which, realistically isn't all that different, except the old scheme has this really large weight to deal with.

So from where I'm sitting, yes different, but it behaves better.

FWIW if the workload was single threads per CPU; the above is also the exact behaviour we'd have without cgroups.

Quoted 6 lines

I see. Thread cardinality and affinity problems make weight based distribution such a pain. I wonder whether this can be better solved by turning it into a two-layer allocation problem - groups to CPUs and then timeshare on CPUs as necessary. That comes with a lot of its own problems but it can, aspirationally at least, approximate global weight distribution and would have better locality properties.

If people want, they can already do this today. I don't see a reason to mandate something like that. That is, combine cpuset and cpu in a v2 hierarchy and you get this.

The main problem with doing something like that is of course that it isn't always clear how many CPUs will be needed for a particular 'job'. So assigning groups to CPUs isn't a straight forward thing.

If I remember, Meta was actually doing some of this. It was dynamically resizing cpusets based on load predictions and the like in order to separate various worloads on the same large machine, right?

Anyway, while it is somewhat tedious to change behaviour, I do think it is worth doing in this case.

maintainer
#58 lore raw
In reply to Peter Zijlstra
Quoted 84 lines · Peter Zijlstra

Hi!

So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.

The problems with weight distribution are related to that infernal global fraction:

             tg->w * grq_i->w
   ge_i->w = ----------------
             \Sum_j grq_j->w

which we've approximated reasonably well by now. However, the immediate consequence of this fraction is that the total group weight (tg->w) gets fragmented across all your CPUs. And at 64 CPUs that means your per-cpu cgroup weight ends up being a nice 19 task worth. And more CPUs more tiny. Combine with the fact that 256 CPU systems are relatively common these days, this becomes painful.

The common 'solution' is to inflate the group weight by 'nr_cpus'; the immediate problem with that is that when all load of a group gets concentrated on a single CPU, the per-cpu cgroup weight becomes insanely large, easily exceeding nice -20.

Additionally there are numerical limits on the max weight you can have before the math starts suffering overflows. As such there is a definite limit on the total group weight. Which has annoyed people ;-)

The first few patches add a knob /debug/sched/cgroup_mode and a few different options on how to deal with this. My favourite is 'concur', but obviously that is also the most expensive one :-/ It adds a tg->tasks counter which makes the update_tg_load_avg() thing more expensive.

I have some ideas but I figured I ought to share these things before sinking more time into it.

On to the hierarchical pick; this has been causing trouble for a very long time. So once again an attempt at flatting it. The basic idea is to keep the full hierarchical load tracking as-is, but keep all the runnable entities in a single level. The immediate concequence of all this is ofcourse that we need to constantly re-compute the effective weight of each entity as things progress.

Reweight is done on:
- enqueue
- pick -- or rather set_next_entity(.first=true)
- tick

So while the {en,de}queue operations are still O(depth) due to the full accounting mess, the pick is now a single level. Removing the intermediate levels that obscure runnability etc.

For testing, I've done a little experiment, I dug out what is colloqually known as a potato. A trusty old Sandybridge 12600k with a RX 580, and ran a game on it. From GOG, I had available 'Shadows: Awakens', a fun title that normally runs really well on this machine (provided you stick to 1080p).

To make it interesting, I added 8 (one for each logical CPU) copies of: 'nice spin.sh'; this results in the game becoming almost unplayable, as in proper terrible.

I used MangoHUD to record a few minutes of playtime for statistics, and then quit the came and re-started it with a shorter slice set (base/10). This results in the game being entirely playable -- not great, but definiltey playable.

Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper) Intel Core i7-2600K
AMD Radeon RX 580

Shadows Awakening (GOG)

default slice(*)

  FPS min  3.8    20.6
      avg 48.0    57.2
      mag 87.4    80.3

FT min 9.4 8.4

      avg  34.5   19.5
      max 107.4   37.2
Quoted 8 lines

FPS (Frames Per Second)
FT (FrameTime)

  [*] Command prefix: 'chrt -o --sched-runtime 280000 0'
      effectively setting 'base_slice_ns/10'

I have not compared to a kernel without flat on, just wanted to run non trivial workloads and play with slice to make sure everything 'works'.

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

I will review the patchset and try to get what is going wrong

Quoted 22 lines

Can also be had:

git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat

 include/linux/cpuset.h |    6
 include/linux/sched.h  |    1
 kernel/cgroup/cpuset.c |   15
 kernel/sched/core.c    |   47 --
 kernel/sched/debug.c   |  171 +++++---
 kernel/sched/fair.c    | 1038 ++++++++++++++++++++++---------------------------
 kernel/sched/pelt.c    |    6
 kernel/sched/sched.h   |   44 --
 8 files changed, 672 insertions(+), 656 deletions(-)

---
Change since v1 ( https://patch.msgid.link/[email protected] ):
- various Sashiko thingies
- rebase atop curren -tip

author maintainer
#59 lore raw
Quoted 18 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Quoted 5 lines

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

I will review the patchset and try to get what is going wrong

Yeah, that is unexpected. Let me go have another look too.

author maintainer
#60 lore raw
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Quoted 7 lines
Quoted 5 lines

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

I will review the patchset and try to get what is going wrong

Yeah, that is unexpected. Let me go have another look too.

So I can reproduce even without the last patch applied. I suspect it is in the cgroup mode patches somewhere. My first suspect is that concur mode thing doing bad things to track the 'global' nr_running thing.

author maintainer
#61 lore raw
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Peter Zijlstra
Quoted 22 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I will review the patchset and try to get what is going wrong

Yeah, that is unexpected. Let me go have another look too.

So I can reproduce even without the last patch applied. I suspect it is in the cgroup mode patches somewhere. My first suspect is that concur mode thing doing bad things to track the 'global' nr_running thing.

Argh, n/m PEBKAC. I'll try this again in the morning :/

maintainer
#62 lore raw
Quoted 29 lines · Peter Zijlstra
Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Peter Zijlstra
Quoted 23 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

I will review the patchset and try to get what is going wrong

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Yeah, that is unexpected. Let me go have another look too.

So I can reproduce even without the last patch applied. I suspect it is in the cgroup mode patches somewhere. My first suspect is that concur mode thing doing bad things to track the 'global' nr_running thing.

Argh, n/m PEBKAC. I'll try this again in the morning :/

Reverting the last patch is enough to recover performance

author maintainer
#63 lore raw
Quoted 32 lines · Vincent Guittot
Quoted 31 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 27 lines · Peter Zijlstra
Quoted 18 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Quoted 5 lines

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

I will review the patchset and try to get what is going wrong

Yeah, that is unexpected. Let me go have another look too.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

So I can reproduce even without the last patch applied. I suspect it is in the cgroup mode patches somewhere. My first suspect is that concur mode thing doing bad things to track the 'global' nr_running thing.

Argh, n/m PEBKAC. I'll try this again in the morning :/

Reverting the last patch is enough to recover performance

Yeah, I was on a fail-streak yesterday. I forgot to copy the kernel image before reboot ...

Lets see if today is better :-)

author maintainer
#64 lore raw
In reply to Vincent Guittot
Quoted 13 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

Quoted 7 lines

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

D'oh :/

kernel/sched/fair.c +1 -1
index e54da4c6c945..77d0e1937f2c 100644
@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
 	struct task_struct *donor = rq->donor;
 	struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
-	struct cfs_rq *cfs_rq = task_cfs_rq(donor);
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	int cse_is_idle, pse_is_idle;
 
 	/*
author maintainer
#65 lore raw
Quoted 30 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

Quoted 7 lines

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

D'oh :/

kernel/sched/fair.c +0 -0
index e54da4c6c945..77d0e1937f2c 100644

kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9077

Quoted 8 lines
kernel/sched/fair.c +1 -1
@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
 	enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
 	struct task_struct *donor = rq->donor;
 	struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
-	struct cfs_rq *cfs_rq = task_cfs_rq(donor);
+	struct cfs_rq *cfs_rq = &rq->cfs;
 	int cse_is_idle, pse_is_idle;
 
 	/*

With that fixed, I now get:

vanilla slice(*)

FPS min 3.0 11.1

    avg  44.7	57.3
    max  88.1	96.2

FT min 9.1 8.0

    avg  41.4	21.0
    max 157.2   53.9

FPS (Frames Per Second)
FT (FrameTime)

Which I suppose shows we now preempt less. Its still significantly better with reduced slice, but not as good as it was.

maintainer
#66 lore raw
In reply to Peter Zijlstra
Quoted 31 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

Quoted 7 lines

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

D'oh :/

kernel/sched/fair.c +0 -0
index e54da4c6c945..77d0e1937f2c 100644

kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9074

Quoted 5 lines
kernel/sched/fair.c +1 -1
@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
        enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
        struct task_struct *donor = rq->donor;
        struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
-       struct cfs_rq *cfs_rq = task_cfs_rq(donor);
+       struct cfs_rq *cfs_rq = &rq->cfs;

I tested this patch on top of the series but it doesn't fix the perf regression on rb5

hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default slice duration

Quoted 3 lines
kernel/sched/fair.c +0 -0
        int cse_is_idle, pse_is_idle;
 
        /*
author maintainer
#67 lore raw
Quoted 41 lines · Vincent Guittot
Quoted 31 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

Quoted 7 lines

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

D'oh :/

kernel/sched/fair.c +0 -0
index e54da4c6c945..77d0e1937f2c 100644

kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9074

Quoted 5 lines
kernel/sched/fair.c +1 -1
@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
        enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
        struct task_struct *donor = rq->donor;
        struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
-       struct cfs_rq *cfs_rq = task_cfs_rq(donor);
+       struct cfs_rq *cfs_rq = &rq->cfs;

I tested this patch on top of the series but it doesn't fix the perf regression on rb5

Quoted 2 lines

hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default slice duration

Weird, I can't reproduce anymore with this fixed :/

I'll try more hackbench variants tomorrow I suppose.

maintainer
#68 lore raw
Quoted 43 lines · Peter Zijlstra
Quoted 41 lines · Vincent Guittot
Quoted 31 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

Quoted 7 lines

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

D'oh :/

kernel/sched/fair.c +0 -0
index e54da4c6c945..77d0e1937f2c 100644

kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9074

Quoted 5 lines
kernel/sched/fair.c +1 -1
@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
        enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
        struct task_struct *donor = rq->donor;
        struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
-       struct cfs_rq *cfs_rq = task_cfs_rq(donor);
+       struct cfs_rq *cfs_rq = &rq->cfs;

I tested this patch on top of the series but it doesn't fix the perf regression on rb5

Quoted 6 lines
Quoted 2 lines

hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default slice duration

Weird, I can't reproduce anymore with this fixed :/

I'll try more hackbench variants tomorrow I suppose.

I tried several conf :
- HMP with EAS enabled
- HMP without EAS enabled (perf cpufreq gov)
- SMP (only the 4 little cores)

All of them show large regressions with hackbench which are almost recovered when increasing the slice from 2.8 to 16ms

maintainer
#69 lore raw
Quoted 52 lines · Vincent Guittot
Quoted 43 lines · Peter Zijlstra
Quoted 41 lines · Vincent Guittot
Quoted 31 lines · Peter Zijlstra
Quoted 13 lines · Vincent Guittot

I haven't reviewed the patches yet but I ran some tests with it while testing sched latency related changes for short slice wakeup preemption. I have some large hackbench regressions with this series on HMP system with and without EAS. those figures are unexpected because the benchs run on root cfs

One example with hackbench 8 groups thread pipe

tip/sched/core  tip/sched/core          +this patchset          +this patchset
slice 2.8ms     16ms                    2.8ms                   16ms

dragonboard rb5 with EAS
0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
0,689(+/- 9.1%) +8%

Quoted 7 lines

radxa orion6 HMP without EAS
0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
1,071(+/-5.9%) -82%

Increasing the slice partly removes regressions but tis is surprising because the bench runs at root cfs and I thought that results will not change in such a case

D'oh :/

kernel/sched/fair.c +0 -0
index e54da4c6c945..77d0e1937f2c 100644

kernel/sched/fair.c static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f · line 9074

Quoted 5 lines
kernel/sched/fair.c +1 -1
@@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
        enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
        struct task_struct *donor = rq->donor;
        struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
-       struct cfs_rq *cfs_rq = task_cfs_rq(donor);
+       struct cfs_rq *cfs_rq = &rq->cfs;

I tested this patch on top of the series but it doesn't fix the perf regression on rb5

Quoted 6 lines
Quoted 2 lines

hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default slice duration

Weird, I can't reproduce anymore with this fixed :/

I'll try more hackbench variants tomorrow I suppose.

Quoted 7 lines

I tried several conf :
- HMP with EAS enabled
- HMP without EAS enabled (perf cpufreq gov)
- SMP (only the 4 little cores)

All of them show large regressions with hackbench which are almost recovered when increasing the slice from 2.8 to 16ms

With patch 10 the vlag value is very often set to the max 3.8ms (the clamp value of 2.8ms slice + 1ms tick) whereas it is usually less than a 1ms without patch 10

#70 lore raw
In reply to Peter Zijlstra
Quoted 4 lines · Peter Zijlstra

Hi!

So cgroup scheduling has always been a pain in the arse. The problems start with weight distribution and end with hierachical picks and it all sucks.

It does..

Not that it is useful info, but we talked briefly about it at OSPM, so thought I'll report back. I gave this a go with my test case from schedqos announcement
[1] of running schbench with kernel build as BACKGROUND noise, but the fairness imposed at group level is preserved (as expected) even if the pick is flattened.

I do actually want a total flat system, ie: disable this whole thing :-)

The problem is to create a system where you want to introduce smart tagging based on tasks, group scheduling becomes a big problem. If a task is set as background or interactive, it has to be global to be enforced otherwise it loses its meaning. And my test case stresses two long running tasks one is interactive but the other is background and group scheduling imposes fairness that breaks the task level tagging. Managing deadline via runtime doesn't help here since they are both always busy tasks; and one must use nice values to manage bandwidth. But nice values are local when autogroup/cgroups are present.

I need to find a simple way to turn this thing off at runtime and properly flatten it ;-)

/runs away

[1] https://lore.kernel.org/lkml/20260415000910.2h5misvwc45bdumu@airbuntu/