Skip to contentSkip to conversation

sched/urgent: zero_vruntime fixes

PATCH Applied Peter Zijlstra 4 participants 10 messages Apr 1 – Apr 2 Scheduler

lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover sched/urgent: zero_vruntime fixes

Hi!

Slightly modified and with a changelog on patches from the thread here:

https://lkml.kernel.org/r/CANDhNCr3ooATiBgcnq8CAZ+AwzvmHeoKhjvfy=awF1RKFHydCA@mail.gmail.com

Where John found stress-ng-yield made mainline sad.

I have retained the tested-by tags even though I've slightly modified the patches; holler if you disagree. Local testing shows its good, and my VM was very quick to panic once I had the right ingredients.

author maintainer
#1 lore raw

PATCH 1/2 sched/fair: Fix zero_vruntime tracking fix

John reported that stress-ng-yield could make his machine unhappy and managed to bisect it to commit b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking").

The combination of yield and that commit was specific enough to hypothesize the following scenario:

Suppose we have 2 runnable tasks, both doing yield. Then one will be eligible and one will not be, because the average position must be in between these two entities.

Therefore, the runnable task will be eligible, and be promoted a full slice (all the tasks do is yield after all). This causes it to jump over the other task and now the other task is eligible and current is no longer. So we schedule.

Since we are runnable, there is no {de,en}queue. All we have is the __{en,de}queue_entity() from {put_prev,set_next}_task(). But per the fingered commit, those two no longer move zero_vruntime.

All that moves zero_vruntime are tick and full {de,en}queue.

This means, that if the two tasks playing leapfrog can reach the critical speed to reach the overflow point inside one tick's worth of time, we're up a creek.

Additionally, when multiple cgroups are involved, there is no guarantee the tick will in fact hit every cgroup in a timely manner. Statistically speaking it will, but that same statistics does not rule out the possibility of one cgroup not getting a tick for a significant amount of time -- however unlikely.

Therefore, just like with the yield() case, force an update at the end of every slice. This ensures the update is never more than a single slice behind and the whole thing is within 2 lag bounds as per the comment on entity_key().

Fixes b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking")
---
 kernel/sched/fair.c |   10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)
kernel/sched/fair.c +3 -7
@@ -707,7 +707,7 @@ void update_zero_vruntime(struct cfs_rq
  * Called in:
  *  - place_entity()      -- before enqueue
  *  - update_entity_lag() -- before dequeue
- *  - entity_tick()
+ *  - update_deadline()   -- slice expiration
  *
  * This means it is one entry 'behind' but that puts it close enough to where
  * the bound on entity_key() is at most two lag bounds.
@@ -1131,6 +1131,7 @@ static bool update_deadline(struct cfs_r
 	 * EEVDF: vd_i = ve_i + r_i / w_i
 	 */
 	se->deadline = se->vruntime + calc_delta_fair(se->slice, se);
+	avg_vruntime(cfs_rq);
 
 	/*
 	 * The task has consumed its request, reschedule.
@@ -5593,11 +5594,6 @@ entity_tick(struct cfs_rq *cfs_rq, struc
 	update_load_avg(cfs_rq, curr, UPDATE_TG);
 	update_cfs_group(curr);
 
-	/*
-	 * Pulls along cfs_rq::zero_vruntime.
-	 */
-	avg_vruntime(cfs_rq);
-
 #ifdef CONFIG_SCHED_HRTICK
 	/*
 	 * queued ticks are scheduled to match the slice, so don't bother
@@ -9128,7 +9124,7 @@ static void yield_task_fair(struct rq *r
 	 */
 	if (entity_eligible(cfs_rq, se)) {
 		se->vruntime = se->deadline;
-		se->deadline += calc_delta_fair(se->slice, se);
+		update_deadline(cfs_rq, se);
 	}
 }
 
maintainer Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] sched/fair: Fix zero_vruntime tracking fix

Quoted 38 lines · Peter Zijlstra

John reported that stress-ng-yield could make his machine unhappy and managed to bisect it to commit b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking").

The combination of yield and that commit was specific enough to hypothesize the following scenario:

Suppose we have 2 runnable tasks, both doing yield. Then one will be eligible and one will not be, because the average position must be in between these two entities.

Therefore, the runnable task will be eligible, and be promoted a full slice (all the tasks do is yield after all). This causes it to jump over the other task and now the other task is eligible and current is no longer. So we schedule.

Since we are runnable, there is no {de,en}queue. All we have is the __{en,de}queue_entity() from {put_prev,set_next}_task(). But per the fingered commit, those two no longer move zero_vruntime.

All that moves zero_vruntime are tick and full {de,en}queue.

This means, that if the two tasks playing leapfrog can reach the critical speed to reach the overflow point inside one tick's worth of time, we're up a creek.

Additionally, when multiple cgroups are involved, there is no guarantee the tick will in fact hit every cgroup in a timely manner. Statistically speaking it will, but that same statistics does not rule out the possibility of one cgroup not getting a tick for a significant amount of time -- however unlikely.

Therefore, just like with the yield() case, force an update at the end of every slice. This ensures the update is never more than a single slice behind and the whole thing is within 2 lag bounds as per the comment on entity_key().

Quoted 5 lines
Fixes b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking")
Quoted 46 lines
---
 kernel/sched/fair.c |   10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)
kernel/sched/fair.c +3 -7
@@ -707,7 +707,7 @@ void update_zero_vruntime(struct cfs_rq
  * Called in:
  *  - place_entity()      -- before enqueue
  *  - update_entity_lag() -- before dequeue
- *  - entity_tick()
+ *  - update_deadline()   -- slice expiration
  *
  * This means it is one entry 'behind' but that puts it close enough to where
  * the bound on entity_key() is at most two lag bounds.
@@ -1131,6 +1131,7 @@ static bool update_deadline(struct cfs_r
         * EEVDF: vd_i = ve_i + r_i / w_i
         */
        se->deadline = se->vruntime + calc_delta_fair(se->slice, se);
+       avg_vruntime(cfs_rq);
 
        /*
         * The task has consumed its request, reschedule.
@@ -5593,11 +5594,6 @@ entity_tick(struct cfs_rq *cfs_rq, struc
        update_load_avg(cfs_rq, curr, UPDATE_TG);
        update_cfs_group(curr);
 
-       /*
-        * Pulls along cfs_rq::zero_vruntime.
-        */
-       avg_vruntime(cfs_rq);
-
 #ifdef CONFIG_SCHED_HRTICK
        /*
         * queued ticks are scheduled to match the slice, so don't bother
@@ -9128,7 +9124,7 @@ static void yield_task_fair(struct rq *r
         */
        if (entity_eligible(cfs_rq, se)) {
                se->vruntime = se->deadline;
-               se->deadline += calc_delta_fair(se->slice, se);
+               update_deadline(cfs_rq, se);
        }
 }
 
merge bot Applied
#3 lore raw

[tip: sched/urgent] sched/fair: Fix zero_vruntime tracking fix

In reply to Peter Zijlstra
The following commit has been merged into the sched/urgent branch of tip:

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     1319ea57529e131822bab56bf417c8edc2db9ae8
Gitweb:        https://git.kernel.org/tip/1319ea57529e131822bab56bf417c8edc2db9ae8
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Wed, 01 Apr 2026 15:20:20 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 02 Apr 2026 13:42:43 +02:00

sched/fair: Fix zero_vruntime tracking fix

John reported that stress-ng-yield could make his machine unhappy and managed to bisect it to commit b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking").

The combination of yield and that commit was specific enough to hypothesize the following scenario:

Suppose we have 2 runnable tasks, both doing yield. Then one will be eligible and one will not be, because the average position must be in between these two entities.

Therefore, the runnable task will be eligible, and be promoted a full slice (all the tasks do is yield after all). This causes it to jump over the other task and now the other task is eligible and current is no longer. So we schedule.

Since we are runnable, there is no {de,en}queue. All we have is the __{en,de}queue_entity() from {put_prev,set_next}_task(). But per the fingered commit, those two no longer move zero_vruntime.

All that moves zero_vruntime are tick and full {de,en}queue.

This means, that if the two tasks playing leapfrog can reach the critical speed to reach the overflow point inside one tick's worth of time, we're up a creek.

Additionally, when multiple cgroups are involved, there is no guarantee the tick will in fact hit every cgroup in a timely manner. Statistically speaking it will, but that same statistics does not rule out the possibility of one cgroup not getting a tick for a significant amount of time -- however unlikely.

Therefore, just like with the yield() case, force an update at the end of every slice. This ensures the update is never more than a single slice behind and the whole thing is within 2 lag bounds as per the comment on entity_key().

---
 kernel/sched/fair.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)
kernel/sched/fair.c +3 -7
index bf948db..ab41147 100644
@@ -707,7 +707,7 @@ void update_zero_vruntime(struct cfs_rq *cfs_rq, s64 delta)
  * Called in:
  *  - place_entity()      -- before enqueue
  *  - update_entity_lag() -- before dequeue
- *  - entity_tick()
+ *  - update_deadline()   -- slice expiration
  *
  * This means it is one entry 'behind' but that puts it close enough to where
  * the bound on entity_key() is at most two lag bounds.
@@ -1131,6 +1131,7 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	 * EEVDF: vd_i = ve_i + r_i / w_i
 	 */
 	se->deadline = se->vruntime + calc_delta_fair(se->slice, se);
+	avg_vruntime(cfs_rq);
 
 	/*
 	 * The task has consumed its request, reschedule.
@@ -5593,11 +5594,6 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
 	update_load_avg(cfs_rq, curr, UPDATE_TG);
 	update_cfs_group(curr);
 
-	/*
-	 * Pulls along cfs_rq::zero_vruntime.
-	 */
-	avg_vruntime(cfs_rq);
-
 #ifdef CONFIG_SCHED_HRTICK
 	/*
 	 * queued ticks are scheduled to match the slice, so don't bother
@@ -9128,7 +9124,7 @@ static void yield_task_fair(struct rq *rq)
 	 */
 	if (entity_eligible(cfs_rq, se)) {
 		se->vruntime = se->deadline;
-		se->deadline += calc_delta_fair(se->slice, se);
+		update_deadline(cfs_rq, se);
 	}
 }
 
author maintainer
#4 lore raw

PATCH 2/2 sched/debug: Fix avg_vruntime() usage

In reply to Peter Zijlstra

John reported that stress-ng-yield could make his machine unhappy and managed to bisect it to commit b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking").

The commit in question changes avg_vruntime() from a function that is a pure reader, to a function that updates variables. This turns an unlocked sched/debug usage of this function from a minor mistake into a data corruptor.

Fixes af4cf40470c2 ("sched/fair: Add cfs_rq::avg_vruntime")
Fixes b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking")
---
 kernel/sched/debug.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/debug.c +3 -1
@@ -902,6 +902,7 @@ static void print_rq(struct seq_file *m,
 void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 {
 	s64 left_vruntime = -1, zero_vruntime, right_vruntime = -1, left_deadline = -1, spread;
+	u64 avruntime;
 	struct sched_entity *last, *first, *root;
 	struct rq *rq = cpu_rq(cpu);
 	unsigned long flags;
@@ -925,6 +926,7 @@ void print_cfs_rq(struct seq_file *m, in
 	if (last)
 		right_vruntime = last->vruntime;
 	zero_vruntime = cfs_rq->zero_vruntime;
+	avruntime = avg_vruntime(cfs_rq);
 	raw_spin_rq_unlock_irqrestore(rq, flags);
 
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "left_deadline",
@@ -934,7 +936,7 @@ void print_cfs_rq(struct seq_file *m, in
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "zero_vruntime",
 			SPLIT_NS(zero_vruntime));
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "avg_vruntime",
-			SPLIT_NS(avg_vruntime(cfs_rq)));
+			SPLIT_NS(avruntime));
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "right_vruntime",
 			SPLIT_NS(right_vruntime));
 	spread = right_vruntime - left_vruntime;
maintainer Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] sched/debug: Fix avg_vruntime() usage

Quoted 31 lines · Peter Zijlstra

John reported that stress-ng-yield could make his machine unhappy and managed to bisect it to commit b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking").

The commit in question changes avg_vruntime() from a function that is a pure reader, to a function that updates variables. This turns an unlocked sched/debug usage of this function from a minor mistake into a data corruptor.

Fixes af4cf40470c2 ("sched/fair: Add cfs_rq::avg_vruntime")
Fixes b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking")
---
 kernel/sched/debug.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/debug.c +1 -0
@@ -902,6 +902,7 @@ static void print_rq(struct seq_file *m,
 void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 {
        s64 left_vruntime = -1, zero_vruntime, right_vruntime = -1, left_deadline = -1, spread;
+       u64 avruntime;
        struct sched_entity *last, *first, *root;
        struct rq *rq = cpu_rq(cpu);
        unsigned long flags;

kernel/sched/debug.c void print_cfs_rq(struct seq_file *m, in · line 929

Quoted 4 lines
kernel/sched/debug.c +1 -0
@@ -925,6 +926,7 @@ void print_cfs_rq(struct seq_file *m, in
        if (last)
                right_vruntime = last->vruntime;
        zero_vruntime = cfs_rq->zero_vruntime;
+       avruntime = avg_vruntime(cfs_rq);

Minor comment:
Do you intentionally save zero_vruntime before callling avg_vruntime() which will update zero_vruntime ?
That could make sense to take a snapshot before being modified by print_cfs_rq() but I'm afraid the call to debugfs will anyway trigger an update before we save and display the value

Quoted 14 lines
kernel/sched/debug.c +1 -1
        raw_spin_rq_unlock_irqrestore(rq, flags);
 
        SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "left_deadline",
@@ -934,7 +936,7 @@ void print_cfs_rq(struct seq_file *m, in
        SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "zero_vruntime",
                        SPLIT_NS(zero_vruntime));
        SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "avg_vruntime",
-                       SPLIT_NS(avg_vruntime(cfs_rq)));
+                       SPLIT_NS(avruntime));
        SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "right_vruntime",
                        SPLIT_NS(right_vruntime));
        spread = right_vruntime - left_vruntime;
author maintainer
#6 lore raw

Re: [PATCH 2/2] sched/debug: Fix avg_vruntime() usage

Quoted 35 lines · Vincent Guittot
Quoted 34 lines · Peter Zijlstra

John reported that stress-ng-yield could make his machine unhappy and managed to bisect it to commit b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking").

The commit in question changes avg_vruntime() from a function that is a pure reader, to a function that updates variables. This turns an unlocked sched/debug usage of this function from a minor mistake into a data corruptor.

Fixes af4cf40470c2 ("sched/fair: Add cfs_rq::avg_vruntime")
Fixes b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking")
---
 kernel/sched/debug.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/debug.c +1 -0
@@ -902,6 +902,7 @@ static void print_rq(struct seq_file *m,
 void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 {
        s64 left_vruntime = -1, zero_vruntime, right_vruntime = -1, left_deadline = -1, spread;
+       u64 avruntime;
        struct sched_entity *last, *first, *root;
        struct rq *rq = cpu_rq(cpu);
        unsigned long flags;
@@ -925,6 +926,7 @@ void print_cfs_rq(struct seq_file *m, in
        if (last)
                right_vruntime = last->vruntime;
        zero_vruntime = cfs_rq->zero_vruntime;
Quoted 8 lines

kernel/sched/debug.c void print_cfs_rq(struct seq_file *m, in · line 929

Quoted 1 line
kernel/sched/debug.c +1 -0
+       avruntime = avg_vruntime(cfs_rq);

Minor comment:
Do you intentionally save zero_vruntime before callling avg_vruntime() which will update zero_vruntime ?
That could make sense to take a snapshot before being modified by print_cfs_rq() but I'm afraid the call to debugfs will anyway trigger an update before we save and display the value

Intentional might be a big word, but yeah, printing the same value twice seemed pointless. This way you can at least see where it came from or something.

Quoted 1 line

Thanks!

merge bot Applied
#7 lore raw

[tip: sched/urgent] sched/debug: Fix avg_vruntime() usage

In reply to Peter Zijlstra
The following commit has been merged into the sched/urgent branch of tip:

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     e08d007f9d813616ce7093600bc4fdb9c9d81d89
Gitweb:        https://git.kernel.org/tip/e08d007f9d813616ce7093600bc4fdb9c9d81d89
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Wed, 01 Apr 2026 15:20:21 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 02 Apr 2026 13:42:43 +02:00

sched/debug: Fix avg_vruntime() usage

John reported that stress-ng-yield could make his machine unhappy and managed to bisect it to commit b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking").

The commit in question changes avg_vruntime() from a function that is a pure reader, to a function that updates variables. This turns an unlocked sched/debug usage of this function from a minor mistake into a data corruptor.

Fixes af4cf40470c2 ("sched/fair: Add cfs_rq::avg_vruntime")
Fixes b3d99f43c72b ("sched/fair: Fix zero_vruntime tracking")
---
 kernel/sched/debug.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/debug.c +3 -1
index b24f40f..15bf45b 100644
@@ -902,6 +902,7 @@ static void print_rq(struct seq_file *m, struct rq *rq, int rq_cpu)
 void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 {
 	s64 left_vruntime = -1, zero_vruntime, right_vruntime = -1, left_deadline = -1, spread;
+	u64 avruntime;
 	struct sched_entity *last, *first, *root;
 	struct rq *rq = cpu_rq(cpu);
 	unsigned long flags;
@@ -925,6 +926,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 	if (last)
 		right_vruntime = last->vruntime;
 	zero_vruntime = cfs_rq->zero_vruntime;
+	avruntime = avg_vruntime(cfs_rq);
 	raw_spin_rq_unlock_irqrestore(rq, flags);
 
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "left_deadline",
@@ -934,7 +936,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "zero_vruntime",
 			SPLIT_NS(zero_vruntime));
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "avg_vruntime",
-			SPLIT_NS(avg_vruntime(cfs_rq)));
+			SPLIT_NS(avruntime));
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "right_vruntime",
 			SPLIT_NS(right_vruntime));
 	spread = right_vruntime - left_vruntime;
author maintainer
#8 lore raw
In reply to Peter Zijlstra
Quoted 11 lines · Peter Zijlstra

Hi!

Slightly modified and with a changelog on patches from the thread here:

https://lkml.kernel.org/r/CANDhNCr3ooATiBgcnq8CAZ+AwzvmHeoKhjvfy=awF1RKFHydCA@mail.gmail.com

Where John found stress-ng-yield made mainline sad.

I have retained the tested-by tags even though I've slightly modified the patches; holler if you disagree. Local testing shows its good, and my VM was very quick to panic once I had the right ingredients.

Should now live in: queue/sched/urgent

#9 lore raw
In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra

Slightly modified and with a changelog on patches from the thread here:

https://lkml.kernel.org/r/CANDhNCr3ooATiBgcnq8CAZ+AwzvmHeoKhjvfy=awF1RKFHydCA@mail.gmail.com

Where John found stress-ng-yield made mainline sad.

I have retained the tested-by tags even though I've slightly modified the patches; holler if you disagree. Local testing shows its good, and my VM was very quick to panic once I had the right ingredients.

Testing ran well overnight! Thank you again for chasing all of these edge cases down so fast!
-john