Skip to contentSkip to conversation

sched/idle: Fix avg_idle saturation by establishing symmetric idle entry hook

PATCH Superseded Masahito S 5 participants 6 messages Apr 17 – Apr 23 Scheduler

lore
Share & tools
lore

Masahito S

author
#0 lore raw

PATCH 1/1 sched/idle: Fix avg_idle saturation by establishing symmetric idle entry hook

update_rq_avg_idle(), called from put_prev_task_idle(), computes rq->avg_idle as rq_clock() - rq->idle_stamp. However, idle_stamp is only set by sched_balance_newidle() when a CPU enters CPU_NEWLY_IDLE through the fair class path. When the idle task is preempted without sched_balance_newidle() having run (boot, hotplug, sched class transitions), idle_stamp remains 0, producing a delta equal to rq_clock() — a value in the billions of nanoseconds — which saturates avg_idle at 2 * max_idle_balance_cost.

This inflated avg_idle prevents sched_balance_newidle() from early-returning (fair.c: avg_idle < max_newidle_lb_cost check), making it overly aggressive. The resulting excess newidle migrations override wake-time placement decisions made by select_idle_sibling(), degrading cache locality that careful placement (recent_used_cpu, select_idle_core, etc.) is designed to preserve.

Fix this by:

1. Adding an idle_stamp validity guard to update_rq_avg_idle(), so that a zero idle_stamp is never used as a timestamp.

2. Setting idle_stamp in set_next_task_idle() when it has not already been set by sched_balance_newidle(). This establishes a symmetric idle entry/exit contract: set_next_task_idle() marks the start of the idle period, put_prev_task_idle() measures and records it via update_rq_avg_idle().

The entry hook preserves idle_stamp if sched_balance_newidle() has already set it, maintaining the existing semantic where balance-attempt duration is included in the idle measurement.

---
 kernel/sched/core.c | 3 +++
 kernel/sched/idle.c | 3 +++
 2 files changed, 6 insertions(+)
kernel/sched/core.c +3 -0
index 496dff740d..ec801f731c 100644
@@ -3633,6 +3633,9 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
 
 void update_rq_avg_idle(struct rq *rq)
 {
+	if (!rq->idle_stamp)
+		return;
+
 	u64 delta = rq_clock(rq) - rq->idle_stamp;
 	u64 max = 2*rq->max_idle_balance_cost;
 
kernel/sched/idle.c +3 -0
index a83be0c834..9ceb7e6224 100644
@@ -491,6 +491,9 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir
 	schedstat_inc(rq->sched_goidle);
 	next->se.exec_start = rq_clock_task(rq);
 
+	if (!rq->idle_stamp)
+		rq->idle_stamp = rq_clock(rq);
+
 	/*
 	 * rq is about to be idle, check if we need to update the
 	 * lost_idle_time of clock_pelt
Signature · 1 line
2.34.1
#1 lore raw
Quoted 27 lines · Masahito S

update_rq_avg_idle(), called from put_prev_task_idle(), computes rq->avg_idle as rq_clock() - rq->idle_stamp. However, idle_stamp is only set by sched_balance_newidle() when a CPU enters CPU_NEWLY_IDLE through the fair class path. When the idle task is preempted without sched_balance_newidle() having run (boot, hotplug, sched class transitions), idle_stamp remains 0, producing a delta equal to rq_clock() — a value in the billions of nanoseconds — which saturates avg_idle at 2 * max_idle_balance_cost.

This inflated avg_idle prevents sched_balance_newidle() from early-returning (fair.c: avg_idle < max_newidle_lb_cost check), making it overly aggressive. The resulting excess newidle migrations override wake-time placement decisions made by select_idle_sibling(), degrading cache locality that careful placement (recent_used_cpu, select_idle_core, etc.) is designed to preserve.

Fix this by:

1. Adding an idle_stamp validity guard to update_rq_avg_idle(), so that a zero idle_stamp is never used as a timestamp.

2. Setting idle_stamp in set_next_task_idle() when it has not already
been set by sched_balance_newidle(). This e> avg_idle at 2 * max_idle_balance_cost.stablishes a symmetric idle entry/exit contract: set_next_task_idle() marks the start of
the idle period, put_prev_task_idle() measures and records it via
update_rq_avg_idle().

Quoted 3 lines

The entry hook preserves idle_stamp if sched_balance_newidle() has already set it, maintaining the existing semantic where balance-attempt duration is included in the idle measurement.

You're missing the sign-off tag.
The actual change looks reasonable to me, albeit the impact surely limited.

Quoted 33 lines
---
 kernel/sched/core.c | 3 +++
 kernel/sched/idle.c | 3 +++
 2 files changed, 6 insertions(+)
kernel/sched/core.c +3 -0
index 496dff740d..ec801f731c 100644
@@ -3633,6 +3633,9 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
 
 void update_rq_avg_idle(struct rq *rq)
 {
+	if (!rq->idle_stamp)
+		return;
+
 	u64 delta = rq_clock(rq) - rq->idle_stamp;
 	u64 max = 2*rq->max_idle_balance_cost;
 
kernel/sched/idle.c +3 -0
index a83be0c834..9ceb7e6224 100644
@@ -491,6 +491,9 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir
 	schedstat_inc(rq->sched_goidle);
 	next->se.exec_start = rq_clock_task(rq);
 
+	if (!rq->idle_stamp)
+		rq->idle_stamp = rq_clock(rq);
+
 	/*
 	 * rq is about to be idle, check if we need to update the
 	 * lost_idle_time of clock_pelt
author
#2 lore raw

PATCH v2 1/1 sched/idle: Fix avg_idle saturation by establishing symmetric idle entry hook

In reply to Masahito S

update_rq_avg_idle(), called from put_prev_task_idle(), computes rq->avg_idle as rq_clock() - rq->idle_stamp. However, idle_stamp is only set by sched_balance_newidle() when a CPU enters CPU_NEWLY_IDLE through the fair class path. When the idle task is preempted without sched_balance_newidle() having run (boot, hotplug, sched class transitions), idle_stamp remains 0, producing a delta equal to rq_clock() — a value in the billions of nanoseconds — which saturates avg_idle at 2 * max_idle_balance_cost.

This inflated avg_idle prevents sched_balance_newidle() from early-returning (fair.c: avg_idle < max_newidle_lb_cost check), making it overly aggressive. The resulting excess newidle migrations override wake-time placement decisions made by select_idle_sibling(), degrading cache locality that careful placement (recent_used_cpu, select_idle_core, etc.) is designed to preserve.

Fix this by:

1. Adding an idle_stamp validity guard to update_rq_avg_idle(), so that a zero idle_stamp is never used as a timestamp.

2. Setting idle_stamp in set_next_task_idle() when it has not already been set by sched_balance_newidle(). This establishes a symmetric idle entry/exit contract: set_next_task_idle() marks the start of the idle period, put_prev_task_idle() measures and records it via update_rq_avg_idle().

The entry hook preserves idle_stamp if sched_balance_newidle() has already set it, maintaining the existing semantic where balance-attempt duration is included in the idle measurement.

---
Changes in v2:
- Added missing Signed-off-by tag (no functional changes). Thanks to Eric Naim and Christian Loehle for pointing this out.

 kernel/sched/core.c | 3 +++
 kernel/sched/idle.c | 3 +++
 2 files changed, 6 insertions(+)
kernel/sched/core.c +3 -0
index 496dff740d..ec801f731c 100644
@@ -3633,6 +3633,9 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
 
 void update_rq_avg_idle(struct rq *rq)
 {
+	if (!rq->idle_stamp)
+		return;
+
 	u64 delta = rq_clock(rq) - rq->idle_stamp;
 	u64 max = 2*rq->max_idle_balance_cost;
 
kernel/sched/idle.c +3 -0
index a83be0c834..9ceb7e6224 100644
@@ -491,6 +491,9 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir
 	schedstat_inc(rq->sched_goidle);
 	next->se.exec_start = rq_clock_task(rq);
 
+	if (!rq->idle_stamp)
+		rq->idle_stamp = rq_clock(rq);
+
 	/*
 	 * rq is about to be idle, check if we need to update the
 	 * lost_idle_time of clock_pelt
Signature · 1 line
2.34.1
#3 lore raw

Hello Masahito,

Quoted 8 lines · Masahito S

update_rq_avg_idle(), called from put_prev_task_idle(), computes rq->avg_idle as rq_clock() - rq->idle_stamp. However, idle_stamp is only set by sched_balance_newidle() when a CPU enters CPU_NEWLY_IDLE through the fair class path. When the idle task is preempted without sched_balance_newidle() having run (boot, hotplug, sched class transitions), idle_stamp remains 0, producing a delta equal to rq_clock() — a value in the billions of nanoseconds — which saturates avg_idle at 2 * max_idle_balance_cost.

But these are rare cases right?

Hotplug would anyways trigger a load balance when the CPU comes online and the avg_idle will stabilize thereafter.

Boot happens once so it should be fine to saturate the counter at boot, and idle task being preempted implies we are calling put_prev_task_idle(). For idle task to be picked again it has to go though the pick for rest of the scheduler classes which would do a newidle balance at fair right?

Maybe there is some case with sched-ext where this saturates but then the counter only becomes relevant when the sched-ext scheduler is unloaded.

With PROXY_EXEC, we do switch to idle between schedule() to take the current tasks off the CPU so this does have some merit.

Quoted 38 lines

This inflated avg_idle prevents sched_balance_newidle() from early-returning (fair.c: avg_idle < max_newidle_lb_cost check), making it overly aggressive. The resulting excess newidle migrations override wake-time placement decisions made by select_idle_sibling(), degrading cache locality that careful placement (recent_used_cpu, select_idle_core, etc.) is designed to preserve.

Fix this by:

1. Adding an idle_stamp validity guard to update_rq_avg_idle(), so that a zero idle_stamp is never used as a timestamp.

2. Setting idle_stamp in set_next_task_idle() when it has not already been set by sched_balance_newidle(). This establishes a symmetric idle entry/exit contract: set_next_task_idle() marks the start of the idle period, put_prev_task_idle() measures and records it via update_rq_avg_idle().

The entry hook preserves idle_stamp if sched_balance_newidle() has already set it, maintaining the existing semantic where balance-attempt duration is included in the idle measurement.

---
Changes in v2:
- Added missing Signed-off-by tag (no functional changes). Thanks to Eric Naim and Christian Loehle for pointing this out.

 kernel/sched/core.c | 3 +++
 kernel/sched/idle.c | 3 +++
 2 files changed, 6 insertions(+)
kernel/sched/core.c +0 -0
index 496dff740d..ec801f731c 100644

kernel/sched/core.c static inline void ttwu_do_wakeup(struct task_struct *p) · line 3638

Quoted 6 lines
kernel/sched/core.c +3 -0
@@ -3633,6 +3633,9 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
 
 void update_rq_avg_idle(struct rq *rq)
 {
+       if (!rq->idle_stamp)
+               return;
+

I think this makes sense since we can be forced into idle and we don't want to account that.

Quoted 8 lines
kernel/sched/core.c +0 -0
        u64 delta = rq_clock(rq) - rq->idle_stamp;
        u64 max = 2*rq->max_idle_balance_cost;
 
kernel/sched/idle.c +0 -0
index a83be0c834..9ceb7e6224 100644

kernel/sched/idle.c static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir · line 496

Quoted 6 lines
kernel/sched/idle.c +3 -0
@@ -491,6 +491,9 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir
        schedstat_inc(rq->sched_goidle);
        next->se.exec_start = rq_clock_task(rq);
 
+       if (!rq->idle_stamp)
+               rq->idle_stamp = rq_clock(rq);
+

I don't think this is required because we can switch the donor to idle task for PROXY_EXEC and we don't want to account that as a short idle time unless there is another case I'm missing.

Quoted 3 lines
kernel/sched/idle.c +0 -0
        /*
         * rq is about to be idle, check if we need to update the
         * lost_idle_time of clock_pelt
Signature · 2 lines
Thanks and Regards,
Prateek
#4 lore raw
In reply to Masahito S
Quoted 27 lines · Masahito S

update_rq_avg_idle(), called from put_prev_task_idle(), computes rq->avg_idle as rq_clock() - rq->idle_stamp. However, idle_stamp is only set by sched_balance_newidle() when a CPU enters CPU_NEWLY_IDLE through the fair class path. When the idle task is preempted without sched_balance_newidle() having run (boot, hotplug, sched class transitions), idle_stamp remains 0, producing a delta equal to rq_clock() — a value in the billions of nanoseconds — which saturates avg_idle at 2 * max_idle_balance_cost.

This inflated avg_idle prevents sched_balance_newidle() from early-returning (fair.c: avg_idle < max_newidle_lb_cost check), making it overly aggressive. The resulting excess newidle migrations override wake-time placement decisions made by select_idle_sibling(), degrading cache locality that careful placement (recent_used_cpu, select_idle_core, etc.) is designed to preserve.

Fix this by:

1. Adding an idle_stamp validity guard to update_rq_avg_idle(), so that a zero idle_stamp is never used as a timestamp.

2. Setting idle_stamp in set_next_task_idle() when it has not already been set by sched_balance_newidle(). This establishes a symmetric idle entry/exit contract: set_next_task_idle() marks the start of the idle period, put_prev_task_idle() measures and records it via update_rq_avg_idle().

Quoted 5 lines

The entry hook preserves idle_stamp if sched_balance_newidle() has already set it, maintaining the existing semantic where balance-attempt duration is included in the idle measurement.

Should this have

Fixes 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an

idle CPU")

Quoted 37 lines

---
Changes in v2:
- Added missing Signed-off-by tag (no functional changes). Thanks to Eric Naim and Christian Loehle for pointing this out.

 kernel/sched/core.c | 3 +++
 kernel/sched/idle.c | 3 +++
 2 files changed, 6 insertions(+)
kernel/sched/core.c +3 -0
index 496dff740d..ec801f731c 100644
@@ -3633,6 +3633,9 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
 
 void update_rq_avg_idle(struct rq *rq)
 {
+	if (!rq->idle_stamp)
+		return;
+
 	u64 delta = rq_clock(rq) - rq->idle_stamp;
 	u64 max = 2*rq->max_idle_balance_cost;
 
kernel/sched/idle.c +3 -0
index a83be0c834..9ceb7e6224 100644
@@ -491,6 +491,9 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir
 	schedstat_inc(rq->sched_goidle);
 	next->se.exec_start = rq_clock_task(rq);
 
+	if (!rq->idle_stamp)
+		rq->idle_stamp = rq_clock(rq);
+
 	/*
 	 * rq is about to be idle, check if we need to update the
 	 * lost_idle_time of clock_pelt
Signature · 2 lines
Regards,
  Eric
maintainer
#5 lore raw
In reply to Masahito S
Quoted 7 lines · Masahito S

update_rq_avg_idle(), called from put_prev_task_idle(), computes rq->avg_idle as rq_clock() - rq->idle_stamp. However, idle_stamp is only set by sched_balance_newidle() when a CPU enters CPU_NEWLY_IDLE through the fair class path. When the idle task is preempted without sched_balance_newidle() having run (boot, hotplug, sched class transitions), idle_stamp remains 0, producing a delta equal to

Having avg_idle sets to 2*rq->max_idle_balance_cost after boot or hotplug seems a good initial value

what do you mean by sched class transition ? pick_next_task_fair()-> sched_balance_newidle() is always called before picking an idle task

Quoted 21 lines

rq_clock() — a value in the billions of nanoseconds — which saturates avg_idle at 2 * max_idle_balance_cost.

This inflated avg_idle prevents sched_balance_newidle() from early-returning (fair.c: avg_idle < max_newidle_lb_cost check), making it overly aggressive. The resulting excess newidle migrations override wake-time placement decisions made by select_idle_sibling(), degrading cache locality that careful placement (recent_used_cpu, select_idle_core, etc.) is designed to preserve.

Fix this by:

1. Adding an idle_stamp validity guard to update_rq_avg_idle(), so that a zero idle_stamp is never used as a timestamp.

2. Setting idle_stamp in set_next_task_idle() when it has not already been set by sched_balance_newidle(). This establishes a symmetric idle entry/exit contract: set_next_task_idle() marks the start of the idle period, put_prev_task_idle() measures and records it via update_rq_avg_idle().

Quoted 3 lines

The entry hook preserves idle_stamp if sched_balance_newidle() has already set it, maintaining the existing semantic where balance-attempt duration is included in the idle measurement.

Signed-off is missing

Quoted 36 lines
---
 kernel/sched/core.c | 3 +++
 kernel/sched/idle.c | 3 +++
 2 files changed, 6 insertions(+)
kernel/sched/core.c +3 -0
index 496dff740d..ec801f731c 100644
@@ -3633,6 +3633,9 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
 
 void update_rq_avg_idle(struct rq *rq)
 {
+       if (!rq->idle_stamp)
+               return;
+
        u64 delta = rq_clock(rq) - rq->idle_stamp;
        u64 max = 2*rq->max_idle_balance_cost;
 
kernel/sched/idle.c +3 -1
index a83be0c834..9ceb7e6224 100644
@@ -491,6 +491,9 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir
        schedstat_inc(rq->sched_goidle);
        next->se.exec_start = rq_clock_task(rq);
 
+       if (!rq->idle_stamp)
+               rq->idle_stamp = rq_clock(rq);
+
        /*
         * rq is about to be idle, check if we need to update the
         * lost_idle_time of clock_pelt
--

2.34.1