Skip to contentSkip to conversation

sched/core: Skip rq->avg_idle update without a valid idle_stamp

PATCH Superseded Shubhang Kaushik (Ampere) 3 participants 4 messages Aug 7 Scheduler

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 sched/core: Skip rq->avg_idle update without a valid idle_stamp

Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and into put_prev_task_idle(), so that the idle interval is consumed whenever the idle task is switched out.

The wakeup-side accounting that it replaced only updated rq->avg_idle when rq->idle_stamp was non-zero. The new helper lost that validity check and unconditionally computes:

	rq_clock(rq) - rq->idle_stamp

If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is not a valid idle duration and can immediately drive rq->avg_idle to its clamp.

This can happen when the scheduler switches to the idle task through a path that did not set rq->idle_stamp via newidle_balance(), for example during find_proxy_task() or force-idling.

Restore the idle_stamp validity check in update_rq_avg_idle() and skip the rq->avg_idle update when there is no measured idle interval.

Fixes 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU")

---
Temporary tracing under hackbench load confirmed that update_rq_avg_idle() can be reached with rq->idle_stamp == 0. Hackbench showed no material regression versus v7.2-rc5 mainline.

Related discussion:
https://lore.kernel.org/r/[email protected]

This is a narrower variant of the earlier proposal. It keeps the rq->idle_stamp guard in update_rq_avg_idle(), but intentionally does not stamp idle entry from set_next_task_idle(), preserving the existing newidle accounting model and avoiding forced/proxy idle accounting concerns.
---
Changes in v2:
- Add Reviewed-by from Prateek.
- Mention find_proxy_task() and force-idling as examples of paths that

    can switch to the idle task without a valid rq->idle_stamp.

- Cc John Stultz.

Link to v1: https://lore.kernel.org/r/[email protected]

---
 kernel/sched/core.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
kernel/sched/core.c +8 -2
index 96226707c2f6135341aa779b8262f113e103d8ad..d8c9a80ffa83ac680bb479f67357ac1ec4a6d55e 100644
@@ -3732,11 +3732,17 @@ static inline void ttwu_do_wakeup(struct task_struct *p)
 
 void update_rq_avg_idle(struct rq *rq)
 {
-	u64 delta = rq_clock(rq) - rq->idle_stamp;
-	u64 max = 2*rq->max_idle_balance_cost;
+	u64 idle_stamp = rq->idle_stamp;
+	u64 delta, max;
+
+	if (unlikely(!idle_stamp))
+		return;
+
+	delta = rq_clock(rq) - idle_stamp;
 
 	update_avg(&rq->avg_idle, delta);
 
+	max = 2 * rq->max_idle_balance_cost;
 	if (rq->avg_idle > max)
 		rq->avg_idle = max;
 	rq->idle_stamp = 0;

---
base-commit: c0a27675eaf08255017b3cabc28c99c0cd71f468
change-id: 20260728-master-55cd7cc13290

Best regards,

Signature · 1 line
Shubhang Kaushik (Ampere) <[email protected]>
Acked-by on v2 1/1
#1 lore raw
Quoted 31 lines · Shubhang Kaushik (Ampere)

Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and into put_prev_task_idle(), so that the idle interval is consumed whenever the idle task is switched out.

The wakeup-side accounting that it replaced only updated rq->avg_idle when rq->idle_stamp was non-zero. The new helper lost that validity check and unconditionally computes:

        rq_clock(rq) - rq->idle_stamp

If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is not a valid idle duration and can immediately drive rq->avg_idle to its clamp.

This can happen when the scheduler switches to the idle task through a path that did not set rq->idle_stamp via newidle_balance(), for example during find_proxy_task() or force-idling.

Restore the idle_stamp validity check in update_rq_avg_idle() and skip the rq->avg_idle update when there is no measured idle interval.

Fixes 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU")

---
Temporary tracing under hackbench load confirmed that update_rq_avg_idle() can be reached with rq->idle_stamp == 0. Hackbench showed no material regression versus v7.2-rc5 mainline.

Quoted 8 lines

Related discussion:
https://lore.kernel.org/r/[email protected]

This is a narrower variant of the earlier proposal. It keeps the rq->idle_stamp guard in update_rq_avg_idle(), but intentionally does not stamp idle entry from set_next_task_idle(), preserving the existing newidle accounting model and avoiding forced/proxy idle accounting concerns.

Thanks for sending this out!

#2 lore raw
In reply to Shubhang Kaushik (Ampere)
Quoted 3 lines · Shubhang Kaushik (Ampere)

This can happen when the scheduler switches to the idle task through a path that did not set rq->idle_stamp via newidle_balance(), for example during find_proxy_task() or force-idling.

There is a path that needs no config option, and I suspect it is what your hackbench tracing actually hit: sched_balance_newidle() returns before it stamps.

	if (this_rq->ttwu_pending)
		return 0;
	...
	this_rq->idle_stamp = rq_clock(this_rq);

The early return sits above the assignment, and its own comment says the task will be enqueued when switching to idle. So the rq goes idle with idle_stamp == 0 and leaves idle again as soon as the pending wakeup is processed, which is exactly when put_prev_task_idle() consumes the stamp. A wakeup-heavy load like hackbench should hit that constantly, whereas find_proxy_task() and force-idling need proxy exec or CONFIG_SCHED_CORE. (Force-idle is the sched_core_enabled(rq) return at the idle: label in pick_next_task_fair(), which is also above the newidle call.)

Worth naming ttwu_pending in the changelog? It makes the bug config-independent, which seems relevant given the Fixes: tag.

The fix itself looks equivalent to what 4b603f1551a7 removed: the old ttwu_do_activate() code was wrapped in if (rq->idle_stamp), and skipping the trailing rq->idle_stamp = 0 is a no-op when the stamp is already zero. update_rq_avg_idle() has just the one caller, so nothing else changes.

Nit: unlikely(!idle_stamp) may be the wrong way round if ttwu_pending is the common trigger.

Thanks,
Zhan Xusheng

author
#3 lore raw

Hi Zhan,

Quoted 24 lines · Zhan Xusheng
Quoted 3 lines · Shubhang Kaushik (Ampere)

This can happen when the scheduler switches to the idle task through a path that did not set rq->idle_stamp via newidle_balance(), for example during find_proxy_task() or force-idling.

There is a path that needs no config option, and I suspect it is what your hackbench tracing actually hit: sched_balance_newidle() returns before it stamps.

	if (this_rq->ttwu_pending)
		return 0;
	...
	this_rq->idle_stamp = rq_clock(this_rq);

The early return sits above the assignment, and its own comment says the task will be enqueued when switching to idle. So the rq goes idle with idle_stamp == 0 and leaves idle again as soon as the pending wakeup is processed, which is exactly when put_prev_task_idle() consumes the stamp. A wakeup-heavy load like hackbench should hit that constantly, whereas find_proxy_task() and force-idling need proxy exec or CONFIG_SCHED_CORE. (Force-idle is the sched_core_enabled(rq) return at the idle: label in pick_next_task_fair(), which is also above the newidle call.)

Quoted 9 lines

Worth naming ttwu_pending in the changelog? It makes the bug config-independent, which seems relevant given the Fixes: tag.

The fix itself looks equivalent to what 4b603f1551a7 removed: the old ttwu_do_activate() code was wrapped in if (rq->idle_stamp), and skipping the trailing rq->idle_stamp = 0 is a no-op when the stamp is already zero. update_rq_avg_idle() has just the one caller, so nothing else changes.

Thanks for the suggestion, that makes sense. The ttwu_pending path is a better example since it does not depend on proxy exec or core scheduling, and it is likely what hackbench hit.

I will update the changelog to lead with sched_balance_newidle() returning before setting idle_stamp when ttwu_pending is set.

Quoted 2 lines

Nit: unlikely(!idle_stamp) may be the wrong way round if ttwu_pending is the common trigger.

Ack.

Quoted 4 lines

Thanks,
Zhan Xusheng

Regards,
Shubhang Kaushik