sched/core: Skip rq->avg_idle update without a valid idle_stamp
PATCH v2 1/1 sched/core: Skip rq->avg_idle update without a valid idle_stamp
Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and into put_prev_task_idle(), so that the idle interval is consumed whenever the idle task is switched out.
The wakeup-side accounting that it replaced only updated rq->avg_idle when rq->idle_stamp was non-zero. The new helper lost that validity check and unconditionally computes:
rq_clock(rq) - rq->idle_stamp
If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is not a valid idle duration and can immediately drive rq->avg_idle to its clamp.
This can happen when the scheduler switches to the idle task through a path that did not set rq->idle_stamp via newidle_balance(), for example during find_proxy_task() or force-idling.
Restore the idle_stamp validity check in update_rq_avg_idle() and skip the rq->avg_idle update when there is no measured idle interval.
---
Temporary tracing under hackbench load confirmed that update_rq_avg_idle() can be reached with rq->idle_stamp == 0. Hackbench showed no material regression versus v7.2-rc5 mainline.
Related discussion:
https://lore.kernel.org/r/[email protected]
This is a narrower variant of the earlier proposal. It keeps the rq->idle_stamp guard in update_rq_avg_idle(), but intentionally does not stamp idle entry from set_next_task_idle(), preserving the existing newidle accounting model and avoiding forced/proxy idle accounting concerns.
---
Changes in v2:
- Add Reviewed-by from Prateek.
- Mention find_proxy_task() and force-idling as examples of paths that
can switch to the idle task without a valid rq->idle_stamp.
- Cc John Stultz.
Link to v1: https://lore.kernel.org/r/[email protected]
--- kernel/sched/core.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
kernel/sched/core.c +8 -2
@@ -3732,11 +3732,17 @@ static inline void ttwu_do_wakeup(struct task_struct *p)void update_rq_avg_idle (struct rq *rq) { -u64 delta =rq_clock (rq) - rq->idle_stamp; -u64 max =2 *rq->max_idle_balance_cost; +u64 idle_stamp = rq->idle_stamp; +u64 delta, max; + +if (unlikely (!idle_stamp)) +return ; + + delta =rq_clock (rq) - idle_stamp;update_avg (&rq->avg_idle, delta); + max =2 * rq->max_idle_balance_cost;if (rq->avg_idle > max) rq->avg_idle = max; rq->idle_stamp =0 ;
---
base-commit: c0a27675eaf08255017b3cabc28c99c0cd71f468
change-id: 20260728-master-55cd7cc13290
Best regards,
Signature · 1 line
Shubhang Kaushik (Ampere) <[email protected]>
Quoted 31 lines · Shubhang Kaushik (Ampere)
Commit
4b603f1551a73("sched: Updaterq->avg_idlewhen a task is moved to an idle CPU") movedrq->avg_idleaccounting out of the wakeup path and intoput_prev_task_idle(), so that the idle interval is consumed whenever the idle task is switched out.The wakeup-side accounting that it replaced only updated
rq->avg_idlewhenrq->idle_stampwas non-zero. The new helper lost that validity check and unconditionally computes:rq_clock(rq) - rq->idle_stampIf
rq->idle_stampis zero, this usesrq_clock(rq)as the sample. That is not a valid idle duration and can immediately driverq->avg_idleto its clamp.This can happen when the scheduler switches to the idle task through a path that did not set
rq->idle_stampvianewidle_balance(), for example duringfind_proxy_task()or force-idling.Restore the
idle_stampvalidity check inupdate_rq_avg_idle()and skip therq->avg_idleupdate when there is no measured idle interval.Fixes 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU")Reviewed-by K Prateek Nayak <[email protected]>Signed-off-by Shubhang Kaushik (Ampere) <[email protected]>---
Temporary tracing under hackbench load confirmed thatupdate_rq_avg_idle()can be reached withrq->idle_stamp== 0. Hackbench showed no material regression versus v7.2-rc5 mainline.
Quoted 8 lines
Related discussion:
https://lore.kernel.org/r/[email protected]This is a narrower variant of the earlier proposal. It keeps the
rq->idle_stampguard inupdate_rq_avg_idle(), but intentionally does not stamp idle entry fromset_next_task_idle(), preserving the existing newidle accounting model and avoiding forced/proxy idle accounting concerns.
Thanks for sending this out!
Quoted 3 lines · Shubhang Kaushik (Ampere)
This can happen when the scheduler switches to the idle task through a path that did not set
rq->idle_stampvianewidle_balance(), for example duringfind_proxy_task()or force-idling.
There is a path that needs no config option, and I suspect it is what your hackbench tracing actually hit: sched_balance_newidle() returns before it stamps.
if (this_rq->ttwu_pending) return 0; ... this_rq->idle_stamp = rq_clock(this_rq);
The early return sits above the assignment, and its own comment says the task will be enqueued when switching to idle. So the rq goes idle with idle_stamp == 0 and leaves idle again as soon as the pending wakeup is processed, which is exactly when put_prev_task_idle() consumes the stamp. A wakeup-heavy load like hackbench should hit that constantly, whereas find_proxy_task() and force-idling need proxy exec or CONFIG_SCHED_CORE. (Force-idle is the sched_core_enabled(rq) return at the idle: label in pick_next_task_fair(), which is also above the newidle call.)
Worth naming ttwu_pending in the changelog? It makes the bug config-independent, which seems relevant given the Fixes: tag.
The fix itself looks equivalent to what 4b603f1551a7 removed: the old ttwu_do_activate() code was wrapped in if (rq->idle_stamp), and skipping the trailing rq->idle_stamp = 0 is a no-op when the stamp is already zero. update_rq_avg_idle() has just the one caller, so nothing else changes.
Nit: unlikely(!idle_stamp) may be the wrong way round if ttwu_pending is the common trigger.
Thanks,
Zhan Xusheng
Hi Zhan,
Quoted 24 lines · Zhan Xusheng
Quoted 3 lines · Shubhang Kaushik (Ampere)
This can happen when the scheduler switches to the idle task through a path that did not set
rq->idle_stampvianewidle_balance(), for example duringfind_proxy_task()or force-idling.There is a path that needs no config option, and I suspect it is what your hackbench tracing actually hit:
sched_balance_newidle()returns before it stamps.if (this_rq->ttwu_pending) return 0; ... this_rq->idle_stamp = rq_clock(this_rq);The early return sits above the assignment, and its own comment says the task will be enqueued when switching to idle. So the rq goes idle with
idle_stamp== 0 and leaves idle again as soon as the pending wakeup is processed, which is exactly whenput_prev_task_idle()consumes the stamp. A wakeup-heavy load like hackbench should hit that constantly, whereasfind_proxy_task()and force-idling need proxy exec orCONFIG_SCHED_CORE. (Force-idle is thesched_core_enabled(rq)return at the idle: label inpick_next_task_fair(), which is also above the newidle call.)
Quoted 9 lines
Worth naming
ttwu_pendingin the changelog? It makes the bug config-independent, which seems relevant given the Fixes: tag.The fix itself looks equivalent to what
4b603f1551a7removed: the oldttwu_do_activate()code was wrapped in if (rq->idle_stamp), and skipping the trailingrq->idle_stamp= 0 is a no-op when the stamp is already zero.update_rq_avg_idle()has just the one caller, so nothing else changes.
Thanks for the suggestion, that makes sense. The ttwu_pending path is a better example since it does not depend on proxy exec or core scheduling, and it is likely what hackbench hit.
I will update the changelog to lead with sched_balance_newidle() returning before setting idle_stamp when ttwu_pending is set.
Quoted 2 lines
Nit:
unlikely(!idle_stamp)may be the wrong way round ifttwu_pendingis the common trigger.
Ack.
Quoted 4 lines
Thanks,
Zhan Xusheng
Regards,
Shubhang Kaushik