sched/core: Tweak wait_task_inactive() to force dequeue sched_delayed tasks
PATCH v3 1/1 sched/core: Tweak wait_task_inactive() to force dequeue sched_delayed tasks
It was reported that in 6.12, smpboot_create_threads() was taking much longer then in 6.6.
I narrowed down the call path to:
smpboot_create_threads()
-> kthread_create_on_cpu()
-> kthread_bind()
-> __kthread_bind_mask()
->wait_task_inactive()Where in wait_task_inactive() we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.
I noticed disabling the DELAY_DEQUEUE sched feature recovered the performance, and it seems the newly create tasks are usually sched_delayed and left on the runqueue.
So in wait_task_inactive() when we see the task p->se.sched_delayed, manually dequeue the sched_delayed task with DEQUEUE_DELAYED, so we don't have to constantly wait a tick.
This seems to work, but I've only lightly tested it, so I'd love close review and feedback in case I've missed something in wait_task_inactive(), or if there is a simpler alternative approach.
NOTE: Peter did highlight[1] his general distaste for the kthread_bind() through wait_task_inactive() functions, which suggests a deeper rework might be better, but I'm not familiar enough with all its users to have a sense of how that might be done, and this fix seems to address the problem and be more easily backported to 6.12-stable, so I wanted to submit it again, as a potentially more short-term solution.
[1]: https://lore.kernel.org/lkml/[email protected]/
---
v2:
* Rework & simplify the check as suggested by K Prateek Nayak
* Added Reported-by tag for proper attribution
v3:
* Add Fixed-by: and Tested-by tag suggested by Prateek
--- kernel/sched/core.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sched/core.c +6 -0
@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state* just go back and repeat. */ rq =task_rq_lock (p, &rf); +/* +* If task is sched_delayed, force dequeue it, to avoid always +* hitting the tick timeout in the queued case +*/ +if (p->se.sched_delayed) +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED );trace_sched_wait_task (p); running =task_on_cpu (rq, p); queued =task_on_rq_queued (p);
Signature · 1 line
2.49.0.901.g37484f566f-goog
Hi John,
Quoted 10 lines · On Tue, Apr 29, 2025 at 08:07:26AM -0700 John Stultz wrote:
It was reported that in 6.12,
smpboot_create_threads()was taking much longer then in 6.6.I narrowed down the call path to:
smpboot_create_threads()
->kthread_create_on_cpu()-> kthread_bind() -> __kthread_bind_mask() ->wait_task_inactive()
Quoted 8 lines
Where in
wait_task_inactive()we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.I noticed disabling the
DELAY_DEQUEUEsched feature recovered the performance, and it seems the newly create tasks are usuallysched_delayedand left on the runqueue.
This seems odd to me. Maybe I'm just misunderstanding something but I don't see how newly created tasks should have accumulated enough runtime to have negative lag that needs to be decayed.
That said, I think it does make sense to dequeue in this case.
Cheers,
Phil
Quoted 67 lines
So in
wait_task_inactive()when we see the taskp->se.sched_delayed, manually dequeue thesched_delayedtask withDEQUEUE_DELAYED, so we don't have to constantly wait a tick.This seems to work, but I've only lightly tested it, so I'd love close review and feedback in case I've missed something in
wait_task_inactive(), or if there is a simpler alternative approach.NOTE: Peter did highlight[1] his general distaste for the
kthread_bind()throughwait_task_inactive()functions, which suggests a deeper rework might be better, but I'm not familiar enough with all its users to have a sense of how that might be done, and this fix seems to address the problem and be more easily backported to 6.12-stable, so I wanted to submit it again, as a potentially more short-term solution.[1]: https://lore.kernel.org/lkml/[email protected]/
Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Juri Lelli <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Segall <[email protected]>Cc Mel Gorman <[email protected]>Cc Valentin Schneider <[email protected]>Cc K Prateek Nayak <[email protected]>Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")Reported-by [email protected]Tested-by K Prateek Nayak <[email protected]>Signed-off-by John Stultz <[email protected]>---
v2:
* Rework & simplify the check as suggested by K Prateek Nayak
* Added Reported-by tag for proper attribution
v3:
* Add Fixed-by: and Tested-by tag suggested by Prateek--- kernel/sched/core.c | 6 ++++++ 1 file changed, 6 insertions(+)kernel/sched/core.c +6
-0@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state* just go back and repeat. */ rq =task_rq_lock (p, &rf); +/* +* If task is sched_delayed, force dequeue it, to avoid always +* hitting the tick timeout in the queued case +*/ +if (p->se.sched_delayed) +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED );trace_sched_wait_task (p); running =task_on_cpu (rq, p); queued =task_on_rq_queued (p);Signature · 1 line
2.49.0.901.g37484f566f-goog
Peter Zijlstra
maintainerQuoted 22 lines · Phil Auld
Hi John,
Quoted 10 lines · On Tue, Apr 29, 2025 at 08:07:26AM -0700 John Stultz wrote:
It was reported that in 6.12,
smpboot_create_threads()was taking much longer then in 6.6.I narrowed down the call path to:
smpboot_create_threads()
->kthread_create_on_cpu()-> kthread_bind() -> __kthread_bind_mask() ->wait_task_inactive()Quoted 8 lines
Where in
wait_task_inactive()we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.I noticed disabling the
DELAY_DEQUEUEsched feature recovered the performance, and it seems the newly create tasks are usuallysched_delayedand left on the runqueue.
Quoted 5 lines
This seems odd to me. Maybe I'm just misunderstanding something but I don't see how newly created tasks should have accumulated enough runtime to have negative lag that needs to be decayed.
That said, I think it does make sense to dequeue in this case.
Well, they start at 0, any runtime will likely push them negative.
Quoted 23 lines · On Wed, Apr 30, 2025 at 02:44:25PM +0200 Peter Zijlstra wrote:
Quoted 22 lines · Phil Auld
Hi John,
Quoted 18 lines · On Tue, Apr 29, 2025 at 08:07:26AM -0700 John Stultz wrote:
It was reported that in 6.12,
smpboot_create_threads()was taking much longer then in 6.6.I narrowed down the call path to:
smpboot_create_threads()
->kthread_create_on_cpu()-> kthread_bind() -> __kthread_bind_mask() ->wait_task_inactive()Where in
wait_task_inactive()we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.I noticed disabling the
DELAY_DEQUEUEsched feature recovered the performance, and it seems the newly create tasks are usuallysched_delayedand left on the runqueue.
Quoted 8 lines
Quoted 5 lines
This seems odd to me. Maybe I'm just misunderstanding something but I don't see how newly created tasks should have accumulated enough runtime to have negative lag that needs to be decayed.
That said, I think it does make sense to dequeue in this case.
Well, they start at 0, any runtime will likely push them negative.
I thought they "made a request" and got a slice when entering the competition so would not immediately go negative when executing. It's now been a while since I read the paper though...
Starting at 0 (service that it ought to have is none) and going immediately negative seems to imply never having positive lag. But, like I said, probably just misunderstanding something :)
Cheers,
Phil
Peter Zijlstra
maintainerQuoted 15 lines · John Stultz
It was reported that in 6.12,
smpboot_create_threads()was taking much longer then in 6.6.I narrowed down the call path to:
smpboot_create_threads()
->kthread_create_on_cpu()-> kthread_bind() -> __kthread_bind_mask() ->wait_task_inactive()Where in
wait_task_inactive()we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.
Quoted 8 lines
I noticed disabling the
DELAY_DEQUEUEsched feature recovered the performance, and it seems the newly create tasks are usuallysched_delayedand left on the runqueue.So in
wait_task_inactive()when we see the taskp->se.sched_delayed, manually dequeue thesched_delayedtask withDEQUEUE_DELAYED, so we don't have to constantly wait a tick.
---
(that is, I'll trim the Changelog a this point, seeing how the rest is 'discussion')
Quoted 4 lines
This seems to work, but I've only lightly tested it, so I'd love close review and feedback in case I've missed something in
wait_task_inactive(), or if there is a simpler alternative approach.
There might be. I suspect:
queued = task_on_rq_queued() && !p->se.sched_delayed;
might just work, but that is indeed pushing things quite far. That gets us into the position of changing ->cpus_allowed while still enqueued, and while it all might just work out, it is fairly tricky and not worth the mental pain.
Quoted 7 lines
NOTE: Peter did highlight[1] his general distaste for the
kthread_bind()throughwait_task_inactive()functions, which suggests a deeper rework might be better, but I'm not familiar enough with all its users to have a sense of how that might be done, and this fix seems to address the problem and be more easily backported to 6.12-stable, so I wanted to submit it again, as a potentially more short-term solution.
Right, so my distaste is with wait_task_inactive() for basically random waiting for the condition to become true. The neater solution would be a completion of sorts, but then we need the dequeue path to do a wakeup and urgh.
So yeah, this is what we have.
Quoted 7 lines
kernel/sched/core.c +0
-0@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state* just go back and repeat. */
kernel/sched/core.c unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state · line 2294
Quoted 10 lines
kernel/sched/core.c +6
-0rq =task_rq_lock (p, &rf); +/* +* If task is sched_delayed, force dequeue it, to avoid always +* hitting the tick timeout in the queued case +*/ +if (p->se.sched_delayed) +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED );trace_sched_wait_task (p); running =task_on_cpu (rq, p); queued =task_on_rq_queued (p);
Lets just do this. I'll to stick it in queue/sched/core.
Quoted 25 lines · Peter Zijlstra
Quoted 15 lines · John Stultz
It was reported that in 6.12,
smpboot_create_threads()was taking much longer then in 6.6.I narrowed down the call path to:
smpboot_create_threads()
->kthread_create_on_cpu()-> kthread_bind() -> __kthread_bind_mask() ->wait_task_inactive()Where in
wait_task_inactive()we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.Quoted 8 lines
I noticed disabling the
DELAY_DEQUEUEsched feature recovered the performance, and it seems the newly create tasks are usuallysched_delayedand left on the runqueue.So in
wait_task_inactive()when we see the taskp->se.sched_delayed, manually dequeue thesched_delayedtask withDEQUEUE_DELAYED, so we don't have to constantly wait a tick.
Quoted 5 lines
---
(that is, I'll trim the Changelog a this point, seeing how the rest is 'discussion')
Ah, thanks. I've noted you tweaking my commit messages before merging, so I'll try to do better about leaving ephemeral notes (and Cc lists, apparently) after the "---" fold.
My apologies for the trouble!
Quoted 11 lines
Quoted 11 lines
kernel/sched/core.c +3
-0@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state* just go back and repeat. */ rq =task_rq_lock (p, &rf); +/* +* If task is sched_delayed, force dequeue it, to avoid always +* hitting the tick timeout in the queued case
Quoted 8 lines
kernel/sched/core.c unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state · line 2294
Quoted 6 lines
kernel/sched/core.c +3
-0+*/ +if (p->se.sched_delayed) +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED );trace_sched_wait_task (p); running =task_on_cpu (rq, p); queued =task_on_rq_queued (p);Lets just do this. I'll to stick it in queue/sched/core.
Ok, thanks so much!
-john
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b7ca5743a2604156d6083b88cefacef983f3a3a6 Gitweb: https://git.kernel.org/tip/b7ca5743a2604156d6083b88cefacef983f3a3a6 Author: John Stultz <[email protected]> AuthorDate: Tue, 29 Apr 2025 08:07:26 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 30 Apr 2025 14:45:41 +02:00
sched/core: Tweak wait_task_inactive() to force dequeue sched_delayed tasks
It was reported that in 6.12, smpboot_create_threads() was taking much longer then in 6.6.
I narrowed down the call path to:
smpboot_create_threads()
-> kthread_create_on_cpu()
-> kthread_bind()
-> __kthread_bind_mask()
->wait_task_inactive()Where in wait_task_inactive() we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.
I noticed disabling the DELAY_DEQUEUE sched feature recovered the performance, and it seems the newly create tasks are usually sched_delayed and left on the runqueue.
So in wait_task_inactive() when we see the task p->se.sched_delayed, manually dequeue the sched_delayed task with DEQUEUE_DELAYED, so we don't have to constantly wait a tick.
--- kernel/sched/core.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sched/core.c +6 -0
@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state* just go back and repeat. */ rq =task_rq_lock (p, &rf); +/* +* If task is sched_delayed, force dequeue it, to avoid always +* hitting the tick timeout in the queued case +*/ +if (p->se.sched_delayed) +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED );trace_sched_wait_task (p); running =task_on_cpu (rq, p); queued =task_on_rq_queued (p);