Skip to contentSkip to conversation

sched/core: Tweak wait_task_inactive() to force dequeue sched_delayed tasks

PATCH Applied John Stultz 4 participants 7 messages Apr 29, 2025 – May 8, 2025 Scheduler

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 sched/core: Tweak wait_task_inactive() to force dequeue sched_delayed tasks

It was reported that in 6.12, smpboot_create_threads() was taking much longer then in 6.6.

I narrowed down the call path to:
smpboot_create_threads()
-> kthread_create_on_cpu()

    -> kthread_bind()
       -> __kthread_bind_mask()
          ->wait_task_inactive()

Where in wait_task_inactive() we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.

I noticed disabling the DELAY_DEQUEUE sched feature recovered the performance, and it seems the newly create tasks are usually sched_delayed and left on the runqueue.

So in wait_task_inactive() when we see the task p->se.sched_delayed, manually dequeue the sched_delayed task with DEQUEUE_DELAYED, so we don't have to constantly wait a tick.

This seems to work, but I've only lightly tested it, so I'd love close review and feedback in case I've missed something in wait_task_inactive(), or if there is a simpler alternative approach.

NOTE: Peter did highlight[1] his general distaste for the kthread_bind() through wait_task_inactive() functions, which suggests a deeper rework might be better, but I'm not familiar enough with all its users to have a sense of how that might be done, and this fix seems to address the problem and be more easily backported to 6.12-stable, so I wanted to submit it again, as a potentially more short-term solution.

[1]: https://lore.kernel.org/lkml/[email protected]/

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Ben Segall <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc K Prateek Nayak <[email protected]>
Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")
Reported-by [email protected]

---
v2:
* Rework & simplify the check as suggested by K Prateek Nayak
* Added Reported-by tag for proper attribution
v3:
* Add Fixed-by: and Tested-by tag suggested by Prateek

---
 kernel/sched/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/core.c +6 -0
index c81cf642dba05..b986cd2fb19b7 100644
@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
 		 * just go back and repeat.
 		 */
 		rq = task_rq_lock(p, &rf);
+		/*
+		 * If task is sched_delayed, force dequeue it, to avoid always
+		 * hitting the tick timeout in the queued case
+		 */
+		if (p->se.sched_delayed)
+			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		trace_sched_wait_task(p);
 		running = task_on_cpu(rq, p);
 		queued = task_on_rq_queued(p);
Signature · 1 line
2.49.0.901.g37484f566f-goog
#1 lore raw

Hi John,

Quoted 10 lines · On Tue, Apr 29, 2025 at 08:07:26AM -0700 John Stultz wrote:

It was reported that in 6.12, smpboot_create_threads() was taking much longer then in 6.6.

I narrowed down the call path to:
smpboot_create_threads()
-> kthread_create_on_cpu()

    -> kthread_bind()
       -> __kthread_bind_mask()
          ->wait_task_inactive()
Quoted 8 lines

Where in wait_task_inactive() we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.

I noticed disabling the DELAY_DEQUEUE sched feature recovered the performance, and it seems the newly create tasks are usually sched_delayed and left on the runqueue.

This seems odd to me. Maybe I'm just misunderstanding something but I don't see how newly created tasks should have accumulated enough runtime to have negative lag that needs to be decayed.

That said, I think it does make sense to dequeue in this case.

Cheers,
Phil

Quoted 67 lines

So in wait_task_inactive() when we see the task p->se.sched_delayed, manually dequeue the sched_delayed task with DEQUEUE_DELAYED, so we don't have to constantly wait a tick.

This seems to work, but I've only lightly tested it, so I'd love close review and feedback in case I've missed something in wait_task_inactive(), or if there is a simpler alternative approach.

NOTE: Peter did highlight[1] his general distaste for the kthread_bind() through wait_task_inactive() functions, which suggests a deeper rework might be better, but I'm not familiar enough with all its users to have a sense of how that might be done, and this fix seems to address the problem and be more easily backported to 6.12-stable, so I wanted to submit it again, as a potentially more short-term solution.

[1]: https://lore.kernel.org/lkml/[email protected]/

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Vincent Guittot <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Ben Segall <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc K Prateek Nayak <[email protected]>
Fixes 152e11f6df29 ("sched/fair: Implement delayed dequeue")
Reported-by [email protected]

---
v2:
* Rework & simplify the check as suggested by K Prateek Nayak
* Added Reported-by tag for proper attribution
v3:
* Add Fixed-by: and Tested-by tag suggested by Prateek

---
 kernel/sched/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/core.c +6 -0
index c81cf642dba05..b986cd2fb19b7 100644
@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
 		 * just go back and repeat.
 		 */
 		rq = task_rq_lock(p, &rf);
+		/*
+		 * If task is sched_delayed, force dequeue it, to avoid always
+		 * hitting the tick timeout in the queued case
+		 */
+		if (p->se.sched_delayed)
+			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		trace_sched_wait_task(p);
 		running = task_on_cpu(rq, p);
 		queued = task_on_rq_queued(p);
Signature · 1 line
2.49.0.901.g37484f566f-goog
maintainer
#2 lore raw
Quoted 22 lines · Phil Auld

Hi John,

Quoted 10 lines · On Tue, Apr 29, 2025 at 08:07:26AM -0700 John Stultz wrote:

It was reported that in 6.12, smpboot_create_threads() was taking much longer then in 6.6.

I narrowed down the call path to:
smpboot_create_threads()
-> kthread_create_on_cpu()

    -> kthread_bind()
       -> __kthread_bind_mask()
          ->wait_task_inactive()
Quoted 8 lines

Where in wait_task_inactive() we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.

I noticed disabling the DELAY_DEQUEUE sched feature recovered the performance, and it seems the newly create tasks are usually sched_delayed and left on the runqueue.

Quoted 5 lines

This seems odd to me. Maybe I'm just misunderstanding something but I don't see how newly created tasks should have accumulated enough runtime to have negative lag that needs to be decayed.

That said, I think it does make sense to dequeue in this case.

Well, they start at 0, any runtime will likely push them negative.

#3 lore raw
Quoted 23 lines · On Wed, Apr 30, 2025 at 02:44:25PM +0200 Peter Zijlstra wrote:
Quoted 22 lines · Phil Auld

Hi John,

Quoted 18 lines · On Tue, Apr 29, 2025 at 08:07:26AM -0700 John Stultz wrote:

It was reported that in 6.12, smpboot_create_threads() was taking much longer then in 6.6.

I narrowed down the call path to:
smpboot_create_threads()
-> kthread_create_on_cpu()

    -> kthread_bind()
       -> __kthread_bind_mask()
          ->wait_task_inactive()

Where in wait_task_inactive() we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.

I noticed disabling the DELAY_DEQUEUE sched feature recovered the performance, and it seems the newly create tasks are usually sched_delayed and left on the runqueue.

Quoted 8 lines
Quoted 5 lines

This seems odd to me. Maybe I'm just misunderstanding something but I don't see how newly created tasks should have accumulated enough runtime to have negative lag that needs to be decayed.

That said, I think it does make sense to dequeue in this case.

Well, they start at 0, any runtime will likely push them negative.

I thought they "made a request" and got a slice when entering the competition so would not immediately go negative when executing. It's now been a while since I read the paper though...

Starting at 0 (service that it ought to have is none) and going immediately negative seems to imply never having positive lag. But, like I said, probably just misunderstanding something :)

Cheers,
Phil

maintainer
#4 lore raw
In reply to John Stultz
Quoted 15 lines · John Stultz

It was reported that in 6.12, smpboot_create_threads() was taking much longer then in 6.6.

I narrowed down the call path to:
smpboot_create_threads()
-> kthread_create_on_cpu()

    -> kthread_bind()
       -> __kthread_bind_mask()
          ->wait_task_inactive()

Where in wait_task_inactive() we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.

Quoted 8 lines

I noticed disabling the DELAY_DEQUEUE sched feature recovered the performance, and it seems the newly create tasks are usually sched_delayed and left on the runqueue.

So in wait_task_inactive() when we see the task p->se.sched_delayed, manually dequeue the sched_delayed task with DEQUEUE_DELAYED, so we don't have to constantly wait a tick.

---

(that is, I'll trim the Changelog a this point, seeing how the rest is 'discussion')

Quoted 4 lines

This seems to work, but I've only lightly tested it, so I'd love close review and feedback in case I've missed something in wait_task_inactive(), or if there is a simpler alternative approach.

There might be. I suspect:

	queued = task_on_rq_queued() && !p->se.sched_delayed;

might just work, but that is indeed pushing things quite far. That gets us into the position of changing ->cpus_allowed while still enqueued, and while it all might just work out, it is fairly tricky and not worth the mental pain.

Quoted 7 lines

NOTE: Peter did highlight[1] his general distaste for the kthread_bind() through wait_task_inactive() functions, which suggests a deeper rework might be better, but I'm not familiar enough with all its users to have a sense of how that might be done, and this fix seems to address the problem and be more easily backported to 6.12-stable, so I wanted to submit it again, as a potentially more short-term solution.

Right, so my distaste is with wait_task_inactive() for basically random waiting for the condition to become true. The neater solution would be a completion of sorts, but then we need the dequeue path to do a wakeup and urgh.

So yeah, this is what we have.

Quoted 7 lines
kernel/sched/core.c +0 -0
index c81cf642dba05..b986cd2fb19b7 100644
@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
 		 * just go back and repeat.
 		 */

kernel/sched/core.c unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state · line 2294

Quoted 10 lines
kernel/sched/core.c +6 -0
 		rq = task_rq_lock(p, &rf);
+		/*
+		 * If task is sched_delayed, force dequeue it, to avoid always
+		 * hitting the tick timeout in the queued case
+		 */
+		if (p->se.sched_delayed)
+			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		trace_sched_wait_task(p);
 		running = task_on_cpu(rq, p);
 		queued = task_on_rq_queued(p);

Lets just do this. I'll to stick it in queue/sched/core.

#5 lore raw
Quoted 25 lines · Peter Zijlstra
Quoted 15 lines · John Stultz

It was reported that in 6.12, smpboot_create_threads() was taking much longer then in 6.6.

I narrowed down the call path to:
smpboot_create_threads()
-> kthread_create_on_cpu()

    -> kthread_bind()
       -> __kthread_bind_mask()
          ->wait_task_inactive()

Where in wait_task_inactive() we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.

Quoted 8 lines

I noticed disabling the DELAY_DEQUEUE sched feature recovered the performance, and it seems the newly create tasks are usually sched_delayed and left on the runqueue.

So in wait_task_inactive() when we see the task p->se.sched_delayed, manually dequeue the sched_delayed task with DEQUEUE_DELAYED, so we don't have to constantly wait a tick.

Quoted 5 lines

---

(that is, I'll trim the Changelog a this point, seeing how the rest is 'discussion')

Ah, thanks. I've noted you tweaking my commit messages before merging, so I'll try to do better about leaving ephemeral notes (and Cc lists, apparently) after the "---" fold.
My apologies for the trouble!

Quoted 11 lines
Quoted 11 lines
kernel/sched/core.c +3 -0
index c81cf642dba05..b986cd2fb19b7 100644
@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
               * just go back and repeat.
               */
              rq = task_rq_lock(p, &rf);
+             /*
+              * If task is sched_delayed, force dequeue it, to avoid always
+              * hitting the tick timeout in the queued case
Quoted 8 lines

kernel/sched/core.c unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state · line 2294

Quoted 6 lines
kernel/sched/core.c +3 -0
+              */
+             if (p->se.sched_delayed)
+                     dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
              trace_sched_wait_task(p);
              running = task_on_cpu(rq, p);
              queued = task_on_rq_queued(p);

Lets just do this. I'll to stick it in queue/sched/core.

Ok, thanks so much!
-john

merge bot Applied
#6 lore raw
In reply to John Stultz
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     b7ca5743a2604156d6083b88cefacef983f3a3a6
Gitweb:        https://git.kernel.org/tip/b7ca5743a2604156d6083b88cefacef983f3a3a6
Author:        John Stultz <[email protected]>
AuthorDate:    Tue, 29 Apr 2025 08:07:26 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 30 Apr 2025 14:45:41 +02:00

sched/core: Tweak wait_task_inactive() to force dequeue sched_delayed tasks

It was reported that in 6.12, smpboot_create_threads() was taking much longer then in 6.6.

I narrowed down the call path to:
smpboot_create_threads()
-> kthread_create_on_cpu()

    -> kthread_bind()
       -> __kthread_bind_mask()
          ->wait_task_inactive()

Where in wait_task_inactive() we were regularly hitting the queued case, which sets a 1 tick timeout, which when called multiple times in a row, accumulates quickly into a long delay.

I noticed disabling the DELAY_DEQUEUE sched feature recovered the performance, and it seems the newly create tasks are usually sched_delayed and left on the runqueue.

So in wait_task_inactive() when we see the task p->se.sched_delayed, manually dequeue the sched_delayed task with DEQUEUE_DELAYED, so we don't have to constantly wait a tick.

---
 kernel/sched/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/core.c +6 -0
index 79692f8..a3507ed 100644
@@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
 		 * just go back and repeat.
 		 */
 		rq = task_rq_lock(p, &rf);
+		/*
+		 * If task is sched_delayed, force dequeue it, to avoid always
+		 * hitting the tick timeout in the queued case
+		 */
+		if (p->se.sched_delayed)
+			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 		trace_sched_wait_task(p);
 		running = task_on_cpu(rq, p);
 		queued = task_on_rq_queued(p);