Skip to contentSkip to conversation

sched/core: Fix PSI inconsistent task state splats with DELAY_DEQUEUE

PATCH Applied K Prateek Nayak 4 participants 20 messages Oct 10, 2024 – Oct 28, 2024

Subsystems (2)
lore
Share & tools
lore

K Prateek Nayak

author reviewer
#0 lore raw

cover sched/core: Fix PSI inconsistent task state splats with DELAY_DEQUEUE

After the introduction of DELAY_DEQUEUE, PSI consistently started warning about inconsistent task state early into the boot. This could be root-caused to three issues that the three patches respectively solve:

o PSI signals not being dequeued when the task is blocked, but also delayed since psi_sched_switch() considered "!task_on_rq_queued()" as the task being blocked but a delayed task will remain queued on the runqueue until it is picked again and goes through a full dequeue.

o enqueue_task() not using the ENQUEUE_WAKEUP alongside ENQUEUE_DELAYED in ttwu_runnable(). Since psi_enqueue() only considers (in terms of enqueue flags):

    (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)

... as a wakeup, the lack of ENQUEUE_WAKEUP can misguide psi_enqueue() which only clears TSK_IOWAIT flag on wakeups.

o When a delayed task is migrated by the load balancer, the requeue or the wakeup context may be aware that the task has migrated between it blocking and it waking up. This is necessary to be communicated to PSI which forgoes clearing TSK_IOWAIT since it expects the psi_.*dequeue() to have cleared it during migration.

The series correctly communicates the blocked status of a delayed task to psi_dequeue(), adds the ENQUEUE_WAKEUP flag during a requeue in ttwu_runnable(), re-arranges the psi_enqueue() to be called after a "p->sched_class->enqueue_task()", and notify psi_enqueue() of a migration in delayed state using "p->migration_flags" to maintain the task state consistently.

This series was previously posted as one large diff at
https://lore.kernel.org/lkml/[email protected]/
and was tested by Johannes. The tags on the diff have been carried to this series.

This series is based on:

git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core

at commit 7266f0a6d3bb ("fs/bcachefs: Fix __wait_on_freeing_inode() definition of waitqueue entry")

Any and all feedback is greatly appreciated.

--
K Prateek Nayak (2):
sched/core: Add ENQUEUE_WAKEUP flag alongside ENQUEUE_DELAYED sched/core: Indicate a sched_delayed task was migrated before wakeup

Peter Zijlstra (1):
sched/core: Dequeue PSI signals for blocked tasks that are delayed

 kernel/sched/core.c  | 25 ++++++++++++++++++++++---
 kernel/sched/sched.h |  1 +
 kernel/sched/stats.h | 10 ++++++++++
 3 files changed, 33 insertions(+), 3 deletions(-)
Signature · 1 line
2.34.1
author reviewer
#1 lore raw

PATCH 1/3 sched/core: Dequeue PSI signals for blocked tasks that are delayed

From: Peter Zijlstra <[email protected]>

psi_dequeue() in for blocked task expects psi_sched_switch() to clear the TSK_.*RUNNING PSI flags and set the TSK_IOWAIT flags however psi_sched_switch() uses "!task_on_rq_queued(prev)" to detect if the task is blocked or still runnable which is no longer true with DELAY_DEQUEUE since a blocking task can be left queued on the runqueue.

This can lead to PSI splats similar to:

psi: inconsistent task state! task=... cpu=... psi_flags=4 clear=0 set=4

when the task is requeued since the TSK_RUNNING flag was not cleared when the task was blocked.

Explicitly communicate that the task was blocked to psi_sched_switch() even if it was delayed and is still on the runqueue.

[ prateek: Broke off the relevant part from [1], commit message ]

---
 kernel/sched/core.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/sched/core.c +3 -1
index 43e453ab7e20..88cbfc671fb6 100644
@@ -6537,6 +6537,7 @@ static void __sched notrace __schedule(int sched_mode)
 	 * as a preemption by schedule_debug() and RCU.
 	 */
 	bool preempt = sched_mode > SM_NONE;
+	bool block = false;
 	unsigned long *switch_count;
 	unsigned long prev_state;
 	struct rq_flags rf;
@@ -6622,6 +6623,7 @@ static void __sched notrace __schedule(int sched_mode)
 			 * After this, schedule() must not care about p->state any more.
 			 */
 			block_task(rq, prev, flags);
+			block = true;
 		}
 		switch_count = &prev->nvcsw;
 	}
@@ -6667,7 +6669,7 @@ static void __sched notrace __schedule(int sched_mode)
 
 		migrate_disable_switch(rq, prev);
 		psi_account_irqtime(rq, prev, next);
-		psi_sched_switch(prev, next, !task_on_rq_queued(prev));
+		psi_sched_switch(prev, next, block);
 
 		trace_sched_switch(preempt, prev, next, prev_state);
 
Signature · 1 line
2.34.1
Acked-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] sched/core: Dequeue PSI signals for blocked tasks that are delayed

Quoted 20 lines · K Prateek Nayak

From: Peter Zijlstra <[email protected]>

psi_dequeue() in for blocked task expects psi_sched_switch() to clear the TSK_.*RUNNING PSI flags and set the TSK_IOWAIT flags however psi_sched_switch() uses "!task_on_rq_queued(prev)" to detect if the task is blocked or still runnable which is no longer true with DELAY_DEQUEUE since a blocking task can be left queued on the runqueue.

This can lead to PSI splats similar to:

psi: inconsistent task state! task=... cpu=... psi_flags=4 clear=0 set=4

when the task is requeued since the TSK_RUNNING flag was not cleared when the task was blocked.

Explicitly communicate that the task was blocked to psi_sched_switch() even if it was delayed and is still on the runqueue.

[ prateek: Broke off the relevant part from [1], commit message ]

Quoted 7 lines
author reviewer
#3 lore raw

PATCH 2/3 sched/core: Add ENQUEUE_WAKEUP flag alongside ENQUEUE_DELAYED

In reply to K Prateek Nayak

With the fixup in dequeuing of PSI signals for delayed tasks, a new inconsistent PSI task state splat was discovered during boot similar to:

psi: inconsistent task state! task=... cpu=... psi_flags=5 clear=4 set=1

Tracking the PSI changes along with task's state revealed the following series of events:

psi_task_switch: psi_flags=4 clear=4 set=1 # sched_delayed is set to 1
psi_enqueue: psi_flags=1 clear=0 set=4 # requeue of delayed entity via ENQUEUE_DELAYED
psi_task_switch: psi_flags=5 clear=4 set=1 # task is blocked again but 1 is already set
psi: inconsistent task state! task=... cpu=... psi_flags=5 clear=4 set=1

The TSK_IOWAIT flag was never cleared onrequeue since psi_enqueue() only clears it on a "wakeup" which, in term of enqueue flags, is defined as:

    (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)

Add ENQUEUE_WAKEUP alongside ENQUEUE_DELAYED for requeue through ttwu_runnable(). psi_enqueue() is the only observer of this flag in the requeue path and it pairs with the DEQUEUE_SLEEP in block_task().

---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index 88cbfc671fb6..52be38021ebb 100644
@@ -3733,7 +3733,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)
 	if (task_on_rq_queued(p)) {
 		update_rq_clock(rq);
 		if (p->se.sched_delayed)
-			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED);
+			enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_WAKEUP | ENQUEUE_DELAYED);
 		if (!task_on_cpu(rq, p)) {
 			/*
 			 * When on_rq && !on_cpu the task is preempted, see if
Signature · 1 line
2.34.1
author reviewer
#4 lore raw

PATCH 3/3 sched/core: Indicate a sched_delayed task was migrated before wakeup

In reply to K Prateek Nayak

Since sched_delayed tasks remain on "rq->cfs_tasks" list even after blocking, they can be migrated from one runqueue to another in a delayed state by the load balancer. When they are eventually requeued or woken up on the same CPU via the try_to_wake_up() path, the eventual activation is clueless about the migration. This trips the PSI accounting since, in terms of enqueue flags, PSI only considers the following as a wakeup for PSI accounting:

    (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)

This can lead inconsistent PSI task state splat similar to:

psi: inconsistent task state! task=... cpu=... psi_flags=4 clear=. set=4 # Without partial fixup from this patch
psi: inconsistent task state! task=... cpu=... psi_flags=0 clear=4 set=1 # With partial fixup from this patch

Tracking the PSI changes along with task's state revealed the following series of events:

psi_task_switch: psi_flags=4 clear=4 set=1 # sched_delayed is set to 1
psi_dequeue: psi_flags=1 clear=1 set=0 # dequeued for migration
psi_enqueue: psi_flags=0 clear=0 set=4 # enqueued after migration
psi_enqueue: psi_flags=4 clear=1 set=4 # wakeup after migration
psi: inconsistent task state! task=... cpu=... psi_flags=4 clear=1 set=4

Moving psi_enqueue() to after "p->sched_class->enqueue_task()" and skipping enqueue until the delayed task is actually woken up (referred to partial fixup previously) changes the above scenario to the following:

psi_task_switch: psi_flags=4 clear=4 set=1 # sched_delayed is set to 1
psi_dequeue: psi_flags=1 clear=1 set=0 # dequeued for migration
psi_enqueue: psi_flags=0 clear=0 set=0 # enqueued after migration, sched delayed
psi_enqueue: psi_flags=0 clear=1 set=4 # wakeup of delayed task
psi: inconsistent task state! task=... cpu=... psi_flags=0 clear=1 set=4

psi_enqueue() tries to clear the TSK_IOWAIT since it believes the task has not migrated due to the lack of ENQUEUE_MIGRATED flag in case of a requeue or a full wakeup on "p->wake_cpu", but in-fact TSK_IOWAIT was cleared during dequeue for migration and was never set again.

Define "DELAYED_MIGRATED" and set it in "p->migration_flags" when a delayed task is migrated. This flag is consumed when the delayed entity is finally woken up, and psi_enqueue() is notified of the migration.

---
 kernel/sched/core.c  | 19 ++++++++++++++++++-
 kernel/sched/sched.h |  1 +
 kernel/sched/stats.h | 10 ++++++++++
 3 files changed, 29 insertions(+), 1 deletion(-)
kernel/sched/core.c +18 -1
index 52be38021ebb..1a353fa69a54 100644
@@ -2009,12 +2009,19 @@ unsigned long get_wchan(struct task_struct *p)
 
 void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 {
+	bool wakee_not_migrated = (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED);
+
 	if (!(flags & ENQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
 	if (!(flags & ENQUEUE_RESTORE)) {
 		sched_info_enqueue(rq, p);
-		psi_enqueue(p, (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED));
+
+		/* Notify PSI that the task was migrated in a delayed state before wakeup. */
+		if ((p->migration_flags & DELAYED_MIGRATED) && !task_on_rq_migrating(p)) {
+			wakee_not_migrated = false;
+			p->migration_flags &= ~DELAYED_MIGRATED;
+		}
 	}
 
 	p->sched_class->enqueue_task(rq, p, flags);
@@ -2023,6 +2030,8 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	 * ->sched_delayed.
 	 */
 	uclamp_rq_inc(rq, p);
+	if (!(flags & ENQUEUE_RESTORE))
+		psi_enqueue(p, wakee_not_migrated);
 
 	if (sched_core_enabled(rq))
 		sched_core_enqueue(rq, p);
@@ -2042,6 +2051,14 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & DEQUEUE_SAVE)) {
 		sched_info_dequeue(rq, p);
 		psi_dequeue(p, flags & DEQUEUE_SLEEP);
+
+		/*
+		 * Indicate that a sched_delayed task was migrated.
+		 * enqueue_task() needs this for correct accounting
+		 * when the delayed task eventually wakes up.
+		 */
+		if (p->se.sched_delayed && task_on_rq_migrating(p))
+			p->migration_flags |= DELAYED_MIGRATED;
 	}
 
 	/*
kernel/sched/sched.h +1 -0
index b1c3588a8f00..2dc2c4cb4f5f 100644
@@ -1326,6 +1326,7 @@ static inline int cpu_of(struct rq *rq)
 }
 
 #define MDF_PUSH		0x01
+#define DELAYED_MIGRATED	0x02 /* Task was migrated when in DELAYED_DEQUEUE state */
 
 static inline bool is_migration_disabled(struct task_struct *p)
 {
kernel/sched/stats.h +10 -0
index 237780aa3c53..06a2c6d3ec1e 100644
@@ -129,6 +129,13 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)
 	if (static_branch_likely(&psi_disabled))
 		return;
 
+	/*
+	 * Delayed task is not ready to run yet!
+	 * Wait for a requeue before accounting.
+	 */
+	if (p->se.sched_delayed)
+		return;
+
 	if (p->in_memstall)
 		set |= TSK_MEMSTALL_RUNNING;
 
@@ -148,6 +155,9 @@ static inline void psi_dequeue(struct task_struct *p, bool sleep)
 	if (static_branch_likely(&psi_disabled))
 		return;
 
+	/* Delayed task can only be dequeued for migration. */
+	WARN_ON_ONCE(p->se.sched_delayed && sleep);
+
 	/*
 	 * A voluntary sleep is a dequeue followed by a task switch. To
 	 * avoid walking all ancestors twice, psi_task_switch() handles
Signature · 1 line
2.34.1
#5 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

Hi Prateek,

patches 1 and 2 make obvious sense to me.

kernel/sched/stats.h static inline void psi_enqueue(struct task_struct *p, bool wakeup) · line 137

Quoted 10 lines · K Prateek Nayak
kernel/sched/stats.h +6 -0
@@ -129,6 +129,13 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)
 	if (static_branch_likely(&psi_disabled))
 		return;
 
+	/*
+	 * Delayed task is not ready to run yet!
+	 * Wait for a requeue before accounting.
+	 */
+	if (p->se.sched_delayed)
+		return;

This one is problematic. It clears sleeping state (memstall, iowait) during the dequeue of the migration but doesn't restore it until the wakeup, which could presumably be much later. This leaves a gap in the accounting.

psi really wants the dequeue and enqueue of the migration, even when a task is delay-dequeued. We just have to get the context parsing right to not confuse migration queues with wakeups.

I'll try to come up with a suitable solution as well, please don't apply this one for now.

maintainer
#6 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

Quoted 2 lines · Johannes Weiner

I'll try to come up with a suitable solution as well, please don't apply this one for now.

I'll make sure it doesn't end up in tip as-is.

#7 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Johannes Weiner

I'll try to come up with a suitable solution as well, please don't apply this one for now.

I'll make sure it doesn't end up in tip as-is.

Thanks.

This would be a replacement patch for #2 and #3 that handles migration of delayed tasks. It's slightly more invasive on the psi callback side, but I think it keeps the sched core bits simpler. Thoughts?

---

From d72a665d7c7c7d9c806424f473d13452754471d3 Mon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Thu, 10 Oct 2024 14:37:43 -0400
Subject: [PATCH] sched: psi: handle delayed-dequeue task migration

Since sched_delayed tasks remain queued even after blocking, the load balancer can migrate them between runqueues while PSI considers them to be asleep. As a result, it misreads the migration requeue followed by a wakeup as a double queue:

psi: inconsistent task state! task=... cpu=... psi_flags=4 clear=. set=4

First, call psi_enqueue() after p->sched_class->enqueue_task(). A wakeup will clear p->se.sched_delayed while a migration will not, so psi can use that flag to tell them apart.

Then teach psi to migrate any "sleep" state when delayed-dequeue tasks are being migrated.

Delayed-dequeue tasks can be revived by ttwu_runnable(), which will call down with a new ENQUEUE_DELAYED. Instead of further complicating the wakeup conditional in enqueue_task(), identify migration contexts instead and default to wakeup handling for all other cases.

---
 kernel/sched/core.c  | 12 +++++------
 kernel/sched/stats.h | 48 ++++++++++++++++++++++++++++++--------------
 2 files changed, 39 insertions(+), 21 deletions(-)
kernel/sched/core.c +6 -6
index 88cbfc671fb6..527502a86ff9 100644
@@ -2012,11 +2012,6 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & ENQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
-	if (!(flags & ENQUEUE_RESTORE)) {
-		sched_info_enqueue(rq, p);
-		psi_enqueue(p, (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED));
-	}
-
 	p->sched_class->enqueue_task(rq, p, flags);
 	/*
 	 * Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear
@@ -2024,6 +2019,11 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	 */
 	uclamp_rq_inc(rq, p);
 
+	if (!(flags & ENQUEUE_RESTORE)) {
+		sched_info_enqueue(rq, p);
+		psi_enqueue(p, flags & ENQUEUE_MIGRATED);
+	}
+
 	if (sched_core_enabled(rq))
 		sched_core_enqueue(rq, p);
 }
@@ -2041,7 +2041,7 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
 	if (!(flags & DEQUEUE_SAVE)) {
 		sched_info_dequeue(rq, p);
-		psi_dequeue(p, flags & DEQUEUE_SLEEP);
+		psi_dequeue(p, !(flags & DEQUEUE_SLEEP));
 	}
 
 	/*
kernel/sched/stats.h +33 -15

86 lines not on this page · raw message

Signature · 1 line
2.46.2
author reviewer Tested-by
#8 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

Hello Johannes,

Quoted 40 lines · Johannes Weiner
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Johannes Weiner

I'll try to come up with a suitable solution as well, please don't apply this one for now.

I'll make sure it doesn't end up in tip as-is.

Thanks.

This would be a replacement patch for #2 and #3 that handles migration of delayed tasks. It's slightly more invasive on the psi callback side, but I think it keeps the sched core bits simpler. Thoughts?

---

From d72a665d7c7c7d9c806424f473d13452754471d3 Mon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Thu, 10 Oct 2024 14:37:43 -0400
Subject: [PATCH] sched: psi: handle delayed-dequeue task migration

Since sched_delayed tasks remain queued even after blocking, the load balancer can migrate them between runqueues while PSI considers them to be asleep. As a result, it misreads the migration requeue followed by a wakeup as a double queue:

psi: inconsistent task state! task=... cpu=... psi_flags=4 clear=. set=4

First, call psi_enqueue() after p->sched_class->enqueue_task(). A wakeup will clear p->se.sched_delayed while a migration will not, so psi can use that flag to tell them apart.

Then teach psi to migrate any "sleep" state when delayed-dequeue tasks are being migrated.

Delayed-dequeue tasks can be revived by ttwu_runnable(), which will call down with a new ENQUEUE_DELAYED. Instead of further complicating the wakeup conditional in enqueue_task(), identify migration contexts instead and default to wakeup handling for all other cases.

Quoted 6 lines

This approach looks good to me as well! Thank you. I added this on top of Patch 1 and I haven't seen any PSI splats after my stress test. Feel free to add:

Signature · 2 lines
Thanks and Regards,
Prateek
Quoted 6 lines
---
  kernel/sched/core.c  | 12 +++++------
  kernel/sched/stats.h | 48 ++++++++++++++++++++++++++++++--------------
  2 files changed, 39 insertions(+), 21 deletions(-)

[..snip..]

maintainer
#9 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

In reply to Johannes Weiner
Quoted 2 lines · Johannes Weiner

This would be a replacement patch for #2 and #3 that handles migration of delayed tasks.

Thanks!

Quoted 2 lines

It's slightly more invasive on the psi callback
side, but I think it keeps the sched core bits simpler. Thoughts?

I wouldn't mind if psi_{en,de}queue() get the full flags argument in the future. For now the one boolean seems to work, but perhaps it makes more sense to just pass the flags along in their entirety.

#10 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Johannes Weiner

It's slightly more invasive on the psi callback
side, but I think it keeps the sched core bits simpler. Thoughts?

I wouldn't mind if psi_{en,de}queue() get the full flags argument in the future. For now the one boolean seems to work, but perhaps it makes more sense to just pass the flags along in their entirety.

Something like this?

I like it better too. There is a weird asymmetry between passing ENQ_MIGRATED to one and !ENQ_SLEEP to the other both as "migrate".

No strong preference for whether the ENQUEUE_RESTORE check should be in caller or callee, but I figured if we pass the flags anyway...

I toyed with a separate branch for ENQUEUE_INITIAL. But it saves one branch during fork while adding one to repeat enqueues. The latter should be hotter on average, so I removed it again.

Completely untested. But if it looks good, I'll send a proper patch.

---

kernel/sched/core.c +6 -6
index 527502a86ff9..42cf181bf3ab 100644
@@ -2019,10 +2019,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	 */
 	uclamp_rq_inc(rq, p);
 
-	if (!(flags & ENQUEUE_RESTORE)) {
+	psi_enqueue(p, flags);
+
+	if (!(flags & ENQUEUE_RESTORE))
 		sched_info_enqueue(rq, p);
-		psi_enqueue(p, flags & ENQUEUE_MIGRATED);
-	}
 
 	if (sched_core_enabled(rq))
 		sched_core_enqueue(rq, p);
@@ -2039,10 +2039,10 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & DEQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
-	if (!(flags & DEQUEUE_SAVE)) {
+	if (!(flags & DEQUEUE_SAVE))
 		sched_info_dequeue(rq, p);
-		psi_dequeue(p, !(flags & DEQUEUE_SLEEP));
-	}
+
+	psi_dequeue(p, flags);
 
 	/*
 	 * Must be before ->dequeue_task() because ->dequeue_task() can 'fail'
kernel/sched/stats.h +19 -10
index 767e098a3bd1..8ee0add5a48a 100644
@@ -127,21 +127,25 @@ static inline void psi_account_irqtime(struct rq *rq, struct task_struct *curr,
  * go through migration requeues. In this case, *sleeping* states need
  * to be transferred.
  */
-static inline void psi_enqueue(struct task_struct *p, bool migrate)
+static inline void psi_enqueue(struct task_struct *p, int flags)
 {
 	int clear = 0, set = 0;
 
 	if (static_branch_likely(&psi_disabled))
 		return;
 
+	/* Same runqueue, nothing changed for psi */
+	if (flags & ENQUEUE_RESTORE)
+		return;
+
 	if (p->se.sched_delayed) {
 		/* CPU migration of "sleeping" task */
-		SCHED_WARN_ON(!migrate);
+		SCHED_WARN_ON(!(flags & ENQUEUE_MIGRATED));
 		if (p->in_memstall)
 			set |= TSK_MEMSTALL;
 		if (p->in_iowait)
 			set |= TSK_IOWAIT;
-	} else if (migrate) {
+	} else if (flags & ENQUEUE_MIGRATED) {
 		/* CPU migration of runnable task */
 		set = TSK_RUNNING;
 		if (p->in_memstall)
@@ -158,17 +162,14 @@ static inline void psi_enqueue(struct task_struct *p, bool migrate)
 	psi_task_change(p, clear, set);
 }
 
-static inline void psi_dequeue(struct task_struct *p, bool migrate)
+static inline void psi_dequeue(struct task_struct *p, int flags)
 {
 	if (static_branch_likely(&psi_disabled))
 		return;
 
-	/*
-	 * When migrating a task to another CPU, clear all psi
-	 * state. The enqueue callback above will work it out.
-	 */
-	if (migrate)
-		psi_task_change(p, p->psi_flags, 0);
+	/* Same runqueue, nothing changed for psi */
+	if (flags & DEQUEUE_SAVE)
+		return;
 
 	/*
 	 * A voluntary sleep is a dequeue followed by a task switch. To
@@ -176,6 +177,14 @@ static inline void psi_dequeue(struct task_struct *p, bool migrate)
 	 * TSK_RUNNING and TSK_IOWAIT for us when it moves TSK_ONCPU.
 	 * Do nothing here.
 	 */
+	if (flags & DEQUEUE_SLEEP)
+		return;
+
+	/*
+	 * When migrating a task to another CPU, clear all psi
+	 * state. The enqueue callback above will work it out.
+	 */
+	psi_task_change(p, p->psi_flags, 0);
 }
 
 static inline void psi_ttwu_dequeue(struct task_struct *p)
maintainer
#11 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

Quoted 1 line · Johannes Weiner

Something like this?

Yeah, something like that indeed :-)

Quoted 2 lines

I like it better too. There is a weird asymmetry between passing ENQ_MIGRATED to one and !ENQ_SLEEP to the other both as "migrate".

So I have a note to stare at the whole {EN,DE}QUEUE_MIGRATING, ENQUEUE_MIGRATED and task_on_rq_migrating() situation, it has the feeling that something could be done to clean up there.

Quoted 2 lines

No strong preference for whether the ENQUEUE_RESTORE check should be in caller or callee, but I figured if we pass the flags anyway...

Right, and this way it's behind the static key, so win, right :-)

Quoted 5 lines

I toyed with a separate branch for ENQUEUE_INITIAL. But it saves one branch during fork while adding one to repeat enqueues. The latter should be hotter on average, so I removed it again.

Completely untested. But if it looks good, I'll send a proper patch.

Sure. Thanks for doing this.

#12 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Johannes Weiner

Completely untested. But if it looks good, I'll send a proper patch.

Sure. Thanks for doing this.

Ok here goes. Built, booted and runtime-tested.

---

From 91f230caa0119877cb861047e1af1371bf00d908 Mon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Fri, 11 Oct 2024 06:18:07 -0400
Subject: [PATCH] sched: psi: pass enqueue/dequeue flags to psi callbacks
directly

What psi needs to do on each enqueue and dequeue has gotten more subtle, and the generic sched code trying to distill this into a bool for the callbacks is awkward.

Pass the flags directly and let psi parse them. For that to work, the #include "stats.h" (which has the psi callback implementations) needs to be below the flag definitions in "sched.h". Move that section further down, next to some of the other accounting stuff.

This also puts the ENQUEUE_SAVE/RESTORE branch behind the psi jump label, slightly reducing overhead when PSI=y but runtime disabled.

---
 kernel/sched/core.c  | 12 +++++-----
 kernel/sched/sched.h | 56 ++++++++++++++++++++++----------------------
 kernel/sched/stats.h | 29 +++++++++++++++--------
 3 files changed, 53 insertions(+), 44 deletions(-)
kernel/sched/core.c +6 -6
index 527502a86ff9..42cf181bf3ab 100644
@@ -2019,10 +2019,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	 */
 	uclamp_rq_inc(rq, p);
 
-	if (!(flags & ENQUEUE_RESTORE)) {
+	psi_enqueue(p, flags);
+
+	if (!(flags & ENQUEUE_RESTORE))
 		sched_info_enqueue(rq, p);
-		psi_enqueue(p, flags & ENQUEUE_MIGRATED);
-	}
 
 	if (sched_core_enabled(rq))
 		sched_core_enqueue(rq, p);
@@ -2039,10 +2039,10 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & DEQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
-	if (!(flags & DEQUEUE_SAVE)) {
+	if (!(flags & DEQUEUE_SAVE))
 		sched_info_dequeue(rq, p);
-		psi_dequeue(p, !(flags & DEQUEUE_SLEEP));
-	}
+
+	psi_dequeue(p, flags);
 
 	/*
 	 * Must be before ->dequeue_task() because ->dequeue_task() can 'fail'
kernel/sched/sched.h +28 -28

68 lines not on this page · raw message

kernel/sched/stats.h +19 -10
index 767e098a3bd1..8ee0add5a48a 100644
@@ -127,21 +127,25 @@ static inline void psi_account_irqtime(struct rq *rq, struct task_struct *curr,
  * go through migration requeues. In this case, *sleeping* states need
  * to be transferred.
  */
-static inline void psi_enqueue(struct task_struct *p, bool migrate)
+static inline void psi_enqueue(struct task_struct *p, int flags)
 {
 	int clear = 0, set = 0;
 
 	if (static_branch_likely(&psi_disabled))
 		return;
 
+	/* Same runqueue, nothing changed for psi */
+	if (flags & ENQUEUE_RESTORE)
+		return;
+
 	if (p->se.sched_delayed) {
 		/* CPU migration of "sleeping" task */
-		SCHED_WARN_ON(!migrate);
+		SCHED_WARN_ON(!(flags & ENQUEUE_MIGRATED));
 		if (p->in_memstall)
 			set |= TSK_MEMSTALL;
 		if (p->in_iowait)
 			set |= TSK_IOWAIT;
-	} else if (migrate) {
+	} else if (flags & ENQUEUE_MIGRATED) {
 		/* CPU migration of runnable task */
 		set = TSK_RUNNING;
 		if (p->in_memstall)
@@ -158,17 +162,14 @@ static inline void psi_enqueue(struct task_struct *p, bool migrate)
 	psi_task_change(p, clear, set);
 }
 
-static inline void psi_dequeue(struct task_struct *p, bool migrate)
+static inline void psi_dequeue(struct task_struct *p, int flags)
 {
 	if (static_branch_likely(&psi_disabled))
 		return;
 
-	/*
-	 * When migrating a task to another CPU, clear all psi
-	 * state. The enqueue callback above will work it out.
-	 */
-	if (migrate)
-		psi_task_change(p, p->psi_flags, 0);
+	/* Same runqueue, nothing changed for psi */
+	if (flags & DEQUEUE_SAVE)
+		return;
 
 	/*
 	 * A voluntary sleep is a dequeue followed by a task switch. To
@@ -176,6 +177,14 @@ static inline void psi_dequeue(struct task_struct *p, bool migrate)
 	 * TSK_RUNNING and TSK_IOWAIT for us when it moves TSK_ONCPU.
 	 * Do nothing here.
 	 */
+	if (flags & DEQUEUE_SLEEP)
+		return;
+
+	/*
+	 * When migrating a task to another CPU, clear all psi
+	 * state. The enqueue callback above will work it out.
+	 */
+	psi_task_change(p, p->psi_flags, 0);
 }
 
 static inline void psi_ttwu_dequeue(struct task_struct *p)
Signature · 1 line
2.47.0
author reviewer Tested-by
#13 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

Hello Johannes,

Quoted 25 lines · Johannes Weiner
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Johannes Weiner

Completely untested. But if it looks good, I'll send a proper patch.

Sure. Thanks for doing this.

Ok here goes. Built, booted and runtime-tested.

---

From 91f230caa0119877cb861047e1af1371bf00d908 Mon Sep 17 00:00:00 2001
From: Johannes Weiner <[email protected]>
Date: Fri, 11 Oct 2024 06:18:07 -0400
Subject: [PATCH] sched: psi: pass enqueue/dequeue flags to psi callbacks
directly

What psi needs to do on each enqueue and dequeue has gotten more subtle, and the generic sched code trying to distill this into a bool for the callbacks is awkward.

Pass the flags directly and let psi parse them. For that to work, the #include "stats.h" (which has the psi callback implementations) needs to be below the flag definitions in "sched.h". Move that section further down, next to some of the other accounting stuff.

Quoted 6 lines

This also puts the ENQUEUE_SAVE/RESTORE branch behind the psi jump label, slightly reducing overhead when PSI=y but runtime disabled.

I tested this series on top of tip:sched/urgent at commit cd9626e9ebc7 ("sched/fair: Fix external p->on_rq users") and did not observe any splats related to PSI or otherwise. Please feel free to add:

Signature · 2 lines
Thanks and Regards,
Prateek
Quoted 7 lines
---
  kernel/sched/core.c  | 12 +++++-----
  kernel/sched/sched.h | 56 ++++++++++++++++++++++----------------------
  kernel/sched/stats.h | 29 +++++++++++++++--------
  3 files changed, 53 insertions(+), 44 deletions(-)

[..snip..]

#14 lore raw

[tip: sched/core] sched: psi: pass enqueue/dequeue flags to psi callbacks directly

In reply to Johannes Weiner
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     1a6151017ee5a30cb2d959f110ab18fc49646467
Gitweb:        https://git.kernel.org/tip/1a6151017ee5a30cb2d959f110ab18fc49646467
Author:        Johannes Weiner <[email protected]>
AuthorDate:    Mon, 14 Oct 2024 10:43:58 -04:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 26 Oct 2024 09:28:38 +02:00

sched: psi: pass enqueue/dequeue flags to psi callbacks directly

What psi needs to do on each enqueue and dequeue has gotten more subtle, and the generic sched code trying to distill this into a bool for the callbacks is awkward.

Pass the flags directly and let psi parse them. For that to work, the #include "stats.h" (which has the psi callback implementations) needs to be below the flag definitions in "sched.h". Move that section further down, next to some of the other accounting stuff.

This also puts the ENQUEUE_SAVE/RESTORE branch behind the psi jump label, slightly reducing overhead when PSI=y but runtime disabled.

---
 kernel/sched/core.c  | 12 ++++-----
 kernel/sched/sched.h | 56 +++++++++++++++++++++----------------------
 kernel/sched/stats.h | 29 ++++++++++++++--------
 3 files changed, 53 insertions(+), 44 deletions(-)
kernel/sched/core.c +6 -6
index 9bad282..c57a79e 100644
@@ -2024,10 +2024,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	 */
 	uclamp_rq_inc(rq, p);
 
-	if (!(flags & ENQUEUE_RESTORE)) {
+	psi_enqueue(p, flags);
+
+	if (!(flags & ENQUEUE_RESTORE))
 		sched_info_enqueue(rq, p);
-		psi_enqueue(p, flags & ENQUEUE_MIGRATED);
-	}
 
 	if (sched_core_enabled(rq))
 		sched_core_enqueue(rq, p);
@@ -2044,10 +2044,10 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & DEQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
-	if (!(flags & DEQUEUE_SAVE)) {
+	if (!(flags & DEQUEUE_SAVE))
 		sched_info_dequeue(rq, p);
-		psi_dequeue(p, !(flags & DEQUEUE_SLEEP));
-	}
+
+	psi_dequeue(p, flags);
 
 	/*
 	 * Must be before ->dequeue_task() because ->dequeue_task() can 'fail'
kernel/sched/sched.h +28 -28

68 lines not on this page · raw message

kernel/sched/stats.h +19 -10
index 767e098..8ee0add 100644
@@ -127,21 +127,25 @@ static inline void psi_account_irqtime(struct rq *rq, struct task_struct *curr,
  * go through migration requeues. In this case, *sleeping* states need
  * to be transferred.
  */
-static inline void psi_enqueue(struct task_struct *p, bool migrate)
+static inline void psi_enqueue(struct task_struct *p, int flags)
 {
 	int clear = 0, set = 0;
 
 	if (static_branch_likely(&psi_disabled))
 		return;
 
+	/* Same runqueue, nothing changed for psi */
+	if (flags & ENQUEUE_RESTORE)
+		return;
+
 	if (p->se.sched_delayed) {
 		/* CPU migration of "sleeping" task */
-		SCHED_WARN_ON(!migrate);
+		SCHED_WARN_ON(!(flags & ENQUEUE_MIGRATED));
 		if (p->in_memstall)
 			set |= TSK_MEMSTALL;
 		if (p->in_iowait)
 			set |= TSK_IOWAIT;
-	} else if (migrate) {
+	} else if (flags & ENQUEUE_MIGRATED) {
 		/* CPU migration of runnable task */
 		set = TSK_RUNNING;
 		if (p->in_memstall)
@@ -158,17 +162,14 @@ static inline void psi_enqueue(struct task_struct *p, bool migrate)
 	psi_task_change(p, clear, set);
 }
 
-static inline void psi_dequeue(struct task_struct *p, bool migrate)
+static inline void psi_dequeue(struct task_struct *p, int flags)
 {
 	if (static_branch_likely(&psi_disabled))
 		return;
 
-	/*
-	 * When migrating a task to another CPU, clear all psi
-	 * state. The enqueue callback above will work it out.
-	 */
-	if (migrate)
-		psi_task_change(p, p->psi_flags, 0);
+	/* Same runqueue, nothing changed for psi */
+	if (flags & DEQUEUE_SAVE)
+		return;
 
 	/*
 	 * A voluntary sleep is a dequeue followed by a task switch. To
@@ -176,6 +177,14 @@ static inline void psi_dequeue(struct task_struct *p, bool migrate)
 	 * TSK_RUNNING and TSK_IOWAIT for us when it moves TSK_ONCPU.
 	 * Do nothing here.
 	 */
+	if (flags & DEQUEUE_SLEEP)
+		return;
+
+	/*
+	 * When migrating a task to another CPU, clear all psi
+	 * state. The enqueue callback above will work it out.
+	 */
+	psi_task_change(p, p->psi_flags, 0);
 }
 
 static inline void psi_ttwu_dequeue(struct task_struct *p)
#15 lore raw

[tip: sched/urgent] Since sched_delayed tasks remain queued even after blocking, the load

In reply to Johannes Weiner
The following commit has been merged into the sched/urgent branch of tip:

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     f2c9767170bead8d0ceb9c01d074c6916350310d
Gitweb:        https://git.kernel.org/tip/f2c9767170bead8d0ceb9c01d074c6916350310d
Author:        Johannes Weiner <[email protected]>
AuthorDate:    Fri, 11 Oct 2024 10:49:33 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Fri, 11 Oct 2024 10:49:33 +02:00

Since sched_delayed tasks remain queued even after blocking, the load balancer can migrate them between runqueues while PSI considers them to be asleep. As a result, it misreads the migration requeue followed by a wakeup as a double queue:

psi: inconsistent task state! task=... cpu=... psi_flags=4 clear=. set=4

First, call psi_enqueue() after p->sched_class->enqueue_task(). A wakeup will clear p->se.sched_delayed while a migration will not, so psi can use that flag to tell them apart.

Then teach psi to migrate any "sleep" state when delayed-dequeue tasks are being migrated.

Delayed-dequeue tasks can be revived by ttwu_runnable(), which will call down with a new ENQUEUE_DELAYED. Instead of further complicating the wakeup conditional in enqueue_task(), identify migration contexts instead and default to wakeup handling for all other cases.

---
 kernel/sched/core.c  | 12 +++++------
 kernel/sched/stats.h | 48 +++++++++++++++++++++++++++++--------------
 2 files changed, 39 insertions(+), 21 deletions(-)
kernel/sched/core.c +6 -6
index 9e09140..71232f8 100644
@@ -2012,11 +2012,6 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & ENQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
-	if (!(flags & ENQUEUE_RESTORE)) {
-		sched_info_enqueue(rq, p);
-		psi_enqueue(p, (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED));
-	}
-
 	p->sched_class->enqueue_task(rq, p, flags);
 	/*
 	 * Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear
@@ -2024,6 +2019,11 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	 */
 	uclamp_rq_inc(rq, p);
 
+	if (!(flags & ENQUEUE_RESTORE)) {
+		sched_info_enqueue(rq, p);
+		psi_enqueue(p, flags & ENQUEUE_MIGRATED);
+	}
+
 	if (sched_core_enabled(rq))
 		sched_core_enqueue(rq, p);
 }
@@ -2041,7 +2041,7 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
 	if (!(flags & DEQUEUE_SAVE)) {
 		sched_info_dequeue(rq, p);
-		psi_dequeue(p, flags & DEQUEUE_SLEEP);
+		psi_dequeue(p, !(flags & DEQUEUE_SLEEP));
 	}
 
 	/*
kernel/sched/stats.h +33 -15

86 lines not on this page · raw message

#16 lore raw

[tip: sched/urgent] sched/psi: Fix mistaken CPU pressure indication after corrupted task state bug

In reply to Johannes Weiner
The following commit has been merged into the sched/urgent branch of tip:

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     c6508124193d42bbc3224571eb75bfa4c1821fbb
Gitweb:        https://git.kernel.org/tip/c6508124193d42bbc3224571eb75bfa4c1821fbb
Author:        Johannes Weiner <[email protected]>
AuthorDate:    Fri, 11 Oct 2024 10:49:33 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Mon, 14 Oct 2024 09:11:42 +02:00

sched/psi: Fix mistaken CPU pressure indication after corrupted task state bug

Since sched_delayed tasks remain queued even after blocking, the load balancer can migrate them between runqueues while PSI considers them to be asleep. As a result, it misreads the migration requeue followed by a wakeup as a double queue:

psi: inconsistent task state! task=... cpu=... psi_flags=4 clear=. set=4

First, call psi_enqueue() after p->sched_class->enqueue_task(). A wakeup will clear p->se.sched_delayed while a migration will not, so psi can use that flag to tell them apart.

Then teach psi to migrate any "sleep" state when delayed-dequeue tasks are being migrated.

Delayed-dequeue tasks can be revived by ttwu_runnable(), which will call down with a new ENQUEUE_DELAYED. Instead of further complicating the wakeup conditional in enqueue_task(), identify migration contexts instead and default to wakeup handling for all other cases.

It's not just the warning in dmesg, the task state corruption causes a permanent CPU pressure indication, which messes with workload/machine health monitoring.

---
 kernel/sched/core.c  | 12 +++++------
 kernel/sched/stats.h | 48 +++++++++++++++++++++++++++++--------------
 2 files changed, 39 insertions(+), 21 deletions(-)
kernel/sched/core.c +6 -6
index 9e09140..71232f8 100644
@@ -2012,11 +2012,6 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & ENQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
-	if (!(flags & ENQUEUE_RESTORE)) {
-		sched_info_enqueue(rq, p);
-		psi_enqueue(p, (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED));
-	}
-
 	p->sched_class->enqueue_task(rq, p, flags);
 	/*
 	 * Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear
@@ -2024,6 +2019,11 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	 */
 	uclamp_rq_inc(rq, p);
 
+	if (!(flags & ENQUEUE_RESTORE)) {
+		sched_info_enqueue(rq, p);
+		psi_enqueue(p, flags & ENQUEUE_MIGRATED);
+	}
+
 	if (sched_core_enabled(rq))
 		sched_core_enqueue(rq, p);
 }
@@ -2041,7 +2041,7 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
 	if (!(flags & DEQUEUE_SAVE)) {
 		sched_info_dequeue(rq, p);
-		psi_dequeue(p, flags & DEQUEUE_SLEEP);
+		psi_dequeue(p, !(flags & DEQUEUE_SLEEP));
 	}
 
 	/*
kernel/sched/stats.h +33 -15

86 lines not on this page · raw message

author reviewer
#17 lore raw

Re: [PATCH 3/3] sched/core: Indicate a sched_delayed task was migrated before wakeup

In reply to Johannes Weiner

Hello Johannes,

Quoted 21 lines · Johannes Weiner

Hi Prateek,

patches 1 and 2 make obvious sense to me.

Review static inline void psi_enqueue(struct task_struct *p, bool wakeup) · line 137

Quoted 10 lines · K Prateek Nayak
(quoted diff) +6 -0
@@ -129,6 +129,13 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)
  	if (static_branch_likely(&psi_disabled))
  		return;
 
+	/*
+	 * Delayed task is not ready to run yet!
+	 * Wait for a requeue before accounting.
+	 */
+	if (p->se.sched_delayed)
+		return;

This one is problematic. It clears sleeping state (memstall, iowait) during the dequeue of the migration but doesn't restore it until the wakeup, which could presumably be much later. This leaves a gap in the accounting.

Quoted 3 lines

psi really wants the dequeue and enqueue of the migration, even when a task is delay-dequeued. We just have to get the context parsing right to not confuse migration queues with wakeups.

I see! I was confused in my interpretation of the expectations. Sorry about that! Can you please give the attached patch a try on top of the first two patches of series. It survived my (decently stressy) stress test but it is not as extensive the ones you have :)

Quoted 3 lines

I'll try to come up with a suitable solution as well, please don't apply this one for now.

Thank you Peter for holding off the patches and sorry for the late surprise!

Signature · 2 lines
Thanks and Regards,
Prateek

From 4ad25e7e6a0eb539c5544a81bb3b22ca7cc05fe0 Mon Sep 17 00:00:00 2001
From: K Prateek Nayak <[email protected]>
Date: Thu, 10 Oct 2024 15:31:41 +0000
Subject: [PATCH v1.1 3/3] sched/core: Move PSI flags when delayed task is migrated

Since sched_delayed tasks remain on "rq->cfs_tasks" list even after blocking, they can be migrated from one runqueue to another in a delayed state by the load balancer. When they are eventually requeued or woken up on the same CPU via the try_to_wake_up() path, the eventual activation is clueless about the migration. This trips the PSI accounting since, in terms of enqueue flags, PSI only considers the following as a wakeup for PSI accounting:

    (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)

This can lead inconsistent PSI task state splat similar to:

psi: inconsistent task state! task=... cpu=... psi_flags=4 clear=. set=4

Migrate the PSI flags as well when delayed tasks are migrated. Since psi_dequeue() clears p->psi_flags, use the same to cache the original PSI flags and requeue it after enqueue during migration.

Note: Delayed tasks will not track TSK_RUNNING or TSK_MEMSTALL_RUNNING signals despite being queued on the runqueue , however they'll still track and migrate TSK_IOWAIT and TSK_MEMSTALL signals.

---
Changelog v1..v1.1:

o Alternate approach that migrates the PSI flags too when the delayed task is migrated by the load balancer.

---
 kernel/sched/core.c  |  9 ++++++---
 kernel/sched/stats.h | 22 ++++++++++++++++++++++
 2 files changed, 28 insertions(+), 3 deletions(-)
kernel/sched/core.c +6 -3
index 52be38021ebb..c3ac63d3eb9d 100644
@@ -2012,10 +2012,8 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	if (!(flags & ENQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
-	if (!(flags & ENQUEUE_RESTORE)) {
+	if (!(flags & ENQUEUE_RESTORE))
 		sched_info_enqueue(rq, p);
-		psi_enqueue(p, (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED));
-	}
 
 	p->sched_class->enqueue_task(rq, p, flags);
 	/*
@@ -2023,6 +2021,11 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 	 * ->sched_delayed.
 	 */
 	uclamp_rq_inc(rq, p);
+	if (!(flags & ENQUEUE_RESTORE)) {
+		/* Tasks can only remain dealyed after an enqueue if they are migrating */
+		WARN_ON_ONCE(p->se.sched_delayed && !task_on_rq_migrating(p));
+		psi_enqueue(p, (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED));
+	}
 
 	if (sched_core_enabled(rq))
 		sched_core_enqueue(rq, p);
kernel/sched/stats.h +22 -0
index 237780aa3c53..d267a187304c 100644
@@ -129,6 +129,15 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)
 	if (static_branch_likely(&psi_disabled))
 		return;
 
+	if (p->se.sched_delayed) {
+		unsigned int flags = p->psi_flags;
+
+		/* Restore the state saved by psi_dequeue() */
+		p->psi_flags = 0;
+		psi_task_change(p, 0, flags);
+		return;
+	}
+
 	if (p->in_memstall)
 		set |= TSK_MEMSTALL_RUNNING;
 
@@ -145,9 +154,14 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)
 
 static inline void psi_dequeue(struct task_struct *p, bool sleep)
 {
+	unsigned int flags = p->psi_flags;
+
 	if (static_branch_likely(&psi_disabled))
 		return;
 
+	/* Delayed task can only be dequeued for migration. */
+	WARN_ON_ONCE(p->se.sched_delayed && sleep);
+
 	/*
 	 * A voluntary sleep is a dequeue followed by a task switch. To
 	 * avoid walking all ancestors twice, psi_task_switch() handles
@@ -158,6 +172,14 @@ static inline void psi_dequeue(struct task_struct *p, bool sleep)
 		return;
 
 	psi_task_change(p, p->psi_flags, 0);
+
+	/*
+	 * Since the delayed entity is migrating, the PSI flags need to
+	 * be migrated too. Since p->psi_flags is not cleared, cache the
+	 * previous state there for psi_enqueue() to restore.
+	 */
+	if (p->se.sched_delayed)
+		p->psi_flags = flags;
 }
 
 static inline void psi_ttwu_dequeue(struct task_struct *p)
Signature · 1 line
2.34.1
maintainer
#18 lore raw
In reply to K Prateek Nayak
Quoted 31 lines · K Prateek Nayak

After the introduction of DELAY_DEQUEUE, PSI consistently started warning about inconsistent task state early into the boot. This could be root-caused to three issues that the three patches respectively solve:

o PSI signals not being dequeued when the task is blocked, but also delayed since psi_sched_switch() considered "!task_on_rq_queued()" as the task being blocked but a delayed task will remain queued on the runqueue until it is picked again and goes through a full dequeue.

o enqueue_task() not using the ENQUEUE_WAKEUP alongside ENQUEUE_DELAYED in ttwu_runnable(). Since psi_enqueue() only considers (in terms of enqueue flags):

    (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)

... as a wakeup, the lack of ENQUEUE_WAKEUP can misguide psi_enqueue() which only clears TSK_IOWAIT flag on wakeups.

o When a delayed task is migrated by the load balancer, the requeue or the wakeup context may be aware that the task has migrated between it blocking and it waking up. This is necessary to be communicated to PSI which forgoes clearing TSK_IOWAIT since it expects the psi_.*dequeue() to have cleared it during migration.

The series correctly communicates the blocked status of a delayed task to psi_dequeue(), adds the ENQUEUE_WAKEUP flag during a requeue in ttwu_runnable(), re-arranges the psi_enqueue() to be called after a "p->sched_class->enqueue_task()", and notify psi_enqueue() of a migration in delayed state using "p->migration_flags" to maintain the task state consistently.

Quoted 4 lines

This series was previously posted as one large diff at
https://lore.kernel.org/lkml/[email protected]/
and was tested by Johannes. The tags on the diff have been carried to this series.

Thanks!

I've renamed DELAYED_MIGRATED to MF_DELAYED, and made a note to go rename the MDF_PUSH thing to something consistent.

I've stuck then in queue.git sched/urgent along with a few other fixes and I will hopefully push the lot into tip soon.

author reviewer
#19 lore raw

Hello Peter,

Quoted 37 lines · Peter Zijlstra
Quoted 31 lines · K Prateek Nayak

After the introduction of DELAY_DEQUEUE, PSI consistently started warning about inconsistent task state early into the boot. This could be root-caused to three issues that the three patches respectively solve:

o PSI signals not being dequeued when the task is blocked, but also delayed since psi_sched_switch() considered "!task_on_rq_queued()" as the task being blocked but a delayed task will remain queued on the runqueue until it is picked again and goes through a full dequeue.

o enqueue_task() not using the ENQUEUE_WAKEUP alongside ENQUEUE_DELAYED in ttwu_runnable(). Since psi_enqueue() only considers (in terms of enqueue flags):

     (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)

... as a wakeup, the lack of ENQUEUE_WAKEUP can misguide psi_enqueue() which only clears TSK_IOWAIT flag on wakeups.

o When a delayed task is migrated by the load balancer, the requeue or the wakeup context may be aware that the task has migrated between it blocking and it waking up. This is necessary to be communicated to PSI which forgoes clearing TSK_IOWAIT since it expects the psi_.*dequeue() to have cleared it during migration.

The series correctly communicates the blocked status of a delayed task to psi_dequeue(), adds the ENQUEUE_WAKEUP flag during a requeue in ttwu_runnable(), re-arranges the psi_enqueue() to be called after a "p->sched_class->enqueue_task()", and notify psi_enqueue() of a migration in delayed state using "p->migration_flags" to maintain the task state consistently.

Quoted 4 lines

This series was previously posted as one large diff at
https://lore.kernel.org/lkml/[email protected]/
and was tested by Johannes. The tags on the diff have been carried to this series.

Quoted 4 lines

Thanks!

I've renamed DELAYED_MIGRATED to MF_DELAYED, and made a note to go rename the MDF_PUSH thing to something consistent.

Ack. DELAYED_MIGRATED was coined in spur of the moment; MF_DELAYED fits the "migration_flags" bill well.

Quoted 3 lines

I've stuck then in queue.git sched/urgent along with a few other fixes and I will hopefully push the lot into tip soon.

Thank you for picking up the patches! Let me know if there are any surprises (there shouldn't be any after the thorough testing by Johannes :) - thank you again for that)

--
Thanks and Regards,
Prateek