Skip to contentSkip to conversation

FAILED: patch "[PATCH] sched/core: Make core-sched flips wait for in-flight" failed to apply to 6.1-stable tree

PATCH New [email protected] 2 participants 2 messages Sep 3 – Sep 9

Subsystems (2)
Stable branch Scheduler
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

The patch below does not apply to the 6.1-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

To reproduce the conflict and resubmit, you may use the following commands:

git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.1.y
git checkout FETCH_HEAD
git cherry-pick -x f3629c63a4af3e491381780bc6c123cb498c4c40
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2026090311-expediter-squire-a01e@gregkh' --subject-prefix 'PATCH 6.1.y' 'HEAD^..'

Possible dependencies:

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

From f3629c63a4af3e491381780bc6c123cb498c4c40 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Fri, 7 Aug 2026 11:02:17 -1000
Subject: [PATCH] sched/core: Make core-sched flips wait for in-flight
selections

Core scheduling's pick_next_task() operates on all sibling rqs under one acquisition of the shared core-wide lock. A ->pick_task() that releases the rq lock leaves every sibling __lock momentarily free, letting __sched_core_flip(false) complete mid-selection and rebind rq_lockp() under it. The selection resumes on the split locks, touching sibling state it no longer protects, and __schedule() finally releases a lock that was never taken while leaking the one that was.

Count in-flight core-wide selections in the leader's rq->core_pick_in_flight and make __sched_core_flip() wait for the count to drain. The count only changes under the shared lock, which the flip holds while sampling, so no other ordering is needed. The wait can repeat while selections overlap, but the flip backs off between samples and flips are rare cookie-lifetime events.

sched_core_cpu_deactivate() moves the count to the new leader - a stale copy left behind would bias it forever if that CPU later returns as its own leader.

Fixes 539f65125d20 ("sched: Add core wide task selection and scheduling")
Cc [email protected] # v5.14+
Signed-off-by Tejun Heo <[email protected]>
Acked-by Peter Zijlstra (Intel) <[email protected]>
kernel/sched/core.c +22 -0
index 84ef83316562..0130463798f8 100644
@@ -443,6 +443,17 @@ static void __sched_core_flip(bool enabled)
 
 		sched_core_lock(cpu, &flags);
 
+		/*
+		 * A core-wide selection may have the shared rq lock temporarily
+		 * released by a lock-dropping ->pick_task(). Flipping would
+		 * rebind rq_lockp() under it. Wait it out.
+		 */
+		while (cpu_rq(cpu)->core->core_pick_in_flight) {
+			sched_core_unlock(cpu, &flags);
+			cpu_relax();
+			sched_core_lock(cpu, &flags);
+		}
+
 		for_each_cpu(t, smt_mask)
 			cpu_rq(t)->core_enabled = enabled;
 

@@ -6236,6 +6247,8 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 		return __pick_next_task(rq, rf);
 	}
 
+	rq->core->core_pick_in_flight++;
+
 	/*
 	 * If there were no {en,de}queues since we picked (IOW, the task
 	 * pointers are all still valid), and we haven't scheduled the last

@@ -6450,6 +6463,7 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 	}
 
 out_set_next:
+	rq->core->core_pick_in_flight--;
 	put_prev_set_next_task(rq, rq->donor, next);
 	if (rq->core->core_forceidle_count && next == rq->idle)
 		queue_core_balance(rq);

@@ -6644,6 +6658,13 @@ static void sched_core_cpu_deactivate(unsigned int cpu)
 	core_rq->core_forceidle_seq        = rq->core_forceidle_seq;
 	core_rq->core_forceidle_occupation = rq->core_forceidle_occupation;
 
+	/*
+	 * A stale leftover would bias the count forever if this CPU later
+	 * returns as its own leader. Move, don't copy.
+	 */
+	core_rq->core_pick_in_flight       = rq->core_pick_in_flight;
+	rq->core_pick_in_flight            = 0;
+
 	/*
 	 * Accounting edge for forced idle is handled in pick_next_task().
 	 * Don't need another one here, since the hotplug thread shouldn't

@@ -9060,6 +9081,7 @@ void __init sched_init(void)
 		rq->core_forceidle_count = 0;
 		rq->core_forceidle_occupation = 0;
 		rq->core_forceidle_start = 0;
+		rq->core_pick_in_flight = 0;
 
 		rq->core_cookie = 0UL;
 #endif
kernel/sched/sched.h +1 -0
index 56acf502ba26..450b6a04669f 100644
@@ -1358,6 +1358,7 @@ struct rq {
 	unsigned int		core_forceidle_seq;
 	unsigned int		core_forceidle_occupation;
 	u64			core_forceidle_start;
+	unsigned int		core_pick_in_flight;
 #endif /* CONFIG_SCHED_CORE */
 
 	/* Scratch cpumask to be temporarily used under rq_lock */
maintainer
#1 lore raw

PATCH 1/1 sched/core: Make core-sched flips wait for in-flight selections

From: Tejun Heo <[email protected]>

[ Upstream commit f3629c63a4af3e491381780bc6c123cb498c4c40 ]

Core scheduling's pick_next_task() operates on all sibling rqs under one acquisition of the shared core-wide lock. A ->pick_task() that releases the rq lock leaves every sibling __lock momentarily free, letting __sched_core_flip(false) complete mid-selection and rebind rq_lockp() under it. The selection resumes on the split locks, touching sibling state it no longer protects, and __schedule() finally releases a lock that was never taken while leaking the one that was.

Count in-flight core-wide selections in the leader's rq->core_pick_in_flight and make __sched_core_flip() wait for the count to drain. The count only changes under the shared lock, which the flip holds while sampling, so no other ordering is needed. The wait can repeat while selections overlap, but the flip backs off between samples and flips are rare cookie-lifetime events.

sched_core_cpu_deactivate() moves the count to the new leader - a stale copy left behind would bias it forever if that CPU later returns as its own leader.

Fixes 539f65125d20 ("sched: Add core wide task selection and scheduling")
Cc [email protected] # v5.14+
Signed-off-by Tejun Heo <[email protected]>
Acked-by Peter Zijlstra (Intel) <[email protected]>
[ Moved the counter decrement from out_set_next to out to accommodate 6.6’s cached-selection path. ]
Signed-off-by Sasha Levin <[email protected]>
---
 kernel/sched/core.c  | 22 ++++++++++++++++++++++
 kernel/sched/sched.h |  1 +
 2 files changed, 23 insertions(+)
kernel/sched/core.c +22 -0
index fa9008e5d8bd6..6c7759dae216f 100644
@@ -345,6 +345,17 @@ static void __sched_core_flip(bool enabled)
 
 		sched_core_lock(cpu, &flags);
 
+		/*
+		 * A core-wide selection may have the shared rq lock temporarily
+		 * released by a lock-dropping ->pick_task(). Flipping would
+		 * rebind rq_lockp() under it. Wait it out.
+		 */
+		while (cpu_rq(cpu)->core->core_pick_in_flight) {
+			sched_core_unlock(cpu, &flags);
+			cpu_relax();
+			sched_core_lock(cpu, &flags);
+		}
+
 		for_each_cpu(t, smt_mask)
 			cpu_rq(t)->core_enabled = enabled;
 

@@ -5959,6 +5970,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 		return __pick_next_task(rq, prev, rf);
 	}
 
+	rq->core->core_pick_in_flight++;
+
 	/*
 	 * If there were no {en,de}queues since we picked (IOW, the task
 	 * pointers are all still valid), and we haven't scheduled the last

@@ -6152,6 +6165,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 out_set_next:
 	set_next_task(rq, next);
 out:
+	rq->core->core_pick_in_flight--;
 	if (rq->core->core_forceidle_count && next == rq->idle)
 		queue_core_balance(rq);
 

@@ -6344,6 +6358,13 @@ static void sched_core_cpu_deactivate(unsigned int cpu)
 	core_rq->core_forceidle_seq        = rq->core_forceidle_seq;
 	core_rq->core_forceidle_occupation = rq->core_forceidle_occupation;
 
+	/*
+	 * A stale leftover would bias the count forever if this CPU later
+	 * returns as its own leader. Move, don't copy.
+	 */
+	core_rq->core_pick_in_flight       = rq->core_pick_in_flight;
+	rq->core_pick_in_flight            = 0;
+
 	/*
 	 * Accounting edge for forced idle is handled in pick_next_task().
 	 * Don't need another one here, since the hotplug thread shouldn't

@@ -9834,6 +9855,7 @@ void __init sched_init(void)
 		rq->core_forceidle_count = 0;
 		rq->core_forceidle_occupation = 0;
 		rq->core_forceidle_start = 0;
+		rq->core_pick_in_flight = 0;
 
 		rq->core_cookie = 0UL;
 #endif
kernel/sched/sched.h +1 -0
index a6c2b8f3045ee..accff24e091e4 100644
@@ -1151,6 +1151,7 @@ struct rq {
 	unsigned int		core_forceidle_seq;
 	unsigned int		core_forceidle_occupation;
 	u64			core_forceidle_start;
+	unsigned int		core_pick_in_flight;
 #endif
 };
Signature · 1 line
2.53.0