Skip to contentSkip to conversation

sched/fair: Clear throttle state when leaving the fair class

PATCH Applied Daehyeon Ko 5 participants 7 messages Oct 22 – Nov 6

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched/fair: Clear throttle state when leaving the fair class

Task-based CFS bandwidth throttling removes a task's fair sched_entity while the scheduler core still considers the task queued. p->throttled carries the fair-specific state across dequeue/enqueue operations.

DEQUEUE_CLASS removes the task from cfs_rq->throttled_limbo_list but retains the bit. If the last finite CFS quota is disabled while the task runs in another class, it is absent from the unthrottle walk and the bit survives.
Returning to fair while cfs_bandwidth_used() is false then performs a real fair enqueue with p->throttled still set.

After bandwidth is re-enabled, a class-change dequeue treats the stale bit as limbo state and skips __dequeue_task(). p->se.on_rq therefore remains set, and put_prev_task_fair() places the entity in the EEVDF tree while the task moves to RT. On exit, the fair picker can select the TASK_DEAD task. __schedule() then returns to do_task_dead(), which hits BUG().

Keep p->throttled across migrations and group moves, but clear it on DEQUEUE_CLASS because fair-specific state must not outlive fair-class membership. A later return to fair uses the normal enqueue path. That path still respects a throttled cfs_rq.

A KASAN, LOCKDEP and DEBUG_LIST v7.3 kernel reproduced the do_task_dead() BUG in 3/3 root and 3/3 delegated uid-1000 runs with CapPrm=CapEff=0 and RLIMIT_RTPRIO=1. The fix passed both matrices plus a 3/3 class-switch control with bandwidth continuously enabled, without a scheduler warning, KASAN report, lockdep splat or crash.

An AI coding assistant analyzed the state mismatch and drafted the initial reproducer, fix candidate and this changelog. The source paths, trigger, controls and results were rechecked against the cited revisions.

Fixes e1fad12dcb66 ("sched/fair: Switch to task based throttle model")
---
 kernel/sched/fair.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/fair.c +2 -2
index 0feec172fd37eb..524fe88bbb381a 100644
@@ -6957,8 +6957,8 @@ static void dequeue_throttled_task(struct task_struct *p, int flags)
 	WARN_ON_ONCE(p->se.on_rq);
 	list_del_init(&p->throttle_node);
 
-	/* task blocked after throttled */
-	if (flags & DEQUEUE_SLEEP) {
+	/* Task blocked after throttling or is leaving the fair class. */
+	if (flags & (DEQUEUE_SLEEP | DEQUEUE_CLASS)) {
 		p->throttled = false;
 		return;
 	}
Signature · 1 line
2.55.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 29 lines · Daehyeon Ko

Task-based CFS bandwidth throttling removes a task's fair sched_entity while the scheduler core still considers the task queued. p->throttled carries the fair-specific state across dequeue/enqueue operations.

DEQUEUE_CLASS removes the task from cfs_rq->throttled_limbo_list but retains the bit. If the last finite CFS quota is disabled while the task runs in another class, it is absent from the unthrottle walk and the bit survives.
Returning to fair while cfs_bandwidth_used() is false then performs a real fair enqueue with p->throttled still set.

After bandwidth is re-enabled, a class-change dequeue treats the stale bit as limbo state and skips __dequeue_task(). p->se.on_rq therefore remains set, and put_prev_task_fair() places the entity in the EEVDF tree while the task moves to RT. On exit, the fair picker can select the TASK_DEAD task. __schedule() then returns to do_task_dead(), which hits BUG().

Keep p->throttled across migrations and group moves, but clear it on DEQUEUE_CLASS because fair-specific state must not outlive fair-class membership. A later return to fair uses the normal enqueue path. That path still respects a throttled cfs_rq.

A KASAN, LOCKDEP and DEBUG_LIST v7.3 kernel reproduced the do_task_dead() BUG in 3/3 root and 3/3 delegated uid-1000 runs with CapPrm=CapEff=0 and RLIMIT_RTPRIO=1. The fix passed both matrices plus a 3/3 class-switch control with bandwidth continuously enabled, without a scheduler warning, KASAN report, lockdep splat or crash.

Quoted 7 lines

An AI coding assistant analyzed the state mismatch and drafted the initial reproducer, fix candidate and this changelog. The source paths, trigger, controls and results were rechecked against the cited revisions.

Fixes e1fad12dcb66 ("sched/fair: Switch to task based throttle model")

Thanks!

Quoted 21 lines
---
 kernel/sched/fair.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/fair.c +2 -2
index 0feec172fd37eb..524fe88bbb381a 100644
@@ -6957,8 +6957,8 @@ static void dequeue_throttled_task(struct task_struct *p, int flags)
 	WARN_ON_ONCE(p->se.on_rq);
 	list_del_init(&p->throttle_node);
 
-	/* task blocked after throttled */
-	if (flags & DEQUEUE_SLEEP) {
+	/* Task blocked after throttling or is leaving the fair class. */
+	if (flags & (DEQUEUE_SLEEP | DEQUEUE_CLASS)) {
 		p->throttled = false;
 		return;
 	}
Signature · 1 line
2.55.0
#2 lore raw
In reply to Daehyeon Ko
Quoted 29 lines · Daehyeon Ko

Task-based CFS bandwidth throttling removes a task's fair sched_entity while the scheduler core still considers the task queued. p->throttled carries the fair-specific state across dequeue/enqueue operations.

DEQUEUE_CLASS removes the task from cfs_rq->throttled_limbo_list but retains the bit. If the last finite CFS quota is disabled while the task runs in another class, it is absent from the unthrottle walk and the bit survives.
Returning to fair while cfs_bandwidth_used() is false then performs a real fair enqueue with p->throttled still set.

After bandwidth is re-enabled, a class-change dequeue treats the stale bit as limbo state and skips __dequeue_task(). p->se.on_rq therefore remains set, and put_prev_task_fair() places the entity in the EEVDF tree while the task moves to RT. On exit, the fair picker can select the TASK_DEAD task. __schedule() then returns to do_task_dead(), which hits BUG().

Keep p->throttled across migrations and group moves, but clear it on DEQUEUE_CLASS because fair-specific state must not outlive fair-class membership. A later return to fair uses the normal enqueue path. That path still respects a throttled cfs_rq.

A KASAN, LOCKDEP and DEBUG_LIST v7.3 kernel reproduced the do_task_dead() BUG in 3/3 root and 3/3 delegated uid-1000 runs with CapPrm=CapEff=0 and RLIMIT_RTPRIO=1. The fix passed both matrices plus a 3/3 class-switch control with bandwidth continuously enabled, without a scheduler warning, KASAN report, lockdep splat or crash.

Quoted 8 lines

An AI coding assistant analyzed the state mismatch and drafted the initial reproducer, fix candidate and this changelog. The source paths, trigger, controls and results were rechecked against the cited revisions.

Fixes e1fad12dcb66 ("sched/fair: Switch to task based throttle model")

---

Yep, sched_class switch lets you smuggle ->throttled into !cfs_bandwidth_used(), and then that check on enqueue_throttled lets you smuggle it onto actually being enqueued, which leads to corruption. That's fun.

I think we need to use switched_from_fair though - if we are currently TASK_ON_RQ_MIGRATING, sched_class changes will skip going into dequeue_task_fair at all, so we will not see the DEQUEUE_CLASS. Triggering a migration on a throttled task can still happen I'm fairly sure.

It's too bad, since it would be nice to just add to this existing touchpoint, but it's at least it's trivial:

kernel/sched/fair.c +4 -0
index cfc982205b98c..db57ac52eb7f7 100644
@@ -15500,10 +15500,14 @@ static void switching_from_fair(struct rq *rq, struct task_struct *p)
 }
 
 static void switched_from_fair(struct rq *rq, struct task_struct *p)
 {
 	detach_task_cfs_rq(p);
+
+#ifdef CONFIG_CFS_BANDWIDTH
+	p->throttled = false;
+#endif
 }
 
 static void switched_to_fair(struct rq *rq, struct task_struct *p)
 {
 	WARN_ON_ONCE(p->se.sched_delayed);

(Or we could pull it out into its own function for cleanliness if it seems worthwhile)

#3 lore raw
Quoted 40 lines · Benjamin Segall
Quoted 29 lines · Daehyeon Ko

Task-based CFS bandwidth throttling removes a task's fair sched_entity while the scheduler core still considers the task queued. p->throttled carries the fair-specific state across dequeue/enqueue operations.

DEQUEUE_CLASS removes the task from cfs_rq->throttled_limbo_list but retains the bit. If the last finite CFS quota is disabled while the task runs in another class, it is absent from the unthrottle walk and the bit survives.
Returning to fair while cfs_bandwidth_used() is false then performs a real fair enqueue with p->throttled still set.

After bandwidth is re-enabled, a class-change dequeue treats the stale bit as limbo state and skips __dequeue_task(). p->se.on_rq therefore remains set, and put_prev_task_fair() places the entity in the EEVDF tree while the task moves to RT. On exit, the fair picker can select the TASK_DEAD task. __schedule() then returns to do_task_dead(), which hits BUG().

Keep p->throttled across migrations and group moves, but clear it on DEQUEUE_CLASS because fair-specific state must not outlive fair-class membership. A later return to fair uses the normal enqueue path. That path still respects a throttled cfs_rq.

A KASAN, LOCKDEP and DEBUG_LIST v7.3 kernel reproduced the do_task_dead() BUG in 3/3 root and 3/3 delegated uid-1000 runs with CapPrm=CapEff=0 and RLIMIT_RTPRIO=1. The fix passed both matrices plus a 3/3 class-switch control with bandwidth continuously enabled, without a scheduler warning, KASAN report, lockdep splat or crash.

Quoted 8 lines

An AI coding assistant analyzed the state mismatch and drafted the initial reproducer, fix candidate and this changelog. The source paths, trigger, controls and results were rechecked against the cited revisions.

Fixes e1fad12dcb66 ("sched/fair: Switch to task based throttle model")

---

Quoted 8 lines

Yep, sched_class switch lets you smuggle ->throttled into !cfs_bandwidth_used(), and then that check on enqueue_throttled lets you smuggle it onto actually being enqueued, which leads to corruption. That's fun.

I think we need to use switched_from_fair though - if we are currently TASK_ON_RQ_MIGRATING, sched_class changes will skip going into dequeue_task_fair at all, so we will not see the DEQUEUE_CLASS.

sched class change serializes with migration I think, with task_rq_lock() which waits for migration to settle, so this should not happen?

reviewer Reviewed-by
#4 lore raw
Quoted 46 lines · Aaron Lu
Quoted 45 lines · Benjamin Segall
Quoted 29 lines · Daehyeon Ko

Task-based CFS bandwidth throttling removes a task's fair sched_entity while the scheduler core still considers the task queued. p->throttled carries the fair-specific state across dequeue/enqueue operations.

DEQUEUE_CLASS removes the task from cfs_rq->throttled_limbo_list but retains the bit. If the last finite CFS quota is disabled while the task runs in another class, it is absent from the unthrottle walk and the bit survives.
Returning to fair while cfs_bandwidth_used() is false then performs a real fair enqueue with p->throttled still set.

After bandwidth is re-enabled, a class-change dequeue treats the stale bit as limbo state and skips __dequeue_task(). p->se.on_rq therefore remains set, and put_prev_task_fair() places the entity in the EEVDF tree while the task moves to RT. On exit, the fair picker can select the TASK_DEAD task. __schedule() then returns to do_task_dead(), which hits BUG().

Keep p->throttled across migrations and group moves, but clear it on DEQUEUE_CLASS because fair-specific state must not outlive fair-class membership. A later return to fair uses the normal enqueue path. That path still respects a throttled cfs_rq.

A KASAN, LOCKDEP and DEBUG_LIST v7.3 kernel reproduced the do_task_dead() BUG in 3/3 root and 3/3 delegated uid-1000 runs with CapPrm=CapEff=0 and RLIMIT_RTPRIO=1. The fix passed both matrices plus a 3/3 class-switch control with bandwidth continuously enabled, without a scheduler warning, KASAN report, lockdep splat or crash.

Quoted 8 lines

An AI coding assistant analyzed the state mismatch and drafted the initial reproducer, fix candidate and this changelog. The source paths, trigger, controls and results were rechecked against the cited revisions.

Fixes e1fad12dcb66 ("sched/fair: Switch to task based throttle model")

---

Yep, sched_class switch lets you smuggle ->throttled into !cfs_bandwidth_used(), and then that check on enqueue_throttled lets you smuggle it onto actually being enqueued, which leads to corruption. That's fun.

Quoted 6 lines
Quoted 3 lines

I think we need to use switched_from_fair though - if we are currently TASK_ON_RQ_MIGRATING, sched_class changes will skip going into dequeue_task_fair at all, so we will not see the DEQUEUE_CLASS.

sched class change serializes with migration I think, with task_rq_lock() which waits for migration to settle, so this should not happen?

Right, of course, sigh. So then this should indeed be fine.

#5 lore raw
In reply to Benjamin Segall

Hello Ben,

Quoted 10 lines · Benjamin Segall

Yep, sched_class switch lets you smuggle ->throttled into !cfs_bandwidth_used(), and then that check on enqueue_throttled lets you smuggle it onto actually being enqueued, which leads to corruption. That's fun.

I think we need to use switched_from_fair though - if we are currently TASK_ON_RQ_MIGRATING, sched_class changes will skip going into dequeue_task_fair at all, so we will not see the DEQUEUE_CLASS. Triggering a migration on a throttled task can still happen I'm fairly sure.

We cannot grab the task_rq_lock() when the task is ON_RQ_MIGRATING and guard(sched_change) should never see TASK_ON_RQ_MIGRATING - the task is either fully blocked (!p->on_rq) or is queued (p->on_rq = 1) on a CPU when we are doing a class change.

Signature · 2 lines
Thanks and Regards,
Prateek
merge bot Applied
#6 lore raw
In reply to Daehyeon Ko
The following commit has been merged into the sched/urgent branch of tip:

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     c282051fc2cae2ea54770ebe435dbea090272c25
Gitweb:        https://git.kernel.org/tip/c282051fc2cae2ea54770ebe435dbea090272c25
Author:        Daehyeon Ko <[email protected]>
AuthorDate:    Fri, 23 Oct 2026 08:52:33 +09:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Fri, 06 Nov 2026 10:04:04 +01:00

sched/fair: Clear throttle state when leaving the fair class

Task-based CFS bandwidth throttling removes a task's fair sched_entity while the scheduler core still considers the task queued. p->throttled carries the fair-specific state across dequeue/enqueue operations.

DEQUEUE_CLASS removes the task from cfs_rq->throttled_limbo_list but retains the bit. If the last finite CFS quota is disabled while the task runs in another class, it is absent from the unthrottle walk and the bit survives.
Returning to fair while cfs_bandwidth_used() is false then performs a real fair enqueue with p->throttled still set.

After bandwidth is re-enabled, a class-change dequeue treats the stale bit as limbo state and skips __dequeue_task(). p->se.on_rq therefore remains set, and put_prev_task_fair() places the entity in the EEVDF tree while the task moves to RT. On exit, the fair picker can select the TASK_DEAD task. __schedule() then returns to do_task_dead(), which hits BUG().

Keep p->throttled across migrations and group moves, but clear it on DEQUEUE_CLASS because fair-specific state must not outlive fair-class membership. A later return to fair uses the normal enqueue path. That path still respects a throttled cfs_rq.

A KASAN, LOCKDEP and DEBUG_LIST v7.3 kernel reproduced the do_task_dead() BUG in 3/3 root and 3/3 delegated uid-1000 runs with CapPrm=CapEff=0 and RLIMIT_RTPRIO=1. The fix passed both matrices plus a 3/3 class-switch control with bandwidth continuously enabled, without a scheduler warning, KASAN report, lockdep splat or crash.

An AI coding assistant analyzed the state mismatch and drafted the initial reproducer, fix candidate and this changelog. The source paths, trigger, controls and results were rechecked against the cited revisions.

---
 kernel/sched/fair.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/fair.c +2 -2
index cfc9822..ad41453 100644
@@ -6964,8 +6964,8 @@ static void dequeue_throttled_task(struct task_struct *p, int flags)
 	WARN_ON_ONCE(p->se.on_rq);
 	list_del_init(&p->throttle_node);
 
-	/* task blocked after throttled */
-	if (flags & DEQUEUE_SLEEP) {
+	/* Task blocked after throttling or is leaving the fair class: */
+	if (flags & (DEQUEUE_SLEEP | DEQUEUE_CLASS)) {
 		p->throttled = false;
 		return;
 	}