sched_ext: TASK_DEAD tasks must be switched into SCX on ops_enable
PATCH 1/1 sched_ext: TASK_DEAD tasks must be switched into SCX on ops_enable
During scx_ops_enable(), SCX needs to invoke the sleepable ops.init_task() on every task. To do this, it does get_task_struct() on each iterated task, drop the lock and then call ops.init_task().
However, a TASK_DEAD task may already have lost all its usage count and be waiting for RCU grace period to be freed. If get_task_struct() is called on such task, use-after-free can happen. To avoid such situations, scx_ops_enable() skips initialization of TASK_DEAD tasks, which seems safe as they are never going to be scheduled again.
Unfortunately, a racing sched_setscheduler(2) can grab the task before the task is unhashed and then continue to e.g. move the task from RT to SCX after TASK_DEAD is set and ops_enable skipped the task. As the task hasn't gone through scx_ops_init_task(), scx_ops_enable_task() called from switching_to_scx() triggers the following warning:
sched_ext: Invalid task state transition 0 -> 3 for stress-ng-race-[2872]
WARNING: CPU: 6 PID: 2367 at kernel/sched/ext.c:3327 scx_ops_enable_task+0x18f/0x1f0 ... RIP: 0010:scx_ops_enable_task+0x18f/0x1f0 ... switching_to_scx+0x13/0xa0 __sched_setscheduler+0x84e/0xa50 do_sched_setscheduler+0x104/0x1c0 __x64_sys_sched_setscheduler+0x18/0x30 do_syscall_64+0x7b/0x140 entry_SYSCALL_64_after_hwframe+0x76/0x7e
As in the ops_disable path, it just doesn't seem like a good idea to leave any task in an inconsistent state, even when the task is dead. The root cause is ops_enable not being able to tell reliably whether a task is truly dead (no one else is looking at it and it's about to be freed) and was testing TASK_DEAD instead. Fix it by testing the task's usage count directly.
- ops_init no longer ignores TASK_DEAD tasks. As now all users iterate all tasks, @include_dead is removed from scx_task_iter_next_locked() along with dead task filtering.
- tryget_task_struct() is added. Tasks are skipped iff tryget_task_struct() fails.
--- include/linux/sched/task.h | 5 +++++ kernel/sched/ext.c | 30 +++++++++++++----------------- 2 files changed, 18 insertions(+), 17 deletions(-)
include/linux/sched/task.h +5 -0
@@ -120,6 +120,11 @@ static inline struct task_struct *get_tareturn t; } +static inline struct task_struct *tryget_task_struct (struct task_struct *t) +{ +return refcount_inc_not_zero (&t->usage) ? t :NULL ; +} +extern void __put_task_struct (struct task_struct *t);extern void __put_task_struct_rcu_cb (struct rcu_head *rhp);
kernel/sched/ext.c +13 -17
@@ -1243,11 +1243,10 @@ static struct task_struct *scx_task_iter* whether they would like to filter out dead tasks. See scx_task_iter_init() * for details. */ -static struct task_struct * -scx_task_iter_next_locked (struct scx_task_iter *iter,bool include_dead) +static struct task_struct *scx_task_iter_next_locked (struct scx_task_iter *iter) {struct task_struct *p; -retry: +scx_task_iter_rq_unlock (iter);while ((p =scx_task_iter_next (iter))) {
@@ -1285,16 +1284,6 @@ retry: iter->rq =task_rq_lock (p, &iter->rf); iter->locked = p; -/* -* If we see %TASK_DEAD, @p already disabled preemption, is about to do -* the final __schedule(), won't ever need to be scheduled again and can -* thus be safely ignored. If we don't see %TASK_DEAD, @p can't enter -* the final __schedle() while we're locking its rq and thus will stay -* alive until the rq is unlocked. -*/ -if (!include_dead &&READ_ONCE (p->__state) ==TASK_DEAD ) - goto retry; -return p; }
@@ -4054,7 +4043,7 @@ static void scx_ops_disable_workfn(struc* The BPF scheduler is going away. All tasks including %TASK_DEAD ones * must be switched out and exited synchronously. */ -while ((p =scx_task_iter_next_locked (&sti, true))) { +while ((p =scx_task_iter_next_locked (&sti))) {const struct sched_class *old_class = p->sched_class;struct sched_enq_and_set_ctx ctx;
@@ -4685,8 +4674,15 @@ static int scx_ops_enable(struct sched_espin_lock_irq (&scx_tasks_lock);scx_task_iter_init (&sti); -while ((p =scx_task_iter_next_locked (&sti, false))) { -get_task_struct (p); +while ((p =scx_task_iter_next_locked (&sti))) { +/* +* @p may already be dead, have lost all its usages counts and +* be waiting for RCU grace period before being freed. @p can't +* be initialized for SCX in such cases and should be ignored. +*/ +if (!tryget_task_struct (p)) +continue ; +scx_task_iter_rq_unlock (&sti);spin_unlock_irq (&scx_tasks_lock);
@@ -4739,7 +4735,7 @@ static int scx_ops_enable(struct sched_eWRITE_ONCE (scx_switching_all, !(ops->flags &SCX_OPS_SWITCH_PARTIAL ));scx_task_iter_init (&sti); -while ((p =scx_task_iter_next_locked (&sti, false))) { +while ((p =scx_task_iter_next_locked (&sti))) {const struct sched_class *old_class = p->sched_class;struct sched_enq_and_set_ctx ctx;
Tejun Heo
AppliedQuoted 39 lines · Tejun Heo
During
scx_ops_enable(), SCX needs to invoke the sleepableops.init_task()on every task. To do this, it doesget_task_struct()on each iterated task, drop the lock and then callops.init_task().However, a
TASK_DEADtask may already have lost all its usage count and be waiting for RCU grace period to be freed. Ifget_task_struct()is called on such task, use-after-free can happen. To avoid such situations,scx_ops_enable()skips initialization ofTASK_DEADtasks, which seems safe as they are never going to be scheduled again.Unfortunately, a racing
sched_setscheduler(2)can grab the task before the task is unhashed and then continue to e.g. move the task from RT to SCX afterTASK_DEADis set andops_enableskipped the task. As the task hasn't gone throughscx_ops_init_task(),scx_ops_enable_task()called fromswitching_to_scx()triggers the following warning:
sched_ext: Invalid task state transition 0 -> 3 for stress-ng-race-[2872]WARNING: CPU: 6 PID: 2367 at kernel/sched/ext.c:3327 scx_ops_enable_task+0x18f/0x1f0 ... RIP: 0010:scx_ops_enable_task+0x18f/0x1f0 ... switching_to_scx+0x13/0xa0 __sched_setscheduler+0x84e/0xa50 do_sched_setscheduler+0x104/0x1c0 __x64_sys_sched_setscheduler+0x18/0x30 do_syscall_64+0x7b/0x140 entry_SYSCALL_64_after_hwframe+0x76/0x7eAs in the
ops_disablepath, it just doesn't seem like a good idea to leave any task in an inconsistent state, even when the task is dead. The root cause isops_enablenot being able to tell reliably whether a task is truly dead (no one else is looking at it and it's about to be freed) and was testingTASK_DEADinstead. Fix it by testing the task's usage count directly.-
ops_initno longer ignoresTASK_DEADtasks. As now all users iterate all tasks, @include_dead is removed fromscx_task_iter_next_locked()along with dead task filtering.
Quoted 6 lines
-
tryget_task_struct()is added. Tasks are skipped ifftryget_task_struct()fails.Signed-off-by Tejun Heo <[email protected]>Cc David Vernet <[email protected]>Cc Peter Zijlstra <[email protected]>
Applied to sched_ext/for-6.12.
Thanks.
Signature · 1 line
tejun