sched_ext: Assorted sub-scheduler and cgroup fixes
Subsystems (3)
Tejun Heo
reviewerPATCH 1/1 sched_ext: Assorted sub-scheduler and cgroup fixes
This patchset contains three sub-scheduler fixes and a cgroup lock ordering deadlock fix.
0001-sched_ext-Reject-setting-disallow-from-init_task-out.patch 0002-sched_ext-Take-cgroup_lock-first-in-scx_cgroup_lock.patch 0003-sched_ext-Skip-sub-disable-teardown-for-never-linked.patch 0004-sched_ext-Don-t-enable-non-ext-tasks-in-the-sub-sche.patch
- 0001 makes ops.init_task() setting p->scx.disallow outside the enable path fail the scheduler. The sub-scheduler disable path could reach the load-path-only policy revert and silently rewrite a live task's policy.
- 0002 fixes a three-way deadlock among scx_cgroup_lock(), cgroup teardown draining the cpu controller's files, and a concurrent cpu.weight write. Present in released kernels since v6.18 and marked for stable.
- 0003 fixes a use-after-free when a sub-scheduler whose enable failed before linking is torn down concurrently with root disable.
- 0004 gates task enabling on the ext class in the sub-scheduler task loops. A non-ext task under an SCX_OPS_SWITCH_PARTIAL root could be marked ENABLED while staying on its own class, corrupting the task state machine.
This is against sched_ext/for-7.2-fixes (0e2f4ab68a89) and also available in the following git branch:
https://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-sub-cgroup-fixes
diffstat follows.
include/linux/sched/ext.h | 10 +++++----- kernel/sched/ext/ext.c | 47 ++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 43 insertions(+), 14 deletions(-)
--
tejun
Tejun Heo
reviewerPATCH 1/4 sched_ext: Reject setting disallow from init_task outside the enable path
The p->scx.disallow revert assumes the root enable path, where the switching loop reads the reverted policy right afterwards and leaves the task off SCX. The sub-scheduler disable path also reaches it when re-initializing the returned tasks on a root parent. Nothing reads the policy there: the task is enabled on root anyway and keeps running on the ext class with a silently rewritten policy.
Kill the sched instead, matching the fork and non-root branches, and update the disallow documentation, which equated !fork with the load path and pointed at a stale debugfs path for nr_rejected.
--- include/linux/sched/ext.h | 10 +++++----- kernel/sched/ext/ext.c | 3 +++ 2 files changed, 8 insertions(+), 5 deletions(-)
include/linux/sched/ext.h +5 -5
@@ -244,11 +244,11 @@ struct sched_ext_entity {* to %SCHED_EXT with -%EACCES. * * Can be set from ops.init_task() while the BPF scheduler is being -* loaded (!scx_init_task_args->fork). If set and the task's policy is -* already %SCHED_EXT, the task's policy is rejected and forcefully -* reverted to %SCHED_NORMAL. The number of such events are reported -* through /sys/kernel/debug/sched_ext::nr_rejected. Setting this flag -* during fork is not allowed. +* loaded. If set and the task's policy is already %SCHED_EXT, the +* task's policy is rejected and forcefully reverted to %SCHED_NORMAL. +* The number of such events are reported through +* /sys/kernel/sched_ext/nr_rejected. Setting this flag from any other +* ops.init_task() invocation, such as during fork, fails the scheduler. */ bool disallow;/* reject switching into SCX */
kernel/sched/ext/ext.c +3 -0
@@ -3606,6 +3606,9 @@ static int __scx_init_task(struct scx_sched *sch, struct task_struct *p, bool fo }else if (unlikely (fork)) {scx_error (sch,"ops.init_task() set task->scx.disallow for %s[%d] during fork" , p->comm, p->pid); + }else if (unlikely (scx_enable_state () !=SCX_ENABLING )) { +scx_error (sch,"ops.init_task() set task->scx.disallow for %s[%d] outside the enable path" , + p->comm, p->pid); }else {struct rq *rq;struct rq_flags rf;
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 2/4 sched_ext: Take cgroup_lock() first in scx_cgroup_lock()
In reply to Tejun Heoscx_cgroup_lock() write-locks scx_cgroup_ops_rwsem and then takes cgroup_lock(), which can deadlock through kernfs:
scx enable/disable cgroup rmdir cpu.weight write ------------------ ------------ ----------------
cgroup_lock()
percpu_down_write(rwsem)
cgroup_lock()
kernfs_get_active() percpu_down_read(rwsem)
kernfs_drain()
The enable path waits for the rmdir to release cgroup_mutex. The rmdir, deactivating the cpu controller's files, waits in kernfs_drain() for the write's active reference. The write, in scx_group_set_weight(), waits for the rwsem behind the pending writer.
Take cgroup_lock() first. The set_* paths take no cgroup locks inside the read side, so a pending write-lock then only waits for read sections that always run to completion, and no dependency from the rwsem back to cgroup_mutex remains.
--- kernel/sched/ext/ext.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
kernel/sched/ext/ext.c +7 -2
@@ -4580,20 +4580,25 @@ static struct cgroup *root_cgroup(void) return &cgrp_dfl_root.cgrp; } +/* +* cgroup_lock() must nest outside the rwsem write side: a writer waiting +* for cgroup_mutex deadlocks with cgroup teardown, which holds it while +* draining a set_* file write blocked on the rwsem behind the writer. +*/ static void scx_cgroup_lock (void ) { +cgroup_lock ();#ifdef CONFIG_EXT_GROUP_SCHED percpu_down_write (&scx_cgroup_ops_rwsem);#endif -cgroup_lock (); }static void scx_cgroup_unlock (void ) { -cgroup_unlock ();#ifdef CONFIG_EXT_GROUP_SCHED percpu_up_write (&scx_cgroup_ops_rwsem);#endif +cgroup_unlock (); }#else /* CONFIG_EXT_GROUP_SCHED || CONFIG_EXT_SUB_SCHED */ static inline struct cgroup *root_cgroup (void ) {return NULL ; }
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 3/4 sched_ext: Skip sub-disable teardown for never-linked sub-schedulers
In reply to Tejun HeoA sub-scheduler enable can fail before scx_link_sched() links the sched into the hierarchy, e.g. when the parent is already being disabled, and cleanup still runs the full scx_sub_disable().
That is racy against root disable: drain_descendants() is the only ordering between a sub's disable-time task walk and root disable's all-task teardown, and an unlinked sub is invisible to it. Root's teardown can thus run between the never-linked sub's drain and its walk, exiting every task to no scheduler.
The walk then trips the membership WARN and re-homes the exited tasks onto the dying hierarchy, a use-after-free.
Skip the cgroup ownership reset and the task walk if @sch was never linked, indicated by the empty ->sibling as unlinking only happens later in the same function. The membership WARN remains valid: a linked sub is always waited on by an ancestor's drain.
--- kernel/sched/ext/ext.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
kernel/sched/ext/ext.c +12 -2
@@ -5937,6 +5937,15 @@ static void scx_sub_disable(struct scx_sched *sch)percpu_down_write (&scx_fork_rwsem);scx_cgroup_lock (); +/* +* An enable that failed before scx_link_sched() never owned a cgroup or +* task and won't be waited on by an ancestor's drain_descendants(). +* Nothing to reparent and walking the tasks can misbehave as the task +* ownership invariant (either owned by self or parent) does not hold. +*/ +if (list_empty (&sch->sibling)) + goto dump; +set_cgroup_sched (sch_cgroup (sch), parent);scx_task_iter_start (&sti, sch->cgrp);
@@ -5949,8 +5958,8 @@ static void scx_sub_disable(struct scx_sched *sch)continue ;/* -* By the time control reaches here, all descendant schedulers -* should already have been disabled. +* By the time control reaches here, all linked descendant +* schedulers should have been disabled. */ WARN_ON_ONCE (!scx_task_on_sched (sch, p));
@@ -6017,6 +6026,7 @@ static void scx_sub_disable(struct scx_sched *sch) }scx_task_iter_stop (&sti); +dump:scx_disable_dump (sch);scx_cgroup_unlock ();
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 4/4 sched_ext: Don't enable non-ext tasks in the sub-sched task loops
In reply to Tejun HeoRoot enable and scx_post_fork() enable a task only if it's on the ext class. Tasks on other classes, possible under an SCX_OPS_SWITCH_PARTIAL root, are left READY and enabled by switching_to_scx() when they switch over. The sub enable-commit pass and the sub-disable re-home loop enable unconditionally, so a fair-class READY task in the subtree becomes ENABLED while not on sched_ext. A later switch to SCHED_EXT then trips the task state validation WARN (ENABLED with the previous state not READY) and calls ops.enable() a second time.
Gate scx_enable_task() on the task's class in both loops.
--- kernel/sched/ext/ext.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
kernel/sched/ext/ext.c +16 -5
@@ -6010,15 +6010,22 @@ static void scx_sub_disable(struct scx_sched *sch)/* * $p is initialized for $parent and still attached to * @sch. Disable and exit for @sch, switch over to -* $parent, override the state to READY to account for -* $p having already been initialized, and then enable. +* $parent and override the state to READY to account +* for $p having already been initialized. */ scx_disable_and_exit_task (sch, p);scx_set_task_state (p,SCX_TASK_INIT_BEGIN );scx_set_task_state (p,SCX_TASK_INIT );scx_set_task_sched (p, parent);scx_set_task_state (p,SCX_TASK_READY ); -scx_enable_task (parent, p); + +/* +* A task on a non-ext class, possible under an +* %SCX_OPS_SWITCH_PARTIAL root, stays READY and is +* enabled by switching_to_scx() if it switches over. +*/ +if (p->sched_class == &ext_sched_class) +scx_enable_task (parent, p); }task_rq_unlock (rq, p, &rf);
@@ -7726,10 +7733,14 @@ static void scx_sub_enable_workfn(struct kthread_work *work)/* * $p is now only initialized for @sch and READY, which -* is what we want. Assign it to @sch and enable. +* is what we want. Assign it to @sch and, if it's on +* the ext class, enable. A non-ext task, possible under +* an %SCX_OPS_SWITCH_PARTIAL root, stays READY and is +* enabled by switching_to_scx() if it switches over. */ scx_set_task_sched (p, sch); -scx_enable_task (sch, p); +if (p->sched_class == &ext_sched_class) +scx_enable_task (sch, p); p->scx.flags &= ~SCX_TASK_SUB_INIT ; }
Signature · 1 line
2.55.0
Andrea Righi
reviewerHi Tejun,
Quoted 16 lines · Tejun Heo
This patchset contains three sub-scheduler fixes and a cgroup lock ordering deadlock fix.
0001-sched_ext-Reject-setting-disallow-from-init_task-out.patch 0002-sched_ext-Take-cgroup_lock-first-in-scx_cgroup_lock.patch 0003-sched_ext-Skip-sub-disable-teardown-for-never-linked.patch 0004-sched_ext-Don-t-enable-non-ext-tasks-in-the-sub-sche.patch
- 0001 makes
ops.init_task()settingp->scx.disallowoutside the enable path fail the scheduler. The sub-scheduler disable path could reach the load-path-only policy revert and silently rewrite a live task's policy.- 0002 fixes a three-way deadlock among
scx_cgroup_lock(), cgroup teardown draining the cpu controller's files, and a concurrent cpu.weight write. Present in released kernels since v6.18 and marked for stable.
Quoted 7 lines
- 0003 fixes a use-after-free when a sub-scheduler whose enable failed before linking is torn down concurrently with root disable.
- 0004 gates task enabling on the ext class in the sub-scheduler task loops. A non-ext task under an
SCX_OPS_SWITCH_PARTIALroot could be marked ENABLED while staying on its own class, corrupting the task state machine.
I randomly triggered the following with this applied, but it looks unrelated to these changes:
[ 140.251732] BUG: unable to handle page fault for address: 0000000000020037 [ 140.251914] #PF: supervisor write access in kernel mode [ 140.252020] #PF: error_code(0x0002) - not-present page [ 140.252163] PGD 0 P4D 0 [ 140.252237] Oops: Oops: 0002 [#1] SMP NOPTI [ 140.252339] CPU: 5 UID: 0 PID: 103 Comm: kworker/u65:1 Not tainted 7.1.0-virtme #1 PREEMPT(full) [ 140.252575] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 140.252704] Workqueue: events_unbound scx_watchdog_workfn [ 140.252766] Sched_ext: qmap (enabled+all), task: runnable_at=-1ms [ 140.252768] RIP: 0010:scx_claim_exit+0xfd/0x190 [ 140.252897] Code: 02 00 74 24 65 8b 05 4a ac ce 02 85 c0 75 19 65 8b 05 27 68 ce 02 a9 ff ff ff 7f 75 0b 65 8b 05 a1 a8 ce 02 85 c0 75 46 31 c0 <f0> 45 0f b1 a7 28 04 00 00 75 14 41 c6 87 b4 03 00 00 01 49 8d bf [ 140.253102] RSP: 0018:ffffc900003f7d10 EFLAGS: 00010046 [ 140.253160] RAX: 0000000000000000 RBX: ffff888125ed5000 RCX: ffffea0004768bb2 [ 140.253232] RDX: 000000008881093f RSI: ffffffff83685140 RDI: ffff888100c45440 [ 140.253316] RBP: 0000000000000001 R08: ffffffff8411c438 R09: ffff888100c45440 [ 140.253406] R10: 00000000000003e5 R11: ffff888100c44780 R12: 0000000000000044 [ 140.253501] R13: 0000000000000000 R14: 0000000000000016 R15: 000000000001fc0f [ 140.253595] FS: 0000000000000000(0000) GS:ffff8882b379a000(0000) knlGS:0000000000000000 [ 140.253681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 140.253791] CR2: 0000000000020037 CR3: 000000000364c000 CR4: 0000000000750ef0 [ 140.253865] PKRU: 55555554 [ 140.253903] Call Trace: [ 140.253966] <TASK> [ 140.253996] scx_vexit+0x3a/0x170 [ 140.254041] __scx_exit+0x51/0x70 [ 140.254087] scx_watchdog_workfn+0x25f/0x360 [ 140.254156] ? process_scheduled_works+0x219/0x510 [ 140.254208] process_scheduled_works+0x265/0x510 [ 140.254272] worker_thread+0x226/0x360 [ 140.254311] ? __pfx_worker_thread+0x10/0x10 [ 140.254386] kthread+0x10c/0x140 [ 140.254425] ? __pfx_kthread+0x10/0x10 [ 140.254465] ret_from_fork+0x189/0x330 [ 140.254521] ? __pfx_kthread+0x10/0x10 [ 140.254562] ret_from_fork_asm+0x1a/0x30 [ 140.254608] </TASK> [ 140.254633] Modules linked in: [ 140.254689] CR2: 0000000000020037 [ 140.254733] ---[ end trace 0000000000000000 ]---
This might be a pre-existent bug, I'll investigate a bit.
Thanks,
-Andrea
Quoted 14 lines
This is against sched_ext/for-7.2-fixes (
0e2f4ab68a89) and also available in the following git branch:https://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-sub-cgroup-fixes
diffstat follows.
include/linux/sched/ext.h | 10 +++++----- kernel/sched/ext/ext.c | 47 ++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 43 insertions(+), 14 deletions(-)--
tejun
Tejun Heo
reviewerHello, Andrea.
Quoted 2 lines · Andrea Righi
I randomly triggered the following with this applied, but it looks unrelated to these changes:
...
Quoted 1 line
This might be a pre-existent bug, I'll investigate a bit.
It is pre-existing. Cgroup migration doesn't update p->scx.sched, so a task moved out of a sub-scheduler's cgroup keeps pointing at it. The sub-scheduler's disable path only walks its own subtree and misses the task, and the task ends up holding a dangling pointer once the scheduler is freed. In your crash, the watchdog read a garbage timeout from the recycled memory, declared a stall, and the exit propagation then chased a garbage ->children pointer into scx_claim_exit().
Reproducer, on this branch, KASAN reports use-after-free within seconds:
1. Run scx_qmap.
2. Put a CPU hog in cgroup A.
3. scx_qmap -c /sys/fs/cgroup/A
4. Move the hog to a different cgroup.
5. Kill the sub-scheduler.
I have a fix series which re-homes tasks on cgroup migration in the works and will post it soon.
Thanks.
--
tejun
Quoted 18 lines · Tejun Heo
Hello, Andrea.
Quoted 2 lines · Andrea Righi
I randomly triggered the following with this applied, but it looks unrelated to these changes:
...
Quoted 1 line
This might be a pre-existent bug, I'll investigate a bit.
It is pre-existing. Cgroup migration doesn't update
p->scx.sched, so a task moved out of a sub-scheduler's cgroup keeps pointing at it. The sub-scheduler's disable path only walks its own subtree and misses the task, and the task ends up holding a dangling pointer once the scheduler is freed. In your crash, the watchdog read a garbage timeout from the recycled memory, declared a stall, and the exit propagation then chased a garbage ->children pointer intoscx_claim_exit().Reproducer, on this branch, KASAN reports use-after-free within seconds:
Quoted 8 lines
1. Run
scx_qmap.
2. Put a CPU hog in cgroup A.
3.scx_qmap-c/sys/fs/cgroup/A
4. Move the hog to a different cgroup.
5. Kill the sub-scheduler.I have a fix series which re-homes tasks on cgroup migration in the works and will post it soon.
Oh I see, thanks for the details. About this patch series, everything looks good to me, so:
-Andrea
Tejun Heo
reviewerHello,
Applied 1-4 to sched_ext/for-7.2-fixes with Andrea's Reviewed-by added.
Thanks.
--
tejun