Skip to contentSkip to conversation

sched_ext: Assorted sub-scheduler and cgroup fixes

PATCH In review Tejun Heo 2 participants 9 messages Jul 16 – Jul 18

Subsystems (3)
lore
Share & tools
lore

Tejun Heo

author reviewer
#0 lore raw

PATCH 1/1 sched_ext: Assorted sub-scheduler and cgroup fixes

This patchset contains three sub-scheduler fixes and a cgroup lock ordering deadlock fix.

0001-sched_ext-Reject-setting-disallow-from-init_task-out.patch 0002-sched_ext-Take-cgroup_lock-first-in-scx_cgroup_lock.patch 0003-sched_ext-Skip-sub-disable-teardown-for-never-linked.patch 0004-sched_ext-Don-t-enable-non-ext-tasks-in-the-sub-sche.patch

- 0001 makes ops.init_task() setting p->scx.disallow outside the enable path fail the scheduler. The sub-scheduler disable path could reach the load-path-only policy revert and silently rewrite a live task's policy.

- 0002 fixes a three-way deadlock among scx_cgroup_lock(), cgroup teardown draining the cpu controller's files, and a concurrent cpu.weight write. Present in released kernels since v6.18 and marked for stable.

- 0003 fixes a use-after-free when a sub-scheduler whose enable failed before linking is torn down concurrently with root disable.

- 0004 gates task enabling on the ext class in the sub-scheduler task loops. A non-ext task under an SCX_OPS_SWITCH_PARTIAL root could be marked ENABLED while staying on its own class, corrupting the task state machine.

This is against sched_ext/for-7.2-fixes (0e2f4ab68a89) and also available in the following git branch:

https://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-sub-cgroup-fixes

diffstat follows.

 include/linux/sched/ext.h | 10 +++++-----
 kernel/sched/ext/ext.c    | 47 ++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 43 insertions(+), 14 deletions(-)

--
tejun

author reviewer
#1 lore raw

PATCH 1/4 sched_ext: Reject setting disallow from init_task outside the enable path

The p->scx.disallow revert assumes the root enable path, where the switching loop reads the reverted policy right afterwards and leaves the task off SCX. The sub-scheduler disable path also reaches it when re-initializing the returned tasks on a root parent. Nothing reads the policy there: the task is enabled on root anyway and keeps running on the ext class with a silently rewritten policy.

Kill the sched instead, matching the fork and non-root branches, and update the disallow documentation, which equated !fork with the load path and pointed at a stale debugfs path for nr_rejected.

Fixes 337ec00b1d9c ("sched_ext: Implement cgroup sub-sched enabling and disabling")
---
 include/linux/sched/ext.h | 10 +++++-----
 kernel/sched/ext/ext.c    |  3 +++
 2 files changed, 8 insertions(+), 5 deletions(-)
include/linux/sched/ext.h +5 -5
index 20b2343aa344..87e353f7e011 100644
@@ -244,11 +244,11 @@ struct sched_ext_entity {
 	 * to %SCHED_EXT with -%EACCES.
 	 *
 	 * Can be set from ops.init_task() while the BPF scheduler is being
-	 * loaded (!scx_init_task_args->fork). If set and the task's policy is
-	 * already %SCHED_EXT, the task's policy is rejected and forcefully
-	 * reverted to %SCHED_NORMAL. The number of such events are reported
-	 * through /sys/kernel/debug/sched_ext::nr_rejected. Setting this flag
-	 * during fork is not allowed.
+	 * loaded. If set and the task's policy is already %SCHED_EXT, the
+	 * task's policy is rejected and forcefully reverted to %SCHED_NORMAL.
+	 * The number of such events are reported through
+	 * /sys/kernel/sched_ext/nr_rejected. Setting this flag from any other
+	 * ops.init_task() invocation, such as during fork, fails the scheduler.
 	 */
 	bool			disallow;	/* reject switching into SCX */
 
kernel/sched/ext/ext.c +3 -0
index e3fa7b2fac9d..46f135bddd46 100644
@@ -3606,6 +3606,9 @@ static int __scx_init_task(struct scx_sched *sch, struct task_struct *p, bool fo
 		} else if (unlikely(fork)) {
 			scx_error(sch, "ops.init_task() set task->scx.disallow for %s[%d] during fork",
 				  p->comm, p->pid);
+		} else if (unlikely(scx_enable_state() != SCX_ENABLING)) {
+			scx_error(sch, "ops.init_task() set task->scx.disallow for %s[%d] outside the enable path",
+				  p->comm, p->pid);
 		} else {
 			struct rq *rq;
 			struct rq_flags rf;
Signature · 1 line
2.55.0
author reviewer
#2 lore raw

PATCH 2/4 sched_ext: Take cgroup_lock() first in scx_cgroup_lock()

In reply to Tejun Heo

scx_cgroup_lock() write-locks scx_cgroup_ops_rwsem and then takes cgroup_lock(), which can deadlock through kernfs:

  scx enable/disable         cgroup rmdir           cpu.weight write
  ------------------         ------------           ----------------

cgroup_lock()
percpu_down_write(rwsem)
cgroup_lock()
kernfs_get_active() percpu_down_read(rwsem)
kernfs_drain()

The enable path waits for the rmdir to release cgroup_mutex. The rmdir, deactivating the cpu controller's files, waits in kernfs_drain() for the write's active reference. The write, in scx_group_set_weight(), waits for the rwsem behind the pending writer.

Take cgroup_lock() first. The set_* paths take no cgroup locks inside the read side, so a pending write-lock then only waits for read sections that always run to completion, and no dependency from the rwsem back to cgroup_mutex remains.

Fixes a5bd6ba30b33 ("sched_ext: Use cgroup_lock/unlock() to synchronize against cgroup operations")
Cc [email protected] # v6.18+
---
 kernel/sched/ext/ext.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
kernel/sched/ext/ext.c +7 -2
index 46f135bddd46..58898cd0727b 100644
@@ -4580,20 +4580,25 @@ static struct cgroup *root_cgroup(void)
 	return &cgrp_dfl_root.cgrp;
 }
 
+/*
+ * cgroup_lock() must nest outside the rwsem write side: a writer waiting
+ * for cgroup_mutex deadlocks with cgroup teardown, which holds it while
+ * draining a set_* file write blocked on the rwsem behind the writer.
+ */
 static void scx_cgroup_lock(void)
 {
+	cgroup_lock();
 #ifdef CONFIG_EXT_GROUP_SCHED
 	percpu_down_write(&scx_cgroup_ops_rwsem);
 #endif
-	cgroup_lock();
 }
 
 static void scx_cgroup_unlock(void)
 {
-	cgroup_unlock();
 #ifdef CONFIG_EXT_GROUP_SCHED
 	percpu_up_write(&scx_cgroup_ops_rwsem);
 #endif
+	cgroup_unlock();
 }
 #else	/* CONFIG_EXT_GROUP_SCHED || CONFIG_EXT_SUB_SCHED */
 static inline struct cgroup *root_cgroup(void) { return NULL; }
Signature · 1 line
2.55.0
author reviewer
#3 lore raw

PATCH 3/4 sched_ext: Skip sub-disable teardown for never-linked sub-schedulers

In reply to Tejun Heo

A sub-scheduler enable can fail before scx_link_sched() links the sched into the hierarchy, e.g. when the parent is already being disabled, and cleanup still runs the full scx_sub_disable().

That is racy against root disable: drain_descendants() is the only ordering between a sub's disable-time task walk and root disable's all-task teardown, and an unlinked sub is invisible to it. Root's teardown can thus run between the never-linked sub's drain and its walk, exiting every task to no scheduler.

The walk then trips the membership WARN and re-homes the exited tasks onto the dying hierarchy, a use-after-free.

Skip the cgroup ownership reset and the task walk if @sch was never linked, indicated by the empty ->sibling as unlinking only happens later in the same function. The membership WARN remains valid: a linked sub is always waited on by an ancestor's drain.

Fixes 337ec00b1d9c ("sched_ext: Implement cgroup sub-sched enabling and disabling")
---
 kernel/sched/ext/ext.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
kernel/sched/ext/ext.c +12 -2
index 58898cd0727b..98dd7df88db4 100644
@@ -5937,6 +5937,15 @@ static void scx_sub_disable(struct scx_sched *sch)
 	percpu_down_write(&scx_fork_rwsem);
 	scx_cgroup_lock();
 
+	/*
+	 * An enable that failed before scx_link_sched() never owned a cgroup or
+	 * task and won't be waited on by an ancestor's drain_descendants().
+	 * Nothing to reparent and walking the tasks can misbehave as the task
+	 * ownership invariant (either owned by self or parent) does not hold.
+	 */
+	if (list_empty(&sch->sibling))
+		goto dump;
+
 	set_cgroup_sched(sch_cgroup(sch), parent);
 
 	scx_task_iter_start(&sti, sch->cgrp);
@@ -5949,8 +5958,8 @@ static void scx_sub_disable(struct scx_sched *sch)
 			continue;
 
 		/*
-		 * By the time control reaches here, all descendant schedulers
-		 * should already have been disabled.
+		 * By the time control reaches here, all linked descendant
+		 * schedulers should have been disabled.
 		 */
 		WARN_ON_ONCE(!scx_task_on_sched(sch, p));
 
@@ -6017,6 +6026,7 @@ static void scx_sub_disable(struct scx_sched *sch)
 	}
 	scx_task_iter_stop(&sti);
 
+dump:
 	scx_disable_dump(sch);
 
 	scx_cgroup_unlock();
Signature · 1 line
2.55.0
author reviewer
#4 lore raw

PATCH 4/4 sched_ext: Don't enable non-ext tasks in the sub-sched task loops

In reply to Tejun Heo

Root enable and scx_post_fork() enable a task only if it's on the ext class. Tasks on other classes, possible under an SCX_OPS_SWITCH_PARTIAL root, are left READY and enabled by switching_to_scx() when they switch over. The sub enable-commit pass and the sub-disable re-home loop enable unconditionally, so a fair-class READY task in the subtree becomes ENABLED while not on sched_ext. A later switch to SCHED_EXT then trips the task state validation WARN (ENABLED with the previous state not READY) and calls ops.enable() a second time.

Gate scx_enable_task() on the task's class in both loops.

Fixes 337ec00b1d9c ("sched_ext: Implement cgroup sub-sched enabling and disabling")
---
 kernel/sched/ext/ext.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
kernel/sched/ext/ext.c +16 -5
index 98dd7df88db4..18183062f751 100644
@@ -6010,15 +6010,22 @@ static void scx_sub_disable(struct scx_sched *sch)
 			/*
 			 * $p is initialized for $parent and still attached to
 			 * @sch. Disable and exit for @sch, switch over to
-			 * $parent, override the state to READY to account for
-			 * $p having already been initialized, and then enable.
+			 * $parent and override the state to READY to account
+			 * for $p having already been initialized.
 			 */
 			scx_disable_and_exit_task(sch, p);
 			scx_set_task_state(p, SCX_TASK_INIT_BEGIN);
 			scx_set_task_state(p, SCX_TASK_INIT);
 			scx_set_task_sched(p, parent);
 			scx_set_task_state(p, SCX_TASK_READY);
-			scx_enable_task(parent, p);
+
+			/*
+			 * A task on a non-ext class, possible under an
+			 * %SCX_OPS_SWITCH_PARTIAL root, stays READY and is
+			 * enabled by switching_to_scx() if it switches over.
+			 */
+			if (p->sched_class == &ext_sched_class)
+				scx_enable_task(parent, p);
 		}
 
 		task_rq_unlock(rq, p, &rf);
@@ -7726,10 +7733,14 @@ static void scx_sub_enable_workfn(struct kthread_work *work)
 
 			/*
 			 * $p is now only initialized for @sch and READY, which
-			 * is what we want. Assign it to @sch and enable.
+			 * is what we want. Assign it to @sch and, if it's on
+			 * the ext class, enable. A non-ext task, possible under
+			 * an %SCX_OPS_SWITCH_PARTIAL root, stays READY and is
+			 * enabled by switching_to_scx() if it switches over.
 			 */
 			scx_set_task_sched(p, sch);
-			scx_enable_task(sch, p);
+			if (p->sched_class == &ext_sched_class)
+				scx_enable_task(sch, p);
 
 			p->scx.flags &= ~SCX_TASK_SUB_INIT;
 		}
Signature · 1 line
2.55.0
reviewer
#5 lore raw
In reply to Tejun Heo

Hi Tejun,

Quoted 16 lines · Tejun Heo

This patchset contains three sub-scheduler fixes and a cgroup lock ordering deadlock fix.

0001-sched_ext-Reject-setting-disallow-from-init_task-out.patch 0002-sched_ext-Take-cgroup_lock-first-in-scx_cgroup_lock.patch 0003-sched_ext-Skip-sub-disable-teardown-for-never-linked.patch 0004-sched_ext-Don-t-enable-non-ext-tasks-in-the-sub-sche.patch

- 0001 makes ops.init_task() setting p->scx.disallow outside the enable path fail the scheduler. The sub-scheduler disable path could reach the load-path-only policy revert and silently rewrite a live task's policy.

- 0002 fixes a three-way deadlock among scx_cgroup_lock(), cgroup teardown draining the cpu controller's files, and a concurrent cpu.weight write. Present in released kernels since v6.18 and marked for stable.

Quoted 7 lines

- 0003 fixes a use-after-free when a sub-scheduler whose enable failed before linking is torn down concurrently with root disable.

- 0004 gates task enabling on the ext class in the sub-scheduler task loops. A non-ext task under an SCX_OPS_SWITCH_PARTIAL root could be marked ENABLED while staying on its own class, corrupting the task state machine.

I randomly triggered the following with this applied, but it looks unrelated to these changes:

  [  140.251732] BUG: unable to handle page fault for address: 0000000000020037
  [  140.251914] #PF: supervisor write access in kernel mode
  [  140.252020] #PF: error_code(0x0002) - not-present page
  [  140.252163] PGD 0 P4D 0
  [  140.252237] Oops: Oops: 0002 [#1] SMP NOPTI
  [  140.252339] CPU: 5 UID: 0 PID: 103 Comm: kworker/u65:1 Not tainted 7.1.0-virtme #1 PREEMPT(full)
  [  140.252575] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
  [  140.252704] Workqueue: events_unbound scx_watchdog_workfn
  [  140.252766] Sched_ext: qmap (enabled+all), task: runnable_at=-1ms
  [  140.252768] RIP: 0010:scx_claim_exit+0xfd/0x190
  [  140.252897] Code: 02 00 74 24 65 8b 05 4a ac ce 02 85 c0 75 19 65 8b 05 27 68 ce 02 a9 ff ff ff 7f
  75 0b 65 8b 05 a1 a8 ce 02 85 c0 75 46 31 c0 <f0> 45 0f b1 a7 28 04 00 00 75 14 41 c6 87 b4 03 00 00 01
  49 8d bf
  [  140.253102] RSP: 0018:ffffc900003f7d10 EFLAGS: 00010046
  [  140.253160] RAX: 0000000000000000 RBX: ffff888125ed5000 RCX: ffffea0004768bb2
  [  140.253232] RDX: 000000008881093f RSI: ffffffff83685140 RDI: ffff888100c45440
  [  140.253316] RBP: 0000000000000001 R08: ffffffff8411c438 R09: ffff888100c45440
  [  140.253406] R10: 00000000000003e5 R11: ffff888100c44780 R12: 0000000000000044
  [  140.253501] R13: 0000000000000000 R14: 0000000000000016 R15: 000000000001fc0f
  [  140.253595] FS:  0000000000000000(0000) GS:ffff8882b379a000(0000) knlGS:0000000000000000
  [  140.253681] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  140.253791] CR2: 0000000000020037 CR3: 000000000364c000 CR4: 0000000000750ef0
  [  140.253865] PKRU: 55555554
  [  140.253903] Call Trace:
  [  140.253966]  <TASK>
  [  140.253996]  scx_vexit+0x3a/0x170
  [  140.254041]  __scx_exit+0x51/0x70
  [  140.254087]  scx_watchdog_workfn+0x25f/0x360
  [  140.254156]  ? process_scheduled_works+0x219/0x510
  [  140.254208]  process_scheduled_works+0x265/0x510
  [  140.254272]  worker_thread+0x226/0x360
  [  140.254311]  ? __pfx_worker_thread+0x10/0x10
  [  140.254386]  kthread+0x10c/0x140
  [  140.254425]  ? __pfx_kthread+0x10/0x10
  [  140.254465]  ret_from_fork+0x189/0x330
  [  140.254521]  ? __pfx_kthread+0x10/0x10
  [  140.254562]  ret_from_fork_asm+0x1a/0x30
  [  140.254608]  </TASK>
  [  140.254633] Modules linked in:
  [  140.254689] CR2: 0000000000020037
  [  140.254733] ---[ end trace 0000000000000000 ]---

This might be a pre-existent bug, I'll investigate a bit.

Thanks,
-Andrea

Quoted 14 lines

This is against sched_ext/for-7.2-fixes (0e2f4ab68a89) and also available in the following git branch:

https://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-sub-cgroup-fixes

diffstat follows.

 include/linux/sched/ext.h | 10 +++++-----
 kernel/sched/ext/ext.c    | 47 ++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 43 insertions(+), 14 deletions(-)

--
tejun

author reviewer
#6 lore raw

Hello, Andrea.

Quoted 2 lines · Andrea Righi

I randomly triggered the following with this applied, but it looks unrelated to these changes:

...

Quoted 1 line

This might be a pre-existent bug, I'll investigate a bit.

It is pre-existing. Cgroup migration doesn't update p->scx.sched, so a task moved out of a sub-scheduler's cgroup keeps pointing at it. The sub-scheduler's disable path only walks its own subtree and misses the task, and the task ends up holding a dangling pointer once the scheduler is freed. In your crash, the watchdog read a garbage timeout from the recycled memory, declared a stall, and the exit propagation then chased a garbage ->children pointer into scx_claim_exit().

Reproducer, on this branch, KASAN reports use-after-free within seconds:

1. Run scx_qmap.
2. Put a CPU hog in cgroup A.
3. scx_qmap -c /sys/fs/cgroup/A
4. Move the hog to a different cgroup.
5. Kill the sub-scheduler.

I have a fix series which re-homes tasks on cgroup migration in the works and will post it soon.

Thanks.

--
tejun

reviewer Reviewed-by
#7 lore raw
Quoted 18 lines · Tejun Heo

Hello, Andrea.

Quoted 2 lines · Andrea Righi

I randomly triggered the following with this applied, but it looks unrelated to these changes:

...

Quoted 1 line

This might be a pre-existent bug, I'll investigate a bit.

It is pre-existing. Cgroup migration doesn't update p->scx.sched, so a task moved out of a sub-scheduler's cgroup keeps pointing at it. The sub-scheduler's disable path only walks its own subtree and misses the task, and the task ends up holding a dangling pointer once the scheduler is freed. In your crash, the watchdog read a garbage timeout from the recycled memory, declared a stall, and the exit propagation then chased a garbage ->children pointer into scx_claim_exit().

Reproducer, on this branch, KASAN reports use-after-free within seconds:

Quoted 8 lines

1. Run scx_qmap.
2. Put a CPU hog in cgroup A.
3. scx_qmap -c /sys/fs/cgroup/A
4. Move the hog to a different cgroup.
5. Kill the sub-scheduler.

I have a fix series which re-homes tasks on cgroup migration in the works and will post it soon.

Oh I see, thanks for the details. About this patch series, everything looks good to me, so:

-Andrea