Skip to contentSkip to conversation

sched_ext: Fix deadlock with PSI trigger creation

PATCH In review Matt Fleming 4 participants 8 messages Jul 10 – Jul 12

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched_ext: Fix deadlock with PSI trigger creation

From: Matt Fleming <[email protected]>

scx_root_enable_workfn() currently takes scx_fork_rwsem for writing before acquiring cgroup_mutex. Since commit a5b98009f16d ("sched/psi:
fix race between file release and pressure write"), pressure_write() holds cgroup_mutex across psi_trigger_create(), which may call kthread_create() for the psimon kthread. kthreadd's fork then enters scx_pre_fork() and waits for the read side of scx_fork_rwsem.

This results in a deadlock. The enable worker holds scx_fork_rwsem and waits for cgroup_mutex, while the PSI writer holds cgroup_mutex and waits for psimon creation to complete. Any concurrent fork blocks on scx_pre_fork() behind the enable worker.

The hung-task detector captured all three sides of the deadlock:

scx_enable_help:
__mutex_lock
scx_enable_workfn
kthread_worker_fn

systemd:
wait_for_completion_killable
__kthread_create_on_node
kthread_create_on_node
psi_trigger_create
pressure_write
kernfs_fop_write_iter

python3:
percpu_rwsem_wait
__percpu_down_read
scx_pre_fork
sched_fork
copy_process
kernel_clone

It also identified systemd as the likely owner of the mutex on which scx_enable_help was blocked.

We reproduced this on a 128-CPU AMD EPYC 7713 by enabling scx_lavd concurrently with writes to cgroup PSI trigger files. Unrelated tasks piled up in scx_pre_fork() and process creation on the box stopped.

Fix the inversion by acquiring cgroup_mutex before scx_fork_rwsem in scx_root_enable_workfn() and releasing them in reverse order, while preserving the existing exclusion around cgroup and task initialisation.

Fixes a5b98009f16d ("sched/psi: fix race between file release and pressure write")
---
 kernel/sched/ext/ext.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
kernel/sched/ext/ext.c +5 -3
index 691d53fe0f64..ba89eafe7964 100644
@@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	/*
 	 * Lock out forks, cgroup on/offlining and moves before opening the
 	 * floodgate so that they don't wander into the operations prematurely.
+	 * cgroup_mutex must nest outside scx_fork_rwsem because cgroup file
+	 * operations may create kthreads while holding cgroup_mutex.
 	 */
+	scx_cgroup_lock();
 	percpu_down_write(&scx_fork_rwsem);
 
 	WARN_ON_ONCE(scx_init_task_enabled);
@@ -7216,7 +7219,6 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	 * while tasks are being initialized so that scx_cgroup_can_attach()
 	 * never sees uninitialized tasks.
 	 */
-	scx_cgroup_lock();
 	set_cgroup_sched(sch_cgroup(sch), sch);
 	ret = scx_cgroup_init(sch);
 	if (ret)
@@ -7283,8 +7285,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 		put_task_struct(p);
 	}
 	scx_task_iter_stop(&sti);
-	scx_cgroup_unlock();
 	percpu_up_write(&scx_fork_rwsem);
+	scx_cgroup_unlock();
 
 	/*
 	 * All tasks are READY. It's safe to turn on scx_enabled() and switch
@@ -7369,8 +7371,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	return;
 
 err_disable_unlock_all:
-	scx_cgroup_unlock();
 	percpu_up_write(&scx_fork_rwsem);
+	scx_cgroup_unlock();
 	/* we'll soon enter disable path, keep bypass on */
 err_disable:
 	mutex_unlock(&scx_enable_mutex);
Signature · 1 line
2.43.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The patch fixes an ABBA deadlock between cgroup_mutex and scx_fork_rwsem in scx_root_enable_workfn(), but leaves the exact same lock inversion unpatched in scx_root_disable(), scx_sub_disable(), and scx_sub_enable_workfn(). --

--- Patch [1]: [PATCH] sched_ext: Fix deadlock with PSI trigger creation --- commit aacffeaa1e10b05888e4970e698ee8d1863dbbbd
Author: Matt Fleming <[email protected]>

sched_ext: Fix deadlock with PSI trigger creation

This commit fixes an ABBA deadlock between cgroup_mutex and scx_fork_rwsem in scx_root_enable_workfn(). It prevents deadlocks that can occur when concurrent writes to cgroup PSI trigger files attempt to create kthreads.

kernel/sched/ext/ext.c static void scx_root_enable_workfn(struct kthread_work *work) · line 7200

Quoted 13 lines
kernel/sched/ext/ext.c +3 -0
index 691d53fe0f648..ba89eafe79643 100644
@@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	/*
 	 * Lock out forks, cgroup on/offlining and moves before opening the
 	 * floodgate so that they don't wander into the operations prematurely.
+	 * cgroup_mutex must nest outside scx_fork_rwsem because cgroup file
+	 * operations may create kthreads while holding cgroup_mutex.
 	 */
+	scx_cgroup_lock();
 	percpu_down_write(&scx_fork_rwsem);

[Severity: High]
This is a pre-existing issue, but does this same lock inversion still exist in other scheduler functions?

For example, scx_sub_disable() takes scx_fork_rwsem before scx_cgroup_lock():

kernel/sched/ext/ext.c:scx_sub_disable() {
    ...
    percpu_down_write(&scx_fork_rwsem);
    scx_cgroup_lock();
    ...
}

scx_root_disable() also takes the locks in the old order:

kernel/sched/ext/ext.c:scx_root_disable() {
    ...
    percpu_down_write(&scx_fork_rwsem);
    ...
    scx_cgroup_lock();
    ...
}

And scx_sub_enable_workfn() appears to do the same:

kernel/sched/ext/ext.c:scx_sub_enable_workfn() {
    ...
    percpu_down_write(&scx_fork_rwsem);
    scx_cgroup_lock();
    ...
}

Could a concurrent PSI trigger write still cause a deadlock if it races with these other paths?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer
#2 lore raw
In reply to Matt Fleming

Hi Matt,

Quoted 44 lines · Matt Fleming

From: Matt Fleming <[email protected]>

scx_root_enable_workfn() currently takes scx_fork_rwsem for writing before acquiring cgroup_mutex. Since commit a5b98009f16d ("sched/psi:
fix race between file release and pressure write"), pressure_write() holds cgroup_mutex across psi_trigger_create(), which may call kthread_create() for the psimon kthread. kthreadd's fork then enters scx_pre_fork() and waits for the read side of scx_fork_rwsem.

This results in a deadlock. The enable worker holds scx_fork_rwsem and waits for cgroup_mutex, while the PSI writer holds cgroup_mutex and waits for psimon creation to complete. Any concurrent fork blocks on scx_pre_fork() behind the enable worker.

The hung-task detector captured all three sides of the deadlock:

scx_enable_help:
__mutex_lock
scx_enable_workfn
kthread_worker_fn

systemd:
wait_for_completion_killable
__kthread_create_on_node
kthread_create_on_node
psi_trigger_create
pressure_write
kernfs_fop_write_iter

python3:
percpu_rwsem_wait
__percpu_down_read
scx_pre_fork
sched_fork
copy_process
kernel_clone

It also identified systemd as the likely owner of the mutex on which scx_enable_help was blocked.

We reproduced this on a 128-CPU AMD EPYC 7713 by enabling scx_lavd concurrently with writes to cgroup PSI trigger files. Unrelated tasks piled up in scx_pre_fork() and process creation on the box stopped.

Quoted 7 lines

Fix the inversion by acquiring cgroup_mutex before scx_fork_rwsem in scx_root_enable_workfn() and releasing them in reverse order, while preserving the existing exclusion around cgroup and task initialisation.

Fixes a5b98009f16d ("sched/psi: fix race between file release and pressure write")

This seems to introduce the following (running the sched_ext kselftests):

[   28.963575] ======================================================
[   28.963670] WARNING: possible circular locking dependency detected
[   28.963752] 7.1.0-virtme #1 Not tainted
[   28.963804] ------------------------------------------------------
[   28.963887] sched_ext_helpe/2619 is trying to acquire lock:
[   28.963954] ffffffff83685240 (scx_cgroup_ops_rwsem){+.+.}-{0:0}, at: scx_root_disable+0x45d/0x840
[   28.964071]
[   28.964071] but task is already holding lock:
[   28.964151] ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[   28.964260]
[   28.964260] which lock already depends on the new lock.
[   28.964260]
[   28.964355]
[   28.964355] the existing dependency chain (in reverse order) is:
[   28.964455]
[   28.964455] -> #2 (scx_fork_rwsem){++++}-{0:0}:
[   28.964539]        percpu_down_write+0x49/0x150
[   28.964610]        scx_root_enable_workfn+0x5d0/0xd30
[   28.964679]        kthread_worker_fn+0x121/0x360
[   28.964749]        kthread+0x10c/0x140
[   28.964802]        ret_from_fork+0x189/0x330
[   28.964872]        ret_from_fork_asm+0x1a/0x30
[   28.964941]
[   28.964941] -> #1 (cgroup_mutex){+.+.}-{4:4}:
[   28.965025]        __mutex_lock+0xbe/0xd80
[   28.965070]        scx_root_enable_workfn+0x5c4/0xd30
[   28.965138]        kthread_worker_fn+0x121/0x360
[   28.965210]        kthread+0x10c/0x140
[   28.965263]        ret_from_fork+0x189/0x330
[   28.965331]        ret_from_fork_asm+0x1a/0x30
[   28.965402]
[   28.965402] -> #0 (scx_cgroup_ops_rwsem){+.+.}-{0:0}:
[   28.965485]        __lock_acquire+0x14c5/0x2a40
[   28.965554]        lock_acquire+0xd3/0x280
[   28.965608]        percpu_down_write+0x49/0x150
[   28.965677]        scx_root_disable+0x45d/0x840
[   28.965745]        kthread_worker_fn+0x121/0x360
[   28.965815]        kthread+0x10c/0x140
[   28.965867]        ret_from_fork+0x189/0x330
[   28.965935]        ret_from_fork_asm+0x1a/0x30
[   28.966002]
[   28.966002] other info that might help us debug this:
[   28.966002]
[   28.966097] Chain exists of:
[   28.966097]   scx_cgroup_ops_rwsem --> cgroup_mutex --> scx_fork_rwsem
[   28.966097]
[   28.966233]  Possible unsafe locking scenario:
[   28.966233]
[   28.966314]        CPU0                    CPU1
[   28.966379]        ----                    ----
[   28.966444]   lock(scx_fork_rwsem);
[   28.966496]                                lock(cgroup_mutex);
[   28.966579]                                lock(scx_fork_rwsem);
[   28.966661]   lock(scx_cgroup_ops_rwsem);
[   28.966713]
[   28.966713]  *** DEADLOCK ***
[   28.966713]
[   28.966794] 2 locks held by sched_ext_helpe/2619:
[   28.966861]  #0: ffffffff83682818 (scx_enable_mutex){+.+.}-{4:4}, at: scx_root_disable+0xba/0x840
[   28.966974]  #1: ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[   28.967090]
[   28.967090] stack backtrace:
[   28.967158] CPU: 8 UID: 0 PID: 2619 Comm: sched_ext_helpe Not tainted 7.1.0-virtme #1 PREEMPT(full)
[   28.967164] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   28.967166] Call Trace:
[   28.967169]  <TASK>
[   28.967172]  dump_stack_lvl+0x6d/0xa0
[   28.967175]  print_circular_bug+0x2e1/0x300
[   28.967179]  check_noncircular+0x144/0x170
[   28.967182]  __lock_acquire+0x14c5/0x2a40
[   28.967202]  ? scx_root_disable+0x45d/0x840
[   28.967204]  lock_acquire+0xd3/0x280
[   28.967207]  ? scx_root_disable+0x45d/0x840
[   28.967209]  percpu_down_write+0x49/0x150
[   28.967212]  ? scx_root_disable+0x45d/0x840
[   28.967213]  scx_root_disable+0x45d/0x840
[   28.967219]  ? kthread_worker_fn+0x51/0x360
[   28.967221]  kthread_worker_fn+0x121/0x360
[   28.967223]  ? __pfx_scx_disable_workfn+0x10/0x10
[   28.967224]  ? __pfx_kthread_worker_fn+0x10/0x10
[   28.967227]  kthread+0x10c/0x140
[   28.967228]  ? __pfx_kthread+0x10/0x10
[   28.967229]  ret_from_fork+0x189/0x330
[   28.967231]  ? __pfx_kthread+0x10/0x10
[   28.967232]  ret_from_fork_asm+0x1a/0x30
[   28.967235]  </TASK>

Thanks,
-Andrea

Quoted 50 lines
---
 kernel/sched/ext/ext.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
kernel/sched/ext/ext.c +5 -3
index 691d53fe0f64..ba89eafe7964 100644
@@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	/*
 	 * Lock out forks, cgroup on/offlining and moves before opening the
 	 * floodgate so that they don't wander into the operations prematurely.
+	 * cgroup_mutex must nest outside scx_fork_rwsem because cgroup file
+	 * operations may create kthreads while holding cgroup_mutex.
 	 */
+	scx_cgroup_lock();
 	percpu_down_write(&scx_fork_rwsem);
 
 	WARN_ON_ONCE(scx_init_task_enabled);
@@ -7216,7 +7219,6 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	 * while tasks are being initialized so that scx_cgroup_can_attach()
 	 * never sees uninitialized tasks.
 	 */
-	scx_cgroup_lock();
 	set_cgroup_sched(sch_cgroup(sch), sch);
 	ret = scx_cgroup_init(sch);
 	if (ret)
@@ -7283,8 +7285,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 		put_task_struct(p);
 	}
 	scx_task_iter_stop(&sti);
-	scx_cgroup_unlock();
 	percpu_up_write(&scx_fork_rwsem);
+	scx_cgroup_unlock();
 
 	/*
 	 * All tasks are READY. It's safe to turn on scx_enabled() and switch
@@ -7369,8 +7371,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	return;
 
 err_disable_unlock_all:
-	scx_cgroup_unlock();
 	percpu_up_write(&scx_fork_rwsem);
+	scx_cgroup_unlock();
 	/* we'll soon enter disable path, keep bypass on */
 err_disable:
 	mutex_unlock(&scx_enable_mutex);
Signature · 1 line
2.43.0
reviewer
#3 lore raw
Quoted 57 lines · Andrea Righi

Hi Matt,

Quoted 44 lines · Matt Fleming

From: Matt Fleming <[email protected]>

scx_root_enable_workfn() currently takes scx_fork_rwsem for writing before acquiring cgroup_mutex. Since commit a5b98009f16d ("sched/psi:
fix race between file release and pressure write"), pressure_write() holds cgroup_mutex across psi_trigger_create(), which may call kthread_create() for the psimon kthread. kthreadd's fork then enters scx_pre_fork() and waits for the read side of scx_fork_rwsem.

This results in a deadlock. The enable worker holds scx_fork_rwsem and waits for cgroup_mutex, while the PSI writer holds cgroup_mutex and waits for psimon creation to complete. Any concurrent fork blocks on scx_pre_fork() behind the enable worker.

The hung-task detector captured all three sides of the deadlock:

scx_enable_help:
__mutex_lock
scx_enable_workfn
kthread_worker_fn

systemd:
wait_for_completion_killable
__kthread_create_on_node
kthread_create_on_node
psi_trigger_create
pressure_write
kernfs_fop_write_iter

python3:
percpu_rwsem_wait
__percpu_down_read
scx_pre_fork
sched_fork
copy_process
kernel_clone

It also identified systemd as the likely owner of the mutex on which scx_enable_help was blocked.

We reproduced this on a 128-CPU AMD EPYC 7713 by enabling scx_lavd concurrently with writes to cgroup PSI trigger files. Unrelated tasks piled up in scx_pre_fork() and process creation on the box stopped.

Quoted 7 lines

Fix the inversion by acquiring cgroup_mutex before scx_fork_rwsem in scx_root_enable_workfn() and releasing them in reverse order, while preserving the existing exclusion around cgroup and task initialisation.

Fixes a5b98009f16d ("sched/psi: fix race between file release and pressure write")

This seems to introduce the following (running the sched_ext kselftests):

Quoted 11 lines
[   28.963575] ======================================================
[   28.963670] WARNING: possible circular locking dependency detected
[   28.963752] 7.1.0-virtme #1 Not tainted
[   28.963804] ------------------------------------------------------
[   28.963887] sched_ext_helpe/2619 is trying to acquire lock:
[   28.963954] ffffffff83685240 (scx_cgroup_ops_rwsem){+.+.}-{0:0}, at: scx_root_disable+0x45d/0x840
[   28.964071]
[   28.964071] but task is already holding lock:
[   28.964151] ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[   28.964260]
[   28.964260] which lock already depends on the new lock.

Looking more at this, I think we should fix the PSI path instead, sched_ext locking order is consistently using:

scx_fork_rwsem -> scx_cgroup_ops_rwsem -> cgroup_mutex

Maybe we can rework pressure_write() to pin the cgroup and file context, then drop both cgroup_mutex and kernfs active protection before creating the PSI kthread?

Thanks,
-Andrea

Quoted 129 lines
[   28.964260]
[   28.964355]
[   28.964355] the existing dependency chain (in reverse order) is:
[   28.964455]
[   28.964455] -> #2 (scx_fork_rwsem){++++}-{0:0}:
[   28.964539]        percpu_down_write+0x49/0x150
[   28.964610]        scx_root_enable_workfn+0x5d0/0xd30
[   28.964679]        kthread_worker_fn+0x121/0x360
[   28.964749]        kthread+0x10c/0x140
[   28.964802]        ret_from_fork+0x189/0x330
[   28.964872]        ret_from_fork_asm+0x1a/0x30
[   28.964941]
[   28.964941] -> #1 (cgroup_mutex){+.+.}-{4:4}:
[   28.965025]        __mutex_lock+0xbe/0xd80
[   28.965070]        scx_root_enable_workfn+0x5c4/0xd30
[   28.965138]        kthread_worker_fn+0x121/0x360
[   28.965210]        kthread+0x10c/0x140
[   28.965263]        ret_from_fork+0x189/0x330
[   28.965331]        ret_from_fork_asm+0x1a/0x30
[   28.965402]
[   28.965402] -> #0 (scx_cgroup_ops_rwsem){+.+.}-{0:0}:
[   28.965485]        __lock_acquire+0x14c5/0x2a40
[   28.965554]        lock_acquire+0xd3/0x280
[   28.965608]        percpu_down_write+0x49/0x150
[   28.965677]        scx_root_disable+0x45d/0x840
[   28.965745]        kthread_worker_fn+0x121/0x360
[   28.965815]        kthread+0x10c/0x140
[   28.965867]        ret_from_fork+0x189/0x330
[   28.965935]        ret_from_fork_asm+0x1a/0x30
[   28.966002]
[   28.966002] other info that might help us debug this:
[   28.966002]
[   28.966097] Chain exists of:
[   28.966097]   scx_cgroup_ops_rwsem --> cgroup_mutex --> scx_fork_rwsem
[   28.966097]
[   28.966233]  Possible unsafe locking scenario:
[   28.966233]
[   28.966314]        CPU0                    CPU1
[   28.966379]        ----                    ----
[   28.966444]   lock(scx_fork_rwsem);
[   28.966496]                                lock(cgroup_mutex);
[   28.966579]                                lock(scx_fork_rwsem);
[   28.966661]   lock(scx_cgroup_ops_rwsem);
[   28.966713]
[   28.966713]  *** DEADLOCK ***
[   28.966713]
[   28.966794] 2 locks held by sched_ext_helpe/2619:
[   28.966861]  #0: ffffffff83682818 (scx_enable_mutex){+.+.}-{4:4}, at: scx_root_disable+0xba/0x840
[   28.966974]  #1: ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[   28.967090]
[   28.967090] stack backtrace:
[   28.967158] CPU: 8 UID: 0 PID: 2619 Comm: sched_ext_helpe Not tainted 7.1.0-virtme #1 PREEMPT(full)
[   28.967164] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   28.967166] Call Trace:
[   28.967169]  <TASK>
[   28.967172]  dump_stack_lvl+0x6d/0xa0
[   28.967175]  print_circular_bug+0x2e1/0x300
[   28.967179]  check_noncircular+0x144/0x170
[   28.967182]  __lock_acquire+0x14c5/0x2a40
[   28.967202]  ? scx_root_disable+0x45d/0x840
[   28.967204]  lock_acquire+0xd3/0x280
[   28.967207]  ? scx_root_disable+0x45d/0x840
[   28.967209]  percpu_down_write+0x49/0x150
[   28.967212]  ? scx_root_disable+0x45d/0x840
[   28.967213]  scx_root_disable+0x45d/0x840
[   28.967219]  ? kthread_worker_fn+0x51/0x360
[   28.967221]  kthread_worker_fn+0x121/0x360
[   28.967223]  ? __pfx_scx_disable_workfn+0x10/0x10
[   28.967224]  ? __pfx_kthread_worker_fn+0x10/0x10
[   28.967227]  kthread+0x10c/0x140
[   28.967228]  ? __pfx_kthread+0x10/0x10
[   28.967229]  ret_from_fork+0x189/0x330
[   28.967231]  ? __pfx_kthread+0x10/0x10
[   28.967232]  ret_from_fork_asm+0x1a/0x30
[   28.967235]  </TASK>

Thanks,
-Andrea

Quoted 50 lines
---
 kernel/sched/ext/ext.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
kernel/sched/ext/ext.c +5 -3
index 691d53fe0f64..ba89eafe7964 100644
@@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	/*
 	 * Lock out forks, cgroup on/offlining and moves before opening the
 	 * floodgate so that they don't wander into the operations prematurely.
+	 * cgroup_mutex must nest outside scx_fork_rwsem because cgroup file
+	 * operations may create kthreads while holding cgroup_mutex.
 	 */
+	scx_cgroup_lock();
 	percpu_down_write(&scx_fork_rwsem);
 
 	WARN_ON_ONCE(scx_init_task_enabled);
@@ -7216,7 +7219,6 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	 * while tasks are being initialized so that scx_cgroup_can_attach()
 	 * never sees uninitialized tasks.
 	 */
-	scx_cgroup_lock();
 	set_cgroup_sched(sch_cgroup(sch), sch);
 	ret = scx_cgroup_init(sch);
 	if (ret)
@@ -7283,8 +7285,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 		put_task_struct(p);
 	}
 	scx_task_iter_stop(&sti);
-	scx_cgroup_unlock();
 	percpu_up_write(&scx_fork_rwsem);
+	scx_cgroup_unlock();
 
 	/*
 	 * All tasks are READY. It's safe to turn on scx_enabled() and switch
@@ -7369,8 +7371,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)
 	return;
 
 err_disable_unlock_all:
-	scx_cgroup_unlock();
 	percpu_up_write(&scx_fork_rwsem);
+	scx_cgroup_unlock();
 	/* we'll soon enter disable path, keep bypass on */
 err_disable:
 	mutex_unlock(&scx_enable_mutex);
Signature · 1 line
2.43.0
maintainer
#4 lore raw

PATCH 1/1 cgroup: Create the psimon kthread outside of cgroup_mutex

In reply to Matt Fleming

a5b98009f16d ("sched/psi: fix race between file release and pressure write") made pressure_write() hold cgroup_mutex across psi_trigger_create(), which forks the psimon kthread for the first rtpoll trigger. As kthread creation depends on the whole fork path, the commit inadvertently created a lot of unwanted locking dependencies from cgroup_mutex.

sched_ext got hit by one: its enable path blocks forks and then grabs cgroup_mutex, so a pressure write racing a scheduler enable deadlocks, with every other fork piling up behind.

Fix it by splitting trigger creation so that the worker is forked with cgroup_mutex dropped and the kernfs active reference left broken. The latter matters because rmdir and cgroup.pressure writes drain active references under cgroup_mutex. Publishing the trigger last keeps error reporting synchronous and preserves the of->priv lifetime rules.

The trigger registered in the first stage pins the group's rtpoll machinery across the unlocked window, leaving only creation races to resolve. The catch-up poll on installation covers scheduling attempts dropped while there was no worker.

Fixes a5b98009f16d ("sched/psi: fix race between file release and pressure write")
Cc Johannes Weiner <[email protected]>
Cc Edward Adam Davis <[email protected]>
Cc Chen Ridong <[email protected]>

---
Matt, your reordering trades one deadlock for another: CLONE_INTO_CGROUP forks grab cgroup_mutex inside the scx_fork_rwsem read section, so an enable racing such a clone deadlocks the other way around. The fork has to move out of the locked sections instead. Can you verify this fixes the deadlock in your setup?

 include/linux/psi.h    |    4 ++
 kernel/cgroup/cgroup.c |   23 +++++++++++++++-
 kernel/sched/psi.c     |   69 +++++++++++++++++++++++++++++++++++++------------
 3 files changed, 78 insertions(+), 18 deletions(-)
include/linux/psi.h +3 -1
@@ -25,7 +25,9 @@ void psi_memstall_leave(unsigned long *f
 int psi_show(struct seq_file *s, struct psi_group *group, enum psi_res res);
 struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
 				       enum psi_res res, struct file *file,
-				       struct kernfs_open_file *of);
+				       struct kernfs_open_file *of,
+				       bool *need_rtpoll_worker);
+int psi_trigger_create_rtpoll_worker(struct psi_group *group);
 void psi_trigger_destroy(struct psi_trigger *t);
 
 __poll_t psi_trigger_poll(void **trigger_ptr, struct file *file,
kernel/cgroup/cgroup.c +22 -1
@@ -3996,6 +3996,7 @@ static ssize_t pressure_write(struct ker
 	struct psi_trigger *new;
 	struct cgroup *cgrp;
 	struct psi_group *psi;
+	bool need_rtpoll_worker;
 	ssize_t ret = 0;
 
 	cgrp = cgroup_kn_lock_live(of->kn, false);
@@ -4015,12 +4016,32 @@ static ssize_t pressure_write(struct ker
 	}
 
 	psi = cgroup_psi(cgrp);
-	new = psi_trigger_create(psi, buf, res, of->file, of);
+	new = psi_trigger_create(psi, buf, res, of->file, of,
+				 &need_rtpoll_worker);
 	if (IS_ERR(new)) {
 		ret = PTR_ERR(new);
 		goto out_unlock;
 	}
 
+	/*
+	 * The worker fork must run with neither cgroup_mutex nor the file's
+	 * kernfs active reference held. The latter is broken since
+	 * cgroup_kn_lock_live(). @of->priv may be released while unlocked, so
+	 * recheck before publishing @new.
+	 */
+	if (need_rtpoll_worker) {
+		cgroup_unlock();
+		ret = psi_trigger_create_rtpoll_worker(psi);
+		cgroup_lock();
+
+		if (!ret && !of->priv)
+			ret = -ENODEV;
+		if (ret) {
+			psi_trigger_destroy(new);
+			goto out_unlock;
+		}
+	}
+
 	smp_store_release(&ctx->psi.trigger, new);
 
 out_unlock:
kernel/sched/psi.c +53 -16

112 lines not on this page · raw message

#5 lore raw

Re: [PATCH cgroup/for-7.2-fixes] cgroup: Create the psimon kthread outside of cgroup_mutex

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but the psi rtpoll_timer can be re-armed after it is deleted and subsequently execute after the psi_group is freed, causing a Use-After-Free. --

--- Patch [1]: [PATCH cgroup/for-7.2-fixes] cgroup: Create the psimon kthread outside of cgroup_mutex --- commit 8790bcf0b23cb2132be9400b975c7f18fd1147b7
Author: Tejun Heo <[email protected]>

cgroup: Create the psimon kthread outside of cgroup_mutex

This commit splits the trigger creation process so the rtpoll worker thread is forked without holding cgroup_mutex. This resolves a deadlock between cgroup_mutex and the fork path during sched_ext enablement.

Quoted 4 lines
kernel/sched/psi.c +0 -0
index 848955f8893dd..b9f207c920edb 100644

[ ... ]

Quoted 6 lines
kernel/sched/psi.c +1 -1
@@ -1565,12 +1592,22 @@ static ssize_t psi_write(struct file *file, const char __user *user_buf,
 		return -EBUSY;
 	}
 
-	new = psi_trigger_create(&psi_system, buf, res, file, NULL);
+	new = psi_trigger_create(&psi_system, buf, res, file, NULL,

kernel/sched/psi.c static ssize_t psi_write(struct file *file, const char __user *user_buf, · line 1605

Quoted 10 lines
kernel/sched/psi.c +5 -0
+				 &need_rtpoll_worker);
 	if (IS_ERR(new)) {
 		mutex_unlock(&seq->lock);
 		return PTR_ERR(new);
 	}
 
+	if (need_rtpoll_worker) {
+		ret = psi_trigger_create_rtpoll_worker(&psi_system);
+		if (ret) {
+			psi_trigger_destroy(new);

[Severity: High]
This isn't a bug introduced by this patch, but can the psi rtpoll_timer be re-armed after it is deleted and subsequently execute after the psi_group is freed, causing a use-after-free?

In psi_trigger_destroy(), the timer is deleted before synchronize_rcu():

psi_trigger_destroy() {
    rcu_assign_pointer(group->rtpoll_task, NULL);
    timer_delete(&group->rtpoll_timer);
    ...
    synchronize_rcu();
}

If Thread A calls psi_schedule_rtpoll_work() and gets delayed or preempted after reading rtpoll_task inside the RCU read-side critical section:

psi_schedule_rtpoll_work() {
    rcu_read_lock();
    task = rcu_dereference(group->rtpoll_task);
    ...
    // Thread A delayed here
}

Thread B could then run psi_trigger_destroy(), delete the timer, and block in synchronize_rcu() waiting for Thread A. When Thread A resumes, it sees the task is non-NULL and calls mod_timer():

    if (likely(task))
        mod_timer(&group->rtpoll_timer, jiffies + delay);

This re-arms the timer after it was already deleted by Thread B.

Later, when the cgroup is removed and psi_cgroup_free() is called, the psi_group is freed without deleting this leaked timer. When the timer finally expires, poll_timer_fn() will attempt to access the freed group memory.

Quoted 8 lines
kernel/sched/psi.c +5 -0
+			mutex_unlock(&seq->lock);
+			return ret;
+		}
+	}
+
 	smp_store_release(&seq->private, new);
 	mutex_unlock(&seq->lock);
 
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#6 lore raw

Re: [PATCH cgroup/for-7.2-fixes] cgroup: Create the psimon kthread outside of cgroup_mutex

In reply to Tejun Heo
Quoted 5 lines · Tejun Heo

Matt, your reordering trades one deadlock for another: CLONE_INTO_CGROUP forks grab cgroup_mutex inside the scx_fork_rwsem read section, so an enable racing such a clone deadlocks the other way around. The fork has to move out of the locked sections instead. Can you verify this fixes the deadlock in your setup?

Thanks for fixing this. I'll test this out ASAP and report back.