sched_ext: Fix deadlock with PSI trigger creation
Subsystems (5)
PATCH 1/1 sched_ext: Fix deadlock with PSI trigger creation
From: Matt Fleming <[email protected]>
scx_root_enable_workfn() currently takes scx_fork_rwsem for writing before acquiring cgroup_mutex. Since commit a5b98009f16d ("sched/psi:
fix race between file release and pressure write"), pressure_write() holds cgroup_mutex across psi_trigger_create(), which may call kthread_create() for the psimon kthread. kthreadd's fork then enters scx_pre_fork() and waits for the read side of scx_fork_rwsem.
This results in a deadlock. The enable worker holds scx_fork_rwsem and waits for cgroup_mutex, while the PSI writer holds cgroup_mutex and waits for psimon creation to complete. Any concurrent fork blocks on scx_pre_fork() behind the enable worker.
The hung-task detector captured all three sides of the deadlock:
scx_enable_help:
__mutex_lock
scx_enable_workfn
kthread_worker_fn
systemd:
wait_for_completion_killable
__kthread_create_on_node
kthread_create_on_node
psi_trigger_create
pressure_write
kernfs_fop_write_iter
python3:
percpu_rwsem_wait
__percpu_down_read
scx_pre_fork
sched_fork
copy_process
kernel_clone
It also identified systemd as the likely owner of the mutex on which scx_enable_help was blocked.
We reproduced this on a 128-CPU AMD EPYC 7713 by enabling scx_lavd concurrently with writes to cgroup PSI trigger files. Unrelated tasks piled up in scx_pre_fork() and process creation on the box stopped.
Fix the inversion by acquiring cgroup_mutex before scx_fork_rwsem in scx_root_enable_workfn() and releasing them in reverse order, while preserving the existing exclusion around cgroup and task initialisation.
--- kernel/sched/ext/ext.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
kernel/sched/ext/ext.c +5 -3
@@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)/* * Lock out forks, cgroup on/offlining and moves before opening the * floodgate so that they don't wander into the operations prematurely. +* cgroup_mutex must nest outside scx_fork_rwsem because cgroup file +* operations may create kthreads while holding cgroup_mutex. */ +scx_cgroup_lock ();percpu_down_write (&scx_fork_rwsem);WARN_ON_ONCE (scx_init_task_enabled);
@@ -7216,7 +7219,6 @@ static void scx_root_enable_workfn(struct kthread_work *work)* while tasks are being initialized so that scx_cgroup_can_attach() * never sees uninitialized tasks. */ -scx_cgroup_lock ();set_cgroup_sched (sch_cgroup (sch), sch); ret =scx_cgroup_init (sch);if (ret)
@@ -7283,8 +7285,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)put_task_struct (p); }scx_task_iter_stop (&sti); -scx_cgroup_unlock ();percpu_up_write (&scx_fork_rwsem); +scx_cgroup_unlock ();/* * All tasks are READY. It's safe to turn on scx_enabled() and switch
@@ -7369,8 +7371,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)return ; err_disable_unlock_all: -scx_cgroup_unlock ();percpu_up_write (&scx_fork_rwsem); +scx_cgroup_unlock ();/* we'll soon enter disable path, keep bypass on */ err_disable:mutex_unlock (&scx_enable_mutex);
Signature · 1 line
2.43.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] The patch fixes an ABBA deadlock between cgroup_mutex and scx_fork_rwsem in scx_root_enable_workfn(), but leaves the exact same lock inversion unpatched in scx_root_disable(), scx_sub_disable(), and scx_sub_enable_workfn(). --
--- Patch [1]: [PATCH] sched_ext: Fix deadlock with PSI trigger creation --- commit aacffeaa1e10b05888e4970e698ee8d1863dbbbd
Author: Matt Fleming <[email protected]>
sched_ext: Fix deadlock with PSI trigger creation
This commit fixes an ABBA deadlock between cgroup_mutex and scx_fork_rwsem in scx_root_enable_workfn(). It prevents deadlocks that can occur when concurrent writes to cgroup PSI trigger files attempt to create kthreads.
kernel/sched/ext/ext.c static void scx_root_enable_workfn(struct kthread_work *work) · line 7200
Quoted 13 lines
kernel/sched/ext/ext.c +3
-0@@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)/* * Lock out forks, cgroup on/offlining and moves before opening the * floodgate so that they don't wander into the operations prematurely. +* cgroup_mutex must nest outside scx_fork_rwsem because cgroup file +* operations may create kthreads while holding cgroup_mutex. */ +scx_cgroup_lock ();percpu_down_write (&scx_fork_rwsem);
[Severity: High]
This is a pre-existing issue, but does this same lock inversion still exist in other scheduler functions?
For example, scx_sub_disable() takes scx_fork_rwsem before scx_cgroup_lock():
kernel/sched/ext/ext.c:scx_sub_disable() {
...
percpu_down_write(&scx_fork_rwsem);
scx_cgroup_lock();
...
}scx_root_disable() also takes the locks in the old order:
kernel/sched/ext/ext.c:scx_root_disable() {
...
percpu_down_write(&scx_fork_rwsem);
...
scx_cgroup_lock();
...
}And scx_sub_enable_workfn() appears to do the same:
kernel/sched/ext/ext.c:scx_sub_enable_workfn() {
...
percpu_down_write(&scx_fork_rwsem);
scx_cgroup_lock();
...
}Could a concurrent PSI trigger write still cause a deadlock if it races with these other paths?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Andrea Righi
reviewerHi Matt,
Quoted 44 lines · Matt Fleming
From: Matt Fleming <[email protected]>
scx_root_enable_workfn()currently takesscx_fork_rwsemfor writing before acquiringcgroup_mutex. Since commita5b98009f16d("sched/psi:
fix race between file release and pressure write"),pressure_write()holdscgroup_mutexacrosspsi_trigger_create(), which may callkthread_create()for the psimon kthread. kthreadd's fork then entersscx_pre_fork()and waits for the read side ofscx_fork_rwsem.This results in a deadlock. The enable worker holds
scx_fork_rwsemand waits forcgroup_mutex, while the PSI writer holdscgroup_mutexand waits for psimon creation to complete. Any concurrent fork blocks onscx_pre_fork()behind the enable worker.The hung-task detector captured all three sides of the deadlock:
scx_enable_help:
__mutex_lock
scx_enable_workfn
kthread_worker_fnsystemd:
wait_for_completion_killable
__kthread_create_on_node
kthread_create_on_node
psi_trigger_create
pressure_write
kernfs_fop_write_iterpython3:
percpu_rwsem_wait
__percpu_down_read
scx_pre_fork
sched_fork
copy_process
kernel_cloneIt also identified systemd as the likely owner of the mutex on which
scx_enable_helpwas blocked.We reproduced this on a 128-CPU AMD EPYC 7713 by enabling
scx_lavdconcurrently with writes to cgroup PSI trigger files. Unrelated tasks piled up inscx_pre_fork()and process creation on the box stopped.
Quoted 7 lines
Fix the inversion by acquiring
cgroup_mutexbeforescx_fork_rwseminscx_root_enable_workfn()and releasing them in reverse order, while preserving the existing exclusion around cgroup and task initialisation.Fixes a5b98009f16d ("sched/psi: fix race between file release and pressure write")Signed-off-by Matt Fleming <[email protected]>
This seems to introduce the following (running the sched_ext kselftests):
[ 28.963575] ======================================================
[ 28.963670] WARNING: possible circular locking dependency detected
[ 28.963752] 7.1.0-virtme #1 Not tainted
[ 28.963804] ------------------------------------------------------
[ 28.963887] sched_ext_helpe/2619 is trying to acquire lock:
[ 28.963954] ffffffff83685240 (scx_cgroup_ops_rwsem){+.+.}-{0:0}, at: scx_root_disable+0x45d/0x840
[ 28.964071]
[ 28.964071] but task is already holding lock:
[ 28.964151] ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[ 28.964260]
[ 28.964260] which lock already depends on the new lock.
[ 28.964260]
[ 28.964355]
[ 28.964355] the existing dependency chain (in reverse order) is:
[ 28.964455]
[ 28.964455] -> #2 (scx_fork_rwsem){++++}-{0:0}:
[ 28.964539] percpu_down_write+0x49/0x150
[ 28.964610] scx_root_enable_workfn+0x5d0/0xd30
[ 28.964679] kthread_worker_fn+0x121/0x360
[ 28.964749] kthread+0x10c/0x140
[ 28.964802] ret_from_fork+0x189/0x330
[ 28.964872] ret_from_fork_asm+0x1a/0x30
[ 28.964941]
[ 28.964941] -> #1 (cgroup_mutex){+.+.}-{4:4}:
[ 28.965025] __mutex_lock+0xbe/0xd80
[ 28.965070] scx_root_enable_workfn+0x5c4/0xd30
[ 28.965138] kthread_worker_fn+0x121/0x360
[ 28.965210] kthread+0x10c/0x140
[ 28.965263] ret_from_fork+0x189/0x330
[ 28.965331] ret_from_fork_asm+0x1a/0x30
[ 28.965402]
[ 28.965402] -> #0 (scx_cgroup_ops_rwsem){+.+.}-{0:0}:
[ 28.965485] __lock_acquire+0x14c5/0x2a40
[ 28.965554] lock_acquire+0xd3/0x280
[ 28.965608] percpu_down_write+0x49/0x150
[ 28.965677] scx_root_disable+0x45d/0x840
[ 28.965745] kthread_worker_fn+0x121/0x360
[ 28.965815] kthread+0x10c/0x140
[ 28.965867] ret_from_fork+0x189/0x330
[ 28.965935] ret_from_fork_asm+0x1a/0x30
[ 28.966002]
[ 28.966002] other info that might help us debug this:
[ 28.966002]
[ 28.966097] Chain exists of:
[ 28.966097] scx_cgroup_ops_rwsem --> cgroup_mutex --> scx_fork_rwsem
[ 28.966097]
[ 28.966233] Possible unsafe locking scenario:
[ 28.966233]
[ 28.966314] CPU0 CPU1
[ 28.966379] ---- ----
[ 28.966444] lock(scx_fork_rwsem);
[ 28.966496] lock(cgroup_mutex);
[ 28.966579] lock(scx_fork_rwsem);
[ 28.966661] lock(scx_cgroup_ops_rwsem);
[ 28.966713]
[ 28.966713] *** DEADLOCK ***
[ 28.966713]
[ 28.966794] 2 locks held by sched_ext_helpe/2619:
[ 28.966861] #0: ffffffff83682818 (scx_enable_mutex){+.+.}-{4:4}, at: scx_root_disable+0xba/0x840
[ 28.966974] #1: ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[ 28.967090]
[ 28.967090] stack backtrace:
[ 28.967158] CPU: 8 UID: 0 PID: 2619 Comm: sched_ext_helpe Not tainted 7.1.0-virtme #1 PREEMPT(full)
[ 28.967164] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 28.967166] Call Trace:
[ 28.967169] <TASK>
[ 28.967172] dump_stack_lvl+0x6d/0xa0
[ 28.967175] print_circular_bug+0x2e1/0x300
[ 28.967179] check_noncircular+0x144/0x170
[ 28.967182] __lock_acquire+0x14c5/0x2a40
[ 28.967202] ? scx_root_disable+0x45d/0x840
[ 28.967204] lock_acquire+0xd3/0x280
[ 28.967207] ? scx_root_disable+0x45d/0x840
[ 28.967209] percpu_down_write+0x49/0x150
[ 28.967212] ? scx_root_disable+0x45d/0x840
[ 28.967213] scx_root_disable+0x45d/0x840
[ 28.967219] ? kthread_worker_fn+0x51/0x360
[ 28.967221] kthread_worker_fn+0x121/0x360
[ 28.967223] ? __pfx_scx_disable_workfn+0x10/0x10
[ 28.967224] ? __pfx_kthread_worker_fn+0x10/0x10
[ 28.967227] kthread+0x10c/0x140
[ 28.967228] ? __pfx_kthread+0x10/0x10
[ 28.967229] ret_from_fork+0x189/0x330
[ 28.967231] ? __pfx_kthread+0x10/0x10
[ 28.967232] ret_from_fork_asm+0x1a/0x30
[ 28.967235] </TASK>Thanks,
-Andrea
Quoted 50 lines
--- kernel/sched/ext/ext.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)kernel/sched/ext/ext.c +5
-3@@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)/* * Lock out forks, cgroup on/offlining and moves before opening the * floodgate so that they don't wander into the operations prematurely. +* cgroup_mutex must nest outside scx_fork_rwsem because cgroup file +* operations may create kthreads while holding cgroup_mutex. */ +scx_cgroup_lock ();percpu_down_write (&scx_fork_rwsem);WARN_ON_ONCE (scx_init_task_enabled);@@ -7216,7 +7219,6 @@ static void scx_root_enable_workfn(struct kthread_work *work)* while tasks are being initialized so that scx_cgroup_can_attach() * never sees uninitialized tasks. */ -scx_cgroup_lock ();set_cgroup_sched (sch_cgroup (sch), sch); ret =scx_cgroup_init (sch);if (ret)@@ -7283,8 +7285,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)put_task_struct (p); }scx_task_iter_stop (&sti); -scx_cgroup_unlock ();percpu_up_write (&scx_fork_rwsem); +scx_cgroup_unlock ();/* * All tasks are READY. It's safe to turn on scx_enabled() and switch @@ -7369,8 +7371,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)return ; err_disable_unlock_all: -scx_cgroup_unlock ();percpu_up_write (&scx_fork_rwsem); +scx_cgroup_unlock ();/* we'll soon enter disable path, keep bypass on */ err_disable:mutex_unlock (&scx_enable_mutex);Signature · 1 line
2.43.0
Andrea Righi
reviewerQuoted 57 lines · Andrea Righi
Hi Matt,
Quoted 44 lines · Matt Fleming
From: Matt Fleming <[email protected]>
scx_root_enable_workfn()currently takesscx_fork_rwsemfor writing before acquiringcgroup_mutex. Since commita5b98009f16d("sched/psi:
fix race between file release and pressure write"),pressure_write()holdscgroup_mutexacrosspsi_trigger_create(), which may callkthread_create()for the psimon kthread. kthreadd's fork then entersscx_pre_fork()and waits for the read side ofscx_fork_rwsem.This results in a deadlock. The enable worker holds
scx_fork_rwsemand waits forcgroup_mutex, while the PSI writer holdscgroup_mutexand waits for psimon creation to complete. Any concurrent fork blocks onscx_pre_fork()behind the enable worker.The hung-task detector captured all three sides of the deadlock:
scx_enable_help:
__mutex_lock
scx_enable_workfn
kthread_worker_fnsystemd:
wait_for_completion_killable
__kthread_create_on_node
kthread_create_on_node
psi_trigger_create
pressure_write
kernfs_fop_write_iterpython3:
percpu_rwsem_wait
__percpu_down_read
scx_pre_fork
sched_fork
copy_process
kernel_cloneIt also identified systemd as the likely owner of the mutex on which
scx_enable_helpwas blocked.We reproduced this on a 128-CPU AMD EPYC 7713 by enabling
scx_lavdconcurrently with writes to cgroup PSI trigger files. Unrelated tasks piled up inscx_pre_fork()and process creation on the box stopped.Quoted 7 lines
Fix the inversion by acquiring
cgroup_mutexbeforescx_fork_rwseminscx_root_enable_workfn()and releasing them in reverse order, while preserving the existing exclusion around cgroup and task initialisation.Fixes a5b98009f16d ("sched/psi: fix race between file release and pressure write")Signed-off-by Matt Fleming <[email protected]>This seems to introduce the following (running the
sched_extkselftests):
Quoted 11 lines
[ 28.963575] ======================================================
[ 28.963670] WARNING: possible circular locking dependency detected
[ 28.963752] 7.1.0-virtme #1 Not tainted
[ 28.963804] ------------------------------------------------------
[ 28.963887] sched_ext_helpe/2619 is trying to acquire lock:
[ 28.963954] ffffffff83685240 (scx_cgroup_ops_rwsem){+.+.}-{0:0}, at: scx_root_disable+0x45d/0x840
[ 28.964071]
[ 28.964071] but task is already holding lock:
[ 28.964151] ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840
[ 28.964260]
[ 28.964260] which lock already depends on the new lock.Looking more at this, I think we should fix the PSI path instead, sched_ext locking order is consistently using:
scx_fork_rwsem -> scx_cgroup_ops_rwsem -> cgroup_mutex
Maybe we can rework pressure_write() to pin the cgroup and file context, then drop both cgroup_mutex and kernfs active protection before creating the PSI kthread?
Thanks,
-Andrea
Quoted 129 lines
[ 28.964260] [ 28.964355] [ 28.964355] the existing dependency chain (in reverse order) is: [ 28.964455] [ 28.964455] -> #2 (scx_fork_rwsem){++++}-{0:0}: [ 28.964539] percpu_down_write+0x49/0x150 [ 28.964610] scx_root_enable_workfn+0x5d0/0xd30 [ 28.964679] kthread_worker_fn+0x121/0x360 [ 28.964749] kthread+0x10c/0x140 [ 28.964802] ret_from_fork+0x189/0x330 [ 28.964872] ret_from_fork_asm+0x1a/0x30 [ 28.964941] [ 28.964941] -> #1 (cgroup_mutex){+.+.}-{4:4}: [ 28.965025] __mutex_lock+0xbe/0xd80 [ 28.965070] scx_root_enable_workfn+0x5c4/0xd30 [ 28.965138] kthread_worker_fn+0x121/0x360 [ 28.965210] kthread+0x10c/0x140 [ 28.965263] ret_from_fork+0x189/0x330 [ 28.965331] ret_from_fork_asm+0x1a/0x30 [ 28.965402] [ 28.965402] -> #0 (scx_cgroup_ops_rwsem){+.+.}-{0:0}: [ 28.965485] __lock_acquire+0x14c5/0x2a40 [ 28.965554] lock_acquire+0xd3/0x280 [ 28.965608] percpu_down_write+0x49/0x150 [ 28.965677] scx_root_disable+0x45d/0x840 [ 28.965745] kthread_worker_fn+0x121/0x360 [ 28.965815] kthread+0x10c/0x140 [ 28.965867] ret_from_fork+0x189/0x330 [ 28.965935] ret_from_fork_asm+0x1a/0x30 [ 28.966002] [ 28.966002] other info that might help us debug this: [ 28.966002] [ 28.966097] Chain exists of: [ 28.966097] scx_cgroup_ops_rwsem --> cgroup_mutex --> scx_fork_rwsem [ 28.966097] [ 28.966233] Possible unsafe locking scenario: [ 28.966233] [ 28.966314] CPU0 CPU1 [ 28.966379] ---- ---- [ 28.966444] lock(scx_fork_rwsem); [ 28.966496] lock(cgroup_mutex); [ 28.966579] lock(scx_fork_rwsem); [ 28.966661] lock(scx_cgroup_ops_rwsem); [ 28.966713] [ 28.966713] *** DEADLOCK *** [ 28.966713] [ 28.966794] 2 locks held by sched_ext_helpe/2619: [ 28.966861] #0: ffffffff83682818 (scx_enable_mutex){+.+.}-{4:4}, at: scx_root_disable+0xba/0x840 [ 28.966974] #1: ffffffff83682910 (scx_fork_rwsem){++++}-{0:0}, at: scx_root_disable+0x160/0x840 [ 28.967090] [ 28.967090] stack backtrace: [ 28.967158] CPU: 8 UID: 0 PID: 2619 Comm: sched_ext_helpe Not tainted 7.1.0-virtme #1 PREEMPT(full) [ 28.967164] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 28.967166] Call Trace: [ 28.967169] <TASK> [ 28.967172] dump_stack_lvl+0x6d/0xa0 [ 28.967175] print_circular_bug+0x2e1/0x300 [ 28.967179] check_noncircular+0x144/0x170 [ 28.967182] __lock_acquire+0x14c5/0x2a40 [ 28.967202] ? scx_root_disable+0x45d/0x840 [ 28.967204] lock_acquire+0xd3/0x280 [ 28.967207] ? scx_root_disable+0x45d/0x840 [ 28.967209] percpu_down_write+0x49/0x150 [ 28.967212] ? scx_root_disable+0x45d/0x840 [ 28.967213] scx_root_disable+0x45d/0x840 [ 28.967219] ? kthread_worker_fn+0x51/0x360 [ 28.967221] kthread_worker_fn+0x121/0x360 [ 28.967223] ? __pfx_scx_disable_workfn+0x10/0x10 [ 28.967224] ? __pfx_kthread_worker_fn+0x10/0x10 [ 28.967227] kthread+0x10c/0x140 [ 28.967228] ? __pfx_kthread+0x10/0x10 [ 28.967229] ret_from_fork+0x189/0x330 [ 28.967231] ? __pfx_kthread+0x10/0x10 [ 28.967232] ret_from_fork_asm+0x1a/0x30 [ 28.967235] </TASK>Thanks,
-AndreaQuoted 50 lines
--- kernel/sched/ext/ext.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)kernel/sched/ext/ext.c +5
-3@@ -7193,7 +7193,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)/* * Lock out forks, cgroup on/offlining and moves before opening the * floodgate so that they don't wander into the operations prematurely. +* cgroup_mutex must nest outside scx_fork_rwsem because cgroup file +* operations may create kthreads while holding cgroup_mutex. */ +scx_cgroup_lock ();percpu_down_write (&scx_fork_rwsem);WARN_ON_ONCE (scx_init_task_enabled);@@ -7216,7 +7219,6 @@ static void scx_root_enable_workfn(struct kthread_work *work)* while tasks are being initialized so that scx_cgroup_can_attach() * never sees uninitialized tasks. */ -scx_cgroup_lock ();set_cgroup_sched (sch_cgroup (sch), sch); ret =scx_cgroup_init (sch);if (ret)@@ -7283,8 +7285,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)put_task_struct (p); }scx_task_iter_stop (&sti); -scx_cgroup_unlock ();percpu_up_write (&scx_fork_rwsem); +scx_cgroup_unlock ();/* * All tasks are READY. It's safe to turn on scx_enabled() and switch @@ -7369,8 +7371,8 @@ static void scx_root_enable_workfn(struct kthread_work *work)return ; err_disable_unlock_all: -scx_cgroup_unlock ();percpu_up_write (&scx_fork_rwsem); +scx_cgroup_unlock ();/* we'll soon enter disable path, keep bypass on */ err_disable:mutex_unlock (&scx_enable_mutex);Signature · 1 line
2.43.0
Tejun Heo
maintainerPATCH 1/1 cgroup: Create the psimon kthread outside of cgroup_mutex
In reply to Matt Fleminga5b98009f16d ("sched/psi: fix race between file release and pressure write") made pressure_write() hold cgroup_mutex across psi_trigger_create(), which forks the psimon kthread for the first rtpoll trigger. As kthread creation depends on the whole fork path, the commit inadvertently created a lot of unwanted locking dependencies from cgroup_mutex.
sched_ext got hit by one: its enable path blocks forks and then grabs cgroup_mutex, so a pressure write racing a scheduler enable deadlocks, with every other fork piling up behind.
Fix it by splitting trigger creation so that the worker is forked with cgroup_mutex dropped and the kernfs active reference left broken. The latter matters because rmdir and cgroup.pressure writes drain active references under cgroup_mutex. Publishing the trigger last keeps error reporting synchronous and preserves the of->priv lifetime rules.
The trigger registered in the first stage pins the group's rtpoll machinery across the unlocked window, leaving only creation races to resolve. The catch-up poll on installation covers scheduling attempts dropped while there was no worker.
---
Matt, your reordering trades one deadlock for another: CLONE_INTO_CGROUP forks grab cgroup_mutex inside the scx_fork_rwsem read section, so an enable racing such a clone deadlocks the other way around. The fork has to move out of the locked sections instead. Can you verify this fixes the deadlock in your setup?
include/linux/psi.h | 4 ++ kernel/cgroup/cgroup.c | 23 +++++++++++++++- kernel/sched/psi.c | 69 +++++++++++++++++++++++++++++++++++++------------ 3 files changed, 78 insertions(+), 18 deletions(-)
include/linux/psi.h +3 -1
@@ -25,7 +25,9 @@ void psi_memstall_leave(unsigned long *fint psi_show (struct seq_file *s,struct psi_group *group,enum psi_res res);struct psi_trigger *psi_trigger_create (struct psi_group *group,char *buf,enum psi_res res,struct file *file, -struct kernfs_open_file *of); +struct kernfs_open_file *of, +bool *need_rtpoll_worker); +int psi_trigger_create_rtpoll_worker (struct psi_group *group);void psi_trigger_destroy (struct psi_trigger *t); __poll_t psi_trigger_poll(void **trigger_ptr, struct file *file,
kernel/cgroup/cgroup.c +22 -1
@@ -3996,6 +3996,7 @@ static ssize_t pressure_write(struct kerstruct psi_trigger *new;struct cgroup *cgrp;struct psi_group *psi; +bool need_rtpoll_worker;ssize_t ret =0 ; cgrp =cgroup_kn_lock_live (of->kn, false);
@@ -4015,12 +4016,32 @@ static ssize_t pressure_write(struct ker } psi =cgroup_psi (cgrp); - new =psi_trigger_create (psi, buf, res, of->file, of); + new =psi_trigger_create (psi, buf, res, of->file, of, + &need_rtpoll_worker);if (IS_ERR (new)) { ret =PTR_ERR (new); goto out_unlock; } +/* +* The worker fork must run with neither cgroup_mutex nor the file's +* kernfs active reference held. The latter is broken since +* cgroup_kn_lock_live(). @of->priv may be released while unlocked, so +* recheck before publishing @new. +*/ +if (need_rtpoll_worker) { +cgroup_unlock (); + ret =psi_trigger_create_rtpoll_worker (psi); +cgroup_lock (); + +if (!ret && !of->priv) + ret = -ENODEV ; +if (ret) { +psi_trigger_destroy (new); + goto out_unlock; + } + } +smp_store_release (&ctx->psi.trigger, new); out_unlock:
kernel/sched/psi.c +53 -16
112 lines not on this page · raw message
[email protected]
AI reviewRe: [PATCH cgroup/for-7.2-fixes] cgroup: Create the psimon kthread outside of cgroup_mutex
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but the psi rtpoll_timer can be re-armed after it is deleted and subsequently execute after the psi_group is freed, causing a Use-After-Free. --
--- Patch [1]: [PATCH cgroup/for-7.2-fixes] cgroup: Create the psimon kthread outside of cgroup_mutex --- commit 8790bcf0b23cb2132be9400b975c7f18fd1147b7
Author: Tejun Heo <[email protected]>
cgroup: Create the psimon kthread outside of cgroup_mutex
This commit splits the trigger creation process so the rtpoll worker thread is forked without holding cgroup_mutex. This resolves a deadlock between cgroup_mutex and the fork path during sched_ext enablement.
Quoted 4 lines
kernel/sched/psi.c +0
-0
[ ... ]
Quoted 6 lines
kernel/sched/psi.c +1
-1@@ -1565,12 +1592,22 @@ static ssize_t psi_write(struct file *file, const char __user *user_buf,return -EBUSY ; } - new =psi_trigger_create (&psi_system, buf, res, file,NULL ); + new = psi_trigger_create(&psi_system, buf, res, file, NULL,
kernel/sched/psi.c static ssize_t psi_write(struct file *file, const char __user *user_buf, · line 1605
Quoted 10 lines
kernel/sched/psi.c +5
-0+ &need_rtpoll_worker);if (IS_ERR (new)) {mutex_unlock (&seq->lock);return PTR_ERR (new); } +if (need_rtpoll_worker) { + ret =psi_trigger_create_rtpoll_worker (&psi_system); +if (ret) { +psi_trigger_destroy (new);
[Severity: High]
This isn't a bug introduced by this patch, but can the psi rtpoll_timer be re-armed after it is deleted and subsequently execute after the psi_group is freed, causing a use-after-free?
In psi_trigger_destroy(), the timer is deleted before synchronize_rcu():
psi_trigger_destroy() {
rcu_assign_pointer(group->rtpoll_task, NULL);
timer_delete(&group->rtpoll_timer);
...
synchronize_rcu();
}If Thread A calls psi_schedule_rtpoll_work() and gets delayed or preempted after reading rtpoll_task inside the RCU read-side critical section:
psi_schedule_rtpoll_work() {
rcu_read_lock();
task = rcu_dereference(group->rtpoll_task);
...
// Thread A delayed here
}Thread B could then run psi_trigger_destroy(), delete the timer, and block in synchronize_rcu() waiting for Thread A. When Thread A resumes, it sees the task is non-NULL and calls mod_timer():
if (likely(task))
mod_timer(&group->rtpoll_timer, jiffies + delay);This re-arms the timer after it was already deleted by Thread B.
Later, when the cgroup is removed and psi_cgroup_free() is called, the psi_group is freed without deleting this leaked timer. When the timer finally expires, poll_timer_fn() will attempt to access the freed group memory.
Quoted 8 lines
kernel/sched/psi.c +5
-0+mutex_unlock (&seq->lock); +return ret; + } + } +smp_store_release (&seq->private, new);mutex_unlock (&seq->lock);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH cgroup/for-7.2-fixes] cgroup: Create the psimon kthread outside of cgroup_mutex
In reply to Tejun HeoQuoted 5 lines · Tejun Heo
Matt, your reordering trades one deadlock for another:
CLONE_INTO_CGROUPforks grabcgroup_mutexinside thescx_fork_rwsemread section, so an enable racing such a clone deadlocks the other way around. The fork has to move out of the locked sections instead. Can you verify this fixes the deadlock in your setup?
Thanks for fixing this. I'll test this out ASAP and report back.
Tejun Heo
maintainerRe: [PATCH cgroup/for-7.2-fixes] cgroup: Create the psimon kthread outside of cgroup_mutex
In reply to Tejun HeoHello,
Superseded by the v2 posting, now a two patch series with an additional rtpoll_timer UAF fix:
https://lore.kernel.org/r/[email protected]
Thanks.
--
tejun