sched_ext: sync disable_irq_work in bpf_scx_unreg()
Subsystems (2)
PATCH 1/1 sched_ext: sync disable_irq_work in bpf_scx_unreg()
When unregistered my self-written scx scheduler, the following panic occurs [1].
The root cause is that the JIT page backing ops->quiescent() is freed before all callers of that function have stopped.
The expected ordering during teardown is:
bitmap_zero(sch->has_op) + synchronize_rcu()
-> guarantees no CPU will ever call sch->ops.* again
-> only THEN free the BPF struct_ops JIT pagebpf_scx_unreg() is supposed to enforce the order, but after commit f4a6c506d118 ("sched_ext: Always bounce scx_disable() through irq_work"), disable_work is no longer queued directly, causing kthread_flush_work() to be a noop. Thus, the caller drops the struct_ops map too early and poisoned with AARCH64_BREAK_FAULT before disable_workfn ever execute.
So the subsequent dequeue_task() still sees SCX_HAS_OP(sch, quiescent) as true and calls ops.quiescent, which hit on the poisoned page and BRK panic.
Fix it by syncing disable_irq_work first, so disable_work is guaranteed to be queued before waiting for it.
---
[1]:
[ 188.572805] sched_ext: BPF scheduler "invariant_0.1.0_aarch64_unknown_linux_gnu_debug" enabled [ 229.923133] Kernel text patching generated an invalid instruction at 0xffff80009bc2c1f8! [ 229.923146] Internal error: Oops - BRK: 00000000f2000100 [#1] SMP [ 230.077871] CPU: 48 UID: 0 PID: 1760 Comm: kworker/u583:7 Not tainted 7.0.0+ #3 PREEMPT(full) [ 230.086677] Hardware name: NVIDIA GB200 NVL/P3809-BMC, BIOS 02.05.12 20251107 [ 230.093972] Workqueue: events_unbound bpf_map_free_deferred [ 230.099675] Sched_ext: invariant_0.1.0_aarch64_unknown_linux_gnu_debug (disabling), task: runnable_at=-174ms [ 230.116843] pc : 0xffff80009bc2c1f8 [ 230.120406] lr : dequeue_task_scx+0x270/0x2d0 [ 230.217749] Call trace: [ 230.228515] 0xffff80009bc2c1f8 (P) [ 230.232077] dequeue_task+0x84/0x188 [ 230.235728] sched_change_begin+0x1dc/0x250 [ 230.240000] __set_cpus_allowed_ptr_locked+0x17c/0x240 [ 230.245250] __set_cpus_allowed_ptr+0x74/0xf0 [ 230.249701] ___migrate_enable+0x4c/0xa0 [ 230.253707] bpf_map_free_deferred+0x1a4/0x1b0 [ 230.258246] process_one_work+0x184/0x540 [ 230.262342] worker_thread+0x19c/0x348 [ 230.266170] kthread+0x13c/0x150 [ 230.269465] ret_from_fork+0x10/0x20 [ 230.281393] Code: d4202000 d4202000 d4202000 d4202000 (d4202000) [ 230.287621] ---[ end trace 0000000000000000 ]--- [ 231.160046] Kernel panic - not syncing: Oops - BRK: Fatal exception in interrupt
Best regards,
Richard Cheng.
--- kernel/sched/ext.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sched/ext.c +6 -0
@@ -7349,6 +7349,12 @@ static void bpf_scx_unreg(void *kdata, struct bpf_link *link)struct scx_sched *sch =rcu_dereference_protected (ops->priv, true);scx_disable (sch,SCX_EXIT_UNREG ); +/* +* sch->disable_work might still not queued, causing kthread_flush_work() +* as a noop. Syncing the irq_work first is required to guarantee the +* kthread work has been queued before waiting for it. +*/ +irq_work_sync (&sch->disable_irq_work);kthread_flush_work (&sch->disable_work);RCU_INIT_POINTER (ops->priv,NULL );kobject_put (&sch->kobj);
Signature · 1 line
2.43.0
Hi Richard,
Quoted 65 lines · Richard Cheng
When unregistered my self-written scx scheduler, the following panic occurs [1].
The root cause is that the JIT page backing
ops->quiescent()is freed before all callers of that function have stopped.The expected ordering during teardown is:
bitmap_zero(sch->has_op) + synchronize_rcu() -> guarantees no CPU will ever call sch->ops.* again -> only THEN free the BPF struct_ops JIT page
bpf_scx_unreg()is supposed to enforce the order, but after commitf4a6c506d118("sched_ext: Always bouncescx_disable()throughirq_work"),disable_workis no longer queued directly, causingkthread_flush_work()to be a noop. Thus, the caller drops thestruct_opsmap too early and poisoned withAARCH64_BREAK_FAULTbeforedisable_workfnever execute.So the subsequent
dequeue_task()still seesSCX_HAS_OP(sch, quiescent)as true and calls ops.quiescent, which hit on the poisoned page and BRK panic.Fix it by syncing
disable_irq_workfirst, sodisable_workis guaranteed to be queued before waiting for it.Fixes f4a6c506d118 ("sched_ext: Always bounce scx_disable() through irq_work")Signed-off-by Richard Cheng <[email protected]>---
[1]:[ 188.572805] sched_ext: BPF scheduler "invariant_0.1.0_aarch64_unknown_linux_gnu_debug" enabled [ 229.923133] Kernel text patching generated an invalid instruction at 0xffff80009bc2c1f8! [ 229.923146] Internal error: Oops - BRK: 00000000f2000100 [#1] SMP [ 230.077871] CPU: 48 UID: 0 PID: 1760 Comm: kworker/u583:7 Not tainted 7.0.0+ #3 PREEMPT(full) [ 230.086677] Hardware name: NVIDIA GB200 NVL/P3809-BMC, BIOS 02.05.12 20251107 [ 230.093972] Workqueue: events_unbound bpf_map_free_deferred [ 230.099675] Sched_ext: invariant_0.1.0_aarch64_unknown_linux_gnu_debug (disabling), task: runnable_at=-174ms [ 230.116843] pc : 0xffff80009bc2c1f8 [ 230.120406] lr : dequeue_task_scx+0x270/0x2d0 [ 230.217749] Call trace: [ 230.228515] 0xffff80009bc2c1f8 (P) [ 230.232077] dequeue_task+0x84/0x188 [ 230.235728] sched_change_begin+0x1dc/0x250 [ 230.240000] __set_cpus_allowed_ptr_locked+0x17c/0x240 [ 230.245250] __set_cpus_allowed_ptr+0x74/0xf0 [ 230.249701] ___migrate_enable+0x4c/0xa0 [ 230.253707] bpf_map_free_deferred+0x1a4/0x1b0 [ 230.258246] process_one_work+0x184/0x540 [ 230.262342] worker_thread+0x19c/0x348 [ 230.266170] kthread+0x13c/0x150 [ 230.269465] ret_from_fork+0x10/0x20 [ 230.281393] Code: d4202000 d4202000 d4202000 d4202000 (d4202000) [ 230.287621] ---[ end trace 0000000000000000 ]--- [ 231.160046] Kernel panic - not syncing: Oops - BRK: Fatal exception in interruptBest regards,
Richard Cheng.--- kernel/sched/ext.c | 6 ++++++ 1 file changed, 6 insertions(+)kernel/sched/ext.c +0
-0
kernel/sched/ext.c static void bpf_scx_unreg(void *kdata, struct bpf_link *link) · line 7354
Quoted 6 lines
kernel/sched/ext.c +3
-0@@ -7349,6 +7349,12 @@ static void bpf_scx_unreg(void *kdata, struct bpf_link *link)struct scx_sched *sch =rcu_dereference_protected (ops->priv, true);scx_disable (sch,SCX_EXIT_UNREG ); +/* +* sch->disable_work might still not queued, causing kthread_flush_work() +* as a noop. Syncing the irq_work first is required to guarantee the
nit, maybe rephrase: sch->disable_work might not have been queued yet, causing kthread_flush_work() to be a no-op.
Quoted 9 lines
kernel/sched/ext.c +3
-0+* kthread work has been queued before waiting for it. +*/ +irq_work_sync (&sch->disable_irq_work);kthread_flush_work (&sch->disable_work);RCU_INIT_POINTER (ops->priv,NULL );kobject_put (&sch->kobj);Signature · 1 line
2.43.0
I can't reproduce it locally, but from a logical perspective it makes sense. Nice catch!
Thanks,
-Andrea
Hi Richard,
Quoted 2 lines · Richard Cheng
When unregistered my self-written scx scheduler, the following panic occurs [1].
Nit: you've placed the panic log [1] below the --- separator. Content below this line will not be preserved in the comment msg. Could you please move it into the commit msg body and send a v2 patch?
Quoted 20 lines
The root cause is that the JIT page backing
ops->quiescent()is freed before all callers of that function have stopped.The expected ordering during teardown is:
bitmap_zero(sch->has_op) + synchronize_rcu() -> guarantees no CPU will ever call sch->ops.* again -> only THEN free the BPF struct_ops JIT page
bpf_scx_unreg()is supposed to enforce the order, but after commitf4a6c506d118("sched_ext: Always bouncescx_disable()throughirq_work"),disable_workis no longer queued directly, causingkthread_flush_work()to be a noop. Thus, the caller drops thestruct_opsmap too early and poisoned withAARCH64_BREAK_FAULTbeforedisable_workfnever execute.So the subsequent
dequeue_task()still seesSCX_HAS_OP(sch, quiescent)as true and calls ops.quiescent, which hit on the poisoned page and BRK panic.
Quoted 5 lines
Fix it by syncing
disable_irq_workfirst, sodisable_workis guaranteed to be queued before waiting for it.Fixes f4a6c506d118 ("sched_ext: Always bounce scx_disable() through irq_work")Signed-off-by Richard Cheng <[email protected]>
Thanks for the fix, and the logic looks correct to me.
Also, scx_root_enable_workfn() has the same pattern in its error path:
scx_error(sch, "scx_root_enable() failed (%d)", ret); kthread_flush_work(&sch->disable_work);
The comment above indicates that this flush is meant to "ensure that error is reported before init completion". However, because scx_error() goes through scx_vexit() -> irq_work_queue(), the flush can be a no-op here as well. The same applies to the sub-scheduler enable error path.
Should those be fixed in the same patch? Tejun, Andrea, wdyt? Thanks
[...]
kernel/sched/ext.c static void bpf_scx_unreg(void *kdata, struct bpf_link *link) · line 7354
Quoted 11 lines
kernel/sched/ext.c +3
-0@@ -7349,6 +7349,12 @@ static void bpf_scx_unreg(void *kdata, struct bpf_link *link)struct scx_sched *sch =rcu_dereference_protected (ops->priv, true);scx_disable (sch,SCX_EXIT_UNREG ); +/* +* sch->disable_work might still not queued, causing kthread_flush_work() +* as a noop. Syncing the irq_work first is required to guarantee the
Perhaps s/noop/no-op/? Though it's just a matter of taste. ^_^
Quoted 10 lines
kernel/sched/ext.c +3
-0+* kthread work has been queued before waiting for it. +*/ +irq_work_sync (&sch->disable_irq_work);kthread_flush_work (&sch->disable_work);RCU_INIT_POINTER (ops->priv,NULL );kobject_put (&sch->kobj);Signature · 1 line
2.43.0
Signature · 2 lines
Cheers, Cheng-Yang
Tejun Heo
reviewerHello,
Quoted 4 lines · Cheng-Yang Chou
Also,
scx_root_enable_workfn()has the same pattern in its error path:scx_error(sch, "scx_root_enable() failed (%d)", ret); kthread_flush_work(&sch->disable_work);
Yeah, can you please add a helper - e.g. flush_disable_work() - to package sync and flush and use that in both places?
Quoted 5 lines
kernel/sched/ext.c static void bpf_scx_unreg(void *kdata, struct bpf_link *link) · line 7354
Quoted 3 lines
kernel/sched/ext.c +3
-0+/* +* sch->disable_work might still not queued, causing kthread_flush_work() +* as a noop. Syncing the irq_work first is required to guarantee the Perhaps s/noop/no-op/? Though it's just a matter of taste. ^_^
noop is used widely in the kernel. In general, I don't think we need this level of language policing.
Thanks.
Signature · 1 line
tejun
Quoted 10 lines · Cheng-Yang Chou
Hi Richard,
Quoted 2 lines · Richard Cheng
When unregistered my self-written scx scheduler, the following panic occurs [1].
Nit: you've placed the panic log [1] below the --- separator. Content below this line will not be preserved in the comment msg. Could you please move it into the commit msg body and send a v2 patch?
Hi Cheng-Yang,
Sure thing, thanks.
Quoted 35 lines
Quoted 20 lines
The root cause is that the JIT page backing
ops->quiescent()is freed before all callers of that function have stopped.The expected ordering during teardown is:
bitmap_zero(sch->has_op) + synchronize_rcu() -> guarantees no CPU will ever call sch->ops.* again -> only THEN free the BPF struct_ops JIT page
bpf_scx_unreg()is supposed to enforce the order, but after commitf4a6c506d118("sched_ext: Always bouncescx_disable()throughirq_work"),disable_workis no longer queued directly, causingkthread_flush_work()to be a noop. Thus, the caller drops thestruct_opsmap too early and poisoned withAARCH64_BREAK_FAULTbeforedisable_workfnever execute.So the subsequent
dequeue_task()still seesSCX_HAS_OP(sch, quiescent)as true and calls ops.quiescent, which hit on the poisoned page and BRK panic.Quoted 5 lines
Fix it by syncing
disable_irq_workfirst, sodisable_workis guaranteed to be queued before waiting for it.Fixes f4a6c506d118 ("sched_ext: Always bounce scx_disable() through irq_work")Signed-off-by Richard Cheng <[email protected]>Thanks for the fix, and the logic looks correct to me.
Reviewed-by Cheng-Yang Chou <[email protected]>Also,
scx_root_enable_workfn()has the same pattern in its error path:scx_error(sch, "scx_root_enable() failed (%d)", ret); kthread_flush_work(&sch->disable_work);
Quoted 7 lines
The comment above indicates that this flush is meant to "ensure that error is reported before init completion". However, because
scx_error()goes throughscx_vexit()->irq_work_queue(), the flush can be a no-op here as well. The same applies to the sub-scheduler enable error path.Should those be fixed in the same patch? Tejun, Andrea, wdyt? Thanks
Sure I'll amend that part in v2.
Quoted 15 lines
[...]
kernel/sched/ext.c static void bpf_scx_unreg(void *kdata, struct bpf_link *link) · line 7354
Quoted 11 lines
kernel/sched/ext.c +3
-0@@ -7349,6 +7349,12 @@ static void bpf_scx_unreg(void *kdata, struct bpf_link *link)struct scx_sched *sch =rcu_dereference_protected (ops->priv, true);scx_disable (sch,SCX_EXIT_UNREG ); +/* +* sch->disable_work might still not queued, causing kthread_flush_work() +* as a noop. Syncing the irq_work first is required to guarantee the Perhaps s/noop/no-op/? Though it's just a matter of taste. ^_^
Quoted 14 lines
Quoted 10 lines
kernel/sched/ext.c +3
-0+* kthread work has been queued before waiting for it. +*/ +irq_work_sync (&sch->disable_irq_work);kthread_flush_work (&sch->disable_work);RCU_INIT_POINTER (ops->priv,NULL );kobject_put (&sch->kobj);Signature · 1 line
2.43.0Signature · 2 lines
Cheers, Cheng-Yang
- Richard
Quoted 10 lines · Tejun Heo
Hello,
Quoted 4 lines · Cheng-Yang Chou
Also,
scx_root_enable_workfn()has the same pattern in its error path:scx_error(sch, "scx_root_enable() failed (%d)", ret); kthread_flush_work(&sch->disable_work);Yeah, can you please add a helper - e.g.
flush_disable_work()- to package sync and flush and use that in both places?
Hello Tejun,
No problem I'll add this part in v2.
Quoted 7 lines
Quoted 6 lines
kernel/sched/ext.c static void bpf_scx_unreg(void *kdata, struct bpf_link *link) · line 7354
Quoted 4 lines
kernel/sched/ext.c +3
-0+/* +* sch->disable_work might still not queued, causing kthread_flush_work() +* as a noop. Syncing the irq_work first is required to guarantee the Perhaps s/noop/no-op/? Though it's just a matter of taste. ^_^
Quoted 7 lines
noop is used widely in the kernel. In general, I don't think we need this level of language policing.
Thanks.
Signature · 1 line
tejun
Thanks.
- Richard