WARNING at kernel/sched/ext.c:3227 scx_cgroup_move_task+0xa8/0xb0
Subsystems (2)
Hi,
We're hitting a WARN_ON_ONCE in scx_cgroup_move_task() while running scx_lavd. The stack is:
scx_cgroup_move_task+0xa8/0xb0
sched_move_task+0x134/0x290
cpu_cgroup_attach+0x39/0x70
cgroup_migrate_execute+0x37d/0x450
cgroup_update_dfl_csses+0x1e3/0x270
cgroup_subtree_control_write+0x3e7/0x440The trigger is systemd's [email protected] startup. The user manager writes +cpu +memory +pids to its own subtree_control. Because +cpu is already inherited, the cpu task_group doesn't change -- but find_css_set() returns a css_set whose subsys[cpu] pointer differs from the source (different css object, same tg_cgrp()). That sets the cpu bit in mgctx->ss_mask, so both cpu_cgroup_can_attach() and cpu_cgroup_attach() run.
scx_cgroup_can_attach() checks "from == to" in tg_cgrp() and skips the task, leaving cgrp_moving_from NULL. But cpu_cgroup_attach() calls sched_move_task() unconditionally, so scx_cgroup_move_task() fires and WARNs on the NULL.
I managed to reproduce by cycling systemctl stop/start [email protected]. The hit rate varies with css_set cache state.
Any thoughts on the best way to resolve this?
Thanks,
Matt
Tejun Heo
reviewerPATCH 1/1 sched_ext: Don't warn on NULL cgrp_moving_from in scx_cgroup_move_task()
A WARN fires when systemd's user manager writes "+cpu +memory +pids" to its own subtree_control while a sched_ext scheduler is loaded:
WARNING: at kernel/sched/ext.c:3227 scx_cgroup_move_task+0xa8/0xb0 scx_cgroup_move_task+0xa8/0xb0 sched_move_task+0x134/0x290 cpu_cgroup_attach+0x39/0x70 cgroup_migrate_execute+0x37d/0x450 cgroup_update_dfl_csses+0x1e3/0x270 cgroup_subtree_control_write+0x3e7/0x440
scx_cgroup_can_attach() arms cgrp_moving_from only when a task's cpu cgroup changes. It can still be NULL when scx_cgroup_move_task() runs, through this sequence:
Step Result --------------------------------- ---------------------------------- 1. cpu enabled on cgroup G cpu css = A 2. cpu toggled off then on for G A killed, B created (same cgroup) 3. an exiting task keeps A alive migration skips it, A now stale 4. +memory migrates G stale A vs current B pulls cpu in 5. cpu attach runs for all tasks hits a live, cpu-unchanged task 6. scx_cgroup_move_task() on it cgrp_moving_from NULL -> WARN
The mismatch is that scx_cgroup_can_attach() keys on cgroup identity while migration drives the move on css identity, so a NULL cgrp_moving_from here is a legitimate css-only migration, not a missing prep.
The call is already gated on cgrp_moving_from, so just drop the warning. ops.cgroup_prep_move() and ops.cgroup_move() stay paired.
--- kernel/sched/ext.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/sched/ext.c +6 -4
@@ -4293,11 +4293,13 @@ void scx_cgroup_move_task(struct task_struct *p)return ;/* -* @p must have ops.cgroup_prep_move() called on it and thus -* cgrp_moving_from set. +* scx_cgroup_can_attach() sets cgrp_moving_from only when the task's +* cgroup changes. Migration keys off css rather than cgroup identity, +* so it can hand an unchanged-cgroup task here with cgrp_moving_from +* NULL. Nothing to report to the BPF scheduler then, so skip it and +* keep prep_move and move paired. */ -if (SCX_HAS_OP (sch, cgroup_move) && - !WARN_ON_ONCE (!p->scx.cgrp_moving_from)) +if (SCX_HAS_OP (sch, cgroup_move) && p->scx.cgrp_moving_from)SCX_CALL_OP_TASK (sch, cgroup_move,task_rq (p), p, p->scx.cgrp_moving_from,tg_cgrp (task_group (p)));
Re: [PATCH sched_ext/for-7.1-fixes] sched_ext: Don't warn on NULL cgrp_moving_from in scx_cgroup_move_task()
Hello,
Quoted 28 lines · Tejun Heo
A WARN fires when systemd's user manager writes "+cpu +memory +pids" to its own
subtree_controlwhile asched_extscheduler is loaded:WARNING: at kernel/sched/ext.c:3227 scx_cgroup_move_task+0xa8/0xb0 scx_cgroup_move_task+0xa8/0xb0 sched_move_task+0x134/0x290 cpu_cgroup_attach+0x39/0x70 cgroup_migrate_execute+0x37d/0x450 cgroup_update_dfl_csses+0x1e3/0x270 cgroup_subtree_control_write+0x3e7/0x440
scx_cgroup_can_attach()armscgrp_moving_fromonly when a task's cpu cgroup changes. It can still be NULL whenscx_cgroup_move_task()runs, through this sequence:Step Result --------------------------------- ---------------------------------- 1. cpu enabled on cgroup G cpu css = A 2. cpu toggled off then on for G A killed, B created (same cgroup) 3. an exiting task keeps A alive migration skips it, A now stale 4. +memory migrates G stale A vs current B pulls cpu in 5. cpu attach runs for all tasks hits a live, cpu-unchanged task 6. scx_cgroup_move_task() on it cgrp_moving_from NULL -> WARNThe mismatch is that
scx_cgroup_can_attach()keys on cgroup identity while migration drives the move on css identity, so a NULLcgrp_moving_fromhere is a legitimate css-only migration, not a missing prep.
Quoted 8 lines
The call is already gated on
cgrp_moving_from, so just drop the warning.ops.cgroup_prep_move()andops.cgroup_move()stay paired.Fixes 819513666966 ("sched_ext: Add cgroup support")Cc [email protected] # v6.12+Reported-by Matt Fleming <[email protected]>Signed-off-by Tejun Heo <[email protected]>
Makes sense to me.
Thanks,
-Andrea
Quoted 26 lines
--- kernel/sched/ext.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)kernel/sched/ext.c +6
-4@@ -4293,11 +4293,13 @@ void scx_cgroup_move_task(struct task_struct *p)return ;/* -* @p must have ops.cgroup_prep_move() called on it and thus -* cgrp_moving_from set. +* scx_cgroup_can_attach() sets cgrp_moving_from only when the task's +* cgroup changes. Migration keys off css rather than cgroup identity, +* so it can hand an unchanged-cgroup task here with cgrp_moving_from +* NULL. Nothing to report to the BPF scheduler then, so skip it and +* keep prep_move and move paired. */ -if (SCX_HAS_OP (sch, cgroup_move) && - !WARN_ON_ONCE (!p->scx.cgrp_moving_from)) +if (SCX_HAS_OP (sch, cgroup_move) && p->scx.cgrp_moving_from)SCX_CALL_OP_TASK (sch, cgroup_move,task_rq (p), p, p->scx.cgrp_moving_from,tg_cgrp (task_group (p)));
Re: [PATCH sched_ext/for-7.1-fixes] sched_ext: Don't warn on NULL cgrp_moving_from in scx_cgroup_move_task()
In reply to Tejun HeoApplied to sched_ext/for-7.1-fixes.
Thanks.
--
tejun