sched_ext: Cgroup migration and op delivery for sub-schedulers
Subsystems (3)
Tejun Heo
maintainerPATCH 1/1 sched_ext: Cgroup migration and op delivery for sub-schedulers
Hello,
A task's sched must match its cgroup's: each sub-scheduler serves the cgroup2 subtree it is attached to and the root scheduler serves the rest. Cgroup migration currently breaks this. A task moved across a sub-scheduler boundary keeps its old sched, leading to wrong-sched scheduling and, once the stale sched is freed, a use-after-free. Cgroup ops are also always delivered to the root scheduler no matter which sched serves the cgroup.
This patchset makes both follow the sub-scheduler topology:
- cgroup gains a task migration notifier (0001). sched_ext uses it to re-home tasks whose migration crosses a sched boundary: the destination sched runs the fallible ops.init_task() before the migration commits, so a rejection fails the cgroup.procs write (0002-0004).
- cgroup_init/exit and the knob ops are delivered to the sched each task_group is on. The move ops go to the task's sched, only for migrations that don't re-home it (0005).
- Sub-scheduler enable claims the subtree's cgroups from the parent and disable returns them. A parent that fails to re-init a returned cgroup fails in turn and the cgroups move up the chain, ending at the root scheduler (0006).
- scx_qmap consumes cgroup weights through ops.cgroup_set_weight() and gains fault injection modes to exercise the failure paths (0007-0008).
Verified with a trace-based suite on the demo scheduler covering op delivery and routing, handover failures, multi-level punt chains, cgroup1 and stress.
Based on sched_ext/for-7.3 (7c2cd767705d).
Tejun Heo (8):
cgroup: Add cgroup_task_notifier and task migration events
sched_ext: Factor out scx_rehome_task() and scx_punt_task()
sched_ext: Relocate scx_cgroup_enabled
sched_ext: Re-home tasks on cgroup migration
sched_ext: Deliver cgroup ops to each task_group's sched
sched_ext: Hand over cgroups at sub-scheduler enable/disable tools/sched_ext: scx_qmap - Consume cgroup weights through set_weight tools/sched_ext: scx_qmap - Add init fault injection modes
Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-cgroup-migration
include/linux/cgroup.h | 26 +++ include/linux/sched/ext.h | 15 ++ kernel/cgroup/cgroup.c | 93 ++++++++- kernel/sched/ext/ext.c | 247 +++++++++++++++++------- kernel/sched/ext/internal.h | 44 ++++- kernel/sched/ext/sub.c | 418 +++++++++++++++++++++++++++++++++++++---- tools/sched_ext/scx_qmap.bpf.c | 95 ++++++++-- tools/sched_ext/scx_qmap.c | 110 +---------- tools/sched_ext/scx_qmap.h | 5 +- 9 files changed, 819 insertions(+), 234 deletions(-)
--
tejun
Tejun Heo
maintainerPATCH 1/8 cgroup: Add cgroup_task_notifier and task migration events
A subsystem can attach to the cgroup hierarchy itself, independent of which controllers are enabled where - BPF hooks already behave this way and sched_ext sub-schedulers do too. Controller callbacks can't track task migrations for them: sched_ext must re-home a task whose migration crosses a sub-scheduler boundary, but the cpu controller's attach callbacks fire only when the task_group changes and miss moves whenever the controller topology is coarser than the sub-scheduler topology.
Add cgroup_task_notifier with per-task migration events mirroring the can_attach/attach/cancel_attach phases so that a consumer which prepares per-task state can also veto a migration: CGROUP_TASK_MIGRATING fires pre-commit, CGROUP_TASK_MIGRATED post-commit and CGROUP_TASK_MIGRATE_CANCELED unwinds a failed migration. Only migrations that change a task's dfl cgroup are reported.
--- include/linux/cgroup.h | 26 ++++++++++++ kernel/cgroup/cgroup.c | 93 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 110 insertions(+), 9 deletions(-)
include/linux/cgroup.h +26 -0
@@ -82,12 +82,38 @@ enum cgroup_lifetime_events { CGROUP_LIFETIME_OFFLINE, }; +/* +* Events on cgroup_task_notifier, data is struct cgroup_task_migrate_ctx. +* MIGRATING fires per task before the migration commits and an error return +* from the chain fails the migration, in which case tasks that were already +* notified receive MIGRATE_CANCELED. MIGRATED fires per task after the +* migration is committed and can't fail. Only migrations that change a task's +* dfl cgroup are reported. +*/ +enum cgroup_task_events { +CGROUP_TASK_MIGRATING , +CGROUP_TASK_MIGRATED , +CGROUP_TASK_MIGRATE_CANCELED , +}; + +/* +* @src_dcgrp and @dst_dcgrp are @task's dfl cgroups before and after the +* migration. @src_dcgrp is NULL for CGROUP_TASK_MIGRATED as per-task sources +* are not tracked past the commit point. +*/ +struct cgroup_task_migrate_ctx { +struct task_struct *task; +struct cgroup *src_dcgrp; +struct cgroup *dst_dcgrp; +}; +extern struct file_system_type cgroup_fs_type;extern struct cgroup_root cgrp_dfl_root;extern struct css_set init_css_set;extern struct mutex cgroup_mutex;extern spinlock_t css_set_lock;extern struct blocking_notifier_head cgroup_lifetime_notifier; +extern struct blocking_notifier_head cgroup_task_notifier;#define SUBSYS (_x) extern struct cgroup_subsys _x ## _cgrp_subsys;#include <linux/cgroup_subsys.h>
kernel/cgroup/cgroup.c +84 -9
136 lines not on this page · raw message
Signature · 1 line
2.55.0
Tejun Heo
maintainerPATCH 2/8 sched_ext: Factor out scx_rehome_task() and scx_punt_task()
In reply to Tejun HeoFactor out scx_rehome_task() and scx_punt_task() from the sub-disable re-home loop and scx_fail_parent(). The upcoming cgroup migration re-homing also needs scx_rehome_task(). No functional changes.
--- kernel/sched/ext/sub.c | 84 ++++++++++++++++++++++++++++-------------- 1 file changed, 56 insertions(+), 28 deletions(-)
kernel/sched/ext/sub.c +56 -28
@@ -758,6 +758,57 @@ void drain_descendants(struct scx_sched *sch)wait_event (scx_unlink_waitq,list_empty (&sch->children)); } +/** +* scx_rehome_task - Move a task to a sched it has been initialized for +* @to: sched taking over @p, @p's init on it already complete +* @p: task to re-home +* +* Exit @p from its current sched and switch it over to @to, overriding the +* state to %SCX_TASK_READY to account for the already completed init. A task +* on a non-ext class, possible under an %SCX_OPS_SWITCH_PARTIAL root, stays +* %READY and is enabled by switching_to_scx() if it switches over. +*/ +static void scx_rehome_task (struct scx_sched *to,struct task_struct *p) +{ +lockdep_assert_held (&p->pi_lock); +lockdep_assert_rq_held (task_rq (p)); + + scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_disable_and_exit_task (scx_task_sched (p), p); +scx_set_task_state (p,SCX_TASK_INIT_BEGIN ); +scx_set_task_state (p,SCX_TASK_INIT ); +scx_set_task_sched (p, to); +scx_set_task_state (p,SCX_TASK_READY ); +if (p->sched_class == &ext_sched_class) +scx_enable_task (to, p); + } +} + +/** +* scx_punt_task - Hand a task to a failed sched without initialization +* @to: failed and bypassed sched taking custody of @p +* @p: task to punt +* +* Take @p off its current sched and put it on @to at %SCX_TASK_NONE. @to is +* dying and its teardown will re-home @p properly. +* +* Used when @to must take over @p but failed to initialize it. Bypass keeps +* scheduling decisions away from @to but @p can still trigger its task ops, +* which may confuse the BPF side. @to is dying anyway. The exit paths skip +* %NONE tasks (see __scx_disable_and_exit_task() and switched_from_scx()). +*/ +static void scx_punt_task (struct scx_sched *to,struct task_struct *p) +{ +lockdep_assert_held (&p->pi_lock); +lockdep_assert_rq_held (task_rq (p)); +WARN_ON_ONCE (!READ_ONCE (to->bypass_depth)); + + scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { +scx_disable_and_exit_task (scx_task_sched (p), p); +scx_set_task_sched (p, to); + } +} +static void scx_fail_parent (struct scx_sched *sch,struct task_struct *failed,s32 fail_code) {
@@ -769,9 +820,9 @@ static void scx_fail_parent(struct scx_sched *sch, fail_code, failed->comm, failed->pid);/* -* Once $parent is bypassed, it's safe to put SCX_TASK_NONE tasks into -* it. This may cause downstream failures on the BPF side but $parent is -* dying anyway. +* Once $parent is bypassed, tasks can be punted into it. This may +* cause downstream failures on the BPF side but $parent is dying +* anyway. */ scx_bypass (parent, true);
@@ -780,10 +831,7 @@ static void scx_fail_parent(struct scx_sched *sch,if (scx_task_on_sched (parent, p))continue ; - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { -scx_disable_and_exit_task (sch, p); -scx_set_task_sched (p, parent); - } +scx_punt_task (parent, p); }scx_task_iter_stop (&sti); }
@@ -881,27 +929,7 @@ void scx_sub_disable(struct scx_sched *sch)continue ; } - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) { -/* -* $p is initialized for $parent and still attached to -* @sch. Disable and exit for @sch, switch over to -* $parent and override the state to READY to account -* for $p having already been initialized. -*/ -scx_disable_and_exit_task (sch, p); -scx_set_task_state (p,SCX_TASK_INIT_BEGIN ); -scx_set_task_state (p,SCX_TASK_INIT ); -scx_set_task_sched (p, parent); -scx_set_task_state (p,SCX_TASK_READY ); - -/* -* A task on a non-ext class, possible under an -* %SCX_OPS_SWITCH_PARTIAL root, stays READY and is -* enabled by switching_to_scx() if it switches over. -*/ -if (p->sched_class == &ext_sched_class) -scx_enable_task (parent, p); - } +scx_rehome_task (parent, p);task_rq_unlock (rq, p, &rf);put_task_struct (p);
Signature · 1 line
2.55.0
Tejun Heo
maintainerPATCH 3/8 sched_ext: Relocate scx_cgroup_enabled
In reply to Tejun Heoscx_cgroup_enabled is in the CONFIG_EXT_GROUP_SCHED block. The upcoming cgroup migration re-homing needs the gate outside the block. Move the definition and flag flips outside CONFIG_EXT_GROUP_SCHED. No functional changes.
--- kernel/sched/ext/ext.c | 18 ++++++++++++------ kernel/sched/ext/internal.h | 1 + 2 files changed, 13 insertions(+), 6 deletions(-)
kernel/sched/ext/ext.c +12 -6
@@ -80,6 +80,14 @@ static bool scx_switching_all;DEFINE_STATIC_KEY_FALSE (__scx_switched_all);static DEFINE_STATIC_KEY_FALSE (__scx_tid_to_task_enabled); +/* +* Gates cgroup ops delivery. Set at the end of the cgroup init phase of root +* enable and cleared before root disable starts tearing down tasks, both under +* scx_cgroup_lock(). Holding cgroup_lock() and seeing %true guarantees no race +* against root tearing down tasks. +*/ +bool scx_cgroup_enabled; +/* * True once SCX_OPS_TID_TO_TASK has been negotiated with the root scheduler * and the tid->task table is live. Wraps the static key so callers don't
@@ -4309,7 +4317,6 @@ bool scx_can_stop_tick(struct rq *rq)#ifdef CONFIG_EXT_GROUP_SCHED DEFINE_STATIC_PERCPU_RWSEM (scx_cgroup_ops_rwsem); -static bool scx_cgroup_enabled;void scx_tg_init (struct task_group *tg) {
@@ -4689,8 +4696,6 @@ static void scx_cgroup_exit(struct scx_sched *sch) {struct cgroup_subsys_state *css; - scx_cgroup_enabled = false; -/* * scx_tg_on/offline() are excluded through cgroup_lock(). If we walk * cgroups and exit all the inited ones, all online cgroups are exited.
@@ -4745,9 +4750,6 @@ static int scx_cgroup_init(struct scx_sched *sch) tg->scx.flags |=SCX_TG_INITED ; } -WARN_ON_ONCE (scx_cgroup_enabled); - scx_cgroup_enabled = true; -return 0 ; }
@@ -5971,6 +5973,7 @@ static void scx_root_disable(struct scx_sched *sch)* doesn't race against scx_disable_and_exit_task(). */ scx_cgroup_lock (); + scx_cgroup_enabled = false;scx_cgroup_exit (sch);scx_cgroup_unlock ();
@@ -7234,6 +7237,9 @@ static void scx_root_enable_workfn(struct kthread_work *work)if (ret) goto err_disable_unlock_all; +WARN_ON_ONCE (scx_cgroup_enabled); + scx_cgroup_enabled = true; +scx_task_iter_start (&sti,NULL );while ((p =scx_task_iter_next_locked (&sti))) {/*
kernel/sched/ext/internal.h +1 -0
@@ -1949,6 +1949,7 @@ __printf(3, 0) s32 scx_bstr_format(struct scx_sched *sch, struct scx_bstr_buf *bextern raw_spinlock_t scx_sched_lock;extern struct mutex scx_enable_mutex;extern struct percpu_rw_semaphore scx_fork_rwsem; +extern bool scx_cgroup_enabled;extern raw_spinlock_t scx_exit_bstr_buf_lock;extern struct scx_bstr_buf scx_exit_bstr_buf;#ifdef CONFIG_EXT_SUB_SCHED
Signature · 1 line
2.55.0
Tejun Heo
maintainerPATCH 4/8 sched_ext: Re-home tasks on cgroup migration
In reply to Tejun HeoA task's sched (p->scx.sched) must match its cgroup's owner (cgrp->scx_sched). cgroup migration breaks the invariant:scx_cgroup_move_task() only fires root's ops.cgroup_move() and never re-homes the task, leading to wrong-sched scheduling and, once the stale sched is freed, a use-after-free.
Hook into the new cgroup task migration events and re-home each task whose destination cgroup is owned by a different sched. The events map naturally to the transfer: MIGRATING runs the fallible init for the destination sched, letting it reject the migration the same way ops.cgroup_prep_move() can, MIGRATED does the re-home, which can't fail, and CANCELED undoes the init when the migration falls through.
Pre-commit, the task's task_group still reflects the source, so __scx_init_task() grows an explicit cgroup argument for the migration path to hand ops.init_task() the destination cgroup.
--- kernel/sched/ext/ext.c | 29 +++++--- kernel/sched/ext/internal.h | 3 +- kernel/sched/ext/sub.c | 135 ++++++++++++++++++++++++++++++++++-- 3 files changed, 152 insertions(+), 15 deletions(-)
kernel/sched/ext/ext.c +21 -8
@@ -3485,15 +3485,28 @@ static struct cgroup *tg_cgrp(struct task_group *tg) return &cgrp_dfl_root.cgrp; } -#define SCX_INIT_TASK_ARGS_CGROUP (tg) .cgroup = tg_cgrp(tg), +#define SCX_INIT_TASK_ARGS_CGROUP (cgrp) .cgroup = (cgrp),#else /* CONFIG_EXT_GROUP_SCHED */ -#define SCX_INIT_TASK_ARGS_CGROUP (tg) +#define SCX_INIT_TASK_ARGS_CGROUP (cgrp)#endif /* CONFIG_EXT_GROUP_SCHED */ -int __scx_init_task (struct scx_sched *sch,struct task_struct *p,bool fork) +/** +* __scx_init_task - Initialize a task for a sched +* @sch: sched to initialize @p for +* @p: task of interest +* @cgrp: cgroup @p is joining, %NULL for @p's current task_group's cgroup +* @fork: %true if @p is being forked +* +* Pre-commit cgroup migration passes @cgrp explicitly as @p's task_group +* still reflects the source. +* +* Return 0 on success, -errno on failure. +*/ +int __scx_init_task (struct scx_sched *sch,struct task_struct *p, +struct cgroup *cgrp,bool fork) {int ret;
@@ -3501,7 +3514,7 @@ int __scx_init_task(struct scx_sched *sch, struct task_struct *p, bool fork)if (SCX_HAS_OP (sch, init_task)) {struct scx_init_task_args args = { -SCX_INIT_TASK_ARGS_CGROUP (task_group (p)) +SCX_INIT_TASK_ARGS_CGROUP (cgrp ?:tg_cgrp (task_group (p))) .fork = fork, };
@@ -3747,7 +3760,7 @@ int scx_fork(struct task_struct *p, struct kernel_clone_args *kargs)struct scx_sched *sch = scx_root;#endif scx_set_task_state (p,SCX_TASK_INIT_BEGIN ); - ret =__scx_init_task (sch, p, true); + ret =__scx_init_task (sch, p,NULL , true);if (unlikely (ret)) {scx_set_task_state (p,SCX_TASK_NONE );return ret;
@@ -5969,8 +5982,8 @@ static void scx_root_disable(struct scx_sched *sch)WRITE_ONCE (scx_switching_all, false);/* -* Shut down cgroup support before tasks so that the cgroup attach path -* doesn't race against scx_disable_and_exit_task(). +* Shut down cgroup support before tasks so that the cgroup attach and +* migration paths don't race against scx_disable_and_exit_task(). */ scx_cgroup_lock (); scx_cgroup_enabled = false;
@@ -7263,7 +7276,7 @@ static void scx_root_enable_workfn(struct kthread_work *work)scx_set_task_state (p,SCX_TASK_INIT_BEGIN );scx_task_iter_unlock (&sti); - ret =__scx_init_task (sch, p, false); + ret =__scx_init_task (sch, p,NULL , false);scx_task_iter_relock (&sti, p);
kernel/sched/ext/internal.h +2 -1
@@ -1920,7 +1920,8 @@ void scx_flush_dispatch_buf(struct scx_sched *sch, struct rq *rq);void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags);void schedule_dsq_reenq (struct scx_sched *sch,struct scx_dispatch_q *dsq,u64 reenq_flags,struct rq *locked_rq); -int __scx_init_task (struct scx_sched *sch,struct task_struct *p,bool fork); +int __scx_init_task (struct scx_sched *sch,struct task_struct *p, +struct cgroup *cgrp,bool fork);void scx_enable_task (struct scx_sched *sch,struct task_struct *p);void __scx_disable_and_exit_task (struct scx_sched *sch,struct task_struct *p);void scx_sub_init_cancel_task (struct scx_sched *sch,struct task_struct *p);
kernel/sched/ext/sub.c +129 -6
161 lines not on this page · raw message
Signature · 1 line
2.55.0
Andrea Righi
reviewerRe: [PATCH 4/8] sched_ext: Re-home tasks on cgroup migration
Hi Tejun,
Quoted 13 lines · Tejun Heo
A task's sched (
p->scx.sched) must match its cgroup's owner (cgrp->scx_sched). cgroup migration breaks the invariant:scx_cgroup_move_task()only fires root'sops.cgroup_move()and never re-homes the task, leading to wrong-sched scheduling and, once the stale sched is freed, a use-after-free.Hook into the new cgroup task migration events and re-home each task whose destination cgroup is owned by a different sched. The events map naturally to the transfer: MIGRATING runs the fallible init for the destination sched, letting it reject the migration the same way
ops.cgroup_prep_move()can, MIGRATED does the re-home, which can't fail, and CANCELED undoes the init when the migration falls through.
Quoted 5 lines
Pre-commit, the task's
task_groupstill reflects the source, so__scx_init_task()grows an explicit cgroup argument for the migration path to handops.init_task()the destination cgroup.Signed-off-by Tejun Heo <[email protected]>
Should we add:
Thanks,
-Andrea
Quoted 266 lines
--- kernel/sched/ext/ext.c | 29 +++++--- kernel/sched/ext/internal.h | 3 +- kernel/sched/ext/sub.c | 135 ++++++++++++++++++++++++++++++++++-- 3 files changed, 152 insertions(+), 15 deletions(-)kernel/sched/ext/ext.c +21
-8@@ -3485,15 +3485,28 @@ static struct cgroup *tg_cgrp(struct task_group *tg) return &cgrp_dfl_root.cgrp; } -#define SCX_INIT_TASK_ARGS_CGROUP (tg) .cgroup = tg_cgrp(tg), +#define SCX_INIT_TASK_ARGS_CGROUP (cgrp) .cgroup = (cgrp),#else /* CONFIG_EXT_GROUP_SCHED */ -#define SCX_INIT_TASK_ARGS_CGROUP (tg) +#define SCX_INIT_TASK_ARGS_CGROUP (cgrp)#endif /* CONFIG_EXT_GROUP_SCHED */ -int __scx_init_task (struct scx_sched *sch,struct task_struct *p,bool fork) +/** +* __scx_init_task - Initialize a task for a sched +* @sch: sched to initialize @p for +* @p: task of interest +* @cgrp: cgroup @p is joining, %NULL for @p's current task_group's cgroup +* @fork: %true if @p is being forked +* +* Pre-commit cgroup migration passes @cgrp explicitly as @p's task_group +* still reflects the source. +* +* Return 0 on success, -errno on failure. +*/ +int __scx_init_task (struct scx_sched *sch,struct task_struct *p, +struct cgroup *cgrp,bool fork) {int ret;@@ -3501,7 +3514,7 @@ int __scx_init_task(struct scx_sched *sch, struct task_struct *p, bool fork)if (SCX_HAS_OP (sch, init_task)) {struct scx_init_task_args args = { -SCX_INIT_TASK_ARGS_CGROUP (task_group (p)) +SCX_INIT_TASK_ARGS_CGROUP (cgrp ?:tg_cgrp (task_group (p))) .fork = fork, };@@ -3747,7 +3760,7 @@ int scx_fork(struct task_struct *p, struct kernel_clone_args *kargs)struct scx_sched *sch = scx_root;#endif scx_set_task_state (p,SCX_TASK_INIT_BEGIN ); - ret =__scx_init_task (sch, p, true); + ret =__scx_init_task (sch, p,NULL , true);if (unlikely (ret)) {scx_set_task_state (p,SCX_TASK_NONE );return ret;@@ -5969,8 +5982,8 @@ static void scx_root_disable(struct scx_sched *sch)WRITE_ONCE (scx_switching_all, false);/* -* Shut down cgroup support before tasks so that the cgroup attach path -* doesn't race against scx_disable_and_exit_task(). +* Shut down cgroup support before tasks so that the cgroup attach and +* migration paths don't race against scx_disable_and_exit_task(). */ scx_cgroup_lock (); scx_cgroup_enabled = false;@@ -7263,7 +7276,7 @@ static void scx_root_enable_workfn(struct kthread_work *work)scx_set_task_state (p,SCX_TASK_INIT_BEGIN );scx_task_iter_unlock (&sti); - ret =__scx_init_task (sch, p, false); + ret =__scx_init_task (sch, p,NULL , false);scx_task_iter_relock (&sti, p);kernel/sched/ext/internal.h +2
-1@@ -1920,7 +1920,8 @@ void scx_flush_dispatch_buf(struct scx_sched *sch, struct rq *rq);void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags);void schedule_dsq_reenq (struct scx_sched *sch,struct scx_dispatch_q *dsq,u64 reenq_flags,struct rq *locked_rq); -int __scx_init_task (struct scx_sched *sch,struct task_struct *p,bool fork); +int __scx_init_task (struct scx_sched *sch,struct task_struct *p, +struct cgroup *cgrp,bool fork);void scx_enable_task (struct scx_sched *sch,struct task_struct *p);void __scx_disable_and_exit_task (struct scx_sched *sch,struct task_struct *p);void scx_sub_init_cancel_task (struct scx_sched *sch,struct task_struct *p);kernel/sched/ext/sub.c +129
-6@@ -906,7 +906,7 @@ void scx_sub_disable(struct scx_sched *sch)* parent. A child can't directly affect the parent through its * own failures. */ - ret =__scx_init_task (parent, p, false); + ret =__scx_init_task (parent, p,NULL , false);if (ret) {scx_fail_parent (sch, p, ret);put_task_struct (p);@@ -1212,7 +1212,7 @@ void scx_sub_enable_workfn(struct kthread_work *work)* As $p is still on $parent, it can't be transitioned to INIT. * Let's worry about task state later. Use __scx_init_task(). */ - ret =__scx_init_task (sch, p, false); + ret =__scx_init_task (sch, p,NULL , false);if (ret) goto abort;@@ -1336,6 +1336,99 @@ void scx_sub_enable_workfn(struct kthread_work *work) cmd->ret =0 ; } +/** +* scx_cgroup_task_migrating - Prepare a task for a cgroup migration +* @ctx: migration being prepared +* +* A task's sched must match its cgroup's owner, so a migration that crosses a +* sched boundary re-homes the task once committed. Run the fallible part here, +* before the migration commits: initialize the task for the destination sched. +* A rejection fails the cgroup.procs write. +*/ +static s32 scx_cgroup_task_migrating (struct cgroup_task_migrate_ctx *ctx) +{ +struct task_struct *p = ctx->task; +struct scx_sched *to; +int ret; + +/* +* Cleared under scx_cgroup_lock() before root disable starts tearing +* down tasks. As cgroup_mutex is held, a set flag guarantees that the +* teardown loop is not running concurrently. +*/ +if (!scx_cgroup_enabled) +return NOTIFY_OK ; + + to = ctx->dst_dcgrp->scx_sched; +if (scx_task_on_sched (to, p)) +return NOTIFY_OK ; + + ret =__scx_init_task (to, p, ctx->dst_dcgrp, false); +if (ret) +return notifier_from_errno (ret); + +return NOTIFY_OK ; +} + +/** +* scx_cgroup_task_migrated - Re-home a task that changed cgroups +* @ctx: committed migration +* +* Move the task to its new cgroup's sched, which scx_cgroup_task_migrating() +* already initialized it for. Can't fail. +* +* This is safe against all phases of the destination sched's destruction. A +* disable resets cgroup ownership to the parent and re-homes tasks in one +* scx_cgroup_lock() section. If that section already ran, the destination would +* be the parent. Otherwise, the re-home loop is still ahead and guaranteed to +* visit the task, now in the destination cgroup. +*/ +static void scx_cgroup_task_migrated (struct cgroup_task_migrate_ctx *ctx) +{ +struct task_struct *p = ctx->task; +struct scx_sched *to; +struct rq *rq; +struct rq_flags rf; + +if (!scx_cgroup_enabled) +return ; + + to = ctx->dst_dcgrp->scx_sched; +if (scx_task_on_sched (to, p)) +return ; + + rq =task_rq_lock (p, &rf); +scx_rehome_task (to, p); +task_rq_unlock (rq, p, &rf); +} + +/** +* scx_cgroup_task_migrate_canceled - Undo migration preparation +* @ctx: canceled migration +* +* The migration failed after scx_cgroup_task_migrating() initialized the task +* for the destination sched. The task stays on its current sched in the source +* cgroup. Undo the destination's init. +*/ +static void scx_cgroup_task_migrate_canceled (struct cgroup_task_migrate_ctx *ctx) +{ +struct task_struct *p = ctx->task; +struct scx_sched *to; +struct rq *rq; +struct rq_flags rf; + +if (!scx_cgroup_enabled) +return ; + + to = ctx->dst_dcgrp->scx_sched; +if (scx_task_on_sched (to, p)) +return ; + + rq =task_rq_lock (p, &rf); +scx_sub_init_cancel_task (to, p); +task_rq_unlock (rq, p, &rf); +} +static s32 scx_cgroup_lifetime_notify (struct notifier_block *nb,unsigned long action,void *data) {@@ -1367,12 +1460,42 @@ static struct notifier_block scx_cgroup_lifetime_nb = { .notifier_call = scx_cgroup_lifetime_notify, }; -static s32 __init scx_cgroup_lifetime_notifier_init (void ) +static s32 scx_cgroup_task_notify (struct notifier_block *nb, +unsigned long action,void *data) +{ +struct cgroup_task_migrate_ctx *ctx = data; + +switch (action) { +case CGROUP_TASK_MIGRATING : +return scx_cgroup_task_migrating (ctx); +case CGROUP_TASK_MIGRATED : +scx_cgroup_task_migrated (ctx); +break ; +case CGROUP_TASK_MIGRATE_CANCELED : +scx_cgroup_task_migrate_canceled (ctx); +break ; + } + +return NOTIFY_OK ; +} + +static struct notifier_block scx_cgroup_task_nb = { + .notifier_call = scx_cgroup_task_notify, +}; + +static s32 __init scx_cgroup_notifier_init (void ) { -return blocking_notifier_chain_register (&cgroup_lifetime_notifier, - &scx_cgroup_lifetime_nb); +s32 ret; + + ret =blocking_notifier_chain_register (&cgroup_lifetime_notifier, + &scx_cgroup_lifetime_nb); +if (ret) +return ret; + +return blocking_notifier_chain_register (&cgroup_task_notifier, + &scx_cgroup_task_nb); } -core_initcall (scx_cgroup_lifetime_notifier_init); +core_initcall (scx_cgroup_notifier_init);static void scx_pstack_recursion (struct bpf_prog *prog,const char *op) {Signature · 1 line
2.55.0
Tejun Heo
maintainerRe: [PATCH 4/8] sched_ext: Re-home tasks on cgroup migration
Hello,
Quoted 4 lines · Andrea Righi
Should we add:
Fixes 337ec00b1d9c ("sched_ext: Implement cgroup sub-sched enabling and disabling")
I added Closes but skipped Fixed. The feature wasn't complete anyway and it still isn't fully complete (the restricted affinity problem still needs solving), and there's a lot of code churns and dependencies inbetween. I don't wanna tag Fixes and trigger stable backports.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerRe: [PATCH 4/8] sched_ext: Re-home tasks on cgroup migration
Quoted 5 lines · Tejun Heo
Hello,
Quoted 2 lines · Andrea Righi
Should we add:
Quoted 7 lines
Quoted 2 lines
Fixes 337ec00b1d9c ("sched_ext: Implement cgroup sub-sched enabling and disabling")I added Closes but skipped Fixed. The feature wasn't complete anyway and it still isn't fully complete (the restricted affinity problem still needs solving), and there's a lot of code churns and dependencies inbetween. I don't wanna tag Fixes and trigger stable backports.
Yeah, makes sense. Closes or Link is enough, at least we have a reference to the issue.
Thanks,
-Andrea
Tejun Heo
maintainerPATCH 5/8 sched_ext: Deliver cgroup ops to each task_group's sched
In reply to Tejun HeoWith sub-schedulers claiming cgroup subtrees, cgroup ops must be delivered to each task_group's sched rather than always to root. Add tg->scx.sched to track which sched initialized the task_group. It is set and cleared together with SCX_TG_INITED.
Deliver the ops accordingly:
- ops.cgroup_exit() goes to the sched whose ops.cgroup_init() it pairs with.
- ops.cgroup_prep_move/move/cancel_move() go to the task's sched, and only for moves that don't re-home the task. A re-homing move is reported through the ops.exit_task/init_task() pair instead. The cgroups passed to the move ops can be outside the sched's inited set as the cpu controller can be coarser than the sub-scheduler topology.
- Knobs of a cgroup belong to the parent, so ops.set_weight/idle/bandwidth() go to the parent task_group's sched.
All task_groups currently resolve to the root sched, so no behavior changes until sub-schedulers start claiming cgroups.
While at it, scx_cgroup_init() is restructured so both paths share the recording.
--- include/linux/sched/ext.h | 2 + kernel/sched/ext/ext.c | 157 ++++++++++++++++++++++++++---------- kernel/sched/ext/internal.h | 34 +++++++- 3 files changed, 146 insertions(+), 47 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -298,6 +298,8 @@ static inline bool scx_rcu_cpu_stall(const struct cpumask *stalled_mask) { returstruct scx_task_group {#ifdef CONFIG_EXT_GROUP_SCHED +struct scx_sched *sched; +u32 flags;/* SCX_TG_* */ u32 weight;u64 bw_period_us;
kernel/sched/ext/ext.c +114 -43
268 lines not on this page · raw message
kernel/sched/ext/internal.h +30 -4
@@ -645,8 +645,14 @@ struct sched_ext_ops {* @cgrp: cgroup being initialized * @args: init arguments, see the struct definition * -* Either the BPF scheduler is being loaded or @cgrp created, initialize -* @cgrp for sched_ext. This operation may block. +* Initialize @cgrp for sched_ext, delivered to @cgrp's sched either +* when the BPF scheduler is being loaded or when @cgrp is created. This +* operation may block. +* +* When the BPF scheduler is being loaded or cgroups are being handed +* over, @cgrp may already have been removed by userspace: a removed +* cgroup stays schedulable until its dying tasks finish their final +* context switches. * * Return 0 for success, -errno for failure. An error return while * loading will abort loading of the BPF scheduler. During cgroup
@@ -659,8 +665,13 @@ struct sched_ext_ops {* @cgroup_exit: Exit a cgroup * @cgrp: cgroup being exited * -* Either the BPF scheduler is being unloaded or @cgrp destroyed, exit -* @cgrp for sched_ext. This operation my block. +* Exit @cgrp for sched_ext, delivered to the sched whose +* ops.cgroup_init() it pairs with, either when the BPF scheduler is +* being unloaded or when @cgrp is destroyed. This operation may block. +* +* For a destroyed @cgrp, delivery follows the last scheduling event on +* it: a removed cgroup stays schedulable until its dying tasks finish +* their final context switches. */ void (*cgroup_exit)(struct cgroup *cgrp);
@@ -673,6 +684,12 @@ struct sched_ext_ops {* Prepare @p for move from cgroup @from to @to. This operation may * block and can be used for allocations. * +* The cgroup_move ops are delivered to @p's sched, and only for moves +* that don't re-home @p. A re-homing move is reported through +* ops.exit_task() and ops.init_task() instead. @from and @to can +* reference cgroups the sched never received ops.cgroup_init() for, as +* the cpu controller can be coarser than the sub-scheduler topology. +* * Return 0 for success, -errno for failure. An error return aborts the * migration. */
@@ -708,6 +725,11 @@ struct sched_ext_ops {* @weight: new weight [1..10000] * * Update @cgrp's weight to @weight. +* +* Knobs of a cgroup belong to the parent, so the set_* ops are +* delivered to @cgrp's parent's sched. That sched may never have seen +* ops.cgroup_init() for @cgrp - at a sub-scheduler attach point, the +* parent sched tracks @cgrp through ops.sub_attach() instead. */ void (*cgroup_set_weight)(struct cgroup *cgrp,u32 weight);
@@ -728,6 +750,8 @@ struct sched_ext_ops {* burst temporarily. The specific control mechanism and thus the * interpretation of @period_us and burstiness is up to the BPF * scheduler. +* +* Delivery follows the same rule as cgroup_set_weight(). */ void (*cgroup_set_bandwidth)(struct cgroup *cgrp,u64 period_us,u64 quota_us,u64 burst_us);
@@ -740,6 +764,8 @@ struct sched_ext_ops {* Update @cgrp's idle state to @idle. This callback is invoked when * a cgroup transitions between idle and non-idle states, allowing the * BPF scheduler to adjust its behavior accordingly. +* +* Delivery follows the same rule as cgroup_set_weight(). */ void (*cgroup_set_idle)(struct cgroup *cgrp,bool idle);
Signature · 1 line
2.55.0
Tejun Heo
maintainerPATCH 6/8 sched_ext: Hand over cgroups at sub-scheduler enable/disable
In reply to Tejun HeoSub-schedulers don't get cgroups yet: every task_group is inited on the root sched and the routing added by the previous patches always resolves to it. Add the handover: an enabling sub-scheduler takes over the cgroups in its subtree and a disabling one returns them to its parent.
scx_cgroup_claim_subtree() runs while the sub enables, after the subtree's cgrp->scx_sched's are set and before any task is claimed. It inits each subtree task_group on the sub, exits it from the parent and updates tg->scx.sched. A failed ops.cgroup_init() unwinds the sub-side inits and aborts the enable with the parent untouched.
Disabling reverses it with scx_cgroup_return_subtree(): exit each cgroup from the sub, then re-init it on the parent with the current tg->scx.* values, resyncing weight and bandwidth changes made while the sub had it. When a re-init fails, the parent is failed and the remaining task_groups still transfer uninited and get no cgroup ops - the same punting done for tasks. The dying parent's own disable moves them onward.
The handover walks include dying but not yet offlined task_groups, the same as root's bulk walks: a removed cgroup keeps hosting scheduling events until its dying tasks finish their final context switches, and its ops.cgroup_exit() must follow the last of them. tg on/offlining is excluded through cgroup_lock(), so either ordering against an rmdir of a subtree cgroup delivers balanced init/exit pairs.
--- include/linux/sched/ext.h | 13 +++ kernel/sched/ext/ext.c | 45 +++++--- kernel/sched/ext/internal.h | 6 ++ kernel/sched/ext/sub.c | 199 ++++++++++++++++++++++++++++++++++++ 4 files changed, 250 insertions(+), 13 deletions(-)
include/linux/sched/ext.h +13 -0
@@ -298,6 +298,19 @@ static inline bool scx_rcu_cpu_stall(const struct cpumask *stalled_mask) { returstruct scx_task_group {#ifdef CONFIG_EXT_GROUP_SCHED +/* +* The sched this tg is on, NULL if none. SCX_TG_INITED tracks whether +* ops.cgroup_init() succeeded on it. When a child sched exits and its +* tgs move to the parent, a failed init leaves the tg on the parent +* with INITED clear (see scx_cgroup_return_subtree()). +* +* This is tracked separately from cgrp->scx_sched because the tg +* hierarchy can diverge from the cgroup2 hierarchy in both lifetime and +* shape. A tg stays online past its cgroup's removal while the +* cgrp->scx_sched rewrites visit only live cgroups, leaving a removed +* cgroup's pointer stale. The cpu controller can also be mounted on +* cgroup1. +*/ struct scx_sched *sched;u32 flags;/* SCX_TG_* */
kernel/sched/ext/ext.c +32 -13
99 lines not on this page · raw message
kernel/sched/ext/internal.h +6 -0
@@ -649,6 +649,11 @@ struct sched_ext_ops {* when the BPF scheduler is being loaded or when @cgrp is created. This * operation may block. * +* Cgroup handovers also generate these ops: an enabling sub-scheduler +* receives ops.cgroup_init() for every cgroup in its subtree while the +* previous sched receives ops.cgroup_exit(), and disabling reverses the +* two. +* * When the BPF scheduler is being loaded or cgroups are being handed * over, @cgrp may already have been removed by userspace: a removed * cgroup stays schedulable until its dying tasks finish their final
@@ -1722,6 +1727,7 @@ enum scx_kick_flags {enum scx_tg_flags {SCX_TG_ONLINE =1U <<0 ,SCX_TG_INITED =1U <<1 , +SCX_TG_SUB_INIT =1U <<2 ,/* see scx_cgroup_claim_subtree() */ };enum scx_enable_state {
kernel/sched/ext/sub.c +199 -0
217 lines not on this page · raw message
Signature · 1 line
2.55.0
Tejun Heo
maintainerPATCH 7/8 tools/sched_ext: scx_qmap - Consume cgroup weights through set_weight
In reply to Tejun HeoWith the set_* ops delivered to the parent's sched, a parent qmap instance now receives cgroup_set_weight for its child subs' attach points. Update the matching sub_sched_ctx weight and redistribute() in-kernel, and drop the userspace feed_weights() polling. This exercises the knob routing end to end.
The self weight is fixed at 100: a cgroup's weight is its parent's knob and not the scheduler's own business. This drops the self-weight polling and the repartition PROG_RUN poke with it.
sub_attach seeds the slot with the cgroup's current weight, read through bpf_cgroup_from_id(), so a weight set before the sub attaches is picked up. A write racing the attach can still be lost until the next value-changing cpu.weight write. Acceptable for a demo.
While at it, add a traced ops.cgroup_move() so tests can observe move delivery.
--- tools/sched_ext/scx_qmap.bpf.c | 80 +++++++++++++++++++++----- tools/sched_ext/scx_qmap.c | 102 +-------------------------------- tools/sched_ext/scx_qmap.h | 3 +- 3 files changed, 69 insertions(+), 116 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +66 -14
@@ -1019,10 +1019,32 @@ s32 BPF_STRUCT_OPS(qmap_cgroup_init, struct cgroup *cgrp, struct scx_cgroup_initreturn 0 ; } +static void redistribute (void ); +void BPF_STRUCT_OPS (qmap_cgroup_set_weight ,struct cgroup *cgrp,u32 weight) { +u64 cgid = cgrp->kn->id; +s32 i; + +QMAP_TOUCH_ARENA (); +if (print_msgs) -bpf_printk ("CGRP SET %llu weight=%u" , cgrp->kn->id, weight); +bpf_printk ("CGRP SET %llu weight=%u" , cgid, weight); + +/* +* Knobs belong to the parent, so this op carries the child subs' +* attach point weights. Adjust the matching sub's share of the cid +* partition. Other cgroups don't participate in the split. +*/ +for (i =0 ; i <MAX_SUB_SCHEDS ; i++) { +if (qa.sub_sched_ctxs[i].cgroup_id != cgid) +continue ; +if (qa.sub_sched_ctxs[i].weight != weight) { + qa.sub_sched_ctxs[i].weight = weight; +redistribute (); + } +break ; + } } void BPF_STRUCT_OPS(qmap_cgroup_set_bandwidth, struct cgroup *cgrp,
@@ -1033,6 +1055,14 @@ void BPF_STRUCT_OPS(qmap_cgroup_set_bandwidth, struct cgroup *cgrp, cgrp->kn->id, period_us, quota_us, burst_us); } +void BPF_STRUCT_OPS (qmap_cgroup_move ,struct task_struct *p, +struct cgroup *from,struct cgroup *to) +{ +if (print_msgs) +bpf_printk ("CGRP MOVE %d %llu -> %llu" , + p->pid, from->kn->id, to->kn->id); +} +void BPF_STRUCT_OPS (qmap_update_idle ,s32 cid,bool idle) {QMAP_TOUCH_ARENA ();
@@ -1336,10 +1366,12 @@ __noinline void compute_partition(void) }/* -* Snapshot membership and weights so the sum_w and share loops agree. -* A mid-compute change would otherwise wrap nr_shared negative. +* Snapshot membership and weights so the sum_w and share loops agree. A +* mid-compute change would otherwise wrap nr_shared negative. The self +* weight is fixed at the default: a cgroup's weight is its parent's +* knob, not the scheduler's own business. */ - self_w = qa.self_weight ?:100 ; + self_w =100 ; bpf_for(i, 0, MAX_SUB_SCHEDS) { cgid_snap[i] = qa.sub_sched_ctxs[i].cgroup_id; w_snap[i] = cgid_snap[i] ? (qa.sub_sched_ctxs[i].weight ?:100 ) :0 ;
@@ -1565,7 +1597,7 @@ __noinline void apply_partition(void)/* * Recompute the split off the node's held caps and apply it. The contexts this -* runs from (the sub-sched callbacks, the userspace poke, the rr timer) are not +* runs from (the sub-sched and cgroup callbacks, the rr timer) are not * serialized by the kernel, so a single runner does the work. A caller that * finds the guard held leaves part_pending set; the holder drains it before * releasing, with the rr timer as a backstop.
@@ -1592,14 +1624,6 @@ static void redistribute(void)part_end (); } -/* userspace pokes this (PROG_RUN) to resplit after a cpu.weight change */ -SEC("syscall") -int repartition (void *ctx) -{ -redistribute (); -return 0 ; -} -/* * Userspace pokes this (PROG_RUN) to bring alloc_ns[] current before reading * it for the stats display. Skipping when the partition guard is held is
@@ -1849,6 +1873,33 @@ void BPF_STRUCT_OPS(qmap_exit, struct scx_exit_info *ei)UEI_RECORD (uei, ei); } +/* +* Seed a new sub slot with the cgroup's current weight. The kernel delivers +* ops.cgroup_set_weight() only on value-changing writes, so a weight set +* before the sub attached would otherwise go unnoticed. +*/ +static u32 cgrp_cur_weight (u64 cgid) +{ +struct cgroup_subsys_state *css; +struct cgroup *cgrp; +u32 weight =100 ; + + cgrp =bpf_cgroup_from_id (cgid); +if (!cgrp) +return weight; + + css =BPF_CORE_READ (cgrp, subsys[cpu_cgrp_id]); +if (css) { + struct task_group *tg = container_of(css, struct task_group, css); +u32 w =BPF_CORE_READ (tg, scx.weight); + +if (w) + weight = w; + } +bpf_cgroup_release (cgrp); +return weight; +} +s32 BPF_STRUCT_OPS (qmap_sub_attach ,struct scx_sub_attach_args *args) {s32 i;
@@ -1862,7 +1913,7 @@ s32 BPF_STRUCT_OPS(qmap_sub_attach, struct scx_sub_attach_args *args)continue ; qa.sub_sched_ctxs[i].cgroup_id = args->ops->sub_cgroup_id; - qa.sub_sched_ctxs[i].weight =100 ;/* until userspace feeds it */ + qa.sub_sched_ctxs[i].weight =cgrp_cur_weight (args->ops->sub_cgroup_id); qa.nr_sub_scheds++;bpf_printk ("attaching sub-sched[%d] on %s" , i, args->cgroup_path);redistribute ();
@@ -1926,6 +1977,7 @@ SCX_OPS_CID_DEFINE(qmap_ops, .cgroup_init = (void *)qmap_cgroup_init, .cgroup_set_weight = (void *)qmap_cgroup_set_weight, .cgroup_set_bandwidth = (void *)qmap_cgroup_set_bandwidth, + .cgroup_move = (void *)qmap_cgroup_move, .sub_attach = (void *)qmap_sub_attach, .sub_detach = (void *)qmap_sub_detach, .sub_caps_updated = (void *)qmap_sub_caps_updated,
tools/sched_ext/scx_qmap.c +2 -100
138 lines not on this page · raw message
tools/sched_ext/scx_qmap.h +1 -2
@@ -90,7 +90,7 @@ struct task_ctx;/* per-direct-child state for the sub-scheduler */ struct sub_sched_ctx {u64 cgroup_id; -u32 weight;/* cpu.weight, fed by userspace */ +u32 weight;/* cpu.weight, seeded at attach, then set_weight */ u64 nr_dsps;struct qmap_cmask granted_cids;/* cids granted excl to this child */ struct qmap_cmask prev_granted;/* last grant, for delta calculation */
@@ -147,7 +147,6 @@ struct qmap_arena {struct sub_sched_ctx sub_sched_ctxs[MAX_SUB_SCHEDS ];/* per-child context */ u64 nr_sub_scheds;/* number of attached children */ -u32 self_weight;/* this node's cpu.weight, fed by userspace */ /* bpf-internal per-cid state */ u8 cid_shared[SCX_QMAP_MAX_CPUS ];/* per cid: 1 if held shared (ENQ_IMMED-only) */
Signature · 1 line
2.55.0
Tejun Heo
maintainerPATCH 8/8 tools/sched_ext: scx_qmap - Add init fault injection modes
In reply to Tejun HeoAdd -J init-fail which makes ops.init_task() fail with -ENOMEM for tasks whose comm starts with "qmfail", and -J cgrp-init-fail which does the same in ops.cgroup_init() for cgroups named "qmfail*".
The former exercises the migration veto path: the cgroup.procs write must fail with the injected errno while the destination sched stays up and the task stays put. The latter exercises the ownership-return failure path: a parent failing to re-init a returned cgroup leaves it unowned, and moves and set_* ops against it must be skipped instead of dereferencing the missing owner. Matching on "qmfail" names keeps the injecting scheduler's own enable unaffected.
--- tools/sched_ext/scx_qmap.bpf.c | 15 +++++++++++++++ tools/sched_ext/scx_qmap.c | 8 +++++++- tools/sched_ext/scx_qmap.h | 2 ++ 3 files changed, 24 insertions(+), 1 deletion(-)
tools/sched_ext/scx_qmap.bpf.c +15 -0
@@ -888,6 +888,10 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init_task, struct task_struct *p,struct task_ctx_stor_val *v;task_ctx_t *taskc; +if (qa.inject_mode ==QMAP_INJ_INIT_FAIL && + !bpf_strncmp (p->comm,6 ,"qmfail" )) +return -ENOMEM ; +if (p->tgid == disallow_tgid) p->scx.disallow = true;
@@ -1012,10 +1016,21 @@ void BPF_STRUCT_OPS(qmap_dump_task, struct scx_dump_ctx *dctx, struct task_strucs32 BPF_STRUCT_OPS (qmap_cgroup_init ,struct cgroup *cgrp,struct scx_cgroup_init_args *args) { +QMAP_TOUCH_ARENA (); +if (print_msgs)bpf_printk ("CGRP INIT %llu weight=%u period=%lu quota=%ld burst=%lu" , cgrp->kn->id, args->weight, args->bw_period_us, args->bw_quota_us, args->bw_burst_us); + +if (qa.inject_mode ==QMAP_INJ_CGRP_INIT_FAIL ) { +char name[7 ] = {}; + +bpf_probe_read_kernel_str (name,sizeof (name), cgrp->kn->name); +if (!bpf_strncmp (name,6 ,"qmfail" )) +return -ENOMEM ; + } +return 0 ; }
tools/sched_ext/scx_qmap.c +7 -1
@@ -69,7 +69,9 @@ const char help_fmt[] =" -C MODE cid-override test (shuffle|bad-dup|bad-range|bad-mono)\n" " -i SEC Stats interval, seconds (default 5)\n" " -R MS Round-robin period for time-shared cpus, ms (default 200)\n" -" -J MODE Fault injection (wrong-cid: dispatch to a cid not held)\n" +" -J MODE Fault injection (wrong-cid: dispatch to a cid not held,\n" +" init-fail/cgrp-init-fail: fail init_task/cgroup_init for\n" +" \"qmfail*\" comms/cgroups)\n" " -v Print libbpf debug messages\n" " -h Display this help and exit\n" ;
@@ -391,6 +393,10 @@ int main(int argc, char **argv)case 'J' :if (!strcmp (optarg,"wrong-cid" )) inject_mode =QMAP_INJ_WRONG_CID ; +else if (!strcmp (optarg,"init-fail" )) + inject_mode =QMAP_INJ_INIT_FAIL ; +else if (!strcmp (optarg,"cgrp-init-fail" )) + inject_mode =QMAP_INJ_CGRP_INIT_FAIL ;else inject_mode =strtoul (optarg,NULL ,0 );break ;
tools/sched_ext/scx_qmap.h +2 -0
@@ -64,6 +64,8 @@ struct qmap_fifo {enum qmap_inject {QMAP_INJ_OFF =0 ,QMAP_INJ_WRONG_CID =1 ,/* dispatch to a cid we don't hold */ +QMAP_INJ_INIT_FAIL =2 ,/* fail init_task for "qmfail*" comms */ +QMAP_INJ_CGRP_INIT_FAIL =3 ,/* fail cgroup_init for "qmfail*" cgroups */ };/*
Signature · 1 line
2.55.0
Hi Tejun,
Quoted 26 lines · Tejun Heo
Hello,
A task's sched must match its cgroup's: each sub-scheduler serves the cgroup2 subtree it is attached to and the root scheduler serves the rest. Cgroup migration currently breaks this. A task moved across a sub-scheduler boundary keeps its old sched, leading to wrong-sched scheduling and, once the stale sched is freed, a use-after-free. Cgroup ops are also always delivered to the root scheduler no matter which sched serves the cgroup.
This patchset makes both follow the sub-scheduler topology:
- cgroup gains a task migration notifier (0001).
sched_extuses it to re-home tasks whose migration crosses a sched boundary: the destination sched runs the fallibleops.init_task()before the migration commits, so a rejection fails the cgroup.procs write (0002-0004).- cgroup_init/exit and the knob ops are delivered to the sched each
task_groupis on. The move ops go to the task's sched, only for migrations that don't re-home it (0005).- Sub-scheduler enable claims the subtree's cgroups from the parent and disable returns them. A parent that fails to re-init a returned cgroup fails in turn and the cgroups move up the chain, ending at the root scheduler (0006).
Quoted 6 lines
-
scx_qmapconsumes cgroup weights throughops.cgroup_set_weight()and gains fault injection modes to exercise the failure paths (0007-0008).Verified with a trace-based suite on the demo scheduler covering op delivery and routing, handover failures, multi-level punt chains, cgroup1 and stress.
I left a comment on PATCH 4/8, but overall the series looks good to me.
I also confirm that, with this applied, I can no longer reproduce the use-after-free triggered by moving a task out of a sub-scheduler's cgroup and then terminating the sub-scheduler.
Thanks,
-Andrea
Quoted 28 lines
Based on sched_ext/for-7.3 (
7c2cd767705d).Tejun Heo (8):
cgroup: Addcgroup_task_notifierand task migration events
sched_ext: Factor outscx_rehome_task()andscx_punt_task()
sched_ext: Relocatescx_cgroup_enabled
sched_ext: Re-home tasks on cgroup migration
sched_ext: Deliver cgroup ops to each task_group's sched
sched_ext: Hand over cgroups at sub-scheduler enable/disabletools/sched_ext:scx_qmap- Consume cgroup weights throughset_weighttools/sched_ext:scx_qmap- Add init fault injection modesGit tree:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.gitscx-cgroup-migrationinclude/linux/cgroup.h | 26 +++ include/linux/sched/ext.h | 15 ++ kernel/cgroup/cgroup.c | 93 ++++++++- kernel/sched/ext/ext.c | 247 +++++++++++++++++------- kernel/sched/ext/internal.h | 44 ++++- kernel/sched/ext/sub.c | 418 +++++++++++++++++++++++++++++++++++++---- tools/sched_ext/scx_qmap.bpf.c | 95 ++++++++-- tools/sched_ext/scx_qmap.c | 110 +---------- tools/sched_ext/scx_qmap.h | 5 +- 9 files changed, 819 insertions(+), 234 deletions(-)--
tejun
Tejun Heo
maintainerApplied 1-8 to sched_ext/for-7.3.
Thanks.
--
tejun