sched_ext: Sub-scheduler follow-ups
Subsystems (2)
Tejun Heo
reviewerPATCH v2 1/1 sched_ext: Sub-scheduler follow-ups
Hello,
v2: - Host scx_dispatch_sched() in a new inlines.h rather than at the end of
internal.h, which formed a circular include with cid.h (sashiko AI), and drop sub.h's now-unused cid.h include.
v1: https://lore.kernel.org/r/[email protected]
Follow-ups to the sub-scheduler capability series, addressing two things Andrea raised while reviewing v5[1]: sched teardown drained queued ecaps syncs with an msleep() poll loop, and with CONFIG_EXT_SUB_SCHED=y hot paths keep paying for sub-sched bookkeeping even when no sub-scheduler exists.
Patch 0001 replaces the teardown poll loop with direct removal from the llist. Patches 0002-0003 are prep and 0004 adds the scx_has_subs static key and gates the sub-sched portions of hot paths on it.
[1] https://lore.kernel.org/r/[email protected]
Based on sched_ext/for-7.3 (35f9cbbacb67).
This patchset contains the following 4 patches.
0001 sched_ext: Remove queued ecaps syncs directly on sched teardown 0002 sched_ext: Move scx_dispatch_sched() to a new inlines.h 0003 sched_ext: Gate sub_dispatch_prev with CONFIG_EXT_SUB_SCHED 0004 sched_ext: Add the scx_has_subs static key and gate sub-sched hot paths
The patchset is also available in the following git branch:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-sub-followups-v2
diffstat follows. Thanks.
kernel/sched/build_policy.c | 1 + kernel/sched/ext/ext.c | 7 ++- kernel/sched/ext/idle.c | 10 ++++ kernel/sched/ext/inlines.h | 128 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/ext/internal.h | 19 ++++++- kernel/sched/ext/sub.c | 89 ++++++++++++++++++++---------- kernel/sched/ext/sub.h | 125 ++++++------------------------------------ kernel/sched/sched.h | 3 +- 8 files changed, 240 insertions(+), 142 deletions(-)
--
tejun
Tejun Heo
reviewerPATCH 1/4 sched_ext: Remove queued ecaps syncs directly on sched teardown
scx_discard_ecaps_to_sync() waited for balance_one() to consume a dying sched's queued ecaps sync, polling with resched_cpu() + msleep(). The wait is unbounded - the ext dl_server forces picks through sustained fair or RT load only while ext tasks are queued, so an ext-idle cpu monopolized by a higher class can stall the teardown indefinitely.
Remove the node directly instead: take all queued nodes, drop the dying sched's and resplice the rest. Consumption runs under the rq lock and batch nodes read as on-list throughout, so the producer-side dedup stays correct.
A node that an in-flight scx_process_sync_ecaps() batch holds across a dispatch-induced rq unlock still needs a wait, but one bounded by that batch completing rather than by a future balance.
--- kernel/sched/ext/ext.c | 2 +- kernel/sched/ext/sub.c | 67 ++++++++++++++++++++++++++---------------- 2 files changed, 43 insertions(+), 26 deletions(-)
kernel/sched/ext/ext.c +1 -1
@@ -4877,7 +4877,7 @@ static void scx_sched_free_rcu_work(struct work_struct *work)*/ WARN_ON_ONCE (!list_empty (&pcpu->deferred_reenq_local.node)); -/* retire the queued ecaps syncs so the pcpu can be freed */ +/* remove the queued ecaps sync so the pcpu can be freed */ scx_discard_ecaps_to_sync (cpu, pcpu);/*
kernel/sched/ext/sub.c +42 -25
101 lines not on this page · raw message
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 2/4 sched_ext: Move scx_dispatch_sched() to a new inlines.h
In reply to Tejun Heoscx_dispatch_sched() is common dispatch machinery and looks out of place in sub.h, but it needs scx_cpu_arg() from cid.h and can't move into internal.h without creating a circular include. Add inlines.h on top of internal.h and cid.h, and move the function there. The function was sub.h's only cid.h user, so drop that include. Pure code move, no functional change.
v2: Host the function in a new inlines.h instead of at internal.h's tail,
which formed a circular include with cid.h. Drop sub.h's now-unused cid.h include. (sashiko AI)
--- kernel/sched/build_policy.c | 1 + kernel/sched/ext/ext.c | 1 + kernel/sched/ext/inlines.h | 125 ++++++++++++++++++++++++++++++++++++ kernel/sched/ext/sub.c | 1 + kernel/sched/ext/sub.h | 110 ------------------------------- 5 files changed, 128 insertions(+), 110 deletions(-) create mode 100644 kernel/sched/ext/inlines.h
kernel/sched/build_policy.c +1 -0
@@ -67,6 +67,7 @@# include "ext/arena.h" # include "ext/idle.h" # include "ext/sub.h" +# include "ext/inlines.h" # include "ext/ext.c" # include "ext/cid.c" # include "ext/arena.c"
kernel/sched/ext/ext.c +1 -0
@@ -20,6 +20,7 @@#include "arena.h" #include "idle.h" #include "sub.h" +#include "inlines.h" DEFINE_RAW_SPINLOCK (scx_sched_lock);
kernel/sched/ext/inlines.h new file +125 -0
125 lines not on this page · raw message
kernel/sched/ext/sub.c +1 -0
@@ -18,6 +18,7 @@#include "cid.h" #include "arena.h" #include "sub.h" +#include "inlines.h" #ifdef CONFIG_EXT_SUB_SCHED
kernel/sched/ext/sub.h +0 -110
120 lines not on this page · raw message
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 3/4 sched_ext: Gate sub_dispatch_prev with CONFIG_EXT_SUB_SCHED
In reply to Tejun Heorq->scx.sub_dispatch_prev is sub-sched-only but was left unconditional. Move it into the CONFIG_EXT_SUB_SCHED block next to ecaps_to_sync and gate its updates.
--- kernel/sched/ext/inlines.h | 19 +++++++++++-------- kernel/sched/sched.h | 3 +-- 2 files changed, 12 insertions(+), 10 deletions(-)
kernel/sched/ext/inlines.h +11 -8
@@ -74,16 +74,19 @@ scx_dispatch_sched(struct scx_sched *sch, struct rq *rq, do { dspc->nr_tasks = 0; - if (nested) { - SCX_CALL_OP(sch, dispatch, rq, scx_cpu_arg(cpu), - prev_on_sch ? prev : NULL); - } else { - /* stash @prev so that nested invocations can access it */ +#ifdef CONFIG_EXT_SUB_SCHED + /* stash @prev so that nested invocations can access it */ + if (!nested) rq->scx.sub_dispatch_prev = prev; - SCX_CALL_OP(sch, dispatch, rq, scx_cpu_arg(cpu), - prev_on_sch ? prev : NULL); +#endif + + SCX_CALL_OP(sch, dispatch, rq, scx_cpu_arg(cpu), + prev_on_sch ? prev : NULL); + +#ifdef CONFIG_EXT_SUB_SCHED + if (!nested) rq->scx.sub_dispatch_prev = NULL; - } +#endif scx_flush_dispatch_buf(sch, rq);
kernel/sched/sched.h +1 -2
@@ -814,6 +814,7 @@ struct scx_rq {u64 clock;/* current per-rq clock -- see scx_bpf_now() */ #ifdef CONFIG_EXT_SUB_SCHED struct llist_head ecaps_to_sync;/* pending ecaps syncs */ +struct task_struct *sub_dispatch_prev;#endif cpumask_var_t cpus_to_sync;bool kick_sync_pending;
@@ -821,8 +822,6 @@ struct scx_rq {struct list_head sched_pcpus_to_kick;/* see kick_cpus_irq_workfn() */ -struct task_struct *sub_dispatch_prev; -raw_spinlock_t deferred_reenq_lock;u64 deferred_reenq_locals_seq;struct list_head deferred_reenq_locals;/* scheds requesting reenq of local DSQ */
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 4/4 sched_ext: Add the scx_has_subs static key and gate sub-sched hot paths
In reply to Tejun HeoWith CONFIG_EXT_SUB_SCHED=y but no sub-scheduler attached - the common case
- hot paths still pay for sub-sched bookkeeping. Gate it behind __scx_has_subs, a static key counting live sub-schedulers, so that a root-only system stops paying.
Most conversions are simple skip-if-no-sub tests. scx_idle_notify() is special - it's a hierarchy walk, so give it a fast path which notifies the root directly using the same tests as the walk. A pending SCX_RQ_SUB_IDLE_RENOTIFY can be ignored as no sub can be owed one and the caller clears the flag either way.
--- kernel/sched/ext/ext.c | 4 ++++ kernel/sched/ext/idle.c | 10 ++++++++++ kernel/sched/ext/internal.h | 19 +++++++++++++++++-- kernel/sched/ext/sub.c | 21 +++++++++++++++++++-- kernel/sched/ext/sub.h | 15 +++++++++++++++ 5 files changed, 65 insertions(+), 4 deletions(-)
kernel/sched/ext/ext.c +4 -0
@@ -4920,6 +4920,10 @@ static void scx_sched_free_rcu_work(struct work_struct *work)scx_arena_pool_destroy (sch);if (sch->arena_map)bpf_map_put (sch->arena_map); + +/* @sch is completely inactive by now */ +scx_dec_has_subs (sch); +kfree (sch); }
kernel/sched/ext/idle.c +10 -0
@@ -746,6 +746,16 @@ static void scx_idle_notify(struct rq *rq, bool idle, bool do_notify, bool root_lockdep_assert_rq_held (rq); +/* with no sub-sched, only the root can be owed a notification */ +if (!scx_has_subs ()) { +struct scx_sched *sch = scx_root; + +if ((do_notify || root_renotify) && +SCX_HAS_OP (sch, update_idle) && !scx_bypassing (sch, cpu)) +SCX_CALL_OP (sch, update_idle, rq, cid, idle); +return ; + } + pos =scx_next_descendant_pre (NULL , scx_root);while (pos) {bool forced = false;
kernel/sched/ext/internal.h +17 -2
@@ -2110,7 +2110,7 @@ do { \*/ #define SCX_CALL_OP_TASK(sch, op, locked_rq, task, args...) \ do { \ - WARN_ON_ONCE((sch) != scx_task_sched_rcu(task)); \ + WARN_ON_ONCE(scx_has_subs() && (sch) != scx_task_sched_rcu(task)); \ __SCX_CALL_OP_TASK((sch), ops, op, locked_rq, task, ##args); \ } while (0)
@@ -2125,7 +2125,7 @@ do { \ #define SCX_CALL_OP_TASK_RET(sch, op, locked_rq, task, args...) \ ({ \ __typeof__((sch)->ops.op(task, ##args)) __ret; \ - WARN_ON_ONCE((sch) != scx_task_sched_rcu(task)); \ + WARN_ON_ONCE(scx_has_subs() && (sch) != scx_task_sched_rcu(task)); \ WARN_ON_ONCE(current->scx.kf_tasks[0]); \ current->scx.kf_tasks[0] = task; \ __ret = SCX_CALL_OP_RET((sch), op, locked_rq, task, ##args); \
@@ -2165,6 +2165,19 @@ static inline bool scx_bypassing(struct scx_sched *sch, s32 cpu) }#ifdef CONFIG_EXT_SUB_SCHED +DECLARE_STATIC_KEY_FALSE (__scx_has_subs); + +/** +* scx_has_subs - Whether any sub-scheduler exists +* +* Gates the sub-sched portions of hot paths so that a root-only system doesn't +* pay for them. See scx_sub_enable_workfn() and scx_sched_free_rcu_work(). +*/ +static inline bool scx_has_subs (void ) +{ +return static_branch_unlikely (&__scx_has_subs); +} +/** * scx_task_sched - Find scx_sched scheduling a task * @p: task of interest
@@ -2259,6 +2272,8 @@ static inline struct scx_sched *scx_parent(struct scx_sched *sch)return NULL ; }#else /* CONFIG_EXT_SUB_SCHED */ +static inline bool scx_has_subs (void ) {return false; } +static inline struct scx_sched *scx_task_sched (const struct task_struct *p) { return rcu_dereference_protected(scx_root,
kernel/sched/ext/sub.c +19 -2
@@ -22,6 +22,12 @@#ifdef CONFIG_EXT_SUB_SCHED +/* +* On while any sub-scheduler exists so that a root-only system doesn't pay for +* the sub-sched portions of hot paths. See scx_has_subs(). +*/ +DEFINE_STATIC_KEY_FALSE (__scx_has_subs); +/** * scx_skip_subtree_pre - Skip @pos's subtree in a pre-order walk * @pos: current position
@@ -236,6 +242,9 @@ void scx_init_root_caps(struct scx_sched *sch)struct scx_dispatch_q *scx_local_or_reject_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) { +if (!scx_has_subs ()) +return &rq->scx.local_dsq; +s32 cid =__scx_cpu_to_cid (cpu_of (rq));struct scx_sched *asch = rq->scx.remote_activate_sch ?: sch;u64 needed =scx_caps_for_enq (*enq_flags);
@@ -314,7 +323,7 @@ void scx_reenq_reject(struct rq *rq)lockdep_assert_rq_held (rq); -if (list_empty (&rq->scx.reject_dsq.list)) +if (!scx_has_subs () ||list_empty (&rq->scx.reject_dsq.list))return ;/*
@@ -497,7 +506,7 @@ void scx_process_sync_ecaps(struct rq *rq, struct task_struct *prev)lockdep_assert_rq_held (rq); -if (likely (llist_empty (&rq->scx.ecaps_to_sync))) +if (!scx_has_subs () ||likely (llist_empty (&rq->scx.ecaps_to_sync)))return ;/*
@@ -1016,10 +1025,18 @@ void scx_sub_enable_workfn(struct kthread_work *work)kobject_get (&parent->kobj);raw_spin_unlock_irq (&scx_sched_lock); +/* +* Flip the hot-path gates before ops->priv is published - the sub's +* programs can e.g. kick cpus from that point on. The matching dec is +* at the end of scx_sched_free_rcu_work(). +*/ +static_branch_inc (&__scx_has_subs); +/* scx_alloc_and_add_sched() consumes @cgrp whether it succeeds or not */ sch =scx_alloc_and_add_sched (cmd, cgrp, parent);kobject_put (&parent->kobj);if (IS_ERR (sch)) { +static_branch_dec (&__scx_has_subs); ret =PTR_ERR (sch); goto out_unlock; }
kernel/sched/ext/sub.h +15 -0
@@ -44,6 +44,13 @@ static inline const char *sch_cgrp_path(struct scx_sched *sch) return sch->cgrp_path; } +/* a dying sub's hot-path influence ends in scx_sched_free_rcu_work() */ +static inline void scx_dec_has_subs (struct scx_sched *sch) +{ +if (sch->level) +static_branch_dec (&__scx_has_subs); +} +#else /* CONFIG_EXT_SUB_SCHED */ static inline struct scx_sched *scx_next_descendant_pre (struct scx_sched *pos,struct scx_sched *root) {return pos ?NULL : root; }
@@ -66,6 +73,7 @@ static inline void scx_discard_stale_ecaps_syncs(void) {}static inline struct scx_dispatch_q *scx_local_or_reject_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) {return &rq->scx.local_dsq; }static inline bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p) {return false; }static inline void scx_reenq_reject (struct rq *rq) {} +static inline void scx_dec_has_subs (struct scx_sched *sch) {}#endif /* CONFIG_EXT_SUB_SCHED */
@@ -96,6 +104,10 @@ static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed) {u64 ecaps; +/* no sub-scheds, no missing caps */ +if (!scx_has_subs ()) +return 0 ; +/* root holds every cap on every cpu */ if (!sch->level)return 0 ;
@@ -156,6 +168,9 @@ static inline u64 scx_caps_implied(u64 cap)/* may @p keep running on @rq's cpu? requires baseline cpu access */ static inline bool scx_task_can_stay_on_cpu (struct rq *rq,struct task_struct *p) { +if (!scx_has_subs ()) +return true; +/* a migration-disabled task is let in without caps, keep it likewise */ if (unlikely (is_migration_disabled (p)))return true;
Signature · 1 line
2.55.0
Tejun Heo
reviewerApplied 1-4 to sched_ext/for-7.3.
Thanks.
--
tejun