sched_ext: Topological CPU IDs and cid-form struct_ops
Subsystems (2)
Tejun Heo
reviewerPATCH v3 1/1 sched_ext: Topological CPU IDs and cid-form struct_ops
Hello,
v3 (all from the Sashiko AI review at
https://sashiko.dev/#/patchset/20260424172721.3458520-1-tj%40kernel.org):
- cid: drop leaked cpus_read_lock() on scx_cid_init() failure; BUILD_BUG_ON tightened to NR_CPUS<=8192 to match the BPF cmask helpers' CMASK_MAX_WORDS coverage.
- bpf-struct-size: use offsetof() in struct_size() to match the kernel <linux/overflow.h> macro semantics (no inflation from trailing struct padding).
- cmask: cmask_copy_from_kernel() validates src->base==0 via probe-read; nr_bits check is bit-level rather than rounded-up word-count.
- cid-qmap-idle: qmap_init() refuses to load when scx_bpf_nr_cids() exceeds SCX_QMAP_MAX_CPUS; the task_ctx flex array would otherwise overflow into the next slab entry.
v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]
This patchset introduces topological CPU IDs (cids) - dense, topology-ordered cpu identifiers - and an alternative cid-form struct_ops type that lets BPF schedulers operate in cid space directly.
Key pieces:
- cid space: scx_cid_init() walks nodes * LLCs * cores * threads and packs a dense cid mapping. The mapping can be overridden via scx_bpf_cid_override(). See "Topological CPU IDs" in ext_cid.h for the model.
- cmask: a base-windowed bitmap over cid space. Kernel and BPF helpers with identical semantics. Used by scx_qmap for per-task affinity and idle-cid tracking; meant to be the substrate for sub-sched cid allocation.
- bpf_sched_ext_ops_cid: a parallel struct_ops type whose callbacks take cids/cmasks instead of cpus/cpumasks. Kernel translates at the boundary via scx_cpu_arg() / scx_cpu_ret(); the two struct types share offsets up through @priv (verified by BUILD_BUG_ON) so the union view in scx_sched works without function-pointer casts. Sub-sched support is tied to
cid-form: validate_ops() rejects cpu-form sub-scheds and cpu-form roots that expose sub_attach / sub_detach.
- cid-form kfuncs: scx_bpf_kick_cid, scx_bpf_cidperf_{cap,cur,set}, scx_bpf_cid_curr, scx_bpf_task_cid, scx_bpf_this_cid, scx_bpf_nr_{cids,online_cids}, scx_bpf_cid_to_cpu, scx_bpf_cpu_to_cid. A cid-form program may not call cpu-only kfuncs (enforced at verifier load via scx_kfunc_context_filter); the reverse is intentionally permissive to ease migration.
- scx_qmap port: scx_qmap is converted to cid-form. It uses the cmask-based idle picker, per-task cid-space cpus_allowed, and cid-form kfuncs throughout. Sub-sched dispatching via scx_bpf_sub_dispatch() continues to work.
v3 re-tested on the 16-cpu QEMU: cid-form scx_qmap under stress-ng plus reload cycles, hotplug auto-restart, and sub-sched (root scx_qmap + cgroup-scoped scx_qmap child). Clean.
Based on sched_ext/for-7.2 (4939721aad2e).
0001-sched_ext-Add-ext_types.h-for-early-subsystem-wide-d.patch 0002-sched_ext-Rename-ops_cpu_valid-to-scx_cpu_valid-and-.patch 0003-sched_ext-Move-scx_exit-scx_error-and-friends-to-ext.patch 0004-sched_ext-Shift-scx_kick_cpu-validity-check-to-scx_b.patch 0005-sched_ext-Relocate-cpu_acquire-cpu_release-to-end-of.patch 0006-sched_ext-Make-scx_enable-take-scx_enable_cmd.patch 0007-sched_ext-Add-topological-CPU-IDs-cids.patch 0008-sched_ext-Add-scx_bpf_cid_override-kfunc.patch 0009-tools-sched_ext-Add-struct_size-helpers-to-common.bp.patch 0010-sched_ext-Add-cmask-a-base-windowed-bitmap-over-cid-.patch 0011-sched_ext-Add-cid-form-kfunc-wrappers-alongside-cpu-.patch 0012-sched_ext-Add-bpf_sched_ext_ops_cid-struct_ops-type.patch 0013-sched_ext-Forbid-cpu-form-kfuncs-from-cid-form-sched.patch 0014-tools-sched_ext-scx_qmap-Restart-on-hotplug-instead-.patch 0015-tools-sched_ext-scx_qmap-Add-cmask-based-idle-tracki.patch 0016-tools-sched_ext-scx_qmap-Port-to-cid-form-struct_ops.patch 0017-sched_ext-Require-cid-form-struct_ops-for-sub-sched-.patch
Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-cid-v3
kernel/sched/build_policy.c | 3 + kernel/sched/ext.c | 651 ++++++++++++++++++++++++++---- kernel/sched/ext_cid.c | 409 +++++++++++++++++++ kernel/sched/ext_cid.h | 164 ++++++++ kernel/sched/ext_idle.c | 8 +- kernel/sched/ext_internal.h | 205 +++++++--- kernel/sched/ext_types.h | 104 +++++ tools/sched_ext/include/scx/cid.bpf.h | 667 +++++++++++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 23 ++ tools/sched_ext/include/scx/compat.bpf.h | 24 ++ tools/sched_ext/scx_qmap.bpf.c | 346 +++++++++------- tools/sched_ext/scx_qmap.c | 70 +++- tools/sched_ext/scx_qmap.h | 2 +- 13 files changed, 2391 insertions(+), 285 deletions(-)
Thanks.
--
tejun
Tejun Heo
reviewerPATCH 1/17 sched_ext: Add ext_types.h for early subsystem-wide defs
Introduce kernel/sched/ext_types.h as the early-def header for the sched_ext compilation unit. Included from kernel/sched/build_policy.c before ext_internal.h so every later header and source in the unit sees its content without re-inclusion. Later patches add their types here (struct scx_cid_topo, scx_cmask, scx_cid_shard, etc.) so the subsystem has one place to stash types shared across the TU.
Move enum scx_consts (SCX_DSP_DFL_MAX_BATCH, SCX_WATCHDOG_MAX_TIMEOUT, SCX_SUB_MAX_DEPTH, etc.) here as the initial content. Ops-facing content stays in ext_internal.h.
--- kernel/sched/build_policy.c | 1 + kernel/sched/ext_internal.h | 32 --------------------------- kernel/sched/ext_types.h | 43 +++++++++++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+), 32 deletions(-) create mode 100644 kernel/sched/ext_types.h
kernel/sched/build_policy.c +1 -0
@@ -59,6 +59,7 @@#ifdef CONFIG_SCHED_CLASS_EXT # include <linux/btf_ids.h> +# include "ext_types.h" # include "ext_internal.h" # include "ext_idle.h" # include "ext.c"
kernel/sched/ext_internal.h +0 -32
@@ -8,38 +8,6 @@#define SCX_OP_IDX (op) (offsetof(struct sched_ext_ops, op) / sizeof(void (*)(void)))#define SCX_MOFF_IDX (moff) ((moff) / sizeof(void (*)(void))) -enum scx_consts { -SCX_DSP_DFL_MAX_BATCH =32 , -SCX_DSP_MAX_LOOPS =32 , -SCX_WATCHDOG_MAX_TIMEOUT =30 *HZ , - -/* per-CPU chunk size for p->scx.tid allocation, see scx_alloc_tid() */ -SCX_TID_CHUNK =1024 , - -SCX_EXIT_BT_LEN =64 , -SCX_EXIT_MSG_LEN =1024 , -SCX_EXIT_DUMP_DFL_LEN =32768 , - -SCX_CPUPERF_ONE =SCHED_CAPACITY_SCALE , - -/* -* Iterating all tasks may take a while. Periodically drop -* scx_tasks_lock to avoid causing e.g. CSD and RCU stalls. -*/ -SCX_TASK_ITER_BATCH =32 , - -SCX_BYPASS_HOST_NTH =2 , - -SCX_BYPASS_LB_DFL_INTV_US =500 *USEC_PER_MSEC , -SCX_BYPASS_LB_DONOR_PCT =125 , -SCX_BYPASS_LB_MIN_DELTA_DIV =4 , -SCX_BYPASS_LB_BATCH =256 , - -SCX_REENQ_LOCAL_MAX_REPEAT =256 , - -SCX_SUB_MAX_DEPTH =4 , -}; -enum scx_exit_kind {SCX_EXIT_NONE ,SCX_EXIT_DONE ,
kernel/sched/ext_types.h new file +43 -0
43 lines not on this page · raw message
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 2/17 sched_ext: Rename ops_cpu_valid() to scx_cpu_valid() and expose it
In reply to Tejun HeoRename the static ext.c helper and declare it in ext_internal.h so ext_idle.c and the upcoming cid code can call it directly instead of relying on build_policy.c textual inclusion.
Pure rename and visibility change.
--- kernel/sched/ext.c | 22 +++++++++++----------- kernel/sched/ext_idle.c | 6 +++--- kernel/sched/ext_internal.h | 2 ++ 3 files changed, 16 insertions(+), 14 deletions(-)
kernel/sched/ext.c +11 -11
@@ -1062,7 +1062,7 @@ static inline bool __cpu_valid(s32 cpu) }/** -* ops_cpu_valid - Verify a cpu number, to be used on ops input args +* scx_cpu_valid - Verify a cpu number, to be used on ops input args * @sch: scx_sched to abort on error * @cpu: cpu number which came from a BPF ops * @where: extra information reported on error
@@ -1071,7 +1071,7 @@ static inline bool __cpu_valid(s32 cpu)* Verify that it is in range and one of the possible cpus. If invalid, trigger * an ops error. */ -static bool ops_cpu_valid (struct scx_sched *sch,s32 cpu,const char *where) +bool scx_cpu_valid (struct scx_sched *sch,s32 cpu,const char *where) {if (__cpu_valid (cpu)) {return true;
@@ -1686,7 +1686,7 @@ static struct scx_dispatch_q *find_dsq_for_dispatch(struct scx_sched *sch,if ((dsq_id &SCX_DSQ_LOCAL_ON ) ==SCX_DSQ_LOCAL_ON ) {s32 cpu = dsq_id &SCX_DSQ_LOCAL_CPU_MASK ; -if (!ops_cpu_valid (sch, cpu,"in SCX_DSQ_LOCAL_ON dispatch verdict" )) +if (!scx_cpu_valid (sch, cpu,"in SCX_DSQ_LOCAL_ON dispatch verdict" ))return find_global_dsq (sch, tcpu);return &cpu_rq (cpu)->scx.local_dsq;
@@ -3269,7 +3269,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flagthis_rq ()->scx.in_select_cpu = false; p->scx.selected_cpu = cpu; *ddsp_taskp =NULL ; -if (ops_cpu_valid (sch, cpu,"from ops.select_cpu()" )) +if (scx_cpu_valid (sch, cpu,"from ops.select_cpu()" ))return cpu;else return prev_cpu;
@@ -8791,7 +8791,7 @@ static void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags)struct rq *this_rq;unsigned long irq_flags; -if (!ops_cpu_valid (sch, cpu,NULL )) +if (!scx_cpu_valid (sch, cpu,NULL ))return ;local_irq_save (irq_flags);
@@ -8888,7 +8888,7 @@ __bpf_kfunc s32 scx_bpf_dsq_nr_queued(u64 dsq_id, const struct bpf_prog_aux *aux }else if ((dsq_id &SCX_DSQ_LOCAL_ON ) ==SCX_DSQ_LOCAL_ON ) {s32 cpu = dsq_id &SCX_DSQ_LOCAL_CPU_MASK ; -if (ops_cpu_valid (sch, cpu,NULL )) { +if (scx_cpu_valid (sch, cpu,NULL )) { ret =READ_ONCE (cpu_rq (cpu)->scx.local_dsq.nr); goto out; }
@@ -9277,7 +9277,7 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cap(s32 cpu, const struct bpf_prog_aux *aux)guard (rcu)(); sch =scx_prog_sched (aux); -if (likely (sch) &&ops_cpu_valid (sch, cpu,NULL )) +if (likely (sch) &&scx_cpu_valid (sch, cpu,NULL ))return arch_scale_cpu_capacity (cpu);else return SCX_CPUPERF_ONE ;
@@ -9305,7 +9305,7 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cur(s32 cpu, const struct bpf_prog_aux *aux)guard (rcu)(); sch =scx_prog_sched (aux); -if (likely (sch) &&ops_cpu_valid (sch, cpu,NULL )) +if (likely (sch) &&scx_cpu_valid (sch, cpu,NULL ))return arch_scale_freq_capacity (cpu);else return SCX_CPUPERF_ONE ;
@@ -9341,7 +9341,7 @@ __bpf_kfunc void scx_bpf_cpuperf_set(s32 cpu, u32 perf, const struct bpf_prog_aureturn ; } -if (ops_cpu_valid (sch, cpu,NULL )) { +if (scx_cpu_valid (sch, cpu,NULL )) {struct rq *rq =cpu_rq (cpu), *locked_rq =scx_locked_rq ();struct rq_flags rf;
@@ -9454,7 +9454,7 @@ __bpf_kfunc struct rq *scx_bpf_cpu_rq(s32 cpu, const struct bpf_prog_aux *aux)if (unlikely (!sch))return NULL ; -if (!ops_cpu_valid (sch, cpu,NULL )) +if (!scx_cpu_valid (sch, cpu,NULL ))return NULL ;if (!sch->warned_deprecated_rq) {
@@ -9511,7 +9511,7 @@ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_if (unlikely (!sch))return NULL ; -if (!ops_cpu_valid (sch, cpu,NULL )) +if (!scx_cpu_valid (sch, cpu,NULL ))return NULL ;return rcu_dereference (cpu_rq (cpu)->curr);
kernel/sched/ext_idle.c +3 -3
@@ -916,7 +916,7 @@ static s32 select_cpu_from_kfunc(struct scx_sched *sch, struct task_struct *p,bool we_locked = false;s32 cpu; -if (!ops_cpu_valid (sch, prev_cpu,NULL )) +if (!scx_cpu_valid (sch, prev_cpu,NULL ))return -EINVAL ;if (!check_builtin_idle_enabled (sch))
@@ -989,7 +989,7 @@ __bpf_kfunc s32 scx_bpf_cpu_node(s32 cpu, const struct bpf_prog_aux *aux)guard (rcu)(); sch =scx_prog_sched (aux); -if (unlikely (!sch) || !ops_cpu_valid (sch, cpu,NULL )) +if (unlikely (!sch) || !scx_cpu_valid (sch, cpu,NULL ))return NUMA_NO_NODE ;return cpu_to_node (cpu); }
@@ -1271,7 +1271,7 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu, const struct bpf_prog_if (!check_builtin_idle_enabled (sch))return false; -if (!ops_cpu_valid (sch, cpu,NULL )) +if (!scx_cpu_valid (sch, cpu,NULL ))return false;return scx_idle_test_and_clear_cpu (cpu);
kernel/sched/ext_internal.h +2 -0
@@ -1352,6 +1352,8 @@ DECLARE_PER_CPU(struct rq *, scx_locked_rq_state);int scx_kfunc_context_filter (const struct bpf_prog *prog,u32 kfunc_id); +bool scx_cpu_valid (struct scx_sched *sch,s32 cpu,const char *where); +/* * Return the rq currently locked from an scx callback, or NULL if no rq is * locked.
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 3/17 sched_ext: Move scx_exit(), scx_error() and friends to ext_internal.h
In reply to Tejun HeoThings shared across multiple .c files belong in a header. scx_exit() / scx_error() (and their scx_vexit() / scx_verror() siblings) are already called from ext_idle.c and the upcoming ext_cid.c, and it was only build_policy.c's textual inclusion of ext.c that made the references resolve. Move the whole family to ext_internal.h.
Pure visibility change.
--- kernel/sched/ext.c | 13 ++++--------- kernel/sched/ext_internal.h | 8 ++++++++ 2 files changed, 12 insertions(+), 9 deletions(-)
kernel/sched/ext.c +4 -9
@@ -231,12 +231,10 @@ static void run_deferred(struct rq *rq);static bool task_dead_and_done (struct task_struct *p);static void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags);static void scx_disable (struct scx_sched *sch,enum scx_exit_kind kind); -static bool scx_vexit (struct scx_sched *sch,enum scx_exit_kind kind, -s64 exit_code,const char *fmt,va_list args); -static __printf(4, 5) bool scx_exit(struct scx_sched *sch, - enum scx_exit_kind kind, s64 exit_code, - const char *fmt, ...) +__printf(4, 5) bool scx_exit(struct scx_sched *sch, +enum scx_exit_kind kind,s64 exit_code, +const char *fmt, ...) {va_list args;bool ret;
@@ -248,9 +246,6 @@ static __printf(4, 5) bool scx_exit(struct scx_sched *sch,return ret; } -#define scx_error(sch, fmt, args...) scx_exit((sch), SCX_EXIT_ERROR, 0, fmt, ##args) -#define scx_verror (sch, fmt, args) scx_vexit((sch), SCX_EXIT_ERROR, 0, fmt, args) -#define SCX_HAS_OP (sch, op) test_bit(SCX_OP_IDX(op), (sch)->has_op)static long jiffies_delta_msecs (unsigned long at,unsigned long now)
@@ -6432,7 +6427,7 @@ static void scx_disable_irq_workfn(struct irq_work *irq_work)kthread_queue_work (sch->helper, &sch->disable_work); } -static bool scx_vexit (struct scx_sched *sch, +bool scx_vexit (struct scx_sched *sch,enum scx_exit_kind kind,s64 exit_code,const char *fmt,va_list args) {
kernel/sched/ext_internal.h +8 -0
@@ -1354,6 +1354,14 @@ int scx_kfunc_context_filter(const struct bpf_prog *prog, u32 kfunc_id);bool scx_cpu_valid (struct scx_sched *sch,s32 cpu,const char *where); +bool scx_vexit (struct scx_sched *sch,enum scx_exit_kind kind,s64 exit_code, +const char *fmt,va_list args); +__printf(4, 5) bool scx_exit(struct scx_sched *sch, enum scx_exit_kind kind, +s64 exit_code,const char *fmt, ...); + +#define scx_verror (sch, fmt, args) scx_vexit((sch), SCX_EXIT_ERROR, 0, fmt, args) +#define scx_error(sch, fmt, args...) scx_exit((sch), SCX_EXIT_ERROR, 0, fmt, ##args) +/* * Return the rq currently locked from an scx callback, or NULL if no rq is * locked.
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 4/17 sched_ext: Shift scx_kick_cpu() validity check to scx_bpf_kick_cpu()
In reply to Tejun HeoCallers that already know the cpu is valid shouldn't have to pay for a redundant check. scx_kick_cpu() is called from the in-kernel balance loop break-out path with the current cpu (trivially valid) and from scx_bpf_kick_cpu() with a BPF-supplied cpu that does need validation. Move the check out of scx_kick_cpu() into scx_bpf_kick_cpu() so the backend is reusable by callers that have already validated.
--- kernel/sched/ext.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
kernel/sched/ext.c +1 -4
@@ -8786,9 +8786,6 @@ static void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags)struct rq *this_rq;unsigned long irq_flags; -if (!scx_cpu_valid (sch, cpu,NULL )) -return ; -local_irq_save (irq_flags); this_rq =this_rq ();
@@ -8851,7 +8848,7 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags, const struct bpf_prog_auxguard (rcu)(); sch =scx_prog_sched (aux); -if (likely (sch)) +if (likely (sch) &&scx_cpu_valid (sch, cpu,NULL ))scx_kick_cpu (sch, cpu, flags); }
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 5/17 sched_ext: Relocate cpu_acquire/cpu_release to end of struct sched_ext_ops
In reply to Tejun Heocpu_acquire and cpu_release are deprecated and slated for removal. Move their declarations to the end of struct sched_ext_ops so an upcoming cid-form struct (sched_ext_ops_cid) can omit them entirely without disturbing the offsets of the shared fields.
Switch the two SCX_HAS_OP() callers for these ops to direct field checks since the relocated ops sit outside the SCX_OPI_END range covered by the has_op bitmap.
scx_kf_allow_flags[] auto-sizes to the highest used SCX_OP_IDX, so SCX_OP_IDX(cpu_release) moving to a higher index just enlarges the sparse array; the lookup logic is unchanged.
--- kernel/sched/ext.c | 4 +-- kernel/sched/ext_internal.h | 54 ++++++++++++++++++++++--------------- 2 files changed, 34 insertions(+), 24 deletions(-)
kernel/sched/ext.c +2 -2
@@ -2823,7 +2823,7 @@ static int balance_one(struct rq *rq, struct task_struct *prev)* core. This callback complements ->cpu_release(), which is * emitted in switch_class(). */ -if (SCX_HAS_OP (sch, cpu_acquire)) +if (sch->ops.cpu_acquire)SCX_CALL_OP (sch, cpu_acquire, rq, cpu,NULL ); rq->scx.cpu_released = false; }
@@ -2969,7 +2969,7 @@ static void switch_class(struct rq *rq, struct task_struct *next)* next time that balance_one() is invoked. */ if (!rq->scx.cpu_released) { -if (SCX_HAS_OP (sch, cpu_release)) { +if (sch->ops.cpu_release) {struct scx_cpu_release_args args = { .reason =preempt_reason_from_class (next_class), .task = next,
kernel/sched/ext_internal.h +32 -22
66 lines not on this page · raw message
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 6/17 sched_ext: Make scx_enable() take scx_enable_cmd
In reply to Tejun HeoPass struct scx_enable_cmd to scx_enable() rather than unpacking @ops at every call site and re-packing into a fresh cmd inside. bpf_scx_reg() now builds the cmd on its stack and hands it in; scx_enable() just wires up the kthread work and waits.
Relocate struct scx_enable_cmd above scx_alloc_and_add_sched() so upcoming patches that also want the cmd can see it.
No behavior change.
--- kernel/sched/ext.c | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-)
kernel/sched/ext.c +23 -23
@@ -6507,6 +6507,19 @@ static struct scx_sched_pnode *alloc_pnode(struct scx_sched *sch, int node)return pnode; } +/* +* scx_enable() is offloaded to a dedicated system-wide RT kthread to avoid +* starvation. During the READY -> ENABLED task switching loop, the calling +* thread's sched_class gets switched from fair to ext. As fair has higher +* priority than ext, the calling thread can be indefinitely starved under +* fair-class saturation, leading to a system hang. +*/ +struct scx_enable_cmd { +struct kthread_work work; +struct sched_ext_ops *ops; +int ret; +}; +/* * Allocate and initialize a new scx_sched. @cgrp's reference is always * consumed whether the function succeeds or fails.
@@ -6749,19 +6762,6 @@ static int validate_ops(struct scx_sched *sch, const struct sched_ext_ops *ops)return 0 ; } -/* -* scx_enable() is offloaded to a dedicated system-wide RT kthread to avoid -* starvation. During the READY -> ENABLED task switching loop, the calling -* thread's sched_class gets switched from fair to ext. As fair has higher -* priority than ext, the calling thread can be indefinitely starved under -* fair-class saturation, leading to a system hang. -*/ -struct scx_enable_cmd { -struct kthread_work work; -struct sched_ext_ops *ops; -int ret; -}; -static void scx_root_enable_workfn (struct kthread_work *work) { struct scx_enable_cmd *cmd = container_of(work, struct scx_enable_cmd, work);
@@ -7346,11 +7346,10 @@ static s32 __init scx_cgroup_lifetime_notifier_init(void)core_initcall (scx_cgroup_lifetime_notifier_init);#endif /* CONFIG_EXT_SUB_SCHED */ -static s32 scx_enable (struct sched_ext_ops *ops,struct bpf_link *link) +static s32 scx_enable (struct scx_enable_cmd *cmd,struct bpf_link *link) {static struct kthread_worker *helper;static DEFINE_MUTEX (helper_mutex); -struct scx_enable_cmd cmd;if (!cpumask_equal (housekeeping_cpumask (HK_TYPE_DOMAIN ), cpu_possible_mask)) {
@@ -7374,16 +7373,15 @@ static s32 scx_enable(struct sched_ext_ops *ops, struct bpf_link *link) }#ifdef CONFIG_EXT_SUB_SCHED -if (ops->sub_cgroup_id >1 ) -kthread_init_work (&cmd.work, scx_sub_enable_workfn); +if (cmd->ops->sub_cgroup_id >1 ) +kthread_init_work (&cmd->work, scx_sub_enable_workfn); else#endif /* CONFIG_EXT_SUB_SCHED */ -kthread_init_work (&cmd.work, scx_root_enable_workfn); - cmd.ops = ops; +kthread_init_work (&cmd->work, scx_root_enable_workfn); -kthread_queue_work (READ_ONCE (helper), &cmd.work); -kthread_flush_work (&cmd.work); -return cmd.ret; +kthread_queue_work (READ_ONCE (helper), &cmd->work); +kthread_flush_work (&cmd->work); +return cmd->ret; }
@@ -7555,7 +7553,9 @@ static int bpf_scx_check_member(const struct btf_type *t,static int bpf_scx_reg (void *kdata,struct bpf_link *link) { -return scx_enable (kdata, link); +struct scx_enable_cmd cmd = { .ops = kdata }; + +return scx_enable (&cmd, link); }static void bpf_scx_unreg (void *kdata,struct bpf_link *link)
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 7/17 sched_ext: Add topological CPU IDs (cids)
In reply to Tejun HeoRaw cpu numbers are clumsy for sharding and cross-sched communication, especially from BPF. The space is sparse, numerical closeness doesn't track topological closeness (x86 hyperthreading often scatters SMT siblings), and a range of cpu ids doesn't describe anything meaningful. Sub-sched support makes this acute: cpu allocation, revocation, and state constantly flow across sub-scheds. Passing whole cpumasks scales poorly (every op scans 4K bits) and cpumasks are awkward in BPF.
cids assign every cpu a dense, topology-ordered id. CPUs sharing a core, LLC, or NUMA node occupy contiguous cid ranges, so a topology unit becomes a (start, length) slice. Communication passes slices; BPF can process a u64 word of cids at a time.
Build the mapping once at root enable by walking online cpus node -> LLC -> core. Possible-but-not-online cpus tail the space with no-topo cids. Expose kfuncs to map cpu <-> cid in either direction and to query each cid's topology metadata.
v2: Use kzalloc_objs()/kmalloc_objs() for the three allocs in
scx_cid_arrays_alloc() (Cheng-Yang Chou).
v3: scx_cid_init() failure path now drops cpus_read_lock();
BUILD_BUG_ON tightened to match BPF cmask helpers' NR_CPUS<=8192. (Sashiko)
--- kernel/sched/build_policy.c | 2 + kernel/sched/ext.c | 18 ++ kernel/sched/ext_cid.c | 301 +++++++++++++++++++++++ kernel/sched/ext_cid.h | 129 ++++++++++ kernel/sched/ext_types.h | 23 ++ tools/sched_ext/include/scx/common.bpf.h | 3 + 6 files changed, 476 insertions(+) create mode 100644 kernel/sched/ext_cid.c create mode 100644 kernel/sched/ext_cid.h
kernel/sched/build_policy.c +2 -0
@@ -61,8 +61,10 @@# include <linux/btf_ids.h> # include "ext_types.h" # include "ext_internal.h" +# include "ext_cid.h" # include "ext_idle.h" # include "ext.c" +# include "ext_cid.c" # include "ext_idle.c" #endif
kernel/sched/ext.c +18 -0
@@ -6820,6 +6820,18 @@ static void scx_root_enable_workfn(struct kthread_work *work)*/ cpus_read_lock (); +/* +* Build the cid mapping before publishing scx_root. The cid kfuncs +* dereference the cid arrays unconditionally once scx_prog_sched() +* returns non-NULL; the rcu_assign_pointer() below pairs with their +* rcu_dereference() to make the populated arrays visible. +*/ + ret =scx_cid_init (sch); +if (ret) { +cpus_read_unlock (); + goto err_disable; + } +/* * Make the scheduler instance visible. Must be inside cpus_read_lock(). * See handle_hotplug().
@@ -9888,6 +9900,12 @@ static int __init scx_init(void)return ret; } + ret =scx_cid_kfunc_init (); +if (ret) { +pr_err ("sched_ext: Failed to register cid kfuncs (%d)\n" , ret); +return ret; + } + ret =register_bpf_struct_ops (&bpf_sched_ext_ops, sched_ext_ops);if (ret) {pr_err ("sched_ext: Failed to register struct_ops (%d)\n" , ret);
kernel/sched/ext_cid.c new file +301 -0
301 lines not on this page · raw message
kernel/sched/ext_cid.h new file +129 -0
129 lines not on this page · raw message
kernel/sched/ext_types.h +23 -0
@@ -40,4 +40,27 @@ enum scx_consts { SCX_SUB_MAX_DEPTH = 4, }; +/* +* Per-cid topology info. For each topology level (core, LLC, node), records +* the first cid in the unit and its global index. Global indices are +* consecutive integers assigned in cid-walk order, so e.g. core_idx ranges +* over [0, nr_cores_at_init) with no gaps. No-topo cids have all fields set +* to -1. +* +* @core_cid: first cid of this cid's core (smt-sibling group) +* @core_idx: global index of that core, in [0, nr_cores_at_init) +* @llc_cid: first cid of this cid's LLC +* @llc_idx: global index of that LLC, in [0, nr_llcs_at_init) +* @node_cid: first cid of this cid's NUMA node +* @node_idx: global index of that node, in [0, nr_nodes_at_init) +*/ +struct scx_cid_topo { +s32 core_cid; +s32 core_idx; +s32 llc_cid; +s32 llc_idx; +s32 node_cid; +s32 node_idx; +}; +#endif /* _KERNEL_SCHED_EXT_TYPES_H */
tools/sched_ext/include/scx/common.bpf.h +3 -0
@@ -102,6 +102,9 @@ struct task_struct *scx_bpf_cpu_curr(s32 cpu) __ksym __weak;struct task_struct *scx_bpf_tid_to_task (u64 tid) __ksym__weak ; u64 scx_bpf_now(void) __ksym __weak; void scx_bpf_events(struct scx_event_stats *events, size_t events__sz) __ksym __weak; +s32 scx_bpf_cpu_to_cid(s32 cpu) __ksym __weak; +s32 scx_bpf_cid_to_cpu(s32 cid) __ksym __weak; +void scx_bpf_cid_topo(s32 cid, struct scx_cid_topo *out) __ksym __weak;/* * Use the following as @it__iter when calling scx_bpf_dsq_move[_vtime]() from
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 8/17 sched_ext: Add scx_bpf_cid_override() kfunc
In reply to Tejun HeoThe auto-probed cid mapping reflects the kernel's view of topology (node -> LLC -> core), but a BPF scheduler may want a different layout - to align cid slices with its own partitioning, or to work around how the kernel reports a particular machine.
Add scx_bpf_cid_override(), callable from ops.init() of the root scheduler. It validates the caller-supplied cpu->cid array and replaces the in-place mapping; topo info is invalidated. A compat.bpf.h wrapper silently no-ops on kernels that lack the kfunc.
A new SCX_KF_ALLOW_INIT bit in the kfunc context filter restricts the kfunc to ops.init() at verifier load time.
--- kernel/sched/ext.c | 16 +++-- kernel/sched/ext_cid.c | 75 +++++++++++++++++++++++- kernel/sched/ext_cid.h | 1 + tools/sched_ext/include/scx/compat.bpf.h | 12 ++++ 4 files changed, 97 insertions(+), 7 deletions(-)
kernel/sched/ext.c +10 -6
@@ -9755,10 +9755,11 @@ static const struct btf_kfunc_id_set scx_kfunc_set_any = {*/ enum scx_kf_allow_flags {SCX_KF_ALLOW_UNLOCKED =1 <<0 , -SCX_KF_ALLOW_CPU_RELEASE =1 <<1 , -SCX_KF_ALLOW_DISPATCH =1 <<2 , -SCX_KF_ALLOW_ENQUEUE =1 <<3 , -SCX_KF_ALLOW_SELECT_CPU =1 <<4 , +SCX_KF_ALLOW_INIT =1 <<1 , +SCX_KF_ALLOW_CPU_RELEASE =1 <<2 , +SCX_KF_ALLOW_DISPATCH =1 <<3 , +SCX_KF_ALLOW_ENQUEUE =1 <<4 , +SCX_KF_ALLOW_SELECT_CPU =1 <<5 , };/*
@@ -9786,7 +9787,7 @@ static const u32 scx_kf_allow_flags[] = { [SCX_OP_IDX(sub_detach)] = SCX_KF_ALLOW_UNLOCKED, [SCX_OP_IDX(cpu_online)] = SCX_KF_ALLOW_UNLOCKED, [SCX_OP_IDX(cpu_offline)] = SCX_KF_ALLOW_UNLOCKED, - [SCX_OP_IDX(init)] = SCX_KF_ALLOW_UNLOCKED, + [SCX_OP_IDX(init)] = SCX_KF_ALLOW_UNLOCKED | SCX_KF_ALLOW_INIT, [SCX_OP_IDX(exit)] = SCX_KF_ALLOW_UNLOCKED, };
@@ -9801,6 +9802,7 @@ static const u32 scx_kf_allow_flags[] = {int scx_kfunc_context_filter (const struct bpf_prog *prog,u32 kfunc_id) {bool in_unlocked =btf_id_set8_contains (&scx_kfunc_ids_unlocked, kfunc_id); +bool in_init =btf_id_set8_contains (&scx_kfunc_ids_init, kfunc_id);bool in_select_cpu =btf_id_set8_contains (&scx_kfunc_ids_select_cpu, kfunc_id);bool in_enqueue =btf_id_set8_contains (&scx_kfunc_ids_enqueue_dispatch, kfunc_id);bool in_dispatch =btf_id_set8_contains (&scx_kfunc_ids_dispatch, kfunc_id);
@@ -9810,7 +9812,7 @@ int scx_kfunc_context_filter(const struct bpf_prog *prog, u32 kfunc_id)u32 moff, flags;/* Not an SCX kfunc - allow. */ -if (!(in_unlocked || in_select_cpu || in_enqueue || in_dispatch || +if (!(in_unlocked || in_init || in_select_cpu || in_enqueue || in_dispatch || in_cpu_release || in_idle || in_any))return 0 ;
@@ -9846,6 +9848,8 @@ int scx_kfunc_context_filter(const struct bpf_prog *prog, u32 kfunc_id)if ((flags &SCX_KF_ALLOW_UNLOCKED ) && in_unlocked)return 0 ; +if ((flags &SCX_KF_ALLOW_INIT ) && in_init) +return 0 ;if ((flags &SCX_KF_ALLOW_CPU_RELEASE ) && in_cpu_release)return 0 ;if ((flags &SCX_KF_ALLOW_DISPATCH ) && in_dispatch)
kernel/sched/ext_cid.c +74 -1
93 lines not on this page · raw message
kernel/sched/ext_cid.h +1 -0
@@ -49,6 +49,7 @@ struct scx_sched;extern s16 *scx_cid_to_cpu_tbl;extern s16 *scx_cpu_to_cid_tbl;extern struct scx_cid_topo *scx_cid_topo; +extern struct btf_id_set8 scx_kfunc_ids_init;s32 scx_cid_init (struct scx_sched *sch);int scx_cid_kfunc_init (void );
tools/sched_ext/include/scx/compat.bpf.h +12 -0
@@ -121,6 +121,18 @@ static inline bool scx_bpf_sub_dispatch(u64 cgroup_id)return false; } +/* +* v7.2: scx_bpf_cid_override() for explicit cpu->cid mapping. Ignore if +* missing. +*/ +void scx_bpf_cid_override___compat(const s32 *cpu_to_cid, u32 cpu_to_cid__sz) __ksym __weak; + +static inline void scx_bpf_cid_override (const s32 *cpu_to_cid,u32 cpu_to_cid__sz) +{ +if (bpf_ksym_exists (scx_bpf_cid_override___compat)) +return scx_bpf_cid_override___compat (cpu_to_cid, cpu_to_cid__sz); +} +/** * __COMPAT_is_enq_cpu_selected - Test if SCX_ENQ_CPU_SELECTED is on * in a compatible way. We will preserve this __COMPAT helper until v6.16.
Signature · 1 line
2.54.0
Andrea Righi
reviewerRe: [PATCH 08/17] sched_ext: Add scx_bpf_cid_override() kfunc
Hi Tejun,
Quoted 15 lines · Tejun Heo
The auto-probed cid mapping reflects the kernel's view of topology (node -> LLC -> core), but a BPF scheduler may want a different layout - to align cid slices with its own partitioning, or to work around how the kernel reports a particular machine.
Add
scx_bpf_cid_override(), callable fromops.init()of the root scheduler. It validates the caller-suppliedcpu->cidarray and replaces the in-place mapping; topo info is invalidated. Acompat.bpf.hwrapper silently no-ops on kernels that lack the kfunc.A new
SCX_KF_ALLOW_INITbit in the kfunc context filter restricts the kfunc toops.init()at verifier load time.Signed-off-by Tejun Heo <[email protected]>Reviewed-by Cheng-Yang Chou <[email protected]>
...
Quoted 50 lines
kernel/sched/ext_cid.c +50
-0+/** +* scx_bpf_cid_override - Install an explicit cpu->cid mapping +* @cpu_to_cid: array of nr_cpu_ids s32 entries (cid for each cpu) +* @cpu_to_cid__sz: must be nr_cpu_ids * sizeof(s32) bytes +* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs +* +* May only be called from ops.init() of the root scheduler. Replace the +* topology-probed cid mapping with the caller-provided one. Each possible cpu +* must map to a unique cid in [0, num_possible_cpus()). Topo info is cleared. +* On invalid input, trigger scx_error() to abort the scheduler. +*/ +__bpf_kfunc void scx_bpf_cid_override(const s32 *cpu_to_cid, u32 cpu_to_cid__sz, +const struct bpf_prog_aux *aux) +{ + cpumask_var_t seen __free(free_cpumask_var) = CPUMASK_VAR_NULL; +struct scx_sched *sch; +bool alloced; +s32 cpu, cid; + +/* GFP_KERNEL alloc must happen before the rcu read section */ + alloced =zalloc_cpumask_var (&seen,GFP_KERNEL ); + +guard (rcu)(); + + sch =scx_prog_sched (aux); +if (unlikely (!sch)) +return ; + +if (!alloced) { +scx_error (sch,"scx_bpf_cid_override: failed to allocate cpumask" ); +return ; + } + +if (scx_parent (sch)) { +scx_error (sch,"scx_bpf_cid_override() only allowed from root sched" ); +return ; + } + +if (cpu_to_cid__sz != nr_cpu_ids *sizeof (s32)) { +scx_error (sch,"scx_bpf_cid_override: expected %zu bytes, got %u" , + nr_cpu_ids *sizeof (s32), cpu_to_cid__sz); +return ; + } + +for_each_possible_cpu (cpu) { +s32 c = cpu_to_cid[cpu]; + +if (!cid_valid (sch, c)) +return ; +if (cpumask_test_and_set_cpu (c, seen)) {
kernel/sched/ext_cid.c s32 scx_cid_init(struct scx_sched *sch) · line 272
Quoted 10 lines
kernel/sched/ext_cid.c +10
-0+scx_error (sch,"cid %d assigned to multiple cpus" , c); +return ; + } + scx_cpu_to_cid_tbl[cpu] = c; + scx_cid_to_cpu_tbl[c] = cpu; + } + +/* Invalidate stale topo info - the override carries no topology. */ +for (cid =0 ; cid <num_possible_cpus (); cid++) + scx_cid_topo[cid] =SCX_CID_TOPO_NEG ;
Considering that the topology info is wiped when scx_bpf_cid_override() is used, should we error if a scheduler is also trying to use scx_bpf_cid_topo() (i.e., setting a flag or similar)?
Thanks,
-Andrea
Tejun Heo
reviewerRe: [PATCH 08/17] sched_ext: Add scx_bpf_cid_override() kfunc
Hello,
I don't think we need to gate it. The override clears scx_cid_topo[] to SCX_CID_TOPO_NEG, so subsequent scx_bpf_cid_topo() lookups return the well-defined "no topo" sentinel. The scheduler that overrode the mapping has already opted out of the auto-probed topology.
Thanks.
--
tejun
Andrea Righi
reviewerRe: [PATCH 08/17] sched_ext: Add scx_bpf_cid_override() kfunc
Quoted 6 lines · Tejun Heo
Hello,
I don't think we need to gate it. The override clears scx_cid_topo[] to
SCX_CID_TOPO_NEG, so subsequentscx_bpf_cid_topo()lookups return the well-defined "no topo" sentinel. The scheduler that overrode the mapping has already opted out of the auto-probed topology.
Ok, makes sense. At the end if a scheduler overrides the topology and then uses scx_bpf_cid_topo(), it's probably reasonable to return "no topology".
Thanks,
-Andrea
Tejun Heo
reviewerPATCH 9/17 tools/sched_ext: Add struct_size() helpers to common.bpf.h
In reply to Tejun HeoAdd flex_array_size(), struct_size() and struct_size_t() to scx/common.bpf.h so BPF schedulers can size flex-array-containing structs the same way kernel code does. These are abbreviated forms of the <linux/overflow.h> macros.
v3: Use offsetof() instead of sizeof() in struct_size() to match kernel
semantics (no inflation from trailing struct padding). (Sashiko)
--- tools/sched_ext/include/scx/common.bpf.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
tools/sched_ext/include/scx/common.bpf.h +10 -0
@@ -1043,6 +1043,16 @@ static inline u64 scx_clock_irq(u32 cpu)return irqt ?BPF_CORE_READ (irqt, total) :0 ; } +/* Abbreviated forms of <linux/overflow.h>'s struct_size() family. */ +#define flex_array_size (p, member, count) \ + ((count) * sizeof(*(p)->member)) + +#define struct_size (p, member, count) \ + (offsetof(typeof(*(p)), member) + flex_array_size(p, member, count)) + +#define struct_size_t (type, member, count) \ + struct_size((type *)NULL, member, count) +#include "compat.bpf.h" #include "enums.bpf.h"
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 10/17 sched_ext: Add cmask, a base-windowed bitmap over cid space
In reply to Tejun HeoSub-scheduler code built on cids needs bitmaps scoped to a slice of cid space (e.g. the idle cids of a shard). A cpumask sized for NR_CPUS wastes most of its bits for a small window and is awkward in BPF.
scx_cmask covers [base, base + nr_bits). bits[] is aligned to the global 64-cid grid: bits[0] spans [base & ~63, (base & ~63) + 64). Any two cmasks therefore address bits[] against the same global windows, so cross-cmask word ops reduce to
dest->bits[i] OP= operand->bits[i - delta]
with no bit-shifting, at the cost of up to one extra storage word for head misalignment. This alignment guarantee is the reason binary ops can stay word-level; every mutating helper preserves it.
Kernel side in ext_cid.[hc]; BPF side in tools/sched_ext/include/scx/ cid.bpf.h. BPF side drops the scx_ prefix (redundant in BPF code) and adds the extra helpers that basic idle-cpu selection needs.
No callers yet.
v2: Narrow to helpers that will be used in the planned changes;
set/bit/find/zero ops will be added as usage develops.
v3: cmask_copy_from_kernel: validate src->base == 0 via probe-read;
bit-level nr_bits check instead of round-up word count. (Sashiko)
--- kernel/sched/ext_cid.h | 25 + kernel/sched/ext_types.h | 38 ++ tools/sched_ext/include/scx/cid.bpf.h | 667 +++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 1 + 4 files changed, 731 insertions(+) create mode 100644 tools/sched_ext/include/scx/cid.bpf.h
kernel/sched/ext_cid.h +25 -0
@@ -127,4 +127,29 @@ static inline s32 scx_cpu_to_cid(struct scx_sched *sch, s32 cpu)return __scx_cpu_to_cid (cpu); } +static inline bool __scx_cmask_contains (const struct scx_cmask *m,u32 cid) +{ +return likely (cid >= m->base && cid < m->base + m->nr_bits); +} + +/* Word in bits[] covering @cid. @cid must satisfy __scx_cmask_contains(). */ +static inline u64 *__scx_cmask_word (const struct scx_cmask *m,u32 cid) +{ +return (u64 *)&m->bits[cid /64 - m->base /64 ]; +} + +static inline void scx_cmask_init (struct scx_cmask *m,u32 base,u32 nr_bits) +{ + m->base = base; + m->nr_bits = nr_bits; +memset (m->bits,0 ,SCX_CMASK_NR_WORDS (nr_bits) *sizeof (u64)); +} + +static inline void __scx_cmask_set (struct scx_cmask *m,u32 cid) +{ +if (!__scx_cmask_contains (m, cid)) +return ; + *__scx_cmask_word (m, cid) |=BIT_U64 (cid &63 ); +} +#endif /* _KERNEL_SCHED_EXT_CID_H */
kernel/sched/ext_types.h +38 -0
@@ -63,4 +63,42 @@ struct scx_cid_topo {s32 node_idx; }; +/* +* cmask: variable-length, base-windowed bitmap over cid space +* ----------------------------------------------------------- +* +* A cmask covers the cid range [base, base + nr_bits). bits[] is aligned to the +* global 64-cid grid: bits[0] spans [base & ~63, (base & ~63) + 64), so the +* first (base & 63) bits of bits[0] are head padding and any tail past base + +* nr_bits is tail padding. Both must stay zero for the lifetime of the mask; +* all mutating helpers preserve that invariant. +* +* Grid alignment means two cmasks always address bits[] against the same global +* 64-cid windows, so cross-cmask word ops (AND, OR, ...) reduce to +* +* dst->bits[i] OP= src->bits[i - delta] +* +* with no bit-shifting, regardless of how the two bases relate mod 64. +*/ +struct scx_cmask { +u32 base; +u32 nr_bits; + DECLARE_FLEX_ARRAY(u64, bits); +}; + +/* +* Number of u64 words of bits[] storage that covers @nr_bits regardless of base +* alignment. The +1 absorbs up to 63 bits of head padding when base is not +* 64-aligned - always allocating one extra word beats branching on base or +* splitting the compute. +*/ +#define SCX_CMASK_NR_WORDS (nr_bits) (((nr_bits) + 63) / 64 + 1) + +/* +* Define an on-stack cmask for up to @cap_bits. @name is a struct scx_cmask * +* aliasing zero-initialized storage; call scx_cmask_init() to set base/nr_bits. +*/ +#define SCX_CMASK_DEFINE (name, cap_bits) \ + DEFINE_RAW_FLEX(struct scx_cmask, name, bits, SCX_CMASK_NR_WORDS(cap_bits)) +#endif /* _KERNEL_SCHED_EXT_TYPES_H */
tools/sched_ext/include/scx/cid.bpf.h new file +667 -0
667 lines not on this page · raw message
tools/sched_ext/include/scx/common.bpf.h +1 -0
@@ -1055,5 +1055,6 @@ static inline u64 scx_clock_irq(u32 cpu)#include "compat.bpf.h" #include "enums.bpf.h" +#include "cid.bpf.h" #endif /* __SCX_COMMON_BPF_H */
Signature · 1 line
2.54.0
Re: [PATCH 10/17] sched_ext: Add cmask, a base-windowed bitmap over cid space
Quoted 87 lines · Tejun Heo
tools/sched_ext/include/scx/cid.bpf.h new file +87
-0+/* +* x86 BPF JIT rejects BPF_OR | BPF_FETCH and BPF_AND | BPF_FETCH on arena +* pointers (see bpf_jit_supports_insn() in arch/x86/net/bpf_jit_comp.c). Only +* BPF_CMPXCHG / BPF_XCHG / BPF_ADD with FETCH are allowed. Implement +* test_and_{set,clear} and the atomic set/clear via a cmpxchg loop. +* +* CMASK_CAS_TRIES is far above what any non-pathological contention needs. +* Exhausting it means the bit update was lost, which corrupts the caller's view +* of the bitmap, so raise scx_bpf_error() to abort the scheduler. +*/ +#define CMASK_CAS_TRIES 1024 + +static __always_inline void cmask_set(struct scx_cmask __arena *m, u32 cid) +{ + u64 __arena *w; +u64 bit, old, new; +u32 i; + +if (!__cmask_contains (m, cid)) +return ; + w =__cmask_word (m, cid); + bit =BIT_U64 (cid &63 ); + bpf_for(i, 0, CMASK_CAS_TRIES) { + old = *w; +if (old & bit) +return ; + new = old | bit; +if (__sync_val_compare_and_swap (w, old, new) == old) +return ; + } +scx_bpf_error ("cmask_set CAS exhausted at cid %u" , cid); +} + +static __always_inline void cmask_clear(struct scx_cmask __arena *m, u32 cid) +{ + u64 __arena *w; +u64 bit, old, new; +u32 i; + +if (!__cmask_contains (m, cid)) +return ; + w =__cmask_word (m, cid); + bit =BIT_U64 (cid &63 ); + bpf_for(i, 0, CMASK_CAS_TRIES) { + old = *w; +if (!(old & bit)) +return ; + new = old & ~bit; +if (__sync_val_compare_and_swap (w, old, new) == old) +return ; + } +scx_bpf_error ("cmask_clear CAS exhausted at cid %u" , cid); +} + +static __always_inline bool cmask_test_and_set(struct scx_cmask __arena *m, u32 cid) +{ + u64 __arena *w; +u64 bit, old, new; +u32 i; + +if (!__cmask_contains (m, cid)) +return false; + w =__cmask_word (m, cid); + bit =BIT_U64 (cid &63 ); + bpf_for(i, 0, CMASK_CAS_TRIES) { + old = *w; +if (old & bit) +return true; + new = old | bit; +if (__sync_val_compare_and_swap (w, old, new) == old) +return false; + } +scx_bpf_error ("cmask_test_and_set CAS exhausted at cid %u" , cid); +return false; +} + +static __always_inline bool cmask_test_and_clear(struct scx_cmask __arena *m, u32 cid) +{ + u64 __arena *w; +u64 bit, old, new; +u32 i; + +if (!__cmask_contains (m, cid)) +return false; + w =__cmask_word (m, cid); + bit =BIT_U64 (cid &63 ); + bpf_for(i, 0, CMASK_CAS_TRIES) {
tools/sched_ext/include/scx/cid.bpf.h line 164
Quoted 10 lines
tools/sched_ext/include/scx/cid.bpf.h new file +10
-0+ old = *w; +if (!(old & bit)) +return false; + new = old & ~bit; +if (__sync_val_compare_and_swap (w, old, new) == old) +return true; + } +scx_bpf_error ("cmask_test_and_clear CAS exhausted at cid %u" , cid); +return false; +}
Exiting a BPF scheduler when CAS retries fail seems too brutal, while it is extremely rare to happen. What about adding a kfunc for the slow path that runs only when the CAS retry fails? For example, scx_bpf_cmask_test_and_clear( ) does the same thing as cmask_test_and_clear(), but it never fails. cmask_test_and_clear() calls scx_bpf_cmask_test_and_clear( ) only when the CAS retry fails. In this way, we can use the fast path in the BPF implementation while ensuring the operation succeeds.
Quoted 29 lines
tools/sched_ext/include/scx/cid.bpf.h new file +29
-0+/** +* cmask_next_set - find the first set bit at or after @cid +* @m: cmask to search +* @cid: starting cid (clamped to @m->base if below) +* +* Returns the smallest set cid in [@cid, @m->base + @m->nr_bits), or +* @m->base + @m->nr_bits if none (the out-of-range sentinel matches the +* termination condition used by cmask_for_each()). +*/ +static __always_inline u32 cmask_next_set(const struct scx_cmask __arena *m, u32 cid) +{ +u32 end = m->base + m->nr_bits; +u32 base = m->base /64 ; +u32 last_wi = (end -1 ) /64 - base; +u32 start_wi, start_bit, i; + +if (cid < m->base) + cid = m->base; +if (cid >= end) +return end; + + start_wi = cid /64 - base; + start_bit = cid &63 ; + + bpf_for(i, 0, CMASK_MAX_WORDS) { +u32 wi = start_wi + i; +u64 word; +u32 found; +
tools/sched_ext/include/scx/cid.bpf.h line 410
Quoted 10 lines
tools/sched_ext/include/scx/cid.bpf.h new file +10
-0+if (wi > last_wi) +break ; + + word = m->bits[wi]; +if (i ==0 ) + word &=GENMASK_U64 (63 , start_bit); +if (!word) +continue ; + + found = (base + wi) *64 +__builtin_ctzll (word);
Some compiler versions (e.g., clang-18 or older) don’t support __builtin_ctzll(). To handle this gracefully, there is already a wrapper, ctzll(), in common.bpf.h. So, I suggest using ctzll() for compatibility.
Tejun Heo
reviewerRe: [PATCH 10/17] sched_ext: Add cmask, a base-windowed bitmap over cid space
Hello,
I'd rather raise the loop count than punt to a kernel slow-path. On multi-socket Sapphire Rapids, banging on a shared cacheline with kernel atomics hard enough can stall the machine to the point of hard lockups. We don't know what BPF will do with these helpers and they can pretty easily trigger similar conditions, so giving them the ability to loop indefinitely in the kernel is exactly what we want to avoid - we want to fail hard when this happens.
Bumping CMASK_CAS_TRIES to 1<<23 in v4 so abort fires only after seconds of real spinning. As a follow-up, I want to add a kfunc to let the BPF loops bail immediately when sch->aborting is set, so the abort path doesn't keep banging the cacheline while the kernel is tearing the scheduler down.
Switching to ctzll() too, thanks.
Thanks.
--
tejun
Tejun Heo
reviewerPATCH 11/17 sched_ext: Add cid-form kfunc wrappers alongside cpu-form
In reply to Tejun Heocpumask is awkward from BPF and unusable from arena; cid/cmask work in both. Sub-sched enqueue will need cmask. Without full cid coverage a scheduler has to mix cid and cpu forms, which is a subtle-bug factory. Close the gap with a cid-native interface.
Pair every cpu-form kfunc that takes a cpu id with a cid-form equivalent (kick, task placement, cpuperf query/set, per-cpu current task, nr-cpu-ids). Add two cid-natives with no cpu-form sibling:scx_bpf_this_cid() (cid of the running cpu, scx equivalent of bpf_get_smp_processor_id) and scx_bpf_nr_online_cids().
scx_bpf_cpu_rq is deprecated; no cid-form counterpart. NUMA node info is reachable via scx_bpf_cid_topo() on the BPF side.
Each cid-form wrapper is a thin cid -> cpu translation that delegates to the cpu path, registered in the same context sets so usage constraints match.
--- kernel/sched/ext.c | 184 +++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 9 ++ 2 files changed, 193 insertions(+)
kernel/sched/ext.c +184 -0
@@ -8864,6 +8864,28 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags, const struct bpf_prog_auxscx_kick_cpu (sch, cpu, flags); } +/** +* scx_bpf_kick_cid - Trigger reschedule on the CPU mapped to @cid +* @cid: cid to kick +* @flags: %SCX_KICK_* flags +* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs +* +* cid-addressed equivalent of scx_bpf_kick_cpu(). +*/ +__bpf_kfunc void scx_bpf_kick_cid(s32 cid, u64 flags, const struct bpf_prog_aux *aux) +{ +struct scx_sched *sch; +s32 cpu; + +guard (rcu)(); + sch =scx_prog_sched (aux); +if (unlikely (!sch)) +return ; + cpu =scx_cid_to_cpu (sch, cid); +if (cpu >=0 ) +scx_kick_cpu (sch, cpu, flags); +} +/** * scx_bpf_dsq_nr_queued - Return the number of queued tasks * @dsq_id: id of the DSQ
@@ -9287,6 +9309,29 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cap(s32 cpu, const struct bpf_prog_aux *aux)return SCX_CPUPERF_ONE ; } +/** +* scx_bpf_cidperf_cap - Query the maximum relative capacity of the CPU at @cid +* @cid: cid of the CPU to query +* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs +* +* cid-addressed equivalent of scx_bpf_cpuperf_cap(). +*/ +__bpf_kfunc u32 scx_bpf_cidperf_cap(s32 cid, const struct bpf_prog_aux *aux) +{ +struct scx_sched *sch; +s32 cpu; + +guard (rcu)(); + + sch =scx_prog_sched (aux); +if (unlikely (!sch)) +return SCX_CPUPERF_ONE ; + cpu =scx_cid_to_cpu (sch, cid); +if (cpu <0 ) +return SCX_CPUPERF_ONE ; +return arch_scale_cpu_capacity (cpu); +} +/** * scx_bpf_cpuperf_cur - Query the current relative performance of a CPU * @cpu: CPU of interest
@@ -9315,6 +9360,29 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cur(s32 cpu, const struct bpf_prog_aux *aux)return SCX_CPUPERF_ONE ; } +/** +* scx_bpf_cidperf_cur - Query the current performance of the CPU at @cid +* @cid: cid of the CPU to query +* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs +* +* cid-addressed equivalent of scx_bpf_cpuperf_cur(). +*/ +__bpf_kfunc u32 scx_bpf_cidperf_cur(s32 cid, const struct bpf_prog_aux *aux) +{ +struct scx_sched *sch; +s32 cpu; + +guard (rcu)(); + + sch =scx_prog_sched (aux); +if (unlikely (!sch)) +return SCX_CPUPERF_ONE ; + cpu =scx_cid_to_cpu (sch, cid); +if (cpu <0 ) +return SCX_CPUPERF_ONE ; +return arch_scale_freq_capacity (cpu); +} +/** * scx_bpf_cpuperf_set - Set the relative performance target of a CPU * @cpu: CPU of interest
@@ -9375,6 +9443,31 @@ __bpf_kfunc void scx_bpf_cpuperf_set(s32 cpu, u32 perf, const struct bpf_prog_au } } +/** +* scx_bpf_cidperf_set - Set the performance target of the CPU at @cid +* @cid: cid of the CPU to target +* @perf: target performance level [0, %SCX_CPUPERF_ONE] +* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs +* +* cid-addressed equivalent of scx_bpf_cpuperf_set(). +*/ +__bpf_kfunc void scx_bpf_cidperf_set(s32 cid, u32 perf, +const struct bpf_prog_aux *aux) +{ +struct scx_sched *sch; +s32 cpu; + +guard (rcu)(); + + sch =scx_prog_sched (aux); +if (unlikely (!sch)) +return ; + cpu =scx_cid_to_cpu (sch, cid); +if (cpu <0 ) +return ; +scx_bpf_cpuperf_set (cpu, perf, aux); +} +/** * scx_bpf_nr_node_ids - Return the number of possible node IDs *
@@ -9395,6 +9488,47 @@ __bpf_kfunc u32 scx_bpf_nr_cpu_ids(void)return nr_cpu_ids; } +/** +* scx_bpf_nr_cids - Return the size of the cid space +* +* Equals num_possible_cpus(). All valid cids are in [0, return value). +*/ +__bpf_kfunc u32 scx_bpf_nr_cids(void) +{ +return num_possible_cpus (); +} + +/** +* scx_bpf_nr_online_cids - Return current count of online CPUs in cid space +* +* Return num_online_cpus(). The standard model restarts the scheduler on +* hotplug, which lets schedulers treat [0, nr_online_cids) as the online +* range. Schedulers that prefer to handle hotplug without a restart should +* install a custom mapping via scx_bpf_cid_override() and track onlining +* through the ops.cid_online / ops.cid_offline callbacks. +*/ +__bpf_kfunc u32 scx_bpf_nr_online_cids(void) +{ +return num_online_cpus (); +} + +/** +* scx_bpf_this_cid - Return the cid of the CPU this program is running on +* +* cid-addressed equivalent of bpf_get_smp_processor_id() for scx programs. +* The current cpu is trivially valid, so this is just a table lookup. Return +* -EINVAL if called from a non-SCX program before any scheduler has ever +* been enabled (the cid table is still unallocated at that point). +*/ +__bpf_kfunc s32 scx_bpf_this_cid(void) +{ +s16 *tbl =READ_ONCE (scx_cpu_to_cid_tbl); + +if (!tbl) +return -EINVAL ; +return tbl[raw_smp_processor_id ()]; +} +/** * scx_bpf_get_possible_cpumask - Get a referenced kptr to cpu_possible_mask */
@@ -9443,6 +9577,23 @@ __bpf_kfunc s32 scx_bpf_task_cpu(const struct task_struct *p)return task_cpu (p); } +/** +* scx_bpf_task_cid - cid a task is currently associated with +* @p: task of interest +* +* cid-addressed equivalent of scx_bpf_task_cpu(). task_cpu(p) is always a +* valid cpu, so this is just a table lookup. Return -EINVAL if called from +* a non-SCX program before any scheduler has ever been enabled. +*/ +__bpf_kfunc s32 scx_bpf_task_cid(const struct task_struct *p) +{ +s16 *tbl =READ_ONCE (scx_cpu_to_cid_tbl); + +if (!tbl) +return -EINVAL ; +return tbl[task_cpu (p)]; +} +/** * scx_bpf_cpu_rq - Fetch the rq of a CPU * @cpu: CPU of the rq
@@ -9521,6 +9672,30 @@ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, const struct bpf_prog_return rcu_dereference (cpu_rq (cpu)->curr); } +/** +* scx_bpf_cid_curr - Return the curr task on the CPU at @cid +* @cid: cid of interest +* @aux: implicit BPF argument to access bpf_prog_aux hidden from BPF progs +* +* cid-addressed equivalent of scx_bpf_cpu_curr(). Callers must hold RCU +* read lock (KF_RCU). +*/ +__bpf_kfunc struct task_struct *scx_bpf_cid_curr(s32 cid, const struct bpf_prog_aux *aux) +{ +struct scx_sched *sch; +s32 cpu; + +guard (rcu)(); + + sch =scx_prog_sched (aux); +if (unlikely (!sch)) +return NULL ; + cpu =scx_cid_to_cpu (sch, cid); +if (cpu <0 ) +return NULL ; +return rcu_dereference (cpu_rq (cpu)->curr); +} +/** * scx_bpf_tid_to_task - Look up a task by its scx tid * @tid: task ID previously read from p->scx.tid
@@ -9708,6 +9883,7 @@ BTF_KFUNCS_START(scx_kfunc_ids_any)BTF_ID_FLAGS (func, scx_bpf_task_set_slice,KF_IMPLICIT_ARGS |KF_RCU );BTF_ID_FLAGS (func, scx_bpf_task_set_dsq_vtime,KF_IMPLICIT_ARGS |KF_RCU ); BTF_ID_FLAGS(func, scx_bpf_kick_cpu, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_kick_cid, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_dsq_nr_queued, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_destroy_dsq, KF_IMPLICIT_ARGS)BTF_ID_FLAGS (func, scx_bpf_dsq_peek,KF_IMPLICIT_ARGS |KF_RCU_PROTECTED |KF_RET_NULL )
@@ -9722,16 +9898,24 @@ BTF_ID_FLAGS(func, scx_bpf_dump_bstr, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_cpuperf_cap, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_cpuperf_cur, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_cpuperf_set, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_cidperf_cap, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_cidperf_cur, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_cidperf_set, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_nr_node_ids) BTF_ID_FLAGS(func, scx_bpf_nr_cpu_ids) +BTF_ID_FLAGS(func, scx_bpf_nr_cids) +BTF_ID_FLAGS(func, scx_bpf_nr_online_cids) +BTF_ID_FLAGS(func, scx_bpf_this_cid) BTF_ID_FLAGS(func, scx_bpf_get_possible_cpumask, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_put_cpumask, KF_RELEASE) BTF_ID_FLAGS(func, scx_bpf_task_running, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_task_cpu, KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_task_cid, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_cpu_rq, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, scx_bpf_locked_rq, KF_IMPLICIT_ARGS | KF_RET_NULL) BTF_ID_FLAGS(func, scx_bpf_cpu_curr, KF_IMPLICIT_ARGS | KF_RET_NULL | KF_RCU_PROTECTED) +BTF_ID_FLAGS(func, scx_bpf_cid_curr, KF_IMPLICIT_ARGS | KF_RET_NULL | KF_RCU_PROTECTED) BTF_ID_FLAGS(func, scx_bpf_tid_to_task, KF_RET_NULL | KF_RCU_PROTECTED) BTF_ID_FLAGS(func, scx_bpf_now)BTF_ID_FLAGS (func, scx_bpf_events)
tools/sched_ext/include/scx/common.bpf.h +9 -0
@@ -105,6 +105,15 @@ void scx_bpf_events(struct scx_event_stats *events, size_t events__sz) __ksym __ s32 scx_bpf_cpu_to_cid(s32 cpu) __ksym __weak; s32 scx_bpf_cid_to_cpu(s32 cid) __ksym __weak; void scx_bpf_cid_topo(s32 cid, struct scx_cid_topo *out) __ksym __weak; +void scx_bpf_kick_cid(s32 cid, u64 flags) __ksym __weak; +s32 scx_bpf_task_cid(const struct task_struct *p) __ksym __weak; +s32 scx_bpf_this_cid(void) __ksym __weak; +struct task_struct *scx_bpf_cid_curr (s32 cid) __ksym__weak ; +u32 scx_bpf_nr_cids(void) __ksym __weak; +u32 scx_bpf_nr_online_cids(void) __ksym __weak; +u32 scx_bpf_cidperf_cap(s32 cid) __ksym __weak; +u32 scx_bpf_cidperf_cur(s32 cid) __ksym __weak; +void scx_bpf_cidperf_set(s32 cid, u32 perf) __ksym __weak;/* * Use the following as @it__iter when calling scx_bpf_dsq_move[_vtime]() from
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 12/17 sched_ext: Add bpf_sched_ext_ops_cid struct_ops type
In reply to Tejun Heocpumask is awkward from BPF and unusable from arena; cid/cmask work in both. Sub-sched enqueue will need cmask. Without a full cid interface, schedulers end up mixing forms - a subtle-bug factory.
Add sched_ext_ops_cid, which mirrors sched_ext_ops with cid/cmask replacing cpu/cpumask in the topology-carrying callbacks. cpu_acquire/cpu_release are deprecated and absent; a prior patch moved them past @priv so the cid-form can omit them without disturbing shared-field offsets.
The two structs share byte-identical layout up to @priv, so the existing bpf_scx init/check hooks, has_op bitmap, and scx_kf_allow_flags[] are offset-indexed and apply to both. BUILD_BUG_ON in scx_init() pins the shared-field and renamed-callback offsets so any future drift trips at boot.
The kernel<->BPF boundary translates between cpu and cid:
- A static key, enabled on cid-form sched load, gates the translation so cpu-form schedulers pay nothing.
- dispatch, update_idle, cpu_online/offline and dump_cpu translate the cpu arg at the callsite.
- select_cpu also translates the returned cid back to a cpu.
- set_cpumask is wrapped to synthesize a cmask in a per-cpu scratch before calling the cid-form callback.
All scheds in a hierarchy share one form. The static key drives the hot-path branch.
v2: Use struct_size() for the set_cmask_scratch percpu alloc. Move
cid-shard fields and assertions into the later cid-shard patch.
v3: Drop static on scx_set_cmask_scratch; add extern in ext_internal.h.
--- kernel/sched/ext.c | 281 +++++++++++++++++++++-- kernel/sched/ext_cid.c | 37 ++- kernel/sched/ext_cid.h | 9 + kernel/sched/ext_idle.c | 2 +- kernel/sched/ext_internal.h | 109 ++++++++- tools/sched_ext/include/scx/compat.bpf.h | 12 + 6 files changed, 429 insertions(+), 21 deletions(-)
kernel/sched/ext.c +263 -18
@@ -513,6 +513,33 @@ do { \update_locked_rq (__prev_locked_rq); \ }while (0 ) +/* +* Flipped on enable per sch->is_cid_type. Declared in ext_internal.h so +* subsystem inlines can read it. +*/ +DEFINE_STATIC_KEY_FALSE (__scx_is_cid_type); + +/* +* scx_cpu_arg() wraps a cpu arg being handed to an SCX op. For cid-form +* schedulers it resolves to the matching cid; for cpu-form it passes @cpu +* through. scx_cpu_ret() is the inverse for a cpu/cid returned from an op +* (currently only ops.select_cpu); it validates the BPF-supplied cid and +* triggers scx_error() on @sch if invalid. +*/ +static s32 scx_cpu_arg (s32 cpu) +{ +if (scx_is_cid_type ()) +return __scx_cpu_to_cid (cpu); +return cpu; +} + +static s32 scx_cpu_ret (struct scx_sched *sch,s32 cpu_or_cid) +{ +if (cpu_or_cid <0 || !scx_is_cid_type ()) +return cpu_or_cid; +return scx_cid_to_cpu (sch, cpu_or_cid); +} + #define SCX_CALL_OP_RET(sch, op, locked_rq, args...) \ ({ \ struct rq *__prev_locked_rq; \
@@ -574,6 +601,41 @@ do { \ __ret; \ }) +/** +* scx_call_op_set_cpumask - invoke ops.set_cpumask / ops_cid.set_cmask for @task +* @sch: scx_sched being invoked +* @rq: rq to update as the currently-locked rq, or NULL +* @task: task whose affinity is changing +* @cpumask: new cpumask +* +* For cid-form schedulers, translate @cpumask to a cmask via the per-cpu +* scratch in ext_cid.c and dispatch through the ops_cid union view. Caller +* must hold @rq's rq lock so this_cpu_ptr is stable across the call. +*/ +static inline void scx_call_op_set_cpumask (struct scx_sched *sch,struct rq *rq, +struct task_struct *task, +const struct cpumask *cpumask) +{ +WARN_ON_ONCE (current->scx.kf_tasks[0 ]); + current->scx.kf_tasks[0 ] = task; +if (rq) +update_locked_rq (rq); + +if (scx_is_cid_type ()) { +struct scx_cmask *cmask =this_cpu_ptr (scx_set_cmask_scratch); + +lockdep_assert_irqs_disabled (); +scx_cpumask_to_cmask (cpumask, cmask); + sch->ops_cid.set_cmask (task, cmask); + }else { + sch->ops.set_cpumask (task, cpumask); + } + +if (rq) +update_locked_rq (NULL ); + current->scx.kf_tasks[0 ] =NULL ; +} +/* see SCX_CALL_OP_TASK() */ static __always_inline bool scx_kf_arg_task_ok(struct scx_sched *sch,struct task_struct *p)
@@ -1679,7 +1741,7 @@ static struct scx_dispatch_q *find_dsq_for_dispatch(struct scx_sched *sch,return &rq->scx.local_dsq;if ((dsq_id &SCX_DSQ_LOCAL_ON ) ==SCX_DSQ_LOCAL_ON ) { -s32 cpu = dsq_id &SCX_DSQ_LOCAL_CPU_MASK ; +s32 cpu =scx_cpu_ret (sch, dsq_id &SCX_DSQ_LOCAL_CPU_MASK );if (!scx_cpu_valid (sch, cpu,"in SCX_DSQ_LOCAL_ON dispatch verdict" ))return find_global_dsq (sch, tcpu);
@@ -2761,11 +2823,13 @@ scx_dispatch_sched(struct scx_sched *sch, struct rq *rq, dspc->nr_tasks =0 ;if (nested) { -SCX_CALL_OP (sch, dispatch, rq, cpu, prev_on_sch ? prev :NULL ); +SCX_CALL_OP (sch, dispatch, rq,scx_cpu_arg (cpu), + prev_on_sch ? prev :NULL ); }else {/* stash @prev so that nested invocations can access it */ rq->scx.sub_dispatch_prev = prev; -SCX_CALL_OP (sch, dispatch, rq, cpu, prev_on_sch ? prev :NULL ); +SCX_CALL_OP (sch, dispatch, rq,scx_cpu_arg (cpu), + prev_on_sch ? prev :NULL ); rq->scx.sub_dispatch_prev =NULL ; }
@@ -3260,7 +3324,9 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag *ddsp_taskp = p;this_rq ()->scx.in_select_cpu = true; - cpu =SCX_CALL_OP_TASK_RET (sch, select_cpu,NULL , p, prev_cpu, wake_flags); + cpu =SCX_CALL_OP_TASK_RET (sch, select_cpu,NULL , p, +scx_cpu_arg (prev_cpu), wake_flags); + cpu =scx_cpu_ret (sch, cpu);this_rq ()->scx.in_select_cpu = false; p->scx.selected_cpu = cpu; *ddsp_taskp =NULL ;
@@ -3310,7 +3376,7 @@ static void set_cpus_allowed_scx(struct task_struct *p,* designation pointless. Cast it away when calling the operation. */ if (SCX_HAS_OP (sch, set_cpumask)) -SCX_CALL_OP_TASK (sch, set_cpumask,task_rq (p), p, (struct cpumask *)p->cpus_ptr); +scx_call_op_set_cpumask (sch,task_rq (p), p, (struct cpumask *)p->cpus_ptr); }static void handle_hotplug (struct rq *rq,bool online)
@@ -3332,9 +3398,9 @@ static void handle_hotplug(struct rq *rq, bool online)scx_idle_update_selcpu_topology (&sch->ops);if (online &&SCX_HAS_OP (sch, cpu_online)) -SCX_CALL_OP (sch, cpu_online,NULL , cpu); +SCX_CALL_OP (sch, cpu_online,NULL ,scx_cpu_arg (cpu));else if (!online &&SCX_HAS_OP (sch, cpu_offline)) -SCX_CALL_OP (sch, cpu_offline,NULL , cpu); +SCX_CALL_OP (sch, cpu_offline,NULL ,scx_cpu_arg (cpu));else scx_exit(sch, SCX_EXIT_UNREG_KERN, SCX_ECODE_ACT_RESTART | SCX_ECODE_RSN_HOTPLUG,
@@ -3919,7 +3985,7 @@ static void switching_to_scx(struct rq *rq, struct task_struct *p)* different scheduler class. Keep the BPF scheduler up-to-date. */ if (SCX_HAS_OP (sch, set_cpumask)) -SCX_CALL_OP_TASK (sch, set_cpumask, rq, p, (struct cpumask *)p->cpus_ptr); +scx_call_op_set_cpumask (sch, rq, p, (struct cpumask *)p->cpus_ptr); }static void switched_from_scx (struct rq *rq,struct task_struct *p)
@@ -5945,6 +6011,7 @@ static void scx_root_disable(struct scx_sched *sch)/* no task is on scx, turn off all the switches and flush in-progress calls */ static_branch_disable (&__scx_enabled); +static_branch_disable (&__scx_is_cid_type);if (sch->ops.flags &SCX_OPS_TID_TO_TASK )static_branch_disable (&__scx_tid_to_task_enabled);bitmap_zero (sch->has_op,SCX_OPI_END );
@@ -6360,7 +6427,7 @@ static void scx_dump_state(struct scx_sched *sch, struct scx_exit_info *ei, used =seq_buf_used (&ns);if (SCX_HAS_OP (sch, dump_cpu)) {ops_dump_init (&ns," " ); -SCX_CALL_OP (sch, dump_cpu, rq, &dctx, cpu, idle); +SCX_CALL_OP (sch, dump_cpu, rq, &dctx,scx_cpu_arg (cpu), idle);ops_dump_exit (); }
@@ -6516,7 +6583,11 @@ static struct scx_sched_pnode *alloc_pnode(struct scx_sched *sch, int node)*/ struct scx_enable_cmd {struct kthread_work work; -struct sched_ext_ops *ops; +union { +struct sched_ext_ops *ops; +struct sched_ext_ops_cid *ops_cid; + }; +bool is_cid_type;int ret; };
@@ -6524,10 +6595,11 @@ struct scx_enable_cmd {* Allocate and initialize a new scx_sched. @cgrp's reference is always * consumed whether the function succeeds or fails. */ -static struct scx_sched *scx_alloc_and_add_sched (struct sched_ext_ops *ops, +static struct scx_sched *scx_alloc_and_add_sched (struct scx_enable_cmd *cmd,struct cgroup *cgrp,struct scx_sched *parent) { +struct sched_ext_ops *ops = cmd->ops;struct scx_sched *sch;s32 level = parent ? parent->level +1 :0 ;s32 node, cpu, ret, bypass_fail_cpu = nr_cpu_ids;
@@ -6619,7 +6691,18 @@ static struct scx_sched *scx_alloc_and_add_sched(struct sched_ext_ops *ops, ret = -ENOMEM ; goto err_free_lb_cpumask; } - sch->ops = *ops; +/* +* Copy ops through the right union view. For cid-form the source is +* struct sched_ext_ops_cid which lacks the trailing cpu_acquire/ +* cpu_release; those stay zero from kzalloc. +*/ +if (cmd->is_cid_type) { + sch->ops_cid = *cmd->ops_cid; + sch->is_cid_type = true; + }else { + sch->ops = *cmd->ops; + } +rcu_assign_pointer (ops->priv, sch); sch->kobj.kset = scx_kset;
@@ -6756,7 +6839,12 @@ static int validate_ops(struct scx_sched *sch, const struct sched_ext_ops *ops)return -EINVAL ; } -if (ops->cpu_acquire || ops->cpu_release) +/* +* cid-form's struct is shorter and doesn't include the cpu_acquire / +* cpu_release tail; reading those fields off a cid-form @ops would +* run past the BPF allocation. Skip for cid-form. +*/ +if (!sch->is_cid_type && (ops->cpu_acquire || ops->cpu_release))pr_warn ("ops->cpu_acquire/release() are deprecated, use sched_switch TP instead\n" );return 0 ;
@@ -6792,12 +6880,15 @@ static void scx_root_enable_workfn(struct kthread_work *work)#ifdef CONFIG_EXT_SUB_SCHED cgroup_get (cgrp);#endif - sch =scx_alloc_and_add_sched (ops, cgrp,NULL ); + sch =scx_alloc_and_add_sched (cmd, cgrp,NULL );if (IS_ERR (sch)) { ret =PTR_ERR (sch); goto err_free_tid_hash; } +if (sch->is_cid_type) +static_branch_enable (&__scx_is_cid_type); +/* * Transition to ENABLING and clear exit info to arm the disable path. * Failure triggers full disabling from here on.
@@ -7119,7 +7210,7 @@ static void scx_sub_enable_workfn(struct kthread_work *work)raw_spin_unlock_irq (&scx_sched_lock);/* scx_alloc_and_add_sched() consumes @cgrp whether it succeeds or not */ - sch =scx_alloc_and_add_sched (ops, cgrp, parent); + sch =scx_alloc_and_add_sched (cmd, cgrp, parent);kobject_put (&parent->kobj);if (IS_ERR (sch)) { ret =PTR_ERR (sch);
@@ -7570,6 +7661,13 @@ static int bpf_scx_reg(void *kdata, struct bpf_link *link)return scx_enable (&cmd, link); } +static int bpf_scx_reg_cid (void *kdata,struct bpf_link *link) +{ +struct scx_enable_cmd cmd = { .ops_cid = kdata, .is_cid_type = true }; + +return scx_enable (&cmd, link); +} +static void bpf_scx_unreg (void *kdata,struct bpf_link *link) {struct sched_ext_ops *ops = kdata;
@@ -7701,6 +7799,73 @@ static struct bpf_struct_ops bpf_sched_ext_ops = { .cfi_stubs = &__bpf_ops_sched_ext_ops }; +/* +* cid-form cfi stubs. Stubs whose signatures match the cpu-form (param types +* identical, only param names differ across structs) are reused; only +* set_cmask needs a fresh stub since the second argument type differs. +*/ +static void sched_ext_ops_cid__set_cmask (struct task_struct *p, +const struct scx_cmask *cmask) {} + +static struct sched_ext_ops_cid __bpf_ops_sched_ext_ops_cid = { + .select_cid = sched_ext_ops__select_cpu, + .enqueue = sched_ext_ops__enqueue, + .dequeue = sched_ext_ops__dequeue, + .dispatch = sched_ext_ops__dispatch, + .tick = sched_ext_ops__tick, + .runnable = sched_ext_ops__runnable, + .running = sched_ext_ops__running, + .stopping = sched_ext_ops__stopping, + .quiescent = sched_ext_ops__quiescent, + .yield = sched_ext_ops__yield, + .core_sched_before = sched_ext_ops__core_sched_before, + .set_weight = sched_ext_ops__set_weight, + .set_cmask = sched_ext_ops_cid__set_cmask, + .update_idle = sched_ext_ops__update_idle, + .init_task = sched_ext_ops__init_task, + .exit_task = sched_ext_ops__exit_task, + .enable = sched_ext_ops__enable, + .disable = sched_ext_ops__disable, +#ifdef CONFIG_EXT_GROUP_SCHED + .cgroup_init = sched_ext_ops__cgroup_init, + .cgroup_exit = sched_ext_ops__cgroup_exit, + .cgroup_prep_move = sched_ext_ops__cgroup_prep_move, + .cgroup_move = sched_ext_ops__cgroup_move, + .cgroup_cancel_move = sched_ext_ops__cgroup_cancel_move, + .cgroup_set_weight = sched_ext_ops__cgroup_set_weight, + .cgroup_set_bandwidth = sched_ext_ops__cgroup_set_bandwidth, + .cgroup_set_idle = sched_ext_ops__cgroup_set_idle, +#endif + .sub_attach = sched_ext_ops__sub_attach, + .sub_detach = sched_ext_ops__sub_detach, + .cid_online = sched_ext_ops__cpu_online, + .cid_offline = sched_ext_ops__cpu_offline, + .init = sched_ext_ops__init, + .exit = sched_ext_ops__exit, + .dump = sched_ext_ops__dump, + .dump_cid = sched_ext_ops__dump_cpu, + .dump_task = sched_ext_ops__dump_task, +}; + +/* +* The cid-form struct_ops shares all bpf_struct_ops hooks with the cpu form. +* init_member, check_member, reg, unreg, etc. process kdata as the byte block +* verified to match by the BUILD_BUG_ON checks in scx_init(). +*/ +static struct bpf_struct_ops bpf_sched_ext_ops_cid = { + .verifier_ops = &bpf_scx_verifier_ops, + .reg = bpf_scx_reg_cid, + .unreg = bpf_scx_unreg, + .check_member = bpf_scx_check_member, + .init_member = bpf_scx_init_member, + .init = bpf_scx_init, + .update = bpf_scx_update, + .validate = bpf_scx_validate, + .name ="sched_ext_ops_cid" , + .owner =THIS_MODULE , + .cfi_stubs = &__bpf_ops_sched_ext_ops_cid +}; +/******************************************************************************** * System integration and init.
@@ -8912,7 +9077,7 @@ __bpf_kfunc s32 scx_bpf_dsq_nr_queued(u64 dsq_id, const struct bpf_prog_aux *aux ret =READ_ONCE (this_rq ()->scx.local_dsq.nr); goto out; }else if ((dsq_id &SCX_DSQ_LOCAL_ON ) ==SCX_DSQ_LOCAL_ON ) { -s32 cpu = dsq_id &SCX_DSQ_LOCAL_CPU_MASK ; +s32 cpu =scx_cpu_ret (sch, dsq_id &SCX_DSQ_LOCAL_CPU_MASK );if (scx_cpu_valid (sch, cpu,NULL )) { ret =READ_ONCE (cpu_rq (cpu)->scx.local_dsq.nr);
@@ -10019,8 +10184,15 @@ int scx_kfunc_context_filter(const struct bpf_prog *prog, u32 kfunc_id)/* * Non-SCX struct_ops: SCX kfuncs are not permitted. -*/ -if (prog->aux->st_ops != &bpf_sched_ext_ops) +* +* Both bpf_sched_ext_ops (cpu-form) and bpf_sched_ext_ops_cid +* (cid-form) are valid SCX struct_ops. Member offsets match between +* the two (verified by BUILD_BUG_ON in scx_init()), so the shared +* scx_kf_allow_flags[] table indexed by SCX_MOFF_IDX(moff) applies to +* both. +*/ +if (prog->aux->st_ops != &bpf_sched_ext_ops && + prog->aux->st_ops != &bpf_sched_ext_ops_cid)return -EACCES ;/* SCX struct_ops: check the per-op allow list. */
@@ -10050,6 +10222,73 @@ static int __init scx_init(void) {int ret; +/* +* sched_ext_ops_cid mirrors sched_ext_ops up to and including @priv. +* Both bpf_scx_init_member() and bpf_scx_check_member() use offsets +* from struct sched_ext_ops; sched_ext_ops_cid relies on those offsets +* matching for the shared fields. Catch any drift at boot. +*/ +#define CID_OFFSET_MATCH (cpu_field, cid_field) \ + BUILD_BUG_ON(offsetof(struct sched_ext_ops, cpu_field) != \ + offsetof(struct sched_ext_ops_cid, cid_field)) +/* data fields used by bpf_scx_init_member() */ +CID_OFFSET_MATCH (dispatch_max_batch, dispatch_max_batch); +CID_OFFSET_MATCH (flags, flags); +CID_OFFSET_MATCH (name, name); +CID_OFFSET_MATCH (timeout_ms, timeout_ms); +CID_OFFSET_MATCH (exit_dump_len, exit_dump_len); +CID_OFFSET_MATCH (hotplug_seq, hotplug_seq); +CID_OFFSET_MATCH (sub_cgroup_id, sub_cgroup_id); +/* shared callbacks: the union view requires byte-for-byte offset match */ +CID_OFFSET_MATCH (enqueue, enqueue); +CID_OFFSET_MATCH (dequeue, dequeue); +CID_OFFSET_MATCH (dispatch, dispatch); +CID_OFFSET_MATCH (tick, tick); +CID_OFFSET_MATCH (runnable, runnable); +CID_OFFSET_MATCH (running, running); +CID_OFFSET_MATCH (stopping, stopping); +CID_OFFSET_MATCH (quiescent, quiescent); +CID_OFFSET_MATCH (yield, yield); +CID_OFFSET_MATCH (core_sched_before, core_sched_before); +CID_OFFSET_MATCH (set_weight, set_weight); +CID_OFFSET_MATCH (update_idle, update_idle); +CID_OFFSET_MATCH (init_task, init_task); +CID_OFFSET_MATCH (exit_task, exit_task); +CID_OFFSET_MATCH (enable, enable); +CID_OFFSET_MATCH (disable, disable); +CID_OFFSET_MATCH (dump, dump); +CID_OFFSET_MATCH (dump_task, dump_task); +CID_OFFSET_MATCH (sub_attach, sub_attach); +CID_OFFSET_MATCH (sub_detach, sub_detach); +CID_OFFSET_MATCH (init, init); +CID_OFFSET_MATCH (exit, exit); +#ifdef CONFIG_EXT_GROUP_SCHED +CID_OFFSET_MATCH (cgroup_init, cgroup_init); +CID_OFFSET_MATCH (cgroup_exit, cgroup_exit); +CID_OFFSET_MATCH (cgroup_prep_move, cgroup_prep_move); +CID_OFFSET_MATCH (cgroup_move, cgroup_move); +CID_OFFSET_MATCH (cgroup_cancel_move, cgroup_cancel_move); +CID_OFFSET_MATCH (cgroup_set_weight, cgroup_set_weight); +CID_OFFSET_MATCH (cgroup_set_bandwidth, cgroup_set_bandwidth); +CID_OFFSET_MATCH (cgroup_set_idle, cgroup_set_idle); +#endif +/* renamed callbacks must occupy the same slot as their cpu-form sibling */ +CID_OFFSET_MATCH (select_cpu, select_cid); +CID_OFFSET_MATCH (set_cpumask, set_cmask); +CID_OFFSET_MATCH (cpu_online, cid_online); +CID_OFFSET_MATCH (cpu_offline, cid_offline); +CID_OFFSET_MATCH (dump_cpu, dump_cid); +/* @priv tail must align since both share the same data block */ +CID_OFFSET_MATCH (priv, priv); +/* +* cid-form must end exactly at @priv - validate_ops() skips +* cpu_acquire/cpu_release for cid-form because reading those fields +* past the BPF allocation would be UB. +*/ +BUILD_BUG_ON (sizeof (struct sched_ext_ops_cid ) != + offsetofend(struct sched_ext_ops, priv)); +#undef CID_OFFSET_MATCH +/* * kfunc registration can't be done from init_sched_ext_class() as * register_btf_kfunc_id_set() needs most of the system to be up.
@@ -10100,6 +10339,12 @@ static int __init scx_init(void)return ret; } + ret =register_bpf_struct_ops (&bpf_sched_ext_ops_cid, sched_ext_ops_cid); +if (ret) { +pr_err ("sched_ext: Failed to register cid struct_ops (%d)\n" , ret); +return ret; + } + ret =register_pm_notifier (&scx_pm_notifier);if (ret) {pr_err ("sched_ext: Failed to register PM notifier (%d)\n" , ret);
kernel/sched/ext_cid.c +36 -1
@@ -7,6 +7,14 @@*/ #include <linux/cacheinfo.h> +/* +* Per-cpu scratch cmask used by scx_call_op_set_cpumask() to synthesize a +* cmask from a cpumask. Allocated alongside the cid arrays on first enable +* and never freed. Sized to the full cid space. Caller holds rq lock so +* this_cpu_ptr is safe. +*/ +struct scx_cmask __percpu *scx_set_cmask_scratch; +/* * cid tables. *
@@ -46,6 +54,7 @@ static s32 scx_cid_arrays_alloc(void)u32 npossible =num_possible_cpus ();s16 *cid_to_cpu, *cpu_to_cid;struct scx_cid_topo *cid_topo; +struct scx_cmask __percpu *set_cmask_scratch;if (scx_cid_to_cpu_tbl)return 0 ;
@@ -53,17 +62,22 @@ static s32 scx_cid_arrays_alloc(void) cid_to_cpu =kzalloc_objs (*scx_cid_to_cpu_tbl, npossible,GFP_KERNEL ); cpu_to_cid =kzalloc_objs (*scx_cpu_to_cid_tbl, nr_cpu_ids,GFP_KERNEL ); cid_topo =kmalloc_objs (*scx_cid_topo, npossible,GFP_KERNEL ); + set_cmask_scratch =__alloc_percpu (struct_size (set_cmask_scratch, bits, +SCX_CMASK_NR_WORDS (npossible)), +sizeof (u64)); -if (!cid_to_cpu || !cpu_to_cid || !cid_topo) { +if (!cid_to_cpu || !cpu_to_cid || !cid_topo || !set_cmask_scratch) {kfree (cid_to_cpu);kfree (cpu_to_cid);kfree (cid_topo); +free_percpu (set_cmask_scratch);return -ENOMEM ; }WRITE_ONCE (scx_cid_to_cpu_tbl, cid_to_cpu);WRITE_ONCE (scx_cpu_to_cid_tbl, cpu_to_cid);WRITE_ONCE (scx_cid_topo, cid_topo); +WRITE_ONCE (scx_set_cmask_scratch, set_cmask_scratch);return 0 ; }
@@ -208,6 +222,27 @@ s32 scx_cid_init(struct scx_sched *sch)return 0 ; } +/** +* scx_cpumask_to_cmask - Translate a kernel cpumask into a cmask +* @src: source cpumask +* @dst: cmask to write +* +* Initialize @dst to cover the full cid space [0, num_possible_cpus()) and +* set the bit for each cid whose cpu is in @src. +*/ +void scx_cpumask_to_cmask (const struct cpumask *src,struct scx_cmask *dst) +{ +s32 cpu; + +scx_cmask_init (dst,0 ,num_possible_cpus ()); +for_each_cpu (cpu, src) { +s32 cid =__scx_cpu_to_cid (cpu); + +if (cid >=0 ) +__scx_cmask_set (dst, cid); + } +} +__bpf_kfunc_start_defs ();/**
kernel/sched/ext_cid.h +9 -0
@@ -53,6 +53,7 @@ extern struct btf_id_set8 scx_kfunc_ids_init;s32 scx_cid_init (struct scx_sched *sch);int scx_cid_kfunc_init (void ); +void scx_cpumask_to_cmask (const struct cpumask *src,struct scx_cmask *dst);/** * cid_valid - Verify a cid value, to be used on ops input args
@@ -127,6 +128,14 @@ static inline s32 scx_cpu_to_cid(struct scx_sched *sch, s32 cpu)return __scx_cpu_to_cid (cpu); } +/** +* scx_is_cid_type - Test whether the active scheduler hierarchy is cid-form +*/ +static inline bool scx_is_cid_type (void ) +{ +return static_branch_unlikely (&__scx_is_cid_type); +} +static inline bool __scx_cmask_contains (const struct scx_cmask *m,u32 cid) {return likely (cid >= m->base && cid < m->base + m->nr_bits);
kernel/sched/ext_idle.c +1 -1
@@ -788,7 +788,7 @@ void __scx_update_idle(struct rq *rq, bool idle, bool do_notify)*/ if (SCX_HAS_OP (sch, update_idle) && do_notify && !scx_bypassing (sch,cpu_of (rq))) -SCX_CALL_OP (sch, update_idle, rq,cpu_of (rq), idle); +SCX_CALL_OP (sch, update_idle, rq,scx_cpu_arg (cpu_of (rq)), idle); }static void reset_idle_masks (struct sched_ext_ops *ops)
kernel/sched/ext_internal.h +108 -1
127 lines not on this page · raw message
tools/sched_ext/include/scx/compat.bpf.h +12 -0
@@ -446,4 +446,16 @@ static inline void scx_bpf_dsq_reenq(u64 dsq_id, u64 reenq_flags) __VA_ARGS__, \ }; +/* +* Define a cid-form sched_ext_ops. Programs targeting this struct_ops type +* use cid-form callback signatures (select_cid, set_cmask, cid_online/offline, +* dispatch with cid arg, etc.) and may only call the cid-form scx_bpf_* +* kfuncs (kick_cid, task_cid, this_cid, ...). +*/ +#define SCX_OPS_CID_DEFINE (__name, ...) \ + SEC(".struct_ops.link") \ + struct sched_ext_ops_cid __name = { \ + __VA_ARGS__, \ + }; +#endif /* __SCX_COMPAT_BPF_H */
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 13/17 sched_ext: Forbid cpu-form kfuncs from cid-form schedulers
In reply to Tejun Heocid and cpu are both small s32s, trivially confused when a cid-form scheduler calls a cpu-keyed kfunc. Reject cid-form programs that reference any kfunc in the new scx_kfunc_ids_cpu_only at verifier load time.
The reverse direction is intentionally permissive: cpu-form schedulers can freely call cid-form kfuncs to ease a gradual cpumask -> cid migration.
The check sits in scx_kfunc_context_filter() right after the SCX struct_ops gate and before the any/idle allow and per-op allow-list checks, so it catches cpu-only kfuncs regardless of which set they belong to (any, idle, or select_cpu).
v2: Sync per-entry kfunc flags with their primary declarations (Zhao).
pahole intersects flags across BTF_ID_FLAGS() occurrences, so omitting them drops the flags globally.
--- kernel/sched/ext.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+)
kernel/sched/ext.c +51 -0
@@ -10095,6 +10095,47 @@ static const struct btf_kfunc_id_set scx_kfunc_set_any = { .filter = scx_kfunc_context_filter, }; +/* +* cpu-form kfuncs that are forbidden from cid-form schedulers +* (bpf_sched_ext_ops_cid). Programs targeting the cid struct_ops type must +* use the cid-form alternative (cid/cmask kfuncs). +* +* Membership overlaps with scx_kfunc_ids_{any,idle,select_cpu}; the filter +* tests this set independently and rejects matches before the per-op +* allow-list check runs. +* +* pahole/resolve_btfids scans every BTF_ID_FLAGS() at build time and +* intersects flags across duplicate entries, so each entry must carry the +* same flags as the kfunc's primary declaration; otherwise the flags get +* dropped globally. +*/ +BTF_KFUNCS_START(scx_kfunc_ids_cpu_only) +BTF_ID_FLAGS(func, scx_bpf_kick_cpu, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_task_cpu, KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_cpu_rq, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_cpu_curr, KF_IMPLICIT_ARGS | KF_RET_NULL | KF_RCU_PROTECTED) +BTF_ID_FLAGS(func, scx_bpf_cpu_node, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_cpuperf_cap, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_cpuperf_cur, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_cpuperf_set, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_get_possible_cpumask, KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_get_online_cpumask, KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_put_cpumask, KF_RELEASE) +BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_IMPLICIT_ARGS | KF_RCU) +BTF_ID_FLAGS(func, __scx_bpf_select_cpu_and, KF_IMPLICIT_ARGS | KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_select_cpu_and, KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_IMPLICIT_ARGS | KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask_node, KF_IMPLICIT_ARGS | KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_IMPLICIT_ARGS | KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask_node, KF_IMPLICIT_ARGS | KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE) +BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_IMPLICIT_ARGS | KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu_node, KF_IMPLICIT_ARGS | KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_IMPLICIT_ARGS | KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu_node, KF_IMPLICIT_ARGS | KF_RCU) +BTF_KFUNCS_END(scx_kfunc_ids_cpu_only) +/* * Per-op kfunc allow flags. Each bit corresponds to a context-sensitive kfunc * group; an op may permit zero or more groups, with the union expressed in
@@ -10158,6 +10199,7 @@ int scx_kfunc_context_filter(const struct bpf_prog *prog, u32 kfunc_id)bool in_cpu_release =btf_id_set8_contains (&scx_kfunc_ids_cpu_release, kfunc_id);bool in_idle =btf_id_set8_contains (&scx_kfunc_ids_idle, kfunc_id);bool in_any =btf_id_set8_contains (&scx_kfunc_ids_any, kfunc_id); +bool in_cpu_only =btf_id_set8_contains (&scx_kfunc_ids_cpu_only, kfunc_id);u32 moff, flags;/* Not an SCX kfunc - allow. */
@@ -10195,6 +10237,15 @@ int scx_kfunc_context_filter(const struct bpf_prog *prog, u32 kfunc_id) prog->aux->st_ops != &bpf_sched_ext_ops_cid)return -EACCES ; +/* +* cid-form schedulers must use cid/cmask kfuncs. cid and cpu are both +* small s32s and trivially confused, so cpu-only kfuncs are rejected at +* load time. The reverse (cpu-form calling cid-form kfuncs) is +* intentionally permissive to ease gradual cpumask -> cid migration. +*/ +if (prog->aux->st_ops == &bpf_sched_ext_ops_cid && in_cpu_only) +return -EACCES ; +/* SCX struct_ops: check the per-op allow list. */ if (in_any || in_idle)return 0 ;
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 14/17 tools/sched_ext: scx_qmap: Restart on hotplug instead of cpu_online/offline
In reply to Tejun HeoThe cid mapping is built from the online cpu set at scheduler enable and stays valid for that set; routine hotplug invalidates it. The default cid behavior is to restart the scheduler so the mapping gets rebuilt against the new online set, and that requires not implementing cpu_online / cpu_offline (which suppress the kernel's ACT_RESTART).
Drop the two ops along with their print_cpus() helper - the cluster view was only useful as a hotplug demo and is meaningless over the dense cid space the scheduler will move to. Wire main() to handle the ACT_RESTART exit by reopening the skel and reattaching, matching the pattern in scx_simple / scx_central / scx_flatcg etc. Reset optind so getopt re-parses argv into the fresh skel rodata each iteration.
--- tools/sched_ext/scx_qmap.bpf.c | 62 ---------------------------------- tools/sched_ext/scx_qmap.c | 13 +++---- 2 files changed, 7 insertions(+), 68 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +0 -62
@@ -843,63 +843,6 @@ void BPF_STRUCT_OPS(qmap_cgroup_set_bandwidth, struct cgroup *cgrp, cgrp->kn->id, period_us, quota_us, burst_us); } -/* -* Print out the online and possible CPU map using bpf_printk() as a -* demonstration of using the cpumask kfuncs and ops.cpu_on/offline(). -*/ -static void print_cpus (void ) -{ -const struct cpumask *possible, *online; -s32 cpu; -char buf[128 ] ="" , *p; -int idx; - - possible =scx_bpf_get_possible_cpumask (); - online =scx_bpf_get_online_cpumask (); - - idx =0 ; - bpf_for(cpu, 0, scx_bpf_nr_cpu_ids()) { - if (!(p = MEMBER_VPTR(buf, [idx++]))) -break ; -if (bpf_cpumask_test_cpu (cpu, online)) - *p++ ='O' ; -else if (bpf_cpumask_test_cpu (cpu, possible)) - *p++ ='X' ; -else - *p++ =' ' ; - -if ((cpu &7 ) ==7 ) { - if (!(p = MEMBER_VPTR(buf, [idx++]))) -break ; - *p++ ='|' ; - } - } - buf[sizeof (buf) -1 ] ='\0' ; - -scx_bpf_put_cpumask (online); -scx_bpf_put_cpumask (possible); - -bpf_printk ("CPUS: |%s" , buf); -} - -void BPF_STRUCT_OPS (qmap_cpu_online ,s32 cpu) -{ -if (print_msgs) { -bpf_printk ("CPU %d coming online" , cpu); -/* @cpu is already online at this point */ -print_cpus (); - } -} - -void BPF_STRUCT_OPS (qmap_cpu_offline ,s32 cpu) -{ -if (print_msgs) { -bpf_printk ("CPU %d going offline" , cpu); -/* @cpu is still online at this point */ -print_cpus (); - } -} -struct monitor_timer {struct bpf_timer timer; };
@@ -1078,9 +1021,6 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init) slab[i].next_free = (i +1 < max_tasks) ? &slab[i +1 ] :NULL ; qa.task_free_head = &slab[0 ]; -if (print_msgs && !sub_cgroup_id) -print_cpus (); - ret =scx_bpf_create_dsq (SHARED_DSQ ,-1 );if (ret) {scx_bpf_error ("failed to create DSQ %d (%d)" ,SHARED_DSQ , ret);
@@ -1174,8 +1114,6 @@ SCX_OPS_DEFINE(qmap_ops, .cgroup_set_bandwidth = (void *)qmap_cgroup_set_bandwidth, .sub_attach = (void *)qmap_sub_attach, .sub_detach = (void *)qmap_sub_detach, - .cpu_online = (void *)qmap_cpu_online, - .cpu_offline = (void *)qmap_cpu_offline, .init = (void *)qmap_init, .exit = (void *)qmap_exit, .timeout_ms = 5000U,
tools/sched_ext/scx_qmap.c +7 -6
@@ -67,12 +67,14 @@ int main(int argc, char **argv)struct bpf_link *link;struct qmap_arena *qa;__u32 test_error_cnt =0 ; +__u64 ecode;int opt;libbpf_set_print (libbpf_print_fn);signal (SIGINT , sigint_handler);signal (SIGTERM , sigint_handler); - +restart: + optind =1 ; skel =SCX_OPS_OPEN (qmap_ops, scx_qmap); skel->rodata->slice_ns =__COMPAT_ENUM_OR_ZERO ("scx_public_consts" ,"SCX_SLICE_DFL" );
@@ -184,11 +186,10 @@ int main(int argc, char **argv) }bpf_link__destroy (link); -UEI_REPORT (skel, uei); + ecode =UEI_REPORT (skel, uei);scx_qmap__destroy (skel); -/* -* scx_qmap implements ops.cpu_on/offline() and doesn't need to restart -* on CPU hotplug events. -*/ + +if (UEI_ECODE_RESTART (ecode)) + goto restart;return 0 ; }
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 15/17 tools/sched_ext: scx_qmap: Add cmask-based idle tracking and cid-based idle pick
In reply to Tejun HeoSwitch qmap's idle-cpu picker from scx_bpf_pick_idle_cpu() to a BPF-side bitmap scan, still under cpu-form struct_ops. qa_idle_cids tracks idle cids (updated in update_idle / cpu_offline) and each task's taskc->cpus_allowed tracks its allowed cids (built in set_cpumask / init_task); select_cpu / enqueue scan the intersection for an idle cid. Callbacks translate cpu <-> cid on entry; cid-qmap-port drops those translations.
The scan is barebone - no core preference or other topology-aware picks like the in-kernel picker - but qmap is a demo and this is enough to exercise the plumbing.
v3: qmap_init() refuses to load when nr_cids exceeds SCX_QMAP_MAX_CPUS;
task_ctx's flex array would otherwise overflow into the next slab entry. (Sashiko)
--- tools/sched_ext/scx_qmap.bpf.c | 137 +++++++++++++++++++++++++++++---- 1 file changed, 121 insertions(+), 16 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +121 -16
@@ -72,6 +72,13 @@ struct { struct qmap_arena __arena qa; +/* +* Global idle-cid tracking, maintained via update_idle / cpu_offline and +* scanned by the direct-dispatch path. Allocated in qmap_init() from one +* arena page, sized to the full cid space. +*/ +struct scx_cmask __arena *qa_idle_cids; +/* Per-queue locks. Each in its own .data section as bpf_res_spin_lock requires. */ __hidden struct bpf_res_spin_lock qa_q_lock0 SEC(".data.qa_q_lock0"); __hidden struct bpf_res_spin_lock qa_q_lock1 SEC(".data.qa_q_lock1");
@@ -132,8 +139,18 @@ struct task_ctx {bool force_local;/* Dispatch directly to local_dsq */ bool highpri;u64 core_sched_seq; +struct scx_cmask cpus_allowed;/* per-task affinity in cid space */ }; +/* +* Slab stride for task_ctx. cpus_allowed's flex array bits[] overlaps the +* tail bytes appended per entry; struct_size() gives the actual per-entry +* footprint. +*/ +#define TASK_CTX_STRIDE \ + struct_size_t(struct task_ctx, cpus_allowed.bits, \ + CMASK_NR_WORDS(SCX_QMAP_MAX_CPUS)) +/* All task_ctx pointers are arena pointers. */ typedef struct task_ctx __arena task_ctx_t;
@@ -161,20 +178,37 @@ static int qmap_spin_lock(struct bpf_res_spin_lock *lock)return 0 ; } -static s32 pick_direct_dispatch_cpu (struct task_struct *p,s32 prev_cpu) +/* +* Try prev_cpu's cid, then scan taskc->cpus_allowed AND qa_idle_cids +* round-robin from prev_cid + 1. Atomic claim retries on race; bounded +* by IDLE_PICK_RETRIES to keep the verifier's insn budget in check. +*/ +#define IDLE_PICK_RETRIES 16 + +static s32 pick_direct_dispatch_cpu (struct task_struct *p,s32 prev_cpu, +task_ctx_t *taskc) { -s32 cpu; +u32 nr_cids =scx_bpf_nr_cids (); +s32 prev_cid, cid; +u32 i;if (!always_enq_immed && p->nr_cpus_allowed ==1 )return prev_cpu; -if (scx_bpf_test_and_clear_cpu_idle (prev_cpu)) + prev_cid =scx_bpf_cpu_to_cid (prev_cpu); +if (cmask_test_and_clear (qa_idle_cids, prev_cid))return prev_cpu; - cpu =scx_bpf_pick_idle_cpu (p->cpus_ptr,0 ); -if (cpu >=0 ) -return cpu; - + cid = prev_cid; + bpf_for(i, 0, IDLE_PICK_RETRIES) { + cid =cmask_next_and_set_wrap (&taskc->cpus_allowed, + qa_idle_cids, cid +1 ); +barrier_var (cid); +if (cid >= nr_cids) +return -1 ; +if (cmask_test_and_clear (qa_idle_cids, cid)) +return scx_bpf_cid_to_cpu (cid); + }return -1 ; }
@@ -286,7 +320,7 @@ s32 BPF_STRUCT_OPS(qmap_select_cpu, struct task_struct *p,if (p->scx.weight <2 && !(p->flags &PF_KTHREAD ))return prev_cpu; - cpu =pick_direct_dispatch_cpu (p, prev_cpu); + cpu =pick_direct_dispatch_cpu (p, prev_cpu, taskc);if (cpu >=0 ) { taskc->force_local = true;
@@ -379,7 +413,7 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)/* if select_cpu() wasn't called, try direct dispatch */ if (!__COMPAT_is_enq_cpu_selected (enq_flags) && - (cpu =pick_direct_dispatch_cpu (p,scx_bpf_task_cpu (p))) >=0 ) { + (cpu =pick_direct_dispatch_cpu (p,scx_bpf_task_cpu (p), taskc)) >=0 ) {__sync_fetch_and_add (&qa.nr_ddsp_from_enq,1 );scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | cpu, slice_ns, enq_flags);return ;
@@ -726,6 +760,10 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init_task, struct task_struct *p, taskc->force_local = false; taskc->highpri = false; taskc->core_sched_seq =0 ; +cmask_init (&taskc->cpus_allowed,0 ,scx_bpf_nr_cids ()); +bpf_rcu_read_lock (); +cmask_from_cpumask (&taskc->cpus_allowed, p->cpus_ptr); +bpf_rcu_read_unlock (); v =bpf_task_storage_get (&task_ctx_stor, p,NULL ,BPF_LOCAL_STORAGE_GET_F_CREATE );
@@ -843,6 +881,48 @@ void BPF_STRUCT_OPS(qmap_cgroup_set_bandwidth, struct cgroup *cgrp, cgrp->kn->id, period_us, quota_us, burst_us); } +void BPF_STRUCT_OPS (qmap_update_idle ,s32 cpu,bool idle) +{ +s32 cid =scx_bpf_cpu_to_cid (cpu); + +QMAP_TOUCH_ARENA (); +if (cid <0 ) +return ; +if (idle) +cmask_set (qa_idle_cids, cid); +else +cmask_clear (qa_idle_cids, cid); +} + +/* +* The cpumask received here is kernel-address memory; walk it bit by bit +* (bpf_cpumask_test_cpu handles the access), convert each set cpu to its +* cid, and populate the arena-resident taskc cmask. +*/ +void BPF_STRUCT_OPS (qmap_set_cpumask ,struct task_struct *p, +const struct cpumask *cpumask) +{ +task_ctx_t *taskc; +u32 nr_cpu_ids =scx_bpf_nr_cpu_ids (); +s32 cpu; + + taskc =lookup_task_ctx (p); +if (!taskc) +return ; + +cmask_zero (&taskc->cpus_allowed); + + bpf_for(cpu, 0, nr_cpu_ids) { +s32 cid; + +if (!bpf_cpumask_test_cpu (cpu, cpumask)) +continue ; + cid =scx_bpf_cpu_to_cid (cpu); +if (cid >=0 ) +__cmask_set (&taskc->cpus_allowed, cid); + } +} +struct monitor_timer {struct bpf_timer timer; };
@@ -992,34 +1072,57 @@ static int lowpri_timerfn(void *map, int *key, struct bpf_timer *timer)s32 BPF_STRUCT_OPS_SLEEPABLE (qmap_init ) { -task_ctx_t *slab; + u8 __arena *slab;u32 nr_pages, key =0 , i;struct bpf_timer *timer;s32 ret; +if (scx_bpf_nr_cids () >SCX_QMAP_MAX_CPUS ) { +scx_bpf_error ("nr_cids=%u exceeds SCX_QMAP_MAX_CPUS=%d" , +scx_bpf_nr_cids (),SCX_QMAP_MAX_CPUS ); +return -EINVAL ; + } +/* * Allocate the task_ctx slab in arena and thread the entire slab onto -* the free list. max_tasks is set by userspace before load. +* the free list. max_tasks is set by userspace before load. Each entry +* is TASK_CTX_STRIDE bytes - task_ctx's trailing cpus_allowed flex +* array extends into the stride tail. */ if (!max_tasks) {scx_bpf_error ("max_tasks must be > 0" );return -EINVAL ; } - nr_pages = (max_tasks *sizeof (struct task_ctx ) +PAGE_SIZE -1 ) /PAGE_SIZE ; + nr_pages = (max_tasks *TASK_CTX_STRIDE +PAGE_SIZE -1 ) /PAGE_SIZE ; slab =bpf_arena_alloc_pages (&arena,NULL , nr_pages,NUMA_NO_NODE ,0 );if (!slab) {scx_bpf_error ("failed to allocate task_ctx slab" );return -ENOMEM ; } - qa.task_ctxs = slab; + qa.task_ctxs = (task_ctx_t *)slab;bpf_for (i,0 ,5 ) qa.fifos[i].idx = i; -bpf_for (i,0 , max_tasks) - slab[i].next_free = (i + 1 < max_tasks) ? &slab[i + 1] : NULL; - qa.task_free_head = &slab[0 ]; + bpf_for(i, 0, max_tasks) { +task_ctx_t *cur = (task_ctx_t *)(slab + i *TASK_CTX_STRIDE ); +task_ctx_t *next = (i +1 < max_tasks) ? + (task_ctx_t *)(slab + (i +1 ) *TASK_CTX_STRIDE ) :NULL ; + cur->next_free = next; + } + qa.task_free_head = (task_ctx_t *)slab; + +/* +* Allocate and initialize the idle cmask. Starts empty - update_idle +* fills it as cpus enter idle. +*/ + qa_idle_cids =bpf_arena_alloc_pages (&arena,NULL ,1 ,NUMA_NO_NODE ,0 ); +if (!qa_idle_cids) { +scx_bpf_error ("failed to allocate idle cmask" ); +return -ENOMEM ; + } +cmask_init (qa_idle_cids,0 ,scx_bpf_nr_cids ()); ret =scx_bpf_create_dsq (SHARED_DSQ ,-1 );if (ret) {
@@ -1104,6 +1207,8 @@ SCX_OPS_DEFINE(qmap_ops, .dispatch = (void *)qmap_dispatch, .tick = (void *)qmap_tick, .core_sched_before = (void *)qmap_core_sched_before, + .set_cpumask = (void *)qmap_set_cpumask, + .update_idle = (void *)qmap_update_idle, .init_task = (void *)qmap_init_task, .exit_task = (void *)qmap_exit_task, .dump = (void *)qmap_dump,
Signature · 1 line
2.54.0
Tejun Heo
reviewerPATCH 16/17 tools/sched_ext: scx_qmap: Port to cid-form struct_ops
In reply to Tejun HeoFlip qmap's struct_ops to bpf_sched_ext_ops_cid. The kernel now passes cids and cmasks to callbacks directly, so the per-callback cpu<->cid translations that the prior patch added drop out and cpu_ctxs[] is reindexed by cid. Cpu-form kfunc calls switch to their cid-form counterparts.
The cpu-only kfuncs (idle/any pick, cpumask iteration) have no cid substitute. Their callers already moved to cmask scans against qa_idle_cids and taskc->cpus_allowed in the prior patch, so the kfunc calls drop here without behavior changes.
set_cmask is wired up via cmask_copy_from_kernel() to copy the kernel-supplied cmask into the arena-resident taskc cmask. The cpuperf monitor iterates the cid-form perf kfuncs.
--- tools/sched_ext/scx_qmap.bpf.c | 231 +++++++++++++++++---------------- tools/sched_ext/scx_qmap.c | 59 ++++++++- tools/sched_ext/scx_qmap.h | 2 +- 3 files changed, 177 insertions(+), 115 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +121 -110
@@ -52,6 +52,28 @@ const volatile bool always_enq_immed;const volatile u32 immed_stress_nth;const volatile u32 max_tasks; +/* +* Optional cid-override test harness. When cid_override_mode is non-zero, +* qmap_init() calls scx_bpf_cid_override() with the caller-supplied arrays +* to exercise the kfunc's acceptance and error paths. +* +* 0 = disabled +* 1 = valid reverse mapping +* 2 = invalid: duplicate cid assignment +* 3 = invalid: non-monotonic shard_start +*/ +const volatile u32 cid_override_mode; +const volatile u32 cid_override_nr_cpus; +const volatile u32 cid_override_nr_shards; +/* +* Arrays live in bss (writable) because scx_bpf_cid_override()'s BPF +* verifier signature treats its len-paired pointer as read/write - rodata +* fails verification with "write into map forbidden". Userspace populates +* them before SCX_OPS_LOAD, same as rodata, and nothing writes them after. +*/ +s32 cid_override_cpu_to_cid[SCX_QMAP_MAX_CPUS ]; +s32 cid_override_shard_start[SCX_QMAP_MAX_CPUS ]; +UEI_DEFINE (uei);/*
@@ -179,25 +201,24 @@ static int qmap_spin_lock(struct bpf_res_spin_lock *lock) }/* -* Try prev_cpu's cid, then scan taskc->cpus_allowed AND qa_idle_cids -* round-robin from prev_cid + 1. Atomic claim retries on race; bounded -* by IDLE_PICK_RETRIES to keep the verifier's insn budget in check. +* Try prev_cid, then scan taskc->cpus_allowed AND qa_idle_cids round-robin +* from prev_cid + 1. Atomic claim retries on race; bounded by +* IDLE_PICK_RETRIES to keep the verifier's insn budget in check. */ #define IDLE_PICK_RETRIES 16 -static s32 pick_direct_dispatch_cpu (struct task_struct *p,s32 prev_cpu, +static s32 pick_direct_dispatch_cid (struct task_struct *p,s32 prev_cid,task_ctx_t *taskc) {u32 nr_cids =scx_bpf_nr_cids (); -s32 prev_cid, cid; +s32 cid;u32 i;if (!always_enq_immed && p->nr_cpus_allowed ==1 ) -return prev_cpu; +return prev_cid; - prev_cid =scx_bpf_cpu_to_cid (prev_cpu);if (cmask_test_and_clear (qa_idle_cids, prev_cid)) -return prev_cpu; +return prev_cid; cid = prev_cid; bpf_for(i, 0, IDLE_PICK_RETRIES) {
@@ -207,7 +228,7 @@ static s32 pick_direct_dispatch_cpu(struct task_struct *p, s32 prev_cpu,if (cid >= nr_cids)return -1 ;if (cmask_test_and_clear (qa_idle_cids, cid)) -return scx_bpf_cid_to_cpu (cid); +return cid; }return -1 ; }
@@ -308,25 +329,25 @@ static void qmap_fifo_remove(task_ctx_t *taskc)bpf_res_spin_unlock (lock); } -s32 BPF_STRUCT_OPS (qmap_select_cpu ,struct task_struct *p, -s32 prev_cpu,u64 wake_flags) +s32 BPF_STRUCT_OPS (qmap_select_cid ,struct task_struct *p, +s32 prev_cid,u64 wake_flags) {task_ctx_t *taskc; -s32 cpu; +s32 cid;if (!(taskc =lookup_task_ctx (p))) -return prev_cpu; +return prev_cid;if (p->scx.weight <2 && !(p->flags &PF_KTHREAD )) -return prev_cpu; +return prev_cid; - cpu =pick_direct_dispatch_cpu (p, prev_cpu, taskc); + cid =pick_direct_dispatch_cid (p, prev_cid, taskc); -if (cpu >=0 ) { +if (cid >=0 ) { taskc->force_local = true; -return cpu; +return cid; }else { -return prev_cpu; +return prev_cid; } }
@@ -350,12 +371,12 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)static u32 user_cnt, kernel_cnt;task_ctx_t *taskc;int idx =weight_to_idx (p->scx.weight); -s32 cpu; +s32 cid;if (enq_flags &SCX_ENQ_REENQ ) {__sync_fetch_and_add (&qa.nr_reenqueued,1 ); -if (scx_bpf_task_cpu (p) ==0 ) -__sync_fetch_and_add (&qa.nr_reenqueued_cpu0,1 ); +if (scx_bpf_task_cid (p) ==0 ) +__sync_fetch_and_add (&qa.nr_reenqueued_cid0,1 ); }if (p->flags &PF_KTHREAD ) {
@@ -388,14 +409,14 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)if (!(++immed_stress_cnt % immed_stress_nth)) { taskc->force_local = false; -scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON |scx_bpf_task_cpu (p), +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON |scx_bpf_task_cid (p), slice_ns, enq_flags);return ; } }/* -* If qmap_select_cpu() is telling us to or this is the last runnable +* If qmap_select_cid() is telling us to or this is the last runnable * task on the CPU, enqueue locally. */ if (taskc->force_local) {
@@ -411,11 +432,11 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)return ; } -/* if select_cpu() wasn't called, try direct dispatch */ +/* if select_cid() wasn't called, try direct dispatch */ if (!__COMPAT_is_enq_cpu_selected (enq_flags) && - (cpu =pick_direct_dispatch_cpu (p,scx_bpf_task_cpu (p), taskc)) >=0 ) { + (cid =pick_direct_dispatch_cid (p,scx_bpf_task_cid (p), taskc)) >=0 ) {__sync_fetch_and_add (&qa.nr_ddsp_from_enq,1 ); -scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | cpu, slice_ns, enq_flags); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | cid, slice_ns, enq_flags);return ; }
@@ -423,15 +444,16 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)* If the task was re-enqueued due to the CPU being preempted by a * higher priority scheduling class, just re-enqueue the task directly * on the global DSQ. As we want another CPU to pick it up, find and -* kick an idle CPU. +* kick an idle cid. */ if (enq_flags &SCX_ENQ_REENQ ) { -s32 cpu; +s32 cid;scx_bpf_dsq_insert (p,SHARED_DSQ ,0 , enq_flags); - cpu =scx_bpf_pick_idle_cpu (p->cpus_ptr,0 ); -if (cpu >=0 ) -scx_bpf_kick_cpu (cpu,SCX_KICK_IDLE ); + cid =cmask_next_and_set_wrap (&taskc->cpus_allowed, + qa_idle_cids,0 ); +if (cid <scx_bpf_nr_cids ()) +scx_bpf_kick_cid (cid,SCX_KICK_IDLE );return ; }
@@ -483,7 +505,8 @@ static void update_core_sched_head_seq(struct task_struct *p)static bool dispatch_highpri (bool from_timer) {struct task_struct *p; -s32 this_cpu =bpf_get_smp_processor_id (); +s32 this_cid =scx_bpf_this_cid (); +u32 nr_cids =scx_bpf_nr_cids ();/* scan SHARED_DSQ and move highpri tasks to HIGHPRI_DSQ */ bpf_for_each (scx_dsq, p,SHARED_DSQ ,0 ) {
@@ -502,21 +525,29 @@ static bool dispatch_highpri(bool from_timer) }/* -* Scan HIGHPRI_DSQ and dispatch until a task that can run on this CPU -* is found. +* Scan HIGHPRI_DSQ and dispatch until a task that can run here is +* found. Prefer this_cid if the task allows it; otherwise RR-scan the +* task's cpus_allowed starting after this_cid. */ bpf_for_each (scx_dsq, p,HIGHPRI_DSQ ,0 ) { +task_ctx_t *taskc;bool dispatched = false; -s32 cpu; +s32 cid; -if (bpf_cpumask_test_cpu (this_cpu, p->cpus_ptr)) - cpu = this_cpu; +if (!(taskc =lookup_task_ctx (p))) +return false; + +if (cmask_test (&taskc->cpus_allowed, this_cid)) + cid = this_cid;else - cpu =scx_bpf_pick_any_cpu (p->cpus_ptr,0 ); + cid =cmask_next_set_wrap (&taskc->cpus_allowed, + this_cid +1 ); +if (cid >= nr_cids) +continue ; -if (scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | cpu, +if (scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | cid,SCX_ENQ_PREEMPT )) { -if (cpu == this_cpu) { +if (cid == this_cid) { dispatched = true;__sync_fetch_and_add (&qa.nr_expedited_local,1 ); }else {
@@ -535,7 +566,7 @@ static bool dispatch_highpri(bool from_timer)return false; } -void BPF_STRUCT_OPS (qmap_dispatch ,s32 cpu,struct task_struct *prev) +void BPF_STRUCT_OPS (qmap_dispatch ,s32 cid,struct task_struct *prev) {struct task_struct *p; struct cpu_ctx __arena *cpuc;
@@ -563,7 +594,7 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev) } } - cpuc = &qa.cpu_ctxs[bpf_get_smp_processor_id ()]; + cpuc = &qa.cpu_ctxs[scx_bpf_this_cid ()];for (i =0 ; i <5 ; i++) {/* Advance the dispatch cursor and pick the fifo. */
@@ -628,8 +659,8 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)* document this class of issue -- other schedulers * seeing similar warnings can use this as a reference. */ -if (!bpf_cpumask_test_cpu (cpu, p->cpus_ptr)) -scx_bpf_kick_cpu (scx_bpf_task_cpu (p),0 ); +if (!cmask_test (&taskc->cpus_allowed, cid)) +scx_bpf_kick_cid (scx_bpf_task_cid (p),0 ); batch--; cpuc->dsp_cnt--;
@@ -668,7 +699,7 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)void BPF_STRUCT_OPS (qmap_tick ,struct task_struct *p) { - struct cpu_ctx __arena *cpuc = &qa.cpu_ctxs[bpf_get_smp_processor_id()]; + struct cpu_ctx __arena *cpuc = &qa.cpu_ctxs[scx_bpf_this_cid()];int idx;/*
@@ -680,7 +711,7 @@ void BPF_STRUCT_OPS(qmap_tick, struct task_struct *p) idx =weight_to_idx (cpuc->avg_weight); cpuc->cpuperf_target = qidx_to_cpuperf_target[idx]; -scx_bpf_cpuperf_set (scx_bpf_task_cpu (p), cpuc->cpuperf_target); +scx_bpf_cidperf_set (scx_bpf_task_cid (p), cpuc->cpuperf_target); }/*
@@ -828,9 +859,9 @@ void BPF_STRUCT_OPS(qmap_dump, struct scx_dump_ctx *dctx) } } -void BPF_STRUCT_OPS (qmap_dump_cpu ,struct scx_dump_ctx *dctx,s32 cpu,bool idle) +void BPF_STRUCT_OPS (qmap_dump_cid ,struct scx_dump_ctx *dctx,s32 cid,bool idle) { - struct cpu_ctx __arena *cpuc = &qa.cpu_ctxs[cpu]; + struct cpu_ctx __arena *cpuc = &qa.cpu_ctxs[cid];if (suppress_dump || idle)return ;
@@ -881,46 +912,24 @@ void BPF_STRUCT_OPS(qmap_cgroup_set_bandwidth, struct cgroup *cgrp, cgrp->kn->id, period_us, quota_us, burst_us); } -void BPF_STRUCT_OPS (qmap_update_idle ,s32 cpu,bool idle) +void BPF_STRUCT_OPS (qmap_update_idle ,s32 cid,bool idle) { -s32 cid =scx_bpf_cpu_to_cid (cpu); -QMAP_TOUCH_ARENA (); -if (cid <0 ) -return ;if (idle)cmask_set (qa_idle_cids, cid);else cmask_clear (qa_idle_cids, cid); } -/* -* The cpumask received here is kernel-address memory; walk it bit by bit -* (bpf_cpumask_test_cpu handles the access), convert each set cpu to its -* cid, and populate the arena-resident taskc cmask. -*/ -void BPF_STRUCT_OPS (qmap_set_cpumask ,struct task_struct *p, -const struct cpumask *cpumask) +void BPF_STRUCT_OPS (qmap_set_cmask ,struct task_struct *p, +const struct scx_cmask *cmask) {task_ctx_t *taskc; -u32 nr_cpu_ids =scx_bpf_nr_cpu_ids (); -s32 cpu; taskc =lookup_task_ctx (p);if (!taskc)return ; - -cmask_zero (&taskc->cpus_allowed); - - bpf_for(cpu, 0, nr_cpu_ids) { -s32 cid; - -if (!bpf_cpumask_test_cpu (cpu, cpumask)) -continue ; - cid =scx_bpf_cpu_to_cid (cpu); -if (cid >=0 ) -__cmask_set (&taskc->cpus_allowed, cid); - } +cmask_copy_from_kernel (&taskc->cpus_allowed, cmask); }struct monitor_timer {
@@ -935,59 +944,49 @@ struct { } monitor_timer SEC(".maps");/* -* Print out the min, avg and max performance levels of CPUs every second to -* demonstrate the cpuperf interface. +* Aggregate cidperf across the first nr_online_cids cids. Post-hotplug +* the first-N-are-online invariant drifts, so some cap/cur values may +* be stale. For this demo monitor that's fine; the scheduler exits on +* the enable-time hotplug_seq mismatch and userspace restarts, which +* rebuilds the layout. */ static void monitor_cpuperf (void ) { -u32 nr_cpu_ids; +u32 nr_online =scx_bpf_nr_online_cids ();u64 cap_sum =0 , cur_sum =0 , cur_min =SCX_CPUPERF_ONE , cur_max =0 ;u64 target_sum =0 , target_min =SCX_CPUPERF_ONE , target_max =0 ; -const struct cpumask *online; -int i, nr_online_cpus =0 ; - - nr_cpu_ids =scx_bpf_nr_cpu_ids (); - online =scx_bpf_get_online_cpumask (); - - bpf_for(i, 0, nr_cpu_ids) { - struct cpu_ctx __arena *cpuc = &qa.cpu_ctxs[i]; -u32 cap, cur; +s32 cid; -if (!bpf_cpumask_test_cpu (i, online)) -continue ; - nr_online_cpus++; +QMAP_TOUCH_ARENA (); -/* collect the capacity and current cpuperf */ - cap =scx_bpf_cpuperf_cap (i); - cur =scx_bpf_cpuperf_cur (i); + bpf_for(cid, 0, nr_online) { + struct cpu_ctx __arena *cpuc = &qa.cpu_ctxs[cid]; +u32 cap =scx_bpf_cidperf_cap (cid); +u32 cur =scx_bpf_cidperf_cur (cid); +u32 target; cur_min = cur < cur_min ? cur : cur_min; cur_max = cur > cur_max ? cur : cur_max; -/* -* $cur is relative to $cap. Scale it down accordingly so that -* it's in the same scale as other CPUs and $cur_sum/$cap_sum -* makes sense. -*/ - cur_sum += cur * cap /SCX_CPUPERF_ONE ; + cur_sum += (u64 )cur * cap /SCX_CPUPERF_ONE ; cap_sum += cap; -/* collect target */ - cur = cpuc->cpuperf_target; - target_sum += cur; - target_min = cur < target_min ? cur : target_min; - target_max = cur > target_max ? cur : target_max; + target = cpuc->cpuperf_target; + target_sum += target; + target_min = target < target_min ? target : target_min; + target_max = target > target_max ? target : target_max; } +if (!nr_online || !cap_sum) +return ; + qa.cpuperf_min = cur_min; qa.cpuperf_avg = cur_sum *SCX_CPUPERF_ONE / cap_sum; qa.cpuperf_max = cur_max; qa.cpuperf_target_min = target_min; - qa.cpuperf_target_avg = target_sum / nr_online_cpus; + qa.cpuperf_target_avg = target_sum / nr_online; qa.cpuperf_target_max = target_max; - -scx_bpf_put_cpumask (online); }/*
@@ -1083,6 +1082,18 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)return -EINVAL ; } +/* +* cid-override test hook. Must run before anything that reads the +* cid space (scx_bpf_nr_cids, cmask_init, etc.). On invalid input, +* the kfunc calls scx_error() which aborts the scheduler. +*/ +if (cid_override_mode) { +scx_bpf_cid_override ((const s32 *)cid_override_cpu_to_cid, + cid_override_nr_cpus *sizeof (s32), + (const s32 *)cid_override_shard_start, + cid_override_nr_shards *sizeof (s32)); + } +/* * Allocate the task_ctx slab in arena and thread the entire slab onto * the free list. max_tasks is set by userspace before load. Each entry
@@ -1199,20 +1210,20 @@ void BPF_STRUCT_OPS(qmap_sub_detach, struct scx_sub_detach_args *args) } } -SCX_OPS_DEFINE(qmap_ops, +SCX_OPS_CID_DEFINE(qmap_ops, .flags = SCX_OPS_ENQ_EXITING | SCX_OPS_TID_TO_TASK, - .select_cpu = (void *)qmap_select_cpu, + .select_cid = (void *)qmap_select_cid, .enqueue = (void *)qmap_enqueue, .dequeue = (void *)qmap_dequeue, .dispatch = (void *)qmap_dispatch, .tick = (void *)qmap_tick, .core_sched_before = (void *)qmap_core_sched_before, - .set_cpumask = (void *)qmap_set_cpumask, + .set_cmask = (void *)qmap_set_cmask, .update_idle = (void *)qmap_update_idle, .init_task = (void *)qmap_init_task, .exit_task = (void *)qmap_exit_task, .dump = (void *)qmap_dump, - .dump_cpu = (void *)qmap_dump_cpu, + .dump_cid = (void *)qmap_dump_cid, .dump_task = (void *)qmap_dump_task, .cgroup_init = (void *)qmap_cgroup_init, .cgroup_set_weight = (void *)qmap_cgroup_set_weight,
tools/sched_ext/scx_qmap.c +55 -4
96 lines not on this page · raw message
tools/sched_ext/scx_qmap.h +1 -1
@@ -45,7 +45,7 @@ struct qmap_fifo {struct qmap_arena {/* userspace-visible stats */ -__u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_reenqueued_cpu0; +__u64 nr_enqueued, nr_dispatched, nr_reenqueued, nr_reenqueued_cid0;__u64 nr_dequeued, nr_ddsp_from_enq;__u64 nr_core_sched_execed;__u64 nr_expedited_local, nr_expedited_remote;
Signature · 1 line
2.54.0
Re: [PATCH 16/17] tools/sched_ext: scx_qmap: Port to cid-form struct_ops
Quoted 1 line · Tejun Heo
/* @@ -1083,6 +1082,18 @@s32·BPF_STRUCT_OPS_SLEEPABLE(qmap_init) › › return·-EINVAL; › } +› /*+› ·*·cid- override·test·hook.·Must·run·before·anything·that·reads·the+› ·*·cid·space·(scx_bpf_nr_cids,·cmask_init,·etc.).·On·invalid·input,+› ·*·the·kfunc·calls·scx_error()·which·aborts·the·scheduler.+› ·*/+› if·(cid_override_mode)·{+› › scx_bpf_cid_override((const·s32·*)cid_override_cpu_to_cid,+› › › › ·····cid_override_nr_cpus·*·sizeof(s32),+› › › › ·····(const·s32·*)cid_override_shard_start,+› › › › ·····cid_override_nr_shards·*·sizeof(s32));+› }+
This cause the following compilation error due to argument mismatch:
scx_qmap.bpf.c:1093:10: error: too many arguments to function call, expected 2, have 4
1091 | scx_bpf_cid_override((const s32
*)cid_override_cpu_to_cid,
| ~~~~~~~~~~~~~~~~~~~~
1092 | cid_override_nr_cpus *
sizeof(s32),
1093 | (const s32
*)cid_override_shard_start,
|
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1094 | cid_override_nr_shards *
sizeof(s32));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~/home/changwoo/ws-multics69/dev/linux-tj/tools/sched_ext/include/scx/compat.bpf.h:130:20: note:
'scx_bpf_cid_override' declared here
130 | static inline void scx_bpf_cid_override(const s32 *cpu_to_cid,
u32 cpu_to_cid__sz)
| ^
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The correct one should be as follows:
Quoted 2 lines
scx_bpf_cid_override((const·s32·*)cid_override_cpu_to_cid,+› › › › ·····cid_override_nr_cpus * sizeof(s32));
Andrea Righi
reviewerRe: [PATCH 16/17] tools/sched_ext: scx_qmap: Port to cid-form struct_ops
Hello,
Quoted 35 lines · Changwoo Min
Quoted 10 lines · Tejun Heo
/* @@ -1083,6 +1082,18 @@s32·BPF_STRUCT_OPS_SLEEPABLE(qmap_init) › › return·-EINVAL; › } +› /*+› ·*·cid- override·test·hook.·Must·run·before·anything·that·reads·the+› ·*·cid·space·(scx_bpf_nr_cids,·cmask_init,·etc.).·On·invalid·input,+› ·*·the·kfunc·calls·scx_error()·which·aborts·the·scheduler.+› ·*/+› if·(cid_override_mode)·{+› › scx_bpf_cid_override((const·s32·*)cid_override_cpu_to_cid,+› › › › ·····cid_override_nr_cpus·*·sizeof(s32),+› › › › ·····(const·s32·*)cid_override_shard_start,+› › › › ·····cid_override_nr_shards·*·sizeof(s32));+› }+
This cause the following compilation error due to argument mismatch:
scx_qmap.bpf.c:1093:10: error: too many arguments to function call, expected 2, have 4
1091 | scx_bpf_cid_override((const s32
*)cid_override_cpu_to_cid,| ~~~~~~~~~~~~~~~~~~~~1092 |
cid_override_nr_cpus*
sizeof(s32),
1093 | (const s32
*)cid_override_shard_start,| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1094 | cid_override_nr_shards * sizeof(s32)); | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/changwoo/ws-multics69/dev/linux-tj/tools/sched_ext/include/scx/compat.bpf.h:130:20:
note:
'scx_bpf_cid_override' declared here130 | static inline void scx_bpf_cid_override(const s32
*cpu_to_cid, u32cpu_to_cid__sz)
Quoted 8 lines
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The correct one should be as follows:
Quoted 2 lines
scx_bpf_cid_override((const·s32·*)cid_override_cpu_to_cid,+› › › › ·····cid_override_nr_cpus * sizeof(s32));
Reviewed-by Changwoo Min <[email protected]>
And after fixing scx_bpf_cid_override() I'm also getting this with scx_qmap -C shuffle:
0: R1=ctx() R10=fp0
; s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init) @ scx_qmap.bpf.c:1069
0: (b4) w1 = 0 ; R1=0
; u32 nr_pages, key = 0, i; @ scx_qmap.bpf.c:1072
1: (63) *(u32 *)(r10 -4) = r1 ; R1=0 R10=fp0 fp-8=0000????
; if (scx_bpf_nr_cids() > SCX_QMAP_MAX_CPUS) { @ scx_qmap.bpf.c:1076
2: (85) call scx_bpf_nr_cids#110275 ; R0=scalar()
3: (a6) if w0 < 0x401 goto pc+14 18: R10=fp0 fp-8=0000pppp
; if (cid_override_mode) { @ scx_qmap.bpf.c:1087
18: (18) r1 = 0xffffc90000322260 ; R1=map_value(map=scx_qmap.rodata,ks=4,vs=964,imm=608)
20: (61) r1 = *(u32 *)(r1 +0) ;
21: (05) goto pc+0
; scx_bpf_nr_cpu_ids() * (u32)sizeof(s32)); @ scx_qmap.bpf.c:1090
22: (85) call scx_bpf_nr_cpu_ids#110276 ; R0=scalar()
; if (bpf_ksym_exists(scx_bpf_cid_override___compat)) @ compat.bpf.h:132
23: (18) r1 = 0xffffffff81464430 ; R1=rdonly_mem(sz=0)
25: (15) if r1 == 0x0 goto pc+5 ; R1=rdonly_mem(sz=0)
; scx_bpf_nr_cpu_ids() * (u32)sizeof(s32)); @ scx_qmap.bpf.c:1090
26: (64) w0 <<= 2 ; R0=scalar(smin=0,smax=umax=umax32=0xfffffffc,smax32=0x7ffffffc,var_off=(0x0; 0xfffffffc))
; return scx_bpf_cid_override___compat(cpu_to_cid, cpu_to_cid__sz); @ compat.bpf.h:133
27: (18) r1 = 0xffffc90001526000 ; R1=map_value(map=scx_qmap.bss,ks=4,vs=4128)
29: (bc) w2 = w0 ; R0=scalar(id=2,smin=0,smax=umax=umax32=0xfffffffc,smax32=0x7ffffffc,var_off=(0x0; 0xfffffffc)) R2=scalar(id=2,smin=0,smax=umax=umax32=0xfffffffc,smax32=0x7ffffffc,var_off=(0x0; 0xfffffffc))
30: (85) call scx_bpf_cid_override#110197
R2 unbounded memory access, use 'var &= const' or 'if (var < const)'
arg#0 arg#1 memory, len pair leads to invalid memory access
processed 28 insns (limit 1000000) max_states_per_insn 0 total_states 2 peak_states 2 mark_read 0
The following seems to fix everything for me.
Thanks,
-Andrea
tools/sched_ext/scx_qmap.bpf.c | 26 +++++++++++++++++--------- tools/sched_ext/scx_qmap.c | 16 ++-------------- 2 files changed, 19 insertions(+), 23 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +17 -9
@@ -63,8 +63,6 @@ const volatile u32 max_tasks;* 3 = invalid: non-monotonic shard_start */ const volatile u32 cid_override_mode; -const volatile u32 cid_override_nr_cpus; -const volatile u32 cid_override_nr_shards;/* * Arrays live in bss (writable) because scx_bpf_cid_override()'s BPF * verifier signature treats its len-paired pointer as read/write - rodata
@@ -72,7 +70,6 @@ const volatile u32 cid_override_nr_shards;* them before SCX_OPS_LOAD, same as rodata, and nothing writes them after. */ s32 cid_override_cpu_to_cid[SCX_QMAP_MAX_CPUS ]; -s32 cid_override_shard_start[SCX_QMAP_MAX_CPUS ];UEI_DEFINE (uei);
@@ -1073,12 +1070,25 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init) { u8 __arena *slab;u32 nr_pages, key =0 , i; +u32 nr_cids, nr_cpu_ids;struct bpf_timer *timer;s32 ret; -if (scx_bpf_nr_cids () >SCX_QMAP_MAX_CPUS ) { + nr_cids =scx_bpf_nr_cids (); + nr_cpu_ids =scx_bpf_nr_cpu_ids (); + +/* +* Separate compares so the verifier tracks each upper bound; needed for +* scx_bpf_cid_override(ptr, nr_cpu_ids * sizeof(s32)) vs bss array size. +*/ +if (nr_cids >SCX_QMAP_MAX_CPUS ) {scx_bpf_error ("nr_cids=%u exceeds SCX_QMAP_MAX_CPUS=%d" , -scx_bpf_nr_cids (),SCX_QMAP_MAX_CPUS ); + nr_cids,SCX_QMAP_MAX_CPUS ); +return -EINVAL ; + } +if (nr_cpu_ids >SCX_QMAP_MAX_CPUS ) { +scx_bpf_error ("nr_cpu_ids=%u exceeds SCX_QMAP_MAX_CPUS=%d" , + nr_cpu_ids,SCX_QMAP_MAX_CPUS );return -EINVAL ; }
@@ -1089,9 +1099,7 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)*/ if (cid_override_mode) {scx_bpf_cid_override ((const s32 *)cid_override_cpu_to_cid, - cid_override_nr_cpus *sizeof (s32), - (const s32 *)cid_override_shard_start, - cid_override_nr_shards *sizeof (s32)); + nr_cpu_ids * (u32 )sizeof (s32)); }/*
@@ -1133,7 +1141,7 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)scx_bpf_error ("failed to allocate idle cmask" );return -ENOMEM ; } -cmask_init (qa_idle_cids,0 ,scx_bpf_nr_cids ()); +cmask_init (qa_idle_cids,0 , nr_cids); ret =scx_bpf_create_dsq (SHARED_DSQ ,-1 );if (ret) {
tools/sched_ext/scx_qmap.c +2 -14
@@ -155,7 +155,6 @@ int main(int argc, char **argv)case 'C' : {u32 nr_cpus =libbpf_num_possible_cpus ();u32 mode, i; -s32 shard_sz =4 ;if (!strcmp (optarg,"shuffle" )) mode =1 ;
@@ -168,7 +167,6 @@ int main(int argc, char **argv)return 1 ; } skel->rodata->cid_override_mode = mode; - skel->rodata->cid_override_nr_cpus = nr_cpus;/* shuffle: reversed cpu_to_cid, bad-dup: dup cid 0, bad-mono: identity */ for (i =0 ; i < nr_cpus; i++) {
@@ -179,19 +177,9 @@ int main(int argc, char **argv) }if (mode ==2 && nr_cpus >=2 ) skel->bss->cid_override_cpu_to_cid[1 ] =0 ; +if (mode ==3 ) + skel->bss->cid_override_cpu_to_cid[0 ] = (s32 )nr_cpus; -/* shards of shard_sz each */ - skel->rodata->cid_override_nr_shards = (nr_cpus + shard_sz -1 ) / shard_sz; -for (i =0 ; i < skel->rodata->cid_override_nr_shards; i++) - skel->bss->cid_override_shard_start[i] = i * shard_sz; - -if (mode ==3 && skel->rodata->cid_override_nr_shards >=3 ) { -/* swap [1] and [2] so shard_start is not monotonically increasing */ -s32 tmp = skel->bss->cid_override_shard_start[1 ]; - skel->bss->cid_override_shard_start[1 ] = - skel->bss->cid_override_shard_start[2 ]; - skel->bss->cid_override_shard_start[2 ] = tmp; - }break ; }case 'v' :
Tejun Heo
reviewerRe: [PATCH 16/17] tools/sched_ext: scx_qmap: Port to cid-form struct_ops
Hello,
Thanks. The qmap caller was written against a follow-up patch that got cut from v3. v4 will fold in your fix.
Thanks.
--
tejun
Tejun Heo
reviewerPATCH 17/17 sched_ext: Require cid-form struct_ops for sub-sched support
In reply to Tejun HeoSub-scheduler support is tied to the cid-form struct_ops: sub_attach / sub_detach will communicate allocation via cmask, and the hierarchy assumes all participants share a single topological cid space. A cpu-form root that accepts sub-scheds would need cpu <-> cid translation on every cross-sched interaction, defeating the purpose.
Enforce this at validate_ops():
- A sub-scheduler (scx_parent(sch) non-NULL) must be cid-form.
- A root that exposes sub_attach / sub_detach must be cid-form.
scx_qmap, which is currently the only scheduler demoing sub-sched support, was converted to cid-form in the preceding patch, so this doesn't cause breakage.
--- kernel/sched/ext.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
kernel/sched/ext.c +17 -0
@@ -6847,6 +6847,23 @@ static int validate_ops(struct scx_sched *sch, const struct sched_ext_ops *ops)if (!sch->is_cid_type && (ops->cpu_acquire || ops->cpu_release))pr_warn ("ops->cpu_acquire/release() are deprecated, use sched_switch TP instead\n" ); +/* +* Sub-scheduler support is tied to the cid-form struct_ops. A sub-sched +* attaches through a cid-form-only interface (sub_attach/sub_detach), +* and a root that accepts sub-scheds must expose cid-form state to +* them. Reject cpu-form schedulers on either side. +*/ +if (!sch->is_cid_type) { +if (scx_parent (sch)) { +scx_error (sch,"sub-sched requires cid-form struct_ops" ); +return -EINVAL ; + } +if (ops->sub_attach || ops->sub_detach) { +scx_error (sch,"sub_attach/sub_detach requires cid-form struct_ops" ); +return -EINVAL ; + } + } +return 0 ; }
Signature · 1 line
2.54.0
Thank you, Tejun, for the patchset.
I like it! I made some comments on patches 10 and 16.
Quoted 102 lines · Tejun Heo
Hello,
v3 (all from the Sashiko AI review at
https://sashiko.dev/#/patchset/20260424172721.3458520-1-tj%40kernel.org):- cid: drop leaked
cpus_read_lock()onscx_cid_init()failure;BUILD_BUG_ONtightened to NR_CPUS<=8192 to match the BPF cmask helpers'CMASK_MAX_WORDScoverage.
- bpf-struct-size: useoffsetof()instruct_size()to match the kernel <linux/overflow.h> macro semantics (no inflation from trailing struct padding).
- cmask:cmask_copy_from_kernel()validates src->base==0 via probe-read;nr_bitscheck is bit-level rather than rounded-up word-count.
- cid-qmap-idle:qmap_init()refuses to load whenscx_bpf_nr_cids()exceedsSCX_QMAP_MAX_CPUS; thetask_ctxflex array would otherwise overflow into the next slab entry.v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]This patchset introduces topological CPU IDs (cids) - dense, topology-ordered cpu identifiers - and an alternative cid-form
struct_opstype that lets BPF schedulers operate in cid space directly.Key pieces:
- cid space:
scx_cid_init()walks nodes * LLCs * cores * threads and packs a dense cid mapping. The mapping can be overridden viascx_bpf_cid_override(). See "Topological CPU IDs" inext_cid.hfor the model.- cmask: a base-windowed bitmap over cid space. Kernel and BPF helpers with identical semantics. Used by
scx_qmapfor per-task affinity and idle-cid tracking; meant to be the substrate for sub-sched cid allocation.-
bpf_sched_ext_ops_cid: a parallelstruct_opstype whose callbacks take cids/cmasks instead of cpus/cpumasks. Kernel translates at the boundary viascx_cpu_arg()/scx_cpu_ret(); the two struct types share offsets up through @priv (verified byBUILD_BUG_ON) so the union view inscx_schedworks without function-pointer casts. Sub-sched support is tied to
cid-form:validate_ops()rejects cpu-form sub-scheds and cpu-form roots that exposesub_attach/sub_detach.- cid-form kfuncs:
scx_bpf_kick_cid, scx_bpf_cidperf_{cap,cur,set},scx_bpf_cid_curr,scx_bpf_task_cid,scx_bpf_this_cid, scx_bpf_nr_{cids,online_cids},scx_bpf_cid_to_cpu,scx_bpf_cpu_to_cid. A cid-form program may not call cpu-only kfuncs (enforced at verifier load viascx_kfunc_context_filter); the reverse is intentionally permissive to ease migration.-
scx_qmapport:scx_qmapis converted to cid-form. It uses the cmask-based idle picker, per-task cid-spacecpus_allowed, and cid-form kfuncs throughout. Sub-sched dispatching viascx_bpf_sub_dispatch()continues to work.v3 re-tested on the 16-cpu QEMU: cid-form
scx_qmapunder stress-ng plus reload cycles, hotplug auto-restart, and sub-sched (rootscx_qmap+ cgroup-scopedscx_qmapchild). Clean.Based on sched_ext/for-7.2 (
4939721aad2e).0001-sched_ext-Add-ext_types.h-for-early-subsystem-wide-d.patch 0002-sched_ext-Rename-ops_cpu_valid-to-scx_cpu_valid-and-.patch 0003-sched_ext-Move-scx_exit-scx_error-and-friends-to-ext.patch 0004-sched_ext-Shift-scx_kick_cpu-validity-check-to-scx_b.patch 0005-sched_ext-Relocate-cpu_acquire-cpu_release-to-end-of.patch 0006-sched_ext-Make-scx_enable-take-scx_enable_cmd.patch 0007-sched_ext-Add-topological-CPU-IDs-cids.patch 0008-sched_ext-Add-scx_bpf_cid_override-kfunc.patch 0009-tools-sched_ext-Add-struct_size-helpers-to-common.bp.patch 0010-sched_ext-Add-cmask-a-base-windowed-bitmap-over-cid-.patch 0011-sched_ext-Add-cid-form-kfunc-wrappers-alongside-cpu-.patch 0012-sched_ext-Add-bpf_sched_ext_ops_cid-struct_ops-type.patch 0013-sched_ext-Forbid-cpu-form-kfuncs-from-cid-form-sched.patch 0014-tools-sched_ext-scx_qmap-Restart-on-hotplug-instead-.patch 0015-tools-sched_ext-scx_qmap-Add-cmask-based-idle-tracki.patch 0016-tools-sched_ext-scx_qmap-Port-to-cid-form-struct_ops.patch 0017-sched_ext-Require-cid-form-struct_ops-for-sub-sched-.patch
Git tree:
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.gitscx-cid-v3kernel/sched/build_policy.c | 3 + kernel/sched/ext.c | 651 ++++++++++++++++++++++++++---- kernel/sched/ext_cid.c | 409 +++++++++++++++++++ kernel/sched/ext_cid.h | 164 ++++++++ kernel/sched/ext_idle.c | 8 +- kernel/sched/ext_internal.h | 205 +++++++--- kernel/sched/ext_types.h | 104 +++++ tools/sched_ext/include/scx/cid.bpf.h | 667 +++++++++++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 23 ++ tools/sched_ext/include/scx/compat.bpf.h | 24 ++ tools/sched_ext/scx_qmap.bpf.c | 346 +++++++++------- tools/sched_ext/scx_qmap.c | 70 +++- tools/sched_ext/scx_qmap.h | 2 +- 13 files changed, 2391 insertions(+), 285 deletions(-)Thanks.
--
tejun
Andrea Righi
reviewerHi Tejun,
Quoted 44 lines · Tejun Heo
Hello,
v3 (all from the Sashiko AI review at
https://sashiko.dev/#/patchset/20260424172721.3458520-1-tj%40kernel.org):- cid: drop leaked
cpus_read_lock()onscx_cid_init()failure;BUILD_BUG_ONtightened to NR_CPUS<=8192 to match the BPF cmask helpers'CMASK_MAX_WORDScoverage.
- bpf-struct-size: useoffsetof()instruct_size()to match the kernel <linux/overflow.h> macro semantics (no inflation from trailing struct padding).
- cmask:cmask_copy_from_kernel()validates src->base==0 via probe-read;nr_bitscheck is bit-level rather than rounded-up word-count.
- cid-qmap-idle:qmap_init()refuses to load whenscx_bpf_nr_cids()exceedsSCX_QMAP_MAX_CPUS; thetask_ctxflex array would otherwise overflow into the next slab entry.v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]This patchset introduces topological CPU IDs (cids) - dense, topology-ordered cpu identifiers - and an alternative cid-form
struct_opstype that lets BPF schedulers operate in cid space directly.Key pieces:
- cid space:
scx_cid_init()walks nodes * LLCs * cores * threads and packs a dense cid mapping. The mapping can be overridden viascx_bpf_cid_override(). See "Topological CPU IDs" inext_cid.hfor the model.- cmask: a base-windowed bitmap over cid space. Kernel and BPF helpers with identical semantics. Used by
scx_qmapfor per-task affinity and idle-cid tracking; meant to be the substrate for sub-sched cid allocation.-
bpf_sched_ext_ops_cid: a parallelstruct_opstype whose callbacks take cids/cmasks instead of cpus/cpumasks. Kernel translates at the boundary viascx_cpu_arg()/scx_cpu_ret(); the two struct types share offsets up through @priv (verified byBUILD_BUG_ON) so the union view inscx_schedworks without function-pointer casts. Sub-sched support is tied to
cid-form:validate_ops()rejects cpu-form sub-scheds and cpu-form roots that exposesub_attach/sub_detach.
Quoted 6 lines
- cid-form kfuncs:
scx_bpf_kick_cid, scx_bpf_cidperf_{cap,cur,set},scx_bpf_cid_curr,scx_bpf_task_cid,scx_bpf_this_cid, scx_bpf_nr_{cids,online_cids},scx_bpf_cid_to_cpu,scx_bpf_cpu_to_cid. A cid-form program may not call cpu-only kfuncs (enforced at verifier load viascx_kfunc_context_filter); the reverse is intentionally permissive to ease migration.
So, IIUC scx schedulers attached to bpf_sched_ext_ops_cid can't use the built-in idle CPU selection kfuncs (ext_idle.c), right?
And that also means sub-sched support => no built-in idle CPU selection. That's a bit unfortunate... I guess we could implement a similar logic in cid/cmask space, maybe in BPF.
Thanks,
-Andrea
Quoted 54 lines · Andrea Righi
Hi Tejun,
Quoted 44 lines · Tejun Heo
Hello,
v3 (all from the Sashiko AI review at
https://sashiko.dev/#/patchset/20260424172721.3458520-1-tj%40kernel.org):- cid: drop leaked
cpus_read_lock()onscx_cid_init()failure;BUILD_BUG_ONtightened to NR_CPUS<=8192 to match the BPF cmask helpers'CMASK_MAX_WORDScoverage.
- bpf-struct-size: useoffsetof()instruct_size()to match the kernel <linux/overflow.h> macro semantics (no inflation from trailing struct padding).
- cmask:cmask_copy_from_kernel()validates src->base==0 via probe-read;nr_bitscheck is bit-level rather than rounded-up word-count.
- cid-qmap-idle:qmap_init()refuses to load whenscx_bpf_nr_cids()exceedsSCX_QMAP_MAX_CPUS; thetask_ctxflex array would otherwise overflow into the next slab entry.v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]This patchset introduces topological CPU IDs (cids) - dense, topology-ordered cpu identifiers - and an alternative cid-form
struct_opstype that lets BPF schedulers operate in cid space directly.Key pieces:
- cid space:
scx_cid_init()walks nodes * LLCs * cores * threads and packs a dense cid mapping. The mapping can be overridden viascx_bpf_cid_override(). See "Topological CPU IDs" inext_cid.hfor the model.- cmask: a base-windowed bitmap over cid space. Kernel and BPF helpers with identical semantics. Used by
scx_qmapfor per-task affinity and idle-cid tracking; meant to be the substrate for sub-sched cid allocation.-
bpf_sched_ext_ops_cid: a parallelstruct_opstype whose callbacks take cids/cmasks instead of cpus/cpumasks. Kernel translates at the boundary viascx_cpu_arg()/scx_cpu_ret(); the two struct types share offsets up through @priv (verified byBUILD_BUG_ON) so the union view inscx_schedworks without function-pointer casts. Sub-sched support is tied to
cid-form:validate_ops()rejects cpu-form sub-scheds and cpu-form roots that exposesub_attach/sub_detach.Quoted 6 lines
- cid-form kfuncs:
scx_bpf_kick_cid, scx_bpf_cidperf_{cap,cur,set},scx_bpf_cid_curr,scx_bpf_task_cid,scx_bpf_this_cid, scx_bpf_nr_{cids,online_cids},scx_bpf_cid_to_cpu,scx_bpf_cpu_to_cid. A cid-form program may not call cpu-only kfuncs (enforced at verifier load viascx_kfunc_context_filter); the reverse is intentionally permissive to ease migration.
Quoted 6 lines
So, IIUC scx schedulers attached to
bpf_sched_ext_ops_cidcan't use the built-in idle CPU selection kfuncs (ext_idle.c), right?And that also means sub-sched support => no built-in idle CPU selection. That's a bit unfortunate... I guess we could implement a similar logic in cid/cmask space, maybe in BPF.
And apart from this and the other comments in PATCH 8/17 and PATCH 16/17, everything else looks good to me.
Thanks,
-Andrea
Tejun Heo
reviewerHello,
Yeah. Expanding the in-kernel idle selector to be sub-sched-aware is hard - it has to respect each sub-sched's cid window, partition constraints, etc. Cmask in arena makes the same logic a lot easier to express on the BPF side, so I think the right path is building that infra in BPF rather than extending ext_idle.c.
Thanks.
--
tejun