sched_ext: Enhance built-in idle selection with allowed CPUs
Subsystems (4)
Andrea Righi
reviewerPATCH v6 1/1 sched_ext: Enhance built-in idle selection with allowed CPUs
Many scx schedulers implement their own hard or soft-affinity rules to support topology characteristics, such as heterogeneous architectures (e.g., big.LITTLE, P-cores/E-cores), or to categorize tasks based on specific properties (e.g., running certain tasks only in a subset of CPUs).
Currently, there is no mechanism that allows to use the built-in idle CPU selection policy to an arbitrary subset of CPUs. As a result, schedulers often implement their own idle CPU selection policies, which are typically similar to one another, leading to a lot of code duplication.
To address this, extend the built-in idle CPU selection policy introducing the concept of allowed CPUs.
With this concept, BPF schedulers can apply the built-in idle CPU selection policy to a subset of allowed CPUs, allowing them to implement their own hard/soft-affinity rules while still using the topology optimizations of the built-in policy, preventing code duplication across different schedulers.
To implement this introduce a new helper kfunc scx_bpf_select_cpu_and() that accepts a cpumask of allowed CPUs:
s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu,
u64 wake_flags, const struct cpumask *cpus_allowed, u64 flags);
Example usage
=============
s32 BPF_STRUCT_OPS(foo_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
{
const struct cpumask *cpus = task_allowed_cpus(p) ?: p->cpus_ptr;
s32 cpu;
cpu = scx_bpf_select_cpu_and(p, prev_cpu, wake_flags, cpus, 0);
if (cpu >= 0) {
scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
return cpu;
}
return prev_cpu;
}Results
=======
Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of scx_bpfland that uses the new helper scx_bpf_select_cpu_and() and 0xff00 as allowed domain:
$ vng --cpu 16,sockets=4,cores=4,threads=1
...
$ stress-ng -c 16
...
$ htop
...
0[ 0.0%] 8[||||||||||||||||||||||||100.0%]
1[ 0.0%] 9[||||||||||||||||||||||||100.0%]
2[ 0.0%] 10[||||||||||||||||||||||||100.0%]
3[ 0.0%] 11[||||||||||||||||||||||||100.0%]
4[ 0.0%] 12[||||||||||||||||||||||||100.0%]
5[ 0.0%] 13[||||||||||||||||||||||||100.0%]
6[ 0.0%] 14[||||||||||||||||||||||||100.0%]
7[ 0.0%] 15[||||||||||||||||||||||||100.0%]With scx_bpf_select_cpu_dfl() tasks would be distributed evenly across all the available CPUs.
ChangeLog v5 -> v6:
- prevent redundant cpumask_subset() + cpumask_equal() checks in all patches
- remove cpumask_subset() + cpumask_and() combo with local cpumasks, as cpumask_and() alone is generally more efficient
- cleanup patches to prevent unnecessary function renames
ChangeLog v4 -> v5:
- simplify code to compute the temporary task's cpumasks (and)
ChangeLog v3 -> v4:
- keep p->nr_cpus_allowed optimizations (skip cpumask operations when the task can run on all CPUs)
- allow to call scx_bpf_select_cpu_and() also from ops.enqueue() and modify the kselftest to cover this case as well
- rebase to the latest sched_ext/for-6.15
ChangeLog v2 -> v3:
- incrementally refactor scx_select_cpu_dfl() to accept idle flags and an arbitrary allowed cpumask
- build scx_bpf_select_cpu_and() on top of the existing logic
- re-arrange scx_select_cpu_dfl() prototype, aligning the first three arguments with select_task_rq()
- do not use "domain" for the allowed cpumask to avoid potential ambiguity with sched_domain
ChangeLog v1 -> v2:
- rename scx_bpf_select_cpu_pref() to scx_bpf_select_cpu_and() and always select idle CPUs strictly within the allowed domain
- rename preferred CPUs -> allowed CPU
- drop %SCX_PICK_IDLE_IN_PREF (not required anymore)
- deprecate scx_bpf_select_cpu_dfl() in favor of scx_bpf_select_cpu_and() and provide all the required backward compatibility boilerplate
Andrea Righi (6):
sched_ext: idle: Extend topology optimizations to all tasks
sched_ext: idle: Explicitly pass allowed cpumask to scx_select_cpu_dfl()
sched_ext: idle: Accept an arbitrary cpumask in scx_select_cpu_dfl()
sched_ext: idle: Introduce scx_bpf_select_cpu_and() selftests/sched_ext: Add test for scx_bpf_select_cpu_and()
sched_ext: idle: Deprecate scx_bpf_select_cpu_dfl()
Documentation/scheduler/sched-ext.rst | 11 +- kernel/sched/ext.c | 6 +- kernel/sched/ext_idle.c | 196 ++++++++++++++++----- kernel/sched/ext_idle.h | 3 +- tools/sched_ext/include/scx/common.bpf.h | 5 +- tools/sched_ext/include/scx/compat.bpf.h | 37 ++++ tools/sched_ext/scx_flatcg.bpf.c | 12 +- tools/sched_ext/scx_simple.bpf.c | 9 +- tools/testing/selftests/sched_ext/Makefile | 1 + .../testing/selftests/sched_ext/allowed_cpus.bpf.c | 121 +++++++++++++ tools/testing/selftests/sched_ext/allowed_cpus.c | 57 ++++++ .../selftests/sched_ext/enq_select_cpu_fails.bpf.c | 12 +- .../selftests/sched_ext/enq_select_cpu_fails.c | 2 +- tools/testing/selftests/sched_ext/exit.bpf.c | 6 +- .../sched_ext/select_cpu_dfl_nodispatch.bpf.c | 13 +- .../sched_ext/select_cpu_dfl_nodispatch.c | 2 +- 16 files changed, 404 insertions(+), 89 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/allowed_cpus.bpf.c create mode 100644 tools/testing/selftests/sched_ext/allowed_cpus.c
Andrea Righi
reviewerPATCH 1/6 sched_ext: idle: Extend topology optimizations to all tasks
The built-in idle selection policy, scx_select_cpu_dfl(), always prioritizes picking idle CPUs within the same LLC or NUMA node, but these optimizations are currently applied only when a task has no CPU affinity constraints.
This is done primarily for efficiency, as it avoids the overhead of updating a cpumask every time we need to select an idle CPU (which can be costly in large SMP systems).
However, this approach limits the effectiveness of the built-in idle policy and results in inconsistent behavior, as affinity-restricted tasks don't benefit from topology-aware optimizations.
To address this, modify the policy to apply LLC and NUMA-aware optimizations even when a task is constrained to a subset of CPUs.
We can still avoid updating the cpumasks by checking if the subset of LLC and node CPUs are contained in the subset of allowed CPUs usable by the task (which is true in most of the cases - for tasks that don't have affinity constratints).
Moreover, use temporary local per-CPU cpumasks to determine the LLC and node subsets, minimizing potential overhead even on large SMP systems.
--- kernel/sched/ext_idle.c | 72 ++++++++++++++++++++++++++++------------- 1 file changed, 49 insertions(+), 23 deletions(-)
kernel/sched/ext_idle.c +49 -23
@@ -46,6 +46,12 @@ static struct scx_idle_cpus scx_idle_global_masks;*/ static struct scx_idle_cpus **scx_idle_node_masks; +/* +* Local per-CPU cpumasks (used to generate temporary idle cpumasks). +*/ +static DEFINE_PER_CPU(cpumask_var_t, local_llc_idle_cpumask); +static DEFINE_PER_CPU(cpumask_var_t, local_numa_idle_cpumask); +/* * Return the idle masks associated to a target @node. *
@@ -391,6 +397,14 @@ void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops)static_branch_disable_cpuslocked (&scx_selcpu_topo_numa); } +/* +* Return true if @p can run on all possible CPUs, false otherwise. +*/ +static inline bool task_affinity_all (const struct task_struct *p) +{ +return p->nr_cpus_allowed >=num_possible_cpus (); +} +/* * Built-in CPU idle selection policy: *
@@ -426,8 +440,7 @@ void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops)*/ s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags,u64 flags) { -const struct cpumask *llc_cpus =NULL ; -const struct cpumask *numa_cpus =NULL ; +const struct cpumask *llc_cpus =NULL , *numa_cpus =NULL ;int node =scx_cpu_node_if_enabled (prev_cpu);s32 cpu;
@@ -437,22 +450,27 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64rcu_read_lock ();/* -* Determine the scheduling domain only if the task is allowed to run -* on all CPUs. -* -* This is done primarily for efficiency, as it avoids the overhead of -* updating a cpumask every time we need to select an idle CPU (which -* can be costly in large SMP systems), but it also aligns logically: -* if a task's scheduling domain is restricted by user-space (through -* CPU affinity), the task will simply use the flat scheduling domain -* defined by user-space. +* Determine the subset of CPUs that the task can use in its +* current LLC and node. */ -if (p->nr_cpus_allowed >=num_possible_cpus ()) { -if (static_branch_maybe (CONFIG_NUMA , &scx_selcpu_topo_numa)) - numa_cpus =numa_span (prev_cpu); +if (static_branch_maybe (CONFIG_NUMA , &scx_selcpu_topo_numa)) { +struct cpumask *local_cpus =this_cpu_cpumask_var_ptr (local_numa_idle_cpumask); +const struct cpumask *cpus =numa_span (prev_cpu); + +if (task_affinity_all (p)) + numa_cpus = cpus; +else if (cpus &&cpumask_and (local_cpus, p->cpus_ptr, cpus)) + numa_cpus = local_cpus; + } -if (static_branch_maybe (CONFIG_SCHED_MC , &scx_selcpu_topo_llc)) - llc_cpus =llc_span (prev_cpu); +if (static_branch_maybe (CONFIG_SCHED_MC , &scx_selcpu_topo_llc)) { +struct cpumask *local_cpus =this_cpu_cpumask_var_ptr (local_llc_idle_cpumask); +const struct cpumask *cpus =llc_span (prev_cpu); + +if (task_affinity_all (p)) + llc_cpus = cpus; +else if (cpus &&cpumask_and (local_cpus, p->cpus_ptr, cpus)) + llc_cpus = local_cpus; }/*
@@ -598,7 +616,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64*/ void scx_idle_init_masks (void ) { -int node; +int i;/* Allocate global idle cpumasks */ BUG_ON (!alloc_cpumask_var (&scx_idle_global_masks.cpu,GFP_KERNEL ));
@@ -609,13 +627,21 @@ void scx_idle_init_masks(void) sizeof(*scx_idle_node_masks), GFP_KERNEL);BUG_ON (!scx_idle_node_masks); -for_each_node (node) { - scx_idle_node_masks[node] =kzalloc_node (sizeof (**scx_idle_node_masks), -GFP_KERNEL , node); -BUG_ON (!scx_idle_node_masks[node]); +for_each_node (i) { + scx_idle_node_masks[i] =kzalloc_node (sizeof (**scx_idle_node_masks), +GFP_KERNEL , i); +BUG_ON (!scx_idle_node_masks[i]); + +BUG_ON (!alloc_cpumask_var_node (&scx_idle_node_masks[i]->cpu,GFP_KERNEL , i)); +BUG_ON (!alloc_cpumask_var_node (&scx_idle_node_masks[i]->smt,GFP_KERNEL , i)); + } -BUG_ON (!alloc_cpumask_var_node (&scx_idle_node_masks[node]->cpu,GFP_KERNEL , node)); -BUG_ON (!alloc_cpumask_var_node (&scx_idle_node_masks[node]->smt,GFP_KERNEL , node)); +/* Allocate local per-cpu idle cpumasks */ +for_each_possible_cpu (i) { +BUG_ON (!alloc_cpumask_var_node (&per_cpu (local_llc_idle_cpumask, i), +GFP_KERNEL ,cpu_to_node (i))); +BUG_ON (!alloc_cpumask_var_node (&per_cpu (local_numa_idle_cpumask, i), +GFP_KERNEL ,cpu_to_node (i))); } }
Signature · 1 line
2.48.1
Andrea Righi
reviewerPATCH 2/6 sched_ext: idle: Explicitly pass allowed cpumask to scx_select_cpu_dfl()
In reply to Andrea RighiModify scx_select_cpu_dfl() to take the allowed cpumask as an explicit argument, instead of implicitly using @p->cpus_ptr.
This prepares for future changes where arbitrary cpumasks may be passed to the built-in idle CPU selection policy.
This is a pure refactoring with no functional changes.
--- kernel/sched/ext.c | 2 +- kernel/sched/ext_idle.c | 19 ++++++++++--------- kernel/sched/ext_idle.h | 3 ++- 3 files changed, 13 insertions(+), 11 deletions(-)
kernel/sched/ext.c +1 -1
@@ -3395,7 +3395,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag }else {s32 cpu; - cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags,0 ); + cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags, p->cpus_ptr,0 );if (cpu >=0 ) { p->scx.slice =SCX_SLICE_DFL ; p->scx.ddsp_dsq_id =SCX_DSQ_LOCAL ;
kernel/sched/ext_idle.c +10 -9
@@ -438,7 +438,8 @@ static inline bool task_affinity_all(const struct task_struct *p)* NOTE: tasks that can only run on 1 CPU are excluded by this logic, because * we never call ops.select_cpu() for them, see select_task_rq(). */ -s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags,u64 flags) +s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags, +const struct cpumask *cpus_allowed,u64 flags) {const struct cpumask *llc_cpus =NULL , *numa_cpus =NULL ;int node =scx_cpu_node_if_enabled (prev_cpu);
@@ -457,9 +458,9 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64struct cpumask *local_cpus =this_cpu_cpumask_var_ptr (local_numa_idle_cpumask);const struct cpumask *cpus =numa_span (prev_cpu); -if (task_affinity_all (p)) +if (cpus_allowed == p->cpus_ptr &&task_affinity_all (p)) numa_cpus = cpus; -else if (cpus &&cpumask_and (local_cpus, p->cpus_ptr, cpus)) +else if (cpus &&cpumask_and (local_cpus, cpus_allowed, cpus)) numa_cpus = local_cpus; }
@@ -467,9 +468,9 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64struct cpumask *local_cpus =this_cpu_cpumask_var_ptr (local_llc_idle_cpumask);const struct cpumask *cpus =llc_span (prev_cpu); -if (task_affinity_all (p)) +if (cpus_allowed == p->cpus_ptr &&task_affinity_all (p)) llc_cpus = cpus; -else if (cpus &&cpumask_and (local_cpus, p->cpus_ptr, cpus)) +else if (cpus &&cpumask_and (local_cpus, cpus_allowed, cpus)) llc_cpus = local_cpus; }
@@ -508,7 +509,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64 cpu_rq(cpu)->scx.local_dsq.nr == 0 && (!(flags & SCX_PICK_IDLE_IN_NODE) || (waker_node == node)) && !cpumask_empty(idle_cpumask(waker_node)->cpu)) { -if (cpumask_test_cpu (cpu, p->cpus_ptr)) +if (cpumask_test_cpu (cpu, cpus_allowed)) goto out_unlock; } }
@@ -553,7 +554,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64* begin in prev_cpu's node and proceed to other nodes in * order of increasing distance. */ - cpu =scx_pick_idle_cpu (p->cpus_ptr, node, flags |SCX_PICK_IDLE_CORE ); + cpu =scx_pick_idle_cpu (cpus_allowed, node, flags |SCX_PICK_IDLE_CORE );if (cpu >=0 ) goto out_unlock;
@@ -601,7 +602,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64* in prev_cpu's node and proceed to other nodes in order of * increasing distance. */ - cpu =scx_pick_idle_cpu (p->cpus_ptr, node, flags); + cpu =scx_pick_idle_cpu (cpus_allowed, node, flags);if (cpu >=0 ) goto out_unlock;
@@ -857,7 +858,7 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, goto prev_cpu;#ifdef CONFIG_SMP - cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags,0 ); + cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags, p->cpus_ptr,0 );if (cpu >=0 ) { *is_idle = true;return cpu;
kernel/sched/ext_idle.h +2 -1
@@ -27,7 +27,8 @@ static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node }#endif /* CONFIG_SMP */ -s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags,u64 flags); +s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags, +const struct cpumask *cpus_allowed,u64 flags);void scx_idle_enable (struct sched_ext_ops *ops);void scx_idle_disable (void );int scx_idle_init (void );
Signature · 1 line
2.48.1
Tejun Heo
reviewerRe: [PATCH 2/6] sched_ext: idle: Explicitly pass allowed cpumask to scx_select_cpu_dfl()
Hello,
On Fri, Mar 21, 2025 at 11:10:48PM +0100, Andrea Righi wrote:
...
kernel/sched/ext_idle.c s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64 · line 462
Quoted 12 lines
kernel/sched/ext_idle.c +3
-1+s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags, +const struct cpumask *cpus_allowed,u64 flags) {const struct cpumask *llc_cpus =NULL , *numa_cpus =NULL ;int node =scx_cpu_node_if_enabled (prev_cpu);@@ -457,9 +458,9 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64struct cpumask *local_cpus =this_cpu_cpumask_var_ptr (local_numa_idle_cpumask);const struct cpumask *cpus =numa_span (prev_cpu); -if (task_affinity_all (p)) +if (cpus_allowed == p->cpus_ptr &&task_affinity_all (p)) numa_cpus = cpus;
Note that this test isn't quite correct. While the error isn't introduced by this patchset, this becomes a lot more prominent with the series. p->nr_cpus_allowed tracks the number of CPUs in p->cpus_mask. p->cpus_ptr can point away from p->cpus_mask without updating p->nr_cpus_allowed, so the condition that should be checked is p->cpus_ptr == &p->cpus_mask && p->nr_cpus_allowed == num_possible_cpus().
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerRe: [PATCH 2/6] sched_ext: idle: Explicitly pass allowed cpumask to scx_select_cpu_dfl()
Quoted 13 lines · Tejun Heo
Hello,
On Fri, Mar 21, 2025 at 11:10:48PM +0100, Andrea Righi wrote:
...Quoted 9 lines
kernel/sched/ext_idle.c +2
-0+s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags, +const struct cpumask *cpus_allowed,u64 flags) {const struct cpumask *llc_cpus =NULL , *numa_cpus =NULL ;int node =scx_cpu_node_if_enabled (prev_cpu);@@ -457,9 +458,9 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64struct cpumask *local_cpus =this_cpu_cpumask_var_ptr (local_numa_idle_cpumask);const struct cpumask *cpus =numa_span (prev_cpu);
Quoted 10 lines
kernel/sched/ext_idle.c s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64 · line 462
Quoted 3 lines
kernel/sched/ext_idle.c +1
-1-if (task_affinity_all (p)) +if (cpus_allowed == p->cpus_ptr &&task_affinity_all (p)) numa_cpus = cpus;Note that this test isn't quite correct. While the error isn't introduced by this patchset, this becomes a lot more prominent with the series.
p->nr_cpus_allowedtracks the number of CPUs inp->cpus_mask.p->cpus_ptrcan point away fromp->cpus_maskwithout updatingp->nr_cpus_allowed, so the condition that should be checked isp->cpus_ptr==&p->cpus_mask&&p->nr_cpus_allowed==num_possible_cpus().
Thanks for pointing this out. Considering that, it's more clear (and less bug prone) to just use NULL when the caller doesn't want to specify an additional cpumask. Will change it in the next version.
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 3/6 sched_ext: idle: Accept an arbitrary cpumask in scx_select_cpu_dfl()
In reply to Andrea RighiMany scx schedulers implement their own hard or soft-affinity rules to support topology characteristics, such as heterogeneous architectures (e.g., big.LITTLE, P-cores/E-cores), or to categorize tasks based on specific properties (e.g., running certain tasks only in a subset of CPUs).
Currently, there is no mechanism that allows to use the built-in idle CPU selection policy to an arbitrary subset of CPUs. As a result, schedulers often implement their own idle CPU selection policies, which are typically similar to one another, leading to a lot of code duplication.
To address this, modify scx_select_cpu_dfl() to accept an arbitrary cpumask, that can be used by the BPF schedulers to apply the existent built-in idle CPU selection policy to a subset of allowed CPUs.
With this concept the idle CPU selection policy becomes the following:
- always prioritize CPUs from fully idle SMT cores (if SMT is enabled),
- select the same CPU if it's idle and in the allowed CPUs,
- select an idle CPU within the same LLC, if the LLC cpumask is a subset of the allowed CPUs,
- select an idle CPU within the same node, if the node cpumask is a subset of the allowed CPUs,
- select an idle CPU within the allowed CPUs.
This functionality will be exposed through a dedicated kfunc in a separate patch.
--- kernel/sched/ext_idle.c | 68 +++++++++++++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 12 deletions(-)
kernel/sched/ext_idle.c +56 -12
@@ -49,6 +49,7 @@ static struct scx_idle_cpus **scx_idle_node_masks;/* * Local per-CPU cpumasks (used to generate temporary idle cpumasks). */ +static DEFINE_PER_CPU(cpumask_var_t, local_idle_cpumask); static DEFINE_PER_CPU(cpumask_var_t, local_llc_idle_cpumask); static DEFINE_PER_CPU(cpumask_var_t, local_numa_idle_cpumask);
@@ -417,13 +418,15 @@ static inline bool task_affinity_all(const struct task_struct *p)* branch prediction optimizations. * * 3. Pick a CPU within the same LLC (Last-Level Cache): -* - if the above conditions aren't met, pick a CPU that shares the same LLC -* to maintain cache locality. +* - if the above conditions aren't met, pick a CPU that shares the same +* LLC, if the LLC domain is a subset of @cpus_allowed, to maintain +* cache locality. * * 4. Pick a CPU within the same NUMA node, if enabled: -* - choose a CPU from the same NUMA node to reduce memory access latency. +* - choose a CPU from the same NUMA node, if the node cpumask is a +* subset of @cpus_allowed, to reduce memory access latency. * -* 5. Pick any idle CPU usable by the task. +* 5. Pick any idle CPU within the @cpus_allowed domain. * * Step 3 and 4 are performed only if the system has, respectively, * multiple LLCs / multiple NUMA nodes (see scx_selcpu_topo_llc and
@@ -442,9 +445,43 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, const struct cpumask *cpus_allowed, u64 flags) {const struct cpumask *llc_cpus =NULL , *numa_cpus =NULL ; -int node =scx_cpu_node_if_enabled (prev_cpu); +const struct cpumask *allowed = p->cpus_ptr; +int node;s32 cpu; +preempt_disable (); + +/* +* Determine the subset of CPUs usable by @p within @cpus_allowed. +*/ +if (cpus_allowed != p->cpus_ptr) { +struct cpumask *local_cpus =this_cpu_cpumask_var_ptr (local_idle_cpumask); + +if (task_affinity_all (p)) { + allowed = cpus_allowed; + }else if (cpumask_and (local_cpus, cpus_allowed, p->cpus_ptr)) { + allowed = local_cpus; + }else { + cpu = -EBUSY ; + goto out_enable; + } + } + +/* +* If @prev_cpu is not in the allowed domain, try to assign a new +* arbitrary CPU usable by the task in the allowed domain. +*/ +if (!cpumask_test_cpu (prev_cpu, allowed)) { + cpu =cpumask_any_and_distribute (p->cpus_ptr, allowed); +if (cpu < nr_cpu_ids) { + prev_cpu = cpu; + }else { + cpu = -EBUSY ; + goto out_enable; + } + } + node =scx_cpu_node_if_enabled (prev_cpu); +/* * This is necessary to protect llc_cpus. */
@@ -453,14 +490,17 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,/* * Determine the subset of CPUs that the task can use in its * current LLC and node. +* +* If the task can run on all CPUs, use the node and LLC cpumasks +* directly. */ if (static_branch_maybe (CONFIG_NUMA , &scx_selcpu_topo_numa)) {struct cpumask *local_cpus =this_cpu_cpumask_var_ptr (local_numa_idle_cpumask);const struct cpumask *cpus =numa_span (prev_cpu); -if (cpus_allowed == p->cpus_ptr &&task_affinity_all (p)) +if (allowed == p->cpus_ptr &&task_affinity_all (p)) numa_cpus = cpus; -else if (cpus &&cpumask_and (local_cpus, cpus_allowed, cpus)) +else if (cpus &&cpumask_and (local_cpus, allowed, cpus)) numa_cpus = local_cpus; }
@@ -468,9 +508,9 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,struct cpumask *local_cpus =this_cpu_cpumask_var_ptr (local_llc_idle_cpumask);const struct cpumask *cpus =llc_span (prev_cpu); -if (cpus_allowed == p->cpus_ptr &&task_affinity_all (p)) +if (allowed == p->cpus_ptr &&task_affinity_all (p)) llc_cpus = cpus; -else if (cpus &&cpumask_and (local_cpus, cpus_allowed, cpus)) +else if (cpus &&cpumask_and (local_cpus, allowed, cpus)) llc_cpus = local_cpus; }
@@ -509,7 +549,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, cpu_rq(cpu)->scx.local_dsq.nr == 0 && (!(flags & SCX_PICK_IDLE_IN_NODE) || (waker_node == node)) && !cpumask_empty(idle_cpumask(waker_node)->cpu)) { -if (cpumask_test_cpu (cpu, cpus_allowed)) +if (cpumask_test_cpu (cpu, allowed)) goto out_unlock; } }
@@ -554,7 +594,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,* begin in prev_cpu's node and proceed to other nodes in * order of increasing distance. */ - cpu =scx_pick_idle_cpu (cpus_allowed, node, flags |SCX_PICK_IDLE_CORE ); + cpu =scx_pick_idle_cpu (allowed, node, flags |SCX_PICK_IDLE_CORE );if (cpu >=0 ) goto out_unlock;
@@ -602,12 +642,14 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,* in prev_cpu's node and proceed to other nodes in order of * increasing distance. */ - cpu =scx_pick_idle_cpu (cpus_allowed, node, flags); + cpu =scx_pick_idle_cpu (allowed, node, flags);if (cpu >=0 ) goto out_unlock; out_unlock:rcu_read_unlock (); +out_enable: +preempt_enable ();return cpu; }
@@ -639,6 +681,8 @@ void scx_idle_init_masks(void)/* Allocate local per-cpu idle cpumasks */ for_each_possible_cpu(i) { + BUG_ON(!alloc_cpumask_var_node(&per_cpu(local_idle_cpumask, i), + GFP_KERNEL, cpu_to_node(i))); BUG_ON(!alloc_cpumask_var_node(&per_cpu(local_llc_idle_cpumask, i), GFP_KERNEL, cpu_to_node(i))); BUG_ON(!alloc_cpumask_var_node(&per_cpu(local_numa_idle_cpumask, i),
Signature · 1 line
2.48.1
Tejun Heo
reviewerRe: [PATCH 3/6] sched_ext: idle: Accept an arbitrary cpumask in scx_select_cpu_dfl()
On Fri, Mar 21, 2025 at 11:10:49PM +0100, Andrea Righi wrote:
...
kernel/sched/ext_idle.c s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, · line 482
Quoted 13 lines
kernel/sched/ext_idle.c +13
-0+/* +* If @prev_cpu is not in the allowed domain, try to assign a new +* arbitrary CPU usable by the task in the allowed domain. +*/ +if (!cpumask_test_cpu (prev_cpu, allowed)) { + cpu =cpumask_any_and_distribute (p->cpus_ptr, allowed); +if (cpu < nr_cpu_ids) { + prev_cpu = cpu; + }else { + cpu = -EBUSY ; + goto out_enable; + } + }
Would it be better to clear it to -1 and disable @prev_cpu optimizations if negative? Not a big deal, so please feel free to push back but things like wake_sync optimization become a bit weird with @prev_cpu set to some random CPU and down the line if we want to allow e.g. preferring previous idle CPU even when the sibling CPU isn't idle which seems to help with some workloads, this can become tricky.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerRe: [PATCH 3/6] sched_ext: idle: Accept an arbitrary cpumask in scx_select_cpu_dfl()
Quoted 14 lines · Tejun Heo
On Fri, Mar 21, 2025 at 11:10:49PM +0100, Andrea Righi wrote:
...Quoted 12 lines
kernel/sched/ext_idle.c +12
-0+/* +* If @prev_cpu is not in the allowed domain, try to assign a new +* arbitrary CPU usable by the task in the allowed domain. +*/ +if (!cpumask_test_cpu (prev_cpu, allowed)) { + cpu =cpumask_any_and_distribute (p->cpus_ptr, allowed); +if (cpu < nr_cpu_ids) { + prev_cpu = cpu; + }else { + cpu = -EBUSY ; + goto out_enable; + }
Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ }Would it be better to clear it to -1 and disable @prev_cpu optimizations if negative? Not a big deal, so please feel free to push back but things like
wake_syncoptimization become a bit weird with @prev_cpu set to some random CPU and down the line if we want to allow e.g. preferring previous idle CPU even when the sibling CPU isn't idle which seems to help with some workloads, this can become tricky.
Maybe a better strategy would be to try with prev_cpu = smp_processor_id(), if it's in the subset p->cpus_ptr & allowed, which might be beneficial for some waker->wakee scenarios, otherwise jump directly to the end, with cpu = scx_pick_idle_cpu(allowed, node, flags).
Thanks,
-Andrea
Andrea Righi
reviewerPATCH 4/6 sched_ext: idle: Introduce scx_bpf_select_cpu_and()
In reply to Andrea RighiProvide a new kfunc, scx_bpf_select_cpu_and(), that can be used to apply the built-in idle CPU selection policy to a subset of allowed CPU.
This new helper is basically an extension of scx_bpf_select_cpu_dfl(). However, when an idle CPU can't be found, it returns a negative value instead of @prev_cpu, aligning its behavior more closely with scx_bpf_pick_idle_cpu().
It also accepts %SCX_PICK_IDLE_* flags, which can be used to enforce strict selection to @prev_cpu's node (%SCX_PICK_IDLE_IN_NODE), or to request only a full-idle SMT core (%SCX_PICK_IDLE_CORE), while applying the built-in selection logic.
With this helper, BPF schedulers can apply the built-in idle CPU selection policy restricted to any arbitrary subset of CPUs.
Example usage
=============
Possible usage in ops.select_cpu():
s32 BPF_STRUCT_OPS(foo_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
{
const struct cpumask *cpus = task_allowed_cpus(p) ?: p->cpus_ptr;
s32 cpu;
cpu = scx_bpf_select_cpu_and(p, prev_cpu, wake_flags, cpus, 0);
if (cpu >= 0) {
scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
return cpu;
}
return prev_cpu;
}Results
=======
Load distribution on a 4 sockets, 4 cores per socket system, simulated using virtme-ng, running a modified version of scx_bpfland that uses scx_bpf_select_cpu_and() with 0xff00 as the allowed subset of CPUs:
$ vng --cpu 16,sockets=4,cores=4,threads=1 ... $ stress-ng -c 16 ... $ htop ... 0[ 0.0%] 8[||||||||||||||||||||||||100.0%] 1[ 0.0%] 9[||||||||||||||||||||||||100.0%] 2[ 0.0%] 10[||||||||||||||||||||||||100.0%] 3[ 0.0%] 11[||||||||||||||||||||||||100.0%] 4[ 0.0%] 12[||||||||||||||||||||||||100.0%] 5[ 0.0%] 13[||||||||||||||||||||||||100.0%] 6[ 0.0%] 14[||||||||||||||||||||||||100.0%] 7[ 0.0%] 15[||||||||||||||||||||||||100.0%]
With scx_bpf_select_cpu_dfl() tasks would be distributed evenly across all the available CPUs.
--- kernel/sched/ext.c | 1 + kernel/sched/ext_idle.c | 43 ++++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 2 ++ 3 files changed, 46 insertions(+)
kernel/sched/ext.c +1 -0
@@ -465,6 +465,7 @@ struct sched_ext_ops {* idle CPU tracking and the following helpers become unavailable: * * - scx_bpf_select_cpu_dfl() +* - scx_bpf_select_cpu_and() * - scx_bpf_test_and_clear_cpu_idle() * - scx_bpf_pick_idle_cpu() *
kernel/sched/ext_idle.c +43 -0
55 lines not on this page · raw message
tools/sched_ext/include/scx/common.bpf.h +2 -0
@@ -48,6 +48,8 @@ static inline void ___vmlinux_h_sanity_check___(void) s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym; s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym; +s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64 wake_flags, + const struct cpumask *cpus_allowed, u64 flags) __ksym __weak; void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym __weak; void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym __weak; u32 scx_bpf_dispatch_nr_slots(void) __ksym;
Signature · 1 line
2.48.1
Tejun Heo
reviewerRe: [PATCH 4/6] sched_ext: idle: Introduce scx_bpf_select_cpu_and()
Hello,
On Fri, Mar 21, 2025 at 11:10:50PM +0100, Andrea Righi wrote:
...
Quoted 12 lines
kernel/sched/ext_idle.c +12
-0+__bpf_kfunc s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64 wake_flags, +const struct cpumask *cpus_allowed,u64 flags) +{ +s32 cpu; + +if (!ops_cpu_valid (prev_cpu,NULL )) +return -EINVAL ; + +if (!check_builtin_idle_enabled ()) +return -EBUSY ; + +if (!scx_kf_allowed (SCX_KF_SELECT_CPU |SCX_KF_ENQUEUE ))
kernel/sched/ext_idle.c __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, · line 957
Quoted 10 lines
kernel/sched/ext_idle.c +10
-0+return -EPERM ; + +#ifdef CONFIG_SMP + cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags, cpus_allowed, flags); +#else + cpu = -EBUSY ; +#endif + +return cpu; +}
Later in the series, I find scx_bpf_select_cpu_and() being called with p->cpus_ptr really confusing. scx_bpf_select_cpu_and() is always constrained by p->cpus_ptr (except for the currently buggy case where p->nr_cpus_allowed is used while p->cpus_ptr is overridden), so what does it mean to call scx_bpf_select_cpu_and() with p->cpus_ptr as @cpus_allowed? I'd much prefer if the convention in such cases is calling with NULL @cpus_allowed. @cpus_allowed is the extra mask to and to p->cpus_ptr when searching for an idle CPU. If we're going to use p->cpus_ptr, we just don't have the extra cpumask to and.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerRe: [PATCH 4/6] sched_ext: idle: Introduce scx_bpf_select_cpu_and()
Quoted 28 lines · Tejun Heo
Hello,
On Fri, Mar 21, 2025 at 11:10:50PM +0100, Andrea Righi wrote:
...Quoted 12 lines
kernel/sched/ext_idle.c +12
-0+__bpf_kfunc s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64 wake_flags, +const struct cpumask *cpus_allowed,u64 flags) +{ +s32 cpu; + +if (!ops_cpu_valid (prev_cpu,NULL )) +return -EINVAL ; + +if (!check_builtin_idle_enabled ()) +return -EBUSY ; + +if (!scx_kf_allowed (SCX_KF_SELECT_CPU |SCX_KF_ENQUEUE ))kernel/sched/ext_idle.c __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, · line 957
Quoted 10 lines
kernel/sched/ext_idle.c +10
-0+return -EPERM ; + +#ifdef CONFIG_SMP + cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags, cpus_allowed, flags); +#else + cpu = -EBUSY ; +#endif + +return cpu; +}Later in the series, I find
scx_bpf_select_cpu_and()being called with
Quoted 8 lines
p->cpus_ptrreally confusing.scx_bpf_select_cpu_and()is always constrained byp->cpus_ptr(except for the currently buggy case wherep->nr_cpus_allowedis used whilep->cpus_ptris overridden), so what does it mean to callscx_bpf_select_cpu_and()withp->cpus_ptras @cpus_allowed? I'd much prefer if the convention in such cases is calling with NULL @cpus_allowed. @cpus_allowed is the extra mask to and top->cpus_ptrwhen searching for an idle CPU. If we're going to usep->cpus_ptr, we just don't have the extra cpumask to and.
Exactly, as mentioned in a previous email I also agree that using NULL as @cpus_allowed would be much more clear and less bug prone. Will change that.
-Andrea
Andrea Righi
reviewerPATCH 5/6 selftests/sched_ext: Add test for scx_bpf_select_cpu_and()
In reply to Andrea RighiAdd a selftest to validate the behavior of the built-in idle CPU selection policy applied to a subset of allowed CPUs, using scx_bpf_select_cpu_and().
--- tools/testing/selftests/sched_ext/Makefile | 1 + .../selftests/sched_ext/allowed_cpus.bpf.c | 121 ++++++++++++++++++ .../selftests/sched_ext/allowed_cpus.c | 57 +++++++++ 3 files changed, 179 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/allowed_cpus.bpf.c create mode 100644 tools/testing/selftests/sched_ext/allowed_cpus.c
tools/testing/selftests/sched_ext/Makefile +1 -0
@@ -173,6 +173,7 @@ auto-test-targets := \ maybe_null \ minimal \ numa \ + allowed_cpus \ prog_run \ reload_loop \ select_cpu_dfl \
tools/testing/selftests/sched_ext/allowed_cpus.bpf.c new file +121 -0
121 lines not on this page · raw message
tools/testing/selftests/sched_ext/allowed_cpus.c new file +57 -0
57 lines not on this page · raw message
Signature · 1 line
2.48.1
Andrea Righi
reviewerPATCH 6/6 sched_ext: idle: Deprecate scx_bpf_select_cpu_dfl()
In reply to Andrea RighiWith the introduction of scx_bpf_select_cpu_and(), we can deprecate scx_bpf_select_cpu_dfl(), as it offers only a subset of features and it's also more consistent with other idle-related APIs (returning a negative value when no idle CPU is found).
Therefore, mark scx_bpf_select_cpu_dfl() as deprecated (printing a warning when it's used), update all the scheduler examples and kselftests to adopt the new API, and ensure backward (source and binary) compatibility by providing the necessary macros and hooks.
Support for scx_bpf_select_cpu_dfl() can be maintained until v6.17.
--- Documentation/scheduler/sched-ext.rst | 11 +++--- kernel/sched/ext.c | 3 +- kernel/sched/ext_idle.c | 18 ++------- tools/sched_ext/include/scx/common.bpf.h | 3 +- tools/sched_ext/include/scx/compat.bpf.h | 37 +++++++++++++++++++ tools/sched_ext/scx_flatcg.bpf.c | 12 +++--- tools/sched_ext/scx_simple.bpf.c | 9 +++-- .../sched_ext/enq_select_cpu_fails.bpf.c | 12 +----- .../sched_ext/enq_select_cpu_fails.c | 2 +- tools/testing/selftests/sched_ext/exit.bpf.c | 6 ++- .../sched_ext/select_cpu_dfl_nodispatch.bpf.c | 13 +++---- .../sched_ext/select_cpu_dfl_nodispatch.c | 2 +- 12 files changed, 73 insertions(+), 55 deletions(-)
Documentation/scheduler/sched-ext.rst +5 -6
@@ -142,15 +142,14 @@ optional. The following modified excerpt is from s32 prev_cpu, u64 wake_flags) { s32 cpu; - /* Need to initialize or the BPF verifier will reject the program */ - bool direct = false; - cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &direct); - - if (direct) + cpu = scx_bpf_select_cpu_and(p, prev_cpu, wake_flags, p->cpus_ptr, 0); + if (cpu >= 0) scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0); + return cpu; + } - return cpu; + return prev_cpu; } /*
kernel/sched/ext.c +1 -2
@@ -464,13 +464,12 @@ struct sched_ext_ops {* state. By default, implementing this operation disables the built-in * idle CPU tracking and the following helpers become unavailable: * -* - scx_bpf_select_cpu_dfl() * - scx_bpf_select_cpu_and() * - scx_bpf_test_and_clear_cpu_idle() * - scx_bpf_pick_idle_cpu() * * The user also must implement ops.select_cpu() as the default -* implementation relies on scx_bpf_select_cpu_dfl(). +* implementation relies on scx_bpf_select_cpu_and(). * * Specify the %SCX_OPS_KEEP_BUILTIN_IDLE flag to keep the built-in idle * tracking.
kernel/sched/ext_idle.c +4 -14
@@ -872,26 +872,16 @@ __bpf_kfunc int scx_bpf_cpu_node(s32 cpu)#endif } -/** -* scx_bpf_select_cpu_dfl - The default implementation of ops.select_cpu() -* @p: task_struct to select a CPU for -* @prev_cpu: CPU @p was on previously -* @wake_flags: %SCX_WAKE_* flags -* @is_idle: out parameter indicating whether the returned CPU is idle -* -* Can only be called from ops.select_cpu() if the built-in CPU selection is -* enabled - ops.update_idle() is missing or %SCX_OPS_KEEP_BUILTIN_IDLE is set. -* @p, @prev_cpu and @wake_flags match ops.select_cpu(). -* -* Returns the picked CPU with *@is_idle indicating whether the picked CPU is -* currently idle and thus a good candidate for direct dispatching. -*/ +/* Provided for backward binary compatibility, will be removed in v6.17. */ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu,u64 wake_flags,bool *is_idle) {#ifdef CONFIG_SMP s32 cpu;#endif +printk_deferred_once (KERN_WARNING +"sched_ext: scx_bpf_select_cpu_dfl() deprecated in favor of scx_bpf_select_cpu_and()" ); +if (!ops_cpu_valid (prev_cpu,NULL )) goto prev_cpu;
tools/sched_ext/include/scx/common.bpf.h +2 -1
@@ -47,7 +47,8 @@ static inline void ___vmlinux_h_sanity_check___(void) } s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym; -s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym; +s32 scx_bpf_select_cpu_dfl(struct task_struct *p, + s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym __weak; s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64 wake_flags, const struct cpumask *cpus_allowed, u64 flags) __ksym __weak; void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym __weak;
tools/sched_ext/include/scx/compat.bpf.h +37 -0
@@ -225,6 +225,43 @@ static inline bool __COMPAT_is_enq_cpu_selected(u64 enq_flags) scx_bpf_pick_any_cpu_node(cpus_allowed, node, flags) : \ scx_bpf_pick_any_cpu(cpus_allowed, flags)) +/** +* scx_bpf_select_cpu_dfl - The default implementation of ops.select_cpu(). +* We will preserve this compatible helper until v6.17. +* +* @p: task_struct to select a CPU for +* @prev_cpu: CPU @p was on previously +* @wake_flags: %SCX_WAKE_* flags +* @is_idle: out parameter indicating whether the returned CPU is idle +* +* Can only be called from ops.select_cpu() if the built-in CPU selection is +* enabled - ops.update_idle() is missing or %SCX_OPS_KEEP_BUILTIN_IDLE is set. +* @p, @prev_cpu and @wake_flags match ops.select_cpu(). +* +* Returns the picked CPU with *@is_idle indicating whether the picked CPU is +* currently idle and thus a good candidate for direct dispatching. +*/ +#define scx_bpf_select_cpu_dfl (p, prev_cpu, wake_flags, is_idle) \ +({ \ + s32 __cpu; \ + \ + if (bpf_ksym_exists(scx_bpf_select_cpu_and)) { \ + __cpu = scx_bpf_select_cpu_and((p), (prev_cpu), (wake_flags), \ + (p)->cpus_ptr, 0); \ + if (__cpu >= 0) { \ + *(is_idle) = true; \ + } else { \ + *(is_idle) = false; \ + __cpu = (prev_cpu); \ + } \ + } else { \ + __cpu = scx_bpf_select_cpu_dfl((p), (prev_cpu), \ + (wake_flags), (is_idle)); \ + } \ + \ + __cpu; \ +}) +/* * Define sched_ext_ops. This may be expanded to define multiple variants for * backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH().
tools/sched_ext/scx_flatcg.bpf.c +6 -6
@@ -317,15 +317,12 @@ static void set_bypassed_at(struct task_struct *p, struct fcg_task_ctx *taskc)s32 BPF_STRUCT_OPS (fcg_select_cpu ,struct task_struct *p,s32 prev_cpu,u64 wake_flags) {struct fcg_task_ctx *taskc; -bool is_idle = false;s32 cpu; - cpu =scx_bpf_select_cpu_dfl (p, prev_cpu, wake_flags, &is_idle); - taskc =bpf_task_storage_get (&task_ctx, p,0 ,0 );if (!taskc) {scx_bpf_error ("task_ctx lookup failed" ); -return cpu; +return prev_cpu; }/*
@@ -333,13 +330,16 @@ s32 BPF_STRUCT_OPS(fcg_select_cpu, struct task_struct *p, s32 prev_cpu, u64 wake* idle. Follow it and charge the cgroup later in fcg_stopping() after * the fact. */ -if (is_idle) { + cpu =scx_bpf_select_cpu_and (p, prev_cpu, wake_flags, p->cpus_ptr,0 ); +if (cpu >=0 ) {set_bypassed_at (p, taskc);stat_inc (FCG_STAT_LOCAL );scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL ,SCX_SLICE_DFL ,0 ); + +return cpu; } -return cpu; +return prev_cpu; }void BPF_STRUCT_OPS (fcg_enqueue ,struct task_struct *p,u64 enq_flags)
tools/sched_ext/scx_simple.bpf.c +5 -4
@@ -54,16 +54,17 @@ static void stat_inc(u32 idx)s32 BPF_STRUCT_OPS (simple_select_cpu ,struct task_struct *p,s32 prev_cpu,u64 wake_flags) { -bool is_idle = false;s32 cpu; - cpu =scx_bpf_select_cpu_dfl (p, prev_cpu, wake_flags, &is_idle); -if (is_idle) { + cpu =scx_bpf_select_cpu_and (p, prev_cpu, wake_flags, p->cpus_ptr,0 ); +if (cpu >=0 ) {stat_inc (0 );/* count local queueing */ scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL ,SCX_SLICE_DFL ,0 ); + +return cpu; } -return cpu; +return prev_cpu; }void BPF_STRUCT_OPS (simple_enqueue ,struct task_struct *p,u64 enq_flags)
tools/testing/selftests/sched_ext/enq_select_cpu_fails.bpf.c +1 -11
@@ -9,10 +9,6 @@ char _license[] SEC("license") = "GPL"; -/* Manually specify the signature until the kfunc is added to the scx repo. */ -s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, - bool *found) __ksym; -s32 BPF_STRUCT_OPS (enq_select_cpu_fails_select_cpu ,struct task_struct *p,s32 prev_cpu,u64 wake_flags) {
@@ -22,14 +18,8 @@ s32 BPF_STRUCT_OPS(enq_select_cpu_fails_select_cpu, struct task_struct *p,void BPF_STRUCT_OPS (enq_select_cpu_fails_enqueue ,struct task_struct *p,u64 enq_flags) { -/* -* Need to initialize the variable or the verifier will fail to load. -* Improving these semantics is actively being worked on. -*/ -bool found = false; -/* Can only call from ops.select_cpu() */ -scx_bpf_select_cpu_dfl (p,0 ,0 , &found); +scx_bpf_select_cpu_and (p,0 ,0 , p->cpus_ptr,0 );scx_bpf_dsq_insert (p,SCX_DSQ_GLOBAL ,SCX_SLICE_DFL , enq_flags); }
tools/testing/selftests/sched_ext/enq_select_cpu_fails.c +1 -1
@@ -52,7 +52,7 @@ static void cleanup(void *ctx)struct scx_test enq_select_cpu_fails = { .name ="enq_select_cpu_fails" , - .description ="Verify we fail to call scx_bpf_select_cpu_dfl() " + .description ="Verify we fail to call scx_bpf_select_cpu_and() " "from ops.enqueue()" , .setup = setup, .run = run,
tools/testing/selftests/sched_ext/exit.bpf.c +4 -2
@@ -20,12 +20,14 @@ UEI_DEFINE(uei);s32 BPF_STRUCT_OPS (exit_select_cpu ,struct task_struct *p,s32 prev_cpu,u64 wake_flags) { -bool found; +s32 cpu;if (exit_point ==EXIT_SELECT_CPU )EXIT_CLEANLY (); -return scx_bpf_select_cpu_dfl (p, prev_cpu, wake_flags, &found); + cpu =scx_bpf_select_cpu_and (p, prev_cpu, wake_flags, p->cpus_ptr,0 ); + +return cpu >=0 ? cpu : prev_cpu; }void BPF_STRUCT_OPS (exit_enqueue ,struct task_struct *p,u64 enq_flags)
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c +6 -7
@@ -27,10 +27,6 @@ struct { __type(value, struct task_ctx); } task_ctx_stor SEC(".maps"); -/* Manually specify the signature until the kfunc is added to the scx repo. */ -s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, - bool *found) __ksym; -s32 BPF_STRUCT_OPS (select_cpu_dfl_nodispatch_select_cpu ,struct task_struct *p,s32 prev_cpu,u64 wake_flags) {
@@ -43,10 +39,13 @@ s32 BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_select_cpu, struct task_struct *p,return -ESRCH ; } - cpu =scx_bpf_select_cpu_dfl (p, prev_cpu, wake_flags, - &tctx->force_local); + cpu =scx_bpf_select_cpu_and (p, prev_cpu, wake_flags, p->cpus_ptr,0 ); +if (cpu >=0 ) { + tctx->force_local = true; +return cpu; + } -return cpu; +return prev_cpu; } void BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_enqueue, struct task_struct *p,
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.c +1 -1
@@ -66,7 +66,7 @@ static void cleanup(void *ctx)struct scx_test select_cpu_dfl_nodispatch = { .name ="select_cpu_dfl_nodispatch" , - .description ="Verify behavior of scx_bpf_select_cpu_dfl() in " + .description ="Verify behavior of scx_bpf_select_cpu_and() in " "ops.select_cpu()" , .setup = setup, .run = run,
Signature · 1 line
2.48.1
Tejun Heo
reviewerRe: [PATCH 6/6] sched_ext: idle: Deprecate scx_bpf_select_cpu_dfl()
Hello,
Quoted 11 lines · Andrea Righi
With the introduction of
scx_bpf_select_cpu_and(), we can deprecatescx_bpf_select_cpu_dfl(), as it offers only a subset of features and it's also more consistent with other idle-related APIs (returning a negative value when no idle CPU is found).Therefore, mark
scx_bpf_select_cpu_dfl()as deprecated (printing a warning when it's used), update all the scheduler examples and kselftests to adopt the new API, and ensure backward (source and binary) compatibility by providing the necessary macros and hooks.Support for
scx_bpf_select_cpu_dfl()can be maintained until v6.17.
Do we need to deprecate it?
...
Quoted 5 lines
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c +0
-1@@ -43,10 +39,13 @@ s32 BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_select_cpu, struct task_struct *p,return -ESRCH ; } - cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags,
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c s32 BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_select_cpu, struct task_struct *p, · line 49
Quoted 10 lines
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c +6
-2- &tctx->force_local); + cpu =scx_bpf_select_cpu_and (p, prev_cpu, wake_flags, p->cpus_ptr,0 ); +if (cpu >=0 ) { + tctx->force_local = true; +return cpu; + } -return cpu; +return prev_cpu; }
scx_bpf_select_cpu_dfl() is simpler for simple cases. I don't see a pressing need to convert everybody to _and().
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerRe: [PATCH 6/6] sched_ext: idle: Deprecate scx_bpf_select_cpu_dfl()
Quoted 30 lines · Tejun Heo
Hello,
Quoted 11 lines · Andrea Righi
With the introduction of
scx_bpf_select_cpu_and(), we can deprecatescx_bpf_select_cpu_dfl(), as it offers only a subset of features and it's also more consistent with other idle-related APIs (returning a negative value when no idle CPU is found).Therefore, mark
scx_bpf_select_cpu_dfl()as deprecated (printing a warning when it's used), update all the scheduler examples and kselftests to adopt the new API, and ensure backward (source and binary) compatibility by providing the necessary macros and hooks.Support for
scx_bpf_select_cpu_dfl()can be maintained until v6.17.Do we need to deprecate it?
...
Quoted 12 lines
tools/testing/selftests/sched_ext/select_cpu_dfl_nodispatch.bpf.c +5
-2@@ -43,10 +39,13 @@ s32 BPF_STRUCT_OPS(select_cpu_dfl_nodispatch_select_cpu, struct task_struct *p,return -ESRCH ; } - cpu =scx_bpf_select_cpu_dfl (p, prev_cpu, wake_flags, - &tctx->force_local); + cpu =scx_bpf_select_cpu_and (p, prev_cpu, wake_flags, p->cpus_ptr,0 ); +if (cpu >=0 ) { + tctx->force_local = true; +return cpu; + }
Quoted 6 lines
Documentation/scheduler/sched-ext.rst optional. The following modified excerpt is from · line 153
Quoted 3 lines
Documentation/scheduler/sched-ext.rst +1
-1- return cpu; + return prev_cpu; }
scx_bpf_select_cpu_dfl()is simpler for simple cases. I don't see a pressing need to convert everybody to_and().
Yeah, I don't have strong opinions on this, I included this patch mostly to show that we can get rid of a kfunc if we want, but we don't really have to and it's probably less work to just keep it. I'll drop this patch in the next version.
Thanks for the review!
-Andrea
Changwoo Min
reviewerHi Andrea,
Looks great to me.
Thanks!
Changwoo Min
Quoted 130 lines · Andrea Righi
Many scx schedulers implement their own hard or soft-affinity rules to support topology characteristics, such as heterogeneous architectures (e.g., big.LITTLE, P-cores/E-cores), or to categorize tasks based on specific properties (e.g., running certain tasks only in a subset of CPUs).
Currently, there is no mechanism that allows to use the built-in idle CPU selection policy to an arbitrary subset of CPUs. As a result, schedulers often implement their own idle CPU selection policies, which are typically similar to one another, leading to a lot of code duplication.
To address this, extend the built-in idle CPU selection policy introducing ]the concept of allowed CPUs.
With this concept, BPF schedulers can apply the built-in idle CPU selection policy to a subset of allowed CPUs, allowing them to implement their own hard/soft-affinity rules while still using the topology optimizations of the built-in policy, preventing code duplication across different schedulers.
To implement this introduce a new helper kfunc
scx_bpf_select_cpu_and()that accepts a cpumask of allowed CPUs:s32 scx_bpf_select_cpu_and(struct
task_struct*p, s32prev_cpu,u64 wake_flags, const struct cpumask *cpus_allowed, u64 flags);Example usage
=============s32 BPF_STRUCT_OPS(foo_select_cpu, struct
task_struct*p,s32 prev_cpu, u64 wake_flags) { const struct cpumask *cpus = task_allowed_cpus(p) ?: p->cpus_ptr; s32 cpu; cpu = scx_bpf_select_cpu_and(p, prev_cpu, wake_flags, cpus, 0); if (cpu >= 0) { scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0); return cpu; } return prev_cpu; }Results
=======Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of
scx_bpflandthat uses the new helperscx_bpf_select_cpu_and()and0xff00as allowed domain:$ vng --cpu 16,sockets=4,cores=4,threads=1 ... $ stress-ng -c 16 ... $ htop ... 0[ 0.0%] 8[||||||||||||||||||||||||100.0%] 1[ 0.0%] 9[||||||||||||||||||||||||100.0%] 2[ 0.0%] 10[||||||||||||||||||||||||100.0%] 3[ 0.0%] 11[||||||||||||||||||||||||100.0%] 4[ 0.0%] 12[||||||||||||||||||||||||100.0%] 5[ 0.0%] 13[||||||||||||||||||||||||100.0%] 6[ 0.0%] 14[||||||||||||||||||||||||100.0%] 7[ 0.0%] 15[||||||||||||||||||||||||100.0%]With
scx_bpf_select_cpu_dfl()tasks would be distributed evenly across all the available CPUs.ChangeLog v5 -> v6:
- prevent redundantcpumask_subset()+cpumask_equal()checks in all patches
- removecpumask_subset()+cpumask_and()combo with local cpumasks, ascpumask_and()alone is generally more efficient
- cleanup patches to prevent unnecessary function renamesChangeLog v4 -> v5:
- simplify code to compute the temporary task's cpumasks (and)ChangeLog v3 -> v4:
- keepp->nr_cpus_allowedoptimizations (skip cpumask operations when the task can run on all CPUs)
- allow to callscx_bpf_select_cpu_and()also fromops.enqueue()and modify the kselftest to cover this case as well
- rebase to the latest sched_ext/for-6.15ChangeLog v2 -> v3:
- incrementally refactorscx_select_cpu_dfl()to accept idle flags and an arbitrary allowed cpumask
- buildscx_bpf_select_cpu_and()on top of the existing logic
- re-arrangescx_select_cpu_dfl()prototype, aligning the first three arguments withselect_task_rq()
- do not use "domain" for the allowed cpumask to avoid potential ambiguity withsched_domainChangeLog v1 -> v2:
- renamescx_bpf_select_cpu_pref()toscx_bpf_select_cpu_and()and always select idle CPUs strictly within the allowed domain
- rename preferred CPUs -> allowed CPU
- drop %SCX_PICK_IDLE_IN_PREF (not required anymore)
- deprecatescx_bpf_select_cpu_dfl()in favor ofscx_bpf_select_cpu_and()and provide all the required backward compatibility boilerplateAndrea Righi (6):
sched_ext: idle: Extend topology optimizations to all tasks
sched_ext: idle: Explicitly pass allowed cpumask toscx_select_cpu_dfl()
sched_ext: idle: Accept an arbitrary cpumask inscx_select_cpu_dfl()
sched_ext: idle: Introducescx_bpf_select_cpu_and()selftests/sched_ext: Add test forscx_bpf_select_cpu_and()
sched_ext: idle: Deprecatescx_bpf_select_cpu_dfl()Documentation/scheduler/sched-ext.rst | 11 +- kernel/sched/ext.c | 6 +- kernel/sched/ext_idle.c | 196 ++++++++++++++++----- kernel/sched/ext_idle.h | 3 +- tools/sched_ext/include/scx/common.bpf.h | 5 +- tools/sched_ext/include/scx/compat.bpf.h | 37 ++++ tools/sched_ext/scx_flatcg.bpf.c | 12 +- tools/sched_ext/scx_simple.bpf.c | 9 +- tools/testing/selftests/sched_ext/Makefile | 1 + .../testing/selftests/sched_ext/allowed_cpus.bpf.c | 121 +++++++++++++ tools/testing/selftests/sched_ext/allowed_cpus.c | 57 ++++++ .../selftests/sched_ext/enq_select_cpu_fails.bpf.c | 12 +- .../selftests/sched_ext/enq_select_cpu_fails.c | 2 +- tools/testing/selftests/sched_ext/exit.bpf.c | 6 +- .../sched_ext/select_cpu_dfl_nodispatch.bpf.c | 13 +- .../sched_ext/select_cpu_dfl_nodispatch.c | 2 +- 16 files changed, 404 insertions(+), 89 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/allowed_cpus.bpf.c create mode 100644 tools/testing/selftests/sched_ext/allowed_cpus.c