sched_ext: Enhance built-in idle selection with preferred CPUs
Subsystems (4)
Andrea Righi
reviewerPATCH 1/1 sched_ext: Enhance built-in idle selection with preferred CPUs
Many scx schedulers define their own concept of scheduling domains to represent topology characteristics, such as heterogeneous architectures (e.g., big.LITTLE, P-cores/E-cores), or to categorize tasks based on specific properties (e.g., setting the soft-affinity of certain tasks to a subset of CPUs).
Currently, there is no mechanism to share these domains with the built-in idle CPU selection policy. As a result, schedulers often implement their own idle CPU selection policies, which are typically similar to one another, leading to a lot of code duplication.
To address this, extend the built-in idle CPU selection policy introducing the concept of preferred CPUs.
With this concept, BPF schedulers can apply the built-in idle CPU selection policy to a subset of preferred CPUs, allowing them to implement their own scheduling domains while still using the topology optimizations optimizations of the built-in policy, preventing code duplication across different schedulers.
To implement this, introduce a new helper kfunc scx_bpf_select_cpu_pref() that allows to specify a cpumask of preferred CPUs:
s32 scx_bpf_select_cpu_pref(struct task_struct *p,
const struct cpumask *preferred_cpus, s32 prev_cpu, u64 wake_flags, u64 flags);
Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF that can be used to enforce selection strictly within the preferred domain.
Example usage
=============
s32 BPF_STRUCT_OPS(foo_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
{
const struct cpumask *dom = task_domain(p) ?: p->cpus_ptr;
s32 cpu;
/*
* Pick an idle CPU in the task's domain. If no CPU is found,
* extend the search outside the domain.
*/
cpu = scx_bpf_select_cpu_pref(p, dom, prev_cpu, wake_flags, 0);
if (cpu >= 0) {
scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
return cpu;
}
return prev_cpu;
}Results
=======
Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of scx_bpfland that uses the new helper scx_bpf_select_cpu_pref() and 0xff00 as preferred domain:
$ vng --cpu 16,sockets=4,cores=4,threads=1
Starting 12 CPU hogs to fill the preferred domain:
$ stress-ng -c 12
...
0[|||||||||||||||||||||||100.0%] 8[||||||||||||||||||||||||100.0%]
1[| 1.3%] 9[||||||||||||||||||||||||100.0%]
2[|||||||||||||||||||||||100.0%] 10[||||||||||||||||||||||||100.0%]
3[|||||||||||||||||||||||100.0%] 11[||||||||||||||||||||||||100.0%]
4[|||||||||||||||||||||||100.0%] 12[||||||||||||||||||||||||100.0%]
5[|| 2.6%] 13[||||||||||||||||||||||||100.0%]
6[| 0.6%] 14[||||||||||||||||||||||||100.0%]
7| 0.0%] 15[||||||||||||||||||||||||100.0%]Passing %SCX_PICK_IDLE_IN_PREF to scx_bpf_select_cpu_pref() to enforce strict selection on the preferred CPUs (with the same workload):
0[ 0.0%] 8[||||||||||||||||||||||||100.0%]
1[ 0.0%] 9[||||||||||||||||||||||||100.0%]
2[ 0.0%] 10[||||||||||||||||||||||||100.0%]
3[ 0.0%] 11[||||||||||||||||||||||||100.0%]
4[ 0.0%] 12[||||||||||||||||||||||||100.0%]
5[ 0.0%] 13[||||||||||||||||||||||||100.0%]
6[ 0.0%] 14[||||||||||||||||||||||||100.0%]
7[ 0.0%] 15[||||||||||||||||||||||||100.0%]Andrea Righi (4):
sched_ext: idle: Honor idle flags in the built-in idle selection policy
sched_ext: idle: Introduce the concept of preferred CPUs
sched_ext: idle: Introduce scx_bpf_select_cpu_pref() selftests/sched_ext: Add test for scx_bpf_select_cpu_pref()
kernel/sched/ext.c | 4 +- kernel/sched/ext_idle.c | 235 ++++++++++++++++++---- kernel/sched/ext_idle.h | 3 +- tools/sched_ext/include/scx/common.bpf.h | 2 + tools/sched_ext/include/scx/compat.h | 1 + tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/pref_cpus.bpf.c | 95 +++++++++ tools/testing/selftests/sched_ext/pref_cpus.c | 58 ++++++ 8 files changed, 354 insertions(+), 45 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.bpf.c create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.c
Andrea Righi
reviewerPATCH 1/4 sched_ext: idle: Honor idle flags in the built-in idle selection policy
Enable passing idle flags (%SCX_PICK_IDLE_*) to scx_select_cpu_dfl(), to enforce strict selection criteria, such as selecting an idle CPU strictly within @prev_cpu's node or choosing only a fully idle SMT core.
This functionality will be exposed through a dedicated kfunc in a separate patch.
--- kernel/sched/ext.c | 2 +- kernel/sched/ext_idle.c | 41 ++++++++++++++++++++++++++++++----------- kernel/sched/ext_idle.h | 2 +- 3 files changed, 32 insertions(+), 13 deletions(-)
kernel/sched/ext.c +1 -1
@@ -3396,7 +3396,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flagbool found;s32 cpu; - cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags, &found); + cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags,0 , &found); p->scx.selected_cpu = cpu;if (found) { p->scx.slice =SCX_SLICE_DFL ;
kernel/sched/ext_idle.c +30 -11
98 lines not on this page · raw message
kernel/sched/ext_idle.h +1 -1
@@ -27,7 +27,7 @@ static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node }#endif /* CONFIG_SMP */ -s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags,bool *found); +s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags,u64 flags,bool *found);void scx_idle_enable (struct sched_ext_ops *ops);void scx_idle_disable (void );int scx_idle_init (void );
Signature · 1 line
2.48.1
Andrea Righi
reviewerPATCH 2/4 sched_ext: idle: Introduce the concept of preferred CPUs
In reply to Andrea RighiMany scx schedulers define their own concept of scheduling domains to represent topology characteristics, such as heterogeneous architectures (e.g., big.LITTLE, P-cores/E-cores), or to categorize tasks based on specific properties (e.g., setting the soft-affinity of certain tasks to a subset of CPUs).
Currently, there is no mechanism to share these domains with the built-in idle CPU selection policy. As a result, schedulers often implement their own idle CPU selection policies, which are typically similar to one another, leading to a lot of code duplication.
To address this, introduce the concept of preferred domain (represented as a cpumask) that can be used by the BPF schedulers to apply the built-in idle CPU selection policy to a subset of preferred CPUs.
With this concept the idle CPU selection policy becomes the following:
- always prioritize CPUs from fully idle SMT cores (if SMT is enabled),
- select the same CPU if it's idle and in the preferred domain,
- select an idle CPU within the same LLC domain, if the LLC domain is a subset of the preferred domain,
- select an idle CPU within the same node, if the node domain is a subset of the preferred domain,
- select an idle CPU within the preferred domain,
- select any idle CPU usable by the task.
Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF, that enforces strict selection within the preferred domain. Without this flag, the preferred domain is treated as a soft constraint: idle CPUs outside the preferred domain can be considered if the preferred domain is fully busy.
If the preferred domain is empty or NULL, the behavior of the built-in idle CPU selection policy remains unchanged.
This only introduces the core concept of preferred domain. This functionality will be exposed through a dedicated kfunc in a separate patch.
--- kernel/sched/ext.c | 3 +- kernel/sched/ext_idle.c | 142 ++++++++++++++++++++------- kernel/sched/ext_idle.h | 3 +- tools/sched_ext/include/scx/compat.h | 1 + 4 files changed, 111 insertions(+), 38 deletions(-)
kernel/sched/ext.c +2 -1
@@ -807,6 +807,7 @@ enum scx_deq_flags {enum scx_pick_idle_cpu_flags {SCX_PICK_IDLE_CORE =1LLU <<0 ,/* pick a CPU whose SMT siblings are also idle */ SCX_PICK_IDLE_IN_NODE =1LLU <<1 ,/* pick a CPU in the same target NUMA node */ +SCX_PICK_IDLE_IN_PREF =1LLU <<2 ,/* pick a CPU in the preferred domain */ };enum scx_kick_flags {
@@ -3396,7 +3397,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flagbool found;s32 cpu; - cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags,0 , &found); + cpu =scx_select_cpu_dfl (p,NULL , prev_cpu, wake_flags,0 , &found); p->scx.selected_cpu = cpu;if (found) { p->scx.slice =SCX_SLICE_DFL ;
kernel/sched/ext_idle.c +106 -36
229 lines not on this page · raw message
kernel/sched/ext_idle.h +2 -1
@@ -27,7 +27,8 @@ static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node }#endif /* CONFIG_SMP */ -s32 scx_select_cpu_dfl (struct task_struct *p,s32 prev_cpu,u64 wake_flags,u64 flags,bool *found); +s32 scx_select_cpu_dfl (struct task_struct *p,const struct cpumask *preferred_cpus, +s32 prev_cpu,u64 wake_flags,u64 flags,bool *found);void scx_idle_enable (struct sched_ext_ops *ops);void scx_idle_disable (void );int scx_idle_init (void );
tools/sched_ext/include/scx/compat.h +1 -0
@@ -120,6 +120,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field#define SCX_PICK_IDLE_CORE SCX_PICK_IDLE_FLAG(SCX_PICK_IDLE_CORE)#define SCX_PICK_IDLE_IN_NODE SCX_PICK_IDLE_FLAG(SCX_PICK_IDLE_IN_NODE) +#define SCX_PICK_IDLE_IN_PREF SCX_PICK_IDLE_FLAG(SCX_PICK_IDLE_IN_PREF)static inline long scx_hotplug_seq (void ) {
Signature · 1 line
2.48.1
Andrea Righi
reviewerPATCH 3/4 sched_ext: idle: Introduce scx_bpf_select_cpu_pref()
In reply to Andrea RighiProvide a new kfunc that can be used to apply the built-in idle CPU selection policy to a subset of preferred CPU:
s32 scx_bpf_select_cpu_pref(struct task_struct *p,
const struct cpumask *preferred_cpus, s32 prev_cpu, u64 wake_flags, u64 flags);
This new helper is basically an extension of scx_bpf_select_cpu_dfl(). However, when an idle CPU can't be found, it returns a negative value instead of @prev_cpu, aligning its behavior more closely with scx_bpf_pick_idle_cpu().
It also accepts %SCX_PICK_IDLE_* flags, which can be used to enforce strict selection to the preferred CPUs (%SCX_PICK_IDLE_IN_PREF) or to @prev_cpu's node (%SCX_PICK_IDLE_IN_NODE), or to request only a full-idle SMT core (%SCX_PICK_IDLE_CORE), while applying the built-in selection logic.
With this helper, BPF schedulers can apply the built-in idle CPU selection policy to a generic CPU domain, with strict or soft selection requirements.
In the future we can also consider to deprecate scx_bpf_select_cpu_dfl() and replace it with scx_bpf_select_cpu_pref(), as the latter provides the same functionality, with the addition of the preferred domain logic.
Example usage
=============
Possible usage in ops.select_cpu():
s32 BPF_STRUCT_OPS(foo_select_cpu, struct task_struct *p,
s32 prev_cpu, u64 wake_flags)
{
const struct cpumask *dom = task_domain(p) ?: p->cpus_ptr;
s32 cpu;
/*
* Pick an idle CPU in the task's domain. If no CPU is found,
* extend the search outside the domain.
*/
cpu = scx_bpf_select_cpu_pref(p, dom, prev_cpu, wake_flags, 0);
if (cpu >= 0) {
scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
return cpu;
}
return prev_cpu;
}Results
=======
Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of scx_bpfland that uses the new helper scx_bpf_select_cpu_pref() and 0xff00 as preferred domain:
$ vng --cpu 16,sockets=4,cores=4,threads=1
Starting 12 CPU hogs to fill the preferred domain:
$ stress-ng -c 12
...
0[|||||||||||||||||||||||100.0%] 8[||||||||||||||||||||||||100.0%]
1[| 1.3%] 9[||||||||||||||||||||||||100.0%]
2[|||||||||||||||||||||||100.0%] 10[||||||||||||||||||||||||100.0%]
3[|||||||||||||||||||||||100.0%] 11[||||||||||||||||||||||||100.0%]
4[|||||||||||||||||||||||100.0%] 12[||||||||||||||||||||||||100.0%]
5[|| 2.6%] 13[||||||||||||||||||||||||100.0%]
6[| 0.6%] 14[||||||||||||||||||||||||100.0%]
7| 0.0%] 15[||||||||||||||||||||||||100.0%]Passing %SCX_PICK_IDLE_IN_PREF to scx_bpf_select_cpu_pref() to enforce strict selection on the preferred CPUs (with the same workload):
0[ 0.0%] 8[||||||||||||||||||||||||100.0%]
1[ 0.0%] 9[||||||||||||||||||||||||100.0%]
2[ 0.0%] 10[||||||||||||||||||||||||100.0%]
3[ 0.0%] 11[||||||||||||||||||||||||100.0%]
4[ 0.0%] 12[||||||||||||||||||||||||100.0%]
5[ 0.0%] 13[||||||||||||||||||||||||100.0%]
6[ 0.0%] 14[||||||||||||||||||||||||100.0%]
7[ 0.0%] 15[||||||||||||||||||||||||100.0%]--- kernel/sched/ext.c | 1 + kernel/sched/ext_idle.c | 60 ++++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 2 + 3 files changed, 63 insertions(+)
kernel/sched/ext.c +1 -0
@@ -465,6 +465,7 @@ struct sched_ext_ops {* idle CPU tracking and the following helpers become unavailable: * * - scx_bpf_select_cpu_dfl() +* - scx_bpf_select_cpu_pref() * - scx_bpf_test_and_clear_cpu_idle() * - scx_bpf_pick_idle_cpu() *
kernel/sched/ext_idle.c +60 -0
72 lines not on this page · raw message
tools/sched_ext/include/scx/common.bpf.h +2 -0
@@ -47,6 +47,8 @@ static inline void ___vmlinux_h_sanity_check___(void) } s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym; +s32 scx_bpf_select_cpu_pref(struct task_struct *p, const struct cpumask *preferred_cpus, + s32 prev_cpu, u64 wake_flags, u64 flags) __ksym __weak; s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym; void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym __weak; void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym __weak;
Signature · 1 line
2.48.1
Changwoo Min
reviewerRe: [PATCH 3/4] sched_ext: idle: Introduce scx_bpf_select_cpu_pref()
Quoted 147 lines · Andrea Righi
Provide a new kfunc that can be used to apply the built-in idle CPU selection policy to a subset of preferred CPU:
s32 scx_bpf_select_cpu_pref(struct
task_struct*p,const struct cpumask *preferred_cpus, s32 prev_cpu, u64 wake_flags, u64 flags);This new helper is basically an extension of
scx_bpf_select_cpu_dfl(). However, when an idle CPU can't be found, it returns a negative value instead of @prev_cpu, aligning its behavior more closely withscx_bpf_pick_idle_cpu().It also accepts %SCX_PICK_IDLE_* flags, which can be used to enforce strict selection to the preferred CPUs (%SCX_PICK_IDLE_IN_PREF) or to @prev_cpu's node (%SCX_PICK_IDLE_IN_NODE), or to request only a full-idle SMT core (%SCX_PICK_IDLE_CORE), while applying the built-in selection logic.
With this helper, BPF schedulers can apply the built-in idle CPU selection policy to a generic CPU domain, with strict or soft selection requirements.
In the future we can also consider to deprecate
scx_bpf_select_cpu_dfl()and replace it withscx_bpf_select_cpu_pref(), as the latter provides the same functionality, with the addition of the preferred domain logic.Example usage
=============Possible usage in
ops.select_cpu():s32 BPF_STRUCT_OPS(foo_select_cpu, struct
task_struct*p,s32 prev_cpu, u64 wake_flags) { const struct cpumask *dom = task_domain(p) ?: p->cpus_ptr; s32 cpu; /* * Pick an idle CPU in the task's domain. If no CPU is found, * extend the search outside the domain. */ cpu = scx_bpf_select_cpu_pref(p, dom, prev_cpu, wake_flags, 0); if (cpu >= 0) { scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0); return cpu; } return prev_cpu; }Results
=======Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of
scx_bpflandthat uses the new helperscx_bpf_select_cpu_pref()and0xff00as preferred domain:$ vng --cpu 16,sockets=4,cores=4,threads=1Starting 12 CPU hogs to fill the preferred domain:
$ stress-ng -c 12 ... 0[|||||||||||||||||||||||100.0%] 8[||||||||||||||||||||||||100.0%] 1[| 1.3%] 9[||||||||||||||||||||||||100.0%] 2[|||||||||||||||||||||||100.0%] 10[||||||||||||||||||||||||100.0%] 3[|||||||||||||||||||||||100.0%] 11[||||||||||||||||||||||||100.0%] 4[|||||||||||||||||||||||100.0%] 12[||||||||||||||||||||||||100.0%] 5[|| 2.6%] 13[||||||||||||||||||||||||100.0%] 6[| 0.6%] 14[||||||||||||||||||||||||100.0%] 7| 0.0%] 15[||||||||||||||||||||||||100.0%]Passing %SCX_PICK_IDLE_IN_PREF to
scx_bpf_select_cpu_pref()to enforce strict selection on the preferred CPUs (with the same workload):0[ 0.0%] 8[||||||||||||||||||||||||100.0%] 1[ 0.0%] 9[||||||||||||||||||||||||100.0%] 2[ 0.0%] 10[||||||||||||||||||||||||100.0%] 3[ 0.0%] 11[||||||||||||||||||||||||100.0%] 4[ 0.0%] 12[||||||||||||||||||||||||100.0%] 5[ 0.0%] 13[||||||||||||||||||||||||100.0%] 6[ 0.0%] 14[||||||||||||||||||||||||100.0%] 7[ 0.0%] 15[||||||||||||||||||||||||100.0%]Signed-off-by Andrea Righi <[email protected]>--- kernel/sched/ext.c | 1 + kernel/sched/ext_idle.c | 60 ++++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 2 + 3 files changed, 63 insertions(+)kernel/sched/ext.c +1
-0@@ -465,6 +465,7 @@ struct sched_ext_ops {* idle CPU tracking and the following helpers become unavailable: * * - scx_bpf_select_cpu_dfl() +* - scx_bpf_select_cpu_pref() * - scx_bpf_test_and_clear_cpu_idle() * - scx_bpf_pick_idle_cpu() * kernel/sched/ext_idle.c +36
-0@@ -907,6 +907,65 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu,return prev_cpu; } +/** +* scx_bpf_select_cpu_pref - Pick an idle CPU usable by task @p, +* prioritizing those in @preferred_cpus +* @p: task_struct to select a CPU for +* @preferred_cpus: cpumask of preferred CPUs +* @prev_cpu: CPU @p was on previously +* @wake_flags: %SCX_WAKE_* flags +* @flags: %SCX_PICK_IDLE* flags +* +* Can only be called from ops.select_cpu() if the built-in CPU selection is +* enabled - ops.update_idle() is missing or %SCX_OPS_KEEP_BUILTIN_IDLE is set. +* @p, @prev_cpu and @wake_flags match ops.select_cpu(). +* +* Returns the selected idle CPU, which will be automatically awakened upon +* returning from ops.select_cpu() and can be used for direct dispatch, or +* a negative value if no idle CPU is available. +*/ +__bpf_kfunc s32 scx_bpf_select_cpu_pref(struct task_struct *p, +const struct cpumask *preferred_cpus, +s32 prev_cpu,u64 wake_flags,u64 flags) +{ +#ifdef CONFIG_SMP +struct cpumask *preferred =NULL ; +bool is_idle = false; +#endif + +if (!ops_cpu_valid (prev_cpu,NULL )) +return -EINVAL ; + +if (!check_builtin_idle_enabled ()) +return -EBUSY ; + +if (!scx_kf_allowed (SCX_KF_SELECT_CPU )) +return -EPERM ; + +#ifdef CONFIG_SMP
kernel/sched/ext_idle.c __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, · line 955
Quoted 10 lines
kernel/sched/ext_idle.c +10
-0+preempt_disable (); + +/* +* As an optimization, do not update the local idle mask when +* p->cpus_ptr is passed directly in @preferred_cpus. +*/ +if (preferred_cpus != p->cpus_ptr) { + preferred =this_cpu_cpumask_var_ptr (local_idle_cpumask); +if (!cpumask_and (preferred, p->cpus_ptr, preferred_cpus)) + preferred =NULL ;
I think it would be better to move cpumask_and() inside scx_select_cpu_dfl() because scx_select_cpu_dfl() assumes that anyway. That will make the code easier to read and avoid potential mistakes when extending scx_select_cpu_dfl() in the future.
Quoted 37 lines
kernel/sched/ext_idle.c +14
-0+ } + prev_cpu =scx_select_cpu_dfl (p, preferred, prev_cpu, wake_flags, flags, &is_idle); +if (!is_idle) + prev_cpu = -EBUSY ; + +preempt_enable (); +#else + prev_cpu = -EBUSY ; +#endif + +return prev_cpu; +} +/** * scx_bpf_get_idle_cpumask_node - Get a referenced kptr to the * idle-tracking per-CPU cpumask of a target NUMA node. @@ -1215,6 +1274,7 @@ static const struct btf_kfunc_id_set scx_kfunc_set_idle = {BTF_KFUNCS_START (scx_kfunc_ids_select_cpu)BTF_ID_FLAGS (func ,scx_bpf_select_cpu_dfl ,KF_RCU ) +BTF_ID_FLAGS (func, scx_bpf_select_cpu_pref,KF_RCU )BTF_KFUNCS_END (scx_kfunc_ids_select_cpu) static const struct btf_kfunc_id_set scx_kfunc_set_select_cpu = {tools/sched_ext/include/scx/common.bpf.h +2
-0@@ -47,6 +47,8 @@ static inline void ___vmlinux_h_sanity_check___(void) } s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym; +s32 scx_bpf_select_cpu_pref(struct task_struct *p, const struct cpumask *preferred_cpus, + s32 prev_cpu, u64 wake_flags, u64 flags) __ksym __weak; s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym; void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym __weak; void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym __weak;
Andrea Righi
reviewerRe: [PATCH 3/4] sched_ext: idle: Introduce scx_bpf_select_cpu_pref()
Hi Changwoo,
On Fri, Mar 07, 2025 at 12:15:04PM +0900, Changwoo Min wrote:
...
Quoted 27 lines
Quoted 27 lines
kernel/sched/ext_idle.c +27
-0+__bpf_kfunc s32 scx_bpf_select_cpu_pref(struct task_struct *p, +const struct cpumask *preferred_cpus, +s32 prev_cpu,u64 wake_flags,u64 flags) +{ +#ifdef CONFIG_SMP +struct cpumask *preferred =NULL ; +bool is_idle = false; +#endif + +if (!ops_cpu_valid (prev_cpu,NULL )) +return -EINVAL ; + +if (!check_builtin_idle_enabled ()) +return -EBUSY ; + +if (!scx_kf_allowed (SCX_KF_SELECT_CPU )) +return -EPERM ; + +#ifdef CONFIG_SMP +preempt_disable (); + +/* +* As an optimization, do not update the local idle mask when +* p->cpus_ptr is passed directly in @preferred_cpus. +*/ +if (preferred_cpus != p->cpus_ptr) { + preferred =this_cpu_cpumask_var_ptr (local_idle_cpumask);
Quoted 8 lines
kernel/sched/ext_idle.c __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, · line 955
Quoted 2 lines
kernel/sched/ext_idle.c +2
-0+if (!cpumask_and (preferred, p->cpus_ptr, preferred_cpus)) + preferred =NULL ;I think it would be better to move
cpumask_and()insidescx_select_cpu_dfl()becausescx_select_cpu_dfl()assumes that anyway. That will make the code easier to read and avoid potential mistakes when extendingscx_select_cpu_dfl()in the future.
I agree, will do this in the next version.
Thanks!
-Andrea
Andrea Righi
reviewerPATCH 4/4 selftests/sched_ext: Add test for scx_bpf_select_cpu_pref()
In reply to Andrea RighiAdd a selftest to validate the behavior of the built-in idle CPU selection policy with preferred CPUs, using scx_bpf_select_cpu_pref().
--- tools/testing/selftests/sched_ext/Makefile | 1 + .../selftests/sched_ext/pref_cpus.bpf.c | 95 +++++++++++++++++++ tools/testing/selftests/sched_ext/pref_cpus.c | 58 +++++++++++ 3 files changed, 154 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.bpf.c create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.c
tools/testing/selftests/sched_ext/Makefile +1 -0
@@ -173,6 +173,7 @@ auto-test-targets := \ maybe_null \ minimal \ numa \ + pref_cpus \ prog_run \ reload_loop \ select_cpu_dfl \
tools/testing/selftests/sched_ext/pref_cpus.bpf.c new file +95 -0
95 lines not on this page · raw message
tools/testing/selftests/sched_ext/pref_cpus.c new file +58 -0
58 lines not on this page · raw message
Signature · 1 line
2.48.1
Tejun Heo
reviewerHello,
Quoted 9 lines · Andrea Righi
To implement this, introduce a new helper kfunc
scx_bpf_select_cpu_pref()that allows to specify a cpumask of preferred CPUs:s32 scx_bpf_select_cpu_pref(struct
task_struct*p,const struct cpumask *preferred_cpus, s32 prev_cpu, u64 wake_flags, u64 flags);Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF that can be used to enforce selection strictly within the preferred domain.
Would something like scx_bpf_select_cpu_and() work which is only allowed pick in the intersection (ie. always SCX_PICK_IDLE_IN_PREF). I'm not sure how much more beneficial a built-in two-level mechanism is especially given that it wouldn't be too uncommon to need multi-level pick - e.g. within l3 then within numa node and so on.
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerHi Tejun,
Quoted 10 lines · Tejun Heo
Hello,
Quoted 7 lines · Andrea Righi
To implement this, introduce a new helper kfunc
scx_bpf_select_cpu_pref()that allows to specify a cpumask of preferred CPUs:s32 scx_bpf_select_cpu_pref(struct
task_struct*p,const struct cpumask *preferred_cpus, s32 prev_cpu, u64 wake_flags, u64 flags);
Quoted 8 lines
Quoted 2 lines
Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF that can be used to enforce selection strictly within the preferred domain.
Would something like
scx_bpf_select_cpu_and()work which is only allowed pick in the intersection (ie. alwaysSCX_PICK_IDLE_IN_PREF). I'm not sure how much more beneficial a built-in two-level mechanism is especially given that it wouldn't be too uncommon to need multi-level pick - e.g. within l3 then within numa node and so on.
Just to make sure I understand, you mean provide two separate kfuncs:scx_bpf_select_cpu_and() and scx_bpf_select_cpu_pref(), instead of introducing the flag?
Thanks,
-Andrea
Tejun Heo
reviewerQuoted 3 lines · Andrea Righi
Just to make sure I understand, you mean provide two separate kfuncs:
scx_bpf_select_cpu_and()andscx_bpf_select_cpu_pref(), instead of introducing the flag?
Oh I meant just having scx_bpf_select_cpu_and(). The caller can just call it twice for _pref() behavior, right?
Thanks.
Signature · 1 line
tejun
Andrea Righi
reviewerQuoted 7 lines · Tejun Heo
Quoted 3 lines · Andrea Righi
Just to make sure I understand, you mean provide two separate kfuncs:
scx_bpf_select_cpu_and()andscx_bpf_select_cpu_pref(), instead of introducing the flag?Oh I meant just having
scx_bpf_select_cpu_and(). The caller can just call it twice for_pref()behavior, right?
Oh I see, you call it for the pref CPUs first and then for all the CPUs to get the same behavior (similar to what we do with the SMT idle cores). Yeah, that can work. Good idea!
Thanks,
-Andrea
Changwoo Min
reviewerHi Andrea,
Thank you for submitting the patch set.
Quoted 11 lines · Andrea Righi
Many scx schedulers define their own concept of scheduling domains to represent topology characteristics, such as heterogeneous architectures (e.g., big.LITTLE, P-cores/E-cores), or to categorize tasks based on specific properties (e.g., setting the soft-affinity of certain tasks to a subset of CPUs).
Currently, there is no mechanism to share these domains with the built-in idle CPU selection policy. As a result, schedulers often implement their own idle CPU selection policies, which are typically similar to one another, leading to a lot of code duplication.
Quoted 7 lines
To address this, extend the built-in idle CPU selection policy introducing the concept of preferred CPUs.
With this concept, BPF schedulers can apply the built-in idle CPU selection policy to a subset of preferred CPUs, allowing them to implement their own scheduling domains while still using the topology optimizations optimizations of the built-in policy, preventing code duplication across
Typo here. There are two "optimizations".
Quoted 86 lines
different schedulers.
To implement this, introduce a new helper kfunc
scx_bpf_select_cpu_pref()that allows to specify a cpumask of preferred CPUs:s32 scx_bpf_select_cpu_pref(struct
task_struct*p,const struct cpumask *preferred_cpus, s32 prev_cpu, u64 wake_flags, u64 flags);Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF that can be used to enforce selection strictly within the preferred domain.
Example usage
=============s32 BPF_STRUCT_OPS(foo_select_cpu, struct
task_struct*p,s32 prev_cpu, u64 wake_flags) { const struct cpumask *dom = task_domain(p) ?: p->cpus_ptr; s32 cpu; /* * Pick an idle CPU in the task's domain. If no CPU is found, * extend the search outside the domain. */ cpu = scx_bpf_select_cpu_pref(p, dom, prev_cpu, wake_flags, 0); if (cpu >= 0) { scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0); return cpu; } return prev_cpu; }Results
=======Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of
scx_bpflandthat uses the new helperscx_bpf_select_cpu_pref()and0xff00as preferred domain:$ vng --cpu 16,sockets=4,cores=4,threads=1Starting 12 CPU hogs to fill the preferred domain:
$ stress-ng -c 12 ... 0[|||||||||||||||||||||||100.0%] 8[||||||||||||||||||||||||100.0%] 1[| 1.3%] 9[||||||||||||||||||||||||100.0%] 2[|||||||||||||||||||||||100.0%] 10[||||||||||||||||||||||||100.0%] 3[|||||||||||||||||||||||100.0%] 11[||||||||||||||||||||||||100.0%] 4[|||||||||||||||||||||||100.0%] 12[||||||||||||||||||||||||100.0%] 5[|| 2.6%] 13[||||||||||||||||||||||||100.0%] 6[| 0.6%] 14[||||||||||||||||||||||||100.0%] 7| 0.0%] 15[||||||||||||||||||||||||100.0%]Passing %SCX_PICK_IDLE_IN_PREF to
scx_bpf_select_cpu_pref()to enforce strict selection on the preferred CPUs (with the same workload):0[ 0.0%] 8[||||||||||||||||||||||||100.0%] 1[ 0.0%] 9[||||||||||||||||||||||||100.0%] 2[ 0.0%] 10[||||||||||||||||||||||||100.0%] 3[ 0.0%] 11[||||||||||||||||||||||||100.0%] 4[ 0.0%] 12[||||||||||||||||||||||||100.0%] 5[ 0.0%] 13[||||||||||||||||||||||||100.0%] 6[ 0.0%] 14[||||||||||||||||||||||||100.0%] 7[ 0.0%] 15[||||||||||||||||||||||||100.0%]Andrea Righi (4):
sched_ext: idle: Honor idle flags in the built-in idle selection policy
sched_ext: idle: Introduce the concept of preferred CPUs
sched_ext: idle: Introducescx_bpf_select_cpu_pref()selftests/sched_ext: Add test forscx_bpf_select_cpu_pref()kernel/sched/ext.c | 4 +- kernel/sched/ext_idle.c | 235 ++++++++++++++++++---- kernel/sched/ext_idle.h | 3 +- tools/sched_ext/include/scx/common.bpf.h | 2 + tools/sched_ext/include/scx/compat.h | 1 + tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/pref_cpus.bpf.c | 95 +++++++++ tools/testing/selftests/sched_ext/pref_cpus.c | 58 ++++++ 8 files changed, 354 insertions(+), 45 deletions(-)create mode 100644
tools/testing/selftests/sched_ext/pref_cpus.bpf.ccreate mode 100644tools/testing/selftests/sched_ext/pref_cpus.c