Skip to contentSkip to conversation

sched_ext: Enhance built-in idle selection with preferred CPUs

PATCH Superseded Andrea Righi 3 participants 12 messages Mar 6, 2025 – Mar 7, 2025

Subsystems (4)

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Andrea Righi

author reviewer
#0 lore raw

PATCH 1/1 sched_ext: Enhance built-in idle selection with preferred CPUs

Many scx schedulers define their own concept of scheduling domains to represent topology characteristics, such as heterogeneous architectures (e.g., big.LITTLE, P-cores/E-cores), or to categorize tasks based on specific properties (e.g., setting the soft-affinity of certain tasks to a subset of CPUs).

Currently, there is no mechanism to share these domains with the built-in idle CPU selection policy. As a result, schedulers often implement their own idle CPU selection policies, which are typically similar to one another, leading to a lot of code duplication.

To address this, extend the built-in idle CPU selection policy introducing the concept of preferred CPUs.

With this concept, BPF schedulers can apply the built-in idle CPU selection policy to a subset of preferred CPUs, allowing them to implement their own scheduling domains while still using the topology optimizations optimizations of the built-in policy, preventing code duplication across different schedulers.

To implement this, introduce a new helper kfunc scx_bpf_select_cpu_pref() that allows to specify a cpumask of preferred CPUs:

s32 scx_bpf_select_cpu_pref(struct task_struct *p,

			    const struct cpumask *preferred_cpus,
			    s32 prev_cpu, u64 wake_flags, u64 flags);

Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF that can be used to enforce selection strictly within the preferred domain.

Example usage
=============

s32 BPF_STRUCT_OPS(foo_select_cpu, struct task_struct *p,

		   s32 prev_cpu, u64 wake_flags)
{
	const struct cpumask *dom = task_domain(p) ?: p->cpus_ptr;
	s32 cpu;

	/*
	 * Pick an idle CPU in the task's domain. If no CPU is found,
	 * extend the search outside the domain.
	 */
	cpu = scx_bpf_select_cpu_pref(p, dom, prev_cpu, wake_flags, 0);
	if (cpu >= 0) {
		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
		return cpu;
	}

	return prev_cpu;
}

Results
=======

Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of scx_bpfland that uses the new helper scx_bpf_select_cpu_pref() and 0xff00 as preferred domain:

 $ vng --cpu 16,sockets=4,cores=4,threads=1

Starting 12 CPU hogs to fill the preferred domain:

 $ stress-ng -c 12
 ...
    0[|||||||||||||||||||||||100.0%]   8[||||||||||||||||||||||||100.0%]
    1[|                        1.3%]   9[||||||||||||||||||||||||100.0%]
    2[|||||||||||||||||||||||100.0%]  10[||||||||||||||||||||||||100.0%]
    3[|||||||||||||||||||||||100.0%]  11[||||||||||||||||||||||||100.0%]
    4[|||||||||||||||||||||||100.0%]  12[||||||||||||||||||||||||100.0%]
    5[||                       2.6%]  13[||||||||||||||||||||||||100.0%]
    6[|                        0.6%]  14[||||||||||||||||||||||||100.0%]
    7|                         0.0%]  15[||||||||||||||||||||||||100.0%]

Passing %SCX_PICK_IDLE_IN_PREF to scx_bpf_select_cpu_pref() to enforce strict selection on the preferred CPUs (with the same workload):

    0[                         0.0%]   8[||||||||||||||||||||||||100.0%]
    1[                         0.0%]   9[||||||||||||||||||||||||100.0%]
    2[                         0.0%]  10[||||||||||||||||||||||||100.0%]
    3[                         0.0%]  11[||||||||||||||||||||||||100.0%]
    4[                         0.0%]  12[||||||||||||||||||||||||100.0%]
    5[                         0.0%]  13[||||||||||||||||||||||||100.0%]
    6[                         0.0%]  14[||||||||||||||||||||||||100.0%]
    7[                         0.0%]  15[||||||||||||||||||||||||100.0%]

Andrea Righi (4):
sched_ext: idle: Honor idle flags in the built-in idle selection policy
sched_ext: idle: Introduce the concept of preferred CPUs
sched_ext: idle: Introduce scx_bpf_select_cpu_pref() selftests/sched_ext: Add test for scx_bpf_select_cpu_pref()

 kernel/sched/ext.c                                |   4 +-
 kernel/sched/ext_idle.c                           | 235 ++++++++++++++++++----
 kernel/sched/ext_idle.h                           |   3 +-
 tools/sched_ext/include/scx/common.bpf.h          |   2 +
 tools/sched_ext/include/scx/compat.h              |   1 +
 tools/testing/selftests/sched_ext/Makefile        |   1 +
 tools/testing/selftests/sched_ext/pref_cpus.bpf.c |  95 +++++++++
 tools/testing/selftests/sched_ext/pref_cpus.c     |  58 ++++++
 8 files changed, 354 insertions(+), 45 deletions(-)
 create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.c
author reviewer
#1 lore raw

PATCH 1/4 sched_ext: idle: Honor idle flags in the built-in idle selection policy

Enable passing idle flags (%SCX_PICK_IDLE_*) to scx_select_cpu_dfl(), to enforce strict selection criteria, such as selecting an idle CPU strictly within @prev_cpu's node or choosing only a fully idle SMT core.

This functionality will be exposed through a dedicated kfunc in a separate patch.

---
 kernel/sched/ext.c      |  2 +-
 kernel/sched/ext_idle.c | 41 ++++++++++++++++++++++++++++++-----------
 kernel/sched/ext_idle.h |  2 +-
 3 files changed, 32 insertions(+), 13 deletions(-)
kernel/sched/ext.c +1 -1
index debcd1cf2de9b..5cd878bbd0e39 100644
@@ -3396,7 +3396,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag
 		bool found;
 		s32 cpu;
 
-		cpu = scx_select_cpu_dfl(p, prev_cpu, wake_flags, &found);
+		cpu = scx_select_cpu_dfl(p, prev_cpu, wake_flags, 0, &found);
 		p->scx.selected_cpu = cpu;
 		if (found) {
 			p->scx.slice = SCX_SLICE_DFL;
kernel/sched/ext_idle.c +30 -11

98 lines not on this page · raw message

kernel/sched/ext_idle.h +1 -1
index 68c4307ce4f6f..5c1db6b315f7a 100644
@@ -27,7 +27,7 @@ static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node
 }
 #endif /* CONFIG_SMP */
 
-s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *found);
+s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64 flags, bool *found);
 void scx_idle_enable(struct sched_ext_ops *ops);
 void scx_idle_disable(void);
 int scx_idle_init(void);
Signature · 1 line
2.48.1
author reviewer
#2 lore raw

PATCH 2/4 sched_ext: idle: Introduce the concept of preferred CPUs

In reply to Andrea Righi

Many scx schedulers define their own concept of scheduling domains to represent topology characteristics, such as heterogeneous architectures (e.g., big.LITTLE, P-cores/E-cores), or to categorize tasks based on specific properties (e.g., setting the soft-affinity of certain tasks to a subset of CPUs).

Currently, there is no mechanism to share these domains with the built-in idle CPU selection policy. As a result, schedulers often implement their own idle CPU selection policies, which are typically similar to one another, leading to a lot of code duplication.

To address this, introduce the concept of preferred domain (represented as a cpumask) that can be used by the BPF schedulers to apply the built-in idle CPU selection policy to a subset of preferred CPUs.

With this concept the idle CPU selection policy becomes the following:
- always prioritize CPUs from fully idle SMT cores (if SMT is enabled),
- select the same CPU if it's idle and in the preferred domain,
- select an idle CPU within the same LLC domain, if the LLC domain is a subset of the preferred domain,
- select an idle CPU within the same node, if the node domain is a subset of the preferred domain,
- select an idle CPU within the preferred domain,
- select any idle CPU usable by the task.

Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF, that enforces strict selection within the preferred domain. Without this flag, the preferred domain is treated as a soft constraint: idle CPUs outside the preferred domain can be considered if the preferred domain is fully busy.

If the preferred domain is empty or NULL, the behavior of the built-in idle CPU selection policy remains unchanged.

This only introduces the core concept of preferred domain. This functionality will be exposed through a dedicated kfunc in a separate patch.

---
 kernel/sched/ext.c                   |   3 +-
 kernel/sched/ext_idle.c              | 142 ++++++++++++++++++++-------
 kernel/sched/ext_idle.h              |   3 +-
 tools/sched_ext/include/scx/compat.h |   1 +
 4 files changed, 111 insertions(+), 38 deletions(-)
kernel/sched/ext.c +2 -1
index 5cd878bbd0e39..a28ddd7655ba8 100644
@@ -807,6 +807,7 @@ enum scx_deq_flags {
 enum scx_pick_idle_cpu_flags {
 	SCX_PICK_IDLE_CORE	= 1LLU << 0,	/* pick a CPU whose SMT siblings are also idle */
 	SCX_PICK_IDLE_IN_NODE	= 1LLU << 1,	/* pick a CPU in the same target NUMA node */
+	SCX_PICK_IDLE_IN_PREF	= 1LLU << 2,	/* pick a CPU in the preferred domain */
 };
 
 enum scx_kick_flags {
@@ -3396,7 +3397,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag
 		bool found;
 		s32 cpu;
 
-		cpu = scx_select_cpu_dfl(p, prev_cpu, wake_flags, 0, &found);
+		cpu = scx_select_cpu_dfl(p, NULL, prev_cpu, wake_flags, 0, &found);
 		p->scx.selected_cpu = cpu;
 		if (found) {
 			p->scx.slice = SCX_SLICE_DFL;
kernel/sched/ext_idle.c +106 -36

229 lines not on this page · raw message

kernel/sched/ext_idle.h +2 -1
index 5c1db6b315f7a..386bde7e8ee3e 100644
@@ -27,7 +27,8 @@ static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node
 }
 #endif /* CONFIG_SMP */
 
-s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, u64 flags, bool *found);
+s32 scx_select_cpu_dfl(struct task_struct *p, const struct cpumask *preferred_cpus,
+		       s32 prev_cpu, u64 wake_flags, u64 flags, bool *found);
 void scx_idle_enable(struct sched_ext_ops *ops);
 void scx_idle_disable(void);
 int scx_idle_init(void);
tools/sched_ext/include/scx/compat.h +1 -0
index 35c67c5174ac0..f9c06079b3a86 100644
@@ -120,6 +120,7 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field
 
 #define SCX_PICK_IDLE_CORE SCX_PICK_IDLE_FLAG(SCX_PICK_IDLE_CORE)
 #define SCX_PICK_IDLE_IN_NODE SCX_PICK_IDLE_FLAG(SCX_PICK_IDLE_IN_NODE)
+#define SCX_PICK_IDLE_IN_PREF SCX_PICK_IDLE_FLAG(SCX_PICK_IDLE_IN_PREF)
 
 static inline long scx_hotplug_seq(void)
 {
Signature · 1 line
2.48.1
author reviewer
#3 lore raw

PATCH 3/4 sched_ext: idle: Introduce scx_bpf_select_cpu_pref()

In reply to Andrea Righi

Provide a new kfunc that can be used to apply the built-in idle CPU selection policy to a subset of preferred CPU:

s32 scx_bpf_select_cpu_pref(struct task_struct *p,

			    const struct cpumask *preferred_cpus,
			    s32 prev_cpu, u64 wake_flags, u64 flags);

This new helper is basically an extension of scx_bpf_select_cpu_dfl(). However, when an idle CPU can't be found, it returns a negative value instead of @prev_cpu, aligning its behavior more closely with scx_bpf_pick_idle_cpu().

It also accepts %SCX_PICK_IDLE_* flags, which can be used to enforce strict selection to the preferred CPUs (%SCX_PICK_IDLE_IN_PREF) or to @prev_cpu's node (%SCX_PICK_IDLE_IN_NODE), or to request only a full-idle SMT core (%SCX_PICK_IDLE_CORE), while applying the built-in selection logic.

With this helper, BPF schedulers can apply the built-in idle CPU selection policy to a generic CPU domain, with strict or soft selection requirements.

In the future we can also consider to deprecate scx_bpf_select_cpu_dfl() and replace it with scx_bpf_select_cpu_pref(), as the latter provides the same functionality, with the addition of the preferred domain logic.

Example usage
=============

Possible usage in ops.select_cpu():

s32 BPF_STRUCT_OPS(foo_select_cpu, struct task_struct *p,

		  s32 prev_cpu, u64 wake_flags)
{
	const struct cpumask *dom = task_domain(p) ?: p->cpus_ptr;
	s32 cpu;

	/*
	 * Pick an idle CPU in the task's domain. If no CPU is found,
	 * extend the search outside the domain.
	 */
	cpu = scx_bpf_select_cpu_pref(p, dom, prev_cpu, wake_flags, 0);
	if (cpu >= 0) {
		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
		return cpu;
	}

	return prev_cpu;
}

Results
=======

Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of scx_bpfland that uses the new helper scx_bpf_select_cpu_pref() and 0xff00 as preferred domain:

 $ vng --cpu 16,sockets=4,cores=4,threads=1

Starting 12 CPU hogs to fill the preferred domain:

 $ stress-ng -c 12
 ...
    0[|||||||||||||||||||||||100.0%]   8[||||||||||||||||||||||||100.0%]
    1[|                        1.3%]   9[||||||||||||||||||||||||100.0%]
    2[|||||||||||||||||||||||100.0%]  10[||||||||||||||||||||||||100.0%]
    3[|||||||||||||||||||||||100.0%]  11[||||||||||||||||||||||||100.0%]
    4[|||||||||||||||||||||||100.0%]  12[||||||||||||||||||||||||100.0%]
    5[||                       2.6%]  13[||||||||||||||||||||||||100.0%]
    6[|                        0.6%]  14[||||||||||||||||||||||||100.0%]
    7|                         0.0%]  15[||||||||||||||||||||||||100.0%]

Passing %SCX_PICK_IDLE_IN_PREF to scx_bpf_select_cpu_pref() to enforce strict selection on the preferred CPUs (with the same workload):

    0[                         0.0%]   8[||||||||||||||||||||||||100.0%]
    1[                         0.0%]   9[||||||||||||||||||||||||100.0%]
    2[                         0.0%]  10[||||||||||||||||||||||||100.0%]
    3[                         0.0%]  11[||||||||||||||||||||||||100.0%]
    4[                         0.0%]  12[||||||||||||||||||||||||100.0%]
    5[                         0.0%]  13[||||||||||||||||||||||||100.0%]
    6[                         0.0%]  14[||||||||||||||||||||||||100.0%]
    7[                         0.0%]  15[||||||||||||||||||||||||100.0%]
---
 kernel/sched/ext.c                       |  1 +
 kernel/sched/ext_idle.c                  | 60 ++++++++++++++++++++++++
 tools/sched_ext/include/scx/common.bpf.h |  2 +
 3 files changed, 63 insertions(+)
kernel/sched/ext.c +1 -0
index a28ddd7655ba8..8ee4818de908b 100644
@@ -465,6 +465,7 @@ struct sched_ext_ops {
 	 * idle CPU tracking and the following helpers become unavailable:
 	 *
 	 * - scx_bpf_select_cpu_dfl()
+	 * - scx_bpf_select_cpu_pref()
 	 * - scx_bpf_test_and_clear_cpu_idle()
 	 * - scx_bpf_pick_idle_cpu()
 	 *
kernel/sched/ext_idle.c +60 -0

72 lines not on this page · raw message

tools/sched_ext/include/scx/common.bpf.h +2 -0
index dc4333d23189f..a33e709ec12ab 100644
@@ -47,6 +47,8 @@ static inline void ___vmlinux_h_sanity_check___(void)
 }
 
 s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym;
+s32 scx_bpf_select_cpu_pref(struct task_struct *p, const struct cpumask *preferred_cpus,
+			    s32 prev_cpu, u64 wake_flags, u64 flags) __ksym __weak;
 s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym;
 void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym __weak;
 void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym __weak;
Signature · 1 line
2.48.1
reviewer
#4 lore raw

Re: [PATCH 3/4] sched_ext: idle: Introduce scx_bpf_select_cpu_pref()

Quoted 147 lines · Andrea Righi

Provide a new kfunc that can be used to apply the built-in idle CPU selection policy to a subset of preferred CPU:

s32 scx_bpf_select_cpu_pref(struct task_struct *p,

			    const struct cpumask *preferred_cpus,
			    s32 prev_cpu, u64 wake_flags, u64 flags);

This new helper is basically an extension of scx_bpf_select_cpu_dfl(). However, when an idle CPU can't be found, it returns a negative value instead of @prev_cpu, aligning its behavior more closely with scx_bpf_pick_idle_cpu().

It also accepts %SCX_PICK_IDLE_* flags, which can be used to enforce strict selection to the preferred CPUs (%SCX_PICK_IDLE_IN_PREF) or to @prev_cpu's node (%SCX_PICK_IDLE_IN_NODE), or to request only a full-idle SMT core (%SCX_PICK_IDLE_CORE), while applying the built-in selection logic.

With this helper, BPF schedulers can apply the built-in idle CPU selection policy to a generic CPU domain, with strict or soft selection requirements.

In the future we can also consider to deprecate scx_bpf_select_cpu_dfl() and replace it with scx_bpf_select_cpu_pref(), as the latter provides the same functionality, with the addition of the preferred domain logic.

Example usage
=============

Possible usage in ops.select_cpu():

s32 BPF_STRUCT_OPS(foo_select_cpu, struct task_struct *p,

		  s32 prev_cpu, u64 wake_flags)
{
	const struct cpumask *dom = task_domain(p) ?: p->cpus_ptr;
	s32 cpu;

	/*
	 * Pick an idle CPU in the task's domain. If no CPU is found,
	 * extend the search outside the domain.
	 */
	cpu = scx_bpf_select_cpu_pref(p, dom, prev_cpu, wake_flags, 0);
	if (cpu >= 0) {
		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
		return cpu;
	}

	return prev_cpu;
}

Results
=======

Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of scx_bpfland that uses the new helper scx_bpf_select_cpu_pref() and 0xff00 as preferred domain:

  $ vng --cpu 16,sockets=4,cores=4,threads=1

Starting 12 CPU hogs to fill the preferred domain:

  $ stress-ng -c 12
  ...
     0[|||||||||||||||||||||||100.0%]   8[||||||||||||||||||||||||100.0%]
     1[|                        1.3%]   9[||||||||||||||||||||||||100.0%]
     2[|||||||||||||||||||||||100.0%]  10[||||||||||||||||||||||||100.0%]
     3[|||||||||||||||||||||||100.0%]  11[||||||||||||||||||||||||100.0%]
     4[|||||||||||||||||||||||100.0%]  12[||||||||||||||||||||||||100.0%]
     5[||                       2.6%]  13[||||||||||||||||||||||||100.0%]
     6[|                        0.6%]  14[||||||||||||||||||||||||100.0%]
     7|                         0.0%]  15[||||||||||||||||||||||||100.0%]

Passing %SCX_PICK_IDLE_IN_PREF to scx_bpf_select_cpu_pref() to enforce strict selection on the preferred CPUs (with the same workload):

     0[                         0.0%]   8[||||||||||||||||||||||||100.0%]
     1[                         0.0%]   9[||||||||||||||||||||||||100.0%]
     2[                         0.0%]  10[||||||||||||||||||||||||100.0%]
     3[                         0.0%]  11[||||||||||||||||||||||||100.0%]
     4[                         0.0%]  12[||||||||||||||||||||||||100.0%]
     5[                         0.0%]  13[||||||||||||||||||||||||100.0%]
     6[                         0.0%]  14[||||||||||||||||||||||||100.0%]
     7[                         0.0%]  15[||||||||||||||||||||||||100.0%]
---
  kernel/sched/ext.c                       |  1 +
  kernel/sched/ext_idle.c                  | 60 ++++++++++++++++++++++++
  tools/sched_ext/include/scx/common.bpf.h |  2 +
  3 files changed, 63 insertions(+)
kernel/sched/ext.c +1 -0
index a28ddd7655ba8..8ee4818de908b 100644
@@ -465,6 +465,7 @@ struct sched_ext_ops {
  	 * idle CPU tracking and the following helpers become unavailable:
  	 *
  	 * - scx_bpf_select_cpu_dfl()
+	 * - scx_bpf_select_cpu_pref()
  	 * - scx_bpf_test_and_clear_cpu_idle()
  	 * - scx_bpf_pick_idle_cpu()
  	 *
kernel/sched/ext_idle.c +36 -0
index 9b002e109404b..24cba7ddceec4 100644
@@ -907,6 +907,65 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu,
  	return prev_cpu;
  }
 
+/**
+ * scx_bpf_select_cpu_pref - Pick an idle CPU usable by task @p,
+ *			     prioritizing those in @preferred_cpus
+ * @p: task_struct to select a CPU for
+ * @preferred_cpus: cpumask of preferred CPUs
+ * @prev_cpu: CPU @p was on previously
+ * @wake_flags: %SCX_WAKE_* flags
+ * @flags: %SCX_PICK_IDLE* flags
+ *
+ * Can only be called from ops.select_cpu() if the built-in CPU selection is
+ * enabled - ops.update_idle() is missing or %SCX_OPS_KEEP_BUILTIN_IDLE is set.
+ * @p, @prev_cpu and @wake_flags match ops.select_cpu().
+ *
+ * Returns the selected idle CPU, which will be automatically awakened upon
+ * returning from ops.select_cpu() and can be used for direct dispatch, or
+ * a negative value if no idle CPU is available.
+ */
+__bpf_kfunc s32 scx_bpf_select_cpu_pref(struct task_struct *p,
+					const struct cpumask *preferred_cpus,
+					s32 prev_cpu, u64 wake_flags, u64 flags)
+{
+#ifdef CONFIG_SMP
+	struct cpumask *preferred = NULL;
+	bool is_idle = false;
+#endif
+
+	if (!ops_cpu_valid(prev_cpu, NULL))
+		return -EINVAL;
+
+	if (!check_builtin_idle_enabled())
+		return -EBUSY;
+
+	if (!scx_kf_allowed(SCX_KF_SELECT_CPU))
+		return -EPERM;
+
+#ifdef CONFIG_SMP

kernel/sched/ext_idle.c __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, · line 955

Quoted 10 lines
kernel/sched/ext_idle.c +10 -0
+	preempt_disable();
+
+	/*
+	 * As an optimization, do not update the local idle mask when
+	 * p->cpus_ptr is passed directly in @preferred_cpus.
+	 */
+	if (preferred_cpus != p->cpus_ptr) {
+		preferred = this_cpu_cpumask_var_ptr(local_idle_cpumask);
+		if (!cpumask_and(preferred, p->cpus_ptr, preferred_cpus))
+			preferred = NULL;

I think it would be better to move cpumask_and() inside scx_select_cpu_dfl() because scx_select_cpu_dfl() assumes that anyway. That will make the code easier to read and avoid potential mistakes when extending scx_select_cpu_dfl() in the future.

Quoted 37 lines
kernel/sched/ext_idle.c +14 -0
+	}
+	prev_cpu = scx_select_cpu_dfl(p, preferred, prev_cpu, wake_flags, flags, &is_idle);
+	if (!is_idle)
+		prev_cpu = -EBUSY;
+
+	preempt_enable();
+#else
+	prev_cpu = -EBUSY;
+#endif
+
+	return prev_cpu;
+}
+
  /**
   * scx_bpf_get_idle_cpumask_node - Get a referenced kptr to the
   * idle-tracking per-CPU cpumask of a target NUMA node.
@@ -1215,6 +1274,7 @@ static const struct btf_kfunc_id_set scx_kfunc_set_idle = {
 
  BTF_KFUNCS_START(scx_kfunc_ids_select_cpu)
  BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_RCU)
+BTF_ID_FLAGS(func, scx_bpf_select_cpu_pref, KF_RCU)
  BTF_KFUNCS_END(scx_kfunc_ids_select_cpu)
 
  static const struct btf_kfunc_id_set scx_kfunc_set_select_cpu = {
tools/sched_ext/include/scx/common.bpf.h +2 -0
index dc4333d23189f..a33e709ec12ab 100644
@@ -47,6 +47,8 @@ static inline void ___vmlinux_h_sanity_check___(void)
  }
 
  s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym;
+s32 scx_bpf_select_cpu_pref(struct task_struct *p, const struct cpumask *preferred_cpus,
+			    s32 prev_cpu, u64 wake_flags, u64 flags) __ksym __weak;
  s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym;
  void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym __weak;
  void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym __weak;
author reviewer
#5 lore raw

Re: [PATCH 3/4] sched_ext: idle: Introduce scx_bpf_select_cpu_pref()

Hi Changwoo,

On Fri, Mar 07, 2025 at 12:15:04PM +0900, Changwoo Min wrote:
...

Quoted 27 lines
Quoted 27 lines
kernel/sched/ext_idle.c +27 -0
+__bpf_kfunc s32 scx_bpf_select_cpu_pref(struct task_struct *p,
+					const struct cpumask *preferred_cpus,
+					s32 prev_cpu, u64 wake_flags, u64 flags)
+{
+#ifdef CONFIG_SMP
+	struct cpumask *preferred = NULL;
+	bool is_idle = false;
+#endif
+
+	if (!ops_cpu_valid(prev_cpu, NULL))
+		return -EINVAL;
+
+	if (!check_builtin_idle_enabled())
+		return -EBUSY;
+
+	if (!scx_kf_allowed(SCX_KF_SELECT_CPU))
+		return -EPERM;
+
+#ifdef CONFIG_SMP
+	preempt_disable();
+
+	/*
+	 * As an optimization, do not update the local idle mask when
+	 * p->cpus_ptr is passed directly in @preferred_cpus.
+	 */
+	if (preferred_cpus != p->cpus_ptr) {
+		preferred = this_cpu_cpumask_var_ptr(local_idle_cpumask);
Quoted 8 lines

kernel/sched/ext_idle.c __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, · line 955

Quoted 2 lines
kernel/sched/ext_idle.c +2 -0
+		if (!cpumask_and(preferred, p->cpus_ptr, preferred_cpus))
+			preferred = NULL;

I think it would be better to move cpumask_and() inside scx_select_cpu_dfl() because scx_select_cpu_dfl() assumes that anyway. That will make the code easier to read and avoid potential mistakes when extending scx_select_cpu_dfl() in the future.

I agree, will do this in the next version.

Thanks!
-Andrea

author reviewer
#6 lore raw

PATCH 4/4 selftests/sched_ext: Add test for scx_bpf_select_cpu_pref()

In reply to Andrea Righi

Add a selftest to validate the behavior of the built-in idle CPU selection policy with preferred CPUs, using scx_bpf_select_cpu_pref().

---
 tools/testing/selftests/sched_ext/Makefile    |  1 +
 .../selftests/sched_ext/pref_cpus.bpf.c       | 95 +++++++++++++++++++
 tools/testing/selftests/sched_ext/pref_cpus.c | 58 +++++++++++
 3 files changed, 154 insertions(+)
 create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.bpf.c
 create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.c
tools/testing/selftests/sched_ext/Makefile +1 -0
index f4531327b8e76..44fd180111389 100644
@@ -173,6 +173,7 @@ auto-test-targets :=			\
 	maybe_null			\
 	minimal				\
 	numa				\
+	pref_cpus			\
 	prog_run			\
 	reload_loop			\
 	select_cpu_dfl			\
tools/testing/selftests/sched_ext/pref_cpus.bpf.c new file +95 -0

95 lines not on this page · raw message

tools/testing/selftests/sched_ext/pref_cpus.c new file +58 -0

58 lines not on this page · raw message

Signature · 1 line
2.48.1
reviewer
#7 lore raw
In reply to Andrea Righi

Hello,

Quoted 9 lines · Andrea Righi

To implement this, introduce a new helper kfunc scx_bpf_select_cpu_pref() that allows to specify a cpumask of preferred CPUs:

s32 scx_bpf_select_cpu_pref(struct task_struct *p,

			    const struct cpumask *preferred_cpus,
			    s32 prev_cpu, u64 wake_flags, u64 flags);

Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF that can be used to enforce selection strictly within the preferred domain.

Would something like scx_bpf_select_cpu_and() work which is only allowed pick in the intersection (ie. always SCX_PICK_IDLE_IN_PREF). I'm not sure how much more beneficial a built-in two-level mechanism is especially given that it wouldn't be too uncommon to need multi-level pick - e.g. within l3 then within numa node and so on.

Thanks.

Signature · 1 line
tejun
author reviewer
#8 lore raw

Hi Tejun,

Quoted 10 lines · Tejun Heo

Hello,

Quoted 7 lines · Andrea Righi

To implement this, introduce a new helper kfunc scx_bpf_select_cpu_pref() that allows to specify a cpumask of preferred CPUs:

s32 scx_bpf_select_cpu_pref(struct task_struct *p,

			    const struct cpumask *preferred_cpus,
			    s32 prev_cpu, u64 wake_flags, u64 flags);
Quoted 8 lines
Quoted 2 lines

Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF that can be used to enforce selection strictly within the preferred domain.

Would something like scx_bpf_select_cpu_and() work which is only allowed pick in the intersection (ie. always SCX_PICK_IDLE_IN_PREF). I'm not sure how much more beneficial a built-in two-level mechanism is especially given that it wouldn't be too uncommon to need multi-level pick - e.g. within l3 then within numa node and so on.

Just to make sure I understand, you mean provide two separate kfuncs:
scx_bpf_select_cpu_and() and scx_bpf_select_cpu_pref(), instead of introducing the flag?

Thanks,
-Andrea

reviewer
#9 lore raw
Quoted 3 lines · Andrea Righi

Just to make sure I understand, you mean provide two separate kfuncs:
scx_bpf_select_cpu_and() and scx_bpf_select_cpu_pref(), instead of introducing the flag?

Oh I meant just having scx_bpf_select_cpu_and(). The caller can just call it twice for _pref() behavior, right?

Thanks.

Signature · 1 line
tejun
author reviewer
#10 lore raw
Quoted 7 lines · Tejun Heo
Quoted 3 lines · Andrea Righi

Just to make sure I understand, you mean provide two separate kfuncs:
scx_bpf_select_cpu_and() and scx_bpf_select_cpu_pref(), instead of introducing the flag?

Oh I meant just having scx_bpf_select_cpu_and(). The caller can just call it twice for _pref() behavior, right?

Oh I see, you call it for the pref CPUs first and then for all the CPUs to get the same behavior (similar to what we do with the SMT idle cores). Yeah, that can work. Good idea!

Thanks,
-Andrea

reviewer
#11 lore raw
In reply to Andrea Righi

Hi Andrea,

Thank you for submitting the patch set.

Quoted 11 lines · Andrea Righi

Many scx schedulers define their own concept of scheduling domains to represent topology characteristics, such as heterogeneous architectures (e.g., big.LITTLE, P-cores/E-cores), or to categorize tasks based on specific properties (e.g., setting the soft-affinity of certain tasks to a subset of CPUs).

Currently, there is no mechanism to share these domains with the built-in idle CPU selection policy. As a result, schedulers often implement their own idle CPU selection policies, which are typically similar to one another, leading to a lot of code duplication.

Quoted 7 lines

To address this, extend the built-in idle CPU selection policy introducing the concept of preferred CPUs.

With this concept, BPF schedulers can apply the built-in idle CPU selection policy to a subset of preferred CPUs, allowing them to implement their own scheduling domains while still using the topology optimizations optimizations of the built-in policy, preventing code duplication across

Typo here. There are two "optimizations".

Quoted 86 lines

different schedulers.

To implement this, introduce a new helper kfunc scx_bpf_select_cpu_pref() that allows to specify a cpumask of preferred CPUs:

s32 scx_bpf_select_cpu_pref(struct task_struct *p,

			    const struct cpumask *preferred_cpus,
			    s32 prev_cpu, u64 wake_flags, u64 flags);

Moreover, introduce the new idle flag %SCX_PICK_IDLE_IN_PREF that can be used to enforce selection strictly within the preferred domain.

Example usage
=============

s32 BPF_STRUCT_OPS(foo_select_cpu, struct task_struct *p,

		   s32 prev_cpu, u64 wake_flags)
{
	const struct cpumask *dom = task_domain(p) ?: p->cpus_ptr;
	s32 cpu;

	/*
	 * Pick an idle CPU in the task's domain. If no CPU is found,
	 * extend the search outside the domain.
	 */
	cpu = scx_bpf_select_cpu_pref(p, dom, prev_cpu, wake_flags, 0);
	if (cpu >= 0) {
		scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
		return cpu;
	}

	return prev_cpu;
}

Results
=======

Load distribution on a 4 sockets / 4 cores per socket system, simulated using virtme-ng, running a modified version of scx_bpfland that uses the new helper scx_bpf_select_cpu_pref() and 0xff00 as preferred domain:

  $ vng --cpu 16,sockets=4,cores=4,threads=1

Starting 12 CPU hogs to fill the preferred domain:

  $ stress-ng -c 12
  ...
     0[|||||||||||||||||||||||100.0%]   8[||||||||||||||||||||||||100.0%]
     1[|                        1.3%]   9[||||||||||||||||||||||||100.0%]
     2[|||||||||||||||||||||||100.0%]  10[||||||||||||||||||||||||100.0%]
     3[|||||||||||||||||||||||100.0%]  11[||||||||||||||||||||||||100.0%]
     4[|||||||||||||||||||||||100.0%]  12[||||||||||||||||||||||||100.0%]
     5[||                       2.6%]  13[||||||||||||||||||||||||100.0%]
     6[|                        0.6%]  14[||||||||||||||||||||||||100.0%]
     7|                         0.0%]  15[||||||||||||||||||||||||100.0%]

Passing %SCX_PICK_IDLE_IN_PREF to scx_bpf_select_cpu_pref() to enforce strict selection on the preferred CPUs (with the same workload):

     0[                         0.0%]   8[||||||||||||||||||||||||100.0%]
     1[                         0.0%]   9[||||||||||||||||||||||||100.0%]
     2[                         0.0%]  10[||||||||||||||||||||||||100.0%]
     3[                         0.0%]  11[||||||||||||||||||||||||100.0%]
     4[                         0.0%]  12[||||||||||||||||||||||||100.0%]
     5[                         0.0%]  13[||||||||||||||||||||||||100.0%]
     6[                         0.0%]  14[||||||||||||||||||||||||100.0%]
     7[                         0.0%]  15[||||||||||||||||||||||||100.0%]

Andrea Righi (4):
sched_ext: idle: Honor idle flags in the built-in idle selection policy
sched_ext: idle: Introduce the concept of preferred CPUs
sched_ext: idle: Introduce scx_bpf_select_cpu_pref() selftests/sched_ext: Add test for scx_bpf_select_cpu_pref()

  kernel/sched/ext.c                                |   4 +-
  kernel/sched/ext_idle.c                           | 235 ++++++++++++++++++----
  kernel/sched/ext_idle.h                           |   3 +-
  tools/sched_ext/include/scx/common.bpf.h          |   2 +
  tools/sched_ext/include/scx/compat.h              |   1 +
  tools/testing/selftests/sched_ext/Makefile        |   1 +
  tools/testing/selftests/sched_ext/pref_cpus.bpf.c |  95 +++++++++
  tools/testing/selftests/sched_ext/pref_cpus.c     |  58 ++++++
  8 files changed, 354 insertions(+), 45 deletions(-)

create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.bpf.c create mode 100644 tools/testing/selftests/sched_ext/pref_cpus.c