Skip to contentSkip to conversation

sched_ext: Make kfunc filters support struct_ops context to reduce runtime overhead

RFC PATCH In review Cheng-Yang Chou 1 participant 4 messages Apr 6

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover sched_ext: Make kfunc filters support struct_ops context to reduce runtime overhead

Hi all,

First of all, thanks to Juntong for letting me continue his work! Since he is out of bandwidth to work on, and Andrea pointed out that this is a feature that may benefit the sched_ext community (thanks Andrea as well).

In this patchset I add myself as the author, while keeping Juntong as Co-developed-by/Signed-off-by to honor his work.

Please let me know if this is not appropriate!

---

This patchset makes kfunc filters support the use of struct_ops context information to enforce SCX kfunc restrictions at BPF load time instead of at runtime.

With the improved kfunc filters, SCX no longer needs the mask-based runtime kfunc call restriction, so this patch series removes it.

Test plan:
- All sched_ext selftests passed.
- Tested all demo schedulers (except scx_flatcg) and multiple upstream SCX schedulers (scx_beerland, scx_bpfland, scx_cosmos, scx_flash, scx_lavd) under stress (stress-ng --cpu 0 --timeout 10s) with no crashes.

[RFC]
I noticed some selftests are still missing, but I am not sure what kind of tests to implement here. Any suggestions would be appreciated.

Thanks,
Cheng-Yang

---

[Cheng-Yang Chou]
Changes in v4:
- Rebase on tj/for-7.1 branch
- Remove TRACING access to select_cpu kfuncs. They call task_rq_lock() internally and are unsafe from arbitrary tracing contexts
- Update scx_bpf_task_cgroup() comment to reflect that the restriction no longer guarantees rq is locked, only that @p is one of the tasks being operated on
- Add prog->aux->st_ops check in the context filter to avoid incorrect kfunc restrictions in non-SCX struct_ops where member offsets carry a different meaning
- Fix ops.enqueue() to also allow select_cpu kfuncs (SCX_OPS_KF_SELECT_CPU), so it can call kfuncs such as scx_bpf_select_cpu_dfl()
- Allow select_cpu kfuncs from BPF_PROG_TYPE_SYSCALL progs in the filter
- Update scx_kf_allowed_if_unlocked() to use scx_locked_rq() and kf_tasks[0] now that kf_mask is removed
- Drop selftests update patch: ops.enqueue() now allows select_cpu kfuncs, so the expected-failure logic is no longer valid
- Use SCX_KFUNCS_* macros to deduplicate kfunc IDs across per-group sets and the aggregated set (Tejun Heo)
- Fix [SCX_OP_IDX(dump)] to use SCX_OPS_KF_UNLOCKED instead of SCX_OPS_KF_DISPATCH, since ops.dump() is a sleepable callback (Tejun Heo)
- Fix SCX_OPS_KF_UNLOCKED enum value to 1 << 0 (Tejun Heo)
- Fix non-SCX struct_ops filter to deny rq-locked kfuncs and only allow unlocked kfuncs (Tejun Heo)
- Fix missed SCX_KF_REST in scx_prio_less() (Andrea Righi)
- Merge "Declare context-sensitive kfunc groups" and "Add scx_kfunc_ids_ops_context_sensitive" into a single commit (Tejun Heo)
- Link to v3:
https://lore.kernel.org/r/AM6PR03MB50806070E3D56208DDB8131699C22@AM6PR03MB5080.eurprd03.prod.outlook.com/

[Juntong Deng]
Changes in v3:
- Change scx_kfunc_ids_ops_context to the more descriptive scx_kfunc_ids_ops_context_sensitive
- Change the target to sched_ext/for-6.15
- Remove RFC tag
- Link to v2:
https://lore.kernel.org/r/AM6PR03MB5080855B90C3FE9B6C4243B099FE2@AM6PR03MB5080.eurprd03.prod.outlook.com/

[Juntong Deng]
Changes in v2:
- Use completely new design in filtering
- Link to v1:
https://lore.kernel.org/r/AM6PR03MB5080261D024B49D26F3FFF0099F72@AM6PR03MB5080.eurprd03.prod.outlook.com/

---

Cheng-Yang Chou (2):
sched_ext: Add verifier-time kfunc context filter
sched_ext: Remove runtime kfunc mask enforcement

 include/linux/sched/ext.h   |  27 ---
 kernel/sched/ext.c          | 430 +++++++++++++++++-------------------
 kernel/sched/ext_idle.c     |  20 +-
 kernel/sched/ext_internal.h |  71 +++++-
 4 files changed, 288 insertions(+), 260 deletions(-)
Signature · 1 line
2.48.1
#1 lore raw

PATCH v4 1/2 sched_ext: Add verifier-time kfunc context filter

Introduce verifier-time kfunc context restrictions for SCX, using the struct_ops context information in struct bpf_prog_aux.

Per-group kfunc sets are defined via SCX_KFUNCS_* macros in ext_internal.h, keeping the per-group lookup sets and the aggregated filtered set automatically in sync.

The groups are:
- SELECT_CPU: ops.select_cpu() and ops.enqueue()
- ENQUEUE_DISPATCH: ops.enqueue() and ops.dispatch()
- DISPATCH_ONLY: ops.dispatch() (rq-locked only)
- DISPATCH_UNLOCKED: shared by DISPATCH and UNLOCKED
- CPU_RELEASE: ops.cpu_release()
- UNLOCKED_ONLY: sleepable ops

scx_ops_context_flags[] maps each op to its permitted groups.

The filter handles three cases:
- SYSCALL progs may call unlocked and select_cpu kfuncs
- non-SCX struct_ops may call only unlocked kfuncs
- SCX struct_ops may call whatever their op's flags allow.

The four per-group BTF set registrations are replaced by a single scx_kfunc_set_ops_context_sensitive covering both struct_ops and syscall program types.

Previously, __scx_bpf_select_cpu_and(), scx_bpf_select_cpu_and(), and scx_bpf_select_cpu_dfl() were accessible from BPF_PROG_TYPE_TRACING programs via scx_kfunc_ids_idle. These kfuncs are no longer available to TRACING programs, as they internally call task_rq_lock() and cannot be used safely from arbitrary tracing contexts where pi_lock state is unknown.

---
 kernel/sched/ext.c          | 206 ++++++++++++++++++++++++++----------
 kernel/sched/ext_idle.c     |  13 ++-
 kernel/sched/ext_internal.h |  64 +++++++++++
 3 files changed, 227 insertions(+), 56 deletions(-)
kernel/sched/ext.c +153 -53
index b757b853b42b..525592851b5b 100644
@@ -613,12 +613,8 @@ static __always_inline bool scx_kf_allowed(struct scx_sched *sch, u32 mask)
 
 /* see SCX_CALL_OP_TASK() */
 static __always_inline bool scx_kf_allowed_on_arg_tasks(struct scx_sched *sch,
-							u32 mask,
 							struct task_struct *p)
 {
-	if (!scx_kf_allowed(sch, mask))
-		return false;
-
 	if (unlikely((p != current->scx.kf_tasks[0] &&
 		      p != current->scx.kf_tasks[1]))) {
 		scx_error(sch, "called on a task not being operated on");
@@ -8493,24 +8489,11 @@ __bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux *
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(scx_kfunc_ids_dispatch)
-BTF_ID_FLAGS(func, scx_bpf_dispatch_nr_slots, KF_IMPLICIT_ARGS)
-BTF_ID_FLAGS(func, scx_bpf_dispatch_cancel, KF_IMPLICIT_ARGS)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move_to_local, KF_IMPLICIT_ARGS)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move_to_local___v2, KF_IMPLICIT_ARGS)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move_set_slice, KF_RCU)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move_set_vtime, KF_RCU)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move, KF_RCU)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move_vtime, KF_RCU)
-#ifdef CONFIG_EXT_SUB_SCHED
-BTF_ID_FLAGS(func, scx_bpf_sub_dispatch, KF_IMPLICIT_ARGS)
-#endif
+SCX_KFUNCS_DISPATCH_ONLY
+SCX_KFUNCS_DISPATCH_UNLOCKED
+SCX_KFUNCS_SUB_DISPATCH
 BTF_KFUNCS_END(scx_kfunc_ids_dispatch)
 
-static const struct btf_kfunc_id_set scx_kfunc_set_dispatch = {
-	.owner			= THIS_MODULE,
-	.set			= &scx_kfunc_ids_dispatch,
-};
-
 __bpf_kfunc_start_defs();
 
 /**
@@ -8543,14 +8526,9 @@ __bpf_kfunc u32 scx_bpf_reenqueue_local(const struct bpf_prog_aux *aux)
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(scx_kfunc_ids_cpu_release)
-BTF_ID_FLAGS(func, scx_bpf_reenqueue_local, KF_IMPLICIT_ARGS)
+SCX_KFUNCS_CPU_RELEASE
 BTF_KFUNCS_END(scx_kfunc_ids_cpu_release)
 
-static const struct btf_kfunc_id_set scx_kfunc_set_cpu_release = {
-	.owner			= THIS_MODULE,
-	.set			= &scx_kfunc_ids_cpu_release,
-};
-
 __bpf_kfunc_start_defs();
 
 /**
@@ -8611,18 +8589,10 @@ __bpf_kfunc s32 scx_bpf_create_dsq(u64 dsq_id, s32 node, const struct bpf_prog_a
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(scx_kfunc_ids_unlocked)
-BTF_ID_FLAGS(func, scx_bpf_create_dsq, KF_IMPLICIT_ARGS | KF_SLEEPABLE)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move_set_slice, KF_RCU)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move_set_vtime, KF_RCU)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move, KF_RCU)
-BTF_ID_FLAGS(func, scx_bpf_dsq_move_vtime, KF_RCU)
+SCX_KFUNCS_UNLOCKED_ONLY
+SCX_KFUNCS_DISPATCH_UNLOCKED
 BTF_KFUNCS_END(scx_kfunc_ids_unlocked)
 
-static const struct btf_kfunc_id_set scx_kfunc_set_unlocked = {
-	.owner			= THIS_MODULE,
-	.set			= &scx_kfunc_ids_unlocked,
-};
-
 __bpf_kfunc_start_defs();
 
 /**
@@ -9528,8 +9498,11 @@ __bpf_kfunc void scx_bpf_events(struct scx_event_stats *events,
  * from the scheduler's POV. SCX operations should use this function to
  * determine @p's current cgroup as, unlike following @p->cgroups,
  * @p->sched_task_group is protected by @p's rq lock and thus atomic w.r.t. all
- * rq-locked operations. Can be called on the parameter tasks of rq-locked
- * operations. The restriction guarantees that @p's rq is locked by the caller.
+ * rq-locked operations. Can be called on the parameter tasks of any task-based
+ * SCX op. The restriction guarantees that @p is one of the tasks being operated
+ * on by the current SCX callback. When the rq is not held (e.g. ops.select_cpu()
+ * called from try_to_wake_up()), @p->sched_task_group is stable under @p's
+ * pi_lock.
  */
 __bpf_kfunc struct cgroup *scx_bpf_task_cgroup(struct task_struct *p,
 					       const struct bpf_prog_aux *aux)
@@ -9544,7 +9517,7 @@ __bpf_kfunc struct cgroup *scx_bpf_task_cgroup(struct task_struct *p,
 	if (unlikely(!sch))
 		goto out;
 
-	if (!scx_kf_allowed_on_arg_tasks(sch, __SCX_KF_RQ_LOCKED, p))
+	if (!scx_kf_allowed_on_arg_tasks(sch, p))
 		goto out;
 
 	cgrp = tg_cgrp(tg);
@@ -9557,6 +9530,141 @@ __bpf_kfunc struct cgroup *scx_bpf_task_cgroup(struct task_struct *p,
 
 __bpf_kfunc_end_defs();
 
+/*
+ * Map each SCX operation to its allowed kfunc groups. Indexed by
+ * SCX_MOFF_IDX(offsetof(struct sched_ext_ops, op)).
+ */
+const u32 scx_ops_context_flags[] = {
+	[SCX_OP_IDX(select_cpu)]		= SCX_OPS_KF_SELECT_CPU | SCX_OPS_KF_ENQUEUE,
+	[SCX_OP_IDX(enqueue)]			= SCX_OPS_KF_ENQUEUE | SCX_OPS_KF_SELECT_CPU,
+	[SCX_OP_IDX(dequeue)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(dispatch)]			= SCX_OPS_KF_DISPATCH | SCX_OPS_KF_ENQUEUE,
+	[SCX_OP_IDX(tick)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(runnable)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(running)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(stopping)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(quiescent)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(yield)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(core_sched_before)]		= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(set_weight)]		= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(set_cpumask)]		= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(update_idle)]		= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(cpu_acquire)]		= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(cpu_release)]		= SCX_OPS_KF_CPU_RELEASE,
+	[SCX_OP_IDX(init_task)]			= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(exit_task)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(enable)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(disable)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(dump)]			= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(dump_cpu)]			= SCX_OPS_KF_ANY,
+	[SCX_OP_IDX(dump_task)]			= SCX_OPS_KF_ANY,
+#ifdef CONFIG_EXT_GROUP_SCHED
+	[SCX_OP_IDX(cgroup_init)]		= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(cgroup_exit)]		= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(cgroup_prep_move)]		= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(cgroup_move)]		= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(cgroup_cancel_move)]	= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(cgroup_set_weight)]		= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(cgroup_set_bandwidth)]	= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(cgroup_set_idle)]		= SCX_OPS_KF_UNLOCKED,
+#endif	/* CONFIG_EXT_GROUP_SCHED */
+	[SCX_OP_IDX(sub_attach)]		= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(sub_detach)]		= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(cpu_online)]		= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(cpu_offline)]		= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(init)]			= SCX_OPS_KF_UNLOCKED,
+	[SCX_OP_IDX(exit)]			= SCX_OPS_KF_UNLOCKED,
+};
+
+/*
+ * Aggregated set of all context-sensitive kfuncs. Each kfunc appears exactly
+ * once; the shared macros from ext_internal.h keep this in sync with the
+ * per-group sets used for lookup in the filter below.
+ */
+BTF_KFUNCS_START(scx_kfunc_ids_ops_context_sensitive)
+SCX_KFUNCS_SELECT_CPU
+SCX_KFUNCS_ENQUEUE_DISPATCH
+SCX_KFUNCS_DISPATCH_ONLY
+SCX_KFUNCS_SUB_DISPATCH
+SCX_KFUNCS_CPU_RELEASE
+SCX_KFUNCS_UNLOCKED_ONLY
+SCX_KFUNCS_DISPATCH_UNLOCKED
+BTF_KFUNCS_END(scx_kfunc_ids_ops_context_sensitive)
+
+static int scx_kfunc_ids_ops_context_sensitive_filter(const struct bpf_prog *prog,
+						      u32 kfunc_id)
+{
+	u32 moff, flags;
+
+	if (!btf_id_set8_contains(&scx_kfunc_ids_ops_context_sensitive, kfunc_id))
+		return 0;
+
+	/*
+	 * Allow unlocked kfuncs (both sleepable UNLOCKED_ONLY and non-sleepable
+	 * DISPATCH_UNLOCKED) and select_cpu kfuncs from syscall prog type
+	 * (e.g. BPF test_run() call). None of these require the rq lock.
+	 */
+	if (prog->type == BPF_PROG_TYPE_SYSCALL &&
+	    (btf_id_set8_contains(&scx_kfunc_ids_unlocked, kfunc_id) ||
+	     btf_id_set8_contains(&scx_kfunc_ids_select_cpu, kfunc_id)))
+		return 0;
+
+	/*
+	 * add_subprog_and_kfunc() collects all kfunc calls, including dead
+	 * code guarded by bpf_ksym_exists(), before check_attach_btf_id()
+	 * sets prog->aux->st_ops. Allow all kfuncs when st_ops is not yet
+	 * set; do_check_main() re-runs the filter with st_ops set and
+	 * enforces the actual restrictions.
+	 */
+	if (prog->type == BPF_PROG_TYPE_STRUCT_OPS && !prog->aux->st_ops)
+		return 0;
+
+	/*
+	 * Non-SCX struct_ops: only allow unlocked kfuncs. All other
+	 * context-sensitive kfuncs assume the rq lock is held, which wouldn't
+	 * be true for other struct_ops implementations.
+	 */
+	if (prog->type == BPF_PROG_TYPE_STRUCT_OPS &&
+	    prog->aux->st_ops != &bpf_sched_ext_ops)
+		return btf_id_set8_contains(&scx_kfunc_ids_unlocked, kfunc_id) ?
+		       0 : -EACCES;
+
+	/* SCX struct_ops: check which kfunc groups are allowed for this op */
+	if (prog->type != BPF_PROG_TYPE_STRUCT_OPS)
+		return -EACCES;
+
+	moff = prog->aux->attach_st_ops_member_off;
+	flags = scx_ops_context_flags[SCX_MOFF_IDX(moff)];
+
+	if ((flags & SCX_OPS_KF_UNLOCKED) &&
+	    btf_id_set8_contains(&scx_kfunc_ids_unlocked, kfunc_id))
+		return 0;
+
+	if ((flags & SCX_OPS_KF_CPU_RELEASE) &&
+	    btf_id_set8_contains(&scx_kfunc_ids_cpu_release, kfunc_id))
+		return 0;
+
+	if ((flags & SCX_OPS_KF_DISPATCH) &&
+	    btf_id_set8_contains(&scx_kfunc_ids_dispatch, kfunc_id))
+		return 0;
+
+	if ((flags & SCX_OPS_KF_ENQUEUE) &&
+	    btf_id_set8_contains(&scx_kfunc_ids_enqueue_dispatch, kfunc_id))
+		return 0;
+
+	if ((flags & SCX_OPS_KF_SELECT_CPU) &&
+	    btf_id_set8_contains(&scx_kfunc_ids_select_cpu, kfunc_id))
+		return 0;
+
+	return -EACCES;
+}
+
+static const struct btf_kfunc_id_set scx_kfunc_set_ops_context_sensitive = {
+	.owner			= THIS_MODULE,
+	.set			= &scx_kfunc_ids_ops_context_sensitive,
+	.filter			= scx_kfunc_ids_ops_context_sensitive_filter,
+};
+
 BTF_KFUNCS_START(scx_kfunc_ids_any)
 BTF_ID_FLAGS(func, scx_bpf_task_set_slice, KF_IMPLICIT_ARGS | KF_RCU);
 BTF_ID_FLAGS(func, scx_bpf_task_set_dsq_vtime, KF_IMPLICIT_ARGS | KF_RCU);
@@ -9605,23 +9713,15 @@ static int __init scx_init(void)
 	 * kfunc registration can't be done from init_sched_ext_class() as
 	 * register_btf_kfunc_id_set() needs most of the system to be up.
 	 *
-	 * Some kfuncs are context-sensitive and can only be called from
-	 * specific SCX ops. They are grouped into BTF sets accordingly.
-	 * Unfortunately, BPF currently doesn't have a way of enforcing such
-	 * restrictions. Eventually, the verifier should be able to enforce
-	 * them. For now, register them the same and make each kfunc explicitly
-	 * check using scx_kf_allowed().
+	 * Context-sensitive kfuncs are grouped and filtered via
+	 * scx_kfunc_set_ops_context_sensitive, which uses the BPF verifier's
+	 * struct_ops context information to enforce per-op restrictions at
+	 * load time instead of via runtime checks.
 	 */
 	if ((ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
-					     &scx_kfunc_set_enqueue_dispatch)) ||
-	    (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
-					     &scx_kfunc_set_dispatch)) ||
-	    (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
-					     &scx_kfunc_set_cpu_release)) ||
-	    (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
-					     &scx_kfunc_set_unlocked)) ||
+					     &scx_kfunc_set_ops_context_sensitive)) ||
 	    (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL,
-					     &scx_kfunc_set_unlocked)) ||
+					     &scx_kfunc_set_ops_context_sensitive)) ||
 	    (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
 					     &scx_kfunc_set_any)) ||
 	    (ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
kernel/sched/ext_idle.c +10 -3
index ecf7e09b54ae..8f9f3d33c223 100644
@@ -1469,11 +1469,18 @@ BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu_node, KF_IMPLICIT_ARGS | KF_RCU)
 BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_IMPLICIT_ARGS | KF_RCU)
 BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu_node, KF_IMPLICIT_ARGS | KF_RCU)
 BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_IMPLICIT_ARGS | KF_RCU)
-BTF_ID_FLAGS(func, __scx_bpf_select_cpu_and, KF_IMPLICIT_ARGS | KF_RCU)
-BTF_ID_FLAGS(func, scx_bpf_select_cpu_and, KF_RCU)
-BTF_ID_FLAGS(func, scx_bpf_select_cpu_dfl, KF_IMPLICIT_ARGS | KF_RCU)
 BTF_KFUNCS_END(scx_kfunc_ids_idle)
 
+/*
+ * select_cpu kfuncs are context-sensitive: they can only be called from
+ * ops.select_cpu(), ops.enqueue(), or an unlocked context (e.g. BPF
+ * test_run() call). They are filtered by scx_kfunc_set_ops_context_sensitive
+ * in ext.c; this set is used only for grouping / lookup in the filter.
+ */
+BTF_KFUNCS_START(scx_kfunc_ids_select_cpu)
+SCX_KFUNCS_SELECT_CPU
+BTF_KFUNCS_END(scx_kfunc_ids_select_cpu)
+
 static const struct btf_kfunc_id_set scx_kfunc_set_idle = {
 	.owner			= THIS_MODULE,
 	.set			= &scx_kfunc_ids_idle,
kernel/sched/ext_internal.h +64 -0

70 lines not on this page · raw message

Signature · 1 line
2.48.1
#2 lore raw

PATCH v4 2/2 sched_ext: Remove runtime kfunc mask enforcement

In reply to Cheng-Yang Chou

Now that scx_kfunc_set_ops_context_sensitive enforces kfunc context at BPF load time, the per-task runtime enforcement via scx_kf_mask is redundant. Remove it.

Update scx_kf_allowed_if_unlocked() to check scx_locked_rq() and current->scx.kf_tasks[0] directly. The kf_tasks[0] check is needed because ops.select_cpu() can be called from try_to_wake_up() with p->pi_lock held, where rq is NULL but the context is not truly unlocked.

---
 include/linux/sched/ext.h   |  27 -----
 kernel/sched/ext.c          | 229 +++++++++---------------------------
 kernel/sched/ext_idle.c     |   7 +-
 kernel/sched/ext_internal.h |   9 +-
 4 files changed, 66 insertions(+), 206 deletions(-)
include/linux/sched/ext.h +0 -27
index 602dc83cab36..720f0423bc70 100644
@@ -147,32 +147,6 @@ enum scx_ent_dsq_flags {
 	SCX_TASK_DSQ_ON_PRIQ	= 1 << 0, /* task is queued on the priority queue of a dsq */
 };
 
-/*
- * Mask bits for scx_entity.kf_mask. Not all kfuncs can be called from
- * everywhere and the following bits track which kfunc sets are currently
- * allowed for %current. This simple per-task tracking works because SCX ops
- * nest in a limited way. BPF will likely implement a way to allow and disallow
- * kfuncs depending on the calling context which will replace this manual
- * mechanism. See scx_kf_allow().
- */
-enum scx_kf_mask {
-	SCX_KF_UNLOCKED		= 0,	  /* sleepable and not rq locked */
-	/* ENQUEUE and DISPATCH may be nested inside CPU_RELEASE */
-	SCX_KF_CPU_RELEASE	= 1 << 0, /* ops.cpu_release() */
-	/*
-	 * ops.dispatch() may release rq lock temporarily and thus ENQUEUE and
-	 * SELECT_CPU may be nested inside. ops.dequeue (in REST) may also be
-	 * nested inside DISPATCH.
-	 */
-	SCX_KF_DISPATCH		= 1 << 1, /* ops.dispatch() */
-	SCX_KF_ENQUEUE		= 1 << 2, /* ops.enqueue() and ops.select_cpu() */
-	SCX_KF_SELECT_CPU	= 1 << 3, /* ops.select_cpu() */
-	SCX_KF_REST		= 1 << 4, /* other rq-locked operations */
-
-	__SCX_KF_RQ_LOCKED	= SCX_KF_CPU_RELEASE | SCX_KF_DISPATCH |
-				  SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
-	__SCX_KF_TERMINAL	= SCX_KF_ENQUEUE | SCX_KF_SELECT_CPU | SCX_KF_REST,
-};
 
 enum scx_dsq_lnode_flags {
 	SCX_DSQ_LNODE_ITER_CURSOR = 1 << 0,
@@ -221,7 +195,6 @@ struct sched_ext_entity {
 	s32			sticky_cpu;
 	s32			holding_cpu;
 	s32			selected_cpu;
-	u32			kf_mask;	/* see scx_kf_mask above */
 	struct task_struct	*kf_tasks[2];	/* see SCX_CALL_OP_TASK() */
 
 	struct list_head	runnable_node;	/* rq->scx.runnable_list */
kernel/sched/ext.c +56 -173

598 lines not on this page · raw message

kernel/sched/ext_idle.c +2 -5
index 8f9f3d33c223..1f5d82a6bec3 100644
@@ -787,9 +787,8 @@ void __scx_update_idle(struct rq *rq, bool idle, bool do_notify)
 	 * either enqueue() sees the idle bit or update_idle() sees the task
 	 * that enqueue() queued.
 	 */
-	if (SCX_HAS_OP(sch, update_idle) && do_notify &&
-	    !scx_bypassing(sch, cpu_of(rq)))
-		SCX_CALL_OP(sch, SCX_KF_REST, update_idle, rq, cpu_of(rq), idle);
+	if (SCX_HAS_OP(sch, update_idle) && do_notify && !scx_bypassing(sch, cpu))
+		SCX_CALL_OP(sch, update_idle, rq, cpu_of(rq), idle);
 }
 
 static void reset_idle_masks(struct sched_ext_ops *ops)
@@ -933,8 +932,6 @@ static s32 select_cpu_from_kfunc(struct scx_sched *sch, struct task_struct *p,
 	if (scx_kf_allowed_if_unlocked()) {
 		rq = task_rq_lock(p, &rf);
 	} else {
-		if (!scx_kf_allowed(sch, SCX_KF_SELECT_CPU | SCX_KF_ENQUEUE))
-			return -EPERM;
 		rq = scx_locked_rq();
 	}
 
kernel/sched/ext_internal.h +8 -1
index 71c59bf1258b..bacab2c17efa 100644
@@ -1438,7 +1438,14 @@ static inline struct rq *scx_locked_rq(void)
 
 static inline bool scx_kf_allowed_if_unlocked(void)
 {
-	return !current->scx.kf_mask;
+	/*
+	 * We're in an unlocked context only if no rq is locked via SCX_CALL_OP
+	 * and we're not inside a task-based SCX op (which sets kf_tasks[0] but
+	 * may have rq==NULL, e.g. ops.select_cpu called from try_to_wake_up
+	 * where p->pi_lock is already held - calling task_rq_lock() there
+	 * would deadlock).
+	 */
+	return !scx_locked_rq() && !current->scx.kf_tasks[0];
 }
 
 static inline bool scx_bypassing(struct scx_sched *sch, s32 cpu)
Signature · 1 line
2.48.1
#3 lore raw
In reply to Cheng-Yang Chou
Quoted 4 lines · Cheng-Yang Chou

This patchset makes kfunc filters support the use of struct_ops context information to enforce SCX kfunc restrictions at BPF load time instead of at runtime.

Quoted 9 lines

With the improved kfunc filters, SCX no longer needs the mask-based runtime kfunc call restriction, so this patch series removes it.

Test plan:
- All sched_ext selftests passed.
- Tested all demo schedulers (except scx_flatcg) and multiple upstream SCX schedulers (scx_beerland, scx_bpfland, scx_cosmos, scx_flash, scx_lavd) under stress (stress-ng --cpu 0 --timeout 10s) with no crashes.

While testing this patchset with scx_flatcg, I encountered the following error. I then switched back to the tj/for-7.1 branch and tested again, but saw the same error.

Therefore, I believe this is not caused by this patchset.

For both tests (patchset and tj/for-7.1 branch), I have applied the patches Andrea mentioned before:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=175b45ed343a9c547b5f45293d3ea08d38a7b6f4
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=61bbcfb50514a8a94e035a7349697a3790ab4783
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7c405fb3279b39244b260b54f1bd6488689ae235

Log:

root@virtme-ng:/home/eric-wcnlab/dev/sched_ext/tools/sched_ext# stress-ng --cpu 0 --timeout 10s & sudo ./build/bin/scx_flatcg
[snip]
[   94.113441] sched_ext: BPF scheduler "flatcg" disabled (runnable task stall)
[   94.113783] sched_ext: flatcg: stress-ng[623] failed to run for 45.289s
[   94.114026]    scx_exit+0xa3/0xd0
[   94.114189]    scx_watchdog_workfn+0x4e9/0x9c0
[   94.114417]    process_scheduled_works+0x922/0x1930
[   94.114644]    worker_thread+0x4e4/0xd90
[   94.114822]    kthread+0x3d4/0x4a0
[   94.114979]    ret_from_fork+0x557/0x7b0
[   94.115135]    ret_from_fork_asm+0x1a/0x30

stress-ng: info: [623] dispatching hogs: 24 cpu

DEBUG DUMP
================================================================================

flatcg: root
kworker/u96:3[168] triggered exit kind 1026:
runnable task stall (stress-ng[623] failed to run for 45.289s)

Backtrace:
  scx_exit+0xa3/0xd0
  scx_watchdog_workfn+0x4e9/0x9c0
  process_scheduled_works+0x922/0x1930
  worker_thread+0x4e4/0xd90
  kthread+0x3d4/0x4a0
  ret_from_fork+0x557/0x7b0
  ret_from_fork_asm+0x1a/0x30

CPU states
----------

CPU 4 : nr_run=1 flags=0x1 cpu_rel=0 ops_qseq=11 ksync=456
curr=swapper/4[0] class=idle_sched_class

R stress-ng[623] *root -45289ms

      scx_state/flags=3/0x3 dsq_flags=0x1 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=0x1
      dsq_vtime=539683794 slice=20000000 weight=100
      cpus=ffffff no_mig=0

    asm_sysvec_apic_timer_interrupt+0x1a/0x20
    ftrace_likely_update+0x6c/0xc0
    __rcu_read_unlock+0xb5/0x5f0
    kernfs_root+0xf5/0x220
    kernfs_active+0x7d/0x130
    kernfs_dop_revalidate+0xa9/0x6c0
    lookup_fast+0x4e0/0x6e0
    link_path_walk+0x695/0x20e0
    path_openat+0x23d/0x2f30
    do_file_open+0x26e/0x3e0
    do_sys_openat2+0x131/0x3e0
    do_sys_open+0x9d/0x100
    __x64_sys_openat+0x9c/0x100
    do_syscall_64+0xfc/0x670
    entry_SYSCALL_64_after_hwframe+0x77/0x7f

CPU 5 : nr_run=1 flags=0x1 cpu_rel=0 ops_qseq=12 ksync=1510

          curr=kworker/u96:3[168] class=ext_sched_class

*R kworker/u96:3[168] *root +0ms

      scx_state/flags=3/0xd dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=(n/a)
      dsq_vtime=46302895 slice=20000000 weight=100
      cpus=ffffff no_mig=0

    scx_dump_state+0x12de/0x1b60
    scx_disable_irq_workfn+0xce/0x110
    irq_work_single+0x14c/0x250
    irq_work_run_list+0xa2/0xf0
    irq_work_run+0x4d/0xb0
    __sysvec_irq_work+0x4c/0x300
    sysvec_irq_work+0x8a/0xb0
    asm_sysvec_irq_work+0x1a/0x20
    scx_watchdog_workfn+0x55d/0x9c0
    process_scheduled_works+0x922/0x1930
    worker_thread+0x4e4/0xd90
    kthread+0x3d4/0x4a0
    ret_from_fork+0x557/0x7b0
    ret_from_fork_asm+0x1a/0x30

Event counters
--------------

              SCX_EV_SELECT_CPU_FALLBACK:                0
       SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE:                0
               SCX_EV_DISPATCH_KEEP_LAST:                1
                 SCX_EV_ENQ_SKIP_EXITING:                0
      SCX_EV_ENQ_SKIP_MIGRATION_DISABLED:                0
                      SCX_EV_REENQ_IMMED:                0
               SCX_EV_REENQ_LOCAL_REPEAT:                0
                 SCX_EV_REFILL_SLICE_DFL:                0
                  SCX_EV_BYPASS_DURATION:         34046683
                  SCX_EV_BYPASS_DISPATCH:                0
                  SCX_EV_BYPASS_ACTIVATE:                1
                 SCX_EV_INSERT_NOT_OWNED:                0
              SCX_EV_SUB_BYPASS_DISPATCH:                0

================================================================================

EXIT: runnable task stall (stress-ng[623] failed to run for 45.289s)

root@virtme-ng:/home/eric-wcnlab/dev/sched_ext/tools/sched_ext# stress-ng: info:  [623] skipped: 0
stress-ng: info:  [623] passed: 24: cpu (24)
stress-ng: info:  [623] failed: 0
stress-ng: info:  [623] metrics untrustworthy: 0
stress-ng: info:  [623] successful run completed in 10.47 secs
Signature · 2 lines
Thanks,
Cheng-Yang