sched_ext: Implement cgroup sub-scheduler support
Subsystems (7)
Tejun Heo
maintainerRFC PATCH 1/1 sched_ext: Implement cgroup sub-scheduler support
This patchset implements cgroup sub-scheduler support for sched_ext, enabling multiple schedulers to operate hierarchically within the cgroup tree. This capability supports multi-tenant server environments and other scenarios where systems must be partitioned to serve distinct workloads, each requiring specialized scheduling policies.
Traditional approaches rely on hard partitioning via cpuset, but this approach lacks the dynamism required by modern workloads. Users typically care less about specific CPU assignments and more about optimizations available on larger machines: opportunistic over-commit, improved latency for critical workloads while preserving bandwidth fairness, control mechanisms beyond simple CPU time (such as memory bandwidth isolation), and intelligent placement to optimize cache locality.
The cgroup sub-scheduler approach enables schedulers to attach anywhere in the cgroup hierarchy, with parent schedulers dynamically controlling CPU allocation to their children. This design provides BPF-driven flexibility while eliminating the constraints of hard partitioning.
This early-stage implementation demonstrates the fundamental building blocks for hierarchical scheduler operation. While the enqueue path and other components require further development, this patchset establishes the core mechanisms for nested scheduler operation and is developed enough to showcase all essential components.
The framework supports scheduling hierarchies up to SCX_SUB_MAX_DEPTH levels (currently set to 4). Enable and disable operations selectively bypass only tasks within the affected subtree, minimizing system-wide disruptions while providing reasonable isolation for child scheduler failures.
To see how this looks from the BPF scheduler perspective, examine the scx_qmap.bpf.c changes in the final patch, which demonstrate simple nested dispatch implementation.
Patch Organization:
Standalone fixes (01-07):
01 sched_ext: Use rhashtable_lookup() instead of rhashtable_lookup_fast() 02 sched_ext: Improve SCX_KF_DISPATCH comment
03 sched_ext: Fix stray scx_root usage in task_can_run_on()
04 sched_ext: Use bitfields for boolean warning flags
05 sched_ext: Add SCX_EFLAG_INITIALIZED to indicate successful initialization 06 sched_ext: Make qmap dump operation non-destructive
07 tools/sched_ext: scx_qmap: Make debug output quieter by default
Preparation patches (08-22):
08 sched_ext: Separate out scx_kick_cpu() and add sch to its name 09 sched_ext: Add the sch parameter to __bstr_format()
10 sched_ext: Add the sch parameter to ext_idle helpers
11 sched_ext: Drop kf_cpu_valid
12 sched_ext: Add the sch parameter to scx_dsq_insert_preamble() 13 sched_ext: Drop scx_kf_exit() and scx_kf_error()
14 sched_ext: Misc updates around scx_sched instance pointer handling 15 sched_ext: Keep dying tasks on a separate list
16 sched_ext: Implement cgroup subtree iteration for scx_task_iter 17 sched_ext: Add kargs to scx_fork()
18 sched/core: Swap the order between sched_post_fork() and wake_up_new_task() 19 cgroup: Expose some cgroup helpers
20 sched_ext: Update p->scx.disallow warning in scx_init_task() 21 sched_ext: Minor reorganization of enable/disable paths
22 sched_ext: Factor out scx_claim_exit() from scx_disable()
Core sub-scheduler implementation (23-40):
23 sched_ext: Introduce cgroup sub-sched support
24 HACK_NOT_FOR_UPSTREAM: BPF: Implement prog grouping hack
25 sched_ext: Introduce scx_task_sched()/_rcu()
26 sched_ext: Introduce scx_prog_sched()
27 sched_ext: Ignore insertions of not owned tasks into sub-sched DSQs 28 sched_ext: scx_dsq_move() should validate the task belongs to the dsq 29 sched_ext: Refactor task init/exit helpers
30 sched_ext: Make scx_prio_less() handle multiple schedulers
31 sched_ext: Move bypass_depth into scx_sched
32 sched_ext: Make bypass mode sub-sched aware
33 sched_ext: Factor out scx_dispatch_sched()
34 sched_ext: When calling ops.dispatch(), prev must be on the correct sched 35 sched_ext: Dispatch from all scx_sched instances
36 sched_ext: Move scx_dsp_ctx and scx_dsp_max_batch into scx_sched 37 sched_ext: Make watchdog sub-sched aware
38 sched_ext: Convert scx_dump_state() spinlock to raw spinlock
39 sched_ext: Support dumping multiple schedulers and add scheduler identification 40 sched_ext: Implement cgroup sub-sched enabling and disabling
Nested dispatch implementation (41-46):
41 HACK_NOT_FOR_UPSTREAM: sched_ext: Work around @aux__prog prototype mismatch 42 sched_ext: Wrap global DSQs in per-node structure
43 sched_ext: Add bypass DSQ for sub-schedulers
44 sched_ext: Factor out scx_link_sched() and scx_unlink_sched() 45 sched_ext: Add rhashtable lookup for sub-schedulers
46 sched_ext: Add basic building blocks for nested sub-scheduler dispatching
Implementation Notes:
- Patches 01-07: Independent fixes to be separated after merge window
- Patches 08-22: Infrastructure preparation (mostly sched_ext, one cgroup change)
- Patch 23: Skeletal sub-scheduler support (create/destroy only)
- Patches 24,41: Temporary BPF hacks requiring proper upstream solution
- Patches 25-39: Task migration and multi-scheduler operation mechanisms
- Patch 40: Full sub-scheduler enable/disable with ops.dispatch() support (enqueue path not yet implemented)
- Patches 42-46: Nested dispatch infrastructure and scx_bpf_dispatch_sched()
The patches are available in the git repository:git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-sub-sched
include/linux/bpf.h | 5 include/linux/cgroup-defs.h | 4 include/linux/cgroup.h | 65 include/linux/sched.h | 2 include/linux/sched/ext.h | 21 init/Kconfig | 4 kernel/bpf/syscall.c | 23 kernel/cgroup/cgroup-internal.h | 6 kernel/cgroup/cgroup.c | 55 kernel/exit.c | 1 kernel/fork.c | 6 kernel/sched/core.c | 2 kernel/sched/ext.c | 2362 ++++++++++++++++++++++++------- kernel/sched/ext.h | 4 kernel/sched/ext_idle.c | 197 ++ kernel/sched/ext_internal.h | 223 ++ kernel/sched/sched.h | 7 tools/sched_ext/include/scx/common.bpf.h | 90 - tools/sched_ext/include/scx/compat.bpf.h | 7 tools/sched_ext/scx_qmap.bpf.c | 146 + tools/sched_ext/scx_qmap.c | 36 21 files changed, 2556 insertions(+), 710 deletions(-)
Tejun Heo
maintainerPATCH 1/46 sched_ext: Use rhashtable_lookup() instead of rhashtable_lookup_fast()
The find_user_dsq() function is called from contexts that are already under RCU read lock protection. Switch from rhashtable_lookup_fast() to rhashtable_lookup() to avoid redundant RCU locking.
--- kernel/sched/ext.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/ext.c +1 -1
@@ -207,7 +207,7 @@ static struct scx_dispatch_q *find_global_dsq(struct task_struct *p)static struct scx_dispatch_q *find_user_dsq (struct scx_sched *sch,u64 dsq_id) { -return rhashtable_lookup_fast (&sch->dsq_hash, &dsq_id, dsq_hash_params); +return rhashtable_lookup (&sch->dsq_hash, &dsq_id, dsq_hash_params); }/*
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 2/46 sched_ext: Improve SCX_KF_DISPATCH comment
In reply to Tejun HeoThe comment for SCX_KF_DISPATCH was incomplete and didn't explain that ops.dispatch() may temporarily release the rq lock, allowing ENQUEUE and SELECT_CPU operations to be nested inside DISPATCH contexts.
Update the comment to clarify this nesting behavior and provide better context for when these operations can occur within dispatch.
--- include/linux/sched/ext.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
include/linux/sched/ext.h +5 -1
@@ -108,7 +108,11 @@ enum scx_kf_mask {SCX_KF_UNLOCKED =0 ,/* sleepable and not rq locked */ /* ENQUEUE and DISPATCH may be nested inside CPU_RELEASE */ SCX_KF_CPU_RELEASE =1 <<0 ,/* ops.cpu_release() */ -/* ops.dequeue (in REST) may be nested inside DISPATCH */ +/* +* ops.dispatch() may release rq lock temporarily and thus ENQUEUE and +* SELECT_CPU may be nested inside. ops.dequeue (in REST) may also be +* nested inside DISPATCH. +*/ SCX_KF_DISPATCH =1 <<1 ,/* ops.dispatch() */ SCX_KF_ENQUEUE =1 <<2 ,/* ops.enqueue() and ops.select_cpu() */ SCX_KF_SELECT_CPU = 1 << 3, /* ops.select_cpu() */
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 3/46 sched_ext: Fix stray scx_root usage in task_can_run_on_remote_rq()
In reply to Tejun Heotask_can_run_on_remote_rq() takes @sch but it is using scx_root when incrementing SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE, which is inconsistent and gets in the way of implementing multiple scheduler support. Use @sch instead. As currently scx_root is the only possible scheduler instance, this doesn't cause any behavior changes.
--- kernel/sched/ext.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/sched/ext.c +1 -2
@@ -1622,8 +1622,7 @@ static bool task_can_run_on_remote_rq(struct scx_sched *sch,if (!scx_rq_online (rq)) {if (enforce) -__scx_add_event (scx_root, -SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE ,1 ); +__scx_add_event (sch,SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE ,1 );return false; }
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 4/46 sched_ext: Use bitfields for boolean warning flags
In reply to Tejun HeoConvert warned_zero_slice and warned_deprecated_rq in scx_sched struct to single-bit bitfields to reduce struct size.
--- kernel/sched/ext_internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/ext_internal.h +2 -2
@@ -871,8 +871,8 @@ struct scx_sched {struct scx_dispatch_q **global_dsqs;struct scx_sched_pcpu __percpu *pcpu; -bool warned_zero_slice; -bool warned_deprecated_rq; + bool warned_zero_slice:1; + bool warned_deprecated_rq:1;atomic_t exit_kind;struct scx_exit_info *exit_info;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 5/46 sched_ext: Add SCX_EFLAG_INITIALIZED to indicate successful ops.init()
In reply to Tejun Heoops.exit() may be called even if the loading failed before ops.init() finishes successfully. This is because ops.exit() allows rich exit info communication. Add SCX_EFLAG_INITIALIZED flag to scx_exit_info.flags to indicate whether ops.init() finished successfully.
This enables BPF schedulers to distinguish between exit scenarios and handle cleanup appropriately based on initialization state.
--- kernel/sched/ext.c | 1 + kernel/sched/ext_internal.h | 13 +++++++++++++ 2 files changed, 14 insertions(+)
kernel/sched/ext.c +1 -0
@@ -4554,6 +4554,7 @@ static int scx_enable(struct sched_ext_ops *ops, struct bpf_link *link)scx_error (sch,"ops.init() failed (%d)" , ret); goto err_disable; } + sch->exit_info->flags |=SCX_EFLAG_INITIALIZED ; }for (i =SCX_OPI_CPU_HOTPLUG_BEGIN ; i <SCX_OPI_CPU_HOTPLUG_END ; i++)
kernel/sched/ext_internal.h +13 -0
@@ -62,6 +62,16 @@ enum scx_exit_code { SCX_ECODE_ACT_RESTART = 1LLU << 48, }; +enum scx_exit_flags { +/* +* ops.exit() may be called even if the loading failed before ops.init() +* finishes successfully. This is because ops.exit() allows rich exit +* info communication. The following flag indicates whether ops.init() +* finished successfully. +*/ +SCX_EFLAG_INITIALIZED , +}; +/* * scx_exit_info is passed to ops.exit() to describe why the BPF scheduler is * being disabled.
@@ -73,6 +83,9 @@ struct scx_exit_info {/* exit code if gracefully exiting */ s64 exit_code; +/* %SCX_EFLAG_* */ +u64 flags; +/* textual representation of the above */ const char *reason;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 6/46 sched_ext: Make qmap dump operation non-destructive
In reply to Tejun HeoThe qmap dump operation was destructively consuming queue entries while displaying them. As dump can be triggered anytime, this can easily lead to stalls. Add a temporary dump_store queue and modify the dump logic to pop entries, display them, and then restore them back to the original queue. This allows dump operations to be performed without affecting the scheduler's queue state.
Note that if racing against new enqueues during dump, ordering can get mixed up, but this is acceptable for debugging purposes.
--- tools/sched_ext/scx_qmap.bpf.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
tools/sched_ext/scx_qmap.bpf.c +17 -1
@@ -56,7 +56,8 @@ struct qmap { queue1 SEC(".maps"), queue2 SEC(".maps"), queue3 SEC(".maps"), - queue4 SEC(".maps"); + queue4 SEC(".maps"), + dump_store SEC(".maps");struct { __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
@@ -578,11 +579,26 @@ void BPF_STRUCT_OPS(qmap_dump, struct scx_dump_ctx *dctx)return ;scx_bpf_dump ("QMAP FIFO[%d]:" , i); + +/* +* Dump can be invoked anytime and there is no way to iterate in +* a non-destructive way. Pop and store in dump_store and then +* restore afterwards. If racing against new enqueues, ordering +* can get mixed up. +*/ bpf_repeat(4096) {if (bpf_map_pop_elem (fifo, &pid))break ; +bpf_map_push_elem (&dump_store, &pid,0 );scx_bpf_dump (" %d" , pid); } + + bpf_repeat(4096) { +if (bpf_map_pop_elem (&dump_store, &pid)) +break ; +bpf_map_push_elem (fifo, &pid,0 ); + } +scx_bpf_dump ("\n" ); } }
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 7/46 tools/sched_ext: scx_qmap: Make debug output quieter by default
In reply to Tejun Heoscx_qmap currently outputs verbose debug messages including cgroup operations and CPU online/offline events by default, which can be noisy during normal operation. While the existing -P option controls DSQ dumps and event statistics, there's no way to suppress the other debug messages.
Split the debug output controls to make scx_qmap quieter by default. The -P option continues to control DSQ dumps and event statistics (print_dsqs_and_events), while a new -M option controls debug messages like cgroup operations and CPU events (print_msgs). This allows users to run scx_qmap with minimal output and selectively enable debug information as needed.
--- tools/sched_ext/scx_qmap.bpf.c | 80 +++++++++++++++++++--------------- tools/sched_ext/scx_qmap.c | 12 +++-- 2 files changed, 53 insertions(+), 39 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +45 -35
@@ -39,7 +39,8 @@ const volatile u32 stall_kernel_nth;const volatile u32 dsp_inf_loop_after;const volatile u32 dsp_batch;const volatile bool highpri_boosting; -const volatile bool print_shared_dsq; +const volatile bool print_dsqs_and_events; +const volatile bool print_msgs;const volatile s32 disallow_tgid;const volatile bool suppress_dump;
@@ -633,22 +634,25 @@ void BPF_STRUCT_OPS(qmap_dump_task, struct scx_dump_ctx *dctx, struct task_strucs32 BPF_STRUCT_OPS (qmap_cgroup_init ,struct cgroup *cgrp,struct scx_cgroup_init_args *args) { -bpf_printk ("CGRP INIT %llu weight=%u period=%lu quota=%ld burst=%lu" , - cgrp->kn->id, args->weight, args->bw_period_us, - args->bw_quota_us, args->bw_burst_us); +if (print_msgs) +bpf_printk ("CGRP INIT %llu weight=%u period=%lu quota=%ld burst=%lu" , + cgrp->kn->id, args->weight, args->bw_period_us, + args->bw_quota_us, args->bw_burst_us);return 0 ; }void BPF_STRUCT_OPS (qmap_cgroup_set_weight ,struct cgroup *cgrp,u32 weight) { -bpf_printk ("CGRP SET %llu weight=%u" , cgrp->kn->id, weight); +if (print_msgs) +bpf_printk ("CGRP SET %llu weight=%u" , cgrp->kn->id, weight); }void BPF_STRUCT_OPS (qmap_cgroup_set_bandwidth ,struct cgroup *cgrp,u64 period_us,u64 quota_us,u64 burst_us) { -bpf_printk ("CGRP SET %llu period=%lu quota=%ld burst=%lu" , cgrp->kn->id, - period_us, quota_us, burst_us); +if (print_msgs) +bpf_printk ("CGRP SET %llu period=%lu quota=%ld burst=%lu" , + cgrp->kn->id, period_us, quota_us, burst_us); }/*
@@ -692,16 +696,20 @@ static void print_cpus(void)void BPF_STRUCT_OPS (qmap_cpu_online ,s32 cpu) { -bpf_printk ("CPU %d coming online" , cpu); -/* @cpu is already online at this point */ -print_cpus (); +if (print_msgs) { +bpf_printk ("CPU %d coming online" , cpu); +/* @cpu is already online at this point */ +print_cpus (); + } }void BPF_STRUCT_OPS (qmap_cpu_offline ,s32 cpu) { -bpf_printk ("CPU %d going offline" , cpu); -/* @cpu is still online at this point */ -print_cpus (); +if (print_msgs) { +bpf_printk ("CPU %d going offline" , cpu); +/* @cpu is still online at this point */ +print_cpus (); + } }struct monitor_timer {
@@ -799,35 +807,36 @@ static void dump_shared_dsq(void)static int monitor_timerfn (void *map,int *key,struct bpf_timer *timer) { -struct scx_event_stats events; -bpf_rcu_read_lock ();dispatch_highpri (true);bpf_rcu_read_unlock ();monitor_cpuperf (); -if (print_shared_dsq) +if (print_dsqs_and_events) { +struct scx_event_stats events; +dump_shared_dsq (); -__COMPAT_scx_bpf_events (&events,sizeof (events)); - -bpf_printk ("%35s: %lld" ,"SCX_EV_SELECT_CPU_FALLBACK" , -scx_read_event (&events,SCX_EV_SELECT_CPU_FALLBACK )); -bpf_printk ("%35s: %lld" ,"SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE" , -scx_read_event (&events,SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE )); -bpf_printk ("%35s: %lld" ,"SCX_EV_DISPATCH_KEEP_LAST" , -scx_read_event (&events,SCX_EV_DISPATCH_KEEP_LAST )); -bpf_printk ("%35s: %lld" ,"SCX_EV_ENQ_SKIP_EXITING" , -scx_read_event (&events,SCX_EV_ENQ_SKIP_EXITING )); -bpf_printk ("%35s: %lld" ,"SCX_EV_REFILL_SLICE_DFL" , -scx_read_event (&events,SCX_EV_REFILL_SLICE_DFL )); -bpf_printk ("%35s: %lld" ,"SCX_EV_BYPASS_DURATION" , -scx_read_event (&events,SCX_EV_BYPASS_DURATION )); -bpf_printk ("%35s: %lld" ,"SCX_EV_BYPASS_DISPATCH" , -scx_read_event (&events,SCX_EV_BYPASS_DISPATCH )); -bpf_printk ("%35s: %lld" ,"SCX_EV_BYPASS_ACTIVATE" , -scx_read_event (&events,SCX_EV_BYPASS_ACTIVATE )); +__COMPAT_scx_bpf_events (&events,sizeof (events)); + +bpf_printk ("%35s: %lld" ,"SCX_EV_SELECT_CPU_FALLBACK" , +scx_read_event (&events,SCX_EV_SELECT_CPU_FALLBACK )); +bpf_printk ("%35s: %lld" ,"SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE" , +scx_read_event (&events,SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE )); +bpf_printk ("%35s: %lld" ,"SCX_EV_DISPATCH_KEEP_LAST" , +scx_read_event (&events,SCX_EV_DISPATCH_KEEP_LAST )); +bpf_printk ("%35s: %lld" ,"SCX_EV_ENQ_SKIP_EXITING" , +scx_read_event (&events,SCX_EV_ENQ_SKIP_EXITING )); +bpf_printk ("%35s: %lld" ,"SCX_EV_REFILL_SLICE_DFL" , +scx_read_event (&events,SCX_EV_REFILL_SLICE_DFL )); +bpf_printk ("%35s: %lld" ,"SCX_EV_BYPASS_DURATION" , +scx_read_event (&events,SCX_EV_BYPASS_DURATION )); +bpf_printk ("%35s: %lld" ,"SCX_EV_BYPASS_DISPATCH" , +scx_read_event (&events,SCX_EV_BYPASS_DISPATCH )); +bpf_printk ("%35s: %lld" ,"SCX_EV_BYPASS_ACTIVATE" , +scx_read_event (&events,SCX_EV_BYPASS_ACTIVATE )); + }bpf_timer_start (timer,ONE_SEC_IN_NS ,0 );return 0 ;
@@ -839,7 +848,8 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)struct bpf_timer *timer;s32 ret; -print_cpus (); +if (print_msgs) +print_cpus (); ret =scx_bpf_create_dsq (SHARED_DSQ ,-1 );if (ret)
tools/sched_ext/scx_qmap.c +8 -4
@@ -20,7 +20,7 @@ const char help_fmt[] ="See the top-level comment in .bpf.c for more details.\n" "\n" "Usage: %s [-s SLICE_US] [-e COUNT] [-t COUNT] [-T COUNT] [-l COUNT] [-b COUNT]\n" -" [-P] [-d PID] [-D LEN] [-p] [-v]\n" +" [-P] [-M] [-d PID] [-D LEN] [-p] [-v]\n" "\n" " -s SLICE_US Override slice duration\n" " -e COUNT Trigger scx_bpf_error() after COUNT enqueues\n"
@@ -28,7 +28,8 @@ const char help_fmt[] =" -T COUNT Stall every COUNT'th kernel thread\n" " -l COUNT Trigger dispatch infinite looping after COUNT dispatches\n" " -b COUNT Dispatch upto COUNT tasks together\n" -" -P Print out DSQ content to trace_pipe every second, use with -b\n" +" -P Print out DSQ content and event counters to trace_pipe every second\n" +" -M Print out debug messages to trace_pipe\n" " -H Boost nice -20 tasks in SHARED_DSQ, use with -b\n" " -d PID Disallow a process from switching into SCHED_EXT (-1 for self)\n" " -D LEN Set scx_exit_info.dump buffer length\n"
@@ -66,7 +67,7 @@ int main(int argc, char **argv) skel->rodata->slice_ns =__COMPAT_ENUM_OR_ZERO ("scx_public_consts" ,"SCX_SLICE_DFL" ); -while ((opt =getopt (argc, argv,"s:e:t:T:l:b:PHd:D:Spvh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:l:b:PMHd:D:Spvh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -87,7 +88,10 @@ int main(int argc, char **argv) skel->rodata->dsp_batch =strtoul (optarg,NULL ,0 );break ;case 'P' : - skel->rodata->print_shared_dsq = true; + skel->rodata->print_dsqs_and_events = true; +break ; +case 'M' : + skel->rodata->print_msgs = true;break ;case 'H' : skel->rodata->highpri_boosting = true;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 8/46 sched_ext: Separate out scx_kick_cpu() and add @sch to it
In reply to Tejun HeoIn preparation for multiple scheduler support, separate out scx_kick_cpu() from scx_bpf_kick_cpu() and add the @sch parameter to it. scx_bpf_kick_cpu() now acquires an RCU read lock, reads $scx_root, and calls scx_kick_cpu() with it if non-NULL. The passed in @sch parameter is not used yet.
Internal uses of scx_bpf_kick_cpu() are converted to scx_kick_cpu(). Where $sch is available, it's used. In the pick_task_scx() path where no associated scheduler can be identified, $scx_root is used directly. Note that $scx_root cannot be NULL in this case.
--- kernel/sched/ext.c | 43 +++++++++++++++++++++++++++---------------- 1 file changed, 27 insertions(+), 16 deletions(-)
kernel/sched/ext.c +27 -16
@@ -136,7 +136,7 @@ static struct kset *scx_kset;#include <trace/events/sched_ext.h> static void process_ddsp_deferred_locals (struct rq *rq); -static void scx_bpf_kick_cpu (s32 cpu,u64 flags); +static void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags);static void scx_vexit (struct scx_sched *sch,enum scx_exit_kind kind,s64 exit_code,const char *fmt,va_list args);
@@ -2125,10 +2125,10 @@ static int balance_one(struct rq *rq, struct task_struct *prev)* balance(), we want to complete this scheduling cycle and then * start a new one. IOW, we want to call resched_curr() on the * next, most likely idle, task, not the current one. Use -* scx_bpf_kick_cpu() for deferred kicking. +* scx_kick_cpu() for deferred kicking. */ if (unlikely (!--nr_loops)) { -scx_bpf_kick_cpu (cpu_of (rq),0 ); +scx_kick_cpu (sch,cpu_of (rq),0 );break ; } }while (dspc->nr_tasks);
@@ -2417,7 +2417,8 @@ static struct task_struct *pick_task_scx(struct rq *rq) p =first_local_task (rq);if (!p) {if (kick_idle) -scx_bpf_kick_cpu (cpu_of (rq),SCX_KICK_IDLE ); +scx_kick_cpu (rcu_dereference_sched (scx_root), +cpu_of (rq),SCX_KICK_IDLE );return NULL ; }
@@ -3721,7 +3722,7 @@ static void scx_clear_softlockup(void)* * - pick_next_task() suppresses zero slice warning. * -* - scx_bpf_kick_cpu() is disabled to avoid irq_work malfunction during PM +* - scx_kick_cpu() is disabled to avoid irq_work malfunction during PM * operations. * * - scx_prio_less() reverts to the default core_sched_at order.
@@ -5809,17 +5810,7 @@ static const struct btf_kfunc_id_set scx_kfunc_set_unlocked = {__bpf_kfunc_start_defs (); -/** -* scx_bpf_kick_cpu - Trigger reschedule on a CPU -* @cpu: cpu to kick -* @flags: %SCX_KICK_* flags -* -* Kick @cpu into rescheduling. This can be used to wake up an idle CPU or -* trigger rescheduling on a busy CPU. This can be called from any online -* scx_ops operation and the actual kicking is performed asynchronously through -* an irq work. -*/ -__bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags) +static void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags) {struct rq *this_rq;unsigned long irq_flags;
@@ -5872,6 +5863,26 @@ __bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags)local_irq_restore (irq_flags); } +/** +* scx_bpf_kick_cpu - Trigger reschedule on a CPU +* @cpu: cpu to kick +* @flags: %SCX_KICK_* flags +* +* Kick @cpu into rescheduling. This can be used to wake up an idle CPU or +* trigger rescheduling on a busy CPU. This can be called from any online +* scx_ops operation and the actual kicking is performed asynchronously through +* an irq work. +*/ +__bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags) +{ +struct scx_sched *sch; + +guard (rcu)(); + sch =rcu_dereference (scx_root); +if (likely (sch)) +scx_kick_cpu (sch, cpu, flags); +} +/** * scx_bpf_dsq_nr_queued - Return the number of queued tasks * @dsq_id: id of the DSQ
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 9/46 sched_ext: Add the @sch parameter to __bstr_format()
In reply to Tejun HeoIn preparation for multiple scheduler support, add the @sch parameter to __bstr_format() and update the callers to read $scx_root, verify that it's not NULL and pass it in. The passed in @sch parameter is not used yet.
--- kernel/sched/ext.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-)
kernel/sched/ext.c +21 -7
@@ -6063,8 +6063,9 @@ __bpf_kfunc void bpf_iter_scx_dsq_destroy(struct bpf_iter_scx_dsq *it)__bpf_kfunc_end_defs (); -static s32 __bstr_format (u64 *data_buf,char *line_buf,size_t line_size, -char *fmt,unsigned long long *data,u32 data__sz) +static s32 __bstr_format (struct scx_sched *sch,u64 *data_buf,char *line_buf, +size_t line_size,char *fmt,unsigned long long *data, +u32 data__sz) {struct bpf_bprintf_data bprintf_data = { .get_bin_args = true };s32 ret;
@@ -6099,10 +6100,10 @@ static s32 __bstr_format(u64 *data_buf, char *line_buf, size_t line_size,return ret; } -static s32 bstr_format (struct scx_bstr_buf *buf, +static s32 bstr_format (struct scx_sched *sch,struct scx_bstr_buf *buf,char *fmt,unsigned long long *data,u32 data__sz) { -return __bstr_format (buf->data, buf->line,sizeof (buf->line), +return __bstr_format (sch, buf->data, buf->line,sizeof (buf->line), fmt, data, data__sz); }
@@ -6121,10 +6122,13 @@ __bpf_kfunc_start_defs(); __bpf_kfunc void scx_bpf_exit_bstr(s64 exit_code, char *fmt,unsigned long long *data,u32 data__sz) { +struct scx_sched *sch;unsigned long flags;raw_spin_lock_irqsave (&scx_exit_bstr_buf_lock, flags); -if (bstr_format (&scx_exit_bstr_buf, fmt, data, data__sz) >=0 ) + sch =rcu_dereference_bh (scx_root); +if (likely (sch) && +bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 )scx_kf_exit (SCX_EXIT_UNREG_BPF , exit_code,"%s" , scx_exit_bstr_buf.line);raw_spin_unlock_irqrestore (&scx_exit_bstr_buf_lock, flags); }
@@ -6141,10 +6145,13 @@ __bpf_kfunc void scx_bpf_exit_bstr(s64 exit_code, char *fmt, __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data,u32 data__sz) { +struct scx_sched *sch;unsigned long flags;raw_spin_lock_irqsave (&scx_exit_bstr_buf_lock, flags); -if (bstr_format (&scx_exit_bstr_buf, fmt, data, data__sz) >=0 ) + sch =rcu_dereference_bh (scx_root); +if (likely (sch) && +bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 )scx_kf_exit (SCX_EXIT_ERROR_BPF ,0 ,"%s" , scx_exit_bstr_buf.line);raw_spin_unlock_irqrestore (&scx_exit_bstr_buf_lock, flags); }
@@ -6164,17 +6171,24 @@ __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data, __bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data,u32 data__sz) { +struct scx_sched *sch;struct scx_dump_data *dd = &scx_dump_data;struct scx_bstr_buf *buf = &dd->buf;s32 ret; +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return ; +if (raw_smp_processor_id () != dd->cpu) {scx_kf_error ("scx_bpf_dump() must only be called from ops.dump() and friends" );return ; }/* append the formatted string to the line buf */ - ret =__bstr_format (buf->data, buf->line + dd->cursor, + ret =__bstr_format (sch, buf->data, buf->line + dd->cursor,sizeof (buf->line) - dd->cursor, fmt, data, data__sz);if (ret <0 ) { dump_line(dd->s, "%s[!] (\"%s\", %p, %u) failed to format (%d)",
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 10/46 sched_ext: Add the @sch parameter to ext_idle helpers
In reply to Tejun HeoIn preparation for multiple scheduler support, add the @sch parameter to validate_node(), check_builtin_idle_enabled() and select_cpu_from_kfunc(), and update their callers to read $scx_root, verify that it's not NULL and pass it in. The passed in @sch parameter is not used yet.
--- kernel/sched/ext_idle.c | 109 ++++++++++++++++++++++++++++++++++------ 1 file changed, 94 insertions(+), 15 deletions(-)
kernel/sched/ext_idle.c +94 -15
@@ -819,7 +819,7 @@ void scx_idle_disable(void)* Helpers that can be called from the BPF scheduler. */ -static int validate_node (int node) +static int validate_node (struct scx_sched *sch,int node) {if (!static_branch_likely (&scx_builtin_idle_per_node)) {scx_kf_error ("per-node idle tracking is disabled" );
@@ -847,7 +847,7 @@ static int validate_node(int node)__bpf_kfunc_start_defs (); -static bool check_builtin_idle_enabled (void ) +static bool check_builtin_idle_enabled (struct scx_sched *sch) {if (static_branch_likely (&scx_builtin_idle_enabled))return true;
@@ -856,7 +856,8 @@ static bool check_builtin_idle_enabled(void)return false; } -static s32 select_cpu_from_kfunc (struct task_struct *p,s32 prev_cpu,u64 wake_flags, +static s32 select_cpu_from_kfunc (struct scx_sched *sch,struct task_struct *p, +s32 prev_cpu,u64 wake_flags,const struct cpumask *allowed,u64 flags) {struct rq *rq;
@@ -866,7 +867,7 @@ static s32 select_cpu_from_kfunc(struct task_struct *p, s32 prev_cpu, u64 wake_fif (!kf_cpu_valid (prev_cpu,NULL ))return -EINVAL ; -if (!check_builtin_idle_enabled ()) +if (!check_builtin_idle_enabled (sch))return -EBUSY ;/*
@@ -946,15 +947,21 @@ __bpf_kfunc int scx_bpf_cpu_node(s32 cpu) __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu,u64 wake_flags,bool *is_idle) { +struct scx_sched *sch;s32 cpu; - cpu =select_cpu_from_kfunc (p, prev_cpu, wake_flags,NULL ,0 ); +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return -ENODEV ; + + cpu =select_cpu_from_kfunc (sch, p, prev_cpu, wake_flags,NULL ,0 );if (cpu >=0 ) { *is_idle = true;return cpu; } *is_idle = false; -return prev_cpu; }
@@ -981,7 +988,16 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, __bpf_kfunc s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64 wake_flags,const struct cpumask *cpus_allowed,u64 flags) { -return select_cpu_from_kfunc (p, prev_cpu, wake_flags, cpus_allowed, flags); +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return -ENODEV ; + +return select_cpu_from_kfunc (sch, p, prev_cpu, wake_flags, + cpus_allowed, flags); }/**
@@ -995,7 +1011,15 @@ __bpf_kfunc s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64*/ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) { - node =validate_node (node); +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return cpu_none_mask; + + node =validate_node (sch, node);if (node <0 )return cpu_none_mask;
@@ -1011,12 +1035,20 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask_node(int node)*/ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) { +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return cpu_none_mask; +if (static_branch_unlikely (&scx_builtin_idle_per_node)) {scx_kf_error ("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled" );return cpu_none_mask; } -if (!check_builtin_idle_enabled ()) +if (!check_builtin_idle_enabled (sch))return cpu_none_mask;return idle_cpumask (NUMA_NO_NODE )->cpu;
@@ -1034,7 +1066,15 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void)*/ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) { - node =validate_node (node); +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return cpu_none_mask; + + node =validate_node (sch, node);if (node <0 )return cpu_none_mask;
@@ -1054,12 +1094,20 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask_node(int node)*/ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void) { +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return cpu_none_mask; +if (static_branch_unlikely (&scx_builtin_idle_per_node)) {scx_kf_error ("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled" );return cpu_none_mask; } -if (!check_builtin_idle_enabled ()) +if (!check_builtin_idle_enabled (sch))return cpu_none_mask;if (sched_smt_active ())
@@ -1095,7 +1143,15 @@ __bpf_kfunc void scx_bpf_put_idle_cpumask(const struct cpumask *idle_mask)*/ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) { -if (!check_builtin_idle_enabled ()) +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return false; + +if (!check_builtin_idle_enabled (sch))return false;if (!kf_cpu_valid (cpu,NULL ))
@@ -1126,7 +1182,15 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) __bpf_kfunc s32 scx_bpf_pick_idle_cpu_node(const struct cpumask *cpus_allowed,int node,u64 flags) { - node =validate_node (node); +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return -ENODEV ; + + node =validate_node (sch, node);if (node <0 )return node;
@@ -1158,12 +1222,20 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu_node(const struct cpumask *cpus_allowed, __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed,u64 flags) { +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return -ENODEV ; +if (static_branch_maybe (CONFIG_NUMA , &scx_builtin_idle_per_node)) {scx_kf_error ("per-node idle tracking is enabled" );return -EBUSY ; } -if (!check_builtin_idle_enabled ()) +if (!check_builtin_idle_enabled (sch))return -EBUSY ;return scx_pick_idle_cpu (cpus_allowed,NUMA_NO_NODE , flags);
@@ -1193,9 +1265,16 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed, __bpf_kfunc s32 scx_bpf_pick_any_cpu_node(const struct cpumask *cpus_allowed,int node,u64 flags) { +struct scx_sched *sch;s32 cpu; - node =validate_node (node); +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return -ENODEV ; + + node =validate_node (sch, node);if (node <0 )return node;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 11/46 sched_ext: Drop kf_cpu_valid()
In reply to Tejun HeoThe intention behind kf_cpu_valid() was that when called from kfuncs, kf_cpu_valid() would be able to implicitly determine the scx_sched instance being operated on and thus wouldn't need @sch passed in explicitly. This turned out to be unnecessarily complicated to implement and not have justifiable practical benefits. Replace kf_cpu_valid() usages with ops_cpu_valid() which takes explicit @sch.
Callers which don't have $sch available in the context are updated to read $scx_root under RCU read lock, verify that it's not NULL and pass it in.
scx_bpf_cpu_rq() is restructured to use guard(rcu)() instead of explicit rcu_read_[un]lock().
--- kernel/sched/ext.c | 67 ++++++++++++++++++++++++----------------- kernel/sched/ext_idle.c | 12 +++++--- 2 files changed, 48 insertions(+), 31 deletions(-)
kernel/sched/ext.c +40 -27
@@ -736,23 +736,6 @@ static bool ops_cpu_valid(struct scx_sched *sch, s32 cpu, const char *where) } } -/** -* kf_cpu_valid - Verify a CPU number, to be used on kfunc input args -* @cpu: cpu number which came from a BPF ops -* @where: extra information reported on error -* -* The same as ops_cpu_valid() but @sch is implicit. -*/ -static bool kf_cpu_valid (u32 cpu,const char *where) -{ -if (__cpu_valid (cpu)) { -return true; - }else { -scx_kf_error ("invalid CPU %d%s%s" , cpu, where ?" " :"" , where ?:"" ); -return false; - } -} -/** * ops_sanitize_err - Sanitize a -errno value * @sch: scx_sched to error out on error
@@ -5815,7 +5798,7 @@ static void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags)struct rq *this_rq;unsigned long irq_flags; -if (!kf_cpu_valid (cpu,NULL )) +if (!ops_cpu_valid (sch, cpu,NULL ))return ;local_irq_save (irq_flags);
@@ -6224,7 +6207,12 @@ __bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data,*/ __bpf_kfunc u32 scx_bpf_cpuperf_cap(s32 cpu) { -if (kf_cpu_valid (cpu,NULL )) +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (likely (sch) &&ops_cpu_valid (sch, cpu,NULL ))return arch_scale_cpu_capacity (cpu);else return SCX_CPUPERF_ONE ;
@@ -6246,7 +6234,12 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cap(s32 cpu)*/ __bpf_kfunc u32 scx_bpf_cpuperf_cur(s32 cpu) { -if (kf_cpu_valid (cpu,NULL )) +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (likely (sch) &&ops_cpu_valid (sch, cpu,NULL ))return arch_scale_freq_capacity (cpu);else return SCX_CPUPERF_ONE ;
@@ -6268,12 +6261,20 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cur(s32 cpu)*/ __bpf_kfunc void scx_bpf_cpuperf_set(s32 cpu, u32 perf) { +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (sch); +if (unlikely (!sch)) +return ; +if (unlikely (perf >SCX_CPUPERF_ONE )) {scx_kf_error ("Invalid cpuperf target %u for CPU %d" , perf, cpu);return ; } -if (kf_cpu_valid (cpu,NULL )) { +if (ops_cpu_valid (sch, cpu,NULL )) {struct rq *rq =cpu_rq (cpu), *locked_rq =scx_locked_rq ();struct rq_flags rf;
@@ -6379,18 +6380,21 @@ __bpf_kfunc struct rq *scx_bpf_cpu_rq(s32 cpu) {struct scx_sched *sch; -if (!kf_cpu_valid (cpu,NULL )) -return NULL ; +guard (rcu)(); -rcu_read_lock (); sch =rcu_dereference (scx_root); -if (likely (sch) && !sch->warned_deprecated_rq) { +if (unlikely (!sch)) +return NULL ; + +if (!ops_cpu_valid (sch, cpu,NULL )) +return NULL ; + +if (!sch->warned_deprecated_rq) {printk_deferred (KERN_WARNING "sched_ext: %s() is deprecated; " "use scx_bpf_locked_rq() when holding rq lock " "or scx_bpf_cpu_curr() to read remote curr safely.\n" , __func__); sch->warned_deprecated_rq = true; } -rcu_read_unlock ();return cpu_rq (cpu); }
@@ -6425,8 +6429,17 @@ __bpf_kfunc struct rq *scx_bpf_locked_rq(void)*/ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu) { -if (!kf_cpu_valid (cpu,NULL )) +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch))return NULL ; + +if (!ops_cpu_valid (sch, cpu,NULL )) +return NULL ; +return rcu_dereference (cpu_rq (cpu)->curr); }
kernel/sched/ext_idle.c +8 -4
@@ -864,7 +864,7 @@ static s32 select_cpu_from_kfunc(struct scx_sched *sch, struct task_struct *p,struct rq_flags rf;s32 cpu; -if (!kf_cpu_valid (prev_cpu,NULL )) +if (!ops_cpu_valid (sch, prev_cpu,NULL ))return -EINVAL ;if (!check_builtin_idle_enabled (sch))
@@ -923,9 +923,13 @@ static s32 select_cpu_from_kfunc(struct scx_sched *sch, struct task_struct *p,*/ __bpf_kfunc int scx_bpf_cpu_node(s32 cpu) { -if (!kf_cpu_valid (cpu,NULL )) -return NUMA_NO_NODE ; +struct scx_sched *sch; + +guard (rcu)(); + sch =rcu_dereference (scx_root); +if (unlikely (!sch) || !ops_cpu_valid (sch, cpu,NULL )) +return NUMA_NO_NODE ;return cpu_to_node (cpu); }
@@ -1154,7 +1158,7 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu)if (!check_builtin_idle_enabled (sch))return false; -if (!kf_cpu_valid (cpu,NULL )) +if (!ops_cpu_valid (sch, cpu,NULL ))return false;return scx_idle_test_and_clear_cpu (cpu);
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 12/46 sched_ext: Add the @sch parameter to scx_dsq_insert_preamble/commit()
In reply to Tejun HeoIn preparation for multiple scheduler support, add the @sch parameter to scx_dsq_insert_preamble/commit() and update the callers to read $scx_root and pass it in. The passed in @sch parameter is not used yet.
--- kernel/sched/ext.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-)
kernel/sched/ext.c +22 -7
@@ -5240,7 +5240,8 @@ void __init init_sched_ext_class(void)/******************************************************************************** * Helpers that can be called from the BPF scheduler. */ -static bool scx_dsq_insert_preamble (struct task_struct *p,u64 enq_flags) +static bool scx_dsq_insert_preamble (struct scx_sched *sch,struct task_struct *p, +u64 enq_flags) {if (!scx_kf_allowed (SCX_KF_ENQUEUE |SCX_KF_DISPATCH ))return false;
@@ -5260,8 +5261,8 @@ static bool scx_dsq_insert_preamble(struct task_struct *p, u64 enq_flags)return true; } -static void scx_dsq_insert_commit (struct task_struct *p,u64 dsq_id, -u64 enq_flags) +static void scx_dsq_insert_commit (struct scx_sched *sch,struct task_struct *p, +u64 dsq_id,u64 enq_flags) {struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx);struct task_struct *ddsp_task;
@@ -5325,7 +5326,14 @@ __bpf_kfunc_start_defs(); __bpf_kfunc void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice,u64 enq_flags) { -if (!scx_dsq_insert_preamble (p, enq_flags)) +struct scx_sched *sch; + +guard (rcu)(); + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return ; + +if (!scx_dsq_insert_preamble (sch, p, enq_flags))return ;if (slice)
@@ -5333,7 +5341,7 @@ __bpf_kfunc void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 sliceelse p->scx.slice = p->scx.slice ?:1 ; -scx_dsq_insert_commit (p, dsq_id, enq_flags); +scx_dsq_insert_commit (sch, p, dsq_id, enq_flags); }/**
@@ -5360,7 +5368,14 @@ __bpf_kfunc void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice __bpf_kfunc void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 dsq_id,u64 slice,u64 vtime,u64 enq_flags) { -if (!scx_dsq_insert_preamble (p, enq_flags)) +struct scx_sched *sch; + +guard (rcu)(); + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return ; + +if (!scx_dsq_insert_preamble (sch, p, enq_flags))return ;if (slice)
@@ -5370,7 +5385,7 @@ __bpf_kfunc void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 dsq_id, p->scx.dsq_vtime = vtime; -scx_dsq_insert_commit (p, dsq_id, enq_flags |SCX_ENQ_DSQ_PRIQ ); +scx_dsq_insert_commit (sch, p, dsq_id, enq_flags |SCX_ENQ_DSQ_PRIQ ); }__bpf_kfunc_end_defs ();
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 13/46 sched_ext: Drop scx_kf_exit() and scx_kf_error()
In reply to Tejun HeoThe intention behind scx_kf_exit/error() was that when called from kfuncs, scx_kf_exit/error() would be able to implicitly determine the scx_sched instance being operated on and thus wouldn't need the @sch parameter passed in explicitly. This turned out to be unnecessarily complicated to implement and not have enough practical benefits. Replace scx_kf_exit/error() usages with scx_exit/error() which take an explicit @sch parameter.
- Add the @sch parameter to scx_kf_allowed(), scx_kf_allowed_on_arg_tasks, mark_direct_dispatch() and other intermediate functions transitively.
- In callers that don't already have @sch available, grab RCU, read $scx_root, verify it's not NULL and use it.
--- kernel/sched/ext.c | 126 +++++++++++++++++++++++----------------- kernel/sched/ext_idle.c | 25 +++++--- 2 files changed, 88 insertions(+), 63 deletions(-)
kernel/sched/ext.c +72 -54
@@ -151,24 +151,7 @@ static __printf(4, 5) void scx_exit(struct scx_sched *sch,va_end (args); } -static __printf(3, 4) void scx_kf_exit(enum scx_exit_kind kind, s64 exit_code, -const char *fmt, ...) -{ -struct scx_sched *sch; -va_list args; - -rcu_read_lock (); - sch =rcu_dereference (scx_root); -if (sch) { -va_start (args, fmt); -scx_vexit (sch, kind, exit_code, fmt, args); -va_end (args); - } -rcu_read_unlock (); -} - #define scx_error(sch, fmt, args...) scx_exit((sch), SCX_EXIT_ERROR, 0, fmt, ##args) -#define scx_kf_error(fmt, args...) scx_kf_exit(SCX_EXIT_ERROR, 0, fmt, ##args)#define SCX_HAS_OP (sch, op) test_bit(SCX_OP_IDX(op), (sch)->has_op)
@@ -329,11 +312,11 @@ do { \ })/* @mask is constant, always inline to cull unnecessary branches */ -static __always_inline bool scx_kf_allowed(u32 mask) +static __always_inline bool scx_kf_allowed(struct scx_sched *sch, u32 mask) {if (unlikely (!(current->scx.kf_mask & mask))) { -scx_kf_error ("kfunc with mask 0x%x called from an operation only allowing 0x%x" , - mask, current->scx.kf_mask); +scx_error (sch,"kfunc with mask 0x%x called from an operation only allowing 0x%x" , + mask, current->scx.kf_mask);return false; }
@@ -346,13 +329,13 @@ static __always_inline bool scx_kf_allowed(u32 mask)*/ if (unlikely (highest_bit (mask) ==SCX_KF_CPU_RELEASE && (current->scx.kf_mask &higher_bits (SCX_KF_CPU_RELEASE )))) { -scx_kf_error ("cpu_release kfunc called from a nested operation" ); +scx_error (sch,"cpu_release kfunc called from a nested operation" );return false; }if (unlikely (highest_bit (mask) ==SCX_KF_DISPATCH && (current->scx.kf_mask &higher_bits (SCX_KF_DISPATCH )))) { -scx_kf_error ("dispatch kfunc called from a nested operation" ); +scx_error (sch,"dispatch kfunc called from a nested operation" );return false; }
@@ -360,15 +343,16 @@ static __always_inline bool scx_kf_allowed(u32 mask) }/* see SCX_CALL_OP_TASK() */ -static __always_inline bool scx_kf_allowed_on_arg_tasks(u32 mask, +static __always_inline bool scx_kf_allowed_on_arg_tasks(struct scx_sched *sch, +u32 mask,struct task_struct *p) { -if (!scx_kf_allowed (mask)) +if (!scx_kf_allowed (sch, mask))return false;if (unlikely ((p != current->scx.kf_tasks[0 ] && p != current->scx.kf_tasks[1 ]))) { -scx_kf_error ("called on a task not being operated on" ); +scx_error (sch,"called on a task not being operated on" );return false; }
@@ -1115,7 +1099,8 @@ static struct scx_dispatch_q *find_dsq_for_dispatch(struct scx_sched *sch,return dsq; } -static void mark_direct_dispatch (struct task_struct *ddsp_task, +static void mark_direct_dispatch (struct scx_sched *sch, +struct task_struct *ddsp_task,struct task_struct *p,u64 dsq_id,u64 enq_flags) {
@@ -1129,10 +1114,10 @@ static void mark_direct_dispatch(struct task_struct *ddsp_task,/* @p must match the task on the enqueue path */ if (unlikely (p != ddsp_task)) {if (IS_ERR (ddsp_task)) -scx_kf_error ("%s[%d] already direct-dispatched" , +scx_error (sch,"%s[%d] already direct-dispatched" , p->comm, p->pid);else -scx_kf_error ("scheduling for %s[%d] but trying to direct-dispatch %s[%d]" , +scx_error (sch,"scheduling for %s[%d] but trying to direct-dispatch %s[%d]" , ddsp_task->comm, ddsp_task->pid, p->comm, p->pid);return ;
@@ -5243,18 +5228,18 @@ void __init init_sched_ext_class(void)static bool scx_dsq_insert_preamble (struct scx_sched *sch,struct task_struct *p,u64 enq_flags) { -if (!scx_kf_allowed (SCX_KF_ENQUEUE |SCX_KF_DISPATCH )) +if (!scx_kf_allowed (sch,SCX_KF_ENQUEUE |SCX_KF_DISPATCH ))return false;lockdep_assert_irqs_disabled ();if (unlikely (!p)) { -scx_kf_error ("called with NULL task" ); +scx_error (sch,"called with NULL task" );return false; }if (unlikely (enq_flags & __SCX_ENQ_INTERNAL_MASK)) { -scx_kf_error ("invalid enq_flags 0x%llx" , enq_flags); +scx_error (sch,"invalid enq_flags 0x%llx" , enq_flags);return false; }
@@ -5269,12 +5254,12 @@ static void scx_dsq_insert_commit(struct scx_sched *sch, struct task_struct *p, ddsp_task =__this_cpu_read (direct_dispatch_task);if (ddsp_task) { -mark_direct_dispatch (ddsp_task, p, dsq_id, enq_flags); +mark_direct_dispatch (sch, ddsp_task, p, dsq_id, enq_flags);return ; }if (unlikely (dspc->cursor >= scx_dsp_max_batch)) { -scx_kf_error ("dispatch buffer overflow" ); +scx_error (sch,"dispatch buffer overflow" );return ; }
@@ -5410,7 +5395,8 @@ static bool scx_dsq_move(struct bpf_iter_scx_dsq_kern *kit,bool in_balance;unsigned long flags; -if (!scx_kf_allowed_if_unlocked () && !scx_kf_allowed (SCX_KF_DISPATCH )) +if (!scx_kf_allowed_if_unlocked () && + !scx_kf_allowed (sch,SCX_KF_DISPATCH ))return false;/*
@@ -5495,7 +5481,15 @@ __bpf_kfunc_start_defs();*/ __bpf_kfunc u32 scx_bpf_dispatch_nr_slots(void) { -if (!scx_kf_allowed (SCX_KF_DISPATCH )) +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return 0 ; + +if (!scx_kf_allowed (sch,SCX_KF_DISPATCH ))return 0 ;return scx_dsp_max_batch -__this_cpu_read (scx_dsp_ctx->cursor);
@@ -5510,14 +5504,21 @@ __bpf_kfunc u32 scx_bpf_dispatch_nr_slots(void) __bpf_kfunc void scx_bpf_dispatch_cancel(void) {struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx); +struct scx_sched *sch; -if (!scx_kf_allowed (SCX_KF_DISPATCH )) +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return ; + +if (!scx_kf_allowed (sch,SCX_KF_DISPATCH ))return ;if (dspc->cursor >0 ) dspc->cursor--;else -scx_kf_error ("dispatch buffer underflow" ); +scx_error (sch,"dispatch buffer underflow" ); }/**
@@ -5540,7 +5541,7 @@ __bpf_kfunc bool scx_bpf_dsq_move_to_local(u64 dsq_id)struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx);struct scx_dispatch_q *dsq; -if (!scx_kf_allowed (SCX_KF_DISPATCH )) +if (!scx_kf_allowed (sch,SCX_KF_DISPATCH ))return false;flush_dispatch_buf (sch, dspc->rq);
@@ -5687,12 +5688,18 @@ __bpf_kfunc_start_defs();*/ __bpf_kfunc u32 scx_bpf_reenqueue_local(void) { +struct scx_sched *sch;LIST_HEAD (tasks);u32 nr_enqueued =0 ;struct rq *rq;struct task_struct *p, *n; -if (!scx_kf_allowed (SCX_KF_CPU_RELEASE )) +guard (rcu)(); + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return 0 ; + +if (!scx_kf_allowed (sch,SCX_KF_CPU_RELEASE ))return 0 ; rq =cpu_rq (smp_processor_id ());
@@ -5837,7 +5844,7 @@ static void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags)struct rq *target_rq =cpu_rq (cpu);if (unlikely (flags & (SCX_KICK_PREEMPT |SCX_KICK_WAIT ))) -scx_kf_error ("PREEMPT/WAIT cannot be used with SCX_KICK_IDLE" ); +scx_error (sch,"PREEMPT/WAIT cannot be used with SCX_KICK_IDLE" );if (raw_spin_rq_trylock (target_rq)) {if (can_skip_idle_kick (target_rq)) {
@@ -6070,20 +6077,20 @@ static s32 __bstr_format(struct scx_sched *sch, u64 *data_buf, char *line_buf,if (data__sz %8 || data__sz >MAX_BPRINTF_VARARGS *8 || (data__sz && !data)) { -scx_kf_error ("invalid data=%p and data__sz=%u" , (void *)data, data__sz); +scx_error (sch,"invalid data=%p and data__sz=%u" , (void *)data, data__sz);return -EINVAL ; } ret =copy_from_kernel_nofault (data_buf, data, data__sz);if (ret <0 ) { -scx_kf_error ("failed to read data fields (%d)" , ret); +scx_error (sch,"failed to read data fields (%d)" , ret);return ret; } ret =bpf_bprintf_prepare (fmt,UINT_MAX , data_buf, data__sz /8 , &bprintf_data);if (ret <0 ) { -scx_kf_error ("format preparation failed (%d)" , ret); +scx_error (sch,"format preparation failed (%d)" , ret);return ret; }
@@ -6091,7 +6098,7 @@ static s32 __bstr_format(struct scx_sched *sch, u64 *data_buf, char *line_buf, bprintf_data.bin_args);bpf_bprintf_cleanup (&bprintf_data);if (ret <0 ) { -scx_kf_error ("(\"%s\", %p, %u) failed to format" , fmt, data, data__sz); +scx_error (sch,"(\"%s\", %p, %u) failed to format" , fmt, data, data__sz);return ret; }
@@ -6127,7 +6134,7 @@ __bpf_kfunc void scx_bpf_exit_bstr(s64 exit_code, char *fmt, sch =rcu_dereference_bh (scx_root);if (likely (sch) &&bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 ) -scx_kf_exit (SCX_EXIT_UNREG_BPF , exit_code,"%s" , scx_exit_bstr_buf.line); +scx_exit (sch,SCX_EXIT_UNREG_BPF , exit_code,"%s" , scx_exit_bstr_buf.line);raw_spin_unlock_irqrestore (&scx_exit_bstr_buf_lock, flags); }
@@ -6150,7 +6157,7 @@ __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data, sch =rcu_dereference_bh (scx_root);if (likely (sch) &&bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 ) -scx_kf_exit (SCX_EXIT_ERROR_BPF ,0 ,"%s" , scx_exit_bstr_buf.line); +scx_exit (sch,SCX_EXIT_ERROR_BPF ,0 ,"%s" , scx_exit_bstr_buf.line);raw_spin_unlock_irqrestore (&scx_exit_bstr_buf_lock, flags); }
@@ -6181,7 +6188,7 @@ __bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data,return ;if (raw_smp_processor_id () != dd->cpu) { -scx_kf_error ("scx_bpf_dump() must only be called from ops.dump() and friends" ); +scx_error (sch,"scx_bpf_dump() must only be called from ops.dump() and friends" );return ; }
@@ -6285,7 +6292,7 @@ __bpf_kfunc void scx_bpf_cpuperf_set(s32 cpu, u32 perf)return ;if (unlikely (perf >SCX_CPUPERF_ONE )) { -scx_kf_error ("Invalid cpuperf target %u for CPU %d" , perf, cpu); +scx_error (sch,"Invalid cpuperf target %u for CPU %d" , perf, cpu);return ; }
@@ -6298,7 +6305,7 @@ __bpf_kfunc void scx_bpf_cpuperf_set(s32 cpu, u32 perf)* to the corresponding CPU to prevent ABBA deadlocks. */ if (locked_rq && rq != locked_rq) { -scx_kf_error ("Invalid target CPU %d" , cpu); +scx_error (sch,"Invalid target CPU %d" , cpu);return ; }
@@ -6422,16 +6429,20 @@ __bpf_kfunc struct rq *scx_bpf_cpu_rq(s32 cpu)*/ __bpf_kfunc struct rq *scx_bpf_locked_rq(void) { +struct scx_sched *sch;struct rq *rq; -preempt_disable (); +guard (preempt)(); + + sch =rcu_dereference_sched (scx_root); +if (unlikely (!sch)) +return NULL ; + rq =scx_locked_rq ();if (!rq) { -preempt_enable (); -scx_kf_error ("accessing rq without holding rq lock" ); +scx_error (sch,"accessing rq without holding rq lock" );return NULL ; } -preempt_enable ();return rq; }
@@ -6474,8 +6485,15 @@ __bpf_kfunc struct cgroup *scx_bpf_task_cgroup(struct task_struct *p) {struct task_group *tg = p->sched_task_group;struct cgroup *cgrp = &cgrp_dfl_root.cgrp; +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) + goto out; -if (!scx_kf_allowed_on_arg_tasks (__SCX_KF_RQ_LOCKED, p)) +if (!scx_kf_allowed_on_arg_tasks (sch, __SCX_KF_RQ_LOCKED, p)) goto out; cgrp =tg_cgrp (tg);
kernel/sched/ext_idle.c +16 -9
@@ -822,7 +822,7 @@ void scx_idle_disable(void)static int validate_node (struct scx_sched *sch,int node) {if (!static_branch_likely (&scx_builtin_idle_per_node)) { -scx_kf_error ("per-node idle tracking is disabled" ); +scx_error (sch,"per-node idle tracking is disabled" );return -EOPNOTSUPP ; }
@@ -832,13 +832,13 @@ static int validate_node(struct scx_sched *sch, int node)/* Make sure node is in a valid range */ if (node <0 || node >= nr_node_ids) { -scx_kf_error ("invalid node %d" , node); +scx_error (sch,"invalid node %d" , node);return -EINVAL ; }/* Make sure the node is part of the set of possible nodes */ if (!node_possible (node)) { -scx_kf_error ("unavailable node %d" , node); +scx_error (sch,"unavailable node %d" , node);return -EINVAL ; }
@@ -852,7 +852,7 @@ static bool check_builtin_idle_enabled(struct scx_sched *sch)if (static_branch_likely (&scx_builtin_idle_enabled))return true; -scx_kf_error ("built-in idle tracking is disabled" ); +scx_error (sch,"built-in idle tracking is disabled" );return false; }
@@ -880,7 +880,7 @@ static s32 select_cpu_from_kfunc(struct scx_sched *sch, struct task_struct *p,if (scx_kf_allowed_if_unlocked ()) { rq =task_rq_lock (p, &rf); }else { -if (!scx_kf_allowed (SCX_KF_SELECT_CPU |SCX_KF_ENQUEUE )) +if (!scx_kf_allowed (sch,SCX_KF_SELECT_CPU |SCX_KF_ENQUEUE ))return -EPERM ; rq =scx_locked_rq (); }
@@ -1048,7 +1048,7 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void)return cpu_none_mask;if (static_branch_unlikely (&scx_builtin_idle_per_node)) { -scx_kf_error ("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled" ); +scx_error (sch,"SCX_OPS_BUILTIN_IDLE_PER_NODE enabled" );return cpu_none_mask; }
@@ -1107,7 +1107,7 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void)return cpu_none_mask;if (static_branch_unlikely (&scx_builtin_idle_per_node)) { -scx_kf_error ("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled" ); +scx_error (sch,"SCX_OPS_BUILTIN_IDLE_PER_NODE enabled" );return cpu_none_mask; }
@@ -1235,7 +1235,7 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed,return -ENODEV ;if (static_branch_maybe (CONFIG_NUMA , &scx_builtin_idle_per_node)) { -scx_kf_error ("per-node idle tracking is enabled" ); +scx_error (sch,"per-node idle tracking is enabled" );return -EBUSY ; }
@@ -1316,10 +1316,17 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu_node(const struct cpumask *cpus_allowed, __bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed,u64 flags) { +struct scx_sched *sch;s32 cpu; +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return -ENODEV ; +if (static_branch_maybe (CONFIG_NUMA , &scx_builtin_idle_per_node)) { -scx_kf_error ("per-node idle tracking is enabled" ); +scx_error (sch,"per-node idle tracking is enabled" );return -EBUSY ; }
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 14/46 sched_ext: Misc updates around scx_sched instance pointer
In reply to Tejun HeoIn preparation for multiple scheduler support:
- Add the @sch parameter to find_global_dsq() and refill_task_slice_dfl().
- Restructure scx_allow_ttwu_queue() and make it read scx_root into $sch.
- Make RCU protection in scx_dsq_move() and scx_bpf_dsq_move_to_local() explicit.
--- kernel/sched/ext.c | 62 ++++++++++++++++++++++++++++++---------------- 1 file changed, 40 insertions(+), 22 deletions(-)
kernel/sched/ext.c +40 -22
@@ -181,10 +181,9 @@ static bool u32_before(u32 a, u32 b)return (s32)(a - b) <0 ; } -static struct scx_dispatch_q *find_global_dsq (struct task_struct *p) +static struct scx_dispatch_q *find_global_dsq (struct scx_sched *sch, +struct task_struct *p) { -struct scx_sched *sch = scx_root; -return sch->global_dsqs[cpu_to_node (task_cpu (p))]; }
@@ -880,10 +879,10 @@ static void dsq_mod_nr(struct scx_dispatch_q *dsq, s32 delta)WRITE_ONCE (dsq->nr, dsq->nr + delta); } -static void refill_task_slice_dfl (struct task_struct *p) +static void refill_task_slice_dfl (struct scx_sched *sch,struct task_struct *p) { p->scx.slice =SCX_SLICE_DFL ; -__scx_add_event (scx_root,SCX_EV_REFILL_SLICE_DFL ,1 ); +__scx_add_event (sch,SCX_EV_REFILL_SLICE_DFL ,1 ); } static void dispatch_enqueue(struct scx_sched *sch, struct scx_dispatch_q *dsq,
@@ -901,7 +900,7 @@ static void dispatch_enqueue(struct scx_sched *sch, struct scx_dispatch_q *dsq,scx_error (sch,"attempting to dispatch to a destroyed dsq" );/* fall back to the global dsq */ raw_spin_unlock (&dsq->lock); - dsq =find_global_dsq (p); + dsq =find_global_dsq (sch, p);raw_spin_lock (&dsq->lock); } }
@@ -1080,20 +1079,20 @@ static struct scx_dispatch_q *find_dsq_for_dispatch(struct scx_sched *sch,s32 cpu = dsq_id &SCX_DSQ_LOCAL_CPU_MASK ;if (!ops_cpu_valid (sch, cpu,"in SCX_DSQ_LOCAL_ON dispatch verdict" )) -return find_global_dsq (p); +return find_global_dsq (sch, p);return &cpu_rq (cpu)->scx.local_dsq; }if (dsq_id ==SCX_DSQ_GLOBAL ) - dsq =find_global_dsq (p); + dsq =find_global_dsq (sch, p);else dsq =find_user_dsq (sch, dsq_id);if (unlikely (!dsq)) {scx_error (sch,"non-existent DSQ 0x%llx for %s[%d]" , dsq_id, p->comm, p->pid); -return find_global_dsq (p); +return find_global_dsq (sch, p); }return dsq;
@@ -1272,15 +1271,15 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,* higher priority it becomes from scx_prio_less()'s POV. */ touch_core_sched (rq, p); -refill_task_slice_dfl (p); +refill_task_slice_dfl (sch, p); local_norefill:dispatch_enqueue (sch, &rq->scx.local_dsq, p, enq_flags);return ; global:touch_core_sched (rq, p);/* see the comment in local: */ -refill_task_slice_dfl (p); -dispatch_enqueue (sch,find_global_dsq (p), p, enq_flags); +refill_task_slice_dfl (sch, p); +dispatch_enqueue (sch,find_global_dsq (sch, p), p, enq_flags); }static bool task_runnable (const struct task_struct *p)
@@ -1692,7 +1691,7 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch, dst_rq = container_of(dst_dsq, struct rq, scx.local_dsq);if (src_rq != dst_rq &&unlikely (!task_can_run_on_remote_rq (sch, p, dst_rq, true))) { - dst_dsq =find_global_dsq (p); + dst_dsq =find_global_dsq (sch, p); dst_rq = src_rq; } }else {
@@ -1848,7 +1847,7 @@ static void dispatch_to_local_dsq(struct scx_sched *sch, struct rq *rq,if (src_rq != dst_rq &&unlikely (!task_can_run_on_remote_rq (sch, p, dst_rq, true))) { -dispatch_enqueue (sch,find_global_dsq (p), p, +dispatch_enqueue (sch,find_global_dsq (sch, p), p, enq_flags |SCX_ENQ_CLEAR_OPSS );return ; }
@@ -2380,7 +2379,7 @@ static struct task_struct *pick_task_scx(struct rq *rq)if (keep_prev) { p = prev;if (!p->scx.slice) -refill_task_slice_dfl (p); +refill_task_slice_dfl (rcu_dereference_sched (scx_root), p); }else { p =first_local_task (rq);if (!p) {
@@ -2391,14 +2390,14 @@ static struct task_struct *pick_task_scx(struct rq *rq) }if (unlikely (!p->scx.slice)) { -struct scx_sched *sch = scx_root; +struct scx_sched *sch =rcu_dereference_sched (scx_root);if (!scx_rq_bypassing (rq) && !sch->warned_zero_slice) {printk_deferred (KERN_WARNING "sched_ext: %s[%d] has zero slice in %s()\n" , p->comm, p->pid, __func__); sch->warned_zero_slice = true; } -refill_task_slice_dfl (p); +refill_task_slice_dfl (sch, p); } }
@@ -2487,7 +2486,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag cpu =scx_select_cpu_dfl (p, prev_cpu, wake_flags,NULL ,0 );if (cpu >=0 ) { -refill_task_slice_dfl (p); +refill_task_slice_dfl (sch, p); p->scx.ddsp_dsq_id =SCX_DSQ_LOCAL ; }else { cpu = prev_cpu;
@@ -3572,9 +3571,22 @@ bool task_should_scx(int policy)bool scx_allow_ttwu_queue (const struct task_struct *p) { -return !scx_enabled () || - (scx_root->ops.flags &SCX_OPS_ALLOW_QUEUED_WAKEUP ) || - p->sched_class != &ext_sched_class; +struct scx_sched *sch; + +if (!scx_enabled ()) +return true; + + sch =rcu_dereference_sched (scx_root); +if (unlikely (!sch)) +return true; + +if (scx_root->ops.flags &SCX_OPS_ALLOW_QUEUED_WAKEUP ) +return true; + +if (unlikely (p->sched_class != &ext_sched_class)) +return true; + +return false; }/**
@@ -5537,9 +5549,15 @@ __bpf_kfunc void scx_bpf_dispatch_cancel(void)*/ __bpf_kfunc bool scx_bpf_dsq_move_to_local(u64 dsq_id) { -struct scx_sched *sch = scx_root;struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx);struct scx_dispatch_q *dsq; +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (unlikely (!sch)) +return false;if (!scx_kf_allowed (sch,SCX_KF_DISPATCH ))return false;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 15/46 sched_ext: Keep dying tasks on a separate list
In reply to Tejun Heosched_ext needs to be able to iterate all tasks until the task finishes its final scheduling event which can be after it is dropped out of pid hash. The scx_tasks list keeps track of all tasks between fork and free for this purpose.
The planned cgroup sub-scheduler support requires selectively iterating tasks belonging to cgroup subtrees. While the live tasks can be iterated using css_task_iter, there is a similar gap after removal from css. As there won't be too many tasks between release and free at any moment, this can be reasonably addressed by scanning dying tasks looking for tasks which are inside the cgroup subtree.
In preparation, split out scx_tasks lists into two - scx_live_tasks and scx_dying_tasks. scx_task_iter is updated to first iterate scx_live_tasks and then scx_dying_tasks. As tasks can only move from the former to the latter, tasks can't escape iteration; however, it's possible for newly dying tasks to appear twice during iteration. The users of scx_task_iter are updated to handle duplicate iterations.
--- include/linux/sched/ext.h | 2 ++ kernel/exit.c | 1 + kernel/sched/ext.c | 70 +++++++++++++++++++++++++++++---------- 3 files changed, 55 insertions(+), 18 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -207,6 +207,7 @@ struct sched_ext_entity {struct list_head tasks_node; }; +void sched_ext_exit (struct task_struct *p);void sched_ext_free (struct task_struct *p);void print_scx_info (const char *log_lvl,struct task_struct *p);void scx_softlockup (u32 dur_s);
@@ -214,6 +215,7 @@ bool scx_rcu_cpu_stall(void);#else /* !CONFIG_SCHED_CLASS_EXT */ +static inline void sched_ext_exit (struct task_struct *p) {}static inline void sched_ext_free (struct task_struct *p) {}static inline void print_scx_info (const char *log_lvl,struct task_struct *p) {}static inline void scx_softlockup (u32 dur_s) {}
kernel/exit.c +1 -0
@@ -962,6 +962,7 @@ void __noreturn do_exit(long code)exit_thread (tsk);sched_autogroup_exit_task (tsk); +sched_ext_exit (tsk);cgroup_exit (tsk);/*
kernel/sched/ext.c +52 -18
152 lines not on this page · raw message
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 16/46 sched_ext: Implement cgroup subtree iteration for scx_task_iter
In reply to Tejun HeoFor the planned cgroup sub-scheduler support, enable/disable operations are going to be subtree specific and iterating all tasks in the system for those operations can be unnecessarily expensive and disruptive.
cgroup already has mechanisms to perform subtree task iterations. Implement cgroup subtree iteration for scx_task_iter:
- Add optional @cgrp to scx_task_iter_start() which enables cgroup subtree iteration.
- Make scx_task_iter use combination of css_next_descendant_pre() and css_task_iter to iterate all live tasks for cgroup iterations.
- After live task iteration is finished, scan scx_dying_tasks and only visit tasks that are in the cgroup subtree. As scx_dying_tasks is most likely really short, this should be pretty cheap.
--- kernel/sched/ext.c | 76 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 67 insertions(+), 9 deletions(-)
kernel/sched/ext.c +67 -9
@@ -451,11 +451,17 @@ struct scx_task_iter {struct rq_flags rf;u32 cnt;bool list_locked; +#ifdef CONFIG_CGROUPS +struct cgroup *cgrp; +struct cgroup_subsys_state *css_pos; +struct css_task_iter css_iter; +#endif };/** * scx_task_iter_start - Lock scx_tasks_lock and start a task iteration * @iter: iterator to init +* @cgrp: Optional root of cgroup subhierarchy to iterate * * Initialize @iter and return with scx_tasks_lock held. Once initialized, @iter * must eventually be stopped with scx_task_iter_stop().
@@ -469,8 +475,14 @@ struct scx_task_iter {* All tasks which existed when the iteration started are guaranteed to be * visited as long as they still exist. Tasks which exit while iteration is in * progress may be visited twice. The caller must be able to handle such cases. +* +* @if @cgrp is NULL, scx_live_tasks are walked followed by scx_dying_tasks. If +* @cgrp is not NULL, @cgrp's tasks are walked using css_task_iter followed by +* scx_dying_tasks. To guarantee that all tasks are visited at least once, the +* caller must be holding scx_fork_rwsem. In the cgroup case, the caller must +* also be holding scx_cgroup_rwsem to prevent cgroup task migrations. */ -static void scx_task_iter_start (struct scx_task_iter *iter) +static void scx_task_iter_start (struct scx_task_iter *iter,struct cgroup *cgrp) {BUILD_BUG_ON (__SCX_DSQ_ITER_ALL_FLAGS & ((1U << __SCX_DSQ_LNODE_PRIV_SHIFT) -1 ));
@@ -478,8 +490,20 @@ static void scx_task_iter_start(struct scx_task_iter *iter)spin_lock_irq (&scx_tasks_lock); iter->head = &scx_live_tasks; +#ifdef CONFIG_CGROUPS +if (cgrp) { + iter->cgrp = cgrp; + iter->css_pos =css_next_descendant_pre (NULL , &iter->cgrp->self); +css_task_iter_start (iter->css_pos,0 , &iter->css_iter); +/* walking cgroup tasks instead, skip scx_live_tasks */ + iter->head = &scx_dying_tasks; + }else { + iter->cgrp =NULL ; + iter->css_pos =NULL ; + } +#endif iter->cursor = (struct sched_ext_entity ){ .flags =SCX_TASK_CURSOR }; -list_add (&iter->cursor.tasks_node, &scx_live_tasks); +list_add (&iter->cursor.tasks_node, iter->head); iter->locked_task =NULL ; iter->cnt =0 ; iter->list_locked = true;
@@ -530,6 +554,8 @@ static void __scx_task_iter_maybe_relock(struct scx_task_iter *iter)static void scx_task_iter_stop (struct scx_task_iter *iter) {__scx_task_iter_maybe_relock (iter); +if (iter->css_pos) +css_task_iter_end (&iter->css_iter);list_del_init (&iter->cursor.tasks_node);scx_task_iter_unlock (iter); }
@@ -557,13 +583,45 @@ static struct task_struct *scx_task_iter_next(struct scx_task_iter *iter)__scx_task_iter_maybe_relock (iter); } retry: + +#ifdef CONFIG_CGROUPS +/* +* For cgroup iterations, use css_task_iter for live tasks. iter->head +* is already set to scx_dying_tasks. +*/ +while (iter->css_pos) { +struct task_struct *p; + + p =css_task_iter_next (&iter->css_iter); +if (p) +return p; + +css_task_iter_end (&iter->css_iter); + iter->css_pos =css_next_descendant_pre (iter->css_pos, + &iter->cgrp->self); +if (iter->css_pos) +css_task_iter_start (iter->css_pos,0 , &iter->css_iter); + } +#endif +list_for_each_entry (pos, cursor, tasks_node) { + struct task_struct *p = container_of(pos, struct task_struct, scx); +if (&pos->tasks_node == iter->head)break ; -if (!(pos->flags &SCX_TASK_CURSOR )) { -list_move (cursor, &pos->tasks_node); - return container_of(pos, struct task_struct, scx); - } +if (pos->flags &SCX_TASK_CURSOR ) +continue ; +#ifdef CONFIG_CGROUPS +/* +* For cgroup iterations, this loop is only used for iterating +* dying tasks. Filter out tasks which aren't in the target +* subtree. +*/ +if (iter->cgrp && !task_under_cgroup_hierarchy (p, iter->cgrp)) +continue ; +#endif +list_move (cursor, &pos->tasks_node); +return p; }if (iter->head == &scx_live_tasks) {
@@ -3936,7 +3994,7 @@ static void scx_disable_workfn(struct kthread_work *work) scx_init_task_enabled = false; -scx_task_iter_start (&sti); +scx_task_iter_start (&sti,NULL );while ((p =scx_task_iter_next_locked (&sti))) {/* @p may be being visited twice, doesn't matter */ const struct sched_class *old_class = p->sched_class;
@@ -4639,7 +4697,7 @@ static int scx_enable(struct sched_ext_ops *ops, struct bpf_link *link)if (ret) goto err_disable_unlock_all; -scx_task_iter_start (&sti); +scx_task_iter_start (&sti,NULL );while ((p =scx_task_iter_next_locked (&sti))) {/* * Task iteration may visit the same task twice when racing
@@ -4688,7 +4746,7 @@ static int scx_enable(struct sched_ext_ops *ops, struct bpf_link *link)* scx_tasks_lock. */ percpu_down_write (&scx_fork_rwsem); -scx_task_iter_start (&sti); +scx_task_iter_start (&sti,NULL );while ((p =scx_task_iter_next_locked (&sti))) {/* @p may be being visited twice, doesn't matter */ const struct sched_class *old_class = p->sched_class;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 17/46 sched_ext: Add @kargs to scx_fork()
In reply to Tejun HeoMake sched_cgroup_fork() pass @kargs to scx_fork(). This will be used to determine @p's cgroup for cgroup sub-sched support.
--- kernel/sched/core.c | 2 +- kernel/sched/ext.c | 2 +- kernel/sched/ext.h | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-)
kernel/sched/core.c +1 -1
@@ -4805,7 +4805,7 @@ int sched_cgroup_fork(struct task_struct *p, struct kernel_clone_args *kargs) p->sched_class->task_fork (p);raw_spin_unlock_irqrestore (&p->pi_lock, flags); -return scx_fork (p); +return scx_fork (p, kargs); }void sched_cancel_fork (struct task_struct *p)
kernel/sched/ext.c +1 -1
@@ -2950,7 +2950,7 @@ void scx_pre_fork(struct task_struct *p)percpu_down_read (&scx_fork_rwsem); } -int scx_fork (struct task_struct *p) +int scx_fork (struct task_struct *p,struct kernel_clone_args *kargs) {percpu_rwsem_assert_held (&scx_fork_rwsem);
kernel/sched/ext.h +2 -2
@@ -11,7 +11,7 @@void scx_tick (struct rq *rq);void init_scx_entity (struct sched_ext_entity *scx);void scx_pre_fork (struct task_struct *p); -int scx_fork (struct task_struct *p); +int scx_fork (struct task_struct *p,struct kernel_clone_args *kargs);void scx_post_fork (struct task_struct *p);void scx_cancel_fork (struct task_struct *p);bool scx_can_stop_tick (struct rq *rq);
@@ -44,7 +44,7 @@ bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,static inline void scx_tick (struct rq *rq) {}static inline void scx_pre_fork (struct task_struct *p) {} -static inline int scx_fork (struct task_struct *p) {return 0 ; } +static inline int scx_fork (struct task_struct *p,struct kernel_clone_args *kargs) {return 0 ; }static inline void scx_post_fork (struct task_struct *p) {}static inline void scx_cancel_fork (struct task_struct *p) {}static inline u32 scx_cpuperf_target (s32 cpu) {return 0 ; }
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 18/46 sched/core: Swap the order between sched_post_fork() and cgroup_post_fork()
In reply to Tejun HeoThe planned sched_ext cgroup sub-scheduler support needs the newly forked task to be associated with its cgroup in its post_fork() hook. There is no existing ordering requirement between the two now. Swap them and note the new ordering requirement.
--- kernel/fork.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/fork.c +5 -1
@@ -2415,8 +2415,12 @@ __latent_entropy struct task_struct *copy_process(fd_install (pidfd, pidfile);proc_fork_connector (p); -sched_post_fork (p); +/* +* sched_ext needs @p to be associated with its cgroup in its post_fork +* hook. cgroup_post_fork() should come before sched_post_fork(). +*/ cgroup_post_fork (p, args); +sched_post_fork (p);perf_event_fork (p);trace_task_newtask (p, clone_flags);
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 19/46 cgroup: Expose some cgroup helpers
In reply to Tejun HeoExpose the following through cgroup.h:
- cgroup_on_dfl()
- cgroup_is_dead()
- cgroup_for_each_live_child()
- cgroup_for_each_live_descendant_pre()
- cgroup_for_each_live_descendant_post()
Until now, these didn't need to be exposed because controllers only cared about the css hierarchy. The planned sched_ext hierarchical scheduler support will be based on the default cgroup hierarchy, which is in line with the existing BPF cgroup support, and thus needs these exposed.
--- include/linux/cgroup.h | 65 ++++++++++++++++++++++++++++++++- kernel/cgroup/cgroup-internal.h | 6 --- kernel/cgroup/cgroup.c | 55 ---------------------------- 3 files changed, 63 insertions(+), 63 deletions(-)
include/linux/cgroup.h +63 -2
@@ -41,6 +41,14 @@ struct kernel_clone_args;#ifdef CONFIG_CGROUPS +/* +* To avoid confusing the compiler (and generating warnings) with code +* that attempts to access what would be a 0-element array (i.e. sized +* to a potentially empty array when CGROUP_SUBSYS_COUNT == 0), this +* constant expression can be added. +*/ +#define CGROUP_HAS_SUBSYS_CONFIG (CGROUP_SUBSYS_COUNT > 0) +enum css_task_iter_flags {CSS_TASK_ITER_PROCS = (1U <<0 ),/* walk only threadgroup leaders */ CSS_TASK_ITER_THREADED = (1U << 1), /* walk all threaded css_sets in the domain */
@@ -75,6 +83,7 @@ enum cgroup_lifetime_events {extern struct file_system_type cgroup_fs_type;extern struct cgroup_root cgrp_dfl_root;extern struct css_set init_css_set; +extern struct mutex cgroup_mutex;extern spinlock_t css_set_lock;extern struct blocking_notifier_head cgroup_lifetime_notifier;
@@ -102,6 +111,8 @@ extern struct blocking_notifier_head cgroup_lifetime_notifier;#define cgroup_subsys_on_dfl (ss) \ static_branch_likely(&ss ## _on_dfl_key) +bool cgroup_on_dfl (const struct cgroup *cgrp); +bool css_has_online_children (struct cgroup_subsys_state *css);struct cgroup_subsys_state *css_from_id (int id,struct cgroup_subsys *ss); struct cgroup_subsys_state *cgroup_e_css(struct cgroup *cgroup,
@@ -272,6 +283,32 @@ void css_task_iter_end(struct css_task_iter *it);for ((pos) =css_next_descendant_post (NULL , (css)); (pos); \ (pos) =css_next_descendant_post ((pos), (css))) +/* iterate over child cgrps, lock should be held throughout iteration */ +#define cgroup_for_each_live_child (child, cgrp) \ + list_for_each_entry((child), &(cgrp)->self.children, self.sibling) \ + if (({ lockdep_assert_held(&cgroup_mutex); \ + cgroup_is_dead(child); })) \ + ; \ + else + +/* walk live descendants in pre order */ +#define cgroup_for_each_live_descendant_pre (dsct, d_css, cgrp) \ + css_for_each_descendant_pre((d_css), cgroup_css((cgrp), NULL)) \ + if (({ lockdep_assert_held(&cgroup_mutex); \ + (dsct) = (d_css)->cgroup; \ + cgroup_is_dead(dsct); })) \ + ; \ + else + +/* walk live descendants in postorder */ +#define cgroup_for_each_live_descendant_post (dsct, d_css, cgrp) \ + css_for_each_descendant_post((d_css), cgroup_css((cgrp), NULL)) \ + if (({ lockdep_assert_held(&cgroup_mutex); \ + (dsct) = (d_css)->cgroup; \ + cgroup_is_dead(dsct); })) \ + ; \ + else +/** * cgroup_taskset_for_each - iterate cgroup_taskset * @task: the loop cursor
@@ -334,6 +371,27 @@ static inline u64 cgroup_id(const struct cgroup *cgrp)return cgrp->kn->id; } +/** +* cgroup_css - obtain a cgroup's css for the specified subsystem +* @cgrp: the cgroup of interest +* @ss: the subsystem of interest (%NULL returns @cgrp->self) +* +* Return @cgrp's css (cgroup_subsys_state) associated with @ss. This +* function must be called either under cgroup_mutex or rcu_read_lock() and +* the caller is responsible for pinning the returned css if it wants to +* keep accessing it outside the said locks. This function may return +* %NULL if @cgrp doesn't have @subsys_id enabled. +*/ +static inline struct cgroup_subsys_state *cgroup_css (struct cgroup *cgrp, +struct cgroup_subsys *ss) +{ +if (CGROUP_HAS_SUBSYS_CONFIG && ss) +return rcu_dereference_check (cgrp->subsys[ss->id], +lockdep_is_held (&cgroup_mutex)); +else +return &cgrp->self; +} +/** * css_is_dying - test whether the specified css is dying * @css: target css
@@ -365,6 +423,11 @@ static inline bool css_is_self(struct cgroup_subsys_state *css)return false; } +static inline bool cgroup_is_dead (const struct cgroup *cgrp) +{ +return !(cgrp->self.flags &CSS_ONLINE ); +} +static inline void cgroup_get (struct cgroup *cgrp) {css_get (&cgrp->self);
@@ -380,8 +443,6 @@ static inline void cgroup_put(struct cgroup *cgrp)css_put (&cgrp->self); } -extern struct mutex cgroup_mutex; -static inline void cgroup_lock (void ) {mutex_lock (&cgroup_mutex);
kernel/cgroup/cgroup-internal.h +0 -6
@@ -184,11 +184,6 @@ extern bool cgrp_dfl_visible;for ((ssid) =0 ; (ssid) <CGROUP_SUBSYS_COUNT && \ (((ss) = cgroup_subsys[ssid]) || true); (ssid)++) -static inline bool cgroup_is_dead(const struct cgroup *cgrp) -{ -return !(cgrp->self.flags &CSS_ONLINE ); -} - static inline bool notify_on_release(const struct cgroup *cgrp) {return test_bit (CGRP_NOTIFY_ON_RELEASE , &cgrp->flags);
@@ -222,7 +217,6 @@ static inline void get_css_set(struct css_set *cset) }bool cgroup_ssid_enabled (int ssid); -bool cgroup_on_dfl (const struct cgroup *cgrp);struct cgroup_root *cgroup_root_from_kf (struct kernfs_root *kf_root); struct cgroup *task_cgroup_from_root(struct task_struct *task,
kernel/cgroup/cgroup.c +0 -55
73 lines not on this page · raw message
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 20/46 sched_ext: Update p->scx.disallow warning in scx_init_task()
In reply to Tejun Heo- Always trigger the warning if p->scx.disallow is set for fork inits. There is no reason to set it during forks.
- Flip the positions of if/else arms to ease adding error conditions.
--- kernel/sched/ext.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/sched/ext.c +4 -4
@@ -2825,7 +2825,10 @@ static int scx_init_task(struct task_struct *p, struct task_group *tg, bool forkscx_set_task_state (p,SCX_TASK_INIT );if (p->scx.disallow) { -if (!fork) { +if (unlikely (fork)) { +scx_error (sch,"ops.init_task() set task->scx.disallow for %s[%d] during fork" , + p->comm, p->pid); + }else {struct rq *rq;struct rq_flags rf;
@@ -2844,9 +2847,6 @@ static int scx_init_task(struct task_struct *p, struct task_group *tg, bool fork }task_rq_unlock (rq, p, &rf); - }else if (p->policy ==SCHED_EXT ) { -scx_error (sch,"ops.init_task() set task->scx.disallow for %s[%d] during fork" , - p->comm, p->pid); } }
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 21/46 sched_ext: Minor reorganization of enable/disable path
In reply to Tejun Heo- Separate out scx_root_disable() from scx_disable_workfn().
- Rename scx_enable() to scx_root_enable().
- Add @sch to scx_disable(). The callers are now responsible for providing the scx_sched to disable.
No functional changes.
--- kernel/sched/ext.c | 70 ++++++++++++++++++++++++++-------------------- 1 file changed, 40 insertions(+), 30 deletions(-)
kernel/sched/ext.c +40 -30
@@ -3018,8 +3018,8 @@ void sched_ext_free(struct task_struct *p)spin_unlock_irqrestore (&scx_tasks_lock, flags);/* -* @p is off scx_tasks and wholly ours. scx_enable()'s READY -> ENABLED -* transitions can't race us. Disable ops for @p. +* @p is off scx_tasks and wholly ours. scx_root_enable()'s READY -> +* ENABLED transitions can't race us. Disable ops for @p. */ if (scx_get_task_state (p) !=SCX_TASK_NONE ) {struct rq_flags rf;
@@ -3933,24 +3933,12 @@ static const char *scx_exit_reason(enum scx_exit_kind kind) } } -static void scx_disable_workfn (struct kthread_work *work) +static void scx_root_disable (struct scx_sched *sch) { - struct scx_sched *sch = container_of(work, struct scx_sched, disable_work);struct scx_exit_info *ei = sch->exit_info;struct scx_task_iter sti;struct task_struct *p; -int kind, cpu; - - kind =atomic_read (&sch->exit_kind); -while (true) { -if (kind ==SCX_EXIT_DONE )/* already disabled? */ -return ; -WARN_ON_ONCE (kind ==SCX_EXIT_NONE ); -if (atomic_try_cmpxchg (&sch->exit_kind, &kind,SCX_EXIT_DONE )) -break ; - } - ei->kind = kind; - ei->reason =scx_exit_reason (ei->kind); +int cpu;/* guarantee forward progress by bypassing scx_ops */ scx_bypass (true);
@@ -4078,21 +4066,35 @@ static void scx_disable_workfn(struct kthread_work *work)scx_bypass (false); } -static void scx_disable (enum scx_exit_kind kind) +static void scx_disable_workfn (struct kthread_work *work) +{ + struct scx_sched *sch = container_of(work, struct scx_sched, disable_work); +struct scx_exit_info *ei = sch->exit_info; +int kind; + + kind =atomic_read (&sch->exit_kind); +while (true) { +if (kind ==SCX_EXIT_DONE )/* already disabled? */ +return ; +WARN_ON_ONCE (kind ==SCX_EXIT_NONE ); +if (atomic_try_cmpxchg (&sch->exit_kind, &kind,SCX_EXIT_DONE )) +break ; + } + ei->kind = kind; + ei->reason =scx_exit_reason (ei->kind); + +scx_root_disable (sch); +} + +static void scx_disable (struct scx_sched *sch,enum scx_exit_kind kind) {int none =SCX_EXIT_NONE ; -struct scx_sched *sch;if (WARN_ON_ONCE (kind ==SCX_EXIT_NONE || kind ==SCX_EXIT_DONE )) kind =SCX_EXIT_ERROR ; -rcu_read_lock (); - sch =rcu_dereference (scx_root); -if (sch) { -atomic_try_cmpxchg (&sch->exit_kind, &none, kind); -kthread_queue_work (sch->helper, &sch->disable_work); - } -rcu_read_unlock (); +atomic_try_cmpxchg (&sch->exit_kind, &none, kind); +kthread_queue_work (sch->helper, &sch->disable_work); }static void dump_newline (struct seq_buf *s)
@@ -4558,7 +4560,7 @@ static int validate_ops(struct scx_sched *sch, const struct sched_ext_ops *ops)return 0 ; } -static int scx_enable (struct sched_ext_ops *ops,struct bpf_link *link) +static int scx_root_enable (struct sched_ext_ops *ops,struct bpf_link *link) {struct scx_sched *sch;struct scx_task_iter sti;
@@ -4808,7 +4810,7 @@ static int scx_enable(struct sched_ext_ops *ops, struct bpf_link *link)* Flush scx_disable_work to ensure that error is reported before init * completion. sch's base reference will be put by bpf_scx_unreg(). */ -scx_error (sch,"scx_enable() failed (%d)" , ret); +scx_error (sch,"scx_root_enable() failed (%d)" , ret);kthread_flush_work (&sch->disable_work);return 0 ; }
@@ -4940,7 +4942,7 @@ static int bpf_scx_check_member(const struct btf_type *t,static int bpf_scx_reg (void *kdata,struct bpf_link *link) { -return scx_enable (kdata, link); +return scx_root_enable (kdata, link); }static void bpf_scx_unreg (void *kdata,struct bpf_link *link)
@@ -4948,7 +4950,7 @@ static void bpf_scx_unreg(void *kdata, struct bpf_link *link)struct sched_ext_ops *ops = kdata;struct scx_sched *sch = ops->priv; -scx_disable (SCX_EXIT_UNREG ); +scx_disable (sch,SCX_EXIT_UNREG );kthread_flush_work (&sch->disable_work);kobject_put (&sch->kobj); }
@@ -5074,7 +5076,15 @@ static struct bpf_struct_ops bpf_sched_ext_ops = {static void sysrq_handle_sched_ext_reset (u8 key) { -scx_disable (SCX_EXIT_SYSRQ ); +struct scx_sched *sch; + +rcu_read_lock (); + sch =rcu_dereference (scx_root); +if (likely (sch)) +scx_disable (sch,SCX_EXIT_SYSRQ ); +else +pr_info ("sched_ext: BPF schedulers not loaded\n" ); +rcu_read_unlock (); }static const struct sysrq_key_op sysrq_sched_ext_reset_op = {
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 22/46 sched_ext: Factor out scx_claim_exit() from scx_disable()
In reply to Tejun HeoBoth scx_disable() and scx_vexit() use cmpxchg on sch->exit_kind to claim exit ownership but the latter lacks some sanity checks. Factor out scx_claim_exit() from scx_disable() and use it in both places. This will also be used to implement sub-sched disabling.
--- kernel/sched/ext.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
kernel/sched/ext.c +9 -5
@@ -4086,15 +4086,20 @@ static void scx_disable_workfn(struct kthread_work *work)scx_root_disable (sch); } -static void scx_disable (struct scx_sched *sch,enum scx_exit_kind kind) +static bool scx_claim_exit (struct scx_sched *sch,enum scx_exit_kind kind) {int none =SCX_EXIT_NONE ;if (WARN_ON_ONCE (kind ==SCX_EXIT_NONE || kind ==SCX_EXIT_DONE )) kind =SCX_EXIT_ERROR ; -atomic_try_cmpxchg (&sch->exit_kind, &none, kind); -kthread_queue_work (sch->helper, &sch->disable_work); +return atomic_try_cmpxchg (&sch->exit_kind, &none, kind); +} + +static void scx_disable (struct scx_sched *sch,enum scx_exit_kind kind) +{ +if (scx_claim_exit (sch, kind)) +kthread_queue_work (sch->helper, &sch->disable_work); }static void dump_newline (struct seq_buf *s)
@@ -4412,9 +4417,8 @@ static void scx_vexit(struct scx_sched *sch, const char *fmt, va_list args) {struct scx_exit_info *ei = sch->exit_info; -int none =SCX_EXIT_NONE ; -if (!atomic_try_cmpxchg (&sch->exit_kind, &none, kind)) +if (!scx_claim_exit (sch, kind))return ; ei->exit_code = exit_code;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 23/46 sched_ext: Introduce cgroup sub-sched support
In reply to Tejun HeoA system often runs multiple workloads especially in multi-tenant server environments where a system is split into partitions servicing separate more-or-less independent workloads each requiring an application-specific scheduler. To support such and other use cases, sched_ext is in the process of growing multiple scheduler support.
When partitioning a system in terms of CPUs for such use cases, an oft-taken approach is hard partitioning the system using cpuset. While it would be possible to tie sched_ext multiple scheduler support to cpuset partitions, such an approach would have fundamental limitations stemming from the lack of dynamism and flexibility.
Users often don't care which specific CPUs are assigned to which workload and want to take advantage of optimizations which are enabled by running workloads on a larger machine - e.g. opportunistic over-commit, improving latency critical workload characteristics while maintaining bandwidth fairness, employing control mechanisms based on different criteria than on-CPU time for e.g. flexible memory bandwidth isolation, packing similar parts from different workloads on same L3s to improve cache efficiency, and so on.
As this sort of dynamic behaviors are impossible or difficult to implement with hard partitioning, sched_ext is implementing cgroup sub-sched support where schedulers can be attached to the cgroup hierarchy and a parent scheduler is responsible for controlling the CPUs that each child can use at any given moment. This makes CPU distribution dynamically controlled by BPF allowing high flexibility.
This patch adds the skeletal sched_ext cgroup sub-sched support:
- sched_ext_ops.sub_cgroup_id and .sub_attach/detach() are added. Non-zero sub_cgroup_id indicates that the scheduler is to be attached to the identified cgroup. A sub-sched is attached to the cgroup iff the nearest ancestor scheduler implements .sub_attach() and grants the attachment. Max nesting depth is limited by SCX_SUB_MAX_DEPTH.
- When a scheduler exits, all its descendant schedulers are exited together. Also, cgroup.scx_sched added which points to the effective scheduler instance for the cgroup. This is updated on scheduler init/exit and inherited on cgroup online. When a cgroup is offlined, the attached scheduler is automatically exited.
- Sub-sched support is gated on CONFIG_EXT_SUB_SCHED which is automatically enabled if both SCX and cgroups are enabled. Sub-sched support is not tied to the CPU controller but rather the cgroup hierarchy itself. This is intentional as the support for cpu.weight and cpu.max based resource control is orthogonal to sub-sched support. Note that CONFIG_CGROUPS around cgroup subtree iteration support for scx_task_iter is replaced with CONFIG_EXT_SUB_SCHED for consistency.
- This allows loading sub-scheds and most framework operations such as propagating disable down the hierarchy work. However, sub-scheds are not operational yet and all tasks stay with the root sched. This will serve as the basis for building up full sub-sched support.
- DSQs point to the scx_sched they belong to.
- scx_qmap is updated to allow attachment of sub-scheds and also serving as sub-scheds.
--- include/linux/cgroup-defs.h | 4 + include/linux/sched/ext.h | 3 + init/Kconfig | 4 + kernel/sched/ext.c | 503 +++++++++++++++++++++++++++++++-- kernel/sched/ext_internal.h | 65 ++++- tools/sched_ext/scx_qmap.bpf.c | 9 +- tools/sched_ext/scx_qmap.c | 13 +- 7 files changed, 567 insertions(+), 34 deletions(-)
include/linux/cgroup-defs.h +4 -0
@@ -17,6 +17,7 @@#include <linux/refcount.h> #include <linux/percpu-refcount.h> #include <linux/percpu-rwsem.h> +#include <linux/sched.h> #include <linux/u64_stats_sync.h> #include <linux/workqueue.h> #include <linux/bpf-cgroup-defs.h>
@@ -590,6 +591,9 @@ struct cgroup {#ifdef CONFIG_BPF_SYSCALL struct bpf_local_storage __rcu *bpf_cgrp_storage;#endif +#ifdef CONFIG_EXT_SUB_SCHED +struct scx_sched __rcu *scx_sched; +#endif /* All ancestors including self */ struct cgroup *ancestors[];
include/linux/sched/ext.h +3 -0
@@ -65,6 +65,7 @@ struct scx_dispatch_q {u64 id;struct rhash_head hash_node;struct llist_node free_node; +struct scx_sched *sched;struct rcu_head rcu; };
@@ -136,6 +137,8 @@ struct scx_dsq_list_node {u32 priv;/* can be used by iter cursor */ }; +struct scx_sched ; +/* * The following is embedded in task_struct and contains all fields necessary * for a task to be scheduled by SCX.
init/Kconfig +4 -0
@@ -1137,6 +1137,10 @@ config EXT_GROUP_SCHED endif #CGROUP_SCHED +config EXT_SUB_SCHED + def_bool y + depends on SCHED_CLASS_EXT + config SCHED_MM_CID def_bool y depends on SMP && RSEQ
kernel/sched/ext.c +477 -26
742 lines not on this page · raw message
kernel/sched/ext_internal.h +59 -6
143 lines not on this page · raw message
tools/sched_ext/scx_qmap.bpf.c +8 -1
@@ -41,6 +41,7 @@ const volatile u32 dsp_batch;const volatile bool highpri_boosting;const volatile bool print_dsqs_and_events;const volatile bool print_msgs; +const volatile u64 sub_cgroup_id;const volatile s32 disallow_tgid;const volatile bool suppress_dump;
@@ -848,7 +849,7 @@ s32 BPF_STRUCT_OPS_SLEEPABLE(qmap_init)struct bpf_timer *timer;s32 ret; -if (print_msgs) +if (print_msgs && !sub_cgroup_id)print_cpus (); ret =scx_bpf_create_dsq (SHARED_DSQ ,-1 );
@@ -874,6 +875,11 @@ void BPF_STRUCT_OPS(qmap_exit, struct scx_exit_info *ei)UEI_RECORD (uei, ei); } +s32 BPF_STRUCT_OPS (qmap_sub_attach ,struct scx_sub_attach_args *args) +{ +return 0 ; +} + SCX_OPS_DEFINE(qmap_ops, .select_cpu = (void *)qmap_select_cpu, .enqueue = (void *)qmap_enqueue,
@@ -889,6 +895,7 @@ SCX_OPS_DEFINE(qmap_ops, .cgroup_init = (void *)qmap_cgroup_init, .cgroup_set_weight = (void *)qmap_cgroup_set_weight, .cgroup_set_bandwidth = (void *)qmap_cgroup_set_bandwidth, + .sub_attach = (void *)qmap_sub_attach, .cpu_online = (void *)qmap_cpu_online, .cpu_offline = (void *)qmap_cpu_offline, .init = (void *)qmap_init,
tools/sched_ext/scx_qmap.c +12 -1
@@ -10,6 +10,7 @@#include <inttypes.h> #include <signal.h> #include <libgen.h> +#include <sys/stat.h> #include <bpf/bpf.h> #include <scx/common.h> #include "scx_qmap.bpf.skel.h"
@@ -67,7 +68,7 @@ int main(int argc, char **argv) skel->rodata->slice_ns =__COMPAT_ENUM_OR_ZERO ("scx_public_consts" ,"SCX_SLICE_DFL" ); -while ((opt =getopt (argc, argv,"s:e:t:T:l:b:PMHd:D:Spvh" )) !=-1 ) { +while ((opt =getopt (argc, argv,"s:e:t:T:l:b:PMHc:d:D:Spvh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -96,6 +97,16 @@ int main(int argc, char **argv)case 'H' : skel->rodata->highpri_boosting = true;break ; +case 'c' : { +struct stat st; +if (stat (optarg, &st) <0 ) { +perror ("stat" ); +return 1 ; + } + skel->struct_ops.qmap_ops->sub_cgroup_id = st.st_ino; + skel->rodata->sub_cgroup_id = st.st_ino; +break ; + }case 'd' : skel->rodata->disallow_tgid =strtol (optarg,NULL ,0 );if (skel->rodata->disallow_tgid <0 )
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 24/46 HACK_NOT_FOR_UPSTREAM: BPF: Implement prog grouping hack
In reply to Tejun HeoHopefully, we can have something better instead.
NOT_SIGNED_OFF
--- include/linux/bpf.h | 5 +++++ include/linux/sched.h | 2 ++ kernel/bpf/syscall.c | 23 +++++++++++++++++++++++ kernel/sched/ext.c | 36 ++++++++++++++++++++++++++++++++++++ tools/sched_ext/scx_qmap.c | 13 +++++++++++++ 5 files changed, 79 insertions(+)
include/linux/bpf.h +5 -0
@@ -1581,6 +1581,11 @@ struct bpf_stream_stage {struct bpf_prog_aux {atomic64_t refcnt; + +/* XXX - See kernel/sched/ext.c::scx_sub_enable() */ +u64 priv_user; +void *priv; +u32 used_map_cnt;u32 used_btf_cnt;u32 max_ctx_offset;
include/linux/sched.h +2 -0
@@ -1596,6 +1596,8 @@ struct task_struct {struct bpf_local_storage __rcu *bpf_storage;/* Used for BPF run context */ struct bpf_run_ctx *bpf_ctx; +/* XXX - See kernel/sched/ext.c::scx_sub_enable() */ +u64 bpf_prog_aux_priv;#endif /* Used by BPF for per-TASK xdp storage */ struct bpf_net_context *bpf_net_context;
kernel/bpf/syscall.c +23 -0
@@ -2761,6 +2761,27 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) } } +static int prog_aux_priv_param_set (const char *input,const struct kernel_param *kp) +{ +return kstrtoull (input,0 , ¤t->bpf_prog_aux_priv); +} + +static int prog_aux_priv_param_get (char *buf,const struct kernel_param *kp) +{ +return scnprintf (buf,PAGE_SIZE ,"%llu\n" , current->bpf_prog_aux_priv); +} + +static const struct kernel_param_ops prog_aux_priv_param_ops = { + .set = prog_aux_priv_param_set, + .get = prog_aux_priv_param_get, +}; + +#undef MODULE_PARAM_PREFIX +#define MODULE_PARAM_PREFIX "bpf." +module_param_cb (prog_aux_priv, &prog_aux_priv_param_ops,NULL ,0664 ); +MODULE_PARM_DESC ("prog_aux_priv" , +"Set prog->aux->priv to this value for all BPF programs loaded by %current" ); +/* last field in 'union bpf_attr' used by this command */ #define BPF_PROG_LOAD_LAST_FIELD fd_array_cnt
@@ -2898,6 +2919,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) prog->expected_attach_type = attr->expected_attach_type; prog->sleepable = !!(attr->prog_flags &BPF_F_SLEEPABLE ); +/* XXX - See kernel/sched/ext.c::scx_sub_enable() */ + prog->aux->priv_user = current->bpf_prog_aux_priv; prog->aux->attach_btf = attach_btf; prog->aux->attach_btf_id = attr->attach_btf_id; prog->aux->dst_prog = dst_prog;
kernel/sched/ext.c +36 -0
@@ -4116,6 +4116,24 @@ static void scx_sub_disable(struct scx_sched *sch)if (sch->ops.exit)SCX_CALL_OP (sch,SCX_KF_UNLOCKED , exit,NULL , sch->exit_info); + +/* +* XXX - NULL prog->aux->priv is interpreted as scx_root, so use an +* ERR_PTR value to mark the associated progs dead. Note that this is +* racy as e.g. a tracepoint program associated with a scheduler which +* hasn't finished scx_sub_enable() yet may end up affecting scx_root +* inadvertently. Plug the hole when this hack is replaced with a proper +* BPF construct. +*/ +u32 prog_id =0 ; +struct bpf_prog *prog; +while ((prog =bpf_prog_get_curr_or_next (&prog_id))) { +if (prog->aux->priv == sch) +RCU_INIT_POINTER (prog->aux->priv,ERR_PTR (-ENODEV )); +bpf_prog_put (prog); + prog_id++; + } + kobject_del(&sch->kobj); }#else /* CONFIG_EXT_SUB_SCHED */
@@ -5148,6 +5166,24 @@ static int scx_sub_enable(struct sched_ext_ops *ops, struct bpf_link *link) goto err_disable; } +/* +* XXX - We want all BPF programs loaded together with this scheduler +* instance to point to this scheduler instance. BPF currently doesn't +* have such feature so work around with a hack. The loading userspace +* thread sets %current->bpf_prog_aux_priv to the associated cgroup ID +* which gets transferred to bpf->aux->priv_user in bpf_prog_load(). +* Here, we can find all progs that have the matching cgroup ID and set +* their prog->aux->priv to $sch. +*/ +u32 prog_id =0 ; +struct bpf_prog *prog; +while ((prog =bpf_prog_get_curr_or_next (&prog_id))) { +if (prog->aux->priv_user ==cgroup_id (cgrp)) +rcu_assign_pointer (prog->aux->priv, sch); +bpf_prog_put (prog); + prog_id++; + } +if (sch->ops.init) { ret =SCX_CALL_OP_RET (sch,SCX_KF_UNLOCKED , init,NULL );if (ret) {
tools/sched_ext/scx_qmap.c +13 -0
@@ -99,12 +99,25 @@ int main(int argc, char **argv)break ;case 'c' : {struct stat st; +int fd, len; +char buf[19 ];if (stat (optarg, &st) <0 ) {perror ("stat" );return 1 ; } skel->struct_ops.qmap_ops->sub_cgroup_id = st.st_ino; skel->rodata->sub_cgroup_id = st.st_ino; + fd =open ("/sys/module/bpf/parameters/prog_aux_priv" ,O_RDWR ); +if (fd <0 ) { +perror ("open(\"/sys/module/bpf/parameters/prog_aux_priv\")" ); +return 1 ; + } + len =snprintf (buf,sizeof (buf),"0x%lx" , st.st_ino); +if (write (fd, buf, len) != len) { +perror ("write(\"/sys/module/bpf/parameters/prog_aux_priv\")" ); +return 1 ; + } +close (fd);break ; }case 'd' :
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 25/46 sched_ext: Introduce scx_task_sched[_rcu]()
In reply to Tejun HeoIn preparation of multiple scheduler support, add p->scx.sched which points to the scx_sched instance that the task is scheduled by, which is currently always scx_root. Add scx_task_sched[_rcu]() accessors which return the assocaited scx_sched of the specified task and replace the raw scx_root dereferences with it where applicable.
As scx_root is still the only scheduler, this shouldn't introduce user-visible behavior changes.
--- include/linux/sched/ext.h | 7 ++++ kernel/sched/ext.c | 68 +++++++++++++++++++++++-------------- kernel/sched/ext_internal.h | 42 +++++++++++++++++++++++ 3 files changed, 92 insertions(+), 25 deletions(-)
include/linux/sched/ext.h +7 -0
@@ -144,6 +144,13 @@ struct scx_sched;* for a task to be scheduled by SCX. */ struct sched_ext_entity { +#ifdef CONFIG_CGROUPS +/* +* Associated scx_sched. Updated either during fork or while holding +* both p->pi_lock and rq lock. +*/ +struct scx_sched __rcu *sched; +#endif struct scx_dispatch_q *dsq;struct scx_dsq_list_node dsq_list;/* dispatch order */ struct rb_node dsq_priq;/* p->scx.dsq_vtime order */
kernel/sched/ext.c +43 -25
224 lines not on this page · raw message
kernel/sched/ext_internal.h +42 -0
51 lines not on this page · raw message
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 26/46 sched_ext: Introduce scx_prog_sched()
In reply to Tejun HeoIn preparation of multiple scheduler support, add add scx_prog_sched() accessor which returns the assocaited scx_sched given the magic BPF parameter @aux__prog on kfuncs.
As scx_root is still the only scheduler, this shouldn't introduce user-visible behavior changes.
--- kernel/sched/ext.c | 110 +++++++++++++++++++++++------------- kernel/sched/ext_idle.c | 72 +++++++++++++++-------- kernel/sched/ext_internal.h | 33 +++++++++++ 3 files changed, 154 insertions(+), 61 deletions(-)
kernel/sched/ext.c +72 -38
@@ -5902,6 +5902,7 @@ __bpf_kfunc_start_defs();* @dsq_id: DSQ to insert into * @slice: duration @p can run for in nsecs, 0 to keep the current value * @enq_flags: SCX_ENQ_* +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Insert @p into the FIFO queue of the DSQ identified by @dsq_id. It is safe to * call this function spuriously. Can be called from ops.enqueue(),
@@ -5932,12 +5933,13 @@ __bpf_kfunc_start_defs();* scx_bpf_kick_cpu() to trigger scheduling. */ __bpf_kfunc void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice, -u64 enq_flags) +u64 enq_flags, +const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return ;
@@ -5979,7 +5981,13 @@ __bpf_kfunc void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 dsq_id,struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (scx_root); +/* +* FIXME - This should be scx_prog_sched() so that we can verify that +* the calling scheduler has authority over @p. However, due to BPF +* funcation all parameter count limit, we can't add @aux__prog to this +* function. This can be fixed by packing the parameters into a struct. +*/ + sch =scx_task_sched (p);if (unlikely (!sch))return ;
@@ -6099,16 +6107,17 @@ __bpf_kfunc_start_defs();/** * scx_bpf_dispatch_nr_slots - Return the number of remaining dispatch slots +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Can only be called from ops.dispatch(). */ -__bpf_kfunc u32 scx_bpf_dispatch_nr_slots(void) +__bpf_kfunc u32 scx_bpf_dispatch_nr_slots(const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return 0 ;
@@ -6120,18 +6129,19 @@ __bpf_kfunc u32 scx_bpf_dispatch_nr_slots(void)/** * scx_bpf_dispatch_cancel - Cancel the latest dispatch +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Cancel the latest dispatch. Can be called multiple times to cancel further * dispatches. Can only be called from ops.dispatch(). */ -__bpf_kfunc void scx_bpf_dispatch_cancel(void) +__bpf_kfunc void scx_bpf_dispatch_cancel(const struct bpf_prog_aux *aux__prog) {struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx);struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return ;
@@ -6147,6 +6157,7 @@ __bpf_kfunc void scx_bpf_dispatch_cancel(void)/** * scx_bpf_dsq_move_to_local - move a task from a DSQ to the current CPU's local DSQ * @dsq_id: DSQ to move task from +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Move a task from the non-local DSQ identified by @dsq_id to the current CPU's * local DSQ for execution. Can only be called from ops.dispatch().
@@ -6158,7 +6169,8 @@ __bpf_kfunc void scx_bpf_dispatch_cancel(void)* Returns %true if a task has been moved, %false if there isn't any task to * move. */ -__bpf_kfunc bool scx_bpf_dsq_move_to_local(u64 dsq_id) +__bpf_kfunc bool scx_bpf_dsq_move_to_local(u64 dsq_id, +const struct bpf_prog_aux *aux__prog) {struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx);struct scx_dispatch_q *dsq;
@@ -6166,7 +6178,7 @@ __bpf_kfunc bool scx_bpf_dsq_move_to_local(u64 dsq_id)guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return false;
@@ -6310,12 +6322,13 @@ __bpf_kfunc_start_defs();/** * scx_bpf_reenqueue_local - Re-enqueue tasks on a local DSQ +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Iterate over all of the tasks currently enqueued on the local DSQ of the * caller's CPU, and re-enqueue them in the BPF scheduler. Returns the number of * processed tasks. Can only be called from ops.cpu_release(). */ -__bpf_kfunc u32 scx_bpf_reenqueue_local(void) +__bpf_kfunc u32 scx_bpf_reenqueue_local(const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;LIST_HEAD (tasks);
@@ -6324,7 +6337,7 @@ __bpf_kfunc u32 scx_bpf_reenqueue_local(void)struct task_struct *p, *n;guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return 0 ;
@@ -6389,11 +6402,13 @@ __bpf_kfunc_start_defs();* scx_bpf_create_dsq - Create a custom DSQ * @dsq_id: DSQ to create * @node: NUMA node to allocate from +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Create a custom DSQ identified by @dsq_id. Can be called from any sleepable * scx callback, and any BPF_PROG_TYPE_SYSCALL prog. */ -__bpf_kfunc s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) +__bpf_kfunc s32 scx_bpf_create_dsq(u64 dsq_id, s32 node, +const struct bpf_prog_aux *aux__prog) {struct scx_dispatch_q *dsq;struct scx_sched *sch;
@@ -6412,7 +6427,7 @@ __bpf_kfunc s32 scx_bpf_create_dsq(u64 dsq_id, s32 node)rcu_read_lock (); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (sch) {init_dsq (dsq, dsq_id, sch); ret = rhashtable_lookup_insert_fast(&sch->dsq_hash, &dsq->hash_node,
@@ -6501,18 +6516,20 @@ static void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags)* scx_bpf_kick_cpu - Trigger reschedule on a CPU * @cpu: cpu to kick * @flags: %SCX_KICK_* flags +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Kick @cpu into rescheduling. This can be used to wake up an idle CPU or * trigger rescheduling on a busy CPU. This can be called from any online * scx_ops operation and the actual kicking is performed asynchronously through * an irq work. */ -__bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags) +__bpf_kfunc void scx_bpf_kick_cpu(s32 cpu, u64 flags, +const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (likely (sch))scx_kick_cpu (sch, cpu, flags); }
@@ -6592,10 +6609,11 @@ __bpf_kfunc void scx_bpf_destroy_dsq(u64 dsq_id)* tasks which are already queued when this function is invoked. */ __bpf_kfunc int bpf_iter_scx_dsq_new(struct bpf_iter_scx_dsq *it, u64 dsq_id, -u64 flags) +u64 flags, +const struct bpf_prog_aux *aux__prog) {struct bpf_iter_scx_dsq_kern *kit = (void *)it; -struct scx_sched *sch; +struct scx_sched *sch =scx_prog_sched (aux__prog);BUILD_BUG_ON (sizeof (struct bpf_iter_scx_dsq_kern ) >sizeof (struct bpf_iter_scx_dsq ));
@@ -6608,7 +6626,6 @@ __bpf_kfunc int bpf_iter_scx_dsq_new(struct bpf_iter_scx_dsq *it, u64 dsq_id,*/ kit->dsq =NULL ; - sch =rcu_dereference_check (scx_root,rcu_read_lock_bh_held ());if (unlikely (!sch))return -ENODEV ;
@@ -6749,18 +6766,20 @@ __bpf_kfunc_start_defs();* @fmt: error message format string * @data: format string parameters packaged using ___bpf_fill() macro * @data__sz: @data len, must end in '__sz' for the verifier +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Indicate that the BPF scheduler wants to exit gracefully, and initiate ops * disabling. */ __bpf_kfunc void scx_bpf_exit_bstr(s64 exit_code, char *fmt, -unsigned long long *data,u32 data__sz) +unsigned long long *data,u32 data__sz, +const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;unsigned long flags;raw_spin_lock_irqsave (&scx_exit_bstr_buf_lock, flags); - sch =rcu_dereference_bh (scx_root); + sch =scx_prog_sched (aux__prog);if (likely (sch) &&bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 )scx_exit (sch,SCX_EXIT_UNREG_BPF , exit_code,"%s" , scx_exit_bstr_buf.line);
@@ -6772,18 +6791,20 @@ __bpf_kfunc void scx_bpf_exit_bstr(s64 exit_code, char *fmt,* @fmt: error message format string * @data: format string parameters packaged using ___bpf_fill() macro * @data__sz: @data len, must end in '__sz' for the verifier +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Indicate that the BPF scheduler encountered a fatal error and initiate ops * disabling. */ __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data, -u32 data__sz) +u32 data__sz, +const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;unsigned long flags;raw_spin_lock_irqsave (&scx_exit_bstr_buf_lock, flags); - sch =rcu_dereference_bh (scx_root); + sch =scx_prog_sched (aux__prog);if (likely (sch) &&bstr_format (sch, &scx_exit_bstr_buf, fmt, data, data__sz) >=0 )scx_exit (sch,SCX_EXIT_ERROR_BPF ,0 ,"%s" , scx_exit_bstr_buf.line);
@@ -6795,6 +6816,7 @@ __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data,* @fmt: format string * @data: format string parameters packaged using ___bpf_fill() macro * @data__sz: @data len, must end in '__sz' for the verifier +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * To be called through scx_bpf_dump() helper from ops.dump(), dump_cpu() and * dump_task() to generate extra debug dump specific to the BPF scheduler.
@@ -6803,7 +6825,8 @@ __bpf_kfunc void scx_bpf_error_bstr(char *fmt, unsigned long long *data,* multiple calls. The last line is automatically terminated. */ __bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data, -u32 data__sz) +u32 data__sz, +const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;struct scx_dump_data *dd = &scx_dump_data;
@@ -6812,7 +6835,7 @@ __bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data,guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return ;
@@ -6851,18 +6874,19 @@ __bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data,/** * scx_bpf_cpuperf_cap - Query the maximum relative capacity of a CPU * @cpu: CPU of interest +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Return the maximum relative capacity of @cpu in relation to the most * performant CPU in the system. The return value is in the range [1, * %SCX_CPUPERF_ONE]. See scx_bpf_cpuperf_cur(). */ -__bpf_kfunc u32 scx_bpf_cpuperf_cap(s32 cpu) +__bpf_kfunc u32 scx_bpf_cpuperf_cap(s32 cpu, const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (likely (sch) &&ops_cpu_valid (sch, cpu,NULL ))return arch_scale_cpu_capacity (cpu);else
@@ -6872,6 +6896,7 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cap(s32 cpu)/** * scx_bpf_cpuperf_cur - Query the current relative performance of a CPU * @cpu: CPU of interest +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Return the current relative performance of @cpu in relation to its maximum. * The return value is in the range [1, %SCX_CPUPERF_ONE].
@@ -6883,13 +6908,13 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cap(s32 cpu)* * The result is in the range [1, %SCX_CPUPERF_ONE]. */ -__bpf_kfunc u32 scx_bpf_cpuperf_cur(s32 cpu) +__bpf_kfunc u32 scx_bpf_cpuperf_cur(s32 cpu, const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (likely (sch) &&ops_cpu_valid (sch, cpu,NULL ))return arch_scale_freq_capacity (cpu);else
@@ -6900,6 +6925,7 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cur(s32 cpu)* scx_bpf_cpuperf_set - Set the relative performance target of a CPU * @cpu: CPU of interest * @perf: target performance level [0, %SCX_CPUPERF_ONE] +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Set the target performance level of @cpu to @perf. @perf is in linear * relative scale between 0 and %SCX_CPUPERF_ONE. This determines how the
@@ -6910,13 +6936,14 @@ __bpf_kfunc u32 scx_bpf_cpuperf_cur(s32 cpu)* use. Consult hardware and cpufreq documentation for more information. The * current performance level can be monitored using scx_bpf_cpuperf_cur(). */ -__bpf_kfunc void scx_bpf_cpuperf_set(s32 cpu, u32 perf) +__bpf_kfunc void scx_bpf_cpuperf_set(s32 cpu, u32 perf, +const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (sch); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return ;
@@ -7026,14 +7053,16 @@ __bpf_kfunc s32 scx_bpf_task_cpu(const struct task_struct *p)/** * scx_bpf_cpu_rq - Fetch the rq of a CPU * @cpu: CPU of the rq +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs */ -__bpf_kfunc struct rq *scx_bpf_cpu_rq(s32 cpu) +__bpf_kfunc struct rq *scx_bpf_cpu_rq(s32 cpu, +const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return NULL ;
@@ -7052,18 +7081,19 @@ __bpf_kfunc struct rq *scx_bpf_cpu_rq(s32 cpu)/** * scx_bpf_locked_rq - Return the rq currently locked by SCX +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Returns the rq if a rq lock is currently held by SCX. * Otherwise emits an error and returns NULL. */ -__bpf_kfunc struct rq *scx_bpf_locked_rq(void) +__bpf_kfunc struct rq *scx_bpf_locked_rq(const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;struct rq *rq;guard (preempt)(); - sch =rcu_dereference_sched (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return NULL ;
@@ -7079,16 +7109,18 @@ __bpf_kfunc struct rq *scx_bpf_locked_rq(void)/** * scx_bpf_cpu_curr - Return remote CPU's curr task * @cpu: CPU of interest +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * Callers must hold RCU read lock (KF_RCU). */ -__bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu) +__bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu, +const struct bpf_prog_aux *aux__prog) {struct scx_sched *sch;guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch))return NULL ;
@@ -7101,6 +7133,7 @@ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu)/** * scx_bpf_task_cgroup - Return the sched cgroup of a task * @p: task of interest +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs * * @p->sched_task_group->css.cgroup represents the cgroup @p is associated with * from the scheduler's POV. SCX operations should use this function to
@@ -7110,7 +7143,8 @@ __bpf_kfunc struct task_struct *scx_bpf_cpu_curr(s32 cpu)* operations. The restriction guarantees that @p's rq is locked by the caller. */ #ifdef CONFIG_CGROUP_SCHED -__bpf_kfunc struct cgroup *scx_bpf_task_cgroup(struct task_struct *p) +__bpf_kfunc struct cgroup *scx_bpf_task_cgroup(struct task_struct *p, +const struct bpf_prog_aux *aux__prog) {struct task_group *tg = p->sched_task_group;struct cgroup *cgrp = &cgrp_dfl_root.cgrp;
@@ -7118,7 +7152,7 @@ __bpf_kfunc struct cgroup *scx_bpf_task_cgroup(struct task_struct *p)guard (rcu)(); - sch =rcu_dereference (scx_root); + sch =scx_prog_sched (aux__prog);if (unlikely (!sch)) goto out;
kernel/sched/ext_idle.c +49 -23
257 lines not on this page · raw message
kernel/sched/ext_internal.h +33 -0
@@ -1157,6 +1157,32 @@ static inline struct scx_sched *scx_task_sched_rcu(const struct task_struct *p)rcu_read_lock_bh_held () || rcu_read_lock_sched_held()); } + +/** +* scx_prog_sched - Find scx_sched associated with a BPF prog +* @aux: aux__prog passed in from BPF to a kfunc +* +* To be called from kfuncs. Return the scheduler instance associated with the +* BPF program given the special kfunc argument aux__prog. The returned +* scx_sched is RCU protected. +*/ +static struct scx_sched *scx_prog_sched (const struct bpf_prog_aux *aux) +{ +struct scx_sched *sch; + + sch =rcu_dereference_check (aux->priv, +rcu_read_lock_bh_held () || +rcu_read_lock_sched_held ()); +if (unlikely (IS_ERR (sch))) +return NULL ; + +if (sch) +return sch; + +return rcu_dereference_check (scx_root, +rcu_read_lock_bh_held () || +rcu_read_lock_sched_held ()); +}#else /* CONFIG_EXT_SUB_SCHED */ static inline struct scx_sched *scx_task_sched (const struct task_struct *p) {
@@ -1170,4 +1196,11 @@ static inline struct scx_sched *scx_task_sched_rcu(const struct task_struct *p)rcu_read_lock_bh_held () || rcu_read_lock_sched_held()); } + +static struct scx_sched *scx_prog_sched (const struct bpf_prog_aux *aux) +{ +return rcu_dereference_check (scx_root, +rcu_read_lock_bh_held () || +rcu_read_lock_sched_held ()); +}#endif /* CONFIG_EXT_SUB_SCHED */
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 27/46 sched_ext: Ignore insertions of not-owned tasks into DSQs
In reply to Tejun HeoAs BPF schedulers are allowed to ignore dequeues, after a sub-sched enabling moved tasks out of the parent scheduler, the parent scheduler may try to dispatch an already moved task. As this doesn't necessarily indicate a malfunction, ignore and count such attempts.
--- kernel/sched/ext.c | 9 +++++++++ kernel/sched/ext_internal.h | 12 ++++++++++++ 2 files changed, 21 insertions(+)
kernel/sched/ext.c +9 -0
@@ -2125,6 +2125,12 @@ static void finish_dispatch(struct scx_sched *sch, struct rq *rq,if ((opss &SCX_OPSS_QSEQ_MASK ) != qseq_at_dispatch)return ; +/* see SCX_EV_INSERT_NOT_OWNED definition */ +if (unlikely (sch !=rcu_access_pointer (p->scx.sched))) { +__scx_add_event (sch,SCX_EV_INSERT_NOT_OWNED ,1 ); +return ; + } +/* * While we know @p is accessible, we don't yet have a claim on * it - the BPF scheduler is allowed to dispatch tasks
@@ -3739,6 +3745,7 @@ static ssize_t scx_attr_events_show(struct kobject *kobj, at +=scx_attr_event_show (buf, at, &events,SCX_EV_BYPASS_DURATION ); at +=scx_attr_event_show (buf, at, &events,SCX_EV_BYPASS_DISPATCH ); at +=scx_attr_event_show (buf, at, &events,SCX_EV_BYPASS_ACTIVATE ); + at +=scx_attr_event_show (buf, at, &events,SCX_EV_INSERT_NOT_OWNED );return at; }SCX_ATTR (events);
@@ -4631,6 +4638,7 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)scx_dump_event (s, &events,SCX_EV_BYPASS_DURATION );scx_dump_event (s, &events,SCX_EV_BYPASS_DISPATCH );scx_dump_event (s, &events,SCX_EV_BYPASS_ACTIVATE ); +scx_dump_event (s, &events,SCX_EV_INSERT_NOT_OWNED );if (seq_buf_has_overflowed (&s) && dump_len >=sizeof (trunc_marker)) memcpy(ei->dump + dump_len - sizeof(trunc_marker),
@@ -7247,6 +7255,7 @@ static void scx_read_events(struct scx_sched *sch, struct scx_event_stats *eventscx_agg_event (events, e_cpu,SCX_EV_BYPASS_DURATION );scx_agg_event (events, e_cpu,SCX_EV_BYPASS_DISPATCH );scx_agg_event (events, e_cpu,SCX_EV_BYPASS_ACTIVATE ); +scx_agg_event (events, e_cpu,SCX_EV_INSERT_NOT_OWNED ); } }
kernel/sched/ext_internal.h +12 -0
@@ -895,6 +895,18 @@ struct scx_event_stats {* The number of times the bypassing mode has been activated. */ s64 SCX_EV_BYPASS_ACTIVATE ; + +/* +* The number of times the scheduler attempted to insert a task that it +* doesn't own into a DSQ. Such attempts are ignored. +* +* As BPF schedulers are allowed to ignore dequeues, it's difficult to +* tell whether such an attempt is from a scheduler malfunction or an +* ignored dequeue around sub-sched enabling. If this count keeps going +* up regardless of sub-sched enabling, it likely indicates a bug in the +* scheduler. +*/ +s64 SCX_EV_INSERT_NOT_OWNED ; };struct scx_sched_pcpu {
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 28/46 sched_ext: scx_dsq_move() should validate the task belongs to the right scheduler
In reply to Tejun Heoscx_bpf_dsq_move[_vtime]() calls scx_dsq_move() to move task from a DSQ to another. However, @p doesn't necessarily have to come form the containing iteration and can thus be a task which belongs to another scx_sched. Verify that @p is on the same scx_sched as the DSQ being iterated.
--- kernel/sched/ext.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/sched/ext.c +6 -1
@@ -6027,8 +6027,8 @@ static const struct btf_kfunc_id_set scx_kfunc_set_enqueue_dispatch = {static bool scx_dsq_move (struct bpf_iter_scx_dsq_kern *kit,struct task_struct *p,u64 dsq_id,u64 enq_flags) { -struct scx_sched *sch = scx_root;struct scx_dispatch_q *src_dsq = kit->dsq, *dst_dsq; +struct scx_sched *sch = src_dsq->sched;struct rq *this_rq, *src_rq, *locked_rq;bool dispatched = false;bool in_balance;
@@ -6038,6 +6038,11 @@ static bool scx_dsq_move(struct bpf_iter_scx_dsq_kern *kit, !scx_kf_allowed(sch, SCX_KF_DISPATCH))return false; +if (unlikely (sch !=scx_task_sched (p))) { +scx_error (sch,"scx_bpf_dsq_move[_vtime]() on %s[%d] but the task belongs to a different scheduler" , + p->comm, p->pid); + } +/* * Can be called from either ops.dispatch() locking this_rq() or any * context where no rq lock is held. If latter, lock @p's task_rq which
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 29/46 sched_ext: Refactor task init/exit helpers
In reply to Tejun Heo- Add the @sch parameter to scx_init_task() and drop @tg as it can be obtained from @p. Separate out __scx_init_task() which does everything except for the task state transition.
- Add the @sch parameter to scx_enable_task(). Separate out __scx_enable_task() which does everything except for the task state transition.
- Add the @sch parameter to scx_disable_task().
- Rename scx_exit_task() to scx_disable_and_exit_task() and separate out __scx_disable_and_exit_task() which does everything except for the task state transition.
While some task state transitions are relocated, no meaningful behavior changes are expected.
--- kernel/sched/ext.c | 60 ++++++++++++++++++++++++++++++---------------- 1 file changed, 39 insertions(+), 21 deletions(-)
kernel/sched/ext.c +39 -21
@@ -2901,9 +2901,9 @@ static void scx_set_task_state(struct task_struct *p, enum scx_task_state state) p->scx.flags |= state <<SCX_TASK_STATE_SHIFT ; } -static int scx_init_task (struct task_struct *p,struct task_group *tg,bool fork) +static int __scx_init_task (struct scx_sched *sch,struct task_struct *p,bool fork) { -struct scx_sched *sch = scx_root; +struct task_group *tg =task_group (p);int ret; p->scx.disallow = false;
@@ -2922,8 +2922,6 @@ static int scx_init_task(struct task_struct *p, struct task_group *tg, bool fork } } -scx_set_task_state (p,SCX_TASK_INIT ); -if (p->scx.disallow) {if (unlikely (scx_parent (sch))) { scx_error(sch, "non-root ops.init_task() set task->scx.disallow for %s[%d]",
@@ -2957,9 +2955,18 @@ static int scx_init_task(struct task_struct *p, struct task_group *tg, bool forkreturn 0 ; } -static void scx_enable_task (struct task_struct *p) +static int scx_init_task (struct scx_sched *sch,struct task_struct *p,bool fork) +{ +int ret; + + ret =__scx_init_task (sch, p, fork); +if (!ret) +scx_set_task_state (p,SCX_TASK_INIT ); +return ret; +} + +static void __scx_enable_task (struct scx_sched *sch,struct task_struct *p) { -struct scx_sched *sch = scx_root;struct rq *rq =task_rq (p);u32 weight;
@@ -2978,16 +2985,20 @@ static void scx_enable_task(struct task_struct *p)if (SCX_HAS_OP (sch, enable))SCX_CALL_OP_TASK (sch,SCX_KF_REST , enable, rq, p); -scx_set_task_state (p,SCX_TASK_ENABLED );if (SCX_HAS_OP (sch, set_weight))SCX_CALL_OP_TASK (sch,SCX_KF_REST , set_weight, rq, p, p->scx.weight); } -static void scx_disable_task (struct task_struct *p) +static void scx_enable_task (struct scx_sched *sch,struct task_struct *p) +{ +__scx_enable_task (sch, p); +scx_set_task_state (p,SCX_TASK_ENABLED ); +} + +static void scx_disable_task (struct scx_sched *sch,struct task_struct *p) { -struct scx_sched *sch = scx_root;struct rq *rq =task_rq (p);lockdep_assert_rq_held (rq);
@@ -2998,9 +3009,9 @@ static void scx_disable_task(struct task_struct *p)scx_set_task_state (p,SCX_TASK_READY ); } -static void scx_exit_task (struct task_struct *p) +static void __scx_disable_and_exit_task (struct scx_sched *sch, +struct task_struct *p) { -struct scx_sched *sch =scx_task_sched (p);struct scx_exit_task_args args = { .cancelled = false, };
@@ -3017,7 +3028,7 @@ static void scx_exit_task(struct task_struct *p)case SCX_TASK_READY :break ;case SCX_TASK_ENABLED : -scx_disable_task (p); +scx_disable_task (sch, p);break ;default :WARN_ON_ONCE (true);
@@ -3027,6 +3038,13 @@ static void scx_exit_task(struct task_struct *p)if (SCX_HAS_OP (sch, exit_task))SCX_CALL_OP_TASK (sch,SCX_KF_REST , exit_task,task_rq (p), p, &args); +} + +static void scx_disable_and_exit_task (struct scx_sched *sch, +struct task_struct *p) +{ +__scx_disable_and_exit_task (sch, p); +scx_set_task_sched (p,NULL );scx_set_task_state (p,SCX_TASK_NONE ); }
@@ -3062,7 +3080,7 @@ int scx_fork(struct task_struct *p, struct kernel_clone_args *kargs)percpu_rwsem_assert_held (&scx_fork_rwsem);if (scx_init_task_enabled) { - ret =scx_init_task (p,task_group (p), true); + ret =scx_init_task (scx_root, p, true);if (!ret)scx_set_task_sched (p, scx_root);return ret;
@@ -3086,7 +3104,7 @@ void scx_post_fork(struct task_struct *p)struct rq *rq; rq =task_rq_lock (p, &rf); -scx_enable_task (p); +scx_enable_task (scx_task_sched (p), p);task_rq_unlock (rq, p, &rf); } }
@@ -3106,7 +3124,7 @@ void scx_cancel_fork(struct task_struct *p) rq =task_rq_lock (p, &rf);WARN_ON_ONCE (scx_get_task_state (p) >=SCX_TASK_READY ); -scx_exit_task (p); +scx_disable_and_exit_task (scx_task_sched (p), p);task_rq_unlock (rq, p, &rf); }
@@ -3137,7 +3155,7 @@ void sched_ext_free(struct task_struct *p)struct rq *rq; rq =task_rq_lock (p, &rf); -scx_exit_task (p); +scx_disable_and_exit_task (scx_task_sched (p), p);task_rq_unlock (rq, p, &rf); } }
@@ -3163,7 +3181,7 @@ static void switching_to_scx(struct rq *rq, struct task_struct *p) {struct scx_sched *sch =scx_task_sched (p); -scx_enable_task (p); +scx_enable_task (sch, p);/* * set_cpus_allowed_scx() is not called while @p is associated with a
@@ -3176,7 +3194,7 @@ static void switching_to_scx(struct rq *rq, struct task_struct *p)static void switched_from_scx (struct rq *rq,struct task_struct *p) { -scx_disable_task (p); +scx_disable_task (scx_task_sched (p), p); }static void wakeup_preempt_scx (struct rq *rq,struct task_struct *p,int wake_flags) {}
@@ -4201,7 +4219,7 @@ static void scx_root_disable(struct scx_sched *sch)/* * Shut down cgroup support before tasks so that the cgroup attach path -* doesn't race against scx_exit_task(). +* doesn't race against scx_disable_and_exit_task(). */ scx_cgroup_lock ();scx_cgroup_exit (sch);
@@ -4234,7 +4252,7 @@ static void scx_root_disable(struct scx_sched *sch)sched_enq_and_set_task (&ctx);check_class_changed (task_rq (p), p, old_class, p->prio); -scx_exit_task (p); +scx_disable_and_exit_task (scx_task_sched (p), p); }scx_task_iter_stop (&sti);
@@ -5022,7 +5040,7 @@ static int scx_root_enable(struct sched_ext_ops *ops, struct bpf_link *link)scx_task_iter_unlock (&sti); - ret =scx_init_task (p,task_group (p), false); + ret =scx_init_task (sch, p, false);if (ret) {put_task_struct (p);scx_task_iter_stop (&sti);
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 30/46 sched_ext: Make scx_prio_less() handle multiple schedulers
In reply to Tejun HeoCall ops.core_sched_before() iff both tasks belong to the same scx_sched. Otherwise, use timestamp based ordering.
--- kernel/sched/ext.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/ext.c +4 -3
@@ -2602,16 +2602,17 @@ static struct task_struct *pick_task_scx(struct rq *rq)bool scx_prio_less (const struct task_struct *a,const struct task_struct *b,bool in_fi) { -struct scx_sched *sch = scx_root; +struct scx_sched *sch_a =scx_task_sched (a); +struct scx_sched *sch_b =scx_task_sched (b);/* * The const qualifiers are dropped from task_struct pointers when * calling ops.core_sched_before(). Accesses are controlled by the * verifier. */ -if (SCX_HAS_OP (sch, core_sched_before) && +if (sch_a == sch_b &&SCX_HAS_OP (sch_a, core_sched_before) && !scx_rq_bypassing (task_rq (a))) -return SCX_CALL_OP_2TASKS_RET (sch,SCX_KF_REST , core_sched_before, +return SCX_CALL_OP_2TASKS_RET (sch_a,SCX_KF_REST , core_sched_before,NULL , (struct task_struct *)a, (struct task_struct *)b);
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 31/46 sched_ext: Move bypass_depth into scx_sched
In reply to Tejun HeoIn preparation of multiple scheduler support, make bypass state per-scx_sched:
- Replace global scx_bypass_depth with scx_sched_pcpu->bypass_depth.
- SCX_RQ_BYPASSING is replaced with SCX_SCHED_PCPU_BYPASSING which is stored in newly added scx_sched_pcpu->flags.
- Add @sch to scx_bypass().
- Rename scx_rq_bypassing() to scx_bypassing() and replace @rq with @cpu as the state is now in scx_sched_pcpu.
No behavior changes expected.
--- kernel/sched/ext.c | 100 ++++++++++++++++++++---------------- kernel/sched/ext_idle.c | 3 +- kernel/sched/ext_internal.h | 12 ++++- kernel/sched/sched.h | 1 - 4 files changed, 67 insertions(+), 49 deletions(-)
kernel/sched/ext.c +55 -45
@@ -51,7 +51,6 @@ DEFINE_STATIC_PERCPU_RWSEM(scx_fork_rwsem);static atomic_t scx_enable_state_var =ATOMIC_INIT (SCX_DISABLED );static unsigned long scx_in_softlockup;static atomic_t scx_breather_depth =ATOMIC_INIT (0 ); -static int scx_bypass_depth;static bool scx_init_task_enabled;static bool scx_switching_all;DEFINE_STATIC_KEY_FALSE (__scx_switched_all);
@@ -1382,7 +1381,7 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,if (!scx_rq_online (rq)) goto local; -if (scx_rq_bypassing (rq)) { +if (scx_bypassing (sch,cpu_of (rq))) {__scx_add_event (sch,SCX_EV_BYPASS_DISPATCH ,1 ); goto global; }
@@ -2218,7 +2217,8 @@ static int balance_one(struct rq *rq, struct task_struct *prev)* See scx_disable_workfn() for the explanation on the bypassing * test. */ -if (prev_on_rq && prev->scx.slice && !scx_rq_bypassing (rq)) { +if (prev_on_rq && prev->scx.slice && + !scx_bypassing (sch,cpu_of (rq))) { rq->scx.flags |=SCX_RQ_BAL_KEEP ; goto has_tasks; }
@@ -2232,7 +2232,7 @@ static int balance_one(struct rq *rq, struct task_struct *prev) goto has_tasks;if (unlikely (!SCX_HAS_OP (sch, dispatch)) || -scx_rq_bypassing (rq) || !scx_rq_online (rq)) +scx_bypassing (sch,cpu_of (rq)) || !scx_rq_online (rq)) goto no_tasks; dspc->rq = rq;
@@ -2282,7 +2282,8 @@ static int balance_one(struct rq *rq, struct task_struct *prev)* %SCX_OPS_ENQ_LAST is in effect. */ if (prev_on_rq && - (!(sch->ops.flags &SCX_OPS_ENQ_LAST ) ||scx_rq_bypassing (rq))) { + (!(sch->ops.flags &SCX_OPS_ENQ_LAST ) || +scx_bypassing (sch,cpu_of (rq)))) { rq->scx.flags |=SCX_RQ_BAL_KEEP ;__scx_add_event (sch,SCX_EV_DISPATCH_KEEP_LAST ,1 ); goto has_tasks;
@@ -2477,7 +2478,7 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,* forcing a different task. Leave it at the head of the local * DSQ. */ -if (p->scx.slice && !scx_rq_bypassing (rq)) { +if (p->scx.slice && !scx_bypassing (sch,cpu_of (rq))) {dispatch_enqueue (sch, &rq->scx.local_dsq, p,SCX_ENQ_HEAD ); goto switch_class;
@@ -2568,7 +2569,8 @@ static struct task_struct *pick_task_scx(struct rq *rq)if (unlikely (!p->scx.slice)) {struct scx_sched *sch =scx_task_sched (p); -if (!scx_rq_bypassing (rq) && !sch->warned_zero_slice) { +if (!scx_bypassing (sch,cpu_of (rq)) && + !sch->warned_zero_slice) {printk_deferred (KERN_WARNING "sched_ext: %s[%d] has zero slice in %s()\n" , p->comm, p->pid, __func__); sch->warned_zero_slice = true;
@@ -2611,7 +2613,7 @@ bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,* verifier. */ if (sch_a == sch_b &&SCX_HAS_OP (sch_a, core_sched_before) && - !scx_rq_bypassing (task_rq (a))) + !scx_bypassing (sch_a,task_cpu (a))) return SCX_CALL_OP_2TASKS_RET(sch_a, SCX_KF_REST, core_sched_before, NULL, (struct task_struct *)a,
@@ -2624,7 +2626,7 @@ bool scx_prio_less(const struct task_struct *a, const struct task_struct *b,static int select_task_rq_scx (struct task_struct *p,int prev_cpu,int wake_flags) {struct scx_sched *sch =scx_task_sched (p); -bool rq_bypass; +bool bypassing;/* * sched_exec() calls with %WF_EXEC when @p is about to exec(2) as it
@@ -2639,8 +2641,8 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flagif (unlikely (wake_flags &WF_EXEC ))return prev_cpu; - rq_bypass =scx_rq_bypassing (task_rq (p)); -if (likely (SCX_HAS_OP (sch, select_cpu)) && !rq_bypass) { + bypassing =scx_bypassing (sch,task_cpu (p)); +if (likely (SCX_HAS_OP (sch, select_cpu)) && !bypassing) {s32 cpu;struct task_struct **ddsp_taskp;
@@ -2670,7 +2672,7 @@ static int select_task_rq_scx(struct task_struct *p, int prev_cpu, int wake_flag } p->scx.selected_cpu = cpu; -if (rq_bypass) +if (bypassing)__scx_add_event (sch,SCX_EV_BYPASS_DISPATCH ,1 );return cpu; }
@@ -2833,7 +2835,7 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)* While disabling, always resched and refresh core-sched timestamp as * we can't trust the slice management or ops.core_sched_before(). */ -if (scx_rq_bypassing (rq)) { +if (scx_bypassing (sch,cpu_of (rq))) { curr->scx.slice =0 ;touch_core_sched (rq, curr); }else if (SCX_HAS_OP (sch, tick)) {
@@ -3217,13 +3219,14 @@ int scx_check_setscheduler(struct task_struct *p, int policy)bool scx_can_stop_tick (struct rq *rq) {struct task_struct *p = rq->curr; - -if (scx_rq_bypassing (rq)) -return false; +struct scx_sched *sch =scx_task_sched (p);if (p->sched_class != &ext_sched_class)return true; +if (scx_bypassing (sch,cpu_of (rq))) +return false; +/* * @rq can dispatch from different DSQs, so we can't tell whether it * needs the tick or not by looking at nr_running. Allow stopping ticks
@@ -3942,40 +3945,37 @@ static void scx_clear_softlockup(void)* * - scx_prio_less() reverts to the default core_sched_at order. */ -static void scx_bypass (bool bypass) +static void scx_bypass (struct scx_sched *sch,bool bypass) {static DEFINE_RAW_SPINLOCK (bypass_lock);static unsigned long bypass_timestamp; -struct scx_sched *sch;unsigned long flags;int cpu;raw_spin_lock_irqsave (&bypass_lock, flags); - sch =rcu_dereference_bh (scx_root);if (bypass) { - scx_bypass_depth++; -WARN_ON_ONCE (scx_bypass_depth <=0 ); -if (scx_bypass_depth !=1 ) + sch->bypass_depth++; +WARN_ON_ONCE (sch->bypass_depth <=0 ); +if (sch->bypass_depth !=1 ) goto unlock; bypass_timestamp =ktime_get_ns (); -if (sch) -scx_add_event (sch,SCX_EV_BYPASS_ACTIVATE ,1 ); +scx_add_event (sch,SCX_EV_BYPASS_ACTIVATE ,1 ); }else { - scx_bypass_depth--; -WARN_ON_ONCE (scx_bypass_depth <0 ); -if (scx_bypass_depth !=0 ) + sch->bypass_depth--; +WARN_ON_ONCE (sch->bypass_depth <0 ); +if (sch->bypass_depth !=0 ) goto unlock; -if (sch) -scx_add_event (sch,SCX_EV_BYPASS_DURATION , -ktime_get_ns () - bypass_timestamp); +scx_add_event (sch,SCX_EV_BYPASS_DURATION , +ktime_get_ns () - bypass_timestamp); } -atomic_inc (&scx_breather_depth); +if (!scx_parent (sch)) +atomic_inc (&scx_breather_depth);/* * No task property is changing. We just need to make sure all currently -* queued tasks are re-queued according to the new scx_rq_bypassing() +* queued tasks are re-queued according to the new scx_bypassing() * state. As an optimization, walk each rq's runnable_list instead of * the scx_tasks list. *
@@ -3984,22 +3984,23 @@ static void scx_bypass(bool bypass)*/ for_each_possible_cpu (cpu) {struct rq *rq =cpu_rq (cpu); +struct scx_sched_pcpu *pcpu =per_cpu_ptr (sch->pcpu, cpu);struct task_struct *p, *n;raw_spin_rq_lock (rq);if (bypass) { -WARN_ON_ONCE (rq->scx.flags &SCX_RQ_BYPASSING ); - rq->scx.flags |=SCX_RQ_BYPASSING ; +WARN_ON_ONCE (pcpu->flags &SCX_SCHED_PCPU_BYPASSING ); + pcpu->flags |=SCX_SCHED_PCPU_BYPASSING ; }else { -WARN_ON_ONCE (!(rq->scx.flags &SCX_RQ_BYPASSING )); - rq->scx.flags &= ~SCX_RQ_BYPASSING ; +WARN_ON_ONCE (!(pcpu->flags &SCX_SCHED_PCPU_BYPASSING )); + pcpu->flags &= ~SCX_SCHED_PCPU_BYPASSING ; }/* * We need to guarantee that no tasks are on the BPF scheduler * while bypassing. Either we see enabled or the enable path -* sees scx_rq_bypassing() before moving tasks to SCX. +* sees scx_bypassing() before moving tasks to SCX. */ if (!scx_enabled ()) {raw_spin_rq_unlock (rq);
@@ -4029,7 +4030,8 @@ static void scx_bypass(bool bypass)raw_spin_rq_unlock (rq); } -atomic_dec (&scx_breather_depth); +if (!scx_parent (sch)) +atomic_dec (&scx_breather_depth); unlock:raw_spin_unlock_irqrestore (&bypass_lock, flags);scx_clear_softlockup ();
@@ -4192,7 +4194,7 @@ static void scx_root_disable(struct scx_sched *sch)int cpu;/* guarantee forward progress and disable all descendants */ -scx_bypass (true); +scx_bypass (sch, true);scx_propagate_disable_and_flush (sch);switch (scx_set_enable_state (SCX_DISABLING )) {
@@ -4324,7 +4326,7 @@ static void scx_root_disable(struct scx_sched *sch)WARN_ON_ONCE (scx_set_enable_state (SCX_DISABLED ) !=SCX_DISABLING ); done: -scx_bypass (false); +scx_bypass (sch, false); }static void scx_disable_workfn (struct kthread_work *work)
@@ -4985,7 +4987,7 @@ static int scx_root_enable(struct sched_ext_ops *ops, struct bpf_link *link)* scheduling) may not function correctly before all tasks are switched. * Init in bypass mode to guarantee forward progress. */ -scx_bypass (true); +scx_bypass (sch, true);for (i =SCX_OPI_NORMAL_BEGIN ; i <SCX_OPI_NORMAL_END ; i++)if (((void (**)(void ))ops)[i])
@@ -5096,7 +5098,7 @@ static int scx_root_enable(struct sched_ext_ops *ops, struct bpf_link *link)scx_task_iter_stop (&sti);percpu_up_write (&scx_fork_rwsem); -scx_bypass (false); +scx_bypass (sch, false);if (!scx_tryset_enable_state (SCX_ENABLED ,SCX_ENABLING )) {WARN_ON_ONCE (atomic_read (&sch->exit_kind) ==SCX_EXIT_NONE );
@@ -5800,6 +5802,14 @@ void print_scx_info(const char *log_lvl, struct task_struct *p)static int scx_pm_handler (struct notifier_block *nb,unsigned long event,void *ptr) { +struct scx_sched *sch; + +guard (rcu)(); + + sch =rcu_dereference (scx_root); +if (!sch) +return NOTIFY_OK ; +/* * SCX schedulers often have userspace components which are sometimes * involved in critial scheduling paths. PM operations involve freezing
@@ -5810,12 +5820,12 @@ static int scx_pm_handler(struct notifier_block *nb, unsigned long event, void *case PM_HIBERNATION_PREPARE :case PM_SUSPEND_PREPARE :case PM_RESTORE_PREPARE : -scx_bypass (true); +scx_bypass (sch, true);break ;case PM_POST_HIBERNATION :case PM_POST_SUSPEND :case PM_POST_RESTORE : -scx_bypass (false); +scx_bypass (sch, false);break ; }
@@ -6508,7 +6518,7 @@ static void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags)* lead to irq_work_queue() malfunction such as infinite busy wait for * IRQ status update. Suppress kicking. */ -if (scx_rq_bypassing (this_rq)) +if (scx_bypassing (sch,cpu_of (this_rq))) goto out;/*
kernel/sched/ext_idle.c +2 -1
@@ -768,7 +768,8 @@ void __scx_update_idle(struct rq *rq, bool idle, bool do_notify)* either enqueue() sees the idle bit or update_idle() sees the task * that enqueue() queued. */ -if (SCX_HAS_OP (sch, update_idle) && do_notify && !scx_rq_bypassing (rq)) +if (SCX_HAS_OP (sch, update_idle) && do_notify && + !scx_bypassing (sch,cpu_of (rq)))SCX_CALL_OP (sch,SCX_KF_REST , update_idle, rq,cpu_of (rq), idle); }
kernel/sched/ext_internal.h +10 -2
@@ -909,7 +909,13 @@ struct scx_event_stats {s64 SCX_EV_INSERT_NOT_OWNED ; }; +enum scx_sched_pcpu_flags { +SCX_SCHED_PCPU_BYPASSING =1LLU <<0 , +}; +struct scx_sched_pcpu { +u64 flags;/* protected by rq lock */ +/* * The event counters are in a per-CPU variable to minimize the * accounting overhead. A system-wide view on the event counter is
@@ -934,6 +940,7 @@ struct scx_sched {struct scx_dispatch_q **global_dsqs;struct scx_sched_pcpu __percpu *pcpu; +s32 bypass_depth;s32 level; bool warned_zero_slice:1; bool warned_deprecated_rq:1;
@@ -1138,9 +1145,10 @@ static inline bool scx_kf_allowed_if_unlocked(void)return !current->scx.kf_mask; } -static inline bool scx_rq_bypassing (struct rq *rq) +static inline bool scx_bypassing (struct scx_sched *sch,s32 cpu) { -return unlikely (rq->scx.flags &SCX_RQ_BYPASSING ); +return unlikely (per_cpu_ptr (sch->pcpu, cpu)->flags & +SCX_SCHED_PCPU_BYPASSING ); }#ifdef CONFIG_EXT_SUB_SCHED
kernel/sched/sched.h +0 -1
@@ -755,7 +755,6 @@ enum scx_rq_flags {SCX_RQ_CAN_STOP_TICK =1 <<1 ,SCX_RQ_BAL_PENDING =1 <<2 ,/* balance hasn't run yet */ SCX_RQ_BAL_KEEP =1 <<3 ,/* balance decided to keep current */ -SCX_RQ_BYPASSING =1 <<4 ,SCX_RQ_CLK_VALID =1 <<5 ,/* RQ clock is fresh and valid */ SCX_RQ_IN_WAKEUP = 1 << 16,
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 32/46 sched_ext: Make bypass mode sub-sched aware
In reply to Tejun HeoBypass mode is used to simplify enable and disable paths and guarantee forward progress when something goes wrong. When enabled, all tasks skip BPF scheduling and fall back to simple in-kernel FIFO scheduling. While this global behavior can be used as-is when dealing with sub-scheds, that would allow any sub-sched instance to affect the whole system in a significantly disruptive manner.
Make bypass mode hierarchical instead. An scx_sched bypasses if itself or any of its ancestors are in the bypass mode.
--- kernel/sched/ext.c | 38 +++++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-)
kernel/sched/ext.c +31 -7
@@ -3949,6 +3949,7 @@ static void scx_bypass(struct scx_sched *sch, bool bypass) {static DEFINE_RAW_SPINLOCK (bypass_lock);static unsigned long bypass_timestamp; +struct scx_sched *pos;unsigned long flags;int cpu;
@@ -3970,6 +3971,24 @@ static void scx_bypass(struct scx_sched *sch, bool bypass) ktime_get_ns() - bypass_timestamp); } +/* +* Bypass state is propagated to all descendants - an scx_sched bypasses +* if itself or any of its ancestors are in bypass mode. +*/ +raw_spin_lock (&scx_sched_lock); +scx_for_each_descendant_pre (pos, sch) { +if (pos == sch) +continue ; +if (bypass) { + pos->bypass_depth++; +WARN_ON_ONCE (pos->bypass_depth <=0 ); + }else { + pos->bypass_depth--; +WARN_ON_ONCE (pos->bypass_depth <0 ); + } + } +raw_spin_unlock (&scx_sched_lock); +if (!scx_parent (sch))atomic_inc (&scx_breather_depth);
@@ -3984,18 +4003,20 @@ static void scx_bypass(struct scx_sched *sch, bool bypass)*/ for_each_possible_cpu (cpu) {struct rq *rq =cpu_rq (cpu); -struct scx_sched_pcpu *pcpu =per_cpu_ptr (sch->pcpu, cpu);struct task_struct *p, *n;raw_spin_rq_lock (rq); -if (bypass) { -WARN_ON_ONCE (pcpu->flags &SCX_SCHED_PCPU_BYPASSING ); - pcpu->flags |=SCX_SCHED_PCPU_BYPASSING ; - }else { -WARN_ON_ONCE (!(pcpu->flags &SCX_SCHED_PCPU_BYPASSING )); - pcpu->flags &= ~SCX_SCHED_PCPU_BYPASSING ; +raw_spin_lock (&scx_sched_lock); +scx_for_each_descendant_pre (pos, sch) { +struct scx_sched_pcpu *pcpu =per_cpu_ptr (pos->pcpu, cpu); + +if (pos->bypass_depth) + pcpu->flags |=SCX_SCHED_PCPU_BYPASSING ; +else + pcpu->flags &= ~SCX_SCHED_PCPU_BYPASSING ; } +raw_spin_unlock (&scx_sched_lock);/* * We need to guarantee that no tasks are on the BPF scheduler
@@ -4018,6 +4039,9 @@ static void scx_bypass(struct scx_sched *sch, bool bypass) scx.runnable_node) {struct sched_enq_and_set_ctx ctx; +if (!scx_is_descendant (scx_task_sched (p), sch)) +continue ; +/* cycling deq/enq is enough, see the function comment */ sched_deq_and_put_task (p,DEQUEUE_SAVE |DEQUEUE_MOVE , &ctx);sched_enq_and_set_task (&ctx);
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 33/46 sched_ext: Factor out scx_dispatch_sched()
In reply to Tejun HeoIn preparation of multiple scheduler support, factor out scx_dispatch_sched() from balance_one(). The function boundary makes remembering $prev_on_scx and $prev_on_rq less useful. Open code $prev_on_scx in balance_one() and $prev_on_rq in both balance_one() and scx_dispatch_sched().
No functional changes.
--- kernel/sched/ext.c | 115 ++++++++++++++++++++++++--------------------- 1 file changed, 62 insertions(+), 53 deletions(-)
kernel/sched/ext.c +62 -53
@@ -2178,14 +2178,68 @@ static void flush_dispatch_buf(struct scx_sched *sch, struct rq *rq) dspc->cursor =0 ; } -static int balance_one (struct rq *rq,struct task_struct *prev) +static bool scx_dispatch_sched (struct scx_sched *sch,struct rq *rq, +struct task_struct *prev) { -struct scx_sched *sch = scx_root;struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx);bool prev_on_scx = prev->sched_class == &ext_sched_class; -bool prev_on_rq = prev->scx.flags &SCX_TASK_QUEUED ;int nr_loops =SCX_DSP_MAX_LOOPS ; +if (consume_global_dsq (sch, rq)) +return true; + +if (unlikely (!SCX_HAS_OP (sch, dispatch)) || +scx_bypassing (sch,cpu_of (rq)) || !scx_rq_online (rq)) +return false; + + dspc->rq = rq; + +/* +* The dispatch loop. Because flush_dispatch_buf() may drop the rq lock, +* the local DSQ might still end up empty after a successful +* ops.dispatch(). If the local DSQ is empty even after ops.dispatch() +* produced some tasks, retry. The BPF scheduler may depend on this +* looping behavior to simplify its implementation. +*/ +do { + dspc->nr_tasks =0 ; + +SCX_CALL_OP (sch,SCX_KF_DISPATCH , dispatch, rq, +cpu_of (rq), prev_on_scx ? prev :NULL ); + +flush_dispatch_buf (sch, rq); + +if ((prev->scx.flags &SCX_TASK_QUEUED ) && prev->scx.slice) { + rq->scx.flags |=SCX_RQ_BAL_KEEP ; +return true; + } +if (rq->scx.local_dsq.nr) +return true; +if (consume_global_dsq (sch, rq)) +return true; + +/* +* ops.dispatch() can trap us in this loop by repeatedly +* dispatching ineligible tasks. Break out once in a while to +* allow the watchdog to run. As IRQ can't be enabled in +* balance(), we want to complete this scheduling cycle and then +* start a new one. IOW, we want to call resched_curr() on the +* next, most likely idle, task, not the current one. Use +* __scx_bpf_kick_cpu() for deferred kicking. +*/ +if (unlikely (!--nr_loops)) { +scx_kick_cpu (sch,cpu_of (rq),0 ); +break ; + } + }while (dspc->nr_tasks); + +return false; +} + +static int balance_one (struct rq *rq,struct task_struct *prev) +{ +struct scx_sched *sch = scx_root; +lockdep_assert_rq_held (rq); rq->scx.flags |=SCX_RQ_IN_BALANCE ; rq->scx.flags &= ~(SCX_RQ_BAL_PENDING |SCX_RQ_BAL_KEEP );
@@ -2204,7 +2258,7 @@ static int balance_one(struct rq *rq, struct task_struct *prev) rq->scx.cpu_released = false; } -if (prev_on_scx) { +if (prev->sched_class == &ext_sched_class) {update_curr_scx (rq);/*
@@ -2217,7 +2271,7 @@ static int balance_one(struct rq *rq, struct task_struct *prev)* See scx_disable_workfn() for the explanation on the bypassing * test. */ -if (prev_on_rq && prev->scx.slice && +if ((prev->scx.flags &SCX_TASK_QUEUED ) && prev->scx.slice && !scx_bypassing (sch,cpu_of (rq))) { rq->scx.flags |=SCX_RQ_BAL_KEEP ; goto has_tasks;
@@ -2228,60 +2282,15 @@ static int balance_one(struct rq *rq, struct task_struct *prev)if (rq->scx.local_dsq.nr) goto has_tasks; -if (consume_global_dsq (sch, rq)) +/* dispatch @sch */ +if (scx_dispatch_sched (sch, rq, prev)) goto has_tasks; -if (unlikely (!SCX_HAS_OP (sch, dispatch)) || -scx_bypassing (sch,cpu_of (rq)) || !scx_rq_online (rq)) - goto no_tasks; - - dspc->rq = rq; - -/* -* The dispatch loop. Because flush_dispatch_buf() may drop the rq lock, -* the local DSQ might still end up empty after a successful -* ops.dispatch(). If the local DSQ is empty even after ops.dispatch() -* produced some tasks, retry. The BPF scheduler may depend on this -* looping behavior to simplify its implementation. -*/ -do { - dspc->nr_tasks =0 ; - -SCX_CALL_OP (sch,SCX_KF_DISPATCH , dispatch, rq, -cpu_of (rq), prev_on_scx ? prev :NULL ); - -flush_dispatch_buf (sch, rq); - -if (prev_on_rq && prev->scx.slice) { - rq->scx.flags |=SCX_RQ_BAL_KEEP ; - goto has_tasks; - } -if (rq->scx.local_dsq.nr) - goto has_tasks; -if (consume_global_dsq (sch, rq)) - goto has_tasks; - -/* -* ops.dispatch() can trap us in this loop by repeatedly -* dispatching ineligible tasks. Break out once in a while to -* allow the watchdog to run. As IRQ can't be enabled in -* balance(), we want to complete this scheduling cycle and then -* start a new one. IOW, we want to call resched_curr() on the -* next, most likely idle, task, not the current one. Use -* scx_kick_cpu() for deferred kicking. -*/ -if (unlikely (!--nr_loops)) { -scx_kick_cpu (sch,cpu_of (rq),0 ); -break ; - } - }while (dspc->nr_tasks); - -no_tasks:/* * Didn't find another task to run. Keep running @prev unless * %SCX_OPS_ENQ_LAST is in effect. */ -if (prev_on_rq && +if ((prev->scx.flags &SCX_TASK_QUEUED ) && (!(sch->ops.flags &SCX_OPS_ENQ_LAST ) ||scx_bypassing (sch,cpu_of (rq)))) { rq->scx.flags |=SCX_RQ_BAL_KEEP ;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 34/46 sched_ext: When calling ops.dispatch() @prev must be on the same scx_sched
In reply to Tejun HeoThe @prev parameter passed into ops.dispatch() is expected to be on the same sched. Passing in @prev which isn't on the sched can spuriously trigger failures that can kill the scheduler. Pass in @prev iff it's on the same sched.
--- kernel/sched/ext.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/sched/ext.c +3 -2
@@ -2182,8 +2182,9 @@ static bool scx_dispatch_sched(struct scx_sched *sch, struct rq *rq, struct task_struct *prev) {struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx); -bool prev_on_scx = prev->sched_class == &ext_sched_class;int nr_loops =SCX_DSP_MAX_LOOPS ; +bool prev_on_sch = (prev->sched_class == &ext_sched_class) && + sch ==rcu_access_pointer (prev->scx.sched);if (consume_global_dsq (sch, rq))return true;
@@ -2205,7 +2206,7 @@ static bool scx_dispatch_sched(struct scx_sched *sch, struct rq *rq, dspc->nr_tasks =0 ;SCX_CALL_OP (sch,SCX_KF_DISPATCH , dispatch, rq, -cpu_of (rq), prev_on_scx ? prev :NULL ); +cpu_of (rq), prev_on_sch ? prev :NULL );flush_dispatch_buf (sch, rq);
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 35/46 sched_ext: Dispatch from all scx_sched instances
In reply to Tejun HeoThe cgroup sub-sched support involves invasive changes to many areas of sched_ext. The overall scaffolding is now in place and the next step is implementing sub-sched enable/disable.
To enable partial testing and verification, update balance_one() to dispatch from all scx_sched instances until it finds a task to run. This should keep scheduling working when sub-scheds are enabled with tasks on them. This will be replaced by BPF-driven hierarchical operation.
--- kernel/sched/ext.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
kernel/sched/ext.c +8 -4
@@ -2239,7 +2239,7 @@ static bool scx_dispatch_sched(struct scx_sched *sch, struct rq *rq,static int balance_one (struct rq *rq,struct task_struct *prev) { -struct scx_sched *sch = scx_root; +struct scx_sched *sch = scx_root, *pos;lockdep_assert_rq_held (rq); rq->scx.flags |=SCX_RQ_IN_BALANCE ;
@@ -2283,9 +2283,13 @@ static int balance_one(struct rq *rq, struct task_struct *prev)if (rq->scx.local_dsq.nr) goto has_tasks; -/* dispatch @sch */ -if (scx_dispatch_sched (sch, rq, prev)) - goto has_tasks; +/* +* TEMPORARY - Dispatch all scheds. This will be replaced by BPF-driven +* hierarchical operation. +*/ +list_for_each_entry_rcu (pos, &scx_sched_all, all) +if (scx_dispatch_sched (pos, rq, prev)) + goto has_tasks;/* * Didn't find another task to run. Keep running @prev unless
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 36/46 sched_ext: Move scx_dsp_ctx and scx_dsp_max_batch into scx_sched
In reply to Tejun Heoscx_dsp_ctx and scx_dsp_max_batch are global variables used in the dispatch path. In prepration for multiple scheduler support, move the former into scx_sched_pcpu and the latter into scx_sched. No user-visible behavior changes intended.
--- kernel/sched/ext.c | 56 ++++++++++--------------------------- kernel/sched/ext_internal.h | 18 ++++++++++++ 2 files changed, 33 insertions(+), 41 deletions(-)
kernel/sched/ext.c +15 -41
@@ -102,25 +102,6 @@ static const struct rhashtable_params dsq_hash_params = {static LLIST_HEAD (dsqs_to_free); -/* dispatch buf */ -struct scx_dsp_buf_ent { -struct task_struct *task; -unsigned long qseq; -u64 dsq_id; -u64 enq_flags; -}; - -static u32 scx_dsp_max_batch; - -struct scx_dsp_ctx { -struct rq *rq; -u32 cursor; -u32 nr_tasks; -struct scx_dsp_buf_ent buf[]; -}; - -static struct scx_dsp_ctx __percpu *scx_dsp_ctx; -/* string formatting from BPF */ struct scx_bstr_buf {u64 data[MAX_BPRINTF_VARARGS ];
@@ -2164,7 +2145,7 @@ static void finish_dispatch(struct scx_sched *sch, struct rq *rq,static void flush_dispatch_buf (struct scx_sched *sch,struct rq *rq) { -struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx); +struct scx_dsp_ctx *dspc = &this_cpu_ptr (sch->pcpu)->dsp_ctx;u32 u;for (u =0 ; u < dspc->cursor; u++) {
@@ -2181,7 +2162,7 @@ static void flush_dispatch_buf(struct scx_sched *sch, struct rq *rq)static bool scx_dispatch_sched (struct scx_sched *sch,struct rq *rq,struct task_struct *prev) { -struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx); +struct scx_dsp_ctx *dspc = &this_cpu_ptr (sch->pcpu)->dsp_ctx;int nr_loops =SCX_DSP_MAX_LOOPS ;bool prev_on_sch = (prev->sched_class == &ext_sched_class) && sch ==rcu_access_pointer (prev->scx.sched);
@@ -4356,10 +4337,6 @@ static void scx_root_disable(struct scx_sched *sch)*/ kobject_del (&sch->kobj); -free_percpu (scx_dsp_ctx); - scx_dsp_ctx =NULL ; - scx_dsp_max_batch =0 ; -mutex_unlock (&scx_enable_mutex);WARN_ON_ONCE (scx_set_enable_state (SCX_DISABLED ) !=SCX_DISABLING );
@@ -4785,7 +4762,10 @@ static struct scx_sched *scx_alloc_and_add_sched(struct sched_ext_ops *ops, sch->global_dsqs[node] = dsq; } - sch->pcpu = alloc_percpu(struct scx_sched_pcpu); + sch->dsp_max_batch = ops->dispatch_max_batch ?:SCX_DSP_DFL_MAX_BATCH ; + sch->pcpu = __alloc_percpu(struct_size_t(struct scx_sched_pcpu, + dsp_ctx.buf, sch->dsp_max_batch), + __alignof__(struct scx_sched_pcpu ));if (!sch->pcpu) goto err_free_gdsqs;
@@ -4999,16 +4979,6 @@ static int scx_root_enable(struct sched_ext_ops *ops, struct bpf_link *link)if (ret) goto err_disable; -WARN_ON_ONCE (scx_dsp_ctx); - scx_dsp_max_batch = ops->dispatch_max_batch ?:SCX_DSP_DFL_MAX_BATCH ; - scx_dsp_ctx = __alloc_percpu(struct_size_t(struct scx_dsp_ctx, buf, - scx_dsp_max_batch), - __alignof__(struct scx_dsp_ctx )); -if (!scx_dsp_ctx) { - ret = -ENOMEM ; - goto err_disable; - } -if (ops->timeout_ms) timeout =msecs_to_jiffies (ops->timeout_ms);else
@@ -5947,7 +5917,7 @@ static bool scx_dsq_insert_preamble(struct scx_sched *sch, struct task_struct *pstatic void scx_dsq_insert_commit (struct scx_sched *sch,struct task_struct *p,u64 dsq_id,u64 enq_flags) { -struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx); +struct scx_dsp_ctx *dspc = &this_cpu_ptr (sch->pcpu)->dsp_ctx;struct task_struct *ddsp_task; ddsp_task =__this_cpu_read (direct_dispatch_task);
@@ -5956,7 +5926,7 @@ static void scx_dsq_insert_commit(struct scx_sched *sch, struct task_struct *p,return ; } -if (unlikely (dspc->cursor >= scx_dsp_max_batch)) { +if (unlikely (dspc->cursor >= sch->dsp_max_batch)) {scx_error (sch,"dispatch buffer overflow" );return ; }
@@ -6204,7 +6174,7 @@ __bpf_kfunc u32 scx_bpf_dispatch_nr_slots(const struct bpf_prog_aux *aux__prog)if (!scx_kf_allowed (sch,SCX_KF_DISPATCH ))return 0 ; -return scx_dsp_max_batch -__this_cpu_read (scx_dsp_ctx->cursor); +return sch->dsp_max_batch -__this_cpu_read (sch->pcpu->dsp_ctx.cursor); }/**
@@ -6216,8 +6186,8 @@ __bpf_kfunc u32 scx_bpf_dispatch_nr_slots(const struct bpf_prog_aux *aux__prog)*/ __bpf_kfunc void scx_bpf_dispatch_cancel(const struct bpf_prog_aux *aux__prog) { -struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx);struct scx_sched *sch; +struct scx_dsp_ctx *dspc;guard (rcu)();
@@ -6228,6 +6198,8 @@ __bpf_kfunc void scx_bpf_dispatch_cancel(const struct bpf_prog_aux *aux__prog)if (!scx_kf_allowed (sch,SCX_KF_DISPATCH ))return ; + dspc = &this_cpu_ptr (sch->pcpu)->dsp_ctx; +if (dspc->cursor >0 ) dspc->cursor--;else
@@ -6252,9 +6224,9 @@ __bpf_kfunc void scx_bpf_dispatch_cancel(const struct bpf_prog_aux *aux__prog) __bpf_kfunc bool scx_bpf_dsq_move_to_local(u64 dsq_id,const struct bpf_prog_aux *aux__prog) { -struct scx_dsp_ctx *dspc =this_cpu_ptr (scx_dsp_ctx);struct scx_dispatch_q *dsq;struct scx_sched *sch; +struct scx_dsp_ctx *dspc;guard (rcu)();
@@ -6265,6 +6237,8 @@ __bpf_kfunc bool scx_bpf_dsq_move_to_local(u64 dsq_id,if (!scx_kf_allowed (sch,SCX_KF_DISPATCH ))return false; + dspc = &this_cpu_ptr (sch->pcpu)->dsp_ctx; +flush_dispatch_buf (sch, dspc->rq); dsq =find_user_dsq (sch, dsq_id);
kernel/sched/ext_internal.h +18 -0
@@ -913,6 +913,21 @@ enum scx_sched_pcpu_flags { SCX_SCHED_PCPU_BYPASSING = 1LLU << 0, }; +/* dispatch buf */ +struct scx_dsp_buf_ent { +struct task_struct *task; +unsigned long qseq; +u64 dsq_id; +u64 enq_flags; +}; + +struct scx_dsp_ctx { +struct rq *rq; +u32 cursor; +u32 nr_tasks; +struct scx_dsp_buf_ent buf[]; +}; +struct scx_sched_pcpu {u64 flags;/* protected by rq lock */
@@ -922,6 +937,8 @@ struct scx_sched_pcpu {* constructed when requested by scx_bpf_events(). */ struct scx_event_stats event_stats; + +struct scx_dsp_ctx dsp_ctx; };struct scx_sched {
@@ -941,6 +958,7 @@ struct scx_sched {struct scx_sched_pcpu __percpu *pcpu;s32 bypass_depth; +u32 dsp_max_batch;s32 level; bool warned_zero_slice:1; bool warned_deprecated_rq:1;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 37/46 sched_ext: Make watchdog sub-sched aware
In reply to Tejun HeoCurrently, the watchdog checks all tasks as if they are all on scx_root. Move scx_watchdog_timeout inside scx_sched and make check_rq_for_timeouts() use the timeout from the scx_sched associated with each task. refresh_watchdog() is added, which determines the timer interval as half of the shortest watchdog timeouts of all scheds and arms or disarms it as necessary. Every scx_sched instance has equivalent or better detection latency while sharing the same timer.
--- kernel/sched/ext.c | 74 ++++++++++++++++++++++++------------- kernel/sched/ext_internal.h | 7 ++++ 2 files changed, 56 insertions(+), 25 deletions(-)
kernel/sched/ext.c +49 -25
@@ -66,11 +66,10 @@ static atomic_long_t scx_hotplug_seq = ATOMIC_LONG_INIT(0);static atomic_long_t scx_enable_seq =ATOMIC_LONG_INIT (0 );/* -* The maximum amount of time in jiffies that a task may be runnable without -* being scheduled on a CPU. If this timeout is exceeded, it will trigger -* scx_error(). +* Watchdog interval. All scx_sched's share a single watchdog timer and the +* interval is half of the shortest sch->watchdog_timeout. */ -static unsigned long scx_watchdog_timeout; +static unsigned long scx_watchdog_interval;/* * The last time the delayed work was run. This delayed work relies on
@@ -2761,10 +2760,11 @@ static bool check_rq_for_timeouts(struct rq *rq) goto out_unlock;list_for_each_entry (p, &rq->scx.runnable_list, scx.runnable_node) { +struct scx_sched *sch =scx_task_sched (p);unsigned long last_runnable = p->scx.runnable_at;if (unlikely (time_after (jiffies, - last_runnable + scx_watchdog_timeout))) { + last_runnable + sch->watchdog_timeout))) {u32 dur_ms =jiffies_to_msecs (jiffies - last_runnable); scx_exit(sch, SCX_EXIT_ERROR_STALL, 0,
@@ -2781,6 +2781,7 @@ static bool check_rq_for_timeouts(struct rq *rq)static void scx_watchdog_workfn (struct work_struct *work) { +unsigned long intv;int cpu;WRITE_ONCE (scx_watchdog_timestamp, jiffies);
@@ -2791,28 +2792,31 @@ static void scx_watchdog_workfn(struct work_struct *work)cond_resched (); } -queue_delayed_work (system_unbound_wq,to_delayed_work (work), - scx_watchdog_timeout /2 ); + + intv =READ_ONCE (scx_watchdog_interval); +if (intv <ULONG_MAX ) +queue_delayed_work (system_unbound_wq,to_delayed_work (work), + intv); }void scx_tick (struct rq *rq) { -struct scx_sched *sch; +struct scx_sched *root;unsigned long last_check;if (!scx_enabled ())return ; - sch =rcu_dereference_bh (scx_root); -if (unlikely (!sch)) + root =rcu_dereference_bh (scx_root); +if (unlikely (!root))return ; last_check =READ_ONCE (scx_watchdog_timestamp);if (unlikely (time_after (jiffies, - last_check +READ_ONCE (scx_watchdog_timeout)))) { + last_check +READ_ONCE (root->watchdog_timeout)))) {u32 dur_ms =jiffies_to_msecs (jiffies - last_check); -scx_exit (sch,SCX_EXIT_ERROR_STALL ,0 , +scx_exit (root,SCX_EXIT_ERROR_STALL ,0 ,"watchdog failed to check in for %u.%03us" , dur_ms /1000 , dur_ms %1000 ); }
@@ -4108,6 +4112,26 @@ static const char *scx_exit_reason(enum scx_exit_kind kind) } } +static void refresh_watchdog (void ) +{ +struct scx_sched *sch; +unsigned long intv =ULONG_MAX ; + +/* take the shortest timeout and use its half for watchdog interval */ +rcu_read_lock (); +list_for_each_entry_rcu (sch, &scx_sched_all, all) + intv =max (min (intv, sch->watchdog_timeout /2 ),1 ); +rcu_read_unlock (); + +WRITE_ONCE (scx_watchdog_timestamp, jiffies); +WRITE_ONCE (scx_watchdog_interval, intv); + +if (intv <ULONG_MAX ) +mod_delayed_work (system_unbound_wq, &scx_watchdog_work, intv); +else +cancel_delayed_work_sync (&scx_watchdog_work); +} +#ifdef CONFIG_EXT_SUB_SCHED static DECLARE_WAIT_QUEUE_HEAD (scx_unlink_waitq);
@@ -4159,6 +4183,8 @@ static void scx_sub_disable(struct scx_sched *sch)list_del_rcu (&sch->all);raw_spin_unlock_irq (&scx_sched_lock); +refresh_watchdog (); +mutex_unlock (&scx_enable_mutex);/*
@@ -4316,12 +4342,12 @@ static void scx_root_disable(struct scx_sched *sch)if (sch->ops.exit)SCX_CALL_OP (sch,SCX_KF_UNLOCKED , exit,NULL , ei); -cancel_delayed_work_sync (&scx_watchdog_work); -raw_spin_lock_irq (&scx_sched_lock);list_del_rcu (&sch->all);raw_spin_unlock_irq (&scx_sched_lock); +refresh_watchdog (); +/* * scx_root clearing must be inside cpus_read_lock(). See * handle_hotplug().
@@ -4780,6 +4806,11 @@ static struct scx_sched *scx_alloc_and_add_sched(struct sched_ext_ops *ops, sch->ancestors[level] = sch; sch->level = level; +if (ops->timeout_ms) + sch->watchdog_timeout =msecs_to_jiffies (ops->timeout_ms); +else + sch->watchdog_timeout =SCX_WATCHDOG_MAX_TIMEOUT ; +atomic_set (&sch->exit_kind,SCX_EXIT_NONE );init_irq_work (&sch->error_irq_work, scx_error_irq_workfn);kthread_init_work (&sch->disable_work, scx_disable_workfn);
@@ -4899,7 +4930,6 @@ static int scx_root_enable(struct sched_ext_ops *ops, struct bpf_link *link)struct scx_sched *sch;struct scx_task_iter sti;struct task_struct *p; -unsigned long timeout;int i, cpu, ret; if (!cpumask_equal(housekeeping_cpumask(HK_TYPE_DOMAIN),
@@ -4953,6 +4983,8 @@ static int scx_root_enable(struct sched_ext_ops *ops, struct bpf_link *link)list_add_tail_rcu (&sch->all, &scx_sched_all);raw_spin_unlock_irq (&scx_sched_lock); +refresh_watchdog (); +scx_idle_enable (ops);if (sch->ops.init) {
@@ -4979,16 +5011,6 @@ static int scx_root_enable(struct sched_ext_ops *ops, struct bpf_link *link)if (ret) goto err_disable; -if (ops->timeout_ms) - timeout =msecs_to_jiffies (ops->timeout_ms); -else - timeout =SCX_WATCHDOG_MAX_TIMEOUT ; - -WRITE_ONCE (scx_watchdog_timeout, timeout); -WRITE_ONCE (scx_watchdog_timestamp, jiffies); -queue_delayed_work (system_unbound_wq, &scx_watchdog_work, - scx_watchdog_timeout /2 ); -/* * Once __scx_enabled is set, %current can be switched to SCX anytime. * This can lead to stalls as some BPF schedulers (e.g. userspace
@@ -5215,6 +5237,8 @@ static int scx_sub_enable(struct sched_ext_ops *ops, struct bpf_link *link)list_add_tail_rcu (&sch->all, &scx_sched_all);raw_spin_unlock_irq (&scx_sched_lock); +refresh_watchdog (); +if (sch->level >=SCX_SUB_MAX_DEPTH ) {scx_error (sch,"max nesting depth %d violated" ,SCX_SUB_MAX_DEPTH );
kernel/sched/ext_internal.h +7 -0
@@ -974,6 +974,13 @@ struct scx_sched {struct kset *sub_kset;#endif /* CONFIG_EXT_SUB_SCHED */ +/* +* The maximum amount of time in jiffies that a task may be runnable +* without being scheduled on a CPU. If this timeout is exceeded, it +* will trigger scx_error(). +*/ +unsigned long watchdog_timeout; +atomic_t exit_kind;struct scx_exit_info *exit_info;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 38/46 sched_ext: Convert scx_dump_state() spinlock to raw spinlock
In reply to Tejun HeoThe scx_dump_state() function uses a regular spinlock to serialize access. In a subsequent patch, this function will be called while holding scx_sched_lock, which is a raw spinlock, creating a lock nesting violation.
Convert the dump_lock to a raw spinlock and use the guard macro for cleaner lock management.
--- kernel/sched/ext.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
kernel/sched/ext.c +2 -5
@@ -4565,7 +4565,7 @@ static void scx_dump_task(struct seq_buf *s, struct scx_dump_ctx *dctx,static void scx_dump_state (struct scx_exit_info *ei,size_t dump_len) { -static DEFINE_SPINLOCK (dump_lock); +static DEFINE_RAW_SPINLOCK (dump_lock);static const char trunc_marker[] ="\n\n~~~~ TRUNCATED ~~~~\n" ;struct scx_sched *sch = scx_root;struct scx_dump_ctx dctx = {
@@ -4577,11 +4577,10 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len) };struct seq_buf s;struct scx_event_stats events; -unsigned long flags;char *buf;int cpu; -spin_lock_irqsave (&dump_lock, flags); +guard (raw_spinlock_irqsave)(&dump_lock);seq_buf_init (&s, ei->dump, dump_len);
@@ -4705,8 +4704,6 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)if (seq_buf_has_overflowed (&s) && dump_len >=sizeof (trunc_marker))memcpy (ei->dump + dump_len -sizeof (trunc_marker), trunc_marker,sizeof (trunc_marker)); - -spin_unlock_irqrestore (&dump_lock, flags); }static void scx_error_irq_workfn (struct irq_work *irq_work)
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 39/46 sched_ext: Support dumping multiple schedulers and add scheduler identification
In reply to Tejun HeoExtend scx_dump_state() to support multiple schedulers and improve task identification in dumps. The function now takes a specific scheduler to dump and can optionally filter tasks by scheduler.
scx_dump_task() now displays which scheduler each task belongs to, using "*" to mark tasks owned by the scheduler being dumped. Sub-schedulers are identified with their level and cgroup ID.
The SysRq-D handler now iterates through all active schedulers under scx_sched_lock and dumps each one separately. For SysRq-D dumps, only tasks owned by each scheduler are dumped to avoid redundancy since all schedulers are being dumped. Error-triggered dumps continue to dump all tasks since only that specific scheduler is being dumped.
--- kernel/sched/ext.c | 54 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 11 deletions(-)
kernel/sched/ext.c +43 -11
@@ -4520,22 +4520,34 @@ static void ops_dump_exit(void) scx_dump_data.cpu =-1 ; } -static void scx_dump_task (struct seq_buf *s,struct scx_dump_ctx *dctx, +static void scx_dump_task (struct scx_sched *sch, +struct seq_buf *s,struct scx_dump_ctx *dctx,struct task_struct *p,char marker) {static unsigned long bt[SCX_EXIT_BT_LEN ]; -struct scx_sched *sch = scx_root; +struct scx_sched *task_sch =scx_task_sched (p); +const char *own_marker; +char sch_id_buf[32 ];char dsq_id_buf[19 ] ="(n/a)" ;unsigned long ops_state =atomic_long_read (&p->scx.ops_state);unsigned int bt_len =0 ; + own_marker = task_sch == sch ?"*" :"" ; + +if (task_sch->level ==0 ) +scnprintf (sch_id_buf,sizeof (sch_id_buf),"root" ); +else +scnprintf (sch_id_buf,sizeof (sch_id_buf),"sub%d-%llu" , + task_sch->level, task_sch->ops.sub_cgroup_id); +if (p->scx.dsq)scnprintf (dsq_id_buf,sizeof (dsq_id_buf),"0x%llx" , (unsigned long long )p->scx.dsq->id);dump_newline (s); -dump_line (s," %c%c %s[%d] %+ldms" , +dump_line (s," %c%c %s[%d] %s%s %+ldms" , marker,task_state_to_char (p), p->comm, p->pid, + own_marker, sch_id_buf,jiffies_delta_msecs (p->scx.runnable_at, dctx->at_jiffies)); dump_line(s, " scx_state/flags=%u/0x%x dsq_flags=0x%x ops_state/qseq=%lu/%lu", scx_get_task_state(p), p->scx.flags & ~SCX_TASK_STATE_MASK,
@@ -4563,11 +4575,18 @@ static void scx_dump_task(struct seq_buf *s, struct scx_dump_ctx *dctx, } } -static void scx_dump_state (struct scx_exit_info *ei,size_t dump_len) +/* +* Dump scheduler state. If @dump_all_tasks is true, dump all tasks regardless +* of which scheduler they belong to. If false, only dump tasks owned by @sch. +* For SysRq-D dumps, @dump_all_tasks=false since all schedulers are dumped +* separately. For error dumps, @dump_all_tasks=true since only the failing +* scheduler is dumped. +*/ +static void scx_dump_state (struct scx_sched *sch,struct scx_exit_info *ei, +size_t dump_len,bool dump_all_tasks) {static DEFINE_RAW_SPINLOCK (dump_lock);static const char trunc_marker[] ="\n\n~~~~ TRUNCATED ~~~~\n" ; -struct scx_sched *sch = scx_root;struct scx_dump_ctx dctx = { .kind = ei->kind, .exit_code = ei->exit_code,
@@ -4584,6 +4603,14 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)seq_buf_init (&s, ei->dump, dump_len); +#ifdef CONFIG_EXT_SUB_SCHED +if (sch->level ==0 ) +dump_line (&s,"%s: root" , sch->ops.name); +else +dump_line (&s,"%s: sub%d-%llu %s" , + sch->ops.name, sch->level, sch->ops.sub_cgroup_id, + sch->cgrp_path); +#endif if (ei->kind ==SCX_EXIT_NONE ) {dump_line (&s,"Debug dump triggered by %s" , ei->reason); }else {
@@ -4676,11 +4703,13 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)seq_buf_set_overflow (&s); } -if (rq->curr->sched_class == &ext_sched_class) -scx_dump_task (&s, &dctx, rq->curr,'*' ); +if (rq->curr->sched_class == &ext_sched_class && + (dump_all_tasks ||scx_task_sched (rq->curr) == sch)) +scx_dump_task (sch, &s, &dctx, rq->curr,'*' );list_for_each_entry (p, &rq->scx.runnable_list, scx.runnable_node) -scx_dump_task (&s, &dctx, p,' ' ); +if (dump_all_tasks ||scx_task_sched (p) == sch) +scx_dump_task (sch, &s, &dctx, p,' ' ); next:rq_unlock (rq, &rf); }
@@ -4712,7 +4741,7 @@ static void scx_error_irq_workfn(struct irq_work *irq_work)struct scx_exit_info *ei = sch->exit_info;if (ei->kind >=SCX_EXIT_ERROR ) -scx_dump_state (ei, sch->ops.exit_dump_len); +scx_dump_state (sch, ei, sch->ops.exit_dump_len, true);kthread_queue_work (sch->helper, &sch->disable_work); }
@@ -5662,9 +5691,12 @@ static const struct sysrq_key_op sysrq_sched_ext_reset_op = {static void sysrq_handle_sched_ext_dump (u8 key) {struct scx_exit_info ei = { .kind =SCX_EXIT_NONE , .reason ="SysRq-D" }; +struct scx_sched *sch; -if (scx_enabled ()) -scx_dump_state (&ei,0 ); +guard (raw_spinlock_irqsave)(&scx_sched_lock); + +list_for_each_entry_rcu (sch, &scx_sched_all, all) +scx_dump_state (sch, &ei,0 , false); }static const struct sysrq_key_op sysrq_sched_ext_dump_op = {
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 40/46 sched_ext: Implement cgroup sub-sched enabling and disabling
In reply to Tejun HeoThe preceding changes implemented the framework to support cgroup sub-scheds and updated scheduling paths and kfuncs so that they have minimal but working support for sub-scheds. However, actual sub-sched enabling/disabling hasn't been implemented yet and all tasks stayed on scx_root.
Implement cgroup sub-sched enabling and disabling to actually activate sub-scheds:
- Both enable and disable operations bypass only the tasks in the subtree of the child being enabled or disabled to limit disruptions.
- When enabling, all candidate tasks are first initialized for the child sched. Once that succeeds, the tasks are exited for the parent and then switched over to the child. This adds a bit of complication but guarantees that child scheduler failures are always contained.
- Disabling works the same way in the other direction. However, when the parent may fail to initialize a task, disabling is propagated up to the parent. While this means that a parent sched fail due to a child sched event, the failure can only originate from the parent itself (its ops.init_task()). The only effect a malfunctioning child can have on the parent is attempting to move the tasks back to the parent.
After this change, although not all the necessary mechanisms are in place yet, sub-scheds can take control of their tasks and schedule them.
--- include/linux/sched/ext.h | 1 + kernel/sched/ext.c | 281 +++++++++++++++++++++++++++++++++++++- 2 files changed, 276 insertions(+), 6 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -74,6 +74,7 @@ enum scx_ent_flags {SCX_TASK_QUEUED =1 <<0 ,/* on ext runqueue */ SCX_TASK_RESET_RUNNABLE_AT =1 <<2 ,/* runnable_at should be reset */ SCX_TASK_DEQD_FOR_SLEEP =1 <<3 ,/* last dequeue was for SLEEP */ +SCX_TASK_SUB_INIT =1 <<4 ,/* task being initialized for a sub sched */ SCX_TASK_STATE_SHIFT =8 ,/* bit 8 and 9 are used to carry scx_task_state */ SCX_TASK_STATE_BITS = 2,
kernel/sched/ext.c +275 -6
373 lines not on this page · raw message
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 41/46 HACK_NOT_FOR_UPSTREAM: sched_ext: Work around @aux__prog prototype mismatch
In reply to Tejun HeoHide them using macros. This breaks binary compatibility. There should be a better way.
NOT_SIGNED_OFF
--- tools/sched_ext/include/scx/common.bpf.h | 88 ++++++++++++++++-------- tools/sched_ext/include/scx/compat.bpf.h | 7 +- tools/sched_ext/scx_qmap.bpf.c | 6 +- 3 files changed, 69 insertions(+), 32 deletions(-)
tools/sched_ext/include/scx/common.bpf.h +60 -28
@@ -58,54 +58,86 @@ static inline void ___vmlinux_h_sanity_check___(void) "bpftool generated vmlinux.h is missing high bits for 64bit enums, upgrade clang and pahole"); } -s32 scx_bpf_create_dsq(u64 dsq_id, s32 node) __ksym; +/* +* XXX TEMPORARY - Each macro that has the same name as the preceding kfunc is +* to work around aux__prog issue. This causes a whole bunch of other problems. +* This is to work around just enough for testing and verification for now. +*/ +s32 scx_bpf_create_dsq(u64 dsq_id, s32 node, const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_create_dsq (dsq_id, node) scx_bpf_create_dsq((dsq_id), (node), NULL) s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *is_idle) __ksym; s32 scx_bpf_select_cpu_and(struct task_struct *p, s32 prev_cpu, u64 wake_flags, const struct cpumask *cpus_allowed, u64 flags) __ksym __weak; -void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym __weak; +void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags, + const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_dsq_insert (p, dsq_id, slice, enq_flags) scx_bpf_dsq_insert((p), (dsq_id), (slice), (enq_flags), NULL) void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym __weak; -u32 scx_bpf_dispatch_nr_slots(void) __ksym; -void scx_bpf_dispatch_cancel(void) __ksym; -bool scx_bpf_dsq_move_to_local(u64 dsq_id) __ksym __weak; +u32 scx_bpf_dispatch_nr_slots(const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_dispatch_nr_slots () scx_bpf_dispatch_nr_slots(NULL) +void scx_bpf_dispatch_cancel(const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_dispatch_cancel () scx_bpf_dispatch_cancel(NULL) +bool scx_bpf_dsq_move_to_local(u64 dsq_id, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_dsq_move_to_local (dsq_id) scx_bpf_dsq_move_to_local(dsq_id, NULL) void scx_bpf_dsq_move_set_slice(struct bpf_iter_scx_dsq *it__iter, u64 slice) __ksym __weak; void scx_bpf_dsq_move_set_vtime(struct bpf_iter_scx_dsq *it__iter, u64 vtime) __ksym __weak; bool scx_bpf_dsq_move(struct bpf_iter_scx_dsq *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __ksym __weak; bool scx_bpf_dsq_move_vtime(struct bpf_iter_scx_dsq *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __ksym __weak; -u32 scx_bpf_reenqueue_local(void) __ksym; -void scx_bpf_kick_cpu(s32 cpu, u64 flags) __ksym; +u32 scx_bpf_reenqueue_local(const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_reenqueue_local () scx_bpf_reenqueue_local(NULL) +void scx_bpf_kick_cpu(s32 cpu, u64 flags, const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_kick_cpu (cpu, flags) scx_bpf_kick_cpu((cpu), (flags), NULL) s32 scx_bpf_dsq_nr_queued(u64 dsq_id) __ksym; void scx_bpf_destroy_dsq(u64 dsq_id) __ksym; -int bpf_iter_scx_dsq_new(struct bpf_iter_scx_dsq *it, u64 dsq_id, u64 flags) __ksym __weak; +int bpf_iter_scx_dsq_new(struct bpf_iter_scx_dsq *it, u64 dsq_id, u64 flags, const struct bpf_prog_aux *aux__prog) __ksym __weak; +//#define bpf_iter_scx_dsq_new(it, dsq_id, flags) bpf_iter_scx_dsq_new((it), (dsq_id), (flags)) struct task_struct *bpf_iter_scx_dsq_next (struct bpf_iter_scx_dsq *it) __ksym__weak ; void bpf_iter_scx_dsq_destroy(struct bpf_iter_scx_dsq *it) __ksym __weak; -void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz) __ksym __weak; -void scx_bpf_error_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym; -void scx_bpf_dump_bstr(char *fmt, unsigned long long *data, u32 data_len) __ksym __weak; -u32 scx_bpf_cpuperf_cap(s32 cpu) __ksym __weak; -u32 scx_bpf_cpuperf_cur(s32 cpu) __ksym __weak; -void scx_bpf_cpuperf_set(s32 cpu, u32 perf) __ksym __weak; +void scx_bpf_exit_bstr(s64 exit_code, char *fmt, unsigned long long *data, u32 data__sz, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_exit_bstr (exit_code, fmt, data, data__sz) scx_bpf_exit_bstr((exit_code), (fmt), (data), (data__sz), NULL) +void scx_bpf_error_bstr(char *fmt, unsigned long long *data, u32 data__sz, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_error_bstr (fmt, data, data__sz) scx_bpf_error_bstr((fmt), (data), (data__sz), NULL) +void scx_bpf_dump_bstr(char *fmt, unsigned long long *data, u32 data__sz, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_dump_bstr (fmt, data, data__sz) scx_bpf_dump_bstr((fmt), (data), (data__sz), NULL) +u32 scx_bpf_cpuperf_cap(s32 cpu, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_cpuperf_cap (cpu) scx_bpf_cpuperf_cap((cpu), NULL) +u32 scx_bpf_cpuperf_cur(s32 cpu, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_cpuperf_cur (cpu) scx_bpf_cpuperf_cur((cpu), NULL) +void scx_bpf_cpuperf_set(s32 cpu, u32 perf, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_cpuperf_set (cpu, perf) scx_bpf_cpuperf_set((cpu), (perf), NULL) u32 scx_bpf_nr_node_ids(void) __ksym __weak; u32 scx_bpf_nr_cpu_ids(void) __ksym __weak; -int scx_bpf_cpu_node(s32 cpu) __ksym __weak; +int scx_bpf_cpu_node(s32 cpu, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_cpu_node (cpu) scx_bpf_cpu_node((cpu), NULL)const struct cpumask *scx_bpf_get_possible_cpumask (void ) __ksym__weak ;const struct cpumask *scx_bpf_get_online_cpumask (void ) __ksym__weak ; void scx_bpf_put_cpumask(const struct cpumask *cpumask) __ksym __weak; -const struct cpumask *scx_bpf_get_idle_cpumask_node (int node) __ksym__weak ; -const struct cpumask *scx_bpf_get_idle_cpumask (void ) __ksym; -const struct cpumask *scx_bpf_get_idle_smtmask_node (int node) __ksym__weak ; -const struct cpumask *scx_bpf_get_idle_smtmask (void ) __ksym; -void scx_bpf_put_idle_cpumask(const struct cpumask *cpumask) __ksym; -bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) __ksym; -s32 scx_bpf_pick_idle_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) __ksym __weak; -s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; -s32 scx_bpf_pick_any_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) __ksym __weak; -s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; +const struct cpumask *scx_bpf_get_idle_cpumask_node (int node,const struct bpf_prog_aux *aux__prog) __ksym__weak ; +#define scx_bpf_get_idle_cpumask_node (node) scx_bpf_get_idle_cpumask_node((node), NULL) +const struct cpumask *scx_bpf_get_idle_cpumask (const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_get_idle_cpumask () scx_bpf_get_idle_cpumask(NULL) +const struct cpumask *scx_bpf_get_idle_smtmask_node (int node,const struct bpf_prog_aux *aux__prog) __ksym__weak ; +#define scx_bpf_get_idle_smtmask_node (node) scx_bpf_get_idle_smtmask_node((node)) +const struct cpumask *scx_bpf_get_idle_smtmask (const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_get_idle_smtmask () scx_bpf_get_idle_smtmask(NULL) +void scx_bpf_put_idle_cpumask(const struct cpumask *cpumask, const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_put_idle_cpumask (cpumask) scx_bpf_put_idle_cpumask((cpumask), NULL) +bool scx_bpf_test_and_clear_cpu_idle(s32 cpu, const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_test_and_clear_cpu_idle (cpu) scx_bpf_test_and_clear_cpu_idle((cpu), NULL) +s32 scx_bpf_pick_idle_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_pick_idle_cpu_node (cpus_allowed, node, flags) scx_bpf_pick_idle_cpu_node((cpus_allowed), (node), (flags), NULL) +s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags, const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_pick_idle_cpu (cpus_allowed, flags) scx_bpf_pick_idle_cpu((cpus_allowed), (flags), NULL) +s32 scx_bpf_pick_any_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_pick_any_cpu_node (cpus_allowed, node, flags) scx_bpf_pick_any_cpu_node((cpus_allowed), (node), (flags), NULL) +s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags, const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_pick_any_cpu (cpus_allowed, flags) scx_bpf_pick_any_cpu((cpus_allowed), (flags), NULL) bool scx_bpf_task_running(const struct task_struct *p) __ksym; s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym; -struct rq *scx_bpf_cpu_rq (s32 cpu) __ksym; -struct rq *scx_bpf_locked_rq (void ) __ksym; +struct rq *scx_bpf_cpu_rq (s32 cpu,const struct bpf_prog_aux *aux__prog) __ksym; +#define scx_bpf_cpu_rq (cpu) scx_bpf_cpu_rq((cpu), NULL)struct task_struct *scx_bpf_cpu_curr (s32 cpu) __ksym__weak ; -struct cgroup *scx_bpf_task_cgroup (struct task_struct *p) __ksym__weak ; +struct cgroup *scx_bpf_task_cgroup (struct task_struct *p,const struct bpf_prog_aux *aux__prog) __ksym__weak ; +#define scx_bpf_task_cgroup (p) scx_bpf_task_cgroup((p), NULL) u64 scx_bpf_now(void) __ksym __weak; void scx_bpf_events(struct scx_event_stats *events, size_t events__sz) __ksym __weak;
tools/sched_ext/include/scx/compat.bpf.h +6 -1
@@ -40,6 +40,11 @@ bool scx_bpf_dispatch_from_dsq___compat(struct bpf_iter_scx_dsq *it__iter, struc bool scx_bpf_dispatch_vtime_from_dsq___compat(struct bpf_iter_scx_dsq *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __ksym __weak; int bpf_cpumask_populate(struct cpumask *dst, void *src, size_t src__sz) __ksym __weak; +/* +* XXX TEMPORARY - The followings conflict with prog__aux wrappers. Comment them +* out for now. +*/ +/* #define scx_bpf_dsq_insert(p, dsq_id, slice, enq_flags) \ (bpf_ksym_exists(scx_bpf_dsq_insert) ? \ scx_bpf_dsq_insert((p), (dsq_id), (slice), (enq_flags)) : \
@@ -54,7 +59,7 @@ int bpf_cpumask_populate(struct cpumask *dst, void *src, size_t src__sz) __ksym(bpf_ksym_exists(scx_bpf_dsq_move_to_local) ? \ scx_bpf_dsq_move_to_local((dsq_id)) : \ scx_bpf_consume___compat((dsq_id))) - +*/ #define __COMPAT_scx_bpf_dsq_move_set_slice (it__iter, slice) \ (bpf_ksym_exists(scx_bpf_dsq_move_set_slice) ? \ scx_bpf_dsq_move_set_slice((it__iter), (slice)) : \
tools/sched_ext/scx_qmap.bpf.c +3 -3
@@ -312,7 +312,7 @@ static bool dispatch_highpri(bool from_timer)s32 this_cpu =bpf_get_smp_processor_id ();/* scan SHARED_DSQ and move highpri tasks to HIGHPRI_DSQ */ -bpf_for_each (scx_dsq, p,SHARED_DSQ ,0 ) { +bpf_for_each (scx_dsq, p,SHARED_DSQ ,0 ,NULL ) {static u64 highpri_seq;struct task_ctx *tctx;
@@ -334,7 +334,7 @@ static bool dispatch_highpri(bool from_timer)* Scan HIGHPRI_DSQ and dispatch until a task that can run on this CPU * is found. */ -bpf_for_each (scx_dsq, p,HIGHPRI_DSQ ,0 ) { +bpf_for_each (scx_dsq, p,HIGHPRI_DSQ ,0 ,NULL ) {bool dispatched = false;s32 cpu;
@@ -801,7 +801,7 @@ static void dump_shared_dsq(void)bpf_printk ("Dumping %d tasks in SHARED_DSQ in reverse order" , nr);bpf_rcu_read_lock (); -bpf_for_each (scx_dsq, p,SHARED_DSQ ,SCX_DSQ_ITER_REV ) +bpf_for_each (scx_dsq, p,SHARED_DSQ ,SCX_DSQ_ITER_REV ,NULL )bpf_printk ("%s[%d]" , p->comm, p->pid);bpf_rcu_read_unlock (); }
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 42/46 sched_ext: Wrap global DSQs in per-node structure
In reply to Tejun HeoCurrently, global DSQs are stored as an array of pointers to scx_dispatch_q structs, one per NUMA node. This is limiting for future enhancements that may require additional per-node data structures.
Introduce scx_sched_pnode structure to wrap the global DSQ. This change replaces the global_dsqs array with a pnode array of scx_sched_pnode structs and updates all references to access global_dsq through the pnode wrapper.
This refactoring maintains NUMA-aware allocation and provides foundation for adding more per-node data structures with no functional impact.
--- kernel/sched/ext.c | 33 ++++++++++++++++----------------- kernel/sched/ext_internal.h | 6 +++++- 2 files changed, 21 insertions(+), 18 deletions(-)
kernel/sched/ext.c +16 -17
@@ -274,7 +274,7 @@ static bool scx_is_descendant(struct scx_sched *sch, struct scx_sched *ancestor)static struct scx_dispatch_q *find_global_dsq (struct scx_sched *sch,struct task_struct *p) { -return sch->global_dsqs[cpu_to_node (task_cpu (p))]; +return &sch->pnode[cpu_to_node (task_cpu (p))]->global_dsq; }static struct scx_dispatch_q *find_user_dsq (struct scx_sched *sch,u64 dsq_id)
@@ -1965,7 +1965,7 @@ static bool consume_global_dsq(struct scx_sched *sch, struct rq *rq) {int node =cpu_to_node (cpu_of (rq)); -return consume_dispatch_q (sch, rq, sch->global_dsqs[node]); +return consume_dispatch_q (sch, rq, &sch->pnode[node]->global_dsq); }/**
@@ -3733,8 +3733,8 @@ static void scx_sched_free_rcu_work(struct work_struct *work)free_percpu (sch->pcpu);for_each_node_state (node,N_POSSIBLE ) -kfree (sch->global_dsqs[node]); -kfree (sch->global_dsqs); +kfree (sch->pnode[node]); +kfree (sch->pnode);rhashtable_walk_enter (&sch->dsq_hash, &rht_iter); do {
@@ -4935,24 +4935,23 @@ static struct scx_sched *scx_alloc_and_add_sched(struct sched_ext_ops *ops,if (ret <0 ) goto err_free_ei; - sch->global_dsqs =kcalloc (nr_node_ids,sizeof (sch->global_dsqs[0 ]), -GFP_KERNEL ); -if (!sch->global_dsqs) { + sch->pnode =kcalloc (nr_node_ids,sizeof (sch->pnode[0 ]),GFP_KERNEL ); +if (!sch->pnode) { ret = -ENOMEM ; goto err_free_hash; }for_each_node_state (node,N_POSSIBLE ) { -struct scx_dispatch_q *dsq; +struct scx_sched_pnode *pnode; - dsq =kzalloc_node (sizeof (*dsq),GFP_KERNEL , node); -if (!dsq) { + pnode =kzalloc_node (sizeof (*pnode),GFP_KERNEL , node); +if (!pnode) { ret = -ENOMEM ; - goto err_free_gdsqs; + goto err_free_pnode; } -init_dsq (dsq,SCX_DSQ_GLOBAL , sch); - sch->global_dsqs[node] = dsq; +init_dsq (&pnode->global_dsq,SCX_DSQ_GLOBAL , sch); + sch->pnode[node] = pnode; } sch->dsp_max_batch = ops->dispatch_max_batch ?:SCX_DSP_DFL_MAX_BATCH ;
@@ -4960,7 +4959,7 @@ static struct scx_sched *scx_alloc_and_add_sched(struct sched_ext_ops *ops, dsp_ctx.buf, sch->dsp_max_batch), __alignof__(struct scx_sched_pcpu));if (!sch->pcpu) - goto err_free_gdsqs; + goto err_free_pnode; sch->helper =kthread_run_worker (0 ,"sched_ext_helper" );if (!sch->helper)
@@ -5031,10 +5030,10 @@ static struct scx_sched *scx_alloc_and_add_sched(struct sched_ext_ops *ops,kthread_stop (sch->helper->task); err_free_pcpu:free_percpu (sch->pcpu); -err_free_gdsqs: +err_free_pnode:for_each_node_state (node,N_POSSIBLE ) -kfree (sch->global_dsqs[node]); -kfree (sch->global_dsqs); +kfree (sch->pnode[node]); +kfree (sch->pnode); err_free_hash:rhashtable_free_and_destroy (&sch->dsq_hash,NULL ,NULL ); err_free_ei:
kernel/sched/ext_internal.h +5 -1
@@ -941,6 +941,10 @@ struct scx_sched_pcpu {struct scx_dsp_ctx dsp_ctx; }; +struct scx_sched_pnode { +struct scx_dispatch_q global_dsq; +}; +struct scx_sched {struct sched_ext_ops ops; DECLARE_BITMAP(has_op, SCX_OPI_END);
@@ -954,7 +958,7 @@ struct scx_sched {* per-node split isn't sufficient, it can be further split. */ struct rhashtable dsq_hash; -struct scx_dispatch_q **global_dsqs; +struct scx_sched_pnode **pnode;struct scx_sched_pcpu __percpu *pcpu;s32 bypass_depth;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 43/46 sched_ext: Add bypass DSQ for sub-schedulers
In reply to Tejun HeoWhen sub-schedulers bypass, we want to isolate their impact on the system while ensuring forward progress. Add a bypass DSQ to route tasks from bypassing sub-schedulers to their nearest non-bypassing ancestor.
Each per-node structure gets a bypass DSQ (SCX_DSQ_SUB_BYPASS). Tasks from bypassing descendants are collected there and consumed with a simple alternating policy between regular dispatch and bypass DSQ.
Add SCX_EV_SUB_BYPASS_DISPATCH event counter to track bypass DSQ usage for observability.
In the future, we can add ops flags and kfuncs to allow BPF schedulers to control bypass scheduling if needed.
--- include/linux/sched/ext.h | 2 + kernel/sched/ext.c | 91 +++++++++++++++++++++++++++++++++---- kernel/sched/ext_internal.h | 9 ++++ kernel/sched/sched.h | 3 ++ 4 files changed, 95 insertions(+), 10 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -48,6 +48,8 @@ enum scx_dsq_id_flags {SCX_DSQ_LOCAL =SCX_DSQ_FLAG_BUILTIN |2 ,SCX_DSQ_LOCAL_ON =SCX_DSQ_FLAG_BUILTIN |SCX_DSQ_FLAG_LOCAL_ON ,SCX_DSQ_LOCAL_CPU_MASK =0xffffffffLLU , + + SCX_DSQ_SUB_BYPASS = SCX_DSQ_FLAG_BUILTIN | 3, };/*
kernel/sched/ext.c +81 -10
167 lines not on this page · raw message
kernel/sched/ext_internal.h +9 -0
@@ -907,6 +907,12 @@ struct scx_event_stats {* scheduler. */ s64 SCX_EV_INSERT_NOT_OWNED ; + +/* +* The number of times tasks from bypassing descendants are scheduled +* from sub_bypass_dsq's. +*/ +s64 SCX_EV_SUB_BYPASS_DISPATCH ; };enum scx_sched_pcpu_flags {
@@ -943,6 +949,9 @@ struct scx_sched_pcpu {struct scx_sched_pnode {struct scx_dispatch_q global_dsq; +#ifdef CONFIG_EXT_SUB_SCHED +struct scx_dispatch_q sub_bypass_dsq; +#endif /* CONFIG_EXT_SUB_SCHED */ };struct scx_sched {
kernel/sched/sched.h +3 -0
@@ -780,6 +780,9 @@ struct scx_rq {struct balance_callback deferred_bal_cb;struct irq_work deferred_irq_work;struct irq_work kick_cpus_irq_work; +#ifdef CONFIG_EXT_SUB_SCHED +bool dsp_sub_bypass_toggle; +#endif /* CONFIG_EXT_SUB_SCHED */ };#endif /* CONFIG_SCHED_CLASS_EXT */
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 44/46 sched_ext: Factor out scx_link_sched() and scx_unlink_sched()
In reply to Tejun HeoFactor out scx_link_sched() and scx_unlink_sched() functions to reduce code duplication in the scheduler enable/disable paths.
No functional change.
--- kernel/sched/ext.c | 53 +++++++++++++++++++++++++++------------------- 1 file changed, 31 insertions(+), 22 deletions(-)
kernel/sched/ext.c +31 -22
@@ -4221,6 +4221,33 @@ static void refresh_watchdog(void)cancel_delayed_work_sync (&scx_watchdog_work); } +static void scx_link_sched (struct scx_sched *sch) +{ + scoped_guard(raw_spinlock_irq, &scx_sched_lock) { +#ifdef CONFIG_EXT_SUB_SCHED +struct scx_sched *parent =scx_parent (sch); +if (parent) +list_add_tail (&sch->sibling, &parent->children); +#endif /* CONFIG_EXT_SUB_SCHED */ +list_add_tail_rcu (&sch->all, &scx_sched_all); + } + +refresh_watchdog (); +} + +static void scx_unlink_sched (struct scx_sched *sch) +{ + scoped_guard(raw_spinlock_irq, &scx_sched_lock) { +#ifdef CONFIG_EXT_SUB_SCHED +if (scx_parent (sch)) +list_del_init (&sch->sibling); +#endif /* CONFIG_EXT_SUB_SCHED */ +list_del_rcu (&sch->all); + } + +refresh_watchdog (); +} +#ifdef CONFIG_EXT_SUB_SCHED static DECLARE_WAIT_QUEUE_HEAD (scx_unlink_waitq);
@@ -4385,12 +4412,7 @@ static void scx_sub_disable(struct scx_sched *sch)*/ synchronize_rcu_expedited (); -raw_spin_lock_irq (&scx_sched_lock); -list_del_init (&sch->sibling); -list_del_rcu (&sch->all); -raw_spin_unlock_irq (&scx_sched_lock); - -refresh_watchdog (); +scx_unlink_sched (sch);mutex_unlock (&scx_enable_mutex);
@@ -4549,11 +4571,7 @@ static void scx_root_disable(struct scx_sched *sch)if (sch->ops.exit)SCX_CALL_OP (sch,SCX_KF_UNLOCKED , exit,NULL , ei); -raw_spin_lock_irq (&scx_sched_lock); -list_del_rcu (&sch->all); -raw_spin_unlock_irq (&scx_sched_lock); - -refresh_watchdog (); +scx_unlink_sched (sch);/* * scx_root clearing must be inside cpus_read_lock(). See
@@ -5215,11 +5233,7 @@ static int scx_root_enable(struct sched_ext_ops *ops, struct bpf_link *link)*/ rcu_assign_pointer (scx_root, sch); -raw_spin_lock_irq (&scx_sched_lock); -list_add_tail_rcu (&sch->all, &scx_sched_all); -raw_spin_unlock_irq (&scx_sched_lock); - -refresh_watchdog (); +scx_link_sched (sch);scx_idle_enable (ops);
@@ -5486,12 +5500,7 @@ static int scx_sub_enable(struct sched_ext_ops *ops, struct bpf_link *link) goto out_put_cgrp; } -raw_spin_lock_irq (&scx_sched_lock); -list_add_tail (&sch->sibling, &parent->children); -list_add_tail_rcu (&sch->all, &scx_sched_all); -raw_spin_unlock_irq (&scx_sched_lock); - -refresh_watchdog (); +scx_link_sched (sch);if (sch->level >=SCX_SUB_MAX_DEPTH ) { scx_error(sch, "max nesting depth %d violated",
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 45/46 sched_ext: Add rhashtable lookup for sub-schedulers
In reply to Tejun HeoAdd rhashtable-based lookup for sub-schedulers indexed by cgroup_id to enable efficient scheduler discovery in preparation for multiple scheduler support. The hash table allows quick lookup of the appropriate scheduler instance when processing tasks from different cgroups.
This extends scx_link_sched() to register sub-schedulers in the hash table and scx_unlink_sched() to remove them. A new scx_find_sub_sched() function provides the lookup interface.
--- kernel/sched/ext.c | 54 +++++++++++++++++++++++++++++++++---- kernel/sched/ext_internal.h | 2 ++ 2 files changed, 51 insertions(+), 5 deletions(-)
kernel/sched/ext.c +49 -5
@@ -27,6 +27,16 @@ struct scx_sched __rcu *scx_root;*/ static LIST_HEAD (scx_sched_all); +#ifdef CONFIG_EXT_SUB_SCHED +static const struct rhashtable_params scx_sched_hash_params = { + .key_len = sizeof_field(struct scx_sched, ops.sub_cgroup_id), + .key_offset = offsetof(struct scx_sched, ops.sub_cgroup_id), + .head_offset = offsetof(struct scx_sched , hash_node), +}; + +static struct rhashtable scx_sched_hash; +#endif +/* * - We want to visit and perform sleepable operations on every task. *
@@ -201,6 +211,12 @@ static struct scx_sched *scx_parent(struct scx_sched *sch) }#ifdef CONFIG_EXT_SUB_SCHED +static struct scx_sched *scx_find_sub_sched (u64 cgroup_id) +{ +return rhashtable_lookup (&scx_sched_hash, &cgroup_id, + scx_sched_hash_params); +} +static void scx_set_task_sched (struct task_struct *p,struct scx_sched *sch) {rcu_assign_pointer (p->scx.sched, sch);
@@ -233,6 +249,11 @@ static struct scx_sched *scx_next_descendant_pre(struct scx_sched *pos, return NULL; }#else /* CONFIG_EXT_SUB_SCHED */ +static struct scx_sched *scx_find_sub_sched (u64 cgroup_id) +{ +return NULL ; +} +static void scx_set_task_sched (struct task_struct *p,struct scx_sched *sch) { }
@@ -4221,26 +4242,41 @@ static void refresh_watchdog(void)cancel_delayed_work_sync (&scx_watchdog_work); } -static void scx_link_sched (struct scx_sched *sch) +static int scx_link_sched (struct scx_sched *sch) { scoped_guard(raw_spinlock_irq, &scx_sched_lock) {#ifdef CONFIG_EXT_SUB_SCHED struct scx_sched *parent =scx_parent (sch); -if (parent) +int ret; + +if (parent) { + ret =rhashtable_lookup_insert_fast (&scx_sched_hash, + &sch->hash_node, scx_sched_hash_params); +if (ret) { +scx_error (sch,"failed to insert into scx_sched_hash (%d)" , ret); +return ret; + } +list_add_tail (&sch->sibling, &parent->children); + }#endif /* CONFIG_EXT_SUB_SCHED */ +list_add_tail_rcu (&sch->all, &scx_sched_all); }refresh_watchdog (); +return 0 ; }static void scx_unlink_sched (struct scx_sched *sch) { scoped_guard(raw_spinlock_irq, &scx_sched_lock) {#ifdef CONFIG_EXT_SUB_SCHED -if (scx_parent (sch)) +if (scx_parent (sch)) { +rhashtable_remove_fast (&scx_sched_hash, &sch->hash_node, + scx_sched_hash_params);list_del_init (&sch->sibling); + }#endif /* CONFIG_EXT_SUB_SCHED */ list_del_rcu (&sch->all); }
@@ -5233,7 +5269,9 @@ static int scx_root_enable(struct sched_ext_ops *ops, struct bpf_link *link)*/ rcu_assign_pointer (scx_root, sch); -scx_link_sched (sch); + ret =scx_link_sched (sch); +if (ret) + goto err_disable;scx_idle_enable (ops);
@@ -5500,7 +5538,9 @@ static int scx_sub_enable(struct sched_ext_ops *ops, struct bpf_link *link) goto out_put_cgrp; } -scx_link_sched (sch); + ret =scx_link_sched (sch); +if (ret) + goto err_disable;if (sch->level >=SCX_SUB_MAX_DEPTH ) { scx_error(sch, "max nesting depth %d violated",
@@ -6287,6 +6327,10 @@ void __init init_sched_ext_class(void)register_sysrq_key ('S' , &sysrq_sched_ext_reset_op);register_sysrq_key ('D' , &sysrq_sched_ext_dump_op);INIT_DELAYED_WORK (&scx_watchdog_work, scx_watchdog_workfn); + +#ifdef CONFIG_EXT_SUB_SCHED +BUG_ON (rhashtable_init (&scx_sched_hash, &scx_sched_hash_params)); +#endif /* CONFIG_EXT_SUB_SCHED */ }
kernel/sched/ext_internal.h +2 -0
@@ -980,6 +980,8 @@ struct scx_sched {struct list_head all;#ifdef CONFIG_EXT_SUB_SCHED +struct rhash_head hash_node; +struct list_head children;struct list_head sibling;struct cgroup *cgrp;
Signature · 1 line
2.51.0
Tejun Heo
maintainerPATCH 46/46 sched_ext: Add basic building blocks for nested sub-scheduler dispatching
In reply to Tejun HeoThis is an early-stage partial implementation that demonstrates the core building blocks for nested sub-scheduler dispatching. While significant work remains in the enqueue path and other areas, this patch establishes the fundamental mechanisms needed for hierarchical scheduler operation.
The key building blocks introduced include:
- Private stack support for ops.dispatch() to prevent stack overflow when walking down nested schedulers during dispatch operations
- scx_bpf_sub_dispatch() kfunc that allows parent schedulers to trigger dispatch operations on their direct child schedulers
- Proper parent-child relationship validation to ensure dispatch requests are only made to legitimate child schedulers
- Updated scx_dispatch_sched() to handle both nested and non-nested invocations with appropriate kf_mask handling
The qmap scheduler is updated to demonstrate the functionality by calling scx_bpf_sub_dispatch() on registered child schedulers when it has no tasks in its own queues.
--- kernel/sched/ext.c | 116 ++++++++++++++++++++--- kernel/sched/sched.h | 3 + tools/sched_ext/include/scx/common.bpf.h | 2 + tools/sched_ext/scx_qmap.bpf.c | 37 +++++++- 4 files changed, 145 insertions(+), 13 deletions(-)
kernel/sched/ext.c +104 -12
@@ -2253,8 +2253,14 @@ static void flush_dispatch_buf(struct scx_sched *sch, struct rq *rq) dspc->cursor =0 ; } -static bool scx_dispatch_sched (struct scx_sched *sch,struct rq *rq, -struct task_struct *prev) +/* +* One user of this function is scx_bpf_dispatch() which can be called +* recursively as sub-sched dispatches nest. Always inline to reduce stack usage +* from the call frame. +*/ +static __always_inline bool +scx_dispatch_sched (struct scx_sched *sch,struct rq *rq, +struct task_struct *prev,bool nested) {struct scx_dsp_ctx *dspc = &this_cpu_ptr (sch->pcpu)->dsp_ctx;int nr_loops =SCX_DSP_MAX_LOOPS ;
@@ -2280,8 +2286,23 @@ static bool scx_dispatch_sched(struct scx_sched *sch, struct rq *rq, do { dspc->nr_tasks = 0; - SCX_CALL_OP(sch, SCX_KF_DISPATCH, dispatch, rq, - cpu_of(rq), prev_on_sch ? prev : NULL); + if (nested) { + /* + * If nested, don't update kf_mask as the originating + * invocation would already have set it up. + */ + SCX_CALL_OP(sch, 0, dispatch, rq, + cpu_of(rq), prev_on_sch ? prev : NULL); + } else { + /* + * If not nested, stash @prev so that nested invocations + * can access it. + */ + rq->scx.sub_dispatch_prev = prev; + SCX_CALL_OP(sch, SCX_KF_DISPATCH, dispatch, rq, + cpu_of(rq), prev_on_sch ? prev : NULL); + rq->scx.sub_dispatch_prev = NULL; + } flush_dispatch_buf(sch, rq);
@@ -2314,7 +2335,7 @@ static bool scx_dispatch_sched(struct scx_sched *sch, struct rq *rq,static int balance_one (struct rq *rq,struct task_struct *prev) { -struct scx_sched *sch = scx_root, *pos; +struct scx_sched *sch = scx_root;lockdep_assert_rq_held (rq); rq->scx.flags |=SCX_RQ_IN_BALANCE ;
@@ -2358,13 +2379,8 @@ static int balance_one(struct rq *rq, struct task_struct *prev)if (rq->scx.local_dsq.nr) goto has_tasks; -/* -* TEMPORARY - Dispatch all scheds. This will be replaced by BPF-driven -* hierarchical operation. -*/ -list_for_each_entry_rcu (pos, &scx_sched_all, all) -if (scx_dispatch_sched (pos, rq, prev)) - goto has_tasks; +if (scx_dispatch_sched (sch, rq, prev, false)) + goto has_tasks;/* * Didn't find another task to run. Keep running @prev unless
@@ -5883,6 +5899,20 @@ static int bpf_scx_init_member(const struct btf_type *t,return 0 ; } +#ifdef CONFIG_EXT_SUB_SCHED +static void scx_pstack_recursion_on_dispatch (struct bpf_prog *prog) +{ +struct scx_sched *sch; + +guard (rcu)(); + sch =scx_prog_sched (prog->aux); +if (unlikely (!sch)) +return ; + +scx_error (sch,"dispatch recursion detected" ); +} +#endif /* CONFIG_EXT_SUB_SCHED */ +static int bpf_scx_check_member (const struct btf_type *t,const struct btf_member *member,const struct bpf_prog *prog)
@@ -5908,6 +5938,22 @@ static int bpf_scx_check_member(const struct btf_type *t,return -EINVAL ; } +#ifdef CONFIG_EXT_SUB_SCHED +/* +* Enable private stack for operations that can nest along the +* hierarchy. +* +* XXX - Ideally, we should only do this for scheds that allow +* sub-scheds and sub-scheds themselves but I don't know how to access +* struct_ops from here. +*/ +switch (moff) { +case offsetof(struct sched_ext_ops , dispatch): + prog->aux->priv_stack_requested = true; + prog->aux->recursion_detected = scx_pstack_recursion_on_dispatch; + } +#endif /* CONFIG_EXT_SUB_SCHED */ +return 0 ; }
@@ -6799,6 +6845,49 @@ __bpf_kfunc bool scx_bpf_dsq_move_vtime(struct bpf_iter_scx_dsq *it__iter, p, dsq_id, enq_flags | SCX_ENQ_DSQ_PRIQ); } +#ifdef CONFIG_EXT_SUB_SCHED +/** +* scx_bpf_sub_dispatch - Trigger dispatching on a child scheduler +* @cgroup_id: cgroup ID of the child scheduler to dispatch +* @aux__prog: magic BPF argument to access bpf_prog_aux hidden from BPF progs +* +* Allows a parent scheduler to trigger dispatching on one of its direct +* child schedulers. The child scheduler runs its dispatch operation to +* move tasks from dispatch queues to the local runqueue. +* +* Returns: true on success, false if cgroup_id is invalid, not a direct +* child, or caller lacks dispatch permission. +*/ +__bpf_kfunc bool scx_bpf_sub_dispatch(u64 cgroup_id, +const struct bpf_prog_aux *aux__prog) +{ +struct rq *this_rq =this_rq (); +struct scx_sched *parent, *child; + +guard (rcu)(); + parent =scx_prog_sched (aux__prog); +if (unlikely (!parent)) +return false; + +if (!scx_kf_allowed (parent,SCX_KF_DISPATCH )) +return false; + + child =scx_find_sub_sched (cgroup_id); + +if (unlikely (!child)) +return false; + +if (unlikely (scx_parent (child) != parent)) { +scx_error (parent,"trying to dispatch a distant sub-sched on cgroup %llu" , + cgroup_id); +return false; + } + +return scx_dispatch_sched (child, this_rq, this_rq->scx.sub_dispatch_prev, + true); +} +#endif /* CONFIG_EXT_SUB_SCHED */ +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (scx_kfunc_ids_dispatch)
@@ -6809,6 +6898,9 @@ BTF_ID_FLAGS(func, scx_bpf_dsq_move_set_slice) BTF_ID_FLAGS(func, scx_bpf_dsq_move_set_vtime) BTF_ID_FLAGS(func, scx_bpf_dsq_move, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_dsq_move_vtime, KF_RCU) +#ifdef CONFIG_EXT_SUB_SCHED +BTF_ID_FLAGS(func, scx_bpf_sub_dispatch) +#endif BTF_KFUNCS_END (scx_kfunc_ids_dispatch) static const struct btf_kfunc_id_set scx_kfunc_set_dispatch = {
kernel/sched/sched.h +3 -0
@@ -777,6 +777,9 @@ struct scx_rq {cpumask_var_t cpus_to_preempt;cpumask_var_t cpus_to_wait;unsigned long pnt_seq; + +struct task_struct *sub_dispatch_prev; +struct balance_callback deferred_bal_cb;struct irq_work deferred_irq_work;struct irq_work kick_cpus_irq_work;
tools/sched_ext/include/scx/common.bpf.h +2 -0
@@ -140,6 +140,8 @@ struct cgroup *scx_bpf_task_cgroup(struct task_struct *p, const struct bpf_prog_#define scx_bpf_task_cgroup (p) scx_bpf_task_cgroup((p), NULL) u64 scx_bpf_now(void) __ksym __weak; void scx_bpf_events(struct scx_event_stats *events, size_t events__sz) __ksym __weak; +bool scx_bpf_sub_dispatch(u64 cgroup_id, const struct bpf_prog_aux *aux__prog) __ksym __weak; +#define scx_bpf_sub_dispatch (cgroup_id) scx_bpf_sub_dispatch((cgroup_id), NULL)/* * Use the following as @it__iter when calling scx_bpf_dsq_move[_vtime]() from
tools/sched_ext/scx_qmap.bpf.c +36 -1
@@ -48,6 +48,9 @@ const volatile bool suppress_dump;u64 nr_highpri_queued;u32 test_error_cnt; +#define MAX_SUB_SCHEDS 8 +u64 sub_sched_cgroup_ids[MAX_SUB_SCHEDS ]; +UEI_DEFINE (uei);struct qmap {
@@ -452,6 +455,12 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev) cpuc->dsp_cnt =0 ; } +for (i =0 ; i <MAX_SUB_SCHEDS ; i++) { +if (sub_sched_cgroup_ids[i] && +scx_bpf_sub_dispatch (sub_sched_cgroup_ids[i])) +return ; + } +/* * No other tasks. @prev will keep running. Update its core_sched_seq as * if the task were enqueued and dispatched immediately.
@@ -877,7 +886,32 @@ void BPF_STRUCT_OPS(qmap_exit, struct scx_exit_info *ei)s32 BPF_STRUCT_OPS (qmap_sub_attach ,struct scx_sub_attach_args *args) { -return 0 ; +int i; + +for (i =0 ; i <MAX_SUB_SCHEDS ; i++) { +if (!sub_sched_cgroup_ids[i]) { + sub_sched_cgroup_ids[i] = args->ops->sub_cgroup_id; +bpf_printk ("attaching sub-sched[%d] on %s" , + i, args->cgroup_path); +return 0 ; + } + } + +return -ENOSPC ; +} + +void BPF_STRUCT_OPS (qmap_sub_detach ,struct scx_sub_detach_args *args) +{ +int i; + +for (i =0 ; i <MAX_SUB_SCHEDS ; i++) { +if (sub_sched_cgroup_ids[i] == args->ops->sub_cgroup_id) { + sub_sched_cgroup_ids[i] =0 ; +bpf_printk ("detaching sub-sched[%d] on %s" , + i, args->cgroup_path); +break ; + } + } } SCX_OPS_DEFINE(qmap_ops,
@@ -896,6 +930,7 @@ SCX_OPS_DEFINE(qmap_ops, .cgroup_set_weight = (void *)qmap_cgroup_set_weight, .cgroup_set_bandwidth = (void *)qmap_cgroup_set_bandwidth, .sub_attach = (void *)qmap_sub_attach, + .sub_detach = (void *)qmap_sub_detach, .cpu_online = (void *)qmap_cpu_online, .cpu_offline = (void *)qmap_cpu_offline, .init = (void *)qmap_init,
Signature · 1 line
2.51.0