sched_ext: Bandwidth-limited rescue execution for stranded tasks
Subsystems (2)
Tejun Heo
reviewerPATCH 1/1 sched_ext: Bandwidth-limited rescue execution for stranded tasks
Hello,
A sub-scheduler holds only the cids its parent granted and nothing guarantees that they cover its tasks' affinities. A task that can run on none of them currently has no good ending: the scheduler's insert is cap-rejected and reenqueued until the repeat limit ejects the scheduler, or the task stalls into the watchdog.
This series adds kernel-side rescue execution. A scheduler marks an insert that may be cap-rejected with SCX_ENQ_RESCUE and the kernel, instead of bouncing it, runs the task on the target cpu at a small configured bandwidth. A rescue starts out non-disruptive. Schedulers that hold caps on the cid largely stay in control of the cpu - a preemption cap, for example, still lets its holder preempt the rescuee. When a rescue goes unserved for too long, it escalates into protected execution.
When a cpu's rescue queue stays oversaturated past the overload threshold, the kernel ejects the sub-scheduler with the highest recent rescue consumption on that cpu instead of misblaming the waiter's owner.
- 0001-0005 are preps: naming, helper visibility, dsq-move flag vetting and the IGNORE_CAPS preemption waiver.
- 0006-0007 synchronize slice/dsq_vtime writes at insertion commit and add SCX_TASK_PROTECTED, the groundwork that lets a kernel-granted slice survive scheduler writes.
- 0008-0009 add the rescue machinery and the overload ejection.
- 0010 syncs the tools autogen enum headers.
- 0011-0012 fix scx_qmap's pinned-task direct dispatch and add rescue support as the demonstration consumer.
0001-sched_ext-Rename-scx_local_or_reject_dsq-to-scx_reso.patch 0002-sched_ext-Make-several-ext.c-helpers-available-outsi.patch 0003-sched_ext-Factor-out-__scx_bpf_now.patch 0004-sched_ext-Reject-internal-enq_flags-in-the-dsq-move-.patch 0005-sched_ext-Make-SCX_ENQ_IGNORE_CAPS-waive-the-preempt.patch 0006-sched_ext-Synchronize-slice-and-dsq_vtime-writes.patch 0007-sched_ext-Add-SCX_TASK_PROTECTED.patch 0008-sched_ext-Add-bandwidth-limited-rescue-execution-for.patch 0009-sched_ext-Eject-the-top-rescue-consumer-on-overload.patch 0010-sched_ext-Sync-tools-autogen-enum-headers.patch 0011-sched_ext-scx_qmap-Idle-check-pinned-tasks-before-di.patch 0012-sched_ext-scx_qmap-Add-rescue-support.patch
Based on sched_ext/for-7.3 (ee7aece60817).
Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-rescue-v1
include/linux/sched/ext.h | 7 + kernel/sched/ext/ext.c | 690 ++++++++++++++++-------- kernel/sched/ext/internal.h | 78 ++- kernel/sched/ext/sub.c | 496 ++++++++++++++++- kernel/sched/ext/sub.h | 41 +- kernel/sched/ext/types.h | 12 + kernel/sched/sched.h | 13 + tools/sched_ext/include/scx/enum_defs.autogen.h | 16 + tools/sched_ext/include/scx/enums.autogen.bpf.h | 3 + tools/sched_ext/include/scx/enums.autogen.h | 1 + tools/sched_ext/scx_qmap.bpf.c | 88 +-- tools/sched_ext/scx_qmap.c | 23 +- tools/sched_ext/scx_qmap.h | 1 + 13 files changed, 1186 insertions(+), 283 deletions(-)
--
tejun
Tejun Heo
reviewerPATCH 1/12 sched_ext: Rename scx_local_or_reject_dsq() to scx_resolve_local_dsq()
The following rescue execution addition gives the function a third possible destination, making a name that enumerates the outcomes a poor fit. Rename to the destination-neutral scx_resolve_local_dsq(). No functional changes.
--- kernel/sched/ext/ext.c | 4 ++-- kernel/sched/ext/sub.c | 6 +++--- kernel/sched/ext/sub.h | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-)
kernel/sched/ext/ext.c +2 -2
@@ -1507,7 +1507,7 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq,bool is_rq_owned = false;if (dsq->id ==SCX_DSQ_LOCAL ) { - dsq =scx_local_or_reject_dsq (sch, rq, p, &enq_flags); + dsq =scx_resolve_local_dsq (sch, rq, p, &enq_flags); is_rq_owned = true; }
@@ -2279,7 +2279,7 @@ static void move_local_task_to_local_dsq(struct scx_sched *sch,struct scx_dispatch_q *src_dsq, struct rq *dst_rq) { - struct scx_dispatch_q *dst_dsq = scx_local_or_reject_dsq(sch, dst_rq, p, &enq_flags); + struct scx_dispatch_q *dst_dsq = scx_resolve_local_dsq(sch, dst_rq, p, &enq_flags);/* @dsq is locked and @p is on @dst_rq */ lockdep_assert_held (&src_dsq->lock);
kernel/sched/ext/sub.c +3 -3
@@ -230,7 +230,7 @@ void scx_init_root_caps(struct scx_sched *sch) }/** -* scx_local_or_reject_dsq - Pick the local or reject DSQ for an insert +* scx_resolve_local_dsq - Pick the local or reject DSQ for an insert * @sch: enqueuing sub-sched * @rq: rq whose local DSQ @p targets * @p: task being inserted
@@ -246,8 +246,8 @@ void scx_init_root_caps(struct scx_sched *sch)* to and run by its nearest non-bypassing ancestor. If root is bypassing, it * always holds all caps. */ -struct scx_dispatch_q *scx_local_or_reject_dsq (struct scx_sched *sch,struct rq *rq, -struct task_struct *p,u64 *enq_flags) +struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq, +struct task_struct *p,u64 *enq_flags) {if (!scx_has_subs ())return &rq->scx.local_dsq;
kernel/sched/ext/sub.h +3 -3
@@ -34,8 +34,8 @@ void scx_online_ecaps(struct rq *rq);void scx_offline_ecaps (struct rq *rq);void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu);void scx_discard_stale_ecaps_syncs (void ); -struct scx_dispatch_q *scx_local_or_reject_dsq (struct scx_sched *sch,struct rq *rq, -struct task_struct *p,u64 *enq_flags); +struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq, +struct task_struct *p,u64 *enq_flags);bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p);void scx_reenq_reject (struct rq *rq);
@@ -84,7 +84,7 @@ static inline void scx_online_ecaps(struct rq *rq) {}static inline void scx_offline_ecaps (struct rq *rq) {}static inline void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu) {}static inline void scx_discard_stale_ecaps_syncs (void ) {} -static inline struct scx_dispatch_q *scx_local_or_reject_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) {return &rq->scx.local_dsq; } +static inline struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) {return &rq->scx.local_dsq; }static inline bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p) {return false; }static inline void scx_reenq_reject (struct rq *rq) {}static inline void scx_dec_has_subs (struct scx_sched *sch) {}
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 2/12 sched_ext: Make several ext.c helpers available outside ext.c
In reply to Tejun Heoset_task_slice(), task_unlink_from_dsq(), move_local_task_to_local_dsq(), init_dsq() and dump_line() will be used outside ext.c. Add the scx_ prefix and declare them in internal.h. The scx_sched_all list will also be used outside ext.c, drop its static. No functional changes.
--- kernel/sched/ext/ext.c | 172 +++++++++++++++++------------------- kernel/sched/ext/internal.h | 8 ++ 2 files changed, 90 insertions(+), 90 deletions(-)
kernel/sched/ext/ext.c +82 -90
@@ -38,7 +38,7 @@ struct scx_sched __rcu *scx_root;* All scheds, writers must hold both scx_enable_mutex and scx_sched_lock. * Readers can hold either or rcu_read_lock(). */ -static LIST_HEAD (scx_sched_all); +LIST_HEAD (scx_sched_all);#ifdef CONFIG_EXT_SUB_SCHED const struct rhashtable_params scx_sched_hash_params = {
@@ -900,7 +900,7 @@ struct task_struct *scx_task_iter_next_locked(struct scx_task_iter *iter)* @kind: a kind of event to dump */ #define scx_dump_event (s, events, kind) do { \ - dump_line(&(s), "%40s: %16lld", #kind, (events)->kind); \ + scx_dump_line(&(s), "%40s: %16lld", #kind, (events)->kind); \ } while (0)
@@ -1248,7 +1248,7 @@ static void set_task_slice_keep_oob(struct task_struct *p, u64 slice) }/* set @p's slice, superseding any pending out-of-band request */ -static void set_task_slice (struct task_struct *p,u64 slice) +void scx_set_task_slice (struct task_struct *p,u64 slice) {set_task_slice_keep_oob (p, slice);clear_task_slice_oob (p);
@@ -1495,7 +1495,7 @@ static void rq_owned_post_enq(struct scx_sched *sch, struct rq *rq,if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->curr && rq->curr->sched_class == &ext_sched_class) { -set_task_slice (rq->curr,0 ); +scx_set_task_slice (rq->curr,0 );resched_curr (rq); } }
@@ -1637,8 +1637,7 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq,atomic_long_set_release (&p->scx.ops_state,SCX_OPSS_NONE ); } -static void task_unlink_from_dsq (struct task_struct *p, -struct scx_dispatch_q *dsq) +void scx_task_unlink_from_dsq (struct task_struct *p,struct scx_dispatch_q *dsq) {WARN_ON_ONCE (list_empty (&p->scx.dsq_list.node));
@@ -1695,7 +1694,7 @@ void scx_dispatch_dequeue(struct rq *rq, struct task_struct *p)*/ if (p->scx.holding_cpu <0 ) {/* @p must still be on @dsq, dequeue */ -task_unlink_from_dsq (p, dsq); +scx_task_unlink_from_dsq (p, dsq); }else {/* * We're racing against dispatch_to_local_dsq() which already
@@ -1722,7 +1721,7 @@ static void dispatch_dequeue_locked(struct task_struct *p,lockdep_assert_rq_held (task_rq (p));lockdep_assert_held (&dsq->lock); -task_unlink_from_dsq (p, dsq); +scx_task_unlink_from_dsq (p, dsq); p->scx.dsq =NULL ; }
@@ -2236,7 +2235,7 @@ static void yield_task_scx(struct rq *rq)if (SCX_HAS_OP (sch, yield))SCX_CALL_OP_2TASKS_RET (sch, yield, rq, p,NULL );else -set_task_slice (p,0 ); +scx_set_task_slice (p,0 ); }static bool yield_to_task_scx (struct rq *rq,struct task_struct *to)
@@ -2274,10 +2273,9 @@ static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p, int wake_flscx_schedule_reenq_local (rq,0 ); } -static void move_local_task_to_local_dsq (struct scx_sched *sch, -struct task_struct *p,u64 enq_flags, -struct scx_dispatch_q *src_dsq, -struct rq *dst_rq) +void scx_move_local_task_to_local_dsq (struct scx_sched *sch,struct task_struct *p, +u64 enq_flags,struct scx_dispatch_q *src_dsq, +struct rq *dst_rq) {struct scx_dispatch_q *dst_dsq =scx_resolve_local_dsq (sch, dst_rq, p, &enq_flags);
@@ -2458,7 +2456,7 @@ static bool unlink_dsq_and_switch_rq_lock(struct task_struct *p,lockdep_assert_rq_held (locked_rq);WARN_ON_ONCE (p->scx.holding_cpu >=0 ); -task_unlink_from_dsq (p, dsq); +scx_task_unlink_from_dsq (p, dsq); p->scx.holding_cpu = cpu;raw_spin_unlock (&dsq->lock);
@@ -2529,9 +2527,8 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch,if (dst_dsq->id ==SCX_DSQ_LOCAL ) {/* @p is going from a non-local DSQ to a local DSQ */ if (src_rq == dst_rq) { -task_unlink_from_dsq (p, src_dsq); -move_local_task_to_local_dsq (sch, p, enq_flags, - src_dsq, dst_rq); +scx_task_unlink_from_dsq (p, src_dsq); +scx_move_local_task_to_local_dsq (sch, p, enq_flags, src_dsq, dst_rq);raw_spin_unlock (&src_dsq->lock); }else {raw_spin_unlock (&src_dsq->lock);
@@ -2581,8 +2578,8 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq,break ;if (rq == task_rq) { -task_unlink_from_dsq (p, dsq); -move_local_task_to_local_dsq (sch, p, enq_flags, dsq, rq); +scx_task_unlink_from_dsq (p, dsq); +scx_move_local_task_to_local_dsq (sch, p, enq_flags, dsq, rq);raw_spin_unlock (&dsq->lock);return true; }
@@ -3530,7 +3527,7 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)* we can't trust the slice management or ops.core_sched_before(). */ if (scx_bypassing (sch,cpu_of (rq))) { -set_task_slice (curr,0 ); +scx_set_task_slice (curr,0 );touch_core_sched (rq, curr); }else if (SCX_HAS_OP (sch, tick)) {SCX_CALL_OP_TASK (sch, tick, rq, curr);
@@ -3685,7 +3682,7 @@ static void scx_disable_task(struct scx_sched *sch, struct task_struct *p)* control, after ops.disable() has observed their final values. */ p->scx.dsq_vtime =0 ; -set_task_slice (p,0 ); +scx_set_task_slice (p,0 ); p->scx.reenq_cnt =0 ;/*
@@ -4184,7 +4181,7 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)if ((reenq_flags &SCX_REENQ_CAP_REVOKE ) && rq->curr->sched_class == &ext_sched_class &&scx_task_reenq_on_cap_revoke (rq, rq->curr)) { -set_task_slice (rq->curr,0 ); +scx_set_task_slice (rq->curr,0 );resched_curr (rq); }
@@ -4748,8 +4745,7 @@ DEFINE_SCHED_CLASS(ext) = {#endif }; -static s32 init_dsq (struct scx_dispatch_q *dsq,u64 dsq_id, -struct scx_sched *sch) +s32 scx_init_dsq (struct scx_dispatch_q *dsq,u64 dsq_id,struct scx_sched *sch) {s32 cpu;
@@ -6375,7 +6371,7 @@ static void dump_newline(struct seq_buf *s)seq_buf_putc (s,'\n' ); } -static __printf(2, 3) void dump_line(struct seq_buf *s, const char *fmt, ...) +__printf(2, 3) void scx_dump_line(struct seq_buf *s, const char *fmt, ...) {va_list args;
@@ -6407,7 +6403,7 @@ static void dump_stack_trace(struct seq_buf *s, const char *prefix,unsigned int i;for (i =0 ; i < len; i++) -dump_line (s,"%s%pS" , prefix, (void *)bt[i]); +scx_dump_line (s,"%s%pS" , prefix, (void *)bt[i]); }static void ops_dump_init (struct seq_buf *s,const char *prefix)
@@ -6457,7 +6453,7 @@ static void ops_dump_flush(void)*/ c = *end; *end ='\0' ; -dump_line (dd->s,"%s%s" , dd->prefix, line); +scx_dump_line (dd->s,"%s%s" , dd->prefix, line);if (c =='\0' )break ;
@@ -6501,21 +6497,19 @@ static void scx_dump_task(struct scx_sched *sch, struct seq_buf *s, struct scx_d (unsigned long long)p->scx.dsq->id);dump_newline (s); -dump_line (s," %c%c %s[%d] %s%s %+ldms" , - marker,task_state_to_char (p), p->comm, p->pid, - own_marker, sch_id_buf, -jiffies_delta_msecs (p->scx.runnable_at, dctx->at_jiffies)); -dump_line (s," scx_state/flags=%u/0x%x dsq_flags=0x%x ops_state/qseq=%lu/%lu" , -scx_get_task_state (p) >>SCX_TASK_STATE_SHIFT , - p->scx.flags & ~SCX_TASK_STATE_MASK , - p->scx.dsq_flags, ops_state &SCX_OPSS_STATE_MASK , - ops_state >>SCX_OPSS_QSEQ_SHIFT ); -dump_line (s," sticky/holding_cpu=%d/%d dsq_id=%s" , - p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf); -dump_line (s," dsq_vtime=%llu slice=%llu weight=%u" , - p->scx.dsq_vtime, p->scx.slice, p->scx.weight); -dump_line (s," cpus=%*pb no_mig=%u" ,cpumask_pr_args (p->cpus_ptr), - p->migration_disabled); +scx_dump_line (s," %c%c %s[%d] %s%s %+ldms" , + marker,task_state_to_char (p), p->comm, p->pid, own_marker, sch_id_buf, +jiffies_delta_msecs (p->scx.runnable_at, dctx->at_jiffies)); +scx_dump_line (s," scx_state/flags=%u/0x%x dsq_flags=0x%x ops_state/qseq=%lu/%lu" , +scx_get_task_state (p) >>SCX_TASK_STATE_SHIFT , + p->scx.flags & ~SCX_TASK_STATE_MASK , p->scx.dsq_flags, + ops_state &SCX_OPSS_STATE_MASK , ops_state >>SCX_OPSS_QSEQ_SHIFT ); +scx_dump_line (s," sticky/holding_cpu=%d/%d dsq_id=%s" , + p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf); +scx_dump_line (s," dsq_vtime=%llu slice=%llu weight=%u" , + p->scx.dsq_vtime, p->scx.slice, p->scx.weight); +scx_dump_line (s," cpus=%*pb no_mig=%u" ,cpumask_pr_args (p->cpus_ptr), + p->migration_disabled);if (SCX_HAS_OP (sch, dump_task)) {ops_dump_init (s," " );
@@ -6563,28 +6557,26 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,seq_buf_init (&ns, buf, avail);dump_newline (&ns); -dump_line (&ns,"CPU %-4d: nr_run=%u flags=0x%x cpu_rel=%d ops_qseq=%lu ksync=%lu" , - cpu, rq->scx.nr_running, rq->scx.flags, - rq->scx.cpu_released, rq->scx.ops_qseq, - rq->scx.kick_sync); -dump_line (&ns," curr=%s[%d] class=%ps" , - rq->curr->comm, rq->curr->pid, - rq->curr->sched_class); +scx_dump_line (&ns,"CPU %-4d: nr_run=%u flags=0x%x cpu_rel=%d ops_qseq=%lu ksync=%lu" , + cpu, rq->scx.nr_running, rq->scx.flags, rq->scx.cpu_released, + rq->scx.ops_qseq, rq->scx.kick_sync); +scx_dump_line (&ns," curr=%s[%d] class=%ps" , + rq->curr->comm, rq->curr->pid, rq->curr->sched_class);if (!cpumask_empty (pcpu->cpus_to_kick)) -dump_line (&ns," cpus_to_kick : %*pb" , -cpumask_pr_args (pcpu->cpus_to_kick)); +scx_dump_line (&ns," cpus_to_kick : %*pb" , +cpumask_pr_args (pcpu->cpus_to_kick));if (!cpumask_empty (pcpu->cpus_to_kick_if_idle)) -dump_line (&ns," idle_to_kick : %*pb" , -cpumask_pr_args (pcpu->cpus_to_kick_if_idle)); +scx_dump_line (&ns," idle_to_kick : %*pb" , +cpumask_pr_args (pcpu->cpus_to_kick_if_idle));if (!cpumask_empty (pcpu->cpus_to_preempt)) -dump_line (&ns," cpus_to_preempt: %*pb" , -cpumask_pr_args (pcpu->cpus_to_preempt)); +scx_dump_line (&ns," cpus_to_preempt: %*pb" , +cpumask_pr_args (pcpu->cpus_to_preempt));if (!cpumask_empty (pcpu->cpus_to_wait)) -dump_line (&ns," cpus_to_wait : %*pb" , -cpumask_pr_args (pcpu->cpus_to_wait)); +scx_dump_line (&ns," cpus_to_wait : %*pb" , +cpumask_pr_args (pcpu->cpus_to_wait));if (!cpumask_empty (rq->scx.cpus_to_sync)) -dump_line (&ns," cpus_to_sync : %*pb" , -cpumask_pr_args (rq->scx.cpus_to_sync)); +scx_dump_line (&ns," cpus_to_sync : %*pb" , +cpumask_pr_args (rq->scx.cpus_to_sync)); used =seq_buf_used (&ns);if (SCX_HAS_OP (sch, dump_cpu)) {
@@ -6652,25 +6644,25 @@ static void scx_dump_state(struct scx_sched *sch, struct scx_exit_info *ei,#ifdef CONFIG_EXT_SUB_SCHED if (sch->level ==0 ) -dump_line (&s,"%s: root" , sch->ops.name); +scx_dump_line (&s,"%s: root" , sch->ops.name);else -dump_line (&s,"%s: sub%d-%llu %s" , - sch->ops.name, sch->level, sch->ops.sub_cgroup_id, - sch->cgrp_path); +scx_dump_line (&s,"%s: sub%d-%llu %s" , + sch->ops.name, sch->level, sch->ops.sub_cgroup_id, + sch->cgrp_path);#endif if (ei->kind ==SCX_EXIT_NONE ) { -dump_line (&s,"Debug dump triggered by %s" , ei->reason); +scx_dump_line (&s,"Debug dump triggered by %s" , ei->reason); }else {if (ei->exit_cpu >=0 ) -dump_line (&s,"%s[%d] triggered exit kind %d on CPU %d:" , - current->comm, current->pid, ei->kind, - ei->exit_cpu); +scx_dump_line (&s,"%s[%d] triggered exit kind %d on CPU %d:" , + current->comm, current->pid, ei->kind, + ei->exit_cpu);else -dump_line (&s,"%s[%d] triggered exit kind %d:" , - current->comm, current->pid, ei->kind); -dump_line (&s," %s (%s)" , ei->reason, ei->msg); +scx_dump_line (&s,"%s[%d] triggered exit kind %d:" , + current->comm, current->pid, ei->kind); +scx_dump_line (&s," %s (%s)" , ei->reason, ei->msg);dump_newline (&s); -dump_line (&s,"Backtrace:" ); +scx_dump_line (&s,"Backtrace:" );dump_stack_trace (&s," " , ei->bt, ei->bt_len); }
@@ -6681,8 +6673,8 @@ static void scx_dump_state(struct scx_sched *sch, struct scx_exit_info *ei, }dump_newline (&s); -dump_line (&s,"CPU states" ); -dump_line (&s,"----------" ); +scx_dump_line (&s,"CPU states" ); +scx_dump_line (&s,"----------" );/* * Dump stalled CPUs first so they aren't lost to dump truncation, then
@@ -6705,8 +6697,8 @@ static void scx_dump_state(struct scx_sched *sch, struct scx_exit_info *ei, }dump_newline (&s); -dump_line (&s,"Event counters" ); -dump_line (&s,"--------------" ); +scx_dump_line (&s,"Event counters" ); +scx_dump_line (&s,"--------------" );scx_read_events (sch, &events);#define SCX_EVENT (name) scx_dump_event(s, &events, name)
@@ -6815,7 +6807,7 @@ static struct scx_sched_pnode *alloc_pnode(struct scx_sched *sch, int node)if (!pnode)return NULL ; -if (init_dsq (&pnode->global_dsq,SCX_DSQ_GLOBAL , sch)) { +if (scx_init_dsq (&pnode->global_dsq,SCX_DSQ_GLOBAL , sch)) {kfree (pnode);return NULL ; }
@@ -6876,7 +6868,7 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, }for_each_possible_cpu (cpu) { - ret =init_dsq (scx_bypass_dsq (sch, cpu),SCX_DSQ_BYPASS , sch); + ret =scx_init_dsq (scx_bypass_dsq (sch, cpu),SCX_DSQ_BYPASS , sch);if (ret) { bypass_fail_cpu = cpu; goto err_free_pcpu;
@@ -7509,7 +7501,7 @@ static void scx_root_enable_workfn(struct kthread_work *work) queue_flags |=DEQUEUE_CLASS ; scoped_guard (sched_change, p, queue_flags) { -set_task_slice (p,READ_ONCE (sch->slice_dfl)); +scx_set_task_slice (p,READ_ONCE (sch->slice_dfl)); p->sched_class = new_class; } }
@@ -8159,7 +8151,7 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_rif (cur_class == &ext_sched_class) {if (likely (!scx_missing_caps (pcpu->sch, cpu,scx_caps_for_preempt (pcpu->sch, rq)))) -set_task_slice (rq->curr,0 ); +scx_set_task_slice (rq->curr,0 );else __scx_add_event (pcpu->sch,SCX_EV_SUB_PREEMPT_DENIED ,1 );
@@ -8363,9 +8355,9 @@ void __init init_sched_ext_class(void)int n =cpu_to_node (cpu);/* local_dsq's sch will be set during scx_root_enable() */ -BUG_ON (init_dsq (&rq->scx.local_dsq,SCX_DSQ_LOCAL ,NULL )); +BUG_ON (scx_init_dsq (&rq->scx.local_dsq,SCX_DSQ_LOCAL ,NULL ));#ifdef CONFIG_EXT_SUB_SCHED -BUG_ON (init_dsq (&rq->scx.reject_dsq,SCX_DSQ_REJECT ,NULL )); +BUG_ON (scx_init_dsq (&rq->scx.reject_dsq,SCX_DSQ_REJECT ,NULL ));#endif INIT_LIST_HEAD (&rq->scx.runnable_list);
@@ -8522,7 +8514,7 @@ __bpf_kfunc bool scx_bpf_dsq_insert___v2(struct task_struct *p, u64 dsq_id,return false;if (slice) -set_task_slice (p, slice); +scx_set_task_slice (p, slice);else set_task_slice_keep_oob (p, p->scx.slice ?:1 );
@@ -8548,7 +8540,7 @@ static bool scx_dsq_insert_vtime(struct scx_sched *sch, struct task_struct *p,return false;if (slice) -set_task_slice (p, slice); +scx_set_task_slice (p, slice);else set_task_slice_keep_oob (p, p->scx.slice ?:1 );
@@ -8732,7 +8724,7 @@ static bool scx_dsq_move(struct bpf_iter_scx_dsq_kern *kit,if (kit->cursor.flags & __SCX_DSQ_ITER_HAS_VTIME) p->scx.dsq_vtime = kit->vtime;if (kit->cursor.flags & __SCX_DSQ_ITER_HAS_SLICE) -set_task_slice (p, kit->slice); +scx_set_task_slice (p, kit->slice);/* execute move */ locked_rq =move_task_between_dsqs (sch, p, enq_flags, src_dsq, dst_dsq);
@@ -9055,10 +9047,10 @@ __bpf_kfunc s32 scx_bpf_create_dsq(u64 dsq_id, s32 node, const struct bpf_prog_areturn -ENOMEM ;/* -* init_dsq() must be called in GFP_KERNEL context. Init it with NULL -* @sch and update afterwards. +* scx_init_dsq() must be called in GFP_KERNEL context. Init it with +* NULL @sch and update afterwards. */ - ret =init_dsq (dsq, dsq_id,NULL ); + ret =scx_init_dsq (dsq, dsq_id,NULL );if (ret) {kfree (dsq);return ret;
@@ -9152,7 +9144,7 @@ __bpf_kfunc bool scx_bpf_task_set_slice(struct task_struct *p, u64 slice, unlikely(scx_missing_caps(sch, cpu_of(locked_rq), SCX_CAP_BASE)))__scx_add_event (sch,SCX_EV_SLICE_DENIED ,1 ); else -set_task_slice (p ,slice ); +scx_set_task_slice (p ,slice );return true; }
@@ -9701,8 +9693,8 @@ __bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data, ret =__bstr_format (sch, buf->data, buf->line + dd->cursor,sizeof (buf->line) - dd->cursor, fmt, data, data__sz);if (ret <0 ) { -dump_line (dd->s,"%s[!] (\"%s\", %p, %u) failed to format (%d)" , - dd->prefix, fmt, data, data__sz, ret); +scx_dump_line (dd->s,"%s[!] (\"%s\", %p, %u) failed to format (%d)" , + dd->prefix, fmt, data, data__sz, ret);return ; }
kernel/sched/ext/internal.h +8 -0
@@ -1968,14 +1968,21 @@ void scx_task_iter_start(struct scx_task_iter *iter, struct cgroup *cgrp);void scx_task_iter_unlock (struct scx_task_iter *iter);void scx_task_iter_stop (struct scx_task_iter *iter);struct task_struct *scx_task_iter_next_locked (struct scx_task_iter *iter); +void scx_set_task_slice (struct task_struct *p,u64 slice); +void scx_task_unlink_from_dsq (struct task_struct *p,struct scx_dispatch_q *dsq);void scx_dispatch_dequeue (struct rq *rq,struct task_struct *p);void scx_do_enqueue_task (struct rq *rq,struct task_struct *p,u64 enq_flags,int sticky_cpu); +void scx_move_local_task_to_local_dsq (struct scx_sched *sch,struct task_struct *p, +u64 enq_flags,struct scx_dispatch_q *src_dsq, +struct rq *dst_rq);bool scx_consume_dispatch_q (struct scx_sched *sch,struct rq *rq,struct scx_dispatch_q *dsq,u64 enq_flags);bool scx_consume_global_dsq (struct scx_sched *sch,struct rq *rq);bool scx_rq_online (struct rq *rq);void scx_flush_dispatch_buf (struct scx_sched *sch,struct rq *rq); +s32 scx_init_dsq (struct scx_dispatch_q *dsq,u64 dsq_id,struct scx_sched *sch); +__printf(2, 3) void scx_dump_line(struct seq_buf *s, const char *fmt, ...);void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags);void schedule_dsq_reenq (struct scx_sched *sch,struct scx_dispatch_q *dsq,u64 reenq_flags,struct rq *locked_rq);
@@ -2014,6 +2021,7 @@ extern bool scx_cgroup_enabled;#ifdef CONFIG_EXT_SUB_SCHED extern const struct rhashtable_params scx_sched_hash_params;extern struct rhashtable scx_sched_hash; +extern struct list_head scx_sched_all;extern struct scx_sched *scx_enabling_sub_sched;#endif
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 02/12] sched_ext: Make several ext.c helpers available outside ext.c
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Unconditional definition of `scx_sched_all` lacks a prototype when `CONFIG_EXT_SUB_SCHED` is…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Unconditional definition of scx_sched_all lacks a prototype when CONFIG_EXT_SUB_SCHED is disabled, causing a build warning. --
--- Patch [2]: [PATCH 02/12] sched_ext: Make several ext.c helpers available outside ext.c --- commit 2fbc86143bb66368e04e606cb09e7ffe2b987b3d
Author: Tejun Heo <[email protected]>
sched_ext: Make several ext.c helpers available outside ext.c
This commit exposes several helper functions and the scx_sched_all list by removing their static keywords and adding declarations in internal.h. This allows them to be utilized outside of ext.c.
Quoted 8 lines
kernel/sched/ext/ext.c +1
-1@@ -38,7 +38,7 @@ struct scx_sched __rcu *scx_root;* All scheds, writers must hold both scx_enable_mutex and scx_sched_lock. * Readers can hold either or rcu_read_lock(). */ -static LIST_HEAD (scx_sched_all); +LIST_HEAD (scx_sched_all);
[ ... ]
kernel/sched/ext/internal.h extern bool scx_cgroup_enabled; · line 2024
Quoted 7 lines
kernel/sched/ext/internal.h +1
-0@@ -2014,6 +2021,7 @@ extern bool scx_cgroup_enabled;#ifdef CONFIG_EXT_SUB_SCHED extern const struct rhashtable_params scx_sched_hash_params;extern struct rhashtable scx_sched_hash; +extern struct list_head scx_sched_all;
[Severity: Low]
Will this trigger a missing prototype warning when CONFIG_EXT_SUB_SCHED is disabled?
Because scx_sched_all is defined unconditionally in ext.c, building the kernel without CONFIG_EXT_SUB_SCHED hides this extern declaration in internal.h from the compiler.
Should this declaration be moved outside of the ifdef block to prevent compiler or sparse warnings?
Quoted 2 lines
kernel/sched/ext/internal.h +0
-0extern struct scx_sched *scx_enabling_sub_sched;#endif
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Tejun Heo
reviewerPATCH 3/12 sched_ext: Factor out __scx_bpf_now()
In reply to Tejun Heoscx_bpf_now() couples the valid-or-fresh rq clock read to the current rq. The read is useful for kernel-internal timing against a specific rq, including a remotely locked one. Factor it out into __scx_bpf_now().
--- kernel/sched/ext/ext.c | 59 ++++++++++++++++++------------------- kernel/sched/ext/internal.h | 1 + 2 files changed, 30 insertions(+), 30 deletions(-)
kernel/sched/ext/ext.c +29 -30
@@ -10159,6 +10159,27 @@ __bpf_kfunc struct task_struct *scx_bpf_tid_to_task(u64 tid) return container_of(scx, struct task_struct, scx); } +u64 __scx_bpf_now (struct rq *rq) +{ +/* the caller must be on @rq's cpu or hold its lock */ +lockdep_assert ((rq ==this_rq () && !preemptible ()) || +lockdep_is_held (__rq_lockp (rq))); + +if (smp_load_acquire (&rq->scx.flags) &SCX_RQ_CLK_VALID ) { +/* if the rq clock is valid, use the cached rq clock */ +return READ_ONCE (rq->scx.clock); + }else { +/* +* Otherwise, return a fresh rq clock. +* +* The rq clock is updated outside of the rq lock. +* In this case, keep the updated rq clock invalid so the next +* read outside the rq lock gets a fresh rq clock. +*/ +return sched_clock_cpu (cpu_of (rq)); + } +} +/** * scx_bpf_now - Returns a high-performance monotonically non-decreasing * clock for the current CPU. The clock returned is in nanoseconds.
@@ -10189,36 +10210,14 @@ __bpf_kfunc struct task_struct *scx_bpf_tid_to_task(u64 tid)*/ __bpf_kfunc u64 scx_bpf_now(void) { -struct rq *rq; -u64 clock; - -preempt_disable (); - - rq =this_rq (); -if (smp_load_acquire (&rq->scx.flags) &SCX_RQ_CLK_VALID ) { -/* -* If the rq clock is valid, use the cached rq clock. -* -* Note that scx_bpf_now() is re-entrant between a process -* context and an interrupt context (e.g., timer interrupt). -* However, we don't need to consider the race between them -* because such race is not observable from a caller. -*/ - clock =READ_ONCE (rq->scx.clock); - }else { -/* -* Otherwise, return a fresh rq clock. -* -* The rq clock is updated outside of the rq lock. -* In this case, keep the updated rq clock invalid so the next -* kfunc call outside the rq lock gets a fresh rq clock. -*/ - clock =sched_clock_cpu (cpu_of (rq)); - } - -preempt_enable (); - -return clock; +/* +* Note that scx_bpf_now() is re-entrant between a process context and +* an interrupt context (e.g., timer interrupt). However, we don't need +* to consider the race between them because such race is not observable +* from a caller. +*/ +guard (preempt)(); +return __scx_bpf_now (this_rq ()); }static void scx_read_events (struct scx_sched *sch,struct scx_event_stats *events)
kernel/sched/ext/internal.h +1 -0
@@ -1984,6 +1984,7 @@ void scx_flush_dispatch_buf(struct scx_sched *sch, struct rq *rq);s32 scx_init_dsq (struct scx_dispatch_q *dsq,u64 dsq_id,struct scx_sched *sch); __printf(2, 3) void scx_dump_line(struct seq_buf *s, const char *fmt, ...);void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags); +u64 __scx_bpf_now (struct rq *rq);void schedule_dsq_reenq (struct scx_sched *sch,struct scx_dispatch_q *dsq,u64 reenq_flags,struct rq *locked_rq); int __scx_init_task(struct scx_sched *sch, struct task_struct *p,
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 4/12 sched_ext: Reject internal enq_flags in the dsq move kfuncs
In reply to Tejun HeoThe dsq insert kfuncs reject __SCX_ENQ_INTERNAL_MASK bits in scx_dsq_insert_preamble() instead of scx_vet_enq_flags(). A scheduler can smuggle internal flags such as SCX_ENQ_CLEAR_OPSS through the dsq move kfuncs and corrupt the dispatch protocol. Move the rejection into scx_vet_enq_flags(). The vtime move wrapper OR'd the internal SCX_ENQ_DSQ_PRIQ bit into enq_flags before the vet; the bit now goes in inside scx_dsq_move() after the vet.
--- kernel/sched/ext/ext.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
kernel/sched/ext/ext.c +13 -8
@@ -8393,6 +8393,11 @@ static bool scx_vet_enq_flags(struct scx_sched *sch, u64 dsq_id, u64 *enq_flags)bool is_local = dsq_id ==SCX_DSQ_LOCAL || (dsq_id &SCX_DSQ_LOCAL_ON ) ==SCX_DSQ_LOCAL_ON ; +if (unlikely (*enq_flags & __SCX_ENQ_INTERNAL_MASK)) { +scx_error (sch,"invalid enq_flags 0x%llx" , *enq_flags); +return false; + } +if (*enq_flags &SCX_ENQ_IMMED ) {if (unlikely (!is_local)) {scx_error (sch,"SCX_ENQ_IMMED on a non-local DSQ 0x%llx" , dsq_id);
@@ -8415,11 +8420,6 @@ static bool scx_dsq_insert_preamble(struct scx_sched *sch, struct task_struct *preturn false; } -if (unlikely (*enq_flags & __SCX_ENQ_INTERNAL_MASK)) { -scx_error (sch,"invalid enq_flags 0x%llx" , *enq_flags); -return false; - } -/* see SCX_EV_INSERT_NOT_OWNED definition */ if (unlikely (!scx_task_on_sched (sch, p))) {__scx_add_event (sch,SCX_EV_INSERT_NOT_OWNED ,1 );
@@ -8651,7 +8651,8 @@ static const struct btf_kfunc_id_set scx_kfunc_set_enqueue_dispatch = { };static bool scx_dsq_move (struct bpf_iter_scx_dsq_kern *kit, -struct task_struct *p,u64 dsq_id,u64 enq_flags) +struct task_struct *p,u64 dsq_id,u64 enq_flags, +bool priq) {struct scx_dispatch_q *src_dsq = kit->dsq, *dst_dsq;struct scx_sched *sch;
@@ -8673,6 +8674,10 @@ static bool scx_dsq_move(struct bpf_iter_scx_dsq_kern *kit,if (!scx_vet_enq_flags (sch, dsq_id, &enq_flags))return false; +/* internal bit, can only go in after @enq_flags is vetted */ +if (priq) + enq_flags |=SCX_ENQ_DSQ_PRIQ ; +/* * If the BPF scheduler keeps calling this function repeatedly, it can * cause similar live-lock conditions as scx_consume_dispatch_q().
@@ -8929,7 +8934,7 @@ __bpf_kfunc bool scx_bpf_dsq_move(struct bpf_iter_scx_dsq *it__iter, u64 enq_flags) {return scx_dsq_move ((struct bpf_iter_scx_dsq_kern *)it__iter, - p, dsq_id, enq_flags); + p, dsq_id, enq_flags, false); }/**
@@ -8954,7 +8959,7 @@ __bpf_kfunc bool scx_bpf_dsq_move_vtime(struct bpf_iter_scx_dsq *it__iter, u64 enq_flags) {return scx_dsq_move ((struct bpf_iter_scx_dsq_kern *)it__iter, - p, dsq_id, enq_flags |SCX_ENQ_DSQ_PRIQ ); + p, dsq_id, enq_flags, true); }__bpf_kfunc_end_defs ();
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 5/12 sched_ext: Make SCX_ENQ_IGNORE_CAPS waive the preemption cap too
In reply to Tejun HeoSCX_ENQ_IGNORE_CAPS is kernel-internal and marks a placement the kernel forces. scx_caps_for_enq() waives the enqueue cap for it, but a PREEMPT insert still picks up the preemption cap requirement from scx_caps_for_preempt(). Update scx_caps_for_preempt() to take enq_flags and require nothing when SCX_ENQ_IGNORE_CAPS is set.
--- kernel/sched/ext/ext.c | 5 +++-- kernel/sched/ext/sub.c | 2 +- kernel/sched/ext/sub.h | 9 ++++++--- 3 files changed, 10 insertions(+), 6 deletions(-)
kernel/sched/ext/ext.c +3 -2
@@ -8149,8 +8149,9 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_rif (kickable && !scx_missing_caps (pcpu->sch, cpu,SCX_CAP_BASE )) {if (cpumask_test_cpu (cpu, pcpu->cpus_to_preempt)) {if (cur_class == &ext_sched_class) { -if (likely (!scx_missing_caps (pcpu->sch, cpu, -scx_caps_for_preempt (pcpu->sch, rq)))) +u64 caps =scx_caps_for_preempt (pcpu->sch, rq,0 ); + +if (likely (!scx_missing_caps (pcpu->sch, cpu, caps)))scx_set_task_slice (rq->curr,0 );else __scx_add_event(pcpu->sch,
kernel/sched/ext/sub.c +1 -1
@@ -262,7 +262,7 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r* @p's owner (@sch). Check caps against the scheduling sched. */ if (*enq_flags &SCX_ENQ_PREEMPT ) - needed |=scx_caps_for_preempt (asch, rq); + needed |=scx_caps_for_preempt (asch, rq, *enq_flags); missing =scx_missing_caps (asch,cpu_of (rq), needed);/* requirements met */
kernel/sched/ext/sub.h +6 -3
@@ -140,7 +140,7 @@ static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed)static inline u64 scx_caps_for_enq (u64 enq_flags) {/* a restored task must be put into the local DSQ regardless of caps */ -if (enq_flags &SCX_ENQ_IGNORE_CAPS ) +if (unlikely (enq_flags &SCX_ENQ_IGNORE_CAPS ))return 0 ;if (enq_flags &SCX_ENQ_IMMED )return SCX_CAP_ENQ_IMMED ;
@@ -156,10 +156,13 @@ static inline u64 scx_caps_for_task(struct task_struct *p) }/* the cap @sch needs to preempt @rq's current task, 0 if none */ -static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq) +static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq,u64 enq_flags) {struct task_struct *curr = rq->curr; +/* a kernel-forced placement preempts regardless of caps */ +if (unlikely (enq_flags &SCX_ENQ_IGNORE_CAPS )) +return 0 ;/* a non-ext task can't be preempted by ext, own-subtree needs no cap */ if (curr->sched_class != &ext_sched_class ||scx_is_descendant (scx_task_sched (curr), sch))
@@ -195,7 +198,7 @@ static inline bool scx_task_can_stay_on_cpu(struct rq *rq, struct task_struct *p#else /* CONFIG_EXT_SUB_SCHED */ static inline u64 scx_missing_caps (struct scx_sched *sch,s32 cpu,u64 needed) {return 0 ; } -static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq) {return 0 ; } +static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq,u64 enq_flags) {return 0 ; }static inline bool scx_task_can_stay_on_cpu (struct rq *rq,struct task_struct *p) {return true; }#endif /* CONFIG_EXT_SUB_SCHED */
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 6/12 sched_ext: Synchronize slice and dsq_vtime writes
In reply to Tejun Heop->scx.slice and p->scx.dsq_vtime writes have no synchronization rules. The dsq insert kfuncs write both fields synchronously from whatever context they're called in - a direct dispatch from ops.select_cpu() writes with only pi_lock held - and, as the kfuncs are safe to call spuriously with the invalid dispatch discarded later, a scheduler can modify any task's slice by spuriously calling them. The latter stands in the way of an upcoming patch which adds kernel-granted slices that the schedulers must not be able to modify.
Give both fields explicit rules. While the task is running, sleeping or queued on an rq-owned DSQ, the rq lock protects them - these are the states where the kernel consumes the slice. While queued on a user DSQ or on the BPF side, the kernel neither consumes nor decides on the fields and every writer acts for the BPF scheduler - synchronizing the writers is the scheduler's responsibility and whichever write lands last wins.
To conform, an insert kfunc no longer writes the fields when called. The values travel with the dispatch and take effect when the task is inserted. A discarded dispatch has no side effects. The rq lock rule is asserted at the slice store.
--- include/linux/sched/ext.h | 2 + kernel/sched/ext/ext.c | 184 ++++++++++++++++++++++-------------- kernel/sched/ext/internal.h | 4 + 3 files changed, 120 insertions(+), 70 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -192,6 +192,8 @@ struct sched_ext_entity {atomic_long_t ops_state;u64 ddsp_dsq_id;u64 ddsp_enq_flags; +u64 ddsp_slice; +u64 ddsp_vtime;struct scx_dsq_list_node dsq_list;/* dispatch order */ struct rb_node dsq_priq;/* p->scx.dsq_vtime order */ u32 dsq_seq;
kernel/sched/ext/ext.c +114 -70
395 lines not on this page · raw message
kernel/sched/ext/internal.h +4 -0
@@ -1246,6 +1246,8 @@ struct scx_dsp_buf_ent {struct task_struct *task;unsigned long qseq;u64 dsq_id; +u64 slice; +u64 vtime;u64 enq_flags; };
@@ -1680,6 +1682,8 @@ enum scx_enq_flags {SCX_ENQ_NESTED =1LLU <<58 ,SCX_ENQ_GDSQ_FALLBACK =1LLU <<59 ,/* fell back to global DSQ */ SCX_ENQ_IGNORE_CAPS =1LLU <<60 ,/* admit to local DSQ ignoring caps */ +SCX_ENQ_APPLY_SLICE =1LLU <<61 ,/* apply carried slice/vtime at insertion */ + SCX_ENQ_SLICE_DFL = 1LLU << 62, /* carried slice is a default refill */ }; enum scx_deq_flags {
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 7/12 sched_ext: Add SCX_TASK_PROTECTED
In reply to Tejun HeoA BPF scheduler can displace any of its tasks at will - cut a running one's slice with an SCX_ENQ_PREEMPT dispatch, an SCX_KICK_PREEMPT kick or a direct shortening, and jump a queued one with HEAD insertions. Sometimes the kernel needs a slice and a DSQ position to stick regardless.
Add SCX_TASK_PROTECTED, guarding both:
- The slice becomes immutable. Every scheduler-reachable write is refused and counted as SCX_EV_SLICE_DENIED. Higher scheduling classes are unaffected. PREEMPT|IMMED can't preempt a running protected task and gets reenqueued.
- A protected task that reached the head of its DSQ keeps it - HEAD insertions land behind the leading run of protected tasks and reenqueue sweeps skip them. Only rq-owned DSQs can hold protected tasks, so the walk runs only for them.
The bit lives in p->scx.flags so that both the refusal and the head walk read it under the rq lock that protects it.
Protection ends when the slice is consumed, when the task leaves the rq except for a save/restore on the running task, on a yield, when the scheduler enters bypass, and when the task leaves scx. The flag is kernel-internal and not used yet.
--- include/linux/sched/ext.h | 1 + kernel/sched/ext/ext.c | 165 +++++++++++++++++++++++++++++++----- kernel/sched/ext/internal.h | 2 +- 3 files changed, 144 insertions(+), 24 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -102,6 +102,7 @@ enum scx_ent_flags {SCX_TASK_DEQD_FOR_SLEEP =1 <<3 ,/* last dequeue was for SLEEP */ SCX_TASK_SUB_INIT =1 <<4 ,/* task being initialized for a sub sched */ SCX_TASK_IMMED =1 <<5 ,/* task is on local DSQ with %SCX_ENQ_IMMED */ +SCX_TASK_PROTECTED =1 <<6 ,/* slice and DSQ head position protected */ /* * Bits 8 to 10 are used to carry task state:
kernel/sched/ext/ext.c +142 -23
284 lines not on this page · raw message
kernel/sched/ext/internal.h +1 -1
@@ -1972,7 +1972,7 @@ void scx_task_iter_start(struct scx_task_iter *iter, struct cgroup *cgrp);void scx_task_iter_unlock (struct scx_task_iter *iter);void scx_task_iter_stop (struct scx_task_iter *iter);struct task_struct *scx_task_iter_next_locked (struct scx_task_iter *iter); -void scx_set_task_slice (struct task_struct *p,u64 slice); +bool scx_set_task_slice (struct task_struct *p,u64 slice);void scx_task_unlink_from_dsq (struct task_struct *p,struct scx_dispatch_q *dsq);void scx_dispatch_dequeue (struct rq *rq,struct task_struct *p); void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 8/12 sched_ext: Add bandwidth-limited rescue execution for stranded tasks
In reply to Tejun HeoA local DSQ insert lacking the needed caps is diverted to the reject DSQ and bounced back through ops.enqueue() so the scheduler can re-decide. That recovery assumes the scheduler has somewhere legal to send the task. When it doesn't, e.g. when the task's affinity is restricted to cids delegated away, the task starves until the stall watchdog ejects the scheduler. An exiting task is worse - it skips ops.enqueue() and the rejection becomes a self-requeuing cycle that burns the CPU until the watchdog fires.
Add SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the insert would be rejected for missing caps, the kernel takes over and runs the task on the target CPU without consulting the owning scheduler. The kernel sets the flag itself when enqueueing an exiting task.
Rescue is a last-resort forward-progress backstop with a persistent disadvantage, not a way around cap enforcement. A per-CPU token bucket accrues rescue_bandwidth_ppt (default 2%) of CPU time and rescues run one at a time in arrival order. Each is granted a slice of the rescue_quantum_us (default 5ms) quantum divided across the waiters, waits at the tail of the local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival once the slice is served.
The schedulers keep their normal control over an admitted rescuee and may preempt or reslice it. Service is measured on CPU time actually received, so neither shortens the rescue. Prolonged denial escalates - the remaining slice turns into protected execution (SCX_TASK_PROTECTED) and the rescuee preempts the current task. Escalation is paced by the same bucket, and delivered service converges on the configured bandwidth no matter how aggressively the schedulers dispatch.
Both knobs are root-only and SCX_RESCUE_DISABLE turns rescue off, making SCX_ENQ_RESCUE inserts reject as usual.
--- include/linux/sched/ext.h | 1 + kernel/sched/ext/ext.c | 112 +++++++++-- kernel/sched/ext/internal.h | 56 +++++- kernel/sched/ext/sub.c | 376 ++++++++++++++++++++++++++++++++++-- kernel/sched/ext/sub.h | 26 +++ kernel/sched/ext/types.h | 9 + kernel/sched/sched.h | 12 ++ 7 files changed, 559 insertions(+), 33 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -59,6 +59,7 @@ enum scx_dsq_id_flags {SCX_DSQ_LOCAL =SCX_DSQ_FLAG_BUILTIN |2 ,SCX_DSQ_BYPASS =SCX_DSQ_FLAG_BUILTIN |3 ,SCX_DSQ_REJECT =SCX_DSQ_FLAG_BUILTIN |4 ,/* internal - see find_dsq_for_dispatch() */ +SCX_DSQ_RESCUE =SCX_DSQ_FLAG_BUILTIN |5 ,/* internal - see find_dsq_for_dispatch() */ SCX_DSQ_LOCAL_ON =SCX_DSQ_FLAG_BUILTIN |SCX_DSQ_FLAG_LOCAL_ON , SCX_DSQ_LOCAL_CPU_MASK = 0xffffffffLLU, };
kernel/sched/ext/ext.c +95 -17
250 lines not on this page · raw message
kernel/sched/ext/internal.h +55 -1
104 lines not on this page · raw message
kernel/sched/ext/sub.c +361 -15
411 lines not on this page · raw message
kernel/sched/ext/sub.h +26 -0
@@ -38,6 +38,13 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r struct task_struct *p, u64 *enq_flags);bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p);void scx_reenq_reject (struct rq *rq); +void scx_rescue_charge (struct rq *rq,s64 delta_exec); +void scx_rescue_end (struct rq *rq); +bool scx_rescue_keep (struct rq *rq,struct task_struct *p); +void scx_rescue_flush (struct rq *rq); +void scx_rescue_dump (struct seq_buf *s,struct rq *rq); +void scx_rescue_set_knobs (struct scx_sched *sch); +void scx_rescue_init (struct rq *rq);/* * cgrp->scx_sched is written by root/sub enable/disable under all of
@@ -87,6 +94,13 @@ static inline void scx_discard_stale_ecaps_syncs(void) {}static inline struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) {return &rq->scx.local_dsq; }static inline bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p) {return false; }static inline void scx_reenq_reject (struct rq *rq) {} +static inline void scx_rescue_charge (struct rq *rq,s64 delta_exec) {} +static inline void scx_rescue_end (struct rq *rq) {} +static inline bool scx_rescue_keep (struct rq *rq,struct task_struct *p) {return false; } +static inline void scx_rescue_flush (struct rq *rq) {} +static inline void scx_rescue_dump (struct seq_buf *s,struct rq *rq) {} +static inline void scx_rescue_set_knobs (struct scx_sched *sch) {} +static inline void scx_rescue_init (struct rq *rq) {}static inline void scx_dec_has_subs (struct scx_sched *sch) {}#endif /* CONFIG_EXT_SUB_SCHED */
@@ -195,11 +209,23 @@ static inline bool scx_task_can_stay_on_cpu(struct rq *rq, struct task_struct *p return likely(!scx_missing_caps(scx_task_sched(p), cpu_of(rq), SCX_CAP_BASE)); } +/* the task admitted for rescue on @rq, NULL if none */ +static inline struct task_struct *scx_rescuee (struct rq *rq) +{ +lockdep_assert_rq_held (rq); + +if (!scx_has_subs ()) +return NULL ; + +return rq->scx.rescue.curr; +} +#else /* CONFIG_EXT_SUB_SCHED */ static inline u64 scx_missing_caps (struct scx_sched *sch,s32 cpu,u64 needed) {return 0 ; }static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq,u64 enq_flags) {return 0 ; }static inline bool scx_task_can_stay_on_cpu (struct rq *rq,struct task_struct *p) {return true; } +static inline struct task_struct *scx_rescuee (struct rq *rq) {return NULL ; }#endif /* CONFIG_EXT_SUB_SCHED */
kernel/sched/ext/types.h +9 -0
@@ -19,6 +19,15 @@ enum scx_consts {SCX_DSP_MAX_LOOPS =32 ,SCX_WATCHDOG_MAX_TIMEOUT =30 *HZ , +/* rescue knob defaults and limits, see scx_rescue_timerfn() */ +SCX_RESCUE_DFL_BW_PPT =20 ,/* parts per thousand, 2% */ +SCX_RESCUE_MAX_BW_PPT =250 ,/* 25% */ +SCX_RESCUE_DISABLE =U32_MAX ,/* disables rescue */ +SCX_RESCUE_DFL_QUANTUM_US =5000 , +SCX_RESCUE_MIN_QUANTUM_US =1000 , +SCX_RESCUE_MAX_QUANTUM_US =100000 , +SCX_RESCUE_MIN_SLICE_US =1000 ,/* floor of the divided slice */ +/* per-CPU chunk size for p->scx.tid allocation, see scx_alloc_tid() */ SCX_TID_CHUNK = 1024,
kernel/sched/sched.h +12 -0
@@ -794,10 +794,22 @@ enum scx_rq_flags { SCX_RQ_IN_BALANCE = 1 << 17, }; +/* per-rq rescue execution state, see scx_rescue_timerfn() */ +struct scx_rq_rescue { +struct scx_dispatch_q dsq;/* stranded tasks awaiting rescue */ +s64 budget;/* execution token bucket, ns */ +u64 clock;/* last budget accrual timestamp */ +struct task_struct *curr;/* task being rescued, one at a time */ +s64 slice;/* curr's admitted slice */ +u64 exec_snap;/* sum_exec_runtime at admission */ +struct timer_list timer;/* paces admission and escalation */ +}; +struct scx_rq {struct scx_dispatch_q local_dsq;#ifdef CONFIG_EXT_SUB_SCHED struct scx_dispatch_q reject_dsq;/* staging for cap-rejected tasks */ +struct scx_rq_rescue rescue;#endif struct list_head runnable_list;/* runnable tasks on this rq */ struct list_head ddsp_deferred_locals;/* deferred ddsps from enq */
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 9/12 sched_ext: Eject the top rescue consumer on overload
In reply to Tejun HeoWhen rescue demand on a cpu persistently exceeds the configured bandwidth, tasks age on that cpu's rescue DSQ until the stall watchdog fires. The watchdog blames the waiting task's owner, but the misbehaving party is whoever floods the queue, not whoever happens to time out.
Track each sched's recent rescue consumption per cpu as a decaying average. Once the oldest waiter on a cpu's rescue DSQ has been queued past a threshold derived from the rescue knobs (4s at the defaults), the rescue timer ejects the sub with the highest recent consumption on that cpu with SCX_EXIT_ERROR_RESCUE. With no recent consumer there is no victim and nothing is ejected - the generic stall watchdog eventually blames the waiter's owner instead. Ejections on a cpu are spaced one threshold apart so the freed bandwidth can drain the backlog before another sub is judged.
The overload check only wins the race against the stall watchdog when the watchdog timeout clears the threshold, and a single in-budget wait must not cross the trigger on its own. Warn on a scheduler whose timeout doesn't fit and on knobs whose funding period exceeds half the threshold.
--- include/linux/sched/ext.h | 3 + kernel/sched/ext/ext.c | 2 + kernel/sched/ext/internal.h | 9 +++ kernel/sched/ext/sub.c | 136 +++++++++++++++++++++++++++++++++--- kernel/sched/ext/types.h | 3 + kernel/sched/sched.h | 1 + 6 files changed, 143 insertions(+), 11 deletions(-)
include/linux/sched/ext.h +3 -0
@@ -215,6 +215,9 @@ struct sched_ext_entity {#ifdef CONFIG_SCHED_CORE u64 core_sched_at;/* see scx_prio_less() */ #endif +#ifdef CONFIG_EXT_SUB_SCHED +unsigned long rescue_at;/* queued on a rescue DSQ at, jiffies */ +#endif /* * Unique non-zero task ID assigned at fork. Persists across exec and
kernel/sched/ext/ext.c +2 -0
@@ -6140,6 +6140,8 @@ static const char *scx_exit_reason(enum scx_exit_kind kind)return "runnable task stall" ;case SCX_EXIT_ERROR_REENQ :return "reenqueue limit" ; +case SCX_EXIT_ERROR_RESCUE : +return "rescue bandwidth overload" ;default :return "<UNKNOWN>" ; }
kernel/sched/ext/internal.h +9 -0
@@ -57,6 +57,7 @@ enum scx_exit_kind {SCX_EXIT_ERROR_BPF ,/* ERROR but triggered through scx_bpf_error() */ SCX_EXIT_ERROR_STALL ,/* watchdog detected stalled runnable tasks */ SCX_EXIT_ERROR_REENQ ,/* task hit reenqueue limit without running */ + SCX_EXIT_ERROR_RESCUE, /* ejected for overloading rescue execution */ };/*
@@ -1340,6 +1341,14 @@ struct scx_sched_pcpu {bool idle_renotify;/* effective caps as of the last sub_ecaps_updated() delivery */ u64 reported_ecaps; + +/* +* Decaying rescue runtime consumed on this cpu, see +* scx_rescue_decay_avg(). Overload on this cpu ejects the sub with the +* largest value. Accessed only under this cpu's rq lock. +*/ +u64 rescue_avg; +unsigned long rescue_avg_at;/* last decay, jiffies */ #endif /*
kernel/sched/ext/sub.c +125 -11
214 lines not on this page · raw message
kernel/sched/ext/types.h +3 -0
@@ -27,6 +27,9 @@ enum scx_consts {SCX_RESCUE_MIN_QUANTUM_US =1000 ,SCX_RESCUE_MAX_QUANTUM_US =100000 ,SCX_RESCUE_MIN_SLICE_US =1000 ,/* floor of the divided slice */ +SCX_RESCUE_OVERLOAD_MULT =16 ,/* overload threshold in funding periods */ +SCX_RESCUE_MIN_OVERLOAD_MS =1000 , +SCX_RESCUE_MAX_OVERLOAD_MS =15000 ,/* per-CPU chunk size for p->scx.tid allocation, see scx_alloc_tid() */ SCX_TID_CHUNK = 1024,
kernel/sched/sched.h +1 -0
@@ -803,6 +803,7 @@ struct scx_rq_rescue {s64 slice;/* curr's admitted slice */ u64 exec_snap;/* sum_exec_runtime at admission */ struct timer_list timer;/* paces admission and escalation */ +unsigned long kill_at;/* last ejection, init before any */ };struct scx_rq {
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 09/12] sched_ext: Eject the top rescue consumer on overload
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The `time_before()` check in `scx_rescue_check_overload()` suffers from a 32-bit jiffies…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The time_before() check in scx_rescue_check_overload() suffers from a 32-bit jiffies wraparound vulnerability, disabling rescue overload protection for 24.8-day intervals on systems with 32-bit unsigned long. --
--- Patch [9]: [PATCH 09/12] sched_ext: Eject the top rescue consumer on overload --- commit 50e2ffc187a8450dacf680dcf11b98c2167b7621
Author: Tejun Heo <[email protected]>
sched_ext: Eject the top rescue consumer on overload
This patch tracks each scheduler's recent rescue consumption per CPU as a decaying average. If the oldest waiter on a CPU's rescue DSQ waits past a configured threshold, the sub-scheduler with the highest recent consumption is ejected. This prevents the generic stall watchdog from unfairly blaming the waiting task's owner instead of the misbehaving party.
Quoted 4 lines
kernel/sched/ext/sub.c +0
-0
[ ... ]
Quoted 29 lines
kernel/sched/ext/sub.c +25
-0@@ -434,6 +459,63 @@ static bool scx_rescue_try_admit(struct rq *rq, struct task_struct *p)return false; } +/** +* scx_rescue_check_overload - Eject the top rescue consumer on a stuck rescue +* @rq: rq whose rescue timer fired +* +* If the oldest waiter on @rq's rescue DSQ has been queued for too long, rescue +* demand on this cpu persistently exceeds the configured bandwidth. Eject the +* sub with the highest recent rescue consumption instead of letting the +* scheduler stall path blame the waiter's owner, who may just be crowded out. +*/ +static void scx_rescue_check_overload (struct rq *rq) +{ +struct scx_sched *victim =NULL , *pos; +struct task_struct *p; +int cpu =cpu_of (rq); +u64 max_avg =0 ; +u32 dur_ms; + +lockdep_assert_rq_held (rq); + + p = list_first_entry_or_null(&rq->scx.rescue.dsq.list, struct task_struct, + scx.dsq_list.node); +if (!p) +return ; + +/* has the head waiter been queued for longer than the threshold? */
kernel/sched/ext/sub.c static bool scx_rescue_try_admit(struct rq *rq, struct task_struct *p) · line 496
Quoted 10 lines
kernel/sched/ext/sub.c +10
-0+if (time_before (jiffies, p->scx.rescue_at + scx_rescue_overload_after)) +return ; + +/* +* Grace period after the last ejection on this cpu - the freed +* bandwidth gets one threshold's worth of time to drain the backlog +* before another sub is judged. +*/ +if (time_before (jiffies, rq->scx.rescue.kill_at + scx_rescue_overload_after)) +return ;
[Severity: Medium]
Does this time_before() check have an issue with jiffies wraparound on 32-bit systems?
If no rescue ejection occurs for LONG_MAX jiffies (about 24.8 days at HZ=1000 on 32-bit systems), jiffies - rq->scx.rescue.kill_at will exceed LONG_MAX. Because the time_before() macro uses signed arithmetic, the difference wraps around to a negative value.
This causes time_before() to incorrectly evaluate to true, falsely trapping the CPU in a continuous grace period. Would this disable overload ejections for the next 24.8 days and cause the scheduler to fall back to the generic stall watchdog instead?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Tejun Heo
reviewerPATCH 10/12 sched_ext: Sync tools autogen enum headers
In reply to Tejun HeoMechanical regeneration for the enumerators added by the slice write rework and the rescue machinery. BPF schedulers reference SCX_ENQ_RESCUE, the rest are enum_defs presence markers.
--- tools/sched_ext/include/scx/enum_defs.autogen.h | 16 ++++++++++++++++ tools/sched_ext/include/scx/enums.autogen.bpf.h | 3 +++ tools/sched_ext/include/scx/enums.autogen.h | 1 + 3 files changed, 20 insertions(+)
tools/sched_ext/include/scx/enum_defs.autogen.h +16 -0
@@ -26,6 +26,16 @@#define HAVE_SCX_DSP_DFL_MAX_BATCH #define HAVE_SCX_DSP_MAX_LOOPS #define HAVE_SCX_WATCHDOG_MAX_TIMEOUT +#define HAVE_SCX_RESCUE_DFL_BW_PPT +#define HAVE_SCX_RESCUE_MAX_BW_PPT +#define HAVE_SCX_RESCUE_DISABLE +#define HAVE_SCX_RESCUE_DFL_QUANTUM_US +#define HAVE_SCX_RESCUE_MIN_QUANTUM_US +#define HAVE_SCX_RESCUE_MAX_QUANTUM_US +#define HAVE_SCX_RESCUE_MIN_SLICE_US +#define HAVE_SCX_RESCUE_OVERLOAD_MULT +#define HAVE_SCX_RESCUE_MIN_OVERLOAD_MS +#define HAVE_SCX_RESCUE_MAX_OVERLOAD_MS #define HAVE_SCX_TID_CHUNK #define HAVE_SCX_EXIT_BT_LEN #define HAVE_SCX_EXIT_MSG_LEN
@@ -53,6 +63,7 @@#define HAVE_SCX_DSQ_LOCAL #define HAVE_SCX_DSQ_BYPASS #define HAVE_SCX_DSQ_REJECT +#define HAVE_SCX_DSQ_RESCUE #define HAVE_SCX_DSQ_LOCAL_ON #define HAVE_SCX_DSQ_LOCAL_CPU_MASK #define HAVE_SCX_DSQ_ITER_REV
@@ -71,6 +82,7 @@#define HAVE_SCX_ENQ_CPU_SELECTED #define HAVE_SCX_ENQ_PREEMPT #define HAVE_SCX_ENQ_IMMED +#define HAVE_SCX_ENQ_RESCUE #define HAVE_SCX_ENQ_REENQ #define HAVE_SCX_ENQ_LAST #define HAVE___SCX_ENQ_INTERNAL_MASK
@@ -79,6 +91,8 @@#define HAVE_SCX_ENQ_NESTED #define HAVE_SCX_ENQ_GDSQ_FALLBACK #define HAVE_SCX_ENQ_IGNORE_CAPS +#define HAVE_SCX_ENQ_APPLY_SLICE +#define HAVE_SCX_ENQ_SLICE_DFL #define HAVE_SCX_TASK_DSQ_ON_PRIQ #define HAVE_SCX_TASK_QUEUED #define HAVE_SCX_TASK_IN_CUSTODY
@@ -86,6 +100,7 @@#define HAVE_SCX_TASK_DEQD_FOR_SLEEP #define HAVE_SCX_TASK_SUB_INIT #define HAVE_SCX_TASK_IMMED +#define HAVE_SCX_TASK_PROTECTED #define HAVE_SCX_TASK_STATE_SHIFT #define HAVE_SCX_TASK_STATE_BITS #define HAVE_SCX_TASK_STATE_MASK
@@ -120,6 +135,7 @@#define HAVE_SCX_EXIT_ERROR_BPF #define HAVE_SCX_EXIT_ERROR_STALL #define HAVE_SCX_EXIT_ERROR_REENQ +#define HAVE_SCX_EXIT_ERROR_RESCUE #define HAVE_SCX_KF_ALLOW_UNLOCKED #define HAVE_SCX_KF_ALLOW_INIT_CIDS #define HAVE_SCX_KF_ALLOW_CPU_RELEASE
tools/sched_ext/include/scx/enums.autogen.bpf.h +3 -0
@@ -124,6 +124,9 @@ const volatile u64 __SCX_ENQ_PREEMPT __weak;const volatile u64 __SCX_ENQ_IMMED__weak ;#define SCX_ENQ_IMMED __SCX_ENQ_IMMED +const volatile u64 __SCX_ENQ_RESCUE__weak ; +#define SCX_ENQ_RESCUE __SCX_ENQ_RESCUE +const volatile u64 __SCX_ENQ_REENQ__weak ;#define SCX_ENQ_REENQ __SCX_ENQ_REENQ
tools/sched_ext/include/scx/enums.autogen.h +1 -0
@@ -45,6 +45,7 @@SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_HEAD ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_PREEMPT ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_IMMED ); \ +SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_RESCUE ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_REENQ ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_LAST ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_CLEAR_OPSS ); \
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 11/12 sched_ext: scx_qmap - Idle-check pinned tasks before direct dispatch
In reply to Tejun Heopick_direct_dispatch_cid() sent a pinned task straight to its only cpu without an idle check. An insert onto a time-shared cid is forced IMMED, which may not wait on a busy cpu - the kernel bounces the task back to ops.enqueue(), the shortcut re-inserts it, and the loop runs into the reenqueue repeat limit.
Run pinned tasks through the same idle test as everyone else and queue them when the cpu is busy. always_enq_immed forces IMMED on every local insert and already skipped the shortcut for the same reason.
--- tools/sched_ext/scx_qmap.bpf.c | 3 --- 1 file changed, 3 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +0 -3
@@ -230,9 +230,6 @@ static s32 pick_direct_dispatch_cid(struct task_struct *p, s32 prev_cid,s32 cid;u32 i; -if (!always_enq_immed && p->nr_cpus_allowed ==1 ) -return prev_cid; -if (cmask_test (prev_cid, &qa.self_cids.mask) &&cmask_test_and_clear (prev_cid, &qa.idle_cids.mask))return prev_cid;
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 11/12] sched_ext: scx_qmap - Idle-check pinned tasks before direct dispatch
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The BPF rodata variable `always_enq_immed` was made unused by the patch and remains as dead code. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The BPF rodata variable always_enq_immed was made unused by the patch and remains as dead code. --
--- Patch [11]: [PATCH 11/12] sched_ext: scx_qmap - Idle-check pinned tasks before direct dispatch --- commit 8281293388fde26e27e2d80b5bd65cbae1161f07
Author: Tejun Heo <[email protected]>
sched_ext: scx_qmap - Idle-check pinned tasks before direct dispatch
This commit updates pick_direct_dispatch_cid() to perform idle checks on pinned tasks before direct dispatching. Previously, pinned tasks were sent straight to their only CPU without an idle check, which could lead to bounce loops if the CPU was busy.
tools/sched_ext/scx_qmap.bpf.c static s32 pick_direct_dispatch_cid(struct task_struct *p, s32 prev_cid, · line 235
Quoted 11 lines
tools/sched_ext/scx_qmap.bpf.c +0
-3@@ -230,9 +230,6 @@ static s32 pick_direct_dispatch_cid(struct task_struct *p, s32 prev_cid,s32 cid;u32 i; -if (!always_enq_immed && p->nr_cpus_allowed ==1 ) -return prev_cid; -
[Severity: Low]
With the removal of this check, the BPF rodata variable always_enq_immed is no longer read anywhere in the BPF program.
Should the declaration and its userspace assignment also be removed to clean up the remaining dead code?
tools/sched_ext/scx_qmap.bpf.c:
const volatile bool always_enq_immed;
tools/sched_ext/scx_qmap.c:main()
...
case 'I':
skel->rodata->always_enq_immed = true;
...Quoted 3 lines
tools/sched_ext/scx_qmap.bpf.c +0
-0if (cmask_test (prev_cid, &qa.self_cids.mask) &&cmask_test_and_clear (prev_cid, &qa.idle_cids.mask))return prev_cid;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Tejun Heo
reviewerPATCH 12/12 sched_ext: scx_qmap - Add rescue support
In reply to Tejun HeoA sched holds only the cids its parent granted and nothing guarantees that they cover its tasks' affinities. A task that can run on none of them has nowhere to go and qmap stalls out: it force-inserts the task onto its first allowed cid, but the kernel bounces the insert back and the task parks in SHARED_DSQ, which is drained only on self cids it can't run on.
Set SCX_ENQ_RESCUE on these inserts so the kernel diverts such tasks to its rescue path instead of bouncing them. The force-insert covers scheds with and without children and fires on re-enqueues, and the SHARED_DSQ scan on every dispatch rescues tasks stranded there - the enqueue-time check misses a task whose cids were lost while it was already queued. The wrong-cid fault injection carries the flag too and doubles as a deterministic rescue-traffic generator.
-B and -q set the root-only rescue bandwidth and quantum ops knobs. -B 0 maps to SCX_RESCUE_DISABLE and turns rescue off kernel-side. Rescue inserts are counted and reported in the hier stats line.
--- tools/sched_ext/scx_qmap.bpf.c | 85 +++++++++++++++++++++++----------- tools/sched_ext/scx_qmap.c | 23 +++++++-- tools/sched_ext/scx_qmap.h | 1 + 3 files changed, 79 insertions(+), 30 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +58 -27
@@ -449,31 +449,33 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;/* -* A node with children delegates most cids. A task of ours that can run -* on none of our self cids (e.g. a per-NUMA kthread pinned to delegated -* cids) would starve in SHARED/FIFO since we never pull those on a -* delegated cid. Force it onto its first allowed cid's local DSQ with -* needs_immed(): if we hold access there it runs, else the kernel -* rejects and bounces it back via REENQ_CAP. Best-effort -* anti-starvation nudge. +* A task of ours that can run on none of our self cids - the parent +* didn't grant them or we delegated them to children - would starve in +* SHARED/FIFO since we only pull from those on self cids. +* +* Force it onto its first allowed cid's local DSQ. If we hold that cid +* it runs. Otherwise the insert carries SCX_ENQ_RESCUE and the kernel +* diverts the task to its rescue path. */ -if (qa.nr_sub_scheds && !(enq_flags &SCX_ENQ_REENQ ) && - !cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) { +if (!cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) {s32 c =cmask_next_set_wrap (&taskc->cpus_allowed,0 );if (c >=0 && c <scx_bpf_nr_cids ()) { taskc->force_local = false; +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 );scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | c, slice_ns, - enq_flags |needs_immed (c)); + enq_flags |needs_immed (c) |SCX_ENQ_RESCUE );return ; } }/* * Fault injection: deliberately dispatch one of our own tasks to a cid -* we don't hold. The kernel cap check must reject it and re-enqueue -* with SCX_TASK_REENQ_CAP, so nr_inject_attempts tracks nr_reenq_cap -* and proves delivery-time enforcement. Throttled. +* we don't hold. The inserts carry SCX_ENQ_RESCUE and divert to the +* kernel rescue path, a deterministic rescue-traffic generator. Under +* -B 0 the kernel cap check rejects and re-enqueues them instead, so +* nr_inject_attempts tracks nr_reenq_cap 1:1 and proves delivery-time +* enforcement. Throttled. */ if (qa.inject_mode ==QMAP_INJ_WRONG_CID && p->nr_cpus_allowed >1 && !(enq_flags &SCX_ENQ_REENQ )) {
@@ -484,8 +486,9 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)if (bad >=0 &&cmask_test (bad, &taskc->cpus_allowed)) {__sync_fetch_and_add (&qa.nr_inject_attempts,1 ); -scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | bad, - slice_ns, enq_flags); +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 ); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | bad, slice_ns, + enq_flags |SCX_ENQ_RESCUE );return ; } }
@@ -588,29 +591,47 @@ static void update_core_sched_head_seq(struct task_struct *p) }/* +* One pass over SHARED_DSQ: rescue stranded tasks and boost highpri ones. A +* task whose cids were lost while it was queued in the fifos would strand on +* SHARED_DSQ, which is consumed only on self cids it can't run on - move it to +* the kernel rescue path. One whose cids were lost after the highpri cull is +* likewise rescued out of HIGHPRI_DSQ below. +* * To demonstrate the use of scx_bpf_dsq_move(), implement silly selective -* priority boosting mechanism by scanning SHARED_DSQ looking for highpri tasks, -* moving them to HIGHPRI_DSQ and then consuming them first. This makes minor -* difference only when dsp_batch is larger than 1. +* priority boosting mechanism by moving highpri tasks to HIGHPRI_DSQ and then +* consuming them first. This makes minor difference only when dsp_batch is +* larger than 1. * * scx_bpf_dsq_move[_vtime]() are allowed both from ops.dispatch() and * non-rq-lock holding BPF programs. As demonstration, this function is called * from qmap_dispatch() and monitor_timerfn(). */ -static bool dispatch_highpri (bool from_timer) +static bool scan_shared_dsq (bool from_timer) {struct task_struct *p;s32 this_cid =scx_bpf_this_cid ();u32 nr_cids =scx_bpf_nr_cids (); -/* scan SHARED_DSQ and move highpri tasks to HIGHPRI_DSQ */ +/* rescue strands and move highpri tasks to HIGHPRI_DSQ */ bpf_for_each (scx_dsq, p,SHARED_DSQ ,0 ) {static u64 highpri_seq;task_ctx_t *taskc; +s32 c;if (!(taskc =lookup_task_ctx (p)))return false; +/* stranded? rescue - it can't be dispatched here either way */ +if (!cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) { + c =cmask_next_set_wrap (&taskc->cpus_allowed,0 ); +if (c >=0 && c <scx_bpf_nr_cids ()) { +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 ); +scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | c, +needs_immed (c) |SCX_ENQ_RESCUE ); + } +continue ; + } +if (taskc->highpri) {/* exercise the set_*() and vtime interface too */ scx_bpf_dsq_move_set_slice (BPF_FOR_EACH_ITER , slice_ns *2 );
@@ -640,8 +661,17 @@ static bool dispatch_highpri(bool from_timer) cid =cmask_next_and_set_wrap (&taskc->cpus_allowed, &qa.self_cids.mask, this_cid +1 ); -if (cid >= nr_cids) +if (cid >= nr_cids) { +/* stranded after the cull - rescue it from here */ +s32 c =cmask_next_set_wrap (&taskc->cpus_allowed,0 ); + +if (c >=0 && c < nr_cids) { +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 ); +scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | c, +needs_immed (c) |SCX_ENQ_RESCUE ); + }continue ; + }if (scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | cid,SCX_ENQ_PREEMPT |needs_immed (cid))) {
@@ -670,10 +700,9 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev) struct cpu_ctx __arena *cpuc;task_ctx_t *taskc;u32 batch = dsp_batch ?:1 ; -s32 owner; -s32 i; +s32 owner, i; -if (dispatch_highpri (false)) +if (scan_shared_dsq (false))return ;/*
@@ -785,11 +814,10 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev)*/ if (!cmask_test (cid, &taskc->cpus_allowed))scx_bpf_kick_cid (scx_bpf_task_cid (p),0 ); - batch--; cpuc->dsp_cnt--;if (!batch || !scx_bpf_dispatch_nr_slots ()) { -if (dispatch_highpri (false)) +if (scan_shared_dsq (false))return ;scx_bpf_dsq_move_to_local (SHARED_DSQ ,needs_immed (cid));return ;
@@ -801,6 +829,9 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev) cpuc->dsp_cnt =0 ; } +if (scan_shared_dsq (false)) +return ; +/* * No other tasks. @prev will keep running. Update its core_sched_seq as * if the task were enqueued and dispatched immediately.
@@ -1185,7 +1216,7 @@ static void dump_shared_dsq(void)static int monitor_timerfn (void *map,int *key,struct bpf_timer *timer) {bpf_rcu_read_lock (); -dispatch_highpri (true); +scan_shared_dsq (true);bpf_rcu_read_unlock ();monitor_cpuperf ();
tools/sched_ext/scx_qmap.c +20 -3
@@ -72,6 +72,8 @@ const char help_fmt[] =" -J MODE Fault injection (wrong-cid: dispatch to a cid not held,\n" " init-fail/cgrp-init-fail: fail init_task/cpuctl_init for\n" " \"qmfail*\" comms/cgroups)\n" +" -B PPT Rescue bandwidth in parts per thousand, 0 disables (root only, default 20)\n" +" -q US Rescue batch quantum in microseconds (root only, default 5000)\n" " -v Print libbpf debug messages\n" " -h Display this help and exit\n" ;
@@ -106,6 +108,7 @@ struct hier_prev {u64 nr_reenq_cap;u64 nr_reenq_immed;u64 nr_inject_attempts; +u64 nr_rescue_dsp; };/* current wall-clock time as "HH:MM:SS" for the startup and interval headers */
@@ -187,14 +190,16 @@ static void print_hier(struct qmap_arena *qa, struct hier_prev *prev, u64 own_cg }format_cid_ranges (qa,CID_SHARED , ranges,sizeof (ranges)); -printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu\n" , +printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu rescue=+%llu\n" , (unsigned long long )qa->nr_sub_scheds, qa->part.nr_excl, ranges, rr, (unsigned long long )(qa->nr_reenq_cap - prev->nr_reenq_cap), (unsigned long long )(qa->nr_reenq_immed - prev->nr_reenq_immed), - (unsigned long long )(qa->nr_inject_attempts - prev->nr_inject_attempts)); + (unsigned long long )(qa->nr_inject_attempts - prev->nr_inject_attempts), + (unsigned long long )(qa->nr_rescue_dsp - prev->nr_rescue_dsp)); prev->nr_reenq_cap = qa->nr_reenq_cap; prev->nr_reenq_immed = qa->nr_reenq_immed; prev->nr_inject_attempts = qa->nr_inject_attempts; + prev->nr_rescue_dsp = qa->nr_rescue_dsp;printf ("hier : %-4s %10s %4s %6s %8s %s\n" ,"" ,"cgroup" ,"w" ,"alloc" ,"disp/s" ,"cids" );
@@ -256,7 +261,8 @@ int main(int argc, char **argv) skel->rodata->slice_ns =__COMPAT_ENUM_OR_ZERO ("scx_public_consts" ,"SCX_SLICE_DFL" ); skel->rodata->max_tasks =16384 ; -while ((opt =getopt (argc, argv,"s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:vh" )) !=-1 ) { +while ((opt =getopt (argc, argv, +"s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:B:q:vh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -400,6 +406,17 @@ int main(int argc, char **argv)else inject_mode =strtoul (optarg,NULL ,0 );break ; +case 'B' : { +u32 ppt =strtoul (optarg,NULL ,0 ); + +if (!ppt) + ppt =__COMPAT_ENUM_OR_ZERO ("scx_consts" ,"SCX_RESCUE_DISABLE" ); + skel->struct_ops.qmap_ops->rescue_bandwidth_ppt = ppt; +break ; + } +case 'q' : + skel->struct_ops.qmap_ops->rescue_quantum_us =strtoul (optarg,NULL ,0 ); +break ;case 'v' : verbose = true;break ;
tools/sched_ext/scx_qmap.h +1 -0
@@ -175,6 +175,7 @@ struct qmap_arena {u64 nr_reenq_cap;/* SCX_TASK_REENQ_CAP bounces */ u64 nr_reenq_immed;/* SCX_TASK_REENQ_IMMED bounces */ u64 nr_inject_attempts;/* fault-injection: dispatches to an unheld cid */ +u64 nr_rescue_dsp;/* SCX_ENQ_RESCUE dispatch attempts */ u32 inject_mode;/* fault-injection mode (QMAP_INJ_*) */ };
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 12/12] sched_ext: scx_qmap - Add rescue support
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Removing the `!(enq_flags & SCX_ENQ_REENQ)` check for stranded tasks causes an infinite…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Removing the !(enq_flags & SCX_ENQ_REENQ) check for stranded tasks causes an infinite enqueue-reject-reenqueue loop when rescue is disabled. --
--- Patch [12]: [PATCH 12/12] sched_ext: scx_qmap - Add rescue support --- commit d8697209a4441d32aa069b5cede51b062fddcc4c
Author: Tejun Heo <[email protected]>
sched_ext: scx_qmap - Add rescue support
This commit adds rescue support to scx_qmap for tasks that have no self cids they can run on. It updates the qmap_enqueue and qmap_dispatch logic to use the SCX_ENQ_RESCUE flag and scan the shared queue to redirect stranded tasks.
Quoted 16 lines
tools/sched_ext/scx_qmap.bpf.c +1
-7@@ -449,31 +449,33 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;/* -* A node with children delegates most cids. A task of ours that can run -* on none of our self cids (e.g. a per-NUMA kthread pinned to delegated -* cids) would starve in SHARED/FIFO since we never pull those on a -* delegated cid. Force it onto its first allowed cid's local DSQ with -* needs_immed(): if we hold access there it runs, else the kernel -* rejects and bounces it back via REENQ_CAP. Best-effort -* anti-starvation nudge. +* A task of ours that can run on none of our self cids - the parent
tools/sched_ext/scx_qmap.bpf.c void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) · line 460
Quoted 10 lines
tools/sched_ext/scx_qmap.bpf.c +7
-2+* didn't grant them or we delegated them to children - would starve in +* SHARED/FIFO since we only pull from those on self cids. +* +* Force it onto its first allowed cid's local DSQ. If we hold that cid +* it runs. Otherwise the insert carries SCX_ENQ_RESCUE and the kernel +* diverts the task to its rescue path. */ -if (qa.nr_sub_scheds && !(enq_flags &SCX_ENQ_REENQ ) && - !cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) { +if (!cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) {
[Severity: High]
Does removing the !(enq_flags & SCX_ENQ_REENQ) check here cause an infinite bounce loop when rescue is disabled kernel-side?
If scx_qmap is executed with -B 0, the kernel capability check rejects the insert and bounces the task back to ops.enqueue() with the SCX_ENQ_REENQ flag.
Without the check for SCX_ENQ_REENQ, qmap_enqueue() will immediately hit this condition again and force-insert the task back to the target CPU.
Could this lead to an endless, unthrottled loop that locks up the core?
Quoted 11 lines
tools/sched_ext/scx_qmap.bpf.c +2
-1s32 c =cmask_next_set_wrap (&taskc->cpus_allowed,0 );if (c >=0 && c <scx_bpf_nr_cids ()) { taskc->force_local = false; +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 );scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | c, slice_ns, - enq_flags |needs_immed (c)); + enq_flags |needs_immed (c) |SCX_ENQ_RESCUE );return ; } }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Tejun Heo
reviewerRe: [PATCH 12/12] sched_ext: scx_qmap - Add rescue support
Quoted 4 lines · On Sat, Aug 01, 2026 at 09:06:02AM +0000, [email protected] wrote:
[Severity: High]
Does removing the !(enq_flags &SCX_ENQ_REENQ) check here cause an infinite bounce loop when rescue is disabled kernel-side?
Quoted 8 lines
If
scx_qmapis executed with -B 0, the kernel capability check rejects the insert and bounces the task back toops.enqueue()with theSCX_ENQ_REENQflag.Without the check for
SCX_ENQ_REENQ,qmap_enqueue()will immediately hit this condition again and force-insert the task back to the target CPU.Could this lead to an endless, unthrottled loop that locks up the core?
The loop is bounded. Each bounce goes back through scx_do_enqueue_task() which counts it in p->scx.reenq_cnt, and crossing SCX_REENQ_MAX_REPEAT ejects the owning scheduler. With -B 0, a stranded task takes down qmap instead of starving or burning the CPU, which is the intended failure mode when rescue is turned off.
Thanks.
--
tejun