sched_ext: Bandwidth-limited rescue execution for stranded tasks
Subsystems (2)
Tejun Heo
reviewerPATCH v2 1/1 sched_ext: Bandwidth-limited rescue execution for stranded tasks
Hello,
v2: Changes from the sashiko AI review:
- Declare scx_sched_all outside the CONFIG_EXT_SUB_SCHED block, the definition is unconditional (0002).
- Track the overload grace and usage-decay timestamps in jiffies_64 so that 32-bit arithmetic on them can't wrap (0009).
- Drop the now-unread always_enq_immed rodata mirror (0011).
v1: https://lore.kernel.org/r/[email protected]
A sub-scheduler holds only the cids its parent granted and nothing guarantees that they cover its tasks' affinities. A task that can run on none of them currently has no good ending: the scheduler's insert is cap-rejected and reenqueued until the repeat limit ejects the scheduler, or the task stalls into the watchdog.
This series adds kernel-side rescue execution. A scheduler marks an insert that may be cap-rejected with SCX_ENQ_RESCUE and the kernel, instead of bouncing it, runs the task on the target cpu at a small configured bandwidth. A rescue starts out non-disruptive. Schedulers that hold caps on the cid largely stay in control of the cpu - a preemption cap, for example, still lets its holder preempt the rescuee. When a rescue goes unserved for too long, it escalates into protected execution.
When a cpu's rescue queue stays oversaturated past the overload threshold, the kernel ejects the sub-scheduler with the highest recent rescue consumption on that cpu instead of misblaming the waiter's owner.
- 0001-0005 are preps: naming, helper visibility, dsq-move flag vetting and the IGNORE_CAPS preemption waiver.
- 0006-0007 synchronize slice/dsq_vtime writes at insertion commit and add SCX_TASK_PROTECTED, the groundwork that lets a kernel-granted slice survive scheduler writes.
- 0008-0009 add the rescue machinery and the overload ejection.
- 0010 syncs the tools autogen enum headers.
- 0011-0012 fix scx_qmap's pinned-task direct dispatch and add rescue support as the demonstration consumer.
0001-sched_ext-Rename-scx_local_or_reject_dsq-to-scx_reso.patch 0002-sched_ext-Make-several-ext.c-helpers-available-outsi.patch 0003-sched_ext-Factor-out-__scx_bpf_now.patch 0004-sched_ext-Reject-internal-enq_flags-in-the-dsq-move-.patch 0005-sched_ext-Make-SCX_ENQ_IGNORE_CAPS-waive-the-preempt.patch 0006-sched_ext-Synchronize-slice-and-dsq_vtime-writes.patch 0007-sched_ext-Add-SCX_TASK_PROTECTED.patch 0008-sched_ext-Add-bandwidth-limited-rescue-execution-for.patch 0009-sched_ext-Eject-the-top-rescue-consumer-on-overload.patch 0010-sched_ext-Sync-tools-autogen-enum-headers.patch 0011-sched_ext-scx_qmap-Idle-check-pinned-tasks-before-di.patch 0012-sched_ext-scx_qmap-Add-rescue-support.patch
Based on sched_ext/for-7.3 (c5b9316cf3d5).
Git tree: git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext.git scx-rescue-v2
include/linux/sched/ext.h | 7 + kernel/sched/ext/ext.c | 690 ++++++++++++++++-------- kernel/sched/ext/internal.h | 78 ++- kernel/sched/ext/sub.c | 497 ++++++++++++++++- kernel/sched/ext/sub.h | 41 +- kernel/sched/ext/types.h | 12 + kernel/sched/sched.h | 13 + tools/sched_ext/include/scx/enum_defs.autogen.h | 16 + tools/sched_ext/include/scx/enums.autogen.bpf.h | 3 + tools/sched_ext/include/scx/enums.autogen.h | 1 + tools/sched_ext/scx_qmap.bpf.c | 89 +-- tools/sched_ext/scx_qmap.c | 24 +- tools/sched_ext/scx_qmap.h | 1 + 13 files changed, 1187 insertions(+), 285 deletions(-)
--
tejun
Tejun Heo
reviewerPATCH 1/12 sched_ext: Rename scx_local_or_reject_dsq() to scx_resolve_local_dsq()
The following rescue execution addition gives the function a third possible destination, making a name that enumerates the outcomes a poor fit. Rename to the destination-neutral scx_resolve_local_dsq(). No functional changes.
--- kernel/sched/ext/ext.c | 4 ++-- kernel/sched/ext/sub.c | 6 +++--- kernel/sched/ext/sub.h | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-)
kernel/sched/ext/ext.c +2 -2
@@ -1507,7 +1507,7 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq,bool is_rq_owned = false;if (dsq->id ==SCX_DSQ_LOCAL ) { - dsq =scx_local_or_reject_dsq (sch, rq, p, &enq_flags); + dsq =scx_resolve_local_dsq (sch, rq, p, &enq_flags); is_rq_owned = true; }
@@ -2279,7 +2279,7 @@ static void move_local_task_to_local_dsq(struct scx_sched *sch,struct scx_dispatch_q *src_dsq, struct rq *dst_rq) { - struct scx_dispatch_q *dst_dsq = scx_local_or_reject_dsq(sch, dst_rq, p, &enq_flags); + struct scx_dispatch_q *dst_dsq = scx_resolve_local_dsq(sch, dst_rq, p, &enq_flags);/* @dsq is locked and @p is on @dst_rq */ lockdep_assert_held (&src_dsq->lock);
kernel/sched/ext/sub.c +3 -3
@@ -230,7 +230,7 @@ void scx_init_root_caps(struct scx_sched *sch) }/** -* scx_local_or_reject_dsq - Pick the local or reject DSQ for an insert +* scx_resolve_local_dsq - Pick the local or reject DSQ for an insert * @sch: enqueuing sub-sched * @rq: rq whose local DSQ @p targets * @p: task being inserted
@@ -246,8 +246,8 @@ void scx_init_root_caps(struct scx_sched *sch)* to and run by its nearest non-bypassing ancestor. If root is bypassing, it * always holds all caps. */ -struct scx_dispatch_q *scx_local_or_reject_dsq (struct scx_sched *sch,struct rq *rq, -struct task_struct *p,u64 *enq_flags) +struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq, +struct task_struct *p,u64 *enq_flags) {if (!scx_has_subs ())return &rq->scx.local_dsq;
kernel/sched/ext/sub.h +3 -3
@@ -34,8 +34,8 @@ void scx_online_ecaps(struct rq *rq);void scx_offline_ecaps (struct rq *rq);void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu);void scx_discard_stale_ecaps_syncs (void ); -struct scx_dispatch_q *scx_local_or_reject_dsq (struct scx_sched *sch,struct rq *rq, -struct task_struct *p,u64 *enq_flags); +struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq, +struct task_struct *p,u64 *enq_flags);bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p);void scx_reenq_reject (struct rq *rq);
@@ -84,7 +84,7 @@ static inline void scx_online_ecaps(struct rq *rq) {}static inline void scx_offline_ecaps (struct rq *rq) {}static inline void scx_discard_ecaps_to_sync (s32 cpu,struct scx_sched_pcpu *pcpu) {}static inline void scx_discard_stale_ecaps_syncs (void ) {} -static inline struct scx_dispatch_q *scx_local_or_reject_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) {return &rq->scx.local_dsq; } +static inline struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) {return &rq->scx.local_dsq; }static inline bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p) {return false; }static inline void scx_reenq_reject (struct rq *rq) {}static inline void scx_dec_has_subs (struct scx_sched *sch) {}
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 2/12 sched_ext: Make several ext.c helpers available outside ext.c
In reply to Tejun Heoset_task_slice(), task_unlink_from_dsq(), move_local_task_to_local_dsq(), init_dsq() and dump_line() will be used outside ext.c. Add the scx_ prefix and declare them in internal.h. The scx_sched_all list will also be used outside ext.c, drop its static. No functional changes.
v2: Declare scx_sched_all outside the CONFIG_EXT_SUB_SCHED block - the
definition is unconditional. (sashiko AI)
--- kernel/sched/ext/ext.c | 172 +++++++++++++++++------------------- kernel/sched/ext/internal.h | 8 ++ 2 files changed, 90 insertions(+), 90 deletions(-)
kernel/sched/ext/ext.c +82 -90
@@ -38,7 +38,7 @@ struct scx_sched __rcu *scx_root;* All scheds, writers must hold both scx_enable_mutex and scx_sched_lock. * Readers can hold either or rcu_read_lock(). */ -static LIST_HEAD (scx_sched_all); +LIST_HEAD (scx_sched_all);#ifdef CONFIG_EXT_SUB_SCHED const struct rhashtable_params scx_sched_hash_params = {
@@ -900,7 +900,7 @@ struct task_struct *scx_task_iter_next_locked(struct scx_task_iter *iter)* @kind: a kind of event to dump */ #define scx_dump_event (s, events, kind) do { \ - dump_line(&(s), "%40s: %16lld", #kind, (events)->kind); \ + scx_dump_line(&(s), "%40s: %16lld", #kind, (events)->kind); \ } while (0)
@@ -1248,7 +1248,7 @@ static void set_task_slice_keep_oob(struct task_struct *p, u64 slice) }/* set @p's slice, superseding any pending out-of-band request */ -static void set_task_slice (struct task_struct *p,u64 slice) +void scx_set_task_slice (struct task_struct *p,u64 slice) {set_task_slice_keep_oob (p, slice);clear_task_slice_oob (p);
@@ -1495,7 +1495,7 @@ static void rq_owned_post_enq(struct scx_sched *sch, struct rq *rq,if ((enq_flags &SCX_ENQ_PREEMPT ) && p != rq->curr && rq->curr->sched_class == &ext_sched_class) { -set_task_slice (rq->curr,0 ); +scx_set_task_slice (rq->curr,0 );resched_curr (rq); } }
@@ -1637,8 +1637,7 @@ static void scx_dispatch_enqueue(struct scx_sched *sch, struct rq *rq,atomic_long_set_release (&p->scx.ops_state,SCX_OPSS_NONE ); } -static void task_unlink_from_dsq (struct task_struct *p, -struct scx_dispatch_q *dsq) +void scx_task_unlink_from_dsq (struct task_struct *p,struct scx_dispatch_q *dsq) {WARN_ON_ONCE (list_empty (&p->scx.dsq_list.node));
@@ -1695,7 +1694,7 @@ void scx_dispatch_dequeue(struct rq *rq, struct task_struct *p)*/ if (p->scx.holding_cpu <0 ) {/* @p must still be on @dsq, dequeue */ -task_unlink_from_dsq (p, dsq); +scx_task_unlink_from_dsq (p, dsq); }else {/* * We're racing against dispatch_to_local_dsq() which already
@@ -1722,7 +1721,7 @@ static void dispatch_dequeue_locked(struct task_struct *p,lockdep_assert_rq_held (task_rq (p));lockdep_assert_held (&dsq->lock); -task_unlink_from_dsq (p, dsq); +scx_task_unlink_from_dsq (p, dsq); p->scx.dsq =NULL ; }
@@ -2236,7 +2235,7 @@ static void yield_task_scx(struct rq *rq)if (SCX_HAS_OP (sch, yield))SCX_CALL_OP_2TASKS_RET (sch, yield, rq, p,NULL );else -set_task_slice (p,0 ); +scx_set_task_slice (p,0 ); }static bool yield_to_task_scx (struct rq *rq,struct task_struct *to)
@@ -2274,10 +2273,9 @@ static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p, int wake_flscx_schedule_reenq_local (rq,0 ); } -static void move_local_task_to_local_dsq (struct scx_sched *sch, -struct task_struct *p,u64 enq_flags, -struct scx_dispatch_q *src_dsq, -struct rq *dst_rq) +void scx_move_local_task_to_local_dsq (struct scx_sched *sch,struct task_struct *p, +u64 enq_flags,struct scx_dispatch_q *src_dsq, +struct rq *dst_rq) {struct scx_dispatch_q *dst_dsq =scx_resolve_local_dsq (sch, dst_rq, p, &enq_flags);
@@ -2458,7 +2456,7 @@ static bool unlink_dsq_and_switch_rq_lock(struct task_struct *p,lockdep_assert_rq_held (locked_rq);WARN_ON_ONCE (p->scx.holding_cpu >=0 ); -task_unlink_from_dsq (p, dsq); +scx_task_unlink_from_dsq (p, dsq); p->scx.holding_cpu = cpu;raw_spin_unlock (&dsq->lock);
@@ -2529,9 +2527,8 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch,if (dst_dsq->id ==SCX_DSQ_LOCAL ) {/* @p is going from a non-local DSQ to a local DSQ */ if (src_rq == dst_rq) { -task_unlink_from_dsq (p, src_dsq); -move_local_task_to_local_dsq (sch, p, enq_flags, - src_dsq, dst_rq); +scx_task_unlink_from_dsq (p, src_dsq); +scx_move_local_task_to_local_dsq (sch, p, enq_flags, src_dsq, dst_rq);raw_spin_unlock (&src_dsq->lock); }else {raw_spin_unlock (&src_dsq->lock);
@@ -2581,8 +2578,8 @@ bool scx_consume_dispatch_q(struct scx_sched *sch, struct rq *rq,break ;if (rq == task_rq) { -task_unlink_from_dsq (p, dsq); -move_local_task_to_local_dsq (sch, p, enq_flags, dsq, rq); +scx_task_unlink_from_dsq (p, dsq); +scx_move_local_task_to_local_dsq (sch, p, enq_flags, dsq, rq);raw_spin_unlock (&dsq->lock);return true; }
@@ -3530,7 +3527,7 @@ static void task_tick_scx(struct rq *rq, struct task_struct *curr, int queued)* we can't trust the slice management or ops.core_sched_before(). */ if (scx_bypassing (sch,cpu_of (rq))) { -set_task_slice (curr,0 ); +scx_set_task_slice (curr,0 );touch_core_sched (rq, curr); }else if (SCX_HAS_OP (sch, tick)) {SCX_CALL_OP_TASK (sch, tick, rq, curr);
@@ -3685,7 +3682,7 @@ static void scx_disable_task(struct scx_sched *sch, struct task_struct *p)* control, after ops.disable() has observed their final values. */ p->scx.dsq_vtime =0 ; -set_task_slice (p,0 ); +scx_set_task_slice (p,0 ); p->scx.reenq_cnt =0 ;/*
@@ -4184,7 +4181,7 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)if ((reenq_flags &SCX_REENQ_CAP_REVOKE ) && rq->curr->sched_class == &ext_sched_class &&scx_task_reenq_on_cap_revoke (rq, rq->curr)) { -set_task_slice (rq->curr,0 ); +scx_set_task_slice (rq->curr,0 );resched_curr (rq); }
@@ -4748,8 +4745,7 @@ DEFINE_SCHED_CLASS(ext) = {#endif }; -static s32 init_dsq (struct scx_dispatch_q *dsq,u64 dsq_id, -struct scx_sched *sch) +s32 scx_init_dsq (struct scx_dispatch_q *dsq,u64 dsq_id,struct scx_sched *sch) {s32 cpu;
@@ -6375,7 +6371,7 @@ static void dump_newline(struct seq_buf *s)seq_buf_putc (s,'\n' ); } -static __printf(2, 3) void dump_line(struct seq_buf *s, const char *fmt, ...) +__printf(2, 3) void scx_dump_line(struct seq_buf *s, const char *fmt, ...) {va_list args;
@@ -6407,7 +6403,7 @@ static void dump_stack_trace(struct seq_buf *s, const char *prefix,unsigned int i;for (i =0 ; i < len; i++) -dump_line (s,"%s%pS" , prefix, (void *)bt[i]); +scx_dump_line (s,"%s%pS" , prefix, (void *)bt[i]); }static void ops_dump_init (struct seq_buf *s,const char *prefix)
@@ -6457,7 +6453,7 @@ static void ops_dump_flush(void)*/ c = *end; *end ='\0' ; -dump_line (dd->s,"%s%s" , dd->prefix, line); +scx_dump_line (dd->s,"%s%s" , dd->prefix, line);if (c =='\0' )break ;
@@ -6501,21 +6497,19 @@ static void scx_dump_task(struct scx_sched *sch, struct seq_buf *s, struct scx_d (unsigned long long)p->scx.dsq->id);dump_newline (s); -dump_line (s," %c%c %s[%d] %s%s %+ldms" , - marker,task_state_to_char (p), p->comm, p->pid, - own_marker, sch_id_buf, -jiffies_delta_msecs (p->scx.runnable_at, dctx->at_jiffies)); -dump_line (s," scx_state/flags=%u/0x%x dsq_flags=0x%x ops_state/qseq=%lu/%lu" , -scx_get_task_state (p) >>SCX_TASK_STATE_SHIFT , - p->scx.flags & ~SCX_TASK_STATE_MASK , - p->scx.dsq_flags, ops_state &SCX_OPSS_STATE_MASK , - ops_state >>SCX_OPSS_QSEQ_SHIFT ); -dump_line (s," sticky/holding_cpu=%d/%d dsq_id=%s" , - p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf); -dump_line (s," dsq_vtime=%llu slice=%llu weight=%u" , - p->scx.dsq_vtime, p->scx.slice, p->scx.weight); -dump_line (s," cpus=%*pb no_mig=%u" ,cpumask_pr_args (p->cpus_ptr), - p->migration_disabled); +scx_dump_line (s," %c%c %s[%d] %s%s %+ldms" , + marker,task_state_to_char (p), p->comm, p->pid, own_marker, sch_id_buf, +jiffies_delta_msecs (p->scx.runnable_at, dctx->at_jiffies)); +scx_dump_line (s," scx_state/flags=%u/0x%x dsq_flags=0x%x ops_state/qseq=%lu/%lu" , +scx_get_task_state (p) >>SCX_TASK_STATE_SHIFT , + p->scx.flags & ~SCX_TASK_STATE_MASK , p->scx.dsq_flags, + ops_state &SCX_OPSS_STATE_MASK , ops_state >>SCX_OPSS_QSEQ_SHIFT ); +scx_dump_line (s," sticky/holding_cpu=%d/%d dsq_id=%s" , + p->scx.sticky_cpu, p->scx.holding_cpu, dsq_id_buf); +scx_dump_line (s," dsq_vtime=%llu slice=%llu weight=%u" , + p->scx.dsq_vtime, p->scx.slice, p->scx.weight); +scx_dump_line (s," cpus=%*pb no_mig=%u" ,cpumask_pr_args (p->cpus_ptr), + p->migration_disabled);if (SCX_HAS_OP (sch, dump_task)) {ops_dump_init (s," " );
@@ -6563,28 +6557,26 @@ static void scx_dump_cpu(struct scx_sched *sch, struct seq_buf *s,seq_buf_init (&ns, buf, avail);dump_newline (&ns); -dump_line (&ns,"CPU %-4d: nr_run=%u flags=0x%x cpu_rel=%d ops_qseq=%lu ksync=%lu" , - cpu, rq->scx.nr_running, rq->scx.flags, - rq->scx.cpu_released, rq->scx.ops_qseq, - rq->scx.kick_sync); -dump_line (&ns," curr=%s[%d] class=%ps" , - rq->curr->comm, rq->curr->pid, - rq->curr->sched_class); +scx_dump_line (&ns,"CPU %-4d: nr_run=%u flags=0x%x cpu_rel=%d ops_qseq=%lu ksync=%lu" , + cpu, rq->scx.nr_running, rq->scx.flags, rq->scx.cpu_released, + rq->scx.ops_qseq, rq->scx.kick_sync); +scx_dump_line (&ns," curr=%s[%d] class=%ps" , + rq->curr->comm, rq->curr->pid, rq->curr->sched_class);if (!cpumask_empty (pcpu->cpus_to_kick)) -dump_line (&ns," cpus_to_kick : %*pb" , -cpumask_pr_args (pcpu->cpus_to_kick)); +scx_dump_line (&ns," cpus_to_kick : %*pb" , +cpumask_pr_args (pcpu->cpus_to_kick));if (!cpumask_empty (pcpu->cpus_to_kick_if_idle)) -dump_line (&ns," idle_to_kick : %*pb" , -cpumask_pr_args (pcpu->cpus_to_kick_if_idle)); +scx_dump_line (&ns," idle_to_kick : %*pb" , +cpumask_pr_args (pcpu->cpus_to_kick_if_idle));if (!cpumask_empty (pcpu->cpus_to_preempt)) -dump_line (&ns," cpus_to_preempt: %*pb" , -cpumask_pr_args (pcpu->cpus_to_preempt)); +scx_dump_line (&ns," cpus_to_preempt: %*pb" , +cpumask_pr_args (pcpu->cpus_to_preempt));if (!cpumask_empty (pcpu->cpus_to_wait)) -dump_line (&ns," cpus_to_wait : %*pb" , -cpumask_pr_args (pcpu->cpus_to_wait)); +scx_dump_line (&ns," cpus_to_wait : %*pb" , +cpumask_pr_args (pcpu->cpus_to_wait));if (!cpumask_empty (rq->scx.cpus_to_sync)) -dump_line (&ns," cpus_to_sync : %*pb" , -cpumask_pr_args (rq->scx.cpus_to_sync)); +scx_dump_line (&ns," cpus_to_sync : %*pb" , +cpumask_pr_args (rq->scx.cpus_to_sync)); used =seq_buf_used (&ns);if (SCX_HAS_OP (sch, dump_cpu)) {
@@ -6652,25 +6644,25 @@ static void scx_dump_state(struct scx_sched *sch, struct scx_exit_info *ei,#ifdef CONFIG_EXT_SUB_SCHED if (sch->level ==0 ) -dump_line (&s,"%s: root" , sch->ops.name); +scx_dump_line (&s,"%s: root" , sch->ops.name);else -dump_line (&s,"%s: sub%d-%llu %s" , - sch->ops.name, sch->level, sch->ops.sub_cgroup_id, - sch->cgrp_path); +scx_dump_line (&s,"%s: sub%d-%llu %s" , + sch->ops.name, sch->level, sch->ops.sub_cgroup_id, + sch->cgrp_path);#endif if (ei->kind ==SCX_EXIT_NONE ) { -dump_line (&s,"Debug dump triggered by %s" , ei->reason); +scx_dump_line (&s,"Debug dump triggered by %s" , ei->reason); }else {if (ei->exit_cpu >=0 ) -dump_line (&s,"%s[%d] triggered exit kind %d on CPU %d:" , - current->comm, current->pid, ei->kind, - ei->exit_cpu); +scx_dump_line (&s,"%s[%d] triggered exit kind %d on CPU %d:" , + current->comm, current->pid, ei->kind, + ei->exit_cpu);else -dump_line (&s,"%s[%d] triggered exit kind %d:" , - current->comm, current->pid, ei->kind); -dump_line (&s," %s (%s)" , ei->reason, ei->msg); +scx_dump_line (&s,"%s[%d] triggered exit kind %d:" , + current->comm, current->pid, ei->kind); +scx_dump_line (&s," %s (%s)" , ei->reason, ei->msg);dump_newline (&s); -dump_line (&s,"Backtrace:" ); +scx_dump_line (&s,"Backtrace:" );dump_stack_trace (&s," " , ei->bt, ei->bt_len); }
@@ -6681,8 +6673,8 @@ static void scx_dump_state(struct scx_sched *sch, struct scx_exit_info *ei, }dump_newline (&s); -dump_line (&s,"CPU states" ); -dump_line (&s,"----------" ); +scx_dump_line (&s,"CPU states" ); +scx_dump_line (&s,"----------" );/* * Dump stalled CPUs first so they aren't lost to dump truncation, then
@@ -6705,8 +6697,8 @@ static void scx_dump_state(struct scx_sched *sch, struct scx_exit_info *ei, }dump_newline (&s); -dump_line (&s,"Event counters" ); -dump_line (&s,"--------------" ); +scx_dump_line (&s,"Event counters" ); +scx_dump_line (&s,"--------------" );scx_read_events (sch, &events);#define SCX_EVENT (name) scx_dump_event(s, &events, name)
@@ -6815,7 +6807,7 @@ static struct scx_sched_pnode *alloc_pnode(struct scx_sched *sch, int node)if (!pnode)return NULL ; -if (init_dsq (&pnode->global_dsq,SCX_DSQ_GLOBAL , sch)) { +if (scx_init_dsq (&pnode->global_dsq,SCX_DSQ_GLOBAL , sch)) {kfree (pnode);return NULL ; }
@@ -6876,7 +6868,7 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd, }for_each_possible_cpu (cpu) { - ret =init_dsq (scx_bypass_dsq (sch, cpu),SCX_DSQ_BYPASS , sch); + ret =scx_init_dsq (scx_bypass_dsq (sch, cpu),SCX_DSQ_BYPASS , sch);if (ret) { bypass_fail_cpu = cpu; goto err_free_pcpu;
@@ -7509,7 +7501,7 @@ static void scx_root_enable_workfn(struct kthread_work *work) queue_flags |=DEQUEUE_CLASS ; scoped_guard (sched_change, p, queue_flags) { -set_task_slice (p,READ_ONCE (sch->slice_dfl)); +scx_set_task_slice (p,READ_ONCE (sch->slice_dfl)); p->sched_class = new_class; } }
@@ -8160,7 +8152,7 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_rif (cur_class == &ext_sched_class) {if (likely (!scx_missing_caps (pcpu->sch, cpu,scx_caps_for_preempt (pcpu->sch, rq)))) -set_task_slice (rq->curr,0 ); +scx_set_task_slice (rq->curr,0 );else __scx_add_event (pcpu->sch,SCX_EV_SUB_PREEMPT_DENIED ,1 );
@@ -8364,9 +8356,9 @@ void __init init_sched_ext_class(void)int n =cpu_to_node (cpu);/* local_dsq's sch will be set during scx_root_enable() */ -BUG_ON (init_dsq (&rq->scx.local_dsq,SCX_DSQ_LOCAL ,NULL )); +BUG_ON (scx_init_dsq (&rq->scx.local_dsq,SCX_DSQ_LOCAL ,NULL ));#ifdef CONFIG_EXT_SUB_SCHED -BUG_ON (init_dsq (&rq->scx.reject_dsq,SCX_DSQ_REJECT ,NULL )); +BUG_ON (scx_init_dsq (&rq->scx.reject_dsq,SCX_DSQ_REJECT ,NULL ));#endif INIT_LIST_HEAD (&rq->scx.runnable_list);
@@ -8523,7 +8515,7 @@ __bpf_kfunc bool scx_bpf_dsq_insert___v2(struct task_struct *p, u64 dsq_id,return false;if (slice) -set_task_slice (p, slice); +scx_set_task_slice (p, slice);else set_task_slice_keep_oob (p, p->scx.slice ?:1 );
@@ -8549,7 +8541,7 @@ static bool scx_dsq_insert_vtime(struct scx_sched *sch, struct task_struct *p,return false;if (slice) -set_task_slice (p, slice); +scx_set_task_slice (p, slice);else set_task_slice_keep_oob (p, p->scx.slice ?:1 );
@@ -8733,7 +8725,7 @@ static bool scx_dsq_move(struct bpf_iter_scx_dsq_kern *kit,if (kit->cursor.flags & __SCX_DSQ_ITER_HAS_VTIME) p->scx.dsq_vtime = kit->vtime;if (kit->cursor.flags & __SCX_DSQ_ITER_HAS_SLICE) -set_task_slice (p, kit->slice); +scx_set_task_slice (p, kit->slice);/* execute move */ locked_rq =move_task_between_dsqs (sch, p, enq_flags, src_dsq, dst_dsq);
@@ -9056,10 +9048,10 @@ __bpf_kfunc s32 scx_bpf_create_dsq(u64 dsq_id, s32 node, const struct bpf_prog_areturn -ENOMEM ;/* -* init_dsq() must be called in GFP_KERNEL context. Init it with NULL -* @sch and update afterwards. +* scx_init_dsq() must be called in GFP_KERNEL context. Init it with +* NULL @sch and update afterwards. */ - ret =init_dsq (dsq, dsq_id,NULL ); + ret =scx_init_dsq (dsq, dsq_id,NULL );if (ret) {kfree (dsq);return ret;
@@ -9153,7 +9145,7 @@ __bpf_kfunc bool scx_bpf_task_set_slice(struct task_struct *p, u64 slice, unlikely(scx_missing_caps(sch, cpu_of(locked_rq), SCX_CAP_BASE)))__scx_add_event (sch,SCX_EV_SLICE_DENIED ,1 ); else -set_task_slice (p ,slice ); +scx_set_task_slice (p ,slice );return true; }
@@ -9702,8 +9694,8 @@ __bpf_kfunc void scx_bpf_dump_bstr(char *fmt, unsigned long long *data, ret =__bstr_format (sch, buf->data, buf->line + dd->cursor,sizeof (buf->line) - dd->cursor, fmt, data, data__sz);if (ret <0 ) { -dump_line (dd->s,"%s[!] (\"%s\", %p, %u) failed to format (%d)" , - dd->prefix, fmt, data, data__sz, ret); +scx_dump_line (dd->s,"%s[!] (\"%s\", %p, %u) failed to format (%d)" , + dd->prefix, fmt, data, data__sz, ret);return ; }
kernel/sched/ext/internal.h +8 -0
@@ -1968,14 +1968,21 @@ void scx_task_iter_start(struct scx_task_iter *iter, struct cgroup *cgrp);void scx_task_iter_unlock (struct scx_task_iter *iter);void scx_task_iter_stop (struct scx_task_iter *iter);struct task_struct *scx_task_iter_next_locked (struct scx_task_iter *iter); +void scx_set_task_slice (struct task_struct *p,u64 slice); +void scx_task_unlink_from_dsq (struct task_struct *p,struct scx_dispatch_q *dsq);void scx_dispatch_dequeue (struct rq *rq,struct task_struct *p);void scx_do_enqueue_task (struct rq *rq,struct task_struct *p,u64 enq_flags,int sticky_cpu); +void scx_move_local_task_to_local_dsq (struct scx_sched *sch,struct task_struct *p, +u64 enq_flags,struct scx_dispatch_q *src_dsq, +struct rq *dst_rq);bool scx_consume_dispatch_q (struct scx_sched *sch,struct rq *rq,struct scx_dispatch_q *dsq,u64 enq_flags);bool scx_consume_global_dsq (struct scx_sched *sch,struct rq *rq);bool scx_rq_online (struct rq *rq);void scx_flush_dispatch_buf (struct scx_sched *sch,struct rq *rq); +s32 scx_init_dsq (struct scx_dispatch_q *dsq,u64 dsq_id,struct scx_sched *sch); +__printf(2, 3) void scx_dump_line(struct seq_buf *s, const char *fmt, ...);void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags);void schedule_dsq_reenq (struct scx_sched *sch,struct scx_dispatch_q *dsq,u64 reenq_flags,struct rq *locked_rq);
@@ -2011,6 +2018,7 @@ extern raw_spinlock_t scx_sched_lock;extern struct mutex scx_enable_mutex;extern struct percpu_rw_semaphore scx_fork_rwsem;extern bool scx_cgroup_enabled; +extern struct list_head scx_sched_all;#ifdef CONFIG_EXT_SUB_SCHED extern const struct rhashtable_params scx_sched_hash_params;extern struct rhashtable scx_sched_hash;
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 3/12 sched_ext: Factor out __scx_bpf_now()
In reply to Tejun Heoscx_bpf_now() couples the valid-or-fresh rq clock read to the current rq. The read is useful for kernel-internal timing against a specific rq, including a remotely locked one. Factor it out into __scx_bpf_now().
--- kernel/sched/ext/ext.c | 59 ++++++++++++++++++------------------- kernel/sched/ext/internal.h | 1 + 2 files changed, 30 insertions(+), 30 deletions(-)
kernel/sched/ext/ext.c +29 -30
@@ -10160,6 +10160,27 @@ __bpf_kfunc struct task_struct *scx_bpf_tid_to_task(u64 tid) return container_of(scx, struct task_struct, scx); } +u64 __scx_bpf_now (struct rq *rq) +{ +/* the caller must be on @rq's cpu or hold its lock */ +lockdep_assert ((rq ==this_rq () && !preemptible ()) || +lockdep_is_held (__rq_lockp (rq))); + +if (smp_load_acquire (&rq->scx.flags) &SCX_RQ_CLK_VALID ) { +/* if the rq clock is valid, use the cached rq clock */ +return READ_ONCE (rq->scx.clock); + }else { +/* +* Otherwise, return a fresh rq clock. +* +* The rq clock is updated outside of the rq lock. +* In this case, keep the updated rq clock invalid so the next +* read outside the rq lock gets a fresh rq clock. +*/ +return sched_clock_cpu (cpu_of (rq)); + } +} +/** * scx_bpf_now - Returns a high-performance monotonically non-decreasing * clock for the current CPU. The clock returned is in nanoseconds.
@@ -10190,36 +10211,14 @@ __bpf_kfunc struct task_struct *scx_bpf_tid_to_task(u64 tid)*/ __bpf_kfunc u64 scx_bpf_now(void) { -struct rq *rq; -u64 clock; - -preempt_disable (); - - rq =this_rq (); -if (smp_load_acquire (&rq->scx.flags) &SCX_RQ_CLK_VALID ) { -/* -* If the rq clock is valid, use the cached rq clock. -* -* Note that scx_bpf_now() is re-entrant between a process -* context and an interrupt context (e.g., timer interrupt). -* However, we don't need to consider the race between them -* because such race is not observable from a caller. -*/ - clock =READ_ONCE (rq->scx.clock); - }else { -/* -* Otherwise, return a fresh rq clock. -* -* The rq clock is updated outside of the rq lock. -* In this case, keep the updated rq clock invalid so the next -* kfunc call outside the rq lock gets a fresh rq clock. -*/ - clock =sched_clock_cpu (cpu_of (rq)); - } - -preempt_enable (); - -return clock; +/* +* Note that scx_bpf_now() is re-entrant between a process context and +* an interrupt context (e.g., timer interrupt). However, we don't need +* to consider the race between them because such race is not observable +* from a caller. +*/ +guard (preempt)(); +return __scx_bpf_now (this_rq ()); }static void scx_read_events (struct scx_sched *sch,struct scx_event_stats *events)
kernel/sched/ext/internal.h +1 -0
@@ -1984,6 +1984,7 @@ void scx_flush_dispatch_buf(struct scx_sched *sch, struct rq *rq);s32 scx_init_dsq (struct scx_dispatch_q *dsq,u64 dsq_id,struct scx_sched *sch); __printf(2, 3) void scx_dump_line(struct seq_buf *s, const char *fmt, ...);void scx_kick_cpu (struct scx_sched *sch,s32 cpu,u64 flags); +u64 __scx_bpf_now (struct rq *rq);void schedule_dsq_reenq (struct scx_sched *sch,struct scx_dispatch_q *dsq,u64 reenq_flags,struct rq *locked_rq); int __scx_init_task(struct scx_sched *sch, struct task_struct *p,
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 4/12 sched_ext: Reject internal enq_flags in the dsq move kfuncs
In reply to Tejun HeoThe dsq insert kfuncs reject __SCX_ENQ_INTERNAL_MASK bits in scx_dsq_insert_preamble() instead of scx_vet_enq_flags(). A scheduler can smuggle internal flags such as SCX_ENQ_CLEAR_OPSS through the dsq move kfuncs and corrupt the dispatch protocol. Move the rejection into scx_vet_enq_flags(). The vtime move wrapper OR'd the internal SCX_ENQ_DSQ_PRIQ bit into enq_flags before the vet; the bit now goes in inside scx_dsq_move() after the vet.
--- kernel/sched/ext/ext.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
kernel/sched/ext/ext.c +13 -8
@@ -8394,6 +8394,11 @@ static bool scx_vet_enq_flags(struct scx_sched *sch, u64 dsq_id, u64 *enq_flags)bool is_local = dsq_id ==SCX_DSQ_LOCAL || (dsq_id &SCX_DSQ_LOCAL_ON ) ==SCX_DSQ_LOCAL_ON ; +if (unlikely (*enq_flags & __SCX_ENQ_INTERNAL_MASK)) { +scx_error (sch,"invalid enq_flags 0x%llx" , *enq_flags); +return false; + } +if (*enq_flags &SCX_ENQ_IMMED ) {if (unlikely (!is_local)) {scx_error (sch,"SCX_ENQ_IMMED on a non-local DSQ 0x%llx" , dsq_id);
@@ -8416,11 +8421,6 @@ static bool scx_dsq_insert_preamble(struct scx_sched *sch, struct task_struct *preturn false; } -if (unlikely (*enq_flags & __SCX_ENQ_INTERNAL_MASK)) { -scx_error (sch,"invalid enq_flags 0x%llx" , *enq_flags); -return false; - } -/* see SCX_EV_INSERT_NOT_OWNED definition */ if (unlikely (!scx_task_on_sched (sch, p))) {__scx_add_event (sch,SCX_EV_INSERT_NOT_OWNED ,1 );
@@ -8652,7 +8652,8 @@ static const struct btf_kfunc_id_set scx_kfunc_set_enqueue_dispatch = { };static bool scx_dsq_move (struct bpf_iter_scx_dsq_kern *kit, -struct task_struct *p,u64 dsq_id,u64 enq_flags) +struct task_struct *p,u64 dsq_id,u64 enq_flags, +bool priq) {struct scx_dispatch_q *src_dsq = kit->dsq, *dst_dsq;struct scx_sched *sch;
@@ -8674,6 +8675,10 @@ static bool scx_dsq_move(struct bpf_iter_scx_dsq_kern *kit,if (!scx_vet_enq_flags (sch, dsq_id, &enq_flags))return false; +/* internal bit, can only go in after @enq_flags is vetted */ +if (priq) + enq_flags |=SCX_ENQ_DSQ_PRIQ ; +/* * If the BPF scheduler keeps calling this function repeatedly, it can * cause similar live-lock conditions as scx_consume_dispatch_q().
@@ -8930,7 +8935,7 @@ __bpf_kfunc bool scx_bpf_dsq_move(struct bpf_iter_scx_dsq *it__iter, u64 enq_flags) {return scx_dsq_move ((struct bpf_iter_scx_dsq_kern *)it__iter, - p, dsq_id, enq_flags); + p, dsq_id, enq_flags, false); }/**
@@ -8955,7 +8960,7 @@ __bpf_kfunc bool scx_bpf_dsq_move_vtime(struct bpf_iter_scx_dsq *it__iter, u64 enq_flags) {return scx_dsq_move ((struct bpf_iter_scx_dsq_kern *)it__iter, - p, dsq_id, enq_flags |SCX_ENQ_DSQ_PRIQ ); + p, dsq_id, enq_flags, true); }__bpf_kfunc_end_defs ();
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 5/12 sched_ext: Make SCX_ENQ_IGNORE_CAPS waive the preemption cap too
In reply to Tejun HeoSCX_ENQ_IGNORE_CAPS is kernel-internal and marks a placement the kernel forces. scx_caps_for_enq() waives the enqueue cap for it, but a PREEMPT insert still picks up the preemption cap requirement from scx_caps_for_preempt(). Update scx_caps_for_preempt() to take enq_flags and require nothing when SCX_ENQ_IGNORE_CAPS is set.
--- kernel/sched/ext/ext.c | 5 +++-- kernel/sched/ext/sub.c | 2 +- kernel/sched/ext/sub.h | 9 ++++++--- 3 files changed, 10 insertions(+), 6 deletions(-)
kernel/sched/ext/ext.c +3 -2
@@ -8150,8 +8150,9 @@ static bool kick_one_cpu(s32 cpu, struct scx_sched_pcpu *pcpu, struct rq *this_rif (kickable && !scx_missing_caps (pcpu->sch, cpu,SCX_CAP_BASE )) {if (cpumask_test_cpu (cpu, pcpu->cpus_to_preempt)) {if (cur_class == &ext_sched_class) { -if (likely (!scx_missing_caps (pcpu->sch, cpu, -scx_caps_for_preempt (pcpu->sch, rq)))) +u64 caps =scx_caps_for_preempt (pcpu->sch, rq,0 ); + +if (likely (!scx_missing_caps (pcpu->sch, cpu, caps)))scx_set_task_slice (rq->curr,0 );else __scx_add_event(pcpu->sch,
kernel/sched/ext/sub.c +1 -1
@@ -262,7 +262,7 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r* @p's owner (@sch). Check caps against the scheduling sched. */ if (*enq_flags &SCX_ENQ_PREEMPT ) - needed |=scx_caps_for_preempt (asch, rq); + needed |=scx_caps_for_preempt (asch, rq, *enq_flags); missing =scx_missing_caps (asch,cpu_of (rq), needed);/* requirements met */
kernel/sched/ext/sub.h +6 -3
@@ -140,7 +140,7 @@ static inline u64 scx_missing_caps(struct scx_sched *sch, s32 cpu, u64 needed)static inline u64 scx_caps_for_enq (u64 enq_flags) {/* a restored task must be put into the local DSQ regardless of caps */ -if (enq_flags &SCX_ENQ_IGNORE_CAPS ) +if (unlikely (enq_flags &SCX_ENQ_IGNORE_CAPS ))return 0 ;if (enq_flags &SCX_ENQ_IMMED )return SCX_CAP_ENQ_IMMED ;
@@ -156,10 +156,13 @@ static inline u64 scx_caps_for_task(struct task_struct *p) }/* the cap @sch needs to preempt @rq's current task, 0 if none */ -static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq) +static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq,u64 enq_flags) {struct task_struct *curr = rq->curr; +/* a kernel-forced placement preempts regardless of caps */ +if (unlikely (enq_flags &SCX_ENQ_IGNORE_CAPS )) +return 0 ;/* a non-ext task can't be preempted by ext, own-subtree needs no cap */ if (curr->sched_class != &ext_sched_class ||scx_is_descendant (scx_task_sched (curr), sch))
@@ -195,7 +198,7 @@ static inline bool scx_task_can_stay_on_cpu(struct rq *rq, struct task_struct *p#else /* CONFIG_EXT_SUB_SCHED */ static inline u64 scx_missing_caps (struct scx_sched *sch,s32 cpu,u64 needed) {return 0 ; } -static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq) {return 0 ; } +static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq,u64 enq_flags) {return 0 ; }static inline bool scx_task_can_stay_on_cpu (struct rq *rq,struct task_struct *p) {return true; }#endif /* CONFIG_EXT_SUB_SCHED */
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 6/12 sched_ext: Synchronize slice and dsq_vtime writes
In reply to Tejun Heop->scx.slice and p->scx.dsq_vtime writes have no synchronization rules. The dsq insert kfuncs write both fields synchronously from whatever context they're called in - a direct dispatch from ops.select_cpu() writes with only pi_lock held - and, as the kfuncs are safe to call spuriously with the invalid dispatch discarded later, a scheduler can modify any task's slice by spuriously calling them. The latter stands in the way of an upcoming patch which adds kernel-granted slices that the schedulers must not be able to modify.
Give both fields explicit rules. While the task is running, sleeping or queued on an rq-owned DSQ, the rq lock protects them - these are the states where the kernel consumes the slice. While queued on a user DSQ or on the BPF side, the kernel neither consumes nor decides on the fields and every writer acts for the BPF scheduler - synchronizing the writers is the scheduler's responsibility and whichever write lands last wins.
To conform, an insert kfunc no longer writes the fields when called. The values travel with the dispatch and take effect when the task is inserted. A discarded dispatch has no side effects. The rq lock rule is asserted at the slice store.
--- include/linux/sched/ext.h | 2 + kernel/sched/ext/ext.c | 184 ++++++++++++++++++++++-------------- kernel/sched/ext/internal.h | 4 + 3 files changed, 120 insertions(+), 70 deletions(-)
include/linux/sched/ext.h +2 -0
@@ -192,6 +192,8 @@ struct sched_ext_entity {atomic_long_t ops_state;u64 ddsp_dsq_id;u64 ddsp_enq_flags; +u64 ddsp_slice; +u64 ddsp_vtime;struct scx_dsq_list_node dsq_list;/* dispatch order */ struct rb_node dsq_priq;/* p->scx.dsq_vtime order */ u32 dsq_seq;
kernel/sched/ext/ext.c +114 -70
395 lines not on this page · raw message
kernel/sched/ext/internal.h +4 -0
@@ -1246,6 +1246,8 @@ struct scx_dsp_buf_ent {struct task_struct *task;unsigned long qseq;u64 dsq_id; +u64 slice; +u64 vtime;u64 enq_flags; };
@@ -1680,6 +1682,8 @@ enum scx_enq_flags {SCX_ENQ_NESTED =1LLU <<58 ,SCX_ENQ_GDSQ_FALLBACK =1LLU <<59 ,/* fell back to global DSQ */ SCX_ENQ_IGNORE_CAPS =1LLU <<60 ,/* admit to local DSQ ignoring caps */ +SCX_ENQ_APPLY_SLICE =1LLU <<61 ,/* apply carried slice/vtime at insertion */ + SCX_ENQ_SLICE_DFL = 1LLU << 62, /* carried slice is a default refill */ }; enum scx_deq_flags {
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 7/12 sched_ext: Add SCX_TASK_PROTECTED
In reply to Tejun HeoA BPF scheduler can displace any of its tasks at will - cut a running one's slice with an SCX_ENQ_PREEMPT dispatch, an SCX_KICK_PREEMPT kick or a direct shortening, and jump a queued one with HEAD insertions. Sometimes the kernel needs a slice and a DSQ position to stick regardless.
Add SCX_TASK_PROTECTED, guarding both:
- The slice becomes immutable. Every scheduler-reachable write is refused and counted as SCX_EV_SLICE_DENIED. Higher scheduling classes are unaffected. PREEMPT|IMMED can't preempt a running protected task and gets reenqueued.
- A protected task that reached the head of its DSQ keeps it - HEAD insertions land behind the leading run of protected tasks and reenqueue sweeps skip them. Only rq-owned DSQs can hold protected tasks, so the walk runs only for them.
The bit lives in p->scx.flags so that both the refusal and the head walk read it under the rq lock that protects it.
Protection ends when the slice is consumed, when the task leaves the rq except for a save/restore on the running task, on a yield, when the scheduler enters bypass, and when the task leaves scx. The flag is kernel-internal and not used yet.
--- include/linux/sched/ext.h | 1 + kernel/sched/ext/ext.c | 165 +++++++++++++++++++++++++++++++----- kernel/sched/ext/internal.h | 2 +- 3 files changed, 144 insertions(+), 24 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -102,6 +102,7 @@ enum scx_ent_flags {SCX_TASK_DEQD_FOR_SLEEP =1 <<3 ,/* last dequeue was for SLEEP */ SCX_TASK_SUB_INIT =1 <<4 ,/* task being initialized for a sub sched */ SCX_TASK_IMMED =1 <<5 ,/* task is on local DSQ with %SCX_ENQ_IMMED */ +SCX_TASK_PROTECTED =1 <<6 ,/* slice and DSQ head position protected */ /* * Bits 8 to 10 are used to carry task state:
kernel/sched/ext/ext.c +142 -23
284 lines not on this page · raw message
kernel/sched/ext/internal.h +1 -1
@@ -1972,7 +1972,7 @@ void scx_task_iter_start(struct scx_task_iter *iter, struct cgroup *cgrp);void scx_task_iter_unlock (struct scx_task_iter *iter);void scx_task_iter_stop (struct scx_task_iter *iter);struct task_struct *scx_task_iter_next_locked (struct scx_task_iter *iter); -void scx_set_task_slice (struct task_struct *p,u64 slice); +bool scx_set_task_slice (struct task_struct *p,u64 slice);void scx_task_unlink_from_dsq (struct task_struct *p,struct scx_dispatch_q *dsq);void scx_dispatch_dequeue (struct rq *rq,struct task_struct *p); void scx_do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 8/12 sched_ext: Add bandwidth-limited rescue execution for stranded tasks
In reply to Tejun HeoA local DSQ insert lacking the needed caps is diverted to the reject DSQ and bounced back through ops.enqueue() so the scheduler can re-decide. That recovery assumes the scheduler has somewhere legal to send the task. When it doesn't, e.g. when the task's affinity is restricted to cids delegated away, the task starves until the stall watchdog ejects the scheduler. An exiting task is worse - it skips ops.enqueue() and the rejection becomes a self-requeuing cycle that burns the CPU until the watchdog fires.
Add SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the insert would be rejected for missing caps, the kernel takes over and runs the task on the target CPU without consulting the owning scheduler. The kernel sets the flag itself when enqueueing an exiting task.
Rescue is a last-resort forward-progress backstop with a persistent disadvantage, not a way around cap enforcement. A per-CPU token bucket accrues rescue_bandwidth_ppt (default 2%) of CPU time and rescues run one at a time in arrival order. Each is granted a slice of the rescue_quantum_us (default 5ms) quantum divided across the waiters, waits at the tail of the local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival once the slice is served.
The schedulers keep their normal control over an admitted rescuee and may preempt or reslice it. Service is measured on CPU time actually received, so neither shortens the rescue. Prolonged denial escalates - the remaining slice turns into protected execution (SCX_TASK_PROTECTED) and the rescuee preempts the current task. Escalation is paced by the same bucket, and delivered service converges on the configured bandwidth no matter how aggressively the schedulers dispatch.
Both knobs are root-only and SCX_RESCUE_DISABLE turns rescue off, making SCX_ENQ_RESCUE inserts reject as usual.
--- include/linux/sched/ext.h | 1 + kernel/sched/ext/ext.c | 112 +++++++++-- kernel/sched/ext/internal.h | 56 +++++- kernel/sched/ext/sub.c | 376 ++++++++++++++++++++++++++++++++++-- kernel/sched/ext/sub.h | 26 +++ kernel/sched/ext/types.h | 9 + kernel/sched/sched.h | 12 ++ 7 files changed, 559 insertions(+), 33 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -59,6 +59,7 @@ enum scx_dsq_id_flags {SCX_DSQ_LOCAL =SCX_DSQ_FLAG_BUILTIN |2 ,SCX_DSQ_BYPASS =SCX_DSQ_FLAG_BUILTIN |3 ,SCX_DSQ_REJECT =SCX_DSQ_FLAG_BUILTIN |4 ,/* internal - see find_dsq_for_dispatch() */ +SCX_DSQ_RESCUE =SCX_DSQ_FLAG_BUILTIN |5 ,/* internal - see find_dsq_for_dispatch() */ SCX_DSQ_LOCAL_ON =SCX_DSQ_FLAG_BUILTIN |SCX_DSQ_FLAG_LOCAL_ON , SCX_DSQ_LOCAL_CPU_MASK = 0xffffffffLLU, };
kernel/sched/ext/ext.c +95 -17
250 lines not on this page · raw message
kernel/sched/ext/internal.h +55 -1
104 lines not on this page · raw message
kernel/sched/ext/sub.c +361 -15
411 lines not on this page · raw message
kernel/sched/ext/sub.h +26 -0
@@ -38,6 +38,13 @@ struct scx_dispatch_q *scx_resolve_local_dsq(struct scx_sched *sch, struct rq *r struct task_struct *p, u64 *enq_flags);bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p);void scx_reenq_reject (struct rq *rq); +void scx_rescue_charge (struct rq *rq,s64 delta_exec); +void scx_rescue_end (struct rq *rq); +bool scx_rescue_keep (struct rq *rq,struct task_struct *p); +void scx_rescue_flush (struct rq *rq); +void scx_rescue_dump (struct seq_buf *s,struct rq *rq); +void scx_rescue_set_knobs (struct scx_sched *sch); +void scx_rescue_init (struct rq *rq);/* * cgrp->scx_sched is written by root/sub enable/disable under all of
@@ -87,6 +94,13 @@ static inline void scx_discard_stale_ecaps_syncs(void) {}static inline struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) {return &rq->scx.local_dsq; }static inline bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p) {return false; }static inline void scx_reenq_reject (struct rq *rq) {} +static inline void scx_rescue_charge (struct rq *rq,s64 delta_exec) {} +static inline void scx_rescue_end (struct rq *rq) {} +static inline bool scx_rescue_keep (struct rq *rq,struct task_struct *p) {return false; } +static inline void scx_rescue_flush (struct rq *rq) {} +static inline void scx_rescue_dump (struct seq_buf *s,struct rq *rq) {} +static inline void scx_rescue_set_knobs (struct scx_sched *sch) {} +static inline void scx_rescue_init (struct rq *rq) {}static inline void scx_dec_has_subs (struct scx_sched *sch) {}#endif /* CONFIG_EXT_SUB_SCHED */
@@ -195,11 +209,23 @@ static inline bool scx_task_can_stay_on_cpu(struct rq *rq, struct task_struct *p return likely(!scx_missing_caps(scx_task_sched(p), cpu_of(rq), SCX_CAP_BASE)); } +/* the task admitted for rescue on @rq, NULL if none */ +static inline struct task_struct *scx_rescuee (struct rq *rq) +{ +lockdep_assert_rq_held (rq); + +if (!scx_has_subs ()) +return NULL ; + +return rq->scx.rescue.curr; +} +#else /* CONFIG_EXT_SUB_SCHED */ static inline u64 scx_missing_caps (struct scx_sched *sch,s32 cpu,u64 needed) {return 0 ; }static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq,u64 enq_flags) {return 0 ; }static inline bool scx_task_can_stay_on_cpu (struct rq *rq,struct task_struct *p) {return true; } +static inline struct task_struct *scx_rescuee (struct rq *rq) {return NULL ; }#endif /* CONFIG_EXT_SUB_SCHED */
kernel/sched/ext/types.h +9 -0
@@ -19,6 +19,15 @@ enum scx_consts {SCX_DSP_MAX_LOOPS =32 ,SCX_WATCHDOG_MAX_TIMEOUT =30 *HZ , +/* rescue knob defaults and limits, see scx_rescue_timerfn() */ +SCX_RESCUE_DFL_BW_PPT =20 ,/* parts per thousand, 2% */ +SCX_RESCUE_MAX_BW_PPT =250 ,/* 25% */ +SCX_RESCUE_DISABLE =U32_MAX ,/* disables rescue */ +SCX_RESCUE_DFL_QUANTUM_US =5000 , +SCX_RESCUE_MIN_QUANTUM_US =1000 , +SCX_RESCUE_MAX_QUANTUM_US =100000 , +SCX_RESCUE_MIN_SLICE_US =1000 ,/* floor of the divided slice */ +/* per-CPU chunk size for p->scx.tid allocation, see scx_alloc_tid() */ SCX_TID_CHUNK = 1024,
kernel/sched/sched.h +12 -0
@@ -794,10 +794,22 @@ enum scx_rq_flags { SCX_RQ_IN_BALANCE = 1 << 17, }; +/* per-rq rescue execution state, see scx_rescue_timerfn() */ +struct scx_rq_rescue { +struct scx_dispatch_q dsq;/* stranded tasks awaiting rescue */ +s64 budget;/* execution token bucket, ns */ +u64 clock;/* last budget accrual timestamp */ +struct task_struct *curr;/* task being rescued, one at a time */ +s64 slice;/* curr's admitted slice */ +u64 exec_snap;/* sum_exec_runtime at admission */ +struct timer_list timer;/* paces admission and escalation */ +}; +struct scx_rq {struct scx_dispatch_q local_dsq;#ifdef CONFIG_EXT_SUB_SCHED struct scx_dispatch_q reject_dsq;/* staging for cap-rejected tasks */ +struct scx_rq_rescue rescue;#endif struct list_head runnable_list;/* runnable tasks on this rq */ struct list_head ddsp_deferred_locals;/* deferred ddsps from enq */
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 9/12 sched_ext: Eject the top rescue consumer on overload
In reply to Tejun HeoWhen rescue demand on a cpu persistently exceeds the configured bandwidth, tasks age on that cpu's rescue DSQ until the stall watchdog fires. The watchdog blames the waiting task's owner, but the misbehaving party is whoever floods the queue, not whoever happens to time out.
Track each sched's recent rescue consumption per cpu as a decaying average. Once the oldest waiter on a cpu's rescue DSQ has been queued past a threshold derived from the rescue knobs (4s at the defaults), the rescue timer ejects the sub with the highest recent consumption on that cpu with SCX_EXIT_ERROR_RESCUE. With no recent consumer there is no victim and nothing is ejected - the generic stall watchdog eventually blames the waiter's owner instead. Ejections on a cpu are spaced one threshold apart so the freed bandwidth can drain the backlog before another sub is judged.
The overload check only wins the race against the stall watchdog when the watchdog timeout clears the threshold, and a single in-budget wait must not cross the trigger on its own. Warn on a scheduler whose timeout doesn't fit and on knobs whose funding period exceeds half the threshold.
v2: - Track kill_at in jiffies_64 - on 32-bit, the time_before() grace check
wraps 2^31 ticks after the last ejection and suppresses ejections. (sashiko AI)
- Track rescue_avg_at in jiffies_64 likewise - the unsigned long decay delta truncates mod 2^32 on 32-bit and can revive a weeks-old usage average in the victim pick.
--- include/linux/sched/ext.h | 3 + kernel/sched/ext/ext.c | 2 + kernel/sched/ext/internal.h | 9 +++ kernel/sched/ext/sub.c | 137 +++++++++++++++++++++++++++++++++--- kernel/sched/ext/types.h | 3 + kernel/sched/sched.h | 1 + 6 files changed, 144 insertions(+), 11 deletions(-)
include/linux/sched/ext.h +3 -0
@@ -215,6 +215,9 @@ struct sched_ext_entity {#ifdef CONFIG_SCHED_CORE u64 core_sched_at;/* see scx_prio_less() */ #endif +#ifdef CONFIG_EXT_SUB_SCHED +unsigned long rescue_at;/* queued on a rescue DSQ at, jiffies */ +#endif /* * Unique non-zero task ID assigned at fork. Persists across exec and
kernel/sched/ext/ext.c +2 -0
@@ -6140,6 +6140,8 @@ static const char *scx_exit_reason(enum scx_exit_kind kind)return "runnable task stall" ;case SCX_EXIT_ERROR_REENQ :return "reenqueue limit" ; +case SCX_EXIT_ERROR_RESCUE : +return "rescue bandwidth overload" ;default :return "<UNKNOWN>" ; }
kernel/sched/ext/internal.h +9 -0
@@ -57,6 +57,7 @@ enum scx_exit_kind {SCX_EXIT_ERROR_BPF ,/* ERROR but triggered through scx_bpf_error() */ SCX_EXIT_ERROR_STALL ,/* watchdog detected stalled runnable tasks */ SCX_EXIT_ERROR_REENQ ,/* task hit reenqueue limit without running */ + SCX_EXIT_ERROR_RESCUE, /* ejected for overloading rescue execution */ };/*
@@ -1340,6 +1341,14 @@ struct scx_sched_pcpu {bool idle_renotify;/* effective caps as of the last sub_ecaps_updated() delivery */ u64 reported_ecaps; + +/* +* Decaying rescue runtime consumed on this cpu, see +* scx_rescue_decay_avg(). Overload on this cpu ejects the sub with the +* largest value. Accessed only under this cpu's rq lock. +*/ +u64 rescue_avg; +u64 rescue_avg_at;/* last decay, jiffies_64 */ #endif /*
kernel/sched/ext/sub.c +126 -11
215 lines not on this page · raw message
kernel/sched/ext/types.h +3 -0
@@ -27,6 +27,9 @@ enum scx_consts {SCX_RESCUE_MIN_QUANTUM_US =1000 ,SCX_RESCUE_MAX_QUANTUM_US =100000 ,SCX_RESCUE_MIN_SLICE_US =1000 ,/* floor of the divided slice */ +SCX_RESCUE_OVERLOAD_MULT =16 ,/* overload threshold in funding periods */ +SCX_RESCUE_MIN_OVERLOAD_MS =1000 , +SCX_RESCUE_MAX_OVERLOAD_MS =15000 ,/* per-CPU chunk size for p->scx.tid allocation, see scx_alloc_tid() */ SCX_TID_CHUNK = 1024,
kernel/sched/sched.h +1 -0
@@ -803,6 +803,7 @@ struct scx_rq_rescue {s64 slice;/* curr's admitted slice */ u64 exec_snap;/* sum_exec_runtime at admission */ struct timer_list timer;/* paces admission and escalation */ +u64 kill_at;/* last ejection, init before any */ };struct scx_rq {
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 10/12 sched_ext: Sync tools autogen enum headers
In reply to Tejun HeoMechanical regeneration for the enumerators added by the slice write rework and the rescue machinery. BPF schedulers reference SCX_ENQ_RESCUE, the rest are enum_defs presence markers.
--- tools/sched_ext/include/scx/enum_defs.autogen.h | 16 ++++++++++++++++ tools/sched_ext/include/scx/enums.autogen.bpf.h | 3 +++ tools/sched_ext/include/scx/enums.autogen.h | 1 + 3 files changed, 20 insertions(+)
tools/sched_ext/include/scx/enum_defs.autogen.h +16 -0
@@ -26,6 +26,16 @@#define HAVE_SCX_DSP_DFL_MAX_BATCH #define HAVE_SCX_DSP_MAX_LOOPS #define HAVE_SCX_WATCHDOG_MAX_TIMEOUT +#define HAVE_SCX_RESCUE_DFL_BW_PPT +#define HAVE_SCX_RESCUE_MAX_BW_PPT +#define HAVE_SCX_RESCUE_DISABLE +#define HAVE_SCX_RESCUE_DFL_QUANTUM_US +#define HAVE_SCX_RESCUE_MIN_QUANTUM_US +#define HAVE_SCX_RESCUE_MAX_QUANTUM_US +#define HAVE_SCX_RESCUE_MIN_SLICE_US +#define HAVE_SCX_RESCUE_OVERLOAD_MULT +#define HAVE_SCX_RESCUE_MIN_OVERLOAD_MS +#define HAVE_SCX_RESCUE_MAX_OVERLOAD_MS #define HAVE_SCX_TID_CHUNK #define HAVE_SCX_EXIT_BT_LEN #define HAVE_SCX_EXIT_MSG_LEN
@@ -53,6 +63,7 @@#define HAVE_SCX_DSQ_LOCAL #define HAVE_SCX_DSQ_BYPASS #define HAVE_SCX_DSQ_REJECT +#define HAVE_SCX_DSQ_RESCUE #define HAVE_SCX_DSQ_LOCAL_ON #define HAVE_SCX_DSQ_LOCAL_CPU_MASK #define HAVE_SCX_DSQ_ITER_REV
@@ -71,6 +82,7 @@#define HAVE_SCX_ENQ_CPU_SELECTED #define HAVE_SCX_ENQ_PREEMPT #define HAVE_SCX_ENQ_IMMED +#define HAVE_SCX_ENQ_RESCUE #define HAVE_SCX_ENQ_REENQ #define HAVE_SCX_ENQ_LAST #define HAVE___SCX_ENQ_INTERNAL_MASK
@@ -79,6 +91,8 @@#define HAVE_SCX_ENQ_NESTED #define HAVE_SCX_ENQ_GDSQ_FALLBACK #define HAVE_SCX_ENQ_IGNORE_CAPS +#define HAVE_SCX_ENQ_APPLY_SLICE +#define HAVE_SCX_ENQ_SLICE_DFL #define HAVE_SCX_TASK_DSQ_ON_PRIQ #define HAVE_SCX_TASK_QUEUED #define HAVE_SCX_TASK_IN_CUSTODY
@@ -86,6 +100,7 @@#define HAVE_SCX_TASK_DEQD_FOR_SLEEP #define HAVE_SCX_TASK_SUB_INIT #define HAVE_SCX_TASK_IMMED +#define HAVE_SCX_TASK_PROTECTED #define HAVE_SCX_TASK_STATE_SHIFT #define HAVE_SCX_TASK_STATE_BITS #define HAVE_SCX_TASK_STATE_MASK
@@ -120,6 +135,7 @@#define HAVE_SCX_EXIT_ERROR_BPF #define HAVE_SCX_EXIT_ERROR_STALL #define HAVE_SCX_EXIT_ERROR_REENQ +#define HAVE_SCX_EXIT_ERROR_RESCUE #define HAVE_SCX_KF_ALLOW_UNLOCKED #define HAVE_SCX_KF_ALLOW_INIT_CIDS #define HAVE_SCX_KF_ALLOW_CPU_RELEASE
tools/sched_ext/include/scx/enums.autogen.bpf.h +3 -0
@@ -124,6 +124,9 @@ const volatile u64 __SCX_ENQ_PREEMPT __weak;const volatile u64 __SCX_ENQ_IMMED__weak ;#define SCX_ENQ_IMMED __SCX_ENQ_IMMED +const volatile u64 __SCX_ENQ_RESCUE__weak ; +#define SCX_ENQ_RESCUE __SCX_ENQ_RESCUE +const volatile u64 __SCX_ENQ_REENQ__weak ;#define SCX_ENQ_REENQ __SCX_ENQ_REENQ
tools/sched_ext/include/scx/enums.autogen.h +1 -0
@@ -45,6 +45,7 @@SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_HEAD ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_PREEMPT ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_IMMED ); \ +SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_RESCUE ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_REENQ ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_LAST ); \SCX_ENUM_SET (skel, scx_enq_flags,SCX_ENQ_CLEAR_OPSS ); \
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 11/12 sched_ext: scx_qmap - Idle-check pinned tasks before direct dispatch
In reply to Tejun Heopick_direct_dispatch_cid() sent a pinned task straight to its only cpu without an idle check. An insert onto a time-shared cid is forced IMMED, which may not wait on a busy cpu - the kernel bounces the task back to ops.enqueue(), the shortcut re-inserts it, and the loop runs into the reenqueue repeat limit.
Run pinned tasks through the same idle test as everyone else and queue them when the cpu is busy. always_enq_immed forces IMMED on every local insert and already skipped the shortcut for the same reason. The check was the last reader of the always_enq_immed rodata mirror, so drop the variable. The -I option still sets SCX_OPS_ALWAYS_ENQ_IMMED.
v2: Drop the now-unread always_enq_immed rodata mirror. (sashiko AI)
--- tools/sched_ext/scx_qmap.bpf.c | 4 ---- tools/sched_ext/scx_qmap.c | 1 - 2 files changed, 5 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +0 -4
@@ -66,7 +66,6 @@ const volatile bool print_msgs;const volatile u64 sub_cgroup_id;const volatile s32 disallow_tgid;const volatile bool suppress_dump; -const volatile bool always_enq_immed;const volatile u32 immed_stress_nth;const volatile u32 max_tasks;
@@ -230,9 +229,6 @@ static s32 pick_direct_dispatch_cid(struct task_struct *p, s32 prev_cid,s32 cid;u32 i; -if (!always_enq_immed && p->nr_cpus_allowed ==1 ) -return prev_cid; -if (cmask_test (prev_cid, &qa.self_cids.mask) &&cmask_test_and_clear (prev_cid, &qa.idle_cids.mask))return prev_cid;
tools/sched_ext/scx_qmap.c +0 -1
@@ -315,7 +315,6 @@ int main(int argc, char **argv) skel->struct_ops.qmap_ops->flags |=SCX_OPS_SWITCH_PARTIAL ;break ;case 'I' : - skel->rodata->always_enq_immed = true; skel->struct_ops.qmap_ops->flags |=SCX_OPS_ALWAYS_ENQ_IMMED ;break ;case 'F' :
Signature · 1 line
2.55.0
Tejun Heo
reviewerPATCH 12/12 sched_ext: scx_qmap - Add rescue support
In reply to Tejun HeoA sched holds only the cids its parent granted and nothing guarantees that they cover its tasks' affinities. A task that can run on none of them has nowhere to go and qmap stalls out: it force-inserts the task onto its first allowed cid, but the kernel bounces the insert back and the task parks in SHARED_DSQ, which is drained only on self cids it can't run on.
Set SCX_ENQ_RESCUE on these inserts so the kernel diverts such tasks to its rescue path instead of bouncing them. The force-insert covers scheds with and without children and fires on re-enqueues, and the SHARED_DSQ scan on every dispatch rescues tasks stranded there - the enqueue-time check misses a task whose cids were lost while it was already queued. The wrong-cid fault injection carries the flag too and doubles as a deterministic rescue-traffic generator.
-B and -q set the root-only rescue bandwidth and quantum ops knobs. -B 0 maps to SCX_RESCUE_DISABLE and turns rescue off kernel-side. Rescue inserts are counted and reported in the hier stats line.
--- tools/sched_ext/scx_qmap.bpf.c | 85 +++++++++++++++++++++++----------- tools/sched_ext/scx_qmap.c | 23 +++++++-- tools/sched_ext/scx_qmap.h | 1 + 3 files changed, 79 insertions(+), 30 deletions(-)
tools/sched_ext/scx_qmap.bpf.c +58 -27
@@ -448,31 +448,33 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags) taskc->core_sched_seq = qa.core_sched_tail_seqs[idx]++;/* -* A node with children delegates most cids. A task of ours that can run -* on none of our self cids (e.g. a per-NUMA kthread pinned to delegated -* cids) would starve in SHARED/FIFO since we never pull those on a -* delegated cid. Force it onto its first allowed cid's local DSQ with -* needs_immed(): if we hold access there it runs, else the kernel -* rejects and bounces it back via REENQ_CAP. Best-effort -* anti-starvation nudge. +* A task of ours that can run on none of our self cids - the parent +* didn't grant them or we delegated them to children - would starve in +* SHARED/FIFO since we only pull from those on self cids. +* +* Force it onto its first allowed cid's local DSQ. If we hold that cid +* it runs. Otherwise the insert carries SCX_ENQ_RESCUE and the kernel +* diverts the task to its rescue path. */ -if (qa.nr_sub_scheds && !(enq_flags &SCX_ENQ_REENQ ) && - !cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) { +if (!cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) {s32 c =cmask_next_set_wrap (&taskc->cpus_allowed,0 );if (c >=0 && c <scx_bpf_nr_cids ()) { taskc->force_local = false; +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 );scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | c, slice_ns, - enq_flags |needs_immed (c)); + enq_flags |needs_immed (c) |SCX_ENQ_RESCUE );return ; } }/* * Fault injection: deliberately dispatch one of our own tasks to a cid -* we don't hold. The kernel cap check must reject it and re-enqueue -* with SCX_TASK_REENQ_CAP, so nr_inject_attempts tracks nr_reenq_cap -* and proves delivery-time enforcement. Throttled. +* we don't hold. The inserts carry SCX_ENQ_RESCUE and divert to the +* kernel rescue path, a deterministic rescue-traffic generator. Under +* -B 0 the kernel cap check rejects and re-enqueues them instead, so +* nr_inject_attempts tracks nr_reenq_cap 1:1 and proves delivery-time +* enforcement. Throttled. */ if (qa.inject_mode ==QMAP_INJ_WRONG_CID && p->nr_cpus_allowed >1 && !(enq_flags &SCX_ENQ_REENQ )) {
@@ -483,8 +485,9 @@ void BPF_STRUCT_OPS(qmap_enqueue, struct task_struct *p, u64 enq_flags)if (bad >=0 &&cmask_test (bad, &taskc->cpus_allowed)) {__sync_fetch_and_add (&qa.nr_inject_attempts,1 ); -scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | bad, - slice_ns, enq_flags); +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 ); +scx_bpf_dsq_insert (p,SCX_DSQ_LOCAL_ON | bad, slice_ns, + enq_flags |SCX_ENQ_RESCUE );return ; } }
@@ -587,29 +590,47 @@ static void update_core_sched_head_seq(struct task_struct *p) }/* +* One pass over SHARED_DSQ: rescue stranded tasks and boost highpri ones. A +* task whose cids were lost while it was queued in the fifos would strand on +* SHARED_DSQ, which is consumed only on self cids it can't run on - move it to +* the kernel rescue path. One whose cids were lost after the highpri cull is +* likewise rescued out of HIGHPRI_DSQ below. +* * To demonstrate the use of scx_bpf_dsq_move(), implement silly selective -* priority boosting mechanism by scanning SHARED_DSQ looking for highpri tasks, -* moving them to HIGHPRI_DSQ and then consuming them first. This makes minor -* difference only when dsp_batch is larger than 1. +* priority boosting mechanism by moving highpri tasks to HIGHPRI_DSQ and then +* consuming them first. This makes minor difference only when dsp_batch is +* larger than 1. * * scx_bpf_dsq_move[_vtime]() are allowed both from ops.dispatch() and * non-rq-lock holding BPF programs. As demonstration, this function is called * from qmap_dispatch() and monitor_timerfn(). */ -static bool dispatch_highpri (bool from_timer) +static bool scan_shared_dsq (bool from_timer) {struct task_struct *p;s32 this_cid =scx_bpf_this_cid ();u32 nr_cids =scx_bpf_nr_cids (); -/* scan SHARED_DSQ and move highpri tasks to HIGHPRI_DSQ */ +/* rescue strands and move highpri tasks to HIGHPRI_DSQ */ bpf_for_each (scx_dsq, p,SHARED_DSQ ,0 ) {static u64 highpri_seq;task_ctx_t *taskc; +s32 c;if (!(taskc =lookup_task_ctx (p)))return false; +/* stranded? rescue - it can't be dispatched here either way */ +if (!cmask_intersects (&taskc->cpus_allowed, &qa.self_cids.mask)) { + c =cmask_next_set_wrap (&taskc->cpus_allowed,0 ); +if (c >=0 && c <scx_bpf_nr_cids ()) { +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 ); +scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | c, +needs_immed (c) |SCX_ENQ_RESCUE ); + } +continue ; + } +if (taskc->highpri) {/* exercise the set_*() and vtime interface too */ scx_bpf_dsq_move_set_slice (BPF_FOR_EACH_ITER , slice_ns *2 );
@@ -639,8 +660,17 @@ static bool dispatch_highpri(bool from_timer) cid =cmask_next_and_set_wrap (&taskc->cpus_allowed, &qa.self_cids.mask, this_cid +1 ); -if (cid >= nr_cids) +if (cid >= nr_cids) { +/* stranded after the cull - rescue it from here */ +s32 c =cmask_next_set_wrap (&taskc->cpus_allowed,0 ); + +if (c >=0 && c < nr_cids) { +__sync_fetch_and_add (&qa.nr_rescue_dsp,1 ); +scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | c, +needs_immed (c) |SCX_ENQ_RESCUE ); + }continue ; + }if (scx_bpf_dsq_move (BPF_FOR_EACH_ITER , p,SCX_DSQ_LOCAL_ON | cid,SCX_ENQ_PREEMPT |needs_immed (cid))) {
@@ -669,10 +699,9 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev) struct cpu_ctx __arena *cpuc;task_ctx_t *taskc;u32 batch = dsp_batch ?:1 ; -s32 owner; -s32 i; +s32 owner, i; -if (dispatch_highpri (false)) +if (scan_shared_dsq (false))return ;/*
@@ -784,11 +813,10 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev)*/ if (!cmask_test (cid, &taskc->cpus_allowed))scx_bpf_kick_cid (scx_bpf_task_cid (p),0 ); - batch--; cpuc->dsp_cnt--;if (!batch || !scx_bpf_dispatch_nr_slots ()) { -if (dispatch_highpri (false)) +if (scan_shared_dsq (false))return ;scx_bpf_dsq_move_to_local (SHARED_DSQ ,needs_immed (cid));return ;
@@ -800,6 +828,9 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cid, struct task_struct *prev) cpuc->dsp_cnt =0 ; } +if (scan_shared_dsq (false)) +return ; +/* * No other tasks. @prev will keep running. Update its core_sched_seq as * if the task were enqueued and dispatched immediately.
@@ -1184,7 +1215,7 @@ static void dump_shared_dsq(void)static int monitor_timerfn (void *map,int *key,struct bpf_timer *timer) {bpf_rcu_read_lock (); -dispatch_highpri (true); +scan_shared_dsq (true);bpf_rcu_read_unlock ();monitor_cpuperf ();
tools/sched_ext/scx_qmap.c +20 -3
@@ -72,6 +72,8 @@ const char help_fmt[] =" -J MODE Fault injection (wrong-cid: dispatch to a cid not held,\n" " init-fail/cgrp-init-fail: fail init_task/cpuctl_init for\n" " \"qmfail*\" comms/cgroups)\n" +" -B PPT Rescue bandwidth in parts per thousand, 0 disables (root only, default 20)\n" +" -q US Rescue batch quantum in microseconds (root only, default 5000)\n" " -v Print libbpf debug messages\n" " -h Display this help and exit\n" ;
@@ -106,6 +108,7 @@ struct hier_prev {u64 nr_reenq_cap;u64 nr_reenq_immed;u64 nr_inject_attempts; +u64 nr_rescue_dsp; };/* current wall-clock time as "HH:MM:SS" for the startup and interval headers */
@@ -187,14 +190,16 @@ static void print_hier(struct qmap_arena *qa, struct hier_prev *prev, u64 own_cg }format_cid_ranges (qa,CID_SHARED , ranges,sizeof (ranges)); -printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu\n" , +printf ("hier : nsub=%llu excl=%u shared=%s rr=%s reenq cap/immed +%llu/+%llu inj=+%llu rescue=+%llu\n" , (unsigned long long )qa->nr_sub_scheds, qa->part.nr_excl, ranges, rr, (unsigned long long )(qa->nr_reenq_cap - prev->nr_reenq_cap), (unsigned long long )(qa->nr_reenq_immed - prev->nr_reenq_immed), - (unsigned long long )(qa->nr_inject_attempts - prev->nr_inject_attempts)); + (unsigned long long )(qa->nr_inject_attempts - prev->nr_inject_attempts), + (unsigned long long )(qa->nr_rescue_dsp - prev->nr_rescue_dsp)); prev->nr_reenq_cap = qa->nr_reenq_cap; prev->nr_reenq_immed = qa->nr_reenq_immed; prev->nr_inject_attempts = qa->nr_inject_attempts; + prev->nr_rescue_dsp = qa->nr_rescue_dsp;printf ("hier : %-4s %10s %4s %6s %8s %s\n" ,"" ,"cgroup" ,"w" ,"alloc" ,"disp/s" ,"cids" );
@@ -256,7 +261,8 @@ int main(int argc, char **argv) skel->rodata->slice_ns =__COMPAT_ENUM_OR_ZERO ("scx_public_consts" ,"SCX_SLICE_DFL" ); skel->rodata->max_tasks =16384 ; -while ((opt =getopt (argc, argv,"s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:vh" )) !=-1 ) { +while ((opt =getopt (argc, argv, +"s:e:t:T:l:b:N:PMHc:d:D:SpIF:C:i:R:J:B:q:vh" )) !=-1 ) {switch (opt) {case 's' : skel->rodata->slice_ns =strtoull (optarg,NULL ,0 ) *1000 ;
@@ -399,6 +405,17 @@ int main(int argc, char **argv)else inject_mode =strtoul (optarg,NULL ,0 );break ; +case 'B' : { +u32 ppt =strtoul (optarg,NULL ,0 ); + +if (!ppt) + ppt =__COMPAT_ENUM_OR_ZERO ("scx_consts" ,"SCX_RESCUE_DISABLE" ); + skel->struct_ops.qmap_ops->rescue_bandwidth_ppt = ppt; +break ; + } +case 'q' : + skel->struct_ops.qmap_ops->rescue_quantum_us =strtoul (optarg,NULL ,0 ); +break ;case 'v' : verbose = true;break ;
tools/sched_ext/scx_qmap.h +1 -0
@@ -175,6 +175,7 @@ struct qmap_arena {u64 nr_reenq_cap;/* SCX_TASK_REENQ_CAP bounces */ u64 nr_reenq_immed;/* SCX_TASK_REENQ_IMMED bounces */ u64 nr_inject_attempts;/* fault-injection: dispatches to an unheld cid */ +u64 nr_rescue_dsp;/* SCX_ENQ_RESCUE dispatch attempts */ u32 inject_mode;/* fault-injection mode (QMAP_INJ_*) */ };
Signature · 1 line
2.55.0
Andrea Righi
reviewerRe: [PATCH 08/12] sched_ext: Add bandwidth-limited rescue execution for stranded tasks
In reply to Tejun HeoHi Tejun,
Quoted 34 lines · Tejun Heo
A local DSQ insert lacking the needed caps is diverted to the reject DSQ and bounced back through
ops.enqueue()so the scheduler can re-decide. That recovery assumes the scheduler has somewhere legal to send the task. When it doesn't, e.g. when the task's affinity is restricted to cids delegated away, the task starves until the stall watchdog ejects the scheduler. An exiting task is worse - it skipsops.enqueue()and the rejection becomes a self-requeuing cycle that burns the CPU until the watchdog fires.Add
SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the insert would be rejected for missing caps, the kernel takes over and runs the task on the target CPU without consulting the owning scheduler. The kernel sets the flag itself when enqueueing an exiting task.Rescue is a last-resort forward-progress backstop with a persistent disadvantage, not a way around cap enforcement. A per-CPU token bucket accrues
rescue_bandwidth_ppt(default 2%) of CPU time and rescues run one at a time in arrival order. Each is granted a slice of therescue_quantum_us(default 5ms) quantum divided across the waiters, waits at the tail of the local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival once the slice is served.The schedulers keep their normal control over an admitted rescuee and may preempt or reslice it. Service is measured on CPU time actually received, so neither shortens the rescue. Prolonged denial escalates - the remaining slice turns into protected execution (
SCX_TASK_PROTECTED) and the rescuee preempts the current task. Escalation is paced by the same bucket, and delivered service converges on the configured bandwidth no matter how aggressively the schedulers dispatch.Both knobs are root-only and
SCX_RESCUE_DISABLEturns rescue off, makingSCX_ENQ_RESCUEinserts reject as usual.Signed-off-by Tejun Heo <[email protected]>---
...
Quoted 29 lines
kernel/sched/ext/internal.h +21
-0@@ -924,6 +924,37 @@ struct sched_ext_ops {*/ u32 cid_shard_size; +/** +* @rescue_bandwidth_ppt: Rescue execution bandwidth in parts per thousand +* +* The fraction of each CPU's time that may be consumed running tasks +* from its rescue DSQ. A higher bandwidth admits and escalates rescues +* faster, see @rescue_quantum_us. +* +* Only the root scheduler's value is used. 0 means the default of 20 +* (2%). May not exceed 250 (25%). %SCX_RESCUE_DISABLE disables rescue - +* %SCX_ENQ_RESCUE inserts are then rejected like any other insert +* lacking the caps. +*/ +u32 rescue_bandwidth_ppt; + +/** +* @rescue_quantum_us: Rescue execution quantum in microseconds +* +* How much CPU time each rescue gets. Rescues run one at a time per CPU +* and admissions are paced to keep rescue execution within +* @rescue_bandwidth_ppt - with the defaults, one 5ms rescue every +* 250ms. A crowded queue round-robins on the quantum divided across the
kernel/sched/ext/internal.h struct sched_ext_ops { · line 957
Quoted 10 lines
kernel/sched/ext/internal.h +10
-0+* waiters, floored at 1ms. A stuck rescue eventually escalates to +* forced execution. A larger quantum interrupts the CPU less often but +* for longer and spaces rescues further apart. +* +* Only the root scheduler's value is used. 0 means the default (5000). +* Non-zero values must be within [1000, 100000]. Values too short for +* the kernel to meter are lifted silently. +*/ +u32 rescue_quantum_us; +
Not a blocker, but should we add compatibility handling for these two optional ops fields in tools/sched_ext/include/scx/compat.h?
Otherwise the later scx_qmap patch would break the qmap build with older kernels.
Thanks,
-Andrea
Tejun Heo
reviewerPATCH v2 8/12 sched_ext: Add bandwidth-limited rescue execution for stranded tasks
In reply to Tejun HeoA local DSQ insert lacking the needed caps is diverted to the reject DSQ and bounced back through ops.enqueue() so the scheduler can re-decide. That recovery assumes the scheduler has somewhere legal to send the task. When it doesn't, e.g. when the task's affinity is restricted to cids delegated away, the task starves until the stall watchdog ejects the scheduler. An exiting task is worse - it skips ops.enqueue() and the rejection becomes a self-requeuing cycle that burns the CPU until the watchdog fires.
Add SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the insert would be rejected for missing caps, the kernel takes over and runs the task on the target CPU without consulting the owning scheduler. The kernel sets the flag itself when enqueueing an exiting task.
Rescue is a last-resort forward-progress backstop with a persistent disadvantage, not a way around cap enforcement. A per-CPU token bucket accrues rescue_bandwidth_ppt (default 2%) of CPU time and rescues run one at a time in arrival order. Each is granted a slice of the rescue_quantum_us (default 5ms) quantum divided across the waiters, waits at the tail of the local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival once the slice is served.
The schedulers keep their normal control over an admitted rescuee and may preempt or reslice it. Service is measured on CPU time actually received, so neither shortens the rescue. Prolonged denial escalates - the remaining slice turns into protected execution (SCX_TASK_PROTECTED) and the rescuee preempts the current task. Escalation is paced by the same bucket, and delivered service converges on the configured bandwidth no matter how aggressively the schedulers dispatch.
Both knobs are root-only and SCX_RESCUE_DISABLE turns rescue off, making SCX_ENQ_RESCUE inserts reject as usual.
v2: - Add SCX_OPS_OPEN() fix-ups for the new ops fields so cpu-form
schedulers setting them still load on older kernels. (Andrea)
--- include/linux/sched/ext.h | 1 kernel/sched/ext/ext.c | 112 ++++++++-- kernel/sched/ext/internal.h | 56 +++++ kernel/sched/ext/sub.c | 376 +++++++++++++++++++++++++++++++++-- kernel/sched/ext/sub.h | 26 ++ kernel/sched/ext/types.h | 9 kernel/sched/sched.h | 12 + tools/sched_ext/include/scx/compat.h | 11 + 8 files changed, 570 insertions(+), 33 deletions(-)
include/linux/sched/ext.h +1 -0
@@ -59,6 +59,7 @@ enum scx_dsq_id_flags {SCX_DSQ_LOCAL =SCX_DSQ_FLAG_BUILTIN |2 ,SCX_DSQ_BYPASS =SCX_DSQ_FLAG_BUILTIN |3 ,SCX_DSQ_REJECT =SCX_DSQ_FLAG_BUILTIN |4 ,/* internal - see find_dsq_for_dispatch() */ +SCX_DSQ_RESCUE =SCX_DSQ_FLAG_BUILTIN |5 ,/* internal - see find_dsq_for_dispatch() */ SCX_DSQ_LOCAL_ON =SCX_DSQ_FLAG_BUILTIN |SCX_DSQ_FLAG_LOCAL_ON , SCX_DSQ_LOCAL_CPU_MASK = 0xffffffffLLU, };
kernel/sched/ext/ext.c +95 -17
250 lines not on this page · raw message
kernel/sched/ext/internal.h +55 -1
104 lines not on this page · raw message
kernel/sched/ext/sub.c +361 -15
411 lines not on this page · raw message
kernel/sched/ext/sub.h +26 -0
@@ -38,6 +38,13 @@ struct scx_dispatch_q *scx_resolve_local struct task_struct *p, u64 *enq_flags);bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p);void scx_reenq_reject (struct rq *rq); +void scx_rescue_charge (struct rq *rq,s64 delta_exec); +void scx_rescue_end (struct rq *rq); +bool scx_rescue_keep (struct rq *rq,struct task_struct *p); +void scx_rescue_flush (struct rq *rq); +void scx_rescue_dump (struct seq_buf *s,struct rq *rq); +void scx_rescue_set_knobs (struct scx_sched *sch); +void scx_rescue_init (struct rq *rq);/* * cgrp->scx_sched is written by root/sub enable/disable under all of
@@ -87,6 +94,13 @@ static inline void scx_discard_stale_ecastatic inline struct scx_dispatch_q *scx_resolve_local_dsq (struct scx_sched *sch,struct rq *rq,struct task_struct *p,u64 *enq_flags) {return &rq->scx.local_dsq; }static inline bool scx_task_reenq_on_cap_revoke (struct rq *rq,struct task_struct *p) {return false; }static inline void scx_reenq_reject (struct rq *rq) {} +static inline void scx_rescue_charge (struct rq *rq,s64 delta_exec) {} +static inline void scx_rescue_end (struct rq *rq) {} +static inline bool scx_rescue_keep (struct rq *rq,struct task_struct *p) {return false; } +static inline void scx_rescue_flush (struct rq *rq) {} +static inline void scx_rescue_dump (struct seq_buf *s,struct rq *rq) {} +static inline void scx_rescue_set_knobs (struct scx_sched *sch) {} +static inline void scx_rescue_init (struct rq *rq) {}static inline void scx_dec_has_subs (struct scx_sched *sch) {}#endif /* CONFIG_EXT_SUB_SCHED */
@@ -195,11 +209,23 @@ static inline bool scx_task_can_stay_on_ return likely(!scx_missing_caps(scx_task_sched(p), cpu_of(rq), SCX_CAP_BASE)); } +/* the task admitted for rescue on @rq, NULL if none */ +static inline struct task_struct *scx_rescuee (struct rq *rq) +{ +lockdep_assert_rq_held (rq); + +if (!scx_has_subs ()) +return NULL ; + +return rq->scx.rescue.curr; +} +#else /* CONFIG_EXT_SUB_SCHED */ static inline u64 scx_missing_caps (struct scx_sched *sch,s32 cpu,u64 needed) {return 0 ; }static inline u64 scx_caps_for_preempt (struct scx_sched *sch,struct rq *rq,u64 enq_flags) {return 0 ; }static inline bool scx_task_can_stay_on_cpu (struct rq *rq,struct task_struct *p) {return true; } +static inline struct task_struct *scx_rescuee (struct rq *rq) {return NULL ; }#endif /* CONFIG_EXT_SUB_SCHED */
kernel/sched/ext/types.h +9 -0
@@ -19,6 +19,15 @@ enum scx_consts {SCX_DSP_MAX_LOOPS =32 ,SCX_WATCHDOG_MAX_TIMEOUT =30 *HZ , +/* rescue knob defaults and limits, see scx_rescue_timerfn() */ +SCX_RESCUE_DFL_BW_PPT =20 ,/* parts per thousand, 2% */ +SCX_RESCUE_MAX_BW_PPT =250 ,/* 25% */ +SCX_RESCUE_DISABLE =U32_MAX ,/* disables rescue */ +SCX_RESCUE_DFL_QUANTUM_US =5000 , +SCX_RESCUE_MIN_QUANTUM_US =1000 , +SCX_RESCUE_MAX_QUANTUM_US =100000 , +SCX_RESCUE_MIN_SLICE_US =1000 ,/* floor of the divided slice */ +/* per-CPU chunk size for p->scx.tid allocation, see scx_alloc_tid() */ SCX_TID_CHUNK = 1024,
kernel/sched/sched.h +12 -0
@@ -794,10 +794,22 @@ enum scx_rq_flags { SCX_RQ_IN_BALANCE = 1 << 17, }; +/* per-rq rescue execution state, see scx_rescue_timerfn() */ +struct scx_rq_rescue { +struct scx_dispatch_q dsq;/* stranded tasks awaiting rescue */ +s64 budget;/* execution token bucket, ns */ +u64 clock;/* last budget accrual timestamp */ +struct task_struct *curr;/* task being rescued, one at a time */ +s64 slice;/* curr's admitted slice */ +u64 exec_snap;/* sum_exec_runtime at admission */ +struct timer_list timer;/* paces admission and escalation */ +}; +struct scx_rq {struct scx_dispatch_q local_dsq;#ifdef CONFIG_EXT_SUB_SCHED struct scx_dispatch_q reject_dsq;/* staging for cap-rejected tasks */ +struct scx_rq_rescue rescue;#endif struct list_head runnable_list;/* runnable tasks on this rq */ struct list_head ddsp_deferred_locals;/* deferred ddsps from enq */
tools/sched_ext/include/scx/compat.h +11 -0
@@ -175,6 +175,7 @@ static inline long scx_hotplug_seq(void)* - v6.17: ops.cgroup_set_bandwidth() * - v6.19: ops.cgroup_set_idle() * - v7.1: ops.sub_attach(), ops.sub_detach(), ops.sub_cgroup_id +* - v7.3: ops.rescue_bandwidth_ppt, ops.rescue_quantum_us */ #define __SCX_OPS_OPEN (__ops_name, __scx_name, __ops_struct) ({ \ struct __scx_name *__oskel; \
@@ -218,6 +219,16 @@ static inline long scx_hotplug_seq(void)fprintf (stderr,"WARNING: kernel doesn't support ops.sub_cgroup_id\n" ); \ __skel->struct_ops.__ops_name->sub_cgroup_id =0 ; \ } \ +if (__skel->struct_ops.__ops_name->rescue_bandwidth_ppt >0 && \ + !__COMPAT_struct_has_field ("sched_ext_ops" ,"rescue_bandwidth_ppt" )) { \ +fprintf (stderr,"WARNING: kernel doesn't support ops.rescue_bandwidth_ppt\n" ); \ + __skel->struct_ops.__ops_name->rescue_bandwidth_ppt =0 ; \ + } \ +if (__skel->struct_ops.__ops_name->rescue_quantum_us >0 && \ + !__COMPAT_struct_has_field ("sched_ext_ops" ,"rescue_quantum_us" )) { \ +fprintf (stderr,"WARNING: kernel doesn't support ops.rescue_quantum_us\n" ); \ + __skel->struct_ops.__ops_name->rescue_quantum_us =0 ; \ + } \ __skel; \ })
Hi Tejun,
Quoted 28 lines · Tejun Heo
Hello,
v2: Changes from the sashiko AI review:
- Declare
scx_sched_alloutside theCONFIG_EXT_SUB_SCHEDblock, the definition is unconditional (0002).- Track the overload grace and usage-decay timestamps in
jiffies_64so that 32-bit arithmetic on them can't wrap (0009).- Drop the now-unread
always_enq_immedrodata mirror (0011).v1: https://lore.kernel.org/r/[email protected]
A sub-scheduler holds only the cids its parent granted and nothing guarantees that they cover its tasks' affinities. A task that can run on none of them currently has no good ending: the scheduler's insert is cap-rejected and reenqueued until the repeat limit ejects the scheduler, or the task stalls into the watchdog.
This series adds kernel-side rescue execution. A scheduler marks an insert that may be cap-rejected with
SCX_ENQ_RESCUEand the kernel, instead of bouncing it, runs the task on the target cpu at a small configured bandwidth. A rescue starts out non-disruptive. Schedulers that hold caps on the cid largely stay in control of the cpu - a preemption cap, for example, still lets its holder preempt the rescuee. When a rescue goes unserved for too long, it escalates into protected execution.
Quoted 3 lines
When a cpu's rescue queue stays oversaturated past the overload threshold, the kernel ejects the sub-scheduler with the highest recent rescue consumption on that cpu instead of misblaming the waiter's owner.
With the updated patch 08/12 everything looks good to me. For the whole series:
Thanks,
-Andrea
Tejun Heo
reviewerQuoted 4 lines · Tejun Heo
0001-sched_ext-Rename-scx_local_or_reject_dsq-to-scx_reso.patch 0002-sched_ext-Make-several-ext.c-helpers-available-outsi.patch 0003-sched_ext-Factor-out-__scx_bpf_now.patch 0004-sched_ext-Reject-internal-enq_flags-in-the-dsq-move-.patch
Quoted 8 lines
0005-sched_ext-Make-SCX_ENQ_IGNORE_CAPS-waive-the-preempt.patch 0006-sched_ext-Synchronize-slice-and-dsq_vtime-writes.patch 0007-sched_ext-Add-SCX_TASK_PROTECTED.patch 0008-sched_ext-Add-bandwidth-limited-rescue-execution-for.patch 0009-sched_ext-Eject-the-top-rescue-consumer-on-overload.patch 0010-sched_ext-Sync-tools-autogen-enum-headers.patch 0011-sched_ext-scx_qmap-Idle-check-pinned-tasks-before-di.patch 0012-sched_ext-scx_qmap-Add-rescue-support.patch
Applied 1-12 (8 in its v2 form) to sched_ext/for-7.3 with Andrea's Reviewed-by added.
Thanks.
--
tejun