Add a deadline server for sched_ext tasks
Subsystems (4)
v6 cover Add a deadline server for sched_ext tasks
sched_ext tasks currently are starved by RT hoggers especially since RT throttling was replaced by deadline servers to boost only CFS tasks. Several users in the community have reported issues with RT stalling sched_ext tasks. Add a sched_ext deadline server as well so that sched_ext tasks are also boosted and do not suffer starvation.
2 kselftest are provided to verify the starvation fixes and bandwidth allocation is looking correct.
v5->v6: mostly no changes versus v5.
- Added Acks to few patches.
- Fixes to few nits suggested by Tejun.
v4->v5:
- Added a kselftest (total_bw) to sched_ext to verify bandwidth values from debugfs.
- Address comment from Andrea about redundant rq clock invalidation.
v3->v4:
- Fixed issues with hotplugged CPUs having their DL server bandwidth altered due to loading SCX.
- Fixed other issues.
- Rebased on Linus master.
- All sched_ext kselftests reliably pass now, also verified that the total_bw in debugfs (CONFIG_SCHED_DEBUG) is conserved with these patches.
v2->v3:
- Removed code duplication in debugfs. Made ext interface separate.
- Fixed issue where rq_lock_irqsave was not used in the relinquish patch.
- Fixed running bw accounting issue in dl_server_remove_params.
Link to v1: https://lore.kernel.org/all/[email protected]/
Link to v2: https://lore.kernel.org/all/[email protected]/
Link to v3: https://lore.kernel.org/all/[email protected]/
Link to v4: https://lore.kernel.org/all/[email protected]/
Link to v5: https://lore.kernel.org/all/[email protected]/
Andrea Righi (2):
sched/deadline: Add support to remove DLserver's bandwidth
contribution
selftests/sched_ext: Add test for sched_ext dl_server
Joel Fernandes (12):
sched/debug: Fix updating of ppos on server write ops sched/debug: Stop and start server based on if it was active sched/deadline: Clear the defer params
sched/deadline: Prevent setting server as started if params couldn't
be applied
sched/deadline: Return EBUSY if dl_bw_cpus is zero
sched: Add support to pick functions to take rf
sched: Add a server arg to dl_server_update_idle_time() sched/ext: Add a DL server for sched_ext tasks sched/debug: Add support to change sched_ext server params sched/ext: Relinquish DL server reservations when not needed sched/deadline: Fix DL server crash in inactive_timer callback selftests/sched_ext: Add test for DL server total_bw consistency
include/linux/sched.h | 2 +- kernel/sched/core.c | 19 +- kernel/sched/deadline.c | 86 ++++-- kernel/sched/debug.c | 165 ++++++++-- kernel/sched/ext.c | 120 +++++++- kernel/sched/fair.c | 15 +- kernel/sched/idle.c | 4 +- kernel/sched/rt.c | 2 +- kernel/sched/sched.h | 13 +- kernel/sched/stop_task.c | 2 +- tools/testing/selftests/sched_ext/Makefile | 2 + .../selftests/sched_ext/rt_stall.bpf.c | 23 ++ tools/testing/selftests/sched_ext/rt_stall.c | 213 +++++++++++++ tools/testing/selftests/sched_ext/total_bw.c | 286 ++++++++++++++++++ 14 files changed, 869 insertions(+), 83 deletions(-) create mode 100644 tools/testing/selftests/sched_ext/rt_stall.bpf.c create mode 100644 tools/testing/selftests/sched_ext/rt_stall.c create mode 100644 tools/testing/selftests/sched_ext/total_bw.c
Signature · 1 line
2.34.1
PATCH v6 1/14 sched/debug: Fix updating of ppos on server write ops
Updating "ppos" on error conditions does not make much sense. The pattern is to return the error code directly without modifying the position, or modify the position on success and return the number of bytes written.
Since on success, the return value of apply is 0, there is no point in modifying ppos either. Fix it by removing all this and just returning error code or number of bytes written on success.
--- kernel/sched/debug.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/sched/debug.c +4 -3
@@ -350,8 +350,8 @@ static ssize_t sched_fair_server_write(struct file *filp, const char __user *ubulong cpu = (long ) ((struct seq_file *) filp->private_data)->private;struct rq *rq =cpu_rq (cpu);u64 runtime, period; +int retval =0 ;size_t err; -int retval;u64 value; err =kstrtoull_from_user (ubuf, cnt,10 , &value);
@@ -387,8 +387,6 @@ static ssize_t sched_fair_server_write(struct file *filp, const char __user *ubu } retval =dl_server_apply_params (&rq->fair_server, runtime, period,0 ); -if (retval) - cnt = retval;if (!runtime) printk_deferred("Fair server disabled in CPU %d, system may crash due to starvation.\n",
@@ -396,6 +394,9 @@ static ssize_t sched_fair_server_write(struct file *filp, const char __user *ubuif (rq->cfs.h_nr_queued)dl_server_start (&rq->fair_server); + +if (retval <0 ) +return retval; } *ppos += cnt;
Signature · 1 line
2.34.1
PATCH v6 2/14 sched/debug: Stop and start server based on if it was active
In reply to Joel FernandesCurrently the DL server interface for applying parameters checks CFS-internals to identify if the server is active. This is error-prone and makes it difficult when adding new servers in the future.
Fix it, by using dl_server_active() which is also used by the DL server code to determine if the DL server was started.
--- kernel/sched/debug.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/sched/debug.c +5 -2
@@ -359,6 +359,8 @@ static ssize_t sched_fair_server_write(struct file *filp, const char __user *ubureturn err; scoped_guard (rq_lock_irqsave, rq) { +bool is_active; + runtime = rq->fair_server.dl_runtime; period = rq->fair_server.dl_period;
@@ -381,7 +383,8 @@ static ssize_t sched_fair_server_write(struct file *filp, const char __user *ubureturn -EINVAL ; } -if (rq->cfs.h_nr_queued) { + is_active =dl_server_active (&rq->fair_server); +if (is_active) {update_rq_clock (rq);dl_server_stop (&rq->fair_server); }
@@ -392,7 +395,7 @@ static ssize_t sched_fair_server_write(struct file *filp, const char __user *ubuprintk_deferred ("Fair server disabled in CPU %d, system may crash due to starvation.\n" ,cpu_of (rq)); -if (rq->cfs.h_nr_queued) +if (is_active)dl_server_start (&rq->fair_server);if (retval <0 )
Signature · 1 line
2.34.1
PATCH v6 3/14 sched/deadline: Clear the defer params
In reply to Joel FernandesThe defer params were not cleared in __dl_clear_params. Clear them.
Without this is some of my test cases are flaking and the DL timer is not starting correctly AFAICS.
--- kernel/sched/deadline.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/sched/deadline.c +3 -0
@@ -3431,6 +3431,9 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se) dl_se->dl_non_contending =0 ; dl_se->dl_overrun =0 ; dl_se->dl_server =0 ; + dl_se->dl_defer =0 ; + dl_se->dl_defer_running =0 ; + dl_se->dl_defer_armed =0 ;#ifdef CONFIG_RT_MUTEXES dl_se->pi_se = dl_se;
Signature · 1 line
2.34.1
PATCH v6 4/14 sched/deadline: Prevent setting server as started if params couldn't be applied
In reply to Joel FernandesThe following call trace fails to set dl_server_apply_params() as dl_bw_cpus() is 0 during CPU onlining.
[ 11.878356] ------------[ cut here ]------------ [ 11.882592] <TASK> [ 11.882685] enqueue_task_scx+0x190/0x280 [ 11.882802] ttwu_do_activate+0xaa/0x2a0 [ 11.882925] try_to_wake_up+0x371/0x600 [ 11.883047] cpuhp_bringup_ap+0xd6/0x170 [ 11.883172] cpuhp_invoke_callback+0x142/0x540 [ 11.883327] _cpu_up+0x15b/0x270 [ 11.883450] cpu_up+0x52/0xb0 [ 11.883576] cpu_subsys_online+0x32/0x120 [ 11.883704] online_store+0x98/0x130 [ 11.883824] kernfs_fop_write_iter+0xeb/0x170 [ 11.883972] vfs_write+0x2c7/0x430 [ 11.884091] ksys_write+0x70/0xe0 [ 11.884209] do_syscall_64+0xd6/0x250 [ 11.884327] ? clear_bhb_loop+0x40/0x90 [ 11.884443] entry_SYSCALL_64_after_hwframe+0x77/0x7f
It is too early to start the server. Simply defer the starting of the server to the next enqueue if dl_server_apply_params() returns an error. In any case, we should not pretend like the server started and it does seem to mess up with the sched_ext CPU hotplug test.
With this, the sched_ext hotplug test reliably passes.
--- kernel/sched/deadline.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/sched/deadline.c +3 -3
@@ -1656,8 +1656,8 @@ void dl_server_start(struct sched_dl_entity *dl_se)u64 runtime =50 *NSEC_PER_MSEC ;u64 period =1000 *NSEC_PER_MSEC ; -dl_server_apply_params (dl_se, runtime, period,1 ); - +if (WARN_ON_ONCE (dl_server_apply_params (dl_se, runtime, period,1 ))) +return ; dl_se->dl_server =1 ; dl_se->dl_defer =1 ;setup_new_dl_entity (dl_se);
@@ -1674,7 +1674,7 @@ void dl_server_start(struct sched_dl_entity *dl_se)void dl_server_stop (struct sched_dl_entity *dl_se) { -if (!dl_se->dl_runtime) +if (!dl_se->dl_runtime || !dl_se->dl_server_active)return ;dequeue_dl_entity (dl_se,DEQUEUE_SLEEP );
Signature · 1 line
2.34.1
PATCH v6 5/14 sched/deadline: Return EBUSY if dl_bw_cpus is zero
In reply to Joel FernandesHotplugged CPUs coming online do an enqueue but are not a part of any root domain containing cpu_active() CPUs. So in this case, don't mess with accounting and we can retry later. Without this patch, we see crashes with sched_ext selftest's hotplug test due to divide by zero.
--- kernel/sched/deadline.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/sched/deadline.c +6 -1
@@ -1725,7 +1725,12 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio cpus =dl_bw_cpus (cpu); cap =dl_bw_capacity (cpu); -if (__dl_overflow (dl_b, cap, old_bw, new_bw)) +/* +* Hotplugged CPUs coming online do an enqueue but are not a part of any +* root domain containing cpu_active() CPUs. So in this case, don't mess +* with accounting and we can retry later. +*/ +if (!cpus ||__dl_overflow (dl_b, cap, old_bw, new_bw))return -EBUSY ;if (init) {
Signature · 1 line
2.34.1
Andrea Righi
reviewerRe: [PATCH v6 05/14] sched/deadline: Return EBUSY if dl_bw_cpus is zero
Hi Joel,
Quoted 16 lines · Joel Fernandes
Hotplugged CPUs coming online do an enqueue but are not a part of any root domain containing
cpu_active()CPUs. So in this case, don't mess with accounting and we can retry later. Without this patch, we see crashes withsched_extselftest's hotplug test due to divide by zero.Signed-off-by Joel Fernandes <[email protected]>--- kernel/sched/deadline.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)kernel/sched/deadline.c +0
-0@@ -1725,7 +1725,12 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio cpus =dl_bw_cpus (cpu);
kernel/sched/deadline.c int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio · line 1734
Quoted 10 lines
kernel/sched/deadline.c +6
-1cap =dl_bw_capacity (cpu); -if (__dl_overflow (dl_b, cap, old_bw, new_bw)) +/* +* Hotplugged CPUs coming online do an enqueue but are not a part of any +* root domain containing cpu_active() CPUs. So in this case, don't mess +* with accounting and we can retry later. +*/ +if (!cpus ||__dl_overflow (dl_b, cap, old_bw, new_bw))return -EBUSY ;
I can trigger the following with the hotplug sched_ext selftest:
[ 61.731069] ------------[ cut here ]------------ [ 61.731241] WARNING: CPU: 4 PID: 2191 at kernel/sched/deadline.c:1591 dl_server_start+0x9c/0xb0 ... [ 61.731552] Sched_ext: hotplug_cbs (enabled+all), task: runnable_at=-1ms [ 61.731555] RIP: 0010:dl_server_start+0x9c/0xb0 ... [ 61.732216] Call Trace: [ 61.732239] <TASK> [ 61.732262] enqueue_task_scx+0x2bb/0x350 [ 61.732298] enqueue_task+0x2e/0xd0 [ 61.732333] ttwu_do_activate+0xa4/0x2d0 [ 61.732368] try_to_wake_up+0x2a2/0x8e0 [ 61.732406] cpuhp_bringup_ap+0x72/0x250 [ 61.732441] ? __pfx_cpuhp_bringup_ap+0x10/0x10 [ 61.732478] cpuhp_invoke_callback+0x1b0/0x650 [ 61.732523] __cpuhp_invoke_callback_range+0x7e/0xf0 [ 61.732566] _cpu_up+0xea/0x1e0 [ 61.732601] cpu_up+0xc3/0xd0 [ 61.732635] cpu_subsys_online+0x4b/0xd0 [ 61.732678] device_online+0x49/0x80 [ 61.732714] online_store+0x9a/0xd0 [ 61.732744] ? sysfs_kf_write+0x2b/0x70 [ 61.732780] kernfs_fop_write_iter+0x137/0x200 [ 61.732822] vfs_write+0x264/0x5e0 [ 61.732867] ksys_write+0x79/0xf0 [ 61.732901] do_syscall_64+0xbb/0x370 [ 61.732939] entry_SYSCALL_64_after_hwframe+0x77/0x7f
I think we can simply ignore the !cpus case, as it's safe to skip hotplugged CPUs that perform an enqueue when coming online, without triggering a warning, and we can always retry applying the server params later.
Maybe return -EAGAIN when !cpus and handle that gracefully in dl_server_start()?
Something along these lines (on top of this patch). What do you think?
Thanks,
-Andrea
kernel/sched/deadline.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +10 -2
@@ -1587,9 +1587,14 @@ void dl_server_start(struct sched_dl_entity *dl_se)if (!dl_server (dl_se)) {u64 runtime =50 *NSEC_PER_MSEC ;u64 period =1000 *NSEC_PER_MSEC ; +int err; -if (WARN_ON_ONCE (dl_server_apply_params (dl_se, runtime, period,1 ))) + err =dl_server_apply_params (dl_se, runtime, period,1 ); +if (err == -EAGAIN )return ; + +WARN_ON_ONCE (err); + dl_se->dl_server =1 ; dl_se->dl_defer =1 ;setup_new_dl_entity (dl_se);
@@ -1662,7 +1667,10 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio* root domain containing cpu_active() CPUs. So in this case, don't mess * with accounting and we can retry later. */ -if (!cpus ||__dl_overflow (dl_b, cap, old_bw, new_bw)) +if (!cpus) +return -EAGAIN ; + +if (__dl_overflow (dl_b, cap, old_bw, new_bw))return -EBUSY ;if (init) {
PATCH v6 6/14 sched: Add support to pick functions to take rf
In reply to Joel FernandesSome pick functions like the internal pick_next_task_fair() already take rf but some others dont. We need this for scx's server pick function. Prepare for this by having pick functions accept it.
--- include/linux/sched.h | 2 +- kernel/sched/core.c | 16 ++++++++-------- kernel/sched/deadline.c | 8 ++++---- kernel/sched/ext.c | 2 +- kernel/sched/fair.c | 13 ++++++++----- kernel/sched/idle.c | 2 +- kernel/sched/rt.c | 2 +- kernel/sched/sched.h | 7 ++++--- kernel/sched/stop_task.c | 2 +- 9 files changed, 29 insertions(+), 25 deletions(-)
include/linux/sched.h +1 -1
@@ -634,7 +634,7 @@ struct sched_rt_entity { } __randomize_layout;typedef bool (*dl_server_has_tasks_f )(struct sched_dl_entity *); -typedef struct task_struct *(*dl_server_pick_f )(struct sched_dl_entity *); +typedef struct task_struct *(*dl_server_pick_f )(struct sched_dl_entity *,void *);struct sched_dl_entity {struct rb_node rb_node;
kernel/sched/core.c +8 -8
@@ -6049,7 +6049,7 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)/* Assume the next prioritized class is idle_sched_class */ if (!p) { - p =pick_task_idle (rq); + p =pick_task_idle (rq, rf);put_prev_set_next_task (rq, prev, p); }
@@ -6061,11 +6061,11 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)for_each_active_class (class) {if (class->pick_next_task) { - p = class->pick_next_task (rq, prev); + p = class->pick_next_task (rq, prev, rf);if (p)return p; }else { - p = class->pick_task (rq); + p = class->pick_task (rq, rf);if (p) {put_prev_set_next_task (rq, prev, p);return p;
@@ -6095,7 +6095,7 @@ static inline bool cookie_match(struct task_struct *a, struct task_struct *b)return a->core_cookie == b->core_cookie; } -static inline struct task_struct *pick_task (struct rq *rq) +static inline struct task_struct *pick_task (struct rq *rq,struct rq_flags *rf) {const struct sched_class *class;struct task_struct *p;
@@ -6103,7 +6103,7 @@ static inline struct task_struct *pick_task(struct rq *rq) rq->dl_server =NULL ;for_each_active_class (class) { - p = class->pick_task (rq); + p = class->pick_task (rq, rf);if (p)return p; }
@@ -6203,7 +6203,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* and there are no cookied tasks running on siblings. */ if (!need_sync) { - next =pick_task (rq); + next =pick_task (rq, rf);if (!next->core_cookie) { rq->core_pick =NULL ; rq->core_dl_server =NULL ;
@@ -6234,7 +6234,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)if (i != cpu && (rq_i != rq->core || !core_clock_updated))update_rq_clock (rq_i); - rq_i->core_pick = p =pick_task (rq_i); + rq_i->core_pick = p =pick_task (rq_i, rf); rq_i->core_dl_server = rq_i->dl_server;if (!max ||prio_less (max, p, fi_before))
@@ -6256,7 +6256,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)if (cookie) p =sched_core_find (rq_i, cookie);if (!p) - p = idle_sched_class.pick_task (rq_i); + p = idle_sched_class.pick_task (rq_i, rf); } rq_i->core_pick = p;
kernel/sched/deadline.c +4 -4
@@ -2424,7 +2424,7 @@ static struct sched_dl_entity *pick_next_dl_entity(struct dl_rq *dl_rq)* __pick_next_task_dl - Helper to pick the next -deadline task to run. * @rq: The runqueue to pick the next task from. */ -static struct task_struct *__pick_task_dl (struct rq *rq) +static struct task_struct *__pick_task_dl (struct rq *rq,struct rq_flags *rf) {struct sched_dl_entity *dl_se;struct dl_rq *dl_rq = &rq->dl;
@@ -2438,7 +2438,7 @@ static struct task_struct *__pick_task_dl(struct rq *rq)WARN_ON_ONCE (!dl_se);if (dl_server (dl_se)) { - p = dl_se->server_pick_task (dl_se); + p = dl_se->server_pick_task (dl_se, rf);if (!p) {if (dl_server_active (dl_se)) { dl_se->dl_yielded =1 ;
@@ -2454,9 +2454,9 @@ static struct task_struct *__pick_task_dl(struct rq *rq)return p; } -static struct task_struct *pick_task_dl (struct rq *rq) +static struct task_struct *pick_task_dl (struct rq *rq,struct rq_flags *rf) { -return __pick_task_dl (rq); +return __pick_task_dl (rq, rf); }static void put_prev_task_dl (struct rq *rq,struct task_struct *p,struct task_struct *next)
kernel/sched/ext.c +1 -1
@@ -3397,7 +3397,7 @@ static struct task_struct *first_local_task(struct rq *rq) struct task_struct, scx.dsq_list.node); } -static struct task_struct *pick_task_scx (struct rq *rq) +static struct task_struct *pick_task_scx (struct rq *rq,struct rq_flags *rf) {struct task_struct *prev = rq->curr;struct task_struct *p;
kernel/sched/fair.c +8 -5
@@ -8852,7 +8852,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, intresched_curr_lazy (rq); } -static struct task_struct *pick_task_fair (struct rq *rq) +static struct task_struct *pick_task_fair (struct rq *rq,struct rq_flags *rf) {struct sched_entity *se;struct cfs_rq *cfs_rq;
@@ -8890,7 +8890,7 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rfint new_tasks; again: - p =pick_task_fair (rq); + p =pick_task_fair (rq, rf);if (!p) goto idle; se = &p->se;
@@ -8969,7 +8969,8 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rfreturn NULL ; } -static struct task_struct *__pick_next_task_fair (struct rq *rq,struct task_struct *prev) +static struct task_struct *__pick_next_task_fair (struct rq *rq,struct task_struct *prev, +struct rq_flags *rf) {return pick_next_task_fair (rq, prev,NULL ); }
@@ -8979,9 +8980,11 @@ static bool fair_server_has_tasks(struct sched_dl_entity *dl_se)return !!dl_se->rq->cfs.nr_queued; } -static struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se) +static struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se,void *flags) { -return pick_task_fair (dl_se->rq); +struct rq_flags *rf = flags; + +return pick_task_fair (dl_se->rq, rf); }void fair_server_init (struct rq *rq)
kernel/sched/idle.c +1 -1
@@ -463,7 +463,7 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir next->se.exec_start =rq_clock_task (rq); } -struct task_struct *pick_task_idle (struct rq *rq) +struct task_struct *pick_task_idle (struct rq *rq,struct rq_flags *rf) {scx_update_idle (rq, true, false);return rq->idle;
kernel/sched/rt.c +1 -1
@@ -1744,7 +1744,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq)return rt_task_of (rt_se); } -static struct task_struct *pick_task_rt (struct rq *rq) +static struct task_struct *pick_task_rt (struct rq *rq,struct rq_flags *rf) {struct task_struct *p;
kernel/sched/sched.h +4 -3
@@ -2401,7 +2401,7 @@ struct sched_class {void (*wakeup_preempt)(struct rq *rq,struct task_struct *p,int flags);int (*balance)(struct rq *rq,struct task_struct *prev,struct rq_flags *rf); -struct task_struct *(*pick_task)(struct rq *rq); +struct task_struct *(*pick_task)(struct rq *rq,struct rq_flags *rf);/* * Optional! When implemented pick_next_task() should be equivalent to: *
@@ -2411,7 +2411,8 @@ struct sched_class {* set_next_task_first(next); * } */ -struct task_struct *(*pick_next_task)(struct rq *rq,struct task_struct *prev); +struct task_struct *(*pick_next_task)(struct rq *rq,struct task_struct *prev, +struct rq_flags *rf);void (*put_prev_task)(struct rq *rq,struct task_struct *p,struct task_struct *next);void (*set_next_task)(struct rq *rq,struct task_struct *p,bool first);
@@ -2574,7 +2575,7 @@ static inline bool sched_fair_runnable(struct rq *rq) }extern struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf); -extern struct task_struct *pick_task_idle (struct rq *rq); +extern struct task_struct *pick_task_idle (struct rq *rq,struct rq_flags *rf);#define SCA_CHECK 0x01#define SCA_MIGRATE_DISABLE 0x02
kernel/sched/stop_task.c +1 -1
@@ -33,7 +33,7 @@ static void set_next_task_stop(struct rq *rq, struct task_struct *stop, bool fir stop->se.exec_start =rq_clock_task (rq); } -static struct task_struct *pick_task_stop (struct rq *rq) +static struct task_struct *pick_task_stop (struct rq *rq,struct rq_flags *rf) {if (!sched_stop_runnable (rq))return NULL ;
Signature · 1 line
2.34.1
PATCH v6 7/14 sched: Add a server arg to dl_server_update_idle_time()
In reply to Joel FernandesSince we are adding more servers, make dl_server_update_idle_time() accept a server argument than a specific server.
--- kernel/sched/deadline.c | 17 +++++++++-------- kernel/sched/fair.c | 2 +- kernel/sched/idle.c | 2 +- kernel/sched/sched.h | 3 ++- 4 files changed, 13 insertions(+), 11 deletions(-)
kernel/sched/deadline.c +9 -8
@@ -1609,28 +1609,29 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64* as time available for the fair server, avoiding a penalty for the * rt scheduler that did not consumed that time. */ -void dl_server_update_idle_time (struct rq *rq,struct task_struct *p) +void dl_server_update_idle_time (struct rq *rq,struct task_struct *p, +struct sched_dl_entity *rq_dl_server) {s64 delta_exec, scaled_delta_exec; -if (!rq->fair_server.dl_defer) +if (!rq_dl_server->dl_defer)return ;/* no need to discount more */ -if (rq->fair_server.runtime <0 ) +if (rq_dl_server->runtime <0 )return ; delta_exec =rq_clock_task (rq) - p->se.exec_start;if (delta_exec <0 )return ; - scaled_delta_exec =dl_scaled_delta_exec (rq, &rq->fair_server, delta_exec); + scaled_delta_exec =dl_scaled_delta_exec (rq, rq_dl_server, delta_exec); - rq->fair_server.runtime -= scaled_delta_exec; + rq_dl_server->runtime -= scaled_delta_exec; -if (rq->fair_server.runtime <0 ) { - rq->fair_server.dl_defer_running =0 ; - rq->fair_server.runtime =0 ; +if (rq_dl_server->runtime <0 ) { + rq_dl_server->dl_defer_running =0 ; + rq_dl_server->runtime =0 ; } p->se.exec_start =rq_clock_task (rq);
kernel/sched/fair.c +1 -1
@@ -7026,7 +7026,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (!rq_h_nr_queued && rq->cfs.h_nr_queued) {/* Account for idle runtime */ if (!rq->nr_running) -dl_server_update_idle_time (rq, rq->curr); +dl_server_update_idle_time (rq, rq->curr, &rq->fair_server);dl_server_start (&rq->fair_server); }
kernel/sched/idle.c +1 -1
@@ -451,7 +451,7 @@ static void wakeup_preempt_idle(struct rq *rq, struct task_struct *p, int flags)static void put_prev_task_idle (struct rq *rq,struct task_struct *prev,struct task_struct *next) { -dl_server_update_idle_time (rq, prev); +dl_server_update_idle_time (rq, prev, &rq->fair_server);scx_update_idle (rq, false, true); }
kernel/sched/sched.h +2 -1
@@ -386,7 +386,8 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq, dl_server_pick_f pick_task);extern void dl_server_update_idle_time (struct rq *rq, -struct task_struct *p); +struct task_struct *p, +struct sched_dl_entity *rq_dl_server);extern void fair_server_init (struct rq *rq);extern void __dl_server_attach_root (struct sched_dl_entity *dl_se,struct rq *rq); extern int dl_server_apply_params(struct sched_dl_entity *dl_se,
Signature · 1 line
2.34.1
PATCH v6 8/14 sched/ext: Add a DL server for sched_ext tasks
In reply to Joel Fernandessched_ext currently suffers starvation due to RT. The same workload when converted to EXT can get zero runtime if RT is 100% running, causing EXT processes to stall. Fix it by adding a DL server for EXT.
A kselftest is also provided later to verify:
./runner -t rt_stall
===== START =====
TEST: rt_stall
DESCRIPTION: Verify that RT tasks cannot stall SCHED_EXT tasks OUTPUT:
TAP version 13
1..1
ok 1 PASS: CFS task got more than 4.00% of runtime
--- kernel/sched/core.c | 3 ++ kernel/sched/deadline.c | 2 +- kernel/sched/ext.c | 62 +++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 2 ++ 4 files changed, 66 insertions(+), 3 deletions(-)
kernel/sched/core.c +3 -0
@@ -8650,6 +8650,9 @@ void __init sched_init(void)hrtick_rq_init (rq);atomic_set (&rq->nr_iowait,0 );fair_server_init (rq); +#ifdef CONFIG_SCHED_CLASS_EXT +ext_server_init (rq); +#endif #ifdef CONFIG_SCHED_CORE rq->core = rq;
kernel/sched/deadline.c +1 -1
@@ -1570,7 +1570,7 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64* The fair server (sole dl_server) does not account for real-time * workload because it is running fair work. */ -if (dl_se == &rq->fair_server) +if (dl_se == &rq->fair_server || dl_se == &rq->ext_server)return ;#ifdef CONFIG_RT_GROUP_SCHED
kernel/sched/ext.c +60 -2
98 lines not on this page · raw message
kernel/sched/sched.h +2 -0
@@ -389,6 +389,7 @@ extern void dl_server_update_idle_time(struct rq *rq,struct task_struct *p, struct sched_dl_entity *rq_dl_server);extern void fair_server_init (struct rq *rq); +extern void ext_server_init (struct rq *rq);extern void __dl_server_attach_root (struct sched_dl_entity *dl_se,struct rq *rq);extern int dl_server_apply_params (struct sched_dl_entity *dl_se,u64 runtime,u64 period,bool init);
@@ -1137,6 +1138,7 @@ struct rq {#endif struct sched_dl_entity fair_server; +struct sched_dl_entity ext_server;#ifdef CONFIG_FAIR_GROUP_SCHED /* list of leaf cfs_rq on this CPU: */
Signature · 1 line
2.34.1
PATCH v6 9/14 sched/debug: Add support to change sched_ext server params
In reply to Joel FernandesWhen a sched_ext server is loaded, tasks in CFS are converted to run in sched_ext class. Add support to modify the ext server parameters similar to how the fair server parameters are modified.
Re-use common code between ext and fair servers as needed.
--- kernel/sched/debug.c | 153 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 129 insertions(+), 24 deletions(-)
kernel/sched/debug.c +129 -24
@@ -341,14 +341,16 @@ enum dl_param { DL_PERIOD, }; -static unsigned long fair_server_period_max = (1UL <<22 ) *NSEC_PER_USEC ;/* ~4 seconds */ -static unsigned long fair_server_period_min = (100 ) *NSEC_PER_USEC ;/* 100 us */ +static unsigned long dl_server_period_max = (1UL <<22 ) *NSEC_PER_USEC ;/* ~4 seconds */ +static unsigned long dl_server_period_min = (100 ) *NSEC_PER_USEC ;/* 100 us */ -static ssize_t sched_fair_server_write (struct file *filp,const char __user *ubuf, -size_t cnt,loff_t *ppos,enum dl_param param) +static ssize_t sched_server_write_common (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos,enum dl_param param, +void *server) {long cpu = (long ) ((struct seq_file *) filp->private_data)->private;struct rq *rq =cpu_rq (cpu); +struct sched_dl_entity *dl_se = (struct sched_dl_entity *)server;u64 runtime, period;int retval =0 ;size_t err;
@@ -361,8 +363,8 @@ static ssize_t sched_fair_server_write(struct file *filp, const char __user *ubu scoped_guard (rq_lock_irqsave, rq) {bool is_active; - runtime = rq->fair_server.dl_runtime; - period = rq->fair_server.dl_period; + runtime = dl_se->dl_runtime; + period = dl_se->dl_period;switch (param) {case DL_RUNTIME :
@@ -378,25 +380,25 @@ static ssize_t sched_fair_server_write(struct file *filp, const char __user *ubu }if (runtime > period || - period > fair_server_period_max || - period < fair_server_period_min) { + period > dl_server_period_max || + period < dl_server_period_min) {return -EINVAL ; } - is_active =dl_server_active (&rq->fair_server); + is_active =dl_server_active (dl_se);if (is_active) {update_rq_clock (rq); -dl_server_stop (&rq->fair_server); +dl_server_stop (dl_se); } - retval =dl_server_apply_params (&rq->fair_server, runtime, period,0 ); + retval =dl_server_apply_params (dl_se, runtime, period,0 );if (!runtime) -printk_deferred ("Fair server disabled in CPU %d, system may crash due to starvation.\n" , -cpu_of (rq)); +printk_deferred ("%s server disabled on CPU %d, system may crash due to starvation.\n" , + server == &rq->fair_server ?"Fair" :"Ext" ,cpu_of (rq));if (is_active) -dl_server_start (&rq->fair_server); +dl_server_start (dl_se);if (retval <0 )return retval;
@@ -406,36 +408,46 @@ static ssize_t sched_fair_server_write(struct file *filp, const char __user *ubureturn cnt; } -static size_t sched_fair_server_show (struct seq_file *m,void *v,enum dl_param param) + + +static size_t sched_server_show_common (struct seq_file *m,void *v,enum dl_param param, +void *server) { -unsigned long cpu = (unsigned long ) m->private; -struct rq *rq =cpu_rq (cpu); +struct sched_dl_entity *dl_se = (struct sched_dl_entity *)server;u64 value;switch (param) {case DL_RUNTIME : - value = rq->fair_server.dl_runtime; + value = dl_se->dl_runtime;break ;case DL_PERIOD : - value = rq->fair_server.dl_period; + value = dl_se->dl_period;break ; }seq_printf (m,"%llu\n" , value);return 0 ; - } + +static ssize_t sched_fair_server_runtime_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) { -return sched_fair_server_write (filp, ubuf, cnt, ppos,DL_RUNTIME ); +long cpu = (long ) ((struct seq_file *) filp->private_data)->private; +struct rq *rq =cpu_rq (cpu); + +return sched_server_write_common (filp, ubuf, cnt, ppos,DL_RUNTIME , + &rq->fair_server); }static int sched_fair_server_runtime_show (struct seq_file *m,void *v) { -return sched_fair_server_show (m, v,DL_RUNTIME ); +unsigned long cpu = (unsigned long ) m->private; +struct rq *rq =cpu_rq (cpu); + +return sched_server_show_common (m, v,DL_RUNTIME , &rq->fair_server); }static int sched_fair_server_runtime_open (struct inode *inode,struct file *filp)
@@ -451,16 +463,55 @@ static const struct file_operations fair_server_runtime_fops = { .release = single_release, }; +static ssize_t +sched_ext_server_runtime_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) +{ +long cpu = (long ) ((struct seq_file *) filp->private_data)->private; +struct rq *rq =cpu_rq (cpu); + +return sched_server_write_common (filp, ubuf, cnt, ppos,DL_RUNTIME , + &rq->ext_server); +} + +static int sched_ext_server_runtime_show (struct seq_file *m,void *v) +{ +unsigned long cpu = (unsigned long ) m->private; +struct rq *rq =cpu_rq (cpu); + +return sched_server_show_common (m, v,DL_RUNTIME , &rq->ext_server); +} + +static int sched_ext_server_runtime_open (struct inode *inode,struct file *filp) +{ +return single_open (filp, sched_ext_server_runtime_show, inode->i_private); +} + +static const struct file_operations ext_server_runtime_fops = { + .open = sched_ext_server_runtime_open, + .write = sched_ext_server_runtime_write, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +static ssize_t sched_fair_server_period_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) { -return sched_fair_server_write (filp, ubuf, cnt, ppos,DL_PERIOD ); +long cpu = (long ) ((struct seq_file *) filp->private_data)->private; +struct rq *rq =cpu_rq (cpu); + +return sched_server_write_common (filp, ubuf, cnt, ppos,DL_PERIOD , + &rq->fair_server); }static int sched_fair_server_period_show (struct seq_file *m,void *v) { -return sched_fair_server_show (m, v,DL_PERIOD ); +unsigned long cpu = (unsigned long ) m->private; +struct rq *rq =cpu_rq (cpu); + +return sched_server_show_common (m, v,DL_PERIOD , &rq->fair_server); }static int sched_fair_server_period_open (struct inode *inode,struct file *filp)
@@ -476,6 +527,38 @@ static const struct file_operations fair_server_period_fops = { .release = single_release, }; +static ssize_t +sched_ext_server_period_write (struct file *filp,const char __user *ubuf, +size_t cnt,loff_t *ppos) +{ +long cpu = (long ) ((struct seq_file *) filp->private_data)->private; +struct rq *rq =cpu_rq (cpu); + +return sched_server_write_common (filp, ubuf, cnt, ppos,DL_PERIOD , + &rq->ext_server); +} + +static int sched_ext_server_period_show (struct seq_file *m,void *v) +{ +unsigned long cpu = (unsigned long ) m->private; +struct rq *rq =cpu_rq (cpu); + +return sched_server_show_common (m, v,DL_PERIOD , &rq->ext_server); +} + +static int sched_ext_server_period_open (struct inode *inode,struct file *filp) +{ +return single_open (filp, sched_ext_server_period_show, inode->i_private); +} + +static const struct file_operations ext_server_period_fops = { + .open = sched_ext_server_period_open, + .write = sched_ext_server_period_write, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +static struct dentry *debugfs_sched;static void debugfs_fair_server_init (void )
@@ -499,6 +582,27 @@ static void debugfs_fair_server_init(void) } } +static void debugfs_ext_server_init (void ) +{ +struct dentry *d_ext; +unsigned long cpu; + + d_ext =debugfs_create_dir ("ext_server" , debugfs_sched); +if (!d_ext) +return ; + +for_each_possible_cpu (cpu) { +struct dentry *d_cpu; +char buf[32 ]; + +snprintf (buf,sizeof (buf),"cpu%lu" , cpu); + d_cpu =debugfs_create_dir (buf, d_ext); + +debugfs_create_file ("runtime" ,0644 , d_cpu, (void *) cpu, &ext_server_runtime_fops); +debugfs_create_file ("period" ,0644 , d_cpu, (void *) cpu, &ext_server_period_fops); + } +} +static __init int sched_init_debug (void ) {struct dentry __maybe_unused *numa;
@@ -539,6 +643,7 @@ static __init int sched_init_debug(void)debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init (); +debugfs_ext_server_init ();return 0 ; }
Signature · 1 line
2.34.1
PATCH v6 10/14 sched/deadline: Add support to remove DLserver's bandwidth contribution
In reply to Joel FernandesFrom: Andrea Righi <[email protected]>
During switching from sched_ext to FAIR tasks and vice-versa, we need support for removing the bandwidth contribution of either DL server. Add support for the same.
--- kernel/sched/deadline.c | 31 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 2 files changed, 32 insertions(+)
kernel/sched/deadline.c +31 -0
@@ -1744,6 +1744,12 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 periodl_rq_change_utilization (rq, dl_se, new_bw); } +/* Clear these so that the dl_server is reinitialized */ +if (new_bw ==0 ) { + dl_se->dl_defer =0 ; + dl_se->dl_server =0 ; + } + dl_se->dl_runtime = runtime; dl_se->dl_deadline = period; dl_se->dl_period = period;
@@ -1757,6 +1763,31 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perioreturn retval; } +/** +* dl_server_remove_params - Remove bandwidth reservation for a DL server +* @dl_se: The DL server entity to remove bandwidth for +* +* This function removes the bandwidth reservation for a DL server entity, +* cleaning up all bandwidth accounting and server state. +* +* Returns: 0 on success, negative error code on failure +*/ +int dl_server_remove_params (struct sched_dl_entity *dl_se) +{ +if (!dl_se->dl_server) +return 0 ;/* Already disabled */ + +/* +* First dequeue if still queued. It should not be queued since +* we call this only after the last dl_server_stop(). +*/ +if (WARN_ON_ONCE (on_dl_rq (dl_se))) +dequeue_dl_entity (dl_se,DEQUEUE_SLEEP ); + +/* Remove bandwidth reservation */ +return dl_server_apply_params (dl_se,0 , dl_se->dl_period, false); +} +/* * Update the current task's runtime statistics (provided it is still * a -deadline task and has not been removed from the dl_rq).
kernel/sched/sched.h +1 -0
@@ -393,6 +393,7 @@ extern void ext_server_init(struct rq *rq);extern void __dl_server_attach_root (struct sched_dl_entity *dl_se,struct rq *rq);extern int dl_server_apply_params (struct sched_dl_entity *dl_se,u64 runtime,u64 period,bool init); +extern int dl_server_remove_params (struct sched_dl_entity *dl_se);static inline bool dl_server_active (struct sched_dl_entity *dl_se) {
Signature · 1 line
2.34.1
PATCH v6 11/14 sched/ext: Relinquish DL server reservations when not needed
In reply to Joel FernandesI tested loading a test SCX program and verifying the bandwidth both before and after applying the patch:
Without patch:
Before loading scx:
.dl_bw->total_bw : 1887408
After unloading scx:
.dl_bw->total_bw : 3774816
After patch:
Before loading scx:
.dl_bw->total_bw : 1887408
After unloading scx:
.dl_bw->total_bw : 1887408
--- kernel/sched/ext.c | 56 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 47 insertions(+), 9 deletions(-)
kernel/sched/ext.c +47 -9
@@ -4965,6 +4965,31 @@ static void scx_disable_workfn(struct kthread_work *work) scx_init_task_enabled = false; +for_each_possible_cpu (cpu) { +struct rq *rq =cpu_rq (cpu); +struct rq_flags rf; + +/* +* Invalidate all the rq clocks to prevent getting outdated +* rq clocks from a previous scx scheduler. +*/ +scx_rq_clock_invalidate (rq); + +/* +* We are unloading the sched_ext scheduler, we do not need its +* DL server bandwidth anymore, remove it for all CPUs. Whenever +* the first SCX task is enqueued (when scx is re-loaded), its DL +* server bandwidth will be re-initialized. +*/ +rq_lock_irqsave (rq, &rf); +update_rq_clock (rq); +if (dl_server_active (&rq->ext_server)) +dl_server_stop (&rq->ext_server); +dl_server_remove_params (&rq->ext_server); +rq_unlock_irqrestore (rq, &rf); + } + +scx_task_iter_start (&sti);while ((p =scx_task_iter_next_locked (&sti))) {const struct sched_class *old_class = p->sched_class;
@@ -4988,15 +5013,6 @@ static void scx_disable_workfn(struct kthread_work *work)scx_task_iter_stop (&sti);percpu_up_write (&scx_fork_rwsem); -/* -* Invalidate all the rq clocks to prevent getting outdated -* rq clocks from a previous scx scheduler. -*/ -for_each_possible_cpu (cpu) { -struct rq *rq =cpu_rq (cpu); -scx_rq_clock_invalidate (rq); - } -/* no task is on scx, turn off all the switches and flush in-progress calls */ static_branch_disable (&__scx_enabled);bitmap_zero (sch->has_op,SCX_OPI_END );
@@ -5730,6 +5746,28 @@ static int scx_enable(struct sched_ext_ops *ops, struct bpf_link *link)check_class_changed (task_rq (p), p, old_class, p->prio); }scx_task_iter_stop (&sti); + +if (scx_switching_all) { +for_each_possible_cpu (cpu) { +struct rq *rq =cpu_rq (cpu); +struct rq_flags rf; + +/* +* We are switching all fair tasks to the sched_ext scheduler, +* we do not need fair server's DL bandwidth anymore, remove it +* for all CPUs. Whenever the first CFS task is enqueued (when +* scx is unloaded), the fair server's DL bandwidth will be +* re-initialized. +*/ +rq_lock_irqsave (rq, &rf); +update_rq_clock (rq); +if (dl_server_active (&rq->fair_server)) +dl_server_stop (&rq->fair_server); +dl_server_remove_params (&rq->fair_server); +rq_unlock_irqrestore (rq, &rf); + } + } +percpu_up_write (&scx_fork_rwsem);scx_bypass (false);
Signature · 1 line
2.34.1
PATCH v6 12/14 selftests/sched_ext: Add test for sched_ext dl_server
In reply to Joel FernandesFrom: Andrea Righi <[email protected]>
Add a selftest to validate the correct behavior of the deadline server for the ext_sched_class.
[ Joel: Replaced occurences of CFS in the test with EXT. ]
--- tools/testing/selftests/sched_ext/Makefile | 1 + .../selftests/sched_ext/rt_stall.bpf.c | 23 ++ tools/testing/selftests/sched_ext/rt_stall.c | 213 ++++++++++++++++++ 3 files changed, 237 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/rt_stall.bpf.c create mode 100644 tools/testing/selftests/sched_ext/rt_stall.c
tools/testing/selftests/sched_ext/Makefile +1 -0
@@ -182,6 +182,7 @@ auto-test-targets := \ select_cpu_dispatch_bad_dsq \ select_cpu_dispatch_dbl_dsp \ select_cpu_vtime \ + rt_stall \ test_example \ testcase-targets := $(addsuffix .o,$(addprefix $(SCXOBJ_DIR)/,$(auto-test-targets)))
tools/testing/selftests/sched_ext/rt_stall.bpf.c new file +23 -0
@@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* A scheduler that verified if RT tasks can stall SCHED_EXT tasks. +* +* Copyright (c) 2025 NVIDIA Corporation. +*/ + +#include <scx/common.bpf.h> + +char _license[] SEC("license") = "GPL"; + +UEI_DEFINE (uei); + +void BPF_STRUCT_OPS (rt_stall_exit ,struct scx_exit_info *ei) +{ +UEI_RECORD (uei, ei); +} + +SEC(".struct_ops.link") +struct sched_ext_ops rt_stall_ops = { + .exit = (void *)rt_stall_exit, + .name ="rt_stall" , +};
tools/testing/selftests/sched_ext/rt_stall.c new file +213 -0
213 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v6 13/14 sched/deadline: Fix DL server crash in inactive_timer callback
In reply to Joel FernandesWhen sched_ext is rapidly disabled/enabled (the reload_loop selftest), the following crash is observed. This happens because the timer handler could not be cancelled and still fires even though the dl_server bandwidth may have been removed via dl_server_remove_params(). hrtimer_try_to_cancel() does not guarantee timer cancellation. This results in a NULL pointer dereference as 'p' is bogus for a dl_se.
I think this happens because the timer may be about to run, but its softirq has not executed yet. Because of that hrtimer_try_to_cancel() cannot prevent the timer from being canceled, however dl_server is still set to 0 by dl_server_apply_params(). When the timer handler eventually runs, it crashes.
[ 24.771835] BUG: kernel NULL pointer dereference, address: 000000000000006c [ 24.772097] #PF: supervisor read access in kernel mode [ 24.772248] #PF: error_code(0x0000) - not-present page [ 24.772404] PGD 0 P4D 0 [ 24.772499] Oops: Oops: 0000 [#1] SMP PTI [ 24.772614] CPU: 9 UID: 0 PID: 0 Comm: swapper/9 [..] #74 PREEMPT(voluntary) [ 24.772932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), [...] [ 24.773149] Sched_ext: maximal (disabling) [ 24.773944] RSP: 0018:ffffb162c0348ee0 EFLAGS: 00010046 [ 24.774100] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88d4412f1800 [ 24.774302] RDX: 0000000000000001 RSI: 0000000000000010 RDI: ffffffffac939240 [ 24.774498] RBP: ffff88d47e65b940 R08: 0000000000000010 R09: 00000008bad3370a [ 24.774742] R10: 0000000000000000 R11: ffffffffa9f159d0 R12: ffff88d47e65b900 [ 24.774962] R13: ffff88d47e65b960 R14: ffff88d47e66a340 R15: ffff88d47e66aed0 [ 24.775182] FS: 0000000000000000(0000) GS:ffff88d4d1d56000(0000) knlGS:[...] [ 24.775392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 24.775579] CR2: 000000000000006c CR3: 0000000002bb0003 CR4: 0000000000770ef0 [ 24.775810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 24.776023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 24.776225] PKRU: 55555554 [ 24.776292] Call Trace: [ 24.776373] <IRQ> [ 24.776453] ? __pfx_inactive_task_timer+0x10/0x10 [ 24.776591] __hrtimer_run_queues+0xf1/0x270 [ 24.776744] hrtimer_interrupt+0xfa/0x220 [ 24.776847] __sysvec_apic_timer_interrupt+0x4d/0x190 [ 24.776988] sysvec_apic_timer_interrupt+0x69/0x80 [ 24.777132] </IRQ> [ 24.777194] <TASK> [ 24.777256] asm_sysvec_apic_timer_interrupt+0x1a/0x20
Fix, by also checking the DL server's has_task pointer which only exists for server tasks. This fixes the crash.
--- kernel/sched/deadline.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/sched/deadline.c +9 -3
@@ -1822,7 +1822,13 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)struct rq_flags rf;struct rq *rq; -if (!dl_server (dl_se)) { +/* +* It is possible that after dl_server_apply_params(), the dl_se->dl_server == 0, +* but the inactive timer is still queued and could not get canceled. Double check +* by looking at ->server_has_tasks to make sure we're dealing with a non-server +* here. Otherwise p may be bogus and we'll crash. +*/ +if (!dl_server (dl_se) && !dl_se->server_has_tasks) { p =dl_task_of (dl_se); rq =task_rq_lock (p, &rf); }else {
@@ -1833,7 +1839,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)sched_clock_tick ();update_rq_clock (rq); -if (dl_server (dl_se)) +if (dl_server (dl_se) || dl_se->server_has_tasks) goto no_task;if (!dl_task (p) ||READ_ONCE (p->__state) ==TASK_DEAD ) {
@@ -1861,7 +1867,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) dl_se->dl_non_contending =0 ; unlock: -if (!dl_server (dl_se)) { +if (!dl_server (dl_se) && !dl_se->server_has_tasks) {task_rq_unlock (rq, p, &rf);put_task_struct (p); }else {
Signature · 1 line
2.34.1
PATCH v6 14/14 selftests/sched_ext: Add test for DL server total_bw consistency
In reply to Joel FernandesAdd a new kselftest to verify that the total_bw value in /sys/kernel/debug/sched/debug remains consistent across all CPUs under different sched_ext BPF program states:
1. Before a BPF scheduler is loaded
2. While a BPF scheduler is loaded and active
3. After a BPF scheduler is unloaded
The test runs CPU stress threads to ensure DL server bandwidth values stabilize before checking consistency. This helps catch potential issues with DL server bandwidth accounting during sched_ext transitions.
--- tools/testing/selftests/sched_ext/Makefile | 1 + tools/testing/selftests/sched_ext/total_bw.c | 286 +++++++++++++++++++ 2 files changed, 287 insertions(+) create mode 100644 tools/testing/selftests/sched_ext/total_bw.c
tools/testing/selftests/sched_ext/Makefile +1 -0
@@ -184,6 +184,7 @@ auto-test-targets := \ select_cpu_vtime \ rt_stall \ test_example \ + total_bw \ testcase-targets := $(addsuffix .o,$(addprefix $(SCXOBJ_DIR)/,$(auto-test-targets)))
tools/testing/selftests/sched_ext/total_bw.c new file +286 -0
286 lines not on this page · raw message
Signature · 1 line
2.34.1