Hierarchical Constant Bandwidth Server
Subsystems (2)
v4 cover Hierarchical Constant Bandwidth Server
Hello,
This is the v4 for Hierarchical Constant Bandwidth Server, aiming at replacing the current RT_GROUP_SCHED mechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ . You can find the previous versions of this patchset at the bottom of the page, in particular version 1 which talks in more detail what this patchset is all about and how it is implemented.
This v4 version reworks some of the patches as suggested by Juri Lelli and Markus Elfring. Follows the list of changes:
- General refactorings, cleanups, removal of unnecessary ifdeffy and comments.
- Add Documentation for HCBS, with how-tos and some theoretical background.
- Change names/definitions of active groups:
- A **live** group is one that is accounted for bw and tasks can be attached.
- An **active** group is a **live** group with tasks running inside.
- Add correct cleanup of allocated memory in alloc_rt_sched_group (on allocation failure), even tho free_rt_sched_group is called on error.
- Fix computing of new bandwidth values in dl_init_tg.
- Fix check in dl_check_tg to use capacity scaling.
- Fix wakeup_preempt_rt to check if curr is a DEADLINE task.
- Update inc/dec_dl_tasks to account for served runqueues regardless of the server type.
- Update add_nr_running to update root domains and perform tracing only if the given runqueue is global.
- Introduce server_try_pull_task, as server_has_task gets removed in kernel version 6.18. This is needed to perform a pull on HCBS server replenish.
- Introduce RELEASE_LOCK macro for cleaner guard-based lock code.
- Move debug BUG_ONs to separate patches, since they are not meant to be used as asserts. The last two patches are not meant to be incorporated in the kernel, but are just used to introduce debug asserts for easier testing of expected preconditions when executing some functions.
The testing system has also been updated to get rid of the closed-source software dependency to generate the tasksets and their valid configurations. More on that on the Testing section.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Summary of the patches:
1-4) Preparation patches, so that the RT classes' code can be used both
for normal and cgroup scheduling.
5-16) Implementation of HCBS, no migration and only one level hierarchy.
The old RT_GROUP_SCHED code is removed.
17-18) Remove cgroups v1 in favour of v2.
19) Add support for deeper hierarchies.
20-25) Add support for tasks migration.
26) Documentation for HCBS.
27-28) Debug BUG_ONs optional patches.
Updates from v3:
- Rebase to latest tip/master.
- General rebasing/cleanup.
- Add Documentation.
- Define **live** and **active** groups.
- Introduce server_try_pull_task in place of the removed server_has_task.
- Introduce RELEASE_LOCK helper macro for guard-based locking.
- Update inc/dec_dl_tasks to account for served runqueues regardless of the server type.
- Fix computing of new bandwidth values in dl_init_tg.
- Fix check in dl_check_tg to use capacity scaling.
- Fix wakeup_preempt_rt to check if curr is a DEADLINE task.
Updates from v2:
- Rebase to latest tip/master.
- Remove fair-servers' bw reclaiming.
- Fix a check which prevented execution of wakeup_preempt code.
- Fix a priority check in group_pull_rt_task between tasks of different groups.
- Rework allocation/deallocation code for rt-cgroups.
- Update signatures for some group related migration functions.
- Add documentation for wakeup_preempt preemption rules.
Updates from v1:
- Rebase to latest tip/master.
- Add migration code.
- Split big patches for more readability.
- Refactor code to use guarded locks where applicable.
- Remove unnecessary patches from v1 which have been addressed differently by mainline updates.
- Remove unnecessary checks and general code cleanup.
Notes:
Task migration support needs some extra work to reduce its invasiveness, especially patches 24-25. Patches 27-28 are completely optional and are not meant to be included in the final patchset: they just add some invasive BUG_ONs that assert some preconditions expected on some function calls.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Testing v4:
We are still using the tests published in version 3 for the evaluation of the patchset (refer to the v3 cover letter for more details). For HCBS v4 the so called "Taskset" tests have been updated to use rt-app as a runner, while the tasksets + configurations themselves are now generated using a completely new open source tool: EVA-rt-Engine (https://github.com/Yurand2000/EVA-rt-Engine).
The tests are available at https://github.com/Yurand2000/HCBS-Test-Suite . Refer to the README of the repository for more details.
Follow these steps to test HCBS v4:
- Get the HCBS patch up and running. Any kernel/disto should work effortlessly.
- Get, compile and _install_ the tests.
- Run the go_rt.sh script to set the frequency of the CPUs to a fixed value and disable hyperthreading and power saving features.
- Run the run_tests.sh full script, to run the whole test suite.
Notes:
While you may have rt-app installed in your system, the testing suite comes with its own rt-app version bundled.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Future Work:
While we wait for more comments, and expect stuff to break, we will work on completing the currently partial/untested, implementation of HCBS with different runtimes per CPU, instead of having the same runtime allocated on all CPUs, to include it in a future RCF.
Future patches:
- HCBS with different runtimes per CPU.
- capacity aware bandwidth reservation.
- enable/disable dl_servers when a CPU goes online/offline.
Have a nice day,
Yuri
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Yuri Andriaccio (11):
sched/rt: Disable RT_GROUP_SCHED
sched/rt: Remove rq field in struct rt_rq
sched/rt: Implement dl-server operations for rt-cgroups. sched/rt: Update task event callbacks for HCBS scheduling sched/rt: Allow zeroing the runtime of the root control group sched/rt: Remove support for cgroups-v1
sched/deadline: Introduce dl_server_try_pull_f sched/core: Execute enqueued balance callbacks when migrating task
betweeen cgroups
Documentation: Update documentation for real-time cgroups [DEBUG] sched/rt: Add debug BUG_ONs for pre-migration code [DEBUG] sched/rt: Add debug BUG_ONs in migration code.
luca abeni (17):
sched/deadline: Do not access dl_se->rq directly sched/deadline: Distinct between dl_rq and my_q
sched/rt: Pass an rt_rq instead of an rq where needed sched/rt: Move some functions from rt.c to sched.h sched/rt: Introduce HCBS specific structs in task_group sched/core: Initialize HCBS specific structures. sched/deadline: Add dl_init_tg
sched/rt: Add {alloc/free}_rt_sched_group
sched/deadline: Account rt-cgroups bandwidth in deadline tasks
schedulability tests.
sched/rt: Update rt-cgroup schedulability checks sched/rt: Remove old RT_GROUP_SCHED data structures sched/core: Cgroup v2 support
sched/deadline: Allow deeper hierarchies of RT cgroups sched/rt: Add rt-cgroup migration
sched/rt: Add HCBS migration related checks and function calls sched/deadline: Fix HCBS migrations on server stop sched/core: Execute enqueued balance callbacks when changing allowed
CPUs
Documentation/scheduler/sched-rt-group.rst | 500 +++- include/linux/cleanup.h | 3 + include/linux/sched.h | 13 +- kernel/sched/autogroup.c | 4 +- kernel/sched/core.c | 63 +- kernel/sched/deadline.c | 257 +- kernel/sched/debug.c | 6 - kernel/sched/fair.c | 10 +- kernel/sched/rt.c | 3097 ++++++++++---------- kernel/sched/sched.h | 176 +- kernel/sched/syscalls.c | 6 +- 11 files changed, 2346 insertions(+), 1789 deletions(-)
base-commit: 6a23ae0a96a600d1d12557add110e0bb6e32730c --
2.51.0
RFC PATCH v4 1/28 sched/deadline: Do not access dl_se->rq directly
From: luca abeni <[email protected]>
Make deadline.c code access the runqueue of a scheduling entity saved in the sched_dl_entity data structure. This allows future patches to save different runqueues in sched_dl_entity other than the global runqueues.
Move dl_server_apply_params call in sched_init_dl_servers as the rq_of_dl_se function will return the correct deadline entity only if the dl_server flag is set.
Add a WARN_ON on the return value of dl_server_apply_params in sched_init_dl_servers as this function may fail if the kernel is not configured correctly.
--- kernel/sched/deadline.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-)
kernel/sched/deadline.c +14 -13
@@ -874,7 +874,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)* and arm the defer timer. */ if (dl_se->dl_defer && !dl_se->dl_defer_running && -dl_time_before (rq_clock (dl_se->rq), dl_se->deadline - dl_se->runtime)) { +dl_time_before (rq_clock (rq), dl_se->deadline - dl_se->runtime)) {if (!is_dl_boosted (dl_se)) {/*
@@ -1176,11 +1176,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_* of time. The dl_server_min_res serves as a limit to avoid * forwarding the timer for a too small amount of time. */ -if (dl_time_before (rq_clock (dl_se->rq), +if (dl_time_before (rq_clock (rq), (dl_se->deadline - dl_se->runtime - dl_server_min_res))) {/* reset the defer timer */ - fw = dl_se->deadline -rq_clock (dl_se->rq) - dl_se->runtime; + fw = dl_se->deadline -rq_clock (rq) - dl_se->runtime;hrtimer_forward_now (timer,ns_to_ktime (fw));return HRTIMER_RESTART ;
@@ -1191,7 +1191,7 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_enqueue_dl_entity (dl_se,ENQUEUE_REPLENISH ); -if (!dl_task (dl_se->rq->curr) ||dl_entity_preempt (dl_se, &dl_se->rq->curr->dl)) +if (!dl_task (rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl))resched_curr (rq);__push_dl_task (rq, rf);
@@ -1459,7 +1459,7 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64hrtimer_try_to_cancel (&dl_se->dl_timer); -replenish_dl_new_period (dl_se, dl_se->rq); +replenish_dl_new_period (dl_se, rq);/* * Not being able to start the timer seems problematic. If it could not
@@ -1572,23 +1572,24 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec) {/* 0 runtime = fair server disabled */ if (dl_se->dl_runtime) -update_curr_dl_se (dl_se->rq, dl_se, delta_exec); +update_curr_dl_se (rq_of_dl_se (dl_se), dl_se, delta_exec); }void dl_server_start (struct sched_dl_entity *dl_se) { -struct rq *rq = dl_se->rq; +struct rq *rq;if (!dl_server (dl_se) || dl_se->dl_server_active)return ; + rq =rq_of_dl_se (dl_se);if (WARN_ON_ONCE (!cpu_online (cpu_of (rq))))return ; dl_se->dl_server_active =1 ;enqueue_dl_entity (dl_se,ENQUEUE_WAKEUP ); -if (!dl_task (dl_se->rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl)) -resched_curr (dl_se->rq); +if (!dl_task (rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl)) +resched_curr (rq); }void dl_server_stop (struct sched_dl_entity *dl_se)
@@ -1628,9 +1629,9 @@ void sched_init_dl_servers(void)WARN_ON (dl_server (dl_se)); -dl_server_apply_params (dl_se, runtime, period,1 ); - dl_se->dl_server =1 ; +WARN_ON (dl_server_apply_params (dl_se, runtime, period,1 )); + dl_se->dl_defer =1 ;setup_new_dl_entity (dl_se); }
@@ -1655,7 +1656,7 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio {u64 old_bw = init ?0 :to_ratio (dl_se->dl_period, dl_se->dl_runtime);u64 new_bw =to_ratio (period, runtime); -struct rq *rq = dl_se->rq; +struct rq *rq =rq_of_dl_se (dl_se);int cpu =cpu_of (rq);struct dl_bw *dl_b;unsigned long cap;
@@ -1732,7 +1733,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) p =dl_task_of (dl_se); rq =task_rq_lock (p, &rf); }else { - rq = dl_se->rq; + rq =rq_of_dl_se (dl_se);rq_lock (rq, &rf); }
Signature · 1 line
2.51.0
Re: [RFC PATCH v4 01/28] sched/deadline: Do not access dl_se->rq directly
Hello!
And Happy New Year! :)
Quoted 9 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Make
deadline.ccode access the runqueue of a scheduling entity saved in thesched_dl_entitydata structure. This allows future patches to save different runqueues insched_dl_entityother than the global runqueues.Move
dl_server_apply_paramscall insched_init_dl_serversas therq_of_dl_sefunction will return the correct deadline entity only if thedl_serverflag is set.
s/deadline entity/runqueue/ in the above, right?
Quoted 6 lines
Add a
WARN_ONon the return value ofdl_server_apply_paramsinsched_init_dl_serversas this function may fail if the kernel is not configured correctly.Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>
Apart from that
Thanks!
Juri
RFC PATCH v4 2/28 sched/deadline: Distinct between dl_rq and my_q
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Split the single runqueue pointer in sched_dl_entity into two separate pointers, following the existing pattern used by sched_rt_entity:
- dl_rq: Points to the deadline runqueue where this entity is queued (global runqueue).
- my_q: Points to the runqueue that this entity serves (for servers).
This distinction is currently redundant for the fair_server (both point to the same CPU's structures), but is essential for future RT cgroup support where deadline servers will be queued on the global dl_rq while serving tasks from cgroup-specific runqueues.
Update rq_of_dl_se() to use container_of() to recover the global rq from dl_rq, and update fair.c to explicitly use my_q (local rq) when accessing the served runqueue.
Update dl_server_init() to take a dl_rq pointer (use to retrieve the global runqueue where the dl_server is scheduled) and a rq pointer (for the local runqueue served by the server).
--- include/linux/sched.h | 6 ++++-- kernel/sched/deadline.c | 11 +++++++---- kernel/sched/fair.c | 4 ++-- kernel/sched/sched.h | 3 ++- 4 files changed, 15 insertions(+), 9 deletions(-)
include/linux/sched.h +4 -2
@@ -729,12 +729,14 @@ struct sched_dl_entity {* Bits for DL-server functionality. Also see the comment near * dl_server_update(). * -* @rq the runqueue this server is for +* @dl_rq the runqueue on which this entity is (to be) queued +* @my_q the runqueue "owned" by this entity * * @server_has_tasks() returns true if @server_pick return a * runnable task. */ -struct rq *rq; +struct dl_rq *dl_rq; +struct rq *my_q;dl_server_pick_f server_pick_task;#ifdef CONFIG_RT_MUTEXES
kernel/sched/deadline.c +7 -4
@@ -75,11 +75,12 @@ static inline struct rq *rq_of_dl_rq(struct dl_rq *dl_rq)static inline struct rq *rq_of_dl_se (struct sched_dl_entity *dl_se) { -struct rq *rq = dl_se->rq; +struct rq *rq;if (!dl_server (dl_se)) rq =task_rq (dl_task_of (dl_se)); - +else + rq = container_of(dl_se->dl_rq, struct rq, dl);return rq; }
@@ -1604,10 +1605,12 @@ void dl_server_stop(struct sched_dl_entity *dl_se) dl_se->dl_server_active =0 ; } -void dl_server_init (struct sched_dl_entity *dl_se,struct rq *rq, +void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq, +struct rq *served_rq,dl_server_pick_f pick_task) { - dl_se->rq = rq; + dl_se->dl_rq = dl_rq; + dl_se->my_q = served_rq; dl_se->server_pick_task = pick_task; }
kernel/sched/fair.c +2 -2
@@ -8959,7 +8959,7 @@ static struct task_struct *__pick_next_task_fair(struct rq *rq, struct task_strustatic struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se) { -return pick_task_fair (dl_se->rq); +return pick_task_fair (dl_se->my_q); }void fair_server_init (struct rq *rq)
@@ -8968,7 +8968,7 @@ void fair_server_init(struct rq *rq)init_dl_entity (dl_se); -dl_server_init (dl_se, rq, fair_server_pick_task); +dl_server_init (dl_se, &rq->dl, rq, fair_server_pick_task); }/*
kernel/sched/sched.h +2 -1
@@ -408,7 +408,8 @@ extern s64 dl_scaled_delta_exec(struct rq *rq, struct sched_dl_entity *dl_se, s6extern void dl_server_update (struct sched_dl_entity *dl_se,s64 delta_exec);extern void dl_server_start (struct sched_dl_entity *dl_se);extern void dl_server_stop (struct sched_dl_entity *dl_se); -extern void dl_server_init (struct sched_dl_entity *dl_se,struct rq *rq, +extern void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq, +struct rq *served_rq,dl_server_pick_f pick_task);extern void sched_init_dl_servers (void );
Signature · 1 line
2.51.0
Re: [RFC PATCH v4 02/28] sched/deadline: Distinct between dl_rq and my_q
Hello,
Think we want s/Distinct/Distinguish/ in $SUBJECT.
Quoted 18 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Split the single runqueue pointer in
sched_dl_entityinto two separate pointers, following the existing pattern used bysched_rt_entity:-
dl_rq: Points to the deadline runqueue where this entity is queued (global runqueue).
-my_q: Points to the runqueue that this entity serves (for servers).This distinction is currently redundant for the
fair_server(both point to the same CPU's structures), but is essential for future RT cgroup support where deadline servers will be queued on the globaldl_rqwhile serving tasks from cgroup-specific runqueues.Update
rq_of_dl_se()to usecontainer_of()to recover the global rq fromdl_rq, and updatefair.cto explicitly usemy_q(local rq) when accessing the served runqueue.
Quoted 6 lines
Update
dl_server_init()to take adl_rqpointer (use to retrieve the global runqueue where thedl_serveris scheduled) and a rq pointer (for the local runqueue served by the server).Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>
Thanks,
Juri
RFC PATCH v4 3/28 sched/rt: Pass an rt_rq instead of an rq where needed
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make rt.c code access the runqueue through the rt_rq data structure rather than passing an rq pointer directly. This allows future patches to define rt_rq data structures which do not refer only to the global runqueue, but also to local cgroup runqueues (as rt_rq will not be always equal to &rq->rt).
Add checks in rt_queue_{push/pull}_tasks to make sure that the given rt_rq object refers to a global runqueue and not any local one.
--- kernel/sched/rt.c | 99 ++++++++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 45 deletions(-)
kernel/sched/rt.c +54 -45
@@ -370,9 +370,9 @@ static inline void rt_clear_overload(struct rq *rq)cpumask_clear_cpu (rq->cpu, rq->rd->rto_mask); } -static inline int has_pushable_tasks (struct rq *rq) +static inline int has_pushable_tasks (struct rt_rq *rt_rq) { -return !plist_head_empty (&rq->rt.pushable_tasks); +return !plist_head_empty (&rt_rq->pushable_tasks); } static DEFINE_PER_CPU(struct balance_callback, rt_push_head);
@@ -381,50 +381,54 @@ static DEFINE_PER_CPU(struct balance_callback, rt_pull_head);static void push_rt_tasks (struct rq *);static void pull_rt_task (struct rq *); -static inline void rt_queue_push_tasks (struct rq *rq) +static inline void rt_queue_push_tasks (struct rt_rq *rt_rq) { -if (!has_pushable_tasks (rq)) + struct rq *rq = container_of(rt_rq, struct rq, rt); + +if (!has_pushable_tasks (rt_rq))return ;queue_balance_callback (rq, &per_cpu (rt_push_head, rq->cpu), push_rt_tasks); } -static inline void rt_queue_pull_task (struct rq *rq) +static inline void rt_queue_pull_task (struct rt_rq *rt_rq) { + struct rq *rq = container_of(rt_rq, struct rq, rt); +queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); } -static void enqueue_pushable_task (struct rq *rq,struct task_struct *p) +static void enqueue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) { -plist_del (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);plist_node_init (&p->pushable_tasks, p->prio); -plist_add (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_add (&p->pushable_tasks, &rt_rq->pushable_tasks);/* Update the highest prio pushable task */ -if (p->prio < rq->rt.highest_prio.next) - rq->rt.highest_prio.next = p->prio; +if (p->prio < rt_rq->highest_prio.next) + rt_rq->highest_prio.next = p->prio; -if (!rq->rt.overloaded) { -rt_set_overload (rq); - rq->rt.overloaded =1 ; +if (!rt_rq->overloaded) { +rt_set_overload (rq_of_rt_rq (rt_rq)); + rt_rq->overloaded =1 ; } } -static void dequeue_pushable_task (struct rq *rq,struct task_struct *p) +static void dequeue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) { -plist_del (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);/* Update the new highest prio pushable task */ -if (has_pushable_tasks (rq)) { - p =plist_first_entry (&rq->rt.pushable_tasks, +if (has_pushable_tasks (rt_rq)) { + p =plist_first_entry (&rt_rq->pushable_tasks, struct task_struct, pushable_tasks); - rq->rt.highest_prio.next = p->prio; + rt_rq->highest_prio.next = p->prio; }else { - rq->rt.highest_prio.next =MAX_RT_PRIO -1 ; + rt_rq->highest_prio.next =MAX_RT_PRIO -1 ; -if (rq->rt.overloaded) { -rt_clear_overload (rq); - rq->rt.overloaded =0 ; +if (rt_rq->overloaded) { +rt_clear_overload (rq_of_rt_rq (rt_rq)); + rt_rq->overloaded =0 ; } } }
@@ -1431,6 +1435,7 @@ static void enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags) { struct sched_rt_entity *rt_se = &p->rt; +struct rt_rq *rt_rq =rt_rq_of_se (rt_se);if (flags &ENQUEUE_WAKEUP ) rt_se->timeout =0 ;
@@ -1444,17 +1449,18 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)return ;if (!task_current (rq, p) && p->nr_cpus_allowed >1 ) -enqueue_pushable_task (rq, p); +enqueue_pushable_task (rt_rq, p); }static bool dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) {struct sched_rt_entity *rt_se = &p->rt; +struct rt_rq *rt_rq =rt_rq_of_se (rt_se);update_curr_rt (rq);dequeue_rt_entity (rt_se, flags); -dequeue_pushable_task (rq, p); +dequeue_pushable_task (rt_rq, p);return true; }
@@ -1639,14 +1645,14 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)static inline void set_next_task_rt (struct rq *rq,struct task_struct *p,bool first) {struct sched_rt_entity *rt_se = &p->rt; -struct rt_rq *rt_rq = &rq->rt; +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt); p->se.exec_start =rq_clock_task (rq);if (on_rt_rq (&p->rt))update_stats_wait_end_rt (rt_rq, rt_se);/* The running task is never eligible for pushing */ -dequeue_pushable_task (rq, p); +dequeue_pushable_task (rt_rq, p);if (!first)return ;
@@ -1659,7 +1665,7 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool fif (rq->donor->sched_class != &rt_sched_class)update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); -rt_queue_push_tasks (rq); +rt_queue_push_tasks (rt_rq); }static struct sched_rt_entity *pick_next_rt_entity (struct rt_rq *rt_rq)
@@ -1710,7 +1716,7 @@ static struct task_struct *pick_task_rt(struct rq *rq)static void put_prev_task_rt (struct rq *rq,struct task_struct *p,struct task_struct *next) {struct sched_rt_entity *rt_se = &p->rt; -struct rt_rq *rt_rq = &rq->rt; +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt);if (on_rt_rq (&p->rt))update_stats_wait_start_rt (rt_rq, rt_se);
@@ -1726,7 +1732,7 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s* if it is still active */ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 ) -enqueue_pushable_task (rq, p); +enqueue_pushable_task (rt_rq, p); }/* Only try algorithms three times */
@@ -1736,16 +1742,16 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s* Return the highest pushable rq's task, which is suitable to be executed * on the CPU, NULL otherwise */ -static struct task_struct *pick_highest_pushable_task (struct rq *rq,int cpu) +static struct task_struct *pick_highest_pushable_task (struct rt_rq *rt_rq,int cpu) { -struct plist_head *head = &rq->rt.pushable_tasks; +struct plist_head *head = &rt_rq->pushable_tasks;struct task_struct *p; -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (rt_rq))return NULL ;plist_for_each_entry (p, head, pushable_tasks) { -if (task_is_pushable (rq, p, cpu)) +if (task_is_pushable (rq_of_rt_rq (rt_rq), p, cpu))return p; }
@@ -1845,14 +1851,15 @@ static int find_lowest_rq(struct task_struct *task)return -1 ; } -static struct task_struct *pick_next_pushable_task (struct rq *rq) +static struct task_struct *pick_next_pushable_task (struct rt_rq *rt_rq) { +struct rq *rq =rq_of_rt_rq (rt_rq);struct task_struct *p; -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (rt_rq))return NULL ; - p =plist_first_entry (&rq->rt.pushable_tasks, + p =plist_first_entry (&rt_rq->pushable_tasks, struct task_struct, pushable_tasks);BUG_ON (rq->cpu !=task_cpu (p));
@@ -1905,7 +1912,7 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)*/ if (unlikely (is_migration_disabled (task) || !cpumask_test_cpu (lowest_rq->cpu, &task->cpus_mask) || - task !=pick_next_pushable_task (rq))) { + task !=pick_next_pushable_task (&rq->rt))) {double_unlock_balance (rq, lowest_rq); lowest_rq =NULL ;
@@ -1939,7 +1946,7 @@ static int push_rt_task(struct rq *rq, bool pull)if (!rq->rt.overloaded)return 0 ; - next_task =pick_next_pushable_task (rq); + next_task =pick_next_pushable_task (&rq->rt);if (!next_task)return 0 ;
@@ -2014,7 +2021,7 @@ static int push_rt_task(struct rq *rq, bool pull)* run-queue and is also still the next task eligible for * pushing. */ - task =pick_next_pushable_task (rq); + task =pick_next_pushable_task (&rq->rt);if (task == next_task) {/* * The task hasn't migrated, and is still the next
@@ -2202,7 +2209,7 @@ void rto_push_irq_work_func(struct irq_work *work)* We do not need to grab the lock to check for has_pushable_tasks. * When it gets updated, a check is made if a push is possible. */ -if (has_pushable_tasks (rq)) { +if (has_pushable_tasks (&rq->rt)) {raw_spin_rq_lock (rq);while (push_rt_task (rq, true)) ;
@@ -2231,6 +2238,7 @@ static void pull_rt_task(struct rq *this_rq)int this_cpu = this_rq->cpu, cpu;bool resched = false;struct task_struct *p, *push_task; +struct rt_rq *src_rt_rq;struct rq *src_rq;int rt_overload_count =rt_overloaded (this_rq);
@@ -2260,6 +2268,7 @@ static void pull_rt_task(struct rq *this_rq)continue ; src_rq =cpu_rq (cpu); + src_rt_rq = &src_rq->rt;/* * Don't bother taking the src_rq->lock if the next highest
@@ -2268,7 +2277,7 @@ static void pull_rt_task(struct rq *this_rq)* logically higher, the src_rq will push this task away. * And if its going logically lower, we do not care */ -if (src_rq->rt.highest_prio.next >= +if (src_rt_rq->highest_prio.next >= this_rq->rt.highest_prio.curr)continue ;
@@ -2284,7 +2293,7 @@ static void pull_rt_task(struct rq *this_rq)* We can pull only a task, which is pushable * on its rq, and no others. */ - p =pick_highest_pushable_task (src_rq, this_cpu); + p =pick_highest_pushable_task (src_rt_rq, this_cpu);/* * Do we have an RT task that preempts
@@ -2390,7 +2399,7 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)if (!task_on_rq_queued (p) || rq->rt.rt_nr_running)return ; -rt_queue_pull_task (rq); +rt_queue_pull_task (rt_rq_of_se (&p->rt)); }void __init init_sched_rt_class (void )
@@ -2426,7 +2435,7 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)*/ if (task_on_rq_queued (p)) {if (p->nr_cpus_allowed >1 && rq->rt.overloaded) -rt_queue_push_tasks (rq); +rt_queue_push_tasks (rt_rq_of_se (&p->rt));if (p->prio < rq->donor->prio &&cpu_online (cpu_of (rq)))resched_curr (rq); }
@@ -2448,7 +2457,7 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)* may need to pull tasks to this runqueue. */ if (oldprio < p->prio) -rt_queue_pull_task (rq); +rt_queue_pull_task (rt_rq_of_se (&p->rt));/* * If there's a higher priority task waiting to run
--
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v4 03/28] sched/rt: Pass an rt_rq instead of an rq where needed
Hello,
Quoted 34 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Make
rt.ccode access the runqueue through thert_rqdata structure rather than passing an rq pointer directly. This allows future patches to definert_rqdata structures which do not refer only to the global runqueue, but also to local cgroup runqueues (asrt_rqwill not be always equal to&rq->rt).Add checks in rt_queue_{push/pull}_tasks to make sure that the given
rt_rqobject refers to a global runqueue and not any local one.Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>--- kernel/sched/rt.c | 99 ++++++++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 45 deletions(-)kernel/sched/rt.c +2
-2@@ -370,9 +370,9 @@ static inline void rt_clear_overload(struct rq *rq)cpumask_clear_cpu (rq->cpu, rq->rd->rto_mask); } -static inline int has_pushable_tasks (struct rq *rq) +static inline int has_pushable_tasks (struct rt_rq *rt_rq) { -return !plist_head_empty (&rq->rt.pushable_tasks); +return !plist_head_empty (&rt_rq->pushable_tasks); } static DEFINE_PER_CPU(struct balance_callback, rt_push_head);
kernel/sched/rt.c static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); · line 386
Quoted 8 lines
kernel/sched/rt.c +2
-2@@ -381,50 +381,54 @@ static DEFINE_PER_CPU(struct balance_callback, rt_pull_head);static void push_rt_tasks (struct rq *);static void pull_rt_task (struct rq *); -static inline void rt_queue_push_tasks (struct rq *rq) +static inline void rt_queue_push_tasks (struct rt_rq *rt_rq) { -if (!has_pushable_tasks (rq)) + struct rq *rq = container_of(rt_rq, struct rq, rt);
Might be a minor thing, but I just noticed
https://elixir.bootlin.com/linux/v6.18-rc6/source/include/linux/container_of.h#L17
so, we might want to use container_of_const() here and below. Not that it makes a difference for these cases, but the indication for new code seems strict.
Peter, not sure if you have a preference here?
Thanks,
Juri
RFC PATCH v4 4/28 sched/rt: Move some functions from rt.c to sched.h
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make the following functions/macros be non-static and move them in sched.h, so that they can be also used in other source files:
- rt_entity_is_task()
- rt_task_of()
- rq_of_rt_rq()
- rt_rq_of_se()
- rq_of_rt_se()
There are no functional changes. This is needed by future patches.
--- kernel/sched/rt.c | 56 ------------------------------------------ kernel/sched/sched.h | 58 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 56 deletions(-)
kernel/sched/rt.c +0 -56
@@ -166,36 +166,6 @@ static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)hrtimer_cancel (&rt_b->rt_period_timer); } -#define rt_entity_is_task (rt_se) (!(rt_se)->my_q) - -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ -WARN_ON_ONCE (!rt_entity_is_task (rt_se)); - - return container_of(rt_se, struct task_struct, rt); -} - -static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) -{ -/* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */ -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -} - -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) -{ -WARN_ON (!rt_group_sched_enabled () && rt_se->rt_rq->tg != &root_task_group); -return rt_se->rt_rq; -} - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct rt_rq *rt_rq = rt_se->rt_rq; - -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -} -void unregister_rt_sched_group (struct task_group *tg) {if (!rt_group_sched_enabled ())
@@ -294,32 +264,6 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)#else /* !CONFIG_RT_GROUP_SCHED: */ -#define rt_entity_is_task (rt_se) (1) - -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ - return container_of(rt_se, struct task_struct, rt); -} - -static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) -{ - return container_of(rt_rq, struct rq, rt); -} - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct task_struct *p =rt_task_of (rt_se); - -return task_rq (p); -} - -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) -{ -struct rq *rq =rq_of_rt_se (rt_se); - -return &rq->rt; -} -void unregister_rt_sched_group (struct task_group *tg) { }void free_rt_sched_group (struct task_group *tg) { }
kernel/sched/sched.h +58 -0
64 lines not on this page · raw message
--
2.51.0
Re: [RFC PATCH v4 04/28] sched/rt: Move some functions from rt.c to sched.h
Hello,
Quoted 11 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Make the following functions/macros be non-static and move them in
sched.h, so that they can be also used in other source files:-
rt_entity_is_task()
-rt_task_of()
-rq_of_rt_rq()
-rt_rq_of_se()
-rq_of_rt_se()
Quoted 4 lines
There are no functional changes. This is needed by future patches.
Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>
Same question about container_of as for previous patch (even if this is just moving and not adding new code). But, let's see what we decide to do about that.
Thanks,
Juri
RFC PATCH v4 5/28 sched/rt: Disable RT_GROUP_SCHED
In reply to Yuri AndriaccioDisable the old RT_GROUP_SCHED scheduler. Note that this does not completely remove all the RT_GROUP_SCHED functionality, just unhooks it and removes most of the relevant functions. Some of the RT_GROUP_SCHED functions are kept because they will be adapted for the HCBS scheduling.
Most notably:
- Disable the initialization of the rt_bandwidth for group scheduling.
- Unhook any functionality for RT_GROUP_SCHED in normal rt.c code, leaving only non-group functionality.
- Remove group related field initialization in init_rt_rq().
- Remove all the unhooked (and so unused) functions from RT_GROUP_SCHED.
- Remove all allocation/deallocation code for rt-groups, always returning failure on allocation.
- Update inc/dec_rt_tasks active tasks' counters, as rt scheduling entities now only represent a single task, and not a group of tasks anymore.
--- kernel/sched/core.c | 6 - kernel/sched/deadline.c | 34 -- kernel/sched/debug.c | 6 - kernel/sched/rt.c | 862 ++-------------------------------------- kernel/sched/sched.h | 15 +- kernel/sched/syscalls.c | 13 - 6 files changed, 29 insertions(+), 907 deletions(-)
kernel/sched/core.c +0 -6
@@ -8691,11 +8691,6 @@ void __init sched_init(void)init_defrootdomain (); -#ifdef CONFIG_RT_GROUP_SCHED -init_rt_bandwidth (&root_task_group.rt_bandwidth, -global_rt_period (),global_rt_runtime ()); -#endif /* CONFIG_RT_GROUP_SCHED */ -#ifdef CONFIG_CGROUP_SCHED task_group_cache =KMEM_CACHE (task_group,0 );
@@ -8747,7 +8742,6 @@ void __init sched_init(void)* starts working after scheduler_running, which is not the case * yet. */ - rq->rt.rt_runtime =global_rt_runtime ();init_tg_rt_entry (&root_task_group, &rq->rt,NULL , i,NULL );#endif rq->sd =NULL ;
kernel/sched/deadline.c +0 -34
@@ -1500,40 +1500,6 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64if (!is_leftmost (dl_se, &rq->dl))resched_curr (rq); } - -/* -* The fair server (sole dl_server) does not account for real-time -* workload because it is running fair work. -*/ -if (dl_se == &rq->fair_server) -return ; - -#ifdef CONFIG_RT_GROUP_SCHED -/* -* Because -- for now -- we share the rt bandwidth, we need to -* account our runtime there too, otherwise actual rt tasks -* would be able to exceed the shared quota. -* -* Account to the root rt group for now. -* -* The solution we're working towards is having the RT groups scheduled -* using deadline servers -- however there's a few nasties to figure -* out before that can happen. -*/ -if (rt_bandwidth_enabled ()) { -struct rt_rq *rt_rq = &rq->rt; - -raw_spin_lock (&rt_rq->rt_runtime_lock); -/* -* We'll let actual RT tasks worry about the overflow here, we -* have our own CBS to keep us inline; only account when RT -* bandwidth is relevant. -*/ -if (sched_rt_bandwidth_account (rt_rq)) - rt_rq->rt_time += delta_exec; -raw_spin_unlock (&rt_rq->rt_runtime_lock); - } -#endif /* CONFIG_RT_GROUP_SCHED */ }/*
kernel/sched/debug.c +0 -6
@@ -890,12 +890,6 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)PU (rt_nr_running); -#ifdef CONFIG_RT_GROUP_SCHED -P (rt_throttled); -PN (rt_time); -PN (rt_runtime); -#endif -#undef PN#undef PU#undef P
kernel/sched/rt.c +26 -836
1,076 lines not on this page · raw message
kernel/sched/sched.h +3 -12
@@ -815,7 +815,7 @@ struct scx_rq {static inline int rt_bandwidth_enabled (void ) { -return sysctl_sched_rt_runtime >=0 ; +return 0 ; }/* RT IPI pull logic requires IRQ_WORK */
@@ -855,7 +855,7 @@ struct rt_rq {static inline bool rt_rq_is_runnable (struct rt_rq *rt_rq) { -return rt_rq->rt_queued && rt_rq->rt_nr_running; +return rt_rq->rt_nr_running; }/* Deadline class' related fields in a runqueue */
@@ -2572,7 +2572,7 @@ static inline bool sched_dl_runnable(struct rq *rq)static inline bool sched_rt_runnable (struct rq *rq) { -return rq->rt.rt_queued >0 ; +return rq->rt.rt_nr_running >0 ; }static inline bool sched_fair_runnable (struct rq *rq)
@@ -2683,9 +2683,6 @@ extern void resched_curr(struct rq *rq);extern void resched_curr_lazy (struct rq *rq);extern void resched_cpu (int cpu); -extern void init_rt_bandwidth (struct rt_bandwidth *rt_b,u64 period,u64 runtime); -extern bool sched_rt_bandwidth_account (struct rt_rq *rt_rq); -extern void init_dl_entity (struct sched_dl_entity *dl_se);extern void init_cfs_throttle_work (struct task_struct *p);
@@ -3072,12 +3069,8 @@ extern void set_rq_offline(struct rq *rq);extern bool sched_smp_initialized;#ifdef CONFIG_RT_GROUP_SCHED -#define rt_entity_is_task (rt_se) (!(rt_se)->my_q) -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) { -WARN_ON_ONCE (!rt_entity_is_task (rt_se)); - return container_of(rt_se, struct task_struct, rt); }
@@ -3102,8 +3095,6 @@ static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)return rt_rq->rq; }#else -#define rt_entity_is_task (rt_se) (1) -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) { return container_of(rt_se, struct task_struct, rt);
kernel/sched/syscalls.c +0 -13
@@ -626,19 +626,6 @@ int __sched_setscheduler(struct task_struct *p, change:if (user) { -#ifdef CONFIG_RT_GROUP_SCHED -/* -* Do not allow real-time tasks into groups that have no runtime -* assigned. -*/ -if (rt_group_sched_enabled () && -rt_bandwidth_enabled () &&rt_policy (policy) && -task_group (p)->rt_bandwidth.rt_runtime ==0 && - !task_group_is_autogroup (task_group (p))) { - retval = -EPERM ; - goto unlock; - } -#endif /* CONFIG_RT_GROUP_SCHED */ if (dl_bandwidth_enabled () &&dl_policy (policy) && !(attr->sched_flags &SCHED_FLAG_SUGOV )) {cpumask_t *span = rq->rd->span;
Signature · 1 line
2.51.0
Re: [RFC PATCH v4 05/28] sched/rt: Disable RT_GROUP_SCHED
Hello,
Quoted 18 lines · Yuri Andriaccio
Disable the old
RT_GROUP_SCHEDscheduler. Note that this does not completely remove all theRT_GROUP_SCHEDfunctionality, just unhooks it and removes most of the relevant functions. Some of theRT_GROUP_SCHEDfunctions are kept because they will be adapted for the HCBS scheduling.Most notably:
- Disable the initialization of the
rt_bandwidthfor group scheduling.
- Unhook any functionality forRT_GROUP_SCHEDin normalrt.ccode, leaving only non-group functionality.
- Remove group related field initialization ininit_rt_rq().
- Remove all the unhooked (and so unused) functions fromRT_GROUP_SCHED.
- Remove all allocation/deallocation code for rt-groups, always returning failure on allocation.
- Update inc/dec_rt_tasks active tasks' counters, as rt scheduling entities now only represent a single task, and not a group of tasks anymore.
Quoted 1 line
Signed-off-by Yuri Andriaccio <[email protected]>
Thanks,
Juri
RFC PATCH v4 6/28 sched/rt: Remove rq field in struct rt_rq
In reply to Yuri AndriaccioThe rq field now is just caching the pointer to the global runqueue of the given rt_rq, so it is unnecessary as the global runqueue can be retrieved in other ways.
--- kernel/sched/rt.c | 7 ++----- kernel/sched/sched.h | 19 +++++++++++++------ 2 files changed, 15 insertions(+), 11 deletions(-)
kernel/sched/rt.c +2 -5
@@ -100,10 +100,7 @@ void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq, struct sched_rt_entity *rt_se, int cpu, struct sched_rt_entity *parent) { - struct rq *rq = cpu_rq(cpu); - rt_rq->highest_prio.curr = MAX_RT_PRIO-1; - rt_rq->rq = rq; rt_rq->tg = tg; tg->rt_rq[cpu] = rt_rq;
@@ -183,7 +180,7 @@ static void pull_rt_task(struct rq *);static inline void rt_queue_push_tasks (struct rt_rq *rt_rq) { - struct rq *rq = container_of(rt_rq, struct rq, rt); +struct rq *rq =served_rq_of_rt_rq (rt_rq);if (!has_pushable_tasks (rt_rq))return ;
@@ -193,7 +190,7 @@ static inline void rt_queue_push_tasks(struct rt_rq *rt_rq)static inline void rt_queue_pull_task (struct rt_rq *rt_rq) { - struct rq *rq = container_of(rt_rq, struct rq, rt); +struct rq *rq =served_rq_of_rt_rq (rt_rq);queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); }
kernel/sched/sched.h +13 -6
@@ -3074,11 +3074,16 @@ static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se) return container_of(rt_se, struct task_struct, rt); } +static inline struct rq *served_rq_of_rt_rq (struct rt_rq *rt_rq) +{ +WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); + return container_of(rt_rq, struct rq, rt); +} +static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) {/* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */ -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; +return cpu_rq (served_rq_of_rt_rq (rt_rq)->cpu); }static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se)
@@ -3089,10 +3094,7 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) { -struct rt_rq *rt_rq = rt_se->rt_rq; - -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; +return rq_of_rt_rq (rt_se->rt_rq); }#else static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se)
@@ -3100,6 +3102,11 @@ static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se) return container_of(rt_se, struct task_struct, rt); } +static inline struct rq *served_rq_of_rt_rq (struct rt_rq *rt_rq) +{ + return container_of(rt_rq, struct rq, rt); +} +static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) { return container_of(rt_rq, struct rq, rt);
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v4 06/28] sched/rt: Remove rq field in struct rt_rq
Hello,
Quoted 3 lines · Yuri Andriaccio
The rq field now is just caching the pointer to the global runqueue of the given
rt_rq, so it is unnecessary as the global runqueue can be retrieved in other ways.
OK, but it's not actually removed, is it? Looks like it's re-purposed for later usage? If that's the case maybe $SUBJECT should say so (and maybe changelog point this out as well), just so we don't create confusion.
Thanks,
Juri
RFC PATCH v4 7/28 sched/rt: Introduce HCBS specific structs in task_group
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Each task_group manages a number of new objects:
- a sched_dl_entity/dl_server for each CPU
- a dl_bandwidth object to keep track of its allocated dl_bandwidth
--- kernel/sched/sched.h | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
kernel/sched/sched.h +20 -2
@@ -319,6 +319,13 @@ struct rt_bandwidth {unsigned int rt_period_active; }; +struct dl_bandwidth { +raw_spinlock_t dl_runtime_lock; +u64 dl_runtime; +u64 dl_period; +}; + +static inline int dl_bandwidth_enabled (void ) {return sysctl_sched_rt_runtime >=0 ;
@@ -492,10 +499,17 @@ struct task_group {#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED +/* +* Each task group manages a different scheduling entity per CPU, i.e. a +* different deadline server, and a runqueue per CPU. All the dl-servers +* share the same dl_bandwidth object. +*/ struct sched_rt_entity **rt_se; +struct sched_dl_entity **dl_se;struct rt_rq **rt_rq;struct rt_bandwidth rt_bandwidth; +struct dl_bandwidth dl_bandwidth;#endif struct scx_task_group scx;
@@ -845,12 +859,16 @@ struct rt_rq {raw_spinlock_t rt_runtime_lock;unsigned int rt_nr_boosted; - -struct rq *rq;/* this is always top-level rq, cache? */ #endif #ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */ #endif + +/* +* The cgroup's served runqueue if the rt_rq entity belongs to a cgroup, +* otherwise the top-level global runqueue. +*/ +struct rq *rq; };static inline bool rt_rq_is_runnable (struct rt_rq *rt_rq)
--
2.51.0
Re: [RFC PATCH v4 07/28] sched/rt: Introduce HCBS specific structs in task_group
Hello,
Quoted 5 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Each
task_groupmanages a number of new objects:
- a sched_dl_entity/dl_server for each CPU
- adl_bandwidthobject to keep track of its allocateddl_bandwidth
The changelog feels a little dry. I wonder if both changelog and maybe documentation (in-code and/or later documentation patch) can benifit from some kind of diagram depicting how these new and existing data structure interact with each other and what they are useful for.
Quoted 7 lines
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>
Anyway, the patch itself looks OK to me.
Thanks,
Juri
RFC PATCH v4 8/28 sched/core: Initialize HCBS specific structures.
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
- Update autogroups' creation/destruction to use the new data structures.
- Initialize the default bandwidth for rt-cgroups (sched_init).
- Initialize rt-scheduler's specific data structures for the root control group (sched_init).
- Change signature of init_tg_rt_entry to accept a rq pointer for the local runqueue and a dl_se pointer for the global runqueue / dl_server entity, update the function to initialize the newly added HCBS fields.
- Add utility functions to check (and get) if a rt_rq entity is connected to a rt-cgroup.
--- kernel/sched/autogroup.c | 4 ++-- kernel/sched/core.c | 11 +++++++++-- kernel/sched/deadline.c | 8 ++++++++ kernel/sched/rt.c | 11 ----------- kernel/sched/sched.h | 30 +++++++++++++++++++++++++++--- 5 files changed, 46 insertions(+), 18 deletions(-)
kernel/sched/autogroup.c +2 -2
@@ -52,7 +52,7 @@ static inline void autogroup_destroy(struct kref *kref)#ifdef CONFIG_RT_GROUP_SCHED /* We've redirected RT tasks to the root task group... */ - ag->tg->rt_se =NULL ; + ag->tg->dl_se =NULL ; ag->tg->rt_rq =NULL ;#endif sched_release_group (ag->tg);
@@ -109,7 +109,7 @@ static inline struct autogroup *autogroup_create(void)* the policy change to proceed. */ free_rt_sched_group (tg); - tg->rt_se = root_task_group.rt_se; + tg->dl_se = root_task_group.dl_se; tg->rt_rq = root_task_group.rt_rq;#endif /* CONFIG_RT_GROUP_SCHED */ tg->autogroup = ag;
kernel/sched/core.c +9 -2
@@ -8680,7 +8680,7 @@ void __init sched_init(void)scx_tg_init (&root_task_group);#endif /* CONFIG_EXT_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED - root_task_group.rt_se = (struct sched_rt_entity **)ptr; + root_task_group.dl_se = (struct sched_dl_entity **)ptr; ptr += nr_cpu_ids *sizeof (void **); root_task_group.rt_rq = (struct rt_rq **)ptr;
@@ -8691,6 +8691,11 @@ void __init sched_init(void)init_defrootdomain (); +#ifdef CONFIG_RT_GROUP_SCHED +init_dl_bandwidth (&root_task_group.dl_bandwidth, +global_rt_period (),global_rt_runtime ()); +#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_CGROUP_SCHED task_group_cache =KMEM_CACHE (task_group,0 );
@@ -8742,7 +8747,9 @@ void __init sched_init(void)* starts working after scheduler_running, which is not the case * yet. */ -init_tg_rt_entry (&root_task_group, &rq->rt,NULL , i,NULL ); + rq->rt.tg = &root_task_group; + root_task_group.rt_rq[i] = &rq->rt; + root_task_group.dl_se[i] =NULL ;#endif rq->sd =NULL ; rq->rd =NULL ;
kernel/sched/deadline.c +8 -0
@@ -513,6 +513,14 @@ static inline int is_leftmost(struct sched_dl_entity *dl_se, struct dl_rq *dl_rqstatic void init_dl_rq_bw_ratio (struct dl_rq *dl_rq); +void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime) +{ +raw_spin_lock_init (&dl_b->dl_runtime_lock); + dl_b->dl_period = period; + dl_b->dl_runtime = runtime; +} + +void init_dl_bw (struct dl_bw *dl_b) {raw_spin_lock_init (&dl_b->lock);
kernel/sched/rt.c +0 -11
@@ -96,17 +96,6 @@ void free_rt_sched_group(struct task_group *tg)return ; } -void init_tg_rt_entry (struct task_group *tg,struct rt_rq *rt_rq, -struct sched_rt_entity *rt_se,int cpu, -struct sched_rt_entity *parent) -{ - rt_rq->highest_prio.curr =MAX_RT_PRIO -1 ; - rt_rq->tg = tg; - - tg->rt_rq[cpu] = rt_rq; - tg->rt_se[cpu] = rt_se; -} -int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent) {if (!rt_group_sched_enabled ())
kernel/sched/sched.h +27 -3
@@ -601,9 +601,6 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);extern void unthrottle_cfs_rq (struct cfs_rq *cfs_rq);extern bool cfs_task_bw_constrained (struct task_struct *p); -extern void init_tg_rt_entry (struct task_group *tg,struct rt_rq *rt_rq, -struct sched_rt_entity *rt_se,int cpu, -struct sched_rt_entity *parent);extern int sched_group_set_rt_runtime (struct task_group *tg,long rt_runtime_us);extern int sched_group_set_rt_period (struct task_group *tg,u64 rt_period_us);extern long sched_group_rt_runtime (struct task_group *tg);
@@ -2701,6 +2698,7 @@ extern void resched_curr(struct rq *rq);extern void resched_curr_lazy (struct rq *rq);extern void resched_cpu (int cpu); +void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime);extern void init_dl_entity (struct sched_dl_entity *dl_se);extern void init_cfs_throttle_work (struct task_struct *p);
@@ -3114,6 +3112,22 @@ static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se) {return rq_of_rt_rq (rt_se->rt_rq); } + +static inline int is_dl_group (struct rt_rq *rt_rq) +{ +return rt_rq->tg != &root_task_group; +} + +/* +* Return the scheduling entity of this group of tasks. +*/ +static inline struct sched_dl_entity *dl_group_of (struct rt_rq *rt_rq) +{ +if (WARN_ON_ONCE (!is_dl_group (rt_rq))) +return NULL ; + +return rt_rq->tg->dl_se[served_rq_of_rt_rq (rt_rq)->cpu]; +}#else static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) {
@@ -3143,6 +3157,16 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)return &rq->rt; } + +static inline int is_dl_group (struct rt_rq *rt_rq) +{ +return 0 ; +} + +static inline struct sched_dl_entity *dl_group_of (struct rt_rq *rt_rq) +{ +return NULL ; +}#endif DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq,
Signature · 1 line
2.51.0
Re: [RFC PATCH v4 08/28] sched/core: Initialize HCBS specific structures.
Hello,
Quoted 11 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
- Update autogroups' creation/destruction to use the new data structures.
- Initialize the default bandwidth for rt-cgroups (sched_init).
- Initialize rt-scheduler's specific data structures for the root control group (sched_init).
- Change signature ofinit_tg_rt_entryto accept a rq pointer for the local runqueue and adl_sepointer for the global runqueue /dl_serverentity, update the function to initialize the newly added HCBS fields.
- Add utility functions to check (and get) if art_rqentity is connected to a rt-cgroup.
I personally prefer a more discursive style, but yeah I'm definitely nitpicking, so feel free to ignore. :)
Quoted 7 lines
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>
Thanks,
Juri
RFC PATCH v4 9/28 sched/deadline: Add dl_init_tg
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add dl_init_tg to initialize and/or update a rt-cgroup dl_server and to also account the allocated bandwidth. This function is currently unhooked and will be later used to allocate bandwidth to rt-cgroups.
Add lock guard for raw_spin_rq_lock_irq for cleaner code.
--- kernel/sched/deadline.c | 31 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 5 +++++ 2 files changed, 36 insertions(+)
kernel/sched/deadline.c +31 -0
@@ -340,6 +340,37 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)cancel_dl_timer (dl_se, &dl_se->inactive_timer); } +#ifdef CONFIG_RT_GROUP_SCHED +void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period) +{ + struct rq *rq = container_of(dl_se->dl_rq, struct rq, dl); +int is_active; +u64 new_bw; + +guard (raw_spin_rq_lock_irq)(rq); + is_active = dl_se->my_q->rt.rt_nr_running >0 ; + +update_rq_clock (rq); +dl_server_stop (dl_se); + + new_bw =to_ratio (rt_period, rt_runtime); +dl_rq_change_utilization (rq, dl_se, new_bw); + + dl_se->dl_runtime = rt_runtime; + dl_se->dl_deadline = rt_period; + dl_se->dl_period = rt_period; + + dl_se->runtime =0 ; + dl_se->deadline =0 ; + + dl_se->dl_bw = new_bw; + dl_se->dl_density = new_bw; + +if (is_active) +dl_server_start (dl_se); +} +#endif +static void dl_change_utilization (struct task_struct *p,u64 new_bw) {WARN_ON_ONCE (p->dl.flags &SCHED_FLAG_SUGOV );
kernel/sched/sched.h +5 -0
@@ -419,6 +419,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,struct rq *served_rq, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void ); +extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period);extern void dl_server_update_idle_time (struct rq *rq,struct task_struct *p);
@@ -1922,6 +1923,10 @@ DEFINE_LOCK_GUARD_1(rq_lock_irqsave, struct rq,rq_unlock_irqrestore (_T->lock, &_T->rf), struct rq_flags rf) +DEFINE_LOCK_GUARD_1(raw_spin_rq_lock_irq, struct rq, + raw_spin_rq_lock_irq(_T->lock), + raw_spin_rq_unlock_irq(_T->lock)) + static inline struct rq *this_rq_lock_irq(struct rq_flags *rf) __acquires(rq->lock) {
Signature · 1 line
2.51.0
Re: [RFC PATCH v4 09/28] sched/deadline: Add dl_init_tg
Hello,
Quoted 8 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Add
dl_init_tgto initialize and/or update a rt-cgroupdl_serverand to also account the allocated bandwidth. This function is currently unhooked and will be later used to allocate bandwidth to rt-cgroups.Add lock guard for
raw_spin_rq_lock_irqfor cleaner code.
Quoted 7 lines
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>
Thanks,
Juri
RFC PATCH v4 10/28 sched/rt: Add {alloc/free}_rt_sched_group
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
- Add allocation and deallocation code for rt-cgroups.
- Declare dl_server specific functions (only skeleton, but no implementation yet), needed by the deadline servers to be called when trying to schedule.
--- kernel/sched/rt.c | 125 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 123 insertions(+), 2 deletions(-)
kernel/sched/rt.c +123 -2
@@ -92,19 +92,140 @@ void unregister_rt_sched_group(struct task_group *tg)void free_rt_sched_group (struct task_group *tg) { +int i; +unsigned long flags; +if (!rt_group_sched_enabled ())return ; + +if (!tg->dl_se || !tg->rt_rq) +return ; + +for_each_possible_cpu (i) { +if (!tg->dl_se[i] || !tg->rt_rq[i]) +continue ; + +/* +* Shutdown the dl_server and free it +* +* Since the dl timer is going to be cancelled, +* we risk to never decrease the running bw... +* Fix this issue by changing the group runtime +* to 0 immediately before freeing it. +*/ +dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); + +raw_spin_rq_lock_irqsave (cpu_rq (i), flags); +hrtimer_cancel (&tg->dl_se[i]->dl_timer); +raw_spin_rq_unlock_irqrestore (cpu_rq (i), flags); +kfree (tg->dl_se[i]); + +/* Free the local per-cpu runqueue */ +kfree (served_rq_of_rt_rq (tg->rt_rq[i])); + } + +kfree (tg->rt_rq); +kfree (tg->dl_se); +} + +static struct task_struct *rt_server_pick (struct sched_dl_entity *dl_se) +{ +return NULL ; +} + +static inline void __rt_rq_free (struct rt_rq **rt_rq) +{ +int i; + +for_each_possible_cpu (i) { +kfree (served_rq_of_rt_rq (rt_rq[i])); + } + +kfree (rt_rq); +} + +DEFINE_FREE(rt_rq_free, struct rt_rq **, if (_T) __rt_rq_free(_T)) + +static inline void __dl_se_free (struct sched_dl_entity **dl_se) +{ +int i; + +for_each_possible_cpu (i) { +kfree (dl_se[i]); + } + +kfree (dl_se); } +DEFINE_FREE(dl_se_free, struct sched_dl_entity **, if (_T) __dl_se_free(_T)) +int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent) { +/* Instantiate automatic cleanup in event of kalloc fail */ + struct rt_rq **tg_rt_rq __free(rt_rq_free) = NULL; + struct sched_dl_entity *dl_se, **tg_dl_se __free(dl_se_free) = NULL; +struct rq *s_rq; +int i; +if (!rt_group_sched_enabled ())return 1 ; +/* Allocate all necessary resources beforehand */ + tg_rt_rq =kcalloc (nr_cpu_ids,sizeof (struct rt_rq *),GFP_KERNEL ); +if (!tg_rt_rq) +return 0 ; + + tg_dl_se =kcalloc (nr_cpu_ids, +sizeof (struct sched_dl_entity *),GFP_KERNEL ); +if (!tg_dl_se) +return 0 ; + +for_each_possible_cpu (i) { + s_rq =kzalloc_node (sizeof (struct rq ), +GFP_KERNEL ,cpu_to_node (i)); +if (!s_rq) +return 0 ; + + dl_se =kzalloc_node (sizeof (struct sched_dl_entity ), +GFP_KERNEL ,cpu_to_node (i)); +if (!dl_se) { +kfree (s_rq); +return 0 ; + } + + tg_rt_rq[i] = &s_rq->rt; + tg_dl_se[i] = dl_se; + } + + tg->rt_rq =no_free_ptr (tg_rt_rq); + tg->dl_se =no_free_ptr (tg_dl_se); + +/* Initialize the allocated resources now. */ +init_dl_bandwidth (&tg->dl_bandwidth,0 ,0 ); + +for_each_possible_cpu (i) { + s_rq =served_rq_of_rt_rq (tg->rt_rq[i]); + dl_se = tg->dl_se[i]; + +init_rt_rq (&s_rq->rt); + s_rq->cpu = i; + s_rq->rt.tg = tg; + +init_dl_entity (dl_se); + dl_se->dl_runtime = tg->dl_bandwidth.dl_runtime; + dl_se->dl_period = tg->dl_bandwidth.dl_period; + dl_se->dl_deadline = dl_se->dl_period; + dl_se->dl_bw =to_ratio (dl_se->dl_period, dl_se->dl_runtime); + dl_se->dl_density =to_ratio (dl_se->dl_period, dl_se->dl_runtime); + dl_se->dl_server =1 ; + +dl_server_init (dl_se, &cpu_rq (i)->dl, s_rq, rt_server_pick); + } +return 1 ; } -#else /* !CONFIG_RT_GROUP_SCHED: */ +#else /* !CONFIG_RT_GROUP_SCHED */ void unregister_rt_sched_group (struct task_group *tg) { }
@@ -114,7 +235,7 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent) {return 1 ; } -#endif /* !CONFIG_RT_GROUP_SCHED */ +#endif /* CONFIG_RT_GROUP_SCHED */ static inline bool need_pull_rt_task (struct rq *rq,struct task_struct *prev) {
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v4 10/28] sched/rt: Add {alloc/free}_rt_sched_group
Hello,
Quoted 15 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
- Add allocation and deallocation code for rt-cgroups.
- Declaredl_serverspecific functions (only skeleton, but no implementation yet), needed by the deadline servers to be called when trying to schedule.Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>---
...
Quoted 36 lines
kernel/sched/rt.c +29
-0+static inline void __dl_se_free (struct sched_dl_entity **dl_se) +{ +int i; + +for_each_possible_cpu (i) { +kfree (dl_se[i]); + } + +kfree (dl_se); } +DEFINE_FREE(dl_se_free, struct sched_dl_entity **, if (_T) __dl_se_free(_T)) +int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent) { +/* Instantiate automatic cleanup in event of kalloc fail */ + struct rt_rq **tg_rt_rq __free(rt_rq_free) = NULL; + struct sched_dl_entity *dl_se, **tg_dl_se __free(dl_se_free) = NULL; +struct rq *s_rq; +int i; +if (!rt_group_sched_enabled ())return 1 ; +/* Allocate all necessary resources beforehand */ + tg_rt_rq =kcalloc (nr_cpu_ids,sizeof (struct rt_rq *),GFP_KERNEL ); +if (!tg_rt_rq) +return 0 ; + + tg_dl_se =kcalloc (nr_cpu_ids, +sizeof (struct sched_dl_entity *),GFP_KERNEL ); +if (!tg_dl_se) +return 0 ; + +for_each_possible_cpu (i) { + s_rq = kzalloc_node(sizeof(struct rq),
kernel/sched/rt.c void unregister_rt_sched_group(struct task_group *tg) · line 194
Quoted 10 lines
kernel/sched/rt.c +10
-0+ GFP_KERNEL, cpu_to_node(i)); +if (!s_rq) +return 0 ; + + dl_se =kzalloc_node (sizeof (struct sched_dl_entity ), +GFP_KERNEL ,cpu_to_node (i)); +if (!dl_se) { +kfree (s_rq); +return 0 ; + }
Would it be possible and make sense to use the __free() approach in the above two cases as well?
Thanks,
Juri
RFC PATCH v4 11/28 sched/deadline: Account rt-cgroups bandwidth in deadline tasks schedulability tests.
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Account the rt-cgroups hierarchy's reserved bandwidth in the schedulability test of deadline entities. This mechanism allows to completely reserve portion of the rt-bandwidth to rt-cgroups even if they do not use all of it.
Account for the rt-cgroups' reserved bandwidth also when changing the total dedicated bandwidth for real time tasks.
--- kernel/sched/deadline.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
kernel/sched/deadline.c +18 -3
@@ -213,8 +213,15 @@ void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)static inline bool __dl_overflow (struct dl_bw *dl_b,unsigned long cap,u64 old_bw,u64 new_bw) { +u64 dl_groups_root =0 ; + +#ifdef CONFIG_RT_GROUP_SCHED + dl_groups_root =to_ratio (root_task_group.dl_bandwidth.dl_period, + root_task_group.dl_bandwidth.dl_runtime); +#endif return dl_b->bw !=-1 && -cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw; +cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw + +cap_scale (dl_groups_root, cap); } static inline
@@ -3147,10 +3154,16 @@ int sched_dl_global_validate(void)u64 period =global_rt_period ();u64 new_bw =to_ratio (period, runtime);u64 cookie = ++dl_cookie; +u64 dl_groups_root =0 ;struct dl_bw *dl_b; -int cpu, cpus, ret =0 ; +int cpu, cap, cpus, ret =0 ;unsigned long flags; +#ifdef CONFIG_RT_GROUP_SCHED + dl_groups_root =to_ratio (root_task_group.dl_bandwidth.dl_period, + root_task_group.dl_bandwidth.dl_runtime); +#endif +/* * Here we want to check the bandwidth not being set to some * value smaller than the currently allocated bandwidth in
@@ -3163,10 +3176,12 @@ int sched_dl_global_validate(void) goto next; dl_b =dl_bw_of (cpu); + cap =dl_bw_capacity (cpu); cpus =dl_bw_cpus (cpu);raw_spin_lock_irqsave (&dl_b->lock, flags); -if (new_bw * cpus < dl_b->total_bw) +if (new_bw * cpus < dl_b->total_bw + +cap_scale (dl_groups_root, cap)) ret = -EBUSY ;raw_spin_unlock_irqrestore (&dl_b->lock, flags);
--
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v4 11/28] sched/deadline: Account rt-cgroups bandwidth in deadline tasks schedulability tests.
Hello,
Quoted 46 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Account the rt-cgroups hierarchy's reserved bandwidth in the schedulability test of deadline entities. This mechanism allows to completely reserve portion of the rt-bandwidth to rt-cgroups even if they do not use all of it.
Account for the rt-cgroups' reserved bandwidth also when changing the total dedicated bandwidth for real time tasks.
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>--- kernel/sched/deadline.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)kernel/sched/deadline.c +8
-1@@ -213,8 +213,15 @@ void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)static inline bool __dl_overflow (struct dl_bw *dl_b,unsigned long cap,u64 old_bw,u64 new_bw) { +u64 dl_groups_root =0 ; + +#ifdef CONFIG_RT_GROUP_SCHED + dl_groups_root =to_ratio (root_task_group.dl_bandwidth.dl_period, + root_task_group.dl_bandwidth.dl_runtime); +#endif return dl_b->bw !=-1 && -cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw; +cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw + +cap_scale (dl_groups_root, cap); } static inline@@ -3147,10 +3154,16 @@ int sched_dl_global_validate(void)u64 period =global_rt_period ();u64 new_bw =to_ratio (period, runtime);u64 cookie = ++dl_cookie;
kernel/sched/deadline.c int sched_dl_global_validate(void) · line 3165
Quoted 10 lines
kernel/sched/deadline.c +6
-1+u64 dl_groups_root =0 ;struct dl_bw *dl_b; -int cpu, cpus, ret =0 ; +int cpu, cap, cpus, ret =0 ;unsigned long flags; +#ifdef CONFIG_RT_GROUP_SCHED + dl_groups_root =to_ratio (root_task_group.dl_bandwidth.dl_period, + root_task_group.dl_bandwidth.dl_runtime); +#endif
Can we move these ifdefs in some header and use helper functions instead?
Thanks,
Juri
RFC PATCH v4 12/28 sched/rt: Implement dl-server operations for rt-cgroups.
In reply to Yuri Andriaccio- Implement rt_server_pick, the callback that deadline servers use to pick a task to schedule.
- rt_server_pick(): pick the next runnable rt task and tell the scheduler that it is going to be scheduled next.
- Let enqueue/dequeue_task_rt function start/stop the attached deadline server when the first/last task is enqueued/dequeued on a specific rq/server.
- Change update_curr_rt to perform a deadline server update if the updated task is served by non-root group.
- Update inc/dec_dl_tasks to account the number of active tasks in the local runqueue for rt-cgroups servers, as their local runqueue is different from the global runqueue, and thus when a rt-group server is activated/deactivated, the number of served tasks' must be added/removed. This uses nr_running to be compatible with future dl-server interfaces.
- Update inc/dec_rt_prio_smp to change a rq's cpupri only if the rt_rq is the global runqueue, since cgroups are scheduled via their dl-server priority.
- Update inc/dec_rt_tasks to account for waking/sleeping tasks on the global runqueue, when the task runs on the root cgroup, or its local dl server is active. The accounting is not done when servers are throttled, as they will add/sub the number of tasks running when they get enqueued/dequeued. For rt cgroups, account for the number of active tasks in the nr_running field of the local runqueue (add/sub_nr_running), as this number is used when a dl server is enqueued/dequeued.
- Update set_task_rq to record the dl_rq, tracking which deadline server manages a task.
- Update set_task_rq to not use the parent field anymore, as it is unused by this patchset's code. Remove the unused parent field from sched_rt_entity.
--- include/linux/sched.h | 1 - kernel/sched/deadline.c | 8 +++++ kernel/sched/rt.c | 68 ++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 8 ++++- 4 files changed, 79 insertions(+), 6 deletions(-)
include/linux/sched.h +0 -1
@@ -629,7 +629,6 @@ struct sched_rt_entity {struct sched_rt_entity *back;#ifdef CONFIG_RT_GROUP_SCHED -struct sched_rt_entity *parent;/* rq on which this entity is (to be) queued: */ struct rt_rq *rt_rq;/* rq "owned" by this entity/group: */
kernel/sched/deadline.c +8 -0
@@ -1847,6 +1847,10 @@ void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)if (!dl_server (dl_se))add_nr_running (rq_of_dl_rq (dl_rq),1 ); +else if (rq_of_dl_se (dl_se) != dl_se->my_q) { +WARN_ON (dl_se->my_q->rt.rt_nr_running != dl_se->my_q->nr_running); +add_nr_running (rq_of_dl_rq (dl_rq), dl_se->my_q->nr_running); + }inc_dl_deadline (dl_rq, deadline); }
@@ -1859,6 +1863,10 @@ void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)if (!dl_server (dl_se))sub_nr_running (rq_of_dl_rq (dl_rq),1 ); +else if (rq_of_dl_se (dl_se) != dl_se->my_q) { +WARN_ON (dl_se->my_q->rt.rt_nr_running != dl_se->my_q->nr_running); +sub_nr_running (rq_of_dl_rq (dl_rq), dl_se->my_q->nr_running); + }dec_dl_deadline (dl_rq, dl_se->deadline); }
kernel/sched/rt.c +64 -4
125 lines not on this page · raw message
kernel/sched/sched.h +7 -1
@@ -2203,7 +2203,7 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)if (!rt_group_sched_enabled ()) tg = &root_task_group; p->rt.rt_rq = tg->rt_rq[cpu]; - p->rt.parent = tg->rt_se[cpu]; + p->dl.dl_rq = &cpu_rq (cpu)->dl;#endif /* CONFIG_RT_GROUP_SCHED */ }
@@ -2750,6 +2750,9 @@ static inline void add_nr_running(struct rq *rq, unsigned count)unsigned prev_nr = rq->nr_running; rq->nr_running = prev_nr + count; +if (rq !=cpu_rq (rq->cpu)) +return ; +if (trace_sched_update_nr_running_tp_enabled ()) {call_trace_sched_update_nr_running (rq, count); }
@@ -2763,6 +2766,9 @@ static inline void add_nr_running(struct rq *rq, unsigned count)static inline void sub_nr_running (struct rq *rq,unsigned count) { rq->nr_running -= count; +if (rq !=cpu_rq (rq->cpu)) +return ; +if (trace_sched_update_nr_running_tp_enabled ()) {call_trace_sched_update_nr_running (rq, -count); }
--
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v4 12/28] sched/rt: Implement dl-server operations for rt-cgroups.
Hello,
Quoted 47 lines · Yuri Andriaccio
- Implement
rt_server_pick, the callback that deadline servers use to pick a task to schedule.
-rt_server_pick(): pick the next runnable rt task and tell the scheduler that it is going to be scheduled next.- Let enqueue/dequeue_task_rt function start/stop the attached deadline server when the first/last task is enqueued/dequeued on a specific rq/server.
- Change
update_curr_rtto perform a deadline server update if the updated task is served by non-root group.- Update inc/dec_dl_tasks to account the number of active tasks in the local runqueue for rt-cgroups servers, as their local runqueue is different from the global runqueue, and thus when a rt-group server is activated/deactivated, the number of served tasks' must be added/removed. This uses
nr_runningto be compatible with future dl-server interfaces.- Update inc/dec_rt_prio_smp to change a rq's cpupri only if the
rt_rqis the global runqueue, since cgroups are scheduled via their dl-server priority.- Update inc/dec_rt_tasks to account for waking/sleeping tasks on the global runqueue, when the task runs on the root cgroup, or its local dl server is active. The accounting is not done when servers are throttled, as they will add/sub the number of tasks running when they get enqueued/dequeued. For rt cgroups, account for the number of active tasks in the
nr_runningfield of the local runqueue (add/sub_nr_running), as this number is used when a dl server is enqueued/dequeued.- Update
set_task_rqto record thedl_rq, tracking which deadline server manages a task.- Update
set_task_rqto not use the parent field anymore, as it is unused by this patchset's code. Remove the unused parent field fromsched_rt_entity.Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by luca abeni <[email protected]>Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>---
...
kernel/sched/rt.c enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags) · line 773
Quoted 7 lines
/*
kernel/sched/rt.c +2
-0@@ -725,6 +769,14 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)check_schedstat_required ();update_stats_wait_start_rt (rt_rq_of_se (rt_se), rt_se); +/* Task arriving in an idle group of tasks. */ + if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) &&
^^^
Shall we be using rt_group_sched_enabled() here and elsewhere instead?
Thanks,
Juri
RFC PATCH v4 13/28 sched/rt: Update task event callbacks for HCBS scheduling
In reply to Yuri AndriaccioUpdate wakeup_preempt_rt, switched_{from/to}_rt and prio_changed_rt with rt-cgroup's specific preemption rules:
- In wakeup_preempt_rt(), whenever a task wakes up, it must be checked if it is served by a deadline server or it lives on the global runqueue. Preemption rules (as documented in the function), change based on the current task and woken task runqueue:
- If both tasks are FIFO/RR tasks on the global runqueue, or the same cgroup, run as normal.
- If woken is inside a cgroup, but curr is a FIFO task on the global runqueue, always preempt. If curr is a DEADLINE task, check if the dl server preempts curr.
- If both tasks are FIFO/RR tasks in served but different groups, check whether the woken server preempts the current server.
- In switched_from_rt(), perform a pull only on the global runqueue, and do nothing if the task is inside a group. This will change when migrations will be added.
- In switched_to_rt(), queue a push only on the global runqueue, while perform a priority check when the task switching is inside a group. This will change also when migrations will be added.
- In prio_changed_rt(), queue a pull only on the global runqueue, if the task is not queued. If the task is queued, run preemption checks only if both the prio changed task and curr are in the same cgroup.
Update sched_rt_can_attach() to check if a task can be attached to a given cgroup. For now the check only consists in checking if the group has non-zero bandwidth. Remove the tsk argument from sched_rt_can_attach, as it is unused.
Change cpu_cgroup_can_attach() to check if the attachee is a FIFO/RR task before attaching it to a cgroup.
Update __sched_setscheduler() to perform checks when trying to switch to FIFO/RR for a task inside a cgroup, as the group needs to have runtime allocated.
Update task_is_throttled_rt() for SCHED_CORE, returning the is_throttled value of the server if present, while global rt-tasks are never throttled.
--- kernel/sched/core.c | 2 +- kernel/sched/rt.c | 104 +++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 2 +- kernel/sched/syscalls.c | 13 +++++ 4 files changed, 108 insertions(+), 13 deletions(-)
kernel/sched/core.c +1 -1
@@ -9318,7 +9318,7 @@ static int cpu_cgroup_can_attach(struct cgroup_taskset *tset) goto scx_check;cgroup_taskset_for_each (task, css, tset) { -if (!sched_rt_can_attach (css_tg (css), task)) +if (rt_task (task) && !sched_rt_can_attach (css_tg (css)))return -EINVAL ; } scx_check:
kernel/sched/rt.c +93 -11
182 lines not on this page · raw message
kernel/sched/sched.h +1 -1
@@ -606,7 +606,7 @@ extern int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us)extern int sched_group_set_rt_period (struct task_group *tg,u64 rt_period_us);extern long sched_group_rt_runtime (struct task_group *tg);extern long sched_group_rt_period (struct task_group *tg); -extern int sched_rt_can_attach (struct task_group *tg,struct task_struct *tsk); +extern int sched_rt_can_attach (struct task_group *tg);extern struct task_group *sched_create_group (struct task_group *parent); extern void sched_online_group(struct task_group *tg,
kernel/sched/syscalls.c +13 -0
@@ -626,6 +626,19 @@ int __sched_setscheduler(struct task_struct *p, change:if (user) { +#ifdef CONFIG_RT_GROUP_SCHED +/* +* Do not allow real-time tasks into groups that have no runtime +* assigned. +*/ +if (rt_group_sched_enabled () && +dl_bandwidth_enabled () &&rt_policy (policy) && + !sched_rt_can_attach (task_group (p)) && + !task_group_is_autogroup (task_group (p))) { + retval = -EPERM ; + goto unlock; + } +#endif if (dl_bandwidth_enabled () &&dl_policy (policy) && !(attr->sched_flags &SCHED_FLAG_SUGOV )) {cpumask_t *span = rq->rd->span;
--
2.51.0
kernel test robot
build botRe: [RFC PATCH v4 13/28] sched/rt: Update task event callbacks for HCBS scheduling
kernel test robot noticed the following build errors:
Hi Yuri,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on linus/master]
[also build test ERROR on v6.18]
[cannot apply to tip/sched/core tip/master peterz-queue/sched/core tip/auto-latest next-20251211] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Yuri-Andriaccio/sched-deadline-Distinct-between-dl_rq-and-my_q/20251201-214223 base: linus/master patch link: https://lore.kernel.org/r/20251201124205.11169-14-yurand2000%40gmail.com patch subject: [RFC PATCH v4 13/28] sched/rt: Update task event callbacks for HCBS scheduling config: x86_64-randconfig-121-20251211 (https://download.01.org/0day-ci/archive/20251211/[email protected]/config) compiler: gcc-14 (Debian 14.2.0-19) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251211/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from kernel/sched/build_policy.c:52:
kernel/sched/rt.c: In function 'prio_changed_rt':
>> kernel/sched/rt.c:1860:40: error: 'struct rt_rq' has no member named 'tg'
1860 | rt_rq_of_se(&p->rt)->tg != rt_rq_of_se(&rq->curr->rt)->tg)
| ^~
kernel/sched/rt.c:1860:74: error: 'struct rt_rq' has no member named 'tg'
1860 | rt_rq_of_se(&p->rt)->tg != rt_rq_of_se(&rq->curr->rt)->tg)
| ^~vim +1860 kernel/sched/rt.c
1826
1827 /*
1828 * Priority of the task has changed. This may cause
1829 * us to initiate a push or pull.
1830 */
1831 static void
1832 prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)
1833 {
1834 struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
1835
1836 if (!task_on_rq_queued(p))
1837 return;
1838
1839 if (task_current_donor(rq, p)) {
1840 /*
1841 * If our priority decreases while running, we
1842 * may need to pull tasks to this runqueue.
1843 */
1844 if (!IS_ENABLED(CONFIG_RT_GROUP_SCHED) && oldprio < p->prio)
1845 rt_queue_pull_task(rt_rq);
1846
1847 /*
1848 * If there's a higher priority task waiting to run
1849 * then reschedule.
1850 */
1851 if (p->prio > rt_rq->highest_prio.curr)
1852 resched_curr(rq);
1853 } else {
1854 /*
1855 * This task is not running, thus we check against the currently
1856 * running task for preemption. We can preempt only if both tasks are
1857 * in the same cgroup or on the global runqueue.
1858 */
1859 if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) &&
> 1860 rt_rq_of_se(&p->rt)->tg != rt_rq_of_se(&rq->curr->rt)->tg)
1861 return;
1862
1863 /*
1864 * This task is not running, but if it is
1865 * greater than the current running task
1866 * then reschedule.
1867 */
1868 if (p->prio < rq->donor->prio)
1869 resched_curr(rq);
1870 }
1871 }
1872
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH v4 13/28] sched/rt: Update task event callbacks for HCBS scheduling
In reply to Yuri Andriacciokernel test robot noticed the following build warnings:
Hi Yuri,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on linus/master]
[also build test WARNING on v6.18]
[cannot apply to tip/sched/core tip/master peterz-queue/sched/core tip/auto-latest next-20251211] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Yuri-Andriaccio/sched-deadline-Distinct-between-dl_rq-and-my_q/20251201-214223 base: linus/master patch link: https://lore.kernel.org/r/20251201124205.11169-14-yurand2000%40gmail.com patch subject: [RFC PATCH v4 13/28] sched/rt: Update task event callbacks for HCBS scheduling config: x86_64-randconfig-122-20251211 (https://download.01.org/0day-ci/archive/20251211/[email protected]/config) compiler: gcc-12 (Debian 12.4.0-5) 12.4.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251211/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
kernel/sched/build_policy.c: note: in included file:
kernel/sched/rt.c:1679:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/rt.c:1679:25: sparse: struct task_struct *
kernel/sched/rt.c:1679:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/rt.c:450:39: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/rt.c:450:39: sparse: expected struct task_struct *donor
kernel/sched/rt.c:450:39: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/rt.c:839:31: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/rt.c:839:31: sparse: expected struct task_struct *p
kernel/sched/rt.c:839:31: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/rt.c:1191:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@
kernel/sched/rt.c:1191:9: sparse: expected struct sched_domain *[assigned] sd
kernel/sched/rt.c:1191:9: sparse: got struct sched_domain [noderef] __rcu *parent
kernel/sched/rt.c:858:14: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct task_struct *curr @@ got struct task_struct [noderef] __rcu * @@
kernel/sched/rt.c:858:14: sparse: expected struct task_struct *curr
kernel/sched/rt.c:858:14: sparse: got struct task_struct [noderef] __rcu *
kernel/sched/rt.c:859:15: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu * @@
kernel/sched/rt.c:859:15: sparse: expected struct task_struct *donor
kernel/sched/rt.c:859:15: sparse: got struct task_struct [noderef] __rcu *
kernel/sched/rt.c:920:45: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/rt.c:920:45: sparse: expected struct task_struct *p
kernel/sched/rt.c:920:45: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/rt.c:962:39: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/rt.c:962:39: sparse: expected struct task_struct *donor
kernel/sched/rt.c:962:39: sparse: got struct task_struct [noderef] __rcu *donor
>> kernel/sched/rt.c:988:40: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct sched_rt_entity *rt_se @@ got struct sched_rt_entity [noderef] __rcu * @@ kernel/sched/rt.c:988:40: sparse: expected struct sched_rt_entity *rt_se kernel/sched/rt.c:988:40: sparse: got struct sched_rt_entity [noderef] __rcu * kernel/sched/rt.c:989:57: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct sched_rt_entity *rt_se @@ got struct sched_rt_entity [noderef] __rcu * @@ kernel/sched/rt.c:989:57: sparse: expected struct sched_rt_entity *rt_se kernel/sched/rt.c:989:57: sparse: got struct sched_rt_entity [noderef] __rcu * kernel/sched/rt.c:990:39: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *curr @@ kernel/sched/rt.c:990:39: sparse: expected struct task_struct *p kernel/sched/rt.c:990:39: sparse: got struct task_struct [noderef] __rcu *curr >> kernel/sched/rt.c:991:28: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_dl_entity *[assigned] curr_dl_se @@ got struct sched_dl_entity [noderef] __rcu * @@
kernel/sched/rt.c:991:28: sparse: expected struct sched_dl_entity *[assigned] curr_dl_se
kernel/sched/rt.c:991:28: sparse: got struct sched_dl_entity [noderef] __rcu *
kernel/sched/rt.c:1032:64: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/rt.c:1032:64: sparse: expected struct task_struct *tsk
kernel/sched/rt.c:1032:64: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/rt.c:1359:40: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *task @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/rt.c:1359:40: sparse: expected struct task_struct *task
kernel/sched/rt.c:1359:40: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/rt.c:1382:13: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/rt.c:1382:13: sparse: struct task_struct *
kernel/sched/rt.c:1382:13: sparse: struct task_struct [noderef] __rcu *
kernel/sched/rt.c:1730:54: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/rt.c:1730:54: sparse: expected struct task_struct *tsk
kernel/sched/rt.c:1730:54: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/rt.c:1732:40: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/rt.c:1732:40: sparse: expected struct task_struct *p
kernel/sched/rt.c:1732:40: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/rt.c:1732:62: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/rt.c:1732:62: sparse: expected struct task_struct *p
kernel/sched/rt.c:1732:62: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/rt.c:1860:63: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct sched_rt_entity *rt_se @@ got struct sched_rt_entity [noderef] __rcu * @@
kernel/sched/rt.c:1860:63: sparse: expected struct sched_rt_entity *rt_se
kernel/sched/rt.c:1860:63: sparse: got struct sched_rt_entity [noderef] __rcu *
kernel/sched/build_policy.c: note: in included file:
kernel/sched/deadline.c:2701:23: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:2701:23: sparse: expected struct task_struct *p
kernel/sched/deadline.c:2701:23: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:2711:13: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/deadline.c:2711:13: sparse: struct task_struct *
kernel/sched/deadline.c:2711:13: sparse: struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:2817:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/deadline.c:2817:25: sparse: struct task_struct *
kernel/sched/deadline.c:2817:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:2319:42: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct sched_dl_entity const *b @@ got struct sched_dl_entity [noderef] __rcu * @@
kernel/sched/deadline.c:2319:42: sparse: expected struct sched_dl_entity const *b
kernel/sched/deadline.c:2319:42: sparse: got struct sched_dl_entity [noderef] __rcu *
kernel/sched/deadline.c:2329:38: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/deadline.c:2329:38: sparse: expected struct task_struct *tsk
kernel/sched/deadline.c:2329:38: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/deadline.c:1241:32: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/deadline.c:1241:32: sparse: expected struct task_struct *p
kernel/sched/deadline.c:1241:32: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/deadline.c:1241:71: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct sched_dl_entity const *b @@ got struct sched_dl_entity [noderef] __rcu * @@
kernel/sched/deadline.c:1241:71: sparse: expected struct sched_dl_entity const *b
kernel/sched/deadline.c:1241:71: sparse: got struct sched_dl_entity [noderef] __rcu *
kernel/sched/deadline.c:1339:23: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:1339:23: sparse: expected struct task_struct *p
kernel/sched/deadline.c:1339:23: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:1604:24: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/deadline.c:1604:24: sparse: expected struct task_struct *p
kernel/sched/deadline.c:1604:24: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/deadline.c:1604:63: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct sched_dl_entity const *b @@ got struct sched_dl_entity [noderef] __rcu * @@
kernel/sched/deadline.c:1604:63: sparse: expected struct sched_dl_entity const *b
kernel/sched/deadline.c:1604:63: sparse: got struct sched_dl_entity [noderef] __rcu *
kernel/sched/deadline.c:1719:39: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:1719:39: sparse: expected struct task_struct *donor
kernel/sched/deadline.c:1719:39: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:2537:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@
kernel/sched/deadline.c:2537:9: sparse: expected struct sched_domain *[assigned] sd
kernel/sched/deadline.c:2537:9: sparse: got struct sched_domain [noderef] __rcu *parent
kernel/sched/deadline.c:2205:14: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct task_struct *curr @@ got struct task_struct [noderef] __rcu * @@
kernel/sched/deadline.c:2205:14: sparse: expected struct task_struct *curr
kernel/sched/deadline.c:2205:14: sparse: got struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:2206:15: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu * @@
kernel/sched/deadline.c:2206:15: sparse: expected struct task_struct *donor
kernel/sched/deadline.c:2206:15: sparse: got struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:2281:43: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:2281:43: sparse: expected struct task_struct *p
kernel/sched/deadline.c:2281:43: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:2862:38: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/deadline.c:2862:38: sparse: expected struct task_struct *tsk
kernel/sched/deadline.c:2862:38: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/deadline.c:2864:23: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/deadline.c:2864:23: sparse: expected struct task_struct *p
kernel/sched/deadline.c:2864:23: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/deadline.c:2866:44: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct sched_dl_entity const *b @@ got struct sched_dl_entity [noderef] __rcu * @@
kernel/sched/deadline.c:2866:44: sparse: expected struct sched_dl_entity const *b
kernel/sched/deadline.c:2866:44: sparse: got struct sched_dl_entity [noderef] __rcu *
kernel/sched/deadline.c:3060:23: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/deadline.c:3060:23: sparse: struct task_struct [noderef] __rcu *
kernel/sched/deadline.c:3060:23: sparse: struct task_struct *
kernel/sched/deadline.c:3106:32: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/build_policy.c: note: in included file:
kernel/sched/syscalls.c:206:22: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/syscalls.c:206:22: sparse: struct task_struct [noderef] __rcu *
kernel/sched/syscalls.c:206:22: sparse: struct task_struct *
kernel/sched/syscalls.c:251:48: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/syscalls.c:251:48: sparse: struct task_struct [noderef] __rcu *
kernel/sched/syscalls.c:251:48: sparse: struct task_struct *
kernel/sched/build_policy.c: note: in included file:
kernel/sched/sched.h:2297:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2297:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2297:25: sparse: struct task_struct *
kernel/sched/sched.h:2297:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2297:25: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2297:25: sparse: struct task_struct *
kernel/sched/sched.h:2308:26: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2308:26: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2308:26: sparse: struct task_struct *
kernel/sched/sched.h:2297:25: sparse: sparse: incompatible types in comparison expression (different address spaces):
vim +988 kernel/sched/rt.c
956
957 /*
958 * Preempt the current task with a newly woken task if needed:
959 */
960 static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)
961 {
962 struct task_struct *donor = rq->donor;
963 struct sched_dl_entity *woken_dl_se = NULL;
964 struct sched_dl_entity *curr_dl_se = NULL;
965
966 if (!rt_group_sched_enabled())
967 goto no_group_sched;
968
969 /*
970 * Preemption checks are different if the waking task and the current task
971 * are running on the global runqueue or in a cgroup. The following rules
972 * apply:
973 * - dl-tasks (and equally dl_servers) always preempt FIFO/RR tasks.
974 * - if curr is a FIFO/RR task inside a cgroup (i.e. run by a
975 * dl_server), or curr is a DEADLINE task and waking is a FIFO/RR task
976 * on the root cgroup, do nothing.
977 * - if waking is inside a cgroup but curr is a FIFO/RR task in the root
978 * cgroup, always reschedule.
979 * - if they are both on the global runqueue, run the standard code. 980 * - if they are both in the same cgroup, check for tasks priorities. 981 * - if they are both in a cgroup, but not the same one, check whether the 982 * woken task's dl_server preempts the current's dl_server. 983 * - if curr is a DEADLINE task and waking is in a cgroup, check whether 984 * the woken task's server preempts curr. 985 */ 986 if (is_dl_group(rt_rq_of_se(&p->rt))) 987 woken_dl_se = dl_group_of(rt_rq_of_se(&p->rt)); > 988 if (is_dl_group(rt_rq_of_se(&rq->curr->rt))) 989 curr_dl_se = dl_group_of(rt_rq_of_se(&rq->curr->rt)); 990 else if (task_has_dl_policy(rq->curr)) > 991 curr_dl_se = &rq->curr->dl;
992
993 if (woken_dl_se != NULL && curr_dl_se != NULL) {
994 if (woken_dl_se == curr_dl_se) {
995 if (p->prio < rq->curr->prio)
996 resched_curr(rq);
997
998 return;
999 }
1000
1001 if (dl_entity_preempt(woken_dl_se, curr_dl_se))
1002 resched_curr(rq);
1003
1004 return;
1005
1006 } else if (woken_dl_se != NULL) {
1007 resched_curr(rq);
1008 return;
1009
1010 } else if (curr_dl_se != NULL) {
1011 return;
1012 }
1013
1014 no_group_sched:
1015 if (p->prio < donor->prio) {
1016 resched_curr(rq);
1017 return;
1018 }
1019
1020 /*
1021 * If:
1022 *
1023 * - the newly woken task is of equal priority to the current task
1024 * - the newly woken task is non-migratable while current is migratable
1025 * - current will be preempted on the next reschedule
1026 *
1027 * we should check to see if current can readily move to a different
1028 * cpu. If so, we will reschedule to allow the push logic to try
1029 * to move current somewhere else, making room for our non-migratable
1030 * task.
1031 */
1032 if (p->prio == donor->prio && !test_tsk_need_resched(rq->curr))
1033 check_preempt_equal_prio(rq, p);
1034 }
1035
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Juri Lelli
maintainerRe: [RFC PATCH v4 13/28] sched/rt: Update task event callbacks for HCBS scheduling
In reply to Yuri AndriaccioHello,
Quoted 47 lines · Yuri Andriaccio
Update
wakeup_preempt_rt, switched_{from/to}_rt andprio_changed_rtwith rt-cgroup's specific preemption rules:- In
wakeup_preempt_rt(), whenever a task wakes up, it must be checked if it is served by a deadline server or it lives on the global runqueue. Preemption rules (as documented in the function), change based on the current task and woken task runqueue:
- If both tasks are FIFO/RR tasks on the global runqueue, or the same cgroup, run as normal.
- If woken is inside a cgroup, but curr is a FIFO task on the global runqueue, always preempt. If curr is a DEADLINE task, check if the dl server preempts curr.
- If both tasks are FIFO/RR tasks in served but different groups, check whether the woken server preempts the current server.
- Inswitched_from_rt(), perform a pull only on the global runqueue, and do nothing if the task is inside a group. This will change when migrations will be added.
- Inswitched_to_rt(), queue a push only on the global runqueue, while perform a priority check when the task switching is inside a group. This will change also when migrations will be added.
- Inprio_changed_rt(), queue a pull only on the global runqueue, if the task is not queued. If the task is queued, run preemption checks only if both the prio changed task and curr are in the same cgroup.Update
sched_rt_can_attach()to check if a task can be attached to a given cgroup. For now the check only consists in checking if the group has non-zero bandwidth. Remove the tsk argument fromsched_rt_can_attach, as it is unused.Change
cpu_cgroup_can_attach()to check if the attachee is a FIFO/RR task before attaching it to a cgroup.Update
__sched_setscheduler()to perform checks when trying to switch to FIFO/RR for a task inside a cgroup, as the group needs to have runtime allocated.Update
task_is_throttled_rt()forSCHED_CORE, returning theis_throttledvalue of the server if present, while global rt-tasks are never throttled.Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by luca abeni <[email protected]>Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>---
...
kernel/sched/rt.c prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio) · line 1845
Quoted 8 lines
kernel/sched/rt.c +2
-2@@ -1777,16 +1841,25 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)* If our priority decreases while running, we * may need to pull tasks to this runqueue. */ -if (oldprio < p->prio) -rt_queue_pull_task (rt_rq_of_se (&p->rt)); +if (!IS_ENABLED (CONFIG_RT_GROUP_SCHED ) && oldprio < p->prio) +rt_queue_pull_task (rt_rq);
Does this mean priority decreases never trigger pulls when CONFIG_RT_ SCHED_GROUP is enabled, even for global runqueue?
Thanks,
Juri
RFC PATCH v4 14/28 sched/rt: Update rt-cgroup schedulability checks
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Update sched_group_rt_runtime/period and sched_group_set_rt_runtime/period to use the newly defined data structures and perform necessary checks to update both the runtime and period of a given group.
The set functions call tg_set_rt_bandwidth() which is also updated:
- Use the newly added HCBS dl_bandwidth structure instead of rt_bandwidth.
- Update __rt_schedulable() to check for numerical issues:
- Prevent a non-zero runtime that is too small, since a non-zero very small runtime will make the servers behave as they had zero runtime.
- Since some computation use signed integers, the period might be so big that when read as a signed integer becomes a negative number, and we don't want that. If the period satisfies this prerequisite, also the runtime will do, since the runtime is always less than or equal to the period.
- Update tg_rt_schedulable(), used when walking the cgroup tree to check if all invariants are met:
- Update most of the instructions to obtain data from the newly added data structures (dl_bandwidth).
- If the task group is the root group, run a total bandwidth check with the newly added dl_check_tg() function.
- After all checks are successful, if the changed group is not the root cgroup, update the assigned runtime and period to all the local deadline servers.
- Additionally use a mutex guard instead of manually locking/unlocking.
Add dl_check_tg(), which performs an admission control test similar to __dl_overflow, but this time we are updating the cgroup's total bandwidth rather than scheduling a new DEADLINE task or updating a non-cgroup deadline server.
Finally, prevent creation of a cgroup hierarchy with depth greater than two, as this will be addressed in a future patch. A depth two hierarchy is sufficient for now for testing the patchset.
--- kernel/sched/core.c | 6 ++++ kernel/sched/deadline.c | 46 +++++++++++++++++++++++---- kernel/sched/rt.c | 70 +++++++++++++++++++++++------------------ kernel/sched/sched.h | 1 + 4 files changed, 87 insertions(+), 36 deletions(-)
kernel/sched/core.c +6 -0
@@ -9253,6 +9253,12 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)return &root_task_group.css; } +/* Do not allow cpu_cgroup hierachies with depth greater than 2. */ +#ifdef CONFIG_RT_GROUP_SCHED +if (parent != &root_task_group) +return ERR_PTR (-EINVAL ); +#endif + tg =sched_create_group (parent);if (IS_ERR (tg))return ERR_PTR (-ENOMEM );
kernel/sched/deadline.c +40 -6
59 lines not on this page · raw message
kernel/sched/rt.c +40 -30
149 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -419,6 +419,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,struct rq *served_rq, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void ); +extern int dl_check_tg (unsigned long total);extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period); extern void dl_server_update_idle_time(struct rq *rq,
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v4 14/28] sched/rt: Update rt-cgroup schedulability checks
Hello,
Quoted 44 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Update sched_group_rt_runtime/period and sched_group_set_rt_runtime/period to use the newly defined data structures and perform necessary checks to update both the runtime and period of a given group.
The set functions call
tg_set_rt_bandwidth()which is also updated:
- Use the newly added HCBSdl_bandwidthstructure instead ofrt_bandwidth.
- Update__rt_schedulable()to check for numerical issues:
- Prevent a non-zero runtime that is too small, since a non-zero very small runtime will make the servers behave as they had zero runtime.
- Since some computation use signed integers, the period might be so big that when read as a signed integer becomes a negative number, and we don't want that. If the period satisfies this prerequisite, also the runtime will do, since the runtime is always less than or equal to the period.
- Updatetg_rt_schedulable(), used when walking the cgroup tree to check if all invariants are met:
- Update most of the instructions to obtain data from the newly added data structures (dl_bandwidth).
- If the task group is the root group, run a total bandwidth check with the newly addeddl_check_tg()function.
- After all checks are successful, if the changed group is not the root cgroup, update the assigned runtime and period to all the local deadline servers.
- Additionally use a mutex guard instead of manually locking/unlocking.Add
dl_check_tg(), which performs an admission control test similar to__dl_overflow, but this time we are updating the cgroup's total bandwidth rather than scheduling a new DEADLINE task or updating a non-cgroup deadline server.Finally, prevent creation of a cgroup hierarchy with depth greater than two, as this will be addressed in a future patch. A depth two hierarchy is sufficient for now for testing the patchset.
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>---
...
Quoted 20 lines
#ifdef
CONFIG_RT_GROUP_SCHEDkernel/sched/deadline.c +19
-0+int dl_check_tg (unsigned long total) +{ +unsigned long flags; +int which_cpu; +int cap; +struct dl_bw *dl_b; +u64 gen = ++dl_cookie; + +for_each_possible_cpu (which_cpu) { +rcu_read_lock_sched (); + +if (!dl_bw_visited (which_cpu, gen)) { + cap =dl_bw_capacity (which_cpu); + dl_b =dl_bw_of (which_cpu); + +raw_spin_lock_irqsave (&dl_b->lock, flags); + +if (dl_b->bw !=-1 && +cap_scale (dl_b->bw, cap) < dl_b->total_bw +cap_scale (total, cap)) {
kernel/sched/deadline.c void cancel_inactive_timer(struct sched_dl_entity *dl_se) · line 385
Quoted 10 lines
kernel/sched/deadline.c +10
-0+raw_spin_unlock_irqrestore (&dl_b->lock, flags); +rcu_read_unlock_sched (); + +return 0 ; + } + +raw_spin_unlock_irqrestore (&dl_b->lock, flags); + } + +rcu_read_unlock_sched ();
I believe we can use lock guards in the above?
...
Quoted 8 lines
kernel/sched/rt.c +4
-0@@ -2108,6 +2107,20 @@ static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime) .rt_runtime = runtime, }; +/* +* Since we truncate DL_SCALE bits, make sure we're at least +* that big. +*/
kernel/sched/rt.c static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime) · line 2123
Quoted 10 lines
kernel/sched/rt.c +10
-0+if (runtime !=0 && runtime < (1ULL <<DL_SCALE )) +return -EINVAL ; + +/* +* Since we use the MSB for wrap-around and sign issues, make +* sure it's not set (mind that period can be equal to zero). +*/ +if (period & (1ULL <<63 )) +return -EINVAL ; +
This is the same as in __checkparam_dl(), is it? Maybe we can create an helper?
Thanks,
Juri
RFC PATCH v4 15/28 sched/rt: Allow zeroing the runtime of the root control group
In reply to Yuri AndriaccioIn HCBS, the root control group follows the already existing rules for rt-task scheduling. As such, it does not make use of the deadline servers to account for runtime, or any other HCBS specific code and features.
While the runtime of SCHED_DEADLINE tasks depends on the global bandwidth reserved for rt_tasks, the runtime of SCHED_FIFO/SCHED_RR tasks is limited by the activation of fair-servers (as the RT_THROTTLING mechanism has been removed in favour of them), thus their maximum bandwidth depends solely on the fair-server settings (which are thightly related to the global bandwidth reserved for rt-tasks) and the amount of SCHED_OTHER workload to run (recall that if no SCHED_OTHER tasks are running, the FIFO/RR tasks may fully utilize the CPU).
The values of runtime and period in the root cgroup's cpu controller do not affect, by design of HCBS, the fair-server settings and similar (consequently they do not affect the scheduling of FIFO/RR tasks in the root cgroup), but they are just used to reserve a portion of the SCHED_DEADLINE bandwidth to the scheduling of rt-cgroups. These values only affect child cgroups, their deadline servers and their assigned FIFO/RR tasks.
--- kernel/sched/rt.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
kernel/sched/rt.c +4 -7
@@ -2134,13 +2134,6 @@ static int tg_set_rt_bandwidth(struct task_group *tg,static DEFINE_MUTEX (rt_constraints_mutex);int i, err =0 ; -/* -* Disallowing the root group RT runtime is BAD, it would disallow the -* kernel creating (and or operating) RT threads. -*/ -if (tg == &root_task_group && rt_runtime ==0 ) -return -EINVAL ; -/* No period doesn't make any sense. */ if (rt_period ==0 )return -EINVAL ;
@@ -2227,6 +2220,10 @@ static int sched_rt_global_constraints(void)int sched_rt_can_attach (struct task_group *tg) { +/* Allow executing in the root cgroup regardless of allowed bandwidth */ +if (tg == &root_task_group) +return 1 ; +/* Don't accept real-time tasks when there is no way for them to run */ if (rt_group_sched_enabled () && tg->dl_bandwidth.dl_runtime ==0 )return 0 ;
--
2.51.0
Re: [RFC PATCH v4 15/28] sched/rt: Allow zeroing the runtime of the root control group
Hello,
Quoted 21 lines · Yuri Andriaccio
In HCBS, the root control group follows the already existing rules for rt-task scheduling. As such, it does not make use of the deadline servers to account for runtime, or any other HCBS specific code and features.
While the runtime of
SCHED_DEADLINEtasks depends on the global bandwidth reserved forrt_tasks, the runtime of SCHED_FIFO/SCHED_RR tasks is limited by the activation of fair-servers (as theRT_THROTTLINGmechanism has been removed in favour of them), thus their maximum bandwidth depends solely on the fair-server settings (which are thightly related to the global bandwidth reserved for rt-tasks) and the amount ofSCHED_OTHERworkload to run (recall that if noSCHED_OTHERtasks are running, the FIFO/RR tasks may fully utilize the CPU).The values of runtime and period in the root cgroup's cpu controller do not affect, by design of HCBS, the fair-server settings and similar (consequently they do not affect the scheduling of FIFO/RR tasks in the root cgroup), but they are just used to reserve a portion of the
SCHED_DEADLINEbandwidth to the scheduling of rt-cgroups. These values only affect child cgroups, their deadline servers and their assigned FIFO/RR tasks.
Quoted 1 line
Signed-off-by Yuri Andriaccio <[email protected]>
Thanks,
Juri
RFC PATCH v4 16/28 sched/rt: Remove old RT_GROUP_SCHED data structures
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Completely remove the old RT_GROUP_SCHED's functions and data structures:
- Remove the fields back and my_q from sched_rt_entity.
- Remove the rt_bandwidth data structure.
- Remove the field rt_bandwidth from task_group.
- Remove the rt_bandwidth_enabled function.
- Remove the fields rt_queued, rt_throttled, rt_time, rt_runtime, rt_runtime_lock and rt_nr_boosted from rt_rq.
All of the removed fields and data are similarly represented in previously added fields in rq, rt_rq, dl_bandwidth and in the dl server themselves.
--- include/linux/sched.h | 3 --- kernel/sched/sched.h | 26 -------------------------- 2 files changed, 29 deletions(-)
include/linux/sched.h +0 -3
@@ -627,12 +627,9 @@ struct sched_rt_entity {unsigned short on_rq;unsigned short on_list; -struct sched_rt_entity *back;#ifdef CONFIG_RT_GROUP_SCHED /* rq on which this entity is (to be) queued: */ struct rt_rq *rt_rq; -/* rq "owned" by this entity/group: */ -struct rt_rq *my_q;#endif } __randomize_layout;
kernel/sched/sched.h +0 -26
@@ -310,15 +310,6 @@ struct rt_prio_array {struct list_head queue[MAX_RT_PRIO ]; }; -struct rt_bandwidth { -/* nests inside the rq lock: */ -raw_spinlock_t rt_runtime_lock; -ktime_t rt_period; -u64 rt_runtime; -struct hrtimer rt_period_timer; -unsigned int rt_period_active; -}; -struct dl_bandwidth {raw_spinlock_t dl_runtime_lock;u64 dl_runtime;
@@ -510,7 +501,6 @@ struct task_group {struct sched_dl_entity **dl_se;struct rt_rq **rt_rq; -struct rt_bandwidth rt_bandwidth;struct dl_bandwidth dl_bandwidth;#endif
@@ -826,11 +816,6 @@ struct scx_rq { };#endif /* CONFIG_SCHED_CLASS_EXT */ -static inline int rt_bandwidth_enabled (void ) -{ -return 0 ; -} -/* RT IPI pull logic requires IRQ_WORK */ #if defined(CONFIG_IRQ_WORK ) && defined(CONFIG_SMP )# define HAVE_RT_PUSH_IPI
@@ -848,17 +833,6 @@ struct rt_rq {bool overloaded;struct plist_head pushable_tasks; -int rt_queued; - -#ifdef CONFIG_RT_GROUP_SCHED -int rt_throttled; -u64 rt_time;/* consumed RT time, goes up in update_curr_rt */ -u64 rt_runtime;/* allotted RT time, "slice" from rt_bandwidth, RT sharing/balancing */ -/* Nests inside the rq lock: */ -raw_spinlock_t rt_runtime_lock; - -unsigned int rt_nr_boosted; -#endif #ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */ #endif
--
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v4 16/28] sched/rt: Remove old RT_GROUP_SCHED data structures
Hello,
Quoted 11 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Completely remove the old RT_GROUP_SCHED's functions and data structures:
- Remove the fields back and
my_qfromsched_rt_entity.
- Remove thert_bandwidthdata structure.
- Remove the fieldrt_bandwidthfromtask_group.
- Remove thert_bandwidth_enabledfunction.
- Remove the fieldsrt_queued,rt_throttled,rt_time,rt_runtime,rt_runtime_lockandrt_nr_boostedfromrt_rq.
Quoted 7 lines
All of the removed fields and data are similarly represented in previously added fields in rq,
rt_rq,dl_bandwidthand in the dl server themselves.Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>---
This looks all good.
While we are cleaning up stuff, maybe we could remove/update this comment as well
https://elixir.bootlin.com/linux/v6.18-rc6/source/kernel/sched/sched.h#L334
Best,
Juri
RFC PATCH v4 17/28 sched/core: Cgroup v2 support
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make rt_runtime_us and rt_period_us virtual files accessible also to the cgroup v2 controller, effectively enabling the RT_GROUP_SCHED mechanism to cgroups v2.
--- kernel/sched/core.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/sched/core.c +12 -0
@@ -10293,6 +10293,18 @@ static struct cftype cpu_files[] = { .write = cpu_uclamp_max_write, },#endif /* CONFIG_UCLAMP_TASK_GROUP */ +#ifdef CONFIG_RT_GROUP_SCHED + { + .name = "rt_runtime_us", + .read_s64 = cpu_rt_runtime_read, + .write_s64 = cpu_rt_runtime_write, + }, + { + .name = "rt_period_us", + .read_u64 = cpu_rt_period_read_uint, + .write_u64 = cpu_rt_period_write_uint, + }, +#endif /* CONFIG_RT_GROUP_SCHED */ { } /* terminate */ };
Signature · 1 line
2.51.0
Re: [RFC PATCH v4 17/28] sched/core: Cgroup v2 support
Hello,
Quoted 9 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Make
rt_runtime_usandrt_period_usvirtual files accessible also to the cgroup v2 controller, effectively enabling theRT_GROUP_SCHEDmechanism to cgroups v2.Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>---
Thanks,
Juri
RFC PATCH v4 18/28 sched/rt: Remove support for cgroups-v1
In reply to Yuri AndriaccioDisable control files for cgroups-v1, and allow only cgroups-v2. This should simplify maintaining the code, also because cgroups-v1 are deprecated.
Set the default rt-cgroups runtime to zero, otherwise a cgroup-v1 kernel will not be able to start SCHED_DEADLINE tasks. The bandwidth for rt-cgroups must then be manually assigned after the kernel boots.
--- kernel/sched/core.c | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-)
kernel/sched/core.c +1 -19
@@ -8693,7 +8693,7 @@ void __init sched_init(void)#ifdef CONFIG_RT_GROUP_SCHED init_dl_bandwidth (&root_task_group.dl_bandwidth, -global_rt_period (),global_rt_runtime ()); +global_rt_period (),0 );#endif /* CONFIG_RT_GROUP_SCHED */ #ifdef CONFIG_CGROUP_SCHED
@@ -10032,20 +10032,6 @@ static struct cftype cpu_legacy_files[] = { };#ifdef CONFIG_RT_GROUP_SCHED -static struct cftype rt_group_files[] = { - { - .name ="rt_runtime_us" , - .read_s64 = cpu_rt_runtime_read, - .write_s64 = cpu_rt_runtime_write, - }, - { - .name ="rt_period_us" , - .read_u64 = cpu_rt_period_read_uint, - .write_u64 = cpu_rt_period_write_uint, - }, - { }/* Terminate */ -}; -# ifdef CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED DEFINE_STATIC_KEY_FALSE (rt_group_sched);# else
@@ -10071,10 +10057,6 @@ __setup("rt_group_sched=", setup_rt_group_sched);static int __init cpu_rt_group_init (void ) { -if (!rt_group_sched_enabled ()) -return 0 ; - -WARN_ON (cgroup_add_legacy_cftypes (&cpu_cgrp_subsys, rt_group_files));return 0 ; }subsys_initcall (cpu_rt_group_init);
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v4 18/28] sched/rt: Remove support for cgroups-v1
Hello,
Quoted 3 lines · Yuri Andriaccio
Disable control files for cgroups-v1, and allow only cgroups-v2. This should simplify maintaining the code, also because cgroups-v1 are deprecated.
I still wonder if we want/need to be a little more verbose about why we think this is good, as it is quite a change. But, I'll let Peter and others to chime in.
Quoted 6 lines
Set the default rt-cgroups runtime to zero, otherwise a cgroup-v1 kernel will not be able to start
SCHED_DEADLINEtasks. The bandwidth for rt-cgroups must then be manually assigned after the kernel boots.Signed-off-by Yuri Andriaccio <[email protected]>---
...
kernel/sched/core.c __setup("rt_group_sched=", setup_rt_group_sched); · line 10062
Quoted 9 lines
static int
__initcpu_rt_group_init(void)
{kernel/sched/core.c +0
-4-if (!rt_group_sched_enabled ()) -return 0 ; - -WARN_ON (cgroup_add_legacy_cftypes (&cpu_cgrp_subsys, rt_group_files));return 0 ; }subsys_initcall (cpu_rt_group_init);
Looks like the whole function can be removed?
Thanks,
Juri
RFC PATCH v4 19/28 sched/deadline: Allow deeper hierarchies of RT cgroups
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Remove the check for depth two only cgroup hierachies.
Introduce the concept of live and active groups:
- A group is live if it is a leaf group or if all its children have zero runtime.
- A live group with non-zero runtime can be used to schedule tasks.
- A live group with running tasks is deemed active.
- A non-live group cannot be used to run tasks, but it is only used for bandwidth accounting, i.e. the sum of its children bandwidth must be less than or equal to the bandwidth of the parent. This change allows to use cgroups for bandwidth management for different users.
- While the root cgroup specifies the total allocatable bandwidth of rt cgroups, a further accounting is performed to keep track of the live bandwidth, i.e. the sum of the bandwidth of live groups. The hierarchy invariant states that the live bandwidth must always be less than or equal to the total allocatable bw.
Add is_live_sched_group() and sched_group_has_live_siblings() in deadline.c. These utility functions are used by dl_init_tg to perform updates only when necessary:
- Only live groups may update the active dl bandwidth of dl entities (call to dl_rq_change_utilization), while non-live groups must not use servers, and thus must not change the active dl bandwidth.
- The total bandwidth accounting must be changed to follow the live/non-live rules:
- When disabling (runtime zero) the last child of a group, the parent becomes a live group, and so the parent's bw must be accounted back.
- When enabling (runtime non-zero) the first child, the parent becomes a non-live group, and so the parent's bandwidth must be removed.
Update free_rt_sched_group() to only zero out the runtime of non-zeroed servers. This is also necessary to force the bandwidth accounting of live groups.
Update tg_set_rt_bandwidth() to change the runtime of a group to a non-zero value only if its parent is inactive, thus forcing it to become non-live if it was precedently (it would've already been non-live if a sibling cgroup was live).
Update sched_rt_can_attach() to allow attaching only on live groups.
Update dl_init_tg() to take a task_group pointer and a cpu's id rather than passing directly the pointer to the cpu's deadline server. The task_group pointer is necessary to check and update the live bandwidth accounting.
--- kernel/sched/core.c | 6 ---- kernel/sched/deadline.c | 61 ++++++++++++++++++++++++++++++++++++++--- kernel/sched/rt.c | 16 +++++++++-- kernel/sched/sched.h | 3 +- 4 files changed, 72 insertions(+), 14 deletions(-)
kernel/sched/core.c +0 -6
@@ -9253,12 +9253,6 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)return &root_task_group.css; } -/* Do not allow cpu_cgroup hierachies with depth greater than 2. */ -#ifdef CONFIG_RT_GROUP_SCHED -if (parent != &root_task_group) -return ERR_PTR (-EINVAL ); -#endif - tg =sched_create_group (parent);if (IS_ERR (tg))return ERR_PTR (-ENOMEM );
kernel/sched/deadline.c +57 -4
82 lines not on this page · raw message
kernel/sched/rt.c +13 -3
@@ -113,7 +113,8 @@ void free_rt_sched_group(struct task_group *tg)* Fix this issue by changing the group runtime * to 0 immediately before freeing it. */ -dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); +if (tg->dl_se[i]->dl_runtime) +dl_init_tg (tg, i,0 , tg->dl_se[i]->dl_period);raw_spin_rq_lock_irqsave (cpu_rq (i), flags);hrtimer_cancel (&tg->dl_se[i]->dl_timer);
@@ -2134,6 +2135,14 @@ static int tg_set_rt_bandwidth(struct task_group *tg,static DEFINE_MUTEX (rt_constraints_mutex);int i, err =0 ; +/* +* Do not allow to set a RT runtime > 0 if the parent has RT tasks +* (and is not the root group) +*/ +if (rt_runtime && tg != &root_task_group && + tg->parent != &root_task_group &&tg_has_rt_tasks (tg->parent)) +return -EINVAL ; +/* No period doesn't make any sense. */ if (rt_period ==0 )return -EINVAL ;
@@ -2157,7 +2166,7 @@ static int tg_set_rt_bandwidth(struct task_group *tg,return 0 ;for_each_possible_cpu (i) { -dl_init_tg (tg->dl_se[i], rt_runtime, rt_period); +dl_init_tg (tg, i, rt_runtime, rt_period); }return 0 ;
@@ -2228,7 +2237,8 @@ int sched_rt_can_attach(struct task_group *tg)if (rt_group_sched_enabled () && tg->dl_bandwidth.dl_runtime ==0 )return 0 ; - return 1; +/* tasks can be attached only if the taskgroup has no live children. */ + return (int)is_live_sched_group(tg); }#else /* !CONFIG_RT_GROUP_SCHED */
kernel/sched/sched.h +2 -1
@@ -411,7 +411,8 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void );extern int dl_check_tg (unsigned long total); -extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period); +extern void dl_init_tg (struct task_group *tg,int cpu,u64 rt_runtime,u64 rt_period); +extern bool is_live_sched_group (struct task_group *tg);extern void dl_server_update_idle_time (struct rq *rq,struct task_struct *p);
--
2.51.0
RFC PATCH v4 20/28 sched/rt: Add rt-cgroup migration
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
When the runtime is exhausted in a RT CGroup, the scheduler checks for another non-throttled runqueue and, if available, migrates the tasks.
The bandwidth (runtime/period) chosen for a certain cgroup is replicated on every core of the system, therefore, in an SMP system with M cores, the total available bandwidth is the given runtime/period multiplied by M.
--- kernel/sched/rt.c | 469 +++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 10 + 2 files changed, 479 insertions(+)
kernel/sched/rt.c +469 -0
@@ -1,3 +1,4 @@ +#pragma GCC diagnostic ignored "-Wunused-function"// SPDX-License-Identifier: GPL-2.0 /* * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -84,6 +85,8 @@ void init_rt_rq(struct rt_rq *rt_rq)plist_head_init (&rt_rq->pushable_tasks); } +static void group_pull_rt_task (struct rt_rq *this_rt_rq); +#ifdef CONFIG_RT_GROUP_SCHED void unregister_rt_sched_group (struct task_group *tg)
@@ -319,6 +322,46 @@ static inline void rt_queue_pull_task(struct rt_rq *rt_rq)queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); } +#ifdef CONFIG_RT_GROUP_SCHED +static DEFINE_PER_CPU(struct balance_callback, rt_group_push_head); +static DEFINE_PER_CPU(struct balance_callback, rt_group_pull_head); +static void group_push_rt_tasks_callback (struct rq *); +static void group_pull_rt_task_callback (struct rq *); + +static void rt_queue_push_from_group (struct rt_rq *rt_rq) +{ +struct rq *rq =served_rq_of_rt_rq (rt_rq); +struct rq *global_rq =cpu_rq (rq->cpu); + +if (global_rq->rq_to_push_from) +return ; + +if (!has_pushable_tasks (rt_rq)) +return ; + + global_rq->rq_to_push_from = rq; +queue_balance_callback (global_rq, &per_cpu (rt_group_push_head, global_rq->cpu), + group_push_rt_tasks_callback); +} + +static void rt_queue_pull_to_group (struct rt_rq *rt_rq) +{ +struct rq *rq =served_rq_of_rt_rq (rt_rq); +struct rq *global_rq =cpu_rq (rq->cpu); +struct sched_dl_entity *dl_se =dl_group_of (rt_rq); + +if (dl_se->dl_throttled || global_rq->rq_to_pull_to) +return ; + + global_rq->rq_to_pull_to = rq; +queue_balance_callback (global_rq, &per_cpu (rt_group_pull_head, global_rq->cpu), + group_pull_rt_task_callback); +} +#else /* !CONFIG_RT_GROUP_SCHED */ +static inline void rt_queue_push_from_group (struct rt_rq *rt_rq) {}; +static inline void rt_queue_pull_to_group (struct rt_rq *rt_rq) {}; +#endif /* CONFIG_RT_GROUP_SCHED */ +static void enqueue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) {plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);
@@ -1721,6 +1764,432 @@ static void pull_rt_task(struct rq *this_rq)resched_curr (this_rq); } +#ifdef CONFIG_RT_GROUP_SCHED +/* +* Find the lowest priority runqueue among the runqueues of the same +* task group. Unlike find_lowest_rt(), this does not mean that the +* lowest priority cpu is running tasks from this runqueue. +*/ +static int group_find_lowest_rt_rq (struct task_struct *task,struct rt_rq *task_rt_rq) +{ +struct sched_domain *sd; +struct cpumask mask, *lowest_mask = &mask; +struct sched_dl_entity *dl_se; +struct rt_rq *rt_rq; +int prio, lowest_prio; +int cpu, this_cpu =smp_processor_id (); + +if (task->nr_cpus_allowed ==1 ) +return -1 ;/* No other targets possible */ + + lowest_prio = task->prio -1 ; +cpumask_clear (lowest_mask); +for_each_cpu_and (cpu, cpu_online_mask, task->cpus_ptr) { + dl_se = task_rt_rq->tg->dl_se[cpu]; + rt_rq = &dl_se->my_q->rt; + prio = rt_rq->highest_prio.curr; + +/* +* If we're on asym system ensure we consider the different capacities +* of the CPUs when searching for the lowest_mask. +*/ +if (dl_se->dl_throttled || !rt_task_fits_capacity (task, cpu)) +continue ; + +if (prio >= lowest_prio) { +if (prio > lowest_prio) { +cpumask_clear (lowest_mask); + lowest_prio = prio; + } + +cpumask_set_cpu (cpu, lowest_mask); + } + } + +if (cpumask_empty (lowest_mask)) +return -1 ; + +/* +* At this point we have built a mask of CPUs representing the +* lowest priority tasks in the system. Now we want to elect +* the best one based on our affinity and topology. +* +* We prioritize the last CPU that the task executed on since +* it is most likely cache-hot in that location. +*/ + cpu =task_cpu (task); +if (cpumask_test_cpu (cpu, lowest_mask)) +return cpu; + +/* +* Otherwise, we consult the sched_domains span maps to figure +* out which CPU is logically closest to our hot cache data. +*/ +if (!cpumask_test_cpu (this_cpu, lowest_mask)) + this_cpu =-1 ;/* Skip this_cpu opt if not among lowest */ + +rcu_read_lock (); +for_each_domain (cpu, sd) { +if (sd->flags &SD_WAKE_AFFINE ) { +int best_cpu; + +/* +* "this_cpu" is cheaper to preempt than a +* remote processor. +*/ +if (this_cpu !=-1 && +cpumask_test_cpu (this_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return this_cpu; + } + + best_cpu =cpumask_any_and_distribute (lowest_mask, +sched_domain_span (sd)); +if (best_cpu < nr_cpu_ids) { +rcu_read_unlock (); +return best_cpu; + } + } + } +rcu_read_unlock (); + +/* +* And finally, if there were no matches within the domains +* just give the caller *something* to work with from the compatible +* locations. +*/ +if (this_cpu !=-1 ) +return this_cpu; + + cpu =cpumask_any_distribute (lowest_mask); +if (cpu < nr_cpu_ids) +return cpu; + +return -1 ; +} + +/* +* Find and lock the lowest priority runqueue among the runqueues +* of the same task group. Unlike find_lock_lowest_rt(), this does not +* mean that the lowest priority cpu is running tasks from this runqueue. +*/ +static struct rt_rq *group_find_lock_lowest_rt_rq (struct task_struct *task,struct rt_rq *rt_rq) +{ +struct rq *rq =rq_of_rt_rq (rt_rq); +struct rq *lowest_rq; +struct rt_rq *lowest_rt_rq =NULL ; +struct sched_dl_entity *lowest_dl_se; +int tries, cpu; + +for (tries =0 ; tries <RT_MAX_TRIES ; tries++) { + cpu =group_find_lowest_rt_rq (task, rt_rq); + +if ((cpu ==-1 ) || (cpu == rq->cpu)) +break ; + + lowest_dl_se = rt_rq->tg->dl_se[cpu]; + lowest_rt_rq = &lowest_dl_se->my_q->rt; + lowest_rq =cpu_rq (cpu); + +if (lowest_rt_rq->highest_prio.curr <= task->prio) { +/* +* Target rq has tasks of equal or higher priority, +* retrying does not release any lock and is unlikely +* to yield a different result. +*/ + lowest_rt_rq =NULL ; +break ; + } + +/* if the prio of this runqueue changed, try again */ +if (double_lock_balance (rq, lowest_rq)) { +/* +* We had to unlock the run queue. In +* the mean time, task could have +* migrated already or had its affinity changed. +* Also make sure that it wasn't scheduled on its rq. +* It is possible the task was scheduled, set +* "migrate_disabled" and then got preempted, so we must +* check the task migration disable flag here too. +*/ +if (unlikely (is_migration_disabled (task) || + lowest_dl_se->dl_throttled || + !cpumask_test_cpu (lowest_rq->cpu, &task->cpus_mask) || + task !=pick_next_pushable_task (rt_rq))) { + +double_unlock_balance (rq, lowest_rq); + lowest_rt_rq =NULL ; +break ; + } + } + +/* If this rq is still suitable use it. */ +if (lowest_rt_rq->highest_prio.curr > task->prio) +break ; + +/* try again */ +double_unlock_balance (rq, lowest_rq); + lowest_rt_rq =NULL ; + } + +return lowest_rt_rq; +} + +static int group_push_rt_task (struct rt_rq *rt_rq,bool pull) +{ +struct rq *rq =rq_of_rt_rq (rt_rq); +struct task_struct *next_task; +struct rq *lowest_rq; +struct rt_rq *lowest_rt_rq; +int ret =0 ; + +if (!rt_rq->overloaded) +return 0 ; + + next_task =pick_next_pushable_task (rt_rq); +if (!next_task) +return 0 ; + +retry: +if (is_migration_disabled (next_task)) { +struct task_struct *push_task =NULL ; +int cpu; + +if (!pull || rq->push_busy) +return 0 ; + +/* +* If the current task does not belong to the same task group +* we cannot push it away. +*/ +if (rq->curr->sched_task_group != rt_rq->tg) +return 0 ; + +/* +* Invoking group_find_lowest_rt_rq() on anything but an RT task doesn't +* make sense. Per the above priority check, curr has to +* be of higher priority than next_task, so no need to +* reschedule when bailing out. +* +* Note that the stoppers are masqueraded as SCHED_FIFO +* (cf. sched_set_stop_task()), so we can't rely on rt_task(). +*/ +if (rq->curr->sched_class != &rt_sched_class) +return 0 ; + + cpu =group_find_lowest_rt_rq (rq->curr, rt_rq); +if (cpu ==-1 || cpu == rq->cpu) +return 0 ; + +/* +* Given we found a CPU with lower priority than @next_task, +* therefore it should be running. However we cannot migrate it +* to this other CPU, instead attempt to push the current +* running task on this CPU away. +*/ + push_task =get_push_task (rq); +if (push_task) { +preempt_disable (); +raw_spin_rq_unlock (rq); +stop_one_cpu_nowait (rq->cpu, push_cpu_stop, + push_task, &rq->push_work); +preempt_enable (); +raw_spin_rq_lock (rq); + } + +return 0 ; + } + +if (WARN_ON (next_task == rq->curr)) +return 0 ; + +/* We might release rq lock */ +get_task_struct (next_task); + +/* group_find_lock_lowest_rq locks the rq if found */ + lowest_rt_rq =group_find_lock_lowest_rt_rq (next_task, rt_rq); +if (!lowest_rt_rq) { +struct task_struct *task; +/* +* group_find_lock_lowest_rt_rq releases rq->lock +* so it is possible that next_task has migrated. +* +* We need to make sure that the task is still on the same +* run-queue and is also still the next task eligible for +* pushing. +*/ + task =pick_next_pushable_task (rt_rq); +if (task == next_task) { +/* +* The task hasn't migrated, and is still the next +* eligible task, but we failed to find a run-queue +* to push it to. Do not retry in this case, since +* other CPUs will pull from us when ready. +*/ + goto out; + } + +if (!task) +/* No more tasks, just exit */ + goto out; + +/* +* Something has shifted, try again. +*/ +put_task_struct (next_task); + next_task = task; + goto retry; + } + + lowest_rq =rq_of_rt_rq (lowest_rt_rq); + +move_queued_task_locked (rq, lowest_rq, next_task); +resched_curr (lowest_rq); + ret =1 ; + +double_unlock_balance (rq, lowest_rq); +out: +put_task_struct (next_task); + +return ret; +} + +static void group_pull_rt_task (struct rt_rq *this_rt_rq) +{ +struct rq *this_rq =rq_of_rt_rq (this_rt_rq); +int this_cpu = this_rq->cpu, cpu; +bool resched = false; +struct task_struct *p, *push_task =NULL ; +struct rt_rq *src_rt_rq; +struct rq *src_rq; +struct sched_dl_entity *src_dl_se; + +for_each_online_cpu (cpu) { +if (this_cpu == cpu) +continue ; + + src_dl_se = this_rt_rq->tg->dl_se[cpu]; + src_rt_rq = &src_dl_se->my_q->rt; + +if (src_rt_rq->rt_nr_running <=1 && !src_dl_se->dl_throttled) +continue ; + + src_rq =rq_of_rt_rq (src_rt_rq); + +/* +* Don't bother taking the src_rq->lock if the next highest +* task is known to be lower-priority than our current task. +* This may look racy, but if this value is about to go +* logically higher, the src_rq will push this task away. +* And if its going logically lower, we do not care +*/ +if (src_rt_rq->highest_prio.next >= + this_rt_rq->highest_prio.curr) +continue ; + +/* +* We can potentially drop this_rq's lock in +* double_lock_balance, and another CPU could +* alter this_rq +*/ + push_task =NULL ; +double_lock_balance (this_rq, src_rq); + +/* +* We can pull only a task, which is pushable +* on its rq, and no others. +*/ + p =pick_highest_pushable_task (src_rt_rq, this_cpu); + +/* +* Do we have an RT task that preempts +* the to-be-scheduled task? +*/ +if (p && (p->prio < this_rt_rq->highest_prio.curr)) { +WARN_ON (p == src_rq->curr); +WARN_ON (!task_on_rq_queued (p)); + +/* +* There's a chance that p is higher in priority +* than what's currently running on its CPU. +* This is just that p is waking up and hasn't +* had a chance to schedule. We only pull +* p if it is lower in priority than the +* current task on the run queue +*/ +if (src_rq->curr->sched_task_group == this_rt_rq->tg && + p->prio < src_rq->curr->prio) + goto skip; + +if (is_migration_disabled (p)) { +/* +* If the current task does not belong to the same task group +* we cannot push it away. +*/ +if (src_rq->curr->sched_task_group != this_rt_rq->tg) + goto skip; + + push_task =get_push_task (src_rq); + }else { +move_queued_task_locked (src_rq, this_rq, p); + resched = true; + } +/* +* We continue with the search, just in +* case there's an even higher prio task +* in another runqueue. (low likelihood +* but possible) +*/ + } +skip: +double_unlock_balance (this_rq, src_rq); + +if (push_task) { +preempt_disable (); +raw_spin_rq_unlock (this_rq); +stop_one_cpu_nowait (src_rq->cpu, push_cpu_stop, + push_task, &src_rq->push_work); +preempt_enable (); +raw_spin_rq_lock (this_rq); + } + } + +if (resched) +resched_curr (this_rq); +} + +static void group_push_rt_tasks (struct rt_rq *rt_rq) +{ +while (group_push_rt_task (rt_rq, false)) + ; +} + +static void group_push_rt_tasks_callback (struct rq *global_rq) +{ +struct rt_rq *rt_rq = &global_rq->rq_to_push_from->rt; + +if ((rt_rq->rt_nr_running >1 ) || + (dl_group_of (rt_rq)->dl_throttled ==1 )) { + +while (group_push_rt_task (rt_rq, false)) + ; + } + + global_rq->rq_to_push_from =NULL ; +} + +static void group_pull_rt_task_callback (struct rq *global_rq) +{ +struct rt_rq *rt_rq = &global_rq->rq_to_pull_to->rt; + +group_pull_rt_task (rt_rq); + global_rq->rq_to_pull_to =NULL ; +} +#else /* !CONFIG_RT_GROUP_SCHED */ +static void group_pull_rt_task (struct rt_rq *this_rt_rq) { } +static void group_push_rt_tasks (struct rt_rq *rt_rq) { } +#endif /* CONFIG_RT_GROUP_SCHED */ +/* * If we are not running and we are not going to reschedule soon, we should * try to push tasks away now
kernel/sched/sched.h +10 -0
@@ -1311,6 +1311,16 @@ struct rq {call_single_data_t cfsb_csd;struct list_head cfsb_csd_list;#endif + +#ifdef CONFIG_RT_GROUP_SCHED +/* +* Balance callbacks operate only on global runqueues. +* These pointers allow referencing cgroup specific runqueues +* for balancing operations. +*/ +struct rq *rq_to_push_from; +struct rq *rq_to_pull_to; +#endif };#ifdef CONFIG_FAIR_GROUP_SCHED
Signature · 1 line
2.51.0
RFC PATCH v4 21/28 sched/rt: Add HCBS migration related checks and function calls
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add HCBS related checks and operations to allow rt-task migration, differentiating between cgroup's tasks or tasks that run on the global runqueue.
--- kernel/sched/rt.c | 61 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 47 insertions(+), 14 deletions(-)
kernel/sched/rt.c +47 -14
@@ -1,4 +1,3 @@ -#pragma GCC diagnostic ignored "-Wunused-function"// SPDX-License-Identifier: GPL-2.0 /* * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -892,6 +891,11 @@ select_task_rq_rt(struct task_struct *p, int cpu, int flags)struct rq *rq;bool test; +/* Just return the task_cpu for processes inside task groups */ +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) && +is_dl_group (rt_rq_of_se (&p->rt))) + goto out; +/* For anything but wake ups, just return the task_cpu */ if (!(flags & (WF_TTWU |WF_FORK ))) goto out;
@@ -991,7 +995,10 @@ static int balance_rt(struct rq *rq, struct task_struct *p, struct rq_flags *rf)* not yet started the picking loop. */ rq_unpin_lock (rq, rf); -pull_rt_task (rq); +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq_of_se (&p->rt))) +group_pull_rt_task (rt_rq_of_se (&p->rt)); +else +pull_rt_task (rq);rq_repin_lock (rq, rf); }
@@ -1100,7 +1107,10 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool fif (rq->donor->sched_class != &rt_sched_class)update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); -rt_queue_push_tasks (rt_rq); +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +rt_queue_push_from_group (rt_rq); +else +rt_queue_push_tasks (rt_rq); }static struct sched_rt_entity *pick_next_rt_entity (struct rt_rq *rt_rq)
@@ -1153,6 +1163,13 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s*/ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 )enqueue_pushable_task (rt_rq, p); + +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) { +struct sched_dl_entity *dl_se =dl_group_of (rt_rq); + +if (dl_se->dl_throttled) +rt_queue_push_from_group (rt_rq); + } }/* Only try algorithms three times */
@@ -2196,6 +2213,7 @@ static void group_push_rt_tasks(struct rt_rq *rt_rq) { }*/ static void task_woken_rt (struct rq *rq,struct task_struct *p) { +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt);bool need_to_push = !task_on_cpu (rq, p) && !test_tsk_need_resched (rq->curr) && p->nr_cpus_allowed > 1 &&
@@ -2203,7 +2221,12 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) (rq->curr->nr_cpus_allowed <2 || rq->donor->prio <= p->prio); -if (need_to_push) +if (!need_to_push) +return ; + +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +group_push_rt_tasks (rt_rq); +else push_rt_tasks (rq); }
@@ -2243,7 +2266,9 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)if (!task_on_rq_queued (p) || rt_rq->rt_nr_running)return ; -if (!IS_ENABLED (CONFIG_RT_GROUP_SCHED ) || !is_dl_group (rt_rq)) +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +rt_queue_pull_to_group (rt_rq); +else rt_queue_pull_task (rt_rq); }
@@ -2272,6 +2297,13 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)*/ if (task_current (rq, p)) {update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); + +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq_of_se (&p->rt))) { +struct sched_dl_entity *dl_se =dl_group_of (rt_rq_of_se (&p->rt)); + + p->dl_server = dl_se; + } +return ; }
@@ -2281,13 +2313,10 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)* then see if we can move to another run queue. */ if (task_on_rq_queued (p)) { -if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) { -if (p->prio < rq->curr->prio) -resched_curr (rq); - }else { -if (p->nr_cpus_allowed >1 && rq->rt.overloaded) -rt_queue_push_tasks (rt_rq_of_se (&p->rt)); - } +if (!is_dl_group (rt_rq) && p->nr_cpus_allowed >1 && rq->rt.overloaded) +rt_queue_push_tasks (rt_rq); +else if (is_dl_group (rt_rq) && rt_rq->overloaded) +rt_queue_push_from_group (rt_rq);if (p->prio < rq->donor->prio &&cpu_online (cpu_of (rq)))resched_curr (rq);
@@ -2311,8 +2340,12 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)* If our priority decreases while running, we * may need to pull tasks to this runqueue. */ -if (!IS_ENABLED (CONFIG_RT_GROUP_SCHED ) && oldprio < p->prio) -rt_queue_pull_task (rt_rq); +if (oldprio < p->prio) { +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +rt_queue_pull_to_group (rt_rq); +else +rt_queue_pull_task (rt_rq); + }/* * If there's a higher priority task waiting to run
--
2.51.0
RFC PATCH v4 22/28 sched/deadline: Introduce dl_server_try_pull_f
In reply to Yuri AndriaccioIntroduce a new deadline server callback, dl_server_try_pull_f, that attempts to pull some tasks from other runqueues and returns true if after this operation the runqueue is not empty.
This function is needed by some scheduling algorithms to guarantee that they are work-conserving (i.e. whenever there is an idle CPU and a ready task, this must be immediately scheduled there) or to enforce some other properties of the scheduling algorithm used on the served runqueue (for example, for a fixed-priority scheduler the m highest-priority tasks must be scheduled).
The function is called whenever the dl_server_timer (the runtime replenishment timer) expires and the deadline server is recharged. The idea behind this callback is that since the deadline server is being unthrottled and is becoming able to serve its runqueue, it should pull tasks from the other runqueues (if there are no runnable tasks on its own runqueue, or if the tasks on its runqueue have low priority).
The interface of the function provides a sched_dl_entity pointer, i.e. the deadline server and expects to return true if there are runnable tasks on that server (also just pulled from other runqueues), false otherwise.
The replenishment timer callback, if there are no runnable tasks for a given server, will replenish its bandwidth and then stop it.
This callback is not relevant for fair deadline servers.
This fixes the test case where a single hog process, in a cgroup with reservation 10ms/100ms, runs (without this patch) on only one CPU, while (with this patch), by definition of the global Fixed Priority scheduling algorithm, it must run on all (up to a utilization of 100ms/100ms) the CPUs of the machine.
--- include/linux/sched.h | 3 ++- kernel/sched/deadline.c | 12 ++++++++++++ kernel/sched/fair.c | 8 +++++++- kernel/sched/rt.c | 13 ++++++++++++- kernel/sched/sched.h | 6 ++++++ 5 files changed, 39 insertions(+), 3 deletions(-)
include/linux/sched.h +2 -1
@@ -633,7 +633,7 @@ struct sched_rt_entity {#endif } __randomize_layout; -typedef bool (*dl_server_has_tasks_f )(struct sched_dl_entity *); +typedef bool (*dl_server_try_pull_f )(struct sched_dl_entity *);typedef struct task_struct *(*dl_server_pick_f )(struct sched_dl_entity *);struct sched_dl_entity {
@@ -734,6 +734,7 @@ struct sched_dl_entity {struct dl_rq *dl_rq;struct rq *my_q;dl_server_pick_f server_pick_task; +dl_server_try_pull_f server_try_pull_task;#ifdef CONFIG_RT_MUTEXES /*
kernel/sched/deadline.c +12 -0
@@ -1295,6 +1295,7 @@ static const u64 dl_server_min_res = 1 * NSEC_PER_MSEC;static enum hrtimer_restart dl_server_timer (struct hrtimer *timer,struct sched_dl_entity *dl_se) {struct rq *rq =rq_of_dl_se (dl_se); +bool is_active;u64 fw; scoped_guard (rq_lock, rq) {
@@ -1309,6 +1310,15 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_if (!dl_se->dl_runtime)return HRTIMER_NORESTART ; +rq_unpin_lock (rq, rf); + is_active = dl_se->server_try_pull_task (dl_se); +rq_repin_lock (rq, rf); +if (!is_active) { +replenish_dl_entity (dl_se); +dl_server_stop (dl_se); +return HRTIMER_NORESTART ; + } +if (dl_se->dl_defer_armed) {/* * First check if the server could consume runtime in background.
@@ -1712,10 +1722,12 @@ void dl_server_stop(struct sched_dl_entity *dl_se)void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq,struct rq *served_rq, +dl_server_try_pull_f try_pull_task,dl_server_pick_f pick_task) { dl_se->dl_rq = dl_rq; dl_se->my_q = served_rq; + dl_se->server_try_pull_task = try_pull_task; dl_se->server_pick_task = pick_task; }
kernel/sched/fair.c +7 -1
@@ -8957,6 +8957,11 @@ static struct task_struct *__pick_next_task_fair(struct rq *rq, struct task_strureturn pick_next_task_fair (rq, prev,NULL ); } +static bool fair_server_try_pull_task (struct sched_dl_entity *dl_se) +{ +return true; +} +static struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se) {return pick_task_fair (dl_se->my_q);
@@ -8968,7 +8973,8 @@ void fair_server_init(struct rq *rq)init_dl_entity (dl_se); -dl_server_init (dl_se, &rq->dl, rq, fair_server_pick_task); +dl_server_init (dl_se, &rq->dl, rq, + fair_server_try_pull_task, fair_server_pick_task); }/*
kernel/sched/rt.c +12 -1
@@ -134,6 +134,16 @@ void free_rt_sched_group(struct task_group *tg)static struct sched_rt_entity *pick_next_rt_entity (struct rt_rq *rt_rq);static inline void set_next_task_rt (struct rq *rq,struct task_struct *p,bool first); +static bool rt_server_try_pull (struct sched_dl_entity *dl_se) +{ +struct rt_rq *rt_rq = &dl_se->my_q->rt; + +if (dl_se->my_q->rt.rt_nr_running ==0 ) +group_pull_rt_task (rt_rq); + +return dl_se->my_q->rt.rt_nr_running >0 ; +} +static struct task_struct *rt_server_pick (struct sched_dl_entity *dl_se) {struct rt_rq *rt_rq = &dl_se->my_q->rt;
@@ -235,7 +245,8 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent) dl_se->dl_density =to_ratio (dl_se->dl_period, dl_se->dl_runtime); dl_se->dl_server =1 ; -dl_server_init (dl_se, &cpu_rq (i)->dl, s_rq, rt_server_pick); +dl_server_init (dl_se, &cpu_rq (i)->dl, s_rq, + rt_server_try_pull, rt_server_pick); }return 1 ;
kernel/sched/sched.h +6 -0
@@ -363,6 +363,11 @@ extern s64 dl_scaled_delta_exec(struct rq *rq, struct sched_dl_entity *dl_se, s6* * dl_se::rq -- runqueue we belong to. * +* dl_se::server_try_pull() -- used on bandwidth enforcement; the server has a +* chance to pull tasks from the other runqueues, +* otherwise it is stopped if there is no task to +* run. +* * dl_se::server_pick() -- nested pick_next_task(); we yield the period if this * returns NULL. *
@@ -408,6 +413,7 @@ extern void dl_server_start(struct sched_dl_entity *dl_se);extern void dl_server_stop (struct sched_dl_entity *dl_se);extern void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq,struct rq *served_rq, +dl_server_try_pull_f try_pull_task,dl_server_pick_f pick_task);extern void sched_init_dl_servers (void );extern int dl_check_tg (unsigned long total);
Signature · 1 line
2.51.0
RFC PATCH v4 23/28 sched/deadline: Fix HCBS migrations on server stop
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Do not unthrottle a non-fair-server dl_server_stop(), since it ends up being stopped when throttled (we try to migrate all the RT tasks away from it).
Notes:
This is a temporary workaround, but it will be hopefully removed in favor of less invasive code.
--- kernel/sched/deadline.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +4 -2
@@ -1714,9 +1714,11 @@ void dl_server_stop(struct sched_dl_entity *dl_se)return ;dequeue_dl_entity (dl_se,DEQUEUE_SLEEP ); -hrtimer_try_to_cancel (&dl_se->dl_timer); +if (dl_se == &rq_of_dl_se (dl_se)->fair_server) { +hrtimer_try_to_cancel (&dl_se->dl_timer); + dl_se->dl_throttled =0 ; + } dl_se->dl_defer_armed =0 ; - dl_se->dl_throttled =0 ; dl_se->dl_server_active =0 ; }
--
2.51.0
RFC PATCH v4 24/28 sched/core: Execute enqueued balance callbacks when changing allowed CPUs
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Execute balancing callbacks when setting the affinity of a task, since the HCBS scheduler may request balancing of throttled dl_servers to fully utilize the server's bandwidth.
--- kernel/sched/core.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/sched/core.c +13 -0
@@ -2915,6 +2915,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flagif (cpumask_test_cpu (task_cpu (p), &p->cpus_mask) || (task_current_donor (rq, p) && !task_current (rq, p))) {struct task_struct *push_task =NULL ; +struct balance_callback *head;if ((flags &SCA_MIGRATE_ENABLE ) && (p->migration_flags &MDF_PUSH ) && !rq->push_busy) {
@@ -2933,11 +2934,13 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag }preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf);if (push_task) {stop_one_cpu_nowait (rq->cpu, push_cpu_stop, p, &rq->push_work); } +balance_callbacks (rq, head);preempt_enable ();if (complete)
@@ -2992,6 +2995,8 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag }if (task_on_cpu (rq, p) ||READ_ONCE (p->__state) ==TASK_WAKING ) { +struct balance_callback *head; +/* * MIGRATE_ENABLE gets here because 'p == current', but for * anything else we cannot do is_migration_disabled(), punt
@@ -3005,16 +3010,19 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag p->migration_flags &= ~MDF_PUSH ;preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf);if (!stop_pending) {stop_one_cpu_nowait (cpu_of (rq), migration_cpu_stop, &pending->arg, &pending->stop_work); } +balance_callbacks (rq, head);preempt_enable ();if (flags &SCA_MIGRATE_ENABLE )return 0 ; }else { +struct balance_callback *head;if (!is_migration_disabled (p)) {if (task_on_rq_queued (p))
@@ -3025,7 +3033,12 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag complete = true; } } + +preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf); +balance_callbacks (rq, head); +preempt_enable ();if (complete)complete_all (&pending->done);
--
2.51.0
RFC PATCH v4 25/28 sched/core: Execute enqueued balance callbacks when migrating task betweeen cgroups
In reply to Yuri AndriaccioExecute balancing callbacks when migrating task between cgroups, since the HCBS scheduler, similarly to the previous patch, may request balancing of throttled dl_servers to fully utilize the server's bandwidth.
Introduce the RELEASE_LOCK helper macro to explicitly unlock a guard-based lock. The macro calls the destructor function of the lock and invalidates it so that the lock is not unlocked twice when the lock variable goes out of scope.
--- include/linux/cleanup.h | 3 +++ kernel/sched/core.c | 7 +++++++ 2 files changed, 10 insertions(+)
include/linux/cleanup.h +3 -0
@@ -518,4 +518,7 @@ __DEFINE_LOCK_GUARD_0(_name, _lock) #define DEFINE_LOCK_GUARD_1_COND(X...) CONCATENATE(DEFINE_LOCK_GUARD_1_COND_, COUNT_ARGS(X))(X) +#define RELEASE_LOCK (_name, _var) \ + ({ class_##_name##_destructor(&_var); no_free_ptr(_var.lock); }) +#endif /* _LINUX_CLEANUP_H */
kernel/sched/core.c +7 -0
@@ -9223,6 +9223,7 @@ void sched_move_task(struct task_struct *tsk, bool for_autogroup) {int queued, running, queue_flags =DEQUEUE_SAVE |DEQUEUE_MOVE |DEQUEUE_NOCLOCK ; +struct balance_callback *head;struct rq *rq;CLASS (task_rq_lock, rq_guard)(tsk);
@@ -9253,6 +9254,12 @@ void sched_move_task(struct task_struct *tsk, bool for_autogroup)*/ resched_curr (rq); } + +preempt_disable (); + head =splice_balance_callbacks (rq); +RELEASE_LOCK (task_rq_lock, rq_guard); +balance_callbacks (rq, head); +preempt_enable (); } static struct cgroup_subsys_state *
--
2.51.0
RFC PATCH v4 26/28 Documentation: Update documentation for real-time cgroups
In reply to Yuri AndriaccioUpdate the RT_GROUP_SCHED specific documentation. Give a brief theoretical background for Hierarchical Constant Bandwidth Server (HCBS). Document how the HCBS is implemented in the kernel and how the RT_GROUP_SCHED behaves now compared to the version which this patchset replaces.
--- Documentation/scheduler/sched-rt-group.rst | 500 ++++++++++++++++++--- 1 file changed, 426 insertions(+), 74 deletions(-)
Documentation/scheduler/sched-rt-group.rst +426 -74
@@ -53,9 +53,12 @@ CPU time is divided by means of specifying how much time can be spent running in a given period. We allocate this "run time" for each real-time group which the other real-time groups will not be permitted to use. -Any time not allocated to a real-time group will be used to run normal priority -tasks (SCHED_OTHER). Any allocated run time not used will also be picked up by -SCHED_OTHER. +Each real-time group runs at the same priority as SCHED_DEADLINE, thus they +share and contend the SCHED_DEADLINE allowed bandwidth. Any time not allocated +to a real-time group (and SCHED_DEADLINE tasks) will be used to run both +SCHED_FIFO/SCHED_RR, normal priority tasks (SCHED_OTHER), and SCHED_EXT tasks, +following the usual priorities. Any allocated run time not used will also be +picked up by the other scheduling classes, in the same order as before. Let's consider an example: a frame fixed real-time renderer must deliver 25 frames a second, which yields a period of 0.04s per frame. Now say it will also
@@ -73,10 +76,6 @@ The remaining CPU time will be used for user input and other tasks. Because real-time tasks have explicitly allocated the CPU time they need to perform their tasks, buffer underruns in the graphics or audio can be eliminated. -NOTE: the above example is not fully implemented yet. We still -lack an EDF scheduler to make non-uniform periods usable. - - 2. The Interface ================
@@ -86,40 +85,92 @@ lack an EDF scheduler to make non-uniform periods usable. The system wide settings are configured under the /proc virtual file system: -/proc/sys/kernel/sched_rt_period_us: +``/proc/sys/kernel/sched_rt_period_us``: The scheduling period that is equivalent to 100% CPU bandwidth. -/proc/sys/kernel/sched_rt_runtime_us: - A global limit on how much time real-time scheduling may use. This is always - less or equal to the period_us, as it denotes the time allocated from the - period_us for the real-time tasks. Without CONFIG_RT_GROUP_SCHED enabled, - this only serves for admission control of deadline tasks. With - CONFIG_RT_GROUP_SCHED=y it also signifies the total bandwidth available to - all real-time groups. +``/proc/sys/kernel/sched_rt_runtime_us``: + A global limit on how much time real-time scheduling may use (SCHED_DEADLINE + tasks + real-time groups). This is always less or equal to the period_us, as + it denotes the time allocated from the period_us for the real-time tasks. + Without **CONFIG_RT_GROUP_SCHED** enabled, this only serves for admission + control of deadline tasks. With **CONFIG_RT_GROUP_SCHED=y** it also signifies + the total bandwidth available to both real-time groups and deadline tasks. * Time is specified in us because the interface is s32. This gives an operating range from 1us to about 35 minutes. - * sched_rt_period_us takes values from 1 to INT_MAX. - * sched_rt_runtime_us takes values from -1 to sched_rt_period_us. + * ``sched_rt_period_us`` takes values from 1 to INT_MAX. + * ``sched_rt_runtime_us`` takes values from -1 to ``sched_rt_period_us``. * A run time of -1 specifies runtime == period, ie. no limit. - * sched_rt_runtime_us/sched_rt_period_us > 0.05 inorder to preserve + * ``sched_rt_runtime_us/sched_rt_period_us`` > 0.05 inorder to preserve bandwidth for fair dl_server. For accurate value check average of - runtime/period in /sys/kernel/debug/sched/fair_server/cpuX/ - - -2.2 Default behaviour ---------------------- - -The default values for sched_rt_period_us (1000000 or 1s) and -sched_rt_runtime_us (950000 or 0.95s). This gives 0.05s to be used by -SCHED_OTHER (non-RT tasks). These defaults were chosen so that a run-away -real-time tasks will not lock up the machine but leave a little time to recover -it. By setting runtime to -1 you'd get the old behaviour back. - -By default all bandwidth is assigned to the root group and new groups get the -period from /proc/sys/kernel/sched_rt_period_us and a run time of 0. If you -want to assign bandwidth to another group, reduce the root group's bandwidth -and assign some or all of the difference to another group. + runtime/period in ``/sys/kernel/debug/sched/fair_server/cpuX/`` + +The default value for ``sched_rt_period_us`` is 1000000 (or 1s) and for +``sched_rt_runtime_us`` is 950000 (or 0.95s). This gives a minimum of 0.05s to +be used by SCHED_FIFO/SCHED_RR and non-RT tasks (SCHED_OTHER, SCHED_EXT), while +0.95s are the maximum to be used by SCHED_DEADLINE, and rt-cgroups if enabled. + +2.2 Cgroup settings +------------------- + +Enabling **CONFIG_RT_GROUP_SCHED** lets you explicitly allocate real CPU +bandwidth to task groups. + +This uses the cgroup virtual file system and the CPU controller for cgroups. +Enabling the controller for the hierarchy creates two files: + +* ``<cgroup>/cpu.rt_period_us``, the scheduling period of the group. +* ``<cgroup>/cpu.rt_runtime_us``, the maximum runtime each CPU will provide + every period. + + .. tip:: + For more information on working with control groups, you should read + *Documentation/admin-guide/cgroup-v1/cgroups.rst* as well. + .. + +By default the root cgroup has the same period of +``/proc/sys/kernel/sched_rt_period_us``, which is 1s, and a runtime of zero, so +that rt-cgroup is *soft-disabled* by default, and all the runtime is available +for SCHED_DEADLINE tasks only. New groups instead get both period and runtime of +zero. + +2.3 Cgroup Hierarchy and Behaviours +----------------------------------- + +With HCBS, cgroups may act either as task runners or bandwidth reservation: + +* A bandwidth reservation cgroup (such as the root control group), has the + purpose to reserve a portion of the total real-time bandwidth for its sub-tree + of groups. A group in this state cannot run SCHED_FIFO/SCHED_RR tasks. + + .. important:: + The *root control group* behaviour is different from the other cgroups, as + its job is to reserve bandwidth for the whole group hierarchy, but it can + also run rt tasks. This is an exception: FIFO/RR tasks running in the + root cgroup follow the same rules as FIFO/RR tasks in a kernel which has + **CONFIG_RT_GROUP_SCHED=n**, and the bandwidth reservation is instead a + feature connected to HCBS, that acts on the cgroup tree. + .. + +* A *live* group instead can be used to run FIFO/RR tasks, with the given + bandwidth parameters: each CPU is served a *potentially continuous* runtime of + ``<cgroup>/cpu.rt_runtime_us`` every period ``<cgroup>/cpu.rt_period_us``. It + is important to notice that increasing the period but leaving the bandwidth + constant changes the behaviour of the cgroup's servers, as the bandwidth given + overall is the same, but it is given in longer bursts (and longer slices of no + bandwidth). + +More specifically on *live* and non-*live*: + +* A group is deemed *live* if it is a leaf of the groups' hierarchy or all of + its children have runtime 0. +* *Live* groups are the only groups allowed to run real-time tasks. A SCHED_FIFO + task cannot be migrated in a non-*live* group, neither a task inside this + group can change scheduling policy to SCHED_FIFO/SCHED_RR if the group is not + *live*. +* Non-*live* groups are only used for bandwidth reservation. +* Group's bandwidth follow this invariant: the sum of the bandwidths of a + group's children is always less than or equal to the group's bandwidth. Real-time group scheduling means you have to assign a portion of total CPU bandwidth to the group before it will accept real-time tasks. Therefore you will
@@ -128,63 +179,364 @@ done that, even if the user has the rights to run processes with real-time priority! -2.3 Basis for grouping tasks ----------------------------- +3. Theoretical Background +========================= + + + .. BIG FAT WARNING ****************************************************** + + .. warning:: + + This section contains a (not-thorough) summary on deadline/hierarchical + scheduling theory, and how it applies to real-time control groups. + The reader can "safely" skip to Section 4 if only interested in seeing + how the scheduling policy can be used. Anyway, we strongly recommend + to come back here and continue reading (once the urge for testing is + satisfied :P) to be sure of fully understanding all technical details. + + .. ************************************************************************ + +The real-time cgroup scheduler is based upon the **Hierarchical Constant +Bandwidth Server** (HCBS) [1] *Compositional Scheduling Framework* (CSF). A +**CSF** is a framework where global (system-level) timing properties can be +established by composing independently (specified and) analyzed local +(component-level) timing properties [5]. + +For HCBS (related to the Linux kernel), the compositional framework consists of +two parts: + +* The *scheduling components*, which are the basic units of the scheduling. In + the kernel these are the single cgroups along with the tasks that must be run + inside. + +* The *scheduling resources*, which are the CPUs of the machine. + +HCBS is a *hierarchical scheduling framework*, where the scheduling components +form a hierarchy and resources are allocated from parent components to its child +components in the hierarchy. + +The Chapter is organized as follows: **Section 3.1** gives basic real-time +theory definitions that are used throughout the whole section. **Section 3.2** +talks about the HCBS framework, giving a general idea on how this is structured. +**Section 3.3** introduces the MPR model, one of the many models which may be +used for the analysis of the scheduling components and the computation of the +minimum required scheduling resources for a given component. **Section 3.4** +shows the schedulability test for MPR on the HCBS framework. **Section 3.5** +shows how to convert a MPR interface to a HCBS compatible resource reservation +for a component. Finally, **Section 3.6** lists other interesting models which +could be used for the component analysis in HCBS. + +3.1 Basic Definitions +--------------------- + +*We borrow the same definitions given in the* ``sched_deadline`` *document, which +are very briefly summarized here, and new ones, needed by the following content, +are added.* + +A typical real-time task is composed of a repetition of computation phases (task +instances, or jobs) which are activated on a periodic or sporadic fashion. For +our purposes, real-time tasks are characterized by three parameters: + +* Worst Case Execution Time (WCET): the maximum execution time among all jobs. +* Relative Deadline (D): the maximum time each job must be completed, relative + to the release time of the job. +* Inter-Arrival Period (P): the exact/minimum (for periodic/sporadic tasks) time + between each consecutive job. + +3.2 Hierarchical Constant Bandwidth Server (HCBS) [1] +----------------------------------------------------- + +As mentioned, HCBS is a *hierarchical scheduling framework*: + +* The framework hierarchy follows the same hierarchy of cgroups. Cgroups may + have two roles, either bandwidth reservation for children cgroups, or they may + be *live*, i.e. run tasks (but not both). The root cgroup, for the kernel's + implementation of HCBS, acts only as bandwidth reservation (but as written in + this document it has also different uses outside of the hierarchical + framework). +* The cgroup tree is internally flattened, for ease of scheduling, to a + two-level hierarchy, since only the *live* groups are of interest and all the + necessary information for their scheduling lies in their interface (there is + no need for the reservation components). +* The hierarchical framework, now on two levels, consists then of a first level + of cgroups, and a second level of tasks that are run inside these groups. +* The scheduling of components is performed using global Earliest Deadline First + (gEDF), SCHED_DEADLINE in the kernel, following the bandwidth reservation of + each group. +* Whenever a component is scheduled, a local scheduler picks which of the tasks + of the cgroup to run. The scheduling policy is global Fixed Priority (gFP), + SCHED_FIFO/SCHED_RR in the kernel. + +3.3 Multiprocessor Periodic Resource (MPR) model +------------------------------------------------ + +A Multiprocessor Periodic Resource (MPR) model [2] **u = <Pi, Theta, m'>** +specifies that an identical, unit-capacity multiprocessor platform collectively +provides **Theta** units of resource every **Pi** time units, where the +**Theta** time units are supplied with concurrency at most **m'**. + +This theoretical model is one of the many models that can abstract the +interface of our real-time cgroups: let **m'** be the number of CPUs of the +machine, let **Theta** be **m' * <cgroup>/cpu.rt_runtime_us** and **Pi** be +**<cgroup>/cpu.rt_period_us**. -Enabling CONFIG_RT_GROUP_SCHED lets you explicitly allocate real -CPU bandwidth to task groups. +Let's introduce the concept of Supply Bound Function (SBF). A SBF is a function +which outputs a lower bound for the processor supply provided in a given time +interval, given a resource supply model. For a completely dedicated CPU, the SBF +function is simply the identity function, as it will always provide **t** units +of computation for an interval of length **t**. The situation gets slightly more +complicated for the MPR model or any of the other model listed in section 3.6. -This uses the cgroup virtual file system and "<cgroup>/cpu.rt_runtime_us" -to control the CPU time reserved for each control group. +The **SBF(t)** for a MPR model **u = <Pi, Theta, m'>** is:: -For more information on working with control groups, you should read -Documentation/admin-guide/cgroup-v1/cgroups.rst as well. + | 0 if t' < 0 + | + SBF_u(t) = | floor(t' / PI) * Theta + | + max(0, m' * x - (m' * Pi - Theta) if t' >= 0 and 1 <= x <= y + | + | floor(t' / PI) * Theta + | + max(0, m' * x - (m' * Pi - Theta) else + | - (m' - beta) -Group settings are checked against the following limits in order to keep the -configuration schedulable: +where:: - \Sum_{i} runtime_{i} / global_period <= global_runtime / global_period + alpha = floor(Theta / m') + beta = Theta - m' * alpha + t' = t - (Pi - ceil(Theta / m')) + x = t' - (Pi * floor(t' / Pi)) + y = Pi - floor(Theta / m') -For now, this can be simplified to just the following (but see Future plans): +Briefly, this function models that the server's bandwidth is given as late as +possible, so describing the worst case possible for the supplied bandwidth. - \Sum_{i} runtime_{i} <= global_runtime +3.4 Schedulability for MPR on global Fixed-Priority +--------------------------------------------------- +Let's introduce the concept of Demand Bound Function (DBF). A DBF is a function +that, given a taskset, a scheduling algorithm and an interval of time, outputs +the worst resource demand for that interval of time. -3. Future plans -=============== +It is easy to see that, given a DBF and a SBF, we can deem a component/taskset +schedulable if, for every time interval t >= 0, it is possible to demonstrate +that: -There is work in progress to make the scheduling period for each group -("<cgroup>/cpu.rt_period_us") configurable as well. + DBF(t) <= SBF(t) -The constraint on the period is that a subgroup must have a smaller or -equal period to its parent. But realistically its not very useful _yet_ -as its prone to starvation without deadline scheduling. +We have the Supply Bound Function for our given MPR model, so we are missing the +Demand Bound Function for a given taskset that is being scheduled using global +Fixed Priority. -Consider two sibling groups A and B; both have 50% bandwidth, but A's -period is twice the length of B's. +3.4.1 Schedulability Analysis for global Fixed Priority +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -* group A: period=100000us, runtime=50000us +Bertogna, Cirinei and Lipari [6] have derived a schedulability test for global +Fixed Priority (gFP) on multi-processor platforms. In this test (called +*BCL_gFP* test) we can consider all the CPUs to be dedicated to the scheduling. - - this runs for 0.05s once every 0.1s + A taskset **Tau** is schedulable with gFP on a multiprocessor platform + composed of **m'** identical processors if for each task **tau_k in Tau**: -* group B: period= 50000us, runtime=25000us + Sum(for i < k)( min(W_i(D_k), D_k - C_k + 1) ) < m' * (D_k - C_k + 1) - - this runs for 0.025s twice every 0.1s (or once every 0.05 sec). + where **W_i(t)** is the workload of task **tau_i** over a time interval **t**: -This means that currently a while (1) loop in A will run for the full period of -B and can starve B's tasks (assuming they are of lower priority) for a whole -period. + W_i(t) = N_i(t) * C_i + min(C_i, t + D_i - C_i - N_i(t) * P_i) -The next project will be SCHED_EDF (Earliest Deadline First scheduling) to bring -full deadline scheduling to the linux kernel. Deadline scheduling the above -groups and treating end of the period as a deadline will ensure that they both -get their allocated time. + and **N_i(t)** is the number of activations of task **tau_i** that complete in + a time interval **t**: -Implementing SCHED_EDF might take a while to complete. Priority Inheritance is -the biggest challenge as the current linux PI infrastructure is geared towards -the limited static priority levels 0-99. With deadline scheduling you need to -do deadline inheritance (since priority is inversely proportional to the -deadline delta (deadline - now)). + N_i(t) = floor( (t + D_i - C_i) / P_i ) + + while the **min** term is the contribution of the carried-out job in the + interval **t**, i.e. that job that does not completely fit in the interval + **t**, but starts inside the interval after all the jobs that complete. + +3.4.2 From BCL_gFP to the Demand Bound Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We can then derive the DBF from this test: + + DBF_gFP(tau_k) = Sum(for i < k)( min(W_i(D_k), D_k - C_k + 1) ) + m' * (C_k - 1) + +Briefly, the first sum component, the same in the BCL_gFP test, describes the +maximum interference that higher priority task give to the analysed task. The +workload is upperbounded by ``(D_k - C_K + 1)`` because we are only interested +in the interference in the slack time, while for the ``C_k`` time we are +requiring that all the CPUs are fully available, as the single job needs `C_k` +(non overlapping) time units to run. + +The demand bound function from Bertogna et al. is only defined on a single time +(i.e. the deadline of the task in analysis) instead of all possible times as +this is the minimum argument to demonstrate schedulability on global Fixed +Priority. + +3.4.3 Putting it all togheter +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A component **C**, on **m'** processors, running a taskset **Tau = { tau_1 = +(C_1, D_1, P_1), ..., tau_n = (C_n, D_n, P_n) }** of **n** sporadic tasks, is +schedulable under gFP using an MPR model **u = <Pi, Theta, m'>**, if for all +tasks **tau_k in Tau**: + + DBF_gFP(tau_k) <= SBF_u(D_K) + +3.5 From MPR to deadline servers +-------------------------------- + +Since there exist no algorithm to schedule MPR interfaces, a tecnique was +developed to transform MPR interfaces into periodic tasks, so that a +number of periodic servers which respect the tasks requirements can be used for +the scheduling of the MPR interface and associated tasks. + +Let **u = <Pi, Theta, m>** be a MPR interface, let **a = Theta - m * floor(Theta +/ m)**, let **k = floor(a)**. Define a transformation from **u** to a periodic +taskset **Tau_u = { tau_1 = (C_1, D_1, P_1), ..., tau_m' = (C_m', D_m', P_m') +}**, where: + + **tau_1 = ... = tau_k = (floor(Theta / m') + 1, Pi, Pi)** + + **tau_k+1 = (floor(Theta / m') + a - k * floor(a/k), Pi, Pi)** + + **tau_k+2 = ... = tau_m' = (floor(Theta / m'), Pi, Pi)** + +This periodic taskset of servers **Tau_u** can be scheduled on any number of +processors with concurrency at most **m'**. + +For real-time control groups, it is possible to just consider a slightly more +demanding taskset **Tau_u'**, where each task **tau_i** is defined as follows: + + **tau_i = (ceil(Theta / m'), Pi, Pi)** + +3.6 Other models +---------------- + +There exist many other theoretical models in literature which are used to +describe a hierarchical scheduling framework on multi-core architectures. +Notable examples are the Multi Supply Function (MSF) abstraction [3], the +Parallel Supply Function (PSF) abstraction [4] and the Bounded Delay +Multipartition (BDM) [7]. + +3.7 References +-------------- + 1 - L. Abeni, A. Balsini, and T. Cucinotta, “Container-based real-time + scheduling in the Linux kernel,” SIGBED Rev., vol. 16, no. 3, pp. 33-38, + Nov. 2019, doi: 10.1145/3373400.3373405. + 2 - A. Easwaran, I. Shin, and I. Lee, “Optimal virtual cluster-based + multiprocessor scheduling,” Real-Time Syst, vol. 43, no. 1, pp. 25-59, + Sept. 2009, doi: 10.1007/s11241-009-9073-x. + 3 - E. Bini, G. Buttazzo, and M. Bertogna, “The Multi Supply Function + Abstraction for Multiprocessors,” in 2009 15th IEEE International + Conference on Embedded and Real-Time Computing Systems and Applications, + Aug. 2009, pp. 294-302. doi: 10.1109/RTCSA.2009.39. + 4 - E. Bini, B. Marko, and S. K. Baruah, “The Parallel Supply Function + Abstraction for a Virtual Multiprocessor,” in Scheduling, S. Albers, S. K. + Baruah, R. H. Möhring, and K. Pruhs, Eds., in Dagstuhl Seminar Proceedings + (DagSemProc), vol. 10071. Dagstuhl, Germany: Schloss Dagstuhl - + Leibniz-Zentrum für Informatik, 2010, pp. 1-14. doi: + 10.4230/DagSemProc.10071.14. + 5 - I. Shin and I. Lee, “Compositional real-time scheduling framework,” in + 25th IEEE International Real-Time Systems Symposium, Dec. 2004, pp. 57-67. + doi: 10.1109/REAL.2004.15. + 6 - M. Bertogna, M. Cirinei, and G. Lipari, “Schedulability Analysis of Global + Scheduling Algorithms on Multiprocessor Platforms,” IEEE Transactions on + Parallel and Distributed Systems, vol. 20, no. 4, pp. 553-566, Apr. 2009, + doi: 10.1109/TPDS.2008.129. + 7 - G. Lipari and E. Bini, “A Framework for Hierarchical Scheduling on + Multiprocessors: From Application Requirements to Run-Time Allocation,” in + 2010 31st IEEE Real-Time Systems Symposium, Nov. 2010, pp. 249-258. doi: + 10.1109/RTSS.2010.12. + + +4. Using Real-Time cgroups +========================== + +4.1 CGroup Setup +---------------- -This means the whole PI machinery will have to be reworked - and that is one of -the most complex pieces of code we have. +The following is a brief guide to the use of Real-Time Control Groups. + +Of course, real-time control groups require mounting of the cgroup file system. +We have decided to only support cgroups v2, so make sure you mount the v2 +controller for the cgroup hierarchy. + +Additionally the real-time cgroups require the CPU controller for the cgroups to +be enabled:: + + # Assume the cgroup file system is mounted at /sys/fs/cgroup + > echo "+cpu" > /sys/fs/cgroup/cgroup.subtree_control + +The CPU controller can only be mounted if there is no SCHED_FIFO/SCHED_RR task +scheduled in any cgroup other than the root control group. + +The root control group has no bandwidth allocated by default, so make sure to +allocate some bandwidth so that it can be used by the other cgroups. More on +that in the following section... + +4.2 Bandwidth Allocation for groups +----------------------------------- + +Allocating bandwidth to a cgroup is a fundamental step to run real-time +workload. The cgroup filesystem exposes two files: + +* ``<cgroup>/cpu.rt_runtime_us``: which specifies the cgroups' runtime in + microseconds. +* ``<cgroup>/cpu.rt_period_us``: which specifies the cgroups' period in + microseconds. + +Both files are readable and writable, and their default value is zero. By +definition, the specified runtime must be always less than or equal to the +period. Additionally, an admission test checks if the bandwidth invariant is +respected (i.e. sum of children's bandwidth <= parent's bandwidth). + +The root control group files instead control and reserve the SCHED_DEADLINE +bandwidth allocated to real-time cgroups, since real-time groups compete and +share the same bandwidth allocated to SCHED_DEADLINE tasks. + +4.3 Running real-time tasks in groups +------------------------------------- + +To run tasks in real-time groups it is just necessary to change a tasks +scheduling policy to SCHED_FIFO/SCHED_RR and migrate it into the group. If the +group is not allowed to run real-time tasks because of incorrect configuration, +either migrating a SCHED_FIFO/SCHED_RR task into the group or changing +scheduling policy to a task already inside the group will fail:: + + # assume there is a task of PID 42 running + # change its scheduling policy to SCHED_FIFO, priority 99 + > chrt -f -p 99 42 + + # migrate the task to a cgroup + > echo 42 > /sys/fs/cgroup/<my-cgroup>/cgroup.procs + +4.4 Special case: the root control group +---------------------------------------- + +The root cgroup is special, compared to the other cgroups, as its tasks are not +managed by the HCBS algorithm, rather they just use the original +SCHED_FIFO/SCHED_RR policies (as if CONFIG_RT_GROUP_SCHED was disabled). As +mentioned, its bandwidth files are just used to control how much of the +SCHED_DEADLINE bandwidth is allocated to cgroups. + +4.5 Guarantees and Special Behaviours +------------------------------------- + +Real-time cgroups are run at the same priority level of SCHED_DEADLINE tasks. +Since this is the highest priority scheduling policy, and since the Constant +Bandwidth Server (CBS) enforces that the specified bandwidth requirements for +both groups and tasks cannot be overrun, real-time groups have the same +guarantees that SCHED_DEADLINE tasks have, i.e. they will be necessarily +supplied by the amount of bandwidth requested (whenever the admission tests +pass). + +This means that, since SCHED_FIFO/SCHED_RR tasks (scheduled in the root control +group) are not subject to bandwidth controls, they are run at a lower priority +than the cgroups' counterparts. Nonetheless, a minimum amount of bandwidth, if +reserved, will always be available to run SCHED_FIFO/SCHED_RR workloads in the +root cgroup, while they will be able to use more runtime if any of the +SCHED_DEADLINE tasks or servers use less than their specified amount of +bandwidth. SCHED_OTHER tasks are instead scheduled as normal, at lower priority +than real-time workloads. + +The aforementioned behaviour differs from the preceding RT_GROUP_SCHED +implementation, but this is necessary to give actual guarantees to the amount of +bandwidth given to rt-cgroups.
--
2.51.0
RFC PATCH v4 27/28 sched/rt: Add debug BUG_ONs for pre-migration code
In reply to Yuri AndriaccioAdd debug BUG_ONs in rt_queue_push/pull_task(s). Can be safely added after all the pre-migration patches.
These are extra asserts which are only useful to debug the kernel code and are not meant to be part of the final patchset.
--- kernel/sched/rt.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sched/rt.c +6 -0
@@ -319,6 +319,9 @@ static inline void rt_queue_push_tasks(struct rt_rq *rt_rq) {struct rq *rq =served_rq_of_rt_rq (rt_rq); +BUG_ON (rt_rq ==NULL ); +BUG_ON (rq !=cpu_rq (rq->cpu)); +if (!has_pushable_tasks (rt_rq))return ;
@@ -329,6 +332,9 @@ static inline void rt_queue_pull_task(struct rt_rq *rt_rq) {struct rq *rq =served_rq_of_rt_rq (rt_rq); +BUG_ON (rt_rq ==NULL ); +BUG_ON (rq !=cpu_rq (rq->cpu)); +queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); }
--
2.51.0
RFC PATCH v4 28/28 sched/rt: Add debug BUG_ONs in migration code.
In reply to Yuri AndriaccioAdd debug BUG_ONs in group specific migration functions. Can be safely added after all the migration patches.
These are extra asserts which are only useful to debug the kernel code and are not meant to be part of the final patchset.
--- kernel/sched/rt.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
kernel/sched/rt.c +25 -0
@@ -349,6 +349,9 @@ static void rt_queue_push_from_group(struct rt_rq *rt_rq)struct rq *rq =served_rq_of_rt_rq (rt_rq);struct rq *global_rq =cpu_rq (rq->cpu); +BUG_ON (rt_rq ==NULL ); +BUG_ON (rq == global_rq); +if (global_rq->rq_to_push_from)return ;
@@ -366,6 +369,10 @@ static void rt_queue_pull_to_group(struct rt_rq *rt_rq)struct rq *global_rq =cpu_rq (rq->cpu);struct sched_dl_entity *dl_se =dl_group_of (rt_rq); +BUG_ON (rt_rq ==NULL ); +BUG_ON (!is_dl_group (rt_rq)); +BUG_ON (rq == global_rq); +if (dl_se->dl_throttled || global_rq->rq_to_pull_to)return ;
@@ -1393,6 +1400,8 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)*/ static int push_rt_task (struct rq *rq,bool pull) { +BUG_ON (is_dl_group (&rq->rt)); +struct task_struct *next_task;struct rq *lowest_rq;int ret =0 ;
@@ -1689,6 +1698,8 @@ void rto_push_irq_work_func(struct irq_work *work)static void pull_rt_task (struct rq *this_rq) { +BUG_ON (is_dl_group (&this_rq->rt)); +int this_cpu = this_rq->cpu, cpu;bool resched = false;struct task_struct *p, *push_task;
@@ -1813,6 +1824,8 @@ static int group_find_lowest_rt_rq(struct task_struct *task, struct rt_rq *task_int prio, lowest_prio;int cpu, this_cpu =smp_processor_id (); +BUG_ON (task->sched_task_group != task_rt_rq->tg); +if (task->nr_cpus_allowed ==1 )return -1 ;/* No other targets possible */
@@ -1915,6 +1928,8 @@ static struct rt_rq *group_find_lock_lowest_rt_rq(struct task_struct *task, strustruct sched_dl_entity *lowest_dl_se;int tries, cpu; +BUG_ON (task->sched_task_group != rt_rq->tg); +for (tries =0 ; tries <RT_MAX_TRIES ; tries++) { cpu =group_find_lowest_rt_rq (task, rt_rq);
@@ -1971,6 +1986,8 @@ static struct rt_rq *group_find_lock_lowest_rt_rq(struct task_struct *task, strustatic int group_push_rt_task (struct rt_rq *rt_rq,bool pull) { +BUG_ON (!is_dl_group (rt_rq)); +struct rq *rq =rq_of_rt_rq (rt_rq);struct task_struct *next_task;struct rq *lowest_rq;
@@ -2090,6 +2107,8 @@ static int group_push_rt_task(struct rt_rq *rt_rq, bool pull)static void group_pull_rt_task (struct rt_rq *this_rt_rq) { +BUG_ON (!is_dl_group (this_rt_rq)); +struct rq *this_rq =rq_of_rt_rq (this_rt_rq);int this_cpu = this_rq->cpu, cpu;bool resched = false;
@@ -2202,6 +2221,9 @@ static void group_push_rt_tasks_callback(struct rq *global_rq) {struct rt_rq *rt_rq = &global_rq->rq_to_push_from->rt; +BUG_ON (global_rq->rq_to_push_from ==NULL ); +BUG_ON (served_rq_of_rt_rq (rt_rq) == global_rq); +if ((rt_rq->rt_nr_running >1 ) || (dl_group_of (rt_rq)->dl_throttled ==1 )) {
@@ -2216,6 +2238,9 @@ static void group_pull_rt_task_callback(struct rq *global_rq) {struct rt_rq *rt_rq = &global_rq->rq_to_pull_to->rt; +BUG_ON (global_rq->rq_to_pull_to ==NULL ); +BUG_ON (served_rq_of_rt_rq (rt_rq) == global_rq); +group_pull_rt_task (rt_rq); global_rq->rq_to_pull_to =NULL ; }
--
2.51.0