Hierarchical Constant Bandwidth Server
Subsystems (2)
v3 cover Hierarchical Constant Bandwidth Server
Hello,
This is the v3 for Hierarchical Constant Bandwidth Server, aiming at replacing the current RT_GROUP_SCHED mechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ . You can find the previous versions of this patchset at the bottom of the page, in particular version 1 which talks in more detail what this patchset is all about and how it is implemented.
This v3 version further reworks some of the patches as suggested by Juri Lelli. While most of the work is refactorings, the following were also changed:
- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/
- A left-over check which prevented execution of some of wakeup_preempt code has been removed.
- Cgroup pull code was erroneusly comparing cgroup with non-cgroup tasks, now it has been fixed.
- The allocation/deallocation code for rt cgroups has been checked and reworked to make sure that resources are managed correctly in all the code paths.
- Some signatures of cgroup migration related functions where changed to match more closely to their non-group counterparts.
- Descriptions and documentation were added where necessary, in particular for preemption rules in wakeup_preempt.
For this v3 version we've also polished the testing system we are using and made it public for testers to run on their own machines. The source code can be found at https://github.com/Yurand2000/HCBS-Test-Suite , along with a README that explains how to use it. Nonetheless I've reported a description of the tools and instruction later in the page.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Summary of the patches:
1-4) Preparation patches, so that the RT classes' code can be used both
for normal and cgroup scheduling.
5-15) Implementation of HCBS, no migration and only one level hierarchy.
The old RT_GROUP_SCHED code is removed.
16-17) Remove cgroups v1 in favour of v2.
18) Add support for deeper hierarchies.
19-24) Add support for tasks migration.
Updates from v2:
- Rebase to latest tip/master.
- Remove fair-servers' bw reclaiming.
- Fix a check which prevented execution of wakeup_preempt code.
- Fix a priority check in group_pull_rt_task between tasks of different groups.
- Rework allocation/deallocation code for rt-cgroups.
- Update signatures for some group related migration functions.
- Add documentation for wakeup_preempt preemption rules.
Updates from v1:
- Rebase to latest tip/master.
- Add migration code.
- Split big patches for more readability.
- Refactor code to use guarded locks where applicable.
- Remove unnecessary patches from v1 which have been addressed differently by mainline updates.
- Remove unnecessary checks and general code cleanup.
Notes:
Task migration support needs some extra work to reduce its invasiveness, especially patches 21-22.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Testing v3:
The HCBS mechanism has been evaluated on several syntetic tests which are designed to stress the HCBS scheduler and verify that non-interference and mathematical schedulability guarantees are really enforced by the scheduling algorithm.
The test suite currently runs different categories of tests:
- Constraints, which are tasked to assert that hard constraints, such as schedulability conditions, are respected.
- Regression, to check that HCBS does not break anything that already exists.
- Stress, to repeatedly invoke the scheduler in all the exposed interfaces, with the goal to detect bugs and more importantly race conditions.
- Time, simple benchmarks to assert that the dl_servers work correctly, i.e. they allocate the correct amount of bandwidth, and that migration code allows to fully utilize the cgroup's allocated bw.
- Taskset: given a set of (generated) periodic tasks and their bandwidth requirements, schedulability analyses are performed to decide whether or not a given hardware configuration can run the taskset. In particular, for each taskset, a HCBS's cgroup configuration along with the number of necessary CPUs is generated. These are mathematically guaranteed to be schedulable. The next step of this test suite is to configure cgroups as computed and to run the taskset, to verify that the HCBS implementation works as intended and that the scheduling overheads are within reasonable bounds.
The source code can be found at https://github.com/Yurand2000/HCBS-Test-Suite . The README file should explain most if not all questions, but I'm writing briefly the pipeline to run these tests here:
- Get the HCBS patch up and running. Any kernel/disto should work effortlessly.
- Get, compile and _install_ the tests.
- Download the additional taskset files and extract them in the _install_ folder. You can find them here:
https://github.com/Yurand2000/HCBS-Test-Suite/releases/tag/250926
- Run the run_tests.sh full script, to run the whole test suite.
Expect a total runtime of ~3 hours. The script will automatically mount the cgroup and debug filesystems (if not already mounted) and will move all the already running SCHED_FIFO/SCHED_RR tasks in the root cgroup, so that the cgroups' CPU controller can be mounted. It will additionally try to reserve all the possible rt-bandwidth for cgroups (i.e. 90%) to run all the later tests, so make sure that there are no running SCHED_DEADLINE tasks if the script fails to setup.
Some tests specifically need a minimum amount of CPU cores, up to a maximum of eight. If your machine has less CPUs then the tests will simply be skipped.
Notes:
The tasksets minimal requirements were computed using a closed-source software, explaining why the tasksets are supplied separately. A open-source analyser is being written to update this step in the future and also allow for more customization for the testers.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Future Work:
While we wait for more comments, and expect stuff to break, we will work on completing the currently partial/untested, implementation of HCBS with different runtimes per CPU, instead of having the same runtime allocated on all CPUs, to include it in a future RCF.
Future patches:
- HCBS with different runtimes per CPU.
- capacity aware bandwidth reservation.
- enable/disable dl_servers when a CPU goes online/offline.
Have a nice day,
Yuri
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Yuri Andriaccio (6):
sched/rt: Disable RT_GROUP_SCHED
sched/rt: Add rt-cgroups' dl-servers operations. sched/rt: Update task event callbacks for HCBS scheduling sched/rt: Allow zeroing the runtime of the root control group sched/rt: Remove support for cgroups-v1
sched/core: Execute enqueued balance callbacks when migrating task
betweeen cgroups
luca abeni (18):
sched/deadline: Do not access dl_se->rq directly sched/deadline: Distinct between dl_rq and my_q
sched/rt: Pass an rt_rq instead of an rq where needed sched/rt: Move some functions from rt.c to sched.h sched/rt: Introduce HCBS specific structs in task_group sched/core: Initialize root_task_group
sched/deadline: Add dl_init_tg
sched/rt: Add {alloc/free}_rt_sched_group
sched/deadline: Account rt-cgroups bandwidth in deadline tasks
schedulability tests.
sched/rt: Update rt-cgroup schedulability checks sched/rt: Remove old RT_GROUP_SCHED data structures sched/core: Cgroup v2 support
sched/deadline: Allow deeper hierarchies of RT cgroups sched/rt: Add rt-cgroup migration
sched/rt: Add HCBS migration related checks and function calls sched/deadline: Make rt-cgroup's servers pull tasks on timer
replenishment
sched/deadline: Fix HCBS migrations on server stop sched/core: Execute enqueued balance callbacks when changing allowed
CPUs
include/linux/sched.h | 10 +- kernel/sched/autogroup.c | 4 +- kernel/sched/core.c | 65 +- kernel/sched/deadline.c | 251 +++- kernel/sched/debug.c | 6 - kernel/sched/fair.c | 6 +- kernel/sched/rt.c | 3069 +++++++++++++++++++------------------- kernel/sched/sched.h | 150 +- kernel/sched/syscalls.c | 6 +- 9 files changed, 1850 insertions(+), 1717 deletions(-)
base-commit: cec1e6e5d1ab33403b809f79cd20d6aff124ccfe
Signature · 1 line
2.51.0
RFC PATCH v3 1/24 sched/deadline: Do not access dl_se->rq directly
From: luca abeni <[email protected]>
Make deadline.c code access the runqueue of a scheduling entity saved in the sched_dl_entity data structure. This allows future patches to save different runqueues in sched_dl_entity other than the global runqueues.
--- kernel/sched/deadline.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-)
kernel/sched/deadline.c +14 -13
@@ -874,7 +874,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)* and arm the defer timer. */ if (dl_se->dl_defer && !dl_se->dl_defer_running && -dl_time_before (rq_clock (dl_se->rq), dl_se->deadline - dl_se->runtime)) { +dl_time_before (rq_clock (rq), dl_se->deadline - dl_se->runtime)) {if (!is_dl_boosted (dl_se) && dl_se->server_has_tasks (dl_se)) {/*
@@ -1184,11 +1184,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_* of time. The dl_server_min_res serves as a limit to avoid * forwarding the timer for a too small amount of time. */ -if (dl_time_before (rq_clock (dl_se->rq), +if (dl_time_before (rq_clock (rq), (dl_se->deadline - dl_se->runtime - dl_server_min_res))) {/* reset the defer timer */ - fw = dl_se->deadline -rq_clock (dl_se->rq) - dl_se->runtime; + fw = dl_se->deadline -rq_clock (rq) - dl_se->runtime;hrtimer_forward_now (timer,ns_to_ktime (fw));return HRTIMER_RESTART ;
@@ -1199,7 +1199,7 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_enqueue_dl_entity (dl_se,ENQUEUE_REPLENISH ); -if (!dl_task (dl_se->rq->curr) ||dl_entity_preempt (dl_se, &dl_se->rq->curr->dl)) +if (!dl_task (rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl))resched_curr (rq);__push_dl_task (rq, rf);
@@ -1467,7 +1467,7 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64hrtimer_try_to_cancel (&dl_se->dl_timer); -replenish_dl_new_period (dl_se, dl_se->rq); +replenish_dl_new_period (dl_se, rq);/* * Not being able to start the timer seems problematic. If it could not
@@ -1581,21 +1581,22 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)/* 0 runtime = fair server disabled */ if (dl_se->dl_runtime) { dl_se->dl_server_idle =0 ; -update_curr_dl_se (dl_se->rq, dl_se, delta_exec); +update_curr_dl_se (rq_of_dl_se (dl_se), dl_se, delta_exec); } }void dl_server_start (struct sched_dl_entity *dl_se) { -struct rq *rq = dl_se->rq; +struct rq *rq;if (!dl_server (dl_se) || dl_se->dl_server_active)return ; dl_se->dl_server_active =1 ;enqueue_dl_entity (dl_se,ENQUEUE_WAKEUP ); -if (!dl_task (dl_se->rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl)) -resched_curr (dl_se->rq); + rq =rq_of_dl_se (dl_se); +if (!dl_task (rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl)) +resched_curr (rq); }void dl_server_stop (struct sched_dl_entity *dl_se)
@@ -1651,9 +1652,9 @@ void sched_init_dl_servers(void)WARN_ON (dl_server (dl_se)); -dl_server_apply_params (dl_se, runtime, period,1 ); - dl_se->dl_server =1 ; +WARN_ON (dl_server_apply_params (dl_se, runtime, period,1 )); + dl_se->dl_defer =1 ;setup_new_dl_entity (dl_se); }
@@ -1678,7 +1679,7 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio {u64 old_bw = init ?0 :to_ratio (dl_se->dl_period, dl_se->dl_runtime);u64 new_bw =to_ratio (period, runtime); -struct rq *rq = dl_se->rq; +struct rq *rq =rq_of_dl_se (dl_se);int cpu =cpu_of (rq);struct dl_bw *dl_b;unsigned long cap;
@@ -1755,7 +1756,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) p =dl_task_of (dl_se); rq =task_rq_lock (p, &rf); }else { - rq = dl_se->rq; + rq =rq_of_dl_se (dl_se);rq_lock (rq, &rf); }
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 01/24] sched/deadline: Do not access dl_se->rq directly
Hello,
Quoted 10 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Make
deadline.ccode access the runqueue of a scheduling entity saved in thesched_dl_entitydata structure. This allows future patches to save different runqueues insched_dl_entityother than the global runqueues.Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>--- kernel/sched/deadline.c | 27 ++++++++++++++-------------
Nit, I believe there are a couple of dl_se->rq direct usages in fair.c left after this change. The are outside deadline and they are indeed modified anyway by next patch. Just for your consideration in case you want to do something about them from completeness.
...
kernel/sched/deadline.c void sched_init_dl_servers(void) · line 1657
Quoted 9 lines
kernel/sched/deadline.c +2
-2@@ -1651,9 +1652,9 @@ void sched_init_dl_servers(void)WARN_ON (dl_server (dl_se)); -dl_server_apply_params (dl_se, runtime, period,1 ); - dl_se->dl_server =1 ; +WARN_ON (dl_server_apply_params (dl_se, runtime, period,1 )); +
The move after dl_server is initialized is indeed needed, but maybe we should mention the small functional change (also the additon of WARN) in the changelog.
Best,
Juri
RFC PATCH v3 2/24 sched/deadline: Distinct between dl_rq and my_q
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Create two fields for runqueues in sched_dl_entity to make a distinction between the global runqueue and the runqueue which the dl_server serves.
--- include/linux/sched.h | 6 ++++-- kernel/sched/deadline.c | 11 +++++++---- kernel/sched/fair.c | 6 +++--- kernel/sched/sched.h | 3 ++- 4 files changed, 16 insertions(+), 10 deletions(-)
include/linux/sched.h +4 -2
@@ -727,12 +727,14 @@ struct sched_dl_entity {* Bits for DL-server functionality. Also see the comment near * dl_server_update(). * -* @rq the runqueue this server is for +* @dl_rq the runqueue on which this entity is (to be) queued +* @my_q the runqueue "owned" by this entity * * @server_has_tasks() returns true if @server_pick return a * runnable task. */ -struct rq *rq; +struct dl_rq *dl_rq; +struct rq *my_q;dl_server_has_tasks_f server_has_tasks;dl_server_pick_f server_pick_task;
kernel/sched/deadline.c +7 -4
@@ -75,11 +75,12 @@ static inline struct rq *rq_of_dl_rq(struct dl_rq *dl_rq)static inline struct rq *rq_of_dl_se (struct sched_dl_entity *dl_se) { -struct rq *rq = dl_se->rq; +struct rq *rq;if (!dl_server (dl_se)) rq =task_rq (dl_task_of (dl_se)); - +else + rq = container_of(dl_se->dl_rq, struct rq, dl);return rq; }
@@ -1625,11 +1626,13 @@ static bool dl_server_stopped(struct sched_dl_entity *dl_se)return false; } -void dl_server_init (struct sched_dl_entity *dl_se,struct rq *rq, +void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq, +struct rq *served_rq,dl_server_has_tasks_f has_tasks,dl_server_pick_f pick_task) { - dl_se->rq = rq; + dl_se->dl_rq = dl_rq; + dl_se->my_q = served_rq; dl_se->server_has_tasks = has_tasks; dl_se->server_pick_task = pick_task; }
kernel/sched/fair.c +3 -3
@@ -8861,12 +8861,12 @@ static struct task_struct *__pick_next_task_fair(struct rq *rq, struct task_strustatic bool fair_server_has_tasks (struct sched_dl_entity *dl_se) { -return !!dl_se->rq->cfs.nr_queued; +return !!dl_se->my_q->cfs.nr_queued; }static struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se) { -return pick_task_fair (dl_se->rq); +return pick_task_fair (dl_se->my_q); }void fair_server_init (struct rq *rq)
@@ -8875,7 +8875,7 @@ void fair_server_init(struct rq *rq)init_dl_entity (dl_se); -dl_server_init (dl_se, rq, fair_server_has_tasks, fair_server_pick_task); +dl_server_init (dl_se, &rq->dl, rq, fair_server_has_tasks, fair_server_pick_task); }/*
kernel/sched/sched.h +2 -1
@@ -382,7 +382,8 @@ extern s64 dl_scaled_delta_exec(struct rq *rq, struct sched_dl_entity *dl_se, s6extern void dl_server_update (struct sched_dl_entity *dl_se,s64 delta_exec);extern void dl_server_start (struct sched_dl_entity *dl_se);extern void dl_server_stop (struct sched_dl_entity *dl_se); -extern void dl_server_init (struct sched_dl_entity *dl_se,struct rq *rq, +extern void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq, +struct rq *served_rq,dl_server_has_tasks_f has_tasks,dl_server_pick_f pick_task);extern void sched_init_dl_servers (void );
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 02/24] sched/deadline: Distinct between dl_rq and my_q
Hello,
Quoted 7 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Create two fields for runqueues in
sched_dl_entityto make a distinction between the global runqueue and the runqueue which thedl_serverserves.Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>
The change looks good to me. Maybe we only want to be a little more verbose in the changelog, e.g. the following?
Split the single rq pointer in sched_dl_entity into two separate pointers, following the existing pattern used by sched_rt_entity:
- dl_rq: Points to the deadline runqueue where this entity is queued
- my_q: Points to the runqueue that this entity serves (for servers)
This distinction is currently redundant for the fair_server (both point to the same CPU's structures), but is essential for future RT cgroup support where deadline servers will be queued on the global dl_rq while serving tasks from cgroup-specific runqueues.
Update rq_of_dl_se() to use container_of() to recover the rq from dl_rq, and update fair.c to explicitly use my_q when accessing the served runqueue.
Thanks,
Juri
RFC PATCH v3 3/24 sched/rt: Pass an rt_rq instead of an rq where needed
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make rt.c code access the runqueue through the rt_rq data structure rather than passing an rq pointer directly. This allows future patches to define rt_rq data structures which do not refer only to the global runqueue, but also to local cgroup runqueues (rt_rq is not always equal to &rq->rt).
--- kernel/sched/rt.c | 99 ++++++++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 45 deletions(-)
kernel/sched/rt.c +54 -45
@@ -370,9 +370,9 @@ static inline void rt_clear_overload(struct rq *rq)cpumask_clear_cpu (rq->cpu, rq->rd->rto_mask); } -static inline int has_pushable_tasks (struct rq *rq) +static inline int has_pushable_tasks (struct rt_rq *rt_rq) { -return !plist_head_empty (&rq->rt.pushable_tasks); +return !plist_head_empty (&rt_rq->pushable_tasks); } static DEFINE_PER_CPU(struct balance_callback, rt_push_head);
@@ -381,50 +381,54 @@ static DEFINE_PER_CPU(struct balance_callback, rt_pull_head);static void push_rt_tasks (struct rq *);static void pull_rt_task (struct rq *); -static inline void rt_queue_push_tasks (struct rq *rq) +static inline void rt_queue_push_tasks (struct rt_rq * rt_rq) { -if (!has_pushable_tasks (rq)) + struct rq *rq = container_of(rt_rq, struct rq, rt); + +if (!has_pushable_tasks (rt_rq))return ;queue_balance_callback (rq, &per_cpu (rt_push_head, rq->cpu), push_rt_tasks); } -static inline void rt_queue_pull_task (struct rq *rq) +static inline void rt_queue_pull_task (struct rt_rq * rt_rq) { + struct rq *rq = container_of(rt_rq, struct rq, rt); +queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); } -static void enqueue_pushable_task (struct rq *rq,struct task_struct *p) +static void enqueue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) { -plist_del (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);plist_node_init (&p->pushable_tasks, p->prio); -plist_add (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_add (&p->pushable_tasks, &rt_rq->pushable_tasks);/* Update the highest prio pushable task */ -if (p->prio < rq->rt.highest_prio.next) - rq->rt.highest_prio.next = p->prio; +if (p->prio < rt_rq->highest_prio.next) + rt_rq->highest_prio.next = p->prio; -if (!rq->rt.overloaded) { -rt_set_overload (rq); - rq->rt.overloaded =1 ; +if (!rt_rq->overloaded) { +rt_set_overload (rq_of_rt_rq (rt_rq)); + rt_rq->overloaded =1 ; } } -static void dequeue_pushable_task (struct rq *rq,struct task_struct *p) +static void dequeue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) { -plist_del (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);/* Update the new highest prio pushable task */ -if (has_pushable_tasks (rq)) { - p =plist_first_entry (&rq->rt.pushable_tasks, +if (has_pushable_tasks (rt_rq)) { + p =plist_first_entry (&rt_rq->pushable_tasks, struct task_struct, pushable_tasks); - rq->rt.highest_prio.next = p->prio; + rt_rq->highest_prio.next = p->prio; }else { - rq->rt.highest_prio.next =MAX_RT_PRIO -1 ; + rt_rq->highest_prio.next =MAX_RT_PRIO -1 ; -if (rq->rt.overloaded) { -rt_clear_overload (rq); - rq->rt.overloaded =0 ; +if (rt_rq->overloaded) { +rt_clear_overload (rq_of_rt_rq (rt_rq)); + rt_rq->overloaded =0 ; } } }
@@ -1431,6 +1435,7 @@ static void enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags) { struct sched_rt_entity *rt_se = &p->rt; +struct rt_rq *rt_rq =rt_rq_of_se (rt_se);if (flags &ENQUEUE_WAKEUP ) rt_se->timeout =0 ;
@@ -1444,17 +1449,18 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)return ;if (!task_current (rq, p) && p->nr_cpus_allowed >1 ) -enqueue_pushable_task (rq, p); +enqueue_pushable_task (rt_rq, p); }static bool dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) {struct sched_rt_entity *rt_se = &p->rt; +struct rt_rq *rt_rq =rt_rq_of_se (rt_se);update_curr_rt (rq);dequeue_rt_entity (rt_se, flags); -dequeue_pushable_task (rq, p); +dequeue_pushable_task (rt_rq, p);return true; }
@@ -1639,14 +1645,14 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)static inline void set_next_task_rt (struct rq *rq,struct task_struct *p,bool first) {struct sched_rt_entity *rt_se = &p->rt; -struct rt_rq *rt_rq = &rq->rt; +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt); p->se.exec_start =rq_clock_task (rq);if (on_rt_rq (&p->rt))update_stats_wait_end_rt (rt_rq, rt_se);/* The running task is never eligible for pushing */ -dequeue_pushable_task (rq, p); +dequeue_pushable_task (rt_rq, p);if (!first)return ;
@@ -1659,7 +1665,7 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool fif (rq->donor->sched_class != &rt_sched_class)update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); -rt_queue_push_tasks (rq); +rt_queue_push_tasks (rt_rq); }static struct sched_rt_entity *pick_next_rt_entity (struct rt_rq *rt_rq)
@@ -1710,7 +1716,7 @@ static struct task_struct *pick_task_rt(struct rq *rq)static void put_prev_task_rt (struct rq *rq,struct task_struct *p,struct task_struct *next) {struct sched_rt_entity *rt_se = &p->rt; -struct rt_rq *rt_rq = &rq->rt; +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt);if (on_rt_rq (&p->rt))update_stats_wait_start_rt (rt_rq, rt_se);
@@ -1726,7 +1732,7 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s* if it is still active */ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 ) -enqueue_pushable_task (rq, p); +enqueue_pushable_task (rt_rq, p); }/* Only try algorithms three times */
@@ -1736,16 +1742,16 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s* Return the highest pushable rq's task, which is suitable to be executed * on the CPU, NULL otherwise */ -static struct task_struct *pick_highest_pushable_task (struct rq *rq,int cpu) +static struct task_struct *pick_highest_pushable_task (struct rt_rq *rt_rq,int cpu) { -struct plist_head *head = &rq->rt.pushable_tasks; +struct plist_head *head = &rt_rq->pushable_tasks;struct task_struct *p; -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (rt_rq))return NULL ;plist_for_each_entry (p, head, pushable_tasks) { -if (task_is_pushable (rq, p, cpu)) +if (task_is_pushable (rq_of_rt_rq (rt_rq), p, cpu))return p; }
@@ -1845,14 +1851,15 @@ static int find_lowest_rq(struct task_struct *task)return -1 ; } -static struct task_struct *pick_next_pushable_task (struct rq *rq) +static struct task_struct *pick_next_pushable_task (struct rt_rq *rt_rq) { +struct rq *rq =rq_of_rt_rq (rt_rq);struct task_struct *p; -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (rt_rq))return NULL ; - p =plist_first_entry (&rq->rt.pushable_tasks, + p =plist_first_entry (&rt_rq->pushable_tasks, struct task_struct, pushable_tasks);BUG_ON (rq->cpu !=task_cpu (p));
@@ -1905,7 +1912,7 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)*/ if (unlikely (is_migration_disabled (task) || !cpumask_test_cpu (lowest_rq->cpu, &task->cpus_mask) || - task !=pick_next_pushable_task (rq))) { + task !=pick_next_pushable_task (&rq->rt))) {double_unlock_balance (rq, lowest_rq); lowest_rq =NULL ;
@@ -1939,7 +1946,7 @@ static int push_rt_task(struct rq *rq, bool pull)if (!rq->rt.overloaded)return 0 ; - next_task =pick_next_pushable_task (rq); + next_task =pick_next_pushable_task (&rq->rt);if (!next_task)return 0 ;
@@ -2014,7 +2021,7 @@ static int push_rt_task(struct rq *rq, bool pull)* run-queue and is also still the next task eligible for * pushing. */ - task =pick_next_pushable_task (rq); + task =pick_next_pushable_task (&rq->rt);if (task == next_task) {/* * The task hasn't migrated, and is still the next
@@ -2202,7 +2209,7 @@ void rto_push_irq_work_func(struct irq_work *work)* We do not need to grab the lock to check for has_pushable_tasks. * When it gets updated, a check is made if a push is possible. */ -if (has_pushable_tasks (rq)) { +if (has_pushable_tasks (&rq->rt)) {raw_spin_rq_lock (rq);while (push_rt_task (rq, true)) ;
@@ -2231,6 +2238,7 @@ static void pull_rt_task(struct rq *this_rq)int this_cpu = this_rq->cpu, cpu;bool resched = false;struct task_struct *p, *push_task; +struct rt_rq *src_rt_rq;struct rq *src_rq;int rt_overload_count =rt_overloaded (this_rq);
@@ -2260,6 +2268,7 @@ static void pull_rt_task(struct rq *this_rq)continue ; src_rq =cpu_rq (cpu); + src_rt_rq = &src_rq->rt;/* * Don't bother taking the src_rq->lock if the next highest
@@ -2268,7 +2277,7 @@ static void pull_rt_task(struct rq *this_rq)* logically higher, the src_rq will push this task away. * And if its going logically lower, we do not care */ -if (src_rq->rt.highest_prio.next >= +if (src_rt_rq->highest_prio.next >= this_rq->rt.highest_prio.curr)continue ;
@@ -2284,7 +2293,7 @@ static void pull_rt_task(struct rq *this_rq)* We can pull only a task, which is pushable * on its rq, and no others. */ - p =pick_highest_pushable_task (src_rq, this_cpu); + p =pick_highest_pushable_task (src_rt_rq, this_cpu);/* * Do we have an RT task that preempts
@@ -2390,7 +2399,7 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)if (!task_on_rq_queued (p) || rq->rt.rt_nr_running)return ; -rt_queue_pull_task (rq); +rt_queue_pull_task (rt_rq_of_se (&p->rt)); }void __init init_sched_rt_class (void )
@@ -2426,7 +2435,7 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)*/ if (task_on_rq_queued (p)) {if (p->nr_cpus_allowed >1 && rq->rt.overloaded) -rt_queue_push_tasks (rq); +rt_queue_push_tasks (rt_rq_of_se (&p->rt));if (p->prio < rq->donor->prio &&cpu_online (cpu_of (rq)))resched_curr (rq); }
@@ -2448,7 +2457,7 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)* may need to pull tasks to this runqueue. */ if (oldprio < p->prio) -rt_queue_pull_task (rq); +rt_queue_pull_task (rt_rq_of_se (&p->rt));/* * If there's a higher priority task waiting to run
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 03/24] sched/rt: Pass an rt_rq instead of an rq where needed
Hello,
Quoted 30 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Make
rt.ccode access the runqueue through thert_rqdata structure rather than passing an rq pointer directly. This allows future patches to definert_rqdata structures which do not refer only to the global runqueue, but also to local cgroup runqueues (rt_rqis not always equal to&rq->rt).Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>--- kernel/sched/rt.c | 99 ++++++++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 45 deletions(-)kernel/sched/rt.c +2
-2@@ -370,9 +370,9 @@ static inline void rt_clear_overload(struct rq *rq)cpumask_clear_cpu (rq->cpu, rq->rd->rto_mask); } -static inline int has_pushable_tasks (struct rq *rq) +static inline int has_pushable_tasks (struct rt_rq *rt_rq) { -return !plist_head_empty (&rq->rt.pushable_tasks); +return !plist_head_empty (&rt_rq->pushable_tasks); } static DEFINE_PER_CPU(struct balance_callback, rt_push_head);
kernel/sched/rt.c static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); · line 384
Quoted 5 lines
kernel/sched/rt.c +1
-1@@ -381,50 +381,54 @@ static DEFINE_PER_CPU(struct balance_callback, rt_pull_head);static void push_rt_tasks (struct rq *);static void pull_rt_task (struct rq *); -static inline void rt_queue_push_tasks (struct rq *rq) +static inline void rt_queue_push_tasks (struct rt_rq * rt_rq)
Nit, remove space ^^^
kernel/sched/rt.c static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); · line 394
Quoted 12 lines
kernel/sched/rt.c +4
-2{ -if (!has_pushable_tasks (rq)) + struct rq *rq = container_of(rt_rq, struct rq, rt); + +if (!has_pushable_tasks (rt_rq))return ;queue_balance_callback (rq, &per_cpu (rt_push_head, rq->cpu), push_rt_tasks); } -static inline void rt_queue_pull_task (struct rq *rq) +static inline void rt_queue_pull_task (struct rt_rq * rt_rq)
Nit, remove space ^^^
kernel/sched/rt.c static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); · line 396
Quoted 2 lines
kernel/sched/rt.c +1
-0{ + struct rq *rq = container_of(rt_rq, struct rq, rt);
Still need to check with the full series (and it's OK up until here), but I guess this needs special attention with tasks inside groups to make sure we don't queue callbacks in rqs representing groups?
kernel/sched/rt.c static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); · line 399
Quoted 3 lines
kernel/sched/rt.c +1
-0+queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); }
Thanks,
Juri
RFC PATCH v3 4/24 sched/rt: Move some functions from rt.c to sched.h
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make the following functions be non-static and move them in sched.h, so that they can be used also in other source files:
- rt_task_of()
- rq_of_rt_rq()
- rt_rq_of_se()
- rq_of_rt_se()
There are no functional changes. This is needed by future patches.
--- kernel/sched/rt.c | 52 ------------------------------------------ kernel/sched/sched.h | 54 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 52 deletions(-)
kernel/sched/rt.c +0 -52
@@ -168,34 +168,6 @@ static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)#define rt_entity_is_task (rt_se) (!(rt_se)->my_q) -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ -WARN_ON_ONCE (!rt_entity_is_task (rt_se)); - - return container_of(rt_se, struct task_struct, rt); -} - -static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) -{ -/* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */ -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -} - -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) -{ -WARN_ON (!rt_group_sched_enabled () && rt_se->rt_rq->tg != &root_task_group); -return rt_se->rt_rq; -} - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct rt_rq *rt_rq = rt_se->rt_rq; - -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -} -void unregister_rt_sched_group (struct task_group *tg) {if (!rt_group_sched_enabled ())
@@ -296,30 +268,6 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)#define rt_entity_is_task (rt_se) (1) -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ - return container_of(rt_se, struct task_struct, rt); -} - -static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) -{ - return container_of(rt_rq, struct rq, rt); -} - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct task_struct *p =rt_task_of (rt_se); - -return task_rq (p); -} - -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) -{ -struct rq *rq =rq_of_rt_se (rt_se); - -return &rq->rt; -} -void unregister_rt_sched_group (struct task_group *tg) { }void free_rt_sched_group (struct task_group *tg) { }
kernel/sched/sched.h +54 -0
60 lines not on this page · raw message
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 04/24] sched/rt: Move some functions from rt.c to sched.h
Hello,
Quoted 23 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Make the following functions be non-static and move them in
sched.h, so that they can be used also in other source files:
-rt_task_of()
-rq_of_rt_rq()
-rt_rq_of_se()
-rq_of_rt_se()There are no functional changes. This is needed by future patches.
Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>--- kernel/sched/rt.c | 52 ------------------------------------------ kernel/sched/sched.h | 54 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 52 deletions(-)kernel/sched/rt.c +0
-0
kernel/sched/rt.c static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b) · line 169
Quoted 2 lines
kernel/sched/rt.c +0
-0@@ -168,34 +168,6 @@ static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)#define rt_entity_is_task (rt_se) (!(rt_se)->my_q)
Do we want to move this macro as well, so that rt_task_of() doesn't need to change?
Quoted 6 lines
kernel/sched/rt.c +0
-6-static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ -WARN_ON_ONCE (!rt_entity_is_task (rt_se)); - - return container_of(rt_se, struct task_struct, rt); -}
...
kernel/sched/rt.c
Quoted 7 lines
kernel/sched/rt.c +7
-0+#ifdef CONFIG_RT_GROUP_SCHED +static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) +{ +WARN_ON_ONCE (rt_se->my_q); + + return container_of(rt_se, struct task_struct, rt); +}
Thanks,
Juri
RFC PATCH v3 5/24 sched/rt: Disable RT_GROUP_SCHED
In reply to Yuri AndriaccioDisable the old RT_GROUP_SCHED scheduler. Note that this does not completely remove all the RT_GROUP_SCHED functionality, just unhooks it and removes most of the relevant functions. Some of the RT_GROUP_SCHED functions are kept because they will be adapted for the HCBS scheduling.
--- kernel/sched/core.c | 6 - kernel/sched/deadline.c | 34 -- kernel/sched/debug.c | 6 - kernel/sched/rt.c | 848 ++-------------------------------------- kernel/sched/sched.h | 11 +- kernel/sched/syscalls.c | 13 - 6 files changed, 26 insertions(+), 892 deletions(-)
kernel/sched/core.c +0 -6
@@ -8721,11 +8721,6 @@ void __init sched_init(void)init_defrootdomain (); -#ifdef CONFIG_RT_GROUP_SCHED -init_rt_bandwidth (&root_task_group.rt_bandwidth, -global_rt_period (),global_rt_runtime ()); -#endif /* CONFIG_RT_GROUP_SCHED */ -#ifdef CONFIG_CGROUP_SCHED task_group_cache =KMEM_CACHE (task_group,0 );
@@ -8777,7 +8772,6 @@ void __init sched_init(void)* starts working after scheduler_running, which is not the case * yet. */ - rq->rt.rt_runtime =global_rt_runtime ();init_tg_rt_entry (&root_task_group, &rq->rt,NULL , i,NULL );#endif rq->sd =NULL ;
kernel/sched/deadline.c +0 -34
@@ -1508,40 +1508,6 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64if (!is_leftmost (dl_se, &rq->dl))resched_curr (rq); } - -/* -* The fair server (sole dl_server) does not account for real-time -* workload because it is running fair work. -*/ -if (dl_se == &rq->fair_server) -return ; - -#ifdef CONFIG_RT_GROUP_SCHED -/* -* Because -- for now -- we share the rt bandwidth, we need to -* account our runtime there too, otherwise actual rt tasks -* would be able to exceed the shared quota. -* -* Account to the root rt group for now. -* -* The solution we're working towards is having the RT groups scheduled -* using deadline servers -- however there's a few nasties to figure -* out before that can happen. -*/ -if (rt_bandwidth_enabled ()) { -struct rt_rq *rt_rq = &rq->rt; - -raw_spin_lock (&rt_rq->rt_runtime_lock); -/* -* We'll let actual RT tasks worry about the overflow here, we -* have our own CBS to keep us inline; only account when RT -* bandwidth is relevant. -*/ -if (sched_rt_bandwidth_account (rt_rq)) - rt_rq->rt_time += delta_exec; -raw_spin_unlock (&rt_rq->rt_runtime_lock); - } -#endif /* CONFIG_RT_GROUP_SCHED */ }/*
kernel/sched/debug.c +0 -6
@@ -890,12 +890,6 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)PU (rt_nr_running); -#ifdef CONFIG_RT_GROUP_SCHED -P (rt_throttled); -PN (rt_time); -PN (rt_runtime); -#endif -#undef PN#undef PU#undef P
kernel/sched/rt.c +23 -825
1,073 lines not on this page · raw message
kernel/sched/sched.h +3 -8
@@ -787,7 +787,7 @@ struct scx_rq {static inline int rt_bandwidth_enabled (void ) { -return sysctl_sched_rt_runtime >=0 ; +return 0 ; }/* RT IPI pull logic requires IRQ_WORK */
@@ -827,7 +827,7 @@ struct rt_rq {static inline bool rt_rq_is_runnable (struct rt_rq *rt_rq) { -return rt_rq->rt_queued && rt_rq->rt_nr_running; +return rt_rq->rt_nr_running; }/* Deadline class' related fields in a runqueue */
@@ -2543,7 +2543,7 @@ static inline bool sched_dl_runnable(struct rq *rq)static inline bool sched_rt_runnable (struct rq *rq) { -return rq->rt.rt_queued >0 ; +return rq->rt.rt_nr_running >0 ; }static inline bool sched_fair_runnable (struct rq *rq)
@@ -2654,9 +2654,6 @@ extern void resched_curr(struct rq *rq);extern void resched_curr_lazy (struct rq *rq);extern void resched_cpu (int cpu); -extern void init_rt_bandwidth (struct rt_bandwidth *rt_b,u64 period,u64 runtime); -extern bool sched_rt_bandwidth_account (struct rt_rq *rt_rq); -extern void init_dl_entity (struct sched_dl_entity *dl_se);#define BW_SHIFT 20
@@ -3043,8 +3040,6 @@ extern bool sched_smp_initialized;#ifdef CONFIG_RT_GROUP_SCHED static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) { -WARN_ON_ONCE (rt_se->my_q); - return container_of(rt_se, struct task_struct, rt); }
kernel/sched/syscalls.c +0 -13
@@ -626,19 +626,6 @@ int __sched_setscheduler(struct task_struct *p, change:if (user) { -#ifdef CONFIG_RT_GROUP_SCHED -/* -* Do not allow real-time tasks into groups that have no runtime -* assigned. -*/ -if (rt_group_sched_enabled () && -rt_bandwidth_enabled () &&rt_policy (policy) && -task_group (p)->rt_bandwidth.rt_runtime ==0 && - !task_group_is_autogroup (task_group (p))) { - retval = -EPERM ; - goto unlock; - } -#endif /* CONFIG_RT_GROUP_SCHED */ if (dl_bandwidth_enabled () &&dl_policy (policy) && !(attr->sched_flags &SCHED_FLAG_SUGOV )) {cpumask_t *span = rq->rd->span;
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 05/24] sched/rt: Disable RT_GROUP_SCHED
Hello,
Quoted 106 lines · Yuri Andriaccio
Disable the old
RT_GROUP_SCHEDscheduler. Note that this does not completely remove all theRT_GROUP_SCHEDfunctionality, just unhooks it and removes most of the relevant functions. Some of theRT_GROUP_SCHEDfunctions are kept because they will be adapted for the HCBS scheduling.Signed-off-by Yuri Andriaccio <[email protected]>--- kernel/sched/core.c | 6 - kernel/sched/deadline.c | 34 -- kernel/sched/debug.c | 6 - kernel/sched/rt.c | 848 ++-------------------------------------- kernel/sched/sched.h | 11 +- kernel/sched/syscalls.c | 13 - 6 files changed, 26 insertions(+), 892 deletions(-)kernel/sched/core.c +0
-6@@ -8721,11 +8721,6 @@ void __init sched_init(void)init_defrootdomain (); -#ifdef CONFIG_RT_GROUP_SCHED -init_rt_bandwidth (&root_task_group.rt_bandwidth, -global_rt_period (),global_rt_runtime ()); -#endif /* CONFIG_RT_GROUP_SCHED */ -#ifdef CONFIG_CGROUP_SCHED task_group_cache =KMEM_CACHE (task_group,0 );@@ -8777,7 +8772,6 @@ void __init sched_init(void)* starts working after scheduler_running, which is not the case * yet. */ - rq->rt.rt_runtime =global_rt_runtime ();init_tg_rt_entry (&root_task_group, &rq->rt,NULL , i,NULL );#endif rq->sd =NULL ;kernel/sched/deadline.c +0
-34@@ -1508,40 +1508,6 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64if (!is_leftmost (dl_se, &rq->dl))resched_curr (rq); } - -/* -* The fair server (sole dl_server) does not account for real-time -* workload because it is running fair work. -*/ -if (dl_se == &rq->fair_server) -return ; - -#ifdef CONFIG_RT_GROUP_SCHED -/* -* Because -- for now -- we share the rt bandwidth, we need to -* account our runtime there too, otherwise actual rt tasks -* would be able to exceed the shared quota. -* -* Account to the root rt group for now. -* -* The solution we're working towards is having the RT groups scheduled -* using deadline servers -- however there's a few nasties to figure -* out before that can happen. -*/ -if (rt_bandwidth_enabled ()) { -struct rt_rq *rt_rq = &rq->rt; - -raw_spin_lock (&rt_rq->rt_runtime_lock); -/* -* We'll let actual RT tasks worry about the overflow here, we -* have our own CBS to keep us inline; only account when RT -* bandwidth is relevant. -*/ -if (sched_rt_bandwidth_account (rt_rq)) - rt_rq->rt_time += delta_exec; -raw_spin_unlock (&rt_rq->rt_runtime_lock); - } -#endif /* CONFIG_RT_GROUP_SCHED */ }/* kernel/sched/debug.c +0
-6@@ -890,12 +890,6 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)PU (rt_nr_running); -#ifdef CONFIG_RT_GROUP_SCHED -P (rt_throttled); -PN (rt_time); -PN (rt_runtime); -#endif -#undef PN#undef PU#undef Pkernel/sched/rt.c +0
-0
kernel/sched/rt.c line 1
Quoted 1 line
kernel/sched/rt.c +1
-0@@ -1,3 +1,4 @@ +#pragma GCC diagnostic ignored "-Wunused-function"
Uh oh, guess this goes away with later patches? It's anyway not very nice unfortunately and it also breaks the SPDX licence first requirement.
kernel/sched/rt.c line 4
Quoted 3 lines
kernel/sched/rt.c +0
-0// SPDX-License-Identifier: GPL-2.0 /* * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
So, this leaves only a skeleton of the current RT_GROUP implementation. I believe cgroup ABI will still be there, but it won't have an effect. Since this is part of an atomic all-or-nothing set of changes, maybe it's OK? Guess people can get confused if for some reason they end up using a kernel with patches only up to this one. :-)
Best,
Juri
RFC PATCH v3 6/24 sched/rt: Introduce HCBS specific structs in task_group
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Each task_group manages a number of new objects:
- a sched_dl_entity/dl_server for each CPU
- a dl_bandwidth object to keep track of its allocated dl_bandwidth
--- kernel/sched/sched.h | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
kernel/sched/sched.h +16 -2
@@ -319,6 +319,13 @@ struct rt_bandwidth {unsigned int rt_period_active; }; +struct dl_bandwidth { +raw_spinlock_t dl_runtime_lock; +u64 dl_runtime; +u64 dl_period; +}; + +static inline int dl_bandwidth_enabled (void ) {return sysctl_sched_rt_runtime >=0 ;
@@ -467,10 +474,17 @@ struct task_group {#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED +/* +* Each task group manages a different scheduling entity per CPU, i.e. a +* different deadline server, and a runqueue per CPU. All the dl-servers +* share the same dl_bandwidth object. +*/ struct sched_rt_entity **rt_se; +struct sched_dl_entity **dl_se;struct rt_rq **rt_rq;struct rt_bandwidth rt_bandwidth; +struct dl_bandwidth dl_bandwidth;#endif struct scx_task_group scx;
@@ -817,12 +831,12 @@ struct rt_rq {raw_spinlock_t rt_runtime_lock;unsigned int rt_nr_boosted; - -struct rq *rq;/* this is always top-level rq, cache? */ #endif #ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */ #endif + +struct rq *rq;/* cgroup's runqueue if the rt_rq entity belongs to a cgroup, otherwise top-level rq */ };static inline bool rt_rq_is_runnable (struct rt_rq *rt_rq)
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 06/24] sched/rt: Introduce HCBS specific structs in task_group
Hello,
Quoted 14 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Each
task_groupmanages a number of new objects:
- a sched_dl_entity/dl_server for each CPU
- adl_bandwidthobject to keep track of its allocateddl_bandwidthCo-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>---
...
Quoted 20 lines
kernel/sched/sched.h +7
-0@@ -467,10 +474,17 @@ struct task_group {#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED +/* +* Each task group manages a different scheduling entity per CPU, i.e. a +* different deadline server, and a runqueue per CPU. All the dl-servers +* share the same dl_bandwidth object. +*/ struct sched_rt_entity **rt_se; +struct sched_dl_entity **dl_se;struct rt_rq **rt_rq;struct rt_bandwidth rt_bandwidth; +struct dl_bandwidth dl_bandwidth;#endif struct scx_task_group scx;@@ -817,12 +831,12 @@ struct rt_rq {raw_spinlock_t rt_runtime_lock;
kernel/sched/sched.h struct rt_rq { · line 839
Quoted 10 lines
kernel/sched/sched.h +2
-2unsigned int rt_nr_boosted; - -struct rq *rq;/* this is always top-level rq, cache? */ #endif #ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */ #endif + +struct rq *rq;/* cgroup's runqueue if the rt_rq entity belongs to a cgroup, otherwise top-level rq */
The comment really wants to span multiple lines, maybe above the pointer.
Thanks,
Juri
RFC PATCH v3 7/24 sched/core: Initialize root_task_group
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add the initialization function for task_group's dl_servers. Initialize the default bandwidth for rt-cgroups and the root control group. Add utility functions to check (and get) if a rt_rq entity is connected to a real-time cgroup.
--- kernel/sched/autogroup.c | 4 ++-- kernel/sched/core.c | 9 +++++++-- kernel/sched/deadline.c | 8 ++++++++ kernel/sched/rt.c | 18 ++++++++---------- kernel/sched/sched.h | 32 +++++++++++++++++++++++++++++--- 5 files changed, 54 insertions(+), 17 deletions(-)
kernel/sched/autogroup.c +2 -2
@@ -52,7 +52,7 @@ static inline void autogroup_destroy(struct kref *kref)#ifdef CONFIG_RT_GROUP_SCHED /* We've redirected RT tasks to the root task group... */ - ag->tg->rt_se =NULL ; + ag->tg->dl_se =NULL ; ag->tg->rt_rq =NULL ;#endif sched_release_group (ag->tg);
@@ -109,7 +109,7 @@ static inline struct autogroup *autogroup_create(void)* the policy change to proceed. */ free_rt_sched_group (tg); - tg->rt_se = root_task_group.rt_se; + tg->dl_se = root_task_group.dl_se; tg->rt_rq = root_task_group.rt_rq;#endif /* CONFIG_RT_GROUP_SCHED */ tg->autogroup = ag;
kernel/sched/core.c +7 -2
@@ -8710,7 +8710,7 @@ void __init sched_init(void)scx_tg_init (&root_task_group);#endif /* CONFIG_EXT_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED - root_task_group.rt_se = (struct sched_rt_entity **)ptr; + root_task_group.dl_se = (struct sched_dl_entity **)ptr; ptr += nr_cpu_ids *sizeof (void **); root_task_group.rt_rq = (struct rt_rq **)ptr;
@@ -8721,6 +8721,11 @@ void __init sched_init(void)init_defrootdomain (); +#ifdef CONFIG_RT_GROUP_SCHED +init_dl_bandwidth (&root_task_group.dl_bandwidth, +global_rt_period (),global_rt_runtime ()); +#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_CGROUP_SCHED task_group_cache =KMEM_CACHE (task_group,0 );
@@ -8772,7 +8777,7 @@ void __init sched_init(void)* starts working after scheduler_running, which is not the case * yet. */ -init_tg_rt_entry (&root_task_group, &rq->rt,NULL , i,NULL ); +init_tg_rt_entry (&root_task_group, rq,NULL , i,NULL );#endif rq->sd =NULL ; rq->rd =NULL ;
kernel/sched/deadline.c +8 -0
@@ -513,6 +513,14 @@ static inline int is_leftmost(struct sched_dl_entity *dl_se, struct dl_rq *dl_rqstatic void init_dl_rq_bw_ratio (struct dl_rq *dl_rq); +void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime) +{ +raw_spin_lock_init (&dl_b->dl_runtime_lock); + dl_b->dl_period = period; + dl_b->dl_runtime = runtime; +} + +void init_dl_bw (struct dl_bw *dl_b) {raw_spin_lock_init (&dl_b->lock);
kernel/sched/rt.c +8 -10
@@ -97,18 +97,16 @@ void free_rt_sched_group(struct task_group *tg)return ; } -void init_tg_rt_entry (struct task_group *tg,struct rt_rq *rt_rq, -struct sched_rt_entity *rt_se,int cpu, -struct sched_rt_entity *parent) +void init_tg_rt_entry (struct task_group *tg,struct rq *served_rq, +struct sched_dl_entity *dl_se,int cpu, +struct sched_dl_entity *parent) { -struct rq *rq =cpu_rq (cpu); + served_rq->rt.highest_prio.curr =MAX_RT_PRIO -1 ; + served_rq->rt.rq =cpu_rq (cpu); + served_rq->rt.tg = tg; - rt_rq->highest_prio.curr =MAX_RT_PRIO -1 ; - rt_rq->rq = rq; - rt_rq->tg = tg; - - tg->rt_rq[cpu] = rt_rq; - tg->rt_se[cpu] = rt_se; + tg->rt_rq[cpu] = &served_rq->rt; + tg->dl_se[cpu] = dl_se; }int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent)
kernel/sched/sched.h +29 -3
@@ -576,9 +576,9 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);extern void unthrottle_cfs_rq (struct cfs_rq *cfs_rq);extern bool cfs_task_bw_constrained (struct task_struct *p); -extern void init_tg_rt_entry (struct task_group *tg,struct rt_rq *rt_rq, -struct sched_rt_entity *rt_se,int cpu, -struct sched_rt_entity *parent); +extern void init_tg_rt_entry (struct task_group *tg,struct rq *s_rq, +struct sched_dl_entity *rt_se,int cpu, +struct sched_dl_entity *parent);extern int sched_group_set_rt_runtime (struct task_group *tg,long rt_runtime_us);extern int sched_group_set_rt_period (struct task_group *tg,u64 rt_period_us);extern long sched_group_rt_runtime (struct task_group *tg);
@@ -2668,6 +2668,7 @@ extern void resched_curr(struct rq *rq);extern void resched_curr_lazy (struct rq *rq);extern void resched_cpu (int cpu); +void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime);extern void init_dl_entity (struct sched_dl_entity *dl_se);#define BW_SHIFT 20
@@ -3077,6 +3078,21 @@ static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); return rt_rq->rq; } + +static inline int is_dl_group (struct rt_rq *rt_rq) +{ +return rt_rq->tg != &root_task_group; +} + +/* +* Return the scheduling entity of this group of tasks. +*/ +static inline struct sched_dl_entity *dl_group_of (struct rt_rq *rt_rq) +{ +BUG_ON (!is_dl_group (rt_rq)); + +return rt_rq->tg->dl_se[cpu_of (rt_rq->rq)]; +}#else static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) {
@@ -3101,6 +3117,16 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)return &rq->rt; } + +static inline int is_dl_group (struct rt_rq *rt_rq) +{ +return 0 ; +} + +static inline struct sched_dl_entity *dl_group_of (struct rt_rq *rt_rq) +{ +return NULL ; +}#endif DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq,
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 07/24] sched/core: Initialize root_task_group
Hello,
Quoted 6 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Add the initialization function for task_group's
dl_servers. Initialize the default bandwidth for rt-cgroups and the root control group. Add utility functions to check (and get) if art_rqentity is connected to a real-time cgroup.
It's a bit of a nit, but this patch does quite more than $SUBJECT. So, I'd say please either reword $SUBJECT or consider splitting it further.
...
Quoted 6 lines
kernel/sched/sched.h +2
-0@@ -3077,6 +3078,21 @@ static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group);return rt_rq->rq; } + +static inline int is_dl_group (struct rt_rq *rt_rq)
kernel/sched/sched.h static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se) · line 3092
Quoted 10 lines
kernel/sched/sched.h +10
-0+{ +return rt_rq->tg != &root_task_group; +} + +/* +* Return the scheduling entity of this group of tasks. +*/ +static inline struct sched_dl_entity *dl_group_of (struct rt_rq *rt_rq) +{ +BUG_ON (!is_dl_group (rt_rq));
BUG_ON() should generally be avoided, if possible. Could a WARN_ON_ONCE() plus returning NULL work here?
kernel/sched/sched.h static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se) · line 3095
Quoted 3 lines
kernel/sched/sched.h +3
-0+ +return rt_rq->tg->dl_se[cpu_of (rt_rq->rq)]; +}
Thanks,
Juri
RFC PATCH v3 8/24 sched/deadline: Add dl_init_tg
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add dl_init_tg to initialize and/or update a rt-cgroup dl_server and to also account the allocated bandwidth.
--- kernel/sched/deadline.c | 33 +++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 2 files changed, 34 insertions(+)
kernel/sched/deadline.c +33 -0
@@ -340,6 +340,39 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)cancel_dl_timer (dl_se, &dl_se->inactive_timer); } +#ifdef CONFIG_RT_GROUP_SCHED +void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period) +{ + struct rq *rq = container_of(dl_se->dl_rq, struct rq, dl); +int is_active; +u64 new_bw; + +raw_spin_rq_lock_irq (rq); + is_active = dl_se->my_q->rt.rt_nr_running >0 ; + +update_rq_clock (rq); +dl_server_stop (dl_se); + + new_bw =to_ratio (dl_se->dl_period, dl_se->dl_runtime); +dl_rq_change_utilization (rq, dl_se, new_bw); + + dl_se->dl_runtime = rt_runtime; + dl_se->dl_deadline = rt_period; + dl_se->dl_period = rt_period; + + dl_se->runtime =0 ; + dl_se->deadline =0 ; + + dl_se->dl_bw = new_bw; + dl_se->dl_density = new_bw; + +if (is_active) +dl_server_start (dl_se); + +raw_spin_rq_unlock_irq (rq); +} +#endif +static void dl_change_utilization (struct task_struct *p,u64 new_bw) {WARN_ON_ONCE (p->dl.flags &SCHED_FLAG_SUGOV );
kernel/sched/sched.h +1 -0
@@ -394,6 +394,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,dl_server_has_tasks_f has_tasks, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void ); +extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period);extern void dl_server_update_idle_time (struct rq *rq,struct task_struct *p);
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 08/24] sched/deadline: Add dl_init_tg
Hello,
Quoted 22 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Add
dl_init_tgto initialize and/or update a rt-cgroupdl_serverand to also account the allocated bandwidth.Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>--- kernel/sched/deadline.c | 33 +++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 2 files changed, 34 insertions(+)kernel/sched/deadline.c +0
-0
kernel/sched/deadline.c void cancel_inactive_timer(struct sched_dl_entity *dl_se) · line 344
Quoted 5 lines
kernel/sched/deadline.c +2
-0@@ -340,6 +340,39 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)cancel_dl_timer (dl_se, &dl_se->inactive_timer); } +#ifdef CONFIG_RT_GROUP_SCHED +void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period)
This is not called from anywhere at this point, is it? I wonder if we want to introduce dead code for later usage or if it would make more sense to introduce it together with the calling sites.
kernel/sched/deadline.c void cancel_inactive_timer(struct sched_dl_entity *dl_se) · line 356
Quoted 12 lines
kernel/sched/deadline.c +12
-0+{ + struct rq *rq = container_of(dl_se->dl_rq, struct rq, dl); +int is_active; +u64 new_bw; + +raw_spin_rq_lock_irq (rq); + is_active = dl_se->my_q->rt.rt_nr_running >0 ; + +update_rq_clock (rq); +dl_server_stop (dl_se); + + new_bw =to_ratio (dl_se->dl_period, dl_se->dl_runtime);
Is this correct? Not sure yet where/how dl_init_tg() is called, but don't we want to compute new_bw using rt_runtime and rt_period?
Thanks,
Juri
RFC PATCH v3 9/24 sched/rt: Add {alloc/free}_rt_sched_group
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add allocation and deallocation code for rt-cgroups.
Declare dl_server specific functions (no implementation yet), needed by the allocation code.
--- kernel/sched/rt.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+)
kernel/sched/rt.c +89 -0
@@ -93,8 +93,43 @@ void unregister_rt_sched_group(struct task_group *tg)void free_rt_sched_group (struct task_group *tg) { +int i; +unsigned long flags; +struct rq *served_rq; +if (!rt_group_sched_enabled ())return ; + +if (!tg->dl_se || !tg->rt_rq) +return ; + +for_each_possible_cpu (i) { +if (!tg->dl_se[i] || !tg->rt_rq[i]) +continue ; + +/* +* Shutdown the dl_server and free it +* +* Since the dl timer is going to be cancelled, +* we risk to never decrease the running bw... +* Fix this issue by changing the group runtime +* to 0 immediately before freeing it. +*/ +dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); + +raw_spin_rq_lock_irqsave (cpu_rq (i), flags); +BUG_ON (tg->rt_rq[i]->rt_nr_running); +hrtimer_cancel (&tg->dl_se[i]->dl_timer); +raw_spin_rq_unlock_irqrestore (cpu_rq (i), flags); +kfree (tg->dl_se[i]); + +/* Free the local per-cpu runqueue */ + served_rq = container_of(tg->rt_rq[i], struct rq, rt); +kfree (served_rq); + } + +kfree (tg->rt_rq); +kfree (tg->dl_se); } void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq,
@@ -109,11 +144,65 @@ void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, tg->dl_se[cpu] = dl_se; } +static bool rt_server_has_tasks (struct sched_dl_entity *dl_se) +{ +return false; +} + +static struct task_struct *rt_server_pick (struct sched_dl_entity *dl_se) +{ +return NULL ; +} +int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent) { +struct rq *s_rq; +struct sched_dl_entity *dl_se; +int i; +if (!rt_group_sched_enabled ())return 1 ; + tg->rt_rq =kcalloc (nr_cpu_ids,sizeof (struct rt_rq *),GFP_KERNEL ); +if (!tg->rt_rq) +return 0 ; + + tg->dl_se =kcalloc (nr_cpu_ids,sizeof (dl_se),GFP_KERNEL ); +if (!tg->dl_se) { +kfree (tg->rt_rq); + tg->rt_rq =NULL ; +return 0 ; + } + +init_dl_bandwidth (&tg->dl_bandwidth,0 ,0 ); + +for_each_possible_cpu (i) { + s_rq =kzalloc_node (sizeof (struct rq ), +GFP_KERNEL ,cpu_to_node (i)); +if (!s_rq) +return 0 ; + + dl_se =kzalloc_node (sizeof (struct sched_dl_entity ), +GFP_KERNEL ,cpu_to_node (i)); +if (!dl_se) { +kfree (s_rq); +return 0 ; + } + +init_rt_rq (&s_rq->rt); +init_dl_entity (dl_se); + dl_se->dl_runtime = tg->dl_bandwidth.dl_runtime; + dl_se->dl_period = tg->dl_bandwidth.dl_period; + dl_se->dl_deadline = dl_se->dl_period; + dl_se->dl_bw =to_ratio (dl_se->dl_period, dl_se->dl_runtime); + dl_se->dl_density =to_ratio (dl_se->dl_period, dl_se->dl_runtime); + dl_se->dl_server =1 ; + +dl_server_init (dl_se, &cpu_rq (i)->dl, s_rq, rt_server_has_tasks, rt_server_pick); + +init_tg_rt_entry (tg, s_rq, dl_se, i, parent->dl_se[i]); + } +return 1 ; }
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 09/24] sched/rt: Add {alloc/free}_rt_sched_group
Hello,
Quoted 41 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Add allocation and deallocation code for rt-cgroups.
Declaredl_serverspecific functions (no implementation yet), needed by the allocation code.Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>--- kernel/sched/rt.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+)kernel/sched/rt.c +14
-0@@ -93,8 +93,43 @@ void unregister_rt_sched_group(struct task_group *tg)void free_rt_sched_group (struct task_group *tg) { +int i; +unsigned long flags; +struct rq *served_rq; +if (!rt_group_sched_enabled ())return ; + +if (!tg->dl_se || !tg->rt_rq) +return ; + +for_each_possible_cpu (i) { +if (!tg->dl_se[i] || !tg->rt_rq[i]) +continue ; + +/* +* Shutdown the dl_server and free it
kernel/sched/rt.c void unregister_rt_sched_group(struct task_group *tg) · line 121
Quoted 10 lines
kernel/sched/rt.c +10
-0+* +* Since the dl timer is going to be cancelled, +* we risk to never decrease the running bw... +* Fix this issue by changing the group runtime +* to 0 immediately before freeing it. +*/ +dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); + +raw_spin_rq_lock_irqsave (cpu_rq (i), flags); +BUG_ON (tg->rt_rq[i]->rt_nr_running);
Crashing is probably a bit harsh. What didn't happen that should have happened if there are still tasks in the group at this point? Can we maybe WARN and try to recover w/o crashing?
Quoted 11 lines
kernel/sched/rt.c +11
-0+hrtimer_cancel (&tg->dl_se[i]->dl_timer); +raw_spin_rq_unlock_irqrestore (cpu_rq (i), flags); +kfree (tg->dl_se[i]); + +/* Free the local per-cpu runqueue */ + served_rq = container_of(tg->rt_rq[i], struct rq, rt); +kfree (served_rq); + } + +kfree (tg->rt_rq); +kfree (tg->dl_se);
...
Quoted 4 lines
int
alloc_rt_sched_group(struct task_group *tg, struct task_group *parent){kernel/sched/rt.c +2
-0+struct rq *s_rq; +struct sched_dl_entity *dl_se;
kernel/sched/rt.c void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, · line 170
Quoted 10 lines
kernel/sched/rt.c +7
-0+int i; +if (!rt_group_sched_enabled ())return 1 ; + tg->rt_rq =kcalloc (nr_cpu_ids,sizeof (struct rt_rq *),GFP_KERNEL ); +if (!tg->rt_rq) +return 0 ; + + tg->dl_se =kcalloc (nr_cpu_ids,sizeof (dl_se),GFP_KERNEL );
Nit. Maybe sizeof(struct sched_dl_entity *) is more clear and consistent with the above (current form is still correct, of course).
Quoted 10 lines
kernel/sched/rt.c +10
-0+if (!tg->dl_se) { +kfree (tg->rt_rq); + tg->rt_rq =NULL ; +return 0 ; + } + +init_dl_bandwidth (&tg->dl_bandwidth,0 ,0 ); + +for_each_possible_cpu (i) { + s_rq = kzalloc_node(sizeof(struct rq),
kernel/sched/rt.c void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, · line 190
Quoted 10 lines
kernel/sched/rt.c +10
-0+ GFP_KERNEL, cpu_to_node(i)); +if (!s_rq) +return 0 ; + + dl_se =kzalloc_node (sizeof (struct sched_dl_entity ), +GFP_KERNEL ,cpu_to_node (i)); +if (!dl_se) { +kfree (s_rq); +return 0 ; + }
Are we not leaking if allocation failure happens mid way during this for loop? Do we free data structures allocated for previous CPUs?
Quoted 7 lines
kernel/sched/rt.c +7
-0+ +init_rt_rq (&s_rq->rt); +init_dl_entity (dl_se); + dl_se->dl_runtime = tg->dl_bandwidth.dl_runtime; + dl_se->dl_period = tg->dl_bandwidth.dl_period; + dl_se->dl_deadline = dl_se->dl_period; + dl_se->dl_bw =to_ratio (dl_se->dl_period, dl_se->dl_runtime);
kernel/sched/rt.c void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, · line 207
Quoted 10 lines
kernel/sched/rt.c +8
-0+ dl_se->dl_density =to_ratio (dl_se->dl_period, dl_se->dl_runtime); + dl_se->dl_server =1 ; + +dl_server_init (dl_se, &cpu_rq (i)->dl, s_rq, rt_server_has_tasks, rt_server_pick); + +init_tg_rt_entry (tg, s_rq, dl_se, i, parent->dl_se[i]); + } +return 1 ; }
Thanks,
Juri
RFC PATCH v3 10/24 sched/deadline: Account rt-cgroups bandwidth in deadline tasks schedulability tests.
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Account the rt-cgroups hierarchy's reserved bandwidth in the schedulability test of deadline entities. This mechanism allows to completely reserve portion of the rt-bandwidth to rt-cgroups even if they do not use all of it.
Account for the rt-cgroups' reserved bandwidth also when changing the total dedicated bandwidth for real time tasks.
--- kernel/sched/deadline.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
kernel/sched/deadline.c +18 -3
@@ -213,8 +213,15 @@ void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)static inline bool __dl_overflow (struct dl_bw *dl_b,unsigned long cap,u64 old_bw,u64 new_bw) { +u64 dl_groups_root =0 ; + +#ifdef CONFIG_RT_GROUP_SCHED + dl_groups_root =to_ratio (root_task_group.dl_bandwidth.dl_period, + root_task_group.dl_bandwidth.dl_runtime); +#endif return dl_b->bw !=-1 && -cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw; +cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw + +cap_scale (dl_groups_root, cap); } static inline
@@ -3150,10 +3157,16 @@ int sched_dl_global_validate(void)u64 period =global_rt_period ();u64 new_bw =to_ratio (period, runtime);u64 cookie = ++dl_cookie; +u64 dl_groups_root =0 ;struct dl_bw *dl_b; -int cpu, cpus, ret =0 ; +int cpu, cap, cpus, ret =0 ;unsigned long flags; +#ifdef CONFIG_RT_GROUP_SCHED + dl_groups_root =to_ratio (root_task_group.dl_bandwidth.dl_period, + root_task_group.dl_bandwidth.dl_runtime); +#endif +/* * Here we want to check the bandwidth not being set to some * value smaller than the currently allocated bandwidth in
@@ -3166,10 +3179,12 @@ int sched_dl_global_validate(void) goto next; dl_b =dl_bw_of (cpu); + cap =dl_bw_capacity (cpu); cpus =dl_bw_cpus (cpu);raw_spin_lock_irqsave (&dl_b->lock, flags); -if (new_bw * cpus < dl_b->total_bw) +if (new_bw * cpus < dl_b->total_bw + +cap_scale (dl_groups_root, cap)) ret = -EBUSY ;raw_spin_unlock_irqrestore (&dl_b->lock, flags);
Signature · 1 line
2.51.0
RFC PATCH v3 11/24 sched/rt: Add rt-cgroups' dl-servers operations.
In reply to Yuri AndriaccioImplement the servers' functions that pick the next eligible task to run. Enable/Disable dl-servers on task enqueue/dequeue when necessary. Update dl-servers on task update.
Account the number of active rt-tasks in the cgroups' specific runqueue. Account the number of active rt-tasks on the global counter of active tasks when a cgroup is enqueued/dequeued (dl-server started/stopped).
Update rq's cpuprio only if the cgroup's is root control group. Record which dl_server is managing a task when it changes runqueue.
--- kernel/sched/deadline.c | 16 ++++++--- kernel/sched/rt.c | 79 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 3 +- 3 files changed, 85 insertions(+), 13 deletions(-)
kernel/sched/deadline.c +12 -4
@@ -1869,9 +1869,13 @@ void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)u64 deadline = dl_se->deadline; dl_rq->dl_nr_running++; - -if (!dl_server (dl_se)) +if (!dl_server (dl_se)) {add_nr_running (rq_of_dl_rq (dl_rq),1 ); + }else if (dl_se != &rq_of_dl_rq (dl_rq)->fair_server) { +struct rt_rq *rt_rq = &dl_se->my_q->rt; + +add_nr_running (rq_of_dl_rq (dl_rq), rt_rq->rt_nr_running); + }inc_dl_deadline (dl_rq, deadline); }
@@ -1881,9 +1885,13 @@ void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq) {WARN_ON (!dl_rq->dl_nr_running); dl_rq->dl_nr_running--; - -if (!dl_server (dl_se)) +if (!dl_server (dl_se)) {sub_nr_running (rq_of_dl_rq (dl_rq),1 ); + }else if (dl_se != &rq_of_dl_rq (dl_rq)->fair_server) { +struct rt_rq *rt_rq = &dl_se->my_q->rt; + +sub_nr_running (rq_of_dl_rq (dl_rq), rt_rq->rt_nr_running); + }dec_dl_deadline (dl_rq, dl_se->deadline); }
kernel/sched/rt.c +71 -8
153 lines not on this page · raw message
kernel/sched/sched.h +2 -1
@@ -2170,7 +2170,7 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)if (!rt_group_sched_enabled ()) tg = &root_task_group; p->rt.rt_rq = tg->rt_rq[cpu]; - p->rt.parent = tg->rt_se[cpu]; + p->dl.dl_rq = &cpu_rq (cpu)->dl;#endif /* CONFIG_RT_GROUP_SCHED */ }
@@ -2726,6 +2726,7 @@ static inline void add_nr_running(struct rq *rq, unsigned count)static inline void sub_nr_running (struct rq *rq,unsigned count) { +BUG_ON (rq->nr_running < count); rq->nr_running -= count;if (trace_sched_update_nr_running_tp_enabled ()) {call_trace_sched_update_nr_running (rq, -count);
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 11/24] sched/rt: Add rt-cgroups' dl-servers operations.
Hello,
Quoted 8 lines · Yuri Andriaccio
Implement the servers' functions that pick the next eligible task to run. Enable/Disable dl-servers on task enqueue/dequeue when necessary. Update dl-servers on task update.
Account the number of active rt-tasks in the cgroups' specific runqueue. Account the number of active rt-tasks on the global counter of active tasks when a cgroup is enqueued/dequeued (dl-server started/stopped).
Update rq's cpuprio only if the cgroup's is root control group. Record whichdl_serveris managing a task when it changes runqueue.
Changelog looks a little dry. Claude suggests a little rewrite like
---
sched/rt: Implement dl-server operations for rt-cgroups
Implement the dl-server backend that enables rt-cgroups to run as deadline servers. This allows RT tasks within a cgroup to be scheduled according to the cgroup's allocated bandwidth using deadline scheduling.
The implementation consists of three main parts:
1) Server task selection callbacks:
- rt_server_has_tasks(): Check if the rt_rq has runnable tasks
- rt_server_pick(): Pick and set the next RT task from the cgroup's rt_rq when the server gets CPU time
2) Server lifecycle management:
- Start the dl-server when the first RT task enqueues to an idle rt-cgroup
- Stop the dl-server when the last RT task dequeues from an rt-cgroup
- Update the server's consumed runtime in update_curr_rt() via dl_server_update()
3) Per-cpu priority and nr_running accounting:
- Only update rq->cpupri for the root rt_rq (not for cgroup rt_rqs) since cgroups are scheduled via their dl-server priority
- For cgroup rt_rqs, update global nr_running only when the dl-server is active (not throttled), as the server acts as the runnable entity
- Bulk update nr_running when the server starts/stops based on the rt_rq's current rt_nr_running count
The rt.parent field is removed as the new implementation doesn't use hierarchical RT scheduling entities. Instead, tasks record their dl_rq to track which dl-server manages them.
---
Which seems to correspond to what this patch does. If that's the case, however, I wonder if we are maybe doing too many things at once?
Quoted 18 lines
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by luca abeni <[email protected]>Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>--- kernel/sched/deadline.c | 16 ++++++--- kernel/sched/rt.c | 79 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 3 +- 3 files changed, 85 insertions(+), 13 deletions(-)kernel/sched/deadline.c +0
-0
kernel/sched/deadline.c void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq) · line 1874
Quoted 8 lines
kernel/sched/deadline.c +2
-2@@ -1869,9 +1869,13 @@ void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)u64 deadline = dl_se->deadline; dl_rq->dl_nr_running++; - -if (!dl_server (dl_se)) +if (!dl_server (dl_se)) {add_nr_running (rq_of_dl_rq (dl_rq),1 ); + }else if (dl_se != &rq_of_dl_rq (dl_rq)->fair_server) {
I fear this condition might get unwieldy with the addition of new servers (e.g. sched_scx).
Quoted 8 lines
kernel/sched/deadline.c +4
-0+struct rt_rq *rt_rq = &dl_se->my_q->rt; + +add_nr_running (rq_of_dl_rq (dl_rq), rt_rq->rt_nr_running); + }inc_dl_deadline (dl_rq, deadline); }
kernel/sched/deadline.c void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq) · line 1890
Quoted 8 lines
kernel/sched/deadline.c +2
-2@@ -1881,9 +1885,13 @@ void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq) {WARN_ON (!dl_rq->dl_nr_running); dl_rq->dl_nr_running--; - -if (!dl_server (dl_se)) +if (!dl_server (dl_se)) {sub_nr_running (rq_of_dl_rq (dl_rq),1 ); + }else if (dl_se != &rq_of_dl_rq (dl_rq)->fair_server) {
Ditto.
Quoted 22 lines
kernel/sched/deadline.c +4
-0+struct rt_rq *rt_rq = &dl_se->my_q->rt; + +sub_nr_running (rq_of_dl_rq (dl_rq), rt_rq->rt_nr_running); + }dec_dl_deadline (dl_rq, dl_se->deadline); }kernel/sched/rt.c +4
-1@@ -144,14 +144,27 @@ void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, tg->dl_se[cpu] = dl_se; } +static struct task_struct *_pick_next_task_rt (struct rt_rq *rt_rq); +static inline void set_next_task_rt (struct rq *rq,struct task_struct *p,bool first); +static bool rt_server_has_tasks (struct sched_dl_entity *dl_se) { -return false; +return !!dl_se->my_q->rt.rt_nr_running;
kernel/sched/rt.c void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, · line 161
Quoted 10 lines
kernel/sched/rt.c +5
-1}static struct task_struct *rt_server_pick (struct sched_dl_entity *dl_se) { -return NULL ; +struct rt_rq *rt_rq = &dl_se->my_q->rt; +struct rq *rq =rq_of_rt_rq (rt_rq); +struct task_struct *p; + +if (dl_se->my_q->rt.rt_nr_running ==0 )
Can't we use rt_server_has_tasks()?
Quoted 6 lines
kernel/sched/rt.c +6
-0+return NULL ; + + p =_pick_next_task_rt (rt_rq); +set_next_task_rt (rq, p, true); + +return p;
...
kernel/sched/rt.c static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq) · line 1019
Quoted 12 lines
kernel/sched/rt.c +5
-2@@ -953,9 +1011,14 @@ static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq)return next; } -static struct task_struct *_pick_next_task_rt (struct rq *rq) +static struct task_struct *_pick_next_task_rt (struct rt_rq *rt_rq) { -return NULL ; +struct sched_rt_entity *rt_se; + + rt_se =pick_next_rt_entity (rt_rq); +BUG_ON (!rt_se);
Can we WARN and recover somehow?
Quoted 19 lines
kernel/sched/rt.c +3
-1+ +return rt_task_of (rt_se); }static struct task_struct *pick_task_rt (struct rq *rq)@@ -965,7 +1028,7 @@ static struct task_struct *pick_task_rt(struct rq *rq)if (!sched_rt_runnable (rq))return NULL ; - p =_pick_next_task_rt (rq); + p =_pick_next_task_rt (&rq->rt);return p; }kernel/sched/sched.h +0
-0
kernel/sched/sched.h static inline void set_task_rq(struct task_struct *p, unsigned int cpu) · line 2173
Quoted 5 lines
kernel/sched/sched.h +1
-1@@ -2170,7 +2170,7 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)if (!rt_group_sched_enabled ()) tg = &root_task_group; p->rt.rt_rq = tg->rt_rq[cpu]; - p->rt.parent = tg->rt_se[cpu]; + p->dl.dl_rq = &cpu_rq (cpu)->dl;
Guess rt.parent is then removed in a subsequent patch? Do we want to consolidate the cleanup?
kernel/sched/sched.h static inline void add_nr_running(struct rq *rq, unsigned count) · line 2729
Quoted 8 lines
kernel/sched/sched.h +1
-0#endif /* CONFIG_RT_GROUP_SCHED */ }@@ -2726,6 +2726,7 @@ static inline void add_nr_running(struct rq *rq, unsigned count)static inline void sub_nr_running (struct rq *rq,unsigned count) { +BUG_ON (rq->nr_running < count);
Can we WARN and recover somehow?
kernel/sched/sched.h static inline void add_nr_running(struct rq *rq, unsigned count) · line 2732
Quoted 3 lines
kernel/sched/sched.h +0
-0rq->nr_running -= count;if (trace_sched_update_nr_running_tp_enabled ()) {call_trace_sched_update_nr_running (rq, -count);
Thanks,
Juri
RFC PATCH v3 12/24 sched/rt: Update task event callbacks for HCBS scheduling
In reply to Yuri AndriaccioUpdate wakeup_preempt_rt, switched_{from/to}_rt and prio_changed_rt with rt-cgroup's specific preemption rules.
Add checks whether a rt-task can be attached or not to a rt-cgroup. Update task_is_throttled_rt for SCHED_CORE.
--- kernel/sched/core.c | 2 +- kernel/sched/rt.c | 88 ++++++++++++++++++++++++++++++++++++++--- kernel/sched/syscalls.c | 13 ++++++ 3 files changed, 96 insertions(+), 7 deletions(-)
kernel/sched/core.c +1 -1
@@ -9346,7 +9346,7 @@ static int cpu_cgroup_can_attach(struct cgroup_taskset *tset) goto scx_check;cgroup_taskset_for_each (task, css, tset) { -if (!sched_rt_can_attach (css_tg (css), task)) +if (rt_task (task) && !sched_rt_can_attach (css_tg (css), task))return -EINVAL ; } scx_check:
kernel/sched/rt.c +82 -6
148 lines not on this page · raw message
kernel/sched/syscalls.c +13 -0
@@ -626,6 +626,19 @@ int __sched_setscheduler(struct task_struct *p, change:if (user) { +#ifdef CONFIG_RT_GROUP_SCHED +/* +* Do not allow real-time tasks into groups that have no runtime +* assigned. +*/ +if (rt_group_sched_enabled () && +dl_bandwidth_enabled () &&rt_policy (policy) && + !sched_rt_can_attach (task_group (p), p) && + !task_group_is_autogroup (task_group (p))) { + retval = -EPERM ; + goto unlock; + } +#endif if (dl_bandwidth_enabled () &&dl_policy (policy) && !(attr->sched_flags &SCHED_FLAG_SUGOV )) {cpumask_t *span = rq->rd->span;
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 12/24] sched/rt: Update task event callbacks for HCBS scheduling
Hello,
Quoted 4 lines · Yuri Andriaccio
Update
wakeup_preempt_rt, switched_{from/to}_rt andprio_changed_rtwith rt-cgroup's specific preemption rules.
Add checks whether a rt-task can be attached or not to a rt-cgroup. Updatetask_is_throttled_rtforSCHED_CORE.
A little dry. :) This is telling the what, can you please add also the why and how?
Quoted 31 lines
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by luca abeni <[email protected]>Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>--- kernel/sched/core.c | 2 +- kernel/sched/rt.c | 88 ++++++++++++++++++++++++++++++++++++++--- kernel/sched/syscalls.c | 13 ++++++ 3 files changed, 96 insertions(+), 7 deletions(-)kernel/sched/core.c +1
-1@@ -9346,7 +9346,7 @@ static int cpu_cgroup_can_attach(struct cgroup_taskset *tset) goto scx_check;cgroup_taskset_for_each (task, css, tset) { -if (!sched_rt_can_attach (css_tg (css), task)) +if (rt_task (task) && !sched_rt_can_attach (css_tg (css), task))return -EINVAL ; } scx_check:kernel/sched/rt.c +0
-0
kernel/sched/rt.c static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) · line 949
Quoted 4 lines
kernel/sched/rt.c +1
-0@@ -946,6 +946,50 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) {struct task_struct *donor = rq->donor; +if (!rt_group_sched_enabled ())
Should we actually be using rt_group_sched_enabled() (instead of ifdeffery) everywhere to check if group scheduling is enabled and differentiate behavior?
kernel/sched/rt.c static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) · line 956
Quoted 7 lines
kernel/sched/rt.c +7
-0+ goto no_group_sched; + +/* +* Preemption checks are different if the waking task and the current +* task are running on the global runqueue or in a cgroup. +* The following rules apply: +* - dl-tasks (and equally dl_servers) always preempt FIFO/RR tasks.
We already touched upon this and maybe we can leave it for later, but the fact that dl_servers always preempt FIFO/RR is going to be a change of behavior wrt legacy RT group scheduling. Legacy users might have expectations that setting an high static priority means those tasks will always be scheduled first and this will break that. Maybe we want to convince them to change their assumptions, but we need to at least cleary document the new behavior (and why it is right/better - if it is indeed :). Not sure if we can also find a way to be "back-compatible" and if we want to do that.
Quoted 6 lines
kernel/sched/rt.c +6
-0+* - if curr is inside a cgroup (i.e. run by a dl_server) and +* waking is not, do nothing. +* - if waking is inside a cgroup but not curr, always reschedule. +* - if they are both on the global runqueue, run the standard code. +* - if they are both in the same cgroup, check for tasks priorities. +* - if they are both in a cgroup, but not the same one, check whether
kernel/sched/rt.c static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) · line 972
Quoted 10 lines
kernel/sched/rt.c +10
-0+* the woken task's dl_server preempts the current's dl_server. +*/ +if (is_dl_group (rt_rq_of_se (&p->rt)) && +is_dl_group (rt_rq_of_se (&rq->curr->rt))) { +struct sched_dl_entity *woken_dl_se, *curr_dl_se; + + woken_dl_se =dl_group_of (rt_rq_of_se (&p->rt)); + curr_dl_se =dl_group_of (rt_rq_of_se (&rq->curr->rt)); + +if (rt_rq_of_se (&p->rt)->tg ==rt_rq_of_se (&rq->curr->rt)->tg) {
What about checking if woken_dl_se and curr_dl_se are the same dl_se?
Quoted 23 lines
kernel/sched/rt.c +20
-0+if (p->prio < rq->curr->prio) +resched_curr (rq); + +return ; + } + +if (dl_entity_preempt (woken_dl_se, curr_dl_se)) +resched_curr (rq); + +return ; + + }else if (is_dl_group (rt_rq_of_se (&p->rt))) { +resched_curr (rq); +return ; + + }else if (is_dl_group (rt_rq_of_se (&rq->curr->rt))) { +return ; + } + +no_group_sched:if (p->prio < donor->prio) {resched_curr (rq);return ;
...
kernel/sched/rt.c static void switched_to_rt(struct rq *rq, struct task_struct *p) · line 1805
Quoted 10 lines
kernel/sched/rt.c +4
-0@@ -1750,8 +1797,17 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)* then see if we can move to another run queue. */ if (task_on_rq_queued (p)) { + +#ifndef CONFIG_RT_GROUP_SCHED if (p->nr_cpus_allowed >1 && rq->rt.overloaded)rt_queue_push_tasks (rt_rq_of_se (&p->rt)); +#else +if (rt_rq_of_se (&p->rt)->overloaded) {
Is this empty because intra-group migration is coming with some future change? Even if that's the case, I believe this wants a comment explaining why this branch is empty.
Quoted 8 lines
kernel/sched/rt.c +5
-0+ } else { +if (p->prio < rq->curr->prio) +resched_curr (rq); + } +#endif if (p->prio < rq->donor->prio &&cpu_online (cpu_of (rq)))resched_curr (rq); }
...
Quoted 18 lines
kernel/sched/rt.c +9
-0@@ -1876,7 +1943,16 @@ static unsigned int get_rr_interval_rt(struct rq *rq, struct task_struct *task)#ifdef CONFIG_SCHED_CORE static int task_is_throttled_rt (struct task_struct *p,int cpu) { +#ifdef CONFIG_RT_GROUP_SCHED +struct rt_rq *rt_rq; + + rt_rq =task_group (p)->rt_rq[cpu]; +WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); + +return dl_group_of (rt_rq)->dl_throttled; +#else return 0 ; +#endif }#endif /* CONFIG_SCHED_CORE */
kernel/sched/rt.c static int sched_rt_global_constraints(void) · line 2207
Quoted 1 line
kernel/sched/rt.c +0
-0@@ -2131,7 +2207,7 @@ static int sched_rt_global_constraints(void)int sched_rt_can_attach (struct task_group *tg,struct task_struct *tsk)
tsk argument is not going to be used anymore with this, please remove it. Also maybe rename to something closer to what the function actually checks for, e.g. [sched_]rt_group_has_runtime.
kernel/sched/rt.c static int sched_rt_global_constraints(void) · line 2211
Quoted 5 lines
kernel/sched/rt.c +1
-1{/* Don't accept real-time tasks when there is no way for them to run */ -if (rt_group_sched_enabled () &&rt_task (tsk) && tg->rt_bandwidth.rt_runtime ==0 ) +if (rt_group_sched_enabled () && tg->dl_bandwidth.dl_runtime ==0 )return 0 ;
Thanks,
Juri
RFC PATCH v3 13/24 sched/rt: Update rt-cgroup schedulability checks
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Update schedulability checks and setup of runtime/period for rt-cgroups.
--- kernel/sched/core.c | 6 ++++ kernel/sched/deadline.c | 46 +++++++++++++++++++++++---- kernel/sched/rt.c | 70 +++++++++++++++++++++++------------------ kernel/sched/sched.h | 1 + 4 files changed, 87 insertions(+), 36 deletions(-)
kernel/sched/core.c +6 -0
@@ -9281,6 +9281,12 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)return &root_task_group.css; } +/* Do not allow cpu_cgroup hierachies with depth greater than 2. */ +#ifdef CONFIG_RT_GROUP_SCHED +if (parent != &root_task_group) +return ERR_PTR (-EINVAL ); +#endif + tg =sched_create_group (parent);if (IS_ERR (tg))return ERR_PTR (-ENOMEM );
kernel/sched/deadline.c +40 -6
59 lines not on this page · raw message
kernel/sched/rt.c +40 -30
149 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -394,6 +394,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,dl_server_has_tasks_f has_tasks, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void ); +extern int dl_check_tg (unsigned long total);extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period); extern void dl_server_update_idle_time(struct rq *rq,
Signature · 1 line
2.51.0
Re: [RFC PATCH v3 13/24] sched/rt: Update rt-cgroup schedulability checks
…
Quoted 2 lines
+++
b/kernel/sched/deadline.c(quoted diff) +0
-0@@ -340,6 +340,39 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
…
Quoted 3 lines
(quoted diff) +3
-0+#ifdef CONFIG_RT_GROUP_SCHED +void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period) +{
…
Quoted 3 lines
(quoted diff) +3
-0+ u64 new_bw; + + raw_spin_rq_lock_irq(rq);
…
Review
Quoted 3 lines
(quoted diff) +3
-0+ raw_spin_rq_unlock_irq(rq); +} +#endif
…
How do you think about to define and use a corresponding lock guard?
Regards,
Markus
Juri Lelli
maintainerRe: [RFC PATCH v3 13/24] sched/rt: Update rt-cgroup schedulability checks
In reply to Yuri AndriaccioHello,
Quoted 3 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Update schedulability checks and setup of runtime/period for rt-cgroups.
So, it looks like changelogs are all too minimal and dry. Having a more comprehensive (but concise) description of what the patch does, why it does it and how, will help reviewing and is essential in the future to trace back design decisions and issues. Please consider this for current and previous/subsequent patches.
Quoted 19 lines
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>--- kernel/sched/core.c | 6 ++++ kernel/sched/deadline.c | 46 +++++++++++++++++++++++---- kernel/sched/rt.c | 70 +++++++++++++++++++++++------------------ kernel/sched/sched.h | 1 + 4 files changed, 87 insertions(+), 36 deletions(-)kernel/sched/core.c +0
-0
kernel/sched/core.c cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) · line 9284
Quoted 4 lines
kernel/sched/core.c +1
-0@@ -9281,6 +9281,12 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)return &root_task_group.css; } +/* Do not allow cpu_cgroup hierachies with depth greater than 2. */
I believe this limit gets removed with later patches, but since we have it here, we should state why we have the limit in place.
Quoted 32 lines
kernel/sched/core.c +5
-0+#ifdef CONFIG_RT_GROUP_SCHED +if (parent != &root_task_group) +return ERR_PTR (-EINVAL ); +#endif + tg =sched_create_group (parent);if (IS_ERR (tg))return ERR_PTR (-ENOMEM );kernel/sched/deadline.c +15
-0@@ -347,7 +347,47 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)cancel_dl_timer (dl_se, &dl_se->inactive_timer); } +/* +* Used for dl_bw check and update, used under sched_rt_handler()::mutex and +* sched_domains_mutex. +*/ +u64 dl_cookie; +#ifdef CONFIG_RT_GROUP_SCHED +int dl_check_tg (unsigned long total) +{ +unsigned long flags; +int which_cpu; +int cpus; +struct dl_bw *dl_b; +u64 gen = ++dl_cookie; + +for_each_possible_cpu (which_cpu) {
kernel/sched/deadline.c void cancel_inactive_timer(struct sched_dl_entity *dl_se) · line 375
Quoted 10 lines
kernel/sched/deadline.c +10
-0+rcu_read_lock_sched (); + +if (!dl_bw_visited (which_cpu, gen)) { + cpus =dl_bw_cpus (which_cpu); + dl_b =dl_bw_of (which_cpu); + +raw_spin_lock_irqsave (&dl_b->lock, flags); + +if (dl_b->bw !=-1 && + dl_b->bw * cpus < dl_b->total_bw + total * cpus) {
Does this need to use cap_scale()?
Quoted 15 lines
kernel/sched/deadline.c +15
-0+raw_spin_unlock_irqrestore (&dl_b->lock, flags); +rcu_read_unlock_sched (); + +return 0 ; + } + +raw_spin_unlock_irqrestore (&dl_b->lock, flags); + } + +rcu_read_unlock_sched (); + } + +return 1 ; +} +
...
kernel/sched/deadline.c static int tg_rt_schedulable(struct task_group *tg, void *data) · line 2033
Quoted 8 lines
kernel/sched/deadline.c +2
-2@@ -2034,8 +2029,8 @@ static int tg_rt_schedulable(struct task_group *tg, void *data)unsigned long total, sum =0 ;u64 period, runtime; - period =ktime_to_ns (tg->rt_bandwidth.rt_period); - runtime = tg->rt_bandwidth.rt_runtime; + period = tg->dl_bandwidth.dl_period; + runtime = tg->dl_bandwidth.dl_runtime;
Just as an example, this is the kind of important change (rt_bandwidth -> dl_bandwidth) that usually deserves to be explicitly mentioned in the changelog.
...
kernel/sched/deadline.c static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime) · line 2106
Quoted 12 lines
kernel/sched/deadline.c +8
-0@@ -2097,6 +2096,20 @@ static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime) .rt_runtime = runtime, }; +/* +* Since we truncate DL_SCALE bits, make sure we're at least +* that big. +*/ +if (runtime !=0 && runtime < (1ULL <<DL_SCALE )) +return -EINVAL ; + +/*
^
Nit, fix alignment.
kernel/sched/deadline.c static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime) · line 2112
Quoted 6 lines
kernel/sched/deadline.c +6
-0+* Since we use the MSB for wrap-around and sign issues, make +* sure it's not set (mind that period can be equal to zero). +*/ +if (period & (1ULL <<63 )) +return -EINVAL ; +
Thanks,
Juri
RFC PATCH v3 14/24 sched/rt: Allow zeroing the runtime of the root control group
In reply to Yuri AndriaccioIn HCBS, the root control group follows the already existing rules for rt-task scheduling. As such, it does not make use of the deadline servers to account for runtime, or any other HCBS specific code and features.
While the runtime of SCHED_DEADLINE tasks depends on the global bandwidth reserved for rt_tasks, the runtime of SCHED_FIFO/SCHED_RR tasks is limited by the activation of fair-servers (as the RT_THROTTLING mechanism has been removed in favour of them), thus their maximum bandwidth depends solely on the fair-server settings (which are thightly related to the global bandwdith reserved for rt-tasks) and the amount of SCHED_OTHER workload to run (recall that if no SCHED_OTHER tasks are running, the FIFO/RR tasks may fully utilize the CPU).
The values of runtime and period in the root cgroup's cpu controller do not affect, by design of HCBS, the fair-server settings and similar (consequently they do not affect the scheduling of FIFO/RR tasks in the root cgroup), buy they are just used to reserve a portion of the SCHED_DEADLINE bandwidth to the scheduling of rt-cgroups. These values only affect child cgroups, their deadline servers and their assigned FIFO/RR tasks.
--- kernel/sched/rt.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
kernel/sched/rt.c +4 -7
@@ -2123,13 +2123,6 @@ static int tg_set_rt_bandwidth(struct task_group *tg,static DEFINE_MUTEX (rt_constraints_mutex);int i, err =0 ; -/* -* Disallowing the root group RT runtime is BAD, it would disallow the -* kernel creating (and or operating) RT threads. -*/ -if (tg == &root_task_group && rt_runtime ==0 ) -return -EINVAL ; -/* No period doesn't make any sense. */ if (rt_period ==0 )return -EINVAL ;
@@ -2216,6 +2209,10 @@ static int sched_rt_global_constraints(void)int sched_rt_can_attach (struct task_group *tg,struct task_struct *tsk) { +/* Allow executing in the root cgroup regardless of allowed bandwidth */ +if (tg == &root_task_group) +return 1 ; +/* Don't accept real-time tasks when there is no way for them to run */ if (rt_group_sched_enabled () && tg->dl_bandwidth.dl_runtime ==0 )return 0 ;
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 14/24] sched/rt: Allow zeroing the runtime of the root control group
Hello,
Quoted 9 lines · Yuri Andriaccio
In HCBS, the root control group follows the already existing rules for rt-task scheduling. As such, it does not make use of the deadline servers to account for runtime, or any other HCBS specific code and features.
While the runtime of
SCHED_DEADLINEtasks depends on the global bandwidth reserved forrt_tasks, the runtime of SCHED_FIFO/SCHED_RR tasks is limited by the activation of fair-servers (as theRT_THROTTLINGmechanism has been removed in favour of them), thus their maximum bandwidth depends solely on the fair-server settings (which are thightly related to the global bandwdith
^ bandwidth
Quoted 7 lines
reserved for rt-tasks) and the amount of
SCHED_OTHERworkload to run (recall that if noSCHED_OTHERtasks are running, the FIFO/RR tasks may fully utilize the CPU).The values of runtime and period in the root cgroup's cpu controller do not affect, by design of HCBS, the fair-server settings and similar (consequently they do not affect the scheduling of FIFO/RR tasks in the root cgroup), buy they
^ but
Quoted 6 lines
are just used to reserve a portion of the
SCHED_DEADLINEbandwidth to the scheduling of rt-cgroups. These values only affect child cgroups, their deadline servers and their assigned FIFO/RR tasks.Signed-off-by Yuri Andriaccio <[email protected]>---
Thanks,
Juri
RFC PATCH v3 15/24 sched/rt: Remove old RT_GROUP_SCHED data structures
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Completely remove the old RT_GROUP_SCHED's functions and data structures.
--- include/linux/sched.h | 4 ---- kernel/sched/rt.c | 1 - kernel/sched/sched.h | 26 -------------------------- 3 files changed, 31 deletions(-)
include/linux/sched.h +0 -4
@@ -624,13 +624,9 @@ struct sched_rt_entity {unsigned short on_rq;unsigned short on_list; -struct sched_rt_entity *back;#ifdef CONFIG_RT_GROUP_SCHED -struct sched_rt_entity *parent;/* rq on which this entity is (to be) queued: */ struct rt_rq *rt_rq; -/* rq "owned" by this entity/group: */ -struct rt_rq *my_q;#endif } __randomize_layout;
kernel/sched/rt.c +0 -1
@@ -1,4 +1,3 @@ -#pragma GCC diagnostic ignored "-Wunused-function"// SPDX-License-Identifier: GPL-2.0 /* * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
kernel/sched/sched.h +0 -26
@@ -310,15 +310,6 @@ struct rt_prio_array {struct list_head queue[MAX_RT_PRIO ]; }; -struct rt_bandwidth { -/* nests inside the rq lock: */ -raw_spinlock_t rt_runtime_lock; -ktime_t rt_period; -u64 rt_runtime; -struct hrtimer rt_period_timer; -unsigned int rt_period_active; -}; -struct dl_bandwidth {raw_spinlock_t dl_runtime_lock;u64 dl_runtime;
@@ -485,7 +476,6 @@ struct task_group {struct sched_dl_entity **dl_se;struct rt_rq **rt_rq; -struct rt_bandwidth rt_bandwidth;struct dl_bandwidth dl_bandwidth;#endif
@@ -801,11 +791,6 @@ struct scx_rq { };#endif /* CONFIG_SCHED_CLASS_EXT */ -static inline int rt_bandwidth_enabled (void ) -{ -return 0 ; -} -/* RT IPI pull logic requires IRQ_WORK */ #if defined(CONFIG_IRQ_WORK ) && defined(CONFIG_SMP )# define HAVE_RT_PUSH_IPI
@@ -823,17 +808,6 @@ struct rt_rq {bool overloaded;struct plist_head pushable_tasks; -int rt_queued; - -#ifdef CONFIG_RT_GROUP_SCHED -int rt_throttled; -u64 rt_time;/* consumed RT time, goes up in update_curr_rt */ -u64 rt_runtime;/* allotted RT time, "slice" from rt_bandwidth, RT sharing/balancing */ -/* Nests inside the rq lock: */ -raw_spinlock_t rt_runtime_lock; - -unsigned int rt_nr_boosted; -#endif #ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */ #endif
Signature · 1 line
2.51.0
RFC PATCH v3 16/24 sched/core: Cgroup v2 support
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make rt_runtime_us and rt_period_us virtual files accessible also to the cgroup v2 controller, effectively enabling the RT_GROUP_SCHED mechanism to cgroups v2.
--- kernel/sched/core.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/sched/core.c +12 -0
@@ -10323,6 +10323,18 @@ static struct cftype cpu_files[] = { .write = cpu_uclamp_max_write, },#endif /* CONFIG_UCLAMP_TASK_GROUP */ +#ifdef CONFIG_RT_GROUP_SCHED + { + .name = "rt_runtime_us", + .read_s64 = cpu_rt_runtime_read, + .write_s64 = cpu_rt_runtime_write, + }, + { + .name = "rt_period_us", + .read_u64 = cpu_rt_period_read_uint, + .write_u64 = cpu_rt_period_write_uint, + }, +#endif /* CONFIG_RT_GROUP_SCHED */ { } /* terminate */ };
Signature · 1 line
2.51.0
RFC PATCH v3 17/24 sched/rt: Remove support for cgroups-v1
In reply to Yuri AndriaccioDisable control files for cgroups-v1, and allow only cgroups-v2. This should simplify maintaining the code, also because cgroups-v1 are deprecated.
Set the default rt-cgroups runtime to zero, otherwise a cgroup-v1 kernel will not be able to start SCHED_DEADLINE tasks. The bandwidth for rt-cgroups must then be manually assigned after the kernel boots.
--- kernel/sched/core.c | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-)
kernel/sched/core.c +1 -19
@@ -8723,7 +8723,7 @@ void __init sched_init(void)#ifdef CONFIG_RT_GROUP_SCHED init_dl_bandwidth (&root_task_group.dl_bandwidth, -global_rt_period (),global_rt_runtime ()); +global_rt_period (),0 );#endif /* CONFIG_RT_GROUP_SCHED */ #ifdef CONFIG_CGROUP_SCHED
@@ -10062,20 +10062,6 @@ static struct cftype cpu_legacy_files[] = { };#ifdef CONFIG_RT_GROUP_SCHED -static struct cftype rt_group_files[] = { - { - .name ="rt_runtime_us" , - .read_s64 = cpu_rt_runtime_read, - .write_s64 = cpu_rt_runtime_write, - }, - { - .name ="rt_period_us" , - .read_u64 = cpu_rt_period_read_uint, - .write_u64 = cpu_rt_period_write_uint, - }, - { }/* Terminate */ -}; -# ifdef CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED DEFINE_STATIC_KEY_FALSE (rt_group_sched);# else
@@ -10101,10 +10087,6 @@ __setup("rt_group_sched=", setup_rt_group_sched);static int __init cpu_rt_group_init (void ) { -if (!rt_group_sched_enabled ()) -return 0 ; - -WARN_ON (cgroup_add_legacy_cftypes (&cpu_cgrp_subsys, rt_group_files));return 0 ; }subsys_initcall (cpu_rt_group_init);
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 17/24] sched/rt: Remove support for cgroups-v1
Hello,
Quoted 9 lines · Yuri Andriaccio
Disable control files for cgroups-v1, and allow only cgroups-v2. This should simplify maintaining the code, also because cgroups-v1 are deprecated.
Set the default rt-cgroups runtime to zero, otherwise a cgroup-v1 kernel will not be able to start
SCHED_DEADLINEtasks. The bandwidth for rt-cgroups must then be manually assigned after the kernel boots.Signed-off-by Yuri Andriaccio <[email protected]>---
The change looks good, but I think this series is currently missing a substantial Docs update. admin-guide cgroup section and scheduler needs some changes, I believe. Please consider doing so for next revision.
Thanks,
Juri
RFC PATCH v3 18/24 sched/deadline: Allow deeper hierarchies of RT cgroups
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Allow creation of cgroup hierachies with depth greater than two. Add check to prevent attaching tasks to a child cgroup of an active cgroup (i.e. with a running FIFO/RR task).
Add check to prevent attaching tasks to cgroups which have children with non-zero runtime.
Update rt-cgroups allocated bandwidth accounting for nested cgroup hierachies.
--- kernel/sched/core.c | 6 ----- kernel/sched/deadline.c | 51 +++++++++++++++++++++++++++++++++++++---- kernel/sched/rt.c | 16 ++++++++++--- kernel/sched/sched.h | 3 ++- 4 files changed, 62 insertions(+), 14 deletions(-)
kernel/sched/core.c +0 -6
@@ -9281,12 +9281,6 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)return &root_task_group.css; } -/* Do not allow cpu_cgroup hierachies with depth greater than 2. */ -#ifdef CONFIG_RT_GROUP_SCHED -if (parent != &root_task_group) -return ERR_PTR (-EINVAL ); -#endif - tg =sched_create_group (parent);if (IS_ERR (tg))return ERR_PTR (-ENOMEM );
kernel/sched/deadline.c +47 -4
72 lines not on this page · raw message
kernel/sched/rt.c +13 -3
@@ -114,7 +114,8 @@ void free_rt_sched_group(struct task_group *tg)* Fix this issue by changing the group runtime * to 0 immediately before freeing it. */ -dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); +if (tg->dl_se[i]->dl_runtime) +dl_init_tg (tg, i,0 , tg->dl_se[i]->dl_period);raw_spin_rq_lock_irqsave (cpu_rq (i), flags);BUG_ON (tg->rt_rq[i]->rt_nr_running);
@@ -2122,6 +2123,14 @@ static int tg_set_rt_bandwidth(struct task_group *tg,static DEFINE_MUTEX (rt_constraints_mutex);int i, err =0 ; +/* +* Do not allow to set a RT runtime > 0 if the parent has RT tasks +* (and is not the root group) +*/ +if (rt_runtime && (tg != &root_task_group) && (tg->parent != &root_task_group) &&tg_has_rt_tasks (tg->parent)) { +return -EINVAL ; + } +/* No period doesn't make any sense. */ if (rt_period ==0 )return -EINVAL ;
@@ -2145,7 +2154,7 @@ static int tg_set_rt_bandwidth(struct task_group *tg,return 0 ;for_each_possible_cpu (i) { -dl_init_tg (tg->dl_se[i], rt_runtime, rt_period); +dl_init_tg (tg, i, rt_runtime, rt_period); }return 0 ;
@@ -2216,7 +2225,8 @@ int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk)if (rt_group_sched_enabled () && tg->dl_bandwidth.dl_runtime ==0 )return 0 ; - return 1; +/* tasks can be attached only if the taskgroup has no active children. */ + return (int)is_active_sched_group(tg); }#else /* !CONFIG_RT_GROUP_SCHED: */
kernel/sched/sched.h +2 -1
@@ -386,7 +386,8 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void );extern int dl_check_tg (unsigned long total); -extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period); +extern void dl_init_tg (struct task_group *tg,int cpu,u64 rt_runtime,u64 rt_period); +extern bool is_active_sched_group (struct task_group *tg);extern void dl_server_update_idle_time (struct rq *rq,struct task_struct *p);
Signature · 1 line
2.51.0
Juri Lelli
maintainerRe: [RFC PATCH v3 18/24] sched/deadline: Allow deeper hierarchies of RT cgroups
Hello,
Quoted 41 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Allow creation of cgroup hierachies with depth greater than two. Add check to prevent attaching tasks to a child cgroup of an active cgroup (i.e. with a running FIFO/RR task).
Add check to prevent attaching tasks to cgroups which have children with non-zero runtime.
Update rt-cgroups allocated bandwidth accounting for nested cgroup hierachies.Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>--- kernel/sched/core.c | 6 ----- kernel/sched/deadline.c | 51 +++++++++++++++++++++++++++++++++++++---- kernel/sched/rt.c | 16 ++++++++++--- kernel/sched/sched.h | 3 ++- 4 files changed, 62 insertions(+), 14 deletions(-)kernel/sched/core.c +0
-6@@ -9281,12 +9281,6 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)return &root_task_group.css; } -/* Do not allow cpu_cgroup hierachies with depth greater than 2. */ -#ifdef CONFIG_RT_GROUP_SCHED -if (parent != &root_task_group) -return ERR_PTR (-EINVAL ); -#endif - tg =sched_create_group (parent);if (IS_ERR (tg))return ERR_PTR (-ENOMEM );kernel/sched/deadline.c +0
-0
kernel/sched/deadline.c int dl_check_tg(unsigned long total) · line 391
Quoted 5 lines
kernel/sched/deadline.c +1
-1@@ -388,11 +388,42 @@ int dl_check_tg(unsigned long total)return 1 ; } -void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period) +bool is_active_sched_group (struct task_group *tg)
I wonder if the function name could be misleading, as this checks runtime and not if there are tasks in the group.
Quoted 9 lines
kernel/sched/deadline.c +8
-0{ +struct task_group *child; +bool is_active =1 ; + +// if there are no children, this is a leaf group, thus it is active +list_for_each_entry_rcu (child, &tg->children, siblings) { +if (child->dl_bandwidth.dl_runtime >0 ) { + is_active =0 ; + }
kernel/sched/deadline.c int dl_check_tg(unsigned long total) · line 410
Quoted 10 lines
kernel/sched/deadline.c +10
-0+ } +return is_active; +} + +static inline bool sched_group_has_active_siblings (struct task_group *tg) +{ +struct task_group *child; +bool has_active_siblings =0 ; + +// if there are no children, this is a leaf group, thus it is active
Copy-pasta from above? :) Also not the correct comment style.
Quoted 34 lines
kernel/sched/deadline.c +18
-3+list_for_each_entry_rcu (child, &tg->parent->children, siblings) { +if (child != tg && child->dl_bandwidth.dl_runtime >0 ) { + has_active_siblings =1 ; + } + } +return has_active_siblings; +} + +void dl_init_tg (struct task_group *tg,int cpu,u64 rt_runtime,u64 rt_period) +{ +struct sched_dl_entity *dl_se = tg->dl_se[cpu]; struct rq *rq = container_of(dl_se->dl_rq, struct rq, dl); -int is_active; -u64 new_bw; +int is_active, is_active_group; +u64 old_runtime, new_bw; + + is_active_group =is_active_sched_group (tg);raw_spin_rq_lock_irq (rq); is_active = dl_se->my_q->rt.rt_nr_running >0 ;@@ -400,8 +431,10 @@ void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period)update_rq_clock (rq);dl_server_stop (dl_se); + old_runtime = dl_se->dl_runtime; new_bw =to_ratio (dl_se->dl_period, dl_se->dl_runtime); -dl_rq_change_utilization (rq, dl_se, new_bw); +if (is_active_group) +dl_rq_change_utilization (rq, dl_se, new_bw); dl_se->dl_runtime = rt_runtime; dl_se->dl_deadline = rt_period;
kernel/sched/deadline.c void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period) · line 449
Quoted 4 lines
kernel/sched/deadline.c +1
-0@@ -413,6 +446,16 @@ void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period) dl_se->dl_bw = new_bw; dl_se->dl_density = new_bw; +// add/remove the parent's bw
Comment style is not correct. Also the comment itself is not very much informative. What about something like (IIUC)
/*
* Handle parent bandwidth accounting when child runtime changes:
* - Disabling the last active child: parent becomes a leaf group,
* so add the parent's bandwidth back to active accounting
* - Enabling the first child: parent becomes a non-leaf group,
* so remove the parent's bandwidth from active accounting
* Only leaf groups (those without active children) should have
* non-zero bandwidth.
*/
kernel/sched/deadline.c void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period) · line 458
Quoted 9 lines
kernel/sched/deadline.c +9
-0+if (tg->parent && tg->parent != &root_task_group) + { +if (rt_runtime ==0 && old_runtime !=0 && !sched_group_has_active_siblings (tg)) { +__add_rq_bw (tg->parent->dl_se[cpu]->dl_bw, dl_se->dl_rq); + }else if (rt_runtime !=0 && old_runtime ==0 && !sched_group_has_active_siblings (tg)) { +__sub_rq_bw (tg->parent->dl_se[cpu]->dl_bw, dl_se->dl_rq); + } + } +
Thanks,
Juri
RFC PATCH v3 19/24 sched/rt: Add rt-cgroup migration
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
When the runtime is exhausted in a RT CGroup, the scheduler checks for another non-throttled runqueue and, if available, migrates the tasks.
The bandwidth (runtime/period) chosen for a certain cgroup is replicated on every core of the system, therefore, in an SMP system with M cores, the total available bandwidth is the given runtime/period multiplied by M.
--- kernel/sched/rt.c | 495 +++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 10 + 2 files changed, 505 insertions(+)
kernel/sched/rt.c +495 -0
@@ -1,3 +1,4 @@ +#pragma GCC diagnostic ignored "-Wunused-function"// SPDX-License-Identifier: GPL-2.0 /* * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -84,6 +85,8 @@ void init_rt_rq(struct rt_rq *rt_rq)plist_head_init (&rt_rq->pushable_tasks); } +static void group_pull_rt_task (struct rt_rq *this_rt_rq); +#ifdef CONFIG_RT_GROUP_SCHED void unregister_rt_sched_group (struct task_group *tg)
@@ -136,6 +139,7 @@ void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, struct sched_dl_entity *dl_se, int cpu, struct sched_dl_entity *parent) { + served_rq->cpu = cpu; served_rq->rt.highest_prio.curr =MAX_RT_PRIO -1 ; served_rq->rt.rq =cpu_rq (cpu); served_rq->rt.tg = tg;
@@ -299,6 +303,53 @@ static inline void rt_queue_pull_task(struct rt_rq * rt_rq)queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); } +#ifdef CONFIG_RT_GROUP_SCHED +static DEFINE_PER_CPU(struct balance_callback, rt_group_push_head); +static DEFINE_PER_CPU(struct balance_callback, rt_group_pull_head); +static void group_push_rt_tasks_callback (struct rq *); +static void group_pull_rt_task_callback (struct rq *); + +static void rt_queue_push_from_group (struct rt_rq *rt_rq) +{ + struct rq *rq = container_of(rt_rq, struct rq, rt); +struct rq *global_rq =cpu_rq (rq->cpu); + +BUG_ON (rt_rq ==NULL ); +BUG_ON (rt_rq->rq != global_rq); + +if (global_rq->rq_to_push_from) +return ; + +if (!has_pushable_tasks (rt_rq)) +return ; + + global_rq->rq_to_push_from = rq; +queue_balance_callback (global_rq, &per_cpu (rt_group_push_head, global_rq->cpu), + group_push_rt_tasks_callback); +} + +static void rt_queue_pull_to_group (struct rt_rq *rt_rq) +{ + struct rq *rq = container_of(rt_rq, struct rq, rt); +struct rq *global_rq =cpu_rq (rq->cpu); +struct sched_dl_entity *dl_se =dl_group_of (rt_rq); + +BUG_ON (rt_rq ==NULL ); +BUG_ON (!is_dl_group (rt_rq)); +BUG_ON (rt_rq->rq != global_rq); + +if (dl_se->dl_throttled || global_rq->rq_to_pull_to) +return ; + + global_rq->rq_to_pull_to = rq; +queue_balance_callback (global_rq, &per_cpu (rt_group_pull_head, global_rq->cpu), + group_pull_rt_task_callback); +} +#else +static inline void rt_queue_push_from_group (struct rt_rq *rt_rq) {}; +static inline void rt_queue_pull_to_group (struct rt_rq *rt_rq) {}; +#endif +static void enqueue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) {plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);
@@ -1303,6 +1354,8 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)*/ static int push_rt_task (struct rq *rq,bool pull) { +BUG_ON (is_dl_group (&rq->rt)); +struct task_struct *next_task;struct rq *lowest_rq;int ret =0 ;
@@ -1599,6 +1652,8 @@ void rto_push_irq_work_func(struct irq_work *work)static void pull_rt_task (struct rq *this_rq) { +BUG_ON (is_dl_group (&this_rq->rt)); +int this_cpu = this_rq->cpu, cpu;bool resched = false;struct task_struct *p, *push_task;
@@ -1708,6 +1763,446 @@ static void pull_rt_task(struct rq *this_rq)resched_curr (this_rq); } +#ifdef CONFIG_RT_GROUP_SCHED +/* +* Find the lowest priority runqueue among the runqueues of the same +* task group. Unlike find_lowest_rt(), this does not mean that the +* lowest priority cpu is running tasks from this runqueue. +*/ +static int group_find_lowest_rt_rq (struct task_struct *task,struct rt_rq * task_rt_rq) +{ +struct sched_domain *sd; +struct cpumask mask, *lowest_mask = &mask; +struct sched_dl_entity *dl_se; +struct rt_rq *rt_rq; +int prio, lowest_prio; +int cpu, this_cpu =smp_processor_id (); + +BUG_ON (task->sched_task_group != task_rt_rq->tg); + +if (task->nr_cpus_allowed ==1 ) +return -1 ;/* No other targets possible */ + + lowest_prio = task->prio -1 ; +cpumask_clear (lowest_mask); +for_each_cpu_and (cpu, cpu_online_mask, task->cpus_ptr) { + dl_se = task_rt_rq->tg->dl_se[cpu]; + rt_rq = &dl_se->my_q->rt; + prio = rt_rq->highest_prio.curr; + +/* +* If we're on asym system ensure we consider the different capacities +* of the CPUs when searching for the lowest_mask. +*/ +if (dl_se->dl_throttled || !rt_task_fits_capacity (task, cpu)) +continue ; + +if (prio >= lowest_prio) { +if (prio > lowest_prio) { +cpumask_clear (lowest_mask); + lowest_prio = prio; + } + +cpumask_set_cpu (cpu, lowest_mask); + } + } + +if (cpumask_empty (lowest_mask)) +return -1 ; + +/* +* At this point we have built a mask of CPUs representing the +* lowest priority tasks in the system. Now we want to elect +* the best one based on our affinity and topology. +* +* We prioritize the last CPU that the task executed on since +* it is most likely cache-hot in that location. +*/ + cpu =task_cpu (task); +if (cpumask_test_cpu (cpu, lowest_mask)) +return cpu; + +/* +* Otherwise, we consult the sched_domains span maps to figure +* out which CPU is logically closest to our hot cache data. +*/ +if (!cpumask_test_cpu (this_cpu, lowest_mask)) + this_cpu =-1 ;/* Skip this_cpu opt if not among lowest */ + +rcu_read_lock (); +for_each_domain (cpu, sd) { +if (sd->flags &SD_WAKE_AFFINE ) { +int best_cpu; + +/* +* "this_cpu" is cheaper to preempt than a +* remote processor. +*/ +if (this_cpu !=-1 && +cpumask_test_cpu (this_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return this_cpu; + } + + best_cpu =cpumask_any_and_distribute (lowest_mask, +sched_domain_span (sd)); +if (best_cpu < nr_cpu_ids) { +rcu_read_unlock (); +return best_cpu; + } + } + } +rcu_read_unlock (); + +/* +* And finally, if there were no matches within the domains +* just give the caller *something* to work with from the compatible +* locations. +*/ +if (this_cpu !=-1 ) +return this_cpu; + + cpu =cpumask_any_distribute (lowest_mask); +if (cpu < nr_cpu_ids) +return cpu; + +return -1 ; +} + +/* +* Find and lock the lowest priority runqueue among the runqueues +* of the same task group. Unlike find_lock_lowest_rt(), this does not +* mean that the lowest priority cpu is running tasks from this runqueue. +*/ +static struct rt_rq *group_find_lock_lowest_rt_rq (struct task_struct *task,struct rt_rq *rt_rq) +{ +struct rq *rq =rq_of_rt_rq (rt_rq); +struct rq *lowest_rq; +struct rt_rq *lowest_rt_rq =NULL ; +struct sched_dl_entity *lowest_dl_se; +int tries, cpu; + +BUG_ON (task->sched_task_group != rt_rq->tg); + +for (tries =0 ; tries <RT_MAX_TRIES ; tries++) { + cpu =group_find_lowest_rt_rq (task, rt_rq); + +if ((cpu ==-1 ) || (cpu == rq->cpu)) +break ; + + lowest_dl_se = rt_rq->tg->dl_se[cpu]; + lowest_rt_rq = &lowest_dl_se->my_q->rt; + lowest_rq =cpu_rq (cpu); + +if (lowest_rt_rq->highest_prio.curr <= task->prio) { +/* +* Target rq has tasks of equal or higher priority, +* retrying does not release any lock and is unlikely +* to yield a different result. +*/ + lowest_rt_rq =NULL ; +break ; + } + +/* if the prio of this runqueue changed, try again */ +if (double_lock_balance (rq, lowest_rq)) { +/* +* We had to unlock the run queue. In +* the mean time, task could have +* migrated already or had its affinity changed. +* Also make sure that it wasn't scheduled on its rq. +* It is possible the task was scheduled, set +* "migrate_disabled" and then got preempted, so we must +* check the task migration disable flag here too. +*/ +if (unlikely (is_migration_disabled (task) || + lowest_dl_se->dl_throttled || + !cpumask_test_cpu (lowest_rq->cpu, &task->cpus_mask) || + task !=pick_next_pushable_task (rt_rq))) { + +double_unlock_balance (rq, lowest_rq); + lowest_rt_rq =NULL ; +break ; + } + } + +/* If this rq is still suitable use it. */ +if (lowest_rt_rq->highest_prio.curr > task->prio) +break ; + +/* try again */ +double_unlock_balance (rq, lowest_rq); + lowest_rt_rq =NULL ; + } + +return lowest_rt_rq; +} + +static int group_push_rt_task (struct rt_rq *rt_rq,bool pull) +{ +BUG_ON (!is_dl_group (rt_rq)); + +struct rq *rq =rq_of_rt_rq (rt_rq); +struct task_struct *next_task; +struct rq *lowest_rq; +struct rt_rq *lowest_rt_rq; +int ret =0 ; + +if (!rt_rq->overloaded) +return 0 ; + + next_task =pick_next_pushable_task (rt_rq); +if (!next_task) +return 0 ; + +retry: +if (is_migration_disabled (next_task)) { +struct task_struct *push_task =NULL ; +int cpu; + +if (!pull || rq->push_busy) +return 0 ; + +/* +* If the current task does not belong to the same task group +* we cannot push it away. +*/ +if (rq->curr->sched_task_group != rt_rq->tg) +return 0 ; + +/* +* Invoking group_find_lowest_rt_rq() on anything but an RT task doesn't +* make sense. Per the above priority check, curr has to +* be of higher priority than next_task, so no need to +* reschedule when bailing out. +* +* Note that the stoppers are masqueraded as SCHED_FIFO +* (cf. sched_set_stop_task()), so we can't rely on rt_task(). +*/ +if (rq->curr->sched_class != &rt_sched_class) +return 0 ; + + cpu =group_find_lowest_rt_rq (rq->curr, rt_rq); +if (cpu ==-1 || cpu == rq->cpu) +return 0 ; + +/* +* Given we found a CPU with lower priority than @next_task, +* therefore it should be running. However we cannot migrate it +* to this other CPU, instead attempt to push the current +* running task on this CPU away. +*/ + push_task =get_push_task (rq); +if (push_task) { +preempt_disable (); +raw_spin_rq_unlock (rq); +stop_one_cpu_nowait (rq->cpu, push_cpu_stop, + push_task, &rq->push_work); +preempt_enable (); +raw_spin_rq_lock (rq); + } + +return 0 ; + } + +if (WARN_ON (next_task == rq->curr)) +return 0 ; + +/* We might release rq lock */ +get_task_struct (next_task); + +/* group_find_lock_lowest_rq locks the rq if found */ + lowest_rt_rq =group_find_lock_lowest_rt_rq (next_task, rt_rq); +if (!lowest_rt_rq) { +struct task_struct *task; +/* +* group_find_lock_lowest_rt_rq releases rq->lock +* so it is possible that next_task has migrated. +* +* We need to make sure that the task is still on the same +* run-queue and is also still the next task eligible for +* pushing. +*/ + task =pick_next_pushable_task (rt_rq); +if (task == next_task) { +/* +* The task hasn't migrated, and is still the next +* eligible task, but we failed to find a run-queue +* to push it to. Do not retry in this case, since +* other CPUs will pull from us when ready. +*/ + goto out; + } + +if (!task) +/* No more tasks, just exit */ + goto out; + +/* +* Something has shifted, try again. +*/ +put_task_struct (next_task); + next_task = task; + goto retry; + } + + lowest_rq =rq_of_rt_rq (lowest_rt_rq); + +move_queued_task_locked (rq, lowest_rq, next_task); +resched_curr (lowest_rq); + ret =1 ; + +double_unlock_balance (rq, lowest_rq); +out: +put_task_struct (next_task); + +return ret; +} + +static void group_pull_rt_task (struct rt_rq *this_rt_rq) +{ +BUG_ON (!is_dl_group (this_rt_rq)); + +struct rq *this_rq =rq_of_rt_rq (this_rt_rq); +int this_cpu = this_rq->cpu, cpu; +bool resched = false; +struct task_struct *p, *push_task =NULL ; +struct rt_rq *src_rt_rq; +struct rq *src_rq; +struct sched_dl_entity *src_dl_se; + +for_each_online_cpu (cpu) { +if (this_cpu == cpu) +continue ; + + src_dl_se = this_rt_rq->tg->dl_se[cpu]; + src_rt_rq = &src_dl_se->my_q->rt; + +if (src_rt_rq->rt_nr_running <=1 && !src_dl_se->dl_throttled) +continue ; + + src_rq =rq_of_rt_rq (src_rt_rq); + +/* +* Don't bother taking the src_rq->lock if the next highest +* task is known to be lower-priority than our current task. +* This may look racy, but if this value is about to go +* logically higher, the src_rq will push this task away. +* And if its going logically lower, we do not care +*/ +if (src_rt_rq->highest_prio.next >= + this_rt_rq->highest_prio.curr) +continue ; + +/* +* We can potentially drop this_rq's lock in +* double_lock_balance, and another CPU could +* alter this_rq +*/ + push_task =NULL ; +double_lock_balance (this_rq, src_rq); + +/* +* We can pull only a task, which is pushable +* on its rq, and no others. +*/ + p =pick_highest_pushable_task (src_rt_rq, this_cpu); + +/* +* Do we have an RT task that preempts +* the to-be-scheduled task? +*/ +if (p && (p->prio < this_rt_rq->highest_prio.curr)) { +WARN_ON (p == src_rq->curr); +WARN_ON (!task_on_rq_queued (p)); + +/* +* There's a chance that p is higher in priority +* than what's currently running on its CPU. +* This is just that p is waking up and hasn't +* had a chance to schedule. We only pull +* p if it is lower in priority than the +* current task on the run queue +*/ +if (src_rq->curr->sched_task_group == this_rt_rq->tg && + p->prio < src_rq->curr->prio) + goto skip; + +if (is_migration_disabled (p)) { +/* +* If the current task does not belong to the same task group +* we cannot push it away. +*/ +if (src_rq->curr->sched_task_group != this_rt_rq->tg) + goto skip; + + push_task =get_push_task (src_rq); + }else { +move_queued_task_locked (src_rq, this_rq, p); + resched = true; + } +/* +* We continue with the search, just in +* case there's an even higher prio task +* in another runqueue. (low likelihood +* but possible) +*/ + } +skip: +double_unlock_balance (this_rq, src_rq); + +if (push_task) { +preempt_disable (); +raw_spin_rq_unlock (this_rq); +stop_one_cpu_nowait (src_rq->cpu, push_cpu_stop, + push_task, &src_rq->push_work); +preempt_enable (); +raw_spin_rq_lock (this_rq); + } + } + +if (resched) +resched_curr (this_rq); +} + +static void group_push_rt_tasks (struct rt_rq *rt_rq) +{ +while (group_push_rt_task (rt_rq, false)) + ; +} + +static void group_push_rt_tasks_callback (struct rq *global_rq) +{ +struct rt_rq *rt_rq = &global_rq->rq_to_push_from->rt; + +BUG_ON (global_rq->rq_to_push_from ==NULL ); +BUG_ON (rt_rq->rq != global_rq); + +if ((rt_rq->rt_nr_running >1 ) || + (dl_group_of (rt_rq)->dl_throttled ==1 )) { + +while (group_push_rt_task (rt_rq, false)) + ; + } + + global_rq->rq_to_push_from =NULL ; +} + +static void group_pull_rt_task_callback (struct rq *global_rq) +{ +struct rt_rq *rt_rq = &global_rq->rq_to_pull_to->rt; + +BUG_ON (global_rq->rq_to_pull_to ==NULL ); +BUG_ON (rt_rq->rq != global_rq); + +group_pull_rt_task (rt_rq); + global_rq->rq_to_pull_to =NULL ; +} +#else /* CONFIG_RT_GROUP_SCHED */ +static void group_pull_rt_task (struct rt_rq *this_rt_rq) { } +static void group_push_rt_tasks (struct rt_rq *rt_rq) { } +#endif /* CONFIG_RT_GROUP_SCHED */ +/* * If we are not running and we are not going to reschedule soon, we should * try to push tasks away now
kernel/sched/sched.h +10 -0
@@ -1282,6 +1282,16 @@ struct rq {call_single_data_t cfsb_csd;struct list_head cfsb_csd_list;#endif + +#ifdef CONFIG_RT_GROUP_SCHED +/* +* Balance callbacks operate only on global runqueues. +* These pointers allow referencing cgroup specific runqueues +* for balancing operations. +*/ +struct rq *rq_to_push_from; +struct rq *rq_to_pull_to; +#endif };#ifdef CONFIG_FAIR_GROUP_SCHED
Signature · 1 line
2.51.0
RFC PATCH v3 20/24 sched/rt: Add HCBS migration related checks and function calls
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add HCBS related checks and operations to allow rt-task migration, differentiating between cgroup's tasks or tasks that run on the global runqueue.
--- kernel/sched/rt.c | 69 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 53 insertions(+), 16 deletions(-)
kernel/sched/rt.c +53 -16
@@ -884,6 +884,11 @@ select_task_rq_rt(struct task_struct *p, int cpu, int flags)struct rq *rq;bool test; +/* Just return the task_cpu for processes inside task groups */ +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) && +is_dl_group (rt_rq_of_se (&p->rt))) + goto out; +/* For anything but wake ups, just return the task_cpu */ if (!(flags & (WF_TTWU |WF_FORK ))) goto out;
@@ -983,7 +988,10 @@ static int balance_rt(struct rq *rq, struct task_struct *p, struct rq_flags *rf)* not yet started the picking loop. */ rq_unpin_lock (rq, rf); -pull_rt_task (rq); +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq_of_se (&p->rt))) +group_pull_rt_task (rt_rq_of_se (&p->rt)); +else +pull_rt_task (rq);rq_repin_lock (rq, rf); }
@@ -1085,7 +1093,10 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool fif (rq->donor->sched_class != &rt_sched_class)update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); -rt_queue_push_tasks (rt_rq); +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +rt_queue_push_from_group (rt_rq); +else +rt_queue_push_tasks (rt_rq); }static struct sched_rt_entity *pick_next_rt_entity (struct rt_rq *rt_rq)
@@ -1148,6 +1159,13 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s*/ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 )enqueue_pushable_task (rt_rq, p); + +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) { +struct sched_dl_entity *dl_se =dl_group_of (rt_rq); + +if (dl_se->dl_throttled) +rt_queue_push_from_group (rt_rq); + } }/* Only try algorithms three times */
@@ -2209,6 +2227,7 @@ static void group_push_rt_tasks(struct rt_rq *rt_rq) { }*/ static void task_woken_rt (struct rq *rq,struct task_struct *p) { +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt);bool need_to_push = !task_on_cpu (rq, p) && !test_tsk_need_resched (rq->curr) && p->nr_cpus_allowed > 1 &&
@@ -2216,7 +2235,12 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) (rq->curr->nr_cpus_allowed <2 || rq->donor->prio <= p->prio); -if (need_to_push) +if (!need_to_push) +return ; + +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +group_push_rt_tasks (rt_rq); +else push_rt_tasks (rq); }
@@ -2256,7 +2280,9 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)if (!task_on_rq_queued (p) || rt_rq->rt_nr_running)return ; -if (!IS_ENABLED (CONFIG_RT_GROUP_SCHED )) +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +rt_queue_pull_to_group (rt_rq); +else rt_queue_pull_task (rt_rq); }
@@ -2277,12 +2303,21 @@ void __init init_sched_rt_class(void)*/ static void switched_to_rt (struct rq *rq,struct task_struct *p) { +struct rt_rq *rt_rq; +/* * If we are running, update the avg_rt tracking, as the running time * will now on be accounted into the latter. */ if (task_current (rq, p)) {update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); + +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq_of_se (&p->rt))) { +struct sched_dl_entity *dl_se =dl_group_of (rt_rq_of_se (&p->rt)); + + p->dl_server = dl_se; + } +return ; }
@@ -2292,17 +2327,15 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)* then see if we can move to another run queue. */ if (task_on_rq_queued (p)) { - -#ifndef CONFIG_RT_GROUP_SCHED -if (p->nr_cpus_allowed >1 && rq->rt.overloaded) -rt_queue_push_tasks (rt_rq_of_se (&p->rt)); -#else -if (rt_rq_of_se (&p->rt)->overloaded) { - }else { -if (p->prio < rq->curr->prio) -resched_curr (rq); + rt_rq =rt_rq_of_se (&p->rt); +if (!is_dl_group (rt_rq) && p->nr_cpus_allowed >1 && rq->rt.overloaded) { +rt_queue_push_tasks (rt_rq); +return ; + }else if (is_dl_group (rt_rq) && rt_rq->overloaded) { +rt_queue_push_from_group (rt_rq); +return ; } -#endif +if (p->prio < rq->donor->prio &&cpu_online (cpu_of (rq)))resched_curr (rq); }
@@ -2325,8 +2358,12 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)* If our priority decreases while running, we * may need to pull tasks to this runqueue. */ -if (!IS_ENABLED (CONFIG_RT_GROUP_SCHED ) && oldprio < p->prio) -rt_queue_pull_task (rt_rq); +if (oldprio < p->prio) { +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +rt_queue_pull_to_group (rt_rq); +else +rt_queue_pull_task (rt_rq); + }/* * If there's a higher priority task waiting to run
Signature · 1 line
2.51.0
RFC PATCH v3 21/24 sched/deadline: Make rt-cgroup's servers pull tasks on timer replenishment
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
--- kernel/sched/deadline.c | 6 +++++- kernel/sched/rt.c | 6 +++++- 2 files changed, 10 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +5 -1
@@ -1290,6 +1290,7 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_ {struct rq *rq =rq_of_dl_se (dl_se);u64 fw; +bool has_tasks; scoped_guard (rq_lock, rq) {struct rq_flags *rf = &scope.rf;
@@ -1303,7 +1304,10 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_if (!dl_se->dl_runtime)return HRTIMER_NORESTART ; -if (!dl_se->server_has_tasks (dl_se)) { +rq_unpin_lock (rq, rf); + has_tasks = dl_se->server_has_tasks (dl_se); +rq_repin_lock (rq, rf); +if (!has_tasks) {replenish_dl_entity (dl_se);dl_server_stopped (dl_se);return HRTIMER_NORESTART ;
kernel/sched/rt.c +5 -1
@@ -1,4 +1,3 @@ -#pragma GCC diagnostic ignored "-Wunused-function"// SPDX-License-Identifier: GPL-2.0 /* * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -153,6 +152,11 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool fstatic bool rt_server_has_tasks (struct sched_dl_entity *dl_se) { +#ifdef CONFIG_SMP +struct rt_rq *rt_rq = &dl_se->my_q->rt; + +group_pull_rt_task (rt_rq); +#endif return !!dl_se->my_q->rt.rt_nr_running; }
Signature · 1 line
2.51.0
RFC PATCH v3 22/24 sched/deadline: Fix HCBS migrations on server stop
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Do not unthrottle a non-fair-server dl_server_stop(), since it ends up being stopped when throttled (we try to migrate all the RT tasks away from it).
Notes:
This is a temporary workaround, but it will be hopefully removed in favor of less invasive code.
--- kernel/sched/deadline.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +4 -2
@@ -1707,9 +1707,11 @@ void dl_server_stop(struct sched_dl_entity *dl_se)return ;dequeue_dl_entity (dl_se,DEQUEUE_SLEEP ); -hrtimer_try_to_cancel (&dl_se->dl_timer); +if (dl_se == &rq_of_dl_se (dl_se)->fair_server) { +hrtimer_try_to_cancel (&dl_se->dl_timer); + dl_se->dl_throttled =0 ; + } dl_se->dl_defer_armed =0 ; - dl_se->dl_throttled =0 ; dl_se->dl_server_active =0 ; }
Signature · 1 line
2.51.0
RFC PATCH v3 23/24 sched/core: Execute enqueued balance callbacks when changing allowed CPUs
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Execute balancing callbacks when setting the affinity of a task, since the HCBS scheduler may request balancing of throttled dl_servers to fully utilize the server's bandwidth.
--- kernel/sched/core.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/sched/core.c +13 -0
@@ -2950,6 +2950,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flagif (cpumask_test_cpu (task_cpu (p), &p->cpus_mask) || (task_current_donor (rq, p) && !task_current (rq, p))) {struct task_struct *push_task =NULL ; +struct balance_callback *head;if ((flags &SCA_MIGRATE_ENABLE ) && (p->migration_flags &MDF_PUSH ) && !rq->push_busy) {
@@ -2968,11 +2969,13 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag }preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf);if (push_task) {stop_one_cpu_nowait (rq->cpu, push_cpu_stop, p, &rq->push_work); } +balance_callbacks (rq, head);preempt_enable ();if (complete)
@@ -3027,6 +3030,8 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag }if (task_on_cpu (rq, p) ||READ_ONCE (p->__state) ==TASK_WAKING ) { +struct balance_callback *head; +/* * MIGRATE_ENABLE gets here because 'p == current', but for * anything else we cannot do is_migration_disabled(), punt
@@ -3040,16 +3045,19 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag p->migration_flags &= ~MDF_PUSH ;preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf);if (!stop_pending) {stop_one_cpu_nowait (cpu_of (rq), migration_cpu_stop, &pending->arg, &pending->stop_work); } +balance_callbacks (rq, head);preempt_enable ();if (flags &SCA_MIGRATE_ENABLE )return 0 ; }else { +struct balance_callback *head;if (!is_migration_disabled (p)) {if (task_on_rq_queued (p))
@@ -3060,7 +3068,12 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag complete = true; } } + +preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf); +balance_callbacks (rq, head); +preempt_enable ();if (complete)complete_all (&pending->done);
Signature · 1 line
2.51.0
RFC PATCH v3 24/24 sched/core: Execute enqueued balance callbacks when migrating task betweeen cgroups
In reply to Yuri AndriaccioExecute balancing callbacks when migrating task between cgroups, since the HCBS scheduler, similarly to the previous patch, may request balancing of throttled dl_servers to fully utilize the server's bandwidth.
--- kernel/sched/core.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/core.c +9 -2
@@ -9251,10 +9251,11 @@ void sched_move_task(struct task_struct *tsk, bool for_autogroup) {int queued, running, queue_flags =DEQUEUE_SAVE |DEQUEUE_MOVE |DEQUEUE_NOCLOCK ; +struct balance_callback *head;struct rq *rq; +struct rq_flags rf; -CLASS (task_rq_lock, rq_guard)(tsk); - rq = rq_guard.rq; + rq =task_rq_lock (tsk, &rf);update_rq_clock (rq);
@@ -9281,6 +9282,12 @@ void sched_move_task(struct task_struct *tsk, bool for_autogroup)*/ resched_curr (rq); } + +preempt_disable (); + head =splice_balance_callbacks (rq); +task_rq_unlock (rq, tsk, &rf); +balance_callbacks (rq, head); +preempt_enable (); } static struct cgroup_subsys_state *
Signature · 1 line
2.51.0
Juri Lelli
maintainerHello!
Quoted 11 lines · Yuri Andriaccio
Hello,
This is the v3 for Hierarchical Constant Bandwidth Server, aiming at replacing the current
RT_GROUP_SCHEDmechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ . You can find the previous versions of this patchset at the bottom of the page, in particular version 1 which talks in more detail what this patchset is all about and how it is implemented.
Quoted 6 lines
This v3 version further reworks some of the patches as suggested by Juri Lelli. While most of the work is refactorings, the following were also changed:
- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/
Peter wasn't indeed happy with that patch, but I am not sure we finished that discussion. Both myself and Luca had further objections to what Peter said, but not further replies after (which can very well be a sign that he is still adamnt in saying no go away :). Peter?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20250904091217.78de3dde@luca64/
...
Quoted 5 lines
For this v3 version we've also polished the testing system we are using and made it public for testers to run on their own machines. The source code can be found at https://github.com/Yurand2000/HCBS-Test-Suite , along with a README that explains how to use it. Nonetheless I've reported a description of the tools and instruction later in the page.
Thanks for this. Quite cool. I tried to run the tests, but it looks like the migration set brought my qemu 8 CPUs system to an halt (had to kill the thing). I will try again.
I have been working a bit in trying to come up with a testing framework for SCHED_DEADLINE, which I didn't unfortunately posted yet mostly because I was waiting for the discussion about the patch mentioned above to settle (which would require adaptation in the tests that check for bandwidth limits). You can find the idea here [1]. It's thought to be an addition to kselftests. I believe your test suite can be extended to cover the tests I implemented and more, so I am not super attached to my attempt, but it would be good (I think) to converge to something that we can maintain for the future, so maybe have a plan to possibly merge the suites. What do you think?
1 - https://github.com/jlelli/linux/commits/experimental/deadline-selftests-scripts/
...
Quoted 6 lines
Testing v3:
The HCBS mechanism has been evaluated on several syntetic tests which are designed to stress the HCBS scheduler and verify that non-interference and mathematical schedulability guarantees are really enforced by the scheduling algorithm.
...
Quoted 4 lines
The tasksets minimal requirements were computed using a closed-source software, explaining why the tasksets are supplied separately. A open-source analyser is being written to update this step in the future and also allow for more customization for the testers.
On this (generation of random taskset and corresponding schedulability checks) I also started working on a different tool, rt-audit.
https://github.com/jlelli/rt-audit
It's very simple, the tool, the random generation and the checks. But I honestly like the idea that it's based on rt-app. Please take a look if you have a chance and tell me what you think. Again, I feel it would be good to converge towards something open that we can maintain.
I will be trying to find time to continue testing and reviewing this RFC of course.
Thanks,
Juri
Juri Lelli
maintainerOn 02/10/25 10:00, Juri Lelli wrote:
...
Quoted 2 lines
I will be trying to find time to continue testing and reviewing this RFC of course.
I provided comments up to the last part of the series that implements migration. I will now stop on purpose and wait for replies/new versions of the patches I reviewed. I believe we could try to leave migration "for later" while we solidify the basics. What do you think?
Thanks,
Juri
Hi,
Quoted 4 lines
I provided comments up to the last part of the series that implements migration. I will now stop on purpose and wait for replies/new versions of the patches I reviewed. I believe we could try to leave migration "for later" while we solidify the basics. What do you think?
Thanks a lot for your comments. It also makes sense to us to solidify the basics before going on to the migration code and later updates. We have sent the migration related patches just to show the end goal of this patchset and give some intuition of possible use cases and demonstration on the provided guarantees. We will work on your comments and submit a v4 patchset soon.
Have a nice day,
Yuri
Juri Lelli
maintainerQuoted 21 lines · Juri Lelli
Hello!
Quoted 11 lines · Yuri Andriaccio
Hello,
This is the v3 for Hierarchical Constant Bandwidth Server, aiming at replacing the current
RT_GROUP_SCHEDmechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ . You can find the previous versions of this patchset at the bottom of the page, in particular version 1 which talks in more detail what this patchset is all about and how it is implemented.Quoted 6 lines
This v3 version further reworks some of the patches as suggested by Juri Lelli. While most of the work is refactorings, the following were also changed:
- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/
Quoted 7 lines
Peter wasn't indeed happy with that patch, but I am not sure we finished that discussion. Both myself and Luca had further objections to what Peter said, but not further replies after (which can very well be a sign that he is still adamnt in saying no go away :). Peter?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20250904091217.78de3dde@luca64/
I had a quick chat with Peter on IRC about this. We now seem to agree that a third option would be to move to explicitly account dl-server(s), correspondingly moving from a 95% to 100% limit. That would also make our life easier in the future with additional dl-servers (e.g. scx-server).
What do you think?
Hi Juri,
On Mon, 20 Oct 2025 11:40:22 +0200
Juri Lelli <[email protected]> wrote:
[...]
Quoted 16 lines
Quoted 7 lines
Quoted 6 lines
- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/Quoted 8 lines
Peter wasn't indeed happy with that patch, but I am not sure we finished that discussion. Both myself and Luca had further objections to what Peter said, but not further replies after (which can very well be a sign that he is still adamnt in saying no go away :). Peter?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20250904091217.78de3dde@luca64/
Quoted 7 lines
I had a quick chat with Peter on IRC about this. We now seem to agree that a third option would be to move to explicitly account dl-server(s), correspondingly moving from a 95% to 100% limit. That would also make our life easier in the future with additional dl-servers (e.g. scx-server).
What do you think?
This looks like another good solution, thanks!
So, if I understand well with this approach /proc/sys/kernel/sched_rt_{runtime, period}_us would be set to 100% as a default, right?
It is often useful to know what is the maximum CPU utilization that can be guaranteed to real-time tasks... With this approach, it would be 100% - <dl_server utilization>, but this can change when scx servers are added... What about making this information available to userspace programs? (maybe /proc/sys/kernel/sched_rt_{runtime, period}_us could provide such information? Or is it better to add a new interface?)
Thanks,
Luca
Juri Lelli
maintainerQuoted 35 lines · luca abeni
Hi Juri,
On Mon, 20 Oct 2025 11:40:22 +0200
Juri Lelli <[email protected]> wrote:
[...]Quoted 16 lines
Quoted 7 lines
Quoted 6 lines
- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/Quoted 8 lines
Peter wasn't indeed happy with that patch, but I am not sure we finished that discussion. Both myself and Luca had further objections to what Peter said, but not further replies after (which can very well be a sign that he is still adamnt in saying no go away :). Peter?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20250904091217.78de3dde@luca64/Quoted 7 lines
I had a quick chat with Peter on IRC about this. We now seem to agree that a third option would be to move to explicitly account dl-server(s), correspondingly moving from a 95% to 100% limit. That would also make our life easier in the future with additional dl-servers (e.g. scx-server).
What do you think?
This looks like another good solution, thanks!
So, if I understand well with this approach
/proc/sys/kernel/sched_rt_{runtime, period}_us would be set to 100% as a default, right?
Quoted 6 lines
It is often useful to know what is the maximum CPU utilization that can be guaranteed to real-time tasks... With this approach, it would be 100% - <
dl_serverutilization>, but this can change when scx servers are added... What about making this information available to userspace programs? (maybe/proc/sys/kernel/sched_rt_{runtime, period}_us could provide such information? Or is it better to add a new interface?)
Not sure. If we set it to 100% by default (as you suggest, which makes sense to me) I wonder what would be a usecase/need to set it to less than 100% later on. We have the debug interface for tweaking dl-servers and sched_rt_ interface doesn't distinguish between DEADLINE and RT anyway (so no way to leave some "bandwidth" around for RT tasks).
Maybe it's an interface we want to start deprecating and we can come up with something better and/or more useful? Peter?