Hierarchical Constant Bandwidth Server
v2 cover Hierarchical Constant Bandwidth Server
Hello,
This is the v2 for Hierarchical Constant Bandwidth Server, aiming at replacing the current RT_GROUP_SCHED mechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ . You can find the v1 of this patchset at the bottom of the page, which talks in more detail what this patchset is all about and how it is implemented.
The big update for this v2 version is the addition of migration code, which allows to migrate tasks between different CPUs (following of course affinity settings).
As requested, we've split the big patches in smaller chunks in order to improve in readability. Additionally, it has been rebased on the latest tip/master to keep up with the latest scheduler updates and new features of dl_servers.
Last but not least, the first patch, which has been presented separately at
https://lore.kernel.org/all/[email protected]/ , is necessary to fully utilize the deadline bandwidth while keeping the fair-servers active. You can refer to the aforementioned link for details. The issue presented in this patch also reflects in HCBS: in the current version of the kernel, by default, 5% of the realtime bandwidth is reserved for fair-servers, 5% is not usable, and only the remaining 90% could be used by deadline tasks, or in our case, by HCBS dl_servers. The first patch addresses this issue and allows to fully utilize the default 95% of bandwidth for rt-tasks/servers.
Summary of the patches:
1) Account fair-servers bw separately from other dl tasks and servers bw.
2-5) Preparation patches, so that the RT classes' code can be used both
for normal and cgroup scheduling.
6-15) Implementation of HCBS, no migration and only one level hierarchy.
The old RT_GROUP_SCHED code is removed.
16-18) Remove cgroups v1 in favour of v2.
19) Add support for deeper hierarchies.
20-25) Add support for tasks migration.
Updates from v1:
- Rebase to tip/master.
- Add migration code.
- Split big patches for more readability.
- Refactor code to use guarded locks where applicable.
- Remove unnecessary patches from v1 which have been addressed differently by mainline updates.
- Remove unnecessary checks and general code cleanup.
Notes:
Task migration support needs some extra work to reduce its invasiveness, especially patches 22-23.
Testing v2:
The HCBS mechanism has been further evaluated on two fully-fledged distros, instead of virtual machines, demonstrating stability in this latest version. A small suite of regression tests shows that the newly added mechanism does not break fair-servers and other scheduling mechanisms. Stress tests show that our implementation is robust while time-based tests demonstrate that the theoretical analysis of real-time tasksets matches with the implementation.
The tests can be found at https://github.com/Yurand2000/HCBS-rust-initrd . The executables are essentially the same as the ones mentioned in the v1 version, minor some updates. You can refer to that for additional details.
Future Work:
We want to further test this patchset, and provide a more commented description of the test suite so that it can be fully automated for testing also by other people. Additionally, we will finish the currently partial/untested, implementation of HCBS with different runtimes per CPU, instead of having the same runtime allocated on all CPUs, to include it in a future RCF.
Future patches:
- HCBS with different runtimes per CPU.
- capacity aware bandwidth reservation.
- enable/disable dl_servers when a CPU goes online/offline.
Have a nice day,
Yuri
v1: https://lore.kernel.org/all/[email protected]/
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Yuri Andriaccio (6):
sched/deadline: Remove fair-servers from real-time task's bandwidth
accounting
sched/rt: Disable RT_GROUP_SCHED
sched/deadline: Account rt-cgroups bandwidth in
sched_dl_global_validate
sched/rt: Remove support for cgroups-v1
sched/rt: Zero rt-cgroups default bandwidth
sched/core: Execute enqueued balance callbacks when migrating task
betweeen cgroups
luca abeni (19):
sched/deadline: Do not access dl_se->rq directly sched/deadline: Distinct between dl_rq and my_q
sched/rt: Pass an rt_rq instead of an rq where needed sched/rt: Move some functions from rt.c to sched.h sched/rt: Introduce HCBS specific structs in task_group sched/deadline: Account rt-cgroups bandwidth in deadline tasks
schedulability tests.
sched/core: Initialize root_task_group
sched/deadline: Add dl_init_tg
sched/rt: Add {alloc/free}_rt_sched_group and dl_server specific
functions
sched/rt: Add HCBS related checks and operations for rt tasks sched/rt: Update rt-cgroup schedulability checks sched/rt: Remove old RT_GROUP_SCHED data structures sched/core: Cgroup v2 support
sched/deadline: Allow deeper hierarchies of RT cgroups sched/rt: Add rt-cgroup migration
sched/rt: add HCBS migration related checks and function calls sched/deadline: Make rt-cgroup's servers pull tasks on timer
replenishment
sched/deadline: Fix HCBS migrations on server stop sched/core: Execute enqueued balance callbacks when changing allowed
CPUs
include/linux/sched.h | 10 +- kernel/sched/autogroup.c | 4 +- kernel/sched/core.c | 68 +- kernel/sched/deadline.c | 311 ++-- kernel/sched/debug.c | 6 - kernel/sched/fair.c | 6 +- kernel/sched/rt.c | 3024 ++++++++++++++++++-------------------- kernel/sched/sched.h | 140 +- kernel/sched/syscalls.c | 6 +- kernel/sched/topology.c | 8 - 10 files changed, 1829 insertions(+), 1754 deletions(-)
Signature · 1 line
2.50.1
RFC PATCH v2 1/25 sched/deadline: Remove fair-servers from real-time task's bandwidth accounting
Fair-servers are currently used in place of the old RT_THROTTLING mechanism to prevent the starvation of SCHED_OTHER (and other lower priority) tasks when real-time FIFO/RR processes are trying to fully utilize the CPU. To allow the RT_THROTTLING mechanism, the maximum allocatable bandwidth for real-time tasks has been limited to 95% of the CPU-time.
The RT_THROTTLING mechanism is now removed in favor of fair-servers, which are currently set to use, as expected, 5% of the CPU-time. Still, they share the same bandwidth that allows to run real-time tasks, and which is still set to 95% of the total CPU-time. This means that by removing the RT_THROTTLING mechanism, the bandwidth remaning for real-time SCHED_DEADLINE tasks and other dl-servers (FIFO/RR are not affected) is only 90%.
This patch reclaims the 5% lost CPU-time, which is definitely reserved for SCHED_OTHER tasks, but should not be accounted togheter with the other real-time tasks. More generally, the fair-servers' bandwidth must not be accounted with other real-time tasks.
Updates:
- Make the fair-servers' bandwidth not be accounted into the total allocated bandwidth for real-time tasks.
- Remove the admission control test when allocating a fair-server.
- Do not account for fair-servers in the GRUB's bandwidth reclaiming mechanism.
- Limit the max bandwidth to (BW_UNIT - max_rt_bw) when changing the parameters of a fair-server, preventing overcommitment.
- Add dl_bw_fair, which computes the total allocated bandwidth of the fair-servers in the given root-domain.
- Update admission tests (in sched_dl_global_validate) when changing the maximum allocatable bandwidth for real-time tasks, preventing overcommitment.
Since the fair-server's bandwidth can be changed through debugfs, it has not been enforced that a fair-server's bw must be always equal to (BW_UNIT - max_rt_bw), rather it must be less or equal to this value. This allows retaining the fair-servers' settings changed through the debugfs when chaning the max_rt_bw.
This also means that in order to increase the maximum bandwidth for real-time tasks, the bw of fair-servers must be first decreased through debugfs otherwise admission tests will fail, and viceversa, to increase the bw of fair-servers, the bw of real-time tasks must be reduced beforehand.
This v2 version addresses the compilation error on i386 reported at:
https://lore.kernel.org/oe-kbuild-all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
--- kernel/sched/deadline.c | 66 ++++++++++++++++++----------------------- kernel/sched/sched.h | 1 - kernel/sched/topology.c | 8 ----- 3 files changed, 29 insertions(+), 46 deletions(-)
kernel/sched/deadline.c +29 -37
@@ -141,6 +141,24 @@ static inline int dl_bw_cpus(int i)return cpus; } +static inline u64 dl_bw_fair (int i) +{ +struct root_domain *rd =cpu_rq (i)->rd; +u64 fair_server_bw =0 ; + +RCU_LOCKDEP_WARN (!rcu_read_lock_sched_held (), +"sched RCU must be held" ); + +if (cpumask_subset (rd->span, cpu_active_mask)) + i =cpumask_first (rd->span); + +for_each_cpu_and (i, rd->span, cpu_active_mask) { + fair_server_bw +=cpu_rq (i)->fair_server.dl_bw; + } + +return fair_server_bw; +} +static inline unsigned long __dl_bw_capacity (const struct cpumask *mask) {unsigned long cap =0 ;
@@ -1657,25 +1675,9 @@ void sched_init_dl_servers(void) } } -void __dl_server_attach_root (struct sched_dl_entity *dl_se,struct rq *rq) -{ -u64 new_bw = dl_se->dl_bw; -int cpu =cpu_of (rq); -struct dl_bw *dl_b; - - dl_b =dl_bw_of (cpu_of (rq)); -guard (raw_spinlock)(&dl_b->lock); - -if (!dl_bw_cpus (cpu)) -return ; - -__dl_add (dl_b, new_bw,dl_bw_cpus (cpu)); -} -int dl_server_apply_params (struct sched_dl_entity *dl_se,u64 runtime,u64 period,bool init) { -u64 old_bw = init ?0 :to_ratio (dl_se->dl_period, dl_se->dl_runtime); -u64 new_bw =to_ratio (period, runtime); +u64 max_bw, new_bw =to_ratio (period, runtime);struct rq *rq = dl_se->rq;int cpu =cpu_of (rq);struct dl_bw *dl_b;
@@ -1688,17 +1690,14 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio cpus =dl_bw_cpus (cpu); cap =dl_bw_capacity (cpu); + max_bw =div64_ul (cap_scale (BW_UNIT - dl_b->bw, cap), (unsigned long )cpus); -if (__dl_overflow (dl_b, cap, old_bw, new_bw)) +if (new_bw > max_bw)return -EBUSY ;if (init) {__add_rq_bw (new_bw, &rq->dl); -__dl_add (dl_b, new_bw, cpus); }else { -__dl_sub (dl_b, dl_se->dl_bw, cpus); -__dl_add (dl_b, new_bw, cpus); -dl_rq_change_utilization (rq, dl_se, new_bw); }
@@ -2939,17 +2938,6 @@ void dl_clear_root_domain(struct root_domain *rd) rd->dl_bw.total_bw =0 ;for_each_cpu (i, rd->span)cpu_rq (i)->dl.extra_bw =cpu_rq (i)->dl.max_bw; - -/* -* dl_servers are not tasks. Since dl_add_task_root_domain ignores -* them, we need to account for them here explicitly. -*/ -for_each_cpu (i, rd->span) { -struct sched_dl_entity *dl_se = &cpu_rq (i)->fair_server; - -if (dl_server (dl_se) &&cpu_active (i)) -__dl_add (&rd->dl_bw, dl_se->dl_bw,dl_bw_cpus (i)); - } }void dl_clear_root_domain_cpu (int cpu)
@@ -3133,9 +3121,10 @@ int sched_dl_global_validate(void)u64 period =global_rt_period ();u64 new_bw =to_ratio (period, runtime);u64 cookie = ++dl_cookie; +u64 fair_bw;struct dl_bw *dl_b; -int cpu, cpus, ret =0 ; -unsigned long flags; +int cpu, ret =0 ; +unsigned long cap, flags;/* * Here we want to check the bandwidth not being set to some
@@ -3149,10 +3138,13 @@ int sched_dl_global_validate(void) goto next; dl_b =dl_bw_of (cpu); - cpus =dl_bw_cpus (cpu); + cap =dl_bw_capacity (cpu); + fair_bw =dl_bw_fair (cpu);raw_spin_lock_irqsave (&dl_b->lock, flags); -if (new_bw * cpus < dl_b->total_bw) +if (cap_scale (new_bw, cap) < dl_b->total_bw) + ret = -EBUSY ; +if (cap_scale (new_bw, cap) + fair_bw >cap_scale (BW_UNIT , cap)) ret = -EBUSY ;raw_spin_unlock_irqrestore (&dl_b->lock, flags);
kernel/sched/sched.h +0 -1
@@ -390,7 +390,6 @@ extern void sched_init_dl_servers(void);extern void dl_server_update_idle_time (struct rq *rq,struct task_struct *p);extern void fair_server_init (struct rq *rq); -extern void __dl_server_attach_root (struct sched_dl_entity *dl_se,struct rq *rq);extern int dl_server_apply_params (struct sched_dl_entity *dl_se,u64 runtime,u64 period,bool init);
kernel/sched/topology.c +0 -8
@@ -500,14 +500,6 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)if (cpumask_test_cpu (rq->cpu, cpu_active_mask))set_rq_online (rq); -/* -* Because the rq is not a task, dl_add_task_root_domain() did not -* move the fair server bw to the rd if it already started. -* Add it now. -*/ -if (rq->fair_server.dl_server) -__dl_server_attach_root (&rq->fair_server, rq); -rq_unlock_irqrestore (rq, &rf);if (old_rd)
Signature · 1 line
2.50.1
RFC PATCH v2 2/25 sched/deadline: Do not access dl_se->rq directly
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make deadline.c code access the runqueue of a scheduling entity saved in the sched_dl_entity data structure. This allows future patches to save different runqueues in sched_dl_entity other than the global runqueues.
--- kernel/sched/deadline.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-)
kernel/sched/deadline.c +14 -13
@@ -892,7 +892,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)* and arm the defer timer. */ if (dl_se->dl_defer && !dl_se->dl_defer_running && -dl_time_before (rq_clock (dl_se->rq), dl_se->deadline - dl_se->runtime)) { +dl_time_before (rq_clock (rq), dl_se->deadline - dl_se->runtime)) {if (!is_dl_boosted (dl_se) && dl_se->server_has_tasks (dl_se)) {/*
@@ -1202,11 +1202,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_* of time. The dl_server_min_res serves as a limit to avoid * forwarding the timer for a too small amount of time. */ -if (dl_time_before (rq_clock (dl_se->rq), +if (dl_time_before (rq_clock (rq), (dl_se->deadline - dl_se->runtime - dl_server_min_res))) {/* reset the defer timer */ - fw = dl_se->deadline -rq_clock (dl_se->rq) - dl_se->runtime; + fw = dl_se->deadline -rq_clock (rq) - dl_se->runtime;hrtimer_forward_now (timer,ns_to_ktime (fw));return HRTIMER_RESTART ;
@@ -1217,7 +1217,7 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_enqueue_dl_entity (dl_se,ENQUEUE_REPLENISH ); -if (!dl_task (dl_se->rq->curr) ||dl_entity_preempt (dl_se, &dl_se->rq->curr->dl)) +if (!dl_task (rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl))resched_curr (rq);__push_dl_task (rq, rf);
@@ -1485,7 +1485,7 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64hrtimer_try_to_cancel (&dl_se->dl_timer); -replenish_dl_new_period (dl_se, dl_se->rq); +replenish_dl_new_period (dl_se, rq);/* * Not being able to start the timer seems problematic. If it could not
@@ -1597,21 +1597,22 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)/* 0 runtime = fair server disabled */ if (dl_se->dl_runtime) { dl_se->dl_server_idle =0 ; -update_curr_dl_se (dl_se->rq, dl_se, delta_exec); +update_curr_dl_se (rq_of_dl_se (dl_se), dl_se, delta_exec); } }void dl_server_start (struct sched_dl_entity *dl_se) { -struct rq *rq = dl_se->rq; +struct rq *rq;if (!dl_server (dl_se) || dl_se->dl_server_active)return ; dl_se->dl_server_active =1 ;enqueue_dl_entity (dl_se,ENQUEUE_WAKEUP ); -if (!dl_task (dl_se->rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl)) -resched_curr (dl_se->rq); + rq =rq_of_dl_se (dl_se); +if (!dl_task (rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl)) +resched_curr (rq); }void dl_server_stop (struct sched_dl_entity *dl_se)
@@ -1667,9 +1668,9 @@ void sched_init_dl_servers(void)WARN_ON (dl_server (dl_se)); -dl_server_apply_params (dl_se, runtime, period,1 ); - dl_se->dl_server =1 ; +BUG_ON (dl_server_apply_params (dl_se, runtime, period,1 )); + dl_se->dl_defer =1 ;setup_new_dl_entity (dl_se); }
@@ -1678,7 +1679,7 @@ void sched_init_dl_servers(void)int dl_server_apply_params (struct sched_dl_entity *dl_se,u64 runtime,u64 period,bool init) {u64 max_bw, new_bw =to_ratio (period, runtime); -struct rq *rq = dl_se->rq; +struct rq *rq =rq_of_dl_se (dl_se);int cpu =cpu_of (rq);struct dl_bw *dl_b;unsigned long cap;
@@ -1752,7 +1753,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) p =dl_task_of (dl_se); rq =task_rq_lock (p, &rf); }else { - rq = dl_se->rq; + rq =rq_of_dl_se (dl_se);rq_lock (rq, &rf); }
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 02/25] sched/deadline: Do not access dl_se->rq directly
Hi,
On 31/07/25 12:55, Yuri Andriaccio wrote:
...
kernel/sched/deadline.c void sched_init_dl_servers(void) · line 1672
Quoted 8 lines
kernel/sched/deadline.c +1
-2@@ -1667,9 +1668,9 @@ void sched_init_dl_servers(void)WARN_ON (dl_server (dl_se)); -dl_server_apply_params (dl_se, runtime, period,1 ); - dl_se->dl_server =1 ; +BUG_ON (dl_server_apply_params (dl_se, runtime, period,1 ));
A WARN_ON(), with possibly a recover strategy, is usually to prefer.
Thanks,
Juri
RFC PATCH v2 3/25 sched/deadline: Distinct between dl_rq and my_q
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Create two fields for runqueues in sched_dl_entity to make a distinction between the global runqueue and the runqueue which the dl_server serves.
--- include/linux/sched.h | 6 ++++-- kernel/sched/deadline.c | 11 +++++++---- kernel/sched/fair.c | 6 +++--- kernel/sched/sched.h | 3 ++- 4 files changed, 16 insertions(+), 10 deletions(-)
include/linux/sched.h +4 -2
@@ -724,12 +724,14 @@ struct sched_dl_entity {* Bits for DL-server functionality. Also see the comment near * dl_server_update(). * -* @rq the runqueue this server is for +* @dl_rq the runqueue on which this entity is (to be) queued +* @my_q the runqueue "owned" by this entity * * @server_has_tasks() returns true if @server_pick return a * runnable task. */ -struct rq *rq; +struct dl_rq *dl_rq; +struct rq *my_q;dl_server_has_tasks_f server_has_tasks;dl_server_pick_f server_pick_task;
kernel/sched/deadline.c +7 -4
@@ -75,11 +75,12 @@ static inline struct rq *rq_of_dl_rq(struct dl_rq *dl_rq)static inline struct rq *rq_of_dl_se (struct sched_dl_entity *dl_se) { -struct rq *rq = dl_se->rq; +struct rq *rq;if (!dl_server (dl_se)) rq =task_rq (dl_task_of (dl_se)); - +else + rq = container_of(dl_se->dl_rq, struct rq, dl);return rq; }
@@ -1641,11 +1642,13 @@ static bool dl_server_stopped(struct sched_dl_entity *dl_se)return false; } -void dl_server_init (struct sched_dl_entity *dl_se,struct rq *rq, +void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq, +struct rq *served_rq,dl_server_has_tasks_f has_tasks,dl_server_pick_f pick_task) { - dl_se->rq = rq; + dl_se->dl_rq = dl_rq; + dl_se->my_q = served_rq; dl_se->server_has_tasks = has_tasks; dl_se->server_pick_task = pick_task; }
kernel/sched/fair.c +3 -3
@@ -8861,12 +8861,12 @@ static struct task_struct *__pick_next_task_fair(struct rq *rq, struct task_strustatic bool fair_server_has_tasks (struct sched_dl_entity *dl_se) { -return !!dl_se->rq->cfs.nr_queued; +return !!dl_se->my_q->cfs.nr_queued; }static struct task_struct *fair_server_pick_task (struct sched_dl_entity *dl_se) { -return pick_task_fair (dl_se->rq); +return pick_task_fair (dl_se->my_q); }void fair_server_init (struct rq *rq)
@@ -8875,7 +8875,7 @@ void fair_server_init(struct rq *rq)init_dl_entity (dl_se); -dl_server_init (dl_se, rq, fair_server_has_tasks, fair_server_pick_task); +dl_server_init (dl_se, &rq->dl, rq, fair_server_has_tasks, fair_server_pick_task); }/*
kernel/sched/sched.h +2 -1
@@ -382,7 +382,8 @@ extern s64 dl_scaled_delta_exec(struct rq *rq, struct sched_dl_entity *dl_se, s6extern void dl_server_update (struct sched_dl_entity *dl_se,s64 delta_exec);extern void dl_server_start (struct sched_dl_entity *dl_se);extern void dl_server_stop (struct sched_dl_entity *dl_se); -extern void dl_server_init (struct sched_dl_entity *dl_se,struct rq *rq, +extern void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq, +struct rq *served_rq,dl_server_has_tasks_f has_tasks,dl_server_pick_f pick_task);extern void sched_init_dl_servers (void );
Signature · 1 line
2.50.1
RFC PATCH v2 4/25 sched/rt: Pass an rt_rq instead of an rq where needed
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make rt.c code access the runqueue through the rt_rq data structure rather than passing an rq pointer directly. This allows future patches to define rt_rq data structures which do not refer only to the global runqueue, but also to local cgroup runqueues (rt_rq is not always equal to &rq->rt).
--- kernel/sched/rt.c | 83 +++++++++++++++++++++++++---------------------- 1 file changed, 44 insertions(+), 39 deletions(-)
kernel/sched/rt.c +44 -39
@@ -370,9 +370,9 @@ static inline void rt_clear_overload(struct rq *rq)cpumask_clear_cpu (rq->cpu, rq->rd->rto_mask); } -static inline int has_pushable_tasks (struct rq *rq) +static inline int has_pushable_tasks (struct rt_rq *rt_rq) { -return !plist_head_empty (&rq->rt.pushable_tasks); +return !plist_head_empty (&rt_rq->pushable_tasks); } static DEFINE_PER_CPU(struct balance_callback, rt_push_head);
@@ -383,7 +383,7 @@ static void pull_rt_task(struct rq *);static inline void rt_queue_push_tasks (struct rq *rq) { -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (&rq->rt))return ;queue_balance_callback (rq, &per_cpu (rt_push_head, rq->cpu), push_rt_tasks);
@@ -394,37 +394,37 @@ static inline void rt_queue_pull_task(struct rq *rq)queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); } -static void enqueue_pushable_task (struct rq *rq,struct task_struct *p) +static void enqueue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) { -plist_del (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);plist_node_init (&p->pushable_tasks, p->prio); -plist_add (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_add (&p->pushable_tasks, &rt_rq->pushable_tasks);/* Update the highest prio pushable task */ -if (p->prio < rq->rt.highest_prio.next) - rq->rt.highest_prio.next = p->prio; +if (p->prio < rt_rq->highest_prio.next) + rt_rq->highest_prio.next = p->prio; -if (!rq->rt.overloaded) { -rt_set_overload (rq); - rq->rt.overloaded =1 ; +if (!rt_rq->overloaded) { +rt_set_overload (rq_of_rt_rq (rt_rq)); + rt_rq->overloaded =1 ; } } -static void dequeue_pushable_task (struct rq *rq,struct task_struct *p) +static void dequeue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) { -plist_del (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);/* Update the new highest prio pushable task */ -if (has_pushable_tasks (rq)) { - p =plist_first_entry (&rq->rt.pushable_tasks, +if (has_pushable_tasks (rt_rq)) { + p =plist_first_entry (&rt_rq->pushable_tasks, struct task_struct, pushable_tasks); - rq->rt.highest_prio.next = p->prio; + rt_rq->highest_prio.next = p->prio; }else { - rq->rt.highest_prio.next =MAX_RT_PRIO -1 ; + rt_rq->highest_prio.next =MAX_RT_PRIO -1 ; -if (rq->rt.overloaded) { -rt_clear_overload (rq); - rq->rt.overloaded =0 ; +if (rt_rq->overloaded) { +rt_clear_overload (rq_of_rt_rq (rt_rq)); + rt_rq->overloaded =0 ; } } }
@@ -1431,6 +1431,7 @@ static void enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags) { struct sched_rt_entity *rt_se = &p->rt; +struct rt_rq *rt_rq =rt_rq_of_se (rt_se);if (flags &ENQUEUE_WAKEUP ) rt_se->timeout =0 ;
@@ -1444,17 +1445,18 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)return ;if (!task_current (rq, p) && p->nr_cpus_allowed >1 ) -enqueue_pushable_task (rq, p); +enqueue_pushable_task (rt_rq, p); }static bool dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) {struct sched_rt_entity *rt_se = &p->rt; +struct rt_rq *rt_rq =rt_rq_of_se (rt_se);update_curr_rt (rq);dequeue_rt_entity (rt_se, flags); -dequeue_pushable_task (rq, p); +dequeue_pushable_task (rt_rq, p);return true; }
@@ -1639,14 +1641,14 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)static inline void set_next_task_rt (struct rq *rq,struct task_struct *p,bool first) {struct sched_rt_entity *rt_se = &p->rt; -struct rt_rq *rt_rq = &rq->rt; +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt); p->se.exec_start =rq_clock_task (rq);if (on_rt_rq (&p->rt))update_stats_wait_end_rt (rt_rq, rt_se);/* The running task is never eligible for pushing */ -dequeue_pushable_task (rq, p); +dequeue_pushable_task (rt_rq, p);if (!first)return ;
@@ -1710,7 +1712,7 @@ static struct task_struct *pick_task_rt(struct rq *rq)static void put_prev_task_rt (struct rq *rq,struct task_struct *p,struct task_struct *next) {struct sched_rt_entity *rt_se = &p->rt; -struct rt_rq *rt_rq = &rq->rt; +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt);if (on_rt_rq (&p->rt))update_stats_wait_start_rt (rt_rq, rt_se);
@@ -1726,7 +1728,7 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s* if it is still active */ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 ) -enqueue_pushable_task (rq, p); +enqueue_pushable_task (rt_rq, p); }/* Only try algorithms three times */
@@ -1736,16 +1738,16 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s* Return the highest pushable rq's task, which is suitable to be executed * on the CPU, NULL otherwise */ -static struct task_struct *pick_highest_pushable_task (struct rq *rq,int cpu) +static struct task_struct *pick_highest_pushable_task (struct rt_rq *rt_rq,int cpu) { -struct plist_head *head = &rq->rt.pushable_tasks; +struct plist_head *head = &rt_rq->pushable_tasks;struct task_struct *p; -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (rt_rq))return NULL ;plist_for_each_entry (p, head, pushable_tasks) { -if (task_is_pushable (rq, p, cpu)) +if (task_is_pushable (rq_of_rt_rq (rt_rq), p, cpu))return p; }
@@ -1845,14 +1847,15 @@ static int find_lowest_rq(struct task_struct *task)return -1 ; } -static struct task_struct *pick_next_pushable_task (struct rq *rq) +static struct task_struct *pick_next_pushable_task (struct rt_rq *rt_rq) { +struct rq *rq =rq_of_rt_rq (rt_rq);struct task_struct *p; -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (rt_rq))return NULL ; - p =plist_first_entry (&rq->rt.pushable_tasks, + p =plist_first_entry (&rt_rq->pushable_tasks, struct task_struct, pushable_tasks);BUG_ON (rq->cpu !=task_cpu (p));
@@ -1905,7 +1908,7 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)*/ if (unlikely (is_migration_disabled (task) || !cpumask_test_cpu (lowest_rq->cpu, &task->cpus_mask) || - task !=pick_next_pushable_task (rq))) { + task !=pick_next_pushable_task (&rq->rt))) {double_unlock_balance (rq, lowest_rq); lowest_rq =NULL ;
@@ -1939,7 +1942,7 @@ static int push_rt_task(struct rq *rq, bool pull)if (!rq->rt.overloaded)return 0 ; - next_task =pick_next_pushable_task (rq); + next_task =pick_next_pushable_task (&rq->rt);if (!next_task)return 0 ;
@@ -2014,7 +2017,7 @@ static int push_rt_task(struct rq *rq, bool pull)* run-queue and is also still the next task eligible for * pushing. */ - task =pick_next_pushable_task (rq); + task =pick_next_pushable_task (&rq->rt);if (task == next_task) {/* * The task hasn't migrated, and is still the next
@@ -2202,7 +2205,7 @@ void rto_push_irq_work_func(struct irq_work *work)* We do not need to grab the lock to check for has_pushable_tasks. * When it gets updated, a check is made if a push is possible. */ -if (has_pushable_tasks (rq)) { +if (has_pushable_tasks (&rq->rt)) {raw_spin_rq_lock (rq);while (push_rt_task (rq, true)) ;
@@ -2231,6 +2234,7 @@ static void pull_rt_task(struct rq *this_rq)int this_cpu = this_rq->cpu, cpu;bool resched = false;struct task_struct *p, *push_task; +struct rt_rq *src_rt_rq;struct rq *src_rq;int rt_overload_count =rt_overloaded (this_rq);
@@ -2260,6 +2264,7 @@ static void pull_rt_task(struct rq *this_rq)continue ; src_rq =cpu_rq (cpu); + src_rt_rq = &src_rq->rt;/* * Don't bother taking the src_rq->lock if the next highest
@@ -2268,7 +2273,7 @@ static void pull_rt_task(struct rq *this_rq)* logically higher, the src_rq will push this task away. * And if its going logically lower, we do not care */ -if (src_rq->rt.highest_prio.next >= +if (src_rt_rq->highest_prio.next >= this_rq->rt.highest_prio.curr)continue ;
@@ -2284,7 +2289,7 @@ static void pull_rt_task(struct rq *this_rq)* We can pull only a task, which is pushable * on its rq, and no others. */ - p =pick_highest_pushable_task (src_rq, this_cpu); + p =pick_highest_pushable_task (src_rt_rq, this_cpu);/* * Do we have an RT task that preempts
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 04/25] sched/rt: Pass an rt_rq instead of an rq where needed
Hi!
On 31/07/25 12:55, Yuri Andriaccio wrote:
...
kernel/sched/rt.c static void pull_rt_task(struct rq *); · line 384
Quoted 3 lines
kernel/sched/rt.c +0
-0@@ -383,7 +383,7 @@ static void pull_rt_task(struct rq *);static inline void rt_queue_push_tasks (struct rq *rq)
Can't the above also take an rt_rq?
kernel/sched/rt.c static void pull_rt_task(struct rq *); · line 389
Quoted 6 lines
kernel/sched/rt.c +1
-1{ -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (&rq->rt))return ;queue_balance_callback (rq, &per_cpu (rt_push_head, rq->cpu), push_rt_tasks);
Thanks,
Juri
RFC PATCH v2 5/25 sched/rt: Move some functions from rt.c to sched.h
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make the following functions be non-static and move them in sched.h, so that they can be used also in other source files:
- rt_task_of()
- rq_of_rt_rq()
- rt_rq_of_se()
- rq_of_rt_se()
There are no functional changes. This is needed by future patches.
--- kernel/sched/rt.c | 52 -------------------------------------------- kernel/sched/sched.h | 51 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 52 deletions(-)
kernel/sched/rt.c +0 -52
@@ -168,34 +168,6 @@ static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)#define rt_entity_is_task (rt_se) (!(rt_se)->my_q) -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ -WARN_ON_ONCE (!rt_entity_is_task (rt_se)); - - return container_of(rt_se, struct task_struct, rt); -} - -static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) -{ -/* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */ -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -} - -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) -{ -WARN_ON (!rt_group_sched_enabled () && rt_se->rt_rq->tg != &root_task_group); -return rt_se->rt_rq; -} - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct rt_rq *rt_rq = rt_se->rt_rq; - -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -} -void unregister_rt_sched_group (struct task_group *tg) {if (!rt_group_sched_enabled ())
@@ -296,30 +268,6 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)#define rt_entity_is_task (rt_se) (1) -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ - return container_of(rt_se, struct task_struct, rt); -} - -static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) -{ - return container_of(rt_rq, struct rq, rt); -} - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct task_struct *p =rt_task_of (rt_se); - -return task_rq (p); -} - -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) -{ -struct rq *rq =rq_of_rt_se (rt_se); - -return &rq->rt; -} -void unregister_rt_sched_group (struct task_group *tg) { }void free_rt_sched_group (struct task_group *tg) { }
kernel/sched/sched.h +51 -0
57 lines not on this page · raw message
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 05/25] sched/rt: Move some functions from rt.c to sched.h
Hi!
Quoted 43 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Make the following functions be non-static and move them in
sched.h, so that they can be used also in other source files:
-rt_task_of()
-rq_of_rt_rq()
-rt_rq_of_se()
-rq_of_rt_se()There are no functional changes. This is needed by future patches.
Signed-off-by luca abeni <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>--- kernel/sched/rt.c | 52 -------------------------------------------- kernel/sched/sched.h | 51 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 52 deletions(-)kernel/sched/rt.c +0
-17@@ -168,34 +168,6 @@ static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)#define rt_entity_is_task (rt_se) (!(rt_se)->my_q) -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ -WARN_ON_ONCE (!rt_entity_is_task (rt_se)); - - return container_of(rt_se, struct task_struct, rt); -} - -static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) -{ -/* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */ -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -} - -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) -{ -WARN_ON (!rt_group_sched_enabled () && rt_se->rt_rq->tg != &root_task_group);
kernel/sched/rt.c static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b) · line 197
Quoted 10 lines
kernel/sched/rt.c +0
-10-return rt_se->rt_rq; -} - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct rt_rq *rt_rq = rt_se->rt_rq; - -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -}
Looks like we are losing WARN_ON checks with the change? Why is that?
...
Quoted 5 lines
kernel/sched/sched.h +0
-0
kernel/sched/sched.h extern void set_rq_offline(struct rq *rq); · line 3050
Quoted 9 lines
kernel/sched/sched.h +6
-0@@ -3042,6 +3042,57 @@ extern void set_rq_offline(struct rq *rq);extern bool sched_smp_initialized; +#ifdef CONFIG_RT_GROUP_SCHED +static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) +{ +#ifdef CONFIG_SCHED_DEBUG +WARN_ON_ONCE (rt_se->my_q); +#endif
And gaining this one above. :)
Thanks,
Juri
RFC PATCH v2 6/25 sched/rt: Disable RT_GROUP_SCHED
In reply to Yuri AndriaccioDisable the old RT_GROUP_SCHED scheduler. Note that this does not completely remove all the RT_GROUP_SCHED functionality, just unhooks it and removes most of the relevant functions. Some of the RT_GROUP_SCHED functions are kept because they will be adapted for the HCBS scheduling.
--- kernel/sched/core.c | 6 - kernel/sched/deadline.c | 34 -- kernel/sched/debug.c | 6 - kernel/sched/rt.c | 848 ++-------------------------------------- kernel/sched/sched.h | 9 +- kernel/sched/syscalls.c | 13 - 6 files changed, 26 insertions(+), 890 deletions(-)
kernel/sched/core.c +0 -6
@@ -8714,11 +8714,6 @@ void __init sched_init(void)init_defrootdomain (); -#ifdef CONFIG_RT_GROUP_SCHED -init_rt_bandwidth (&root_task_group.rt_bandwidth, -global_rt_period (),global_rt_runtime ()); -#endif /* CONFIG_RT_GROUP_SCHED */ -#ifdef CONFIG_CGROUP_SCHED task_group_cache =KMEM_CACHE (task_group,0 );
@@ -8770,7 +8765,6 @@ void __init sched_init(void)* starts working after scheduler_running, which is not the case * yet. */ - rq->rt.rt_runtime =global_rt_runtime ();init_tg_rt_entry (&root_task_group, &rq->rt,NULL , i,NULL );#endif rq->sd =NULL ;
kernel/sched/deadline.c +0 -34
@@ -1524,40 +1524,6 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64if (!is_leftmost (dl_se, &rq->dl))resched_curr (rq); } - -/* -* The fair server (sole dl_server) does not account for real-time -* workload because it is running fair work. -*/ -if (dl_se == &rq->fair_server) -return ; - -#ifdef CONFIG_RT_GROUP_SCHED -/* -* Because -- for now -- we share the rt bandwidth, we need to -* account our runtime there too, otherwise actual rt tasks -* would be able to exceed the shared quota. -* -* Account to the root rt group for now. -* -* The solution we're working towards is having the RT groups scheduled -* using deadline servers -- however there's a few nasties to figure -* out before that can happen. -*/ -if (rt_bandwidth_enabled ()) { -struct rt_rq *rt_rq = &rq->rt; - -raw_spin_lock (&rt_rq->rt_runtime_lock); -/* -* We'll let actual RT tasks worry about the overflow here, we -* have our own CBS to keep us inline; only account when RT -* bandwidth is relevant. -*/ -if (sched_rt_bandwidth_account (rt_rq)) - rt_rq->rt_time += delta_exec; -raw_spin_unlock (&rt_rq->rt_runtime_lock); - } -#endif /* CONFIG_RT_GROUP_SCHED */ }/*
kernel/sched/debug.c +0 -6
@@ -892,12 +892,6 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)PU (rt_nr_running); -#ifdef CONFIG_RT_GROUP_SCHED -P (rt_throttled); -PN (rt_time); -PN (rt_runtime); -#endif -#undef PN#undef PU#undef P
kernel/sched/rt.c +23 -825
1,073 lines not on this page · raw message
kernel/sched/sched.h +3 -6
@@ -789,7 +789,7 @@ struct scx_rq {static inline int rt_bandwidth_enabled (void ) { -return sysctl_sched_rt_runtime >=0 ; +return 0 ; }/* RT IPI pull logic requires IRQ_WORK */
@@ -829,7 +829,7 @@ struct rt_rq {static inline bool rt_rq_is_runnable (struct rt_rq *rt_rq) { -return rt_rq->rt_queued && rt_rq->rt_nr_running; +return rt_rq->rt_nr_running; }/* Deadline class' related fields in a runqueue */
@@ -2545,7 +2545,7 @@ static inline bool sched_dl_runnable(struct rq *rq)static inline bool sched_rt_runnable (struct rq *rq) { -return rq->rt.rt_queued >0 ; +return rq->rt.rt_nr_running >0 ; }static inline bool sched_fair_runnable (struct rq *rq)
@@ -2656,9 +2656,6 @@ extern void resched_curr(struct rq *rq);extern void resched_curr_lazy (struct rq *rq);extern void resched_cpu (int cpu); -extern void init_rt_bandwidth (struct rt_bandwidth *rt_b,u64 period,u64 runtime); -extern bool sched_rt_bandwidth_account (struct rt_rq *rt_rq); -extern void init_dl_entity (struct sched_dl_entity *dl_se);#define BW_SHIFT 20
kernel/sched/syscalls.c +0 -13
@@ -626,19 +626,6 @@ int __sched_setscheduler(struct task_struct *p, change:if (user) { -#ifdef CONFIG_RT_GROUP_SCHED -/* -* Do not allow real-time tasks into groups that have no runtime -* assigned. -*/ -if (rt_group_sched_enabled () && -rt_bandwidth_enabled () &&rt_policy (policy) && -task_group (p)->rt_bandwidth.rt_runtime ==0 && - !task_group_is_autogroup (task_group (p))) { - retval = -EPERM ; - goto unlock; - } -#endif /* CONFIG_RT_GROUP_SCHED */ if (dl_bandwidth_enabled () &&dl_policy (policy) && !(attr->sched_flags &SCHED_FLAG_SUGOV )) {cpumask_t *span = rq->rd->span;
Signature · 1 line
2.50.1
RFC PATCH v2 7/25 sched/rt: Introduce HCBS specific structs in task_group
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Each task_group manages a number of new objects:
- a sched_dl_entity/dl_server for each CPU
- a dl_bandwidth object to keep track of its allocated dl_bandwidth
--- kernel/sched/sched.h | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
kernel/sched/sched.h +15 -2
@@ -319,6 +319,13 @@ struct rt_bandwidth {unsigned int rt_period_active; }; +struct dl_bandwidth { +raw_spinlock_t dl_runtime_lock; +u64 dl_runtime; +u64 dl_period; +}; + +static inline int dl_bandwidth_enabled (void ) {return sysctl_sched_rt_runtime >=0 ;
@@ -467,9 +474,15 @@ struct task_group {#ifdef CONFIG_RT_GROUP_SCHED struct sched_rt_entity **rt_se; +/* +* The scheduling entities for the task group are managed as a single +* sched_dl_entity, each of them sharing the same dl_bandwidth. +*/ +struct sched_dl_entity **dl_se;struct rt_rq **rt_rq;struct rt_bandwidth rt_bandwidth; +struct dl_bandwidth dl_bandwidth;#endif #ifdef CONFIG_EXT_GROUP_SCHED
@@ -819,12 +832,12 @@ struct rt_rq {raw_spinlock_t rt_runtime_lock;unsigned int rt_nr_boosted; - -struct rq *rq;/* this is always top-level rq, cache? */ #endif #ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */ #endif + +struct rq *rq;/* cgroup's runqueue if the rt_rq entity belongs to a cgroup, otherwise top-level rq */ };static inline bool rt_rq_is_runnable (struct rt_rq *rt_rq)
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 07/25] sched/rt: Introduce HCBS specific structs in task_group
Hi!
On 31/07/25 12:55, Yuri Andriaccio wrote:
...
kernel/sched/sched.h struct task_group { · line 481
Quoted 9 lines
kernel/sched/sched.h +5
-0@@ -467,9 +474,15 @@ struct task_group {#ifdef CONFIG_RT_GROUP_SCHED struct sched_rt_entity **rt_se; +/* +* The scheduling entities for the task group are managed as a single +* sched_dl_entity, each of them sharing the same dl_bandwidth. +*/ +struct sched_dl_entity **dl_se;
This is actually one sched_dl_entity per CPU, right? If that is the case the comment is a little misleading I am afraid (mentioning single entitiy).
Thanks,
Juri
RFC PATCH v2 8/25 sched/deadline: Account rt-cgroups bandwidth in deadline tasks schedulability tests.
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Account the rt-cgroups hierarchy's reserved bandwidth in the schedulability test of deadline entities. This mechanism allows to completely reserve portion of the rt-bandwidth to rt-cgroups even if they do not use all of it.
--- kernel/sched/deadline.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/deadline.c +8 -1
@@ -231,8 +231,15 @@ void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)static inline bool __dl_overflow (struct dl_bw *dl_b,unsigned long cap,u64 old_bw,u64 new_bw) { +u64 dl_groups_root =0 ; + +#ifdef CONFIG_RT_GROUP_SCHED + dl_groups_root =to_ratio (root_task_group.dl_bandwidth.dl_period, + root_task_group.dl_bandwidth.dl_runtime); +#endif return dl_b->bw !=-1 && -cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw; +cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw + +cap_scale (dl_groups_root, cap); } static inline
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 08/25] sched/deadline: Account rt-cgroups bandwidth in deadline tasks schedulability tests.
Hi!
Quoted 6 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Account the rt-cgroups hierarchy's reserved bandwidth in the schedulability test of deadline entities. This mechanism allows to completely reserve portion of the rt-bandwidth to rt-cgroups even if they do not use all of it.
Quoted 8 lines
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>---
I wonder if, from a series flow perspective, would make sense to move this one and the following after the changes that deal with groups initialization. Also, please consider a little bit of squashing.
Thanks,
Juri
RFC PATCH v2 9/25 sched/deadline: Account rt-cgroups bandwidth in sched_dl_global_validate
In reply to Yuri Andriacciosched_dl_global_validate (similarly to the previous patch) must take into account the rt-cgroups' reserved bandwidth.
--- kernel/sched/deadline.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/sched/deadline.c +7 -1
@@ -3098,11 +3098,17 @@ int sched_dl_global_validate(void)u64 period =global_rt_period ();u64 new_bw =to_ratio (period, runtime);u64 cookie = ++dl_cookie; +u64 dl_groups_root =0 ;u64 fair_bw;struct dl_bw *dl_b;int cpu, ret =0 ;unsigned long cap, flags; +#ifdef CONFIG_RT_GROUP_SCHED + dl_groups_root =to_ratio (root_task_group.dl_bandwidth.dl_period, + root_task_group.dl_bandwidth.dl_runtime); +#endif +/* * Here we want to check the bandwidth not being set to some * value smaller than the currently allocated bandwidth in
@@ -3119,7 +3125,7 @@ int sched_dl_global_validate(void) fair_bw =dl_bw_fair (cpu);raw_spin_lock_irqsave (&dl_b->lock, flags); -if (cap_scale (new_bw, cap) < dl_b->total_bw) +if (cap_scale (new_bw, cap) < dl_b->total_bw +cap_scale (dl_groups_root, cap)) ret = -EBUSY ;if (cap_scale (new_bw, cap) + fair_bw >cap_scale (BW_UNIT , cap)) ret = -EBUSY ;
Signature · 1 line
2.50.1
RFC PATCH v2 10/25 sched/core: Initialize root_task_group
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add the initialization function for task_group's dl_servers. Initialize the default bandwidth for rt-cgroups and the root control group. Add utility functions to check (and get) if a rt_rq entity is connected to a real-time cgroup.
--- kernel/sched/autogroup.c | 4 ++-- kernel/sched/core.c | 9 +++++++-- kernel/sched/deadline.c | 8 ++++++++ kernel/sched/rt.c | 18 ++++++++---------- kernel/sched/sched.h | 32 +++++++++++++++++++++++++++++--- 5 files changed, 54 insertions(+), 17 deletions(-)
kernel/sched/autogroup.c +2 -2
@@ -52,7 +52,7 @@ static inline void autogroup_destroy(struct kref *kref)#ifdef CONFIG_RT_GROUP_SCHED /* We've redirected RT tasks to the root task group... */ - ag->tg->rt_se =NULL ; + ag->tg->dl_se =NULL ; ag->tg->rt_rq =NULL ;#endif sched_release_group (ag->tg);
@@ -109,7 +109,7 @@ static inline struct autogroup *autogroup_create(void)* the policy change to proceed. */ free_rt_sched_group (tg); - tg->rt_se = root_task_group.rt_se; + tg->dl_se = root_task_group.dl_se; tg->rt_rq = root_task_group.rt_rq;#endif /* CONFIG_RT_GROUP_SCHED */ tg->autogroup = ag;
kernel/sched/core.c +7 -2
@@ -8703,7 +8703,7 @@ void __init sched_init(void)scx_tg_init (&root_task_group);#endif /* CONFIG_EXT_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED - root_task_group.rt_se = (struct sched_rt_entity **)ptr; + root_task_group.dl_se = (struct sched_dl_entity **)ptr; ptr += nr_cpu_ids *sizeof (void **); root_task_group.rt_rq = (struct rt_rq **)ptr;
@@ -8714,6 +8714,11 @@ void __init sched_init(void)init_defrootdomain (); +#ifdef CONFIG_RT_GROUP_SCHED +init_dl_bandwidth (&root_task_group.dl_bandwidth, +global_rt_period (),global_rt_runtime ()); +#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_CGROUP_SCHED task_group_cache =KMEM_CACHE (task_group,0 );
@@ -8765,7 +8770,7 @@ void __init sched_init(void)* starts working after scheduler_running, which is not the case * yet. */ -init_tg_rt_entry (&root_task_group, &rq->rt,NULL , i,NULL ); +init_tg_rt_entry (&root_task_group, rq,NULL , i,NULL );#endif rq->sd =NULL ; rq->rd =NULL ;
kernel/sched/deadline.c +8 -0
@@ -538,6 +538,14 @@ static inline int is_leftmost(struct sched_dl_entity *dl_se, struct dl_rq *dl_rqstatic void init_dl_rq_bw_ratio (struct dl_rq *dl_rq); +void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime) +{ +raw_spin_lock_init (&dl_b->dl_runtime_lock); + dl_b->dl_period = period; + dl_b->dl_runtime = runtime; +} + +void init_dl_bw (struct dl_bw *dl_b) {raw_spin_lock_init (&dl_b->lock);
kernel/sched/rt.c +8 -10
@@ -97,18 +97,16 @@ void free_rt_sched_group(struct task_group *tg)return ; } -void init_tg_rt_entry (struct task_group *tg,struct rt_rq *rt_rq, -struct sched_rt_entity *rt_se,int cpu, -struct sched_rt_entity *parent) +void init_tg_rt_entry (struct task_group *tg,struct rq *served_rq, +struct sched_dl_entity *dl_se,int cpu, +struct sched_dl_entity *parent) { -struct rq *rq =cpu_rq (cpu); + served_rq->rt.highest_prio.curr =MAX_RT_PRIO -1 ; + served_rq->rt.rq =cpu_rq (cpu); + served_rq->rt.tg = tg; - rt_rq->highest_prio.curr =MAX_RT_PRIO -1 ; - rt_rq->rq = rq; - rt_rq->tg = tg; - - tg->rt_rq[cpu] = rt_rq; - tg->rt_se[cpu] = rt_se; + tg->rt_rq[cpu] = &served_rq->rt; + tg->dl_se[cpu] = dl_se; }int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent)
kernel/sched/sched.h +29 -3
@@ -577,9 +577,9 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);extern void unthrottle_cfs_rq (struct cfs_rq *cfs_rq);extern bool cfs_task_bw_constrained (struct task_struct *p); -extern void init_tg_rt_entry (struct task_group *tg,struct rt_rq *rt_rq, -struct sched_rt_entity *rt_se,int cpu, -struct sched_rt_entity *parent); +extern void init_tg_rt_entry (struct task_group *tg,struct rq *s_rq, +struct sched_dl_entity *rt_se,int cpu, +struct sched_dl_entity *parent);extern int sched_group_set_rt_runtime (struct task_group *tg,long rt_runtime_us);extern int sched_group_set_rt_period (struct task_group *tg,u64 rt_period_us);extern long sched_group_rt_runtime (struct task_group *tg);
@@ -2669,6 +2669,7 @@ extern void resched_curr(struct rq *rq);extern void resched_curr_lazy (struct rq *rq);extern void resched_cpu (int cpu); +void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime);extern void init_dl_entity (struct sched_dl_entity *dl_se);#define BW_SHIFT 20
@@ -3077,6 +3078,21 @@ static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se) return rt_rq->rq; } + +static inline int is_dl_group (struct rt_rq *rt_rq) +{ +return rt_rq->tg != &root_task_group; +} + +/* +* Return the scheduling entity of this group of tasks. +*/ +static inline struct sched_dl_entity *dl_group_of (struct rt_rq *rt_rq) +{ +BUG_ON (!is_dl_group (rt_rq)); + +return rt_rq->tg->dl_se[cpu_of (rt_rq->rq)]; +}#else static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) {
@@ -3101,6 +3117,16 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)return &rq->rt; } + +static inline int is_dl_group (struct rt_rq *rt_rq) +{ +return 0 ; +} + +static inline struct sched_dl_entity *dl_group_of (struct rt_rq *rt_rq) +{ +return NULL ; +}#endif DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq,
Signature · 1 line
2.50.1
RFC PATCH v2 11/25 sched/deadline: Add dl_init_tg
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
This function is used to initialize and/or update a rt-cgroup dl_server, also accounting for the allocated bandwidth.
--- kernel/sched/deadline.c | 33 +++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 2 files changed, 34 insertions(+)
kernel/sched/deadline.c +33 -0
@@ -365,6 +365,39 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)cancel_dl_timer (dl_se, &dl_se->inactive_timer); } +#ifdef CONFIG_RT_GROUP_SCHED +void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period) +{ + struct rq *rq = container_of(dl_se->dl_rq, struct rq, dl); +int is_active; +u64 new_bw; + +raw_spin_rq_lock_irq (rq); + is_active = dl_se->my_q->rt.rt_nr_running >0 ; + +update_rq_clock (rq); +dl_server_stop (dl_se); + + new_bw =to_ratio (dl_se->dl_period, dl_se->dl_runtime); +dl_rq_change_utilization (rq, dl_se, new_bw); + + dl_se->dl_runtime = rt_runtime; + dl_se->dl_deadline = rt_period; + dl_se->dl_period = rt_period; + + dl_se->runtime =0 ; + dl_se->deadline =0 ; + + dl_se->dl_bw = new_bw; + dl_se->dl_density = new_bw; + +if (is_active) +dl_server_start (dl_se); + +raw_spin_rq_unlock_irq (rq); +} +#endif +static void dl_change_utilization (struct task_struct *p,u64 new_bw) {WARN_ON_ONCE (p->dl.flags &SCHED_FLAG_SUGOV );
kernel/sched/sched.h +1 -0
@@ -394,6 +394,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,dl_server_has_tasks_f has_tasks, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void ); +extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period);extern void dl_server_update_idle_time (struct rq *rq,struct task_struct *p);
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 11/25] sched/deadline: Add dl_init_tg
Hi!
Quoted 4 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
This function is used to initialize and/or update a rt-cgroup
dl_server, also accounting for the allocated bandwidth.
This function/this patch are usually frowned up [1].
Thanks,
Juri
1 - https://elixir.bootlin.com/linux/v6.16/source/Documentation/process/submitting-patches.rst#L94
RFC PATCH v2 12/25 sched/rt: Add {alloc/free}_rt_sched_group and dl_server specific functions
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add allocation and deallocation code for rt-cgroups. Add rt dl_server's specific functions that pick the next eligible task to run.
--- kernel/sched/rt.c | 107 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 104 insertions(+), 3 deletions(-)
kernel/sched/rt.c +104 -3
@@ -93,8 +93,39 @@ void unregister_rt_sched_group(struct task_group *tg)void free_rt_sched_group (struct task_group *tg) { +int i; +if (!rt_group_sched_enabled ())return ; + +for_each_possible_cpu (i) { +if (tg->dl_se) { +unsigned long flags; + +/* +* Since the dl timer is going to be cancelled, +* we risk to never decrease the running bw... +* Fix this issue by changing the group runtime +* to 0 immediately before freeing it. +*/ +dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); +raw_spin_rq_lock_irqsave (cpu_rq (i), flags); +BUG_ON (tg->rt_rq[i]->rt_nr_running); +raw_spin_rq_unlock_irqrestore (cpu_rq (i), flags); + +hrtimer_cancel (&tg->dl_se[i]->dl_timer); +kfree (tg->dl_se[i]); + } +if (tg->rt_rq) { +struct rq *served_rq; + + served_rq = container_of(tg->rt_rq[i], struct rq, rt); +kfree (served_rq); + } + } + +kfree (tg->rt_rq); +kfree (tg->dl_se); } void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq,
@@ -109,12 +140,77 @@ void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, tg->dl_se[cpu] = dl_se; } +static bool rt_server_has_tasks (struct sched_dl_entity *dl_se) +{ +return !!dl_se->my_q->rt.rt_nr_running; +} + +static struct task_struct *_pick_next_task_rt (struct rt_rq *rt_rq); +static inline void set_next_task_rt (struct rq *rq,struct task_struct *p,bool first); +static struct task_struct *rt_server_pick (struct sched_dl_entity *dl_se) +{ +struct rt_rq *rt_rq = &dl_se->my_q->rt; +struct rq *rq =rq_of_rt_rq (rt_rq); +struct task_struct *p; + +if (dl_se->my_q->rt.rt_nr_running ==0 ) +return NULL ; + + p =_pick_next_task_rt (rt_rq); +set_next_task_rt (rq, p, true); + +return p; +} +int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent) { +struct rq *s_rq; +struct sched_dl_entity *dl_se; +int i; +if (!rt_group_sched_enabled ())return 1 ; + tg->rt_rq =kcalloc (nr_cpu_ids,sizeof (struct rt_rq *),GFP_KERNEL ); +if (!tg->rt_rq) + goto err; + tg->dl_se =kcalloc (nr_cpu_ids,sizeof (dl_se),GFP_KERNEL ); +if (!tg->dl_se) + goto err; + +init_dl_bandwidth (&tg->dl_bandwidth,0 ,0 ); + +for_each_possible_cpu (i) { + s_rq =kzalloc_node (sizeof (struct rq ), +GFP_KERNEL ,cpu_to_node (i)); +if (!s_rq) + goto err; + + dl_se =kzalloc_node (sizeof (struct sched_dl_entity ), +GFP_KERNEL ,cpu_to_node (i)); +if (!dl_se) + goto err_free_rq; + +init_rt_rq (&s_rq->rt); +init_dl_entity (dl_se); + dl_se->dl_runtime = tg->dl_bandwidth.dl_runtime; + dl_se->dl_period = tg->dl_bandwidth.dl_period; + dl_se->dl_deadline = dl_se->dl_period; + dl_se->dl_bw =to_ratio (dl_se->dl_period, dl_se->dl_runtime); + dl_se->dl_density =to_ratio (dl_se->dl_period, dl_se->dl_runtime); + dl_se->dl_server =1 ; + +dl_server_init (dl_se, &cpu_rq (i)->dl, s_rq, rt_server_has_tasks, rt_server_pick); + +init_tg_rt_entry (tg, s_rq, dl_se, i, parent->dl_se[i]); + } +return 1 ; + +err_free_rq: +kfree (s_rq); +err: +return 0 ; }#else /* !CONFIG_RT_GROUP_SCHED: */
@@ -860,9 +956,14 @@ static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq)return next; } -static struct task_struct *_pick_next_task_rt (struct rq *rq) +static struct task_struct *_pick_next_task_rt (struct rt_rq *rt_rq) { -return NULL ; +struct sched_rt_entity *rt_se; + + rt_se =pick_next_rt_entity (rt_rq); +BUG_ON (!rt_se); + +return rt_task_of (rt_se); }static struct task_struct *pick_task_rt (struct rq *rq)
@@ -872,7 +973,7 @@ static struct task_struct *pick_task_rt(struct rq *rq)if (!sched_rt_runnable (rq))return NULL ; - p =_pick_next_task_rt (rq); + p =_pick_next_task_rt (&rq->rt);return p; }
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 12/25] sched/rt: Add {alloc/free}_rt_sched_group and dl_server specific functions
Hi!
Quoted 33 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Add allocation and deallocation code for rt-cgroups. Add rt dl_server's specific functions that pick the next eligible task to run.
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>--- kernel/sched/rt.c | 107 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 104 insertions(+), 3 deletions(-)kernel/sched/rt.c +7
-0@@ -93,8 +93,39 @@ void unregister_rt_sched_group(struct task_group *tg)void free_rt_sched_group (struct task_group *tg) { +int i; +if (!rt_group_sched_enabled ())return ; + +for_each_possible_cpu (i) { +if (tg->dl_se) { +unsigned long flags; +
kernel/sched/rt.c void unregister_rt_sched_group(struct task_group *tg) · line 114
Quoted 10 lines
kernel/sched/rt.c +10
-0+/* +* Since the dl timer is going to be cancelled, +* we risk to never decrease the running bw... +* Fix this issue by changing the group runtime +* to 0 immediately before freeing it. +*/ +dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); +raw_spin_rq_lock_irqsave (cpu_rq (i), flags); +BUG_ON (tg->rt_rq[i]->rt_nr_running); +raw_spin_rq_unlock_irqrestore (cpu_rq (i), flags);
So here we always call dl_init_tg for cpu 0, is it correct?
Also I wonder if the lock shouldn't cover both init and subsequent check.
Quoted 48 lines
kernel/sched/rt.c +39
-0+ +hrtimer_cancel (&tg->dl_se[i]->dl_timer); +kfree (tg->dl_se[i]); + } +if (tg->rt_rq) { +struct rq *served_rq; + + served_rq = container_of(tg->rt_rq[i], struct rq, rt); +kfree (served_rq); + } + } + +kfree (tg->rt_rq); +kfree (tg->dl_se); } void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq,@@ -109,12 +140,77 @@ void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, tg->dl_se[cpu] = dl_se; } +static bool rt_server_has_tasks (struct sched_dl_entity *dl_se) +{ +return !!dl_se->my_q->rt.rt_nr_running; +} + +static struct task_struct *_pick_next_task_rt (struct rt_rq *rt_rq); +static inline void set_next_task_rt (struct rq *rq,struct task_struct *p,bool first); +static struct task_struct *rt_server_pick (struct sched_dl_entity *dl_se) +{ +struct rt_rq *rt_rq = &dl_se->my_q->rt; +struct rq *rq =rq_of_rt_rq (rt_rq); +struct task_struct *p; + +if (dl_se->my_q->rt.rt_nr_running ==0 ) +return NULL ; + + p =_pick_next_task_rt (rt_rq); +set_next_task_rt (rq, p, true); + +return p; +} +int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent) { +struct rq *s_rq; +struct sched_dl_entity *dl_se; +int i;
kernel/sched/rt.c void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, · line 179
Quoted 10 lines
kernel/sched/rt.c +7
-0+if (!rt_group_sched_enabled ())return 1 ; + tg->rt_rq =kcalloc (nr_cpu_ids,sizeof (struct rt_rq *),GFP_KERNEL ); +if (!tg->rt_rq) + goto err; + tg->dl_se =kcalloc (nr_cpu_ids,sizeof (dl_se),GFP_KERNEL ); +if (!tg->dl_se) + goto err;
Don't we need to free the array allocated above if we fail here?
Thanks,
Juri
RFC PATCH v2 13/25 sched/rt: Add HCBS related checks and operations for rt tasks
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add checks wheter a task belongs to the root cgroup or a rt-cgroup, since HCBS reuses the rt classes' scheduler, and operate accordingly where needed.
--- kernel/sched/core.c | 3 + kernel/sched/deadline.c | 16 ++++- kernel/sched/rt.c | 147 +++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 6 +- kernel/sched/syscalls.c | 13 ++++ 5 files changed, 171 insertions(+), 14 deletions(-)
kernel/sched/core.c +3 -0
@@ -2196,6 +2196,9 @@ void wakeup_preempt(struct rq *rq, struct task_struct *p, int flags) {struct task_struct *donor = rq->donor; +if (is_dl_group (rt_rq_of_se (&p->rt)) &&task_has_rt_policy (p)) +resched_curr (rq); +if (p->sched_class == donor->sched_class) donor->sched_class->wakeup_preempt (rq, p, flags);else if (sched_class_above (p->sched_class, donor->sched_class))
kernel/sched/deadline.c +14 -2
@@ -1866,7 +1866,13 @@ void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)u64 deadline = dl_se->deadline; dl_rq->dl_nr_running++; -add_nr_running (rq_of_dl_rq (dl_rq),1 ); +if (!dl_server (dl_se) || dl_se == &rq_of_dl_rq (dl_rq)->fair_server) { +add_nr_running (rq_of_dl_rq (dl_rq),1 ); + }else { +struct rt_rq *rt_rq = &dl_se->my_q->rt; + +add_nr_running (rq_of_dl_rq (dl_rq), rt_rq->rt_nr_running); + }inc_dl_deadline (dl_rq, deadline); }
@@ -1876,7 +1882,13 @@ void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq) {WARN_ON (!dl_rq->dl_nr_running); dl_rq->dl_nr_running--; -sub_nr_running (rq_of_dl_rq (dl_rq),1 ); +if (!dl_server (dl_se) || dl_se == &rq_of_dl_rq (dl_rq)->fair_server) { +sub_nr_running (rq_of_dl_rq (dl_rq),1 ); + }else { +struct rt_rq *rt_rq = &dl_se->my_q->rt; + +sub_nr_running (rq_of_dl_rq (dl_rq), rt_rq->rt_nr_running); + }dec_dl_deadline (dl_rq, dl_se->deadline); }
kernel/sched/rt.c +139 -8
281 lines not on this page · raw message
kernel/sched/sched.h +2 -4
@@ -2171,7 +2171,7 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)if (!rt_group_sched_enabled ()) tg = &root_task_group; p->rt.rt_rq = tg->rt_rq[cpu]; - p->rt.parent = tg->rt_se[cpu]; + p->dl.dl_rq = &cpu_rq (cpu)->dl;#endif /* CONFIG_RT_GROUP_SCHED */ }
@@ -2727,6 +2727,7 @@ static inline void add_nr_running(struct rq *rq, unsigned count)static inline void sub_nr_running (struct rq *rq,unsigned count) { +BUG_ON (rq->nr_running < count); rq->nr_running -= count;if (trace_sched_update_nr_running_tp_enabled ()) {call_trace_sched_update_nr_running (rq, -count);
@@ -3057,9 +3058,6 @@ extern bool sched_smp_initialized;#ifdef CONFIG_RT_GROUP_SCHED static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) { -#ifdef CONFIG_SCHED_DEBUG -WARN_ON_ONCE (rt_se->my_q); -#endif return container_of(rt_se, struct task_struct, rt); }
kernel/sched/syscalls.c +13 -0
@@ -626,6 +626,19 @@ int __sched_setscheduler(struct task_struct *p, change:if (user) { +#ifdef CONFIG_RT_GROUP_SCHED +/* +* Do not allow real-time tasks into groups that have no runtime +* assigned. +*/ +if (rt_group_sched_enabled () && +dl_bandwidth_enabled () &&rt_policy (policy) && +task_group (p)->dl_bandwidth.dl_runtime ==0 && + !task_group_is_autogroup (task_group (p))) { + retval = -EPERM ; + goto unlock; + } +#endif if (dl_bandwidth_enabled () &&dl_policy (policy) && !(attr->sched_flags &SCHED_FLAG_SUGOV )) {cpumask_t *span = rq->rd->span;
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 13/25] sched/rt: Add HCBS related checks and operations for rt tasks
Hi!
Quoted 25 lines · Yuri Andriaccio
From: luca abeni <[email protected]>
Add checks wheter a task belongs to the root cgroup or a rt-cgroup, since HCBS reuses the rt classes' scheduler, and operate accordingly where needed.
Co-developed-by Alessio Balsini <[email protected]>Signed-off-by Alessio Balsini <[email protected]>Co-developed-by Andrea Parri <[email protected]>Signed-off-by Andrea Parri <[email protected]>Co-developed-by Yuri Andriaccio <[email protected]>Signed-off-by Yuri Andriaccio <[email protected]>Signed-off-by luca abeni <[email protected]>--- kernel/sched/core.c | 3 + kernel/sched/deadline.c | 16 ++++- kernel/sched/rt.c | 147 +++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 6 +- kernel/sched/syscalls.c | 13 ++++ 5 files changed, 171 insertions(+), 14 deletions(-)kernel/sched/core.c +0
-0
kernel/sched/core.c void wakeup_preempt(struct rq *rq, struct task_struct *p, int flags) · line 2200
Quoted 5 lines
kernel/sched/core.c +2
-0@@ -2196,6 +2196,9 @@ void wakeup_preempt(struct rq *rq, struct task_struct *p, int flags) {struct task_struct *donor = rq->donor; +if (is_dl_group (rt_rq_of_se (&p->rt)) &&task_has_rt_policy (p)) +resched_curr (rq);
Why this unconditional resched for tasks in groups?
...
kernel/sched/core.c enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags) · line 755
Quoted 13 lines
kernel/sched/core.c +8
-0@@ -715,6 +744,14 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)check_schedstat_required ();update_stats_wait_start_rt (rt_rq_of_se (rt_se), rt_se); +/* Task arriving in an idle group of tasks. */ +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) && +is_dl_group (rt_rq) && rt_rq->rt_nr_running ==0 ) { +struct sched_dl_entity *dl_se =dl_group_of (rt_rq); + +dl_server_start (dl_se); + } +enqueue_rt_entity (rt_se, flags);
Is it OK to start the server before the first task is enqueued in an idle group?
...
kernel/sched/core.c static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) · line 941
Quoted 7 lines
kernel/sched/core.c +3
-0@@ -891,6 +936,34 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) {struct task_struct *donor = rq->donor; +if (!rt_group_sched_enabled ()) + goto no_group_sched; +
I think the below deserves a comment detailing the rules of preemption (inside/outside groups, etc.).
kernel/sched/core.c static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) · line 952
Quoted 11 lines
kernel/sched/core.c +11
-0+if (is_dl_group (rt_rq_of_se (&p->rt)) && +is_dl_group (rt_rq_of_se (&rq->curr->rt))) { +struct sched_dl_entity *dl_se, *curr_dl_se; + + dl_se =dl_group_of (rt_rq_of_se (&p->rt)); + curr_dl_se =dl_group_of (rt_rq_of_se (&rq->curr->rt)); + +if (dl_entity_preempt (dl_se, curr_dl_se)) { +resched_curr (rq); +return ; + }else if (!dl_entity_preempt (curr_dl_se, dl_se)) {
Isn't this condition implied by the above?
...
Quoted 13 lines
kernel/sched/core.c +13
-0+#ifdef CONFIG_RT_GROUP_SCHED +static int group_push_rt_task (struct rt_rq *rt_rq) +{ +struct rq *rq =rq_of_rt_rq (rt_rq); + +if (is_dl_group (rt_rq)) +return 0 ; + +return push_rt_task (rq, false); +} + +static void group_push_rt_tasks (struct rt_rq *rt_rq) +{
kernel/sched/core.c static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)
Quoted 10 lines
kernel/sched/core.c +10
-0+while (group_push_rt_task (rt_rq)) + ; +} +#else +static void group_push_rt_tasks (struct rt_rq *rt_rq) +{ +push_rt_tasks (rq_of_rt_rq (rt_rq)); +} +#endif +
Hummm, maybe too much for a single patch? I am a little lost at this point. :))
Thanks,
Juri
RFC PATCH v2 14/25 sched/rt: Update rt-cgroup schedulability checks
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Update schedulability checks and setup of runtime/period for rt-cgroups.
--- kernel/sched/core.c | 6 ++++ kernel/sched/deadline.c | 46 ++++++++++++++++++++++---- kernel/sched/rt.c | 72 +++++++++++++++++++++++------------------ kernel/sched/sched.h | 1 + 4 files changed, 88 insertions(+), 37 deletions(-)
kernel/sched/core.c +6 -0
@@ -9277,6 +9277,12 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)return &root_task_group.css; } +/* Do not allow cpu_cgroup hierachies with depth greater than 2. */ +#ifdef CONFIG_RT_GROUP_SCHED +if (parent != &root_task_group) +return ERR_PTR (-EINVAL ); +#endif + tg =sched_create_group (parent);if (IS_ERR (tg))return ERR_PTR (-ENOMEM );
kernel/sched/deadline.c +40 -6
59 lines not on this page · raw message
kernel/sched/rt.c +41 -31
157 lines not on this page · raw message
kernel/sched/sched.h +1 -0
@@ -394,6 +394,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,dl_server_has_tasks_f has_tasks, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void ); +extern int dl_check_tg (unsigned long total);extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period); extern void dl_server_update_idle_time(struct rq *rq,
Signature · 1 line
2.50.1
RFC PATCH v2 15/25 sched/rt: Remove old RT_GROUP_SCHED data structures
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Completely remove the old RT_GROUP_SCHED's functions and data structures.
--- include/linux/sched.h | 4 ---- kernel/sched/rt.c | 1 - kernel/sched/sched.h | 26 -------------------------- 3 files changed, 31 deletions(-)
include/linux/sched.h +0 -4
@@ -621,13 +621,9 @@ struct sched_rt_entity {unsigned short on_rq;unsigned short on_list; -struct sched_rt_entity *back;#ifdef CONFIG_RT_GROUP_SCHED -struct sched_rt_entity *parent;/* rq on which this entity is (to be) queued: */ struct rt_rq *rt_rq; -/* rq "owned" by this entity/group: */ -struct rt_rq *my_q;#endif } __randomize_layout;
kernel/sched/rt.c +0 -1
@@ -1,4 +1,3 @@ -#pragma GCC diagnostic ignored "-Wunused-function"// SPDX-License-Identifier: GPL-2.0 /* * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
kernel/sched/sched.h +0 -26
@@ -310,15 +310,6 @@ struct rt_prio_array {struct list_head queue[MAX_RT_PRIO ]; }; -struct rt_bandwidth { -/* nests inside the rq lock: */ -raw_spinlock_t rt_runtime_lock; -ktime_t rt_period; -u64 rt_runtime; -struct hrtimer rt_period_timer; -unsigned int rt_period_active; -}; -struct dl_bandwidth {raw_spinlock_t dl_runtime_lock;u64 dl_runtime;
@@ -483,7 +474,6 @@ struct task_group {struct sched_dl_entity **dl_se;struct rt_rq **rt_rq; -struct rt_bandwidth rt_bandwidth;struct dl_bandwidth dl_bandwidth;#endif
@@ -802,11 +792,6 @@ struct scx_rq { };#endif /* CONFIG_SCHED_CLASS_EXT */ -static inline int rt_bandwidth_enabled (void ) -{ -return 0 ; -} -/* RT IPI pull logic requires IRQ_WORK */ #if defined(CONFIG_IRQ_WORK ) && defined(CONFIG_SMP )# define HAVE_RT_PUSH_IPI
@@ -824,17 +809,6 @@ struct rt_rq {bool overloaded;struct plist_head pushable_tasks; -int rt_queued; - -#ifdef CONFIG_RT_GROUP_SCHED -int rt_throttled; -u64 rt_time;/* consumed RT time, goes up in update_curr_rt */ -u64 rt_runtime;/* allotted RT time, "slice" from rt_bandwidth, RT sharing/balancing */ -/* Nests inside the rq lock: */ -raw_spinlock_t rt_runtime_lock; - -unsigned int rt_nr_boosted; -#endif #ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */ #endif
Signature · 1 line
2.50.1
RFC PATCH v2 16/25 sched/core: Cgroup v2 support
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make rt_runtime_us and rt_period_us virtual files accessible also to the cgroup v2 controller, effectively enabling the RT_GROUP_SCHED mechanism to cgroups v2.
--- kernel/sched/core.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/sched/core.c +12 -0
@@ -10300,6 +10300,18 @@ static struct cftype cpu_files[] = { .write = cpu_uclamp_max_write, },#endif /* CONFIG_UCLAMP_TASK_GROUP */ +#ifdef CONFIG_RT_GROUP_SCHED + { + .name = "rt_runtime_us", + .read_s64 = cpu_rt_runtime_read, + .write_s64 = cpu_rt_runtime_write, + }, + { + .name = "rt_period_us", + .read_u64 = cpu_rt_period_read_uint, + .write_u64 = cpu_rt_period_write_uint, + }, +#endif /* CONFIG_RT_GROUP_SCHED */ { } /* terminate */ };
Signature · 1 line
2.50.1
RFC PATCH v2 17/25 sched/rt: Remove support for cgroups-v1
In reply to Yuri AndriaccioDisable control files for cgroups-v1, and allow only cgroups-v2. This should simplify maintaining the code, also because cgroups-v1 are deprecated.
--- kernel/sched/core.c | 18 ------------------ 1 file changed, 18 deletions(-)
kernel/sched/core.c +0 -18
@@ -10039,20 +10039,6 @@ static struct cftype cpu_legacy_files[] = { };#ifdef CONFIG_RT_GROUP_SCHED -static struct cftype rt_group_files[] = { - { - .name ="rt_runtime_us" , - .read_s64 = cpu_rt_runtime_read, - .write_s64 = cpu_rt_runtime_write, - }, - { - .name ="rt_period_us" , - .read_u64 = cpu_rt_period_read_uint, - .write_u64 = cpu_rt_period_write_uint, - }, - { }/* Terminate */ -}; -# ifdef CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED DEFINE_STATIC_KEY_FALSE (rt_group_sched);# else
@@ -10078,10 +10064,6 @@ __setup("rt_group_sched=", setup_rt_group_sched);static int __init cpu_rt_group_init (void ) { -if (!rt_group_sched_enabled ()) -return 0 ; - -WARN_ON (cgroup_add_legacy_cftypes (&cpu_cgrp_subsys, rt_group_files));return 0 ; }subsys_initcall (cpu_rt_group_init);
Signature · 1 line
2.50.1
RFC PATCH v2 18/25 sched/rt: Zero rt-cgroups default bandwidth
In reply to Yuri AndriaccioSet the default rt-cgroups runtime to zero, otherwise a cgroup-v1 kernel will not be able to start SCHED_DEADLINE tasks. The bandwidth for rt-cgroups must then be manually assigned after the kernel boots.
Allow zeroing the runtime of the root control group. This runtime only affects the available bandwidth of the rt-cgroup hierarchy but not the SCHED_FIFO / SCHED_RR tasks on the global runqueue.
Notes:
Disabling the root control group bandwidth should not cause any side effect, as SCHED_FIFO / SCHED_RR tasks do not depend on it since the introduction of fair_servers.
--- kernel/sched/core.c | 4 ++-- kernel/sched/rt.c | 13 +++++-------- kernel/sched/syscalls.c | 2 +- 3 files changed, 8 insertions(+), 11 deletions(-)
kernel/sched/core.c +2 -2
@@ -8719,7 +8719,7 @@ void __init sched_init(void)#ifdef CONFIG_RT_GROUP_SCHED init_dl_bandwidth (&root_task_group.dl_bandwidth, -global_rt_period (),global_rt_runtime ()); +global_rt_period (),0 );#endif /* CONFIG_RT_GROUP_SCHED */ #ifdef CONFIG_CGROUP_SCHED
@@ -9348,7 +9348,7 @@ static int cpu_cgroup_can_attach(struct cgroup_taskset *tset) goto scx_check;cgroup_taskset_for_each (task, css, tset) { -if (!sched_rt_can_attach (css_tg (css), task)) +if (rt_task (task) && !sched_rt_can_attach (css_tg (css), task))return -EINVAL ; } scx_check:
kernel/sched/rt.c +5 -8
@@ -2122,13 +2122,6 @@ static int tg_set_rt_bandwidth(struct task_group *tg,static DEFINE_MUTEX (rt_constraints_mutex);int i, err =0 ; -/* -* Disallowing the root group RT runtime is BAD, it would disallow the -* kernel creating (and or operating) RT threads. -*/ -if (tg == &root_task_group && rt_runtime ==0 ) -return -EINVAL ; -/* No period doesn't make any sense. */ if (rt_period ==0 )return -EINVAL ;
@@ -2215,8 +2208,12 @@ static int sched_rt_global_constraints(void)int sched_rt_can_attach (struct task_group *tg,struct task_struct *tsk) { +/* Allow executing in the root cgroup regardless of allowed bandwidth */ +if (tg == &root_task_group) +return 1 ; +/* Don't accept real-time tasks when there is no way for them to run */ -if (rt_group_sched_enabled () &&rt_task (tsk) && tg->dl_bandwidth.dl_runtime ==0 ) +if (rt_group_sched_enabled () && tg->dl_bandwidth.dl_runtime ==0 )return 0 ;return 1 ;
kernel/sched/syscalls.c +1 -1
@@ -633,7 +633,7 @@ int __sched_setscheduler(struct task_struct *p,*/ if (rt_group_sched_enabled () &&dl_bandwidth_enabled () &&rt_policy (policy) && -task_group (p)->dl_bandwidth.dl_runtime ==0 && + !sched_rt_can_attach (task_group (p), p) && !task_group_is_autogroup (task_group (p))) { retval = -EPERM ; goto unlock;
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 18/25] sched/rt: Zero rt-cgroups default bandwidth
Hi!
Quoted 2 lines · Yuri Andriaccio
Set the default rt-cgroups runtime to zero, otherwise a cgroup-v1 kernel will not be able to start
SCHED_DEADLINEtasks. The bandwidth for rt-cgroups must
Well, we disabled v1 support at this point with the previous patch, didn't we? :)
Quoted 10 lines
then be manually assigned after the kernel boots.
Allow zeroing the runtime of the root control group. This runtime only affects the available bandwidth of the rt-cgroup hierarchy but not the
SCHED_FIFO/SCHED_RRtasks on the global runqueue.Notes:
Disabling the root control group bandwidth should not cause any side effect, asSCHED_FIFO/SCHED_RRtasks do not depend on it since the introduction offair_servers.
I believe this deserves proper documentation.
Thanks,
Juri
RFC PATCH v2 19/25 sched/deadline: Allow deeper hierarchies of RT cgroups
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Allow creation of cgroup hierachies with depth greater than two. Add check to prevent attaching tasks to a child cgroup of an active cgroup (i.e. with a running FIFO/RR task).
Add check to prevent attaching tasks to cgroups which have children with non-zero runtime.
Update rt-cgroups allocated bandwidth accounting for nested cgroup hierachies.
--- kernel/sched/core.c | 6 ----- kernel/sched/deadline.c | 51 +++++++++++++++++++++++++++++++++++++---- kernel/sched/rt.c | 25 +++++++++++++++++--- kernel/sched/sched.h | 2 +- 4 files changed, 70 insertions(+), 14 deletions(-)
kernel/sched/core.c +0 -6
@@ -9277,12 +9277,6 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)return &root_task_group.css; } -/* Do not allow cpu_cgroup hierachies with depth greater than 2. */ -#ifdef CONFIG_RT_GROUP_SCHED -if (parent != &root_task_group) -return ERR_PTR (-EINVAL ); -#endif - tg =sched_create_group (parent);if (IS_ERR (tg))return ERR_PTR (-ENOMEM );
kernel/sched/deadline.c +47 -4
72 lines not on this page · raw message
kernel/sched/rt.c +22 -3
@@ -107,7 +107,8 @@ void free_rt_sched_group(struct task_group *tg)* Fix this issue by changing the group runtime * to 0 immediately before freeing it. */ -dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); +if (tg->dl_se[i]->dl_runtime) +dl_init_tg (tg, i,0 , tg->dl_se[i]->dl_period);raw_spin_rq_lock_irqsave (cpu_rq (i), flags);BUG_ON (tg->rt_rq[i]->rt_nr_running);raw_spin_rq_unlock_irqrestore (cpu_rq (i), flags);
@@ -2122,6 +2123,14 @@ static int tg_set_rt_bandwidth(struct task_group *tg,static DEFINE_MUTEX (rt_constraints_mutex);int i, err =0 ; +/* +* Do not allow to set a RT runtime > 0 if the parent has RT tasks +* (and is not the root group) +*/ +if (rt_runtime && (tg != &root_task_group) && (tg->parent != &root_task_group) &&tg_has_rt_tasks (tg->parent)) { +return -EINVAL ; + } +/* No period doesn't make any sense. */ if (rt_period ==0 )return -EINVAL ;
@@ -2145,7 +2154,7 @@ static int tg_set_rt_bandwidth(struct task_group *tg,return 0 ;for_each_possible_cpu (i) { -dl_init_tg (tg->dl_se[i], rt_runtime, rt_period); +dl_init_tg (tg, i, rt_runtime, rt_period); }return 0 ;
@@ -2208,6 +2217,9 @@ static int sched_rt_global_constraints(void)int sched_rt_can_attach (struct task_group *tg,struct task_struct *tsk) { +struct task_group *child; +int can_attach =1 ; +/* Allow executing in the root cgroup regardless of allowed bandwidth */ if (tg == &root_task_group)return 1 ;
@@ -2216,7 +2228,14 @@ int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk)if (rt_group_sched_enabled () && tg->dl_bandwidth.dl_runtime ==0 )return 0 ; - return 1; +/* If one of the children has runtime > 0, cannot attach RT tasks! */ +list_for_each_entry_rcu (child, &tg->children, siblings) { +if (child->dl_bandwidth.dl_runtime) { + can_attach =0 ; + } + } + + return can_attach; }#else /* !CONFIG_RT_GROUP_SCHED: */
kernel/sched/sched.h +1 -1
@@ -386,7 +386,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void );extern int dl_check_tg (unsigned long total); -extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period); +extern void dl_init_tg (struct task_group *tg,int cpu,u64 rt_runtime,u64 rt_period);extern void dl_server_update_idle_time (struct rq *rq,struct task_struct *p);
Signature · 1 line
2.50.1
Juri Lelli
maintainerRe: [RFC PATCH v2 19/25] sched/deadline: Allow deeper hierarchies of RT cgroups
Hi!
On 31/07/25 12:55, Yuri Andriaccio wrote:
...
Quoted 36 lines
kernel/sched/deadline.c +26
-1@@ -406,11 +406,42 @@ int dl_check_tg(unsigned long total)return 1 ; } -void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period) +static inline bool is_active_sched_group (struct task_group *tg) { +struct task_group *child; +bool is_active =1 ; + +// if there are no children, this is a leaf group, thus it is active +list_for_each_entry_rcu (child, &tg->children, siblings) { +if (child->dl_bandwidth.dl_runtime >0 ) { + is_active =0 ; + } + } +return is_active; +} + +static inline bool sched_group_has_active_siblings (struct task_group *tg) +{ +struct task_group *child; +bool has_active_siblings =0 ; + +// if there are no children, this is a leaf group, thus it is active +list_for_each_entry_rcu (child, &tg->parent->children, siblings) { +if (child != tg && child->dl_bandwidth.dl_runtime >0 ) { + has_active_siblings =1 ; + } + } +return has_active_siblings; +}
...
kernel/sched/deadline.c int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk) · line 2236
Quoted 11 lines
kernel/sched/deadline.c +6
-1@@ -2216,7 +2228,14 @@ int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk)if (rt_group_sched_enabled () && tg->dl_bandwidth.dl_runtime ==0 )return 0 ; -return 1 ; +/* If one of the children has runtime > 0, cannot attach RT tasks! */ +list_for_each_entry_rcu (child, &tg->children, siblings) { +if (child->dl_bandwidth.dl_runtime) { + can_attach =0 ; + } + }
Can we maybe reuse some of the methods above?
Thanks,
Juri
RFC PATCH v2 20/25 sched/rt: Add rt-cgroup migration
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
When the runtime is exhausted in a RT CGroup, the scheduler checks for another non-throttled runqueue and, if available, migrates the tasks.
The bandwidth (runtime/period) chosen for a certain cgroup is replicated on every core of the system, therefore, in an SMP system with M cores, the total available bandwidth is the given runtime/period multiplied by M.
--- kernel/sched/rt.c | 471 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 5 + 2 files changed, 468 insertions(+), 8 deletions(-)
kernel/sched/rt.c +463 -8
@@ -1,3 +1,4 @@ +#pragma GCC diagnostic ignored "-Wunused-function"// SPDX-License-Identifier: GPL-2.0 /* * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -84,6 +85,8 @@ void init_rt_rq(struct rt_rq *rt_rq)plist_head_init (&rt_rq->pushable_tasks); } +static void group_pull_rt_task (struct rt_rq *this_rt_rq); +#ifdef CONFIG_RT_GROUP_SCHED void unregister_rt_sched_group (struct task_group *tg)
@@ -289,6 +292,45 @@ static inline void rt_queue_pull_task(struct rq *rq)queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); } +#ifdef CONFIG_RT_GROUP_SCHED +static DEFINE_PER_CPU(struct balance_callback, rt_group_push_head); +static DEFINE_PER_CPU(struct balance_callback, rt_group_pull_head); +static void push_group_rt_tasks (struct rq *); +static void pull_group_rt_tasks (struct rq *); + +static void rt_queue_push_from_group (struct rq *rq,struct rt_rq *rt_rq) +{ +BUG_ON (rt_rq ==NULL ); +BUG_ON (rt_rq->rq != rq); + +if (rq->rq_to_push_from) +return ; + + rq->rq_to_push_from = container_of(rt_rq, struct rq, rt); +queue_balance_callback (rq, &per_cpu (rt_group_push_head, rq->cpu), + push_group_rt_tasks); +} + +static void rt_queue_pull_to_group (struct rq *rq,struct rt_rq *rt_rq) +{ +struct sched_dl_entity *dl_se =dl_group_of (rt_rq); + +BUG_ON (rt_rq ==NULL ); +BUG_ON (!is_dl_group (rt_rq)); +BUG_ON (rt_rq->rq != rq); + +if (dl_se->dl_throttled || rq->rq_to_pull_to) +return ; + + rq->rq_to_pull_to = container_of(rt_rq, struct rq, rt); +queue_balance_callback (rq, &per_cpu (rt_group_pull_head, rq->cpu), + pull_group_rt_tasks); +} +#else +static inline void rt_queue_push_from_group (struct rq *rq,struct rt_rq *rt_rq) {}; +static inline void rt_queue_pull_to_group (struct rq *rq,struct rt_rq *rt_rq) {}; +#endif +static void enqueue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) {plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);
@@ -1277,6 +1319,8 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)*/ static int push_rt_task (struct rq *rq,bool pull) { +BUG_ON (is_dl_group (&rq->rt)); +struct task_struct *next_task;struct rq *lowest_rq;int ret =0 ;
@@ -1573,6 +1617,8 @@ void rto_push_irq_work_func(struct irq_work *work)static void pull_rt_task (struct rq *this_rq) { +BUG_ON (is_dl_group (&this_rq->rt)); +int this_cpu = this_rq->cpu, cpu;bool resched = false;struct task_struct *p, *push_task;
@@ -1683,27 +1729,436 @@ static void pull_rt_task(struct rq *this_rq) }#ifdef CONFIG_RT_GROUP_SCHED -static int group_push_rt_task (struct rt_rq *rt_rq) +/* +* Find the lowest priority runqueue among the runqueues of the same +* task group. Unlike find_lowest_rt(), this does not mean that the +* lowest priority cpu is running tasks from this runqueue. +*/ +static int group_find_lowest_rt_rq (struct task_struct *task,struct rt_rq * task_rt_rq) +{ +struct sched_domain *sd; +struct cpumask mask, *lowest_mask = &mask; +struct sched_dl_entity *dl_se; +struct rt_rq *rt_rq; +int prio, lowest_prio; +int cpu, this_cpu =smp_processor_id (); + +BUG_ON (task->sched_task_group != task_rt_rq->tg); + +if (task->nr_cpus_allowed ==1 ) +return -1 ;/* No other targets possible */ + + lowest_prio = task->prio -1 ; +cpumask_clear (lowest_mask); +for_each_cpu_and (cpu, cpu_online_mask, task->cpus_ptr) { + dl_se = task_rt_rq->tg->dl_se[cpu]; + rt_rq = &dl_se->my_q->rt; + prio = rt_rq->highest_prio.curr; + +/* +* If we're on asym system ensure we consider the different capacities +* of the CPUs when searching for the lowest_mask. +*/ +if (dl_se->dl_throttled || !rt_task_fits_capacity (task, cpu)) +continue ; + +if (prio >= lowest_prio) { +if (prio > lowest_prio) { +cpumask_clear (lowest_mask); + lowest_prio = prio; + } + +cpumask_set_cpu (cpu, lowest_mask); + } + } + +if (cpumask_empty (lowest_mask)) +return -1 ; + +/* +* At this point we have built a mask of CPUs representing the +* lowest priority tasks in the system. Now we want to elect +* the best one based on our affinity and topology. +* +* We prioritize the last CPU that the task executed on since +* it is most likely cache-hot in that location. +*/ + cpu =task_cpu (task); +if (cpumask_test_cpu (cpu, lowest_mask)) +return cpu; + +/* +* Otherwise, we consult the sched_domains span maps to figure +* out which CPU is logically closest to our hot cache data. +*/ +if (!cpumask_test_cpu (this_cpu, lowest_mask)) + this_cpu =-1 ;/* Skip this_cpu opt if not among lowest */ + +rcu_read_lock (); +for_each_domain (cpu, sd) { +if (sd->flags &SD_WAKE_AFFINE ) { +int best_cpu; + +/* +* "this_cpu" is cheaper to preempt than a +* remote processor. +*/ +if (this_cpu !=-1 && +cpumask_test_cpu (this_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return this_cpu; + } + + best_cpu =cpumask_any_and_distribute (lowest_mask, +sched_domain_span (sd)); +if (best_cpu < nr_cpu_ids) { +rcu_read_unlock (); +return best_cpu; + } + } + } +rcu_read_unlock (); + +/* +* And finally, if there were no matches within the domains +* just give the caller *something* to work with from the compatible +* locations. +*/ +if (this_cpu !=-1 ) +return this_cpu; + + cpu =cpumask_any_distribute (lowest_mask); +if (cpu < nr_cpu_ids) +return cpu; + +return -1 ; +} + +/* +* Find and lock the lowest priority runqueue among the runqueues +* of the same task group. Unlike find_lock_lowest_rt(), this does not +* mean that the lowest priority cpu is running tasks from this runqueue. +*/ +static struct rt_rq *group_find_lock_lowest_rt_rq (struct task_struct *task,struct rt_rq *rt_rq) +{ +struct rq *rq =rq_of_rt_rq (rt_rq); +struct rq *lowest_rq; +struct rt_rq *lowest_rt_rq =NULL ; +struct sched_dl_entity *lowest_dl_se; +int tries, cpu; + +BUG_ON (task->sched_task_group != rt_rq->tg); + +for (tries =0 ; tries <RT_MAX_TRIES ; tries++) { + cpu =group_find_lowest_rt_rq (task, rt_rq); + +if ((cpu ==-1 ) || (cpu == rq->cpu)) +break ; + + lowest_dl_se = rt_rq->tg->dl_se[cpu]; + lowest_rt_rq = &lowest_dl_se->my_q->rt; + lowest_rq =cpu_rq (cpu); + +if (lowest_rt_rq->highest_prio.curr <= task->prio) { +/* +* Target rq has tasks of equal or higher priority, +* retrying does not release any lock and is unlikely +* to yield a different result. +*/ + lowest_rt_rq =NULL ; +break ; + } + +/* if the prio of this runqueue changed, try again */ +if (double_lock_balance (rq, lowest_rq)) { +/* +* We had to unlock the run queue. In +* the mean time, task could have +* migrated already or had its affinity changed. +* Also make sure that it wasn't scheduled on its rq. +* It is possible the task was scheduled, set +* "migrate_disabled" and then got preempted, so we must +* check the task migration disable flag here too. +*/ +if (unlikely (is_migration_disabled (task) || + lowest_dl_se->dl_throttled || + !cpumask_test_cpu (lowest_rq->cpu, &task->cpus_mask) || + task !=pick_next_pushable_task (rt_rq))) { + +double_unlock_balance (rq, lowest_rq); + lowest_rt_rq =NULL ; +break ; + } + } + +/* If this rq is still suitable use it. */ +if (lowest_rt_rq->highest_prio.curr > task->prio) +break ; + +/* try again */ +double_unlock_balance (rq, lowest_rq); + lowest_rt_rq =NULL ; + } + +return lowest_rt_rq; +} + +static int group_push_rt_task (struct rt_rq *rt_rq,bool pull) { +BUG_ON (!is_dl_group (rt_rq)); +struct rq *rq =rq_of_rt_rq (rt_rq); +struct task_struct *next_task; +struct rq *lowest_rq; +struct rt_rq *lowest_rt_rq; +int ret =0 ; + +if (!rt_rq->overloaded) +return 0 ; + + next_task =pick_next_pushable_task (rt_rq); +if (!next_task) +return 0 ; + +retry: +if (is_migration_disabled (next_task)) { +struct task_struct *push_task =NULL ; +int cpu; + +if (!pull || rq->push_busy) +return 0 ; + +/* +* If the current task does not belong to the same task group +* we cannot push it away. +*/ +if (rq->curr->sched_task_group != rt_rq->tg) +return 0 ; + +/* +* Invoking group_find_lowest_rt_rq() on anything but an RT task doesn't +* make sense. Per the above priority check, curr has to +* be of higher priority than next_task, so no need to +* reschedule when bailing out. +* +* Note that the stoppers are masqueraded as SCHED_FIFO +* (cf. sched_set_stop_task()), so we can't rely on rt_task(). +*/ +if (rq->curr->sched_class != &rt_sched_class) +return 0 ; + + cpu =group_find_lowest_rt_rq (rq->curr, rt_rq); +if (cpu ==-1 || cpu == rq->cpu) +return 0 ; + +/* +* Given we found a CPU with lower priority than @next_task, +* therefore it should be running. However we cannot migrate it +* to this other CPU, instead attempt to push the current +* running task on this CPU away. +*/ + push_task =get_push_task (rq); +if (push_task) { +preempt_disable (); +raw_spin_rq_unlock (rq); +stop_one_cpu_nowait (rq->cpu, push_cpu_stop, + push_task, &rq->push_work); +preempt_enable (); +raw_spin_rq_lock (rq); + } -if (is_dl_group (rt_rq))return 0 ; + } + +if (WARN_ON (next_task == rq->curr)) +return 0 ; + +/* We might release rq lock */ +get_task_struct (next_task); + +/* group_find_lock_lowest_rq locks the rq if found */ + lowest_rt_rq =group_find_lock_lowest_rt_rq (next_task, rt_rq); +if (!lowest_rt_rq) { +struct task_struct *task; +/* +* group_find_lock_lowest_rt_rq releases rq->lock +* so it is possible that next_task has migrated. +* +* We need to make sure that the task is still on the same +* run-queue and is also still the next task eligible for +* pushing. +*/ + task =pick_next_pushable_task (rt_rq); +if (task == next_task) { +/* +* The task hasn't migrated, and is still the next +* eligible task, but we failed to find a run-queue +* to push it to. Do not retry in this case, since +* other CPUs will pull from us when ready. +*/ + goto out; + } + +if (!task) +/* No more tasks, just exit */ + goto out; + +/* +* Something has shifted, try again. +*/ +put_task_struct (next_task); + next_task = task; + goto retry; + } + + lowest_rq =rq_of_rt_rq (lowest_rt_rq); + +move_queued_task_locked (rq, lowest_rq, next_task); +resched_curr (lowest_rq); + ret =1 ; + +double_unlock_balance (rq, lowest_rq); +out: +put_task_struct (next_task); + +return ret; +} + +static void group_pull_rt_task (struct rt_rq *this_rt_rq) +{ +BUG_ON (!is_dl_group (this_rt_rq)); + +struct rq *this_rq =rq_of_rt_rq (this_rt_rq); +int this_cpu = this_rq->cpu, cpu; +bool resched = false; +struct task_struct *p, *push_task =NULL ; +struct rt_rq *src_rt_rq; +struct rq *src_rq; +struct sched_dl_entity *src_dl_se; + +for_each_online_cpu (cpu) { +if (this_cpu == cpu) +continue ; -return push_rt_task (rq, false); + src_dl_se = this_rt_rq->tg->dl_se[cpu]; + src_rt_rq = &src_dl_se->my_q->rt; + +if (src_rt_rq->rt_nr_running <=1 && !src_dl_se->dl_throttled) +continue ; + + src_rq =rq_of_rt_rq (src_rt_rq); + +/* +* Don't bother taking the src_rq->lock if the next highest +* task is known to be lower-priority than our current task. +* This may look racy, but if this value is about to go +* logically higher, the src_rq will push this task away. +* And if its going logically lower, we do not care +*/ +if (src_rt_rq->highest_prio.next >= + this_rt_rq->highest_prio.curr) +continue ; + +/* +* We can potentially drop this_rq's lock in +* double_lock_balance, and another CPU could +* alter this_rq +*/ + push_task =NULL ; +double_lock_balance (this_rq, src_rq); + +/* +* We can pull only a task, which is pushable +* on its rq, and no others. +*/ + p =pick_highest_pushable_task (src_rt_rq, this_cpu); + +/* +* Do we have an RT task that preempts +* the to-be-scheduled task? +*/ +if (p && (p->prio < this_rt_rq->highest_prio.curr)) { +WARN_ON (p == src_rq->curr); +WARN_ON (!task_on_rq_queued (p)); + +/* +* There's a chance that p is higher in priority +* than what's currently running on its CPU. +* This is just that p is waking up and hasn't +* had a chance to schedule. We only pull +* p if it is lower in priority than the +* current task on the run queue +*/ +if (p->prio < src_rq->curr->prio) + goto skip; + +if (is_migration_disabled (p)) { +/* +* If the current task does not belong to the same task group +* we cannot push it away. +*/ +if (src_rq->curr->sched_task_group != this_rt_rq->tg) + goto skip; + + push_task =get_push_task (src_rq); + }else { +move_queued_task_locked (src_rq, this_rq, p); + resched = true; + } +/* +* We continue with the search, just in +* case there's an even higher prio task +* in another runqueue. (low likelihood +* but possible) +*/ + } +skip: +double_unlock_balance (this_rq, src_rq); + +if (push_task) { +preempt_disable (); +raw_spin_rq_unlock (this_rq); +stop_one_cpu_nowait (src_rq->cpu, push_cpu_stop, + push_task, &src_rq->push_work); +preempt_enable (); +raw_spin_rq_lock (this_rq); + } + } + +if (resched) +resched_curr (this_rq); }static void group_push_rt_tasks (struct rt_rq *rt_rq) { -while (group_push_rt_task (rt_rq)) +while (group_push_rt_task (rt_rq, false)) ; } -#else -static void group_push_rt_tasks (struct rt_rq *rt_rq) + +static void push_group_rt_tasks (struct rq *rq) { -push_rt_tasks (rq_of_rt_rq (rt_rq)); +BUG_ON (rq->rq_to_push_from ==NULL ); + +if ((rq->rq_to_push_from->rt.rt_nr_running >1 ) || + (dl_group_of (&rq->rq_to_push_from->rt)->dl_throttled ==1 )) { +group_push_rt_task (&rq->rq_to_push_from->rt, false); + } + + rq->rq_to_push_from =NULL ; } -#endif + +static void pull_group_rt_tasks (struct rq *rq) +{ +BUG_ON (rq->rq_to_pull_to ==NULL ); +BUG_ON (rq->rq_to_pull_to->rt.rq != rq); + +group_pull_rt_task (&rq->rq_to_pull_to->rt); + rq->rq_to_pull_to =NULL ; +} +#else /* CONFIG_RT_GROUP_SCHED */ +static void group_pull_rt_task (struct rt_rq *this_rt_rq) { } +static void group_push_rt_tasks (struct rt_rq *rt_rq) { } +#endif /* CONFIG_RT_GROUP_SCHED */ /* * If we are not running and we are not going to reschedule soon, we should
kernel/sched/sched.h +5 -0
@@ -1282,6 +1282,11 @@ struct rq {call_single_data_t cfsb_csd;struct list_head cfsb_csd_list;#endif + +#ifdef CONFIG_RT_GROUP_SCHED +struct rq *rq_to_push_from; +struct rq *rq_to_pull_to; +#endif };#ifdef CONFIG_FAIR_GROUP_SCHED
Signature · 1 line
2.50.1
RFC PATCH v2 21/25 sched/rt: add HCBS migration related checks and function calls
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add HCBS related checks and operations to allow rt-task migration, differentiating between cgroup's tasks or tasks that run on the global runqueue.
--- kernel/sched/rt.c | 63 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 15 deletions(-)
kernel/sched/rt.c +48 -15
@@ -865,6 +865,11 @@ select_task_rq_rt(struct task_struct *p, int cpu, int flags)struct rq *rq;bool test; +/* Just return the task_cpu for processes inside task groups */ +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) && +is_dl_group (rt_rq_of_se (&p->rt))) + goto out; +/* For anything but wake ups, just return the task_cpu */ if (!(flags & (WF_TTWU |WF_FORK ))) goto out;
@@ -964,7 +969,10 @@ static int balance_rt(struct rq *rq, struct task_struct *p, struct rq_flags *rf)* not yet started the picking loop. */ rq_unpin_lock (rq, rf); -pull_rt_task (rq); +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq_of_se (&p->rt))) +group_pull_rt_task (rt_rq_of_se (&p->rt)); +else +pull_rt_task (rq);rq_repin_lock (rq, rf); }
@@ -1050,7 +1058,10 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool fif (rq->donor->sched_class != &rt_sched_class)update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); -rt_queue_push_tasks (rq); +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +rt_queue_push_from_group (rq, rt_rq); +else +rt_queue_push_tasks (rq); }static struct sched_rt_entity *pick_next_rt_entity (struct rt_rq *rt_rq)
@@ -1113,6 +1124,13 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s*/ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 )enqueue_pushable_task (rt_rq, p); + +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) { +struct sched_dl_entity *dl_se =dl_group_of (rt_rq); + +if (dl_se->dl_throttled) +rt_queue_push_from_group (rq, rt_rq); + } }/* Only try algorithms three times */
@@ -2174,8 +2192,13 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) (rq->curr->nr_cpus_allowed <2 || rq->donor->prio <= p->prio); -if (need_to_push) +if (!need_to_push) +return ; + +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq))group_push_rt_tasks (rt_rq); +else +push_rt_tasks (rq); }/* Assumes rq->lock is held */
@@ -2216,7 +2239,9 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)if (!task_on_rq_queued (p) || rt_rq->rt_nr_running)return ; -if (!IS_ENABLED (CONFIG_RT_GROUP_SCHED )) +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +rt_queue_pull_to_group (rq, rt_rq); +else rt_queue_pull_task (rq); }
@@ -2243,6 +2268,13 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)*/ if (task_current (rq, p)) {update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); + +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq_of_se (&p->rt))) { +struct sched_dl_entity *dl_se =dl_group_of (rt_rq_of_se (&p->rt)); + + p->dl_server = dl_se; + } +return ; }
@@ -2252,17 +2284,14 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)* then see if we can move to another run queue. */ if (task_on_rq_queued (p)) { - -#ifndef CONFIG_RT_GROUP_SCHED -if (p->nr_cpus_allowed >1 && rq->rt.overloaded) +if (!is_dl_group (rt_rq_of_se (&p->rt)) && p->nr_cpus_allowed >1 && rq->rt.overloaded) {rt_queue_push_tasks (rq); -#else -if (rt_rq_of_se (&p->rt)->overloaded) { - }else { -if (p->prio < rq->curr->prio) -resched_curr (rq); +return ; + }else if (is_dl_group (rt_rq_of_se (&p->rt)) &&rt_rq_of_se (&p->rt)->overloaded) { +rt_queue_push_from_group (rq,rt_rq_of_se (&p->rt)); +return ; } -#endif +if (p->prio < rq->donor->prio &&cpu_online (cpu_of (rq)))resched_curr (rq); }
@@ -2285,8 +2314,12 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)* If our priority decreases while running, we * may need to pull tasks to this runqueue. */ -if (!IS_ENABLED (CONFIG_RT_GROUP_SCHED ) && oldprio < p->prio) -rt_queue_pull_task (rq); +if (oldprio < p->prio) { +if (IS_ENABLED (CONFIG_RT_GROUP_SCHED ) &&is_dl_group (rt_rq)) +rt_queue_pull_to_group (rq, rt_rq); +else +rt_queue_pull_task (rq); + }/* * If there's a higher priority task waiting to run
Signature · 1 line
2.50.1
RFC PATCH v2 22/25 sched/deadline: Make rt-cgroup's servers pull tasks on timer replenishment
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
--- kernel/sched/deadline.c | 6 +++++- kernel/sched/rt.c | 6 +++++- 2 files changed, 10 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +5 -1
@@ -1308,6 +1308,7 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_ {struct rq *rq =rq_of_dl_se (dl_se);u64 fw; +bool has_tasks; scoped_guard (rq_lock, rq) {struct rq_flags *rf = &scope.rf;
@@ -1321,7 +1322,10 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_if (!dl_se->dl_runtime)return HRTIMER_NORESTART ; -if (!dl_se->server_has_tasks (dl_se)) { +rq_unpin_lock (rq, rf); + has_tasks = dl_se->server_has_tasks (dl_se); +rq_repin_lock (rq, rf); +if (!has_tasks) {replenish_dl_entity (dl_se);dl_server_stopped (dl_se);return HRTIMER_NORESTART ;
kernel/sched/rt.c +5 -1
@@ -1,4 +1,3 @@ -#pragma GCC diagnostic ignored "-Wunused-function"// SPDX-License-Identifier: GPL-2.0 /* * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -145,6 +144,11 @@ void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq,static bool rt_server_has_tasks (struct sched_dl_entity *dl_se) { +#ifdef CONFIG_SMP +struct rt_rq *rt_rq = &dl_se->my_q->rt; + +group_pull_rt_task (rt_rq); +#endif return !!dl_se->my_q->rt.rt_nr_running; }
Signature · 1 line
2.50.1
RFC PATCH v2 23/25 sched/deadline: Fix HCBS migrations on server stop
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Do not unthrottle a non-fair-server dl_server_stop(), since it ends up being stopped when throttled (we try to migrate all the RT tasks away from it).
Notes:
This is a temporary workaround, but it will be hopefully removed in favor of less invasive code.
--- kernel/sched/deadline.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +4 -2
@@ -1723,9 +1723,11 @@ void dl_server_stop(struct sched_dl_entity *dl_se)return ;dequeue_dl_entity (dl_se,DEQUEUE_SLEEP ); -hrtimer_try_to_cancel (&dl_se->dl_timer); +if (dl_se == &rq_of_dl_se (dl_se)->fair_server) { +hrtimer_try_to_cancel (&dl_se->dl_timer); + dl_se->dl_throttled =0 ; + } dl_se->dl_defer_armed =0 ; - dl_se->dl_throttled =0 ; dl_se->dl_server_active =0 ; }
Signature · 1 line
2.50.1
RFC PATCH v2 24/25 sched/core: Execute enqueued balance callbacks when changing allowed CPUs
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Execute balancing callbacks when setting the affinity of a task, since the HCBS scheduler may request balancing of throttled dl_servers to fully utilize the server's bandwidth.
--- kernel/sched/core.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/sched/core.c +13 -0
@@ -2947,6 +2947,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flagif (cpumask_test_cpu (task_cpu (p), &p->cpus_mask) || (task_current_donor (rq, p) && !task_current (rq, p))) {struct task_struct *push_task =NULL ; +struct balance_callback *head;if ((flags &SCA_MIGRATE_ENABLE ) && (p->migration_flags &MDF_PUSH ) && !rq->push_busy) {
@@ -2965,11 +2966,13 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag }preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf);if (push_task) {stop_one_cpu_nowait (rq->cpu, push_cpu_stop, p, &rq->push_work); } +balance_callbacks (rq, head);preempt_enable ();if (complete)
@@ -3024,6 +3027,8 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag }if (task_on_cpu (rq, p) ||READ_ONCE (p->__state) ==TASK_WAKING ) { +struct balance_callback *head; +/* * MIGRATE_ENABLE gets here because 'p == current', but for * anything else we cannot do is_migration_disabled(), punt
@@ -3037,16 +3042,19 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag p->migration_flags &= ~MDF_PUSH ;preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf);if (!stop_pending) {stop_one_cpu_nowait (cpu_of (rq), migration_cpu_stop, &pending->arg, &pending->stop_work); } +balance_callbacks (rq, head);preempt_enable ();if (flags &SCA_MIGRATE_ENABLE )return 0 ; }else { +struct balance_callback *head;if (!is_migration_disabled (p)) {if (task_on_rq_queued (p))
@@ -3057,7 +3065,12 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag complete = true; } } + +preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf); +balance_callbacks (rq, head); +preempt_enable ();if (complete)complete_all (&pending->done);
Signature · 1 line
2.50.1
RFC PATCH v2 25/25 sched/core: Execute enqueued balance callbacks when migrating task betweeen cgroups
In reply to Yuri AndriaccioExecute balancing callbacks when migrating task between cgroups, since the HCBS scheduler, similarly to the previous patch, may request balancing of throttled dl_servers to fully utilize the server's bandwidth.
--- kernel/sched/core.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/core.c +9 -2
@@ -9247,10 +9247,11 @@ void sched_move_task(struct task_struct *tsk, bool for_autogroup) {int queued, running, queue_flags =DEQUEUE_SAVE |DEQUEUE_MOVE |DEQUEUE_NOCLOCK ; +struct balance_callback *head;struct rq *rq; +struct rq_flags rf; -CLASS (task_rq_lock, rq_guard)(tsk); - rq = rq_guard.rq; + rq =task_rq_lock (tsk, &rf);update_rq_clock (rq);
@@ -9277,6 +9278,12 @@ void sched_move_task(struct task_struct *tsk, bool for_autogroup)*/ resched_curr (rq); } + +preempt_disable (); + head =splice_balance_callbacks (rq); +task_rq_unlock (rq, tsk, &rf); +balance_callbacks (rq, head); +preempt_enable (); } static struct cgroup_subsys_state *
Signature · 1 line
2.50.1
Juri Lelli
maintainerHi,
Quoted 38 lines · Yuri Andriaccio
Hello,
This is the v2 for Hierarchical Constant Bandwidth Server, aiming at replacing the current
RT_GROUP_SCHEDmechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ . You can find the v1 of this patchset at the bottom of the page, which talks in more detail what this patchset is all about and how it is implemented.The big update for this v2 version is the addition of migration code, which allows to migrate tasks between different CPUs (following of course affinity settings).
As requested, we've split the big patches in smaller chunks in order to improve in readability. Additionally, it has been rebased on the latest tip/master to keep up with the latest scheduler updates and new features of
dl_servers.Last but not least, the first patch, which has been presented separately at
https://lore.kernel.org/all/[email protected]/ , is necessary to fully utilize the deadline bandwidth while keeping the fair-servers active. You can refer to the aforementioned link for details. The issue presented in this patch also reflects in HCBS: in the current version of the kernel, by default, 5% of the realtime bandwidth is reserved for fair-servers, 5% is not usable, and only the remaining 90% could be used by deadline tasks, or in our case, by HCBSdl_servers. The first patch addresses this issue and allows to fully utilize the default 95% of bandwidth for rt-tasks/servers.Summary of the patches:
1) Account fair-servers bw separately from other dl tasks and servers bw.
2-5) Preparation patches, so that the RT classes' code can be used both
for normal and cgroup scheduling.
6-15) Implementation of HCBS, no migration and only one level hierarchy.
The oldRT_GROUP_SCHEDcode is removed.
16-18) Remove cgroups v1 in favour of v2.
19) Add support for deeper hierarchies.
20-25) Add support for tasks migration.
Quoted 2 lines
Updates from v1:
- Rebase to tip/master.
Would you mind sharing the baseline sha this set applies to? It looks like it doesn't apply cleanly anymore.
Thanks!
Juri
Hi,
Quoted 2 lines
Would you mind sharing the baseline sha this set applies to? It looks like it doesn't apply cleanly anymore.
The patchset should apply cleanly on top of commit "Merge tag 'sysctl-6.17-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl", with sha 4b290aae788e06561754b28c6842e4080957d3f7 .
Have a nice day,
Yuri