Hierarchical Constant Bandwidth Server
Subsystems (6)
v7 cover Hierarchical Constant Bandwidth Server
Hello,
This is the seventh RFC for Hierarchical Constant Bandwidth Server, aiming at replacing the current RT_GROUP_SCHED mechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25 and OSPM26
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ and the latest updates and discussions at https://lwn.net/Articles/1078696/. You can find the previous versions of this patchset at the bottom of the page, in particular version 1 which talks in more detail what this patchset is all about and how it is implemented.
Version v7 introduces a lot of changes since v6. First off there have been a lot of bug fixes from the comments by the reviewers and the automated AI tool Sashiko, which surprisingly caught a lot of issues. Later, we've finally had time to add features which were planned a long time ago. The following are the most meaningful updates since v6 (the complete changelog can be found on the github readme):
- Rebase to kernel version 7.4-rc1.
- Address (most of) bugs as reported in the v6 reviews.
- Introduce dl_freq_invariant flag for dl_servers.
+ Force fair/ext servers to be frequency/capacity invariant, while allow HCBS servers to scale their consumed runtime.
+ This enables capacity-aware bandwidth reservation also for HCBS.
- Rework cgroup-v2 interface:
+ Rename cpu.rt.internal to cpu.rt.effective.local
+ Fix internal runtime rounding errors.
- Update cgroup balance callbacks to allow balancing operation of multiple cgroups at the same time.
- Allow unprivileged FIFO/RR tasks inside cgroups.
- Allow HRTICK_DL for HCBS servers.
- Integrate cpu and cpuset controllers.
- Integrate HCBS with hotplug/hotunplug operations.
- Introduce dl_server overrides (previously known as multi-runtime/multi-CPU).
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Unprivileged FIFO/RR tasks in cgroups:
Version v7 adds a simple patch to the sched_setattr syscall which allows non-root processes to execute FIFO/RR tasks in already configured HCBS cgroups. The reasoning follows from the fact that only the administrator is capable of creating and managing the real-time bandwidth, and, given the isolation properties of SCHED_DEADLINE, regular users, tied in that specific bandwidth, cannot cause starvation or steal time to other processes/users.
A non-root user cannot migrate its FIFO/RR task anywhere else in the cgroup hierarchy due to permissions, thus it is also impossible to move a user-created HCBS-FIFO task in the root cgroup and steal more bandwidth than the admin allowed.
cpu and cpuset controllers integration:
It has been requested by the maintainers to integrate HCBS with the cpuset cgroup controller, and it made a lot of sense since this enables easier integration with the rest of the scheduler classes in the class-switch code paths. The integration forces the rt-cgroup to have as active the same CPUs as those that cpuset.effective.local allows, while all the other dl_servers are by construction never activated. To simplify the code, the administrator must first setup the cpuset cpus and later setup the HCBS bandwidth. The current integration disallows changing allowed cpus while the HCBS bandwidth is reserved (thus the admin must first remove the HCBS reservation, change the cpuset, and re-request real-time bandwidth).
dl_server override feature:
Following the cpuset integration, we integrated the multi-runtime/multi-CPU
feature: the admin can request the override of the default runtime/period pair for a specific set of CPUs. The user interface works still by writing onto the cpu.rt.max file with a new format, which follows the nested-keyed standard. The original <runtime>|max <period> format is still supported, while the new nested-keyed format is default runtime=<value_us> period=<value_us> or default runtime=max for the default config values as in the original, <cpuset> runtime=<value_us> period=<value_us> to set a specific override and <cpuset> default to reset it. Note that the cpusets must be subsets of the allowed cpus in cpuset.effective.local, and that overrides with runtime zero are disallowed. These strict rules enforce that the active CPUs are exactly those allowed by the cpuset controller, no more no less. The only exception is a cgroup with zero runtime, which allows execution only if all CPUs are overridden, i.e. the same restriction as above (zero runtime cgroups simplifies setup in some cases).
A final note is about max cgroups, which are disallowed to have overrides, i.e. a cgroup is either delegating its FIFO/RR to its nearest active parent or it is managing these itself, but cannot have a mixed behaviour.
hotplug/hotunplug:
We've also fixed the hotplug/hotunplug failures. Hotunplug is allowed only if there are no active HCBS cgroups on the specific core that is going offline. Hotplug does not cause issues but HCBS reservations will need to be updated to make use of the newly enabled core (the cgroups' HCBS specific cpumask is only updated when setting the cgroups' bandwidth).
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Summary of the patches:
1) Introduce dl_freq_invariant flag.
2-7) Preparation patches, so that the RT classes' code can be used both
for normal and cgroup scheduling.
8-18) Implementation of HCBS, no migration.
The old RT_GROUP_SCHED code is removed.
15) Remove support for cgroup-v1.
16) Implement cgroup-v2 cpu.rt.max interface.
19-23) Add support for tasks migration.
24) Allow unprivileged FIFO/RR in rt-cgroups.
25) Enable HRTICK_DL for HCBS.
26) Documentation for HCBS (no cpuset integration yet).
27) Integrate cpuset and cpu controllers.
28) Fix CPU hotplug/hotunplug operations for rt-cgroups.
29) Introduce multi-cpu (dl_server override) support.
Updates from v6:
- Update to kernel version 7.1.
- Refactorings and general cleanups.
- Removal of substantial duplicated code.
- Express more locking constraints in code.
- New cpu.rt.max interface.
- Refactoring of migration code to reduce code duplication.
Updates from v5:
- Rebase to latest master.
- General rebasing/cleanup.
- More locking contraints expressed in code.
- New cpu.rt.max interface.
- Refactoring of migration code.
Updates from v4:
- Rebase to latest tip/master.
- General rebasing/cleanup.
- Update default sysctl_sched_rt_runtime to 1s, same as the period.
- Fix non-deferred deadline server replenishment logic.
- Add missing RCU read sections.
- Account HCBS servers along with their tasks when the servers are active.
- Release bandwidth resources early in unregister_rt_sched_group.
- Drop server_try_pull_task as it is now redundant.
- Remove dl_server_stop call in dequeue_task_rt.
- Update to reuse __checkparam_dl for deadline servers.
Updates from v3:
- Rebase to latest tip/master.
- General rebasing/cleanup.
- Add Documentation.
- Define **live** and **active** groups.
- Introduce server_try_pull_task in place of the removed server_has_task.
- Introduce RELEASE_LOCK helper macro for guard-based locking.
- Update inc/dec_dl_tasks to account for served runqueues regardless of the server type.
- Fix computing of new bandwidth values in dl_init_tg.
- Fix check in dl_check_tg to use capacity scaling.
- Fix wakeup_preempt_rt to check if curr is a DEADLINE task.
Updates from v2:
- Rebase to latest tip/master.
- Remove fair-servers' bw reclaiming.
- Fix a check which prevented execution of wakeup_preempt code.
- Fix a priority check in group_pull_rt_task between tasks of different groups.
- Rework allocation/deallocation code for rt-cgroups.
- Update signatures for some group related migration functions.
- Add documentation for wakeup_preempt preemption rules.
Updates from v1:
- Rebase to latest tip/master.
- Add migration code.
- Split big patches for more readability.
- Refactor code to use guarded locks where applicable.
- Remove unnecessary patches from v1 which have been addressed differently by mainline updates.
- Remove unnecessary checks and general code cleanup.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Testing v7:
The patchset has been tested with a suite of tests tailored to stress all the implemented functionalities.
The tests are available at https://github.com/Yurand2000/HCBS-Test-Suite . Refer to the README of the repository for more details.
Follow these steps to test HCBS v7:
- Get the HCBS patch up and running. Any kernel/disto should work effortlessly.
- Get, compile and _install_ the tests.
- Run the go_rt.sh script to set the frequency of the CPUs to a fixed value and disable hyperthreading and power saving features.
- Run the run_tests.sh full script, to run the whole test suite.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Future Work:
This latest RFC, along with all the new features, is stable on our limited hardware. The patchset is nonetheless potentially lacking testing of some specific code paths and features.
We surely expect comments on the cpuset integration and the dl_server override feature, in the hope to find a better solution for the current limitations.
There is still some pending issues to solve and work to do:
- PI-boosting will cause starvation if cgroups have zero (internal) runtime. There are many solutions, such as bandwidth inheritance or forcing cgroups to not have zero runtime, but this has yet to be investigated. We are open for advice on the matter.
- As Tejun Heo pointed out, it maybe be possible to remove the forceful global reservation of cgroup bandwidth in the root cgroup, while still being compliant with real-time schedulability theory and related SCHED_DEADLINE admission tests.
- There are some potential performance improvements in the migration code that are yet to be implemented (rto_masks).
- Integration with the old v1 cgroup? Maybe disallow HCBS/RT_GROUP_SCHED in cgroups-v1 entirely? There has also been ongoing discussion at
https://lore.kernel.org/all/[email protected]/
Have a nice day,
Yuri
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v5: https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/all/[email protected]/
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Yuri Andriaccio (18):
sched/deadline: Introduce dl_freq_invariant flag for fair/ext servers sched/rt: Disable RT_GROUP_SCHED
sched/rt: Remove unnecessary runqueue pointer in struct rt_rq sched/rt: Add {alloc/unregister/free}_rt_sched_group sched/rt: Implement dl-server operations for rt-cgroups. sched/rt: Update task event callbacks for HCBS scheduling sched/rt: Remove support for cgroups-v1
sched/rt: Update task's RT runqueue when switching scheduling class sched/rt: Add HCBS migration code to related functions sched/rt: Hook HCBS migration functions
sched/rt: Try pull task on empty server pick.
sched/core: Execute enqueued balance callbacks after
migrate_disable_switch
sched/syscalls: Allow unpriviledged tasks to set RT priority inside
cgroups.
sched/rt: Enable HRTICK_DL for HCBS servers.
Documentation: Update documentation for real-time cgroups sched/rt: Integrate cpuset and cpu controllers
sched/rt: Prevent CPU offline operation on active rt-cgroups. sched/rt: Introduce multi-cpu support
luca abeni (11):
sched/deadline: Do not access dl_se->rq directly sched/deadline: Distinguish between dl_rq and my_q sched/rt: Pass an rt_rq instead of an rq where needed sched/rt: Move functions from rt.c to sched.h
sched/rt: Introduce HCBS specific structs in task_group sched/core: Initialize HCBS specific structures. sched/deadline: Add dl_init_tg
sched/deadline: Account rt-cgroups bandwidth in deadline tasks
schedulability tests.
sched/rt: Update rt-cgroup schedulability checks sched/rt: Remove old RT_GROUP_SCHED data structures sched/core: Execute enqueued balance callbacks when changing allowed
CPUs
Documentation/scheduler/sched-rt-group.rst | 471 +++- include/linux/cpuset.h | 2 + include/linux/rcupdate.h | 1 + include/linux/sched.h | 16 +- kernel/cgroup/cpuset.c | 31 + kernel/sched/autogroup.c | 4 +- kernel/sched/core.c | 239 +- kernel/sched/deadline.c | 324 ++- kernel/sched/debug.c | 6 - kernel/sched/ext/ext.c | 4 +- kernel/sched/fair.c | 4 +- kernel/sched/rt.c | 2725 +++++++++++--------- kernel/sched/sched.h | 225 +- kernel/sched/syscalls.c | 37 +- 14 files changed, 2540 insertions(+), 1549 deletions(-)
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4 --
2.56.0
RFC PATCH v7 1/29 sched/deadline: Introduce dl_freq_invariant flag for fair/ext servers
The fair/ext servers are by design frequency and capacity invariant, i.e. their maximum allowed runtime is fixed, regardless of the frequency the CPU is
running. (See also https://lore.kernel.org/all/[email protected]/)
This inevitably broke non-fair dl_servers, which would want to instead scale their execution time to be consistent with CBS and GRUB-PA.
The newly added flag allows deadline servers to adopt the frequency invariance behaviour as fair/ext servers.
--- include/linux/sched.h | 4 ++++ kernel/sched/deadline.c | 5 ++++- 2 files changed, 8 insertions(+), 1 deletion(-)
include/linux/sched.h +4 -0
@@ -706,6 +706,9 @@ struct sched_dl_entity {* contributes to the root domain's total_bw. Only meaningful for server * entities (@dl_server == 1). Allows toggling the reservation on/off * without losing the configured @dl_runtime/@dl_period. +* +* @dl_freq_invariant tells if this server's runtime must NOT be scaled +* by the CPU's frequency and capacity. */ unsigned int dl_throttled : 1; unsigned int dl_yielded : 1;
@@ -718,6 +721,7 @@ struct sched_dl_entity { unsigned int dl_defer_running : 1; unsigned int dl_defer_idle : 1; unsigned int dl_bw_attached : 1; + unsigned int dl_freq_invariant : 1;/* * Bandwidth enforcement timer. Each -deadline task has its
kernel/sched/deadline.c +4 -1
@@ -1432,7 +1432,7 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64return ; scaled_delta_exec = delta_exec; -if (!dl_server (dl_se)) +if (!dl_server (dl_se) || !dl_se->dl_freq_invariant) scaled_delta_exec =dl_scaled_delta_exec (rq, dl_se, delta_exec); dl_se->runtime -= scaled_delta_exec;
@@ -1862,6 +1862,7 @@ void sched_init_dl_servers(void) dl_se->dl_server =1 ; dl_se->dl_defer =1 ; + dl_se->dl_freq_invariant =1 ;setup_new_dl_entity (dl_se);#ifdef CONFIG_SCHED_CLASS_EXT
@@ -1873,6 +1874,7 @@ void sched_init_dl_servers(void) dl_se->dl_server =1 ; dl_se->dl_defer =1 ; + dl_se->dl_freq_invariant =1 ;setup_new_dl_entity (dl_se);/*
@@ -3957,6 +3959,7 @@ static void __dl_clear_params(struct sched_dl_entity *dl_se) dl_se->dl_defer =0 ; dl_se->dl_defer_running =0 ; dl_se->dl_defer_armed =0 ; + dl_se->dl_freq_invariant =0 ;#ifdef CONFIG_RT_MUTEXES dl_se->pi_se = dl_se;
--
2.56.0
RFC PATCH v7 2/29 sched/deadline: Do not access dl_se->rq directly
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make deadline.c code access the runqueue of a scheduling entity saved in the sched_dl_entity data structure. This allows future patches to save different runqueues in sched_dl_entity other than the global runqueues.
Move dl_server_apply_params call in sched_init_dl_servers as the rq_of_dl_se function will return the correct deadline entity only if the dl_server flag is set.
Add a WARN_ON on the return value of dl_server_apply_params in sched_init_dl_servers as this function may fail if the kernel is not configured correctly.
--- kernel/sched/deadline.c | 46 ++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 23 deletions(-)
kernel/sched/deadline.c +23 -23
@@ -869,7 +869,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)* and arm the defer timer. */ if (dl_se->dl_defer && !dl_se->dl_defer_running && -dl_time_before (rq_clock (dl_se->rq), dl_se->deadline - dl_se->runtime)) { +dl_time_before (rq_clock (rq), dl_se->deadline - dl_se->runtime)) {if (!is_dl_boosted (dl_se)) {/*
@@ -1171,11 +1171,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_* of time. The dl_server_min_res serves as a limit to avoid * forwarding the timer for a too small amount of time. */ -if (dl_time_before (rq_clock (dl_se->rq), +if (dl_time_before (rq_clock (rq), (dl_se->deadline - dl_se->runtime - dl_server_min_res))) {/* reset the defer timer */ - fw = dl_se->deadline -rq_clock (dl_se->rq) - dl_se->runtime; + fw = dl_se->deadline -rq_clock (rq) - dl_se->runtime;hrtimer_forward_now (timer,ns_to_ktime (fw));return HRTIMER_RESTART ;
@@ -1186,7 +1186,7 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_enqueue_dl_entity (dl_se,ENQUEUE_REPLENISH ); -if (!dl_task (dl_se->rq->curr) ||dl_entity_preempt (dl_se, &dl_se->rq->curr->dl)) +if (!dl_task (rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl))resched_curr (rq);__push_dl_task (rq, rf);
@@ -1482,7 +1482,7 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64hrtimer_try_to_cancel (&dl_se->dl_timer); -replenish_dl_new_period (dl_se, dl_se->rq); +replenish_dl_new_period (dl_se, rq);if (idle) dl_se->dl_defer_idle =1 ;
@@ -1579,14 +1579,14 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64void dl_server_update_idle (struct sched_dl_entity *dl_se,s64 delta_exec) {if (dl_se->dl_server_active && dl_se->dl_runtime && dl_se->dl_defer) -update_curr_dl_se (dl_se->rq, dl_se, delta_exec); +update_curr_dl_se (rq_of_dl_se (dl_se), dl_se, delta_exec); }void dl_server_update (struct sched_dl_entity *dl_se,s64 delta_exec) {/* 0 runtime = fair server disabled */ if (dl_se->dl_server_active && dl_se->dl_runtime) -update_curr_dl_se (dl_se->rq, dl_se, delta_exec); +update_curr_dl_se (rq_of_dl_se (dl_se), dl_se, delta_exec); }/*
@@ -1795,7 +1795,7 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)*/ void dl_server_start (struct sched_dl_entity *dl_se) { -struct rq *rq = dl_se->rq; +struct rq *rq; dl_se->dl_defer_idle =0 ; if (!dl_server(dl_se) || dl_se->dl_server_active || !dl_se->dl_runtime ||
@@ -1805,16 +1805,16 @@ void dl_server_start(struct sched_dl_entity *dl_se)/* * Update the current task to 'now'. */ + rq =rq_of_dl_se (dl_se); rq->donor->sched_class->update_curr (rq); -if (WARN_ON_ONCE (!cpu_online (cpu_of (rq))))return ;trace_sched_dl_server_start_tp (dl_se,cpu_of (rq),dl_get_type (dl_se, rq)); dl_se->dl_server_active =1 ;enqueue_dl_entity (dl_se,ENQUEUE_WAKEUP ); -if (!dl_task (dl_se->rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl)) -resched_curr (dl_se->rq); +if (!dl_task (rq->curr) ||dl_entity_preempt (dl_se, &rq->curr->dl)) +resched_curr (rq); }void dl_server_stop (struct sched_dl_entity *dl_se)
@@ -1858,9 +1858,9 @@ void sched_init_dl_servers(void)WARN_ON (dl_server (dl_se)); -dl_server_apply_params (dl_se, runtime, period,1 ); - dl_se->dl_server =1 ; +WARN_ON (dl_server_apply_params (dl_se, runtime, period,1 )); + dl_se->dl_defer =1 ; dl_se->dl_freq_invariant =1 ;setup_new_dl_entity (dl_se);
@@ -1870,9 +1870,9 @@ void sched_init_dl_servers(void)WARN_ON (dl_server (dl_se)); -dl_server_apply_params (dl_se, runtime, period,1 ); - dl_se->dl_server =1 ; +WARN_ON (dl_server_apply_params (dl_se, runtime, period,1 )); + dl_se->dl_defer =1 ; dl_se->dl_freq_invariant =1 ;setup_new_dl_entity (dl_se);
@@ -1910,7 +1910,7 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 periou64 old_bw = (init || !dl_se->dl_bw_attached) ?0 :to_ratio (dl_se->dl_period, dl_se->dl_runtime);u64 new_bw =to_ratio (period, runtime); -struct rq *rq = dl_se->rq; +struct rq *rq =rq_of_dl_se (dl_se);int cpu =cpu_of (rq);struct dl_bw *dl_b;unsigned long cap;
@@ -1957,7 +1957,7 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 periostatic int __dl_server_attach_bw_locked (struct sched_dl_entity *dl_se,struct dl_bw *dl_b,int cpus) { -struct rq *rq = dl_se->rq; +struct rq *rq =rq_of_dl_se (dl_se);unsigned long cap;/*
@@ -1985,7 +1985,7 @@ static int __dl_server_attach_bw_locked(struct sched_dl_entity *dl_se,static void __dl_server_detach_bw_locked (struct sched_dl_entity *dl_se,struct dl_bw *dl_b,int cpus) { -struct rq *rq = dl_se->rq; +struct rq *rq =rq_of_dl_se (dl_se);/* * If the server is still active (on_rq), dequeue it via
@@ -2030,7 +2030,7 @@ static void __dl_server_detach_bw_locked(struct sched_dl_entity *dl_se,*/ int dl_server_attach_bw (struct sched_dl_entity *dl_se) { -struct rq *rq = dl_se->rq; +struct rq *rq =rq_of_dl_se (dl_se);int cpu =cpu_of (rq);struct dl_bw *dl_b;int cpus, ret;
@@ -2071,7 +2071,7 @@ int dl_server_attach_bw(struct sched_dl_entity *dl_se)*/ void dl_server_detach_bw (struct sched_dl_entity *dl_se) { -int cpu =cpu_of (dl_se->rq); +int cpu =cpu_of (rq_of_dl_se (dl_se));struct dl_bw *dl_b;int cpus;
@@ -2096,12 +2096,12 @@ void dl_server_detach_bw(struct sched_dl_entity *dl_se)int dl_server_swap_bw (struct sched_dl_entity *detach_se,struct sched_dl_entity *attach_se) { -struct rq *rq = detach_se->rq; +struct rq *rq =rq_of_dl_se (detach_se);int cpu =cpu_of (rq);struct dl_bw *dl_b;int cpus, ret; -WARN_ON_ONCE (attach_se->rq != rq); +WARN_ON_ONCE (rq_of_dl_se (attach_se) != rq); scoped_guard (raw_spinlock, &dl_bw_of(cpu)->lock) { dl_b =dl_bw_of (cpu);
@@ -2162,7 +2162,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer) p =dl_task_of (dl_se); rq =task_rq_lock (p, &rf); }else { - rq = dl_se->rq; + rq =rq_of_dl_se (dl_se);rq_lock (rq, &rf); }
Signature · 1 line
2.56.0
RFC PATCH v7 3/29 sched/deadline: Distinguish between dl_rq and my_q
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Split the single runqueue pointer in sched_dl_entity into two separate pointers, following the existing pattern used by sched_rt_entity:
- dl_rq: Points to the deadline runqueue where this entity is queued (global runqueue).
- my_q: Points to the runqueue that this entity serves (for servers).
This distinction is currently redundant for the fair_server and ext_servers (both point to the same CPU's structures), but is essential for future RT cgroup support where deadline servers will be queued on the global dl_rq while serving tasks from cgroup-specific runqueues.
Update rq_of_dl_se() to use container_of_const() to recover the global rq from dl_rq, and update fair.c and ext.c to explicitly use my_q (local rq) when accessing the served runqueue.
Update dl_server_init() to take a dl_rq pointer (use to retrieve the global runqueue where the dl_server is scheduled) and a rq pointer (for the local runqueue served by the server).
--- include/linux/sched.h | 6 ++++-- kernel/sched/deadline.c | 17 ++++++++++++----- kernel/sched/ext/ext.c | 4 ++-- kernel/sched/fair.c | 4 ++-- kernel/sched/sched.h | 3 ++- 5 files changed, 22 insertions(+), 12 deletions(-)
include/linux/sched.h +4 -2
@@ -742,9 +742,11 @@ struct sched_dl_entity {* Bits for DL-server functionality. Also see the comment near * dl_server_update(). * -* @rq the runqueue this server is for +* @dl_rq the runqueue on which this entity is (to be) queued +* @my_q the runqueue "owned" by this entity */ -struct rq *rq; +struct dl_rq *dl_rq; +struct rq *my_q;dl_server_pick_f server_pick_task;#ifdef CONFIG_RT_MUTEXES
kernel/sched/deadline.c +12 -5
@@ -65,10 +65,12 @@ static inline struct rq *rq_of_dl_rq(struct dl_rq *dl_rq)static inline struct rq *rq_of_dl_se (struct sched_dl_entity *dl_se) { -struct rq *rq = dl_se->rq; +struct rq *rq;if (!dl_server (dl_se)) rq =task_rq (dl_task_of (dl_se)); +else + rq = container_of_const(dl_se->dl_rq, struct rq, dl);return rq; }
@@ -1819,11 +1821,14 @@ void dl_server_start(struct sched_dl_entity *dl_se)void dl_server_stop (struct sched_dl_entity *dl_se) { +struct rq *rq; +if (!dl_server (dl_se) || !dl_server_active (dl_se))return ; -trace_sched_dl_server_stop_tp (dl_se,cpu_of (dl_se->rq), -dl_get_type (dl_se, dl_se->rq)); + rq =rq_of_dl_se (dl_se); +trace_sched_dl_server_stop_tp (dl_se,cpu_of (rq), +dl_get_type (dl_se, rq));dequeue_dl_entity (dl_se,DEQUEUE_SLEEP );hrtimer_try_to_cancel (&dl_se->dl_timer); dl_se->dl_defer_armed =0 ;
@@ -1832,10 +1837,12 @@ void dl_server_stop(struct sched_dl_entity *dl_se) dl_se->dl_server_active =0 ; } -void dl_server_init (struct sched_dl_entity *dl_se,struct rq *rq, +void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq, +struct rq *served_rq,dl_server_pick_f pick_task) { - dl_se->rq = rq; + dl_se->dl_rq = dl_rq; + dl_se->my_q = served_rq; dl_se->server_pick_task = pick_task; }
kernel/sched/ext/ext.c +2 -2
@@ -4099,7 +4099,7 @@ ext_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)if (!scx_enabled ())return NULL ; -return do_pick_task_scx (dl_se->rq, rf, true); +return do_pick_task_scx (dl_se->my_q, rf, true); }/*
@@ -4111,7 +4111,7 @@ void ext_server_init(struct rq *rq)init_dl_entity (dl_se); -dl_server_init (dl_se, rq, ext_server_pick_task); +dl_server_init (dl_se, &rq->dl, rq, ext_server_pick_task); }#ifdef CONFIG_SCHED_CORE
kernel/sched/fair.c +2 -2
@@ -10311,7 +10311,7 @@ static struct task_struct * fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf) __must_hold(__rq_lockp(dl_se->rq)) { - return pick_task_fair(dl_se->rq, rf); + return pick_task_fair(dl_se->my_q, rf); }void fair_server_init (struct rq *rq)
@@ -10320,7 +10320,7 @@ void fair_server_init(struct rq *rq)init_dl_entity (dl_se); -dl_server_init (dl_se, rq, fair_server_pick_task); +dl_server_init (dl_se, &rq->dl, rq, fair_server_pick_task); }/*
kernel/sched/sched.h +2 -1
@@ -412,7 +412,8 @@ extern void dl_server_update_idle(struct sched_dl_entity *dl_se, s64 delta_exec)extern void dl_server_update (struct sched_dl_entity *dl_se,s64 delta_exec);extern void dl_server_start (struct sched_dl_entity *dl_se);extern void dl_server_stop (struct sched_dl_entity *dl_se); -extern void dl_server_init (struct sched_dl_entity *dl_se,struct rq *rq, +extern void dl_server_init (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq, +struct rq *served_rq,dl_server_pick_f pick_task);extern void sched_init_dl_servers (void );
--
2.56.0
RFC PATCH v7 4/29 sched/rt: Pass an rt_rq instead of an rq where needed
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make rt.c code access the runqueue through the rt_rq data structure rather than passing an rq pointer directly. This allows future patches to define rt_rq data structures which do not refer only to the global runqueue, but also to local cgroup runqueues (as rt_rq will not be always equal to &rq->rt).
--- kernel/sched/rt.c | 174 +++++++++++++++++++++++++++------------------- 1 file changed, 104 insertions(+), 70 deletions(-)
kernel/sched/rt.c +104 -70
@@ -370,9 +370,9 @@ static inline void rt_clear_overload(struct rq *rq)cpumask_clear_cpu (rq->cpu, rq->rd->rto_mask); } -static inline int has_pushable_tasks (struct rq *rq) +static inline int has_pushable_tasks (struct rt_rq *rt_rq) { -return !plist_head_empty (&rq->rt.pushable_tasks); +return !plist_head_empty (&rt_rq->pushable_tasks); } static DEFINE_PER_CPU(struct balance_callback, rt_push_head);
@@ -381,50 +381,66 @@ static DEFINE_PER_CPU(struct balance_callback, rt_pull_head);static void push_rt_tasks (struct rq *);static void pull_rt_task (struct rq *); -static inline void rt_queue_push_tasks (struct rq *rq) +static inline void rt_queue_push_tasks (struct rt_rq *rt_rq) { -if (!has_pushable_tasks (rq)) + struct rq *rq = container_of_const(rt_rq, struct rq, rt); + +if (!has_pushable_tasks (rt_rq))return ;queue_balance_callback (rq, &per_cpu (rt_push_head, rq->cpu), push_rt_tasks); } -static inline void rt_queue_pull_task (struct rq *rq) +static inline void rt_queue_pull_task (struct rt_rq *rt_rq) { + struct rq *rq = container_of_const(rt_rq, struct rq, rt); +queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); } -static void enqueue_pushable_task (struct rq *rq,struct task_struct *p) +static void push_rt_rq_tasks (struct rt_rq *rt_rq); + +static void push_rt_tasks (struct rq *global_rq) { +push_rt_rq_tasks (&global_rq->rt); +} + +static void pull_rt_rq_task (struct rt_rq *this_rt_rq); + +static void pull_rt_task (struct rq *global_rq) { +pull_rt_rq_task (&global_rq->rt); +} + +static void enqueue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) { -plist_del (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);plist_node_init (&p->pushable_tasks, p->prio); -plist_add (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_add (&p->pushable_tasks, &rt_rq->pushable_tasks);/* Update the highest prio pushable task */ -if (p->prio < rq->rt.highest_prio.next) - rq->rt.highest_prio.next = p->prio; +if (p->prio < rt_rq->highest_prio.next) + rt_rq->highest_prio.next = p->prio; -if (!rq->rt.overloaded) { -rt_set_overload (rq); - rq->rt.overloaded =1 ; +if (!rt_rq->overloaded) { +rt_set_overload (rq_of_rt_rq (rt_rq)); + rt_rq->overloaded =1 ; } } -static void dequeue_pushable_task (struct rq *rq,struct task_struct *p) +static void dequeue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) { -plist_del (&p->pushable_tasks, &rq->rt.pushable_tasks); +plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);/* Update the new highest prio pushable task */ -if (has_pushable_tasks (rq)) { - p =plist_first_entry (&rq->rt.pushable_tasks, +if (has_pushable_tasks (rt_rq)) { + p =plist_first_entry (&rt_rq->pushable_tasks, struct task_struct, pushable_tasks); - rq->rt.highest_prio.next = p->prio; + rt_rq->highest_prio.next = p->prio; }else { - rq->rt.highest_prio.next =MAX_RT_PRIO -1 ; + rt_rq->highest_prio.next =MAX_RT_PRIO -1 ; -if (rq->rt.overloaded) { -rt_clear_overload (rq); - rq->rt.overloaded =0 ; +if (rt_rq->overloaded) { +rt_clear_overload (rq_of_rt_rq (rt_rq)); + rt_rq->overloaded =0 ; } } }
@@ -1436,6 +1452,7 @@ static void enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags) { struct sched_rt_entity *rt_se = &p->rt; +struct rt_rq *rt_rq =rt_rq_of_se (rt_se);if (flags &ENQUEUE_WAKEUP ) rt_se->timeout =0 ;
@@ -1449,17 +1466,18 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)return ;if (!task_current (rq, p) && p->nr_cpus_allowed >1 ) -enqueue_pushable_task (rq, p); +enqueue_pushable_task (rt_rq, p); }static bool dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) {struct sched_rt_entity *rt_se = &p->rt; +struct rt_rq *rt_rq =rt_rq_of_se (rt_se);update_curr_rt (rq);dequeue_rt_entity (rt_se, flags); -dequeue_pushable_task (rq, p); +dequeue_pushable_task (rt_rq, p);return true; }
@@ -1498,7 +1516,7 @@ static void yield_task_rt(struct rq *rq)requeue_task_rt (rq, rq->donor,0 ); } -static int find_lowest_rq (struct task_struct *task); +static int find_lowest_rt_rq (struct task_struct *task);static int select_task_rq_rt (struct task_struct *p,int cpu,int flags)
@@ -1548,7 +1566,7 @@ select_task_rq_rt(struct task_struct *p, int cpu, int flags) (curr->nr_cpus_allowed <2 || donor->prio <= p->prio);if (test || !rt_task_fits_capacity (p, cpu)) { -int target =find_lowest_rq (p); +int target =find_lowest_rt_rq (p);/* * Bail out if we were forcing a migration to find a better
@@ -1612,7 +1630,7 @@ static int balance_rt(struct rq *rq, struct rq_flags *rf)* not yet started the picking loop. */ rq_unpin_lock (rq, rf); -pull_rt_task (rq); +pull_rt_rq_task (&rq->rt);rq_repin_lock (rq, rf); }
@@ -1657,7 +1675,7 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)static inline void set_next_task_rt (struct rq *rq,struct task_struct *p,enum snt_e type) {struct sched_rt_entity *rt_se = &p->rt; -struct rt_rq *rt_rq = &rq->rt; +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt);if (type ==SNT_REPICK )return ;
@@ -1667,7 +1685,7 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, enum supdate_stats_wait_end_rt (rt_rq, rt_se);/* The running task is never eligible for pushing */ -dequeue_pushable_task (rq, p); +dequeue_pushable_task (rt_rq, p);if (type !=SNT_PICK )return ;
@@ -1680,7 +1698,7 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, enum sif (rq->donor->sched_class != &rt_sched_class)update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); -rt_queue_push_tasks (rq); +rt_queue_push_tasks (rt_rq); }static struct sched_rt_entity *pick_next_rt_entity (struct rt_rq *rt_rq)
@@ -1731,7 +1749,7 @@ static struct task_struct *pick_task_rt(struct rq *rq, struct rq_flags *rf)static void put_prev_task_rt (struct rq *rq,struct task_struct *p,struct task_struct *next) {struct sched_rt_entity *rt_se = &p->rt; -struct rt_rq *rt_rq = &rq->rt; +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt);if (on_rt_rq (&p->rt))update_stats_wait_start_rt (rt_rq, rt_se);
@@ -1747,7 +1765,7 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s* if it is still active */ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 ) -enqueue_pushable_task (rq, p); +enqueue_pushable_task (rt_rq, p); }/* Only try algorithms three times */
@@ -1757,16 +1775,16 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s* Return the highest pushable rq's task, which is suitable to be executed * on the CPU, NULL otherwise */ -static struct task_struct *pick_highest_pushable_task (struct rq *rq,int cpu) +static struct task_struct *pick_highest_pushable_task (struct rt_rq *rt_rq,int cpu) { -struct plist_head *head = &rq->rt.pushable_tasks; +struct plist_head *head = &rt_rq->pushable_tasks;struct task_struct *p; -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (rt_rq))return NULL ;plist_for_each_entry (p, head, pushable_tasks) { -if (task_is_pushable (rq, p, cpu)) +if (task_is_pushable (rq_of_rt_rq (rt_rq), p, cpu))return p; }
@@ -1775,7 +1793,7 @@ static struct task_struct *pick_highest_pushable_task(struct rq *rq, int cpu) static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask); -static int find_lowest_rq (struct task_struct *task) +static int find_lowest_rt_rq (struct task_struct *task) {struct sched_domain *sd;struct cpumask *lowest_mask =this_cpu_cpumask_var_ptr (local_cpu_mask);
@@ -1866,12 +1884,13 @@ static int find_lowest_rq(struct task_struct *task)return -1 ; } -static struct task_struct *pick_next_pushable_task (struct rq *rq) +static struct task_struct *pick_next_pushable_task (struct rt_rq *rt_rq) { -struct plist_head *head = &rq->rt.pushable_tasks; +struct rq *rq =rq_of_rt_rq (rt_rq); +struct plist_head *head = &rt_rq->pushable_tasks;struct task_struct *i, *p =NULL ; -if (!has_pushable_tasks (rq)) +if (!has_pushable_tasks (rt_rq))return NULL ;plist_for_each_entry (i, head, pushable_tasks) {
@@ -1897,14 +1916,15 @@ static struct task_struct *pick_next_pushable_task(struct rq *rq) }/* Will lock the rq it finds */ -static struct rq *find_lock_lowest_rq (struct task_struct *task,struct rq *rq) +static struct rt_rq *find_lock_lowest_rt_rq (struct task_struct *task,struct rt_rq *rt_rq) { +struct rq *rq =rq_of_rt_rq (rt_rq);struct rq *lowest_rq =NULL ;int tries;int cpu;for (tries =0 ; tries <RT_MAX_TRIES ; tries++) { - cpu =find_lowest_rq (task); + cpu =find_lowest_rt_rq (task);if ((cpu ==-1 ) || (cpu == rq->cpu))break ;
@@ -1935,7 +1955,7 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)*/ if (unlikely (is_migration_disabled (task) || !cpumask_test_cpu (lowest_rq->cpu, &task->cpus_mask) || - task !=pick_next_pushable_task (rq))) { + task !=pick_next_pushable_task (rt_rq))) {double_unlock_balance (rq, lowest_rq); lowest_rq =NULL ;
@@ -1952,7 +1972,16 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq) lowest_rq =NULL ; } -return lowest_rq; +return &lowest_rq->rt; +} + +static struct rq *find_lock_lowest_rq (struct task_struct *task,struct rq *rq) { +struct rt_rq *rt_rq =find_lock_lowest_rt_rq (task, &rq->rt); + +if (!rt_rq) +return NULL ; + +return rq_of_rt_rq (rt_rq); }/*
@@ -1960,16 +1989,17 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)* running task can migrate over to a CPU that is running a task * of lesser priority. */ -static int push_rt_task (struct rq *rq,bool pull) +static int push_rt_rq_task (struct rt_rq *rt_rq,bool pull) {struct task_struct *next_task; -struct rq *lowest_rq; +struct rq *lowest_rq, *rq =rq_of_rt_rq (rt_rq); +struct rt_rq *lowest_rt_rq;int ret =0 ; -if (!rq->rt.overloaded) +if (!rt_rq->overloaded)return 0 ; - next_task =pick_next_pushable_task (rq); + next_task =pick_next_pushable_task (rt_rq);if (!next_task)return 0 ;
@@ -1992,7 +2022,7 @@ static int push_rt_task(struct rq *rq, bool pull)return 0 ;/* -* Invoking find_lowest_rq() on anything but an RT task doesn't +* Invoking find_lowest_rt_rq() on anything but an RT task doesn't * make sense. Per the above priority check, curr has to * be of higher priority than next_task, so no need to * reschedule when bailing out.
@@ -2003,7 +2033,7 @@ static int push_rt_task(struct rq *rq, bool pull)if (rq->donor->sched_class != &rt_sched_class)return 0 ; - cpu =find_lowest_rq (rq->curr); + cpu =find_lowest_rt_rq (rq->curr);if (cpu ==-1 || cpu == rq->cpu)return 0 ;
@@ -2032,19 +2062,19 @@ static int push_rt_task(struct rq *rq, bool pull)/* We might release rq lock */ get_task_struct (next_task); -/* find_lock_lowest_rq locks the rq if found */ - lowest_rq =find_lock_lowest_rq (next_task, rq); -if (!lowest_rq) { +/* find_lock_lowest_rt_rq locks the rq if found */ + lowest_rt_rq =find_lock_lowest_rt_rq (next_task, rt_rq); +if (!lowest_rt_rq) {struct task_struct *task;/* -* find_lock_lowest_rq releases rq->lock +* find_lock_lowest_rt_rq releases rq->lock * so it is possible that next_task has migrated. * * We need to make sure that the task is still on the same * run-queue and is also still the next task eligible for * pushing. */ - task =pick_next_pushable_task (rq); + task =pick_next_pushable_task (rt_rq);if (task == next_task) {/* * The task hasn't migrated, and is still the next
@@ -2067,6 +2097,7 @@ static int push_rt_task(struct rq *rq, bool pull) goto retry; } + lowest_rq =rq_of_rt_rq (lowest_rt_rq);move_queued_task_locked (rq, lowest_rq, next_task);resched_curr (lowest_rq); ret =1 ;
@@ -2078,10 +2109,10 @@ static int push_rt_task(struct rq *rq, bool pull)return ret; } -static void push_rt_tasks (struct rq *rq) +static void push_rt_rq_tasks (struct rt_rq *rt_rq) { -/* push_rt_task will return true if it moved an RT */ -while (push_rt_task (rq, false)) +/* push_rt_rq_task will return true if it moved an RT */ +while (push_rt_rq_task (rt_rq, false)) ; }
@@ -2237,9 +2268,9 @@ void rto_push_irq_work_func(struct irq_work *work)* We do not need to grab the lock to check for has_pushable_tasks. * When it gets updated, a check is made if a push is possible. */ -if (has_pushable_tasks (rq)) { +if (has_pushable_tasks (&rq->rt)) {raw_spin_rq_lock (rq); -while (push_rt_task (rq, true)) +while (push_rt_rq_task (&rq->rt, true)) ;raw_spin_rq_unlock (rq); }
@@ -2261,11 +2292,13 @@ void rto_push_irq_work_func(struct irq_work *work) }#endif /* HAVE_RT_PUSH_IPI */ -static void pull_rt_task (struct rq *this_rq) +static void pull_rt_rq_task (struct rt_rq *this_rt_rq) { +struct rq *this_rq =rq_of_rt_rq (this_rt_rq);int this_cpu = this_rq->cpu, cpu;bool resched = false;struct task_struct *p, *push_task; +struct rt_rq *src_rt_rq;struct rq *src_rq;int rt_overload_count =rt_overloaded (this_rq);
@@ -2295,6 +2328,7 @@ static void pull_rt_task(struct rq *this_rq)continue ; src_rq =cpu_rq (cpu); + src_rt_rq = &src_rq->rt;/* * Don't bother taking the src_rq->lock if the next highest
@@ -2303,8 +2337,8 @@ static void pull_rt_task(struct rq *this_rq)* logically higher, the src_rq will push this task away. * And if its going logically lower, we do not care */ -if (src_rq->rt.highest_prio.next >= - this_rq->rt.highest_prio.curr) +if (src_rt_rq->highest_prio.next >= + this_rt_rq->highest_prio.curr)continue ;/*
@@ -2319,13 +2353,13 @@ static void pull_rt_task(struct rq *this_rq)* We can pull only a task, which is pushable * on its rq, and no others. */ - p =pick_highest_pushable_task (src_rq, this_cpu); + p =pick_highest_pushable_task (src_rt_rq, this_cpu);/* * Do we have an RT task that preempts * the to-be-scheduled task? */ -if (p && (p->prio < this_rq->rt.highest_prio.curr)) { +if (p && (p->prio < this_rt_rq->highest_prio.curr)) {WARN_ON (p == src_rq->curr);WARN_ON (!task_on_rq_queued (p));
@@ -2384,7 +2418,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) rq->donor->prio <= p->prio);if (need_to_push) -push_rt_tasks (rq); +push_rt_rq_tasks (rt_rq_of_se (&p->rt)); }/* Assumes rq->lock is held */
@@ -2425,7 +2459,7 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)if (!task_on_rq_queued (p) || rq->rt.rt_nr_running)return ; -rt_queue_pull_task (rq); +rt_queue_pull_task (rt_rq_of_se (&p->rt)); }void __init init_sched_rt_class (void )
@@ -2460,8 +2494,8 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)* then see if we can move to another run queue. */ if (task_on_rq_queued (p)) { -if (p->nr_cpus_allowed >1 && rq->rt.overloaded) -rt_queue_push_tasks (rq); +if (p->nr_cpus_allowed >1 &&rt_rq_of_se (&p->rt)->overloaded) +rt_queue_push_tasks (rt_rq_of_se (&p->rt));if (p->prio < rq->donor->prio &&cpu_online (cpu_of (rq)))resched_curr (rq); }
@@ -2486,7 +2520,7 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, u64 oldprio)* may need to pull tasks to this runqueue. */ if (oldprio < p->prio) -rt_queue_pull_task (rq); +rt_queue_pull_task (rt_rq_of_se (&p->rt));/* * If there's a higher priority task waiting to run
--
2.56.0
RFC PATCH v7 5/29 sched/rt: Move functions from rt.c to sched.h
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Make the following functions/macros be non-static and move them in sched.h, so that they can be also used in other source files:
- rt_entity_is_task()
- rt_task_of()
- rq_of_rt_rq()
- rt_rq_of_se()
- rq_of_rt_se()
There are no functional changes, apart from the use of container_of_const() instead of container_of() where applicable. This is needed by future patches.
--- kernel/sched/rt.c | 56 ------------------------------------------ kernel/sched/sched.h | 58 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 56 deletions(-)
kernel/sched/rt.c +0 -56
@@ -166,36 +166,6 @@ static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)hrtimer_cancel (&rt_b->rt_period_timer); } -#define rt_entity_is_task (rt_se) (!(rt_se)->my_q) - -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ -WARN_ON_ONCE (!rt_entity_is_task (rt_se)); - - return container_of(rt_se, struct task_struct, rt); -} - -static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) -{ -/* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */ -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -} - -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) -{ -WARN_ON (!rt_group_sched_enabled () && rt_se->rt_rq->tg != &root_task_group); -return rt_se->rt_rq; -} - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct rt_rq *rt_rq = rt_se->rt_rq; - -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -} -void unregister_rt_sched_group (struct task_group *tg) {if (!rt_group_sched_enabled ())
@@ -294,32 +264,6 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)#else /* !CONFIG_RT_GROUP_SCHED: */ -#define rt_entity_is_task (rt_se) (1) - -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) -{ - return container_of(rt_se, struct task_struct, rt); -} - -static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) -{ - return container_of(rt_rq, struct rq, rt); -} - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct task_struct *p =rt_task_of (rt_se); - -return task_rq (p); -} - -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) -{ -struct rq *rq =rq_of_rt_se (rt_se); - -return &rq->rt; -} -void unregister_rt_sched_group (struct task_group *tg) { }void free_rt_sched_group (struct task_group *tg) { }
kernel/sched/sched.h +58 -0
64 lines not on this page · raw message
Signature · 1 line
2.56.0
RFC PATCH v7 6/29 sched/rt: Disable RT_GROUP_SCHED
In reply to Yuri AndriaccioDisable the old RT_GROUP_SCHED scheduler. Note that this does not completely remove all the RT_GROUP_SCHED functionality, just unhooks it and removes most of the relevant functions. Some of the RT_GROUP_SCHED functions are kept because they will be adapted for the HCBS scheduling.
Most notably:
- Disable the initialization of the rt_bandwidth for group scheduling.
- Unhook any functionality for RT_GROUP_SCHED in normal rt.c code, leaving only non-group functionality.
- Remove group related field initialization in init_rt_rq().
- Remove all the unhooked (and so unused) functions from RT_GROUP_SCHED.
- Remove all allocation/deallocation code for rt-groups, always returning failure on allocation.
- Update inc/dec_rt_tasks active tasks' counters, as rt scheduling entities now only represent a single task, and not a group of tasks anymore.
- Remove unused rq_of_rt_se function.
--- kernel/sched/core.c | 6 - kernel/sched/deadline.c | 34 -- kernel/sched/debug.c | 6 - kernel/sched/rt.c | 857 ++-------------------------------------- kernel/sched/sched.h | 32 +- kernel/sched/syscalls.c | 13 - 6 files changed, 35 insertions(+), 913 deletions(-)
kernel/sched/core.c +0 -6
@@ -8948,11 +8948,6 @@ void __init sched_init(void)init_defrootdomain (); -#ifdef CONFIG_RT_GROUP_SCHED -init_rt_bandwidth (&root_task_group.rt_bandwidth, -global_rt_period (),global_rt_runtime ()); -#endif /* CONFIG_RT_GROUP_SCHED */ -#ifdef CONFIG_CGROUP_SCHED task_group_cache =KMEM_CACHE (task_group,0 );
@@ -9004,7 +8999,6 @@ void __init sched_init(void)* starts working after scheduler_running, which is not the case * yet. */ - rq->rt.rt_runtime =global_rt_runtime ();init_tg_rt_entry (&root_task_group, &rq->rt,NULL , i,NULL );#endif rq->next_class = &idle_sched_class;
kernel/sched/deadline.c +0 -34
@@ -1534,40 +1534,6 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64 }else {trace_sched_dl_update_tp (dl_se,cpu_of (rq),dl_get_type (dl_se, rq)); } - -/* -* The dl_server does not account for real-time workload because it -* is running fair work. -*/ -if (dl_se->dl_server) -return ; - -#ifdef CONFIG_RT_GROUP_SCHED -/* -* Because -- for now -- we share the rt bandwidth, we need to -* account our runtime there too, otherwise actual rt tasks -* would be able to exceed the shared quota. -* -* Account to the root rt group for now. -* -* The solution we're working towards is having the RT groups scheduled -* using deadline servers -- however there's a few nasties to figure -* out before that can happen. -*/ -if (rt_bandwidth_enabled ()) { -struct rt_rq *rt_rq = &rq->rt; - -raw_spin_lock (&rt_rq->rt_runtime_lock); -/* -* We'll let actual RT tasks worry about the overflow here, we -* have our own CBS to keep us inline; only account when RT -* bandwidth is relevant. -*/ -if (sched_rt_bandwidth_account (rt_rq)) - rt_rq->rt_time += delta_exec; -raw_spin_unlock (&rt_rq->rt_runtime_lock); - } -#endif /* CONFIG_RT_GROUP_SCHED */ }/*
kernel/sched/debug.c +0 -6
@@ -1144,12 +1144,6 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)PU (rt_nr_running); -#ifdef CONFIG_RT_GROUP_SCHED -P (rt_throttled); -PN (rt_time); -PN (rt_runtime); -#endif -#undef PN#undef PU#undef P
kernel/sched/rt.c +31 -826
1,057 lines not on this page · raw message
kernel/sched/sched.h +4 -28
@@ -872,7 +872,7 @@ struct scx_rq {static inline int rt_bandwidth_enabled (void ) { -return sysctl_sched_rt_runtime >=0 ; +return 0 ; }/* RT IPI pull logic requires IRQ_WORK */
@@ -912,7 +912,7 @@ struct rt_rq {static inline bool rt_rq_is_runnable (struct rt_rq *rt_rq) { -return rt_rq->rt_queued && rt_rq->rt_nr_running; +return rt_rq->rt_nr_running; }/* Deadline class' related fields in a runqueue */
@@ -2939,7 +2939,7 @@ static inline bool sched_dl_runnable(struct rq *rq)static inline bool sched_rt_runnable (struct rq *rq) { -return rq->rt.rt_queued >0 ; +return rq->rt.rt_nr_running >0 ; }static inline bool sched_fair_runnable (struct rq *rq)
@@ -3050,9 +3050,6 @@ extern void resched_curr(struct rq *rq);extern void resched_curr_lazy (struct rq *rq);extern void resched_cpu (int cpu); -extern void init_rt_bandwidth (struct rt_bandwidth *rt_b,u64 period,u64 runtime); -extern bool sched_rt_bandwidth_account (struct rt_rq *rt_rq); -extern void init_dl_entity (struct sched_dl_entity *dl_se);extern void init_cfs_throttle_work (struct task_struct *p);
@@ -3496,12 +3493,8 @@ extern void set_rq_offline(struct rq *rq);extern bool sched_smp_initialized;#ifdef CONFIG_RT_GROUP_SCHED -#define rt_entity_is_task (rt_se) (!(rt_se)->my_q) -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) { -WARN_ON_ONCE (!rt_entity_is_task (rt_se)); - return container_of(rt_se, struct task_struct, rt); }
@@ -3517,17 +3510,7 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)WARN_ON (!rt_group_sched_enabled () && rt_se->rt_rq->tg != &root_task_group);return rt_se->rt_rq; } - -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct rt_rq *rt_rq = rt_se->rt_rq; - -WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; -}#else -#define rt_entity_is_task (rt_se) (1) -static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) { return container_of(rt_se, struct task_struct, rt);
@@ -3538,16 +3521,9 @@ static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq) return container_of(rt_rq, struct rq, rt); } -static inline struct rq *rq_of_rt_se (struct sched_rt_entity *rt_se) -{ -struct task_struct *p =rt_task_of (rt_se); - -return task_rq (p); -} -static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) { -struct rq *rq =rq_of_rt_se (rt_se); +struct rq *rq =task_rq (rt_task_of (rt_se));return &rq->rt; }
kernel/sched/syscalls.c +0 -13
@@ -606,19 +606,6 @@ int __sched_setscheduler(struct task_struct *p, change:if (user) { -#ifdef CONFIG_RT_GROUP_SCHED -/* -* Do not allow real-time tasks into groups that have no runtime -* assigned. -*/ -if (rt_group_sched_enabled () && -rt_bandwidth_enabled () &&rt_policy (policy) && -task_group (p)->rt_bandwidth.rt_runtime ==0 && - !task_group_is_autogroup (task_group (p))) { - retval = -EPERM ; - goto unlock; - } -#endif /* CONFIG_RT_GROUP_SCHED */ if (dl_bandwidth_enabled () &&dl_policy (policy) && !(attr->sched_flags &SCHED_FLAG_SUGOV )) {cpumask_t *span = rq->rd->span;
Signature · 1 line
2.56.0
RFC PATCH v7 7/29 sched/rt: Remove unnecessary runqueue pointer in struct rt_rq
In reply to Yuri AndriaccioRemove the rq field in struct rt_rq.
The rq field now is just caching the pointer to the global runqueue of the given rt_rq, so it is unnecessary as the global runqueue can be retrieved in other ways.
Introduce global_rq_of_rt_rq to retrieve the global runqueue which serves a rt_rq's dl_server.
Rework rq_of_rt_rq to retrieve the runqueue a rt_rq is serving.
--- kernel/sched/rt.c | 21 +++++++++------------ kernel/sched/sched.h | 16 ++++++++++++---- 2 files changed, 21 insertions(+), 16 deletions(-)
kernel/sched/rt.c +9 -12
@@ -101,10 +101,7 @@ void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq, struct sched_rt_entity *rt_se, int cpu, struct sched_rt_entity *parent) { - struct rq *rq = cpu_rq(cpu); - rt_rq->highest_prio.curr = MAX_RT_PRIO-1; - rt_rq->rq = rq; rt_rq->tg = tg; tg->rt_rq[cpu] = rt_rq;
@@ -184,7 +181,7 @@ static void pull_rt_task(struct rq *);static inline void rt_queue_push_tasks (struct rt_rq *rt_rq) { - struct rq *rq = container_of_const(rt_rq, struct rq, rt); +struct rq *rq =global_rq_of_rt_rq (rt_rq);if (!has_pushable_tasks (rt_rq))return ;
@@ -194,7 +191,7 @@ static inline void rt_queue_push_tasks(struct rt_rq *rt_rq)static inline void rt_queue_pull_task (struct rt_rq *rt_rq) { - struct rq *rq = container_of_const(rt_rq, struct rq, rt); +struct rq *rq =global_rq_of_rt_rq (rt_rq);queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); }
@@ -222,7 +219,7 @@ static void enqueue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p) rt_rq->highest_prio.next = p->prio;if (!rt_rq->overloaded) { -rt_set_overload (rq_of_rt_rq (rt_rq)); +rt_set_overload (global_rq_of_rt_rq (rt_rq)); rt_rq->overloaded =1 ; } }
@@ -240,7 +237,7 @@ static void dequeue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p) rt_rq->highest_prio.next =MAX_RT_PRIO -1 ;if (rt_rq->overloaded) { -rt_clear_overload (rq_of_rt_rq (rt_rq)); +rt_clear_overload (global_rq_of_rt_rq (rt_rq)); rt_rq->overloaded =0 ; } }
@@ -495,7 +492,7 @@ update_stats_wait_start_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se)if (!stats)return ; -__update_stats_wait_start (rq_of_rt_rq (rt_rq), p, stats); +__update_stats_wait_start (global_rq_of_rt_rq (rt_rq), p, stats); } static inline void
@@ -512,7 +509,7 @@ update_stats_enqueue_sleeper_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_if (!stats)return ; -__update_stats_enqueue_sleeper (rq_of_rt_rq (rt_rq), p, stats); +__update_stats_enqueue_sleeper (global_rq_of_rt_rq (rt_rq), p, stats); } static inline void
@@ -540,7 +537,7 @@ update_stats_wait_end_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se)if (!stats)return ; -__update_stats_wait_end (rq_of_rt_rq (rt_rq), p, stats); +__update_stats_wait_end (global_rq_of_rt_rq (rt_rq), p, stats); } static inline void
@@ -564,11 +561,11 @@ update_stats_dequeue_rt(struct rt_rq *rt_rq, struct sched_rt_entity *rt_se, state =READ_ONCE (p->__state);if (state &TASK_INTERRUPTIBLE )__schedstat_set (p->stats.sleep_start, -rq_clock (rq_of_rt_rq (rt_rq))); +rq_clock (global_rq_of_rt_rq (rt_rq)));if (state &TASK_UNINTERRUPTIBLE )__schedstat_set (p->stats.block_start, -rq_clock (rq_of_rt_rq (rt_rq))); +rq_clock (global_rq_of_rt_rq (rt_rq))); } }
kernel/sched/sched.h +12 -4
@@ -902,8 +902,6 @@ struct rt_rq {raw_spinlock_t rt_runtime_lock;unsigned int rt_nr_boosted; - -struct rq *rq;/* this is always top-level rq, cache? */ #endif #ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */
@@ -3500,9 +3498,14 @@ static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)static inline struct rq *rq_of_rt_rq (struct rt_rq *rt_rq) { -/* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */ WARN_ON (!rt_group_sched_enabled () && rt_rq->tg != &root_task_group); -return rt_rq->rq; + return container_of_const(rt_rq, struct rq, rt); +} + +static inline struct rq *global_rq_of_rt_rq (struct rt_rq *rt_rq) +{ +/* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */ +return cpu_rq (rq_of_rt_rq (rt_rq)->cpu); }static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se)
@@ -3521,6 +3524,11 @@ static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq) return container_of(rt_rq, struct rq, rt); } +static inline struct rq *global_rq_of_rt_rq (struct rt_rq *rt_rq) +{ + return container_of_const(rt_rq, struct rq, rt); +} +static inline struct rt_rq *rt_rq_of_se (struct sched_rt_entity *rt_se) {struct rq *rq =task_rq (rt_task_of (rt_se));
--
2.56.0
RFC PATCH v7 8/29 sched/rt: Introduce HCBS specific structs in task_group
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add an array of sched_dl_entity objects in task_group.
Create the dl_bandwidth struct and add a field for it in task_group.
Add a rq pointer field in struct rt_rq.
---
For each CPU on the host system, the task_group manages a sched_dl_entity and a rt_rq object, which in turn keeps a pointer to its locally managed runqueue. The sched_dl_entity object manages the deadline server which will be scheduled for execution on the CPU, while the rt_rq object is instead used to reference the local runqueue's specific data and entities and it is used when an actual task must be scheduled when the CPU is given to the dl_server.
The dl_bandwidth object keeps track of the currently allocated bandwidth for the cgroup and the currently active context. RT-cgroups can either run tasks themselves or can delegate the scheduling of their tasks to their parent, the active_context field keeps track of which cgroup is serving the tasks.
--- kernel/sched/sched.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
kernel/sched/sched.h +22 -0
@@ -322,6 +322,15 @@ struct rt_bandwidth {unsigned int rt_period_active; }; +struct dl_bandwidth { +raw_spinlock_t dl_runtime_lock; +u64 dl_runtime; +u64 dl_internal_runtime; +u64 dl_period; +struct task_group *active_context; +}; + +static inline int dl_bandwidth_enabled (void ) {return sysctl_sched_rt_runtime >=0 ;
@@ -499,10 +508,17 @@ struct task_group {#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED +/* +* Each task group manages a different scheduling entity per CPU, i.e. a +* different deadline server, and a runqueue per CPU. All the dl-servers +* share the same dl_bandwidth object. +*/ struct sched_rt_entity **rt_se; +struct sched_dl_entity **dl_se;struct rt_rq **rt_rq;struct rt_bandwidth rt_bandwidth; +struct dl_bandwidth dl_bandwidth;#endif struct scx_task_group scx;
@@ -906,6 +922,12 @@ struct rt_rq {#ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */ #endif + +/* +* The cgroup's served runqueue if the rt_rq entity belongs to a cgroup, +* otherwise the top-level global runqueue. +*/ +struct rq *rq; };static inline bool rt_rq_is_runnable (struct rt_rq *rt_rq)
--
2.56.0
RFC PATCH v7 9/29 sched/core: Initialize HCBS specific structures.
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Update autogroups' creation/destruction to use the new data structures.
Initialize the default bandwidth for rt-cgroups (sched_init).
Initialize rt-scheduler's specific data structures for the root control group (sched_init).
Remove init_tg_rt_entry in favour of manual setup of the necessary data structures in sched_init.
Add utility functions to check (and get) if a rt_rq entity is connected to a rt-cgroup.
Add read/write accessors for dl_bandwidth.
Add dl_bw_lock_of_tg macro to reference the a task group dl_bandwidth's spinlock.
--- kernel/sched/autogroup.c | 4 ++-- kernel/sched/core.c | 11 ++++++++-- kernel/sched/deadline.c | 11 ++++++++++ kernel/sched/rt.c | 45 ++++++++++++++++++++++++++++------------ kernel/sched/sched.h | 38 ++++++++++++++++++++++++++++++--- 5 files changed, 89 insertions(+), 20 deletions(-)
kernel/sched/autogroup.c +2 -2
@@ -52,7 +52,7 @@ static inline void autogroup_destroy(struct kref *kref)#ifdef CONFIG_RT_GROUP_SCHED /* We've redirected RT tasks to the root task group... */ - ag->tg->rt_se =NULL ; + ag->tg->dl_se =NULL ; ag->tg->rt_rq =NULL ;#endif sched_release_group (ag->tg);
@@ -109,7 +109,7 @@ static inline struct autogroup *autogroup_create(void)* the policy change to proceed. */ free_rt_sched_group (tg); - tg->rt_se = root_task_group.rt_se; + tg->dl_se = root_task_group.dl_se; tg->rt_rq = root_task_group.rt_rq;#endif /* CONFIG_RT_GROUP_SCHED */ tg->autogroup = ag;
kernel/sched/core.c +9 -2
@@ -8938,7 +8938,7 @@ void __init sched_init(void)#ifdef CONFIG_RT_GROUP_SCHED ptr +=2 * nr_cpu_ids *sizeof (void **); ptr = (unsigned long )kzalloc (ptr,GFP_NOWAIT ); - root_task_group.rt_se = (struct sched_rt_entity **)ptr; + root_task_group.dl_se = (struct sched_dl_entity **)ptr; ptr += nr_cpu_ids *sizeof (void **); root_task_group.rt_rq = (struct rt_rq **)ptr;
@@ -8948,6 +8948,11 @@ void __init sched_init(void)init_defrootdomain (); +#ifdef CONFIG_RT_GROUP_SCHED +init_dl_bandwidth (&root_task_group.dl_bandwidth, +global_rt_period (),0 , &root_task_group); +#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_CGROUP_SCHED task_group_cache =KMEM_CACHE (task_group,0 );
@@ -8999,7 +9004,9 @@ void __init sched_init(void)* starts working after scheduler_running, which is not the case * yet. */ -init_tg_rt_entry (&root_task_group, &rq->rt,NULL , i,NULL ); + rq->rt.tg = &root_task_group; + root_task_group.rt_rq[i] = &rq->rt; + root_task_group.dl_se[i] =NULL ;#endif rq->next_class = &idle_sched_class;
kernel/sched/deadline.c +11 -0
@@ -508,6 +508,17 @@ static inline int is_leftmost(struct sched_dl_entity *dl_se, struct dl_rq *dl_rqstatic void init_dl_rq_bw_ratio (struct dl_rq *dl_rq); +void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime, +struct task_group *active_context) +{ +raw_spin_lock_init (&dl_b->dl_runtime_lock); + dl_b->dl_period = period; + dl_b->dl_runtime = runtime; + dl_b->dl_internal_runtime =0 ; + dl_b->active_context = active_context; +} + +void init_dl_bw (struct dl_bw *dl_b) {raw_spin_lock_init (&dl_b->lock);
kernel/sched/rt.c +32 -13
66 lines not on this page · raw message
kernel/sched/sched.h +35 -3
@@ -598,9 +598,6 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);extern void unthrottle_cfs_rq (struct cfs_rq *cfs_rq);extern bool cfs_task_bw_constrained (struct task_struct *p); -extern void init_tg_rt_entry (struct task_group *tg,struct rt_rq *rt_rq, -struct sched_rt_entity *rt_se,int cpu, -struct sched_rt_entity *parent);extern int sched_group_set_rt_runtime (struct task_group *tg,long rt_runtime_us);extern int sched_group_set_rt_period (struct task_group *tg,u64 rt_period_us);extern long sched_group_rt_runtime (struct task_group *tg);
@@ -3070,6 +3067,8 @@ extern void resched_curr(struct rq *rq);extern void resched_curr_lazy (struct rq *rq);extern void resched_cpu (int cpu); +extern void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime, +struct task_group *active_context);extern void init_dl_entity (struct sched_dl_entity *dl_se);extern void init_cfs_throttle_work (struct task_struct *p);
@@ -3512,6 +3511,9 @@ extern void set_rq_offline(struct rq *rq);extern bool sched_smp_initialized; +extern const struct dl_bandwidth *dl_bandwidth_read (struct task_group *tg); +extern struct dl_bandwidth *dl_bandwidth_write (struct task_group *tg); +#ifdef CONFIG_RT_GROUP_SCHED static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) {
@@ -3535,6 +3537,24 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)WARN_ON (!rt_group_sched_enabled () && rt_se->rt_rq->tg != &root_task_group); return rt_se->rt_rq; } + +static inline int is_dl_group (struct rt_rq *rt_rq) +{ +return rt_rq->tg != &root_task_group; +} + +/* +* Return the scheduling entity of this group of tasks. +*/ +static inline struct sched_dl_entity *dl_group_of (struct rt_rq *rt_rq) +{ +if (WARN_ON_ONCE (!is_dl_group (rt_rq))) +return NULL ; + +return rt_rq->tg->dl_se[rq_of_rt_rq (rt_rq)->cpu]; +} + +#define dl_bw_lock_of_tg (tg) (&(tg)->dl_bandwidth.dl_runtime_lock)#else static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se) {
@@ -3557,6 +3577,18 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)return &rq->rt; } + +static inline int is_dl_group (struct rt_rq *rt_rq) +{ +return 0 ; +} + +static inline struct sched_dl_entity *dl_group_of (struct rt_rq *rt_rq) +{ +return NULL ; +} + +#define dl_bw_lock_of_tg (tg) ((raw_spinlock_t*)NULL)#endif DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq,
Signature · 1 line
2.56.0
RFC PATCH v7 10/29 sched/deadline: Add dl_init_tg
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Add dl_init_tg to initialize and/or update a rt-cgroup dl_server and to also account the allocated bandwidth. This function is currently unhooked and will be later used to allocate bandwidth to rt-cgroups.
Add lock guard for raw_spin_rq_lock_irq for cleaner code.
--- kernel/sched/deadline.c | 31 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 2 files changed, 32 insertions(+)
kernel/sched/deadline.c +31 -0
@@ -335,6 +335,37 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)cancel_dl_timer (dl_se, &dl_se->inactive_timer); } +#ifdef CONFIG_RT_GROUP_SCHED +void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period) +{ + struct rq *rq = container_of_const(dl_se->dl_rq, struct rq, dl); +int is_active; +u64 new_bw; + +guard (rq_lock_irq)(rq); + is_active = dl_se->my_q->rt.rt_nr_running >0 ; + +update_rq_clock (rq); +dl_server_stop (dl_se); + + new_bw =to_ratio (rt_period, rt_runtime); +dl_rq_change_utilization (rq, dl_se, new_bw); + + dl_se->dl_runtime = rt_runtime; + dl_se->dl_deadline = rt_period; + dl_se->dl_period = rt_period; + + dl_se->runtime =0 ; + dl_se->deadline =0 ; + + dl_se->dl_bw = new_bw; + dl_se->dl_density = new_bw; + +if (is_active) +dl_server_start (dl_se); +} +#endif +static void dl_change_utilization (struct task_struct *p,u64 new_bw) {WARN_ON_ONCE (p->dl.flags &SCHED_FLAG_SUGOV );
kernel/sched/sched.h +1 -0
@@ -425,6 +425,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,struct rq *served_rq, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void ); +extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period);extern void fair_server_init (struct rq *rq);extern void ext_server_init (struct rq *rq);
Signature · 1 line
2.56.0
RFC PATCH v7 11/29 sched/rt: Add {alloc/unregister/free}_rt_sched_group
In reply to Yuri AndriaccioAdd allocation and deallocation code for rt-cgroups.
Declare dl_server specific functions (only skeleton, but no implementation yet), needed by the deadline servers to be called when trying to schedule.
Initialize a cgroup's active context to that of its parent.
--- kernel/sched/rt.c | 156 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 154 insertions(+), 2 deletions(-)
kernel/sched/rt.c +154 -2
@@ -120,24 +120,176 @@ struct dl_bandwidth *dl_bandwidth_write(struct task_group *tg)void unregister_rt_sched_group (struct task_group *tg) { +int i; + +if (!rt_group_sched_enabled ()) +return ; + +if (!tg->dl_se || !tg->rt_rq) +return ; +for_each_possible_cpu (i) { +if (!tg->dl_se[i] || !tg->rt_rq[i]) +continue ; + +if (tg->dl_se[i]->dl_runtime) +dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); + } }void free_rt_sched_group (struct task_group *tg) { +int i; +if (!rt_group_sched_enabled ())return ; + +if (!tg->dl_se || !tg->rt_rq) +return ; + +for_each_possible_cpu (i) { +if (!tg->dl_se[i] || !tg->rt_rq[i]) +continue ; + +/* +* Shutdown the dl_server and free it +* +* Since the dl timer is going to be cancelled, +* we risk to never decrease the running bw... +* Fix this issue by changing the group runtime +* to 0 immediately before freeing it. +*/ +if (tg->dl_se[i]->dl_runtime) +dl_init_tg (tg->dl_se[i],0 , tg->dl_se[i]->dl_period); + +hrtimer_cancel (&tg->dl_se[i]->dl_timer); +hrtimer_cancel (&tg->dl_se[i]->inactive_timer); +kfree (tg->dl_se[i]); + +/* Free the local per-cpu runqueue */ +kfree (rq_of_rt_rq (tg->rt_rq[i])); + } + +kfree (tg->rt_rq); +kfree (tg->dl_se); } +static inline void __rt_rq_free (struct rt_rq **rt_rq) +{ +int i; + +for_each_possible_cpu (i) { +if (rt_rq[i]) +kfree (rq_of_rt_rq (rt_rq[i])); + } + +kfree (rt_rq); +} + +DEFINE_FREE(rt_rq_free, struct rt_rq **, if (_T) __rt_rq_free(_T)) + +static inline void __dl_se_free (struct sched_dl_entity **dl_se) +{ +int i; + +for_each_possible_cpu (i) { +kfree (dl_se[i]); + } + +kfree (dl_se); +} + +DEFINE_FREE(dl_se_free, struct sched_dl_entity **, if (_T) __dl_se_free(_T)) + +static int __alloc_rt_sched_group_data (struct task_group *tg) { +/* Instantiate automatic cleanup in event of kalloc fail */ + struct rt_rq **tg_rt_rq __free(rt_rq_free) = NULL; + struct sched_dl_entity **tg_dl_se __free(dl_se_free) = NULL; + struct sched_dl_entity *dl_se __free(kfree) = NULL; + struct rq *s_rq __free(kfree) = NULL; +int i; + + tg_rt_rq =kcalloc (nr_cpu_ids,sizeof (struct rt_rq *),GFP_KERNEL ); +if (!tg_rt_rq) +return 0 ; + + tg_dl_se =kcalloc (nr_cpu_ids, +sizeof (struct sched_dl_entity *),GFP_KERNEL ); +if (!tg_dl_se) +return 0 ; + +for_each_possible_cpu (i) { + s_rq =kzalloc_node (sizeof (struct rq ), +GFP_KERNEL ,cpu_to_node (i)); +if (!s_rq) +return 0 ; + + dl_se =kzalloc_node (sizeof (struct sched_dl_entity ), +GFP_KERNEL ,cpu_to_node (i)); +if (!dl_se) +return 0 ; + + tg_rt_rq[i] = &no_free_ptr (s_rq)->rt; + tg_dl_se[i] =no_free_ptr (dl_se); + } + + tg->rt_rq =no_free_ptr (tg_rt_rq); + tg->dl_se =no_free_ptr (tg_dl_se); + +return 1 ; +} + +static struct task_struct *rt_server_pick (struct sched_dl_entity *dl_se,struct rq_flags *rf); +int alloc_rt_sched_group (struct task_group *tg,struct task_group *parent) { +struct sched_dl_entity *dl_se; +struct rq *s_rq; +int i; +if (!rt_group_sched_enabled ())return 1 ; +/* Allocate all necessary resources beforehand */ +if (!__alloc_rt_sched_group_data (tg)) +return 0 ; + +/* Initialize the allocated resources now. */ + scoped_guard(raw_spinlock_irq, dl_bw_lock_of_tg(parent)) { +init_dl_bandwidth (&tg->dl_bandwidth,0 ,RUNTIME_INF , +dl_bandwidth_read (parent)->active_context); + } + +for_each_possible_cpu (i) { + s_rq =rq_of_rt_rq (tg->rt_rq[i]); + dl_se = tg->dl_se[i]; + +init_rt_rq (&s_rq->rt); + s_rq->cpu = i; + s_rq->rt.tg = tg; + +init_dl_entity (dl_se); + dl_se->dl_runtime =0 ; + dl_se->dl_deadline =0 ; + dl_se->dl_period =0 ; + dl_se->runtime =0 ; + dl_se->deadline =0 ; + dl_se->dl_bw =to_ratio (dl_se->dl_period, dl_se->dl_runtime); + dl_se->dl_density =to_ratio (dl_se->dl_deadline, dl_se->dl_runtime); + dl_se->dl_server =1 ; + dl_se->dl_bw_attached =1 ; +dl_server_init (dl_se, &cpu_rq (i)->dl, s_rq, rt_server_pick); + } +return 1 ; } -#else /* !CONFIG_RT_GROUP_SCHED: */ +static struct task_struct *rt_server_pick (struct sched_dl_entity *dl_se,struct rq_flags *rf) +{ +return NULL ; +} + +#else /* !CONFIG_RT_GROUP_SCHED */ void unregister_rt_sched_group (struct task_group *tg) { }
@@ -147,7 +299,7 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent) {return 1 ; } -#endif /* !CONFIG_RT_GROUP_SCHED */ +#endif /* CONFIG_RT_GROUP_SCHED */ static inline bool need_pull_rt_task (struct rq *rq,struct task_struct *prev) {
--
2.56.0
RFC PATCH v7 12/29 sched/deadline: Account rt-cgroups bandwidth in deadline tasks schedulability tests.
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Account the rt-cgroups hierarchy's reserved bandwidth in the schedulability test of deadline entities. This mechanism allows to completely reserve portion of the rt-bandwidth to rt-cgroups even if they do not use all of it.
Account for the rt-cgroups' reserved bandwidth also when changing the total dedicated bandwidth for real time tasks.
--- kernel/sched/deadline.c | 58 +++++++++++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 17 deletions(-)
kernel/sched/deadline.c +41 -17
@@ -127,14 +127,17 @@ static inline struct dl_bw *dl_bw_of(int i)return &cpu_rq (i)->rd->dl_bw; } -static inline int dl_bw_cpus (int i) +static inline int __dl_bw_cpus (const struct cpumask *mask) { -struct root_domain *rd =cpu_rq (i)->rd; +return cpumask_weight_and (mask, cpu_active_mask); +} +static inline int dl_bw_cpus (int i) +{RCU_LOCKDEP_WARN (!rcu_read_lock_sched_held (),"sched RCU must be held" ); -return cpumask_weight_and (rd->span, cpu_active_mask); +return __dl_bw_cpus (cpu_rq (i)->rd->span); }static inline unsigned long __dl_bw_capacity (const struct cpumask *mask)
@@ -205,11 +208,25 @@ void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)__dl_update (dl_b, -((s32 )tsk_bw / cpus)); } +static inline u64 get_dl_groups_bw (void ) +{ +#ifdef CONFIG_RT_GROUP_SCHED +const struct dl_bandwidth *dl_b; + +guard (raw_spinlock_irqsave)(dl_bw_lock_of_tg (&root_task_group)); + dl_b =dl_bandwidth_read (&root_task_group); +return to_ratio (dl_b->dl_period, dl_b->dl_runtime); +#else +return 0 ; +#endif +} +static inline bool -__dl_overflow (struct dl_bw *dl_b,unsigned long cap,u64 old_bw,u64 new_bw) +__dl_overflow (struct dl_bw *dl_b,unsigned long cap,int cpus,u64 old_bw,u64 new_bw) {return dl_b->bw !=-1 && -cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw; +cap_scale (dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw + +get_dl_groups_bw () * cpus; } static inline
@@ -1937,7 +1954,7 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio cpus =dl_bw_cpus (cpu); cap =dl_bw_capacity (cpu); -if (__dl_overflow (dl_b, cap, old_bw, new_bw)) +if (__dl_overflow (dl_b, cap, cpus, old_bw, new_bw))return -EBUSY ;if (init) {
@@ -1984,7 +2001,7 @@ static int __dl_server_attach_bw_locked(struct sched_dl_entity *dl_se,*/ if (cpu_active (cpu_of (rq))) { cap =dl_bw_capacity (cpu_of (rq)); -if (__dl_overflow (dl_b, cap,0 , dl_se->dl_bw)) +if (__dl_overflow (dl_b, cap, cpus,0 , dl_se->dl_bw))return -EBUSY ;__dl_add (dl_b, dl_se->dl_bw, cpus); }
@@ -3710,8 +3727,9 @@ int sched_dl_global_validate(void)u64 period =global_rt_period ();u64 new_bw =to_ratio (period, runtime);u64 cookie = ++dl_cookie; +u64 dl_groups_root =get_dl_groups_bw ();struct dl_bw *dl_b; -int cpu, cpus, ret =0 ; +int cpu, cap, cpus, ret =0 ;unsigned long flags;/*
@@ -3726,10 +3744,12 @@ int sched_dl_global_validate(void) goto next; dl_b =dl_bw_of (cpu); + cap =dl_bw_capacity (cpu); cpus =dl_bw_cpus (cpu);raw_spin_lock_irqsave (&dl_b->lock, flags); -if (new_bw * cpus < dl_b->total_bw) +if (cap_scale (new_bw, cap) < dl_b->total_bw + + dl_groups_root * cpus) ret = -EBUSY ;raw_spin_unlock_irqrestore (&dl_b->lock, flags);
@@ -3823,13 +3843,13 @@ int sched_dl_overflow(struct task_struct *p, int policy, cap =dl_bw_capacity (cpu);if (dl_policy (policy) && !task_has_dl_policy (p) && - !__dl_overflow (dl_b, cap,0 , new_bw)) { + !__dl_overflow (dl_b, cap, cpus,0 , new_bw)) {if (hrtimer_active (&p->dl.inactive_timer))__dl_sub (dl_b, p->dl.dl_bw, cpus);__dl_add (dl_b, new_bw, cpus); err =0 ; }else if (dl_policy (policy) &&task_has_dl_policy (p) && - !__dl_overflow (dl_b, cap, p->dl.dl_bw, new_bw)) { + !__dl_overflow (dl_b, cap, cpus, p->dl.dl_bw, new_bw)) {/* * XXX this is slightly incorrect: when the task * utilization decreases, we should delay the total
@@ -4007,13 +4027,14 @@ int dl_cpuset_cpumask_can_shrink(const struct cpumask *cur, {unsigned long flags, cap;struct dl_bw *cur_dl_b; -int ret =1 ; +int cpus, ret =1 ;rcu_read_lock_sched (); cur_dl_b =dl_bw_of (cpumask_any (cur)); cap =__dl_bw_capacity (trial); + cpus =__dl_bw_cpus (trial);raw_spin_lock_irqsave (&cur_dl_b->lock, flags); -if (__dl_overflow (cur_dl_b, cap,0 ,0 )) +if (__dl_overflow (cur_dl_b, cap, cpus,0 ,0 )) ret =0 ;raw_spin_unlock_irqrestore (&cur_dl_b->lock, flags);rcu_read_unlock_sched ();
@@ -4030,6 +4051,7 @@ enum dl_bw_request {static int dl_bw_manage (enum dl_bw_request req,int cpu,u64 dl_bw) {unsigned long flags, cap; +int cpus;struct dl_bw *dl_b;bool overflow =0 ;u64 dl_server_bw =0 ;
@@ -4039,12 +4061,13 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)raw_spin_lock_irqsave (&dl_b->lock, flags); cap =dl_bw_capacity (cpu); + cpus =dl_bw_cpus (cpu);switch (req) {case dl_bw_req_free:__dl_sub (dl_b, dl_bw,dl_bw_cpus (cpu));break ;case dl_bw_req_alloc: - overflow =__dl_overflow (dl_b, cap,0 , dl_bw); + overflow =__dl_overflow (dl_b, cap, cpus,0 , dl_bw);if (!overflow) {/*
@@ -4053,7 +4076,7 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)* We will free resources in the source root_domain * later on (see set_cpus_allowed_dl()). */ -__dl_add (dl_b, dl_bw,dl_bw_cpus (cpu)); +__dl_add (dl_b, dl_bw, cpus); }break ;case dl_bw_req_deactivate:
@@ -4063,6 +4086,7 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)* turn cpu off?). */ cap -=arch_scale_cpu_capacity (cpu); + cpus -=1 ;/* * cpu is going offline and NORMAL and EXT tasks will be
@@ -4083,8 +4107,8 @@ static int dl_bw_manage(enum dl_bw_request req, int cpu, u64 dl_bw)* wise thing to do. As said above, cpu is not offline * yet, so account for that. */ -if (dl_bw_cpus (cpu) -1 ) - overflow =__dl_overflow (dl_b, cap, dl_server_bw,0 ); +if (cpus) + overflow =__dl_overflow (dl_b, cap, cpus, dl_server_bw,0 );else overflow =1 ; }
--
2.56.0
RFC PATCH v7 13/29 sched/rt: Implement dl-server operations for rt-cgroups.
In reply to Yuri AndriaccioImplement rt_server_pick, the callback that deadline servers use to pick a task to schedule.
rt_server_pick(): pick the next runnable rt task and tell the scheduler that it is going to be scheduled next.
Let enqueue_task_rt function start the attached deadline server when the first task is enqueued on a specific rq/server.
The server is not symmetrically stopped in dequeue_task_rt as it is stopped when server_pick_task returns NULL (see deadline.c).
Change update_curr_rt to perform a deadline server update if the updated task is served by non-root group.
Update {enqueue/dequeue}_pushable_task and rt_{set/clear}_overload to only set the CPU-wise overload flag only if the root runqueues are overloaded, but not for HCBS runqueues.
Update inc/dec_dl_tasks to account the number of active tasks in the local runqueue for rt-cgroups servers, as their local runqueue is different from the global runqueue, and thus when a rt-group server is activated/deactivated, the number of served tasks' must be added/removed. This uses nr_running to be compatible with future dl-server interfaces. Account also the deadline server so that it is picked for shutdown when its runqueue is empty (future patches will try to pull tasks before stopping).
Update inc/dec_rt_prio_smp to change a rq's cpupri only if the rt_rq is the global runqueue, since cgroups are scheduled via their dl-server priority.
Update inc/dec_rt_tasks to account for waking/sleeping tasks on the global runqueue, when the task runs on the root cgroup, or its local dl server is active. The accounting is not done when servers are throttled, as they will add/sub the number of tasks running when they get enqueued/dequeued. For rt cgroups, account for the number of active tasks in the nr_running field of the local runqueue (add/sub_nr_running), as this number is used when a dl server is enqueued/dequeued.
Update set_task_rq to record the rt_rq of the cgroup's active_context, tracking where to schedule the given task.
Update set_task_rq to record the dl_rq, tracking which deadline server manages a task.
Update set_task_rq to not use the parent field anymore, as it is unused by this patchset's code. Remove the unused parent field from sched_rt_entity.
--- include/linux/sched.h | 1 - kernel/sched/deadline.c | 8 +++++ kernel/sched/rt.c | 70 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 17 +++++----- 4 files changed, 80 insertions(+), 16 deletions(-)
include/linux/sched.h +0 -1
@@ -629,7 +629,6 @@ struct sched_rt_entity {struct sched_rt_entity *back;#ifdef CONFIG_RT_GROUP_SCHED -struct sched_rt_entity *parent;/* rq on which this entity is (to be) queued: */ struct rt_rq *rt_rq;/* rq "owned" by this entity/group: */
kernel/sched/deadline.c +8 -0
@@ -2293,6 +2293,10 @@ void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)if (!dl_server (dl_se))add_nr_running (rq_of_dl_rq (dl_rq),1 ); +else if (rq_of_dl_se (dl_se) != dl_se->my_q) { +WARN_ON (dl_se->my_q->rt.rt_nr_running != dl_se->my_q->nr_running); +add_nr_running (rq_of_dl_rq (dl_rq), dl_se->my_q->nr_running +1 ); + }inc_dl_deadline (dl_rq, deadline); }
@@ -2305,6 +2309,10 @@ void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)if (!dl_server (dl_se))sub_nr_running (rq_of_dl_rq (dl_rq),1 ); +else if (rq_of_dl_se (dl_se) != dl_se->my_q) { +WARN_ON (dl_se->my_q->rt.rt_nr_running != dl_se->my_q->nr_running); +sub_nr_running (rq_of_dl_rq (dl_rq), dl_se->my_q->nr_running +1 ); + }dec_dl_deadline (dl_rq, dl_se->deadline); }
kernel/sched/rt.c +62 -8
156 lines not on this page · raw message
kernel/sched/sched.h +10 -7
@@ -2427,15 +2427,12 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)#endif #ifdef CONFIG_RT_GROUP_SCHED -/* -* p->rt.rt_rq is NULL initially and it is easier to assign -* root_task_group's rt_rq than switching in rt_rq_of_se() -* Clobbers tg(!) -*/ if (!rt_group_sched_enabled ()) tg = &root_task_group; - p->rt.rt_rq = tg->rt_rq[cpu]; - p->rt.parent = tg->rt_se[cpu]; + +guard (raw_spinlock_irqsave)(&tg->dl_bandwidth.dl_runtime_lock); + p->rt.rt_rq = tg->dl_bandwidth.active_context->rt_rq[cpu]; + p->dl.dl_rq = &cpu_rq (cpu)->dl;#endif /* CONFIG_RT_GROUP_SCHED */ }
@@ -3116,6 +3113,9 @@ static inline void add_nr_running(struct rq *rq, unsigned count)unsigned prev_nr = rq->nr_running; rq->nr_running = prev_nr + count; +if (rq !=cpu_rq (rq->cpu)) +return ; +if (trace_sched_update_nr_running_tp_enabled ()) {call_trace_sched_update_nr_running (rq, count); }
@@ -3129,6 +3129,9 @@ static inline void add_nr_running(struct rq *rq, unsigned count)static inline void sub_nr_running (struct rq *rq,unsigned count) { rq->nr_running -= count; +if (rq !=cpu_rq (rq->cpu)) +return ; +if (trace_sched_update_nr_running_tp_enabled ()) {call_trace_sched_update_nr_running (rq, -count); }
--
2.56.0
RFC PATCH v7 14/29 sched/rt: Update task event callbacks for HCBS scheduling
In reply to Yuri AndriaccioUpdate wakeup_preempt_rt, switched_{from/to}_rt and prio_changed_rt with rt-cgroup's specific preemption rules:
- In wakeup_preempt_rt(), whenever a task wakes up, it must be checked if it is served by a deadline server or it lives on the global runqueue. Preemption rules (as documented in the function), change based on the current task's donor and woken task runqueue:
- If both tasks are FIFO/RR tasks on the global runqueue, or the same cgroup, run as normal.
- If woken is inside a cgroup, but donor is a FIFO task on the global runqueue, always preempt. If donor is a DEADLINE task, check if the dl server preempts donor.
- If both tasks are FIFO/RR tasks in served but different groups, check whether the woken server preempts the donor server.
- In prio_changed_rt(), if the task is not running, only run preemption checks if the running task resides on the same task group of the task that changed priority.
Update sched_rt_can_attach() to check if a task can be attached to a given cgroup. For now the check only consists in checking if the group has non-zero bandwidth. Remove the tsk argument from sched_rt_can_attach, as it is unused.
Change cpu_cgroup_can_attach() to check if the attachee is a FIFO/RR task before attaching it to a cgroup.
Update __sched_setscheduler() to perform checks when trying to switch to FIFO/RR for a task inside a cgroup, as the group needs to have runtime allocated.
Update task_is_throttled_rt() for SCHED_CORE, returning the is_throttled value of the server if present, while global rt-tasks are never throttled.
Update migration functions to ignore cgroups migration, to be implemented in later patches.
--- kernel/sched/core.c | 2 +- kernel/sched/rt.c | 108 +++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 2 +- kernel/sched/syscalls.c | 12 +++++ 4 files changed, 115 insertions(+), 9 deletions(-)
kernel/sched/core.c +1 -1
@@ -9577,7 +9577,7 @@ static int cpu_cgroup_can_attach(struct cgroup_taskset *tset) goto scx_check;cgroup_taskset_for_each (task, css, tset) { -if (!sched_rt_can_attach (css_tg (css), task)) +if (rt_task (task) && !sched_rt_can_attach (css_tg (css)))return -EINVAL ; } scx_check:
kernel/sched/rt.c +101 -7
211 lines not on this page · raw message
kernel/sched/sched.h +1 -1
@@ -603,7 +603,7 @@ extern int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us)extern int sched_group_set_rt_period (struct task_group *tg,u64 rt_period_us);extern long sched_group_rt_runtime (struct task_group *tg);extern long sched_group_rt_period (struct task_group *tg); -extern int sched_rt_can_attach (struct task_group *tg,struct task_struct *tsk); +extern int sched_rt_can_attach (struct task_group *tg);extern struct task_group *sched_create_group (struct task_group *parent); extern void sched_online_group(struct task_group *tg,
kernel/sched/syscalls.c +12 -0
@@ -606,6 +606,18 @@ int __sched_setscheduler(struct task_struct *p, change:if (user) { +/* +* Do not allow real-time tasks into groups that have no runtime +* assigned. +*/ +if (rt_group_sched_enabled () && +dl_bandwidth_enabled () &&rt_policy (policy) && + !sched_rt_can_attach (task_group (p)) && + !task_group_is_autogroup (task_group (p))) { + retval = -EPERM ; + goto unlock; + } +if (dl_bandwidth_enabled () &&dl_policy (policy) && !(attr->sched_flags &SCHED_FLAG_SUGOV )) {cpumask_t *span = rq->rd->span;
--
2.56.0
RFC PATCH v7 15/29 sched/rt: Remove support for cgroups-v1
In reply to Yuri AndriaccioDisable control files for cgroups-v1.
Remove cpu_rt_group_init function and functions related to the cgroup-v1 control files 'rt_runtime_us' and 'rt_period_us'.
--- kernel/sched/core.c | 50 -------------------------------------------- kernel/sched/rt.c | 49 +------------------------------------------ kernel/sched/sched.h | 4 ---- 3 files changed, 1 insertion(+), 102 deletions(-)
kernel/sched/core.c +0 -50
@@ -10185,32 +10185,6 @@ static int cpu_burst_write_u64(struct cgroup_subsys_state *css, }#endif /* CONFIG_GROUP_SCHED_BANDWIDTH */ -#ifdef CONFIG_RT_GROUP_SCHED -static int cpu_rt_runtime_write (struct cgroup_subsys_state *css, -struct cftype *cft,s64 val) -{ -return sched_group_set_rt_runtime (css_tg (css), val); -} - -static s64 cpu_rt_runtime_read (struct cgroup_subsys_state *css, -struct cftype *cft) -{ -return sched_group_rt_runtime (css_tg (css)); -} - -static int cpu_rt_period_write_uint (struct cgroup_subsys_state *css, -struct cftype *cftype,u64 rt_period_us) -{ -return sched_group_set_rt_period (css_tg (css), rt_period_us); -} - -static u64 cpu_rt_period_read_uint (struct cgroup_subsys_state *css, -struct cftype *cft) -{ -return sched_group_rt_period (css_tg (css)); -} -#endif /* CONFIG_RT_GROUP_SCHED */ -#ifdef CONFIG_GROUP_SCHED_WEIGHT static s64 cpu_idle_read_s64 (struct cgroup_subsys_state *css,struct cftype *cft)
@@ -10288,20 +10262,6 @@ static struct cftype cpu_legacy_files[] = { };#ifdef CONFIG_RT_GROUP_SCHED -static struct cftype rt_group_files[] = { - { - .name ="rt_runtime_us" , - .read_s64 = cpu_rt_runtime_read, - .write_s64 = cpu_rt_runtime_write, - }, - { - .name ="rt_period_us" , - .read_u64 = cpu_rt_period_read_uint, - .write_u64 = cpu_rt_period_write_uint, - }, - { }/* Terminate */ -}; -# ifdef CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED DEFINE_STATIC_KEY_FALSE (rt_group_sched);# else
@@ -10324,16 +10284,6 @@ static int __init setup_rt_group_sched(char *str)return 1 ; }__setup ("rt_group_sched=" , setup_rt_group_sched); - -static int __init cpu_rt_group_init (void ) -{ -if (!rt_group_sched_enabled ()) -return 0 ; - -WARN_ON (cgroup_add_legacy_cftypes (&cpu_cgrp_subsys, rt_group_files)); -return 0 ; -} -subsys_initcall (cpu_rt_group_init);#endif /* CONFIG_RT_GROUP_SCHED */ static int cpu_extra_stat_show(struct seq_file *sf,
kernel/sched/rt.c +1 -48
58 lines not on this page · raw message
kernel/sched/sched.h +0 -4
@@ -599,10 +599,6 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);extern void unthrottle_cfs_rq (struct cfs_rq *cfs_rq);extern bool cfs_task_bw_constrained (struct task_struct *p); -extern int sched_group_set_rt_runtime (struct task_group *tg,long rt_runtime_us); -extern int sched_group_set_rt_period (struct task_group *tg,u64 rt_period_us); -extern long sched_group_rt_runtime (struct task_group *tg); -extern long sched_group_rt_period (struct task_group *tg);extern int sched_rt_can_attach (struct task_group *tg);extern struct task_group *sched_create_group (struct task_group *parent);
Signature · 1 line
2.56.0
RFC PATCH v7 16/29 sched/rt: Update rt-cgroup schedulability checks
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Introduce cgroup-v2 control files:
- cpu.rt.max:
Get/set the bandwidth of the given cgroup, or inherith from parent.
- cpu.rt.effective.local:
Get the actual remaning bandwidth for the group, removing the bw of the group's children.
Introduce a number of functions to update the cgroup settings across the whole hierarchy:
- tg_subtree_has_rt_tasks()
Checks if the active context rooted at tg is running rt workload.
Child groups which do not share the same active context are ignored.
- tg_compute_children_bw()
Computes the total bandwidth of the active context rooted at tg minus the root of the context itself.
- tg_rt_schedulable()
Runs admission tests for the current cgroup tree and the given bandwidth update.
- tg_update_active_context()
Updates the active context of a given subtree with a new one.
- tg_rt_bandwidth() / tg_rt_internal_bandwidth()
Read the max (internal) bandwidth set to the cgroup.
- tg_set_rt_bandwidth()
Set the bandwidth of the group.
Update sched_rt_can_attach to run only tasks in the root cgroup or HCBS cgroups which have non-zero runtime.
Update and reuse __checkparam_dl to check for numerical issues regarding the dl_server's parameters.
Add from_ratio function to convert from period and bw to runtime, inverse of the to_ratio function.
Add dl_check_tg(), which performs an admission control test similar to __dl_overflow, but this time we are updating the cgroup's total bandwidth rather than scheduling a new DEADLINE task or updating a non-cgroup deadline server.
Add rcu_sched lock guard for rcu_read_{lock/unlock}_sched. Add sched_domains lock guard for sched_domains_mutex_{lock/unlock}. Add lock/unlock methods for sched_rt_handler_mutex and its lock guard.
Add asserts for held sched_domains_mutex and sched_rt_handler_mutex.
--- include/linux/rcupdate.h | 1 + include/linux/sched.h | 2 + kernel/sched/core.c | 47 +++++ kernel/sched/deadline.c | 61 ++++-- kernel/sched/rt.c | 403 ++++++++++++++++++++++++++++++++------- kernel/sched/sched.h | 17 +- kernel/sched/syscalls.c | 2 +- 7 files changed, 447 insertions(+), 86 deletions(-)
include/linux/rcupdate.h +1 -0
@@ -1214,6 +1214,7 @@ extern int rcu_expedited;extern int rcu_normal; DEFINE_LOCK_GUARD_0(rcu, rcu_read_lock(), rcu_read_unlock()) +DEFINE_LOCK_GUARD_0(rcu_sched, rcu_read_lock_sched(), rcu_read_unlock_sched()) DECLARE_LOCK_GUARD_0_ATTRS(rcu, __acquires_shared(RCU), __releases_shared(RCU))#endif /* __LINUX_RCUPDATE_H */
include/linux/sched.h +2 -0
@@ -2531,4 +2531,6 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable()) +DEFINE_LOCK_GUARD_0(sched_domains, sched_domains_mutex_lock(), sched_domains_mutex_unlock()) +#endif
kernel/sched/core.c +47 -0
59 lines not on this page · raw message
kernel/sched/deadline.c +50 -11
112 lines not on this page · raw message
kernel/sched/rt.c +330 -73
504 lines not on this page · raw message
kernel/sched/sched.h +16 -1
@@ -366,7 +366,7 @@ extern void sched_dl_do_global(void);extern int sched_dl_overflow (struct task_struct *p,int policy,const struct sched_attr *attr);extern void __setparam_dl (struct task_struct *p,const struct sched_attr *attr);extern void __getparam_dl (struct task_struct *p,struct sched_attr *attr,unsigned int flags); -extern bool __checkparam_dl (const struct sched_attr *attr); +extern bool __checkparam_dl (const struct sched_attr *attr,bool allow_zero_runtime);extern bool dl_param_changed (struct task_struct *p,const struct sched_attr *attr);extern int dl_cpuset_cpumask_can_shrink (const struct cpumask *cur,const struct cpumask *trial);extern int dl_bw_deactivate (int cpu);
@@ -425,6 +425,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,struct rq *served_rq, dl_server_pick_f pick_task);extern void sched_init_dl_servers (void ); +extern int dl_check_tg (unsigned long total);extern void dl_init_tg (struct sched_dl_entity *dl_se,u64 rt_runtime,u64 rt_period);extern void fair_server_init (struct rq *rq);
@@ -599,6 +600,12 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);extern void unthrottle_cfs_rq (struct cfs_rq *cfs_rq);extern bool cfs_task_bw_constrained (struct task_struct *p); +extern int tg_rt_bandwidth (struct task_group *tg, +u64 *rt_period_us,u64 *rt_runtime_us); +extern int tg_rt_internal_bandwidth (struct task_group *tg, +u64 *rt_period_us,u64 *rt_runtime_us); +extern int tg_set_rt_bandwidth (struct task_group *tg, +u64 rt_period_us,u64 rt_runtime_us);extern int sched_rt_can_attach (struct task_group *tg);extern struct task_group *sched_create_group (struct task_group *parent);
@@ -2123,6 +2130,14 @@ static inline struct rq *_this_rq_lock_irq(struct rq_flags *rf) __acquires_retreturn rq; } +extern struct mutex sched_rt_handler_mutex; +extern void sched_rt_handler_mutex_lock (void ); +extern void sched_rt_handler_mutex_unlock (void ); + +DEFINE_LOCK_GUARD_0 (sched_rt_handler, +sched_rt_handler_mutex_lock (), + sched_rt_handler_mutex_unlock()) +#ifdef CONFIG_NUMA enum numa_topology_type {
kernel/sched/syscalls.c +1 -1
@@ -528,7 +528,7 @@ int __sched_setscheduler(struct task_struct *p,*/ if (attr->sched_priority >MAX_RT_PRIO -1 )return -EINVAL ; -if ((dl_policy (policy) && !__checkparam_dl (attr)) || +if ((dl_policy (policy) && !__checkparam_dl (attr, false)) || (rt_policy (policy) != (attr->sched_priority !=0 )))return -EINVAL ;
Signature · 1 line
2.56.0
RFC PATCH v7 17/29 sched/rt: Update task's RT runqueue when switching scheduling class
In reply to Yuri Andriaccio--- kernel/sched/rt.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
kernel/sched/rt.c +22 -0
@@ -1926,6 +1926,27 @@ void __init init_sched_rt_class(void) } } +#ifdef CONFIG_RT_GROUP_SCHED +static void switching_to_rt (struct rq *rq,struct task_struct *p) +{ +struct task_group *tg = p->sched_task_group; +int cpu = rq->cpu; + +if (!rt_group_sched_enabled ()) +return ; + +if (tg == &root_task_group) { + p->rt.rt_rq = root_task_group.rt_rq[cpu]; +return ; + } + +guard (raw_spinlock_irqsave)(dl_bw_lock_of_tg (tg)); + p->rt.rt_rq =dl_bandwidth_read (tg)->active_context->rt_rq[cpu]; +} +#else +static void switching_to_rt (struct rq *rq,struct task_struct *p) {} +#endif +/* * When switching a task to RT, we may overload the runqueue * with RT tasks. In this case we try to push them off to
@@ -2127,6 +2148,7 @@ DEFINE_SCHED_CLASS(rt) = { .get_rr_interval = get_rr_interval_rt, + .switching_to = switching_to_rt, .switched_to = switched_to_rt, .prio_changed = prio_changed_rt,
--
2.56.0
RFC PATCH v7 18/29 sched/rt: Remove old RT_GROUP_SCHED data structures
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Completely remove the old RT_GROUP_SCHED's functions and data structures:
- Remove the fields back and my_q from sched_rt_entity.
- Remove the rt_bandwidth data structure.
- Remove the field rt_bandwidth from task_group.
- Remove the rt_bandwidth_enabled function.
- Remove the fields rt_queued, rt_throttled, rt_time, rt_runtime, rt_runtime_lock and rt_nr_boosted from rt_rq.
All of the removed fields and data are similarly represented in previously added fields in rq, rt_rq, dl_bandwidth and in the dl server themselves.
--- include/linux/sched.h | 3 --- kernel/sched/sched.h | 33 --------------------------------- 2 files changed, 36 deletions(-)
include/linux/sched.h +0 -3
@@ -627,12 +627,9 @@ struct sched_rt_entity {unsigned short on_rq;unsigned short on_list; -struct sched_rt_entity *back;#ifdef CONFIG_RT_GROUP_SCHED /* rq on which this entity is (to be) queued: */ struct rt_rq *rt_rq; -/* rq "owned" by this entity/group: */ -struct rt_rq *my_q;#endif } __randomize_layout;
kernel/sched/sched.h +0 -33
@@ -313,15 +313,6 @@ struct rt_prio_array {struct list_head queue[MAX_RT_PRIO ]; }; -struct rt_bandwidth { -/* nests inside the rq lock: */ -raw_spinlock_t rt_runtime_lock; -ktime_t rt_period; -u64 rt_runtime; -struct hrtimer rt_period_timer; -unsigned int rt_period_active; -}; -struct dl_bandwidth {raw_spinlock_t dl_runtime_lock;u64 dl_runtime;
@@ -343,12 +334,6 @@ static inline int dl_bandwidth_enabled(void)* - cache the fraction of bandwidth that is currently allocated in * each root domain; * -* This is all done in the data structure below. It is similar to the -* one used for RT-throttling (rt_bandwidth), with the main difference -* that, since here we are only interested in admission control, we -* do not decrease any runtime while the group "executes", neither we -* need a timer to replenish it. -* * With respect to SMP, bandwidth is given on a per root domain basis, * meaning that: * - bw (< 100%) is the deadline bandwidth of each CPU;
@@ -515,11 +500,9 @@ struct task_group {* different deadline server, and a runqueue per CPU. All the dl-servers * share the same dl_bandwidth object. */ -struct sched_rt_entity **rt_se;struct sched_dl_entity **dl_se;struct rt_rq **rt_rq; -struct rt_bandwidth rt_bandwidth;struct dl_bandwidth dl_bandwidth;#endif
@@ -887,11 +870,6 @@ struct scx_rq { };#endif /* CONFIG_SCHED_CLASS_EXT */ -static inline int rt_bandwidth_enabled (void ) -{ -return 0 ; -} -/* RT IPI pull logic requires IRQ_WORK */ #if defined(CONFIG_IRQ_WORK ) && defined(CONFIG_SMP )# define HAVE_RT_PUSH_IPI
@@ -909,17 +887,6 @@ struct rt_rq {bool overloaded;struct plist_head pushable_tasks; -int rt_queued; - -#ifdef CONFIG_RT_GROUP_SCHED -int rt_throttled; -u64 rt_time;/* consumed RT time, goes up in update_curr_rt */ -u64 rt_runtime;/* allotted RT time, "slice" from rt_bandwidth, RT sharing/balancing */ -/* Nests inside the rq lock: */ -raw_spinlock_t rt_runtime_lock; - -unsigned int rt_nr_boosted; -#endif #ifdef CONFIG_CGROUP_SCHED struct task_group *tg;/* this tg has "this" rt_rq on given CPU for runnable entities */ #endif
--
2.56.0
RFC PATCH v7 19/29 sched/rt: Add HCBS migration code to related functions
In reply to Yuri AndriaccioUpdate rt_queue_{push/pull}_task{s} to differentiate between cgroup and global runqueue balancing.
Introduce new balance callbacks for cgroup migration.
Add rq_to_push_from and rq_to_push_to fields for cgroup related migration.
Balance callbacks are only called on the root runqueues, thus it is necessary to store which non-root runqueues need to be balanced. The rqs to balance are stored as a forward-linked list/stack, with the top pointers accessible on root runqueues.
Update migration functions to specialize for cgroup migration:
- find_lowest_rt_rq():
Scan all the cpus to get the cgroup specific lowest_mask.
- find_lock_lowest_rt_rq():
Use appropriate rt_rqs to differentiate the cgroup being checked. Prevent migration for throttled cgroups.
- push_rt_rq_task():
Allow pushing away for migration disabled tasks only if the tasks belong to the same cgroup.
- pull_rt_rq_task():
Use appropriate rt_rqs and push away for migration disabled only if the task to pull and curr are in the same runqueue.
Add tg_of_se to get the task group a scheduling entity is assigned to.
This is different from the active context of the group.
Add new macros for field access and non-CONFIG_RT_GROUP_SCHED code.
--- kernel/sched/rt.c | 292 +++++++++++++++++++++++++++++++++---------- kernel/sched/sched.h | 24 ++++ 2 files changed, 248 insertions(+), 68 deletions(-)
kernel/sched/rt.c +224 -68
@@ -364,31 +364,83 @@ static inline int has_pushable_tasks(struct rt_rq *rt_rq) static DEFINE_PER_CPU(struct balance_callback, rt_push_head); static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); +static DEFINE_PER_CPU(struct balance_callback, rt_group_push_head); +static DEFINE_PER_CPU(struct balance_callback, rt_group_pull_head);static void push_rt_tasks (struct rq *);static void pull_rt_task (struct rq *); +static void push_group_rt_tasks (struct rq *); +static void pull_group_rt_task (struct rq *); + +#define rq_queue_field_push (global_rq, new_rq, field) \ + do { \ + struct rq *__rq = (global_rq); \ + struct rq *__new_rq = (new_rq); \ + __new_rq->field = __rq->field; \ + __rq->field = __new_rq; \ + } while (0) + +#define rq_queue_field_pop (global_rq, field) \ + ({ \ + struct rq *__rq = (global_rq); \ + struct rq *__head = __rq->field; \ + if (__head) { \ + __rq->field = __head->field; \ + __head->field = NULL; \ + } \ + __head; \ + }) + +#define push_rq_to_push_from (global_rq, new_rq) rq_queue_field_push((global_rq), (new_rq), rq_to_push_from) +#define pop_rq_to_push_from (global_rq) rq_queue_field_pop((global_rq), rq_to_push_from) + +#define push_rq_to_pull_to (global_rq, new_rq) rq_queue_field_push((global_rq), (new_rq), rq_to_pull_to) +#define pop_rq_to_pull_to (global_rq) rq_queue_field_pop((global_rq), rq_to_pull_to)static inline void rt_queue_push_tasks (struct rt_rq *rt_rq) { -struct rq *rq =global_rq_of_rt_rq (rt_rq); - -if (is_dl_group (rt_rq)) -return ; +struct rq *rq =rq_of_rt_rq (rt_rq); +struct rq *global_rq =global_rq_of_rt_rq (rt_rq);if (!has_pushable_tasks (rt_rq))return ; -queue_balance_callback (rq, &per_cpu (rt_push_head, rq->cpu), push_rt_tasks); +if (!rt_group_sched_enabled () || !is_dl_group (rt_rq)) { + +queue_balance_callback (global_rq, + &per_cpu (rt_push_head, rq->cpu), + push_rt_tasks); + }else { + +push_rq_to_push_from (global_rq, rq); +queue_balance_callback (global_rq, + &per_cpu (rt_group_push_head, global_rq->cpu), + push_group_rt_tasks); + } }static inline void rt_queue_pull_task (struct rt_rq *rt_rq) { -struct rq *rq =global_rq_of_rt_rq (rt_rq); +struct rq *rq =rq_of_rt_rq (rt_rq); +struct rq *global_rq =global_rq_of_rt_rq (rt_rq); +struct sched_dl_entity *dl_se; -if (is_dl_group (rt_rq)) -return ; +if (!rt_group_sched_enabled () || !is_dl_group (rt_rq)) { + +queue_balance_callback (global_rq, + &per_cpu (rt_pull_head, rq->cpu), + pull_rt_task); + }else { + + dl_se =dl_group_of (rt_rq); +if (dl_se->dl_throttled) +return ; -queue_balance_callback (rq, &per_cpu (rt_pull_head, rq->cpu), pull_rt_task); +push_rq_to_pull_to (global_rq, rq); +queue_balance_callback (global_rq, + &per_cpu (rt_group_pull_head, global_rq->cpu), + pull_group_rt_task); + } }static void push_rt_rq_tasks (struct rt_rq *rt_rq);
@@ -403,6 +455,31 @@ static void pull_rt_task(struct rq *global_rq) {pull_rt_rq_task (&global_rq->rt); } +static void push_group_rt_tasks (struct rq *global_rq) +{ +struct rq *rq; + +while ((rq =pop_rq_to_push_from (global_rq))) { +struct rt_rq *rt_rq = &rq->rt; + +if (rt_rq->rt_nr_running <=1 && !dl_group_of (rt_rq)->dl_throttled) +continue ; + +push_rt_rq_tasks (rt_rq); + } +} + +static void pull_group_rt_task (struct rq *global_rq) +{ +struct rq *rq; + +while ((rq =pop_rq_to_pull_to (global_rq))) { +struct rt_rq *rt_rq = &rq->rt; + +pull_rt_rq_task (rt_rq); + } +} +static void enqueue_pushable_task (struct rt_rq *rt_rq,struct task_struct *p) {plist_del (&p->pushable_tasks, &rt_rq->pushable_tasks);
@@ -1246,35 +1323,74 @@ static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask);static int find_lowest_rt_rq (struct task_struct *task) {struct sched_domain *sd; -struct cpumask *lowest_mask =this_cpu_cpumask_var_ptr (local_cpu_mask); -int this_cpu =smp_processor_id (); -int cpu =task_cpu (task); -int ret; - -/* Make sure the mask is initialized first */ -if (unlikely (!lowest_mask)) -return -1 ; + cpumask_var_t mask __free(free_cpumask_var) = CPUMASK_VAR_NULL; +struct cpumask *lowest_mask; +struct sched_dl_entity *dl_se; +struct rt_rq *rt_rq, *task_rt_rq =rt_rq_of_se (&task->rt); +int cpu, this_cpu =smp_processor_id (); +int ret, prio, lowest_prio;if (task->nr_cpus_allowed ==1 )return -1 ;/* No other targets possible */ -/* -* If we're on asym system ensure we consider the different capacities -* of the CPUs when searching for the lowest_mask. -*/ -if (sched_asym_cpucap_active ()) { +if (!rt_group_sched_enabled () || !is_dl_group (task_rt_rq)) { + + lowest_mask =this_cpu_cpumask_var_ptr (local_cpu_mask); + +/* Make sure the mask is initialized first */ +if (unlikely (!lowest_mask)) +return -1 ; + +/* +* If we're on asym system ensure we consider the different +* capacities of the CPUs when searching for the lowest_mask. +*/ +if (sched_asym_cpucap_active ()) { + + ret =cpupri_find_fitness (&task_rq (task)->rd->cpupri, + task, lowest_mask, + rt_task_fits_capacity); + }else { + + ret =cpupri_find (&task_rq (task)->rd->cpupri, + task, lowest_mask); + } - ret =cpupri_find_fitness (&task_rq (task)->rd->cpupri, - task, lowest_mask, - rt_task_fits_capacity); +if (!ret) +return -1 ;/* No targets found */ }else { +if (!alloc_cpumask_var (&mask,GFP_KERNEL )) +return -ENOMEM ; + + lowest_prio = task->prio -1 ; + lowest_mask = mask; +cpumask_clear (lowest_mask); +for_each_cpu_and (cpu, cpu_online_mask, task->cpus_ptr) { + dl_se =dl_se_of_tg (task_rt_rq->tg, cpu); + rt_rq = &dl_se->my_q->rt; + prio = rt_rq->highest_prio.curr; - ret =cpupri_find (&task_rq (task)->rd->cpupri, - task, lowest_mask); +/* +* If we're on asym system ensure we consider the +* different capacities of the CPUs when searching for +* the lowest_mask. +*/ +if (dl_se->dl_throttled || !rt_task_fits_capacity (task, cpu)) +continue ; + +if (prio >= lowest_prio) { +if (prio > lowest_prio) { +cpumask_clear (lowest_mask); + lowest_prio = prio; + } + +cpumask_set_cpu (cpu, lowest_mask); + } + } } -if (!ret) -return -1 ;/* No targets found */ +if (cpumask_empty (lowest_mask)) +return -1 ;/* * At this point we have built a mask of CPUs representing the
@@ -1284,6 +1400,7 @@ static int find_lowest_rt_rq(struct task_struct *task)* We prioritize the last CPU that the task executed on since * it is most likely cache-hot in that location. */ + cpu =task_cpu (task);if (cpumask_test_cpu (cpu, lowest_mask))return cpu;
@@ -1294,30 +1411,27 @@ static int find_lowest_rt_rq(struct task_struct *task)if (!cpumask_test_cpu (this_cpu, lowest_mask)) this_cpu =-1 ;/* Skip this_cpu opt if not among lowest */ -rcu_read_lock (); -for_each_domain (cpu, sd) { -if (sd->flags &SD_WAKE_AFFINE ) { +scoped_guard (rcu) { +for_each_domain (cpu, sd) {int best_cpu; +if (!(sd->flags &SD_WAKE_AFFINE )) +continue ; +/* * "this_cpu" is cheaper to preempt than a * remote processor. */ if (this_cpu !=-1 && -cpumask_test_cpu (this_cpu,sched_domain_span (sd))) { -rcu_read_unlock (); +cpumask_test_cpu (this_cpu,sched_domain_span (sd)))return this_cpu; - } best_cpu =cpumask_any_and_distribute (lowest_mask,sched_domain_span (sd)); -if (best_cpu < nr_cpu_ids) { -rcu_read_unlock (); +if (best_cpu < nr_cpu_ids)return best_cpu; - } } } -rcu_read_unlock ();/* * And finally, if there were no matches within the domains
@@ -1368,27 +1482,35 @@ static struct task_struct *pick_next_pushable_task(struct rt_rq *rt_rq)/* Will lock the rq it finds */ static struct rt_rq *find_lock_lowest_rt_rq (struct task_struct *task,struct rt_rq *rt_rq) { -struct rq *rq =rq_of_rt_rq (rt_rq); -struct rq *lowest_rq =NULL ; -int tries; -int cpu; +struct rq *lowest_rq, *rq =global_rq_of_rt_rq (rt_rq); +struct rt_rq *lowest_rt_rq; +struct sched_dl_entity *lowest_dl_se; +int tries, cpu; +bool dl_group; + + dl_group =rt_group_sched_enabled () &&is_dl_group (rt_rq);for (tries =0 ; tries <RT_MAX_TRIES ; tries++) { cpu =find_lowest_rt_rq (task);if ((cpu ==-1 ) || (cpu == rq->cpu)) -break ; +return NULL ; lowest_rq =cpu_rq (cpu); +if (dl_group) { + lowest_dl_se =dl_se_of_tg (rt_rq->tg, cpu); + lowest_rt_rq = &lowest_dl_se->my_q->rt; + }else { + lowest_rt_rq = &lowest_rq->rt; + } -if (lowest_rq->rt.highest_prio.curr <= task->prio) { +if (lowest_rt_rq->highest_prio.curr <= task->prio) {/* * Target rq has tasks of equal or higher priority, * retrying does not release any lock and is unlikely * to yield a different result. */ - lowest_rq =NULL ; -break ; +return NULL ; }/* if the prio of this runqueue changed, try again */
@@ -1404,25 +1526,24 @@ static struct rt_rq *find_lock_lowest_rt_rq(struct task_struct *task, struct rt_* check the task migration disable flag here too. */ if (unlikely (is_migration_disabled (task) || + (dl_group && lowest_dl_se->dl_throttled) || !cpumask_test_cpu (lowest_rq->cpu, &task->cpus_mask) || task !=pick_next_pushable_task (rt_rq))) {double_unlock_balance (rq, lowest_rq); - lowest_rq =NULL ; -break ; +return NULL ; } }/* If this rq is still suitable use it. */ -if (lowest_rq->rt.highest_prio.curr > task->prio) -break ; +if (lowest_rt_rq->highest_prio.curr > task->prio) +return lowest_rt_rq;/* try again */ double_unlock_balance (rq, lowest_rq); - lowest_rq =NULL ; } -return &lowest_rq->rt; +return NULL ; }static struct rq *find_lock_lowest_rq (struct task_struct *task,struct rq *rq) {
@@ -1442,12 +1563,10 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq) {static int push_rt_rq_task (struct rt_rq *rt_rq,bool pull) {struct task_struct *next_task; -struct rq *lowest_rq, *rq =rq_of_rt_rq (rt_rq); +struct rq *lowest_rq, *rq =global_rq_of_rt_rq (rt_rq);struct rt_rq *lowest_rt_rq;int ret =0 ; - -if (is_dl_group (rt_rq)) -return 0 ; +bool dl_group;if (!rt_rq->overloaded)return 0 ;
@@ -1462,7 +1581,8 @@ static int push_rt_rq_task(struct rt_rq *rt_rq, bool pull)* higher priority than current. If that's the case * just reschedule current. */ -if (unlikely (next_task->prio < rq->donor->prio)) { + dl_group =rt_group_sched_enabled () &&is_dl_group (rt_rq); +if (!dl_group &&unlikely (next_task->prio < rq->donor->prio)) {resched_curr (rq);return 0 ; }
@@ -1474,6 +1594,13 @@ static int push_rt_rq_task(struct rt_rq *rt_rq, bool pull)if (!pull || rq->push_busy)return 0 ; +/* +* If the current task does not belong to the same task group +* we cannot push it away. +*/ +if (dl_group && rq->donor->sched_task_group != rt_rq->tg) +return 0 ; +/* * Invoking find_lowest_rt_rq() on anything but an RT task doesn't * make sense. Per the above priority check, curr has to
@@ -1550,7 +1677,7 @@ static int push_rt_rq_task(struct rt_rq *rt_rq, bool pull) goto retry; } - lowest_rq =rq_of_rt_rq (lowest_rt_rq); + lowest_rq =global_rq_of_rt_rq (lowest_rt_rq);move_queued_task_locked (rq, lowest_rq, next_task);resched_curr (lowest_rq); ret =1 ;
@@ -1747,16 +1874,22 @@ void rto_push_irq_work_func(struct irq_work *work)static void pull_rt_rq_task (struct rt_rq *this_rt_rq) { -struct rq *this_rq =rq_of_rt_rq (this_rt_rq); +struct rq * this_rq =global_rq_of_rt_rq (this_rt_rq);int this_cpu = this_rq->cpu, cpu;bool resched = false; -struct task_struct *p, *push_task; +struct task_struct *p, *push_task =NULL ; +struct sched_dl_entity *src_dl_se;struct rt_rq *src_rt_rq;struct rq *src_rq; -int rt_overload_count =rt_overloaded (this_rq); +int rt_overload_count; +const struct cpumask *cpu_mask; +bool dl_group; -if (is_dl_group (&this_rq->rt)) -return ; + dl_group =rt_group_sched_enabled () &&is_dl_group (this_rt_rq); +if (dl_group) + goto group_sched; + + rt_overload_count =rt_overloaded (this_rq);if (likely (!rt_overload_count))return ;
@@ -1779,12 +1912,26 @@ static void pull_rt_rq_task(struct rt_rq *this_rt_rq) }#endif -for_each_cpu (cpu, this_rq->rd->rto_mask) { +group_sched: +if (!dl_group) + cpu_mask = this_rq->rd->rto_mask; +else + cpu_mask = cpu_online_mask; + +for_each_cpu (cpu, cpu_mask) {if (this_cpu == cpu)continue ; src_rq =cpu_rq (cpu); - src_rt_rq = &src_rq->rt; +if (!dl_group) { + src_rt_rq = &src_rq->rt; + }else { + src_dl_se =dl_se_of_tg (this_rt_rq->tg, cpu); + src_rt_rq = &src_dl_se->my_q->rt; + +if (src_rt_rq->rt_nr_running <=1 && !src_dl_se->dl_throttled) +continue ; + }/* * Don't bother taking the src_rq->lock if the next highest
@@ -1825,12 +1972,21 @@ static void pull_rt_rq_task(struct rt_rq *this_rt_rq)* This is just that p is waking up and hasn't * had a chance to schedule. We only pull * p if it is lower in priority than the -* current task on the run queue +* current task on the run queue and p is +* in the same runqueue as donor. */ -if (p->prio < src_rq->donor->prio) +if (tg_of_se (&src_rq->donor->rt) == this_rt_rq->tg && + p->prio < src_rq->donor->prio) goto skip;if (is_migration_disabled (p)) { +/* +* If the current task does not belong to the +* same task group we cannot push it away. +*/ +if (tg_of_se (&src_rq->donor->rt) != this_rt_rq->tg) + goto skip; + push_task =get_push_task (src_rq); }else {move_queued_task_locked (src_rq, this_rq, p);
kernel/sched/sched.h +24 -0
@@ -1403,6 +1403,16 @@ struct rq {struct list_head cfsb_csd_list;#endif +#ifdef CONFIG_RT_GROUP_SCHED +/* +* Balance callbacks operate only on global runqueues. +* These pointers allow referencing cgroup specific runqueues +* for balancing operations. +*/ +struct rq *rq_to_push_from; +struct rq *rq_to_pull_to; +#endif +atomic_t nr_iowait; } __no_randomize_layout;
@@ -3520,6 +3530,11 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)return rt_se->rt_rq; } +static inline struct task_group *tg_of_se (struct sched_rt_entity *rt_se) +{ +return rt_rq_of_se (rt_se)->tg; +} +static inline int is_dl_group (struct rt_rq *rt_rq) {return rt_rq->tg != &root_task_group;
@@ -3536,6 +3551,7 @@ static inline struct sched_dl_entity *dl_group_of(struct rt_rq *rt_rq) return rt_rq->tg->dl_se[rq_of_rt_rq(rt_rq)->cpu]; } +#define dl_se_of_tg (tg, cpu) ((tg)->dl_se[(cpu)])#define dl_bw_lock_of_tg (tg) (&(tg)->dl_bandwidth.dl_runtime_lock)#else static inline struct task_struct *rt_task_of (struct sched_rt_entity *rt_se)
@@ -3560,6 +3576,11 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)return &rq->rt; } +static inline struct task_group *tg_of_se (struct sched_rt_entity *rt_se) +{ +return &root_task_group; +} +static inline int is_dl_group (struct rt_rq *rt_rq) {return 0 ;
@@ -3570,6 +3591,9 @@ static inline struct sched_dl_entity *dl_group_of(struct rt_rq *rt_rq) return NULL; } +#define rq_to_push_from (rq) (rq) +#define rq_to_pull_to (rq) (rq) +#define dl_se_of_tg (tg, cpu) ((struct sched_dl_entity*)NULL)#define dl_bw_lock_of_tg (tg) ((raw_spinlock_t*)NULL)#endif
--
2.56.0
RFC PATCH v7 20/29 sched/rt: Hook HCBS migration functions
In reply to Yuri AndriaccioHook rt-cgroup migration functions:
- select_task_rt_rq
Always return the cpu where the task is scheduled.
- balance_rt
- put_prev_task_rt
If a server is throttled, put_prev_task_rt is invoked and a push is necessary so that the task can keep running on another server if possible.
- switched_to_rt
Keep track of the deadline server that is assigned to the task switching to FIFO/RR priority.
Set rt-cgroups runqueues to online to allow balancing in balance_rt.
--- kernel/sched/rt.c | 42 +++++++++++++++++++++++++++++++++--------- 1 file changed, 33 insertions(+), 9 deletions(-)
kernel/sched/rt.c +33 -9
@@ -266,6 +266,7 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)init_rt_rq (&s_rq->rt); s_rq->cpu = i; + s_rq->online =1 ; s_rq->rt.tg = tg;init_dl_entity (dl_se);
@@ -1017,6 +1018,10 @@ select_task_rq_rt(struct task_struct *p, int cpu, int flags)struct rq *rq;bool test; +/* Just return the task_cpu for processes inside task groups */ +if (is_dl_group (rt_rq_of_se (&p->rt))) + goto out; +/* For anything but wake ups, just return the task_cpu */ if (!(flags & (WF_TTWU |WF_FORK ))) goto out;
@@ -1106,27 +1111,30 @@ static void check_preempt_equal_prio(struct rq *rq, struct task_struct *p)resched_curr (rq); } -static int balance_rt (struct rq *rq,struct rq_flags *rf) +static int balance_rt (struct rq *global_rq,struct rq_flags *rf) {/* * Note, rq->donor may change during rq lock drops, * so don't re-use p across lock drops */ -struct task_struct *p = rq->donor; +struct task_struct *p = global_rq->donor; +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt); -if (!on_rt_rq (&p->rt) &&need_pull_rt_task (rq, p)) { +if (!on_rt_rq (&p->rt) &&need_pull_rt_task (rq_of_rt_rq (rt_rq), p)) {/* * This is OK, because current is on_cpu, which avoids it being * picked for load-balance and preemption/IRQs are still * disabled avoiding further scheduler activity on it and we've * not yet started the picking loop. */ -rq_unpin_lock (rq, rf); -pull_rt_rq_task (&rq->rt); -rq_repin_lock (rq, rf); +rq_unpin_lock (global_rq, rf); +pull_rt_rq_task (rt_rq); +rq_repin_lock (global_rq, rf); } -return sched_stop_runnable (rq) ||sched_dl_runnable (rq) ||sched_rt_runnable (rq); +return sched_stop_runnable (global_rq) || +sched_dl_runnable (global_rq) || +sched_rt_runnable (global_rq); }/*
@@ -1293,6 +1301,13 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s*/ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 )enqueue_pushable_task (rt_rq, p); + +if (is_dl_group (rt_rq)) { +struct sched_dl_entity *dl_se =dl_group_of (rt_rq); + +if (dl_se->dl_throttled) +rt_queue_push_tasks (rt_rq); + } }/* Only try algorithms three times */
@@ -2110,12 +2125,21 @@ static void switching_to_rt(struct rq *rq, struct task_struct *p) {}*/ static void switched_to_rt (struct rq *rq,struct task_struct *p) { +struct rt_rq *rt_rq =rt_rq_of_se (&p->rt); +/* * If we are running, update the avg_rt tracking, as the running time * will now on be accounted into the latter. */ if (task_current (rq, p)) {update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); + +if (is_dl_group (rt_rq)) { +struct sched_dl_entity *dl_se =dl_group_of (rt_rq); + + p->dl_server = dl_se; + } +return ; }
@@ -2125,8 +2149,8 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)* then see if we can move to another run queue. */ if (task_on_rq_queued (p)) { -if (p->nr_cpus_allowed >1 &&rt_rq_of_se (&p->rt)->overloaded) -rt_queue_push_tasks (rt_rq_of_se (&p->rt)); +if (p->nr_cpus_allowed >1 && rt_rq->overloaded) +rt_queue_push_tasks (rt_rq);if (p->prio < rq->donor->prio &&cpu_online (cpu_of (rq)))resched_curr (rq);
Signature · 1 line
2.56.0
RFC PATCH v7 21/29 sched/core: Execute enqueued balance callbacks when changing allowed CPUs
In reply to Yuri AndriaccioFrom: luca abeni <[email protected]>
Execute balancing callbacks when setting the affinity of a task, since the HCBS scheduler may request balancing of throttled dl_servers to fully utilize the server's bandwidth.
--- kernel/sched/core.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/sched/core.c +13 -0
@@ -3019,6 +3019,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flagif (cpumask_test_cpu (task_cpu (p), &p->cpus_mask) || (task_current_donor (rq, p) && !task_current (rq, p))) {struct task_struct *push_task =NULL ; +struct balance_callback *head;if ((flags &SCA_MIGRATE_ENABLE ) && (p->migration_flags &MDF_PUSH ) && !rq->push_busy) {
@@ -3037,11 +3038,13 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag }preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf);if (push_task) {stop_one_cpu_nowait (rq->cpu, push_cpu_stop, p, &rq->push_work); } +balance_callbacks (rq, head);preempt_enable ();if (complete)
@@ -3096,6 +3099,8 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag }if (task_on_cpu (rq, p) ||READ_ONCE (p->__state) ==TASK_WAKING ) { +struct balance_callback *head; +/* * MIGRATE_ENABLE gets here because 'p == current', but for * anything else we cannot do is_migration_disabled(), punt
@@ -3109,16 +3114,19 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag p->migration_flags &= ~MDF_PUSH ;preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf);if (!stop_pending) {stop_one_cpu_nowait (cpu_of (rq), migration_cpu_stop, &pending->arg, &pending->stop_work); } +balance_callbacks (rq, head);preempt_enable ();if (flags &SCA_MIGRATE_ENABLE )return 0 ; }else { +struct balance_callback *head;if (!is_migration_disabled (p)) {if (task_on_rq_queued (p))
@@ -3129,7 +3137,12 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag complete = true; } } + +preempt_disable (); + head =splice_balance_callbacks (rq);task_rq_unlock (rq, p, rf); +balance_callbacks (rq, head); +preempt_enable ();if (complete)complete_all (&pending->done);
--
2.56.0
RFC PATCH v7 22/29 sched/rt: Try pull task on empty server pick.
In reply to Yuri AndriaccioTry to pull task on a server with an empty runqueue before returning NULL (and thus shutting down).
---
When all the servers of a cgroup are throttled, work is pending, and any one of the servers is replenished, it may happen that the runqueue is empty and thus the replenished server is immediately shut down.
The server may try to pull a task so that the cgroup could consume its allocated runtime as soon as it is replenished.
--- kernel/sched/deadline.c | 8 ++++++++ kernel/sched/rt.c | 13 +++++++++++-- 2 files changed, 19 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +8 -0
@@ -2910,6 +2910,14 @@ static struct task_struct *__pick_task_dl(struct rq *rq, struct rq_flags *rf)if (dl_server (dl_se)) { p = dl_se->server_pick_task (dl_se, rf); + +/* +* server_pick_task may drop the runqueue lock, so check if +* something has shifted. +*/ +if (pick_next_dl_entity (dl_rq) != dl_se) + goto again; +if (!p) {dl_server_stop (dl_se); goto again;
kernel/sched/rt.c +11 -2
@@ -286,14 +286,23 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent) }static struct sched_rt_entity *pick_next_rt_entity (struct rt_rq *rt_rq); +static void pull_rt_task (struct rq *);static struct task_struct *rt_server_pick (struct sched_dl_entity *dl_se,struct rq_flags *rf) {struct rt_rq *rt_rq = &dl_se->my_q->rt; +struct rq *global_rq =global_rq_of_rt_rq (rt_rq);struct task_struct *p; -if (!sched_rt_runnable (dl_se->my_q)) -return NULL ; +if (!sched_rt_runnable (dl_se->my_q)) { +rq_unpin_lock (global_rq, rf); +pull_rt_task (rq_of_rt_rq (rt_rq)); +update_rq_clock (global_rq); +rq_repin_lock (global_rq, rf); + +if (!sched_rt_runnable (dl_se->my_q)) +return NULL ; + } p =rt_task_of (pick_next_rt_entity (rt_rq));
Signature · 1 line
2.56.0
RFC PATCH v7 23/29 sched/core: Execute enqueued balance callbacks after migrate_disable_switch
In reply to Yuri AndriaccioExecute balance callbacks after migrate_disable_switch.
Balancing may be requested on the __schedule path, in migrate_disable_switch, when the running task is throttled and then pushed away from its runqueue.
--- kernel/sched/core.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
kernel/sched/core.c +10 -2
@@ -2470,6 +2470,9 @@ do_set_cpus_allowed(struct task_struct *p, struct affinity_context *ctx);static void migrate_disable_switch (struct rq *rq,struct task_struct *p) { +struct rq_flags rf; +struct balance_callback *head; +struct affinity_context ac = { .new_mask =cpumask_of (rq->cpu), .flags =SCA_MIGRATE_DISABLE ,
@@ -2481,8 +2484,13 @@ static void migrate_disable_switch(struct rq *rq, struct task_struct *p)if (p->cpus_ptr != &p->cpus_mask)return ; - scoped_guard (task_rq_lock, p) -do_set_cpus_allowed (p, &ac); + rq =task_rq_lock (p, &rf); + +do_set_cpus_allowed (p, &ac); + + head =splice_balance_callbacks (rq); +task_rq_unlock (rq, p, &rf); +balance_callbacks (rq, head); }void ___migrate_enable (void )
--
2.56.0
RFC PATCH v7 24/29 sched/syscalls: Allow unpriviledged tasks to set RT priority inside cgroups.
In reply to Yuri Andriaccio--- kernel/sched/syscalls.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)
kernel/sched/syscalls.c +19 -7
@@ -444,15 +444,27 @@ static int user_check_sched_setscheduler(struct task_struct *p,if (rt_policy (policy)) {unsigned long rlim_rtprio =task_rlimit (p,RLIMIT_RTPRIO ); +struct task_group *active_context; -/* Can't set/change the rt policy: */ -if (policy != p->policy && !rlim_rtprio) - goto req_priv; +if (!rt_group_sched_enabled ()) + goto root_cgroup_checks; -/* Can't increase priority: */ -if (attr->sched_priority > p->rt_priority && - attr->sched_priority > rlim_rtprio) - goto req_priv; + scoped_guard(raw_spinlock_irq, dl_bw_lock_of_tg(p->sched_task_group)) { + active_context =dl_bandwidth_read (p->sched_task_group)->active_context; + } + +/* Allow changing to the rt policy when not in the root cgroup */ +if (active_context == &root_task_group) { +root_cgroup_checks: +/* Can't set/change the rt policy: */ +if (policy != p->policy && !rlim_rtprio) + goto req_priv; + +/* Can't increase priority: */ +if (attr->sched_priority > p->rt_priority && + attr->sched_priority > rlim_rtprio) + goto req_priv; + } }/*
--
2.56.0
RFC PATCH v7 25/29 sched/rt: Enable HRTICK_DL for HCBS servers.
In reply to Yuri AndriaccioIf HRTICK_DL is enabled, start hrtick timers whenever a rt-task that is served by a rt-cgroup is ticked or enqueued for execution.
--- kernel/sched/deadline.c | 13 +++++++++++-- kernel/sched/rt.c | 30 ++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 3 files changed, 42 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +11 -2
@@ -2841,10 +2841,20 @@ static void start_hrtick_dl(struct rq *rq, struct sched_dl_entity *dl_se) {hrtick_start (rq, dl_se->runtime); } + +int dl_can_start_hrtick (struct rq *rq,struct sched_dl_entity *dl_se) +{ +return dl_se->runtime >0 &&is_leftmost (dl_se, &rq->dl); +}#else /* !CONFIG_SCHED_HRTICK: */ static void start_hrtick_dl (struct rq *rq,struct sched_dl_entity *dl_se) { } + +int dl_can_start_hrtick (struct rq *rq,struct sched_dl_entity *dl_se) +{ +return 0 ; +}#endif /* !CONFIG_SCHED_HRTICK */ /*
@@ -2975,8 +2985,7 @@ static void task_tick_dl(struct rq *rq, struct task_struct *p, int queued)* not being the leftmost task anymore. In that case NEED_RESCHED will * be set and schedule() will start a new hrtick for the next task. */ -if (hrtick_enabled_dl (rq) && queued && p->dl.runtime >0 && -is_leftmost (&p->dl, &rq->dl)) +if (hrtick_enabled_dl (rq) && queued &&dl_can_start_hrtick (rq, &p->dl))start_hrtick_dl (rq, &p->dl); }
kernel/sched/rt.c +30 -0
@@ -1231,6 +1231,17 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)check_preempt_equal_prio (rq, p); } +#ifdef CONFIG_SCHED_HRTICK +static void start_hrtick_rt (struct rq *rq,struct sched_dl_entity *dl_se) +{ +hrtick_start (rq, dl_se->runtime); +} +#else /* !CONFIG_SCHED_HRTICK: */ +static void start_hrtick_dl (struct rq *rq,struct sched_dl_entity *dl_se) +{ +} +#endif /* !CONFIG_SCHED_HRTICK */ +static inline void set_next_task_rt (struct rq *rq,struct task_struct *p,enum snt_e type) {struct sched_rt_entity *rt_se = &p->rt;
@@ -1258,6 +1269,12 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, enum supdate_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 );rt_queue_push_tasks (rt_rq); + +if (!is_dl_group (rt_rq)) +return ; + +if (hrtick_enabled_dl (rq)) +start_hrtick_rt (rq,dl_group_of (rt_rq)); }static struct sched_rt_entity *pick_next_rt_entity (struct rt_rq *rt_rq)
@@ -2254,6 +2271,8 @@ static inline void watchdog(struct rq *rq, struct task_struct *p) { }static void task_tick_rt (struct rq *rq,struct task_struct *p,int queued) {struct sched_rt_entity *rt_se = &p->rt; +struct sched_dl_entity *dl_se; +struct rt_rq *rt_rq =rt_rq_of_se (rt_se);update_curr_rt (rq);update_rt_rq_load_avg (rq_clock_pelt (rq), rq,1 );
@@ -2281,6 +2300,17 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued)resched_curr (rq);return ; } + +if (!is_dl_group (rt_rq)) +return ; + +/* +* update_curr_rt() might have resulted our dl_server to not be anymore +* the leftmost deadline entity, similarly to task_tick_dl. +*/ + dl_se =dl_group_of (rt_rq); +if (hrtick_enabled_dl (rq) && queued &&dl_can_start_hrtick (rq, dl_se)) +start_hrtick_rt (rq, dl_se); }static unsigned int get_rr_interval_rt (struct rq *rq,struct task_struct *task)
kernel/sched/sched.h +1 -0
@@ -356,6 +356,7 @@ extern bool dl_param_changed(struct task_struct *p, const struct sched_attr *attextern int dl_cpuset_cpumask_can_shrink (const struct cpumask *cur,const struct cpumask *trial);extern int dl_bw_deactivate (int cpu);extern s64 dl_scaled_delta_exec (struct rq *rq,struct sched_dl_entity *dl_se,s64 delta_exec); +extern int dl_can_start_hrtick (struct rq *rq,struct sched_dl_entity *dl_se);/* * SCHED_DEADLINE supports servers (nested scheduling) with the following * interface:
Signature · 1 line
2.56.0
RFC PATCH v7 26/29 Documentation: Update documentation for real-time cgroups
In reply to Yuri AndriaccioUpdate the RT_GROUP_SCHED specific documentation. Give a brief theoretical background for Hierarchical Constant Bandwidth Server (HCBS). Document how the HCBS is implemented in the kernel and how the RT_GROUP_SCHED behaves now compared to the version which this patchset replaces.
--- Documentation/scheduler/sched-rt-group.rst | 471 +++++++++++++++++---- 1 file changed, 394 insertions(+), 77 deletions(-)
Documentation/scheduler/sched-rt-group.rst +394 -77
@@ -53,9 +53,12 @@ CPU time is divided by means of specifying how much time can be spent running in a given period. We allocate this "run time" for each real-time group which the other real-time groups will not be permitted to use. -Any time not allocated to a real-time group will be used to run normal priority -tasks (SCHED_OTHER). Any allocated run time not used will also be picked up by -SCHED_OTHER. +Each real-time group runs at the same priority as SCHED_DEADLINE, thus they +share and contend the SCHED_DEADLINE allowed bandwidth. Any time not allocated +to a real-time group (and SCHED_DEADLINE tasks) will be used to run both +SCHED_FIFO/SCHED_RR, normal priority tasks (SCHED_OTHER), and SCHED_EXT tasks, +following the usual priorities. Any allocated run time not used will also be +picked up by the other scheduling classes, in the same order as before. Let's consider an example: a frame fixed real-time renderer must deliver 25 frames a second, which yields a period of 0.04s per frame. Now say it will also
@@ -73,10 +76,6 @@ The remaining CPU time will be used for user input and other tasks. Because real-time tasks have explicitly allocated the CPU time they need to perform their tasks, buffer underruns in the graphics or audio can be eliminated. -NOTE: the above example is not fully implemented yet. We still -lack an EDF scheduler to make non-uniform periods usable. - - 2. The Interface ================
@@ -86,105 +85,423 @@ lack an EDF scheduler to make non-uniform periods usable. The system wide settings are configured under the /proc virtual file system: -/proc/sys/kernel/sched_rt_period_us: +``/proc/sys/kernel/sched_rt_period_us``: The scheduling period that is equivalent to 100% CPU bandwidth. -/proc/sys/kernel/sched_rt_runtime_us: - A global limit on how much time real-time scheduling may use. This is always - less or equal to the period_us, as it denotes the time allocated from the - period_us for the real-time tasks. Without CONFIG_RT_GROUP_SCHED enabled, - this only serves for admission control of deadline tasks. With - CONFIG_RT_GROUP_SCHED=y it also signifies the total bandwidth available to - all real-time groups. +``/proc/sys/kernel/sched_rt_runtime_us``: + A global limit on how much time real-time scheduling may use (SCHED_DEADLINE + tasks + real-time groups). This is always less or equal to the period_us, as + it denotes the time allocated from the period_us for the real-time tasks. + Without **CONFIG_RT_GROUP_SCHED** enabled, this only serves for admission + control of deadline tasks. With **CONFIG_RT_GROUP_SCHED=y** it also signifies + the total bandwidth available to both real-time groups and deadline tasks. * Time is specified in us because the interface is s32. This gives an operating range from 1us to about 35 minutes. - * sched_rt_period_us takes values from 1 to INT_MAX. - * sched_rt_runtime_us takes values from -1 to sched_rt_period_us. - * A run time of -1 specifies runtime == period, ie. no limit. - * sched_rt_runtime_us/sched_rt_period_us > 0.05 inorder to preserve - bandwidth for fair dl_server. For accurate value check average of - runtime/period in /sys/kernel/debug/sched/fair_server/cpuX/ + * ``sched_rt_period_us`` takes values from 1 to INT_MAX. + * ``sched_rt_runtime_us`` takes values from -1 to ``sched_rt_period_us``. + * A run time of -1 specifies runtime == period, i.e., no limit, but also + disables admission tests for SCHED_DEADLINE. + +The default value for both ``sched_rt_period_us`` and ``sched_rt_runtime_us`` is +1000000 (or 1s), while fair-servers and ext-servers have a default runtime of +50ms and default period of 1s, giving a minimum of 0.05s to be used by +SCHED_FIFO/SCHED_RR and non-RT tasks (SCHED_OTHER, SCHED_EXT), while 0.95s are +the maximum to be used by SCHED_DEADLINE, and rt-cgroups if enabled. + +2.2 Cgroup settings +------------------- + +Enabling **CONFIG_RT_GROUP_SCHED** lets you explicitly allocate real CPU +bandwidth to task groups. + + .. warning:: + Real Time Cgroups are only available for cgroups-v2. + .. + +This uses the cgroup virtual file system and the CPU controller for cgroups. +Enabling the controller for the hierarchy creates two files: + +* ``<cgroup>/cpu.rt.max``, which specifies the runtime and period of the group. + The file also accepts a runtime of 'max', specifying that its tasks must be + scheduled using the nearest configured ancestor (or the root cgroup if it is + the nearest non-max ancestor). +* ``<cgroup>/cpu.rt.effective.local``, read-only, returns the runtime and period + actually allocated to the group, excluding that of its children. + + .. tip:: + For more information on working with control groups, you should read + *Documentation/admin-guide/cgroup-v2.rst*. + .. + +By default the root cgroup has the same period of +``/proc/sys/kernel/sched_rt_period_us``, which is 1s, and a runtime of zero, so +that rt-cgroup is *soft-disabled* by default, and all the runtime is available +for SCHED_DEADLINE tasks only. New groups instead get a period of zero and +runtime of 'max' (essentially delegating their tasks' scheduling to the nearest +configured ancestor). + +3. Theoretical Background +========================= + + + .. BIG FAT WARNING ****************************************************** + + .. warning:: + + This section contains a (not-thorough) summary on deadline/hierarchical + scheduling theory, and how it applies to real-time control groups. + The reader can "safely" skip to Section 4 if only interested in seeing + how the scheduling policy can be used. Anyway, we strongly recommend + to come back here and continue reading (once the urge for testing is + satisfied :P) to be sure of fully understanding all technical details. + + .. ************************************************************************ + +The real-time cgroup scheduler is based upon the **Hierarchical Constant +Bandwidth Server** (HCBS) [1] *Compositional Scheduling Framework* (CSF). A +**CSF** is a framework where global (system-level) timing properties can be +established by composing independently (specified and) analyzed local +(component-level) timing properties [5]. + +For HCBS (related to the Linux kernel), the compositional framework consists of +two parts: + +* The *scheduling components*, which are the basic units of the scheduling. In + the kernel these are the single cgroups along with the tasks that must be run + inside. + +* The *scheduling resources*, which are the CPUs of the machine. + +HCBS is a *hierarchical scheduling framework*, where the scheduling components +form a hierarchy and resources are allocated from parent components to its child +components in the hierarchy. + +The Chapter is organized as follows: **Section 3.1** gives basic real-time +theory definitions that are used throughout the whole section. **Section 3.2** +talks about the HCBS framework, giving a general idea on how this is structured. +**Section 3.3** introduces the MPR model, one of the many models which may be +used for the analysis of the scheduling components and the computation of the +minimum required scheduling resources for a given component. **Section 3.4** +shows the schedulability test for MPR on the HCBS framework. **Section 3.5** +shows how to convert a MPR interface to a HCBS compatible resource reservation +for a component. Finally, **Section 3.6** lists other interesting models which +could be used for the component analysis in HCBS. + +3.1 Basic Definitions +--------------------- +*We borrow the same definitions given in the* ``sched_deadline`` *document, +which are very briefly summarized here, and new ones, needed by the following +content, are added.* + +A typical real-time task is composed of a repetition of computation phases (task +instances, or jobs) which are activated on a periodic or sporadic fashion. For +our purposes, real-time tasks are characterized by three parameters: + +* Worst Case Execution Time (WCET): the maximum execution time among all jobs. +* Relative Deadline (D): the maximum time each job must be completed, relative + to the release time of the job. +* Inter-Arrival Period (P): the exact/minimum (for periodic/sporadic tasks) time + between each consecutive job. + +3.2 Hierarchical Constant Bandwidth Server (HCBS) [1] +----------------------------------------------------- + +As mentioned, HCBS is a *hierarchical scheduling framework*: + +* The framework hierarchy follows the same hierarchy of cgroups. Cgroups may + have two roles, either bandwidth reservation for children cgroups, or they may + be *live*, i.e. run tasks (but not both). The root cgroup, for the kernel's + implementation of HCBS, acts only as bandwidth reservation (but as written in + this document it has also different uses outside of the hierarchical + framework). +* The cgroup tree is internally flattened, for ease of scheduling, to a + two-level hierarchy, since only the *live* groups are of interest and all the + necessary information for their scheduling lies in their interface (there is + no need for the reservation components). +* The hierarchical framework, now on two levels, consists then of a first level + of cgroups, and a second level of tasks that are run inside these groups. +* The scheduling of components is performed using global Earliest Deadline First + (gEDF), SCHED_DEADLINE in the kernel, following the bandwidth reservation of + each group. +* Whenever a component is scheduled, a local scheduler picks which of the tasks + of the cgroup to run. The scheduling policy is global Fixed Priority (gFP), + SCHED_FIFO/SCHED_RR in the kernel. -2.2 Default behaviour ---------------------- +3.3 Multiprocessor Periodic Resource (MPR) model +------------------------------------------------ + +A Multiprocessor Periodic Resource (MPR) model [2] **u = <Pi, Theta, m'>** +specifies that an identical, unit-capacity multiprocessor platform collectively +provides **Theta** units of resource every **Pi** time units, where the +**Theta** time units are supplied with concurrency at most **m'**. + +This theoretical model is one of the many models that can abstract the +interface of our real-time cgroups: let **m'** be the number of CPUs of the +machine, let **Theta** be **m' * <cgroup>/cpu.rt_runtime_us** and **Pi** be +**<cgroup>/cpu.rt_period_us**. + +Let's introduce the concept of Supply Bound Function (SBF). A SBF is a function +which outputs a lower bound for the processor supply provided in a given time +interval, given a resource supply model. For a completely dedicated CPU, the SBF +function is simply the identity function, as it will always provide **t** units +of computation for an interval of length **t**. The situation gets slightly more +complicated for the MPR model or any of the other model listed in section 3.6. + +The **SBF(t)** for a MPR model **u = <Pi, Theta, m'>** is:: + + | 0 if t' < 0 + | + SBF_u(t) = | floor(t' / Pi) * Theta + | + max(0, m' * x - (m' * Pi - Theta)) if t' >= 0 and 1 <= x <= y + | + | floor(t' / Pi) * Theta + | + max(0, m' * x - (m' * Pi - Theta)) else + | - (m' - beta) + +where:: + + alpha = floor(Theta / m') + beta = Theta - m' * alpha + t' = t - (Pi - ceil(Theta / m')) + x = t' - (Pi * floor(t' / Pi)) + y = Pi - floor(Theta / m') + +Briefly, this function models that the server's bandwidth is given as late as +possible, so describing the worst case possible for the supplied bandwidth. + +3.4 Schedulability for MPR on global Fixed-Priority +--------------------------------------------------- + +Let's introduce the concept of Demand Bound Function (DBF). A DBF is a function +that, given a taskset, a scheduling algorithm and an interval of time, outputs +the worst resource demand for that interval of time. + +It is easy to see that, given a DBF and a SBF, we can deem a component/taskset +schedulable if, for every time interval t >= 0, it is possible to demonstrate +that: + + DBF(t) <= SBF(t) + +We have the Supply Bound Function for our given MPR model, so we are missing the +Demand Bound Function for a given taskset that is being scheduled using global +Fixed Priority. + +3.4.1 Schedulability Analysis for global Fixed Priority +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Bertogna, Cirinei and Lipari [6] have derived a schedulability test for global +Fixed Priority (gFP) on multi-processor platforms. In this test (called +*BCL_gFP* test) we can consider all the CPUs to be dedicated to the scheduling. + + A taskset **Tau** is schedulable with gFP on a multiprocessor platform + composed of **m'** identical processors if for each task **tau_k in Tau**: + + Sum(for i < k)( min(W_i(D_k), D_k - C_k + 1) ) < m' * (D_k - C_k + 1) + + where **W_i(t)** is the workload of task **tau_i** over a time interval **t**: + + W_i(t) = N_i(t) * C_i + min(C_i, t + D_i - C_i - N_i(t) * P_i) + + and **N_i(t)** is the number of activations of task **tau_i** that complete in + a time interval **t**: -The default values for sched_rt_period_us (1000000 or 1s) and -sched_rt_runtime_us (950000 or 0.95s). This gives 0.05s to be used by -SCHED_OTHER (non-RT tasks). These defaults were chosen so that a run-away -real-time tasks will not lock up the machine but leave a little time to recover -it. By setting runtime to -1 you'd get the old behaviour back. + N_i(t) = floor( (t + D_i - C_i) / P_i ) -By default all bandwidth is assigned to the root group and new groups get the -period from /proc/sys/kernel/sched_rt_period_us and a run time of 0. If you -want to assign bandwidth to another group, reduce the root group's bandwidth -and assign some or all of the difference to another group. + while the **min** term is the contribution of the carried-out job in the + interval **t**, i.e. that job that does not completely fit in the interval + **t**, but starts inside the interval after all the jobs that complete. + +3.4.2 From BCL_gFP to the Demand Bound Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We can then derive the DBF from this test: + + DBF_gFP(tau_k) = Sum(for i < k)( min(W_i(D_k), D_k - C_k + 1) ) + m' * (C_k - 1) + +Briefly, the first sum component, the same in the BCL_gFP test, describes the +maximum interference that higher priority task give to the analysed task. The +workload is upperbounded by ``(D_k - C_K + 1)`` because we are only interested +in the interference in the slack time, while for the ``C_k`` time we are +requiring that all the CPUs are fully available, as the single job needs `C_k` +(non overlapping) time units to run. + +The demand bound function from Bertogna et al. is only defined on a single time +(i.e. the deadline of the task in analysis) instead of all possible times as +this is the minimum argument to demonstrate schedulability on global Fixed +Priority. + +3.4.3 Putting it all together +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A component **C**, on **m'** processors, running a taskset **Tau = { tau_1 = +(C_1, D_1, P_1), ..., tau_n = (C_n, D_n, P_n) }** of **n** sporadic tasks, is +schedulable under gFP using an MPR model **u = <Pi, Theta, m'>**, if for all +tasks **tau_k in Tau**: + + DBF_gFP(tau_k) <= SBF_u(D_k) + +3.5 From MPR to deadline servers +-------------------------------- + +Since there exist no algorithm to schedule MPR interfaces, a tecnique was +developed to transform MPR interfaces into periodic tasks, so that a +number of periodic servers which respect the tasks requirements can be used for +the scheduling of the MPR interface and associated tasks. + +Let **u = <Pi, Theta, m>** be a MPR interface, let **a = Theta - m * floor(Theta +/ m)**, let **k = floor(a)**. Define a transformation from **u** to a periodic +taskset **Tau_u = { tau_1 = (C_1, D_1, P_1), ..., tau_m' = (C_m', D_m', P_m') +}**, where: + + **tau_1 = ... = tau_k = (floor(Theta / m') + 1, Pi, Pi)** + + **tau_k+1 = (floor(Theta / m') + a - k * floor(a/k), Pi, Pi)** + + **tau_k+2 = ... = tau_m' = (floor(Theta / m'), Pi, Pi)** + +This periodic taskset of servers **Tau_u** can be scheduled on any number of +processors with concurrency at most **m'**. + +For real-time control groups, it is possible to just consider a slightly more +demanding taskset **Tau_u'**, where each task **tau_i** is defined as follows: + + **tau_i = (ceil(Theta / m'), Pi, Pi)** + +3.6 Other models +---------------- + +There exist many other theoretical models in literature which are used to +describe a hierarchical scheduling framework on multi-core architectures. +Notable examples are the Multi Supply Function (MSF) abstraction [3], the +Parallel Supply Function (PSF) abstraction [4] and the Bounded Delay +Multipartition (BDM) [7]. + +3.7 References +-------------- + 1 - L. Abeni, A. Balsini, and T. Cucinotta, “Container-based real-time + scheduling in the Linux kernel,” SIGBED Rev., vol. 16, no. 3, pp. 33-38, + Nov. 2019, doi: 10.1145/3373400.3373405. + 2 - A. Easwaran, I. Shin, and I. Lee, “Optimal virtual cluster-based + multiprocessor scheduling,” Real-Time Syst, vol. 43, no. 1, pp. 25-59, + Sept. 2009, doi: 10.1007/s11241-009-9073-x. + 3 - E. Bini, G. Buttazzo, and M. Bertogna, “The Multi Supply Function + Abstraction for Multiprocessors,” in 2009 15th IEEE International + Conference on Embedded and Real-Time Computing Systems and Applications, + Aug. 2009, pp. 294-302. doi: 10.1109/RTCSA.2009.39. + 4 - E. Bini, B. Marko, and S. K. Baruah, “The Parallel Supply Function + Abstraction for a Virtual Multiprocessor,” in Scheduling, S. Albers, S. K. + Baruah, R. H. Möhring, and K. Pruhs, Eds., in Dagstuhl Seminar Proceedings + (DagSemProc), vol. 10071. Dagstuhl, Germany: Schloss Dagstuhl - + Leibniz-Zentrum für Informatik, 2010, pp. 1-14. doi: + 10.4230/DagSemProc.10071.14. + 5 - I. Shin and I. Lee, “Compositional real-time scheduling framework,” in + 25th IEEE International Real-Time Systems Symposium, Dec. 2004, pp. 57-67. + doi: 10.1109/REAL.2004.15. + 6 - M. Bertogna, M. Cirinei, and G. Lipari, “Schedulability Analysis of Global + Scheduling Algorithms on Multiprocessor Platforms,” IEEE Transactions on + Parallel and Distributed Systems, vol. 20, no. 4, pp. 553-566, Apr. 2009, + doi: 10.1109/TPDS.2008.129. + 7 - G. Lipari and E. Bini, “A Framework for Hierarchical Scheduling on + Multiprocessors: From Application Requirements to Run-Time Allocation,” in + 2010 31st IEEE Real-Time Systems Symposium, Nov. 2010, pp. 249-258. doi: + 10.1109/RTSS.2010.12. + + +4. Using Real-Time cgroups +========================== + +4.1 CGroup Setup +---------------- -Real-time group scheduling means you have to assign a portion of total CPU -bandwidth to the group before it will accept real-time tasks. Therefore you will -not be able to run real-time tasks as any user other than root until you have -done that, even if the user has the rights to run processes with real-time -priority! +The following is a brief guide to the use of Real-Time Control Groups. +Of course, real-time control groups require mounting of the cgroup file system. +We have decided to only support cgroups v2, so make sure you mount the v2 +controller for the cgroup hierarchy. -2.3 Basis for grouping tasks ----------------------------- +Additionally the real-time cgroups require the CPU controller for the cgroups to +be enabled:: -Enabling CONFIG_RT_GROUP_SCHED lets you explicitly allocate real -CPU bandwidth to task groups. + # Assume the cgroup file system is mounted at /sys/fs/cgroup + > echo "+cpu" > /sys/fs/cgroup/cgroup.subtree_control -This uses the cgroup virtual file system and "<cgroup>/cpu.rt_runtime_us" -to control the CPU time reserved for each control group. +The CPU controller can only be mounted if there is no SCHED_FIFO/SCHED_RR task +scheduled in any cgroup other than the root control group. -For more information on working with control groups, you should read -Documentation/admin-guide/cgroup-v1/cgroups.rst as well. +The root control group has no bandwidth allocated by default, so make sure to +allocate some bandwidth so that it can be used by the other cgroups. More on +that in the following section... -Group settings are checked against the following limits in order to keep the -configuration schedulable: +4.2 Bandwidth Allocation for groups +----------------------------------- - \Sum_{i} runtime_{i} / global_period <= global_runtime / global_period +Allocating bandwidth to a cgroup is a fundamental step to run real-time +workload. The cgroup filesystem exposes two files: -For now, this can be simplified to just the following (but see Future plans): +* ``<cgroup>/cpu.rt.max``: which specifies the cgroups' runtime and period in + microseconds. +* ``<cgroup>/cpu.rt.effective.local``: read-only, get the cgroups' actualy + runtime and period in microseconds, without its children's bandwidth. - \Sum_{i} runtime_{i} <= global_runtime +By definition, the specified runtime must be always less than or equal to the +period. Additionally, an admission test checks if the bandwidth invariant is +respected (i.e. sum of children's bandwidth <= parent's bandwidth). +The root control group files instead control and reserve the SCHED_DEADLINE +bandwidth allocated to real-time cgroups, since real-time groups compete and +share the same bandwidth allocated to SCHED_DEADLINE tasks. -3. Future plans -=============== +4.3 Running real-time tasks in groups +------------------------------------- -There is work in progress to make the scheduling period for each group -("<cgroup>/cpu.rt_period_us") configurable as well. +To run tasks in real-time groups it is just necessary to change a task's +scheduling policy to SCHED_FIFO/SCHED_RR and migrate it into the group. If the +group is not allowed to run real-time tasks because of incorrect configuration, +either migrating a SCHED_FIFO/SCHED_RR task into the group or changing +scheduling policy to a task already inside the group will fail:: -The constraint on the period is that a subgroup must have a smaller or -equal period to its parent. But realistically its not very useful _yet_ -as its prone to starvation without deadline scheduling. + # assume there is a task of PID 42 running + # change its scheduling policy to SCHED_FIFO, priority 99 + > chrt -f -p 99 42 -Consider two sibling groups A and B; both have 50% bandwidth, but A's -period is twice the length of B's. + # migrate the task to a cgroup + > echo 42 > /sys/fs/cgroup/<my-cgroup>/cgroup.procs -* group A: period=100000us, runtime=50000us +4.4 Special case: the root control group +---------------------------------------- - - this runs for 0.05s once every 0.1s +The root cgroup is special, compared to the other cgroups, as its tasks are not +managed by the HCBS algorithm, rather they just use the original +SCHED_FIFO/SCHED_RR policies (as if CONFIG_RT_GROUP_SCHED was disabled). As +mentioned, its bandwidth files are just used to control how much of the +SCHED_DEADLINE bandwidth is allocated to cgroups. -* group B: period= 50000us, runtime=25000us +Any non-root cgroup configured as 'max' that has the root cgroup as its nearest +non-max ancestor will run its tasks in the root runqueue. - - this runs for 0.025s twice every 0.1s (or once every 0.05 sec). +4.5 Guarantees and Special Behaviours +------------------------------------- -This means that currently a while (1) loop in A will run for the full period of -B and can starve B's tasks (assuming they are of lower priority) for a whole -period. +Real-time cgroups are run at the same priority level of SCHED_DEADLINE tasks. +Since this is the highest priority scheduling policy, and since the Constant +Bandwidth Server (CBS) enforces that the specified bandwidth requirements for +both groups and tasks cannot be overrun, real-time groups have the same +guarantees that SCHED_DEADLINE tasks have, i.e. they will be necessarily +supplied by the amount of bandwidth requested (whenever the admission tests +pass). -The next project will be SCHED_EDF (Earliest Deadline First scheduling) to bring -full deadline scheduling to the linux kernel. Deadline scheduling the above -groups and treating end of the period as a deadline will ensure that they both -get their allocated time. +This means that, since SCHED_FIFO/SCHED_RR tasks (scheduled in the root control +group) are not subject to bandwidth controls, they are run at a lower priority +than the cgroups' counterparts. Nonetheless, a minimum amount of bandwidth, if +reserved, will always be available to run SCHED_FIFO/SCHED_RR workloads in the +root cgroup, while they will be able to use more runtime if any of the +SCHED_DEADLINE tasks or servers use less than their specified amount of +bandwidth. SCHED_OTHER tasks are instead scheduled as normal, at lower priority +than real-time workloads. -Implementing SCHED_EDF might take a while to complete. Priority Inheritance is -the biggest challenge as the current linux PI infrastructure is geared towards -the limited static priority levels 0-99. With deadline scheduling you need to -do deadline inheritance (since priority is inversely proportional to the -deadline delta (deadline - now)). +The aforementioned behaviour differs from the preceding RT_GROUP_SCHED +implementation, but this is necessary to give actual guarantees to the amount of +bandwidth given to rt-cgroups. -This means the whole PI machinery will have to be reworked - and that is one of -the most complex pieces of code we have.
--
2.56.0
RFC PATCH v7 27/29 sched/rt: Integrate cpuset and cpu controllers
In reply to Yuri AndriaccioIntegrate the cpuset and cpu controllers:
- cpuset cannot be changed if the HCBS cgroup is active (non-max) or any of the group's children is active (cpuset_can_change function).
- introduce cpuset_effective_cpus to read the cpuset's effective mask.
- update struct dl_bandwidth to store the effective mask of the cgroup.
- remove the dl_internal_runtime in dl_bandwidth as this value is now per-cpu/server and equal to dl_se->dl_runtime.
- update schedulability checks to account for cpumasks.
- update cpu.rt.effective.local interface to show per-cpu internal values.
--- include/linux/cpuset.h | 2 + kernel/cgroup/cpuset.c | 31 +++++ kernel/sched/core.c | 22 +++- kernel/sched/deadline.c | 13 ++- kernel/sched/rt.c | 249 ++++++++++++++++++++++++++-------------- kernel/sched/sched.h | 9 +- 6 files changed, 237 insertions(+), 89 deletions(-)
include/linux/cpuset.h +2 -0
@@ -81,6 +81,8 @@ extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *maextern void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask);extern bool cpuset_cpus_allowed_fallback (struct task_struct *p);extern int cpuset_num_cpus (struct cgroup *cgroup); +extern void cpuset_effective_cpus (struct cgroup *cgroup,struct cpumask *mask); +extern int cpuset_can_change (struct cgroup *cgroup);extern nodemask_t cpuset_mems_allowed (struct task_struct *p);#define cpuset_current_mems_allowed (current->mems_allowed)void cpuset_init_current_mems_allowed (void );
kernel/cgroup/cpuset.c +31 -0
@@ -2572,6 +2572,10 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs,if (cpumask_equal (cs->cpus_allowed, trialcs->cpus_allowed))return 0 ; +/* Disallow change if subtree has any allocated rt-cgroup */ +if (!cpuset_can_change (cs->css.cgroup)) +return -EINVAL ; +compute_trialcs_excpus (trialcs, cs); trialcs->prs_err =PERR_NONE ;
@@ -2632,6 +2636,10 @@ static int update_exclusive_cpumask(struct cpuset *cs, struct cpuset *trialcs,if (cpumask_equal (cs->exclusive_cpus, trialcs->exclusive_cpus))return 0 ; +/* Disallow change if subtree has any allocated rt-cgroup */ +if (!cpuset_can_change (cs->css.cgroup)) +return -EINVAL ; +/* * Reject the change if there is exclusive CPUs conflict with * the siblings.
@@ -3585,6 +3593,10 @@ static ssize_t cpuset_partition_write(struct kernfs_open_file *of, char *buf,else return -EINVAL ; +/* Disallow change if subtree has any allocated rt-cgroup */ +if (!cpuset_can_change (cs->css.cgroup)) +return -EINVAL ; +mutex_lock (&cpuset_top_mutex);cpus_read_lock ();wait_attach_done_lock ();
@@ -4354,6 +4366,25 @@ int cpuset_num_cpus(struct cgroup *cgrp)return nr; } +void cpuset_effective_cpus (struct cgroup *cgrp,struct cpumask *mask) +{ +struct cpuset *cs; + +if (!is_in_v2_mode ()) { +cpumask_copy (mask, cpu_online_mask); +return ; + } + +guard (rcu)(); + cs =css_cs (cgroup_e_css (cgrp, &cpuset_cgrp_subsys)); +if (!cs) { +cpumask_copy (mask, cpu_online_mask); +return ; + } + +cpumask_copy (mask, cs->effective_cpus); +} +void __init cpuset_init_current_mems_allowed (void ) {nodes_setall (current->mems_allowed);
kernel/sched/core.c +20 -2
@@ -8970,6 +8970,8 @@ void __init sched_init(void)init_defrootdomain ();#ifdef CONFIG_RT_GROUP_SCHED +BUG_ON (!alloc_dl_bandwidth (&root_task_group.dl_bandwidth,GFP_NOWAIT )); +init_dl_bandwidth (&root_task_group.dl_bandwidth,global_rt_period (),0 , &root_task_group);#endif /* CONFIG_RT_GROUP_SCHED */
@@ -10426,6 +10428,19 @@ static void __maybe_unused cpu_period_quota_print(struct seq_file *sf,seq_printf (sf," %ld\n" , period); } +static void __maybe_unused cpu_period_quota_single_print (struct seq_file *sf,int cpu, +long period,long quota) +{ +seq_printf (sf,"%d " , cpu); + +if (quota <0 ) +seq_puts (sf,"runtime=max" ); +else +seq_printf (sf,"runtime=%ld" , quota); + +seq_printf (sf," period=%ld\n" , period); +} +/* caller should put the current value in *@periodp before calling */ static int __maybe_unused cpu_period_quota_parse (char *buf,u64 *period_us_p,u64 *quota_us_p)
@@ -10486,9 +10501,12 @@ static int cpu_rt_internal_show(struct seq_file *sf, void *v) {struct task_group *tg =css_tg (seq_css (sf));u64 period_us, runtime_us; +int cpu; -tg_rt_internal_bandwidth (tg, &period_us, &runtime_us); -cpu_period_quota_print (sf, period_us, runtime_us); +for_each_possible_cpu (cpu) { +tg_rt_internal_bandwidth (tg, cpu, &period_us, &runtime_us); +cpu_period_quota_single_print (sf, cpu, period_us, runtime_us); + }return 0 ; }
kernel/sched/deadline.c +12 -1
@@ -592,14 +592,25 @@ static inline int is_leftmost(struct sched_dl_entity *dl_se, struct dl_rq *dl_rqstatic void init_dl_rq_bw_ratio (struct dl_rq *dl_rq); +struct dl_bandwidth *alloc_dl_bandwidth (struct dl_bandwidth *dl_b,gfp_t gfp_flags) { +if (!alloc_cpumask_var (&dl_b->active_mask, gfp_flags)) +return NULL ; + +return dl_b; +} + +void free_dl_bandwidth (struct dl_bandwidth *dl_b) { +free_cpumask_var (dl_b->active_mask); +} +void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime,struct task_group *active_context) {raw_spin_lock_init (&dl_b->dl_runtime_lock); dl_b->dl_period = period; dl_b->dl_runtime = runtime; - dl_b->dl_internal_runtime =0 ; dl_b->active_context = active_context; +cpumask_clear (dl_b->active_mask); }
kernel/sched/rt.c +165 -84
450 lines not on this page · raw message
kernel/sched/sched.h +7 -2
@@ -316,8 +316,8 @@ struct rt_prio_array {struct dl_bandwidth {raw_spinlock_t dl_runtime_lock;u64 dl_runtime; -u64 dl_internal_runtime;u64 dl_period; +cpumask_var_t active_mask;struct task_group *active_context; };
@@ -586,7 +586,7 @@ extern bool cfs_task_bw_constrained(struct task_struct *p);extern int tg_rt_bandwidth (struct task_group *tg,u64 *rt_period_us,u64 *rt_runtime_us); -extern int tg_rt_internal_bandwidth (struct task_group *tg, +extern int tg_rt_internal_bandwidth (struct task_group *tg,int cpu,u64 *rt_period_us,u64 *rt_runtime_us);extern int tg_set_rt_bandwidth (struct task_group *tg,u64 rt_period_us,u64 rt_runtime_us);
@@ -3054,6 +3054,11 @@ extern void resched_curr(struct rq *rq);extern void resched_curr_lazy (struct rq *rq);extern void resched_cpu (int cpu); +extern struct dl_bandwidth *alloc_dl_bandwidth (struct dl_bandwidth *dl_b,gfp_t gfp_flags); +extern void free_dl_bandwidth (struct dl_bandwidth *dl_b); + +DEFINE_FREE(dl_bandwidth_free, struct dl_bandwidth *, if (_T) free_dl_bandwidth(_T)) +extern void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime,struct task_group *active_context);extern void init_dl_entity (struct sched_dl_entity *dl_se);
Signature · 1 line
2.56.0
RFC PATCH v7 28/29 sched/rt: Prevent CPU offline operation on active rt-cgroups.
In reply to Yuri AndriaccioIntroduce tg_can_offline_cpu and prevent CPU offlining if any rt-cgroup has requested bandwidth on the offlining core.
--- kernel/sched/core.c | 3 +++ kernel/sched/rt.c | 27 +++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 3 files changed, 31 insertions(+)
kernel/sched/core.c +3 -0
@@ -8717,6 +8717,9 @@ int sched_cpu_deactivate(unsigned int cpu)struct rq *rq =cpu_rq (cpu);int ret; +if (!tg_can_offline_cpu (cpu)) +return -EBUSY ; + ret =dl_bw_deactivate (cpu);if (ret)
kernel/sched/rt.c +27 -0
@@ -2853,6 +2853,33 @@ int cpuset_can_change(struct cgroup *cgroup) {return 1 ; } +static int __tg_can_offline_cpu (struct task_group *tg,void *data) { +const struct dl_bandwidth *dl_b; +int *d = data; +int cpu = *d; + +if (tg == &root_task_group) +return 0 ; + +guard (raw_spinlock_irq)(dl_bw_lock_of_tg (tg)); + dl_b =dl_bandwidth_read (tg); +if (dl_b->dl_runtime ==RUNTIME_INF ) +return 0 ; + +return cpumask_test_cpu (cpu, dl_b->active_mask); +} + +int tg_can_offline_cpu (int cpu) { +if (!rt_group_sched_enabled ()) +return 1 ; + +guard (rcu)(); +if (walk_tg_tree (__tg_can_offline_cpu, tg_nop, &cpu)) +return 0 ; + +return 1 ; +} +int sched_rt_can_attach (struct task_group *tg) {struct task_group *ctx;
kernel/sched/sched.h +1 -0
@@ -590,6 +590,7 @@ extern int tg_rt_internal_bandwidth(struct task_group *tg, int cpu, u64 *rt_period_us, u64 *rt_runtime_us);extern int tg_set_rt_bandwidth (struct task_group *tg,u64 rt_period_us,u64 rt_runtime_us); +extern int tg_can_offline_cpu (int cpu);extern int sched_rt_can_attach (struct task_group *tg);extern struct task_group *sched_create_group (struct task_group *parent);
--
2.56.0
RFC PATCH v7 29/29 sched/rt: Introduce multi-cpu support
In reply to Yuri AndriaccioIntroduce per-cpu overrides to server's runtime and period: it is now possible to override a subset of cpus HCBS setup by writing to the cpu.rt.max file.
The new syntax is as follows:
max [<period_us>] - Set whole cgroup to max.
Remove any overrides.
default runtime=max - As above.
<runtime_us> <period_us> - Set default runtime/period.
Keeps overrides.
default runtime=<runtime_us> period=<period_us> - As above.
<cpumask> default - Remove override for the
given cpumask. Resets to the default runtime and period.
<cpumask> runtime=<runtime_us> period=<period_us> - Set override for the
given cpumask.
To keep consistency between the cpumask and the overrides, it is not possible to override a cpu to zero runtime. Shrink the cpuset controller cpumask instead.
If the cgroup defaults to zero runtime, defaulting an override will reset the cpus to zero runtime. A zero runtime cgroup which has all of its cpus overriden to non-zero can be scheduled onto (but for consistency, if not all cpus are overriden, running tasks is disallowed).
Details:
- update struct dl_bandwidth to store per-cpu runtime and period overrides.
- introduce tg_set_rt_bandwidth_override to set a cpuset's override.
- update schedulability/admission checks.
- update cpu.rt.max interface to support per-cpu overrides.
--- kernel/sched/core.c | 89 ++++++++++++++- kernel/sched/deadline.c | 23 ++++ kernel/sched/rt.c | 237 ++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 7 ++ 4 files changed, 331 insertions(+), 25 deletions(-)
kernel/sched/core.c +86 -3
@@ -10494,9 +10494,20 @@ static int cpu_rt_max_show(struct seq_file *sf, void *v) {struct task_group *tg =css_tg (seq_css (sf));u64 period_us, runtime_us; +int cpu;tg_rt_bandwidth (tg, &period_us, &runtime_us);cpu_period_quota_print (sf, period_us, runtime_us); + +if (tg == &root_task_group) +return 0 ; + +for_each_possible_cpu (cpu) { +tg_rt_single_bandwidth (tg, cpu, &period_us, &runtime_us); +if (runtime_us !=RUNTIME_INF ) +cpu_period_quota_single_print (sf, cpu, period_us, runtime_us); + } +return 0 ; }
@@ -10510,19 +10521,91 @@ static int cpu_rt_internal_show(struct seq_file *sf, void *v)tg_rt_internal_bandwidth (tg, cpu, &period_us, &runtime_us); cpu_period_quota_single_print(sf, cpu, period_us, runtime_us); } +return 0 ; } +static int __cpu_rt_max_parse (char *buf,struct cpumask *mask, +u64 *period_us_p,u64 *quota_us_p) +{ +char *tok, *runtime_key, *period_key; + + tok =strsep (&buf," " ); +if (!tok) +return -EINVAL ; + +if (strncmp (tok,"max" ,sizeof ("max" )) ==0 ) { + *quota_us_p =RUNTIME_INF ; +return 0 ; + } + +if (strncmp (tok,"default" ,sizeof ("default" )) ==0 ) { +if (strncmp (buf,"runtime=max" ,sizeof ("runtime=max" )) ==0 ) { + *quota_us_p =RUNTIME_INF ; + *period_us_p =0 ; +return 0 ; + } + +if (!buf) +return -EINVAL ; + + goto parse_keys; + } + +if (buf &&sscanf (buf,"%llu" , period_us_p) ==0 ) { +if (cpulist_parse (tok, mask) !=0 ) +return -EINVAL ; + + goto parse_cpumask; + } + +if (sscanf (tok,"%llu" , quota_us_p) <1 ) +return -EINVAL ; + +return 0 ; + +parse_cpumask: +if (strncmp (buf,"default" ,sizeof ("default" )) ==0 ) { + *quota_us_p =RUNTIME_INF ; + *period_us_p =RUNTIME_INF ; +return 0 ; + } + +parse_keys: + runtime_key =strstr (buf,"runtime=" ); + period_key =strstr (buf,"period=" ); + +if (runtime_key && + period_key && +sscanf (runtime_key,"runtime=%llu" , quota_us_p) ==1 && +sscanf (period_key,"period=%llu" , period_us_p) ==1 ) +return 0 ; + +return -EINVAL ; +} +static ssize_t cpu_rt_max_write (struct kernfs_open_file *of,char *buf,size_t nbytes,loff_t off) {struct task_group *tg =css_tg (of_css (of)); -u64 period_us =0 , runtime_us; + cpumask_var_t mask __free(free_cpumask_var) = CPUMASK_VAR_NULL; +u64 period_us, runtime_us;int ret; - ret =cpu_period_quota_parse (buf, &period_us, &runtime_us); -if (!ret) +if (!alloc_cpumask_var (&mask,GFP_NOWAIT )) +return -ENOMEM ; + +cpumask_clear (mask); + + ret =__cpu_rt_max_parse (buf, mask, &period_us, &runtime_us); +if (ret) +return ret; + +if (cpumask_empty (mask)) ret =tg_set_rt_bandwidth (tg, period_us, runtime_us); +else + ret =tg_set_rt_bandwidth_override (tg, mask, period_us, runtime_us); +return ret ?: nbytes; }#endif /* CONFIG_RT_GROUP_SCHED */
kernel/sched/deadline.c +23 -0
@@ -593,24 +593,47 @@ static inline int is_leftmost(struct sched_dl_entity *dl_se, struct dl_rq *dl_rqstatic void init_dl_rq_bw_ratio (struct dl_rq *dl_rq);struct dl_bandwidth *alloc_dl_bandwidth (struct dl_bandwidth *dl_b,gfp_t gfp_flags) { + u64 *period_overrides __free(kfree) = NULL; + u64 *runtime_overrides __free(kfree) = NULL; + + period_overrides =kcalloc (nr_cpu_ids,sizeof (u64), gfp_flags); +if (!period_overrides) +return NULL ; + + runtime_overrides =kcalloc (nr_cpu_ids,sizeof (u64), gfp_flags); +if (!runtime_overrides) +return NULL ; +if (!alloc_cpumask_var (&dl_b->active_mask, gfp_flags))return NULL ; + dl_b->dl_period_overrides =no_free_ptr (period_overrides); + dl_b->dl_runtime_overrides =no_free_ptr (runtime_overrides); +return dl_b; }void free_dl_bandwidth (struct dl_bandwidth *dl_b) {free_cpumask_var (dl_b->active_mask); +kfree (dl_b->dl_period_overrides); +kfree (dl_b->dl_runtime_overrides); }void init_dl_bandwidth (struct dl_bandwidth *dl_b,u64 period,u64 runtime,struct task_group *active_context) { +int cpu; +raw_spin_lock_init (&dl_b->dl_runtime_lock); dl_b->dl_period = period; dl_b->dl_runtime = runtime; dl_b->active_context = active_context;cpumask_clear (dl_b->active_mask); + +for_each_possible_cpu (cpu) { + dl_b->dl_period_overrides[cpu] =RUNTIME_INF ; + dl_b->dl_runtime_overrides[cpu] =RUNTIME_INF ; + } }
kernel/sched/rt.c +215 -22
318 lines not on this page · raw message
kernel/sched/sched.h +7 -0
@@ -317,6 +317,8 @@ struct dl_bandwidth {raw_spinlock_t dl_runtime_lock;u64 dl_runtime;u64 dl_period; +u64 *dl_runtime_overrides; +u64 *dl_period_overrides;cpumask_var_t active_mask;struct task_group *active_context; };
@@ -586,10 +588,15 @@ extern bool cfs_task_bw_constrained(struct task_struct *p);extern int tg_rt_bandwidth (struct task_group *tg,u64 *rt_period_us,u64 *rt_runtime_us); +extern int tg_rt_single_bandwidth (struct task_group *tg,int cpu, +u64 *rt_period_us,u64 *rt_runtime_us);extern int tg_rt_internal_bandwidth (struct task_group *tg,int cpu,u64 *rt_period_us,u64 *rt_runtime_us);extern int tg_set_rt_bandwidth (struct task_group *tg,u64 rt_period_us,u64 rt_runtime_us); +extern int tg_set_rt_bandwidth_override (struct task_group *tg, +const struct cpumask *rt_mask, +u64 rt_period_us,u64 rt_runtime_us);extern int tg_can_offline_cpu (int cpu);extern int sched_rt_can_attach (struct task_group *tg);
Signature · 1 line
2.56.0
Tejun Heo
maintainerHello, Yuri.
I only looked at the interface and the user-visible behavior, not the implementation.
Quoted 2 lines · Yuri Andriaccio
- Rework cgroup-v2 interface:
+ Rename cpu.rt.internal to cpu.rt.effective.local
The new name still doesn't follow the cgroup2 naming convention. Every .effective and .local file is a suffix on an existing file:
cpuset.cpus.effective, cpuset.cpus.exclusive.effective, memory.events.local, cpu.stat.local. There is no cpu.rt file, so can we go with cpu.rt.max.effective.local as suggested for v6?
The file prints a "CPU runtime=R period=P" line for each possible CPU. It does so even when the value is the same on every CPU, and the CPUs that the budget doesn't cover are listed with 0. 1k CPU machines are becoming wide-spread as we speak. This isn't a scalable interface.
Switching between "max" and a budget is refused while the cgroup or its "max" descendants have RT tasks. That's okay as a restriction for now. Ideally, the switch wouldn't be restricted as long as the budget is sufficient, but that can be left for the future.
Quoted 1 line
cpuandcpusetcontrollers integration:
...
Quoted 4 lines
first setup the
cpusetcpus and later setup the HCBS bandwidth. The current integration disallows changing allowed cpus while the HCBS bandwidth is reserved (thus the admin must first remove the HCBS reservation, change the cpuset, and re-request real-time bandwidth).
...
Quoted 4 lines
dl_serveroverride feature:
Following thecpusetintegration, we integrated the multi-runtime/multi-CPU
feature: the admin can request the override of the default runtime/period pair for a specific set of CPUs.
This is my main objection. v7 sits in the middle on both axes. Per-CPU config is allowed but only to a degree: an override must be within the cpuset, can't be zero and isn't allowed on a "max" cgroup. cpuset is integrated but not fully: the budget covers the cpuset as it was when cpu.rt.max was written, and cpuset changes and CPU offlining are refused while budgets exist.
Can we separate cpu.rt.max from cpuset? I can see the following paths.
1. Make the allocation global. The interface then becomes system-wide, and cpuset and CPU hotplug become transparent for the most part.
2. Make the interface per-CPU. The overrides mostly make it so already. Config entries are keyed by cpulists only, with nested keys like io.max. Reads collapse the CPUs that share a config into one cpulist entry:
0-1,4-5,7 runtime=400000 period=1000000
2-3,6 runtime=800000 period=10000003. If you want simplicity while staying per-CPU, make every cgroup's cpu.rt.max allocate on all CPUs.
In any of these, cpuset and cpu.rt.max are orthogonal, and what users do with intersecting configs is their problem.
On the per-CPU budgets themselves, they don't match cpu.max, and giving a budget per CPU is a bit odd when the consumption isn't per CPU. The threads move around. Maybe it can't be avoided, but I'd like to see a discussion of the rationale behind the design choice and some exploration of alternatives.
Quoted 3 lines
Hotplug does not cause issues but HCBS reservations will need to be updated to make use of the newly enabled core (the cgroups' HCBS specific cpumask is only updated when setting the cgroups' bandwidth).
The cpuset coupling creates all sorts of problems. AFAICS, disabling the cpuset controller above a budgeted cgroup moves it to the ancestor's cpuset and changes its effective CPUs without going through the lockout at all. A newly onlined CPU joins cpuset.cpus.effective and the cgroup's tasks can run there, but the budget doesn't cover it until cpu.rt.max is rewritten. A partition change in a sibling isn't refused and can still move CPUs underneath it.
What users configure on the config surface should always match what's in effect. Interfaces that let internal state deviate from that are almost always bad ideas and long-term headaches. Hopefully, the lockout can go away eventually.
Quoted 4 lines
- PI-boosting will cause starvation if cgroups have zero (internal) runtime. There are many solutions, such as bandwidth inheritance or forcing cgroups to not have zero runtime, but this has yet to be investigated. We are open for advice on the matter.
Zero runtime is a problem without PI too. A budgeted cgroup's own servers get its budget minus what its budgeted children take. When the children take all of it, the tasks running on the cgroup's own servers stall. Those are the internal threads of a threaded cgroup and the tasks in its "max" descendants, and the latter doesn't need threaded mode. The tasks are still accepted because the attach and setscheduler checks look at the configured budget, not what's left.
One option is to treat the RT tasks on a cgroup's servers as usage in admission. Config changes that would leave a cgroup that has RT tasks with zero runtime would be refused, and attach and setscheduler would check what's left instead of the configured budget. There may be better ways.
PI doesn't need to be perfect. Skipping the boost when the target has no runtime is fine as long as nothing stalls.
Thanks.
--
tejun
Hello Tejun,
Thanks for taking a look. I'm answering some of your comments first and at the end I'm proposing a summary of changes that will appear in the next RFC to then "stabilize" the user interface.
Quoted 1 line
The new name still doesn't follow the cgroup2 naming convention.
Thanks for pointing this out, it was a mistake on my end. This will be fixed in the next RFC.
Quoted 1 line
The file prints a "CPU runtime=R period=P" line for each possible CPU.
We will condense the displaying for same runtime/period CPUs and omit zeroed ones.
Quoted 4 lines
Switching between "max" and a budget is refused while the cgroup or its "max" descendants have RT tasks. That's okay as a restriction for now. Ideally, the switch wouldn't be restricted as long as the budget is sufficient, but that can be left for the future.
The restriction is in place to simplify the initial implementation: switching between "max" and non-"max" requires to migrate all tasks from one cgroup's runqueues to another cgroup ones, and could potentially be an expensive operation if there are lots of tasks. If you think the restriction is acceptable (for now), we will keep this unchanged in hte next RFC, and fix it later.
Quoted 1 line
Can we separate cpu.rt.max from cpuset?
Maybe we misunderstood, but in some previous discussion it was asked us to integrate the cpuset and cpu controllers somehow. Your proposal of having the cpu and cpuset controllers orthogonal simplifies things, but it is similar to the previous (cgroup-v1-based) implementation, or before the integration patch itself (i.e. just allow the per-CPU overrides but remove the tighter coupling). What is the final verdict on this?
Quoted 5 lines
On the per-CPU budgets themselves, they don't match cpu.max, and giving a budget per CPU is a bit odd when the consumption isn't per CPU. The threads move around. Maybe it can't be avoided, but I'd like to see a discussion of the rationale behind the design choice and some exploration of alternatives.
I'm not sure if I'm understanding correctly here, but the budget for cpu.rt.max, differently from cpu.max, is meant to be per-CPU and not global. Sure the initial patches have a single setting that is the same on all CPUs, but it was meant to be a per-CPU limit that is the same across all cores. Notice that threads move around, but we prevent migrating them to cores with zero runtime (hence, there is some overlap of functionality between cpusets and rt-cgroups).
The rationale behind this is that, from a real-time theoretical pov, it is necessary to know how much budget and how it is distributed on the CPUs, as changing the number of CPUs or the budget distribution radically changes the schedulability outputs. The most complete and flexible interface is therefore having per-CPU budgets.
This is also necessary to prevent over-provisioning and allowing partitioning of the total bandwidth:
- If implementing solution 1, a request of 10% of budget reserves 10% of runtime on all cores, regardless of cpuset settings. Of course a cpuset mask that is smaller than the total allows limiting the tasks on a subset of CPUs, yet the budget on the unused cores is still allocated, and therefore wasted. This is a case of over-provisioning.
- On the other hand, allowing per-CPU controls allows to potentially partition the total bandwidth in the CPU hierarchy, for example reserving some 10% runtime on some cores, while still allowing other cgroups to make use of the unused 10% bandwidth on the other cores.
From this limited discussion it appears to us that solution 2, which is similar to what we already had but without the "default" case, is the most suitable. Solutions 1 and 3 are more or less what already exists before the cpuset integration patches.
Quoted 3 lines
AFAICS, disabling the cpuset controller above a budgeted cgroup moves it to the ancestor's cpuset and changes its effective CPUs without going through the lockout at all.
You are right, we tried to prevent changes in the cpuset when an active rt-cgroup was involved but this was clearly missed out.
The lockout is in place because, in the current integration, it is necessary to rerun the bandwdith admission test on every cpuset change, which greatly complicates the code. Nonetheless, a more permissive integration still needs to couple updates in both controllers and run some admission tests.
Quoted 2 lines
Skipping the boost when the target has no runtime is fine as long as nothing stalls.
For PI and zero-budget cgroups, if this is an acceptable solution, it seems the most straightforward one: disallow switching to RT for zeroed cgroups and disable zeroing the budget of any parent which has active RT tasks.
- - -
As mentioned at the beginning, here is a summary of what the interface would be after the discussed points:
- cpu.rt.max and cpu.rt.max.effective.local files. When displaying the per-CPU settings, zero runtimes are skipped and same runtime/period cpus are condensed in the same cpuset.
- cpu.rt.max only accepts per-CPU settings, so the global "default" is removed. Removing all overrides means the cgroup is a "max" cgroup, while just a single override means that the cgroup has its own budget.
+ This new interface disallows zero-runtime overrides, as a zero override is equal to an empty override when there is at least one non-zero override. As an example: [<CPU0 10ms 100ms>, <CPU1 0ms 100ms>] is equal to just having [<CPU0 10ms 100ms>]. This seems a bit odd, how to make this better?
- Switching between max and non-max is still disallowed for active real-time cgroups (to be left for the future).
- cpu and cpuset controllers are decoupled, as it was before the cpuset integration patch. The user/admin is responsible for inconsistencies between the two controllers (i.e. empty intersection between active server in the rt-cgroup and the allowed cpuset just disallows executing RT tasks entirely).
- zero "internal" runtime cgroups will disallow RT tasks and prevent priority boosting. Admission controls will disallow zeroing the "internal" budget of a parent which has active tasks running.
Hopefully we can agree on the ideas posted here. If you have any doubts or better suggestions let us know.
Thanks,
Yuri
Juri Lelli
maintainerOn 11/11/26 18:56, Yuri Andriaccio wrote:
...
Quoted 4 lines
-
cpuandcpusetcontrollers are decoupled, as it was before thecpusetintegration patch. The user/admin is responsible for inconsistencies between the two controllers (i.e. empty intersection between active server in the rt-cgroup and the allowed cpuset just disallows executing RT tasks entirely).
Curious to see what Tejun (and others) think about this and the other points, but FWIW I'd also lean towards decoupling.
It looks like the enforcement you need for it is already in the
patchset: RT tasks can't be placed on CPUs where their cgroup's server has no runtime (find_lowest_rt_rq() skips them). So an empty intersection between a cgroup's per-CPU entries and its cpuset just means no RT execution there, and that's the user's configuration, same as an isolated partition with no tasks in it.
Thanks,
Juri
Tejun Heo
maintainerHello, Yuri.
The summary sounds fine for the most part. Some comments below.
Quoted 7 lines · Yuri Andriaccio
-
cpu.rt.maxonly accepts per-CPU settings, so the global "default" is removed. Removing all overrides means the cgroup is a "max" cgroup, while just a single override means that the cgroup has its own budget.
+ This new interface disallows zero-runtime overrides, as a zero override is equal to an empty override when there is at least one non-zero override. As an example: [<CPU0 10ms 100ms>, <CPU1 0ms 100ms>] is equal to just having [<CPU0 10ms 100ms>]. This seems a bit odd, how to make this better?
It's odd because a missing entry means "max" when there are no entries and zero when there are some. Can a missing entry always mean "max", per CPU? On a CPU that a cgroup has no entry for, its tasks would be served by the nearest ancestor which has one. Zero then becomes a regular value which is shown on read, and the two configs in your example are different. The former can't run RT tasks on CPU1. The latter runs them on the ancestor's budget there.
This means that a cgroup can have its own budget on some CPUs and be "max" on the others. That is a fundamental part of the interface, so I think it's best to get it right from the get-go rather than starting with a restriction.
The root has no ancestor to fall back to, so a missing entry can't mean anything there. It has to always carry an entry for every possible CPU, which defaults to zero, and reject "max":
0-127 runtime=0 period=1000000
Quoted 4 lines
-
cpuandcpusetcontrollers are decoupled, as it was before thecpusetintegration patch. The user/admin is responsible for inconsistencies between the two controllers (i.e. empty intersection between active server in the rt-cgroup and the allowed cpuset just disallows executing RT tasks entirely).
Yes, let's decouple them. With that, can the CPU offline lockout go away too? An entry for an offline CPU can stay configured and take effect again when the CPU comes back.
One more thing on the unprivileged FIFO/RR tasks in cgroups. No other cgroup resource config grants privileges that way. e.g. If an rlimit is set on memory, memory.low doesn't do anything to work around it. I think who can use RT policies should be left to system configuration, i.e. RLIMIT_RTPRIO and CAP_SYS_NICE. Can you drop that patch?
Thanks.
--
tejun
Juri Lelli
maintainerHello,
Quoted 3 lines · Tejun Heo
Hello, Yuri.
The summary sounds fine for the most part. Some comments below.
Sorry to interject again, but I had a thought that I would like to share. Also, the very similar names probably don't help keeping track of who said what, but anyway. :P
Quoted 17 lines · Yuri Andriaccio
Quoted 7 lines
-
cpu.rt.maxonly accepts per-CPU settings, so the global "default" is removed. Removing all overrides means the cgroup is a "max" cgroup, while just a single override means that the cgroup has its own budget.
+ This new interface disallows zero-runtime overrides, as a zero override is equal to an empty override when there is at least one non-zero override. As an example: [<CPU0 10ms 100ms>, <CPU1 0ms 100ms>] is equal to just having [<CPU0 10ms 100ms>]. This seems a bit odd, how to make this better?It's odd because a missing entry means "max" when there are no entries and zero when there are some. Can a missing entry always mean "max", per CPU? On a CPU that a cgroup has no entry for, its tasks would be served by the nearest ancestor which has one. Zero then becomes a regular value which is shown on read, and the two configs in your example are different. The former can't run RT tasks on CPU1. The latter runs them on the ancestor's budget there.
Quoted 5 lines
This means that a cgroup can have its own budget on some CPUs and be "max" on the others. That is a fundamental part of the interface, so I think it's best to get it right from the get-go rather than starting with a restriction.
Agree on making zero a regular value and on a missing entry always meaning "max". There is one case where "the ancestor's budget" doesn't exist though, and that is when the walk ends at the root.
The root doesn't have a server. Its cpu.rt.max is the pool of SCHED_DEADLINE bandwidth that child servers are carved from, and FIFO/RR tasks served by the root just run in the root runqueue, outside HCBS (as with !RT_GROUP_SCHED). So take a top level group with
0 runtime=10000 period=100000
no entry for CPU1, and a few while(1) FIFO tasks allowed on CPUs 0-1. On CPU0 they are throttled by the group's server. Once they get to CPU1 (wakeup placement, push/pull, affinity changes) they run in the root runqueue and nothing throttles them, only DL tasks and the fair/ext servers. The group reads as 10% of CPU0, root reads as runtime=100000 everywhere, and the admin reasonably thinks the group is contained. It isn't.
Making a missing entry mean zero at the top would bring back the ambiguity you pointed out, so what about keeping your semantics and adding a validity rule instead:
A cgroup can be served by the root either on all of its CPUs or on none of them.
In practice, a cgroup whose parent is entirely root served (e.g. a top level group) has entries for all possible CPUs or for none. A write that would leave some CPUs served by a server (or zero) and others by the root fails with -EINVAL, like the existing admission checks. Further down, a missing entry is fine and means the parent's context on that CPU, which is always a real server or an explicit zero.
With that, per CPU:
- no entry: served by the parent's context on that CPU
- runtime=0: no RT on that CPU, shown on read
- runtime=N period=P: own server on that CPU
A missing entry never changes meaning; the rule only rejects the configurations where it would escape HCBS. For the example above, the group must say "1 runtime=0 ..." (no RT on CPU1) or give CPU1 a budget. The same config under a parent with servers on CPUs 0-1 is accepted, and CPU1 runs on the parent's server.
What do you think?
Quoted 5 lines
The root has no ancestor to fall back to, so a missing entry can't mean anything there. It has to always carry an entry for every possible CPU, which defaults to zero, and reject "max":
0-127 runtime=0 period=1000000
Also, related to the above: on the root these numbers mean something else (how much DL bandwidth cgroups may reserve, not a server for the root's tasks), and showing them in the same file and format is part of what makes the example misleading. Would you be OK with dropping cpu.rt.max on the root and exposing the parameters through a separately named root-only file (or the existing sysctls)? That way, having no entry all the way up consistently means unmanaged FIFO/RR, and nothing on the root looks like a budget those tasks share.
Thanks!
Juri
Hi Juri,
On Thu, 19 Nov 2026 12:23:48 +0100
Juri Lelli <[email protected]> wrote:
[...]
Quoted 19 lines
Quoted 19 lines · Yuri Andriaccio
Quoted 9 lines
-
cpu.rt.maxonly accepts per-CPU settings, so the global "default" is removed. Removing all overrides means the cgroup is a "max" cgroup, while just a single override means that the cgroup has its own budget.
+ This new interface disallows zero-runtime overrides, as a
zero override is equal to an empty override when there is at least one non-zero override. As an example: [<CPU0 10ms 100ms>, <CPU1 0ms 100ms>] is equal to just having [<CPU0 10ms 100ms>]. This seems a bit odd, how to make this better?It's odd because a missing entry means "max" when there are no entries and zero when there are some. Can a missing entry always mean "max", per CPU? On a CPU that a cgroup has no entry for, its tasks would be served by the nearest ancestor which has one. Zero then becomes a regular value which is shown on read, and the two configs in your example are different. The former can't run RT tasks on CPU1. The latter runs them on the ancestor's budget there.
Quoted 8 lines
Quoted 5 lines
This means that a cgroup can have its own budget on some CPUs and be "max" on the others. That is a fundamental part of the interface, so I think it's best to get it right from the get-go rather than starting with a restriction.
Agree on making zero a regular value and on a missing entry always meaning "max".
I think "0" as a regular value is agreed upon by everyone :) But I would avoid "missing entries meaning max"... See my previous email.
Basically, I think a cgroup should either inherit from the parent for every core, or define a deadline entity (even one with 0 runtime) on every core. Mixing the two is, in my opinion, problematic (for example, it would change the serving reservation when a task migrates from one core to another)).
I hope this "all-or-nothing" approach is acceptable...
[...]
Quoted 6 lines
Making a missing entry mean zero at the top would bring back the ambiguity you pointed out, so what about keeping your semantics and adding a validity rule instead:
A cgroup can be served by the root either on all of its CPUs or on none of them.
As mentioned above, I would be even more drastic:
A cgroup can be served by an ancestor either on all of its CPUs or on none of them.
What do you think? Can this approach be acceptable?
Quoted 8 lines
In practice, a cgroup whose parent is entirely root served (e.g. a top level group) has entries for all possible CPUs or for none. A write that would leave some CPUs served by a server (or zero) and others by the root fails with
-EINVAL, like the existing admission checks. Further down, a missing entry is fine and means the parent's context on that CPU, which is always a real server or an explicit zero.
Quoted 5 lines
With that, per CPU:
- no entry: served by the parent's context on that CPU
- runtime=0: no RT on that CPU, shown on read
- runtime=N period=P: own server on that CPU
If what I wrote above is OK, I would just avoid the "no entry" case, and I would define the interface so that missing entries are not possible (for example, see my previous email: we first set a (runtime, period) for all core, then we can override some of them... But not with "max").
Quoted 6 lines
Quoted 5 lines
The root has no ancestor to fall back to, so a missing entry can't mean anything there. It has to always carry an entry for every possible CPU, which defaults to zero, and reject "max":
0-127 runtime=0 period=1000000
Quoted 8 lines
Also, related to the above: on the root these numbers mean something else (how much DL bandwidth cgroups may reserve, not a server for the root's tasks), and showing them in the same file and format is part of what makes the example misleading. Would you be OK with dropping cpu.rt.max on the root and exposing the parameters through a separately named root-only file (or the existing sysctls)? That way, having no entry all the way up consistently means unmanaged FIFO/RR, and nothing on the root looks like a budget those tasks share.
Regardless of the filenames we decide, the defaults for these settings indicate the fraction of CPU time that can be consumed by all real-time tasks that are not in the root cgroup, right?
Thanks,
Luca
Juri Lelli
maintainerQuoted 49 lines · luca abeni
Hi Juri,
On Thu, 19 Nov 2026 12:23:48 +0100
Juri Lelli <[email protected]> wrote:
[...]Quoted 19 lines
Quoted 19 lines · Yuri Andriaccio
Quoted 9 lines
-
cpu.rt.maxonly accepts per-CPU settings, so the global "default" is removed. Removing all overrides means the cgroup is a "max" cgroup, while just a single override means that the cgroup has its own budget.
+ This new interface disallows zero-runtime overrides, as a
zero override is equal to an empty override when there is at least one non-zero override. As an example: [<CPU0 10ms 100ms>, <CPU1 0ms 100ms>] is equal to just having [<CPU0 10ms 100ms>]. This seems a bit odd, how to make this better?It's odd because a missing entry means "max" when there are no entries and zero when there are some. Can a missing entry always mean "max", per CPU? On a CPU that a cgroup has no entry for, its tasks would be served by the nearest ancestor which has one. Zero then becomes a regular value which is shown on read, and the two configs in your example are different. The former can't run RT tasks on CPU1. The latter runs them on the ancestor's budget there.
Quoted 8 lines
Quoted 5 lines
This means that a cgroup can have its own budget on some CPUs and be "max" on the others. That is a fundamental part of the interface, so I think it's best to get it right from the get-go rather than starting with a restriction.
Agree on making zero a regular value and on a missing entry always meaning "max".
I think "0" as a regular value is agreed upon by everyone :) But I would avoid "missing entries meaning max"... See my previous email.
Basically, I think a cgroup should either inherit from the parent for every core, or define a deadline entity (even one with 0 runtime) on every core. Mixing the two is, in my opinion, problematic (for example, it would change the serving reservation when a task migrates from one core to another)).
I hope this "all-or-nothing" approach is acceptable...[...]
Quoted 4 lines
Making a missing entry mean zero at the top would bring back the ambiguity you pointed out, so what about keeping your semantics and adding a validity rule instead:
Quoted 8 lines
Quoted 2 lines
A cgroup can be served by the root either on all of its CPUs or on none of them.
As mentioned above, I would be even more drastic:
A cgroup can be served by an ancestor either on all of its CPUs or on none of them.What do you think? Can this approach be acceptable?
Yes, works for me. The example I gave is just the extreme case of the problem you describe (the "other reservation" being no reservation at all), and your argument covers the nested case I still allowed.
Tejun, I think this still addresses the original concern: zero is a regular value with a single meaning ("no RT on this CPU"), and no entry only exists for a cgroup that is "max" as a whole. It's not a restriction to relax later, though: a task changing reservation on migration is what we want to rule out.
Quoted 14 lines
Quoted 8 lines
In practice, a cgroup whose parent is entirely root served (e.g. a top level group) has entries for all possible CPUs or for none. A write that would leave some CPUs served by a server (or zero) and others by the root fails with
-EINVAL, like the existing admission checks. Further down, a missing entry is fine and means the parent's context on that CPU, which is always a real server or an explicit zero.Quoted 5 lines
With that, per CPU:
- no entry: served by the parent's context on that CPU
- runtime=0: no RT on that CPU, shown on read
- runtime=N period=P: own server on that CPU
Quoted 5 lines
If what I wrote above is OK, I would just avoid the "no entry" case, and I would define the interface so that missing entries are not possible (for example, see my previous email: we first set a (runtime, period) for all core, then we can override some of them... But not with "max").
OK.
Quoted 15 lines
Quoted 6 lines
Quoted 5 lines
The root has no ancestor to fall back to, so a missing entry can't mean anything there. It has to always carry an entry for every possible CPU, which defaults to zero, and reject "max":
0-127 runtime=0 period=1000000
Quoted 8 lines
Also, related to the above: on the root these numbers mean something else (how much DL bandwidth cgroups may reserve, not a server for the root's tasks), and showing them in the same file and format is part of what makes the example misleading. Would you be OK with dropping cpu.rt.max on the root and exposing the parameters through a separately named root-only file (or the existing sysctls)? That way, having no entry all the way up consistently means unmanaged FIFO/RR, and nothing on the root looks like a budget those tasks share.
Quoted 3 lines
Regardless of the filenames we decide, the defaults for these settings indicate the fraction of CPU time that can be consumed by all real-time tasks that are not in the root cgroup, right?
Not quite, I think? They cap the bandwidth that can be reserved for cgroup servers, so they only cover RT tasks that are served by one. RT tasks in a non-root "max" cgroup (with no configured ancestor) are not in the root cgroup, but run in the root runqueue and aren't limited by those values at all (sched_rt_can_attach() lets them in unconditionally). Which is the kind of confusion I'd like the root interface to avoid.
Thanks,
Juri
Quoted 16 lines · On Thu, 19 Nov 2026 15:36:01 +0100 Juri Lelli <[email protected]> wrote:
Quoted 16 lines
Quoted 15 lines
Quoted 5 lines
The root has no ancestor to fall back to, so a missing entry can't mean anything there. It has to always carry an entry for every possible CPU, which defaults to zero, and reject "max":
0-127 runtime=0 period=1000000
Also, related to the above: on the root these numbers mean something else (how much DL bandwidth cgroups may reserve, not a server for the root's tasks), and showing them in the same file and format is part of what makes the example misleading. Would you be OK with dropping cpu.rt.max on the root and exposing the parameters through a separately named root-only file (or the existing sysctls)? That way, having no entry all the way up consistently means unmanaged FIFO/RR, and nothing on the root looks like a budget those tasks share.
Quoted 8 lines
Quoted 3 lines
Regardless of the filenames we decide, the defaults for these settings indicate the fraction of CPU time that can be consumed by all real-time tasks that are not in the root cgroup, right?
Not quite, I think? They cap the bandwidth that can be reserved for cgroup servers, so they only cover RT tasks that are served by one. RT tasks in a non-root "max" cgroup (with no configured ancestor) are not in the root cgroup, but run in the root runqueue
Ah, right, sorry about the confusion... I was counting them as belonging to the root cgroup, but you are right, they are in different cgroups.
So, yes, I was victim of the confusion you'd like to avoid :)
Luca
Quoted 8 lines
and aren't
limited by those values at all (sched_rt_can_attach()lets them in unconditionally). Which is the kind of confusion I'd like the root interface to avoid.Thanks,
Juri
Hi all,
first of all, sorry about the delay, Yuri and I have been discussing these issues for some time before replying. See our conclusions below.
On Tue, 17 Nov 2026 22:07:04 -1000
Tejun Heo <[email protected]> wrote:
[...]
Quoted 11 lines · Yuri Andriaccio
Quoted 9 lines
-
cpu.rt.maxonly accepts per-CPU settings, so the global "default" is removed. Removing all overrides means the cgroup is a "max" cgroup, while just a single override means that the cgroup has its own budget.
+ This new interface disallows zero-runtime overrides, as a zero
override is equal to an empty override when there is at least one non-zero override. As an example: [<CPU0 10ms 100ms>, <CPU1 0ms 100ms>] is equal to just having [<CPU0 10ms 100ms>]. This seems a 100ms>bit odd, how to make this better?
Quoted 2 lines
It's odd because a missing entry means "max" when there are no entries and zero when there are some.
Yes, we agree this behaviour looks wrong, and should be removed, however...
Quoted 3 lines
Can a missing entry always mean "max",
per CPU? On a CPU that a cgroup has no entry for, its tasks would be served by the nearest ancestor which has one.
... We believe that mixing "max" cores and cores configured with (runtime, period) in the same cgroup is problematic, for various reason. From the theoretical point of view, this would mean that when migrating between cores a task can "move" from one reservation to a different one, creating issues for schedulability analysis. All the (runtime, period) deadline entities in a cgroup form a multi-core reservation, as in MPR
(https://repository.upenn.edu/entities/publication/5cd778fe-4d0b-45a9-ae9e-cef48209bd8a)
if some cores are "max" and some are attached to (runtime, period) deadline entities, the tasks move between two different multi-core reservations, without control.
So, what about a different idea: a cgroup can be "max" or have (runtime,period); when we set (runtime,period), this is set for all the CPU cores, but then we can override (with runtime 0, or different runtime/period) some cores (but we cannot override with "max"). This way, "0" still has one single meaning (no real-time tasks on this core), but we do not mix "max" and "(runtime,period)" in the same cgroup... Would this be acceptable?
Quoted 4 lines
Zero then becomes a
regular value which is shown on read, and the two configs in your example are different. The former can't run RT tasks on CPU1. The latter runs them on the ancestor's budget there.
We believe this property can be valid with the proposal above, too.
[...]
Quoted 5 lines
The root has no ancestor to fall back to, so a missing entry can't mean anything there. It has to always carry an entry for every possible CPU, which defaults to zero, and reject "max":
0-127 runtime=0 period=1000000
We think 0 as a default would be problematic, as it means "no real-time tasks", and 0 in the root means "no real-time tasks in all the cgroups"... Or am I missing your point?
Quoted 9 lines
Quoted 5 lines
-
cpuandcpusetcontrollers are decoupled, as it was before thecpusetintegration patch. The user/admin is responsible for inconsistencies between the two controllers (i.e. empty intersection between active server in the rt-cgroup and the allowed cpuset just disallows executing RT tasks entirely).Yes, let's decouple them. With that, can the CPU offline lockout go away too? An entry for an offline CPU can stay configured and take effect again when the CPU comes back.
Let's decouple them, then. I am not sure about offline CPUs.
Quoted 5 lines
One more thing on the unprivileged FIFO/RR tasks in cgroups. No other cgroup resource config grants privileges that way. e.g. If an rlimit is set on memory, memory.low doesn't do anything to work around it. I think who can use RT policies should be left to system configuration, i.e.
RLIMIT_RTPRIOandCAP_SYS_NICE. Can you drop that patch?
OK, let's drop this patch.
Thanks,
Luca
Tejun Heo
maintainerHello, Luca.
I'm only thinking about this from the interface POV, so please read the following with that in mind.
Quoted 5 lines · luca abeni
... We believe that mixing "max" cores and cores configured with (runtime, period) in the same cgroup is problematic, for various reason. From the theoretical point of view, this would mean that when migrating between cores a task can "move" from one reservation to a different one, creating issues for schedulability analysis.
...
Quoted 4 lines
So, what about a different idea: a cgroup can be "max" or have (runtime,period); when we set (runtime,period), this is set for all the CPU cores, but then we can override (with runtime 0, or different runtime/period) some cores (but we cannot override with "max").
If the interface is per-CPU, I think it should be per-CPU all the way. What a cgroup has on one CPU shouldn't say anything about what it has on another. That seems the most consistent in itself and with the other controllers.
Take Juri's example of a top-level group which has an entry only for CPU0. Its tasks are limited on CPU0 and not on CPU1, which is what the config says. io.max works the same way. A limit on one device says nothing about the other devices, and a device without an entry is at "max". If the group shouldn't run RT tasks on CPU1, it can be given runtime=0 there.
Tasks moving across CPUs doesn't change how the resources are allocated. A task consumes from whatever serves its cgroup on the CPU it's running on.
I'm not too familiar with the schedulability analysis side, so please correct me if I'm missing something. Wouldn't it work if the setups which need the analysis configure the group on all CPUs? That seems like something which can be a convention in how groups are configured rather than forced through the interface, which looks like it'd get clunky whichever shape we pick.
Quoted 3 lines
We think 0 as a default would be problematic, as it means "no real-time tasks", and 0 in the root means "no real-time tasks in all the cgroups"... Or am I missing your point?
Yeah, zero doesn't work there. How about defaulting the root to "max" like the other cgroups? "max" on the root would mean that nothing is set aside for cgroups. RT tasks which aren't under any entry run without a server as they do now, and the budgets of top-level cgroups are admitted against the DL bandwidth directly. An entry on the root would be the total that cgroup servers can reserve on that CPU, and writing one fails if what's already reserved doesn't fit. RT tasks in the root itself keep running without a server either way.
Thanks.
--
tejun
Hi Tejun,
Quoted 4 lines · On Thu, 19 Nov 2026 20:46:18 -1000 Tejun Heo <[email protected]> wrote:
Hello, Luca.
I'm only thinking about this from the interface POV, so please read the following with that in mind.
I understand (I hope :) the interface issue, and I'd like to find a compromise that works from both an interface and a "performance evaluation" perspective.
So below I'm trying to clarify the performance evaluation issues with an example (maybe I am just repeating myself; if so, I am sorry... but I hope discussing this example leads to a solution that works from every perspective).
[...]
Quoted 10 lines
Quoted 5 lines
So, what about a different idea: a cgroup can be "max" or have (runtime,period); when we set (runtime,period), this is set for all the CPU cores, but then we can override (with runtime 0, or different runtime/period) some cores (but we cannot override with "max").
If the interface is per-CPU, I think it should be per-CPU all the way. What a cgroup has on one CPU shouldn't say anything about what it has on another. That seems the most consistent in itself and with the other controllers.
Well, my "mixing" of per-CPU and per-cgroup interfaces is mainly due to the fact that the tasks in the cgroup can be "global" (allowed to migrate on all the cgroup's cores) or pinned to a CPU core... I try to explain with an example, based on a cgroup with 2 cores, core0 and core1:
- If both core0 and core1 are configured with runtime 10ms and period 100ms, then the real-time tasks running in the cgroup are guaranteed to execute for 20ms every 100ms. Based on this, some theoretical analysis can be used to check if the tasks will respect their deadlines, etc
- If both core0 and core1 are configured as "max", then the real-time tasks contained in the cgroup will always execute with the parent cgroup's deadline entities. Knowing the parameters of such deadline entities, and knowing the tasks running in both cgroups, it is again possible to provide some performance guarantees
- if core0 is configured with runtime 10ms and period 100ms and core1 is configured as "max", then real-time tasks migrating to core1 will execute with the parent's deadline entity... But that deadline entity is also shared with other real-time tasks. So, it is unclear how much CPU time the cgroup's real-time tasks are expected to receive. Also, the tasks migrating from core0 to core1 will consume the parent's runtime on core1 (but not on core0), making it difficult to provide performance guarantees for the parent cgroup too.
In other words, if tasks can migrate between cores, performance guarantees should be "per-cgroup", not per-CPU, which makes it difficult to mix "(runtime, period)" and "max" in the same cgroup. If, on the other hand, tasks were pinned to cores or migrations were disabled, the mix wouldn't be an issue.
Quoted 3 lines
Take Juri's example of a top-level group which has an entry only for CPU0. Its tasks are limited on CPU0 and not on CPU1, which is what the config says.
I understand this if tasks are pinned, but I still fail to reason about performance when tasks can freely migrate between CPU0 and CPU1 (I can only think about a very pessimistic evaluation: assuming that all the tasks execute on CPU0 when we evaluate this cgroup's performance, and assuming that they all execute on CPU1 when evaluate the interference on the parent's workload).
Quoted 4 lines
io.max works the same way. A limit on one device says nothing about the other devices, and a device without an entry is at "max". If the group shouldn't run RT tasks on CPU1, it can be given runtime=0 there.
I do not know much about IO device and io.max, sorry... But maybe the difference is that tasks do not "migrate" between different devices?
Quoted 10 lines
Tasks moving across CPUs doesn't change how the resources are allocated. A task consumes from whatever serves its cgroup on the CPU it's running on.
I'm not too familiar with the schedulability analysis side, so please correct me if I'm missing something. Wouldn't it work if the setups which need the analysis configure the group on all CPUs? That seems like something which can be a convention in how groups are configured rather than forced through the interface, which looks like it'd get clunky whichever shape we pick.
Something like leaving the "all-configured or all-max" as a system administrator responsibility, instead of enforcing it, right? I guess this can work, but I had no time to discuss this with Yuri, so I do not know how difficult this can be from the implementation's perspective.
I also fear this kind of solution can increase the probability of misconfiguring the system (ending up with the feature not being used).
Quoted 4 lines
Quoted 3 lines
We think 0 as a default would be problematic, as it means "no real-time tasks", and 0 in the root means "no real-time tasks in all the cgroups"... Or am I missing your point?
Quoted 8 lines
Yeah, zero doesn't work there. How about defaulting the root to "max" like the other cgroups? "max" on the root would mean that nothing is set aside for cgroups. RT tasks which aren't under any entry run without a server as they do now, and the budgets of top-level cgroups are admitted against the DL bandwidth directly. An entry on the root would be the total that cgroup servers can reserve on that CPU, and writing one fails if what's already reserved doesn't fit. RT tasks in the root itself keep running without a server either way.
Yes, I think this is OK. It means that hierarchical scheduling is disabled by default, and the administrator has to write something in the root cgroup to enable it.
Thanks,
Luca
Tejun Heo
maintainerHello, Luca.
Quoted 3 lines · luca abeni
In other words, if tasks can migrate between cores, performance guarantees should be "per-cgroup", not per-CPU, which makes it difficult to mix "(runtime, period)" and "max" in the same cgroup.
Thanks for the example. I can see that the mixed config is difficult to analyze. What I don't follow is why the kernel has to enforce it. Again, this is purely from the interface POV, but wouldn't it be enough for the kernel to provide the mechanism and leave what gets built on top of it to the users?
Quoted 2 lines
I do not know much about IO device and io.max, sorry... But maybe the difference is that tasks do not "migrate" between different devices?
Yeah, that's different, so the analogy isn't perfect. The allocation side isn't different tho. One can construct an analogous, if contorted, case with block devices. e.g. Imagine multiple disks serving an identical read-only data set, with io.max used for bandwidth and performance isolation. A workload which moves across the disks would be essentially the same model as threads moving across CPUs.
Quoted 2 lines
I also fear this kind of solution can increase the probability of misconfiguring the system (ending up with the feature not being used).
I'm not sure baking the constraint into the interface would make people use it more.
Quoted 3 lines
Yes, I think this is OK. It means that hierarchical scheduling is disabled by default, and the administrator has to write something in the root cgroup to enable it.
That's not what I meant. With the root at "max", nothing has to be written to the root. A cgroup can be given a budget right away and the budget is admitted against the DL bandwidth directly. An entry on the root is only needed to set aside bandwidth for cgroups up front.
Thanks.
--
tejun
Hi Tejun,
On Fri, 20 Nov 2026 01:55:31 -1000
Tejun Heo <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 3 lines
Yes, I think this is OK. It means that hierarchical scheduling is disabled by default, and the administrator has to write something in the root cgroup to enable it.
That's not what I meant. With the root at "max", nothing has to be written to the root. A cgroup can be given a budget right away and the budget is admitted against the DL bandwidth directly. An entry on the root is only needed to set aside bandwidth for cgroups up front.
Uhm... I think this might be problematic.
The SCHED_DEADLINE admission control for tasks is based on a "global scheduling" assumption (tasks are free to migrate on all the CPU cores of the root domain). Hence, the admission control is something like
sum_i(runtime_i/period_i) < limit
But the deadline entities serving the real-time tasks in a cgroup are pinned to cores (one dl entity serves the cgroup's RT rq for a core). So, they cannot just be added to the sum mentioned above (which works for dl entities which are free to migrate).
This issue was solved by reserving a fraction of the CPU time (and the root cgroup parameters indicated this fraction) to RT cgropus, so that the SCHED_DEADLINE admission control is still the same (we just decrease "limit" by the fraction reserved to RT cgroups).
I am not sure about how to adapt this admission test to the strategy you propose above.
Luca
Juri Lelli
maintainerHello,
On 09/11/26 17:35, Yuri Andriaccio wrote:
...
Quoted 22 lines
Version v7 introduces a lot of changes since v6. First off there have been a lot of bug fixes from the comments by the reviewers and the automated AI tool Sashiko, which surprisingly caught a lot of issues. Later, we've finally had time to add features which were planned a long time ago. The following are the most meaningful updates since v6 (the complete changelog can be found on the github readme):
- Rebase to kernel version 7.4-rc1.
- Address (most of) bugs as reported in the v6 reviews.
- Introducedl_freq_invariantflag fordl_servers.
+ Force fair/ext servers to be frequency/capacity invariant, while allow HCBS servers to scale their consumed runtime.
+ This enables capacity-aware bandwidth reservation also for HCBS.
- Rework cgroup-v2 interface:
+ Rename cpu.rt.internal to cpu.rt.effective.local
+ Fix internal runtime rounding errors.
- Update cgroup balance callbacks to allow balancing operation of multiple cgroups at the same time.
- Allow unprivileged FIFO/RR tasks inside cgroups.
- AllowHRTICK_DLfor HCBS servers.
- Integratecpuandcpusetcontrollers.
- Integrate HCBS with hotplug/hotunplug operations.
- Introducedl_serveroverrides (previously known as multi-runtime/multi-CPU).
...
Quoted 8 lines
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Testing v7:
The patchset has been tested with a suite of tests tailored to stress all the implemented functionalities.
The tests are available at https://github.com/Yurand2000/HCBS-Test-Suite . Refer to the README of the repository for more details.
Quoted 8 lines
Follow these steps to test HCBS v7:
- Get the HCBS patch up and running. Any kernel/disto should work effortlessly.
- Get, compile and _install_ the tests.
- Run thego_rt.shscript to set the frequency of the CPUs to a fixed value and disable hyperthreading and power saving features.
- Run therun_tests.sh fullscript, to run the whole test suite.- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
I think Sashiko already commented about it, but I had to apply the following to make the series compile with clang.
BTW, since you already received comments from Tejun and Sashiko also raised lots of stuff to look at, shall I wait for a v8 before engaging as well, so that you don't get overwhelmed? :)
Thanks for your work!
Juri
--- kernel/sched/rt.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/rt.c +4 -2
@@ -3113,11 +3113,13 @@ static int sched_rt_global_validate(void)DEFINE_MUTEX (sched_rt_handler_mutex); -void sched_rt_handler_mutex_lock () { +void sched_rt_handler_mutex_lock (void ) +{mutex_lock (&sched_rt_handler_mutex); } -void sched_rt_handler_mutex_unlock () { +void sched_rt_handler_mutex_unlock (void ) +{mutex_unlock (&sched_rt_handler_mutex); }
Hi Juri,
Thanks for your reply. Sure, Sashiko raised a lot of stuff to look at, so it makes more sense to look at the patch once (hopefully) these issues have been fixed.
We would like to also address the interface issues which Tejun pointed out, so if everyone agrees with out latest interface proposal, we will fix Sashiko's issues and implement the final interface as intended for the next RFC v8.
As a side note, some of the raised issues are strictly related on how the cpuset + cpu integration is implemented, so we'd like to have feedback on the interface before fixing them.
Thanks and have a nice day,
Yuri