Skip to contentSkip to conversation

Hierarchical Constant Bandwidth Server

RFC PATCH Superseded Yuri Andriaccio 4 participants 47 messages Sep 29, 2025 – Nov 3, 2025

Subsystems (2)

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v3 cover Hierarchical Constant Bandwidth Server

Hello,

This is the v3 for Hierarchical Constant Bandwidth Server, aiming at replacing the current RT_GROUP_SCHED mechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ . You can find the previous versions of this patchset at the bottom of the page, in particular version 1 which talks in more detail what this patchset is all about and how it is implemented.

This v3 version further reworks some of the patches as suggested by Juri Lelli. While most of the work is refactorings, the following were also changed:
- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/
- A left-over check which prevented execution of some of wakeup_preempt code has been removed.
- Cgroup pull code was erroneusly comparing cgroup with non-cgroup tasks, now it has been fixed.
- The allocation/deallocation code for rt cgroups has been checked and reworked to make sure that resources are managed correctly in all the code paths.
- Some signatures of cgroup migration related functions where changed to match more closely to their non-group counterparts.
- Descriptions and documentation were added where necessary, in particular for preemption rules in wakeup_preempt.

For this v3 version we've also polished the testing system we are using and made it public for testers to run on their own machines. The source code can be found at https://github.com/Yurand2000/HCBS-Test-Suite , along with a README that explains how to use it. Nonetheless I've reported a description of the tools and instruction later in the page.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Summary of the patches:
1-4) Preparation patches, so that the RT classes' code can be used both
for normal and cgroup scheduling.
5-15) Implementation of HCBS, no migration and only one level hierarchy.
The old RT_GROUP_SCHED code is removed.
16-17) Remove cgroups v1 in favour of v2.
18) Add support for deeper hierarchies.
19-24) Add support for tasks migration.

Updates from v2:
- Rebase to latest tip/master.
- Remove fair-servers' bw reclaiming.
- Fix a check which prevented execution of wakeup_preempt code.
- Fix a priority check in group_pull_rt_task between tasks of different groups.
- Rework allocation/deallocation code for rt-cgroups.
- Update signatures for some group related migration functions.
- Add documentation for wakeup_preempt preemption rules.

Updates from v1:
- Rebase to latest tip/master.
- Add migration code.
- Split big patches for more readability.
- Refactor code to use guarded locks where applicable.
- Remove unnecessary patches from v1 which have been addressed differently by mainline updates.
- Remove unnecessary checks and general code cleanup.

Notes:
Task migration support needs some extra work to reduce its invasiveness, especially patches 21-22.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Testing v3:

The HCBS mechanism has been evaluated on several syntetic tests which are designed to stress the HCBS scheduler and verify that non-interference and mathematical schedulability guarantees are really enforced by the scheduling algorithm.

The test suite currently runs different categories of tests:
- Constraints, which are tasked to assert that hard constraints, such as schedulability conditions, are respected.
- Regression, to check that HCBS does not break anything that already exists.
- Stress, to repeatedly invoke the scheduler in all the exposed interfaces, with the goal to detect bugs and more importantly race conditions.
- Time, simple benchmarks to assert that the dl_servers work correctly, i.e. they allocate the correct amount of bandwidth, and that migration code allows to fully utilize the cgroup's allocated bw.
- Taskset: given a set of (generated) periodic tasks and their bandwidth requirements, schedulability analyses are performed to decide whether or not a given hardware configuration can run the taskset. In particular, for each taskset, a HCBS's cgroup configuration along with the number of necessary CPUs is generated. These are mathematically guaranteed to be schedulable. The next step of this test suite is to configure cgroups as computed and to run the taskset, to verify that the HCBS implementation works as intended and that the scheduling overheads are within reasonable bounds.

The source code can be found at https://github.com/Yurand2000/HCBS-Test-Suite . The README file should explain most if not all questions, but I'm writing briefly the pipeline to run these tests here:

- Get the HCBS patch up and running. Any kernel/disto should work effortlessly.
- Get, compile and _install_ the tests.
- Download the additional taskset files and extract them in the _install_ folder. You can find them here:
https://github.com/Yurand2000/HCBS-Test-Suite/releases/tag/250926
- Run the run_tests.sh full script, to run the whole test suite.

Expect a total runtime of ~3 hours. The script will automatically mount the cgroup and debug filesystems (if not already mounted) and will move all the already running SCHED_FIFO/SCHED_RR tasks in the root cgroup, so that the cgroups' CPU controller can be mounted. It will additionally try to reserve all the possible rt-bandwidth for cgroups (i.e. 90%) to run all the later tests, so make sure that there are no running SCHED_DEADLINE tasks if the script fails to setup.

Some tests specifically need a minimum amount of CPU cores, up to a maximum of eight. If your machine has less CPUs then the tests will simply be skipped.

Notes:

The tasksets minimal requirements were computed using a closed-source software, explaining why the tasksets are supplied separately. A open-source analyser is being written to update this step in the future and also allow for more customization for the testers.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Future Work:

While we wait for more comments, and expect stuff to break, we will work on completing the currently partial/untested, implementation of HCBS with different runtimes per CPU, instead of having the same runtime allocated on all CPUs, to include it in a future RCF.

Future patches:
- HCBS with different runtimes per CPU.
- capacity aware bandwidth reservation.
- enable/disable dl_servers when a CPU goes online/offline.

Have a nice day,
Yuri

v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Yuri Andriaccio (6):
sched/rt: Disable RT_GROUP_SCHED
sched/rt: Add rt-cgroups' dl-servers operations. sched/rt: Update task event callbacks for HCBS scheduling sched/rt: Allow zeroing the runtime of the root control group sched/rt: Remove support for cgroups-v1
sched/core: Execute enqueued balance callbacks when migrating task
betweeen cgroups

luca abeni (18):
sched/deadline: Do not access dl_se->rq directly sched/deadline: Distinct between dl_rq and my_q
sched/rt: Pass an rt_rq instead of an rq where needed sched/rt: Move some functions from rt.c to sched.h sched/rt: Introduce HCBS specific structs in task_group sched/core: Initialize root_task_group
sched/deadline: Add dl_init_tg
sched/rt: Add {alloc/free}_rt_sched_group
sched/deadline: Account rt-cgroups bandwidth in deadline tasks
schedulability tests.
sched/rt: Update rt-cgroup schedulability checks sched/rt: Remove old RT_GROUP_SCHED data structures sched/core: Cgroup v2 support
sched/deadline: Allow deeper hierarchies of RT cgroups sched/rt: Add rt-cgroup migration
sched/rt: Add HCBS migration related checks and function calls sched/deadline: Make rt-cgroup's servers pull tasks on timer
replenishment
sched/deadline: Fix HCBS migrations on server stop sched/core: Execute enqueued balance callbacks when changing allowed
CPUs

 include/linux/sched.h    |   10 +-
 kernel/sched/autogroup.c |    4 +-
 kernel/sched/core.c      |   65 +-
 kernel/sched/deadline.c  |  251 +++-
 kernel/sched/debug.c     |    6 -
 kernel/sched/fair.c      |    6 +-
 kernel/sched/rt.c        | 3069 +++++++++++++++++++-------------------
 kernel/sched/sched.h     |  150 +-
 kernel/sched/syscalls.c  |    6 +-
 9 files changed, 1850 insertions(+), 1717 deletions(-)

base-commit: cec1e6e5d1ab33403b809f79cd20d6aff124ccfe

Signature · 1 line
2.51.0
#1 lore raw

RFC PATCH v3 1/24 sched/deadline: Do not access dl_se->rq directly

From: luca abeni <[email protected]>

Make deadline.c code access the runqueue of a scheduling entity saved in the sched_dl_entity data structure. This allows future patches to save different runqueues in sched_dl_entity other than the global runqueues.

---
 kernel/sched/deadline.c | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)
kernel/sched/deadline.c +14 -13
index f25301267e4..5113c3a2633 100644
@@ -874,7 +874,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)
 	 * and arm the defer timer.
 	 */
 	if (dl_se->dl_defer && !dl_se->dl_defer_running &&
-	    dl_time_before(rq_clock(dl_se->rq), dl_se->deadline - dl_se->runtime)) {
+	    dl_time_before(rq_clock(rq), dl_se->deadline - dl_se->runtime)) {
 		if (!is_dl_boosted(dl_se) && dl_se->server_has_tasks(dl_se)) {
 
 			/*
@@ -1184,11 +1184,11 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
 			 * of time. The dl_server_min_res serves as a limit to avoid
 			 * forwarding the timer for a too small amount of time.
 			 */
-			if (dl_time_before(rq_clock(dl_se->rq),
+			if (dl_time_before(rq_clock(rq),
 					   (dl_se->deadline - dl_se->runtime - dl_server_min_res))) {
 
 				/* reset the defer timer */
-				fw = dl_se->deadline - rq_clock(dl_se->rq) - dl_se->runtime;
+				fw = dl_se->deadline - rq_clock(rq) - dl_se->runtime;
 
 				hrtimer_forward_now(timer, ns_to_ktime(fw));
 				return HRTIMER_RESTART;
@@ -1199,7 +1199,7 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
 
 		enqueue_dl_entity(dl_se, ENQUEUE_REPLENISH);
 
-		if (!dl_task(dl_se->rq->curr) || dl_entity_preempt(dl_se, &dl_se->rq->curr->dl))
+		if (!dl_task(rq->curr) || dl_entity_preempt(dl_se, &rq->curr->dl))
 			resched_curr(rq);
 
 		__push_dl_task(rq, rf);
@@ -1467,7 +1467,7 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64
 
 		hrtimer_try_to_cancel(&dl_se->dl_timer);
 
-		replenish_dl_new_period(dl_se, dl_se->rq);
+		replenish_dl_new_period(dl_se, rq);
 
 		/*
 		 * Not being able to start the timer seems problematic. If it could not
@@ -1581,21 +1581,22 @@ void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec)
 	/* 0 runtime = fair server disabled */
 	if (dl_se->dl_runtime) {
 		dl_se->dl_server_idle = 0;
-		update_curr_dl_se(dl_se->rq, dl_se, delta_exec);
+		update_curr_dl_se(rq_of_dl_se(dl_se), dl_se, delta_exec);
 	}
 }
 
 void dl_server_start(struct sched_dl_entity *dl_se)
 {
-	struct rq *rq = dl_se->rq;
+	struct rq *rq;
 
 	if (!dl_server(dl_se) || dl_se->dl_server_active)
 		return;
 
 	dl_se->dl_server_active = 1;
 	enqueue_dl_entity(dl_se, ENQUEUE_WAKEUP);
-	if (!dl_task(dl_se->rq->curr) || dl_entity_preempt(dl_se, &rq->curr->dl))
-		resched_curr(dl_se->rq);
+	rq = rq_of_dl_se(dl_se);
+	if (!dl_task(rq->curr) || dl_entity_preempt(dl_se, &rq->curr->dl))
+		resched_curr(rq);
 }
 
 void dl_server_stop(struct sched_dl_entity *dl_se)
@@ -1651,9 +1652,9 @@ void sched_init_dl_servers(void)
 
 		WARN_ON(dl_server(dl_se));
 
-		dl_server_apply_params(dl_se, runtime, period, 1);
-
 		dl_se->dl_server = 1;
+		WARN_ON(dl_server_apply_params(dl_se, runtime, period, 1));
+
 		dl_se->dl_defer = 1;
 		setup_new_dl_entity(dl_se);
 	}
@@ -1678,7 +1679,7 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio
 {
 	u64 old_bw = init ? 0 : to_ratio(dl_se->dl_period, dl_se->dl_runtime);
 	u64 new_bw = to_ratio(period, runtime);
-	struct rq *rq = dl_se->rq;
+	struct rq *rq = rq_of_dl_se(dl_se);
 	int cpu = cpu_of(rq);
 	struct dl_bw *dl_b;
 	unsigned long cap;
@@ -1755,7 +1756,7 @@ static enum hrtimer_restart inactive_task_timer(struct hrtimer *timer)
 		p = dl_task_of(dl_se);
 		rq = task_rq_lock(p, &rf);
 	} else {
-		rq = dl_se->rq;
+		rq = rq_of_dl_se(dl_se);
 		rq_lock(rq, &rf);
 	}
 
Signature · 1 line
2.51.0
maintainer
#2 lore raw

Re: [RFC PATCH v3 01/24] sched/deadline: Do not access dl_se->rq directly

Hello,

Quoted 10 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Make deadline.c code access the runqueue of a scheduling entity saved in the sched_dl_entity data structure. This allows future patches to save different runqueues in sched_dl_entity other than the global runqueues.

---
 kernel/sched/deadline.c | 27 ++++++++++++++-------------

Nit, I believe there are a couple of dl_se->rq direct usages in fair.c left after this change. The are outside deadline and they are indeed modified anyway by next patch. Just for your consideration in case you want to do something about them from completeness.

...

kernel/sched/deadline.c void sched_init_dl_servers(void) · line 1657

Quoted 9 lines
kernel/sched/deadline.c +2 -2
@@ -1651,9 +1652,9 @@ void sched_init_dl_servers(void)
 
 		WARN_ON(dl_server(dl_se));
 
-		dl_server_apply_params(dl_se, runtime, period, 1);
-
 		dl_se->dl_server = 1;
+		WARN_ON(dl_server_apply_params(dl_se, runtime, period, 1));
+

The move after dl_server is initialized is indeed needed, but maybe we should mention the small functional change (also the additon of WARN) in the changelog.

Best,
Juri

#3 lore raw

RFC PATCH v3 2/24 sched/deadline: Distinct between dl_rq and my_q

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Create two fields for runqueues in sched_dl_entity to make a distinction between the global runqueue and the runqueue which the dl_server serves.

---
 include/linux/sched.h   |  6 ++++--
 kernel/sched/deadline.c | 11 +++++++----
 kernel/sched/fair.c     |  6 +++---
 kernel/sched/sched.h    |  3 ++-
 4 files changed, 16 insertions(+), 10 deletions(-)
include/linux/sched.h +4 -2
index f8188b83335..5022ff99c5d 100644
@@ -727,12 +727,14 @@ struct sched_dl_entity {
 	 * Bits for DL-server functionality. Also see the comment near
 	 * dl_server_update().
 	 *
-	 * @rq the runqueue this server is for
+	 * @dl_rq the runqueue on which this entity is (to be) queued
+	 * @my_q  the runqueue "owned" by this entity
 	 *
 	 * @server_has_tasks() returns true if @server_pick return a
 	 * runnable task.
 	 */
-	struct rq			*rq;
+	struct dl_rq			*dl_rq;
+	struct rq			*my_q;
 	dl_server_has_tasks_f		server_has_tasks;
 	dl_server_pick_f		server_pick_task;
 
kernel/sched/deadline.c +7 -4
index 5113c3a2633..6ff00f71041 100644
@@ -75,11 +75,12 @@ static inline struct rq *rq_of_dl_rq(struct dl_rq *dl_rq)
 
 static inline struct rq *rq_of_dl_se(struct sched_dl_entity *dl_se)
 {
-	struct rq *rq = dl_se->rq;
+	struct rq *rq;
 
 	if (!dl_server(dl_se))
 		rq = task_rq(dl_task_of(dl_se));
-
+	else
+		rq = container_of(dl_se->dl_rq, struct rq, dl);
 	return rq;
 }
 
@@ -1625,11 +1626,13 @@ static bool dl_server_stopped(struct sched_dl_entity *dl_se)
 	return false;
 }
 
-void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
+void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,
+		    struct rq *served_rq,
 		    dl_server_has_tasks_f has_tasks,
 		    dl_server_pick_f pick_task)
 {
-	dl_se->rq = rq;
+	dl_se->dl_rq = dl_rq;
+	dl_se->my_q  = served_rq;
 	dl_se->server_has_tasks = has_tasks;
 	dl_se->server_pick_task = pick_task;
 }
kernel/sched/fair.c +3 -3
index b173a059315..2723086538b 100644
@@ -8861,12 +8861,12 @@ static struct task_struct *__pick_next_task_fair(struct rq *rq, struct task_stru
 
 static bool fair_server_has_tasks(struct sched_dl_entity *dl_se)
 {
-	return !!dl_se->rq->cfs.nr_queued;
+	return !!dl_se->my_q->cfs.nr_queued;
 }
 
 static struct task_struct *fair_server_pick_task(struct sched_dl_entity *dl_se)
 {
-	return pick_task_fair(dl_se->rq);
+	return pick_task_fair(dl_se->my_q);
 }
 
 void fair_server_init(struct rq *rq)
@@ -8875,7 +8875,7 @@ void fair_server_init(struct rq *rq)
 
 	init_dl_entity(dl_se);
 
-	dl_server_init(dl_se, rq, fair_server_has_tasks, fair_server_pick_task);
+	dl_server_init(dl_se, &rq->dl, rq, fair_server_has_tasks, fair_server_pick_task);
 }
 
 /*
kernel/sched/sched.h +2 -1
index be9745d104f..2499b36a3c5 100644
@@ -382,7 +382,8 @@ extern s64 dl_scaled_delta_exec(struct rq *rq, struct sched_dl_entity *dl_se, s6
 extern void dl_server_update(struct sched_dl_entity *dl_se, s64 delta_exec);
 extern void dl_server_start(struct sched_dl_entity *dl_se);
 extern void dl_server_stop(struct sched_dl_entity *dl_se);
-extern void dl_server_init(struct sched_dl_entity *dl_se, struct rq *rq,
+extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,
+		    struct rq *served_rq,
 		    dl_server_has_tasks_f has_tasks,
 		    dl_server_pick_f pick_task);
 extern void sched_init_dl_servers(void);
Signature · 1 line
2.51.0
maintainer
#4 lore raw

Re: [RFC PATCH v3 02/24] sched/deadline: Distinct between dl_rq and my_q

Hello,

Quoted 7 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Create two fields for runqueues in sched_dl_entity to make a distinction between the global runqueue and the runqueue which the dl_server serves.

The change looks good to me. Maybe we only want to be a little more verbose in the changelog, e.g. the following?

Split the single rq pointer in sched_dl_entity into two separate pointers, following the existing pattern used by sched_rt_entity:

- dl_rq: Points to the deadline runqueue where this entity is queued
- my_q: Points to the runqueue that this entity serves (for servers)

This distinction is currently redundant for the fair_server (both point to the same CPU's structures), but is essential for future RT cgroup support where deadline servers will be queued on the global dl_rq while serving tasks from cgroup-specific runqueues.

Update rq_of_dl_se() to use container_of() to recover the rq from dl_rq, and update fair.c to explicitly use my_q when accessing the served runqueue.

Thanks,
Juri

#5 lore raw

RFC PATCH v3 3/24 sched/rt: Pass an rt_rq instead of an rq where needed

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Make rt.c code access the runqueue through the rt_rq data structure rather than passing an rq pointer directly. This allows future patches to define rt_rq data structures which do not refer only to the global runqueue, but also to local cgroup runqueues (rt_rq is not always equal to &rq->rt).

---
 kernel/sched/rt.c | 99 ++++++++++++++++++++++++++---------------------
 1 file changed, 54 insertions(+), 45 deletions(-)
kernel/sched/rt.c +54 -45
index 7936d433373..a7d063d2303 100644
@@ -370,9 +370,9 @@ static inline void rt_clear_overload(struct rq *rq)
 	cpumask_clear_cpu(rq->cpu, rq->rd->rto_mask);
 }
 
-static inline int has_pushable_tasks(struct rq *rq)
+static inline int has_pushable_tasks(struct rt_rq *rt_rq)
 {
-	return !plist_head_empty(&rq->rt.pushable_tasks);
+	return !plist_head_empty(&rt_rq->pushable_tasks);
 }
 
 static DEFINE_PER_CPU(struct balance_callback, rt_push_head);
@@ -381,50 +381,54 @@ static DEFINE_PER_CPU(struct balance_callback, rt_pull_head);
 static void push_rt_tasks(struct rq *);
 static void pull_rt_task(struct rq *);
 
-static inline void rt_queue_push_tasks(struct rq *rq)
+static inline void rt_queue_push_tasks(struct rt_rq * rt_rq)
 {
-	if (!has_pushable_tasks(rq))
+	struct rq *rq = container_of(rt_rq, struct rq, rt);
+
+	if (!has_pushable_tasks(rt_rq))
 		return;
 
 	queue_balance_callback(rq, &per_cpu(rt_push_head, rq->cpu), push_rt_tasks);
 }
 
-static inline void rt_queue_pull_task(struct rq *rq)
+static inline void rt_queue_pull_task(struct rt_rq * rt_rq)
 {
+	struct rq *rq = container_of(rt_rq, struct rq, rt);
+
 	queue_balance_callback(rq, &per_cpu(rt_pull_head, rq->cpu), pull_rt_task);
 }
 
-static void enqueue_pushable_task(struct rq *rq, struct task_struct *p)
+static void enqueue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
 {
-	plist_del(&p->pushable_tasks, &rq->rt.pushable_tasks);
+	plist_del(&p->pushable_tasks, &rt_rq->pushable_tasks);
 	plist_node_init(&p->pushable_tasks, p->prio);
-	plist_add(&p->pushable_tasks, &rq->rt.pushable_tasks);
+	plist_add(&p->pushable_tasks, &rt_rq->pushable_tasks);
 
 	/* Update the highest prio pushable task */
-	if (p->prio < rq->rt.highest_prio.next)
-		rq->rt.highest_prio.next = p->prio;
+	if (p->prio < rt_rq->highest_prio.next)
+		rt_rq->highest_prio.next = p->prio;
 
-	if (!rq->rt.overloaded) {
-		rt_set_overload(rq);
-		rq->rt.overloaded = 1;
+	if (!rt_rq->overloaded) {
+		rt_set_overload(rq_of_rt_rq(rt_rq));
+		rt_rq->overloaded = 1;
 	}
 }
 
-static void dequeue_pushable_task(struct rq *rq, struct task_struct *p)
+static void dequeue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
 {
-	plist_del(&p->pushable_tasks, &rq->rt.pushable_tasks);
+	plist_del(&p->pushable_tasks, &rt_rq->pushable_tasks);
 
 	/* Update the new highest prio pushable task */
-	if (has_pushable_tasks(rq)) {
-		p = plist_first_entry(&rq->rt.pushable_tasks,
+	if (has_pushable_tasks(rt_rq)) {
+		p = plist_first_entry(&rt_rq->pushable_tasks,
 				      struct task_struct, pushable_tasks);
-		rq->rt.highest_prio.next = p->prio;
+		rt_rq->highest_prio.next = p->prio;
 	} else {
-		rq->rt.highest_prio.next = MAX_RT_PRIO-1;
+		rt_rq->highest_prio.next = MAX_RT_PRIO-1;
 
-		if (rq->rt.overloaded) {
-			rt_clear_overload(rq);
-			rq->rt.overloaded = 0;
+		if (rt_rq->overloaded) {
+			rt_clear_overload(rq_of_rt_rq(rt_rq));
+			rt_rq->overloaded = 0;
 		}
 	}
 }
@@ -1431,6 +1435,7 @@ static void
 enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)
 {
 	struct sched_rt_entity *rt_se = &p->rt;
+	struct rt_rq *rt_rq = rt_rq_of_se(rt_se);
 
 	if (flags & ENQUEUE_WAKEUP)
 		rt_se->timeout = 0;
@@ -1444,17 +1449,18 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)
 		return;
 
 	if (!task_current(rq, p) && p->nr_cpus_allowed > 1)
-		enqueue_pushable_task(rq, p);
+		enqueue_pushable_task(rt_rq, p);
 }
 
 static bool dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags)
 {
 	struct sched_rt_entity *rt_se = &p->rt;
+	struct rt_rq *rt_rq = rt_rq_of_se(rt_se);
 
 	update_curr_rt(rq);
 	dequeue_rt_entity(rt_se, flags);
 
-	dequeue_pushable_task(rq, p);
+	dequeue_pushable_task(rt_rq, p);
 
 	return true;
 }
@@ -1639,14 +1645,14 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)
 static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool first)
 {
 	struct sched_rt_entity *rt_se = &p->rt;
-	struct rt_rq *rt_rq = &rq->rt;
+	struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
 
 	p->se.exec_start = rq_clock_task(rq);
 	if (on_rt_rq(&p->rt))
 		update_stats_wait_end_rt(rt_rq, rt_se);
 
 	/* The running task is never eligible for pushing */
-	dequeue_pushable_task(rq, p);
+	dequeue_pushable_task(rt_rq, p);
 
 	if (!first)
 		return;
@@ -1659,7 +1665,7 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool f
 	if (rq->donor->sched_class != &rt_sched_class)
 		update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 0);
 
-	rt_queue_push_tasks(rq);
+	rt_queue_push_tasks(rt_rq);
 }
 
 static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq)
@@ -1710,7 +1716,7 @@ static struct task_struct *pick_task_rt(struct rq *rq)
 static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_struct *next)
 {
 	struct sched_rt_entity *rt_se = &p->rt;
-	struct rt_rq *rt_rq = &rq->rt;
+	struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
 
 	if (on_rt_rq(&p->rt))
 		update_stats_wait_start_rt(rt_rq, rt_se);
@@ -1726,7 +1732,7 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s
 	 * if it is still active
 	 */
 	if (on_rt_rq(&p->rt) && p->nr_cpus_allowed > 1)
-		enqueue_pushable_task(rq, p);
+		enqueue_pushable_task(rt_rq, p);
 }
 
 /* Only try algorithms three times */
@@ -1736,16 +1742,16 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s
  * Return the highest pushable rq's task, which is suitable to be executed
  * on the CPU, NULL otherwise
  */
-static struct task_struct *pick_highest_pushable_task(struct rq *rq, int cpu)
+static struct task_struct *pick_highest_pushable_task(struct rt_rq *rt_rq, int cpu)
 {
-	struct plist_head *head = &rq->rt.pushable_tasks;
+	struct plist_head *head = &rt_rq->pushable_tasks;
 	struct task_struct *p;
 
-	if (!has_pushable_tasks(rq))
+	if (!has_pushable_tasks(rt_rq))
 		return NULL;
 
 	plist_for_each_entry(p, head, pushable_tasks) {
-		if (task_is_pushable(rq, p, cpu))
+		if (task_is_pushable(rq_of_rt_rq(rt_rq), p, cpu))
 			return p;
 	}
 
@@ -1845,14 +1851,15 @@ static int find_lowest_rq(struct task_struct *task)
 	return -1;
 }
 
-static struct task_struct *pick_next_pushable_task(struct rq *rq)
+static struct task_struct *pick_next_pushable_task(struct rt_rq *rt_rq)
 {
+	struct rq *rq = rq_of_rt_rq(rt_rq);
 	struct task_struct *p;
 
-	if (!has_pushable_tasks(rq))
+	if (!has_pushable_tasks(rt_rq))
 		return NULL;
 
-	p = plist_first_entry(&rq->rt.pushable_tasks,
+	p = plist_first_entry(&rt_rq->pushable_tasks,
 			      struct task_struct, pushable_tasks);
 
 	BUG_ON(rq->cpu != task_cpu(p));
@@ -1905,7 +1912,7 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
 			 */
 			if (unlikely(is_migration_disabled(task) ||
 				     !cpumask_test_cpu(lowest_rq->cpu, &task->cpus_mask) ||
-				     task != pick_next_pushable_task(rq))) {
+				     task != pick_next_pushable_task(&rq->rt))) {
 
 				double_unlock_balance(rq, lowest_rq);
 				lowest_rq = NULL;
@@ -1939,7 +1946,7 @@ static int push_rt_task(struct rq *rq, bool pull)
 	if (!rq->rt.overloaded)
 		return 0;
 
-	next_task = pick_next_pushable_task(rq);
+	next_task = pick_next_pushable_task(&rq->rt);
 	if (!next_task)
 		return 0;
 
@@ -2014,7 +2021,7 @@ static int push_rt_task(struct rq *rq, bool pull)
 		 * run-queue and is also still the next task eligible for
 		 * pushing.
 		 */
-		task = pick_next_pushable_task(rq);
+		task = pick_next_pushable_task(&rq->rt);
 		if (task == next_task) {
 			/*
 			 * The task hasn't migrated, and is still the next
@@ -2202,7 +2209,7 @@ void rto_push_irq_work_func(struct irq_work *work)
 	 * We do not need to grab the lock to check for has_pushable_tasks.
 	 * When it gets updated, a check is made if a push is possible.
 	 */
-	if (has_pushable_tasks(rq)) {
+	if (has_pushable_tasks(&rq->rt)) {
 		raw_spin_rq_lock(rq);
 		while (push_rt_task(rq, true))
 			;
@@ -2231,6 +2238,7 @@ static void pull_rt_task(struct rq *this_rq)
 	int this_cpu = this_rq->cpu, cpu;
 	bool resched = false;
 	struct task_struct *p, *push_task;
+	struct rt_rq *src_rt_rq;
 	struct rq *src_rq;
 	int rt_overload_count = rt_overloaded(this_rq);
 
@@ -2260,6 +2268,7 @@ static void pull_rt_task(struct rq *this_rq)
 			continue;
 
 		src_rq = cpu_rq(cpu);
+		src_rt_rq = &src_rq->rt;
 
 		/*
 		 * Don't bother taking the src_rq->lock if the next highest
@@ -2268,7 +2277,7 @@ static void pull_rt_task(struct rq *this_rq)
 		 * logically higher, the src_rq will push this task away.
 		 * And if its going logically lower, we do not care
 		 */
-		if (src_rq->rt.highest_prio.next >=
+		if (src_rt_rq->highest_prio.next >=
 		    this_rq->rt.highest_prio.curr)
 			continue;
 
@@ -2284,7 +2293,7 @@ static void pull_rt_task(struct rq *this_rq)
 		 * We can pull only a task, which is pushable
 		 * on its rq, and no others.
 		 */
-		p = pick_highest_pushable_task(src_rq, this_cpu);
+		p = pick_highest_pushable_task(src_rt_rq, this_cpu);
 
 		/*
 		 * Do we have an RT task that preempts
@@ -2390,7 +2399,7 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)
 	if (!task_on_rq_queued(p) || rq->rt.rt_nr_running)
 		return;
 
-	rt_queue_pull_task(rq);
+	rt_queue_pull_task(rt_rq_of_se(&p->rt));
 }
 
 void __init init_sched_rt_class(void)
@@ -2426,7 +2435,7 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)
 	 */
 	if (task_on_rq_queued(p)) {
 		if (p->nr_cpus_allowed > 1 && rq->rt.overloaded)
-			rt_queue_push_tasks(rq);
+			rt_queue_push_tasks(rt_rq_of_se(&p->rt));
 		if (p->prio < rq->donor->prio && cpu_online(cpu_of(rq)))
 			resched_curr(rq);
 	}
@@ -2448,7 +2457,7 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)
 		 * may need to pull tasks to this runqueue.
 		 */
 		if (oldprio < p->prio)
-			rt_queue_pull_task(rq);
+			rt_queue_pull_task(rt_rq_of_se(&p->rt));
 
 		/*
 		 * If there's a higher priority task waiting to run
Signature · 1 line
2.51.0
maintainer
#6 lore raw

Re: [RFC PATCH v3 03/24] sched/rt: Pass an rt_rq instead of an rq where needed

Hello,

Quoted 30 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Make rt.c code access the runqueue through the rt_rq data structure rather than passing an rq pointer directly. This allows future patches to define rt_rq data structures which do not refer only to the global runqueue, but also to local cgroup runqueues (rt_rq is not always equal to &rq->rt).

---
 kernel/sched/rt.c | 99 ++++++++++++++++++++++++++---------------------
 1 file changed, 54 insertions(+), 45 deletions(-)
kernel/sched/rt.c +2 -2
index 7936d433373..a7d063d2303 100644
@@ -370,9 +370,9 @@ static inline void rt_clear_overload(struct rq *rq)
 	cpumask_clear_cpu(rq->cpu, rq->rd->rto_mask);
 }
 
-static inline int has_pushable_tasks(struct rq *rq)
+static inline int has_pushable_tasks(struct rt_rq *rt_rq)
 {
-	return !plist_head_empty(&rq->rt.pushable_tasks);
+	return !plist_head_empty(&rt_rq->pushable_tasks);
 }
 
 static DEFINE_PER_CPU(struct balance_callback, rt_push_head);

kernel/sched/rt.c static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); · line 384

Quoted 5 lines
kernel/sched/rt.c +1 -1
@@ -381,50 +381,54 @@ static DEFINE_PER_CPU(struct balance_callback, rt_pull_head);
 static void push_rt_tasks(struct rq *);
 static void pull_rt_task(struct rq *);
 
-static inline void rt_queue_push_tasks(struct rq *rq)
+static inline void rt_queue_push_tasks(struct rt_rq * rt_rq)

Nit, remove space ^^^

kernel/sched/rt.c static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); · line 394

Quoted 12 lines
kernel/sched/rt.c +4 -2
 {
-	if (!has_pushable_tasks(rq))
+	struct rq *rq = container_of(rt_rq, struct rq, rt);
+
+	if (!has_pushable_tasks(rt_rq))
 		return;
 
 	queue_balance_callback(rq, &per_cpu(rt_push_head, rq->cpu), push_rt_tasks);
 }
 
-static inline void rt_queue_pull_task(struct rq *rq)
+static inline void rt_queue_pull_task(struct rt_rq * rt_rq)

Nit, remove space ^^^

kernel/sched/rt.c static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); · line 396

Quoted 2 lines
kernel/sched/rt.c +1 -0
 {
+	struct rq *rq = container_of(rt_rq, struct rq, rt);

Still need to check with the full series (and it's OK up until here), but I guess this needs special attention with tasks inside groups to make sure we don't queue callbacks in rqs representing groups?

kernel/sched/rt.c static DEFINE_PER_CPU(struct balance_callback, rt_pull_head); · line 399

Quoted 3 lines
kernel/sched/rt.c +1 -0
+
 	queue_balance_callback(rq, &per_cpu(rt_pull_head, rq->cpu), pull_rt_task);
 }

Thanks,
Juri

#7 lore raw

RFC PATCH v3 4/24 sched/rt: Move some functions from rt.c to sched.h

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Make the following functions be non-static and move them in sched.h, so that they can be used also in other source files:
- rt_task_of()
- rq_of_rt_rq()
- rt_rq_of_se()
- rq_of_rt_se()

There are no functional changes. This is needed by future patches.

---
 kernel/sched/rt.c    | 52 ------------------------------------------
 kernel/sched/sched.h | 54 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 52 deletions(-)
kernel/sched/rt.c +0 -52
index a7d063d2303..a599f63bf7f 100644
@@ -168,34 +168,6 @@ static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)
 
 #define rt_entity_is_task(rt_se) (!(rt_se)->my_q)
 
-static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
-{
-	WARN_ON_ONCE(!rt_entity_is_task(rt_se));
-
-	return container_of(rt_se, struct task_struct, rt);
-}
-
-static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq)
-{
-	/* Cannot fold with non-CONFIG_RT_GROUP_SCHED version, layout */
-	WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
-	return rt_rq->rq;
-}
-
-static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
-{
-	WARN_ON(!rt_group_sched_enabled() && rt_se->rt_rq->tg != &root_task_group);
-	return rt_se->rt_rq;
-}
-
-static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
-{
-	struct rt_rq *rt_rq = rt_se->rt_rq;
-
-	WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
-	return rt_rq->rq;
-}
-
 void unregister_rt_sched_group(struct task_group *tg)
 {
 	if (!rt_group_sched_enabled())
@@ -296,30 +268,6 @@ int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
 
 #define rt_entity_is_task(rt_se) (1)
 
-static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
-{
-	return container_of(rt_se, struct task_struct, rt);
-}
-
-static inline struct rq *rq_of_rt_rq(struct rt_rq *rt_rq)
-{
-	return container_of(rt_rq, struct rq, rt);
-}
-
-static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
-{
-	struct task_struct *p = rt_task_of(rt_se);
-
-	return task_rq(p);
-}
-
-static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
-{
-	struct rq *rq = rq_of_rt_se(rt_se);
-
-	return &rq->rt;
-}
-
 void unregister_rt_sched_group(struct task_group *tg) { }
 
 void free_rt_sched_group(struct task_group *tg) { }
kernel/sched/sched.h +54 -0

60 lines not on this page · raw message

Signature · 1 line
2.51.0
maintainer
#8 lore raw

Re: [RFC PATCH v3 04/24] sched/rt: Move some functions from rt.c to sched.h

Hello,

Quoted 23 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Make the following functions be non-static and move them in sched.h, so that they can be used also in other source files:
- rt_task_of()
- rq_of_rt_rq()
- rt_rq_of_se()
- rq_of_rt_se()

There are no functional changes. This is needed by future patches.

---
 kernel/sched/rt.c    | 52 ------------------------------------------
 kernel/sched/sched.h | 54 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 52 deletions(-)
kernel/sched/rt.c +0 -0
index a7d063d2303..a599f63bf7f 100644

kernel/sched/rt.c static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b) · line 169

Quoted 2 lines
kernel/sched/rt.c +0 -0
@@ -168,34 +168,6 @@ static void destroy_rt_bandwidth(struct rt_bandwidth *rt_b)
 
 #define rt_entity_is_task(rt_se) (!(rt_se)->my_q)

Do we want to move this macro as well, so that rt_task_of() doesn't need to change?

Quoted 6 lines
kernel/sched/rt.c +0 -6
-static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
-{
-	WARN_ON_ONCE(!rt_entity_is_task(rt_se));
-
-	return container_of(rt_se, struct task_struct, rt);
-}

...

kernel/sched/rt.c

Quoted 7 lines
kernel/sched/rt.c +7 -0
+#ifdef CONFIG_RT_GROUP_SCHED
+static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
+{
+	WARN_ON_ONCE(rt_se->my_q);
+
+	return container_of(rt_se, struct task_struct, rt);
+}

Thanks,
Juri

#9 lore raw

RFC PATCH v3 5/24 sched/rt: Disable RT_GROUP_SCHED

In reply to Yuri Andriaccio

Disable the old RT_GROUP_SCHED scheduler. Note that this does not completely remove all the RT_GROUP_SCHED functionality, just unhooks it and removes most of the relevant functions. Some of the RT_GROUP_SCHED functions are kept because they will be adapted for the HCBS scheduling.

---
 kernel/sched/core.c     |   6 -
 kernel/sched/deadline.c |  34 --
 kernel/sched/debug.c    |   6 -
 kernel/sched/rt.c       | 848 ++--------------------------------------
 kernel/sched/sched.h    |  11 +-
 kernel/sched/syscalls.c |  13 -
 6 files changed, 26 insertions(+), 892 deletions(-)
kernel/sched/core.c +0 -6
index ccba6fc3c3f..5791aa1f8c8 100644
@@ -8721,11 +8721,6 @@ void __init sched_init(void)
 
 	init_defrootdomain();
 
-#ifdef CONFIG_RT_GROUP_SCHED
-	init_rt_bandwidth(&root_task_group.rt_bandwidth,
-			global_rt_period(), global_rt_runtime());
-#endif /* CONFIG_RT_GROUP_SCHED */
-
 #ifdef CONFIG_CGROUP_SCHED
 	task_group_cache = KMEM_CACHE(task_group, 0);
 
@@ -8777,7 +8772,6 @@ void __init sched_init(void)
 		 * starts working after scheduler_running, which is not the case
 		 * yet.
 		 */
-		rq->rt.rt_runtime = global_rt_runtime();
 		init_tg_rt_entry(&root_task_group, &rq->rt, NULL, i, NULL);
 #endif
 		rq->sd = NULL;
kernel/sched/deadline.c +0 -34
index 6ff00f71041..277fbaff8b5 100644
@@ -1508,40 +1508,6 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64
 		if (!is_leftmost(dl_se, &rq->dl))
 			resched_curr(rq);
 	}
-
-	/*
-	 * The fair server (sole dl_server) does not account for real-time
-	 * workload because it is running fair work.
-	 */
-	if (dl_se == &rq->fair_server)
-		return;
-
-#ifdef CONFIG_RT_GROUP_SCHED
-	/*
-	 * Because -- for now -- we share the rt bandwidth, we need to
-	 * account our runtime there too, otherwise actual rt tasks
-	 * would be able to exceed the shared quota.
-	 *
-	 * Account to the root rt group for now.
-	 *
-	 * The solution we're working towards is having the RT groups scheduled
-	 * using deadline servers -- however there's a few nasties to figure
-	 * out before that can happen.
-	 */
-	if (rt_bandwidth_enabled()) {
-		struct rt_rq *rt_rq = &rq->rt;
-
-		raw_spin_lock(&rt_rq->rt_runtime_lock);
-		/*
-		 * We'll let actual RT tasks worry about the overflow here, we
-		 * have our own CBS to keep us inline; only account when RT
-		 * bandwidth is relevant.
-		 */
-		if (sched_rt_bandwidth_account(rt_rq))
-			rt_rq->rt_time += delta_exec;
-		raw_spin_unlock(&rt_rq->rt_runtime_lock);
-	}
-#endif /* CONFIG_RT_GROUP_SCHED */
 }
 
 /*
kernel/sched/debug.c +0 -6
index 02e16b70a79..efcf8d82f85 100644
@@ -890,12 +890,6 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)
 
 	PU(rt_nr_running);
 
-#ifdef CONFIG_RT_GROUP_SCHED
-	P(rt_throttled);
-	PN(rt_time);
-	PN(rt_runtime);
-#endif
-
 #undef PN
 #undef PU
 #undef P
kernel/sched/rt.c +23 -825

1,073 lines not on this page · raw message

kernel/sched/sched.h +3 -8
index 831ab576842..2d373a1ba67 100644
@@ -787,7 +787,7 @@ struct scx_rq {
 
 static inline int rt_bandwidth_enabled(void)
 {
-	return sysctl_sched_rt_runtime >= 0;
+	return 0;
 }
 
 /* RT IPI pull logic requires IRQ_WORK */
@@ -827,7 +827,7 @@ struct rt_rq {
 
 static inline bool rt_rq_is_runnable(struct rt_rq *rt_rq)
 {
-	return rt_rq->rt_queued && rt_rq->rt_nr_running;
+	return rt_rq->rt_nr_running;
 }
 
 /* Deadline class' related fields in a runqueue */
@@ -2543,7 +2543,7 @@ static inline bool sched_dl_runnable(struct rq *rq)
 
 static inline bool sched_rt_runnable(struct rq *rq)
 {
-	return rq->rt.rt_queued > 0;
+	return rq->rt.rt_nr_running > 0;
 }
 
 static inline bool sched_fair_runnable(struct rq *rq)
@@ -2654,9 +2654,6 @@ extern void resched_curr(struct rq *rq);
 extern void resched_curr_lazy(struct rq *rq);
 extern void resched_cpu(int cpu);
 
-extern void init_rt_bandwidth(struct rt_bandwidth *rt_b, u64 period, u64 runtime);
-extern bool sched_rt_bandwidth_account(struct rt_rq *rt_rq);
-
 extern void init_dl_entity(struct sched_dl_entity *dl_se);
 
 #define BW_SHIFT		20
@@ -3043,8 +3040,6 @@ extern bool sched_smp_initialized;
 #ifdef CONFIG_RT_GROUP_SCHED
 static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
 {
-	WARN_ON_ONCE(rt_se->my_q);
-
 	return container_of(rt_se, struct task_struct, rt);
 }
 
kernel/sched/syscalls.c +0 -13
index 77ae87f36e8..93a9c03b28e 100644
@@ -626,19 +626,6 @@ int __sched_setscheduler(struct task_struct *p,
 change:
 
 	if (user) {
-#ifdef CONFIG_RT_GROUP_SCHED
-		/*
-		 * Do not allow real-time tasks into groups that have no runtime
-		 * assigned.
-		 */
-		if (rt_group_sched_enabled() &&
-				rt_bandwidth_enabled() && rt_policy(policy) &&
-				task_group(p)->rt_bandwidth.rt_runtime == 0 &&
-				!task_group_is_autogroup(task_group(p))) {
-			retval = -EPERM;
-			goto unlock;
-		}
-#endif /* CONFIG_RT_GROUP_SCHED */
 		if (dl_bandwidth_enabled() && dl_policy(policy) &&
 				!(attr->sched_flags & SCHED_FLAG_SUGOV)) {
 			cpumask_t *span = rq->rd->span;
Signature · 1 line
2.51.0
maintainer
#10 lore raw

Re: [RFC PATCH v3 05/24] sched/rt: Disable RT_GROUP_SCHED

Hello,

Quoted 106 lines · Yuri Andriaccio

Disable the old RT_GROUP_SCHED scheduler. Note that this does not completely remove all the RT_GROUP_SCHED functionality, just unhooks it and removes most of the relevant functions. Some of the RT_GROUP_SCHED functions are kept because they will be adapted for the HCBS scheduling.

---
 kernel/sched/core.c     |   6 -
 kernel/sched/deadline.c |  34 --
 kernel/sched/debug.c    |   6 -
 kernel/sched/rt.c       | 848 ++--------------------------------------
 kernel/sched/sched.h    |  11 +-
 kernel/sched/syscalls.c |  13 -
 6 files changed, 26 insertions(+), 892 deletions(-)
kernel/sched/core.c +0 -6
index ccba6fc3c3f..5791aa1f8c8 100644
@@ -8721,11 +8721,6 @@ void __init sched_init(void)
 
 	init_defrootdomain();
 
-#ifdef CONFIG_RT_GROUP_SCHED
-	init_rt_bandwidth(&root_task_group.rt_bandwidth,
-			global_rt_period(), global_rt_runtime());
-#endif /* CONFIG_RT_GROUP_SCHED */
-
 #ifdef CONFIG_CGROUP_SCHED
 	task_group_cache = KMEM_CACHE(task_group, 0);
 
@@ -8777,7 +8772,6 @@ void __init sched_init(void)
 		 * starts working after scheduler_running, which is not the case
 		 * yet.
 		 */
-		rq->rt.rt_runtime = global_rt_runtime();
 		init_tg_rt_entry(&root_task_group, &rq->rt, NULL, i, NULL);
 #endif
 		rq->sd = NULL;
kernel/sched/deadline.c +0 -34
index 6ff00f71041..277fbaff8b5 100644
@@ -1508,40 +1508,6 @@ static void update_curr_dl_se(struct rq *rq, struct sched_dl_entity *dl_se, s64
 		if (!is_leftmost(dl_se, &rq->dl))
 			resched_curr(rq);
 	}
-
-	/*
-	 * The fair server (sole dl_server) does not account for real-time
-	 * workload because it is running fair work.
-	 */
-	if (dl_se == &rq->fair_server)
-		return;
-
-#ifdef CONFIG_RT_GROUP_SCHED
-	/*
-	 * Because -- for now -- we share the rt bandwidth, we need to
-	 * account our runtime there too, otherwise actual rt tasks
-	 * would be able to exceed the shared quota.
-	 *
-	 * Account to the root rt group for now.
-	 *
-	 * The solution we're working towards is having the RT groups scheduled
-	 * using deadline servers -- however there's a few nasties to figure
-	 * out before that can happen.
-	 */
-	if (rt_bandwidth_enabled()) {
-		struct rt_rq *rt_rq = &rq->rt;
-
-		raw_spin_lock(&rt_rq->rt_runtime_lock);
-		/*
-		 * We'll let actual RT tasks worry about the overflow here, we
-		 * have our own CBS to keep us inline; only account when RT
-		 * bandwidth is relevant.
-		 */
-		if (sched_rt_bandwidth_account(rt_rq))
-			rt_rq->rt_time += delta_exec;
-		raw_spin_unlock(&rt_rq->rt_runtime_lock);
-	}
-#endif /* CONFIG_RT_GROUP_SCHED */
 }
 
 /*
kernel/sched/debug.c +0 -6
index 02e16b70a79..efcf8d82f85 100644
@@ -890,12 +890,6 @@ void print_rt_rq(struct seq_file *m, int cpu, struct rt_rq *rt_rq)
 
 	PU(rt_nr_running);
 
-#ifdef CONFIG_RT_GROUP_SCHED
-	P(rt_throttled);
-	PN(rt_time);
-	PN(rt_runtime);
-#endif
-
 #undef PN
 #undef PU
 #undef P
kernel/sched/rt.c +0 -0
index a599f63bf7f..c625ea45ca7 100644

kernel/sched/rt.c line 1

Quoted 1 line
kernel/sched/rt.c +1 -0
@@ -1,3 +1,4 @@
+#pragma GCC diagnostic ignored "-Wunused-function"

Uh oh, guess this goes away with later patches? It's anyway not very nice unfortunately and it also breaks the SPDX licence first requirement.

kernel/sched/rt.c line 4

Quoted 3 lines
kernel/sched/rt.c +0 -0
 // SPDX-License-Identifier: GPL-2.0
 /*
  * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR

So, this leaves only a skeleton of the current RT_GROUP implementation. I believe cgroup ABI will still be there, but it won't have an effect. Since this is part of an atomic all-or-nothing set of changes, maybe it's OK? Guess people can get confused if for some reason they end up using a kernel with patches only up to this one. :-)

Best,
Juri

#11 lore raw

RFC PATCH v3 6/24 sched/rt: Introduce HCBS specific structs in task_group

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Each task_group manages a number of new objects:
- a sched_dl_entity/dl_server for each CPU
- a dl_bandwidth object to keep track of its allocated dl_bandwidth

---
 kernel/sched/sched.h | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
kernel/sched/sched.h +16 -2
index 2d373a1ba67..59a154505d8 100644
@@ -319,6 +319,13 @@ struct rt_bandwidth {
 	unsigned int		rt_period_active;
 };
 
+struct dl_bandwidth {
+	raw_spinlock_t          dl_runtime_lock;
+	u64                     dl_runtime;
+	u64                     dl_period;
+};
+
+
 static inline int dl_bandwidth_enabled(void)
 {
 	return sysctl_sched_rt_runtime >= 0;
@@ -467,10 +474,17 @@ struct task_group {
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 #ifdef CONFIG_RT_GROUP_SCHED
+	/*
+	 * Each task group manages a different scheduling entity per CPU, i.e. a
+	 * different deadline server, and a runqueue per CPU. All the dl-servers
+	 * share the same dl_bandwidth object.
+	 */
 	struct sched_rt_entity	**rt_se;
+	struct sched_dl_entity	**dl_se;
 	struct rt_rq		**rt_rq;
 
 	struct rt_bandwidth	rt_bandwidth;
+	struct dl_bandwidth	dl_bandwidth;
 #endif
 
 	struct scx_task_group	scx;
@@ -817,12 +831,12 @@ struct rt_rq {
 	raw_spinlock_t		rt_runtime_lock;
 
 	unsigned int		rt_nr_boosted;
-
-	struct rq		*rq; /* this is always top-level rq, cache? */
 #endif
 #ifdef CONFIG_CGROUP_SCHED
 	struct task_group	*tg; /* this tg has "this" rt_rq on given CPU for runnable entities */
 #endif
+
+	struct rq		*rq; /* cgroup's runqueue if the rt_rq entity belongs to a cgroup, otherwise top-level rq */
 };
 
 static inline bool rt_rq_is_runnable(struct rt_rq *rt_rq)
Signature · 1 line
2.51.0
maintainer
#12 lore raw

Re: [RFC PATCH v3 06/24] sched/rt: Introduce HCBS specific structs in task_group

Hello,

Quoted 14 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Each task_group manages a number of new objects:
- a sched_dl_entity/dl_server for each CPU
- a dl_bandwidth object to keep track of its allocated dl_bandwidth

---

...

Quoted 20 lines
kernel/sched/sched.h +7 -0
@@ -467,10 +474,17 @@ struct task_group {
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 
 #ifdef CONFIG_RT_GROUP_SCHED
+	/*
+	 * Each task group manages a different scheduling entity per CPU, i.e. a
+	 * different deadline server, and a runqueue per CPU. All the dl-servers
+	 * share the same dl_bandwidth object.
+	 */
 	struct sched_rt_entity	**rt_se;
+	struct sched_dl_entity	**dl_se;
 	struct rt_rq		**rt_rq;
 
 	struct rt_bandwidth	rt_bandwidth;
+	struct dl_bandwidth	dl_bandwidth;
 #endif
 
 	struct scx_task_group	scx;
@@ -817,12 +831,12 @@ struct rt_rq {
 	raw_spinlock_t		rt_runtime_lock;

kernel/sched/sched.h struct rt_rq { · line 839

Quoted 10 lines
kernel/sched/sched.h +2 -2
 
 	unsigned int		rt_nr_boosted;
-
-	struct rq		*rq; /* this is always top-level rq, cache? */
 #endif
 #ifdef CONFIG_CGROUP_SCHED
 	struct task_group	*tg; /* this tg has "this" rt_rq on given CPU for runnable entities */
 #endif
+
+	struct rq		*rq; /* cgroup's runqueue if the rt_rq entity belongs to a cgroup, otherwise top-level rq */

The comment really wants to span multiple lines, maybe above the pointer.

Thanks,
Juri

#13 lore raw

RFC PATCH v3 7/24 sched/core: Initialize root_task_group

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Add the initialization function for task_group's dl_servers. Initialize the default bandwidth for rt-cgroups and the root control group. Add utility functions to check (and get) if a rt_rq entity is connected to a real-time cgroup.

---
 kernel/sched/autogroup.c |  4 ++--
 kernel/sched/core.c      |  9 +++++++--
 kernel/sched/deadline.c  |  8 ++++++++
 kernel/sched/rt.c        | 18 ++++++++----------
 kernel/sched/sched.h     | 32 +++++++++++++++++++++++++++++---
 5 files changed, 54 insertions(+), 17 deletions(-)
kernel/sched/autogroup.c +2 -2
index cdea931aae3..017eadc0a0a 100644
@@ -52,7 +52,7 @@ static inline void autogroup_destroy(struct kref *kref)
 
 #ifdef CONFIG_RT_GROUP_SCHED
 	/* We've redirected RT tasks to the root task group... */
-	ag->tg->rt_se = NULL;
+	ag->tg->dl_se = NULL;
 	ag->tg->rt_rq = NULL;
 #endif
 	sched_release_group(ag->tg);
@@ -109,7 +109,7 @@ static inline struct autogroup *autogroup_create(void)
 	 * the policy change to proceed.
 	 */
 	free_rt_sched_group(tg);
-	tg->rt_se = root_task_group.rt_se;
+	tg->dl_se = root_task_group.dl_se;
 	tg->rt_rq = root_task_group.rt_rq;
 #endif /* CONFIG_RT_GROUP_SCHED */
 	tg->autogroup = ag;
kernel/sched/core.c +7 -2
index 5791aa1f8c8..e5b4facee24 100644
@@ -8710,7 +8710,7 @@ void __init sched_init(void)
 		scx_tg_init(&root_task_group);
 #endif /* CONFIG_EXT_GROUP_SCHED */
 #ifdef CONFIG_RT_GROUP_SCHED
-		root_task_group.rt_se = (struct sched_rt_entity **)ptr;
+		root_task_group.dl_se = (struct sched_dl_entity **)ptr;
 		ptr += nr_cpu_ids * sizeof(void **);
 
 		root_task_group.rt_rq = (struct rt_rq **)ptr;
@@ -8721,6 +8721,11 @@ void __init sched_init(void)
 
 	init_defrootdomain();
 
+#ifdef CONFIG_RT_GROUP_SCHED
+	init_dl_bandwidth(&root_task_group.dl_bandwidth,
+			global_rt_period(), global_rt_runtime());
+#endif /* CONFIG_RT_GROUP_SCHED */
+
 #ifdef CONFIG_CGROUP_SCHED
 	task_group_cache = KMEM_CACHE(task_group, 0);
 
@@ -8772,7 +8777,7 @@ void __init sched_init(void)
 		 * starts working after scheduler_running, which is not the case
 		 * yet.
 		 */
-		init_tg_rt_entry(&root_task_group, &rq->rt, NULL, i, NULL);
+		init_tg_rt_entry(&root_task_group, rq, NULL, i, NULL);
 #endif
 		rq->sd = NULL;
 		rq->rd = NULL;
kernel/sched/deadline.c +8 -0
index 277fbaff8b5..5c8fe80b585 100644
@@ -513,6 +513,14 @@ static inline int is_leftmost(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq
 
 static void init_dl_rq_bw_ratio(struct dl_rq *dl_rq);
 
+void init_dl_bandwidth(struct dl_bandwidth *dl_b, u64 period, u64 runtime)
+{
+	raw_spin_lock_init(&dl_b->dl_runtime_lock);
+	dl_b->dl_period = period;
+	dl_b->dl_runtime = runtime;
+}
+
+
 void init_dl_bw(struct dl_bw *dl_b)
 {
 	raw_spin_lock_init(&dl_b->lock);
kernel/sched/rt.c +8 -10
index c625ea45ca7..84dbb4853b6 100644
@@ -97,18 +97,16 @@ void free_rt_sched_group(struct task_group *tg)
 		return;
 }
 
-void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq,
-		struct sched_rt_entity *rt_se, int cpu,
-		struct sched_rt_entity *parent)
+void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq,
+		struct sched_dl_entity *dl_se, int cpu,
+		struct sched_dl_entity *parent)
 {
-	struct rq *rq = cpu_rq(cpu);
+	served_rq->rt.highest_prio.curr = MAX_RT_PRIO-1;
+	served_rq->rt.rq = cpu_rq(cpu);
+	served_rq->rt.tg = tg;
 
-	rt_rq->highest_prio.curr = MAX_RT_PRIO-1;
-	rt_rq->rq = rq;
-	rt_rq->tg = tg;
-
-	tg->rt_rq[cpu] = rt_rq;
-	tg->rt_se[cpu] = rt_se;
+	tg->rt_rq[cpu] = &served_rq->rt;
+	tg->dl_se[cpu] = dl_se;
 }
 
 int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
kernel/sched/sched.h +29 -3
index 59a154505d8..d433abe1c9c 100644
@@ -576,9 +576,9 @@ extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
 extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
 extern bool cfs_task_bw_constrained(struct task_struct *p);
 
-extern void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq,
-		struct sched_rt_entity *rt_se, int cpu,
-		struct sched_rt_entity *parent);
+extern void init_tg_rt_entry(struct task_group *tg, struct rq *s_rq,
+		struct sched_dl_entity *rt_se, int cpu,
+		struct sched_dl_entity *parent);
 extern int sched_group_set_rt_runtime(struct task_group *tg, long rt_runtime_us);
 extern int sched_group_set_rt_period(struct task_group *tg, u64 rt_period_us);
 extern long sched_group_rt_runtime(struct task_group *tg);
@@ -2668,6 +2668,7 @@ extern void resched_curr(struct rq *rq);
 extern void resched_curr_lazy(struct rq *rq);
 extern void resched_cpu(int cpu);
 
+void init_dl_bandwidth(struct dl_bandwidth *dl_b, u64 period, u64 runtime);
 extern void init_dl_entity(struct sched_dl_entity *dl_se);
 
 #define BW_SHIFT		20
@@ -3077,6 +3078,21 @@ static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
 	WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
 	return rt_rq->rq;
 }
+
+static inline int is_dl_group(struct rt_rq *rt_rq)
+{
+	return rt_rq->tg != &root_task_group;
+}
+
+/*
+ * Return the scheduling entity of this group of tasks.
+ */
+static inline struct sched_dl_entity *dl_group_of(struct rt_rq *rt_rq)
+{
+	BUG_ON(!is_dl_group(rt_rq));
+
+	return rt_rq->tg->dl_se[cpu_of(rt_rq->rq)];
+}
 #else
 static inline struct task_struct *rt_task_of(struct sched_rt_entity *rt_se)
 {
@@ -3101,6 +3117,16 @@ static inline struct rt_rq *rt_rq_of_se(struct sched_rt_entity *rt_se)
 
 	return &rq->rt;
 }
+
+static inline int is_dl_group(struct rt_rq *rt_rq)
+{
+	return 0;
+}
+
+static inline struct sched_dl_entity *dl_group_of(struct rt_rq *rt_rq)
+{
+	return NULL;
+}
 #endif
 
 DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq,
Signature · 1 line
2.51.0
maintainer
#14 lore raw

Re: [RFC PATCH v3 07/24] sched/core: Initialize root_task_group

Hello,

Quoted 6 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Add the initialization function for task_group's dl_servers. Initialize the default bandwidth for rt-cgroups and the root control group. Add utility functions to check (and get) if a rt_rq entity is connected to a real-time cgroup.

It's a bit of a nit, but this patch does quite more than $SUBJECT. So, I'd say please either reword $SUBJECT or consider splitting it further.

...

Quoted 6 lines
kernel/sched/sched.h +2 -0
@@ -3077,6 +3078,21 @@ static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se)
 	WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
 	return rt_rq->rq;
 }
+
+static inline int is_dl_group(struct rt_rq *rt_rq)

kernel/sched/sched.h static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se) · line 3092

Quoted 10 lines
kernel/sched/sched.h +10 -0
+{
+	return rt_rq->tg != &root_task_group;
+}
+
+/*
+ * Return the scheduling entity of this group of tasks.
+ */
+static inline struct sched_dl_entity *dl_group_of(struct rt_rq *rt_rq)
+{
+	BUG_ON(!is_dl_group(rt_rq));

BUG_ON() should generally be avoided, if possible. Could a WARN_ON_ONCE() plus returning NULL work here?

kernel/sched/sched.h static inline struct rq *rq_of_rt_se(struct sched_rt_entity *rt_se) · line 3095

Quoted 3 lines
kernel/sched/sched.h +3 -0
+
+	return rt_rq->tg->dl_se[cpu_of(rt_rq->rq)];
+}

Thanks,
Juri

#15 lore raw

RFC PATCH v3 8/24 sched/deadline: Add dl_init_tg

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Add dl_init_tg to initialize and/or update a rt-cgroup dl_server and to also account the allocated bandwidth.

---
 kernel/sched/deadline.c | 33 +++++++++++++++++++++++++++++++++
 kernel/sched/sched.h    |  1 +
 2 files changed, 34 insertions(+)
kernel/sched/deadline.c +33 -0
index 5c8fe80b585..34a1494d782 100644
@@ -340,6 +340,39 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
 	cancel_dl_timer(dl_se, &dl_se->inactive_timer);
 }
 
+#ifdef CONFIG_RT_GROUP_SCHED
+void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period)
+{
+	struct rq *rq = container_of(dl_se->dl_rq, struct rq, dl);
+	int is_active;
+	u64 new_bw;
+
+	raw_spin_rq_lock_irq(rq);
+	is_active = dl_se->my_q->rt.rt_nr_running > 0;
+
+	update_rq_clock(rq);
+	dl_server_stop(dl_se);
+
+	new_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
+	dl_rq_change_utilization(rq, dl_se, new_bw);
+
+	dl_se->dl_runtime  = rt_runtime;
+	dl_se->dl_deadline = rt_period;
+	dl_se->dl_period   = rt_period;
+
+	dl_se->runtime = 0;
+	dl_se->deadline = 0;
+
+	dl_se->dl_bw = new_bw;
+	dl_se->dl_density = new_bw;
+
+	if (is_active)
+		dl_server_start(dl_se);
+
+	raw_spin_rq_unlock_irq(rq);
+}
+#endif
+
 static void dl_change_utilization(struct task_struct *p, u64 new_bw)
 {
 	WARN_ON_ONCE(p->dl.flags & SCHED_FLAG_SUGOV);
kernel/sched/sched.h +1 -0
index d433abe1c9c..9853f321363 100644
@@ -394,6 +394,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,
 		    dl_server_has_tasks_f has_tasks,
 		    dl_server_pick_f pick_task);
 extern void sched_init_dl_servers(void);
+extern void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period);
 
 extern void dl_server_update_idle_time(struct rq *rq,
 		    struct task_struct *p);
Signature · 1 line
2.51.0
maintainer
#16 lore raw

Re: [RFC PATCH v3 08/24] sched/deadline: Add dl_init_tg

Hello,

Quoted 22 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Add dl_init_tg to initialize and/or update a rt-cgroup dl_server and to also account the allocated bandwidth.

---
 kernel/sched/deadline.c | 33 +++++++++++++++++++++++++++++++++
 kernel/sched/sched.h    |  1 +
 2 files changed, 34 insertions(+)
kernel/sched/deadline.c +0 -0
index 5c8fe80b585..34a1494d782 100644

kernel/sched/deadline.c void cancel_inactive_timer(struct sched_dl_entity *dl_se) · line 344

Quoted 5 lines
kernel/sched/deadline.c +2 -0
@@ -340,6 +340,39 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
 	cancel_dl_timer(dl_se, &dl_se->inactive_timer);
 }
 
+#ifdef CONFIG_RT_GROUP_SCHED
+void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period)

This is not called from anywhere at this point, is it? I wonder if we want to introduce dead code for later usage or if it would make more sense to introduce it together with the calling sites.

kernel/sched/deadline.c void cancel_inactive_timer(struct sched_dl_entity *dl_se) · line 356

Quoted 12 lines
kernel/sched/deadline.c +12 -0
+{
+	struct rq *rq = container_of(dl_se->dl_rq, struct rq, dl);
+	int is_active;
+	u64 new_bw;
+
+	raw_spin_rq_lock_irq(rq);
+	is_active = dl_se->my_q->rt.rt_nr_running > 0;
+
+	update_rq_clock(rq);
+	dl_server_stop(dl_se);
+
+	new_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);

Is this correct? Not sure yet where/how dl_init_tg() is called, but don't we want to compute new_bw using rt_runtime and rt_period?

Thanks,
Juri

#17 lore raw

RFC PATCH v3 9/24 sched/rt: Add {alloc/free}_rt_sched_group

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Add allocation and deallocation code for rt-cgroups.
Declare dl_server specific functions (no implementation yet), needed by the allocation code.

---
 kernel/sched/rt.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 89 insertions(+)
kernel/sched/rt.c +89 -0
index 84dbb4853b6..3094f59d0c8 100644
@@ -93,8 +93,43 @@ void unregister_rt_sched_group(struct task_group *tg)
 
 void free_rt_sched_group(struct task_group *tg)
 {
+	int i;
+	unsigned long flags;
+	struct rq *served_rq;
+
 	if (!rt_group_sched_enabled())
 		return;
+
+	if (!tg->dl_se || !tg->rt_rq)
+		return;
+
+	for_each_possible_cpu(i) {
+		if (!tg->dl_se[i] || !tg->rt_rq[i])
+			continue;
+
+		/*
+		 * Shutdown the dl_server and free it
+		 *
+		 * Since the dl timer is going to be cancelled,
+		 * we risk to never decrease the running bw...
+		 * Fix this issue by changing the group runtime
+		 * to 0 immediately before freeing it.
+		 */
+		dl_init_tg(tg->dl_se[i], 0, tg->dl_se[i]->dl_period);
+
+		raw_spin_rq_lock_irqsave(cpu_rq(i), flags);
+		BUG_ON(tg->rt_rq[i]->rt_nr_running);
+		hrtimer_cancel(&tg->dl_se[i]->dl_timer);
+		raw_spin_rq_unlock_irqrestore(cpu_rq(i), flags);
+		kfree(tg->dl_se[i]);
+
+		/* Free the local per-cpu runqueue */
+		served_rq = container_of(tg->rt_rq[i], struct rq, rt);
+		kfree(served_rq);
+	}
+
+	kfree(tg->rt_rq);
+	kfree(tg->dl_se);
 }
 
 void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq,
@@ -109,11 +144,65 @@ void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq,
 	tg->dl_se[cpu] = dl_se;
 }
 
+static bool rt_server_has_tasks(struct sched_dl_entity *dl_se)
+{
+	return false;
+}
+
+static struct task_struct *rt_server_pick(struct sched_dl_entity *dl_se)
+{
+	return NULL;
+}
+
 int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
 {
+	struct rq *s_rq;
+	struct sched_dl_entity *dl_se;
+	int i;
+
 	if (!rt_group_sched_enabled())
 		return 1;
 
+	tg->rt_rq = kcalloc(nr_cpu_ids, sizeof(struct rt_rq *), GFP_KERNEL);
+	if (!tg->rt_rq)
+		return 0;
+
+	tg->dl_se = kcalloc(nr_cpu_ids, sizeof(dl_se), GFP_KERNEL);
+	if (!tg->dl_se) {
+		kfree(tg->rt_rq);
+		tg->rt_rq = NULL;
+		return 0;
+	}
+
+	init_dl_bandwidth(&tg->dl_bandwidth, 0, 0);
+
+	for_each_possible_cpu(i) {
+		s_rq = kzalloc_node(sizeof(struct rq),
+				     GFP_KERNEL, cpu_to_node(i));
+		if (!s_rq)
+			return 0;
+
+		dl_se = kzalloc_node(sizeof(struct sched_dl_entity),
+				     GFP_KERNEL, cpu_to_node(i));
+		if (!dl_se) {
+			kfree(s_rq);
+			return 0;
+		}
+
+		init_rt_rq(&s_rq->rt);
+		init_dl_entity(dl_se);
+		dl_se->dl_runtime = tg->dl_bandwidth.dl_runtime;
+		dl_se->dl_period = tg->dl_bandwidth.dl_period;
+		dl_se->dl_deadline = dl_se->dl_period;
+		dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
+		dl_se->dl_density = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
+		dl_se->dl_server = 1;
+
+		dl_server_init(dl_se, &cpu_rq(i)->dl, s_rq, rt_server_has_tasks, rt_server_pick);
+
+		init_tg_rt_entry(tg, s_rq, dl_se, i, parent->dl_se[i]);
+	}
+
 	return 1;
 }
 
Signature · 1 line
2.51.0
maintainer
#18 lore raw

Re: [RFC PATCH v3 09/24] sched/rt: Add {alloc/free}_rt_sched_group

Hello,

Quoted 41 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Add allocation and deallocation code for rt-cgroups.
Declare dl_server specific functions (no implementation yet), needed by the allocation code.

---
 kernel/sched/rt.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 89 insertions(+)
kernel/sched/rt.c +14 -0
index 84dbb4853b6..3094f59d0c8 100644
@@ -93,8 +93,43 @@ void unregister_rt_sched_group(struct task_group *tg)
 
 void free_rt_sched_group(struct task_group *tg)
 {
+	int i;
+	unsigned long flags;
+	struct rq *served_rq;
+
 	if (!rt_group_sched_enabled())
 		return;
+
+	if (!tg->dl_se || !tg->rt_rq)
+		return;
+
+	for_each_possible_cpu(i) {
+		if (!tg->dl_se[i] || !tg->rt_rq[i])
+			continue;
+
+		/*
+		 * Shutdown the dl_server and free it

kernel/sched/rt.c void unregister_rt_sched_group(struct task_group *tg) · line 121

Quoted 10 lines
kernel/sched/rt.c +10 -0
+		 *
+		 * Since the dl timer is going to be cancelled,
+		 * we risk to never decrease the running bw...
+		 * Fix this issue by changing the group runtime
+		 * to 0 immediately before freeing it.
+		 */
+		dl_init_tg(tg->dl_se[i], 0, tg->dl_se[i]->dl_period);
+
+		raw_spin_rq_lock_irqsave(cpu_rq(i), flags);
+		BUG_ON(tg->rt_rq[i]->rt_nr_running);

Crashing is probably a bit harsh. What didn't happen that should have happened if there are still tasks in the group at this point? Can we maybe WARN and try to recover w/o crashing?

Quoted 11 lines
kernel/sched/rt.c +11 -0
+		hrtimer_cancel(&tg->dl_se[i]->dl_timer);
+		raw_spin_rq_unlock_irqrestore(cpu_rq(i), flags);
+		kfree(tg->dl_se[i]);
+
+		/* Free the local per-cpu runqueue */
+		served_rq = container_of(tg->rt_rq[i], struct rq, rt);
+		kfree(served_rq);
+	}
+
+	kfree(tg->rt_rq);
+	kfree(tg->dl_se);

...

Quoted 4 lines

int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent) {

kernel/sched/rt.c +2 -0
+	struct rq *s_rq;
+	struct sched_dl_entity *dl_se;

kernel/sched/rt.c void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, · line 170

Quoted 10 lines
kernel/sched/rt.c +7 -0
+	int i;
+
 	if (!rt_group_sched_enabled())
 		return 1;
 
+	tg->rt_rq = kcalloc(nr_cpu_ids, sizeof(struct rt_rq *), GFP_KERNEL);
+	if (!tg->rt_rq)
+		return 0;
+
+	tg->dl_se = kcalloc(nr_cpu_ids, sizeof(dl_se), GFP_KERNEL);

Nit. Maybe sizeof(struct sched_dl_entity *) is more clear and consistent with the above (current form is still correct, of course).

Quoted 10 lines
kernel/sched/rt.c +10 -0
+	if (!tg->dl_se) {
+		kfree(tg->rt_rq);
+		tg->rt_rq = NULL;
+		return 0;
+	}
+
+	init_dl_bandwidth(&tg->dl_bandwidth, 0, 0);
+
+	for_each_possible_cpu(i) {
+		s_rq = kzalloc_node(sizeof(struct rq),

kernel/sched/rt.c void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, · line 190

Quoted 10 lines
kernel/sched/rt.c +10 -0
+				     GFP_KERNEL, cpu_to_node(i));
+		if (!s_rq)
+			return 0;
+
+		dl_se = kzalloc_node(sizeof(struct sched_dl_entity),
+				     GFP_KERNEL, cpu_to_node(i));
+		if (!dl_se) {
+			kfree(s_rq);
+			return 0;
+		}

Are we not leaking if allocation failure happens mid way during this for loop? Do we free data structures allocated for previous CPUs?

Quoted 7 lines
kernel/sched/rt.c +7 -0
+
+		init_rt_rq(&s_rq->rt);
+		init_dl_entity(dl_se);
+		dl_se->dl_runtime = tg->dl_bandwidth.dl_runtime;
+		dl_se->dl_period = tg->dl_bandwidth.dl_period;
+		dl_se->dl_deadline = dl_se->dl_period;
+		dl_se->dl_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);

kernel/sched/rt.c void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, · line 207

Quoted 10 lines
kernel/sched/rt.c +8 -0
+		dl_se->dl_density = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
+		dl_se->dl_server = 1;
+
+		dl_server_init(dl_se, &cpu_rq(i)->dl, s_rq, rt_server_has_tasks, rt_server_pick);
+
+		init_tg_rt_entry(tg, s_rq, dl_se, i, parent->dl_se[i]);
+	}
+
 	return 1;
 }

Thanks,
Juri

#19 lore raw

RFC PATCH v3 10/24 sched/deadline: Account rt-cgroups bandwidth in deadline tasks schedulability tests.

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Account the rt-cgroups hierarchy's reserved bandwidth in the schedulability test of deadline entities. This mechanism allows to completely reserve portion of the rt-bandwidth to rt-cgroups even if they do not use all of it.

Account for the rt-cgroups' reserved bandwidth also when changing the total dedicated bandwidth for real time tasks.

---
 kernel/sched/deadline.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
kernel/sched/deadline.c +18 -3
index 34a1494d782..754bfe231b4 100644
@@ -213,8 +213,15 @@ void __dl_add(struct dl_bw *dl_b, u64 tsk_bw, int cpus)
 static inline bool
 __dl_overflow(struct dl_bw *dl_b, unsigned long cap, u64 old_bw, u64 new_bw)
 {
+	u64 dl_groups_root = 0;
+
+#ifdef CONFIG_RT_GROUP_SCHED
+	dl_groups_root = to_ratio(root_task_group.dl_bandwidth.dl_period,
+				  root_task_group.dl_bandwidth.dl_runtime);
+#endif
 	return dl_b->bw != -1 &&
-	       cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw;
+	       cap_scale(dl_b->bw, cap) < dl_b->total_bw - old_bw + new_bw
+					+ cap_scale(dl_groups_root, cap);
 }
 
 static inline
@@ -3150,10 +3157,16 @@ int sched_dl_global_validate(void)
 	u64 period = global_rt_period();
 	u64 new_bw = to_ratio(period, runtime);
 	u64 cookie = ++dl_cookie;
+	u64 dl_groups_root = 0;
 	struct dl_bw *dl_b;
-	int cpu, cpus, ret = 0;
+	int cpu, cap, cpus, ret = 0;
 	unsigned long flags;
 
+#ifdef CONFIG_RT_GROUP_SCHED
+	dl_groups_root = to_ratio(root_task_group.dl_bandwidth.dl_period,
+				  root_task_group.dl_bandwidth.dl_runtime);
+#endif
+
 	/*
 	 * Here we want to check the bandwidth not being set to some
 	 * value smaller than the currently allocated bandwidth in
@@ -3166,10 +3179,12 @@ int sched_dl_global_validate(void)
 			goto next;
 
 		dl_b = dl_bw_of(cpu);
+		cap = dl_bw_capacity(cpu);
 		cpus = dl_bw_cpus(cpu);
 
 		raw_spin_lock_irqsave(&dl_b->lock, flags);
-		if (new_bw * cpus < dl_b->total_bw)
+		if (new_bw * cpus < dl_b->total_bw +
+				    cap_scale(dl_groups_root, cap))
 			ret = -EBUSY;
 		raw_spin_unlock_irqrestore(&dl_b->lock, flags);
 
Signature · 1 line
2.51.0
#20 lore raw

RFC PATCH v3 11/24 sched/rt: Add rt-cgroups' dl-servers operations.

In reply to Yuri Andriaccio

Implement the servers' functions that pick the next eligible task to run. Enable/Disable dl-servers on task enqueue/dequeue when necessary. Update dl-servers on task update.
Account the number of active rt-tasks in the cgroups' specific runqueue. Account the number of active rt-tasks on the global counter of active tasks when a cgroup is enqueued/dequeued (dl-server started/stopped).
Update rq's cpuprio only if the cgroup's is root control group. Record which dl_server is managing a task when it changes runqueue.

---
 kernel/sched/deadline.c | 16 ++++++---
 kernel/sched/rt.c       | 79 ++++++++++++++++++++++++++++++++++++-----
 kernel/sched/sched.h    |  3 +-
 3 files changed, 85 insertions(+), 13 deletions(-)
kernel/sched/deadline.c +12 -4
index 754bfe231b4..1293b9a252b 100644
@@ -1869,9 +1869,13 @@ void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 	u64 deadline = dl_se->deadline;
 
 	dl_rq->dl_nr_running++;
-
-	if (!dl_server(dl_se))
+	if (!dl_server(dl_se)) {
 		add_nr_running(rq_of_dl_rq(dl_rq), 1);
+	} else if (dl_se != &rq_of_dl_rq(dl_rq)->fair_server) {
+		struct rt_rq *rt_rq = &dl_se->my_q->rt;
+
+		add_nr_running(rq_of_dl_rq(dl_rq), rt_rq->rt_nr_running);
+	}
 
 	inc_dl_deadline(dl_rq, deadline);
 }
@@ -1881,9 +1885,13 @@ void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 {
 	WARN_ON(!dl_rq->dl_nr_running);
 	dl_rq->dl_nr_running--;
-
-	if (!dl_server(dl_se))
+	if (!dl_server(dl_se)) {
 		sub_nr_running(rq_of_dl_rq(dl_rq), 1);
+	} else if (dl_se != &rq_of_dl_rq(dl_rq)->fair_server) {
+		struct rt_rq *rt_rq = &dl_se->my_q->rt;
+
+		sub_nr_running(rq_of_dl_rq(dl_rq), rt_rq->rt_nr_running);
+	}
 
 	dec_dl_deadline(dl_rq, dl_se->deadline);
 }
kernel/sched/rt.c +71 -8

153 lines not on this page · raw message

kernel/sched/sched.h +2 -1
index 9853f321363..b2c87541257 100644
@@ -2170,7 +2170,7 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)
 	if (!rt_group_sched_enabled())
 		tg = &root_task_group;
 	p->rt.rt_rq  = tg->rt_rq[cpu];
-	p->rt.parent = tg->rt_se[cpu];
+	p->dl.dl_rq  = &cpu_rq(cpu)->dl;
 #endif /* CONFIG_RT_GROUP_SCHED */
 }
 
@@ -2726,6 +2726,7 @@ static inline void add_nr_running(struct rq *rq, unsigned count)
 
 static inline void sub_nr_running(struct rq *rq, unsigned count)
 {
+	BUG_ON(rq->nr_running < count);
 	rq->nr_running -= count;
 	if (trace_sched_update_nr_running_tp_enabled()) {
 		call_trace_sched_update_nr_running(rq, -count);
Signature · 1 line
2.51.0
maintainer
#21 lore raw

Re: [RFC PATCH v3 11/24] sched/rt: Add rt-cgroups' dl-servers operations.

Hello,

Quoted 8 lines · Yuri Andriaccio

Implement the servers' functions that pick the next eligible task to run. Enable/Disable dl-servers on task enqueue/dequeue when necessary. Update dl-servers on task update.
Account the number of active rt-tasks in the cgroups' specific runqueue. Account the number of active rt-tasks on the global counter of active tasks when a cgroup is enqueued/dequeued (dl-server started/stopped).
Update rq's cpuprio only if the cgroup's is root control group. Record which dl_server is managing a task when it changes runqueue.

Changelog looks a little dry. Claude suggests a little rewrite like

---
sched/rt: Implement dl-server operations for rt-cgroups

Implement the dl-server backend that enables rt-cgroups to run as deadline servers. This allows RT tasks within a cgroup to be scheduled according to the cgroup's allocated bandwidth using deadline scheduling.

The implementation consists of three main parts:

1) Server task selection callbacks:
- rt_server_has_tasks(): Check if the rt_rq has runnable tasks
- rt_server_pick(): Pick and set the next RT task from the cgroup's rt_rq when the server gets CPU time

2) Server lifecycle management:
- Start the dl-server when the first RT task enqueues to an idle rt-cgroup
- Stop the dl-server when the last RT task dequeues from an rt-cgroup
- Update the server's consumed runtime in update_curr_rt() via dl_server_update()

3) Per-cpu priority and nr_running accounting:
- Only update rq->cpupri for the root rt_rq (not for cgroup rt_rqs) since cgroups are scheduled via their dl-server priority
- For cgroup rt_rqs, update global nr_running only when the dl-server is active (not throttled), as the server acts as the runnable entity
- Bulk update nr_running when the server starts/stops based on the rt_rq's current rt_nr_running count

The rt.parent field is removed as the new implementation doesn't use hierarchical RT scheduling entities. Instead, tasks record their dl_rq to track which dl-server manages them.
---

Which seems to correspond to what this patch does. If that's the case, however, I wonder if we are maybe doing too many things at once?

Quoted 18 lines
---
 kernel/sched/deadline.c | 16 ++++++---
 kernel/sched/rt.c       | 79 ++++++++++++++++++++++++++++++++++++-----
 kernel/sched/sched.h    |  3 +-
 3 files changed, 85 insertions(+), 13 deletions(-)
kernel/sched/deadline.c +0 -0
index 754bfe231b4..1293b9a252b 100644

kernel/sched/deadline.c void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq) · line 1874

Quoted 8 lines
kernel/sched/deadline.c +2 -2
@@ -1869,9 +1869,13 @@ void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 	u64 deadline = dl_se->deadline;
 
 	dl_rq->dl_nr_running++;
-
-	if (!dl_server(dl_se))
+	if (!dl_server(dl_se)) {
 		add_nr_running(rq_of_dl_rq(dl_rq), 1);
+	} else if (dl_se != &rq_of_dl_rq(dl_rq)->fair_server) {

I fear this condition might get unwieldy with the addition of new servers (e.g. sched_scx).

Quoted 8 lines
kernel/sched/deadline.c +4 -0
+		struct rt_rq *rt_rq = &dl_se->my_q->rt;
+
+		add_nr_running(rq_of_dl_rq(dl_rq), rt_rq->rt_nr_running);
+	}
 
 	inc_dl_deadline(dl_rq, deadline);
 }

kernel/sched/deadline.c void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq) · line 1890

Quoted 8 lines
kernel/sched/deadline.c +2 -2
@@ -1881,9 +1885,13 @@ void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq)
 {
 	WARN_ON(!dl_rq->dl_nr_running);
 	dl_rq->dl_nr_running--;
-
-	if (!dl_server(dl_se))
+	if (!dl_server(dl_se)) {
 		sub_nr_running(rq_of_dl_rq(dl_rq), 1);
+	} else if (dl_se != &rq_of_dl_rq(dl_rq)->fair_server) {

Ditto.

Quoted 22 lines
kernel/sched/deadline.c +4 -0
+		struct rt_rq *rt_rq = &dl_se->my_q->rt;
+
+		sub_nr_running(rq_of_dl_rq(dl_rq), rt_rq->rt_nr_running);
+	}
 
 	dec_dl_deadline(dl_rq, dl_se->deadline);
 }
kernel/sched/rt.c +4 -1
index 3094f59d0c8..d9442f64c6b 100644
@@ -144,14 +144,27 @@ void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq,
 	tg->dl_se[cpu] = dl_se;
 }
 
+static struct task_struct *_pick_next_task_rt(struct rt_rq *rt_rq);
+static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool first);
+
 static bool rt_server_has_tasks(struct sched_dl_entity *dl_se)
 {
-	return false;
+	return !!dl_se->my_q->rt.rt_nr_running;

kernel/sched/rt.c void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq, · line 161

Quoted 10 lines
kernel/sched/rt.c +5 -1
 }
 
 static struct task_struct *rt_server_pick(struct sched_dl_entity *dl_se)
 {
-	return NULL;
+	struct rt_rq *rt_rq = &dl_se->my_q->rt;
+	struct rq *rq = rq_of_rt_rq(rt_rq);
+	struct task_struct *p;
+
+	if (dl_se->my_q->rt.rt_nr_running == 0)

Can't we use rt_server_has_tasks()?

Quoted 6 lines
kernel/sched/rt.c +6 -0
+		return NULL;
+
+	p = _pick_next_task_rt(rt_rq);
+	set_next_task_rt(rq, p, true);
+
+	return p;

...

kernel/sched/rt.c static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq) · line 1019

Quoted 12 lines
kernel/sched/rt.c +5 -2
@@ -953,9 +1011,14 @@ static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq)
 	return next;
 }
 
-static struct task_struct *_pick_next_task_rt(struct rq *rq)
+static struct task_struct *_pick_next_task_rt(struct rt_rq *rt_rq)
 {
-	return NULL;
+	struct sched_rt_entity *rt_se;
+
+	rt_se = pick_next_rt_entity(rt_rq);
+	BUG_ON(!rt_se);

Can we WARN and recover somehow?

Quoted 19 lines
kernel/sched/rt.c +3 -1
+
+	return rt_task_of(rt_se);
 }
 
 static struct task_struct *pick_task_rt(struct rq *rq)
@@ -965,7 +1028,7 @@ static struct task_struct *pick_task_rt(struct rq *rq)
 	if (!sched_rt_runnable(rq))
 		return NULL;
 
-	p = _pick_next_task_rt(rq);
+	p = _pick_next_task_rt(&rq->rt);
 
 	return p;
 }
kernel/sched/sched.h +0 -0
index 9853f321363..b2c87541257 100644

kernel/sched/sched.h static inline void set_task_rq(struct task_struct *p, unsigned int cpu) · line 2173

Quoted 5 lines
kernel/sched/sched.h +1 -1
@@ -2170,7 +2170,7 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)
 	if (!rt_group_sched_enabled())
 		tg = &root_task_group;
 	p->rt.rt_rq  = tg->rt_rq[cpu];
-	p->rt.parent = tg->rt_se[cpu];
+	p->dl.dl_rq  = &cpu_rq(cpu)->dl;

Guess rt.parent is then removed in a subsequent patch? Do we want to consolidate the cleanup?

kernel/sched/sched.h static inline void add_nr_running(struct rq *rq, unsigned count) · line 2729

Quoted 8 lines
kernel/sched/sched.h +1 -0
 #endif /* CONFIG_RT_GROUP_SCHED */
 }
 
@@ -2726,6 +2726,7 @@ static inline void add_nr_running(struct rq *rq, unsigned count)
 
 static inline void sub_nr_running(struct rq *rq, unsigned count)
 {
+	BUG_ON(rq->nr_running < count);

Can we WARN and recover somehow?

kernel/sched/sched.h static inline void add_nr_running(struct rq *rq, unsigned count) · line 2732

Quoted 3 lines
kernel/sched/sched.h +0 -0
 	rq->nr_running -= count;
 	if (trace_sched_update_nr_running_tp_enabled()) {
 		call_trace_sched_update_nr_running(rq, -count);

Thanks,
Juri

#22 lore raw

RFC PATCH v3 12/24 sched/rt: Update task event callbacks for HCBS scheduling

In reply to Yuri Andriaccio

Update wakeup_preempt_rt, switched_{from/to}_rt and prio_changed_rt with rt-cgroup's specific preemption rules.
Add checks whether a rt-task can be attached or not to a rt-cgroup. Update task_is_throttled_rt for SCHED_CORE.

---
 kernel/sched/core.c     |  2 +-
 kernel/sched/rt.c       | 88 ++++++++++++++++++++++++++++++++++++++---
 kernel/sched/syscalls.c | 13 ++++++
 3 files changed, 96 insertions(+), 7 deletions(-)
kernel/sched/core.c +1 -1
index e5b4facee24..2cfbe3b7b17 100644
@@ -9346,7 +9346,7 @@ static int cpu_cgroup_can_attach(struct cgroup_taskset *tset)
 		goto scx_check;
 
 	cgroup_taskset_for_each(task, css, tset) {
-		if (!sched_rt_can_attach(css_tg(css), task))
+		if (rt_task(task) && !sched_rt_can_attach(css_tg(css), task))
 			return -EINVAL;
 	}
 scx_check:
kernel/sched/rt.c +82 -6

148 lines not on this page · raw message

kernel/sched/syscalls.c +13 -0
index 93a9c03b28e..71f20be6f29 100644
@@ -626,6 +626,19 @@ int __sched_setscheduler(struct task_struct *p,
 change:
 
 	if (user) {
+#ifdef CONFIG_RT_GROUP_SCHED
+		/*
+		 * Do not allow real-time tasks into groups that have no runtime
+		 * assigned.
+		 */
+		if (rt_group_sched_enabled() &&
+				dl_bandwidth_enabled() && rt_policy(policy) &&
+				!sched_rt_can_attach(task_group(p), p) &&
+				!task_group_is_autogroup(task_group(p))) {
+			retval = -EPERM;
+			goto unlock;
+		}
+#endif
 		if (dl_bandwidth_enabled() && dl_policy(policy) &&
 				!(attr->sched_flags & SCHED_FLAG_SUGOV)) {
 			cpumask_t *span = rq->rd->span;
Signature · 1 line
2.51.0
maintainer
#23 lore raw

Re: [RFC PATCH v3 12/24] sched/rt: Update task event callbacks for HCBS scheduling

Hello,

Quoted 4 lines · Yuri Andriaccio

Update wakeup_preempt_rt, switched_{from/to}_rt and prio_changed_rt with rt-cgroup's specific preemption rules.
Add checks whether a rt-task can be attached or not to a rt-cgroup. Update task_is_throttled_rt for SCHED_CORE.

A little dry. :) This is telling the what, can you please add also the why and how?

Quoted 31 lines
---
 kernel/sched/core.c     |  2 +-
 kernel/sched/rt.c       | 88 ++++++++++++++++++++++++++++++++++++++---
 kernel/sched/syscalls.c | 13 ++++++
 3 files changed, 96 insertions(+), 7 deletions(-)
kernel/sched/core.c +1 -1
index e5b4facee24..2cfbe3b7b17 100644
@@ -9346,7 +9346,7 @@ static int cpu_cgroup_can_attach(struct cgroup_taskset *tset)
 		goto scx_check;
 
 	cgroup_taskset_for_each(task, css, tset) {
-		if (!sched_rt_can_attach(css_tg(css), task))
+		if (rt_task(task) && !sched_rt_can_attach(css_tg(css), task))
 			return -EINVAL;
 	}
 scx_check:
kernel/sched/rt.c +0 -0
index d9442f64c6b..ce114823fe7 100644

kernel/sched/rt.c static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) · line 949

Quoted 4 lines
kernel/sched/rt.c +1 -0
@@ -946,6 +946,50 @@ static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags)
 {
 	struct task_struct *donor = rq->donor;
 
+	if (!rt_group_sched_enabled())

Should we actually be using rt_group_sched_enabled() (instead of ifdeffery) everywhere to check if group scheduling is enabled and differentiate behavior?

kernel/sched/rt.c static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) · line 956

Quoted 7 lines
kernel/sched/rt.c +7 -0
+		goto no_group_sched;
+
+	/*
+	 * Preemption checks are different if the waking task and the current
+	 * task are running on the global runqueue or in a cgroup.
+	 * The following rules apply:
+	 *   - dl-tasks (and equally dl_servers) always preempt FIFO/RR tasks.

We already touched upon this and maybe we can leave it for later, but the fact that dl_servers always preempt FIFO/RR is going to be a change of behavior wrt legacy RT group scheduling. Legacy users might have expectations that setting an high static priority means those tasks will always be scheduled first and this will break that. Maybe we want to convince them to change their assumptions, but we need to at least cleary document the new behavior (and why it is right/better - if it is indeed :). Not sure if we can also find a way to be "back-compatible" and if we want to do that.

Quoted 6 lines
kernel/sched/rt.c +6 -0
+	 *     - if curr is inside a cgroup (i.e. run by a dl_server) and
+	 *       waking is not, do nothing.
+	 *     - if waking is inside a cgroup but not curr, always reschedule.
+	 *   - if they are both on the global runqueue, run the standard code.
+	 *   - if they are both in the same cgroup, check for tasks priorities.
+	 *   - if they are both in a cgroup, but not the same one, check whether

kernel/sched/rt.c static void wakeup_preempt_rt(struct rq *rq, struct task_struct *p, int flags) · line 972

Quoted 10 lines
kernel/sched/rt.c +10 -0
+	 *     the woken task's dl_server preempts the current's dl_server.
+	 */
+	if (is_dl_group(rt_rq_of_se(&p->rt)) &&
+	    is_dl_group(rt_rq_of_se(&rq->curr->rt))) {
+		struct sched_dl_entity *woken_dl_se, *curr_dl_se;
+
+		woken_dl_se = dl_group_of(rt_rq_of_se(&p->rt));
+		curr_dl_se = dl_group_of(rt_rq_of_se(&rq->curr->rt));
+
+		if (rt_rq_of_se(&p->rt)->tg == rt_rq_of_se(&rq->curr->rt)->tg) {

What about checking if woken_dl_se and curr_dl_se are the same dl_se?

Quoted 23 lines
kernel/sched/rt.c +20 -0
+			if (p->prio < rq->curr->prio)
+				resched_curr(rq);
+
+			return;
+		}
+
+		if (dl_entity_preempt(woken_dl_se, curr_dl_se))
+			resched_curr(rq);
+
+		return;
+
+	} else if (is_dl_group(rt_rq_of_se(&p->rt))) {
+		resched_curr(rq);
+		return;
+
+	} else if (is_dl_group(rt_rq_of_se(&rq->curr->rt))) {
+		return;
+	}
+
+no_group_sched:
 	if (p->prio < donor->prio) {
 		resched_curr(rq);
 		return;

...

kernel/sched/rt.c static void switched_to_rt(struct rq *rq, struct task_struct *p) · line 1805

Quoted 10 lines
kernel/sched/rt.c +4 -0
@@ -1750,8 +1797,17 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)
 	 * then see if we can move to another run queue.
 	 */
 	if (task_on_rq_queued(p)) {
+
+#ifndef CONFIG_RT_GROUP_SCHED
 		if (p->nr_cpus_allowed > 1 && rq->rt.overloaded)
 			rt_queue_push_tasks(rt_rq_of_se(&p->rt));
+#else
+		if (rt_rq_of_se(&p->rt)->overloaded) {

Is this empty because intra-group migration is coming with some future change? Even if that's the case, I believe this wants a comment explaining why this branch is empty.

Quoted 8 lines
kernel/sched/rt.c +5 -0
+		} else {
+			if (p->prio < rq->curr->prio)
+				resched_curr(rq);
+		}
+#endif
 		if (p->prio < rq->donor->prio && cpu_online(cpu_of(rq)))
 			resched_curr(rq);
 	}

...

Quoted 18 lines
kernel/sched/rt.c +9 -0
@@ -1876,7 +1943,16 @@ static unsigned int get_rr_interval_rt(struct rq *rq, struct task_struct *task)
 #ifdef CONFIG_SCHED_CORE
 static int task_is_throttled_rt(struct task_struct *p, int cpu)
 {
+#ifdef CONFIG_RT_GROUP_SCHED
+	struct rt_rq *rt_rq;
+
+	rt_rq = task_group(p)->rt_rq[cpu];
+	WARN_ON(!rt_group_sched_enabled() && rt_rq->tg != &root_task_group);
+
+	return dl_group_of(rt_rq)->dl_throttled;
+#else
 	return 0;
+#endif
 }
 #endif /* CONFIG_SCHED_CORE */
 

kernel/sched/rt.c static int sched_rt_global_constraints(void) · line 2207

Quoted 1 line
kernel/sched/rt.c +0 -0
@@ -2131,7 +2207,7 @@ static int sched_rt_global_constraints(void)
 int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk)

tsk argument is not going to be used anymore with this, please remove it. Also maybe rename to something closer to what the function actually checks for, e.g. [sched_]rt_group_has_runtime.

kernel/sched/rt.c static int sched_rt_global_constraints(void) · line 2211

Quoted 5 lines
kernel/sched/rt.c +1 -1
 {
 	/* Don't accept real-time tasks when there is no way for them to run */
-	if (rt_group_sched_enabled() && rt_task(tsk) && tg->rt_bandwidth.rt_runtime == 0)
+	if (rt_group_sched_enabled() && tg->dl_bandwidth.dl_runtime == 0)
 		return 0;

Thanks,
Juri

#24 lore raw

RFC PATCH v3 13/24 sched/rt: Update rt-cgroup schedulability checks

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Update schedulability checks and setup of runtime/period for rt-cgroups.

---
 kernel/sched/core.c     |  6 ++++
 kernel/sched/deadline.c | 46 +++++++++++++++++++++++----
 kernel/sched/rt.c       | 70 +++++++++++++++++++++++------------------
 kernel/sched/sched.h    |  1 +
 4 files changed, 87 insertions(+), 36 deletions(-)
kernel/sched/core.c +6 -0
index 2cfbe3b7b17..1217f714dd2 100644
@@ -9281,6 +9281,12 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 		return &root_task_group.css;
 	}
 
+	/* Do not allow cpu_cgroup hierachies with depth greater than 2. */
+#ifdef CONFIG_RT_GROUP_SCHED
+	if (parent != &root_task_group)
+		return ERR_PTR(-EINVAL);
+#endif
+
 	tg = sched_create_group(parent);
 	if (IS_ERR(tg))
 		return ERR_PTR(-ENOMEM);
kernel/sched/deadline.c +40 -6

59 lines not on this page · raw message

kernel/sched/rt.c +40 -30

149 lines not on this page · raw message

kernel/sched/sched.h +1 -0
index b2c87541257..97e1e779df9 100644
@@ -394,6 +394,7 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,
 		    dl_server_has_tasks_f has_tasks,
 		    dl_server_pick_f pick_task);
 extern void sched_init_dl_servers(void);
+extern int dl_check_tg(unsigned long total);
 extern void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period);
 
 extern void dl_server_update_idle_time(struct rq *rq,
Signature · 1 line
2.51.0
#25 lore raw

Re: [RFC PATCH v3 13/24] sched/rt: Update rt-cgroup schedulability checks

…

Quoted 2 lines

+++ b/kernel/sched/deadline.c

(quoted diff) +0 -0
@@ -340,6 +340,39 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)

…

Quoted 3 lines
(quoted diff) +3 -0
+#ifdef CONFIG_RT_GROUP_SCHED
+void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period)
+{

…

Quoted 3 lines
(quoted diff) +3 -0
+	u64 new_bw;
+
+	raw_spin_rq_lock_irq(rq);

…

Review

Quoted 3 lines
(quoted diff) +3 -0
+	raw_spin_rq_unlock_irq(rq);
+}
+#endif

…

How do you think about to define and use a corresponding lock guard?

Regards,
Markus

maintainer
#26 lore raw

Re: [RFC PATCH v3 13/24] sched/rt: Update rt-cgroup schedulability checks

In reply to Yuri Andriaccio

Hello,

Quoted 3 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Update schedulability checks and setup of runtime/period for rt-cgroups.

So, it looks like changelogs are all too minimal and dry. Having a more comprehensive (but concise) description of what the patch does, why it does it and how, will help reviewing and is essential in the future to trace back design decisions and issues. Please consider this for current and previous/subsequent patches.

Quoted 19 lines
---
 kernel/sched/core.c     |  6 ++++
 kernel/sched/deadline.c | 46 +++++++++++++++++++++++----
 kernel/sched/rt.c       | 70 +++++++++++++++++++++++------------------
 kernel/sched/sched.h    |  1 +
 4 files changed, 87 insertions(+), 36 deletions(-)
kernel/sched/core.c +0 -0
index 2cfbe3b7b17..1217f714dd2 100644

kernel/sched/core.c cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) · line 9284

Quoted 4 lines
kernel/sched/core.c +1 -0
@@ -9281,6 +9281,12 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 		return &root_task_group.css;
 	}
 
+	/* Do not allow cpu_cgroup hierachies with depth greater than 2. */

I believe this limit gets removed with later patches, but since we have it here, we should state why we have the limit in place.

Quoted 32 lines
kernel/sched/core.c +5 -0
+#ifdef CONFIG_RT_GROUP_SCHED
+	if (parent != &root_task_group)
+		return ERR_PTR(-EINVAL);
+#endif
+
 	tg = sched_create_group(parent);
 	if (IS_ERR(tg))
 		return ERR_PTR(-ENOMEM);
kernel/sched/deadline.c +15 -0
index 1293b9a252b..5d93b3ca030 100644
@@ -347,7 +347,47 @@ void cancel_inactive_timer(struct sched_dl_entity *dl_se)
 	cancel_dl_timer(dl_se, &dl_se->inactive_timer);
 }
 
+/*
+ * Used for dl_bw check and update, used under sched_rt_handler()::mutex and
+ * sched_domains_mutex.
+ */
+u64 dl_cookie;
+
 #ifdef CONFIG_RT_GROUP_SCHED
+int dl_check_tg(unsigned long total)
+{
+	unsigned long flags;
+	int which_cpu;
+	int cpus;
+	struct dl_bw *dl_b;
+	u64 gen = ++dl_cookie;
+
+	for_each_possible_cpu(which_cpu) {

kernel/sched/deadline.c void cancel_inactive_timer(struct sched_dl_entity *dl_se) · line 375

Quoted 10 lines
kernel/sched/deadline.c +10 -0
+		rcu_read_lock_sched();
+
+		if (!dl_bw_visited(which_cpu, gen)) {
+			cpus = dl_bw_cpus(which_cpu);
+			dl_b = dl_bw_of(which_cpu);
+
+			raw_spin_lock_irqsave(&dl_b->lock, flags);
+
+			if (dl_b->bw != -1 &&
+			    dl_b->bw * cpus < dl_b->total_bw + total * cpus) {

Does this need to use cap_scale()?

Quoted 15 lines
kernel/sched/deadline.c +15 -0
+				raw_spin_unlock_irqrestore(&dl_b->lock, flags);
+				rcu_read_unlock_sched();
+
+				return 0;
+			}
+
+			raw_spin_unlock_irqrestore(&dl_b->lock, flags);
+		}
+
+		rcu_read_unlock_sched();
+	}
+
+	return 1;
+}
+

...

kernel/sched/deadline.c static int tg_rt_schedulable(struct task_group *tg, void *data) · line 2033

Quoted 8 lines
kernel/sched/deadline.c +2 -2
@@ -2034,8 +2029,8 @@ static int tg_rt_schedulable(struct task_group *tg, void *data)
 	unsigned long total, sum = 0;
 	u64 period, runtime;
 
-	period = ktime_to_ns(tg->rt_bandwidth.rt_period);
-	runtime = tg->rt_bandwidth.rt_runtime;
+	period  = tg->dl_bandwidth.dl_period;
+	runtime = tg->dl_bandwidth.dl_runtime;

Just as an example, this is the kind of important change (rt_bandwidth -> dl_bandwidth) that usually deserves to be explicitly mentioned in the changelog.

...

kernel/sched/deadline.c static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime) · line 2106

Quoted 12 lines
kernel/sched/deadline.c +8 -0
@@ -2097,6 +2096,20 @@ static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime)
 		.rt_runtime = runtime,
 	};
 
+	/*
+	* Since we truncate DL_SCALE bits, make sure we're at least
+	* that big.
+	*/
+	if (runtime != 0 && runtime < (1ULL << DL_SCALE))
+		return -EINVAL;
+
+	/*
        ^

Nit, fix alignment.

kernel/sched/deadline.c static int __rt_schedulable(struct task_group *tg, u64 period, u64 runtime) · line 2112

Quoted 6 lines
kernel/sched/deadline.c +6 -0
+	* Since we use the MSB for wrap-around and sign issues, make
+	* sure it's not set (mind that period can be equal to zero).
+	*/
+	if (period & (1ULL << 63))
+		return -EINVAL;
+

Thanks,
Juri

#27 lore raw

RFC PATCH v3 14/24 sched/rt: Allow zeroing the runtime of the root control group

In reply to Yuri Andriaccio

In HCBS, the root control group follows the already existing rules for rt-task scheduling. As such, it does not make use of the deadline servers to account for runtime, or any other HCBS specific code and features.

While the runtime of SCHED_DEADLINE tasks depends on the global bandwidth reserved for rt_tasks, the runtime of SCHED_FIFO/SCHED_RR tasks is limited by the activation of fair-servers (as the RT_THROTTLING mechanism has been removed in favour of them), thus their maximum bandwidth depends solely on the fair-server settings (which are thightly related to the global bandwdith reserved for rt-tasks) and the amount of SCHED_OTHER workload to run (recall that if no SCHED_OTHER tasks are running, the FIFO/RR tasks may fully utilize the CPU).

The values of runtime and period in the root cgroup's cpu controller do not affect, by design of HCBS, the fair-server settings and similar (consequently they do not affect the scheduling of FIFO/RR tasks in the root cgroup), buy they are just used to reserve a portion of the SCHED_DEADLINE bandwidth to the scheduling of rt-cgroups. These values only affect child cgroups, their deadline servers and their assigned FIFO/RR tasks.

---
 kernel/sched/rt.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
kernel/sched/rt.c +4 -7
index 7c7622303e8..94693093b70 100644
@@ -2123,13 +2123,6 @@ static int tg_set_rt_bandwidth(struct task_group *tg,
 	static DEFINE_MUTEX(rt_constraints_mutex);
 	int i, err = 0;
 
-	/*
-	 * Disallowing the root group RT runtime is BAD, it would disallow the
-	 * kernel creating (and or operating) RT threads.
-	 */
-	if (tg == &root_task_group && rt_runtime == 0)
-		return -EINVAL;
-
 	/* No period doesn't make any sense. */
 	if (rt_period == 0)
 		return -EINVAL;
@@ -2216,6 +2209,10 @@ static int sched_rt_global_constraints(void)
 
 int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk)
 {
+	/* Allow executing in the root cgroup regardless of allowed bandwidth */
+	if (tg == &root_task_group)
+		return 1;
+
 	/* Don't accept real-time tasks when there is no way for them to run */
 	if (rt_group_sched_enabled() && tg->dl_bandwidth.dl_runtime == 0)
 		return 0;
Signature · 1 line
2.51.0
maintainer
#28 lore raw

Re: [RFC PATCH v3 14/24] sched/rt: Allow zeroing the runtime of the root control group

Hello,

Quoted 9 lines · Yuri Andriaccio

In HCBS, the root control group follows the already existing rules for rt-task scheduling. As such, it does not make use of the deadline servers to account for runtime, or any other HCBS specific code and features.

While the runtime of SCHED_DEADLINE tasks depends on the global bandwidth reserved for rt_tasks, the runtime of SCHED_FIFO/SCHED_RR tasks is limited by the activation of fair-servers (as the RT_THROTTLING mechanism has been removed in favour of them), thus their maximum bandwidth depends solely on the fair-server settings (which are thightly related to the global bandwdith

                                                                    ^
							      bandwidth
Quoted 7 lines

reserved for rt-tasks) and the amount of SCHED_OTHER workload to run (recall that if no SCHED_OTHER tasks are running, the FIFO/RR tasks may fully utilize the CPU).

The values of runtime and period in the root cgroup's cpu controller do not affect, by design of HCBS, the fair-server settings and similar (consequently they do not affect the scheduling of FIFO/RR tasks in the root cgroup), buy they

                                                                           ^
									 but
Quoted 6 lines

are just used to reserve a portion of the SCHED_DEADLINE bandwidth to the scheduling of rt-cgroups. These values only affect child cgroups, their deadline servers and their assigned FIFO/RR tasks.

---

Thanks,
Juri

#29 lore raw

RFC PATCH v3 15/24 sched/rt: Remove old RT_GROUP_SCHED data structures

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Completely remove the old RT_GROUP_SCHED's functions and data structures.

---
 include/linux/sched.h |  4 ----
 kernel/sched/rt.c     |  1 -
 kernel/sched/sched.h  | 26 --------------------------
 3 files changed, 31 deletions(-)
include/linux/sched.h +0 -4
index 5022ff99c5d..379923eba59 100644
@@ -624,13 +624,9 @@ struct sched_rt_entity {
 	unsigned short			on_rq;
 	unsigned short			on_list;
 
-	struct sched_rt_entity		*back;
 #ifdef CONFIG_RT_GROUP_SCHED
-	struct sched_rt_entity		*parent;
 	/* rq on which this entity is (to be) queued: */
 	struct rt_rq			*rt_rq;
-	/* rq "owned" by this entity/group: */
-	struct rt_rq			*my_q;
 #endif
 } __randomize_layout;
 
kernel/sched/rt.c +0 -1
index 94693093b70..1cdc699bbb7 100644
@@ -1,4 +1,3 @@
-#pragma GCC diagnostic ignored "-Wunused-function"
 // SPDX-License-Identifier: GPL-2.0
 /*
  * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
kernel/sched/sched.h +0 -26
index 97e1e779df9..fddb171145e 100644
@@ -310,15 +310,6 @@ struct rt_prio_array {
 	struct list_head queue[MAX_RT_PRIO];
 };
 
-struct rt_bandwidth {
-	/* nests inside the rq lock: */
-	raw_spinlock_t		rt_runtime_lock;
-	ktime_t			rt_period;
-	u64			rt_runtime;
-	struct hrtimer		rt_period_timer;
-	unsigned int		rt_period_active;
-};
-
 struct dl_bandwidth {
 	raw_spinlock_t          dl_runtime_lock;
 	u64                     dl_runtime;
@@ -485,7 +476,6 @@ struct task_group {
 	struct sched_dl_entity	**dl_se;
 	struct rt_rq		**rt_rq;
 
-	struct rt_bandwidth	rt_bandwidth;
 	struct dl_bandwidth	dl_bandwidth;
 #endif
 
@@ -801,11 +791,6 @@ struct scx_rq {
 };
 #endif /* CONFIG_SCHED_CLASS_EXT */
 
-static inline int rt_bandwidth_enabled(void)
-{
-	return 0;
-}
-
 /* RT IPI pull logic requires IRQ_WORK */
 #if defined(CONFIG_IRQ_WORK) && defined(CONFIG_SMP)
 # define HAVE_RT_PUSH_IPI
@@ -823,17 +808,6 @@ struct rt_rq {
 	bool			overloaded;
 	struct plist_head	pushable_tasks;
 
-	int			rt_queued;
-
-#ifdef CONFIG_RT_GROUP_SCHED
-	int			rt_throttled;
-	u64			rt_time; /* consumed RT time, goes up in update_curr_rt */
-	u64			rt_runtime; /* allotted RT time, "slice" from rt_bandwidth, RT sharing/balancing */
-	/* Nests inside the rq lock: */
-	raw_spinlock_t		rt_runtime_lock;
-
-	unsigned int		rt_nr_boosted;
-#endif
 #ifdef CONFIG_CGROUP_SCHED
 	struct task_group	*tg; /* this tg has "this" rt_rq on given CPU for runnable entities */
 #endif
Signature · 1 line
2.51.0
#30 lore raw

RFC PATCH v3 16/24 sched/core: Cgroup v2 support

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Make rt_runtime_us and rt_period_us virtual files accessible also to the cgroup v2 controller, effectively enabling the RT_GROUP_SCHED mechanism to cgroups v2.

---
 kernel/sched/core.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
kernel/sched/core.c +12 -0
index 1217f714dd2..f6e9a4b22aa 100644
@@ -10323,6 +10323,18 @@ static struct cftype cpu_files[] = {
 		.write = cpu_uclamp_max_write,
 	},
 #endif /* CONFIG_UCLAMP_TASK_GROUP */
+#ifdef CONFIG_RT_GROUP_SCHED
+	{
+		.name = "rt_runtime_us",
+		.read_s64 = cpu_rt_runtime_read,
+		.write_s64 = cpu_rt_runtime_write,
+	},
+	{
+		.name = "rt_period_us",
+		.read_u64 = cpu_rt_period_read_uint,
+		.write_u64 = cpu_rt_period_write_uint,
+	},
+#endif /* CONFIG_RT_GROUP_SCHED */
 	{ }	/* terminate */
 };
 
Signature · 1 line
2.51.0
#31 lore raw

RFC PATCH v3 17/24 sched/rt: Remove support for cgroups-v1

In reply to Yuri Andriaccio

Disable control files for cgroups-v1, and allow only cgroups-v2. This should simplify maintaining the code, also because cgroups-v1 are deprecated.

Set the default rt-cgroups runtime to zero, otherwise a cgroup-v1 kernel will not be able to start SCHED_DEADLINE tasks. The bandwidth for rt-cgroups must then be manually assigned after the kernel boots.

---
 kernel/sched/core.c | 20 +-------------------
 1 file changed, 1 insertion(+), 19 deletions(-)
kernel/sched/core.c +1 -19
index f6e9a4b22aa..6f516cdc7bb 100644
@@ -8723,7 +8723,7 @@ void __init sched_init(void)
 
 #ifdef CONFIG_RT_GROUP_SCHED
 	init_dl_bandwidth(&root_task_group.dl_bandwidth,
-			global_rt_period(), global_rt_runtime());
+			global_rt_period(), 0);
 #endif /* CONFIG_RT_GROUP_SCHED */
 
 #ifdef CONFIG_CGROUP_SCHED
@@ -10062,20 +10062,6 @@ static struct cftype cpu_legacy_files[] = {
 };
 
 #ifdef CONFIG_RT_GROUP_SCHED
-static struct cftype rt_group_files[] = {
-	{
-		.name = "rt_runtime_us",
-		.read_s64 = cpu_rt_runtime_read,
-		.write_s64 = cpu_rt_runtime_write,
-	},
-	{
-		.name = "rt_period_us",
-		.read_u64 = cpu_rt_period_read_uint,
-		.write_u64 = cpu_rt_period_write_uint,
-	},
-	{ }	/* Terminate */
-};
-
 # ifdef CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED
 DEFINE_STATIC_KEY_FALSE(rt_group_sched);
 # else
@@ -10101,10 +10087,6 @@ __setup("rt_group_sched=", setup_rt_group_sched);
 
 static int __init cpu_rt_group_init(void)
 {
-	if (!rt_group_sched_enabled())
-		return 0;
-
-	WARN_ON(cgroup_add_legacy_cftypes(&cpu_cgrp_subsys, rt_group_files));
 	return 0;
 }
 subsys_initcall(cpu_rt_group_init);
Signature · 1 line
2.51.0
maintainer
#32 lore raw

Re: [RFC PATCH v3 17/24] sched/rt: Remove support for cgroups-v1

Hello,

Quoted 9 lines · Yuri Andriaccio

Disable control files for cgroups-v1, and allow only cgroups-v2. This should simplify maintaining the code, also because cgroups-v1 are deprecated.

Set the default rt-cgroups runtime to zero, otherwise a cgroup-v1 kernel will not be able to start SCHED_DEADLINE tasks. The bandwidth for rt-cgroups must then be manually assigned after the kernel boots.

---

The change looks good, but I think this series is currently missing a substantial Docs update. admin-guide cgroup section and scheduler needs some changes, I believe. Please consider doing so for next revision.

Thanks,
Juri

#33 lore raw

RFC PATCH v3 18/24 sched/deadline: Allow deeper hierarchies of RT cgroups

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Allow creation of cgroup hierachies with depth greater than two. Add check to prevent attaching tasks to a child cgroup of an active cgroup (i.e. with a running FIFO/RR task).
Add check to prevent attaching tasks to cgroups which have children with non-zero runtime.
Update rt-cgroups allocated bandwidth accounting for nested cgroup hierachies.

---
 kernel/sched/core.c     |  6 -----
 kernel/sched/deadline.c | 51 +++++++++++++++++++++++++++++++++++++----
 kernel/sched/rt.c       | 16 ++++++++++---
 kernel/sched/sched.h    |  3 ++-
 4 files changed, 62 insertions(+), 14 deletions(-)
kernel/sched/core.c +0 -6
index 6f516cdc7bb..d1d7215c4a2 100644
@@ -9281,12 +9281,6 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 		return &root_task_group.css;
 	}
 
-	/* Do not allow cpu_cgroup hierachies with depth greater than 2. */
-#ifdef CONFIG_RT_GROUP_SCHED
-	if (parent != &root_task_group)
-		return ERR_PTR(-EINVAL);
-#endif
-
 	tg = sched_create_group(parent);
 	if (IS_ERR(tg))
 		return ERR_PTR(-ENOMEM);
kernel/sched/deadline.c +47 -4

72 lines not on this page · raw message

kernel/sched/rt.c +13 -3
index 1cdc699bbb7..17ad91261cb 100644
@@ -114,7 +114,8 @@ void free_rt_sched_group(struct task_group *tg)
 		 * Fix this issue by changing the group runtime
 		 * to 0 immediately before freeing it.
 		 */
-		dl_init_tg(tg->dl_se[i], 0, tg->dl_se[i]->dl_period);
+		if (tg->dl_se[i]->dl_runtime)
+			dl_init_tg(tg, i, 0, tg->dl_se[i]->dl_period);
 
 		raw_spin_rq_lock_irqsave(cpu_rq(i), flags);
 		BUG_ON(tg->rt_rq[i]->rt_nr_running);
@@ -2122,6 +2123,14 @@ static int tg_set_rt_bandwidth(struct task_group *tg,
 	static DEFINE_MUTEX(rt_constraints_mutex);
 	int i, err = 0;
 
+	/*
+	 * Do not allow to set a RT runtime > 0 if the parent has RT tasks
+	 * (and is not the root group)
+	 */
+	if (rt_runtime && (tg != &root_task_group) && (tg->parent != &root_task_group) && tg_has_rt_tasks(tg->parent)) {
+		return -EINVAL;
+	}
+
 	/* No period doesn't make any sense. */
 	if (rt_period == 0)
 		return -EINVAL;
@@ -2145,7 +2154,7 @@ static int tg_set_rt_bandwidth(struct task_group *tg,
 		return 0;
 
 	for_each_possible_cpu(i) {
-		dl_init_tg(tg->dl_se[i], rt_runtime, rt_period);
+		dl_init_tg(tg, i, rt_runtime, rt_period);
 	}
 
 	return 0;
@@ -2216,7 +2225,8 @@ int sched_rt_can_attach(struct task_group *tg, struct task_struct *tsk)
 	if (rt_group_sched_enabled() && tg->dl_bandwidth.dl_runtime == 0)
 		return 0;
 
-	return 1;
+	/* tasks can be attached only if the taskgroup has no active children. */
+	return (int)is_active_sched_group(tg);
 }
 
 #else /* !CONFIG_RT_GROUP_SCHED: */
kernel/sched/sched.h +2 -1
index fddb171145e..55631d93e02 100644
@@ -386,7 +386,8 @@ extern void dl_server_init(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq,
 		    dl_server_pick_f pick_task);
 extern void sched_init_dl_servers(void);
 extern int dl_check_tg(unsigned long total);
-extern void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period);
+extern void dl_init_tg(struct task_group *tg, int cpu, u64 rt_runtime, u64 rt_period);
+extern bool is_active_sched_group(struct task_group *tg);
 
 extern void dl_server_update_idle_time(struct rq *rq,
 		    struct task_struct *p);
Signature · 1 line
2.51.0
maintainer
#34 lore raw

Re: [RFC PATCH v3 18/24] sched/deadline: Allow deeper hierarchies of RT cgroups

Hello,

Quoted 41 lines · Yuri Andriaccio

From: luca abeni <[email protected]>

Allow creation of cgroup hierachies with depth greater than two. Add check to prevent attaching tasks to a child cgroup of an active cgroup (i.e. with a running FIFO/RR task).
Add check to prevent attaching tasks to cgroups which have children with non-zero runtime.
Update rt-cgroups allocated bandwidth accounting for nested cgroup hierachies.

---
 kernel/sched/core.c     |  6 -----
 kernel/sched/deadline.c | 51 +++++++++++++++++++++++++++++++++++++----
 kernel/sched/rt.c       | 16 ++++++++++---
 kernel/sched/sched.h    |  3 ++-
 4 files changed, 62 insertions(+), 14 deletions(-)
kernel/sched/core.c +0 -6
index 6f516cdc7bb..d1d7215c4a2 100644
@@ -9281,12 +9281,6 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 		return &root_task_group.css;
 	}
 
-	/* Do not allow cpu_cgroup hierachies with depth greater than 2. */
-#ifdef CONFIG_RT_GROUP_SCHED
-	if (parent != &root_task_group)
-		return ERR_PTR(-EINVAL);
-#endif
-
 	tg = sched_create_group(parent);
 	if (IS_ERR(tg))
 		return ERR_PTR(-ENOMEM);
kernel/sched/deadline.c +0 -0
index 5d93b3ca030..abe11985c41 100644

kernel/sched/deadline.c int dl_check_tg(unsigned long total) · line 391

Quoted 5 lines
kernel/sched/deadline.c +1 -1
@@ -388,11 +388,42 @@ int dl_check_tg(unsigned long total)
 	return 1;
 }
 
-void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period)
+bool is_active_sched_group(struct task_group *tg)

I wonder if the function name could be misleading, as this checks runtime and not if there are tasks in the group.

Quoted 9 lines
kernel/sched/deadline.c +8 -0
 {
+	struct task_group *child;
+	bool is_active = 1;
+
+	// if there are no children, this is a leaf group, thus it is active
+	list_for_each_entry_rcu(child, &tg->children, siblings) {
+		if (child->dl_bandwidth.dl_runtime > 0) {
+			is_active = 0;
+		}

kernel/sched/deadline.c int dl_check_tg(unsigned long total) · line 410

Quoted 10 lines
kernel/sched/deadline.c +10 -0
+	}
+	return is_active;
+}
+
+static inline bool sched_group_has_active_siblings(struct task_group *tg)
+{
+	struct task_group *child;
+	bool has_active_siblings = 0;
+
+	// if there are no children, this is a leaf group, thus it is active

Copy-pasta from above? :) Also not the correct comment style.

Quoted 34 lines
kernel/sched/deadline.c +18 -3
+	list_for_each_entry_rcu(child, &tg->parent->children, siblings) {
+		if (child != tg && child->dl_bandwidth.dl_runtime > 0) {
+			has_active_siblings = 1;
+		}
+	}
+	return has_active_siblings;
+}
+
+void dl_init_tg(struct task_group *tg, int cpu, u64 rt_runtime, u64 rt_period)
+{
+	struct sched_dl_entity *dl_se = tg->dl_se[cpu];
 	struct rq *rq = container_of(dl_se->dl_rq, struct rq, dl);
-	int is_active;
-	u64 new_bw;
+	int is_active, is_active_group;
+	u64 old_runtime, new_bw;
+
+	is_active_group = is_active_sched_group(tg);
 
 	raw_spin_rq_lock_irq(rq);
 	is_active = dl_se->my_q->rt.rt_nr_running > 0;
@@ -400,8 +431,10 @@ void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period)
 	update_rq_clock(rq);
 	dl_server_stop(dl_se);
 
+	old_runtime = dl_se->dl_runtime;
 	new_bw = to_ratio(dl_se->dl_period, dl_se->dl_runtime);
-	dl_rq_change_utilization(rq, dl_se, new_bw);
+	if (is_active_group)
+		dl_rq_change_utilization(rq, dl_se, new_bw);
 
 	dl_se->dl_runtime  = rt_runtime;
 	dl_se->dl_deadline = rt_period;

kernel/sched/deadline.c void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period) · line 449

Quoted 4 lines
kernel/sched/deadline.c +1 -0
@@ -413,6 +446,16 @@ void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period)
 	dl_se->dl_bw = new_bw;
 	dl_se->dl_density = new_bw;
 
+	// add/remove the parent's bw

Comment style is not correct. Also the comment itself is not very much informative. What about something like (IIUC)

/*
* Handle parent bandwidth accounting when child runtime changes:
* - Disabling the last active child: parent becomes a leaf group,
* so add the parent's bandwidth back to active accounting
* - Enabling the first child: parent becomes a non-leaf group,
* so remove the parent's bandwidth from active accounting
* Only leaf groups (those without active children) should have
* non-zero bandwidth.
*/

kernel/sched/deadline.c void dl_init_tg(struct sched_dl_entity *dl_se, u64 rt_runtime, u64 rt_period) · line 458

Quoted 9 lines
kernel/sched/deadline.c +9 -0
+	if (tg->parent && tg->parent != &root_task_group)
+	{
+		if (rt_runtime == 0 && old_runtime != 0 && !sched_group_has_active_siblings(tg)) {
+			__add_rq_bw(tg->parent->dl_se[cpu]->dl_bw, dl_se->dl_rq);
+		} else if (rt_runtime != 0 && old_runtime == 0 && !sched_group_has_active_siblings(tg)) {
+			__sub_rq_bw(tg->parent->dl_se[cpu]->dl_bw, dl_se->dl_rq);
+		}
+	}
+

Thanks,
Juri

#35 lore raw

RFC PATCH v3 19/24 sched/rt: Add rt-cgroup migration

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

When the runtime is exhausted in a RT CGroup, the scheduler checks for another non-throttled runqueue and, if available, migrates the tasks.

The bandwidth (runtime/period) chosen for a certain cgroup is replicated on every core of the system, therefore, in an SMP system with M cores, the total available bandwidth is the given runtime/period multiplied by M.

---
 kernel/sched/rt.c    | 495 +++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h |  10 +
 2 files changed, 505 insertions(+)
kernel/sched/rt.c +495 -0
index 17ad91261cb..1fbcac14fc9 100644
@@ -1,3 +1,4 @@
+#pragma GCC diagnostic ignored "-Wunused-function"
 // SPDX-License-Identifier: GPL-2.0
 /*
  * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -84,6 +85,8 @@ void init_rt_rq(struct rt_rq *rt_rq)
 	plist_head_init(&rt_rq->pushable_tasks);
 }
 
+static void group_pull_rt_task(struct rt_rq *this_rt_rq);
+
 #ifdef CONFIG_RT_GROUP_SCHED
 
 void unregister_rt_sched_group(struct task_group *tg)
@@ -136,6 +139,7 @@ void init_tg_rt_entry(struct task_group *tg, struct rq *served_rq,
 		struct sched_dl_entity *dl_se, int cpu,
 		struct sched_dl_entity *parent)
 {
+	served_rq->cpu = cpu;
 	served_rq->rt.highest_prio.curr = MAX_RT_PRIO-1;
 	served_rq->rt.rq = cpu_rq(cpu);
 	served_rq->rt.tg = tg;
@@ -299,6 +303,53 @@ static inline void rt_queue_pull_task(struct rt_rq * rt_rq)
 	queue_balance_callback(rq, &per_cpu(rt_pull_head, rq->cpu), pull_rt_task);
 }
 
+#ifdef CONFIG_RT_GROUP_SCHED
+static DEFINE_PER_CPU(struct balance_callback, rt_group_push_head);
+static DEFINE_PER_CPU(struct balance_callback, rt_group_pull_head);
+static void group_push_rt_tasks_callback(struct rq *);
+static void group_pull_rt_task_callback(struct rq *);
+
+static void rt_queue_push_from_group(struct rt_rq *rt_rq)
+{
+	struct rq *rq = container_of(rt_rq, struct rq, rt);
+	struct rq *global_rq = cpu_rq(rq->cpu);
+
+	BUG_ON(rt_rq == NULL);
+	BUG_ON(rt_rq->rq != global_rq);
+
+	if (global_rq->rq_to_push_from)
+		return;
+
+	if (!has_pushable_tasks(rt_rq))
+		return;
+
+	global_rq->rq_to_push_from = rq;
+	queue_balance_callback(global_rq, &per_cpu(rt_group_push_head, global_rq->cpu),
+			       group_push_rt_tasks_callback);
+}
+
+static void rt_queue_pull_to_group(struct rt_rq *rt_rq)
+{
+	struct rq *rq = container_of(rt_rq, struct rq, rt);
+	struct rq *global_rq = cpu_rq(rq->cpu);
+	struct sched_dl_entity *dl_se = dl_group_of(rt_rq);
+
+	BUG_ON(rt_rq == NULL);
+	BUG_ON(!is_dl_group(rt_rq));
+	BUG_ON(rt_rq->rq != global_rq);
+
+	if (dl_se->dl_throttled || global_rq->rq_to_pull_to)
+		return;
+
+	global_rq->rq_to_pull_to = rq;
+	queue_balance_callback(global_rq, &per_cpu(rt_group_pull_head, global_rq->cpu),
+			       group_pull_rt_task_callback);
+}
+#else
+static inline void rt_queue_push_from_group(struct rt_rq *rt_rq) {};
+static inline void rt_queue_pull_to_group(struct rt_rq *rt_rq) {};
+#endif
+
 static void enqueue_pushable_task(struct rt_rq *rt_rq, struct task_struct *p)
 {
 	plist_del(&p->pushable_tasks, &rt_rq->pushable_tasks);
@@ -1303,6 +1354,8 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
  */
 static int push_rt_task(struct rq *rq, bool pull)
 {
+	BUG_ON(is_dl_group(&rq->rt));
+
 	struct task_struct *next_task;
 	struct rq *lowest_rq;
 	int ret = 0;
@@ -1599,6 +1652,8 @@ void rto_push_irq_work_func(struct irq_work *work)
 
 static void pull_rt_task(struct rq *this_rq)
 {
+	BUG_ON(is_dl_group(&this_rq->rt));
+
 	int this_cpu = this_rq->cpu, cpu;
 	bool resched = false;
 	struct task_struct *p, *push_task;
@@ -1708,6 +1763,446 @@ static void pull_rt_task(struct rq *this_rq)
 		resched_curr(this_rq);
 }
 
+#ifdef CONFIG_RT_GROUP_SCHED
+/*
+ * Find the lowest priority runqueue among the runqueues of the same
+ * task group. Unlike find_lowest_rt(), this does not mean that the
+ * lowest priority cpu is running tasks from this runqueue.
+ */
+static int group_find_lowest_rt_rq(struct task_struct *task, struct rt_rq* task_rt_rq)
+{
+	struct sched_domain *sd;
+	struct cpumask mask, *lowest_mask = &mask;
+	struct sched_dl_entity *dl_se;
+	struct rt_rq *rt_rq;
+	int prio, lowest_prio;
+	int cpu, this_cpu = smp_processor_id();
+
+	BUG_ON(task->sched_task_group != task_rt_rq->tg);
+
+	if (task->nr_cpus_allowed == 1)
+		return -1; /* No other targets possible */
+
+	lowest_prio = task->prio - 1;
+	cpumask_clear(lowest_mask);
+	for_each_cpu_and(cpu, cpu_online_mask, task->cpus_ptr) {
+		dl_se = task_rt_rq->tg->dl_se[cpu];
+		rt_rq = &dl_se->my_q->rt;
+		prio = rt_rq->highest_prio.curr;
+
+		/*
+		 * If we're on asym system ensure we consider the different capacities
+		 * of the CPUs when searching for the lowest_mask.
+		 */
+		if (dl_se->dl_throttled || !rt_task_fits_capacity(task, cpu))
+			continue;
+
+		if (prio >= lowest_prio) {
+			if (prio > lowest_prio) {
+				cpumask_clear(lowest_mask);
+				lowest_prio = prio;
+			}
+
+			cpumask_set_cpu(cpu, lowest_mask);
+		}
+	}
+
+	if (cpumask_empty(lowest_mask))
+		return -1;
+
+	/*
+	 * At this point we have built a mask of CPUs representing the
+	 * lowest priority tasks in the system.  Now we want to elect
+	 * the best one based on our affinity and topology.
+	 *
+	 * We prioritize the last CPU that the task executed on since
+	 * it is most likely cache-hot in that location.
+	 */
+	cpu = task_cpu(task);
+	if (cpumask_test_cpu(cpu, lowest_mask))
+		return cpu;
+
+	/*
+	 * Otherwise, we consult the sched_domains span maps to figure
+	 * out which CPU is logically closest to our hot cache data.
+	 */
+	if (!cpumask_test_cpu(this_cpu, lowest_mask))
+		this_cpu = -1; /* Skip this_cpu opt if not among lowest */
+
+	rcu_read_lock();
+	for_each_domain(cpu, sd) {
+		if (sd->flags & SD_WAKE_AFFINE) {
+			int best_cpu;
+
+			/*
+			 * "this_cpu" is cheaper to preempt than a
+			 * remote processor.
+			 */
+			if (this_cpu != -1 &&
+			    cpumask_test_cpu(this_cpu, sched_domain_span(sd))) {
+				rcu_read_unlock();
+				return this_cpu;
+			}
+
+			best_cpu = cpumask_any_and_distribute(lowest_mask,
+							      sched_domain_span(sd));
+			if (best_cpu < nr_cpu_ids) {
+				rcu_read_unlock();
+				return best_cpu;
+			}
+		}
+	}
+	rcu_read_unlock();
+
+	/*
+	 * And finally, if there were no matches within the domains
+	 * just give the caller *something* to work with from the compatible
+	 * locations.
+	 */
+	if (this_cpu != -1)
+		return this_cpu;
+
+	cpu = cpumask_any_distribute(lowest_mask);
+	if (cpu < nr_cpu_ids)
+		return cpu;
+
+	return -1;
+}
+
+/*
+ * Find and lock the lowest priority runqueue among the runqueues
+ * of the same task group. Unlike find_lock_lowest_rt(), this does not
+ * mean that the lowest priority cpu is running tasks from this runqueue.
+ */
+static struct rt_rq* group_find_lock_lowest_rt_rq(struct task_struct *task, struct rt_rq *rt_rq)
+{
+	struct rq *rq = rq_of_rt_rq(rt_rq);
+	struct rq *lowest_rq;
+	struct rt_rq *lowest_rt_rq = NULL;
+	struct sched_dl_entity *lowest_dl_se;
+	int tries, cpu;
+
+	BUG_ON(task->sched_task_group != rt_rq->tg);
+
+	for (tries = 0; tries < RT_MAX_TRIES; tries++) {
+		cpu = group_find_lowest_rt_rq(task, rt_rq);
+
+		if ((cpu == -1) || (cpu == rq->cpu))
+			break;
+
+		lowest_dl_se = rt_rq->tg->dl_se[cpu];
+		lowest_rt_rq = &lowest_dl_se->my_q->rt;
+		lowest_rq = cpu_rq(cpu);
+
+		if (lowest_rt_rq->highest_prio.curr <= task->prio) {
+			/*
+			 * Target rq has tasks of equal or higher priority,
+			 * retrying does not release any lock and is unlikely
+			 * to yield a different result.
+			 */
+			lowest_rt_rq = NULL;
+			break;
+		}
+
+		/* if the prio of this runqueue changed, try again */
+		if (double_lock_balance(rq, lowest_rq)) {
+			/*
+			 * We had to unlock the run queue. In
+			 * the mean time, task could have
+			 * migrated already or had its affinity changed.
+			 * Also make sure that it wasn't scheduled on its rq.
+			 * It is possible the task was scheduled, set
+			 * "migrate_disabled" and then got preempted, so we must
+			 * check the task migration disable flag here too.
+			 */
+			if (unlikely(is_migration_disabled(task) ||
+				     lowest_dl_se->dl_throttled ||
+				     !cpumask_test_cpu(lowest_rq->cpu, &task->cpus_mask) ||
+				     task != pick_next_pushable_task(rt_rq))) {
+
+				double_unlock_balance(rq, lowest_rq);
+				lowest_rt_rq = NULL;
+				break;
+			}
+		}
+
+		/* If this rq is still suitable use it. */
+		if (lowest_rt_rq->highest_prio.curr > task->prio)
+			break;
+
+		/* try again */
+		double_unlock_balance(rq, lowest_rq);
+		lowest_rt_rq = NULL;
+	}
+
+	return lowest_rt_rq;
+}
+
+static int group_push_rt_task(struct rt_rq *rt_rq, bool pull)
+{
+	BUG_ON(!is_dl_group(rt_rq));
+
+	struct rq *rq = rq_of_rt_rq(rt_rq);
+	struct task_struct *next_task;
+	struct rq *lowest_rq;
+	struct rt_rq *lowest_rt_rq;
+	int ret = 0;
+
+	if (!rt_rq->overloaded)
+		return 0;
+
+	next_task = pick_next_pushable_task(rt_rq);
+	if (!next_task)
+		return 0;
+
+retry:
+	if (is_migration_disabled(next_task)) {
+		struct task_struct *push_task = NULL;
+		int cpu;
+
+		if (!pull || rq->push_busy)
+			return 0;
+
+		/*
+		 * If the current task does not belong to the same task group
+		 * we cannot push it away.
+		 */
+		if (rq->curr->sched_task_group != rt_rq->tg)
+			return 0;
+
+		/*
+		 * Invoking group_find_lowest_rt_rq() on anything but an RT task doesn't
+		 * make sense. Per the above priority check, curr has to
+		 * be of higher priority than next_task, so no need to
+		 * reschedule when bailing out.
+		 *
+		 * Note that the stoppers are masqueraded as SCHED_FIFO
+		 * (cf. sched_set_stop_task()), so we can't rely on rt_task().
+		 */
+		if (rq->curr->sched_class != &rt_sched_class)
+			return 0;
+
+		cpu = group_find_lowest_rt_rq(rq->curr, rt_rq);
+		if (cpu == -1 || cpu == rq->cpu)
+			return 0;
+
+		/*
+		 * Given we found a CPU with lower priority than @next_task,
+		 * therefore it should be running. However we cannot migrate it
+		 * to this other CPU, instead attempt to push the current
+		 * running task on this CPU away.
+		 */
+		push_task = get_push_task(rq);
+		if (push_task) {
+			preempt_disable();
+			raw_spin_rq_unlock(rq);
+			stop_one_cpu_nowait(rq->cpu, push_cpu_stop,
+					    push_task, &rq->push_work);
+			preempt_enable();
+			raw_spin_rq_lock(rq);
+		}
+
+		return 0;
+	}
+
+	if (WARN_ON(next_task == rq->curr))
+		return 0;
+
+	/* We might release rq lock */
+	get_task_struct(next_task);
+
+	/* group_find_lock_lowest_rq locks the rq if found */
+	lowest_rt_rq = group_find_lock_lowest_rt_rq(next_task, rt_rq);
+	if (!lowest_rt_rq) {
+		struct task_struct *task;
+		/*
+		 * group_find_lock_lowest_rt_rq releases rq->lock
+		 * so it is possible that next_task has migrated.
+		 *
+		 * We need to make sure that the task is still on the same
+		 * run-queue and is also still the next task eligible for
+		 * pushing.
+		 */
+		task = pick_next_pushable_task(rt_rq);
+		if (task == next_task) {
+			/*
+			 * The task hasn't migrated, and is still the next
+			 * eligible task, but we failed to find a run-queue
+			 * to push it to.  Do not retry in this case, since
+			 * other CPUs will pull from us when ready.
+			 */
+			goto out;
+		}
+
+		if (!task)
+			/* No more tasks, just exit */
+			goto out;
+
+		/*
+		 * Something has shifted, try again.
+		 */
+		put_task_struct(next_task);
+		next_task = task;
+		goto retry;
+	}
+
+	lowest_rq = rq_of_rt_rq(lowest_rt_rq);
+
+	move_queued_task_locked(rq, lowest_rq, next_task);
+	resched_curr(lowest_rq);
+	ret = 1;
+
+	double_unlock_balance(rq, lowest_rq);
+out:
+	put_task_struct(next_task);
+
+	return ret;
+}
+
+static void group_pull_rt_task(struct rt_rq *this_rt_rq)
+{
+	BUG_ON(!is_dl_group(this_rt_rq));
+
+	struct rq *this_rq = rq_of_rt_rq(this_rt_rq);
+	int this_cpu = this_rq->cpu, cpu;
+	bool resched = false;
+	struct task_struct *p, *push_task = NULL;
+	struct rt_rq *src_rt_rq;
+	struct rq *src_rq;
+	struct sched_dl_entity *src_dl_se;
+
+	for_each_online_cpu(cpu) {
+		if (this_cpu == cpu)
+			continue;
+
+		src_dl_se = this_rt_rq->tg->dl_se[cpu];
+		src_rt_rq = &src_dl_se->my_q->rt;
+
+		if (src_rt_rq->rt_nr_running <= 1 && !src_dl_se->dl_throttled)
+			continue;
+
+		src_rq = rq_of_rt_rq(src_rt_rq);
+
+		/*
+		 * Don't bother taking the src_rq->lock if the next highest
+		 * task is known to be lower-priority than our current task.
+		 * This may look racy, but if this value is about to go
+		 * logically higher, the src_rq will push this task away.
+		 * And if its going logically lower, we do not care
+		 */
+		if (src_rt_rq->highest_prio.next >=
+		    this_rt_rq->highest_prio.curr)
+			continue;
+
+		/*
+		 * We can potentially drop this_rq's lock in
+		 * double_lock_balance, and another CPU could
+		 * alter this_rq
+		 */
+		push_task = NULL;
+		double_lock_balance(this_rq, src_rq);
+
+		/*
+		 * We can pull only a task, which is pushable
+		 * on its rq, and no others.
+		 */
+		p = pick_highest_pushable_task(src_rt_rq, this_cpu);
+
+		/*
+		 * Do we have an RT task that preempts
+		 * the to-be-scheduled task?
+		 */
+		if (p && (p->prio < this_rt_rq->highest_prio.curr)) {
+			WARN_ON(p == src_rq->curr);
+			WARN_ON(!task_on_rq_queued(p));
+
+			/*
+			 * There's a chance that p is higher in priority
+			 * than what's currently running on its CPU.
+			 * This is just that p is waking up and hasn't
+			 * had a chance to schedule. We only pull
+			 * p if it is lower in priority than the
+			 * current task on the run queue
+			 */
+			if (src_rq->curr->sched_task_group == this_rt_rq->tg &&
+			    p->prio < src_rq->curr->prio)
+				goto skip;
+
+			if (is_migration_disabled(p)) {
+				/*
+				 * If the current task does not belong to the same task group
+				 * we cannot push it away.
+				 */
+				if (src_rq->curr->sched_task_group != this_rt_rq->tg)
+					goto skip;
+
+				push_task = get_push_task(src_rq);
+			} else {
+				move_queued_task_locked(src_rq, this_rq, p);
+				resched = true;
+			}
+			/*
+			 * We continue with the search, just in
+			 * case there's an even higher prio task
+			 * in another runqueue. (low likelihood
+			 * but possible)
+			 */
+		}
+skip:
+		double_unlock_balance(this_rq, src_rq);
+
+		if (push_task) {
+			preempt_disable();
+			raw_spin_rq_unlock(this_rq);
+			stop_one_cpu_nowait(src_rq->cpu, push_cpu_stop,
+					    push_task, &src_rq->push_work);
+			preempt_enable();
+			raw_spin_rq_lock(this_rq);
+		}
+	}
+
+	if (resched)
+		resched_curr(this_rq);
+}
+
+static void group_push_rt_tasks(struct rt_rq *rt_rq)
+{
+	while (group_push_rt_task(rt_rq, false))
+		;
+}
+
+static void group_push_rt_tasks_callback(struct rq *global_rq)
+{
+	struct rt_rq *rt_rq = &global_rq->rq_to_push_from->rt;
+
+	BUG_ON(global_rq->rq_to_push_from == NULL);
+	BUG_ON(rt_rq->rq != global_rq);
+
+	if ((rt_rq->rt_nr_running > 1) ||
+	    (dl_group_of(rt_rq)->dl_throttled == 1)) {
+
+		while (group_push_rt_task(rt_rq, false))
+			;
+	}
+
+	global_rq->rq_to_push_from = NULL;
+}
+
+static void group_pull_rt_task_callback(struct rq *global_rq)
+{
+	struct rt_rq *rt_rq = &global_rq->rq_to_pull_to->rt;
+
+	BUG_ON(global_rq->rq_to_pull_to == NULL);
+	BUG_ON(rt_rq->rq != global_rq);
+
+	group_pull_rt_task(rt_rq);
+	global_rq->rq_to_pull_to = NULL;
+}
+#else /* CONFIG_RT_GROUP_SCHED */
+static void group_pull_rt_task(struct rt_rq *this_rt_rq) { }
+static void group_push_rt_tasks(struct rt_rq *rt_rq) { }
+#endif /* CONFIG_RT_GROUP_SCHED */
+
 /*
  * If we are not running and we are not going to reschedule soon, we should
  * try to push tasks away now
kernel/sched/sched.h +10 -0
index 55631d93e02..4e287aaa72f 100644
@@ -1282,6 +1282,16 @@ struct rq {
 	call_single_data_t	cfsb_csd;
 	struct list_head	cfsb_csd_list;
 #endif
+
+#ifdef CONFIG_RT_GROUP_SCHED
+	/*
+	 * Balance callbacks operate only on global runqueues.
+	 * These pointers allow referencing cgroup specific runqueues
+	 * for balancing operations.
+	 */
+	struct rq		*rq_to_push_from;
+	struct rq		*rq_to_pull_to;
+#endif
 };
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
Signature · 1 line
2.51.0
#36 lore raw

RFC PATCH v3 20/24 sched/rt: Add HCBS migration related checks and function calls

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Add HCBS related checks and operations to allow rt-task migration, differentiating between cgroup's tasks or tasks that run on the global runqueue.

---
 kernel/sched/rt.c | 69 ++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 53 insertions(+), 16 deletions(-)
kernel/sched/rt.c +53 -16
index 1fbcac14fc9..a864f569011 100644
@@ -884,6 +884,11 @@ select_task_rq_rt(struct task_struct *p, int cpu, int flags)
 	struct rq *rq;
 	bool test;
 
+	/* Just return the task_cpu for processes inside task groups */
+	if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) &&
+	    is_dl_group(rt_rq_of_se(&p->rt)))
+		goto out;
+
 	/* For anything but wake ups, just return the task_cpu */
 	if (!(flags & (WF_TTWU | WF_FORK)))
 		goto out;
@@ -983,7 +988,10 @@ static int balance_rt(struct rq *rq, struct task_struct *p, struct rq_flags *rf)
 		 * not yet started the picking loop.
 		 */
 		rq_unpin_lock(rq, rf);
-		pull_rt_task(rq);
+		if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) && is_dl_group(rt_rq_of_se(&p->rt)))
+			group_pull_rt_task(rt_rq_of_se(&p->rt));
+		else
+			pull_rt_task(rq);
 		rq_repin_lock(rq, rf);
 	}
 
@@ -1085,7 +1093,10 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool f
 	if (rq->donor->sched_class != &rt_sched_class)
 		update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 0);
 
-	rt_queue_push_tasks(rt_rq);
+	if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) && is_dl_group(rt_rq))
+		rt_queue_push_from_group(rt_rq);
+	else
+		rt_queue_push_tasks(rt_rq);
 }
 
 static struct sched_rt_entity *pick_next_rt_entity(struct rt_rq *rt_rq)
@@ -1148,6 +1159,13 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s
 	 */
 	if (on_rt_rq(&p->rt) && p->nr_cpus_allowed > 1)
 		enqueue_pushable_task(rt_rq, p);
+
+	if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) && is_dl_group(rt_rq)) {
+		struct sched_dl_entity *dl_se = dl_group_of(rt_rq);
+
+		if (dl_se->dl_throttled)
+			rt_queue_push_from_group(rt_rq);
+	}
 }
 
 /* Only try algorithms three times */
@@ -2209,6 +2227,7 @@ static void group_push_rt_tasks(struct rt_rq *rt_rq) { }
  */
 static void task_woken_rt(struct rq *rq, struct task_struct *p)
 {
+	struct rt_rq *rt_rq = rt_rq_of_se(&p->rt);
 	bool need_to_push = !task_on_cpu(rq, p) &&
 			    !test_tsk_need_resched(rq->curr) &&
 			    p->nr_cpus_allowed > 1 &&
@@ -2216,7 +2235,12 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p)
 			    (rq->curr->nr_cpus_allowed < 2 ||
 			     rq->donor->prio <= p->prio);
 
-	if (need_to_push)
+	if (!need_to_push)
+		return;
+
+	if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) && is_dl_group(rt_rq))
+		group_push_rt_tasks(rt_rq);
+	else
 		push_rt_tasks(rq);
 }
 
@@ -2256,7 +2280,9 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)
 	if (!task_on_rq_queued(p) || rt_rq->rt_nr_running)
 		return;
 
-	if (!IS_ENABLED(CONFIG_RT_GROUP_SCHED))
+	if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) && is_dl_group(rt_rq))
+		rt_queue_pull_to_group(rt_rq);
+	else
 		rt_queue_pull_task(rt_rq);
 }
 
@@ -2277,12 +2303,21 @@ void __init init_sched_rt_class(void)
  */
 static void switched_to_rt(struct rq *rq, struct task_struct *p)
 {
+	struct rt_rq *rt_rq;
+
 	/*
 	 * If we are running, update the avg_rt tracking, as the running time
 	 * will now on be accounted into the latter.
 	 */
 	if (task_current(rq, p)) {
 		update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 0);
+
+		if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) && is_dl_group(rt_rq_of_se(&p->rt))) {
+			struct sched_dl_entity *dl_se = dl_group_of(rt_rq_of_se(&p->rt));
+
+			p->dl_server = dl_se;
+		}
+
 		return;
 	}
 
@@ -2292,17 +2327,15 @@ static void switched_to_rt(struct rq *rq, struct task_struct *p)
 	 * then see if we can move to another run queue.
 	 */
 	if (task_on_rq_queued(p)) {
-
-#ifndef CONFIG_RT_GROUP_SCHED
-		if (p->nr_cpus_allowed > 1 && rq->rt.overloaded)
-			rt_queue_push_tasks(rt_rq_of_se(&p->rt));
-#else
-		if (rt_rq_of_se(&p->rt)->overloaded) {
-		} else {
-			if (p->prio < rq->curr->prio)
-				resched_curr(rq);
+		rt_rq = rt_rq_of_se(&p->rt);
+		if (!is_dl_group(rt_rq) && p->nr_cpus_allowed > 1 && rq->rt.overloaded) {
+			rt_queue_push_tasks(rt_rq);
+			return;
+		} else if (is_dl_group(rt_rq) && rt_rq->overloaded) {
+			rt_queue_push_from_group(rt_rq);
+			return;
 		}
-#endif
+
 		if (p->prio < rq->donor->prio && cpu_online(cpu_of(rq)))
 			resched_curr(rq);
 	}
@@ -2325,8 +2358,12 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, int oldprio)
 		 * If our priority decreases while running, we
 		 * may need to pull tasks to this runqueue.
 		 */
-		if (!IS_ENABLED(CONFIG_RT_GROUP_SCHED) && oldprio < p->prio)
-			rt_queue_pull_task(rt_rq);
+		if (oldprio < p->prio) {
+			if (IS_ENABLED(CONFIG_RT_GROUP_SCHED) && is_dl_group(rt_rq))
+				rt_queue_pull_to_group(rt_rq);
+			else
+				rt_queue_pull_task(rt_rq);
+		}
 
 		/*
 		 * If there's a higher priority task waiting to run
Signature · 1 line
2.51.0
#37 lore raw

RFC PATCH v3 21/24 sched/deadline: Make rt-cgroup's servers pull tasks on timer replenishment

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

---
 kernel/sched/deadline.c | 6 +++++-
 kernel/sched/rt.c       | 6 +++++-
 2 files changed, 10 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +5 -1
index abe11985c41..27a5d5ffd39 100644
@@ -1290,6 +1290,7 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
 {
 	struct rq *rq = rq_of_dl_se(dl_se);
 	u64 fw;
+	bool has_tasks;
 
 	scoped_guard (rq_lock, rq) {
 		struct rq_flags *rf = &scope.rf;
@@ -1303,7 +1304,10 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_
 		if (!dl_se->dl_runtime)
 			return HRTIMER_NORESTART;
 
-		if (!dl_se->server_has_tasks(dl_se)) {
+		rq_unpin_lock(rq, rf);
+		has_tasks = dl_se->server_has_tasks(dl_se);
+		rq_repin_lock(rq, rf);
+		if (!has_tasks) {
 			replenish_dl_entity(dl_se);
 			dl_server_stopped(dl_se);
 			return HRTIMER_NORESTART;
kernel/sched/rt.c +5 -1
index a864f569011..b09f0887e3b 100644
@@ -1,4 +1,3 @@
-#pragma GCC diagnostic ignored "-Wunused-function"
 // SPDX-License-Identifier: GPL-2.0
 /*
  * Real-Time Scheduling Class (mapped to the SCHED_FIFO and SCHED_RR
@@ -153,6 +152,11 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool f
 
 static bool rt_server_has_tasks(struct sched_dl_entity *dl_se)
 {
+#ifdef CONFIG_SMP
+	struct rt_rq *rt_rq = &dl_se->my_q->rt;
+
+	group_pull_rt_task(rt_rq);
+#endif
 	return !!dl_se->my_q->rt.rt_nr_running;
 }
 
Signature · 1 line
2.51.0
#38 lore raw

RFC PATCH v3 22/24 sched/deadline: Fix HCBS migrations on server stop

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Do not unthrottle a non-fair-server dl_server_stop(), since it ends up being stopped when throttled (we try to migrate all the RT tasks away from it).

Notes:
This is a temporary workaround, but it will be hopefully removed in favor of less invasive code.

---
 kernel/sched/deadline.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/deadline.c +4 -2
index 27a5d5ffd39..6435a94159d 100644
@@ -1707,9 +1707,11 @@ void dl_server_stop(struct sched_dl_entity *dl_se)
 		return;
 
 	dequeue_dl_entity(dl_se, DEQUEUE_SLEEP);
-	hrtimer_try_to_cancel(&dl_se->dl_timer);
+	if (dl_se == &rq_of_dl_se(dl_se)->fair_server) {
+		hrtimer_try_to_cancel(&dl_se->dl_timer);
+		dl_se->dl_throttled = 0;
+	}
 	dl_se->dl_defer_armed = 0;
-	dl_se->dl_throttled = 0;
 	dl_se->dl_server_active = 0;
 }
 
Signature · 1 line
2.51.0
#39 lore raw

RFC PATCH v3 23/24 sched/core: Execute enqueued balance callbacks when changing allowed CPUs

In reply to Yuri Andriaccio

From: luca abeni <[email protected]>

Execute balancing callbacks when setting the affinity of a task, since the HCBS scheduler may request balancing of throttled dl_servers to fully utilize the server's bandwidth.

---
 kernel/sched/core.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/sched/core.c +13 -0
index d1d7215c4a2..7639e9abba1 100644
@@ -2950,6 +2950,7 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 	if (cpumask_test_cpu(task_cpu(p), &p->cpus_mask) ||
 	    (task_current_donor(rq, p) && !task_current(rq, p))) {
 		struct task_struct *push_task = NULL;
+		struct balance_callback *head;
 
 		if ((flags & SCA_MIGRATE_ENABLE) &&
 		    (p->migration_flags & MDF_PUSH) && !rq->push_busy) {
@@ -2968,11 +2969,13 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 		}
 
 		preempt_disable();
+		head = splice_balance_callbacks(rq);
 		task_rq_unlock(rq, p, rf);
 		if (push_task) {
 			stop_one_cpu_nowait(rq->cpu, push_cpu_stop,
 					    p, &rq->push_work);
 		}
+		balance_callbacks(rq, head);
 		preempt_enable();
 
 		if (complete)
@@ -3027,6 +3030,8 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 	}
 
 	if (task_on_cpu(rq, p) || READ_ONCE(p->__state) == TASK_WAKING) {
+		struct balance_callback *head;
+
 		/*
 		 * MIGRATE_ENABLE gets here because 'p == current', but for
 		 * anything else we cannot do is_migration_disabled(), punt
@@ -3040,16 +3045,19 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 			p->migration_flags &= ~MDF_PUSH;
 
 		preempt_disable();
+		head = splice_balance_callbacks(rq);
 		task_rq_unlock(rq, p, rf);
 		if (!stop_pending) {
 			stop_one_cpu_nowait(cpu_of(rq), migration_cpu_stop,
 					    &pending->arg, &pending->stop_work);
 		}
+		balance_callbacks(rq, head);
 		preempt_enable();
 
 		if (flags & SCA_MIGRATE_ENABLE)
 			return 0;
 	} else {
+		struct balance_callback *head;
 
 		if (!is_migration_disabled(p)) {
 			if (task_on_rq_queued(p))
@@ -3060,7 +3068,12 @@ static int affine_move_task(struct rq *rq, struct task_struct *p, struct rq_flag
 				complete = true;
 			}
 		}
+
+		preempt_disable();
+		head = splice_balance_callbacks(rq);
 		task_rq_unlock(rq, p, rf);
+		balance_callbacks(rq, head);
+		preempt_enable();
 
 		if (complete)
 			complete_all(&pending->done);
Signature · 1 line
2.51.0
#40 lore raw

RFC PATCH v3 24/24 sched/core: Execute enqueued balance callbacks when migrating task betweeen cgroups

In reply to Yuri Andriaccio

Execute balancing callbacks when migrating task between cgroups, since the HCBS scheduler, similarly to the previous patch, may request balancing of throttled dl_servers to fully utilize the server's bandwidth.

---
 kernel/sched/core.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/core.c +9 -2
index 7639e9abba1..8d99a0c20c6 100644
@@ -9251,10 +9251,11 @@ void sched_move_task(struct task_struct *tsk, bool for_autogroup)
 {
 	int queued, running, queue_flags =
 		DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
+	struct balance_callback *head;
 	struct rq *rq;
+	struct rq_flags rf;
 
-	CLASS(task_rq_lock, rq_guard)(tsk);
-	rq = rq_guard.rq;
+	rq = task_rq_lock(tsk, &rf);
 
 	update_rq_clock(rq);
 
@@ -9281,6 +9282,12 @@ void sched_move_task(struct task_struct *tsk, bool for_autogroup)
 		 */
 		resched_curr(rq);
 	}
+
+	preempt_disable();
+	head = splice_balance_callbacks(rq);
+	task_rq_unlock(rq, tsk, &rf);
+	balance_callbacks(rq, head);
+	preempt_enable();
 }
 
 static struct cgroup_subsys_state *
Signature · 1 line
2.51.0
maintainer
#41 lore raw
In reply to Yuri Andriaccio

Hello!

Quoted 11 lines · Yuri Andriaccio

Hello,

This is the v3 for Hierarchical Constant Bandwidth Server, aiming at replacing the current RT_GROUP_SCHED mechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ . You can find the previous versions of this patchset at the bottom of the page, in particular version 1 which talks in more detail what this patchset is all about and how it is implemented.

Quoted 6 lines

This v3 version further reworks some of the patches as suggested by Juri Lelli. While most of the work is refactorings, the following were also changed:
- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/

Peter wasn't indeed happy with that patch, but I am not sure we finished that discussion. Both myself and Luca had further objections to what Peter said, but not further replies after (which can very well be a sign that he is still adamnt in saying no go away :). Peter?

https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20250904091217.78de3dde@luca64/

...

Quoted 5 lines

For this v3 version we've also polished the testing system we are using and made it public for testers to run on their own machines. The source code can be found at https://github.com/Yurand2000/HCBS-Test-Suite , along with a README that explains how to use it. Nonetheless I've reported a description of the tools and instruction later in the page.

Thanks for this. Quite cool. I tried to run the tests, but it looks like the migration set brought my qemu 8 CPUs system to an halt (had to kill the thing). I will try again.

I have been working a bit in trying to come up with a testing framework for SCHED_DEADLINE, which I didn't unfortunately posted yet mostly because I was waiting for the discussion about the patch mentioned above to settle (which would require adaptation in the tests that check for bandwidth limits). You can find the idea here [1]. It's thought to be an addition to kselftests. I believe your test suite can be extended to cover the tests I implemented and more, so I am not super attached to my attempt, but it would be good (I think) to converge to something that we can maintain for the future, so maybe have a plan to possibly merge the suites. What do you think?

1 - https://github.com/jlelli/linux/commits/experimental/deadline-selftests-scripts/

...

Quoted 6 lines

Testing v3:

The HCBS mechanism has been evaluated on several syntetic tests which are designed to stress the HCBS scheduler and verify that non-interference and mathematical schedulability guarantees are really enforced by the scheduling algorithm.

...

Quoted 4 lines

The tasksets minimal requirements were computed using a closed-source software, explaining why the tasksets are supplied separately. A open-source analyser is being written to update this step in the future and also allow for more customization for the testers.

On this (generation of random taskset and corresponding schedulability checks) I also started working on a different tool, rt-audit.

https://github.com/jlelli/rt-audit

It's very simple, the tool, the random generation and the checks. But I honestly like the idea that it's based on rt-app. Please take a look if you have a chance and tell me what you think. Again, I feel it would be good to converge towards something open that we can maintain.

I will be trying to find time to continue testing and reviewing this RFC of course.

Thanks,
Juri

maintainer
#42 lore raw

On 02/10/25 10:00, Juri Lelli wrote:

...

Quoted 2 lines

I will be trying to find time to continue testing and reviewing this RFC of course.

I provided comments up to the last part of the series that implements migration. I will now stop on purpose and wait for replies/new versions of the patches I reviewed. I believe we could try to leave migration "for later" while we solidify the basics. What do you think?

Thanks,
Juri

#43 lore raw

Hi,

Quoted 4 lines

I provided comments up to the last part of the series that implements migration. I will now stop on purpose and wait for replies/new versions of the patches I reviewed. I believe we could try to leave migration "for later" while we solidify the basics. What do you think?

Thanks a lot for your comments. It also makes sense to us to solidify the basics before going on to the migration code and later updates. We have sent the migration related patches just to show the end goal of this patchset and give some intuition of possible use cases and demonstration on the provided guarantees. We will work on your comments and submit a v4 patchset soon.

Have a nice day,
Yuri

maintainer
#44 lore raw
In reply to Juri Lelli
Quoted 21 lines · Juri Lelli

Hello!

Quoted 11 lines · Yuri Andriaccio

Hello,

This is the v3 for Hierarchical Constant Bandwidth Server, aiming at replacing the current RT_GROUP_SCHED mechanism with something more robust and theoretically sound. The patchset has been presented at OSPM25
(https://retis.sssup.it/ospm-summit/), and a summary of its inner workings can be found at https://lwn.net/Articles/1021332/ . You can find the previous versions of this patchset at the bottom of the page, in particular version 1 which talks in more detail what this patchset is all about and how it is implemented.

Quoted 6 lines

This v3 version further reworks some of the patches as suggested by Juri Lelli. While most of the work is refactorings, the following were also changed:
- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/

Quoted 7 lines

Peter wasn't indeed happy with that patch, but I am not sure we finished that discussion. Both myself and Luca had further objections to what Peter said, but not further replies after (which can very well be a sign that he is still adamnt in saying no go away :). Peter?

https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20250904091217.78de3dde@luca64/

I had a quick chat with Peter on IRC about this. We now seem to agree that a third option would be to move to explicitly account dl-server(s), correspondingly moving from a 95% to 100% limit. That would also make our life easier in the future with additional dl-servers (e.g. scx-server).

What do you think?

#45 lore raw

Hi Juri,

On Mon, 20 Oct 2025 11:40:22 +0200
Juri Lelli <[email protected]> wrote:
[...]

Quoted 16 lines
Quoted 7 lines
Quoted 6 lines

- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/

Quoted 8 lines

Peter wasn't indeed happy with that patch, but I am not sure we finished that discussion. Both myself and Luca had further objections to what Peter said, but not further replies after (which can very well be a sign that he is still adamnt in saying no go away :). Peter?

https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20250904091217.78de3dde@luca64/

Quoted 7 lines

I had a quick chat with Peter on IRC about this. We now seem to agree that a third option would be to move to explicitly account dl-server(s), correspondingly moving from a 95% to 100% limit. That would also make our life easier in the future with additional dl-servers (e.g. scx-server).

What do you think?

This looks like another good solution, thanks!

So, if I understand well with this approach /proc/sys/kernel/sched_rt_{runtime, period}_us would be set to 100% as a default, right?

It is often useful to know what is the maximum CPU utilization that can be guaranteed to real-time tasks... With this approach, it would be 100% - <dl_server utilization>, but this can change when scx servers are added... What about making this information available to userspace programs? (maybe /proc/sys/kernel/sched_rt_{runtime, period}_us could provide such information? Or is it better to add a new interface?)

Thanks,
Luca

maintainer
#46 lore raw
Quoted 35 lines · luca abeni

Hi Juri,

On Mon, 20 Oct 2025 11:40:22 +0200
Juri Lelli <[email protected]> wrote:
[...]

Quoted 16 lines
Quoted 7 lines
Quoted 6 lines

- The first patch which removed fair-servers' bandwidth accounting has been removed, as it was deemed wrong. You can find the last version of this removed patch, just for history reasons, here:
https://lore.kernel.org/all/[email protected]/

Quoted 8 lines

Peter wasn't indeed happy with that patch, but I am not sure we finished that discussion. Both myself and Luca had further objections to what Peter said, but not further replies after (which can very well be a sign that he is still adamnt in saying no go away :). Peter?

https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20250904091217.78de3dde@luca64/

Quoted 7 lines

I had a quick chat with Peter on IRC about this. We now seem to agree that a third option would be to move to explicitly account dl-server(s), correspondingly moving from a 95% to 100% limit. That would also make our life easier in the future with additional dl-servers (e.g. scx-server).

What do you think?

This looks like another good solution, thanks!

So, if I understand well with this approach /proc/sys/kernel/sched_rt_{runtime, period}_us would be set to 100% as a default, right?

Quoted 6 lines

It is often useful to know what is the maximum CPU utilization that can be guaranteed to real-time tasks... With this approach, it would be 100% - <dl_server utilization>, but this can change when scx servers are added... What about making this information available to userspace programs? (maybe /proc/sys/kernel/sched_rt_{runtime, period}_us could provide such information? Or is it better to add a new interface?)

Not sure. If we set it to 100% by default (as you suggest, which makes sense to me) I wonder what would be a usecase/need to set it to less than 100% later on. We have the debug interface for tweaking dl-servers and sched_rt_ interface doesn't distinguish between DEADLINE and RT anyway (so no way to leave some "bandwidth" around for RT tasks).

Maybe it's an interface we want to start deprecating and we can come up with something better and/or more useful? Peter?