Skip to contentSkip to conversation

sched/fair: Delay throttling to kernel exit

RFC PATCH Superseded Valentin Schneider 3 participants 13 messages Nov 30, 2023 – Dec 19, 2023

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Valentin Schneider

author reviewer
#0 lore raw

cover sched/fair: Delay throttling to kernel exit

Hi folks

I apologize for the one-big-patch-to-rule-them-all, I tried splitting but I couldn't get to anything that made sense, so it's one big patch and a small follow-up.

TL;DR of what I'd like to hear about:
o Is there interest in getting this in for !PREEMPT_RT reasons?
o Any ideas on how to make cgroup migration less horrible so we can fully pop out the throttled cfs_rq se's?

Survives a good hour of my testing below on a 4-CPU QEMU system, but I expect the throttled clocks & PELT to be busted.

Testing
=======

setup
+++++
mount -t cgroup -o cpu none /root/cpu

mkdir /root/cpu/cg0
echo 10000 > /root/cpu/cg0/cpu.cfs_period_us
echo 1000 > /root/cpu/cg0/cpu.cfs_quota_us

mkdir /root/cpu/cg0/cg00
mkdir /root/cpu/cg0/cg01

mkdir /root/cpu/cg0/cg00/cg000
mkdir /root/cpu/cg0/cg00/cg001

read.sh
+++++++
while true; do cat /sys/devices/system/cpu/smt/active &>/dev/null; done

repro.sh
++++++++

spawn() {
    ./read.sh &
    PID=$!
    echo "Tracing PID${PID}"
    echo $PID > $1
}

spawn cpu/cg0/tasks
spawn cpu/cg0/tasks
spawn cpu/cg0/tasks
spawn cpu/cg0/tasks

spawn cpu/cg0/cg01/tasks

spawn cpu/cg0/cg00/cg000/tasks
spawn cpu/cg0/cg00/cg001/tasks

sleep 1

kill $(jobs -p)

Valentin Schneider (2):
sched/fair: Only throttle CFS tasks on return to userspace sched/fair: Repurpose cfs_rq_throttled()

 include/linux/sched.h |   2 +
 kernel/sched/core.c   |   6 +-
 kernel/sched/debug.c  |   4 +-
 kernel/sched/fair.c   | 296 +++++++++++++++++++++++++++---------------
 kernel/sched/sched.h  |   8 +-
 5 files changed, 204 insertions(+), 112 deletions(-)

--
2.41.0

author reviewer
#1 lore raw

RFC PATCH 1/2 sched/fair: Only throttle CFS tasks on return to userspace

As reported in [1], CFS bandwidth throttling is a source of headaches in PREEMPT_RT - generally speaking, a throttled CFS task can hold locks that prevent ksoftirqd from running, which prevents replenishing & unthrottling the cfs_rq of said CFS task.

Peter mentioned that there have been discussions on changing /when/ the throttling happens: rather than have it be done immediately upon updating the runtime statistics and realizing the cfs_rq has depleted its quota, we wait for the task to be about to return to userspace.

I'm not aware of the arguments in favour of this for !PREEMPT_RT, but given [1] I had a look into it. Using this approach means no task can be throttled while holding a kernel lock, which solves the locking dependency issue.

Concept & headaches
===================

The original idea was:
o cfs_rq runs out of runtime, flag it
o Take the tasks out of the cfs_rq one by one as they exit the kernel
o Dequeue the cfs_rq once it is empty of tasks (as if doing "normal" throttling)
o Put the tasks back onto it
o At unthrottle, simply re-enqueue the cfs_rq sched_entity and all is well

Cgroup migration kinda breaks all of that. Consider a task moving to a now-fully-throttled cfs_rq: if we just enqueue it onto the throttled se, then it's the same as throttling it, and if it is holding a problematic lock then we're back to square one.
This means we can never fully throttle a cfs_rq, which is making me not so confident in this approach.

Should/can we consider deferring such migrations to the unthrottle (with something like in affine_move_task())? I think failing the migration is not an option due to e.g. autogroups, and IIUC returning from writing to cgroup/tasks means the migration has happened (like updating a task's affinity), which means this would need to be a blocking wait.

CPU migrations within the same taskgroup are similarly problematic (e.g. moving from a cfs_rq that still has a bit of ->runtime left to one that has run out).

Throttle clocks are also kinda busted in this approach, but they aren't my greatest concern at this point :-)

Implementation
==============

cfs_rq's that deplete their ->runtime get flagged for later throttling via ->in_throttle_limbo, and this is propagated down the hierarchy.

Tasks enqueued in a cfs_rq with ->in_throttle_limbo picked for execution get a task_work added, which lets the actual throttling happen in exit_to_user_mode().

Upon unthrottling, tasks are enqueued back onto their respective cfs_rq. Unlike the previous throttling implementation, cfs_rq's can be unthrottled while in a half-throttled state (i.e. some tasks have been removed from them, while others are still enqueued and runnable), so the unthrottling process is a bit more involved.

[1]: https://lore.kernel.org/all/[email protected]/

---
 include/linux/sched.h |   2 +
 kernel/sched/core.c   |   6 +-
 kernel/sched/debug.c  |   4 +-
 kernel/sched/fair.c   | 257 +++++++++++++++++++++++++++++++-----------
 kernel/sched/sched.h  |   7 ++
 5 files changed, 206 insertions(+), 70 deletions(-)
include/linux/sched.h +2 -0
index e4235bbfad776..20194e4c8b0b5 100644
@@ -805,6 +805,8 @@ struct task_struct {
 
 #ifdef CONFIG_CGROUP_SCHED
 	struct task_group		*sched_task_group;
+	struct callback_head            sched_throttle_work;
+	struct list_head                throttle_node;
 #endif
 
 #ifdef CONFIG_UCLAMP_TASK
kernel/sched/core.c +5 -1
index 7a0c16115b79f..a63152072554b 100644
@@ -4504,6 +4504,10 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 	p->se.cfs_rq			= NULL;
 #endif
 
+#ifdef CONFIG_CFS_BANDWIDTH
+	init_cfs_throttle_work(p);
+#endif
+
 #ifdef CONFIG_SCHEDSTATS
 	/* Even if schedstat is disabled, there should not be garbage */
 	memset(&p->stats, 0, sizeof(p->stats));
@@ -10823,7 +10827,7 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota,
 		cfs_rq->runtime_enabled = runtime_enabled;
 		cfs_rq->runtime_remaining = 0;
 
-		if (cfs_rq->throttled)
+		if (cfs_rq->in_throttle_limbo)
 			unthrottle_cfs_rq(cfs_rq);
 	}
 
kernel/sched/debug.c +2 -2
index 4580a450700ec..8867904cf7eee 100644
@@ -695,8 +695,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 #endif
 #endif
 #ifdef CONFIG_CFS_BANDWIDTH
-	SEQ_printf(m, "  .%-30s: %d\n", "throttled",
-			cfs_rq->throttled);
+	SEQ_printf(m, "  .%-30s: %d\n", "in_throttle_limbo",
+			cfs_rq->in_throttle_limbo);
 	SEQ_printf(m, "  .%-30s: %d\n", "throttle_count",
 			cfs_rq->throttle_count);
 #endif
kernel/sched/fair.c +190 -67

385 lines not on this page · raw message

kernel/sched/sched.h +7 -0
index 2e5a95486a422..be29154d93898 100644
@@ -633,8 +633,13 @@ struct cfs_rq {
 	u64			throttled_clock_self_time;
 	int			throttled;
 	int			throttle_count;
+	int                     in_throttle_limbo;
+	/* Temp storage for updating the counts during unthrottling */
+	unsigned int            unthrottled_h_nr_running;
+	unsigned int            unthrottled_idle_h_nr_running;
 	struct list_head	throttled_list;
 	struct list_head	throttled_csd_list;
+	struct list_head	throttled_limbo_list;
 #endif /* CONFIG_CFS_BANDWIDTH */
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 };
@@ -2416,6 +2421,8 @@ extern void init_sched_dl_class(void);
 extern void init_sched_rt_class(void);
 extern void init_sched_fair_class(void);
 
+extern void init_cfs_throttle_work(struct task_struct *p);
+
 extern void reweight_task(struct task_struct *p, int prio);
 
 extern void resched_curr(struct rq *rq);
Signature · 1 line
2.41.0
#2 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

Quoted 5 lines · Valentin Schneider

As reported in [1], CFS bandwidth throttling is a source of headaches in PREEMPT_RT - generally speaking, a throttled CFS task can hold locks that prevent ksoftirqd from running, which prevents replenishing & unthrottling the cfs_rq of said CFS task.

Quoted 8 lines

Peter mentioned that there have been discussions on changing /when/ the throttling happens: rather than have it be done immediately upon updating the runtime statistics and realizing the cfs_rq has depleted its quota, we wait for the task to be about to return to userspace.

I'm not aware of the arguments in favour of this for !PREEMPT_RT, but given [1] I had a look into it. Using this approach means no task can be throttled while holding a kernel lock, which solves the locking dependency issue.

The alternative we've been experimenting with (and still running into other issues that have made it hard to tell if they work) is to still leave the tasks on their cfs_rqs, but instead have two task_timelines or similar per cfs_rq, one of which only has unthrottleable tasks (or partially-throttled child cgroups) on it. Then when picking into a partially-unthrottled cfs_rq you only look at that alternate task_timeline.

This means that we get to skip this per-actually-throttled-task loop:

Quoted 15 lines
kernel/sched/fair.c +10 -0
@@ -5548,7 +5548,61 @@ static int tg_unthrottle_up(struct task_group *tg, void *data)
 {
 	struct rq *rq = data;
 	struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)];
+	struct sched_entity *se = tg->se[cpu_of(rq)];
+	struct cfs_rq *pcfs_rq = cfs_rq_of(se);
+	long task_delta = 0, idle_task_delta = 0;
+	struct task_struct *p, *tmp;
 
+	/*
+	 * Re-enqueue the tasks that have been throttled at this level.
+	 *
+	 * The task count is up-propagated via ->unthrottled_*h_nr_running,
+	 * as we can't purely rely on h_nr_running post-enqueue: the unthrottle
+	 * might happen when a cfs_rq still has some tasks enqueued, either still

kernel/sched/fair.c static int tg_unthrottle_up(struct task_group *tg, void *data) · line 5571

Quoted 10 lines
kernel/sched/fair.c +10 -0
+	 * making their way to userspace, or freshly migrated to it.
+	 */
+	list_for_each_entry_safe(p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) {
+		struct sched_entity *pse = &p->se;
+
+		list_del_init(&p->throttle_node);
+		enqueue_entity(cfs_rq, pse, ENQUEUE_WAKEUP);
+		task_delta++;
+		idle_task_delta += task_has_idle_policy(p);
+	}

The downsides are that you instead have extra operations per enqueue/dequeue/pick (but just an extra list/rbtree operation or check), and that it doesn't do *any* accounting change for a partially dequeued cfs_rq.

I'm going to try putting together a cleaner variant of our version that works via task_work instead of bracketing every relevant entry point. (That design came from when we were trying instead to only do it for tasks holding actual locks)

author reviewer
#3 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

Quoted 16 lines · Benjamin Segall
Quoted 5 lines · Valentin Schneider

As reported in [1], CFS bandwidth throttling is a source of headaches in PREEMPT_RT - generally speaking, a throttled CFS task can hold locks that prevent ksoftirqd from running, which prevents replenishing & unthrottling the cfs_rq of said CFS task.

Quoted 8 lines

Peter mentioned that there have been discussions on changing /when/ the throttling happens: rather than have it be done immediately upon updating the runtime statistics and realizing the cfs_rq has depleted its quota, we wait for the task to be about to return to userspace.

I'm not aware of the arguments in favour of this for !PREEMPT_RT, but given [1] I had a look into it. Using this approach means no task can be throttled while holding a kernel lock, which solves the locking dependency issue.

Quoted 7 lines

The alternative we've been experimenting with (and still running into other issues that have made it hard to tell if they work) is to still leave the tasks on their cfs_rqs, but instead have two task_timelines or similar per cfs_rq, one of which only has unthrottleable tasks (or partially-throttled child cgroups) on it. Then when picking into a partially-unthrottled cfs_rq you only look at that alternate task_timeline.

IIUC then you don't dequeue the cfs_rq's se, but instead rely on the RB tree switch to only have unthrottable tasks running.

Quoted 24 lines

This means that we get to skip this per-actually-throttled-task loop:

Quoted 22 lines
kernel/sched/fair.c +17 -0
@@ -5548,7 +5548,61 @@ static int tg_unthrottle_up(struct task_group *tg, void *data)
 {
     struct rq *rq = data;
     struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)];
+	struct sched_entity *se = tg->se[cpu_of(rq)];
+	struct cfs_rq *pcfs_rq = cfs_rq_of(se);
+	long task_delta = 0, idle_task_delta = 0;
+	struct task_struct *p, *tmp;
 
+	/*
+	 * Re-enqueue the tasks that have been throttled at this level.
+	 *
+	 * The task count is up-propagated via ->unthrottled_*h_nr_running,
+	 * as we can't purely rely on h_nr_running post-enqueue: the unthrottle
+	 * might happen when a cfs_rq still has some tasks enqueued, either still
+	 * making their way to userspace, or freshly migrated to it.
+	 */
+	list_for_each_entry_safe(p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) {
+		struct sched_entity *pse = &p->se;
+
+		list_del_init(&p->throttle_node);
+		enqueue_entity(cfs_rq, pse, ENQUEUE_WAKEUP);
Quoted 9 lines

kernel/sched/fair.c static int tg_unthrottle_up(struct task_group *tg, void *data) · line 5571

Quoted 3 lines
kernel/sched/fair.c +3 -0
+		task_delta++;
+		idle_task_delta += task_has_idle_policy(p);
+	}

The downsides are that you instead have extra operations per enqueue/dequeue/pick (but just an extra list/rbtree operation or check), and that it doesn't do *any* accounting change for a partially dequeued cfs_rq.

I would assume we want to keep the *nr_running as close to reality as possible, given their impact on pick_next_task_fair() & load_balance().

Quoted 4 lines

I'm going to try putting together a cleaner variant of our version that works via task_work instead of bracketing every relevant entry point. (That design came from when we were trying instead to only do it for tasks holding actual locks)

Interesting, thank you for sharing! I assume then the motivation for this is to reduce latencies caused by throttling lock holders?

#4 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

Quoted 11 lines · Valentin Schneider
Quoted 9 lines · Benjamin Segall

Valentin Schneider <[email protected]> writes:

The alternative we've been experimenting with (and still running into other issues that have made it hard to tell if they work) is to still leave the tasks on their cfs_rqs, but instead have two task_timelines or similar per cfs_rq, one of which only has unthrottleable tasks (or partially-throttled child cgroups) on it. Then when picking into a partially-unthrottled cfs_rq you only look at that alternate task_timeline.

Quoted 2 lines

IIUC then you don't dequeue the cfs_rq's se, but instead rely on the RB tree switch to only have unthrottable tasks running.

Correct.

Quoted 35 lines
Quoted 25 lines

This means that we get to skip this per-actually-throttled-task loop:

Quoted 22 lines
kernel/sched/fair.c +17 -0
@@ -5548,7 +5548,61 @@ static int tg_unthrottle_up(struct task_group *tg, void *data)
 {
     struct rq *rq = data;
     struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)];
+	struct sched_entity *se = tg->se[cpu_of(rq)];
+	struct cfs_rq *pcfs_rq = cfs_rq_of(se);
+	long task_delta = 0, idle_task_delta = 0;
+	struct task_struct *p, *tmp;
 
+	/*
+	 * Re-enqueue the tasks that have been throttled at this level.
+	 *
+	 * The task count is up-propagated via ->unthrottled_*h_nr_running,
+	 * as we can't purely rely on h_nr_running post-enqueue: the unthrottle
+	 * might happen when a cfs_rq still has some tasks enqueued, either still
+	 * making their way to userspace, or freshly migrated to it.
+	 */
+	list_for_each_entry_safe(p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) {
+		struct sched_entity *pse = &p->se;
+
+		list_del_init(&p->throttle_node);
+		enqueue_entity(cfs_rq, pse, ENQUEUE_WAKEUP);
Quoted 9 lines

kernel/sched/fair.c static int tg_unthrottle_up(struct task_group *tg, void *data) · line 5571

Quoted 3 lines
kernel/sched/fair.c +3 -0
+		task_delta++;
+		idle_task_delta += task_has_idle_policy(p);
+	}

The downsides are that you instead have extra operations per enqueue/dequeue/pick (but just an extra list/rbtree operation or check), and that it doesn't do *any* accounting change for a partially dequeued cfs_rq.

Quoted 3 lines

I would assume we want to keep the *nr_running as close to reality as possible, given their impact on pick_next_task_fair() & load_balance().

Yeah, and while it's maybe ok for the longer-period load_balance, it's definitely sketchy for the shorter term things like select_idle_sibling. In theory we could duplicate more and more of the accounting... (though that also then becomes questionable amounts of increased overhead on enqueue/dequeue).

Quoted 7 lines
Quoted 4 lines

I'm going to try putting together a cleaner variant of our version that works via task_work instead of bracketing every relevant entry point. (That design came from when we were trying instead to only do it for tasks holding actual locks)

Interesting, thank you for sharing! I assume then the motivation for this is to reduce latencies caused by throttling lock holders?

Yeah, and then we ran into things like percpu-rwsem where lock/unlock accounting doesn't work as well, but kept the basic design.

#5 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

In reply to Valentin Schneider

Alright, this took longer than I intended, but here's the basic version with the duplicate "runqueue" just being a list rather than actual EEVDF or anything sensible.

I also wasn't actually able to adapt it to work via task_work like I thought I could; flagging the entry to the relevant schedule() calls is too critical.

--------

Subject: [RFC PATCH] sched/fair: only throttle CFS tasks in userspace

The basic idea of this implementation is to maintain duplicate runqueues in each cfs_rq that contain duplicate pointers to sched_entitys which should bypass throttling. Then we can skip throttling cfs_rqs that have any such children, and when we pick inside any not-actually-throttled cfs_rq, we only look at this duplicated list.

"Which tasks should bypass throttling" here is "all schedule() calls that don't set a special flag", but could instead involve the lockdep markers (except for the problem of percpu-rwsem and similar) or explicit flags around syscalls and faults, or something else.

This approach avoids any O(tasks) loops, but leaves partially-throttled cfs_rqs still contributing their full h_nr_running to their parents, which might result in worse balancing. Also it adds more (generally still small) overhead to the common enqueue/dequeue/pick paths.

The very basic debug test added is to run a cpusoaker and "cat /sys/kernel/debug/sched_locked_spin" pinned to the same cpu in the same cgroup with a quota < 1 cpu.

---
 include/linux/sched.h |   7 ++
 kernel/entry/common.c |   2 +-
 kernel/entry/kvm.c    |   2 +-
 kernel/sched/core.c   |  20 ++++
 kernel/sched/debug.c  |  28 +++++
 kernel/sched/fair.c   | 232 ++++++++++++++++++++++++++++++++++++++++--
 kernel/sched/sched.h  |   3 +
 7 files changed, 281 insertions(+), 13 deletions(-)
include/linux/sched.h +7 -0
index 8d258162deb0a..3e9e0e308d66d 100644
@@ -301,10 +301,11 @@ extern long schedule_timeout(long timeout);
 extern long schedule_timeout_interruptible(long timeout);
 extern long schedule_timeout_killable(long timeout);
 extern long schedule_timeout_uninterruptible(long timeout);
 extern long schedule_timeout_idle(long timeout);
 asmlinkage void schedule(void);
+asmlinkage void schedule_usermode(void);
 extern void schedule_preempt_disabled(void);
 asmlinkage void preempt_schedule_irq(void);
 #ifdef CONFIG_PREEMPT_RT
  extern void schedule_rtlock(void);
 #endif
@@ -576,10 +577,13 @@ struct sched_entity {
 	struct cfs_rq			*cfs_rq;
 	/* rq "owned" by this entity/group: */
 	struct cfs_rq			*my_q;
 	/* cached value of my_q->h_nr_running */
 	unsigned long			runnable_weight;
+#ifdef CONFIG_CFS_BANDWIDTH
+	struct list_head		kernel_node;
+#endif
 #endif
 
 #ifdef CONFIG_SMP
 	/*
 	 * Per entity load average tracking.
@@ -1562,10 +1566,13 @@ struct task_struct {
 
 #ifdef CONFIG_USER_EVENTS
 	struct user_event_mm		*user_event_mm;
 #endif
 
+#ifdef CONFIG_CFS_BANDWIDTH
+	atomic_t			in_return_to_user;
+#endif
 	/*
 	 * New fields for task_struct should be added above here, so that
 	 * they are included in the randomized portion of task_struct.
 	 */
 	randomized_struct_fields_end
kernel/entry/common.c +1 -1
index d7ee4bc3f2ba3..008c4cf7c09a1 100644
@@ -154,11 +154,11 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
 	while (ti_work & EXIT_TO_USER_MODE_WORK) {
 
 		local_irq_enable_exit_to_user(ti_work);
 
 		if (ti_work & _TIF_NEED_RESCHED)
-			schedule();
+			schedule_usermode(); /* TODO: also all of the arch/* loops that don't use this yet */
 
 		if (ti_work & _TIF_UPROBE)
 			uprobe_notify_resume(regs);
 
 		if (ti_work & _TIF_PATCH_PENDING)
kernel/entry/kvm.c +1 -1
index 2e0f75bcb7fd1..fc4b73de07539 100644
@@ -12,11 +12,11 @@ static int xfer_to_guest_mode_work(struct kvm_vcpu *vcpu, unsigned long ti_work)
 			kvm_handle_signal_exit(vcpu);
 			return -EINTR;
 		}
 
 		if (ti_work & _TIF_NEED_RESCHED)
-			schedule();
+			schedule_usermode();
 
 		if (ti_work & _TIF_NOTIFY_RESUME)
 			resume_user_mode_work(NULL);
 
 		ret = arch_xfer_to_guest_mode_handle_work(vcpu, ti_work);
kernel/sched/core.c +20 -0
index db4be4921e7f0..a7c028fad5a89 100644
@@ -4527,10 +4527,14 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 	INIT_LIST_HEAD(&p->se.group_node);
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	p->se.cfs_rq			= NULL;
 #endif
+#ifdef CONFIG_CFS_BANDWIDTH
+	INIT_LIST_HEAD(&p->se.kernel_node);
+	atomic_set(&p->in_return_to_user, 0);
+#endif
 
 #ifdef CONFIG_SCHEDSTATS
 	/* Even if schedstat is disabled, there should not be garbage */
 	memset(&p->stats, 0, sizeof(p->stats));
 #endif
@@ -6816,10 +6820,26 @@ asmlinkage __visible void __sched schedule(void)
 	__schedule_loop(SM_NONE);
 	sched_update_worker(tsk);
 }
 EXPORT_SYMBOL(schedule);
 
+asmlinkage __visible void __sched schedule_usermode(void)
+{
+#ifdef CONFIG_CFS_BANDWIDTH
+	/*
+	 * This is only atomic because of this simple implementation. We could
+	 * do something with an SM_USER to avoid other-cpu scheduler operations
+	 * racing against these writes.
+	 */
+	atomic_set(&current->in_return_to_user, true);
+	schedule();
+	atomic_set(&current->in_return_to_user, false);
+#else
+	schedule();
+#endif
+}
+
 /*
  * synchronize_rcu_tasks() makes sure that no task is stuck in preempted
  * state (have scheduled out non-voluntarily) by making sure that all
  * tasks have either left the run queue or have gone into user space.
  * As idle tasks do not do either, they must not ever be preempted
kernel/sched/debug.c +28 -0
index 168eecc209b49..5d9d011a0e6fb 100644
@@ -317,10 +317,36 @@ static const struct file_operations sched_verbose_fops = {
 	.write =        sched_verbose_write,
 	.open =         simple_open,
 	.llseek =       default_llseek,
 };
 
+static DEFINE_MUTEX(sched_debug_spin_mutex);
+static int sched_debug_spin_show(struct seq_file *m, void *v) {
+	int count;
+	mutex_lock(&sched_debug_spin_mutex);
+	for (count = 0; count < 1000; count++) {
+		u64 start2;
+		start2 = jiffies;
+		while (jiffies == start2)
+			cpu_relax();
+		schedule();
+	}
+	mutex_unlock(&sched_debug_spin_mutex);
+	return 0;
+}
+static int sched_debug_spin_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, sched_debug_spin_show, NULL);
+}
+
+static const struct file_operations sched_debug_spin_fops = {
+	.open		= sched_debug_spin_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
 static const struct seq_operations sched_debug_sops;
 
 static int sched_debug_open(struct inode *inode, struct file *filp)
 {
 	return seq_open(filp, &sched_debug_sops);
@@ -372,10 +398,12 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
 #endif
 
 	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
 
+	debugfs_create_file("sched_locked_spin", 0444, NULL, NULL,
+			    &sched_debug_spin_fops);
 	return 0;
 }
 late_initcall(sched_init_debug);
 
 #ifdef CONFIG_SMP
kernel/sched/fair.c +221 -11

547 lines not on this page · raw message

kernel/sched/sched.h +3 -0
index e58a54bda77de..5fee806cc4538 100644
@@ -578,10 +578,11 @@ struct cfs_rq {
 	u64			min_vruntime_copy;
 #endif
 
 	struct rb_root_cached	tasks_timeline;
 
+
 	/*
 	 * 'curr' points to currently running entity on this cfs_rq.
 	 * It is set to NULL otherwise (i.e when none are currently running).
 	 */
 	struct sched_entity	*curr;
@@ -656,12 +657,14 @@ struct cfs_rq {
 	u64			throttled_clock_pelt_time;
 	u64			throttled_clock_self;
 	u64			throttled_clock_self_time;
 	int			throttled;
 	int			throttle_count;
+	int			h_kernel_running;
 	struct list_head	throttled_list;
 	struct list_head	throttled_csd_list;
+	struct list_head	kernel_children;;
 #endif /* CONFIG_CFS_BANDWIDTH */
 #endif /* CONFIG_FAIR_GROUP_SCHED */
 };
 
 static inline int rt_bandwidth_enabled(void)
Signature · 1 line
2.43.0.472.g3155946c3a-goog
author reviewer
#6 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

Quoted 8 lines · Benjamin Segall

Alright, this took longer than I intended, but here's the basic version with the duplicate "runqueue" just being a list rather than actual EEVDF or anything sensible.

I also wasn't actually able to adapt it to work via task_work like I thought I could; flagging the entry to the relevant schedule() calls is too critical.

Thanks for sharing this! Have a first set of comments.

Quoted 15 lines

--------

Subject: [RFC PATCH] sched/fair: only throttle CFS tasks in userspace

The basic idea of this implementation is to maintain duplicate runqueues in each cfs_rq that contain duplicate pointers to sched_entitys which should bypass throttling. Then we can skip throttling cfs_rqs that have any such children, and when we pick inside any not-actually-throttled cfs_rq, we only look at this duplicated list.

"Which tasks should bypass throttling" here is "all schedule() calls that don't set a special flag", but could instead involve the lockdep markers (except for the problem of percpu-rwsem and similar) or explicit flags around syscalls and faults, or something else.

Quoted 8 lines

This approach avoids any O(tasks) loops, but leaves partially-throttled cfs_rqs still contributing their full h_nr_running to their parents, which might result in worse balancing. Also it adds more (generally still small) overhead to the common enqueue/dequeue/pick paths.

The very basic debug test added is to run a cpusoaker and "cat /sys/kernel/debug/sched_locked_spin" pinned to the same cpu in the same cgroup with a quota < 1 cpu.

So if I'm trying to compare notes:

The dual tree:
- Can actually throttle cfs_rq's once all tasks are in userspace
- Adds (light-ish) overhead to enqueue/dequeue
- Doesn't keep *nr_running updated when not fully throttled

Whereas dequeue via task_work:
- Can never fully throttle a cfs_rq
- Adds (not so light) overhead to unthrottle
- Keeps *nr_running updated

My thoughts right now are that there might be a way to mix both worlds: go with the dual tree, but subtract from h_nr_running at dequeue_kernel() (sort of doing the count update in throttle_cfs_rq() early) and keep track of in-user tasks via handle_kernel_task_prev() to add this back when unthrottling a not-fully-throttled cfs_rq. I need to actually think about it though :-)

kernel/entry/common.c static unsigned long exit_to_user_mode_loop(struct pt_regs *regs, · line 159

Quoted 8 lines
kernel/entry/common.c +1 -1
@@ -154,11 +154,11 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
      while (ti_work & EXIT_TO_USER_MODE_WORK) {
 
              local_irq_enable_exit_to_user(ti_work);
 
              if (ti_work & _TIF_NEED_RESCHED)
-			schedule();
+			schedule_usermode(); /* TODO: also all of the arch/* loops that don't use this yet */

^^
Interestingly -Werror=comment trips on this

Quoted 24 lines
kernel/entry/common.c +6 -5
              if (ti_work & _TIF_UPROBE)
                      uprobe_notify_resume(regs);
 
              if (ti_work & _TIF_PATCH_PENDING)
@@ -5416,18 +5426,38 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
      se->prev_sum_exec_runtime = se->sum_exec_runtime;
 }
 
 /*
  * Pick the next process, keeping these things in mind, in this order:
- * 1) keep things fair between processes/task groups
- * 2) pick the "next" process, since someone really wants that to run
- * 3) pick the "last" process, for cache locality
- * 4) do not run the "skip" process, if something else is available
+ * 1) If we're inside a throttled cfs_rq, only pick threads in the kernel
+ * 2) keep things fair between processes/task groups
+ * 3) pick the "next" process, since someone really wants that to run
+ * 4) pick the "last" process, for cache locality
+ * 5) do not run the "skip" process, if something else is available
  */
 static struct sched_entity *
-pick_next_entity(struct cfs_rq *cfs_rq)
+pick_next_entity(struct cfs_rq *cfs_rq, bool throttled)

kernel/entry/common.c set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 5448

Quoted 10 lines
kernel/entry/common.c +9 -0
 {
+#ifdef CONFIG_CFS_BANDWIDTH
+	/*
+	 * TODO: This might trigger, I'm not sure/don't remember. Regardless,
+	 * while we do not explicitly handle the case where h_kernel_running
+	 * goes to 0, we will call account/check_cfs_rq_runtime at worst in
+	 * entity_tick and notice that we can now properly do the full
+	 * throttle_cfs_rq.
+	 */
+	WARN_ON_ONCE(list_empty(&cfs_rq->kernel_children));

FWIW this triggers pretty much immediately in my environment (buildroot over QEMU).

Quoted 14 lines
kernel/sched/fair.c +3 -0
@@ -5712,16 +5748,26 @@ static int tg_throttle_down(struct task_group *tg, void *data)
      cfs_rq->throttle_count++;
 
      return 0;
 }
 
+static void enqueue_kernel(struct cfs_rq *cfs_rq, struct sched_entity *se, int count);
+static void dequeue_kernel(struct cfs_rq *cfs_rq, struct sched_entity *se, int count);
+
 static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
 {
      struct rq *rq = rq_of(cfs_rq);
      struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
      struct sched_entity *se;

kernel/sched/fair.c static int tg_throttle_down(struct task_group *tg, void *data) · line 5769

Quoted 10 lines
kernel/sched/fair.c +8 -1
-	long task_delta, idle_task_delta, dequeue = 1;
+	long task_delta, idle_task_delta, kernel_delta, dequeue = 1;
+
+	/*
+	 * We don't actually throttle, though account() will have made sure to
+	 * resched us so that we pick into a kernel task.
+	 */
+	if (cfs_rq->h_kernel_running)
+		return false;
 

So as long as a cfs_rq has at least one task executing in the kernel, we keep its *nr_running intact. We do throttle the whole thing once all the tasks are in userspace (or about to be anyway), and unthrottle_on_enqueue() undoes that.

Quoted 23 lines
kernel/sched/fair.c +17 -0
@@ -8316,15 +8471,44 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int
 
 preempt:
      resched_curr(rq);
 }
 
+static void handle_kernel_task_prev(struct task_struct *prev)
+{
+#ifdef CONFIG_CFS_BANDWIDTH
+	struct sched_entity *se = &prev->se;
+	bool new_kernel = is_kernel_task(prev);
+	bool prev_kernel = !list_empty(&se->kernel_node);
+	/*
+	 * These extra loops are bad and against the whole point of the merged
+	 * PNT, but it's a pain to merge, particularly since we want it to occur
+	 * before check_cfs_runtime.
+	 */
+	if (prev_kernel && !new_kernel) {
+		WARN_ON_ONCE(!se->on_rq); /* dequeue should have removed us */
+		for_each_sched_entity(se) {
+			dequeue_kernel(cfs_rq_of(se), se, 1);
+			if (cfs_rq_throttled(cfs_rq_of(se)))
+				break;

kernel/sched/fair.c static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int · line 8502

Quoted 10 lines
kernel/sched/fair.c +10 -0
+		}
+	} else if (!prev_kernel && new_kernel && se->on_rq) {
+		for_each_sched_entity(se) {
+			enqueue_kernel(cfs_rq_of(se), se, 1);
+			if (cfs_rq_throttled(cfs_rq_of(se)))
+				break;
+		}
+	}
+#endif
+}

I'm trying to grok what the implications of a second tasks_timeline would be on picking - we'd want a RBtree update equivalent to put_prev_entity()'s __enqueue_entity(), but that second tree doesn't follow the same enqueue/dequeue cycle as the first one. Would a __dequeue_entity() + __enqueue_entity() to force a rebalance make sense? That'd also take care of updating the min_vruntime.

#7 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

Quoted 43 lines · Valentin Schneider
Quoted 8 lines · Benjamin Segall

Alright, this took longer than I intended, but here's the basic version with the duplicate "runqueue" just being a list rather than actual EEVDF or anything sensible.

I also wasn't actually able to adapt it to work via task_work like I thought I could; flagging the entry to the relevant schedule() calls is too critical.

Thanks for sharing this! Have a first set of comments.

Quoted 15 lines

--------

Subject: [RFC PATCH] sched/fair: only throttle CFS tasks in userspace

The basic idea of this implementation is to maintain duplicate runqueues in each cfs_rq that contain duplicate pointers to sched_entitys which should bypass throttling. Then we can skip throttling cfs_rqs that have any such children, and when we pick inside any not-actually-throttled cfs_rq, we only look at this duplicated list.

"Which tasks should bypass throttling" here is "all schedule() calls that don't set a special flag", but could instead involve the lockdep markers (except for the problem of percpu-rwsem and similar) or explicit flags around syscalls and faults, or something else.

Quoted 8 lines

This approach avoids any O(tasks) loops, but leaves partially-throttled cfs_rqs still contributing their full h_nr_running to their parents, which might result in worse balancing. Also it adds more (generally still small) overhead to the common enqueue/dequeue/pick paths.

The very basic debug test added is to run a cpusoaker and "cat /sys/kernel/debug/sched_locked_spin" pinned to the same cpu in the same cgroup with a quota < 1 cpu.

So if I'm trying to compare notes:

The dual tree:
- Can actually throttle cfs_rq's once all tasks are in userspace
- Adds (light-ish) overhead to enqueue/dequeue
- Doesn't keep *nr_running updated when not fully throttled

Quoted 4 lines

Whereas dequeue via task_work:
- Can never fully throttle a cfs_rq
- Adds (not so light) overhead to unthrottle
- Keeps *nr_running updated

Yeah, this sounds right. (Though for the task_work version once all the tasks are throttled, the cfs_rq is dequeued like normal, so it doesn't seem like a big deal to me)

Quoted 7 lines

My thoughts right now are that there might be a way to mix both worlds: go with the dual tree, but subtract from h_nr_running at dequeue_kernel() (sort of doing the count update in throttle_cfs_rq() early) and keep track of in-user tasks via handle_kernel_task_prev() to add this back when unthrottling a not-fully-throttled cfs_rq. I need to actually think about it though :-)

Probably, but I had tons of trouble getting the accounting correct as it was :P

Quoted 4 lines
Quoted 4 lines
kernel/sched/fair.c +4 -0
+#ifdef CONFIG_CFS_BANDWIDTH
+	/*
+	 * TODO: This might trigger, I'm not sure/don't remember. Regardless,
+	 * while we do not explicitly handle the case where h_kernel_running
Quoted 8 lines

kernel/sched/fair.c set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 5448

Quoted 5 lines
kernel/sched/fair.c +5 -0
+	 * goes to 0, we will call account/check_cfs_rq_runtime at worst in
+	 * entity_tick and notice that we can now properly do the full
+	 * throttle_cfs_rq.
+	 */
+	WARN_ON_ONCE(list_empty(&cfs_rq->kernel_children));

FWIW this triggers pretty much immediately in my environment (buildroot over QEMU).

Good to know; I feel like this should be avoidable (and definitely should be avoided if it can be), but I might be forgetting some case that's mostly unavoidable. I'll have to poke it when I have more time.

Quoted 35 lines
Quoted 23 lines
kernel/sched/fair.c +17 -0
@@ -8316,15 +8471,44 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int
 
 preempt:
      resched_curr(rq);
 }
 
+static void handle_kernel_task_prev(struct task_struct *prev)
+{
+#ifdef CONFIG_CFS_BANDWIDTH
+	struct sched_entity *se = &prev->se;
+	bool new_kernel = is_kernel_task(prev);
+	bool prev_kernel = !list_empty(&se->kernel_node);
+	/*
+	 * These extra loops are bad and against the whole point of the merged
+	 * PNT, but it's a pain to merge, particularly since we want it to occur
+	 * before check_cfs_runtime.
+	 */
+	if (prev_kernel && !new_kernel) {
+		WARN_ON_ONCE(!se->on_rq); /* dequeue should have removed us */
+		for_each_sched_entity(se) {
+			dequeue_kernel(cfs_rq_of(se), se, 1);
+			if (cfs_rq_throttled(cfs_rq_of(se)))
+				break;

kernel/sched/fair.c static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int · line 8502

Quoted 10 lines
kernel/sched/fair.c +10 -0
+		}
+	} else if (!prev_kernel && new_kernel && se->on_rq) {
+		for_each_sched_entity(se) {
+			enqueue_kernel(cfs_rq_of(se), se, 1);
+			if (cfs_rq_throttled(cfs_rq_of(se)))
+				break;
+		}
+	}
+#endif
+}
Quoted 6 lines

I'm trying to grok what the implications of a second tasks_timeline would be on picking - we'd want a RBtree update equivalent to put_prev_entity()'s __enqueue_entity(), but that second tree doesn't follow the same enqueue/dequeue cycle as the first one. Would a __dequeue_entity() + __enqueue_entity() to force a rebalance make sense? That'd also take care of updating the min_vruntime.

Yeah, the most sensible thing to do would probably be to just have curr not be in the queue, the same as the normal rbtree, and save the "on kernel list" as an on_rq-equivalent bool instead of as !list_empty(kernel_node).

author reviewer
#8 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

Quoted 9 lines · Benjamin Segall
Quoted 8 lines · Valentin Schneider

So if I'm trying to compare notes:

The dual tree:
- Can actually throttle cfs_rq's once all tasks are in userspace
- Adds (light-ish) overhead to enqueue/dequeue
- Doesn't keep *nr_running updated when not fully throttled

Quoted 9 lines
Quoted 4 lines

Whereas dequeue via task_work:
- Can never fully throttle a cfs_rq
- Adds (not so light) overhead to unthrottle
- Keeps *nr_running updated

Yeah, this sounds right. (Though for the task_work version once all the tasks are throttled, the cfs_rq is dequeued like normal, so it doesn't seem like a big deal to me)

In my implementation the cfs_rq's can be dequeued once all of their tasks have been throttled, but the tasks aren't migrated back onto the cfs_rq's until the unthrottle (rather than at the "final" throttle).

That is done because if a random task gets migrated to a throttled & empty cfs_rq, we still need to schedule it 'till it exits the kernel, and we don't want the now-in-userspace task to suddenly become eligible for picking.

That adds these O(ntasks) loops in the unthrottle to re-shape the cfs_rq's. The dual tree approach saves us from doing all that, IIUC unthrottle_on_enqueue() only puts the se's in the task's direct hierarchy back onto the second tree for them to be picked.

Quoted 11 lines
Quoted 7 lines

My thoughts right now are that there might be a way to mix both worlds: go with the dual tree, but subtract from h_nr_running at dequeue_kernel() (sort of doing the count update in throttle_cfs_rq() early) and keep track of in-user tasks via handle_kernel_task_prev() to add this back when unthrottling a not-fully-throttled cfs_rq. I need to actually think about it though :-)

Probably, but I had tons of trouble getting the accounting correct as it was :P

Yeah so did I, cgroups are "fun".

Quoted 11 lines
Quoted 7 lines

I'm trying to grok what the implications of a second tasks_timeline would be on picking - we'd want a RBtree update equivalent to put_prev_entity()'s __enqueue_entity(), but that second tree doesn't follow the same enqueue/dequeue cycle as the first one. Would a __dequeue_entity() + __enqueue_entity() to force a rebalance make sense? That'd also take care of updating the min_vruntime.

Yeah, the most sensible thing to do would probably be to just have curr not be in the queue, the same as the normal rbtree, and save the "on kernel list" as an on_rq-equivalent bool instead of as !list_empty(kernel_node).

Makes sense, I'll have a go at this.

#9 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

In reply to Valentin Schneider
kernel test robot noticed the following build errors:

Hi Valentin,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on tip/sched/core]
[also build test ERROR on tip/master linus/master v6.7-rc4 next-20231205] [cannot apply to tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Valentin-Schneider/sched-fair-Only-throttle-CFS-tasks-on-return-to-userspace/20231201-001908
base:   tip/sched/core
patch link:    https://lore.kernel.org/r/20231130161245.3894682-2-vschneid%40redhat.com
patch subject: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
config: arm-mvebu_v5_defconfig (https://download.01.org/0day-ci/archive/20231206/[email protected]/config)
compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231206/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

kernel/sched/fair.c:8575:30: error: no member named 'in_throttle_limbo' in 'struct cfs_rq'

           if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
                        ~~~~~~~~~~~~~  ^
   include/linux/compiler.h:77:42: note: expanded from macro 'unlikely'
   # define unlikely(x)    __builtin_expect(!!(x), 0)
                                               ^
>> kernel/sched/fair.c:8575:52: error: call to undeclared function 'task_has_throttle_work'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
           if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
                                                             ^
   kernel/sched/fair.c:8575:52: note: did you mean 'init_cfs_throttle_work'?
   kernel/sched/sched.h:2457:13: note: 'init_cfs_throttle_work' declared here
   extern void init_cfs_throttle_work(struct task_struct *p);
               ^
   kernel/sched/fair.c:8576:24: error: no member named 'sched_throttle_work' in 'struct task_struct'
                   task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
                                     ~  ^
   kernel/sched/fair.c:13197:6: warning: no previous prototype for function 'free_fair_sched_group' [-Wmissing-prototypes]
   void free_fair_sched_group(struct task_group *tg) { }
        ^
   kernel/sched/fair.c:13197:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
   void free_fair_sched_group(struct task_group *tg) { }
   ^
   static 
   kernel/sched/fair.c:13199:5: warning: no previous prototype for function 'alloc_fair_sched_group' [-Wmissing-prototypes]
   int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
       ^
   kernel/sched/fair.c:13199:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
   int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
   ^
   static 
   kernel/sched/fair.c:13204:6: warning: no previous prototype for function 'online_fair_sched_group' [-Wmissing-prototypes]
   void online_fair_sched_group(struct task_group *tg) { }
        ^
   kernel/sched/fair.c:13204:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
   void online_fair_sched_group(struct task_group *tg) { }
   ^
   static 
   kernel/sched/fair.c:13206:6: warning: no previous prototype for function 'unregister_fair_sched_group' [-Wmissing-prototypes]
   void unregister_fair_sched_group(struct task_group *tg) { }
        ^

kernel/sched/fair.c:13206:1: note: declare 'static' if the function is not intended to be used outside of this translation unit void unregister_fair_sched_group(struct task_group *tg) { }
^
static
4 warnings and 3 errors generated.

vim +/task_has_throttle_work +8575 kernel/sched/fair.c

8471
8472 struct task_struct *
8473 pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
8474 {
8475 struct cfs_rq *cfs_rq = &rq->cfs;
8476 struct sched_entity *se;
8477 struct task_struct *p;
8478 int new_tasks;
8479
8480 again:
8481 if (!sched_fair_runnable(rq))
8482 goto idle;
8483
8484 #ifdef CONFIG_FAIR_GROUP_SCHED
8485 if (!prev || prev->sched_class != &fair_sched_class)
8486 goto simple;
8487
8488 /*
8489 * Because of the set_next_buddy() in dequeue_task_fair() it is rather
8490 * likely that a next task is from the same cgroup as the current.
8491 *
8492 * Therefore attempt to avoid putting and setting the entire cgroup
8493 * hierarchy, only change the part that actually changes.
8494 */
8495
8496 do {
8497 struct sched_entity *curr = cfs_rq->curr;
8498
8499 /*
8500 * Since we got here without doing put_prev_entity() we also
8501 * have to consider cfs_rq->curr. If it is still a runnable
8502 * entity, update_curr() will update its vruntime, otherwise
8503 * forget we've ever seen it.
8504 */
8505 if (curr) {
8506 if (curr->on_rq)
8507 update_curr(cfs_rq);
8508 else
8509 curr = NULL;
8510
8511 /*
8512 * This call to check_cfs_rq_runtime() will do the
8513 * throttle and dequeue its entity in the parent(s).
8514 * Therefore the nr_running test will indeed
8515 * be correct.
8516 */
8517 if (unlikely(check_cfs_rq_runtime(cfs_rq))) {
8518 cfs_rq = &rq->cfs;
8519
8520 if (!cfs_rq->nr_running)
8521 goto idle;
8522
8523 goto simple;
8524 }
8525 }
8526
8527 se = pick_next_entity(cfs_rq);
8528 cfs_rq = group_cfs_rq(se);
8529 } while (cfs_rq);
8530
8531 p = task_of(se);
8532
8533 if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
8534 task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
8535 /*
8536 * Since we haven't yet done put_prev_entity and if the selected task
8537 * is a different task than we started out with, try and touch the
8538 * least amount of cfs_rqs.
8539 */
8540 if (prev != p) {
8541 struct sched_entity *pse = &prev->se;
8542
8543 while (!(cfs_rq = is_same_group(se, pse))) {
8544 int se_depth = se->depth;
8545 int pse_depth = pse->depth;
8546
8547 if (se_depth <= pse_depth) {
8548 put_prev_entity(cfs_rq_of(pse), pse);
8549 pse = parent_entity(pse);
8550 }
8551 if (se_depth >= pse_depth) {
8552 set_next_entity(cfs_rq_of(se), se);
8553 se = parent_entity(se);
8554 }
8555 }
8556
8557 put_prev_entity(cfs_rq, pse);
8558 set_next_entity(cfs_rq, se);
8559 }
8560
8561 goto done;
8562 simple:
8563 #endif
8564 if (prev)
8565 put_prev_task(rq, prev);
8566
8567 do {
8568 se = pick_next_entity(cfs_rq);
8569 set_next_entity(cfs_rq, se);
8570 cfs_rq = group_cfs_rq(se);
8571 } while (cfs_rq);
8572
8573 p = task_of(se);
8574

> 8575		if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))

8576 task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
8577

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#10 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

In reply to Valentin Schneider
kernel test robot noticed the following build errors:

Hi Valentin,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on tip/sched/core]
[also build test ERROR on tip/master linus/master v6.7-rc4 next-20231205] [cannot apply to tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Valentin-Schneider/sched-fair-Only-throttle-CFS-tasks-on-return-to-userspace/20231201-001908
base:   tip/sched/core
patch link:    https://lore.kernel.org/r/20231130161245.3894682-2-vschneid%40redhat.com
patch subject: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
config: arm-randconfig-002-20231202 (https://download.01.org/0day-ci/archive/20231206/[email protected]/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231206/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

kernel/sched/fair.c:8575:30: error: no member named 'in_throttle_limbo' in 'struct cfs_rq'

    8575 |         if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
         |                      ~~~~~~~~~~~~~  ^
   include/linux/compiler.h:77:42: note: expanded from macro 'unlikely'
      77 | # define unlikely(x)    __builtin_expect(!!(x), 0)
         |                                             ^
   kernel/sched/fair.c:8575:52: error: call to undeclared function 'task_has_throttle_work'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    8575 |         if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
         |                                                           ^
   kernel/sched/fair.c:8575:52: note: did you mean 'init_cfs_throttle_work'?
   kernel/sched/sched.h:2457:13: note: 'init_cfs_throttle_work' declared here
    2457 | extern void init_cfs_throttle_work(struct task_struct *p);
         |             ^
>> kernel/sched/fair.c:8576:24: error: no member named 'sched_throttle_work' in 'struct task_struct'
    8576 |                 task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
         |                                   ~  ^
   kernel/sched/fair.c:13197:6: warning: no previous prototype for function 'free_fair_sched_group' [-Wmissing-prototypes]
    13197 | void free_fair_sched_group(struct task_group *tg) { }
          |      ^
   kernel/sched/fair.c:13197:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
    13197 | void free_fair_sched_group(struct task_group *tg) { }
          | ^
          | static 
   kernel/sched/fair.c:13199:5: warning: no previous prototype for function 'alloc_fair_sched_group' [-Wmissing-prototypes]
    13199 | int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
          |     ^
   kernel/sched/fair.c:13199:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
    13199 | int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
          | ^
          | static 
   kernel/sched/fair.c:13204:6: warning: no previous prototype for function 'online_fair_sched_group' [-Wmissing-prototypes]
    13204 | void online_fair_sched_group(struct task_group *tg) { }
          |      ^
   kernel/sched/fair.c:13204:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
    13204 | void online_fair_sched_group(struct task_group *tg) { }
          | ^
          | static 
   kernel/sched/fair.c:13206:6: warning: no previous prototype for function 'unregister_fair_sched_group' [-Wmissing-prototypes]
    13206 | void unregister_fair_sched_group(struct task_group *tg) { }
          |      ^
   kernel/sched/fair.c:13206:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
    13206 | void unregister_fair_sched_group(struct task_group *tg) { }
          | ^
          | static 

4 warnings and 3 errors generated.

vim +8576 kernel/sched/fair.c

8471
8472 struct task_struct *
8473 pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
8474 {
8475 struct cfs_rq *cfs_rq = &rq->cfs;
8476 struct sched_entity *se;
8477 struct task_struct *p;
8478 int new_tasks;
8479
8480 again:
8481 if (!sched_fair_runnable(rq))
8482 goto idle;
8483
8484 #ifdef CONFIG_FAIR_GROUP_SCHED
8485 if (!prev || prev->sched_class != &fair_sched_class)
8486 goto simple;
8487
8488 /*
8489 * Because of the set_next_buddy() in dequeue_task_fair() it is rather
8490 * likely that a next task is from the same cgroup as the current.
8491 *
8492 * Therefore attempt to avoid putting and setting the entire cgroup
8493 * hierarchy, only change the part that actually changes.
8494 */
8495
8496 do {
8497 struct sched_entity *curr = cfs_rq->curr;
8498
8499 /*
8500 * Since we got here without doing put_prev_entity() we also
8501 * have to consider cfs_rq->curr. If it is still a runnable
8502 * entity, update_curr() will update its vruntime, otherwise
8503 * forget we've ever seen it.
8504 */
8505 if (curr) {
8506 if (curr->on_rq)
8507 update_curr(cfs_rq);
8508 else
8509 curr = NULL;
8510
8511 /*
8512 * This call to check_cfs_rq_runtime() will do the
8513 * throttle and dequeue its entity in the parent(s).
8514 * Therefore the nr_running test will indeed
8515 * be correct.
8516 */
8517 if (unlikely(check_cfs_rq_runtime(cfs_rq))) {
8518 cfs_rq = &rq->cfs;
8519
8520 if (!cfs_rq->nr_running)
8521 goto idle;
8522
8523 goto simple;
8524 }
8525 }
8526
8527 se = pick_next_entity(cfs_rq);
8528 cfs_rq = group_cfs_rq(se);
8529 } while (cfs_rq);
8530
8531 p = task_of(se);
8532
8533 if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
8534 task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
8535 /*
8536 * Since we haven't yet done put_prev_entity and if the selected task
8537 * is a different task than we started out with, try and touch the
8538 * least amount of cfs_rqs.
8539 */
8540 if (prev != p) {
8541 struct sched_entity *pse = &prev->se;
8542
8543 while (!(cfs_rq = is_same_group(se, pse))) {
8544 int se_depth = se->depth;
8545 int pse_depth = pse->depth;
8546
8547 if (se_depth <= pse_depth) {
8548 put_prev_entity(cfs_rq_of(pse), pse);
8549 pse = parent_entity(pse);
8550 }
8551 if (se_depth >= pse_depth) {
8552 set_next_entity(cfs_rq_of(se), se);
8553 se = parent_entity(se);
8554 }
8555 }
8556
8557 put_prev_entity(cfs_rq, pse);
8558 set_next_entity(cfs_rq, se);
8559 }
8560
8561 goto done;
8562 simple:
8563 #endif
8564 if (prev)
8565 put_prev_task(rq, prev);
8566
8567 do {
8568 se = pick_next_entity(cfs_rq);
8569 set_next_entity(cfs_rq, se);
8570 cfs_rq = group_cfs_rq(se);
8571 } while (cfs_rq);
8572
8573 p = task_of(se);
8574
8575 if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))

> 8576			task_work_add(p, &p->sched_throttle_work, TWA_RESUME);

8577

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#11 lore raw

Re: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace

In reply to Valentin Schneider
kernel test robot noticed the following build warnings:

Hi Valentin,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on tip/sched/core]
[also build test WARNING on tip/master linus/master v6.7-rc4 next-20231205] [cannot apply to tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Valentin-Schneider/sched-fair-Only-throttle-CFS-tasks-on-return-to-userspace/20231201-001908
base:   tip/sched/core
patch link:    https://lore.kernel.org/r/20231130161245.3894682-2-vschneid%40redhat.com
patch subject: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
config: i386-randconfig-141-20231202 (https://download.01.org/0day-ci/archive/20231206/[email protected]/config)
compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231206/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> kernel/sched/fair.c:5809:6: warning: variable 'top_cfs_rq' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
           if (!qcfs_rq->in_throttle_limbo)
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~
   kernel/sched/fair.c:5860:6: note: uninitialized use occurs here
           if (top_cfs_rq && !top_cfs_rq->throttled_clock)
               ^~~~~~~~~~
   kernel/sched/fair.c:5809:2: note: remove the 'if' if its condition is always false
           if (!qcfs_rq->in_throttle_limbo)
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   kernel/sched/fair.c:5788:37: note: initialize the variable 'top_cfs_rq' to silence this warning
           struct cfs_rq *qcfs_rq, *top_cfs_rq;
                                              ^
                                               = NULL
   kernel/sched/fair.c:533:19: warning: unused function 'max_vruntime' [-Wunused-function]
   static inline u64 max_vruntime(u64 max_vruntime, u64 vruntime)
                     ^

2 warnings generated.

vim +5809 kernel/sched/fair.c

5782
5783 static void throttle_cfs_rq_work(struct callback_head *work)
5784 {
5785 struct task_struct *p = container_of(work, struct task_struct, sched_throttle_work);
5786 struct sched_entity *se;
5787 struct rq *rq;
5788 struct cfs_rq *qcfs_rq, *top_cfs_rq;
5789 long task_delta, idle_task_delta;
5790
5791 WARN_ON_ONCE(p != current);
5792
5793 /*
5794 * If task is exiting, then there won't be a return to userspace, so we
5795 * don't have to bother with any of this.
5796 */
5797 if ((p->flags & PF_EXITING))
5798 return;
5799
5800 CLASS(task_rq_lock, rq_guard)(p);
5801 rq = rq_guard.rq;
5802 se = &p->se;
5803 qcfs_rq = cfs_rq_of(se);
5804
5805 /*
5806 * If not in limbo, then either replenish has happened or this task got
5807 * migrated out of the throttled cfs_rq, move along
5808 */

> 5809		if (!qcfs_rq->in_throttle_limbo)

5810 goto done;
5811
5812 update_rq_clock(rq);
5813
5814 list_add(&p->throttle_node, &qcfs_rq->throttled_limbo_list);
5815 task_delta = 1;
5816 idle_task_delta = cfs_rq_is_idle(qcfs_rq) ? 1 : 0;
5817
5818 for_each_sched_entity(se) {
5819 qcfs_rq = cfs_rq_of(se);
5820
5821 if (!se->on_rq)
5822 goto done;
5823
5824 dequeue_entity(qcfs_rq, se, DEQUEUE_SLEEP);
5825 qcfs_rq->h_nr_running -= task_delta;
5826 qcfs_rq->idle_h_nr_running -= idle_task_delta;
5827
5828 if (qcfs_rq->in_throttle_limbo)
5829 top_cfs_rq = qcfs_rq;
5830
5831 if (qcfs_rq->load.weight) {
5832 /* Avoid re-evaluating load for this entity: */
5833 se = parent_entity(se);
5834 break;
5835 }
5836 }
5837
5838 for_each_sched_entity(se) {
5839 qcfs_rq = cfs_rq_of(se);
5840 /* throttled entity or throttle-on-deactivate */
5841 if (!se->on_rq)
5842 goto throttle_done;
5843
5844 update_load_avg(qcfs_rq, se, 0);
5845 se_update_runnable(se);
5846 qcfs_rq->h_nr_running -= task_delta;
5847 qcfs_rq->h_nr_running -= idle_task_delta;
5848 }
5849
5850 throttle_done:
5851 /* At this point se is NULL and we are at root level*/
5852 sub_nr_running(rq, 1);
5853 done:
5854 p->sched_throttle_work.next = &p->sched_throttle_work;
5855 /*
5856 * XXX: this assumes the highest in_throttle_limbo cfs_rq in the
5857 * hierarchy is the one which got throttled, which might not always
5858 * be the case.
5859 */
5860 if (top_cfs_rq && !top_cfs_rq->throttled_clock)
5861 top_cfs_rq->throttled_clock = rq_clock(rq);
5862
5863 resched_curr(rq);
5864 }
5865

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author reviewer
#12 lore raw

RFC PATCH 2/2 sched/fair: Repurpose cfs_rq_throttled()

In reply to Valentin Schneider

cfs_rq->throttled is now never set, as cfs_rq's are never fully throttled but rather stay in limbo as tasks are slowly plucked out of them.

Get rid of cfs_rq->throttled, and repurpose cfs_rq_throttled() to use the limbo meaning.

---
 kernel/sched/fair.c  | 57 ++++++--------------------------------------
 kernel/sched/sched.h |  3 +--
 2 files changed, 8 insertions(+), 52 deletions(-)
kernel/sched/fair.c +7 -50
index 27aee13e7ccd9..fd3a0c388fabd 100644
@@ -5497,7 +5497,7 @@ static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
 	if (likely(cfs_rq->runtime_remaining > 0))
 		return;
 
-	if (cfs_rq->throttled || cfs_rq->in_throttle_limbo)
+	if (cfs_rq->in_throttle_limbo)
 		return;
 	/*
 	 * if we're unable to extend our runtime we resched so that the active
@@ -5518,7 +5518,7 @@ void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
 
 static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq)
 {
-	return cfs_bandwidth_used() && cfs_rq->throttled;
+	return cfs_bandwidth_used() && cfs_rq->in_throttle_limbo;
 }
 
 /* check whether cfs_rq, or any parent, is throttled */
@@ -5848,10 +5848,6 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
 
 		qcfs_rq->h_nr_running += task_delta;
 		qcfs_rq->idle_h_nr_running += idle_task_delta;
-
-		/* end evaluation on encountering a throttled cfs_rq */
-		if (cfs_rq_throttled(qcfs_rq))
-			goto unthrottle_throttle;
 	}
 
 	for_each_sched_entity(se) {
@@ -5862,10 +5858,6 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
 
 		qcfs_rq->h_nr_running += task_delta;
 		qcfs_rq->idle_h_nr_running += idle_task_delta;
-
-		/* end evaluation on encountering a throttled cfs_rq */
-		if (cfs_rq_throttled(qcfs_rq))
-			goto unthrottle_throttle;
 	}
 
 	/* At this point se is NULL and we are at root level*/
@@ -5909,7 +5901,7 @@ static void __cfsb_csd_unthrottle(void *arg)
 				 throttled_csd_list) {
 		list_del_init(&cursor->throttled_csd_list);
 
-		if (cfs_rq_throttled(cursor) || cursor->in_throttle_limbo)
+		if (cfs_rq_throttled(cursor))
 			unthrottle_cfs_rq(cursor);
 	}
 
@@ -5949,7 +5941,7 @@ static void unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq)
 {
 	lockdep_assert_rq_held(rq_of(cfs_rq));
 
-	if (SCHED_WARN_ON(!(cfs_rq_throttled(cfs_rq) || cfs_rq->in_throttle_limbo) ||
+	if (SCHED_WARN_ON(!cfs_rq_throttled(cfs_rq) ||
 	    cfs_rq->runtime_remaining <= 0))
 		return;
 
@@ -5982,7 +5974,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
 		 * waiting for tasks to exit the kernel. In this case we still
 		 * want to replenish.
 		 */
-		if (!cfs_rq_throttled(cfs_rq) && !cfs_rq->in_throttle_limbo)
+		if (!cfs_rq_throttled(cfs_rq))
 			goto next;
 
 		/* Already queued for async unthrottle */
@@ -6031,7 +6023,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
 
 		list_del_init(&cfs_rq->throttled_csd_list);
 
-		if (cfs_rq_throttled(cfs_rq) || cfs_rq->in_throttle_limbo)
+		if (cfs_rq_throttled(cfs_rq))
 			unthrottle_cfs_rq(cfs_rq);
 
 		rq_unlock_irqrestore(rq, &rf);
@@ -6230,10 +6222,6 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq)
 	if (!cfs_rq->runtime_enabled || cfs_rq->curr)
 		return;
 
-	/* ensure the group is not already throttled */
-	if (cfs_rq_throttled(cfs_rq))
-		return;
-
 	/* update runtime allocation */
 	account_cfs_rq_runtime(cfs_rq, 0);
 	if (cfs_rq->runtime_remaining <= 0)
@@ -6266,13 +6254,6 @@ static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq)
 	if (likely(!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining > 0))
 		return false;
 
-	/*
-	 * it's possible for a throttled entity to be forced into a running
-	 * state (e.g. set_curr_task), in this case we're finished.
-	 */
-	if (cfs_rq_throttled(cfs_rq))
-		return true;
-
 	return throttle_cfs_rq(cfs_rq);
 }
 
@@ -6705,10 +6686,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 		if (cfs_rq_is_idle(cfs_rq))
 			idle_h_nr_running = 1;
 
-		/* end evaluation on encountering a throttled cfs_rq */
-		if (cfs_rq_throttled(cfs_rq))
-			goto enqueue_throttle;
-
 		flags = ENQUEUE_WAKEUP;
 	}
 
@@ -6724,10 +6701,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 
 		if (cfs_rq_is_idle(cfs_rq))
 			idle_h_nr_running = 1;
-
-		/* end evaluation on encountering a throttled cfs_rq */
-		if (cfs_rq_throttled(cfs_rq))
-			goto enqueue_throttle;
 	}
 
 	/* At this point se is NULL and we are at root level*/
@@ -6750,7 +6723,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 	if (!task_new)
 		update_overutilized_status(rq);
 
-enqueue_throttle:
 	assert_list_leaf_cfs_rq(rq);
 
 	hrtick_update(rq);
@@ -6783,10 +6755,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 		if (cfs_rq_is_idle(cfs_rq))
 			idle_h_nr_running = 1;
 
-		/* end evaluation on encountering a throttled cfs_rq */
-		if (cfs_rq_throttled(cfs_rq))
-			goto dequeue_throttle;
-
 		/* Don't dequeue parent if it has other entities besides us */
 		if (cfs_rq->load.weight) {
 			/* Avoid re-evaluating load for this entity: */
@@ -6815,10 +6783,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 		if (cfs_rq_is_idle(cfs_rq))
 			idle_h_nr_running = 1;
 
-		/* end evaluation on encountering a throttled cfs_rq */
-		if (cfs_rq_throttled(cfs_rq))
-			goto dequeue_throttle;
-
 	}
 
 	/* At this point se is NULL and we are at root level*/
@@ -6828,7 +6792,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 	if (unlikely(!was_sched_idle && sched_idle_rq(rq)))
 		rq->next_balance = jiffies;
 
-dequeue_throttle:
 	util_est_update(&rq->cfs, p, task_sleep);
 	hrtick_update(rq);
 }
@@ -9582,7 +9545,7 @@ static inline int check_misfit_status(struct rq *rq, struct sched_domain *sd)
  * Something like:
  *
  *	{ 0 1 2 3 } { 4 5 6 7 }
- *	        *     * * *
+ *		*     * * *
  *
  * If we were to balance group-wise we'd place two tasks in the first group and
  * two tasks in the second group. Clearly this is undesired as it will overload
@@ -12642,9 +12605,6 @@ static void propagate_entity_cfs_rq(struct sched_entity *se)
 {
 	struct cfs_rq *cfs_rq = cfs_rq_of(se);
 
-	if (cfs_rq_throttled(cfs_rq))
-		return;
-
 	if (!throttled_hierarchy(cfs_rq))
 		list_add_leaf_cfs_rq(cfs_rq);
 
@@ -12656,9 +12616,6 @@ static void propagate_entity_cfs_rq(struct sched_entity *se)
 
 		update_load_avg(cfs_rq, se, UPDATE_TG);
 
-		if (cfs_rq_throttled(cfs_rq))
-			break;
-
 		if (!throttled_hierarchy(cfs_rq))
 			list_add_leaf_cfs_rq(cfs_rq);
 	}
kernel/sched/sched.h +1 -2
index be29154d93898..7f1afee52a776 100644
@@ -631,9 +631,8 @@ struct cfs_rq {
 	u64			throttled_clock_pelt_time;
 	u64			throttled_clock_self;
 	u64			throttled_clock_self_time;
-	int			throttled;
-	int			throttle_count;
 	int                     in_throttle_limbo;
+	int			throttle_count;
 	/* Temp storage for updating the counts during unthrottling */
 	unsigned int            unthrottled_h_nr_running;
 	unsigned int            unthrottled_idle_h_nr_running;
Signature · 1 line
2.41.0