sched/fair: Delay throttling to kernel exit
Subsystems (2)
Valentin Schneider
reviewercover sched/fair: Delay throttling to kernel exit
Hi folks
I apologize for the one-big-patch-to-rule-them-all, I tried splitting but I couldn't get to anything that made sense, so it's one big patch and a small follow-up.
TL;DR of what I'd like to hear about:
o Is there interest in getting this in for !PREEMPT_RT reasons?
o Any ideas on how to make cgroup migration less horrible so we can fully pop out the throttled cfs_rq se's?
Survives a good hour of my testing below on a 4-CPU QEMU system, but I expect the throttled clocks & PELT to be busted.
Testing
=======
setup
+++++
mount -t cgroup -o cpu none /root/cpu
mkdir /root/cpu/cg0
echo 10000 > /root/cpu/cg0/cpu.cfs_period_us
echo 1000 > /root/cpu/cg0/cpu.cfs_quota_us
mkdir /root/cpu/cg0/cg00
mkdir /root/cpu/cg0/cg01
mkdir /root/cpu/cg0/cg00/cg000
mkdir /root/cpu/cg0/cg00/cg001
read.sh
+++++++
while true; do cat /sys/devices/system/cpu/smt/active &>/dev/null; done
repro.sh
++++++++
spawn() {
./read.sh &
PID=$!
echo "Tracing PID${PID}"
echo $PID > $1
}spawn cpu/cg0/tasks
spawn cpu/cg0/tasks
spawn cpu/cg0/tasks
spawn cpu/cg0/tasks
spawn cpu/cg0/cg01/tasks
spawn cpu/cg0/cg00/cg000/tasks
spawn cpu/cg0/cg00/cg001/tasks
sleep 1
kill $(jobs -p)
Valentin Schneider (2):
sched/fair: Only throttle CFS tasks on return to userspace sched/fair: Repurpose cfs_rq_throttled()
include/linux/sched.h | 2 + kernel/sched/core.c | 6 +- kernel/sched/debug.c | 4 +- kernel/sched/fair.c | 296 +++++++++++++++++++++++++++--------------- kernel/sched/sched.h | 8 +- 5 files changed, 204 insertions(+), 112 deletions(-)
--
2.41.0
Valentin Schneider
reviewerRFC PATCH 1/2 sched/fair: Only throttle CFS tasks on return to userspace
As reported in [1], CFS bandwidth throttling is a source of headaches in PREEMPT_RT - generally speaking, a throttled CFS task can hold locks that prevent ksoftirqd from running, which prevents replenishing & unthrottling the cfs_rq of said CFS task.
Peter mentioned that there have been discussions on changing /when/ the throttling happens: rather than have it be done immediately upon updating the runtime statistics and realizing the cfs_rq has depleted its quota, we wait for the task to be about to return to userspace.
I'm not aware of the arguments in favour of this for !PREEMPT_RT, but given [1] I had a look into it. Using this approach means no task can be throttled while holding a kernel lock, which solves the locking dependency issue.
Concept & headaches
===================
The original idea was:
o cfs_rq runs out of runtime, flag it
o Take the tasks out of the cfs_rq one by one as they exit the kernel
o Dequeue the cfs_rq once it is empty of tasks (as if doing "normal" throttling)
o Put the tasks back onto it
o At unthrottle, simply re-enqueue the cfs_rq sched_entity and all is well
Cgroup migration kinda breaks all of that. Consider a task moving to a now-fully-throttled cfs_rq: if we just enqueue it onto the throttled se, then it's the same as throttling it, and if it is holding a problematic lock then we're back to square one.
This means we can never fully throttle a cfs_rq, which is making me not so confident in this approach.
Should/can we consider deferring such migrations to the unthrottle (with something like in affine_move_task())? I think failing the migration is not an option due to e.g. autogroups, and IIUC returning from writing to cgroup/tasks means the migration has happened (like updating a task's affinity), which means this would need to be a blocking wait.
CPU migrations within the same taskgroup are similarly problematic (e.g. moving from a cfs_rq that still has a bit of ->runtime left to one that has run out).
Throttle clocks are also kinda busted in this approach, but they aren't my greatest concern at this point :-)
Implementation
==============
cfs_rq's that deplete their ->runtime get flagged for later throttling via ->in_throttle_limbo, and this is propagated down the hierarchy.
Tasks enqueued in a cfs_rq with ->in_throttle_limbo picked for execution get a task_work added, which lets the actual throttling happen in exit_to_user_mode().
Upon unthrottling, tasks are enqueued back onto their respective cfs_rq. Unlike the previous throttling implementation, cfs_rq's can be unthrottled while in a half-throttled state (i.e. some tasks have been removed from them, while others are still enqueued and runnable), so the unthrottling process is a bit more involved.
[1]: https://lore.kernel.org/all/[email protected]/
--- include/linux/sched.h | 2 + kernel/sched/core.c | 6 +- kernel/sched/debug.c | 4 +- kernel/sched/fair.c | 257 +++++++++++++++++++++++++++++++----------- kernel/sched/sched.h | 7 ++ 5 files changed, 206 insertions(+), 70 deletions(-)
include/linux/sched.h +2 -0
@@ -805,6 +805,8 @@ struct task_struct {#ifdef CONFIG_CGROUP_SCHED struct task_group *sched_task_group; +struct callback_head sched_throttle_work; +struct list_head throttle_node;#endif #ifdef CONFIG_UCLAMP_TASK
kernel/sched/core.c +5 -1
@@ -4504,6 +4504,10 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.cfs_rq =NULL ;#endif +#ifdef CONFIG_CFS_BANDWIDTH +init_cfs_throttle_work (p); +#endif +#ifdef CONFIG_SCHEDSTATS /* Even if schedstat is disabled, there should not be garbage */ memset (&p->stats,0 ,sizeof (p->stats));
@@ -10823,7 +10827,7 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota, cfs_rq->runtime_enabled = runtime_enabled; cfs_rq->runtime_remaining =0 ; -if (cfs_rq->throttled) +if (cfs_rq->in_throttle_limbo)unthrottle_cfs_rq (cfs_rq); }
kernel/sched/debug.c +2 -2
@@ -695,8 +695,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)#endif #endif #ifdef CONFIG_CFS_BANDWIDTH -SEQ_printf (m," .%-30s: %d\n" ,"throttled" , - cfs_rq->throttled); +SEQ_printf (m," .%-30s: %d\n" ,"in_throttle_limbo" , + cfs_rq->in_throttle_limbo);SEQ_printf (m," .%-30s: %d\n" ,"throttle_count" , cfs_rq->throttle_count);#endif
kernel/sched/fair.c +190 -67
385 lines not on this page · raw message
kernel/sched/sched.h +7 -0
@@ -633,8 +633,13 @@ struct cfs_rq {u64 throttled_clock_self_time;int throttled;int throttle_count; +int in_throttle_limbo; +/* Temp storage for updating the counts during unthrottling */ +unsigned int unthrottled_h_nr_running; +unsigned int unthrottled_idle_h_nr_running;struct list_head throttled_list;struct list_head throttled_csd_list; +struct list_head throttled_limbo_list;#endif /* CONFIG_CFS_BANDWIDTH */ #endif /* CONFIG_FAIR_GROUP_SCHED */ };
@@ -2416,6 +2421,8 @@ extern void init_sched_dl_class(void);extern void init_sched_rt_class (void );extern void init_sched_fair_class (void ); +extern void init_cfs_throttle_work (struct task_struct *p); +extern void reweight_task (struct task_struct *p,int prio);extern void resched_curr (struct rq *rq);
Signature · 1 line
2.41.0
Benjamin Segall
reviewerRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
Quoted 5 lines · Valentin Schneider
As reported in [1], CFS bandwidth throttling is a source of headaches in
PREEMPT_RT- generally speaking, a throttled CFS task can hold locks that prevent ksoftirqd from running, which prevents replenishing & unthrottling thecfs_rqof said CFS task.
Quoted 8 lines
Peter mentioned that there have been discussions on changing /when/ the throttling happens: rather than have it be done immediately upon updating the runtime statistics and realizing the
cfs_rqhas depleted its quota, we wait for the task to be about to return to userspace.I'm not aware of the arguments in favour of this for !PREEMPT_RT, but given [1] I had a look into it. Using this approach means no task can be throttled while holding a kernel lock, which solves the locking dependency issue.
The alternative we've been experimenting with (and still running into other issues that have made it hard to tell if they work) is to still leave the tasks on their cfs_rqs, but instead have two task_timelines or similar per cfs_rq, one of which only has unthrottleable tasks (or partially-throttled child cgroups) on it. Then when picking into a partially-unthrottled cfs_rq you only look at that alternate task_timeline.
This means that we get to skip this per-actually-throttled-task loop:
Quoted 15 lines
kernel/sched/fair.c +10
-0@@ -5548,7 +5548,61 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) {struct rq *rq = data;struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of (rq)]; +struct sched_entity *se = tg->se[cpu_of (rq)]; +struct cfs_rq *pcfs_rq =cfs_rq_of (se); +long task_delta =0 , idle_task_delta =0 ; +struct task_struct *p, *tmp; +/* +* Re-enqueue the tasks that have been throttled at this level. +* +* The task count is up-propagated via ->unthrottled_*h_nr_running, +* as we can't purely rely on h_nr_running post-enqueue: the unthrottle +* might happen when a cfs_rq still has some tasks enqueued, either still
kernel/sched/fair.c static int tg_unthrottle_up(struct task_group *tg, void *data) · line 5571
Quoted 10 lines
kernel/sched/fair.c +10
-0+* making their way to userspace, or freshly migrated to it. +*/ +list_for_each_entry_safe (p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) { +struct sched_entity *pse = &p->se; + +list_del_init (&p->throttle_node); +enqueue_entity (cfs_rq, pse,ENQUEUE_WAKEUP ); + task_delta++; + idle_task_delta +=task_has_idle_policy (p); + }
The downsides are that you instead have extra operations per enqueue/dequeue/pick (but just an extra list/rbtree operation or check), and that it doesn't do *any* accounting change for a partially dequeued cfs_rq.
I'm going to try putting together a cleaner variant of our version that works via task_work instead of bracketing every relevant entry point. (That design came from when we were trying instead to only do it for tasks holding actual locks)
Valentin Schneider
reviewerRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
Quoted 16 lines · Benjamin Segall
Quoted 5 lines · Valentin Schneider
As reported in [1], CFS bandwidth throttling is a source of headaches in
PREEMPT_RT- generally speaking, a throttled CFS task can hold locks that prevent ksoftirqd from running, which prevents replenishing & unthrottling thecfs_rqof said CFS task.Quoted 8 lines
Peter mentioned that there have been discussions on changing /when/ the throttling happens: rather than have it be done immediately upon updating the runtime statistics and realizing the
cfs_rqhas depleted its quota, we wait for the task to be about to return to userspace.I'm not aware of the arguments in favour of this for !PREEMPT_RT, but given [1] I had a look into it. Using this approach means no task can be throttled while holding a kernel lock, which solves the locking dependency issue.
Quoted 7 lines
The alternative we've been experimenting with (and still running into other issues that have made it hard to tell if they work) is to still leave the tasks on their
cfs_rqs, but instead have twotask_timelinesor similar percfs_rq, one of which only has unthrottleable tasks (or partially-throttled child cgroups) on it. Then when picking into a partially-unthrottledcfs_rqyou only look at that alternatetask_timeline.
IIUC then you don't dequeue the cfs_rq's se, but instead rely on the RB tree switch to only have unthrottable tasks running.
Quoted 24 lines
This means that we get to skip this per-actually-throttled-task loop:
Quoted 22 lines
kernel/sched/fair.c +17
-0@@ -5548,7 +5548,61 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) {struct rq *rq = data;struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of (rq)]; +struct sched_entity *se = tg->se[cpu_of (rq)]; +struct cfs_rq *pcfs_rq =cfs_rq_of (se); +long task_delta =0 , idle_task_delta =0 ; +struct task_struct *p, *tmp; +/* +* Re-enqueue the tasks that have been throttled at this level. +* +* The task count is up-propagated via ->unthrottled_*h_nr_running, +* as we can't purely rely on h_nr_running post-enqueue: the unthrottle +* might happen when a cfs_rq still has some tasks enqueued, either still +* making their way to userspace, or freshly migrated to it. +*/ +list_for_each_entry_safe (p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) { +struct sched_entity *pse = &p->se; + +list_del_init (&p->throttle_node); +enqueue_entity (cfs_rq, pse,ENQUEUE_WAKEUP );
Quoted 9 lines
kernel/sched/fair.c static int tg_unthrottle_up(struct task_group *tg, void *data) · line 5571
Quoted 3 lines
kernel/sched/fair.c +3
-0+ task_delta++; + idle_task_delta +=task_has_idle_policy (p); + }The downsides are that you instead have extra operations per enqueue/dequeue/pick (but just an extra list/rbtree operation or check), and that it doesn't do *any* accounting change for a partially dequeued
cfs_rq.
I would assume we want to keep the *nr_running as close to reality as possible, given their impact on pick_next_task_fair() & load_balance().
Quoted 4 lines
I'm going to try putting together a cleaner variant of our version that works via
task_workinstead of bracketing every relevant entry point. (That design came from when we were trying instead to only do it for tasks holding actual locks)
Interesting, thank you for sharing! I assume then the motivation for this is to reduce latencies caused by throttling lock holders?
Benjamin Segall
reviewerRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
Quoted 11 lines · Valentin Schneider
Quoted 9 lines · Benjamin Segall
Valentin Schneider <[email protected]> writes:
The alternative we've been experimenting with (and still running into other issues that have made it hard to tell if they work) is to still leave the tasks on their
cfs_rqs, but instead have twotask_timelinesor similar percfs_rq, one of which only has unthrottleable tasks (or partially-throttled child cgroups) on it. Then when picking into a partially-unthrottledcfs_rqyou only look at that alternatetask_timeline.
Quoted 2 lines
IIUC then you don't dequeue the cfs_rq's se, but instead rely on the RB tree switch to only have unthrottable tasks running.
Correct.
Quoted 35 lines
Quoted 25 lines
This means that we get to skip this per-actually-throttled-task loop:
Quoted 22 lines
kernel/sched/fair.c +17
-0@@ -5548,7 +5548,61 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) {struct rq *rq = data;struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of (rq)]; +struct sched_entity *se = tg->se[cpu_of (rq)]; +struct cfs_rq *pcfs_rq =cfs_rq_of (se); +long task_delta =0 , idle_task_delta =0 ; +struct task_struct *p, *tmp; +/* +* Re-enqueue the tasks that have been throttled at this level. +* +* The task count is up-propagated via ->unthrottled_*h_nr_running, +* as we can't purely rely on h_nr_running post-enqueue: the unthrottle +* might happen when a cfs_rq still has some tasks enqueued, either still +* making their way to userspace, or freshly migrated to it. +*/ +list_for_each_entry_safe (p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) { +struct sched_entity *pse = &p->se; + +list_del_init (&p->throttle_node); +enqueue_entity (cfs_rq, pse,ENQUEUE_WAKEUP );Quoted 9 lines
kernel/sched/fair.c static int tg_unthrottle_up(struct task_group *tg, void *data) · line 5571
Quoted 3 lines
kernel/sched/fair.c +3
-0+ task_delta++; + idle_task_delta +=task_has_idle_policy (p); + }The downsides are that you instead have extra operations per enqueue/dequeue/pick (but just an extra list/rbtree operation or check), and that it doesn't do *any* accounting change for a partially dequeued
cfs_rq.
Quoted 3 lines
I would assume we want to keep the
*nr_runningas close to reality as possible, given their impact onpick_next_task_fair()&load_balance().
Yeah, and while it's maybe ok for the longer-period load_balance, it's definitely sketchy for the shorter term things like select_idle_sibling. In theory we could duplicate more and more of the accounting... (though that also then becomes questionable amounts of increased overhead on enqueue/dequeue).
Quoted 7 lines
Quoted 4 lines
I'm going to try putting together a cleaner variant of our version that works via
task_workinstead of bracketing every relevant entry point. (That design came from when we were trying instead to only do it for tasks holding actual locks)Interesting, thank you for sharing! I assume then the motivation for this is to reduce latencies caused by throttling lock holders?
Yeah, and then we ran into things like percpu-rwsem where lock/unlock accounting doesn't work as well, but kept the basic design.
Benjamin Segall
reviewerRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
In reply to Valentin SchneiderAlright, this took longer than I intended, but here's the basic version with the duplicate "runqueue" just being a list rather than actual EEVDF or anything sensible.
I also wasn't actually able to adapt it to work via task_work like I thought I could; flagging the entry to the relevant schedule() calls is too critical.
--------
Subject: [RFC PATCH] sched/fair: only throttle CFS tasks in userspace
The basic idea of this implementation is to maintain duplicate runqueues in each cfs_rq that contain duplicate pointers to sched_entitys which should bypass throttling. Then we can skip throttling cfs_rqs that have any such children, and when we pick inside any not-actually-throttled cfs_rq, we only look at this duplicated list.
"Which tasks should bypass throttling" here is "all schedule() calls that don't set a special flag", but could instead involve the lockdep markers (except for the problem of percpu-rwsem and similar) or explicit flags around syscalls and faults, or something else.
This approach avoids any O(tasks) loops, but leaves partially-throttled cfs_rqs still contributing their full h_nr_running to their parents, which might result in worse balancing. Also it adds more (generally still small) overhead to the common enqueue/dequeue/pick paths.
The very basic debug test added is to run a cpusoaker and "cat /sys/kernel/debug/sched_locked_spin" pinned to the same cpu in the same cgroup with a quota < 1 cpu.
--- include/linux/sched.h | 7 ++ kernel/entry/common.c | 2 +- kernel/entry/kvm.c | 2 +- kernel/sched/core.c | 20 ++++ kernel/sched/debug.c | 28 +++++ kernel/sched/fair.c | 232 ++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 3 + 7 files changed, 281 insertions(+), 13 deletions(-)
include/linux/sched.h +7 -0
@@ -301,10 +301,11 @@ extern long schedule_timeout(long timeout);extern long schedule_timeout_interruptible (long timeout);extern long schedule_timeout_killable (long timeout);extern long schedule_timeout_uninterruptible (long timeout);extern long schedule_timeout_idle (long timeout); asmlinkage void schedule(void); +asmlinkage void schedule_usermode(void);extern void schedule_preempt_disabled (void ); asmlinkage void preempt_schedule_irq(void);#ifdef CONFIG_PREEMPT_RT extern void schedule_rtlock (void );#endif
@@ -576,10 +577,13 @@ struct sched_entity {struct cfs_rq *cfs_rq;/* rq "owned" by this entity/group: */ struct cfs_rq *my_q;/* cached value of my_q->h_nr_running */ unsigned long runnable_weight; +#ifdef CONFIG_CFS_BANDWIDTH +struct list_head kernel_node; +#endif #endif #ifdef CONFIG_SMP /* * Per entity load average tracking.
@@ -1562,10 +1566,13 @@ struct task_struct {#ifdef CONFIG_USER_EVENTS struct user_event_mm *user_event_mm;#endif +#ifdef CONFIG_CFS_BANDWIDTH +atomic_t in_return_to_user; +#endif /* * New fields for task_struct should be added above here, so that * they are included in the randomized portion of task_struct. */ randomized_struct_fields_end
kernel/entry/common.c +1 -1
@@ -154,11 +154,11 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,while (ti_work &EXIT_TO_USER_MODE_WORK ) {local_irq_enable_exit_to_user (ti_work);if (ti_work & _TIF_NEED_RESCHED) -schedule (); +schedule_usermode ();/* TODO: also all of the arch/* loops that don't use this yet */ if (ti_work & _TIF_UPROBE)uprobe_notify_resume (regs);if (ti_work & _TIF_PATCH_PENDING)
kernel/entry/kvm.c +1 -1
@@ -12,11 +12,11 @@ static int xfer_to_guest_mode_work(struct kvm_vcpu *vcpu, unsigned long ti_work)kvm_handle_signal_exit (vcpu);return -EINTR ; }if (ti_work & _TIF_NEED_RESCHED) -schedule (); +schedule_usermode ();if (ti_work & _TIF_NOTIFY_RESUME)resume_user_mode_work (NULL ); ret =arch_xfer_to_guest_mode_handle_work (vcpu, ti_work);
kernel/sched/core.c +20 -0
@@ -4527,10 +4527,14 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)INIT_LIST_HEAD (&p->se.group_node);#ifdef CONFIG_FAIR_GROUP_SCHED p->se.cfs_rq =NULL ;#endif +#ifdef CONFIG_CFS_BANDWIDTH +INIT_LIST_HEAD (&p->se.kernel_node); +atomic_set (&p->in_return_to_user,0 ); +#endif #ifdef CONFIG_SCHEDSTATS /* Even if schedstat is disabled, there should not be garbage */ memset (&p->stats,0 ,sizeof (p->stats));#endif
@@ -6816,10 +6820,26 @@ asmlinkage __visible void __sched schedule(void)__schedule_loop (SM_NONE );sched_update_worker (tsk); }EXPORT_SYMBOL (schedule); +asmlinkage __visible void __sched schedule_usermode(void) +{ +#ifdef CONFIG_CFS_BANDWIDTH +/* +* This is only atomic because of this simple implementation. We could +* do something with an SM_USER to avoid other-cpu scheduler operations +* racing against these writes. +*/ +atomic_set (¤t->in_return_to_user, true); +schedule (); +atomic_set (¤t->in_return_to_user, false); +#else +schedule (); +#endif +} +/* * synchronize_rcu_tasks() makes sure that no task is stuck in preempted * state (have scheduled out non-voluntarily) by making sure that all * tasks have either left the run queue or have gone into user space. * As idle tasks do not do either, they must not ever be preempted
kernel/sched/debug.c +28 -0
@@ -317,10 +317,36 @@ static const struct file_operations sched_verbose_fops = { .write = sched_verbose_write, .open = simple_open, .llseek = default_llseek, }; +static DEFINE_MUTEX (sched_debug_spin_mutex); +static int sched_debug_spin_show (struct seq_file *m,void *v) { +int count; +mutex_lock (&sched_debug_spin_mutex); +for (count =0 ; count <1000 ; count++) { +u64 start2; + start2 = jiffies; +while (jiffies == start2) +cpu_relax (); +schedule (); + } +mutex_unlock (&sched_debug_spin_mutex); +return 0 ; +} +static int sched_debug_spin_open (struct inode *inode,struct file *filp) +{ +return single_open (filp, sched_debug_spin_show,NULL ); +} + +static const struct file_operations sched_debug_spin_fops = { + .open = sched_debug_spin_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +static const struct seq_operations sched_debug_sops;static int sched_debug_open (struct inode *inode,struct file *filp) {return seq_open (filp, &sched_debug_sops);
@@ -372,10 +398,12 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops); +debugfs_create_file ("sched_locked_spin" ,0444 ,NULL ,NULL , + &sched_debug_spin_fops);return 0 ; }late_initcall (sched_init_debug);#ifdef CONFIG_SMP
kernel/sched/fair.c +221 -11
547 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -578,10 +578,11 @@ struct cfs_rq {u64 min_vruntime_copy;#endif struct rb_root_cached tasks_timeline; +/* * 'curr' points to currently running entity on this cfs_rq. * It is set to NULL otherwise (i.e when none are currently running). */ struct sched_entity *curr;
@@ -656,12 +657,14 @@ struct cfs_rq {u64 throttled_clock_pelt_time;u64 throttled_clock_self;u64 throttled_clock_self_time;int throttled;int throttle_count; +int h_kernel_running;struct list_head throttled_list;struct list_head throttled_csd_list; +struct list_head kernel_children;;#endif /* CONFIG_CFS_BANDWIDTH */ #endif /* CONFIG_FAIR_GROUP_SCHED */ };static inline int rt_bandwidth_enabled (void )
Signature · 1 line
2.43.0.472.g3155946c3a-goog
Valentin Schneider
reviewerRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
Quoted 8 lines · Benjamin Segall
Alright, this took longer than I intended, but here's the basic version with the duplicate "runqueue" just being a list rather than actual EEVDF or anything sensible.
I also wasn't actually able to adapt it to work via
task_worklike I thought I could; flagging the entry to the relevantschedule()calls is too critical.
Thanks for sharing this! Have a first set of comments.
Quoted 15 lines
--------
Subject: [RFC PATCH] sched/fair: only throttle CFS tasks in userspace
The basic idea of this implementation is to maintain duplicate runqueues in each
cfs_rqthat contain duplicate pointers tosched_entityswhich should bypass throttling. Then we can skip throttlingcfs_rqsthat have any such children, and when we pick inside any not-actually-throttledcfs_rq, we only look at this duplicated list."Which tasks should bypass throttling" here is "all
schedule()calls that don't set a special flag", but could instead involve the lockdep markers (except for the problem of percpu-rwsem and similar) or explicit flags around syscalls and faults, or something else.
Quoted 8 lines
This approach avoids any O(tasks) loops, but leaves partially-throttled
cfs_rqsstill contributing their fullh_nr_runningto their parents, which might result in worse balancing. Also it adds more (generally still small) overhead to the common enqueue/dequeue/pick paths.The very basic debug test added is to run a cpusoaker and "cat
/sys/kernel/debug/sched_locked_spin" pinned to the same cpu in the same cgroup with a quota < 1 cpu.
So if I'm trying to compare notes:
The dual tree:
- Can actually throttle cfs_rq's once all tasks are in userspace
- Adds (light-ish) overhead to enqueue/dequeue
- Doesn't keep *nr_running updated when not fully throttled
Whereas dequeue via task_work:
- Can never fully throttle a cfs_rq
- Adds (not so light) overhead to unthrottle
- Keeps *nr_running updated
My thoughts right now are that there might be a way to mix both worlds: go with the dual tree, but subtract from h_nr_running at dequeue_kernel() (sort of doing the count update in throttle_cfs_rq() early) and keep track of in-user tasks via handle_kernel_task_prev() to add this back when unthrottling a not-fully-throttled cfs_rq. I need to actually think about it though :-)
kernel/entry/common.c static unsigned long exit_to_user_mode_loop(struct pt_regs *regs, · line 159
Quoted 8 lines
kernel/entry/common.c +1
-1@@ -154,11 +154,11 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,while (ti_work &EXIT_TO_USER_MODE_WORK ) {local_irq_enable_exit_to_user (ti_work);if (ti_work & _TIF_NEED_RESCHED) -schedule (); +schedule_usermode ();/* TODO: also all of the arch/* loops that don't use this yet */
^^
Interestingly -Werror=comment trips on this
Quoted 24 lines
kernel/entry/common.c +6
-5if (ti_work & _TIF_UPROBE)uprobe_notify_resume (regs);if (ti_work & _TIF_PATCH_PENDING)@@ -5416,18 +5426,38 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) se->prev_sum_exec_runtime = se->sum_exec_runtime; }/* * Pick the next process, keeping these things in mind, in this order: -* 1) keep things fair between processes/task groups -* 2) pick the "next" process, since someone really wants that to run -* 3) pick the "last" process, for cache locality -* 4) do not run the "skip" process, if something else is available +* 1) If we're inside a throttled cfs_rq, only pick threads in the kernel +* 2) keep things fair between processes/task groups +* 3) pick the "next" process, since someone really wants that to run +* 4) pick the "last" process, for cache locality +* 5) do not run the "skip" process, if something else is available */ static struct sched_entity * -pick_next_entity (struct cfs_rq *cfs_rq) +pick_next_entity (struct cfs_rq *cfs_rq,bool throttled)
kernel/entry/common.c set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 5448
Quoted 10 lines
kernel/entry/common.c +9
-0{ +#ifdef CONFIG_CFS_BANDWIDTH +/* +* TODO: This might trigger, I'm not sure/don't remember. Regardless, +* while we do not explicitly handle the case where h_kernel_running +* goes to 0, we will call account/check_cfs_rq_runtime at worst in +* entity_tick and notice that we can now properly do the full +* throttle_cfs_rq. +*/ +WARN_ON_ONCE (list_empty (&cfs_rq->kernel_children));
FWIW this triggers pretty much immediately in my environment (buildroot over QEMU).
Quoted 14 lines
kernel/sched/fair.c +3
-0@@ -5712,16 +5748,26 @@ static int tg_throttle_down(struct task_group *tg, void *data) cfs_rq->throttle_count++;return 0 ; } +static void enqueue_kernel (struct cfs_rq *cfs_rq,struct sched_entity *se,int count); +static void dequeue_kernel (struct cfs_rq *cfs_rq,struct sched_entity *se,int count); +static bool throttle_cfs_rq (struct cfs_rq *cfs_rq) {struct rq *rq =rq_of (cfs_rq);struct cfs_bandwidth *cfs_b =tg_cfs_bandwidth (cfs_rq->tg);struct sched_entity *se;
kernel/sched/fair.c static int tg_throttle_down(struct task_group *tg, void *data) · line 5769
Quoted 10 lines
kernel/sched/fair.c +8
-1-long task_delta, idle_task_delta, dequeue =1 ; +long task_delta, idle_task_delta, kernel_delta, dequeue =1 ; + +/* +* We don't actually throttle, though account() will have made sure to +* resched us so that we pick into a kernel task. +*/ +if (cfs_rq->h_kernel_running) +return false;
So as long as a cfs_rq has at least one task executing in the kernel, we keep its *nr_running intact. We do throttle the whole thing once all the tasks are in userspace (or about to be anyway), and unthrottle_on_enqueue() undoes that.
Quoted 23 lines
kernel/sched/fair.c +17
-0@@ -8316,15 +8471,44 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int preempt:resched_curr (rq); } +static void handle_kernel_task_prev (struct task_struct *prev) +{ +#ifdef CONFIG_CFS_BANDWIDTH +struct sched_entity *se = &prev->se; +bool new_kernel =is_kernel_task (prev); +bool prev_kernel = !list_empty (&se->kernel_node); +/* +* These extra loops are bad and against the whole point of the merged +* PNT, but it's a pain to merge, particularly since we want it to occur +* before check_cfs_runtime. +*/ +if (prev_kernel && !new_kernel) { +WARN_ON_ONCE (!se->on_rq);/* dequeue should have removed us */ +for_each_sched_entity (se) { +dequeue_kernel (cfs_rq_of (se), se,1 ); +if (cfs_rq_throttled (cfs_rq_of (se))) +break ;
kernel/sched/fair.c static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int · line 8502
Quoted 10 lines
kernel/sched/fair.c +10
-0+ } + }else if (!prev_kernel && new_kernel && se->on_rq) { +for_each_sched_entity (se) { +enqueue_kernel (cfs_rq_of (se), se,1 ); +if (cfs_rq_throttled (cfs_rq_of (se))) +break ; + } + } +#endif +}
I'm trying to grok what the implications of a second tasks_timeline would be on picking - we'd want a RBtree update equivalent to put_prev_entity()'s __enqueue_entity(), but that second tree doesn't follow the same enqueue/dequeue cycle as the first one. Would a __dequeue_entity() + __enqueue_entity() to force a rebalance make sense? That'd also take care of updating the min_vruntime.
Benjamin Segall
reviewerRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
Quoted 43 lines · Valentin Schneider
Quoted 8 lines · Benjamin Segall
Alright, this took longer than I intended, but here's the basic version with the duplicate "runqueue" just being a list rather than actual EEVDF or anything sensible.
I also wasn't actually able to adapt it to work via
task_worklike I thought I could; flagging the entry to the relevantschedule()calls is too critical.Thanks for sharing this! Have a first set of comments.
Quoted 15 lines
--------
Subject: [RFC PATCH] sched/fair: only throttle CFS tasks in userspace
The basic idea of this implementation is to maintain duplicate runqueues in each
cfs_rqthat contain duplicate pointers tosched_entityswhich should bypass throttling. Then we can skip throttlingcfs_rqsthat have any such children, and when we pick inside any not-actually-throttledcfs_rq, we only look at this duplicated list."Which tasks should bypass throttling" here is "all
schedule()calls that don't set a special flag", but could instead involve the lockdep markers (except for the problem of percpu-rwsem and similar) or explicit flags around syscalls and faults, or something else.Quoted 8 lines
This approach avoids any O(tasks) loops, but leaves partially-throttled
cfs_rqsstill contributing their fullh_nr_runningto their parents, which might result in worse balancing. Also it adds more (generally still small) overhead to the common enqueue/dequeue/pick paths.The very basic debug test added is to run a cpusoaker and "cat
/sys/kernel/debug/sched_locked_spin" pinned to the same cpu in the same cgroup with a quota < 1 cpu.So if I'm trying to compare notes:
The dual tree:
- Can actually throttle cfs_rq's once all tasks are in userspace
- Adds (light-ish) overhead to enqueue/dequeue
- Doesn't keep*nr_runningupdated when not fully throttled
Quoted 4 lines
Whereas dequeue via
task_work:
- Can never fully throttle acfs_rq
- Adds (not so light) overhead to unthrottle
- Keeps*nr_runningupdated
Yeah, this sounds right. (Though for the task_work version once all the tasks are throttled, the cfs_rq is dequeued like normal, so it doesn't seem like a big deal to me)
Quoted 7 lines
My thoughts right now are that there might be a way to mix both worlds: go with the dual tree, but subtract from
h_nr_runningatdequeue_kernel()(sort of doing the count update inthrottle_cfs_rq()early) and keep track of in-user tasks viahandle_kernel_task_prev()to add this back when unthrottling a not-fully-throttledcfs_rq. I need to actually think about it though :-)
Probably, but I had tons of trouble getting the accounting correct as it was :P
Quoted 4 lines
Quoted 4 lines
kernel/sched/fair.c +4
-0+#ifdef CONFIG_CFS_BANDWIDTH +/* +* TODO: This might trigger, I'm not sure/don't remember. Regardless, +* while we do not explicitly handle the case where h_kernel_running
Quoted 8 lines
kernel/sched/fair.c set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 5448
Quoted 5 lines
kernel/sched/fair.c +5
-0+* goes to 0, we will call account/check_cfs_rq_runtime at worst in +* entity_tick and notice that we can now properly do the full +* throttle_cfs_rq. +*/ +WARN_ON_ONCE (list_empty (&cfs_rq->kernel_children));FWIW this triggers pretty much immediately in my environment (buildroot over QEMU).
Good to know; I feel like this should be avoidable (and definitely should be avoided if it can be), but I might be forgetting some case that's mostly unavoidable. I'll have to poke it when I have more time.
Quoted 35 lines
Quoted 23 lines
kernel/sched/fair.c +17
-0@@ -8316,15 +8471,44 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int preempt:resched_curr (rq); } +static void handle_kernel_task_prev (struct task_struct *prev) +{ +#ifdef CONFIG_CFS_BANDWIDTH +struct sched_entity *se = &prev->se; +bool new_kernel =is_kernel_task (prev); +bool prev_kernel = !list_empty (&se->kernel_node); +/* +* These extra loops are bad and against the whole point of the merged +* PNT, but it's a pain to merge, particularly since we want it to occur +* before check_cfs_runtime. +*/ +if (prev_kernel && !new_kernel) { +WARN_ON_ONCE (!se->on_rq);/* dequeue should have removed us */ +for_each_sched_entity (se) { +dequeue_kernel (cfs_rq_of (se), se,1 ); +if (cfs_rq_throttled (cfs_rq_of (se))) +break ;kernel/sched/fair.c static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int · line 8502
Quoted 10 lines
kernel/sched/fair.c +10
-0+ } + }else if (!prev_kernel && new_kernel && se->on_rq) { +for_each_sched_entity (se) { +enqueue_kernel (cfs_rq_of (se), se,1 ); +if (cfs_rq_throttled (cfs_rq_of (se))) +break ; + } + } +#endif +}
Quoted 6 lines
I'm trying to grok what the implications of a second
tasks_timelinewould be on picking - we'd want a RBtree update equivalent to put_prev_entity()'s__enqueue_entity(), but that second tree doesn't follow the same enqueue/dequeue cycle as the first one. Would a__dequeue_entity()+__enqueue_entity()to force a rebalance make sense? That'd also take care of updating themin_vruntime.
Yeah, the most sensible thing to do would probably be to just have curr not be in the queue, the same as the normal rbtree, and save the "on kernel list" as an on_rq-equivalent bool instead of as !list_empty(kernel_node).
Valentin Schneider
reviewerRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
Quoted 9 lines · Benjamin Segall
Quoted 8 lines · Valentin Schneider
So if I'm trying to compare notes:
The dual tree:
- Can actually throttle cfs_rq's once all tasks are in userspace
- Adds (light-ish) overhead to enqueue/dequeue
- Doesn't keep*nr_runningupdated when not fully throttled
Quoted 9 lines
Quoted 4 lines
Whereas dequeue via
task_work:
- Can never fully throttle acfs_rq
- Adds (not so light) overhead to unthrottle
- Keeps*nr_runningupdatedYeah, this sounds right. (Though for the
task_workversion once all the tasks are throttled, thecfs_rqis dequeued like normal, so it doesn't seem like a big deal to me)
In my implementation the cfs_rq's can be dequeued once all of their tasks have been throttled, but the tasks aren't migrated back onto the cfs_rq's until the unthrottle (rather than at the "final" throttle).
That is done because if a random task gets migrated to a throttled & empty cfs_rq, we still need to schedule it 'till it exits the kernel, and we don't want the now-in-userspace task to suddenly become eligible for picking.
That adds these O(ntasks) loops in the unthrottle to re-shape the cfs_rq's. The dual tree approach saves us from doing all that, IIUC unthrottle_on_enqueue() only puts the se's in the task's direct hierarchy back onto the second tree for them to be picked.
Quoted 11 lines
Quoted 7 lines
My thoughts right now are that there might be a way to mix both worlds: go with the dual tree, but subtract from
h_nr_runningatdequeue_kernel()(sort of doing the count update inthrottle_cfs_rq()early) and keep track of in-user tasks viahandle_kernel_task_prev()to add this back when unthrottling a not-fully-throttledcfs_rq. I need to actually think about it though :-)Probably, but I had tons of trouble getting the accounting correct as it was :P
Yeah so did I, cgroups are "fun".
Quoted 11 lines
Quoted 7 lines
I'm trying to grok what the implications of a second
tasks_timelinewould be on picking - we'd want a RBtree update equivalent to put_prev_entity()'s__enqueue_entity(), but that second tree doesn't follow the same enqueue/dequeue cycle as the first one. Would a__dequeue_entity()+__enqueue_entity()to force a rebalance make sense? That'd also take care of updating themin_vruntime.Yeah, the most sensible thing to do would probably be to just have curr not be in the queue, the same as the normal rbtree, and save the "on kernel list" as an on_rq-equivalent bool instead of as !list_empty(kernel_node).
Makes sense, I'll have a go at this.
kernel test robot
build botRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
In reply to Valentin Schneiderkernel test robot noticed the following build errors:
Hi Valentin,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on tip/sched/core]
[also build test ERROR on tip/master linus/master v6.7-rc4 next-20231205] [cannot apply to tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Valentin-Schneider/sched-fair-Only-throttle-CFS-tasks-on-return-to-userspace/20231201-001908 base: tip/sched/core patch link: https://lore.kernel.org/r/20231130161245.3894682-2-vschneid%40redhat.com patch subject: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace config: arm-mvebu_v5_defconfig (https://download.01.org/0day-ci/archive/20231206/[email protected]/config) compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231206/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
kernel/sched/fair.c:8575:30: error: no member named 'in_throttle_limbo' in 'struct cfs_rq'
if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
~~~~~~~~~~~~~ ^
include/linux/compiler.h:77:42: note: expanded from macro 'unlikely'
# define unlikely(x) __builtin_expect(!!(x), 0)
^
>> kernel/sched/fair.c:8575:52: error: call to undeclared function 'task_has_throttle_work'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
^
kernel/sched/fair.c:8575:52: note: did you mean 'init_cfs_throttle_work'?
kernel/sched/sched.h:2457:13: note: 'init_cfs_throttle_work' declared here
extern void init_cfs_throttle_work(struct task_struct *p);
^
kernel/sched/fair.c:8576:24: error: no member named 'sched_throttle_work' in 'struct task_struct'
task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
~ ^
kernel/sched/fair.c:13197:6: warning: no previous prototype for function 'free_fair_sched_group' [-Wmissing-prototypes]
void free_fair_sched_group(struct task_group *tg) { }
^
kernel/sched/fair.c:13197:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
void free_fair_sched_group(struct task_group *tg) { }
^
static
kernel/sched/fair.c:13199:5: warning: no previous prototype for function 'alloc_fair_sched_group' [-Wmissing-prototypes]
int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
^
kernel/sched/fair.c:13199:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
^
static
kernel/sched/fair.c:13204:6: warning: no previous prototype for function 'online_fair_sched_group' [-Wmissing-prototypes]
void online_fair_sched_group(struct task_group *tg) { }
^
kernel/sched/fair.c:13204:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
void online_fair_sched_group(struct task_group *tg) { }
^
static
kernel/sched/fair.c:13206:6: warning: no previous prototype for function 'unregister_fair_sched_group' [-Wmissing-prototypes]
void unregister_fair_sched_group(struct task_group *tg) { }
^ kernel/sched/fair.c:13206:1: note: declare 'static' if the function is not intended to be used outside of this translation unit void unregister_fair_sched_group(struct task_group *tg) { }
^
static
4 warnings and 3 errors generated.
vim +/task_has_throttle_work +8575 kernel/sched/fair.c
8471
8472 struct task_struct *
8473 pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
8474 {
8475 struct cfs_rq *cfs_rq = &rq->cfs;
8476 struct sched_entity *se;
8477 struct task_struct *p;
8478 int new_tasks;
8479
8480 again:
8481 if (!sched_fair_runnable(rq))
8482 goto idle;
8483
8484 #ifdef CONFIG_FAIR_GROUP_SCHED
8485 if (!prev || prev->sched_class != &fair_sched_class)
8486 goto simple;
8487
8488 /*
8489 * Because of the set_next_buddy() in dequeue_task_fair() it is rather
8490 * likely that a next task is from the same cgroup as the current.
8491 *
8492 * Therefore attempt to avoid putting and setting the entire cgroup
8493 * hierarchy, only change the part that actually changes.
8494 */
8495
8496 do {
8497 struct sched_entity *curr = cfs_rq->curr;
8498
8499 /*
8500 * Since we got here without doing put_prev_entity() we also
8501 * have to consider cfs_rq->curr. If it is still a runnable
8502 * entity, update_curr() will update its vruntime, otherwise
8503 * forget we've ever seen it.
8504 */
8505 if (curr) {
8506 if (curr->on_rq)
8507 update_curr(cfs_rq);
8508 else
8509 curr = NULL;
8510
8511 /*
8512 * This call to check_cfs_rq_runtime() will do the
8513 * throttle and dequeue its entity in the parent(s).
8514 * Therefore the nr_running test will indeed
8515 * be correct.
8516 */
8517 if (unlikely(check_cfs_rq_runtime(cfs_rq))) {
8518 cfs_rq = &rq->cfs;
8519
8520 if (!cfs_rq->nr_running)
8521 goto idle;
8522
8523 goto simple;
8524 }
8525 }
8526
8527 se = pick_next_entity(cfs_rq);
8528 cfs_rq = group_cfs_rq(se);
8529 } while (cfs_rq);
8530
8531 p = task_of(se);
8532
8533 if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
8534 task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
8535 /*
8536 * Since we haven't yet done put_prev_entity and if the selected task
8537 * is a different task than we started out with, try and touch the
8538 * least amount of cfs_rqs.
8539 */
8540 if (prev != p) {
8541 struct sched_entity *pse = &prev->se;
8542
8543 while (!(cfs_rq = is_same_group(se, pse))) {
8544 int se_depth = se->depth;
8545 int pse_depth = pse->depth;
8546
8547 if (se_depth <= pse_depth) {
8548 put_prev_entity(cfs_rq_of(pse), pse);
8549 pse = parent_entity(pse);
8550 }
8551 if (se_depth >= pse_depth) {
8552 set_next_entity(cfs_rq_of(se), se);
8553 se = parent_entity(se);
8554 }
8555 }
8556
8557 put_prev_entity(cfs_rq, pse);
8558 set_next_entity(cfs_rq, se);
8559 }
8560
8561 goto done;
8562 simple:
8563 #endif
8564 if (prev)
8565 put_prev_task(rq, prev);
8566
8567 do {
8568 se = pick_next_entity(cfs_rq);
8569 set_next_entity(cfs_rq, se);
8570 cfs_rq = group_cfs_rq(se);
8571 } while (cfs_rq);
8572
8573 p = task_of(se);
8574
> 8575 if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
8576 task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
8577
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
In reply to Valentin Schneiderkernel test robot noticed the following build errors:
Hi Valentin,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on tip/sched/core]
[also build test ERROR on tip/master linus/master v6.7-rc4 next-20231205] [cannot apply to tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Valentin-Schneider/sched-fair-Only-throttle-CFS-tasks-on-return-to-userspace/20231201-001908 base: tip/sched/core patch link: https://lore.kernel.org/r/20231130161245.3894682-2-vschneid%40redhat.com patch subject: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace config: arm-randconfig-002-20231202 (https://download.01.org/0day-ci/archive/20231206/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231206/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
kernel/sched/fair.c:8575:30: error: no member named 'in_throttle_limbo' in 'struct cfs_rq'
8575 | if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
| ~~~~~~~~~~~~~ ^
include/linux/compiler.h:77:42: note: expanded from macro 'unlikely'
77 | # define unlikely(x) __builtin_expect(!!(x), 0)
| ^
kernel/sched/fair.c:8575:52: error: call to undeclared function 'task_has_throttle_work'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
8575 | if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
| ^
kernel/sched/fair.c:8575:52: note: did you mean 'init_cfs_throttle_work'?
kernel/sched/sched.h:2457:13: note: 'init_cfs_throttle_work' declared here
2457 | extern void init_cfs_throttle_work(struct task_struct *p);
| ^
>> kernel/sched/fair.c:8576:24: error: no member named 'sched_throttle_work' in 'struct task_struct'
8576 | task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
| ~ ^
kernel/sched/fair.c:13197:6: warning: no previous prototype for function 'free_fair_sched_group' [-Wmissing-prototypes]
13197 | void free_fair_sched_group(struct task_group *tg) { }
| ^
kernel/sched/fair.c:13197:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
13197 | void free_fair_sched_group(struct task_group *tg) { }
| ^
| static
kernel/sched/fair.c:13199:5: warning: no previous prototype for function 'alloc_fair_sched_group' [-Wmissing-prototypes]
13199 | int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
| ^
kernel/sched/fair.c:13199:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
13199 | int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
| ^
| static
kernel/sched/fair.c:13204:6: warning: no previous prototype for function 'online_fair_sched_group' [-Wmissing-prototypes]
13204 | void online_fair_sched_group(struct task_group *tg) { }
| ^
kernel/sched/fair.c:13204:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
13204 | void online_fair_sched_group(struct task_group *tg) { }
| ^
| static
kernel/sched/fair.c:13206:6: warning: no previous prototype for function 'unregister_fair_sched_group' [-Wmissing-prototypes]
13206 | void unregister_fair_sched_group(struct task_group *tg) { }
| ^
kernel/sched/fair.c:13206:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
13206 | void unregister_fair_sched_group(struct task_group *tg) { }
| ^
| static 4 warnings and 3 errors generated.
vim +8576 kernel/sched/fair.c
8471
8472 struct task_struct *
8473 pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
8474 {
8475 struct cfs_rq *cfs_rq = &rq->cfs;
8476 struct sched_entity *se;
8477 struct task_struct *p;
8478 int new_tasks;
8479
8480 again:
8481 if (!sched_fair_runnable(rq))
8482 goto idle;
8483
8484 #ifdef CONFIG_FAIR_GROUP_SCHED
8485 if (!prev || prev->sched_class != &fair_sched_class)
8486 goto simple;
8487
8488 /*
8489 * Because of the set_next_buddy() in dequeue_task_fair() it is rather
8490 * likely that a next task is from the same cgroup as the current.
8491 *
8492 * Therefore attempt to avoid putting and setting the entire cgroup
8493 * hierarchy, only change the part that actually changes.
8494 */
8495
8496 do {
8497 struct sched_entity *curr = cfs_rq->curr;
8498
8499 /*
8500 * Since we got here without doing put_prev_entity() we also
8501 * have to consider cfs_rq->curr. If it is still a runnable
8502 * entity, update_curr() will update its vruntime, otherwise
8503 * forget we've ever seen it.
8504 */
8505 if (curr) {
8506 if (curr->on_rq)
8507 update_curr(cfs_rq);
8508 else
8509 curr = NULL;
8510
8511 /*
8512 * This call to check_cfs_rq_runtime() will do the
8513 * throttle and dequeue its entity in the parent(s).
8514 * Therefore the nr_running test will indeed
8515 * be correct.
8516 */
8517 if (unlikely(check_cfs_rq_runtime(cfs_rq))) {
8518 cfs_rq = &rq->cfs;
8519
8520 if (!cfs_rq->nr_running)
8521 goto idle;
8522
8523 goto simple;
8524 }
8525 }
8526
8527 se = pick_next_entity(cfs_rq);
8528 cfs_rq = group_cfs_rq(se);
8529 } while (cfs_rq);
8530
8531 p = task_of(se);
8532
8533 if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
8534 task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
8535 /*
8536 * Since we haven't yet done put_prev_entity and if the selected task
8537 * is a different task than we started out with, try and touch the
8538 * least amount of cfs_rqs.
8539 */
8540 if (prev != p) {
8541 struct sched_entity *pse = &prev->se;
8542
8543 while (!(cfs_rq = is_same_group(se, pse))) {
8544 int se_depth = se->depth;
8545 int pse_depth = pse->depth;
8546
8547 if (se_depth <= pse_depth) {
8548 put_prev_entity(cfs_rq_of(pse), pse);
8549 pse = parent_entity(pse);
8550 }
8551 if (se_depth >= pse_depth) {
8552 set_next_entity(cfs_rq_of(se), se);
8553 se = parent_entity(se);
8554 }
8555 }
8556
8557 put_prev_entity(cfs_rq, pse);
8558 set_next_entity(cfs_rq, se);
8559 }
8560
8561 goto done;
8562 simple:
8563 #endif
8564 if (prev)
8565 put_prev_task(rq, prev);
8566
8567 do {
8568 se = pick_next_entity(cfs_rq);
8569 set_next_entity(cfs_rq, se);
8570 cfs_rq = group_cfs_rq(se);
8571 } while (cfs_rq);
8572
8573 p = task_of(se);
8574
8575 if (unlikely(cfs_rq_of(se)->in_throttle_limbo && !task_has_throttle_work(p)))
> 8576 task_work_add(p, &p->sched_throttle_work, TWA_RESUME);
8577
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace
In reply to Valentin Schneiderkernel test robot noticed the following build warnings:
Hi Valentin,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/sched/core]
[also build test WARNING on tip/master linus/master v6.7-rc4 next-20231205] [cannot apply to tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Valentin-Schneider/sched-fair-Only-throttle-CFS-tasks-on-return-to-userspace/20231201-001908 base: tip/sched/core patch link: https://lore.kernel.org/r/20231130161245.3894682-2-vschneid%40redhat.com patch subject: [RFC PATCH 1/2] sched/fair: Only throttle CFS tasks on return to userspace config: i386-randconfig-141-20231202 (https://download.01.org/0day-ci/archive/20231206/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231206/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> kernel/sched/fair.c:5809:6: warning: variable 'top_cfs_rq' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
if (!qcfs_rq->in_throttle_limbo)
^~~~~~~~~~~~~~~~~~~~~~~~~~~
kernel/sched/fair.c:5860:6: note: uninitialized use occurs here
if (top_cfs_rq && !top_cfs_rq->throttled_clock)
^~~~~~~~~~
kernel/sched/fair.c:5809:2: note: remove the 'if' if its condition is always false
if (!qcfs_rq->in_throttle_limbo)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
kernel/sched/fair.c:5788:37: note: initialize the variable 'top_cfs_rq' to silence this warning
struct cfs_rq *qcfs_rq, *top_cfs_rq;
^
= NULL
kernel/sched/fair.c:533:19: warning: unused function 'max_vruntime' [-Wunused-function]
static inline u64 max_vruntime(u64 max_vruntime, u64 vruntime)
^2 warnings generated.
vim +5809 kernel/sched/fair.c
5782
5783 static void throttle_cfs_rq_work(struct callback_head *work)
5784 {
5785 struct task_struct *p = container_of(work, struct task_struct, sched_throttle_work);
5786 struct sched_entity *se;
5787 struct rq *rq;
5788 struct cfs_rq *qcfs_rq, *top_cfs_rq;
5789 long task_delta, idle_task_delta;
5790
5791 WARN_ON_ONCE(p != current);
5792
5793 /*
5794 * If task is exiting, then there won't be a return to userspace, so we
5795 * don't have to bother with any of this.
5796 */
5797 if ((p->flags & PF_EXITING))
5798 return;
5799
5800 CLASS(task_rq_lock, rq_guard)(p);
5801 rq = rq_guard.rq;
5802 se = &p->se;
5803 qcfs_rq = cfs_rq_of(se);
5804
5805 /*
5806 * If not in limbo, then either replenish has happened or this task got
5807 * migrated out of the throttled cfs_rq, move along
5808 */
> 5809 if (!qcfs_rq->in_throttle_limbo)
5810 goto done;
5811
5812 update_rq_clock(rq);
5813
5814 list_add(&p->throttle_node, &qcfs_rq->throttled_limbo_list);
5815 task_delta = 1;
5816 idle_task_delta = cfs_rq_is_idle(qcfs_rq) ? 1 : 0;
5817
5818 for_each_sched_entity(se) {
5819 qcfs_rq = cfs_rq_of(se);
5820
5821 if (!se->on_rq)
5822 goto done;
5823
5824 dequeue_entity(qcfs_rq, se, DEQUEUE_SLEEP);
5825 qcfs_rq->h_nr_running -= task_delta;
5826 qcfs_rq->idle_h_nr_running -= idle_task_delta;
5827
5828 if (qcfs_rq->in_throttle_limbo)
5829 top_cfs_rq = qcfs_rq;
5830
5831 if (qcfs_rq->load.weight) {
5832 /* Avoid re-evaluating load for this entity: */
5833 se = parent_entity(se);
5834 break;
5835 }
5836 }
5837
5838 for_each_sched_entity(se) {
5839 qcfs_rq = cfs_rq_of(se);
5840 /* throttled entity or throttle-on-deactivate */
5841 if (!se->on_rq)
5842 goto throttle_done;
5843
5844 update_load_avg(qcfs_rq, se, 0);
5845 se_update_runnable(se);
5846 qcfs_rq->h_nr_running -= task_delta;
5847 qcfs_rq->h_nr_running -= idle_task_delta;
5848 }
5849
5850 throttle_done:
5851 /* At this point se is NULL and we are at root level*/
5852 sub_nr_running(rq, 1);
5853 done:
5854 p->sched_throttle_work.next = &p->sched_throttle_work;
5855 /*
5856 * XXX: this assumes the highest in_throttle_limbo cfs_rq in the
5857 * hierarchy is the one which got throttled, which might not always
5858 * be the case.
5859 */
5860 if (top_cfs_rq && !top_cfs_rq->throttled_clock)
5861 top_cfs_rq->throttled_clock = rq_clock(rq);
5862
5863 resched_curr(rq);
5864 }
5865
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Valentin Schneider
reviewerRFC PATCH 2/2 sched/fair: Repurpose cfs_rq_throttled()
In reply to Valentin Schneidercfs_rq->throttled is now never set, as cfs_rq's are never fully throttled but rather stay in limbo as tasks are slowly plucked out of them.
Get rid of cfs_rq->throttled, and repurpose cfs_rq_throttled() to use the limbo meaning.
--- kernel/sched/fair.c | 57 ++++++-------------------------------------- kernel/sched/sched.h | 3 +-- 2 files changed, 8 insertions(+), 52 deletions(-)
kernel/sched/fair.c +7 -50
@@ -5497,7 +5497,7 @@ static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)if (likely (cfs_rq->runtime_remaining >0 ))return ; -if (cfs_rq->throttled || cfs_rq->in_throttle_limbo) +if (cfs_rq->in_throttle_limbo)return ;/* * if we're unable to extend our runtime we resched so that the active
@@ -5518,7 +5518,7 @@ void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)static inline int cfs_rq_throttled (struct cfs_rq *cfs_rq) { -return cfs_bandwidth_used () && cfs_rq->throttled; +return cfs_bandwidth_used () && cfs_rq->in_throttle_limbo; }/* check whether cfs_rq, or any parent, is throttled */
@@ -5848,10 +5848,6 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) qcfs_rq->h_nr_running += task_delta; qcfs_rq->idle_h_nr_running += idle_task_delta; - -/* end evaluation on encountering a throttled cfs_rq */ -if (cfs_rq_throttled (qcfs_rq)) - goto unthrottle_throttle; }for_each_sched_entity (se) {
@@ -5862,10 +5858,6 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) qcfs_rq->h_nr_running += task_delta; qcfs_rq->idle_h_nr_running += idle_task_delta; - -/* end evaluation on encountering a throttled cfs_rq */ -if (cfs_rq_throttled (qcfs_rq)) - goto unthrottle_throttle; }/* At this point se is NULL and we are at root level*/
@@ -5909,7 +5901,7 @@ static void __cfsb_csd_unthrottle(void *arg) throttled_csd_list) {list_del_init (&cursor->throttled_csd_list); -if (cfs_rq_throttled (cursor) || cursor->in_throttle_limbo) +if (cfs_rq_throttled (cursor))unthrottle_cfs_rq (cursor); }
@@ -5949,7 +5941,7 @@ static void unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq) {lockdep_assert_rq_held (rq_of (cfs_rq)); -if (SCHED_WARN_ON (!(cfs_rq_throttled (cfs_rq) || cfs_rq->in_throttle_limbo) || +if (SCHED_WARN_ON (!cfs_rq_throttled (cfs_rq) || cfs_rq->runtime_remaining <=0 ))return ;
@@ -5982,7 +5974,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)* waiting for tasks to exit the kernel. In this case we still * want to replenish. */ -if (!cfs_rq_throttled (cfs_rq) && !cfs_rq->in_throttle_limbo) +if (!cfs_rq_throttled (cfs_rq)) goto next;/* Already queued for async unthrottle */
@@ -6031,7 +6023,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)list_del_init (&cfs_rq->throttled_csd_list); -if (cfs_rq_throttled (cfs_rq) || cfs_rq->in_throttle_limbo) +if (cfs_rq_throttled (cfs_rq))unthrottle_cfs_rq (cfs_rq);rq_unlock_irqrestore (rq, &rf);
@@ -6230,10 +6222,6 @@ static void check_enqueue_throttle(struct cfs_rq *cfs_rq)if (!cfs_rq->runtime_enabled || cfs_rq->curr)return ; -/* ensure the group is not already throttled */ -if (cfs_rq_throttled (cfs_rq)) -return ; -/* update runtime allocation */ account_cfs_rq_runtime (cfs_rq,0 );if (cfs_rq->runtime_remaining <=0 )
@@ -6266,13 +6254,6 @@ static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq)if (likely (!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining >0 ))return false; -/* -* it's possible for a throttled entity to be forced into a running -* state (e.g. set_curr_task), in this case we're finished. -*/ -if (cfs_rq_throttled (cfs_rq)) -return true; -return throttle_cfs_rq (cfs_rq); }
@@ -6705,10 +6686,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (cfs_rq_is_idle (cfs_rq)) idle_h_nr_running =1 ; -/* end evaluation on encountering a throttled cfs_rq */ -if (cfs_rq_throttled (cfs_rq)) - goto enqueue_throttle; - flags =ENQUEUE_WAKEUP ; }
@@ -6724,10 +6701,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (cfs_rq_is_idle (cfs_rq)) idle_h_nr_running =1 ; - -/* end evaluation on encountering a throttled cfs_rq */ -if (cfs_rq_throttled (cfs_rq)) - goto enqueue_throttle; }/* At this point se is NULL and we are at root level*/
@@ -6750,7 +6723,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (!task_new)update_overutilized_status (rq); -enqueue_throttle:assert_list_leaf_cfs_rq (rq);hrtick_update (rq);
@@ -6783,10 +6755,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (cfs_rq_is_idle (cfs_rq)) idle_h_nr_running =1 ; -/* end evaluation on encountering a throttled cfs_rq */ -if (cfs_rq_throttled (cfs_rq)) - goto dequeue_throttle; -/* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) {/* Avoid re-evaluating load for this entity: */
@@ -6815,10 +6783,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (cfs_rq_is_idle (cfs_rq)) idle_h_nr_running =1 ; -/* end evaluation on encountering a throttled cfs_rq */ -if (cfs_rq_throttled (cfs_rq)) - goto dequeue_throttle; - }/* At this point se is NULL and we are at root level*/
@@ -6828,7 +6792,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; -dequeue_throttle:util_est_update (&rq->cfs, p, task_sleep);hrtick_update (rq); }
@@ -9582,7 +9545,7 @@ static inline int check_misfit_status(struct rq *rq, struct sched_domain *sd)* Something like: * * { 0 1 2 3 } { 4 5 6 7 } -* * * * * +* * * * * * * If we were to balance group-wise we'd place two tasks in the first group and * two tasks in the second group. Clearly this is undesired as it will overload
@@ -12642,9 +12605,6 @@ static void propagate_entity_cfs_rq(struct sched_entity *se) {struct cfs_rq *cfs_rq =cfs_rq_of (se); -if (cfs_rq_throttled (cfs_rq)) -return ; -if (!throttled_hierarchy (cfs_rq))list_add_leaf_cfs_rq (cfs_rq);
@@ -12656,9 +12616,6 @@ static void propagate_entity_cfs_rq(struct sched_entity *se)update_load_avg (cfs_rq, se,UPDATE_TG ); -if (cfs_rq_throttled (cfs_rq)) -break ; -if (!throttled_hierarchy (cfs_rq))list_add_leaf_cfs_rq (cfs_rq); }
kernel/sched/sched.h +1 -2
@@ -631,9 +631,8 @@ struct cfs_rq {u64 throttled_clock_pelt_time;u64 throttled_clock_self;u64 throttled_clock_self_time; -int throttled; -int throttle_count;int in_throttle_limbo; +int throttle_count;/* Temp storage for updating the counts during unthrottling */ unsigned int unthrottled_h_nr_running;unsigned int unthrottled_idle_h_nr_running;
Signature · 1 line
2.41.0