sched/fair: improve CFS throttle
Subsystems (3)
cover sched/fair: improve CFS throttle
Cong Wang (3):
sched/fair: make CFS bandwidth slice per cpu group sched/fair: accumulate unused cpu time for CFS throttle sched/fair: add tracepoints for cfs throttle
Documentation/scheduler/sched-bwc.txt | 25 ++++++-- include/trace/events/sched.h | 42 +++++++++++++ kernel/sched/core.c | 109 ++++++++++++++++++++++++++++++++-- kernel/sched/fair.c | 76 +++++++++++++++--------- kernel/sched/sched.h | 4 +- 5 files changed, 216 insertions(+), 40 deletions(-)
Signature · 1 line
2.13.0
PATCH 1/3 sched/fair: make CFS bandwidth slice per cpu group
We saw tasks in a CPU cgroup got throttled for many times even when they don't apparently over-burn the CPU's. I tried to trace the cause and noticed that one of the problems here is that the local CPU which got the last chunk of quota doesn't use all of them and prevents other CPUs to reuse the unused CPU time. So reducing sched_cfs_bandwidth_slice_us from the default 5ms to 1ms mostly solves the problem, at least no tasks got throttled after this change in my test case.
However, the sched_cfs_bandwidth_slice_us is a global setting which affects all cgroups. Different cgroups may want different values based on their own workload, 5ms or 1ms is not suitable for all the cgroups. A smaller slice distributes CPU time more fairly at a cost of slightly higher overhead. We have to minimize the impact of the slice change. On the other hand, the global pool filled periodically is per cgroup, each cgroup should have the right to distribute its own quota to the local CPUs with its own preferred frequency.
This patch introduces cpu.cfs_slice_us which allows each cgroup to specify its own slice length without any global impact. And the global sysctl sched_cfs_bandwidth_slice_us now becomes the default value of each cpu.cfs_slice_us. Note, updating this sysctl does not automatically update existing cgroups using a default value, users will have to update each existing cgroup accordingly to make a global change.
--- Documentation/scheduler/sched-bwc.txt | 16 ++++++++---- kernel/sched/core.c | 47 +++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 22 ++++++++++------ kernel/sched/sched.h | 1 + 4 files changed, 73 insertions(+), 13 deletions(-)
Documentation/scheduler/sched-bwc.txt +11 -5
@@ -48,19 +48,25 @@ and return the group to an unconstrained state once more. Any updates to a group's bandwidth specification will result in it becoming unthrottled if it is in a constrained state. -System wide settings --------------------- +Slice +----- For efficiency run-time is transferred between the global pool and CPU local "silos" in a batch fashion. This greatly reduces global accounting pressure on large systems. The amount transferred each time such an update is required is described as the "slice". -This is tunable via procfs: - /proc/sys/kernel/sched_cfs_bandwidth_slice_us (default=5ms) - Larger slice values will reduce transfer overheads, while smaller values allow for more fine-grained consumption. +The per-group file cpu.cfs_slice_us controls the slice length within each CPU +group, different groups could set different values for their own preference. +Its default value is tunable via procfs: + + /proc/sys/kernel/sched_cfs_bandwidth_slice_us (default=5ms) + +Note, updating this file does not automatically update existing groups using +a default slice. + Statistics ---------- A group's bandwidth statistics are exported via 3 fields in cpu.stat.
kernel/sched/core.c +47 -0
65 lines not on this page · raw message
kernel/sched/fair.c +14 -8
@@ -4527,11 +4527,6 @@ static inline u64 default_cfs_period(void)return 100000000ULL ; } -static inline u64 sched_cfs_bandwidth_slice (void ) -{ -return (u64 )sysctl_sched_cfs_bandwidth_slice *NSEC_PER_USEC ; -} -/* * Replenish runtime according to assigned quota and update expiration time. * We use sched_clock_cpu directly instead of rq->clock to avoid adding
@@ -4565,6 +4560,11 @@ static inline u64 cfs_rq_clock_task(struct cfs_rq *cfs_rq)return rq_clock_task (rq_of (cfs_rq)) - cfs_rq->throttled_clock_task_time; } +static inline u64 cfs_bandwidth_slice (struct cfs_bandwidth *cfs_b) +{ +return cfs_b->slice; +} +/* returns 0 on failure to allocate runtime */ static int assign_cfs_rq_runtime (struct cfs_rq *cfs_rq) {
@@ -4573,7 +4573,7 @@ static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq)u64 amount =0 , min_amount, expires;/* note: this is a positive sum as runtime_remaining <= 0 */ - min_amount =sched_cfs_bandwidth_slice () - cfs_rq->runtime_remaining; + min_amount =cfs_bandwidth_slice (cfs_b) - cfs_rq->runtime_remaining;raw_spin_lock (&cfs_b->lock);if (cfs_b->quota ==RUNTIME_INF )
@@ -4992,7 +4992,7 @@ static void __return_cfs_rq_runtime(struct cfs_rq *cfs_rq) cfs_b->runtime += slack_runtime;/* we are under rq->lock, defer unthrottling using a timer */ -if (cfs_b->runtime >sched_cfs_bandwidth_slice () && +if (cfs_b->runtime >cfs_bandwidth_slice (cfs_b) && !list_empty (&cfs_b->throttled_cfs_rq))start_cfs_slack_bandwidth (cfs_b); }
@@ -5019,7 +5019,7 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq)*/ static void do_sched_cfs_slack_timer (struct cfs_bandwidth *cfs_b) { -u64 runtime =0 , slice =sched_cfs_bandwidth_slice (); +u64 runtime =0 , slice =cfs_bandwidth_slice (cfs_b);u64 expires;/* confirm we're still not at a refresh boundary */
@@ -5139,12 +5139,18 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer)return idle ?HRTIMER_NORESTART :HRTIMER_RESTART ; } +static inline u64 cfs_default_bandwidth_slice (void ) +{ +return (u64 )sysctl_sched_cfs_bandwidth_slice *NSEC_PER_USEC ; +} +void init_cfs_bandwidth (struct cfs_bandwidth *cfs_b) {raw_spin_lock_init (&cfs_b->lock); cfs_b->runtime =0 ; cfs_b->quota =RUNTIME_INF ; cfs_b->period =ns_to_ktime (default_cfs_period ()); + cfs_b->slice =cfs_default_bandwidth_slice ();INIT_LIST_HEAD (&cfs_b->throttled_cfs_rq);hrtimer_init (&cfs_b->period_timer,CLOCK_MONOTONIC ,HRTIMER_MODE_ABS_PINNED );
kernel/sched/sched.h +1 -0
@@ -331,6 +331,7 @@ struct cfs_bandwidth {raw_spinlock_t lock;ktime_t period;u64 quota; +u64 slice;u64 runtime;s64 hierarchical_quota;u64 runtime_expires;
Signature · 1 line
2.13.0
PATCH 2/3 sched/fair: accumulate unused cpu time for CFS throttle
In reply to Cong WangWith current implementation of CFS CPU throttle, the unused CPU time in a period is not carried over to the next one. And when the CFS periodical timer is deactivated after detecting idle, these idle periods are not counted either. This causes some apparently counterintuitive CPU throttles when the job in a cpu cgroup mostly sleeps during each period.
This can be reproduced with the reproducer from Ivan Babrou:
https://gist.github.com/bobrik/2030ff040fad360327a5fab7a09c4ff1
(I need to run the binary directly rather using golang to run, otherwise the CPU used by golang itself is also accounted.)
We also observed a similar issue in production where QPS has an interesting anti-correlation with nr_throttled/nr_periods.
This patch addresses it by always accumulating unused CPU time in the global pool, and when the CFS periodical timer is deactivated calculating the number of idle periods and replenishing the missing quotas to the cpu cgroup. To avoid burst, a cap could be specified by user so that quotas won't exceed this cap.
This patch significantly reduces the nr_throttled/nr_periods percentage and throttled time:
Without any patch (quota = 20000, period = 100000):
nr_periods 56
nr_throttled 14
throttled_time 1496288690
With this patch (burst = 10 * quota):
nr_periods 43
nr_throttled 1
throttled_time 74789584
This patch first folds __refill_cfs_bandwidth_runtime() into tg_set_cfs_bandwidth(), then renames and reuses it for do_sched_cfs_period_timer(), where I use cfs_rq->last_active to record the last time when this timer is activated, so that we can calculate how many periods we miss during this.
--- Documentation/scheduler/sched-bwc.txt | 9 +++++ kernel/sched/core.c | 62 ++++++++++++++++++++++++++++++++--- kernel/sched/fair.c | 52 +++++++++++++++++------------ kernel/sched/sched.h | 3 +- 4 files changed, 99 insertions(+), 27 deletions(-)
Documentation/scheduler/sched-bwc.txt +9 -0
@@ -67,6 +67,15 @@ group, different groups could set different values for their own preference. Note, updating this file does not automatically update existing groups using a default slice. +Burst +----- +When a local run queue has no task to run, its unused CPU time is returned +back to the global pool. If the whole group becomes idle, its periodical +timer is halted, unused CPU time during this time is accumulated too. +To avoid burst, a cap should be set by user via cpu.cfs_burst_us, with it +the accumulated CPU time won't exceed this cap. Its default value allows +a group to burn all CPU's with the given quota for a whole period. + Statistics ---------- A group's bandwidth statistics are exported via 3 fields in cpu.stat.
kernel/sched/core.c +57 -5
87 lines not on this page · raw message
kernel/sched/fair.c +31 -21
83 lines not on this page · raw message
kernel/sched/sched.h +2 -1
@@ -333,8 +333,10 @@ struct cfs_bandwidth {u64 quota;u64 slice;u64 runtime; +u64 burst;s64 hierarchical_quota;u64 runtime_expires; +u64 last_active;int idle;int period_active;
@@ -433,7 +435,6 @@ extern void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq, struct sched_entity *parent);extern void init_cfs_bandwidth (struct cfs_bandwidth *cfs_b); -extern void __refill_cfs_bandwidth_runtime (struct cfs_bandwidth *cfs_b);extern void start_cfs_bandwidth (struct cfs_bandwidth *cfs_b);extern void unthrottle_cfs_rq (struct cfs_rq *cfs_rq);
Signature · 1 line
2.13.0
PATCH 3/3 sched/fair: add tracepoints for cfs throttle
In reply to Cong WangThis patch introduces two sched tracepoints to track CFS throttle and unthrottle. The use case is to measure how long each throttle is and to track when a CPU cgroup is throttled and unthrottled.
Sample output:
cfs-722 [000] dN.3 51.477702: sched_cfs_throttle: path=/test cpu=0 runtime_remaining=0 <idle>-0 [000] d.h4 51.536659: sched_cfs_unthrottle: path=/test cpu=0 runtime_remaining=1
--- include/trace/events/sched.h | 42 ++++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 2 ++ 2 files changed, 44 insertions(+)
include/trace/events/sched.h +42 -0
@@ -570,6 +570,48 @@ TRACE_EVENT(sched_wake_idle_without_ipi,TP_printk ("cpu=%d" , __entry->cpu) ); + +#ifdef CONFIG_CFS_BANDWIDTH +DECLARE_EVENT_CLASS (sched_fair, + + TP_PROTO(struct cfs_rq *cfs_rq), + +TP_ARGS (cfs_rq), + +TP_STRUCT__entry ( + __field( s64, runtime_remaining ) + __field( int, cpu ) +__dynamic_array (char, cfs_path, +cgroup_path (cfs_rq->tg->css.cgroup,NULL ,0 ) +1 ) + ), + +TP_fast_assign ( + __entry->runtime_remaining = cfs_rq->runtime_remaining; + __entry->cpu = cpu_of(cfs_rq->rq); +cgroup_path (cfs_rq->tg->css.cgroup, +__get_dynamic_array (cfs_path), + __get_dynamic_array_len(cfs_path)); + ), + +TP_printk ("path=%s cpu=%d runtime_remaining=%lld" ,__get_str (cfs_path), + __entry->cpu, __entry->runtime_remaining) +); + +DEFINE_EVENT (sched_fair, sched_cfs_throttle, + + TP_PROTO(struct cfs_rq *cfs_rq), + +TP_ARGS (cfs_rq) +); + +DEFINE_EVENT (sched_fair, sched_cfs_unthrottle, + + TP_PROTO(struct cfs_rq *cfs_rq), + +TP_ARGS (cfs_rq) +); +#endif /* CONFIG_CFS_BANDWIDTH */ +#endif /* _TRACE_SCHED_H */ /* This part must be outside protection */
kernel/sched/fair.c +2 -0
@@ -4736,6 +4736,7 @@ static void throttle_cfs_rq(struct cfs_rq *cfs_rq) cfs_rq->throttled =1 ; cfs_rq->throttled_clock =rq_clock (rq); +trace_sched_cfs_throttle (cfs_rq);raw_spin_lock (&cfs_b->lock); empty =list_empty (&cfs_b->throttled_cfs_rq);
@@ -4766,6 +4767,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) se = cfs_rq->tg->se[cpu_of (rq)]; cfs_rq->throttled =0 ; +trace_sched_cfs_unthrottle (cfs_rq);update_rq_clock (rq);
Signature · 1 line
2.13.0
A message that is not in the archive (sent off-list or not yet received): CANWdNRCQCteO7L+Of7T2vWBW7GbMHyvyF5aSQMVYNGg_afxNhg@mail.gmail.com
Re: [PATCH 3/3] sched/fair: add tracepoints for cfs throttle
Quoted 4 lines · Ivan Babrou
Quoted 1 line · Cong Wang
Quoted 8 lines
Quoted 4 lines
(quoted diff) +2
-0+ + TP_printk("path=%s cpu=%d runtime_remaining=%lld",
__get_str(cfs_path),
+__entry->cpu,__entry->runtime_remaining)Can you add "[ns]" as the unit to
runtime_remaining? We have it in "sched:sched_stat_runtime".
Hmm, although it is indeed ns, this ->runtime_remaining can be negative, unlike sched_stat_runtime. So I am not sure if I should marked it as "[ns]".
Peter Zijlstra
maintainerRe: [PATCH 3/3] sched/fair: add tracepoints for cfs throttle
In reply to a message not in the archiveQuoted 7 lines · Ivan Babrou
Quoted 4 lines
+ TP_printk("path=%s cpu=%d runtime_remaining=%lld",
__get_str(cfs_path),
+__entry->cpu,__entry->runtime_remaining)Can you add "[ns]" as the unit to
runtime_remaining? We have it in "sched:sched_stat_runtime".
Yeah, don't worry. I hate tracepoints, I regret the existing ones, no new ones will happen.
Re: [PATCH 3/3] sched/fair: add tracepoints for cfs throttle
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Ivan Babrou
Quoted 4 lines
+ TP_printk("path=%s cpu=%d runtime_remaining=%lld",
__get_str(cfs_path),
+__entry->cpu,__entry->runtime_remaining)Can you add "[ns]" as the unit to
runtime_remaining? We have it in "sched:sched_stat_runtime".Yeah, don't worry. I hate tracepoints, I regret the existing ones, no new ones will happen.
No matter you hate them or not, they are useful:
https://github.com/iovisor/bcc/blob/master/tools/runqlat.py
Unlike in sched, tracepoints are welcome in networking:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/include/trace/events/tcp.h
I don't want to change your mind, just want to show the facts.
Peter Zijlstra
maintainerRe: [PATCH 3/3] sched/fair: add tracepoints for cfs throttle
Quoted 6 lines · Cong Wang
Quoted 2 lines · Peter Zijlstra
Yeah, don't worry. I hate tracepoints, I regret the existing ones, no new ones will happen.
No matter you hate them or not, they are useful:
No argument there.
Quoted 2 lines
Unlike in sched, tracepoints are welcome in networking:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/include/trace/events/tcp.h
They are also absolutely disallowed in the vfs..
Quoted 1 line
I don't want to change your mind, just want to show the facts.
The problem with tracepoints is that they can become ABI and you cannot change them without breaking tools. This is a crap situation and I'm fed up with it.
Re: [PATCH 3/3] sched/fair: add tracepoints for cfs throttle
Quoted 4 lines · Peter Zijlstra
The problem with tracepoints is that they can become ABI and you cannot change them without breaking tools. This is a crap situation and I'm fed up with it.
Yeah, I once used perf_event_open() to parse the trace events, I understand how painful it is. Fortunately tools like bcc mostly hides the ABI from end user. To be fair, kprobe has the same problem if not worse, different compilers could inline different kernel functions, not to mention kernel functions could change at any time.
OTOH, the tracepoint text interface is sufficient to help people to inspect kernel internals, like in this case I can observe when a cpu cgroup is throttled or unthrottled. Of course they are not friendly for programming.
Thanks.