tg: count the sum wait time of an task group
RFC PATCH 1/1 tg: count the sum wait time of an task group
Although we can rely on cpuacct to present the cpu usage of task group, it is hard to tell how intense the competition is between these groups on cpu resources.
Monitoring the wait time of each process could cost too much, and there is no good way to accurately represent the conflict with these info, we need the wait time on group dimension.
Thus we introduced group's wait_sum provided by kernel to represent the conflict between task groups, whenever a group's cfs_rq ends waiting, it's wait time accounted to the sum.
The cpu.stat is modified to show the new statistic, like:
nr_periods 0
nr_throttled 0
throttled_time 0
wait_sum 2035098795584
Now we can monitor the changing on wait_sum to tell how suffering a task group is in the fight of cpu resources.
--- kernel/sched/core.c | 2 ++ kernel/sched/fair.c | 4 ++++ kernel/sched/sched.h | 1 + 3 files changed, 7 insertions(+)
kernel/sched/core.c +2 -0
@@ -6787,6 +6787,8 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v)seq_printf (sf,"nr_periods %d\n" , cfs_b->nr_periods);seq_printf (sf,"nr_throttled %d\n" , cfs_b->nr_throttled);seq_printf (sf,"throttled_time %llu\n" , cfs_b->throttled_time); +if (schedstat_enabled ()) +seq_printf (sf,"wait_sum %llu\n" , tg->wait_sum);return 0 ; }
kernel/sched/fair.c +4 -0
@@ -862,6 +862,7 @@ static void update_curr_fair(struct rq *rq)static inline void update_stats_wait_end (struct cfs_rq *cfs_rq,struct sched_entity *se) { +struct task_group *tg;struct task_struct *p;u64 delta;
@@ -882,6 +883,9 @@ static void update_curr_fair(struct rq *rq)return ; }trace_sched_stat_wait (p, delta); + }else { + tg =group_cfs_rq (se)->tg; +__schedstat_add (tg->wait_sum, delta); } __schedstat_set(se->statistics.wait_max,
kernel/sched/sched.h +1 -0
@@ -358,6 +358,7 @@ struct task_group {/* runqueue "owned" by this group on each CPU */ struct cfs_rq **cfs_rq;unsigned long shares; +u64 wait_sum;#ifdef CONFIG_SMP /*
Signature · 1 line
1.8.3.1
Quoted 13 lines · 王贇
kernel/sched/fair.c +1
-0@@ -862,6 +862,7 @@ static void update_curr_fair(struct rq *rq)static inline void update_stats_wait_end (struct cfs_rq *cfs_rq,struct sched_entity *se) { +struct task_group *tg;struct task_struct *p;u64 delta;
kernel/sched/fair.c static void update_curr_fair(struct rq *rq) · line 889
Quoted 7 lines
kernel/sched/fair.c +3
-0@@ -882,6 +883,9 @@ static void update_curr_fair(struct rq *rq)return ; }trace_sched_stat_wait (p, delta); + }else { + tg =group_cfs_rq (se)->tg; +__schedstat_add (tg->wait_sum, delta); }
You're joking right? This patch is both broken and utterly insane.
You're wanting to update an effectively global variable for every schedule action (and its broken because it is without any serialization or atomics).
NAK
Hi, Peter
Quoted 24 lines · Peter Zijlstra
Quoted 13 lines · 王贇
kernel/sched/fair.c +1
-0@@ -862,6 +862,7 @@ static void update_curr_fair(struct rq *rq)static inline void update_stats_wait_end (struct cfs_rq *cfs_rq,struct sched_entity *se) { +struct task_group *tg;struct task_struct *p;u64 delta;kernel/sched/fair.c static void update_curr_fair(struct rq *rq) · line 889
Quoted 7 lines
kernel/sched/fair.c +3
-0@@ -882,6 +883,9 @@ static void update_curr_fair(struct rq *rq)return ; }trace_sched_stat_wait (p, delta); + }else { + tg =group_cfs_rq (se)->tg; +__schedstat_add (tg->wait_sum, delta); }You're joking right? This patch is both broken and utterly insane.
Quoted 3 lines
You're wanting to update an effectively global variable for every schedule action (and its broken because it is without any serialization or atomics).
Thanks for the reply and sorry for the thoughtless, I'll rewrite the code to make it per-cpu variable, then assemble the results on show.
Regards,
Michael Wang
Quoted 3 lines
NAK
PATCH 1/1 tg: show the sum wait time of an task group
In reply to 王贇Although we can rely on cpuacct to present the cpu usage of task group, it is hard to tell how intense the competition is between these groups on cpu resources.
Monitoring the wait time of each process or sched_debug could cost too much, and there is no good way to accurately represent the conflict with these info, we need the wait time on group dimension.
Thus we introduced group's wait_sum represent the conflict between task groups, which is simply sum the wait time of group's cfs_rq.
The 'cpu.stat' is modified to show the statistic, like:
nr_periods 0
nr_throttled 0
throttled_time 0
wait_sum 2035098795584
Now we can monitor the changing on wait_sum to tell how suffering a task group is in the fight of cpu resources.
For example:
(wait_sum - last_wait_sum) * 100 / (nr_cpu * period_ns) == X%
means the task group paid X percentage of period on waiting for the cpu.
---
Since RFC:
redesigned the way to acquire wait_sum
kernel/sched/core.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/sched/core.c +8 -0
@@ -6781,6 +6781,8 @@ static int __cfs_schedulable(struct task_group *tg, u64 period, u64 quota)static int cpu_cfs_stat_show (struct seq_file *sf,void *v) { +int i; +u64 wait_sum =0 ;struct task_group *tg =css_tg (seq_css (sf));struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;
@@ -6788,6 +6790,12 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v)seq_printf (sf,"nr_throttled %d\n" , cfs_b->nr_throttled);seq_printf (sf,"throttled_time %llu\n" , cfs_b->throttled_time); +if (schedstat_enabled ()) { +for_each_possible_cpu (i) + wait_sum += tg->se[i]->statistics.wait_sum; +seq_printf (sf,"wait_sum %llu\n" , wait_sum); + } + return 0; }#endif /* CONFIG_CFS_BANDWIDTH */
Signature · 1 line
1.8.3.1
PATCH v2 1/1 tg: show the sum wait time of an task group
Although we can rely on cpuacct to present the cpu usage of task group, it is hard to tell how intense the competition is between these groups on cpu resources.
Monitoring the wait time of each process or sched_debug could cost too much, and there is no good way to accurately represent the conflict with these info, we need the wait time on group dimension.
Thus we introduced group's wait_sum represent the conflict between task groups, which is simply sum the wait time of group's cfs_rq.
The 'cpu.stat' is modified to show the statistic, like:
nr_periods 0
nr_throttled 0
throttled_time 0
wait_sum 2035098795584
Now we can monitor the changing on wait_sum to tell how suffering a task group is in the fight of cpu resources.
For example:
(wait_sum - last_wait_sum) * 100 / (nr_cpu * period_ns) == X%
means the task group paid X percentage of period on waiting for the cpu.
---
Since v1:
Use schedstat_val to avoid compile error
Check and skip root_task_group
kernel/sched/core.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/sched/core.c +8 -0
@@ -6781,6 +6781,8 @@ static int __cfs_schedulable(struct task_group *tg, u64 period, u64 quota)static int cpu_cfs_stat_show (struct seq_file *sf,void *v) { +int i; +u64 ws =0 ;struct task_group *tg =css_tg (seq_css (sf));struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;
@@ -6788,6 +6790,12 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v)seq_printf (sf,"nr_throttled %d\n" , cfs_b->nr_throttled);seq_printf (sf,"throttled_time %llu\n" , cfs_b->throttled_time); +if (schedstat_enabled () && tg != &root_task_group) { +for_each_possible_cpu (i) + ws +=schedstat_val (tg->se[i]->statistics.wait_sum); +seq_printf (sf,"wait_sum %llu\n" , ws); + } + return 0; }#endif /* CONFIG_CFS_BANDWIDTH */
Signature · 1 line
1.8.3.1
Re: [PATCH v2] tg: show the sum wait time of an task group
Quoted 18 lines · 王贇
Although we can rely on cpuacct to present the cpu usage of task group, it is hard to tell how intense the competition is between these groups on cpu resources.
Monitoring the wait time of each process or
sched_debugcould cost too much, and there is no good way to accurately represent the conflict with these info, we need the wait time on group dimension.Thus we introduced group's
wait_sumrepresent the conflict between task groups, which is simply sum the wait time of group'scfs_rq.The 'cpu.stat' is modified to show the statistic, like:
nr_periods0
nr_throttled0
throttled_time0
wait_sum2035098795584
Quoted 8 lines
Now we can monitor the changing on
wait_sumto tell how suffering a task group is in the fight of cpu resources.For example:
(wait_sum-last_wait_sum) * 100 / (nr_cpu*period_ns) == X%means the task group paid X percentage of period on waiting for the cpu.
Hi, Peter
How do you think about this proposal?
There are situation that tasks in some group suffered much more than others, will be good to have some way to easily locate them.
Regards,
Michael Wang
Quoted 37 lines
Signed-off-by Michael Wang <[email protected]>---
Since v1:
Useschedstat_valto avoid compile error
Check and skiproot_task_groupkernel/sched/core.c | 8 ++++++++ 1 file changed, 8 insertions(+)kernel/sched/core.c +8
-0@@ -6781,6 +6781,8 @@ static int __cfs_schedulable(struct task_group *tg, u64 period, u64 quota) static int cpu_cfs_stat_show(struct seq_file *sf, void *v) { + int i; + u64 ws = 0; struct task_group *tg = css_tg(seq_css(sf)); struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;@@ -6788,6 +6790,12 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v) seq_printf(sf, "nr_throttled %d\n", cfs_b->nr_throttled); seq_printf(sf, "throttled_time %llu\n", cfs_b->throttled_time); + if (schedstat_enabled() && tg != &root_task_group) { + for_each_possible_cpu(i) + ws += schedstat_val(tg->se[i]->statistics.wait_sum); + seq_printf(sf, "wait_sum %llu\n", ws); + } + return 0; } #endif /* CONFIG_CFS_BANDWIDTH */
Re: [PATCH v2] tg: show the sum wait time of an task group
In reply to 王贇Hi, folks
Quoted 18 lines · 王贇
Although we can rely on cpuacct to present the cpu usage of task group, it is hard to tell how intense the competition is between these groups on cpu resources.
Monitoring the wait time of each process or
sched_debugcould cost too much, and there is no good way to accurately represent the conflict with these info, we need the wait time on group dimension.Thus we introduced group's
wait_sumrepresent the conflict between task groups, which is simply sum the wait time of group'scfs_rq.The 'cpu.stat' is modified to show the statistic, like:
nr_periods0
nr_throttled0
throttled_time0
wait_sum2035098795584
Quoted 8 lines
Now we can monitor the changing on
wait_sumto tell how suffering a task group is in the fight of cpu resources.For example:
(wait_sum-last_wait_sum) * 100 / (nr_cpu*period_ns) == X%means the task group paid X percentage of period on waiting for the cpu.
Any comments please?
Regards,
Michael Wang
Quoted 37 lines
Signed-off-by Michael Wang <[email protected]>---
Since v1:
Useschedstat_valto avoid compile error
Check and skiproot_task_groupkernel/sched/core.c | 8 ++++++++ 1 file changed, 8 insertions(+)kernel/sched/core.c +8
-0@@ -6781,6 +6781,8 @@ static int __cfs_schedulable(struct task_group *tg, u64 period, u64 quota) static int cpu_cfs_stat_show(struct seq_file *sf, void *v) { + int i; + u64 ws = 0; struct task_group *tg = css_tg(seq_css(sf)); struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;@@ -6788,6 +6790,12 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v) seq_printf(sf, "nr_throttled %d\n", cfs_b->nr_throttled); seq_printf(sf, "throttled_time %llu\n", cfs_b->throttled_time); + if (schedstat_enabled() && tg != &root_task_group) { + for_each_possible_cpu(i) + ws += schedstat_val(tg->se[i]->statistics.wait_sum); + seq_printf(sf, "wait_sum %llu\n", ws); + } + return 0; } #endif /* CONFIG_CFS_BANDWIDTH */
Peter Zijlstra
maintainerRe: [PATCH v2] tg: show the sum wait time of an task group
In reply to 王贇kernel/sched/core.c static int cpu_cfs_stat_show(struct seq_file *sf, void *v) · line 6793
Quoted 5 lines · 王贇
kernel/sched/core.c +1
-0@@ -6788,6 +6790,12 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v)seq_printf (sf,"nr_throttled %d\n" , cfs_b->nr_throttled);seq_printf (sf,"throttled_time %llu\n" , cfs_b->throttled_time); +if (schedstat_enabled () && tg != &root_task_group) {
I put the variables here.
kernel/sched/core.c static int cpu_cfs_stat_show(struct seq_file *sf, void *v) · line 6795
Quoted 2 lines
kernel/sched/core.c +2
-0+for_each_possible_cpu (i) + ws +=schedstat_val (tg->se[i]->statistics.wait_sum);
This doesn't quite work on 32bit archs, but I'm not sure I care enough to be bothered about that.
Quoted 2 lines
kernel/sched/core.c +2
-0+seq_printf (sf,"wait_sum %llu\n" , ws); + }
Re: [PATCH v2] tg: show the sum wait time of an task group
Quoted 9 lines · Peter Zijlstra
kernel/sched/core.c static int cpu_cfs_stat_show(struct seq_file *sf, void *v) · line 6793
Quoted 5 lines · 王贇
kernel/sched/core.c +1
-0@@ -6788,6 +6790,12 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v)seq_printf (sf,"nr_throttled %d\n" , cfs_b->nr_throttled);seq_printf (sf,"throttled_time %llu\n" , cfs_b->throttled_time); +if (schedstat_enabled () && tg != &root_task_group) {I put the variables here.
Will do that in next version :-)
Quoted 6 lines
kernel/sched/core.c static int cpu_cfs_stat_show(struct seq_file *sf, void *v) · line 6795
Quoted 3 lines
kernel/sched/core.c +2
-0+for_each_possible_cpu (i) + ws +=schedstat_val (tg->se[i]->statistics.wait_sum);This doesn't quite work on 32bit archs, but I'm not sure I care enough to be bothered about that.
Could easily overflow then... hope they won't really care about the group conflicts.
Regards,
Michael Wang
Quoted 3 lines
Quoted 3 lines
kernel/sched/core.c +2
-0+seq_printf (sf,"wait_sum %llu\n" , ws); + }
PATCH v3 1/1 tg: show the sum wait time of an task group
In reply to 王贇Although we can rely on cpuacct to present the cpu usage of task group, it is hard to tell how intense the competition is between these groups on cpu resources.
Monitoring the wait time of each process or sched_debug could cost too much, and there is no good way to accurately represent the conflict with these info, we need the wait time on group dimension.
Thus we introduced group's wait_sum represent the conflict between task groups, which is simply sum the wait time of group's cfs_rq.
The 'cpu.stat' is modified to show the statistic, like:
nr_periods 0
nr_throttled 0
throttled_time 0
wait_sum 2035098795584
Now we can monitor the changing on wait_sum to tell how suffering a task group is in the fight of cpu resources.
For example:
(wait_sum - last_wait_sum) * 100 / (nr_cpu * period_ns) == X%
means the task group paid X percentage of period on waiting for the cpu.
---
Since v2:
Declare variables inside branch (From Peter).
kernel/sched/core.c | 9 +++++++++ 1 file changed, 9 insertions(+)
kernel/sched/core.c +9 -0
@@ -6788,6 +6788,15 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v)seq_printf (sf,"nr_throttled %d\n" , cfs_b->nr_throttled);seq_printf (sf,"throttled_time %llu\n" , cfs_b->throttled_time); +if (schedstat_enabled () && tg != &root_task_group) { +int i; +u64 ws =0 ; + +for_each_possible_cpu (i) + ws +=schedstat_val (tg->se[i]->statistics.wait_sum); +seq_printf (sf,"wait_sum %llu\n" , ws); + } + return 0; }#endif /* CONFIG_CFS_BANDWIDTH */
Signature · 1 line
1.8.3.1
tip-bot for Yun Wang
merge bot[tip:sched/core] sched/debug: Show the sum wait time of a task group
In reply to 王贇sched/debug: Show the sum wait time of a task group
Commit-ID: 3d6c50c27bd6418dceb51642540ecfcb8ca708c2 Gitweb: https://git.kernel.org/tip/3d6c50c27bd6418dceb51642540ecfcb8ca708c2 Author: Yun Wang <[email protected]> AuthorDate: Wed, 4 Jul 2018 11:27:27 +0800 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 25 Jul 2018 11:41:05 +0200
sched/debug: Show the sum wait time of a task group
Although we can rely on cpuacct to present the CPU usage of task groups, it is hard to tell how intense the competition is between these groups on CPU resources.
Monitoring the wait time or sched_debug of each process could be very expensive, and there is no good way to accurately represent the conflict with these info, we need the wait time on group dimension.
Thus we introduce group's wait_sum to represent the resource conflict between task groups, which is simply the sum of the wait time of the group's cfs_rq.
The 'cpu.stat' is modified to show the statistic, like:
nr_periods 0
nr_throttled 0
throttled_time 0
wait_sum 2035098795584
Now we can monitor the changes of wait_sum to tell how much a a task group is suffering in the fight of CPU resources.
For example:
(wait_sum - last_wait_sum) * 100 / (nr_cpu * period_ns) == X%
means the task group paid X percentage of period on waiting for the CPU.
--- kernel/sched/core.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
kernel/sched/core.c +10 -0
@@ -6748,6 +6748,16 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v)seq_printf (sf,"nr_throttled %d\n" , cfs_b->nr_throttled);seq_printf (sf,"throttled_time %llu\n" , cfs_b->throttled_time); +if (schedstat_enabled () && tg != &root_task_group) { +u64 ws =0 ; +int i; + +for_each_possible_cpu (i) + ws +=schedstat_val (tg->se[i]->statistics.wait_sum); + +seq_printf (sf,"wait_sum %llu\n" , ws); + } + return 0; }#endif /* CONFIG_CFS_BANDWIDTH */