latencytop lock usage improvement
Subsystems (4)
cover latencytop lock usage improvement
Hi All,
latencytop is a very nice tool for tracing system latency hotspots, and we heavily use it in our LKP test suites.
However, we found in some benchmark tests, there are very severe lock contentions which hits 70%+ of CPU cycles in perf profile, especially for benchmark involving massive process scheduling on platforms with many CPUs, like running hackbench
"hackbench -g 1408 --process --pipe -l 1875 -s 1024"
on a 2 sockets Xeon E5-2699 machine (44 Cores/88 CPUs) with 64GB RAM.
And due to that, we have to explicitly disable latencytop for those test cases.
By checking the code, we found latencytop use one global spinlock to cover the latency data updating for both global data and per-task ones.
So initially we tried splitting single lock into one global lock and per-task lock for better lock granularity, but the lock contention is only reduced slightly (only 1% drop for perf profile). The reason is the contention is still severe, as the benchmarks cause massive scheduling on the 88 CPUs, and every schedule-in call will try to acquire the lock.
Then we tried to reduce the operations inside the latency_lock's protection (between the spin_lock_irqsave/raw_spin_unlock_irqrestore pair), and also there is only very small improvement, and lock contention keeps high.
At last, we tried adding one extra lazy mode which only update the global latency data when a task exit, while still updating the per-task data in real time. This reduces the lock contention from 70%+ to less than 5% while boost that hackbench case's throughput by 276%.
Please help to review, thanks!
Patch 1/3 adds the missing sysctl description for "latencytop" and I think it could be merged independently.
Patch 2/3 splits latency_lock to global lock and per task lock. And actually, a more aggressive thought is the per-task lock may not be needed as the per-task data is only updated at the enqueueing time for a task, which implies no race condtion for it.
Patch 3/3 implements the lazy mode and update the document.
Thanks,
Feng
Feng Tang (3):
kernel/sysctl: add description for "latencytop"
latencytop: split latency_lock to global lock and per task lock
latencytop: add a lazy mode for updating global data
Documentation/sysctl/kernel.txt | 23 ++++++++++++++++++++ include/linux/latencytop.h | 5 +++++ include/linux/sched.h | 1 + init/init_task.c | 3 +++ kernel/exit.c | 2 ++ kernel/fork.c | 4 ++++ kernel/latencytop.c | 47 +++++++++++++++++++++++++++++++++++------ 7 files changed, 78 insertions(+), 7 deletions(-)
Signature · 1 line
2.7.4
RFC PATCH 1/3 kernel/sysctl: add description for "latencytop"
The body of description is mostly copied from comments in kernel/latencytop.c
--- Documentation/sysctl/kernel.txt | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
Documentation/sysctl/kernel.txt +18 -0
@@ -43,6 +43,7 @@ show up in /proc/sys/kernel: - hyperv_record_panic_msg - kexec_load_disabled - kptr_restrict +- latencytop - l2cr [ PPC only ] - modprobe ==> Documentation/debugging-modules.txt - modules_disabled
@@ -437,6 +438,23 @@ When kptr_restrict is set to (2), kernel pointers printed using ============================================================== +latencytop: + +This value controls whether to start collecting kernel latency +data, it is off (0) by default, and could be switched on (1). +The latency talked here is not the 'traditional' interrupt +latency (which is primarily caused by something else consuming CPU), +but instead, it is the latency an application encounters because +the kernel sleeps on its behalf for various reasons. + +The info is exported via /proc/latency_stats and /proc/<pid>/latency. + +This file shows up only if CONFIG_LATENCYTOP is enabled, and please +be noted that turning it on may bring notable sytstem overhead when +there are massive scheduling in system. + +============================================================== + l2cr: (PPC only) This flag controls the L2 cache of G3 processor boards. If
Signature · 1 line
2.7.4
RFC PATCH 2/3 latencytop: split latency_lock to global lock and per task lock
In reply to Feng TangCurrently there is one global "latency_lock" to cover both the global and per-task latency data updating. Splitting it into one global lock and per-task one will improve lock's granularity and reduce the contention.
--- include/linux/sched.h | 1 + init/init_task.c | 3 +++ kernel/fork.c | 4 ++++ kernel/latencytop.c | 9 +++++---- 4 files changed, 13 insertions(+), 4 deletions(-)
include/linux/sched.h +1 -0
@@ -1095,6 +1095,7 @@ struct task_struct {unsigned long dirty_paused_when;#ifdef CONFIG_LATENCYTOP +raw_spinlock_t latency_lock;int latency_record_count;struct latency_record latency_record[LT_SAVECOUNT ];#endif
init/init_task.c +3 -0
@@ -158,6 +158,9 @@ struct task_struct init_task .numa_group = NULL, .numa_faults = NULL,#endif +#ifdef CONFIG_LATENCYTOP + .latency_lock = __RAW_SPIN_LOCK_UNLOCKED(init_task.latency_lock), +#endif #ifdef CONFIG_KASAN .kasan_depth = 1,#endif
kernel/fork.c +4 -0
@@ -1963,6 +1963,10 @@ static __latent_entropy struct task_struct *copy_process(#ifdef TIF_SYSCALL_EMU clear_tsk_thread_flag (p,TIF_SYSCALL_EMU );#endif + +#ifdef CONFIG_LATENCYTOP +raw_spin_lock_init (&p->latency_lock); +#endif clear_all_latency_tracing (p);/* ok, now we should be set up.. */
kernel/latencytop.c +5 -4
@@ -74,10 +74,10 @@ void clear_all_latency_tracing(struct task_struct *p)if (!latencytop_enabled)return ; -raw_spin_lock_irqsave (&latency_lock, flags); +raw_spin_lock_irqsave (&p->latency_lock, flags);memset (&p->latency_record,0 ,sizeof (p->latency_record)); p->latency_record_count =0 ; -raw_spin_unlock_irqrestore (&latency_lock, flags); +raw_spin_unlock_irqrestore (&p->latency_lock, flags); }static void clear_global_latency_tracing (void )
@@ -194,9 +194,10 @@ __account_scheduler_latency(struct task_struct *tsk, int usecs, int inter)store_stacktrace (tsk, &lat);raw_spin_lock_irqsave (&latency_lock, flags); -account_global_scheduler_latency (tsk, &lat); +raw_spin_unlock_irqrestore (&latency_lock, flags); +raw_spin_lock_irqsave (&tsk->latency_lock, flags);for (i =0 ; i < tsk->latency_record_count; i++) {struct latency_record *mylat;int same =1 ;
@@ -234,7 +235,7 @@ __account_scheduler_latency(struct task_struct *tsk, int usecs, int inter)memcpy (&tsk->latency_record[i], &lat,sizeof (struct latency_record )); out_unlock: -raw_spin_unlock_irqrestore (&latency_lock, flags); +raw_spin_unlock_irqrestore (&tsk->latency_lock, flags); }static int lstats_show (struct seq_file *m,void *v)
Signature · 1 line
2.7.4
RFC PATCH 3/3 latencytop: add a lazy mode for updating global latency data
In reply to Feng Tanglatencytop is a nice tool for tracing system latency hotspots, and we heavily use it in 0day/LKP test suites.
However, When running some scheduler benchmarks like hackbench, we noticed in some cases the global latencytop_lock will occupy around 70% of CPU cycles from perf profile, mainly come from contention for "latency_lock" inside __account_scheduler_latency(), as when system is running with workload that causes massive process scheduling, most of the processes contends for this global lock. Given that, we have to disable the latencytop when running such benchmarks.
Add an extra lazy mode option, which will only update the global latency data when a task exits, and this greatly reduces the possible lock contion for "latency_lock". And with this new lazy mode, the lock contention for latency_lock could be cut from 70% to less than 3% (perf profile data), and there is a hackbench throughput boost :
v5.0 v5.0 + patches
---------------- ---------------------------
540207 +267.6% 1986052 hackbench.throughputThe test we run is on a 2 sockets Xeon E5-2699 machine (44 Cores/88 CPUs) with 64GB RAM, with cmd:
"hackbench -g 1408 --process --pipe -l 1875 -s 1024"
As a new mode is added, the sysctl "kernel.latencytop" and /proc/sys/kernel/latencytop are changed as follows:
0 - Disabled
1 - Enabled (normal mode): update the global data each time task
gets scheduled (same as before the patch)
2 - Enabled (lazy mode): update the global data only when a task
exists
--- Documentation/sysctl/kernel.txt | 17 +++++++++++------ include/linux/latencytop.h | 5 +++++ kernel/exit.c | 2 ++ kernel/latencytop.c | 41 +++++++++++++++++++++++++++++++++++++---- 4 files changed, 55 insertions(+), 10 deletions(-)
Documentation/sysctl/kernel.txt +11 -6
@@ -440,12 +440,17 @@ When kptr_restrict is set to (2), kernel pointers printed using latencytop: -This value controls whether to start collecting kernel latency -data, it is off (0) by default, and could be switched on (1). -The latency talked here is not the 'traditional' interrupt -latency (which is primarily caused by something else consuming CPU), -but instead, it is the latency an application encounters because -the kernel sleeps on its behalf for various reasons. +This value controls whether and how to collect kernel latency +data, it is off (0) by default. The latency talked here is not +the 'traditional' interrupt latency (which is primarily caused by +something else consuming CPU), but instead, it is the latency an +application encounters because the kernel sleeps on its behalf +for various reasons. + +0 - Disabled +1 - Enabled (normal mode): update the global data each time task + gets scheduled +2 - Enabled (lazy mode): update the global data only when a task exists The info is exported via /proc/latency_stats and /proc/<pid>/latency.
include/linux/latencytop.h +5 -0
@@ -41,6 +41,7 @@ void clear_all_latency_tracing(struct task_struct *p);extern int sysctl_latencytop (struct ctl_table *table,int write,void __user *buffer,size_t *lenp,loff_t *ppos); +extern void update_task_latency_data (struct task_struct *tsk);#else static inline void
@@ -52,6 +53,10 @@ static inline void clear_all_latency_tracing(struct task_struct *p) { } +static inline void update_task_latency_data (struct task_struct *tsk) +{ +} +#endif #endif
kernel/exit.c +2 -0
@@ -859,6 +859,8 @@ void __noreturn do_exit(long code) tsk->exit_code = code;taskstats_exit (tsk, group_dead); +update_task_latency_data (tsk); +exit_mm ();if (group_dead)
kernel/latencytop.c +37 -4
65 lines not on this page · raw message
Signature · 1 line
2.7.4
Peter Zijlstra
maintainerQuoted 4 lines · Feng Tang
Hi All,
latencytop is a very nice tool for tracing system latency hotspots, and we heavily use it in our LKP test suites.
What data does latency-top give that perf cannot give you? Ideally we'd remove latencytop entirely.
Hi Peter,
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Feng Tang
Hi All,
latencytop is a very nice tool for tracing system latency hotspots, and we heavily use it in our LKP test suites.
What data does latency-top give that perf cannot give you? Ideally we'd remove latencytop entirely.
Thanks for the review. In 0day/LKP test service, we have many tools for monitoring and analyzing the test results, perf is the most important one, which has the most parts in our auto-generated comparing results. For example to identify spinlock contentions and system hotspots.
latencytop is another tool we used to find why systems go idle, like why workload chose to sleep or waiting for something.
Thanks,
Feng
Peter Zijlstra
maintainerQuoted 12 lines · Feng Tang
Hi Peter,
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Feng Tang
Hi All,
latencytop is a very nice tool for tracing system latency hotspots, and we heavily use it in our LKP test suites.
What data does latency-top give that perf cannot give you? Ideally we'd remove latencytop entirely.
Quoted 7 lines
Thanks for the review. In 0day/LKP test service, we have many tools for monitoring and analyzing the test results, perf is the most important one, which has the most parts in our auto-generated comparing results. For example to identify spinlock contentions and system hotspots.
latencytop is another tool we used to find why systems go idle, like why workload chose to sleep or waiting for something.
You're not answering the question; why can't you use perf for that? ISTR we explicitly added support for things like that.
Hi Peter,
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Feng Tang
Hi Peter,
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Feng Tang
Hi All,
latencytop is a very nice tool for tracing system latency hotspots, and we heavily use it in our LKP test suites.
What data does latency-top give that perf cannot give you? Ideally we'd remove latencytop entirely.
Thanks for the review. In 0day/LKP test service, we have many tools for monitoring and analyzing the test results, perf is the most important one, which has the most parts in our auto-generated comparing results. For example to identify spinlock contentions and system hotspots.
Quoted 5 lines
Quoted 2 lines
latencytop is another tool we used to find why systems go idle, like why workload chose to sleep or waiting for something.
You're not answering the question; why can't you use perf for that? ISTR we explicitly added support for things like that.
I was not very familiar with perf before. And after my last reply, I googled a little, and found "perf sched latency" has the simliar function, except I can't directly get the call chain, any suggestion for this? thanks!
- Feng