Measuring term of acquiring spinlock
Subsystems (9)
RFC PATCH 1/1 Measuring term of acquiring spinlock
Hi Ingo,
At first, I have to say that the patch this mail contains
is *VERY ROUGH*. Because this patch affects not only spinlock
but also wide range of kernel, including VFS, MM, and process management, etc. So I didn't format this patch to adopt to check by checkpatch.pl and not signed.
But I strongly believe that this patch will help some virtualization people (and some non-virtualization people). So I want your advice and comment on this.
* Description
This patch makes the file spinlock_stats on top of the debugfs. When user reads this file, some statistical data related to spinlocks are printed.
e.g.
name: count longest shortest total average
lock: 0 0 0 0 0i8259A_lock: 44 408852 3596 643488 14624i8253_lock: 0 0 0 0 0rtc_lock: 3 63504 6980 77740 25913mce_state_lock: 0 0 0 0 0cmci_discover_lock: 4 33512 20292 94544 23636set_atomicity_lock: 4 3247864 0 7534840 1883710
lock: 0 0 0 0 0ioapic_lock: 221 15512 1848 1264832 5723vector_lock: 82 31312 528 174716 2130gart_lock: 0 0 0 0 0iommu_bitmap_lock: 0 0 0 0 0pgd_lock: 19404 39880 42 23864027 1229cpa_lock: 2383 24836 576 1830496 768
......
The reason why I want to obtain this is that
in the virtualization world, there is the problem
called "Lock Holder Preemption" (LHP).
LHP is well described in this paper,
http://www.amd64.org/fileadmin/user_upload/pub/2008-Friebel-LHP-GI_OS.pdf
LHP is not a trivial problem. e.g. VMware employs coscheduling to avoid this. (source: http://communities.vmware.com/docs/DOC-4960 ) And the paper I linked describes the problem and the way to deal it on Xen environment.
And, this is the important point, some paper including the above measures term of acquiring spinlock.
But the authors of these didn't describe the way to measuring. (This is wonder! Is this the secret voodoo?)
So I implemented the patch to measure the term of acquiring spinlock. The result this patch products is the above.
But this patch contains a lot of trivial and non-trivial problems. So I want your advices and comments.
Below is the list of problems:
1) The way of detecting the data structures of spinlock_t This is a trivial problem.
Current strategy this patch employs is that
modify DEFINE_SPINLOCK() and add __section(.spinlock.data). So spinlock values are located in the section .spinlock.data.
.spinlock.data is the section specially created for spinlock_t. So I thought that this will be the array of spinlock_t.
But this was not. I thinks this caused by some problem related to alignment.
But the my lack of knowledge about ld, I employed temporary way to detect spinlock_t.
The way is that after setting char * value to start of .spinlock.data, increment this, and if this is equal to magic of spinlock, obtain head address of spinlock_t by container_of() macro. This way is implemented in spinlock_stats_show().
Can I make the section .spinlock.data the array of spinlock_t?
2) Affecting to wide range of kernel
Some spinlocks are declarated with __cacheline_aligned_in_smp. These declarations conflict with new DEFINE_SPINLOCK(), because __cacheline_aligned_in_smp contains __section(.data.page_aligned).
So I manually added the code to print these spinlocks to spinlock_stats_show(). Like this:
#define P(spin) do { \
count = atomic64_read(&spin.count); \
total = atomic64_read(&spin.total); \
shortest = atomic64_read(&spin.shortest); \
seq_printf(m, "%s: %ld %ld %ld %ld %ld\n", spin.dep_map.name, count, atomic64_read(&spin.longest), \
(shortest == LONG_MAX) ? 0 : shortest, total, count ? total / count : 0); \
}while(0)
P(bdev_lock);
P(dcache_lock);
P(files_lock);...
This is very ugly, and makes bad effect for maintaining the code.
3) Overhead
This is the result of 'perf bench sched messaging -g 20 -l 1000' Raw kernel:
# Running sched/messaging benchmark...
# 20 sender and receiver processes per group
# 20 groups == 800 processes run
Total time: 6.457 [sec]
Patched kernel:
# 20 sender and receiver processes per group
# 20 groups == 800 processes run
Total time: 6.607 [sec]
It seems that there is a little overhead.
I'm writing this email on the patched kernel,
and I cannot distinguish the overhead from human's perspective.
But I cannot decide the meaning of this overhead by myself. How should I think about this? Can we discard this?
Of course, I isolated the code with ifdef.
And if this patch was merged, I'll add new Kconfig option for this patch.
... these are my proposal. Sorry for long email :-( But, as I mentioned at first, I strongly believe that this patch makes amount of sense!
# And I'm working of 'perf bench mem memcpy' concurrently :)
Below is the patch. Request for comments.
fs/block_dev.c +1 -1
@@ -412,7 +412,7 @@ static int block_fsync(struct file *filp, struct dentry *dentry, int datasync)* pseudo-fs */ -static __cacheline_aligned_in_smp DEFINE_SPINLOCK (bdev_lock ); +__cacheline_aligned_in_smp DEFINE_SPINLOCK_RAW (bdev_lock );static struct kmem_cache * bdev_cachep__read_mostly ;static struct inode *bdev_alloc_inode (struct super_block *sb)
fs/dcache.c +1 -1
@@ -38,7 +38,7 @@int sysctl_vfs_cache_pressure__read_mostly =100 ;EXPORT_SYMBOL_GPL (sysctl_vfs_cache_pressure); -__cacheline_aligned_in_smp DEFINE_SPINLOCK (dcache_lock ); +__cacheline_aligned_in_smp DEFINE_SPINLOCK_RAW (dcache_lock );__cacheline_aligned_in_smp DEFINE_SEQLOCK (rename_lock );EXPORT_SYMBOL (dcache_lock);
fs/file_table.c +1 -1
@@ -31,7 +31,7 @@ struct files_stat_struct files_stat = { };/* public. Not pretty! */ -__cacheline_aligned_in_smp DEFINE_SPINLOCK (files_lock ); +__cacheline_aligned_in_smp DEFINE_SPINLOCK_RAW (files_lock );/* SLAB cache for file structures */ static struct kmem_cache *filp_cachep__read_mostly ;
fs/namespace.c +1 -1
@@ -38,7 +38,7 @@#define HASH_SIZE (1UL << HASH_SHIFT)/* spinlock for vfsmount related operations, inplace of dcache_lock */ -__cacheline_aligned_in_smp DEFINE_SPINLOCK (vfsmount_lock ); +__cacheline_aligned_in_smp DEFINE_SPINLOCK_RAW (vfsmount_lock );static int event;static DEFINE_IDA (mnt_id_ida);
include/asm-generic/vmlinux.lds.h +4 -0
@@ -154,6 +154,10 @@ VMLINUX_SYMBOL(__start___tracepoints) = .; \ *(__tracepoints ) \ VMLINUX_SYMBOL(__stop___tracepoints) = .; \ + . = ALIGN(8); \ + VMLINUX_SYMBOL(__lock_data_start) = .; \ + *(.spinlock.data) \ + VMLINUX_SYMBOL(__lock_data_end) = .; \/* implement dynamic printk debug */ \ . = ALIGN(8); \ VMLINUX_SYMBOL(__start___verbose) = .; \
include/linux/spinlock_api_smp.h +82 -0
150 lines not on this page · raw message
include/linux/spinlock_types.h +11 -0
@@ -29,6 +29,10 @@ typedef struct {#ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map;#endif + +#if defined(CONFIG_DEBUG_SPINLOCK ) && defined(CONFIG_DEBUG_LOCK_ALLOC ) +atomic64_t lock_start, longest, count, total, shortest; +#endif } spinlock_t;#define SPINLOCK_MAGIC 0xdead4ead
@@ -69,6 +73,7 @@ typedef struct { .magic = SPINLOCK_MAGIC, \ .owner = SPINLOCK_OWNER_INIT, \ .owner_cpu = -1, \ + .shortest = ATOMIC64_INIT(LONG_MAX), \ SPIN_DEP_MAP_INIT(lockname) }#define __RW_LOCK_UNLOCKED (lockname) \ (rwlock_t) { .raw_lock = __RAW_RW_LOCK_UNLOCKED, \
@@ -94,7 +99,13 @@ typedef struct {#define SPIN_LOCK_UNLOCKED __SPIN_LOCK_UNLOCKED(old_style_spin_init)#define RW_LOCK_UNLOCKED __RW_LOCK_UNLOCKED(old_style_rw_init) +#if defined(CONFIG_DEBUG_SPINLOCK ) && defined(CONFIG_DEBUG_LOCK_ALLOC ) +#define DEFINE_SPINLOCK (x) spinlock_t __section(.spinlock.data) x = __SPIN_LOCK_UNLOCKED(x) +#define DEFINE_SPINLOCK_RAW (x) spinlock_t x = __SPIN_LOCK_UNLOCKED(x) +#else #define DEFINE_SPINLOCK (x) spinlock_t x = __SPIN_LOCK_UNLOCKED(x) +#endif +#define DEFINE_RWLOCK (x) rwlock_t x = __RW_LOCK_UNLOCKED(x)#endif /* __LINUX_SPINLOCK_TYPES_H */
kernel/fork.c +1 -1
@@ -415,7 +415,7 @@ static inline void mm_free_pgd(struct mm_struct * mm)#define mm_free_pgd (mm)#endif /* CONFIG_MMU */ -__cacheline_aligned_in_smp DEFINE_SPINLOCK (mmlist_lock ); +__cacheline_aligned_in_smp DEFINE_SPINLOCK_RAW (mmlist_lock );#define allocate_mm () (kmem_cache_alloc(mm_cachep, GFP_KERNEL))#define free_mm (mm) (kmem_cache_free(mm_cachep, (mm)))
kernel/pid.c +1 -1
@@ -110,7 +110,7 @@ EXPORT_SYMBOL(is_container_init);* For now it is easier to be safe than to prove it can't happen. */ -static __cacheline_aligned_in_smp DEFINE_SPINLOCK (pidmap_lock ); +__cacheline_aligned_in_smp DEFINE_SPINLOCK_RAW (pidmap_lock );static void free_pidmap (struct upid *upid) {
kernel/spinlock.c +83 -0
92 lines not on this page · raw message
lib/kernel_lock.c +1 -1
@@ -20,7 +20,7 @@* * Don't use in new code. */ -static __cacheline_aligned_in_smp DEFINE_SPINLOCK (kernel_flag ); +__cacheline_aligned_in_smp DEFINE_SPINLOCK_RAW (kernel_flag );/*
Quoted 5 lines · Hitoshi Mitake
* Description
This patch makes the file
spinlock_statson top of the debugfs. When user reads this file, some statistical data related to spinlocks are printed.
hm, are you aware of the lockstat tracepoints? They do something like this - and it utilizes perf events to report these events. See include/trace/events/lockdep.h. Needs CONFIG_LOCK_STAT enabled.
Might make sense to put it into 'perf lock' kind of tool. I think Frederic had such plans.
Ingo
From: Ingo Molnar <[email protected]>
Subject: Re: [PATCH][RFC] Measuring term of acquiring spinlock
Date: Thu, 12 Nov 2009 08:39:09 +0100
Quoted 13 lines · Hitoshi Mitake
Quoted 6 lines
* Description
This patch makes the file
spinlock_statson top of the debugfs. When user reads this file, some statistical data related to spinlocks are printed.hm, are you aware of the lockstat tracepoints? They do something like this - and it utilizes perf events to report these events. See
include/trace/events/lockdep.h. NeedsCONFIG_LOCK_STATenabled.
Wow, I didn't know that. I'll try it.
Quoted 2 lines
Might make sense to put it into 'perf lock' kind of tool. I think Frederic had such plans.
If you want a tester for 'perf lock', please call me, Frederic :)
Quoted 13 lines · Hitoshi Mitake
From: Ingo Molnar <[email protected]>
Subject: Re: [PATCH][RFC] Measuring term of acquiring spinlock
Date: Thu, 12 Nov 2009 08:39:09 +0100Quoted 9 lines · Hitoshi Mitake
Quoted 6 lines
* Description
This patch makes the file
spinlock_statson top of the debugfs. When user reads this file, some statistical data related to spinlocks are printed.
Quoted 6 lines
Quoted 4 lines
hm, are you aware of the lockstat tracepoints? They do something like this - and it utilizes perf events to report these events. See
include/trace/events/lockdep.h. NeedsCONFIG_LOCK_STATenabled.Wow, I didn't know that. I'll try it.
Btw., i think we should rename that tracepoint category from 'lockdep' to 'lock'. It's possible to enable them without CONFIG_PROVE_LOCKING, and they arent high-overhead in that case.
Ingo
Peter Zijlstra
maintainerQuoted 5 lines · Ingo Molnar
Btw., i think we should rename that tracepoint category from 'lockdep' to 'lock'. It's possible to enable them without
CONFIG_PROVE_LOCKING, and they arent high-overhead in that case.
They're still in 'lockdep', and it does still require enabling all of the lock debugging code, so the overhead is far from zero.
Quoted 21 lines · Ingo Molnar
Quoted 18 lines · Hitoshi Mitake
From: Ingo Molnar <[email protected]>
Subject: Re: [PATCH][RFC] Measuring term of acquiring spinlock
Date: Thu, 12 Nov 2009 08:39:09 +0100Quoted 13 lines · Hitoshi Mitake
Quoted 6 lines
* Description
This patch makes the file
spinlock_statson top of the debugfs. When user reads this file, some statistical data related to spinlocks are printed.hm, are you aware of the lockstat tracepoints? They do something like this - and it utilizes perf events to report these events. See
include/trace/events/lockdep.h. NeedsCONFIG_LOCK_STATenabled.
Quoted 7 lines
Quoted 1 line
Wow, I didn't know that. I'll try it.
Btw., i think we should rename that tracepoint category from 'lockdep' to 'lock'. It's possible to enable them without
CONFIG_PROVE_LOCKING, and they arent high-overhead in that case.Ingo
I have a pending patch for that somewhere. I can send it right away.
Quoted 24 lines · Frederic Weisbecker
Quoted 23 lines · Ingo Molnar
Quoted 19 lines · Hitoshi Mitake
From: Ingo Molnar <[email protected]>
Subject: Re: [PATCH][RFC] Measuring term of acquiring spinlock
Date: Thu, 12 Nov 2009 08:39:09 +0100Quoted 13 lines · Hitoshi Mitake
Quoted 6 lines
* Description
This patch makes the file
spinlock_statson top of the debugfs. When user reads this file, some statistical data related to spinlocks are printed.hm, are you aware of the lockstat tracepoints? They do something like this - and it utilizes perf events to report these events. See
include/trace/events/lockdep.h. NeedsCONFIG_LOCK_STATenabled.Wow, I didn't know that. I'll try it.
Quoted 8 lines
Quoted 5 lines
Btw., i think we should rename that tracepoint category from 'lockdep' to 'lock'. It's possible to enable them without
CONFIG_PROVE_LOCKING, and they arent high-overhead in that case.Ingo
I have a pending patch for that somewhere. I can send it right away.
Cool - thanks!
Ingo
PATCH 1/1 tracing: Rename lockdep event subsystem into lock
Lockdep events subsystem gathers various locking related events such as a request, release, contention or acquisition of a lock.
The name of this event subsystem is a bit of a misnomer since these events are not quite related to lockdep but more generally to locking,
ie: these events are not reporting lock dependencies or possible deadlock scenario but pure locking events.
Hence this rename.
---
include/trace/events/{lockdep.h => lock.h} | 8 ++++----
kernel/lockdep.c | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
rename include/trace/events/{lockdep.h => lock.h} (92%)include/trace/events/lock.h renamed from include/trace/events/lockdep.h +4 -4
@@ -1,8 +1,8 @@#undef TRACE_SYSTEM -#define TRACE_SYSTEM lockdep +#define TRACE_SYSTEM lock -#if !defined(_TRACE_LOCKDEP_H) || defined(TRACE_HEADER_MULTI_READ ) -#define _TRACE_LOCKDEP_H +#if !defined(_TRACE_LOCK_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _TRACE_LOCK_H#include <linux/lockdep.h> #include <linux/tracepoint.h>
@@ -90,7 +90,7 @@ TRACE_EVENT(lock_acquired,#endif #endif -#endif /* _TRACE_LOCKDEP_H */ +#endif /* _TRACE_LOCK_H */ /* This part must be outside protection */ #include <trace/define_trace.h>
kernel/lockdep.c +1 -1
@@ -49,7 +49,7 @@#include "lockdep_internals.h" #define CREATE_TRACE_POINTS -#include <trace/events/lockdep.h> +#include <trace/events/lock.h> #ifdef CONFIG_PROVE_LOCKING int prove_locking =1 ;
Signature · 1 line
1.6.2.3
Peter Zijlstra
maintainerRe: [PATCH] tracing: Rename lockdep event subsystem into lock
Quoted 7 lines · Frederic Weisbecker
Lockdep events subsystem gathers various locking related events such as a request, release, contention or acquisition of a lock.
The name of this event subsystem is a bit of a misnomer since these events are not quite related to lockdep but more generally to locking,
ie: these events are not reporting lock dependencies or possible deadlock scenario but pure locking events.
But in order to get them you need pretty much all of lockdep, except PROVE_LOCKING. You get all the lock debugging, the lock tracking, the struct dep_map bloat etc.
But sure, I don't mind renaming the category.
Re: [PATCH] tracing: Rename lockdep event subsystem into lock
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Frederic Weisbecker
Lockdep events subsystem gathers various locking related events such as a request, release, contention or acquisition of a lock.
The name of this event subsystem is a bit of a misnomer since these events are not quite related to lockdep but more generally to locking, ie: these events are not reporting lock dependencies or possible deadlock scenario but pure locking events.
Quoted 5 lines
But in order to get them you need pretty much all of lockdep, except
PROVE_LOCKING. You get all the lock debugging, the lock tracking, the structdep_mapbloat etc.But sure, I don't mind renaming the category.
Yeah, i'd prefer it this way. You are right that most of lockdep.o is still there, but most of the lockdep _overhead_ shouldnt be there - no hashing, no tracking, etc.
it's still nonzero - see 'perf top' of a hackbench session, with LOCK_STAT enabled and PROVE_LOCKING disabled:
------------------------------------------------------------------------------
PerfTop: 14059 irqs/sec kernel:99.8% [1000Hz cycles], (all, 16 CPUs)
------------------------------------------------------------------------------
samples pcnt function DSO
_______ _____ ________________________________ ________________
7320.00 7.9% sched_clock_local [kernel]
7217.00 7.8% lock_acquired [kernel]
5768.00 6.2% trace_hardirqs_off [kernel]
4562.00 4.9% __lock_acquire [kernel]
4304.00 4.6% lock_release [kernel]
3838.00 4.1% lock_acquire [kernel]
3833.00 4.1% look_up_lock_class [kernel]
3561.00 3.8% cpu_clock [kernel]
3283.00 3.5% start_critical_timing [kernel]
2992.00 3.2% __alloc_skb [kernel]
2680.00 2.9% acpi_pm_read [kernel]
2498.00 2.7% sched_clock [kernel]
2409.00 2.6% copy_user_generic_string [kernel]
2016.00 2.2% lock_release_holdtime [kernel]
1899.00 2.0% sched_clock_cpu [kernel]but i think those should be gradually eliminated and improved, as lock statistics could become a quite popular thing.
If the tracepoints are named 'lock', people will expect less overhead, and might end up fixing/improving it. If it's named 'lockdep' on the other hand, the expectation is higher overhead.
Ingo
Re: [PATCH] tracing: Rename lockdep event subsystem into lock
In reply to Peter ZijlstraQuoted 4 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
Lockdep events subsystem gathers various locking related events such as a request, release, contention or acquisition of a lock.
Quoted 8 lines
Quoted 4 lines
The name of this event subsystem is a bit of a misnomer since these events are not quite related to lockdep but more generally to locking,
ie: these events are not reporting lock dependencies or possible deadlock scenario but pure locking events.But in order to get them you need pretty much all of lockdep, except
PROVE_LOCKING. You get all the lock debugging, the lock tracking, the structdep_mapbloat etc.
Sure. But I think an event susbsystem name should reflect its actual purpose despite the machinery that makes it work.
[tip:perf/core] tracing: Rename 'lockdep' event subsystem into 'lock'
In reply to Frederic WeisbeckerCommit-ID: b8b901965dbc6572f365457bb8c6a487f7da5d02 Gitweb: http://git.kernel.org/tip/b8b901965dbc6572f365457bb8c6a487f7da5d02 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 13 Nov 2009 10:06:34 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 13 Nov 2009 10:28:13 +0100
tracing: Rename 'lockdep' event subsystem into 'lock'
Lockdep events subsystem gathers various locking related events such as a request, release, contention or acquisition of a lock.
The name of this event subsystem is a bit of a misnomer since these events are not quite related to lockdep but more generally to locking, ie: these events are not reporting lock dependencies or possible deadlock scenario but pure locking events.
Hence this rename.
LKML-Reference: <[email protected]>
--- include/trace/events/lockdep.h | 8 ++++---- kernel/lockdep.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)
include/trace/events/lockdep.h +4 -4
@@ -1,8 +1,8 @@#undef TRACE_SYSTEM -#define TRACE_SYSTEM lockdep +#define TRACE_SYSTEM lock -#if !defined(_TRACE_LOCKDEP_H) || defined(TRACE_HEADER_MULTI_READ ) -#define _TRACE_LOCKDEP_H +#if !defined(_TRACE_LOCK_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _TRACE_LOCK_H#include <linux/lockdep.h> #include <linux/tracepoint.h>
@@ -90,7 +90,7 @@ TRACE_EVENT(lock_acquired,#endif #endif -#endif /* _TRACE_LOCKDEP_H */ +#endif /* _TRACE_LOCK_H */ /* This part must be outside protection */ #include <trace/define_trace.h>
kernel/lockdep.c +1 -1
@@ -49,7 +49,7 @@#include "lockdep_internals.h" #define CREATE_TRACE_POINTS -#include <trace/events/lockdep.h> +#include <trace/events/lock.h> #ifdef CONFIG_PROVE_LOCKING int prove_locking =1 ;
[tip:perf/core] tracing: Rename 'lockdep' event subsystem into 'lock'
In reply to Frederic WeisbeckerCommit-ID: 67178767b936fb47a3a5e88097cff41ccbda7acb Gitweb: http://git.kernel.org/tip/67178767b936fb47a3a5e88097cff41ccbda7acb Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 13 Nov 2009 10:06:34 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 13 Nov 2009 10:48:27 +0100
tracing: Rename 'lockdep' event subsystem into 'lock'
Lockdep events subsystem gathers various locking related events such as a request, release, contention or acquisition of a lock.
The name of this event subsystem is a bit of a misnomer since these events are not quite related to lockdep but more generally to locking, ie: these events are not reporting lock dependencies or possible deadlock scenario but pure locking events.
Hence this rename.
LKML-Reference: <[email protected]>
---
include/trace/events/{lockdep.h => lock.h} | 8 ++++----
kernel/lockdep.c | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)include/trace/events/lock.h renamed from include/trace/events/lockdep.h +4 -4
@@ -1,8 +1,8 @@#undef TRACE_SYSTEM -#define TRACE_SYSTEM lockdep +#define TRACE_SYSTEM lock -#if !defined(_TRACE_LOCKDEP_H) || defined(TRACE_HEADER_MULTI_READ ) -#define _TRACE_LOCKDEP_H +#if !defined(_TRACE_LOCK_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _TRACE_LOCK_H#include <linux/lockdep.h> #include <linux/tracepoint.h>
@@ -90,7 +90,7 @@ TRACE_EVENT(lock_acquired,#endif #endif -#endif /* _TRACE_LOCKDEP_H */ +#endif /* _TRACE_LOCK_H */ /* This part must be outside protection */ #include <trace/define_trace.h>
kernel/lockdep.c +1 -1
@@ -49,7 +49,7 @@#include "lockdep_internals.h" #define CREATE_TRACE_POINTS -#include <trace/events/lockdep.h> +#include <trace/events/lock.h> #ifdef CONFIG_PROVE_LOCKING int prove_locking =1 ;
Quoted 4 lines · Hitoshi Mitake
Quoted 2 lines
Might make sense to put it into 'perf lock' kind of tool. I think Frederic had such plans.
If you want a tester for 'perf lock', please call me, Frederic :)
Thanks :)
I haven't started it yet, because of some other things I need to finish.
Would you be interested in starting it?
The lock tracepoints are:
- lock_acquire: the lock is requested but not yet "acquired"
- lock_release
- lock_contended: we can't get the lock yet because someone already holds it.
- lock_acquired: we got the lock
So, the point would be to develop a state machine that is able to give us statistics on a per lock granularity basis:
- the total/average/maximum time a lock is contended, waited before being acquired (probably lock_acquired time - lock_acquire time)
- the ratio of contention versus immediate availability => Sequence acquire - acquired means immediate availability. If we have a contended event inside it means it had to wait because of the contention.
- the total/average/maximum time a lock is held.
It would be very nice to also add an option to see the callsites sorted from max to min, if possible with a per boundary granularity: the couple (lock callsite, unlock callsite), still on a per lock basis.
- any idea you could have...
Such a tool would be very useful to profile the kernel locking.
It would be nice to use design close to what perf sched does:
having an lock event structure that provides callbacks for each lock events so that we can easily plug various plugin inside.
It's just a suggestion in case you are interested and have time for that. Otherwise I'll do it later.
Hm?
Thanks,
Frederic.
From: Frederic Weisbecker <[email protected]>
Subject: Re: [PATCH][RFC] Measuring term of acquiring spinlock
Date: Fri, 13 Nov 2009 11:51:27 +0100
Quoted 12 lines · Hitoshi Mitake
Quoted 4 lines
Quoted 2 lines
Might make sense to put it into 'perf lock' kind of tool. I think Frederic had such plans.
If you want a tester for 'perf lock', please call me, Frederic :)
Thanks :)
I haven't started it yet, because of some other things I need to finish.Would you be interested in starting it?
Yes, I'm very interested in it!
Quoted 35 lines
The lock tracepoints are:
-
lock_acquire: the lock is requested but not yet "acquired"
-lock_release
-lock_contended: we can't get the lock yet because someone already holds it.
-lock_acquired: we got the lockSo, the point would be to develop a state machine that is able to give us statistics on a per lock granularity basis:
- the total/average/maximum time a lock is contended, waited before being acquired (probably
lock_acquiredtime -lock_acquiretime)- the ratio of contention versus immediate availability => Sequence acquire - acquired means immediate availability. If we have a contended event inside it means it had to wait because of the contention.
- the total/average/maximum time a lock is held.
It would be very nice to also add an option to see the callsites sorted from max to min, if possible with a per boundary granularity: the couple (lock callsite, unlock callsite), still on a per lock basis.- any idea you could have...
Such a tool would be very useful to profile the kernel locking.
Quoted 8 lines
It would be nice to use design close to what perf sched does:
having an lock event structure that provides callbacks for each lock events so that we can easily plug various plugin inside.It's just a suggestion in case you are interested and have time for that. Otherwise I'll do it later.
Hm?
I'd like to do that.
But I'm an only newbie, so it may take a week (or more). So If you finish this work, please post and disregard me :)
Hitoshi
Quoted 6 lines · Hitoshi Mitake
Quoted 4 lines
Thanks :)
I haven't started it yet, because of some other things I need to finish.Would you be interested in starting it?
Yes, I'm very interested in it!
Great!
Quoted 6 lines
Quoted 6 lines
Such a tool would be very useful to profile the kernel locking.
It would be nice to use design close to what perf sched does:
having an lock event structure that provides callbacks for each lock events so that we can easily plug various plugin inside.
Quoted 7 lines
Quoted 4 lines
It's just a suggestion in case you are interested and have time for that. Otherwise I'll do it later.
Hm?
I'd like to do that.
But I'm an only newbie, so it may take a week (or more).
Don't worry about that. Take your time.
Quoted 1 line
So If you finish this work, please post and disregard me :)
No if you take it I won't start a concurrent work.
Don't hesitate if you have questions. This will be the first tool (at least that I'm aware of) that post-processes the lock event so you may encounter weird things, missing events, unappropriate sequences, or any bad things we haven't yet seen.
And don't forget to use -M on perf record to record
the lock events so that you have them multiplexed across cpus and hence well ordered wrt time. If later we need something that scales better, we can still drop the use of -M and do the reordering from userspace.
Thanks.
From: Frederic Weisbecker <[email protected]>
Subject: Re: [PATCH][RFC] Measuring term of acquiring spinlock
Date: Sun, 15 Nov 2009 03:21:38 +0100
Quoted 34 lines · Hitoshi Mitake
Quoted 6 lines
Quoted 4 lines
Thanks :)
I haven't started it yet, because of some other things I need to finish.Would you be interested in starting it?
Yes, I'm very interested in it!
Great!
Quoted 6 lines
Quoted 6 lines
Such a tool would be very useful to profile the kernel locking.
It would be nice to use design close to what perf sched does:
having an lock event structure that provides callbacks for each lock events so that we can easily plug various plugin inside.Quoted 7 lines
Quoted 4 lines
It's just a suggestion in case you are interested and have time for that. Otherwise I'll do it later.
Hm?
I'd like to do that.
But I'm an only newbie, so it may take a week (or more).Don't worry about that. Take your time.
Quoted 1 line
So If you finish this work, please post and disregard me :)
Quoted 6 lines
No if you take it I won't start a concurrent work.
Don't hesitate if you have questions. This will be the first tool (at least that I'm aware of) that post-processes the lock event so you may encounter weird things, missing events, unappropriate sequences, or any bad things we haven't yet seen.
Thanks. Of course, I'll not hesitate at questioning :)
Quoted 6 lines
And don't forget to use -M on perf record to record
the lock events so that you have them multiplexed across cpus and hence well ordered wrt time. If later we need something that scales better, we can still drop the use of -M and do the reordering from userspace.
Thanks for your advice! I'll do my best.
PATCH 1/2 Does raw_field_ptr() supports __data_loc?
In reply to Frederic WeisbeckerVERY very sorry for my long silence...
Please don't commit this patch, Ingo.
This patch is required to test the next patch for perf lock.
I have a question for Steven Rostedt and Tom Zanussi. Does current raw_field_ptr() support __data_loc?
At 064739bc4b3d7f424b2f25547e6611bcf0132415 , support for the modifier "__data_loc" of format is added.
But, when I wanted to parse format of lock_acquired (or some event else), raw_field_ptr() did not returned correct pointer.
So I modified raw_field_ptr() like this patch. Then raw_field_ptr() works well. Parser was large so I'd like to ask you about this.
I have a question
--- tools/perf/util/trace-event-parse.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)
tools/perf/util/trace-event-parse.c +7 -0
@@ -1925,6 +1925,13 @@ void *raw_field_ptr(struct event *event, const char *name, void *data)if (!field)return NULL ; +if (field->flags &FIELD_IS_STRING ) { +int offset; + offset = *(int *)(data + field->offset); + offset &=0xffff ; +return data + offset; + } +return data + field->offset; }
Signature · 1 line
1.6.5.2
Re: [PATCH 1/2] Does raw_field_ptr() supports __data_loc?
Quoted 10 lines · Hitoshi Mitake
VERY very sorry for my long silence...
Please don't commit this patch, Ingo.
This patch is required to test the next patch for perf lock.I have a question for Steven Rostedt and Tom Zanussi. Does current
raw_field_ptr()support__data_loc?At
064739bc4b3d7f424b2f25547e6611bcf0132415, support for the modifier "__data_loc" of format is added.
I guess this patch handled the nature of the strings, but the current end use is in perl. There is no support of it in raw_field_ptr().
Your patch looks good to me..
Re: [PATCH 1/2] Does raw_field_ptr() supports __data_loc?
In reply to Hitoshi MitakeQuoted 33 lines · Hitoshi Mitake
VERY very sorry for my long silence...
Please don't commit this patch, Ingo.
This patch is required to test the next patch for perf lock.I have a question for Steven Rostedt and Tom Zanussi. Does current
raw_field_ptr()support__data_loc?At
064739bc4b3d7f424b2f25547e6611bcf0132415, support for the modifier "__data_loc" of format is added.But, when I wanted to parse format of
lock_acquired(or some event else),raw_field_ptr()did not returned correct pointer.So I modified
raw_field_ptr()like this patch. Thenraw_field_ptr()works well. Parser was large so I'd like to ask you about this.Cc Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Paul Mackerras <[email protected]>Cc Tom Zanussi <[email protected]>Cc Steven Rostedt <[email protected]>I have a question
--- tools/perf/util/trace-event-parse.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)tools/perf/util/trace-event-parse.c +0
-0
tools/perf/util/trace-event-parse.c void *raw_field_ptr(struct event *event, const char *name, void *data) · line 1934
Quoted 10 lines
tools/perf/util/trace-event-parse.c +7
-0@@ -1925,6 +1925,13 @@ void *raw_field_ptr(struct event *event, const char *name, void *data)if (!field)return NULL ; +if (field->flags &FIELD_IS_STRING ) { +int offset; + offset = *(int *)(data + field->offset); + offset &=0xffff ; +return data + offset; + } +
Hmm, I guess I need to update the code in perf again. The way I did this was to have FIELD_IS_STRING evaluated in the eval_num_arg.
-- Steve
Quoted 3 lines
tools/perf/util/trace-event-parse.c +0
-0return data + field->offset; }
PATCH 2/2 perf lock: New subcommand "lock" to perf for analyzing lock statistics
In reply to Frederic WeisbeckerThis patch adds new subcommand "lock" to perf for analyzing lock usage statistics. Current perf lock is very primitive. This cannot provide the minimum functions. Of course I continue to working on this.
But too big patch is not good thing for you, so I post this.
And I found some important problem, so I'd like to ask your opinion.
For another issue, this patch depends on the previous one.
The previous one is very dirty and temporary, I cannot sign on it, so I cannot sign on this too...
First, it seems that current locks (spinlock, rwlock, mutex) has no numeric ID. So we can't treat rq->lock on CPU 0 and rq->lock on CPU 1 as different things. Symbol name of locks cannot be complete ID.
This is the result of current ugly data structure for lock_stat (data structure for stat per lock in builtin-lock.c).
Hash table will solve the problem of speed,
but it is not a radical solution.
I understand it is hard to implement numeric IDs for locks,
but it is required seriously, do you have some ideas?
Second, there's a lot of lack of information from trace events. For example, current lock event subsystem cannot provide the time between lock_acquired and lock_release.
But this time is already measured in lockdep, and we can obtain it from /proc/lock_stat.
But /proc/lock_stat provides information from boot time only. So I have to modify wide area of kernel including lockdep, may I do this?
Third, siginificant overhead :-(
% perf bench sched messaging # Without perf lock rec
Total time: 0.436 [sec]
% sudo ./perf lock rec perf bench sched messaging # With perf lock rec
Total time: 4.677 [sec]
[ perf record: Woken up 0 times to write data ]
[ perf record: Captured and wrote 106.065 MB perf.data (~4634063 samples) ]
Over 10 times! No one can ignore this...
This is example of using perf lock prof:
% sudo ./perf lock prof # Outputs in pager
------------------------------------------------------------------------------------------
Lock | Acquired | Max wait ns | Min wait ns | Total wait ns |
--------------------------------------------------------------------------------------------
&q->lock 30 0 0 0
&ctx->lock 3912 0 0 0
event_mutex 2 0 0 0
&newf->file_lock 1008 0 0 0
dcache_lock 444 0 0 0
&dentry->d_lock 1164 0 0 0
&ctx->mutex 2 0 0 0
&child->perf_event_mutex 2 0 0 0
&event->child_mutex 18 0 0 0
&f->f_lock 2 0 0 0
&event->mmap_mutex 2 0 0 0
&sb->s_type->i_mutex_key 259 0 0 0
&sem->wait_lock 27205 0 0 0
&(&ip->i_iolock)->mr_lock 130 0 0 0
&(&ip->i_lock)->mr_lock 6376 0 0 0
&parent->list_lock 9149 7367 146 527013
&inode->i_data.tree_lock 12175 0 0 0
&inode->i_data.private_lock 6097 0 0 0--- tools/perf/Makefile | 1 + tools/perf/builtin-lock.c | 449 +++++++++++++++++++++++++++++++++++++++++++++ tools/perf/builtin.h | 1 + tools/perf/perf.c | 1 + 4 files changed, 452 insertions(+), 0 deletions(-) create mode 100644 tools/perf/builtin-lock.c
tools/perf/Makefile +1 -0
@@ -437,6 +437,7 @@ BUILTIN_OBJS += builtin-top.o BUILTIN_OBJS += builtin-trace.o BUILTIN_OBJS += builtin-probe.o BUILTIN_OBJS += builtin-kmem.o +BUILTIN_OBJS += builtin-lock.o PERFLIBS = $(LIB_FILE)
tools/perf/builtin-lock.c new file +449 -0
449 lines not on this page · raw message
tools/perf/builtin.h +1 -0
@@ -29,5 +29,6 @@ extern int cmd_trace(int argc, const char **argv, const char *prefix);extern int cmd_version (int argc,const char **argv,const char *prefix);extern int cmd_probe (int argc,const char **argv,const char *prefix);extern int cmd_kmem (int argc,const char **argv,const char *prefix); +extern int cmd_lock (int argc,const char **argv,const char *prefix);#endif
tools/perf/perf.c +1 -0
@@ -300,6 +300,7 @@ static void handle_internal_command(int argc, const char **argv) { "sched", cmd_sched, 0 }, { "probe", cmd_probe, 0 }, { "kmem", cmd_kmem, 0 }, + { "lock", cmd_lock, 0 }, };unsigned int i;static const char ext[] =STRIP_EXTENSION ;
Signature · 1 line
1.6.5.2
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
Quoted 4 lines · Hitoshi Mitake
This patch adds new subcommand "lock" to perf for analyzing lock usage statistics. Current perf lock is very primitive. This cannot provide the minimum functions. Of course I continue to working on this.
But too big patch is not good thing for you, so I post this.
Oh great!
Yeah, the work can be done incrementally.
Quoted 4 lines
And I found some important problem, so I'd like to ask your opinion.
For another issue, this patch depends on the previous one.
The previous one is very dirty and temporary, I cannot sign on it, so I cannot sign on this too...
The previous one looks rather good actually.
Quoted 9 lines
First, it seems that current locks (spinlock, rwlock, mutex) has no numeric ID. So we can't treat
rq->lockon CPU 0 andrq->lockon CPU 1 as different things. Symbol name of locks cannot be complete ID.
This is the result of current ugly data structure forlock_stat(data structure for stat per lock inbuiltin-lock.c).
Hash table will solve the problem of speed,
but it is not a radical solution.
I understand it is hard to implement numeric IDs for locks,
but it is required seriously, do you have some ideas?
Indeed. I think every lock instance has its own lockdep_map. And this lockdep_map is passed in every lock event but is only used to retrieve the name of the lock.
Why not adding the address of the lockdep_map in the event?
Quoted 7 lines
Second, there's a lot of lack of information from trace events. For example, current lock event subsystem cannot provide the time between
lock_acquiredandlock_release.
But this time is already measured in lockdep, and we can obtain it from/proc/lock_stat.
But/proc/lock_statprovides information from boot time only. So I have to modify wide area of kernel including lockdep, may I do this?
I think this is more something to compute in a state machine:lock_release - lock_acquired event.
This is what we do with sched events in perf sched latency
Also I think we should remove the field that gives the time waited between lock_acquire and lock_acquired. This is more something that should be done in userspace instead of calculating in from the kernel. This brings overhead in the wrong place.
Quoted 7 lines
Third, siginificant overhead :-(
% perf bench sched messaging # Without perf lock rec
Total time: 0.436 [sec]
Quoted 7 lines
% sudo ./perf lock rec perf bench sched messaging # With perf lock rec
Total time: 4.677 [sec][ perf record: Woken up 0 times to write data ]
[ perf record: Captured and wrote 106.065 MB perf.data (~4634063 samples) ]Over 10 times! No one can ignore this...
I think that the lock events are much more sensible than the sched events, and that by nature: these are very high frequency events class, probably the highest among every event classes we have (the worst beeing function tracing :)
But still, you're right, there are certainly various things we need to optimize in this area.
More than 8 times slower is high.
Quoted 16 lines
This is example of using perf lock prof:
% sudo ./perf lock prof # Outputs in pager
------------------------------------------------------------------------------------------
Lock | Acquired | Max wait ns | Min wait ns | Total wait ns |
--------------------------------------------------------------------------------------------&q->lock 30 0 0 0 &ctx->lock 3912 0 0 0 event_mutex 2 0 0 0 &newf->file_lock 1008 0 0 0 dcache_lock 444 0 0 0 &dentry->d_lock 1164 0 0 0 &ctx->mutex 2 0 0 0 &child->perf_event_mutex 2 0 0 0 &event->child_mutex 18 0 0 0 &f->f_lock 2 0 0 0
Quoted 8 lines
&event->mmap_mutex 2 0 0 0
&sb->s_type->i_mutex_key 259 0 0 0
&sem->wait_lock 27205 0 0 0
&(&ip->i_iolock)->mr_lock 130 0 0 0
&(&ip->i_lock)->mr_lock 6376 0 0 0
&parent->list_lock 9149 7367 146 527013
&inode->i_data.tree_lock 12175 0 0 0
&inode->i_data.private_lock 6097 0 0 0Very nice and promising!
I can't wait to try it.
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
Quoted 9 lines · Frederic Weisbecker
Quoted 4 lines · Hitoshi Mitake
This patch adds new subcommand "lock" to perf for analyzing lock usage statistics. Current perf lock is very primitive. This cannot provide the minimum functions. Of course I continue to working on this.
But too big patch is not good thing for you, so I post this.Oh great!
Yeah, the work can be done incrementally.
[...]
Quoted 5 lines
Very nice and promising!
I can't wait to try it.
ok, to ease testing i've created a new (and not yet permanent) topic tree for it to track this new perf feature: tip:perf/lock and pushed it out.
Note: because it's not yet in a final form i have not merged it into tip:master yet - when you are working on these bits you need to do this manually via:
git merge tip/perf/lock
Also, we might need to rebase this branch as it's WIP, so the commit IDs are not permanent yet. But i thought it would be easier to do deltas on this basis.
Hitoshi-san, the patches did not have a Signed-off-by line from you, can i add them for you?
Also, i agree that the performance aspect is probably the most pressing issue. Note that 'perf bench sched messaging' is very locking intense so a 10x slowdown is not entirely unexpected - we still ought to optimize it all some more. 'perf lock' is an excellent testcase for this in any case.
Thanks,
Ingo
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
Quoted 6 lines · Ingo Molnar
Also, i agree that the performance aspect is probably the most pressing issue. Note that 'perf bench sched messaging' is very locking intense so a 10x slowdown is not entirely unexpected - we still ought to optimize it all some more. 'perf lock' is an excellent testcase for this in any case.
Here are some test results to show the overhead of lockdep trace events:
select pagefault mmap Memory par Cont_SW
latency latency latency R/W BD latency
disable ftrace 0 0 0 0 0
enable all ftrace -16.65% -109.80% -93.62% 0.14% -6.94%
enable all ftrace -2.67% 1.08% -3.65% -0.52% -0.68%
except lockdep
We also found big overhead when using kernbench and fio, but we haven't verified whether it's caused by lockdep events.
Thanks,
Xiao
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
From: Xiao Guangrong <[email protected]>
Subject: Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
Date: Mon, 07 Dec 2009 16:38:03 +0800
Hi Xiao,
Quoted 18 lines
Quoted 6 lines · Ingo Molnar
Also, i agree that the performance aspect is probably the most pressing issue. Note that 'perf bench sched messaging' is very locking intense so a 10x slowdown is not entirely unexpected - we still ought to optimize it all some more. 'perf lock' is an excellent testcase for this in any case.
Here are some test results to show the overhead of lockdep trace events:
select pagefault mmap Memory par
Cont_SW
latency latency latency R/W BD latencydisable ftrace 0 0 0 0 0
Quoted 8 lines
enable all ftrace -16.65% -109.80% -93.62% 0.14% -6.94%
enable all ftrace -2.67% 1.08% -3.65% -0.52% -0.68%
except lockdepWe also found big overhead when using kernbench and fio, but we haven't verified whether it's caused by lockdep events.
Thanks for your terrible but important data.
Difference between "enable all ftrace" and "enable all ftrace except lockdep" is significant... This must be reduced.
Thanks
Hitoshi
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
In reply to Xiao GuangrongQuoted 17 lines · Xiao Guangrong
Quoted 6 lines · Ingo Molnar
Also, i agree that the performance aspect is probably the most pressing issue. Note that 'perf bench sched messaging' is very locking intense so a 10x slowdown is not entirely unexpected - we still ought to optimize it all some more. 'perf lock' is an excellent testcase for this in any case.
Here are some test results to show the overhead of lockdep trace events:
select pagefault mmap Memory par
Cont_SW
latency latency latency R/W BD latencydisable ftrace 0 0 0 0 0
Quoted 8 lines
enable all ftrace -16.65% -109.80% -93.62% 0.14% -6.94%
enable all ftrace -2.67% 1.08% -3.65% -0.52% -0.68%
except lockdepWe also found big overhead when using kernbench and fio, but we haven't verified whether it's caused by lockdep events.
Well, it is expected that recording all locking is going to have a substantial overhead. In my measurements, a typical event takes around 250ns per event (note, I've gotten this down to 140ns in recent updates, and even 90ns by disabling integrity checks, but I don't want to disable those checks in production).
Anyway, if you add just 100ns to every lock taken in the kernel, that will definitely increase the overhead. Just enable spin_lock() in the function tracer and watch the performance go down. This is why, when using the function tracer I usually add all locking to the notrace filter. This alone helps tremendously in tracing functions.
-- Steve
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
In reply to Xiao GuangrongQuoted 24 lines · Xiao Guangrong
Quoted 6 lines · Ingo Molnar
Also, i agree that the performance aspect is probably the most pressing issue. Note that 'perf bench sched messaging' is very locking intense so a 10x slowdown is not entirely unexpected - we still ought to optimize it all some more. 'perf lock' is an excellent testcase for this in any case.
Here are some test results to show the overhead of lockdep trace events:
select pagefault mmap Memory par
Cont_SW
latency latency latency R/W BD latencydisable ftrace 0 0 0 0 0
enable all ftrace -16.65% -109.80% -93.62% 0.14% -6.94%
enable all ftrace -2.67% 1.08% -3.65% -0.52% -0.68%
except lockdep
Quoted 5 lines
We also found big overhead when using kernbench and fio, but we haven't verified whether it's caused by lockdep events.
Thanks,
Xiao
This profile has been done using ftrace with perf right? It might be because the lock events are high rate events and fill a lot of perf buffer space. More than other events. In one of your previous mails, you showed us the difference of the size of perf.data by capturing either scheduler events or lock events.
And IIRC, the case of lock events resulted in a 100 MB perf.data whereas it was a small file for sched events.
The overhead in the pagefault and mmap latency could then result in the fact we have much more events to save, walking through much more pages in perf buffer, then faulting more often, etc.
Plus the fact various locks are taken in mmap and fault path, generating more lock events.
Just a guess...
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
Quoted 32 lines · Frederic Weisbecker
Quoted 24 lines · Xiao Guangrong
Quoted 6 lines · Ingo Molnar
Also, i agree that the performance aspect is probably the most pressing issue. Note that 'perf bench sched messaging' is very locking intense so a 10x slowdown is not entirely unexpected - we still ought to optimize it all some more. 'perf lock' is an excellent testcase for this in any case.
Here are some test results to show the overhead of lockdep trace events:
select pagefault mmap Memory par
Cont_SW
latency latency latency R/W BD latencydisable ftrace 0 0 0 0 0
enable all ftrace -16.65% -109.80% -93.62% 0.14% -6.94%
enable all ftrace -2.67% 1.08% -3.65% -0.52% -0.68%
except lockdepQuoted 5 lines
We also found big overhead when using kernbench and fio, but we haven't verified whether it's caused by lockdep events.
Thanks,
Xiao
Quoted 6 lines
This profile has been done using ftrace with perf right? It might be because the lock events are high rate events and fill a lot of perf buffer space. More than other events. In one of your previous mails, you showed us the difference of the size of perf.data by capturing either scheduler events or lock events.
I'm not sure who sent this email actually. May be you or Hitoshi.
But, anyway you got the point :)
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
In reply to Frederic WeisbeckerHi Frederic,
Quoted 1 line · Frederic Weisbecker
This profile has been done using ftrace with perf right?
We not use perf, just operate it by debugfs, the test procedure is like below:
case 1: disable all trace events:
echo nop > debugfs/tracing/current_tracer
echo > debugfs/tracing/set_event
run benchmarks
...
case 2: enable all trace events except lockdep:
echo nop > debugfs/tracing/current_tracer
echo *:* > debugfs/tracing/set_event
echo 0 > debugfs/tracing/events/lockdep/enable
run benchmarks
...
Quoted 4 lines
In one of your previous mails, you showed us the difference of the size of perf.data by capturing either scheduler events or lock events.
It's not my work :-)
Thanks,
Xiao
Quoted 15 lines
And IIRC, the case of lock events resulted in a 100 MB perf.data whereas it was a small file for sched events.
The overhead in the pagefault and mmap latency could then result in the fact we have much more events to save, walking through much more pages in perf buffer, then faulting more often, etc.
Plus the fact various locks are taken in mmap and fault path, generating more lock events.
Just a guess...
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
Subject: Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
Date: Mon, 7 Dec 2009 08:27:52 +0100
Quoted 27 lines · Frederic Weisbecker
Quoted 10 lines · Hitoshi Mitake
Quoted 4 lines
This patch adds new subcommand "lock" to perf for analyzing lock usage statistics. Current perf lock is very primitive. This cannot provide the minimum functions. Of course I continue to working on this.
But too big patch is not good thing for you, so I post this.Oh great!
Yeah, the work can be done incrementally.[...]
Quoted 5 lines
Very nice and promising!
I can't wait to try it.
ok, to ease testing i've created a new (and not yet permanent) topic tree for it to track this new perf feature: tip:perf/lock and pushed it out.
Note: because it's not yet in a final form i have not merged it into tip:master yet - when you are working on these bits you need to do this manually via:
Quoted 5 lines
git merge tip/perf/lock
Also, we might need to rebase this branch as it's WIP, so the commit IDs are not permanent yet. But i thought it would be easier to do deltas on this basis.
Thanks!
Quoted 3 lines
Hitoshi-san, the patches did not have a Signed-off-by line from you, can i add them for you?
Yes of course. I didn't signed
because these were too experimental things for master branch. But I can sign on these as experimental things on experimental branch.
Quoted 6 lines
Also, i agree that the performance aspect is probably the most pressing issue. Note that 'perf bench sched messaging' is very locking intense so a 10x slowdown is not entirely unexpected - we still ought to optimize it all some more. 'perf lock' is an excellent testcase for this in any case.
Yeah, as I described in my reply to Frederic, separating lockdep and lock events for perf lock might be solution for performance problem. I'll try it.
Thanks
Hitoshi
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
In reply to Frederic WeisbeckerFrom: Frederic Weisbecker <[email protected]>
Subject: Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
Date: Mon, 7 Dec 2009 05:41:26 +0100
Frederic, thanks for your comment!
Quoted 8 lines
Quoted 4 lines
And I found some important problem, so I'd like to ask your opinion.
For another issue, this patch depends on the previous one.
The previous one is very dirty and temporary, I cannot sign on it, so I cannot sign on this too...The previous one looks rather good actually.
Thanks for your review in previous mail.
I'm new to perf, so I didn't have confidence.
Your advice is encouraging!
Quoted 14 lines
Quoted 12 lines
First, it seems that current locks (spinlock, rwlock, mutex) has no numeric ID. So we can't treat
rq->lockon CPU 0 andrq->lockon CPU 1 as different things. Symbol name of locks cannot be complete ID.
This is the result of current ugly data structure forlock_stat(data structure for stat per lock inbuiltin-lock.c).
Hash table will solve the problem of speed,
but it is not a radical solution.
I understand it is hard to implement numeric IDs for locks,
but it is required seriously, do you have some ideas?
Quoted 5 lines
Indeed. I think every lock instance has its own
lockdep_map. And thislockdep_mapis passed in every lock event but is only used to retrieve the name of the lock.Why not adding the address of the
lockdep_mapin the event?
It's good idea. Address cannot be used as index of array directly, but dealing with it is far easier than string and low cost.
Quoted 12 lines
Quoted 9 lines
Second, there's a lot of lack of information from trace events. For example, current lock event subsystem cannot provide the time between
lock_acquiredandlock_release.
But this time is already measured in lockdep, and we can obtain it from/proc/lock_stat.
But/proc/lock_statprovides information from boot time only. So I have to modify wide area of kernel including lockdep, may I do this?
Quoted 4 lines
I think this is more something to compute in a state machine:
lock_release-lock_acquiredevent.This is what we do with sched events in perf sched latency
Yeah, tracing state of the lock is smart way. I'll try it.
Quoted 5 lines
Also I think we should remove the field that gives the time waited between
lock_acquireandlock_acquired. This is more something that should be done in userspace instead of calculating in from the kernel. This brings overhead in the wrong place.
I agree. I think we can exploit more information from timestamps.
Quoted 18 lines
Quoted 9 lines
Third, siginificant overhead :-(
% perf bench sched messaging # Without perf lock rec
Total time: 0.436 [sec]Quoted 7 lines
% sudo ./perf lock rec perf bench sched messaging # With perf lock rec
Total time: 4.677 [sec][ perf record: Woken up 0 times to write data ]
[ perf record: Captured and wrote 106.065 MB perf.data (~4634063 samples) ]Over 10 times! No one can ignore this...
Quoted 8 lines
I think that the lock events are much more sensible than the sched events, and that by nature: these are very high frequency events class, probably the highest among every event classes we have (the worst beeing function tracing :)
But still, you're right, there are certainly various things we need to optimize in this area.
More than 8 times slower is high.
It seems that lockdep contains some O(n) codes.
Of course lockdep is important, but analyzing statistics of lock usage is another problem.
I think separating lockdep and lock event for stats can be solution.
Quoted 29 lines
Quoted 18 lines
This is example of using perf lock prof:
% sudo ./perf lock prof # Outputs in pager
------------------------------------------------------------------------------------------
Lock | Acquired | Max wait ns | Min wait ns | Total wait ns |
--------------------------------------------------------------------------------------------&q->lock 30 0 0 0 &ctx->lock 3912 0 0 0 event_mutex 2 0 0 0 &newf->file_lock 1008 0 0 0 dcache_lock 444 0 0 0 &dentry->d_lock 1164 0 0 0 &ctx->mutex 2 0 0 0 &child->perf_event_mutex 2 0 0 0 &event->child_mutex 18 0 0 0 &f->f_lock 2 0 0 0Quoted 8 lines
&event->mmap_mutex 2 0 0 0 &sb->s_type->i_mutex_key 259 0 0 0 &sem->wait_lock 27205 0 0 0 &(&ip->i_iolock)->mr_lock 130 0 0 0 &(&ip->i_lock)->mr_lock 6376 0 0 0 &parent->list_lock 9149 7367 146 527013 &inode->i_data.tree_lock 12175 0 0 0 &inode->i_data.private_lock 6097 0 0 0
Quoted 5 lines
Very nice and promising!
I can't wait to try it.
Thanks! I'll do my best :)
Re: [PATCH 2/2] perf lock: New subcommand "lock" to perf for analyzing lock statistics
Quoted 9 lines · Hitoshi Mitake
From: Frederic Weisbecker <[email protected]>
Quoted 5 lines
Indeed. I think every lock instance has its own
lockdep_map. And thislockdep_mapis passed in every lock event but is only used to retrieve the name of the lock.Why not adding the address of the
lockdep_mapin the event?It's good idea. Address cannot be used as index of array directly, but dealing with it is far easier than string and low cost.
Yeah, dealing with lock strings as identifier is like dealing with the lock class instead of the lock instance. It would require a very tricky state machine to sort it out.
The address of the lockdep map would be useful as an identifier. Probably a hashlist based on it would be a good way: we can use the hashlist from the kernel headers (list.h) in perf tools already.
Quoted 7 lines
Quoted 7 lines
I think that the lock events are much more sensible than the sched events, and that by nature: these are very high frequency events class, probably the highest among every event classes we have (the worst beeing function tracing :)
But still, you're right, there are certainly various things we need to optimize in this area.
Quoted 6 lines
Quoted 1 line
More than 8 times slower is high.
It seems that lockdep contains some O(n) codes.
Of course lockdep is important, but analyzing statistics of lock usage is another problem.
I think separating lockdep and lock event for stats can be solution.
Indeed.
Thanks!