lockdep: Make lockstats counting per cpu
PATCH 1/1 lockdep: Make lockstats counting per cpu
Locking statistics are implemented using global atomic variables. This is usually fine unless some path write them very often.
This is the case for the function and function graph tracers that disable irqs for each entry saved (except if the function tracer is in preempt disabled only mode).
And calls to local_irq_save/restore() increment hardirqs_on_events and hardirqs_off_events stats (or similar stats for redundant versions).
Incrementing these global vars for each function ends up in too much cache bouncing if lockstats are enabled.
To solve this, implement the debug_atomic_*() operations using per cpu vars. We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but local_t operations are.
This version then uses local_t based per cpu counters.
--- kernel/lockdep.c | 28 ++++++++-------- kernel/lockdep_internals.h | 71 +++++++++++++++++++++++++++++++------------- 2 files changed, 64 insertions(+), 35 deletions(-)
kernel/lockdep.c +14 -14
@@ -430,20 +430,20 @@ static struct stack_trace lockdep_init_trace = {/* * Various lockdep statistics: */ -atomic_t chain_lookup_hits; -atomic_t chain_lookup_misses; -atomic_t hardirqs_on_events; -atomic_t hardirqs_off_events; -atomic_t redundant_hardirqs_on; -atomic_t redundant_hardirqs_off; -atomic_t softirqs_on_events; -atomic_t softirqs_off_events; -atomic_t redundant_softirqs_on; -atomic_t redundant_softirqs_off; -atomic_t nr_unused_locks; -atomic_t nr_cyclic_checks; -atomic_t nr_find_usage_forwards_checks; -atomic_t nr_find_usage_backwards_checks; +DEFINE_PER_CPU (local_t, chain_lookup_hits); +DEFINE_PER_CPU (local_t, chain_lookup_misses); +DEFINE_PER_CPU (local_t, hardirqs_on_events); +DEFINE_PER_CPU (local_t, hardirqs_off_events); +DEFINE_PER_CPU (local_t, redundant_hardirqs_on); +DEFINE_PER_CPU (local_t, redundant_hardirqs_off); +DEFINE_PER_CPU (local_t, softirqs_on_events); +DEFINE_PER_CPU (local_t, softirqs_off_events); +DEFINE_PER_CPU (local_t, redundant_softirqs_on); +DEFINE_PER_CPU (local_t, redundant_softirqs_off); +DEFINE_PER_CPU (local_t, nr_unused_locks); +DEFINE_PER_CPU (local_t, nr_cyclic_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_forwards_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_backwards_checks);#endif /*
kernel/lockdep_internals.h +50 -21
79 lines not on this page · raw message
Signature · 1 line
1.6.2.3
PATCH 1/1 lockdep: Make lockstats counting per cpu
Locking statistics are implemented using global atomic variables. This is usually fine unless some path write them very often.
This is the case for the function and function graph tracers that disable irqs for each entry saved (except if the function tracer is in preempt disabled only mode).
And calls to local_irq_save/restore() increment hardirqs_on_events and hardirqs_off_events stats (or similar stats for redundant versions).
Incrementing these global vars for each function ends up in too much cache bouncing if lockstats are enabled.
To solve this, implement the debug_atomic_*() operations using per cpu vars. We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but local_t oprations are.
This version then uses local_t based per cpu counters.
v2: Use per_cpu() instead of get_cpu_var() to fetch the desired cpu vars on debug_atomic_read()
--- kernel/lockdep.c | 28 ++++++++-------- kernel/lockdep_internals.h | 71 +++++++++++++++++++++++++++++++------------- 2 files changed, 64 insertions(+), 35 deletions(-)
kernel/lockdep.c +14 -14
@@ -430,20 +430,20 @@ static struct stack_trace lockdep_init_trace = {/* * Various lockdep statistics: */ -atomic_t chain_lookup_hits; -atomic_t chain_lookup_misses; -atomic_t hardirqs_on_events; -atomic_t hardirqs_off_events; -atomic_t redundant_hardirqs_on; -atomic_t redundant_hardirqs_off; -atomic_t softirqs_on_events; -atomic_t softirqs_off_events; -atomic_t redundant_softirqs_on; -atomic_t redundant_softirqs_off; -atomic_t nr_unused_locks; -atomic_t nr_cyclic_checks; -atomic_t nr_find_usage_forwards_checks; -atomic_t nr_find_usage_backwards_checks; +DEFINE_PER_CPU (local_t, chain_lookup_hits); +DEFINE_PER_CPU (local_t, chain_lookup_misses); +DEFINE_PER_CPU (local_t, hardirqs_on_events); +DEFINE_PER_CPU (local_t, hardirqs_off_events); +DEFINE_PER_CPU (local_t, redundant_hardirqs_on); +DEFINE_PER_CPU (local_t, redundant_hardirqs_off); +DEFINE_PER_CPU (local_t, softirqs_on_events); +DEFINE_PER_CPU (local_t, softirqs_off_events); +DEFINE_PER_CPU (local_t, redundant_softirqs_on); +DEFINE_PER_CPU (local_t, redundant_softirqs_off); +DEFINE_PER_CPU (local_t, nr_unused_locks); +DEFINE_PER_CPU (local_t, nr_cyclic_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_forwards_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_backwards_checks);#endif /*
kernel/lockdep_internals.h +50 -21
79 lines not on this page · raw message
Signature · 1 line
1.6.2.3
I forgot to mention in the title of this patch, this is the "v2".
Quoted 158 lines · Frederic Weisbecker
Locking statistics are implemented using global atomic variables. This is usually fine unless some path write them very often.
This is the case for the function and function graph tracers that disable irqs for each entry saved (except if the function tracer is in preempt disabled only mode).
And calls to local_irq_save/restore() incrementhardirqs_on_eventsandhardirqs_off_eventsstats (or similar stats for redundant versions).Incrementing these global vars for each function ends up in too much cache bouncing if lockstats are enabled.
To solve this, implement the debug_atomic_*() operations using per cpu vars. We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but
local_toprations are.This version then uses
local_tbased per cpu counters.v2: Use
per_cpu()instead ofget_cpu_var()to fetch the desired cpu vars ondebug_atomic_read()Suggested-by Steven Rostedt <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>--- kernel/lockdep.c | 28 ++++++++-------- kernel/lockdep_internals.h | 71 +++++++++++++++++++++++++++++++------------- 2 files changed, 64 insertions(+), 35 deletions(-)kernel/lockdep.c +14
-14@@ -430,20 +430,20 @@ static struct stack_trace lockdep_init_trace = {/* * Various lockdep statistics: */ -atomic_t chain_lookup_hits; -atomic_t chain_lookup_misses; -atomic_t hardirqs_on_events; -atomic_t hardirqs_off_events; -atomic_t redundant_hardirqs_on; -atomic_t redundant_hardirqs_off; -atomic_t softirqs_on_events; -atomic_t softirqs_off_events; -atomic_t redundant_softirqs_on; -atomic_t redundant_softirqs_off; -atomic_t nr_unused_locks; -atomic_t nr_cyclic_checks; -atomic_t nr_find_usage_forwards_checks; -atomic_t nr_find_usage_backwards_checks; +DEFINE_PER_CPU (local_t, chain_lookup_hits); +DEFINE_PER_CPU (local_t, chain_lookup_misses); +DEFINE_PER_CPU (local_t, hardirqs_on_events); +DEFINE_PER_CPU (local_t, hardirqs_off_events); +DEFINE_PER_CPU (local_t, redundant_hardirqs_on); +DEFINE_PER_CPU (local_t, redundant_hardirqs_off); +DEFINE_PER_CPU (local_t, softirqs_on_events); +DEFINE_PER_CPU (local_t, softirqs_off_events); +DEFINE_PER_CPU (local_t, redundant_softirqs_on); +DEFINE_PER_CPU (local_t, redundant_softirqs_off); +DEFINE_PER_CPU (local_t, nr_unused_locks); +DEFINE_PER_CPU (local_t, nr_cyclic_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_forwards_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_backwards_checks);#endif /* kernel/lockdep_internals.h +50
-21@@ -110,29 +110,58 @@ lockdep_count_backward_deps(struct lock_class *class)#endif #ifdef CONFIG_DEBUG_LOCKDEP + +#include <asm/local.h> +/* +* Various lockdep statistics. +* We want them per cpu as they are often accessed in fast path +* and we want to avoid too much cache bouncing. +* We can't use irqsafe per cpu counters as those are not NMI safe, as +* opposite to local_t. +*/ +DECLARE_PER_CPU (local_t, chain_lookup_hits); +DECLARE_PER_CPU (local_t, chain_lookup_misses); +DECLARE_PER_CPU (local_t, hardirqs_on_events); +DECLARE_PER_CPU (local_t, hardirqs_off_events); +DECLARE_PER_CPU (local_t, redundant_hardirqs_on); +DECLARE_PER_CPU (local_t, redundant_hardirqs_off); +DECLARE_PER_CPU (local_t, softirqs_on_events); +DECLARE_PER_CPU (local_t, softirqs_off_events); +DECLARE_PER_CPU (local_t, redundant_softirqs_on); +DECLARE_PER_CPU (local_t, redundant_softirqs_off); +DECLARE_PER_CPU (local_t, nr_unused_locks); +DECLARE_PER_CPU (local_t, nr_cyclic_checks); +DECLARE_PER_CPU (local_t, nr_cyclic_check_recursions); +DECLARE_PER_CPU (local_t, nr_find_usage_forwards_checks); +DECLARE_PER_CPU (local_t, nr_find_usage_forwards_recursions); +DECLARE_PER_CPU (local_t, nr_find_usage_backwards_checks); +DECLARE_PER_CPU (local_t, nr_find_usage_backwards_recursions); + +# define debug_atomic_inc (ptr) { \ + WARN_ON_ONCE(!irq_disabled()); \ + local_t *__ptr = &__get_cpu_var(ptr); \ + local_inc(__ptr); \ +} + +# define debug_atomic_dec (ptr) { \ + WARN_ON_ONCE(!irq_disabled()); \ + local_t *__ptr = &__get_cpu_var(ptr); \ + local_dec(__ptr); \ +} +/* -* Various lockdep statistics: +* It's fine to use local_read from other cpus. Read is racy anyway, +* but it's guaranteed high and low parts are read atomically. */ -extern atomic_t chain_lookup_hits; -extern atomic_t chain_lookup_misses; -extern atomic_t hardirqs_on_events; -extern atomic_t hardirqs_off_events; -extern atomic_t redundant_hardirqs_on; -extern atomic_t redundant_hardirqs_off; -extern atomic_t softirqs_on_events; -extern atomic_t softirqs_off_events; -extern atomic_t redundant_softirqs_on; -extern atomic_t redundant_softirqs_off; -extern atomic_t nr_unused_locks; -extern atomic_t nr_cyclic_checks; -extern atomic_t nr_cyclic_check_recursions; -extern atomic_t nr_find_usage_forwards_checks; -extern atomic_t nr_find_usage_forwards_recursions; -extern atomic_t nr_find_usage_backwards_checks; -extern atomic_t nr_find_usage_backwards_recursions; -# define debug_atomic_inc (ptr) atomic_inc(ptr) -# define debug_atomic_dec (ptr) atomic_dec(ptr) -# define debug_atomic_read (ptr) atomic_read(ptr) +# define debug_atomic_read (ptr) ({ \ + unsigned long long __total = 0; \ + int __cpu; \ + for_each_possible_cpu(__cpu) { \ + local_t *__ptr = &per_cpu(ptr, cpu); \ + __total += local_read(__ptr); \ + } \ + __total; \ +})#else # define debug_atomic_inc (ptr) do { } while (0)# define debug_atomic_dec (ptr) do { } while (0)Signature · 1 line
1.6.2.3
Quoted 4 lines · Frederic Weisbecker
kernel/lockdep.c +4
-0+DEFINE_PER_CPU (local_t, chain_lookup_hits); +DEFINE_PER_CPU (local_t, chain_lookup_misses); +DEFINE_PER_CPU (local_t, hardirqs_on_events); +DEFINE_PER_CPU (local_t, hardirqs_off_events);
kernel/lockdep.c static struct stack_trace lockdep_init_trace = { · line 446
Quoted 10 lines
kernel/lockdep.c +10
-0+DEFINE_PER_CPU (local_t, redundant_hardirqs_on); +DEFINE_PER_CPU (local_t, redundant_hardirqs_off); +DEFINE_PER_CPU (local_t, softirqs_on_events); +DEFINE_PER_CPU (local_t, softirqs_off_events); +DEFINE_PER_CPU (local_t, redundant_softirqs_on); +DEFINE_PER_CPU (local_t, redundant_softirqs_off); +DEFINE_PER_CPU (local_t, nr_unused_locks); +DEFINE_PER_CPU (local_t, nr_cyclic_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_forwards_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_backwards_checks);
btw., i think this should really be cleaned up and put into a helper struct.
'struct lockdep_stats' or so?
Ingo
Quoted 20 lines · Ingo Molnar
Quoted 4 lines · Frederic Weisbecker
kernel/lockdep.c +4
-0+DEFINE_PER_CPU (local_t, chain_lookup_hits); +DEFINE_PER_CPU (local_t, chain_lookup_misses); +DEFINE_PER_CPU (local_t, hardirqs_on_events); +DEFINE_PER_CPU (local_t, hardirqs_off_events);kernel/lockdep.c static struct stack_trace lockdep_init_trace = { · line 446
Quoted 10 lines
kernel/lockdep.c +10
-0+DEFINE_PER_CPU (local_t, redundant_hardirqs_on); +DEFINE_PER_CPU (local_t, redundant_hardirqs_off); +DEFINE_PER_CPU (local_t, softirqs_on_events); +DEFINE_PER_CPU (local_t, softirqs_off_events); +DEFINE_PER_CPU (local_t, redundant_softirqs_on); +DEFINE_PER_CPU (local_t, redundant_softirqs_off); +DEFINE_PER_CPU (local_t, nr_unused_locks); +DEFINE_PER_CPU (local_t, nr_cyclic_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_forwards_checks); +DEFINE_PER_CPU (local_t, nr_find_usage_backwards_checks);btw., i think this should really be cleaned up and put into a helper struct.
Quoted 1 line
'struct
lockdep_stats' or so?
Yep, looks like a good idea.
Quoted 4 lines · Frederic Weisbecker
We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but
local_toperations are.
lockdep shouldn't do NMI, nmi_enter() has an explicit lockdep_off().
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but
local_toperations are.lockdep shouldn't do NMI,
nmi_enter()has an explicitlockdep_off().
Also, more importantly, you simply should never use locks in NMI context to begin with ;-)
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but
local_toperations are.lockdep shouldn't do NMI,
nmi_enter()has an explicitlockdep_off().Also, more importantly, you simply should never use locks in NMI context to begin with ;-)
Right. I just thought ther could be some path that use rcu_read_lock things there or whatever check. I mean, NMI don't need rcu_read_lock but it could call helpers that use it.
Whatever, lockdep is off there. Looks like I can safely use per cpu counters.
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but
local_toperations are.lockdep shouldn't do NMI,
nmi_enter()has an explicitlockdep_off().
Ah ok, thanks.