Skip to contentSkip to conversation

lockdep: Make lockstats counting per cpu

PATCH In review Frederic Weisbecker 3 participants 9 messages Mar 26, 2010

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 lockdep: Make lockstats counting per cpu

Locking statistics are implemented using global atomic variables. This is usually fine unless some path write them very often.

This is the case for the function and function graph tracers that disable irqs for each entry saved (except if the function tracer is in preempt disabled only mode).
And calls to local_irq_save/restore() increment hardirqs_on_events and hardirqs_off_events stats (or similar stats for redundant versions).

Incrementing these global vars for each function ends up in too much cache bouncing if lockstats are enabled.

To solve this, implement the debug_atomic_*() operations using per cpu vars. We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but local_t operations are.

This version then uses local_t based per cpu counters.

---
 kernel/lockdep.c           |   28 ++++++++--------
 kernel/lockdep_internals.h |   71 +++++++++++++++++++++++++++++++-------------
 2 files changed, 64 insertions(+), 35 deletions(-)
kernel/lockdep.c +14 -14
index 65b5f5b..55e60a0 100644
@@ -430,20 +430,20 @@ static struct stack_trace lockdep_init_trace = {
 /*
  * Various lockdep statistics:
  */
-atomic_t chain_lookup_hits;
-atomic_t chain_lookup_misses;
-atomic_t hardirqs_on_events;
-atomic_t hardirqs_off_events;
-atomic_t redundant_hardirqs_on;
-atomic_t redundant_hardirqs_off;
-atomic_t softirqs_on_events;
-atomic_t softirqs_off_events;
-atomic_t redundant_softirqs_on;
-atomic_t redundant_softirqs_off;
-atomic_t nr_unused_locks;
-atomic_t nr_cyclic_checks;
-atomic_t nr_find_usage_forwards_checks;
-atomic_t nr_find_usage_backwards_checks;
+DEFINE_PER_CPU(local_t, chain_lookup_hits);
+DEFINE_PER_CPU(local_t, chain_lookup_misses);
+DEFINE_PER_CPU(local_t, hardirqs_on_events);
+DEFINE_PER_CPU(local_t, hardirqs_off_events);
+DEFINE_PER_CPU(local_t, redundant_hardirqs_on);
+DEFINE_PER_CPU(local_t, redundant_hardirqs_off);
+DEFINE_PER_CPU(local_t, softirqs_on_events);
+DEFINE_PER_CPU(local_t, softirqs_off_events);
+DEFINE_PER_CPU(local_t, redundant_softirqs_on);
+DEFINE_PER_CPU(local_t, redundant_softirqs_off);
+DEFINE_PER_CPU(local_t, nr_unused_locks);
+DEFINE_PER_CPU(local_t, nr_cyclic_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_forwards_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_backwards_checks);
 #endif
 
 /*
kernel/lockdep_internals.h +50 -21

79 lines not on this page · raw message

Signature · 1 line
1.6.2.3
#1 lore raw

PATCH 1/1 lockdep: Make lockstats counting per cpu

Locking statistics are implemented using global atomic variables. This is usually fine unless some path write them very often.

This is the case for the function and function graph tracers that disable irqs for each entry saved (except if the function tracer is in preempt disabled only mode).
And calls to local_irq_save/restore() increment hardirqs_on_events and hardirqs_off_events stats (or similar stats for redundant versions).

Incrementing these global vars for each function ends up in too much cache bouncing if lockstats are enabled.

To solve this, implement the debug_atomic_*() operations using per cpu vars. We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but local_t oprations are.

This version then uses local_t based per cpu counters.

v2: Use per_cpu() instead of get_cpu_var() to fetch the desired cpu vars on debug_atomic_read()

---
 kernel/lockdep.c           |   28 ++++++++--------
 kernel/lockdep_internals.h |   71 +++++++++++++++++++++++++++++++-------------
 2 files changed, 64 insertions(+), 35 deletions(-)
kernel/lockdep.c +14 -14
index 65b5f5b..55e60a0 100644
@@ -430,20 +430,20 @@ static struct stack_trace lockdep_init_trace = {
 /*
  * Various lockdep statistics:
  */
-atomic_t chain_lookup_hits;
-atomic_t chain_lookup_misses;
-atomic_t hardirqs_on_events;
-atomic_t hardirqs_off_events;
-atomic_t redundant_hardirqs_on;
-atomic_t redundant_hardirqs_off;
-atomic_t softirqs_on_events;
-atomic_t softirqs_off_events;
-atomic_t redundant_softirqs_on;
-atomic_t redundant_softirqs_off;
-atomic_t nr_unused_locks;
-atomic_t nr_cyclic_checks;
-atomic_t nr_find_usage_forwards_checks;
-atomic_t nr_find_usage_backwards_checks;
+DEFINE_PER_CPU(local_t, chain_lookup_hits);
+DEFINE_PER_CPU(local_t, chain_lookup_misses);
+DEFINE_PER_CPU(local_t, hardirqs_on_events);
+DEFINE_PER_CPU(local_t, hardirqs_off_events);
+DEFINE_PER_CPU(local_t, redundant_hardirqs_on);
+DEFINE_PER_CPU(local_t, redundant_hardirqs_off);
+DEFINE_PER_CPU(local_t, softirqs_on_events);
+DEFINE_PER_CPU(local_t, softirqs_off_events);
+DEFINE_PER_CPU(local_t, redundant_softirqs_on);
+DEFINE_PER_CPU(local_t, redundant_softirqs_off);
+DEFINE_PER_CPU(local_t, nr_unused_locks);
+DEFINE_PER_CPU(local_t, nr_cyclic_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_forwards_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_backwards_checks);
 #endif
 
 /*
kernel/lockdep_internals.h +50 -21

79 lines not on this page · raw message

Signature · 1 line
1.6.2.3
#2 lore raw

I forgot to mention in the title of this patch, this is the "v2".

Quoted 158 lines · Frederic Weisbecker

Locking statistics are implemented using global atomic variables. This is usually fine unless some path write them very often.

This is the case for the function and function graph tracers that disable irqs for each entry saved (except if the function tracer is in preempt disabled only mode).
And calls to local_irq_save/restore() increment hardirqs_on_events and hardirqs_off_events stats (or similar stats for redundant versions).

Incrementing these global vars for each function ends up in too much cache bouncing if lockstats are enabled.

To solve this, implement the debug_atomic_*() operations using per cpu vars. We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but local_t oprations are.

This version then uses local_t based per cpu counters.

v2: Use per_cpu() instead of get_cpu_var() to fetch the desired cpu vars on debug_atomic_read()

---
 kernel/lockdep.c           |   28 ++++++++--------
 kernel/lockdep_internals.h |   71 +++++++++++++++++++++++++++++++-------------
 2 files changed, 64 insertions(+), 35 deletions(-)
kernel/lockdep.c +14 -14
index 65b5f5b..55e60a0 100644
@@ -430,20 +430,20 @@ static struct stack_trace lockdep_init_trace = {
 /*
  * Various lockdep statistics:
  */
-atomic_t chain_lookup_hits;
-atomic_t chain_lookup_misses;
-atomic_t hardirqs_on_events;
-atomic_t hardirqs_off_events;
-atomic_t redundant_hardirqs_on;
-atomic_t redundant_hardirqs_off;
-atomic_t softirqs_on_events;
-atomic_t softirqs_off_events;
-atomic_t redundant_softirqs_on;
-atomic_t redundant_softirqs_off;
-atomic_t nr_unused_locks;
-atomic_t nr_cyclic_checks;
-atomic_t nr_find_usage_forwards_checks;
-atomic_t nr_find_usage_backwards_checks;
+DEFINE_PER_CPU(local_t, chain_lookup_hits);
+DEFINE_PER_CPU(local_t, chain_lookup_misses);
+DEFINE_PER_CPU(local_t, hardirqs_on_events);
+DEFINE_PER_CPU(local_t, hardirqs_off_events);
+DEFINE_PER_CPU(local_t, redundant_hardirqs_on);
+DEFINE_PER_CPU(local_t, redundant_hardirqs_off);
+DEFINE_PER_CPU(local_t, softirqs_on_events);
+DEFINE_PER_CPU(local_t, softirqs_off_events);
+DEFINE_PER_CPU(local_t, redundant_softirqs_on);
+DEFINE_PER_CPU(local_t, redundant_softirqs_off);
+DEFINE_PER_CPU(local_t, nr_unused_locks);
+DEFINE_PER_CPU(local_t, nr_cyclic_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_forwards_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_backwards_checks);
 #endif
 
 /*
kernel/lockdep_internals.h +50 -21
index a2ee95a..38c8ac7 100644
@@ -110,29 +110,58 @@ lockdep_count_backward_deps(struct lock_class *class)
 #endif
 
 #ifdef CONFIG_DEBUG_LOCKDEP
+
+#include <asm/local.h>
+/*
+ * Various lockdep statistics.
+ * We want them per cpu as they are often accessed in fast path
+ * and we want to avoid too much cache bouncing.
+ * We can't use irqsafe per cpu counters as those are not NMI safe, as
+ * opposite to local_t.
+ */
+DECLARE_PER_CPU(local_t, chain_lookup_hits);
+DECLARE_PER_CPU(local_t, chain_lookup_misses);
+DECLARE_PER_CPU(local_t, hardirqs_on_events);
+DECLARE_PER_CPU(local_t, hardirqs_off_events);
+DECLARE_PER_CPU(local_t, redundant_hardirqs_on);
+DECLARE_PER_CPU(local_t, redundant_hardirqs_off);
+DECLARE_PER_CPU(local_t, softirqs_on_events);
+DECLARE_PER_CPU(local_t, softirqs_off_events);
+DECLARE_PER_CPU(local_t, redundant_softirqs_on);
+DECLARE_PER_CPU(local_t, redundant_softirqs_off);
+DECLARE_PER_CPU(local_t, nr_unused_locks);
+DECLARE_PER_CPU(local_t, nr_cyclic_checks);
+DECLARE_PER_CPU(local_t, nr_cyclic_check_recursions);
+DECLARE_PER_CPU(local_t, nr_find_usage_forwards_checks);
+DECLARE_PER_CPU(local_t, nr_find_usage_forwards_recursions);
+DECLARE_PER_CPU(local_t, nr_find_usage_backwards_checks);
+DECLARE_PER_CPU(local_t, nr_find_usage_backwards_recursions);
+
+# define debug_atomic_inc(ptr)			{	\
+	WARN_ON_ONCE(!irq_disabled());			\
+	local_t *__ptr = &__get_cpu_var(ptr);		\
+	local_inc(__ptr);				\
+}
+
+# define debug_atomic_dec(ptr)			{	\
+	WARN_ON_ONCE(!irq_disabled());			\
+	local_t *__ptr = &__get_cpu_var(ptr);		\
+	local_dec(__ptr);				\
+}
+
 /*
- * Various lockdep statistics:
+ * It's fine to use local_read from other cpus. Read is racy anyway,
+ * but it's guaranteed high and low parts are read atomically.
  */
-extern atomic_t chain_lookup_hits;
-extern atomic_t chain_lookup_misses;
-extern atomic_t hardirqs_on_events;
-extern atomic_t hardirqs_off_events;
-extern atomic_t redundant_hardirqs_on;
-extern atomic_t redundant_hardirqs_off;
-extern atomic_t softirqs_on_events;
-extern atomic_t softirqs_off_events;
-extern atomic_t redundant_softirqs_on;
-extern atomic_t redundant_softirqs_off;
-extern atomic_t nr_unused_locks;
-extern atomic_t nr_cyclic_checks;
-extern atomic_t nr_cyclic_check_recursions;
-extern atomic_t nr_find_usage_forwards_checks;
-extern atomic_t nr_find_usage_forwards_recursions;
-extern atomic_t nr_find_usage_backwards_checks;
-extern atomic_t nr_find_usage_backwards_recursions;
-# define debug_atomic_inc(ptr)		atomic_inc(ptr)
-# define debug_atomic_dec(ptr)		atomic_dec(ptr)
-# define debug_atomic_read(ptr)		atomic_read(ptr)
+# define debug_atomic_read(ptr)		({		\
+	unsigned long long __total = 0;			\
+	int __cpu;					\
+	for_each_possible_cpu(__cpu) {			\
+		local_t *__ptr = &per_cpu(ptr, cpu);	\
+		__total += local_read(__ptr);		\
+	}						\
+	__total;					\
+})
 #else
 # define debug_atomic_inc(ptr)		do { } while (0)
 # define debug_atomic_dec(ptr)		do { } while (0)
Signature · 1 line
1.6.2.3
#3 lore raw
In reply to Frederic Weisbecker
Quoted 4 lines · Frederic Weisbecker
kernel/lockdep.c +4 -0
+DEFINE_PER_CPU(local_t, chain_lookup_hits);
+DEFINE_PER_CPU(local_t, chain_lookup_misses);
+DEFINE_PER_CPU(local_t, hardirqs_on_events);
+DEFINE_PER_CPU(local_t, hardirqs_off_events);

kernel/lockdep.c static struct stack_trace lockdep_init_trace = { · line 446

Quoted 10 lines
kernel/lockdep.c +10 -0
+DEFINE_PER_CPU(local_t, redundant_hardirqs_on);
+DEFINE_PER_CPU(local_t, redundant_hardirqs_off);
+DEFINE_PER_CPU(local_t, softirqs_on_events);
+DEFINE_PER_CPU(local_t, softirqs_off_events);
+DEFINE_PER_CPU(local_t, redundant_softirqs_on);
+DEFINE_PER_CPU(local_t, redundant_softirqs_off);
+DEFINE_PER_CPU(local_t, nr_unused_locks);
+DEFINE_PER_CPU(local_t, nr_cyclic_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_forwards_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_backwards_checks);

btw., i think this should really be cleaned up and put into a helper struct.

'struct lockdep_stats' or so?

Ingo

#4 lore raw
Quoted 20 lines · Ingo Molnar
Quoted 4 lines · Frederic Weisbecker
kernel/lockdep.c +4 -0
+DEFINE_PER_CPU(local_t, chain_lookup_hits);
+DEFINE_PER_CPU(local_t, chain_lookup_misses);
+DEFINE_PER_CPU(local_t, hardirqs_on_events);
+DEFINE_PER_CPU(local_t, hardirqs_off_events);

kernel/lockdep.c static struct stack_trace lockdep_init_trace = { · line 446

Quoted 10 lines
kernel/lockdep.c +10 -0
+DEFINE_PER_CPU(local_t, redundant_hardirqs_on);
+DEFINE_PER_CPU(local_t, redundant_hardirqs_off);
+DEFINE_PER_CPU(local_t, softirqs_on_events);
+DEFINE_PER_CPU(local_t, softirqs_off_events);
+DEFINE_PER_CPU(local_t, redundant_softirqs_on);
+DEFINE_PER_CPU(local_t, redundant_softirqs_off);
+DEFINE_PER_CPU(local_t, nr_unused_locks);
+DEFINE_PER_CPU(local_t, nr_cyclic_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_forwards_checks);
+DEFINE_PER_CPU(local_t, nr_find_usage_backwards_checks);

btw., i think this should really be cleaned up and put into a helper struct.

Quoted 1 line

'struct lockdep_stats' or so?

Yep, looks like a good idea.

#5 lore raw
In reply to Frederic Weisbecker
Quoted 4 lines · Frederic Weisbecker

We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but local_t operations are.

lockdep shouldn't do NMI, nmi_enter() has an explicit lockdep_off().

#6 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but local_t operations are.

lockdep shouldn't do NMI, nmi_enter() has an explicit lockdep_off().

Also, more importantly, you simply should never use locks in NMI context to begin with ;-)

#7 lore raw
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but local_t operations are.

lockdep shouldn't do NMI, nmi_enter() has an explicit lockdep_off().

Also, more importantly, you simply should never use locks in NMI context to begin with ;-)

Right. I just thought ther could be some path that use rcu_read_lock things there or whatever check. I mean, NMI don't need rcu_read_lock but it could call helpers that use it.

Whatever, lockdep is off there. Looks like I can safely use per cpu counters.

#8 lore raw
In reply to Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

We can't use irqsafe per cpu counters for that as these stats might be also written from NMI path, and irqsafe per cpu counters are not NMI safe, but local_t operations are.

lockdep shouldn't do NMI, nmi_enter() has an explicit lockdep_off().

Ah ok, thanks.