Skip to contentSkip to conversation

tracing: Fix lockdep warning in global_clock()

PATCH Superseded Li Zefan 5 participants 10 messages Dec 21, 2009 – Dec 23, 2009 Tracing

lore
Share & tools
lore

Li Zefan

author
#0 lore raw

PATCH 1/1 tracing: Fix lockdep warning in global_clock()

 # echo 1 > events/enable
 # echo global > trace_clock
------------[ cut here ]------------
WARNING: at kernel/lockdep.c:3162 check_flags+0xb2/0x190()
...
---[ end trace 3f86734a89416623 ]---

possible reason: unannotated irqs-on.
...

---
 kernel/trace/trace_clock.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/trace/trace_clock.c +2 -2
index 84a3a7b..11563c9 100644
@@ -83,7 +83,7 @@ u64 notrace trace_clock_global(void)
 	int this_cpu;
 	u64 now;
 
-	raw_local_irq_save(flags);
+	local_irq_save(flags);
 
 	this_cpu = raw_smp_processor_id();
 	now = cpu_clock(this_cpu);
@@ -109,7 +109,7 @@ u64 notrace trace_clock_global(void)
 	arch_spin_unlock(&trace_clock_struct.lock);
 
  out:
-	raw_local_irq_restore(flags);
+	local_irq_restore(flags);
 
 	return now;
 }
Signature · 1 line
1.6.3
#1 lore raw
Quoted 20 lines · Li Zefan
 # echo 1 > events/enable
 # echo global > trace_clock
------------[ cut here ]------------
WARNING: at kernel/lockdep.c:3162 check_flags+0xb2/0x190()
...
---[ end trace 3f86734a89416623 ]---

possible reason: unannotated irqs-on.
...

---
 kernel/trace/trace_clock.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/trace/trace_clock.c +0 -0
index 84a3a7b..11563c9 100644

kernel/trace/trace_clock.c u64 notrace trace_clock_global(void) · line 86

Quoted 5 lines
kernel/trace/trace_clock.c +1 -1
@@ -83,7 +83,7 @@ u64 notrace trace_clock_global(void)
 	int this_cpu;
 	u64 now;
 
-	raw_local_irq_save(flags);
+	local_irq_save(flags);

Hm, wont this cause problems when we trace inside lockdep? Have you tried the lockdep events - do they still work?

Ingo

author
#2 lore raw
Quoted 25 lines · Ingo Molnar
Quoted 23 lines · Li Zefan
 # echo 1 > events/enable
 # echo global > trace_clock
------------[ cut here ]------------
WARNING: at kernel/lockdep.c:3162 check_flags+0xb2/0x190()
...
---[ end trace 3f86734a89416623 ]---

possible reason: unannotated irqs-on.
...

---
 kernel/trace/trace_clock.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/trace/trace_clock.c +0 -0
index 84a3a7b..11563c9 100644
@@ -83,7 +83,7 @@ u64 notrace trace_clock_global(void)
 	int this_cpu;
 	u64 now;
 
Quoted 6 lines

kernel/trace/trace_clock.c u64 notrace trace_clock_global(void) · line 86

Quoted 2 lines
kernel/trace/trace_clock.c +1 -1
-	raw_local_irq_save(flags);
+	local_irq_save(flags);

Hm, wont this cause problems when we trace inside lockdep? Have you tried the lockdep events - do they still work?

Yes, they still work.

trace_clock_global() calls cpu_clock() which calls local_irq_save(), which causes this lockdep warning.

And I noticed this commit:

===============================================
commit 2d452c9b10caeec455eb5e56a0ef4ed485178213

Author: Ingo Molnar <[email protected]>
Date:   Sun Jun 29 15:01:59 2008 +0200

sched: sched_clock_cpu() based cpu_clock(), lockdep fix

Vegard Nossum reported:

> WARNING: at kernel/lockdep.c:2738 check_flags+0x142/0x160()

which happens due to:

unsigned long long cpu_clock(int cpu)
{
unsigned long long clock;
unsigned long flags;

raw_local_irq_save(flags);

as lower level functions can take locks, we must not do that, use proper lockdep-annotated irq save/restore.

kernel/sched_clock.c +2 -2
index ed5a8c4..60094e2 100644
@@ -250,9 +250,9 @@ unsigned long long cpu_clock(int cpu)
        unsigned long long clock;
        unsigned long flags;
 
-       raw_local_irq_save(flags);
+       local_irq_save(flags);
        clock = sched_clock_cpu(cpu);
-       raw_local_irq_restore(flags);
+       local_irq_restore(flags);
 
        return clock;
 }

===============================================

I guess it's still true that lower level functions can take locks?

#3 lore raw
Quoted 15 lines · Li Zefan

+++ b/kernel/sched_clock.c

kernel/sched_clock.c +2 -2
@@ -250,9 +250,9 @@ unsigned long long cpu_clock(int cpu)
        unsigned long long clock;
        unsigned long flags;
 
-       raw_local_irq_save(flags);
+       local_irq_save(flags);
        clock = sched_clock_cpu(cpu);
-       raw_local_irq_restore(flags);
+       local_irq_restore(flags);
 
        return clock;
 }

===============================================

Quoted 1 line

I guess it's still true that lower level functions can take locks?

No, I removed the locks from cpu_clock a while back.

maintainer
#4 lore raw
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Li Zefan

+++ b/kernel/sched_clock.c

kernel/sched_clock.c +2 -2
@@ -250,9 +250,9 @@ unsigned long long cpu_clock(int cpu)
        unsigned long long clock;
        unsigned long flags;
 
-       raw_local_irq_save(flags);
+       local_irq_save(flags);
        clock = sched_clock_cpu(cpu);
-       raw_local_irq_restore(flags);
+       local_irq_restore(flags);
 
Quoted 8 lines
Quoted 5 lines
kernel/sched_clock.c +0 -0
        return clock;
 }

===============================================

I guess it's still true that lower level functions can take locks?

No, I removed the locks from cpu_clock a while back.

The problem isn't with locks, it's with disabling interrupts. Lockdep keeps track of every time interrupts are disabled. If something disables interrupts with raw_* but then calls something that disables interrupts the normal way, lockdep will detect that interrupts are being disabled but they already are disabled. The internal irq disabled variable in lockdep wont match reality, and this will blow up (well, disable lockdep).

So if cpu_clock disables interrupts with local_irq_* then so must trace_clock_global.

-- Steve

author
#5 lore raw
Quoted 30 lines · Steven Rostedt
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Li Zefan

+++ b/kernel/sched_clock.c

kernel/sched_clock.c +2 -2
@@ -250,9 +250,9 @@ unsigned long long cpu_clock(int cpu)
        unsigned long long clock;
        unsigned long flags;
 
-       raw_local_irq_save(flags);
+       local_irq_save(flags);
        clock = sched_clock_cpu(cpu);
-       raw_local_irq_restore(flags);
+       local_irq_restore(flags);
 
Quoted 7 lines
Quoted 5 lines
kernel/sched_clock.c +0 -0
        return clock;
 }

===============================================

I guess it's still true that lower level functions can take locks?

No, I removed the locks from cpu_clock a while back.

The problem isn't with locks, it's with disabling interrupts. Lockdep keeps track of every time interrupts are disabled. If something disables interrupts with raw_* but then calls something that disables interrupts the normal way, lockdep will detect that interrupts are being disabled but they already are disabled. The internal irq disabled variable in lockdep wont match reality, and this will blow up (well, disable lockdep).

Quoted 3 lines

So if cpu_clock disables interrupts with local_irq_* then so must trace_clock_global.

Looking into sched_clock.c, seems the only place that can trigger lockdep is the local_irq_* in cpu_clock(), though I'm not sure for architecture- specific sched_clock().

If so, I think we can use raw_local_irq_* in cpu_clock(). Actually I tried it and the warning is gone.

#6 lore raw

[snip]

Quoted 12 lines
Quoted 11 lines

The problem isn't with locks, it's with disabling interrupts. Lockdep keeps track of every time interrupts are disabled. If something disables interrupts with raw_* but then calls something that disables interrupts the normal way, lockdep will detect that interrupts are being disabled but they already are disabled. The internal irq disabled variable in lockdep wont match reality, and this will blow up (well, disable lockdep).

So if cpu_clock disables interrupts with local_irq_* then so must trace_clock_global.

Quoted 7 lines

Looking into sched_clock.c, seems the only place that can trigger lockdep is the local_irq_* in cpu_clock(), though I'm not sure for architecture- specific sched_clock().

If so, I think we can use raw_local_irq_* in cpu_clock(). Actually I tried it and the warning is gone.

I think the reason is on local_irq_restore():

#define local_irq_restore(flags)			\
	do {						\
		typecheck(unsigned long, flags);	\
		if (raw_irqs_disabled_flags(flags)) {	\
			raw_local_irq_restore(flags);	\
			trace_hardirqs_off();		\                        <======
		} else {				\
			trace_hardirqs_on();		\
			raw_local_irq_restore(flags);	\
		}					\
	} while (0)

If we have a call trace like below:

raw_local_irq_save();
            ...
local_irq_save();
            ...
local_irq_restore();
            ...

raw_locla_irq_save();

Then when we leave from local_irq_restore(), current->hardirqs_enabled will changed by trace_hardirqs_off() to 0, this will break the all consistent.

Thanks,
Yong

#7 lore raw
In reply to Li Zefan
Quoted 3 lines · Li Zefan

Looking into sched_clock.c, seems the only place that can trigger lockdep is the local_irq_* in cpu_clock(), though I'm not sure for architecture- specific sched_clock().

But why would you want to do that? that reduces the lockdep coverage for no good reason, just fix up the tracer.

author
#8 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Li Zefan

Looking into sched_clock.c, seems the only place that can trigger lockdep is the local_irq_* in cpu_clock(), though I'm not sure for architecture- specific sched_clock().

But why would you want to do that? that reduces the lockdep coverage for no good reason, just fix up the tracer.

Then the proper fix is to use local_irq_* in global_clock(), which is what this patch does.

maintainer
#9 lore raw
In reply to Ingo Molnar
Quoted 25 lines · Ingo Molnar
Quoted 23 lines · Li Zefan
 # echo 1 > events/enable
 # echo global > trace_clock
------------[ cut here ]------------
WARNING: at kernel/lockdep.c:3162 check_flags+0xb2/0x190()
...
---[ end trace 3f86734a89416623 ]---

possible reason: unannotated irqs-on.
...

---
 kernel/trace/trace_clock.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/trace/trace_clock.c +0 -0
index 84a3a7b..11563c9 100644
@@ -83,7 +83,7 @@ u64 notrace trace_clock_global(void)
 	int this_cpu;
 	u64 now;
 
Quoted 5 lines

kernel/trace/trace_clock.c u64 notrace trace_clock_global(void) · line 86

Quoted 2 lines
kernel/trace/trace_clock.c +1 -1
-	raw_local_irq_save(flags);
+	local_irq_save(flags);

Hm, wont this cause problems when we trace inside lockdep? Have you tried the lockdep events - do they still work?

Doesn't the lockdep code protect against this type of recursion.

Working with interrupts and spin locks in the latency code, I found that I had to remove all the raw_local_irq_save for the normal local_irq_save. Because if you disable interrupts with raw_* and then anything that is called must also disable with raw_*, if they don't then lockdep will blow up.

-- Steve