Skip to contentSkip to conversation

tracing: Have stack tracer force RCU to be watching

RFC PATCH In review Steven Rostedt 3 participants 9 messages Oct 20, 2015 – Oct 21, 2015 Tracing

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

RFC PATCH 1/1 tracing: Have stack tracer force RCU to be watching

Paul,

I've spent a couple of days debugging this, and finally found that my stack tracer was calling the stack trace code, which calls __module_address() which asserts the below.

Is just calling rcu_irq_enter() and rcu_irq_exit() safe to do everywhere (with interrupts always disabled)? This patch appears to fix the bug.

Peter,

I'm going to be sending a second patch that converts that from a WARN_ON() to an open coded WARN_ON_ONCE(), because WARN_ON() also calls the stack trace code which calls __module_address() and we end up with an infinite warning about it. This prevented me from seeing where the bug actually was, and crashed the box.

-- Steve

Quoted 1 line

From a2d7629048322ae62bff57f34f5f995e25ed234c Mon Sep 17 00:00:00 2001

From: "Steven Rostedt (Red Hat)" <[email protected]>
Date: Tue, 20 Oct 2015 11:38:08 -0400
Subject: [PATCH] tracing: Have stack tracer force RCU to be watching

The stack tracer was triggering the WARN_ON() in module.c:

static void module_assert_mutex_or_preempt(void)
{

 #ifdef CONFIG_LOCKDEP
	if (unlikely(!debug_locks))
		return;

	WARN_ON(!rcu_read_lock_sched_held() &&
		!lockdep_is_held(&module_mutex));

#endif
}

The reason is that the stack tracer traces all function calls, and some of those calls happen while exiting or entering user space and idle. Some of these functions are called after RCU had already stopped watching, as RCU does not watch userspace or idle CPUs.

If a max stack is hit, then the save_stack_trace() is called, which will check module addresses and call module_assert_mutex_or_preempt(), and then trigger the warning. Sad part is, the warning itself will also do a stack trace and tigger the same warning. That probably should be fixed.

The warning was added by 0be964be0d45 "module: Sanitize RCU usage and locking" but this bug has probably been around longer. But it's unlikely to cause much harm, but the new warning causes the system to lock up.

Cc Peter Zijlstra <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
---
 kernel/trace/trace_stack.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/trace/trace_stack.c +7 -0
index b746399ab59c..5f29402bff0f 100644
@@ -88,6 +88,12 @@ check_stack(unsigned long ip, unsigned long *stack)
 	local_irq_save(flags);
 	arch_spin_lock(&max_stack_lock);
 
+	/*
+	 * RCU may not be watching, make it see us.
+	 * The stack trace code uses rcu_sched.
+	 */
+	rcu_irq_enter();
+
 	/* In case another CPU set the tracer_frame on us */
 	if (unlikely(!frame_size))
 		this_size -= tracer_frame;
@@ -169,6 +175,7 @@ check_stack(unsigned long ip, unsigned long *stack)
 	}
 
  out:
+	rcu_irq_exit();
 	arch_spin_unlock(&max_stack_lock);
 	local_irq_restore(flags);
 }
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 10 lines · Steven Rostedt

Paul,

I've spent a couple of days debugging this, and finally found that my stack tracer was calling the stack trace code, which calls __module_address() which asserts the below.

Is just calling rcu_irq_enter() and rcu_irq_exit() safe to do everywhere (with interrupts always disabled)? This patch appears to fix the bug.

Yep! Just don't call it from an NMI handler. And don't call it with interrupts enabled. The patch looks to have interrupts always disabled, and the surrounding code doesn't look like NMI-safe code anyway, so should be OK.

Thanx, Paul

Quoted 80 lines

Peter,

I'm going to be sending a second patch that converts that from a WARN_ON() to an open coded WARN_ON_ONCE(), because WARN_ON() also calls the stack trace code which calls __module_address() and we end up with an infinite warning about it. This prevented me from seeing where the bug actually was, and crashed the box.

-- Steve

From a2d7629048322ae62bff57f34f5f995e25ed234c Mon Sep 17 00:00:00 2001
From: "Steven Rostedt (Red Hat)" <[email protected]>
Date: Tue, 20 Oct 2015 11:38:08 -0400
Subject: [PATCH] tracing: Have stack tracer force RCU to be watching

The stack tracer was triggering the WARN_ON() in module.c:

static void module_assert_mutex_or_preempt(void)
{

 #ifdef CONFIG_LOCKDEP
	if (unlikely(!debug_locks))
		return;

	WARN_ON(!rcu_read_lock_sched_held() &&
		!lockdep_is_held(&module_mutex));

#endif
}

The reason is that the stack tracer traces all function calls, and some of those calls happen while exiting or entering user space and idle. Some of these functions are called after RCU had already stopped watching, as RCU does not watch userspace or idle CPUs.

If a max stack is hit, then the save_stack_trace() is called, which will check module addresses and call module_assert_mutex_or_preempt(), and then trigger the warning. Sad part is, the warning itself will also do a stack trace and tigger the same warning. That probably should be fixed.

The warning was added by 0be964be0d45 "module: Sanitize RCU usage and locking" but this bug has probably been around longer. But it's unlikely to cause much harm, but the new warning causes the system to lock up.

Cc Peter Zijlstra <[email protected]>
Cc "Paul E. McKenney" <[email protected]>
---
 kernel/trace/trace_stack.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/trace/trace_stack.c +7 -0
index b746399ab59c..5f29402bff0f 100644
@@ -88,6 +88,12 @@ check_stack(unsigned long ip, unsigned long *stack)
 	local_irq_save(flags);
 	arch_spin_lock(&max_stack_lock);
 
+	/*
+	 * RCU may not be watching, make it see us.
+	 * The stack trace code uses rcu_sched.
+	 */
+	rcu_irq_enter();
+
 	/* In case another CPU set the tracer_frame on us */
 	if (unlikely(!frame_size))
 		this_size -= tracer_frame;
@@ -169,6 +175,7 @@ check_stack(unsigned long ip, unsigned long *stack)
 	}
 
  out:
+	rcu_irq_exit();
 	arch_spin_unlock(&max_stack_lock);
 	local_irq_restore(flags);
 }
Signature · 1 line
1.8.3.1
author maintainer
#2 lore raw
Quoted 12 lines · On Tue, 20 Oct 2015 13:25:28 -0700 "Paul E. McKenney"…
Quoted 10 lines · Steven Rostedt

Paul,

I've spent a couple of days debugging this, and finally found that my stack tracer was calling the stack trace code, which calls __module_address() which asserts the below.

Is just calling rcu_irq_enter() and rcu_irq_exit() safe to do everywhere (with interrupts always disabled)? This patch appears to fix the bug.

Quoted 7 lines

Yep! Just don't call it from an NMI handler. And don't call it with interrupts enabled. The patch looks to have interrupts always disabled, and the surrounding code doesn't look like NMI-safe code anyway, so should be OK.

Thanx, Paul

Hmm, good point about NMI handler. Right now I think the only thing protecting this from getting in the critical section while in NMI is the check that we are using the task struct stack. But that may not be enough in 32 bit.

I should probably add a "if (in_nmi()) return" somewhere.

-- Steve

#3 lore raw
Quoted 23 lines · Steven Rostedt
Quoted 12 lines · On Tue, 20 Oct 2015 13:25:28 -0700 "Paul E. McKenney"…
Quoted 10 lines · Steven Rostedt

Paul,

I've spent a couple of days debugging this, and finally found that my stack tracer was calling the stack trace code, which calls __module_address() which asserts the below.

Is just calling rcu_irq_enter() and rcu_irq_exit() safe to do everywhere (with interrupts always disabled)? This patch appears to fix the bug.

Quoted 7 lines

Yep! Just don't call it from an NMI handler. And don't call it with interrupts enabled. The patch looks to have interrupts always disabled, and the surrounding code doesn't look like NMI-safe code anyway, so should be OK.

Thanx, Paul

Quoted 6 lines

Hmm, good point about NMI handler. Right now I think the only thing protecting this from getting in the critical section while in NMI is the check that we are using the task struct stack. But that may not be enough in 32 bit.

I should probably add a "if (in_nmi()) return" somewhere.

Please! ;-) ;-) ;-)

Thanx, Paul

author maintainer
#4 lore raw
Quoted 24 lines · On Tue, 20 Oct 2015 16:48:30 -0700 "Paul E. McKenney"…
Quoted 23 lines · Steven Rostedt
Quoted 19 lines · On Tue, 20 Oct 2015 13:25:28 -0700 "Paul E. McKenney"…
Quoted 10 lines · Steven Rostedt

Paul,

I've spent a couple of days debugging this, and finally found that my stack tracer was calling the stack trace code, which calls __module_address() which asserts the below.

Is just calling rcu_irq_enter() and rcu_irq_exit() safe to do everywhere (with interrupts always disabled)? This patch appears to fix the bug.

Yep! Just don't call it from an NMI handler. And don't call it with interrupts enabled. The patch looks to have interrupts always disabled, and the surrounding code doesn't look like NMI-safe code anyway, so should be OK.

Thanx, Paul

Quoted 10 lines
Quoted 6 lines

Hmm, good point about NMI handler. Right now I think the only thing protecting this from getting in the critical section while in NMI is the check that we are using the task struct stack. But that may not be enough in 32 bit.

I should probably add a "if (in_nmi()) return" somewhere.

Please! ;-) ;-) ;-)

That's a separate fix, as it will break elsewhere than just this.

As for my patch, Can I have an Acked-by?

Thanks!

-- Steve

author maintainer
#5 lore raw
Quoted 34 lines · On Tue, 20 Oct 2015 19:55:29 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 31 lines · On Tue, 20 Oct 2015 16:48:30 -0700 "Paul E. McKenney"…
Quoted 29 lines · Steven Rostedt
Quoted 12 lines · On Tue, 20 Oct 2015 13:25:28 -0700 "Paul E. McKenney"…
Quoted 10 lines · Steven Rostedt

Paul,

I've spent a couple of days debugging this, and finally found that my stack tracer was calling the stack trace code, which calls __module_address() which asserts the below.

Is just calling rcu_irq_enter() and rcu_irq_exit() safe to do everywhere (with interrupts always disabled)? This patch appears to fix the bug.

Quoted 7 lines

Yep! Just don't call it from an NMI handler. And don't call it with interrupts enabled. The patch looks to have interrupts always disabled, and the surrounding code doesn't look like NMI-safe code anyway, so should be OK.

Thanx, Paul

Hmm, good point about NMI handler. Right now I think the only thing protecting this from getting in the critical section while in NMI is the check that we are using the task struct stack. But that may not be enough in 32 bit.

I should probably add a "if (in_nmi()) return" somewhere.

Quoted 7 lines
Quoted 3 lines

Please! ;-) ;-) ;-)

That's a separate fix, as it will break elsewhere than just this.

As for my patch, Can I have an Acked-by?

Ping?

-- Steve

#6 lore raw
In reply to Steven Rostedt
Quoted 23 lines · Steven Rostedt
Quoted 12 lines · On Tue, 20 Oct 2015 13:25:28 -0700 "Paul E. McKenney"…
Quoted 10 lines · Steven Rostedt

Paul,

I've spent a couple of days debugging this, and finally found that my stack tracer was calling the stack trace code, which calls __module_address() which asserts the below.

Is just calling rcu_irq_enter() and rcu_irq_exit() safe to do everywhere (with interrupts always disabled)? This patch appears to fix the bug.

Quoted 7 lines

Yep! Just don't call it from an NMI handler. And don't call it with interrupts enabled. The patch looks to have interrupts always disabled, and the surrounding code doesn't look like NMI-safe code anyway, so should be OK.

Thanx, Paul

Quoted 6 lines

Hmm, good point about NMI handler. Right now I think the only thing protecting this from getting in the critical section while in NMI is the check that we are using the task struct stack. But that may not be enough in 32 bit.

I should probably add a "if (in_nmi()) return" somewhere.

But if there's an arch that doesn't use a separate NMI stack, the NMI might cause the largest stack, which would then remain invisible from the stack-tracer.

Should we not instead fix the NMI-safety of this tracer?

author maintainer
#7 lore raw
Quoted 7 lines · On Wed, 21 Oct 2015 10:01:42 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 1 line

I should probably add a "if (in_nmi()) return" somewhere.

But if there's an arch that doesn't use a separate NMI stack, the NMI might cause the largest stack, which would then remain invisible from the stack-tracer.

Should we not instead fix the NMI-safety of this tracer?

We could, but that should be a separate project, as that would require doing everything lockless, which would require a redesign. Is that worth it?

For now, the safe thing to do is the if (in_nmi()), but certainly, if someone gets time to make it NMI safe, we can do that too.

-- Steve

#8 lore raw
Quoted 9 lines · Steven Rostedt
Quoted 6 lines · On Wed, 21 Oct 2015 10:01:42 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 1 line

I should probably add a "if (in_nmi()) return" somewhere.

But if there's an arch that doesn't use a separate NMI stack, the NMI might cause the largest stack, which would then remain invisible from the stack-tracer.

Quoted 5 lines
Quoted 1 line

Should we not instead fix the NMI-safety of this tracer?

We could, but that should be a separate project, as that would require doing everything lockless, which would require a redesign. Is that worth it?

I've no idea on either, not on how hard it would be to fix, nor on if its worth the effort.

I suppose auditing which archs do not have dedicated NMI stacks might be a good first stab at things. x86 still uses an IST for NMIs, right? Andy's been changing things a lot lately, I'm not sure I'm up to date on this.

Quoted 2 lines

For now, the safe thing to do is the if (in_nmi()), but certainly, if someone gets time to make it NMI safe, we can do that too.

Sure, fix current holes first.