Skip to contentSkip to conversation

[2.6.34] tracing: urgent fixes

GIT PULL In review Steven Rostedt 4 participants 14 messages Mar 13, 2010 – Mar 17, 2010

Subsystems (3)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

Ingo,

Li Zefan and Lai Jiangshan told me they were finding various bugs with running a stress test. I asked for this test, and Li sent it to me. Running the test I was also able to trigger the same bugs that they were seeing. Unfortunately, these bugs were very hard to figure out and it took me most of the week to fix them.

These bugs would most likely not happen in normal use, since they all require multiple users writing to the control files of ftrace. This stress test does just that, it creates several processes that each write to a different part of a ftrace control system. One resizes the ring buffer, the other enables and disables various options, another changes the ftrace plugins, another enables and disables various events, another enables and disables the tracer, etc.

Normal use generally has a single user that would only do one of these actions at a time. But that is no excuse for not fixing the issues that this test uncovered, so I took this very seriously.

Even the fix "tracing: Disable buffer switching when starting or stopping trace" may look like it could happen if wakeup tracer is running, but since opening the trace file would cause the wakeup tracer to stop, it actually required another task to change the current tracer to the wakeup tracer while the trace file was being read.

In any event, I've been running Li's ftrace_stress_test for a few hours now, and it seems pretty stable with these patches. I'll continue to run it to see if it finds anything else.

All of these patches I consider urgent and have also Cc'd stable on them.

Please pull the latest tip/tracing/urgent tree, which can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/urgent

Lai Jiangshan (1):
ring-buffer: Move disabled check into preempt disable section

Steven Rostedt (4):
function-graph: Init curr_ret_stack with ret_stack
tracing: Use same local variable when resetting the ring buffer
tracing: Disable buffer switching when starting or stopping trace
tracing: Do not record user stack trace from NMI context

----

 kernel/trace/ftrace.c      |    2 +-
 kernel/trace/ring_buffer.c |   12 ++++++------
 kernel/trace/trace.c       |   24 ++++++++++++++++++++----
 3 files changed, 27 insertions(+), 11 deletions(-)
author maintainer
#1 lore raw

PATCH 1/5 ring-buffer: Move disabled check into preempt disable section

From: Lai Jiangshan <[email protected]>

The ring buffer resizing and resetting relies on a schedule RCU action. The buffers are disabled, a synchronize_sched() is called and then the resize or reset takes place.

But this only works if the disabling of the buffers are within the preempt disabled section, otherwise a window exists that the buffers can be written to while a reset or resize takes place.

LKML-Reference: <[email protected]>

---
 kernel/trace/ring_buffer.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/trace/ring_buffer.c +6 -6
index 8c1b2d2..54191d6 100644
@@ -2232,12 +2232,12 @@ ring_buffer_lock_reserve(struct ring_buffer *buffer, unsigned long length)
 	if (ring_buffer_flags != RB_BUFFERS_ON)
 		return NULL;
 
-	if (atomic_read(&buffer->record_disabled))
-		return NULL;
-
 	/* If we are tracing schedule, we don't want to recurse */
 	resched = ftrace_preempt_disable();
 
+	if (atomic_read(&buffer->record_disabled))
+		goto out_nocheck;
+
 	if (trace_recursive_lock())
 		goto out_nocheck;
 
@@ -2469,11 +2469,11 @@ int ring_buffer_write(struct ring_buffer *buffer,
 	if (ring_buffer_flags != RB_BUFFERS_ON)
 		return -EBUSY;
 
-	if (atomic_read(&buffer->record_disabled))
-		return -EBUSY;
-
 	resched = ftrace_preempt_disable();
 
+	if (atomic_read(&buffer->record_disabled))
+		goto out;
+
 	cpu = raw_smp_processor_id();
 
 	if (!cpumask_test_cpu(cpu, buffer->cpumask))
Signature · 1 line
1.7.0
author maintainer
#2 lore raw

PATCH 2/5 function-graph: Init curr_ret_stack with ret_stack

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

If the graph tracer is active, and a task is forked but the allocating of the processes graph stack fails, it can cause crash later on.

This is due to the temporary stack being NULL, but the curr_ret_stack variable is copied from the parent. If it is not -1, then in ftrace_graph_probe_sched_switch() the following:

	for (index = next->curr_ret_stack; index >= 0; index--)
		next->ret_stack[index].calltime += timestamp;

Will cause a kernel OOPS.

Found with Li Zefan's ftrace_stress_test.

---
 kernel/trace/ftrace.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/trace/ftrace.c +1 -1
index d4d1238..bb53edb 100644
@@ -3349,6 +3349,7 @@ void ftrace_graph_init_task(struct task_struct *t)
 {
 	/* Make sure we do not use the parent ret_stack */
 	t->ret_stack = NULL;
+	t->curr_ret_stack = -1;
 
 	if (ftrace_graph_active) {
 		struct ftrace_ret_stack *ret_stack;
@@ -3358,7 +3359,6 @@ void ftrace_graph_init_task(struct task_struct *t)
 				GFP_KERNEL);
 		if (!ret_stack)
 			return;
-		t->curr_ret_stack = -1;
 		atomic_set(&t->tracing_graph_pause, 0);
 		atomic_set(&t->trace_overrun, 0);
 		t->ftrace_timestamp = 0;
Signature · 1 line
1.7.0
#3 lore raw

Re: [PATCH 2/5] function-graph: Init curr_ret_stack with ret_stack

Quoted 12 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

If the graph tracer is active, and a task is forked but the allocating of the processes graph stack fails, it can cause crash later on.

This is due to the temporary stack being NULL, but the curr_ret_stack variable is copied from the parent. If it is not -1, then in ftrace_graph_probe_sched_switch() the following:

	for (index = next->curr_ret_stack; index >= 0; index--)
		next->ret_stack[index].calltime += timestamp;
Quoted 6 lines

Will cause a kernel OOPS.

Found with Li Zefan's ftrace_stress_test.

Ah, well spotted!

Quoted 28 lines
---
 kernel/trace/ftrace.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/trace/ftrace.c +1 -1
index d4d1238..bb53edb 100644
@@ -3349,6 +3349,7 @@ void ftrace_graph_init_task(struct task_struct *t)
 {
 	/* Make sure we do not use the parent ret_stack */
 	t->ret_stack = NULL;
+	t->curr_ret_stack = -1;
 
 	if (ftrace_graph_active) {
 		struct ftrace_ret_stack *ret_stack;
@@ -3358,7 +3359,6 @@ void ftrace_graph_init_task(struct task_struct *t)
 				GFP_KERNEL);
 		if (!ret_stack)
 			return;
-		t->curr_ret_stack = -1;
 		atomic_set(&t->tracing_graph_pause, 0);
 		atomic_set(&t->trace_overrun, 0);
 		t->ftrace_timestamp = 0;
Signature · 1 line
1.7.0
author maintainer
#4 lore raw

PATCH 3/5 tracing: Use same local variable when resetting the ring buffer

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

In the ftrace code that resets the ring buffer it references the buffer with a local variable, but then uses the tr->buffer as the parameter to reset. If the wakeup tracer is running, which can switch the tr->buffer with the max saved buffer, this can break the requirement of disabling the buffer before the reset.

   buffer = tr->buffer;
   ring_buffer_record_disable(buffer);
   synchronize_sched();
   __tracing_reset(tr->buffer, cpu);

If the tr->buffer is swapped, then the reset is not happening to the buffer that was disabled. This will cause the ring buffer to fail.

Found with Li Zefan's ftrace_stress_test.

---
 kernel/trace/trace.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/trace/trace.c +4 -4
index 6af8d7b..60de37b 100644
@@ -840,10 +840,10 @@ out:
 	mutex_unlock(&trace_types_lock);
 }
 
-static void __tracing_reset(struct trace_array *tr, int cpu)
+static void __tracing_reset(struct ring_buffer *buffer, int cpu)
 {
 	ftrace_disable_cpu();
-	ring_buffer_reset_cpu(tr->buffer, cpu);
+	ring_buffer_reset_cpu(buffer, cpu);
 	ftrace_enable_cpu();
 }
 
@@ -855,7 +855,7 @@ void tracing_reset(struct trace_array *tr, int cpu)
 
 	/* Make sure all commits have finished */
 	synchronize_sched();
-	__tracing_reset(tr, cpu);
+	__tracing_reset(buffer, cpu);
 
 	ring_buffer_record_enable(buffer);
 }
@@ -873,7 +873,7 @@ void tracing_reset_online_cpus(struct trace_array *tr)
 	tr->time_start = ftrace_now(tr->cpu);
 
 	for_each_online_cpu(cpu)
-		__tracing_reset(tr, cpu);
+		__tracing_reset(buffer, cpu);
 
 	ring_buffer_record_enable(buffer);
 }
Signature · 1 line
1.7.0
author maintainer
#5 lore raw

PATCH 4/5 tracing: Disable buffer switching when starting or stopping trace

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

When the trace iterator is read, tracing_start() and tracing_stop() is called to stop tracing while the iterator is processing the trace output.

These functions disable both the standard buffer and the max latency buffer. But if the wakeup tracer is running, it can switch these buffers between the two disables:

  buffer = global_trace.buffer;
  if (buffer)
      ring_buffer_record_disable(buffer);
      <<<--------- swap happens here
  buffer = max_tr.buffer;
  if (buffer)
      ring_buffer_record_disable(buffer);

What happens is that we disabled the same buffer twice. On tracing_start() we can enable the same buffer twice. All ring_buffer_record_disable() must be matched with a ring_buffer_record_enable() or the buffer can be disable permanently, or enable prematurely, and cause a bug where a reset happens while a trace is commiting.

This patch protects these two by taking the ftrace_max_lock to prevent a switch from occurring.

Found with Li Zefan's ftrace_stress_test.

---
 kernel/trace/trace.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
kernel/trace/trace.c +9 -0
index 60de37b..484337d 100644
@@ -950,6 +950,8 @@ void tracing_start(void)
 		goto out;
 	}
 
+	/* Prevent the buffers from switching */
+	arch_spin_lock(&ftrace_max_lock);
 
 	buffer = global_trace.buffer;
 	if (buffer)
@@ -959,6 +961,8 @@ void tracing_start(void)
 	if (buffer)
 		ring_buffer_record_enable(buffer);
 
+	arch_spin_unlock(&ftrace_max_lock);
+
 	ftrace_start();
  out:
 	spin_unlock_irqrestore(&tracing_start_lock, flags);
@@ -980,6 +984,9 @@ void tracing_stop(void)
 	if (trace_stop_count++)
 		goto out;
 
+	/* Prevent the buffers from switching */
+	arch_spin_lock(&ftrace_max_lock);
+
 	buffer = global_trace.buffer;
 	if (buffer)
 		ring_buffer_record_disable(buffer);
@@ -988,6 +995,8 @@ void tracing_stop(void)
 	if (buffer)
 		ring_buffer_record_disable(buffer);
 
+	arch_spin_unlock(&ftrace_max_lock);
+
  out:
 	spin_unlock_irqrestore(&tracing_start_lock, flags);
 }
Signature · 1 line
1.7.0
author maintainer
#6 lore raw

PATCH 5/5 tracing: Do not record user stack trace from NMI context

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

A bug was found with Li Zefan's ftrace_stress_test that caused applications to segfault during the test.

Placing a tracing_off() in the segfault code, and examining several traces, I found that the following was always the case. The lock tracer was enabled (lockdep being required) and userstack was enabled. Testing this out, I just enabled the two, but that was not good enough. I needed to run something else that could trigger it. Running a load like hackbench did not work, but executing a new program would. The following would trigger the segfault within seconds:

  # echo 1 > /debug/tracing/options/userstacktrace
  # echo 1 > /debug/tracing/events/lock/enable
  # while :; do ls > /dev/null ; done

Enabling the function graph tracer and looking at what was happening I finally noticed that all cashes happened just after an NMI.

 1)               |    copy_user_handle_tail() {
 1)               |      bad_area_nosemaphore() {
 1)               |        __bad_area_nosemaphore() {
 1)               |          no_context() {
 1)               |            fixup_exception() {
 1)   0.319 us    |              search_exception_tables();
 1)   0.873 us    |            }
[...]
 1)   0.314 us    |  __rcu_read_unlock();
 1)   0.325 us    |    native_apic_mem_write();
 1)   0.943 us    |  }
 1)   0.304 us    |  rcu_nmi_exit();
[...]
 1)   0.479 us    |  find_vma();
 1)               |  bad_area() {
 1)               |    __bad_area() {

After capturing several traces of failures, all of them happened after an NMI. Curious about this, I added a trace_printk() to the NMI handler to read the regs->ip to see where the NMI happened. In which I found out it was here:

ffffffff8135b660 <page_fault>:

ffffffff8135b660:       48 83 ec 78             sub    $0x78,%rsp
ffffffff8135b664:       e8 97 01 00 00          callq  ffffffff8135b800 <error_entry>

What was happening is that the NMI would happen at the place that a page fault occurred. It would call rcu_read_lock() which was traced by the lock events, and the user_stack_trace would run. This would trigger a page fault inside the NMI. I do not see where the CR2 register is saved or restored in NMI handling. This means that it would corrupt the page fault handling that the NMI interrupted.

The reason the while loop of ls helped trigger the bug, was that each execution of ls would cause lots of pages to be faulted in, and increase the chances of the race happening.

The simple solution is to not allow user stack traces in NMI context. After this patch, I ran the above "ls" test for a couple of hours without any issues. Without this patch, the bug would trigger in less than a minute.

---
 kernel/trace/trace.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/trace/trace.c +7 -0
index 484337d..e52683f 100644
@@ -1284,6 +1284,13 @@ ftrace_trace_userstack(struct ring_buffer *buffer, unsigned long flags, int pc)
 	if (!(trace_flags & TRACE_ITER_USERSTACKTRACE))
 		return;
 
+	/*
+	 * NMIs can not handle page faults, even with fix ups.
+	 * The save user stack can (and often does) fault.
+	 */
+	if (unlikely(in_nmi()))
+		return;
+
 	event = trace_buffer_lock_reserve(buffer, TRACE_USER_STACK,
 					  sizeof(*entry), flags, pc);
 	if (!event)
Signature · 1 line
1.7.0
#7 lore raw

Re: [PATCH 5/5] tracing: Do not record user stack trace from NMI context

Quoted 57 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

A bug was found with Li Zefan's ftrace_stress_test that caused applications to segfault during the test.

Placing a tracing_off() in the segfault code, and examining several traces, I found that the following was always the case. The lock tracer was enabled (lockdep being required) and userstack was enabled. Testing this out, I just enabled the two, but that was not good enough. I needed to run something else that could trigger it. Running a load like hackbench did not work, but executing a new program would. The following would trigger the segfault within seconds:

  # echo 1 > /debug/tracing/options/userstacktrace
  # echo 1 > /debug/tracing/events/lock/enable
  # while :; do ls > /dev/null ; done

Enabling the function graph tracer and looking at what was happening I finally noticed that all cashes happened just after an NMI.

 1)               |    copy_user_handle_tail() {
 1)               |      bad_area_nosemaphore() {
 1)               |        __bad_area_nosemaphore() {
 1)               |          no_context() {
 1)               |            fixup_exception() {
 1)   0.319 us    |              search_exception_tables();
 1)   0.873 us    |            }
[...]
 1)   0.314 us    |  __rcu_read_unlock();
 1)   0.325 us    |    native_apic_mem_write();
 1)   0.943 us    |  }
 1)   0.304 us    |  rcu_nmi_exit();
[...]
 1)   0.479 us    |  find_vma();
 1)               |  bad_area() {
 1)               |    __bad_area() {

After capturing several traces of failures, all of them happened after an NMI. Curious about this, I added a trace_printk() to the NMI handler to read the regs->ip to see where the NMI happened. In which I found out it was here:

ffffffff8135b660 <page_fault>:

ffffffff8135b660:       48 83 ec 78             sub    $0x78,%rsp
ffffffff8135b664:       e8 97 01 00 00          callq  ffffffff8135b800 <error_entry>

What was happening is that the NMI would happen at the place that a page fault occurred. It would call rcu_read_lock() which was traced by the lock events, and the user_stack_trace would run. This would trigger a page fault inside the NMI. I do not see where the CR2 register is saved or restored in NMI handling. This means that it would corrupt the page fault handling that the NMI interrupted.

The reason the while loop of ls helped trigger the bug, was that each execution of ls would cause lots of pages to be faulted in, and increase the chances of the race happening.

Quoted 8 lines

The simple solution is to not allow user stack traces in NMI context. After this patch, I ran the above "ls" test for a couple of hours without any issues. Without this patch, the bug would trigger in less than a minute.

Wow, that's a race :)

In perf this is dealt with a special copy_from_user_nmi() (see in arch/x86/kernel/cpu/perf_event.c)

May be save_stack_trace_user() should use that instead of a __copy_from_user_inatomic() based thing, just to cover such NMI corner race case.

Quoted 26 lines
---
 kernel/trace/trace.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/trace/trace.c +7 -0
index 484337d..e52683f 100644
@@ -1284,6 +1284,13 @@ ftrace_trace_userstack(struct ring_buffer *buffer, unsigned long flags, int pc)
 	if (!(trace_flags & TRACE_ITER_USERSTACKTRACE))
 		return;
 
+	/*
+	 * NMIs can not handle page faults, even with fix ups.
+	 * The save user stack can (and often does) fault.
+	 */
+	if (unlikely(in_nmi()))
+		return;
+
 	event = trace_buffer_lock_reserve(buffer, TRACE_USER_STACK,
 					  sizeof(*entry), flags, pc);
 	if (!event)
Signature · 1 line
1.7.0
author maintainer
#8 lore raw

Re: [PATCH 5/5] tracing: Do not record user stack trace from NMI context

Quoted 10 lines · Frederic Weisbecker

Wow, that's a race :)

In perf this is dealt with a special copy_from_user_nmi() (see in arch/x86/kernel/cpu/perf_event.c)

May be save_stack_trace_user() should use that instead of a __copy_from_user_inatomic() based thing, just to cover such NMI corner race case.

Yeah, we should move the __copy_from_user_nmi() out of the perf code and into the normal uaccess code. Then we could do as you suggest, and have the stack code do:

if (in_nmi())

	__copy_from_user_nmi();
else
	__copy_from_user_inatomic();

Or maybe it would be best to have the __copy_from_user_inatomic() handle it.

-- Steve

author maintainer
#9 lore raw

Re: [PATCH 5/5] tracing: Do not record user stack trace from NMI context

In reply to Frederic Weisbecker
Quoted 10 lines · Frederic Weisbecker

Wow, that's a race :)

In perf this is dealt with a special copy_from_user_nmi() (see in arch/x86/kernel/cpu/perf_event.c)

May be save_stack_trace_user() should use that instead of a __copy_from_user_inatomic() based thing, just to cover such NMI corner race case.

Yeah, we should move the __copy_from_user_nmi() out of the perf code and into the normal uaccess code. Then we could do as you suggest, and have the stack code do:

if (in_nmi())

	__copy_from_user_nmi();
else
	__copy_from_user_inatomic();

Or maybe it would be best to have the __copy_from_user_inatomic() handle it.

-- Steve

#10 lore raw

Re: [PATCH 5/5] tracing: Do not record user stack trace from NMI context

Quoted 17 lines · Steven Rostedt
Quoted 10 lines · Frederic Weisbecker

Wow, that's a race :)

In perf this is dealt with a special copy_from_user_nmi() (see in arch/x86/kernel/cpu/perf_event.c)

May be save_stack_trace_user() should use that instead of a __copy_from_user_inatomic() based thing, just to cover such NMI corner race case.

Yeah, we should move the __copy_from_user_nmi() out of the perf code and into the normal uaccess code. Then we could do as you suggest, and have the stack code do:

Quoted 7 lines

if (in_nmi())

	__copy_from_user_nmi();
else
	__copy_from_user_inatomic();

Or maybe it would be best to have the __copy_from_user_inatomic() handle it.

Yeah, agreed.

#11 lore raw

Re: [PATCH 5/5] tracing: Do not record user stack trace from NMI context

In reply to Steven Rostedt
Quoted 88 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

A bug was found with Li Zefan's ftrace_stress_test that caused applications to segfault during the test.

Placing a tracing_off() in the segfault code, and examining several traces, I found that the following was always the case. The lock tracer was enabled (lockdep being required) and userstack was enabled. Testing this out, I just enabled the two, but that was not good enough. I needed to run something else that could trigger it. Running a load like hackbench did not work, but executing a new program would. The following would trigger the segfault within seconds:

 # echo 1 > /debug/tracing/options/userstacktrace
 # echo 1 > /debug/tracing/events/lock/enable
 # while :; do ls > /dev/null ; done

Enabling the function graph tracer and looking at what was happening I finally noticed that all cashes happened just after an NMI.

 1)               |    copy_user_handle_tail() {
 1)               |      bad_area_nosemaphore() {
 1)               |        __bad_area_nosemaphore() {
 1)               |          no_context() {
 1)               |            fixup_exception() {
 1)   0.319 us    |              search_exception_tables();
 1)   0.873 us    |            }
[...]
 1)   0.314 us    |  __rcu_read_unlock();
 1)   0.325 us    |    native_apic_mem_write();
 1)   0.943 us    |  }
 1)   0.304 us    |  rcu_nmi_exit();

[...]
 1)   0.479 us    |  find_vma();
 1)               |  bad_area() {
 1)               |    __bad_area() {

After capturing several traces of failures, all of them happened after an NMI. Curious about this, I added a trace_printk() to the NMI handler to read the regs->ip to see where the NMI happened. In which I found out it was here:

ffffffff8135b660 <page_fault>:
ffffffff8135b660:       48 83 ec 78             sub    $0x78,%rsp
ffffffff8135b664:       e8 97 01 00 00          callq  ffffffff8135b800 <error_entry>

What was happening is that the NMI would happen at the place that a page fault occurred. It would call rcu_read_lock() which was traced by the lock events, and the user_stack_trace would run. This would trigger a page fault inside the NMI. I do not see where the CR2 register is saved or restored in NMI handling. This means that it would corrupt the page fault handling that the NMI interrupted.

The reason the while loop of ls helped trigger the bug, was that each execution of ls would cause lots of pages to be faulted in, and increase the chances of the race happening.

The simple solution is to not allow user stack traces in NMI context. After this patch, I ran the above "ls" test for a couple of hours without any issues. Without this patch, the bug would trigger in less than a minute.

---
 kernel/trace/trace.c |    7 +++++++

 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/trace/trace.c +7 -1
index 484337d..e52683f 100644
@@ -1284,6 +1284,13 @@ ftrace_trace_userstack(struct ring_buffer *buffer, unsigned long flags, int pc)
        if (!(trace_flags & TRACE_ITER_USERSTACKTRACE))
                return;
 
+       /*
+        * NMIs can not handle page faults, even with fix ups.
+        * The save user stack can (and often does) fault.
+        */
+       if (unlikely(in_nmi()))
+               return;
+
        event = trace_buffer_lock_reserve(buffer, TRACE_USER_STACK,
                                          sizeof(*entry), flags, pc);
        if (!event)
--
Quoted 2 lines

1.7.0

That is some awfully cool detective work. Just one question, if it is so easy to trigger, why did you wrap it in "unlikely"?

author maintainer
#12 lore raw

Re: [PATCH 5/5] tracing: Do not record user stack trace from NMI context

Quoted 10 lines · John Kacur
Quoted 8 lines · Steven Rostedt
kernel/trace/trace.c +0 -0
index 484337d..e52683f 100644
@@ -1284,6 +1284,13 @@ ftrace_trace_userstack(struct ring_buffer *buffer, unsigned long flags, int pc)
       if (!(trace_flags & TRACE_ITER_USERSTACKTRACE))
               return;
 
Quoted 16 lines
Quoted 11 lines
kernel/trace/trace.c +7 -1
+       /*
+        * NMIs can not handle page faults, even with fix ups.
+        * The save user stack can (and often does) fault.
+        */
+       if (unlikely(in_nmi()))
+               return;
+
       event = trace_buffer_lock_reserve(buffer, TRACE_USER_STACK,
                                         sizeof(*entry), flags, pc);
       if (!event)
--
Quoted 2 lines

1.7.0

That is some awfully cool detective work. Just one question, if it is so easy to trigger, why did you wrap it in "unlikely"?

It was not easy to trigger until I had a case to trigger it. Currently, the only time that this should trigger is for the lockdep tracing, which only happens when lockdep is enabled. Most of the time, this should never trigger.

-- Steve

#13 lore raw
In reply to Steven Rostedt
Quoted 47 lines · Steven Rostedt

Ingo,

Li Zefan and Lai Jiangshan told me they were finding various bugs with running a stress test. I asked for this test, and Li sent it to me. Running the test I was also able to trigger the same bugs that they were seeing. Unfortunately, these bugs were very hard to figure out and it took me most of the week to fix them.

These bugs would most likely not happen in normal use, since they all require multiple users writing to the control files of ftrace. This stress test does just that, it creates several processes that each write to a different part of a ftrace control system. One resizes the ring buffer, the other enables and disables various options, another changes the ftrace plugins, another enables and disables various events, another enables and disables the tracer, etc.

Normal use generally has a single user that would only do one of these actions at a time. But that is no excuse for not fixing the issues that this test uncovered, so I took this very seriously.

Even the fix "tracing: Disable buffer switching when starting or stopping trace" may look like it could happen if wakeup tracer is running, but since opening the trace file would cause the wakeup tracer to stop, it actually required another task to change the current tracer to the wakeup tracer while the trace file was being read.

In any event, I've been running Li's ftrace_stress_test for a few hours now, and it seems pretty stable with these patches. I'll continue to run it to see if it finds anything else.

All of these patches I consider urgent and have also Cc'd stable on them.

Please pull the latest tip/tracing/urgent tree, which can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/urgent

Lai Jiangshan (1):
ring-buffer: Move disabled check into preempt disable section

Steven Rostedt (4):
function-graph: Init curr_ret_stack with ret_stack
tracing: Use same local variable when resetting the ring buffer
tracing: Disable buffer switching when starting or stopping trace
tracing: Do not record user stack trace from NMI context

Quoted 5 lines

----

 kernel/trace/ftrace.c      |    2 +-
 kernel/trace/ring_buffer.c |   12 ++++++------
 kernel/trace/trace.c       |   24 ++++++++++++++++++++----
 3 files changed, 27 insertions(+), 11 deletions(-)

Pulled, thanks a lot Steve!

Ingo