Skip to contentSkip to conversation

tracing: recursion protection fixes

PATCH In review Frederic Weisbecker 3 participants 5 messages Apr 19, 2009 – Apr 21, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover tracing: recursion protection fixes

Hi,

Here are two fixes for tracing recursion protection.
The first slightly improves debugging in case of recursion detection. The second fixes a spurious warning and tracing disabling while activating a filter.

The following changes since commit 8e668b5b3455207e4540fc7ccab9ecf70142f288:
Steven Rostedt (1):
tracing: remove format attribute of inline function

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git tracing/recursion

Frederic Weisbecker (2):
tracing/core: Add current context on tracing recursion warning tracing/ring-buffer: unlock recursion protection on discard

 kernel/trace/ring_buffer.c |   25 ++++++++++++++++++++-----
 1 files changed, 20 insertions(+), 5 deletions(-)
#1 lore raw

PATCH v3 1/2 tracing/core: Add current context on tracing recursion warning

In case of tracing recursion detection, we only get the stacktrace. But the current context may be very useful to debug the issue.

This patch adds the softirq/hardirq/nmi context with the warning using lockdep context display to have a familiar output.

v2: Use printk_once()
v3: drop {hardirq,softirq}_context which depend on lockdep,
only keep what is part of current->trace_recursion, sufficient to debug the warning source.

[ Impact: print context necessary to debug recursion ]

---
 kernel/trace/ring_buffer.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/trace/ring_buffer.c +7 -0
index b421b0e..bffde63 100644
@@ -1495,6 +1495,13 @@ static int trace_recursive_lock(void)
 	if (unlikely(current->trace_recursion & (1 << level))) {
 		/* Disable all tracing before we do anything else */
 		tracing_off_permanent();
+
+		printk_once(KERN_WARNING "Tracing recursion: "
+			    "HC[%lu]:SC[%lu]:NMI[%lu]\n",
+			    hardirq_count() >> HARDIRQ_SHIFT,
+			    softirq_count() >> SOFTIRQ_SHIFT,
+			    in_nmi());
+
 		WARN_ON_ONCE(1);
 		return -1;
 	}
Signature · 1 line
1.6.2.3
#2 lore raw

PATCH 2/2 tracing/ring-buffer: unlock recursion protection on discard

In reply to Frederic Weisbecker

The pair of helpers trace_recursive_lock() trace_recursive_unlock() have been introduced recently to provide a generic tracing recursion protection.

They are used in a symetric way:

-trace_recursive_lock() on buffer reserve
-trace_recursive_unlock() on buffer commit

However sometimes, we don't commit but discard en entry to the buffer, ie: in case of filter checking.
Then we must also unlock the recursion protection on discard time, otherwise the tracing gets definetly deactivated and a warning is raised spuriously, such as:

111.119821] ------------[ cut here ]------------

[  111.119829] WARNING: at kernel/trace/ring_buffer.c:1498 ring_buffer_lock_reserve+0x1b7/0x1d0()
[  111.119835] Hardware name: AMILO Li 2727
[  111.119839] Modules linked in:
[  111.119846] Pid: 5731, comm: Xorg Tainted: G        W  2.6.30-rc1 #69
[  111.119851] Call Trace:
[  111.119863]  [<ffffffff8025ce68>] warn_slowpath+0xd8/0x130
[  111.119873]  [<ffffffff8028a30f>] ? __lock_acquire+0x19f/0x1ae0
[  111.119882]  [<ffffffff8028a30f>] ? __lock_acquire+0x19f/0x1ae0
[  111.119891]  [<ffffffff802199b0>] ? native_sched_clock+0x20/0x70
[  111.119899]  [<ffffffff80286dee>] ? put_lock_stats+0xe/0x30
[  111.119906]  [<ffffffff80286eb8>] ? lock_release_holdtime+0xa8/0x150
[  111.119913]  [<ffffffff802c8ae7>] ring_buffer_lock_reserve+0x1b7/0x1d0
[  111.119921]  [<ffffffff802cd110>] trace_buffer_lock_reserve+0x30/0x70
[  111.119930]  [<ffffffff802ce000>] trace_current_buffer_lock_reserve+0x20/0x30
[  111.119939]  [<ffffffff802474e8>] ftrace_raw_event_sched_switch+0x58/0x100
[  111.119948]  [<ffffffff808103b7>] __schedule+0x3a7/0x4cd
[  111.119957]  [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b
[  111.119964]  [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b
[  111.119971]  [<ffffffff80810c08>] schedule+0x18/0x40
[  111.119977]  [<ffffffff80810e09>] preempt_schedule+0x39/0x60
[  111.119985]  [<ffffffff80813bd3>] _read_unlock+0x53/0x60
[  111.119993]  [<ffffffff807259d2>] sock_def_readable+0x72/0x80
[  111.120002]  [<ffffffff807ad5ed>] unix_stream_sendmsg+0x24d/0x3d0
[  111.120011]  [<ffffffff807219a3>] sock_aio_write+0x143/0x160
[  111.120019]  [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b
[  111.120026]  [<ffffffff80721860>] ? sock_aio_write+0x0/0x160
[  111.120033]  [<ffffffff80721860>] ? sock_aio_write+0x0/0x160
[  111.120042]  [<ffffffff8031c283>] do_sync_readv_writev+0xf3/0x140
[  111.120049]  [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b
[  111.120057]  [<ffffffff80276ff0>] ? autoremove_wake_function+0x0/0x40
[  111.120067]  [<ffffffff8045d489>] ? cap_file_permission+0x9/0x10
[  111.120074]  [<ffffffff8045c1e6>] ? security_file_permission+0x16/0x20
[  111.120082]  [<ffffffff8031cab4>] do_readv_writev+0xd4/0x1f0
[  111.120089]  [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b
[  111.120097]  [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b
[  111.120105]  [<ffffffff8031cc18>] vfs_writev+0x48/0x70
[  111.120111]  [<ffffffff8031cd65>] sys_writev+0x55/0xc0
[  111.120119]  [<ffffffff80211e32>] system_call_fastpath+0x16/0x1b
[  111.120125] ---[ end trace 15605f4e98d5ccb5 ]---

[ Impact: fix spurious warning/tracing shutdown ]

---
 kernel/trace/ring_buffer.c |   18 +++++++++++++-----
 1 files changed, 13 insertions(+), 5 deletions(-)
kernel/trace/ring_buffer.c +13 -5
index bffde63..e145969 100644
@@ -1642,6 +1642,14 @@ int ring_buffer_unlock_commit(struct ring_buffer *buffer,
 }
 EXPORT_SYMBOL_GPL(ring_buffer_unlock_commit);
 
+static inline void rb_event_discard(struct ring_buffer_event *event)
+{
+	event->type = RINGBUF_TYPE_PADDING;
+	/* time delta must be non zero */
+	if (!event->time_delta)
+		event->time_delta = 1;
+}
+
 /**
  * ring_buffer_event_discard - discard any event in the ring buffer
  * @event: the event to discard
@@ -1656,10 +1664,8 @@ EXPORT_SYMBOL_GPL(ring_buffer_unlock_commit);
  */
 void ring_buffer_event_discard(struct ring_buffer_event *event)
 {
-	event->type = RINGBUF_TYPE_PADDING;
-	/* time delta must be non zero */
-	if (!event->time_delta)
-		event->time_delta = 1;
+	rb_event_discard(event);
+	trace_recursive_unlock();
 }
 EXPORT_SYMBOL_GPL(ring_buffer_event_discard);
 
@@ -1690,7 +1696,7 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,
 	int cpu;
 
 	/* The event is discarded regardless */
-	ring_buffer_event_discard(event);
+	rb_event_discard(event);
 
 	/*
 	 * This must only be called if the event has not been
@@ -1735,6 +1741,8 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,
 	if (rb_is_commit(cpu_buffer, event))
 		rb_set_commit_to_write(cpu_buffer);
 
+	trace_recursive_unlock();
+
 	/*
 	 * Only the last preempt count needs to restore preemption.
 	 */
Signature · 1 line
1.6.2.3
#3 lore raw
In reply to Frederic Weisbecker
Quoted 13 lines · Frederic Weisbecker

Hi,

Here are two fixes for tracing recursion protection.
The first slightly improves debugging in case of recursion detection. The second fixes a spurious warning and tracing disabling while activating a filter.

The following changes since commit 8e668b5b3455207e4540fc7ccab9ecf70142f288:
Steven Rostedt (1):
tracing: remove format attribute of inline function

are available in the git repository at:

Quoted 8 lines

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git tracing/recursion

Frederic Weisbecker (2):
tracing/core: Add current context on tracing recursion warning tracing/ring-buffer: unlock recursion protection on discard

 kernel/trace/ring_buffer.c |   25 ++++++++++++++++++++-----
 1 files changed, 20 insertions(+), 5 deletions(-)

Pulled, thanks Frederic! Steve, any objections?

Ingo

maintainer
#4 lore raw
Quoted 18 lines · Ingo Molnar
Quoted 15 lines · Frederic Weisbecker

Hi,

Here are two fixes for tracing recursion protection.
The first slightly improves debugging in case of recursion detection. The second fixes a spurious warning and tracing disabling while activating a filter.

The following changes since commit 8e668b5b3455207e4540fc7ccab9ecf70142f288:
Steven Rostedt (1):
tracing: remove format attribute of inline function

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git tracing/recursion

Quoted 8 lines
Quoted 6 lines

Frederic Weisbecker (2):
tracing/core: Add current context on tracing recursion warning tracing/ring-buffer: unlock recursion protection on discard

 kernel/trace/ring_buffer.c |   25 ++++++++++++++++++++-----
 1 files changed, 20 insertions(+), 5 deletions(-)

Pulled, thanks Frederic! Steve, any objections?

Ug, there was a whole block of email that I missed.

No objections here.

-- Steve