tracing: recursion protection fixes
Subsystems (2)
cover tracing: recursion protection fixes
Hi,
Here are two fixes for tracing recursion protection.
The first slightly improves debugging in case of recursion detection. The second fixes a spurious warning and tracing disabling while activating a filter.
The following changes since commit 8e668b5b3455207e4540fc7ccab9ecf70142f288:
Steven Rostedt (1):
tracing: remove format attribute of inline function
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git tracing/recursion
Frederic Weisbecker (2):
tracing/core: Add current context on tracing recursion warning tracing/ring-buffer: unlock recursion protection on discard
kernel/trace/ring_buffer.c | 25 ++++++++++++++++++++----- 1 files changed, 20 insertions(+), 5 deletions(-)
PATCH v3 1/2 tracing/core: Add current context on tracing recursion warning
In case of tracing recursion detection, we only get the stacktrace. But the current context may be very useful to debug the issue.
This patch adds the softirq/hardirq/nmi context with the warning using lockdep context display to have a familiar output.
v2: Use printk_once()
v3: drop {hardirq,softirq}_context which depend on lockdep,
only keep what is part of current->trace_recursion, sufficient to debug the warning source.
[ Impact: print context necessary to debug recursion ]
--- kernel/trace/ring_buffer.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/trace/ring_buffer.c +7 -0
@@ -1495,6 +1495,13 @@ static int trace_recursive_lock(void)if (unlikely (current->trace_recursion & (1 << level))) {/* Disable all tracing before we do anything else */ tracing_off_permanent (); + +printk_once (KERN_WARNING "Tracing recursion: " +"HC[%lu]:SC[%lu]:NMI[%lu]\n" , +hardirq_count () >>HARDIRQ_SHIFT , +softirq_count () >>SOFTIRQ_SHIFT , +in_nmi ()); +WARN_ON_ONCE (1 );return -1 ; }
Signature · 1 line
1.6.2.3
PATCH 2/2 tracing/ring-buffer: unlock recursion protection on discard
In reply to Frederic WeisbeckerThe pair of helpers trace_recursive_lock() trace_recursive_unlock() have been introduced recently to provide a generic tracing recursion protection.
They are used in a symetric way:
-trace_recursive_lock() on buffer reserve
-trace_recursive_unlock() on buffer commit
However sometimes, we don't commit but discard en entry to the buffer, ie: in case of filter checking.
Then we must also unlock the recursion protection on discard time, otherwise the tracing gets definetly deactivated and a warning is raised spuriously, such as:
111.119821] ------------[ cut here ]------------
[ 111.119829] WARNING: at kernel/trace/ring_buffer.c:1498 ring_buffer_lock_reserve+0x1b7/0x1d0() [ 111.119835] Hardware name: AMILO Li 2727 [ 111.119839] Modules linked in: [ 111.119846] Pid: 5731, comm: Xorg Tainted: G W 2.6.30-rc1 #69 [ 111.119851] Call Trace: [ 111.119863] [<ffffffff8025ce68>] warn_slowpath+0xd8/0x130 [ 111.119873] [<ffffffff8028a30f>] ? __lock_acquire+0x19f/0x1ae0 [ 111.119882] [<ffffffff8028a30f>] ? __lock_acquire+0x19f/0x1ae0 [ 111.119891] [<ffffffff802199b0>] ? native_sched_clock+0x20/0x70 [ 111.119899] [<ffffffff80286dee>] ? put_lock_stats+0xe/0x30 [ 111.119906] [<ffffffff80286eb8>] ? lock_release_holdtime+0xa8/0x150 [ 111.119913] [<ffffffff802c8ae7>] ring_buffer_lock_reserve+0x1b7/0x1d0 [ 111.119921] [<ffffffff802cd110>] trace_buffer_lock_reserve+0x30/0x70 [ 111.119930] [<ffffffff802ce000>] trace_current_buffer_lock_reserve+0x20/0x30 [ 111.119939] [<ffffffff802474e8>] ftrace_raw_event_sched_switch+0x58/0x100 [ 111.119948] [<ffffffff808103b7>] __schedule+0x3a7/0x4cd [ 111.119957] [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b [ 111.119964] [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b [ 111.119971] [<ffffffff80810c08>] schedule+0x18/0x40 [ 111.119977] [<ffffffff80810e09>] preempt_schedule+0x39/0x60 [ 111.119985] [<ffffffff80813bd3>] _read_unlock+0x53/0x60 [ 111.119993] [<ffffffff807259d2>] sock_def_readable+0x72/0x80 [ 111.120002] [<ffffffff807ad5ed>] unix_stream_sendmsg+0x24d/0x3d0 [ 111.120011] [<ffffffff807219a3>] sock_aio_write+0x143/0x160 [ 111.120019] [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b [ 111.120026] [<ffffffff80721860>] ? sock_aio_write+0x0/0x160 [ 111.120033] [<ffffffff80721860>] ? sock_aio_write+0x0/0x160 [ 111.120042] [<ffffffff8031c283>] do_sync_readv_writev+0xf3/0x140 [ 111.120049] [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b [ 111.120057] [<ffffffff80276ff0>] ? autoremove_wake_function+0x0/0x40 [ 111.120067] [<ffffffff8045d489>] ? cap_file_permission+0x9/0x10 [ 111.120074] [<ffffffff8045c1e6>] ? security_file_permission+0x16/0x20 [ 111.120082] [<ffffffff8031cab4>] do_readv_writev+0xd4/0x1f0 [ 111.120089] [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b [ 111.120097] [<ffffffff80211b56>] ? ftrace_call+0x5/0x2b [ 111.120105] [<ffffffff8031cc18>] vfs_writev+0x48/0x70 [ 111.120111] [<ffffffff8031cd65>] sys_writev+0x55/0xc0 [ 111.120119] [<ffffffff80211e32>] system_call_fastpath+0x16/0x1b [ 111.120125] ---[ end trace 15605f4e98d5ccb5 ]---
[ Impact: fix spurious warning/tracing shutdown ]
--- kernel/trace/ring_buffer.c | 18 +++++++++++++----- 1 files changed, 13 insertions(+), 5 deletions(-)
kernel/trace/ring_buffer.c +13 -5
@@ -1642,6 +1642,14 @@ int ring_buffer_unlock_commit(struct ring_buffer *buffer, }EXPORT_SYMBOL_GPL (ring_buffer_unlock_commit); +static inline void rb_event_discard (struct ring_buffer_event *event) +{ + event->type =RINGBUF_TYPE_PADDING ; +/* time delta must be non zero */ +if (!event->time_delta) + event->time_delta =1 ; +} +/** * ring_buffer_event_discard - discard any event in the ring buffer * @event: the event to discard
@@ -1656,10 +1664,8 @@ EXPORT_SYMBOL_GPL(ring_buffer_unlock_commit);*/ void ring_buffer_event_discard (struct ring_buffer_event *event) { - event->type =RINGBUF_TYPE_PADDING ; -/* time delta must be non zero */ -if (!event->time_delta) - event->time_delta =1 ; +rb_event_discard (event); +trace_recursive_unlock (); }EXPORT_SYMBOL_GPL (ring_buffer_event_discard);
@@ -1690,7 +1696,7 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,int cpu;/* The event is discarded regardless */ -ring_buffer_event_discard (event); +rb_event_discard (event);/* * This must only be called if the event has not been
@@ -1735,6 +1741,8 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,if (rb_is_commit (cpu_buffer, event))rb_set_commit_to_write (cpu_buffer); +trace_recursive_unlock (); +/* * Only the last preempt count needs to restore preemption. */
Signature · 1 line
1.6.2.3
Quoted 13 lines · Frederic Weisbecker
Hi,
Here are two fixes for tracing recursion protection.
The first slightly improves debugging in case of recursion detection. The second fixes a spurious warning and tracing disabling while activating a filter.The following changes since commit
8e668b5b3455207e4540fc7ccab9ecf70142f288:
Steven Rostedt (1):
tracing: remove format attribute of inline functionare available in the git repository at:
Quoted 8 lines
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.gittracing/recursionFrederic Weisbecker (2):
tracing/core: Add current context on tracing recursion warning tracing/ring-buffer: unlock recursion protection on discardkernel/trace/ring_buffer.c | 25 ++++++++++++++++++++----- 1 files changed, 20 insertions(+), 5 deletions(-)
Pulled, thanks Frederic! Steve, any objections?
Ingo
Steven Rostedt
maintainerQuoted 18 lines · Ingo Molnar
Quoted 15 lines · Frederic Weisbecker
Hi,
Here are two fixes for tracing recursion protection.
The first slightly improves debugging in case of recursion detection. The second fixes a spurious warning and tracing disabling while activating a filter.The following changes since commit
8e668b5b3455207e4540fc7ccab9ecf70142f288:
Steven Rostedt (1):
tracing: remove format attribute of inline functionare available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.gittracing/recursion
Quoted 8 lines
Quoted 6 lines
Frederic Weisbecker (2):
tracing/core: Add current context on tracing recursion warning tracing/ring-buffer: unlock recursion protection on discardkernel/trace/ring_buffer.c | 25 ++++++++++++++++++++----- 1 files changed, 20 insertions(+), 5 deletions(-)Pulled, thanks Frederic! Steve, any objections?
Ug, there was a whole block of email that I missed.
No objections here.
-- Steve