tracing/filters/ringbuffer: updates for 2.6.31
Subsystems (2)
Steven Rostedt
maintainerIngo,
Could you run these through your tests. I ran them through mine and they passed.
These should really make it into 31 since they all fix some bugs.
Please pull the latest tip/tracing/urgent tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/urgent
Li Zefan (2):
tracing/filters: free filter_string in destroy_preds() tracing/filters: fix race between filter setting and module unload
Steven Rostedt (3):
ring-buffer: have benchmark test handle discarded events
ring-buffer: remove unused variable
ring-buffer: use commit counters for commit pointer accounting
----
kernel/trace/ring_buffer.c | 155 ++++++++++++++++----------------- kernel/trace/ring_buffer_benchmark.c | 8 +- kernel/trace/trace_events_filter.c | 28 +++---- 3 files changed, 91 insertions(+), 100 deletions(-)
Steven Rostedt
maintainerPATCH 1/5 ring-buffer: have benchmark test handle discarded events
From: Steven Rostedt <[email protected]>
With the addition of commit:
c7b0930857e2278f2e7714db6294e94c57f623b0
ring-buffer: prevent adding write in discarded area
The ring buffer may now add discarded events when a write passes the end of a buffer page. Before, a discarded event was only added when the tracer deliberately created one. The ring buffer benchmark test does not handle discarded events when it reads the buffer and fails when it encounters one.
Also fix the increment for large data entries (luckily, the test did not add any yet).
[ Impact: fix false failure of ring buffer self test ]
--- kernel/trace/ring_buffer_benchmark.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-)
kernel/trace/ring_buffer_benchmark.c +5 -3
@@ -102,8 +102,10 @@ static enum event_status read_page(int cpu) event = (void *)&rpage->data[i];switch (event->type_len) {case RINGBUF_TYPE_PADDING : -/* We don't expect any padding */ -KILL_TEST (); +/* failed writes may be discarded events */ +if (!event->time_delta) +KILL_TEST (); + inc = event->array[0 ] +4 ;break ;case RINGBUF_TYPE_TIME_EXTEND : inc =8 ;
@@ -119,7 +121,7 @@ static enum event_status read_page(int cpu)KILL_TEST ();break ; } - inc = event->array[0 ]; + inc = event->array[0 ] +4 ;break ;default : entry =ring_buffer_event_data (event);
Signature · 3 lines
1.6.3.1 --
Steven Rostedt
maintainerPATCH 3/5 ring-buffer: use commit counters for commit pointer accounting
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The ring buffer is made up of three sets of pointers.
The head page pointer, which points to the next page for the reader to get.
The commit pointer and commit index, which points to the page and index of the last committed write respectively.
The tail pointer and tail index, which points to the page and the index of the last reserved data respectively (non committed).
The commit pointer is only moved forward by the outer most writer. If a nested writer comes in, it will not move the pointer forward.
The current implementation has a flaw. It assumes that the outer most writer successfully reserved data. There's a small race window where the outer most writer could find the tail pointer, but a nested writer could come in (via interrupt) and move the tail forward, and even the commit forward.
The outer writer would not realized the commit moved forward and the accounting will break.
This patch changes the design to use counters in the per cpu buffers to keep track of commits. The counters are incremented at the start of the commit, and decremented at the end. If the end commit counter is 1, then it moves the commit pointers. A loop is made to check for races between checking and moving the commit pointers. Only the outer commit should move the pointers anyway.
The test of knowing if a reserve is equal to the last commit update is still needed to know for time keeping. The time code is much less racey than the commit updates.
This change not only solves the mentioned race, but also makes the code simpler.
[ Impact: fix commit race and simplify code ]
--- kernel/trace/ring_buffer.c | 154 +++++++++++++++++++++---------------------- 1 files changed, 75 insertions(+), 79 deletions(-)
kernel/trace/ring_buffer.c +75 -79
@@ -415,6 +415,8 @@ struct ring_buffer_per_cpu {unsigned long overrun;unsigned long read;local_t entries; +local_t committing; +local_t commits;u64 write_stamp;u64 read_stamp;atomic_t record_disabled;
@@ -1015,8 +1017,8 @@ rb_event_index(struct ring_buffer_event *event) }static inline int -rb_is_commit (struct ring_buffer_per_cpu *cpu_buffer, -struct ring_buffer_event *event) +rb_event_is_commit (struct ring_buffer_per_cpu *cpu_buffer, +struct ring_buffer_event *event) {unsigned long addr = (unsigned long )event;unsigned long index;
@@ -1029,31 +1031,6 @@ rb_is_commit(struct ring_buffer_per_cpu *cpu_buffer, }static void -rb_set_commit_event (struct ring_buffer_per_cpu *cpu_buffer, -struct ring_buffer_event *event) -{ -unsigned long addr = (unsigned long )event; -unsigned long index; - - index =rb_event_index (event); - addr &=PAGE_MASK ; - -while (cpu_buffer->commit_page->page != (void *)addr) { -if (RB_WARN_ON (cpu_buffer, - cpu_buffer->commit_page == cpu_buffer->tail_page)) -return ; - cpu_buffer->commit_page->page->commit = - cpu_buffer->commit_page->write; -rb_inc_page (cpu_buffer, &cpu_buffer->commit_page); - cpu_buffer->write_stamp = - cpu_buffer->commit_page->page->time_stamp; - } - -/* Now set the commit to the event's index */ -local_set (&cpu_buffer->commit_page->page->commit, index); -} - -static void rb_set_commit_to_write (struct ring_buffer_per_cpu *cpu_buffer) {/*
@@ -1319,15 +1296,6 @@ rb_move_tail(struct ring_buffer_per_cpu *cpu_buffer,rb_reset_tail (cpu_buffer, tail_page, tail, length); -/* -* If this was a commit entry that failed, -* increment that too -*/ -if (tail_page == cpu_buffer->commit_page && - tail ==rb_commit_index (cpu_buffer)) { -rb_set_commit_to_write (cpu_buffer); - } -__raw_spin_unlock (&cpu_buffer->lock);local_irq_restore (flags);
@@ -1377,11 +1345,11 @@ __rb_reserve_next(struct ring_buffer_per_cpu *cpu_buffer,local_inc (&tail_page->entries);/* -* If this is a commit and the tail is zero, then update -* this page's time stamp. +* If this is the first commit on the page, then update +* its timestamp. */ -if (!tail &&rb_is_commit (cpu_buffer, event)) - cpu_buffer->commit_page->page->time_stamp = *ts; +if (!tail) + tail_page->page->time_stamp = *ts;return event; }
@@ -1450,16 +1418,16 @@ rb_add_time_stamp(struct ring_buffer_per_cpu *cpu_buffer,return -EAGAIN ;/* Only a commited time event can update the write stamp */ -if (rb_is_commit (cpu_buffer, event)) { +if (rb_event_is_commit (cpu_buffer, event)) {/* -* If this is the first on the page, then we need to -* update the page itself, and just put in a zero. +* If this is the first on the page, then it was +* updated with the page itself. Try to discard it +* and if we can't just make it zero. */ if (rb_event_index (event)) { event->time_delta = *delta &TS_MASK ; event->array[0 ] = *delta >>TS_SHIFT ; }else { - cpu_buffer->commit_page->page->time_stamp = *ts;/* try to discard, since we do not need this */ if (!rb_try_to_discard (cpu_buffer, event)) {/* nope, just zero it */
@@ -1485,6 +1453,44 @@ rb_add_time_stamp(struct ring_buffer_per_cpu *cpu_buffer,return ret; } +static void rb_start_commit (struct ring_buffer_per_cpu *cpu_buffer) +{ +local_inc (&cpu_buffer->committing); +local_inc (&cpu_buffer->commits); +} + +static void rb_end_commit (struct ring_buffer_per_cpu *cpu_buffer) +{ +unsigned long commits; + +if (RB_WARN_ON (cpu_buffer, + !local_read (&cpu_buffer->committing))) +return ; + + again: + commits =local_read (&cpu_buffer->commits); +/* synchronize with interrupts */ +barrier (); +if (local_read (&cpu_buffer->committing) ==1 ) +rb_set_commit_to_write (cpu_buffer); + +local_dec (&cpu_buffer->committing); + +/* synchronize with interrupts */ +barrier (); + +/* +* Need to account for interrupts coming in between the +* updating of the commit page and the clearing of the +* committing counter. +*/ +if (unlikely (local_read (&cpu_buffer->commits) != commits) && + !local_read (&cpu_buffer->committing)) { +local_inc (&cpu_buffer->committing); + goto again; + } +} +static struct ring_buffer_event *rb_reserve_next_event (struct ring_buffer_per_cpu *cpu_buffer,unsigned long length)
@@ -1494,6 +1500,8 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,int commit =0 ;int nr_loops =0 ; +rb_start_commit (cpu_buffer); + length =rb_calculate_event_length (length); again:/*
@@ -1506,7 +1514,7 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,* Bail! */ if (RB_WARN_ON (cpu_buffer, ++nr_loops >1000 )) -return NULL ; + goto out_fail; ts =rb_time_stamp (cpu_buffer->buffer, cpu_buffer->cpu);
@@ -1537,7 +1545,7 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer, commit =rb_add_time_stamp (cpu_buffer, &ts, &delta);if (commit == -EBUSY ) -return NULL ; + goto out_fail;if (commit == -EAGAIN ) goto again;
@@ -1551,28 +1559,19 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,if (unlikely (PTR_ERR (event) == -EAGAIN )) goto again; -if (!event) { -if (unlikely (commit)) -/* -* Ouch! We needed a timestamp and it was commited. But -* we didn't get our event reserved. -*/ -rb_set_commit_to_write (cpu_buffer); -return NULL ; - } +if (!event) + goto out_fail; -/* -* If the timestamp was commited, make the commit our entry -* now so that we will update it when needed. -*/ -if (unlikely (commit)) -rb_set_commit_event (cpu_buffer, event); -else if (!rb_is_commit (cpu_buffer, event)) +if (!rb_event_is_commit (cpu_buffer, event)) delta =0 ; event->time_delta = delta;return event; + + out_fail: +rb_end_commit (cpu_buffer); +return NULL ; }#define TRACE_RECURSIVE_DEPTH 16
@@ -1682,13 +1681,14 @@ static void rb_commit(struct ring_buffer_per_cpu *cpu_buffer, {local_inc (&cpu_buffer->entries); -/* Only process further if we own the commit */ -if (!rb_is_commit (cpu_buffer, event)) -return ; - - cpu_buffer->write_stamp += event->time_delta; +/* +* The event first in the commit queue updates the +* time stamp. +*/ +if (rb_event_is_commit (cpu_buffer, event)) + cpu_buffer->write_stamp += event->time_delta; -rb_set_commit_to_write (cpu_buffer); +rb_end_commit (cpu_buffer); }/**
@@ -1777,15 +1777,15 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,/* The event is discarded regardless */ rb_event_discard (event); + cpu =smp_processor_id (); + cpu_buffer = buffer->buffers[cpu]; +/* * This must only be called if the event has not been * committed yet. Thus we can assume that preemption * is still disabled. */ -RB_WARN_ON (buffer,preemptible ()); - - cpu =smp_processor_id (); - cpu_buffer = buffer->buffers[cpu]; +RB_WARN_ON (buffer, !local_read (&cpu_buffer->committing));if (!rb_try_to_discard (cpu_buffer, event)) goto out;
@@ -1796,13 +1796,7 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,*/ local_inc (&cpu_buffer->entries); out: -/* -* If a write came in and pushed the tail page -* we still need to update the commit pointer -* if we were the commit. -*/ -if (rb_is_commit (cpu_buffer, event)) -rb_set_commit_to_write (cpu_buffer); +rb_end_commit (cpu_buffer);trace_recursive_unlock ();
@@ -2720,6 +2714,8 @@ rb_reset_cpu(struct ring_buffer_per_cpu *cpu_buffer) cpu_buffer->overrun =0 ; cpu_buffer->read =0 ;local_set (&cpu_buffer->entries,0 ); +local_set (&cpu_buffer->committing,0 ); +local_set (&cpu_buffer->commits,0 ); cpu_buffer->write_stamp =0 ; cpu_buffer->read_stamp =0 ;
Signature · 3 lines
1.6.3.1 --
Steven Rostedt
maintainerPATCH 2/5 ring-buffer: remove unused variable
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
[Impact: fix compiler warning ]
--- kernel/trace/ring_buffer.c | 1 - 1 files changed, 0 insertions(+), 1 deletions(-)
kernel/trace/ring_buffer.c +0 -1
@@ -1233,7 +1233,6 @@ rb_move_tail(struct ring_buffer_per_cpu *cpu_buffer, {struct buffer_page *next_page, *head_page, *reader_page;struct ring_buffer *buffer = cpu_buffer->buffer; -struct ring_buffer_event *event;bool lock_taken = false;unsigned long flags;
Signature · 3 lines
1.6.3.1 --
Steven Rostedt
maintainerPATCH 5/5 tracing/filters: fix race between filter setting and module unload
In reply to Steven RostedtFrom: Li Zefan <[email protected]>
Module unload is protected by event_mutex, while setting filter is protected by filter_mutex. This leads to the race:
echo 'bar == 0 || bar == 10' \ |
> sample/filter |
| insmod sample.ko
add_pred("bar == 0") |
-> n_preds == 1 |
add_pred("bar == 100") |
-> n_preds == 2 |
| rmmod sample.ko
| insmod sample.ko
add_pred("&&") |
-> n_preds == 1 (should be 3) |Now event->filter->preds is corrupted. An then when filter_match_preds() is called, the WARN_ON() in it will be triggered.
To avoid the race, we remove filter_mutex, and replace it with event_mutex.
[ Impact: prevent corruption of filters by module removing and loading ]
LKML-Reference: <[email protected]>
--- kernel/trace/trace_events_filter.c | 27 ++++++++++----------------- 1 files changed, 10 insertions(+), 17 deletions(-)
kernel/trace/trace_events_filter.c +10 -17
@@ -27,8 +27,6 @@#include "trace.h" #include "trace_output.h" -static DEFINE_MUTEX (filter_mutex); -enum filter_op_ids {OP_OR ,
@@ -294,12 +292,12 @@ void print_event_filter(struct ftrace_event_call *call, struct trace_seq *s) {struct event_filter *filter = call->filter; -mutex_lock (&filter_mutex); +mutex_lock (&event_mutex);if (filter->filter_string)trace_seq_printf (s,"%s\n" , filter->filter_string);else trace_seq_printf (s,"none\n" ); -mutex_unlock (&filter_mutex); +mutex_unlock (&event_mutex); } void print_subsystem_event_filter(struct event_subsystem *system,
@@ -307,12 +305,12 @@ void print_subsystem_event_filter(struct event_subsystem *system, {struct event_filter *filter = system->filter; -mutex_lock (&filter_mutex); +mutex_lock (&event_mutex);if (filter->filter_string)trace_seq_printf (s,"%s\n" , filter->filter_string);else trace_seq_printf (s,"none\n" ); -mutex_unlock (&filter_mutex); +mutex_unlock (&event_mutex); } static struct ftrace_event_field *
@@ -434,7 +432,6 @@ static void filter_free_subsystem_preds(struct event_subsystem *system) filter->n_preds =0 ; } -mutex_lock (&event_mutex);list_for_each_entry (call, &ftrace_events, list) {if (!call->define_fields)continue ;
@@ -444,7 +441,6 @@ static void filter_free_subsystem_preds(struct event_subsystem *system)remove_filter_string (call->filter); } } -mutex_unlock (&event_mutex); } static int filter_add_pred_fn(struct filter_parse_state *ps,
@@ -631,7 +627,6 @@ static int filter_add_subsystem_pred(struct filter_parse_state *ps, filter->preds[filter->n_preds] = pred; filter->n_preds++; -mutex_lock (&event_mutex);list_for_each_entry (call, &ftrace_events, list) {if (!call->define_fields)
@@ -642,14 +637,12 @@ static int filter_add_subsystem_pred(struct filter_parse_state *ps, err =filter_add_pred (ps, call, pred);if (err) { -mutex_unlock (&event_mutex);filter_free_subsystem_preds (system);parse_error (ps,FILT_ERR_BAD_SUBSYS_FILTER ,0 ); goto out; }replace_filter_string (call->filter, filter_string); } -mutex_unlock (&event_mutex); out:return err; }
@@ -1076,12 +1069,12 @@ int apply_event_filter(struct ftrace_event_call *call, char *filter_string)struct filter_parse_state *ps; -mutex_lock (&filter_mutex); +mutex_lock (&event_mutex);if (!strcmp (strstrip (filter_string),"0" )) {filter_disable_preds (call);remove_filter_string (call->filter); -mutex_unlock (&filter_mutex); +mutex_unlock (&event_mutex);return 0 ; }
@@ -1109,7 +1102,7 @@ out:postfix_clear (ps);kfree (ps); out_unlock: -mutex_unlock (&filter_mutex); +mutex_unlock (&event_mutex);return err; }
@@ -1121,12 +1114,12 @@ int apply_subsystem_event_filter(struct event_subsystem *system,struct filter_parse_state *ps; -mutex_lock (&filter_mutex); +mutex_lock (&event_mutex);if (!strcmp (strstrip (filter_string),"0" )) {filter_free_subsystem_preds (system);remove_filter_string (system->filter); -mutex_unlock (&filter_mutex); +mutex_unlock (&event_mutex);return 0 ; }
@@ -1154,7 +1147,7 @@ out:postfix_clear (ps);kfree (ps); out_unlock: -mutex_unlock (&filter_mutex); +mutex_unlock (&event_mutex);return err; }
Signature · 3 lines
1.6.3.1 --
Steven Rostedt
maintainerPATCH 4/5 tracing/filters: free filter_string in destroy_preds()
In reply to Steven RostedtFrom: Li Zefan <[email protected]>
filter->filter_string is not freed when unloading a module:
# insmod trace-events-sample.ko # echo "bar < 100" > /mnt/tracing/events/sample/foo_bar/filter # rmmod trace-events-sample.ko
[ Impact: fix memory leak when unloading module ]
LKML-Reference: <[email protected]>
--- kernel/trace/trace_events_filter.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
kernel/trace/trace_events_filter.c +1 -0
@@ -381,6 +381,7 @@ void destroy_preds(struct ftrace_event_call *call)filter_free_pred (filter->preds[i]); }kfree (filter->preds); +kfree (filter->filter_string);kfree (filter); call->filter =NULL ; }
Signature · 3 lines
1.6.3.1 --
Quoted 22 lines · Steven Rostedt
Ingo,
Could you run these through your tests. I ran them through mine and they passed.
These should really make it into 31 since they all fix some bugs.
Please pull the latest tip/tracing/urgent tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/urgentLi Zefan (2):
tracing/filters: freefilter_stringindestroy_preds()tracing/filters: fix race between filter setting and module unloadSteven Rostedt (3):
ring-buffer: have benchmark test handle discarded events
ring-buffer: remove unused variable
ring-buffer: use commit counters for commit pointer accounting
Quoted 5 lines
----
kernel/trace/ring_buffer.c | 155 ++++++++++++++++----------------- kernel/trace/ring_buffer_benchmark.c | 8 +- kernel/trace/trace_events_filter.c | 28 +++---- 3 files changed, 91 insertions(+), 100 deletions(-)
Pulled, thanks Steve!
Ingo
hm, there's a new build failure with this lot:
kernel/trace/ring_buffer.c:1298: error: ‘event’ undeclared (first use in this function) kernel/trace/ring_buffer.c:1298: error: (Each undeclared identifier is reported only once kernel/trace/ring_buffer.c:1298: error: for each function it appears in.)
Ingo
Steven Rostedt
maintainerQuoted 6 lines · Ingo Molnar
hm, there's a new build failure with this lot:
kernel/trace/ring_buffer.c:1298: error: ?event? undeclared (first use in this function)kernel/trace/ring_buffer.c:1298: error: (Each undeclared identifier is reported only oncekernel/trace/ring_buffer.c:1298: error: for each function it appears in.)
Strange, I have a blank line at line 1298:
1294 cpu_buffer->tail_page->page->time_stamp = *ts;
1295 }
1296
1297 rb_reset_tail(cpu_buffer, tail_page, tail, length);
1298
1299 __raw_spin_unlock(&cpu_buffer->lock);
1300 local_irq_restore(flags);
1301
-- Steve
Quoted 11 lines · Steven Rostedt
Quoted 6 lines · Ingo Molnar
hm, there's a new build failure with this lot:
kernel/trace/ring_buffer.c:1298: error: ?event? undeclared (first use in this function)kernel/trace/ring_buffer.c:1298: error: (Each undeclared identifier is reported only oncekernel/trace/ring_buffer.c:1298: error: for each function it appears in.)Strange, I have a blank line at line 1298:
Quoted 8 lines
1294
cpu_buffer->tail_page->page->time_stamp= *ts;
1295 }
1296
1297rb_reset_tail(cpu_buffer, tail_page, tail, length);
1298
1299__raw_spin_unlock(&cpu_buffer->lock);
1300local_irq_restore(flags);
1301
ah. git auto-merge getting confused:
}
kmemcheck_annotate_bitfield(event, bitfield);
rb_reset_tail(cpu_buffer, tail_page, tail, length);it's rare that we see something like that.
Ingo