ring-buffer: optimize to 17% performance increase
Subsystems (2)
Steven Rostedt
maintainerIngo,
This patch series tunes the ring buffer to be a bit faster. I used the ring-buffer-benmark test to help give a good idea on the performance of the buffer. I ran it on an 2.8 GHz 4way box on an idle system. I only wanted to test the write without the reader, since the reader can produce some cacheline bouncing. To do this I inserted the benchmark module with the "disable_reader=1" option.
Note, when I disable the ring buffer and run the test, I get an average of 87 ns. Thus the overhead of the test is 87ns, and I will show both the full time and the 87 subtracted from the time (in parenthesis).
I'm also including the size of the ring_buffer.o object since some changes helped in shrinking the text segments too.
Before the patch series:
benchmark: 307 ns (220 ns)
text data bss dec hex filename
16554 24 12 16590 40ce kernel/trace/ring_buffer.o
commit 1cd8d7358948909ab80b254eb14bcebc555ad417
ring-buffer: remove type parameter from rb_reserve_next_event
benchmark: 302 ns (215 ns)
text data bss dec hex filename
16538 24 12 16574 40be kernel/trace/ring_buffer.o
commit be957c447f7233a67904a1b11eb3ab61e702bf4d
ring-buffer: move calculation of event length
benchmark: 293 ns (206 ns)
text data bss dec hex filename
16490 24 12 16526 408e kernel/trace/ring_buffer.o
commit 0f0c85fc80adbbd2265d89867d743f929d516805
ring-buffer: small optimizations
benchmark: 285 ns (198 ns)
text data bss dec hex filename
16474 24 12 16510 407e kernel/trace/ring_buffer.o
commit 88eb0125362f2ab272cbaf84252cf101ddc2dec9
ring-buffer: use internal time stamp function
benchmark: 282 ns (195 ns)
text data bss dec hex filename
16474 24 12 16510 407e kernel/trace/ring_buffer.o
commit 168b6b1d0594c7866caa73b12f3b8d91075695f2
ring-buffer: move code around to remove some branches
benchmark: 270 ns (183 ns)
text data bss dec hex filename
16490 24 12 16526 408e kernel/trace/ring_buffer.o
Thus we went from an average of 220 ns per recording, to 183 ns. Which is about a 17% performance gain.
For your information:
Adding a reader that reads via pages (like splice), the time jumps to 326 ns.
Adding a reader that reades event by event it jumps to (with lots of overruns)
469 ns.
But disabling the ring buffer, the overhead for the test jumps from 87 ns to 113 ns, making the ring buffer cost with busy reader: 213 ns and 356 ns.
Please pull the latest tip/tracing/ftrace tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/ftrace
Steven Rostedt (5):
ring-buffer: remove type parameter from rb_reserve_next_event
ring-buffer: move calculation of event length
ring-buffer: small optimizations
ring-buffer: use internal time stamp function
ring-buffer: move code around to remove some branches
----
kernel/trace/ring_buffer.c | 63 +++++++++++++++++++++++++------------------- 1 files changed, 36 insertions(+), 27 deletions(-)
Steven Rostedt
maintainerPATCH 1/5 ring-buffer: remove type parameter from rb_reserve_next_event
From: Steven Rostedt <[email protected]>
The rb_reserve_next_event is only called for the data type (type = 0). There is no reason to pass in the type to the function.
Before:
text data bss dec hex filename
16554 24 12 16590 40ce kernel/trace/ring_buffer.o
After:
text data bss dec hex filename
16538 24 12 16574 40be kernel/trace/ring_buffer.o
[ Impact: cleaner, smaller and slightly more efficient code ]
--- kernel/trace/ring_buffer.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/trace/ring_buffer.c +4 -4
@@ -1389,7 +1389,7 @@ rb_add_time_stamp(struct ring_buffer_per_cpu *cpu_buffer,static struct ring_buffer_event *rb_reserve_next_event (struct ring_buffer_per_cpu *cpu_buffer, -unsigned type,unsigned long length) +unsigned long length) {struct ring_buffer_event *event;u64 ts, delta;
@@ -1448,7 +1448,7 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,/* Non commits have zero deltas */ delta =0 ; - event =__rb_reserve_next (cpu_buffer, type, length, &ts); + event =__rb_reserve_next (cpu_buffer,0 , length, &ts);if (PTR_ERR (event) == -EAGAIN ) goto again;
@@ -1556,7 +1556,7 @@ ring_buffer_lock_reserve(struct ring_buffer *buffer, unsigned long length)if (length >BUF_PAGE_SIZE ) goto out; - event =rb_reserve_next_event (cpu_buffer,0 , length); + event =rb_reserve_next_event (cpu_buffer, length);if (!event) goto out;
@@ -1782,7 +1782,7 @@ int ring_buffer_write(struct ring_buffer *buffer, goto out; event_length =rb_calculate_event_length (length); - event =rb_reserve_next_event (cpu_buffer,0 , event_length); + event =rb_reserve_next_event (cpu_buffer, event_length);if (!event) goto out;
Signature · 3 lines
1.6.2.4 --
Steven Rostedt
maintainerPATCH 2/5 ring-buffer: move calculation of event length
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The event length is calculated and passed in to rb_reserve_next_event in two different locations. Having rb_reserve_next_event do the calculations directly makes only one location to do the change and causes the calculation to be inlined by gcc.
Before:
text data bss dec hex filename
16538 24 12 16574 40be kernel/trace/ring_buffer.o
After:
text data bss dec hex filename
16490 24 12 16526 408e kernel/trace/ring_buffer.o
[ Impact: smaller more efficient code ]
--- kernel/trace/ring_buffer.c | 14 +++++++++----- 1 files changed, 9 insertions(+), 5 deletions(-)
kernel/trace/ring_buffer.c +9 -5
@@ -367,6 +367,9 @@ static inline int test_time_stamp(u64 delta)#define BUF_PAGE_SIZE (PAGE_SIZE - BUF_PAGE_HDR_SIZE) +/* Max payload is BUF_PAGE_SIZE - header (8bytes) */ +#define BUF_MAX_DATA_SIZE (BUF_PAGE_SIZE - (sizeof(u32) * 2)) +int ring_buffer_print_page_header (struct trace_seq *s) {struct buffer_data_page field;
@@ -1396,6 +1399,7 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,int commit =0 ;int nr_loops =0 ; + length =rb_calculate_event_length (length); again:/* * We allow for interrupts to reenter here and do a trace.
@@ -1552,8 +1556,7 @@ ring_buffer_lock_reserve(struct ring_buffer *buffer, unsigned long length)if (atomic_read (&cpu_buffer->record_disabled)) goto out; - length =rb_calculate_event_length (length); -if (length >BUF_PAGE_SIZE ) +if (length >BUF_MAX_DATA_SIZE ) goto out; event =rb_reserve_next_event (cpu_buffer, length);
@@ -1758,7 +1761,6 @@ int ring_buffer_write(struct ring_buffer *buffer, {struct ring_buffer_per_cpu *cpu_buffer;struct ring_buffer_event *event; -unsigned long event_length;void *body;int ret = -EBUSY ;int cpu, resched;
@@ -1781,8 +1783,10 @@ int ring_buffer_write(struct ring_buffer *buffer,if (atomic_read (&cpu_buffer->record_disabled)) goto out; - event_length =rb_calculate_event_length (length); - event =rb_reserve_next_event (cpu_buffer, event_length); +if (length >BUF_MAX_DATA_SIZE ) + goto out; + + event =rb_reserve_next_event (cpu_buffer, length);if (!event) goto out;
Signature · 3 lines
1.6.2.4 --
Steven Rostedt
maintainerPATCH 3/5 ring-buffer: small optimizations
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Doing some small changes in the fast path of the ring buffer recording saves over 3% in the ring-buffer-benchmark test.
[ Impact: a little faster ring buffer recording ]
--- kernel/trace/ring_buffer.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/trace/ring_buffer.c +6 -6
@@ -1000,7 +1000,7 @@ rb_event_index(struct ring_buffer_event *event)return (addr & ~PAGE_MASK ) - (PAGE_SIZE -BUF_PAGE_SIZE ); } -static int +static inline int rb_is_commit (struct ring_buffer_per_cpu *cpu_buffer,struct ring_buffer_event *event) {
@@ -1423,9 +1423,9 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,* also be made. But only the entry that did the actual * commit will be something other than zero. */ -if (cpu_buffer->tail_page == cpu_buffer->commit_page && -rb_page_write (cpu_buffer->tail_page) == -rb_commit_index (cpu_buffer)) { +if (likely (cpu_buffer->tail_page == cpu_buffer->commit_page && +rb_page_write (cpu_buffer->tail_page) == +rb_commit_index (cpu_buffer))) { delta = ts - cpu_buffer->write_stamp;
@@ -1436,7 +1436,7 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,if (unlikely (ts < cpu_buffer->write_stamp)) delta =0 ; -if (test_time_stamp (delta)) { +else if (unlikely (test_time_stamp (delta))) { commit =rb_add_time_stamp (cpu_buffer, &ts, &delta);
@@ -1470,7 +1470,7 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,* If the timestamp was commited, make the commit our entry * now so that we will update it when needed. */ -if (commit) +if (unlikely (commit))rb_set_commit_event (cpu_buffer, event);else if (!rb_is_commit (cpu_buffer, event)) delta =0 ;
Signature · 3 lines
1.6.2.4 --
Steven Rostedt
maintainerPATCH 4/5 ring-buffer: use internal time stamp function
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The ring_buffer_time_stamp that is exported adds a little more overhead than is needed for using it internally. This patch adds an internal timestamp function that can be inlined (a single line function) and used internally for the ring buffer.
[ Impact: a little less overhead to the ring buffer ]
--- kernel/trace/ring_buffer.c | 13 +++++++++---- 1 files changed, 9 insertions(+), 4 deletions(-)
kernel/trace/ring_buffer.c +9 -4
@@ -454,13 +454,18 @@ struct ring_buffer_iter {/* Up this if you want to test the TIME_EXTENTS and normalization */ #define DEBUG_SHIFT 0 +static inline u64 rb_time_stamp (struct ring_buffer *buffer,int cpu) +{ +/* shift to debug/test normalization and TIME_EXTENTS */ +return buffer->clock () <<DEBUG_SHIFT ; +} +u64 ring_buffer_time_stamp (struct ring_buffer *buffer,int cpu) {u64 time;preempt_disable_notrace (); -/* shift to debug/test normalization and TIME_EXTENTS */ - time = buffer->clock () <<DEBUG_SHIFT ; + time =rb_time_stamp (buffer, cpu);preempt_enable_no_resched_notrace ();return time;
@@ -1247,7 +1252,7 @@ rb_move_tail(struct ring_buffer_per_cpu *cpu_buffer, cpu_buffer->tail_page = next_page;/* reread the time stamp */ - *ts =ring_buffer_time_stamp (buffer, cpu_buffer->cpu); + *ts =rb_time_stamp (buffer, cpu_buffer->cpu); cpu_buffer->tail_page->page->time_stamp = *ts; }
@@ -1413,7 +1418,7 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,if (RB_WARN_ON (cpu_buffer, ++nr_loops >1000 ))return NULL ; - ts =ring_buffer_time_stamp (cpu_buffer->buffer, cpu_buffer->cpu); + ts =rb_time_stamp (cpu_buffer->buffer, cpu_buffer->cpu);/* * Only the first commit can update the timestamp.
Signature · 3 lines
1.6.2.4 --
Steven Rostedt
maintainerPATCH 5/5 ring-buffer: move code around to remove some branches
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
This is a bit of micro-optimizations. But since the ring buffer is used in tracing every function call, it is an extreme hot path. Every nanosecond counts.
This change shows over 5% improvement in the ring-buffer-benchmark.
[ Impact: more efficient code ]
--- kernel/trace/ring_buffer.c | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-)
kernel/trace/ring_buffer.c +10 -10
@@ -1400,7 +1400,7 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer, unsigned long length) {struct ring_buffer_event *event; -u64 ts, delta; +u64 ts, delta =0 ;int commit =0 ;int nr_loops =0 ;
@@ -1431,20 +1431,21 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,if (likely (cpu_buffer->tail_page == cpu_buffer->commit_page &&rb_page_write (cpu_buffer->tail_page) ==rb_commit_index (cpu_buffer))) { +u64 diff; - delta = ts - cpu_buffer->write_stamp; + diff = ts - cpu_buffer->write_stamp; -/* make sure this delta is calculated here */ +/* make sure this diff is calculated here */ barrier ();/* Did the write stamp get updated already? */ if (unlikely (ts < cpu_buffer->write_stamp)) - delta =0 ; + goto get_event; -else if (unlikely (test_time_stamp (delta))) { + delta = diff; +if (unlikely (test_time_stamp (delta))) { commit =rb_add_time_stamp (cpu_buffer, &ts, &delta); -if (commit == -EBUSY )return NULL ;
@@ -1453,12 +1454,11 @@ rb_reserve_next_event(struct ring_buffer_per_cpu *cpu_buffer,RB_WARN_ON (cpu_buffer, commit <0 ); } - }else -/* Non commits have zero deltas */ - delta =0 ; + } + get_event: event =__rb_reserve_next (cpu_buffer,0 , length, &ts); -if (PTR_ERR (event) == -EAGAIN ) +if (unlikely (PTR_ERR (event) == -EAGAIN )) goto again;if (!event) {
Signature · 3 lines
1.6.2.4 --
Quoted 56 lines · Steven Rostedt
Ingo,
This patch series tunes the ring buffer to be a bit faster. I used the ring-buffer-benmark test to help give a good idea on the performance of the buffer. I ran it on an 2.8 GHz 4way box on an idle system. I only wanted to test the write without the reader, since the reader can produce some cacheline bouncing. To do this I inserted the benchmark module with the "disable_reader=1" option.
Note, when I disable the ring buffer and run the test, I get an average of 87 ns. Thus the overhead of the test is 87ns, and I will show both the full time and the 87 subtracted from the time (in parenthesis).
I'm also including the size of the ring_buffer.o object since some changes helped in shrinking the text segments too.
Before the patch series:
benchmark: 307 ns (220 ns)
text data bss dec hex filename
16554 24 12 16590 40cekernel/trace/ring_buffer.ocommit
1cd8d7358948909ab80b254eb14bcebc555ad417
ring-buffer: remove type parameter fromrb_reserve_next_eventbenchmark: 302 ns (215 ns)
text data bss dec hex filename
16538 24 12 16574 40bekernel/trace/ring_buffer.ocommit
be957c447f7233a67904a1b11eb3ab61e702bf4d
ring-buffer: move calculation of event lengthbenchmark: 293 ns (206 ns)
text data bss dec hex filename
16490 24 12 16526 408ekernel/trace/ring_buffer.ocommit
0f0c85fc80adbbd2265d89867d743f929d516805
ring-buffer: small optimizationsbenchmark: 285 ns (198 ns)
text data bss dec hex filename
16474 24 12 16510 407ekernel/trace/ring_buffer.ocommit
88eb0125362f2ab272cbaf84252cf101ddc2dec9
ring-buffer: use internal time stamp functionbenchmark: 282 ns (195 ns)
text data bss dec hex filename
16474 24 12 16510 407ekernel/trace/ring_buffer.ocommit
168b6b1d0594c7866caa73b12f3b8d91075695f2
ring-buffer: move code around to remove some branches
Quoted 6 lines
benchmark: 270 ns (183 ns)
text data bss dec hex filename
16490 24 12 16526 408ekernel/trace/ring_buffer.oThus we went from an average of 220 ns per recording, to 183 ns. Which is about a 17% performance gain.
Nice!
It's also interesting to see that text size went down when speed went up. I'm wondering how these compiler options:
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_OPTIMIZE_INLINING=y
My guess is that the combo with the highest performance is:
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
# CONFIG_OPTIMIZE_INLINING is not set
Especially if you run it on a fast box with a lot of caches and a modern x86 CPU.
Quoted 25 lines
For your information:
Adding a reader that reads via pages (like splice), the time jumps to 326 ns.
Adding a reader that reades event by event it jumps to (with lots of overruns)
469 ns.But disabling the ring buffer, the overhead for the test jumps from 87 ns to 113 ns, making the ring buffer cost with busy reader: 213 ns and 356 ns.
Please pull the latest tip/tracing/ftrace tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/ftraceSteven Rostedt (5):
ring-buffer: remove type parameter fromrb_reserve_next_event
ring-buffer: move calculation of event length
ring-buffer: small optimizations
ring-buffer: use internal time stamp function
ring-buffer: move code around to remove some branches
Quoted 3 lines
----
kernel/trace/ring_buffer.c | 63 +++++++++++++++++++++++++------------------- 1 files changed, 36 insertions(+), 27 deletions(-)
Pulled, thanks Steve!
Ingo
Steven Rostedt
maintainerQuoted 47 lines · Ingo Molnar
Quoted 44 lines · Steven Rostedt
Before the patch series:
benchmark: 307 ns (220 ns)
text data bss dec hex filename
16554 24 12 16590 40cekernel/trace/ring_buffer.ocommit
1cd8d7358948909ab80b254eb14bcebc555ad417
ring-buffer: remove type parameter fromrb_reserve_next_eventbenchmark: 302 ns (215 ns)
text data bss dec hex filename
16538 24 12 16574 40bekernel/trace/ring_buffer.ocommit
be957c447f7233a67904a1b11eb3ab61e702bf4d
ring-buffer: move calculation of event lengthbenchmark: 293 ns (206 ns)
text data bss dec hex filename
16490 24 12 16526 408ekernel/trace/ring_buffer.ocommit
0f0c85fc80adbbd2265d89867d743f929d516805
ring-buffer: small optimizationsbenchmark: 285 ns (198 ns)
text data bss dec hex filename
16474 24 12 16510 407ekernel/trace/ring_buffer.ocommit
88eb0125362f2ab272cbaf84252cf101ddc2dec9
ring-buffer: use internal time stamp functionbenchmark: 282 ns (195 ns)
text data bss dec hex filename
16474 24 12 16510 407ekernel/trace/ring_buffer.ocommit
168b6b1d0594c7866caa73b12f3b8d91075695f2
ring-buffer: move code around to remove some branchesbenchmark: 270 ns (183 ns)
text data bss dec hex filename
16490 24 12 16526 408ekernel/trace/ring_buffer.o
Quoted 7 lines
Quoted 2 lines
Thus we went from an average of 220 ns per recording, to 183 ns. Which is about a 17% performance gain.
Nice!
It's also interesting to see that text size went down when speed went up. I'm wondering how these compiler options:
But that was not always the case. The biggest boost in performance of the series (the last patch) also increased the size.
Quoted 3 lines
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_OPTIMIZE_INLINING=y
Note, my test runs had both the above configure options disabled. I'll run it again and see how they affect the results:
-- Steve
Steven Rostedt
maintainerQuoted 8 lines · Steven Rostedt
Quoted 3 lines
It's also interesting to see that text size went down when speed went up. I'm wondering how these compiler options:
But that was not always the case. The biggest boost in performance of the series (the last patch) also increased the size.
Quoted 1 line
Quoted 5 lines
Quoted 2 lines
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_OPTIMIZE_INLINING=yNote, my test runs had both the above configure options disabled. I'll run it again and see how they affect the results:
Here's the results:
size=n inline=n 270 size=n inline=y 290 size=y inline=n 315 size=y inline=y 372 (ouch!)
Thus it seems to keep both optimizations off is best for performance.
-- Steve
Steven Rostedt
maintainerQuoted 9 lines · Steven Rostedt
Here's the results:
size=n inline=n 270 size=n inline=y 290 size=y inline=n 315 size=y inline=y 372 (ouch!)Thus it seems to keep both optimizations off is best for performance.
I did not run with ring buffer off on all tests, but the last one brought the overhead of the test up from 87 to 130.
-- Steve
Quoted 16 lines · Steven Rostedt
Quoted 8 lines · Steven Rostedt
Quoted 3 lines
It's also interesting to see that text size went down when speed went up. I'm wondering how these compiler options:
But that was not always the case. The biggest boost in performance of the series (the last patch) also increased the size.
Quoted 1 line
Quoted 5 lines
Quoted 2 lines
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_OPTIMIZE_INLINING=yNote, my test runs had both the above configure options disabled. I'll run it again and see how they affect the results:
Quoted 8 lines
Here's the results:
size=n inline=n 270 size=n inline=y 290 size=y inline=n 315 size=y inline=y 372 (ouch!)Thus it seems to keep both optimizations off is best for performance.
ok. Maybe newer gcc does better with size=y. optimize-inline=n is not a surprise - this is in essence a micro-benchmark where inlining helps - and gcc's inliner isnt too fantastic either.
Ingo
Steven Rostedt
maintainerQuoted 16 lines · Steven Rostedt
Quoted 8 lines · Steven Rostedt
Quoted 3 lines
It's also interesting to see that text size went down when speed went up. I'm wondering how these compiler options:
But that was not always the case. The biggest boost in performance of the series (the last patch) also increased the size.
Quoted 1 line
Quoted 5 lines
Quoted 2 lines
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_OPTIMIZE_INLINING=yNote, my test runs had both the above configure options disabled. I'll run it again and see how they affect the results:
Quoted 8 lines
Here's the results:
size=n inline=n 270 size=n inline=y 290 size=y inline=n 315 size=y inline=y 372 (ouch!)Thus it seems to keep both optimizations off is best for performance.
Versions do make a difference. I just upgraded my distcc gcc on all my distcc boxes from gcc 4.2.2 to 4.4.0 and here's the new results:
size=n inline=n 257 size=n inline=y 259 size=y inline=n 295 size=y inline=y 315
The gcc inlining got a little better, but adding size seems to hurt.
-- Steve