tracing/ring-buffer: updates for tip
Subsystems (2)
Steven Rostedt
maintainerIngo,
Please pull the latest tip/tracing/ftrace tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/ftrace
Steven Rostedt (4):
tracing: have menu default enabled when kernel debug is configured
ring-buffer: only periodically call cond_resched to ring-buffer-benchmark
ring-buffer: add total count in ring-buffer-benchmark
ring-buffer: change WARN_ON from checking preempt_count to preemptible
----
kernel/trace/Kconfig | 1 + kernel/trace/ring_buffer.c | 2 +- kernel/trace/ring_buffer_benchmark.c | 22 ++++++++++++++++++++-- 3 files changed, 22 insertions(+), 3 deletions(-)
Steven Rostedt
maintainerPATCH 1/4 tracing: have menu default enabled when kernel debug is configured
From: Steven Rostedt <[email protected]>
Tracing can be very helpful to debug the kernel. When DEBUG_KERNEL is enabled it is nice to enable the trace menu as well.
This patch only make the tracing menu enabled by default, it does not make any of the tracers enabled. And the menu is only enabled by default if DEBUG_KERNEL is enabled.
[ Impact: show tracing options to those debugging the kernel ]
--- kernel/trace/Kconfig | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
kernel/trace/Kconfig +1 -0
@@ -79,6 +79,7 @@ if TRACING_SUPPORT menuconfig FTRACE bool "Tracers" + default y if DEBUG_KERNEL help Enable the kernel tracing infrastructure.
Signature · 3 lines
1.6.2.4 --
Steven Rostedt
maintainerPATCH 2/4 ring-buffer: only periodically call cond_resched to ring-buffer-benchmark
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Calling cond_resched at every iteration of the loop adds a bit of overhead to the benchmark.
This patch does two things.
1) only calls cond-resched when CONFIG_PREEMPT is not enabled
2) only calls cond-resched after so many traces has been performed.
[ Impact: less overhead to the ring-buffer-benchmark ]
--- kernel/trace/ring_buffer_benchmark.c | 11 +++++++++-- 1 files changed, 9 insertions(+), 2 deletions(-)
kernel/trace/ring_buffer_benchmark.c +9 -2
@@ -218,16 +218,23 @@ static void ring_buffer_producer(void) }do_gettimeofday (&end_tv); -if (consumer && !(++cnt % wakeup_interval)) + cnt++; +if (consumer && !(cnt % wakeup_interval))wake_up_process (consumer); +#ifndef CONFIG_PREEMPT /* * If we are a non preempt kernel, the 10 second run will * stop everything while it runs. Instead, we will call * cond_resched and also add any time that was lost by a * rescedule. +* +* Do a cond resched at the same frequency we would wake up +* the reader. */ -cond_resched (); +if (cnt % wakeup_interval) +cond_resched (); +#endif }while (end_tv.tv_sec < (start_tv.tv_sec +RUN_TIME ) && !kill_test);pr_info ("End ring buffer hammer\n" );
Signature · 3 lines
1.6.2.4 --
Steven Rostedt
maintainerPATCH 3/4 ring-buffer: add total count in ring-buffer-benchmark
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
It is nice to see the overhead of the benchmark test when tracing is disabled. That is, we turn off the ring buffer just to see what the cost of running the loop that calls into the ring buffer is.
Currently, if no entries wer made, we get 0. This is not informative. This patch changes it to check if we had any "missed" (non recorded) events. If so, a total count is also reported.
[ Impact: evaluate the over head of the ring buffer benchmark test ]
--- kernel/trace/ring_buffer_benchmark.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-)
kernel/trace/ring_buffer_benchmark.c +11 -0
@@ -285,6 +285,17 @@ static void ring_buffer_producer(void) avg =1000000 / hit;pr_info ("%ld ns per entry\n" , avg); } + + +if (missed) { +if (time) + missed /= (long )time; + +pr_info ("Total iterations per millisec: %ld\n" , hit + missed); + + avg =1000000 / (hit + missed); +pr_info ("%ld ns per entry\n" , avg); + } }static void wait_to_die (void )
Signature · 3 lines
1.6.2.4 --
Re: [PATCH 3/4] ring-buffer: add total count in ring-buffer-benchmark
kernel/trace/ring_buffer_benchmark.c static void ring_buffer_producer(void) · line 289
Quoted 8 lines · On Fri, 08 May 2009 00:32:53 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/ring_buffer_benchmark.c +2
-0@@ -285,6 +285,17 @@ static void ring_buffer_producer(void) avg =1000000 / hit;pr_info ("%ld ns per entry\n" , avg); } + +
s/\n\n/\n/
kernel/trace/ring_buffer_benchmark.c static void ring_buffer_producer(void) · line 296
Quoted 7 lines
kernel/trace/ring_buffer_benchmark.c +7
-0+if (missed) { +if (time) + missed /= (long )time; + +pr_info ("Total iterations per millisec: %ld\n" , hit + missed); + + avg =1000000 / (hit + missed);
s/1000000/USEC_PER_SEC/?
Hopefully we can't have hit+missed==0.
They're counters and hence should have unsigned types.
But even unsigned up-counters can add to zero if wrapping occurs.
Quoted 3 lines
kernel/trace/ring_buffer_benchmark.c +2
-0+pr_info ("%ld ns per entry\n" , avg); + } }
Steven Rostedt
maintainerRe: [PATCH 3/4] ring-buffer: add total count in ring-buffer-benchmark
Quoted 12 lines · Andrew Morton
kernel/trace/ring_buffer_benchmark.c static void ring_buffer_producer(void) · line 289
Quoted 8 lines · On Fri, 08 May 2009 00:32:53 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/ring_buffer_benchmark.c +2
-0@@ -285,6 +285,17 @@ static void ring_buffer_producer(void) avg =1000000 / hit;pr_info ("%ld ns per entry\n" , avg); } + +s/\n\n/\n/
My pinky must hae been itchy, and hit the enter twice.
Quoted 10 lines
kernel/trace/ring_buffer_benchmark.c static void ring_buffer_producer(void) · line 296
Quoted 8 lines
kernel/trace/ring_buffer_benchmark.c +7
-0+if (missed) { +if (time) + missed /= (long )time; + +pr_info ("Total iterations per millisec: %ld\n" , hit + missed); + + avg =1000000 / (hit + missed);s/1000000/USEC_PER_SEC/?
Heh, after I sent out the patch, I noticed that number, and thought to myself, "oh no, akpm is going to commont on that" ;-)
Quoted 6 lines
Hopefully we can't have hit+missed==0.
They're counters and hence should have unsigned types.
But even unsigned up-counters can add to zero if wrapping occurs.
Eigad! I never thought about an overflow to zero. Will fix.
Thanks!
-- Steve
Quoted 6 lines
Quoted 5 lines
kernel/trace/ring_buffer_benchmark.c +2
-0+pr_info ("%ld ns per entry\n" , avg); + } }
Steven Rostedt
maintainerPATCH 4/4 ring-buffer: change WARN_ON from checking preempt_count to preemptible
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
There's a WARN_ON in the ring buffer code that makes sure preemption is disabled. It checks "!preempt_count()". But when CONFIG_PREEMPT is not enabled, preempt_count() is always zero, and this will trigger the warning.
[ Impact: prevent false warning on non preemptible kernels ]
--- kernel/trace/ring_buffer.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/trace/ring_buffer.c +1 -1
@@ -1688,7 +1688,7 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,* committed yet. Thus we can assume that preemption * is still disabled. */ -RB_WARN_ON (buffer, !preempt_count ()); +RB_WARN_ON (buffer,preemptible ()); cpu =smp_processor_id (); cpu_buffer = buffer->buffers[cpu];
Signature · 3 lines
1.6.2.4 --
Re: [PATCH 4/4] ring-buffer: change WARN_ON from checking preempt_count to preemptible
Quoted 18 lines · On Fri, 08 May 2009 00:32:54 -0400 Steven Rostedt <[email protected]> wrote:
From: Steven Rostedt <[email protected]>
There's a
WARN_ONin the ring buffer code that makes sure preemption is disabled. It checks "!preempt_count()". But whenCONFIG_PREEMPTis not enabled,preempt_count()is always zero, and this will trigger the warning.[ Impact: prevent false warning on non preemptible kernels ]
Signed-off-by Steven Rostedt <[email protected]>--- kernel/trace/ring_buffer.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)kernel/trace/ring_buffer.c +0
-0
kernel/trace/ring_buffer.c void ring_buffer_discard_commit(struct ring_buffer *buffer, · line 1694
Quoted 8 lines
kernel/trace/ring_buffer.c +1
-1@@ -1688,7 +1688,7 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,* committed yet. Thus we can assume that preemption * is still disabled. */ -RB_WARN_ON (buffer, !preempt_count ()); +RB_WARN_ON (buffer,preemptible ()); cpu =smp_processor_id (); cpu_buffer = buffer->buffers[cpu];
smp_processor_id() will warn too.
Steven Rostedt
maintainerRe: [PATCH 4/4] ring-buffer: change WARN_ON from checking preempt_count to preemptible
Quoted 26 lines · Andrew Morton
Quoted 24 lines · On Fri, 08 May 2009 00:32:54 -0400 Steven Rostedt <[email protected]> wrote:
From: Steven Rostedt <[email protected]>
There's a
WARN_ONin the ring buffer code that makes sure preemption is disabled. It checks "!preempt_count()". But whenCONFIG_PREEMPTis not enabled,preempt_count()is always zero, and this will trigger the warning.[ Impact: prevent false warning on non preemptible kernels ]
Signed-off-by Steven Rostedt <[email protected]>--- kernel/trace/ring_buffer.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)kernel/trace/ring_buffer.c +1
-1@@ -1688,7 +1688,7 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,* committed yet. Thus we can assume that preemption * is still disabled. */ -RB_WARN_ON (buffer, !preempt_count ()); +RB_WARN_ON (buffer,preemptible ());
Quoted 5 lines
kernel/trace/ring_buffer.c void ring_buffer_discard_commit(struct ring_buffer *buffer, · line 1694
Quoted 2 lines
kernel/trace/ring_buffer.c +0
-0cpu =smp_processor_id (); cpu_buffer = buffer->buffers[cpu];
smp_processor_id()will warn too.
The difference is that RB_WARN_ON also disables the ring buffer.
-- Steve
Re: [PATCH 4/4] ring-buffer: change WARN_ON from checking preempt_count to preemptible
Quoted 30 lines · Steven Rostedt
Quoted 26 lines · Andrew Morton
Quoted 24 lines · On Fri, 08 May 2009 00:32:54 -0400 Steven Rostedt <[email protected]> wrote:
From: Steven Rostedt <[email protected]>
There's a
WARN_ONin the ring buffer code that makes sure preemption is disabled. It checks "!preempt_count()". But whenCONFIG_PREEMPTis not enabled,preempt_count()is always zero, and this will trigger the warning.[ Impact: prevent false warning on non preemptible kernels ]
Signed-off-by Steven Rostedt <[email protected]>--- kernel/trace/ring_buffer.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)kernel/trace/ring_buffer.c +1
-1@@ -1688,7 +1688,7 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,* committed yet. Thus we can assume that preemption * is still disabled. */ -RB_WARN_ON (buffer, !preempt_count ()); +RB_WARN_ON (buffer,preemptible ());
Quoted 7 lines
Quoted 5 lines
kernel/trace/ring_buffer.c void ring_buffer_discard_commit(struct ring_buffer *buffer, · line 1694
Quoted 2 lines
kernel/trace/ring_buffer.c +0
-0cpu =smp_processor_id (); cpu_buffer = buffer->buffers[cpu];
smp_processor_id()will warn too.The difference is that
RB_WARN_ONalso disables the ring buffer.
Yes, it's a more robust form of warning, in this context.
Ingo
Re: [PATCH 4/4] ring-buffer: change WARN_ON from checking preempt_count to preemptible
In reply to Steven RostedtQuoted 30 lines · On Fri, 8 May 2009 06:53:48 -0400 (EDT) Steven Rostedt <[email protected]>…
Quoted 26 lines · Andrew Morton
Quoted 24 lines · On Fri, 08 May 2009 00:32:54 -0400 Steven Rostedt <[email protected]> wrote:
From: Steven Rostedt <[email protected]>
There's a
WARN_ONin the ring buffer code that makes sure preemption is disabled. It checks "!preempt_count()". But whenCONFIG_PREEMPTis not enabled,preempt_count()is always zero, and this will trigger the warning.[ Impact: prevent false warning on non preemptible kernels ]
Signed-off-by Steven Rostedt <[email protected]>--- kernel/trace/ring_buffer.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)kernel/trace/ring_buffer.c +1
-1@@ -1688,7 +1688,7 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,* committed yet. Thus we can assume that preemption * is still disabled. */ -RB_WARN_ON (buffer, !preempt_count ()); +RB_WARN_ON (buffer,preemptible ());
Quoted 8 lines
Quoted 5 lines
kernel/trace/ring_buffer.c void ring_buffer_discard_commit(struct ring_buffer *buffer, · line 1694
Quoted 2 lines
kernel/trace/ring_buffer.c +0
-0cpu =smp_processor_id (); cpu_buffer = buffer->buffers[cpu];
smp_processor_id()will warn too.The difference is that
RB_WARN_ONalso disables the ring buffer.
Yes, but the kernel will produce two large warning spews for the same thing.
Steven Rostedt
maintainerRe: [PATCH 4/4] ring-buffer: change WARN_ON from checking preempt_count to preemptible
Quoted 19 lines · Andrew Morton
Quoted 18 lines · On Fri, 8 May 2009 06:53:48 -0400 (EDT) Steven Rostedt <[email protected]>…
Quoted 17 lines
Quoted 6 lines
kernel/trace/ring_buffer.c +0
-0kernel/trace/ring_buffer.c void ring_buffer_discard_commit(struct ring_buffer *buffer, · line 1694
Quoted 8 lines
kernel/trace/ring_buffer.c +1
-1@@ -1688,7 +1688,7 @@ void ring_buffer_discard_commit(struct ring_buffer *buffer,* committed yet. Thus we can assume that preemption * is still disabled. */ -RB_WARN_ON (buffer, !preempt_count ()); +RB_WARN_ON (buffer,preemptible ()); cpu =smp_processor_id (); cpu_buffer = buffer->buffers[cpu];
smp_processor_id()will warn too.
Quoted 5 lines
Quoted 2 lines
The difference is that
RB_WARN_ONalso disables the ring buffer.Yes, but the kernel will produce two large warning spews for the same thing.
Hmm, I should go back to my original patch. Which was:
RB_WARN_ON_PREEMPT(buffer, !preempt_count());
and have at the top of the file:
#ifdef CONFIG_PREEMPT
# RB_WARN_ON_PREEMPT(buffer, cond) RB_WARN_ON(buffer, cond) #else
# RB_WARN_ON_PREEMPT(buffer, cond) do { } while (0) #endif
The difference here is not that preemption must be off, but that this must be called from after a ring_buffer_lock_reserve() but not after a ring_buffer_unlock_commit().
We have two methods to discard a change, one is to discard it instead of commiting it. This will try to totally remove the change without leaving holes. The ring buffer is reentrant, and if an interrupt were to happen in between reserve and commit, it could store data after this commit. In which case we have no choice but to leave a hole. But if that did not happen, then we use cmpxchg to reset the tail of the buffer back to the original and not have to worry about NMIs and interrupts.
The other version of discarding can be done after a commit. But it only changes the type of the data and a hole exits. If we did this for all discards, the ring buffer would be wasted with discarded material and the filtering would be useless.
The original code only had the discard with a hole, and that was done after the commit. After converting it over, to the new method, several places still had the discard after the commit, and that broke the ring buffer.
There's no easy way to catch this, and do it fast. Since preemption is always disabled between reserve and commit, I was able to find places that broke this (and even new places that were added) by simply checking if preemption is disabled. Note, preemptible() is actually wrong because it also checks if interrupts are disabled, and if interrupts are disabled but preemption is not, we miss the case as well.
In any case, if this happens then the ring buffer will be corrupted, and it is best to disable it. Having two big warnings will help have these errors stick out more :-)
Actually, the way I've noticed it (when it happened on my laptop) was when I ran the trace and got no output. Then I decided to do a dmesg to see why. If I just saw garbage (or even worse, a crash) then I would not have looked right away at the dmesg. A crash would have prevented me from looking at dmesg if it locked up my laptop.
-- Steve
Quoted 14 lines · Steven Rostedt
Ingo,
Please pull the latest tip/tracing/ftrace tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/ftraceSteven Rostedt (4):
tracing: have menu default enabled when kernel debug is configured
ring-buffer: only periodically callcond_reschedto ring-buffer-benchmark
ring-buffer: add total count in ring-buffer-benchmark
ring-buffer: changeWARN_ONfrom checkingpreempt_countto preemptible
Quoted 6 lines
----
kernel/trace/Kconfig | 1 + kernel/trace/ring_buffer.c | 2 +- kernel/trace/ring_buffer_benchmark.c | 22 ++++++++++++++++++++-- 3 files changed, 22 insertions(+), 3 deletions(-)
Pulled, thanks Steve. (I suspect i'll get the fix for the things noticed by Andrew in the next pull request?)
Ingo