mcount and latency tracing utility -v5
Subsystems (11)
Steven Rostedt
reviewercover mcount and latency tracing utility -v5
[
version 5 of mcount / trace patches:
changes include:
Folded more patches together (mostly the timing stuff)
Some minor fixes to the timing code.
Redesigned the tracer!
- Now all tracers use the same buffer. This cut down tremendously the code duplication).
- consolidated the debugfs files (see below)
This has now gone from a RFC patch series to a true PATCH series.
New! trace-cmd.c - see below
added printk patch to allow printk to happen with runqueue lock held.
]
All released version of these patches can be found at:
http://people.redhat.com/srostedt/tracing/
The following patch series brings to vanilla Linux a bit of the RT kernel trace facility. This incorporates the "-pg" profiling option of gcc that will call the "mcount" function for all functions called in the kernel.
Note: I did investigate using -finstrument-functions but that adds a call to both start and end of a function. Using mcount only does the beginning of the function. mcount alone adds ~13% overhead. The -finstrument-functions added ~19%. Also it caused me to do tricks with inline, because it adds the function calls to inline functions as well.
This patch series implements the code for x86 (32 and 64 bit), but other archs can easily be implemented as well (note: ARM and PPC are already implemented in -rt)
Some Background:
----------------
A while back, Ingo Molnar and William Lee Irwin III created a latency tracer to find problem latency areas in the kernel for the RT patch. This tracer became a very integral part of the RT kernel in solving where latency hot spots were. One of the features that the latency tracer added was a function trace. This function tracer would record all functions that were called (implemented by the gcc "-pg" option) and would show what was called when interrupts or preemption was turned off.
This feature is also very helpful in normal debugging. So it's been talked about taking bits and pieces from the RT latency tracer and bring them to LKML. But no one had the time to do it.
Arnaldo Carvalho de Melo took a crack at it. He pulled out the mcount as well as part of the tracing code and made it generic from the point of the tracing code. I'm not sure why this stopped. Probably because Arnaldo is a very busy man, and his efforts had to be utilized elsewhere.
While I still maintain my own Logdev utility:
http://rostedt.homelinux.com/logdev
I came across a need to do the mcount with logdev too. I was successful but found that it became very dependent on a lot of code. One thing that I liked about my logdev utility was that it was very non-intrusive, and has been easy to port from the Linux 2.0 days. I did not want to burden the logdev patch with the intrusiveness of mcount (not really that intrusive, it just needs to add a "notrace" annotation to functions in the kernel that will cause more conflicts in applying patches for me).
Being close to the holidays, I grabbed Arnaldos old patches and started massaging them into something that could be useful for logdev, and what I found out (and talking this over with Arnaldo too) that this can be much more useful for others as well.
The main thing I changed, was that I made the mcount function itself generic, and not the dependency on the tracing code. That is I added
register_mcount_function()
andclear_mcount_function()
So when ever mcount is enabled and a function is registered that function is called for all functions in the kernel that is not labeled with the "notrace" annotation.
The Simple Tracer:
------------------
To show the power of this I also massaged the tracer code that Arnaldo pulled from the RT patch and made it be a nice example of what can be done with this.
The function that is registered to mcount has the prototype:
void func(unsigned long ip, unsigned long parent_ip);
The ip is the address of the function and parent_ip is the address of the parent function that called it.
The x86_64 version has the assembly call the registered function directly to save having to do a double function call.
To enable mcount, a sysctl is added:
/proc/sys/kernel/mcount_enabled
Once mcount is enabled, when a function is registed, it will be called by all functions. The tracer in this patch series shows how this is done. It adds a directory in the debugfs, called mctracer. With a ctrl file that will allow the user have the tracer register its function. Note, the order of enabling mcount and registering a function is not important, but both must be done to initiate the tracing. That is, you can disable tracing by either disabling mcount or by clearing the registered function.
Only one function may be registered at a time. If another function is registered, it will simply override what ever was there previously.
Here's a simple example of the tracer output:
CPU 2: hackbench:11867 preempt_schedule+0xc/0x84 <-- avc_has_perm_noaudit+0x45d/0x52c CPU 1: hackbench:12052 selinux_file_permission+0x10/0x11c <-- security_file_permission+0x16/0x18 CPU 3: hackbench:12017 update_curr+0xe/0x8b <-- put_prev_task_fair+0x24/0x4c
CPU 2: hackbench:11867 avc_audit+0x16/0x9e3 <-- avc_has_perm+0x51/0x63
CPU 0: hackbench:12019 socket_has_perm+0x16/0x7c <-- selinux_socket_sendmsg+0x27/0x3e CPU 1: hackbench:12052 file_has_perm+0x16/0xbb <-- selinux_file_permission+0x104/0x11c
This is formated like:
CPU <CPU#>: <task-comm>:<task-pid> <function> <-- <parent-function>
Latency Tracer Format:
----------------------
The format used by the RT patch is a bit more complex. It is designed to record a lot of information quickly and dump out a lot too.
There's two versions of the format. Verbose and non-vebose.
verbose:
preemption latency trace v1.1.5 on 2.6.24-rc7-tst
--------------------------------------------------------------------
latency: 89 us, #3/3, CPU#1 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:4)
-----------------
| task: kjournald-600 (uid:0 nice:-5 policy:0 rt_prio:0)
----------------- => started at: _spin_lock_irqsave+0x2a/0x63 <c06310d2>
=> ended at: _spin_unlock_irqrestore+0x32/0x41 <c0631245>
kjournald 600 1 1 00000000 00000000 [397408f1] 0.003ms (+0.079ms): _spin_lock_irqsave+0x2a/0x63 <c06310d2> (scsi_dispatch_cmd+0x155/0x234 [scsi_mod] <f8867c19>)
kjournald 600 1 1 00000000 00000001 [39740940] 0.081ms (+0.005ms): _spin_unlock_irqrestore+0x32/0x41 <c0631245> (scsi_dispatch_cmd+0x1be/0x234 [scsi_mod] <f8867c82>)
kjournald 600 1 1 00000000 00000002 [39740945] 0.087ms (+0.000ms): trace_hardirqs_on_caller+0x74/0x86 <c0508bdc> (_spin_unlock_irqrestore+0x32/0x41 <c0631245>)non-verbose:
preemption latency trace v1.1.5 on 2.6.24-rc7-tst
--------------------------------------------------------------------
latency: 89 us, #3/3, CPU#2 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:4)
-----------------
| task: kjournald-600 (uid:0 nice:-5 policy:0 rt_prio:0)
----------------- => started at: _spin_lock_irqsave+0x2a/0x63 <c06310d2>
=> ended at: _spin_unlock_irqrestore+0x32/0x41 <c0631245>
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
kjournal-600 1d... 3us+: _spin_lock_irqsave+0x2a/0x63 <c06310d2> (scsi_dispatch_cmd+0x155/0x234 [scsi_mod] <f8867c19>)
kjournal-600 1d... 81us+: _spin_unlock_irqrestore+0x32/0x41 <c0631245> (scsi_dispatch_cmd+0x1be/0x234 [scsi_mod] <f8867c82>)
kjournal-600 1d... 87us : trace_hardirqs_on_caller+0x74/0x86 <c0508bdc> (_spin_unlock_irqrestore+0x32/0x41 <c0631245>)Debug FS:
---------
Although enabling and disabling mcount is done through the sysctl:
/proc/sys/kernel/mcount_enabled
The rest of the tracing uses debugfs.
/debugfs/tracing
Here's the available files:
available_tracers
Lists the tracers that are compiled into the kernel and can be echoed into current_tracer
current_tracer
Shows the current tracer that is registered. On bootup this is blank. To set a new tracer, simply echo into this file the name found in available_tracers.
e.g.
echo wakeup > /debugfs/tracing/current_tracer
trace_ctrl
echo 1 to enable the current tracer.
echo 0 to disable it.
Note, the function trace also needs mcount_enabled set, otherwise it will not record.
latency_trace
Outputs the current trace in latency_trace format.
tracing_thresh
echo a number (in usecs) into this to record all traces that are greater than threshold. Only matters for those traces that use a max threshold (preemptoff irqsoff and wakeup)
iter_ctrl
echo "symonly" to not show the instruction pointers in the trace echo "nosymonly" to disable symonly.
echo "verbose" for verbose output from latency format. echo "noverbose" to disable verbose ouput.
cat iter_ctrl to see the current settings.
tracing_max_latency
Holds the current max critical latency. echo 0 to reset and start tracing.
Only holds for those tracers that use a max setting. (preemptoff irqsoff and wakeup)
trace
simple output format of the current trace.
Trace tool:
-----------
The trace_cmd.c found at
http://people.redhat.com/srostedt/tracing/trace-cmd.c
This tool will set up the tracer and allow you to run a program and trace it. (must be root)
e.g.
# ./trace-cmd -f echo hi
# cat /debug/tracing/latency_trace | grep echo | head
echo-4240 1d..1 1076us : ret_from_fork+0x6/0x1c (schedule_tail+0x9/0x5e)
echo-4240 1d..1 1077us : schedule_tail+0x1e/0x5e (finish_task_switch+0xb/0x61)
echo-4240 1d..1 1078us : finish_task_switch+0x24/0x61 (_spin_unlock_irq+0x8/0x40)
echo-4240 1...1 1079us : _spin_unlock_irq+0x27/0x40 (sub_preempt_count+0xd/0x104)
echo-4240 1...1 1080us+: sub_preempt_count+0x8d/0x104 (in_lock_functions+0x8/0x2c)
echo-4240 1d... 1082us : error_code+0x72/0x78 (do_page_fault+0xe/0x7c0)
echo-4240 1d... 1083us : do_page_fault+0x35b/0x7c0 (add_preempt_count+0xc/0x10f)
echo-4240 1d..1 1085us : add_preempt_count+0x53/0x10f (in_lock_functions+0x8/0x2c)
echo-4240 1d..1 1086us : do_page_fault+0x38f/0x7c0 (sub_preempt_count+0xd/0x104)
echo-4240 1d..1 1087us : sub_preempt_count+0x8d/0x104 (in_lock_functions+0x8/0x2c)The available switches are:
-s - context switch tracing
-p - preemption off tracing
-i - interrupts off tracing
-b - both interrupts off and preemption off tracing -w - wakeup timings
-f - function trace
Only -f (function trace) may be enabled with the other tracers, since most the other tracers also have a function trace version if mcount is enabled (yes -f will enable mcount).
Overhead:
---------
Note that having mcount compiled in seems to show a little overhead.
Here's 3 runs of hackbench 50 without the patches:
Time: 2.137
Time: 2.283
Time: 2.245
Avg: 2.221
and here's 3 runs with the patches (without tracing on):
Time: 2.738
Time: 2.469
Time: 2.388
Avg: 2.531
So it is a 13% overhead when enabled (according to hackbench).
But full tracing can cause a bit more problems:
# hackbench 50 Time: 113.350
113.350!!!!!
But this is tracing *every* function call!
Future:
-------
The way the mcount hook is done here, other utilities can easily add their own functions. Just care needs to be made not to call anything that is not marked with notrace, or you will crash the box with recursion. But even the simple tracer adds a "disabled" feature so in case it happens to call something that is not marked with notrace, it is a safety net not to kill the box.
I was originally going to use the relay system to record the data, but that had a chance of calling functions not marked with notrace. But, if for example LTTng wanted to use this, it could disable tracing on a CPU when doing the calls, and this will protect from recusion.
Redesign:
---------
This will be the last series that has each of the traces as a separate buffer. The next series will be more like the RT patch implementation of holding a single buffer for all latency traces. The only disadvantage of this is that you can only perform one type of latency at a time. But this has never been brought up as an issue with the RT patch.
The user will still be able to switch at runtime which type of latency they would like to record.
SystemTap:
----------
One thing that Arnaldo and I discussed last year was using systemtap to add hooks into the kernel to start and stop tracing. kprobes is too heavy to do on all funtion calls, but it would be perfect to add to non hot paths to start the tracer and stop the tracer.
So when debugging the kernel, instead of recompiling with printks or other markers, you could simply use systemtap to place a trace start and stop locations and trace the problem areas to see what is happening.
These are just some of the ideas we have with this. And we are sure others could come up with more.
These patches are for the underlining work. We'll see what happens next.
Steven Rostedt
reviewerPATCH 1/20 printk - dont wakeup klogd with interrupts disabled
[ This patch is added to the series since the wakeup timings trace
may lockup without it. ]
I thought that one could place a printk anywhere without worrying. But it seems that it is not wise to place a printk where the runqueue lock is held.
I just spent two hours debugging why some of my code was locking up, to find that the lockup was caused by some debugging printk's that I had in the scheduler. The printk's were only in rare paths so they shouldn't be too much of a problem, but after I hit the printk the system locked up.
Thinking that it was locking up on my code I went looking down the wrong path. I finally found (after examining an NMI dump) that the lockup happened because printk was trying to wakeup the klogd daemon, which caused a deadlock when the try_to_wakeup code tries to grab the runqueue lock.
Since printks are seldom called with interrupts disabled, we can hold off the waking of klogd if they are. We don't have access to the runqueue locks from printk, but those locks need interrupts disabled in order to be held.
Calling printk with interrupts disabled should only be done for emergencies and debugging anyway.
And with this patch, my code ran fine ;-)
--- kernel/printk.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
kernel/printk.c +12 -4
@@ -595,9 +595,11 @@ static int have_callable_console(void)* @fmt: format string * * This is printk(). It can be called from any context. We want it to work. -* Be aware of the fact that if oops_in_progress is not set, we might try to -* wake klogd up which could deadlock on runqueue lock if printk() is called -* from scheduler code. +* +* Note: if printk() is called with interrupts disabled, it will not wake +* up the klogd. This is to avoid a deadlock from calling printk() in schedule +* with the runqueue lock held and having the wake_up grab the runqueue lock +* as well. * * We try to grab the console_sem. If we succeed, it's easy - we log the output and * call the console drivers. If we fail to get the semaphore we place the output
@@ -978,7 +980,13 @@ void release_console_sem(void) console_locked =0 ;up (&console_sem);spin_unlock_irqrestore (&logbuf_lock, flags); -if (wake_klogd) +/* +* If we try to wake up klogd while printing with the runqueue lock +* held, this will deadlock. We don't have access to the runqueue +* lock from here, but just checking for interrupts disabled +* should be enough. +*/ +if (!irqs_disabled () && wake_klogd)wake_up_klogd (); }EXPORT_SYMBOL (release_console_sem);
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
kernel/printk.c void release_console_sem(void) · line 990
Quoted 2 lines · Steven Rostedt
kernel/printk.c +1
-0+if (!irqs_disabled () && wake_klogd)wake_up_klogd ();
This causes a regression .. When printk is called during an OOPS in kernels without this change then the OOPS will get logged, since the logging process (klogd) is woken to handle the messages.. If you apply this change klogd doesn't wakeup, and hence doesn't log the oops.. So if you remove the wakeup here you have to add it someplace else to maintain the logging ..
(I'm not theorizing here, I have defects logged against this specific piece of code..)
Daniel
Steven Rostedt
reviewerRe: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
Quoted 13 lines · Daniel Walker
kernel/printk.c void release_console_sem(void) · line 990
Quoted 2 lines · Steven Rostedt
kernel/printk.c +1
-0+if (!irqs_disabled () && wake_klogd)wake_up_klogd ();This causes a regression .. When printk is called during an OOPS in kernels without this change then the OOPS will get logged, since the logging process (klogd) is woken to handle the messages.. If you apply this change klogd doesn't wakeup, and hence doesn't log the oops.. So if you remove the wakeup here you have to add it someplace else to maintain the logging ..
Quoted 2 lines
(I'm not theorizing here, I have defects logged against this specific piece of code..)
It wont get woken up anyway. Did you look at wake_up_klogd?
void wake_up_klogd(void)
{
if (!oops_in_progress && waitqueue_active(&log_wait))
wake_up_interruptible(&log_wait);
}So if oops_in_progress is set, then it still wont get woken. Perhaps it got woken some other way? Or is oops_in_progress not set in these oops?
One other solution is to make the runqueue locks visible externally. Like:
in sched.c:
int runqueue_is_locked(void)
{
int cpu = get_cpu();
struct rq *rq = cpu_rq(cpu);
int ret;
ret = spin_is_locked(&rq->lock);
put_cpu();
return ret;
}And in printk we could do:
if (wake_klogd && !runqueue_is_locked()) wake_up_klogd();
This probably is the cleanest solution since it simply prevents the deadlock from occurring.
-- Steve
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
Quoted 46 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 13 lines · Daniel Walker
kernel/printk.c void release_console_sem(void) · line 990
Quoted 2 lines · Steven Rostedt
kernel/printk.c +1
-0+if (!irqs_disabled () && wake_klogd)wake_up_klogd ();This causes a regression .. When printk is called during an OOPS in kernels without this change then the OOPS will get logged, since the logging process (klogd) is woken to handle the messages.. If you apply this change klogd doesn't wakeup, and hence doesn't log the oops.. So if you remove the wakeup here you have to add it someplace else to maintain the logging ..
Quoted 2 lines
(I'm not theorizing here, I have defects logged against this specific piece of code..)
It wont get woken up anyway. Did you look at
wake_up_klogd?void
wake_up_klogd(void){ if (!oops_in_progress && waitqueue_active(&log_wait)) wake_up_interruptible(&log_wait); }So if
oops_in_progressis set, then it still wont get woken. Perhaps it got woken some other way? Or isoops_in_progressnot set in these oops?One other solution is to make the runqueue locks visible externally. Like:
in
sched.c:int
runqueue_is_locked(void){ int cpu = get_cpu(); struct rq *rq = cpu_rq(cpu); int ret; ret = spin_is_locked(&rq->lock); put_cpu(); return ret; }
Quoted 8 lines
And in printk we could do:
if (wake_klogd && !runqueue_is_locked()) wake_up_klogd();This probably is the cleanest solution since it simply prevents the deadlock from occurring.
FYI :
kernel/panic.c
NORET_TYPE void panic(const char * fmt, ...)
{
....bust_spinlocks(1);
...bust_spinlocks(0);
...
In bust_spinlocks :
void __attribute__((weak)) bust_spinlocks(int yes)
{
if (yes) {
++oops_in_progress;
} else {
#ifdef CONFIG_VT
unblank_screen();
#endif
if (--oops_in_progress == 0)
wake_up_klogd();
}
}Where the final wake_up happens while oops_in_progress is 0, but interrupts are still disabled.
So about my previous email, proposing testing oops_in_progress, it just won't work.
Mathieu
Quoted 2 lines
-- Steve
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
In reply to Steven RostedtQuoted 6 lines
if (wake_klogd && !runqueue_is_locked()) wake_up_klogd();This probably is the cleanest solution since it simply prevents the deadlock from occurring.
Do you really need to call it with the runqueue lock held .. There are other issue with the calls at that level.. For instance in -rt these call can actually hang the system in the console laying from inside printk (something I reported more than 6 months ago) .. It would be better to move them around the scheduler rather than inside it..
Daniel
Steven Rostedt
reviewerRe: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
Quoted 4 lines · Daniel Walker
Quoted 4 lines
if (wake_klogd && !runqueue_is_locked()) wake_up_klogd();
Quoted 8 lines
Quoted 2 lines
This probably is the cleanest solution since it simply prevents the deadlock from occurring.
Do you really need to call it with the runqueue lock held .. There are other issue with the calls at that level.. For instance in -rt these call can actually hang the system in the console laying from inside printk (something I reported more than 6 months ago) .. It would be better to move them around the scheduler rather than inside it..
The wakeup hook in schedule is when we find out that we hit our max. I could postpone that somehow, but that would require more glue code than I would like to add.
As for -rt, we've disabled printk for consoles not ATOMIC safe. So the only consoles that will print in atomic sections so far are, earlyprintk and the VGA.
Tim, could you send me your "postponed print" patches. That sounds like something the -rt patch could use.
Thanks,
-- Steve
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
Quoted 3 lines · Steven Rostedt
The wakeup hook in schedule is when we find out that we hit our max. I could postpone that somehow, but that would require more glue code than I would like to add.
Which hook specifically ?
Quoted 3 lines
As for -rt, we've disabled printk for consoles not ATOMIC safe. So the only consoles that will print in atomic sections so far are, earlyprintk and the VGA.
Is this code you added recently for tracing?
Daniel
Steven Rostedt
reviewerRe: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
Quoted 8 lines · Daniel Walker
Quoted 3 lines · Steven Rostedt
The wakeup hook in schedule is when we find out that we hit our max. I could postpone that somehow, but that would require more glue code than I would like to add.
Which hook specifically ?
The wakeup tracer does a print on max when the new task is about to be switched to (in context_switch).
Quoted 6 lines
Quoted 4 lines
As for -rt, we've disabled printk for consoles not ATOMIC safe. So the only consoles that will print in atomic sections so far are, earlyprintk and the VGA.
Is this code you added recently for tracing?
Nope, it's been there for a few releases and had nothing to do with the latency tracer.
<looks at code>
Hmm, seems that only the "hack" fix is in upstream. The Red Hat version has the better fix, but mainline didn't get it. OK, I'll add that to the next -rt release.
-- Steve
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
Quoted 5 lines · Steven Rostedt
Quoted 3 lines · Daniel Walker
On Wed, 2008-01-23 at 13:18 -0500, Steven Rostedt wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
The wakeup hook in schedule is when we find out that we hit our max. I could postpone that somehow, but that would require more glue code than I would like to add.
Which hook specifically ?
The wakeup tracer does a print on max when the new task is about to be switched to (in
context_switch).
You can't move that hook to when the task actually starts running (i.e. at the end of schedule() )..
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
As for -rt, we've disabled printk for consoles not ATOMIC safe. So the only consoles that will print in atomic sections so far are, earlyprintk and the VGA.
Is this code you added recently for tracing?
Quoted 8 lines
Nope, it's been there for a few releases and had nothing to do with the latency tracer.
<looks at code>
Hmm, seems that only the "hack" fix is in upstream. The Red Hat version has the better fix, but mainline didn't get it. OK, I'll add that to the next -rt release.
mainline ? The problem I'm thinking of was directly tracing related..
Daniel
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
Quoted 10 lines · Daniel Walker
Quoted 9 lines · Steven Rostedt
Quoted 7 lines · Daniel Walker
Quoted 3 lines · Steven Rostedt
The wakeup hook in schedule is when we find out that we hit our max. I could postpone that somehow, but that would require more glue code than I would like to add.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Which hook specifically ?
The wakeup tracer does a print on max when the new task is about to be switched to (in
context_switch).You can't move that hook to when the task actually starts running (i.e. at the end of
schedule())..
I mean , why can't you?
Daniel
PATCH 1/1 defer printks in irqs
In reply to Steven RostedtQuoted 2 lines · Steven Rostedt
Tim, could you send me your "postponed print" patches. That sounds like something the -rt patch could use.
Sure. See below.
This patch changes the way the printk code processes output to the console buffer. On some platforms, especially those with a serial console, the way printk output the messages to console caused significant interrupt-off periods. This code introduces two config options to improve that handling.
Without this change, test code run on a 178 MHZ ARM platform showed an interrupt off period for a particularly long printk message of 24 milliseconds. With this patch applied, and the options set to 'Y', the same message only had an interrupt-off period of 482 microseconds.
Portions of this (some parts that reorder locking and irq management primitives) were taken from the RT-preempt patch.
The patch does several things:
1. It moves the location of the irqrestore operations in the printk code, so that the console writing can occur with interrupts enabled
2. It changes the code so that characters are written to the console one-at-a-time. The serial console code holds interrupts disabled for the duration of its output to the serial port. To decrease the interrupt-off time, only a small amount of data is written at a time. (This could be tuned by writing chunks larger than 1 character at a time, but the current solution was easiest, and gives the lowest interrupt-off times.)
3. The printk code was changed so that if it is entered with interrupts disabled, it does not output to console immediately. Rather, a workqueue routine is scheduled to process the console output later (when interrupts are enabled).
4. An init-time routine was created, to prevent trying to schedule the workqueue routine too early in the system boot sequence.
Here's the patch:
---
kernel/printk.c | 92 85 + 7 - 0 ! lib/Kconfig.debug | 42 42 + 0 - 0 ! 2 files changed, 127 insertions(+), 7 deletions(-)
kernel/printk.c +85 -7
@@ -33,11 +33,14 @@#include <linux/bootmem.h> #include <linux/syscalls.h> #include <linux/jiffies.h> +#include <linux/workqueue.h> #include <asm/uaccess.h> #define __LOG_BUF_LEN (1 << CONFIG_LOG_BUF_SHIFT) +#define PRINTK_BUF_SIZE 1024 +/* printk's without a loglevel use this.. */ #define DEFAULT_MESSAGE_LOGLEVEL 4/* KERN_WARNING */
@@ -47,6 +50,10 @@DECLARE_WAIT_QUEUE_HEAD (log_wait); +static void printk_console_flush (struct work_struct *ignored); +static int have_callable_console (void ); +static DECLARE_WORK(printk_console_work, printk_console_flush); +int console_printk[4 ] = {DEFAULT_CONSOLE_LOGLEVEL ,/* console_loglevel */ DEFAULT_MESSAGE_LOGLEVEL ,/* default_message_loglevel */
@@ -220,6 +227,43 @@ static inline void boot_delay_msec(void)#endif /* +* We can't schedule printk console flushing during early boot. +* This flag holds it off until after we've initialized some +* important stuff. +*/ +static int pcf_ok =0 ; + +static int __init init_printk_console_flush (void ) +{ + pcf_ok =1 ; +return 0 ; +} +__initcall (init_printk_console_flush); + +void printk_console_flush (struct work_struct *ignored) +{ +unsigned long flags; +int chars_to_flush; + +spin_lock_irqsave (&logbuf_lock, flags); + chars_to_flush = log_end - con_start; +spin_unlock_irqrestore (&logbuf_lock, flags); + +if (chars_to_flush) { +acquire_console_sem (); + +if ((cpu_online (smp_processor_id ()) || +have_callable_console ())) { + console_may_schedule =0 ; +release_console_sem (); + }else { + console_locked =0 ; +up (&console_sem); + } + } +} + +/* * Return the number of unread characters in the log buffer. */ int log_buf_get_len (void )
@@ -436,8 +480,16 @@ static void __call_console_drivers(unsigfor (con = console_drivers; con; con = con->next) {if ((con->flags &CON_ENABLED ) && con->write && (cpu_online (smp_processor_id ()) || - (con->flags &CON_ANYTIME ))) + (con->flags &CON_ANYTIME ))) { +#ifdef CONFIG_PRINTK_SHORT_IRQ_DISABLE +int i; +for (i=0 ; i<end-start; i++) { + con->write (con, &LOG_BUF (start+i),1 ); + } +#else con->write (con, &LOG_BUF (start), end - start); +#endif + } } }
@@ -633,8 +685,9 @@ asmlinkage int vprintk(const char *fmt,unsigned long flags;int printed_len;char *p; -static char printk_buf[1024 ]; +static char printk_buf[PRINTK_BUF_SIZE ];static int log_level_unknown =1 ; +int defer;boot_delay_msec ();
@@ -649,6 +702,7 @@ asmlinkage int vprintk(const char *fmt,lockdep_off ();spin_lock (&logbuf_lock); printk_cpu =smp_processor_id (); +preempt_enable ();/* Emit the output into the temporary buffer */ printed_len =vscnprintf (printk_buf,sizeof (printk_buf), fmt, args);
@@ -718,6 +772,19 @@ asmlinkage int vprintk(const char *fmt, console_locked =1 ; printk_cpu =UINT_MAX ;spin_unlock (&logbuf_lock); +lockdep_on (); +local_irq_restore (flags); + + defer =0 ; +#ifdef CONFIG_PRINTK_DEFER_IN_IRQ +/* +* If we're in an interrupt and it's not an emergency, +* don't emit to console just yet. +*/ +if ((irqs_disabled () ||in_interrupt ()) && !oops_in_progress) { + defer =1 ; + } +#endif /* * Console drivers may assume that per-cpu resources have
@@ -725,7 +792,8 @@ asmlinkage int vprintk(const char *fmt,* being able to cope (CON_ANYTIME) don't call them until * this CPU is officially up. */ -if (cpu_online (smp_processor_id ()) ||have_callable_console ()) { +if (!defer && (cpu_online (smp_processor_id ()) || +have_callable_console ())) { console_may_schedule =0 ;release_console_sem (); }else {
@@ -733,8 +801,6 @@ asmlinkage int vprintk(const char *fmt, console_locked =0 ;up (&console_sem); } -lockdep_on (); -raw_local_irq_restore (flags); }else {/* * Someone else owns the drivers. We drop the spinlock, which
@@ -747,7 +813,11 @@ asmlinkage int vprintk(const char *fmt,raw_local_irq_restore (flags); } -preempt_enable (); +/* didn't flush console all the way yet? - schedule for later */ +if ((log_end - con_start) && pcf_ok) { +schedule_work (&printk_console_work); + } +return printed_len; }EXPORT_SYMBOL (printk);
@@ -971,13 +1041,21 @@ void release_console_sem(void) _con_start = con_start; _log_end = log_end; con_start = log_end;/* Flush */ + +#ifdef CONFIG_PRINTK_SHORT_IRQ_DISABLE +/* allow irqs during console out */ +spin_unlock_irqrestore (&logbuf_lock, flags); +call_console_drivers (_con_start, _log_end); +#else spin_unlock (&logbuf_lock);call_console_drivers (_con_start, _log_end);local_irq_restore (flags); +#endif } console_locked =0 ; -up (&console_sem);spin_unlock_irqrestore (&logbuf_lock, flags); +up (&console_sem); +if (wake_klogd)wake_up_klogd (); }
lib/Kconfig.debug +42 -0
48 lines not on this page · raw message
Re: [PATCH] defer printks in irqs
Quoted 38 lines · Tim Bird
Quoted 2 lines · Steven Rostedt
Tim, could you send me your "postponed print" patches. That sounds like something the -rt patch could use.
Sure. See below.
This patch changes the way the printk code processes output to the console buffer. On some platforms, especially those with a serial console, the way printk output the messages to console caused significant interrupt-off periods. This code introduces two config options to improve that handling.
Without this change, test code run on a 178 MHZ ARM platform showed an interrupt off period for a particularly long printk message of 24 milliseconds. With this patch applied, and the options set to 'Y', the same message only had an interrupt-off period of 482 microseconds.
Portions of this (some parts that reorder locking and irq management primitives) were taken from the RT-preempt patch.
The patch does several things:
1. It moves the location of the irqrestore operations in the printk code, so that the console writing can occur with interrupts enabled
2. It changes the code so that characters are written to the console one-at-a-time. The serial console code holds interrupts disabled for the duration of its output to the serial port. To decrease the interrupt-off time, only a small amount of data is written at a time. (This could be tuned by writing chunks larger than 1 character at a time, but the current solution was easiest, and gives the lowest interrupt-off times.)
3. The printk code was changed so that if it is entered with interrupts disabled, it does not output to console immediately. Rather, a workqueue routine is scheduled to process the console output later (when interrupts are enabled).
Quoted 4 lines
4. An init-time routine was created, to prevent trying to schedule the workqueue routine too early in the system boot sequence.
Here's the patch:
I suspect these features reduce the change a crash messages makes it out onto the console, but fail to spot any of the copious text mention this critical issue.
Quoted 247 lines
Signed-off-by Tim Bird [email protected]---
kernel/printk.c | 92 85 + 7 - 0 ! lib/Kconfig.debug | 42 42 + 0 - 0 ! 2 files changed, 127 insertions(+), 7 deletions(-)Index:
linux-2.6.24-rc8/kernel/printk.c
===================================================================kernel/printk.c +85
-7@@ -33,11 +33,14 @@#include <linux/bootmem.h> #include <linux/syscalls.h> #include <linux/jiffies.h> +#include <linux/workqueue.h> #include <asm/uaccess.h> #define __LOG_BUF_LEN (1 << CONFIG_LOG_BUF_SHIFT) +#define PRINTK_BUF_SIZE 1024 +/* printk's without a loglevel use this.. */ #define DEFAULT_MESSAGE_LOGLEVEL 4/* KERN_WARNING */ @@ -47,6 +50,10 @@DECLARE_WAIT_QUEUE_HEAD (log_wait); +static void printk_console_flush (struct work_struct *ignored); +static int have_callable_console (void ); +static DECLARE_WORK(printk_console_work, printk_console_flush); +int console_printk[4 ] = {DEFAULT_CONSOLE_LOGLEVEL ,/* console_loglevel */ DEFAULT_MESSAGE_LOGLEVEL ,/* default_message_loglevel */ @@ -220,6 +227,43 @@ static inline void boot_delay_msec(void)#endif /* +* We can't schedule printk console flushing during early boot. +* This flag holds it off until after we've initialized some +* important stuff. +*/ +static int pcf_ok =0 ; + +static int __init init_printk_console_flush (void ) +{ + pcf_ok =1 ; +return 0 ; +} +__initcall (init_printk_console_flush); + +void printk_console_flush (struct work_struct *ignored) +{ +unsigned long flags; +int chars_to_flush; + +spin_lock_irqsave (&logbuf_lock, flags); + chars_to_flush = log_end - con_start; +spin_unlock_irqrestore (&logbuf_lock, flags); + +if (chars_to_flush) { +acquire_console_sem (); + +if ((cpu_online (smp_processor_id ()) || +have_callable_console ())) { + console_may_schedule =0 ; +release_console_sem (); + }else { + console_locked =0 ; +up (&console_sem); + } + } +} + +/* * Return the number of unread characters in the log buffer. */ int log_buf_get_len (void )@@ -436,8 +480,16 @@ static void __call_console_drivers(unsigfor (con = console_drivers; con; con = con->next) {if ((con->flags &CON_ENABLED ) && con->write && (cpu_online (smp_processor_id ()) || - (con->flags &CON_ANYTIME ))) + (con->flags &CON_ANYTIME ))) { +#ifdef CONFIG_PRINTK_SHORT_IRQ_DISABLE +int i; +for (i=0 ; i<end-start; i++) { + con->write (con, &LOG_BUF (start+i),1 ); + } +#else con->write (con, &LOG_BUF (start), end - start); +#endif + } } }@@ -633,8 +685,9 @@ asmlinkage int vprintk(const char *fmt,unsigned long flags;int printed_len;char *p; -static char printk_buf[1024 ]; +static char printk_buf[PRINTK_BUF_SIZE ];static int log_level_unknown =1 ; +int defer;boot_delay_msec ();@@ -649,6 +702,7 @@ asmlinkage int vprintk(const char *fmt,lockdep_off ();spin_lock (&logbuf_lock); printk_cpu =smp_processor_id (); +preempt_enable ();/* Emit the output into the temporary buffer */ printed_len =vscnprintf (printk_buf,sizeof (printk_buf), fmt, args);@@ -718,6 +772,19 @@ asmlinkage int vprintk(const char *fmt, console_locked =1 ; printk_cpu =UINT_MAX ;spin_unlock (&logbuf_lock); +lockdep_on (); +local_irq_restore (flags); + + defer =0 ; +#ifdef CONFIG_PRINTK_DEFER_IN_IRQ +/* +* If we're in an interrupt and it's not an emergency, +* don't emit to console just yet. +*/ +if ((irqs_disabled () ||in_interrupt ()) && !oops_in_progress) { + defer =1 ; + } +#endif /* * Console drivers may assume that per-cpu resources have @@ -725,7 +792,8 @@ asmlinkage int vprintk(const char *fmt,* being able to cope (CON_ANYTIME) don't call them until * this CPU is officially up. */ -if (cpu_online (smp_processor_id ()) ||have_callable_console ()) { +if (!defer && (cpu_online (smp_processor_id ()) || +have_callable_console ())) { console_may_schedule =0 ;release_console_sem (); }else {@@ -733,8 +801,6 @@ asmlinkage int vprintk(const char *fmt, console_locked =0 ;up (&console_sem); } -lockdep_on (); -raw_local_irq_restore (flags); }else {/* * Someone else owns the drivers. We drop the spinlock, which @@ -747,7 +813,11 @@ asmlinkage int vprintk(const char *fmt,raw_local_irq_restore (flags); } -preempt_enable (); +/* didn't flush console all the way yet? - schedule for later */ +if ((log_end - con_start) && pcf_ok) { +schedule_work (&printk_console_work); + } +return printed_len; }EXPORT_SYMBOL (printk);@@ -971,13 +1041,21 @@ void release_console_sem(void) _con_start = con_start; _log_end = log_end; con_start = log_end;/* Flush */ + +#ifdef CONFIG_PRINTK_SHORT_IRQ_DISABLE +/* allow irqs during console out */ +spin_unlock_irqrestore (&logbuf_lock, flags); +call_console_drivers (_con_start, _log_end); +#else spin_unlock (&logbuf_lock);call_console_drivers (_con_start, _log_end);local_irq_restore (flags); +#endif } console_locked =0 ; -up (&console_sem);spin_unlock_irqrestore (&logbuf_lock, flags); +up (&console_sem); +if (wake_klogd)wake_up_klogd (); }Index: linux-2.6.24-rc8/lib/Kconfig.debug
===================================================================lib/Kconfig.debug +42
-0@@ -17,6 +17,48 @@ config ENABLE_WARN_DEPRECATED Disable this to suppress the "warning: 'foo' is deprecated (declared at kernel/power/somefile.c:1234)" messages. +config PRINTK_SHORT_IRQ_DISABLE + bool "Enable interrupts for shorter times during printk output" + default n + depends on PRINTK + help + Selecting this option will cause the kernel to write the + contents of the printk buffer to the console with interrupts + enabled, and in small increments. + + With the default setting of 'n', the kernel attempts to + send printk messages to the console with interrupts disabled, + and it sends the entire contents of the printk buffer in one go. + + For some consoles (notably the serial console, which is commonly + used in embedded products), this results in long interrupt-off + periods. + + Users interested in good real-time scheduling should set this + to 'y'. + +config PRINTK_DEFER_IN_IRQ + bool "Defer writing printk output to console during interrupts" + default n + depends on PRINTK && PRINTK_SHORT_IRQ_DISABLE + help + Selecting this option will cause the kernel to hold off writing + printk messages to the console until interrupts are enabled. + On some systems, printk buffer and console handling can take + a long time. If done with interrupts off for the entire duration + of the operation, this results in long interrupt-off periods. + + With the default setting of 'n', the kernel attempts to + send printk messages to the console whether it is in interrupt + context or not. + + With a setting of 'y', when printk is called with interrupts + disabled, the messages are buffered in the kernel's + log buffer until later. If you are debugging the kernel + with printks in interrupt routines, you should set this + to 'n'. Users interested in good real-time scheduling + should set this to 'y'. + config ENABLE_MUST_CHECK bool "Enable __must_check logic" default y
Re: [PATCH] defer printks in irqs
Quoted 3 lines · Peter Zijlstra
I suspect these features reduce the chance a crash messages makes it out onto the console, but fail to spot any of the copious text mention this critical issue.
Anything not in interrupts would not be deferred.
The following code is meant to avoid deferring in oops as well. I'm not sure if that covers all crash cases or not.
If you know of other cases, and ways to detect them, please advise.
Quoted 9 lines
kernel/printk.c asmlinkage int vprintk(const char *fmt, · line 787
Quoted 9 lines
kernel/printk.c +9
-0+#ifdef CONFIG_PRINTK_DEFER_IN_IRQ +/* +* If we're in an interrupt and it's not an emergency, +* don't emit to console just yet. +*/ +if ((irqs_disabled () ||in_interrupt ()) && !oops_in_progress) { + defer =1 ; + } +#endif
I made the code conditional because printk messages are pretty critical for debugging, and I wanted someone to be aware if they altered the behavior in any way. I suspect that only in embedded, when people are having realtime performance problems, would this change be of interest.
Please recognize that I'm not submitting this for kernel inclusion at this time. Please see the larger thread for context. That said, I appreciate the feedback on the code.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================
Andrew Morton
maintainerRe: [PATCH] defer printks in irqs
Quoted 11 lines · On Thu, 24 Jan 2008 15:23:03 -0800 Tim Bird <[email protected]> wrote:
Quoted 3 lines · Peter Zijlstra
I suspect these features reduce the chance a crash messages makes it out onto the console, but fail to spot any of the copious text mention this critical issue.
Anything not in interrupts would not be deferred.
The following code is meant to avoid deferring in oops as well. I'm not sure if that covers all crash cases or not.
If you know of other cases, and ways to detect them, please advise.
Yes, using oops_in_progress seems an appropriate fix for that.
I assume that another downside is that printk-from-interrupt will now get jumbled up with an interrupted pritk-from-non-interrupt.
Please use checkpatch.
Re: [PATCH] defer printks in irqs
Quoted 11 lines · Andrew Morton
Quoted 10 lines · On Thu, 24 Jan 2008 15:23:03 -0800 Tim Bird <[email protected]> wrote:
Quoted 3 lines · Peter Zijlstra
I suspect these features reduce the chance a crash messages makes it out onto the console, but fail to spot any of the copious text mention this critical issue.
Anything not in interrupts would not be deferred.
The following code is meant to avoid deferring in oops as well. I'm not sure if that covers all crash cases or not.
If you know of other cases, and ways to detect them, please advise.
Quoted 4 lines
Yes, using
oops_in_progressseems an appropriate fix for that.I assume that another downside is that printk-from-interrupt will now get jumbled up with an interrupted pritk-from-non-interrupt.
No. The order of printk messages (whether printk'ed from interrupt context or not) should not be affected. This patch should only affect the timing of when the messages are output to console.
Note that an oops will flush any buffered messages ahead of the oops messages themselves.
Some messages, not in oops, that were printk'ed in interrupt context, may still be in the log buffer but not have made it to console if the machine hangs hard. Because of this (IMO slight) downgrade in debugging utility, I chose to make the changes configurable.
Quoted 1 line
Please use checkpatch.
Will do from now on! Thanks!
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
In reply to Steven RostedtQuoted 46 lines · Steven Rostedt
Quoted 13 lines · Daniel Walker
kernel/printk.c void release_console_sem(void) · line 990
Quoted 2 lines · Steven Rostedt
kernel/printk.c +1
-0+if (!irqs_disabled () && wake_klogd)wake_up_klogd ();This causes a regression .. When printk is called during an OOPS in kernels without this change then the OOPS will get logged, since the logging process (klogd) is woken to handle the messages.. If you apply this change klogd doesn't wakeup, and hence doesn't log the oops.. So if you remove the wakeup here you have to add it someplace else to maintain the logging ..
Quoted 2 lines
(I'm not theorizing here, I have defects logged against this specific piece of code..)
It wont get woken up anyway. Did you look at
wake_up_klogd?void
wake_up_klogd(void){ if (!oops_in_progress && waitqueue_active(&log_wait)) wake_up_interruptible(&log_wait); }So if
oops_in_progressis set, then it still wont get woken. Perhaps it got woken some other way? Or isoops_in_progressnot set in these oops?One other solution is to make the runqueue locks visible externally. Like:
in
sched.c:int
runqueue_is_locked(void){ int cpu = get_cpu(); struct rq *rq = cpu_rq(cpu); int ret; ret = spin_is_locked(&rq->lock); put_cpu(); return ret; }
Quoted 7 lines
And in printk we could do:
if (wake_klogd && !runqueue_is_locked()) wake_up_klogd();This probably is the cleanest solution since it simply prevents the deadlock from occurring.
Yes please... this looks like a solution to me.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
In reply to Daniel WalkerQuoted 13 lines · * Daniel Walker ([email protected]) wrote:
kernel/printk.c void release_console_sem(void) · line 990
Quoted 2 lines · Steven Rostedt
kernel/printk.c +1
-0+if (!irqs_disabled () && wake_klogd)wake_up_klogd ();This causes a regression .. When printk is called during an OOPS in kernels without this change then the OOPS will get logged, since the logging process (klogd) is woken to handle the messages.. If you apply this change klogd doesn't wakeup, and hence doesn't log the oops.. So if you remove the wakeup here you have to add it someplace else to maintain the logging ..
Quoted 3 lines
(I'm not theorizing here, I have defects logged against this specific piece of code..)
Can we change this for :
if (!(irqs_disabled() && !oops_in_progress) && wake_klogd)
wake_up_klogd();
?
(totally untested)
Quoted 2 lines
Daniel
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
Quoted 3 lines · Mathieu Desnoyers
Can we change this for :
if (!(irqs_disabled() && !oops_in_progress) &&wake_klogd)
wake_up_klogd();
!oops_in_progress is already inside wake_up_klogd().
Also, I think you'd want that first '!' inside the parens.
I've been working on a system to handle deferred displaying of printk messages, when printk is called in interrupt context. Currently on a serial console if printk is called with ints disabled, it can hold interrupts off for up to 20 milliseconds.
My solution was to set up a worker thread to handle deferred printk activity. I don't think this works for the oops case, because you don't want to defer the output. However, I can post this code if you think it would help with this problem.
I hesitate to just drop the call to wake_up_klogd() without some kind of mechanism call it later. Waiting until the next printk will work to drain the buffer, but could lead to indefinite intervals with printks waiting to be logged.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================
Re: [PATCH 01/20 -v5] printk - dont wakeup klogd with interrupts disabled
In reply to Mathieu DesnoyersQuoted 18 lines · Mathieu Desnoyers
Quoted 13 lines · * Daniel Walker ([email protected]) wrote:
kernel/printk.c void release_console_sem(void) · line 990
Quoted 2 lines · Steven Rostedt
kernel/printk.c +1
-0+if (!irqs_disabled () && wake_klogd)wake_up_klogd ();This causes a regression .. When printk is called during an OOPS in kernels without this change then the OOPS will get logged, since the logging process (klogd) is woken to handle the messages.. If you apply this change klogd doesn't wakeup, and hence doesn't log the oops.. So if you remove the wakeup here you have to add it someplace else to maintain the logging ..
Quoted 3 lines
(I'm not theorizing here, I have defects logged against this specific piece of code..)
Quoted 5 lines
Can we change this for :
if (!(irqs_disabled() && !oops_in_progress) &&wake_klogd)
wake_up_klogd();?
That's wrong, too.
Just make wake_up_klogd do trylock, if it finds that it can't get neccessary locks, printk(KERN_ALERT) so at least console gets the message, but proceed without the lock.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Steven Rostedt
reviewerPATCH 2/20 Add basic support for gcc profiler instrumentation
In reply to Steven RostedtIf CONFIG_MCOUNT is selected and /proc/sys/kernel/mcount_enabled is set to a non-zero value the mcount routine will be called everytime we enter a kernel function that is not marked with the "notrace" attribute.
The mcount routine will then call a registered function if a function happens to be registered.
[This code has been highly hacked by Steven Rostedt, so don't
blame Arnaldo for all of this ;-) ]
Update:
It is now possible to register more than one mcount function. If only one mcount function is registered, that will be the function that mcount calls directly. If more than one function is registered, then mcount will call a function that will loop through the functions to call.
--- Makefile | 3 arch/x86/Kconfig | 1 arch/x86/kernel/entry_32.S | 25 +++++++ arch/x86/kernel/entry_64.S | 36 +++++++++++ include/linux/linkage.h | 2 include/linux/mcount.h | 38 ++++++++++++ kernel/sysctl.c | 11 +++ lib/Kconfig.debug | 2 lib/Makefile | 2 lib/tracing/Kconfig | 10 +++ lib/tracing/Makefile | 3 lib/tracing/mcount.c | 141 +++++++++++++++++++++++++++++++++++++++++++++ 12 files changed, 274 insertions(+)
Makefile +3 -0
@@ -509,6 +509,9 @@ endif include $(srctree)/arch/$(SRCARCH)/Makefile +ifdef CONFIG_MCOUNT +KBUILD_CFLAGS += -pg +endif ifdef CONFIG_FRAME_POINTER KBUILD_CFLAGS += -fno-omit-frame-pointer -fno-optimize-sibling-calls else
arch/x86/Kconfig +1 -0
@@ -19,6 +19,7 @@ config X86_64 config X86 bool default y + select HAVE_MCOUNT config GENERIC_TIME bool
arch/x86/kernel/entry_32.S +25 -0
@@ -75,6 +75,31 @@ DF_MASK = 0x00000400 NT_MASK = 0x00004000 VM_MASK = 0x00020000 +#ifdef CONFIG_MCOUNT +.globl mcount +mcount: + /* unlikely(mcount_enabled) */ + cmpl $0, mcount_enabled + jnz trace + ret + +trace: + /* taken from glibc */ + pushl %eax + pushl %ecx + pushl %edx + movl 0xc(%esp), %edx + movl 0x4(%ebp), %eax + + call *mcount_trace_function + + popl %edx + popl %ecx + popl %eax + + ret +#endif + #ifdef CONFIG_PREEMPT #define preempt_stop(clobbers) DISABLE_INTERRUPTS(clobbers); TRACE_IRQS_OFF #else
arch/x86/kernel/entry_64.S +36 -0
@@ -53,6 +53,42 @@ .code64 +#ifdef CONFIG_MCOUNT + +ENTRY(mcount) + /* unlikely(mcount_enabled) */ + cmpl $0, mcount_enabled + jnz trace + retq + +trace: + /* taken from glibc */ + subq $0x38, %rsp + movq %rax, (%rsp) + movq %rcx, 8(%rsp) + movq %rdx, 16(%rsp) + movq %rsi, 24(%rsp) + movq %rdi, 32(%rsp) + movq %r8, 40(%rsp) + movq %r9, 48(%rsp) + + movq 0x38(%rsp), %rsi + movq 8(%rbp), %rdi + + call *mcount_trace_function + + movq 48(%rsp), %r9 + movq 40(%rsp), %r8 + movq 32(%rsp), %rdi + movq 24(%rsp), %rsi + movq 16(%rsp), %rdx + movq 8(%rsp), %rcx + movq (%rsp), %rax + addq $0x38, %rsp + + retq +#endif + #ifndef CONFIG_PREEMPT #define retint_kernel retint_restore_args #endif
include/linux/linkage.h +2 -0
@@ -3,6 +3,8 @@#include <asm/linkage.h> +#define notrace __attribute__((no_instrument_function)) +#ifdef __cplusplus#define CPP_ASMLINKAGE extern "C"#else
include/linux/mcount.h new file +38 -0
@@ -0,0 +1,38 @@ +#ifndef _LINUX_MCOUNT_H +#define _LINUX_MCOUNT_H + +#ifdef CONFIG_MCOUNT +extern int mcount_enabled; + +#include <linux/linkage.h> + +#define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0)) +#define CALLER_ADDR1 ((unsigned long)__builtin_return_address(1)) +#define CALLER_ADDR2 ((unsigned long)__builtin_return_address(2)) + +typedef void (*mcount_func_t )(unsigned long ip,unsigned long parent_ip); + +struct mcount_ops { +mcount_func_t func; +struct mcount_ops *next; +}; + +/* +* The mcount_ops must be a static and should also +* be read_mostly. These functions do modify read_mostly variables +* so use them sparely. Never free an mcount_op or modify the +* next pointer after it has been registered. Even after unregistering +* it, the next pointer may still be used internally. +*/ +int register_mcount_function (struct mcount_ops *ops); +int unregister_mcount_function (struct mcount_ops *ops); +void clear_mcount_function (void ); + +extern void mcount (void ); + +#else /* !CONFIG_MCOUNT */ +# define register_mcount_function (ops) do { } while (0) +# define unregister_mcount_function (ops) do { } while (0) +# define clear_mcount_function (ops) do { } while (0) +#endif /* CONFIG_MCOUNT */ +#endif /* _LINUX_MCOUNT_H */
kernel/sysctl.c +11 -0
@@ -46,6 +46,7 @@#include <linux/nfs_fs.h> #include <linux/acpi.h> #include <linux/reboot.h> +#include <linux/mcount.h> #include <asm/uaccess.h> #include <asm/processor.h>
@@ -470,6 +471,16 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = &proc_dointvec, }, +#ifdef CONFIG_MCOUNT + { + .ctl_name = CTL_UNNUMBERED, + .procname = "mcount_enabled", + .data = &mcount_enabled, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, +#endif #ifdef CONFIG_KMOD { .ctl_name = KERN_MODPROBE,
lib/Kconfig.debug +2 -0
@@ -517,4 +517,6 @@ config FAULT_INJECTION_STACKTRACE_FILTER help Provide stacktrace filter for fault-injection capabilities +source lib/tracing/Kconfig + source "samples/Kconfig"
lib/Makefile +2 -0
@@ -66,6 +66,8 @@ obj-$(CONFIG_AUDIT_GENERIC) += audit.o obj-$(CONFIG_SWIOTLB) += swiotlb.o obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o +obj-$(CONFIG_MCOUNT) += tracing/ + lib-$(CONFIG_GENERIC_BUG) += bug.o hostprogs-y := gen_crc32table
lib/tracing/Kconfig new file +10 -0
@@ -0,0 +1,10 @@ + +# Archs that enable MCOUNT should select HAVE_MCOUNT +config HAVE_MCOUNT + bool + +# MCOUNT itself is useless, or will just be added overhead. +# It needs something to register a function with it. +config MCOUNT + bool + select FRAME_POINTER
lib/tracing/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +obj-$(CONFIG_MCOUNT) += libmcount.o + +libmcount-y := mcount.o
lib/tracing/mcount.c new file +141 -0
141 lines not on this page · raw message
Steven Rostedt
reviewerPATCH 3/20 Annotate core code that should not be traced
In reply to Steven RostedtMark with "notrace" functions in core code that should not be traced. The "notrace" attribute will prevent gcc from adding a call to mcount on the annotated funtions.
--- lib/smp_processor_id.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/smp_processor_id.c +1 -1
@@ -7,7 +7,7 @@#include <linux/kallsyms.h> #include <linux/sched.h> -unsigned int debug_smp_processor_id (void ) +notrace unsigned int debug_smp_processor_id(void) {unsigned long preempt_count =preempt_count ();int this_cpu =raw_smp_processor_id ();
Steven Rostedt
reviewerPATCH 4/20 x86_64: notrace annotations
In reply to Steven RostedtAdd "notrace" annotation to x86_64 specific files.
--- arch/x86/kernel/head64.c | 2 +- arch/x86/kernel/setup64.c | 4 ++-- arch/x86/kernel/smpboot_64.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/head64.c +1 -1
@@ -46,7 +46,7 @@ static void __init copy_bootdata(char *r } } -void __init x86_64_start_kernel (char * real_mode_data) +notrace void __init x86_64_start_kernel(char *real_mode_data) {int i;
arch/x86/kernel/setup64.c +2 -2
@@ -114,7 +114,7 @@ void __init setup_per_cpu_areas(void) } } -void pda_init (int cpu) +notrace void pda_init(int cpu) {struct x8664_pda *pda =cpu_pda (cpu);
@@ -197,7 +197,7 @@ DEFINE_PER_CPU(struct orig_ist, orig_ist* 'CPU state barrier', nothing should get across. * A lot of state is already set up in PDA init. */ -void __cpuinit cpu_init (void ) +notrace void __cpuinit cpu_init(void) {int cpu =stack_smp_processor_id ();struct tss_struct *t = &per_cpu (init_tss, cpu);
arch/x86/kernel/smpboot_64.c +1 -1
@@ -317,7 +317,7 @@ static inline void set_cpu_sibling_map(i/* * Setup code on secondary processor (after comming out of the trampoline) */ -void __cpuinit start_secondary (void ) +notrace __cpuinit void start_secondary(void) {/* * Dont put anything before smp_callin(), SMP
Steven Rostedt
reviewerPATCH 5/20 add notrace annotations to vsyscall.
In reply to Steven RostedtAdd the notrace annotations to some of the vsyscall functions.
--- arch/x86/kernel/vsyscall_64.c | 3 ++- arch/x86/vdso/vclock_gettime.c | 15 ++++++++------- arch/x86/vdso/vgetcpu.c | 3 ++- include/asm-x86/vsyscall.h | 3 ++- 4 files changed, 14 insertions(+), 10 deletions(-)
arch/x86/kernel/vsyscall_64.c +2 -1
@@ -42,7 +42,8 @@#include <asm/topology.h> #include <asm/vgtod.h> -#define __vsyscall (nr) __attribute__ ((unused,__section__(".vsyscall_" #nr))) +#define __vsyscall (nr) \ + __attribute__ ((unused, __section__(".vsyscall_" #nr))) notrace#define __syscall_clobber "r11","rcx","memory"#define __pa_vsymbol (x) \ ({unsigned long v; \
arch/x86/vdso/vclock_gettime.c +8 -7
@@ -24,7 +24,7 @@#define gtod vdso_vsyscall_gtod_data -static long vdso_fallback_gettime (long clock,struct timespec *ts) +notrace static long vdso_fallback_gettime(long clock, struct timespec *ts) {long ret; asm("syscall" : "=a" (ret) :
@@ -32,7 +32,7 @@ static long vdso_fallback_gettime(long creturn ret; } -static inline long vgetns (void ) +notrace static inline long vgetns(void) {long v;cycles_t (*vread)(void);
@@ -41,7 +41,7 @@ static inline long vgetns(void)return (v * gtod->clock.mult) >> gtod->clock.shift; } -static noinline int do_realtime(struct timespec *ts) +notrace static noinline int do_realtime(struct timespec *ts) {unsigned long seq, ns; do {
@@ -55,7 +55,8 @@ static noinline int do_realtime(struct t }/* Copy of the version in kernel/time.c which we cannot directly access */ -static void vset_normalized_timespec (struct timespec *ts,long sec,long nsec) +notrace static void +vset_normalized_timespec (struct timespec *ts,long sec,long nsec) {while (nsec >=NSEC_PER_SEC ) { nsec -=NSEC_PER_SEC ;
@@ -69,7 +70,7 @@ static void vset_normalized_timespec(str ts->tv_nsec = nsec; } -static noinline int do_monotonic(struct timespec *ts) +notrace static noinline int do_monotonic(struct timespec *ts) {unsigned long seq, ns, secs; do {
@@ -83,7 +84,7 @@ static noinline int do_monotonic(structreturn 0 ; } -int __vdso_clock_gettime (clockid_t clock,struct timespec *ts) +notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts) {if (likely (gtod->sysctl_enabled && gtod->clock.vread))switch (clock) {
@@ -97,7 +98,7 @@ int __vdso_clock_gettime(clockid_t clockint clock_gettime (clockid_t ,struct timespec *) __attribute__((weak,alias ("__vdso_clock_gettime" ))); -int __vdso_gettimeofday (struct timeval *tv,struct timezone *tz) +notrace int __vdso_gettimeofday(struct timeval *tv, struct timezone *tz) {long ret;if (likely (gtod->sysctl_enabled && gtod->clock.vread)) {
arch/x86/vdso/vgetcpu.c +2 -1
@@ -13,7 +13,8 @@#include <asm/vgtod.h> #include "vextern.h" -long __vdso_getcpu (unsigned *cpu,unsigned *node,struct getcpu_cache *unused) +notrace long +__vdso_getcpu (unsigned *cpu,unsigned *node,struct getcpu_cache *unused) {unsigned int dummy, p;
include/asm-x86/vsyscall.h +2 -1
@@ -24,7 +24,8 @@ enum vsyscall_num { ((unused, __section__ (".vsyscall_gtod_data"),aligned(16)))#define __section_vsyscall_clock __attribute__ \ ((unused, __section__ (".vsyscall_clock"),aligned(16))) -#define __vsyscall_fn __attribute__ ((unused,__section__(".vsyscall_fn"))) +#define __vsyscall_fn \ + __attribute__ ((unused, __section__(".vsyscall_fn"))) notrace#define VGETCPU_RDTSCP 1#define VGETCPU_LSL 2
Steven Rostedt
reviewerPATCH 6/20 add notrace annotations for NMI routines
In reply to Steven RostedtThis annotates NMI functions with notrace. Some tracers may be able to live with this, but some cannot. So we turn off NMI tracing.
One solution might be to make a notrace_nmi which would only turn off NMI tracing if a trace utility needed it off.
--- arch/x86/kernel/nmi_32.c | 2 +- arch/x86/kernel/nmi_64.c | 2 +- arch/x86/kernel/traps_32.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/nmi_32.c +1 -1
@@ -323,7 +323,7 @@ EXPORT_SYMBOL(touch_nmi_watchdog);extern void die_nmi (struct pt_regs *,const char *msg); -__kprobes int nmi_watchdog_tick (struct pt_regs * regs,unsigned reason) +notrace __kprobes int nmi_watchdog_tick(struct pt_regs *regs, unsigned reason) {/*
arch/x86/kernel/nmi_64.c +1 -1
@@ -314,7 +314,7 @@ void touch_nmi_watchdog(void)touch_softlockup_watchdog (); } -int __kprobes nmi_watchdog_tick (struct pt_regs * regs,unsigned reason) +notrace __kprobes int nmi_watchdog_tick(struct pt_regs *regs, unsigned reason) {int sum;int touched =0 ;
arch/x86/kernel/traps_32.c +2 -2
@@ -723,7 +723,7 @@ void __kprobes die_nmi(struct pt_regs *rdo_exit (SIGSEGV ); } -static __kprobes void default_do_nmi (struct pt_regs * regs) +static notrace __kprobes void default_do_nmi(struct pt_regs *regs) {unsigned char reason =0 ;
@@ -763,7 +763,7 @@ static __kprobes void default_do_nmi(strstatic int ignore_nmis; -fastcall __kprobes void do_nmi(struct pt_regs * regs, long error_code) +notrace fastcall __kprobes void do_nmi(struct pt_regs *regs, long error_code) {int cpu;
Re: [PATCH 06/20 -v5] add notrace annotations for NMI routines
Quoted 6 lines · * Steven Rostedt ([email protected]) wrote:
This annotates NMI functions with notrace. Some tracers may be able to live with this, but some cannot. So we turn off NMI tracing.
One solution might be to make a
notrace_nmiwhich would only turn off NMI tracing if a trace utility needed it off.
Is this still needed with the atomic clocksource read ?
Quoted 59 lines
Signed-off-by Arnaldo Carvalho de Melo <[email protected]>Signed-off-by Steven Rostedt <[email protected]>--- arch/x86/kernel/nmi_32.c | 2 +- arch/x86/kernel/nmi_64.c | 2 +- arch/x86/kernel/traps_32.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-)Index:
linux-mcount.git/arch/x86/kernel/nmi_32.c
===================================================================arch/x86/kernel/nmi_32.c +1
-1@@ -323,7 +323,7 @@ EXPORT_SYMBOL(touch_nmi_watchdog);extern void die_nmi (struct pt_regs *,const char *msg); -__kprobes int nmi_watchdog_tick (struct pt_regs * regs,unsigned reason) +notrace __kprobes int nmi_watchdog_tick(struct pt_regs *regs, unsigned reason) {/* Index:
linux-mcount.git/arch/x86/kernel/nmi_64.c
===================================================================arch/x86/kernel/nmi_64.c +1
-1@@ -314,7 +314,7 @@ void touch_nmi_watchdog(void)touch_softlockup_watchdog (); } -int __kprobes nmi_watchdog_tick (struct pt_regs * regs,unsigned reason) +notrace __kprobes int nmi_watchdog_tick(struct pt_regs *regs, unsigned reason) {int sum;int touched =0 ;Index:
linux-mcount.git/arch/x86/kernel/traps_32.c
===================================================================arch/x86/kernel/traps_32.c +2
-2@@ -723,7 +723,7 @@ void __kprobes die_nmi(struct pt_regs *rdo_exit (SIGSEGV ); } -static __kprobes void default_do_nmi (struct pt_regs * regs) +static notrace __kprobes void default_do_nmi(struct pt_regs *regs) {unsigned char reason =0 ;@@ -763,7 +763,7 @@ static __kprobes void default_do_nmi(strstatic int ignore_nmis; -fastcall __kprobes void do_nmi(struct pt_regs * regs, long error_code) +notrace fastcall __kprobes void do_nmi(struct pt_regs *regs, long error_code) {int cpu;Signature · 1 line
--
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Steven Rostedt
reviewerRe: [PATCH 06/20 -v5] add notrace annotations for NMI routines
Quoted 8 lines · Mathieu Desnoyers
Quoted 6 lines · * Steven Rostedt ([email protected]) wrote:
This annotates NMI functions with notrace. Some tracers may be able to live with this, but some cannot. So we turn off NMI tracing.
One solution might be to make a
notrace_nmiwhich would only turn off NMI tracing if a trace utility needed it off.Is this still needed with the atomic clocksource read ?
This never had to do with the clocksource. The tracer itself isn't atomic against NMIs. hmm, it may actually be. It does a tracer disable for the CPU by an atomic_inc and this would prevent the NMI from causing harm.
I'll test without this patch and see what happens when I turn on a high rate of NMIS.
I still want to add a bit more notraces around for the simple reason of cleaning up the output and prehaps speeding up the code a bit.
-- Steve
Steven Rostedt
reviewerRe: [PATCH 06/20 -v5] add notrace annotations for NMI routines
In reply to Mathieu DesnoyersQuoted 9 lines · Mathieu Desnoyers
Quoted 6 lines · * Steven Rostedt ([email protected]) wrote:
This annotates NMI functions with notrace. Some tracers may be able to live with this, but some cannot. So we turn off NMI tracing.
One solution might be to make a
notrace_nmiwhich would only turn off NMI tracing if a trace utility needed it off.Is this still needed with the atomic clocksource read ?
Before you ask again, I've still included this in -v6, simply because I didn't get a chance to test it without this patch. I'll try to remember to do that on Monday.
-- Steve
Re: [PATCH 06/20 -v5] add notrace annotations for NMI routines
Quoted 9 lines · Steven Rostedt
Quoted 7 lines · Mathieu Desnoyers
Quoted 6 lines · * Steven Rostedt ([email protected]) wrote:
This annotates NMI functions with notrace. Some tracers may be able to live with this, but some cannot. So we turn off NMI tracing.
One solution might be to make a
notrace_nmiwhich would only turn off NMI tracing if a trace utility needed it off.
Quoted 6 lines
Quoted 2 lines
Is this still needed with the atomic clocksource read ?
Before you ask again, I've still included this in -v6, simply because I didn't get a chance to test it without this patch. I'll try to remember to do that on Monday.
Only in case you finally have to keep the annotations for the posted
tracer: Then please make it a Kconfig selectable restriction, because there are use cases of the mcount hook where you are interested in NMI invocations.
TiA,
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Steven Rostedt
reviewerRe: [PATCH 06/20 -v5] add notrace annotations for NMI routines
Quoted 13 lines · Jan Kiszka
Quoted 12 lines · Steven Rostedt
Quoted 9 lines · Mathieu Desnoyers
Quoted 6 lines · * Steven Rostedt ([email protected]) wrote:
This annotates NMI functions with notrace. Some tracers may be able to live with this, but some cannot. So we turn off NMI tracing.
One solution might be to make a
notrace_nmiwhich would only turn off NMI tracing if a trace utility needed it off.Is this still needed with the atomic clocksource read ?
Quoted 8 lines
Quoted 3 lines
Before you ask again, I've still included this in -v6, simply because I didn't get a chance to test it without this patch. I'll try to remember to do that on Monday.
Only in case you finally have to keep the annotations for the posted
tracer: Then please make it aKconfigselectable restriction, because there are use cases of the mcount hook where you are interested in NMI invocations.
I've thought about adding a notrace_nmi which would do precisely that. This week I'll take this patch out and turn on a bunch of nmi profiling and see how it holds up.
Thanks,
-- Steve
Steven Rostedt
reviewerPATCH 7/20 handle accurate time keeping over long delays
In reply to Steven RostedtHandle accurate time even if there's a long delay between accumulated clock cycles.
--- arch/powerpc/kernel/time.c | 3 +- arch/x86/kernel/vsyscall_64.c | 5 ++- include/asm-x86/vgtod.h | 2 - include/linux/clocksource.h | 58 ++++++++++++++++++++++++++++++++++++++++-- kernel/time/timekeeping.c | 36 +++++++++++++------------- 5 files changed, 82 insertions(+), 22 deletions(-)
arch/x86/kernel/vsyscall_64.c +4 -1
@@ -86,6 +86,7 @@ void update_vsyscall(struct timespec *wa vsyscall_gtod_data.clock.mask = clock->mask; vsyscall_gtod_data.clock.mult = clock->mult; vsyscall_gtod_data.clock.shift = clock->shift; + vsyscall_gtod_data.clock.cycle_accumulated = clock->cycle_accumulated; vsyscall_gtod_data.wall_time_sec = wall_time->tv_sec; vsyscall_gtod_data.wall_time_nsec = wall_time->tv_nsec; vsyscall_gtod_data.wall_to_monotonic = wall_to_monotonic;
@@ -121,7 +122,7 @@ static __always_inline long time_syscall static __always_inline void do_vgettimeofday(struct timeval * tv) { -cycle_t now, base, mask, cycle_delta; +cycle_t now, base, accumulated, mask, cycle_delta;unsigned seq;unsigned long mult, shift, nsec;cycle_t (*vread)(void);
@@ -135,6 +136,7 @@ static __always_inline void do_vgettimeo } now =vread (); base = __vsyscall_gtod_data.clock.cycle_last; + accumulated = __vsyscall_gtod_data.clock.cycle_accumulated; mask = __vsyscall_gtod_data.clock.mask; mult = __vsyscall_gtod_data.clock.mult; shift = __vsyscall_gtod_data.clock.shift;
@@ -145,6 +147,7 @@ static __always_inline void do_vgettimeo/* calculate interval: */ cycle_delta = (now - base) & mask; + cycle_delta += accumulated;/* convert to nsecs: */ nsec += (cycle_delta * mult) >> shift;
include/asm-x86/vgtod.h +1 -1
@@ -15,7 +15,7 @@ struct vsyscall_gtod_data {struct timezone sys_tz;struct {/* extract of a clocksource struct */ cycle_t (*vread)(void ); -cycle_t cycle_last; +cycle_t cycle_last, cycle_accumulated;cycle_t mask;u32 mult;u32 shift;
include/linux/clocksource.h +56 -2
95 lines not on this page · raw message
kernel/time/timekeeping.c +19 -17
@@ -66,16 +66,10 @@ static struct clocksource *clock; /* poi*/ static inline s64 __get_nsec_offset (void ) { -cycle_t cycle_now, cycle_delta; +cycle_t cycle_delta;s64 ns_offset; -/* read clocksource: */ - cycle_now =clocksource_read (clock); - -/* calculate the delta since the last update_wall_time: */ - cycle_delta = (cycle_now - clock->cycle_last) & clock->mask; - -/* convert to nanoseconds: */ + cycle_delta =clocksource_get_cycles (clock,clocksource_read (clock)); ns_offset =cyc2ns (clock, cycle_delta);return ns_offset;
@@ -195,7 +189,7 @@ static void change_clocksource(void) clock = new; clock->cycle_last = now; - + clock->cycle_accumulated =0 ; clock->error =0 ; clock->xtime_nsec =0 ;clocksource_calculate_interval (clock,NTP_INTERVAL_LENGTH );
@@ -205,9 +199,15 @@ static void change_clocksource(void)printk (KERN_INFO "Time: %s clocksource has been installed.\n" , clock->name); } + +void timekeeping_accumulate (void ) +{ +clocksource_accumulate (clock,clocksource_read (clock)); +}#else static inline void change_clocksource (void ) { }static inline s64 __get_nsec_offset (void ) {return 0 ; } +void timekeeping_accumulate (void ) { }#endif /**
@@ -302,6 +302,7 @@ static int timekeeping_resume(struct systimespec_add_ns (&xtime, timekeeping_suspend_nsecs);/* re-base the last cycle value */ clock->cycle_last =clocksource_read (clock); + clock->cycle_accumulated =0 ; clock->error =0 ; timekeeping_suspended =0 ;write_sequnlock_irqrestore (&xtime_lock, flags);
@@ -448,27 +449,28 @@ static void clocksource_adjust(s64 offse*/ void update_wall_time (void ) { -cycle_t offset; +cycle_t cycle_now;/* Make sure we're fully resumed: */ if (unlikely (timekeeping_suspended))return ;#ifdef CONFIG_GENERIC_TIME - offset = (clocksource_read (clock) - clock->cycle_last) & clock->mask; + cycle_now =clocksource_read (clock);#else - offset = clock->cycle_interval; + cycle_now = clock->cycle_last + clock->cycle_interval;#endif +clocksource_accumulate (clock, cycle_now); + clock->xtime_nsec += (s64 )xtime.tv_nsec << clock->shift;/* normally this loop will run just once, however in the * case of lost or late ticks, it will accumulate correctly. */ -while (offset >= clock->cycle_interval) { +while (clock->cycle_accumulated >= clock->cycle_interval) {/* accumulate one interval */ clock->xtime_nsec += clock->xtime_interval; - clock->cycle_last += clock->cycle_interval; - offset -= clock->cycle_interval; + clock->cycle_accumulated -= clock->cycle_interval;if (clock->xtime_nsec >= (u64 )NSEC_PER_SEC << clock->shift) { clock->xtime_nsec -= (u64 )NSEC_PER_SEC << clock->shift;
@@ -482,13 +484,13 @@ void update_wall_time(void) }/* correct the clock when NTP error is too big */ -clocksource_adjust (offset); +clocksource_adjust (clock->cycle_accumulated);/* store full nanoseconds into xtime */ xtime.tv_nsec = (s64 )clock->xtime_nsec >> clock->shift; clock->xtime_nsec -= (s64 )xtime.tv_nsec << clock->shift; -update_xtime_cache (cyc2ns (clock, offset)); +update_xtime_cache (cyc2ns (clock, clock->cycle_accumulated));/* check to see if there is a new clocksource to use */ change_clocksource ();
arch/powerpc/kernel/time.c +2 -1
@@ -773,7 +773,8 @@ void update_vsyscall(struct timespec *wa stamp_xsec = (u64 ) xtime.tv_nsec *XSEC_PER_SEC ;do_div (stamp_xsec,1000000000 ); stamp_xsec += (u64 ) xtime.tv_sec *XSEC_PER_SEC ; -update_gtod (clock->cycle_last, stamp_xsec, t2x); +update_gtod (clock->cycle_last-clock->cycle_accumulated, + stamp_xsec, t2x); }void update_vsyscall_tz (void )
Steven Rostedt
reviewerPATCH 8/20 initialize the clock source to jiffies clock.
In reply to Steven RostedtThe latency tracer can call clocksource_read very early in bootup and before the clock source variable has been initialized. This results in a crash at boot up (even before earlyprintk is initialized). Since the clock->read variable points to NULL.
This patch simply initializes the clock to use clocksource_jiffies, so that any early user of clocksource_read will not crash.
--- include/linux/clocksource.h | 3 +++ kernel/time/timekeeping.c | 9 +++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
include/linux/clocksource.h +3 -0
@@ -273,6 +273,9 @@ extern struct clocksource* clocksource_gextern void clocksource_change_rating (struct clocksource *cs,int rating);extern void clocksource_resume (void ); +/* used to initialize clock */ +extern struct clocksource clocksource_jiffies; +#ifdef CONFIG_GENERIC_TIME_VSYSCALL extern void update_vsyscall (struct timespec *ts,struct clocksource *c);extern void update_vsyscall_tz (void );
kernel/time/timekeeping.c +7 -2
@@ -53,8 +53,13 @@ static inline void update_xtime_cache(u6timespec_add_ns (&xtime_cache, nsec); } -static struct clocksource *clock;/* pointer to current clocksource */ - +/* +* pointer to current clocksource +* Just in case we use clocksource_read before we initialize +* the actual clock source. Instead of calling a NULL read pointer +* we return jiffies. +*/ +static struct clocksource *clock = &clocksource_jiffies;#ifdef CONFIG_GENERIC_TIME /**
Re: [PATCH 08/20 -v5] initialize the clock source to jiffies clock.
Quoted 7 lines · Steven Rostedt
The latency tracer can call
clocksource_readvery early in bootup and before the clock source variable has been initialized. This results in a crash at boot up (even before earlyprintk is initialized). Since theclock->readvariable points to NULL.This patch simply initializes the clock to use
clocksource_jiffies, so that any early user ofclocksource_readwill not crash.
There's a similar problem with reads of sched_clock() too early on some architectures. This problem reminds me of a fix I need to re-send to the ARM mailing list. That aside, I heartily recommend applying this patch, whether the rest of this system gets mainlined or not.
FWIW:
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================
Steven Rostedt
reviewerPATCH 9/20 add get_monotonic_cycles
In reply to Steven RostedtThe latency tracer needs a way to get an accurate time without grabbing any locks. Locks themselves might call the latency tracer and cause at best a slow down.
This patch adds get_monotonic_cycles that returns cycles from a reliable clock source in a monotonic fashion.
--- include/linux/clocksource.h | 54 +++++++++++++++++++++++++++++++++++++------- kernel/time/timekeeping.c | 26 +++++++++++++++++++-- 2 files changed, 70 insertions(+), 10 deletions(-)
include/linux/clocksource.h +46 -8
@@ -88,8 +88,16 @@ struct clocksource {*/ struct {cycle_t cycle_last, cycle_accumulated; - }____cacheline_aligned_in_smp ; +/* base structure provides lock-free read +* access to a virtualized 64bit counter +* Uses RCU-like update. +*/ +struct { +cycle_t cycle_base_last, cycle_base; + } base[2 ]; +int base_num; + }____cacheline_aligned_in_smp ;u64 xtime_nsec;s64 error;
@@ -175,19 +183,30 @@ static inline cycle_t clocksource_read(s }/** -* clocksource_get_cycles: - Access the clocksource's accumulated cycle value +* clocksource_get_basecycles: - get the clocksource's accumulated cycle value * @cs: pointer to clocksource being read * @now: current cycle value * * Uses the clocksource to return the current cycle_t value. * NOTE!!!: This is different from clocksource_read, because it -* returns the accumulated cycle value! Must hold xtime lock! +* returns a 64bit wide accumulated value. */ static inline cycle_t -clocksource_get_cycles (struct clocksource *cs,cycle_t now) +clocksource_get_basecycles (struct clocksource *cs) { -cycle_t offset = (now - cs->cycle_last) & cs->mask; - offset += cs->cycle_accumulated; +int num; +cycle_t now, offset; + +preempt_disable (); + num = cs->base_num; +/* base_num is shared, and some archs are wacky */ +smp_read_barrier_depends (); + now =clocksource_read (cs); + offset = (now - cs->base[num].cycle_base_last); + offset &= cs->mask; + offset += cs->base[num].cycle_base; +preempt_enable (); +return offset; }
@@ -197,11 +216,27 @@ clocksource_get_cycles(struct clocksourc* @now: current cycle value * * Used to avoids clocksource hardware overflow by periodically -* accumulating the current cycle delta. Must hold xtime write lock! +* accumulating the current cycle delta. Uses RCU-like update, but +* ***still requires the xtime_lock is held for writing!*** */ static inline void clocksource_accumulate (struct clocksource *cs,cycle_t now) { -cycle_t offset = (now - cs->cycle_last) & cs->mask; +/* +* First update the monotonic base portion. +* The dual array update method allows for lock-free reading. +* 'num' is always 1 or 0. +*/ +int num =1 - cs->base_num; +cycle_t offset = (now - cs->base[1 -num].cycle_base_last); + offset &= cs->mask; + cs->base[num].cycle_base = cs->base[1 -num].cycle_base + offset; + cs->base[num].cycle_base_last = now; +/* make sure this array is visible to the world first */ +smp_wmb (); + cs->base_num = num; + +/* Now update the cycle_accumulated portion */ + offset = (now - cs->cycle_last) & cs->mask; cs->cycle_last = now; cs->cycle_accumulated += offset; }
@@ -272,6 +307,9 @@ extern int clocksource_register(struct cextern struct clocksource *clocksource_get_next (void );extern void clocksource_change_rating (struct clocksource *cs,int rating);extern void clocksource_resume (void ); +extern cycle_t get_monotonic_cycles (void ); +extern unsigned long cycles_to_usecs (cycle_t cycles); +extern cycle_t usecs_to_cycles (unsigned long usecs);/* used to initialize clock */ extern struct clocksource clocksource_jiffies;
kernel/time/timekeeping.c +24 -2
@@ -71,10 +71,12 @@ static struct clocksource *clock = &cloc*/ static inline s64 __get_nsec_offset (void ) { -cycle_t cycle_delta; +cycle_t now, cycle_delta;s64 ns_offset; - cycle_delta =clocksource_get_cycles (clock,clocksource_read (clock)); + now =clocksource_read (clock); + cycle_delta = (now - clock->cycle_last) & clock->mask; + cycle_delta += clock->cycle_accumulated; ns_offset =cyc2ns (clock, cycle_delta);return ns_offset;
@@ -103,6 +105,26 @@ static inline void __get_realtime_clock_timespec_add_ns (ts, nsecs); } +cycle_t notrace get_monotonic_cycles(void) +{ +return clocksource_get_basecycles (clock); +} + +unsigned long notrace cycles_to_usecs (cycle_t cycles) +{ +u64 ret =cyc2ns (clock, cycles); + + ret +=NSEC_PER_USEC /2 ;/* For rounding in do_div() */ +do_div (ret,NSEC_PER_USEC ); + +return ret; +} + +cycle_t notrace usecs_to_cycles(unsigned long usecs) +{ +return ns2cyc (clock, (u64 )usecs *1000 ); +} +/** * getnstimeofday - Returns the time of day in a timespec * @ts: pointer to the timespec to be set
Steven Rostedt
reviewerPATCH 10/20 add notrace annotations to timing events
In reply to Steven RostedtThis patch adds notrace annotations to timer functions that will be used by tracing. This helps speed things up and also keeps the ugliness of printing these functions down.
--- arch/x86/kernel/apic_32.c | 2 +- arch/x86/kernel/hpet.c | 2 +- arch/x86/kernel/time_32.c | 2 +- arch/x86/kernel/tsc_32.c | 2 +- arch/x86/kernel/tsc_64.c | 4 ++-- arch/x86/lib/delay_32.c | 6 +++--- drivers/clocksource/acpi_pm.c | 8 ++++---- 7 files changed, 13 insertions(+), 13 deletions(-)
arch/x86/kernel/apic_32.c +1 -1
@@ -577,7 +577,7 @@ static void local_apic_timer_interrupt(v* interrupt as well. Thus we cannot inline the local irq ... ] */ -void fastcall smp_apic_timer_interrupt(struct pt_regs *regs) +notrace fastcall void smp_apic_timer_interrupt(struct pt_regs *regs) {struct pt_regs *old_regs =set_irq_regs (regs);
arch/x86/kernel/hpet.c +1 -1
@@ -295,7 +295,7 @@ static int hpet_legacy_next_event(unsign/* * Clock source related code */ -static cycle_t read_hpet (void ) +static notrace cycle_t read_hpet(void) {return (cycle_t )hpet_readl (HPET_COUNTER ); }
arch/x86/kernel/time_32.c +1 -1
@@ -122,7 +122,7 @@ static int set_rtc_mmss(unsigned long noint timer_ack; -unsigned long profile_pc (struct pt_regs *regs) +notrace unsigned long profile_pc (struct pt_regs *regs) {unsigned long pc =instruction_pointer (regs);
arch/x86/kernel/tsc_32.c +1 -1
@@ -269,7 +269,7 @@ core_initcall(cpufreq_tsc);static unsigned long current_tsc_khz =0 ; -static cycle_t read_tsc (void ) +static notrace cycle_t read_tsc(void) {cycle_t ret;
arch/x86/kernel/tsc_64.c +2 -2
@@ -248,13 +248,13 @@ __setup("notsc", notsc_setup);/* clock source code: */ -static cycle_t read_tsc (void ) +static notrace cycle_t read_tsc(void) {cycle_t ret = (cycle_t )get_cycles_sync ();return ret; } -static cycle_t __vsyscall_fn vread_tsc(void) +static notrace cycle_t __vsyscall_fn vread_tsc(void) {cycle_t ret = (cycle_t )get_cycles_sync ();return ret;
arch/x86/lib/delay_32.c +3 -3
@@ -24,7 +24,7 @@#endif /* simple loop based delay: */ -static void delay_loop (unsigned long loops) +static notrace void delay_loop(unsigned long loops) {int d0;
@@ -39,7 +39,7 @@ static void delay_loop(unsigned long loo }/* TSC based delay: */ -static void delay_tsc (unsigned long loops) +static notrace void delay_tsc(unsigned long loops) {unsigned long bclock, now;
@@ -72,7 +72,7 @@ int read_current_timer(unsigned long *tireturn -1 ; } -void __delay (unsigned long loops) +notrace void __delay(unsigned long loops) {delay_fn (loops); }
drivers/clocksource/acpi_pm.c +4 -4
@@ -30,13 +30,13 @@*/ u32 pmtmr_ioport__read_mostly ; -static inline u32 read_pmtmr (void ) +static inline notrace u32 read_pmtmr(void) {/* mask the output to 24 bits */ return inl (pmtmr_ioport) &ACPI_PM_MASK ; } -u32 acpi_pm_read_verified (void ) +notrace u32 acpi_pm_read_verified(void) {u32 v1 =0 , v2 =0 , v3 =0 ;
@@ -56,12 +56,12 @@ u32 acpi_pm_read_verified(void)return v2; } -static cycle_t acpi_pm_read_slow (void ) +static notrace cycle_t acpi_pm_read_slow(void) {return (cycle_t )acpi_pm_read_verified (); } -static cycle_t acpi_pm_read (void ) +static notrace cycle_t acpi_pm_read(void) {return (cycle_t )read_pmtmr (); }
Steven Rostedt
reviewerPATCH 11/20 mcount based trace in the form of a header file library
In reply to Steven RostedtThis is a simple trace that uses the mcount infrastructure. It is designed to be fast and small, and easy to use. It is useful to record things that happen over a very short period of time, and not to analyze the system in general.
An interface is added to the debugfs
/debugfs/tracing/
This patch adds the following files:
available_tracers
list of available tracers. Currently only "function" is available.
current_tracer
The trace that is currently active. Empty on start up. To switch to a tracer simply echo one of the tracers that are listed in available_tracers:
echo function > /debugfs/tracing/current_tracer
trace_ctrl
echoing "1" into this file starts the mcount function tracing
(if sysctl kernel.mcount_enabled=1)
echoing "0" turns it off.
latency_trace
This file is readonly and holds the result of the trace.
trace
This file outputs a easier to read version of the trace.
iter_ctrl
Controls the way the output of traces look.
So far there's two controls:
echoing in "symonly" will only show the kallsyms variables
without the addresses (if kallsyms was configured)
echoing in "verbose" will change the output to show
a lot more data, but not very easy to understand by humans.
echoing in "nosymonly" turns off symonly. echoing in "noverbose" turns off verbose.
The output of the function_trace file is as follows
"echo noverbose > /debugfs/tracing/iter_ctrl"
preemption latency trace v1.1.5 on 2.6.24-rc7-tst
--------------------------------------------------------------------
latency: 0 us, #419428/4361791, CPU#1 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:4)
-----------------
| task: -0 (uid:0 nice:0 policy:0 rt_prio:0)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
swapper-0 0d.h. 1595128us+: set_normalized_timespec+0x8/0x2d <c043841d> (ktime_get_ts+0x4a/0x4e <c04499d4>)
swapper-0 0d.h. 1595131us+: _spin_lock+0x8/0x18 <c0630690> (hrtimer_interrupt+0x6e/0x1b0 <c0449c56>)Or with verbose turned on:
"echo verbose > /debugfs/tracing/iter_ctrl"
preemption latency trace v1.1.5 on 2.6.24-rc7-tst
--------------------------------------------------------------------
latency: 0 us, #419428/4361791, CPU#1 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:4)
-----------------
| task: -0 (uid:0 nice:0 policy:0 rt_prio:0)
-----------------
swapper 0 0 9 00000000 00000000 [f3675f41] 1595.128ms (+0.003ms): set_normalized_timespec+0x8/0x2d <c043841d> (ktime_get_ts+0x4a/0x4e <c04499d4>)
swapper 0 0 9 00000000 00000001 [f3675f45] 1595.131ms (+0.003ms): _spin_lock+0x8/0x18 <c0630690> (hrtimer_interrupt+0x6e/0x1b0 <c0449c56>)
swapper 0 0 9 00000000 00000002 [f3675f48] 1595.135ms (+0.003ms): _spin_lock+0x8/0x18 <c0630690> (hrtimer_interrupt+0x6e/0x1b0 <c0449c56>)The "trace" file is not affected by the verbose mode, but is by the symonly.
echo "nosymonly" > /debugfs/tracing/iter_ctrl
tracer:
[ 81.479967] CPU 0: bash:3154 register_mcount_function+0x5f/0x66 <ffffffff80337a4d> <-- _spin_unlock_irqrestore+0xe/0x5a <ffffffff8048cc8f> [ 81.479967] CPU 0: bash:3154 _spin_unlock_irqrestore+0x3e/0x5a <ffffffff8048ccbf> <-- sub_preempt_count+0xc/0x7a <ffffffff80233d7b> [ 81.479968] CPU 0: bash:3154 sub_preempt_count+0x30/0x7a <ffffffff80233d9f> <-- in_lock_functions+0x9/0x24 <ffffffff8025a75d> [ 81.479968] CPU 0: bash:3154 vfs_write+0x11d/0x155 <ffffffff8029a043> <-- dnotify_parent+0x12/0x78 <ffffffff802d54fb> [ 81.479968] CPU 0: bash:3154 dnotify_parent+0x2d/0x78 <ffffffff802d5516> <-- _spin_lock+0xe/0x70 <ffffffff8048c910> [ 81.479969] CPU 0: bash:3154 _spin_lock+0x1b/0x70 <ffffffff8048c91d> <-- add_preempt_count+0xe/0x77 <ffffffff80233df7> [ 81.479969] CPU 0: bash:3154 add_preempt_count+0x3e/0x77 <ffffffff80233e27> <-- in_lock_functions+0x9/0x24 <ffffffff8025a75d>
echo "symonly" > /debugfs/tracing/iter_ctrl
tracer:
[ 81.479913] CPU 0: bash:3154 register_mcount_function+0x5f/0x66 <-- _spin_unlock_irqrestore+0xe/0x5a [ 81.479913] CPU 0: bash:3154 _spin_unlock_irqrestore+0x3e/0x5a <-- sub_preempt_count+0xc/0x7a [ 81.479913] CPU 0: bash:3154 sub_preempt_count+0x30/0x7a <-- in_lock_functions+0x9/0x24 [ 81.479914] CPU 0: bash:3154 vfs_write+0x11d/0x155 <-- dnotify_parent+0x12/0x78 [ 81.479914] CPU 0: bash:3154 dnotify_parent+0x2d/0x78 <-- _spin_lock+0xe/0x70 [ 81.479914] CPU 0: bash:3154 _spin_lock+0x1b/0x70 <-- add_preempt_count+0xe/0x77 [ 81.479914] CPU 0: bash:3154 add_preempt_count+0x3e/0x77 <-- in_lock_functions+0x9/0x24
--- lib/Makefile | 1 lib/tracing/Kconfig | 15 lib/tracing/Makefile | 3 lib/tracing/trace_function.c | 72 ++ lib/tracing/tracer.c | 1149 +++++++++++++++++++++++++++++++++++++++++++ lib/tracing/tracer.h | 96 +++ 6 files changed, 1336 insertions(+)
lib/tracing/Kconfig +15 -0
@@ -8,3 +8,18 @@ config HAVE_MCOUNT config MCOUNT bool select FRAME_POINTER + +config TRACING + bool + select DEBUG_FS + +config FUNCTION_TRACER + bool "Profiler instrumentation based tracer" + depends on DEBUG_KERNEL && HAVE_MCOUNT + select MCOUNT + select TRACING + help + Use profiler instrumentation, adding -pg to CFLAGS. This will + insert a call to an architecture specific __mcount routine, + that the debugging mechanism using this facility will hook by + providing a set of inline routines.
lib/tracing/Makefile +3 -0
@@ -1,3 +1,6 @@ obj-$(CONFIG_MCOUNT) += libmcount.o +obj-$(CONFIG_TRACING) += tracer.o +obj-$(CONFIG_FUNCTION_TRACER) += trace_function.o + libmcount-y := mcount.o
lib/tracing/tracer.c new file +1149 -0
1,149 lines not on this page · raw message
lib/tracing/tracer.h new file +96 -0
96 lines not on this page · raw message
lib/Makefile +1 -0
@@ -67,6 +67,7 @@ obj-$(CONFIG_SWIOTLB) += swiotlb.o obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o obj-$(CONFIG_MCOUNT) += tracing/ +obj-$(CONFIG_TRACING) += tracing/ lib-$(CONFIG_GENERIC_BUG) += bug.o
lib/tracing/trace_function.c new file +72 -0
72 lines not on this page · raw message
Steven Rostedt
reviewerPATCH 12/20 Add context switch marker to sched.c
In reply to Steven RostedtAdd marker into context_switch to record the prev and next tasks.
--- kernel/sched.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched.c +2 -0
@@ -1933,6 +1933,8 @@ context_switch(struct rq *rq, struct tasstruct mm_struct *mm, *oldmm;prepare_task_switch (rq, prev, next); +trace_mark (kernel_sched_schedule, +"prev %p next %p" , prev, next); mm = next->mm; oldmm = prev->active_mm;/*
Steven Rostedt
reviewerPATCH 13/20 Make the task State char-string visible to all
In reply to Steven RostedtThe tracer wants to be able to convert the state number into a user visible character. This patch pulls that conversion string out the scheduler into the header. This way if it were to ever change, other parts of the kernel will know.
--- include/linux/sched.h | 2 ++ kernel/sched.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/sched.h +2 -0
@@ -1987,6 +1987,8 @@ static inline void migration_init(void) }#endif +#define TASK_STATE_TO_CHAR_STR "RSDTtZX" +#endif /* __KERNEL__ */ #endif
kernel/sched.c +1 -1
@@ -4890,7 +4890,7 @@ out_unlock:return retval; } -static const char stat_nam[] ="RSDTtZX" ; +static const char stat_nam[] =TASK_STATE_TO_CHAR_STR ;static void show_task (struct task_struct *p) {
Steven Rostedt
reviewerPATCH 14/20 Add tracing of context switches
In reply to Steven RostedtThis patch adds context switch tracing, of the format of:
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
swapper-0 1d..3 137us+: 0:140:R --> 2912:120
sshd-2912 1d..3 216us+: 2912:120:S --> 0:140
swapper-0 1d..3 261us+: 0:140:R --> 2912:120
bash-2920 0d..3 267us+: 2920:120:S --> 0:140
sshd-2912 1d..3 330us!: 2912:120:S --> 0:140
swapper-0 1d..3 2389us+: 0:140:R --> 2847:120
yum-upda-2847 1d..3 2411us!: 2847:120:S --> 0:140
swapper-0 0d..3 11089us+: 0:140:R --> 3139:120
gdm-bina-3139 0d..3 11113us!: 3139:120:S --> 0:140
swapper-0 1d..3 102328us+: 0:140:R --> 2847:120
yum-upda-2847 1d..3 102348us!: 2847:120:S --> 0:140 "sched_switch" is added to /debugfs/tracing/available_tracers
--- lib/tracing/Kconfig | 9 ++ lib/tracing/Makefile | 1 lib/tracing/trace_sched_switch.c | 134 +++++++++++++++++++++++++++++++++++++++ lib/tracing/tracer.c | 32 +++++++++ lib/tracing/tracer.h | 19 +++++ 5 files changed, 194 insertions(+), 1 deletion(-)
lib/tracing/Kconfig +9 -0
@@ -23,3 +23,12 @@ config FUNCTION_TRACER insert a call to an architecture specific __mcount routine, that the debugging mechanism using this facility will hook by providing a set of inline routines. + +config CONTEXT_SWITCH_TRACER + bool "Trace process context switches" + depends on DEBUG_KERNEL + select TRACING + help + This tracer hooks into the context switch and records + all switching of tasks. +
lib/tracing/Makefile +1 -0
@@ -1,6 +1,7 @@ obj-$(CONFIG_MCOUNT) += libmcount.o obj-$(CONFIG_TRACING) += tracer.o +obj-$(CONFIG_CONTEXT_SWITCH_TRACER) += trace_sched_switch.o obj-$(CONFIG_FUNCTION_TRACER) += trace_function.o libmcount-y := mcount.o
lib/tracing/trace_sched_switch.c new file +134 -0
134 lines not on this page · raw message
lib/tracing/tracer.c +32 -0
@@ -52,6 +52,7 @@ enum trace_type { __TRACE_FIRST_TYPE =0 ,TRACE_FN , +TRACE_CTX , __TRACE_LAST_TYPE };
@@ -227,6 +228,24 @@ notrace void tracing_function_trace(stru entry->fn.parent_ip = parent_ip; } +notrace void tracing_sched_switch_trace(struct tracing_trace *tr, +struct tracing_trace_cpu *data, +struct task_struct *prev, +struct task_struct *next, +unsigned long flags) +{ +struct tracing_entry *entry; + + entry =tracing_get_trace_entry (tr, data); +tracing_generic_entry_update (entry, flags); + entry->type =TRACE_CTX ; + entry->ctx.prev_pid = prev->pid; + entry->ctx.prev_prio = prev->prio; + entry->ctx.prev_state = prev->state; + entry->ctx.next_pid = next->pid; + entry->ctx.next_prio = next->prio; +} +enum trace_iterator {TRACE_ITER_SYM_ONLY =1 ,TRACE_ITER_VERBOSE =2 ,
@@ -545,6 +564,8 @@ lat_print_timestamp(struct seq_file *m,seq_puts (m," : " ); } +static const char state_to_char[] =TASK_STATE_TO_CHAR_STR ; + static void notraceprint_lat_fmt (struct seq_file *m,struct tracing_iterator *iter,unsigned int trace_idx,int cpu)
@@ -555,6 +576,7 @@ print_lat_fmt(struct seq_file *m, structunsigned long rel_usecs;int sym_only = !!(trace_flags &TRACE_ITER_SYM_ONLY );int verbose = !!(trace_flags &TRACE_ITER_VERBOSE ); +int S ;if (!next_entry) next_entry = entry;
@@ -581,6 +603,16 @@ print_lat_fmt(struct seq_file *m, structseq_print_ip_sym (m, entry->fn.parent_ip, sym_only);seq_puts (m,")\n" );break ; +case TRACE_CTX : +S = entry->ctx.prev_state <sizeof (state_to_char) ? + state_to_char[entry->ctx.prev_state] :'X' ; +seq_printf (m," %d:%d:%c --> %d:%d\n" , + entry->ctx.prev_pid, + entry->ctx.prev_prio, +S , + entry->ctx.next_pid, + entry->ctx.next_prio); +break ; } }
lib/tracing/tracer.h +18 -1
@@ -10,6 +10,14 @@ struct tracing_function {unsigned long parent_ip; }; +struct tracing_sched_switch { +unsigned int prev_pid; +unsigned char prev_prio; +unsigned char prev_state; +unsigned int next_pid; +unsigned char next_prio; +}; +struct tracing_entry {char type;char cpu;/* who will want to trace more than 256 CPUS? */
@@ -18,7 +26,10 @@ struct tracing_entry {int pid;cycle_t t;char comm[TASK_COMM_LEN ]; -struct tracing_function fn; +union { +struct tracing_function fn; +struct tracing_sched_switch ctx; + }; };struct tracing_trace_cpu {
@@ -82,6 +93,12 @@ void tracing_function_trace(struct traciunsigned long ip, unsigned long parent_ip, unsigned long flags); +void tracing_sched_switch_trace (struct tracing_trace *tr, +struct tracing_trace_cpu *data, +struct task_struct *prev, +struct task_struct *next, +unsigned long flags); +void tracing_start_function_trace (void );void tracing_stop_function_trace (void );
Steven Rostedt
reviewerPATCH 15/20 Generic command line storage
In reply to Steven RostedtSaving the comm of tasks for each trace is very expensive. This patch includes in the context switch hook, a way to store the last 100 command lines of tasks. This table is examined when a trace is to be printed.
Note: The comm may be destroyed if other traces are performed. Later (TBD) patches may simply store this information in the trace itself.
--- lib/tracing/Kconfig | 1 lib/tracing/trace_function.c | 2 lib/tracing/trace_sched_switch.c | 7 ++ lib/tracing/tracer.c | 104 +++++++++++++++++++++++++++++++++++++-- lib/tracing/tracer.h | 6 +- 5 files changed, 114 insertions(+), 6 deletions(-)
lib/tracing/Kconfig +1 -0
@@ -18,6 +18,7 @@ config FUNCTION_TRACER depends on DEBUG_KERNEL && HAVE_MCOUNT select MCOUNT select TRACING + select CONTEXT_SWITCH_TRACER help Use profiler instrumentation, adding -pg to CFLAGS. This will insert a call to an architecture specific __mcount routine,
lib/tracing/trace_function.c +2 -0
@@ -28,11 +28,13 @@ static notrace void function_reset(struc static notrace void start_function_trace(struct tracing_trace *tr) {function_reset (tr); +atomic_inc (&trace_record_cmdline);tracing_start_function_trace (); } static notrace void stop_function_trace(struct tracing_trace *tr) { +atomic_dec (&trace_record_cmdline);tracing_stop_function_trace (); }
lib/tracing/trace_sched_switch.c +7 -0
@@ -30,6 +30,11 @@ static notrace void sched_switch_callbacva_list ap;int cpu; +if (likely (!atomic_read (&trace_record_cmdline))) +return ; + +tracing_record_cmdline (current); +if (likely (!trace_enabled))return ;
@@ -62,6 +67,7 @@ static notrace void sched_switch_reset(s static notrace void start_sched_trace(struct tracing_trace *tr) { +atomic_inc (&trace_record_cmdline);sched_switch_reset (tr); trace_enabled =1 ; }
@@ -69,6 +75,7 @@ static notrace void start_sched_trace(st static notrace void stop_sched_trace(struct tracing_trace *tr) { trace_enabled =0 ; +atomic_dec (&trace_record_cmdline); } static notrace void sched_switch_trace_init(struct tracing_trace *tr)
lib/tracing/tracer.c +100 -4
161 lines not on this page · raw message
lib/tracing/tracer.h +4 -2
@@ -25,7 +25,6 @@ struct tracing_entry {char preempt_count;/* assumes PREEMPT_MASK is 8 bits or less */ int pid;cycle_t t; -char comm[TASK_COMM_LEN ];union {struct tracing_function fn;struct tracing_sched_switch ctx;
@@ -98,13 +97,16 @@ void tracing_sched_switch_trace(struct tstruct task_struct *prev, struct task_struct *next, unsigned long flags); - +void tracing_record_cmdline (struct task_struct *tsk);void tracing_start_function_trace (void );void tracing_stop_function_trace (void );int register_trace (struct trace_types_struct *type);void unregister_trace (struct trace_types_struct *type); +extern atomic_t trace_record_cmdline; +extern atomic_t trace_record_cmdline_disabled; + static inline notrace cycle_t now(void) {return get_monotonic_cycles ();
Steven Rostedt
reviewerPATCH 16/20 trace generic call to schedule switch
In reply to Steven RostedtThis patch adds hooks into the schedule switch tracing to allow other latency traces to hook into the schedule switches.
--- lib/tracing/trace_sched_switch.c | 123 +++++++++++++++++++++++++++++++++------ lib/tracing/tracer.h | 14 ++++ 2 files changed, 119 insertions(+), 18 deletions(-)
lib/tracing/tracer.h +14 -0
@@ -112,4 +112,18 @@ static inline notrace cycle_t now(void) return get_monotonic_cycles(); } +#ifdef CONFIG_CONTEXT_SWITCH_TRACER +typedef void (*tracer_switch_func_t )(void *private, +struct task_struct *prev, +struct task_struct *next); +struct tracer_switch_ops { +tracer_switch_func_t func; +void *private; +struct tracer_switch_ops *next; +}; + +extern int register_tracer_switch (struct tracer_switch_ops *ops); +extern int unregister_tracer_switch (struct tracer_switch_ops *ops); +#endif /* CONFIG_CONTEXT_SWITCH_TRACER */ +#endif /* _LINUX_MCOUNT_TRACER_H */
lib/tracing/trace_sched_switch.c +105 -18
144 lines not on this page · raw message
Steven Rostedt
reviewerPATCH 17/20 Add marker in try_to_wake_up
In reply to Steven RostedtAdd markers into the wakeup code, to allow the tracer to record wakeup timings.
--- kernel/sched.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/sched.c +4 -0
@@ -1633,6 +1633,10 @@ out_set_cpu: out_activate:#endif /* CONFIG_SMP */ +trace_mark (kernel_sched_wakeup, +"p %p rq->curr %p" , + p, rq->curr); +schedstat_inc (p, se.nr_wakeups);if (sync)schedstat_inc (p, se.nr_wakeups_sync);
Steven Rostedt
reviewerPATCH 18/20 mcount tracer for wakeup latency timings.
In reply to Steven RostedtThis patch adds hooks to trace the wake up latency of the highest priority waking task.
"wakeup" is added to /debugfs/tracing/available_tracers
Also added to /debugfs/tracing
tracing_max_latency
holds the current max latency for the wakeup
wakeup_thresh
if set to other than zero, a log will be recorded for every wakeup that takes longer than the number entered in here (usecs for all counters) (deletes previous trace)
Examples:
(with mcount_enabled = 0)
============
preemption latency trace v1.1.5 on 2.6.24-rc8
--------------------------------------------------------------------
latency: 26 us, #2/2, CPU#1 | (M:rt VP:0, KP:0, SP:0 HP:0 #P:2)
-----------------
| task: migration/0-3 (uid:0 nice:-5 policy:1 rt_prio:99)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
quilt-8551 0d..3 0us+: wake_up_process+0x15/0x17 <ffffffff80233e80> (sched_exec+0xc9/0x100 <ffffffff80235343>)
quilt-8551 0d..4 26us : sched_switch_callback+0x73/0x81 <ffffffff80338d2f> (schedule+0x483/0x6d5 <ffffffff8048b3ee>)vim:ft=help
============
(with mcount_enabled = 1)
============
preemption latency trace v1.1.5 on 2.6.24-rc8
--------------------------------------------------------------------
latency: 36 us, #45/45, CPU#0 | (M:rt VP:0, KP:0, SP:0 HP:0 #P:2)
-----------------
| task: migration/1-5 (uid:0 nice:-5 policy:1 rt_prio:99)
-----------------
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
bash-10653 1d..3 0us : wake_up_process+0x15/0x17 <ffffffff80233e80> (sched_exec+0xc9/0x100 <ffffffff80235343>)
bash-10653 1d..3 1us : try_to_wake_up+0x271/0x2e7 <ffffffff80233dcf> (sub_preempt_count+0xc/0x7a <ffffffff8023309e>)
bash-10653 1d..2 2us : try_to_wake_up+0x296/0x2e7 <ffffffff80233df4> (update_rq_clock+0x9/0x20 <ffffffff802303f3>)
bash-10653 1d..2 2us : update_rq_clock+0x1e/0x20 <ffffffff80230408> (__update_rq_clock+0xc/0x90 <ffffffff80230366>)
bash-10653 1d..2 3us : __update_rq_clock+0x1b/0x90 <ffffffff80230375> (sched_clock+0x9/0x29 <ffffffff80214529>)
bash-10653 1d..2 4us : try_to_wake_up+0x2a6/0x2e7 <ffffffff80233e04> (activate_task+0xc/0x3f <ffffffff8022ffca>)
bash-10653 1d..2 4us : activate_task+0x2d/0x3f <ffffffff8022ffeb> (enqueue_task+0xe/0x66 <ffffffff8022ff66>)
bash-10653 1d..2 5us : enqueue_task+0x5b/0x66 <ffffffff8022ffb3> (enqueue_task_rt+0x9/0x3c <ffffffff80233351>)
bash-10653 1d..2 6us : try_to_wake_up+0x2ba/0x2e7 <ffffffff80233e18> (check_preempt_wakeup+0x12/0x99 <ffffffff80234f84>)
[...]
bash-10653 1d..5 33us : tracing_record_cmdline+0xcf/0xd4 <ffffffff80338aad> (_spin_unlock+0x9/0x33 <ffffffff8048d3ec>)
bash-10653 1d..5 34us : _spin_unlock+0x19/0x33 <ffffffff8048d3fc> (sub_preempt_count+0xc/0x7a <ffffffff8023309e>)
bash-10653 1d..4 35us : wakeup_sched_switch+0x65/0x2ff <ffffffff80339f66> (_spin_lock_irqsave+0xc/0xa9 <ffffffff8048d08b>)
bash-10653 1d..4 35us : _spin_lock_irqsave+0x19/0xa9 <ffffffff8048d098> (add_preempt_count+0xe/0x77 <ffffffff8023311a>)
bash-10653 1d..4 36us : sched_switch_callback+0x73/0x81 <ffffffff80338d2f> (schedule+0x483/0x6d5 <ffffffff8048b3ee>)vim:ft=help
============
The [...] was added here to not waste your email box space.
--- lib/tracing/Kconfig | 14 + lib/tracing/Makefile | 1 lib/tracing/trace_wakeup.c | 333 +++++++++++++++++++++++++++++++++++++++++++++ lib/tracing/tracer.c | 131 +++++++++++++++++ lib/tracing/tracer.h | 6 5 files changed, 483 insertions(+), 2 deletions(-)
lib/tracing/Kconfig +14 -0
@@ -9,6 +9,9 @@ config MCOUNT bool select FRAME_POINTER +config TRACER_MAX_TRACE + bool + config TRACING bool select DEBUG_FS
@@ -25,6 +28,17 @@ config FUNCTION_TRACER that the debugging mechanism using this facility will hook by providing a set of inline routines. +config WAKEUP_TRACER + bool "Trace wakeup latencies" + depends on DEBUG_KERNEL + select TRACING + select CONTEXT_SWITCH_TRACER + select TRACER_MAX_TRACE + help + This tracer adds hooks into scheduling to time the latency + of the highest priority task tasks to be scheduled in + after it has worken up. + config CONTEXT_SWITCH_TRACER bool "Trace process context switches" depends on DEBUG_KERNEL
lib/tracing/Makefile +1 -0
@@ -3,5 +3,6 @@ obj-$(CONFIG_MCOUNT) += libmcount.o obj-$(CONFIG_TRACING) += tracer.o obj-$(CONFIG_CONTEXT_SWITCH_TRACER) += trace_sched_switch.o obj-$(CONFIG_FUNCTION_TRACER) += trace_function.o +obj-$(CONFIG_WAKEUP_TRACER) += trace_wakeup.o libmcount-y := mcount.o
lib/tracing/trace_wakeup.c new file +333 -0
333 lines not on this page · raw message
lib/tracing/tracer.c +129 -2
195 lines not on this page · raw message
lib/tracing/tracer.h +6 -0
@@ -69,6 +69,7 @@ struct trace_types_struct {void (*stop)(struct tracing_iterator *iter);void (*ctrl_update)(struct tracing_trace *tr);struct trace_types_struct *next; +int print_max; };struct tracing_iterator {
@@ -107,6 +108,11 @@ void unregister_trace(struct trace_typesextern atomic_t trace_record_cmdline;extern atomic_t trace_record_cmdline_disabled; +extern unsigned long tracing_max_latency; +extern unsigned long tracing_thresh; + +void update_max_tr (struct tracing_trace *tr,struct task_struct *tsk,int cpu); + static inline notrace cycle_t now(void) {return get_monotonic_cycles ();
Steven Rostedt
reviewerPATCH 19/20 Trace irq disabled critical timings
In reply to Steven RostedtThis patch adds latency tracing for critical timings (how long interrupts are disabled for).
"irqsoff" is added to /debugfs/tracing/available_tracers
Note:
tracing_max_latency
also holds the max latency for irqsoff (in usecs).
(default to large number so one must start latency tracing)
tracing_thresh
threshold (in usecs) to always print out if irqs off is detected to be longer than stated here. If irq_thresh is non-zero, then max_irq_latency is ignored.
Here's an example of a trace with mcount_enabled = 0
=======
preemption latency trace v1.1.5 on 2.6.24-rc7
--------------------------------------------------------------------
latency: 100 us, #3/3, CPU#1 | (M:rt VP:0, KP:0, SP:0 HP:0 #P:2)
-----------------
| task: swapper-0 (uid:0 nice:0 policy:0 rt_prio:0)
----------------- => started at: _spin_lock_irqsave+0x2a/0xb7
=> ended at: _spin_unlock_irqrestore+0x32/0x5f
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
swapper-0 1d.s3 0us+: _spin_lock_irqsave+0x2a/0xb7 (e1000_update_stats+0x47/0x64c [e1000])
swapper-0 1d.s3 100us : _spin_unlock_irqrestore+0x32/0x5f (e1000_update_stats+0x641/0x64c [e1000])
swapper-0 1d.s3 100us : trace_hardirqs_on_caller+0x75/0x89 (_spin_unlock_irqrestore+0x32/0x5f)vim:ft=help
=======
And this is a trace with mcount_enabled == 1
=======
preemption latency trace v1.1.5 on 2.6.24-rc7
--------------------------------------------------------------------
latency: 102 us, #12/12, CPU#1 | (M:rt VP:0, KP:0, SP:0 HP:0 #P:2)
-----------------
| task: swapper-0 (uid:0 nice:0 policy:0 rt_prio:0)
----------------- => started at: _spin_lock_irqsave+0x2a/0xb7
=> ended at: _spin_unlock_irqrestore+0x32/0x5f
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
swapper-0 1dNs3 0us+: _spin_lock_irqsave+0x2a/0xb7 (e1000_update_stats+0x47/0x64c [e1000])
swapper-0 1dNs3 46us : e1000_read_phy_reg+0x16/0x225 [e1000] (e1000_update_stats+0x5e2/0x64c [e1000])
swapper-0 1dNs3 46us : e1000_swfw_sync_acquire+0x10/0x99 [e1000] (e1000_read_phy_reg+0x49/0x225 [e1000])
swapper-0 1dNs3 46us : e1000_get_hw_eeprom_semaphore+0x12/0xa6 [e1000] (e1000_swfw_sync_acquire+0x36/0x99 [e1000])
swapper-0 1dNs3 47us : __const_udelay+0x9/0x47 (e1000_read_phy_reg+0x116/0x225 [e1000])
swapper-0 1dNs3 47us+: __delay+0x9/0x50 (__const_udelay+0x45/0x47)
swapper-0 1dNs3 97us : preempt_schedule+0xc/0x84 (__delay+0x4e/0x50)
swapper-0 1dNs3 98us : e1000_swfw_sync_release+0xc/0x55 [e1000] (e1000_read_phy_reg+0x211/0x225 [e1000])
swapper-0 1dNs3 99us+: e1000_put_hw_eeprom_semaphore+0x9/0x35 [e1000] (e1000_swfw_sync_release+0x50/0x55 [e1000])
swapper-0 1dNs3 101us : _spin_unlock_irqrestore+0xe/0x5f (e1000_update_stats+0x641/0x64c [e1000])
swapper-0 1dNs3 102us : _spin_unlock_irqrestore+0x32/0x5f (e1000_update_stats+0x641/0x64c [e1000])
swapper-0 1dNs3 102us : trace_hardirqs_on_caller+0x75/0x89 (_spin_unlock_irqrestore+0x32/0x5f)vim:ft=help
=======
--- arch/x86/kernel/process_64.c | 3 arch/x86/lib/thunk_64.S | 18 + include/asm-x86/irqflags_32.h | 4 include/asm-x86/irqflags_64.h | 4 include/linux/irqflags.h | 37 ++- include/linux/mcount.h | 31 ++- kernel/fork.c | 2 kernel/lockdep.c | 20 +- lib/tracing/Kconfig | 18 + lib/tracing/Makefile | 1 lib/tracing/trace_irqsoff.c | 415 ++++++++++++++++++++++++++++++++++++++++++ lib/tracing/tracer.c | 59 ++++- lib/tracing/tracer.h | 2 13 files changed, 579 insertions(+), 35 deletions(-)
arch/x86/kernel/process_64.c +3 -0
@@ -233,7 +233,10 @@ void cpu_idle (void)*/ local_irq_disable ();enter_idle (); +/* Don't trace irqs off for idle */ +stop_critical_timings ();idle (); +start_critical_timings ();/* In many cases the interrupt that ended idle has already called exit_idle. But some idle loops can be woken up without interrupt. */
arch/x86/lib/thunk_64.S +16 -2
@@ -47,8 +47,22 @@ thunk __up_wakeup,__up #ifdef CONFIG_TRACE_IRQFLAGS - thunk trace_hardirqs_on_thunk,trace_hardirqs_on - thunk trace_hardirqs_off_thunk,trace_hardirqs_off + /* put return address in rdi (arg1) */ + .macro thunk_ra name,func + .globl \name +\name: + CFI_STARTPROC + SAVE_ARGS + /* SAVE_ARGS pushs 9 elements */ + /* the next element would be the rip */ + movq 9*8(%rsp), %rdi + call \func + jmp restore + CFI_ENDPROC + .endm + + thunk_ra trace_hardirqs_on_thunk,trace_hardirqs_on_caller + thunk_ra trace_hardirqs_off_thunk,trace_hardirqs_off_caller #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC
include/asm-x86/irqflags_32.h +2 -2
@@ -139,9 +139,9 @@ static inline int raw_irqs_disabled(voidstatic inline void trace_hardirqs_fixup_flags (unsigned long flags) {if (raw_irqs_disabled_flags (flags)) -trace_hardirqs_off (); +__trace_hardirqs_off ();else -trace_hardirqs_on (); +__trace_hardirqs_on (); }static inline void trace_hardirqs_fixup (void )
include/asm-x86/irqflags_64.h +2 -2
@@ -120,9 +120,9 @@ static inline int raw_irqs_disabled(voidstatic inline void trace_hardirqs_fixup_flags (unsigned long flags) {if (raw_irqs_disabled_flags (flags)) -trace_hardirqs_off (); +__trace_hardirqs_off ();else -trace_hardirqs_on (); +__trace_hardirqs_on (); }static inline void trace_hardirqs_fixup (void )
include/linux/irqflags.h +29 -8
@@ -12,10 +12,21 @@#define _LINUX_TRACE_IRQFLAGS_H#ifdef CONFIG_TRACE_IRQFLAGS -extern void trace_hardirqs_on (void ); -extern void trace_hardirqs_off (void ); +# include <linux/mcount.h> +extern void trace_hardirqs_on_caller (unsigned long ip); +extern void trace_hardirqs_off_caller (unsigned long ip);extern void trace_softirqs_on (unsigned long ip);extern void trace_softirqs_off (unsigned long ip); +extern void trace_hardirqs_on (void ); +extern void trace_hardirqs_off (void ); + static inline void notrace __trace_hardirqs_on(void) + { +trace_hardirqs_on_caller (CALLER_ADDR0 ); + } + static inline void notrace __trace_hardirqs_off(void) + { +trace_hardirqs_off_caller (CALLER_ADDR0 ); + }# define trace_hardirq_context (p) ((p)->hardirq_context)# define trace_softirq_context (p) ((p)->softirq_context)# define trace_hardirqs_enabled (p) ((p)->hardirqs_enabled)
@@ -28,6 +39,8 @@#else # define trace_hardirqs_on () do { } while (0)# define trace_hardirqs_off () do { } while (0) +# define __trace_hardirqs_on () do { } while (0) +# define __trace_hardirqs_off () do { } while (0)# define trace_softirqs_on (ip) do { } while (0)# define trace_softirqs_off (ip) do { } while (0)# define trace_hardirq_context (p) 0
@@ -41,24 +54,32 @@# define INIT_TRACE_IRQFLAGS #endif +#ifdef CONFIG_CRITICAL_IRQSOFF_TIMING +extern void stop_critical_timings (void ); +extern void start_critical_timings (void ); +#else +# define stop_critical_timings () do { } while (0) +# define start_critical_timings () do { } while (0) +#endif +#ifdef CONFIG_TRACE_IRQFLAGS_SUPPORT #include <asm/irqflags.h> #define local_irq_enable () \ - do { trace_hardirqs_on(); raw_local_irq_enable(); } while (0) + do { __trace_hardirqs_on(); raw_local_irq_enable(); } while (0)#define local_irq_disable () \ - do { raw_local_irq_disable(); trace_hardirqs_off(); } while (0) + do { raw_local_irq_disable(); __trace_hardirqs_off(); } while (0)#define local_irq_save (flags) \ - do { raw_local_irq_save(flags); trace_hardirqs_off(); } while (0) + do { raw_local_irq_save(flags); __trace_hardirqs_off(); } while (0)#define local_irq_restore (flags) \ do { \ if (raw_irqs_disabled_flags(flags)) { \ raw_local_irq_restore(flags); \ - trace_hardirqs_off(); \ + __trace_hardirqs_off(); \ } else { \ - trace_hardirqs_on(); \ + __trace_hardirqs_on(); \ raw_local_irq_restore(flags); \ } \ } while (0)
@@ -76,7 +97,7 @@#ifdef CONFIG_TRACE_IRQFLAGS_SUPPORT #define safe_halt () \ do { \ - trace_hardirqs_on(); \ + __trace_hardirqs_on(); \ raw_safe_halt(); \ } while (0)
include/linux/mcount.h +27 -4
@@ -6,10 +6,6 @@ extern int mcount_enabled;#include <linux/linkage.h> -#define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0)) -#define CALLER_ADDR1 ((unsigned long)__builtin_return_address(1)) -#define CALLER_ADDR2 ((unsigned long)__builtin_return_address(2)) -typedef void (*mcount_func_t )(unsigned long ip,unsigned long parent_ip);struct mcount_ops {
@@ -35,4 +31,31 @@ extern void mcount(void);# define unregister_mcount_function (ops) do { } while (0)# define clear_mcount_function (ops) do { } while (0)#endif /* CONFIG_MCOUNT */ + + +#ifdef CONFIG_FRAME_POINTER +/* TODO: need to fix this for ARM */ +# define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0)) +# define CALLER_ADDR1 ((unsigned long)__builtin_return_address(1)) +# define CALLER_ADDR2 ((unsigned long)__builtin_return_address(2)) +# define CALLER_ADDR3 ((unsigned long)__builtin_return_address(3)) +# define CALLER_ADDR4 ((unsigned long)__builtin_return_address(4)) +# define CALLER_ADDR5 ((unsigned long)__builtin_return_address(5)) +#else +# define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0)) +# define CALLER_ADDR1 0UL +# define CALLER_ADDR2 0UL +# define CALLER_ADDR3 0UL +# define CALLER_ADDR4 0UL +# define CALLER_ADDR5 0UL +#endif + +#ifdef CONFIG_CRITICAL_IRQSOFF_TIMING + extern void notrace time_hardirqs_on(unsigned long a0, unsigned long a1); + extern void notrace time_hardirqs_off(unsigned long a0, unsigned long a1); +#else +# define time_hardirqs_on (a0, a1) do { } while (0) +# define time_hardirqs_off (a0, a1) do { } while (0) +#endif +#endif /* _LINUX_MCOUNT_H */
kernel/fork.c +1 -1
@@ -1010,7 +1010,7 @@ static struct task_struct *copy_process(rt_mutex_init_task (p); -#ifdef CONFIG_TRACE_IRQFLAGS +#if defined(CONFIG_TRACE_IRQFLAGS ) && defined(CONFIG_LOCKDEP )DEBUG_LOCKS_WARN_ON (!p->hardirqs_enabled);DEBUG_LOCKS_WARN_ON (!p->softirqs_enabled);#endif
kernel/lockdep.c +18 -2
@@ -39,6 +39,7 @@#include <linux/irqflags.h> #include <linux/utsname.h> #include <linux/hash.h> +#include <linux/mcount.h> #include <asm/sections.h>
@@ -2009,7 +2010,7 @@ void early_boot_irqs_on(void)/* * Hardirqs will be enabled: */ -void trace_hardirqs_on (void ) +void notrace trace_hardirqs_on_caller(unsigned long a0) {struct task_struct *curr = current;unsigned long ip;
@@ -2050,14 +2051,24 @@ void trace_hardirqs_on(void) curr->hardirq_enable_ip = ip; curr->hardirq_enable_event = ++curr->irq_events;debug_atomic_inc (&hardirqs_on_events); +time_hardirqs_on (CALLER_ADDR0 , a0); } +EXPORT_SYMBOL (trace_hardirqs_on_caller); +void notrace trace_hardirqs_on(void) { +trace_hardirqs_on_caller (CALLER_ADDR0 ); +}EXPORT_SYMBOL (trace_hardirqs_on); +void notrace trace_hardirqs_off(void) { +trace_hardirqs_off_caller (CALLER_ADDR0 ); +} +EXPORT_SYMBOL (trace_hardirqs_off); +/* * Hardirqs were disabled: */ -void trace_hardirqs_off (void ) +void notrace trace_hardirqs_off_caller(unsigned long a0) {struct task_struct *curr = current;
@@ -2075,10 +2086,15 @@ void trace_hardirqs_off(void) curr->hardirq_disable_ip = _RET_IP_; curr->hardirq_disable_event = ++curr->irq_events;debug_atomic_inc (&hardirqs_off_events); +time_hardirqs_off (CALLER_ADDR0 , a0); }else debug_atomic_inc (&redundant_hardirqs_off); } +void notrace trace_hardirqs_off(void) { +trace_hardirqs_off_caller (CALLER_ADDR0 ); +} +EXPORT_SYMBOL (trace_hardirqs_off);/*
lib/tracing/Kconfig +18 -0
@@ -28,6 +28,24 @@ config FUNCTION_TRACER that the debugging mechanism using this facility will hook by providing a set of inline routines. +config CRITICAL_IRQSOFF_TIMING + bool "Interrupts-off critical section latency timing" + default n + depends on TRACE_IRQFLAGS_SUPPORT + depends on GENERIC_TIME + select TRACE_IRQFLAGS + select TRACING + select TRACER_MAX_TRACE + help + This option measures the time spent in irqs-off critical + sections, with microsecond accuracy. + + The default measurement method is a maximum search, which is + disabled by default and can be runtime (re-)started + via: + + echo 0 > /debugfs/tracing/tracing_max_latency + config WAKEUP_TRACER bool "Trace wakeup latencies" depends on DEBUG_KERNEL
lib/tracing/Makefile +1 -0
@@ -3,6 +3,7 @@ obj-$(CONFIG_MCOUNT) += libmcount.o obj-$(CONFIG_TRACING) += tracer.o obj-$(CONFIG_CONTEXT_SWITCH_TRACER) += trace_sched_switch.o obj-$(CONFIG_FUNCTION_TRACER) += trace_function.o +obj-$(CONFIG_CRITICAL_IRQSOFF_TIMING) += trace_irqsoff.o obj-$(CONFIG_WAKEUP_TRACER) += trace_wakeup.o libmcount-y := mcount.o
lib/tracing/trace_irqsoff.c new file +415 -0
415 lines not on this page · raw message
lib/tracing/tracer.c +45 -14
87 lines not on this page · raw message
lib/tracing/tracer.h +2 -0
@@ -112,6 +112,8 @@ extern unsigned long tracing_max_latencyextern unsigned long tracing_thresh;void update_max_tr (struct tracing_trace *tr,struct task_struct *tsk,int cpu); +void update_max_tr_single (struct tracing_trace *tr, +struct task_struct *tsk,int cpu); static inline notrace cycle_t now(void) {
Steven Rostedt
reviewerRe: [PATCH 19/20 -v5] Trace irq disabled critical timings
Quoted 30 lines · Steven Rostedt
This patch adds latency tracing for critical timings (how long interrupts are disabled for).
"irqsoff" is added to /debugfs/tracing/available_tracers
Index:
linux-mcount.git/kernel/lockdep.c
===================================================================kernel/lockdep.c +2
-1@@ -39,6 +39,7 @@#include <linux/irqflags.h> #include <linux/utsname.h> #include <linux/hash.h> +#include <linux/mcount.h> #include <asm/sections.h> @@ -2009,7 +2010,7 @@ void early_boot_irqs_on(void)/* * Hardirqs will be enabled: */ -void trace_hardirqs_on (void ) +void notrace trace_hardirqs_on_caller(unsigned long a0) {struct task_struct *curr = current;unsigned long ip;@@ -2050,14 +2051,24 @@ void trace_hardirqs_on(void) curr->hardirq_enable_ip = ip; curr->hardirq_enable_event = ++curr->irq_events;
kernel/lockdep.c void trace_hardirqs_on(void) · line 2062
Quoted 10 lines
kernel/lockdep.c +5
-0debug_atomic_inc (&hardirqs_on_events); +time_hardirqs_on (CALLER_ADDR0 , a0); } +EXPORT_SYMBOL (trace_hardirqs_on_caller); +void notrace trace_hardirqs_on(void) { +trace_hardirqs_on_caller (CALLER_ADDR0 ); +}EXPORT_SYMBOL (trace_hardirqs_on);
vvvvvvvvvvvvvvvv
kernel/lockdep.c void trace_hardirqs_on(void) · line 2067
Quoted 5 lines
kernel/lockdep.c +5
-0+void notrace trace_hardirqs_off(void) { +trace_hardirqs_off_caller (CALLER_ADDR0 ); +} +EXPORT_SYMBOL (trace_hardirqs_off); +
^^^^^^^^^^^^^^^^
Quoted 15 lines
kernel/lockdep.c +2
-1/* * Hardirqs were disabled: */ -void trace_hardirqs_off (void ) +void notrace trace_hardirqs_off_caller(unsigned long a0) {struct task_struct *curr = current;@@ -2075,10 +2086,15 @@ void trace_hardirqs_off(void) curr->hardirq_disable_ip = _RET_IP_; curr->hardirq_disable_event = ++curr->irq_events;debug_atomic_inc (&hardirqs_off_events); +time_hardirqs_off (CALLER_ADDR0 , a0); }else
kernel/lockdep.c void trace_hardirqs_off(void) · line 2100
Quoted 10 lines
kernel/lockdep.c +4
-0debug_atomic_inc (&redundant_hardirqs_off); } +void notrace trace_hardirqs_off(void) { +trace_hardirqs_off_caller (CALLER_ADDR0 ); +} +EXPORT_SYMBOL (trace_hardirqs_off);
Bah, I didn't refresh quilt! I'll update the website with a -v5a with this fix.
-- Steve
Quoted 1 line
/*
Steven Rostedt
reviewerRe: [PATCH 19/20 -v5] Trace irq disabled critical timings
[ Here's the updated patch anyway ]
This patch adds latency tracing for critical timings (how long interrupts are disabled for).
"irqsoff" is added to /debugfs/tracing/available_tracers
Note:
tracing_max_latency
also holds the max latency for irqsoff (in usecs).
(default to large number so one must start latency tracing)
tracing_thresh
threshold (in usecs) to always print out if irqs off is detected to be longer than stated here. If irq_thresh is non-zero, then max_irq_latency is ignored.
Here's an example of a trace with mcount_enabled = 0
=======
preemption latency trace v1.1.5 on 2.6.24-rc7
--------------------------------------------------------------------
latency: 100 us, #3/3, CPU#1 | (M:rt VP:0, KP:0, SP:0 HP:0 #P:2)
-----------------
| task: swapper-0 (uid:0 nice:0 policy:0 rt_prio:0)
----------------- => started at: _spin_lock_irqsave+0x2a/0xb7
=> ended at: _spin_unlock_irqrestore+0x32/0x5f
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
swapper-0 1d.s3 0us+: _spin_lock_irqsave+0x2a/0xb7 (e1000_update_stats+0x47/0x64c [e1000])
swapper-0 1d.s3 100us : _spin_unlock_irqrestore+0x32/0x5f (e1000_update_stats+0x641/0x64c [e1000])
swapper-0 1d.s3 100us : trace_hardirqs_on_caller+0x75/0x89 (_spin_unlock_irqrestore+0x32/0x5f)vim:ft=help
=======
And this is a trace with mcount_enabled == 1
=======
preemption latency trace v1.1.5 on 2.6.24-rc7
--------------------------------------------------------------------
latency: 102 us, #12/12, CPU#1 | (M:rt VP:0, KP:0, SP:0 HP:0 #P:2)
-----------------
| task: swapper-0 (uid:0 nice:0 policy:0 rt_prio:0)
----------------- => started at: _spin_lock_irqsave+0x2a/0xb7
=> ended at: _spin_unlock_irqrestore+0x32/0x5f
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
swapper-0 1dNs3 0us+: _spin_lock_irqsave+0x2a/0xb7 (e1000_update_stats+0x47/0x64c [e1000])
swapper-0 1dNs3 46us : e1000_read_phy_reg+0x16/0x225 [e1000] (e1000_update_stats+0x5e2/0x64c [e1000])
swapper-0 1dNs3 46us : e1000_swfw_sync_acquire+0x10/0x99 [e1000] (e1000_read_phy_reg+0x49/0x225 [e1000])
swapper-0 1dNs3 46us : e1000_get_hw_eeprom_semaphore+0x12/0xa6 [e1000] (e1000_swfw_sync_acquire+0x36/0x99 [e1000])
swapper-0 1dNs3 47us : __const_udelay+0x9/0x47 (e1000_read_phy_reg+0x116/0x225 [e1000])
swapper-0 1dNs3 47us+: __delay+0x9/0x50 (__const_udelay+0x45/0x47)
swapper-0 1dNs3 97us : preempt_schedule+0xc/0x84 (__delay+0x4e/0x50)
swapper-0 1dNs3 98us : e1000_swfw_sync_release+0xc/0x55 [e1000] (e1000_read_phy_reg+0x211/0x225 [e1000])
swapper-0 1dNs3 99us+: e1000_put_hw_eeprom_semaphore+0x9/0x35 [e1000] (e1000_swfw_sync_release+0x50/0x55 [e1000])
swapper-0 1dNs3 101us : _spin_unlock_irqrestore+0xe/0x5f (e1000_update_stats+0x641/0x64c [e1000])
swapper-0 1dNs3 102us : _spin_unlock_irqrestore+0x32/0x5f (e1000_update_stats+0x641/0x64c [e1000])
swapper-0 1dNs3 102us : trace_hardirqs_on_caller+0x75/0x89 (_spin_unlock_irqrestore+0x32/0x5f)vim:ft=help
=======
--- arch/x86/kernel/process_64.c | 3 arch/x86/lib/thunk_64.S | 18 + include/asm-x86/irqflags_32.h | 4 include/asm-x86/irqflags_64.h | 4 include/linux/irqflags.h | 37 ++- include/linux/mcount.h | 31 ++- kernel/fork.c | 2 kernel/lockdep.c | 16 + lib/tracing/Kconfig | 18 + lib/tracing/Makefile | 1 lib/tracing/trace_irqsoff.c | 415 ++++++++++++++++++++++++++++++++++++++++++ lib/tracing/tracer.c | 59 ++++- lib/tracing/tracer.h | 2 13 files changed, 575 insertions(+), 35 deletions(-)
arch/x86/kernel/process_64.c +3 -0
@@ -233,7 +233,10 @@ void cpu_idle (void)*/ local_irq_disable ();enter_idle (); +/* Don't trace irqs off for idle */ +stop_critical_timings ();idle (); +start_critical_timings ();/* In many cases the interrupt that ended idle has already called exit_idle. But some idle loops can be woken up without interrupt. */
arch/x86/lib/thunk_64.S +16 -2
@@ -47,8 +47,22 @@ thunk __up_wakeup,__up #ifdef CONFIG_TRACE_IRQFLAGS - thunk trace_hardirqs_on_thunk,trace_hardirqs_on - thunk trace_hardirqs_off_thunk,trace_hardirqs_off + /* put return address in rdi (arg1) */ + .macro thunk_ra name,func + .globl \name +\name: + CFI_STARTPROC + SAVE_ARGS + /* SAVE_ARGS pushs 9 elements */ + /* the next element would be the rip */ + movq 9*8(%rsp), %rdi + call \func + jmp restore + CFI_ENDPROC + .endm + + thunk_ra trace_hardirqs_on_thunk,trace_hardirqs_on_caller + thunk_ra trace_hardirqs_off_thunk,trace_hardirqs_off_caller #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC
include/asm-x86/irqflags_32.h +2 -2
@@ -139,9 +139,9 @@ static inline int raw_irqs_disabled(voidstatic inline void trace_hardirqs_fixup_flags (unsigned long flags) {if (raw_irqs_disabled_flags (flags)) -trace_hardirqs_off (); +__trace_hardirqs_off ();else -trace_hardirqs_on (); +__trace_hardirqs_on (); }static inline void trace_hardirqs_fixup (void )
include/asm-x86/irqflags_64.h +2 -2
@@ -120,9 +120,9 @@ static inline int raw_irqs_disabled(voidstatic inline void trace_hardirqs_fixup_flags (unsigned long flags) {if (raw_irqs_disabled_flags (flags)) -trace_hardirqs_off (); +__trace_hardirqs_off ();else -trace_hardirqs_on (); +__trace_hardirqs_on (); }static inline void trace_hardirqs_fixup (void )
include/linux/irqflags.h +29 -8
@@ -12,10 +12,21 @@#define _LINUX_TRACE_IRQFLAGS_H#ifdef CONFIG_TRACE_IRQFLAGS -extern void trace_hardirqs_on (void ); -extern void trace_hardirqs_off (void ); +# include <linux/mcount.h> +extern void trace_hardirqs_on_caller (unsigned long ip); +extern void trace_hardirqs_off_caller (unsigned long ip);extern void trace_softirqs_on (unsigned long ip);extern void trace_softirqs_off (unsigned long ip); +extern void trace_hardirqs_on (void ); +extern void trace_hardirqs_off (void ); + static inline void notrace __trace_hardirqs_on(void) + { +trace_hardirqs_on_caller (CALLER_ADDR0 ); + } + static inline void notrace __trace_hardirqs_off(void) + { +trace_hardirqs_off_caller (CALLER_ADDR0 ); + }# define trace_hardirq_context (p) ((p)->hardirq_context)# define trace_softirq_context (p) ((p)->softirq_context)# define trace_hardirqs_enabled (p) ((p)->hardirqs_enabled)
@@ -28,6 +39,8 @@#else # define trace_hardirqs_on () do { } while (0)# define trace_hardirqs_off () do { } while (0) +# define __trace_hardirqs_on () do { } while (0) +# define __trace_hardirqs_off () do { } while (0)# define trace_softirqs_on (ip) do { } while (0)# define trace_softirqs_off (ip) do { } while (0)# define trace_hardirq_context (p) 0
@@ -41,24 +54,32 @@# define INIT_TRACE_IRQFLAGS #endif +#ifdef CONFIG_CRITICAL_IRQSOFF_TIMING +extern void stop_critical_timings (void ); +extern void start_critical_timings (void ); +#else +# define stop_critical_timings () do { } while (0) +# define start_critical_timings () do { } while (0) +#endif +#ifdef CONFIG_TRACE_IRQFLAGS_SUPPORT #include <asm/irqflags.h> #define local_irq_enable () \ - do { trace_hardirqs_on(); raw_local_irq_enable(); } while (0) + do { __trace_hardirqs_on(); raw_local_irq_enable(); } while (0)#define local_irq_disable () \ - do { raw_local_irq_disable(); trace_hardirqs_off(); } while (0) + do { raw_local_irq_disable(); __trace_hardirqs_off(); } while (0)#define local_irq_save (flags) \ - do { raw_local_irq_save(flags); trace_hardirqs_off(); } while (0) + do { raw_local_irq_save(flags); __trace_hardirqs_off(); } while (0)#define local_irq_restore (flags) \ do { \ if (raw_irqs_disabled_flags(flags)) { \ raw_local_irq_restore(flags); \ - trace_hardirqs_off(); \ + __trace_hardirqs_off(); \ } else { \ - trace_hardirqs_on(); \ + __trace_hardirqs_on(); \ raw_local_irq_restore(flags); \ } \ } while (0)
@@ -76,7 +97,7 @@#ifdef CONFIG_TRACE_IRQFLAGS_SUPPORT #define safe_halt () \ do { \ - trace_hardirqs_on(); \ + __trace_hardirqs_on(); \ raw_safe_halt(); \ } while (0)
include/linux/mcount.h +27 -4
@@ -6,10 +6,6 @@ extern int mcount_enabled;#include <linux/linkage.h> -#define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0)) -#define CALLER_ADDR1 ((unsigned long)__builtin_return_address(1)) -#define CALLER_ADDR2 ((unsigned long)__builtin_return_address(2)) -typedef void (*mcount_func_t )(unsigned long ip,unsigned long parent_ip);struct mcount_ops {
@@ -35,4 +31,31 @@ extern void mcount(void);# define unregister_mcount_function (ops) do { } while (0)# define clear_mcount_function (ops) do { } while (0)#endif /* CONFIG_MCOUNT */ + + +#ifdef CONFIG_FRAME_POINTER +/* TODO: need to fix this for ARM */ +# define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0)) +# define CALLER_ADDR1 ((unsigned long)__builtin_return_address(1)) +# define CALLER_ADDR2 ((unsigned long)__builtin_return_address(2)) +# define CALLER_ADDR3 ((unsigned long)__builtin_return_address(3)) +# define CALLER_ADDR4 ((unsigned long)__builtin_return_address(4)) +# define CALLER_ADDR5 ((unsigned long)__builtin_return_address(5)) +#else +# define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0)) +# define CALLER_ADDR1 0UL +# define CALLER_ADDR2 0UL +# define CALLER_ADDR3 0UL +# define CALLER_ADDR4 0UL +# define CALLER_ADDR5 0UL +#endif + +#ifdef CONFIG_CRITICAL_IRQSOFF_TIMING + extern void notrace time_hardirqs_on(unsigned long a0, unsigned long a1); + extern void notrace time_hardirqs_off(unsigned long a0, unsigned long a1); +#else +# define time_hardirqs_on (a0, a1) do { } while (0) +# define time_hardirqs_off (a0, a1) do { } while (0) +#endif +#endif /* _LINUX_MCOUNT_H */
kernel/fork.c +1 -1
@@ -1010,7 +1010,7 @@ static struct task_struct *copy_process(rt_mutex_init_task (p); -#ifdef CONFIG_TRACE_IRQFLAGS +#if defined(CONFIG_TRACE_IRQFLAGS ) && defined(CONFIG_LOCKDEP )DEBUG_LOCKS_WARN_ON (!p->hardirqs_enabled);DEBUG_LOCKS_WARN_ON (!p->softirqs_enabled);#endif
kernel/lockdep.c +14 -2
@@ -39,6 +39,7 @@#include <linux/irqflags.h> #include <linux/utsname.h> #include <linux/hash.h> +#include <linux/mcount.h> #include <asm/sections.h>
@@ -2009,7 +2010,7 @@ void early_boot_irqs_on(void)/* * Hardirqs will be enabled: */ -void trace_hardirqs_on (void ) +void notrace trace_hardirqs_on_caller(unsigned long a0) {struct task_struct *curr = current;unsigned long ip;
@@ -2050,14 +2051,19 @@ void trace_hardirqs_on(void) curr->hardirq_enable_ip = ip; curr->hardirq_enable_event = ++curr->irq_events;debug_atomic_inc (&hardirqs_on_events); +time_hardirqs_on (CALLER_ADDR0 , a0); } +EXPORT_SYMBOL (trace_hardirqs_on_caller); +void notrace trace_hardirqs_on(void) { +trace_hardirqs_on_caller (CALLER_ADDR0 ); +}EXPORT_SYMBOL (trace_hardirqs_on);/* * Hardirqs were disabled: */ -void trace_hardirqs_off (void ) +void notrace trace_hardirqs_off_caller(unsigned long a0) {struct task_struct *curr = current;
@@ -2075,9 +2081,15 @@ void trace_hardirqs_off(void) curr->hardirq_disable_ip = _RET_IP_; curr->hardirq_disable_event = ++curr->irq_events;debug_atomic_inc (&hardirqs_off_events); +time_hardirqs_off (CALLER_ADDR0 , a0); }else debug_atomic_inc (&redundant_hardirqs_off); } +EXPORT_SYMBOL (trace_hardirqs_off_caller); + +void notrace trace_hardirqs_off(void) { +trace_hardirqs_off_caller (CALLER_ADDR0 ); +}EXPORT_SYMBOL (trace_hardirqs_off);
lib/tracing/Kconfig +18 -0
@@ -28,6 +28,24 @@ config FUNCTION_TRACER that the debugging mechanism using this facility will hook by providing a set of inline routines. +config CRITICAL_IRQSOFF_TIMING + bool "Interrupts-off critical section latency timing" + default n + depends on TRACE_IRQFLAGS_SUPPORT + depends on GENERIC_TIME + select TRACE_IRQFLAGS + select TRACING + select TRACER_MAX_TRACE + help + This option measures the time spent in irqs-off critical + sections, with microsecond accuracy. + + The default measurement method is a maximum search, which is + disabled by default and can be runtime (re-)started + via: + + echo 0 > /debugfs/tracing/tracing_max_latency + config WAKEUP_TRACER bool "Trace wakeup latencies" depends on DEBUG_KERNEL
lib/tracing/Makefile +1 -0
@@ -3,6 +3,7 @@ obj-$(CONFIG_MCOUNT) += libmcount.o obj-$(CONFIG_TRACING) += tracer.o obj-$(CONFIG_CONTEXT_SWITCH_TRACER) += trace_sched_switch.o obj-$(CONFIG_FUNCTION_TRACER) += trace_function.o +obj-$(CONFIG_CRITICAL_IRQSOFF_TIMING) += trace_irqsoff.o obj-$(CONFIG_WAKEUP_TRACER) += trace_wakeup.o libmcount-y := mcount.o
lib/tracing/trace_irqsoff.c new file +415 -0
415 lines not on this page · raw message
lib/tracing/tracer.c +45 -14
87 lines not on this page · raw message
lib/tracing/tracer.h +2 -0
@@ -112,6 +112,8 @@ extern unsigned long tracing_max_latencyextern unsigned long tracing_thresh;void update_max_tr (struct tracing_trace *tr,struct task_struct *tsk,int cpu); +void update_max_tr_single (struct tracing_trace *tr, +struct task_struct *tsk,int cpu); static inline notrace cycle_t now(void) {
Re: [PATCH 19/20 -v5] Trace irq disabled critical timings
Hi -
Quoted 3 lines · Steven Rostedt
This patch adds latency tracing for critical timings (how long interrupts are disabled for). [...]
Is there an opportunity here to share effort with latencytop, and ideally to use markers as much as possible for these event hooks?
- FChE
Steven Rostedt
reviewerRe: [PATCH 19/20 -v5] Trace irq disabled critical timings
Quoted 7 lines · Frank Ch. Eigler
Quoted 3 lines · Steven Rostedt
This patch adds latency tracing for critical timings (how long interrupts are disabled for). [...]
Is there an opportunity here to share effort with latencytop, and ideally to use markers as much as possible for these event hooks?
Well, I just found out about latencytop a couple of days ago when Arjan announced it. I took a quick look and it seems to be more of a general collection of information. This tracer pinpoints out specific maxes. Although I may be wrong (I have to admit, I haven't had the time to really look at what LatencyTop does, but I have been using this tracer in -rt for a number of years).
As for using markers, I just piggied back on top of lockdep, since it has the same hooks that I needed. If we were to replace them with markers, then I would say lockdep should use the markers too.
One thing about markers, only one probe may be allowed at a time. Perhaps we should fix this by using something like I did with the mcount hook.
-- Steve
Re: [PATCH 19/20 -v5] Trace irq disabled critical timings
Quoted 17 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 7 lines · Frank Ch. Eigler
Quoted 3 lines · Steven Rostedt
This patch adds latency tracing for critical timings (how long interrupts are disabled for). [...]
Is there an opportunity here to share effort with latencytop, and ideally to use markers as much as possible for these event hooks?
Well, I just found out about latencytop a couple of days ago when Arjan announced it. I took a quick look and it seems to be more of a general collection of information. This tracer pinpoints out specific maxes. Although I may be wrong (I have to admit, I haven't had the time to really look at what LatencyTop does, but I have been using this tracer in -rt for a number of years).
Quoted 7 lines
As for using markers, I just piggied back on top of lockdep, since it has the same hooks that I needed. If we were to replace them with markers, then I would say lockdep should use the markers too.
One thing about markers, only one probe may be allowed at a time. Perhaps we should fix this by using something like I did with the mcount hook.
It's already in my -mm patches (multiple probes support). It's also in my current LTTng patchset.
Mathieu
Quoted 2 lines
-- Steve
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Re: [PATCH 19/20 -v5] Trace irq disabled critical timings
In reply to Frank Ch. EiglerQuoted 9 lines · Frank Ch. Eigler
Hi -
Quoted 3 lines · Steven Rostedt
This patch adds latency tracing for critical timings (how long interrupts are disabled for). [...]
Is there an opportunity here to share effort with latencytop, and ideally to use markers as much as possible for these event hooks?
I'm still thinking latencytop should only record stack traces and do the rest in user-space.
Also, the things the latency tracer and latecytop measure are quite different. the latency tracer measures how long a non-schedulable section is, latencytop measures how a task was scheduled away.
Re: [PATCH 19/20 -v5] Trace irq disabled critical timings
Hi -
Quoted 10 lines · Peter Zijlstra
[...]
Quoted 2 lines
Is there an opportunity here to share effort with latencytop, and ideally to use markers as much as possible for these event hooks?
I'm still thinking latencytop should only record stack traces and do the rest in user-space.
Also, the things the latency tracer and latecytop measure are quite different. the latency tracer measures how long a non-schedulable section is, latencytop measures how a task was scheduled away.
The commonality I imagine is not in what sorts of reports they create, or where they complete the formatting for and end-user. It is solely whether the kernel-side instrumentation sites can be represented by individual markers, so that these particular tools as well as future ones don't need to keep inserting custom code in similar or even the same places.
- FChE
Steven Rostedt
reviewerRe: [PATCH 19/20 -v5] Trace irq disabled critical timings
Quoted 11 lines · Frank Ch. Eigler
Quoted 4 lines
Also, the things the latency tracer and latecytop measure are quite different. the latency tracer measures how long a non-schedulable section is, latencytop measures how a task was scheduled away.
The commonality I imagine is not in what sorts of reports they create, or where they complete the formatting for and end-user. It is solely whether the kernel-side instrumentation sites can be represented by individual markers, so that these particular tools as well as future ones don't need to keep inserting custom code in similar or even the same places.
So basically just having the marker implementation there to let all tracers access to irqs enabled/disabled.
One problem is that the hooks are alse in entry.S, so the tracing ends up being architecture dependent. One solution is to have a "generic" hook for trace_hardirqs_on/off (kind of like what I did here) and add a marker inside that. The hooks in entry.S can call the generic function and have the marker call the registered probes, be that lockdep, latencytap or latency-trace.
Have the generic hook only be compiled in when one of the tracers or lockdep is configured on and they select it.
-- Steve
Re: [PATCH 19/20 -v5] Trace irq disabled critical timings
In reply to Peter ZijlstraQuoted 12 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 9 lines · Frank Ch. Eigler
Hi -
Quoted 3 lines · Steven Rostedt
This patch adds latency tracing for critical timings (how long interrupts are disabled for). [...]
Is there an opportunity here to share effort with latencytop, and ideally to use markers as much as possible for these event hooks?
Quoted 6 lines
I'm still thinking latencytop should only record stack traces and do the rest in user-space.
Also, the things the latency tracer and latecytop measure are quite different. the latency tracer measures how long a non-schedulable section is, latencytop measures how a task was scheduled away.
Hrm, if it's really the only thing latencytop is doing.. ever thought about trying LTTng ? ;) A simple plugin could be written in lttv to perform this kind of analysis.
Mathieu
Quoted 1 line
Signature · 3 lines
Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Steven Rostedt
reviewerPATCH 20/20 trace preempt off critical timings
In reply to Steven RostedtAdd preempt off timings. A lot of kernel core code is taken from the RT patch latency trace that was written by Ingo Molnar.
This adds "preemptoff" and "preemptirqsoff" to /debugfs/tracing/available_tracers
Now instead of just tracing irqs off, preemption off can be selected to be recorded.
When this is selected, it shares the same files as irqs off timings. One can either trace preemption off, irqs off, or one or the other off.
By echoing "preemptoff" into /debugfs/tracing/current_tracer, recording of preempt off only is performed. "irqsoff" will only record the time irqs are disabled, but "preemptirqsoff" will take the total time irqs or preemption are disabled. Runtime switching of these options is now supported by simpling echoing in the appropriate trace name into /debugfs/tracing/current_tracer.
--- arch/x86/kernel/process_32.c | 3 include/linux/irqflags.h | 3 include/linux/mcount.h | 8 + include/linux/preempt.h | 2 kernel/sched.c | 24 +++++ lib/tracing/Kconfig | 25 +++++ lib/tracing/Makefile | 1 lib/tracing/trace_irqsoff.c | 181 +++++++++++++++++++++++++++++++------------ 8 files changed, 195 insertions(+), 52 deletions(-)
lib/tracing/Kconfig +25 -0
@@ -46,6 +46,31 @@ config CRITICAL_IRQSOFF_TIMING echo 0 > /debugfs/tracing/tracing_max_latency + (Note that kernel size and overhead increases with this option + enabled. This option and the preempt-off timing option can be + used together or separately.) + +config CRITICAL_PREEMPT_TIMING + bool "Preemption-off critical section latency timing" + default n + depends on GENERIC_TIME + depends on PREEMPT + select TRACING + select TRACER_MAX_TRACE + help + This option measures the time spent in preemption off critical + sections, with microsecond accuracy. + + The default measurement method is a maximum search, which is + disabled by default and can be runtime (re-)started + via: + + echo 0 > /debugfs/tracing/tracing_max_latency + + (Note that kernel size and overhead increases with this option + enabled. This option and the irqs-off timing option can be + used together or separately.) + config WAKEUP_TRACER bool "Trace wakeup latencies" depends on DEBUG_KERNEL
lib/tracing/Makefile +1 -0
@@ -4,6 +4,7 @@ obj-$(CONFIG_TRACING) += tracer.o obj-$(CONFIG_CONTEXT_SWITCH_TRACER) += trace_sched_switch.o obj-$(CONFIG_FUNCTION_TRACER) += trace_function.o obj-$(CONFIG_CRITICAL_IRQSOFF_TIMING) += trace_irqsoff.o +obj-$(CONFIG_CRITICAL_PREEMPT_TIMING) += trace_irqsoff.o obj-$(CONFIG_WAKEUP_TRACER) += trace_wakeup.o libmcount-y := mcount.o
lib/tracing/trace_irqsoff.c +132 -49
302 lines not on this page · raw message
include/linux/preempt.h +1 -1
@@ -10,7 +10,7 @@#include <linux/linkage.h> #include <linux/list.h> -#ifdef CONFIG_DEBUG_PREEMPT +#if defined(CONFIG_DEBUG_PREEMPT ) || defined(CONFIG_CRITICAL_PREEMPT_TIMING ) extern void fastcall add_preempt_count(int val); extern void fastcall sub_preempt_count(int val);#else
kernel/sched.c +23 -1
@@ -63,6 +63,7 @@#include <linux/reciprocal_div.h> #include <linux/unistd.h> #include <linux/pagemap.h> +#include <linux/mcount.h> #include <asm/tlb.h> #include <asm/irq_regs.h>
@@ -3505,26 +3506,44 @@ void scheduler_tick(void)#endif } -#if defined(CONFIG_PREEMPT ) && defined(CONFIG_DEBUG_PREEMPT ) +#if defined(CONFIG_PREEMPT ) && (defined(CONFIG_DEBUG_PREEMPT ) || \ + defined(CONFIG_CRITICAL_PREEMPT_TIMING )) + +static inline unsigned long get_parent_ip (unsigned long addr) +{ +if (in_lock_functions (addr)) { + addr =CALLER_ADDR2 ; +if (in_lock_functions (addr)) + addr =CALLER_ADDR3 ; + } +return addr; +} void fastcall add_preempt_count(int val) { +#ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? */ if (DEBUG_LOCKS_WARN_ON ((preempt_count () <0 )))return ; +#endif preempt_count () += val; +#ifdef CONFIG_DEBUG_PREEMPT /* * Spinlock count overflowing soon? */ DEBUG_LOCKS_WARN_ON ((preempt_count () &PREEMPT_MASK ) >=PREEMPT_MASK -10 ); +#endif +if (preempt_count () == val) +trace_preempt_off (CALLER_ADDR0 ,get_parent_ip (CALLER_ADDR1 )); }EXPORT_SYMBOL (add_preempt_count); void fastcall sub_preempt_count(int val) { +#ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? */
@@ -3536,7 +3555,10 @@ void fastcall sub_preempt_count(int val)if (DEBUG_LOCKS_WARN_ON ((val <PREEMPT_MASK ) && !(preempt_count () &PREEMPT_MASK )))return ; +#endif +if (preempt_count () == val) +trace_preempt_on (CALLER_ADDR0 ,get_parent_ip (CALLER_ADDR1 ));preempt_count () -= val; }EXPORT_SYMBOL (sub_preempt_count);
arch/x86/kernel/process_32.c +3 -0
@@ -195,7 +195,10 @@ void cpu_idle(void)play_dead ();__get_cpu_var (irq_stat).idle_timestamp = jiffies; +/* Don't trace irqs off for idle */ +stop_critical_timings ();idle (); +start_critical_timings (); }tick_nohz_restart_sched_tick ();preempt_enable_no_resched ();
include/linux/mcount.h +8 -0
@@ -58,4 +58,12 @@ extern void mcount(void);# define time_hardirqs_off (a0, a1) do { } while (0)#endif +#ifdef CONFIG_CRITICAL_PREEMPT_TIMING + extern void notrace trace_preempt_on(unsigned long a0, unsigned long a1); + extern void notrace trace_preempt_off(unsigned long a0, unsigned long a1); +#else +# define trace_preempt_on (a0, a1) do { } while (0) +# define trace_preempt_off (a0, a1) do { } while (0) +#endif +#endif /* _LINUX_MCOUNT_H */
include/linux/irqflags.h +2 -1
@@ -54,7 +54,8 @@# define INIT_TRACE_IRQFLAGS #endif -#ifdef CONFIG_CRITICAL_IRQSOFF_TIMING +#if defined(CONFIG_CRITICAL_IRQSOFF_TIMING ) || \ + defined(CONFIG_CRITICAL_PREEMPT_TIMING )extern void stop_critical_timings (void );extern void start_critical_timings (void );#else