tracing: Fixes for 5.8-rc
Subsystems (3)
Steven Rostedt
maintainercover tracing: Fixes for 5.8-rc
Masami Hiramatsu (2):
tracing/boot: Fix config dependency for synthedic event
tracing: Fix event trigger to accept redundant spaces
Sascha Ortmann (1):
tracing/boottime: Fix kprobe multiple events
Steven Rostedt (VMware) (1):
ring-buffer: Zero out time extend if it is nested and not absolute
----
kernel/trace/ring_buffer.c | 2 +- kernel/trace/trace_boot.c | 10 +++++++--- kernel/trace/trace_events_trigger.c | 21 +++++++++++++++++++-- 3 files changed, 27 insertions(+), 6 deletions(-)
Steven Rostedt
maintainerPATCH 1/4 ring-buffer: Zero out time extend if it is nested and not absolute
From: "Steven Rostedt (VMware)" <[email protected]>
Currently the ring buffer makes events that happen in interrupts that preempt another event have a delta of zero. (Hopefully we can change this soon). But this is to deal with the races of updating a global counter with lockless and nesting functions updating deltas.
With the addition of absolute time stamps, the time extend didn't follow this rule. A time extend can happen if two events happen longer than 2^27 nanoseconds appart, as the delta time field in each event is only 27 bits. If that happens, then a time extend is injected with 2^59 bits of nanoseconds to use (18 years). But if the 2^27 nanoseconds happen between two events, and as it is writing the event, an interrupt triggers, it will see the 2^27 difference as well and inject a time extend of its own. But a recent change made the time extend logic not take into account the nesting, and this can cause two time extend deltas to happen moving the time stamp much further ahead than the current time. This gets all reset when the ring buffer moves to the next page, but that can cause time to appear to go backwards.
This was observed in a trace-cmd recording, and since the data is saved in a file, with trace-cmd report --debug, it was possible to see that this indeed did happen!
bash-52501 110d... 81778.908247: sched_switch: bash:52501 [120] S ==> swapper/110:0 [120] [12770284:0x2e8:64] <idle>-0 110d... 81778.908757: sched_switch: swapper/110:0 [120] R ==> bash:52501 [120] [509947:0x32c:64]
TIME EXTEND: delta:306454770 length:0
bash-52501 110.... 81779.215212: sched_swap_numa: src_pid=52501 src_tgid=52388 src_ngid=52501 src_cpu=110 src_nid=2 dst_pid=52509 dst_tgid=52388 dst_ngid=52501 dst_cpu=49 dst_nid=1 [0:0x378:48]
TIME EXTEND: delta:306458165 length:0
bash-52501 110dNh. 81779.521670: sched_wakeup: migration/110:565 [0] success=1 CPU:110 [0:0x3b4:40]
and at the next page, caused the time to go backwards:
bash-52504 110d... 81779.685411: sched_switch: bash:52504 [120] S ==> swapper/110:0 [120] [8347057:0xfb4:64]
CPU:110 [SUBBUFFER START] [81779379165886:0x1320000]
<idle>-0 110dN.. 81779.379166: sched_wakeup: bash:52504 [120] success=1 CPU:110 [0:0x10:40] <idle>-0 110d... 81779.379167: sched_switch: swapper/110:0 [120] R ==> bash:52504 [120] [1168:0x3c:64]
--- kernel/trace/ring_buffer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/ring_buffer.c +1 -1
@@ -2427,7 +2427,7 @@ rb_update_event(struct ring_buffer_per_cpu *cpu_buffer,if (unlikely (info->add_timestamp)) {bool abs =ring_buffer_time_stamp_abs (cpu_buffer->buffer); - event =rb_add_time_stamp (event, info->delta, abs); + event =rb_add_time_stamp (event, abs ? info->delta : delta, abs); length -=RB_LEN_TIME_EXTEND ; delta =0 ; }
Signature · 1 line
2.26.2
Steven Rostedt
maintainerPATCH 2/4 tracing/boot: Fix config dependency for synthedic event
In reply to Steven RostedtFrom: Masami Hiramatsu <[email protected]>
Since commit 726721a51838 ("tracing: Move synthetic events to a separate file") decoupled synthetic event from histogram, boot-time tracing also has to check CONFIG_SYNTH_EVENT instead of CONFIG_HIST_TRIGGERS.
--- kernel/trace/trace_boot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_boot.c +1 -1
@@ -120,7 +120,7 @@ trace_boot_add_kprobe_event(struct xbc_node *node, const char *event) }#endif -#ifdef CONFIG_HIST_TRIGGERS +#ifdef CONFIG_SYNTH_EVENTS static int __init trace_boot_add_synth_event (struct xbc_node *node,const char *event) {
Signature · 1 line
2.26.2
Steven Rostedt
maintainerPATCH 3/4 tracing: Fix event trigger to accept redundant spaces
In reply to Steven RostedtFrom: Masami Hiramatsu <[email protected]>
Fix the event trigger to accept redundant spaces in the trigger input.
For example, these return -EINVAL
echo " traceon" > events/ftrace/print/trigger
echo "traceon if common_pid == 0" > events/ftrace/print/trigger echo "disable_event:kmem:kmalloc " > events/ftrace/print/trigger
But these are hard to find what is wrong.
To fix this issue, use skip_spaces() to remove spaces in front of actual tokens, and set NULL if there is no token.
--- kernel/trace/trace_events_trigger.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-)
kernel/trace/trace_events_trigger.c +19 -2
@@ -216,11 +216,17 @@ static int event_trigger_regex_open(struct inode *inode, struct file *file)int trigger_process_regex (struct trace_event_file *file,char *buff) { -char *command, *next = buff; +char *command, *next;struct event_command *p;int ret = -EINVAL ; + next = buff =skip_spaces (buff); command =strsep (&next,": \t" ); +if (next) { + next =skip_spaces (next); +if (!*next) + next =NULL ; + } command = (command[0 ] !='!' ) ? command : command +1 ;mutex_lock (&trigger_cmd_mutex);
@@ -630,8 +636,14 @@ event_trigger_callback(struct event_command *cmd_ops,int ret;/* separate the trigger from the filter (t:n [if filter]) */ -if (param &&isdigit (param[0 ])) +if (param &&isdigit (param[0 ])) { trigger =strsep (¶m," \t" ); +if (param) { + param =skip_spaces (param); +if (!*param) + param =NULL ; + } + } trigger_ops = cmd_ops->get_trigger_ops (cmd, trigger);
@@ -1368,6 +1380,11 @@ int event_enable_trigger_func(struct event_command *cmd_ops, trigger =strsep (¶m," \t" );if (!trigger)return -EINVAL ; +if (param) { + param =skip_spaces (param); +if (!*param) + param =NULL ; + } system =strsep (&trigger,":" );if (!trigger)
Signature · 1 line
2.26.2
Steven Rostedt
maintainerPATCH 4/4 tracing/boottime: Fix kprobe multiple events
In reply to Steven RostedtFrom: Sascha Ortmann <[email protected]>
Fix boottime kprobe events to report and abort after each failure when adding probes.
As an example, when we try to set multiprobe kprobe events in bootconfig like this:
ftrace.event.kprobes.vfsevents {
probes = "vfs_read $arg1 $arg2,,
!error! not reported;?", // leads to error
"vfs_write $arg1 $arg2"
}This will not work as expected. After
commit da0f1f4167e3af69e ("tracing/boottime: Fix kprobe event API usage"), the function trace_boot_add_kprobe_event will not produce any error message when adding a probe fails at kprobe_event_gen_cmd_start. Furthermore, we continue to add probes when kprobe_event_gen_cmd_end fails (and kprobe_event_gen_cmd_start did not fail). In this case the function even returns successfully when the last call to kprobe_event_gen_cmd_end is successful.
The behaviour of reporting and aborting after failures is not consistent.
The function trace_boot_add_kprobe_event now reports each failure and stops adding probes immediately.
--- kernel/trace/trace_boot.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
kernel/trace/trace_boot.c +6 -2
@@ -101,12 +101,16 @@ trace_boot_add_kprobe_event(struct xbc_node *node, const char *event)kprobe_event_cmd_init (&cmd, buf,MAX_BUF_LEN ); ret =kprobe_event_gen_cmd_start (&cmd, event, val); -if (ret) +if (ret) { +pr_err ("Failed to generate probe: %s\n" , buf);break ; + } ret =kprobe_event_gen_cmd_end (&cmd); -if (ret) +if (ret) {pr_err ("Failed to add probe: %s\n" , buf); +break ; + } }return ret;
Signature · 1 line
2.26.2