tracing: Fixes for 5.14-rc2
Subsystems (5)
Steven Rostedt
maintainercover tracing: Fixes for 5.14-rc2
- Fix deadloop in ring buffer because of using stale "read" variable
- Fix synthetic event use of field_pos as boolean and not an index
- Fixed histogram special var "cpu" overriding event fields called "cpu"
- Cleaned up error prone logic in alloc_synth_event()
- Removed call to synchronize_rcu_tasks_rude() when not needed
- Removed redundant initialization of a local variable "ret"
- Fixed kernel crash when updating tracepoint callbacks of different priorities.
Colin Ian King (1):
ftrace: Remove redundant initialization of variable ret
Haoran Luo (1):
tracing: Fix bug in rb_per_cpu_empty() that might cause deadloop.
Nicolas Saenz Julienne (1):
ftrace: Avoid synchronize_rcu_tasks_rude() call when not necessary
Steven Rostedt (VMware) (4):
tracing: Synthetic event field_pos is an index not a boolean tracing/histogram: Rename "cpu" to "common_cpu"
tracing: Clean up alloc_synth_event()
tracepoints: Update static_call before tp_funcs when adding a tracepoint
----
Documentation/trace/histogram.rst | 2 +- kernel/trace/ftrace.c | 5 +++-- kernel/trace/ring_buffer.c | 28 ++++++++++++++++++++++++---- kernel/trace/trace.c | 4 ++++ kernel/trace/trace_events_hist.c | 22 ++++++++++++++++------ kernel/trace/trace_events_synth.c | 8 +++----- kernel/trace/trace_synth.h | 2 +- kernel/tracepoint.c | 2 +- 8 files changed, 53 insertions(+), 20 deletions(-)
Steven Rostedt
maintainerPATCH 1/7 tracing: Fix bug in rb_per_cpu_empty() that might cause deadloop.
From: Haoran Luo <[email protected]>
The "rb_per_cpu_empty()" misinterpret the condition (as not-empty) when "head_page" and "commit_page" of "struct ring_buffer_per_cpu" points to the same buffer page, whose "buffer_data_page" is empty and "read" field is non-zero.
An error scenario could be constructed as followed (kernel perspective):
1. All pages in the buffer has been accessed by reader(s) so that all of them will have non-zero "read" field.
2. Read and clear all buffer pages so that "rb_num_of_entries()" will return 0 rendering there's no more data to read. It is also required that the "read_page", "commit_page" and "tail_page" points to the same page, while "head_page" is the next page of them.
3. Invoke "ring_buffer_lock_reserve()" with large enough "length" so that it shot pass the end of current tail buffer page. Now the "head_page", "commit_page" and "tail_page" points to the same page.
4. Discard current event with "ring_buffer_discard_commit()", so that "head_page", "commit_page" and "tail_page" points to a page whose buffer data page is now empty.
When the error scenario has been constructed, "tracing_read_pipe" will be trapped inside a deadloop: "trace_empty()" returns 0 since "rb_per_cpu_empty()" returns 0 when it hits the CPU containing such constructed ring buffer. Then "trace_find_next_entry_inc()" always return NULL since "rb_num_of_entries()" reports there's no more entry to read. Finally "trace_seq_to_user()" returns "-EBUSY" spanking "tracing_read_pipe" back to the start of the "waitagain" loop.
I've also written a proof-of-concept script to construct the scenario and trigger the bug automatically, you can use it to trace and validate my reasoning above:
https://github.com/aegistudio/RingBufferDetonator.git
Tests has been carried out on linux kernel 5.14-rc2 (2734d6c1b1a089fb593ef6a23d4b70903526fe0c), my fixed version of kernel (for testing whether my update fixes the bug) and some older kernels (for range of affected kernels). Test result is also attached to the proof-of-concept repository.
--- kernel/trace/ring_buffer.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
kernel/trace/ring_buffer.c +24 -4
@@ -3880,10 +3880,30 @@ static bool rb_per_cpu_empty(struct ring_buffer_per_cpu *cpu_buffer)if (unlikely (!head))return true; -return reader->read ==rb_page_commit (reader) && - (commit == reader || - (commit == head && - head->read ==rb_page_commit (commit))); +/* Reader should exhaust content in reader page */ +if (reader->read !=rb_page_commit (reader)) +return false; + +/* +* If writers are committing on the reader page, knowing all +* committed content has been read, the ring buffer is empty. +*/ +if (commit == reader) +return true; + +/* +* If writers are committing on a page other than reader page +* and head page, there should always be content to read. +*/ +if (commit != head) +return false; + +/* +* Writers are committing on the head page, we just need +* to care about there're committed data, and the reader will +* swap reader page with head page when it is to read data. +*/ +return rb_page_commit (commit) ==0 ; }/**
Signature · 1 line
2.30.2
Steven Rostedt
maintainerPATCH 2/7 tracing: Synthetic event field_pos is an index not a boolean
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Performing the following:
Quoted 3 lines
# echo 'wakeup_lat s32 pid; u64 delta; char wake_comm[]' > synthetic_events # echo 'hist:keys=pid:__arg__1=common_timestamp.usecs' > events/sched/sched_waking/trigger # echo 'hist:keys=next_pid:pid=next_pid,delta=common_timestamp.usecs-$__arg__1:onmatch(sched.sched_waking).trace(wakeup_lat,$pid,$delta,prev_comm)'\
> events/sched/sched_switch/trigger
Quoted 1 line
# echo 1 > events/synthetic/enable
Crashed the kernel:
BUG: kernel NULL pointer dereference, address: 000000000000001b #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] PREEMPT SMP CPU: 7 PID: 0 Comm: swapper/7 Not tainted 5.13.0-rc5-test+ #104 Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v03.03 07/14/2016 RIP: 0010:strlen+0x0/0x20 Code: f6 82 80 2b 0b bc 20 74 11 0f b6 50 01 48 83 c0 01 f6 82 80 2b 0b bc 20 75 ef c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 <80> 3f 00 74 10 48 89 f8 48 83 c0 01 80 38 9 f8 c3 31 RSP: 0018:ffffaa75000d79d0 EFLAGS: 00010046 RAX: 0000000000000002 RBX: ffff9cdb55575270 RCX: 0000000000000000 RDX: ffff9cdb58c7a320 RSI: ffffaa75000d7b40 RDI: 000000000000001b RBP: ffffaa75000d7b40 R08: ffff9cdb40a4f010 R09: ffffaa75000d7ab8 R10: ffff9cdb4398c700 R11: 0000000000000008 R12: ffff9cdb58c7a320 R13: ffff9cdb55575270 R14: ffff9cdb58c7a000 R15: 0000000000000018 FS: 0000000000000000(0000) GS:ffff9cdb5aa00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000001b CR3: 00000000c0612006 CR4: 00000000001706e0 Call Trace: trace_event_raw_event_synth+0x90/0x1d0 action_trace+0x5b/0x70 event_hist_trigger+0x4bd/0x4e0 ? cpumask_next_and+0x20/0x30 ? update_sd_lb_stats.constprop.0+0xf6/0x840 ? __lock_acquire.constprop.0+0x125/0x550 ? find_held_lock+0x32/0x90 ? sched_clock_cpu+0xe/0xd0 ? lock_release+0x155/0x440 ? update_load_avg+0x8c/0x6f0 ? enqueue_entity+0x18a/0x920 ? __rb_reserve_next+0xe5/0x460 ? ring_buffer_lock_reserve+0x12a/0x3f0 event_triggers_call+0x52/0xe0 trace_event_buffer_commit+0x1ae/0x240 trace_event_raw_event_sched_switch+0x114/0x170 __traceiter_sched_switch+0x39/0x50 __schedule+0x431/0xb00 schedule_idle+0x28/0x40 do_idle+0x198/0x2e0 cpu_startup_entry+0x19/0x20 secondary_startup_64_no_verify+0xc2/0xcb
The reason is that the dynamic events array keeps track of the field position of the fields array, via the field_pos variable in the synth_field structure. Unfortunately, that field is a boolean for some reason, which means any field_pos greater than 1 will be a bug (in this case it was 2).
--- kernel/trace/trace_synth.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_synth.h +1 -1
@@ -14,10 +14,10 @@ struct synth_field {char *name;size_t size;unsigned int offset; +unsigned int field_pos;bool is_signed;bool is_string;bool is_dynamic; -bool field_pos; };struct synth_event {
Signature · 1 line
2.30.2
Steven Rostedt
maintainerPATCH 3/7 tracing/histogram: Rename "cpu" to "common_cpu"
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Currently the histogram logic allows the user to write "cpu" in as an event field, and it will record the CPU that the event happened on.
The problem with this is that there's a lot of events that have "cpu" as a real field, and using "cpu" as the CPU it ran on, makes it impossible to run histograms on the "cpu" field of events.
For example, if I want to have a histogram on the count of the workqueue_queue_work event on its cpu field, running:
Quoted 1 line
# echo 'hist:keys=cpu' > events/workqueue/workqueue_queue_work/trigger
Gives a misleading and wrong result.
Change the command to "common_cpu" as no event should have "common_*" fields as that's a reserved name for fields used by all events. And this makes sense here as common_cpu would be a field used by all events.
Now we can even do:
Quoted 2 lines
# echo 'hist:keys=common_cpu,cpu if cpu < 100' > events/workqueue/workqueue_queue_work/trigger # cat events/workqueue/workqueue_queue_work/hist
# event histogram # # trigger info: hist:keys=common_cpu,cpu:vals=hitcount:sort=hitcount:size=2048 if cpu < 100 [active] #
{ common_cpu: 0, cpu: 2 } hitcount: 1
{ common_cpu: 0, cpu: 4 } hitcount: 1
{ common_cpu: 7, cpu: 7 } hitcount: 1
{ common_cpu: 0, cpu: 7 } hitcount: 1
{ common_cpu: 0, cpu: 1 } hitcount: 1
{ common_cpu: 0, cpu: 6 } hitcount: 2
{ common_cpu: 0, cpu: 5 } hitcount: 2
{ common_cpu: 1, cpu: 1 } hitcount: 4
{ common_cpu: 6, cpu: 6 } hitcount: 4
{ common_cpu: 5, cpu: 5 } hitcount: 14
{ common_cpu: 4, cpu: 4 } hitcount: 26
{ common_cpu: 0, cpu: 0 } hitcount: 39
{ common_cpu: 2, cpu: 2 } hitcount: 184Now for backward compatibility, I added a trick. If "cpu" is used, and the field is not found, it will fall back to "common_cpu" and work as it did before. This way, it will still work for old programs that use "cpu" to get the actual CPU, but if the event has a "cpu" as a field, it will get that event's "cpu" field, which is probably what it wants anyway.
I updated the tracefs/README to include documentation about both the common_timestamp and the common_cpu. This way, if that text is present in the README, then an application can know that common_cpu is supported over just plain "cpu".
--- Documentation/trace/histogram.rst | 2 +- kernel/trace/trace.c | 4 ++++ kernel/trace/trace_events_hist.c | 22 ++++++++++++++++------ 3 files changed, 21 insertions(+), 7 deletions(-)
Documentation/trace/histogram.rst +1 -1
@@ -191,7 +191,7 @@ Documentation written by Tom Zanussi with the event, in nanoseconds. May be modified by .usecs to have timestamps interpreted as microseconds. - cpu int the cpu on which the event occurred. + common_cpu int the cpu on which the event occurred. ====================== ==== ======================================= Extended error information
kernel/trace/trace.c +4 -0
@@ -5609,6 +5609,10 @@ static const char readme_msg[] ="\t [:name=histname1]\n" "\t [:<handler>.<action>]\n" "\t [if <filter>]\n\n" +"\t Note, special fields can be used as well:\n" +"\t common_timestamp - to record current timestamp\n" +"\t common_cpu - to record the CPU the event happened on\n" +"\n" "\t When a matching event is hit, an entry is added to a hash\n" "\t table using the key(s) and value(s) named, and the value of a\n" "\t sum called 'hitcount' is incremented. Keys and values\n"
kernel/trace/trace_events_hist.c +16 -6
@@ -1111,7 +1111,7 @@ static const char *hist_field_name(struct hist_field *field, field->flags & HIST_FIELD_FL_ALIAS) field_name =hist_field_name (field->operands[0 ], ++level); else if (field->flags & HIST_FIELD_FL_CPU) - field_name ="cpu" ; + field_name ="common_cpu" ; else if (field->flags & HIST_FIELD_FL_EXPR || field->flags & HIST_FIELD_FL_VAR_REF) {if (field->system) {
@@ -1991,14 +1991,24 @@ parse_field(struct hist_trigger_data *hist_data, struct trace_event_file *file, hist_data->enable_timestamps = true;if (*flags &HIST_FIELD_FL_TIMESTAMP_USECS ) hist_data->attrs->ts_in_usecs = true; - }else if (strcmp (field_name,"cpu" ) ==0 ) + }else if (strcmp (field_name,"common_cpu" ) ==0 ) *flags |=HIST_FIELD_FL_CPU ;else { field =trace_find_event_field (file->event_call, field_name);if (!field || !field->size) { -hist_err (tr,HIST_ERR_FIELD_NOT_FOUND ,errpos (field_name)); - field =ERR_PTR (-EINVAL ); - goto out; +/* +* For backward compatibility, if field_name +* was "cpu", then we treat this the same as +* common_cpu. +*/ +if (strcmp (field_name,"cpu" ) ==0 ) { + *flags |=HIST_FIELD_FL_CPU ; + }else { +hist_err (tr,HIST_ERR_FIELD_NOT_FOUND , +errpos (field_name)); + field =ERR_PTR (-EINVAL ); + goto out; + } } } out:
@@ -5085,7 +5095,7 @@ static void hist_field_print(struct seq_file *m, struct hist_field *hist_field)seq_printf (m,"%s=" , hist_field->var.name);if (hist_field->flags &HIST_FIELD_FL_CPU ) -seq_puts (m,"cpu" ); +seq_puts (m,"common_cpu" );else if (field_name) {if (hist_field->flags &HIST_FIELD_FL_VAR_REF || hist_field->flags &HIST_FIELD_FL_ALIAS )
Signature · 1 line
2.30.2
Steven Rostedt
maintainerPATCH 4/7 tracing: Clean up alloc_synth_event()
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
alloc_synth_event() currently has the following code to initialize the event fields and dynamic_fields:
for (i = 0, j = 0; i < n_fields; i++) {
event->fields[i] = fields[i];
if (fields[i]->is_dynamic) {
event->dynamic_fields[j] = fields[i];
event->dynamic_fields[j]->field_pos = i;
event->dynamic_fields[j++] = fields[i];
event->n_dynamic_fields++;
}
}1) It would make more sense to have all fields keep track of their field_pos.
2) event->dynmaic_fields[j] is assigned twice for no reason.
3) We can move updating event->n_dynamic_fields outside the loop, and just assign it to j.
This combination makes the code much cleaner.
--- kernel/trace/trace_events_synth.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
kernel/trace/trace_events_synth.c +3 -5
@@ -893,15 +893,13 @@ static struct synth_event *alloc_synth_event(const char *name, int n_fields,dyn_event_init (&event->devent, &synth_event_ops);for (i =0 , j =0 ; i < n_fields; i++) { + fields[i]->field_pos = i; event->fields[i] = fields[i]; -if (fields[i]->is_dynamic) { - event->dynamic_fields[j] = fields[i]; - event->dynamic_fields[j]->field_pos = i; +if (fields[i]->is_dynamic) event->dynamic_fields[j++] = fields[i]; - event->n_dynamic_fields++; - } } + event->n_dynamic_fields = j; event->n_fields = n_fields; out:return event;
Signature · 1 line
2.30.2
Steven Rostedt
maintainerPATCH 5/7 ftrace: Avoid synchronize_rcu_tasks_rude() call when not necessary
In reply to Steven RostedtFrom: Nicolas Saenz Julienne <[email protected]>
synchronize_rcu_tasks_rude() triggers IPIs and forces rescheduling on all CPUs. It is a costly operation and, when targeting nohz_full CPUs, very disrupting (hence the name). So avoid calling it when 'old_hash' doesn't need to be freed.
--- kernel/trace/ftrace.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/trace/ftrace.c +2 -1
@@ -5985,7 +5985,8 @@ ftrace_graph_release(struct inode *inode, struct file *file)* infrastructure to do the synchronization, thus we must do it * ourselves. */ -synchronize_rcu_tasks_rude (); +if (old_hash !=EMPTY_HASH ) +synchronize_rcu_tasks_rude ();free_ftrace_hash (old_hash); }
Signature · 1 line
2.30.2
Steven Rostedt
maintainerPATCH 6/7 ftrace: Remove redundant initialization of variable ret
In reply to Steven RostedtFrom: Colin Ian King <[email protected]>
The variable ret is being initialized with a value that is never read, it is being updated later on. The assignment is redundant and can be removed.
Addresses-Coverity: ("Unused value")
--- kernel/trace/ftrace.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/ftrace.c +1 -1
@@ -7545,7 +7545,7 @@ int ftrace_is_dead(void)*/ int register_ftrace_function (struct ftrace_ops *ops) { -int ret =-1 ; +int ret;ftrace_ops_init (ops);
Signature · 1 line
2.30.2
Steven Rostedt
maintainerPATCH 7/7 tracepoints: Update static_call before tp_funcs when adding a tracepoint
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Because of the significant overhead that retpolines pose on indirect calls, the tracepoint code was updated to use the new "static_calls" that can modify the running code to directly call a function instead of using an indirect caller, and this function can be changed at runtime.
In the tracepoint code that calls all the registered callbacks that are attached to a tracepoint, the following is done:
it_func_ptr = rcu_dereference_raw((&__tracepoint_##name)->funcs);
if (it_func_ptr) {
__data = (it_func_ptr)->data;
static_call(tp_func_##name)(__data, args);
}If there's just a single callback, the static_call is updated to just call that callback directly. Once another handler is added, then the static caller is updated to call the iterator, that simply loops over all the funcs in the array and calls each of the callbacks like the old method using indirect calling.
The issue was discovered with a race between updating the funcs array and updating the static_call. The funcs array was updated first and then the static_call was updated. This is not an issue as long as the first element in the old array is the same as the first element in the new array. But that assumption is incorrect, because callbacks also have a priority field, and if there's a callback added that has a higher priority than the callback on the old array, then it will become the first callback in the new array. This means that it is possible to call the old callback with the new callback data element, which can cause a kernel panic.
static_call = callback1()
funcs[] = {callback1,data1};
callback2 has higher priority than callback1
CPU 1 CPU 2
----- ----- new_funcs = {callback2,data2},
{callback1,data1}rcu_assign_pointer(tp->funcs, new_funcs);
/*
* Now tp->funcs has the new array
* but the static_call still calls callback1
*/
it_func_ptr = tp->funcs [ new_funcs ] data = it_func_ptr->data [ data2 ] static_call(callback1, data); /* Now callback1 is called with * callback2's data */ [ KERNEL PANIC ]
update_static_call(iterator);
To prevent this from happening, always switch the static_call to the iterator before assigning the tp->funcs to the new array. The iterator will always properly match the callback with its data.
To trigger this bug:
In one terminal:
while :; do hackbench 50; done
In another terminal
echo 1 > /sys/kernel/tracing/events/sched/sched_waking/enable
while :; do
echo 1 > /sys/kernel/tracing/set_event_pid;
sleep 0.5
echo 0 > /sys/kernel/tracing/set_event_pid;
sleep 0.5done
And it doesn't take long to crash. This is because the set_event_pid adds a callback to the sched_waking tracepoint with a high priority, which will be called before the sched_waking trace event callback is called.
Note, the removal to a single callback updates the array first, before changing the static_call to single callback, which is the proper order as the first element in the array is the same as what the static_call is being changed to.
--- kernel/tracepoint.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/tracepoint.c +1 -1
@@ -299,8 +299,8 @@ static int tracepoint_add_func(struct tracepoint *tp,* a pointer to it. This array is referenced by __DO_TRACE from * include/linux/tracepoint.h using rcu_dereference_sched(). */ -rcu_assign_pointer (tp->funcs, tp_funcs);tracepoint_update_call (tp, tp_funcs, false); +rcu_assign_pointer (tp->funcs, tp_funcs);static_key_enable (&tp->key);release_probes (old);
Signature · 1 line
2.30.2