tracing: Fixes for v7.3
Subsystems (4)
cover tracing: Fixes for v7.3
tracing fixes for v7.3:
- Fix several tracefs files that did not take the trace_array reference
A trace instance can be created and destroyed in the tracefs "instances" directory via mkdir and rmdir respectively. The instance is represented by a trace_array descriptor. Most tracefs files pass the trace_array as the private data of the inode to the open/read/write functions. Since there is no locking between the time a task opens a file and the deletion of the instance (and the freeing of the trace_array), each open needs to get a reference to the trace_array and each close must remove it. A instance can't be removed if there's any reference taken on its trace_array. The open function uses trace_array_get() that takes a lock (preventing removal of instances) and iterates the list of all existing trace_arrays and if it finds a match, it takes the reference and releases the lock. If it doesn't find a match, it causes the open to return -ENODEV.
There were some added files that did not take the trace_array reference on open that needed to be fixed. Sashiko also correctly pointed out that there were some files that took a reference to an field or element of the trace_array which had a pointer back to the trace_array to take its reference on open. But this leaves a slight race between referencing this element to get the trace_array as the element itself could be freed. To solve this, some helper functions were created to look for trace_arrays with this field or element in the search so that the element did not have to be dereferenced before the trace_array's reference was taken.
- Add a lock around ftrace_ops initialization
When a ftrace_ops is first used by ftrace, some internal initialization is performed on the ops. But if multiple tasks were calling functions that did this initialization, it could race and perform doing the initialization more than once, corrupting the internal data. Add a lock in the initialization code to prevent this from happening.
- Fix splice reads on mmapped buffers
The logic in the ring buffer splice code for mmapped buffers is supposed to do a copy of the memory as the mapped buffers can't be given to splice. But there was an if statement within the copy code that would return a -1 if a request for a full page was done and it wasn't a partial read. This is because this logic was written before mmapped buffers existed and this case didn't make sense at the time. For mmapped buffers it makes perfect sense and by returning early can drop a lot of pages unnecessarily.
- Have the persistent ring buffer validation check nr_subbufs
Sashiko reported that the validation code was relying on the saved nr_subbufs to match the calculated nr_pages + 1 and if they were off, that the code could cause corruption. Sashiko is correct, and the saved nr_subbufs should be validated before assuming it is correct.
- Do not allow more than one instance with the same name on cmdline
If an admin were to add more than one trace instances with the same name they all would be created, but only the first one would be accessible via tracefs. This used to not be allowed but some restructuring of code has since made it possible.
- Fix the race between subbuf resize and trace_pipe_raw readers
If a task was reading trace_pipe_raw while another task was changing the ring buffer subbuf size, it could crash the reader. The trace_pipe_raw readers do get their own copy of the page from the buffer, but the code needs some restructuring to not have the resize of the subbuffers cause issues.
- Cap the size of the mapped (static) ring buffer nr_pages
The meta data used for ring buffer mapped buffers is 32 bit in size. A normal ring buffer could (in theory) have more than 4 billion pages. But this is not allowed by mapped buffers, so enforce it.
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git
trace/fixes
Head SHA1: d80e12156f1fd490adf29a8d28489725a3ac817a
Masami Hiramatsu (Google) (1):
tracing: Fix to avoid creating trace instances with duplicate names
Steven Rostedt (7):
tracing: Have show_event_filters/triggers files take trace array ref
ftrace: Take trace_array reference before accessing its ftrace_ops
ftrace: Synchronize the initialization of ftrace_ops
tracing: Take trace_array reference when opening options file
ring-buffer: Add checking nr_subbufs to persistent ring buffer validation
tracing: Fix comment in tracing_buffers_splice_read()
ring-buffer: Use a macro for static buffer bits
Vincent Donnefort (4):
ring-buffer: Allow splice reads on static buffers
tracing: Fix subbuf resize races with trace_pipe_raw readers
ring-buffer: Cap static ring buffer nr_pages
ring-buffer: Prevent truncation of nr_pages / nr_subbufs
----
include/linux/ftrace.h | 5 +- include/linux/ring_buffer.h | 5 +- kernel/trace/ftrace.c | 70 ++++++---- kernel/trace/ring_buffer.c | 239 +++++++++++++++++++++++------------ kernel/trace/ring_buffer_benchmark.c | 6 +- kernel/trace/trace.c | 171 ++++++++++++++++--------- kernel/trace/trace.h | 14 +- kernel/trace/trace_events.c | 28 +++- kernel/trace/trace_functions.c | 2 +- kernel/trace/trace_stack.c | 2 +- 10 files changed, 357 insertions(+), 185 deletions(-)
PATCH 1/12 tracing: Have show_event_filters/triggers files take trace array ref
From: Steven Rostedt <[email protected]>
The newly added files show_event_filters and show_event_triggers that show all filters or triggers that are set within the trace array do not take a reference for the trace array it is showing. Without taking a reference, the trace_array may be freed via "rmdir" while a task is reading one of theses files. Those files iterate all the events within an instance (trace_array) and nothing prevents that instance from being freed while its data is being read. This causes a use-after-free crash.
Have the open of both those files take the trace_array reference via the trace_array_get() that prevents the trace_array from being freed while the files are opened.
--- kernel/trace/trace_events.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
kernel/trace/trace_events.c +24 -4
@@ -2736,14 +2736,14 @@ static const struct file_operations ftrace_show_event_filters_fops = { .open = ftrace_event_show_filters_open, .read = seq_read, .llseek = seq_lseek, - .release = seq_release, + .release = ftrace_event_release, };static const struct file_operations ftrace_show_event_triggers_fops = { .open = ftrace_event_show_triggers_open, .read = seq_read, .llseek = seq_lseek, - .release = seq_release, + .release = ftrace_event_release, };static const struct file_operations ftrace_set_event_pid_fops = {
@@ -2908,7 +2908,17 @@ ftrace_event_set_open(struct inode *inode, struct file *file)static int ftrace_event_show_filters_open (struct inode *inode,struct file *file) { -return ftrace_event_open (inode, file, &show_show_event_filters_seq_ops); +struct trace_array *tr = inode->i_private; +int ret; + + ret =tracing_check_open_get_tr (tr); +if (ret) +return ret; + + ret =ftrace_event_open (inode, file, &show_show_event_filters_seq_ops); +if (ret <0 ) +trace_array_put (tr); +return ret; }/**
@@ -2922,7 +2932,17 @@ ftrace_event_show_filters_open(struct inode *inode, struct file *file)static int ftrace_event_show_triggers_open (struct inode *inode,struct file *file) { -return ftrace_event_open (inode, file, &show_show_event_triggers_seq_ops); +struct trace_array *tr = inode->i_private; +int ret; + + ret =tracing_check_open_get_tr (tr); +if (ret) +return ret; + + ret =ftrace_event_open (inode, file, &show_show_event_triggers_seq_ops); +if (ret <0 ) +trace_array_put (tr); +return ret; } static int
Signature · 1 line
2.53.0
PATCH 2/12 ftrace: Take trace_array reference before accessing its ftrace_ops
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The trace instance files set_ftrace_filter and set_ftrace_notrace was updated to work with specific trace instances (trace_arrays). The issue is that when these files are opened, there is a small race window where it will use the ftrace_ops from the inode->private pointer to get a reference to the trace_array and then take its reference. The problem is that the ftrace_ops itself could be freed. If the rmdir on the instance happens at the same time the set_ftrace_filter file is opened, the rmdir could have also freed the ftrace_ops and referencing it will cause a use-after-free bug and crash the kernel.
Instead, pass in the trace_array as the file private data (NULL for the top level instance), and then pass both the trace_array and the ftrace_ops to the ftrace_regex_open() function. If the trace_array is NULL, then it just uses the ftrace_ops without the need to take its reference (like normal). If the ftrace_ops is NULL, that is only the case for the top level instance and the global_ops can be used.
This allows the trace_array to have its reference incremented before touching the ftrace_ops that could also be freed when the instance is.
--- include/linux/ftrace.h | 5 +-- kernel/trace/ftrace.c | 57 ++++++++++++++++++++++------------ kernel/trace/trace.h | 5 +-- kernel/trace/trace_functions.c | 2 +- kernel/trace/trace_stack.c | 2 +- 5 files changed, 45 insertions(+), 26 deletions(-)
include/linux/ftrace.h +3 -2
@@ -866,8 +866,9 @@ unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec);unsigned long ftrace_get_addr_curr (struct dyn_ftrace *rec);extern ftrace_func_t ftrace_trace_function; +struct trace_array ; -int ftrace_regex_open (struct ftrace_ops *ops,int flag, +int ftrace_regex_open (struct trace_array *tr,struct ftrace_ops *ops,int flag,struct inode *inode,struct file *file);ssize_t ftrace_filter_write (struct file *file,const char __user *ubuf,size_t cnt,loff_t *ppos);
@@ -1077,7 +1078,7 @@ static inline unsigned long ftrace_location(unsigned long ip)* have them defined when ftrace is not enabled, but these * functions may still be called. Use a macro instead of inline. */ -#define ftrace_regex_open (ops, flag, inod, file) ({ -ENODEV; }) +#define ftrace_regex_open (tr, ops, flag, inode, file) ({ -ENODEV; })#define ftrace_set_early_filter (ops, buf, enable) do { } while (0)#define ftrace_set_filter_ip (ops, ip, remove, reset) ({ -ENODEV; })#define ftrace_set_filter_ips (ops, ips, cnt, remove, reset) ({ -ENODEV; })
kernel/trace/ftrace.c +37 -20
120 lines not on this page · raw message
kernel/trace/trace.h +3 -2
@@ -1340,7 +1340,7 @@ extern void clear_ftrace_function_probes(struct trace_array *tr);int register_ftrace_command (struct ftrace_func_command *cmd);int unregister_ftrace_command (struct ftrace_func_command *cmd); -void ftrace_create_filter_files (struct ftrace_ops *ops, +void ftrace_create_filter_files (struct trace_array *tr,struct dentry *parent);void ftrace_destroy_filter_files (struct ftrace_ops *ops);
@@ -1363,11 +1363,12 @@ static inline void clear_ftrace_function_probes(struct trace_array *tr) { } +static inline void ftrace_create_filter_files (struct trace_array *tr, +struct dentry *parent) { }/* * The ops parameter passed in is usually undefined. * This must be a macro. */ -#define ftrace_create_filter_files (ops, parent) do { } while (0)#define ftrace_destroy_filter_files (ops) do { } while (0)#endif /* CONFIG_FUNCTION_TRACER && CONFIG_DYNAMIC_FTRACE */
kernel/trace/trace_functions.c +1 -1
@@ -101,7 +101,7 @@ int ftrace_create_function_files(struct trace_array *tr,return ret; } -ftrace_create_filter_files (tr->ops, parent); +ftrace_create_filter_files (tr, parent);return 0 ; }
kernel/trace/trace_stack.c +1 -1
@@ -499,7 +499,7 @@ stack_trace_filter_open(struct inode *inode, struct file *file)struct ftrace_ops *ops = inode->i_private;/* Checks for tracefs lockdown */ -return ftrace_regex_open (ops,FTRACE_ITER_FILTER , +return ftrace_regex_open (NULL , ops,FTRACE_ITER_FILTER , inode, file); }
Signature · 1 line
2.53.0
PATCH 3/12 ftrace: Synchronize the initialization of ftrace_ops
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
There's some internal state that ftrace_ops needs to have set, but since it can be declared outside of the ftrace.c code, it calls ftrace_ops_init() on the ops in every global function. The issue is that if two tasks call it on the same ops at the same time it is possible to have the initialization of one corrupt the initialization of the other call.
Create a ops_mutex to use to synchronize every initialization of the ftrace_ops. The mutex is taken within checking the ftrace_ops flag that states it was initializied but the flag is checked again after the mutex has been taken. Checking first outside the mutex allows it to shortcut having to take the mutex. But then the check needs to be done again after the mute is taken in case of races.
Close: https://lore.kernel.org/all/[email protected]/
--- kernel/trace/ftrace.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
kernel/trace/ftrace.c +11 -2
@@ -75,6 +75,8 @@ .func_hash = &opsname.local_hash, \ .local_hash.regex_lock = __MUTEX_INITIALIZER(opsname.local_hash.regex_lock), \ .subop_list = LIST_HEAD_INIT(opsname.subop_list), +/* Used only to synchronize the initialization of ftrace_ops */ +static DEFINE_MUTEX(ops_mutex);#else #define INIT_OPS_HASH (opsname)#endif
@@ -159,11 +161,18 @@ const struct ftrace_ops ftrace_nop_ops = {static inline void ftrace_ops_init (struct ftrace_ops *ops) {#ifdef CONFIG_DYNAMIC_FTRACE -if (!(ops->flags &FTRACE_OPS_FL_INITIALIZED )) { +unsigned long flags =smp_load_acquire (&ops->flags); + +if (!(flags &FTRACE_OPS_FL_INITIALIZED )) { +guard (mutex)(&ops_mutex); +/* Could have been initialized before lock taken */ +if (unlikely (ops->flags &FTRACE_OPS_FL_INITIALIZED )) +return ;mutex_init (&ops->local_hash.regex_lock);INIT_LIST_HEAD (&ops->subop_list); ops->func_hash = &ops->local_hash; - ops->flags |=FTRACE_OPS_FL_INITIALIZED ; + flags = ops->flags |FTRACE_OPS_FL_INITIALIZED ; +smp_store_release (&ops->flags, flags); }#endif }
Signature · 1 line
2.53.0
PATCH 4/12 tracing: Take trace_array reference when opening options file
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The options files do not take the trace_array reference for the options they represent. This could cause a use-after-free kernel crash if one of these files is opened by one task and another task removes the instance that the option is for. Because it doesn't take a reference upon opening, it will not stop the removal which will free the options descriptor that is being used.
As the options are somewhat dynamic in their creation at boot up, each file represents a flag in the trace_array. The trace_array has an array of indexes to represent each of these flags that is stored in the trace_flags_index array. The address of the index array element is used to pass to the inode->i_private pointer. Then that element is read which holds the index (which represents the flag) and then the index is used to calculate the trace_array descriptor from its trace_flags_index array.
One issue is that the index element can not be referenced until the trace_array's reference is taken. To handle this, create a new helper function called: trace_array_options_get() that will iterate all the existing trace_arrays in the ftrace_trace_arrays list (under the trace_types_lock), and compare the passed in address of the index element with the entire array of the trace_array's trace_flags_index array. If it matches, then up the corresponding trace_array's reference and return.
--- kernel/trace/trace.c | 67 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 63 insertions(+), 4 deletions(-)
kernel/trace/trace.c +63 -4
@@ -7842,11 +7842,70 @@ trace_options_core_write(struct file *filp, const char __user *ubuf, size_t cnt,return cnt; } +/* +* The tr_index is the address of a trace_array->trace_flags_index[] +* element that holds the index of the trace flag. But since the +* trace_array reference has not been taken yet, it cannot be referenced +* as it could have been freed by a rmdir of the instance the trace_array +* represents. +* +* Search the list of trace_arrays and compare the tr_index to the +* address of the entire trace_array trace_flags_index array for each +* trace_array in the list. If one is matched, then take the reference +* and return it. If not, the trace_array no longer exits. +*/ +static int trace_array_options_get (void *tr_index) +{ +struct trace_array *tr; +int ret; + + ret =security_locked_down (LOCKDOWN_TRACEFS ); +if (ret) +return ret; + +if (tracing_disabled) +return -ENODEV ; + +guard (mutex)(&trace_types_lock); +list_for_each_entry (tr, &ftrace_trace_arrays, list) { +if (tr_index >= (void *)&tr->trace_flags_index[0 ] && + tr_index < (void *)&tr->trace_flags_index[TRACE_FLAGS_MAX_SIZE ]) +return __trace_array_get (tr); + } +return -ENODEV ; +} + +static int trace_options_open (struct inode *inode,struct file *filp) +{ +void *tr_index = inode->i_private; + +if (trace_array_options_get (tr_index) <0 ) +return -ENODEV ; + + filp->private_data = tr_index; + +return 0 ; +} + +static int trace_options_release (struct inode *inode,struct file *filp) +{ +void *tr_index = filp->private_data; +struct trace_array *tr; +unsigned int index; + +get_tr_index (tr_index, &tr, &index); + +trace_array_put (tr); + +return 0 ; +} +static const struct file_operations trace_options_core_fops = { - .open = tracing_open_generic, - .read = trace_options_core_read, - .write = trace_options_core_write, - .llseek = generic_file_llseek, + .open = trace_options_open, + .read = trace_options_core_read, + .write = trace_options_core_write, + .llseek = generic_file_llseek, + .release = trace_options_release, }; struct dentry *trace_create_file(const char *name,
Signature · 1 line
2.53.0
PATCH 5/12 ring-buffer: Allow splice reads on static buffers
In reply to Steven RostedtFrom: Vincent Donnefort <[email protected]>
ring_buffer_read_page() rejects splice (full=1) reads on static buffers (that is user-mapped, persistent or remote) because !read check assumes unread pages must be swapped. However for those buffers we have no other choice than memcpy the data.
For the memcpy case, only return an error when the writer is still on the reader page for the splice interface to wait.
--- kernel/trace/ring_buffer.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-)
kernel/trace/ring_buffer.c +2 -9
@@ -7193,15 +7193,8 @@ int ring_buffer_read_page(struct trace_buffer *buffer,unsigned int event_size;unsigned int flags =0 ; -/* -* If a full page is expected, this can still be returned -* if there's been a previous partial read and the -* rest of the page can be read and the commit page is off -* the reader page. -*/ -if (full && - (!read || (len < (size - read)) || - cpu_buffer->reader_page == cpu_buffer->commit_page)) +/* If a full page is requested, it cannot be the commit page */ +if (full && cpu_buffer->reader_page == cpu_buffer->commit_page)return -1 ;if (len > (size - read))
Signature · 1 line
2.53.0
PATCH 6/12 ring-buffer: Add checking nr_subbufs to persistent ring buffer validation
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Sashiko reported that the code was using meta->nr_subbufs without making sure that it matched the nr_pages + 1 on data that was assuming the two were the same.
Add a check to the persistent ring buffer validation code to make sure that the saved nr_subbufs matches what we expect.
--- kernel/trace/ring_buffer.c | 5 +++++ 1 file changed, 5 insertions(+)
kernel/trace/ring_buffer.c +5 -0
@@ -1856,6 +1856,11 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,return false; } +if (meta->nr_subbufs != nr_pages +1 ) { +pr_info ("Ring buffer boot meta [%d] invalid nr_subbufs\n" , cpu); +return false; + } + buffers_start = meta->first_buffer; buffers_end = meta->first_buffer + (subbuf_size * meta->nr_subbufs);
Signature · 1 line
2.53.0
PATCH 7/12 tracing: Fix to avoid creating trace instances with duplicate names
In reply to Steven RostedtFrom: "Masami Hiramatsu (Google)" <[email protected]>
Since commit e645535a954a ("tracing: Add option to use memmapped memory for trace boot instance") changed trace_array_get_by_name() to trace_array_create_systems(), enable_instances() does not reuse the same name instance. Therefore, if an administrator mistakenly specifies multiple trace_instance= options with duplicate names, all are created but only the first is accessible via tracefs.
Check whether an instance with the same name already exists before creating a new one, and reject duplicates with a warning.
--- kernel/trace/trace.c | 5 +++++ 1 file changed, 5 insertions(+)
kernel/trace/trace.c +5 -0
@@ -9710,6 +9710,11 @@ __init static void enable_instances(void)if (flag_delim) *flag_delim++ ='\0' ; +if (trace_array_find (name)) { +pr_warn ("Tracing: Instance %s already exists\n" , name); +continue ; + } +if (backup) {if (backup_instance_area (backup, &addr, &size) <0 )continue ;
Signature · 1 line
2.53.0
PATCH 8/12 tracing: Fix subbuf resize races with trace_pipe_raw readers
In reply to Steven RostedtFrom: Vincent Donnefort <[email protected]>
Concurrent subbuffer resizes may crash trace_pipe_raw readers or leak uninitialized memory to userspace due to stale size values.
Modify ring_buffer_alloc_read_page() to handle the resizing of an existing buffer_data_read_page if necessary and add a new ring_buffer_read_page_size(). This new function enables ring-buffer buffer_data_read_page users to not call the racy ring_buffer_subbuf_size_get(). This makes the spare_size member of ftrace_buffer_info redundant.
Finally, handle buffer_data_read_page/reader_page order discrepancy in ring_buffer_read_page(). On a mismatch simply copy manually the data to the buffer_data_read_page.
--- include/linux/ring_buffer.h | 5 +- kernel/trace/ring_buffer.c | 135 ++++++++++++++++++--------- kernel/trace/ring_buffer_benchmark.c | 6 +- kernel/trace/trace.c | 97 +++++++++---------- kernel/trace/trace.h | 9 +- 5 files changed, 144 insertions(+), 108 deletions(-)
include/linux/ring_buffer.h +3 -2
@@ -218,14 +218,15 @@ bool ring_buffer_time_stamp_abs(struct trace_buffer *buffer);size_t ring_buffer_nr_dirty_pages (struct trace_buffer *buffer,int cpu);struct buffer_data_read_page ; -struct buffer_data_read_page * -ring_buffer_alloc_read_page (struct trace_buffer *buffer,int cpu); +int ring_buffer_alloc_read_page (struct trace_buffer *buffer,int cpu, +struct buffer_data_read_page **rpage);void ring_buffer_free_read_page (struct trace_buffer *buffer,int cpu,struct buffer_data_read_page *page);int ring_buffer_read_page (struct trace_buffer *buffer,struct buffer_data_read_page *data_page,size_t len,int cpu,int full);void *ring_buffer_read_page_data (struct buffer_data_read_page *page); +unsigned int ring_buffer_read_page_size (struct buffer_data_read_page *rpage);struct trace_seq ;
kernel/trace/ring_buffer.c +92 -43
258 lines not on this page · raw message
kernel/trace/ring_buffer_benchmark.c +3 -3
@@ -104,7 +104,7 @@ static enum event_status read_event(int cpu)static enum event_status read_page (int cpu) { -struct buffer_data_read_page *bpage; +struct buffer_data_read_page *bpage =NULL ;struct ring_buffer_event *event;struct rb_page *rpage;unsigned long commit;
@@ -114,8 +114,8 @@ static enum event_status read_page(int cpu)int inc;int i; - bpage =ring_buffer_alloc_read_page (buffer, cpu); -if (IS_ERR (bpage)) + ret =ring_buffer_alloc_read_page (buffer, cpu, &bpage); +if (ret <0 )return EVENT_DROPPED ; page_size =ring_buffer_subbuf_size_get (buffer);
kernel/trace/trace.c +42 -55
170 lines not on this page · raw message
kernel/trace/trace.h +4 -5
@@ -745,11 +745,10 @@ static inline int tracing_get_cpu(struct inode *inode)void tracing_reset_cpu (struct array_buffer *buf,int cpu);struct ftrace_buffer_info { -struct trace_iterator iter; -void *spare; -unsigned int spare_cpu; -unsigned int spare_size; -unsigned int read; +struct trace_iterator iter; +struct buffer_data_read_page *spare; +unsigned int spare_cpu; +unsigned int read; };/**
Signature · 1 line
2.53.0
PATCH 9/12 ring-buffer: Cap static ring buffer nr_pages
In reply to Steven RostedtFrom: Vincent Donnefort <[email protected]>
Static ring buffers (i.e. persistent, user-mapped and remote) rely on the bpage::id field. The number of pages for those ring buffers must fit into that variable. Enforce this limit on ring buffer creation or user-mapping.
While at it, prevent nr_pages underflow when allocating a persistent buffer.
--- kernel/trace/ring_buffer.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
kernel/trace/ring_buffer.c +22 -0
@@ -657,6 +657,15 @@ static bool rb_is_static(struct ring_buffer_per_cpu *cpu_buffer)return cpu_buffer->user_mapped || cpu_buffer->remote || cpu_buffer->ring_meta; } +static unsigned long rb_static_max_pages (void ) +{ +/* +* Static ring buffers are using bpage::id and must account for the +* reader page. +*/ +return (1UL <<30 ) -1 ; +} +struct ring_buffer_iter {struct ring_buffer_per_cpu *cpu_buffer;unsigned long head;
@@ -2838,6 +2847,8 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags, size = end - buffers_start; size = size / nr_cpu_ids; +if (size <sizeof (struct ring_buffer_cpu_meta )) + goto fail_free_buffers;/* * The number of sub-buffers (nr_pages) is determined by the * total size allocated minus the meta data size.
@@ -2847,6 +2858,10 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,*/ nr_pages = (size -sizeof (struct ring_buffer_cpu_meta )) / (subbuf_size +sizeof (int )); + +if (nr_pages >rb_static_max_pages ()) + goto fail_free_buffers; +/* Need at least two pages plus the reader page */ if (nr_pages <3 ) goto fail_free_buffers;
@@ -2879,6 +2894,10 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,/* The writer is remote. This ring-buffer is read-only */ atomic_inc (&buffer->record_disabled); nr_pages = desc->nr_page_va -1 ; + +if (nr_pages >rb_static_max_pages ()) + goto fail_free_buffers; +if (nr_pages <2 ) goto fail_free_buffers; }else {
@@ -7841,6 +7860,9 @@ int ring_buffer_map(struct trace_buffer *buffer, int cpu,/* prevent another thread from changing buffer/sub-buffer sizes */ guard (mutex)(&buffer->mutex); +if (cpu_buffer->nr_pages >rb_static_max_pages ()) +return -E2BIG ; + err =rb_alloc_meta_page (cpu_buffer);if (err)return err;
Signature · 1 line
2.53.0
PATCH 10/12 ring-buffer: Prevent truncation of nr_pages / nr_subbufs
In reply to Steven RostedtFrom: Vincent Donnefort <[email protected]>
Although ring_buffer_per_cpu::nr_pages is defined as unsigned long, it is capped to 32-bits in a few places, limiting the operations possible on a very large buffer. Use unsigned long where appropriate and prevent truncation of values using nr_pages (or nr_subbufs).
While at it, subbuf_size must be at least unsigned int.
Note that persistent, remote and user-mapped ring buffers are capping the number of pages to 30 bits already, making "int" safe in many places.
--- kernel/trace/ring_buffer.c | 61 +++++++++++++++++++------------------- 1 file changed, 30 insertions(+), 31 deletions(-)
kernel/trace/ring_buffer.c +30 -31
@@ -1683,7 +1683,7 @@ static void rb_check_pages(struct ring_buffer_per_cpu *cpu_buffer)* This is used to help find the next per cpu subbuffer within a mapped range. */ static unsigned long -rb_range_align_subbuf (unsigned long addr,int subbuf_size,int nr_subbufs) +rb_range_align_subbuf (unsigned long addr,unsigned int subbuf_size,unsigned long nr_subbufs) { addr +=sizeof (struct ring_buffer_cpu_meta ) +sizeof (int ) * nr_subbufs;
@@ -1693,13 +1693,12 @@ rb_range_align_subbuf(unsigned long addr, int subbuf_size, int nr_subbufs)/* * Return the ring_buffer_meta for a given @cpu. */ -static void *rb_range_meta (struct trace_buffer *buffer,int nr_pages,int cpu) +static void *rb_range_meta (struct trace_buffer *buffer,unsigned long nr_pages,int cpu) { -int subbuf_size =rb_subbuf_size (buffer); +unsigned int subbuf_size =rb_subbuf_size (buffer);struct ring_buffer_cpu_meta *meta;struct ring_buffer_meta *bmeta; -unsigned long ptr; -int nr_subbufs; +unsigned long ptr, nr_subbufs; bmeta = buffer->meta;if (!bmeta)
@@ -1745,7 +1744,7 @@ static void *rb_range_meta(struct trace_buffer *buffer, int nr_pages, int cpu)/* Return the start of subbufs given the meta pointer */ static void *rb_subbufs_from_meta (struct ring_buffer_cpu_meta *meta) { -int subbuf_size = meta->subbuf_size; +unsigned int subbuf_size = meta->subbuf_size;unsigned long ptr; ptr = (unsigned long )meta;
@@ -1757,11 +1756,11 @@ static void *rb_subbufs_from_meta(struct ring_buffer_cpu_meta *meta)/* * Return a specific sub-buffer for a given @cpu defined by @idx. */ -static void *rb_range_buffer (struct ring_buffer_per_cpu *cpu_buffer,int idx) +static void *rb_range_buffer (struct ring_buffer_per_cpu *cpu_buffer,unsigned long idx) {struct ring_buffer_cpu_meta *meta; +unsigned int subbuf_size;unsigned long ptr; -int subbuf_size; meta =rb_range_meta (cpu_buffer->buffer,0 , cpu_buffer->cpu);if (!meta)
@@ -1777,7 +1776,7 @@ static void *rb_range_buffer(struct ring_buffer_per_cpu *cpu_buffer, int idx) ptr = (unsigned long )rb_subbufs_from_meta (meta); - ptr += subbuf_size * idx; + ptr += (unsigned long )subbuf_size * idx;if (ptr + subbuf_size > cpu_buffer->buffer->range_addr_end)return NULL ;
@@ -1854,13 +1853,12 @@ static bool rb_meta_init(struct trace_buffer *buffer, int scratch_size)* must be the same. */ static bool rb_cpu_meta_valid (struct ring_buffer_cpu_meta *meta,int cpu, -struct trace_buffer *buffer,int nr_pages, +struct trace_buffer *buffer,unsigned long nr_pages,unsigned long *subbuf_mask) { -int subbuf_size =PAGE_SIZE ;unsigned long buffers_start;unsigned long buffers_end; -int i; +unsigned long i;if (!subbuf_mask)return false;
@@ -1876,7 +1874,7 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu, } buffers_start = meta->first_buffer; - buffers_end = meta->first_buffer + (subbuf_size * meta->nr_subbufs); + buffers_end = meta->first_buffer + (meta->nr_subbufs *PAGE_SIZE );/* Is the head and commit buffers within the range of buffers? */ if (meta->head_buffer < buffers_start ||
@@ -2114,8 +2112,8 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)struct buffer_page *head_page, *orig_head, *orig_reader;struct rb_validation_state state = {0 };bool skip = false; +unsigned long i;int ret; -int i;if (!meta || !meta->head_buffer)return ;
@@ -2166,7 +2164,7 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)rb_validate_buffer (head_page, cpu_buffer, meta, &state,0 , state.ts); }if (i) -pr_info ("Ring buffer [%d] rewound %d pages\n" , cpu_buffer->cpu, i); +pr_info ("Ring buffer [%d] rewound %lu pages\n" , cpu_buffer->cpu, i);/* The last rewound page must be skipped. */ if (head_page != orig_head)
@@ -2250,7 +2248,8 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer) } } -static void rb_range_meta_init (struct trace_buffer *buffer,int nr_pages,int scratch_size) +static void rb_range_meta_init (struct trace_buffer *buffer,unsigned long nr_pages, +int scratch_size) {struct ring_buffer_cpu_meta *meta;unsigned long *subbuf_mask;
@@ -2350,8 +2349,8 @@ static int rbm_show(struct seq_file *m, void *v) rb_meta_subbuf_idx(meta, (void *)meta->head_buffer));seq_printf (m,"commit_buffer: %d\n" ,rb_meta_subbuf_idx (meta, (void *)meta->commit_buffer)); -seq_printf (m,"subbuf_size: %d\n" , meta->subbuf_size); -seq_printf (m,"nr_subbufs: %d\n" , meta->nr_subbufs); +seq_printf (m,"subbuf_size: %u\n" , meta->subbuf_size); +seq_printf (m,"nr_subbufs: %u\n" , meta->nr_subbufs);return 0 ; }
@@ -2436,7 +2435,7 @@ static void *ring_buffer_desc_page(struct ring_buffer_desc *desc, unsigned int p }static int __rb_allocate_pages (struct ring_buffer_per_cpu *cpu_buffer, -long nr_pages,struct list_head *pages) +unsigned long nr_pages,struct list_head *pages) {struct trace_buffer *buffer = cpu_buffer->buffer;struct ring_buffer_cpu_meta *meta =NULL ;
@@ -2564,7 +2563,7 @@ static int rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer, }static struct ring_buffer_per_cpu * -rb_allocate_cpu_buffer (struct trace_buffer *buffer,long nr_pages,int cpu) +rb_allocate_cpu_buffer (struct trace_buffer *buffer,unsigned long nr_pages,int cpu) { struct ring_buffer_per_cpu *cpu_buffer __free(kfree) =alloc_cpu_buffer (cpu);
@@ -2721,8 +2720,8 @@ static void rb_test_inject_invalid_pages(struct trace_buffer *buffer)struct ring_buffer_cpu_meta *meta;struct buffer_data_page *dpage;unsigned long entry_bytes =0 ; +unsigned int subbuf_size;unsigned long ptr; -int subbuf_size;int invalid =0 ;int cpu;int i;
@@ -2792,8 +2791,8 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags, struct ring_buffer_remote *remote) { struct trace_buffer *buffer __free(kfree) = NULL; -long nr_pages; -int subbuf_size; +unsigned int subbuf_size; +unsigned long nr_pages;int bsize;int cpu;int ret;
@@ -5882,12 +5881,12 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)static struct buffer_page *__rb_get_reader_page (struct ring_buffer_per_cpu *cpu_buffer) { -int max_loops = cpu_buffer->ring_meta ? cpu_buffer->nr_pages :3 ; +unsigned long max_loops = cpu_buffer->ring_meta ? cpu_buffer->nr_pages :3 ;struct buffer_page *reader =NULL ; +unsigned long nr_loops =0 ;unsigned long overwrite;unsigned long flags;int missed_events =0 ; -int nr_loops =0 ;bool ret;local_irq_save (flags);
@@ -6205,8 +6204,8 @@ rb_iter_peek(struct ring_buffer_iter *iter, u64 *ts)struct trace_buffer *buffer;struct ring_buffer_per_cpu *cpu_buffer;struct ring_buffer_event *event; -int nr_loops =0 ; -int max_loops; +unsigned long nr_loops =0 ; +unsigned long max_loops;if (ts) *ts =0 ;
@@ -7446,8 +7445,8 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)struct ring_buffer_per_cpu *cpu_buffer;struct buffer_page *bpage, *tmp;unsigned int old_capacity; +unsigned long nr_pages;int old_order; -int nr_pages;int psize;int err;int cpu;
@@ -7629,10 +7628,10 @@ static void rb_setup_ids_meta_page(struct ring_buffer_per_cpu *cpu_buffer, struct buffer_page **subbuf_ids) {struct trace_buffer_meta *meta = cpu_buffer->meta_page; -unsigned int nr_subbufs = cpu_buffer->nr_pages +1 ; +unsigned long nr_subbufs = cpu_buffer->nr_pages +1 ;struct buffer_page *first_subbuf, *subbuf; -int cnt =0 ; -int id =0 ; +unsigned int cnt =0 ; +unsigned int id =0 ; id =rb_page_id (cpu_buffer, cpu_buffer->reader_page, id); subbuf_ids[id++] = cpu_buffer->reader_page;
Signature · 1 line
2.53.0
PATCH 11/12 tracing: Fix comment in tracing_buffers_splice_read()
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The comment about returning an error if the read fails on the first iteration is slightly incorrect. It makes it sound like the only reason it could fail on a later iteration is if the subbuf order changed. That is incorrect, it could also fail if the length passed in was not a multiple of the subbuf size. Fix the comment.
--- kernel/trace/trace.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
kernel/trace/trace.c +7 -5
@@ -7296,11 +7296,13 @@ ssize_t tracing_buffers_splice_read(struct file *file, loff_t *ppos, r =ring_buffer_read_page (ref->buffer, ref->rpage, len, iter->cpu_file,1 ); }else if (!i) {/* -* We failed to read because the length is too small -* or unaligned. If this is the first iteration, it's -* an invalid userspace input. Otherwise, this is due -* to a subbuf order change. Do not report an error -* and just finish the read. +* If this fails to read on the first iteration, it +* means the length was too small and an error should +* be returned to user space. Otherwise, at least +* one sub-buffer was successfully read but this failed +* due to either the length was unaligned or the +* subbuf order changed. Either case, do not report +* an error. */ ret = -EINVAL ; }
Signature · 1 line
2.53.0
PATCH 12/12 ring-buffer: Use a macro for static buffer bits
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Instead of hard coding 30 for the number of bits used for the static buffer ids in two places, create a macro. This way if it changes in the future, it will change in all the locations that use it.
--- kernel/trace/ring_buffer.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/trace/ring_buffer.c +5 -2
@@ -335,6 +335,9 @@ static __always_inline unsigned int rb_read_page_capacity(struct buffer_data_reareturn (PAGE_SIZE << rpage->order) -BUF_PAGE_HDR_SIZE ; } +/* The number of bits for static buffer ids */ +#define RB_STATIC_BITS 30 +/* * Note, the buffer_page list must be first. The buffer pages * are allocated in cache lines, which means that each buffer
@@ -350,7 +353,7 @@ struct buffer_page {local_t entries;/* entries on this page */ unsigned long real_end;/* real end of data */ unsigned order;/* order of the page */ - u32 id:30; /* ID for external mapping */ + u32 id:RB_STATIC_BITS; /* ID for external mapping */ u32 range:1; /* Mapped via a range */struct buffer_data_page *page;/* Actual data page */ };
@@ -663,7 +666,7 @@ static unsigned long rb_static_max_pages(void)* Static ring buffers are using bpage::id and must account for the * reader page. */ -return (1UL <<30 ) -1 ; +return (1UL <<RB_STATIC_BITS ) -1 ; }struct ring_buffer_iter {
Signature · 1 line
2.53.0
Steven Rostedt
maintainerSorry for the duplicate emails. I upgrade my exim4 and something failed which prevented the original patches from going out. I fixed the issue and resent. What I didn't realize was that the original patches were just queued, and when I fixed the issue, it sent the original patches too.
-- Steve