ring-buffer: Updates for 7.4
Subsystems (3)
cover ring-buffer: Updates for 7.4
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git
ring-buffer/for-next
Head SHA1: 453d0f47b174cac387af48d303c503b84f82212e
David Carlier (1):
ring-buffer: Take cpus_read_lock() when changing the subbuf order
Steven Rostedt (1):
ring-buffer: Add a MAINTAINERS category specific for the tracing ring-buffer
Vincent Donnefort (3):
ring-buffer: Unify ring buffer minimum page calculations
ring-buffer: Clean up resize_disabled checks
ring-buffer: Remove unnecessary cpumask_test_cpu
----
MAINTAINERS | 16 ++++++ include/linux/ring_buffer.h | 9 +++- kernel/trace/ring_buffer.c | 102 ++++++++++++++++---------------------- kernel/trace/simple_ring_buffer.c | 4 +- 4 files changed, 68 insertions(+), 63 deletions(-)
PATCH 1/5 ring-buffer: Add a MAINTAINERS category specific for the tracing ring-buffer
From: Steven Rostedt <[email protected]>
The ring buffer code in the tracing directory has always been decoupled from the tracing code. It had a clean abstracted API to allow it to be used by systems outside of tracing. The API even allowed the ring buffer code to be drastically changed without affecting the tracing sub-system.
Recently, Vincent Donnefort has been contributing a lot of work to enhance the ring buffer. He implemented the user space mappings and recently created a version to allow pKVM hypervisors to use the code to trace the hypervisor and expose the events to the user.
Create a separate category for maintaining the ring buffer code so that Vincent can become an official maintainer of it. His work has little to do with the tracing code itself. In the future, he may even be the main maintainer of the ring buffer.
--- MAINTAINERS | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
MAINTAINERS +16 -0
@@ -27794,6 +27794,22 @@ F: kernel/trace/trace_irqsoff.c F: kernel/trace/trace_osnoise.c F: kernel/trace/trace_sched_wakeup.c +TRACING RING BUFFER +M: Steven Rostedt <[email protected]> +M: Vincent Donnefort <[email protected]> +M: Masami Hiramatsu <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +Q: https://patchwork.kernel.org/project/linux-trace-kernel/list/ +T: git git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git +F: Documentation/trace/ring-buffer* +F: Documentation/trace/remotes.rst +F: include/linux/ring-buffer*.h +F: kernel/trace/*ring_buffer*.c +F: kernel/trace/remote*.c +F: tools/testing/selftests/ring-buffer/ + TRADITIONAL CHINESE DOCUMENTATION M: Chen-Yu Yeh <[email protected]> M: Weijie Yuan <[email protected]>
Signature · 1 line
2.53.0
PATCH 2/5 ring-buffer: Unify ring buffer minimum page calculations
In reply to Steven RostedtFrom: Vincent Donnefort <[email protected]>
The codebase scatters hardcoded minimum page limits across multiple places. Introduce RING_BUFFER_MIN_NR_PAGES and rb_calculate_nr_pages() to formalize the requirement of two pages in a ring buffer.
--- include/linux/ring_buffer.h | 9 ++++++++- kernel/trace/ring_buffer.c | 32 +++++++++++++------------------ kernel/trace/simple_ring_buffer.c | 4 ++-- 3 files changed, 23 insertions(+), 22 deletions(-)
include/linux/ring_buffer.h +8 -1
@@ -98,6 +98,13 @@ struct trace_buffer *__ring_buffer_alloc_range(unsigned long size, unsigned flagvoid *ring_buffer_meta_scratch (struct trace_buffer *buffer,unsigned int *size); +/* +* A minimum of 2 pages ensures we can always swap a page out of the ring +* buffer. This does not account for the reader page or any other page +* containing meta data used by static ring buffers. +*/ +#define RING_BUFFER_MIN_NR_PAGES 2UL +/* * Because the ring buffer is generic, if other users of the ring buffer get * traced by ftrace, it can produce lockdep warnings. We need to keep each
@@ -287,7 +294,7 @@ static inline struct ring_buffer_desc *__first_ring_buffer_desc(struct trace_bufstatic inline unsigned long __calc_nr_pages_ring_buffer_desc (size_t size) {/* Takes into account the reader page */ -return max (DIV_ROUND_UP (size,PAGE_SIZE -BUF_PAGE_HDR_SIZE ),2UL ) +1 ; +return max (DIV_ROUND_UP (size,PAGE_SIZE -BUF_PAGE_HDR_SIZE ),RING_BUFFER_MIN_NR_PAGES ) +1 ; }static inline size_t trace_buffer_desc_size (size_t buffer_size,unsigned int nr_cpus)
kernel/trace/ring_buffer.c +13 -19
@@ -642,6 +642,11 @@ static __always_inline unsigned int rb_subbuf_max_data_size(struct trace_bufferreturn rb_subbuf_capacity (buffer) -RB_EVNT_HDR_SIZE -sizeof (event->array[0 ]); } +static unsigned long rb_calculate_nr_pages (struct trace_buffer *buffer,unsigned long size) +{ +return max (DIV_ROUND_UP (size,rb_subbuf_capacity (buffer)),RING_BUFFER_MIN_NR_PAGES ); +} +/** * rb_subbuf_start - Get the start address of a subbuffer * @buffer: A trace buffer
@@ -2864,7 +2869,7 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags, goto fail_free_buffers;/* Need at least two pages plus the reader page */ -if (nr_pages <3 ) +if (nr_pages <RING_BUFFER_MIN_NR_PAGES +1 ) goto fail_free_buffers; again:
@@ -2876,7 +2881,7 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags, ptr += subbuf_size * nr_pages; }if (ptr > end) { -if (nr_pages <=3 ) +if (nr_pages <=RING_BUFFER_MIN_NR_PAGES +1 ) goto fail_free_buffers; nr_pages--; goto again;
@@ -2899,13 +2904,10 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,if (nr_pages >rb_static_max_pages ()) goto fail_free_buffers; -if (nr_pages <2 ) +if (nr_pages <RING_BUFFER_MIN_NR_PAGES ) goto fail_free_buffers; }else { -/* need at least two pages */ - nr_pages =DIV_ROUND_UP (size,rb_subbuf_capacity (buffer)); -if (nr_pages <2 ) - nr_pages =2 ; + nr_pages =rb_calculate_nr_pages (buffer, size); }cpumask_set_cpu (cpu, buffer->cpumask);
@@ -3318,11 +3320,7 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size,mutex_lock (&buffer->mutex);atomic_inc (&buffer->resizing); - nr_pages =DIV_ROUND_UP (size,rb_subbuf_capacity (buffer)); - -/* we need a minimum of two pages */ -if (nr_pages <2 ) - nr_pages =2 ; + nr_pages =rb_calculate_nr_pages (buffer, size);if (cpu_id ==RING_BUFFER_ALL_CPUS ) {/*
@@ -7502,12 +7500,8 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order) }/* Update the number of pages to match the new size */ - nr_pages = old_capacity * buffer->buffers[cpu]->nr_pages; - nr_pages =DIV_ROUND_UP (nr_pages,rb_subbuf_capacity (buffer)); - -/* we need a minimum of two pages */ -if (nr_pages <2 ) - nr_pages =2 ; + nr_pages =rb_calculate_nr_pages (buffer, + old_capacity * buffer->buffers[cpu]->nr_pages); cpu_buffer->nr_pages_to_update = nr_pages;
@@ -8102,7 +8096,7 @@ int trace_rb_cpu_prepare(unsigned int cpu, struct hlist_node *node) }/* allocate minimum pages, user can later expand it */ if (!nr_pages_same) - nr_pages =2 ; + nr_pages =RING_BUFFER_MIN_NR_PAGES ; buffer->buffers[cpu] =rb_allocate_cpu_buffer (buffer, nr_pages, cpu);if (!buffer->buffers[cpu]) {
kernel/trace/simple_ring_buffer.c +2 -2
@@ -383,8 +383,8 @@ int simple_ring_buffer_init_mm(struct simple_rb_per_cpu *cpu_buffer,void *page;int i; -/* At least 1 reader page and two pages in the ring-buffer */ -if (desc->nr_page_va <3 ) +/* At least RING_BUFFER_MIN_NR_PAGES and 1 reader page */ +if (desc->nr_page_va <RING_BUFFER_MIN_NR_PAGES +1 )return -EINVAL ;memset (cpu_buffer,0 ,sizeof (*cpu_buffer));
Signature · 1 line
2.53.0
PATCH 3/5 ring-buffer: Clean up resize_disabled checks
In reply to Steven RostedtFrom: Vincent Donnefort <[email protected]>
ring_buffer_subbuf_order_set() checks for resize_disabled twice under the same buffer->mutex hold. Moreover, this check duplicates the logic in ring_buffer_resize(). Create a common helper rb_resize_disabled() to factor out this code.
--- kernel/trace/ring_buffer.c | 63 +++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 35 deletions(-)
kernel/trace/ring_buffer.c +28 -35
@@ -3282,6 +3282,21 @@ static void update_pages_handler(struct work_struct *work)complete (&cpu_buffer->update_done); } +static bool rb_resize_disabled (struct trace_buffer *buffer,int cpu) +{ +lockdep_assert_held (&buffer->mutex); + +if (cpu !=RING_BUFFER_ALL_CPUS ) +return atomic_read (&buffer->buffers[cpu]->resize_disabled); + +for_each_buffer_cpu (buffer, cpu) { +if (atomic_read (&buffer->buffers[cpu]->resize_disabled)) +return true; + } + +return false; +} +/** * ring_buffer_resize - resize the ring buffer * @buffer: the buffer to resize.
@@ -3322,20 +3337,17 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size, nr_pages =rb_calculate_nr_pages (buffer, size); -if (cpu_id ==RING_BUFFER_ALL_CPUS ) { -/* -* Don't succeed if resizing is disabled, as a reader might be -* manipulating the ring buffer and is expecting a sane state while -* this is true. -*/ -for_each_buffer_cpu (buffer, cpu) { - cpu_buffer = buffer->buffers[cpu]; -if (atomic_read (&cpu_buffer->resize_disabled)) { - err = -EBUSY ; - goto out_err_unlock; - } - } +/* +* Don't succeed if resizing is disabled, as a reader might be +* manipulating the ring buffer and is expecting a sane state while +* this is true. +*/ +if (rb_resize_disabled (buffer, cpu_id)) { + err = -EBUSY ; + goto out_err_unlock; + } +if (cpu_id ==RING_BUFFER_ALL_CPUS ) {/* calculate the pages to update */ for_each_buffer_cpu (buffer, cpu) { cpu_buffer = buffer->buffers[cpu];
@@ -3407,16 +3419,6 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size,if (nr_pages == cpu_buffer->nr_pages) goto out; -/* -* Don't succeed if resizing is disabled, as a reader might be -* manipulating the ring buffer and is expecting a sane state while -* this is true. -*/ -if (atomic_read (&cpu_buffer->resize_disabled)) { - err = -EBUSY ; - goto out_err_unlock; - } - cpu_buffer->nr_pages_to_update = nr_pages - cpu_buffer->nr_pages;
@@ -7471,13 +7473,9 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order) old_capacity =rb_subbuf_capacity (buffer); -/* The mmap fast path reads subbuf_order without buffer->mutex. */ -for_each_buffer_cpu (buffer, cpu) { -if (!cpumask_test_cpu (cpu, buffer->cpumask)) -continue ; -if (atomic_read (&buffer->buffers[cpu]->resize_disabled)) -return -EBUSY ; - } +/* Check it is resizable before we touch subbuf_order */ +if (rb_resize_disabled (buffer,RING_BUFFER_ALL_CPUS )) +return -EBUSY ;atomic_inc (&buffer->record_disabled);
@@ -7494,11 +7492,6 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order) cpu_buffer = buffer->buffers[cpu]; -if (atomic_read (&cpu_buffer->resize_disabled)) { - err = -EBUSY ; - goto error; - } -/* Update the number of pages to match the new size */ nr_pages =rb_calculate_nr_pages (buffer, old_capacity * buffer->buffers[cpu]->nr_pages);
Signature · 1 line
2.53.0
PATCH 4/5 ring-buffer: Remove unnecessary cpumask_test_cpu
In reply to Steven RostedtFrom: Vincent Donnefort <[email protected]>
for_each_buffer_cpu() already iterates over buffer->cpumask. Remove the unnecessary cpumask_test_cpu() found in ring_buffer_subbuf_order_set().
--- kernel/trace/ring_buffer.c | 7 ------- 1 file changed, 7 deletions(-)
kernel/trace/ring_buffer.c +0 -7
@@ -7486,10 +7486,6 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)/* Make sure all new buffers are allocated, before deleting the old ones */ for_each_buffer_cpu (buffer, cpu) { - -if (!cpumask_test_cpu (cpu, buffer->cpumask)) -continue ; - cpu_buffer = buffer->buffers[cpu];/* Update the number of pages to match the new size */
@@ -7516,9 +7512,6 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)struct list_head old_pages;unsigned long flags; -if (!cpumask_test_cpu (cpu, buffer->cpumask)) -continue ; - cpu_buffer = buffer->buffers[cpu];raw_spin_lock_irqsave (&cpu_buffer->reader_lock, flags);
Signature · 1 line
2.53.0
PATCH 5/5 ring-buffer: Take cpus_read_lock() when changing the subbuf order
In reply to Steven RostedtFrom: David Carlier <[email protected]>
A CPU coming online for the first time gets its per CPU buffer allocated by trace_rb_cpu_prepare(), which then adds it to buffer->cpumask. If that happens while ring_buffer_subbuf_order_set() is running, the CPU shows up after the allocation loop and the install loop walks it with an empty new_pages list, corrupting the buffer.
Take cpus_read_lock() as ring_buffer_resize() does.
--- kernel/trace/ring_buffer.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/trace/ring_buffer.c +2 -0
@@ -7464,6 +7464,8 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)if (psize >RB_WRITE_MASK +1 )return -EINVAL ; +guard (cpus_read_lock)(); +/* prevent another thread from changing buffer sizes */ guard (mutex)(&buffer->mutex);
Signature · 1 line
2.53.0