ring-buffer: Allow persistent memory to be user space mmapped
Subsystems (3)
Steven Rostedt
maintainerv3 cover ring-buffer: Allow persistent memory to be user space mmapped
Now that I learned that the memory passed back from reserve_mem is part of the memory allocator and just "reserved" and the memory is already virtually mapped, it can simply use phys_to_virt() on the physical memory that is returned to get the virtual mapping for that memory!
(Thanks Mike!)
That makes things much easier, especially since it means that the memory returned by reserve_mem is no different than the memory retrieved by page_alloc(). This allows that memory to be memory mapped to user space no differently than it is mapped by the normal buffer.
This new series does the following:
- Enforce the memory mapping is page aligned (both the address and the size). If not, it errors out.
- Use phys_to_virt() to get to the virtual memory from the reserve_mem returned addresses. Also use free_reserved_area() to give it back to the buddy allocator when it is freed.
- Treat the buffer allocated via memmap differently. It still needs to be virtually mapped (cannot use phys_to_virt) and it must not be freed nor memory mapped to user space. A new flag is added when a buffer is created this way to prevent it from ever being memory mapped to user space and the ref count is upped so that it can never be freed.
- Use vmap_page_range() instead of using kmalloc_array() to create an array of struct pages for vmap().
- Use flush_kernel_vmap_range() instead of flush_dcache_folio()
- Allow the reserve_mem persistent ring buffer to be memory mapped. There is no difference now with how the memory is mapped to user space, only the accounting of what pages are mapped where is updated as the meta data is different between the two.
Note, the first 4 patches makes the code a bit more correct. Especially since the vunmap() does not give the buffer back to the buddy allocator. I will be looking to get the first 4 patches into this merge window.
The last patch which enables he persistent memory mapping to user space can wait till the 6.16.
Changes since v2: https://lore.kernel.org/all/[email protected]/
- Basically a full rewrite once I found out that you can get the virtual address of the memory returned by reserve_mem via phys_to_virt()!
Steven Rostedt (5):
tracing: Enforce the persistent ring buffer to be page aligned
tracing: Have reserve_mem use phys_to_virt() and separate from memmap buffer
tracing: Use vmap_page_range() to map memmap ring buffer
ring-buffer: Use flush_kernel_vmap_range() over flush_dcache_folio()
ring-buffer: Allow reserve_mem persistent ring buffers to be mmapped
----
Documentation/admin-guide/kernel-parameters.txt | 2 + Documentation/trace/debugging.rst | 2 + kernel/trace/ring_buffer.c | 54 ++++++++++++++++-- kernel/trace/trace.c | 75 ++++++++++++++++--------- kernel/trace/trace.h | 1 + 5 files changed, 102 insertions(+), 32 deletions(-)
Steven Rostedt
maintainerPATCH v3 1/5 tracing: Enforce the persistent ring buffer to be page aligned
From: Steven Rostedt <[email protected]>
Enforce that the address and the size of the memory used by the persistent ring buffer is page aligned. Also update the documentation to reflect this requirement.
--- Documentation/admin-guide/kernel-parameters.txt | 2 ++ Documentation/trace/debugging.rst | 2 ++ kernel/trace/trace.c | 12 ++++++++++++ 3 files changed, 16 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +2 -0
@@ -7241,6 +7241,8 @@ This is just one of many ways that can clear memory. Make sure your system keeps the content of memory across reboots before relying on this option. + NB: Both the mapped address and size must be page aligned for the architecture. + See also Documentation/trace/debugging.rst
Documentation/trace/debugging.rst +2 -0
@@ -136,6 +136,8 @@ kernel, so only the same kernel is guaranteed to work if the mapping is preserved. Switching to a different kernel version may find a different layout and mark the buffer as invalid. +NB: Both the mapped address and size must be page aligned for the architecture. + Using trace_printk() in the boot instance ----------------------------------------- By default, the content of trace_printk() goes into the top level tracing
kernel/trace/trace.c +12 -0
@@ -10774,6 +10774,18 @@ __init static void enable_instances(void) }if (start) { +/* Start and size must be page aligned */ +if (start & ~PAGE_MASK ) { +pr_warn ("Tracing: mapping start addr %lx is not page aligned\n" , + (unsigned long )start); +continue ; + } +if (size & ~PAGE_MASK ) { +pr_warn ("Tracing: mapping size %lx is not page aligned\n" , + (unsigned long )size); +continue ; + } + addr =map_pages (start, size);if (addr) { pr_info("Tracing: mapped boot instance %s at physical memory %pa of size 0x%lx\n",
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v3 2/5 tracing: Have reserve_mem use phys_to_virt() and separate from memmap buffer
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The reserve_mem kernel command line option may pass back a physical address, but the memory is still part of the normal memory just like using memblock_reserve() would be. This means that the physical memory returned by the reserve_mem command line option can be converted directly to virtual memory by simply using phys_to_virt().
When freeing the buffer allocated by reserve_mem, use free_reserved_area().
Because the persistent ring buffer can also be allocated via the memmap option, which *is* different than normal memory as it cannot be added back to the buddy system, it must be treated differently. It still needs to be virtually mapped to have access to it. It also can not be freed nor can it ever be memory mapped to user space.
Create a new trace_array flag called TRACE_ARRAY_FL_MEMMAP which gets set if the buffer is created by the memmap option, and this will prevent the buffer from being memory mapped by user space.
Also increment the ref count for memmap'ed buffers so that they can never be freed.
--- kernel/trace/trace.c | 28 ++++++++++++++++++++++------ kernel/trace/trace.h | 1 + 2 files changed, 23 insertions(+), 6 deletions(-)
kernel/trace/trace.c +22 -6
@@ -8492,6 +8492,10 @@ static int tracing_buffers_mmap(struct file *filp, struct vm_area_struct *vma)struct trace_iterator *iter = &info->iter;int ret =0 ; +/* A memmap'ed buffer is not supported for user space mmap */ +if (iter->tr->flags &TRACE_ARRAY_FL_MEMMAP ) +return -ENODEV ; +/* Currently the boot mapped buffer is not supported for mmap */ if (iter->tr->flags &TRACE_ARRAY_FL_BOOT )return -ENODEV ;
@@ -9601,8 +9605,12 @@ static void free_trace_buffers(struct trace_array *tr)free_trace_buffer (&tr->max_buffer);#endif -if (tr->range_addr_start) -vunmap ((void *)tr->range_addr_start); +if (tr->range_addr_start) { +void *start = (void *)tr->range_addr_start; +void *end = start + tr->range_addr_size; + +free_reserved_area (start, end,0 , tr->range_name); + } }static void init_trace_flags_index (struct trace_array *tr)
@@ -10696,6 +10704,7 @@ static inline void do_allocate_snapshot(const char *name) { }__init static void enable_instances (void ) {struct trace_array *tr; +bool memmap_area = false;char *curr_str;char *name;char *str;
@@ -10764,6 +10773,7 @@ __init static void enable_instances(void) name);continue ; } + memmap_area = true; }else if (tok) {if (!reserve_mem_find_by_name (tok, &start, &size)) { start =0 ;
@@ -10786,7 +10796,10 @@ __init static void enable_instances(void)continue ; } - addr =map_pages (start, size); +if (memmap_area) + addr =map_pages (start, size); +else + addr = (unsigned long )phys_to_virt (start);if (addr) {pr_info ("Tracing: mapped boot instance %s at physical memory %pa of size 0x%lx\n" , name, &start, (unsigned long )size);
@@ -10813,10 +10826,13 @@ __init static void enable_instances(void)update_printk_trace (tr);/* -* If start is set, then this is a mapped buffer, and -* cannot be deleted by user space, so keep the reference -* to it. +* memmap'd buffers can not be freed. */ +if (memmap_area) { + tr->flags |=TRACE_ARRAY_FL_MEMMAP ; + tr->ref++; + } +if (start) { tr->flags |=TRACE_ARRAY_FL_BOOT |TRACE_ARRAY_FL_LAST_BOOT ; tr->range_name =no_free_ptr (rname);
kernel/trace/trace.h +1 -0
@@ -446,6 +446,7 @@ enum {TRACE_ARRAY_FL_BOOT =BIT (1 ),TRACE_ARRAY_FL_LAST_BOOT =BIT (2 ),TRACE_ARRAY_FL_MOD_INIT =BIT (3 ), + TRACE_ARRAY_FL_MEMMAP = BIT(4), };#ifdef CONFIG_MODULES
Signature · 1 line
2.47.2
Steven Rostedt
maintainerPATCH v3 3/5 tracing: Use vmap_page_range() to map memmap ring buffer
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The code to map the physical memory retrieved by memmap currently allocates an array of pages to cover the physical memory and then calls vmap() to map it to a virtual address. Instead of using this temporary array of struct page descriptors, simply use vmap_page_range() that can directly map the contiguous physical memory to a virtual address.
--- kernel/trace/trace.c | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-)
kernel/trace/trace.c +17 -18
@@ -50,6 +50,7 @@#include <linux/irq_work.h> #include <linux/workqueue.h> #include <linux/sort.h> +#include <linux/io.h> /* vmap_page_range() */ #include <asm/setup.h> /* COMMAND_LINE_SIZE */
@@ -9803,29 +9804,27 @@ static int instance_mkdir(const char *name)return ret; } -static u64 map_pages (u64 start,u64 size) +static u64 map_pages (unsigned long start,unsigned long size) { -struct page **pages; -phys_addr_t page_start; -unsigned int page_count; -unsigned int i; -void *vaddr; +unsigned long vmap_start, vmap_end; +struct vm_struct *area; +int ret; - page_count =DIV_ROUND_UP (size,PAGE_SIZE ); + area =get_vm_area (size,VM_IOREMAP ); +if (!area) +return 0 ; - page_start = start; - pages =kmalloc_array (page_count,sizeof (struct page *),GFP_KERNEL ); -if (!pages) -return 0 ; + vmap_start = (unsigned long ) area->addr; + vmap_end = vmap_start + size; -for (i =0 ; i < page_count; i++) { -phys_addr_t addr = page_start + i *PAGE_SIZE ; - pages[i] =pfn_to_page (addr >>PAGE_SHIFT ); - } - vaddr =vmap (pages, page_count,VM_MAP ,PAGE_KERNEL ); -kfree (pages); + ret =vmap_page_range (vmap_start, vmap_end, + start,pgprot_nx (PAGE_KERNEL )); +if (ret <0 ) { +free_vm_area (area); +return 0 ; + } -return (u64 )(unsigned long )vaddr; +return (u64 )vmap_start; }/**
Signature · 1 line
2.47.2
Steven Rostedt
maintainerRe: [PATCH v3 3/5] tracing: Use vmap_page_range() to map memmap ring buffer
kernel/trace/trace.c static int instance_mkdir(const char *name) · line 9809
Quoted 9 lines · On Tue, 01 Apr 2025 16:25:52 -0400 Steven Rostedt <[email protected]> wrote:
kernel/trace/trace.c +2
-6-static u64 map_pages (u64 start,u64 size) +static u64 map_pages (unsigned long start,unsigned long size) { -struct page **pages; -phys_addr_t page_start; -unsigned int page_count; -unsigned int i; -void *vaddr; +unsigned long vmap_start, vmap_end;
Bah, looks like I have whitespace issues. Will send an update to fix this and the change log for patch 4.
Quoted 3 lines
kernel/trace/trace.c +2
-0+struct vm_struct *area; +int ret;
Steven Rostedt
maintainerPATCH v3 4/5 ring-buffer: Use flush_kernel_vmap_range() over flush_dcache_folio()
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Some architectures do not have data cache coherency between user and kernel space. For these architectures, the cache needs to be flushed on both the kernel and user addresses so that user space can see the updates the kernel has made.
Instead of using flush_dcache_folio() and playing with virt_to_folio() within the call to that function, use flush_kernel_vmap_range() which takes the virtual address and does the work for those architectures that need it.
--- kernel/trace/ring_buffer.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/trace/ring_buffer.c +3 -2
@@ -6016,7 +6016,7 @@ static void rb_update_meta_page(struct ring_buffer_per_cpu *cpu_buffer) meta->read = cpu_buffer->read;/* Some archs do not have data cache coherency between kernel and user-space */ -flush_dcache_folio (virt_to_folio (cpu_buffer->meta_page)); +flush_kernel_vmap_range (cpu_buffer->meta_page,PAGE_SIZE ); } static void
@@ -7319,7 +7319,8 @@ int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu) out:/* Some archs do not have data cache coherency between kernel and user-space */ -flush_dcache_folio (virt_to_folio (cpu_buffer->reader_page->page)); +flush_kernel_vmap_range (cpu_buffer->reader_page->page, + buffer->subbuf_size +BUF_PAGE_HDR_SIZE );rb_update_meta_page (cpu_buffer);
Signature · 1 line
2.47.2
Steven Rostedt
maintainerRe: [PATCH v3 4/5] ring-buffer: Use flush_kernel_vmap_range() over flush_dcache_folio()
kernel/trace/ring_buffer.c int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu) · line 7326
Quoted 12 lines · On Tue, 01 Apr 2025 16:25:53 -0400 Steven Rostedt <[email protected]> wrote:
static void
kernel/trace/ring_buffer.c +2
-1@@ -7319,7 +7319,8 @@ int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu) out:/* Some archs do not have data cache coherency between kernel and user-space */ -flush_dcache_folio (virt_to_folio (cpu_buffer->reader_page->page)); +flush_kernel_vmap_range (cpu_buffer->reader_page->page, + buffer->subbuf_size +BUF_PAGE_HDR_SIZE );rb_update_meta_page (cpu_buffer);
This patch actually fixes a bug (I need to update the change log).
The previous code only flushed a page. But if the sub-buffer size was 2 or more pages, it would not flush the rest of the sub-buffer.
-- Steve
Steven Rostedt
maintainerPATCH v3 5/5 ring-buffer: Allow reserve_mem persistent ring buffers to be mmapped
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
When the persistent ring buffer is created from the memory returned by reserve_mem there is nothing prohibiting it to be memory mapped to user space. The memory is the same as the pages allocated by alloc_page().
The way the memory is managed by the ring buffer code is slightly different though and needs to be addressed.
The persistent memory uses the page->id for its own purpose where as the user mmap buffer currently uses that for the subbuf array mapped to user space. If the buffer is a persistent buffer, use the page index into that buffer as the identifier instead of the page->id.
That is, the page->id for a persistent buffer, represents the order of the buffer is in the link list. ->id == 0 means it is the reader page. When a reader page is swapped, the new reader page's ->id gets zero, and the old reader page gets the ->id of the page that it swapped with.
The user space mapping has the ->id is the index of where it was mapped in user space and does not change while it is mapped.
Since the persistent buffer is fixed in its location, the index of where a page is in the memory range can be used as the "id" to put in the meta page array, and it can be mapped in the same order to user space as it is in the persistent memory.
A new rb_page_id() helper function is used to get and set the id depending on if the page is a normal memory allocated buffer or a physical memory mapped buffer.
--- kernel/trace/ring_buffer.c | 49 ++++++++++++++++++++++++++++++++++---- kernel/trace/trace.c | 4 ---- 2 files changed, 45 insertions(+), 8 deletions(-)
kernel/trace/ring_buffer.c +45 -4
@@ -6000,6 +6000,39 @@ static void rb_clear_buffer_page(struct buffer_page *page) page->read =0 ; } +/* +* When the buffer is memory mapped to user space, each sub buffer +* has a unique id that is used by the meta data to tell the user +* where the current reader page is. +* +* For a normal allocated ring buffer, the id is saved in the buffer page +* id field, and updated via this function. +* +* But for a fixed memory mapped buffer, the id is already assigned for +* fixed memory ording in the memory layout and can not be used. Instead +* the index of where the page lies in the memory layout is used. +* +* For the normal pages, set the buffer page id with the passed in @id +* value and return that. +* +* For fixed memory mapped pages, get the page index in the memory layout +* and return that as the id. +*/ +static int rb_page_id (struct ring_buffer_per_cpu *cpu_buffer, +struct buffer_page *bpage,int id) +{ +/* +* For boot buffers, the id is the index, +* otherwise, set the buffer page with this id +*/ +if (cpu_buffer->ring_meta) + id =rb_meta_subbuf_idx (cpu_buffer->ring_meta, bpage->page); +else + bpage->id = id; + +return id; +} +static void rb_update_meta_page (struct ring_buffer_per_cpu *cpu_buffer) {struct trace_buffer_meta *meta = cpu_buffer->meta_page;
@@ -6008,7 +6041,9 @@ static void rb_update_meta_page(struct ring_buffer_per_cpu *cpu_buffer)return ; meta->reader.read = cpu_buffer->reader_page->read; - meta->reader.id = cpu_buffer->reader_page->id; + meta->reader.id =rb_page_id (cpu_buffer, cpu_buffer->reader_page, + cpu_buffer->reader_page->id); + meta->reader.lost_events = cpu_buffer->lost_events; meta->entries =local_read (&cpu_buffer->entries);
@@ -6924,23 +6959,29 @@ static void rb_setup_ids_meta_page(struct ring_buffer_per_cpu *cpu_buffer,struct trace_buffer_meta *meta = cpu_buffer->meta_page;unsigned int nr_subbufs = cpu_buffer->nr_pages +1 ;struct buffer_page *first_subbuf, *subbuf; +int cnt =0 ;int id =0 ; - subbuf_ids[id] = (unsigned long )cpu_buffer->reader_page->page; - cpu_buffer->reader_page->id = id++; + id =rb_page_id (cpu_buffer, cpu_buffer->reader_page, id); + subbuf_ids[id++] = (unsigned long )cpu_buffer->reader_page->page; + cnt++; first_subbuf = subbuf =rb_set_head_page (cpu_buffer);do { + id =rb_page_id (cpu_buffer, subbuf, id); +if (WARN_ON (id >= nr_subbufs))break ; subbuf_ids[id] = (unsigned long )subbuf->page; - subbuf->id = id;rb_inc_page (&subbuf); id++; + cnt++; }while (subbuf != first_subbuf); +WARN_ON (cnt != nr_subbufs); +/* install subbuf ID to kern VA translation */ cpu_buffer->subbuf_ids = subbuf_ids;
kernel/trace/trace.c +0 -4
@@ -8497,10 +8497,6 @@ static int tracing_buffers_mmap(struct file *filp, struct vm_area_struct *vma)if (iter->tr->flags &TRACE_ARRAY_FL_MEMMAP )return -ENODEV ; -/* Currently the boot mapped buffer is not supported for mmap */ -if (iter->tr->flags &TRACE_ARRAY_FL_BOOT ) -return -ENODEV ; - ret =get_snapshot_map (iter->tr);if (ret)return ret;
Signature · 1 line
2.47.2