Skip to contentSkip to conversation

tracing: Clean up persistent ring buffer code

PATCH In review Steven Rostedt 3 participants 11 messages Apr 2, 2025 – Apr 7, 2025

Subsystems (4)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

v6 cover tracing: Clean up persistent ring buffer code

Now that I learned that the memory passed back from reserve_mem is part of the memory allocator and just "reserved" and the memory is already virtually mapped, it can simply use phys_to_virt() on the physical memory that is returned to get the virtual mapping for that memory!
(Thanks Mike!)

That makes things much easier, especially since it means that the memory returned by reserve_mem is no different than the memory retrieved by page_alloc(). This allows that memory to be memory mapped to user space no differently than it is mapped by the normal buffer.

This new series does the following:

- Enforce the memory mapping is page aligned (both the address and the size). If not, it errors out.

- Use phys_to_virt() to get to the virtual memory from the reserve_mem returned addresses. As the memory is already freed via reserve_mem_release_by_name() and it's not mapped by vmap() anymore, the free ring buffer code doesn't need to do anything special for this mapping.

- Treat the buffer allocated via memmap differently. It still needs to be virtually mapped (cannot use phys_to_virt) and it must not be freed nor memory mapped to user space. A new flag is added when a buffer is created this way to prevent it from ever being memory mapped to user space and the ref count is upped so that it can never be freed.

- Use vmap_page_range() instead of using kmalloc_array() to create an array of struct pages for vmap().

- Use flush_kernel_vmap_range() instead of flush_dcache_folio()

Changes since v5: https://lore.kernel.org/linux-trace-kernel/[email protected]/

- Use %pa instead of %lx for start and size sizes (Mike Rapoport)

- Updated change log to use memblock_alloc() instead of memblock_reserve() (Mike Rapoport)

Steven Rostedt (4):
tracing: Enforce the persistent ring buffer to be page aligned
tracing: Have reserve_mem use phys_to_virt() and separate from memmap buffer
tracing: Use vmap_page_range() to map memmap ring buffer
ring-buffer: Use flush_kernel_vmap_range() over flush_dcache_folio()

----

 Documentation/admin-guide/kernel-parameters.txt |  2 +
 Documentation/trace/debugging.rst               |  2 +
 kernel/trace/ring_buffer.c                      |  5 +-
 kernel/trace/trace.c                            | 66 ++++++++++++++++---------
 kernel/trace/trace.h                            |  1 +
 5 files changed, 50 insertions(+), 26 deletions(-)
author maintainer
#1 lore raw

PATCH v6 1/4 tracing: Enforce the persistent ring buffer to be page aligned

From: Steven Rostedt <[email protected]>

Enforce that the address and the size of the memory used by the persistent ring buffer is page aligned. Also update the documentation to reflect this requirement.

---
Change since v5: https://lore.kernel.org/linux-trace-kernel/[email protected]/

- Use %pa instead of %lx for start and size sizes (Mike Rapoport)

 Documentation/admin-guide/kernel-parameters.txt |  2 ++
 Documentation/trace/debugging.rst               |  2 ++
 kernel/trace/trace.c                            | 10 ++++++++++
 3 files changed, 14 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +2 -0
index fb8752b42ec8..71861643ef14 100644
@@ -7241,6 +7241,8 @@
 			This is just one of many ways that can clear memory. Make sure your system
 			keeps the content of memory across reboots before relying on this option.
 
+			NB: Both the mapped address and size must be page aligned for the architecture.
+
 			See also Documentation/trace/debugging.rst
 
 
Documentation/trace/debugging.rst +2 -0
index 54fb16239d70..d54bc500af80 100644
@@ -136,6 +136,8 @@ kernel, so only the same kernel is guaranteed to work if the mapping is
 preserved. Switching to a different kernel version may find a different
 layout and mark the buffer as invalid.
 
+NB: Both the mapped address and size must be page aligned for the architecture.
+
 Using trace_printk() in the boot instance
 -----------------------------------------
 By default, the content of trace_printk() goes into the top level tracing
kernel/trace/trace.c +10 -0
index 14c38fcd6f9e..96129985e81c 100644
@@ -10774,6 +10774,16 @@ __init static void enable_instances(void)
 		}
 
 		if (start) {
+			/* Start and size must be page aligned */
+			if (start & ~PAGE_MASK) {
+				pr_warn("Tracing: mapping start addr %pa is not page aligned\n", &start);
+				continue;
+			}
+			if (size & ~PAGE_MASK) {
+				pr_warn("Tracing: mapping size %pa is not page aligned\n", &size);
+				continue;
+			}
+
 			addr = map_pages(start, size);
 			if (addr) {
 				pr_info("Tracing: mapped boot instance %s at physical memory %pa of size 0x%lx\n",
Signature · 1 line
2.47.2
author maintainer
#2 lore raw

PATCH v6 2/4 tracing: Have reserve_mem use phys_to_virt() and separate from memmap buffer

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

The reserve_mem kernel command line option may pass back a physical address, but the memory is still part of the normal memory just like using memblock_alloc() would be. This means that the physical memory returned by the reserve_mem command line option can be converted directly to virtual memory by simply using phys_to_virt().

When freeing the buffer there's no need to call vunmap() anymore as the memory allocated by reserve_mem is freed by the call to reserve_mem_release_by_name().

Because the persistent ring buffer can also be allocated via the memmap option, which *is* different than normal memory as it cannot be added back to the buddy system, it must be treated differently. It still needs to be virtually mapped to have access to it. It also can not be freed nor can it ever be memory mapped to user space.

Create a new trace_array flag called TRACE_ARRAY_FL_MEMMAP which gets set if the buffer is created by the memmap option, and this will prevent the buffer from being memory mapped by user space.

Also increment the ref count for memmap'ed buffers so that they can never be freed.

---
Changes since v5: https://lore.kernel.org/linux-trace-kernel/[email protected]/

- Updated change log to use memblock_alloc() instead of memblock_reserve() (Mike Rapoport)

 kernel/trace/trace.c | 23 ++++++++++++++++-------
 kernel/trace/trace.h |  1 +
 2 files changed, 17 insertions(+), 7 deletions(-)
kernel/trace/trace.c +16 -7
index 96129985e81c..e58b72e61573 100644
@@ -8492,6 +8492,10 @@ static int tracing_buffers_mmap(struct file *filp, struct vm_area_struct *vma)
 	struct trace_iterator *iter = &info->iter;
 	int ret = 0;
 
+	/* A memmap'ed buffer is not supported for user space mmap */
+	if (iter->tr->flags & TRACE_ARRAY_FL_MEMMAP)
+		return -ENODEV;
+
 	/* Currently the boot mapped buffer is not supported for mmap */
 	if (iter->tr->flags & TRACE_ARRAY_FL_BOOT)
 		return -ENODEV;
@@ -9600,9 +9604,6 @@ static void free_trace_buffers(struct trace_array *tr)
 #ifdef CONFIG_TRACER_MAX_TRACE
 	free_trace_buffer(&tr->max_buffer);
 #endif
-
-	if (tr->range_addr_start)
-		vunmap((void *)tr->range_addr_start);
 }
 
 static void init_trace_flags_index(struct trace_array *tr)
@@ -10696,6 +10697,7 @@ static inline void do_allocate_snapshot(const char *name) { }
 __init static void enable_instances(void)
 {
 	struct trace_array *tr;
+	bool memmap_area = false;
 	char *curr_str;
 	char *name;
 	char *str;
@@ -10764,6 +10766,7 @@ __init static void enable_instances(void)
 					name);
 				continue;
 			}
+			memmap_area = true;
 		} else if (tok) {
 			if (!reserve_mem_find_by_name(tok, &start, &size)) {
 				start = 0;
@@ -10784,7 +10787,10 @@ __init static void enable_instances(void)
 				continue;
 			}
 
-			addr = map_pages(start, size);
+			if (memmap_area)
+				addr = map_pages(start, size);
+			else
+				addr = (unsigned long)phys_to_virt(start);
 			if (addr) {
 				pr_info("Tracing: mapped boot instance %s at physical memory %pa of size 0x%lx\n",
 					name, &start, (unsigned long)size);
@@ -10811,10 +10817,13 @@ __init static void enable_instances(void)
 			update_printk_trace(tr);
 
 		/*
-		 * If start is set, then this is a mapped buffer, and
-		 * cannot be deleted by user space, so keep the reference
-		 * to it.
+		 * memmap'd buffers can not be freed.
 		 */
+		if (memmap_area) {
+			tr->flags |= TRACE_ARRAY_FL_MEMMAP;
+			tr->ref++;
+		}
+
 		if (start) {
 			tr->flags |= TRACE_ARRAY_FL_BOOT | TRACE_ARRAY_FL_LAST_BOOT;
 			tr->range_name = no_free_ptr(rname);
kernel/trace/trace.h +1 -0
index ab7c7a1930cc..9d9dcfad6269 100644
@@ -446,6 +446,7 @@ enum {
 	TRACE_ARRAY_FL_BOOT		= BIT(1),
 	TRACE_ARRAY_FL_LAST_BOOT	= BIT(2),
 	TRACE_ARRAY_FL_MOD_INIT		= BIT(3),
+	TRACE_ARRAY_FL_MEMMAP		= BIT(4),
 };
 
 #ifdef CONFIG_MODULES
Signature · 1 line
2.47.2
author maintainer
#3 lore raw

PATCH v6 3/4 tracing: Use vmap_page_range() to map memmap ring buffer

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

The code to map the physical memory retrieved by memmap currently allocates an array of pages to cover the physical memory and then calls vmap() to map it to a virtual address. Instead of using this temporary array of struct page descriptors, simply use vmap_page_range() that can directly map the contiguous physical memory to a virtual address.

---
 kernel/trace/trace.c | 33 ++++++++++++++++-----------------
 1 file changed, 16 insertions(+), 17 deletions(-)
kernel/trace/trace.c +16 -17
index e58b72e61573..f6531cd3176b 100644
@@ -50,6 +50,7 @@
 #include <linux/irq_work.h>
 #include <linux/workqueue.h>
 #include <linux/sort.h>
+#include <linux/io.h> /* vmap_page_range() */
 
 #include <asm/setup.h> /* COMMAND_LINE_SIZE */
 
@@ -9796,29 +9797,27 @@ static int instance_mkdir(const char *name)
 	return ret;
 }
 
-static u64 map_pages(u64 start, u64 size)
+static u64 map_pages(unsigned long start, unsigned long size)
 {
-	struct page **pages;
-	phys_addr_t page_start;
-	unsigned int page_count;
-	unsigned int i;
-	void *vaddr;
-
-	page_count = DIV_ROUND_UP(size, PAGE_SIZE);
+	unsigned long vmap_start, vmap_end;
+	struct vm_struct *area;
+	int ret;
 
-	page_start = start;
-	pages = kmalloc_array(page_count, sizeof(struct page *), GFP_KERNEL);
-	if (!pages)
+	area = get_vm_area(size, VM_IOREMAP);
+	if (!area)
 		return 0;
 
-	for (i = 0; i < page_count; i++) {
-		phys_addr_t addr = page_start + i * PAGE_SIZE;
-		pages[i] = pfn_to_page(addr >> PAGE_SHIFT);
+	vmap_start = (unsigned long) area->addr;
+	vmap_end = vmap_start + size;
+
+	ret = vmap_page_range(vmap_start, vmap_end,
+			      start, pgprot_nx(PAGE_KERNEL));
+	if (ret < 0) {
+		free_vm_area(area);
+		return 0;
 	}
-	vaddr = vmap(pages, page_count, VM_MAP, PAGE_KERNEL);
-	kfree(pages);
 
-	return (u64)(unsigned long)vaddr;
+	return (u64)vmap_start;
 }
 
 /**
Signature · 1 line
2.47.2
#4 lore raw

Re: [PATCH v6 3/4] tracing: Use vmap_page_range() to map memmap ring buffer

Quoted 14 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

The code to map the physical memory retrieved by memmap currently allocates an array of pages to cover the physical memory and then calls vmap() to map it to a virtual address. Instead of using this temporary array of struct page descriptors, simply use vmap_page_range() that can directly map the contiguous physical memory to a virtual address.

---
 kernel/trace/trace.c | 33 ++++++++++++++++-----------------

...

kernel/trace/trace.c static int instance_mkdir(const char *name) · line 9806

Quoted 1 line
kernel/trace/trace.c +1 -0
+	area = get_vm_area(size, VM_IOREMAP);

non-MMU builds (xtensa:nommu_kc705_defconfig):

kernel/trace/trace.o:(.init.literal+0x250): undefined reference to `get_vm_area'

Guenter

author maintainer
#5 lore raw

Re: [PATCH v6 3/4] tracing: Use vmap_page_range() to map memmap ring buffer

Quoted 1 line · On Sun, 6 Apr 2025 10:39:27 -0700 Guenter Roeck <[email protected]> wrote:

non-MMU builds (xtensa:nommu_kc705_defconfig):

Bah!

Quoted 2 lines

kernel/trace/trace.o:(.init.literal+0x250): undefined reference to `get_vm_area'

Does this fix it?

kernel/trace/trace.c +7 -0
index b581e388a9d9..8ddf6b17215c 100644
@@ -9806,6 +9806,7 @@ static int instance_mkdir(const char *name)
 	return ret;
 }
 
+#ifdef CONFIG_MMU
 static u64 map_pages(unsigned long start, unsigned long size)
 {
 	unsigned long vmap_start, vmap_end;
@@ -9828,6 +9829,12 @@ static u64 map_pages(unsigned long start, unsigned long size)
 
 	return (u64)vmap_start;
 }
+#else
+static inline u64 map_pages(unsigned long start, unsigned long size)
+{
+	return 0;
+}
+#endif
 
 /**
  * trace_array_get_by_name - Create/Lookup a trace array, given its name.

-- Steve

Tested-by
#6 lore raw

Re: [PATCH v6 3/4] tracing: Use vmap_page_range() to map memmap ring buffer

Quoted 7 lines · Steven Rostedt
Quoted 1 line · On Sun, 6 Apr 2025 10:39:27 -0700 Guenter Roeck <[email protected]> wrote:

non-MMU builds (xtensa:nommu_kc705_defconfig):

Bah!

:-)

Quoted 5 lines
Quoted 2 lines

kernel/trace/trace.o:(.init.literal+0x250): undefined reference to `get_vm_area'

Does this fix it?

Yes, it does.

Thanks,
Guenter

Quoted 28 lines
kernel/trace/trace.c +7 -1
index b581e388a9d9..8ddf6b17215c 100644
@@ -9806,6 +9806,7 @@ static int instance_mkdir(const char *name)
  	return ret;
  }
 
+#ifdef CONFIG_MMU
  static u64 map_pages(unsigned long start, unsigned long size)
  {
  	unsigned long vmap_start, vmap_end;
@@ -9828,6 +9829,12 @@ static u64 map_pages(unsigned long start, unsigned long size)
 
  	return (u64)vmap_start;
  }
+#else
+static inline u64 map_pages(unsigned long start, unsigned long size)
+{
+	return 0;
+}
+#endif
 
  /**
   * trace_array_get_by_name - Create/Lookup a trace array, given its name.
 
 
-- Steve
author maintainer
#7 lore raw

PATCH v6 4/4 ring-buffer: Use flush_kernel_vmap_range() over flush_dcache_folio()

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

Some architectures do not have data cache coherency between user and kernel space. For these architectures, the cache needs to be flushed on both the kernel and user addresses so that user space can see the updates the kernel has made.

Instead of using flush_dcache_folio() and playing with virt_to_folio() within the call to that function, use flush_kernel_vmap_range() which takes the virtual address and does the work for those architectures that need it.

This also fixes a bug where the flush of the reader page only flushed one page. If the sub-buffer order is 1 or more, where the sub-buffer size would be greater than a page, it would miss the rest of the sub-buffer content, as the "reader page" is not just a page, but the size of a sub-buffer.

Fixes 117c39200d9d7 ("ring-buffer: Introducing ring-buffer mapping functions");
---
 kernel/trace/ring_buffer.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/trace/ring_buffer.c +3 -2
index f25966b3a1fc..d4b0f7b55cce 100644
@@ -6016,7 +6016,7 @@ static void rb_update_meta_page(struct ring_buffer_per_cpu *cpu_buffer)
 	meta->read = cpu_buffer->read;
 
 	/* Some archs do not have data cache coherency between kernel and user-space */
-	flush_dcache_folio(virt_to_folio(cpu_buffer->meta_page));
+	flush_kernel_vmap_range(cpu_buffer->meta_page, PAGE_SIZE);
 }
 
 static void
@@ -7319,7 +7319,8 @@ int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu)
 
 out:
 	/* Some archs do not have data cache coherency between kernel and user-space */
-	flush_dcache_folio(virt_to_folio(cpu_buffer->reader_page->page));
+	flush_kernel_vmap_range(cpu_buffer->reader_page->page,
+				buffer->subbuf_size + BUF_PAGE_HDR_SIZE);
 
 	rb_update_meta_page(cpu_buffer);
 
Signature · 1 line
2.47.2
#8 lore raw
In reply to Steven Rostedt
Quoted 7 lines · Steven Rostedt

Now that I learned that the memory passed back from reserve_mem is part of the memory allocator and just "reserved" and the memory is already virtually mapped, it can simply use phys_to_virt() on the physical memory that is returned to get the virtual mapping for that memory!
(Thanks Mike!)

Quoted 6 lines

That makes things much easier, especially since it means that the memory returned by reserve_mem is no different than the memory retrieved by page_alloc(). This allows that memory to be memory mapped to user space no differently than it is mapped by the normal buffer.

This new series does the following:

So I've been loosely following this patch series, and I'm confused about one thing.

AFAIU one goal is to have the ftrace persistent ring buffer written to through a memory range returned by vmap_page_range(), and userspace maps the buffer with virtual mappings.

With respect to architectures with aliasing dcache, is the plan:

A) To make sure all persistent ring buffer mappings are aligned on SHMLBA:

Quoting "Documentation/core-api/cachetlb.rst":

Is your port susceptible to virtual aliasing in its D-cache? Well, if your D-cache is virtually indexed, is larger in size than PAGE_SIZE, and does not prevent multiple cache lines for the same physical address from existing at once, you have this problem.

If your D-cache has this problem, first define asm/shmparam.h SHMLBA properly, it should essentially be the size of your virtually addressed D-cache (or if the size is variable, the largest possible size). This setting will force the SYSv IPC layer to only allow user processes to mmap shared memory at address which are a multiple of this value.

or

B) to flush both the kernel and userspace mappings when a ring buffer page is handed over from writer to reader ?

I've seen both approaches being discussed in the recent threads, with some consensus aiming towards (A), but then the code that follows takes approach (B).

AFAIU, it we are aiming for approach (A), then I'm missing where vmap_page_range() guarantees that the _kernel_ virtual mapping is SHMLBA aligned. AFAIU, only user mappings are aligned on SHMLBA.

And if we aiming towards approach (A), then the explicit flushing is not needed when handing over pages from writer to reader, but an SHMLBA alignment should be introduced.

Please let me know if I'm missing something,

Thanks,

Mathieu

Quoted 41 lines

- Enforce the memory mapping is page aligned (both the address and the size). If not, it errors out.

- Use phys_to_virt() to get to the virtual memory from the reserve_mem returned addresses. As the memory is already freed via reserve_mem_release_by_name() and it's not mapped by vmap() anymore, the free ring buffer code doesn't need to do anything special for this mapping.

- Treat the buffer allocated via memmap differently. It still needs to be virtually mapped (cannot use phys_to_virt) and it must not be freed nor memory mapped to user space. A new flag is added when a buffer is created this way to prevent it from ever being memory mapped to user space and the ref count is upped so that it can never be freed.

- Use vmap_page_range() instead of using kmalloc_array() to create an array of struct pages for vmap().

- Use flush_kernel_vmap_range() instead of flush_dcache_folio()

Changes since v5: https://lore.kernel.org/linux-trace-kernel/[email protected]/

- Use %pa instead of %lx for start and size sizes (Mike Rapoport)

- Updated change log to use memblock_alloc() instead of memblock_reserve() (Mike Rapoport)

Steven Rostedt (4):
tracing: Enforce the persistent ring buffer to be page aligned
tracing: Have reserve_mem use phys_to_virt() and separate from memmap buffer
tracing: Use vmap_page_range() to map memmap ring buffer
ring-buffer: Use flush_kernel_vmap_range() over flush_dcache_folio()

----

  Documentation/admin-guide/kernel-parameters.txt |  2 +
  Documentation/trace/debugging.rst               |  2 +
  kernel/trace/ring_buffer.c                      |  5 +-
  kernel/trace/trace.c                            | 66 ++++++++++++++++---------
  kernel/trace/trace.h                            |  1 +
  5 files changed, 50 insertions(+), 26 deletions(-)
Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
author maintainer
#9 lore raw
Quoted 6 lines · On Wed, 2 Apr 2025 10:56:51 -0400 Mathieu Desnoyers…

So I've been loosely following this patch series, and I'm confused about one thing.

AFAIU one goal is to have the ftrace persistent ring buffer written to through a memory range returned by vmap_page_range(), and userspace maps the buffer with virtual mappings.

Note, this series does not have that goal. The memory mapped buffer via vmap_page_range() is not going to ever be mapped to user space. The only time the buffer will be mapped via vmap_page_range() (after this patch series) is when it is allocated via memmap=... which is a major hack and cannot be trusted to be used with memory mappings.

The persistent buffer when mapped with reserve_mem=... is no different than the normal buffer being allocated via page_alloc() and page_address() to get the virtual address. (Note this series still doesn't it make available to be mapped to user space).

Quoted 20 lines

With respect to architectures with aliasing dcache, is the plan:

A) To make sure all persistent ring buffer mappings are aligned on SHMLBA:

Quoting "Documentation/core-api/cachetlb.rst":

Is your port susceptible to virtual aliasing in its D-cache? Well, if your D-cache is virtually indexed, is larger in size than PAGE_SIZE, and does not prevent multiple cache lines for the same physical address from existing at once, you have this problem.

If your D-cache has this problem, first define asm/shmparam.h SHMLBA properly, it should essentially be the size of your virtually addressed D-cache (or if the size is variable, the largest possible size). This setting will force the SYSv IPC layer to only allow user processes to mmap shared memory at address which are a multiple of this value.

Quoted 8 lines

or

B) to flush both the kernel and userspace mappings when a ring buffer page is handed over from writer to reader ?

I've seen both approaches being discussed in the recent threads, with some consensus aiming towards (A), but then the code that follows takes approach (B).

The alignment done is not for cache issues but for memory mapping in general. Having the persistent buffer memory be page aligned and all of it fit nicely in pages is a requirement so that there's nothing truncated.

For the cache issue, (B) is used. Note, all mappings to user space is from the page allocator. As reserve_mem has it allocated but just not free to use.

Quoted 4 lines

AFAIU, it we are aiming for approach (A), then I'm missing where vmap_page_range() guarantees that the _kernel_ virtual mapping is SHMLBA aligned. AFAIU, only user mappings are aligned on SHMLBA.

The code in this patch series makes vmap_page_range() only happen when memmap= is used to allocate the buffer. And then it sets the TRACE_ARRAY_FL_MEMMAP flag which tells the the tracing code "Do not ever memory map his buffer to user space".

Quoted 6 lines

And if we aiming towards approach (A), then the explicit flushing is not needed when handing over pages from writer to reader, but an SHMLBA alignment should be introduced.

Please let me know if I'm missing something,

I guess what you are missing is that the code no longer is aimed at having vmap_page_range() ever get mapped to user space ;-)

-- Steve

#10 lore raw
Quoted 2 lines · Steven Rostedt

On Wed, 2 Apr 2025 10:56:51 -0400
Mathieu Desnoyers <[email protected]> wrote:

[...]

Quoted 4 lines
Quoted 1 line

Please let me know if I'm missing something,

I guess what you are missing is that the code no longer is aimed at having vmap_page_range() ever get mapped to user space ;-)

Thanks for the clarification!

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com