Skip to contentSkip to conversation

Introducing trace buffer mapping by user-space

PATCH In review Vincent Donnefort 6 participants 17 messages May 10, 2024 – Jul 3, 2024

Subsystems (6)
lore
Share & tools
lore

Vincent Donnefort

author maintainer
#0 lore raw

v23 cover Introducing trace buffer mapping by user-space

The tracing ring-buffers can be stored on disk or sent to network without any copy via splice. However the later doesn't allow real time processing of the traces. A solution is to give userspace direct access to the ring-buffer pages via a mapping. An application can now become a consumer of the ring-buffer, in a similar fashion to what trace_pipe offers.

Support for this new feature can already be found in libtracefs from version 1.8, when built with EXTRA_CFLAGS=-DFORCE_MMAP_ENABLE.

Vincent

v22 -> v23:
* Remove VM_IO (Prevent ptrace and does not bring any other useful protection).
* Bring back DONTDUMP (As we removed VM_IO).
* Add a check for root user in map_test.
* Make the s/nr_subbufs check a WARN.

v21 -> v22:
* Remove DONTDUMP (VM_IO implies DONTDUMP already)
* Remove MIXEDMAP (implicit when using vm_insert_page)
* Remove PFNMAP (We do not perform raw PFN mappings and MIXEDMAP is already implicitely set)
* Add comments to justify the VM_* flags

v20 -> v21:
* Collect Ack
* Add .gitignore
* Few nits
* Remove meta-page padding (zero-page not supported by vm_insert_pages)
* Remove single-usage macros
* Move vma flags handling into ring-buffer.c

v19 -> v20:
* Fix typos in documentation.
* Remove useless mmap open and fault callbacks.
* add mm.h include for vm_insert_pages

v18 -> v19:
* Use VM_PFNMAP and vm_insert_pages
* Allocate ring-buffer subbufs with __GFP_COMP
* Pad the meta-page with the zero-page to align on the subbuf_order
* Extend the ring-buffer test with mmap() dedicated suite

v17 -> v18:
* Fix lockdep_assert_held
* Fix spin_lock_init typo
* Fix CONFIG_TRACER_MAX_TRACE typo

v16 -> v17:
* Documentation and comments improvements.
* Create get/put_snapshot_map() for clearer code.
* Replace kzalloc with kcalloc.
* Fix -ENOMEM handling in rb_alloc_meta_page().
* Move flush(cpu_buffer->reader_page) behind the reader lock.
* Move all inc/dec of cpu_buffer->mapped behind reader lock and buffer mutex. (removes READ_ONCE/WRITE_ONCE accesses).

v15 -> v16:
* Add comment for the dcache flush.
* Remove now unnecessary WRITE_ONCE for the meta-page.

v14 -> v15:
* Add meta-page and reader-page flush. Intends to fix the mapping for VIVT and aliasing-VIPT data caches.
* -EPERM on VM_EXEC.
* Fix build warning !CONFIG_TRACER_MAX_TRACE.

v13 -> v14:
* All cpu_buffer->mapped readers use READ_ONCE (except for swap_cpu)
* on unmap, sync meta-page teardown with the reader_lock instead of the synchronize_rcu.
* Add a dedicated spinlock for trace_array ->snapshot and ->mapped. (intends to fix a lockdep issue)
* Add kerneldoc for flags and Reserved fields.
* Add kselftest for snapshot/map mutual exclusion.

v12 -> v13:
* Swap subbufs_{touched,lost} for Reserved fields.
* Add a flag field in the meta-page.
* Fix CONFIG_TRACER_MAX_TRACE.
* Rebase on top of trace/urgent.
* Add a comment for try_unregister_trigger()

v11 -> v12:
* Fix code sample mmap bug.
* Add logging in sample code.
* Reset tracer in selftest.
* Add a refcount for the snapshot users.
* Prevent mapping when there are snapshot users and vice versa.
* Refine the meta-page.
* Fix types in the meta-page.
* Collect Reviewed-by.

v10 -> v11:
* Add Documentation and code sample.
* Add a selftest.
* Move all the update to the meta-page into a single

    rb_update_meta_page().

* rb_update_meta_page() is now called from ring_buffer_map_get_reader() to fix NOBLOCK callers.
* kerneldoc for struct trace_meta_page.
* Add a patch to zero all the ring-buffer allocations.

v9 -> v10:
* Refactor rb_update_meta_page()
* In-loop declaration for foreach_subbuf_page()
* Check for cpu_buffer->mapped overflow

v8 -> v9:
* Fix the unlock path in ring_buffer_map()
* Fix cpu_buffer cast with rb_work_rq->is_cpu_buffer
* Rebase on linux-trace/for-next (3cb3091138ca0921c4569bcf7ffa062519639b6a)

v7 -> v8:
* Drop the subbufs renaming into bpages
* Use subbuf as a name when relevant

v6 -> v7:
* Rebase onto lore.kernel.org/lkml/[email protected]/
* Support for subbufs
* Rename subbufs into bpages

v5 -> v6:
* Rebase on next-20230802.
* (unsigned long) -> (void *) cast for virt_to_page().
* Add a wait for the GET_READER_PAGE ioctl.
* Move writer fields update (overrun/pages_lost/entries/pages_touched) in the irq_work.
* Rearrange id in struct buffer_page.
* Rearrange the meta-page.
* ring_buffer_meta_page -> trace_buffer_meta_page.
* Add meta_struct_len into the meta-page.

v4 -> v5:
* Trivial rebase onto 6.5-rc3 (previously 6.4-rc3)

v3 -> v4:
* Add to the meta-page:
- pages_lost / pages_read (allow to compute how full is the ring-buffer)
- read (allow to compute how many entries can be read)
- A reader_page struct.
* Rename ring_buffer_meta_header -> ring_buffer_meta
* Rename ring_buffer_get_reader_page -> ring_buffer_map_get_reader_page
* Properly consume events on ring_buffer_map_get_reader_page() with rb_advance_reader().

v2 -> v3:
* Remove data page list (for non-consuming read)
** Implies removing order > 0 meta-page
* Add a new meta page field ->read
* Rename ring_buffer_meta_page_header into ring_buffer_meta_header

v1 -> v2:
* Hide data_pages from the userspace struct
* Fix META_PAGE_MAX_PAGES
* Support for order > 0 meta-page
* Add missing page->mapping.

Vincent Donnefort (5):
ring-buffer: Allocate sub-buffers with __GFP_COMP
ring-buffer: Introducing ring-buffer mapping functions
tracing: Allow user-space mapping of the ring-buffer
Documentation: tracing: Add ring-buffer mapping ring-buffer/selftest: Add ring-buffer mapping test

 Documentation/trace/index.rst                 |   1 +
 Documentation/trace/ring-buffer-map.rst       | 106 +++++
 include/linux/ring_buffer.h                   |   6 +
 include/uapi/linux/trace_mmap.h               |  48 ++
 kernel/trace/ring_buffer.c                    | 420 +++++++++++++++++-
 kernel/trace/trace.c                          | 104 ++++-
 kernel/trace/trace.h                          |   1 +
 .../testing/selftests/ring-buffer/.gitignore  |   1 +
 tools/testing/selftests/ring-buffer/Makefile  |   8 +
 tools/testing/selftests/ring-buffer/config    |   2 +
 .../testing/selftests/ring-buffer/map_test.c  | 294 ++++++++++++
 11 files changed, 980 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/trace/ring-buffer-map.rst
 create mode 100644 include/uapi/linux/trace_mmap.h
 create mode 100644 tools/testing/selftests/ring-buffer/.gitignore
 create mode 100644 tools/testing/selftests/ring-buffer/Makefile
 create mode 100644 tools/testing/selftests/ring-buffer/config
 create mode 100644 tools/testing/selftests/ring-buffer/map_test.c

base-commit: 7604256cecef34a82333d9f78262d3180f4eb525

Signature · 1 line
2.45.0.118.g7fe29c98d7-goog
author maintainer
#1 lore raw

PATCH v23 1/5 ring-buffer: Allocate sub-buffers with __GFP_COMP

In preparation for the ring-buffer memory mapping, allocate compound pages for the ring-buffer sub-buffers to enable us to map them to user-space with vm_insert_pages().

kernel/trace/ring_buffer.c +3 -3
index 25476ead681b..cc9ebe593571 100644
@@ -1524,7 +1524,7 @@ static int __rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,
 		list_add(&bpage->list, pages);
 
 		page = alloc_pages_node(cpu_to_node(cpu_buffer->cpu),
-					mflags | __GFP_ZERO,
+					mflags | __GFP_COMP | __GFP_ZERO,
 					cpu_buffer->buffer->subbuf_order);
 		if (!page)
 			goto free_pages;
@@ -1609,7 +1609,7 @@ rb_allocate_cpu_buffer(struct trace_buffer *buffer, long nr_pages, int cpu)
 
 	cpu_buffer->reader_page = bpage;
 
-	page = alloc_pages_node(cpu_to_node(cpu), GFP_KERNEL | __GFP_ZERO,
+	page = alloc_pages_node(cpu_to_node(cpu), GFP_KERNEL | __GFP_COMP | __GFP_ZERO,
 				cpu_buffer->buffer->subbuf_order);
 	if (!page)
 		goto fail_free_reader;
@@ -5579,7 +5579,7 @@ ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu)
 		goto out;
 
 	page = alloc_pages_node(cpu_to_node(cpu),
-				GFP_KERNEL | __GFP_NORETRY | __GFP_ZERO,
+				GFP_KERNEL | __GFP_NORETRY | __GFP_COMP | __GFP_ZERO,
 				cpu_buffer->buffer->subbuf_order);
 	if (!page) {
 		kfree(bpage);
Signature · 1 line
2.45.0.118.g7fe29c98d7-goog
author maintainer
#2 lore raw

PATCH v23 2/5 ring-buffer: Introducing ring-buffer mapping functions

In reply to Vincent Donnefort

In preparation for allowing the user-space to map a ring-buffer, add a set of mapping functions:

ring_buffer_{map,unmap}()

And controls on the ring-buffer:

ring_buffer_map_get_reader() /* swap reader and head */

Mapping the ring-buffer also involves:

A unique ID for each subbuf of the ring-buffer, currently they are only identified through their in-kernel VA.

A meta-page, where are stored ring-buffer statistics and a description for the current reader

The linear mapping exposes the meta-page, and each subbuf of the ring-buffer, ordered following their unique ID, assigned during the first mapping.

Once mapped, no subbuf can get in or out of the ring-buffer: the buffer size will remain unmodified and the splice enabling functions will in reality simply memcpy the data instead of swapping subbufs.

include/linux/ring_buffer.h +6 -0
index dc5ae4e96aee..96d2140b471e 100644
@@ -6,6 +6,8 @@
 #include <linux/seq_file.h>
 #include <linux/poll.h>
 
+#include <uapi/linux/trace_mmap.h>
+
 struct trace_buffer;
 struct ring_buffer_iter;
 
@@ -223,4 +225,8 @@ int trace_rb_cpu_prepare(unsigned int cpu, struct hlist_node *node);
 #define trace_rb_cpu_prepare	NULL
 #endif
 
+int ring_buffer_map(struct trace_buffer *buffer, int cpu,
+		    struct vm_area_struct *vma);
+int ring_buffer_unmap(struct trace_buffer *buffer, int cpu);
+int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu);
 #endif /* _LINUX_RING_BUFFER_H */
include/uapi/linux/trace_mmap.h new file +46 -0

46 lines not on this page · raw message

kernel/trace/ring_buffer.c +411 -3

486 lines not on this page · raw message

Signature · 1 line
2.45.0.118.g7fe29c98d7-goog
Acked-by on v23 2/5
#3 lore raw

Re: [PATCH v23 2/5] ring-buffer: Introducing ring-buffer mapping functions

Quoted 365 lines · Vincent Donnefort

In preparation for allowing the user-space to map a ring-buffer, add a set of mapping functions:

   ring_buffer_{map,unmap}()

And controls on the ring-buffer:

ring_buffer_map_get_reader() /* swap reader and head */

Mapping the ring-buffer also involves:

A unique ID for each subbuf of the ring-buffer, currently they are only identified through their in-kernel VA.

A meta-page, where are stored ring-buffer statistics and a description for the current reader

The linear mapping exposes the meta-page, and each subbuf of the ring-buffer, ordered following their unique ID, assigned during the first mapping.

Once mapped, no subbuf can get in or out of the ring-buffer: the buffer size will remain unmodified and the splice enabling functions will in reality simply memcpy the data instead of swapping subbufs.

include/linux/ring_buffer.h +6 -0
index dc5ae4e96aee..96d2140b471e 100644
@@ -6,6 +6,8 @@
  #include <linux/seq_file.h>
  #include <linux/poll.h>
 
+#include <uapi/linux/trace_mmap.h>
+
  struct trace_buffer;
  struct ring_buffer_iter;
 
@@ -223,4 +225,8 @@ int trace_rb_cpu_prepare(unsigned int cpu, struct hlist_node *node);
  #define trace_rb_cpu_prepare	NULL
  #endif
 
+int ring_buffer_map(struct trace_buffer *buffer, int cpu,
+		    struct vm_area_struct *vma);
+int ring_buffer_unmap(struct trace_buffer *buffer, int cpu);
+int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu);
  #endif /* _LINUX_RING_BUFFER_H */
include/uapi/linux/trace_mmap.h new file +46 -0
new file mode 100644
index 000000000000..b682e9925539
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _TRACE_MMAP_H_
+#define _TRACE_MMAP_H_
+
+#include <linux/types.h>
+
+/**
+ * struct trace_buffer_meta - Ring-buffer Meta-page description
+ * @meta_page_size:	Size of this meta-page.
+ * @meta_struct_len:	Size of this structure.
+ * @subbuf_size:	Size of each sub-buffer.
+ * @nr_subbufs:		Number of subbfs in the ring-buffer, including the reader.
+ * @reader.lost_events:	Number of events lost at the time of the reader swap.
+ * @reader.id:		subbuf ID of the current reader. ID range [0 : @nr_subbufs - 1]
+ * @reader.read:	Number of bytes read on the reader subbuf.
+ * @flags:		Placeholder for now, 0 until new features are supported.
+ * @entries:		Number of entries in the ring-buffer.
+ * @overrun:		Number of entries lost in the ring-buffer.
+ * @read:		Number of entries that have been read.
+ * @Reserved1:		Internal use only.
+ * @Reserved2:		Internal use only.
+ */
+struct trace_buffer_meta {
+	__u32		meta_page_size;
+	__u32		meta_struct_len;
+
+	__u32		subbuf_size;
+	__u32		nr_subbufs;
+
+	struct {
+		__u64	lost_events;
+		__u32	id;
+		__u32	read;
+	} reader;
+
+	__u64	flags;
+
+	__u64	entries;
+	__u64	overrun;
+	__u64	read;
+
+	__u64	Reserved1;
+	__u64	Reserved2;
+};
+
+#endif /* _TRACE_MMAP_H_ */
kernel/trace/ring_buffer.c +175 -3
index cc9ebe593571..849f6942bb96 100644
@@ -9,6 +9,7 @@
  #include <linux/ring_buffer.h>
  #include <linux/trace_clock.h>
  #include <linux/sched/clock.h>
+#include <linux/cacheflush.h>
  #include <linux/trace_seq.h>
  #include <linux/spinlock.h>
  #include <linux/irq_work.h>
@@ -26,6 +27,7 @@
  #include <linux/list.h>
  #include <linux/cpu.h>
  #include <linux/oom.h>
+#include <linux/mm.h>
 
  #include <asm/local64.h>
  #include <asm/local.h>
@@ -338,6 +340,7 @@ struct buffer_page {
  	local_t		 entries;	/* entries on this page */
  	unsigned long	 real_end;	/* real end of data */
  	unsigned	 order;		/* order of the page */
+	u32		 id;		/* ID for external mapping */
  	struct buffer_data_page *page;	/* Actual data page */
  };
 
@@ -484,6 +487,12 @@ struct ring_buffer_per_cpu {
  	u64				read_stamp;
  	/* pages removed since last reset */
  	unsigned long			pages_removed;
+
+	unsigned int			mapped;
+	struct mutex			mapping_lock;
+	unsigned long			*subbuf_ids;	/* ID to subbuf VA */
+	struct trace_buffer_meta	*meta_page;
+
  	/* ring buffer pages to update, > 0 to add, < 0 to remove */
  	long				nr_pages_to_update;
  	struct list_head		new_pages; /* new pages to add */
@@ -1599,6 +1608,7 @@ rb_allocate_cpu_buffer(struct trace_buffer *buffer, long nr_pages, int cpu)
  	init_irq_work(&cpu_buffer->irq_work.work, rb_wake_up_waiters);
  	init_waitqueue_head(&cpu_buffer->irq_work.waiters);
  	init_waitqueue_head(&cpu_buffer->irq_work.full_waiters);
+	mutex_init(&cpu_buffer->mapping_lock);
 
  	bpage = kzalloc_node(ALIGN(sizeof(*bpage), cache_line_size()),
  			    GFP_KERNEL, cpu_to_node(cpu));
@@ -1789,8 +1799,6 @@ bool ring_buffer_time_stamp_abs(struct trace_buffer *buffer)
  	return buffer->time_stamp_abs;
  }
 
-static void rb_reset_cpu(struct ring_buffer_per_cpu *cpu_buffer);
-
  static inline unsigned long rb_page_entries(struct buffer_page *bpage)
  {
  	return local_read(&bpage->entries) & RB_WRITE_MASK;
@@ -5211,6 +5219,22 @@ static void rb_clear_buffer_page(struct buffer_page *page)
  	page->read = 0;
  }
 
+static void rb_update_meta_page(struct ring_buffer_per_cpu *cpu_buffer)
+{
+	struct trace_buffer_meta *meta = cpu_buffer->meta_page;
+
+	meta->reader.read = cpu_buffer->reader_page->read;
+	meta->reader.id = cpu_buffer->reader_page->id;
+	meta->reader.lost_events = cpu_buffer->lost_events;
+
+	meta->entries = local_read(&cpu_buffer->entries);
+	meta->overrun = local_read(&cpu_buffer->overrun);
+	meta->read = cpu_buffer->read;
+
+	/* Some archs do not have data cache coherency between kernel and user-space */
+	flush_dcache_folio(virt_to_folio(cpu_buffer->meta_page));
+}
+
  static void
  rb_reset_cpu(struct ring_buffer_per_cpu *cpu_buffer)
  {
@@ -5255,6 +5279,9 @@ rb_reset_cpu(struct ring_buffer_per_cpu *cpu_buffer)
  	cpu_buffer->lost_events = 0;
  	cpu_buffer->last_overrun = 0;
 
+	if (cpu_buffer->mapped)
+		rb_update_meta_page(cpu_buffer);
+
  	rb_head_page_activate(cpu_buffer);
  	cpu_buffer->pages_removed = 0;
  }
@@ -5469,6 +5496,12 @@ int ring_buffer_swap_cpu(struct trace_buffer *buffer_a,
  	cpu_buffer_a = buffer_a->buffers[cpu];
  	cpu_buffer_b = buffer_b->buffers[cpu];
 
+	/* It's up to the callers to not try to swap mapped buffers */
+	if (WARN_ON_ONCE(cpu_buffer_a->mapped || cpu_buffer_b->mapped)) {
+		ret = -EBUSY;
+		goto out;
+	}
+
  	/* At least make sure the two buffers are somewhat the same */
  	if (cpu_buffer_a->nr_pages != cpu_buffer_b->nr_pages)
  		goto out;
@@ -5733,7 +5766,8 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
  	 * Otherwise, we can simply swap the page with the one passed in.
  	 */
  	if (read || (len < (commit - read)) ||
-	    cpu_buffer->reader_page == cpu_buffer->commit_page) {
+	    cpu_buffer->reader_page == cpu_buffer->commit_page ||
+	    cpu_buffer->mapped) {
  		struct buffer_data_page *rpage = cpu_buffer->reader_page->page;
  		unsigned int rpos = read;
  		unsigned int pos = 0;
@@ -5956,6 +5990,11 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)
 
  		cpu_buffer = buffer->buffers[cpu];
 
+		if (cpu_buffer->mapped) {
+			err = -EBUSY;
+			goto error;
+		}
+
  		/* Update the number of pages to match the new size */
  		nr_pages = old_size * buffer->buffers[cpu]->nr_pages;
  		nr_pages = DIV_ROUND_UP(nr_pages, buffer->subbuf_size);
@@ -6057,6 +6096,375 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)
  }
  EXPORT_SYMBOL_GPL(ring_buffer_subbuf_order_set);
 
+static int rb_alloc_meta_page(struct ring_buffer_per_cpu *cpu_buffer)
+{
+	struct page *page;
+
+	if (cpu_buffer->meta_page)
+		return 0;
+
+	page = alloc_page(GFP_USER | __GFP_ZERO);
+	if (!page)
+		return -ENOMEM;
+
+	cpu_buffer->meta_page = page_to_virt(page);
+
+	return 0;
+}
+
+static void rb_free_meta_page(struct ring_buffer_per_cpu *cpu_buffer)
+{
+	unsigned long addr = (unsigned long)cpu_buffer->meta_page;
+
+	free_page(addr);
+	cpu_buffer->meta_page = NULL;
+}
+
+static void rb_setup_ids_meta_page(struct ring_buffer_per_cpu *cpu_buffer,
+				   unsigned long *subbuf_ids)
+{
+	struct trace_buffer_meta *meta = cpu_buffer->meta_page;
+	unsigned int nr_subbufs = cpu_buffer->nr_pages + 1;
+	struct buffer_page *first_subbuf, *subbuf;
+	int id = 0;
+
+	subbuf_ids[id] = (unsigned long)cpu_buffer->reader_page->page;
+	cpu_buffer->reader_page->id = id++;
+
+	first_subbuf = subbuf = rb_set_head_page(cpu_buffer);
+	do {
+		if (WARN_ON(id >= nr_subbufs))
+			break;
+
+		subbuf_ids[id] = (unsigned long)subbuf->page;
+		subbuf->id = id;
+
+		rb_inc_page(&subbuf);
+		id++;
+	} while (subbuf != first_subbuf);
+
+	/* install subbuf ID to kern VA translation */
+	cpu_buffer->subbuf_ids = subbuf_ids;
+
+	meta->meta_page_size = PAGE_SIZE;
+	meta->meta_struct_len = sizeof(*meta);
+	meta->nr_subbufs = nr_subbufs;
+	meta->subbuf_size = cpu_buffer->buffer->subbuf_size + BUF_PAGE_HDR_SIZE;
+
+	rb_update_meta_page(cpu_buffer);
+}
+
+static struct ring_buffer_per_cpu *
+rb_get_mapped_buffer(struct trace_buffer *buffer, int cpu)
+{
+	struct ring_buffer_per_cpu *cpu_buffer;
+
+	if (!cpumask_test_cpu(cpu, buffer->cpumask))
+		return ERR_PTR(-EINVAL);
+
+	cpu_buffer = buffer->buffers[cpu];
+
+	mutex_lock(&cpu_buffer->mapping_lock);
+
+	if (!cpu_buffer->mapped) {
+		mutex_unlock(&cpu_buffer->mapping_lock);
+		return ERR_PTR(-ENODEV);
+	}
+
+	return cpu_buffer;
+}
+
+static void rb_put_mapped_buffer(struct ring_buffer_per_cpu *cpu_buffer)
+{
+	mutex_unlock(&cpu_buffer->mapping_lock);
+}
+
+/*
+ * Fast-path for rb_buffer_(un)map(). Called whenever the meta-page doesn't need
+ * to be set-up or torn-down.
+ */
+static int __rb_inc_dec_mapped(struct ring_buffer_per_cpu *cpu_buffer,
+			       bool inc)
+{
+	unsigned long flags;
+
+	lockdep_assert_held(&cpu_buffer->mapping_lock);
+
+	if (inc && cpu_buffer->mapped == UINT_MAX)
+		return -EBUSY;
+
+	if (WARN_ON(!inc && cpu_buffer->mapped == 0))
+		return -EINVAL;
+
+	mutex_lock(&cpu_buffer->buffer->mutex);
+	raw_spin_lock_irqsave(&cpu_buffer->reader_lock, flags);
+
+	if (inc)
+		cpu_buffer->mapped++;
+	else
+		cpu_buffer->mapped--;
+
+	raw_spin_unlock_irqrestore(&cpu_buffer->reader_lock, flags);
+	mutex_unlock(&cpu_buffer->buffer->mutex);
+
+	return 0;
+}
+
+/*
+ *   +--------------+  pgoff == 0
+ *   |   meta page  |
+ *   +--------------+  pgoff == 1
+ *   | subbuffer 0  |
+ *   |              |
+ *   +--------------+  pgoff == (1 + (1 << subbuf_order))
+ *   | subbuffer 1  |
+ *   |              |
+ *         ...
+ */
+#ifdef CONFIG_MMU
+static int __rb_map_vma(struct ring_buffer_per_cpu *cpu_buffer,
+			struct vm_area_struct *vma)
+{
+	unsigned long nr_subbufs, nr_pages, vma_pages, pgoff = vma->vm_pgoff;
+	unsigned int subbuf_pages, subbuf_order;
+	struct page **pages;
+	int p = 0, s = 0;

kernel/trace/ring_buffer.c int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order) · line 6241

Quoted 10 lines
kernel/trace/ring_buffer.c +10 -0
+	int err;
+
+	/* Refuse MP_PRIVATE or writable mappings */
+	if (vma->vm_flags & VM_WRITE || vma->vm_flags & VM_EXEC ||
+	    !(vma->vm_flags & VM_MAYSHARE))
+		return -EPERM;
+
+	/*
+	 * Make sure the mapping cannot become writable later. Also tell the VM
+	 * to not touch these pages (VM_DONTCOPY | VM_DONTEXPAND).

Coment a bit outdated.

Signature · 3 lines
Cheers,

David / dhildenb
author maintainer
#4 lore raw

PATCH v23 3/5 tracing: Allow user-space mapping of the ring-buffer

In reply to Vincent Donnefort

Currently, user-space extracts data from the ring-buffer via splice, which is handy for storage or network sharing. However, due to splice limitations, it is imposible to do real-time analysis without a copy.

A solution for that problem is to let the user-space map the ring-buffer directly.

The mapping is exposed via the per-CPU file trace_pipe_raw. The first element of the mapping is the meta-page. It is followed by each subbuffer constituting the ring-buffer, ordered by their unique page ID:

* Meta-page -- include/uapi/linux/trace_mmap.h for a description
* Subbuf ID 0
* Subbuf ID 1

     ...

It is therefore easy to translate a subbuf ID into an offset in the mapping:

  reader_id = meta->reader->id;
  reader_offset = meta->meta_page_size + reader_id * meta->subbuf_size;

When new data is available, the mapper must call a newly introduced ioctl:
TRACE_MMAP_IOCTL_GET_READER. This will update the Meta-page reader ID to point to the next reader containing unread data.

Mapping will prevent snapshot and buffer size modifications.

include/uapi/linux/trace_mmap.h +2 -0
index b682e9925539..bd1066754220 100644
@@ -43,4 +43,6 @@ struct trace_buffer_meta {
 	__u64	Reserved2;
 };
 
+#define TRACE_MMAP_IOCTL_GET_READER		_IO('T', 0x1)
+
 #endif /* _TRACE_MMAP_H_ */
kernel/trace/trace.c +99 -5

146 lines not on this page · raw message

kernel/trace/trace.h +1 -0
index 64450615ca0c..749a182dab48 100644
@@ -336,6 +336,7 @@ struct trace_array {
 	bool			allocated_snapshot;
 	spinlock_t		snapshot_trigger_lock;
 	unsigned int		snapshot;
+	unsigned int		mapped;
 	unsigned long		max_latency;
 #ifdef CONFIG_FSNOTIFY
 	struct dentry		*d_max_latency;
Signature · 1 line
2.45.0.118.g7fe29c98d7-goog
#5 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

On Fri, May 10, 2024 at 03:04:32PM +0100, Vincent Donnefort wrote:
[...]

include/uapi/linux/trace_mmap.h struct trace_buffer_meta { · line 47

Quoted 10 lines
include/uapi/linux/trace_mmap.h +2 -0
index b682e9925539..bd1066754220 100644
@@ -43,4 +43,6 @@ struct trace_buffer_meta {
 	__u64	Reserved2;
 };
 
+#define TRACE_MMAP_IOCTL_GET_READER		_IO('T', 0x1)
+

I'm sorry but among all the numbers this one was probably the least fortunate choice because it collides with TCGETS on most of architectures.

For example, this is how strace output would look like when TRACE_MMAP_IOCTL_GET_READER support is added:

$ strace -e ioctl stty
ioctl(0, TCGETS or TRACE_MMAP_IOCTL_GET_READER, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0

Even though ioctl numbers are inherently not unique, TCGETS is a very traditional one, so it would be great if you could change TRACE_MMAP_IOCTL_GET_READER to avoid this collision.

Given that _IO('T', 0x1) is _IOC(_IOC_NONE, 'T', 0x1, 0), something like _IOC(_IOC_NONE, 'T', 0x1, 0x1) should be OK.

Signature · 1 line
ldv
maintainer
#6 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

Quoted 10 lines · On Sun, 30 Jun 2024 13:53:23 +0300 "Dmitry V. Levin" <[email protected]> wrote:

On Fri, May 10, 2024 at 03:04:32PM +0100, Vincent Donnefort wrote:
[...]

Quoted 8 lines
include/uapi/linux/trace_mmap.h +0 -0
index b682e9925539..bd1066754220 100644
@@ -43,4 +43,6 @@ struct trace_buffer_meta {
 	__u64	Reserved2;
 };
 
Quoted 5 lines

include/uapi/linux/trace_mmap.h struct trace_buffer_meta { · line 47

Quoted 2 lines
include/uapi/linux/trace_mmap.h +2 -0
+#define TRACE_MMAP_IOCTL_GET_READER		_IO('T', 0x1)
+  

I'm sorry but among all the numbers this one was probably the least fortunate choice because it collides with TCGETS on most of architectures.

Hmm, that is unfortunate.

Quoted 7 lines

For example, this is how strace output would look like when TRACE_MMAP_IOCTL_GET_READER support is added:

$ strace -e ioctl stty
ioctl(0, TCGETS or TRACE_MMAP_IOCTL_GET_READER, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
Quoted 6 lines

Even though ioctl numbers are inherently not unique, TCGETS is a very traditional one, so it would be great if you could change TRACE_MMAP_IOCTL_GET_READER to avoid this collision.

Given that _IO('T', 0x1) is _IOC(_IOC_NONE, 'T', 0x1, 0), something like _IOC(_IOC_NONE, 'T', 0x1, 0x1) should be OK.

Well, it may not be too late to update this as it hasn't been officially released in 6.10 yet. It's still only in the -rc and the library doesn't rely on this yet (I've been holding off until 6.10 was officially released before releasing the library that uses it).

I can send a patch this week to update it. Or feel free to send a patch yourself.

Thanks,

-- Steve

#7 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

Quoted 35 lines · Steven Rostedt
Quoted 10 lines · On Sun, 30 Jun 2024 13:53:23 +0300 "Dmitry V. Levin" <[email protected]> wrote:

On Fri, May 10, 2024 at 03:04:32PM +0100, Vincent Donnefort wrote:
[...]

Quoted 8 lines
include/uapi/linux/trace_mmap.h +0 -0
index b682e9925539..bd1066754220 100644
@@ -43,4 +43,6 @@ struct trace_buffer_meta {
  	__u64	Reserved2;
  };
 
Quoted 5 lines

include/uapi/linux/trace_mmap.h struct trace_buffer_meta { · line 47

Quoted 2 lines
include/uapi/linux/trace_mmap.h +2 -0
+#define TRACE_MMAP_IOCTL_GET_READER		_IO('T', 0x1)
+

I'm sorry but among all the numbers this one was probably the least fortunate choice because it collides with TCGETS on most of architectures.

Hmm, that is unfortunate.

Quoted 7 lines

For example, this is how strace output would look like when TRACE_MMAP_IOCTL_GET_READER support is added:

$ strace -e ioctl stty
ioctl(0, TCGETS or TRACE_MMAP_IOCTL_GET_READER, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
Quoted 6 lines

Even though ioctl numbers are inherently not unique, TCGETS is a very traditional one, so it would be great if you could change TRACE_MMAP_IOCTL_GET_READER to avoid this collision.

Given that _IO('T', 0x1) is _IOC(_IOC_NONE, 'T', 0x1, 0), something like _IOC(_IOC_NONE, 'T', 0x1, 0x1) should be OK.

Quoted 7 lines

Well, it may not be too late to update this as it hasn't been officially released in 6.10 yet. It's still only in the -rc and the library doesn't rely on this yet (I've been holding off until 6.10 was officially released before releasing the library that uses it).

I can send a patch this week to update it. Or feel free to send a patch yourself.

You need to reserve an unused ioctl Code and Seq# range within:

Documentation/userspace-api/ioctl/ioctl-number.rst

Otherwise this duplicate will confuse all system call instrumentation tooling.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
maintainer
#8 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

Quoted 6 lines · On Tue, 2 Jul 2024 10:36:03 -0400 Mathieu Desnoyers…
Quoted 2 lines

I can send a patch this week to update it. Or feel free to send a patch yourself.

You need to reserve an unused ioctl Code and Seq# range within:

Documentation/userspace-api/ioctl/ioctl-number.rst

Ug, it's been so long, I completely forgot about that file.

Thanks for catching this.

Quoted 3 lines

Otherwise this duplicate will confuse all system call instrumentation tooling.

Agreed, what if we did this then:

-- Steve

Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index a141e8e65c5d..9a97030c6c8d 100644
@@ -186,6 +186,7 @@ Code  Seq#    Include File                                           Comments
 'Q'   all    linux/soundcard.h
 'R'   00-1F  linux/random.h                                          conflict!
 'R'   01     linux/rfkill.h                                          conflict!
+'R'   20-2F  linux/trace_mmap.h
 'R'   C0-DF  net/bluetooth/rfcomm.h
 'R'   E0     uapi/linux/fsl_mc.h
 'S'   all    linux/cdrom.h                                           conflict!
include/uapi/linux/trace_mmap.h +1 -1
index bd1066754220..c102ef35d11e 100644
@@ -43,6 +43,6 @@ struct trace_buffer_meta {
 	__u64	Reserved2;
 };
 
-#define TRACE_MMAP_IOCTL_GET_READER		_IO('T', 0x1)
+#define TRACE_MMAP_IOCTL_GET_READER		_IO('R', 0x20)
 
 #endif /* _TRACE_MMAP_H_ */
#9 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

Quoted 39 lines · Steven Rostedt
Quoted 6 lines · On Tue, 2 Jul 2024 10:36:03 -0400 Mathieu Desnoyers…
Quoted 2 lines

I can send a patch this week to update it. Or feel free to send a patch yourself.

You need to reserve an unused ioctl Code and Seq# range within:

Documentation/userspace-api/ioctl/ioctl-number.rst

Ug, it's been so long, I completely forgot about that file.

Thanks for catching this.

Quoted 3 lines

Otherwise this duplicate will confuse all system call instrumentation tooling.

Agreed, what if we did this then:

-- Steve

Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
index a141e8e65c5d..9a97030c6c8d 100644
@@ -186,6 +186,7 @@ Code  Seq#    Include File                                           Comments
  'Q'   all    linux/soundcard.h
  'R'   00-1F  linux/random.h                                          conflict!
  'R'   01     linux/rfkill.h                                          conflict!
+'R'   20-2F  linux/trace_mmap.h
  'R'   C0-DF  net/bluetooth/rfcomm.h
  'R'   E0     uapi/linux/fsl_mc.h
  'S'   all    linux/cdrom.h                                           conflict!
include/uapi/linux/trace_mmap.h +0 -0
index bd1066754220..c102ef35d11e 100644

include/uapi/linux/trace_mmap.h struct trace_buffer_meta { · line 46

Quoted 5 lines
include/uapi/linux/trace_mmap.h +1 -1
@@ -43,6 +43,6 @@ struct trace_buffer_meta {
  	__u64	Reserved2;
  };
 
-#define TRACE_MMAP_IOCTL_GET_READER		_IO('T', 0x1)
+#define TRACE_MMAP_IOCTL_GET_READER		_IO('R', 0x20)

Note that user events also has this issue: the ioctl is not reserved in the ioctl-number.rst list. See include/uapi/linux/user_events.h:

#define DIAG_IOC_MAGIC '*'

/* Request to register a user_event */
#define DIAG_IOCSREG _IOWR(DIAG_IOC_MAGIC, 0, struct user_reg *)

/* Request to delete a user_event */
#define DIAG_IOCSDEL _IOW(DIAG_IOC_MAGIC, 1, char *)

/* Requests to unregister a user_event */
#define DIAG_IOCSUNREG _IOW(DIAG_IOC_MAGIC, 2, struct user_unreg*)

Where '*' maps to Code 0x2A. Looking at the list I don't see any conflicts there, but we should definitely add it.

If we use '*' for user events already, perhaps we'd want to consider using the same range for the ring buffer ioctls ? Arguably one is about instrumentation and the other is about ring buffer interaction (data transport), but those are both related to tracing.

Thanks,

Mathieu

Quoted 2 lines
include/uapi/linux/trace_mmap.h +0 -0
  #endif /* _TRACE_MMAP_H_ */
Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
#10 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

On Tue, Jul 02, 2024 at 11:32:53AM -0400, Mathieu Desnoyers wrote:
[...]

Quoted 14 lines

Note that user events also has this issue: the ioctl is not reserved in the ioctl-number.rst list. See include/uapi/linux/user_events.h:

#define DIAG_IOC_MAGIC '*'

/* Request to register a user_event */
#define DIAG_IOCSREG _IOWR(DIAG_IOC_MAGIC, 0, struct user_reg *)

/* Request to delete a user_event */
#define DIAG_IOCSDEL _IOW(DIAG_IOC_MAGIC, 1, char *)

/* Requests to unregister a user_event */
#define DIAG_IOCSUNREG _IOW(DIAG_IOC_MAGIC, 2, struct user_unreg*)

Quoted 7 lines

Where '*' maps to Code 0x2A. Looking at the list I don't see any conflicts there, but we should definitely add it.

If we use '*' for user events already, perhaps we'd want to consider using the same range for the ring buffer ioctls ? Arguably one is about instrumentation and the other is about ring buffer interaction (data transport), but those are both related to tracing.

FWIW, I've checked the list of ioctls known to strace and can confirm that currently there are no more users of 0x2a ioctl code besides these three DIAG_* ioctls.

Signature · 1 line
ldv
maintainer
#11 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

In reply to Mathieu Desnoyers
Quoted 4 lines · On Tue, 2 Jul 2024 11:32:53 -0400 Mathieu Desnoyers…

If we use '*' for user events already, perhaps we'd want to consider using the same range for the ring buffer ioctls ? Arguably one is about instrumentation and the other is about ring buffer interaction (data transport), but those are both related to tracing.

Yeah, but I still rather keep them separate.

Beau, care to send a patch adding an entry into that ioctl document for user events?

-- Steve

#12 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Tue, 2 Jul 2024 11:32:53 -0400 Mathieu Desnoyers…

If we use '*' for user events already, perhaps we'd want to consider using the same range for the ring buffer ioctls ? Arguably one is about instrumentation and the other is about ring buffer interaction (data transport), but those are both related to tracing.

Yeah, but I still rather keep them separate.

No objection, I'm OK either way.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
#13 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

In reply to Steven Rostedt
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · On Tue, 2 Jul 2024 11:32:53 -0400 Mathieu Desnoyers…

If we use '*' for user events already, perhaps we'd want to consider using the same range for the ring buffer ioctls ? Arguably one is about instrumentation and the other is about ring buffer interaction (data transport), but those are both related to tracing.

Quoted 6 lines

Yeah, but I still rather keep them separate.

Beau, care to send a patch adding an entry into that ioctl document for user events?

-- Steve

Sure thing, sent one out [1].

Thanks,
-Beau

1. https://lore.kernel.org/linux-doc/[email protected]/

#14 lore raw

Re: [PATCH v23 3/5] tracing: Allow user-space mapping of the ring-buffer

In reply to Steven Rostedt
Quoted 27 lines · Steven Rostedt
Quoted 6 lines · On Tue, 2 Jul 2024 10:36:03 -0400 Mathieu Desnoyers…
Quoted 2 lines

I can send a patch this week to update it. Or feel free to send a patch yourself.

You need to reserve an unused ioctl Code and Seq# range within:

Documentation/userspace-api/ioctl/ioctl-number.rst

Ug, it's been so long, I completely forgot about that file.

Thanks for catching this.

Quoted 3 lines

Otherwise this duplicate will confuse all system call instrumentation tooling.

Agreed, what if we did this then:

-- Steve

Documentation/userspace-api/ioctl/ioctl-number.rst +0 -0
index a141e8e65c5d..9a97030c6c8d 100644

Documentation/userspace-api/ioctl/ioctl-number.rst Code Seq# Include File Comments · line 192

Quoted 7 lines
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
@@ -186,6 +186,7 @@ Code  Seq#    Include File                                           Comments
 'Q'   all    linux/soundcard.h
 'R'   00-1F  linux/random.h                                          conflict!
 'R'   01     linux/rfkill.h                                          conflict!
+'R'   20-2F  linux/trace_mmap.h
 'R'   C0-DF  net/bluetooth/rfcomm.h
 'R'   E0     uapi/linux/fsl_mc.h
 'S'   all    linux/cdrom.h                                           conflict!

Just in case, I've checked the list of ioctls known to strace and can confirm that there are no users of 'R' ioctl code in 0x20..0x2f range yet.

By the way, this file is definitely not up to date, the 'R' part of it should have contained the following:

'R'   00-1F  uapi/linux/random.h                                     conflict!
'R'   01-02  uapi/linux/rfkill.h                                     conflict!
'R'   01-0D  uapi/misc/fastrpc.h                                     conflict!
'R'   C0-DF  net/bluetooth/rfcomm.h
'R'   E0     uapi/linux/fsl_mc.h
Signature · 1 line
ldv
author maintainer
#15 lore raw

PATCH v23 4/5 Documentation: tracing: Add ring-buffer mapping

In reply to Vincent Donnefort

It is now possible to mmap() a ring-buffer to stream its content. Add some documentation and a code example.

Documentation/trace/index.rst +1 -0
index 5092d6c13af5..0b300901fd75 100644
@@ -29,6 +29,7 @@ Linux Tracing Technologies
    timerlat-tracer
    intel_th
    ring-buffer-design
+   ring-buffer-map
    stm
    sys-t
    coresight/index
Documentation/trace/ring-buffer-map.rst new file +106 -0

106 lines not on this page · raw message

Signature · 1 line
2.45.0.118.g7fe29c98d7-goog
author maintainer
#16 lore raw

PATCH v23 5/5 ring-buffer/selftest: Add ring-buffer mapping test

In reply to Vincent Donnefort

Map a ring-buffer, validate the meta-page before and after emitting few events. Also check ring-buffer mapping boundaries and finally ensure the tracing snapshot is mutually exclusive.

tools/testing/selftests/ring-buffer/.gitignore new file +1 -0
new file mode 100644
index 000000000000..3aed1a2a6c67
@@ -0,0 +1 @@
+map_test
tools/testing/selftests/ring-buffer/Makefile new file +8 -0
new file mode 100644
index 000000000000..627c5fa6d1ab
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+CFLAGS += -Wl,-no-as-needed -Wall
+CFLAGS += $(KHDR_INCLUDES)
+CFLAGS += -D_GNU_SOURCE
+
+TEST_GEN_PROGS = map_test
+
+include ../lib.mk
tools/testing/selftests/ring-buffer/config new file +2 -0
new file mode 100644
index 000000000000..d936f8f00e78
@@ -0,0 +1,2 @@
+CONFIG_FTRACE=y
+CONFIG_TRACER_SNAPSHOT=y
tools/testing/selftests/ring-buffer/map_test.c new file +294 -0

294 lines not on this page · raw message

Signature · 1 line
2.45.0.118.g7fe29c98d7-goog