Skip to contentSkip to conversation

ring-buffer: Memmory mapping the ring buffer to user space

PATCH New Steven Rostedt 1 participant 7 messages May 14, 2024

Subsystems (5)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover ring-buffer: Memmory mapping the ring buffer to user space

ring-buffer/for-next

Head SHA1: fe832be05a8eee5f1488cbcc2c562dd82d079fd6

Steven Rostedt (Google) (1):
ring-buffer: Have mmapped ring buffer keep track of missed events

Vincent Donnefort (5):
ring-buffer: Allocate sub-buffers with __GFP_COMP
ring-buffer: Introducing ring-buffer mapping functions
tracing: Allow user-space mapping of the ring-buffer
Documentation: tracing: Add ring-buffer mapping ring-buffer/selftest: Add ring-buffer mapping test

----

 Documentation/trace/index.rst                  |   1 +
 Documentation/trace/ring-buffer-map.rst        | 106 ++++++
 include/linux/ring_buffer.h                    |   6 +
 include/uapi/linux/trace_mmap.h                |  48 +++
 kernel/trace/ring_buffer.c                     | 471 ++++++++++++++++++++++++-
 kernel/trace/trace.c                           | 104 +++++-
 kernel/trace/trace.h                           |   1 +
 tools/testing/selftests/ring-buffer/.gitignore |   1 +
 tools/testing/selftests/ring-buffer/Makefile   |   8 +
 tools/testing/selftests/ring-buffer/config     |   2 +
 tools/testing/selftests/ring-buffer/map_test.c | 294 +++++++++++++++
 11 files changed, 1026 insertions(+), 16 deletions(-)
 create mode 100644 Documentation/trace/ring-buffer-map.rst
 create mode 100644 include/uapi/linux/trace_mmap.h
 create mode 100644 tools/testing/selftests/ring-buffer/.gitignore
 create mode 100644 tools/testing/selftests/ring-buffer/Makefile
 create mode 100644 tools/testing/selftests/ring-buffer/config
 create mode 100644 tools/testing/selftests/ring-buffer/map_test.c
author maintainer
#1 lore raw

PATCH 1/6 ring-buffer: Allocate sub-buffers with __GFP_COMP

From: Vincent Donnefort <[email protected]>

In preparation for the ring-buffer memory mapping, allocate compound pages for the ring-buffer sub-buffers to enable us to map them to user-space with vm_insert_pages().

---
 kernel/trace/ring_buffer.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/trace/ring_buffer.c +3 -3
index 6511dc3a00da..a1e011733f6a 100644
@@ -1524,7 +1524,7 @@ static int __rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,
 		list_add(&bpage->list, pages);
 
 		page = alloc_pages_node(cpu_to_node(cpu_buffer->cpu),
-					mflags | __GFP_ZERO,
+					mflags | __GFP_COMP | __GFP_ZERO,
 					cpu_buffer->buffer->subbuf_order);
 		if (!page)
 			goto free_pages;
@@ -1609,7 +1609,7 @@ rb_allocate_cpu_buffer(struct trace_buffer *buffer, long nr_pages, int cpu)
 
 	cpu_buffer->reader_page = bpage;
 
-	page = alloc_pages_node(cpu_to_node(cpu), GFP_KERNEL | __GFP_ZERO,
+	page = alloc_pages_node(cpu_to_node(cpu), GFP_KERNEL | __GFP_COMP | __GFP_ZERO,
 				cpu_buffer->buffer->subbuf_order);
 	if (!page)
 		goto fail_free_reader;
@@ -5579,7 +5579,7 @@ ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu)
 		goto out;
 
 	page = alloc_pages_node(cpu_to_node(cpu),
-				GFP_KERNEL | __GFP_NORETRY | __GFP_ZERO,
+				GFP_KERNEL | __GFP_NORETRY | __GFP_COMP | __GFP_ZERO,
 				cpu_buffer->buffer->subbuf_order);
 	if (!page) {
 		kfree(bpage);
Signature · 1 line
2.43.0
author maintainer
#2 lore raw

PATCH 2/6 ring-buffer: Introducing ring-buffer mapping functions

In reply to Steven Rostedt

From: Vincent Donnefort <[email protected]>

In preparation for allowing the user-space to map a ring-buffer, add a set of mapping functions:

ring_buffer_{map,unmap}()

And controls on the ring-buffer:

ring_buffer_map_get_reader() /* swap reader and head */

Mapping the ring-buffer also involves:

A unique ID for each subbuf of the ring-buffer, currently they are only identified through their in-kernel VA.

A meta-page, where are stored ring-buffer statistics and a description for the current reader

The linear mapping exposes the meta-page, and each subbuf of the ring-buffer, ordered following their unique ID, assigned during the first mapping.

Once mapped, no subbuf can get in or out of the ring-buffer: the buffer size will remain unmodified and the splice enabling functions will in reality simply memcpy the data instead of swapping subbufs.

---
 include/linux/ring_buffer.h     |   6 +
 include/uapi/linux/trace_mmap.h |  46 ++++
 kernel/trace/ring_buffer.c      | 414 +++++++++++++++++++++++++++++++-
 3 files changed, 463 insertions(+), 3 deletions(-)
 create mode 100644 include/uapi/linux/trace_mmap.h
include/linux/ring_buffer.h +6 -0
index dc5ae4e96aee..96d2140b471e 100644
@@ -6,6 +6,8 @@
 #include <linux/seq_file.h>
 #include <linux/poll.h>
 
+#include <uapi/linux/trace_mmap.h>
+
 struct trace_buffer;
 struct ring_buffer_iter;
 
@@ -223,4 +225,8 @@ int trace_rb_cpu_prepare(unsigned int cpu, struct hlist_node *node);
 #define trace_rb_cpu_prepare	NULL
 #endif
 
+int ring_buffer_map(struct trace_buffer *buffer, int cpu,
+		    struct vm_area_struct *vma);
+int ring_buffer_unmap(struct trace_buffer *buffer, int cpu);
+int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu);
 #endif /* _LINUX_RING_BUFFER_H */
include/uapi/linux/trace_mmap.h new file +46 -0

46 lines not on this page · raw message

kernel/trace/ring_buffer.c +411 -3

486 lines not on this page · raw message

Signature · 1 line
2.43.0
author maintainer
#3 lore raw

PATCH 3/6 tracing: Allow user-space mapping of the ring-buffer

In reply to Steven Rostedt

From: Vincent Donnefort <[email protected]>

Currently, user-space extracts data from the ring-buffer via splice, which is handy for storage or network sharing. However, due to splice limitations, it is imposible to do real-time analysis without a copy.

A solution for that problem is to let the user-space map the ring-buffer directly.

The mapping is exposed via the per-CPU file trace_pipe_raw. The first element of the mapping is the meta-page. It is followed by each subbuffer constituting the ring-buffer, ordered by their unique page ID:

* Meta-page -- include/uapi/linux/trace_mmap.h for a description
* Subbuf ID 0
* Subbuf ID 1

     ...

It is therefore easy to translate a subbuf ID into an offset in the mapping:

  reader_id = meta->reader->id;
  reader_offset = meta->meta_page_size + reader_id * meta->subbuf_size;

When new data is available, the mapper must call a newly introduced ioctl:
TRACE_MMAP_IOCTL_GET_READER. This will update the Meta-page reader ID to point to the next reader containing unread data.

Mapping will prevent snapshot and buffer size modifications.

---
 include/uapi/linux/trace_mmap.h |   2 +
 kernel/trace/trace.c            | 104 ++++++++++++++++++++++++++++++--
 kernel/trace/trace.h            |   1 +
 3 files changed, 102 insertions(+), 5 deletions(-)
include/uapi/linux/trace_mmap.h +2 -0
index b682e9925539..bd1066754220 100644
@@ -43,4 +43,6 @@ struct trace_buffer_meta {
 	__u64	Reserved2;
 };
 
+#define TRACE_MMAP_IOCTL_GET_READER		_IO('T', 0x1)
+
 #endif /* _TRACE_MMAP_H_ */
kernel/trace/trace.c +99 -5

146 lines not on this page · raw message

kernel/trace/trace.h +1 -0
index 64450615ca0c..749a182dab48 100644
@@ -336,6 +336,7 @@ struct trace_array {
 	bool			allocated_snapshot;
 	spinlock_t		snapshot_trigger_lock;
 	unsigned int		snapshot;
+	unsigned int		mapped;
 	unsigned long		max_latency;
 #ifdef CONFIG_FSNOTIFY
 	struct dentry		*d_max_latency;
Signature · 1 line
2.43.0
author maintainer
#4 lore raw

PATCH 4/6 Documentation: tracing: Add ring-buffer mapping

In reply to Steven Rostedt

From: Vincent Donnefort <[email protected]>

It is now possible to mmap() a ring-buffer to stream its content. Add some documentation and a code example.

---
 Documentation/trace/index.rst           |   1 +
 Documentation/trace/ring-buffer-map.rst | 106 ++++++++++++++++++++++++
 2 files changed, 107 insertions(+)
 create mode 100644 Documentation/trace/ring-buffer-map.rst
Documentation/trace/index.rst +1 -0
index 5092d6c13af5..0b300901fd75 100644
@@ -29,6 +29,7 @@ Linux Tracing Technologies
    timerlat-tracer
    intel_th
    ring-buffer-design
+   ring-buffer-map
    stm
    sys-t
    coresight/index
Documentation/trace/ring-buffer-map.rst new file +106 -0

106 lines not on this page · raw message

Signature · 1 line
2.43.0
author maintainer
#5 lore raw

PATCH 5/6 ring-buffer/selftest: Add ring-buffer mapping test

In reply to Steven Rostedt

From: Vincent Donnefort <[email protected]>

Map a ring-buffer, validate the meta-page before and after emitting few events. Also check ring-buffer mapping boundaries and finally ensure the tracing snapshot is mutually exclusive.

---
 .../testing/selftests/ring-buffer/.gitignore  |   1 +
 tools/testing/selftests/ring-buffer/Makefile  |   8 +
 tools/testing/selftests/ring-buffer/config    |   2 +
 .../testing/selftests/ring-buffer/map_test.c  | 294 ++++++++++++++++++
 4 files changed, 305 insertions(+)
 create mode 100644 tools/testing/selftests/ring-buffer/.gitignore
 create mode 100644 tools/testing/selftests/ring-buffer/Makefile
 create mode 100644 tools/testing/selftests/ring-buffer/config
 create mode 100644 tools/testing/selftests/ring-buffer/map_test.c
tools/testing/selftests/ring-buffer/.gitignore new file +1 -0
new file mode 100644
index 000000000000..3aed1a2a6c67
@@ -0,0 +1 @@
+map_test
tools/testing/selftests/ring-buffer/Makefile new file +8 -0
new file mode 100644
index 000000000000..627c5fa6d1ab
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+CFLAGS += -Wl,-no-as-needed -Wall
+CFLAGS += $(KHDR_INCLUDES)
+CFLAGS += -D_GNU_SOURCE
+
+TEST_GEN_PROGS = map_test
+
+include ../lib.mk
tools/testing/selftests/ring-buffer/config new file +2 -0
new file mode 100644
index 000000000000..d936f8f00e78
@@ -0,0 +1,2 @@
+CONFIG_FTRACE=y
+CONFIG_TRACER_SNAPSHOT=y
tools/testing/selftests/ring-buffer/map_test.c new file +294 -0

294 lines not on this page · raw message

Signature · 1 line
2.43.0
author maintainer
#6 lore raw

PATCH 6/6 ring-buffer: Have mmapped ring buffer keep track of missed events

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

While testing libtracefs on the mmapped ring buffer, the test that checks if missed events are accounted for failed when using the mapped buffer. This is because the mapped page does not update the missed events that were dropped because the writer filled up the ring buffer before the reader could catch it.

Add the missed events to the reader page/sub-buffer when the IOCTL is done and a new reader page is acquired.

Note that all accesses to the reader_page via rb_page_commit() had to be switched to rb_page_size(), and rb_page_size() which was just a copy of rb_page_commit() but now it masks out the RB_MISSED bits. This is needed as the mapped reader page is still active in the ring buffer code and where it reads the commit field of the bpage for the size, it now must mask it otherwise the missed bits that are now set will corrupt the size returned.

Cc Masami Hiramatsu <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Vincent Donnefort <[email protected]>
---
 kernel/trace/ring_buffer.c | 53 +++++++++++++++++++++++++++++++++-----
 1 file changed, 47 insertions(+), 6 deletions(-)
kernel/trace/ring_buffer.c +47 -6
index cb303052bff2..a02c7a52a0f5 100644
@@ -314,6 +314,8 @@ static u64 rb_event_time_stamp(struct ring_buffer_event *event)
 /* Missed count stored at end */
 #define RB_MISSED_STORED	(1 << 30)
 
+#define RB_MISSED_MASK		(3 << 30)
+
 struct buffer_data_page {
 	u64		 time_stamp;	/* page time stamp */
 	local_t		 commit;	/* write committed index */
@@ -2326,7 +2328,7 @@ rb_iter_head_event(struct ring_buffer_iter *iter)
 /* Size is determined by what has been committed */
 static __always_inline unsigned rb_page_size(struct buffer_page *bpage)
 {
-	return rb_page_commit(bpage);
+	return rb_page_commit(bpage) & ~RB_MISSED_MASK;
 }
 
 static __always_inline unsigned
@@ -3953,7 +3955,7 @@ static bool rb_per_cpu_empty(struct ring_buffer_per_cpu *cpu_buffer)
 		return true;
 
 	/* Reader should exhaust content in reader page */
-	if (reader->read != rb_page_commit(reader))
+	if (reader->read != rb_page_size(reader))
 		return false;
 
 	/*
@@ -4424,7 +4426,7 @@ int ring_buffer_iter_empty(struct ring_buffer_iter *iter)
 	return ((iter->head_page == commit_page && iter->head >= commit) ||
 		(iter->head_page == reader && commit_page == head_page &&
 		 head_page->read == commit &&
-		 iter->head == rb_page_commit(cpu_buffer->reader_page)));
+		 iter->head == rb_page_size(cpu_buffer->reader_page)));
 }
 EXPORT_SYMBOL_GPL(ring_buffer_iter_empty);
 
@@ -5753,7 +5755,7 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
 	event = rb_reader_event(cpu_buffer);
 
 	read = reader->read;
-	commit = rb_page_commit(reader);
+	commit = rb_page_size(reader);
 
 	/* Check if any events were dropped */
 	missed_events = cpu_buffer->lost_events;
@@ -5830,7 +5832,7 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
 	} else {
 		/* update the entry counter */
 		cpu_buffer->read += rb_page_entries(reader);
-		cpu_buffer->read_bytes += rb_page_commit(reader);
+		cpu_buffer->read_bytes += rb_page_size(reader);
 
 		/* swap the pages */
 		rb_init_page(bpage);
@@ -6422,6 +6424,8 @@ int ring_buffer_unmap(struct trace_buffer *buffer, int cpu)
 int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu)
 {
 	struct ring_buffer_per_cpu *cpu_buffer;
+	struct buffer_page *reader;
+	unsigned long missed_events;
 	unsigned long reader_size;
 	unsigned long flags;
 
@@ -6448,9 +6452,46 @@ int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu)
 		goto out;
 	}
 
-	if (WARN_ON(!rb_get_reader_page(cpu_buffer)))
+	reader = rb_get_reader_page(cpu_buffer);
+	if (WARN_ON(!reader))
 		goto out;
 
+	/* Check if any events were dropped */
+	missed_events = cpu_buffer->lost_events;
+
+	if (cpu_buffer->reader_page != cpu_buffer->commit_page) {
+		if (missed_events) {
+			struct buffer_data_page *bpage = reader->page;
+			unsigned int commit;
+			/*
+			 * Use the real_end for the data size,
+			 * This gives us a chance to store the lost events
+			 * on the page.
+			 */
+			if (reader->real_end)
+				local_set(&bpage->commit, reader->real_end);
+			/*
+			 * If there is room at the end of the page to save the
+			 * missed events, then record it there.
+			 */
+			commit = rb_page_size(reader);
+			if (buffer->subbuf_size - commit >= sizeof(missed_events)) {
+				memcpy(&bpage->data[commit], &missed_events,
+				       sizeof(missed_events));
+				local_add(RB_MISSED_STORED, &bpage->commit);
+			}
+			local_add(RB_MISSED_EVENTS, &bpage->commit);
+		}
+	} else {
+		/*
+		 * There really shouldn't be any missed events if the commit
+		 * is on the reader page.
+		 */
+		WARN_ON_ONCE(missed_events);
+	}
+
+	cpu_buffer->lost_events = 0;
+
 	goto consume;
 
 out:
Signature · 1 line
2.43.0