relay revamp v8, full patch

This patch splits the current relay into two separate and completely independent pieces: relay and relay_pagewriter. Here's a one-line description of each piece:

relay - the new relay simply accepts pages of trace data from anywhere in the kernel and collects them into per-cpu lists which can be sent by userspace to disk or network destinations via standard read(2), splice(2) and sendfile(2) file operations, after which the pages are returned to their owner(s).

relay_pagewriter - provides a small set of page-writing functions that allow tracers to write into pages it grabs one-by-one from a per-cpu pool of pages and sends them to relay when each is completely written.

There are of course other features e.g. if you don't want to deal with multiple per-cpu files, there's also a 'global' mode, as there always was. relay_pagewriter also supports a 'flight-recorder' mode where the pages aren't sent to relay but rather returned to the end of the pool; at some later point they can be sent to relay via a simple function designed for that purpose. Similarly, the 'early tracing/late setup' functionality has been retained, but is now much simpler in implementation; the fact that these and other features just seem to 'fall out' of the implementation seems to point to the end design being a little nicer.

Note that because of this separation, there's nothing special about relay_pagewriter - it becomes just another source of trace data (via an actual tracer such as blktrace).

overview
========

Basically, the overall effect of the patchset is to split the current relay into two separate and completely independent pieces: relay and relay_pagewriter. Here's a one-line description of each piece:

relay - the new relay simply accepts pages of trace data from anywhere in the kernel and collects them into per-cpu lists which can be sent by userspace to disk or network destinations via standard read(2), splice(2) and sendfile(2) file operations, after which the pages are returned to their owner(s).

relay_pagewriter - provides a small set of page-writing functions that allow tracers to write into pages it grabs one-by-one from a per-cpu pool of pages and sends them to relay when each is completely written.

There are of course other features e.g. if you don't want to deal with multiple per-cpu files, there's also a 'global' mode, as there always was. relay_pagewriter also supports a 'flight-recorder' mode where the pages aren't sent to relay but rather returned to the end of the pool; at some later point they can be sent to relay via a simple function designed for that purpose. Similarly, the 'early tracing/late setup' functionality has been retained, but is now much simpler in implementation; the fact that these and other features just seem to 'fall out' of the implementation seems to point to the end design being a little nicer.

Note that because of this separation, there's nothing special about relay_pagewriter - it becomes just another source of trace data (via an actual tracer such as blktrace).

Here's an overview of relay and relay_pagewriter:

relay
=====

With the new changes, relay really does become just what its name says and and nothing more - it accepts pages from tracers, and relays the data to userspace via read(2) or splice(2) (and therefore sendfile(2)). It doesn't allocate any buffer space and provides no write functions - those are expected to be supplied by any tracer that might want relay pages of trace data to userspace.

After applying the full patch you should be able to use it to stream your trace data to disk or over the network...

Anyway, here's a brief overview of the new API (see code for details):

- relay_open():

Creates a per-cpu relay channel and by default associates debugfs files with each per-cpu 'buffer'. No buffer space is allocated for the 'buffers', rather they collect pages added by tracers in a list which is drained by read()/splice(), etc. Tracers add pages to the 'buffers' using relay_write_page() and relay_write_pages(). One of the parameters to relay_open() is n_pages_wakeup, which specifies that readers should be woken up every time n_pages have been added; if this is 0, readers are never woken up.

- relay_add_page():

Adds a page of trace data to relay. After it has been consumed by userspace, the tracer is notified by the 'relay page' callback function page_released(). The page passed via the callback can then be re-used by the tracer (see for example the pagewriter code, which simply adds the page back into pagewriter's per-cpu page pool). If the page has been stolen instead (if SPLICE_F_MOVE succeeded, which can't happen in current kernels since support for it isn't there), the page_stolen() callback is called, at which point the tracer can allocate a new page to replace the stolen page (see the pagewriter code, which does this too).

- relay_add_partial_page():

Adds a partial (less than or equal to PAGE_SIZE) page of trace data to relay. Sometimes a less-than-full page needs to be added to relay e.g. the last page at the end of a trace; this allows for that.

- relay_add_pages():

The same as relay_add_page(), but adds a set of pages to relay and guarantees that they'll stay together and remain in the same order they were added.

- relay_close():

Releases unread pages to the tracer(s) and frees the channel.

- relay_flush():

Wakes up readers.

- relay_reset():

Releases unread pages to the tracer(s) and resets the channel state.

That's basically the entire kernel API for relay; the userspace API is of course just read(), splice(), and sendfile().

relay_pagewriter
================

One example of a component that can write pages of trace data and send them to relay would be the original relay write functions and buffers (the no-vmap page-based versions), which have been split out into a new file called relay_pagewriter.c and provide one means of writing into pages and feeding them into relay. blktrace and kvmtrace have been 'ported' over to using pagewriter instead of relay directly.

- pagewriter_open():

Allocates a pool of pages for each cpu and creates a relay channel to send them to. Whenever a new page is needed for writing into, the pagewriter object pulls a page out of the appropriate per-cpu pool and makes it available to tracers to write/reserve into via the pagewriter_write() and pagewriter_reserve() functions. When a page is full, the pagewriter object sends the page to relay via relay_add_page() and grabs another page for the next writes. When relay has finished relaying the pages to userspace, it returns the page via pagewriter's page_released() or page_stolen() callbacks; pagewriter simply returns (and allocates a new page if stolen) the page to the pool.

The number of pages allocated per-cpu is specified by the n_pages param. The n_pages_wakeup param specifies that readers should be woken up every n_pages_wakeup pages; pagewriter doesn't actually do anything with this other than pass it on to the relay channel.

The flags param to pagewriter_open() is also passed on to relay_open(); the bottom half is used by relay for its flags and the top half by pagewriter. Here are the pagewriter flags and what they mean:

- PAGEWRITER_PAD_WRITES - this means that the writer doesn't want writes split across pages - if the write won't fit completely within the space remaining in the page, the event will be written in the next page, but before that happens, the write_padding() pagewriter callback is called to give the tracer a chance to write a 'padding event' in the unused space. Currently, this flag only make sense for a tracer using pagewriter_reserve() and must be set if using pagewriter_reserve() (since there's no way to reserve across pages). Also currently, pagewriter_write() always writes across pages, and for tracers using pagewriter_write(), this flag should not be set.

- PAGEWRITER_FLIGHT_MODE - if this flag is set, pagewriter won't send pages to relay as they become full, but will simply send them to the end of the appropriate per-cpu pool and keep track of how many have been written. This continues until the tracer explicitly requests that they be sent to relay and made available to userspace. The function that does this is pagewriter_save_flight_data(). After the pages have been read by userspace, they're returned to pagewriter as usual.

- PAGEWRITER_LATE_SETUP - if this flag is set, pagewriter_open() won't create the relay channel until pagewriter_late_setup() is called. Before that call is made, written pages are tracked in the per-cpu pool as in the PAGEWRITER_FLIGHT_MODE case and similarly, but after the relay channel has been created by pagewriter_late_setup(), the pages are sent to relay and on to userspace (and returned) as usual.

pagewriter callbacks
--------------------

The pagewriter_callbacks param of pagewriter_open() refers to the three available pagewriter callbacks, none of which are required:

- new_page() - gives the tracer a chance to write page headers, track pages, etc.

- switch_page() - allows a tracer to define its own switch_page() 'slow-path' function and therefore gain control over the complete write path. Wouldn't typically be used by normal tracers.

- write_padding() - called when PAGEWRITER_PAD_WRITES is used - allows the tracer to write a 'padding event' if a normal event won't fit. Currently only makes sense when using pagewriter_reserve().

write/reserve functions
-----------------------

- pagewriter_reserve(pagewriter, length):

Reserve length bytes in the current page. If length bytes won't fit into the current page, the reserve happens in the next page. If length bytes won't fit, at least end_reserve (specified via the pagewriter_open param) bytes are guaranteed to be available at the end of the page for the tracer to write a 'padding event' that presumably will fit in end_reserve bytes. The tracer is given an opportunity to write its padding event via the write_padding() callback, which is passed a pointer to start of padding along with the length of the padding. Currently, if using pagewriter_reserve(), the pagewriter should also be opened using the PAGEWRITER_PAD_WRITES flag.

- pagewriter_write(pagewriter, data, length):

Write length bytes into the current page. If the event won't fit into the current page, whatever will fit is written into the current page, and the remainder is written into the next page. If there's no page available for the remainder, the first part won't be written either. pagewriter_write() disables irqs for synchronization; __pagewriter_write() is essentially the same, but only disables preemption.

misc functions
--------------

- pagewriter_flush():

Switches all partially written pages and calls relay_flush().

- pagewriter_close():

Frees the per-cpu pools, closes the associated relay channel, and destroys the pagewriter.

- pagewriter_reset():

Resets the pagewriter to its initial state.

---
 block/blktrace.c                 |   69 +--
 include/linux/blktrace_api.h     |    7 +-
 include/linux/relay.h            |  255 ++-----
 include/linux/relay_pagewriter.h |  295 ++++++++
 kernel/Makefile                  |    2 +-
 kernel/relay.c                   | 1469 +++++++++++++++-----------------------
 kernel/relay_pagewriter.c        |  860 ++++++++++++++++++++++
 virt/kvm/kvm_trace.c             |   82 +--
 8 files changed, 1845 insertions(+), 1194 deletions(-)
 create mode 100644 include/linux/relay_pagewriter.h
 create mode 100644 kernel/relay_pagewriter.c

index 85049a7..19e417c 100644
@@ -35,7 +35,7 @@ static void trace_note(struct blk_trace *bt, pid_t pid, int action,
 {
 	struct blk_io_trace *t;
 
-	t = relay_reserve(bt->rchan, sizeof(*t) + len);
+	t = pagewriter_reserve(bt->pagewriter, sizeof(*t) + len);
 	if (t) {
 		const int cpu = smp_processor_id();
 

@@ -153,7 +153,7 @@ void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
 	if (unlikely(tsk->btrace_seq != blktrace_seq))
 		trace_note_tsk(bt, tsk);
 
-	t = relay_reserve(bt->rchan, sizeof(*t) + pdu_len);
+	t = pagewriter_reserve(bt->pagewriter, sizeof(*t) + pdu_len);
 	if (t) {
 		cpu = smp_processor_id();
 		sequence = per_cpu_ptr(bt->sequence, cpu);

@@ -230,7 +230,7 @@ err:
 
 static void blk_trace_cleanup(struct blk_trace *bt)
 {
-	relay_close(bt->rchan);
+	pagewriter_close(bt->pagewriter);
 	debugfs_remove(bt->msg_file);
 	debugfs_remove(bt->dropped_file);
 	blk_remove_tree(bt->dir);

@@ -268,7 +268,8 @@ static ssize_t blk_dropped_read(struct file *filp, char __user *buffer,
 	struct blk_trace *bt = filp->private_data;
 	char buf[16];
 
-	snprintf(buf, sizeof(buf), "%u\n", atomic_read(&bt->dropped));
+	snprintf(buf, sizeof(buf), "%u\n",
+		 atomic_read(&bt->pagewriter->dropped));
 
 	return simple_read_from_buffer(buffer, count, ppos, buf, strlen(buf));
 }

@@ -317,43 +318,19 @@ static const struct file_operations blk_msg_fops = {
 	.write =	blk_msg_write,
 };
 
-/*
- * Keep track of how many times we encountered a full subbuffer, to aid
- * the user space app in telling how many lost events there were.
- */
-static int blk_subbuf_start_callback(struct rchan_buf *buf, void *subbuf,
-				     void *prev_subbuf, size_t prev_padding)
+static void blk_write_padding_callback(struct pagewriter_buf *buf,
+				       size_t length,
+				       void *reserved)
 {
-	struct blk_trace *bt;
+	struct blk_io_trace *t = reserved;
 
-	if (!relay_buf_full(buf))
-		return 1;
-
-	bt = buf->chan->private_data;
-	atomic_inc(&bt->dropped);
-	return 0;
-}
-
-static int blk_remove_buf_file_callback(struct dentry *dentry)
-{
-	debugfs_remove(dentry);
-	return 0;
-}
-
-static struct dentry *blk_create_buf_file_callback(const char *filename,
-						   struct dentry *parent,
-						   int mode,
-						   struct rchan_buf *buf,
-						   int *is_global)
-{
-	return debugfs_create_file(filename, mode, parent, buf,
-					&relay_file_operations);
+	t->magic = BLK_IO_TRACE_MAGIC | BLK_IO_TRACE_VERSION;
+	t->action = BLK_TN_PADDING;
+	t->pdu_len = length - sizeof(*t);
 }
 
-static struct rchan_callbacks blk_relay_callbacks = {
-	.subbuf_start		= blk_subbuf_start_callback,
-	.create_buf_file	= blk_create_buf_file_callback,
-	.remove_buf_file	= blk_remove_buf_file_callback,
+static struct pagewriter_callbacks blk_pagewriter_callbacks = {
+	.write_padding           = blk_write_padding_callback,
 };
 
 /*

@@ -365,6 +342,7 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 	struct blk_trace *old_bt, *bt = NULL;
 	struct dentry *dir = NULL;
 	int ret, i;
+	int n_pages, n_pages_wakeup;
 
 	if (!buts->buf_size || !buts->buf_nr)
 		return -EINVAL;

@@ -400,7 +378,6 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 
 	bt->dir = dir;
 	bt->dev = dev;
-	atomic_set(&bt->dropped, 0);
 
 	ret = -EIO;
 	bt->dropped_file = debugfs_create_file("dropped", 0444, dir, bt, &blk_dropped_fops);

@@ -411,9 +388,13 @@ int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 	if (!bt->msg_file)
 		goto err;
 
-	bt->rchan = relay_open("trace", dir, buts->buf_size,
-				buts->buf_nr, &blk_relay_callbacks, bt);
-	if (!bt->rchan)
+	n_pages = (buts->buf_size * buts->buf_nr) / PAGE_SIZE;
+	n_pages_wakeup = buts->buf_size / PAGE_SIZE;
+	bt->pagewriter = pagewriter_open("trace", dir, n_pages, n_pages_wakeup,
+					 sizeof(struct blk_io_trace),
+					 &blk_pagewriter_callbacks, bt,
+					 PAGEWRITER_PAD_WRITES);
+	if (!bt->pagewriter)
 		goto err;
 
 	bt->act_mask = buts->act_mask;

@@ -446,8 +427,8 @@ err:
 			debugfs_remove(bt->dropped_file);
 		free_percpu(bt->sequence);
 		free_percpu(bt->msg_data);
-		if (bt->rchan)
-			relay_close(bt->rchan);
+		if (bt->pagewriter)
+			pagewriter_close(bt->pagewriter);
 		kfree(bt);
 	}
 	return ret;

@@ -500,7 +481,7 @@ int blk_trace_startstop(struct request_queue *q, int start)
 	} else {
 		if (bt->trace_state == Blktrace_running) {
 			bt->trace_state = Blktrace_stopped;
-			relay_flush(bt->rchan);
+			pagewriter_flush(bt->pagewriter);
 			ret = 0;
 		}
 	}

index bdf505d..b14e6e4 100644
@@ -3,7 +3,7 @@
 
 #ifdef __KERNEL__
 #include <linux/blkdev.h>
-#include <linux/relay.h>
+#include <linux/relay_pagewriter.h>
 #endif
 
 /*

@@ -62,6 +62,7 @@ enum blktrace_notify {
 	__BLK_TN_PROCESS = 0,		/* establish pid/name mapping */
 	__BLK_TN_TIMESTAMP,		/* include system clock */
 	__BLK_TN_MESSAGE,		/* Character string message */
+	__BLK_TN_PADDING,		/* Padding message */
 };
 
 

@@ -89,6 +90,7 @@ enum blktrace_notify {
 #define BLK_TN_PROCESS		(__BLK_TN_PROCESS | BLK_TC_ACT(BLK_TC_NOTIFY))
 #define BLK_TN_TIMESTAMP	(__BLK_TN_TIMESTAMP | BLK_TC_ACT(BLK_TC_NOTIFY))
 #define BLK_TN_MESSAGE		(__BLK_TN_MESSAGE | BLK_TC_ACT(BLK_TC_NOTIFY))
+#define BLK_TN_PADDING		(__BLK_TN_PADDING | BLK_TC_ACT(BLK_TC_NOTIFY))
 
 #define BLK_IO_TRACE_MAGIC	0x65617400
 #define BLK_IO_TRACE_VERSION	0x07

@@ -144,7 +146,7 @@ struct blk_user_trace_setup {
 #if defined(CONFIG_BLK_DEV_IO_TRACE)
 struct blk_trace {
 	int trace_state;
-	struct rchan *rchan;
+	struct pagewriter *pagewriter;
 	unsigned long *sequence;
 	unsigned char *msg_data;
 	u16 act_mask;

@@ -155,7 +157,6 @@ struct blk_trace {
 	struct dentry *dir;
 	struct dentry *dropped_file;
 	struct dentry *msg_file;
-	atomic_t dropped;
 };
 
 extern int blk_trace_ioctl(struct block_device *, unsigned, char __user *);

index 066550a..81d28ce 100644
@@ -80,7 +80,7 @@ obj-$(CONFIG_PREEMPT_RCU) += rcupreempt.o
 ifeq ($(CONFIG_PREEMPT_RCU),y)
 obj-$(CONFIG_RCU_TRACE) += rcupreempt_trace.o
 endif
-obj-$(CONFIG_RELAY) += relay.o
+obj-$(CONFIG_RELAY) += relay.o relay_pagewriter.o
 obj-$(CONFIG_SYSCTL) += utsname_sysctl.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o

1.5.3.5

Part of the diff is not shown here. View the raw message.