RFC relay revamp 7

Here's the latest version of the 'relay revamp' patchset. Following the last version, I made some more changes in a couple areas, as detailed below. Rather than posting all the individual patches again, I'm just posting the squashed version.

I'm also appending the end-result relay and relay_pagewriter at the end of this mail since it may be hard to see the result through all the individual changes.

Here are descriptions of the two additional patches beyond the original 21 (all rolled into the full patch, and not posted separately):

-- Add relay_add_partial_page() and related.

Tracers don't always send full pages, especially e.g. the final page of a buffer, so relay needs a way to add 'partial pages'. Adding this pretty much required a simplification of the page-switch code in pagewriter, which in turn required some changes to the pagewriter write/reserve functions. The end result is a much cleaner and simpler page-switch mechanism. As part of this, I decided to take advantage of the replaceable page-switch mechanism to get rid of the multiple tests in the previous monolithic default page_switch implementation. The goal is foremost to provide a low-overhead write path, which in this case meant separate switch functions for padded/non-padded/flight/normal logging paths. These end up being pretty small and in any case we should never need more.

Also in the process simplified pagewriter_reserve() by getting rid of the end_reserve param, which can be and is now specified as part of pagewriter_open.

Also added reference counting (using the existing buf kref) to each pagewriter buf which is updated every time a page is added to or returned from relay. This is to disallow the pagewriter from being destroyed while relay still has its pages.

-- Finish up locking.

Finish locking in relay and add locking to relay_pagewriter (basically need to protect the lists if being accessed from any cpu).

--

I've done some testing of the basic stuff via blktrace, which seems to still perform well and without any problems. It seems pretty solid at this point, but needs more testing - in any case, unless I find more bugs or get some review comments, I don't think the code should change much beyond this point.

overview
========

Basically, the overall effect of the patchset is to split the current relay into two separate and completely independent pieces: relay and relay_pagewriter. Here's a one-line description of each piece:

relay - the new relay simply accepts pages of trace data from anywhere in the kernel and collects them into per-cpu lists which can be sent by userspace to disk or network destinations via standard read(2), splice(2) and sendfile(2) file operations, after which the pages are returned to their owner(s).

relay_pagewriter - provides a small set of page-writing functions that allow tracers to write into pages it grabs one-by-one from a per-cpu pool of pages and sends them to relay when each is completely written.

There are of course other features e.g. if you don't want to deal with multiple per-cpu files, there's also a 'global' mode, as there always was. relay_pagewriter also supports a 'flight-recorder' mode where the pages aren't sent to relay but rather returned to the end of the pool; at some later point they can be sent to relay via a simple function designed for that purpose. Similarly, the 'early tracing/late setup' functionality has been retained, but is now much simpler in implementation; the fact that these and other features just seem to 'fall out' of the implementation seems to point to the end design being a little nicer.

Note that because of this separation, there's nothing special about relay_pagewriter - it becomes just another source of trace data (via an actual tracer such as blktrace).

Here's an overview of relay and relay_pagewriter:

relay
=====

With the new changes, relay really does become just what its name says and and nothing more - it accepts pages from tracers, and relays the data to userspace via read(2) or splice(2) (and therefore sendfile(2)). It doesn't allocate any buffer space and provides no write functions - those are expected to be supplied by any tracer that might want relay pages of trace data to userspace.

After applying the full patch you should be able to use it to stream your trace data to disk or over the network...

Anyway, here's a brief overview of the new API (see code for details):

- relay_open():

Creates a per-cpu relay channel and by default associates debugfs files with each per-cpu 'buffer'. No buffer space is allocated for the 'buffers', rather they collect pages added by tracers in a list which is drained by read()/splice(), etc. Tracers add pages to the 'buffers' using relay_write_page() and relay_write_pages(). One of the parameters to relay_open() is n_pages_wakeup, which specifies that readers should be woken up every time n_pages have been added; if this is 0, readers are never woken up.

- relay_add_page():

Adds a page of trace data to relay. After it has been consumed by userspace, the tracer is notified by the 'relay page' callback function page_released(). The page passed via the callback can then be re-used by the tracer (see for example the pagewriter code, which simply adds the page back into pagewriter's per-cpu page pool). If the page has been stolen instead (if SPLICE_F_MOVE succeeded, which can't happen in current kernels since support for it isn't there), the page_stolen() callback is called, at which point the tracer can allocate a new page to replace the stolen page (see the pagewriter code, which does this too).

- relay_add_partial_page():

Adds a partial (less than or equal to PAGE_SIZE) page of trace data to relay. Sometimes a less-than-full page needs to be added to relay e.g. the last page at the end of a trace; this allows for that.

- relay_add_pages():

The same as relay_add_page(), but adds a set of pages to relay and guarantees that they'll stay together and remain in the same order they were added.

- relay_close():

Releases unread pages to the tracer(s) and frees the channel.

- relay_flush():

Wakes up readers.

- relay_reset():

Releases unread pages to the tracer(s) and resets the channel state.

That's basically the entire kernel API for relay; the userspace API is of course just read(), splice(), and sendfile().

relay_pagewriter
================

One example of a component that can write pages of trace data and send them to relay would be the original relay write functions and buffers (the no-vmap page-based versions), which have been split out into a new file called relay_pagewriter.c and provide one means of writing into pages and feeding them into relay. blktrace and kvmtrace have been 'ported' over to using pagewriter instead of relay directly.

- pagewriter_open():

Allocates a pool of pages for each cpu and creates a relay channel to send them to. Whenever a new page is needed for writing into, the pagewriter object pulls a page out of the appropriate per-cpu pool and makes it available to tracers to write/reserve into via the pagewriter_write() and pagewriter_reserve() functions. When a page is full, the pagewriter object sends the page to relay via relay_add_page() and grabs another page for the next writes. When relay has finished relaying the pages to userspace, it returns the page via pagewriter's page_released() or page_stolen() callbacks; pagewriter simply returns (and allocates a new page if stolen) the page to the pool.

The number of pages allocated per-cpu is specified by the n_pages param. The n_pages_wakeup param specifies that readers should be woken up every n_pages_wakeup pages; pagewriter doesn't actually do anything with this other than pass it on to the relay channel.

The flags param to pagewriter_open() is also passed on to relay_open(); the bottom half is used by relay for its flags and the top half by pagewriter. Here are the pagewriter flags and what they mean:

- PAGEWRITER_PAD_WRITES - this means that the writer doesn't want writes split across pages - if the write won't fit completely within the space remaining in the page, the event will be written in the next page, but before that happens, the write_padding() pagewriter callback is called to give the tracer a chance to write a 'padding event' in the unused space. Currently, this flag only make sense for a tracer using pagewriter_reserve() and must be set if using pagewriter_reserve() (since there's no way to reserve across pages). Also currently, pagewriter_write() always writes across pages, and for tracers using pagewriter_write(), this flag should not be set.

- PAGEWRITER_FLIGHT_MODE - if this flag is set, pagewriter won't send pages to relay as they become full, but will simply send them to the end of the appropriate per-cpu pool and keep track of how many have been written. This continues until the tracer explicitly requests that they be sent to relay and made available to userspace. The function that does this is pagewriter_save_flight_data(). After the pages have been read by userspace, they're returned to pagewriter as usual.

- PAGEWRITER_LATE_SETUP - if this flag is set, pagewriter_open() won't create the relay channel until pagewriter_late_setup() is called. Before that call is made, written pages are tracked in the per-cpu pool as in the PAGEWRITER_FLIGHT_MODE case and similarly, but after the relay channel has been created by pagewriter_late_setup(), the pages are sent to relay and on to userspace (and returned) as usual.

pagewriter callbacks
--------------------

The pagewriter_callbacks param of pagewriter_open() refers to the three available pagewriter callbacks, none of which are required:

- new_page() - gives the tracer a chance to write page headers, track pages, etc.

- switch_page() - allows a tracer to define its own switch_page() 'slow-path' function and therefore gain control over the complete write path. Wouldn't typically be used by normal tracers.

- write_padding() - called when PAGEWRITER_PAD_WRITES is used - allows the tracer to write a 'padding event' if a normal event won't fit. Currently only makes sense when using pagewriter_reserve().

write/reserve functions
-----------------------

- pagewriter_reserve(pagewriter, length):

Reserve length bytes in the current page. If length bytes won't fit into the current page, the reserve happens in the next page. If length bytes won't fit, at least end_reserve (specified via the pagewriter_open param) bytes are guaranteed to be available at the end of the page for the tracer to write a 'padding event' that presumably will fit in end_reserve bytes. The tracer is given an opportunity to write its padding event via the write_padding() callback, which is passed a pointer to start of padding along with the length of the padding. Currently, if using pagewriter_reserve(), the pagewriter should also be opened using the PAGEWRITER_PAD_WRITES flag.

- pagewriter_write(pagewriter, data, length):

Write length bytes into the current page. If the event won't fit into the current page, whatever will fit is written into the current page, and the remainder is written into the next page. If there's no page available for the remainder, the first part won't be written either. pagewriter_write() disables irqs for synchronization; __pagewriter_write() is essentially the same, but only disables preemption.

misc functions
--------------

- pagewriter_flush():

Switches all partially written pages and calls relay_flush().

- pagewriter_close():

Frees the per-cpu pools, closes the associated relay channel, and destroys the pagewriter.

- pagewriter_reset():

Resets the pagewriter to its initial state.

Below is the current relay/relay_pagewriter code after applying all patches in the patchset, for convenience.

Tom

@@ -0,0 +1,176 @@
+/*
+ * linux/include/linux/relay.h
+ *
+ * Copyright (C) 2002, 2003 - Tom Zanussi ([email protected]), IBM Corp
+ * Copyright (C) 1999, 2000, 2001, 2002 - Karim Yaghmour ([email protected])
+ * Copyright (C) 2008 - Tom Zanussi ([email protected])
+ *
+ * CONFIG_RELAY definitions and declarations
+ */
+
+#ifndef _LINUX_RELAY_H
+#define _LINUX_RELAY_H
+
+#include <linux/types.h>
+#include <linux/sched.h>
+#include <linux/timer.h>
+#include <linux/wait.h>
+#include <linux/list.h>
+#include <linux/fs.h>
+#include <linux/poll.h>
+#include <linux/kref.h>
+#include <linux/pagevec.h>
+
+/*
+ * relay channel flags
+ */
+#define RCHAN_GLOBAL_BUFFER		0x00000001	/* not using per-cpu */
+
+/*
+ * For page lists
+ */
+struct relay_page {
+	struct page *page;
+	size_t len;
+	struct list_head list;
+	struct relay_page_callbacks *cb;
+	void *private_data;
+};
+
+/*
+ * Per-cpu relay channel buffer
+ */
+struct rchan_buf {
+	struct rchan *chan;		/* associated channel */
+	wait_queue_head_t read_wait;	/* reader wait queue */
+	struct timer_list timer; 	/* reader wake-up timer */
+	struct dentry *dentry;		/* channel file dentry */
+	struct kref kref;		/* channel buffer refcount */
+	struct list_head pages;		/* current set of unconsumed pages */
+	size_t nr_pages;		/* number of unconsumed pages */
+	spinlock_t lock;		/* protect pages list */
+	size_t consumed_offset;		/* bytes consumed in cur page */
+	unsigned int finalized;		/* buffer has been finalized */
+	unsigned int cpu;		/* this buf's cpu */
+} ____cacheline_aligned;
+
+/*
+ * Relay channel data structure
+ */
+struct rchan
+{
+	size_t n_pages_wakeup;		/* wake up readers after filling n */
+	struct rchan_callbacks *cb;	/* client callbacks */
+	struct kref kref;		/* channel refcount */
+	void *private_data;		/* for user-defined data */
+	struct rchan_buf *buf[NR_CPUS]; /* per-cpu channel buffers */
+	struct list_head list;		/* for channel list */
+	struct dentry *parent;		/* parent dentry passed to open */
+	char base_filename[NAME_MAX];	/* saved base filename */
+	unsigned long flags;		/* relay flags for this channel */
+};
+
+/*
+ * Relay channel client callbacks
+ */
+struct rchan_callbacks
+{
+	/*
+	 * create_buf_file - create file to represent a relay channel buffer
+	 * @filename: the name of the file to create
+	 * @parent: the parent of the file to create
+	 * @mode: the mode of the file to create
+	 * @buf: the channel buffer
+	 *
+	 * Called during relay_open(), once for each per-cpu buffer,
+	 * to allow the client to create a file to be used to
+	 * represent the corresponding channel buffer.  If the file is
+	 * created outside of relay, the parent must also exist in
+	 * that filesystem.
+	 *
+	 * The callback should return the dentry of the file created
+	 * to represent the relay buffer.
+	 *
+	 * See Documentation/filesystems/relayfs.txt for more info.
+	 */
+	struct dentry *(*create_buf_file)(const char *filename,
+					  struct dentry *parent,
+					  int mode,
+					  struct rchan_buf *buf);
+
+	/*
+	 * remove_buf_file - remove file representing a relay channel buffer
+	 * @dentry: the dentry of the file to remove
+	 *
+	 * Called during relay_close(), once for each per-cpu buffer,
+	 * to allow the client to remove a file used to represent a
+	 * channel buffer.
+	 *
+	 * The callback should return 0 if successful, negative if not.
+	 */
+	int (*remove_buf_file)(struct dentry *dentry);
+};
+
+/*
+ * Relay page callbacks
+ */
+struct relay_page_callbacks
+{
+	/*
+	 * page_released - notification that a page is ready for re-use
+	 * @page: the released page
+	 * @private_data: user-defined data associated with the page
+	 *
+	 * This callback is a notification that a given page has been
+	 * read by userspace and can be re-used.  Always called in
+	 * user context.
+	 */
+	void (*page_released) (struct page *page, void *private_data);
+
+	/*
+	 * page_released - notification that a page has been stolen
+	 * @page: the stolen page
+	 * @private_data: user-defined data associated with the page
+	 *
+	 * This callback is a notification that a given page has been
+	 * stolen by userspace.  The owner may wish to replace it;
+	 * this gives it the opportunity to do so.  Always called in
+	 * user context.
+	 */
+	void (*page_stolen) (struct page *page, void *private_data);
+};
+
+/*
+ * CONFIG_RELAY kernel API, kernel/relay.c
+ */
+
+extern struct rchan *relay_open(const char *base_filename,
+				struct dentry *parent,
+				size_t n_pages_wakeup,
+				struct rchan_callbacks *cb,
+				void *private_data,
+				unsigned long rchan_flags);
+extern void relay_add_partial_page(struct rchan *chan,
+				   struct page *page,
+				   size_t len,
+				   struct relay_page_callbacks *cb,
+				   void *private_data);
+extern void relay_add_page(struct rchan *chan,
+			   struct page *page,
+			   struct relay_page_callbacks *cb,
+			   void *private_data);
+extern void relay_add_pages(struct rchan *chan,
+			    struct pagevec *pages,
+			    struct relay_page_callbacks *cb,
+			    void *private_data);
+extern void relay_flush(struct rchan *chan);
+extern void relay_close(struct rchan *chan);
+extern void relay_reset(struct rchan *chan);
+
+/*
+ * exported relay file operations, kernel/relay.c
+ */
+extern const struct file_operations relay_file_operations;
+
+#endif /* _LINUX_RELAY_H */
+