Tracer Ring Buffer splice() vs page cache [was: Re: Perf and ftrace [was Re: PyTimechart]]

[CCing memory management specialists]

You bring a very interesting point. Let me describe what I want to achieve, and see what others have to say about it:

I want the ring buffer to allocate pages only at ring buffer creation (never while tracing). There are a few reasons why I want to do that, ranging from improved performance to limited system disturbance.

Now let suppose we have the synchronization mechanism (detailed in the original thread, but not relevant to this part of the discussion) that lets us give the pages to the ring buffer "reader", which sends them to splice() so it can use them as write buffers. Let also suppose that the ring buffer reader blocks until the pages are written to the disk (synchronous write). In my scheme, the reader still has pointers to these pages.

The point you bring here is that when the ring buffer "reader" is woken up, these pages could still be in the page cache. So when the reader gives these pages back to the ring buffer (so they can be used for writing again), the page cache may still hold a reference to them, so the pages in the page cache and the version on disk could be unsynchronized, and therefore this could possibly lead to trace file corruption (in the worse case).

So I have three questions here:

1 - could we enforce removal of these pages from the page cache by calling
"page_cache_release()" before giving these pages back to the ring buffer ?

2 - or maybe is there a page flag we could specify when we allocate them to
ask for these pages to never be put in the page cache ? (but they should be still usable as write buffers)

3 - is there something more we need to do to grab a reference on the pages
before passing them to splice(), so that when we call page_cache_release() they don't get reclaimed ?

Thanks,

Mathieu

Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

And jet you forgot Jens who wrote it ;-)

There is no guarantee it is the pagecache they end up in, it could be a network packet queue, a pipe, or anything that implements .splice_write.

ownership of the page, you're not supposed to touch them again, non of the above three are feasible.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

oops ! thanks for adding him.

Yup, I've looked more deeply at the splice() code, and I now see why things don't fall apart in LTTng currently. My implementation seems to be causing splice() to perform a copy. My ring buffer splice implementation is derived from kernel/relay.c. I override

pipe_buf_operations release op with:

static void ltt_relay_pipe_buf_release(struct pipe_inode_info *pipe,
struct pipe_buffer *pbuf)
{
}

and

splice_pipe_desc spd_release file op with:

static void ltt_relay_page_release(struct splice_pipe_desc *spd, unsigned int i) {
}

My understanding is that by keeping 2 references on the pages (the ring buffer + the pipe), splice safely refuses to move the pages and performs a copy instead.

I'll continue to look into this. One of the things I noticed that that we could possibly use the "steal()" operation to steal the pages back from the page cache to repopulate the ring buffer rather than continuously allocating new pages. If steal() fails for some reasons, then we can fall back on page allocation. I'm not sure it is safe to assume anything about pages being in the page cache though. Maybe the safest route is to just allocate new pages for now.

Thoughts ?

Mathieu

Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Also, suppose it was still in the page-cache and still dirty, a steal() would then punch a hole in the file.

Yes, that seems to be the only sane approach.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

page_cache_pipe_buf_steal starts by doing a wait_on_page_writeback(page); and then does a try_to_release_page(page, GFP_KERNEL). Only if that succeeds is the action of stealing succeeding.

Yes, a good start anyway.

Thanks,

Mathieu

Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

If you're going to wait for writeback I don't really see the advantage of stealing over simply allocating a new page.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

That would allow the ring buffer to use a bounded amount of memory and not pollute the page cache uselessly. When allocating pages as you propose, the tracer will quickly fill and pollute the page cache with trace file pages, which will have a large impact on I/O behavior. But in 99.9999% of use-cases, we don't ever need to access them after they have been saved to disk.

By re-stealing its own pages after waiting for the writeback to complete, the ring buffer would use a bounded amount of pages. If larger buffers are needed, the user just has to specify a larger buffer size.

Thanks,

Mathieu

Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com