Trying to measure performance with splice/vmsplice ....
Hello Jens - any assistance/pointers on 1) and 2) below will be great.I'm willing to test out any sample patch.
Steve,
Quoted 21 lines · Steven J. Magnani
Hi Rick,
Quoted 1 line · Rick Sherm
Q3) When using splice, even though the destination
file is opened in
O_DIRECTmode, the data gets cached. I verified it using vmstat.Quoted 10 lines
r b swpd free buff cache
1 0 0 9358820 116576 2100904./splice_to_splice
r b swpd free buff cache
2 0 0 7228908 116576 4198164I see the same caching issue even if I vmsplice
buffers(simple malloc'd iov) to a pipe and then splice the pipe to a file. The speed is still an issue with vmsplice too.
Quoted 1 line
Quoted 9 lines
One thing is that
O_DIRECTis a hint; not all filesystems bypass the cache. I'm pretty sure ext2 does, and I know fat doesn't.Another variable is whether (and how) your filesystem implements the
splice_writefile operation. The generic one (pipe_to_file) infs/splice.ccopies data to pagecache. The default one goes out tovfs_write()and might stand more of a chance of honoringO_DIRECT.
True.I guess I should have looked harder. It's xfs and xfs's->file_ops points to 'generic_file_splice_read[write]'.Last time I had to 'fdatasync' and then fadvise to mimic 'O_DIRECT'.
Quoted 6 lines
Quoted 1 line
Q4) Also, using splice, you can only transfer 64K
worth of
data(PIPE_BUFFERS*PAGE_SIZE)at a time,correct?.But using stock read/write, I can go upto 1MB buffer. After that I don't see any gain. But still the reduction in system/cpu time is significant.
Quoted 11 lines
I'm not a splicing expert but I did spend some time
recently trying to
improve FTP reception by splicing from a TCP socket to a file. I found that while splicing avoids copying packets to userland, that gain is more than offset by a large increase in calls into the storage stack.It's especially bad with TCP sockets because a typical packet has, say,1460 bytes of data. Since splicing works onPIPE_BUFFERSpages at a time, and packet pages are only about 35% utilized, each cycle to userland I could only move 23 KiB of data at most. Some similar effect may be in play in your case.
Agreed,increasing number of calls will offset the benefit.
But what if:
1)We were to increase the PIPE_BUFFERS from '16' to '64' or 'some value'?
What are the implications in the other parts of the kernel?
2)There was a way to find out if the DMA-out/in from the initial buffer's that were passed are complete so that we are free to recycle them? Callback would be helpful.Obviously, the user-space-app will have to manage it's buffers but atleast we are guranteed that the buffers can be recycled(in other words no worrying about modifying in-flight data that is being DMA'd).
Quoted 2 lines
Regards,
Steven J. Magnani
regards
++Rick
Quoted 2 lines · Rick Sherm
Hello Jens - any assistance/pointers on 1) and 2) below will be great.I'm willing to test out any sample patch.
Recent mail from him has come from [email protected], I cc'd it.
Quoted 55 lines
Steve,
Quoted 21 lines · Steven J. Magnani
Hi Rick,
Quoted 1 line · Rick Sherm
Q3) When using splice, even though the destination
file is opened in
O_DIRECTmode, the data gets cached. I verified it using vmstat.Quoted 10 lines
r b swpd free buff cache
1 0 0 9358820 116576 2100904./splice_to_splice
r b swpd free buff cache
2 0 0 7228908 116576 4198164I see the same caching issue even if I vmsplice
buffers(simple malloc'd iov) to a pipe and then splice the pipe to a file. The speed is still an issue with vmsplice too.
Quoted 1 line
Quoted 9 lines
One thing is that
O_DIRECTis a hint; not all filesystems bypass the cache. I'm pretty sure ext2 does, and I know fat doesn't.Another variable is whether (and how) your filesystem implements the
splice_writefile operation. The generic one (pipe_to_file) infs/splice.ccopies data to pagecache. The default one goes out tovfs_write()and might stand more of a chance of honoringO_DIRECT.True.I guess I should have looked harder. It's xfs and xfs's->file_ops points to 'generic_file_splice_read[write]'.Last time I had to 'fdatasync' and then fadvise to mimic '
O_DIRECT'.Quoted 6 lines
Quoted 1 line
Q4) Also, using splice, you can only transfer 64K
worth of
data(PIPE_BUFFERS*PAGE_SIZE)at a time,correct?.But using stock read/write, I can go upto 1MB buffer. After that I don't see any gain. But still the reduction in system/cpu time is significant.Quoted 11 lines
I'm not a splicing expert but I did spend some time
recently trying to
improve FTP reception by splicing from a TCP socket to a file. I found that while splicing avoids copying packets to userland, that gain is more than offset by a large increase in calls into the storage stack.It's especially bad with TCP sockets because a typical packet has, say,1460 bytes of data. Since splicing works onPIPE_BUFFERSpages at a time, and packet pages are only about 35% utilized, each cycle to userland I could only move 23 KiB of data at most. Some similar effect may be in play in your case.
Quoted 4 lines
Agreed,increasing number of calls will offset the benefit. But what if:
1)We were to increase thePIPE_BUFFERSfrom '16' to '64' or 'some value'? What are the implications in the other parts of the kernel?
This came up recently, one problem is that there a couple of kernel functions having up to 3 stack-based arrays of dimension PIPE_BUFFER. So the stack cost of increasing PIPE_BUFFERS can be quite high. I've thought it might be nice if there was some mechanism for userland apps to be able to request larger PIPE_BUFFERS values, but I haven't pursued this line of thought to see if it's practical.
Quoted 1 line
2)There was a way to find out if the DMA-out/in from the initial buffer's that were passed are complete so that we are free to recycle them? Callback would be helpful.Obviously, the user-space-app will have to manage it's buffers but atleast we are guranteed that the buffers can be recycled(in other words no worrying about modifying in-flight data that is being DMA'd).
It's a neat idea, but it would probably be much easier (and less invasive) to try this sort of pipelining in userland using a ring buffer or ping-pong approach. I'm actually in the middle of something like this with FTP, where I will have a reader thread that puts data from the network into a ring buffer, from which a writer thread moves it to a file.
------------------------------------------------------------------------
Steven J. Magnani "I claim this network for MARS!
www.digidescorp.com Earthling, return my space modulator!"
#include <standard.disclaimer>
Quoted 5 lines · Steven J. Magnani
Quoted 2 lines · Rick Sherm
Hello Jens - any assistance/pointers on 1) and 2) below will be great.I'm willing to test out any sample patch.
Recent mail from him has come from [email protected], I cc'd it.
Goes to the same inbox in the end, so no difference :-)
Quoted 54 lines
Quoted 49 lines
Quoted 19 lines · Rick Sherm
Quoted 1 line
Q3) When using splice, even though the destination
file is opened in
O_DIRECTmode, the data gets cached. I verified it using vmstat.Quoted 10 lines
r b swpd free buff cache
1 0 0 9358820 116576 2100904./splice_to_splice
r b swpd free buff cache
2 0 0 7228908 116576 4198164I see the same caching issue even if I vmsplice
buffers(simple malloc'd iov) to a pipe and then splice the pipe to a file. The speed is still an issue with vmsplice too.
Quoted 1 line
Quoted 9 lines
One thing is that
O_DIRECTis a hint; not all filesystems bypass the cache. I'm pretty sure ext2 does, and I know fat doesn't.Another variable is whether (and how) your filesystem implements the
splice_writefile operation. The generic one (pipe_to_file) infs/splice.ccopies data to pagecache. The default one goes out tovfs_write()and might stand more of a chance of honoringO_DIRECT.True.I guess I should have looked harder. It's xfs and xfs's->file_ops points to 'generic_file_splice_read[write]'.Last time I had to 'fdatasync' and then fadvise to mimic '
O_DIRECT'.Quoted 6 lines
Quoted 1 line
Q4) Also, using splice, you can only transfer 64K
worth of
data(PIPE_BUFFERS*PAGE_SIZE)at a time,correct?.But using stock read/write, I can go upto 1MB buffer. After that I don't see any gain. But still the reduction in system/cpu time is significant.Quoted 11 lines
I'm not a splicing expert but I did spend some time
recently trying to
improve FTP reception by splicing from a TCP socket to a file. I found that while splicing avoids copying packets to userland, that gain is more than offset by a large increase in calls into the storage stack.It's especially bad with TCP sockets because a typical packet has, say,1460 bytes of data. Since splicing works onPIPE_BUFFERSpages at a time, and packet pages are only about 35% utilized, each cycle to userland I could only move 23 KiB of data at most. Some similar effect may be in play in your case.Quoted 4 lines
Agreed,increasing number of calls will offset the benefit. But what if:
1)We were to increase thePIPE_BUFFERSfrom '16' to '64' or 'some value'? What are the implications in the other parts of the kernel?
Quoted 6 lines
This came up recently, one problem is that there a couple of kernel functions having up to 3 stack-based arrays of dimension
PIPE_BUFFER. So the stack cost of increasingPIPE_BUFFERScan be quite high. I've thought it might be nice if there was some mechanism for userland apps to be able to request largerPIPE_BUFFERSvalues, but I haven't pursued this line of thought to see if it's practical.
I still have patches pending for this, making the pipe buffer count settable form user space:
http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=24547ac4d97bebb58caf9ce58bd507a95c812a3f
Let me know if you want to give it a spin on a recent kernel, and I'll update it.
Quoted 8 lines
Quoted 1 line
2)There was a way to find out if the DMA-out/in from the initial buffer's that were passed are complete so that we are free to recycle them? Callback would be helpful.Obviously, the user-space-app will have to manage it's buffers but atleast we are guranteed that the buffers can be recycled(in other words no worrying about modifying in-flight data that is being DMA'd).
It's a neat idea, but it would probably be much easier (and less invasive) to try this sort of pipelining in userland using a ring buffer or ping-pong approach. I'm actually in the middle of something like this with FTP, where I will have a reader thread that puts data from the network into a ring buffer, from which a writer thread moves it to a file.
See vmsplice.c from the splice test tools:
http://brick.kernel.dk/snaps/splice-git-latest.tar.gz
Signature · 1 line
Jens Axboe
Hello Jens,
Quoted 10 lines · Jens Axboe
I still have patches pending for this, making the pipe buffer count
settable form user space:http://git.kernel.dk/?p=linux-2.6-block.git;a=commit;h=24547ac4d97bebb58caf9ce58bd507a95c812a3f
Let me know if you want to give it a spin on a recent kernel, and I'll
update it.
I think we need to adjust 'PIPE_BUFFERS' in default_file_splice_read() also,correct?
Quoted 1 line
Jens Axboe
Thanks