generic_file_sendpage
Subsystems (4)
PATCH 1/1 generic_file_sendpage
This is a generic sendpage() for regular files.
With generic_file_sendpage() it is possible to use sendfile() on file targets, instead of only sending data to sockets.
This implementation is basically an extension of Joern's original patch (http://marc.theaimsgroup.com/?l=linux-kernel&m=109455958522766&w=2) but is honoring signals. I also removed some unnecessary code: no IOVs, no AIO.
qemu-debian:/home/root/sendfile_file# time ./fastcp 100mb test real 0m11.037s user 0m0.010s sys 0m7.600s
qemu-debian:/home/root/sendfile_file# time cp 100mb test real 0m13.342s user 0m0.400s sys 0m9.080s
Comments please,
Jan
fs/ext2/file.c | 1 fs/ext3/file.c | 1 include/linux/fs.h | 2 mm/filemap.c | 170 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 174 insertions(+)
include/linux/fs.h +2 -0
@@ -1502,6 +1502,8 @@ extern ssize_t do_sync_write(struct filessize_t generic_file_write_nolock (struct file *file,const struct iovec *iov,unsigned long nr_segs,loff_t *ppos);extern ssize_t generic_file_sendfile (struct file *,loff_t *,size_t ,read_actor_t ,void *); +extern ssize_t generic_file_sendpage (struct file *,struct page *,int ,size_t , +loff_t *,int );extern void do_generic_mapping_read (struct address_space *mapping,struct file_ra_state *,struct file *,loff_t *,read_descriptor_t *,read_actor_t );
mm/filemap.c +170 -0
188 lines not on this page · raw message
fs/ext2/file.c +1 -0
@@ -53,6 +53,7 @@ struct file_operations ext2_file_operati .readv = generic_file_readv, .writev = generic_file_writev, .sendfile = generic_file_sendfile, + .sendpage = generic_file_sendpage, };#ifdef CONFIG_EXT2_FS_XIP
fs/ext3/file.c +1 -0
@@ -119,6 +119,7 @@ struct file_operations ext3_file_operati .release = ext3_release_file, .fsync = ext3_sync_file, .sendfile = generic_file_sendfile, + .sendpage = generic_file_sendpage, };struct inode_operations ext3_file_inode_operations = {
mm/filemap.c int remove_suid(struct dentry *dentry) · line 1747
Quoted 13 lines · Jan Blunck
This is a generic
sendpage()for regular files.mm/filemap.c +9
-0+static inline size_t +filemap_copy_from_kernel (struct page *page,unsigned long offset, +const char *buf,unsigned bytes) +{ +char *kaddr; + + kaddr =kmap (page); +memcpy (kaddr + offset, buf, bytes); +kunmap (page);
Use kmap_atomic().
Move the flush_dcache_page() into here.
Quoted 9 lines
mm/filemap.c +9
-0+static ssize_t +__generic_kernel_file_write (struct file *file,const char *buf, +size_t count,loff_t *ppos) +{ +struct address_space * mapping = file->f_mapping; +struct address_space_operations *a_ops = mapping->a_ops; +struct inode *inode = mapping->host; +long status =0 ; +loff_t pos;
mm/filemap.c generic_file_direct_write(struct kiocb * · line 1928
Quoted 10 lines
mm/filemap.c +10
-0+struct page *page; +struct page *cached_page =NULL ; +const int isblk =S_ISBLK (inode->i_mode); +ssize_t written; +ssize_t err; +size_t bytes; +struct pagevec lru_pvec; + +/* There is no sane reason to use O_DIRECT */ +BUG_ON (file->f_flags &O_DIRECT );
err, this seems like an easy way for people to make the kernel go BUG.
mm/filemap.c generic_file_direct_write(struct kiocb * · line 1931
Quoted 2 lines
mm/filemap.c +2
-0+if (unlikely (signal_pending (current))) +return -EINTR ;
This doesn't help. The reason we've avoided file-to-file sendfile() is that it can cause applications to get uninterruptibly stuck in the kernel for ages. This code doesn't solve that problem. It needs to handle signal_pending() inside the main loop.
And it probably needs to return a sane value (number of bytes copied) rather than -EINTR.
I don't know if we want to add this feature, really. It's such a specialised thing.
Quoted 5 lines · Andrew Morton
mm/filemap.c generic_file_direct_write(struct kiocb * · line 1928
Quoted 3 lines
mm/filemap.c +3
-0+ +/* There is no sane reason to use O_DIRECT */ +BUG_ON (file->f_flags &O_DIRECT );err, this seems like an easy way for people to make the kernel go BUG.
Is there a sane use for O_DIRECT in combination with sendfile()?
If not, I'd like to change sys_sendfile() and return -EINVAL for O_DIRECT file descriptors.
Quoted 10 lines
mm/filemap.c generic_file_direct_write(struct kiocb * · line 1931
Quoted 2 lines
mm/filemap.c +2
-0+if (unlikely (signal_pending (current))) +return -EINTR ;This doesn't help. The reason we've avoided file-to-file
sendfile()is that it can cause applications to get uninterruptibly stuck in the kernel for ages. This code doesn't solve that problem. It needs to handlesignal_pending()inside the main loop.And it probably needs to return a sane value (number of bytes copied) rather than
-EINTR.
Makes sense.
Quoted 2 lines
I don't know if we want to add this feature, really. It's such a specialised thing.
With union mount and cowlink, there are two users already. cp(1) could use it as well, even if the improvement is quite minimal.
Jörn
Signature · 2 lines
All art is but imitation of nature. -- Lucius Annaeus Seneca
Quoted 5 lines
Quoted 2 lines
I don't know if we want to add this feature, really. It's such a specialised thing.
With union mount and cowlink, there are two users already. cp(1) could use it as well, even if the improvement is quite minimal.
FTP PUT could use this too - currently, only FTPGETs can use sendfile because the target had to be a socket.
(With FTP PUT, the src is a sock, dst is a filedescriptor.)
Jan Engelhardt
Quoted 10 lines · Jan Engelhardt
Quoted 6 lines
Quoted 3 lines
I don't know if we want to add this feature, really. It's such a specialised thing.
With union mount and cowlink, there are two users already. cp(1) could use it as well, even if the improvement is quite minimal.
FTP PUT could use this too - currently, only FTPGETs can use sendfile because the target had to be a socket.
(With FTP PUT, the src is a sock, dst is a filedescriptor.)
No, FTP PUT _cannot_ use it, exactly because sendfile() can't do anything but file sources (and not even all file sources - it can only do regular filesystems that use the page cache).
This is why I want to get rid of sendfile(). It's a fundamentally broken interface. Really. In contrast, the pipe buffers _can_ be used for direct socket->file interfaces.
Now, even pipe buffers obviously won't actually be really "zero-copy":
you'll end up needing one copy to align the result, since the incoming network packets will obviously not be nice page-sized and page-aligned things. But they won't need the "copy to user space" and "copy back from user space into kernel space", so it will be a question of _minimal_ copy (and avoiding unnecessary user space VM work).
Linus
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
With union mount and cowlink, there are two users already. cp(1) could use it as well, even if the improvement is quite minimal.
FTP PUT could use this too - ...
Quoted 7 lines
No, FTP PUT _cannot_ use it, exactly because
sendfile()can't do anything but file sources (and not even all file sources - it can only do regular filesystems that use the page cache).This is why I want to get rid of
sendfile(). It's a fundamentally broken interface. Really. In contrast, the pipe buffers _can_ be used for directsocket->fileinterfaces.
How will userspace access these pipe buffers?
Jan Engelhardt
Quoted 6 lines · Jan Engelhardt
Quoted 4 lines
This is why I want to get rid of
sendfile(). It's a fundamentally broken interface. Really. In contrast, the pipe buffers _can_ be used for directsocket->fileinterfaces.How will userspace access these pipe buffers?
You can fill them from a user-space buffer with "write()", and you can read them into a user-space buffer with "read()".
It's really a pipe.
The idea of the pipe buffers is that you can _also_ send them to other file descriptors, or fill them from other file descriptors, without having to copy the data to/from user space.
Linus
---
See the example patch that I posted months ago (and that I referred to in my other email, here is that thing repeated in case you missed it):
Quoted 11 lines
For anybody interested in zero-copy work, here's a LWN write-up of some of the original discussion:
http://lwn.net/Articles/118750/and my (very ugly) example patch can be found for example here:
http://groups-beta.google.com/group/linux.kernel/msg/782bd9e5cb647207?hl=en&(it's not a a complete implementation, but it shows how to go from a file _to_ a pipe buffer, but not back to a file again).
Quoted 23 lines · Jörn Engel
Quoted 5 lines · Andrew Morton
mm/filemap.c generic_file_direct_write(struct kiocb * · line 1928
Quoted 3 lines
mm/filemap.c +3
-0+ +/* There is no sane reason to use O_DIRECT */ +BUG_ON (file->f_flags &O_DIRECT );err, this seems like an easy way for people to make the kernel go BUG.
Is there a sane use for
O_DIRECTin combination withsendfile()?If not, I'd like to change
sys_sendfile()and return-EINVALforO_DIRECTfile descriptors.Quoted 10 lines
mm/filemap.c generic_file_direct_write(struct kiocb * · line 1931
Quoted 2 lines
mm/filemap.c +2
-0+if (unlikely (signal_pending (current))) +return -EINTR ;This doesn't help. The reason we've avoided file-to-file
sendfile()is that it can cause applications to get uninterruptibly stuck in the kernel for ages. This code doesn't solve that problem. It needs to handlesignal_pending()inside the main loop.And it probably needs to return a sane value (number of bytes copied) rather than
-EINTR.
Quoted 7 lines
Makes sense.
Quoted 2 lines
I don't know if we want to add this feature, really. It's such a specialised thing.
With union mount and cowlink, there are two users already. cp(1) could use it as well, even if the improvement is quite minimal.
you might soon add linux-vserver to the list, as we will be using this for a special version of the COW links (to break unified links on write) ...
best,
Herbert
Quoted 10 lines
Jörn
Signature · 7 lines
All art is but imitation of nature. -- Lucius Annaeus Seneca - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] generic_file_sendpage (updated patch)
In reply to Andrew MortonQuoted 14 lines · Andrew Morton
Review
Quoted 10 lines
(quoted diff) +9
-0+static inline size_t +filemap_copy_from_kernel(struct page *page, unsigned long offset, + const char *buf, unsigned bytes) +{ + char *kaddr; + + kaddr = kmap(page); + memcpy(kaddr + offset, buf, bytes); + kunmap(page);Use
kmap_atomic().
Quoted 2 lines
Move the
flush_dcache_page()into here.
Ok
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ + /* There is no sane reason to use O_DIRECT */ + BUG_ON(file->f_flags & O_DIRECT);err, this seems like an easy way for people to make the kernel go BUG.
Hmm, yeah ... -EINVAL should do it.
Quoted 9 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ if (unlikely(signal_pending(current))) + return -EINTR;This doesn't help. The reason we've avoided file-to-file
sendfile()is that it can cause applications to get uninterruptibly stuck in the kernel for ages. This code doesn't solve that problem. It needs to handlesignal_pending()inside the main loop.
The sendfile() mainloop is in generic_file_sendfile(). generic_file_sendpage() is called via the file_send_actor() from do_generic_file_read(). Therefore generic_file_sendpage() is called for every page which is written to the file. Since signals are checked before every page this is solving the uninterruptible problem. IMHO, the sendpage() should not be interrupted in between a single page.
Quoted 3 lines
And it probably needs to return a sane value (number of bytes copied) rather than
-EINTR.
The return value of sendfile() is not the same as the return value of sendpage(). This is done in file_send_actor().
Quoted 3 lines
I don't know if we want to add this feature, really. It's such a specialised thing.
This issue comes up every now and than. I know of at least two users: COW links and my union mounts. I think this solution is quite good. I tested it under qemu and it is working for me.
Jan
BTW: I'm out for the weekend now.
fs/ext2/file.c | 1 fs/ext3/file.c | 1 include/linux/fs.h | 2 mm/filemap.c | 172 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 176 insertions(+)
include/linux/fs.h +2 -0
@@ -1502,6 +1502,8 @@ extern ssize_t do_sync_write(struct filessize_t generic_file_write_nolock (struct file *file,const struct iovec *iov,unsigned long nr_segs,loff_t *ppos);extern ssize_t generic_file_sendfile (struct file *,loff_t *,size_t ,read_actor_t ,void *); +extern ssize_t generic_file_sendpage (struct file *,struct page *,int ,size_t , +loff_t *,int );extern void do_generic_mapping_read (struct address_space *mapping,struct file_ra_state *,struct file *,loff_t *,read_descriptor_t *,read_actor_t );
mm/filemap.c +172 -0
190 lines not on this page · raw message
fs/ext2/file.c +1 -0
@@ -53,6 +53,7 @@ struct file_operations ext2_file_operati .readv = generic_file_readv, .writev = generic_file_writev, .sendfile = generic_file_sendfile, + .sendpage = generic_file_sendpage, };#ifdef CONFIG_EXT2_FS_XIP
fs/ext3/file.c +1 -0
@@ -119,6 +119,7 @@ struct file_operations ext3_file_operati .release = ext3_release_file, .fsync = ext3_sync_file, .sendfile = generic_file_sendfile, + .sendpage = generic_file_sendpage, };struct inode_operations ext3_file_inode_operations = {
Quoted 3 lines · Andrew Morton
I don't know if we want to add this feature, really. It's such a specialised thing.
It is, in this format, and I agree - I don't want to add it.
What I'd really like to see is somebody taking a look at my old "pipe as a zero-copy buffer" patches, which as an interface allowed arbitrary data to be copied between _any_ file descriptors, and allowed you to do things like mix input sources quite naturally (ie you could write a header first from a user-space buffer, then the contents of a file, and then push the result out to a socket, all with zero-copy).
"sendfile()" in general I think has been a mistake. It's too specialized, and the interface has always sucked. As Andrew pointed out, it actually needs to limit the number of buffers in flight partly because otherwise you have uninterruptible kernel work etc etc.
But more importantly, sendfile() was always broken as a interface for receiving data from anything but a page-cache based filesystem. That means that it's totally useless for a lot of things. The pipe-buffer thing ends up being a totally generic "in-kernel buffer" interface, and is a _lot_ more flexible.
For anybody interested in zero-copy work, here's a LWN write-up of some of the original discussion:
http://lwn.net/Articles/118750/
and my (very ugly) example patch can be found for example here:
http://groups-beta.google.com/group/linux.kernel/msg/782bd9e5cb647207?hl=en&
(it's not a a complete implementation, but it shows how to go from a file _to_ a pipe buffer, but not back to a file again).
I really want to get _rid_ of sendfile, not make any more of it.
Linus
Quoted 3 lines · Linus Torvalds
"
sendfile()" in general I think has been a mistake. It's too specialized, and the interface has always sucked.
Ok, you're right. I will have a look at the pipe buffer stuff.
Quoted 2 lines
As Andrew pointed out, it actually
needs to limit the number of buffers in flight partly because otherwise you have uninterruptible kernel work etc etc.
Hmm, sendfile() uses do_generic_mapping_read() which is calling the file_send_actor() per page. The actor calls ->sendpage(). I don't see any situation how this could lead to uninterruptible kernel work when the sendpage() itself is checking for signals. There are only 2 pages in flight in this case.
Jan