splice, block: Use page pinning and kill ITER_PIPE
Subsystems (17)
v18 cover splice, block: Use page pinning and kill ITER_PIPE
Hi Jens, Al, Christoph,
The first half of this patchset kills off ITER_PIPE to avoid a race between truncate, iov_iter_revert() on the pipe and an as-yet incomplete DMA to a bio with unpinned/unref'ed pages from an O_DIRECT splice read. This causes memory corruption[2]. Instead, we use filemap_splice_read(), which invokes the buffered file reading code and splices from the pagecache into the pipe; direct_splice_read(), which bulk-allocates a buffer, reads into it and then pushes the filled pages into the pipe; or handle it in filesystem-specific code.
(1) Simplify the calculations for the number of pages to be reclaimed in direct_splice_read().
(2) Turn do_splice_to() into a helper so that it can be used by overlayfs and coda to perform the checks on the lower fs.
(3) Provide shmem with its own splice_read to handle non-existent pages in the pagecache. We don't want a ->read_folio() as we don't want to populate holes, but filemap_get_pages() requires it.
(4) Provide overlayfs with its own splice_read to call down to a lower
layer as overlayfs doesn't provide ->read_folio().
(5) Provide coda with its own splice_read to call down to a lower layer as
coda doesn't provide ->read_folio().
(6) Direct ->splice_read to direct_splice_read() in tty, procfs, kernfs and random files as they just copy to the output buffer and don't splice pages.
(7) Change generic_file_splice_read() to just switch between filemap_splice_read() and direct_splice_read() rather than using ITER_PIPE.
(8) Make cifs use generic_file_splice_read().
(9) Remove ITER_PIPE and its paraphernalia as generic_file_splice_read() was the only user.
The second half of the patchset rolls page-pinning out to the bio struct and the block layer, using iov_iter_extract_pages() to get pages and noting with BIO_PAGE_PINNED if the data pages attached to a bio are pinned. If the data pages come from a non-user-backed iterator, then the pages are left unpinned and unref'd, relying on whoever set up the I/O to do the retaining
(10) Don't hold a ref on ZERO_PAGE in iomap_dio_zero().
(11) Fix bio_flagged() so that it doesn't prevent a gcc optimisation.
(12) Make the bio struct carry a pair of flags to indicate the cleanup mode. BIO_NO_PAGE_REF is replaced with BIO_PAGE_REFFED (indicating FOLL_GET was used) and BIO_PAGE_PINNED (indicating FOLL_PIN was used) is added.
BIO_PAGE_REFFED will go away, but at the moment fs/direct-io.c sets it and this series does not fully address that file.
(13) Add a function, bio_release_page(), to release a page appropriately to the cleanup mode indicated by the BIO_PAGE_* flags.
(14) Make bio_iov_iter_get_pages() use iov_iter_extract_pages() to retain the pages appropriately and clean them up later.
(15) Make bio_map_user_iov() also use iov_iter_extract_pages().
I've pushed the patches here also:
https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=iov-extract
David
Changes:
========
ver #18)
- Split out the cifs bits from the patch the switches generic_file_splice_read() over to using the non-ITER_PIPE splicing.
- Don't get/put refs on the zeropage in shmem_splice_read().
ver #17)
- Rename do_splice_to() to vfs_splice_read() and export it so that it can be a helper and make overlayfs and coda use it, allowing duplicate checks to be removed.
ver #16)
- The filemap_get_pages() changes are now upstream.
- filemap_splice_read() and direct_splice_read() are now upstream.
- iov_iter_extract_pages() is now upstream.
ver #15)
- Fixed up some errors in overlayfs_splice_read().
ver #14)
- Some changes to generic_file_buffered_splice_read():
- Rename to filemap_splice_read() and move to mm/filemap.c.
- Create a helper, pipe_head_buf().
- Use init_sync_kiocb().
- Some changes to generic_file_direct_splice_read():
- Use alloc_pages_bulk_array() rather than alloc_pages_bulk_list().
- Use release_pages() instead of __free_page() in a loop.
- Rename to direct_splice_read().
- Rearrange the patches to implement filemap_splice_read() and direct_splice_read() separately to changing generic_file_splice_read().
- Don't call generic_file_splice_read() when there isn't a ->read_folio().
- Insert patches to fix read_folio-less cases:
- Make tty, procfs, kernfs and (u)random use direct_splice_read().
- Make overlayfs and coda call down to a lower layer.
- Give shmem its own splice-read that doesn't insert missing pages.
- Fixed a min() with mixed type args on some arches.
ver #13)
- Only use allocation in advance and ITER_BVEC for DIO read-splice.
- Make buffered read-splice get pages directly from the pagecache.
- Alter filemap_get_pages() & co. so that it doesn't need an iterator.
ver #12)
- Added the missing __bitwise on the iov_iter_extraction_t typedef.
- Rebased on -rc7.
- Don't specify FOLL_PIN to pin_user_pages_fast().
- Inserted patch at front to fix race between DIO read and truncation that caused memory corruption when iov_iter_revert() got called on an ITER_PIPE iterator[2].
- Inserted a patch after that to remove the now-unused ITER_PIPE and its helper functions.
- Removed the ITER_PIPE bits from iov_iter_extract_pages().
ver #11)
- Fix iov_iter_extract_kvec_pages() to include the offset into the page in the returned starting offset.
- Use __bitwise for the extraction flags
ver #10)
- Fix use of i->kvec in iov_iter_extract_bvec_pages() to be i->bvec.
- Drop bio_set_cleanup_mode(), open coding it instead.
ver #9)
- It's now not permitted to use FOLL_PIN outside of mm/, so:
- Change iov_iter_extract_mode() into iov_iter_extract_will_pin() and return true/false instead of FOLL_PIN/0.
- Drop of folio_put_unpin() and page_put_unpin() and instead call unpin_user_page() (and put_page()) directly as necessary.
- Make __bio_release_pages() call bio_release_page() instead of unpin_user_page() as there's no BIO_* -> FOLL_* translation to do.
- Drop the FOLL_* renumbering patch.
- Change extract_flags to extraction_flags.
ver #8)
- Import Christoph Hellwig's changes.
- Split the conversion-to-extraction patch.
- Drop the extract_flags arg from iov_iter_extract_mode().
- Don't default bios to BIO_PAGE_REFFED, but set explicitly.
- Switch FOLL_PIN and FOLL_GET when renumbering so PIN is at bit 0.
- Switch BIO_PAGE_PINNED and BIO_PAGE_REFFED so PINNED is at bit 0.
- We should always be using FOLL_PIN (not FOLL_GET) for DIO, so adjust the patches for that.
ver #7)
- For now, drop the parts to pass the I/O direction to iov_iter_*pages*() as it turned out to be a lot more complicated, with places not setting IOCB_WRITE when they should, for example.
- Drop all the patches that changed things other then the block layer's bio handling. The netfslib and cifs changes can go into a separate patchset.
- Add support for extracting pages from KVEC-type iterators.
- When extracting from BVEC/KVEC, skip over empty vecs at the front.
ver #6)
- Fix write() syscall and co. not setting IOCB_WRITE.
- Added iocb_is_read() and iocb_is_write() to check IOCB_WRITE.
- Use op_is_write() in bio_copy_user_iov().
- Drop the iterator direction checks from smbd_recv().
- Define FOLL_SOURCE_BUF and FOLL_DEST_BUF and pass them in as part of gup_flags to iov_iter_get/extract_pages*().
- Replace iov_iter_get_pages*2() with iov_iter_get_pages*() and remove.
- Add back the function to indicate the cleanup mode.
- Drop the cleanup_mode return arg to iov_iter_extract_pages().
- Provide a helper to clean up a page.
- Renumbered FOLL_GET and FOLL_PIN and made BIO_PAGE_REFFED/PINNED have the same numerical values, enforced with an assertion.
- Converted AF_ALG, SCSI vhost, generic DIO, FUSE, splice to pipe, 9P and NFS.
- Added in the patches to make CIFS do top-to-bottom iterators and use various of the added extraction functions.
- Added a pair of work-in-progess patches to make sk_buff fragments store FOLL_GET and FOLL_PIN.
ver #5)
- Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED and split into own patch.
- Transcribe FOLL_GET/PIN into BIO_PAGE_REFFED/PINNED flags.
- Add patch to allow bio_flagged() to be combined by gcc.
ver #4)
- Drop the patch to move the FOLL_* flags to linux/mm_types.h as they're no longer referenced by linux/uio.h.
- Add ITER_SOURCE/DEST cleanup patches.
- Make iov_iter/netfslib iter extraction patches use ITER_SOURCE/DEST.
- Allow additional gup_flags to be passed into iov_iter_extract_pages().
- Add struct bio patch.
ver #3)
- Switch to using EXPORT_SYMBOL_GPL to prevent indirect 3rd-party access to get/pin_user_pages_fast()[1].
ver #2)
- Rolled the extraction cleanup mode query function into the extraction function, returning the indication through the argument list.
- Fixed patch 4 (extract to scatterlist) to actually use the new extraction API.
Additional patches that got folded in:
Christoph Hellwig (1):
block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted
logic
David Howells (14):
splice: Clean up direct_splice_read() a bit
splice: Make do_splice_to() generic and export it
shmem: Implement splice-read
overlayfs: Implement splice-read
coda: Implement splice-read
tty, proc, kernfs, random: Use direct_splice_read()
splice: Do splice read from a file without using ITER_PIPE
cifs: Use generic_file_splice_read()
iov_iter: Kill ITER_PIPE
iomap: Don't get an reference on ZERO_PAGE for direct I/O block
zeroing
block: Fix bio_flagged() so that gcc can better optimise it
block: Add BIO_PAGE_PINNED and associated infrastructure
block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages
block: convert bio_map_user_iov to use iov_iter_extract_pages
block/bio.c | 29 +-- block/blk-map.c | 22 +- block/blk.h | 12 ++ drivers/char/random.c | 4 +- drivers/tty/tty_io.c | 4 +- fs/cifs/cifsfs.c | 8 +- fs/cifs/cifsfs.h | 3 - fs/cifs/file.c | 16 -- fs/coda/file.c | 29 ++- fs/direct-io.c | 2 + fs/iomap/direct-io.c | 1 - fs/kernfs/file.c | 2 +- fs/overlayfs/file.c | 23 +- fs/proc/inode.c | 4 +- fs/proc/proc_sysctl.c | 2 +- fs/proc_namespace.c | 6 +- fs/splice.c | 76 +++---- include/linux/bio.h | 5 +- include/linux/blk_types.h | 3 +- include/linux/splice.h | 3 + include/linux/uio.h | 14 -- lib/iov_iter.c | 429 +------------------------------------- mm/filemap.c | 4 +- mm/shmem.c | 135 +++++++++++- 24 files changed, 285 insertions(+), 551 deletions(-)
PATCH v18 1/15 splice: Clean up direct_splice_read() a bit
Do a couple of cleanups to direct_splice_read():
(1) Cast to struct page **, not void *.
(2) Simplify the calculation of the number of pages to keep/reclaim in direct_splice_read().
--- fs/splice.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
fs/splice.c +7 -12
@@ -295,7 +295,7 @@ ssize_t direct_splice_read(struct file *in, loff_t *ppos,struct kiocb kiocb;struct page **pages;ssize_t ret; -size_t used, npages, chunk, remain, reclaim; +size_t used, npages, chunk, remain, keep =0 ;int i;/* Work out how much data we can actually add into the pipe */
@@ -309,7 +309,7 @@ ssize_t direct_splice_read(struct file *in, loff_t *ppos,if (!bv)return -ENOMEM ; - pages = (void *)(bv + npages); + pages = (struct page **)(bv + npages); npages =alloc_pages_bulk_array (GFP_USER , npages, pages);if (!npages) {kfree (bv);
@@ -332,11 +332,8 @@ ssize_t direct_splice_read(struct file *in, loff_t *ppos, kiocb.ki_pos = *ppos; ret =call_read_iter (in, &kiocb, &to); - reclaim = npages *PAGE_SIZE ; - remain =0 ;if (ret >0 ) { - reclaim -= ret; - remain = ret; + keep =DIV_ROUND_UP (ret,PAGE_SIZE ); *ppos = kiocb.ki_pos;file_accessed (in); }else if (ret <0 ) {
@@ -349,14 +346,12 @@ ssize_t direct_splice_read(struct file *in, loff_t *ppos, }/* Free any pages that didn't get touched at all. */ - reclaim /=PAGE_SIZE ; -if (reclaim) { - npages -= reclaim; -release_pages (pages + npages, reclaim); - } +if (keep < npages) +release_pages (pages + keep, npages - keep);/* Push the remaining pages into the pipe. */ -for (i =0 ; i < npages; i++) { + remain = ret; +for (i =0 ; i < keep; i++) {struct pipe_buffer *buf =pipe_head_buf (pipe); chunk =min_t (size_t, remain,PAGE_SIZE );
PATCH v18 2/15 splice: Make do_splice_to() generic and export it
In reply to David HowellsRename do_splice_to() to vfs_splice_read() and export it so that it can be used as a helper when calling down to a lower layer filesystem as it performs all the necessary checks[1].
--- fs/splice.c | 27 ++++++++++++++++++++------- include/linux/splice.h | 3 +++ 2 files changed, 23 insertions(+), 7 deletions(-)
fs/splice.c +20 -7
@@ -851,12 +851,24 @@ static long do_splice_from(struct pipe_inode_info *pipe, struct file *out,return out->f_op->splice_write (pipe, out, ppos, len, flags); } -/* -* Attempt to initiate a splice from a file to a pipe. +/** +* vfs_splice_read - Read data from a file and splice it into a pipe +* @in: File to splice from +* @ppos: Input file offset +* @pipe: Pipe to splice to +* @len: Number of bytes to splice +* @flags: Splice modifier flags (SPLICE_F_*) +* +* Splice the requested amount of data from the input file to the pipe. This +* is synchronous as the caller must hold the pipe lock across the entire +* operation. +* +* If successful, it returns the amount of data spliced, 0 if it hit the EOF or +* a hole and a negative error code otherwise. */ -static long do_splice_to (struct file *in,loff_t *ppos, -struct pipe_inode_info *pipe,size_t len, -unsigned int flags) +long vfs_splice_read (struct file *in,loff_t *ppos, +struct pipe_inode_info *pipe,size_t len, +unsigned int flags) {unsigned int p_space;int ret;
@@ -879,6 +891,7 @@ static long do_splice_to(struct file *in, loff_t *ppos,return warn_unsupported (in,"read" );return in->f_op->splice_read (in, ppos, pipe, len, flags); } +EXPORT_SYMBOL_GPL (vfs_splice_read);/** * splice_direct_to_actor - splices data directly between two non-pipes
@@ -949,7 +962,7 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,size_t read_len;loff_t pos = sd->pos, prev_pos = pos; - ret =do_splice_to (in, &pos, pipe, len, flags); + ret =vfs_splice_read (in, &pos, pipe, len, flags);if (unlikely (ret <=0 )) goto out_release;
@@ -1097,7 +1110,7 @@ long splice_file_to_pipe(struct file *in,pipe_lock (opipe); ret =wait_for_space (opipe, flags);if (!ret) - ret =do_splice_to (in, offset, opipe, len, flags); + ret =vfs_splice_read (in, offset, opipe, len, flags);pipe_unlock (opipe);if (ret >0 )wakeup_pipe_readers (opipe);
include/linux/splice.h +3 -0
@@ -76,6 +76,9 @@ extern ssize_t splice_to_pipe(struct pipe_inode_info *, struct splice_pipe_desc *);extern ssize_t add_to_pipe (struct pipe_inode_info *,struct pipe_buffer *); +long vfs_splice_read (struct file *in,loff_t *ppos, +struct pipe_inode_info *pipe,size_t len, +unsigned int flags);extern ssize_t splice_direct_to_actor (struct file *,struct splice_desc *,splice_direct_actor *); extern long do_splice(struct file *in, loff_t *off_in,
PATCH v18 3/15 shmem: Implement splice-read
In reply to David HowellsThe new filemap_splice_read() has an implicit expectation via filemap_get_pages() that ->read_folio() exists if ->readahead() doesn't fully populate the pagecache of the file it is reading from[1], potentially leading to a jump to NULL if this doesn't exist. shmem, however, (and by extension, tmpfs, ramfs and rootfs), doesn't have ->read_folio(),
Work around this by equipping shmem with its own splice-read implementation, based on filemap_splice_read(), but able to paste in zero_page when there's a page missing.
---
Notes:
ver #18)
- Don't take/release a ref on the zero page.
mm/shmem.c | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 134 insertions(+), 1 deletion(-)
mm/shmem.c +134 -1
@@ -2719,6 +2719,139 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)return retval ? retval : error; } +static bool zero_pipe_buf_get (struct pipe_inode_info *pipe, +struct pipe_buffer *buf) +{ +return true; +} + +static void zero_pipe_buf_release (struct pipe_inode_info *pipe, +struct pipe_buffer *buf) +{ +} + +static bool zero_pipe_buf_try_steal (struct pipe_inode_info *pipe, +struct pipe_buffer *buf) +{ +return false; +} + +static const struct pipe_buf_operations zero_pipe_buf_ops = { + .release = zero_pipe_buf_release, + .try_steal = zero_pipe_buf_try_steal, + .get = zero_pipe_buf_get, +}; + +static size_t splice_zeropage_into_pipe (struct pipe_inode_info *pipe, +loff_t fpos,size_t size) +{ +size_t offset = fpos & ~PAGE_MASK ; + + size =min_t (size_t, size,PAGE_SIZE - offset); + +if (!pipe_full (pipe->head, pipe->tail, pipe->max_usage)) { +struct pipe_buffer *buf =pipe_head_buf (pipe); + + *buf = (struct pipe_buffer ) { + .ops = &zero_pipe_buf_ops, + .page =ZERO_PAGE (0 ), + .offset = offset, + .len = size, + }; +get_page (buf->page); + pipe->head++; + } + +return size; +} + +static ssize_t shmem_file_splice_read (struct file *in,loff_t *ppos, +struct pipe_inode_info *pipe, +size_t len,unsigned int flags) +{ +struct inode *inode =file_inode (in); +struct address_space *mapping = inode->i_mapping; +struct folio *folio =NULL ; +size_t total_spliced =0 , used, npages, n, part; +loff_t isize; +int error =0 ; + +/* Work out how much data we can actually add into the pipe */ + used =pipe_occupancy (pipe->head, pipe->tail); + npages =max_t (ssize_t, pipe->max_usage - used,0 ); + len =min_t (size_t, len, npages *PAGE_SIZE ); + +do { +if (*ppos >=i_size_read (inode)) +break ; + + error =shmem_get_folio (inode, *ppos /PAGE_SIZE , &folio,SGP_READ ); +if (error) { +if (error == -EINVAL ) + error =0 ; +break ; + } +if (folio) { +folio_unlock (folio); + +if (folio_test_hwpoison (folio)) { + error = -EIO ; +break ; + } + } + +/* +* i_size must be checked after we know the pages are Uptodate. +* +* Checking i_size after the check allows us to calculate +* the correct value for "nr", which means the zero-filled +* part of the page is not copied back to userspace (unless +* another truncate extends the file - this is desired though). +*/ + isize =i_size_read (inode); +if (unlikely (*ppos >= isize)) +break ; + part =min_t (loff_t, isize - *ppos, len); + +if (folio) { +/* +* If users can be writing to this page using arbitrary +* virtual addresses, take care about potential aliasing +* before reading the page on the kernel side. +*/ +if (mapping_writably_mapped (mapping)) +flush_dcache_folio (folio); +folio_mark_accessed (folio); +/* +* Ok, we have the page, and it's up-to-date, so we can +* now splice it into the pipe. +*/ + n =splice_folio_into_pipe (pipe, folio, *ppos, part); +folio_put (folio); + folio =NULL ; + }else { + n =splice_zeropage_into_pipe (pipe, *ppos, len); + } + +if (!n) +break ; + len -= n; + total_spliced += n; + *ppos += n; + in->f_ra.prev_pos = *ppos; +if (pipe_full (pipe->head, pipe->tail, pipe->max_usage)) +break ; + +cond_resched (); + }while (len); + +if (folio) +folio_put (folio); + +file_accessed (in); +return total_spliced ? total_spliced : error; +} +static loff_t shmem_file_llseek (struct file *file,loff_t offset,int whence) {struct address_space *mapping = file->f_mapping;
@@ -3938,7 +4071,7 @@ static const struct file_operations shmem_file_operations = { .read_iter = shmem_file_read_iter, .write_iter = generic_file_write_iter, .fsync = noop_fsync, - .splice_read = generic_file_splice_read, + .splice_read = shmem_file_splice_read, .splice_write = iter_file_splice_write, .fallocate = shmem_fallocate,#endif
Re: [PATCH v18 03/15] shmem: Implement splice-read
Quoted 1 line · David Howells
\> +static size_t splice_zeropage_into_pipe(...
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 2761
Quoted 8 lines
...
mm/shmem.c +7
-0+ *buf = (struct pipe_buffer ) { + .ops = &zero_pipe_buf_ops, + .page =ZERO_PAGE (0 ), + .offset = offset, + .len = size, + }; +get_page (buf->page);
That
+ get_page(buf->page);
is still there, and now it's doubly wrong because it's never dropped and will eventually overflow that count that shouldn't even be there.
Linus
PATCH v18 4/15 overlayfs: Implement splice-read
In reply to David HowellsImplement splice-read for overlayfs by passing the request down a layer rather than going through generic_file_splice_read() which is going to be changed to assume that ->read_folio() is present on buffered files.
---
Notes:
ver #17)
- Use vfs_splice_read() helper rather than open-coding checks.
ver #15)
- Remove redundant FMODE_CAN_ODIRECT check on real file.
- Do rw_verify_area() on the real file, not the overlay file.
- Fix a file leak.
fs/overlayfs/file.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
fs/overlayfs/file.c +22 -1
@@ -419,6 +419,27 @@ static ssize_t ovl_write_iter(struct kiocb *iocb, struct iov_iter *iter)return ret; } +static ssize_t ovl_splice_read (struct file *in,loff_t *ppos, +struct pipe_inode_info *pipe,size_t len, +unsigned int flags) +{ +const struct cred *old_cred; +struct fd real; +ssize_t ret; + + ret =ovl_real_fdget (in, &real); +if (ret) +return ret; + + old_cred =ovl_override_creds (file_inode (in)->i_sb); + ret =vfs_splice_read (real.file, ppos, pipe, len, flags); +revert_creds (old_cred); +ovl_file_accessed (in); + +fdput (real); +return ret; +} +/* * Calling iter_file_splice_write() directly from overlay's f_op may deadlock * due to lock order inversion between pipe->mutex in iter_file_splice_write()
@@ -695,7 +716,7 @@ const struct file_operations ovl_file_operations = { .fallocate = ovl_fallocate, .fadvise = ovl_fadvise, .flush = ovl_flush, - .splice_read = generic_file_splice_read, + .splice_read = ovl_splice_read, .splice_write = ovl_splice_write, .copy_file_range = ovl_copy_file_range,
PATCH v18 5/15 coda: Implement splice-read
In reply to David HowellsImplement splice-read for coda by passing the request down a layer rather than going through generic_file_splice_read() which is going to be changed to assume that ->read_folio() is present on buffered files.
---
Notes:
ver #17)
- Use vfs_splice_read() helper rather than open-coding checks.
fs/coda/file.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-)
fs/coda/file.c +28 -1
@@ -23,6 +23,7 @@#include <linux/slab.h> #include <linux/uaccess.h> #include <linux/uio.h> +#include <linux/splice.h> #include <linux/coda.h> #include "coda_psdev.h"
@@ -94,6 +95,32 @@ coda_file_write_iter(struct kiocb *iocb, struct iov_iter *to)return ret; } +static ssize_t +coda_file_splice_read (struct file *coda_file,loff_t *ppos, +struct pipe_inode_info *pipe, +size_t len,unsigned int flags) +{ +struct inode *coda_inode =file_inode (coda_file); +struct coda_file_info *cfi =coda_ftoc (coda_file); +struct file *in = cfi->cfi_container; +loff_t ki_pos = *ppos; +ssize_t ret; + + ret =venus_access_intent (coda_inode->i_sb,coda_i2f (coda_inode), + &cfi->cfi_access_intent, + len, ki_pos,CODA_ACCESS_TYPE_READ ); +if (ret) + goto finish_read; + + ret =vfs_splice_read (in, ppos, pipe, len, flags); + +finish_read: +venus_access_intent (coda_inode->i_sb,coda_i2f (coda_inode), + &cfi->cfi_access_intent, + len, ki_pos,CODA_ACCESS_TYPE_READ_FINISH ); +return ret; +} +static void coda_vm_open (struct vm_area_struct *vma) {
@@ -302,5 +329,5 @@ const struct file_operations coda_file_operations = { .open = coda_open, .release = coda_release, .fsync = coda_fsync, - .splice_read = generic_file_splice_read, + .splice_read = coda_file_splice_read, };
PATCH v18 6/15 tty, proc, kernfs, random: Use direct_splice_read()
In reply to David HowellsUse direct_splice_read() for tty, procfs, kernfs and random files rather than going through generic_file_splice_read() as they just copy the file into the output buffer and don't splice pages. This avoids the need for them to have a ->read_folio() to satisfy filemap_splice_read().
--- drivers/char/random.c | 4 ++-- drivers/tty/tty_io.c | 4 ++-- fs/kernfs/file.c | 2 +- fs/proc/inode.c | 4 ++-- fs/proc/proc_sysctl.c | 2 +- fs/proc_namespace.c | 6 +++--- 6 files changed, 11 insertions(+), 11 deletions(-)
drivers/char/random.c +2 -2
@@ -1546,7 +1546,7 @@ const struct file_operations random_fops = { .compat_ioctl = compat_ptr_ioctl, .fasync = random_fasync, .llseek = noop_llseek, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, };
@@ -1557,7 +1557,7 @@ const struct file_operations urandom_fops = { .compat_ioctl = compat_ptr_ioctl, .fasync = random_fasync, .llseek = noop_llseek, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, };
drivers/tty/tty_io.c +2 -2
@@ -466,7 +466,7 @@ static const struct file_operations tty_fops = { .llseek = no_llseek, .read_iter = tty_read, .write_iter = tty_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, .poll = tty_poll, .unlocked_ioctl = tty_ioctl,
@@ -481,7 +481,7 @@ static const struct file_operations console_fops = { .llseek = no_llseek, .read_iter = tty_read, .write_iter = redirected_tty_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, .poll = tty_poll, .unlocked_ioctl = tty_ioctl,
fs/kernfs/file.c +1 -1
@@ -1011,7 +1011,7 @@ const struct file_operations kernfs_file_fops = { .release = kernfs_fop_release, .poll = kernfs_fop_poll, .fsync = noop_fsync, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, };
fs/proc/inode.c +2 -2
@@ -591,7 +591,7 @@ static const struct file_operations proc_iter_file_ops = { .llseek = proc_reg_llseek, .read_iter = proc_reg_read_iter, .write = proc_reg_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .poll = proc_reg_poll, .unlocked_ioctl = proc_reg_unlocked_ioctl, .mmap = proc_reg_mmap,
@@ -617,7 +617,7 @@ static const struct file_operations proc_reg_file_ops_compat = {static const struct file_operations proc_iter_file_ops_compat = { .llseek = proc_reg_llseek, .read_iter = proc_reg_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .write = proc_reg_write, .poll = proc_reg_poll, .unlocked_ioctl = proc_reg_unlocked_ioctl,
fs/proc/proc_sysctl.c +1 -1
@@ -869,7 +869,7 @@ static const struct file_operations proc_sys_file_operations = { .poll = proc_sys_poll, .read_iter = proc_sys_read, .write_iter = proc_sys_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, .llseek = default_llseek, };
fs/proc_namespace.c +3 -3
@@ -324,7 +324,7 @@ static int mountstats_open(struct inode *inode, struct file *file)const struct file_operations proc_mounts_operations = { .open = mounts_open, .read_iter = seq_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .llseek = seq_lseek, .release = mounts_release, .poll = mounts_poll,
@@ -333,7 +333,7 @@ const struct file_operations proc_mounts_operations = {const struct file_operations proc_mountinfo_operations = { .open = mountinfo_open, .read_iter = seq_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .llseek = seq_lseek, .release = mounts_release, .poll = mounts_poll,
@@ -342,7 +342,7 @@ const struct file_operations proc_mountinfo_operations = {const struct file_operations proc_mountstats_operations = { .open = mountstats_open, .read_iter = seq_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .llseek = seq_lseek, .release = mounts_release, };
PATCH v18 7/15 splice: Do splice read from a file without using ITER_PIPE
In reply to David HowellsMake generic_file_splice_read() use filemap_splice_read() and direct_splice_read() rather than using an ITER_PIPE and call_read_iter().
With this, ITER_PIPE is no longer used.
---
Notes:
ver #18)
- Split out the change to cifs to make it use generic_file_splice_read().
- Split out the unexport of filemap_splice_read() (still needed by cifs).
fs/splice.c | 30 +++++++----------------------- 1 file changed, 7 insertions(+), 23 deletions(-)
fs/splice.c +7 -23
@@ -387,29 +387,13 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags) { - struct iov_iter to; -struct kiocb kiocb; -int ret; - -iov_iter_pipe (&to,ITER_DEST , pipe, len); -init_sync_kiocb (&kiocb, in); - kiocb.ki_pos = *ppos; - ret =call_read_iter (in, &kiocb, &to); -if (ret >0 ) { - *ppos = kiocb.ki_pos; -file_accessed (in); - }else if (ret <0 ) { -/* free what was emitted */ -pipe_discard_from (pipe, to.start_head); -/* -* callers of ->splice_read() expect -EAGAIN on -* "can't put anything in there", rather than -EFAULT. -*/ -if (ret == -EFAULT ) - ret = -EAGAIN ; - } - -return ret; + if (unlikely(*ppos >= file_inode(in)->i_sb->s_maxbytes)) + return 0; + if (unlikely(!len)) + return 0; + if (in->f_flags & O_DIRECT) + return direct_splice_read(in, ppos, pipe, len, flags); +return filemap_splice_read (in, ppos, pipe, len, flags); }EXPORT_SYMBOL (generic_file_splice_read);
PATCH v18 8/15 cifs: Use generic_file_splice_read()
In reply to David HowellsMake cifs use generic_file_splice_read() rather than doing it for itself.
As a consequence, filemap_splice_read() no longer needs to be exported.
---
Notes:
ver #18)
- Split out from change to generic_file_splice_read().
fs/cifs/cifsfs.c | 8 ++++---- fs/cifs/cifsfs.h | 3 --- fs/cifs/file.c | 16 ---------------- mm/filemap.c | 1 - 4 files changed, 4 insertions(+), 24 deletions(-)
fs/cifs/cifsfs.c +4 -4
@@ -1359,7 +1359,7 @@ const struct file_operations cifs_file_ops = { .fsync = cifs_fsync, .flush = cifs_flush, .mmap = cifs_file_mmap, - .splice_read = cifs_splice_read, + .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, .llseek = cifs_llseek, .unlocked_ioctl = cifs_ioctl,
@@ -1379,7 +1379,7 @@ const struct file_operations cifs_file_strict_ops = { .fsync = cifs_strict_fsync, .flush = cifs_flush, .mmap = cifs_file_strict_mmap, - .splice_read = cifs_splice_read, + .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, .llseek = cifs_llseek, .unlocked_ioctl = cifs_ioctl,
@@ -1417,7 +1417,7 @@ const struct file_operations cifs_file_nobrl_ops = { .fsync = cifs_fsync, .flush = cifs_flush, .mmap = cifs_file_mmap, - .splice_read = cifs_splice_read, + .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, .llseek = cifs_llseek, .unlocked_ioctl = cifs_ioctl,
@@ -1435,7 +1435,7 @@ const struct file_operations cifs_file_strict_nobrl_ops = { .fsync = cifs_strict_fsync, .flush = cifs_flush, .mmap = cifs_file_strict_mmap, - .splice_read = cifs_splice_read, + .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, .llseek = cifs_llseek, .unlocked_ioctl = cifs_ioctl,
fs/cifs/cifsfs.h +0 -3
@@ -100,9 +100,6 @@ extern ssize_t cifs_strict_readv(struct kiocb *iocb, struct iov_iter *to);extern ssize_t cifs_user_writev (struct kiocb *iocb,struct iov_iter *from);extern ssize_t cifs_direct_writev (struct kiocb *iocb,struct iov_iter *from);extern ssize_t cifs_strict_writev (struct kiocb *iocb,struct iov_iter *from); -extern ssize_t cifs_splice_read (struct file *in,loff_t *ppos, -struct pipe_inode_info *pipe,size_t len, -unsigned int flags);extern int cifs_flock (struct file *pfile,int cmd,struct file_lock *plock);extern int cifs_lock (struct file *,int ,struct file_lock *);extern int cifs_fsync (struct file *,loff_t ,loff_t ,int );
fs/cifs/file.c +0 -16
@@ -5066,19 +5066,3 @@ const struct address_space_operations cifs_addr_ops_smallbuf = { .launder_folio = cifs_launder_folio, .migrate_folio = filemap_migrate_folio, }; - -/* -* Splice data from a file into a pipe. -*/ -ssize_t cifs_splice_read (struct file *in,loff_t *ppos, -struct pipe_inode_info *pipe,size_t len, -unsigned int flags) -{ -if (unlikely (*ppos >=file_inode (in)->i_sb->s_maxbytes)) -return 0 ; -if (unlikely (!len)) -return 0 ; -if (in->f_flags &O_DIRECT ) -return direct_splice_read (in, ppos, pipe, len, flags); -return filemap_splice_read (in, ppos, pipe, len, flags); -}
mm/filemap.c +0 -1
@@ -2967,7 +2967,6 @@ ssize_t filemap_splice_read(struct file *in, loff_t *ppos,return total_spliced ? total_spliced : error; } -EXPORT_SYMBOL (filemap_splice_read); static inline loff_t folio_seek_hole_data(struct xa_state *xas, struct address_space *mapping, struct folio *folio,
Re: [PATCH v18 08/15] cifs: Use generic_file_splice_read()
Quoted 21 lines · David Howells
Make cifs use
generic_file_splice_read()rather than doing it for itself.As a consequence,
filemap_splice_read()no longer needs to be exported.Signed-off-by David Howells <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>cc Jens Axboe <[email protected]>cc Steve French <[email protected]>cc Al Viro <[email protected]>cc David Hildenbrand <[email protected]>cc John Hubbard <[email protected]>---
Notes:
ver #18)
- Split out from change togeneric_file_splice_read().
Quoted 5 lines
fs/cifs/cifsfs.c | 8 ++++---- fs/cifs/cifsfs.h | 3 --- fs/cifs/file.c | 16 ---------------- mm/filemap.c | 1 - 4 files changed, 4 insertions(+), 24 deletions(-)
PATCH v18 9/15 iov_iter: Kill ITER_PIPE
In reply to David HowellsThe ITER_PIPE-type iterator was only used for generic_file_splice_read(), but that has now been switched to either pull pages directly from the pagecache for buffered file splice-reads or to use ITER_BVEC instead for O_DIRECT file splice-reads. This leaves ITER_PIPE unused - so remove it.
--- include/linux/uio.h | 14 -- lib/iov_iter.c | 429 +------------------------------------------- mm/filemap.c | 3 +- 3 files changed, 4 insertions(+), 442 deletions(-)
include/linux/uio.h +0 -14
@@ -11,7 +11,6 @@#include <uapi/linux/uio.h> struct page ; -struct pipe_inode_info ;typedef unsigned int __bitwise iov_iter_extraction_t ;
@@ -25,7 +24,6 @@ enum iter_type {ITER_IOVEC ,ITER_KVEC ,ITER_BVEC , -ITER_PIPE ,ITER_XARRAY ,ITER_DISCARD , ITER_UBUF,
@@ -55,15 +53,10 @@ struct iov_iter {const struct kvec *kvec;const struct bio_vec *bvec;struct xarray *xarray; -struct pipe_inode_info *pipe;void __user *ubuf; };union {unsigned long nr_segs; -struct { -unsigned int head; -unsigned int start_head; - };loff_t xarray_start; }; };
@@ -101,11 +94,6 @@ static inline bool iov_iter_is_bvec(const struct iov_iter *i)return iov_iter_type (i) ==ITER_BVEC ; } -static inline bool iov_iter_is_pipe (const struct iov_iter *i) -{ -return iov_iter_type (i) ==ITER_PIPE ; -} -static inline bool iov_iter_is_discard (const struct iov_iter *i) {return iov_iter_type (i) ==ITER_DISCARD ;
@@ -247,8 +235,6 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, const struct kvec unsigned long nr_segs, size_t count);void iov_iter_bvec (struct iov_iter *i,unsigned int direction,const struct bio_vec *bvec,unsigned long nr_segs,size_t count); -void iov_iter_pipe (struct iov_iter *i,unsigned int direction,struct pipe_inode_info *pipe, -size_t count);void iov_iter_discard (struct iov_iter *i,unsigned int direction,size_t count);void iov_iter_xarray (struct iov_iter *i,unsigned int direction,struct xarray *xarray,loff_t start,size_t count);
lib/iov_iter.c +3 -426
555 lines not on this page · raw message
mm/filemap.c +1 -2
@@ -2690,8 +2690,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,if (unlikely (iocb->ki_pos >=i_size_read (inode)))break ; - error =filemap_get_pages (iocb, iter->count, &fbatch, -iov_iter_is_pipe (iter)); + error =filemap_get_pages (iocb, iter->count, &fbatch, false);if (error <0 )break ;
Al Viro
maintainerRe: [PATCH v18 09/15] iov_iter: Kill ITER_PIPE
Quoted 4 lines · David Howells
The ITER_PIPE-type iterator was only used for
generic_file_splice_read(), but that has now been switched to either pull pages directly from the pagecache for buffered file splice-reads or to useITER_BVECinstead forO_DIRECTfile splice-reads. This leavesITER_PIPEunused - so remove it.
Wonderful, except that now you've got duplicates of ->read_iter() for everyone who wants zero-copy on ->splice_read() ;-/
I understand the attraction of arbitrary seeks on those suckers; ITER_PIPE is a massive headache in that respect. But I really don't like what your approach trades it for.
And you are nowhere near done - consider e.g. NFS. Mainline has it feed ITER_PIPE to nfs_file_read(), which does call generic_file_read_iter() - after
result = nfs_revalidate_mapping(inode, iocb->ki_filp->f_mapping);
Sure, you can add nfs_file_splice_read() that would do what nfs_file_read() does, calling filemap_spice_read() instead of generic_file_read_iter().
Repeat the same for ocfs2 (locking of its own). And orangefs. And XFS (locking, again). And your own AFS, while we are at it. Et sodding cetera - *everything* that uses generic_file_splice_read() with ->read_iter other than generic_file_read_iter() needs review and, quite likely, a ->splice_read() instance of its own.
Al Viro
maintainerRe: [PATCH v18 09/15] iov_iter: Kill ITER_PIPE
Quoted 18 lines · Al Viro
Quoted 4 lines · David Howells
The ITER_PIPE-type iterator was only used for
generic_file_splice_read(), but that has now been switched to either pull pages directly from the pagecache for buffered file splice-reads or to useITER_BVECinstead forO_DIRECTfile splice-reads. This leavesITER_PIPEunused - so remove it.Wonderful, except that now you've got duplicates of ->read_iter() for everyone who wants zero-copy on ->splice_read() ;-/
I understand the attraction of arbitrary seeks on those suckers;
ITER_PIPEis a massive headache in that respect. But I really don't like what your approach trades it for.And you are nowhere near done - consider e.g. NFS. Mainline has it feed
ITER_PIPEtonfs_file_read(), which does callgeneric_file_read_iter()- afterresult = nfs_revalidate_mapping(inode, iocb->ki_filp->f_mapping);
Quoted 8 lines
Sure, you can add
nfs_file_splice_read()that would do whatnfs_file_read()does, callingfilemap_spice_read()instead ofgeneric_file_read_iter().Repeat the same for ocfs2 (locking of its own). And orangefs. And XFS (locking, again). And your own AFS, while we are at it. Et sodding cetera - *everything* that uses
generic_file_splice_read()with ->read_iter other thangeneric_file_read_iter()needs review and, quite likely, a ->splice_read() instance of its own.
Don't get me wrong - I'd love to kill ITER_PIPE off; it's just that tons of ->splice_read() duplicating the corresponding ->read_iter() up to the point where it would call generic_file_read_iter(), modulo ignoring O_DIRECT case and then calling filemap_splice_read() instead... Painful and asking for trouble down the road.
Al Viro
maintainerRe: [PATCH v18 09/15] iov_iter: Kill ITER_PIPE
Quoted 5 lines · Al Viro
Don't get me wrong - I'd love to kill
ITER_PIPEoff; it's just that tons of ->splice_read() duplicating the corresponding ->read_iter() up to the point where it would callgeneric_file_read_iter(), modulo ignoringO_DIRECTcase and then callingfilemap_splice_read()instead... Painful and asking for trouble down the road.
FWIW, broken in this series:
* a bunch of 9p variants (those could probably switch to direct)
* afs
* ceph (very likely)
* ecryptfs (atime propagation)
* dax on various filesystems (probably)
* f2fs
* gfs2
* nfs
* ocfs2
* orangefs
* xfs
* zonefs (probably)
* splice from UDP sockets, unless I'm misreading something.
Your sock_splice_read() still falls back to generic_file_splice_read(), rather than to direct_splice_read() and sockets don't have O_DIRECT in flags. Neither do they have associated page cache ;-)
Sure, we could provide a bunch of ->splice_read(), but then we'd have to make sure that all subsequent changes to matching ->read_iter() get duplicated - unless they are on O_DIRECT-only paths, that is...
Al Viro
maintainerRe: [PATCH v18 09/15] iov_iter: Kill ITER_PIPE
Quoted 26 lines · Al Viro
Quoted 5 lines · Al Viro
Don't get me wrong - I'd love to kill
ITER_PIPEoff; it's just that tons of ->splice_read() duplicating the corresponding ->read_iter() up to the point where it would callgeneric_file_read_iter(), modulo ignoringO_DIRECTcase and then callingfilemap_splice_read()instead... Painful and asking for trouble down the road.FWIW, broken in this series:
* a bunch of 9p variants (those could probably switch to direct)
* afs
* ceph (very likely)
* ecryptfs (atime propagation)
* dax on various filesystems (probably)
* f2fs
* gfs2
* nfs
* ocfs2
* orangefs
* xfs
* zonefs (probably)
* splice from UDP sockets, unless I'm misreading something.
Yoursock_splice_read()still falls back togeneric_file_splice_read(), rather than todirect_splice_read()and sockets don't haveO_DIRECTin flags. Neither do they have associated page cache ;-)
Quoted 3 lines
Sure, we could provide a bunch of ->splice_read(), but then we'd have to make sure that all subsequent changes to matching ->read_iter() get duplicated - unless they are on O_DIRECT-only paths, that is...
OK, how about the following modification of ITER_PIPE?
* buffers might extend past the current position
* buffers (as now) are either data or zero-copy ones.
* advance past the end of data => allocate data-type buffers
and fill with zeroes.
* copying data to area prior to the end of buffers => each
zero-copy-type buffer is converted to a data one; data is copied, but WTF else can you do? If there's a page borrowed from page cache, we can't change its contents anyway.
* zero-copy of a page into an area overlapping the existing
buffers => treat as data copy for the overlapping part and zero-copy the rest (if any) in normal fashion.
* iov_iter_get_pages(): convert everything involved into
data-type buffers first, then give caller references to that. Zero-fill new pages (if any).
* iov_iter_revert() does *NOT* truncate anything
* truncation of stuff past the current position is done bygeneric_file_splice_read() not only in error case, but on success as well - starting at the end of actually read data, of course.
That would make for seekable ITER_PIPE, with zero-copy working for normal cases. I agree that O_DIRECT would probably be better off with ITER_BVEC.
Comments?
PATCH v18 10/15 iomap: Don't get an reference on ZERO_PAGE for direct I/O block zeroing
In reply to David HowellsZERO_PAGE can't go away, no need to hold an extra reference.
--- fs/iomap/direct-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/iomap/direct-io.c +1 -1
@@ -202,7 +202,7 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; -get_page (page); +bio_set_flag (bio,BIO_NO_PAGE_REF );__bio_add_page (bio, page, len,0 );iomap_dio_submit_bio (iter, dio, bio, pos); }
PATCH v18 11/15 block: Fix bio_flagged() so that gcc can better optimise it
In reply to David HowellsFix bio_flagged() so that multiple instances of it, such as:
if (bio_flagged(bio, BIO_PAGE_REFFED) || bio_flagged(bio, BIO_PAGE_PINNED))
can be combined by the gcc optimiser into a single test in assembly (arguably, this is a compiler optimisation issue[1]).
The missed optimisation stems from bio_flagged() comparing the result of the bitwise-AND to zero. This results in an out-of-line bio_release_page() being compiled to something like:
<+0>: mov 0x14(%rdi),%eax <+3>: test $0x1,%al <+5>: jne 0xffffffff816dac53 <bio_release_pages+11> <+7>: test $0x2,%al <+9>: je 0xffffffff816dac5c <bio_release_pages+20> <+11>: movzbl %sil,%esi <+15>: jmp 0xffffffff816daba1 <__bio_release_pages> <+20>: jmp 0xffffffff81d0b800 <__x86_return_thunk>
However, the test is superfluous as the return type is bool. Removing it results in:
<+0>: testb $0x3,0x14(%rdi) <+4>: je 0xffffffff816e4af4 <bio_release_pages+15> <+6>: movzbl %sil,%esi <+10>: jmp 0xffffffff816dab7c <__bio_release_pages> <+15>: jmp 0xffffffff81d0b7c0 <__x86_return_thunk>
instead.
Also, the MOVZBL instruction looks unnecessary[2] - I think it's just 're-booling' the mark_dirty parameter.
--- include/linux/bio.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/bio.h +1 -1
@@ -229,7 +229,7 @@ static inline void bio_cnt_set(struct bio *bio, unsigned int count)static inline bool bio_flagged (struct bio *bio,unsigned int bit) { -return (bio->bi_flags & (1U << bit)) !=0 ; +return bio->bi_flags & (1U << bit); }static inline void bio_set_flag (struct bio *bio,unsigned int bit)
PATCH v18 12/15 block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted logic
In reply to David HowellsFrom: Christoph Hellwig <[email protected]>
Replace BIO_NO_PAGE_REF with a BIO_PAGE_REFFED flag that has the inverted meaning is only set when a page reference has been acquired that needs to be released by bio_release_pages().
---
Notes:
ver #8)
- Split out from another patch [hch].
- Don't default to BIO_PAGE_REFFED [hch].
ver #5)
- Split from patch that uses iov_iter_extract_pages().
block/bio.c | 2 +- block/blk-map.c | 1 + fs/direct-io.c | 2 ++ fs/iomap/direct-io.c | 1 - include/linux/bio.h | 2 +- include/linux/blk_types.h | 2 +- 6 files changed, 6 insertions(+), 4 deletions(-)
block/bio.c +1 -1
@@ -1190,7 +1190,6 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_bvec_done = iter->iov_offset; bio->bi_iter.bi_size = size; -bio_set_flag (bio,BIO_NO_PAGE_REF );bio_set_flag (bio,BIO_CLONED ); }
@@ -1335,6 +1334,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)return 0 ; } +bio_set_flag (bio,BIO_PAGE_REFFED );do { ret =__bio_iov_iter_get_pages (bio, iter); }while (!ret &&iov_iter_count (iter) && !bio_full (bio,0 ));
block/blk-map.c +1 -0
@@ -281,6 +281,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (blk_queue_pci_p2pdma (rq->q)) extraction_flags |=ITER_ALLOW_P2PDMA ; +bio_set_flag (bio,BIO_PAGE_REFFED );while (iov_iter_count (iter)) {struct page **pages, *stack_pages[UIO_FASTIOV ];ssize_t bytes;
fs/direct-io.c +2 -0
@@ -403,6 +403,8 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_end_io = dio_bio_end_aio; else bio->bi_end_io = dio_bio_end_io; +/* for now require references for all pages */ +bio_set_flag (bio,BIO_PAGE_REFFED ); sdio->bio = bio; sdio->logical_offset_in_bio = sdio->cur_page_fs_offset; }
fs/iomap/direct-io.c +0 -1
@@ -202,7 +202,6 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; -bio_set_flag (bio,BIO_NO_PAGE_REF );__bio_add_page (bio, page, len,0 );iomap_dio_submit_bio (iter, dio, bio, pos); }
include/linux/bio.h +1 -1
@@ -488,7 +488,7 @@ void zero_fill_bio(struct bio *bio);static inline void bio_release_pages (struct bio *bio,bool mark_dirty) { -if (!bio_flagged (bio,BIO_NO_PAGE_REF )) +if (bio_flagged (bio,BIO_PAGE_REFFED ))__bio_release_pages (bio, mark_dirty); }
include/linux/blk_types.h +1 -1
@@ -318,7 +318,7 @@ struct bio {* bio flags */ enum { -BIO_NO_PAGE_REF ,/* don't put release vec pages */ +BIO_PAGE_REFFED ,/* put pages in bio_release_pages() */ BIO_CLONED ,/* doesn't own data */ BIO_BOUNCED ,/* bio is a bounce bio */ BIO_QUIET ,/* Make BIO Quiet */
PATCH v18 13/15 block: Add BIO_PAGE_PINNED and associated infrastructure
In reply to David HowellsAdd BIO_PAGE_PINNED to indicate that the pages in a bio are pinned (FOLL_PIN) and that the pin will need removing.
---
Notes:
ver #10)
- Drop bio_set_cleanup_mode(), open coding it instead.
ver #9)
- Only consider pinning in bio_set_cleanup_mode(). Ref'ing pages in struct bio is going away.
- page_put_unpin() is removed; call unpin_user_page() and put_page() directly.
- Use bio_release_page() in __bio_release_pages().
- BIO_PAGE_PINNED and BIO_PAGE_REFFED can't both be set, so use if-else when testing both of them.
ver #8)
- Move the infrastructure to clean up pinned pages to this patch [hch].
- Put BIO_PAGE_PINNED before BIO_PAGE_REFFED as the latter should probably be removed at some point. FOLL_PIN can then be renumbered first.
block/bio.c | 6 +++--- block/blk.h | 12 ++++++++++++ include/linux/bio.h | 3 ++- include/linux/blk_types.h | 1 + 4 files changed, 18 insertions(+), 4 deletions(-)
block/bio.c +3 -3
@@ -1168,7 +1168,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty)bio_for_each_segment_all (bvec, bio, iter_all) {if (mark_dirty && !PageCompound (bvec->bv_page))set_page_dirty_lock (bvec->bv_page); -put_page (bvec->bv_page); +bio_release_page (bio, bvec->bv_page); } }EXPORT_SYMBOL_GPL (__bio_release_pages);
@@ -1488,8 +1488,8 @@ void bio_set_pages_dirty(struct bio *bio)* the BIO and re-dirty the pages in process context. * * It is expected that bio_check_pages_dirty() will wholly own the BIO from -* here on. It will run one put_page() against each page and will run one -* bio_put() against the BIO. +* here on. It will unpin each page and will run one bio_put() against the +* BIO. */ static void bio_dirty_fn (struct work_struct *work);
block/blk.h +12 -0
@@ -432,6 +432,18 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); +/* +* Clean up a page appropriately, where the page may be pinned, may have a +* ref taken on it or neither. +*/ +static inline void bio_release_page (struct bio *bio,struct page *page) +{ +if (bio_flagged (bio,BIO_PAGE_PINNED )) +unpin_user_page (page); +else if (bio_flagged (bio,BIO_PAGE_REFFED )) +put_page (page); +} +struct request_queue *blk_alloc_queue (int node_id);int disk_scan_partitions (struct gendisk *disk,fmode_t mode);
include/linux/bio.h +2 -1
@@ -488,7 +488,8 @@ void zero_fill_bio(struct bio *bio);static inline void bio_release_pages (struct bio *bio,bool mark_dirty) { -if (bio_flagged (bio,BIO_PAGE_REFFED )) +if (bio_flagged (bio,BIO_PAGE_REFFED ) || +bio_flagged (bio,BIO_PAGE_PINNED ))__bio_release_pages (bio, mark_dirty); }
include/linux/blk_types.h +1 -0
@@ -318,6 +318,7 @@ struct bio {* bio flags */ enum { +BIO_PAGE_PINNED ,/* Unpin pages in bio_release_pages() */ BIO_PAGE_REFFED ,/* put pages in bio_release_pages() */ BIO_CLONED ,/* doesn't own data */ BIO_BOUNCED ,/* bio is a bounce bio */
PATCH v18 14/15 block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages
In reply to David HowellsThis will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator.
The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being affected by/visible to the child process).
---
Notes:
ver #10)
- Drop bio_set_cleanup_mode(), open coding it instead.
ver #8)
- Split the patch up a bit [hch].
- We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately.
ver #7)
- Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN.
ver #5)
- Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode.
- Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch.
block/bio.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
block/bio.c +12 -11
@@ -1204,7 +1204,7 @@ static int bio_iov_add_page(struct bio *bio, struct page *page, }if (same_page) -put_page (page); +bio_release_page (bio, page);return 0 ; }
@@ -1218,7 +1218,7 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, queue_max_zone_append_sectors(q), &same_page) != len)return -EINVAL ;if (same_page) -put_page (page); +bio_release_page (bio, page);return 0 ; }
@@ -1229,10 +1229,10 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page,* @bio: bio to add pages to * @iter: iov iterator describing the region to be mapped * -* Pins pages from *iter and appends them to @bio's bvec array. The -* pages will have to be released using put_page() when done. -* For multi-segment *iter, this function only adds pages from the -* next non-empty segment of the iov iterator. +* Extracts pages from *iter and appends them to @bio's bvec array. The pages +* will have to be cleaned up in the way indicated by the BIO_PAGE_PINNED flag. +* For a multi-segment *iter, this function only adds pages from the next +* non-empty segment of the iov iterator. */ static int __bio_iov_iter_get_pages (struct bio *bio,struct iov_iter *iter) {
@@ -1264,9 +1264,9 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size =iov_iter_get_pages (iter, pages, -UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, extraction_flags); + size =iov_iter_extract_pages (iter, &pages, +UINT_MAX - bio->bi_iter.bi_size, + nr_pages, extraction_flags, &offset);if (unlikely (size <=0 ))return size ? size : -EFAULT ;
@@ -1299,7 +1299,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)iov_iter_revert (iter, left); out:while (i < nr_pages) -put_page (pages[i++]); +bio_release_page (bio, pages[i++]);return ret; }
@@ -1334,7 +1334,8 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)return 0 ; } -bio_set_flag (bio,BIO_PAGE_REFFED ); +if (iov_iter_extract_will_pin (iter)) +bio_set_flag (bio,BIO_PAGE_PINNED );do { ret =__bio_iov_iter_get_pages (bio, iter); }while (!ret &&iov_iter_count (iter) && !bio_full (bio,0 ));
PATCH v18 15/15 block: convert bio_map_user_iov to use iov_iter_extract_pages
In reply to David HowellsThis will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator.
The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being visible to/affected by the child process).
---
Notes:
ver #10)
- Drop bio_set_cleanup_mode(), open coding it instead.
ver #8)
- Split the patch up a bit [hch].
- We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately.
ver #7)
- Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN.
ver #5)
- Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode.
- Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch.
block/blk-map.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-)
block/blk-map.c +11 -12
@@ -280,22 +280,21 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (blk_queue_pci_p2pdma (rq->q)) extraction_flags |=ITER_ALLOW_P2PDMA ; +if (iov_iter_extract_will_pin (iter)) +bio_set_flag (bio,BIO_PAGE_PINNED ); -bio_set_flag (bio,BIO_PAGE_REFFED );while (iov_iter_count (iter)) { -struct page **pages, *stack_pages[UIO_FASTIOV ]; +struct page *stack_pages[UIO_FASTIOV ]; +struct page **pages = stack_pages;ssize_t bytes;size_t offs;int npages; -if (nr_vecs <=ARRAY_SIZE (stack_pages)) { - pages = stack_pages; - bytes =iov_iter_get_pages (iter, pages,LONG_MAX , - nr_vecs, &offs, extraction_flags); - }else { - bytes =iov_iter_get_pages_alloc (iter, &pages, -LONG_MAX , &offs, extraction_flags); - } +if (nr_vecs >ARRAY_SIZE (stack_pages)) + pages =NULL ; + + bytes =iov_iter_extract_pages (iter, &pages,LONG_MAX , + nr_vecs, extraction_flags, &offs);if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ; goto out_unmap;
@@ -317,7 +316,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (!bio_add_hw_page (rq->q, bio, page, n, offs, max_sectors, &same_page)) {if (same_page) -put_page (page); +bio_release_page (bio, page);break ; }
@@ -329,7 +328,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,* release the pages we didn't map into the bio, if any */ while (j < npages) -put_page (pages[j++]); +bio_release_page (bio, pages[j++]);if (pages != stack_pages)kvfree (pages);/* couldn't stuff something into bio? */