iov_iter stuff
Subsystems (26)
Al Viro
maintainerRFC PATCH 1/1 iov_iter stuff
There's a bunch of pending iov_iter-related work; most of that had
been posted, but only one part got anything resembling a review. Currently it seems to be working, but it obviously needs review and testing.
It's split into several subseries; the entire series can be observed
as v5.19-rc2..#work.iov_iter_get_pages. Description follows; individual patches will be posted^Wmailbombed in followups.
This stuff is not in -next yet; I'd like to put it there, so if you
see any problems - please yell.
One thing not currently in there, but to be added very soon is
iov_iter_find_pages{,_alloc}() - analogue of iov_iter_get_pages(), except that it only grabs page references for userland-backed flavours. The callers, of course, are responsible for keeping the underlying object(s) alive for as long as they are using the results. Quite a few of iov_iter_get_pages() callers would be fine with that. Moreover, unlike iov_iter_get_pages() this could be allowed for ITER_KVEC, potentially eliminating several places where we special-case the treatment of ITER_KVEC.
Another pending thing is integration with cifs and ceph series (dhowells
and jlayton resp.) and probably io_uring as well.
----------------------------------------------------------------------------
Part 1, #work.9p: [rc1-based]
1/44: 9p: handling Rerror without copy_from_iter_full()
Self-contained fix, should be easy to backport. What happens
there is that arrival of Rerror in response to zerocopy read or readdir ends up with error string in the place where the actual data would've gone in case of success. It needs to be extracted, and copy_from_iter_full() is only for data-source iterators, not for e.g. ITER_PIPE. And ITER_PIPE can be used with those...
----------------------------------------------------------------------------
Part 2, #work.iov_iter: [rc1-based]
Dealing with the overhead in new_sync_read()/new_sync_write(), mostly. Several things there - one is that calculation of iocb flags can be made cheaper, another is that single-segment iovec is sufficiently common to be worth turning into a new iov_iter flavour (ITER_UBUF). With all that, the total size of iov_iter.c goes down, mostly due to removal of magic in iovec copy_page_to_iter()/copy_page_from_iter(). Generic variant works for those nowadays...
This had been posted two weeks ago, got a reasonable amount of comments.
2/44: No need of likely/unlikely on calls of check_copy_size()
not just in uio.h; the thing is inlined and it has unlikely on
all paths leading to return false
3/44: teach iomap_dio_rw() to suppress dsync
new flag for iomap_dio_rw(), telling it to suppress generic_write_sync()
4/44: btrfs: use IOMAP_DIO_NOSYNC
use the above instead of currently used kludges.
5/44: struct file: use anonymous union member for rcuhead and llist
"f_u" might have been an amusing name, but... we expect anon unions to
work.
6/44: iocb: delay evaluation of IS_SYNC(...) until we want to check IOCB_DSYNC
makes iocb_flags() much cheaper, and it's easier to keep track of
the places where it can change.
7/44: keep iocb_flags() result cached in struct file
that, along with the previous commit, reduces the overhead of
new_sync_{read,write}(). struct file doesn't grow - we can keep that thing in the same anon union where rcuhead and llist live; that field gets used only before ->f_count reaches zero while the other two are used only after ->f_count has reached zero.
8/44: copy_page_{to,from}_iter(): switch iovec variants to generic
kmap_local_page() allows that. And it kills quite a bit of
code.
9/44: new iov_iter flavour - ITER_UBUF
iovec analogue, with single segment. That case is fairly common and it
can be handled with less overhead than full-blown iovec.
10/44: switch new_sync_{read,write}() to ITER_UBUF
... and this is why it is so common. Further reduction of overhead
for new_sync_{read,write}().
11/44: iov_iter_bvec_advance(): don't bother with bvec_iter
AFAICS, variant similar to what we do for iovec/kvec generates better
code. Needs profiling, obviously.
----------------------------------------------------------------------------
Part 3, #fixes [-rc2-based]
12/44: fix short copy handling in copy_mc_pipe_to_iter()
Minimal version of fix; it's replaced with prettier one in the next
series, but replacement is not a backport fodder.
----------------------------------------------------------------------------
Part 4, #work.ITER_PIPE [on top of merge of previous branches]
ITER_PIPE handling had never been pretty, but by now it has become really obfuscated and hard to read. Untangle it a bit. Posted last weekend, some brainos fixed since then.
13/44: splice: stop abusing iov_iter_advance() to flush a pipe
A really odd (ab)use of iov_iter_advance() - in case of errorgeneric_file_splice_read() wants to free all pipe buffers ->read_iter() has produced. Yes, forcibly resetting ->head and ->iov_offset to original values and calling iov_iter_advance(i, 0) will trigger pipe_advance(), which will trigger pipe_truncate(), which will free buffers. Or we could just go ahead and free the same buffers; pipe_discard_from() does exactly that, no iov_iter stuff needs to be involved.
14/44: ITER_PIPE: helper for getting pipe buffer by index
In a lot of places we want to find pipe_buffer by index;
expression is convoluted and hard to read. Provide an inline helper for that, convert trivial open-coded cases. Eventually *all* open-coded instances in iov_iter.c will be gone.
15/44: ITER_PIPE: helpers for adding pipe buffers
There are only two kinds of pipe_buffer in the area used by ITER_PIPE.
* anonymous - copy_to_iter() et.al. end up creating those and copying data there. They have zero ->offset, and their ->ops points to default_pipe_page_ops.
* zero-copy ones - those come from copy_page_to_iter(), and page comes from caller. ->offset is also caller-supplied - it might be non-zero. ->ops points to page_cache_pipe_buf_ops.
Move creation and insertion of those into helpers -push_anon(pipe, size) and push_page(pipe, page, offset, size) resp., separating them from the "could we avoid creating a new buffer by merging with the current head?" logics.
16/44: ITER_PIPE: allocate buffers as we go in copy-to-pipe primitives
New helper: append_pipe(). Extends the last buffer if possible,
allocates a new one otherwise. Returns page and offset in it on success, NULL on failure. iov_iter is advanced past the data we've got.
Use that instead of push_pipe() in copy-to-pipe primitives;
they get simpler that way. Handling of short copy (in "mc" one) is done simply by iov_iter_revert() - iov_iter is in consistent state after that one, so we can use that.
17/44: ITER_PIPE: fold push_pipe() into __pipe_get_pages()
Expand the only remaining call of push_pipe() (in__pipe_get_pages()), combine it with the page-collecting loop there. We don't need to bother with i->count checks or calculation of offset in the first page - the caller already has done that.
Note that the only reason it's not a loop doing append_pipe()
is that append_pipe() is advancing, while iov_iter_get_pages() is not. As soon as it switches to saner semantics, this thing will switch to using append_pipe().
18/44: ITER_PIPE: lose iter_head argument of __pipe_get_pages()
Redundant.
19/44: ITER_PIPE: clean pipe_advance() up
Don't bother with pipe_truncate(); adjust the buffer
length just as we decide it'll be the last one, then use pipe_discard_from() to release buffers past that one.
20/44: ITER_PIPE: clean iov_iter_revert()
Fold pipe_truncate() in there, clean the things up.
21/44: ITER_PIPE: cache the type of last buffer
We often need to find whether the last buffer is anon or not, and
currently it's rather clumsy:
check if ->iov_offset is non-zero (i.e. that pipe is not empty)
if so, get the corresponding pipe_buffer and check its ->ops
if it's &default_pipe_buf_ops, we have an anon buffer.Let's replace the use of ->iov_offset (which is nowhere near similar to its role for other flavours) with signed field (->last_offset), with the following rules:
empty, no buffers occupied: 0
anon, with bytes up to N-1 filled: N
zero-copy, with bytes up to N-1 filled: -N
That way abs(i->last_offset) is equal to what used to be in i->iov_offset and empty vs. anon vs. zero-copy can be distinguished by the sign of i->last_offset.
Checks for "should we extend the last buffer or should we start
a new one?" become easier to follow that way.
Note that most of the operations can only be done in a sane
state - i.e. when the pipe has nothing past the current position of iterator. About the only thing that could be done outside of that state is iov_iter_advance(), which transitions to the sane state by truncating the pipe. There are only two cases where we leave the sane state:
1) iov_iter_get_pages()/iov_iter_get_pages_alloc(). Will be
dealt with later, when we make get_pages advancing - the callers are actually happier that way.
2) iov_iter copied, then something is put into the copy. Since
they share the underlying pipe, the original gets behind. When we decide that we are done with the copy (original is not usable until then) we advance the original. direct_io used to be done that way; nowadays it operates on the original and we do iov_iter_revert() to discard the excessive data. At the moment there's nothing in the kernel that could do that to ITER_PIPE iterators, so this reason for insane state is theoretical right now.
22/44: ITER_PIPE: fold data_start() and pipe_space_for_user() together
All their callers are next to each other; all of them want
the total amount of pages and, possibly, the offset in the partial final buffer.
Combine into a new helper (pipe_npages()), fix the
bogosity in pipe_space_for_user(), while we are at it.
----------------------------------------------------------------------------
Part 5, #work.unify_iov_iter_get_pages [on top of previous]
iov_iter_get_pages() and iov_iter_get_pages_alloc() have a lot of code duplication and are bloody hard to read. With some massage duplication can be eliminated, along with some of the cruft accumulated there.
Flavour-independent arguments validation and, for ..._alloc(),
cleanup handling on failure:
23/44: iov_iter_get_pages{,_alloc}(): cap the maxsize with MAX_RW_COUNT 24/44: iov_iter_get_pages_alloc(): lift freeing pages array on failure exits into wrapper 25/44: iov_iter_get_pages(): sanity-check arguments
Mechanically merge parallel ..._get_pages() and ..._get_pages_alloc().
26/44: unify pipe_get_pages() and pipe_get_pages_alloc()
27/44: unify xarray_get_pages() and xarray_get_pages_alloc() 28/44: unify the rest of iov_iter_get_pages()/iov_iter_get_pages_alloc() guts
Decrufting for XARRAY:
29/44: ITER_XARRAY: don't open-code DIV_ROUND_UP()
Decrufting for UBUF/IOVEC/BVEC: that bunch suffers from really convoluted
helpers; untangling those takes a bit of care, so I'd carved that up into fairly small chunks. Could be collapsed together, but...
30/44: iov_iter: lift dealing with maxpages out of first_{iovec,bvec}_segment() 31/44: iov_iter: first_{iovec,bvec}_segment() - simplify a bit 32/44: iov_iter: massage calling conventions for first_{iovec,bvec}_segment() 33/44: found_iovec_segment(): just return address
Decrufting for PIPE:
34/44: fold __pipe_get_pages() into pipe_get_pages()
Now we can finally get a helper encapsulating the array allocations
right way:
35/44: iov_iter: saner helper for page array allocation
----------------------------------------------------------------------------
Part 6, #work.iov_iter_get_pages-advance [on top of previous]
Convert iov_iter_get_pages{,_alloc}() to iterator-advancing semantics.
Most of the callers follow successful ...get_pages... with advance
by the amount it had reported. For some it's unconditional, for some it might end up being less in some cases. All of them would be fine with advancing variants of those primitives - those that might want to advance by less than reported could easily use revert by the difference of those amounts.
Rather than doing a flagday change (they are exported and signatures
remain unchanged), replacement variants are added (iov_iter_get_pages2() and iov_iter_get_pages_alloc2(), initially as wrappers). By the end of the series everything is converted to those and the old ones are removed.
Makes for simpler rules for ITER_PIPE, among other things, and
advancing semantics is consistent with all data-copying primitives. Series is pretty obvious - introduce variants with new semantics, switch users one by one, fold the old variants into new ones.
36/44: iov_iter: advancing variants of iov_iter_get_pages{,_alloc}() 37/44: block: convert to advancing variants of iov_iter_get_pages{,_alloc}() 38/44: iter_to_pipe(): switch to advancing variant of iov_iter_get_pages() 39/44: af_alg_make_sg(): switch to advancing variant of iov_iter_get_pages() 40/44: 9p: convert to advancing variant of iov_iter_get_pages_alloc() 41/44: ceph: switch the last caller of iov_iter_get_pages_alloc() 42/44: get rid of non-advancing variants
----------------------------------------------------------------------------
Part 7, #wort.iov_iter_get_pages [on top of previous]
Trivial followups, with more to be added here...
43/44: pipe_get_pages(): switch to append_pipe()
44/44: expand those iov_iter_advance()...
Overall diffstat:
arch/powerpc/include/asm/uaccess.h | 2 +- arch/s390/include/asm/uaccess.h | 4 +- block/bio.c | 15 +- block/blk-map.c | 7 +- block/fops.c | 8 +- crypto/af_alg.c | 3 +- crypto/algif_hash.c | 5 +- drivers/nvme/target/io-cmd-file.c | 2 +- drivers/vhost/scsi.c | 4 +- fs/aio.c | 2 +- fs/btrfs/file.c | 19 +- fs/btrfs/inode.c | 3 +- fs/ceph/addr.c | 2 +- fs/ceph/file.c | 5 +- fs/cifs/file.c | 8 +- fs/cifs/misc.c | 3 +- fs/direct-io.c | 7 +- fs/fcntl.c | 1 + fs/file_table.c | 17 +- fs/fuse/dev.c | 7 +- fs/fuse/file.c | 7 +- fs/gfs2/file.c | 2 +- fs/io_uring.c | 2 +- fs/iomap/direct-io.c | 21 +- fs/nfs/direct.c | 8 +- fs/open.c | 1 + fs/read_write.c | 6 +- fs/splice.c | 54 +- fs/zonefs/super.c | 2 +- include/linux/fs.h | 21 +- include/linux/iomap.h | 6 + include/linux/pipe_fs_i.h | 29 +- include/linux/uaccess.h | 4 +- include/linux/uio.h | 50 +- lib/iov_iter.c | 993 ++++++++++++++----------------------- mm/shmem.c | 2 +- net/9p/client.c | 125 +---- net/9p/protocol.c | 3 +- net/9p/trans_virtio.c | 37 +- net/core/datagram.c | 3 +- net/core/skmsg.c | 3 +- net/rds/message.c | 3 +- net/tls/tls_sw.c | 4 +- 43 files changed, 599 insertions(+), 911 deletions(-)
Al Viro
maintainerPATCH 3/44 teach iomap_dio_rw() to suppress dsync
In reply to Al ViroNew flag, equivalent to removal of IOCB_DSYNC from iocb flags. This mimics what btrfs is doing (and that's what btrfs will switch to). However, I'm not at all sure that we want to suppress REQ_FUA for those - all btrfs hack really cares about is suppression of generic_write_sync(). For now let's keep the existing behaviour, but I really want to hear more detailed arguments pro or contra.
[folded brain fix from willy]
--- fs/iomap/direct-io.c | 20 +++++++++++--------- include/linux/iomap.h | 6 ++++++ 2 files changed, 17 insertions(+), 9 deletions(-)
fs/iomap/direct-io.c +11 -9
@@ -548,17 +548,19 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, }/* for data sync or sync, we need sync completion processing */ -if (iocb->ki_flags &IOCB_DSYNC ) +if (iocb->ki_flags &IOCB_DSYNC && + !(dio_flags &IOMAP_DIO_NOSYNC )) { dio->flags |=IOMAP_DIO_NEED_SYNC ; -/* -* For datasync only writes, we optimistically try using FUA for -* this IO. Any non-FUA write that occurs will clear this flag, -* hence we know before completion whether a cache flush is -* necessary. -*/ -if ((iocb->ki_flags & (IOCB_DSYNC |IOCB_SYNC )) ==IOCB_DSYNC ) - dio->flags |=IOMAP_DIO_WRITE_FUA ; +/* +* For datasync only writes, we optimistically try +* using FUA for this IO. Any non-FUA write that +* occurs will clear this flag, hence we know before +* completion whether a cache flush is necessary. +*/ +if (!(iocb->ki_flags &IOCB_SYNC )) + dio->flags |=IOMAP_DIO_WRITE_FUA ; + } }if (dio_flags &IOMAP_DIO_OVERWRITE_ONLY ) {
include/linux/iomap.h +6 -0
@@ -353,6 +353,12 @@ struct iomap_dio_ops {*/ #define IOMAP_DIO_PARTIAL (1 << 2) +/* +* The caller will sync the write if needed; do not sync it within +* iomap_dio_rw. Overrides IOMAP_DIO_FORCE_WAIT. +*/ +#define IOMAP_DIO_NOSYNC (1 << 3) +ssize_t iomap_dio_rw (struct kiocb *iocb,struct iov_iter *iter,const struct iomap_ops *ops,const struct iomap_dio_ops *dops,unsigned int dio_flags,void *private,size_t done_before);
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 4/44 btrfs: use IOMAP_DIO_NOSYNC
In reply to Al Viro... instead of messing with iocb flags
--- fs/btrfs/file.c | 17 ----------------- fs/btrfs/inode.c | 3 ++- 2 files changed, 2 insertions(+), 18 deletions(-)
fs/btrfs/file.c +0 -17
@@ -1848,7 +1848,6 @@ static ssize_t check_direct_IO(struct btrfs_fs_info *fs_info,static ssize_t btrfs_direct_write (struct kiocb *iocb,struct iov_iter *from) { -const bool is_sync_write = (iocb->ki_flags &IOCB_DSYNC );struct file *file = iocb->ki_filp;struct inode *inode =file_inode (file);struct btrfs_fs_info *fs_info =btrfs_sb (inode->i_sb);
@@ -1901,15 +1900,6 @@ static ssize_t btrfs_direct_write(struct kiocb *iocb, struct iov_iter *from) goto buffered; } -/* -* We remove IOCB_DSYNC so that we don't deadlock when iomap_dio_rw() -* calls generic_write_sync() (through iomap_dio_complete()), because -* that results in calling fsync (btrfs_sync_file()) which will try to -* lock the inode in exclusive/write mode. -*/ -if (is_sync_write) - iocb->ki_flags &= ~IOCB_DSYNC ; -/* * The iov_iter can be mapped to the same file range we are writing to. * If that's the case, then we will deadlock in the iomap code, because
@@ -1964,13 +1954,6 @@ static ssize_t btrfs_direct_write(struct kiocb *iocb, struct iov_iter *from)btrfs_inode_unlock (inode, ilock_flags); -/* -* Add back IOCB_DSYNC. Our caller, btrfs_file_write_iter(), will do -* the fsync (call generic_write_sync()). -*/ -if (is_sync_write) - iocb->ki_flags |=IOCB_DSYNC ; -/* If 'err' is -ENOTBLK then it means we must fallback to buffered IO. */ if ((err <0 && err != -ENOTBLK ) || !iov_iter_count (from)) goto out;
fs/btrfs/inode.c +2 -1
@@ -8152,7 +8152,8 @@ ssize_t btrfs_dio_rw(struct kiocb *iocb, struct iov_iter *iter, size_t done_befostruct btrfs_dio_data data;return iomap_dio_rw (iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops, -IOMAP_DIO_PARTIAL , &data, done_before); +IOMAP_DIO_PARTIAL |IOMAP_DIO_NOSYNC , + &data, done_before); } static int btrfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 2/44 No need of likely/unlikely on calls of check_copy_size()
In reply to Al Viroit's inline and unlikely() inside of it (including the implicit one in WARN_ON_ONCE()) suffice to convince the compiler that getting false from check_copy_size() is unlikely.
--- arch/powerpc/include/asm/uaccess.h | 2 +- arch/s390/include/asm/uaccess.h | 4 ++-- include/linux/uaccess.h | 4 ++-- include/linux/uio.h | 15 ++++++--------- 4 files changed, 11 insertions(+), 14 deletions(-)
arch/powerpc/include/asm/uaccess.h +1 -1
@@ -348,7 +348,7 @@ copy_mc_to_kernel(void *to, const void *from, unsigned long size)static inline unsigned long __must_check copy_mc_to_user (void __user *to,const void *from,unsigned long n) { -if (likely (check_copy_size (from, n, true))) { +if (check_copy_size (from, n, true)) {if (access_ok (to, n)) {allow_write_to_user (to, n); n =copy_mc_generic ((void *)to, from, n);
arch/s390/include/asm/uaccess.h +2 -2
@@ -39,7 +39,7 @@ _copy_from_user_key(void *to, const void __user *from, unsigned long n, unsignedstatic __always_inline unsigned long __must_check copy_from_user_key (void *to,const void __user *from,unsigned long n,unsigned long key) { -if (likely (check_copy_size (to, n, false))) +if (check_copy_size (to, n, false)) n =_copy_from_user_key (to, from, n, key);return n; }
@@ -50,7 +50,7 @@ _copy_to_user_key(void __user *to, const void *from, unsigned long n, unsigned lstatic __always_inline unsigned long __must_check copy_to_user_key (void __user *to,const void *from,unsigned long n,unsigned long key) { -if (likely (check_copy_size (from, n, true))) +if (check_copy_size (from, n, true)) n =_copy_to_user_key (to, from, n, key);return n; }
include/linux/uaccess.h +2 -2
@@ -148,7 +148,7 @@ _copy_to_user(void __user *, const void *, unsigned long);static __always_inline unsigned long __must_check copy_from_user (void *to,const void __user *from,unsigned long n) { -if (likely (check_copy_size (to, n, false))) +if (check_copy_size (to, n, false)) n =_copy_from_user (to, from, n);return n; }
@@ -156,7 +156,7 @@ copy_from_user(void *to, const void __user *from, unsigned long n)static __always_inline unsigned long __must_check copy_to_user (void __user *to,const void *from,unsigned long n) { -if (likely (check_copy_size (from, n, true))) +if (check_copy_size (from, n, true)) n =_copy_to_user (to, from, n);return n; }
include/linux/uio.h +6 -9
@@ -156,19 +156,17 @@ static inline size_t copy_folio_to_iter(struct folio *folio, size_t offset,static __always_inline __must_check size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { -if (unlikely (!check_copy_size (addr, bytes, true))) -return 0 ; -else +if (check_copy_size (addr, bytes, true))return _copy_to_iter (addr, bytes, i); +return 0 ; }static __always_inline __must_check size_t copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) { -if (unlikely (!check_copy_size (addr, bytes, false))) -return 0 ; -else +if (check_copy_size (addr, bytes, false))return _copy_from_iter (addr, bytes, i); +return 0 ; } static __always_inline __must_check
@@ -184,10 +182,9 @@ bool copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i)static __always_inline __must_check size_t copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) { -if (unlikely (!check_copy_size (addr, bytes, false))) -return 0 ; -else +if (check_copy_size (addr, bytes, false))return _copy_from_iter_nocache (addr, bytes, i); +return 0 ; } static __always_inline __must_check
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 5/44 struct file: use anonymous union member for rcuhead and llist
In reply to Al ViroOnce upon a time we couldn't afford anon unions; these days minimal gcc version had been raised enough to take care of that.
--- fs/file_table.c | 16 ++++++++-------- include/linux/fs.h | 6 +++--- 2 files changed, 11 insertions(+), 11 deletions(-)
fs/file_table.c +8 -8
@@ -45,7 +45,7 @@ static struct percpu_counter nr_files __cacheline_aligned_in_smp;static void file_free_rcu (struct rcu_head *head) { - struct file *f = container_of(head, struct file, f_u.fu_rcuhead); + struct file *f = container_of(head, struct file, f_rcuhead);put_cred (f->f_cred);kmem_cache_free (filp_cachep, f);
@@ -56,7 +56,7 @@ static inline void file_free(struct file *f)security_file_free (f);if (!(f->f_mode &FMODE_NOACCOUNT ))percpu_counter_dec (&nr_files); -call_rcu (&f->f_u.fu_rcuhead, file_free_rcu); +call_rcu (&f->f_rcuhead, file_free_rcu); }/*
@@ -142,7 +142,7 @@ static struct file *__alloc_file(int flags, const struct cred *cred) f->f_cred =get_cred (cred); error =security_file_alloc (f);if (unlikely (error)) { -file_free_rcu (&f->f_u.fu_rcuhead); +file_free_rcu (&f->f_rcuhead);return ERR_PTR (error); }
@@ -341,13 +341,13 @@ static void delayed_fput(struct work_struct *unused)struct llist_node *node =llist_del_all (&delayed_fput_list);struct file *f, *t; -llist_for_each_entry_safe (f, t, node, f_u.fu_llist) +llist_for_each_entry_safe (f, t, node, f_llist)__fput (f); }static void ____fput (struct callback_head *work) { - __fput(container_of(work, struct file, f_u.fu_rcuhead)); +__fput (container_of(work ,struct file ,f_rcuhead )); }/*
@@ -374,8 +374,8 @@ void fput(struct file *file)struct task_struct *task = current;if (likely (!in_interrupt () && !(task->flags &PF_KTHREAD ))) { -init_task_work (&file->f_u.fu_rcuhead, ____fput); -if (!task_work_add (task, &file->f_u.fu_rcuhead,TWA_RESUME )) +init_task_work (&file->f_rcuhead, ____fput); +if (!task_work_add (task, &file->f_rcuhead,TWA_RESUME ))return ;/* * After this task has run exit_task_work(),
@@ -384,7 +384,7 @@ void fput(struct file *file)*/ } -if (llist_add (&file->f_u.fu_llist, &delayed_fput_list)) +if (llist_add (&file->f_llist, &delayed_fput_list))schedule_delayed_work (&delayed_fput_work,1 ); } }
include/linux/fs.h +3 -3
@@ -924,9 +924,9 @@ static inline int ra_has_index(struct file_ra_state *ra, pgoff_t index)struct file {union { -struct llist_node fu_llist; -struct rcu_head fu_rcuhead; - } f_u; +struct llist_node f_llist; +struct rcu_head f_rcuhead; + };struct path f_path;struct inode *f_inode;/* cached value */ const struct file_operations *f_op;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 1/44 9p: handling Rerror without copy_from_iter_full()
In reply to Al Viro p9_client_zc_rpc()/p9_check_zc_errors() are playing fast
and loose with copy_from_iter_full().
Reading from file is done by sending Tread request. Response
consists of fixed-sized header (including the amount of data actually read) followed by the data itself.
For zero-copy case we arrange the things so that the first
11 bytes of reply go into the fixed-sized buffer, with the rest going straight into the pages we want to read into.
What makes the things inconvenient is that sglist describing
what should go where has to be set *before* the reply arrives. As the result, if reply is an error, the things get interesting. On success we get
size[4] Rread tag[2] count[4] data[count] For error layout varies depending upon the protocol variant - in original 9P and 9P2000 it's size[4] Rerror tag[2] len[2] error[len] in 9P2000.U size[4] Rerror tag[2] len[2] error[len] errno[4] in 9P2000.L size[4] Rlerror tag[2] errno[4] The last case is nice and simple - we have an 11-byte response
that fits into the fixed-sized buffer we hoped to get an Rread into. In other two, though, we get a variable-length string spill into the pages we'd prepared for the data to be read.
Had that been in fixed-sized buffer (which is actually 4K),
we would've dealt with that the same way we handle non-zerocopy case. However, for zerocopy it doesn't end up there, so we need to copy it from those pages.
The trouble is, by the time we get around to that, the
references to pages in question are already dropped. As the result, p9_zc_check_errors() tries to get the data using copy_from_iter_full(). Unfortunately, the iov_iter it's trying to read from might *NOT* be capable of that. It is, after all, a data destination, not data source. In particular, if it's an ITER_PIPE one, copy_from_iter_full() will simply fail.
In ->zc_request() itself we do have those pages and dealing with
the problem in there would be a simple matter of memcpy_from_page() into the fixed-sized buffer. Moreover, it isn't hard to recognize the (rare) case when such copying is needed. That way we get rid of p9_zc_check_errors() entirely - p9_check_errors() can be used instead both for zero-copy and non-zero-copy cases.
--- net/9p/client.c | 86 +------------------------------------------ net/9p/trans_virtio.c | 34 +++++++++++++++++ 2 files changed, 35 insertions(+), 85 deletions(-)
net/9p/client.c +1 -85
@@ -550,90 +550,6 @@ static int p9_check_errors(struct p9_client *c, struct p9_req_t *req)return err; } -/** -* p9_check_zc_errors - check 9p packet for error return and process it -* @c: current client instance -* @req: request to parse and check for error conditions -* @uidata: external buffer containing error -* @in_hdrlen: Size of response protocol buffer. -* -* returns error code if one is discovered, otherwise returns 0 -* -* this will have to be more complicated if we have multiple -* error packet types -*/ - -static int p9_check_zc_errors (struct p9_client *c,struct p9_req_t *req, -struct iov_iter *uidata,int in_hdrlen) -{ -int err; -int ecode; -s8 type; -char *ename =NULL ; - - err =p9_parse_header (&req->rc,NULL , &type,NULL ,0 ); -/* dump the response from server -* This should be after parse_header which poplulate pdu_fcall. -*/ -trace_9p_protocol_dump (c, &req->rc); -if (err) { -p9_debug (P9_DEBUG_ERROR ,"couldn't parse header %d\n" , err); -return err; - } - -if (type !=P9_RERROR && type !=P9_RLERROR ) -return 0 ; - -if (!p9_is_proto_dotl (c)) { -/* Error is reported in string format */ -int len; -/* 7 = header size for RERROR; */ -int inline_len = in_hdrlen -7 ; - - len = req->rc.size - req->rc.offset; -if (len > (P9_ZC_HDR_SZ -7 )) { - err = -EFAULT ; - goto out_err; - } - - ename = &req->rc.sdata[req->rc.offset]; -if (len > inline_len) { -/* We have error in external buffer */ -if (!copy_from_iter_full (ename + inline_len, - len - inline_len, uidata)) { - err = -EFAULT ; - goto out_err; - } - } - ename =NULL ; - err =p9pdu_readf (&req->rc, c->proto_version,"s?d" , - &ename, &ecode); -if (err) - goto out_err; - -if (p9_is_proto_dotu (c) && ecode <512 ) - err = -ecode; - -if (!err) { - err =p9_errstr2errno (ename,strlen (ename)); - -p9_debug (P9_DEBUG_9P ,"<<< RERROR (%d) %s\n" , - -ecode, ename); - } -kfree (ename); - }else { - err =p9pdu_readf (&req->rc, c->proto_version,"d" , &ecode); - err = -ecode; - -p9_debug (P9_DEBUG_9P ,"<<< RLERROR (%d)\n" , -ecode); - } -return err; - -out_err: -p9_debug (P9_DEBUG_ERROR ,"couldn't parse error%d\n" , err); -return err; -} -static struct p9_req_t *p9_client_rpc (struct p9_client *c,int8_t type,const char *fmt, ...);
@@ -874,7 +790,7 @@ static struct p9_req_t *p9_client_zc_rpc(struct p9_client *c, int8_t type,if (err <0 ) goto reterr; - err =p9_check_zc_errors (c, req, uidata, in_hdrlen); + err =p9_check_errors (c, req);trace_9p_client_res (c, type, req->rc.tag, err);if (!err)return req;
net/9p/trans_virtio.c +34 -0
@@ -377,6 +377,35 @@ static int p9_get_mapped_pages(struct virtio_chan *chan, } } +static void handle_rerror (struct p9_req_t *req,int in_hdr_len, +size_t offs,struct page **pages) +{ +unsigned size, n; +void *to = req->rc.sdata + in_hdr_len; + +// Fits entirely into the static data? Nothing to do. +if (req->rc.size < in_hdr_len) +return ; + +// Really long error message? Tough, truncate the reply. Might get +// rejected (we can't be arsed to adjust the size encoded in header, +// or string size for that matter), but it wouldn't be anything valid +// anyway. +if (unlikely (req->rc.size >P9_ZC_HDR_SZ )) + req->rc.size =P9_ZC_HDR_SZ ; + +// data won't span more than two pages + size = req->rc.size - in_hdr_len; + n =PAGE_SIZE - offs; +if (size > n) { +memcpy_from_page (to, *pages++, offs, n); + offs =0 ; + to += n; + size -= n; + } +memcpy_from_page (to, *pages, offs, size); +} +/** * p9_virtio_zc_request - issue a zero copy request * @client: client instance issuing the request
@@ -503,6 +532,11 @@ p9_virtio_zc_request(struct p9_client *client, struct p9_req_t *req, kicked =1 ;p9_debug (P9_DEBUG_TRANS ,"virtio request kicked\n" ); err =wait_event_killable (req->wq, req->status >=REQ_STATUS_RCVD ); +// RERROR needs reply (== error string) in static data +if (req->status ==REQ_STATUS_RCVD && +unlikely (req->rc.sdata[4 ] ==P9_RERROR )) +handle_rerror (req, in_hdr_len, offs, in_pages); +/* * Non kernel buffers are pinned, unpin them */
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 6/44 iocb: delay evaluation of IS_SYNC(...) until we want to check IOCB_DSYNC
New helper to be used instead of direct checks for IOCB_DSYNC:iocb_is_dsync(iocb). Checks converted, which allows to avoid the IS_SYNC(iocb->ki_filp->f_mapping->host) part (4 cache lines) from iocb_flags() - it's checked in iocb_is_dsync() instead
--- block/fops.c | 2 +- fs/btrfs/file.c | 2 +- fs/direct-io.c | 2 +- fs/fuse/file.c | 2 +- fs/iomap/direct-io.c | 3 +-- fs/zonefs/super.c | 2 +- include/linux/fs.h | 10 ++++++++-- 7 files changed, 14 insertions(+), 9 deletions(-)
block/fops.c +1 -1
@@ -37,7 +37,7 @@ static unsigned int dio_bio_write_op(struct kiocb *iocb)unsigned int op =REQ_OP_WRITE |REQ_SYNC |REQ_IDLE ;/* avoid the need for a I/O completion work item */ -if (iocb->ki_flags &IOCB_DSYNC ) +if (iocb_is_dsync (iocb)) op |=REQ_FUA ;return op; }
fs/btrfs/file.c +1 -1
@@ -2021,7 +2021,7 @@ ssize_t btrfs_do_write_iter(struct kiocb *iocb, struct iov_iter *from,struct file *file = iocb->ki_filp;struct btrfs_inode *inode =BTRFS_I (file_inode (file));ssize_t num_written, num_sync; -const bool sync = iocb->ki_flags &IOCB_DSYNC ; +const bool sync =iocb_is_dsync (iocb);/* * If the fs flips readonly due to some impossible error, although we
fs/direct-io.c +1 -1
@@ -1210,7 +1210,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode,*/ if (dio->is_async &&iov_iter_rw (iter) ==WRITE ) { retval =0 ; -if (iocb->ki_flags &IOCB_DSYNC ) +if (iocb_is_dsync (iocb)) retval =dio_set_defer_completion (dio);else if (!dio->inode->i_sb->s_dio_done_wq) {/*
fs/fuse/file.c +1 -1
@@ -1042,7 +1042,7 @@ static unsigned int fuse_write_flags(struct kiocb *iocb) {unsigned int flags = iocb->ki_filp->f_flags; -if (iocb->ki_flags &IOCB_DSYNC ) +if (iocb_is_dsync (iocb)) flags |=O_DSYNC ;if (iocb->ki_flags &IOCB_SYNC ) flags |=O_SYNC ;
fs/iomap/direct-io.c +1 -2
@@ -548,8 +548,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, }/* for data sync or sync, we need sync completion processing */ -if (iocb->ki_flags &IOCB_DSYNC && - !(dio_flags &IOMAP_DIO_NOSYNC )) { +if (iocb_is_dsync (iocb) && !(dio_flags &IOMAP_DIO_NOSYNC )) { dio->flags |=IOMAP_DIO_NEED_SYNC ;/*
fs/zonefs/super.c +1 -1
@@ -746,7 +746,7 @@ static ssize_t zonefs_file_dio_append(struct kiocb *iocb, struct iov_iter *from) REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE, GFP_NOFS); bio->bi_iter.bi_sector = zi->i_zsector; bio->bi_ioprio = iocb->ki_ioprio; -if (iocb->ki_flags &IOCB_DSYNC ) +if (iocb_is_dsync (iocb)) bio->bi_opf |=REQ_FUA ; ret =bio_iov_iter_get_pages (bio, from);
include/linux/fs.h +8 -2
@@ -2720,6 +2720,12 @@ extern int vfs_fsync(struct file *file, int datasync);extern int sync_file_range (struct file *file,loff_t offset,loff_t nbytes,unsigned int flags); +static inline bool iocb_is_dsync (const struct kiocb *iocb) +{ +return (iocb->ki_flags &IOCB_DSYNC ) || +IS_SYNC (iocb->ki_filp->f_mapping->host); +} +/* * Sync the bytes written if this was a synchronous write. Expect ki_pos * to already be updated for the write, and will return either the amount
@@ -2727,7 +2733,7 @@ extern int sync_file_range(struct file *file, loff_t offset, loff_t nbytes,*/ static inline ssize_t generic_write_sync (struct kiocb *iocb,ssize_t count) { -if (iocb->ki_flags &IOCB_DSYNC ) { +if (iocb_is_dsync (iocb)) {int ret =vfs_fsync_range (iocb->ki_filp, iocb->ki_pos - count, iocb->ki_pos -1 , (iocb->ki_flags &IOCB_SYNC ) ?0 :1 );
@@ -3262,7 +3268,7 @@ static inline int iocb_flags(struct file *file) res |=IOCB_APPEND ;if (file->f_flags &O_DIRECT ) res |=IOCB_DIRECT ; -if ((file->f_flags &O_DSYNC ) ||IS_SYNC (file->f_mapping->host)) +if (file->f_flags &O_DSYNC ) res |=IOCB_DSYNC ;if (file->f_flags & __O_SYNC) res |=IOCB_SYNC ;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 7/44 keep iocb_flags() result cached in struct file
In reply to Al Viro* calculate at the time we set FMODE_OPENED (do_dentry_open() for normal opens, alloc_file() for pipe()/socket()/etc.)
* update when handling F_SETFL
* keep in a new field - file->f_iocb_flags; since that thing is needed only before the refcount reaches zero, we can put it into the same anon union where ->f_rcuhead and ->f_llist live - those are used only after refcount reaches zero.
--- drivers/nvme/target/io-cmd-file.c | 2 +- fs/aio.c | 2 +- fs/fcntl.c | 1 + fs/file_table.c | 1 + fs/io_uring.c | 2 +- fs/open.c | 1 + include/linux/fs.h | 5 ++--- 7 files changed, 8 insertions(+), 6 deletions(-)
drivers/nvme/target/io-cmd-file.c +1 -1
@@ -112,7 +112,7 @@ static ssize_t nvmet_file_submit_bvec(struct nvmet_req *req, loff_t pos, iocb->ki_pos = pos; iocb->ki_filp = req->ns->file; - iocb->ki_flags = ki_flags |iocb_flags (req->ns->file); + iocb->ki_flags = ki_flags | iocb->ki_filp->f_iocb_flags;return call_iter (iocb, &iter); }
fs/aio.c +1 -1
@@ -1475,7 +1475,7 @@ static int aio_prep_rw(struct kiocb *req, const struct iocb *iocb) req->ki_complete = aio_complete_rw; req->private =NULL ; req->ki_pos = iocb->aio_offset; - req->ki_flags =iocb_flags (req->ki_filp); + req->ki_flags = req->ki_filp->f_iocb_flags;if (iocb->aio_flags &IOCB_FLAG_RESFD ) req->ki_flags |=IOCB_EVENTFD ;if (iocb->aio_flags &IOCB_FLAG_IOPRIO ) {
fs/fcntl.c +1 -0
@@ -78,6 +78,7 @@ static int setfl(int fd, struct file * filp, unsigned long arg) }spin_lock (&filp->f_lock); filp->f_flags = (arg &SETFL_MASK ) | (filp->f_flags & ~SETFL_MASK ); + filp->f_iocb_flags =iocb_flags (filp);spin_unlock (&filp->f_lock); out:
fs/file_table.c +1 -0
@@ -241,6 +241,7 @@ static struct file *alloc_file(const struct path *path, int flags,if ((file->f_mode &FMODE_WRITE ) &&likely (fop->write || fop->write_iter)) file->f_mode |=FMODE_CAN_WRITE ; + file->f_iocb_flags =iocb_flags (file); file->f_mode |=FMODE_OPENED ; file->f_op = fop;if ((file->f_mode & (FMODE_READ |FMODE_WRITE )) ==FMODE_READ )
fs/io_uring.c +1 -1
@@ -4330,7 +4330,7 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode)if (!io_req_ffs_set (req)) req->flags |=io_file_get_flags (file) <<REQ_F_SUPPORT_NOWAIT_BIT ; - kiocb->ki_flags =iocb_flags (file); + kiocb->ki_flags = file->f_iocb_flags; ret =kiocb_set_rw_flags (kiocb, req->rw.flags);if (unlikely (ret))return ret;
fs/open.c +1 -0
@@ -862,6 +862,7 @@ static int do_dentry_open(struct file *f, f->f_mode |=FMODE_CAN_ODIRECT ; f->f_flags &= ~(O_CREAT |O_EXCL |O_NOCTTY |O_TRUNC ); + f->f_iocb_flags =iocb_flags (f);file_ra_state_init (&f->f_ra, f->f_mapping->host->i_mapping);
include/linux/fs.h +2 -3
@@ -926,6 +926,7 @@ struct file {union {struct llist_node f_llist;struct rcu_head f_rcuhead; +unsigned int f_iocb_flags; };struct path f_path;struct inode *f_inode;/* cached value */
@@ -2199,13 +2200,11 @@ static inline bool HAS_UNMAPPED_ID(struct user_namespace *mnt_userns, !gid_valid (i_gid_into_mnt (mnt_userns, inode)); } -static inline int iocb_flags (struct file *file); -static inline void init_sync_kiocb (struct kiocb *kiocb,struct file *filp) { *kiocb = (struct kiocb ) { .ki_filp = filp, - .ki_flags =iocb_flags (filp), + .ki_flags = filp->f_iocb_flags, .ki_ioprio =get_current_ioprio (), }; }
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 10/44 switch new_sync_{read,write}() to ITER_UBUF
In reply to Al Viro--- fs/read_write.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
fs/read_write.c +2 -4
@@ -389,14 +389,13 @@ EXPORT_SYMBOL(rw_verify_area);static ssize_t new_sync_read (struct file *filp,char __user *buf,size_t len,loff_t *ppos) { -struct iovec iov = { .iov_base = buf, .iov_len = len };struct kiocb kiocb;struct iov_iter iter;ssize_t ret;init_sync_kiocb (&kiocb, filp); kiocb.ki_pos = (ppos ? *ppos :0 ); -iov_iter_init (&iter,READ , &iov,1 , len); +iov_iter_ubuf (&iter,READ , buf, len); ret =call_read_iter (filp, &kiocb, &iter);BUG_ON (ret == -EIOCBQUEUED );
@@ -492,14 +491,13 @@ ssize_t vfs_read(struct file *file, char __user *buf, size_t count, loff_t *pos)static ssize_t new_sync_write (struct file *filp,const char __user *buf,size_t len,loff_t *ppos) { -struct iovec iov = { .iov_base = (void __user *)buf, .iov_len = len };struct kiocb kiocb;struct iov_iter iter;ssize_t ret;init_sync_kiocb (&kiocb, filp); kiocb.ki_pos = (ppos ? *ppos :0 ); -iov_iter_init (&iter,WRITE , &iov,1 , len); +iov_iter_ubuf (&iter,WRITE , (void __user *)buf, len); ret =call_write_iter (filp, &kiocb, &iter);BUG_ON (ret == -EIOCBQUEUED );
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 8/44 copy_page_{to,from}_iter(): switch iovec variants to generic
In reply to Al Virowe can do copyin/copyout under kmap_local_page(); it shouldn't overflow the kmap stack - the maximal footprint increase only by one here.
--- lib/iov_iter.c | 191 ++----------------------------------------------- 1 file changed, 4 insertions(+), 187 deletions(-)
lib/iov_iter.c +4 -187
@@ -168,174 +168,6 @@ static int copyin(void *to, const void __user *from, size_t n)return n; } -static size_t copy_page_to_iter_iovec (struct page *page,size_t offset,size_t bytes, -struct iov_iter *i) -{ -size_t skip, copy, left, wanted; -const struct iovec *iov; -char __user *buf; -void *kaddr, *from; - -if (unlikely (bytes > i->count)) - bytes = i->count; - -if (unlikely (!bytes)) -return 0 ; - -might_fault (); - wanted = bytes; - iov = i->iov; - skip = i->iov_offset; - buf = iov->iov_base + skip; - copy =min (bytes, iov->iov_len - skip); - -if (IS_ENABLED (CONFIG_HIGHMEM ) && !fault_in_writeable (buf, copy)) { - kaddr =kmap_atomic (page); - from = kaddr + offset; - -/* first chunk, usually the only one */ - left =copyout (buf, from, copy); - copy -= left; - skip += copy; - from += copy; - bytes -= copy; - -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyout (buf, from, copy); - copy -= left; - skip = copy; - from += copy; - bytes -= copy; - } -if (likely (!bytes)) { -kunmap_atomic (kaddr); - goto done; - } - offset = from - kaddr; - buf += copy; -kunmap_atomic (kaddr); - copy =min (bytes, iov->iov_len - skip); - } -/* Too bad - revert to non-atomic kmap */ - - kaddr =kmap (page); - from = kaddr + offset; - left =copyout (buf, from, copy); - copy -= left; - skip += copy; - from += copy; - bytes -= copy; -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyout (buf, from, copy); - copy -= left; - skip = copy; - from += copy; - bytes -= copy; - } -kunmap (page); - -done: -if (skip == iov->iov_len) { - iov++; - skip =0 ; - } - i->count -= wanted - bytes; - i->nr_segs -= iov - i->iov; - i->iov = iov; - i->iov_offset = skip; -return wanted - bytes; -} - -static size_t copy_page_from_iter_iovec (struct page *page,size_t offset,size_t bytes, -struct iov_iter *i) -{ -size_t skip, copy, left, wanted; -const struct iovec *iov; -char __user *buf; -void *kaddr, *to; - -if (unlikely (bytes > i->count)) - bytes = i->count; - -if (unlikely (!bytes)) -return 0 ; - -might_fault (); - wanted = bytes; - iov = i->iov; - skip = i->iov_offset; - buf = iov->iov_base + skip; - copy =min (bytes, iov->iov_len - skip); - -if (IS_ENABLED (CONFIG_HIGHMEM ) && !fault_in_readable (buf, copy)) { - kaddr =kmap_atomic (page); - to = kaddr + offset; - -/* first chunk, usually the only one */ - left =copyin (to, buf, copy); - copy -= left; - skip += copy; - to += copy; - bytes -= copy; - -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyin (to, buf, copy); - copy -= left; - skip = copy; - to += copy; - bytes -= copy; - } -if (likely (!bytes)) { -kunmap_atomic (kaddr); - goto done; - } - offset = to - kaddr; - buf += copy; -kunmap_atomic (kaddr); - copy =min (bytes, iov->iov_len - skip); - } -/* Too bad - revert to non-atomic kmap */ - - kaddr =kmap (page); - to = kaddr + offset; - left =copyin (to, buf, copy); - copy -= left; - skip += copy; - to += copy; - bytes -= copy; -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyin (to, buf, copy); - copy -= left; - skip = copy; - to += copy; - bytes -= copy; - } -kunmap (page); - -done: -if (skip == iov->iov_len) { - iov++; - skip =0 ; - } - i->count -= wanted - bytes; - i->nr_segs -= iov - i->iov; - i->iov = iov; - i->iov_offset = skip; -return wanted - bytes; -} -#ifdef PIPE_PARANOIA static bool sanity (const struct iov_iter *i) {
@@ -848,24 +680,14 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n)static size_t __copy_page_to_iter (struct page *page,size_t offset,size_t bytes,struct iov_iter *i) { -if (likely (iter_is_iovec (i))) -return copy_page_to_iter_iovec (page, offset, bytes, i); -if (iov_iter_is_bvec (i) ||iov_iter_is_kvec (i) ||iov_iter_is_xarray (i)) { +if (unlikely (iov_iter_is_pipe (i))) { +return copy_page_to_iter_pipe (page, offset, bytes, i); + }else {void *kaddr =kmap_local_page (page);size_t wanted =_copy_to_iter (kaddr + offset, bytes, i);kunmap_local (kaddr);return wanted; } -if (iov_iter_is_pipe (i)) -return copy_page_to_iter_pipe (page, offset, bytes, i); -if (unlikely (iov_iter_is_discard (i))) { -if (unlikely (i->count < bytes)) - bytes = i->count; - i->count -= bytes; -return bytes; - } -WARN_ON (1 ); -return 0 ; } size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes,
@@ -896,17 +718,12 @@ EXPORT_SYMBOL(copy_page_to_iter);size_t copy_page_from_iter (struct page *page,size_t offset,size_t bytes,struct iov_iter *i) { -if (unlikely (!page_copy_sane (page, offset, bytes))) -return 0 ; -if (likely (iter_is_iovec (i))) -return copy_page_from_iter_iovec (page, offset, bytes, i); -if (iov_iter_is_bvec (i) ||iov_iter_is_kvec (i) ||iov_iter_is_xarray (i)) { +if (page_copy_sane (page, offset, bytes)) {void *kaddr =kmap_local_page (page);size_t wanted =_copy_from_iter (kaddr + offset, bytes, i);kunmap_local (kaddr);return wanted; } -WARN_ON (1 );return 0 ; }EXPORT_SYMBOL (copy_page_from_iter);
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 11/44 iov_iter_bvec_advance(): don't bother with bvec_iter
In reply to Al Virodo what we do for iovec/kvec; that ends up generating better code, AFAICS.
--- lib/iov_iter.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
lib/iov_iter.c +14 -9
@@ -870,17 +870,22 @@ static void pipe_advance(struct iov_iter *i, size_t size)static void iov_iter_bvec_advance (struct iov_iter *i,size_t size) { -struct bvec_iter bi; +const struct bio_vec *bvec, *end; - bi.bi_size = i->count; - bi.bi_bvec_done = i->iov_offset; - bi.bi_idx =0 ; -bvec_iter_advance (i->bvec, &bi, size); +if (!i->count) +return ; + i->count -= size; + + size += i->iov_offset; - i->bvec += bi.bi_idx; - i->nr_segs -= bi.bi_idx; - i->count = bi.bi_size; - i->iov_offset = bi.bi_bvec_done; +for (bvec = i->bvec, end = bvec + i->nr_segs; bvec < end; bvec++) { +if (likely (size < bvec->bv_len)) +break ; + size -= bvec->bv_len; + } + i->iov_offset = size; + i->nr_segs -= bvec - i->bvec; + i->bvec = bvec; }static void iov_iter_iovec_advance (struct iov_iter *i,size_t size)
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 12/44 fix short copy handling in copy_mc_pipe_to_iter()
In reply to Al ViroUnlike other copying operations on ITER_PIPE, copy_mc_to_iter() can result in a short copy. In that case we need to trim the unused buffers, as well as the length of partially filled one - it's not enough to set ->head, ->iov_offset and ->count to reflect how much had we copied. Not hard to fix, fortunately...
I'd put a helper (pipe_discard_from(pipe, head)) into pipe_fs_i.h, rather than iov_iter.c - it has nothing to do with iov_iter and having it will allow us to avoid an ugly kludge in fs/splice.c. We could put it into lib/iov_iter.c for now and move it later, but I don't see the point going that way...
--- include/linux/pipe_fs_i.h | 9 +++++++++ lib/iov_iter.c | 15 +++++++++++---- 2 files changed, 20 insertions(+), 4 deletions(-)
include/linux/pipe_fs_i.h +9 -0
@@ -229,6 +229,15 @@ static inline bool pipe_buf_try_steal(struct pipe_inode_info *pipe,return buf->ops->try_steal (pipe, buf); } +static inline void pipe_discard_from (struct pipe_inode_info *pipe, +unsigned int old_head) +{ +unsigned int mask = pipe->ring_size -1 ; + +while (pipe->head > old_head) +pipe_buf_release (pipe, &pipe->bufs[--pipe->head & mask]); +} +/* Differs from PIPE_BUF in that PIPE_SIZE is the length of the actual memory allocation, whereas PIPE_BUF makes atomicity guarantees. */ #define PIPE_SIZE PAGE_SIZE
lib/iov_iter.c +11 -4
@@ -689,6 +689,7 @@ static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes,struct pipe_inode_info *pipe = i->pipe;unsigned int p_mask = pipe->ring_size -1 ;unsigned int i_head; +unsigned int valid = pipe->head;size_t n, off, xfer =0 ;if (!sanity (i))
@@ -702,11 +703,17 @@ static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, rem =copy_mc_to_kernel (p + off, addr + xfer, chunk); chunk -= rem;kunmap_local (p); - i->head = i_head; - i->iov_offset = off + chunk; - xfer += chunk; -if (rem) +if (chunk) { + i->head = i_head; + i->iov_offset = off + chunk; + xfer += chunk; + valid = i_head +1 ; + } +if (rem) { + pipe->bufs[i_head & p_mask].len -= rem; +pipe_discard_from (pipe, valid);break ; + } n -= chunk; off =0 ; i_head++;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 14/44 ITER_PIPE: helper for getting pipe buffer by index
In reply to Al Viropipe_buffer instances of a pipe are organized as a ring buffer, with power-of-2 size. Indices are kept *not* reduced modulo ring size, so the buffer refered to by index N is
pipe->bufs[N & (pipe->ring_size - 1)].
Ring size can change over the lifetime of a pipe, but not while the pipe is locked. So for any iov_iter primitives it's a constant. Original conversion of pipes to this layout went overboard trying to microoptimize that - calculating pipe->ring_size - 1, storing it in a local variable and using through the function. In some cases it might be warranted, but most of the times it only obfuscates what's going on in there.
Introduce a helper (pipe_buf(pipe, N)) that would encapsulate that and use it in the obvious cases. More will follow...
--- lib/iov_iter.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
lib/iov_iter.c +9 -6
@@ -183,13 +183,18 @@ static int copyin(void *to, const void __user *from, size_t n)return n; } +static inline struct pipe_buffer *pipe_buf (const struct pipe_inode_info *pipe, +unsigned int slot) +{ +return &pipe->bufs[slot & (pipe->ring_size -1 )]; +} +#ifdef PIPE_PARANOIA static bool sanity (const struct iov_iter *i) {struct pipe_inode_info *pipe = i->pipe;unsigned int p_head = pipe->head;unsigned int p_tail = pipe->tail; -unsigned int p_mask = pipe->ring_size -1 ;unsigned int p_occupancy =pipe_occupancy (p_head, p_tail);unsigned int i_head = i->head;unsigned int idx;
@@ -201,7 +206,7 @@ static bool sanity(const struct iov_iter *i)if (unlikely (i_head != p_head -1 )) goto Bad;// must be at the last buffer... - p = &pipe->bufs[i_head & p_mask]; + p =pipe_buf (pipe, i_head);if (unlikely (p->offset + p->len != i->iov_offset)) goto Bad;// ... at the end of segment }else {
@@ -386,11 +391,10 @@ static inline bool allocated(struct pipe_buffer *buf)static inline void data_start (const struct iov_iter *i,unsigned int *iter_headp,size_t *offp) { -unsigned int p_mask = i->pipe->ring_size -1 ;unsigned int iter_head = i->head;size_t off = i->iov_offset; -if (off && (!allocated (&i->pipe->bufs[iter_head & p_mask]) || +if (off && (!allocated (pipe_buf (i->pipe, iter_head)) || off ==PAGE_SIZE )) { iter_head++; off =0 ;
@@ -1180,10 +1184,9 @@ unsigned long iov_iter_alignment(const struct iov_iter *i)return iov_iter_alignment_bvec (i);if (iov_iter_is_pipe (i)) { -unsigned int p_mask = i->pipe->ring_size -1 ;size_t size = i->count; -if (size && i->iov_offset &&allocated (&i->pipe->bufs[i->head & p_mask])) +if (size && i->iov_offset &&allocated (pipe_buf (i->pipe, i->head)))return size | i->iov_offset;return size; }
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 13/44 splice: stop abusing iov_iter_advance() to flush a pipe
In reply to Al ViroUse pipe_discard_from() explicitly in generic_file_read_iter(); don't bother with rather non-obvious use of iov_iter_advance() in there.
--- fs/splice.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
fs/splice.c +2 -5
@@ -301,11 +301,9 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, {struct iov_iter to;struct kiocb kiocb; -unsigned int i_head;int ret;iov_iter_pipe (&to,READ , pipe, len); - i_head = to.head;init_sync_kiocb (&kiocb, in); kiocb.ki_pos = *ppos; ret =call_read_iter (in, &kiocb, &to);
@@ -313,9 +311,8 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, *ppos = kiocb.ki_pos;file_accessed (in); }else if (ret <0 ) { - to.head = i_head; - to.iov_offset =0 ; -iov_iter_advance (&to,0 );/* to free what was emitted */ +/* free what was emitted */ +pipe_discard_from (pipe, to.start_head);/* * callers of ->splice_read() expect -EAGAIN on * "can't put anything in there", rather than -EFAULT.
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 15/44 ITER_PIPE: helpers for adding pipe buffers
In reply to Al ViroThere are only two kinds of pipe_buffer in the area used by ITER_PIPE.
1) anonymous - copy_to_iter() et.al. end up creating those and copying data there. They have zero ->offset, and their ->ops points to default_pipe_page_ops.
2) zero-copy ones - those come from copy_page_to_iter(), and page comes from caller. ->offset is also caller-supplied - it might be non-zero. ->ops points to page_cache_pipe_buf_ops.
Move creation and insertion of those into helpers - push_anon(pipe, size) and push_page(pipe, page, offset, size) resp., separating them from the "could we avoid creating a new buffer by merging with the current head?" logics.
--- lib/iov_iter.c | 88 ++++++++++++++++++++++++++------------------------ 1 file changed, 46 insertions(+), 42 deletions(-)
lib/iov_iter.c +46 -42
@@ -231,15 +231,39 @@ static bool sanity(const struct iov_iter *i)#define sanity (i) true#endif +static struct page *push_anon (struct pipe_inode_info *pipe,unsigned size) +{ +struct page *page =alloc_page (GFP_USER ); +if (page) { +struct pipe_buffer *buf =pipe_buf (pipe, pipe->head++); + *buf = (struct pipe_buffer ) { + .ops = &default_pipe_buf_ops, + .page = page, + .offset =0 , + .len = size + }; + } +return page; +} + +static void push_page (struct pipe_inode_info *pipe,struct page *page, +unsigned int offset,unsigned int size) +{ +struct pipe_buffer *buf =pipe_buf (pipe, pipe->head++); + *buf = (struct pipe_buffer ) { + .ops = &page_cache_pipe_buf_ops, + .page = page, + .offset = offset, + .len = size + }; +get_page (page); +} +static size_t copy_page_to_iter_pipe (struct page *page,size_t offset,size_t bytes,struct iov_iter *i) {struct pipe_inode_info *pipe = i->pipe; -struct pipe_buffer *buf; -unsigned int p_tail = pipe->tail; -unsigned int p_mask = pipe->ring_size -1 ; -unsigned int i_head = i->head; -size_t off; +unsigned int head = pipe->head;if (unlikely (bytes > i->count)) bytes = i->count;
@@ -250,32 +274,21 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t byif (!sanity (i))return 0 ; - off = i->iov_offset; - buf = &pipe->bufs[i_head & p_mask]; -if (off) { -if (offset == off && buf->page == page) { -/* merge with the last one */ +if (offset && i->iov_offset == offset) {// could we merge it? +struct pipe_buffer *buf =pipe_buf (pipe, head -1 ); +if (buf->page == page) { buf->len += bytes; i->iov_offset += bytes; - goto out; + i->count -= bytes; +return bytes; } - i_head++; - buf = &pipe->bufs[i_head & p_mask]; } -if (pipe_full (i_head, p_tail, pipe->max_usage)) +if (pipe_full (pipe->head, pipe->tail, pipe->max_usage))return 0 ; - buf->ops = &page_cache_pipe_buf_ops; - buf->flags =0 ; -get_page (page); - buf->page = page; - buf->offset = offset; - buf->len = bytes; - - pipe->head = i_head +1 ; +push_page (pipe, page, offset, bytes); i->iov_offset = offset + bytes; - i->head = i_head; -out: + i->head = head; i->count -= bytes;return bytes; }
@@ -407,8 +420,6 @@ static size_t push_pipe(struct iov_iter *i, size_t size, int *iter_headp, size_t *offp) { struct pipe_inode_info *pipe = i->pipe; -unsigned int p_tail = pipe->tail; -unsigned int p_mask = pipe->ring_size -1 ;unsigned int iter_head;size_t off;ssize_t left;
@@ -423,30 +434,23 @@ static size_t push_pipe(struct iov_iter *i, size_t size, *iter_headp = iter_head; *offp = off;if (off) { +struct pipe_buffer *buf =pipe_buf (pipe, iter_head); + left -=PAGE_SIZE - off;if (left <=0 ) { - pipe->bufs[iter_head & p_mask].len += size; + buf->len += size;return size; } - pipe->bufs[iter_head & p_mask].len =PAGE_SIZE ; - iter_head++; + buf->len =PAGE_SIZE ; } -while (!pipe_full (iter_head, p_tail, pipe->max_usage)) { -struct pipe_buffer *buf = &pipe->bufs[iter_head & p_mask]; -struct page *page =alloc_page (GFP_USER ); +while (!pipe_full (pipe->head, pipe->tail, pipe->max_usage)) { +struct page *page =push_anon (pipe, +min_t (ssize_t, left,PAGE_SIZE ));if (!page)break ; - buf->ops = &default_pipe_buf_ops; - buf->flags =0 ; - buf->page = page; - buf->offset =0 ; - buf->len =min_t (ssize_t, left,PAGE_SIZE ); - left -= buf->len; - iter_head++; - pipe->head = iter_head; - -if (left ==0 ) + left -=PAGE_SIZE ; +if (left <=0 )return size; }return size - left;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 9/44 new iov_iter flavour - ITER_UBUF
In reply to Al ViroEquivalent of single-segment iovec. Initialized by iov_iter_ubuf(), checked for by iter_is_ubuf(), otherwise behaves like ITER_IOVEC ones.
We are going to expose the things like ->write_iter() et.al. to those in subsequent commits.
New predicate (user_backed_iter()) that is true for ITER_IOVEC and ITER_UBUF; places like direct-IO handling should use that for checking that pages we modify after getting them from iov_iter_get_pages() would need to be dirtied.
DO NOT assume that replacing iter_is_iovec() with user_backed_iter() will solve all problems - there's code that uses iter_is_iovec() to decide how to poke around in iov_iter guts and for that the predicate replacement obviously won't suffice.
--- block/fops.c | 6 +-- fs/ceph/file.c | 2 +- fs/cifs/file.c | 2 +- fs/direct-io.c | 2 +- fs/fuse/dev.c | 4 +- fs/fuse/file.c | 2 +- fs/gfs2/file.c | 2 +- fs/iomap/direct-io.c | 2 +- fs/nfs/direct.c | 2 +- include/linux/uio.h | 26 ++++++++++++ lib/iov_iter.c | 94 ++++++++++++++++++++++++++++++++++---------- mm/shmem.c | 2 +- 12 files changed, 113 insertions(+), 33 deletions(-)
block/fops.c +3 -3
@@ -69,7 +69,7 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,if (iov_iter_rw (iter) ==READ ) {bio_init (&bio, bdev, vecs, nr_pages,REQ_OP_READ ); -if (iter_is_iovec (iter)) +if (user_backed_iter (iter)) should_dirty = true; }else {bio_init (&bio, bdev, vecs, nr_pages,dio_bio_write_op (iocb));
@@ -199,7 +199,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, } dio->size =0 ; -if (is_read &&iter_is_iovec (iter)) +if (is_read &&user_backed_iter (iter)) dio->flags |=DIO_SHOULD_DIRTY ;blk_start_plug (&plug);
@@ -331,7 +331,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, dio->size = bio->bi_iter.bi_size;if (is_read) { -if (iter_is_iovec (iter)) { +if (user_backed_iter (iter)) { dio->flags |=DIO_SHOULD_DIRTY ;bio_set_pages_dirty (bio); }
fs/ceph/file.c +1 -1
@@ -1262,7 +1262,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,size_t count =iov_iter_count (iter);loff_t pos = iocb->ki_pos;bool write =iov_iter_rw (iter) ==WRITE ; -bool should_dirty = !write &&iter_is_iovec (iter); +bool should_dirty = !write &&user_backed_iter (iter);if (write &&ceph_snap (file_inode (file)) !=CEPH_NOSNAP )return -EROFS ;
fs/cifs/file.c +1 -1
@@ -4004,7 +4004,7 @@ static ssize_t __cifs_readv(if (!is_sync_kiocb (iocb)) ctx->iocb = iocb; -if (iter_is_iovec (to)) +if (user_backed_iter (to)) ctx->should_dirty = true;if (direct) {
fs/direct-io.c +1 -1
@@ -1245,7 +1245,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode,spin_lock_init (&dio->bio_lock); dio->refcount =1 ; - dio->should_dirty =iter_is_iovec (iter) &&iov_iter_rw (iter) ==READ ; + dio->should_dirty =user_backed_iter (iter) &&iov_iter_rw (iter) ==READ ; sdio.iter = iter; sdio.final_block_in_request = end >> blkbits;
fs/fuse/dev.c +2 -2
@@ -1356,7 +1356,7 @@ static ssize_t fuse_dev_read(struct kiocb *iocb, struct iov_iter *to)if (!fud)return -EPERM ; -if (!iter_is_iovec (to)) +if (!user_backed_iter (to))return -EINVAL ;fuse_copy_init (&cs,1 , to);
@@ -1949,7 +1949,7 @@ static ssize_t fuse_dev_write(struct kiocb *iocb, struct iov_iter *from)if (!fud)return -EPERM ; -if (!iter_is_iovec (from)) +if (!user_backed_iter (from))return -EINVAL ;fuse_copy_init (&cs,0 , from);
fs/fuse/file.c +1 -1
@@ -1465,7 +1465,7 @@ ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter,inode_unlock (inode); } - io->should_dirty = !write &&iter_is_iovec (iter); + io->should_dirty = !write &&user_backed_iter (iter);while (count) {ssize_t nres;fl_owner_t owner = current->files;
fs/gfs2/file.c +1 -1
@@ -780,7 +780,7 @@ static inline bool should_fault_in_pages(struct iov_iter *i,if (!count)return false; -if (!iter_is_iovec (i)) +if (!user_backed_iter (i))return false; size =PAGE_SIZE ;
fs/iomap/direct-io.c +1 -1
@@ -533,7 +533,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomi.flags |=IOMAP_NOWAIT ; } -if (iter_is_iovec (iter)) +if (user_backed_iter (iter)) dio->flags |=IOMAP_DIO_DIRTY ; }else { iomi.flags |=IOMAP_WRITE ;
fs/nfs/direct.c +1 -1
@@ -478,7 +478,7 @@ ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter,if (!is_sync_kiocb (iocb)) dreq->iocb = iocb; -if (iter_is_iovec (iter)) +if (user_backed_iter (iter)) dreq->flags =NFS_ODIRECT_SHOULD_DIRTY ;if (!swap)
include/linux/uio.h +26 -0
@@ -26,6 +26,7 @@ enum iter_type {ITER_PIPE ,ITER_XARRAY ,ITER_DISCARD , + ITER_UBUF, }; struct iov_iter_state {
@@ -38,6 +39,7 @@ struct iov_iter {u8 iter_type;bool nofault;bool data_source; +bool user_backed;size_t iov_offset;size_t count;union {
@@ -46,6 +48,7 @@ struct iov_iter {const struct bio_vec *bvec;struct xarray *xarray;struct pipe_inode_info *pipe; +void __user *ubuf; };union {unsigned long nr_segs;
@@ -70,6 +73,11 @@ static inline void iov_iter_save_state(struct iov_iter *iter, state->nr_segs = iter->nr_segs; } +static inline bool iter_is_ubuf (const struct iov_iter *i) +{ +return iov_iter_type (i) ==ITER_UBUF ; +} +static inline bool iter_is_iovec (const struct iov_iter *i) {return iov_iter_type (i) ==ITER_IOVEC ;
@@ -105,6 +113,11 @@ static inline unsigned char iov_iter_rw(const struct iov_iter *i)return i->data_source ?WRITE :READ ; } +static inline bool user_backed_iter (const struct iov_iter *i) +{ +return i->user_backed; +} +/* * Total number of bytes covered by an iovec. *
@@ -320,4 +333,17 @@ ssize_t __import_iovec(int type, const struct iovec __user *uvec,int import_single_range (int type,void __user *buf,size_t len,struct iovec *iov,struct iov_iter *i); +static inline void iov_iter_ubuf (struct iov_iter *i,unsigned int direction, +void __user *buf,size_t count) +{ +WARN_ON (direction & ~(READ |WRITE )); + *i = (struct iov_iter ) { + .iter_type =ITER_UBUF , + .user_backed = true, + .data_source = direction, + .ubuf = buf, + .count = count + }; +} +#endif
lib/iov_iter.c +74 -20
217 lines not on this page · raw message
mm/shmem.c +1 -1
@@ -2603,7 +2603,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) ret =copy_page_to_iter (page, offset, nr, to);put_page (page); - }else if (iter_is_iovec (to)) { + }else if (!user_backed_iter (to)) {/* * Copy to user tends to be so well optimized, but * clear_user() not so much, that it is noticeably
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 16/44 ITER_PIPE: allocate buffers as we go in copy-to-pipe primitives
In reply to Al ViroNew helper: append_pipe(). Extends the last buffer if possible, allocates a new one otherwise. Returns page and offset in it on success, NULL on failure. iov_iter is advanced past the data we've got.
Use that instead of push_pipe() in copy-to-pipe primitives; they get simpler that way. Handling of short copy (in "mc" one) is done simply by iov_iter_revert() - iov_iter is in consistent state after that one, so we can use that.
--- lib/iov_iter.c | 159 +++++++++++++++++++++++++++++-------------------- 1 file changed, 93 insertions(+), 66 deletions(-)
lib/iov_iter.c +93 -66
@@ -259,6 +259,44 @@ static void push_page(struct pipe_inode_info *pipe, struct page *page,get_page (page); } +static inline bool allocated (struct pipe_buffer *buf) +{ +return buf->ops == &default_pipe_buf_ops; +} + +static struct page *append_pipe (struct iov_iter *i,size_t size,size_t *off) +{ +struct pipe_inode_info *pipe = i->pipe; +size_t offset = i->iov_offset; +struct pipe_buffer *buf; +struct page *page; + +if (offset && offset <PAGE_SIZE ) { +// some space in the last buffer; can we add to it? + buf =pipe_buf (pipe, pipe->head -1 ); +if (allocated (buf)) { + size =min_t (size_t, size,PAGE_SIZE - offset); + buf->len += size; + i->iov_offset += size; + i->count -= size; + *off = offset; +return buf->page; + } + } +// OK, we need a new buffer + *off =0 ; + size =min_t (size_t, size,PAGE_SIZE ); +if (pipe_full (pipe->head, pipe->tail, pipe->max_usage)) +return NULL ; + page =push_anon (pipe, size); +if (!page) +return NULL ; + i->head = pipe->head -1 ; + i->iov_offset = size; + i->count -= size; +return page; +} +static size_t copy_page_to_iter_pipe (struct page *page,size_t offset,size_t bytes,struct iov_iter *i) {
@@ -396,11 +434,6 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, }EXPORT_SYMBOL (iov_iter_init); -static inline bool allocated (struct pipe_buffer *buf) -{ -return buf->ops == &default_pipe_buf_ops; -} -static inline void data_start (const struct iov_iter *i,unsigned int *iter_headp,size_t *offp) {
@@ -459,28 +492,26 @@ static size_t push_pipe(struct iov_iter *i, size_t size,static size_t copy_pipe_to_iter (const void *addr,size_t bytes,struct iov_iter *i) { -struct pipe_inode_info *pipe = i->pipe; -unsigned int p_mask = pipe->ring_size -1 ; -unsigned int i_head;size_t n, off; -if (!sanity (i)) +if (unlikely (bytes > i->count)) + bytes = i->count; +if (unlikely (!bytes))return 0 ; - bytes = n =push_pipe (i, bytes, &i_head, &off); -if (unlikely (!n)) +if (!sanity (i))return 0 ; -do { + + n = bytes; +while (n) { +struct page *page =append_pipe (i, n, &off);size_t chunk =min_t (size_t, n,PAGE_SIZE - off); -memcpy_to_page (pipe->bufs[i_head & p_mask].page, off, addr, chunk); - i->head = i_head; - i->iov_offset = off + chunk; - n -= chunk; +if (!page) +break ; +memcpy_to_page (page, off, addr, chunk); addr += chunk; - off =0 ; - i_head++; - }while (n); - i->count -= bytes; + n -= chunk; + }return bytes; }
@@ -494,31 +525,32 @@ static __wsum csum_and_memcpy(void *to, const void *from, size_t len,static size_t csum_and_copy_to_pipe_iter (const void *addr,size_t bytes,struct iov_iter *i,__wsum *sump) { -struct pipe_inode_info *pipe = i->pipe; -unsigned int p_mask = pipe->ring_size -1 ;__wsum sum = *sump;size_t off =0 ; -unsigned int i_head;size_t r; +if (unlikely (bytes > i->count)) + bytes = i->count; +if (unlikely (!bytes)) +return 0 ; +if (!sanity (i))return 0 ; - bytes =push_pipe (i, bytes, &i_head, &r);while (bytes) { +struct page *page =append_pipe (i, bytes, &r);size_t chunk =min_t (size_t, bytes,PAGE_SIZE - r); -char *p =kmap_local_page (pipe->bufs[i_head & p_mask].page); +char *p; + +if (!page) +break ; + p =kmap_local_page (page); sum =csum_and_memcpy (p + r, addr + off, chunk, sum, off);kunmap_local (p); - i->head = i_head; - i->iov_offset = r + chunk; - bytes -= chunk; off += chunk; - r =0 ; - i_head++; + bytes -= chunk; } *sump = sum; - i->count -= off;return off; }
@@ -550,39 +582,35 @@ static int copyout_mc(void __user *to, const void *from, size_t n)static size_t copy_mc_pipe_to_iter (const void *addr,size_t bytes,struct iov_iter *i) { -struct pipe_inode_info *pipe = i->pipe; -unsigned int p_mask = pipe->ring_size -1 ; -unsigned int i_head; -unsigned int valid = pipe->head; -size_t n, off, xfer =0 ; +size_t off, xfer =0 ; + +if (unlikely (bytes > i->count)) + bytes = i->count; +if (unlikely (!bytes)) +return 0 ;if (!sanity (i))return 0 ; - n =push_pipe (i, bytes, &i_head, &off); -while (n) { -size_t chunk =min_t (size_t, n,PAGE_SIZE - off); -char *p =kmap_local_page (pipe->bufs[i_head & p_mask].page); +while (bytes) { +struct page *page =append_pipe (i, bytes, &off); +size_t chunk =min_t (size_t, bytes,PAGE_SIZE - off);unsigned long rem; +char *p; + +if (!page) +break ; + p =kmap_local_page (page); rem =copy_mc_to_kernel (p + off, addr + xfer, chunk); chunk -= rem;kunmap_local (p); -if (chunk) { - i->head = i_head; - i->iov_offset = off + chunk; - xfer += chunk; - valid = i_head +1 ; - } + xfer += chunk; + bytes -= chunk;if (rem) { - pipe->bufs[i_head & p_mask].len -= rem; -pipe_discard_from (pipe, valid); +iov_iter_revert (i, rem);break ; } - n -= chunk; - off =0 ; - i_head++; } - i->count -= xfer;return xfer; }
@@ -769,30 +797,29 @@ EXPORT_SYMBOL(copy_page_from_iter);static size_t pipe_zero (size_t bytes,struct iov_iter *i) { -struct pipe_inode_info *pipe = i->pipe; -unsigned int p_mask = pipe->ring_size -1 ; -unsigned int i_head;size_t n, off; -if (!sanity (i)) +if (unlikely (bytes > i->count)) + bytes = i->count; +if (unlikely (!bytes))return 0 ; - bytes = n =push_pipe (i, bytes, &i_head, &off); -if (unlikely (!n)) +if (!sanity (i))return 0 ; -do { + n = bytes; +while (n) { +struct page *page =append_pipe (i, n, &off);size_t chunk =min_t (size_t, n,PAGE_SIZE - off); -char *p =kmap_local_page (pipe->bufs[i_head & p_mask].page); +char *p; + +if (!page) +break ; + p =kmap_local_page (page);memset (p + off,0 , chunk);kunmap_local (p); - i->head = i_head; - i->iov_offset = off + chunk; n -= chunk; - off =0 ; - i_head++; - }while (n); - i->count -= bytes; + }return bytes; }
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 17/44 ITER_PIPE: fold push_pipe() into __pipe_get_pages()
In reply to Al Viro Expand the only remaining call of push_pipe() (in__pipe_get_pages()), combine it with the page-collecting loop there.
Note that the only reason it's not a loop doing append_pipe() is that append_pipe() is advancing, while iov_iter_get_pages() is not. As soon as it switches to saner semantics, this thing will switch to using append_pipe().
--- lib/iov_iter.c | 80 ++++++++++++++++---------------------------------- 1 file changed, 25 insertions(+), 55 deletions(-)
lib/iov_iter.c +25 -55
@@ -449,46 +449,6 @@ static inline void data_start(const struct iov_iter *i, *offp = off; } -static size_t push_pipe (struct iov_iter *i,size_t size, -int *iter_headp,size_t *offp) -{ -struct pipe_inode_info *pipe = i->pipe; -unsigned int iter_head; -size_t off; -ssize_t left; - -if (unlikely (size > i->count)) - size = i->count; -if (unlikely (!size)) -return 0 ; - - left = size; -data_start (i, &iter_head, &off); - *iter_headp = iter_head; - *offp = off; -if (off) { -struct pipe_buffer *buf =pipe_buf (pipe, iter_head); - - left -=PAGE_SIZE - off; -if (left <=0 ) { - buf->len += size; -return size; - } - buf->len =PAGE_SIZE ; - } -while (!pipe_full (pipe->head, pipe->tail, pipe->max_usage)) { -struct page *page =push_anon (pipe, -min_t (ssize_t, left,PAGE_SIZE )); -if (!page) -break ; - - left -=PAGE_SIZE ; -if (left <=0 ) -return size; - } -return size - left; -} -static size_t copy_pipe_to_iter (const void *addr,size_t bytes,struct iov_iter *i) {
@@ -1261,23 +1221,33 @@ static inline ssize_t __pipe_get_pages(struct iov_iter *i,size_t maxsize, struct page **pages, int iter_head, - size_t *start) + size_t off) { struct pipe_inode_info *pipe = i->pipe; -unsigned int p_mask = pipe->ring_size -1 ; -ssize_t n =push_pipe (i, maxsize, &iter_head, start); -if (!n) -return -EFAULT ; +ssize_t left = maxsize; - maxsize = n; - n += *start; -while (n >0 ) { -get_page (*pages++ = pipe->bufs[iter_head & p_mask].page); - iter_head++; - n -=PAGE_SIZE ; - } +if (off) { +struct pipe_buffer *buf =pipe_buf (pipe, iter_head); -return maxsize; +get_page (*pages++ = buf->page); + left -=PAGE_SIZE - off; +if (left <=0 ) { + buf->len += maxsize; +return maxsize; + } + buf->len =PAGE_SIZE ; + } +while (!pipe_full (pipe->head, pipe->tail, pipe->max_usage)) { +struct page *page =push_anon (pipe, +min_t (ssize_t, left,PAGE_SIZE )); +if (!page) +break ; +get_page (*pages++ = page); + left -=PAGE_SIZE ; +if (left <=0 ) +return maxsize; + } +return maxsize - left ? : -EFAULT ; } static ssize_t pipe_get_pages(struct iov_iter *i,
@@ -1295,7 +1265,7 @@ static ssize_t pipe_get_pages(struct iov_iter *i, npages =pipe_space_for_user (iter_head, i->pipe->tail, i->pipe); capacity =min (npages, maxpages) *PAGE_SIZE - *start; -return __pipe_get_pages (i,min (maxsize, capacity), pages, iter_head, start); +return __pipe_get_pages (i,min (maxsize, capacity), pages, iter_head, *start); } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
@@ -1491,7 +1461,7 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, p =get_pages_array (npages);if (!p)return -ENOMEM ; - n =__pipe_get_pages (i, maxsize, p, iter_head, start); + n =__pipe_get_pages (i, maxsize, p, iter_head, *start);if (n >0 ) *pages = p;else
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 19/44 ITER_PIPE: clean pipe_advance() up
In reply to Al Viroinstead of setting ->iov_offset for new position and calling pipe_truncate() to adjust ->len of the last buffer and discard everything after it, adjust ->len at the same time we set ->iov_offset and use pipe_discard_from() to deal with buffers past that.
--- lib/iov_iter.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
lib/iov_iter.c +17 -17
@@ -847,27 +847,27 @@ static inline void pipe_truncate(struct iov_iter *i)static void pipe_advance (struct iov_iter *i,size_t size) {struct pipe_inode_info *pipe = i->pipe; -if (size) { -struct pipe_buffer *buf; -unsigned int p_mask = pipe->ring_size -1 ; -unsigned int i_head = i->head; -size_t off = i->iov_offset, left = size; +unsigned int off = i->iov_offset; +if (!off && !size) { +pipe_discard_from (pipe, i->start_head);// discard everything +return ; + } + i->count -= size; +while (1 ) { +struct pipe_buffer *buf =pipe_buf (pipe, i->head);if (off)/* make it relative to the beginning of buffer */ - left += off - pipe->bufs[i_head & p_mask].offset; -while (1 ) { - buf = &pipe->bufs[i_head & p_mask]; -if (left <= buf->len) -break ; - left -= buf->len; - i_head++; + size += off - buf->offset; +if (size <= buf->len) { + buf->len = size; + i->iov_offset = buf->offset + size; +break ; } - i->head = i_head; - i->iov_offset = buf->offset + left; + size -= buf->len; + i->head++; + off =0 ; } - i->count -= size; -/* ... and discard everything past that point */ -pipe_truncate (i); +pipe_discard_from (pipe, i->head +1 );// discard everything past this one }static void iov_iter_bvec_advance (struct iov_iter *i,size_t size)
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 23/44 iov_iter_get_pages{,_alloc}(): cap the maxsize with MAX_RW_COUNT
In reply to Al ViroAll callers can and should handle iov_iter_get_pages() returning fewer pages than requested. All in-kernel ones do. And it makes the arithmetical overflow analysis much simpler...
--- lib/iov_iter.c | 4 ++++ 1 file changed, 4 insertions(+)
lib/iov_iter.c +4 -0
@@ -1367,6 +1367,8 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, maxsize = i->count;if (!maxsize)return 0 ; +if (maxsize >MAX_RW_COUNT ) + maxsize =MAX_RW_COUNT ;if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;
@@ -1485,6 +1487,8 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, maxsize = i->count;if (!maxsize)return 0 ; +if (maxsize >MAX_RW_COUNT ) + maxsize =MAX_RW_COUNT ;if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 18/44 ITER_PIPE: lose iter_head argument of __pipe_get_pages()
In reply to Al Viroit's only used to get to the partial buffer we can add to, and that's always the last one, i.e. pipe->head - 1.
--- lib/iov_iter.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
lib/iov_iter.c +3 -4
@@ -1220,14 +1220,13 @@ EXPORT_SYMBOL(iov_iter_gap_alignment);static inline ssize_t __pipe_get_pages (struct iov_iter *i,size_t maxsize,struct page **pages, -int iter_head,size_t off) {struct pipe_inode_info *pipe = i->pipe;ssize_t left = maxsize;if (off) { -struct pipe_buffer *buf =pipe_buf (pipe, iter_head); +struct pipe_buffer *buf =pipe_buf (pipe, pipe->head -1 );get_page (*pages++ = buf->page); left -=PAGE_SIZE - off;
@@ -1265,7 +1264,7 @@ static ssize_t pipe_get_pages(struct iov_iter *i, npages =pipe_space_for_user (iter_head, i->pipe->tail, i->pipe); capacity =min (npages, maxpages) *PAGE_SIZE - *start; -return __pipe_get_pages (i,min (maxsize, capacity), pages, iter_head, *start); +return __pipe_get_pages (i,min (maxsize, capacity), pages, *start); } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
@@ -1461,7 +1460,7 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, p =get_pages_array (npages);if (!p)return -ENOMEM ; - n =__pipe_get_pages (i, maxsize, p, iter_head, *start); + n =__pipe_get_pages (i, maxsize, p, *start);if (n >0 ) *pages = p;else
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 22/44 ITER_PIPE: fold data_start() and pipe_space_for_user() together
In reply to Al ViroAll their callers are next to each other; all of them want the total amount of pages and, possibly, the offset in the partial final buffer.
Combine into a new helper (pipe_npages()), fix the bogosity in pipe_space_for_user(), while we are at it.
--- include/linux/pipe_fs_i.h | 20 ------------------ lib/iov_iter.c | 44 +++++++++++++++++---------------------- 2 files changed, 19 insertions(+), 45 deletions(-)
include/linux/pipe_fs_i.h +0 -20
@@ -156,26 +156,6 @@ static inline bool pipe_full(unsigned int head, unsigned int tail,return pipe_occupancy (head, tail) >= limit; } -/** -* pipe_space_for_user - Return number of slots available to userspace -* @head: The pipe ring head pointer -* @tail: The pipe ring tail pointer -* @pipe: The pipe info structure -*/ -static inline unsigned int pipe_space_for_user (unsigned int head,unsigned int tail, -struct pipe_inode_info *pipe) -{ -unsigned int p_occupancy, p_space; - - p_occupancy =pipe_occupancy (head, tail); -if (p_occupancy >= pipe->max_usage) -return 0 ; - p_space = pipe->ring_size - p_occupancy; -if (p_space > pipe->max_usage) - p_space = pipe->max_usage; -return p_space; -} -/** * pipe_buf_get - get a reference to a pipe_buffer * @pipe: the pipe that the buffer belongs to
lib/iov_iter.c +19 -25
95 lines not on this page · raw message
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 26/44 unify pipe_get_pages() and pipe_get_pages_alloc()
In reply to Al Viro The differences between those two are
* pipe_get_pages() gets a non-NULL struct page ** value pointing to preallocated array + array size.
* pipe_get_pages_alloc() gets an address of struct page ** variable that contains NULL, allocates the array and (on success) stores its address in that variable.
Not hard to combine - always pass struct page ***, have
the previous pipe_get_pages_alloc() caller pass ~0U as cap for array size.
--- lib/iov_iter.c | 49 +++++++++++++++++-------------------------------- 1 file changed, 17 insertions(+), 32 deletions(-)
lib/iov_iter.c +17 -32
@@ -1187,6 +1187,11 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i) }EXPORT_SYMBOL (iov_iter_gap_alignment); +static struct page **get_pages_array (size_t n) +{ +return kvmalloc_array (n,sizeof (struct page *),GFP_KERNEL ); +} + static inline ssize_t __pipe_get_pages(struct iov_iter *i, size_t maxsize, struct page **pages,
@@ -1220,10 +1225,11 @@ static inline ssize_t __pipe_get_pages(struct iov_iter *i, }static ssize_t pipe_get_pages (struct iov_iter *i, -struct page **pages,size_t maxsize,unsigned maxpages, +struct page ***pages,size_t maxsize,unsigned maxpages,size_t *start) {unsigned int npages, off; +struct page **p;size_t capacity;if (!sanity (i))
@@ -1231,8 +1237,15 @@ static ssize_t pipe_get_pages(struct iov_iter *i, *start = off =pipe_npages (i, &npages); capacity =min (npages, maxpages) *PAGE_SIZE - off; + maxsize =min (maxsize, capacity); + p = *pages; +if (!p) { + *pages = p =get_pages_array (DIV_ROUND_UP (maxsize + off,PAGE_SIZE )); +if (!p) +return -ENOMEM ; + } -return __pipe_get_pages (i,min (maxsize, capacity), pages, off); +return __pipe_get_pages (i, maxsize, p, off); } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
@@ -1394,41 +1407,13 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,return len - *start; }if (iov_iter_is_pipe (i)) -return pipe_get_pages (i, pages, maxsize, maxpages, start); +return pipe_get_pages (i, &pages, maxsize, maxpages, start);if (iov_iter_is_xarray (i))return iter_xarray_get_pages (i, pages, maxsize, maxpages, start);return -EFAULT ; }EXPORT_SYMBOL (iov_iter_get_pages); -static struct page **get_pages_array (size_t n) -{ -return kvmalloc_array (n,sizeof (struct page *),GFP_KERNEL ); -} - -static ssize_t pipe_get_pages_alloc (struct iov_iter *i, -struct page ***pages,size_t maxsize, -size_t *start) -{ -struct page **p; -unsigned int npages, off; -ssize_t n; - -if (!sanity (i)) -return -EFAULT ; - - *start = off =pipe_npages (i, &npages); - n = npages *PAGE_SIZE - off; -if (maxsize > n) - maxsize = n; -else - npages =DIV_ROUND_UP (maxsize + off,PAGE_SIZE ); - *pages = p =get_pages_array (npages); -if (!p) -return -ENOMEM ; -return __pipe_get_pages (i, maxsize, p, off); -} -static ssize_t iter_xarray_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *_start_offset)
@@ -1511,7 +1496,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,return len - *start; }if (iov_iter_is_pipe (i)) -return pipe_get_pages_alloc (i, pages, maxsize, start); +return pipe_get_pages (i, pages, maxsize, ~0U , start);if (iov_iter_is_xarray (i))return iter_xarray_get_pages_alloc (i, pages, maxsize, start);return -EFAULT ;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 28/44 unify the rest of iov_iter_get_pages()/iov_iter_get_pages_alloc() guts
In reply to Al Virosame as for pipes and xarrays; after that iov_iter_get_pages() becomes a wrapper for __iov_iter_get_pages_alloc().
--- lib/iov_iter.c | 86 ++++++++++++++++---------------------------------- 1 file changed, 28 insertions(+), 58 deletions(-)
lib/iov_iter.c +28 -58
@@ -1372,20 +1372,19 @@ static struct page *first_bvec_segment(const struct iov_iter *i,return page; } -ssize_t iov_iter_get_pages (struct iov_iter *i, -struct page **pages,size_t maxsize,unsigned maxpages, -size_t *start) +static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i, +struct page ***pages,size_t maxsize, +unsigned int maxpages,size_t *start) {size_t len;int n, res;if (maxsize > i->count) maxsize = i->count; -if (!maxsize || !maxpages) +if (!maxsize)return 0 ;if (maxsize >MAX_RW_COUNT ) maxsize =MAX_RW_COUNT ; -BUG_ON (!pages);if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;
@@ -1398,80 +1397,51 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, addr =first_iovec_segment (i, &len, start, maxsize, maxpages); n =DIV_ROUND_UP (len,PAGE_SIZE ); - res =get_user_pages_fast (addr, n, gup_flags, pages); +if (!*pages) { + *pages =get_pages_array (n); +if (!*pages) +return -ENOMEM ; + } + res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res;return (res == n ? len : res *PAGE_SIZE ) - *start; }if (iov_iter_is_bvec (i)) { +struct page **p;struct page *page; page =first_bvec_segment (i, &len, start, maxsize, maxpages); n =DIV_ROUND_UP (len,PAGE_SIZE ); + p = *pages; +if (!p) { + *pages = p =get_pages_array (n); +if (!p) +return -ENOMEM ; + }while (n--) -get_page (*pages++ = page++); +get_page (*p++ = page++);return len - *start; }if (iov_iter_is_pipe (i)) -return pipe_get_pages (i, &pages, maxsize, maxpages, start); +return pipe_get_pages (i, pages, maxsize, maxpages, start);if (iov_iter_is_xarray (i)) -return iter_xarray_get_pages (i, &pages, maxsize, maxpages, start); +return iter_xarray_get_pages (i, pages, maxsize, maxpages, + start);return -EFAULT ; } -EXPORT_SYMBOL (iov_iter_get_pages); -static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i, -struct page ***pages,size_t maxsize, +ssize_t iov_iter_get_pages (struct iov_iter *i, +struct page **pages,size_t maxsize,unsigned maxpages,size_t *start) { -struct page **p; -size_t len; -int n, res; - -if (maxsize > i->count) - maxsize = i->count; -if (!maxsize) +if (!maxpages)return 0 ; -if (maxsize >MAX_RW_COUNT ) - maxsize =MAX_RW_COUNT ; - -if (likely (user_backed_iter (i))) { -unsigned int gup_flags =0 ; -unsigned long addr; - -if (iov_iter_rw (i) !=WRITE ) - gup_flags |=FOLL_WRITE ; -if (i->nofault) - gup_flags |=FOLL_NOFAULT ; - - addr =first_iovec_segment (i, &len, start, maxsize, ~0U ); - n =DIV_ROUND_UP (len,PAGE_SIZE ); - *pages = p =get_pages_array (n); -if (!p) -return -ENOMEM ; - res =get_user_pages_fast (addr, n, gup_flags, p); -if (unlikely (res <=0 )) -return res; -return (res == n ? len : res *PAGE_SIZE ) - *start; - } -if (iov_iter_is_bvec (i)) { -struct page *page; +BUG_ON (!pages); - page =first_bvec_segment (i, &len, start, maxsize, ~0U ); - n =DIV_ROUND_UP (len,PAGE_SIZE ); - *pages = p =get_pages_array (n); -if (!p) -return -ENOMEM ; -while (n--) -get_page (*p++ = page++); -return len - *start; - } -if (iov_iter_is_pipe (i)) -return pipe_get_pages (i, pages, maxsize, ~0U , start); -if (iov_iter_is_xarray (i)) -return iter_xarray_get_pages (i, pages, maxsize, ~0U , start); -return -EFAULT ; +return __iov_iter_get_pages_alloc (i, &pages, maxsize, maxpages, start); } +EXPORT_SYMBOL (iov_iter_get_pages); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize,
@@ -1481,7 +1451,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, *pages =NULL ; - len =__iov_iter_get_pages_alloc (i, pages, maxsize, start); + len =__iov_iter_get_pages_alloc (i, pages, maxsize, ~0U , start);if (len <=0 ) {kvfree (*pages); *pages =NULL ;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 30/44 iov_iter: lift dealing with maxpages out of first_{iovec,bvec}_segment()
In reply to Al Viro--- lib/iov_iter.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-)
lib/iov_iter.c +11 -12
@@ -1308,12 +1308,9 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,static unsigned long found_ubuf_segment (unsigned long addr,size_t len, -size_t *size,size_t *start, -unsigned maxpages) +size_t *size,size_t *start) { len += (*start = addr %PAGE_SIZE ); -if (len > maxpages *PAGE_SIZE ) - len = maxpages *PAGE_SIZE ; *size = len;return addr &PAGE_MASK ; }
@@ -1321,14 +1318,14 @@ static unsigned long found_ubuf_segment(unsigned long addr,/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment (const struct iov_iter *i,size_t *size,size_t *start, -size_t maxsize,unsigned maxpages) +size_t maxsize) {size_t skip;long k;if (iter_is_ubuf (i)) {unsigned long addr = (unsigned long )i->ubuf + i->iov_offset; -return found_ubuf_segment (addr, maxsize, size, start, maxpages); +return found_ubuf_segment (addr, maxsize, size, start); }for (k =0 , skip = i->iov_offset; k < i->nr_segs; k++, skip =0 ) {
@@ -1339,7 +1336,7 @@ static unsigned long first_iovec_segment(const struct iov_iter *i,continue ;if (len > maxsize) len = maxsize; -return found_ubuf_segment (addr, len, size, start, maxpages); +return found_ubuf_segment (addr, len, size, start); }BUG ();// if it had been empty, we wouldn't get called }
@@ -1347,7 +1344,7 @@ static unsigned long first_iovec_segment(const struct iov_iter *i,/* must be done on non-empty ITER_BVEC one */ static struct page *first_bvec_segment (const struct iov_iter *i,size_t *size,size_t *start, -size_t maxsize,unsigned maxpages) +size_t maxsize) {struct page *page;size_t skip = i->iov_offset, len;
@@ -1358,8 +1355,6 @@ static struct page *first_bvec_segment(const struct iov_iter *i, skip += i->bvec->bv_offset; page = i->bvec->bv_page + skip /PAGE_SIZE ; len += (*start = skip %PAGE_SIZE ); -if (len > maxpages *PAGE_SIZE ) - len = maxpages *PAGE_SIZE ; *size = len;return page; }
@@ -1387,7 +1382,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (i->nofault) gup_flags |=FOLL_NOFAULT ; - addr =first_iovec_segment (i, &len, start, maxsize, maxpages); + addr =first_iovec_segment (i, &len, start, maxsize); +if (len > maxpages *PAGE_SIZE ) + len = maxpages *PAGE_SIZE ; n =DIV_ROUND_UP (len,PAGE_SIZE );if (!*pages) { *pages =get_pages_array (n);
@@ -1403,7 +1400,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,struct page **p;struct page *page; - page =first_bvec_segment (i, &len, start, maxsize, maxpages); + page =first_bvec_segment (i, &len, start, maxsize); +if (len > maxpages *PAGE_SIZE ) + len = maxpages *PAGE_SIZE ; n =DIV_ROUND_UP (len,PAGE_SIZE ); p = *pages;if (!p) {
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 32/44 iov_iter: massage calling conventions for first_{iovec,bvec}_segment()
In reply to Al ViroPass maxsize by reference, return length via the same.
--- lib/iov_iter.c | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-)
lib/iov_iter.c +15 -22
@@ -1306,26 +1306,22 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); } -static unsigned long found_ubuf_segment (unsigned long addr, -size_t len, -size_t *size,size_t *start) +static unsigned long found_ubuf_segment (unsigned long addr,size_t *start) { *start = addr %PAGE_SIZE ; - *size = len;return addr &PAGE_MASK ; }/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment (const struct iov_iter *i, -size_t *size,size_t *start, -size_t maxsize) +size_t *size,size_t *start) {size_t skip;long k;if (iter_is_ubuf (i)) {unsigned long addr = (unsigned long )i->ubuf + i->iov_offset; -return found_ubuf_segment (addr, maxsize, size, start); +return found_ubuf_segment (addr, start); }for (k =0 , skip = i->iov_offset; k < i->nr_segs; k++, skip =0 ) {
@@ -1334,28 +1330,26 @@ static unsigned long first_iovec_segment(const struct iov_iter *i,if (unlikely (!len))continue ; -if (len > maxsize) - len = maxsize; -return found_ubuf_segment (addr, len, size, start); +if (*size > len) + *size = len; +return found_ubuf_segment (addr, start); }BUG ();// if it had been empty, we wouldn't get called }/* must be done on non-empty ITER_BVEC one */ static struct page *first_bvec_segment (const struct iov_iter *i, -size_t *size,size_t *start, -size_t maxsize) +size_t *size,size_t *start) {struct page *page;size_t skip = i->iov_offset, len; len = i->bvec->bv_len - skip; -if (len > maxsize) - len = maxsize; +if (*size > len) + *size = len; skip += i->bvec->bv_offset; page = i->bvec->bv_page + skip /PAGE_SIZE ; *start = skip %PAGE_SIZE ; - *size = len;return page; }
@@ -1363,7 +1357,6 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start) { - size_t len; int n, res;if (maxsize > i->count)
@@ -1382,8 +1375,8 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (i->nofault) gup_flags |=FOLL_NOFAULT ; - addr =first_iovec_segment (i, &len, start, maxsize); - n =DIV_ROUND_UP (len + *start,PAGE_SIZE ); + addr =first_iovec_segment (i, &maxsize, start); + n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE );if (n > maxpages) n = maxpages;if (!*pages) {
@@ -1394,14 +1387,14 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res; -return min_t (size_t, len, res *PAGE_SIZE - *start); +return min_t (size_t, maxsize, res *PAGE_SIZE - *start); }if (iov_iter_is_bvec (i)) {struct page **p;struct page *page; - page =first_bvec_segment (i, &len, start, maxsize); - n =DIV_ROUND_UP (len + *start,PAGE_SIZE ); + page =first_bvec_segment (i, &maxsize, start); + n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE );if (n > maxpages) n = maxpages; p = *pages;
@@ -1412,7 +1405,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, }for (int k =0 ; k < n; k++)get_page (*p++ = page++); -return min_t (size_t, len, n *PAGE_SIZE - *start); +return min_t (size_t, maxsize, n *PAGE_SIZE - *start); }if (iov_iter_is_pipe (i))return pipe_get_pages (i, pages, maxsize, maxpages, start);
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 33/44 found_iovec_segment(): just return address
In reply to Al Viro... and calculate the offset in the caller
--- lib/iov_iter.c | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-)
lib/iov_iter.c +7 -15
@@ -1306,33 +1306,23 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); } -static unsigned long found_ubuf_segment (unsigned long addr,size_t *start) -{ - *start = addr %PAGE_SIZE ; -return addr &PAGE_MASK ; -} -/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment (const struct iov_iter *i, -size_t *size,size_t *start) +size_t *size) {size_t skip;long k; -if (iter_is_ubuf (i)) { -unsigned long addr = (unsigned long )i->ubuf + i->iov_offset; -return found_ubuf_segment (addr, start); - } +if (iter_is_ubuf (i)) +return (unsigned long )i->ubuf + i->iov_offset;for (k =0 , skip = i->iov_offset; k < i->nr_segs; k++, skip =0 ) { -unsigned long addr = (unsigned long )i->iov[k].iov_base + skip;size_t len = i->iov[k].iov_len - skip; -if (unlikely (!len))continue ;if (*size > len) *size = len; -return found_ubuf_segment (addr, start); +return (unsigned long )i->iov[k].iov_base + skip; }BUG ();// if it had been empty, we wouldn't get called }
@@ -1375,7 +1365,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (i->nofault) gup_flags |=FOLL_NOFAULT ; - addr =first_iovec_segment (i, &maxsize, start); + addr =first_iovec_segment (i, &maxsize); + *start = addr %PAGE_SIZE ; + addr &=PAGE_MASK ; n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE );if (n > maxpages) n = maxpages;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 34/44 fold __pipe_get_pages() into pipe_get_pages()
In reply to Al Viro... and don't mangle maxsize there - turn the loop into counting one instead. Easier to see that we won't run out of array that way. Note that special treatment of the partial buffer in that thing is an artifact of the non-advancing semantics of iov_iter_get_pages() - if not for that, it would be append_pipe(), same as the body of the loop that follows it. IOW, once we make iov_iter_get_pages() advancing, the whole thing will turn into
calculate how many pages do we want
allocate an array (if needed)
call append_pipe() that many times.
--- lib/iov_iter.c | 75 +++++++++++++++++++++++++------------------------- 1 file changed, 38 insertions(+), 37 deletions(-)
lib/iov_iter.c +38 -37
@@ -1192,60 +1192,61 @@ static struct page **get_pages_array(size_t n)return kvmalloc_array (n,sizeof (struct page *),GFP_KERNEL ); } -static inline ssize_t __pipe_get_pages (struct iov_iter *i, -size_t maxsize, -struct page **pages, -size_t off) -{ -struct pipe_inode_info *pipe = i->pipe; -ssize_t left = maxsize; - -if (off) { -struct pipe_buffer *buf =pipe_buf (pipe, pipe->head -1 ); - -get_page (*pages++ = buf->page); - left -=PAGE_SIZE - off; -if (left <=0 ) { - buf->len += maxsize; -return maxsize; - } - buf->len =PAGE_SIZE ; - } -while (!pipe_full (pipe->head, pipe->tail, pipe->max_usage)) { -struct page *page =push_anon (pipe, -min_t (ssize_t, left,PAGE_SIZE )); -if (!page) -break ; -get_page (*pages++ = page); - left -=PAGE_SIZE ; -if (left <=0 ) -return maxsize; - } -return maxsize - left ? : -EFAULT ; -} -static ssize_t pipe_get_pages (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned maxpages,size_t *start) { +struct pipe_inode_info *pipe = i->pipe;unsigned int npages, off;struct page **p; -size_t capacity; +ssize_t left; +int count;if (!sanity (i))return -EFAULT ; *start = off =pipe_npages (i, &npages); - capacity =min (npages, maxpages) *PAGE_SIZE - off; - maxsize =min (maxsize, capacity); + count =DIV_ROUND_UP (maxsize + off,PAGE_SIZE ); +if (count > npages) + count = npages; +if (count > maxpages) + count = maxpages; p = *pages;if (!p) { - *pages = p =get_pages_array (DIV_ROUND_UP (maxsize + off,PAGE_SIZE )); + *pages = p =get_pages_array (count);if (!p)return -ENOMEM ; } -return __pipe_get_pages (i, maxsize, p, off); + left = maxsize; + npages =0 ; +if (off) { +struct pipe_buffer *buf =pipe_buf (pipe, pipe->head -1 ); + +get_page (*p++ = buf->page); + left -=PAGE_SIZE - off; +if (left <=0 ) { + buf->len += maxsize; +return maxsize; + } + buf->len =PAGE_SIZE ; + npages =1 ; + } +for ( ; npages < count; npages++) { +struct page *page; +unsigned int size =min_t (ssize_t, left,PAGE_SIZE ); + +if (pipe_full (pipe->head, pipe->tail, pipe->max_usage)) +break ; + page =push_anon (pipe, size); +if (!page) +break ; +get_page (*p++ = page); + left -= size; + } +if (!npages) +return -EFAULT ; +return maxsize - left; } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 27/44 unify xarray_get_pages() and xarray_get_pages_alloc()
In reply to Al Virosame as for pipes
--- lib/iov_iter.c | 49 ++++++++++--------------------------------------- 1 file changed, 10 insertions(+), 39 deletions(-)
lib/iov_iter.c +10 -39
@@ -1276,7 +1276,7 @@ static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa }static ssize_t iter_xarray_get_pages (struct iov_iter *i, -struct page **pages,size_t maxsize, +struct page ***pages,size_t maxsize,unsigned maxpages,size_t *_start_offset) {unsigned nr, offset;
@@ -1301,7 +1301,13 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,if (count > maxpages) count = maxpages; - nr =iter_xarray_populate_pages (pages, i->xarray, index, count); +if (!*pages) { + *pages =get_pages_array (count); +if (!*pages) +return -ENOMEM ; + } + + nr =iter_xarray_populate_pages (*pages, i->xarray, index, count);if (nr ==0 )return 0 ;
@@ -1409,46 +1415,11 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,if (iov_iter_is_pipe (i))return pipe_get_pages (i, &pages, maxsize, maxpages, start);if (iov_iter_is_xarray (i)) -return iter_xarray_get_pages (i, pages, maxsize, maxpages, start); +return iter_xarray_get_pages (i, &pages, maxsize, maxpages, start);return -EFAULT ; }EXPORT_SYMBOL (iov_iter_get_pages); -static ssize_t iter_xarray_get_pages_alloc (struct iov_iter *i, -struct page ***pages,size_t maxsize, -size_t *_start_offset) -{ -struct page **p; -unsigned nr, offset; -pgoff_t index, count; -size_t size = maxsize; -loff_t pos; - - pos = i->xarray_start + i->iov_offset; - index = pos >>PAGE_SHIFT ; - offset = pos & ~PAGE_MASK ; - *_start_offset = offset; - - count =1 ; -if (size >PAGE_SIZE - offset) { - size -=PAGE_SIZE - offset; - count += size >>PAGE_SHIFT ; - size &= ~PAGE_MASK ; -if (size) - count++; - } - - *pages = p =get_pages_array (count); -if (!p) -return -ENOMEM ; - - nr =iter_xarray_populate_pages (p, i->xarray, index, count); -if (nr ==0 ) -return 0 ; - -return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); -} -static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start)
@@ -1498,7 +1469,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (iov_iter_is_pipe (i))return pipe_get_pages (i, pages, maxsize, ~0U , start);if (iov_iter_is_xarray (i)) -return iter_xarray_get_pages_alloc (i, pages, maxsize, start); +return iter_xarray_get_pages (i, pages, maxsize, ~0U , start);return -EFAULT ; }
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 20/44 ITER_PIPE: clean iov_iter_revert()
In reply to Al ViroFold pipe_truncate() into it, clean up. We can release buffers in the same loop where we walk backwards to the iterator beginning looking for the place where the new position will be.
--- lib/iov_iter.c | 60 ++++++++++++-------------------------------------- 1 file changed, 14 insertions(+), 46 deletions(-)
lib/iov_iter.c +14 -46
@@ -818,32 +818,6 @@ size_t copy_page_from_iter_atomic(struct page *page, unsigned offset, size_t byt }EXPORT_SYMBOL (copy_page_from_iter_atomic); -static inline void pipe_truncate (struct iov_iter *i) -{ -struct pipe_inode_info *pipe = i->pipe; -unsigned int p_tail = pipe->tail; -unsigned int p_head = pipe->head; -unsigned int p_mask = pipe->ring_size -1 ; - -if (!pipe_empty (p_head, p_tail)) { -struct pipe_buffer *buf; -unsigned int i_head = i->head; -size_t off = i->iov_offset; - -if (off) { - buf = &pipe->bufs[i_head & p_mask]; - buf->len = off - buf->offset; - i_head++; - } -while (p_head != i_head) { - p_head--; -pipe_buf_release (pipe, &pipe->bufs[p_head & p_mask]); - } - - pipe->head = p_head; - } -} -static void pipe_advance (struct iov_iter *i,size_t size) {struct pipe_inode_info *pipe = i->pipe;
@@ -938,28 +912,22 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll) i->count += unroll;if (unlikely (iov_iter_is_pipe (i))) {struct pipe_inode_info *pipe = i->pipe; -unsigned int p_mask = pipe->ring_size -1 ; -unsigned int i_head = i->head; -size_t off = i->iov_offset; -while (1 ) { -struct pipe_buffer *b = &pipe->bufs[i_head & p_mask]; -size_t n = off - b->offset; -if (unroll < n) { - off -= unroll; -break ; - } - unroll -= n; -if (!unroll && i_head == i->start_head) { - off =0 ; -break ; +unsigned int head = pipe->head; + +while (head > i->start_head) { +struct pipe_buffer *b =pipe_buf (pipe, --head); +if (unroll < b->len) { + b->len -= unroll; + i->iov_offset = b->offset + b->len; + i->head = head; +return ; } - i_head--; - b = &pipe->bufs[i_head & p_mask]; - off = b->offset + b->len; + unroll -= b->len; +pipe_buf_release (pipe, b); + pipe->head--; } - i->iov_offset = off; - i->head = i_head; -pipe_truncate (i); + i->iov_offset =0 ; + i->head = head;return ; }if (unlikely (iov_iter_is_discard (i)))
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 25/44 iov_iter_get_pages(): sanity-check arguments
In reply to Al Virozero maxpages is bogus, but best treated as "just return 0"; NULL pages, OTOH, should be treated as a hard bug.
get rid of now completely useless checks in xarray_get_pages{,_alloc}().
--- lib/iov_iter.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
lib/iov_iter.c +2 -7
@@ -1271,9 +1271,6 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,size_t size = maxsize;loff_t pos; -if (!size || !maxpages) -return 0 ; - pos = i->xarray_start + i->iov_offset; index = pos >>PAGE_SHIFT ; offset = pos & ~PAGE_MASK ;
@@ -1365,10 +1362,11 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,if (maxsize > i->count) maxsize = i->count; -if (!maxsize) +if (!maxsize || !maxpages)return 0 ;if (maxsize >MAX_RW_COUNT ) maxsize =MAX_RW_COUNT ; +BUG_ON (!pages);if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;
@@ -1441,9 +1439,6 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i,size_t size = maxsize;loff_t pos; -if (!size) -return 0 ; - pos = i->xarray_start + i->iov_offset; index = pos >>PAGE_SHIFT ; offset = pos & ~PAGE_MASK ;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 29/44 ITER_XARRAY: don't open-code DIV_ROUND_UP()
In reply to Al Viro--- lib/iov_iter.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-)
lib/iov_iter.c +1 -9
@@ -1289,15 +1289,7 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, offset = pos & ~PAGE_MASK ; *_start_offset = offset; - count =1 ; -if (size >PAGE_SIZE - offset) { - size -=PAGE_SIZE - offset; - count += size >>PAGE_SHIFT ; - size &= ~PAGE_MASK ; -if (size) - count++; - } - + count =DIV_ROUND_UP (size + offset,PAGE_SIZE );if (count > maxpages) count = maxpages;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 24/44 iov_iter_get_pages_alloc(): lift freeing pages array on failure exits into wrapper
In reply to Al ViroIncidentally, ITER_XARRAY did *not* free the sucker in case when iter_xarray_populate_pages() returned 0...
--- lib/iov_iter.c | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-)
lib/iov_iter.c +22 -16
@@ -1425,15 +1425,10 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, maxsize = n; else npages =DIV_ROUND_UP (maxsize + off,PAGE_SIZE ); - p =get_pages_array (npages); + *pages = p =get_pages_array (npages);if (!p)return -ENOMEM ; - n =__pipe_get_pages (i, maxsize, p, off); -if (n >0 ) - *pages = p; -else -kvfree (p); -return n; +return __pipe_get_pages (i, maxsize, p, off); } static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i,
@@ -1463,10 +1458,9 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, count++; } - p =get_pages_array (count); + *pages = p =get_pages_array (count);if (!p)return -ENOMEM ; - *pages = p; nr =iter_xarray_populate_pages (p, i->xarray, index, count);if (nr ==0 )
@@ -1475,7 +1469,7 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i,return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); } -ssize_t iov_iter_get_pages_alloc (struct iov_iter *i, +static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start) {
@@ -1501,16 +1495,12 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, addr =first_iovec_segment (i, &len, start, maxsize, ~0U ); n =DIV_ROUND_UP (len,PAGE_SIZE ); - p =get_pages_array (n); + *pages = p =get_pages_array (n);if (!p)return -ENOMEM ; res =get_user_pages_fast (addr, n, gup_flags, p); -if (unlikely (res <=0 )) { -kvfree (p); - *pages =NULL ; +if (unlikely (res <=0 ))return res; - } - *pages = p;return (res == n ? len : res *PAGE_SIZE ) - *start; }if (iov_iter_is_bvec (i)) {
@@ -1531,6 +1521,22 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i,return iter_xarray_get_pages_alloc (i, pages, maxsize, start);return -EFAULT ; } + +ssize_t iov_iter_get_pages_alloc (struct iov_iter *i, +struct page ***pages,size_t maxsize, +size_t *start) +{ +ssize_t len; + + *pages =NULL ; + + len =__iov_iter_get_pages_alloc (i, pages, maxsize, start); +if (len <=0 ) { +kvfree (*pages); + *pages =NULL ; + } +return len; +}EXPORT_SYMBOL (iov_iter_get_pages_alloc); size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 31/44 iov_iter: first_{iovec,bvec}_segment() - simplify a bit
In reply to Al ViroWe return length + offset in page via *size. Don't bother - the caller can do that arithmetics just as well; just report the length to it.
--- lib/iov_iter.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
lib/iov_iter.c +11 -11
@@ -1310,7 +1310,7 @@ static unsigned long found_ubuf_segment(unsigned long addr,size_t len, size_t *size, size_t *start) { - len += (*start = addr % PAGE_SIZE); + *start = addr %PAGE_SIZE ; *size = len;return addr &PAGE_MASK ; }
@@ -1354,7 +1354,7 @@ static struct page *first_bvec_segment(const struct iov_iter *i, len = maxsize; skip += i->bvec->bv_offset; page = i->bvec->bv_page + skip /PAGE_SIZE ; - len += (*start = skip %PAGE_SIZE ); + *start = skip %PAGE_SIZE ; *size = len;return page; }
@@ -1383,9 +1383,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, gup_flags |=FOLL_NOFAULT ; addr =first_iovec_segment (i, &len, start, maxsize); -if (len > maxpages *PAGE_SIZE ) - len = maxpages *PAGE_SIZE ; - n =DIV_ROUND_UP (len,PAGE_SIZE ); + n =DIV_ROUND_UP (len + *start,PAGE_SIZE ); +if (n > maxpages) + n = maxpages;if (!*pages) { *pages =get_pages_array (n);if (!*pages)
@@ -1394,25 +1394,25 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res; -return (res == n ? len : res *PAGE_SIZE ) - *start; +return min_t (size_t, len, res *PAGE_SIZE - *start); }if (iov_iter_is_bvec (i)) {struct page **p;struct page *page; page =first_bvec_segment (i, &len, start, maxsize); -if (len > maxpages *PAGE_SIZE ) - len = maxpages *PAGE_SIZE ; - n =DIV_ROUND_UP (len,PAGE_SIZE ); + n =DIV_ROUND_UP (len + *start,PAGE_SIZE ); +if (n > maxpages) + n = maxpages; p = *pages;if (!p) { *pages = p =get_pages_array (n);if (!p)return -ENOMEM ; } -while (n--) +for (int k =0 ; k < n; k++)get_page (*p++ = page++); -return len - *start; +return min_t (size_t, len, n *PAGE_SIZE - *start); }if (iov_iter_is_pipe (i))return pipe_get_pages (i, pages, maxsize, maxpages, start);
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 21/44 ITER_PIPE: cache the type of last buffer
In reply to Al ViroWe often need to find whether the last buffer is anon or not, and currently it's rather clumsy:
check if ->iov_offset is non-zero (i.e. that pipe is not empty) if so, get the corresponding pipe_buffer and check its ->ops if it's &default_pipe_buf_ops, we have an anon buffer.
Let's replace the use of ->iov_offset (which is nowhere near similar to its role for other flavours) with signed field (->last_offset), with the following rules:
empty, no buffers occupied: 0
anon, with bytes up to N-1 filled: N
zero-copy, with bytes up to N-1 filled: -N
That way abs(i->last_offset) is equal to what used to be in i->iov_offset and empty vs. anon vs. zero-copy can be distinguished by the sign of i->last_offset.
Checks for "should we extend the last buffer or should we start
a new one?" become easier to follow that way.
Note that most of the operations can only be done in a sane
state - i.e. when the pipe has nothing past the current position of iterator. About the only thing that could be done outside of that state is iov_iter_advance(), which transitions to the sane state by truncating the pipe. There are only two cases where we leave the sane state:
1) iov_iter_get_pages()/iov_iter_get_pages_alloc(). Will be
dealt with later, when we make get_pages advancing - the callers are actually happier that way.
2) iov_iter copied, then something is put into the copy. Since
they share the underlying pipe, the original gets behind. When we decide that we are done with the copy (original is not usable until then) we advance the original. direct_io used to be done that way; nowadays it operates on the original and we do iov_iter_revert() to discard the excessive data. At the moment there's nothing in the kernel that could do that to ITER_PIPE iterators, so this reason for insane state is theoretical right now.
--- include/linux/uio.h | 5 +++- lib/iov_iter.c | 72 ++++++++++++++++++++++----------------------- 2 files changed, 40 insertions(+), 37 deletions(-)
include/linux/uio.h +4 -1
@@ -40,7 +40,10 @@ struct iov_iter {bool nofault;bool data_source;bool user_backed; -size_t iov_offset; +union { +size_t iov_offset; +int last_offset; + };size_t count;union {const struct iovec *iov;
lib/iov_iter.c +36 -36
173 lines not on this page · raw message
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 35/44 iov_iter: saner helper for page array allocation
In reply to Al ViroAll call sites of get_pages_array() are essenitally identical now. Replace with common helper...
Returns number of slots available in resulting array or 0 on OOM; it's up to the caller to make sure it doesn't ask to zero-entry array (i.e. neither maxpages nor size are allowed to be zero).
--- lib/iov_iter.c | 77 +++++++++++++++++++++----------------------------- 1 file changed, 32 insertions(+), 45 deletions(-)
lib/iov_iter.c +32 -45
@@ -1187,9 +1187,20 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i) }EXPORT_SYMBOL (iov_iter_gap_alignment); -static struct page **get_pages_array (size_t n) +static int want_pages_array (struct page ***res,size_t size, +size_t start,unsigned int maxpages) { -return kvmalloc_array (n,sizeof (struct page *),GFP_KERNEL ); +unsigned int count =DIV_ROUND_UP (size + start,PAGE_SIZE ); + +if (count > maxpages) + count = maxpages; +WARN_ON (!count);// caller should've prevented that +if (!*res) { + *res =kvmalloc_array (count,sizeof (struct page *),GFP_KERNEL ); +if (!*res) +return 0 ; + } +return count; } static ssize_t pipe_get_pages(struct iov_iter *i,
@@ -1197,27 +1208,20 @@ static ssize_t pipe_get_pages(struct iov_iter *i, size_t *start) {struct pipe_inode_info *pipe = i->pipe; -unsigned int npages, off; +unsigned int npages, off, count;struct page **p;ssize_t left; -int count;if (!sanity (i))return -EFAULT ; *start = off =pipe_npages (i, &npages); - count =DIV_ROUND_UP (maxsize + off,PAGE_SIZE ); -if (count > npages) - count = npages; -if (count > maxpages) - count = maxpages; +if (!npages) +return -EFAULT ; + count =want_pages_array (pages, maxsize, off,min (npages, maxpages)); +if (!count) +return -ENOMEM ; p = *pages; -if (!p) { - *pages = p =get_pages_array (count); -if (!p) -return -ENOMEM ; - } - left = maxsize; npages =0 ;if (off) {
@@ -1280,9 +1284,8 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned maxpages, size_t *_start_offset) { - unsigned nr, offset; -pgoff_t index, count; -size_t size = maxsize; + unsigned nr, offset, count; +pgoff_t index;loff_t pos; pos = i->xarray_start + i->iov_offset;
@@ -1290,16 +1293,9 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, offset = pos & ~PAGE_MASK ; *_start_offset = offset; - count =DIV_ROUND_UP (size + offset,PAGE_SIZE ); -if (count > maxpages) - count = maxpages; - -if (!*pages) { - *pages =get_pages_array (count); -if (!*pages) -return -ENOMEM ; - } - + count =want_pages_array (pages, maxsize, offset, maxpages); +if (!count) +return -ENOMEM ; nr =iter_xarray_populate_pages (*pages, i->xarray, index, count);if (nr ==0 )return 0 ;
@@ -1348,7 +1344,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start) { - int n, res; + unsigned int n;if (maxsize > i->count) maxsize = i->count;
@@ -1360,6 +1356,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;unsigned long addr; +int res;if (iov_iter_rw (i) !=WRITE ) gup_flags |=FOLL_WRITE ;
@@ -1369,14 +1366,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, addr =first_iovec_segment (i, &maxsize); *start = addr %PAGE_SIZE ; addr &=PAGE_MASK ; - n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE ); -if (n > maxpages) - n = maxpages; -if (!*pages) { - *pages =get_pages_array (n); -if (!*pages) -return -ENOMEM ; - } + n =want_pages_array (pages, maxsize, *start, maxpages); +if (!n) +return -ENOMEM ; res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res;
@@ -1387,15 +1379,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,struct page *page; page =first_bvec_segment (i, &maxsize, start); - n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE ); -if (n > maxpages) - n = maxpages; + n =want_pages_array (pages, maxsize, *start, maxpages); +if (!n) +return -ENOMEM ; p = *pages; -if (!p) { - *pages = p =get_pages_array (n); -if (!p) -return -ENOMEM ; - }for (int k =0 ; k < n; k++)get_page (*p++ = page++);return min_t (size_t, maxsize, n *PAGE_SIZE - *start);
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 36/44 iov_iter: advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al ViroMost of the users immediately follow successful iov_iter_get_pages() with advancing by the amount it had returned.
Provide inline wrappers doing that, convert trivial open-coded uses of those.
BTW, iov_iter_get_pages() never returns more than it had been asked to; such checks in cifs ought to be removed someday...
--- drivers/vhost/scsi.c | 4 +--- fs/ceph/file.c | 3 +-- fs/cifs/file.c | 6 ++---- fs/cifs/misc.c | 3 +-- fs/direct-io.c | 3 +-- fs/fuse/dev.c | 3 +-- fs/fuse/file.c | 3 +-- fs/nfs/direct.c | 6 ++---- include/linux/uio.h | 20 ++++++++++++++++++++ net/core/datagram.c | 3 +-- net/core/skmsg.c | 3 +-- net/rds/message.c | 3 +-- net/tls/tls_sw.c | 4 +--- 13 files changed, 34 insertions(+), 30 deletions(-)
drivers/vhost/scsi.c +1 -3
@@ -643,14 +643,12 @@ vhost_scsi_map_to_sgl(struct vhost_scsi_cmd *cmd,size_t offset;unsigned int npages =0 ; - bytes =iov_iter_get_pages (iter, pages,LONG_MAX , + bytes =iov_iter_get_pages2 (iter, pages,LONG_MAX ,VHOST_SCSI_PREALLOC_UPAGES , &offset);/* No pages were pinned */ if (bytes <=0 )return bytes <0 ? bytes : -EFAULT ; -iov_iter_advance (iter, bytes); -while (bytes) {unsigned n =min_t (unsigned,PAGE_SIZE - offset, bytes);sg_set_page (sg++, pages[npages++], n, offset);
fs/ceph/file.c +1 -2
@@ -95,12 +95,11 @@ static ssize_t __iter_get_bvecs(struct iov_iter *iter, size_t maxsize,size_t start;int idx =0 ; - bytes =iov_iter_get_pages (iter, pages, maxsize - size, + bytes =iov_iter_get_pages2 (iter, pages, maxsize - size,ITER_GET_BVECS_PAGES , &start);if (bytes <0 )return size ?: bytes; -iov_iter_advance (iter, bytes); size += bytes;for ( ; bytes; idx++, bvec_idx++) {
fs/cifs/file.c +2 -4
@@ -3022,7 +3022,7 @@ cifs_write_from_iter(loff_t offset, size_t len, struct iov_iter *from,if (ctx->direct_io) {ssize_t result; - result =iov_iter_get_pages_alloc ( + result =iov_iter_get_pages_alloc2 ( from, &pagevec, cur_len, &start);if (result <0 ) { cifs_dbg(VFS,
@@ -3036,7 +3036,6 @@ cifs_write_from_iter(loff_t offset, size_t len, struct iov_iter *from,break ; } cur_len = (size_t )result; -iov_iter_advance (from, cur_len); nr_pages = (cur_len + start +PAGE_SIZE -1 ) /PAGE_SIZE ;
@@ -3758,7 +3757,7 @@ cifs_send_async_read(loff_t offset, size_t len, struct cifsFileInfo *open_file,if (ctx->direct_io) {ssize_t result; - result =iov_iter_get_pages_alloc ( + result =iov_iter_get_pages_alloc2 ( &direct_iov, &pagevec, cur_len, &start);if (result <0 ) {
@@ -3774,7 +3773,6 @@ cifs_send_async_read(loff_t offset, size_t len, struct cifsFileInfo *open_file,break ; } cur_len = (size_t )result; -iov_iter_advance (&direct_iov, cur_len); rdata =cifs_readdata_direct_alloc ( pagevec, cifs_uncached_readv_complete);
fs/cifs/misc.c +1 -2
@@ -1022,7 +1022,7 @@ setup_aio_ctx_iter(struct cifs_aio_ctx *ctx, struct iov_iter *iter, int rw) saved_len = count;while (count && npages < max_pages) { - rc =iov_iter_get_pages (iter, pages, count, max_pages, &start); + rc =iov_iter_get_pages2 (iter, pages, count, max_pages, &start);if (rc <0 ) {cifs_dbg (VFS ,"Couldn't get user pages (rc=%zd)\n" , rc);break ;
@@ -1034,7 +1034,6 @@ setup_aio_ctx_iter(struct cifs_aio_ctx *ctx, struct iov_iter *iter, int rw)break ; } -iov_iter_advance (iter, rc); count -= rc; rc += start; cur_npages =DIV_ROUND_UP (rc,PAGE_SIZE );
fs/direct-io.c +1 -2
@@ -169,7 +169,7 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) {ssize_t ret; - ret =iov_iter_get_pages (sdio->iter, dio->pages,LONG_MAX ,DIO_PAGES , + ret =iov_iter_get_pages2 (sdio->iter, dio->pages,LONG_MAX ,DIO_PAGES , &sdio->from);if (ret <0 && sdio->blocks_available && (dio->op ==REQ_OP_WRITE )) {
@@ -191,7 +191,6 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) }if (ret >=0 ) { -iov_iter_advance (sdio->iter, ret); ret += sdio->from; sdio->head =0 ; sdio->tail = (ret +PAGE_SIZE -1 ) /PAGE_SIZE ;
fs/fuse/dev.c +1 -2
@@ -730,14 +730,13 @@ static int fuse_copy_fill(struct fuse_copy_state *cs) } }else {size_t off; - err =iov_iter_get_pages (cs->iter, &page,PAGE_SIZE ,1 , &off); + err =iov_iter_get_pages2 (cs->iter, &page,PAGE_SIZE ,1 , &off);if (err <0 )return err;BUG_ON (!err); cs->len = err; cs->offset = off; cs->pg = page; -iov_iter_advance (cs->iter, err); }return lock_request (cs->req);
fs/fuse/file.c +1 -2
@@ -1401,14 +1401,13 @@ static int fuse_get_user_pages(struct fuse_args_pages *ap, struct iov_iter *ii,while (nbytes < *nbytesp && ap->num_pages < max_pages) {unsigned npages;size_t start; - ret =iov_iter_get_pages (ii, &ap->pages[ap->num_pages], + ret =iov_iter_get_pages2 (ii, &ap->pages[ap->num_pages], *nbytesp - nbytes, max_pages - ap->num_pages, &start);if (ret <0 )break ; -iov_iter_advance (ii, ret); nbytes += ret; ret += start;
fs/nfs/direct.c +2 -4
@@ -364,13 +364,12 @@ static ssize_t nfs_direct_read_schedule_iovec(struct nfs_direct_req *dreq,size_t pgbase;unsigned npages, i; - result =iov_iter_get_pages_alloc (iter, &pagevec, + result =iov_iter_get_pages_alloc2 (iter, &pagevec, rsize, &pgbase);if (result <0 )break ; bytes = result; -iov_iter_advance (iter, bytes); npages = (result + pgbase +PAGE_SIZE -1 ) /PAGE_SIZE ;for (i =0 ; i < npages; i++) {struct nfs_page *req;
@@ -812,13 +811,12 @@ static ssize_t nfs_direct_write_schedule_iovec(struct nfs_direct_req *dreq,size_t pgbase;unsigned npages, i; - result =iov_iter_get_pages_alloc (iter, &pagevec, + result =iov_iter_get_pages_alloc2 (iter, &pagevec, wsize, &pgbase);if (result <0 )break ; bytes = result; -iov_iter_advance (iter, bytes); npages = (result + pgbase +PAGE_SIZE -1 ) /PAGE_SIZE ;for (i =0 ; i < npages; i++) {struct nfs_page *req;
include/linux/uio.h +20 -0
@@ -349,4 +349,24 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +static inline ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages, +size_t maxsize,unsigned maxpages,size_t *start) +{ +ssize_t res =iov_iter_get_pages (i, pages, maxsize, maxpages, start); + +if (res >=0 ) +iov_iter_advance (i, res); +return res; +} + +static inline ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages, +size_t maxsize,size_t *start) +{ +ssize_t res =iov_iter_get_pages_alloc (i, pages, maxsize, start); + +if (res >=0 ) +iov_iter_advance (i, res); +return res; +} +#endif
net/core/datagram.c +1 -2
@@ -629,12 +629,11 @@ int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb,if (frag ==MAX_SKB_FRAGS )return -EMSGSIZE ; - copied =iov_iter_get_pages (from, pages, length, + copied =iov_iter_get_pages2 (from, pages, length,MAX_SKB_FRAGS - frag, &start);if (copied <0 )return -EFAULT ; -iov_iter_advance (from, copied); length -= copied; truesize =PAGE_ALIGN (copied + start);
net/core/skmsg.c +1 -2
@@ -324,14 +324,13 @@ int sk_msg_zerocopy_from_iter(struct sock *sk, struct iov_iter *from, goto out; } - copied =iov_iter_get_pages (from, pages, bytes, maxpages, + copied =iov_iter_get_pages2 (from, pages, bytes, maxpages, &offset);if (copied <=0 ) { ret = -EFAULT ; goto out; } -iov_iter_advance (from, copied); bytes -= copied; msg->sg.size += copied;
net/rds/message.c +1 -2
@@ -391,7 +391,7 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter *size_t start;ssize_t copied; - copied =iov_iter_get_pages (from, &pages,PAGE_SIZE , + copied =iov_iter_get_pages2 (from, &pages,PAGE_SIZE ,1 , &start);if (copied <0 ) {struct mmpin *mmp;
@@ -405,7 +405,6 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * goto err; } total_copied += copied; -iov_iter_advance (from, copied); length -= copied;sg_set_page (sg, pages, copied, start); rm->data.op_nents++;
net/tls/tls_sw.c +1 -3
@@ -1361,7 +1361,7 @@ static int tls_setup_from_iter(struct iov_iter *from, rc = -EFAULT ; goto out; } - copied =iov_iter_get_pages (from, pages, + copied =iov_iter_get_pages2 (from, pages, length, maxpages, &offset);if (copied <=0 ) {
@@ -1369,8 +1369,6 @@ static int tls_setup_from_iter(struct iov_iter *from, goto out; } -iov_iter_advance (from, copied); - length -= copied; size += copied;while (copied) {
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 41/44 ceph: switch the last caller of iov_iter_get_pages_alloc()
In reply to Al Virohere nothing even looks at the iov_iter after the call, so we couldn't care less whether it advances or not.
--- fs/ceph/addr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ceph/addr.c +1 -1
@@ -329,7 +329,7 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)dout ("%s: pos=%llu orig_len=%zu len=%llu\n" , __func__, subreq->start, subreq->len, len);iov_iter_xarray (&iter,READ , &rreq->mapping->i_pages, subreq->start, len); - err =iov_iter_get_pages_alloc (&iter, &pages, len, &page_off); + err =iov_iter_get_pages_alloc2 (&iter, &pages, len, &page_off);if (err <0 ) {dout ("%s: iov_ter_get_pages_alloc returned %d\n" , __func__, err); goto out;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 37/44 block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al Viro... doing revert if we end up not using some pages
--- block/bio.c | 15 ++++++--------- block/blk-map.c | 7 ++++--- 2 files changed, 10 insertions(+), 12 deletions(-)
block/bio.c +6 -9
@@ -1190,7 +1190,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)BUILD_BUG_ON (PAGE_PTRS_PER_BVEC <2 ); pages += entries_left * (PAGE_PTRS_PER_BVEC -1 ); - size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); + size =iov_iter_get_pages2 (iter, pages,LONG_MAX , nr_pages, &offset);if (unlikely (size <=0 ))return size ? size : -EFAULT ;
@@ -1205,6 +1205,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) }else {if (WARN_ON_ONCE (bio_full (bio, len))) {bio_put_pages (pages + i, left, offset); +iov_iter_revert (iter, left);return -EINVAL ; }__bio_add_page (bio, page, len, offset);
@@ -1212,7 +1213,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) offset =0 ; } -iov_iter_advance (iter, size);return 0 ; }
@@ -1227,7 +1227,6 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)ssize_t size, left;unsigned len, i;size_t offset; -int ret =0 ;if (WARN_ON_ONCE (!max_append_sectors))return 0 ;
@@ -1240,7 +1239,7 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)BUILD_BUG_ON (PAGE_PTRS_PER_BVEC <2 ); pages += entries_left * (PAGE_PTRS_PER_BVEC -1 ); - size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); + size =iov_iter_get_pages2 (iter, pages,LONG_MAX , nr_pages, &offset);if (unlikely (size <=0 ))return size ? size : -EFAULT ;
@@ -1252,16 +1251,14 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)if (bio_add_hw_page (q, bio, page, len, offset, max_append_sectors, &same_page) != len) {bio_put_pages (pages + i, left, offset); - ret = -EINVAL ; -break ; +iov_iter_revert (iter, left); +return -EINVAL ; }if (same_page)put_page (page); offset =0 ; } - -iov_iter_advance (iter, size - left); -return ret; +return 0 ; }/**
block/blk-map.c +4 -3
@@ -254,7 +254,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,size_t offs, added =0 ;int npages; - bytes =iov_iter_get_pages_alloc (iter, &pages,LONG_MAX , &offs); + bytes =iov_iter_get_pages_alloc2 (iter, &pages,LONG_MAX , &offs);if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ; goto out_unmap;
@@ -284,7 +284,6 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, bytes -= n; offs =0 ; } -iov_iter_advance (iter, added); }/* * release the pages we didn't map into the bio, if any
@@ -293,8 +292,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,put_page (pages[j++]);kvfree (pages);/* couldn't stuff something into bio? */ -if (bytes) +if (bytes) { +iov_iter_revert (iter, bytes);break ; + } } ret =blk_rq_append_bio (rq, bio);
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 38/44 iter_to_pipe(): switch to advancing variant of iov_iter_get_pages()
In reply to Al Viro... and untangle the cleanup on failure to add into pipe.
--- fs/splice.c | 47 ++++++++++++++++++++++++----------------------- 1 file changed, 24 insertions(+), 23 deletions(-)
fs/splice.c +24 -23
@@ -1160,39 +1160,40 @@ static int iter_to_pipe(struct iov_iter *from, };size_t total =0 ;int ret =0 ; -bool failed = false; -while (iov_iter_count (from) && !failed) { +while (iov_iter_count (from)) {struct page *pages[16 ]; -ssize_t copied; +ssize_t left;size_t start; -int n; +int i, n; - copied =iov_iter_get_pages (from, pages, ~0UL ,16 , &start); -if (copied <=0 ) { - ret = copied; + left =iov_iter_get_pages2 (from, pages, ~0UL ,16 , &start); +if (left <=0 ) { + ret = left;break ; } -for (n =0 ; copied; n++, start =0 ) { -int size =min_t (int, copied,PAGE_SIZE - start); -if (!failed) { - buf.page = pages[n]; - buf.offset = start; - buf.len = size; - ret =add_to_pipe (pipe, &buf); -if (unlikely (ret <0 )) { - failed = true; - }else { -iov_iter_advance (from, ret); - total += ret; - } - }else { -put_page (pages[n]); + n =DIV_ROUND_UP (left + start,PAGE_SIZE ); +for (i =0 ; i < n; i++) { +int size =min_t (int, left,PAGE_SIZE - start); + + buf.page = pages[i]; + buf.offset = start; + buf.len = size; + ret =add_to_pipe (pipe, &buf); +if (unlikely (ret <0 )) { +iov_iter_revert (from, left); +// this one got dropped by add_to_pipe() +while (++i < n) +put_page (pages[i]); + goto out; } - copied -= size; + total += ret; + left -= size; + start =0 ; } } +out:return total ? total : ret; }
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 43/44 pipe_get_pages(): switch to append_pipe()
In reply to Al Vironow that we are advancing the iterator, there's no need to treat the first page separately - just call append_pipe() in a loop.
--- lib/iov_iter.c | 36 ++++++++---------------------------- 1 file changed, 8 insertions(+), 28 deletions(-)
lib/iov_iter.c +8 -28
@@ -1207,10 +1207,10 @@ static ssize_t pipe_get_pages(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned maxpages, size_t *start) { - struct pipe_inode_info *pipe = i->pipe; -unsigned int npages, off, count; + unsigned int npages, count;struct page **p;ssize_t left; +size_t off;if (!sanity (i))return -EFAULT ;
@@ -1222,38 +1222,18 @@ static ssize_t pipe_get_pages(struct iov_iter *i,if (!count)return -ENOMEM ; p = *pages; - left = maxsize; - npages =0 ; -if (off) { -struct pipe_buffer *buf =pipe_buf (pipe, pipe->head -1 ); - -get_page (*p++ = buf->page); - left -=PAGE_SIZE - off; -if (left <=0 ) { - buf->len += maxsize; -iov_iter_advance (i, maxsize); -return maxsize; - } - buf->len =PAGE_SIZE ; - npages =1 ; - } -for ( ; npages < count; npages++) { -struct page *page; -unsigned int size =min_t (ssize_t, left,PAGE_SIZE ); - -if (pipe_full (pipe->head, pipe->tail, pipe->max_usage)) -break ; - page =push_anon (pipe, size); +for (npages =0 , left = maxsize ; npages < count; npages++) { +struct page *page =append_pipe (i, left, &off);if (!page)break ;get_page (*p++ = page); - left -= size; +if (left <=PAGE_SIZE - off) +return maxsize; + left -=PAGE_SIZE - off; }if (!npages)return -EFAULT ; - maxsize -= left; -iov_iter_advance (i, maxsize); -return maxsize; +return maxsize - left; } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 40/44 9p: convert to advancing variant of iov_iter_get_pages_alloc()
In reply to Al Virothat one is somewhat clumsier than usual and needs serious testing.
--- net/9p/client.c | 39 +++++++++++++++++++++++---------------- net/9p/protocol.c | 3 +-- net/9p/trans_virtio.c | 3 ++- 3 files changed, 26 insertions(+), 19 deletions(-)
net/9p/client.c +23 -16
@@ -1491,7 +1491,7 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to,struct p9_client *clnt = fid->clnt;struct p9_req_t *req;int count =iov_iter_count (to); -int rsize, non_zc =0 ; +int rsize, received, non_zc =0 ;char *dataptr; *err =0 ;
@@ -1520,36 +1520,40 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to, }if (IS_ERR (req)) { *err =PTR_ERR (req); +if (!non_zc) +iov_iter_revert (to, count -iov_iter_count (to));return 0 ; } *err =p9pdu_readf (&req->rc, clnt->proto_version, -"D" , &count, &dataptr); +"D" , &received, &dataptr);if (*err) { +if (!non_zc) +iov_iter_revert (to, count -iov_iter_count (to));trace_9p_protocol_dump (clnt, &req->rc);p9_tag_remove (clnt, req);return 0 ; } -if (rsize < count) { -pr_err ("bogus RREAD count (%d > %d)\n" , count, rsize); - count = rsize; +if (rsize < received) { +pr_err ("bogus RREAD count (%d > %d)\n" , received, rsize); + received = rsize; }p9_debug (P9_DEBUG_9P ,"<<< RREAD count %d\n" , count);if (non_zc) { -int n =copy_to_iter (dataptr, count, to); +int n =copy_to_iter (dataptr, received, to); -if (n != count) { +if (n != received) { *err = -EFAULT ;p9_tag_remove (clnt, req);return n; } }else { -iov_iter_advance (to, count); +iov_iter_revert (to, count - received -iov_iter_count (to)); }p9_tag_remove (clnt, req); -return count; +return received; }EXPORT_SYMBOL (p9_client_read_once);
@@ -1567,6 +1571,7 @@ p9_client_write(struct p9_fid *fid, u64 offset, struct iov_iter *from, int *err)while (iov_iter_count (from)) {int count =iov_iter_count (from);int rsize = fid->iounit; +int written;if (!rsize || rsize > clnt->msize -P9_IOHDRSZ ) rsize = clnt->msize -P9_IOHDRSZ ;
@@ -1584,27 +1589,29 @@ p9_client_write(struct p9_fid *fid, u64 offset, struct iov_iter *from, int *err) offset, rsize, from); }if (IS_ERR (req)) { +iov_iter_revert (from, count -iov_iter_count (from)); *err =PTR_ERR (req);break ; } - *err =p9pdu_readf (&req->rc, clnt->proto_version,"d" , &count); + *err =p9pdu_readf (&req->rc, clnt->proto_version,"d" , &written);if (*err) { +iov_iter_revert (from, count -iov_iter_count (from));trace_9p_protocol_dump (clnt, &req->rc);p9_tag_remove (clnt, req);break ; } -if (rsize < count) { -pr_err ("bogus RWRITE count (%d > %d)\n" , count, rsize); - count = rsize; +if (rsize < written) { +pr_err ("bogus RWRITE count (%d > %d)\n" , written, rsize); + written = rsize; }p9_debug (P9_DEBUG_9P ,"<<< RWRITE count %d\n" , count);p9_tag_remove (clnt, req); -iov_iter_advance (from, count); - total += count; - offset += count; +iov_iter_revert (from, count - written -iov_iter_count (from)); + total += written; + offset += written; }return total; }
net/9p/protocol.c +1 -2
@@ -63,9 +63,8 @@ static size_t pdu_write_u(struct p9_fcall *pdu, struct iov_iter *from, size_t size) { size_t len = min(pdu->capacity - pdu->size, size); -struct iov_iter i = *from; -if (!copy_from_iter_full (&pdu->sdata[pdu->size], len, &i)) +if (!copy_from_iter_full (&pdu->sdata[pdu->size], len, from)) len =0 ; pdu->size += len;
net/9p/trans_virtio.c +2 -1
@@ -331,7 +331,7 @@ static int p9_get_mapped_pages(struct virtio_chan *chan,if (err == -ERESTARTSYS )return err; } - n =iov_iter_get_pages_alloc (data, pages, count, offs); + n =iov_iter_get_pages_alloc2 (data, pages, count, offs);if (n <0 )return n; *need_drop =1 ;
@@ -373,6 +373,7 @@ static int p9_get_mapped_pages(struct virtio_chan *chan, (*pages)[index] =kmap_to_page (p); p +=PAGE_SIZE ; } +iov_iter_advance (data, len);return len; } }
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 44/44 expand those iov_iter_advance()...
In reply to Al Viro--- lib/iov_iter.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
lib/iov_iter.c +9 -2
@@ -1284,7 +1284,8 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,return 0 ; maxsize =min_t (size_t, nr *PAGE_SIZE - offset, maxsize); -iov_iter_advance (i, maxsize); + i->iov_offset += maxsize; + i->count -= maxsize;return maxsize; }
@@ -1373,7 +1374,13 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,for (int k =0 ; k < n; k++)get_page (p[k] = page + k); maxsize =min_t (size_t, maxsize, n *PAGE_SIZE - *start); -iov_iter_advance (i, maxsize); + i->count -= maxsize; + i->iov_offset += maxsize; +if (i->iov_offset == i->bvec->bv_len) { + i->iov_offset =0 ; + i->bvec++; + i->nr_segs--; + }return maxsize; }if (iov_iter_is_pipe (i))
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 39/44 af_alg_make_sg(): switch to advancing variant of iov_iter_get_pages()
In reply to Al Viro... and adjust the callers
--- crypto/af_alg.c | 3 +-- crypto/algif_hash.c | 5 +++-- 2 files changed, 4 insertions(+), 4 deletions(-)
crypto/af_alg.c +1 -2
@@ -404,7 +404,7 @@ int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len)ssize_t n;int npages, i; - n =iov_iter_get_pages (iter, sgl->pages, len,ALG_MAX_PAGES , &off); + n =iov_iter_get_pages2 (iter, sgl->pages, len,ALG_MAX_PAGES , &off);if (n <0 )return n;
@@ -1191,7 +1191,6 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, len += err;atomic_add (err, &ctx->rcvused); rsgl->sg_num_bytes = err; -iov_iter_advance (&msg->msg_iter, err); } *outlen = len;
crypto/algif_hash.c +3 -2
@@ -102,11 +102,12 @@ static int hash_sendmsg(struct socket *sock, struct msghdr *msg, err =crypto_wait_req (crypto_ahash_update (&ctx->req), &ctx->wait);af_alg_free_sg (&ctx->sgl); -if (err) +if (err) { +iov_iter_revert (&msg->msg_iter, len); goto unlock; + } copied += len; -iov_iter_advance (&msg->msg_iter, len); } err =0 ;
Signature · 1 line
2.30.2
Al Viro
maintainerPATCH 42/44 get rid of non-advancing variants
In reply to Al Viromechanical change; will be further massaged in subsequent commits
--- include/linux/uio.h | 24 ++---------------------- lib/iov_iter.c | 27 ++++++++++++++++++--------- 2 files changed, 20 insertions(+), 31 deletions(-)
include/linux/uio.h +2 -22
@@ -245,9 +245,9 @@ void iov_iter_pipe(struct iov_iter *i, unsigned int direction, struct pipe_inodevoid iov_iter_discard (struct iov_iter *i,unsigned int direction,size_t count);void iov_iter_xarray (struct iov_iter *i,unsigned int direction,struct xarray *xarray,loff_t start,size_t count); -ssize_t iov_iter_get_pages (struct iov_iter *i,struct page **pages, +ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start); -ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages, +ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start);int iov_iter_npages (const struct iov_iter *i,int maxpages);void iov_iter_restore (struct iov_iter *i,struct iov_iter_state *state);
@@ -349,24 +349,4 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } -static inline ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages, -size_t maxsize,unsigned maxpages,size_t *start) -{ -ssize_t res =iov_iter_get_pages (i, pages, maxsize, maxpages, start); - -if (res >=0 ) -iov_iter_advance (i, res); -return res; -} - -static inline ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages, -size_t maxsize,size_t *start) -{ -ssize_t res =iov_iter_get_pages_alloc (i, pages, maxsize, start); - -if (res >=0 ) -iov_iter_advance (i, res); -return res; -} -#endif
lib/iov_iter.c +18 -9
@@ -1231,6 +1231,7 @@ static ssize_t pipe_get_pages(struct iov_iter *i, left -=PAGE_SIZE - off;if (left <=0 ) { buf->len += maxsize; +iov_iter_advance (i, maxsize);return maxsize; } buf->len =PAGE_SIZE ;
@@ -1250,7 +1251,9 @@ static ssize_t pipe_get_pages(struct iov_iter *i, }if (!npages)return -EFAULT ; -return maxsize - left; + maxsize -= left; +iov_iter_advance (i, maxsize); +return maxsize; } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
@@ -1300,7 +1303,9 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,if (nr ==0 )return 0 ; -return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); + maxsize =min_t (size_t, nr *PAGE_SIZE - offset, maxsize); +iov_iter_advance (i, maxsize); +return maxsize; }/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */
@@ -1372,7 +1377,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res; -return min_t (size_t, maxsize, res *PAGE_SIZE - *start); + maxsize =min_t (size_t, maxsize, res *PAGE_SIZE - *start); +iov_iter_advance (i, maxsize); +return maxsize; }if (iov_iter_is_bvec (i)) {struct page **p;
@@ -1384,8 +1391,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,return -ENOMEM ; p = *pages;for (int k =0 ; k < n; k++) -get_page (*p++ = page++); -return min_t (size_t, maxsize, n *PAGE_SIZE - *start); +get_page (p[k] = page + k); + maxsize =min_t (size_t, maxsize, n *PAGE_SIZE - *start); +iov_iter_advance (i, maxsize); +return maxsize; }if (iov_iter_is_pipe (i))return pipe_get_pages (i, pages, maxsize, maxpages, start);
@@ -1395,7 +1404,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,return -EFAULT ; } -ssize_t iov_iter_get_pages (struct iov_iter *i, +ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start) {
@@ -1405,9 +1414,9 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,return __iov_iter_get_pages_alloc (i, &pages, maxsize, maxpages, start); } -EXPORT_SYMBOL (iov_iter_get_pages); +EXPORT_SYMBOL (iov_iter_get_pages2); -ssize_t iov_iter_get_pages_alloc (struct iov_iter *i, +ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start) {
@@ -1422,7 +1431,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, }return len; } -EXPORT_SYMBOL (iov_iter_get_pages_alloc); +EXPORT_SYMBOL (iov_iter_get_pages_alloc2);size_t csum_and_copy_from_iter (void *addr,size_t bytes,__wsum *csum,struct iov_iter *i)
Signature · 1 line
2.30.2
Quoted 10 lines · Al Viro
13/44: splice: stop abusing
iov_iter_advance()to flush a pipe
A really odd (ab)use ofiov_iter_advance()- in case of errorgeneric_file_splice_read()wants to free all pipe buffers ->read_iter() has produced. Yes, forcibly resetting ->head and ->iov_offset to original values and callingiov_iter_advance(i, 0)will triggerpipe_advance(), which will triggerpipe_truncate(), which will free buffers. Or we could just go ahead and free the same buffers;pipe_discard_from()does exactly that, noiov_iterstuff needs to be involved.
Can ->splice_read() and ->splice_write() be given pipe-class iov_iters rather than pipe_inode_info structs?
David
Al Viro
maintainerQuoted 13 lines · David Howells
Quoted 10 lines · Al Viro
13/44: splice: stop abusing
iov_iter_advance()to flush a pipe
A really odd (ab)use ofiov_iter_advance()- in case of errorgeneric_file_splice_read()wants to free all pipe buffers ->read_iter() has produced. Yes, forcibly resetting ->head and ->iov_offset to original values and callingiov_iter_advance(i, 0)will triggerpipe_advance(), which will triggerpipe_truncate(), which will free buffers. Or we could just go ahead and free the same buffers;pipe_discard_from()does exactly that, noiov_iterstuff needs to be involved.
Quoted 2 lines
Can ->splice_read() and ->splice_write() be given pipe-class
iov_itersrather thanpipe_inode_infostructs?
Huh?
First of all, ->splice_write() is given a pipe as data _source_, which makes ITER_PIPE completely irrelevant - those suckers are data destinations. What's more, it will unlock the pipe, wait and relock once somebody writes to that pipe. And ITER_PIPE very much relies upon the pipe being locked and staying locked.
As for ->splice_read()... We could create the iov_iter in the caller, but... Look at those callers:
pipe_lock(opipe); ret = wait_for_space(opipe, flags); if (!ret) ret = do_splice_to(in, offset, opipe, len, flags); pipe_unlock(opipe);
You can't set it up before wait_for_pace(), obviously - if there's no empty slots, what would that sucker work with? And you can't keep it past pipe_unlock(). So it's limited to do_splice_to(), which is a very thin wrapper for ->splice_read(). So what's the point?
Re: [PATCH 08/44] copy_page_{to,from}_iter(): switch iovec variants to generic
In reply to Al ViroQuoted 224 lines · Al Viro
we can do copyin/copyout under
kmap_local_page(); it shouldn't overflow the kmap stack - the maximal footprint increase only by one here.Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 191 ++----------------------------------------------- 1 file changed, 4 insertions(+), 187 deletions(-)lib/iov_iter.c +3
-186@@ -168,174 +168,6 @@ static int copyin(void *to, const void __user *from, size_t n)return n; } -static size_t copy_page_to_iter_iovec (struct page *page,size_t offset,size_t bytes, -struct iov_iter *i) -{ -size_t skip, copy, left, wanted; -const struct iovec *iov; -char __user *buf; -void *kaddr, *from; - -if (unlikely (bytes > i->count)) - bytes = i->count; - -if (unlikely (!bytes)) -return 0 ; - -might_fault (); - wanted = bytes; - iov = i->iov; - skip = i->iov_offset; - buf = iov->iov_base + skip; - copy =min (bytes, iov->iov_len - skip); - -if (IS_ENABLED (CONFIG_HIGHMEM ) && !fault_in_writeable (buf, copy)) { - kaddr =kmap_atomic (page); - from = kaddr + offset; - -/* first chunk, usually the only one */ - left =copyout (buf, from, copy); - copy -= left; - skip += copy; - from += copy; - bytes -= copy; - -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyout (buf, from, copy); - copy -= left; - skip = copy; - from += copy; - bytes -= copy; - } -if (likely (!bytes)) { -kunmap_atomic (kaddr); - goto done; - } - offset = from - kaddr; - buf += copy; -kunmap_atomic (kaddr); - copy =min (bytes, iov->iov_len - skip); - } -/* Too bad - revert to non-atomic kmap */ - - kaddr =kmap (page); - from = kaddr + offset; - left =copyout (buf, from, copy); - copy -= left; - skip += copy; - from += copy; - bytes -= copy; -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyout (buf, from, copy); - copy -= left; - skip = copy; - from += copy; - bytes -= copy; - } -kunmap (page); - -done: -if (skip == iov->iov_len) { - iov++; - skip =0 ; - } - i->count -= wanted - bytes; - i->nr_segs -= iov - i->iov; - i->iov = iov; - i->iov_offset = skip; -return wanted - bytes; -} - -static size_t copy_page_from_iter_iovec (struct page *page,size_t offset,size_t bytes, -struct iov_iter *i) -{ -size_t skip, copy, left, wanted; -const struct iovec *iov; -char __user *buf; -void *kaddr, *to; - -if (unlikely (bytes > i->count)) - bytes = i->count; - -if (unlikely (!bytes)) -return 0 ; - -might_fault (); - wanted = bytes; - iov = i->iov; - skip = i->iov_offset; - buf = iov->iov_base + skip; - copy =min (bytes, iov->iov_len - skip); - -if (IS_ENABLED (CONFIG_HIGHMEM ) && !fault_in_readable (buf, copy)) { - kaddr =kmap_atomic (page); - to = kaddr + offset; - -/* first chunk, usually the only one */ - left =copyin (to, buf, copy); - copy -= left; - skip += copy; - to += copy; - bytes -= copy; - -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyin (to, buf, copy); - copy -= left; - skip = copy; - to += copy; - bytes -= copy; - } -if (likely (!bytes)) { -kunmap_atomic (kaddr); - goto done; - } - offset = to - kaddr; - buf += copy; -kunmap_atomic (kaddr); - copy =min (bytes, iov->iov_len - skip); - } -/* Too bad - revert to non-atomic kmap */ - - kaddr =kmap (page); - to = kaddr + offset; - left =copyin (to, buf, copy); - copy -= left; - skip += copy; - to += copy; - bytes -= copy; -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyin (to, buf, copy); - copy -= left; - skip = copy; - to += copy; - bytes -= copy; - } -kunmap (page); - -done: -if (skip == iov->iov_len) { - iov++; - skip =0 ; - } - i->count -= wanted - bytes; - i->nr_segs -= iov - i->iov; - i->iov = iov; - i->iov_offset = skip; -return wanted - bytes; -} -#ifdef PIPE_PARANOIA static bool sanity (const struct iov_iter *i) {@@ -848,24 +680,14 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n)static size_t __copy_page_to_iter (struct page *page,size_t offset,size_t bytes,struct iov_iter *i) { -if (likely (iter_is_iovec (i))) -return copy_page_to_iter_iovec (page, offset, bytes, i); -if (iov_iter_is_bvec (i) ||iov_iter_is_kvec (i) ||iov_iter_is_xarray (i)) { +if (unlikely (iov_iter_is_pipe (i))) { +return copy_page_to_iter_pipe (page, offset, bytes, i); + }else {void *kaddr =kmap_local_page (page);size_t wanted =_copy_to_iter (kaddr + offset, bytes, i);kunmap_local (kaddr);return wanted; } -if (iov_iter_is_pipe (i)) -return copy_page_to_iter_pipe (page, offset, bytes, i); -if (unlikely (iov_iter_is_discard (i))) { -if (unlikely (i->count < bytes)) - bytes = i->count; - i->count -= bytes; -return bytes; - } -WARN_ON (1 ); -return 0 ; } size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes,@@ -896,17 +718,12 @@ EXPORT_SYMBOL(copy_page_to_iter);size_t copy_page_from_iter (struct page *page,size_t offset,size_t bytes,struct iov_iter *i) { -if (unlikely (!page_copy_sane (page, offset, bytes))) -return 0 ; -if (likely (iter_is_iovec (i))) -return copy_page_from_iter_iovec (page, offset, bytes, i); -if (iov_iter_is_bvec (i) ||iov_iter_is_kvec (i) ||iov_iter_is_xarray (i)) {
lib/iov_iter.c EXPORT_SYMBOL(copy_page_to_iter); · line 729
Quoted 10 lines
lib/iov_iter.c +1
-1+if (page_copy_sane (page, offset, bytes)) {void *kaddr =kmap_local_page (page);size_t wanted =_copy_from_iter (kaddr + offset, bytes, i);kunmap_local (kaddr);return wanted; } -WARN_ON (1 );return 0 ; }EXPORT_SYMBOL (copy_page_from_iter);
Love it.
Jeff Layton
maintainerRe: [PATCH 09/44] new iov_iter flavour - ITER_UBUF
In reply to Al ViroQuoted 218 lines · Al Viro
Equivalent of single-segment iovec. Initialized by
iov_iter_ubuf(), checked for byiter_is_ubuf(), otherwise behaves likeITER_IOVECones.We are going to expose the things like ->write_iter() et.al. to those in subsequent commits.
New predicate (
user_backed_iter()) that is true forITER_IOVECandITER_UBUF; places like direct-IO handling should use that for checking that pages we modify after getting them fromiov_iter_get_pages()would need to be dirtied.DO NOT assume that replacing
iter_is_iovec()withuser_backed_iter()will solve all problems - there's code that usesiter_is_iovec()to decide how to poke around iniov_iterguts and for that the predicate replacement obviously won't suffice.Signed-off-by Al Viro <[email protected]>--- block/fops.c | 6 +-- fs/ceph/file.c | 2 +- fs/cifs/file.c | 2 +- fs/direct-io.c | 2 +- fs/fuse/dev.c | 4 +- fs/fuse/file.c | 2 +- fs/gfs2/file.c | 2 +- fs/iomap/direct-io.c | 2 +- fs/nfs/direct.c | 2 +- include/linux/uio.h | 26 ++++++++++++ lib/iov_iter.c | 94 ++++++++++++++++++++++++++++++++++---------- mm/shmem.c | 2 +- 12 files changed, 113 insertions(+), 33 deletions(-)block/fops.c +3
-3@@ -69,7 +69,7 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,if (iov_iter_rw (iter) ==READ ) {bio_init (&bio, bdev, vecs, nr_pages,REQ_OP_READ ); -if (iter_is_iovec (iter)) +if (user_backed_iter (iter)) should_dirty = true; }else {bio_init (&bio, bdev, vecs, nr_pages,dio_bio_write_op (iocb));@@ -199,7 +199,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, } dio->size =0 ; -if (is_read &&iter_is_iovec (iter)) +if (is_read &&user_backed_iter (iter)) dio->flags |=DIO_SHOULD_DIRTY ;blk_start_plug (&plug);@@ -331,7 +331,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, dio->size = bio->bi_iter.bi_size;if (is_read) { -if (iter_is_iovec (iter)) { +if (user_backed_iter (iter)) { dio->flags |=DIO_SHOULD_DIRTY ;bio_set_pages_dirty (bio); }fs/ceph/file.c +1
-1@@ -1262,7 +1262,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,size_t count =iov_iter_count (iter);loff_t pos = iocb->ki_pos;bool write =iov_iter_rw (iter) ==WRITE ; -bool should_dirty = !write &&iter_is_iovec (iter); +bool should_dirty = !write &&user_backed_iter (iter);if (write &&ceph_snap (file_inode (file)) !=CEPH_NOSNAP )return -EROFS ;fs/cifs/file.c +1
-1@@ -4004,7 +4004,7 @@ static ssize_t __cifs_readv(if (!is_sync_kiocb (iocb)) ctx->iocb = iocb; -if (iter_is_iovec (to)) +if (user_backed_iter (to)) ctx->should_dirty = true;if (direct) {fs/direct-io.c +1
-1@@ -1245,7 +1245,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode,spin_lock_init (&dio->bio_lock); dio->refcount =1 ; - dio->should_dirty =iter_is_iovec (iter) &&iov_iter_rw (iter) ==READ ; + dio->should_dirty =user_backed_iter (iter) &&iov_iter_rw (iter) ==READ ; sdio.iter = iter; sdio.final_block_in_request = end >> blkbits;fs/fuse/dev.c +2
-2@@ -1356,7 +1356,7 @@ static ssize_t fuse_dev_read(struct kiocb *iocb, struct iov_iter *to)if (!fud)return -EPERM ; -if (!iter_is_iovec (to)) +if (!user_backed_iter (to))return -EINVAL ;fuse_copy_init (&cs,1 , to);@@ -1949,7 +1949,7 @@ static ssize_t fuse_dev_write(struct kiocb *iocb, struct iov_iter *from)if (!fud)return -EPERM ; -if (!iter_is_iovec (from)) +if (!user_backed_iter (from))return -EINVAL ;fuse_copy_init (&cs,0 , from);fs/fuse/file.c +1
-1@@ -1465,7 +1465,7 @@ ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter,inode_unlock (inode); } - io->should_dirty = !write &&iter_is_iovec (iter); + io->should_dirty = !write &&user_backed_iter (iter);while (count) {ssize_t nres;fl_owner_t owner = current->files;fs/gfs2/file.c +1
-1@@ -780,7 +780,7 @@ static inline bool should_fault_in_pages(struct iov_iter *i,if (!count)return false; -if (!iter_is_iovec (i)) +if (!user_backed_iter (i))return false; size =PAGE_SIZE ;fs/iomap/direct-io.c +1
-1@@ -533,7 +533,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomi.flags |=IOMAP_NOWAIT ; } -if (iter_is_iovec (iter)) +if (user_backed_iter (iter)) dio->flags |=IOMAP_DIO_DIRTY ; }else { iomi.flags |=IOMAP_WRITE ;fs/nfs/direct.c +1
-1@@ -478,7 +478,7 @@ ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter,if (!is_sync_kiocb (iocb)) dreq->iocb = iocb; -if (iter_is_iovec (iter)) +if (user_backed_iter (iter)) dreq->flags =NFS_ODIRECT_SHOULD_DIRTY ;if (!swap)include/linux/uio.h +8
-0@@ -26,6 +26,7 @@ enum iter_type {ITER_PIPE ,ITER_XARRAY ,ITER_DISCARD , + ITER_UBUF, }; struct iov_iter_state {@@ -38,6 +39,7 @@ struct iov_iter {u8 iter_type;bool nofault;bool data_source; +bool user_backed;size_t iov_offset;size_t count;union {@@ -46,6 +48,7 @@ struct iov_iter {const struct bio_vec *bvec;struct xarray *xarray;struct pipe_inode_info *pipe; +void __user *ubuf; };union {unsigned long nr_segs;@@ -70,6 +73,11 @@ static inline void iov_iter_save_state(struct iov_iter *iter, state->nr_segs = iter->nr_segs; } +static inline bool iter_is_ubuf (const struct iov_iter *i) +{ +return iov_iter_type (i) ==ITER_UBUF ; +} +static inline bool iter_is_iovec (const struct iov_iter *i) {return iov_iter_type (i) ==ITER_IOVEC ;
include/linux/uio.h static inline unsigned char iov_iter_rw(const struct iov_iter *i) · line 120
Quoted 8 lines
include/linux/uio.h +5
-0@@ -105,6 +113,11 @@ static inline unsigned char iov_iter_rw(const struct iov_iter *i)return i->data_source ?WRITE :READ ; } +static inline bool user_backed_iter (const struct iov_iter *i) +{ +return i->user_backed; +} +
nit: I wonder whether this new boolean is worth it over just checking is_iter_iovec() || is_iter_ubuf. Not a big deal though.
Quoted 266 lines
include/linux/uio.h +13
-0/* * Total number of bytes covered by an iovec. * @@ -320,4 +333,17 @@ ssize_t __import_iovec(int type, const struct iovec __user *uvec,int import_single_range (int type,void __user *buf,size_t len,struct iovec *iov,struct iov_iter *i); +static inline void iov_iter_ubuf (struct iov_iter *i,unsigned int direction, +void __user *buf,size_t count) +{ +WARN_ON (direction & ~(READ |WRITE )); + *i = (struct iov_iter ) { + .iter_type =ITER_UBUF , + .user_backed = true, + .data_source = direction, + .ubuf = buf, + .count = count + }; +} +#endif lib/iov_iter.c +74
-20@@ -16,6 +16,16 @@#define PIPE_PARANOIA /* for now */ +/* covers ubuf and kbuf alike */ +#define iterate_buf (i, n, base, len, off, __p,STEP ) { \ + size_t __maybe_unused off = 0; \ + len = n; \ + base = __p + i->iov_offset; \ + len -= (STEP); \ + i->iov_offset += len; \ + n = len; \ +} +/* covers iovec and kvec alike */ #define iterate_iovec (i, n, base, len, off, __p,STEP ) { \ size_t off = 0; \@@ -110,7 +120,12 @@ __out: \if (unlikely (i->count < n)) \ n = i->count; \if (likely (n)) { \ -if (likely (iter_is_iovec (i))) { \ +if (likely (iter_is_ubuf (i))) { \ + void __user *base; \ +size_t len; \ +iterate_buf (i, n, base, len, off, \ + i->ubuf, (I)) \ + }else if (likely (iter_is_iovec (i))) { \const struct iovec *iov = i->iov; \ void __user *base; \size_t len; \@@ -275,7 +290,11 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t by*/ size_t fault_in_iov_iter_readable (const struct iov_iter *i,size_t size) { -if (iter_is_iovec (i)) { +if (iter_is_ubuf (i)) { +size_t n =min (size,iov_iter_count (i)); + n -=fault_in_readable (i->ubuf + i->iov_offset, n); +return size - n; + }else if (iter_is_iovec (i)) {size_t count =min (size,iov_iter_count (i));const struct iovec *p;size_t skip;@@ -314,7 +333,11 @@ EXPORT_SYMBOL(fault_in_iov_iter_readable);*/ size_t fault_in_iov_iter_writeable (const struct iov_iter *i,size_t size) { -if (iter_is_iovec (i)) { +if (iter_is_ubuf (i)) { +size_t n =min (size,iov_iter_count (i)); + n -=fault_in_safe_writeable (i->ubuf + i->iov_offset, n); +return size - n; + }else if (iter_is_iovec (i)) {size_t count =min (size,iov_iter_count (i));const struct iovec *p;size_t skip;@@ -345,6 +368,7 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, *i = (struct iov_iter ) { .iter_type =ITER_IOVEC , .nofault = false, + .user_backed = true, .data_source = direction, .iov = iov, .nr_segs = nr_segs,@@ -494,7 +518,7 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) {if (unlikely (iov_iter_is_pipe (i)))return copy_pipe_to_iter (addr, bytes, i); -if (iter_is_iovec (i)) +if (user_backed_iter (i))might_fault (); iterate_and_advance(i, bytes, base, len, off, copyout(base, addr + off, len),@@ -576,7 +600,7 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) {if (unlikely (iov_iter_is_pipe (i)))return copy_mc_pipe_to_iter (addr, bytes, i); -if (iter_is_iovec (i)) +if (user_backed_iter (i))might_fault (); __iterate_and_advance(i, bytes, base, len, off, copyout_mc(base, addr + off, len),@@ -594,7 +618,7 @@ size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i)WARN_ON (1 );return 0 ; } -if (iter_is_iovec (i)) +if (user_backed_iter (i))might_fault (); iterate_and_advance(i, bytes, base, len, off, copyin(addr + off, base, len),@@ -882,16 +906,16 @@ void iov_iter_advance(struct iov_iter *i, size_t size) {if (unlikely (i->count < size)) size = i->count; -if (likely (iter_is_iovec (i) ||iov_iter_is_kvec (i))) { +if (likely (iter_is_ubuf (i)) ||unlikely (iov_iter_is_xarray (i))) { + i->iov_offset += size; + i->count -= size; + }else if (likely (iter_is_iovec (i) ||iov_iter_is_kvec (i))) {/* iovec and kvec have identical layouts */ iov_iter_iovec_advance (i, size); }else if (iov_iter_is_bvec (i)) {iov_iter_bvec_advance (i, size); }else if (iov_iter_is_pipe (i)) {pipe_advance (i, size); - }else if (unlikely (iov_iter_is_xarray (i))) { - i->iov_offset += size; - i->count -= size; }else if (iov_iter_is_discard (i)) { i->count -= size; }@@ -938,7 +962,7 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)return ; } unroll -= i->iov_offset; -if (iov_iter_is_xarray (i)) { +if (iov_iter_is_xarray (i) ||iter_is_ubuf (i)) {BUG ();/* We should never go beyond the start of the specified * range since we might then be straying into pages that * aren't pinned. @@ -1129,6 +1153,13 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)unsigned long iov_iter_alignment (const struct iov_iter *i) { +if (likely (iter_is_ubuf (i))) { +size_t size = i->count; +if (size) +return ((unsigned long )i->ubuf + i->iov_offset) | size; +return 0 ; + } +/* iovec and kvec have identical layouts */ if (likely (iter_is_iovec (i) ||iov_iter_is_kvec (i)))return iov_iter_alignment_iovec (i);@@ -1159,6 +1190,9 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i)size_t size = i->count;unsigned k; +if (iter_is_ubuf (i)) +return 0 ; +if (WARN_ON (!iter_is_iovec (i)))return ~0U ;@@ -1287,7 +1321,19 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,return actual; } -/* must be done on non-empty ITER_IOVEC one */ +static unsigned long found_ubuf_segment (unsigned long addr, +size_t len, +size_t *size,size_t *start, +unsigned maxpages) +{ + len += (*start = addr %PAGE_SIZE ); +if (len > maxpages *PAGE_SIZE ) + len = maxpages *PAGE_SIZE ; + *size = len; +return addr &PAGE_MASK ; +} + +/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment (const struct iov_iter *i,size_t *size,size_t *start,size_t maxsize,unsigned maxpages)@@ -1295,6 +1341,11 @@ static unsigned long first_iovec_segment(const struct iov_iter *i,size_t skip;long k; +if (iter_is_ubuf (i)) { +unsigned long addr = (unsigned long )i->ubuf + i->iov_offset; +return found_ubuf_segment (addr, maxsize, size, start, maxpages); + } +for (k =0 , skip = i->iov_offset; k < i->nr_segs; k++, skip =0 ) {unsigned long addr = (unsigned long )i->iov[k].iov_base + skip;size_t len = i->iov[k].iov_len - skip;@@ -1303,11 +1354,7 @@ static unsigned long first_iovec_segment(const struct iov_iter *i,continue ;if (len > maxsize) len = maxsize; - len += (*start = addr %PAGE_SIZE ); -if (len > maxpages *PAGE_SIZE ) - len = maxpages *PAGE_SIZE ; - *size = len; -return addr &PAGE_MASK ; +return found_ubuf_segment (addr, len, size, start, maxpages); }BUG ();// if it had been empty, we wouldn't get called }@@ -1344,7 +1391,7 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,if (!maxsize)return 0 ; -if (likely (iter_is_iovec (i))) { +if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;unsigned long addr;@@ -1470,7 +1517,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i,if (!maxsize)return 0 ; -if (likely (iter_is_iovec (i))) { +if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;unsigned long addr;@@ -1624,6 +1671,11 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages) {if (unlikely (!i->count))return 0 ; +if (likely (iter_is_ubuf (i))) { +unsigned offs =offset_in_page (i->ubuf + i->iov_offset); +int npages =DIV_ROUND_UP (offs + i->count,PAGE_SIZE ); +return min (npages, maxpages); + }/* iovec and kvec have identical layouts */ if (likely (iter_is_iovec (i) ||iov_iter_is_kvec (i)))return iov_npages (i, maxpages);@@ -1862,10 +1914,12 @@ EXPORT_SYMBOL(import_single_range);void iov_iter_restore (struct iov_iter *i,struct iov_iter_state *state) {if (WARN_ON_ONCE (!iov_iter_is_bvec (i) && !iter_is_iovec (i)) && - !iov_iter_is_kvec (i)) + !iov_iter_is_kvec (i) && !iter_is_ubuf (i))return ; i->iov_offset = state->iov_offset; i->count = state->count; +if (iter_is_ubuf (i)) +return ;/* * For the *vec iters, nr_segs + iov is constant - if we increment * the vec, then we also decrement the nr_segs count. Hence we don't mm/shmem.c +0
-0
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 2609
Quoted 8 lines
mm/shmem.c +1
-1@@ -2603,7 +2603,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) ret =copy_page_to_iter (page, offset, nr, to);put_page (page); - }else if (iter_is_iovec (to)) { + }else if (!user_backed_iter (to)) {/* * Copy to user tends to be so well optimized, but * clear_user() not so much, that it is noticeably
The code looks reasonable but is there any real benefit here? It seems like the only user of it so far is new_sync_{read,write}, and both seem to just use it to avoid allocating a single iovec on the stack.
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH 11/44] iov_iter_bvec_advance(): don't bother with bvec_iter
In reply to Al ViroQuoted 34 lines · Al Viro
do what we do for iovec/kvec; that ends up generating better code, AFAICS.
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)lib/iov_iter.c +7
-9@@ -870,17 +870,22 @@ static void pipe_advance(struct iov_iter *i, size_t size)static void iov_iter_bvec_advance (struct iov_iter *i,size_t size) { -struct bvec_iter bi; +const struct bio_vec *bvec, *end; - bi.bi_size = i->count; - bi.bi_bvec_done = i->iov_offset; - bi.bi_idx =0 ; -bvec_iter_advance (i->bvec, &bi, size); +if (!i->count) +return ; + i->count -= size; + + size += i->iov_offset; - i->bvec += bi.bi_idx; - i->nr_segs -= bi.bi_idx; - i->count = bi.bi_size; - i->iov_offset = bi.bi_bvec_done; +for (bvec = i->bvec, end = bvec + i->nr_segs; bvec < end; bvec++) {
lib/iov_iter.c static void pipe_advance(struct iov_iter *i, size_t size) · line 891
Quoted 10 lines
lib/iov_iter.c +7
-0+if (likely (size < bvec->bv_len)) +break ; + size -= bvec->bv_len; + } + i->iov_offset = size; + i->nr_segs -= bvec - i->bvec; + i->bvec = bvec; }static void iov_iter_iovec_advance (struct iov_iter *i,size_t size)
Much simpler to follow, IMO...
Re: [PATCH 12/44] fix short copy handling in copy_mc_pipe_to_iter()
In reply to Al ViroQuoted 63 lines · Al Viro
Unlike other copying operations on
ITER_PIPE,copy_mc_to_iter()can result in a short copy. In that case we need to trim the unused buffers, as well as the length of partially filled one - it's not enough to set ->head, ->iov_offset and ->count to reflect how much had we copied. Not hard to fix, fortunately...I'd put a helper (
pipe_discard_from(pipe, head)) intopipe_fs_i.h, rather thaniov_iter.c- it has nothing to do withiov_iterand having it will allow us to avoid an ugly kludge infs/splice.c. We could put it intolib/iov_iter.cfor now and move it later, but I don't see the point going that way...Fixes ca146f6f091e "lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe()"Signed-off-by Al Viro <[email protected]>--- include/linux/pipe_fs_i.h | 9 +++++++++ lib/iov_iter.c | 15 +++++++++++---- 2 files changed, 20 insertions(+), 4 deletions(-)include/linux/pipe_fs_i.h +9
-0@@ -229,6 +229,15 @@ static inline bool pipe_buf_try_steal(struct pipe_inode_info *pipe,return buf->ops->try_steal (pipe, buf); } +static inline void pipe_discard_from (struct pipe_inode_info *pipe, +unsigned int old_head) +{ +unsigned int mask = pipe->ring_size -1 ; + +while (pipe->head > old_head) +pipe_buf_release (pipe, &pipe->bufs[--pipe->head & mask]); +} +/* Differs from PIPE_BUF in that PIPE_SIZE is the length of the actual memory allocation, whereas PIPE_BUF makes atomicity guarantees. */ #define PIPE_SIZE PAGE_SIZElib/iov_iter.c +5
-4@@ -689,6 +689,7 @@ static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes,struct pipe_inode_info *pipe = i->pipe;unsigned int p_mask = pipe->ring_size -1 ;unsigned int i_head; +unsigned int valid = pipe->head;size_t n, off, xfer =0 ;if (!sanity (i))@@ -702,11 +703,17 @@ static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, rem =copy_mc_to_kernel (p + off, addr + xfer, chunk); chunk -= rem;kunmap_local (p); - i->head = i_head; - i->iov_offset = off + chunk; - xfer += chunk; -if (rem) +if (chunk) { + i->head = i_head; + i->iov_offset = off + chunk; + xfer += chunk;
lib/iov_iter.c static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, · line 719
Quoted 10 lines
lib/iov_iter.c +6
-0+ valid = i_head +1 ; + } +if (rem) { + pipe->bufs[i_head & p_mask].len -= rem; +pipe_discard_from (pipe, valid);break ; + } n -= chunk; off =0 ; i_head++;
Re: [PATCH 13/44] splice: stop abusing iov_iter_advance() to flush a pipe
In reply to Al ViroQuoted 26 lines · Al Viro
Use
pipe_discard_from()explicitly ingeneric_file_read_iter(); don't bother with rather non-obvious use ofiov_iter_advance()in there.Signed-off-by Al Viro <[email protected]>--- fs/splice.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)fs/splice.c +0
-2@@ -301,11 +301,9 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, {struct iov_iter to;struct kiocb kiocb; -unsigned int i_head;int ret;iov_iter_pipe (&to,READ , pipe, len); - i_head = to.head;init_sync_kiocb (&kiocb, in); kiocb.ki_pos = *ppos; ret =call_read_iter (in, &kiocb, &to);@@ -313,9 +311,8 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, *ppos = kiocb.ki_pos;
fs/splice.c ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, · line 318
Quoted 10 lines
fs/splice.c +2
-3file_accessed (in); }else if (ret <0 ) { - to.head = i_head; - to.iov_offset =0 ; -iov_iter_advance (&to,0 );/* to free what was emitted */ +/* free what was emitted */ +pipe_discard_from (pipe, to.start_head);/* * callers of ->splice_read() expect -EAGAIN on * "can't put anything in there", rather than -EFAULT.
Re: [PATCH 14/44] ITER_PIPE: helper for getting pipe buffer by index
In reply to Al ViroQuoted 69 lines · Al Viro
pipe_bufferinstances of a pipe are organized as a ring buffer, with power-of-2 size. Indices are kept *not* reduced modulo ring size, so the buffer refered to by index N ispipe->bufs[N & (pipe->ring_size - 1)].Ring size can change over the lifetime of a pipe, but not while the pipe is locked. So for any
iov_iterprimitives it's a constant. Original conversion of pipes to this layout went overboard trying to microoptimize that - calculatingpipe->ring_size- 1, storing it in a local variable and using through the function. In some cases it might be warranted, but most of the times it only obfuscates what's going on in there.Introduce a helper (
pipe_buf(pipe, N)) that would encapsulate that and use it in the obvious cases. More will follow...Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)lib/iov_iter.c +8
-4@@ -183,13 +183,18 @@ static int copyin(void *to, const void __user *from, size_t n)return n; } +static inline struct pipe_buffer *pipe_buf (const struct pipe_inode_info *pipe, +unsigned int slot) +{ +return &pipe->bufs[slot & (pipe->ring_size -1 )]; +} +#ifdef PIPE_PARANOIA static bool sanity (const struct iov_iter *i) {struct pipe_inode_info *pipe = i->pipe;unsigned int p_head = pipe->head;unsigned int p_tail = pipe->tail; -unsigned int p_mask = pipe->ring_size -1 ;unsigned int p_occupancy =pipe_occupancy (p_head, p_tail);unsigned int i_head = i->head;unsigned int idx;@@ -201,7 +206,7 @@ static bool sanity(const struct iov_iter *i)if (unlikely (i_head != p_head -1 )) goto Bad;// must be at the last buffer... - p = &pipe->bufs[i_head & p_mask]; + p =pipe_buf (pipe, i_head);if (unlikely (p->offset + p->len != i->iov_offset)) goto Bad;// ... at the end of segment }else {@@ -386,11 +391,10 @@ static inline bool allocated(struct pipe_buffer *buf)static inline void data_start (const struct iov_iter *i,unsigned int *iter_headp,size_t *offp) { -unsigned int p_mask = i->pipe->ring_size -1 ;unsigned int iter_head = i->head;size_t off = i->iov_offset; -if (off && (!allocated (&i->pipe->bufs[iter_head & p_mask]) || +if (off && (!allocated (pipe_buf (i->pipe, iter_head)) || off ==PAGE_SIZE )) { iter_head++; off =0 ;@@ -1180,10 +1184,9 @@ unsigned long iov_iter_alignment(const struct iov_iter *i)return iov_iter_alignment_bvec (i);
lib/iov_iter.c unsigned long iov_iter_alignment(const struct iov_iter *i) · line 1192
Quoted 10 lines
lib/iov_iter.c +1
-2if (iov_iter_is_pipe (i)) { -unsigned int p_mask = i->pipe->ring_size -1 ;size_t size = i->count; -if (size && i->iov_offset &&allocated (&i->pipe->bufs[i->head & p_mask])) +if (size && i->iov_offset &&allocated (pipe_buf (i->pipe, i->head)))return size | i->iov_offset;return size; }
Re: [PATCH 15/44] ITER_PIPE: helpers for adding pipe buffers
In reply to Al ViroQuoted 149 lines · Al Viro
There are only two kinds of
pipe_bufferin the area used byITER_PIPE.1) anonymous -
copy_to_iter()et.al. end up creating those and copying data there. They have zero ->offset, and their ->ops points todefault_pipe_page_ops.2) zero-copy ones - those come from
copy_page_to_iter(), and page comes from caller. ->offset is also caller-supplied - it might be non-zero. ->ops points topage_cache_pipe_buf_ops.Move creation and insertion of those into helpers -
push_anon(pipe, size)andpush_page(pipe, page, offset, size)resp., separating them from the "could we avoid creating a new buffer by merging with the current head?" logics.Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 88 ++++++++++++++++++++++++++------------------------ 1 file changed, 46 insertions(+), 42 deletions(-)lib/iov_iter.c +44
-37@@ -231,15 +231,39 @@ static bool sanity(const struct iov_iter *i)#define sanity (i) true#endif +static struct page *push_anon (struct pipe_inode_info *pipe,unsigned size) +{ +struct page *page =alloc_page (GFP_USER ); +if (page) { +struct pipe_buffer *buf =pipe_buf (pipe, pipe->head++); + *buf = (struct pipe_buffer ) { + .ops = &default_pipe_buf_ops, + .page = page, + .offset =0 , + .len = size + }; + } +return page; +} + +static void push_page (struct pipe_inode_info *pipe,struct page *page, +unsigned int offset,unsigned int size) +{ +struct pipe_buffer *buf =pipe_buf (pipe, pipe->head++); + *buf = (struct pipe_buffer ) { + .ops = &page_cache_pipe_buf_ops, + .page = page, + .offset = offset, + .len = size + }; +get_page (page); +} +static size_t copy_page_to_iter_pipe (struct page *page,size_t offset,size_t bytes,struct iov_iter *i) {struct pipe_inode_info *pipe = i->pipe; -struct pipe_buffer *buf; -unsigned int p_tail = pipe->tail; -unsigned int p_mask = pipe->ring_size -1 ; -unsigned int i_head = i->head; -size_t off; +unsigned int head = pipe->head;if (unlikely (bytes > i->count)) bytes = i->count;@@ -250,32 +274,21 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t byif (!sanity (i))return 0 ; - off = i->iov_offset; - buf = &pipe->bufs[i_head & p_mask]; -if (off) { -if (offset == off && buf->page == page) { -/* merge with the last one */ +if (offset && i->iov_offset == offset) {// could we merge it? +struct pipe_buffer *buf =pipe_buf (pipe, head -1 ); +if (buf->page == page) { buf->len += bytes; i->iov_offset += bytes; - goto out; + i->count -= bytes; +return bytes; } - i_head++; - buf = &pipe->bufs[i_head & p_mask]; } -if (pipe_full (i_head, p_tail, pipe->max_usage)) +if (pipe_full (pipe->head, pipe->tail, pipe->max_usage))return 0 ; - buf->ops = &page_cache_pipe_buf_ops; - buf->flags =0 ; -get_page (page); - buf->page = page; - buf->offset = offset; - buf->len = bytes; - - pipe->head = i_head +1 ; +push_page (pipe, page, offset, bytes); i->iov_offset = offset + bytes; - i->head = i_head; -out: + i->head = head; i->count -= bytes;return bytes; }@@ -407,8 +420,6 @@ static size_t push_pipe(struct iov_iter *i, size_t size, int *iter_headp, size_t *offp) { struct pipe_inode_info *pipe = i->pipe; -unsigned int p_tail = pipe->tail; -unsigned int p_mask = pipe->ring_size -1 ;unsigned int iter_head;size_t off;ssize_t left;@@ -423,30 +434,23 @@ static size_t push_pipe(struct iov_iter *i, size_t size, *iter_headp = iter_head; *offp = off;if (off) { +struct pipe_buffer *buf =pipe_buf (pipe, iter_head); + left -=PAGE_SIZE - off;if (left <=0 ) { - pipe->bufs[iter_head & p_mask].len += size; + buf->len += size;return size; } - pipe->bufs[iter_head & p_mask].len =PAGE_SIZE ; - iter_head++; + buf->len =PAGE_SIZE ; } -while (!pipe_full (iter_head, p_tail, pipe->max_usage)) { -struct pipe_buffer *buf = &pipe->bufs[iter_head & p_mask]; -struct page *page =alloc_page (GFP_USER ); +while (!pipe_full (pipe->head, pipe->tail, pipe->max_usage)) { +struct page *page =push_anon (pipe, +min_t (ssize_t, left,PAGE_SIZE ));if (!page)break ; - buf->ops = &default_pipe_buf_ops; - buf->flags =0 ; - buf->page = page; - buf->offset =0 ; - buf->len =min_t (ssize_t, left,PAGE_SIZE );
lib/iov_iter.c static size_t push_pipe(struct iov_iter *i, size_t size, · line 456
Quoted 10 lines
lib/iov_iter.c +2
-5- left -= buf->len; - iter_head++; - pipe->head = iter_head; - -if (left ==0 ) + left -=PAGE_SIZE ; +if (left <=0 )return size; }return size - left;
Not sure I follow all of the buffer handling shenanigans in here, but I think it looks sane.
Re: [PATCH 23/44] iov_iter_get_pages{,_alloc}(): cap the maxsize with MAX_RW_COUNT
In reply to Al ViroQuoted 23 lines · Al Viro
All callers can and should handle
iov_iter_get_pages()returning fewer pages than requested. All in-kernel ones do. And it makes the arithmetical overflow analysis much simpler...Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 4 ++++ 1 file changed, 4 insertions(+)lib/iov_iter.c +2
-0@@ -1367,6 +1367,8 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, maxsize = i->count;if (!maxsize)return 0 ; +if (maxsize >MAX_RW_COUNT ) + maxsize =MAX_RW_COUNT ;if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;
lib/iov_iter.c ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, · line 1494
Quoted 8 lines
lib/iov_iter.c +2
-0@@ -1485,6 +1487,8 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, maxsize = i->count;if (!maxsize)return 0 ; +if (maxsize >MAX_RW_COUNT ) + maxsize =MAX_RW_COUNT ;if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;
Re: [PATCH 24/44] iov_iter_get_pages_alloc(): lift freeing pages array on failure exits into wrapper
In reply to Al ViroQuoted 83 lines · Al Viro
Incidentally,
ITER_XARRAYdid *not* free the sucker in case wheniter_xarray_populate_pages()returned 0...Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-)lib/iov_iter.c +15
-16@@ -1425,15 +1425,10 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, maxsize = n; else npages =DIV_ROUND_UP (maxsize + off,PAGE_SIZE ); - p =get_pages_array (npages); + *pages = p =get_pages_array (npages);if (!p)return -ENOMEM ; - n =__pipe_get_pages (i, maxsize, p, off); -if (n >0 ) - *pages = p; -else -kvfree (p); -return n; +return __pipe_get_pages (i, maxsize, p, off); } static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i,@@ -1463,10 +1458,9 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, count++; } - p =get_pages_array (count); + *pages = p =get_pages_array (count);if (!p)return -ENOMEM ; - *pages = p; nr =iter_xarray_populate_pages (p, i->xarray, index, count);if (nr ==0 )@@ -1475,7 +1469,7 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i,return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); } -ssize_t iov_iter_get_pages_alloc (struct iov_iter *i, +static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start) {@@ -1501,16 +1495,12 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, addr =first_iovec_segment (i, &len, start, maxsize, ~0U ); n =DIV_ROUND_UP (len,PAGE_SIZE ); - p =get_pages_array (n); + *pages = p =get_pages_array (n);if (!p)return -ENOMEM ; res =get_user_pages_fast (addr, n, gup_flags, p); -if (unlikely (res <=0 )) { -kvfree (p); - *pages =NULL ; +if (unlikely (res <=0 ))return res; - } - *pages = p;return (res == n ? len : res *PAGE_SIZE ) - *start; }if (iov_iter_is_bvec (i)) {@@ -1531,6 +1521,22 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i,return iter_xarray_get_pages_alloc (i, pages, maxsize, start);return -EFAULT ; } + +ssize_t iov_iter_get_pages_alloc (struct iov_iter *i, +struct page ***pages,size_t maxsize, +size_t *start) +{ +ssize_t len; + + *pages =NULL ; +
lib/iov_iter.c ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, · line 1542
Quoted 10 lines
lib/iov_iter.c +7
-0+ len =__iov_iter_get_pages_alloc (i, pages, maxsize, start); +if (len <=0 ) { +kvfree (*pages); + *pages =NULL ; + } +return len; +}EXPORT_SYMBOL (iov_iter_get_pages_alloc); size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,
Re: [PATCH 25/44] iov_iter_get_pages(): sanity-check arguments
In reply to Al ViroQuoted 38 lines · Al Viro
zero maxpages is bogus, but best treated as "just return 0"; NULL pages, OTOH, should be treated as a hard bug.
get rid of now completely useless checks in xarray_get_pages{,_alloc}().
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)lib/iov_iter.c +2
-4@@ -1271,9 +1271,6 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,size_t size = maxsize;loff_t pos; -if (!size || !maxpages) -return 0 ; - pos = i->xarray_start + i->iov_offset; index = pos >>PAGE_SHIFT ; offset = pos & ~PAGE_MASK ;@@ -1365,10 +1362,11 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,if (maxsize > i->count) maxsize = i->count; -if (!maxsize) +if (!maxsize || !maxpages)return 0 ;if (maxsize >MAX_RW_COUNT ) maxsize =MAX_RW_COUNT ; +BUG_ON (!pages);if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;
lib/iov_iter.c static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, · line 1444
Quoted 9 lines
lib/iov_iter.c +0
-3@@ -1441,9 +1439,6 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i,size_t size = maxsize;loff_t pos; -if (!size) -return 0 ; - pos = i->xarray_start + i->iov_offset; index = pos >>PAGE_SHIFT ; offset = pos & ~PAGE_MASK ;
Re: [PATCH 26/44] unify pipe_get_pages() and pipe_get_pages_alloc()
In reply to Al ViroQuoted 106 lines · Al Viro
The differences between those two are
*pipe_get_pages()gets a non-NULL struct page ** value pointing to preallocated array + array size.
*pipe_get_pages_alloc()gets an address of struct page ** variable that contains NULL, allocates the array and (on success) stores its address in that variable.Not hard to combine - always pass struct page ***, have
the previouspipe_get_pages_alloc()caller pass ~0U as cap for array size.Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 49 +++++++++++++++++-------------------------------- 1 file changed, 17 insertions(+), 32 deletions(-)lib/iov_iter.c +16
-31@@ -1187,6 +1187,11 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i) }EXPORT_SYMBOL (iov_iter_gap_alignment); +static struct page **get_pages_array (size_t n) +{ +return kvmalloc_array (n,sizeof (struct page *),GFP_KERNEL ); +} + static inline ssize_t __pipe_get_pages(struct iov_iter *i, size_t maxsize, struct page **pages,@@ -1220,10 +1225,11 @@ static inline ssize_t __pipe_get_pages(struct iov_iter *i, }static ssize_t pipe_get_pages (struct iov_iter *i, -struct page **pages,size_t maxsize,unsigned maxpages, +struct page ***pages,size_t maxsize,unsigned maxpages,size_t *start) {unsigned int npages, off; +struct page **p;size_t capacity;if (!sanity (i))@@ -1231,8 +1237,15 @@ static ssize_t pipe_get_pages(struct iov_iter *i, *start = off =pipe_npages (i, &npages); capacity =min (npages, maxpages) *PAGE_SIZE - off; + maxsize =min (maxsize, capacity); + p = *pages; +if (!p) { + *pages = p =get_pages_array (DIV_ROUND_UP (maxsize + off,PAGE_SIZE )); +if (!p) +return -ENOMEM ; + } -return __pipe_get_pages (i,min (maxsize, capacity), pages, off); +return __pipe_get_pages (i, maxsize, p, off); } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,@@ -1394,41 +1407,13 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,return len - *start; }if (iov_iter_is_pipe (i)) -return pipe_get_pages (i, pages, maxsize, maxpages, start); +return pipe_get_pages (i, &pages, maxsize, maxpages, start);if (iov_iter_is_xarray (i))return iter_xarray_get_pages (i, pages, maxsize, maxpages, start);return -EFAULT ; }EXPORT_SYMBOL (iov_iter_get_pages); -static struct page **get_pages_array (size_t n) -{ -return kvmalloc_array (n,sizeof (struct page *),GFP_KERNEL ); -} - -static ssize_t pipe_get_pages_alloc (struct iov_iter *i, -struct page ***pages,size_t maxsize, -size_t *start) -{ -struct page **p; -unsigned int npages, off; -ssize_t n; - -if (!sanity (i)) -return -EFAULT ; - - *start = off =pipe_npages (i, &npages); - n = npages *PAGE_SIZE - off; -if (maxsize > n) - maxsize = n; -else - npages =DIV_ROUND_UP (maxsize + off,PAGE_SIZE ); - *pages = p =get_pages_array (npages); -if (!p) -return -ENOMEM ; -return __pipe_get_pages (i, maxsize, p, off); -} -static ssize_t iter_xarray_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *_start_offset)
lib/iov_iter.c static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, · line 1502
Quoted 8 lines
lib/iov_iter.c +1
-1@@ -1511,7 +1496,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,return len - *start; }if (iov_iter_is_pipe (i)) -return pipe_get_pages_alloc (i, pages, maxsize, start); +return pipe_get_pages (i, pages, maxsize, ~0U , start);if (iov_iter_is_xarray (i))return iter_xarray_get_pages_alloc (i, pages, maxsize, start);return -EFAULT ;
Re: [PATCH 27/44] unify xarray_get_pages() and xarray_get_pages_alloc()
In reply to Al ViroQuoted 84 lines · Al Viro
same as for pipes
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 49 ++++++++++--------------------------------------- 1 file changed, 10 insertions(+), 39 deletions(-)lib/iov_iter.c +9
-38@@ -1276,7 +1276,7 @@ static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa }static ssize_t iter_xarray_get_pages (struct iov_iter *i, -struct page **pages,size_t maxsize, +struct page ***pages,size_t maxsize,unsigned maxpages,size_t *_start_offset) {unsigned nr, offset;@@ -1301,7 +1301,13 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,if (count > maxpages) count = maxpages; - nr =iter_xarray_populate_pages (pages, i->xarray, index, count); +if (!*pages) { + *pages =get_pages_array (count); +if (!*pages) +return -ENOMEM ; + } + + nr =iter_xarray_populate_pages (*pages, i->xarray, index, count);if (nr ==0 )return 0 ;@@ -1409,46 +1415,11 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,if (iov_iter_is_pipe (i))return pipe_get_pages (i, &pages, maxsize, maxpages, start);if (iov_iter_is_xarray (i)) -return iter_xarray_get_pages (i, pages, maxsize, maxpages, start); +return iter_xarray_get_pages (i, &pages, maxsize, maxpages, start);return -EFAULT ; }EXPORT_SYMBOL (iov_iter_get_pages); -static ssize_t iter_xarray_get_pages_alloc (struct iov_iter *i, -struct page ***pages,size_t maxsize, -size_t *_start_offset) -{ -struct page **p; -unsigned nr, offset; -pgoff_t index, count; -size_t size = maxsize; -loff_t pos; - - pos = i->xarray_start + i->iov_offset; - index = pos >>PAGE_SHIFT ; - offset = pos & ~PAGE_MASK ; - *_start_offset = offset; - - count =1 ; -if (size >PAGE_SIZE - offset) { - size -=PAGE_SIZE - offset; - count += size >>PAGE_SHIFT ; - size &= ~PAGE_MASK ; -if (size) - count++; - } - - *pages = p =get_pages_array (count); -if (!p) -return -ENOMEM ; - - nr =iter_xarray_populate_pages (p, i->xarray, index, count); -if (nr ==0 ) -return 0 ; - -return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); -} -static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start)
lib/iov_iter.c static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, · line 1475
Quoted 8 lines
lib/iov_iter.c +1
-1@@ -1498,7 +1469,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (iov_iter_is_pipe (i))return pipe_get_pages (i, pages, maxsize, ~0U , start);if (iov_iter_is_xarray (i)) -return iter_xarray_get_pages_alloc (i, pages, maxsize, start); +return iter_xarray_get_pages (i, pages, maxsize, ~0U , start);return -EFAULT ; }
Re: [PATCH 28/44] unify the rest of iov_iter_get_pages()/iov_iter_get_pages_alloc() guts
In reply to Al ViroQuoted 142 lines · Al Viro
same as for pipes and xarrays; after that
iov_iter_get_pages()becomes a wrapper for__iov_iter_get_pages_alloc().Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 86 ++++++++++++++++---------------------------------- 1 file changed, 28 insertions(+), 58 deletions(-)lib/iov_iter.c +27
-57@@ -1372,20 +1372,19 @@ static struct page *first_bvec_segment(const struct iov_iter *i,return page; } -ssize_t iov_iter_get_pages (struct iov_iter *i, -struct page **pages,size_t maxsize,unsigned maxpages, -size_t *start) +static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i, +struct page ***pages,size_t maxsize, +unsigned int maxpages,size_t *start) {size_t len;int n, res;if (maxsize > i->count) maxsize = i->count; -if (!maxsize || !maxpages) +if (!maxsize)return 0 ;if (maxsize >MAX_RW_COUNT ) maxsize =MAX_RW_COUNT ; -BUG_ON (!pages);if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;@@ -1398,80 +1397,51 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, addr =first_iovec_segment (i, &len, start, maxsize, maxpages); n =DIV_ROUND_UP (len,PAGE_SIZE ); - res =get_user_pages_fast (addr, n, gup_flags, pages); +if (!*pages) { + *pages =get_pages_array (n); +if (!*pages) +return -ENOMEM ; + } + res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res;return (res == n ? len : res *PAGE_SIZE ) - *start; }if (iov_iter_is_bvec (i)) { +struct page **p;struct page *page; page =first_bvec_segment (i, &len, start, maxsize, maxpages); n =DIV_ROUND_UP (len,PAGE_SIZE ); + p = *pages; +if (!p) { + *pages = p =get_pages_array (n); +if (!p) +return -ENOMEM ; + }while (n--) -get_page (*pages++ = page++); +get_page (*p++ = page++);return len - *start; }if (iov_iter_is_pipe (i)) -return pipe_get_pages (i, &pages, maxsize, maxpages, start); +return pipe_get_pages (i, pages, maxsize, maxpages, start);if (iov_iter_is_xarray (i)) -return iter_xarray_get_pages (i, &pages, maxsize, maxpages, start); +return iter_xarray_get_pages (i, pages, maxsize, maxpages, + start);return -EFAULT ; } -EXPORT_SYMBOL (iov_iter_get_pages); -static ssize_t __iov_iter_get_pages_alloc (struct iov_iter *i, -struct page ***pages,size_t maxsize, +ssize_t iov_iter_get_pages (struct iov_iter *i, +struct page **pages,size_t maxsize,unsigned maxpages,size_t *start) { -struct page **p; -size_t len; -int n, res; - -if (maxsize > i->count) - maxsize = i->count; -if (!maxsize) +if (!maxpages)return 0 ; -if (maxsize >MAX_RW_COUNT ) - maxsize =MAX_RW_COUNT ; - -if (likely (user_backed_iter (i))) { -unsigned int gup_flags =0 ; -unsigned long addr; - -if (iov_iter_rw (i) !=WRITE ) - gup_flags |=FOLL_WRITE ; -if (i->nofault) - gup_flags |=FOLL_NOFAULT ; - - addr =first_iovec_segment (i, &len, start, maxsize, ~0U ); - n =DIV_ROUND_UP (len,PAGE_SIZE ); - *pages = p =get_pages_array (n); -if (!p) -return -ENOMEM ; - res =get_user_pages_fast (addr, n, gup_flags, p); -if (unlikely (res <=0 )) -return res; -return (res == n ? len : res *PAGE_SIZE ) - *start; - } -if (iov_iter_is_bvec (i)) { -struct page *page; +BUG_ON (!pages); - page =first_bvec_segment (i, &len, start, maxsize, ~0U ); - n =DIV_ROUND_UP (len,PAGE_SIZE ); - *pages = p =get_pages_array (n); -if (!p) -return -ENOMEM ; -while (n--) -get_page (*p++ = page++); -return len - *start; - } -if (iov_iter_is_pipe (i)) -return pipe_get_pages (i, pages, maxsize, ~0U , start); -if (iov_iter_is_xarray (i)) -return iter_xarray_get_pages (i, pages, maxsize, ~0U , start); -return -EFAULT ; +return __iov_iter_get_pages_alloc (i, &pages, maxsize, maxpages, start); } +EXPORT_SYMBOL (iov_iter_get_pages); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize,
lib/iov_iter.c ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, · line 1457
Quoted 8 lines
lib/iov_iter.c +1
-1@@ -1481,7 +1451,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, *pages =NULL ; - len =__iov_iter_get_pages_alloc (i, pages, maxsize, start); + len =__iov_iter_get_pages_alloc (i, pages, maxsize, ~0U , start);if (len <=0 ) {kvfree (*pages); *pages =NULL ;
Re: [PATCH 29/44] ITER_XARRAY: don't open-code DIV_ROUND_UP()
In reply to Al ViroQuoted 16 lines · Al Viro
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-)lib/iov_iter.c +0
-3@@ -1289,15 +1289,7 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, offset = pos & ~PAGE_MASK ; *_start_offset = offset; - count =1 ; -if (size >PAGE_SIZE - offset) { - size -=PAGE_SIZE - offset;
lib/iov_iter.c static ssize_t iter_xarray_get_pages(struct iov_iter *i, · line 1295
Quoted 10 lines
lib/iov_iter.c +1
-6- count += size >>PAGE_SHIFT ; - size &= ~PAGE_MASK ; -if (size) - count++; - } - + count =DIV_ROUND_UP (size + offset,PAGE_SIZE );if (count > maxpages) count = maxpages;
Re: [PATCH 30/44] iov_iter: lift dealing with maxpages out of first_{iovec,bvec}_segment()
In reply to Al ViroQuoted 79 lines · Al Viro
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-)lib/iov_iter.c +8
-11@@ -1308,12 +1308,9 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,static unsigned long found_ubuf_segment (unsigned long addr,size_t len, -size_t *size,size_t *start, -unsigned maxpages) +size_t *size,size_t *start) { len += (*start = addr %PAGE_SIZE ); -if (len > maxpages *PAGE_SIZE ) - len = maxpages *PAGE_SIZE ; *size = len;return addr &PAGE_MASK ; }@@ -1321,14 +1318,14 @@ static unsigned long found_ubuf_segment(unsigned long addr,/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment (const struct iov_iter *i,size_t *size,size_t *start, -size_t maxsize,unsigned maxpages) +size_t maxsize) {size_t skip;long k;if (iter_is_ubuf (i)) {unsigned long addr = (unsigned long )i->ubuf + i->iov_offset; -return found_ubuf_segment (addr, maxsize, size, start, maxpages); +return found_ubuf_segment (addr, maxsize, size, start); }for (k =0 , skip = i->iov_offset; k < i->nr_segs; k++, skip =0 ) {@@ -1339,7 +1336,7 @@ static unsigned long first_iovec_segment(const struct iov_iter *i,continue ;if (len > maxsize) len = maxsize; -return found_ubuf_segment (addr, len, size, start, maxpages); +return found_ubuf_segment (addr, len, size, start); }BUG ();// if it had been empty, we wouldn't get called }@@ -1347,7 +1344,7 @@ static unsigned long first_iovec_segment(const struct iov_iter *i,/* must be done on non-empty ITER_BVEC one */ static struct page *first_bvec_segment (const struct iov_iter *i,size_t *size,size_t *start, -size_t maxsize,unsigned maxpages) +size_t maxsize) {struct page *page;size_t skip = i->iov_offset, len;@@ -1358,8 +1355,6 @@ static struct page *first_bvec_segment(const struct iov_iter *i, skip += i->bvec->bv_offset; page = i->bvec->bv_page + skip /PAGE_SIZE ; len += (*start = skip %PAGE_SIZE ); -if (len > maxpages *PAGE_SIZE ) - len = maxpages *PAGE_SIZE ; *size = len;return page; }@@ -1387,7 +1382,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (i->nofault) gup_flags |=FOLL_NOFAULT ; - addr =first_iovec_segment (i, &len, start, maxsize, maxpages); + addr =first_iovec_segment (i, &len, start, maxsize); +if (len > maxpages *PAGE_SIZE ) + len = maxpages *PAGE_SIZE ; n =DIV_ROUND_UP (len,PAGE_SIZE );if (!*pages) { *pages =get_pages_array (n);
lib/iov_iter.c static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, · line 1408
Quoted 10 lines
lib/iov_iter.c +3
-1@@ -1403,7 +1400,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,struct page **p;struct page *page; - page =first_bvec_segment (i, &len, start, maxsize, maxpages); + page =first_bvec_segment (i, &len, start, maxsize); +if (len > maxpages *PAGE_SIZE ) + len = maxpages *PAGE_SIZE ; n =DIV_ROUND_UP (len,PAGE_SIZE ); p = *pages;if (!p) {
Re: [PATCH 31/44] iov_iter: first_{iovec,bvec}_segment() - simplify a bit
In reply to Al ViroQuoted 65 lines · Al Viro
We return length + offset in page via *size. Don't bother - the caller can do that arithmetics just as well; just report the length to it.
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)lib/iov_iter.c +9
-9@@ -1310,7 +1310,7 @@ static unsigned long found_ubuf_segment(unsigned long addr,size_t len, size_t *size, size_t *start) { - len += (*start = addr % PAGE_SIZE); + *start = addr %PAGE_SIZE ; *size = len;return addr &PAGE_MASK ; }@@ -1354,7 +1354,7 @@ static struct page *first_bvec_segment(const struct iov_iter *i, len = maxsize; skip += i->bvec->bv_offset; page = i->bvec->bv_page + skip /PAGE_SIZE ; - len += (*start = skip %PAGE_SIZE ); + *start = skip %PAGE_SIZE ; *size = len;return page; }@@ -1383,9 +1383,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, gup_flags |=FOLL_NOFAULT ; addr =first_iovec_segment (i, &len, start, maxsize); -if (len > maxpages *PAGE_SIZE ) - len = maxpages *PAGE_SIZE ; - n =DIV_ROUND_UP (len,PAGE_SIZE ); + n =DIV_ROUND_UP (len + *start,PAGE_SIZE ); +if (n > maxpages) + n = maxpages;if (!*pages) { *pages =get_pages_array (n);if (!*pages)@@ -1394,25 +1394,25 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res; -return (res == n ? len : res *PAGE_SIZE ) - *start; +return min_t (size_t, len, res *PAGE_SIZE - *start); }if (iov_iter_is_bvec (i)) {struct page **p;struct page *page; page =first_bvec_segment (i, &len, start, maxsize); -if (len > maxpages *PAGE_SIZE ) - len = maxpages *PAGE_SIZE ; - n =DIV_ROUND_UP (len,PAGE_SIZE ); + n =DIV_ROUND_UP (len + *start,PAGE_SIZE ); +if (n > maxpages) + n = maxpages; p = *pages;if (!p) { *pages = p =get_pages_array (n);if (!p)
lib/iov_iter.c static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, · line 1418
Quoted 10 lines
lib/iov_iter.c +2
-2return -ENOMEM ; } -while (n--) +for (int k =0 ; k < n; k++)get_page (*p++ = page++); -return len - *start; +return min_t (size_t, len, n *PAGE_SIZE - *start); }if (iov_iter_is_pipe (i))return pipe_get_pages (i, pages, maxsize, maxpages, start);
Re: [PATCH 32/44] iov_iter: massage calling conventions for first_{iovec,bvec}_segment()
In reply to Al ViroQuoted 114 lines · Al Viro
Pass maxsize by reference, return length via the same.
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-)lib/iov_iter.c +14
-21@@ -1306,26 +1306,22 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); } -static unsigned long found_ubuf_segment (unsigned long addr, -size_t len, -size_t *size,size_t *start) +static unsigned long found_ubuf_segment (unsigned long addr,size_t *start) { *start = addr %PAGE_SIZE ; - *size = len;return addr &PAGE_MASK ; }/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment (const struct iov_iter *i, -size_t *size,size_t *start, -size_t maxsize) +size_t *size,size_t *start) {size_t skip;long k;if (iter_is_ubuf (i)) {unsigned long addr = (unsigned long )i->ubuf + i->iov_offset; -return found_ubuf_segment (addr, maxsize, size, start); +return found_ubuf_segment (addr, start); }for (k =0 , skip = i->iov_offset; k < i->nr_segs; k++, skip =0 ) {@@ -1334,28 +1330,26 @@ static unsigned long first_iovec_segment(const struct iov_iter *i,if (unlikely (!len))continue ; -if (len > maxsize) - len = maxsize; -return found_ubuf_segment (addr, len, size, start); +if (*size > len) + *size = len; +return found_ubuf_segment (addr, start); }BUG ();// if it had been empty, we wouldn't get called }/* must be done on non-empty ITER_BVEC one */ static struct page *first_bvec_segment (const struct iov_iter *i, -size_t *size,size_t *start, -size_t maxsize) +size_t *size,size_t *start) {struct page *page;size_t skip = i->iov_offset, len; len = i->bvec->bv_len - skip; -if (len > maxsize) - len = maxsize; +if (*size > len) + *size = len; skip += i->bvec->bv_offset; page = i->bvec->bv_page + skip /PAGE_SIZE ; *start = skip %PAGE_SIZE ; - *size = len;return page; }@@ -1363,7 +1357,6 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start) { - size_t len; int n, res;if (maxsize > i->count)@@ -1382,8 +1375,8 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (i->nofault) gup_flags |=FOLL_NOFAULT ; - addr =first_iovec_segment (i, &len, start, maxsize); - n =DIV_ROUND_UP (len + *start,PAGE_SIZE ); + addr =first_iovec_segment (i, &maxsize, start); + n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE );if (n > maxpages) n = maxpages;if (!*pages) {@@ -1394,14 +1387,14 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res; -return min_t (size_t, len, res *PAGE_SIZE - *start); +return min_t (size_t, maxsize, res *PAGE_SIZE - *start); }if (iov_iter_is_bvec (i)) {struct page **p;struct page *page; - page =first_bvec_segment (i, &len, start, maxsize); - n =DIV_ROUND_UP (len + *start,PAGE_SIZE ); + page =first_bvec_segment (i, &maxsize, start); + n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE );if (n > maxpages) n = maxpages; p = *pages;
lib/iov_iter.c static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, · line 1411
Quoted 8 lines
lib/iov_iter.c +1
-1@@ -1412,7 +1405,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, }for (int k =0 ; k < n; k++)get_page (*p++ = page++); -return min_t (size_t, len, n *PAGE_SIZE - *start); +return min_t (size_t, maxsize, n *PAGE_SIZE - *start); }if (iov_iter_is_pipe (i))return pipe_get_pages (i, pages, maxsize, maxpages, start);
Re: [PATCH 33/44] found_iovec_segment(): just return address
In reply to Al ViroSubject line should read "first_iovec_segment"
Quoted 50 lines · Al Viro
... and calculate the offset in the caller
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-)lib/iov_iter.c +4
-14@@ -1306,33 +1306,23 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); } -static unsigned long found_ubuf_segment (unsigned long addr,size_t *start) -{ - *start = addr %PAGE_SIZE ; -return addr &PAGE_MASK ; -} -/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment (const struct iov_iter *i, -size_t *size,size_t *start) +size_t *size) {size_t skip;long k; -if (iter_is_ubuf (i)) { -unsigned long addr = (unsigned long )i->ubuf + i->iov_offset; -return found_ubuf_segment (addr, start); - } +if (iter_is_ubuf (i)) +return (unsigned long )i->ubuf + i->iov_offset;for (k =0 , skip = i->iov_offset; k < i->nr_segs; k++, skip =0 ) { -unsigned long addr = (unsigned long )i->iov[k].iov_base + skip;size_t len = i->iov[k].iov_len - skip; -if (unlikely (!len))continue ;if (*size > len) *size = len; -return found_ubuf_segment (addr, start); +return (unsigned long )i->iov[k].iov_base + skip; }BUG ();// if it had been empty, we wouldn't get called }
lib/iov_iter.c static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, · line 1373
Quoted 10 lines
lib/iov_iter.c +3
-1@@ -1375,7 +1365,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (i->nofault) gup_flags |=FOLL_NOFAULT ; - addr =first_iovec_segment (i, &maxsize, start); + addr =first_iovec_segment (i, &maxsize); + *start = addr %PAGE_SIZE ; + addr &=PAGE_MASK ; n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE );if (n > maxpages) n = maxpages;
Re: [PATCH 34/44] fold __pipe_get_pages() into pipe_get_pages()
In reply to Al ViroQuoted 109 lines · Al Viro
... and don't mangle maxsize there - turn the loop into counting one instead. Easier to see that we won't run out of array that way. Note that special treatment of the partial buffer in that thing is an artifact of the non-advancing semantics of
iov_iter_get_pages()- if not for that, it would beappend_pipe(), same as the body of the loop that follows it. IOW, once we makeiov_iter_get_pages()advancing, the whole thing will turn into
calculate how many pages do we want
allocate an array (if needed)
callappend_pipe()that many times.Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 75 +++++++++++++++++++++++++------------------------- 1 file changed, 38 insertions(+), 37 deletions(-)lib/iov_iter.c +31
-37@@ -1192,60 +1192,61 @@ static struct page **get_pages_array(size_t n)return kvmalloc_array (n,sizeof (struct page *),GFP_KERNEL ); } -static inline ssize_t __pipe_get_pages (struct iov_iter *i, -size_t maxsize, -struct page **pages, -size_t off) -{ -struct pipe_inode_info *pipe = i->pipe; -ssize_t left = maxsize; - -if (off) { -struct pipe_buffer *buf =pipe_buf (pipe, pipe->head -1 ); - -get_page (*pages++ = buf->page); - left -=PAGE_SIZE - off; -if (left <=0 ) { - buf->len += maxsize; -return maxsize; - } - buf->len =PAGE_SIZE ; - } -while (!pipe_full (pipe->head, pipe->tail, pipe->max_usage)) { -struct page *page =push_anon (pipe, -min_t (ssize_t, left,PAGE_SIZE )); -if (!page) -break ; -get_page (*pages++ = page); - left -=PAGE_SIZE ; -if (left <=0 ) -return maxsize; - } -return maxsize - left ? : -EFAULT ; -} -static ssize_t pipe_get_pages (struct iov_iter *i,struct page ***pages,size_t maxsize,unsigned maxpages,size_t *start) { +struct pipe_inode_info *pipe = i->pipe;unsigned int npages, off;struct page **p; -size_t capacity; +ssize_t left; +int count;if (!sanity (i))return -EFAULT ; *start = off =pipe_npages (i, &npages); - capacity =min (npages, maxpages) *PAGE_SIZE - off; - maxsize =min (maxsize, capacity); + count =DIV_ROUND_UP (maxsize + off,PAGE_SIZE ); +if (count > npages) + count = npages; +if (count > maxpages) + count = maxpages; p = *pages;if (!p) { - *pages = p =get_pages_array (DIV_ROUND_UP (maxsize + off,PAGE_SIZE )); + *pages = p =get_pages_array (count);if (!p)return -ENOMEM ; } -return __pipe_get_pages (i, maxsize, p, off); + left = maxsize; + npages =0 ; +if (off) { +struct pipe_buffer *buf =pipe_buf (pipe, pipe->head -1 ); + +get_page (*p++ = buf->page); + left -=PAGE_SIZE - off; +if (left <=0 ) { + buf->len += maxsize; +return maxsize; + } + buf->len =PAGE_SIZE ; + npages =1 ; + } +for ( ; npages < count; npages++) { +struct page *page; +unsigned int size =min_t (ssize_t, left,PAGE_SIZE ); + +if (pipe_full (pipe->head, pipe->tail, pipe->max_usage)) +break ; + page =push_anon (pipe, size); +if (!page)
lib/iov_iter.c static struct page **get_pages_array(size_t n) · line 1252
Quoted 10 lines
lib/iov_iter.c +7
-0+break ; +get_page (*p++ = page); + left -= size; + } +if (!npages) +return -EFAULT ; +return maxsize - left; } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
Re: [PATCH 35/44] iov_iter: saner helper for page array allocation
In reply to Al ViroQuoted 149 lines · Al Viro
All call sites of
get_pages_array()are essenitally identical now. Replace with common helper...Returns number of slots available in resulting array or 0 on OOM; it's up to the caller to make sure it doesn't ask to zero-entry array (i.e. neither maxpages nor size are allowed to be zero).
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 77 +++++++++++++++++++++----------------------------- 1 file changed, 32 insertions(+), 45 deletions(-)lib/iov_iter.c +31
-40@@ -1187,9 +1187,20 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i) }EXPORT_SYMBOL (iov_iter_gap_alignment); -static struct page **get_pages_array (size_t n) +static int want_pages_array (struct page ***res,size_t size, +size_t start,unsigned int maxpages) { -return kvmalloc_array (n,sizeof (struct page *),GFP_KERNEL ); +unsigned int count =DIV_ROUND_UP (size + start,PAGE_SIZE ); + +if (count > maxpages) + count = maxpages; +WARN_ON (!count);// caller should've prevented that +if (!*res) { + *res =kvmalloc_array (count,sizeof (struct page *),GFP_KERNEL ); +if (!*res) +return 0 ; + } +return count; } static ssize_t pipe_get_pages(struct iov_iter *i,@@ -1197,27 +1208,20 @@ static ssize_t pipe_get_pages(struct iov_iter *i, size_t *start) {struct pipe_inode_info *pipe = i->pipe; -unsigned int npages, off; +unsigned int npages, off, count;struct page **p;ssize_t left; -int count;if (!sanity (i))return -EFAULT ; *start = off =pipe_npages (i, &npages); - count =DIV_ROUND_UP (maxsize + off,PAGE_SIZE ); -if (count > npages) - count = npages; -if (count > maxpages) - count = maxpages; +if (!npages) +return -EFAULT ; + count =want_pages_array (pages, maxsize, off,min (npages, maxpages)); +if (!count) +return -ENOMEM ; p = *pages; -if (!p) { - *pages = p =get_pages_array (count); -if (!p) -return -ENOMEM ; - } - left = maxsize; npages =0 ;if (off) {@@ -1280,9 +1284,8 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned maxpages, size_t *_start_offset) { - unsigned nr, offset; -pgoff_t index, count; -size_t size = maxsize; + unsigned nr, offset, count; +pgoff_t index;loff_t pos; pos = i->xarray_start + i->iov_offset;@@ -1290,16 +1293,9 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, offset = pos & ~PAGE_MASK ; *_start_offset = offset; - count =DIV_ROUND_UP (size + offset,PAGE_SIZE ); -if (count > maxpages) - count = maxpages; - -if (!*pages) { - *pages =get_pages_array (count); -if (!*pages) -return -ENOMEM ; - } - + count =want_pages_array (pages, maxsize, offset, maxpages); +if (!count) +return -ENOMEM ; nr =iter_xarray_populate_pages (*pages, i->xarray, index, count);if (nr ==0 )return 0 ;@@ -1348,7 +1344,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start) { - int n, res; + unsigned int n;if (maxsize > i->count) maxsize = i->count;@@ -1360,6 +1356,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,if (likely (user_backed_iter (i))) {unsigned int gup_flags =0 ;unsigned long addr; +int res;if (iov_iter_rw (i) !=WRITE ) gup_flags |=FOLL_WRITE ;@@ -1369,14 +1366,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, addr =first_iovec_segment (i, &maxsize); *start = addr %PAGE_SIZE ; addr &=PAGE_MASK ; - n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE ); -if (n > maxpages) - n = maxpages; -if (!*pages) { - *pages =get_pages_array (n); -if (!*pages) -return -ENOMEM ; - } + n =want_pages_array (pages, maxsize, *start, maxpages); +if (!n) +return -ENOMEM ; res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res;@@ -1387,15 +1379,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,struct page *page; page =first_bvec_segment (i, &maxsize, start); - n =DIV_ROUND_UP (maxsize + *start,PAGE_SIZE ); -if (n > maxpages) - n = maxpages; + n =want_pages_array (pages, maxsize, *start, maxpages); +if (!n)
lib/iov_iter.c static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, · line 1388
Quoted 10 lines
lib/iov_iter.c +1
-5+return -ENOMEM ; p = *pages; -if (!p) { - *pages = p =get_pages_array (n); -if (!p) -return -ENOMEM ; - }for (int k =0 ; k < n; k++)get_page (*p++ = page++);return min_t (size_t, maxsize, n *PAGE_SIZE - *start);
Re: [PATCH 36/44] iov_iter: advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al ViroQuoted 320 lines · Al Viro
Most of the users immediately follow successful
iov_iter_get_pages()with advancing by the amount it had returned.Provide inline wrappers doing that, convert trivial open-coded uses of those.
BTW,
iov_iter_get_pages()never returns more than it had been asked to; such checks in cifs ought to be removed someday...Signed-off-by Al Viro <[email protected]>--- drivers/vhost/scsi.c | 4 +--- fs/ceph/file.c | 3 +-- fs/cifs/file.c | 6 ++---- fs/cifs/misc.c | 3 +-- fs/direct-io.c | 3 +-- fs/fuse/dev.c | 3 +-- fs/fuse/file.c | 3 +-- fs/nfs/direct.c | 6 ++---- include/linux/uio.h | 20 ++++++++++++++++++++ net/core/datagram.c | 3 +-- net/core/skmsg.c | 3 +-- net/rds/message.c | 3 +-- net/tls/tls_sw.c | 4 +--- 13 files changed, 34 insertions(+), 30 deletions(-)drivers/vhost/scsi.c +1
-3@@ -643,14 +643,12 @@ vhost_scsi_map_to_sgl(struct vhost_scsi_cmd *cmd,size_t offset;unsigned int npages =0 ; - bytes =iov_iter_get_pages (iter, pages,LONG_MAX , + bytes =iov_iter_get_pages2 (iter, pages,LONG_MAX ,VHOST_SCSI_PREALLOC_UPAGES , &offset);/* No pages were pinned */ if (bytes <=0 )return bytes <0 ? bytes : -EFAULT ; -iov_iter_advance (iter, bytes); -while (bytes) {unsigned n =min_t (unsigned,PAGE_SIZE - offset, bytes);sg_set_page (sg++, pages[npages++], n, offset);fs/ceph/file.c +1
-2@@ -95,12 +95,11 @@ static ssize_t __iter_get_bvecs(struct iov_iter *iter, size_t maxsize,size_t start;int idx =0 ; - bytes =iov_iter_get_pages (iter, pages, maxsize - size, + bytes =iov_iter_get_pages2 (iter, pages, maxsize - size,ITER_GET_BVECS_PAGES , &start);if (bytes <0 )return size ?: bytes; -iov_iter_advance (iter, bytes); size += bytes;for ( ; bytes; idx++, bvec_idx++) {fs/cifs/file.c +2
-4@@ -3022,7 +3022,7 @@ cifs_write_from_iter(loff_t offset, size_t len, struct iov_iter *from,if (ctx->direct_io) {ssize_t result; - result =iov_iter_get_pages_alloc ( + result =iov_iter_get_pages_alloc2 ( from, &pagevec, cur_len, &start);if (result <0 ) { cifs_dbg(VFS,@@ -3036,7 +3036,6 @@ cifs_write_from_iter(loff_t offset, size_t len, struct iov_iter *from,break ; } cur_len = (size_t )result; -iov_iter_advance (from, cur_len); nr_pages = (cur_len + start +PAGE_SIZE -1 ) /PAGE_SIZE ;@@ -3758,7 +3757,7 @@ cifs_send_async_read(loff_t offset, size_t len, struct cifsFileInfo *open_file,if (ctx->direct_io) {ssize_t result; - result =iov_iter_get_pages_alloc ( + result =iov_iter_get_pages_alloc2 ( &direct_iov, &pagevec, cur_len, &start);if (result <0 ) {@@ -3774,7 +3773,6 @@ cifs_send_async_read(loff_t offset, size_t len, struct cifsFileInfo *open_file,break ; } cur_len = (size_t )result; -iov_iter_advance (&direct_iov, cur_len); rdata =cifs_readdata_direct_alloc ( pagevec, cifs_uncached_readv_complete);fs/cifs/misc.c +1
-2@@ -1022,7 +1022,7 @@ setup_aio_ctx_iter(struct cifs_aio_ctx *ctx, struct iov_iter *iter, int rw) saved_len = count;while (count && npages < max_pages) { - rc =iov_iter_get_pages (iter, pages, count, max_pages, &start); + rc =iov_iter_get_pages2 (iter, pages, count, max_pages, &start);if (rc <0 ) {cifs_dbg (VFS ,"Couldn't get user pages (rc=%zd)\n" , rc);break ;@@ -1034,7 +1034,6 @@ setup_aio_ctx_iter(struct cifs_aio_ctx *ctx, struct iov_iter *iter, int rw)break ; } -iov_iter_advance (iter, rc); count -= rc; rc += start; cur_npages =DIV_ROUND_UP (rc,PAGE_SIZE );fs/direct-io.c +1
-2@@ -169,7 +169,7 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) {ssize_t ret; - ret =iov_iter_get_pages (sdio->iter, dio->pages,LONG_MAX ,DIO_PAGES , + ret =iov_iter_get_pages2 (sdio->iter, dio->pages,LONG_MAX ,DIO_PAGES , &sdio->from);if (ret <0 && sdio->blocks_available && (dio->op ==REQ_OP_WRITE )) {@@ -191,7 +191,6 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) }if (ret >=0 ) { -iov_iter_advance (sdio->iter, ret); ret += sdio->from; sdio->head =0 ; sdio->tail = (ret +PAGE_SIZE -1 ) /PAGE_SIZE ;fs/fuse/dev.c +1
-2@@ -730,14 +730,13 @@ static int fuse_copy_fill(struct fuse_copy_state *cs) } }else {size_t off; - err =iov_iter_get_pages (cs->iter, &page,PAGE_SIZE ,1 , &off); + err =iov_iter_get_pages2 (cs->iter, &page,PAGE_SIZE ,1 , &off);if (err <0 )return err;BUG_ON (!err); cs->len = err; cs->offset = off; cs->pg = page; -iov_iter_advance (cs->iter, err); }return lock_request (cs->req);fs/fuse/file.c +1
-2@@ -1401,14 +1401,13 @@ static int fuse_get_user_pages(struct fuse_args_pages *ap, struct iov_iter *ii,while (nbytes < *nbytesp && ap->num_pages < max_pages) {unsigned npages;size_t start; - ret =iov_iter_get_pages (ii, &ap->pages[ap->num_pages], + ret =iov_iter_get_pages2 (ii, &ap->pages[ap->num_pages], *nbytesp - nbytes, max_pages - ap->num_pages, &start);if (ret <0 )break ; -iov_iter_advance (ii, ret); nbytes += ret; ret += start;fs/nfs/direct.c +2
-4@@ -364,13 +364,12 @@ static ssize_t nfs_direct_read_schedule_iovec(struct nfs_direct_req *dreq,size_t pgbase;unsigned npages, i; - result =iov_iter_get_pages_alloc (iter, &pagevec, + result =iov_iter_get_pages_alloc2 (iter, &pagevec, rsize, &pgbase);if (result <0 )break ; bytes = result; -iov_iter_advance (iter, bytes); npages = (result + pgbase +PAGE_SIZE -1 ) /PAGE_SIZE ;for (i =0 ; i < npages; i++) {struct nfs_page *req;@@ -812,13 +811,12 @@ static ssize_t nfs_direct_write_schedule_iovec(struct nfs_direct_req *dreq,size_t pgbase;unsigned npages, i; - result =iov_iter_get_pages_alloc (iter, &pagevec, + result =iov_iter_get_pages_alloc2 (iter, &pagevec, wsize, &pgbase);if (result <0 )break ; bytes = result; -iov_iter_advance (iter, bytes); npages = (result + pgbase +PAGE_SIZE -1 ) /PAGE_SIZE ;for (i =0 ; i < npages; i++) {struct nfs_page *req;include/linux/uio.h +20
-0@@ -349,4 +349,24 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +static inline ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages, +size_t maxsize,unsigned maxpages,size_t *start) +{ +ssize_t res =iov_iter_get_pages (i, pages, maxsize, maxpages, start); + +if (res >=0 ) +iov_iter_advance (i, res); +return res; +} + +static inline ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages, +size_t maxsize,size_t *start) +{ +ssize_t res =iov_iter_get_pages_alloc (i, pages, maxsize, start); + +if (res >=0 ) +iov_iter_advance (i, res); +return res; +} +#endif net/core/datagram.c +1
-2@@ -629,12 +629,11 @@ int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb,if (frag ==MAX_SKB_FRAGS )return -EMSGSIZE ; - copied =iov_iter_get_pages (from, pages, length, + copied =iov_iter_get_pages2 (from, pages, length,MAX_SKB_FRAGS - frag, &start);if (copied <0 )return -EFAULT ; -iov_iter_advance (from, copied); length -= copied; truesize =PAGE_ALIGN (copied + start);net/core/skmsg.c +1
-2@@ -324,14 +324,13 @@ int sk_msg_zerocopy_from_iter(struct sock *sk, struct iov_iter *from, goto out; } - copied =iov_iter_get_pages (from, pages, bytes, maxpages, + copied =iov_iter_get_pages2 (from, pages, bytes, maxpages, &offset);if (copied <=0 ) { ret = -EFAULT ; goto out; } -iov_iter_advance (from, copied); bytes -= copied; msg->sg.size += copied;net/rds/message.c +1
-2@@ -391,7 +391,7 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter *size_t start;ssize_t copied; - copied =iov_iter_get_pages (from, &pages,PAGE_SIZE , + copied =iov_iter_get_pages2 (from, &pages,PAGE_SIZE ,1 , &start);if (copied <0 ) {struct mmpin *mmp;@@ -405,7 +405,6 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * goto err; } total_copied += copied; -iov_iter_advance (from, copied); length -= copied;sg_set_page (sg, pages, copied, start); rm->data.op_nents++;net/tls/tls_sw.c +1
-1@@ -1361,7 +1361,7 @@ static int tls_setup_from_iter(struct iov_iter *from, rc = -EFAULT ; goto out; } - copied =iov_iter_get_pages (from, pages, + copied =iov_iter_get_pages2 (from, pages, length, maxpages, &offset);if (copied <=0 ) {
net/tls/tls_sw.c static int tls_setup_from_iter(struct iov_iter *from, · line 1374
Quoted 8 lines
net/tls/tls_sw.c +0
-2@@ -1369,8 +1369,6 @@ static int tls_setup_from_iter(struct iov_iter *from, goto out; } -iov_iter_advance (from, copied); - length -= copied; size += copied;while (copied) {
Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al ViroQuoted 97 lines · Al Viro
... doing revert if we end up not using some pages
Signed-off-by Al Viro <[email protected]>--- block/bio.c | 15 ++++++--------- block/blk-map.c | 7 ++++--- 2 files changed, 10 insertions(+), 12 deletions(-)block/bio.c +6
-9@@ -1190,7 +1190,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)BUILD_BUG_ON (PAGE_PTRS_PER_BVEC <2 ); pages += entries_left * (PAGE_PTRS_PER_BVEC -1 ); - size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); + size =iov_iter_get_pages2 (iter, pages,LONG_MAX , nr_pages, &offset);if (unlikely (size <=0 ))return size ? size : -EFAULT ;@@ -1205,6 +1205,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) }else {if (WARN_ON_ONCE (bio_full (bio, len))) {bio_put_pages (pages + i, left, offset); +iov_iter_revert (iter, left);return -EINVAL ; }__bio_add_page (bio, page, len, offset);@@ -1212,7 +1213,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) offset =0 ; } -iov_iter_advance (iter, size);return 0 ; }@@ -1227,7 +1227,6 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)ssize_t size, left;unsigned len, i;size_t offset; -int ret =0 ;if (WARN_ON_ONCE (!max_append_sectors))return 0 ;@@ -1240,7 +1239,7 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)BUILD_BUG_ON (PAGE_PTRS_PER_BVEC <2 ); pages += entries_left * (PAGE_PTRS_PER_BVEC -1 ); - size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); + size =iov_iter_get_pages2 (iter, pages,LONG_MAX , nr_pages, &offset);if (unlikely (size <=0 ))return size ? size : -EFAULT ;@@ -1252,16 +1251,14 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)if (bio_add_hw_page (q, bio, page, len, offset, max_append_sectors, &same_page) != len) {bio_put_pages (pages + i, left, offset); - ret = -EINVAL ; -break ; +iov_iter_revert (iter, left); +return -EINVAL ; }if (same_page)put_page (page); offset =0 ; } - -iov_iter_advance (iter, size - left); -return ret; +return 0 ; }/** block/blk-map.c +1
-2@@ -254,7 +254,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,size_t offs, added =0 ;int npages; - bytes =iov_iter_get_pages_alloc (iter, &pages,LONG_MAX , &offs); + bytes =iov_iter_get_pages_alloc2 (iter, &pages,LONG_MAX , &offs);if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ; goto out_unmap;@@ -284,7 +284,6 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, bytes -= n; offs =0 ; } -iov_iter_advance (iter, added); }/* * release the pages we didn't map into the bio, if any @@ -293,8 +292,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,put_page (pages[j++]);
block/blk-map.c static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, · line 301
Quoted 10 lines
block/blk-map.c +3
-1kvfree (pages);/* couldn't stuff something into bio? */ -if (bytes) +if (bytes) { +iov_iter_revert (iter, bytes);break ; + } } ret =blk_rq_append_bio (rq, bio);
Re: [PATCH 38/44] iter_to_pipe(): switch to advancing variant of iov_iter_get_pages()
In reply to Al ViroQuoted 65 lines · Al Viro
... and untangle the cleanup on failure to add into pipe.
Signed-off-by Al Viro <[email protected]>--- fs/splice.c | 47 ++++++++++++++++++++++++----------------------- 1 file changed, 24 insertions(+), 23 deletions(-)fs/splice.c +20
-22@@ -1160,39 +1160,40 @@ static int iter_to_pipe(struct iov_iter *from, };size_t total =0 ;int ret =0 ; -bool failed = false; -while (iov_iter_count (from) && !failed) { +while (iov_iter_count (from)) {struct page *pages[16 ]; -ssize_t copied; +ssize_t left;size_t start; -int n; +int i, n; - copied =iov_iter_get_pages (from, pages, ~0UL ,16 , &start); -if (copied <=0 ) { - ret = copied; + left =iov_iter_get_pages2 (from, pages, ~0UL ,16 , &start); +if (left <=0 ) { + ret = left;break ; } -for (n =0 ; copied; n++, start =0 ) { -int size =min_t (int, copied,PAGE_SIZE - start); -if (!failed) { - buf.page = pages[n]; - buf.offset = start; - buf.len = size; - ret =add_to_pipe (pipe, &buf); -if (unlikely (ret <0 )) { - failed = true; - }else { -iov_iter_advance (from, ret); - total += ret; - } - }else { -put_page (pages[n]); + n =DIV_ROUND_UP (left + start,PAGE_SIZE ); +for (i =0 ; i < n; i++) { +int size =min_t (int, left,PAGE_SIZE - start); + + buf.page = pages[i]; + buf.offset = start; + buf.len = size; + ret =add_to_pipe (pipe, &buf); +if (unlikely (ret <0 )) { +iov_iter_revert (from, left); +// this one got dropped by add_to_pipe() +while (++i < n) +put_page (pages[i]); + goto out; }
fs/splice.c static int iter_to_pipe(struct iov_iter *from, · line 1199
Quoted 10 lines
fs/splice.c +4
-1- copied -= size; + total += ret; + left -= size; + start =0 ; } } +out:return total ? total : ret; }
Re: [PATCH 39/44] af_alg_make_sg(): switch to advancing variant of iov_iter_get_pages()
In reply to Al ViroQuoted 38 lines · Al Viro
... and adjust the callers
Signed-off-by Al Viro <[email protected]>--- crypto/af_alg.c | 3 +-- crypto/algif_hash.c | 5 +++-- 2 files changed, 4 insertions(+), 4 deletions(-)crypto/af_alg.c +1
-2@@ -404,7 +404,7 @@ int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len)ssize_t n;int npages, i; - n =iov_iter_get_pages (iter, sgl->pages, len,ALG_MAX_PAGES , &off); + n =iov_iter_get_pages2 (iter, sgl->pages, len,ALG_MAX_PAGES , &off);if (n <0 )return n;@@ -1191,7 +1191,6 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, len += err;atomic_add (err, &ctx->rcvused); rsgl->sg_num_bytes = err; -iov_iter_advance (&msg->msg_iter, err); } *outlen = len;crypto/algif_hash.c +0
-1@@ -102,11 +102,12 @@ static int hash_sendmsg(struct socket *sock, struct msghdr *msg, err =crypto_wait_req (crypto_ahash_update (&ctx->req), &ctx->wait);af_alg_free_sg (&ctx->sgl); -if (err)
crypto/algif_hash.c static int hash_sendmsg(struct socket *sock, struct msghdr *msg, · line 113
Quoted 10 lines
crypto/algif_hash.c +3
-1+if (err) { +iov_iter_revert (&msg->msg_iter, len); goto unlock; + } copied += len; -iov_iter_advance (&msg->msg_iter, len); } err =0 ;
Re: [PATCH 41/44] ceph: switch the last caller of iov_iter_get_pages_alloc()
In reply to Al ViroQuoted 13 lines · Al Viro
here nothing even looks at the
iov_iterafter the call, so we couldn't care less whether it advances or not.Signed-off-by Al Viro <[email protected]>--- fs/ceph/addr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/ceph/addr.c +0
-0
fs/ceph/addr.c static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) · line 335
Quoted 8 lines
fs/ceph/addr.c +1
-1@@ -329,7 +329,7 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)dout ("%s: pos=%llu orig_len=%zu len=%llu\n" , __func__, subreq->start, subreq->len, len);iov_iter_xarray (&iter,READ , &rreq->mapping->i_pages, subreq->start, len); - err =iov_iter_get_pages_alloc (&iter, &pages, len, &page_off); + err =iov_iter_get_pages_alloc2 (&iter, &pages, len, &page_off);if (err <0 ) {dout ("%s: iov_ter_get_pages_alloc returned %d\n" , __func__, err); goto out;
There are some coming changes to make this code use an iter passed in as part of the subreq, at which point we will need to advance this anyway.
Re: [PATCH 42/44] get rid of non-advancing variants
In reply to Al ViroQuoted 129 lines · Al Viro
mechanical change; will be further massaged in subsequent commits
Signed-off-by Al Viro <[email protected]>--- include/linux/uio.h | 24 ++---------------------- lib/iov_iter.c | 27 ++++++++++++++++++--------- 2 files changed, 20 insertions(+), 31 deletions(-)include/linux/uio.h +2
-22@@ -245,9 +245,9 @@ void iov_iter_pipe(struct iov_iter *i, unsigned int direction, struct pipe_inodevoid iov_iter_discard (struct iov_iter *i,unsigned int direction,size_t count);void iov_iter_xarray (struct iov_iter *i,unsigned int direction,struct xarray *xarray,loff_t start,size_t count); -ssize_t iov_iter_get_pages (struct iov_iter *i,struct page **pages, +ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start); -ssize_t iov_iter_get_pages_alloc (struct iov_iter *i,struct page ***pages, +ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start);int iov_iter_npages (const struct iov_iter *i,int maxpages);void iov_iter_restore (struct iov_iter *i,struct iov_iter_state *state);@@ -349,24 +349,4 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } -static inline ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages, -size_t maxsize,unsigned maxpages,size_t *start) -{ -ssize_t res =iov_iter_get_pages (i, pages, maxsize, maxpages, start); - -if (res >=0 ) -iov_iter_advance (i, res); -return res; -} - -static inline ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages, -size_t maxsize,size_t *start) -{ -ssize_t res =iov_iter_get_pages_alloc (i, pages, maxsize, start); - -if (res >=0 ) -iov_iter_advance (i, res); -return res; -} -#endif lib/iov_iter.c +17
-8@@ -1231,6 +1231,7 @@ static ssize_t pipe_get_pages(struct iov_iter *i, left -=PAGE_SIZE - off;if (left <=0 ) { buf->len += maxsize; +iov_iter_advance (i, maxsize);return maxsize; } buf->len =PAGE_SIZE ;@@ -1250,7 +1251,9 @@ static ssize_t pipe_get_pages(struct iov_iter *i, }if (!npages)return -EFAULT ; -return maxsize - left; + maxsize -= left; +iov_iter_advance (i, maxsize); +return maxsize; } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,@@ -1300,7 +1303,9 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,if (nr ==0 )return 0 ; -return min_t (size_t, nr *PAGE_SIZE - offset, maxsize); + maxsize =min_t (size_t, nr *PAGE_SIZE - offset, maxsize); +iov_iter_advance (i, maxsize); +return maxsize; }/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ @@ -1372,7 +1377,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, res =get_user_pages_fast (addr, n, gup_flags, *pages);if (unlikely (res <=0 ))return res; -return min_t (size_t, maxsize, res *PAGE_SIZE - *start); + maxsize =min_t (size_t, maxsize, res *PAGE_SIZE - *start); +iov_iter_advance (i, maxsize); +return maxsize; }if (iov_iter_is_bvec (i)) {struct page **p;@@ -1384,8 +1391,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,return -ENOMEM ; p = *pages;for (int k =0 ; k < n; k++) -get_page (*p++ = page++); -return min_t (size_t, maxsize, n *PAGE_SIZE - *start); +get_page (p[k] = page + k); + maxsize =min_t (size_t, maxsize, n *PAGE_SIZE - *start); +iov_iter_advance (i, maxsize); +return maxsize; }if (iov_iter_is_pipe (i))return pipe_get_pages (i, pages, maxsize, maxpages, start);@@ -1395,7 +1404,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,return -EFAULT ; } -ssize_t iov_iter_get_pages (struct iov_iter *i, +ssize_t iov_iter_get_pages2 (struct iov_iter *i,struct page **pages,size_t maxsize,unsigned maxpages,size_t *start) {@@ -1405,9 +1414,9 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,return __iov_iter_get_pages_alloc (i, &pages, maxsize, maxpages, start); } -EXPORT_SYMBOL (iov_iter_get_pages); +EXPORT_SYMBOL (iov_iter_get_pages2); -ssize_t iov_iter_get_pages_alloc (struct iov_iter *i, +ssize_t iov_iter_get_pages_alloc2 (struct iov_iter *i,struct page ***pages,size_t maxsize,size_t *start) {
lib/iov_iter.c ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, · line 1437
Quoted 8 lines
lib/iov_iter.c +1
-1@@ -1422,7 +1431,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, }return len; } -EXPORT_SYMBOL (iov_iter_get_pages_alloc); +EXPORT_SYMBOL (iov_iter_get_pages_alloc2);size_t csum_and_copy_from_iter (void *addr,size_t bytes,__wsum *csum,struct iov_iter *i)
Jeff Layton
maintainerRe: [PATCH 44/44] expand those iov_iter_advance()...
In reply to Al ViroQuoted 24 lines · Al Viro
Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)lib/iov_iter.c +2
-2@@ -1284,7 +1284,8 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,return 0 ; maxsize =min_t (size_t, nr *PAGE_SIZE - offset, maxsize); -iov_iter_advance (i, maxsize); + i->iov_offset += maxsize; + i->count -= maxsize;return maxsize; }@@ -1373,7 +1374,13 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,for (int k =0 ; k < n; k++)get_page (p[k] = page + k); maxsize =min_t (size_t, maxsize, n *PAGE_SIZE - *start); -iov_iter_advance (i, maxsize);
lib/iov_iter.c static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, · line 1386
Quoted 10 lines
lib/iov_iter.c +7
-0+ i->count -= maxsize; + i->iov_offset += maxsize; +if (i->iov_offset == i->bvec->bv_len) { + i->iov_offset =0 ; + i->bvec++; + i->nr_segs--; + }return maxsize; }if (iov_iter_is_pipe (i))
Why do this? iov_iter_advance makes it clearer as to what's going on here.
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH 43/44] pipe_get_pages(): switch to append_pipe()
In reply to Al ViroQuoted 61 lines · Al Viro
now that we are advancing the iterator, there's no need to treat the first page separately - just call
append_pipe()in a loop.Signed-off-by Al Viro <[email protected]>--- lib/iov_iter.c | 36 ++++++++---------------------------- 1 file changed, 8 insertions(+), 28 deletions(-)lib/iov_iter.c +7
-25@@ -1207,10 +1207,10 @@ static ssize_t pipe_get_pages(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned maxpages, size_t *start) { - struct pipe_inode_info *pipe = i->pipe; -unsigned int npages, off, count; + unsigned int npages, count;struct page **p;ssize_t left; +size_t off;if (!sanity (i))return -EFAULT ;@@ -1222,38 +1222,18 @@ static ssize_t pipe_get_pages(struct iov_iter *i,if (!count)return -ENOMEM ; p = *pages; - left = maxsize; - npages =0 ; -if (off) { -struct pipe_buffer *buf =pipe_buf (pipe, pipe->head -1 ); - -get_page (*p++ = buf->page); - left -=PAGE_SIZE - off; -if (left <=0 ) { - buf->len += maxsize; -iov_iter_advance (i, maxsize); -return maxsize; - } - buf->len =PAGE_SIZE ; - npages =1 ; - } -for ( ; npages < count; npages++) { -struct page *page; -unsigned int size =min_t (ssize_t, left,PAGE_SIZE ); - -if (pipe_full (pipe->head, pipe->tail, pipe->max_usage)) -break ; - page =push_anon (pipe, size); +for (npages =0 , left = maxsize ; npages < count; npages++) { +struct page *page =append_pipe (i, left, &off);if (!page)break ;get_page (*p++ = page); - left -= size; +if (left <=PAGE_SIZE - off) +return maxsize; + left -=PAGE_SIZE - off;
lib/iov_iter.c static ssize_t pipe_get_pages(struct iov_iter *i, · line 1239
Quoted 10 lines
lib/iov_iter.c +1
-3}if (!npages)return -EFAULT ; - maxsize -= left; -iov_iter_advance (i, maxsize); -return maxsize; +return maxsize - left; } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
Quoted 319 lines · Al Viro
There's a bunch of pending iov_iter-related work; most of that had
been posted, but only one part got anything resembling a review. Currently it seems to be working, but it obviously needs review and testing.It's split into several subseries; the entire series can be observed
as v5.19-rc2..#work.iov_iter_get_pages. Description follows; individual patches will be posted^Wmailbombed in followups.This stuff is not in -next yet; I'd like to put it there, so if you
see any problems - please yell.One thing not currently in there, but to be added very soon is
iov_iter_find_pages{,_alloc}() - analogue ofiov_iter_get_pages(), except that it only grabs page references for userland-backed flavours. The callers, of course, are responsible for keeping the underlying object(s) alive for as long as they are using the results. Quite a few ofiov_iter_get_pages()callers would be fine with that. Moreover, unlikeiov_iter_get_pages()this could be allowed forITER_KVEC, potentially eliminating several places where we special-case the treatment ofITER_KVEC.Another pending thing is integration with cifs and ceph series (dhowells
and jlayton resp.) and probablyio_uringas well.----------------------------------------------------------------------------
Part 1, #work.9p: [rc1-based]1/44: 9p: handling Rerror without
copy_from_iter_full()
Self-contained fix, should be easy to backport. What happens
there is that arrival of Rerror in response to zerocopy read or readdir ends up with error string in the place where the actual data would've gone in case of success. It needs to be extracted, andcopy_from_iter_full()is only for data-source iterators, not for e.g.ITER_PIPE. AndITER_PIPEcan be used with those...----------------------------------------------------------------------------
Part 2, #work.iov_iter: [rc1-based]Dealing with the overhead in new_sync_read()/new_sync_write(), mostly. Several things there - one is that calculation of iocb flags can be made cheaper, another is that single-segment iovec is sufficiently common to be worth turning into a new
iov_iterflavour (ITER_UBUF). With all that, the total size ofiov_iter.cgoes down, mostly due to removal of magic in iovec copy_page_to_iter()/copy_page_from_iter(). Generic variant works for those nowadays...This had been posted two weeks ago, got a reasonable amount of comments.
2/44: No need of likely/unlikely on calls of
check_copy_size()
not just inuio.h; the thing is inlined and it has unlikely on
all paths leading to return false3/44: teach
iomap_dio_rw()to suppress dsync
new flag foriomap_dio_rw(), telling it to suppressgeneric_write_sync()4/44: btrfs: use
IOMAP_DIO_NOSYNC
use the above instead of currently used kludges.5/44: struct file: use anonymous union member for rcuhead and llist
"f_u" might have been an amusing name, but... we expect anon unions to
work.6/44: iocb: delay evaluation of
IS_SYNC(...)until we want to checkIOCB_DSYNC
makesiocb_flags()much cheaper, and it's easier to keep track of
the places where it can change.7/44: keep
iocb_flags()result cached in struct file
that, along with the previous commit, reduces the overhead of
new_sync_{read,write}(). struct file doesn't grow - we can keep that thing in the same anon union where rcuhead and llist live; that field gets used only before ->f_count reaches zero while the other two are used only after ->f_count has reached zero.8/44: copy_page_{to,from}_iter(): switch iovec variants to generic
kmap_local_page()allows that. And it kills quite a bit of
code.9/44: new
iov_iterflavour -ITER_UBUF
iovec analogue, with single segment. That case is fairly common and it
can be handled with less overhead than full-blown iovec.10/44: switch new_sync_{read,write}() to
ITER_UBUF
... and this is why it is so common. Further reduction of overhead
for new_sync_{read,write}().11/44:
iov_iter_bvec_advance(): don't bother withbvec_iter
AFAICS, variant similar to what we do for iovec/kvec generates better
code. Needs profiling, obviously.----------------------------------------------------------------------------
Part 3, #fixes [-rc2-based]12/44: fix short copy handling in
copy_mc_pipe_to_iter()
Minimal version of fix; it's replaced with prettier one in the next
series, but replacement is not a backport fodder.----------------------------------------------------------------------------
Part 4, #work.ITER_PIPE [on top of merge of previous branches]
ITER_PIPEhandling had never been pretty, but by now it has become really obfuscated and hard to read. Untangle it a bit. Posted last weekend, some brainos fixed since then.13/44: splice: stop abusing
iov_iter_advance()to flush a pipe
A really odd (ab)use ofiov_iter_advance()- in case of errorgeneric_file_splice_read()wants to free all pipe buffers ->read_iter() has produced. Yes, forcibly resetting ->head and ->iov_offset to original values and callingiov_iter_advance(i, 0)will triggerpipe_advance(), which will triggerpipe_truncate(), which will free buffers. Or we could just go ahead and free the same buffers;pipe_discard_from()does exactly that, noiov_iterstuff needs to be involved.14/44:
ITER_PIPE: helper for getting pipe buffer by indexIn a lot of places we want to find pipe_buffer by index;expression is convoluted and hard to read. Provide an inline helper for that, convert trivial open-coded cases. Eventually *all* open-coded instances in
iov_iter.cwill be gone.15/44:
ITER_PIPE: helpers for adding pipe buffers
There are only two kinds ofpipe_bufferin the area used byITER_PIPE.
* anonymous -copy_to_iter()et.al. end up creating those and copying data there. They have zero ->offset, and their ->ops points todefault_pipe_page_ops.
* zero-copy ones - those come fromcopy_page_to_iter(), and page comes from caller. ->offset is also caller-supplied - it might be non-zero. ->ops points topage_cache_pipe_buf_ops.
Move creation and insertion of those into helpers -push_anon(pipe, size)andpush_page(pipe, page, offset, size)resp., separating them from the "could we avoid creating a new buffer by merging with the current head?" logics.16/44:
ITER_PIPE: allocate buffers as we go in copy-to-pipe primitives
New helper:append_pipe(). Extends the last buffer if possible,
allocates a new one otherwise. Returns page and offset in it on success, NULL on failure.iov_iteris advanced past the data we've got.Use that instead of push_pipe() in copy-to-pipe primitives;they get simpler that way. Handling of short copy (in "mc" one) is done simply by
iov_iter_revert()-iov_iteris in consistent state after that one, so we can use that.17/44:
ITER_PIPE: foldpush_pipe()into__pipe_get_pages()
Expand the only remaining call ofpush_pipe()(in__pipe_get_pages()), combine it with the page-collecting loop there. We don't need to bother withi->countchecks or calculation of offset in the first page - the caller already has done that.
Note that the only reason it's not a loop doingappend_pipe()
is thatappend_pipe()is advancing, whileiov_iter_get_pages()is not. As soon as it switches to saner semantics, this thing will switch to usingappend_pipe().18/44:
ITER_PIPE: loseiter_headargument of__pipe_get_pages()
Redundant.19/44:
ITER_PIPE: cleanpipe_advance()upDon't bother with pipe_truncate(); adjust the bufferlength just as we decide it'll be the last one, then use
pipe_discard_from()to release buffers past that one.20/44:
ITER_PIPE: cleaniov_iter_revert()
Foldpipe_truncate()in there, clean the things up.21/44:
ITER_PIPE: cache the type of last buffer
We often need to find whether the last buffer is anon or not, and
currently it's rather clumsy:check if ->iov_offset is non-zero (i.e. that pipe is not empty) if so, get the corresponding pipe_buffer and check its ->ops if it's &default_pipe_buf_ops, we have an anon buffer.Let's replace the use of ->iov_offset (which is nowhere near similar to its role for other flavours) with signed field (->last_offset), with the following rules:
empty, no buffers occupied: 0
anon, with bytes up to N-1 filled: N
zero-copy, with bytes up to N-1 filled: -N
That wayabs(i->last_offset)is equal to what used to be ini->iov_offsetand empty vs. anon vs. zero-copy can be distinguished by the sign ofi->last_offset.
Checks for "should we extend the last buffer or should we start
a new one?" become easier to follow that way.
Note that most of the operations can only be done in a sane
state - i.e. when the pipe has nothing past the current position of iterator. About the only thing that could be done outside of that state isiov_iter_advance(), which transitions to the sane state by truncating the pipe. There are only two cases where we leave the sane state:
1) iov_iter_get_pages()/iov_iter_get_pages_alloc(). Will be
dealt with later, when we makeget_pagesadvancing - the callers are actually happier that way.
2)iov_itercopied, then something is put into the copy. Since
they share the underlying pipe, the original gets behind. When we decide that we are done with the copy (original is not usable until then) we advance the original.direct_ioused to be done that way; nowadays it operates on the original and we doiov_iter_revert()to discard the excessive data. At the moment there's nothing in the kernel that could do that toITER_PIPEiterators, so this reason for insane state is theoretical right now.22/44:
ITER_PIPE: folddata_start()andpipe_space_for_user()together
All their callers are next to each other; all of them want
the total amount of pages and, possibly, the offset in the partial final buffer.
Combine into a new helper (pipe_npages()), fix the
bogosity inpipe_space_for_user(), while we are at it.----------------------------------------------------------------------------
Part 5, #work.unify_iov_iter_get_pages [on top of previous]
iov_iter_get_pages()andiov_iter_get_pages_alloc()have a lot of code duplication and are bloody hard to read. With some massage duplication can be eliminated, along with some of the cruft accumulated there.Flavour-independent arguments validation and, for ..._alloc(),
cleanup handling on failure:
23/44: iov_iter_get_pages{,_alloc}(): cap the maxsize withMAX_RW_COUNT24/44:iov_iter_get_pages_alloc(): lift freeing pages array on failure exits into wrapper 25/44:iov_iter_get_pages(): sanity-check argumentsMechanically merge parallel ..._get_pages() and ..._get_pages_alloc().
26/44: unifypipe_get_pages()andpipe_get_pages_alloc()
27/44: unifyxarray_get_pages()andxarray_get_pages_alloc()28/44: unify the rest of iov_iter_get_pages()/iov_iter_get_pages_alloc() gutsDecrufting for XARRAY:
29/44:ITER_XARRAY: don't open-codeDIV_ROUND_UP()Decrufting for UBUF/IOVEC/BVEC: that bunch suffers from really convoluted
helpers; untangling those takes a bit of care, so I'd carved that up into fairly small chunks. Could be collapsed together, but...
30/44:iov_iter: lift dealing with maxpages out of first_{iovec,bvec}_segment() 31/44:iov_iter: first_{iovec,bvec}_segment() - simplify a bit 32/44:iov_iter: massage calling conventions for first_{iovec,bvec}_segment() 33/44:found_iovec_segment(): just return addressDecrufting for PIPE:
34/44: fold__pipe_get_pages()intopipe_get_pages()Now we can finally get a helper encapsulating the array allocations
right way:
35/44:iov_iter: saner helper for page array allocation----------------------------------------------------------------------------
Part 6, #work.iov_iter_get_pages-advance [on top of previous]
Convert iov_iter_get_pages{,_alloc}() to iterator-advancing semantics.Most of the callers follow successful ...get_pages... with advance
by the amount it had reported. For some it's unconditional, for some it might end up being less in some cases. All of them would be fine with advancing variants of those primitives - those that might want to advance by less than reported could easily use revert by the difference of those amounts.
Rather than doing a flagday change (they are exported and signatures
remain unchanged), replacement variants are added (iov_iter_get_pages2()andiov_iter_get_pages_alloc2(), initially as wrappers). By the end of the series everything is converted to those and the old ones are removed.Makes for simpler rules for
ITER_PIPE, among other things, and
advancing semantics is consistent with all data-copying primitives. Series is pretty obvious - introduce variants with new semantics, switch users one by one, fold the old variants into new ones.36/44:
iov_iter: advancing variants of iov_iter_get_pages{,_alloc}() 37/44: block: convert to advancing variants of iov_iter_get_pages{,_alloc}() 38/44:iter_to_pipe(): switch to advancing variant ofiov_iter_get_pages()39/44:af_alg_make_sg(): switch to advancing variant ofiov_iter_get_pages()40/44: 9p: convert to advancing variant ofiov_iter_get_pages_alloc()41/44: ceph: switch the last caller ofiov_iter_get_pages_alloc()42/44: get rid of non-advancing variants----------------------------------------------------------------------------
Part 7, #wort.iov_iter_get_pages [on top of previous]
Trivial followups, with more to be added here...43/44:
pipe_get_pages(): switch toappend_pipe()
44/44: expand thoseiov_iter_advance()...Overall diffstat:
arch/powerpc/include/asm/uaccess.h | 2 +- arch/s390/include/asm/uaccess.h | 4 +- block/bio.c | 15 +- block/blk-map.c | 7 +- block/fops.c | 8 +- crypto/af_alg.c | 3 +- crypto/algif_hash.c | 5 +- drivers/nvme/target/io-cmd-file.c | 2 +- drivers/vhost/scsi.c | 4 +- fs/aio.c | 2 +- fs/btrfs/file.c | 19 +- fs/btrfs/inode.c | 3 +- fs/ceph/addr.c | 2 +- fs/ceph/file.c | 5 +- fs/cifs/file.c | 8 +- fs/cifs/misc.c | 3 +- fs/direct-io.c | 7 +- fs/fcntl.c | 1 + fs/file_table.c | 17 +- fs/fuse/dev.c | 7 +- fs/fuse/file.c | 7 +- fs/gfs2/file.c | 2 +- fs/io_uring.c | 2 +- fs/iomap/direct-io.c | 21 +- fs/nfs/direct.c | 8 +- fs/open.c | 1 + fs/read_write.c | 6 +- fs/splice.c | 54 +- fs/zonefs/super.c | 2 +- include/linux/fs.h | 21 +- include/linux/iomap.h | 6 + include/linux/pipe_fs_i.h | 29 +- include/linux/uaccess.h | 4 +- include/linux/uio.h | 50 +- lib/iov_iter.c | 993 ++++++++++++++----------------------- mm/shmem.c | 2 +-
Quoted 8 lines
net/9p/client.c | 125 +---- net/9p/protocol.c | 3 +- net/9p/trans_virtio.c | 37 +- net/core/datagram.c | 3 +- net/core/skmsg.c | 3 +- net/rds/message.c | 3 +- net/tls/tls_sw.c | 4 +- 43 files changed, 599 insertions(+), 911 deletions(-)
I ported the CEPH_MSG_DATA_ITER patches on top of this, and ran ceph through xfstests and it seemed to do just fine. You can add:
Re: [PATCH 08/44] copy_page_{to,from}_iter(): switch iovec variants to generic
In reply to Al ViroQuoted 5 lines · Al Viro
we can do copyin/copyout under
kmap_local_page(); it shouldn't overflow the kmap stack - the maximal footprint increase only by one here.Signed-off-by Al Viro <[email protected]>---
Assuming the WARN_ON(1) removals are intentional,
Quoted 232 lines
lib/iov_iter.c | 191 ++----------------------------------------------- 1 file changed, 4 insertions(+), 187 deletions(-)lib/iov_iter.c +4
-187@@ -168,174 +168,6 @@ static int copyin(void *to, const void __user *from, size_t n)return n; } -static size_t copy_page_to_iter_iovec (struct page *page,size_t offset,size_t bytes, -struct iov_iter *i) -{ -size_t skip, copy, left, wanted; -const struct iovec *iov; -char __user *buf; -void *kaddr, *from; - -if (unlikely (bytes > i->count)) - bytes = i->count; - -if (unlikely (!bytes)) -return 0 ; - -might_fault (); - wanted = bytes; - iov = i->iov; - skip = i->iov_offset; - buf = iov->iov_base + skip; - copy =min (bytes, iov->iov_len - skip); - -if (IS_ENABLED (CONFIG_HIGHMEM ) && !fault_in_writeable (buf, copy)) { - kaddr =kmap_atomic (page); - from = kaddr + offset; - -/* first chunk, usually the only one */ - left =copyout (buf, from, copy); - copy -= left; - skip += copy; - from += copy; - bytes -= copy; - -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyout (buf, from, copy); - copy -= left; - skip = copy; - from += copy; - bytes -= copy; - } -if (likely (!bytes)) { -kunmap_atomic (kaddr); - goto done; - } - offset = from - kaddr; - buf += copy; -kunmap_atomic (kaddr); - copy =min (bytes, iov->iov_len - skip); - } -/* Too bad - revert to non-atomic kmap */ - - kaddr =kmap (page); - from = kaddr + offset; - left =copyout (buf, from, copy); - copy -= left; - skip += copy; - from += copy; - bytes -= copy; -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyout (buf, from, copy); - copy -= left; - skip = copy; - from += copy; - bytes -= copy; - } -kunmap (page); - -done: -if (skip == iov->iov_len) { - iov++; - skip =0 ; - } - i->count -= wanted - bytes; - i->nr_segs -= iov - i->iov; - i->iov = iov; - i->iov_offset = skip; -return wanted - bytes; -} - -static size_t copy_page_from_iter_iovec (struct page *page,size_t offset,size_t bytes, -struct iov_iter *i) -{ -size_t skip, copy, left, wanted; -const struct iovec *iov; -char __user *buf; -void *kaddr, *to; - -if (unlikely (bytes > i->count)) - bytes = i->count; - -if (unlikely (!bytes)) -return 0 ; - -might_fault (); - wanted = bytes; - iov = i->iov; - skip = i->iov_offset; - buf = iov->iov_base + skip; - copy =min (bytes, iov->iov_len - skip); - -if (IS_ENABLED (CONFIG_HIGHMEM ) && !fault_in_readable (buf, copy)) { - kaddr =kmap_atomic (page); - to = kaddr + offset; - -/* first chunk, usually the only one */ - left =copyin (to, buf, copy); - copy -= left; - skip += copy; - to += copy; - bytes -= copy; - -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyin (to, buf, copy); - copy -= left; - skip = copy; - to += copy; - bytes -= copy; - } -if (likely (!bytes)) { -kunmap_atomic (kaddr); - goto done; - } - offset = to - kaddr; - buf += copy; -kunmap_atomic (kaddr); - copy =min (bytes, iov->iov_len - skip); - } -/* Too bad - revert to non-atomic kmap */ - - kaddr =kmap (page); - to = kaddr + offset; - left =copyin (to, buf, copy); - copy -= left; - skip += copy; - to += copy; - bytes -= copy; -while (unlikely (!left && bytes)) { - iov++; - buf = iov->iov_base; - copy =min (bytes, iov->iov_len); - left =copyin (to, buf, copy); - copy -= left; - skip = copy; - to += copy; - bytes -= copy; - } -kunmap (page); - -done: -if (skip == iov->iov_len) { - iov++; - skip =0 ; - } - i->count -= wanted - bytes; - i->nr_segs -= iov - i->iov; - i->iov = iov; - i->iov_offset = skip; -return wanted - bytes; -} -#ifdef PIPE_PARANOIA static bool sanity (const struct iov_iter *i) {@@ -848,24 +680,14 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n)static size_t __copy_page_to_iter (struct page *page,size_t offset,size_t bytes,struct iov_iter *i) { -if (likely (iter_is_iovec (i))) -return copy_page_to_iter_iovec (page, offset, bytes, i); -if (iov_iter_is_bvec (i) ||iov_iter_is_kvec (i) ||iov_iter_is_xarray (i)) { +if (unlikely (iov_iter_is_pipe (i))) { +return copy_page_to_iter_pipe (page, offset, bytes, i); + }else {void *kaddr =kmap_local_page (page);size_t wanted =_copy_to_iter (kaddr + offset, bytes, i);kunmap_local (kaddr);return wanted; } -if (iov_iter_is_pipe (i)) -return copy_page_to_iter_pipe (page, offset, bytes, i); -if (unlikely (iov_iter_is_discard (i))) { -if (unlikely (i->count < bytes)) - bytes = i->count; - i->count -= bytes; -return bytes; - } -WARN_ON (1 ); -return 0 ; } size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes,@@ -896,17 +718,12 @@ EXPORT_SYMBOL(copy_page_to_iter);size_t copy_page_from_iter (struct page *page,size_t offset,size_t bytes,struct iov_iter *i) { -if (unlikely (!page_copy_sane (page, offset, bytes))) -return 0 ; -if (likely (iter_is_iovec (i))) -return copy_page_from_iter_iovec (page, offset, bytes, i); -if (iov_iter_is_bvec (i) ||iov_iter_is_kvec (i) ||iov_iter_is_xarray (i)) { +if (page_copy_sane (page, offset, bytes)) {void *kaddr =kmap_local_page (page);size_t wanted =_copy_from_iter (kaddr + offset, bytes, i);kunmap_local (kaddr);return wanted; } -WARN_ON (1 );return 0 ; }EXPORT_SYMBOL (copy_page_from_iter);Signature · 1 line
2.30.2
Christian Brauner
maintainerRe: [PATCH 09/44] new iov_iter flavour - ITER_UBUF
In reply to Al ViroQuoted 398 lines · Al Viro
Equivalent of single-segment iovec. Initialized by
iov_iter_ubuf(), checked for byiter_is_ubuf(), otherwise behaves likeITER_IOVECones.We are going to expose the things like ->write_iter() et.al. to those in subsequent commits.
New predicate (
user_backed_iter()) that is true forITER_IOVECandITER_UBUF; places like direct-IO handling should use that for checking that pages we modify after getting them fromiov_iter_get_pages()would need to be dirtied.DO NOT assume that replacing
iter_is_iovec()withuser_backed_iter()will solve all problems - there's code that usesiter_is_iovec()to decide how to poke around iniov_iterguts and for that the predicate replacement obviously won't suffice.Signed-off-by Al Viro <[email protected]>--- block/fops.c | 6 +-- fs/ceph/file.c | 2 +- fs/cifs/file.c | 2 +- fs/direct-io.c | 2 +- fs/fuse/dev.c | 4 +- fs/fuse/file.c | 2 +- fs/gfs2/file.c | 2 +- fs/iomap/direct-io.c | 2 +- fs/nfs/direct.c | 2 +- include/linux/uio.h | 26 ++++++++++++ lib/iov_iter.c | 94 ++++++++++++++++++++++++++++++++++---------- mm/shmem.c | 2 +- 12 files changed, 113 insertions(+), 33 deletions(-)block/fops.c +3
-3@@ -69,7 +69,7 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,if (iov_iter_rw (iter) ==READ ) {bio_init (&bio, bdev, vecs, nr_pages,REQ_OP_READ ); -if (iter_is_iovec (iter)) +if (user_backed_iter (iter)) should_dirty = true; }else {bio_init (&bio, bdev, vecs, nr_pages,dio_bio_write_op (iocb));@@ -199,7 +199,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, } dio->size =0 ; -if (is_read &&iter_is_iovec (iter)) +if (is_read &&user_backed_iter (iter)) dio->flags |=DIO_SHOULD_DIRTY ;blk_start_plug (&plug);@@ -331,7 +331,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, dio->size = bio->bi_iter.bi_size;if (is_read) { -if (iter_is_iovec (iter)) { +if (user_backed_iter (iter)) { dio->flags |=DIO_SHOULD_DIRTY ;bio_set_pages_dirty (bio); }fs/ceph/file.c +1
-1@@ -1262,7 +1262,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,size_t count =iov_iter_count (iter);loff_t pos = iocb->ki_pos;bool write =iov_iter_rw (iter) ==WRITE ; -bool should_dirty = !write &&iter_is_iovec (iter); +bool should_dirty = !write &&user_backed_iter (iter);if (write &&ceph_snap (file_inode (file)) !=CEPH_NOSNAP )return -EROFS ;fs/cifs/file.c +1
-1@@ -4004,7 +4004,7 @@ static ssize_t __cifs_readv(if (!is_sync_kiocb (iocb)) ctx->iocb = iocb; -if (iter_is_iovec (to)) +if (user_backed_iter (to)) ctx->should_dirty = true;if (direct) {fs/direct-io.c +1
-1@@ -1245,7 +1245,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode,spin_lock_init (&dio->bio_lock); dio->refcount =1 ; - dio->should_dirty =iter_is_iovec (iter) &&iov_iter_rw (iter) ==READ ; + dio->should_dirty =user_backed_iter (iter) &&iov_iter_rw (iter) ==READ ; sdio.iter = iter; sdio.final_block_in_request = end >> blkbits;fs/fuse/dev.c +2
-2@@ -1356,7 +1356,7 @@ static ssize_t fuse_dev_read(struct kiocb *iocb, struct iov_iter *to)if (!fud)return -EPERM ; -if (!iter_is_iovec (to)) +if (!user_backed_iter (to))return -EINVAL ;fuse_copy_init (&cs,1 , to);@@ -1949,7 +1949,7 @@ static ssize_t fuse_dev_write(struct kiocb *iocb, struct iov_iter *from)if (!fud)return -EPERM ; -if (!iter_is_iovec (from)) +if (!user_backed_iter (from))return -EINVAL ;fuse_copy_init (&cs,0 , from);fs/fuse/file.c +1
-1@@ -1465,7 +1465,7 @@ ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter,inode_unlock (inode); } - io->should_dirty = !write &&iter_is_iovec (iter); + io->should_dirty = !write &&user_backed_iter (iter);while (count) {ssize_t nres;fl_owner_t owner = current->files;fs/gfs2/file.c +1
-1@@ -780,7 +780,7 @@ static inline bool should_fault_in_pages(struct iov_iter *i,if (!count)return false; -if (!iter_is_iovec (i)) +if (!user_backed_iter (i))return false; size =PAGE_SIZE ;fs/iomap/direct-io.c +1
-1@@ -533,7 +533,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomi.flags |=IOMAP_NOWAIT ; } -if (iter_is_iovec (iter)) +if (user_backed_iter (iter)) dio->flags |=IOMAP_DIO_DIRTY ; }else { iomi.flags |=IOMAP_WRITE ;fs/nfs/direct.c +1
-1@@ -478,7 +478,7 @@ ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter,if (!is_sync_kiocb (iocb)) dreq->iocb = iocb; -if (iter_is_iovec (iter)) +if (user_backed_iter (iter)) dreq->flags =NFS_ODIRECT_SHOULD_DIRTY ;if (!swap)include/linux/uio.h +26
-0@@ -26,6 +26,7 @@ enum iter_type {ITER_PIPE ,ITER_XARRAY ,ITER_DISCARD , + ITER_UBUF, }; struct iov_iter_state {@@ -38,6 +39,7 @@ struct iov_iter {u8 iter_type;bool nofault;bool data_source; +bool user_backed;size_t iov_offset;size_t count;union {@@ -46,6 +48,7 @@ struct iov_iter {const struct bio_vec *bvec;struct xarray *xarray;struct pipe_inode_info *pipe; +void __user *ubuf; };union {unsigned long nr_segs;@@ -70,6 +73,11 @@ static inline void iov_iter_save_state(struct iov_iter *iter, state->nr_segs = iter->nr_segs; } +static inline bool iter_is_ubuf (const struct iov_iter *i) +{ +return iov_iter_type (i) ==ITER_UBUF ; +} +static inline bool iter_is_iovec (const struct iov_iter *i) {return iov_iter_type (i) ==ITER_IOVEC ;@@ -105,6 +113,11 @@ static inline unsigned char iov_iter_rw(const struct iov_iter *i)return i->data_source ?WRITE :READ ; } +static inline bool user_backed_iter (const struct iov_iter *i) +{ +return i->user_backed; +} +/* * Total number of bytes covered by an iovec. * @@ -320,4 +333,17 @@ ssize_t __import_iovec(int type, const struct iovec __user *uvec,int import_single_range (int type,void __user *buf,size_t len,struct iovec *iov,struct iov_iter *i); +static inline void iov_iter_ubuf (struct iov_iter *i,unsigned int direction, +void __user *buf,size_t count) +{ +WARN_ON (direction & ~(READ |WRITE )); + *i = (struct iov_iter ) { + .iter_type =ITER_UBUF , + .user_backed = true, + .data_source = direction, + .ubuf = buf, + .count = count + }; +} +#endif lib/iov_iter.c +45
-11@@ -16,6 +16,16 @@#define PIPE_PARANOIA /* for now */ +/* covers ubuf and kbuf alike */ +#define iterate_buf (i, n, base, len, off, __p,STEP ) { \ + size_t __maybe_unused off = 0; \ + len = n; \ + base = __p + i->iov_offset; \ + len -= (STEP); \ + i->iov_offset += len; \ + n = len; \ +} +/* covers iovec and kvec alike */ #define iterate_iovec (i, n, base, len, off, __p,STEP ) { \ size_t off = 0; \@@ -110,7 +120,12 @@ __out: \if (unlikely (i->count < n)) \ n = i->count; \if (likely (n)) { \ -if (likely (iter_is_iovec (i))) { \ +if (likely (iter_is_ubuf (i))) { \ + void __user *base; \ +size_t len; \ +iterate_buf (i, n, base, len, off, \ + i->ubuf, (I)) \ + }else if (likely (iter_is_iovec (i))) { \const struct iovec *iov = i->iov; \ void __user *base; \size_t len; \@@ -275,7 +290,11 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t by*/ size_t fault_in_iov_iter_readable (const struct iov_iter *i,size_t size) { -if (iter_is_iovec (i)) { +if (iter_is_ubuf (i)) { +size_t n =min (size,iov_iter_count (i)); + n -=fault_in_readable (i->ubuf + i->iov_offset, n); +return size - n; + }else if (iter_is_iovec (i)) {size_t count =min (size,iov_iter_count (i));const struct iovec *p;size_t skip;@@ -314,7 +333,11 @@ EXPORT_SYMBOL(fault_in_iov_iter_readable);*/ size_t fault_in_iov_iter_writeable (const struct iov_iter *i,size_t size) { -if (iter_is_iovec (i)) { +if (iter_is_ubuf (i)) { +size_t n =min (size,iov_iter_count (i)); + n -=fault_in_safe_writeable (i->ubuf + i->iov_offset, n); +return size - n; + }else if (iter_is_iovec (i)) {size_t count =min (size,iov_iter_count (i));const struct iovec *p;size_t skip;@@ -345,6 +368,7 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, *i = (struct iov_iter ) { .iter_type =ITER_IOVEC , .nofault = false, + .user_backed = true, .data_source = direction, .iov = iov, .nr_segs = nr_segs,@@ -494,7 +518,7 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) {if (unlikely (iov_iter_is_pipe (i)))return copy_pipe_to_iter (addr, bytes, i); -if (iter_is_iovec (i)) +if (user_backed_iter (i))might_fault (); iterate_and_advance(i, bytes, base, len, off, copyout(base, addr + off, len),@@ -576,7 +600,7 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) {if (unlikely (iov_iter_is_pipe (i)))return copy_mc_pipe_to_iter (addr, bytes, i); -if (iter_is_iovec (i)) +if (user_backed_iter (i))might_fault (); __iterate_and_advance(i, bytes, base, len, off, copyout_mc(base, addr + off, len),@@ -594,7 +618,7 @@ size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i)WARN_ON (1 );return 0 ; } -if (iter_is_iovec (i)) +if (user_backed_iter (i))might_fault (); iterate_and_advance(i, bytes, base, len, off, copyin(addr + off, base, len),@@ -882,16 +906,16 @@ void iov_iter_advance(struct iov_iter *i, size_t size) {if (unlikely (i->count < size)) size = i->count; -if (likely (iter_is_iovec (i) ||iov_iter_is_kvec (i))) { +if (likely (iter_is_ubuf (i)) ||unlikely (iov_iter_is_xarray (i))) { + i->iov_offset += size; + i->count -= size; + }else if (likely (iter_is_iovec (i) ||iov_iter_is_kvec (i))) {/* iovec and kvec have identical layouts */ iov_iter_iovec_advance (i, size); }else if (iov_iter_is_bvec (i)) {iov_iter_bvec_advance (i, size); }else if (iov_iter_is_pipe (i)) {pipe_advance (i, size); - }else if (unlikely (iov_iter_is_xarray (i))) { - i->iov_offset += size; - i->count -= size; }else if (iov_iter_is_discard (i)) { i->count -= size; }@@ -938,7 +962,7 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)return ; } unroll -= i->iov_offset; -if (iov_iter_is_xarray (i)) { +if (iov_iter_is_xarray (i) ||iter_is_ubuf (i)) {BUG ();/* We should never go beyond the start of the specified * range since we might then be straying into pages that * aren't pinned. @@ -1129,6 +1153,13 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)unsigned long iov_iter_alignment (const struct iov_iter *i) { +if (likely (iter_is_ubuf (i))) { +size_t size = i->count; +if (size) +return ((unsigned long )i->ubuf + i->iov_offset) | size; +return 0 ; + } +/* iovec and kvec have identical layouts */ if (likely (iter_is_iovec (i) ||iov_iter_is_kvec (i)))return iov_iter_alignment_iovec (i);@@ -1159,6 +1190,9 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i)size_t size = i->count;unsigned k; +if (iter_is_ubuf (i)) +return 0 ; +if (WARN_ON (!iter_is_iovec (i)))return ~0U ;
lib/iov_iter.c static ssize_t iter_xarray_get_pages(struct iov_iter *i, · line 1329
Quoted 10 lines
lib/iov_iter.c +6
-1@@ -1287,7 +1321,19 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i,return actual; } -/* must be done on non-empty ITER_IOVEC one */ +static unsigned long found_ubuf_segment (unsigned long addr, +size_t len, +size_t *size,size_t *start, +unsigned maxpages) +{ + len += (*start = addr %PAGE_SIZE );
Ugh, I know you just copy-pasted this but can we rewrite this to:
*start = addr % PAGE_SIZE; len += *start;
I think that's easier to read.
Re: [PATCH 11/44] iov_iter_bvec_advance(): don't bother with bvec_iter
In reply to Al ViroQuoted 5 lines · Al Viro
do what we do for iovec/kvec; that ends up generating better code, AFAICS.
Signed-off-by Al Viro <[email protected]>---
Looks good to me,
Re: [PATCH 12/44] fix short copy handling in copy_mc_pipe_to_iter()
In reply to Al ViroQuoted 12 lines · Al Viro
Unlike other copying operations on
ITER_PIPE,copy_mc_to_iter()can result in a short copy. In that case we need to trim the unused buffers, as well as the length of partially filled one - it's not enough to set ->head, ->iov_offset and ->count to reflect how much had we copied. Not hard to fix, fortunately...I'd put a helper (
pipe_discard_from(pipe, head)) intopipe_fs_i.h, rather thaniov_iter.c- it has nothing to do withiov_iterand having it will allow us to avoid an ugly kludge infs/splice.c. We could put it intolib/iov_iter.cfor now and move it later, but I don't see the point going that way...
Quoted 3 lines
Fixes ca146f6f091e "lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe()"Signed-off-by Al Viro <[email protected]>---
Does that need a
or sm?
Re: [PATCH 13/44] splice: stop abusing iov_iter_advance() to flush a pipe
In reply to Al ViroQuoted 5 lines · Al Viro
Use
pipe_discard_from()explicitly ingeneric_file_read_iter(); don't bother with rather non-obvious use ofiov_iter_advance()in there.Signed-off-by Al Viro <[email protected]>---
Looks good to me,
Re: [PATCH 14/44] ITER_PIPE: helper for getting pipe buffer by index
In reply to Al ViroQuoted 13 lines · Al Viro
pipe_bufferinstances of a pipe are organized as a ring buffer, with power-of-2 size. Indices are kept *not* reduced modulo ring size, so the buffer refered to by index N ispipe->bufs[N & (pipe->ring_size - 1)].Ring size can change over the lifetime of a pipe, but not while the pipe is locked. So for any
iov_iterprimitives it's a constant. Original conversion of pipes to this layout went overboard trying to microoptimize that - calculatingpipe->ring_size- 1, storing it in a local variable and using through the function. In some cases it might be warranted, but most of the times it only obfuscates what's going on in there.
Quoted 5 lines
Introduce a helper (
pipe_buf(pipe, N)) that would encapsulate that and use it in the obvious cases. More will follow...Signed-off-by Al Viro <[email protected]>---
Looks good to me,
Al Viro
maintainerRe: [PATCH 08/44] copy_page_{to,from}_iter(): switch iovec variants to generic
In reply to Christian BraunerQuoted 9 lines · Christian Brauner
Quoted 5 lines · Al Viro
we can do copyin/copyout under
kmap_local_page(); it shouldn't overflow the kmap stack - the maximal footprint increase only by one here.Signed-off-by Al Viro <[email protected]>---
Assuming the
WARN_ON(1)removals are intentional,
Deliberate - it shouldn't be any different from what _copy_to_iter() and _copy_from_iter() are ready to handle.
Al Viro
maintainerRe: [PATCH 09/44] new iov_iter flavour - ITER_UBUF
In reply to Jeff LaytonQuoted 3 lines · Jeff Layton
The code looks reasonable but is there any real benefit here? It seems like the only user of it so far is new_sync_{read,write}, and both seem to just use it to avoid allocating a single iovec on the stack.
Not really - for one thing, it's less overhead in data-copying primitives, for another... Jens had plans for it as well. It's not as simple as "just use it whenever you are asked for a single-segment iovec", but...
Al Viro
maintainerRe: [PATCH 09/44] new iov_iter flavour - ITER_UBUF
In reply to Christian BraunerQuoted 10 lines · Christian Brauner
lib/iov_iter.c static ssize_t iter_xarray_get_pages(struct iov_iter *i, · line 1329
Quoted 7 lines
lib/iov_iter.c +6
-1-/* must be done on non-empty ITER_IOVEC one */ +static unsigned long found_ubuf_segment (unsigned long addr, +size_t len, +size_t *size,size_t *start, +unsigned maxpages) +{ + len += (*start = addr %PAGE_SIZE );Ugh, I know you just copy-pasted this but can we rewrite this to:
Quoted 4 lines
*start = addr % PAGE_SIZE; len += *start;I think that's easier to read.
Dealt with later in the series (around the unification and cleanups of iov_iter_get_pages/iov_iter_get_pages_alloc). We could do that first, but I'd rather not mix that massage in here.
Al Viro
maintainer[block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al ViroQuoted 3 lines · Al Viro
... doing revert if we end up not using some pages
Signed-off-by Al Viro <[email protected]>
... and the first half of that thing conflicts with "block: relax direct io memory alignment" in -next...
Joy. It's not hard to redo on top of the commit in there; the question is, how to deal with conflicts?
I can do a backmerge, provided that there's a sane tag or branch to backmerge from. Another fun (if trivial) issue in the same series is around "iov: introduce iov_iter_aligned" (two commits prior).
Jens, Keith, do you have any suggestions? AFAICS, variants include
* tag or branch covering b1a000d3b8ec582da64bb644be633e5a0beffcbf
(I'd rather not grab the entire for-5.20/block for obvious reasons) It sits in the beginning of for-5.20/block, so that should be fairly straightforward, provided that you are not going to do rebases there. If you are, could you put that stuff into an invariant branch, so I'd just pull it?
* feeding the entire iov_iter pile through block.git;
bad idea, IMO, seeing that it contains a lot of stuff far from anything block-related.
* doing a manual conflict resolution on top of my branch
and pushing that out. Would get rid of the problem from -next, but Linus hates that kind of stuff, AFAIK, and with good reasons.
I would prefer the first variant (and that's what I'm
going to do locally for now - just
git tag keith_stuff bf8d08532bc19a14cfb54ae61099dccadefca446 and backmerge from it), but if you would prefer to deal with that differently - please tell.
Al Viro
maintainerRe: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
Quoted 2 lines · Al Viro
... and the first half of that thing conflicts with "block: relax direct io memory alignment" in -next...
BTW, looking at that commit - are you sure that bio_put_pages() on failure exit will do the right thing? We have grabbed a bunch of page references; the amount if DIV_ROUND_UP(offset + size, PAGE_SIZE). And that's before your
size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));
in there. IMO the following would be more obviously correct:
size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset);
if (unlikely(size <= 0))
return size ? size : -EFAULT;
nr_pages = DIV_ROUND_UP(size + offset, PAGE_SIZE);
size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));
for (left = size, i = 0; left > 0; left -= len, i++) {
...
if (ret) {
while (i < nr_pages)
put_page(pages[i++]);
return ret;
}...
and get rid of bio_put_pages() entirely. Objections?
Jens Axboe
maintainerRe: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al ViroQuoted 29 lines · Al Viro
Quoted 3 lines · Al Viro
... doing revert if we end up not using some pages
Signed-off-by Al Viro <[email protected]>... and the first half of that thing conflicts with "block: relax direct io memory alignment" in -next...
Joy. It's not hard to redo on top of the commit in there; the question is, how to deal with conflicts?
I can do a backmerge, provided that there's a sane tag or branch to backmerge from. Another fun (if trivial) issue in the same series is around "iov: introduce
iov_iter_aligned" (two commits prior).Jens, Keith, do you have any suggestions? AFAICS, variants include
* tag or branch coveringb1a000d3b8ec582da64bb644be633e5a0beffcbf
(I'd rather not grab the entire for-5.20/block for obvious reasons) It sits in the beginning of for-5.20/block, so that should be fairly straightforward, provided that you are not going to do rebases there. If you are, could you put that stuff into an invariant branch, so I'd just pull it?
* feeding the entireiov_iterpile through block.git;
bad idea, IMO, seeing that it contains a lot of stuff far from anything block-related.
* doing a manual conflict resolution on top of my branch
and pushing that out. Would get rid of the problem from -next, but Linus hates that kind of stuff, AFAIK, and with good reasons.
Quoted 5 lines
I would prefer the first variant (and that's what I'm
going to do locally for now - just
git tagkeith_stuffbf8d08532bc19a14cfb54ae61099dccadefca446and backmerge from it), but if you would prefer to deal with that differently - please tell.
I'm not going to rebase it, and I can create a tag for that commit for you. Done, it's block-5.20-al. I did the former commit, or we can move the tag so it includes bf8d08532bc19a14cfb54ae61099dccadefca446? That'd be the whole series of that patchset, which is just that one extra patch.
Signature · 1 line
Jens Axboe
Re: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al ViroQuoted 10 lines · Al Viro
Quoted 2 lines · Al Viro
... and the first half of that thing conflicts with "block: relax direct io memory alignment" in -next...
BTW, looking at that commit - are you sure that
bio_put_pages()on failure exit will do the right thing? We have grabbed a bunch of page references; the amount ifDIV_ROUND_UP(offset + size, PAGE_SIZE). And that's before yoursize = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));
Thanks for the catch, it does look like a page reference could get leaked here.
Quoted 15 lines
in there. IMO the following would be more obviously correct:
size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; nr_pages = DIV_ROUND_UP(size + offset, PAGE_SIZE); size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev)); for (left = size, i = 0; left > 0; left -= len, i++) { ... if (ret) { while (i < nr_pages) put_page(pages[i++]); return ret; }
Quoted 3 lines
...
and get rid of
bio_put_pages()entirely. Objections?
I think that makes sense. I'll give your idea a test run tomorrow.
Re: [PATCH 01/44] 9p: handling Rerror without copy_from_iter_full()
In reply to Al Viro+Christian Schoenebeck in Ccs as that concerns qemu as well.
The patch I'm replying to is at
https://lkml.kernel.org/r/[email protected]
Al Viro wrote on Wed, Jun 22, 2022 at 05:15:09AM +0100:
Quoted 42 lines
p9_client_zc_rpc()/p9_check_zc_errors() are playing fast
and loose withcopy_from_iter_full().Reading from file is done by sending Tread request. Response
consists of fixed-sized header (including the amount of data actually read) followed by the data itself.For zero-copy case we arrange the things so that the first
11 bytes of reply go into the fixed-sized buffer, with the rest going straight into the pages we want to read into.What makes the things inconvenient is that sglist describing
what should go where has to be set *before* the reply arrives. As the result, if reply is an error, the things get interesting. On success we getsize[4] Rread tag[2] count[4] data[count] For error layout varies depending upon the protocol variant - in original 9P and 9P2000 it's size[4] Rerror tag[2] len[2] error[len] in 9P2000.U size[4] Rerror tag[2] len[2] error[len] errno[4] in 9P2000.L size[4] Rlerror tag[2] errno[4] The last case is nice and simple - we have an 11-byte responsethat fits into the fixed-sized buffer we hoped to get an Rread into. In other two, though, we get a variable-length string spill into the pages we'd prepared for the data to be read.
Had that been in fixed-sized buffer (which is actually 4K),
we would've dealt with that the same way we handle non-zerocopy case. However, for zerocopy it doesn't end up there, so we need to copy it from those pages.The trouble is, by the time we get around to that, the
references to pages in question are already dropped. As the result,p9_zc_check_errors()tries to get the data usingcopy_from_iter_full(). Unfortunately, theiov_iterit's trying to read from might *NOT* be capable of that. It is, after all, a data destination, not data source. In particular, if it's anITER_PIPEone,copy_from_iter_full()will simply fail.
Quoted 8 lines
In ->zc_request() itself we do have those pages and dealing withthe problem in there would be a simple matter of
memcpy_from_page()into the fixed-sized buffer. Moreover, it isn't hard to recognize the (rare) case when such copying is needed. That way we get rid ofp9_zc_check_errors()entirely -p9_check_errors()can be used instead both for zero-copy and non-zero-copy cases.Signed-off-by Al Viro <[email protected]>
I ran basic tests with this, should be ok given the code path is never used on normal (9p2000.L) workloads.
I also tried 9p2000.u for principle and ... I have no idea if this works but it didn't seem to blow up there at least.
The problem is that 9p2000.u just doesn't work well even without these patches, so I still stand by what I said about 9p2000.u and virtio (zc interface): we really can (and I think should) just say virtio doesn't support 9p2000.u.
(and could then further simplify this)
If you're curious, 9p2000.u hangs without your patch on at least two different code paths (trying to read a huge buffer aborts sending a reply because msize is too small instead of clamping it, that one has a qemu warning message; but there are others ops like copyrange that just fail silently and I didn't investigate)
I'd rather not fool someone into believing we support it when nobody has time to maintain it and it fails almost immediately when user requests some unusual IO patterns... And I definitely don't have time to even try fixing it.
I'll suggest the same thing to qemu lists if we go that way.
Anyway, for anything useful:
--
Dominique
Re: [PATCH 01/44] 9p: handling Rerror without copy_from_iter_full()
In reply to Al Viro(sigh, I'm tired -- said I'd add Christian in Ccs and promply forgot to do it. Sorry for double send to everyone else.)
+Christian Schoenebeck in Ccs as that concerns qemu as well.
The patch I'm replying to is at
https://lkml.kernel.org/r/[email protected]
Al Viro wrote on Wed, Jun 22, 2022 at 05:15:09AM +0100:
Quoted 42 lines
p9_client_zc_rpc()/p9_check_zc_errors() are playing fast
and loose withcopy_from_iter_full().Reading from file is done by sending Tread request. Response
consists of fixed-sized header (including the amount of data actually read) followed by the data itself.For zero-copy case we arrange the things so that the first
11 bytes of reply go into the fixed-sized buffer, with the rest going straight into the pages we want to read into.What makes the things inconvenient is that sglist describing
what should go where has to be set *before* the reply arrives. As the result, if reply is an error, the things get interesting. On success we getsize[4] Rread tag[2] count[4] data[count] For error layout varies depending upon the protocol variant - in original 9P and 9P2000 it's size[4] Rerror tag[2] len[2] error[len] in 9P2000.U size[4] Rerror tag[2] len[2] error[len] errno[4] in 9P2000.L size[4] Rlerror tag[2] errno[4] The last case is nice and simple - we have an 11-byte responsethat fits into the fixed-sized buffer we hoped to get an Rread into. In other two, though, we get a variable-length string spill into the pages we'd prepared for the data to be read.
Had that been in fixed-sized buffer (which is actually 4K),
we would've dealt with that the same way we handle non-zerocopy case. However, for zerocopy it doesn't end up there, so we need to copy it from those pages.The trouble is, by the time we get around to that, the
references to pages in question are already dropped. As the result,p9_zc_check_errors()tries to get the data usingcopy_from_iter_full(). Unfortunately, theiov_iterit's trying to read from might *NOT* be capable of that. It is, after all, a data destination, not data source. In particular, if it's anITER_PIPEone,copy_from_iter_full()will simply fail.
Quoted 8 lines
In ->zc_request() itself we do have those pages and dealing withthe problem in there would be a simple matter of
memcpy_from_page()into the fixed-sized buffer. Moreover, it isn't hard to recognize the (rare) case when such copying is needed. That way we get rid ofp9_zc_check_errors()entirely -p9_check_errors()can be used instead both for zero-copy and non-zero-copy cases.Signed-off-by Al Viro <[email protected]>
I ran basic tests with this, should be ok given the code path is never used on normal (9p2000.L) workloads.
I also tried 9p2000.u for principle and ... I have no idea if this works but it didn't seem to blow up there at least.
The problem is that 9p2000.u just doesn't work well even without these patches, so I still stand by what I said about 9p2000.u and virtio (zc interface): we really can (and I think should) just say virtio doesn't support 9p2000.u.
(and could then further simplify this)
If you're curious, 9p2000.u hangs without your patch on at least two different code paths (trying to read a huge buffer aborts sending a reply because msize is too small instead of clamping it, that one has a qemu warning message; but there are others ops like copyrange that just fail silently and I didn't investigate)
I'd rather not fool someone into believing we support it when nobody has time to maintain it and it fails almost immediately when user requests some unusual IO patterns... And I definitely don't have time to even try fixing it.
I'll suggest the same thing to qemu lists if we go that way.
Anyway, for anything useful:
--
Dominique
Re: [PATCH 40/44] 9p: convert to advancing variant of iov_iter_get_pages_alloc()
In reply to Al ViroAl Viro wrote on Wed, Jun 22, 2022 at 05:15:48AM +0100:
Quoted 1 line
that one is somewhat clumsier than usual and needs serious testing.
code inspection looks good to me, we revert everywhere I think we need to revert for read/write and readdir doesn't need any special treatment. I had a couple of nitpicks on debug messages, but that aside you can add my R-b:
Now for tests though I'm not quite sure what I'm supposed to test to stress the error cases, that'd actually let me detect a failure... Basic stuff seems to work but I don't think I ever got into an error path where that matters -- forcing short reads perhaps?
Quoted 39 lines
Signed-off-by Al Viro <[email protected]>--- net/9p/client.c | 39 +++++++++++++++++++++++---------------- net/9p/protocol.c | 3 +-- net/9p/trans_virtio.c | 3 ++- 3 files changed, 26 insertions(+), 19 deletions(-)net/9p/client.c +6
-2@@ -1491,7 +1491,7 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to,struct p9_client *clnt = fid->clnt;struct p9_req_t *req;int count =iov_iter_count (to); -int rsize, non_zc =0 ; +int rsize, received, non_zc =0 ;char *dataptr; *err =0 ;@@ -1520,36 +1520,40 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to, }if (IS_ERR (req)) { *err =PTR_ERR (req); +if (!non_zc) +iov_iter_revert (to, count -iov_iter_count (to));return 0 ; } *err =p9pdu_readf (&req->rc, clnt->proto_version, -"D" , &count, &dataptr); +"D" , &received, &dataptr);if (*err) { +if (!non_zc) +iov_iter_revert (to, count -iov_iter_count (to));trace_9p_protocol_dump (clnt, &req->rc);p9_tag_remove (clnt, req);return 0 ;
net/9p/client.c p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to, · line 1542
Quoted 10 lines
net/9p/client.c +3
-3} -if (rsize < count) { -pr_err ("bogus RREAD count (%d > %d)\n" , count, rsize); - count = rsize; +if (rsize < received) { +pr_err ("bogus RREAD count (%d > %d)\n" , received, rsize); + received = rsize; }p9_debug (P9_DEBUG_9P ,"<<< RREAD count %d\n" , count);
This probably should be updated to received; we know how much we asked to read already what we want to see here is what the server replied.
Quoted 45 lines
net/9p/client.c +8
-5if (non_zc) { -int n =copy_to_iter (dataptr, count, to); +int n =copy_to_iter (dataptr, received, to); -if (n != count) { +if (n != received) { *err = -EFAULT ;p9_tag_remove (clnt, req);return n; } }else { -iov_iter_advance (to, count); +iov_iter_revert (to, count - received -iov_iter_count (to)); }p9_tag_remove (clnt, req); -return count; +return received; }EXPORT_SYMBOL (p9_client_read_once);@@ -1567,6 +1571,7 @@ p9_client_write(struct p9_fid *fid, u64 offset, struct iov_iter *from, int *err)while (iov_iter_count (from)) {int count =iov_iter_count (from);int rsize = fid->iounit; +int written;if (!rsize || rsize > clnt->msize -P9_IOHDRSZ ) rsize = clnt->msize -P9_IOHDRSZ ;@@ -1584,27 +1589,29 @@ p9_client_write(struct p9_fid *fid, u64 offset, struct iov_iter *from, int *err) offset, rsize, from); }if (IS_ERR (req)) { +iov_iter_revert (from, count -iov_iter_count (from)); *err =PTR_ERR (req);break ; } - *err =p9pdu_readf (&req->rc, clnt->proto_version,"d" , &count); + *err =p9pdu_readf (&req->rc, clnt->proto_version,"d" , &written);if (*err) { +iov_iter_revert (from, count -iov_iter_count (from));trace_9p_protocol_dump (clnt, &req->rc);p9_tag_remove (clnt, req);break ;
net/9p/client.c p9_client_write(struct p9_fid *fid, u64 offset, struct iov_iter *from, int *err) · line 1609
Quoted 10 lines
net/9p/client.c +3
-3} -if (rsize < count) { -pr_err ("bogus RWRITE count (%d > %d)\n" , count, rsize); - count = rsize; +if (rsize < written) { +pr_err ("bogus RWRITE count (%d > %d)\n" , written, rsize); + written = rsize; }p9_debug (P9_DEBUG_9P ,"<<< RWRITE count %d\n" , count);
likewise, please make it dump written.
--
Dominique
Quoted 47 lines
net/9p/client.c +3
-3p9_tag_remove (clnt, req); -iov_iter_advance (from, count); - total += count; - offset += count; +iov_iter_revert (from, count - written -iov_iter_count (from)); + total += written; + offset += written; }return total; }net/9p/protocol.c +1
-2@@ -63,9 +63,8 @@ static size_t pdu_write_u(struct p9_fcall *pdu, struct iov_iter *from, size_t size) { size_t len = min(pdu->capacity - pdu->size, size); -struct iov_iter i = *from; -if (!copy_from_iter_full (&pdu->sdata[pdu->size], len, &i)) +if (!copy_from_iter_full (&pdu->sdata[pdu->size], len, from)) len =0 ; pdu->size += len;net/9p/trans_virtio.c +2
-1@@ -331,7 +331,7 @@ static int p9_get_mapped_pages(struct virtio_chan *chan,if (err == -ERESTARTSYS )return err; } - n =iov_iter_get_pages_alloc (data, pages, count, offs); + n =iov_iter_get_pages_alloc2 (data, pages, count, offs);if (n <0 )return n; *need_drop =1 ;@@ -373,6 +373,7 @@ static int p9_get_mapped_pages(struct virtio_chan *chan, (*pages)[index] =kmap_to_page (p); p +=PAGE_SIZE ; } +iov_iter_advance (data, len);return len; } }
Re: [PATCH 40/44] 9p: convert to advancing variant of iov_iter_get_pages_alloc()
In reply to Al ViroQuoted 3 lines · On Mittwoch, 22. Juni 2022 06:15:48 CEST Al Viro wrote:
that one is somewhat clumsier than usual and needs serious testing.
Signed-off-by Al Viro <[email protected]>
Hi Al,
it took me a bit to find the patch that introduces iov_iter_get_pages_alloc2(), but this patch itself looks fine:
Please give me some days for thorough testing. We recently had 9p broken (with cache=loose) for half a year, so I would like to avoid repetition.
Best regards,
Christian Schoenebeck
Quoted 149 lines
--- net/9p/client.c | 39 +++++++++++++++++++++++---------------- net/9p/protocol.c | 3 +-- net/9p/trans_virtio.c | 3 ++- 3 files changed, 26 insertions(+), 19 deletions(-)net/9p/client.c +23
-16@@ -1491,7 +1491,7 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to, struct p9_client *clnt = fid->clnt;struct p9_req_t *req;int count =iov_iter_count (to); -int rsize, non_zc =0 ; +int rsize, received, non_zc =0 ;char *dataptr; *err =0 ;@@ -1520,36 +1520,40 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to, }if (IS_ERR (req)) { *err =PTR_ERR (req); +if (!non_zc) +iov_iter_revert (to, count -iov_iter_count (to));return 0 ; } *err =p9pdu_readf (&req->rc, clnt->proto_version, -"D" , &count, &dataptr); +"D" , &received, &dataptr);if (*err) { +if (!non_zc) +iov_iter_revert (to, count -iov_iter_count (to));trace_9p_protocol_dump (clnt, &req->rc);p9_tag_remove (clnt, req);return 0 ; } -if (rsize < count) { -pr_err ("bogus RREAD count (%d > %d)\n" , count, rsize); - count = rsize; +if (rsize < received) { +pr_err ("bogus RREAD count (%d > %d)\n" , received, rsize); + received = rsize; }p9_debug (P9_DEBUG_9P ,"<<< RREAD count %d\n" , count);if (non_zc) { -int n =copy_to_iter (dataptr, count, to); +int n =copy_to_iter (dataptr, received, to); -if (n != count) { +if (n != received) { *err = -EFAULT ;p9_tag_remove (clnt, req);return n; } }else { -iov_iter_advance (to, count); +iov_iter_revert (to, count - received -iov_iter_count (to)); }p9_tag_remove (clnt, req); -return count; +return received; }EXPORT_SYMBOL (p9_client_read_once);@@ -1567,6 +1571,7 @@ p9_client_write(struct p9_fid *fid, u64 offset, struct iov_iter *from, int *err) while (iov_iter_count(from)) {int count =iov_iter_count (from);int rsize = fid->iounit; +int written;if (!rsize || rsize > clnt->msize -P9_IOHDRSZ ) rsize = clnt->msize -P9_IOHDRSZ ;@@ -1584,27 +1589,29 @@ p9_client_write(struct p9_fid *fid, u64 offset, struct iov_iter *from, int *err) offset, rsize, from); }if (IS_ERR (req)) { +iov_iter_revert (from, count -iov_iter_count (from)); *err =PTR_ERR (req);break ; } - *err =p9pdu_readf (&req->rc, clnt->proto_version,"d" , &count); + *err =p9pdu_readf (&req->rc, clnt->proto_version,"d" , &written);if (*err) { +iov_iter_revert (from, count -iov_iter_count (from));trace_9p_protocol_dump (clnt, &req->rc);p9_tag_remove (clnt, req);break ; } -if (rsize < count) { -pr_err ("bogus RWRITE count (%d > %d)\n" , count, rsize); - count = rsize; +if (rsize < written) { +pr_err ("bogus RWRITE count (%d > %d)\n" , written, rsize); + written = rsize; }p9_debug (P9_DEBUG_9P ,"<<< RWRITE count %d\n" , count);p9_tag_remove (clnt, req); -iov_iter_advance (from, count); - total += count; - offset += count; +iov_iter_revert (from, count - written -iov_iter_count (from)); + total += written; + offset += written; }return total; }net/9p/protocol.c +1
-2@@ -63,9 +63,8 @@ static size_t pdu_write_u(struct p9_fcall *pdu, struct iov_iter *from, size_t size) { size_t len = min(pdu->capacity - pdu->size, size); -struct iov_iter i = *from; -if (!copy_from_iter_full (&pdu->sdata[pdu->size], len, &i)) +if (!copy_from_iter_full (&pdu->sdata[pdu->size], len, from)) len =0 ; pdu->size += len;net/9p/trans_virtio.c +2
-1@@ -331,7 +331,7 @@ static int p9_get_mapped_pages(struct virtio_chan *chan,if (err == -ERESTARTSYS )return err; } - n =iov_iter_get_pages_alloc (data, pages, count, offs); + n =iov_iter_get_pages_alloc2 (data, pages, count, offs);if (n <0 )return n; *need_drop =1 ;@@ -373,6 +373,7 @@ static int p9_get_mapped_pages(struct virtio_chan *chan, (*pages)[index] =kmap_to_page (p); p +=PAGE_SIZE ; } +iov_iter_advance (data, len);return len; } }
Christian Schoenebeck
reviewerRe: [PATCH 01/44] 9p: handling Rerror without copy_from_iter_full()
In reply to Dominique MartinetQuoted 73 lines · On Freitag, 1. Juli 2022 08:25:49 CEST Dominique Martinet wrote:
(sigh, I'm tired -- said I'd add Christian in Ccs and promply forgot to do it. Sorry for double send to everyone else.)
+Christian Schoenebeck in Ccs as that concerns qemu as well.
The patch I'm replying to is at
https://lkml.kernel.org/r/[email protected]Al Viro wrote on Wed, Jun 22, 2022 at 05:15:09AM +0100:
Quoted 64 lines
p9_client_zc_rpc()/p9_check_zc_errors() are playing fast
and loose with
copy_from_iter_full().Reading from file is done by sending Tread request. Response
consists of fixed-sized header (including the amount of data actually read) followed by the data itself.
For zero-copy case we arrange the things so that the first
11 bytes of reply go into the fixed-sized buffer, with the rest going straight into the pages we want to read into.
What makes the things inconvenient is that sglist describing
what should go where has to be set *before* the reply arrives. As the result, if reply is an error, the things get interesting. On success we get
size[4] Rread tag[2] count[4] data[count]For error layout varies depending upon the protocol variant - in original 9P and 9P2000 it's
size[4] Rerror tag[2] len[2] error[len]in 9P2000.U
size[4] Rerror tag[2] len[2] error[len] errno[4]in 9P2000.L
size[4] Rlerror tag[2] errno[4] The last case is nice and simple - we have an 11-byte responsethat fits into the fixed-sized buffer we hoped to get an Rread into. In other two, though, we get a variable-length string spill into the pages we'd prepared for the data to be read.
Had that been in fixed-sized buffer (which is actually 4K),
we would've dealt with that the same way we handle non-zerocopy case. However, for zerocopy it doesn't end up there, so we need to copy it from those pages.
The trouble is, by the time we get around to that, the
references to pages in question are already dropped. As the result,
p9_zc_check_errors()tries to get the data usingcopy_from_iter_full(). Unfortunately, theiov_iterit's trying to read from might *NOT* be capable of that. It is, after all, a data destination, not data source. In particular, if it's anITER_PIPEone,copy_from_iter_full()will simply fail.In ->zc_request() itself we do have those pages and dealing withthe problem in there would be a simple matter of
memcpy_from_page()into the fixed-sized buffer. Moreover, it isn't hard to recognize the (rare) case when such copying is needed. That way we get rid ofp9_zc_check_errors()entirely -p9_check_errors()can be used instead both for zero-copy and non-zero-copy cases.
Quoted 4 lines
Quoted 1 line
Signed-off-by Al Viro <[email protected]>I ran basic tests with this, should be ok given the code path is never used on normal (9p2000.L) workloads.
I haven't read this patch in detail yet, but upfront: POSIX error strings are like what, max. 128 bytes, no? So my expectation therefore would be that this patch could be further simplified.
Apart from that, I would rename handle_rerror() to something that reflects better what it actually does, e.g. unsparse_error() or cp_rerror_to_sdata().
Quoted 8 lines
I also tried 9p2000.u for principle and ... I have no idea if this works but it didn't seem to blow up there at least.
The problem is that 9p2000.u just doesn't work well even without these patches, so I still stand by what I said about 9p2000.u and virtio (zc interface): we really can (and I think should) just say virtio doesn't support 9p2000.u.
(and could then further simplify this)
Quoted 5 lines
If you're curious, 9p2000.u hangs without your patch on at least two different code paths (trying to read a huge buffer aborts sending a reply because msize is too small instead of clamping it, that one has a qemu warning message; but there are others ops like copyrange that just fail silently and I didn't investigate)
Last time I tested 9p2000.u was with the "remove msize limit" (WIP) patches:
https://lore.kernel.org/all/[email protected]/
Where I did not observe any issue with 9p2000.u.
What msize are we talking about, or can you tell a way to reproduce?
Quoted 5 lines
I'd rather not fool someone into believing we support it when nobody has time to maintain it and it fails almost immediately when user requests some unusual IO patterns... And I definitely don't have time to even try fixing it.
I'll suggest the same thing to qemu lists if we go that way.
Yeah, the situation with 9p2000.u in QEMU is similar in the sense that 9p2000.u is barely used, little contributions, code not in good shape (e.g slower in many aspects in comparison to 9p2000.L), and for that reason I discussed with Greg to deprecate 9p2000.u in QEMU (not done yet). We are not aware about any serious issue with 9p2000.u though.
Best regards,
Christian Schoenebeck
Al Viro
maintainerRe: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Keith BuschQuoted 31 lines · Keith Busch
Quoted 10 lines · Al Viro
Quoted 2 lines · Al Viro
... and the first half of that thing conflicts with "block: relax direct io memory alignment" in -next...
BTW, looking at that commit - are you sure that
bio_put_pages()on failure exit will do the right thing? We have grabbed a bunch of page references; the amount ifDIV_ROUND_UP(offset + size, PAGE_SIZE). And that's before yoursize = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));Thanks for the catch, it does look like a page reference could get leaked here.
Quoted 17 lines
in there. IMO the following would be more obviously correct:
size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; nr_pages = DIV_ROUND_UP(size + offset, PAGE_SIZE); size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev)); for (left = size, i = 0; left > 0; left -= len, i++) { ... if (ret) { while (i < nr_pages) put_page(pages[i++]); return ret; }...
Quoted 4 lines
Quoted 1 line
and get rid of
bio_put_pages()entirely. Objections?I think that makes sense. I'll give your idea a test run tomorrow.
See vfs.git#block-fixes, along with #work.iov_iter_get_pages-3 in there. Seems to work here...
If you are OK with #block-fixes (it's one commit on top of bf8d08532bc1 "iomap: add support for dma aligned direct-io" in block.git), the easiest way to deal with the conflicts would be to have that branch pulled into block.git. Jens, would you be OK with that in terms of tree topology? Provided that patch itself looks sane to you, of course...
FWOW, the patch in question is
commit 863965bb7e52997851af3a107ec3e4d8c7050cbd
Author: Al Viro <[email protected]> Date: Fri Jul 1 13:15:36 2022 -0400
__bio_iov_iter_get_pages(): make sure we don't leak page refs on failure
Calculate the number of pages we'd grabbed before trimming size down. And don't bother with bio_put_pages() - an explicit cleanup loop is easier to follow...
block/bio.c +4 -11
@@ -1151,14 +1151,6 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter)bio_set_flag (bio,BIO_CLONED ); } -static void bio_put_pages (struct page **pages,size_t size,size_t off) -{ -size_t i, nr =DIV_ROUND_UP (size + (off & ~PAGE_MASK ),PAGE_SIZE ); - -for (i =0 ; i < nr; i++) -put_page (pages[i]); -} -static int bio_iov_add_page (struct bio *bio,struct page *page,unsigned int len,unsigned int offset) {
@@ -1228,11 +1220,11 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* the iov data will be picked up in the next bio iteration. */ size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));if (unlikely (size <=0 ))return size ? size : -EFAULT ; + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE ); + size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));for (left = size, i =0 ; left >0 ; left -= len, i++) {struct page *page = pages[i];int ret;
@@ -1245,7 +1237,8 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) ret =bio_iov_add_page (bio, page, len, offset);if (ret) { -bio_put_pages (pages + i, left, offset); +while (i < nr_pages) +put_page (pages[i++]);return ret; } offset =0 ;
Re: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
Quoted 12 lines · Al Viro
(quoted diff) +0
-8-static void bio_put_pages(struct page **pages, size_t size, size_t off) -{ - size_t i, nr = DIV_ROUND_UP(size + (off & ~PAGE_MASK), PAGE_SIZE); - - for (i = 0; i < nr; i++) - put_page(pages[i]); -} - static int bio_iov_add_page(struct bio *bio, struct page *page, unsigned int len, unsigned int offset) {
Review static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1227
Quoted 10 lines
(quoted diff) +2
-2@@ -1228,11 +1220,11 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * the iov data will be picked up in the next bio iteration. */ size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); - if (size > 0) - size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev)); if (unlikely(size <= 0)) return size ? size : -EFAULT; + nr_pages = DIV_ROUND_UP(offset + size, PAGE_SIZE); + size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));
This isn't quite right. The result of the ALIGN_DOWN could be 0, so whatever page we got before would be leaked since unused pages are only released on an add_page error. I was about to reply with a patch that fixes this, but here's the one that I'm currently testing:
---
block/bio.c +21 -19
@@ -1151,14 +1151,6 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter)bio_set_flag (bio,BIO_CLONED ); } -static void bio_put_pages (struct page **pages,size_t size,size_t off) -{ -size_t i, nr =DIV_ROUND_UP (size + (off & ~PAGE_MASK ),PAGE_SIZE ); - -for (i =0 ; i < nr; i++) -put_page (pages[i]); -} -static int bio_iov_add_page (struct bio *bio,struct page *page,unsigned int len,unsigned int offset) {
@@ -1208,9 +1200,10 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt;struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt;struct page **pages = (struct page **)bv; +unsigned len, i =0 ;ssize_t size, left; -unsigned len, i;size_t offset; +int ret;/* * Move page array up in the allocated memory for the bio vecs as far as
@@ -1228,14 +1221,19 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* the iov data will be picked up in the next bio iteration. */ size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) +if (size >0 ) { + nr_pages =DIV_ROUND_UP (size + offset,PAGE_SIZE ); size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev)); -if (unlikely (size <=0 )) -return size ? size : -EFAULT ; + }else + nr_pages =0 ; + +if (unlikely (size <=0 )) { + ret = size ? size : -EFAULT ; + goto out; + } -for (left = size, i =0 ; left >0 ; left -= len, i++) { +for (left = size; left >0 ; left -= len, i++) {struct page *page = pages[i]; -int ret; len =min_t (size_t,PAGE_SIZE - offset, left);if (bio_op (bio) ==REQ_OP_ZONE_APPEND )
@@ -1244,15 +1242,19 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)else ret =bio_iov_add_page (bio, page, len, offset); -if (ret) { -bio_put_pages (pages + i, left, offset); -return ret; - } +if (ret) + goto out; offset =0 ; }iov_iter_advance (iter, size); -return 0 ; +out: +while (i < nr_pages) { +put_page (pages[i]); + i++; + } + +return ret; }/**
--
Al Viro
maintainerRe: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
Quoted 22 lines · Keith Busch
Quoted 21 lines · Al Viro
block/bio.c +1
-10-static void bio_put_pages (struct page **pages,size_t size,size_t off) -{ -size_t i, nr =DIV_ROUND_UP (size + (off & ~PAGE_MASK ),PAGE_SIZE ); - -for (i =0 ; i < nr; i++) -put_page (pages[i]); -} -static int bio_iov_add_page (struct bio *bio,struct page *page,unsigned int len,unsigned int offset) {@@ -1228,11 +1220,11 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* the iov data will be picked up in the next bio iteration. */ size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));if (unlikely (size <=0 ))return size ? size : -EFAULT ; + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE );
Quoted 6 lines
Review static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1227
Quoted 1 line
(quoted diff) +1
-0+ size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));This isn't quite right. The result of the
ALIGN_DOWNcould be 0, so whatever page we got before would be leaked since unused pages are only released on anadd_pageerror. I was about to reply with a patch that fixes this, but here's the one that I'm currently testing:
AFAICS, result is broken; you might end up consuming some data and leaving iterator not advanced at all. With no way for the caller to tell which way it went.
Al Viro
maintainerRe: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
Quoted 30 lines · Al Viro
Quoted 22 lines · Keith Busch
Quoted 21 lines · Al Viro
(quoted diff) +1
-10-static void bio_put_pages(struct page **pages, size_t size, size_t off) -{ - size_t i, nr = DIV_ROUND_UP(size + (off & ~PAGE_MASK), PAGE_SIZE); - - for (i = 0; i < nr; i++) - put_page(pages[i]); -} - static int bio_iov_add_page(struct bio *bio, struct page *page, unsigned int len, unsigned int offset) {@@ -1228,11 +1220,11 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * the iov data will be picked up in the next bio iteration. */ size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); - if (size > 0) - size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev)); if (unlikely(size <= 0)) return size ? size : -EFAULT; + nr_pages = DIV_ROUND_UP(offset + size, PAGE_SIZE);Quoted 6 lines
Review static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1227
Quoted 1 line
(quoted diff) +1
-0+ size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));This isn't quite right. The result of the
ALIGN_DOWNcould be 0, so whatever page we got before would be leaked since unused pages are only released on anadd_pageerror. I was about to reply with a patch that fixes this, but here's the one that I'm currently testing:
Quoted 3 lines
AFAICS, result is broken; you might end up consuming some data and leaving iterator not advanced at all. With no way for the caller to tell which way it went.
How about the following?
commit 5e3e9769404de54734c110b2040bdb93593e0f1b
Author: Al Viro <[email protected]> Date: Fri Jul 1 13:15:36 2022 -0400
__bio_iov_iter_get_pages(): make sure we don't leak page refs on failure
Calculate the number of pages we'd grabbed before trimming size down. And don't bother with bio_put_pages() - an explicit cleanup loop is easier to follow...
block/bio.c +9 -17
@@ -1151,14 +1151,6 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter)bio_set_flag (bio,BIO_CLONED ); } -static void bio_put_pages (struct page **pages,size_t size,size_t off) -{ -size_t i, nr =DIV_ROUND_UP (size + (off & ~PAGE_MASK ),PAGE_SIZE ); - -for (i =0 ; i < nr; i++) -put_page (pages[i]); -} -static int bio_iov_add_page (struct bio *bio,struct page *page,unsigned int len,unsigned int offset) {
@@ -1211,6 +1203,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)ssize_t size, left;unsigned len, i;size_t offset; +int ret;/* * Move page array up in the allocated memory for the bio vecs as far as
@@ -1228,14 +1221,13 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* the iov data will be picked up in the next bio iteration. */ size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));if (unlikely (size <=0 ))return size ? size : -EFAULT ; + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE ); + size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));for (left = size, i =0 ; left >0 ; left -= len, i++) {struct page *page = pages[i]; -int ret; len =min_t (size_t,PAGE_SIZE - offset, left);if (bio_op (bio) ==REQ_OP_ZONE_APPEND )
@@ -1244,15 +1236,15 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)else ret =bio_iov_add_page (bio, page, len, offset); -if (ret) { -bio_put_pages (pages + i, left, offset); -return ret; - } +if (ret) +break ; offset =0 ; } +while (i < nr_pages) +put_page (pages[i++]); -iov_iter_advance (iter, size); -return 0 ; +iov_iter_advance (iter, size - left); +return ret; }/**
Re: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
Quoted 8 lines · Al Viro
Quoted 7 lines · Al Viro
Quoted 3 lines
page we got before would be leaked since unused pages are only released on an
add_pageerror. I was about to reply with a patch that fixes this, but here's the one that I'm currently testing:AFAICS, result is broken; you might end up consuming some data and leaving iterator not advanced at all. With no way for the caller to tell which way it went.
I think I see what you mean, though the issue with a non-advancing iterator on a partially filled bio was happening prior to this patch.
Quoted 1 line
How about the following?
This looks close to what I was about to respond with. Just a couple issues below:
Quoted 12 lines
block/bio.c +0
-8-static void bio_put_pages (struct page **pages,size_t size,size_t off) -{ -size_t i, nr =DIV_ROUND_UP (size + (off & ~PAGE_MASK ),PAGE_SIZE ); - -for (i =0 ; i < nr; i++) -put_page (pages[i]); -} -static int bio_iov_add_page (struct bio *bio,struct page *page,unsigned int len,unsigned int offset) {
block/bio.c static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1206
Quoted 4 lines
block/bio.c +1
-0@@ -1211,6 +1203,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)ssize_t size, left;unsigned len, i;size_t offset; +int ret;
'ret' might never be initialized if 'size' aligns down to 0.
block/bio.c static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1228
Quoted 13 lines
block/bio.c +2
-2/* * Move page array up in the allocated memory for the bio vecs as far as @@ -1228,14 +1221,13 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* the iov data will be picked up in the next bio iteration. */ size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));if (unlikely (size <=0 ))return size ? size : -EFAULT ; + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE ); + size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));
We still need to return EFAULT if size becomes 0 because that's the only way bio_iov_iter_get_pages()'s loop will break out in this condition.
Re: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al ViroQuoted 30 lines · Al Viro
Quoted 22 lines · Keith Busch
Quoted 21 lines · Al Viro
block/bio.c +1
-10-static void bio_put_pages (struct page **pages,size_t size,size_t off) -{ -size_t i, nr =DIV_ROUND_UP (size + (off & ~PAGE_MASK ),PAGE_SIZE ); - -for (i =0 ; i < nr; i++) -put_page (pages[i]); -} -static int bio_iov_add_page (struct bio *bio,struct page *page,unsigned int len,unsigned int offset) {@@ -1228,11 +1220,11 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* the iov data will be picked up in the next bio iteration. */ size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));if (unlikely (size <=0 ))return size ? size : -EFAULT ; + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE );Quoted 6 lines
Review static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1227
Quoted 1 line
(quoted diff) +1
-0+ size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));This isn't quite right. The result of the
ALIGN_DOWNcould be 0, so whatever page we got before would be leaked since unused pages are only released on anadd_pageerror. I was about to reply with a patch that fixes this, but here's the one that I'm currently testing:
Quoted 3 lines
AFAICS, result is broken; you might end up consuming some data and leaving iterator not advanced at all. With no way for the caller to tell which way it went.
Looks like the possibility of a non-advancing iterator goes all the way back to the below commit.
commit 576ed9135489c723fb39b97c4e2c73428d06dd78
Author: Christoph Hellwig <[email protected]>
Date: Thu Sep 20 08:28:21 2018 +0200
block: use bio_add_page in bio_iov_iter_get_pages
I guess the error condition never occured, and nor should it if the bio's available vectors is accounted for correctly.
Al Viro
maintainerRe: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Keith BuschQuoted 28 lines · Keith Busch
Quoted 8 lines · Al Viro
Quoted 7 lines · Al Viro
Quoted 3 lines
page we got before would be leaked since unused pages are only released on an
add_pageerror. I was about to reply with a patch that fixes this, but here's the one that I'm currently testing:AFAICS, result is broken; you might end up consuming some data and leaving iterator not advanced at all. With no way for the caller to tell which way it went.
I think I see what you mean, though the issue with a non-advancing iterator on a partially filled bio was happening prior to this patch.
Quoted 1 line
How about the following?
This looks close to what I was about to respond with. Just a couple issues below:
Quoted 10 lines
block/bio.c +0
-8-static void bio_put_pages (struct page **pages,size_t size,size_t off) -{ -size_t i, nr =DIV_ROUND_UP (size + (off & ~PAGE_MASK ),PAGE_SIZE ); - -for (i =0 ; i < nr; i++) -put_page (pages[i]); -} -static int bio_iov_add_page (struct bio *bio,struct page *page,unsigned int len,unsigned int offset)
Quoted 8 lines
block/bio.c static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1206
Quoted 6 lines
{
block/bio.c +1
-0@@ -1211,6 +1203,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)ssize_t size, left;unsigned len, i;size_t offset; +int ret;'ret' might never be initialized if 'size' aligns down to 0.
Point.
Quoted 12 lines
Quoted 12 lines
block/bio.c +1
-2/* * Move page array up in the allocated memory for the bio vecs as far as @@ -1228,14 +1221,13 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* the iov data will be picked up in the next bio iteration. */ size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));if (unlikely (size <=0 ))return size ? size : -EFAULT ; + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE );
Quoted 4 lines
block/bio.c static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1228
Quoted 1 line
block/bio.c +1
-0+ size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));We still need to return EFAULT if size becomes 0 because that's the only way bio_iov_iter_get_pages()'s loop will break out in this condition.
I really don't like these calling conventions ;-/
What do you want to happen if we have that align-down to reduce size? IOW, what should be the state after that loop in the caller?
Re: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
Quoted 8 lines · Al Viro
Quoted 7 lines · Keith Busch
Quoted 7 lines
block/bio.c +1
-2size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));if (unlikely (size <=0 ))return size ? size : -EFAULT ; + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE );
Quoted 6 lines
Quoted 4 lines
block/bio.c static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1228
Quoted 1 line
block/bio.c +1
-0+ size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));We still need to return EFAULT if size becomes 0 because that's the only way bio_iov_iter_get_pages()'s loop will break out in this condition.
I really don't like these calling conventions ;-/
No argument here; I'm just working in the space as I found it. :)
Quoted 2 lines
What do you want to happen if we have that align-down to reduce size? IOW, what should be the state after that loop in the caller?
We fill up the bio to bi_max_vecs. If there are more pages than vectors, then the bio is submitted as-is with the partially drained iov_iter. The remainder of the iov is left for a subsequent bio to deal with.
The ALIGN_DOWN is required because I've replaced the artificial kernel aligment with the underlying hardware's alignment. The hardware's alignment is usually smaller than a block size. If the last bvec has a non-block aligned offset, then it has to be truncated down, which could result in a 0 size when bi_vcnt is already non-zero. If that happens, we just submit the bio as-is, and allocate a new one to deal with the rest of the iov.
Al Viro
maintainerRe: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
Quoted 25 lines · Keith Busch
Quoted 8 lines · Al Viro
Quoted 7 lines · Keith Busch
Quoted 7 lines
block/bio.c +1
-2size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));if (unlikely (size <=0 ))return size ? size : -EFAULT ; + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE );Quoted 6 lines
Quoted 4 lines
block/bio.c static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1228
Quoted 1 line
block/bio.c +1
-0+ size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));We still need to return EFAULT if size becomes 0 because that's the only way bio_iov_iter_get_pages()'s loop will break out in this condition.
I really don't like these calling conventions ;-/
No argument here; I'm just working in the space as I found it. :)
Quoted 2 lines
What do you want to happen if we have that align-down to reduce size? IOW, what should be the state after that loop in the caller?
We fill up the bio to
bi_max_vecs. If there are more pages than vectors, then the bio is submitted as-is with the partially drainediov_iter. The remainder of the iov is left for a subsequent bio to deal with.
Quoted 6 lines
The
ALIGN_DOWNis required because I've replaced the artificial kernel aligment with the underlying hardware's alignment. The hardware's alignment is usually smaller than a block size. If the last bvec has a non-block aligned offset, then it has to be truncated down, which could result in a 0 size whenbi_vcntis already non-zero. If that happens, we just submit the bio as-is, and allocate a new one to deal with the rest of the iov.
Wait a sec. Looks like you are dealing with the round-down in the wrong place - it applies to the *total* you've packed into the bio, no matter how it is distributed over the segments you've gathered it from. Looks like it would be more natural to handle it after the loop in the caller, would it not?
I.e.
while bio is not full
grab pages
if got nothing
break
pack pages into bio
if can't add a page (bio_add_hw_page() failure)
drop the ones not shoved there break
see how much had we packed into the sucker
if not a multiple of logical block size
trim the bio, dropping what needs to be dropped iov_iter_revert()
if nothing's packed
return -EINVAL if it was a failed bio_add_hw_page() return -EFAULT otherwise
return 0
Re: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
Quoted 47 lines · Al Viro
Quoted 25 lines · Keith Busch
Quoted 8 lines · Al Viro
Quoted 7 lines · Keith Busch
Quoted 7 lines
block/bio.c +1
-2size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); -if (size >0 ) - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));if (unlikely (size <=0 ))return size ? size : -EFAULT ; + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE );Quoted 6 lines
Quoted 4 lines
block/bio.c static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) · line 1228
Quoted 1 line
block/bio.c +1
-0+ size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev));We still need to return EFAULT if size becomes 0 because that's the only way bio_iov_iter_get_pages()'s loop will break out in this condition.
I really don't like these calling conventions ;-/
No argument here; I'm just working in the space as I found it. :)
Quoted 2 lines
What do you want to happen if we have that align-down to reduce size? IOW, what should be the state after that loop in the caller?
We fill up the bio to
bi_max_vecs. If there are more pages than vectors, then the bio is submitted as-is with the partially drainediov_iter. The remainder of the iov is left for a subsequent bio to deal with.Quoted 6 lines
The
ALIGN_DOWNis required because I've replaced the artificial kernel aligment with the underlying hardware's alignment. The hardware's alignment is usually smaller than a block size. If the last bvec has a non-block aligned offset, then it has to be truncated down, which could result in a 0 size whenbi_vcntis already non-zero. If that happens, we just submit the bio as-is, and allocate a new one to deal with the rest of the iov.Wait a sec. Looks like you are dealing with the round-down in the wrong place - it applies to the *total* you've packed into the bio, no matter how it is distributed over the segments you've gathered it from. Looks like it would be more natural to handle it after the loop in the caller, would it not?
I.e.
while bio is not full
grab pages
if got nothing
break
pack pages into bio
if can't add a page (bio_add_hw_page()failure)
drop the ones not shoved there break
Quoted 8 lines
see how much had we packed into the sucker
if not a multiple of logical block size
trim the bio, dropping what needs to be droppediov_iter_revert()
if nothing's packed
return-EINVALif it was a failedbio_add_hw_page()return-EFAULTotherwise
return 0
Validating user requests gets really messy if we allow arbitrary segment lengths. This particular patch just enables arbitrary address alignment, but segment size is still required to be a block size. You found the commit that enforces that earlier, "iov: introduce iov_iter_aligned", two commits prior.
The rest of the logic simplifies when we are guaranteed segment size is a block size mulitple: truncating a segment at a block boundary ensures both sides are block size aligned, and we don't even need to consult lower level queue limits, like segment count or segment length. The bio is valid after this step, and can be split into valid bios later if needed.
Dominique Martinet
maintainerRe: [PATCH 01/44] 9p: handling Rerror without copy_from_iter_full()
In reply to Christian SchoenebeckChristian Schoenebeck wrote on Fri, Jul 01, 2022 at 06:02:31PM +0200:
Quoted 14 lines
Quoted 8 lines
I also tried 9p2000.u for principle and ... I have no idea if this works but it didn't seem to blow up there at least.
The problem is that 9p2000.u just doesn't work well even without these patches, so I still stand by what I said about 9p2000.u and virtio (zc interface): we really can (and I think should) just say virtio doesn't support 9p2000.u.
(and could then further simplify this)Quoted 5 lines
If you're curious, 9p2000.u hangs without your patch on at least two different code paths (trying to read a huge buffer aborts sending a reply because msize is too small instead of clamping it, that one has a qemu warning message; but there are others ops like copyrange that just fail silently and I didn't investigate)
Quoted 5 lines
Last time I tested 9p2000.u was with the "remove msize limit" (WIP) patches:
https://lore.kernel.org/all/[email protected]/
Where I did not observe any issue with 9p2000.u.What msize are we talking about, or can you tell a way to reproduce?
I just ran fsstress on amount -t 9p -o cache=none,trans=virtio,version=9p2000.u mount on v5.19-rc2:
fsstress -d /mnt/fstress -n 1000 -v
If that doesn't reproduce for you (and you care) I can turn some more logs on, but from the look of it it could very well be msize related, I just didn't check as I don't expect any real user
--
Dominique
Jens Axboe
maintainerRe: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al ViroQuoted 40 lines · Al Viro
Quoted 31 lines · Keith Busch
Quoted 10 lines · Al Viro
Quoted 2 lines · Al Viro
... and the first half of that thing conflicts with "block: relax direct io memory alignment" in -next...
BTW, looking at that commit - are you sure that
bio_put_pages()on failure exit will do the right thing? We have grabbed a bunch of page references; the amount ifDIV_ROUND_UP(offset + size, PAGE_SIZE). And that's before yoursize = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));Thanks for the catch, it does look like a page reference could get leaked here.
Quoted 17 lines
in there. IMO the following would be more obviously correct:
size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; nr_pages = DIV_ROUND_UP(size + offset, PAGE_SIZE); size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev)); for (left = size, i = 0; left > 0; left -= len, i++) { ... if (ret) { while (i < nr_pages) put_page(pages[i++]); return ret; }...
Quoted 4 lines
Quoted 1 line
and get rid of
bio_put_pages()entirely. Objections?I think that makes sense. I'll give your idea a test run tomorrow.
See vfs.git#block-fixes, along with #work.iov_iter_get_pages-3 in there. Seems to work here...
Quoted 6 lines
If you are OK with #block-fixes (it's one commit on top of
bf8d08532bc1"iomap: add support for dma aligned direct-io" in block.git), the easiest way to deal with the conflicts would be to have that branch pulled into block.git. Jens, would you be OK with that in terms of tree topology? Provided that patch itself looks sane to you, of course...
I'm fine with that approach. Don't have too much time to look into this stuff just yet, but looks like you and Keith are getting it sorted out. I'll check in on emails later this weekend and we can get it pulled in at that point when you guys deem it ready to check/pull.
Signature · 1 line
Jens Axboe
Al Viro
maintainerRe: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Keith BuschQuoted 4 lines · Keith Busch
Validating user requests gets really messy if we allow arbitrary segment lengths. This particular patch just enables arbitrary address alignment, but segment size is still required to be a block size. You found the commit that enforces that earlier, "iov: introduce
iov_iter_aligned", two commits prior.
BTW, where do you check it for this caller?
fs/zonefs/super.c:786: ret = bio_iov_iter_get_pages(bio, from);
Incidentally, we have an incorrect use of iov_iter_truncate() in that one (compare with iomap case, where we reexpand it afterwards)...
I still don't get the logics of those round-downs. You've *already* verified that each segment is a multiple of logical block size. And you are stuffing as much as you can into bio, covering the data for as many segments as you can. Sure, you might end up e.g. running into an unmapped page at wrong offset (since your requirements for initial offsets might be milder than logical block size). Or you might run out of pages bio would take. Either might put the end of bio at the wrong offset.
So why not trim it down *after* you are done adding pages into it? And do it once, outside of the loop. IDGI... Validation is already done; I'm not suggesting to allow weird segment lengths or to change behaviour of your iov_iter_is_aligned() in any other way.
Put it another way, is there any possibility for __bio_iov_iter_get_pages() to do a non-trivial round-down on anything other than the last iteration of that loop?
Re: [block.git conflicts] Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
Quoted 12 lines · Al Viro
Quoted 4 lines · Keith Busch
Validating user requests gets really messy if we allow arbitrary segment lengths. This particular patch just enables arbitrary address alignment, but segment size is still required to be a block size. You found the commit that enforces that earlier, "iov: introduce
iov_iter_aligned", two commits prior.BTW, where do you check it for this caller?
fs/zonefs/super.c:786: ret = bio_iov_iter_get_pages(bio, from);Incidentally, we have an incorrect use of
iov_iter_truncate()in that one (compare with iomap case, where we reexpand it afterwards)...
Quoted 7 lines
I still don't get the logics of those round-downs. You've *already* verified that each segment is a multiple of logical block size. And you are stuffing as much as you can into bio, covering the data for as many segments as you can. Sure, you might end up e.g. running into an unmapped page at wrong offset (since your requirements for initial offsets might be milder than logical block size). Or you might run out of pages bio would take. Either might put the end of bio at the wrong offset.
It is strange this function allows the possibility that bio_iov_add_page() can fail. There's no reason to grab more pages that exceed the bio_full() condition (ignoring the special ZONE_APPEND case for the moment).
I think it will make more sense if we clean that part up first so the size for all successfully gotten pages can skip subsequent bio add page checks, and make the error handling unnecessary.
Quoted 4 lines
So why not trim it down *after* you are done adding pages into it? And do it once, outside of the loop. IDGI... Validation is already done; I'm not suggesting to allow weird segment lengths or to change behaviour of your
iov_iter_is_aligned()in any other way.
I may have misunderstood your previous suggestion, but I still think this is the right way to go. The ALIGN_DOWN in its current location ensures the size we're appending to the bio is acceptable before we even start. It's easier to prevent adding pages to a bio IO than to back them out later. The latter would need something like a special cased version of bio_truncate().
Anyway, I have some changes testing right now that I think will fix up the issues you've raised, and make the rest a bit more clear. I'll send them for consideration this weekend if all is succesful.
Quoted 3 lines
Put it another way, is there any possibility for
__bio_iov_iter_get_pages()to do a non-trivial round-down on anything other than the last iteration of that loop?
Christian Schoenebeck
reviewerRe: [PATCH 01/44] 9p: handling Rerror without copy_from_iter_full()
In reply to Dominique MartinetQuoted 26 lines · On Freitag, 1. Juli 2022 23:00:06 CEST Dominique Martinet wrote:
Christian Schoenebeck wrote on Fri, Jul 01, 2022 at 06:02:31PM +0200:
Quoted 14 lines
Quoted 8 lines
I also tried 9p2000.u for principle and ... I have no idea if this works but it didn't seem to blow up there at least.
The problem is that 9p2000.u just doesn't work well even without these patches, so I still stand by what I said about 9p2000.u and virtio (zc interface): we really can (and I think should) just say virtio doesn't support 9p2000.u.
(and could then further simplify this)Quoted 5 lines
If you're curious, 9p2000.u hangs without your patch on at least two different code paths (trying to read a huge buffer aborts sending a reply because msize is too small instead of clamping it, that one has a qemu warning message; but there are others ops like copyrange that just fail silently and I didn't investigate)
Quoted 6 lines
Last time I tested 9p2000.u was with the "remove msize limit" (WIP) patches:
https://lore.kernel.org/all/[email protected]/
Where I did not observe any issue with 9p2000.u.What msize are we talking about, or can you tell a way to reproduce?
I just ran fsstress on a
mount -t 9p -o cache=none,trans=virtio,version=9p2000.umount on v5.19-rc2:
Quoted 5 lines
fsstress -d
/mnt/fstress-n 1000 -vIf that doesn't reproduce for you (and you care) I can turn some more logs on, but from the look of it it could very well be msize related, I just didn't check as I don't expect any real user
Confirmed. :/ I tested with various kernel versions (also w/wo "remove msize limit" WIP patches), different combinations of msize and cache options. They all start to hang the fsstress app with 9p2000.u protocol version, sometimes sooner, sometimes later.
BTW, fsstress does not pass with 9p2000.L here either, it would not hang, but it consistently terminates fsstress with (cache=none|mmap):
posix_memalign: Invalid argument
@Greg: JFYI
Best regards,
Christian Schoenebeck
Re: [PATCH 40/44] 9p: convert to advancing variant of iov_iter_get_pages_alloc()
In reply to Christian SchoenebeckQuoted 11 lines · On Freitag, 1. Juli 2022 15:47:24 CEST Christian Schoenebeck wrote:
Quoted 3 lines · On Mittwoch, 22. Juni 2022 06:15:48 CEST Al Viro wrote:
that one is somewhat clumsier than usual and needs serious testing.
Signed-off-by Al Viro <[email protected]>Hi Al,
it took me a bit to find the patch that introduces
iov_iter_get_pages_alloc2(), but this patch itself looks fine:Reviewed-by Christian Schoenebeck <[email protected]>
Re: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}()
In reply to Al ViroQuoted 3 lines · Al Viro
... doing revert if we end up not using some pages
With the version from latest vfs.git#for-next as from [1] (which differs from this one) I see with LLVM-14:
5618: clang -Wp,-MMD,block/.bio.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generate
d/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Qunused-arguments -fmacro-prefix-map=./= -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-
PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 --target=x86_64-linux-gnu -fintegrated-as -Werror=un known-warning-option -Werror=ignored-optimization-argument -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-loops=1 -mno-80387 -mno-fp
-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mretpoline-external-th
unk -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Wimplicit-fallthrough -Wno-
gnu -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-stack-clash-protection -pg -mfentry -DCC_USING_NOP_MCOUNT -DCC_USING_FENTRY -fno-lto -flto=thin -fspli
t-lto-unit -fvisibility=hidden -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wcast-function-type -fno-strict-overflow -fno-stack-check -Werror=date-time -Werr or=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-pointer-to-enum-cast -Wno-tautological-constant-out
-of-range-compare -Wno-unaligned-access -g -gdwarf-5 -DKBUILD_MODFILE='"block/bio"' -DKBUILD_BASENAME='"bio"' -DKBUILD_MODNAME='"bio"' -D__KBUILD_MODNAME=kmod_bio -
c -o block/bio.o block/bio.c
[ ... ]
5635:block/bio.c:1232:6: warning: variable 'i' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
5636- if (unlikely(!size)) {
5637- ^~~~~~~~~~~~~~~5638-./include/linux/compiler.h:78:22: note: expanded from macro 'unlikely'
5639-# define unlikely(x) __builtin_expect(!!(x), 0)
5640- ^~~~~~~~~~~~~~~~~~~~~~~~~~5641-block/bio.c:1254:9: note: uninitialized use occurs here
5642- while (i < nr_pages)
5643- ^5644-block/bio.c:1232:2: note: remove the 'if' if its condition is always false
5645- if (unlikely(!size)) {
5646- ^~~~~~~~~~~~~~~~~~~~~~5647-block/bio.c:1202:17: note: initialize the variable 'i' to silence this warning
5648- unsigned len, i;
5649- ^
5650- = 0
Patch from [1] is attached.
Regards,
-Sedat-
Quoted 106 lines
Signed-off-by Al Viro <[email protected]>--- block/bio.c | 15 ++++++--------- block/blk-map.c | 7 ++++--- 2 files changed, 10 insertions(+), 12 deletions(-)block/bio.c +6
-9@@ -1190,7 +1190,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)BUILD_BUG_ON (PAGE_PTRS_PER_BVEC <2 ); pages += entries_left * (PAGE_PTRS_PER_BVEC -1 ); - size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); + size =iov_iter_get_pages2 (iter, pages,LONG_MAX , nr_pages, &offset);if (unlikely (size <=0 ))return size ? size : -EFAULT ;@@ -1205,6 +1205,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) }else {if (WARN_ON_ONCE (bio_full (bio, len))) {bio_put_pages (pages + i, left, offset); +iov_iter_revert (iter, left);return -EINVAL ; }__bio_add_page (bio, page, len, offset);@@ -1212,7 +1213,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) offset =0 ; } -iov_iter_advance (iter, size);return 0 ; }@@ -1227,7 +1227,6 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)ssize_t size, left;unsigned len, i;size_t offset; -int ret =0 ;if (WARN_ON_ONCE (!max_append_sectors))return 0 ;@@ -1240,7 +1239,7 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)BUILD_BUG_ON (PAGE_PTRS_PER_BVEC <2 ); pages += entries_left * (PAGE_PTRS_PER_BVEC -1 ); - size =iov_iter_get_pages (iter, pages,LONG_MAX , nr_pages, &offset); + size =iov_iter_get_pages2 (iter, pages,LONG_MAX , nr_pages, &offset);if (unlikely (size <=0 ))return size ? size : -EFAULT ;@@ -1252,16 +1251,14 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)if (bio_add_hw_page (q, bio, page, len, offset, max_append_sectors, &same_page) != len) {bio_put_pages (pages + i, left, offset); - ret = -EINVAL ; -break ; +iov_iter_revert (iter, left); +return -EINVAL ; }if (same_page)put_page (page); offset =0 ; } - -iov_iter_advance (iter, size - left); -return ret; +return 0 ; }/** block/blk-map.c +4
-4@@ -254,7 +254,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,size_t offs, added =0 ;int npages; - bytes =iov_iter_get_pages_alloc (iter, &pages,LONG_MAX , &offs); + bytes =iov_iter_get_pages_alloc2 (iter, &pages,LONG_MAX , &offs);if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ; goto out_unmap;@@ -284,7 +284,6 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, bytes -= n; offs =0 ; } -iov_iter_advance (iter, added); }/* * release the pages we didn't map into the bio, if any @@ -293,8 +292,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,put_page (pages[j++]);kvfree (pages);/* couldn't stuff something into bio? */ -if (bytes) +if (bytes) { +iov_iter_revert (iter, bytes);break ; + } } ret =blk_rq_append_bio (rq, bio); --
Quoted 2 lines
2.30.2
From 9a6469060316674230c0666c5706f7097e9278bb Mon Sep 17 00:00:00 2001
From: Al Viro <[email protected]>
Date: Thu, 9 Jun 2022 10:37:57 -0400
Subject: [PATCH] block: convert to advancing variants of
iov_iter_get_pages{,_alloc}()
... doing revert if we end up not using some pages
--- block/bio.c | 25 ++++++++++++++----------- block/blk-map.c | 7 ++++--- 2 files changed, 18 insertions(+), 14 deletions(-)
block/bio.c +14 -11
@@ -1200,7 +1200,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)struct page **pages = (struct page **)bv;ssize_t size, left;unsigned len, i; -size_t offset; +size_t offset, trim;int ret =0 ;/*
@@ -1218,16 +1218,19 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)* result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size =iov_iter_get_pages (iter, pages,UINT_MAX - bio->bi_iter.bi_size, + size =iov_iter_get_pages2 (iter, pages,UINT_MAX - bio->bi_iter.bi_size, nr_pages, &offset); -if (size >0 ) { - nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE ); - size =ALIGN_DOWN (size,bdev_logical_block_size (bio->bi_bdev)); - }else - nr_pages =0 ; - -if (unlikely (size <=0 )) { - ret = size ? size : -EFAULT ; +if (unlikely (size <=0 )) +return size ? size : -EFAULT ; + + nr_pages =DIV_ROUND_UP (offset + size,PAGE_SIZE ); + + trim = size & (bdev_logical_block_size (bio->bi_bdev) -1 ); +iov_iter_revert (iter, trim); + + size -= trim; +if (unlikely (!size)) { + ret = -EFAULT ; goto out; }
@@ -1246,7 +1249,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) offset =0 ; } -iov_iter_advance (iter, size - left); +iov_iter_revert (iter, left); out:while (i < nr_pages)put_page (pages[i++]);
block/blk-map.c +4 -3
@@ -254,7 +254,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,size_t offs, added =0 ;int npages; - bytes =iov_iter_get_pages_alloc (iter, &pages,LONG_MAX , &offs); + bytes =iov_iter_get_pages_alloc2 (iter, &pages,LONG_MAX , &offs);if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ; goto out_unmap;
@@ -284,7 +284,6 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, bytes -= n; offs =0 ; } -iov_iter_advance (iter, added); }/* * release the pages we didn't map into the bio, if any
@@ -293,8 +292,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,put_page (pages[j++]);kvfree (pages);/* couldn't stuff something into bio? */ -if (bytes) +if (bytes) { +iov_iter_revert (iter, bytes);break ; + } } ret =blk_rq_append_bio (rq, bio);
Signature · 1 line
2.36.1
Alexander Gordeev
maintainerRe: [PATCH 9/44] new iov_iter flavour - ITER_UBUF
In reply to Al ViroQuoted 12 lines · Al Viro
Equivalent of single-segment iovec. Initialized by
iov_iter_ubuf(), checked for byiter_is_ubuf(), otherwise behaves likeITER_IOVECones.We are going to expose the things like ->write_iter() et.al. to those in subsequent commits.
New predicate (
user_backed_iter()) that is true forITER_IOVECandITER_UBUF; places like direct-IO handling should use that for checking that pages we modify after getting them fromiov_iter_get_pages()would need to be dirtied.
Quoted 7 lines
DO NOT assume that replacing
iter_is_iovec()withuser_backed_iter()will solve all problems - there's code that usesiter_is_iovec()to decide how to poke around iniov_iterguts and for that the predicate replacement obviously won't suffice.Signed-off-by Al Viro <[email protected]>
Hi Al,
This changes causes sendfile09 LTP testcase fail in linux-next (up to next-20220727) on s390. In fact, not this change exactly, but rather 92d4d18eecb9 ("new iov_iter flavour - ITER_UBUF") - which differs from what is posted here.
AFAICT page_cache_pipe_buf_confirm() encounters !PageUptodate() and !page->mapping page and returns -ENODATA.
I am going to narrow the testcase and get more details, but please let me know if I am missing something.
Thanks!
Al Viro
maintainerRe: [PATCH 9/44] new iov_iter flavour - ITER_UBUF
Quoted 28 lines · Alexander Gordeev
Quoted 12 lines · Al Viro
Equivalent of single-segment iovec. Initialized by
iov_iter_ubuf(), checked for byiter_is_ubuf(), otherwise behaves likeITER_IOVECones.We are going to expose the things like ->write_iter() et.al. to those in subsequent commits.
New predicate (
user_backed_iter()) that is true forITER_IOVECandITER_UBUF; places like direct-IO handling should use that for checking that pages we modify after getting them fromiov_iter_get_pages()would need to be dirtied.Quoted 7 lines
DO NOT assume that replacing
iter_is_iovec()withuser_backed_iter()will solve all problems - there's code that usesiter_is_iovec()to decide how to poke around iniov_iterguts and for that the predicate replacement obviously won't suffice.Signed-off-by Al Viro <[email protected]>Hi Al,
This changes causes sendfile09 LTP testcase fail in linux-next (up to next-20220727) on s390. In fact, not this change exactly, but rather
92d4d18eecb9("newiov_iterflavour -ITER_UBUF") - which differs from what is posted here.
Quoted 5 lines
AFAICT
page_cache_pipe_buf_confirm()encounters !PageUptodate() and !page->mapping page and returns-ENODATA.I am going to narrow the testcase and get more details, but please let me know if I am missing something.
Grrr....
- } else if (iter_is_iovec(to)) {
+ } else if (!user_backed_iter(to)) {
in mm/shmem.c. Spot the typo...
Could you check if replacing that line with
} else if (user_backed_iter(to)) {fixes the breakage?
Alexander Gordeev
maintainerRe: [PATCH 9/44] new iov_iter flavour - ITER_UBUF
Quoted 18 lines · Al Viro
Quoted 7 lines
Hi Al,
This changes causes sendfile09 LTP testcase fail in linux-next (up to next-20220727) on s390. In fact, not this change exactly, but rather
92d4d18eecb9("newiov_iterflavour -ITER_UBUF") - which differs from what is posted here.Quoted 5 lines
AFAICT
page_cache_pipe_buf_confirm()encounters !PageUptodate() and !page->mapping page and returns-ENODATA.I am going to narrow the testcase and get more details, but please let me know if I am missing something.
Grrr....
(quoted diff) +1
-1- } else if (iter_is_iovec(to)) { + } else if (!user_backed_iter(to)) {
Quoted 6 lines
in
mm/shmem.c. Spot the typo...Could you check if replacing that line with
} else if (user_backed_iter(to)) {fixes the breakage?
Yes, it does! So just to be sure - this is the fix:
mm/shmem.c +1 -1
@@ -2626,7 +2626,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) ret =copy_page_to_iter (page, offset, nr, to);put_page (page); - }else if (!user_backed_iter (to)) { + }else if (user_backed_iter (to)) {/* * Copy to user tends to be so well optimized, but * clear_user() not so much, that it is noticeably
Thanks!
Al Viro
maintainerRe: [PATCH 9/44] new iov_iter flavour - ITER_UBUF
Quoted 19 lines · Alexander Gordeev
Quoted 18 lines · Al Viro
Quoted 7 lines
Hi Al,
This changes causes sendfile09 LTP testcase fail in linux-next (up to next-20220727) on s390. In fact, not this change exactly, but rather
92d4d18eecb9("newiov_iterflavour -ITER_UBUF") - which differs from what is posted here.Quoted 5 lines
AFAICT
page_cache_pipe_buf_confirm()encounters !PageUptodate() and !page->mapping page and returns-ENODATA.I am going to narrow the testcase and get more details, but please let me know if I am missing something.
Grrr....
(quoted diff) +1
-1- } else if (iter_is_iovec(to)) { + } else if (!user_backed_iter(to)) {
Quoted 8 lines
Quoted 6 lines
in
mm/shmem.c. Spot the typo...Could you check if replacing that line with
} else if (user_backed_iter(to)) {fixes the breakage?
Yes, it does! So just to be sure - this is the fix:
FWIW, there'd been another braino, caught by test from Hugh Dickins; this one in ITER_PIPE: allocate buffers as we go in copy-to-pipe primitives
Incremental follows; folded and pushed out.
lib/iov_iter.c +2 -2
@@ -469,7 +469,7 @@ static size_t copy_pipe_to_iter(const void *addr, size_t bytes,struct page *page =append_pipe (i, n, &off); chunk =min_t (size_t, n,PAGE_SIZE - off);if (!page) -break ; +return bytes - n;memcpy_to_page (page, off, addr, chunk); addr += chunk; }
@@ -774,7 +774,7 @@ static size_t pipe_zero(size_t bytes, struct iov_iter *i)char *p;if (!page) -break ; +return bytes - n; chunk =min_t (size_t, n,PAGE_SIZE - off); p =kmap_local_page (page);memset (p + off,0 , chunk);
mm/shmem.c +1 -1
@@ -2603,7 +2603,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) ret =copy_page_to_iter (page, offset, nr, to);put_page (page); - }else if (!user_backed_iter (to)) { + }else if (user_backed_iter (to)) {/* * Copy to user tends to be so well optimized, but * clear_user() not so much, that it is noticeably
Re: [PATCH 09/44] new iov_iter flavour - ITER_UBUF
In reply to Al ViroYou need to modify dup_iter() also. That will go through the:
return new->iov = kmemdup(new->iov, new->nr_segs * sizeof(struct iovec), flags);
case with a ubuf-class iterators, which will clobber new->ubuf.
David
Al Viro
maintainerRe: [PATCH 09/44] new iov_iter flavour - ITER_UBUF
Quoted 9 lines · David Howells
You need to modify
dup_iter()also. That will go through the:return new->iov = kmemdup(new->iov, new->nr_segs * sizeof(struct iovec), flags);case with a ubuf-class iterators, which will clobber
new->ubuf.David
Fixed, folded and pushed out. Incremental:
lib/iov_iter.c +2 -3
@@ -1659,17 +1659,16 @@ const void *dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags)WARN_ON (1 );return NULL ; } -if (unlikely (iov_iter_is_discard (new) ||iov_iter_is_xarray (new))) -return NULL ;if (iov_iter_is_bvec (new))return new->bvec =kmemdup (new->bvec, new->nr_segs *sizeof (struct bio_vec ), flags); -else +else if (iov_iter_is_kvec (new) ||iter_is_iovec (new))/* iovec and kvec have identical layout */ return new->iov =kmemdup (new->iov, new->nr_segs *sizeof (struct iovec ), flags); +return NULL ; }EXPORT_SYMBOL (dup_iter);
Re: [PATCH 09/44] new iov_iter flavour - ITER_UBUF
lib/iov_iter.c const void *dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) · line 1666
Quoted 6 lines · Al Viro
lib/iov_iter.c +1
-1if (iov_iter_is_bvec (new))return new->bvec =kmemdup (new->bvec, new->nr_segs *sizeof (struct bio_vec ), flags); -else +else if (iov_iter_is_kvec (new) ||iter_is_iovec (new))
The else is redundant.
David