iov_iter: Add extraction functions
Subsystems (2)
RFC PATCH 1/1 iov_iter: Add extraction functions
Hi Al, Jeff,
Here's a replacement for the extract_iter_to_iter() patch I had previously. It's a WIP, some bits aren't fully implemented, though some bits I have tested and got to work, but if you could take a look and see if you're okay with the interface.
I think I've addressed most of Al's comments. The page-pinning is conditional on certain types of iterator, and a number of the iterator types just extract to the same thing. It should now handle kvec-class iterators that refer to vmalloc'd data.
I've also added extraction to scatterlist (which I'll need for doing various crypto things) and extraction to ib_sge which could be used in cifs/smb RDMA, bypassing the conversion-to-scatterlist step.
As mentioned, there are bits that aren't fully implemented, let alone tested.
David
---iov_iter: Add extraction functions
Add extraction functions to extract the page content from an I/O iterator to one of three destinations:
(1) extract_iter_to_iter()
Builds a new iterator from the source iterator such that the new iterator remains valid if the source iterator gets deallocated, such as can happen in asynchronous I/O when -EIOCBQUEUED is returned.
For UBUF/IOVEC-class iterators the output iteratior will be BVEC-class and data/buffer pages are pinned to prevent them being moved, swapped out or discarded for the duration.
For XARRAY-class iterators, the new iterator is copied and then trimmed; no page pinning is done. For BVEC- and KVEC-class iterators, the bio_vec/kvec table is copied and trimmed; again no page pinning.
KVEC-class iterators with vmalloc'd areas should work.
DISCARD- and PIPE-class iterators are not currently supported and incur an error.
The extraction function fills out a "cleanup" record that can then be passed to iov_iter_clean_up() once the I/O is complete. This will undo any pinning and free any allocated bufferage.
Tested with DIO read: IOVEC, UBUF.
Tested just dumping iterator: BVEC, KVEC.
Untested: XARRAY.
(2) extract_iter_to_sg()
Similar to above, but builds a scatterlist and attaches it to an sg_table instead of a new iterator. Returns an indication if the pages were pinned.
The caller is responsible for freeing the scatterlist and unpinning any pages.
Tested to dump sglist: IOVEC, UBUF, KVEC.
Untested: BVEC, XARRAY.
(3) extract_iter_to_rdma()
Similar again, but fills in an ib_sge struct array, mapping the pages to DMA addresses appropriate to the specified driver.
The caller provides the ib_sge array, so it's up to the caller to clean that up. No page-pinning is done as ib_sge does not provide a place to note the source page(s), so only BVEC-, KVEC- and XARRAY-class iterators are supported for now. One possible solution to that is to pass the iterator through extract_iter_to_iter() first.
Completely untested and needs more modification to deal with errors from ib_dma_map_page().
Note that for the moment, none of these advance the source iterator.
An additional function, iov_iter_flush_dcache(), is provided to do dcache flushing over the source buffer, using the information provided in the extracted iterator and the cleanup record. This works for BVEC-class iterators, but is incompletely implemented for KVEC-class and probably wrong for XARRAY-class.
---
include/linux/uio2.h new file +45 -0
@@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* iov_iter extractors +* +* Copyright (C) 2022 Red Hat, Inc. All Rights Reserved. +* Written by David Howells ([email protected]) +*/ + +#ifndef _LINUX_UIO2_H +#define _LINUX_UIO2_H + +#include <linux/uio.h> +#include <linux/dma-direction.h> + +struct sg_table ; +struct ib_device ; +struct ib_sge ; + +/* +* Cleanup information for an extracted iterator. +*/ +struct iov_iter_cleanup { +union { +struct bio_vec *bv; +struct kvec *kv; + }; +unsigned int nr_segs; +bool pages_pinned; +enum iter_type type:8 ; +}; + +ssize_t extract_iter_to_iter (struct iov_iter *iter,size_t len, +struct iov_iter *to, +struct iov_iter_cleanup *cleanup); +ssize_t extract_iter_to_sg (struct iov_iter *iter,size_t len, +struct sg_table *sgtable,bool *pinned); +ssize_t extract_iter_to_rdma (struct iov_iter *iter,size_t len, +struct ib_device *device,u32 local_dma_lkey, +enum dma_data_direction direction, +struct ib_sge *sge,unsigned int max_sge, +unsigned int *nr_sge); +void iov_iter_flush_dcache (struct iov_iter *iter, +struct iov_iter_cleanup *cleanup); +void iov_iter_clean_up (struct iov_iter_cleanup *cleanup); + +#endif /* _LINUX_UIO2_H */
lib/Makefile +1 -1
@@ -44,7 +44,7 @@ obj-y += lockref.o obj-y += bcd.o sort.o parser.o debug_locks.o random32.o \ bust_spinlocks.o kasprintf.o bitmap.o scatterlist.o \ - list_sort.o uuid.o iov_iter.o clz_ctz.o \ + list_sort.o uuid.o iov_iter.o iov_iter_extract.o clz_ctz.o \ bsearch.o find_bit.o llist.o memweight.o kfifo.o \ percpu-refcount.o rhashtable.o base64.o \ once.o refcount.o usercopy.o errseq.o bucket_locks.o \
lib/iov_iter_extract.c new file +653 -0
653 lines not on this page · raw message
kernel test robot
build bot[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in…
Hi David,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on linus/master]
[also build test WARNING on v6.0-rc4 next-20220909]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/David-Howells/iov_iter-Add-extraction-functions/20220910-072102 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ce888220d5c7a805e0e155302a318d5d23e62950 config: s390-randconfig-r044-20220907 (https://download.01.org/0day-ci/archive/20220910/[email protected]/config) compiler: clang version 16.0.0 (https://github.com/llvm/llvm-project 1546df49f5a6d09df78f569e4137ddb365a3e827)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install s390 cross compiling tool for clang build
# apt-get install binutils-s390x-linux-gnu
# https://github.com/intel-lab-lkp/linux/commit/a8df8a897880379087a0a9fc4f6fc1391e352217
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review David-Howells/iov_iter-Add-extraction-functions/20220910-072102 git checkout a8df8a897880379087a0a9fc4f6fc1391e352217
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=s390 SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
All warnings (new ones prefixed by >>):
In file included from lib/iov_iter_extract.c:16:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:75:
include/asm-generic/io.h:547:31: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __raw_readb(PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:560:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __le16_to_cpu((__le16 __force)__raw_readw(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/big_endian.h:37:59: note: expanded from macro '__le16_to_cpu'
#define __le16_to_cpu(x) __swab16((__force __u16)(__le16)(x))
^
include/uapi/linux/swab.h:102:54: note: expanded from macro '__swab16'
#define __swab16(x) (__u16)__builtin_bswap16((__u16)(x))
^
In file included from lib/iov_iter_extract.c:16:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:75:
include/asm-generic/io.h:573:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/big_endian.h:35:59: note: expanded from macro '__le32_to_cpu'
#define __le32_to_cpu(x) __swab32((__force __u32)(__le32)(x))
^
include/uapi/linux/swab.h:115:54: note: expanded from macro '__swab32'
#define __swab32(x) (__u32)__builtin_bswap32((__u32)(x))
^
In file included from lib/iov_iter_extract.c:16:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:75:
include/asm-generic/io.h:584:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writeb(value, PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:594:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writew((u16 __force)cpu_to_le16(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:604:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writel((u32 __force)cpu_to_le32(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:692:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
readsb(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:700:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
readsw(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:708:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
readsl(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:717:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
writesb(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:726:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
writesw(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:735:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
writesl(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
>> lib/iov_iter_extract.c:348:3: warning: variable 'ret' is uninitialized when used here [-Wuninitialized]
ret += len;
^~~
lib/iov_iter_extract.c:329:13: note: initialize the variable 'ret' to silence this warning
ssize_t ret;
^
= 013 warnings generated.
vim +/ret +348 lib/iov_iter_extract.c
314
315 /*
316 * Extract the pages from an XARRAY-class iterator and add them to the
317 * destination buffer. The pages are not pinned.
318 */
319 static ssize_t iov_iter_extract_xarray(struct iov_iter *iter,
320 void *array, unsigned int array_max,
321 ssize_t maxsize,
322 enum iter_extract_dest dest)
323 {
324 struct xarray *xa = iter->xarray;
325 struct folio *folio;
326 unsigned int ix;
327 loff_t start = iter->xarray_start + iter->iov_offset;
328 pgoff_t index = start / PAGE_SIZE;
329 ssize_t ret;
330 size_t offset, len;
331 XA_STATE(xas, xa, index);
332
333 rcu_read_lock();
334 xas_for_each(&xas, folio, ULONG_MAX) {
335 if (xas_retry(&xas, folio))
336 continue;
337 if (WARN_ON(xa_is_value(folio)))
338 break;
339 if (WARN_ON(folio_test_hugetlb(folio)))
340 break;
341
342 offset = offset_in_folio(folio, start);
343 len = min_t(size_t, maxsize, folio_size(folio) - offset);
344
345 ix = extract_contig_pages(array, folio_page(folio, 0),
346 offset, len, dest);
347 maxsize -= len;
> 348 ret += len; 349 if (ix >= array_max) {
350 WARN_ON_ONCE(ix > array_max);
351 break;
352 }
353
354 if (maxsize <= 0)
355 break;
356 }
357
358 rcu_read_unlock();
359 terminate_array(array, dest);
360 return ret;
361 }
362
Signature · 2 lines
0-DAY CI Kernel Test Service https://01.org/lkp
Al Viro
maintainerQuoted 12 lines · David Howells
Hi Al, Jeff,
Here's a replacement for the
extract_iter_to_iter()patch I had previously. It's a WIP, some bits aren't fully implemented, though some bits I have tested and got to work, but if you could take a look and see if you're okay with the interface.I think I've addressed most of Al's comments. The page-pinning is conditional on certain types of iterator, and a number of the iterator types just extract to the same thing. It should now handle kvec-class iterators that refer to vmalloc'd data.
Quoted 5 lines
I've also added extraction to scatterlist (which I'll need for doing various crypto things) and extraction to
ib_sgewhich could be used in cifs/smb RDMA, bypassing the conversion-to-scatterlist step.As mentioned, there are bits that aren't fully implemented, let alone tested.
IDGI. Essentially, you are passing a callback disguised as enum, only to lose any type safety. How is it better than "iov_iter_get_pages2() into a fixed-sized array and handle the result" done in a loop? No need to advance it (iov_iter_get_page2() auto-advances), *way* fewer conditional branches and no need to share anything between the 3 functions you are after...
lib/iov_iter_extract.c line 478
Quoted 2 lines
lib/iov_iter_extract.c new file +2
-0+ssize_t extract_iter_to_sg (struct iov_iter *iter,size_t len, +struct sg_table *sgtable,bool *pages_pinned)
Your *pages_pinned is user_backed_iter(iter), isn't it?
Quoted 2 lines · Al Viro
How is it better than "
iov_iter_get_pages2()into a fixed-sized array and handle the result" done in a loop?
I did that first - and you objected on the basis that I shouldn't be taking page refs if it's not necessary, and under some circumstances some of the pages pointed to by the iterator might not be something that I should attempt to take a ref on.
David
Quoted 2 lines · Al Viro
IDGI. Essentially, you are passing a callback disguised as enum, only to lose any type safety.
Granted, but this is how struct iov_iter works. I'm keeping the enum inside the internal functions where it can't be got at by external code.
Now, I could just copy the code three times, once for iter=>bvec, once for iter=>sg and once for iter=>rdma. I can also grant that iter=>bvec is really only needed for UBUF/IOVEC-class iterators; I think I can make the assumption that kernel-supplied buffers are good for the lifetime of the operation.
Quoted 2 lines
How is it better than "
iov_iter_get_pages2()into a fixed-sized array and handle the result" done in a loop?
Note that iov_iter_get_pages2() doesn't handle KVEC-class iterators, which this code does - for kmalloc'd, vmalloc'd and vmap'd memory and for global and stack variables. What I've written gets the physical addresses but doesn't/can't pin it (which probably means I can't just move my code into iov_iter_get_pages2()).
Further, my code also treats multipage folios as single units which iov_iter_get_pages2() also doesn't - at least from XARRAY-class iterators.
The UBUF-/IOVEC-extraction code doesn't handle multipage folios because get_user_pages_fast() doesn't - though perhaps it will need to at some point.
David
Quoted 5 lines · David Howells
Note that
iov_iter_get_pages2()doesn't handle KVEC-class iterators, which this code does - for kmalloc'd, vmalloc'd and vmap'd memory and for global and stack variables. What I've written gets the physical addresses but doesn't/can't pin it (which probably means I can't just move my code intoiov_iter_get_pages2()).
Please look at the long discussion on Ira's patches for iov_iter_pin_pages where we went through this over and over again. We need to handle all of these in the common helper.
Quoted 5 lines
Further, my code also treats multipage folios as single units which
iov_iter_get_pages2()also doesn't - at least from XARRAY-class iterators.The UBUF-/IOVEC-extraction code doesn't handle multipage folios because
get_user_pages_fast()doesn't - though perhaps it will need to at some point.
Yes, we need to be smarter about folios. But that is nothing magic about network file systems. I have old patches that added a (barely working) variant of get_user_pages_fast that exported bio_vecs and I think willy planned to look into a folio based version of get/pin user pages. But one thing at a time. I think the work from Ira to switch to proper pinning and sort out the weirdnesses for non-users buffer types is the first priority. After that proper folio or large contiguous range support is another item. But none of that is helped by parallel infrastructure.