Skip to contentSkip to conversation

iov_iter: Add extraction functions

RFC PATCH In review David Howells 4 participants 6 messages Sep 9, 2022 – Oct 18, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 iov_iter: Add extraction functions

Hi Al, Jeff,

Here's a replacement for the extract_iter_to_iter() patch I had previously. It's a WIP, some bits aren't fully implemented, though some bits I have tested and got to work, but if you could take a look and see if you're okay with the interface.

I think I've addressed most of Al's comments. The page-pinning is conditional on certain types of iterator, and a number of the iterator types just extract to the same thing. It should now handle kvec-class iterators that refer to vmalloc'd data.

I've also added extraction to scatterlist (which I'll need for doing various crypto things) and extraction to ib_sge which could be used in cifs/smb RDMA, bypassing the conversion-to-scatterlist step.

As mentioned, there are bits that aren't fully implemented, let alone tested.

David
---
iov_iter: Add extraction functions

Add extraction functions to extract the page content from an I/O iterator to one of three destinations:

(1) extract_iter_to_iter()

Builds a new iterator from the source iterator such that the new iterator remains valid if the source iterator gets deallocated, such as can happen in asynchronous I/O when -EIOCBQUEUED is returned.

For UBUF/IOVEC-class iterators the output iteratior will be BVEC-class and data/buffer pages are pinned to prevent them being moved, swapped out or discarded for the duration.

For XARRAY-class iterators, the new iterator is copied and then trimmed; no page pinning is done. For BVEC- and KVEC-class iterators, the bio_vec/kvec table is copied and trimmed; again no page pinning.

KVEC-class iterators with vmalloc'd areas should work.

DISCARD- and PIPE-class iterators are not currently supported and incur an error.

The extraction function fills out a "cleanup" record that can then be passed to iov_iter_clean_up() once the I/O is complete. This will undo any pinning and free any allocated bufferage.

Tested with DIO read: IOVEC, UBUF.
Tested just dumping iterator: BVEC, KVEC.
Untested: XARRAY.

(2) extract_iter_to_sg()

Similar to above, but builds a scatterlist and attaches it to an sg_table instead of a new iterator. Returns an indication if the pages were pinned.

The caller is responsible for freeing the scatterlist and unpinning any pages.

Tested to dump sglist: IOVEC, UBUF, KVEC.
Untested: BVEC, XARRAY.

(3) extract_iter_to_rdma()

Similar again, but fills in an ib_sge struct array, mapping the pages to DMA addresses appropriate to the specified driver.

The caller provides the ib_sge array, so it's up to the caller to clean that up. No page-pinning is done as ib_sge does not provide a place to note the source page(s), so only BVEC-, KVEC- and XARRAY-class iterators are supported for now. One possible solution to that is to pass the iterator through extract_iter_to_iter() first.

Completely untested and needs more modification to deal with errors from ib_dma_map_page().

Note that for the moment, none of these advance the source iterator.

An additional function, iov_iter_flush_dcache(), is provided to do dcache flushing over the source buffer, using the information provided in the extracted iterator and the cleanup record. This works for BVEC-class iterators, but is incompletely implemented for KVEC-class and probably wrong for XARRAY-class.

---

include/linux/uio2.h new file +45 -0
new file mode 100644
index 000000000000..bc3e5ea96de7
@@ -0,0 +1,45 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* iov_iter extractors
+ *
+ * Copyright (C) 2022 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells ([email protected])
+ */
+
+#ifndef _LINUX_UIO2_H
+#define _LINUX_UIO2_H
+
+#include <linux/uio.h>
+#include <linux/dma-direction.h>
+
+struct sg_table;
+struct ib_device;
+struct ib_sge;
+
+/*
+ * Cleanup information for an extracted iterator.
+ */
+struct iov_iter_cleanup {
+	union {
+		struct bio_vec	*bv;
+		struct kvec	*kv;
+	};
+	unsigned int	nr_segs;
+	bool		pages_pinned;
+	enum iter_type	type:8;
+};
+
+ssize_t extract_iter_to_iter(struct iov_iter *iter, size_t len,
+			     struct iov_iter *to,
+			     struct iov_iter_cleanup *cleanup);
+ssize_t extract_iter_to_sg(struct iov_iter *iter, size_t len,
+			   struct sg_table *sgtable, bool *pinned);
+ssize_t extract_iter_to_rdma(struct iov_iter *iter, size_t len,
+			     struct ib_device *device, u32 local_dma_lkey,
+			     enum dma_data_direction direction,
+			     struct ib_sge *sge, unsigned int max_sge,
+			     unsigned int *nr_sge);
+void iov_iter_flush_dcache(struct iov_iter *iter,
+			   struct iov_iter_cleanup *cleanup);
+void iov_iter_clean_up(struct iov_iter_cleanup *cleanup);
+
+#endif /* _LINUX_UIO2_H */
lib/Makefile +1 -1
index 5927d7fa0806..7d46c1a68322 100644
@@ -44,7 +44,7 @@ obj-y	+= lockref.o
 
 obj-y += bcd.o sort.o parser.o debug_locks.o random32.o \
 	 bust_spinlocks.o kasprintf.o bitmap.o scatterlist.o \
-	 list_sort.o uuid.o iov_iter.o clz_ctz.o \
+	 list_sort.o uuid.o iov_iter.o iov_iter_extract.o clz_ctz.o \
 	 bsearch.o find_bit.o llist.o memweight.o kfifo.o \
 	 percpu-refcount.o rhashtable.o base64.o \
 	 once.o refcount.o usercopy.o errseq.o bucket_locks.o \
lib/iov_iter_extract.c new file +653 -0

653 lines not on this page · raw message

#1 lore raw
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in…

Hi David,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on linus/master]
[also build test WARNING on v6.0-rc4 next-20220909]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/David-Howells/iov_iter-Add-extraction-functions/20220910-072102
base:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ce888220d5c7a805e0e155302a318d5d23e62950
config: s390-randconfig-r044-20220907 (https://download.01.org/0day-ci/archive/20220910/[email protected]/config)
compiler: clang version 16.0.0 (https://github.com/llvm/llvm-project 1546df49f5a6d09df78f569e4137ddb365a3e827)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install s390 cross compiling tool for clang build
# apt-get install binutils-s390x-linux-gnu
# https://github.com/intel-lab-lkp/linux/commit/a8df8a897880379087a0a9fc4f6fc1391e352217
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review David-Howells/iov_iter-Add-extraction-functions/20220910-072102 git checkout a8df8a897880379087a0a9fc4f6fc1391e352217
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=s390 SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable

All warnings (new ones prefixed by >>):

In file included from lib/iov_iter_extract.c:16:
In file included from include/linux/scatterlist.h:9:
In file included from arch/s390/include/asm/io.h:75:
include/asm-generic/io.h:547:31: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]

           val = __raw_readb(PCI_IOBASE + addr);
                             ~~~~~~~~~~ ^
   include/asm-generic/io.h:560:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           val = __le16_to_cpu((__le16 __force)__raw_readw(PCI_IOBASE + addr));
                                                           ~~~~~~~~~~ ^
   include/uapi/linux/byteorder/big_endian.h:37:59: note: expanded from macro '__le16_to_cpu'
   #define __le16_to_cpu(x) __swab16((__force __u16)(__le16)(x))
                                                             ^
   include/uapi/linux/swab.h:102:54: note: expanded from macro '__swab16'
   #define __swab16(x) (__u16)__builtin_bswap16((__u16)(x))
                                                        ^
   In file included from lib/iov_iter_extract.c:16:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/s390/include/asm/io.h:75:
   include/asm-generic/io.h:573:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
                                                           ~~~~~~~~~~ ^
   include/uapi/linux/byteorder/big_endian.h:35:59: note: expanded from macro '__le32_to_cpu'
   #define __le32_to_cpu(x) __swab32((__force __u32)(__le32)(x))
                                                             ^
   include/uapi/linux/swab.h:115:54: note: expanded from macro '__swab32'
   #define __swab32(x) (__u32)__builtin_bswap32((__u32)(x))
                                                        ^
   In file included from lib/iov_iter_extract.c:16:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/s390/include/asm/io.h:75:
   include/asm-generic/io.h:584:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           __raw_writeb(value, PCI_IOBASE + addr);
                               ~~~~~~~~~~ ^
   include/asm-generic/io.h:594:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           __raw_writew((u16 __force)cpu_to_le16(value), PCI_IOBASE + addr);
                                                         ~~~~~~~~~~ ^
   include/asm-generic/io.h:604:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           __raw_writel((u32 __force)cpu_to_le32(value), PCI_IOBASE + addr);
                                                         ~~~~~~~~~~ ^
   include/asm-generic/io.h:692:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           readsb(PCI_IOBASE + addr, buffer, count);
                  ~~~~~~~~~~ ^
   include/asm-generic/io.h:700:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           readsw(PCI_IOBASE + addr, buffer, count);
                  ~~~~~~~~~~ ^
   include/asm-generic/io.h:708:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           readsl(PCI_IOBASE + addr, buffer, count);
                  ~~~~~~~~~~ ^
   include/asm-generic/io.h:717:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           writesb(PCI_IOBASE + addr, buffer, count);
                   ~~~~~~~~~~ ^
   include/asm-generic/io.h:726:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           writesw(PCI_IOBASE + addr, buffer, count);
                   ~~~~~~~~~~ ^
   include/asm-generic/io.h:735:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           writesl(PCI_IOBASE + addr, buffer, count);
                   ~~~~~~~~~~ ^
>> lib/iov_iter_extract.c:348:3: warning: variable 'ret' is uninitialized when used here [-Wuninitialized]
                   ret += len;
                   ^~~
   lib/iov_iter_extract.c:329:13: note: initialize the variable 'ret' to silence this warning
           ssize_t ret;
                      ^
                       = 0

13 warnings generated.

vim +/ret +348 lib/iov_iter_extract.c

314
315 /*
316 * Extract the pages from an XARRAY-class iterator and add them to the
317 * destination buffer. The pages are not pinned.
318 */
319 static ssize_t iov_iter_extract_xarray(struct iov_iter *iter,

   320					       void *array, unsigned int array_max,
   321					       ssize_t maxsize,
   322					       enum iter_extract_dest dest)
   323	{
   324		struct xarray *xa = iter->xarray;
   325		struct folio *folio;
   326		unsigned int ix;
   327		loff_t start = iter->xarray_start + iter->iov_offset;
   328		pgoff_t index = start / PAGE_SIZE;
   329		ssize_t ret;
   330		size_t offset, len;
   331		XA_STATE(xas, xa, index);
   332	
   333		rcu_read_lock();
   334		xas_for_each(&xas, folio, ULONG_MAX) {
   335			if (xas_retry(&xas, folio))
   336				continue;
   337			if (WARN_ON(xa_is_value(folio)))
   338				break;
   339			if (WARN_ON(folio_test_hugetlb(folio)))
   340				break;
   341	
   342			offset = offset_in_folio(folio, start);
   343			len = min_t(size_t, maxsize, folio_size(folio) - offset);
   344	
   345			ix = extract_contig_pages(array, folio_page(folio, 0),
   346						  offset, len, dest);
   347			maxsize -= len;
 > 348			ret += len;

349 if (ix >= array_max) {
350 WARN_ON_ONCE(ix > array_max);
351 break;
352 }
353
354 if (maxsize <= 0)
355 break;
356 }
357
358 rcu_read_unlock();
359 terminate_array(array, dest);
360 return ret;
361 }
362

Signature · 2 lines
0-DAY CI Kernel Test Service
https://01.org/lkp
maintainer
#2 lore raw
In reply to David Howells
Quoted 12 lines · David Howells

Hi Al, Jeff,

Here's a replacement for the extract_iter_to_iter() patch I had previously. It's a WIP, some bits aren't fully implemented, though some bits I have tested and got to work, but if you could take a look and see if you're okay with the interface.

I think I've addressed most of Al's comments. The page-pinning is conditional on certain types of iterator, and a number of the iterator types just extract to the same thing. It should now handle kvec-class iterators that refer to vmalloc'd data.

Quoted 5 lines

I've also added extraction to scatterlist (which I'll need for doing various crypto things) and extraction to ib_sge which could be used in cifs/smb RDMA, bypassing the conversion-to-scatterlist step.

As mentioned, there are bits that aren't fully implemented, let alone tested.

IDGI. Essentially, you are passing a callback disguised as enum, only to lose any type safety. How is it better than "iov_iter_get_pages2() into a fixed-sized array and handle the result" done in a loop? No need to advance it (iov_iter_get_page2() auto-advances), *way* fewer conditional branches and no need to share anything between the 3 functions you are after...

lib/iov_iter_extract.c line 478

Quoted 2 lines
lib/iov_iter_extract.c new file +2 -0
+ssize_t extract_iter_to_sg(struct iov_iter *iter, size_t len,
+			   struct sg_table *sgtable, bool *pages_pinned)

Your *pages_pinned is user_backed_iter(iter), isn't it?

#3 lore raw
Quoted 2 lines · Al Viro

How is it better than "iov_iter_get_pages2() into a fixed-sized array and handle the result" done in a loop?

I did that first - and you objected on the basis that I shouldn't be taking page refs if it's not necessary, and under some circumstances some of the pages pointed to by the iterator might not be something that I should attempt to take a ref on.

David

#4 lore raw
In reply to Al Viro
Quoted 2 lines · Al Viro

IDGI. Essentially, you are passing a callback disguised as enum, only to lose any type safety.

Granted, but this is how struct iov_iter works. I'm keeping the enum inside the internal functions where it can't be got at by external code.

Now, I could just copy the code three times, once for iter=>bvec, once for iter=>sg and once for iter=>rdma. I can also grant that iter=>bvec is really only needed for UBUF/IOVEC-class iterators; I think I can make the assumption that kernel-supplied buffers are good for the lifetime of the operation.

Quoted 2 lines

How is it better than "iov_iter_get_pages2() into a fixed-sized array and handle the result" done in a loop?

Note that iov_iter_get_pages2() doesn't handle KVEC-class iterators, which this code does - for kmalloc'd, vmalloc'd and vmap'd memory and for global and stack variables. What I've written gets the physical addresses but doesn't/can't pin it (which probably means I can't just move my code into iov_iter_get_pages2()).

Further, my code also treats multipage folios as single units which iov_iter_get_pages2() also doesn't - at least from XARRAY-class iterators.

The UBUF-/IOVEC-extraction code doesn't handle multipage folios because get_user_pages_fast() doesn't - though perhaps it will need to at some point.

David

#5 lore raw
Quoted 5 lines · David Howells

Note that iov_iter_get_pages2() doesn't handle KVEC-class iterators, which this code does - for kmalloc'd, vmalloc'd and vmap'd memory and for global and stack variables. What I've written gets the physical addresses but doesn't/can't pin it (which probably means I can't just move my code into iov_iter_get_pages2()).

Please look at the long discussion on Ira's patches for iov_iter_pin_pages where we went through this over and over again. We need to handle all of these in the common helper.

Quoted 5 lines

Further, my code also treats multipage folios as single units which iov_iter_get_pages2() also doesn't - at least from XARRAY-class iterators.

The UBUF-/IOVEC-extraction code doesn't handle multipage folios because get_user_pages_fast() doesn't - though perhaps it will need to at some point.

Yes, we need to be smarter about folios. But that is nothing magic about network file systems. I have old patches that added a (barely working) variant of get_user_pages_fast that exported bio_vecs and I think willy planned to look into a folio based version of get/pin user pages. But one thing at a time. I think the work from Ira to switch to proper pinning and sort out the weirdnesses for non-users buffer types is the first priority. After that proper folio or large contiguous range support is another item. But none of that is helped by parallel infrastructure.