Skip to contentSkip to conversation

bio bounce buffering fixes

PATCH In review Christoph Hellwig 2 participants 9 messages Jul 14 – Jul 15

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hi all,

this series fixes a few issues with the bio bounce buffering code.

The patches are on top of the "block: validate direct I/O memory alignment" series from Keith.

A git tree with that already applied is available here for convenience:

git://git.infradead.org/users/hch/misc.git block-bounce-fix

Gitweb:

https://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/block-bounce-fix

Diffstat:

 block/bio.c         |   53 +++++++++++-----------------------------------------
 include/linux/uio.h |    1 
 lib/iov_iter.c      |   37 ++++++++++++++++++++++++++++++++++--
 3 files changed, 48 insertions(+), 43 deletions(-)
#1 lore raw

PATCH 1/3 block: handle huge zero folios in bio_free_folios

When CONFIG_PERSISTENT_HUGE_ZERO_FOLIO is enabled, iomap_dio_zero() can add a huge zero folio to a zeroing bio, which needs special treatment in bio_free_folios by also checking is_huge_zero_folio() in addition to is_zero_folio().

Fixes 8dd5e7c75d7b ("block: add helpers to bounce buffer an iov_iter into bios")
---
 block/bio.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/bio.c +1 -1
index faad41a72ac7..23f739d5321d 100644
@@ -1352,7 +1352,7 @@ static void bio_free_folios(struct bio *bio)
 	bio_for_each_bvec_all(bv, bio, i) {
 		struct folio *folio = bvec_folio(bv);
 
-		if (!is_zero_folio(folio))
+		if (!is_zero_folio(folio) && !is_huge_zero_folio(folio))
 			folio_put(folio);
 	}
 }
Signature · 1 line
2.53.0
#2 lore raw

PATCH 2/3 block: simplify minsize alignment in bio_iov_iter_bounce_write

In reply to Christoph Hellwig

Don't allow partial copies to create unaligned length, which the file systems can't handle, and remove thee need for the separate bio_iov_iter_align_down() pass.

---
 block/bio.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
block/bio.c +6 -1
index 23f739d5321d..7f707ecf328e 100644
@@ -1399,7 +1399,12 @@ static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
 			 * bytes as @this_len is queued but only advanced
 			 * less than that.
 			 * Need to compensate that for the revert.
+			 *
+			 * Round down the amount copied to the minimum size /
+			 * aligned required by the file system so that we don't
+			 * create unaligned bios.
 			 */
+			copied = rounddown(copied, minsize);
 			iov_iter_revert(iter, bio->bi_iter.bi_size - this_len +
 					copied);
 			bio_free_folios(bio);
@@ -1410,7 +1415,7 @@ static int bio_iov_iter_bounce_write(struct bio *bio, struct iov_iter *iter,
 
 	if (!bio->bi_iter.bi_size)
 		return -ENOMEM;
-	return bio_iov_iter_align_down(bio, iter, minsize - 1);
+	return 0;
 }
 
 static int bio_iov_iter_bounce_read(struct bio *bio, struct iov_iter *iter,
Signature · 1 line
2.53.0
#3 lore raw

PATCH 3/3 block,iov_iter: move bio_iov_iter_align_down into iov_iter_extract_bvecs

In reply to Christoph Hellwig

Add a len_align_mask argument to iov_iter_extract_bvecs() and align the
extracted length in this core routine. This ensures that the returned
bvec array is directly usable by the caller, and thus fixes two bugs in bio_iov_iter_bounce_read() where the bounce bvec size is not updated to the adjusted value, and the bounce folio is not freed when we can't form a valid bio at all.

Fixes e7b8b3c5b2a6 ("block: align down bounces bios")
---
 block/bio.c         | 44 +++++---------------------------------------
 include/linux/uio.h |  1 +
 lib/iov_iter.c      | 37 +++++++++++++++++++++++++++++++++++--
 3 files changed, 41 insertions(+), 41 deletions(-)
block/bio.c +5 -39
index 7f707ecf328e..659678cd0843 100644
@@ -1185,41 +1185,6 @@ void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter)
 	bio_set_flag(bio, BIO_CLONED);
 }
 
-/*
- * Aligns the bio size to the len_align_mask, releasing excessive bio vecs that
- * __bio_iov_iter_get_pages may have inserted, and reverts the trimmed length
- * for the next iteration.
- */
-static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter,
-			    unsigned len_align_mask)
-{
-	size_t nbytes = bio->bi_iter.bi_size & len_align_mask;
-
-	if (!nbytes)
-		return 0;
-
-	iov_iter_revert(iter, nbytes);
-	bio->bi_iter.bi_size -= nbytes;
-	do {
-		struct bio_vec *bv = &bio->bi_io_vec[bio->bi_vcnt - 1];
-
-		if (nbytes < bv->bv_len) {
-			bv->bv_len -= nbytes;
-			break;
-		}
-
-		if (bio_flagged(bio, BIO_PAGE_PINNED))
-			unpin_user_page(bv->bv_page);
-
-		bio->bi_vcnt--;
-		nbytes -= bv->bv_len;
-	} while (nbytes);
-
-	if (!bio->bi_vcnt)
-		return -EFAULT;
-	return 0;
-}
-
 #ifdef CONFIG_DEBUG_KERNEL
 static inline bool bio_iov_bvec_aligned(const struct bio *bio,
 					unsigned mem_align_mask)
@@ -1305,7 +1270,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
 		ret = iov_iter_extract_bvecs(iter, bio->bi_io_vec,
 				BIO_MAX_SIZE - bio->bi_iter.bi_size,
 				&bio->bi_vcnt, bio->bi_max_vecs,
-				mem_align_mask, flags);
+				mem_align_mask, len_align_mask, flags);
 		if (ret <= 0) {
 			/*
 			 * A misaligned vector fails the whole I/O.  Release any
@@ -1326,7 +1291,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
 
 	if (is_pci_p2pdma_page(bio->bi_io_vec->bv_page))
 		bio->bi_opf |= REQ_NOMERGE;
-	return bio_iov_iter_align_down(bio, iter, len_align_mask);
+	return 0;
 }
 
 static struct folio *folio_alloc_greedy(gfp_t gfp, size_t *size,
@@ -1432,7 +1397,8 @@ static int bio_iov_iter_bounce_read(struct bio *bio, struct iov_iter *iter,
 		ssize_t ret;
 
 		ret = iov_iter_extract_bvecs(iter, bio->bi_io_vec + 1, len,
-				&bio->bi_vcnt, bio->bi_max_vecs - 1, 0, 0);
+				&bio->bi_vcnt, bio->bi_max_vecs - 1, 0,
+				minsize - 1, 0);
 		if (ret <= 0) {
 			if (!bio->bi_vcnt) {
 				folio_put(folio);
@@ -1453,7 +1419,7 @@ static int bio_iov_iter_bounce_read(struct bio *bio, struct iov_iter *iter,
 	bvec_set_folio(&bio->bi_io_vec[0], folio, bio->bi_iter.bi_size, 0);
 	if (iov_iter_extract_will_pin(iter))
 		bio_set_flag(bio, BIO_PAGE_PINNED);
-	return bio_iov_iter_align_down(bio, iter, minsize - 1);
+	return 0;
 }
 
 /**
include/linux/uio.h +1 -0
index fe2e985d74d2..fb1a161fa765 100644
@@ -399,6 +399,7 @@ ssize_t iov_iter_extract_pages(struct iov_iter *i, struct page ***pages,
 ssize_t iov_iter_extract_bvecs(struct iov_iter *iter, struct bio_vec *bv,
 		size_t max_size, unsigned short *nr_vecs,
 		unsigned short max_vecs, unsigned mem_align_mask,
+		unsigned len_align_mask,
 		iov_iter_extraction_t extraction_flags);
 
 /**
lib/iov_iter.c +35 -2
index 34a52e9ba9e1..bb6084672971 100644
@@ -1897,6 +1897,36 @@ static unsigned int get_contig_folio_len(struct page **pages,
 
 #define PAGE_PTRS_PER_BVEC     (sizeof(struct bio_vec) / sizeof(struct page *))
 
+static ssize_t bvec_iter_align_down(struct iov_iter *iter, struct bio_vec *bvec,
+		size_t size, unsigned short *nr_vecs, unsigned len_align_mask)
+{
+	size_t nbytes = size & len_align_mask;
+
+	if (!nbytes)
+		return size;
+
+	iov_iter_revert(iter, nbytes);
+	size -= nbytes;
+	do {
+		struct bio_vec *bv = &bvec[*nr_vecs - 1];
+
+		if (nbytes < bv->bv_len) {
+			bv->bv_len -= nbytes;
+			break;
+		}
+
+		if (iov_iter_extract_will_pin(iter))
+			unpin_user_page(bv->bv_page);
+
+		(*nr_vecs)--;
+		nbytes -= bv->bv_len;
+	} while (nbytes);
+
+	if (*nr_vecs == 0)
+		return -EFAULT;
+	return size;
+}
+
 /**
  * iov_iter_extract_bvecs - Extract bvecs from an iterator
  * @iter:	the iterator to extract from
@@ -1906,6 +1936,7 @@ static unsigned int get_contig_folio_len(struct page **pages,
  * @max_vecs:	maximum vectors in @bv, including those filled before calling
  * @mem_align_mask:	reject with -EINVAL if the source address or
  *		length is not aligned to this mask
+ * @len_align_mask: the mask to align the total size to, 0 for any length
  * @extraction_flags: flags to qualify request
  *
  * Like iov_iter_extract_pages(), but returns physically contiguous ranges
@@ -1918,7 +1949,7 @@ static unsigned int get_contig_folio_len(struct page **pages,
 ssize_t iov_iter_extract_bvecs(struct iov_iter *iter, struct bio_vec *bv,
 		size_t max_size, unsigned short *nr_vecs,
 		unsigned short max_vecs, unsigned mem_align_mask,
-		iov_iter_extraction_t extraction_flags)
+		unsigned len_align_mask, iov_iter_extraction_t extraction_flags)
 {
 	unsigned long start = (unsigned long)iter_iov_addr(iter);
 	unsigned short entries_left = max_vecs - *nr_vecs;
@@ -1960,11 +1991,13 @@ ssize_t iov_iter_extract_bvecs(struct iov_iter *iter, struct bio_vec *bv,
 		offset = 0;
 	}
 
+	size -= left;
 	iov_iter_revert(iter, left);
 	if (iov_iter_extract_will_pin(iter)) {
 		while (i < nr_pages)
 			unpin_user_page(pages[i++]);
 	}
-	return size - left;
+
+	return bvec_iter_align_down(iter, bv, size, nr_vecs, len_align_mask);
 }
 EXPORT_SYMBOL_GPL(iov_iter_extract_bvecs);
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH 3/3] block,iov_iter: move bio_iov_iter_align_down into iov_iter_extract_bvecs

Quoted 11 lines · Christoph Hellwig

ssize_t iov_iter_extract_bvecs(struct iov_iter *iter, struct bio_vec *bv,

lib/iov_iter.c +1 -1
 		size_t max_size, unsigned short *nr_vecs,
 		unsigned short max_vecs, unsigned mem_align_mask,
-		iov_iter_extraction_t extraction_flags)
+		unsigned len_align_mask, iov_iter_extraction_t extraction_flags)
 {
 	unsigned long start = (unsigned long)iter_iov_addr(iter);
 	unsigned short entries_left = max_vecs - *nr_vecs;
@@ -1960,11 +1991,13 @@ ssize_t iov_iter_extract_bvecs(struct iov_iter *iter, struct bio_vec *bv,
 		offset = 0;
 	}

lib/iov_iter.c ssize_t iov_iter_extract_bvecs(struct iov_iter *iter, struct bio_vec *bv, · line 2001

Quoted 10 lines
lib/iov_iter.c +3 -1
 
+	size -= left;
 	iov_iter_revert(iter, left);
 	if (iov_iter_extract_will_pin(iter)) {
 		while (i < nr_pages)
 			unpin_user_page(pages[i++]);
 	}
-	return size - left;
+
+	return bvec_iter_align_down(iter, bv, size, nr_vecs, len_align_mask);

I don't think we can do this alignment inline with building the bio_vec. The len_align_mask is usually the logical block size. We can submit sub-sector sized vectors so aligning down after extracting such a vector will result in a zero size for that iteration.

#5 lore raw

Re: [PATCH 3/3] block,iov_iter: move bio_iov_iter_align_down into iov_iter_extract_bvecs

Quoted 8 lines · Keith Busch

lib/iov_iter.c ssize_t iov_iter_extract_bvecs(struct iov_iter *iter, struct bio_vec *bv, · line 2001

Quoted 3 lines
lib/iov_iter.c +2 -1
-	return size - left;
+
+	return bvec_iter_align_down(iter, bv, size, nr_vecs, len_align_mask);

I don't think we can do this alignment inline with building the bio_vec. The len_align_mask is usually the logical block size. We can submit sub-sector sized vectors so aligning down after extracting such a vector will result in a zero size for that iteration.

Yeah, we need to support that. Kinda interesting that xfstests did not hit it, though. So back to the previous version.

#6 lore raw

Re: [PATCH 3/3] block,iov_iter: move bio_iov_iter_align_down into iov_iter_extract_bvecs

Quoted 2 lines · Christoph Hellwig

Yeah, we need to support that. Kinda interesting that xfstests did not hit it, though. So back to the previous version.

Here's a simple test for this case if you want to try. This should work for any file or raw disk with nvme backing storage.

It prints "Success" before this series, and "Bad Address" after.

virt-discontig-prp-test.c:
---
#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif

#include <err.h>
#include <errno.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/uio.h>

int main(int argc, char **argv)

{
	struct iovec iov[3];
	void *buf;
	int fd;

	if (argc < 2)
		errx(EINVAL, "usage: %s <file>", argv[0]);

	fd = open(argv[1], O_RDWR | O_TRUNC | O_DIRECT);
	if (fd < 0)
		err(EXIT_FAILURE, "failed to open %s", argv[1]);

	if (posix_memalign((void **)&buf, 4096, 8 * 4096))
		err(EXIT_FAILURE, "failed to allocate buffer");

	iov[0].iov_base = buf + (4096 - 256);
	iov[0].iov_len = 256;

	iov[1].iov_base = buf + 8192;
	iov[1].iov_len = 8192;

	iov[2].iov_base = buf + 20480;
	iov[2].iov_len = 4096 - 256;

	preadv(fd, iov, 3, 0);
	perror("preadv");
	return errno;
}

--

#7 lore raw

Re: [PATCH 3/3] block,iov_iter: move bio_iov_iter_align_down into iov_iter_extract_bvecs

Quoted 8 lines · Keith Busch
Quoted 2 lines · Christoph Hellwig

Yeah, we need to support that. Kinda interesting that xfstests did not hit it, though. So back to the previous version.

Here's a simple test for this case if you want to try. This should work for any file or raw disk with nvme backing storage.

It prints "Success" before this series, and "Bad Address" after.

Yes, assuming the NVMe doesn't support SGLs and has a 512 byte block size.. I guess this is going into blktessts/xfstests once the query API lands?

#8 lore raw

Re: [PATCH 3/3] block,iov_iter: move bio_iov_iter_align_down into iov_iter_extract_bvecs

Quoted 11 lines · Christoph Hellwig
Quoted 8 lines · Keith Busch
Quoted 2 lines · Christoph Hellwig

Yeah, we need to support that. Kinda interesting that xfstests did not hit it, though. So back to the previous version.

Here's a simple test for this case if you want to try. This should work for any file or raw disk with nvme backing storage.

It prints "Success" before this series, and "Bad Address" after.

Yes, assuming the NVMe doesn't support SGLs and has a 512 byte

The same test should be applicable to 4k LBS and SGL capable too!

Quoted 2 lines

block size.. I guess this is going into blktessts/xfstests once the query API lands?

It's a simplfied version of this one:

https://github.com/linux-blktests/blktests/blob/master/src/dio-offsets.c#L269

Though that is incorrectly expecting failure since older kernels didn't support sub-sector io vectors. If only we exported the necessary attributes to tell user space what's allowed... :)