Skip to contentSkip to conversation

direct-io dma alignment

PATCH Applied Keith Busch 11 participants 42 messages Jun 10, 2022 – Jul 25, 2022

Subsystems (8)
VFS Block layer Library code s390 +4 more subsystems
lore
Share & tools
lore
#0 lore raw

v6 cover direct-io dma alignment

From: Keith Busch <[email protected]>

The previous version is available here:

https://lore.kernel.org/linux-block/[email protected]/T/#m0a93b6392038aad6144e066fb5ada2cbf316f78e

Changes from the previous are all trivial changes:

Fixed a typo, s/bvev_/bdev_/

Updated commit messages and reviews

Updated code comments

Added a new comment for request_queue dma_alignement expressly documenting the consequences of setting this. All existing users of this attribute were checked to ensure they are on the correct side of those consequences.

Keith Busch (11):
block: fix infinite loop for invalid zone append block/bio: remove duplicate append pages code
block: export dma_alignment attribute
block: introduce bdev_dma_alignment helper
block: add a helper function for dio alignment block/merge: count bytes instead of sectors block/bounce: count bytes instead of sectors
iov: introduce iov_iter_aligned
block: introduce bdev_iter_is_aligned helper
block: relax direct io memory alignment
iomap: add support for dma aligned direct-io

 Documentation/ABI/stable/sysfs-block |   9 +++
 block/bio.c                          | 114 ++++++++++++---------------
 block/blk-merge.c                    |  41 ++++++----
 block/blk-sysfs.c                    |   7 ++
 block/bounce.c                       |  13 ++-
 block/fops.c                         |  16 ++--
 fs/iomap/direct-io.c                 |   4 +-
 include/linux/blkdev.h               |  17 ++++
 include/linux/uio.h                  |   2 +
 lib/iov_iter.c                       |  92 +++++++++++++++++++++
 10 files changed, 224 insertions(+), 91 deletions(-)
Signature · 1 line
2.30.2
#1 lore raw

PATCH v6 1/11 block: fix infinite loop for invalid zone append

From: Keith Busch <[email protected]>

Returning 0 early from __bio_iov_append_get_pages() for the max_append_sectors warning just creates an infinite loop since 0 means success, and the bio will never fill from the unadvancing iov_iter. We could turn the return into an error value, but it will already be turned into an error value later on, so just remove the warning. Clearly no one ever hit it anyway.

Fixes 0512a75b98f84 ("block: Introduce REQ_OP_ZONE_APPEND")
---
 block/bio.c | 3 ---
 1 file changed, 3 deletions(-)
block/bio.c +0 -3
index f92d0223247b..d481d5e4fe47 100644
@@ -1229,9 +1229,6 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)
 	size_t offset;
 	int ret = 0;
 
-	if (WARN_ON_ONCE(!max_append_sectors))
-		return 0;
-
 	/*
 	 * Move page array up in the allocated memory for the bio vecs as far as
 	 * possible so that we can start filling biovecs from the beginning
Signature · 1 line
2.30.2
#2 lore raw

PATCH v6 2/11 block/bio: remove duplicate append pages code

In reply to Keith Busch

From: Keith Busch <[email protected]>

The getting pages setup for zone append and normal IO are identical. Use common code for each.

---
 block/bio.c | 102 ++++++++++++++++++++++------------------------------
 1 file changed, 42 insertions(+), 60 deletions(-)
block/bio.c +42 -60
index d481d5e4fe47..5618c6a4b3a3 100644
@@ -1159,6 +1159,37 @@ static void bio_put_pages(struct page **pages, size_t size, size_t off)
 		put_page(pages[i]);
 }
 
+static int bio_iov_add_page(struct bio *bio, struct page *page,
+		unsigned int len, unsigned int offset)
+{
+	bool same_page = false;
+
+	if (!__bio_try_merge_page(bio, page, len, offset, &same_page)) {
+		if (WARN_ON_ONCE(bio_full(bio, len)))
+			return -EINVAL;
+		__bio_add_page(bio, page, len, offset);
+		return 0;
+	}
+
+	if (same_page)
+		put_page(page);
+	return 0;
+}
+
+static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page,
+		unsigned int len, unsigned int offset)
+{
+	struct request_queue *q = bdev_get_queue(bio->bi_bdev);
+	bool same_page = false;
+
+	if (bio_add_hw_page(q, bio, page, len, offset,
+			queue_max_zone_append_sectors(q), &same_page) != len)
+		return -EINVAL;
+	if (same_page)
+		put_page(page);
+	return 0;
+}
+
 #define PAGE_PTRS_PER_BVEC     (sizeof(struct bio_vec) / sizeof(struct page *))
 
 /**
@@ -1177,7 +1208,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)
 	unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt;
 	struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt;
 	struct page **pages = (struct page **)bv;
-	bool same_page = false;
 	ssize_t size, left;
 	unsigned len, i;
 	size_t offset;
@@ -1186,7 +1216,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)
 	 * Move page array up in the allocated memory for the bio vecs as far as
 	 * possible so that we can start filling biovecs from the beginning
 	 * without overwriting the temporary page array.
-	*/
+	 */
 	BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2);
 	pages += entries_left * (PAGE_PTRS_PER_BVEC - 1);
 
@@ -1196,18 +1226,18 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)
 
 	for (left = size, i = 0; left > 0; left -= len, i++) {
 		struct page *page = pages[i];
+		int ret;
 
 		len = min_t(size_t, PAGE_SIZE - offset, left);
+		if (bio_op(bio) == REQ_OP_ZONE_APPEND)
+			ret = bio_iov_add_zone_append_page(bio, page, len,
+					offset);
+		else
+			ret = bio_iov_add_page(bio, page, len, offset);
 
-		if (__bio_try_merge_page(bio, page, len, offset, &same_page)) {
-			if (same_page)
-				put_page(page);
-		} else {
-			if (WARN_ON_ONCE(bio_full(bio, len))) {
-				bio_put_pages(pages + i, left, offset);
-				return -EINVAL;
-			}
-			__bio_add_page(bio, page, len, offset);
+		if (ret) {
+			bio_put_pages(pages + i, left, offset);
+			return ret;
 		}
 		offset = 0;
 	}
@@ -1216,51 +1246,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)
 	return 0;
 }
 
-static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter)
-{
-	unsigned short nr_pages = bio->bi_max_vecs - bio->bi_vcnt;
-	unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt;
-	struct request_queue *q = bdev_get_queue(bio->bi_bdev);
-	unsigned int max_append_sectors = queue_max_zone_append_sectors(q);
-	struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt;
-	struct page **pages = (struct page **)bv;
-	ssize_t size, left;
-	unsigned len, i;
-	size_t offset;
-	int ret = 0;
-
-	/*
-	 * Move page array up in the allocated memory for the bio vecs as far as
-	 * possible so that we can start filling biovecs from the beginning
-	 * without overwriting the temporary page array.
-	 */
-	BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2);
-	pages += entries_left * (PAGE_PTRS_PER_BVEC - 1);
-
-	size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset);
-	if (unlikely(size <= 0))
-		return size ? size : -EFAULT;
-
-	for (left = size, i = 0; left > 0; left -= len, i++) {
-		struct page *page = pages[i];
-		bool same_page = false;
-
-		len = min_t(size_t, PAGE_SIZE - offset, left);
-		if (bio_add_hw_page(q, bio, page, len, offset,
-				max_append_sectors, &same_page) != len) {
-			bio_put_pages(pages + i, left, offset);
-			ret = -EINVAL;
-			break;
-		}
-		if (same_page)
-			put_page(page);
-		offset = 0;
-	}
-
-	iov_iter_advance(iter, size - left);
-	return ret;
-}
-
 /**
  * bio_iov_iter_get_pages - add user or kernel pages to a bio
  * @bio: bio to add pages to
@@ -1295,10 +1280,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)
 	}
 
 	do {
-		if (bio_op(bio) == REQ_OP_ZONE_APPEND)
-			ret = __bio_iov_append_get_pages(bio, iter);
-		else
-			ret = __bio_iov_iter_get_pages(bio, iter);
+		ret = __bio_iov_iter_get_pages(bio, iter);
 	} while (!ret && iov_iter_count(iter) && !bio_full(bio, 0));
 
 	/* don't account direct I/O as memory stall */
Signature · 1 line
2.30.2
#3 lore raw

PATCH v6 3/11 block: export dma_alignment attribute

In reply to Keith Busch

From: Keith Busch <[email protected]>

User space may want to know how to align their buffers to avoid bouncing. Export the queue attribute.

---
 Documentation/ABI/stable/sysfs-block | 9 +++++++++
 block/blk-sysfs.c                    | 7 +++++++
 2 files changed, 16 insertions(+)
Documentation/ABI/stable/sysfs-block +9 -0
index e8797cd09aff..cd14ecb3c9a5 100644
@@ -260,6 +260,15 @@ Description:
 		for discards, and don't read this file.
 
 
+What:		/sys/block/<disk>/queue/dma_alignment
+Date:		May 2022
+Contact:	[email protected]
+Description:
+		Reports the alignment that user space addresses must have to be
+		used for raw block device access with O_DIRECT and other driver
+		specific passthrough mechanisms.
+
+
 What:		/sys/block/<disk>/queue/fua
 Date:		May 2018
 Contact:	[email protected]
block/blk-sysfs.c +7 -0
index 88bd41d4cb59..14607565d781 100644
@@ -274,6 +274,11 @@ static ssize_t queue_virt_boundary_mask_show(struct request_queue *q, char *page
 	return queue_var_show(q->limits.virt_boundary_mask, page);
 }
 
+static ssize_t queue_dma_alignment_show(struct request_queue *q, char *page)
+{
+	return queue_var_show(queue_dma_alignment(q), page);
+}
+
 #define QUEUE_SYSFS_BIT_FNS(name, flag, neg)				\
 static ssize_t								\
 queue_##name##_show(struct request_queue *q, char *page)		\
@@ -606,6 +611,7 @@ QUEUE_RO_ENTRY(queue_dax, "dax");
 QUEUE_RW_ENTRY(queue_io_timeout, "io_timeout");
 QUEUE_RW_ENTRY(queue_wb_lat, "wbt_lat_usec");
 QUEUE_RO_ENTRY(queue_virt_boundary_mask, "virt_boundary_mask");
+QUEUE_RO_ENTRY(queue_dma_alignment, "dma_alignment");
 
 #ifdef CONFIG_BLK_DEV_THROTTLING_LOW
 QUEUE_RW_ENTRY(blk_throtl_sample_time, "throttle_sample_time");
@@ -667,6 +673,7 @@ static struct attribute *queue_attrs[] = {
 	&blk_throtl_sample_time_entry.attr,
 #endif
 	&queue_virt_boundary_mask_entry.attr,
+	&queue_dma_alignment_entry.attr,
 	NULL,
 };
 
Signature · 1 line
2.30.2
#4 lore raw

PATCH v6 4/11 block: introduce bdev_dma_alignment helper

In reply to Keith Busch

From: Keith Busch <[email protected]>

Preparing for upcoming dma_alignment users that have a block_device, but don't need the request_queue.

---
 include/linux/blkdev.h | 5 +++++
 1 file changed, 5 insertions(+)
include/linux/blkdev.h +5 -0
index 608d577734c2..ab7e6aa17954 100644
@@ -1366,6 +1366,11 @@ static inline int queue_dma_alignment(const struct request_queue *q)
 	return q ? q->dma_alignment : 511;
 }
 
+static inline unsigned int bdev_dma_alignment(struct block_device *bdev)
+{
+	return queue_dma_alignment(bdev_get_queue(bdev));
+}
+
 static inline int blk_rq_aligned(struct request_queue *q, unsigned long addr,
 				 unsigned int len)
 {
Signature · 1 line
2.30.2
#5 lore raw

PATCH v6 5/11 block: add a helper function for dio alignment

In reply to Keith Busch

From: Keith Busch <[email protected]>

This will make it easier to add more complex acceptable alignment criteria in the future.

---
 block/fops.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
block/fops.c +10 -6
index d6b3276a6c68..9d32df6fc315 100644
@@ -42,6 +42,13 @@ static unsigned int dio_bio_write_op(struct kiocb *iocb)
 	return op;
 }
 
+static bool blkdev_dio_unaligned(struct block_device *bdev, loff_t pos,
+			      struct iov_iter *iter)
+{
+	return ((pos | iov_iter_alignment(iter)) &
+	    (bdev_logical_block_size(bdev) - 1));
+}
+
 #define DIO_INLINE_BIO_VECS 4
 
 static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,
@@ -54,8 +61,7 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb,
 	struct bio bio;
 	ssize_t ret;
 
-	if ((pos | iov_iter_alignment(iter)) &
-	    (bdev_logical_block_size(bdev) - 1))
+	if (blkdev_dio_unaligned(bdev, pos, iter))
 		return -EINVAL;
 
 	if (nr_pages <= DIO_INLINE_BIO_VECS)
@@ -173,8 +179,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
 	loff_t pos = iocb->ki_pos;
 	int ret = 0;
 
-	if ((pos | iov_iter_alignment(iter)) &
-	    (bdev_logical_block_size(bdev) - 1))
+	if (blkdev_dio_unaligned(bdev, pos, iter))
 		return -EINVAL;
 
 	if (iocb->ki_flags & IOCB_ALLOC_CACHE)
@@ -298,8 +303,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
 	loff_t pos = iocb->ki_pos;
 	int ret = 0;
 
-	if ((pos | iov_iter_alignment(iter)) &
-	    (bdev_logical_block_size(bdev) - 1))
+	if (blkdev_dio_unaligned(bdev, pos, iter))
 		return -EINVAL;
 
 	if (iocb->ki_flags & IOCB_ALLOC_CACHE)
Signature · 1 line
2.30.2
#6 lore raw

Re: [PATCHv6 05/11] block: add a helper function for dio alignment

block/fops.c static unsigned int dio_bio_write_op(struct kiocb *iocb) · line 50

Quoted 6 lines · Keith Busch
block/fops.c +6 -0
+static bool blkdev_dio_unaligned(struct block_device *bdev, loff_t pos,
+			      struct iov_iter *iter)
+{
+	return ((pos | iov_iter_alignment(iter)) &
+	    (bdev_logical_block_size(bdev) - 1));
+}

A nit: if you have to repost this patch series, please leave out the outer parentheses from the return statement. Otherwise this patch looks good to me.

Thanks,

Bart.

#7 lore raw

PATCH v6 6/11 block/merge: count bytes instead of sectors

In reply to Keith Busch

From: Keith Busch <[email protected]>

Individual bv_len's may not be a sector size.

---
 block/blk-merge.c | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
block/blk-merge.c +24 -17
index 7771dacc99cb..3874619ba136 100644
@@ -201,11 +201,11 @@ static inline unsigned get_max_segment_size(const struct request_queue *q,
  * @nsegs:    [in,out] Number of segments in the bio being built. Incremented
  *            by the number of segments from @bv that may be appended to that
  *            bio without exceeding @max_segs
- * @sectors:  [in,out] Number of sectors in the bio being built. Incremented
- *            by the number of sectors from @bv that may be appended to that
- *            bio without exceeding @max_sectors
+ * @bytes:    [in,out] Number of bytes in the bio being built. Incremented
+ *            by the number of bytes from @bv that may be appended to that
+ *            bio without exceeding @max_bytes
  * @max_segs: [in] upper bound for *@nsegs
- * @max_sectors: [in] upper bound for *@sectors
+ * @max_bytes: [in] upper bound for *@bytes
  *
  * When splitting a bio, it can happen that a bvec is encountered that is too
  * big to fit in a single segment and hence that it has to be split in the
@@ -216,10 +216,10 @@ static inline unsigned get_max_segment_size(const struct request_queue *q,
  */
 static bool bvec_split_segs(const struct request_queue *q,
 			    const struct bio_vec *bv, unsigned *nsegs,
-			    unsigned *sectors, unsigned max_segs,
-			    unsigned max_sectors)
+			    unsigned *bytes, unsigned max_segs,
+			    unsigned max_bytes)
 {
-	unsigned max_len = (min(max_sectors, UINT_MAX >> 9) - *sectors) << 9;
+	unsigned max_len = min(max_bytes, UINT_MAX) - *bytes;
 	unsigned len = min(bv->bv_len, max_len);
 	unsigned total_len = 0;
 	unsigned seg_size = 0;
@@ -237,7 +237,7 @@ static bool bvec_split_segs(const struct request_queue *q,
 			break;
 	}
 
-	*sectors += total_len >> 9;
+	*bytes += total_len;
 
 	/* tell the caller to split the bvec if it is too big to fit */
 	return len > 0 || bv->bv_len > max_len;
@@ -269,8 +269,8 @@ static struct bio *blk_bio_segment_split(struct request_queue *q,
 {
 	struct bio_vec bv, bvprv, *bvprvp = NULL;
 	struct bvec_iter iter;
-	unsigned nsegs = 0, sectors = 0;
-	const unsigned max_sectors = get_max_io_size(q, bio);
+	unsigned nsegs = 0, bytes = 0;
+	const unsigned max_bytes = get_max_io_size(q, bio) << 9;
 	const unsigned max_segs = queue_max_segments(q);
 
 	bio_for_each_bvec(bv, bio, iter) {
@@ -282,12 +282,12 @@ static struct bio *blk_bio_segment_split(struct request_queue *q,
 			goto split;
 
 		if (nsegs < max_segs &&
-		    sectors + (bv.bv_len >> 9) <= max_sectors &&
+		    bytes + bv.bv_len <= max_bytes &&
 		    bv.bv_offset + bv.bv_len <= PAGE_SIZE) {
 			nsegs++;
-			sectors += bv.bv_len >> 9;
-		} else if (bvec_split_segs(q, &bv, &nsegs, &sectors, max_segs,
-					 max_sectors)) {
+			bytes += bv.bv_len;
+		} else if (bvec_split_segs(q, &bv, &nsegs, &bytes, max_segs,
+					   max_bytes)) {
 			goto split;
 		}
 
@@ -300,13 +300,20 @@ static struct bio *blk_bio_segment_split(struct request_queue *q,
 split:
 	*segs = nsegs;
 
+	/*
+	 * Individual bvecs might not be logical block aligned. Round down the
+	 * split size so that each bio is properly block size aligned, even if
+	 * we do not use the full hardware limits.
+	 */
+	bytes = ALIGN_DOWN(bytes, queue_logical_block_size(q));
+
 	/*
 	 * Bio splitting may cause subtle trouble such as hang when doing sync
 	 * iopoll in direct IO routine. Given performance gain of iopoll for
 	 * big IO can be trival, disable iopoll when split needed.
 	 */
 	bio_clear_polled(bio);
-	return bio_split(bio, sectors, GFP_NOIO, bs);
+	return bio_split(bio, bytes >> SECTOR_SHIFT, GFP_NOIO, bs);
 }
 
 /**
@@ -375,7 +382,7 @@ EXPORT_SYMBOL(blk_queue_split);
 unsigned int blk_recalc_rq_segments(struct request *rq)
 {
 	unsigned int nr_phys_segs = 0;
-	unsigned int nr_sectors = 0;
+	unsigned int bytes = 0;
 	struct req_iterator iter;
 	struct bio_vec bv;
 
@@ -398,7 +405,7 @@ unsigned int blk_recalc_rq_segments(struct request *rq)
 	}
 
 	rq_for_each_bvec(bv, rq, iter)
-		bvec_split_segs(rq->q, &bv, &nr_phys_segs, &nr_sectors,
+		bvec_split_segs(rq->q, &bv, &nr_phys_segs, &bytes,
 				UINT_MAX, UINT_MAX);
 	return nr_phys_segs;
 }
Signature · 1 line
2.30.2
#8 lore raw

Re: [PATCHv6 06/11] block/merge: count bytes instead of sectors

block/blk-merge.c static struct bio *blk_bio_segment_split(struct request_queue *q, · line 273

Quoted 8 lines · Keith Busch
block/blk-merge.c +2 -2
@@ -269,8 +269,8 @@ static struct bio *blk_bio_segment_split(struct request_queue *q,
  {
  	struct bio_vec bv, bvprv, *bvprvp = NULL;
  	struct bvec_iter iter;
-	unsigned nsegs = 0, sectors = 0;
-	const unsigned max_sectors = get_max_io_size(q, bio);
+	unsigned nsegs = 0, bytes = 0;
+	const unsigned max_bytes = get_max_io_size(q, bio) << 9;

How about using SECTOR_SHIFT instead of 9?

Thanks,

Bart.

#9 lore raw

PATCH v6 7/11 block/bounce: count bytes instead of sectors

In reply to Keith Busch

From: Keith Busch <[email protected]>

Individual bv_len's may not be a sector size.

---
 block/bounce.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
block/bounce.c +10 -3
index 8f7b6fe3b4db..fbadf179601f 100644
@@ -205,19 +205,26 @@ void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig)
 	int rw = bio_data_dir(*bio_orig);
 	struct bio_vec *to, from;
 	struct bvec_iter iter;
-	unsigned i = 0;
+	unsigned i = 0, bytes = 0;
 	bool bounce = false;
-	int sectors = 0;
+	int sectors;
 
 	bio_for_each_segment(from, *bio_orig, iter) {
 		if (i++ < BIO_MAX_VECS)
-			sectors += from.bv_len >> 9;
+			bytes += from.bv_len;
 		if (PageHighMem(from.bv_page))
 			bounce = true;
 	}
 	if (!bounce)
 		return;
 
+	/*
+	 * Individual bvecs might not be logical block aligned. Round down
+	 * the split size so that each bio is properly block size aligned,
+	 * even if we do not use the full hardware limits.
+	 */
+	sectors = ALIGN_DOWN(bytes, queue_logical_block_size(q)) >>
+			SECTOR_SHIFT;
 	if (sectors < bio_sectors(*bio_orig)) {
 		bio = bio_split(*bio_orig, sectors, GFP_NOIO, &bounce_bio_split);
 		bio_chain(bio, *bio_orig);
Signature · 1 line
2.30.2
#11 lore raw

Re: [PATCHv6 07/11] block/bounce: count bytes instead of sectors

In reply to Keith Busch
Quoted 35 lines · Keith Busch

From: Keith Busch <[email protected]>

Individual bv_len's may not be a sector size.

---
  block/bounce.c | 13 ++++++++++---
  1 file changed, 10 insertions(+), 3 deletions(-)
block/bounce.c +3 -3
index 8f7b6fe3b4db..fbadf179601f 100644
@@ -205,19 +205,26 @@ void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig)
  	int rw = bio_data_dir(*bio_orig);
  	struct bio_vec *to, from;
  	struct bvec_iter iter;
-	unsigned i = 0;
+	unsigned i = 0, bytes = 0;
  	bool bounce = false;
-	int sectors = 0;
+	int sectors;
 
  	bio_for_each_segment(from, *bio_orig, iter) {
  		if (i++ < BIO_MAX_VECS)
-			sectors += from.bv_len >> 9;
+			bytes += from.bv_len;
  		if (PageHighMem(from.bv_page))
  			bounce = true;
  	}
  	if (!bounce)
  		return;
 

block/bounce.c void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig) · line 230

Quoted 10 lines
block/bounce.c +7 -0
+	/*
+	 * Individual bvecs might not be logical block aligned. Round down
+	 * the split size so that each bio is properly block size aligned,
+	 * even if we do not use the full hardware limits.
+	 */
+	sectors = ALIGN_DOWN(bytes, queue_logical_block_size(q)) >>
+			SECTOR_SHIFT;
  	if (sectors < bio_sectors(*bio_orig)) {
  		bio = bio_split(*bio_orig, sectors, GFP_NOIO, &bounce_bio_split);
  		bio_chain(bio, *bio_orig);

Do I see correctly that there are two changes in this patch: counting bytes instead of sectors and also splitting at logical block boundaries instead of a 512-byte boundary? Should this patch perhaps be split?

Thanks,

Bart.

maintainer
#12 lore raw

Re: [PATCHv6 07/11] block/bounce: count bytes instead of sectors

Quoted 40 lines · Bart Van Assche
Quoted 39 lines · Keith Busch

From: Keith Busch <[email protected]>

Individual bv_len's may not be a sector size.

---
  block/bounce.c | 13 ++++++++++---
  1 file changed, 10 insertions(+), 3 deletions(-)
block/bounce.c +9 -3
index 8f7b6fe3b4db..fbadf179601f 100644
@@ -205,19 +205,26 @@ void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig)
  	int rw = bio_data_dir(*bio_orig);
  	struct bio_vec *to, from;
  	struct bvec_iter iter;
-	unsigned i = 0;
+	unsigned i = 0, bytes = 0;
  	bool bounce = false;
-	int sectors = 0;
+	int sectors;
  	bio_for_each_segment(from, *bio_orig, iter) {
  		if (i++ < BIO_MAX_VECS)
-			sectors += from.bv_len >> 9;
+			bytes += from.bv_len;
  		if (PageHighMem(from.bv_page))
  			bounce = true;
  	}
  	if (!bounce)
  		return;
+	/*
+	 * Individual bvecs might not be logical block aligned. Round down
+	 * the split size so that each bio is properly block size aligned,
+	 * even if we do not use the full hardware limits.
+	 */
+	sectors = ALIGN_DOWN(bytes, queue_logical_block_size(q)) >>
Quoted 8 lines

block/bounce.c void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig) · line 230

Quoted 4 lines
block/bounce.c +1 -0
+			SECTOR_SHIFT;
  	if (sectors < bio_sectors(*bio_orig)) {
  		bio = bio_split(*bio_orig, sectors, GFP_NOIO, &bounce_bio_split);
  		bio_chain(bio, *bio_orig);

Do I see correctly that there are two changes in this patch: counting bytes instead of sectors and also splitting at logical block boundaries instead of a 512-byte boundary? Should this patch perhaps be split?

The code previously would only split on a bvec boundary. All bvecs were logical block sized, so that part is not changing. We just needed to be able to split mid-bvec since the series enables unaligned offsets to match hardware capabilities.

#13 lore raw

PATCH v6 8/11 iov: introduce iov_iter_aligned

In reply to Keith Busch

From: Keith Busch <[email protected]>

The existing iov_iter_alignment() function returns the logical OR of address and length. For cases where address and length need to be considered separately, introduce a helper function that a caller can specificy length and address masks that indicate if the iov is unaligned.

---
 include/linux/uio.h |  2 +
 lib/iov_iter.c      | 92 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 94 insertions(+)
include/linux/uio.h +2 -0
index 739285fe5a2f..34ba4a731179 100644
@@ -219,6 +219,8 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i);
 #endif
 
 size_t iov_iter_zero(size_t bytes, struct iov_iter *);
+bool iov_iter_is_aligned(const struct iov_iter *i, unsigned addr_mask,
+			unsigned len_mask);
 unsigned long iov_iter_alignment(const struct iov_iter *i);
 unsigned long iov_iter_gap_alignment(const struct iov_iter *i);
 void iov_iter_init(struct iov_iter *i, unsigned int direction, const struct iovec *iov,
lib/iov_iter.c +92 -0

98 lines not on this page · raw message

Signature · 1 line
2.30.2
#14 lore raw

PATCH v6 9/11 block: introduce bdev_iter_is_aligned helper

In reply to Keith Busch

From: Keith Busch <[email protected]>

Provide a convenient function for this repeatable coding pattern.

---
 include/linux/blkdev.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/linux/blkdev.h +7 -0
index ab7e6aa17954..fb5c177708d5 100644
@@ -1371,6 +1371,13 @@ static inline unsigned int bdev_dma_alignment(struct block_device *bdev)
 	return queue_dma_alignment(bdev_get_queue(bdev));
 }
 
+static inline bool bdev_iter_is_aligned(struct block_device *bdev,
+					struct iov_iter *iter)
+{
+	return iov_iter_is_aligned(iter, bdev_dma_alignment(bdev),
+				   bdev_logical_block_size(bdev) - 1);
+}
+
 static inline int blk_rq_aligned(struct request_queue *q, unsigned long addr,
 				 unsigned int len)
 {
Signature · 1 line
2.30.2
#15 lore raw

PATCH v6 10/11 block: relax direct io memory alignment

In reply to Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size. User space can discover the alignment requirements from the dma_alignment queue attribute.

User space can specify any hardware compatible DMA offset for each segment, but every segment length is still required to be a multiple of the block size.

---
 block/bio.c            | 9 +++++++++
 block/fops.c           | 4 ++--
 include/linux/blkdev.h | 5 +++++
 3 files changed, 16 insertions(+), 2 deletions(-)
block/bio.c +9 -0
index 5618c6a4b3a3..551f1d12208b 100644
@@ -1220,7 +1220,16 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)
 	BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2);
 	pages += entries_left * (PAGE_PTRS_PER_BVEC - 1);
 
+	/*
+	 * Each segment in the iov is required to be a block size multiple.
+	 * However, we may not be able to get the entire segment if it spans
+	 * more pages than bi_max_vecs allows, so we have to ALIGN_DOWN the
+	 * result to ensure the bio's total size is correct. The remainder of
+	 * the iov data will be picked up in the next bio iteration.
+	 */
 	size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset);
+	if (size > 0)
+		size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev));
 	if (unlikely(size <= 0))
 		return size ? size : -EFAULT;
 
block/fops.c +2 -2
index 9d32df6fc315..86d3cab9bf93 100644
@@ -45,8 +45,8 @@ static unsigned int dio_bio_write_op(struct kiocb *iocb)
 static bool blkdev_dio_unaligned(struct block_device *bdev, loff_t pos,
 			      struct iov_iter *iter)
 {
-	return ((pos | iov_iter_alignment(iter)) &
-	    (bdev_logical_block_size(bdev) - 1));
+	return pos & (bdev_logical_block_size(bdev) - 1) ||
+		!bdev_iter_is_aligned(bdev, iter);
 }
 
 #define DIO_INLINE_BIO_VECS 4
include/linux/blkdev.h +5 -0
index fb5c177708d5..914c613d81da 100644
@@ -425,6 +425,11 @@ struct request_queue {
 	unsigned long		nr_requests;	/* Max # of requests */
 
 	unsigned int		dma_pad_mask;
+	/*
+	 * Drivers that set dma_alignment to less than 511 must be prepared to
+	 * handle individual bvec's that are not a multiple of a SECTOR_SIZE
+	 * due to possible offsets.
+	 */
 	unsigned int		dma_alignment;
 
 #ifdef CONFIG_BLK_INLINE_ENCRYPTION
Signature · 1 line
2.30.2
#16 lore raw

PATCH v6 11/11 iomap: add support for dma aligned direct-io

In reply to Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +2 -2
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
 		return -EINVAL;
 
 	if (iomap->type == IOMAP_UNWRITTEN) {
Signature · 1 line
2.30.2
#17 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 5 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct
io instead of requiring addresses be aligned to the block size.

Hi Keith,

Our s390 PV guests recently started failing to boot from a -next host, and git blame brought me here.

As near as I have been able to tell, we start tripping up on this code from patch 9 [1] that gets invoked with this patch:

Quoted 7 lines
	for (k = 0; k < i->nr_segs; k++, skip = 0) {
		size_t len = i->iov[k].iov_len - skip;

		if (len > size)
			len = size;
		if (len & len_mask)
			return false;

The iovec we're failing on has two segments, one with a len of x200 (and base of x...000) and another with a len of xe00 (and a base of x...200), while len_mask is of course xfff.

So before I go any further on what we might have broken, do you happen to have any suggestions what might be going on here, or something I should try?

Thanks,
Eric

[1] https://lore.kernel.org/r/[email protected]/

Quoted 32 lines
---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct
 iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits =
 blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
fs/iomap/direct-io.c +2 -1
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct
 iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
 		return -EINVAL;
 

if (iomap->type == IOMAP_UNWRITTEN) {

maintainer
#18 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 23 lines · Eric Farman
Quoted 5 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct
io instead of requiring addresses be aligned to the block size.

Hi Keith,

Our s390 PV guests recently started failing to boot from a -next host, and git blame brought me here.

As near as I have been able to tell, we start tripping up on this code from patch 9 [1] that gets invoked with this patch:

Quoted 7 lines
	for (k = 0; k < i->nr_segs; k++, skip = 0) {
		size_t len = i->iov[k].iov_len - skip;

		if (len > size)
			len = size;
		if (len & len_mask)
			return false;
Quoted 7 lines

The iovec we're failing on has two segments, one with a len of x200 (and base of x...000) and another with a len of xe00 (and a base of x...200), while len_mask is of course xfff.

So before I go any further on what we might have broken, do you happen to have any suggestions what might be going on here, or something I should try?

Thanks for the notice, sorry for the trouble. This check wasn't intended to have any difference from the previous code with respect to the vector lengths.

Could you tell me if you're accessing this through the block device direct-io, or through iomap filesystem?

maintainer
#19 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 32 lines · Keith Busch
Quoted 23 lines · Eric Farman
Quoted 5 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct
io instead of requiring addresses be aligned to the block size.

Hi Keith,

Our s390 PV guests recently started failing to boot from a -next host, and git blame brought me here.

As near as I have been able to tell, we start tripping up on this code from patch 9 [1] that gets invoked with this patch:

Quoted 7 lines
	for (k = 0; k < i->nr_segs; k++, skip = 0) {
		size_t len = i->iov[k].iov_len - skip;

		if (len > size)
			len = size;
		if (len & len_mask)
			return false;
Quoted 7 lines

The iovec we're failing on has two segments, one with a len of x200 (and base of x...000) and another with a len of xe00 (and a base of x...200), while len_mask is of course xfff.

So before I go any further on what we might have broken, do you happen to have any suggestions what might be going on here, or something I should try?

Quoted 5 lines

Thanks for the notice, sorry for the trouble. This check wasn't intended to have any difference from the previous code with respect to the vector lengths.

Could you tell me if you're accessing this through the block device direct-io, or through iomap filesystem?

If using iomap, the previous check was this:

	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
	unsigned int align = iov_iter_alignment(dio->submit.iter);
	...
	if ((pos | length | align) & ((1 << blkbits) - 1))
		return -EINVAL;

And if raw block device, it was this:

	if ((pos | iov_iter_alignment(iter)) &
	    (bdev_logical_block_size(bdev) - 1))
		return -EINVAL;

The result of "iov_iter_alignment()" would include "0xe00 | 0x200" in your example, and checked against 0xfff should have been failing prior to this patch. Unless I'm missing something...

#20 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 40 lines · Keith Busch
Quoted 39 lines · Keith Busch
Quoted 37 lines · Eric Farman
Quoted 6 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for
direct
io instead of requiring addresses be aligned to the block size.

Hi Keith,

Our s390 PV guests recently started failing to boot from a -next host,
and git blame brought me here.

As near as I have been able to tell, we start tripping up on this code
from patch 9 [1] that gets invoked with this patch:

Quoted 7 lines
	for (k = 0; k < i->nr_segs; k++, skip = 0) {
		size_t len = i->iov[k].iov_len - skip;

		if (len > size)
			len = size;
		if (len & len_mask)
			return false;

The iovec we're failing on has two segments, one with a len of x200
(and base of x...000) and another with a len of xe00 (and a base of
x...200), while len_mask is of course xfff.

So before I go any further on what we might have broken, do you happen
to have any suggestions what might be going on here, or something I
should try?

Quoted 8 lines
Quoted 8 lines

Thanks for the notice, sorry for the trouble. This check wasn't intended to
have any difference from the previous code with respect to the vector lengths.

Could you tell me if you're accessing this through the block device direct-io,
or through iomap filesystem?

Reasonably certain the failure's on iomap. I'd reverted the subject patch from next-20220622 and got things in working order.

Quoted 11 lines

If using iomap, the previous check was this:

unsigned int blkbits =

blksize_bits(bdev_logical_block_size(iomap->bdev));
	unsigned int align = iov_iter_alignment(dio->submit.iter);
	...
	if ((pos | length | align) & ((1 << blkbits) - 1))
		return -EINVAL;

...

Quoted 5 lines

The result of "iov_iter_alignment()" would include "0xe00 | 0x200" in your
example, and checked against 0xfff should have been failing prior to this
patch. Unless I'm missing something...

Nope, you're not. I didn't look back at what the old check was doing, just saw "0xe00 and 0x200" and thought "oh there's one page" instead of noting the code was or'ing them. My bad.

That was the last entry in my trace before the guest gave up, as everything else through this code up to that point seemed okay. I'll pick up the working case and see if I can get a clearer picture between the two.

#21 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 83 lines · Eric Farman
Quoted 50 lines · Keith Busch
Quoted 49 lines · Keith Busch
Quoted 36 lines · Eric Farman
Quoted 7 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for
direct
io instead of requiring addresses be aligned to the block size.

Hi Keith,

Our s390 PV guests recently started failing to boot from a -next
host,
and git blame brought me here.

As near as I have been able to tell, we start tripping up on this
code
from patch 9 [1] that gets invoked with this patch:

Quoted 7 lines
	for (k = 0; k < i->nr_segs; k++, skip = 0) {
		size_t len = i->iov[k].iov_len - skip;

		if (len > size)
			len = size;
		if (len & len_mask)
			return false;

The iovec we're failing on has two segments, one with a len of x200
(and base of x...000) and another with a len of xe00 (and a base
of
x...200), while len_mask is of course xfff.

Quoted 6 lines

So before I go any further on what we might have broken, do you happen
to have any suggestions what might be going on here, or something
I
should try?

Thanks for the notice, sorry for the trouble. This check wasn't intended to
have any difference from the previous code with respect to the vector lengths.

Quoted 4 lines
Quoted 4 lines

Could you tell me if you're accessing this through the block device
direct-io,
or through iomap filesystem?

Reasonably certain the failure's on iomap. I'd reverted the subject patch from next-20220622 and got things in working order.

Quoted 10 lines

If using iomap, the previous check was this:

unsigned int blkbits =

blksize_bits(bdev_logical_block_size(iomap->bdev));
	unsigned int align = iov_iter_alignment(dio->submit.iter);
	...
	if ((pos | length | align) & ((1 << blkbits) - 1))
		return -EINVAL;

...

Quoted 7 lines

The result of "iov_iter_alignment()" would include "0xe00 | 0x200" in
your
example, and checked against 0xfff should have been failing prior to
this
patch. Unless I'm missing something...

Nope, you're not. I didn't look back at what the old check was doing, just saw "0xe00 and 0x200" and thought "oh there's one page" instead of
noting the code was or'ing them. My bad.

Quoted 5 lines

That was the last entry in my trace before the guest gave up, as everything else through this code up to that point seemed okay. I'll pick up the working case and see if I can get a clearer picture between
the two.

Looking over the trace again, I realize I did dump iov_iter_alignment() as a comparator, and I see one pass through that had a non-zero response but bdev_iter_is_aligned() returned true...

count = x1000
iov_offset = x0
nr_segs = 1
iov_len = x1000 (len_mask = xfff)
iov_base = x...200 (addr_mask = x1ff)

That particular pass through is in the middle of the stuff it tried to do, so I don't know if that's the cause or not but it strikes me as unusual. Will look into that tomorrow and report back.

Quoted 1 line
#22 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 108 lines · Eric Farman
Quoted 89 lines · Eric Farman
Quoted 53 lines · Keith Busch
Quoted 52 lines · Keith Busch
Quoted 38 lines · Eric Farman
Quoted 8 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device
for
direct
io instead of requiring addresses be aligned to the block size.

Hi Keith,

Our s390 PV guests recently started failing to boot from a -next
host,
and git blame brought me here.

As near as I have been able to tell, we start tripping up on this
code
from patch 9 [1] that gets invoked with this patch:

Quoted 7 lines
	for (k = 0; k < i->nr_segs; k++, skip = 0) {
		size_t len = i->iov[k].iov_len - skip;

		if (len > size)
			len = size;
		if (len & len_mask)
			return false;

The iovec we're failing on has two segments, one with a len of
x200
(and base of x...000) and another with a len of xe00 (and a base
of
x...200), while len_mask is of course xfff.

Quoted 7 lines

So before I go any further on what we might have broken, do you
happen
to have any suggestions what might be going on here, or something
I
should try?

Thanks for the notice, sorry for the trouble. This check wasn't intended to
have any difference from the previous code with respect to the vector lengths.

Quoted 4 lines
Quoted 4 lines

Could you tell me if you're accessing this through the block device
direct-io,
or through iomap filesystem?

Reasonably certain the failure's on iomap. I'd reverted the subject patch from next-20220622 and got things in working order.

Quoted 10 lines

If using iomap, the previous check was this:

unsigned int blkbits =

blksize_bits(bdev_logical_block_size(iomap->bdev));
	unsigned int align = iov_iter_alignment(dio->submit.iter);
	...
	if ((pos | length | align) & ((1 << blkbits) - 1))
		return -EINVAL;

...

Quoted 8 lines

The result of "iov_iter_alignment()" would include "0xe00 | 0x200"
in
your
example, and checked against 0xfff should have been failing prior to
this
patch. Unless I'm missing something...

Nope, you're not. I didn't look back at what the old check was doing,
just saw "0xe00 and 0x200" and thought "oh there's one page" instead
of
noting the code was or'ing them. My bad.

Quoted 6 lines

That was the last entry in my trace before the guest gave up, as everything else through this code up to that point seemed okay. I'll
pick up the working case and see if I can get a clearer picture between
the two.

Looking over the trace again, I realize I did dump iov_iter_alignment()
as a comparator, and I see one pass through that had a non-zero response but bdev_iter_is_aligned() returned true...

count = x1000
iov_offset = x0
nr_segs = 1
iov_len = x1000 (len_mask = xfff)
iov_base = x...200 (addr_mask = x1ff)

Quoted 5 lines

That particular pass through is in the middle of the stuff it tried to
do, so I don't know if that's the cause or not but it strikes me as unusual. Will look into that tomorrow and report back.

Apologies, it took me an extra day to get back to this, but it is indeed this pass through that's causing our boot failures. I note that the old code (in iomap_dio_bio_iter), did:

        if ((pos | length | align) & ((1 << blkbits) - 1))
                return -EINVAL;

With blkbits equal to 12, the resulting mask was 0x0fff against an align value (from iov_iter_alignment) of x200 kicks us out.

The new code (in iov_iter_aligned_iovec), meanwhile, compares this:

                if ((unsigned long)(i->iov[k].iov_base + skip) &
addr_mask)
                        return false;

iov_base (and the output of the old iov_iter_aligned_iovec() routine) is x200, but since addr_mask is x1ff this check provides a different response than it used to.

To check this, I changed the comparator to len_mask (almost certainly not the right answer since addr_mask is then unused, but it was good for a quick test), and our PV guests are able to boot again with -next running in the host.

Thanks,
Eric

maintainer
#23 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 17 lines · Eric Farman

Apologies, it took me an extra day to get back to this, but it is indeed this pass through that's causing our boot failures. I note that the old code (in iomap_dio_bio_iter), did:

        if ((pos | length | align) & ((1 << blkbits) - 1))
                return -EINVAL;

With blkbits equal to 12, the resulting mask was 0x0fff against an align value (from iov_iter_alignment) of x200 kicks us out.

The new code (in iov_iter_aligned_iovec), meanwhile, compares this:

                if ((unsigned long)(i->iov[k].iov_base + skip) &
addr_mask)
                        return false;
Quoted 8 lines

iov_base (and the output of the old iov_iter_aligned_iovec() routine) is x200, but since addr_mask is x1ff this check provides a different response than it used to.

To check this, I changed the comparator to len_mask (almost certainly not the right answer since addr_mask is then unused, but it was good for a quick test), and our PV guests are able to boot again with -next running in the host.

This raises more questions for me. It sounds like your process used to get an EINVAL error, and it wants to continue getting an EINVAL error instead of letting the direct-io request proceed. Is that correct? If so, could you provide more details on what issue occurs with dispatching this request?

If you really need to restrict address' alignment to the storage's logical block size, I think your storage driver needs to set the dma_alignment queue limit to that value.

#24 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 22 lines · On Mon, 27 Jun 2022 09:36:56 -0600 Keith Busch <[email protected]> wrote:
Quoted 21 lines · Eric Farman

Apologies, it took me an extra day to get back to this, but it is indeed this pass through that's causing our boot failures. I note that the old code (in iomap_dio_bio_iter), did:

        if ((pos | length | align) & ((1 << blkbits) - 1))
                return -EINVAL;

With blkbits equal to 12, the resulting mask was 0x0fff against an align value (from iov_iter_alignment) of x200 kicks us out.

The new code (in iov_iter_aligned_iovec), meanwhile, compares this:

                if ((unsigned long)(i->iov[k].iov_base + skip) &
addr_mask)
                        return false;

iov_base (and the output of the old iov_iter_aligned_iovec() routine) is x200, but since addr_mask is x1ff this check provides a different response than it used to.

Quoted 8 lines
Quoted 4 lines

To check this, I changed the comparator to len_mask (almost certainly not the right answer since addr_mask is then unused, but it was good for a quick test), and our PV guests are able to boot again with -next running in the host.

This raises more questions for me. It sounds like your process used to get an EINVAL error, and it wants to continue getting an EINVAL error instead of letting the direct-io request proceed. Is that correct?

Is my understanding as well. But I'm not familiar enough with the code to tell where and how that -EINVAL gets handled.

BTW let me just point out that the bounce buffering via swiotlb needed for PV is not unlikely to mess up the alignment of things. But I'm not sure if that is relevant here.

Regards,
Halil

Quoted 6 lines

If so, could you
provide more details on what issue occurs with dispatching this request?

If you really need to restrict address' alignment to the storage's logical block size, I think your storage driver needs to set the dma_alignment queue limit to that value.

#25 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 47 lines · Halil Pasic
Quoted 34 lines · On Mon, 27 Jun 2022 09:36:56 -0600 Keith Busch <[email protected]> wrote:
Quoted 25 lines · Eric Farman

Apologies, it took me an extra day to get back to this, but it is indeed this pass through that's causing our boot failures. I note that
the old code (in iomap_dio_bio_iter), did:

        if ((pos | length | align) & ((1 << blkbits) - 1))
                return -EINVAL;

With blkbits equal to 12, the resulting mask was 0x0fff against an
align value (from iov_iter_alignment) of x200 kicks us out.

The new code (in iov_iter_aligned_iovec), meanwhile, compares this:

                if ((unsigned long)(i->iov[k].iov_base + skip) &
addr_mask)
                        return false;

iov_base (and the output of the old iov_iter_aligned_iovec() routine)
is x200, but since addr_mask is x1ff this check provides a different
response than it used to.

Quoted 7 lines

To check this, I changed the comparator to len_mask (almost certainly
not the right answer since addr_mask is then unused, but it was good
for a quick test), and our PV guests are able to boot again with -next
running in the host.

Quoted 5 lines

This raises more questions for me. It sounds like your process used to get an
EINVAL error, and it wants to continue getting an EINVAL error instead of
letting the direct-io request proceed. Is that correct?

Is my understanding as well. But I'm not familiar enough with the code to
tell where and how that -EINVAL gets handled.

Quoted 12 lines

BTW let me just point out that the bounce buffering via swiotlb needed
for PV is not unlikely to mess up the alignment of things. But I'm not
sure if that is relevant here.

Regards,
Halil

Quoted 3 lines

If so, could you
provide more details on what issue occurs with dispatching this request?

This error occurs reading the initial boot record for a guest, stating QEMU was unable to read block zero from the device. The code that complains doesn't appear to have anything that says "oh, got EINVAL, try it this other way" but I haven't chased down if/where something in between is expecting that and handling it in some unique way. I -think-
I have an easier reproducer now, so maybe I'd be able to get a better
answer to this question.

Quoted 6 lines
Quoted 6 lines

If you really need to restrict address' alignment to the storage's logical
block size, I think your storage driver needs to set the dma_alignment queue
limit to that value.

It's possible that there's a problem in the virtio stack here, but the failing configuration is a qcow image on the host rootfs, so it's not using any distinct driver. The bdev request queue that ends up being used is the same allocated out of blk_alloc_queue, so changing dma_alignment there wouldn't work.

#26 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 42 lines · Eric Farman
Quoted 41 lines · Halil Pasic
Quoted 38 lines · On Mon, 27 Jun 2022 09:36:56 -0600 Keith Busch <[email protected]> wrote:
Quoted 36 lines · Eric Farman

Apologies, it took me an extra day to get back to this, but it is
indeed this pass through that's causing our boot failures. I note
that
the old code (in iomap_dio_bio_iter), did:

        if ((pos | length | align) & ((1 << blkbits) - 1))
                return -EINVAL;

With blkbits equal to 12, the resulting mask was 0x0fff against an
align value (from iov_iter_alignment) of x200 kicks us out.

The new code (in iov_iter_aligned_iovec), meanwhile, compares this:

                if ((unsigned long)(i->iov[k].iov_base + skip)
&
addr_mask)
                        return false;

iov_base (and the output of the old iov_iter_aligned_iovec() routine)
is x200, but since addr_mask is x1ff this check provides a different
response than it used to.

To check this, I changed the comparator to len_mask (almost certainly
not the right answer since addr_mask is then unused, but it was good
for a quick test), and our PV guests are able to boot again with
-next
running in the host.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

This raises more questions for me. It sounds like your process used
to get an
EINVAL error, and it wants to continue getting an EINVAL error instead of
letting the direct-io request proceed. Is that correct?

Sort of. In the working case, I see a set of iovecs come through with different counts:

base count
0000 0001
0000 0200
0000 0400
0000 0800
0000 1000
0001 1000
0200 1000 << Change occurs here
0400 1000
0800 1000
1000 1000

EINVAL was being returned for any of these iovecs except the page- aligned ones. Once the x200 request returns 0, the remainder of the above list was skipped and the requests continue elsewhere on the file.

Still not sure how our request is getting us into this process. We're simply asking to read a single block, but that's somewhere within an image file.

Quoted 10 lines
Quoted 10 lines

Is my understanding as well. But I'm not familiar enough with the code to
tell where and how that -EINVAL gets handled.

BTW let me just point out that the bounce buffering via swiotlb needed
for PV is not unlikely to mess up the alignment of things. But I'm not
sure if that is relevant here.

It's true that PV guests were the first to trip over this, but I've since been able to reproduce this with a normal guest. So long as the image file is connected with cache.direct=true, it's unbootable. That should absolve the swiotlb bits from being at fault here.

Quoted 27 lines
Quoted 7 lines

Regards,
Halil

Quoted 3 lines

If so, could you
provide more details on what issue occurs with dispatching this request?

This error occurs reading the initial boot record for a guest, stating
QEMU was unable to read block zero from the device. The code that complains doesn't appear to have anything that says "oh, got EINVAL, try it this other way" but I haven't chased down if/where something in
between is expecting that and handling it in some unique way. I -think-
I have an easier reproducer now, so maybe I'd be able to get a
better
answer to this question.

Quoted 6 lines
Quoted 6 lines

If you really need to restrict address' alignment to the storage's
logical
block size, I think your storage driver needs to set the dma_alignment queue
limit to that value.

Quoted 3 lines

It's possible that there's a problem in the virtio stack here, but the
failing configuration is a qcow image on the host rootfs

(on an ext4 filesystem)

Quoted 4 lines

, so it's not
using any distinct driver. The bdev request queue that ends up being used is the same allocated out of blk_alloc_queue, so changing dma_alignment there wouldn't work.

maintainer
#27 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 15 lines · Eric Farman

Sort of. In the working case, I see a set of iovecs come through with different counts:

base count
0000 0001
0000 0200
0000 0400
0000 0800
0000 1000
0001 1000
0200 1000 << Change occurs here
0400 1000
0800 1000
1000 1000

Quoted 7 lines

EINVAL was being returned for any of these iovecs except the page- aligned ones. Once the x200 request returns 0, the remainder of the above list was skipped and the requests continue elsewhere on the file.

Still not sure how our request is getting us into this process. We're simply asking to read a single block, but that's somewhere within an image file.

I thought this was sounding like some kind of corruption. I tested ext4 on various qemu devices with 4k logical block sizes, and it all looks okay there.

What block driver are you observing this with?

#28 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 28 lines · Keith Busch
Quoted 21 lines · Eric Farman

Sort of. In the working case, I see a set of iovecs come through with
different counts:

base count
0000 0001
0000 0200
0000 0400
0000 0800
0000 1000
0001 1000
0200 1000 << Change occurs here
0400 1000
0800 1000
1000 1000

EINVAL was being returned for any of these iovecs except the page- aligned ones. Once the x200 request returns 0, the remainder of the above list was skipped and the requests continue elsewhere on the file.

Quoted 5 lines

Still not sure how our request is getting us into this process. We're
simply asking to read a single block, but that's somewhere within an
image file.

Quoted 6 lines

I thought this was sounding like some kind of corruption. I tested ext4 on
various qemu devices with 4k logical block sizes, and it all looks okay there.

What block driver are you observing this with?

s390 dasd

This made me think to change my rootfs, and of course the problem goes away once on something like a SCSI volume.

So crawling through the dasd (instead of virtio) driver and I finally find the point where a change to dma_alignment (which you mentioned earlier) would actually fit.

Such a change fixes this for me, so I'll run it by our DASD guys. Thanks for your help and patience.

Eric

maintainer
#29 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 11 lines · Eric Farman

s390 dasd

This made me think to change my rootfs, and of course the problem goes away once on something like a SCSI volume.

So crawling through the dasd (instead of virtio) driver and I finally find the point where a change to dma_alignment (which you mentioned earlier) would actually fit.

Such a change fixes this for me, so I'll run it by our DASD guys. Thanks for your help and patience.

I'm assuming there's some driver or device requirement that's making this necessary. Is the below driver change what you're looking for? If so, I think you might want this regardless of this direct-io patch just because other interfaces like blk_rq_map_user_iov() and blk_rq_aligned() align to it.

---

drivers/s390/block/dasd_fba.c +1 -0
index 60be7f7bf2d1..5c79fb02cded 100644
@@ -780,6 +780,7 @@ static void dasd_fba_setup_blk_queue(struct dasd_block *block)
 	/* With page sized segments each segment can be translated into one idaw/tidaw */
 	blk_queue_max_segment_size(q, PAGE_SIZE);
 	blk_queue_segment_boundary(q, PAGE_SIZE - 1);
+	blk_queue_dma_alignment(q, PAGE_SIZE - 1);
 
 	q->limits.discard_granularity = logical_block_size;
 

--

#30 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 12 lines · Keith Busch
Quoted 11 lines · Eric Farman

s390 dasd

This made me think to change my rootfs, and of course the problem goes
away once on something like a SCSI volume.

So crawling through the dasd (instead of virtio) driver and I finally
find the point where a change to dma_alignment (which you mentioned earlier) would actually fit.

Quoted 6 lines
Quoted 2 lines

Such a change fixes this for me, so I'll run it by our DASD guys. Thanks for your help and patience.

I'm assuming there's some driver or device requirement that's making this
necessary. Is the below driver change what you're looking for?

Yup, that's exactly what I have (in dasd_eckd.c) and indeed gets things working again. Need to scrounge up some FBA volumes to test that configuration and the change there.

Quoted 5 lines

If so, I think
you might want this regardless of this direct-io patch just because other
interfaces like blk_rq_map_user_iov() and blk_rq_aligned() align to it.

Good point.

Quoted 18 lines

---

drivers/s390/block/dasd_fba.c +0 -0
index 60be7f7bf2d1..5c79fb02cded 100644
@@ -780,6 +780,7 @@ static void dasd_fba_setup_blk_queue(struct
 dasd_block *block)
 	/* With page sized segments each segment can be translated into

one idaw/tidaw */

drivers/s390/block/dasd_fba.c +1 -1
 	blk_queue_max_segment_size(q, PAGE_SIZE);
 	blk_queue_segment_boundary(q, PAGE_SIZE - 1);
+	blk_queue_dma_alignment(q, PAGE_SIZE - 1);
 
 	q->limits.discard_granularity = logical_block_size;
 
--
#31 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

In reply to Keith Busch

CCing Stefan, Jan.

Below is necessary (in dasd-eckd.c) to make kvm boot working again (and this seems to be the right thing anyway).

Am 29.06.22 um 21:07 schrieb Keith Busch:

Quoted 32 lines · Eric Farman
Quoted 11 lines

s390 dasd

This made me think to change my rootfs, and of course the problem goes away once on something like a SCSI volume.

So crawling through the dasd (instead of virtio) driver and I finally find the point where a change to dma_alignment (which you mentioned earlier) would actually fit.

Such a change fixes this for me, so I'll run it by our DASD guys. Thanks for your help and patience.

I'm assuming there's some driver or device requirement that's making this necessary. Is the below driver change what you're looking for? If so, I think you might want this regardless of this direct-io patch just because other interfaces like blk_rq_map_user_iov() and blk_rq_aligned() align to it.

---

drivers/s390/block/dasd_fba.c +1 -1
index 60be7f7bf2d1..5c79fb02cded 100644
@@ -780,6 +780,7 @@ static void dasd_fba_setup_blk_queue(struct dasd_block *block)
  	/* With page sized segments each segment can be translated into one idaw/tidaw */
  	blk_queue_max_segment_size(q, PAGE_SIZE);
  	blk_queue_segment_boundary(q, PAGE_SIZE - 1);
+	blk_queue_dma_alignment(q, PAGE_SIZE - 1);
 
  	q->limits.discard_granularity = logical_block_size;
 
--
#32 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

In reply to Keith Busch

[+f2fs list and maintainers]

Quoted 24 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 259

Quoted 9 lines
fs/iomap/direct-io.c +2 -1
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
 		return -EINVAL;
 
 	if (iomap->type == IOMAP_UNWRITTEN) {

I noticed that this patch is going to break the following logic in f2fs_should_use_dio() in fs/f2fs/file.c:

	/*
	 * Direct I/O not aligned to the disk's logical_block_size will be
	 * attempted, but will fail with -EINVAL.
	 *
	 * f2fs additionally requires that direct I/O be aligned to the
	 * filesystem block size, which is often a stricter requirement.
	 * However, f2fs traditionally falls back to buffered I/O on requests
	 * that are logical_block_size-aligned but not fs-block aligned.
	 *
	 * The below logic implements this behavior.
	 */
	align = iocb->ki_pos | iov_iter_alignment(iter);
	if (!IS_ALIGNED(align, i_blocksize(inode)) &&
	    IS_ALIGNED(align, bdev_logical_block_size(inode->i_sb->s_bdev)))
		return false;

	return true;

So, f2fs assumes that __iomap_dio_rw() returns an error if the I/O isn't logical block aligned. This patch changes that. The result is that DIO will sometimes proceed in cases where the I/O doesn't have the fs block alignment required by f2fs for all DIO.

Does anyone have any thoughts about what f2fs should be doing here? I think it's weird that f2fs has different behaviors for different degrees of
misalignment: fail with EINVAL if not logical block aligned, else fallback to buffered I/O if not fs block aligned. I think it should be one convention or the other. Any opinions about which one it should be?

(Note: if you blame the above code, it was written by me. But I was just preserving the existing behavior; I don't know the original motivation.)

- Eric

maintainer
#33 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 63 lines · Eric Biggers

[+f2fs list and maintainers]

Quoted 24 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 259

Quoted 9 lines
fs/iomap/direct-io.c +2 -1
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
 		return -EINVAL;
 
 	if (iomap->type == IOMAP_UNWRITTEN) {

I noticed that this patch is going to break the following logic in f2fs_should_use_dio() in fs/f2fs/file.c:

	/*
	 * Direct I/O not aligned to the disk's logical_block_size will be
	 * attempted, but will fail with -EINVAL.
	 *
	 * f2fs additionally requires that direct I/O be aligned to the
	 * filesystem block size, which is often a stricter requirement.
	 * However, f2fs traditionally falls back to buffered I/O on requests
	 * that are logical_block_size-aligned but not fs-block aligned.
	 *
	 * The below logic implements this behavior.
	 */
	align = iocb->ki_pos | iov_iter_alignment(iter);
	if (!IS_ALIGNED(align, i_blocksize(inode)) &&
	    IS_ALIGNED(align, bdev_logical_block_size(inode->i_sb->s_bdev)))
		return false;

	return true;

So, f2fs assumes that __iomap_dio_rw() returns an error if the I/O isn't logical block aligned. This patch changes that. The result is that DIO will sometimes proceed in cases where the I/O doesn't have the fs block alignment required by f2fs for all DIO.

Quoted 5 lines

Does anyone have any thoughts about what f2fs should be doing here? I think it's weird that f2fs has different behaviors for different degrees of
misalignment: fail with EINVAL if not logical block aligned, else fallback to buffered I/O if not fs block aligned. I think it should be one convention or the other. Any opinions about which one it should be?

It looks like f2fs just falls back to buffered IO for this condition without reaching the new code in iomap_dio_bio_iter(). btrfs does the same thing (check_direct_IO()). Is that a problem?

#34 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 64 lines · Keith Busch
Quoted 63 lines · Eric Biggers

[+f2fs list and maintainers]

Quoted 24 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 259

Quoted 9 lines
fs/iomap/direct-io.c +2 -1
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
 		return -EINVAL;
 
 	if (iomap->type == IOMAP_UNWRITTEN) {

I noticed that this patch is going to break the following logic in f2fs_should_use_dio() in fs/f2fs/file.c:

	/*
	 * Direct I/O not aligned to the disk's logical_block_size will be
	 * attempted, but will fail with -EINVAL.
	 *
	 * f2fs additionally requires that direct I/O be aligned to the
	 * filesystem block size, which is often a stricter requirement.
	 * However, f2fs traditionally falls back to buffered I/O on requests
	 * that are logical_block_size-aligned but not fs-block aligned.
	 *
	 * The below logic implements this behavior.
	 */
	align = iocb->ki_pos | iov_iter_alignment(iter);
	if (!IS_ALIGNED(align, i_blocksize(inode)) &&
	    IS_ALIGNED(align, bdev_logical_block_size(inode->i_sb->s_bdev)))
		return false;

	return true;

So, f2fs assumes that __iomap_dio_rw() returns an error if the I/O isn't logical block aligned. This patch changes that. The result is that DIO will sometimes proceed in cases where the I/O doesn't have the fs block alignment required by f2fs for all DIO.

Quoted 8 lines
Quoted 5 lines

Does anyone have any thoughts about what f2fs should be doing here? I think it's weird that f2fs has different behaviors for different degrees of
misalignment: fail with EINVAL if not logical block aligned, else fallback to buffered I/O if not fs block aligned. I think it should be one convention or the other. Any opinions about which one it should be?

It looks like f2fs just falls back to buffered IO for this condition without reaching the new code in iomap_dio_bio_iter().

No. It's a bit subtle, so read the code and what I'm saying carefully. f2fs only supports 4K aligned DIO and normally falls back to buffered I/O; however, for DIO that is *very* misaligned (not even LBS aligned) it returns EINVAL instead. And it relies on __iomap_dio_rw() returning that EINVAL.

Relying on __iomap_dio_rw() in that way is definitely a bad design on f2fs's part (and I messed that up when switching f2fs from fs/direct-io.c to iomap). The obvious fix is to just have f2fs do the LBS alignment check itself.

But I think that f2fs shouldn't have different behavior for different levels of misalignment in the first place, so I was wondering if anyone had any thoughts on which behavior (EINVAL or fallback to buffered I/O) should be standardized on in all cases, at least for f2fs. There was some discussion about this sort of thing for ext4 several years ago on the thread
https://lore.kernel.org/linux-ext4/[email protected]/T/#u,
but it didn't really reach a conclusion. I'm wondering if the f2fs maintainers have any thoughts about why the f2fs behavior is as it is. I.e. is it just accidental, or are there specific reasons...

- Eric

maintainer
#35 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 71 lines · Eric Biggers
Quoted 70 lines · Keith Busch
Quoted 68 lines · Eric Biggers

[+f2fs list and maintainers]

Quoted 24 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 259

Quoted 9 lines
fs/iomap/direct-io.c +2 -1
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
 		return -EINVAL;
 
 	if (iomap->type == IOMAP_UNWRITTEN) {

I noticed that this patch is going to break the following logic in f2fs_should_use_dio() in fs/f2fs/file.c:

	/*
	 * Direct I/O not aligned to the disk's logical_block_size will be
	 * attempted, but will fail with -EINVAL.
	 *
	 * f2fs additionally requires that direct I/O be aligned to the
	 * filesystem block size, which is often a stricter requirement.
	 * However, f2fs traditionally falls back to buffered I/O on requests
	 * that are logical_block_size-aligned but not fs-block aligned.
	 *
	 * The below logic implements this behavior.
	 */
	align = iocb->ki_pos | iov_iter_alignment(iter);
	if (!IS_ALIGNED(align, i_blocksize(inode)) &&
	    IS_ALIGNED(align, bdev_logical_block_size(inode->i_sb->s_bdev)))
		return false;

	return true;

So, f2fs assumes that __iomap_dio_rw() returns an error if the I/O isn't logical block aligned. This patch changes that. The result is that DIO will sometimes proceed in cases where the I/O doesn't have the fs block alignment required by f2fs for all DIO.

Does anyone have any thoughts about what f2fs should be doing here? I think it's weird that f2fs has different behaviors for different degrees of
misalignment: fail with EINVAL if not logical block aligned, else fallback to buffered I/O if not fs block aligned. I think it should be one convention or the other. Any opinions about which one it should be?

Quoted 7 lines
Quoted 2 lines

It looks like f2fs just falls back to buffered IO for this condition without reaching the new code in iomap_dio_bio_iter().

No. It's a bit subtle, so read the code and what I'm saying carefully. f2fs only supports 4K aligned DIO and normally falls back to buffered I/O; however, for DIO that is *very* misaligned (not even LBS aligned) it returns EINVAL instead. And it relies on __iomap_dio_rw() returning that EINVAL.

Okay, I understand the code flow now.

I tested f2fs direct io with every possible alignment, and it is successful for all hardware dma supported alignments and continues to return EINVAL for unaligned. Is the concern that it's now returning success in scenarios that used to fail? Or is there some other 4k f2fs constraint that I haven't found yet?

Quoted 15 lines

Relying on __iomap_dio_rw() in that way is definitely a bad design on f2fs's part (and I messed that up when switching f2fs from fs/direct-io.c to iomap). The obvious fix is to just have f2fs do the LBS alignment check itself.

But I think that f2fs shouldn't have different behavior for different levels of misalignment in the first place, so I was wondering if anyone had any thoughts on which behavior (EINVAL or fallback to buffered I/O) should be standardized on in all cases, at least for f2fs. There was some discussion about this sort of thing for ext4 several years ago on the thread
https://lore.kernel.org/linux-ext4/[email protected]/T/#u,
but it didn't really reach a conclusion. I'm wondering if the f2fs maintainers have any thoughts about why the f2fs behavior is as it is. I.e. is it just accidental, or are there specific reasons...

- Eric

#36 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 80 lines · Keith Busch
Quoted 71 lines · Eric Biggers
Quoted 70 lines · Keith Busch
Quoted 68 lines · Eric Biggers

[+f2fs list and maintainers]

Quoted 24 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 259

Quoted 9 lines
fs/iomap/direct-io.c +2 -1
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
 		return -EINVAL;
 
 	if (iomap->type == IOMAP_UNWRITTEN) {

I noticed that this patch is going to break the following logic in f2fs_should_use_dio() in fs/f2fs/file.c:

	/*
	 * Direct I/O not aligned to the disk's logical_block_size will be
	 * attempted, but will fail with -EINVAL.
	 *
	 * f2fs additionally requires that direct I/O be aligned to the
	 * filesystem block size, which is often a stricter requirement.
	 * However, f2fs traditionally falls back to buffered I/O on requests
	 * that are logical_block_size-aligned but not fs-block aligned.
	 *
	 * The below logic implements this behavior.
	 */
	align = iocb->ki_pos | iov_iter_alignment(iter);
	if (!IS_ALIGNED(align, i_blocksize(inode)) &&
	    IS_ALIGNED(align, bdev_logical_block_size(inode->i_sb->s_bdev)))
		return false;

	return true;

So, f2fs assumes that __iomap_dio_rw() returns an error if the I/O isn't logical block aligned. This patch changes that. The result is that DIO will sometimes proceed in cases where the I/O doesn't have the fs block alignment required by f2fs for all DIO.

Does anyone have any thoughts about what f2fs should be doing here? I think it's weird that f2fs has different behaviors for different degrees of
misalignment: fail with EINVAL if not logical block aligned, else fallback to buffered I/O if not fs block aligned. I think it should be one convention or the other. Any opinions about which one it should be?

Quoted 7 lines
Quoted 2 lines

It looks like f2fs just falls back to buffered IO for this condition without reaching the new code in iomap_dio_bio_iter().

No. It's a bit subtle, so read the code and what I'm saying carefully. f2fs only supports 4K aligned DIO and normally falls back to buffered I/O; however, for DIO that is *very* misaligned (not even LBS aligned) it returns EINVAL instead. And it relies on __iomap_dio_rw() returning that EINVAL.

Quoted 7 lines

Okay, I understand the code flow now.

I tested f2fs direct io with every possible alignment, and it is successful for all hardware dma supported alignments and continues to return EINVAL for unaligned. Is the concern that it's now returning success in scenarios that used to fail? Or is there some other 4k f2fs constraint that I haven't found yet?

f2fs has a lot of different options, and edge cases that only occur occasionally such as data blocks being moved by garbage collection. So just because misaligned DIO appears to work doesn't necessarily mean that it actually works. Maybe the f2fs developers can elaborate on the reason that f2fs always requires 4K alignment for DIO.

- Eric

#37 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

In reply to Eric Biggers
Quoted 83 lines · Eric Biggers
Quoted 64 lines · Keith Busch
Quoted 63 lines · Eric Biggers

[+f2fs list and maintainers]

Quoted 24 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 259

Quoted 9 lines
fs/iomap/direct-io.c +2 -1
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
 		return -EINVAL;
 
 	if (iomap->type == IOMAP_UNWRITTEN) {

I noticed that this patch is going to break the following logic in f2fs_should_use_dio() in fs/f2fs/file.c:

	/*
	 * Direct I/O not aligned to the disk's logical_block_size will be
	 * attempted, but will fail with -EINVAL.
	 *
	 * f2fs additionally requires that direct I/O be aligned to the
	 * filesystem block size, which is often a stricter requirement.
	 * However, f2fs traditionally falls back to buffered I/O on requests
	 * that are logical_block_size-aligned but not fs-block aligned.
	 *
	 * The below logic implements this behavior.
	 */
	align = iocb->ki_pos | iov_iter_alignment(iter);
	if (!IS_ALIGNED(align, i_blocksize(inode)) &&
	    IS_ALIGNED(align, bdev_logical_block_size(inode->i_sb->s_bdev)))
		return false;

	return true;

So, f2fs assumes that __iomap_dio_rw() returns an error if the I/O isn't logical block aligned. This patch changes that. The result is that DIO will sometimes proceed in cases where the I/O doesn't have the fs block alignment required by f2fs for all DIO.

Quoted 8 lines
Quoted 5 lines

Does anyone have any thoughts about what f2fs should be doing here? I think it's weird that f2fs has different behaviors for different degrees of
misalignment: fail with EINVAL if not logical block aligned, else fallback to buffered I/O if not fs block aligned. I think it should be one convention or the other. Any opinions about which one it should be?

It looks like f2fs just falls back to buffered IO for this condition without reaching the new code in iomap_dio_bio_iter().

No. It's a bit subtle, so read the code and what I'm saying carefully. f2fs only supports 4K aligned DIO and normally falls back to buffered I/O; however, for DIO that is *very* misaligned (not even LBS aligned) it returns EINVAL instead. And it relies on __iomap_dio_rw() returning that EINVAL.

Relying on __iomap_dio_rw() in that way is definitely a bad design on f2fs's part (and I messed that up when switching f2fs from fs/direct-io.c to iomap). The obvious fix is to just have f2fs do the LBS alignment check itself.

Quoted 9 lines

But I think that f2fs shouldn't have different behavior for different levels of misalignment in the first place, so I was wondering if anyone had any thoughts on which behavior (EINVAL or fallback to buffered I/O) should be standardized on in all cases, at least for f2fs. There was some discussion about this sort of thing for ext4 several years ago on the thread
https://lore.kernel.org/linux-ext4/[email protected]/T/#u,
but it didn't really reach a conclusion. I'm wondering if the f2fs maintainers have any thoughts about why the f2fs behavior is as it is. I.e. is it just accidental, or are there specific reasons...

If there's a generic way to deal with this, I have no objection to follow it. Initially, I remember I was trying to match the ext4 rule, but at some point, I lost the track.

Quoted 2 lines

- Eric

#38 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

In reply to Eric Biggers
Quoted 30 lines · Eric Biggers

[+f2fs list and maintainers]

Quoted 27 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
Quoted 3 lines

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 256

Quoted 3 lines
fs/iomap/direct-io.c +2 -1
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))

How does this change intersect with "make statx() return DIO alignment information" ? Will the new STATX_DIOALIGN implementations have to be adjusted to set stx_dio_mem_align = bdev_dma_alignment(...)?

I'm guessing the answer is yes, but I haven't seen any patches on the list to do that, but more and more these days email behaves like a flood of UDP traffic... :(

--D

Quoted 40 lines

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 259

Quoted 3 lines
fs/iomap/direct-io.c +0 -0
 		return -EINVAL;
 
 	if (iomap->type == IOMAP_UNWRITTEN) {

I noticed that this patch is going to break the following logic in f2fs_should_use_dio() in fs/f2fs/file.c:

	/*
	 * Direct I/O not aligned to the disk's logical_block_size will be
	 * attempted, but will fail with -EINVAL.
	 *
	 * f2fs additionally requires that direct I/O be aligned to the
	 * filesystem block size, which is often a stricter requirement.
	 * However, f2fs traditionally falls back to buffered I/O on requests
	 * that are logical_block_size-aligned but not fs-block aligned.
	 *
	 * The below logic implements this behavior.
	 */
	align = iocb->ki_pos | iov_iter_alignment(iter);
	if (!IS_ALIGNED(align, i_blocksize(inode)) &&
	    IS_ALIGNED(align, bdev_logical_block_size(inode->i_sb->s_bdev)))
		return false;

	return true;

So, f2fs assumes that __iomap_dio_rw() returns an error if the I/O isn't logical block aligned. This patch changes that. The result is that DIO will sometimes proceed in cases where the I/O doesn't have the fs block alignment required by f2fs for all DIO.

Does anyone have any thoughts about what f2fs should be doing here? I think it's weird that f2fs has different behaviors for different degrees of
misalignment: fail with EINVAL if not logical block aligned, else fallback to buffered I/O if not fs block aligned. I think it should be one convention or the other. Any opinions about which one it should be?

(Note: if you blame the above code, it was written by me. But I was just preserving the existing behavior; I don't know the original motivation.)

- Eric

#39 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 39 lines · Darrick J. Wong
Quoted 30 lines · Eric Biggers

[+f2fs list and maintainers]

Quoted 27 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
Quoted 3 lines

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 256

Quoted 3 lines
fs/iomap/direct-io.c +2 -1
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))

How does this change intersect with "make statx() return DIO alignment information" ? Will the new STATX_DIOALIGN implementations have to be adjusted to set stx_dio_mem_align = bdev_dma_alignment(...)?

Quoted 4 lines

I'm guessing the answer is yes, but I haven't seen any patches on the list to do that, but more and more these days email behaves like a flood of UDP traffic... :(

Yes. I haven't done that in the STATX_DIOALIGN patchset yet because I've been basing it on upstream, which doesn't yet have this iomap patch. I haven't been expecting STATX_DIOALIGN to make 5.20, given that it's a new UAPI that needs time to be properly reviewed, plus I've just been busy with other things. So I've been planning to make the above change after this patch lands upstream.

- Eric

#40 lore raw

Re: [PATCHv6 11/11] iomap: add support for dma aligned direct-io

Quoted 45 lines · Eric Biggers
Quoted 39 lines · Darrick J. Wong
Quoted 30 lines · Eric Biggers

[+f2fs list and maintainers]

Quoted 27 lines · Keith Busch

From: Keith Busch <[email protected]>

Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size.

---
 fs/iomap/direct-io.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/iomap/direct-io.c +0 -1
index 370c3241618a..5d098adba443 100644
@@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	struct inode *inode = iter->inode;
 	unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev));
 	unsigned int fs_block_size = i_blocksize(inode), pad;
-	unsigned int align = iov_iter_alignment(dio->submit.iter);
 	loff_t length = iomap_length(iter);
 	loff_t pos = iter->pos;
 	unsigned int bio_opf;
@@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
 	size_t copied = 0;
 	size_t orig_count;
 
Quoted 3 lines

fs/iomap/direct-io.c static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · line 256

Quoted 3 lines
fs/iomap/direct-io.c +2 -1
-	if ((pos | length | align) & ((1 << blkbits) - 1))
+	if ((pos | length) & ((1 << blkbits) - 1) ||
+	    !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))

How does this change intersect with "make statx() return DIO alignment information" ? Will the new STATX_DIOALIGN implementations have to be adjusted to set stx_dio_mem_align = bdev_dma_alignment(...)?

Quoted 4 lines

I'm guessing the answer is yes, but I haven't seen any patches on the list to do that, but more and more these days email behaves like a flood of UDP traffic... :(

Quoted 5 lines

Yes. I haven't done that in the STATX_DIOALIGN patchset yet because I've been basing it on upstream, which doesn't yet have this iomap patch. I haven't been expecting STATX_DIOALIGN to make 5.20, given that it's a new UAPI that needs time to be properly reviewed, plus I've just been busy with other things. So I've been planning to make the above change after this patch lands upstream.

<nod> Ok, I'm looking forward to it. Thank you for your work on statx! :)

--D

Quoted 1 line

- Eric

maintainer Applied
#41 lore raw
In reply to Keith Busch
Quoted 9 lines · Keith Busch

From: Keith Busch <[email protected]>

The previous version is available here:

https://lore.kernel.org/linux-block/[email protected]/T/#m0a93b6392038aad6144e066fb5ada2cbf316f78e

Changes from the previous are all trivial changes:

[...]

Applied, thanks!

[01/11] block: fix infinite loop for invalid zone append

        commit: 1180b55c93f6b060ad930db151fe6d2b425f9215
[02/11] block/bio: remove duplicate append pages code
        commit: 7a2b81b95a89e578343b1c944ddd64d1b14ee49a
[03/11] block: export dma_alignment attribute
        commit: 5f507439f051daaa1e3273ff536afda3ad1f1505
[04/11] block: introduce bdev_dma_alignment helper
        commit: 24b10a6e0bc22619535b0ed982b7735910981661
[05/11] block: add a helper function for dio alignment
        commit: 8a39418810a65f0bcbe559261ef011fe0e298eeb
[06/11] block/merge: count bytes instead of sectors
        commit: 4ff782f24a4cad4b033d0f4f6e38cd50e0d463b0
[07/11] block/bounce: count bytes instead of sectors
        commit: 4b5310470e72d77c9b52f8544b98aa8cf77d956f
[08/11] iov: introduce iov_iter_aligned
        commit: ab7c0c3abb2e5fa15655e4b87bb7b937ca7e18c3
[09/11] block: introduce bdev_iter_is_aligned helper
        commit: 72230944b7a53280c1f351a0d5cafed12732ec21
[10/11] block: relax direct io memory alignment
        commit: 84f970d415ef4d048e664ac308792eb93d0152fc
[11/11] iomap: add support for dma aligned direct-io
        commit: 40e11e7a6cc74f11b5ca23ceefec7c84af5c4c73

Best regards,

Signature · 1 line
Jens Axboe