Skip to contentSkip to conversation

io_import_fixed cleanups and optimisation

PATCH Applied Pavel Begunkov 3 participants 15 messages Apr 17, 2025 io_uring

lore
Share & tools
lore
#0 lore raw

cover io_import_fixed cleanups and optimisation

io_import_fixed() cleanups topped with the nr segs optimisation patch from Nitesh. Doesn't have the kbuf part, assuming Jens will add it on top.

Based on io_uring-6.15

Nitesh Shetty (1):
io_uring/rsrc: send exact nr_segs for fixed buffer

Pavel Begunkov (3):
io_uring/rsrc: don't skip offset calculation
io_uring/rsrc: separate kbuf offset adjustments
io_uring/rsrc: refactor io_import_fixed

 io_uring/rsrc.c | 74 ++++++++++++++++++++-----------------------------
 1 file changed, 30 insertions(+), 44 deletions(-)
Signature · 1 line
2.48.1
#1 lore raw

PATCH 1/4 io_uring/rsrc: don't skip offset calculation

Don't optimise for requests with offset=0. Large registered buffers are the preference and hence the user is likely to pass an offset, and the adjustments are not expensive and will be made even cheaper in following patches.

---
 io_uring/rsrc.c | 75 ++++++++++++++++++++++++-------------------------
 1 file changed, 37 insertions(+), 38 deletions(-)
io_uring/rsrc.c +37 -38
index b36c8825550e..4d62897d1c89 100644
@@ -1036,6 +1036,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 			   struct io_mapped_ubuf *imu,
 			   u64 buf_addr, size_t len)
 {
+	const struct bio_vec *bvec;
 	size_t offset;
 	int ret;
 
@@ -1054,47 +1055,45 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 	offset = buf_addr - imu->ubuf;
 	iov_iter_bvec(iter, ddir, imu->bvec, imu->nr_bvecs, offset + len);
 
-	if (offset) {
-		/*
-		 * Don't use iov_iter_advance() here, as it's really slow for
-		 * using the latter parts of a big fixed buffer - it iterates
-		 * over each segment manually. We can cheat a bit here for user
-		 * registered nodes, because we know that:
-		 *
-		 * 1) it's a BVEC iter, we set it up
-		 * 2) all bvecs are the same in size, except potentially the
-		 *    first and last bvec
-		 *
-		 * So just find our index, and adjust the iterator afterwards.
-		 * If the offset is within the first bvec (or the whole first
-		 * bvec, just use iov_iter_advance(). This makes it easier
-		 * since we can just skip the first segment, which may not
-		 * be folio_size aligned.
-		 */
-		const struct bio_vec *bvec = imu->bvec;
+	/*
+	 * Don't use iov_iter_advance() here, as it's really slow for
+	 * using the latter parts of a big fixed buffer - it iterates
+	 * over each segment manually. We can cheat a bit here for user
+	 * registered nodes, because we know that:
+	 *
+	 * 1) it's a BVEC iter, we set it up
+	 * 2) all bvecs are the same in size, except potentially the
+	 *    first and last bvec
+	 *
+	 * So just find our index, and adjust the iterator afterwards.
+	 * If the offset is within the first bvec (or the whole first
+	 * bvec, just use iov_iter_advance(). This makes it easier
+	 * since we can just skip the first segment, which may not
+	 * be folio_size aligned.
+	 */
+	bvec = imu->bvec;
 
-		/*
-		 * Kernel buffer bvecs, on the other hand, don't necessarily
-		 * have the size property of user registered ones, so we have
-		 * to use the slow iter advance.
-		 */
-		if (offset < bvec->bv_len) {
-			iter->count -= offset;
-			iter->iov_offset = offset;
-		} else if (imu->is_kbuf) {
-			iov_iter_advance(iter, offset);
-		} else {
-			unsigned long seg_skip;
+	/*
+	 * Kernel buffer bvecs, on the other hand, don't necessarily
+	 * have the size property of user registered ones, so we have
+	 * to use the slow iter advance.
+	 */
+	if (offset < bvec->bv_len) {
+		iter->count -= offset;
+		iter->iov_offset = offset;
+	} else if (imu->is_kbuf) {
+		iov_iter_advance(iter, offset);
+	} else {
+		unsigned long seg_skip;
 
-			/* skip first vec */
-			offset -= bvec->bv_len;
-			seg_skip = 1 + (offset >> imu->folio_shift);
+		/* skip first vec */
+		offset -= bvec->bv_len;
+		seg_skip = 1 + (offset >> imu->folio_shift);
 
-			iter->bvec += seg_skip;
-			iter->nr_segs -= seg_skip;
-			iter->count -= bvec->bv_len + offset;
-			iter->iov_offset = offset & ((1UL << imu->folio_shift) - 1);
-		}
+		iter->bvec += seg_skip;
+		iter->nr_segs -= seg_skip;
+		iter->count -= bvec->bv_len + offset;
+		iter->iov_offset = offset & ((1UL << imu->folio_shift) - 1);
 	}
 
 	return 0;
Signature · 1 line
2.48.1
#2 lore raw

PATCH 2/4 io_uring/rsrc: separate kbuf offset adjustments

In reply to Pavel Begunkov

Kernel registered buffers are special because segments are not uniform in size, and we have a bunch of optimisations based on that uniformity for normal buffers. Handle kbuf separately, it'll be cleaner this way.

---
 io_uring/rsrc.c | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
io_uring/rsrc.c +7 -12
index 4d62897d1c89..fddde8ffe81e 100644
@@ -1048,11 +1048,14 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 	if (!(imu->dir & (1 << ddir)))
 		return -EFAULT;
 
-	/*
-	 * Might not be a start of buffer, set size appropriately
-	 * and advance us to the beginning.
-	 */
 	offset = buf_addr - imu->ubuf;
+
+	if (imu->is_kbuf) {
+		iov_iter_bvec(iter, ddir, imu->bvec, imu->nr_bvecs, offset + len);
+		iov_iter_advance(iter, offset);
+		return 0;
+	}
+
 	iov_iter_bvec(iter, ddir, imu->bvec, imu->nr_bvecs, offset + len);
 
 	/*
@@ -1072,17 +1075,9 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 	 * be folio_size aligned.
 	 */
 	bvec = imu->bvec;
-
-	/*
-	 * Kernel buffer bvecs, on the other hand, don't necessarily
-	 * have the size property of user registered ones, so we have
-	 * to use the slow iter advance.
-	 */
 	if (offset < bvec->bv_len) {
 		iter->count -= offset;
 		iter->iov_offset = offset;
-	} else if (imu->is_kbuf) {
-		iov_iter_advance(iter, offset);
 	} else {
 		unsigned long seg_skip;
 
Signature · 1 line
2.48.1
#3 lore raw

PATCH 3/4 io_uring/rsrc: refactor io_import_fixed

In reply to Pavel Begunkov

io_import_fixed is a mess. Even though we know the final len of the iterator, we still assign offset + len and do some magic after to correct for that.

Do offset calculation first and finalise it with iov_iter_bvec at the end.

---
 io_uring/rsrc.c | 24 +++++++-----------------
 1 file changed, 7 insertions(+), 17 deletions(-)
io_uring/rsrc.c +7 -17
index fddde8ffe81e..5cf854318b1d 100644
@@ -1037,6 +1037,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 			   u64 buf_addr, size_t len)
 {
 	const struct bio_vec *bvec;
+	unsigned nr_segs;
 	size_t offset;
 	int ret;
 
@@ -1056,8 +1057,6 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 		return 0;
 	}
 
-	iov_iter_bvec(iter, ddir, imu->bvec, imu->nr_bvecs, offset + len);
-
 	/*
 	 * Don't use iov_iter_advance() here, as it's really slow for
 	 * using the latter parts of a big fixed buffer - it iterates
@@ -1067,30 +1066,21 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 	 * 1) it's a BVEC iter, we set it up
 	 * 2) all bvecs are the same in size, except potentially the
 	 *    first and last bvec
-	 *
-	 * So just find our index, and adjust the iterator afterwards.
-	 * If the offset is within the first bvec (or the whole first
-	 * bvec, just use iov_iter_advance(). This makes it easier
-	 * since we can just skip the first segment, which may not
-	 * be folio_size aligned.
 	 */
 	bvec = imu->bvec;
-	if (offset < bvec->bv_len) {
-		iter->count -= offset;
-		iter->iov_offset = offset;
-	} else {
+	if (offset >= bvec->bv_len) {
 		unsigned long seg_skip;
 
 		/* skip first vec */
 		offset -= bvec->bv_len;
 		seg_skip = 1 + (offset >> imu->folio_shift);
-
-		iter->bvec += seg_skip;
-		iter->nr_segs -= seg_skip;
-		iter->count -= bvec->bv_len + offset;
-		iter->iov_offset = offset & ((1UL << imu->folio_shift) - 1);
+		bvec += seg_skip;
+		offset &= (1UL << imu->folio_shift) - 1;
 	}
 
+	nr_segs = imu->nr_bvecs - (bvec - imu->bvec);
+	iov_iter_bvec(iter, ddir, bvec, nr_segs, len);
+	iter->iov_offset = offset;
 	return 0;
 }
 
Signature · 1 line
2.48.1
#4 lore raw

PATCH 4/4 io_uring/rsrc: send exact nr_segs for fixed buffer

In reply to Pavel Begunkov

From: Nitesh Shetty <[email protected]>

Sending exact nr_segs, avoids bio split check and processing in block layer, which takes around 5%[1] of overall CPU utilization.

In our setup, we see overall improvement of IOPS from 7.15M to 7.65M [2] and 5% less CPU utilization.

[1]

     3.52%  io_uring         [kernel.kallsyms]     [k] bio_split_rw_at
     1.42%  io_uring         [kernel.kallsyms]     [k] bio_split_rw
     0.62%  io_uring         [kernel.kallsyms]     [k] bio_submit_split

[2]
sudo taskset -c 0,1 ./t/io_uring -b512 -d128 -c32 -s32 -p1 -F1 -B1 -n2 -r4 /dev/nvme0n1 /dev/nvme1n1

[Pavel: fixed for kbuf, rebased and reworked on top of cleanups]
---
 io_uring/rsrc.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
io_uring/rsrc.c +4 -2
index 5cf854318b1d..4099b8225670 100644
@@ -1037,6 +1037,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 			   u64 buf_addr, size_t len)
 {
 	const struct bio_vec *bvec;
+	size_t folio_mask;
 	unsigned nr_segs;
 	size_t offset;
 	int ret;
@@ -1067,6 +1068,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 	 * 2) all bvecs are the same in size, except potentially the
 	 *    first and last bvec
 	 */
+	folio_mask = (1UL << imu->folio_shift) - 1;
 	bvec = imu->bvec;
 	if (offset >= bvec->bv_len) {
 		unsigned long seg_skip;
@@ -1075,10 +1077,10 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 		offset -= bvec->bv_len;
 		seg_skip = 1 + (offset >> imu->folio_shift);
 		bvec += seg_skip;
-		offset &= (1UL << imu->folio_shift) - 1;
+		offset &= folio_mask;
 	}
 
-	nr_segs = imu->nr_bvecs - (bvec - imu->bvec);
+	nr_segs = (offset + len + folio_mask) >> imu->folio_shift;
 	iov_iter_bvec(iter, ddir, bvec, nr_segs, len);
 	iter->iov_offset = offset;
 	return 0;
Signature · 1 line
2.48.1
#5 lore raw

Re: [PATCH 4/4] io_uring/rsrc: send exact nr_segs for fixed buffer

Quoted 1 line · Pavel Begunkov

From: Nitesh Shetty <[email protected]>

...

Quoted 21 lines
io_uring/rsrc.c +2 -0
index 5cf854318b1d..4099b8225670 100644
@@ -1037,6 +1037,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
  			   u64 buf_addr, size_t len)
  {
  	const struct bio_vec *bvec;
+	size_t folio_mask;
  	unsigned nr_segs;
  	size_t offset;
  	int ret;
@@ -1067,6 +1068,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
  	 * 2) all bvecs are the same in size, except potentially the
  	 *    first and last bvec
  	 */
+	folio_mask = (1UL << imu->folio_shift) - 1;
  	bvec = imu->bvec;
  	if (offset >= bvec->bv_len) {
  		unsigned long seg_skip;

io_uring/rsrc.c static int io_import_fixed(int ddir, struct iov_iter *iter, · line 1083

Quoted 9 lines
io_uring/rsrc.c +2 -2
@@ -1075,10 +1077,10 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
  		offset -= bvec->bv_len;
  		seg_skip = 1 + (offset >> imu->folio_shift);
  		bvec += seg_skip;
-		offset &= (1UL << imu->folio_shift) - 1;
+		offset &= folio_mask;
  	}
 
-	nr_segs = imu->nr_bvecs - (bvec - imu->bvec);
+	nr_segs = (offset + len + folio_mask) >> imu->folio_shift;

Nitesh, let me know if you're happy with this version.

Quoted 3 lines
io_uring/rsrc.c +0 -0
  	iov_iter_bvec(iter, ddir, bvec, nr_segs, len);
  	iter->iov_offset = offset;
  	return 0;
Signature · 1 line
Pavel Begunkov
#6 lore raw

Re: [PATCH 4/4] io_uring/rsrc: send exact nr_segs for fixed buffer

Quoted 26 lines · Pavel Begunkov
Quoted 1 line · Pavel Begunkov

From: Nitesh Shetty <[email protected]>

...

Quoted 23 lines
io_uring/rsrc.c +2 -0
index 5cf854318b1d..4099b8225670 100644
@@ -1037,6 +1037,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 			   u64 buf_addr, size_t len)
 {
 	const struct bio_vec *bvec;
+	size_t folio_mask;
 	unsigned nr_segs;
 	size_t offset;
 	int ret;
@@ -1067,6 +1068,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 	 * 2) all bvecs are the same in size, except potentially the
 	 *    first and last bvec
 	 */
+	folio_mask = (1UL << imu->folio_shift) - 1;
 	bvec = imu->bvec;
 	if (offset >= bvec->bv_len) {
 		unsigned long seg_skip;
@@ -1075,10 +1077,10 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 		offset -= bvec->bv_len;
 		seg_skip = 1 + (offset >> imu->folio_shift);
Quoted 9 lines

io_uring/rsrc.c static int io_import_fixed(int ddir, struct iov_iter *iter, · line 1083

Quoted 6 lines
io_uring/rsrc.c +2 -2
 		bvec += seg_skip;
-		offset &= (1UL << imu->folio_shift) - 1;
+		offset &= folio_mask;
 	}
-	nr_segs = imu->nr_bvecs - (bvec - imu->bvec);
+	nr_segs = (offset + len + folio_mask) >> imu->folio_shift;

Nitesh, let me know if you're happy with this version.

This looks great to me, I tested this series and see the improvement in IOPS from 7.15 to 7.65M here.

Thanks,
Nitesh Shetty

maintainer Applied
#13 lore raw
In reply to Pavel Begunkov
Quoted 10 lines · Pavel Begunkov

io_import_fixed() cleanups topped with the nr segs optimisation patch from Nitesh. Doesn't have the kbuf part, assuming Jens will add it on top.

Based on io_uring-6.15

Nitesh Shetty (1):
io_uring/rsrc: send exact nr_segs for fixed buffer

[...]

Applied, thanks!

[1/4] io_uring/rsrc: don't skip offset calculation

      commit: 1ac571288822253db32196c49f240739148417e3
[2/4] io_uring/rsrc: separate kbuf offset adjustments
      commit: 50169d07548441e3033b9bbaa06e573e7224f140
[3/4] io_uring/rsrc: refactor io_import_fixed
      commit: 59852ebad954c8a3ac8b746930c2ea60febe797a
[4/4] io_uring/rsrc: send exact nr_segs for fixed buffer
      commit: c5a173d9864a597288c6acd091315cfc3b32ca1c

Best regards,

Signature · 1 line
Jens Axboe
#7 lore raw

Re: [PATCH 4/4] io_uring/rsrc: send exact nr_segs for fixed buffer

In reply to Nitesh Shetty
Quoted 36 lines · Nitesh Shetty
Quoted 26 lines · Pavel Begunkov
Quoted 1 line · Pavel Begunkov

From: Nitesh Shetty <[email protected]>

...

Quoted 23 lines
io_uring/rsrc.c +2 -0
index 5cf854318b1d..4099b8225670 100644
@@ -1037,6 +1037,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 			   u64 buf_addr, size_t len)
 {
 	const struct bio_vec *bvec;
+	size_t folio_mask;
 	unsigned nr_segs;
 	size_t offset;
 	int ret;
@@ -1067,6 +1068,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 	 * 2) all bvecs are the same in size, except potentially the
 	 *    first and last bvec
 	 */
+	folio_mask = (1UL << imu->folio_shift) - 1;
 	bvec = imu->bvec;
 	if (offset >= bvec->bv_len) {
 		unsigned long seg_skip;
@@ -1075,10 +1077,10 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
 		offset -= bvec->bv_len;
 		seg_skip = 1 + (offset >> imu->folio_shift);
Quoted 9 lines

io_uring/rsrc.c static int io_import_fixed(int ddir, struct iov_iter *iter, · line 1083

Quoted 6 lines
io_uring/rsrc.c +2 -2
 		bvec += seg_skip;
-		offset &= (1UL << imu->folio_shift) - 1;
+		offset &= folio_mask;
 	}
-	nr_segs = imu->nr_bvecs - (bvec - imu->bvec);
+	nr_segs = (offset + len + folio_mask) >> imu->folio_shift;

Nitesh, let me know if you're happy with this version.

Quoted 3 lines

This looks great to me, I tested this series and see the improvement in IOPS from 7.15 to 7.65M here.

There is corner case where this might not work,
This happens when there is a first bvec has non zero offset. Let's say bv_offset = 256, len = 512, iov_offset = 3584 (512*7, 8th IO), here we expect IO to have 2 segments with present codebase, but this patch set produces 1 segment.

So having a fix like this solves the issue,
+ nr_segs = (offset + len + bvec->bv_offset + folio_mask) >> imu->folio_shift;

Note:
I am investigating whether this is a valid case or not, because having a 512 byte IO with 256 byte alignment feel odd. So have sent one patch for that as well[1]. If that patch[1] is upstreamed then above case is taken care of, so we can use this series as it is.

Thanks,
Nitesh

[1]
https://lore.kernel.org/all/[email protected]/

#8 lore raw

Re: [PATCH 4/4] io_uring/rsrc: send exact nr_segs for fixed buffer

Quoted 41 lines · Nitesh Shetty
Quoted 36 lines · Nitesh Shetty
Quoted 26 lines · Pavel Begunkov
Quoted 1 line · Pavel Begunkov

From: Nitesh Shetty <[email protected]>

...

Quoted 23 lines
io_uring/rsrc.c +2 -0
index 5cf854318b1d..4099b8225670 100644
@@ -1037,6 +1037,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
                u64 buf_addr, size_t len)
 {
     const struct bio_vec *bvec;
+    size_t folio_mask;
     unsigned nr_segs;
     size_t offset;
     int ret;
@@ -1067,6 +1068,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
      * 2) all bvecs are the same in size, except potentially the
      *    first and last bvec
      */
+    folio_mask = (1UL << imu->folio_shift) - 1;
     bvec = imu->bvec;
     if (offset >= bvec->bv_len) {
         unsigned long seg_skip;
@@ -1075,10 +1077,10 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
         offset -= bvec->bv_len;
         seg_skip = 1 + (offset >> imu->folio_shift);
Quoted 9 lines

io_uring/rsrc.c static int io_import_fixed(int ddir, struct iov_iter *iter, · line 1083

Quoted 6 lines
io_uring/rsrc.c +2 -2
         bvec += seg_skip;
-        offset &= (1UL << imu->folio_shift) - 1;
+        offset &= folio_mask;
     }
-    nr_segs = imu->nr_bvecs - (bvec - imu->bvec);
+    nr_segs = (offset + len + folio_mask) >> imu->folio_shift;

Nitesh, let me know if you're happy with this version.

Quoted 3 lines

This looks great to me, I tested this series and see the improvement in IOPS from 7.15 to 7.65M here.

Quoted 8 lines

There is corner case where this might not work,
This happens when there is a first bvec has non zero offset. Let's say bv_offset = 256, len = 512, iov_offset = 3584 (512*7, 8th IO), here we expect IO to have 2 segments with present codebase, but this patch set produces 1 segment.

So having a fix like this solves the issue,
+    nr_segs = (offset + len + bvec->bv_offset + folio_mask) >> imu->folio_shift;

Ah yes, looks like the right fix up. We can make it nicer, but that's for later. It'd also be great to have a test for it.

Quoted 3 lines

Note:
I am investigating whether this is a valid case or not, because having a 512 byte IO with 256 byte alignment feel odd. So have sent one patch for

Block might filter it out, but for example net/ doesn't care, fs as well. IIUC what you mean, either way we definitely should correct that.

Quoted 10 lines

that as well[1]. If that patch[1] is upstreamed then above case is taken care of, so we can use this series as it is.

Thanks,
Nitesh

[1]
https://lore.kernel.org/all/[email protected]/

Signature · 1 line
Pavel Begunkov
maintainer
#9 lore raw

Re: [PATCH 4/4] io_uring/rsrc: send exact nr_segs for fixed buffer

Quoted 55 lines · Pavel Begunkov
Quoted 41 lines · Nitesh Shetty
Quoted 36 lines · Nitesh Shetty
Quoted 26 lines · Pavel Begunkov
Quoted 1 line · Pavel Begunkov

From: Nitesh Shetty <[email protected]>

...

Quoted 23 lines
io_uring/rsrc.c +2 -0
index 5cf854318b1d..4099b8225670 100644
@@ -1037,6 +1037,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
               u64 buf_addr, size_t len)
 {
    const struct bio_vec *bvec;
+    size_t folio_mask;
    unsigned nr_segs;
    size_t offset;
    int ret;
@@ -1067,6 +1068,7 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
     * 2) all bvecs are the same in size, except potentially the
     *    first and last bvec
     */
+    folio_mask = (1UL << imu->folio_shift) - 1;
    bvec = imu->bvec;
    if (offset >= bvec->bv_len) {
        unsigned long seg_skip;
@@ -1075,10 +1077,10 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
        offset -= bvec->bv_len;
        seg_skip = 1 + (offset >> imu->folio_shift);
Quoted 9 lines

io_uring/rsrc.c static int io_import_fixed(int ddir, struct iov_iter *iter, · line 1083

Quoted 6 lines
io_uring/rsrc.c +2 -2
        bvec += seg_skip;
-        offset &= (1UL << imu->folio_shift) - 1;
+        offset &= folio_mask;
    }
-    nr_segs = imu->nr_bvecs - (bvec - imu->bvec);
+    nr_segs = (offset + len + folio_mask) >> imu->folio_shift;

Nitesh, let me know if you're happy with this version.

Quoted 3 lines

This looks great to me, I tested this series and see the improvement in IOPS from 7.15 to 7.65M here.

Quoted 8 lines

There is corner case where this might not work,
This happens when there is a first bvec has non zero offset. Let's say bv_offset = 256, len = 512, iov_offset = 3584 (512*7, 8th IO), here we expect IO to have 2 segments with present codebase, but this patch set produces 1 segment.

So having a fix like this solves the issue,
+ nr_segs = (offset + len + bvec->bv_offset + folio_mask) >> imu->folio_shift;

Ah yes, looks like the right fix up. We can make it nicer, but that's for later. It'd also be great to have a test for it.

Quoted 7 lines
Quoted 3 lines

Note:
I am investigating whether this is a valid case or not, because having a 512 byte IO with 256 byte alignment feel odd. So have sent one patch for

Block might filter it out, but for example net/ doesn't care, fs as well. IIUC what you mean, either way we definitely should correct that.

I just tested it, and yes it certainly blows up... Can also confirm that the corrected nr_segs calculation does the right thing, doesn't end up underestimating the segment count by 1 in that case.

I'll turn the test case into something we can add to liburing, and fold in that change.

Signature · 1 line
Jens Axboe
maintainer
#10 lore raw

Re: [PATCH 4/4] io_uring/rsrc: send exact nr_segs for fixed buffer

Quoted 2 lines · Jens Axboe

I'll turn the test case into something we can add to liburing, and fold in that change.

Here's what I tested, fwiw, and it reliably blows up pre the fixup. I'll turn it into a normal test case, and then folks can add more invariants to this one if they wish.

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#include <liburing.h>

static struct iovec vec;

static int read_it(struct io_uring *ring, int fd, int len, int off)

{
	struct io_uring_sqe *sqe;
	struct io_uring_cqe *cqe;
	int ret;

	sqe = io_uring_get_sqe(ring);
	io_uring_prep_read_fixed(sqe, fd, vec.iov_base + off, len, 0, 0);
	sqe->user_data = 1;

	io_uring_submit(ring);

	ret = io_uring_wait_cqe(ring, &cqe);
	if (ret) {
		fprintf(stderr, "wait %d\n", ret);
		return 1;
	}
	if (cqe->res < 0) {
		fprintf(stderr, "cqe res %s\n", strerror(-cqe->res));
		return 1;
	}
	if (cqe->res != len) {
		fprintf(stderr, "Bad read amount: %d\n", cqe->res);
		return 1;
	}
	io_uring_cqe_seen(ring, cqe);
	return 0;
}

static int test(struct io_uring *ring, int fd, int vec_off)

{
	struct iovec v = vec;
	int ret;

	v.iov_base += vec_off;
	v.iov_len -= vec_off;
	ret = io_uring_register_buffers(ring, &v, 1);
	if (ret) {
		fprintf(stderr, "Vec register: %d\n", ret);
		return 1;
	}

	ret = read_it(ring, fd, 4096, vec_off);
	if (ret) {
		fprintf(stderr, "4096 0 failed\n");
		return 1;
	}
	ret = read_it(ring, fd, 8192, 4096);
	if (ret) {
		fprintf(stderr, "8192 4096 failed\n");
		return 1;
	}
	ret = read_it(ring, fd, 4096, 4096);
	if (ret) {
		fprintf(stderr, "4096 4096 failed\n");
		return 1;
	}
	
	io_uring_unregister_buffers(ring);
	return 0;
}

int main(int argc, char *argv[])

{
	struct io_uring ring;
	int fd, ret;

	if (argc == 1) {
		printf("%s: <file>\n", argv[0]);
		return 1;
	}

	fd = open(argv[1], O_RDONLY | O_DIRECT);
	if (fd < 0) {
		perror("open");
		return 1;
	}

	posix_memalign(&vec.iov_base, 4096, 512*1024);
	vec.iov_len = 512*1024;

	io_uring_queue_init(32, &ring, 0);

	ret = test(&ring, fd, 0);
	if (ret) {
		fprintf(stderr, "test 0 failed\n");
		return 1;
	}

	ret = test(&ring, fd, 512);
	if (ret) {
		fprintf(stderr, "test 512 failed\n");
		return 1;
	}

	close(fd);
	io_uring_queue_exit(&ring);
	return 0;
}
Signature · 1 line
Jens Axboe
#11 lore raw

Re: [PATCH 4/4] io_uring/rsrc: send exact nr_segs for fixed buffer

Quoted 7 lines · Jens Axboe
Quoted 2 lines · Jens Axboe

I'll turn the test case into something we can add to liburing, and fold in that change.

Here's what I tested, fwiw, and it reliably blows up pre the fixup. I'll turn it into a normal test case, and then folks can add more invariants to this one if they wish.

Awesome, thanks!

Signature · 1 line
Pavel Begunkov
maintainer
#12 lore raw

Re: [PATCH 4/4] io_uring/rsrc: send exact nr_segs for fixed buffer

Quoted 10 lines · Pavel Begunkov
Quoted 7 lines · Jens Axboe
Quoted 2 lines · Jens Axboe

I'll turn the test case into something we can add to liburing, and fold in that change.

Here's what I tested, fwiw, and it reliably blows up pre the fixup. I'll turn it into a normal test case, and then folks can add more invariants to this one if they wish.

Awesome, thanks!

Done:

https://git.kernel.dk/cgit/liburing/commit/?id=d700f83631a780182c3a3f1315e926d5547acedb

Signature · 1 line
Jens Axboe
#14 lore raw
In reply to Pavel Begunkov
Quoted 9 lines · Pavel Begunkov

io_import_fixed() cleanups topped with the nr segs optimisation patch from Nitesh. Doesn't have the kbuf part, assuming Jens will add it on top.

Based on io_uring-6.15

Nitesh Shetty (1):
io_uring/rsrc: send exact nr_segs for fixed buffer

Quoted 11 lines

Pavel Begunkov (3):
io_uring/rsrc: don't skip offset calculation
io_uring/rsrc: separate kbuf offset adjustments
io_uring/rsrc: refactor io_import_fixed

io_uring/rsrc.c | 74 ++++++++++++++++++++-----------------------------

1 file changed, 30 insertions(+), 44 deletions(-)
Signature · 1 line
2.48.1

Reviewed and Tested by: Nitesh Shetty <[email protected]>