Skip to content

uring_cmd copy avoidance

Jens Axboe io-uring 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest4 patches 5 repliesIn review
  2. v34 patches 5 repliesSuperseded
  3. v2 RFC4 patches 13 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v1 RFC3 patches1 replySuperseded

Changes in v4 from the cover letter

- Add REQ_F_SQE_COPY flag, so that ->sqe_copy() can only be called once and so that the IO_URING_F_INLINE WARN_ON_ONCE()/-EFAULT check actually works.
- Call io_req_sqe_copy() for any link member
- Update patch 4 commit message
- Rebase on 6.16-rc1

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4io_uring: add IO_URING_F_INLINE issue flaginclude/linux/io_uring_types.h io_uring/io_uring.c+9 -51 reply
2/4io_uring: add struct io_cold_def->sqe_copy() methodinclude/linux/io_uring_types.h io_uring/io_uring.c io_uring/opdef.h+29 -22 replies
3/4io_uring/uring_cmd: get rid of io_uring_cmd_prep_setup()io_uring/uring_cmd.c+13 -210 replies
4/4io_uring/uring_cmd: implement ->sqe_copy() to avoid unnecessary copiesio_uring/opdef.c io_uring/uring_cmd.c io_uring/uring_cmd.h+14 -92 replies

Reviews trailers per patch

All 4 patches have a Reviewed-by or Acked-by.

PatchCaleb Sander MateosAnuj Gupta
1/4R-b
2/4R-b
3/4R-b R-b
4/4R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v4 thread

Cover letter [PATCHSET v4 0/4] uring_cmd copy avoidance

Hi,

Currently uring_cmd unconditionally copies the SQE at prep time, as it has no other choice - the SQE data must remain stable after submit. This can lead to excessive memory bandwidth being used for that copy, as passthrough will often use 128b SQEs, and efficiency concerns as those copies will potentially use quite a lot of CPU cycles as well.

As a quick test, running the current -git kernel on a box with 23 NVMe drives doing passthrough IO, memcpy() is the highest cycle user at 9.05%, which is all off the uring_cmd prep path. The test case is a 512b random read, which runs at 91-92M IOPS.

With these patches, memcpy() is gone from the profiles, and it runs at 98-99M IOPS, or about 7-8% faster.

Before:

IOPS=91.12M, BW=44.49GiB/s, IOS/call=32/32
IOPS=91.16M, BW=44.51GiB/s, IOS/call=32/32
IOPS=91.18M, BW=44.52GiB/s, IOS/call=31/32
IOPS=91.92M, BW=44.88GiB/s, IOS/call=32/32
IOPS=91.88M, BW=44.86GiB/s, IOS/call=32/32
IOPS=91.82M, BW=44.83GiB/s, IOS/call=32/31
IOPS=91.52M, BW=44.69GiB/s, IOS/call=32/32

with the top perf report -g --no-children being:

+ 9.07% io_uring [kernel.kallsyms] [k] memcpy

and after:

# bash run-peak-pass.sh
[...]
IOPS=99.30M, BW=48.49GiB/s, IOS/call=32/32
IOPS=99.27M, BW=48.47GiB/s, IOS/call=31/32
IOPS=99.60M, BW=48.63GiB/s, IOS/call=32/32
IOPS=99.68M, BW=48.67GiB/s, IOS/call=32/31
IOPS=99.80M, BW=48.73GiB/s, IOS/call=31/32
IOPS=99.84M, BW=48.75GiB/s, IOS/call=32/32

with memcpy not even in profiles. If you do the actual math of 100M requests per second, and 128b of copying per IOP, then it's almost 12GB/sec of reduced memory bandwidth.

Even for lower IOPS production testing, Caleb reports that memcpy() overhead is in the realm of 1.1% of CPU time.

I think this approach is saner, and in fact it can be extended to reduce over-eager copies in other spots. For now I just did uring_cmd, and verified that the memcpy's are still gone from my test.

Can also be found here:

https://git.kernel.dk/cgit/linux/log/?h=uring_cmd.2

 include/linux/io_uring_types.h |  5 ++++
 io_uring/io_uring.c            | 39 ++++++++++++++++++++++++------
 io_uring/opdef.c               |  1 +
 io_uring/opdef.h               |  1 +
 io_uring/uring_cmd.c           | 43 +++++++++++++++-------------------
 io_uring/uring_cmd.h           |  1 +
 6 files changed, 59 insertions(+), 31 deletions(-)

Since v3:
- Add REQ_F_SQE_COPY flag, so that ->sqe_copy() can only be called once and so that the IO_URING_F_INLINE WARN_ON_ONCE()/-EFAULT check actually works.
- Call io_req_sqe_copy() for any link member
- Update patch 4 commit message
- Rebase on 6.16-rc1

Signature · 1 line
Jens Axboe