Skip to content

ublk: allow off-daemon zero-copy buffer registration

Caleb Sander Mateos linux-block 2 versions – Applied

Status from this message: Applied, thanks!

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest14 patches 11 repliesApplied
  2. v18 patches 17 repliesSuperseded

Changes in v2 from the cover letter

- Add 2 fix patches
- Optimize buffer unregistration too
- Cache-align ublk_io
- Restore check for zero copy support in ublk_unregister_io_buf()
- Check for registered buffer count overflow

Patches in v2 Applied

#Subject and filesSizeRepliesApplied
1/14ublk: use vmalloc for ublk_device's __queuesdrivers/block/ublk_drv.c+2 -21 reply
2/14ublk: remove struct ublk_rq_datadrivers/block/ublk_drv.c+71 -601 reply
3/14ublk: check cmd_op firstdrivers/block/ublk_drv.c+6 -60 replies
4/14ublk: handle UBLK_IO_FETCH_REQ earlierdrivers/block/ublk_drv.c+12 -90 replies
5/14ublk: remove task variable from __ublk_ch_uring_cmd()drivers/block/ublk_drv.c+1 -30 replies
6/14ublk: consolidate UBLK_IO_FLAG_{ACTIVE,OWNED_BY_SRV} checksdrivers/block/ublk_drv.c+1 -40 replies
7/14ublk: move ublk_prep_cancel() to case UBLK_IO_COMMIT_AND_FETCH_REQdrivers/block/ublk_drv.c+3 -50 replies
8/14ublk: don't take ublk_queue in ublk_unregister_io_buf()drivers/block/ublk_drv.c+3 -31 reply
9/14ublk: allow UBLK_IO_(UN)REGISTER_IO_BUF on any taskdrivers/block/ublk_drv.c include/uapi/linux/ublk_cmd.h+32 -51 reply
10/14ublk: return early if blk_should_fake_timeout()drivers/block/ublk_drv.c+3 -21 reply
11/14ublk: optimize UBLK_IO_REGISTER_IO_BUF on daemon taskdrivers/block/ublk_drv.c+61 -91 reply
12/14ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon taskdrivers/block/ublk_drv.c+8 -11 reply
13/14ublk: remove ubq checks from ublk_{get,put}_req_ref()drivers/block/ublk_drv.c+11 -241 reply
14/14ublk: cache-align struct ublk_iodrivers/block/ublk_drv.c+1 -11 reply

Reviews trailers per patch

All 14 patches have a Reviewed-by or Acked-by.

PatchMing Lei
1/14R-b
2/14R-b
3/14R-b
4/14R-b
5/14R-b
6/14R-b
7/14R-b
8/14R-b
9/14R-b
10/14R-b
11/14R-b
12/14R-b
13/14R-b
14/14R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 00/14] ublk: allow off-daemon zero-copy buffer registration

Currently ublk zero-copy requires ublk request buffers to be registered and unregistered by the ublk I/O's daemon task. However, as currently implemented, there is no reason for this restriction. Registration looks up the request via the ublk device's tagset rather than the daemon-local ublk_io structure and takes an atomic reference to prevent racing with dispatch or completion of the request. Ming has expressed interest in relaxing this restriction[1] so the ublk server can offload the I/O operation that uses the zero-copy buffer to another thread.

Additionally, optimize the buffer registration for the common case where the buffer is registered and unregistered by the daemon task. Skip the expensive atomic reference count increment and decrement and the several pointer dereferences involved in looking up the request on the tagset. On our ublk server threads, this results in a 24% decrease in the CPU time spent in the ublk functions on a 4K zero-copy read workload, from 8.75% to 6.68%.

Two preliminary fixes are included as well:
- Move the ublk request reference count from ublk_rq_data (a tail allocation of struct request) to ublk_io to prevent a use after free if the struct request is freed concurrently with taking a reference.
- Don't allocate physically contiguous memory for __queues, which can be very large. Doubling the size of struct ublk_io caused ENOMEM errors when adding a ublk device before this change.

[1]: https://lore.kernel.org/linux-block/aAmYJxaV1-yWEMRo@fedora/

v2:
- Add 2 fix patches
- Optimize buffer unregistration too
- Cache-align ublk_io
- Restore check for zero copy support in ublk_unregister_io_buf()
- Check for registered buffer count overflow

Caleb Sander Mateos (14):
ublk: use vmalloc for ublk_device's __queues
ublk: remove struct ublk_rq_data
ublk: check cmd_op first
ublk: handle UBLK_IO_FETCH_REQ earlier
ublk: remove task variable from __ublk_ch_uring_cmd()
ublk: consolidate UBLK_IO_FLAG_{ACTIVE,OWNED_BY_SRV} checks
ublk: move ublk_prep_cancel() to case UBLK_IO_COMMIT_AND_FETCH_REQ
ublk: don't take ublk_queue in ublk_unregister_io_buf()
ublk: allow UBLK_IO_(UN)REGISTER_IO_BUF on any task
ublk: return early if blk_should_fake_timeout()
ublk: optimize UBLK_IO_REGISTER_IO_BUF on daemon task
ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon task
ublk: remove ubq checks from ublk_{get,put}_req_ref()
ublk: cache-align struct ublk_io

 drivers/block/ublk_drv.c      | 283 +++++++++++++++++++++-------------
 include/uapi/linux/ublk_cmd.h |  10 ++
 2 files changed, 187 insertions(+), 106 deletions(-)
Signature · 1 line
2.45.2