ublk: allow off-daemon zero-copy buffer registration
Status from this message: Applied, thanks!
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v2 Latest14 patches 11 repliesApplied
- v18 patches 17 repliesSuperseded
Changes in v2 from the cover letter
- Add 2 fix patches
- Optimize buffer unregistration too
- Cache-align ublk_io
- Restore check for zero copy support in ublk_unregister_io_buf()
- Check for registered buffer count overflow
Patches in v2 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/14 | ublk: use vmalloc for ublk_device's __queuesdrivers/ | +2 | 1 reply | |
| 2/14 | ublk: remove struct ublk_rq_datadrivers/ | +71 | 1 reply | |
| 3/14 | ublk: check cmd_op firstdrivers/ | +6 | 0 replies | |
| 4/14 | ublk: handle UBLK_IO_FETCH_REQ earlierdrivers/ | +12 | 0 replies | |
| 5/14 | ublk: remove task variable from __ublk_ch_uring_cmd()drivers/ | +1 | 0 replies | |
| 6/14 | ublk: consolidate UBLK_IO_FLAG_{ACTIVE,OWNED_BY_SRV} checksdrivers/ | +1 | 0 replies | |
| 7/14 | ublk: move ublk_prep_cancel() to case UBLK_IO_COMMIT_AND_FETCH_REQdrivers/ | +3 | 0 replies | |
| 8/14 | ublk: don't take ublk_queue in ublk_unregister_io_buf()drivers/ | +3 | 1 reply | |
| 9/14 | ublk: allow UBLK_IO_(UN)REGISTER_IO_BUF on any taskdrivers/ include/ | +32 | 1 reply | |
| 10/14 | ublk: return early if blk_should_fake_timeout()drivers/ | +3 | 1 reply | |
| 11/14 | ublk: optimize UBLK_IO_REGISTER_IO_BUF on daemon taskdrivers/ | +61 | 1 reply | |
| 12/14 | ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon taskdrivers/ | +8 | 1 reply | |
| 13/14 | ublk: remove ubq checks from ublk_{get,put}_req_ref()drivers/ | +11 | 1 reply | |
| 14/14 | ublk: cache-align struct ublk_iodrivers/ | +1 | 1 reply |
Reviews trailers per patch
All 14 patches have a Reviewed-by or Acked-by.
| Patch | Ming Lei |
|---|---|
| 1/14 | R-b |
| 2/14 | R-b |
| 3/14 | R-b |
| 4/14 | R-b |
| 5/14 | R-b |
| 6/14 | R-b |
| 7/14 | R-b |
| 8/14 | R-b |
| 9/14 | R-b |
| 10/14 | R-b |
| 11/14 | R-b |
| 12/14 | R-b |
| 13/14 | R-b |
| 14/14 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH v2 00/14] ublk: allow off-daemon zero-copy buffer registration
Currently ublk zero-copy requires ublk request buffers to be registered and unregistered by the ublk I/O's daemon task. However, as currently implemented, there is no reason for this restriction. Registration looks up the request via the ublk device's tagset rather than the daemon-local ublk_io structure and takes an atomic reference to prevent racing with dispatch or completion of the request. Ming has expressed interest in relaxing this restriction[1] so the ublk server can offload the I/O operation that uses the zero-copy buffer to another thread.
Additionally, optimize the buffer registration for the common case where the buffer is registered and unregistered by the daemon task. Skip the expensive atomic reference count increment and decrement and the several pointer dereferences involved in looking up the request on the tagset. On our ublk server threads, this results in a 24% decrease in the CPU time spent in the ublk functions on a 4K zero-copy read workload, from 8.75% to 6.68%.
Two preliminary fixes are included as well:
- Move the ublk request reference count from ublk_rq_data (a tail allocation of struct request) to ublk_io to prevent a use after free if the struct request is freed concurrently with taking a reference.
- Don't allocate physically contiguous memory for __queues, which can be very large. Doubling the size of struct ublk_io caused ENOMEM errors when adding a ublk device before this change.
[1]: https://lore.kernel.org/linux-block/aAmYJxaV1-yWEMRo@fedora/
v2:
- Add 2 fix patches
- Optimize buffer unregistration too
- Cache-align ublk_io
- Restore check for zero copy support in ublk_unregister_io_buf()
- Check for registered buffer count overflow
Caleb Sander Mateos (14):
ublk: use vmalloc for ublk_device's __queues
ublk: remove struct ublk_rq_data
ublk: check cmd_op first
ublk: handle UBLK_IO_FETCH_REQ earlier
ublk: remove task variable from __ublk_ch_uring_cmd()
ublk: consolidate UBLK_IO_FLAG_{ACTIVE,OWNED_BY_SRV} checks
ublk: move ublk_prep_cancel() to case UBLK_IO_COMMIT_AND_FETCH_REQ
ublk: don't take ublk_queue in ublk_unregister_io_buf()
ublk: allow UBLK_IO_(UN)REGISTER_IO_BUF on any task
ublk: return early if blk_should_fake_timeout()
ublk: optimize UBLK_IO_REGISTER_IO_BUF on daemon task
ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon task
ublk: remove ubq checks from ublk_{get,put}_req_ref()
ublk: cache-align struct ublk_io
drivers/block/ublk_drv.c | 283 +++++++++++++++++++++------------- include/uapi/linux/ublk_cmd.h | 10 ++ 2 files changed, 187 insertions(+), 106 deletions(-)
Signature · 1 line
2.45.2