ublk: cleanup & improvement & zc follow-up
Subsystems (4)
v2 cover ublk: cleanup & improvement & zc follow-up
Hello Jens,
The 1st two fixes ublk_abort_requests(), and adds comment on handling ubq->canceling.
The 3rd ~ 5th patches are cleanup.
The 6th and 7th are zc-followup.
The 8th & 9th patch implements ->queue_rqs() and improves IOPS by > 10%.
The last two patches are self-test for ->queue_rqs() & segment parameter change.
V2:
- use io_uring_cmd_to_pdu() (Uday)
- improve zc document (Caleb)
- consolidate segment parameter interface (Caleb)
- add reviewed-by
- add one fix and comment on ubq->canceling
- fix one hang bug in V2
Ming Lei (11):
ublk: make sure ubq->canceling is set when queue is frozen
ublk: comment on ubq->canceling handling in ublk_queue_rq()
ublk: remove two unused fields from 'struct ublk_queue'
ublk: add helper of ublk_need_map_io()
ublk: call io_uring_cmd_to_pdu to get uring_cmd pdu
ublk: add segment parameter
ublk: document zero copy feature
ublk: implement ->queue_rqs()
ublk: rename ublk_rq_task_work_cb as ublk_cmd_tw_cb
selftests: ublk: add more tests for covering MQ
selftests: ublk: add test for checking zero copy related parameter
Documentation/block/ublk.rst | 35 ++- drivers/block/ublk_drv.c | 214 ++++++++++++++---- include/uapi/linux/ublk_cmd.h | 25 ++ tools/testing/selftests/ublk/Makefile | 4 + tools/testing/selftests/ublk/null.c | 11 +- tools/testing/selftests/ublk/test_common.sh | 6 + .../testing/selftests/ublk/test_generic_02.sh | 44 ++++ .../testing/selftests/ublk/test_generic_03.sh | 28 +++ tools/testing/selftests/ublk/test_loop_01.sh | 14 +- tools/testing/selftests/ublk/test_loop_03.sh | 14 +- tools/testing/selftests/ublk/test_loop_05.sh | 28 +++ .../testing/selftests/ublk/test_stress_01.sh | 6 +- .../testing/selftests/ublk/test_stress_02.sh | 6 +- .../testing/selftests/ublk/test_stripe_01.sh | 14 +- .../testing/selftests/ublk/test_stripe_03.sh | 30 +++ 15 files changed, 397 insertions(+), 82 deletions(-) create mode 100755 tools/testing/selftests/ublk/test_generic_02.sh create mode 100755 tools/testing/selftests/ublk/test_generic_03.sh create mode 100755 tools/testing/selftests/ublk/test_loop_05.sh create mode 100755 tools/testing/selftests/ublk/test_stripe_03.sh
Signature · 1 line
2.47.0
PATCH v2 1/11 ublk: make sure ubq->canceling is set when queue is frozen
Now ublk driver depends on ubq->canceling for deciding if the request can be dispatched via uring_cmd & io_uring_cmd_complete_in_task().
Once ubq->canceling is set, the uring_cmd can be done via ublk_cancel_cmd() and io_uring_cmd_done().
So set ubq->canceling when queue is frozen, this way makes sure that the flag can be observed from ublk_queue_rq() reliably, and avoids use-after-free on uring_cmd.
--- drivers/block/ublk_drv.c | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-)
drivers/block/ublk_drv.c +29 -10
@@ -1446,17 +1446,27 @@ static void ublk_abort_queue(struct ublk_device *ub, struct ublk_queue *ubq) } } +/* Must be called when queue is frozen */ +static bool ublk_mark_queue_canceling (struct ublk_queue *ubq) +{ +bool canceled; + +spin_lock (&ubq->cancel_lock); + canceled = ubq->canceling; +if (!canceled) + ubq->canceling = true; +spin_unlock (&ubq->cancel_lock); + +return canceled; +} +static bool ublk_abort_requests (struct ublk_device *ub,struct ublk_queue *ubq) { +bool was_canceled = ubq->canceling;struct gendisk *disk; -spin_lock (&ubq->cancel_lock); -if (ubq->canceling) { -spin_unlock (&ubq->cancel_lock); +if (was_canceled)return false; - } - ubq->canceling = true; -spin_unlock (&ubq->cancel_lock);spin_lock (&ub->lock); disk = ub->ub_disk;
@@ -1468,14 +1478,23 @@ static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq)if (!disk)return false; -/* Now we are serialized with ublk_queue_rq() */ +/* +* Now we are serialized with ublk_queue_rq() +* +* Make sure that ubq->canceling is set when queue is frozen, +* because ublk_queue_rq() has to rely on this flag for avoiding to +* touch completed uring_cmd +*/ blk_mq_quiesce_queue (disk->queue); -/* abort queue is for making forward progress */ -ublk_abort_queue (ub, ubq); + was_canceled =ublk_mark_queue_canceling (ubq); +if (!was_canceled) { +/* abort queue is for making forward progress */ +ublk_abort_queue (ub, ubq); + }blk_mq_unquiesce_queue (disk->queue);put_device (disk_to_dev (disk)); -return true; +return !was_canceled; } static void ublk_cancel_cmd(struct ublk_queue *ubq, struct ublk_io *io,
Signature · 1 line
2.47.0
PATCH v2 2/11 ublk: comment on ubq->canceling handling in ublk_queue_rq()
In reply to Ming LeiIn ublk_queue_rq(), ubq->canceling has to be handled after ->fail_io and ->force_abort are dealt with, otherwise the request may not be failed when deleting disk.
Add comment on this usage.
--- drivers/block/ublk_drv.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/block/ublk_drv.c +5 -0
@@ -1310,6 +1310,11 @@ static blk_status_t ublk_queue_rq(struct blk_mq_hw_ctx *hctx,if (ublk_nosrv_should_queue_io (ubq) &&unlikely (ubq->force_abort))return BLK_STS_IOERR ; +/* +* ->canceling has to be handled after ->force_abort and ->fail_io +* is dealt with, otherwise this request may not be failed in case +* of recovery, and cause hang when deleting disk +*/ if (unlikely (ubq->canceling)) {__ublk_abort_rq (ubq, rq);return BLK_STS_OK ;
Signature · 1 line
2.47.0
PATCH v2 3/11 ublk: remove two unused fields from 'struct ublk_queue'
In reply to Ming LeiRemove two unused fields(io_addr & max_io_sz) from struct ublk_queue.
--- drivers/block/ublk_drv.c | 2 -- 1 file changed, 2 deletions(-)
drivers/block/ublk_drv.c +0 -2
@@ -143,8 +143,6 @@ struct ublk_queue {struct task_struct *ubq_daemon;char *io_cmd_buf; -unsigned long io_addr;/* mapped vm address */ -unsigned int max_io_sz;bool force_abort;bool timeout;bool canceling;
Signature · 1 line
2.47.0
PATCH v2 4/11 ublk: add helper of ublk_need_map_io()
In reply to Ming Leiublk_need_map_io() is more readable.
--- drivers/block/ublk_drv.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/block/ublk_drv.c +9 -4
@@ -596,6 +596,11 @@ static inline bool ublk_support_user_copy(const struct ublk_queue *ubq)return ubq->flags & (UBLK_F_USER_COPY |UBLK_F_SUPPORT_ZERO_COPY ); } +static inline bool ublk_need_map_io (const struct ublk_queue *ubq) +{ +return !ublk_support_user_copy (ubq); +} +static inline bool ublk_need_req_ref (const struct ublk_queue *ubq) {/*
@@ -923,7 +928,7 @@ static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req, {const unsigned int rq_bytes =blk_rq_bytes (req); -if (ublk_support_user_copy (ubq)) +if (!ublk_need_map_io (ubq))return rq_bytes;/*
@@ -947,7 +952,7 @@ static int ublk_unmap_io(const struct ublk_queue *ubq, {const unsigned int rq_bytes =blk_rq_bytes (req); -if (ublk_support_user_copy (ubq)) +if (!ublk_need_map_io (ubq))return rq_bytes;if (ublk_need_unmap_req (req)) {
@@ -1867,7 +1872,7 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd,if (io->flags &UBLK_IO_FLAG_OWNED_BY_SRV ) goto out; -if (!ublk_support_user_copy (ubq)) { +if (ublk_need_map_io (ubq)) {/* * FETCH_RQ has to provide IO buffer if NEED GET * DATA is not enabled
@@ -1889,7 +1894,7 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd,if (!(io->flags &UBLK_IO_FLAG_OWNED_BY_SRV )) goto out; -if (!ublk_support_user_copy (ubq)) { +if (ublk_need_map_io (ubq)) {/* * COMMIT_AND_FETCH_REQ has to provide IO buffer if * NEED GET DATA is not enabled or it is Read IO.
Signature · 1 line
2.47.0
PATCH v2 5/11 ublk: call io_uring_cmd_to_pdu to get uring_cmd pdu
In reply to Ming LeiCall io_uring_cmd_to_pdu() to get uring_cmd pdu, and one big benefit is the automatic pdu size build check.
--- drivers/block/ublk_drv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/ublk_drv.c +1 -1
@@ -1040,7 +1040,7 @@ static blk_status_t ublk_setup_iod(struct ublk_queue *ubq, struct request *req)static inline struct ublk_uring_cmd_pdu *ublk_get_uring_cmd_pdu (struct io_uring_cmd *ioucmd) { -return (struct ublk_uring_cmd_pdu *)&ioucmd->pdu; + return io_uring_cmd_to_pdu(ioucmd, struct ublk_uring_cmd_pdu); }static inline bool ubq_daemon_is_dying (struct ublk_queue *ubq)
Signature · 1 line
2.47.0
Re: [PATCH V2 05/11] ublk: call io_uring_cmd_to_pdu to get uring_cmd pdu
Quoted 6 lines · Ming Lei
Call
io_uring_cmd_to_pdu()to geturing_cmdpdu, and one big benefit is the automatic pdu size build check.Suggested-by Uday Shankar <[email protected]>Signed-off-by Ming Lei <[email protected]>
PATCH v2 6/11 ublk: add segment parameter
In reply to Ming LeiIO split is usually bad in io_uring world, since -EAGAIN is caused and IO handling may have to fallback to io-wq, this way does hurt performance.
ublk starts to support zero copy recently, for avoiding unnecessary IO split, ublk driver's segment limit should be aligned with backend device's segment limit.
Another reason is that io_buffer_register_bvec() needs to allocate bvecs, which number is aligned with ublk request segment number, so that big memory allocation can be avoided by setting reasonable max_segments limit.
So add segment parameter for providing ublk server chance to align segment limit with backend, and keep it reasonable from implementation viewpoint.
--- drivers/block/ublk_drv.c | 20 +++++++++++++++++++- include/uapi/linux/ublk_cmd.h | 25 +++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 1 deletion(-)
drivers/block/ublk_drv.c +19 -1
@@ -74,7 +74,7 @@#define UBLK_PARAM_TYPE_ALL \ (UBLK_PARAM_TYPE_BASIC | UBLK_PARAM_TYPE_DISCARD | \ UBLK_PARAM_TYPE_DEVT | UBLK_PARAM_TYPE_ZONED | \ - UBLK_PARAM_TYPE_DMA_ALIGN) + UBLK_PARAM_TYPE_DMA_ALIGN | UBLK_PARAM_TYPE_SEGMENT)struct ublk_rq_data {struct kref ref;
@@ -580,6 +580,18 @@ static int ublk_validate_params(const struct ublk_device *ub)return -EINVAL ; } +if (ub->params.types &UBLK_PARAM_TYPE_SEGMENT ) { +const struct ublk_param_segment *p = &ub->params.seg; + +if (!is_power_of_2 (p->seg_boundary_mask +1 )) +return -EINVAL ; + +if (p->seg_boundary_mask +1 <UBLK_MIN_SEGMENT_SIZE ) +return -EINVAL ; +if (p->max_segment_size <UBLK_MIN_SEGMENT_SIZE ) +return -EINVAL ; + } +return 0 ; }
@@ -2370,6 +2382,12 @@ static int ublk_ctrl_start_dev(struct ublk_device *ub, struct io_uring_cmd *cmd)if (ub->params.types &UBLK_PARAM_TYPE_DMA_ALIGN ) lim.dma_alignment = ub->params.dma.alignment; +if (ub->params.types &UBLK_PARAM_TYPE_SEGMENT ) { + lim.seg_boundary_mask = ub->params.seg.seg_boundary_mask; + lim.max_segment_size = ub->params.seg.max_segment_size; + lim.max_segments = ub->params.seg.max_segments; + } +if (wait_for_completion_interruptible (&ub->completion) !=0 )return -EINTR ;
include/uapi/linux/ublk_cmd.h +25 -0
@@ -410,6 +410,29 @@ struct ublk_param_dma_align {__u8 pad[4 ]; }; +#define UBLK_MIN_SEGMENT_SIZE 4096 +/* +* If any one of the three segment parameter is set as 0, the behavior is +* undefined. +*/ +struct ublk_param_segment { +/* +* seg_boundary_mask + 1 needs to be power_of_2(), and the sum has +* to be >= UBLK_MIN_SEGMENT_SIZE(4096) +*/ +__u64 seg_boundary_mask; + +/* +* max_segment_size could be override by virt_boundary_mask, so be +* careful when setting both. +* +* max_segment_size has to be >= UBLK_MIN_SEGMENT_SIZE(4096) +*/ +__u32 max_segment_size; +__u16 max_segments; +__u8 pad[2 ]; +}; +struct ublk_params {/* * Total length of parameters, userspace has to set 'len' for both
@@ -423,6 +446,7 @@ struct ublk_params {#define UBLK_PARAM_TYPE_DEVT (1 << 2)#define UBLK_PARAM_TYPE_ZONED (1 << 3)#define UBLK_PARAM_TYPE_DMA_ALIGN (1 << 4) +#define UBLK_PARAM_TYPE_SEGMENT (1 << 5)__u32 types;/* types of parameter included */ struct ublk_param_basic basic;
@@ -430,6 +454,7 @@ struct ublk_params {struct ublk_param_devt devt;struct ublk_param_zoned zoned;struct ublk_param_dma_align dma; + struct ublk_param_segment seg; };#endif
Signature · 1 line
2.47.0
PATCH v2 7/11 ublk: document zero copy feature
In reply to Ming LeiAdd words to explain how zero copy feature works, and why it has to be trusted for handling IO read command.
--- Documentation/block/ublk.rst | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-)
Documentation/block/ublk.rst +26 -9
@@ -309,18 +309,35 @@ with specified IO tag in the command data: ``UBLK_IO_COMMIT_AND_FETCH_REQ`` to the server, ublkdrv needs to copy the server buffer (pages) read to the IO request pages. -Future development -================== - Zero copy --------- -Zero copy is a generic requirement for nbd, fuse or similar drivers. A -problem [#xiaoguang]_ Xiaoguang mentioned is that pages mapped to userspace -can't be remapped any more in kernel with existing mm interfaces. This can -occurs when destining direct IO to ``/dev/ublkb*``. Also, he reported that -big requests (IO size >= 256 KB) may benefit a lot from zero copy. - +ublk zero copy relies on io_uring's fixed kernel buffer, which provides +two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`. + +ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call +`io_buffer_register_bvec()` for ublk server to register client request +buffer into io_uring buffer table, then ublk server can submit io_uring +IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF` +calls `io_buffer_unregister_bvec()` to unregister the buffer, which is +guaranteed to be live between calling `io_buffer_register_bvec()` and +`io_buffer_unregister_bvec()`. Any io_uring operation which supports this +kind of kernel buffer will grab one reference of the buffer until the +operation is completed. + +ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and +be trusted, because it is ublk server's responsibility to make sure IO buffer +filled with data for handling read command, and ublk server has to return +correct result to ublk driver when handling READ command, and the result +has to match with how many bytes filled to the IO buffer. Otherwise, +uninitialized kernel IO buffer will be exposed to client application. + +ublk server needs to align the parameter of `struct ublk_param_dma_align` +with backend for zero copy to work correctly. + +For reaching best IO performance, ublk server should align its segment +parameter of `struct ublk_param_segment` with backend for avoiding +unnecessary IO split, which usually hurts io_uring performance. References ==========
Signature · 1 line
2.47.0
PATCH v2 8/11 ublk: implement ->queue_rqs()
In reply to Ming LeiImplement ->queue_rqs() for improving perf in case of MQ.
In this way, we just need to call io_uring_cmd_complete_in_task() once for whole IO batch, then both io_uring and ublk server can get exact batch from ublk frontend.
Follows IOPS improvement:
- tests
tools/testing/selftests/ublk/kublk add -t null -q 2 [-z] fio/t/io_uring -p0 /dev/ublkb0
- results:
more than 10% IOPS boost observed
Pass all ublk selftests, especially the io dispatch order test.
--- drivers/block/ublk_drv.c | 131 +++++++++++++++++++++++++++++++++------ 1 file changed, 111 insertions(+), 20 deletions(-)
drivers/block/ublk_drv.c +111 -20
@@ -81,6 +81,20 @@ struct ublk_rq_data { };struct ublk_uring_cmd_pdu { +/* +* Store requests in same batch temporarily for queuing them to +* daemon context. +* +* It should have been stored to request payload, but we do want +* to avoid extra pre-allocation, and uring_cmd payload is always +* free for us +*/ +struct request *req_list; + +/* +* The following two are valid in this cmd whole lifetime, and +* setup in ublk uring_cmd handler +*/ struct ublk_queue *ubq;u16 tag; };
@@ -1170,14 +1184,12 @@ static inline void __ublk_abort_rq(struct ublk_queue *ubq,blk_mq_end_request (rq,BLK_STS_IOERR ); } -static void ublk_rq_task_work_cb (struct io_uring_cmd *cmd, -unsigned int issue_flags) +static void ublk_dispatch_req (struct ublk_queue *ubq, +struct io_uring_cmd *cmd, +struct request *req, +unsigned int issue_flags) { -struct ublk_uring_cmd_pdu *pdu =ublk_get_uring_cmd_pdu (cmd); -struct ublk_queue *ubq = pdu->ubq; -int tag = pdu->tag; -struct request *req =blk_mq_tag_to_rq ( - ubq->dev->tag_set.tags[ubq->q_id], tag); +int tag = req->tag;struct ublk_io *io = &ubq->ios[tag];unsigned int mapped_bytes;
@@ -1252,6 +1264,18 @@ static void ublk_rq_task_work_cb(struct io_uring_cmd *cmd,ubq_complete_io_cmd (io,UBLK_IO_RES_OK , issue_flags); } +static void ublk_rq_task_work_cb (struct io_uring_cmd *cmd, +unsigned int issue_flags) +{ +struct ublk_uring_cmd_pdu *pdu =ublk_get_uring_cmd_pdu (cmd); +struct ublk_queue *ubq = pdu->ubq; +int tag = pdu->tag; +struct request *req =blk_mq_tag_to_rq ( + ubq->dev->tag_set.tags[ubq->q_id], tag); + +ublk_dispatch_req (ubq, cmd, req, issue_flags); +} +static void ublk_queue_cmd (struct ublk_queue *ubq,struct request *rq) {struct ublk_io *io = &ubq->ios[rq->tag];
@@ -1259,6 +1283,35 @@ static void ublk_queue_cmd(struct ublk_queue *ubq, struct request *rq)io_uring_cmd_complete_in_task (io->cmd, ublk_rq_task_work_cb); } +static void ublk_cmd_list_tw_cb (struct io_uring_cmd *cmd, +unsigned int issue_flags) +{ +struct ublk_uring_cmd_pdu *pdu =ublk_get_uring_cmd_pdu (cmd); +struct request *rq = pdu->req_list; +struct ublk_queue *ubq = rq->mq_hctx->driver_data; +struct request *next; + +while (rq) { +struct ublk_io *io = &ubq->ios[rq->tag]; + + next = rq->rq_next; + rq->rq_next =NULL ; +ublk_dispatch_req (ubq, io->cmd, rq, issue_flags); + rq = next; + } +} + +static void ublk_queue_cmd_list (struct ublk_queue *ubq,struct rq_list *l) +{ +struct request *rq =rq_list_peek (l); +struct ublk_io *io = &ubq->ios[rq->tag]; +struct ublk_uring_cmd_pdu *pdu =ublk_get_uring_cmd_pdu (io->cmd); + + pdu->req_list = rq; +rq_list_init (l); +io_uring_cmd_complete_in_task (io->cmd, ublk_cmd_list_tw_cb); +} +static enum blk_eh_timer_return ublk_timeout (struct request *rq) {struct ublk_queue *ubq = rq->mq_hctx->driver_data;
@@ -1297,21 +1350,12 @@ static enum blk_eh_timer_return ublk_timeout(struct request *rq)return BLK_EH_RESET_TIMER ; } -static blk_status_t ublk_queue_rq (struct blk_mq_hw_ctx *hctx, -const struct blk_mq_queue_data *bd) +static blk_status_t ublk_prep_req (struct ublk_queue *ubq,struct request *rq) { -struct ublk_queue *ubq = hctx->driver_data; -struct request *rq = bd->rq;blk_status_t res; -if (unlikely (ubq->fail_io)) { +if (unlikely (ubq->fail_io))return BLK_STS_TARGET ; - } - -/* fill iod to slot in io cmd buffer */ - res =ublk_setup_iod (ubq, rq); -if (unlikely (res !=BLK_STS_OK )) -return BLK_STS_IOERR ;/* With recovery feature enabled, force_abort is set in * ublk_stop_dev() before calling del_gendisk(). We have to
@@ -1325,6 +1369,29 @@ static blk_status_t ublk_queue_rq(struct blk_mq_hw_ctx *hctx,if (ublk_nosrv_should_queue_io (ubq) &&unlikely (ubq->force_abort))return BLK_STS_IOERR ; +if (unlikely (ubq->canceling)) +return BLK_STS_IOERR ; + +/* fill iod to slot in io cmd buffer */ + res =ublk_setup_iod (ubq, rq); +if (unlikely (res !=BLK_STS_OK )) +return BLK_STS_IOERR ; + +blk_mq_start_request (rq); +return BLK_STS_OK ; +} + +static blk_status_t ublk_queue_rq (struct blk_mq_hw_ctx *hctx, +const struct blk_mq_queue_data *bd) +{ +struct ublk_queue *ubq = hctx->driver_data; +struct request *rq = bd->rq; +blk_status_t res; + + res =ublk_prep_req (ubq, rq); +if (res !=BLK_STS_OK ) +return res; +/* * ->canceling has to be handled after ->force_abort and ->fail_io * is dealt with, otherwise this request may not be failed in case
@@ -1335,12 +1402,35 @@ static blk_status_t ublk_queue_rq(struct blk_mq_hw_ctx *hctx,return BLK_STS_OK ; } -blk_mq_start_request (bd->rq);ublk_queue_cmd (ubq, rq); -return BLK_STS_OK ; } +static void ublk_queue_rqs (struct rq_list *rqlist) +{ +struct rq_list requeue_list = { }; +struct rq_list submit_list = { }; +struct ublk_queue *ubq =NULL ; +struct request *req; + +while ((req =rq_list_pop (rqlist))) { +struct ublk_queue *this_q = req->mq_hctx->driver_data; + +if (ubq && ubq != this_q && !rq_list_empty (&submit_list)) +ublk_queue_cmd_list (ubq, &submit_list); + ubq = this_q; + +if (ublk_prep_req (ubq, req) ==BLK_STS_OK ) +rq_list_add_tail (&submit_list, req); +else +rq_list_add_tail (&requeue_list, req); + } + +if (ubq && !rq_list_empty (&submit_list)) +ublk_queue_cmd_list (ubq, &submit_list); + *rqlist = requeue_list; +} +static int ublk_init_hctx (struct blk_mq_hw_ctx *hctx,void *driver_data,unsigned int hctx_idx) {
@@ -1353,6 +1443,7 @@ static int ublk_init_hctx(struct blk_mq_hw_ctx *hctx, void *driver_data,static const struct blk_mq_ops ublk_mq_ops = { .queue_rq = ublk_queue_rq, + .queue_rqs = ublk_queue_rqs, .init_hctx = ublk_init_hctx, .timeout = ublk_timeout, };
Signature · 1 line
2.47.0
PATCH v2 9/11 ublk: rename ublk_rq_task_work_cb as ublk_cmd_tw_cb
In reply to Ming LeiThe new name is aligned with ublk_cmd_list_tw_cb(), and looks more readable.
--- drivers/block/ublk_drv.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/block/ublk_drv.c +3 -3
@@ -1264,8 +1264,8 @@ static void ublk_dispatch_req(struct ublk_queue *ubq,ubq_complete_io_cmd (io,UBLK_IO_RES_OK , issue_flags); } -static void ublk_rq_task_work_cb (struct io_uring_cmd *cmd, -unsigned int issue_flags) +static void ublk_cmd_tw_cb (struct io_uring_cmd *cmd, +unsigned int issue_flags) {struct ublk_uring_cmd_pdu *pdu =ublk_get_uring_cmd_pdu (cmd);struct ublk_queue *ubq = pdu->ubq;
@@ -1280,7 +1280,7 @@ static void ublk_queue_cmd(struct ublk_queue *ubq, struct request *rq) {struct ublk_io *io = &ubq->ios[rq->tag]; -io_uring_cmd_complete_in_task (io->cmd, ublk_rq_task_work_cb); +io_uring_cmd_complete_in_task (io->cmd, ublk_cmd_tw_cb); } static void ublk_cmd_list_tw_cb(struct io_uring_cmd *cmd,
Signature · 1 line
2.47.0
PATCH v2 10/11 selftests: ublk: add more tests for covering MQ
In reply to Ming LeiAdd test test_generic_02.sh for covering IO dispatch order in case of MQ.
Especially we just support ->queue_rqs() which may affect IO dispatch order.
Add test_loop_05.sh and test_stripe_03.sh for covering MQ.
--- tools/testing/selftests/ublk/Makefile | 3 ++ tools/testing/selftests/ublk/test_common.sh | 6 +++ .../testing/selftests/ublk/test_generic_02.sh | 44 +++++++++++++++++++ tools/testing/selftests/ublk/test_loop_01.sh | 14 +++--- tools/testing/selftests/ublk/test_loop_03.sh | 14 +++--- tools/testing/selftests/ublk/test_loop_05.sh | 28 ++++++++++++ .../testing/selftests/ublk/test_stress_01.sh | 6 +-- .../testing/selftests/ublk/test_stress_02.sh | 6 +-- .../testing/selftests/ublk/test_stripe_01.sh | 14 +++--- .../testing/selftests/ublk/test_stripe_03.sh | 30 +++++++++++++ 10 files changed, 132 insertions(+), 33 deletions(-) create mode 100755 tools/testing/selftests/ublk/test_generic_02.sh create mode 100755 tools/testing/selftests/ublk/test_loop_05.sh create mode 100755 tools/testing/selftests/ublk/test_stripe_03.sh
tools/testing/selftests/ublk/Makefile +3 -0
@@ -4,6 +4,7 @@ CFLAGS += -O3 -Wl,-no-as-needed -Wall -I $(top_srcdir) LDLIBS += -lpthread -lm -luring TEST_PROGS := test_generic_01.sh +TEST_PROGS += test_generic_02.sh TEST_PROGS += test_null_01.sh TEST_PROGS += test_null_02.sh
@@ -11,8 +12,10 @@ TEST_PROGS += test_loop_01.sh TEST_PROGS += test_loop_02.sh TEST_PROGS += test_loop_03.sh TEST_PROGS += test_loop_04.sh +TEST_PROGS += test_loop_05.sh TEST_PROGS += test_stripe_01.sh TEST_PROGS += test_stripe_02.sh +TEST_PROGS += test_stripe_03.sh TEST_PROGS += test_stress_01.sh TEST_PROGS += test_stress_02.sh
tools/testing/selftests/ublk/test_common.sh +6 -0
@@ -23,6 +23,12 @@ _get_disk_dev_t() { echo $(( (major & 0xfff) << 20 | (minor & 0xfffff) )) } +_run_fio_verify_io() { + fio --name=verify --rw=randwrite --direct=1 --ioengine=libaio \ + --bs=8k --iodepth=32 --verify=crc32c --do_verify=1 \ + --verify_state_save=0 "$@" > /dev/null +} + _create_backfile() { local my_size=$1 local my_file
tools/testing/selftests/ublk/test_generic_02.sh new file +44 -0
44 lines not on this page · raw message
tools/testing/selftests/ublk/test_loop_01.sh +5 -9
@@ -6,6 +6,10 @@ TID="loop_01" ERR_CODE=0 +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" +fi + _prep_test "loop" "write and verify test" backfile_0=$(_create_backfile 256M)
@@ -14,15 +18,7 @@ dev_id=$(_add_ublk_dev -t loop "$backfile_0") _check_add_dev $TID $? "${backfile_0}" # run fio over the ublk disk -fio --name=write_and_verify \ - --filename=/dev/ublkb"${dev_id}" \ - --ioengine=libaio --iodepth=16 \ - --rw=write \ - --size=256M \ - --direct=1 \ - --verify=crc32c \ - --do_verify=1 \ - --bs=4k > /dev/null 2>&1 +_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=256M ERR_CODE=$? _cleanup_test "loop"
tools/testing/selftests/ublk/test_loop_03.sh +5 -9
@@ -6,6 +6,10 @@ TID="loop_03" ERR_CODE=0 +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" +fi + _prep_test "loop" "write and verify over zero copy" backfile_0=$(_create_backfile 256M)
@@ -13,15 +17,7 @@ dev_id=$(_add_ublk_dev -t loop -z "$backfile_0") _check_add_dev $TID $? "$backfile_0" # run fio over the ublk disk -fio --name=write_and_verify \ - --filename=/dev/ublkb"${dev_id}" \ - --ioengine=libaio --iodepth=64 \ - --rw=write \ - --size=256M \ - --direct=1 \ - --verify=crc32c \ - --do_verify=1 \ - --bs=4k > /dev/null 2>&1 +_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=256M ERR_CODE=$? _cleanup_test "loop"
tools/testing/selftests/ublk/test_loop_05.sh new file +28 -0
@@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh + +TID="loop_05" +ERR_CODE=0 + +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" +fi + +_prep_test "loop" "write and verify test" + +backfile_0=$(_create_backfile 256M) + +dev_id=$(_add_ublk_dev -q 2 -t loop "$backfile_0") +_check_add_dev $TID $? "${backfile_0}" + +# run fio over the ublk disk +_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=256M +ERR_CODE=$? + +_cleanup_test "loop" + +_remove_backfile "$backfile_0" + +_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stress_01.sh +3 -3
@@ -27,20 +27,20 @@ ublk_io_and_remove() _prep_test "stress" "run IO and remove device" -ublk_io_and_remove 8G -t null +ublk_io_and_remove 8G -t null -q 4 ERR_CODE=$? if [ ${ERR_CODE} -ne 0 ]; then _show_result $TID $ERR_CODE fi BACK_FILE=$(_create_backfile 256M) -ublk_io_and_remove 256M -t loop "${BACK_FILE}" +ublk_io_and_remove 256M -t loop -q 4 "${BACK_FILE}" ERR_CODE=$? if [ ${ERR_CODE} -ne 0 ]; then _show_result $TID $ERR_CODE fi -ublk_io_and_remove 256M -t loop -z "${BACK_FILE}" +ublk_io_and_remove 256M -t loop -q 4 -z "${BACK_FILE}" ERR_CODE=$? _cleanup_test "stress" _remove_backfile "${BACK_FILE}"
tools/testing/selftests/ublk/test_stress_02.sh +3 -3
@@ -27,20 +27,20 @@ ublk_io_and_kill_daemon() _prep_test "stress" "run IO and kill ublk server" -ublk_io_and_kill_daemon 8G -t null +ublk_io_and_kill_daemon 8G -t null -q 4 ERR_CODE=$? if [ ${ERR_CODE} -ne 0 ]; then _show_result $TID $ERR_CODE fi BACK_FILE=$(_create_backfile 256M) -ublk_io_and_kill_daemon 256M -t loop "${BACK_FILE}" +ublk_io_and_kill_daemon 256M -t loop -q 4 "${BACK_FILE}" ERR_CODE=$? if [ ${ERR_CODE} -ne 0 ]; then _show_result $TID $ERR_CODE fi -ublk_io_and_kill_daemon 256M -t loop -z "${BACK_FILE}" +ublk_io_and_kill_daemon 256M -t loop -q 4 -z "${BACK_FILE}" ERR_CODE=$? _cleanup_test "stress" _remove_backfile "${BACK_FILE}"
tools/testing/selftests/ublk/test_stripe_01.sh +5 -9
@@ -6,6 +6,10 @@ TID="stripe_01" ERR_CODE=0 +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" +fi + _prep_test "stripe" "write and verify test" backfile_0=$(_create_backfile 256M)
@@ -15,15 +19,7 @@ dev_id=$(_add_ublk_dev -t stripe "$backfile_0" "$backfile_1") _check_add_dev $TID $? "${backfile_0}" # run fio over the ublk disk -fio --name=write_and_verify \ - --filename=/dev/ublkb"${dev_id}" \ - --ioengine=libaio --iodepth=32 \ - --rw=write \ - --size=512M \ - --direct=1 \ - --verify=crc32c \ - --do_verify=1 \ - --bs=4k > /dev/null 2>&1 +_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=512M ERR_CODE=$? _cleanup_test "stripe"
tools/testing/selftests/ublk/test_stripe_03.sh new file +30 -0
@@ -0,0 +1,30 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh + +TID="stripe_03" +ERR_CODE=0 + +if ! _have_program fio; then + exit "$UBLK_SKIP_CODE" +fi + +_prep_test "stripe" "write and verify test" + +backfile_0=$(_create_backfile 256M) +backfile_1=$(_create_backfile 256M) + +dev_id=$(_add_ublk_dev -q 2 -t stripe "$backfile_0" "$backfile_1") +_check_add_dev $TID $? "${backfile_0}" + +# run fio over the ublk disk +_run_fio_verify_io --filename=/dev/ublkb"${dev_id}" --size=512M +ERR_CODE=$? + +_cleanup_test "stripe" + +_remove_backfile "$backfile_0" +_remove_backfile "$backfile_1" + +_show_result $TID $ERR_CODE
Signature · 1 line
2.47.0
PATCH v2 11/11 selftests: ublk: add test for checking zero copy related parameter
In reply to Ming Leiublk zero copy usually requires to set dma and segment parameter correctly, so hard-code null target's dma & segment parameter in non-default value, and verify if they are setup correctly by ublk driver.
--- tools/testing/selftests/ublk/Makefile | 1 + tools/testing/selftests/ublk/null.c | 11 +++++++- .../testing/selftests/ublk/test_generic_03.sh | 28 +++++++++++++++++++ 3 files changed, 39 insertions(+), 1 deletion(-) create mode 100755 tools/testing/selftests/ublk/test_generic_03.sh
tools/testing/selftests/ublk/Makefile +1 -0
@@ -5,6 +5,7 @@ LDLIBS += -lpthread -lm -luring TEST_PROGS := test_generic_01.sh TEST_PROGS += test_generic_02.sh +TEST_PROGS += test_generic_03.sh TEST_PROGS += test_null_01.sh TEST_PROGS += test_null_02.sh
tools/testing/selftests/ublk/null.c +10 -1
@@ -17,7 +17,8 @@ static int ublk_null_tgt_init(const struct dev_ctx *ctx, struct ublk_dev *dev) dev->tgt.dev_size = dev_size; dev->tgt.params = (struct ublk_params ) { - .types =UBLK_PARAM_TYPE_BASIC , + .types =UBLK_PARAM_TYPE_BASIC |UBLK_PARAM_TYPE_DMA_ALIGN | +UBLK_PARAM_TYPE_SEGMENT , .basic = { .logical_bs_shift =9 , .physical_bs_shift =12 ,
@@ -26,6 +27,14 @@ static int ublk_null_tgt_init(const struct dev_ctx *ctx, struct ublk_dev *dev) .max_sectors = info->max_io_buf_bytes >> 9, .dev_sectors = dev_size >> 9, }, + .dma = { + .alignment = 4095, + }, + .seg = { + .seg_boundary_mask = 4095, + .max_segment_size = 32 << 10, + .max_segments = 32, + }, };if (info->flags &UBLK_F_SUPPORT_ZERO_COPY )
tools/testing/selftests/ublk/test_generic_03.sh new file +28 -0
@@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh + +TID="generic_03" +ERR_CODE=0 + +_prep_test "null" "check dma & segment limits for zero copy" + +dev_id=$(_add_ublk_dev -t null -z) +_check_add_dev $TID $? + +sysfs_path=/sys/block/ublkb"${dev_id}" +dma_align=$(cat "$sysfs_path"/queue/dma_alignment) +max_segments=$(cat "$sysfs_path"/queue/max_segments) +max_segment_size=$(cat "$sysfs_path"/queue/max_segment_size) +if [ "$dma_align" != "4095" ]; then + ERR_CODE=255 +fi +if [ "$max_segments" != "32" ]; then + ERR_CODE=255 +fi +if [ "$max_segment_size" != "32768" ]; then + ERR_CODE=255 +fi +_cleanup_test "null" +_show_result $TID $ERR_CODE
Signature · 1 line
2.47.0
Quoted 8 lines · Ming Lei
The 1st two fixes
ublk_abort_requests(), and adds comment on handlingubq->canceling.The 3rd ~ 5th patches are cleanup.
The 6th and 7th are zc-followup.
[...]
Applied, thanks!
[01/11] ublk: make sure ubq->canceling is set when queue is frozen
commit: 9c397ab202da574da65be6f4901d6260ad28d420
[02/11] ublk: comment on ubq->canceling handling in ublk_queue_rq()
commit: 2563451efc04e6e32b089696d221498277b5addb
[03/11] ublk: remove two unused fields from 'struct ublk_queue'
commit: b4ce4eae5c0e3afcdf4a1fb8c85e5c171a2ea5db
[04/11] ublk: add helper of ublk_need_map_io()
commit: 8a799a1616dd34724d9283ca0c4b96003da53d6a
[05/11] ublk: call io_uring_cmd_to_pdu to get uring_cmd pdu
commit: 639cf001d7cca2bdaccff3f545d80394b6b36a53
[06/11] ublk: add segment parameter
commit: e14e391fc9cc38a54f7528077930eae4f573b58e
[07/11] ublk: document zero copy feature
commit: 9f7862ebfb05db6ce88f6eeb0e6144cda284cdc2
[08/11] ublk: implement ->queue_rqs()
commit: d7aad08075f13bd9eb9cc73df831278f6c843aa4
[09/11] ublk: rename ublk_rq_task_work_cb as ublk_cmd_tw_cb
commit: 48c74741b43f77e7fef74f2ad2bb714bdf1675e1
[10/11] selftests: ublk: add more tests for covering MQ
commit: 7e1121b53e27978c5e33f241e87f4ffee45febd9
[11/11] selftests: ublk: add test for checking zero copy related parameter
commit: 888ba37f212022926462d7d0b993ee5354a30907Best regards,
Signature · 1 line
Jens Axboe