ublk: add support for UBLK_IO_NEED_GET_DATA
Subsystems (2)
v3 cover ublk: add support for UBLK_IO_NEED_GET_DATA
1. Introduction:UBLK_IO_NEED_GET_DATA is a new ublk IO command. It is designed for a user application who wants to allocate IO buffer and set IO buffer address only after it receives an IO request from ublksrv. This is a reasonable scenario because these users may use a RPC framework as one IO backend to handle IO requests passed from ublksrv. And a RPC framework may allocate its own buffer(or memory pool).
This new feature (UBLK_F_NEED_GET_DATA) is optional for ublk users. Related userspace code has been added in ublksrv[1] as one pull request.
We have add some test cases in ublksrv and all of them pass. The performance result shows that this new feature does bring additional latency because one IO is issued back to ublk_drv once again to copy data from bio vectors to user-provided data buffer.
2. Background:
For now, ublk requires the user to set IO buffer address in advance(with last UBLK_IO_COMMIT_AND_FETCH_REQ command)so the user has to pre-allocate IO buffer.
For READ requests, this work flow looks good because the data copy happens after user application gets a cqe and the kernel copies data. So user application can allocate IO buffer, copy data to be read into it, and issues a sqe with the newly allocated IO buffer.
However, for WRITE requests, this work flow looks weird because the data copy happens in a task_work before the user application gets one cqe. So it is inconvenient for users who allocates(or fetch from a memory pool)buffer after it gets one request(and know the actual data size). For these users, they have to memcpy from ublksrv's pre-allocated buffer to their internal buffer(such as RPC buffer). We think this additional memcpy could be a bottleneck and it is avoidable.
2. Design:
Consider add a new feature flag: UBLK_F_NEED_GET_DATA.
If user sets this new flag(through libublksrv) and pass it to kernel driver, ublk kernel driver should returns a cqe with UBLK_IO_RES_NEED_GET_DATA after a new blk-mq WRITE request comes.
A user application now can allocate data buffer for writing and pass its address in UBLK_IO_NEED_GET_DATA command after ublk kernel driver returns cqe with UBLK_IO_RES_NEED_GET_DATA.
After the kernel side gets the sqe (UBLK_IO_NEED_GET_DATA command), it now copies(address pinned, indeed) data to be written from bio vectors to newly returned IO data buffer.
Finally, the kernel side returns UBLK_IO_RES_OK and ublksrv handles the IO request as usual.The new feature: UBLK_F_NEED_GET_DATA is enabled on demand ublksrv still can pre-allocate data buffers with task_work.
3. Evaluation:
Related userspace code and tests have been added in ublksrv[1] as one pull request. We evaluate performance based on this PR.
We have tested write latency with:
(1) No UBLK_F_NEED_GET_DATA(the old commit) as baseline
(2) UBLK_F_NEED_GET_DATA enabled/disabled
on demo_null and demo_event of newest ublksrv project.
Config of fio:bs=4k, iodepth=1, numjobs=1, rw=write/randwrite, direct=1, ioengine=libaio.
Here is the comparison of lat(usec) in fio:
demo_null:
write: 28.74(baseline) -- 28.77(disable) -- 57.20(enable) randwrite: 27.81(baseline) -- 28.51(disable) -- 54.81(enable)
demo_event:
write: 46.45(baseline) -- 43.31(disable) -- 75.50(enable) randwrite: 45.39(baseline) -- 43.66(disable) -- 76.02(enable)
Looks like:
(1) UBLK_F_NEED_GET_DATA does not introduce additional overhead when comparing baseline and disable.
(2) enabling UBLK_F_NEED_GET_DATA adds about two times more latency than disabling it. And it is reasonable since the IO goes through the total ublk IO stack(ubd_drv <--> ublksrv) once again.
(3) demo_null and demo_event are both null targets. And I think this overhead is not too heavy if real data handling backend is used.
Without UBLK_IO_NEED_GET_DATA, an additional memcpy(from pre-allocated ublksrv buffer to user's buffer) is necessary for a WRITE request. However, UBLK_IO_NEED_GET_DATA does bring addtional syscall (io_uring_enter). To prove the value of UBLK_IO_NEED_GET_DATA, we test the single IO latency (usec) of demo_null with:
(1) UBLK_F_NEED_GET_DATA disabled; additional memcpy
(2) UBLK_F_NEED_GET_DATA enabled
Config of fio:iodepth=1, numjobs=1, rw=randwrite, direct=1, ioengine=libaio.
For block size, we choose 4k/64k/128k/256k/512k/1m. Note that with 1m block size, the original IO request will be split into two blk-mq requests.
Here is the comparison of lat(usec) in fio:
2 memcpy, w/o NEED_GET_DATA 1 memcpy, w/ NEED_GET_DATA
4k-randwrite: 9.65 10.06 64k-randwrite: 15.19 13.38 128k-randwrite: 19.47 17.77 256k-randwrite: 32.63 25.33 512k-randwrite: 90.57 46.08 1m-randwrite: 177.06 117.26
We find that with bigger block size, cases with one memcpy w/ NEED_GET_DATA result in lower latency than cases with two memcpy w/o NEED_GET_DATA. Therefore, we think NEED_GET_DATA is suitable for bigger block size, such as 512B or 1MB.
[1] https://github.com/ming1/ubdsrv
Since V2:
(1) UBLK_IO_NEED_GET_DATA should support both built-in task_work_add() and io_uring_cmd_complete_in_task()
Since V1:
(1) Add tests to compare (1)2 memcpy, w/o NEED_GET_DATA and (2)1 memcpy, w/ NEED_GET_DATA to show value of UBLK_IO_NEED_GET_DATA.
(2) rebase on the newest version of ublk_drv
ZiyangZhang (2):
ublk_cmd.h: add one new ublk command: UBLK_IO_NEED_GET_DATA
ublk_drv: add support for UBLK_IO_NEED_GET_DATA
drivers/block/ublk_drv.c | 96 +++++++++++++++++++++++++++++++---- include/uapi/linux/ublk_cmd.h | 18 +++++++ 2 files changed, 105 insertions(+), 9 deletions(-)
Signature · 1 line
2.34.1
PATCH v3 1/2 ublk_cmd.h: add one new ublk command: UBLK_IO_NEED_GET_DATA
Add one new ublk command: UBLK_IO_NEED_GET_DATA. It is prepared for a new feature designed for a user application who wants to allocate IO buffer and set IO buffer address only after it receives an IO request from ublksrv.
--- include/uapi/linux/ublk_cmd.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
include/uapi/linux/ublk_cmd.h +18 -0
@@ -28,12 +28,21 @@* this IO request, request's handling result is committed to ublk * driver, meantime FETCH_REQ is piggyback, and FETCH_REQ has to be * handled before completing io request. +* +* NEED_GET_DATA: only used for write requests to set io addr and copy data +* When NEED_GET_DATA is set, ublksrv has to issue UBLK_IO_NEED_GET_DATA +* command after ublk driver returns UBLK_IO_RES_NEED_GET_DATA. +* +* It is only used if ublksrv set UBLK_F_NEED_GET_DATA flag +* while starting a ublk device. */ #define UBLK_IO_FETCH_REQ 0x20#define UBLK_IO_COMMIT_AND_FETCH_REQ 0x21 +#define UBLK_IO_NEED_GET_DATA 0x22/* only ABORT means that no re-fetch */ #define UBLK_IO_RES_OK 0 +#define UBLK_IO_RES_NEED_GET_DATA 1#define UBLK_IO_RES_ABORT (-ENODEV)#define UBLKSRV_CMD_BUF_OFFSET 0
@@ -54,6 +63,15 @@*/ #define UBLK_F_URING_CMD_COMP_IN_TASK (1ULL << 1) +/* +* User should issue io cmd again for write requests to +* set io buffer address and copy data from bio vectors +* to the userspace io buffer. +* +* In this mode, task_work is not used. +*/ +#define UBLK_F_NEED_GET_DATA (1UL << 2) +/* device state */ #define UBLK_S_DEV_DEAD 0#define UBLK_S_DEV_LIVE 1
Signature · 1 line
2.34.1
Re: [PATCH V3 1/2] ublk_cmd.h: add one new ublk command: UBLK_IO_NEED_GET_DATA
Quoted 6 lines · ZiyangZhang
Add one new ublk command:
UBLK_IO_NEED_GET_DATA. It is prepared for a new feature designed for a user application who wants to allocate IO buffer and set IO buffer address only after it receives an IO request from ublksrv.Signed-off-by ZiyangZhang <[email protected]>
Thanks,
Ming
PATCH v3 2/2 ublk_drv: add support for UBLK_IO_NEED_GET_DATA
In reply to ZiyangZhangUBLK_IO_NEED_GET_DATA is one ublk IO command. It is designed for a user application who wants to allocate IO buffer and set IO buffer address only after it receives an IO request from ublksrv. This is a reasonable scenario because these users may use a RPC framework as one IO backend to handle IO requests passed from ublksrv. And a RPC framework may allocate its own buffer(or memory pool).
This new feature (UBLK_F_NEED_GET_DATA) is optional for ublk users. Related userspace code has been added in ublksrv[1] as one pull request.
Test cases for this feature are added in ublksrv and all the tests pass. The performance result shows that this new feature does bring additional latency because one IO is issued back to ublk_drv once again to copy data from bio vectors to user-provided data buffer. UBLK_IO_NEED_GET_DATA is suitable for bigger block size such as 512B or 1MB.
[1] https://github.com/ming1/ubdsrv
--- drivers/block/ublk_drv.c | 96 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 87 insertions(+), 9 deletions(-)
drivers/block/ublk_drv.c +87 -9
@@ -47,7 +47,9 @@#define UBLK_MINORS (1U << MINORBITS)/* All UBLK_F_* have to be included into UBLK_F_ALL */ -#define UBLK_F_ALL (UBLK_F_SUPPORT_ZERO_COPY | UBLK_F_URING_CMD_COMP_IN_TASK) +#define UBLK_F_ALL (UBLK_F_SUPPORT_ZERO_COPY \ + | UBLK_F_URING_CMD_COMP_IN_TASK \ + | UBLK_F_NEED_GET_DATA)struct ublk_rq_data {struct callback_head work;
@@ -86,6 +88,15 @@ struct ublk_uring_cmd_pdu {*/ #define UBLK_IO_FLAG_ABORTED 0x04 +/* +* UBLK_IO_FLAG_NEED_GET_DATA is set because IO command requires +* get data buffer address from ublksrv. +* +* Then, bio data could be copied into this data buffer for a WRITE request +* after the IO command is issued again and UBLK_IO_FLAG_NEED_GET_DATA is unset. +*/ +#define UBLK_IO_FLAG_NEED_GET_DATA 0x08 +struct ublk_io {/* userspace buffer address from io cmd */ __u64 addr;
@@ -168,6 +179,13 @@ static inline bool ublk_can_use_task_work(const struct ublk_queue *ubq)return false; } +static inline bool ublk_need_get_data (const struct ublk_queue *ubq) +{ +if (ubq->flags &UBLK_F_NEED_GET_DATA ) +return true; +return false; +} +static struct ublk_device *ublk_get_device (struct ublk_device *ub) {if (kobject_get_unless_zero (&ub->cdev_dev.kobj))
@@ -509,6 +527,21 @@ static void __ublk_fail_req(struct ublk_io *io, struct request *req) } } +static void ubq_complete_io_cmd (struct ublk_io *io,int res) +{ +/* mark this cmd owned by ublksrv */ + io->flags |=UBLK_IO_FLAG_OWNED_BY_SRV ; + +/* +* clear ACTIVE since we are done with this sqe/cmd slot +* We can only accept io cmd in case of being not active. +*/ + io->flags &= ~UBLK_IO_FLAG_ACTIVE ; + +/* tell ublksrv one io request is coming */ +io_uring_cmd_done (io->cmd, res,0 ); +} +#define UBLK_REQUEUE_DELAY_MS 3static inline void __ublk_rq_task_work (struct request *req)
@@ -531,6 +564,20 @@ static inline void __ublk_rq_task_work(struct request *req)return ; } +if (ublk_need_get_data (ubq) && + (req_op (req) ==REQ_OP_WRITE || +req_op (req) ==REQ_OP_FLUSH ) && + !(io->flags &UBLK_IO_FLAG_NEED_GET_DATA )) { + + io->flags |=UBLK_IO_FLAG_NEED_GET_DATA ; + +pr_devel ("%s: ublk_need_get_data. op %d, qid %d tag %d io_flags %x\n" , + __func__, io->cmd->cmd_op, ubq->q_id, req->tag, io->flags); + +ubq_complete_io_cmd (io,UBLK_IO_RES_NEED_GET_DATA ); +return ; + } + mapped_bytes =ublk_map_io (ubq, req, io);/* partially mapped, update io descriptor */
@@ -553,17 +600,13 @@ static inline void __ublk_rq_task_work(struct request *req) mapped_bytes >>9 ; } -/* mark this cmd owned by ublksrv */ - io->flags |=UBLK_IO_FLAG_OWNED_BY_SRV ; -/* -* clear ACTIVE since we are done with this sqe/cmd slot -* We can only accept io cmd in case of being not active. +* Anyway, we have handled UBLK_IO_NEED_GET_DATA for WRITE/FLUSH requests, +* or we did nothing for other types requests. */ - io->flags &= ~UBLK_IO_FLAG_ACTIVE ; + io->flags &= ~UBLK_IO_FLAG_NEED_GET_DATA ; -/* tell ublksrv one io request is coming */ -io_uring_cmd_done (io->cmd,UBLK_IO_RES_OK ,0 ); +ubq_complete_io_cmd (io,UBLK_IO_RES_OK ); }static void ublk_rq_task_work_cb (struct io_uring_cmd *cmd)
@@ -846,6 +889,25 @@ static void ublk_mark_io_ready(struct ublk_device *ub, struct ublk_queue *ubq)mutex_unlock (&ub->mutex); } +static void ublk_handle_need_get_data (struct ublk_device *ub,int q_id, +int tag,struct io_uring_cmd *cmd) +{ +struct ublk_queue *ubq =ublk_get_queue (ub, q_id); +struct request *req =blk_mq_tag_to_rq (ub->tag_set.tags[q_id], tag); + +if (ublk_can_use_task_work (ubq)) { +struct ublk_rq_data *data =blk_mq_rq_to_pdu (req); + +/* should not fail since we call it just in ubq->ubq_daemon */ +task_work_add (ubq->ubq_daemon, &data->work,TWA_SIGNAL_NO_IPI ); + }else { +struct ublk_uring_cmd_pdu *pdu =ublk_get_uring_cmd_pdu (cmd); + + pdu->req = req; +io_uring_cmd_complete_in_task (cmd, ublk_rq_task_work_cb); + } +} +static int ublk_ch_uring_cmd (struct io_uring_cmd *cmd,unsigned int issue_flags) {struct ublksrv_io_cmd *ub_cmd = (struct ublksrv_io_cmd *)cmd->cmd;
@@ -884,6 +946,14 @@ static int ublk_ch_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) goto out; } +/* +* ensure that the user issues UBLK_IO_NEED_GET_DATA +* iff the driver have set the UBLK_IO_FLAG_NEED_GET_DATA. +*/ +if ((!!(io->flags &UBLK_IO_FLAG_NEED_GET_DATA )) + ^ (cmd_op ==UBLK_IO_NEED_GET_DATA )) + goto out; +switch (cmd_op) {case UBLK_IO_FETCH_REQ :/* UBLK_IO_FETCH_REQ is only allowed before queue is setup */
@@ -917,6 +987,14 @@ static int ublk_ch_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) io->cmd = cmd;ublk_commit_completion (ub, ub_cmd);break ; +case UBLK_IO_NEED_GET_DATA : +if (!(io->flags &UBLK_IO_FLAG_OWNED_BY_SRV )) + goto out; + io->addr = ub_cmd->addr; + io->cmd = cmd; + io->flags |=UBLK_IO_FLAG_ACTIVE ; +ublk_handle_need_get_data (ub, ub_cmd->q_id, ub_cmd->tag, cmd); +break ;default : goto out; }
Signature · 1 line
2.34.1
Re: [PATCH V3 2/2] ublk_drv: add support for UBLK_IO_NEED_GET_DATA
Quoted 116 lines · ZiyangZhang
UBLK_IO_NEED_GET_DATAis one ublk IO command. It is designed for a user application who wants to allocate IO buffer and set IO buffer address only after it receives an IO request from ublksrv. This is a reasonable scenario because these users may use a RPC framework as one IO backend to handle IO requests passed from ublksrv. And a RPC framework may allocate its own buffer(or memory pool).This new feature (
UBLK_F_NEED_GET_DATA) is optional for ublk users. Related userspace code has been added in ublksrv[1] as one pull request.Test cases for this feature are added in ublksrv and all the tests pass. The performance result shows that this new feature does bring additional latency because one IO is issued back to
ublk_drvonce again to copy data from bio vectors to user-provided data buffer.UBLK_IO_NEED_GET_DATAis suitable for bigger block size such as 512B or 1MB.[1] https://github.com/ming1/ubdsrv
Signed-off-by ZiyangZhang <[email protected]>--- drivers/block/ublk_drv.c | 96 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 87 insertions(+), 9 deletions(-)drivers/block/ublk_drv.c +48
-2@@ -47,7 +47,9 @@#define UBLK_MINORS (1U << MINORBITS)/* All UBLK_F_* have to be included into UBLK_F_ALL */ -#define UBLK_F_ALL (UBLK_F_SUPPORT_ZERO_COPY | UBLK_F_URING_CMD_COMP_IN_TASK) +#define UBLK_F_ALL (UBLK_F_SUPPORT_ZERO_COPY \ + | UBLK_F_URING_CMD_COMP_IN_TASK \ + | UBLK_F_NEED_GET_DATA)struct ublk_rq_data {struct callback_head work;@@ -86,6 +88,15 @@ struct ublk_uring_cmd_pdu {*/ #define UBLK_IO_FLAG_ABORTED 0x04 +/* +* UBLK_IO_FLAG_NEED_GET_DATA is set because IO command requires +* get data buffer address from ublksrv. +* +* Then, bio data could be copied into this data buffer for a WRITE request +* after the IO command is issued again and UBLK_IO_FLAG_NEED_GET_DATA is unset. +*/ +#define UBLK_IO_FLAG_NEED_GET_DATA 0x08 +struct ublk_io {/* userspace buffer address from io cmd */ __u64 addr;@@ -168,6 +179,13 @@ static inline bool ublk_can_use_task_work(const struct ublk_queue *ubq)return false; } +static inline bool ublk_need_get_data (const struct ublk_queue *ubq) +{ +if (ubq->flags &UBLK_F_NEED_GET_DATA ) +return true; +return false; +} +static struct ublk_device *ublk_get_device (struct ublk_device *ub) {if (kobject_get_unless_zero (&ub->cdev_dev.kobj))@@ -509,6 +527,21 @@ static void __ublk_fail_req(struct ublk_io *io, struct request *req) } } +static void ubq_complete_io_cmd (struct ublk_io *io,int res) +{ +/* mark this cmd owned by ublksrv */ + io->flags |=UBLK_IO_FLAG_OWNED_BY_SRV ; + +/* +* clear ACTIVE since we are done with this sqe/cmd slot +* We can only accept io cmd in case of being not active. +*/ + io->flags &= ~UBLK_IO_FLAG_ACTIVE ; + +/* tell ublksrv one io request is coming */ +io_uring_cmd_done (io->cmd, res,0 ); +} +#define UBLK_REQUEUE_DELAY_MS 3static inline void __ublk_rq_task_work (struct request *req)@@ -531,6 +564,20 @@ static inline void __ublk_rq_task_work(struct request *req)return ; } +if (ublk_need_get_data (ubq) && + (req_op (req) ==REQ_OP_WRITE || +req_op (req) ==REQ_OP_FLUSH ) && + !(io->flags &UBLK_IO_FLAG_NEED_GET_DATA )) { + + io->flags |=UBLK_IO_FLAG_NEED_GET_DATA ; + +pr_devel ("%s: ublk_need_get_data. op %d, qid %d tag %d io_flags %x\n" , + __func__, io->cmd->cmd_op, ubq->q_id, req->tag, io->flags); + +ubq_complete_io_cmd (io,UBLK_IO_RES_NEED_GET_DATA ); +return ; + } + mapped_bytes =ublk_map_io (ubq, req, io);/* partially mapped, update io descriptor */ @@ -553,17 +600,13 @@ static inline void __ublk_rq_task_work(struct request *req) mapped_bytes >>9 ; } -/* mark this cmd owned by ublksrv */
drivers/block/ublk_drv.c static inline void __ublk_rq_task_work(struct request *req) · line 607
Quoted 10 lines
drivers/block/ublk_drv.c +3
-5- io->flags |=UBLK_IO_FLAG_OWNED_BY_SRV ; -/* -* clear ACTIVE since we are done with this sqe/cmd slot -* We can only accept io cmd in case of being not active. +* Anyway, we have handled UBLK_IO_NEED_GET_DATA for WRITE/FLUSH requests, +* or we did nothing for other types requests. */ - io->flags &= ~UBLK_IO_FLAG_ACTIVE ; + io->flags &= ~UBLK_IO_FLAG_NEED_GET_DATA ;
Please move clearing UBLK_IO_FLAG_NEED_GET_DATA into ublk_handle_need_get_data().
UBLK_IO_FLAG_NEED_GET_DATA should only be touched in case that UBLK_F_NEED_GET_DATA is set, also it becomes more readable.
Otherwise, this patch looks fine.
Thanks,
Ming
Re: [PATCH V3 2/2] ublk_drv: add support for UBLK_IO_NEED_GET_DATA
Quoted 130 lines · Ming Lei
Quoted 116 lines · ZiyangZhang
UBLK_IO_NEED_GET_DATAis one ublk IO command. It is designed for a user application who wants to allocate IO buffer and set IO buffer address only after it receives an IO request from ublksrv. This is a reasonable scenario because these users may use a RPC framework as one IO backend to handle IO requests passed from ublksrv. And a RPC framework may allocate its own buffer(or memory pool).This new feature (
UBLK_F_NEED_GET_DATA) is optional for ublk users. Related userspace code has been added in ublksrv[1] as one pull request.Test cases for this feature are added in ublksrv and all the tests pass. The performance result shows that this new feature does bring additional latency because one IO is issued back to
ublk_drvonce again to copy data from bio vectors to user-provided data buffer.UBLK_IO_NEED_GET_DATAis suitable for bigger block size such as 512B or 1MB.[1] https://github.com/ming1/ubdsrv
Signed-off-by ZiyangZhang <[email protected]>--- drivers/block/ublk_drv.c | 96 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 87 insertions(+), 9 deletions(-)drivers/block/ublk_drv.c +48
-2@@ -47,7 +47,9 @@#define UBLK_MINORS (1U << MINORBITS)/* All UBLK_F_* have to be included into UBLK_F_ALL */ -#define UBLK_F_ALL (UBLK_F_SUPPORT_ZERO_COPY | UBLK_F_URING_CMD_COMP_IN_TASK) +#define UBLK_F_ALL (UBLK_F_SUPPORT_ZERO_COPY \ + | UBLK_F_URING_CMD_COMP_IN_TASK \ + | UBLK_F_NEED_GET_DATA)struct ublk_rq_data {struct callback_head work;@@ -86,6 +88,15 @@ struct ublk_uring_cmd_pdu {*/ #define UBLK_IO_FLAG_ABORTED 0x04 +/* +* UBLK_IO_FLAG_NEED_GET_DATA is set because IO command requires +* get data buffer address from ublksrv. +* +* Then, bio data could be copied into this data buffer for a WRITE request +* after the IO command is issued again and UBLK_IO_FLAG_NEED_GET_DATA is unset. +*/ +#define UBLK_IO_FLAG_NEED_GET_DATA 0x08 +struct ublk_io {/* userspace buffer address from io cmd */ __u64 addr;@@ -168,6 +179,13 @@ static inline bool ublk_can_use_task_work(const struct ublk_queue *ubq)return false; } +static inline bool ublk_need_get_data (const struct ublk_queue *ubq) +{ +if (ubq->flags &UBLK_F_NEED_GET_DATA ) +return true; +return false; +} +static struct ublk_device *ublk_get_device (struct ublk_device *ub) {if (kobject_get_unless_zero (&ub->cdev_dev.kobj))@@ -509,6 +527,21 @@ static void __ublk_fail_req(struct ublk_io *io, struct request *req) } } +static void ubq_complete_io_cmd (struct ublk_io *io,int res) +{ +/* mark this cmd owned by ublksrv */ + io->flags |=UBLK_IO_FLAG_OWNED_BY_SRV ; + +/* +* clear ACTIVE since we are done with this sqe/cmd slot +* We can only accept io cmd in case of being not active. +*/ + io->flags &= ~UBLK_IO_FLAG_ACTIVE ; + +/* tell ublksrv one io request is coming */ +io_uring_cmd_done (io->cmd, res,0 ); +} +#define UBLK_REQUEUE_DELAY_MS 3static inline void __ublk_rq_task_work (struct request *req)@@ -531,6 +564,20 @@ static inline void __ublk_rq_task_work(struct request *req)return ; } +if (ublk_need_get_data (ubq) && + (req_op (req) ==REQ_OP_WRITE || +req_op (req) ==REQ_OP_FLUSH ) && + !(io->flags &UBLK_IO_FLAG_NEED_GET_DATA )) { + + io->flags |=UBLK_IO_FLAG_NEED_GET_DATA ; + +pr_devel ("%s: ublk_need_get_data. op %d, qid %d tag %d io_flags %x\n" , + __func__, io->cmd->cmd_op, ubq->q_id, req->tag, io->flags); + +ubq_complete_io_cmd (io,UBLK_IO_RES_NEED_GET_DATA ); +return ; + } + mapped_bytes =ublk_map_io (ubq, req, io);/* partially mapped, update io descriptor */ @@ -553,17 +600,13 @@ static inline void __ublk_rq_task_work(struct request *req) mapped_bytes >>9 ; } -/* mark this cmd owned by ublksrv */ drivers/block/ublk_drv.c static inline void __ublk_rq_task_work(struct request *req) · line 607
Quoted 10 lines
drivers/block/ublk_drv.c +3
-5- io->flags |=UBLK_IO_FLAG_OWNED_BY_SRV ; -/* -* clear ACTIVE since we are done with this sqe/cmd slot -* We can only accept io cmd in case of being not active. +* Anyway, we have handled UBLK_IO_NEED_GET_DATA for WRITE/FLUSH requests, +* or we did nothing for other types requests. */ - io->flags &= ~UBLK_IO_FLAG_ACTIVE ; + io->flags &= ~UBLK_IO_FLAG_NEED_GET_DATA ;Please move clearing
UBLK_IO_FLAG_NEED_GET_DATAintoublk_handle_need_get_data().
Quoted 5 lines
UBLK_IO_FLAG_NEED_GET_DATAshould only be touched in case thatUBLK_F_NEED_GET_DATAis set, also it becomes more readable.Otherwise, this patch looks fine.
Ok, it will be fixed in V4.
Thanks,
Zhang
Quoted 67 lines · ZiyangZhang
1. Introduction:
UBLK_IO_NEED_GET_DATAis a new ublk IO command. It is designed for a user application who wants to allocate IO buffer and set IO buffer address only after it receives an IO request from ublksrv. This is a reasonable scenario because these users may use a RPC framework as one IO backend to handle IO requests passed from ublksrv. And a RPC framework may allocate its own buffer(or memory pool).This new feature (
UBLK_F_NEED_GET_DATA) is optional for ublk users. Related userspace code has been added in ublksrv[1] as one pull request.We have add some test cases in ublksrv and all of them pass. The performance result shows that this new feature does bring additional latency because one IO is issued back to
ublk_drvonce again to copy data from bio vectors to user-provided data buffer.2. Background:
For now, ublk requires the user to set IO buffer address inadvance(with last UBLK_IO_COMMIT_AND_FETCH_REQ command)so the user has to pre-allocate IO buffer.For READ requests, this work flow looks good because the data copy happens after user application gets a cqe and the kernel copies data. So user application can allocate IO buffer, copy data to be read into it, and issues a sqe with the newly allocated IO buffer.
However, for WRITE requests, this work flow looks weird because the data copy happens in a
task_workbefore the user application gets one cqe. So it is inconvenient for users who allocates(or fetch from a memory pool)buffer after it gets one request(and know the actual data size). For these users, they have to memcpy from ublksrv's pre-allocated buffer to their internal buffer(such as RPC buffer). We think this additional memcpy could be a bottleneck and it is avoidable.2. Design:
Consider add a new feature flag:UBLK_F_NEED_GET_DATA.If user sets this new flag(through libublksrv) and pass it to kernel driver, ublk kernel driver should returns a cqe with
UBLK_IO_RES_NEED_GET_DATAafter a new blk-mq WRITE request comes.A user application now can allocate data buffer for writing and pass its address in
UBLK_IO_NEED_GET_DATAcommand after ublk kernel driver returns cqe withUBLK_IO_RES_NEED_GET_DATA.After the kernel side gets the sqe (
UBLK_IO_NEED_GET_DATAcommand), it nowcopies(address pinned, indeed)data to be written from bio vectors to newly returned IO data buffer.Finally, the kernel side returns
UBLK_IO_RES_OKand ublksrv handles the IO request as usual.The new feature:UBLK_F_NEED_GET_DATAis enabled on demand ublksrv still can pre-allocate data buffers withtask_work.3. Evaluation:
Related userspace code and tests have been added in ublksrv[1] as one pull request. We evaluate performance based on this PR.We have tested write latency with:
(1) NoUBLK_F_NEED_GET_DATA(the old commit)as baseline
(2)UBLK_F_NEED_GET_DATAenabled/disabled
ondemo_nullanddemo_eventof newest ublksrv project.Config of fio:bs=4k, iodepth=1, numjobs=1, rw=write/randwrite, direct=1, ioengine=libaio.
Here is the comparison of lat(usec) in fio:
Quoted 7 lines
demo_null:write: 28.74(baseline) -- 28.77(disable) -- 57.20(enable) randwrite: 27.81(baseline) -- 28.51(disable) -- 54.81(enable)
demo_event:write: 46.45(baseline) -- 43.31(disable) -- 75.50(enable) randwrite: 45.39(baseline) -- 43.66(disable) -- 76.02(enable)
The data is interesting, and I guess the enable latency data could become much less if 64 iodepth & 16 batch is used
--iodepth=64 --iodepth_batch_submit=16 --iodepth_batch_complete_min=16
Quoted 34 lines
Looks like:
(1)UBLK_F_NEED_GET_DATAdoes not introduce additional overhead when comparing baseline and disable.
(2) enablingUBLK_F_NEED_GET_DATAadds about two times more latency than disabling it. And it is reasonable since the IO goes through the total ublk IOstack(ubd_drv <--> ublksrv)once again.
(3)demo_nullanddemo_eventare both null targets. And I think this overhead is not too heavy if real data handling backend is used.Without
UBLK_IO_NEED_GET_DATA, an additional memcpy(from pre-allocated ublksrv buffer to user's buffer) is necessary for a WRITE request. However,UBLK_IO_NEED_GET_DATAdoes bring addtional syscall (io_uring_enter). To prove the value ofUBLK_IO_NEED_GET_DATA, we test the single IO latency (usec) ofdemo_nullwith:
(1)UBLK_F_NEED_GET_DATAdisabled; additional memcpy
(2)UBLK_F_NEED_GET_DATAenabledConfig of fio:iodepth=1, numjobs=1, rw=randwrite, direct=1, ioengine=libaio.
For block size, we choose 4k/64k/128k/256k/512k/1m. Note that with 1m block size, the original IO request will be split into two blk-mq requests.
Here is the comparison of lat(usec) in fio:
2 memcpy, w/o
NEED_GET_DATA1 memcpy, w/NEED_GET_DATA4k-randwrite: 9.65 10.06 64k-randwrite: 15.19 13.38 128k-randwrite: 19.47 17.77 256k-randwrite: 32.63 25.33 512k-randwrite: 90.57 46.08 1m-randwrite: 177.06 117.26
Quoted 4 lines
We find that with bigger block size, cases with one memcpy w/
NEED_GET_DATAresult in lower latency than cases with two memcpy w/oNEED_GET_DATA. Therefore, we thinkNEED_GET_DATAis suitable for bigger block size, such as 512B or 1MB.
With 64 iodepth and submit/completion batching, I think NEED_GET_DATA could become less.
Anyway, it is very helpful to share the test data, nice job!
Thanks,
Ming
Quoted 72 lines · Ming Lei
Quoted 71 lines · ZiyangZhang
1. Introduction:
UBLK_IO_NEED_GET_DATAis a new ublk IO command. It is designed for a user application who wants to allocate IO buffer and set IO buffer address only after it receives an IO request from ublksrv. This is a reasonable scenario because these users may use a RPC framework as one IO backend to handle IO requests passed from ublksrv. And a RPC framework may allocate its own buffer(or memory pool).This new feature (
UBLK_F_NEED_GET_DATA) is optional for ublk users. Related userspace code has been added in ublksrv[1] as one pull request.We have add some test cases in ublksrv and all of them pass. The performance result shows that this new feature does bring additional latency because one IO is issued back to
ublk_drvonce again to copy data from bio vectors to user-provided data buffer.2. Background:
For now, ublk requires the user to set IO buffer address inadvance(with last UBLK_IO_COMMIT_AND_FETCH_REQ command)so the user has to pre-allocate IO buffer.For READ requests, this work flow looks good because the data copy happens after user application gets a cqe and the kernel copies data. So user application can allocate IO buffer, copy data to be read into it, and issues a sqe with the newly allocated IO buffer.
However, for WRITE requests, this work flow looks weird because the data copy happens in a
task_workbefore the user application gets one cqe. So it is inconvenient for users who allocates(or fetch from a memory pool)buffer after it gets one request(and know the actual data size). For these users, they have to memcpy from ublksrv's pre-allocated buffer to their internal buffer(such as RPC buffer). We think this additional memcpy could be a bottleneck and it is avoidable.2. Design:
Consider add a new feature flag:UBLK_F_NEED_GET_DATA.If user sets this new flag(through libublksrv) and pass it to kernel driver, ublk kernel driver should returns a cqe with
UBLK_IO_RES_NEED_GET_DATAafter a new blk-mq WRITE request comes.A user application now can allocate data buffer for writing and pass its address in
UBLK_IO_NEED_GET_DATAcommand after ublk kernel driver returns cqe withUBLK_IO_RES_NEED_GET_DATA.After the kernel side gets the sqe (
UBLK_IO_NEED_GET_DATAcommand), it nowcopies(address pinned, indeed)data to be written from bio vectors to newly returned IO data buffer.Finally, the kernel side returns
UBLK_IO_RES_OKand ublksrv handles the IO request as usual.The new feature:UBLK_F_NEED_GET_DATAis enabled on demand ublksrv still can pre-allocate data buffers withtask_work.3. Evaluation:
Related userspace code and tests have been added in ublksrv[1] as one pull request. We evaluate performance based on this PR.We have tested write latency with:
(1) NoUBLK_F_NEED_GET_DATA(the old commit)as baseline
(2)UBLK_F_NEED_GET_DATAenabled/disabled
ondemo_nullanddemo_eventof newest ublksrv project.Config of fio:bs=4k, iodepth=1, numjobs=1, rw=write/randwrite, direct=1, ioengine=libaio.
Here is the comparison of lat(usec) in fio:
demo_null:write: 28.74(baseline) -- 28.77(disable) -- 57.20(enable) randwrite: 27.81(baseline) -- 28.51(disable) -- 54.81(enable)
Quoted 7 lines
Quoted 3 lines
demo_event:write: 46.45(baseline) -- 43.31(disable) -- 75.50(enable) randwrite: 45.39(baseline) -- 43.66(disable) -- 76.02(enable)The data is interesting, and I guess the enable latency data could become much less if 64 iodepth & 16 batch is used
--iodepth=64 --iodepth_batch_submit=16 --iodepth_batch_complete_min=16
Yes, multiple UBLK_IO_NEED_GET_DATA cmds can share the same io_uring_enter() syscall with bigger iodepth & batch.
Quoted 40 lines
Quoted 35 lines
Looks like:
(1)UBLK_F_NEED_GET_DATAdoes not introduce additional overhead when comparing baseline and disable.
(2) enablingUBLK_F_NEED_GET_DATAadds about two times more latency than disabling it. And it is reasonable since the IO goes through the total ublk IOstack(ubd_drv <--> ublksrv)once again.
(3)demo_nullanddemo_eventare both null targets. And I think this overhead is not too heavy if real data handling backend is used.Without
UBLK_IO_NEED_GET_DATA, an additional memcpy(from pre-allocated ublksrv buffer to user's buffer) is necessary for a WRITE request. However,UBLK_IO_NEED_GET_DATAdoes bring addtional syscall (io_uring_enter). To prove the value ofUBLK_IO_NEED_GET_DATA, we test the single IO latency (usec) ofdemo_nullwith:
(1)UBLK_F_NEED_GET_DATAdisabled; additional memcpy
(2)UBLK_F_NEED_GET_DATAenabledConfig of fio:iodepth=1, numjobs=1, rw=randwrite, direct=1, ioengine=libaio.
For block size, we choose 4k/64k/128k/256k/512k/1m. Note that with 1m block size, the original IO request will be split into two blk-mq requests.
Here is the comparison of lat(usec) in fio:
2 memcpy, w/o
NEED_GET_DATA1 memcpy, w/NEED_GET_DATA4k-randwrite: 9.65 10.06 64k-randwrite: 15.19 13.38 128k-randwrite: 19.47 17.77 256k-randwrite: 32.63 25.33 512k-randwrite: 90.57 46.08 1m-randwrite: 177.06 117.26Quoted 4 lines
We find that with bigger block size, cases with one memcpy w/
NEED_GET_DATAresult in lower latency than cases with two memcpy w/oNEED_GET_DATA. Therefore, we thinkNEED_GET_DATAis suitable for bigger block size, such as 512B or 1MB.
Quoted 4 lines
With 64 iodepth and submit/completion batching, I think
NEED_GET_DATAcould become less.Anyway, it is very helpful to share the test data, nice job!
Thanks.
I have also noticed that with 128 iodepth, 16 batch(in test cases of ublksrv), NEED_GET_DATA behaves well.
Thanks,
Zhang.