Skip to contentSkip to conversation

ublk: restore auto buf unregister refcount optimization

PATCH Applied Caleb Sander Mateos 3 participants 5 messages Jan 28 – Jan 30

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ublk: restore auto buf unregister refcount optimization

Commit 1ceeedb59749 ("ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon task") optimized ublk request buffer unregistration to use a non-atomic reference count decrement when performed on the ublk_io's daemon task. The optimization applied to auto buffer unregistration, which happens as part of handling UBLK_IO_COMMIT_AND_FETCH_REQ on the daemon task. However, commit b749965edda8 ("ublk: remove ublk_commit_and_fetch()") reordered the ublk_sub_req_ref() for the completed request before the io_buffer_unregister_bvec() call. As a result, task_registered_buffers is already 0 when io_buffer_unregister_bvec() calls ublk_io_release() and the non-atomic refcount optimization doesn't apply.
Move the io_buffer_unregister_bvec() call back to before ublk_need_complete_req() to restore the reference counting optimization.

Fixes b749965edda8 ("ublk: remove ublk_commit_and_fetch()")
---
 drivers/block/ublk_drv.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/block/ublk_drv.c +2 -2
index 7981decd1cee..f864a0f2f572 100644
@@ -3243,15 +3243,15 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
 		if (ret)
 			goto out;
 		io->res = result;
 		req = ublk_fill_io_cmd(io, cmd);
 		ret = ublk_config_io_buf(ub, io, cmd, addr, &buf_idx);
+		if (buf_idx != UBLK_INVALID_BUF_IDX)
+			io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
 		compl = ublk_need_complete_req(ub, io);
 
 		/* can't touch 'ublk_io' any more */
-		if (buf_idx != UBLK_INVALID_BUF_IDX)
-			io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
 		if (req_op(req) == REQ_OP_ZONE_APPEND)
 			req->__sector = addr;
 		if (compl)
 			__ublk_complete_rq(req, io, ublk_dev_need_map_io(ub), NULL);
 
Signature · 1 line
2.45.2
#1 lore raw
Quoted 31 lines · Caleb Sander Mateos

Commit 1ceeedb59749 ("ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon task") optimized ublk request buffer unregistration to use a non-atomic reference count decrement when performed on the ublk_io's daemon task. The optimization applied to auto buffer unregistration, which happens as part of handling UBLK_IO_COMMIT_AND_FETCH_REQ on the daemon task. However, commit b749965edda8 ("ublk: remove ublk_commit_and_fetch()") reordered the ublk_sub_req_ref() for the completed request before the io_buffer_unregister_bvec() call. As a result, task_registered_buffers is already 0 when io_buffer_unregister_bvec() calls ublk_io_release() and the non-atomic refcount optimization doesn't apply.
Move the io_buffer_unregister_bvec() call back to before ublk_need_complete_req() to restore the reference counting optimization.

Fixes b749965edda8 ("ublk: remove ublk_commit_and_fetch()")
---
 drivers/block/ublk_drv.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/block/ublk_drv.c +1 -0
index 7981decd1cee..f864a0f2f572 100644
@@ -3243,15 +3243,15 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
                if (ret)
                        goto out;
                io->res = result;
                req = ublk_fill_io_cmd(io, cmd);
                ret = ublk_config_io_buf(ub, io, cmd, addr, &buf_idx);
+               if (buf_idx != UBLK_INVALID_BUF_IDX)

drivers/block/ublk_drv.c static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, · line 3256

Quoted 10 lines
drivers/block/ublk_drv.c +1 -2
+                       io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
                compl = ublk_need_complete_req(ub, io);
 
                /* can't touch 'ublk_io' any more */
-               if (buf_idx != UBLK_INVALID_BUF_IDX)
-                       io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
                if (req_op(req) == REQ_OP_ZONE_APPEND)
                        req->__sector = addr;
                if (compl)
                        __ublk_complete_rq(req, io, ublk_dev_need_map_io(ub), NULL);

I also noticed that the "can't touch 'ublk_io' any more" comment doesn't make much sense, as __ublk_complete_rq() still accesses (and even mutates) the struct ublk_io. Am I misunderstanding the comment? It looks like this might be a race condition for UBLK_U_IO_COMMIT_IO_CMDS, as __ublk_complete_rq() is called without holding the ublk_io spinlock.

Thanks,
Caleb

#2 lore raw
Quoted 36 lines · Caleb Sander Mateos
Quoted 34 lines · Caleb Sander Mateos

Commit 1ceeedb59749 ("ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon task") optimized ublk request buffer unregistration to use a non-atomic reference count decrement when performed on the ublk_io's daemon task. The optimization applied to auto buffer unregistration, which happens as part of handling UBLK_IO_COMMIT_AND_FETCH_REQ on the daemon task. However, commit b749965edda8 ("ublk: remove ublk_commit_and_fetch()") reordered the ublk_sub_req_ref() for the completed request before the io_buffer_unregister_bvec() call. As a result, task_registered_buffers is already 0 when io_buffer_unregister_bvec() calls ublk_io_release() and the non-atomic refcount optimization doesn't apply.
Move the io_buffer_unregister_bvec() call back to before ublk_need_complete_req() to restore the reference counting optimization.

Fixes b749965edda8 ("ublk: remove ublk_commit_and_fetch()")
---
 drivers/block/ublk_drv.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/block/ublk_drv.c +2 -0
index 7981decd1cee..f864a0f2f572 100644
@@ -3243,15 +3243,15 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
                if (ret)
                        goto out;
                io->res = result;
                req = ublk_fill_io_cmd(io, cmd);
                ret = ublk_config_io_buf(ub, io, cmd, addr, &buf_idx);
+               if (buf_idx != UBLK_INVALID_BUF_IDX)
+                       io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
                compl = ublk_need_complete_req(ub, io);
 
Quoted 11 lines

drivers/block/ublk_drv.c static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, · line 3256

Quoted 7 lines
drivers/block/ublk_drv.c +0 -2
                /* can't touch 'ublk_io' any more */
-               if (buf_idx != UBLK_INVALID_BUF_IDX)
-                       io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
                if (req_op(req) == REQ_OP_ZONE_APPEND)
                        req->__sector = addr;
                if (compl)
                        __ublk_complete_rq(req, io, ublk_dev_need_map_io(ub), NULL);

I also noticed that the "can't touch 'ublk_io' any more" comment doesn't make much sense, as __ublk_complete_rq() still accesses (and even mutates) the struct ublk_io. Am I misunderstanding the comment?

Yes, it can be removed, originally this code block may be reused for
BATCH_IO, but finally it doesn't work toward this way, so can you remove
it in this patch given it is introduced in b749965edda8 ("ublk: remove ublk_commit_and_fetch()")?

Quoted 3 lines

It looks like this might be a race condition for UBLK_U_IO_COMMIT_IO_CMDS, as __ublk_complete_rq() is called without holding the ublk_io spinlock.

It is actually fine for __ublk_complete_rq() to manipulate io->res lockless:

1) UBLK_IO_FLAG_OWNED_BY_SRV is cleared & checked with io->lock, so any new UBLK_U_IO_COMMIT_IO_CMDS will be failed

2) the current IO request isn't completed yet, so new io command handling won't be started.

Thanks,
Ming

Reviewed-by on v1 1/1
#3 lore raw
In reply to Caleb Sander Mateos
Quoted 13 lines · Caleb Sander Mateos

Commit 1ceeedb59749 ("ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon task") optimized ublk request buffer unregistration to use a non-atomic reference count decrement when performed on the ublk_io's daemon task. The optimization applied to auto buffer unregistration, which happens as part of handling UBLK_IO_COMMIT_AND_FETCH_REQ on the daemon task. However, commit b749965edda8 ("ublk: remove ublk_commit_and_fetch()") reordered the ublk_sub_req_ref() for the completed request before the io_buffer_unregister_bvec() call. As a result, task_registered_buffers is already 0 when io_buffer_unregister_bvec() calls ublk_io_release() and the non-atomic refcount optimization doesn't apply.
Move the io_buffer_unregister_bvec() call back to before ublk_need_complete_req() to restore the reference counting optimization.

Quoted 2 lines
Fixes b749965edda8 ("ublk: remove ublk_commit_and_fetch()")

Thanks,
Ming

maintainer Applied
#4 lore raw
In reply to Caleb Sander Mateos
Quoted 13 lines · Caleb Sander Mateos

Commit 1ceeedb59749 ("ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon task") optimized ublk request buffer unregistration to use a non-atomic reference count decrement when performed on the ublk_io's daemon task. The optimization applied to auto buffer unregistration, which happens as part of handling UBLK_IO_COMMIT_AND_FETCH_REQ on the daemon task. However, commit b749965edda8 ("ublk: remove ublk_commit_and_fetch()") reordered the ublk_sub_req_ref() for the completed request before the io_buffer_unregister_bvec() call. As a result, task_registered_buffers is already 0 when io_buffer_unregister_bvec() calls ublk_io_release() and the non-atomic refcount optimization doesn't apply.
Move the io_buffer_unregister_bvec() call back to before ublk_need_complete_req() to restore the reference counting optimization.

Quoted 1 line

[...]

Applied, thanks!

[1/1] ublk: restore auto buf unregister refcount optimization

      commit: ad5f2e2908c9b79a86529281a48e94d644d43dc7

Best regards,

Signature · 1 line
Jens Axboe