io_uring: set plug tags for same file
Subsystems (2)
PATCH v3 1/1 io_uring: set plug tags for same file
From: Keith Busch <[email protected]>
io_uring tries to optimize allocating tags by hinting to the plug how many it expects to need for a batch instead of allocating each tag individually. But io_uring submission queueus may have a mix of many devices for io, so the number of io's counted may be overestimated. This can lead to allocating too many tags, which adds overhead to finding that many contiguous tags, freeing up the ones we didn't use, and may starve out other users that can actually use them.
When starting a new batch of uring commands, count only commands that match the file descriptor of the first seen for this optimization. This avoids have to call the unlikely "blk_mq_free_plug_rqs()" at the end of a submission when multiple devices are used in a batch.
---v2->v3
The previous attempted to split the setup and submit further, but was requested to go back to this simpler version.
block/blk-core.c | 49 +++++++++++++++------------------- block/blk-mq.c | 6 +++-- include/linux/blkdev.h | 6 ----- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 37 ++++++++++++++++++------- 5 files changed, 54 insertions(+), 45 deletions(-)
block/blk-core.c +22 -27
@@ -1043,32 +1043,6 @@ int kblockd_mod_delayed_work_on(int cpu, struct delayed_work *dwork, }EXPORT_SYMBOL (kblockd_mod_delayed_work_on); -void blk_start_plug_nr_ios (struct blk_plug *plug,unsigned short nr_ios) -{ -struct task_struct *tsk = current; - -/* -* If this is a nested plug, don't actually assign it. -*/ -if (tsk->plug) -return ; - - plug->mq_list =NULL ; - plug->cached_rq =NULL ; - plug->nr_ios = min_t(unsigned short, nr_ios, BLK_MAX_REQUEST_COUNT); - plug->rq_count =0 ; - plug->multiple_queues = false; - plug->has_elevator = false; - plug->nowait = false; -INIT_LIST_HEAD (&plug->cb_list); - -/* -* Store ordering should not be needed here, since a potential -* preempt will imply a full memory barrier -*/ - tsk->plug = plug; -} -/** * blk_start_plug - initialize blk_plug and track it inside the task_struct * @plug: The &struct blk_plug that needs to be initialized
@@ -1094,7 +1068,28 @@ void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)*/ void blk_start_plug (struct blk_plug *plug) { -blk_start_plug_nr_ios (plug,1 ); +struct task_struct *tsk = current; + +/* +* If this is a nested plug, don't actually assign it. +*/ +if (tsk->plug) +return ; + + plug->mq_list =NULL ; + plug->cached_rq =NULL ; + plug->nr_ios =1 ; + plug->rq_count =0 ; + plug->multiple_queues = false; + plug->has_elevator = false; + plug->nowait = false; +INIT_LIST_HEAD (&plug->cb_list); + +/* +* Store ordering should not be needed here, since a potential +* preempt will imply a full memory barrier +*/ + tsk->plug = plug; }EXPORT_SYMBOL (blk_start_plug);
block/blk-mq.c +4 -2
@@ -523,7 +523,8 @@ static struct request *blk_mq_rq_cache_fill(struct request_queue *q, .q = q, .flags = flags, .cmd_flags = opf, - .nr_tags = plug->nr_ios, + .nr_tags = min_t(unsigned int, plug->nr_ios, + BLK_MAX_REQUEST_COUNT), .cached_rq = &plug->cached_rq, };struct request *rq;
@@ -2859,7 +2860,8 @@ static struct request *blk_mq_get_new_requests(struct request_queue *q,rq_qos_throttle (q, bio);if (plug) { - data.nr_tags = plug->nr_ios; + data.nr_tags = min_t(unsigned int, plug->nr_ios, +BLK_MAX_REQUEST_COUNT ); plug->nr_ios =1 ; data.cached_rq = &plug->cached_rq; }
include/linux/blkdev.h +0 -6
@@ -984,7 +984,6 @@ struct blk_plug_cb {extern struct blk_plug_cb *blk_check_plugged (blk_plug_cb_fn unplug,void *data,int size);extern void blk_start_plug (struct blk_plug *); -extern void blk_start_plug_nr_ios (struct blk_plug *,unsigned short );extern void blk_finish_plug (struct blk_plug *);void __blk_flush_plug (struct blk_plug *plug,bool from_schedule);
@@ -1000,11 +999,6 @@ long nr_blockdev_pages(void);struct blk_plug { }; -static inline void blk_start_plug_nr_ios (struct blk_plug *plug, -unsigned short nr_ios) -{ -} -static inline void blk_start_plug (struct blk_plug *plug) { }
include/linux/io_uring_types.h +1 -0
@@ -175,6 +175,7 @@ struct io_submit_state {bool need_plug;unsigned short submit_nr;unsigned int cqes_count; +int fd;struct blk_plug plug;struct io_uring_cqe cqes[16 ]; };
io_uring/io_uring.c +27 -10
@@ -2195,18 +2195,25 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req,return -EINVAL ;if (def->needs_file) { -struct io_submit_state *state = &ctx->submit_state; - req->cqe.fd =READ_ONCE (sqe->fd);/* * Plug now if we have more than 2 IO left after this, and the * target is potentially a read/write to block based storage. */ -if (state->need_plug && def->plug) { - state->plug_started = true; - state->need_plug = false; -blk_start_plug_nr_ios (&state->plug, state->submit_nr); +if (def->plug) { +struct io_submit_state *state = &ctx->submit_state; + +if (state->need_plug) { + state->plug_started = true; + state->need_plug = false; + state->fd = req->cqe.fd; +blk_start_plug (&state->plug); + }else if (state->plug_started && + state->fd == req->cqe.fd && + !state->link.head) { + state->plug.nr_ios++; + } } }
@@ -2267,7 +2274,8 @@ static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, }static inline int io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req, -const struct io_uring_sqe *sqe) +const struct io_uring_sqe *sqe, +struct io_wq_work_list *req_list) __must_hold(&ctx->uring_lock) {struct io_submit_link *link = &ctx->submit_state.link;
@@ -2315,7 +2323,7 @@ static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,return 0 ; } -io_queue_sqe (req); +wq_list_add_tail (&req->comp_list, req_list);return 0 ; }
@@ -2400,6 +2408,8 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) __must_hold(&ctx->uring_lock) {unsigned int entries =io_sqring_entries (ctx); +struct io_wq_work_list req_list; +struct io_kiocb *req;unsigned int left;int ret;
@@ -2410,9 +2420,9 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr)io_get_task_refs (left);io_submit_state_start (&ctx->submit_state, left); +INIT_WQ_LIST (&req_list); do { const struct io_uring_sqe *sqe; - struct io_kiocb *req; if (unlikely(!io_alloc_req(ctx, &req))) break;
@@ -2425,13 +2435,20 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr)* Continue submitting even for sqe failure if the * ring was setup with IORING_SETUP_SUBMIT_ALL */ -if (unlikely (io_submit_sqe (ctx, req, sqe)) && +if (unlikely (io_submit_sqe (ctx, req, sqe, &req_list)) && !(ctx->flags &IORING_SETUP_SUBMIT_ALL )) { left--;break ; } }while (--left); +while (req_list.first) { + req = container_of(req_list.first, struct io_kiocb, comp_list); + req_list.first = req->comp_list.next; + req->comp_list.next =NULL ; +io_queue_sqe (req); + } +if (unlikely (left)) { ret -= left;/* try again if it submitted nothing and can't allocate a req */
Signature · 1 line
2.34.1
kernel test robot
build botkernel test robot noticed the following build errors:
Hi Keith,
kernel test robot noticed the following build errors:
[auto build test ERROR on axboe-block/for-next]
[also build test ERROR on linus/master v6.5-rc4 next-20230731] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Keith-Busch/io_uring-set-plug-tags-for-same-file/20230801-044107 base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git for-next patch link: https://lore.kernel.org/r/20230731203932.2083468-1-kbusch%40meta.com patch subject: [PATCHv3] io_uring: set plug tags for same file config: openrisc-randconfig-r014-20230731 (https://download.01.org/0day-ci/archive/20230801/[email protected]/config) compiler: or1k-linux-gcc (GCC) 12.3.0 reproduce: (https://download.01.org/0day-ci/archive/20230801/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
io_uring/io_uring.c: In function 'io_init_req':
>> io_uring/io_uring.c:2228:44: error: 'struct blk_plug' has no member named 'nr_ios'
2228 | state->plug.nr_ios++;
| ^vim +2228 io_uring/io_uring.c
2150
2151 static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req,
2152 const struct io_uring_sqe *sqe)
2153 __must_hold(&ctx->uring_lock)
2154 {
2155 const struct io_issue_def *def;
2156 unsigned int sqe_flags;
2157 int personality;
2158 u8 opcode;
2159
2160 /* req is partially pre-initialised, see io_preinit_req() */
2161 req->opcode = opcode = READ_ONCE(sqe->opcode);
2162 /* same numerical values with corresponding REQ_F_*, safe to copy */
2163 req->flags = sqe_flags = READ_ONCE(sqe->flags);
2164 req->cqe.user_data = READ_ONCE(sqe->user_data);
2165 req->file = NULL;
2166 req->rsrc_node = NULL;
2167 req->task = current;
2168
2169 if (unlikely(opcode >= IORING_OP_LAST)) {
2170 req->opcode = 0;
2171 return -EINVAL;
2172 }
2173 def = &io_issue_defs[opcode];
2174 if (unlikely(sqe_flags & ~SQE_COMMON_FLAGS)) {
2175 /* enforce forwards compatibility on users */
2176 if (sqe_flags & ~SQE_VALID_FLAGS)
2177 return -EINVAL;
2178 if (sqe_flags & IOSQE_BUFFER_SELECT) {
2179 if (!def->buffer_select)
2180 return -EOPNOTSUPP;
2181 req->buf_index = READ_ONCE(sqe->buf_group);
2182 }
2183 if (sqe_flags & IOSQE_CQE_SKIP_SUCCESS)
2184 ctx->drain_disabled = true;
2185 if (sqe_flags & IOSQE_IO_DRAIN) {
2186 if (ctx->drain_disabled)
2187 return -EOPNOTSUPP;
2188 io_init_req_drain(req);
2189 }
2190 }
2191 if (unlikely(ctx->restricted || ctx->drain_active || ctx->drain_next)) {
2192 if (ctx->restricted && !io_check_restriction(ctx, req, sqe_flags))
2193 return -EACCES;
2194 /* knock it to the slow queue path, will be drained there */
2195 if (ctx->drain_active)
2196 req->flags |= REQ_F_FORCE_ASYNC;
2197 /* if there is no link, we're at "next" request and need to drain */
2198 if (unlikely(ctx->drain_next) && !ctx->submit_state.link.head) {
2199 ctx->drain_next = false;
2200 ctx->drain_active = true;
2201 req->flags |= REQ_F_IO_DRAIN | REQ_F_FORCE_ASYNC;
2202 }
2203 }
2204
2205 if (!def->ioprio && sqe->ioprio)
2206 return -EINVAL;
2207 if (!def->iopoll && (ctx->flags & IORING_SETUP_IOPOLL))
2208 return -EINVAL;
2209
2210 if (def->needs_file) {
2211 req->cqe.fd = READ_ONCE(sqe->fd);
2212
2213 /*
2214 * Plug now if we have more than 2 IO left after this, and the
2215 * target is potentially a read/write to block based storage.
2216 */
2217 if (def->plug) {
2218 struct io_submit_state *state = &ctx->submit_state;
2219
2220 if (state->need_plug) {
2221 state->plug_started = true;
2222 state->need_plug = false;
2223 state->fd = req->cqe.fd;
2224 blk_start_plug(&state->plug);
2225 } else if (state->plug_started &&
2226 state->fd == req->cqe.fd &&
2227 !state->link.head) {
> 2228 state->plug.nr_ios++; 2229 }
2230 }
2231 }
2232
2233 personality = READ_ONCE(sqe->personality);
2234 if (personality) {
2235 int ret;
2236
2237 req->creds = xa_load(&ctx->personalities, personality);
2238 if (!req->creds)
2239 return -EINVAL;
2240 get_cred(req->creds);
2241 ret = security_uring_override_creds(req->creds);
2242 if (ret) {
2243 put_cred(req->creds);
2244 return ret;
2245 }
2246 req->flags |= REQ_F_CREDS;
2247 }
2248
2249 return def->prep(req, sqe);
2250 }
2251
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Quoted 10 lines · Keith Busch
From: Keith Busch <[email protected]>
io_uringtries to optimize allocating tags by hinting to the plug how many it expects to need for a batch instead of allocating each tag individually. Butio_uringsubmission queueus may have a mix of many devices for io, so the number of io's counted may be overestimated. This can lead to allocating too many tags, which adds overhead to finding that many contiguous tags, freeing up the ones we didn't use, and may starve out other users that can actually use them.
Quoted 4 lines
When starting a new batch of uring commands, count only commands that match the file descriptor of the first seen for this optimization. This avoids have to call the unlikely "
blk_mq_free_plug_rqs()" at the end of a submission when multiple devices are used in a batch.
looks good and doesn't blow up on tests
Quoted 238 lines
Signed-off-by Keith Busch <[email protected]>---
v2->v3The previous attempted to split the setup and submit further, but was requested to go back to this simpler version.
block/blk-core.c | 49 +++++++++++++++------------------- block/blk-mq.c | 6 +++-- include/linux/blkdev.h | 6 ----- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 37 ++++++++++++++++++------- 5 files changed, 54 insertions(+), 45 deletions(-)block/blk-core.c +22
-27@@ -1043,32 +1043,6 @@ int kblockd_mod_delayed_work_on(int cpu, struct delayed_work *dwork, }EXPORT_SYMBOL (kblockd_mod_delayed_work_on); -void blk_start_plug_nr_ios (struct blk_plug *plug,unsigned short nr_ios) -{ -struct task_struct *tsk = current; - -/* -* If this is a nested plug, don't actually assign it. -*/ -if (tsk->plug) -return ; - - plug->mq_list =NULL ; - plug->cached_rq =NULL ; - plug->nr_ios = min_t(unsigned short, nr_ios, BLK_MAX_REQUEST_COUNT); - plug->rq_count =0 ; - plug->multiple_queues = false; - plug->has_elevator = false; - plug->nowait = false; -INIT_LIST_HEAD (&plug->cb_list); - -/* -* Store ordering should not be needed here, since a potential -* preempt will imply a full memory barrier -*/ - tsk->plug = plug; -} -/** * blk_start_plug - initialize blk_plug and track it inside the task_struct * @plug: The &struct blk_plug that needs to be initialized @@ -1094,7 +1068,28 @@ void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)*/ void blk_start_plug (struct blk_plug *plug) { -blk_start_plug_nr_ios (plug,1 ); +struct task_struct *tsk = current; + +/* +* If this is a nested plug, don't actually assign it. +*/ +if (tsk->plug) +return ; + + plug->mq_list =NULL ; + plug->cached_rq =NULL ; + plug->nr_ios =1 ; + plug->rq_count =0 ; + plug->multiple_queues = false; + plug->has_elevator = false; + plug->nowait = false; +INIT_LIST_HEAD (&plug->cb_list); + +/* +* Store ordering should not be needed here, since a potential +* preempt will imply a full memory barrier +*/ + tsk->plug = plug; }EXPORT_SYMBOL (blk_start_plug);block/blk-mq.c +4
-2@@ -523,7 +523,8 @@ static struct request *blk_mq_rq_cache_fill(struct request_queue *q, .q = q, .flags = flags, .cmd_flags = opf, - .nr_tags = plug->nr_ios, + .nr_tags = min_t(unsigned int, plug->nr_ios, + BLK_MAX_REQUEST_COUNT), .cached_rq = &plug->cached_rq, };struct request *rq;@@ -2859,7 +2860,8 @@ static struct request *blk_mq_get_new_requests(struct request_queue *q,rq_qos_throttle (q, bio);if (plug) { - data.nr_tags = plug->nr_ios; + data.nr_tags = min_t(unsigned int, plug->nr_ios, +BLK_MAX_REQUEST_COUNT ); plug->nr_ios =1 ; data.cached_rq = &plug->cached_rq; }include/linux/blkdev.h +0
-6@@ -984,7 +984,6 @@ struct blk_plug_cb {extern struct blk_plug_cb *blk_check_plugged (blk_plug_cb_fn unplug,void *data,int size);extern void blk_start_plug (struct blk_plug *); -extern void blk_start_plug_nr_ios (struct blk_plug *,unsigned short );extern void blk_finish_plug (struct blk_plug *);void __blk_flush_plug (struct blk_plug *plug,bool from_schedule);@@ -1000,11 +999,6 @@ long nr_blockdev_pages(void);struct blk_plug { }; -static inline void blk_start_plug_nr_ios (struct blk_plug *plug, -unsigned short nr_ios) -{ -} -static inline void blk_start_plug (struct blk_plug *plug) { }include/linux/io_uring_types.h +1
-0@@ -175,6 +175,7 @@ struct io_submit_state {bool need_plug;unsigned short submit_nr;unsigned int cqes_count; +int fd;struct blk_plug plug;struct io_uring_cqe cqes[16 ]; };io_uring/io_uring.c +27
-10@@ -2195,18 +2195,25 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req,return -EINVAL ;if (def->needs_file) { -struct io_submit_state *state = &ctx->submit_state; - req->cqe.fd =READ_ONCE (sqe->fd);/* * Plug now if we have more than 2 IO left after this, and the * target is potentially a read/write to block based storage. */ -if (state->need_plug && def->plug) { - state->plug_started = true; - state->need_plug = false; -blk_start_plug_nr_ios (&state->plug, state->submit_nr); +if (def->plug) { +struct io_submit_state *state = &ctx->submit_state; + +if (state->need_plug) { + state->plug_started = true; + state->need_plug = false; + state->fd = req->cqe.fd; +blk_start_plug (&state->plug); + }else if (state->plug_started && + state->fd == req->cqe.fd && + !state->link.head) { + state->plug.nr_ios++; + } } }@@ -2267,7 +2274,8 @@ static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, }static inline int io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req, -const struct io_uring_sqe *sqe) +const struct io_uring_sqe *sqe, +struct io_wq_work_list *req_list) __must_hold(&ctx->uring_lock) {struct io_submit_link *link = &ctx->submit_state.link;@@ -2315,7 +2323,7 @@ static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,return 0 ; } -io_queue_sqe (req); +wq_list_add_tail (&req->comp_list, req_list);return 0 ; }@@ -2400,6 +2408,8 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) __must_hold(&ctx->uring_lock) {unsigned int entries =io_sqring_entries (ctx); +struct io_wq_work_list req_list; +struct io_kiocb *req;unsigned int left;int ret;@@ -2410,9 +2420,9 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr)io_get_task_refs (left);io_submit_state_start (&ctx->submit_state, left); +INIT_WQ_LIST (&req_list); do { const struct io_uring_sqe *sqe; - struct io_kiocb *req; if (unlikely(!io_alloc_req(ctx, &req))) break;@@ -2425,13 +2435,20 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr)* Continue submitting even for sqe failure if the * ring was setup with IORING_SETUP_SUBMIT_ALL */ -if (unlikely (io_submit_sqe (ctx, req, sqe)) && +if (unlikely (io_submit_sqe (ctx, req, sqe, &req_list)) && !(ctx->flags &IORING_SETUP_SUBMIT_ALL )) { left--;break ; } }while (--left); +while (req_list.first) { + req = container_of(req_list.first, struct io_kiocb, comp_list); + req_list.first = req->comp_list.next; + req->comp_list.next =NULL ; +io_queue_sqe (req); + } +if (unlikely (left)) { ret -= left;/* try again if it submitted nothing and can't allocate a req */
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 10 lines · Keith Busch
From: Keith Busch <[email protected]>
io_uringtries to optimize allocating tags by hinting to the plug how many it expects to need for a batch instead of allocating each tag individually. Butio_uringsubmission queueus may have a mix of many devices for io, so the number of io's counted may be overestimated. This can lead to allocating too many tags, which adds overhead to finding that many contiguous tags, freeing up the ones we didn't use, and may starve out other users that can actually use them.
Quoted 4 lines
When starting a new batch of uring commands, count only commands that match the file descriptor of the first seen for this optimization. This avoids have to call the unlikely "
blk_mq_free_plug_rqs()" at the end of a submission when multiple devices are used in a batch.
Wanted to run this through both the peak IOPS and networking testing, started with the former. Here's a peak run with -git + pending 6.5 changes + pending 6.6 changes:
IOPS=125.88M, BW=61.46GiB/s, IOS/call=32/31
IOPS=125.39M, BW=61.23GiB/s, IOS/call=31/31
IOPS=124.97M, BW=61.02GiB/s, IOS/call=32/32
IOPS=124.60M, BW=60.84GiB/s, IOS/call=32/32
IOPS=124.27M, BW=60.68GiB/s, IOS/call=31/31
IOPS=124.00M, BW=60.54GiB/s, IOS/call=32/31
and here's one with the patch:
IOPS=121.69M, BW=59.42GiB/s, IOS/call=31/32
IOPS=121.26M, BW=59.21GiB/s, IOS/call=32/32
IOPS=120.87M, BW=59.02GiB/s, IOS/call=31/31
IOPS=120.87M, BW=59.02GiB/s, IOS/call=32/32
IOPS=121.02M, BW=59.09GiB/s, IOS/call=32/32
IOPS=121.63M, BW=59.39GiB/s, IOS/call=31/32
IOPS=121.48M, BW=59.32GiB/s, IOS/call=31/31
Running a quick profile, here's the top diff:
# Baseline Delta Abs Shared Object Symbol
# ........ ......... ................ ...........................................
#
6.69% -3.03% [kernel.vmlinux] [k] io_issue_sqe
9.65% +2.30% [nvme] [k] nvme_poll_cq
4.86% -1.55% [kernel.vmlinux] [k] io_submit_sqes
4.61% +1.40% [kernel.vmlinux] [k] blk_mq_submit_bio
4.79% -0.98% [kernel.vmlinux] [k] io_read
0.56% +0.97% [kernel.vmlinux] [k] blkdev_dio_unaligned.isra.0
3.61% +0.52% [kernel.vmlinux] [k] dma_unmap_page_attrs
2.04% -0.45% [kernel.vmlinux] [k] blk_add_rq_to_plugNote that this is perf.data.old being the kernel with your patch, and perf.data being the "stock" kernel mentioned above. The main thing looks like spending more time in io_issue_sqe() and io_submit_sqes(), and converserly we're spending less time polling. Usually for profiling a polled workload, having more time in the polling function is good, as it shows us we're spending less time everywhere else.
This is what I'm using:
sudo t/io_uring -p1 -d128 -b512 -s32 -c32 -F1 -B1 -R0 -X1 -n24 -P1 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1 /dev/nvme9n1 /dev/nvme10n1 /dev/nvme11n1 /dev/nvme12n1 /dev/nvme13n1 /dev/nvme14n1 /dev/nvme15n1 /dev/nvme16n1 /dev/nvme17n1 /dev/nvme18n1 /dev/nvme19n1 /dev/nvme20n1 /dev/nvme21n1 /dev/nvme22n1 /dev/nvme23n1
which is submitting 32 requests at the time. Obviously we don't expect a win in this case, as each thread is handling just a single NVMe device. The stock kernel will not over-allocate in this case.
If we change that to -n12 instead, meaning each will drive two devices, here's what the stock kernel gets:
IOPS=60.95M, BW=29.76GiB/s, IOS/call=31/32
IOPS=60.99M, BW=29.78GiB/s, IOS/call=31/32
IOPS=60.96M, BW=29.77GiB/s, IOS/call=31/31
IOPS=60.95M, BW=29.76GiB/s, IOS/call=31/31
IOPS=60.91M, BW=29.74GiB/s, IOS/call=32/32
and with the patch:
IOPS=59.64M, BW=29.12GiB/s, IOS/call=32/31
IOPS=59.63M, BW=29.12GiB/s, IOS/call=31/32
IOPS=59.57M, BW=29.09GiB/s, IOS/call=31/31
IOPS=59.57M, BW=29.09GiB/s, IOS/call=32/32
IOPS=59.65M, BW=29.12GiB/s, IOS/call=31/31
Now these are both obviously lower, but I haven't done anything to ensure that the two-drives-per-poller workload is optimized. For all I know, the numa layout is now messed up too. Just as a caveat, but they are comparable to each other.
Perf diff again looks similar, note that this time it's perf.data.old that's the stock kernel and perf.data that's the one with your patch:
3.51% +2.84% [kernel.vmlinux] [k] io_issue_sqe
3.24% +1.35% [kernel.vmlinux] [k] io_submit_sqesWith the kernel without your patch, I was looking for tag flush overhead but didn't find much:
0.02% io_uring [kernel.vmlinux] [k] blk_mq_free_plug_rqs
Outside of the peak worry with the patch, do you have a workload that we should test this on?
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
3.51% +2.84% [kernel.vmlinux] [k] io_issue_sqe 3.24% +1.35% [kernel.vmlinux] [k] io_submit_sqesWith the kernel without your patch, I was looking for tag flush overhead but didn't find much:
0.02% io_uring [kernel.vmlinux] [k] blk_mq_free_plug_rqsOutside of the peak worry with the patch, do you have a workload that we should test this on?
Thanks for the insights! I had tested simply 4 nvme drives with 1 thread, default everything:
./t/io_uring /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
Which appeared to show a very small improvement with this patch on my test vm. I'll test more to see where the tipping point is, and also see if there's any other ways to reduce time spent in io_issue_sqe.