Add plug based request allocation batching
Subsystems (3)
Jens Axboe
maintainerv2 cover Add plug based request allocation batching
Hi,
Even if the caller knows that N requests will be submitted, we still have to go all the way to tag allocation for each one. That's somewhat inefficient, when we could just grab as many tags as we need initially instead.
This small series allows request caching in the blk_plug. We add a new helper for passing in how many requests we expect to submit, blk_start_plug_nr_ios(), and then we can use that information to populate the cache on the first request allocation. Subsequent queue attempts can then just grab a pre-allocated request out of the cache rather than go into the tag allocation again.
This brings single core performance on my setup from:
taskset -c 0,16 t/io_uring -b512 -d128 -s32 -c32 -p1 -F1 -B1 -n2 /dev/nvme1n1 /dev/nvme2n1 Added file /dev/nvme1n1 (submitter 0)
Added file /dev/nvme2n1 (submitter 1)
polled=1, fixedbufs=1, register_files=1, buffered=0, QD=128
Engine=io_uring, sq_ring=128, cq_ring=256
submitter=0, tid=1206
submitter=1, tid=1207
IOPS=5806400, BW=2835MiB/s, IOS/call=32/32, inflight=(77 32)
IOPS=5860352, BW=2861MiB/s, IOS/call=32/31, inflight=(102 128)
IOPS=5844800, BW=2853MiB/s, IOS/call=32/32, inflight=(102 32)
to:
taskset -c 0,16 t/io_uring -b512 -d128 -s32 -c32 -p1 -F1 -B1 -n2 /dev/nvme1n1 /dev/nvme2n1 Added file /dev/nvme1n1 (submitter 0)
Added file /dev/nvme2n1 (submitter 1)
polled=1, fixedbufs=1, register_files=1, buffered=0, QD=128
Engine=io_uring, sq_ring=128, cq_ring=256
submitter=0, tid=1220
submitter=1, tid=1221
IOPS=6061248, BW=2959MiB/s, IOS/call=32/31, inflight=(114 82)
IOPS=6100672, BW=2978MiB/s, IOS/call=32/31, inflight=(128 128)
IOPS=6082496, BW=2969MiB/s, IOS/call=32/32, inflight=(77 38)
which is about a 4-5% improvement in single core performance. Looking at profiles, we're spending _less_ time in sbitmap even though we're doing more work.
V2:
- Remove hunk from another patch in patch 2
- Make the plugging limits block layer private
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v2 1/3 block: bump max plugged deferred size from 16 to 32
Particularly for NVMe with efficient deferred submission for many requests, there are nice benefits to be seen by bumping the default max plug count from 16 to 32. This is especially true for virtualized setups, where the submit part is more expensive. But can be noticed even on native hardware.
Reduce the multiple queue factor from 4 to 2, since we're changing the default size.
While changing it, move the defines into the block layer private header. These aren't values that anyone outside of the block layer uses, or should use.
--- block/blk-mq.c | 4 ++-- block/blk.h | 6 ++++++ include/linux/blkdev.h | 2 -- 3 files changed, 8 insertions(+), 4 deletions(-)
block/blk-mq.c +2 -2
@@ -2145,14 +2145,14 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq) }/* -* Allow 4x BLK_MAX_REQUEST_COUNT requests on plug queue for multiple +* Allow 2x BLK_MAX_REQUEST_COUNT requests on plug queue for multiple * queues. This is important for md arrays to benefit from merging * requests. */ static inline unsigned short blk_plug_max_rq_count (struct blk_plug *plug) {if (plug->multiple_queues) -return BLK_MAX_REQUEST_COUNT *4 ; +return BLK_MAX_REQUEST_COUNT *2 ;return BLK_MAX_REQUEST_COUNT ; }
block/blk.h +6 -0
@@ -222,6 +222,12 @@ bool blk_bio_list_merge(struct request_queue *q, struct list_head *list,void blk_account_io_start (struct request *req);void blk_account_io_done (struct request *req,u64 now); +/* +* Plug flush limits +*/ +#define BLK_MAX_REQUEST_COUNT 32 +#define BLK_PLUG_FLUSH_SIZE (128 * 1024) +/* * Internal elevator interface */
include/linux/blkdev.h +0 -2
@@ -727,8 +727,6 @@ struct blk_plug {bool multiple_queues;bool nowait; }; -#define BLK_MAX_REQUEST_COUNT 16 -#define BLK_PLUG_FLUSH_SIZE (128 * 1024)struct blk_plug_cb ;typedef void (*blk_plug_cb_fn )(struct blk_plug_cb *,bool );
Signature · 1 line
2.33.0
Re: [PATCH 1/3] block: bump max plugged deferred size from 16 to 32
Quoted 5 lines · Jens Axboe
Particularly for NVMe with efficient deferred submission for many requests, there are nice benefits to be seen by bumping the default max plug count from 16 to 32. This is especially true for virtualized setups, where the submit part is more expensive. But can be noticed even on native hardware.
It might be worth to throw in a comment how ths value was chosen.
Otherwise this looks good:
Jens Axboe
maintainerPATCH v2 2/3 block: pre-allocate requests if plug is started and is a batch
In reply to Jens AxboeThe caller typically has a good (or even exact) idea of how many requests it needs to submit. We can make the request/tag allocation a lot more efficient if we just allocate N requests/tags upfront when we queue the first bio from the batch.
Provide a new plug start helper that allows the caller to specify how many IOs are expected. This sets plug->nr_ios, and we can use that for smarter request allocation. The plug provides a holding spot for requests, and request allocation will check it before calling into the normal request allocation path.
The blk_finish_plug() is called, check if there are unused requests and free them. This should not happen in normal operations. The exception is if we get merging, then we may be left with requests that need freeing when done.
This raises the per-core performance on my setup from ~5.8M to ~6.1M IOPS.
--- block/blk-core.c | 47 ++++++++++++++++------------ block/blk-mq.c | 70 ++++++++++++++++++++++++++++++++++-------- block/blk-mq.h | 5 +++ include/linux/blk-mq.h | 5 ++- include/linux/blkdev.h | 15 ++++++++- 5 files changed, 109 insertions(+), 33 deletions(-)
block/blk-core.c +28 -19
@@ -1624,6 +1624,31 @@ int kblockd_mod_delayed_work_on(int cpu, struct delayed_work *dwork, }EXPORT_SYMBOL (kblockd_mod_delayed_work_on); +void blk_start_plug_nr_ios (struct blk_plug *plug,unsigned short nr_ios) +{ +struct task_struct *tsk = current; + +/* +* If this is a nested plug, don't actually assign it. +*/ +if (tsk->plug) +return ; + +INIT_LIST_HEAD (&plug->mq_list); + plug->cached_rq =NULL ; + plug->nr_ios = min_t(unsigned short, nr_ios, BLK_MAX_REQUEST_COUNT); + plug->rq_count =0 ; + plug->multiple_queues = false; + plug->nowait = false; +INIT_LIST_HEAD (&plug->cb_list); + +/* +* Store ordering should not be needed here, since a potential +* preempt will imply a full memory barrier +*/ + tsk->plug = plug; +} +/** * blk_start_plug - initialize blk_plug and track it inside the task_struct * @plug: The &struct blk_plug that needs to be initialized
@@ -1649,25 +1674,7 @@ EXPORT_SYMBOL(kblockd_mod_delayed_work_on);*/ void blk_start_plug (struct blk_plug *plug) { -struct task_struct *tsk = current; - -/* -* If this is a nested plug, don't actually assign it. -*/ -if (tsk->plug) -return ; - -INIT_LIST_HEAD (&plug->mq_list); -INIT_LIST_HEAD (&plug->cb_list); - plug->rq_count =0 ; - plug->multiple_queues = false; - plug->nowait = false; - -/* -* Store ordering should not be needed here, since a potential -* preempt will imply a full memory barrier -*/ - tsk->plug = plug; +blk_start_plug_nr_ios (plug,1 ); }EXPORT_SYMBOL (blk_start_plug);
@@ -1719,6 +1726,8 @@ void blk_flush_plug_list(struct blk_plug *plug, bool from_schedule)if (!list_empty (&plug->mq_list))blk_mq_flush_plug_list (plug, from_schedule); +if (unlikely (!from_schedule && plug->cached_rq)) +blk_mq_free_plug_rqs (plug); }/**
block/blk-mq.c +58 -12
126 lines not on this page · raw message
block/blk-mq.h +5 -0
@@ -125,6 +125,7 @@ extern int __blk_mq_register_dev(struct device *dev, struct request_queue *q);extern int blk_mq_sysfs_register (struct request_queue *q);extern void blk_mq_sysfs_unregister (struct request_queue *q);extern void blk_mq_hctx_kobj_init (struct blk_mq_hw_ctx *hctx); +void blk_mq_free_plug_rqs (struct blk_plug *plug);void blk_mq_release (struct request_queue *q);
@@ -152,6 +153,10 @@ struct blk_mq_alloc_data {unsigned int shallow_depth;unsigned int cmd_flags; +/* allocate multiple requests/tags in one go */ +unsigned int nr_tags; +struct request **cached_rq; +/* input & output parameter */ struct blk_mq_ctx *ctx;struct blk_mq_hw_ctx *hctx;
include/linux/blk-mq.h +4 -1
@@ -90,7 +90,10 @@ struct request {struct bio *bio;struct bio *biotail; -struct list_head queuelist; +union { +struct list_head queuelist; +struct request *rq_next; + };/* * The hash is used inside the scheduler, and killed once the
include/linux/blkdev.h +14 -1
@@ -722,10 +722,17 @@ extern void blk_set_queue_dying(struct request_queue *);*/ struct blk_plug {struct list_head mq_list;/* blk-mq requests */ -struct list_head cb_list;/* md requires an unplug callback */ + +/* if ios_left is > 1, we can batch tag/rq allocations */ +struct request *cached_rq; +unsigned short nr_ios; +unsigned short rq_count; +bool multiple_queues;bool nowait; + +struct list_head cb_list;/* md requires an unplug callback */ };struct blk_plug_cb ;
@@ -738,6 +745,7 @@ struct blk_plug_cb {extern struct blk_plug_cb *blk_check_plugged (blk_plug_cb_fn unplug,void *data,int size);extern void blk_start_plug (struct blk_plug *); +extern void blk_start_plug_nr_ios (struct blk_plug *,unsigned short );extern void blk_finish_plug (struct blk_plug *);extern void blk_flush_plug_list (struct blk_plug *,bool );
@@ -772,6 +780,11 @@ long nr_blockdev_pages(void);struct blk_plug { }; +static inline void blk_start_plug_nr_ios (struct blk_plug *plug, +unsigned short nr_ios) +{ +} +static inline void blk_start_plug (struct blk_plug *plug) { }
Signature · 1 line
2.33.0
Jens Axboe
maintainerPATCH v2 3/3 io_uring: inform block layer of how many requests we are submitting
In reply to Jens AxboeThe block layer can use this knowledge to make smarter decisions on how to handle the request, if it knows that N more may be coming. Switch to using blk_start_plug_nr_ios() to pass in that information.
--- fs/io_uring.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
fs/io_uring.c +3 -1
@@ -316,6 +316,7 @@ struct io_submit_state {bool plug_started;bool need_plug; +unsigned short submit_nr;struct blk_plug plug; };
@@ -7027,7 +7028,7 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req,if (state->need_plug && io_op_defs[opcode].plug) { state->plug_started = true; state->need_plug = false; -blk_start_plug (&state->plug); +blk_start_plug_nr_ios (&state->plug, state->submit_nr); } req->file = io_file_get(ctx, req, READ_ONCE(sqe->fd),
@@ -7148,6 +7149,7 @@ static void io_submit_state_start(struct io_submit_state *state, { state->plug_started = false; state->need_plug = max_ios >2 ; + state->submit_nr = max_ios;/* set only head, no need to init link_last in advance */ state->link.head =NULL ; }
Signature · 1 line
2.33.0