passthru block optimizations
Jens Axboe
maintainercover passthru block optimizations
Hi,
Currently passthru IO is slower than bdev O_DIRECT. One of the reasons is that we do two allocations for each IO:
- One alloc+free for the page array for mapping the data
- One alloc+free of the bio
Let passthru IO dip into the bio cache to eliminate that one, and use UIO_FASTIOV to gate whether we need to alloc+free the page array for mapping purposes.
This closes about half of the gap between passthru and bdev dio for me. If we can sanely wire up completion batching for passthru, then that would almost fully close the gap. Outside of that, the main missing feature for passthru is the ability to use registered buffers with io_uring, as the per-io get_user_pages() is a large cycle consumer as well.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/3 block: shrink rq_map_data a bit
We don't need full ints for several of these members. Change the page_order and nr_entries to unsigned shorts, and the true/false from_user and null_mapped to booleans.
This shrinks the struct from 32 to 24 bytes on 64-bit archs.
--- block/blk-map.c | 2 +- include/linux/blk-mq.h | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)
block/blk-map.c +1 -1
@@ -158,7 +158,7 @@ static int bio_copy_user_iov(struct request *rq, struct rq_map_data *map_data,bio_init (bio,NULL , bio->bi_inline_vecs, nr_pages,req_op (rq));if (map_data) { - nr_pages =1 << map_data->page_order; + nr_pages =1U << map_data->page_order; i = map_data->offset /PAGE_SIZE ; }while (len) {
include/linux/blk-mq.h +4 -4
@@ -964,11 +964,11 @@ blk_status_t blk_insert_cloned_request(struct request *rq);struct rq_map_data {struct page **pages; -int page_order; -int nr_entries;unsigned long offset; -int null_mapped; -int from_user; +unsigned short page_order; +unsigned short nr_entries; +bool null_mapped; +bool from_user; }; int blk_rq_map_user(struct request_queue *, struct request *,
Signature · 1 line
2.35.1
Jens Axboe
maintainerPATCH 2/3 block: enable bio caching use for passthru IO
In reply to Jens Axboebdev based polled O_DIRECT is currently quite a bit faster than passthru on the same device, and one of the reaons is that we're not able to use the bio caching for passthru IO.
If REQ_POLLED is set on the request, use the fs bio set for grabbing a bio from the caches, if available. This saves 5-6% of CPU over head for polled passthru IO.
--- block/blk-map.c | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-)
block/blk-map.c +25 -8
@@ -231,6 +231,16 @@ static int bio_copy_user_iov(struct request *rq, struct rq_map_data *map_data,return ret; } +static void bio_map_put (struct bio *bio) +{ +if (bio->bi_opf &REQ_ALLOC_CACHE ) { +bio_put (bio); + }else { +bio_uninit (bio); +kfree (bio); + } +} +static int bio_map_user_iov (struct request *rq,struct iov_iter *iter,gfp_t gfp_mask) {
@@ -243,10 +253,19 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,if (!iov_iter_count (iter))return -EINVAL ; - bio =bio_kmalloc (nr_vecs, gfp_mask); -if (!bio) -return -ENOMEM ; -bio_init (bio,NULL , bio->bi_inline_vecs, nr_vecs,req_op (rq)); +if (rq->cmd_flags &REQ_POLLED ) { +blk_opf_t opf = rq->cmd_flags |REQ_ALLOC_CACHE ; + + bio =bio_alloc_bioset (NULL , nr_vecs, opf, gfp_mask, + &fs_bio_set); +if (!bio) +return -ENOMEM ; + }else { + bio =bio_kmalloc (nr_vecs, gfp_mask); +if (!bio) +return -ENOMEM ; +bio_init (bio,NULL , bio->bi_inline_vecs, nr_vecs,req_op (rq)); + }while (iov_iter_count (iter)) {struct page **pages;
@@ -304,8 +323,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, out_unmap:bio_release_pages (bio, false); -bio_uninit (bio); -kfree (bio); +bio_map_put (bio);return ret; }
@@ -610,8 +628,7 @@ int blk_rq_unmap_user(struct bio *bio) next_bio = bio; bio = bio->bi_next; -bio_uninit (next_bio); -kfree (next_bio); +bio_map_put (next_bio); }return ret;
Signature · 1 line
2.35.1
Jens Axboe
maintainerPATCH 3/3 block: use on-stack page vec for <= UIO_FASTIOV
In reply to Jens AxboeAvoid a kmalloc+kfree for each page array, if we only have a few pages that are mapped. An alloc+free for each IO is quite expensive, and it's pretty pointless if we're only dealing with 1 or a few vecs.
Use UIO_FASTIOV like we do in other spots to set a sane limit for how big of an IO we want to avoid allocations for.
--- block/blk-map.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
block/blk-map.c +11 -3
@@ -268,12 +268,19 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, }while (iov_iter_count (iter)) { -struct page **pages; +struct page **pages, *stack_pages[UIO_FASTIOV ];ssize_t bytes;size_t offs, added =0 ;int npages; - bytes =iov_iter_get_pages_alloc (iter, &pages,LONG_MAX , &offs); +if (nr_vecs <ARRAY_SIZE (stack_pages)) { + pages = stack_pages; + bytes =iov_iter_get_pages (iter, pages,LONG_MAX , + nr_vecs, &offs); + }else { + bytes =iov_iter_get_pages_alloc (iter, &pages,LONG_MAX , + &offs); + }if (unlikely (bytes <=0 )) { ret = bytes ? bytes : -EFAULT ; goto out_unmap;
@@ -310,7 +317,8 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,*/ while (j < npages)put_page (pages[j++]); -kvfree (pages); +if (pages != stack_pages) +kvfree (pages);/* couldn't stuff something into bio? */ if (bytes)break ;
Signature · 1 line
2.35.1
Re: [PATCH 1/3] block: shrink rq_map_data a bit
In reply to Jens AxboeQuoted 7 lines · Jens Axboe
We don't need full ints for several of these members. Change the
page_orderandnr_entriesto unsigned shorts, and the true/falsefrom_userandnull_mappedto booleans.This shrinks the struct from 32 to 24 bytes on 64-bit archs.
Signed-off-by Jens Axboe <[email protected]>
Looks good.
-ck
Re: [PATCH 2/3] block: enable bio caching use for passthru IO
In reply to Jens AxboeQuoted 9 lines · Jens Axboe
bdev based polled
O_DIRECTis currently quite a bit faster than passthru on the same device, and one of the reaons is that we're not able to use the bio caching for passthru IO.If
REQ_POLLEDis set on the request, use the fs bio set for grabbing a bio from the caches, if available. This saves 5-6% of CPU over head for polled passthru IO.Signed-off-by Jens Axboe <[email protected]>
Looks good.
-ck
Re: [PATCH 3/3] block: use on-stack page vec for <= UIO_FASTIOV
In reply to Jens AxboeQuoted 4 lines · Jens Axboe
Avoid a kmalloc+kfree for each page array, if we only have a few pages that are mapped. An alloc+free for each IO is quite expensive, and it's pretty pointless if we're only dealing with 1 or a few vecs.
Quoted 8 lines
Use
UIO_FASTIOVlike we do in other spots to set a sane limit for how big of an IO we want to avoid allocations for.Signed-off-by Jens Axboe <[email protected]>--- block/blk-map.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
Looks good.
-ck
Re: [PATCH 2/3] block: enable bio caching use for passthru IO
In reply to Jens AxboeQuoted 7 lines · Jens Axboe
bdev based polled
O_DIRECTis currently quite a bit faster than passthru on the same device, and one of the reaons is that we're not able to use the bio caching for passthru IO.If
REQ_POLLEDis set on the request, use the fs bio set for grabbing a bio from the caches, if available. This saves 5-6% of CPU over head for polled passthru IO.
For passthru path, bio is always freed in the task-context (and not in irq) so must this be tied to polled-io only?
Jens Axboe
maintainerRe: [PATCH 2/3] block: enable bio caching use for passthru IO
Quoted 11 lines · Kanchan Joshi
Quoted 7 lines · Jens Axboe
bdev based polled
O_DIRECTis currently quite a bit faster than passthru on the same device, and one of the reaons is that we're not able to use the bio caching for passthru IO.If
REQ_POLLEDis set on the request, use the fs bio set for grabbing a bio from the caches, if available. This saves 5-6% of CPU over head for polled passthru IO.For passthru path, bio is always freed in the task-context (and not in irq) so must this be tied to polled-io only?
Right, that's why it's tied to polled. If polling gets cleared, then it will be freed normally on completion rather than inserted into the cache.
I do have patches for irq bio caching too, that'll work fine with io_uring:
I'll post that too, would be nice if we can figure out a clean way to do this. I have posted it before, iirc.
Signature · 1 line
Jens Axboe