Enable bio recycling for polled IO
Subsystems (4)
Jens Axboe
maintainerv4 cover Enable bio recycling for polled IO
Hi,
This is v4 of this patchset, and Yet Another method of achieving the same goal. This one moves into the direction of my old cpu-alloc-cache branch, where the caches are just per-cpu. The trouble with those is that we need to make this specific to polled IO to lose the IRQ safety of them, otherwise it's not a real win and we're better off just using the slab allocator smarts. This is combined with Christoph's idea to make it per bio_set, and retains the flagging of the kiocb for having the IO issuer tell the below layer whether the cache can be safely used or not.
Another change from last is that we can now grossly simplify the io_uring side, as we don't need locking for the cache and async retries are no longer interesting there. This is combined with a block layer change that clears BIO_PERCPU_CACHE if we clear the HIPRI flag.
The tldr; here is that we get about a 10% bump in polled performance with this patchset, as we can recycle bio structures essentially for free. Outside of that, explanations in each patch. I've also got an iomap patch, but trying to keep this single user until there's agreement on the direction.
Against for-5.15/io_uring, and can also be found in my io_uring-bio-cache.4 branch.
block/bio.c | 170 +++++++++++++++++++++++++++++++++---- block/blk-core.c | 5 +- fs/block_dev.c | 6 +- fs/io_uring.c | 2 +- include/linux/bio.h | 23 +++-- include/linux/blk_types.h | 1 + include/linux/cpuhotplug.h | 1 + include/linux/fs.h | 2 + 8 files changed, 185 insertions(+), 25 deletions(-)
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v4 1/6 bio: optimize initialization of a bio
The memset() used is measurably slower in targeted benchmarks. Get rid of it and fill in the bio manually, in a separate helper.
--- block/bio.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
block/bio.c +30 -1
@@ -238,6 +238,35 @@ static void bio_free(struct bio *bio) } } +static inline void __bio_init (struct bio *bio) +{ + bio->bi_next =NULL ; + bio->bi_bdev =NULL ; + bio->bi_opf =0 ; + bio->bi_flags = bio->bi_ioprio = bio->bi_write_hint =0 ; + bio->bi_status =0 ; + bio->bi_iter.bi_sector =0 ; + bio->bi_iter.bi_size =0 ; + bio->bi_iter.bi_idx =0 ; + bio->bi_iter.bi_bvec_done =0 ; + bio->bi_end_io =NULL ; + bio->bi_private =NULL ; +#ifdef CONFIG_BLK_CGROUP + bio->bi_blkg =NULL ; + bio->bi_issue.value =0 ; +#ifdef CONFIG_BLK_CGROUP_IOCOST + bio->bi_iocost_cost =0 ; +#endif +#endif +#ifdef CONFIG_BLK_INLINE_ENCRYPTION + bio->bi_crypt_context =NULL ; +#endif +#ifdef CONFIG_BLK_DEV_INTEGRITY + bio->bi_integrity =NULL ; +#endif + bio->bi_vcnt =0 ; +} +/* * Users of this function have their own bio allocation. Subsequently, * they must remember to pair any call to bio_init() with bio_uninit()
@@ -246,7 +275,7 @@ static void bio_free(struct bio *bio)void bio_init (struct bio *bio,struct bio_vec *table,unsigned short max_vecs) { -memset (bio,0 ,sizeof (*bio)); +__bio_init (bio);atomic_set (&bio->__bi_remaining,1 );atomic_set (&bio->__bi_cnt,1 );
Signature · 1 line
2.32.0
Re: [PATCH 1/6] bio: optimize initialization of a bio
Quoted 2 lines · Jens Axboe
The
memset()used is measurably slower in targeted benchmarks. Get rid of it and fill in the bio manually, in a separate helper.
If you have some numbers if would be great to throw them in here.
Quoted 1 line
+static inline void
__bio_init(struct bio *bio)
Why is this split from bio_init and what are the criteria where an initialization goes?
block/bio.c static void bio_free(struct bio *bio) · line 246
Quoted 1 line
block/bio.c +1
-0+ bio->bi_flags = bio->bi_ioprio = bio->bi_write_hint =0 ;
Please keep each initialization on a separate line.
Jens Axboe
maintainerRe: [PATCH 1/6] bio: optimize initialization of a bio
Quoted 5 lines · Christoph Hellwig
Quoted 2 lines · Jens Axboe
The
memset()used is measurably slower in targeted benchmarks. Get rid of it and fill in the bio manually, in a separate helper.If you have some numbers if would be great to throw them in here.
It's about 1% of the overhead of the alloc after the cache, which comes later in the series.
Percent│ return __underlying_memset(p, c, size);
│ lea 0x8(%r8),%rdi
│ bio_alloc_kiocb():
2.18 │ cmove %rax,%r9
│ memset():
│ mov %r8,%rcx
│ and $0xfffffffffffffff8,%rdi
│ movq $0x0,(%r8)
│ sub %rdi,%rcx
│ add $0x60,%ecx
│ shr $0x3,%ecx
55.02 │ rep stos %rax,%es:(%rdi)This is on AMD, might look different on Intel, the manual clear seems like a nice win on both. As a minor detail, avoids things like re-setting bio->bi_pool for cached entries, as it never changes.
Quoted 4 lines
Quoted 1 line
+static inline void
__bio_init(struct bio *bio)Why is this split from
bio_initand what are the criteria where an initialization goes?
Got rid of the helper.
Quoted 3 lines
block/bio.c static void bio_free(struct bio *bio) · line 246
Quoted 1 line
block/bio.c +1
-0+ bio->bi_flags = bio->bi_ioprio = bio->bi_write_hint =0 ;Please keep each initialization on a separate line.
Done
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v4 2/6 fs: add kiocb alloc cache flag
In reply to Jens AxboeIf this kiocb can safely use the polled bio allocation cache, then this flag must be set.
--- include/linux/fs.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/fs.h +2 -0
@@ -319,6 +319,8 @@ enum rw_hint {/* iocb->ki_waitq is valid */ #define IOCB_WAITQ (1 << 19)#define IOCB_NOIO (1 << 20) +/* can use bio alloc cache */ +#define IOCB_ALLOC_CACHE (1 << 21)struct kiocb {struct file *ki_filp;
Signature · 1 line
2.32.0
Re: [PATCH 2/6] fs: add kiocb alloc cache flag
Quoted 2 lines · Jens Axboe
If this kiocb can safely use the polled bio allocation cache, then this flag must be set.
Looks fine, but it might be worth to define the semantics a little bit better.
Jens Axboe
maintainerRe: [PATCH 2/6] fs: add kiocb alloc cache flag
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Jens Axboe
If this kiocb can safely use the polled bio allocation cache, then this flag must be set.
Looks fine, but it might be worth to define the semantics a little bit better.
Sure, I'll expand that explanation a bit.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v4 3/6 bio: add allocation cache abstraction
In reply to Jens AxboeAdd a per-cpu bio_set cache for bio allocations, enabling us to quickly recycle them instead of going through the slab allocator. This cache isn't IRQ safe, and hence is only really suitable for polled IO.
Very simple - keeps a count of bio's in the cache, and maintains a max of 512 with a slack of 64. If we get above max + slack, we drop slack number of bio's.
--- block/bio.c | 139 +++++++++++++++++++++++++++++++++---- include/linux/bio.h | 23 ++++-- include/linux/blk_types.h | 1 + include/linux/cpuhotplug.h | 1 + 4 files changed, 144 insertions(+), 20 deletions(-)
block/bio.c +124 -15
@@ -25,6 +25,11 @@#include "blk.h" #include "blk-rq-qos.h" +struct bio_alloc_cache { +struct bio_list free_list; +unsigned int nr; +}; + static struct biovec_slab { int nr_vecs; char *name;
@@ -620,6 +625,69 @@ void guard_bio_eod(struct bio *bio)bio_truncate (bio, maxsector <<9 ); } +#define ALLOC_CACHE_MAX 512 +#define ALLOC_CACHE_SLACK 64 + +static void bio_alloc_cache_prune (struct bio_alloc_cache *cache, +unsigned int nr) +{ +struct bio *bio; +unsigned int i; + + i =0 ; +while ((bio =bio_list_pop (&cache->free_list)) !=NULL ) { + cache->nr--; +bio_free (bio); +if (++i == nr) +break ; + } +} + +static int bio_cpu_dead (unsigned int cpu,struct hlist_node *node) +{ +struct bio_set *bs; + + bs = hlist_entry_safe(node, struct bio_set, cpuhp_dead); +if (bs->cache) { +struct bio_alloc_cache *cache =per_cpu_ptr (bs->cache, cpu); + +bio_alloc_cache_prune (cache,-1U ); + } +return 0 ; +} + +static void bio_alloc_cache_destroy (struct bio_set *bs) +{ +int cpu; + +if (!bs->cache) +return ; + +preempt_disable (); +cpuhp_state_remove_instance_nocalls (CPUHP_BIO_DEAD , &bs->cpuhp_dead); +for_each_possible_cpu (cpu) { +struct bio_alloc_cache *cache; + + cache =per_cpu_ptr (bs->cache, cpu); +bio_alloc_cache_prune (cache,-1U ); + } +preempt_enable (); +free_percpu (bs->cache); +} + +static inline bool __bio_put (struct bio *bio) +{ +if (!bio_flagged (bio,BIO_REFFED )) +return true; + +BIO_BUG_ON (!atomic_read (&bio->__bi_cnt)); + +/* +* last put frees it +*/ +return atomic_dec_and_test (&bio->__bi_cnt); +} +/** * bio_put - release a reference to a bio * @bio: bio to release reference to
@@ -630,16 +698,21 @@ void guard_bio_eod(struct bio *bio)**/ void bio_put (struct bio *bio) { -if (!bio_flagged (bio,BIO_REFFED )) -bio_free (bio); -else { -BIO_BUG_ON (!atomic_read (&bio->__bi_cnt)); +if (unlikely (!__bio_put (bio))) +return ; -/* -* last put frees it -*/ -if (atomic_dec_and_test (&bio->__bi_cnt)) -bio_free (bio); +if (bio_flagged (bio,BIO_PERCPU_CACHE )) { +struct bio_alloc_cache *cache; + +bio_uninit (bio); + cache =per_cpu_ptr (bio->bi_pool->cache,get_cpu ()); +bio_list_add_head (&cache->free_list, bio); + cache->nr++; +if (cache->nr >ALLOC_CACHE_MAX +ALLOC_CACHE_SLACK ) +bio_alloc_cache_prune (cache,ALLOC_CACHE_SLACK ); +put_cpu (); + }else { +bio_free (bio); } }EXPORT_SYMBOL (bio_put);
@@ -1531,6 +1604,7 @@ int biovec_init_pool(mempool_t *pool, int pool_entries)*/ void bioset_exit (struct bio_set *bs) { +bio_alloc_cache_destroy (bs);if (bs->rescue_workqueue)destroy_workqueue (bs->rescue_workqueue); bs->rescue_workqueue =NULL ;
@@ -1592,12 +1666,18 @@ int bioset_init(struct bio_set *bs, biovec_init_pool(&bs->bvec_pool, pool_size)) goto bad; -if (!(flags &BIOSET_NEED_RESCUER )) -return 0 ; - - bs->rescue_workqueue =alloc_workqueue ("bioset" ,WQ_MEM_RECLAIM ,0 ); -if (!bs->rescue_workqueue) - goto bad; +if (flags &BIOSET_NEED_RESCUER ) { + bs->rescue_workqueue =alloc_workqueue ("bioset" , +WQ_MEM_RECLAIM ,0 ); +if (!bs->rescue_workqueue) + goto bad; + } +if (flags &BIOSET_PERCPU_CACHE ) { + bs->cache = alloc_percpu(struct bio_alloc_cache); +if (!bs->cache) + goto bad; +cpuhp_state_add_instance_nocalls (CPUHP_BIO_DEAD , &bs->cpuhp_dead); + }return 0 ; bad:
@@ -1624,6 +1704,32 @@ int bioset_init_from_src(struct bio_set *bs, struct bio_set *src) }EXPORT_SYMBOL (bioset_init_from_src); +struct bio *bio_alloc_kiocb (struct kiocb *kiocb,gfp_t gfp, +unsigned short nr_vecs,struct bio_set *bs) +{ +struct bio_alloc_cache *cache =NULL ; +struct bio *bio; + +if (!(kiocb->ki_flags &IOCB_ALLOC_CACHE ) || nr_vecs >BIO_INLINE_VECS ) + goto normal_alloc; + + cache =per_cpu_ptr (bs->cache,get_cpu ()); + bio =bio_list_pop (&cache->free_list); +if (bio) { + cache->nr--; +put_cpu (); +bio_init (bio, nr_vecs ? bio->bi_inline_vecs :NULL , nr_vecs); +bio_set_flag (bio,BIO_PERCPU_CACHE ); +return bio; + } +put_cpu (); +normal_alloc: + bio =bio_alloc_bioset (gfp, nr_vecs, bs); +if (cache) +bio_set_flag (bio,BIO_PERCPU_CACHE ); +return bio; +} +static int __init init_bio (void ) {int i;
@@ -1638,6 +1744,9 @@ static int __init init_bio(void) SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL); } +cpuhp_setup_state_multi (CPUHP_BIO_DEAD ,"block/bio:dead" ,NULL , + bio_cpu_dead); +if (bioset_init (&fs_bio_set,BIO_POOL_SIZE ,0 ,BIOSET_NEED_BVECS ))panic ("bio: can't allocate bios\n" );
include/linux/bio.h +18 -5
@@ -401,6 +401,7 @@ static inline struct bio *bio_next_split(struct bio *bio, int sectors,enum {BIOSET_NEED_BVECS =BIT (0 ),BIOSET_NEED_RESCUER =BIT (1 ), +BIOSET_PERCPU_CACHE =BIT (2 ), };extern int bioset_init (struct bio_set *,unsigned int ,unsigned int ,int flags);extern void bioset_exit (struct bio_set *);
@@ -409,6 +410,8 @@ extern int bioset_init_from_src(struct bio_set *bs, struct bio_set *src);struct bio *bio_alloc_bioset (gfp_t gfp,unsigned short nr_iovecs,struct bio_set *bs); +struct bio *bio_alloc_kiocb (struct kiocb *kiocb,gfp_t ,unsigned short nr_vecs, +struct bio_set *bs);struct bio *bio_kmalloc (gfp_t gfp_mask,unsigned short nr_iovecs);extern void bio_put (struct bio *);
@@ -652,18 +655,22 @@ static inline struct bio *bio_list_peek(struct bio_list *bl)return bl->head; } -static inline struct bio *bio_list_pop (struct bio_list *bl) +static inline void bio_list_del_head (struct bio_list *bl,struct bio *head) { -struct bio *bio = bl->head; - -if (bio) { +if (head) { bl->head = bl->head->bi_next;if (!bl->head) bl->tail =NULL ; - bio->bi_next =NULL ; + head->bi_next =NULL ; } +} + +static inline struct bio *bio_list_pop (struct bio_list *bl) +{ +struct bio *bio = bl->head; +bio_list_del_head (bl, bio);return bio; }
@@ -699,6 +706,12 @@ struct bio_set {struct kmem_cache *bio_slab;unsigned int front_pad; +/* +* per-cpu bio alloc cache and notifier +*/ +struct bio_alloc_cache __percpu *cache; +struct hlist_node cpuhp_dead; +mempool_t bio_pool;mempool_t bvec_pool;#if defined(CONFIG_BLK_DEV_INTEGRITY )
include/linux/blk_types.h +1 -0
@@ -301,6 +301,7 @@ enum {BIO_TRACKED ,/* set if bio goes through the rq_qos path */ BIO_REMAPPED ,BIO_ZONE_WRITE_LOCKED ,/* Owns a zoned device zone write lock */ +BIO_PERCPU_CACHE ,/* can participate in per-cpu alloc cache */ BIO_FLAG_LAST };
include/linux/cpuhotplug.h +1 -0
@@ -46,6 +46,7 @@ enum cpuhp_state {CPUHP_ARM_OMAP_WAKE_DEAD ,CPUHP_IRQ_POLL_DEAD ,CPUHP_BLOCK_SOFTIRQ_DEAD , +CPUHP_BIO_DEAD ,CPUHP_ACPI_CPUDRV_DEAD ,CPUHP_S390_PFAULT_DEAD , CPUHP_BLK_MQ_DEAD,
Signature · 1 line
2.32.0
Re: [PATCH 3/6] bio: add allocation cache abstraction
block/bio.c void guard_bio_eod(struct bio *bio) · line 637
Quoted 4 lines · Jens Axboe
block/bio.c +4
-0+struct bio *bio; +unsigned int i; + + i =0 ;
Initialize at declaration time?
block/bio.c void guard_bio_eod(struct bio *bio) · line 689
Quoted 12 lines
block/bio.c +12
-0+static inline bool __bio_put (struct bio *bio) +{ +if (!bio_flagged (bio,BIO_REFFED )) +return true; + +BIO_BUG_ON (!atomic_read (&bio->__bi_cnt)); + +/* +* last put frees it +*/ +return atomic_dec_and_test (&bio->__bi_cnt); +}
Please avoid this helper, we can trivially do the check inside of bio_put:
if (bio_flagged(bio, BIO_REFFED)) {
BIO_BUG_ON(!atomic_read(&bio->__bi_cnt));
if (!atomic_dec_and_test(&bio->__bi_cnt))
return;
}block/bio.c void guard_bio_eod(struct bio *bio) · line 711
Quoted 9 lines
block/bio.c +8
-1-bio_free (bio); +if (bio_flagged (bio,BIO_PERCPU_CACHE )) { +struct bio_alloc_cache *cache; + +bio_uninit (bio); + cache =per_cpu_ptr (bio->bi_pool->cache,get_cpu ()); +bio_list_add_head (&cache->free_list, bio); + cache->nr++; +if (cache->nr >ALLOC_CACHE_MAX +ALLOC_CACHE_SLACK )
Folding the increment as a prefix here would make the increment and test semantics a little more obvious.
Quoted 14 lines
block/bio.c +14
-0+struct bio *bio_alloc_kiocb (struct kiocb *kiocb,gfp_t gfp, +unsigned short nr_vecs,struct bio_set *bs) +{ +struct bio_alloc_cache *cache =NULL ; +struct bio *bio; + +if (!(kiocb->ki_flags &IOCB_ALLOC_CACHE ) || nr_vecs >BIO_INLINE_VECS ) + goto normal_alloc; + + cache =per_cpu_ptr (bs->cache,get_cpu ()); + bio =bio_list_pop (&cache->free_list); +if (bio) { + cache->nr--; +put_cpu ();
block/bio.c int bioset_init_from_src(struct bio_set *bs, struct bio_set *src) · line 1730
Quoted 10 lines
block/bio.c +10
-0+bio_init (bio, nr_vecs ? bio->bi_inline_vecs :NULL , nr_vecs); +bio_set_flag (bio,BIO_PERCPU_CACHE ); +return bio; + } +put_cpu (); +normal_alloc: + bio =bio_alloc_bioset (gfp, nr_vecs, bs); +if (cache) +bio_set_flag (bio,BIO_PERCPU_CACHE ); +return bio;
The goto here is pretty obsfucating and adds an extra patch to the fast path.
Also I don't think we need the gfp argument here at all - it should always be GFP_KERNEL. In fact I plan to kill these arguments from much of the block layer as the places that need NOIO of NOFS semantics can and should move to set that in the per-thread context.
Quoted 13 lines
include/linux/bio.h +3
-5-static inline struct bio *bio_list_pop (struct bio_list *bl) +static inline void bio_list_del_head (struct bio_list *bl,struct bio *head) { -struct bio *bio = bl->head; - -if (bio) { +if (head) { bl->head = bl->head->bi_next;if (!bl->head) bl->tail =NULL ; - bio->bi_next =NULL ; + head->bi_next =NULL ;
include/linux/bio.h static inline struct bio *bio_list_peek(struct bio_list *bl) · line 675
Quoted 10 lines
include/linux/bio.h +6
-0} +} + +static inline struct bio *bio_list_pop (struct bio_list *bl) +{ +struct bio *bio = bl->head; +bio_list_del_head (bl, bio);return bio; }
No need for this change.
include/linux/bio.h struct bio_set { · line 714
Quoted 12 lines
include/linux/bio.h +6
-0@@ -699,6 +706,12 @@ struct bio_set {struct kmem_cache *bio_slab;unsigned int front_pad; +/* +* per-cpu bio alloc cache and notifier +*/ +struct bio_alloc_cache __percpu *cache; +struct hlist_node cpuhp_dead; +
I'd keep the hotplug list entry at the end instead of bloating the cache line used in the fast path.
Jens Axboe
maintainerRe: [PATCH 3/6] bio: add allocation cache abstraction
Quoted 7 lines · Christoph Hellwig
block/bio.c void guard_bio_eod(struct bio *bio) · line 637
Quoted 4 lines · Jens Axboe
block/bio.c +4
-0+struct bio *bio; +unsigned int i; + + i =0 ;Initialize at declaration time?
Sure, done.
Quoted 15 lines
block/bio.c void guard_bio_eod(struct bio *bio) · line 689
Quoted 12 lines
block/bio.c +12
-0+static inline bool __bio_put (struct bio *bio) +{ +if (!bio_flagged (bio,BIO_REFFED )) +return true; + +BIO_BUG_ON (!atomic_read (&bio->__bi_cnt)); + +/* +* last put frees it +*/ +return atomic_dec_and_test (&bio->__bi_cnt); +}Please avoid this helper, we can trivially do the check inside of
bio_put:
Quoted 5 lines
if (bio_flagged(bio, BIO_REFFED)) {
BIO_BUG_ON(!atomic_read(&bio->__bi_cnt));
if (!atomic_dec_and_test(&bio->__bi_cnt))
return;
}Done
Quoted 4 lines
Quoted 4 lines
block/bio.c +3
-1-bio_free (bio); +if (bio_flagged (bio,BIO_PERCPU_CACHE )) { +struct bio_alloc_cache *cache; +
Quoted 8 lines
block/bio.c void guard_bio_eod(struct bio *bio) · line 711
Quoted 5 lines
block/bio.c +5
-0+bio_uninit (bio); + cache =per_cpu_ptr (bio->bi_pool->cache,get_cpu ()); +bio_list_add_head (&cache->free_list, bio); + cache->nr++; +if (cache->nr >ALLOC_CACHE_MAX +ALLOC_CACHE_SLACK )Folding the increment as a prefix here would make the increment and test semantics a little more obvious.
I don't really care that deeply about it, but I generally prefer keeping them separate as it makes it easier to read (for me). But I'll change it.
Quoted 19 lines
Quoted 19 lines
block/bio.c +19
-0+struct bio *bio_alloc_kiocb (struct kiocb *kiocb,gfp_t gfp, +unsigned short nr_vecs,struct bio_set *bs) +{ +struct bio_alloc_cache *cache =NULL ; +struct bio *bio; + +if (!(kiocb->ki_flags &IOCB_ALLOC_CACHE ) || nr_vecs >BIO_INLINE_VECS ) + goto normal_alloc; + + cache =per_cpu_ptr (bs->cache,get_cpu ()); + bio =bio_list_pop (&cache->free_list); +if (bio) { + cache->nr--; +put_cpu (); +bio_init (bio, nr_vecs ? bio->bi_inline_vecs :NULL , nr_vecs); +bio_set_flag (bio,BIO_PERCPU_CACHE ); +return bio; + } +put_cpu ();
Quoted 8 lines
block/bio.c int bioset_init_from_src(struct bio_set *bs, struct bio_set *src) · line 1730
Quoted 5 lines
block/bio.c +5
-0+normal_alloc: + bio =bio_alloc_bioset (gfp, nr_vecs, bs); +if (cache) +bio_set_flag (bio,BIO_PERCPU_CACHE ); +return bio;The goto here is pretty obsfucating and adds an extra patch to the fast path.
I don't agree, and it's not the fast path - the fast path is popping off a bio off the list, not hitting the allocator.
Quoted 4 lines
Also I don't think we need the gfp argument here at all - it should always be
GFP_KERNEL. In fact I plan to kill these arguments from much of the block layer as the places that need NOIO of NOFS semantics can and should move to set that in the per-thread context.
Yeah, I actually kept it on purpose for future users, but let's just kill it as both potential use cases I have use GFP_KERNEL anyway.
Quoted 20 lines
Quoted 20 lines
include/linux/bio.h +8
-5-static inline struct bio *bio_list_pop (struct bio_list *bl) +static inline void bio_list_del_head (struct bio_list *bl,struct bio *head) { -struct bio *bio = bl->head; - -if (bio) { +if (head) { bl->head = bl->head->bi_next;if (!bl->head) bl->tail =NULL ; - bio->bi_next =NULL ; + head->bi_next =NULL ; } +} + +static inline struct bio *bio_list_pop (struct bio_list *bl) +{ +struct bio *bio = bl->head;
Quoted 5 lines
include/linux/bio.h static inline struct bio *bio_list_peek(struct bio_list *bl) · line 675
Quoted 3 lines
include/linux/bio.h +1
-0+bio_list_del_head (bl, bio);return bio; }No need for this change.
Leftover from earlier series, killed it now.
Quoted 4 lines
Quoted 4 lines
include/linux/bio.h +0
-0@@ -699,6 +706,12 @@ struct bio_set {struct kmem_cache *bio_slab;unsigned int front_pad;
Quoted 9 lines
include/linux/bio.h struct bio_set { · line 714
Quoted 6 lines
include/linux/bio.h +6
-0+/* +* per-cpu bio alloc cache and notifier +*/ +struct bio_alloc_cache __percpu *cache; +struct hlist_node cpuhp_dead; +I'd keep the hotplug list entry at the end instead of bloating the cache line used in the fast path.
Good point, moved to the end.
Signature · 1 line
Jens Axboe
Re: [PATCH 3/6] bio: add allocation cache abstraction
Quoted 19 lines · Jens Axboe
Quoted 10 lines
Quoted 10 lines
(quoted diff) +10
-0+ cache = per_cpu_ptr(bs->cache, get_cpu()); + bio = bio_list_pop(&cache->free_list); + if (bio) { + cache->nr--; + put_cpu(); + bio_init(bio, nr_vecs ? bio->bi_inline_vecs : NULL, nr_vecs); + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio; + } + put_cpu();Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+normal_alloc: + bio = bio_alloc_bioset(gfp, nr_vecs, bs); + if (cache) + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio;The goto here is pretty obsfucating and adds an extra patch to the fast path.
Quoted 2 lines
I don't agree, and it's not the fast path - the fast path is popping off a bio off the list, not hitting the allocator.
Oh, I see you special case the list pop return now. Still seems much easier to follow to avoid the goto, the cache initialization and the conditional in the no bio found in the list case (see patch below).
block/bio.c +3 -5
@@ -1707,11 +1707,11 @@ EXPORT_SYMBOL(bioset_init_from_src);struct bio *bio_alloc_kiocb (struct kiocb *kiocb,gfp_t gfp,unsigned short nr_vecs,struct bio_set *bs) { -struct bio_alloc_cache *cache =NULL ; +struct bio_alloc_cache *cache;struct bio *bio;if (!(kiocb->ki_flags &IOCB_ALLOC_CACHE ) || nr_vecs >BIO_INLINE_VECS ) - goto normal_alloc; +return bio_alloc_bioset (gfp, nr_vecs, bs); cache =per_cpu_ptr (bs->cache,get_cpu ()); bio =bio_list_pop (&cache->free_list);
@@ -1723,10 +1723,8 @@ struct bio *bio_alloc_kiocb(struct kiocb *kiocb, gfp_t gfp,return bio; }put_cpu (); -normal_alloc: bio =bio_alloc_bioset (gfp, nr_vecs, bs); -if (cache) -bio_set_flag (bio,BIO_PERCPU_CACHE ); +bio_set_flag (bio,BIO_PERCPU_CACHE );return bio; }
Jens Axboe
maintainerRe: [PATCH 3/6] bio: add allocation cache abstraction
Quoted 46 lines · Christoph Hellwig
Quoted 19 lines · Jens Axboe
Quoted 10 lines
Quoted 10 lines
(quoted diff) +10
-0+ cache = per_cpu_ptr(bs->cache, get_cpu()); + bio = bio_list_pop(&cache->free_list); + if (bio) { + cache->nr--; + put_cpu(); + bio_init(bio, nr_vecs ? bio->bi_inline_vecs : NULL, nr_vecs); + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio; + } + put_cpu();Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+normal_alloc: + bio = bio_alloc_bioset(gfp, nr_vecs, bs); + if (cache) + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio;The goto here is pretty obsfucating and adds an extra patch to the fast path.
Quoted 2 lines
I don't agree, and it's not the fast path - the fast path is popping off a bio off the list, not hitting the allocator.
Oh, I see you special case the list pop return now. Still seems much easier to follow to avoid the goto, the cache initialization and the conditional in the no bio found in the list case (see patch below).
block/bio.c +2
-2@@ -1707,11 +1707,11 @@ EXPORT_SYMBOL(bioset_init_from_src);struct bio *bio_alloc_kiocb (struct kiocb *kiocb,gfp_t gfp,unsigned short nr_vecs,struct bio_set *bs) { -struct bio_alloc_cache *cache =NULL ; +struct bio_alloc_cache *cache;struct bio *bio;if (!(kiocb->ki_flags &IOCB_ALLOC_CACHE ) || nr_vecs >BIO_INLINE_VECS ) - goto normal_alloc; +return bio_alloc_bioset (gfp, nr_vecs, bs); cache =per_cpu_ptr (bs->cache,get_cpu ()); bio =bio_list_pop (&cache->free_list);
block/bio.c struct bio *bio_alloc_kiocb(struct kiocb *kiocb, gfp_t gfp, · line 1729
Quoted 10 lines
block/bio.c +1
-3@@ -1723,10 +1723,8 @@ struct bio *bio_alloc_kiocb(struct kiocb *kiocb, gfp_t gfp,return bio; }put_cpu (); -normal_alloc: bio =bio_alloc_bioset (gfp, nr_vecs, bs); -if (cache) -bio_set_flag (bio,BIO_PERCPU_CACHE ); +bio_set_flag (bio,BIO_PERCPU_CACHE );return bio; }
Sure, if that'll shut it down, I'll make the edit :-)
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v4 4/6 block: clear BIO_PERCPU_CACHE flag if polling isn't supported
In reply to Jens AxboeThe bio alloc cache relies on the fact that a polled bio will complete in process context, clear the cacheable flag if we disable polling for a given bio.
--- block/blk-core.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
block/blk-core.c +4 -1
@@ -833,8 +833,11 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) } } -if (!test_bit (QUEUE_FLAG_POLL , &q->queue_flags)) +if (!test_bit (QUEUE_FLAG_POLL , &q->queue_flags)) { +/* can't support alloc cache if we turn off polling */ +bio_clear_flag (bio,BIO_PERCPU_CACHE ); bio->bi_opf &= ~REQ_HIPRI ; + }switch (bio_op (bio)) {case REQ_OP_DISCARD :
Signature · 1 line
2.32.0
Jens Axboe
maintainerPATCH v4 5/6 io_uring: enable use of bio alloc cache
In reply to Jens AxboeMark polled IO as being safe for dipping into the bio allocation cache, in case the targeted bio_set has it enabled.
This brings an IOPOLL gen2 Optane QD=128 workload from ~3.0M IOPS to ~3.3M IOPS.
--- fs/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io_uring.c +1 -1
@@ -2667,7 +2667,7 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) !kiocb->ki_filp->f_op->iopoll)return -EOPNOTSUPP ; - kiocb->ki_flags |=IOCB_HIPRI ; + kiocb->ki_flags |=IOCB_HIPRI |IOCB_ALLOC_CACHE ; kiocb->ki_complete = io_complete_rw_iopoll; req->iopoll_completed =0 ; }else {
Signature · 1 line
2.32.0
Jens Axboe
maintainerPATCH v4 6/6 block: enable use of bio allocation cache
In reply to Jens AxboeInitialize the bio_set used for IO with per-cpu bio caching enabled, and use the new bio_alloc_kiocb() helper to dip into that cache.
--- fs/block_dev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
fs/block_dev.c +4 -2
@@ -385,7 +385,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, (bdev_logical_block_size(bdev) - 1))return -EINVAL ; - bio =bio_alloc_bioset (GFP_KERNEL , nr_pages, &blkdev_dio_pool); + bio =bio_alloc_kiocb (iocb,GFP_KERNEL , nr_pages, &blkdev_dio_pool); dio = container_of(bio, struct blkdev_dio, bio); dio->is_sync = is_sync =is_sync_kiocb (iocb);
@@ -513,7 +513,9 @@ blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter)static __init int blkdev_init (void ) { -return bioset_init (&blkdev_dio_pool,4 , offsetof(struct blkdev_dio , bio),BIOSET_NEED_BVECS ); +return bioset_init (&blkdev_dio_pool,4 , + offsetof(struct blkdev_dio , bio), +BIOSET_NEED_BVECS |BIOSET_PERCPU_CACHE ); }module_init (blkdev_init);
Signature · 1 line
2.32.0
Re: [PATCH 6/6] block: enable use of bio allocation cache
Quoted 4 lines · Jens Axboe
Initialize the
bio_setused for IO with per-cpu bio caching enabled, and use the newbio_alloc_kiocb()helper to dip into that cache.Signed-off-by Jens Axboe <[email protected]>
This seems to be pretty much my patch as-is..
Jens Axboe
maintainerRe: [PATCH 6/6] block: enable use of bio allocation cache
Quoted 7 lines · Christoph Hellwig
Quoted 4 lines · Jens Axboe
Initialize the
bio_setused for IO with per-cpu bio caching enabled, and use the newbio_alloc_kiocb()helper to dip into that cache.Signed-off-by Jens Axboe <[email protected]>This seems to be pretty much my patch as-is..
It is, sorry I'll fix the attribution.
Signature · 1 line
Jens Axboe