block: try slab allocation in bio_alloc_bioset() before mempool
PATCH v2 1/1 block: try slab allocation in bio_alloc_bioset() before mempool
When the per-CPU bio cache is enabled but empty, bio_alloc_percpu_cache() returns NULL and bio_alloc_bioset() falls straight through to the mempool fallback:
if (unlikely(!bio)) {
if (!(saved_gfp & __GFP_DIRECT_RECLAIM))
return NULL;
...
}For non-sleeping allocations (no __GFP_DIRECT_RECLAIM) this returns NULL without ever attempting a slab allocation, even when there is plenty of free memory.
Commit b520c4eef83d ("block: split bio_alloc_bioset more clearly into a fast and slowpath") introduced this. Before it, a percpu cache miss fell through to mempool_alloc(), which attempted the underlying slab allocation first and only failed when that slab allocation failed. The restructuring dropped the slab attempt that non-sleeping callers of a cache-enabled bioset (such as the default fs_bio_set used by bio_alloc()) relied on.
Try a slab allocation with optimistic GFP_ flags before falling back to the mempool whenever the bio is still NULL, so both the cache-empty and non-cache paths share the same slab attempt. This restores the previous behavior for non-sleeping allocations.
--- block/bio.c | 8 ++++++++ 1 file changed, 8 insertions(+)
block/bio.c +8 -0
@@ -555,6 +555,14 @@ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs, bio =bio_alloc_percpu_cache (bs); }else { opf &= ~REQ_ALLOC_CACHE ; + } + +/* +* For a bioset without a percpu cache, or when the percpu cache was +* empty, try a slab allocation with optimistic GFP_ flags before +* falling back to the mempool. +*/ +if (!bio) { p =kmem_cache_alloc (bs->bio_slab, gfp);if (p) bio = p + bs->front_pad;
Signature · 1 line
2.39.3
Looks good:
Gentle ping...
Quoted 52 lines · Joseph Qi
When the per-CPU bio cache is enabled but empty,
bio_alloc_percpu_cache()returns NULL andbio_alloc_bioset()falls straight through to the mempool fallback:if (unlikely(!bio)) { if (!(saved_gfp & __GFP_DIRECT_RECLAIM)) return NULL; ... }For non-sleeping allocations (no
__GFP_DIRECT_RECLAIM) this returns NULL without ever attempting a slab allocation, even when there is plenty of free memory.Commit
b520c4eef83d("block: splitbio_alloc_biosetmore clearly into a fast and slowpath") introduced this. Before it, a percpu cache miss fell through tomempool_alloc(), which attempted the underlying slab allocation first and only failed when that slab allocation failed. The restructuring dropped the slab attempt that non-sleeping callers of a cache-enabled bioset (such as the defaultfs_bio_setused bybio_alloc()) relied on.Try a slab allocation with optimistic
GFP_flags before falling back to the mempool whenever the bio is still NULL, so both the cache-empty and non-cache paths share the same slab attempt. This restores the previous behavior for non-sleeping allocations.Fixes b520c4eef83d ("block: split bio_alloc_bioset more clearly into a fast and slowpath")Suggested-by Christoph Hellwig <[email protected]>Signed-off-by Joseph Qi <[email protected]>--- block/bio.c | 8 ++++++++ 1 file changed, 8 insertions(+)block/bio.c +8
-0@@ -555,6 +555,14 @@ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs, bio =bio_alloc_percpu_cache (bs); }else { opf &= ~REQ_ALLOC_CACHE ; + } + +/* +* For a bioset without a percpu cache, or when the percpu cache was +* empty, try a slab allocation with optimistic GFP_ flags before +* falling back to the mempool. +*/ +if (!bio) { p =kmem_cache_alloc (bs->bio_slab, gfp);if (p) bio = p + bs->front_pad;
Quoted 11 lines · Joseph Qi
When the per-CPU bio cache is enabled but empty,
bio_alloc_percpu_cache()returns NULL andbio_alloc_bioset()falls straight through to the mempool fallback:if (unlikely(!bio)) { if (!(saved_gfp & __GFP_DIRECT_RECLAIM)) return NULL; ... }[...]
Applied, thanks!
[1/1] block: try slab allocation in bio_alloc_bioset() before mempool
commit: 447cfed6d700bfbd5a7120f8ea5821a0e1191667
Best regards,
Signature · 1 line
Jens Axboe