Skip to contentSkip to conversation

block: try slab allocation in bio_alloc_bioset() before mempool

PATCH Applied Joseph Qi 3 participants 4 messages Jul 9 – Jul 16 Block layer

lore
Share & tools
lore

Joseph Qi

author
#0 lore raw

PATCH v2 1/1 block: try slab allocation in bio_alloc_bioset() before mempool

When the per-CPU bio cache is enabled but empty, bio_alloc_percpu_cache() returns NULL and bio_alloc_bioset() falls straight through to the mempool fallback:

    if (unlikely(!bio)) {
        if (!(saved_gfp & __GFP_DIRECT_RECLAIM))
            return NULL;
        ...
    }

For non-sleeping allocations (no __GFP_DIRECT_RECLAIM) this returns NULL without ever attempting a slab allocation, even when there is plenty of free memory.

Commit b520c4eef83d ("block: split bio_alloc_bioset more clearly into a fast and slowpath") introduced this. Before it, a percpu cache miss fell through to mempool_alloc(), which attempted the underlying slab allocation first and only failed when that slab allocation failed. The restructuring dropped the slab attempt that non-sleeping callers of a cache-enabled bioset (such as the default fs_bio_set used by bio_alloc()) relied on.

Try a slab allocation with optimistic GFP_ flags before falling back to the mempool whenever the bio is still NULL, so both the cache-empty and non-cache paths share the same slab attempt. This restores the previous behavior for non-sleeping allocations.

Fixes b520c4eef83d ("block: split bio_alloc_bioset more clearly into a fast and slowpath")
---
 block/bio.c | 8 ++++++++
 1 file changed, 8 insertions(+)
block/bio.c +8 -0
index f2a5f4d0a9672..982f2e47a4218 100644
@@ -555,6 +555,14 @@ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs,
 		bio = bio_alloc_percpu_cache(bs);
 	} else {
 		opf &= ~REQ_ALLOC_CACHE;
+	}
+
+	/*
+	 * For a bioset without a percpu cache, or when the percpu cache was
+	 * empty, try a slab allocation with optimistic GFP_ flags before
+	 * falling back to the mempool.
+	 */
+	if (!bio) {
 		p = kmem_cache_alloc(bs->bio_slab, gfp);
 		if (p)
 			bio = p + bs->front_pad;
Signature · 1 line
2.39.3
author
#2 lore raw
In reply to Joseph Qi

Gentle ping...

Quoted 52 lines · Joseph Qi

When the per-CPU bio cache is enabled but empty, bio_alloc_percpu_cache() returns NULL and bio_alloc_bioset() falls straight through to the mempool fallback:

    if (unlikely(!bio)) {
        if (!(saved_gfp & __GFP_DIRECT_RECLAIM))
            return NULL;
        ...
    }

For non-sleeping allocations (no __GFP_DIRECT_RECLAIM) this returns NULL without ever attempting a slab allocation, even when there is plenty of free memory.

Commit b520c4eef83d ("block: split bio_alloc_bioset more clearly into a fast and slowpath") introduced this. Before it, a percpu cache miss fell through to mempool_alloc(), which attempted the underlying slab allocation first and only failed when that slab allocation failed. The restructuring dropped the slab attempt that non-sleeping callers of a cache-enabled bioset (such as the default fs_bio_set used by bio_alloc()) relied on.

Try a slab allocation with optimistic GFP_ flags before falling back to the mempool whenever the bio is still NULL, so both the cache-empty and non-cache paths share the same slab attempt. This restores the previous behavior for non-sleeping allocations.

Fixes b520c4eef83d ("block: split bio_alloc_bioset more clearly into a fast and slowpath")
---
 block/bio.c | 8 ++++++++
 1 file changed, 8 insertions(+)
block/bio.c +8 -0
index f2a5f4d0a9672..982f2e47a4218 100644
@@ -555,6 +555,14 @@ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs,
 		bio = bio_alloc_percpu_cache(bs);
 	} else {
 		opf &= ~REQ_ALLOC_CACHE;
+	}
+
+	/*
+	 * For a bioset without a percpu cache, or when the percpu cache was
+	 * empty, try a slab allocation with optimistic GFP_ flags before
+	 * falling back to the mempool.
+	 */
+	if (!bio) {
 		p = kmem_cache_alloc(bs->bio_slab, gfp);
 		if (p)
 			bio = p + bs->front_pad;
maintainer Applied
#3 lore raw
In reply to Joseph Qi
Quoted 11 lines · Joseph Qi

When the per-CPU bio cache is enabled but empty, bio_alloc_percpu_cache() returns NULL and bio_alloc_bioset() falls straight through to the mempool fallback:

    if (unlikely(!bio)) {
        if (!(saved_gfp & __GFP_DIRECT_RECLAIM))
            return NULL;
        ...
    }

[...]

Applied, thanks!

[1/1] block: try slab allocation in bio_alloc_bioset() before mempool

      commit: 447cfed6d700bfbd5a7120f8ea5821a0e1191667

Best regards,

Signature · 1 line
Jens Axboe