btrfs: allocate eb-attached btree pages as movable
PATCH v2 1/1 btrfs: allocate eb-attached btree pages as movable
Extent buffer pages allocated by alloc_extent_buffer() are attached to btree_inode->i_mapping (the buffer_tree path), reach the LRU, and are served by the btree_migrate_folio aops in fs/btrfs/disk-io.c. They are migratable in practice once their owning extent buffer hits refs == 1, which happens naturally. The buddy allocator classifies them by GFP, however, and bare GFP_NOFS lands them in MIGRATE_UNMOVABLE pageblocks. The result: every btree_inode page we read in pins an unmovable pageblock from the page-superblock allocator's perspective, even though the page itself can be moved.
Have each caller of btrfs_alloc_page_array, btrfs_alloc_folio_array, and alloc_eb_folio_array pass in the full GFP mask directly, instead of having the functions calculate it from boolean flags.
The alloc_extent_buffer call site passes GFP_NOFS | __GFP_NOFAIL | __GFP_MOVABLE. All other call sites pass plain GFP_NOFS.
Three categories of caller stay on bare GFP_NOFS, deliberately:
- alloc_dummy_extent_buffer / btrfs_clone_extent_buffer: the resulting eb is EXTENT_BUFFER_UNMAPPED, folio->mapping stays NULL, the folios never enter LRU, never get migrate_folio aops. Tagging them __GFP_MOVABLE would violate the page allocator's migrability contract and they would defeat compaction in MOVABLE pageblocks where isolate_migratepages_block skips non-LRU non-movable_ops pages outright.
- btrfs_alloc_page_array callers in fs/btrfs/raid56.c (stripe pages), fs/btrfs/inode.c (encoded reads), fs/btrfs/ioctl.c (uring encoded reads), fs/btrfs/relocation.c (relocation buffers): same contract violation. raid56 stripe_pages additionally persist in the stripe cache (RBIO_CACHE_SIZE=1024) well beyond a single I/O, so they are not transient enough to hand-wave the contract.
- btrfs_alloc_folio_array caller in fs/btrfs/scrub.c (stripe folios): same -- stripe->folios[] are private buffers freed via folio_put in release_scrub_stripe.
This change targets the dominant fragmentation source observed on the page-superblock series: ~28 GB of btree_inode pages parked across many tainted superpageblocks on a 250 GB test system with btrfs root, preventing 1 GiB hugepage allocation from those regions. With the movable hint, those pages now land in MOVABLE pageblocks where the existing background defragger drains them through the standard PB_has_movable gate, no LRU-sample fallback needed.
---
v2: pass the gfp mask directly to each function from the callers (thanks Boris)
fs/btrfs/extent_io.c | 43 +++++++++++++++++++++++-------------------- fs/btrfs/extent_io.h | 4 ++-- fs/btrfs/inode.c | 2 +- fs/btrfs/ioctl.c | 2 +- fs/btrfs/raid56.c | 6 +++--- fs/btrfs/relocation.c | 2 +- fs/btrfs/scrub.c | 3 ++- 7 files changed, 33 insertions(+), 29 deletions(-)
fs/btrfs/extent_io.c +23 -20
@@ -620,24 +620,24 @@ static void end_bbio_data_read(struct btrfs_bio *bbio) }/* -* Populate every free slot in a provided array with folios using GFP_NOFS. +* Populate every free slot in a provided array with folios. * -* @nr_folios: number of folios to allocate -* @order: the order of the folios to be allocated -* @folio_array: the array to fill with folios; any existing non-NULL entries in -* the array will be skipped +* @nr_folios: number of folios to allocate +* @order: folio order +* @folio_array: array to fill with folios; non-NULL entries are skipped +* @gfp: GFP flags for the allocation * * Return: 0 if all folios were able to be allocated; * -ENOMEM otherwise, the partially allocated folios would be freed and * the array slots zeroed */ int btrfs_alloc_folio_array (unsigned int nr_folios,unsigned int order, -struct folio **folio_array) +struct folio **folio_array,gfp_t gfp) {for (int i =0 ; i < nr_folios; i++) {if (folio_array[i])continue ; - folio_array[i] =folio_alloc (GFP_NOFS , order); + folio_array[i] =folio_alloc (gfp, order);if (!folio_array[i]) goto error; }
@@ -652,21 +652,19 @@ int btrfs_alloc_folio_array(unsigned int nr_folios, unsigned int order, }/* -* Populate every free slot in a provided array with pages, using GFP_NOFS. +* Populate every free slot in a provided array with pages. * -* @nr_pages: number of pages to allocate -* @page_array: the array to fill with pages; any existing non-null entries in -* the array will be skipped -* @nofail: whether using __GFP_NOFAIL flag +* @nr_pages: number of pages to allocate +* @page_array: array to fill; non-NULL entries are skipped +* @gfp: GFP flags for the allocation * * Return: 0 if all pages were able to be allocated; * -ENOMEM otherwise, the partially allocated pages would be freed and * the array slots zeroed */ int btrfs_alloc_page_array (unsigned int nr_pages,struct page **page_array, -bool nofail) +gfp_t gfp) { -const gfp_t gfp = nofail ? (GFP_NOFS | __GFP_NOFAIL) :GFP_NOFS ;unsigned int allocated;for (allocated =0 ; allocated < nr_pages;) {
@@ -690,13 +688,13 @@ int btrfs_alloc_page_array(unsigned int nr_pages, struct page **page_array,* * For now, the folios populated are always in order 0 (aka, single page). */ -static int alloc_eb_folio_array (struct extent_buffer *eb,bool nofail) +static int alloc_eb_folio_array (struct extent_buffer *eb,gfp_t gfp) {struct page *page_array[INLINE_EXTENT_BUFFER_PAGES ] = {0 };int num_pages =num_extent_pages (eb);int ret; - ret =btrfs_alloc_page_array (num_pages, page_array, nofail); + ret =btrfs_alloc_page_array (num_pages, page_array, gfp);if (ret <0 )return ret;
@@ -3097,7 +3095,7 @@ struct extent_buffer *btrfs_clone_extent_buffer(const struct extent_buffer *src)*/ set_bit (EXTENT_BUFFER_UNMAPPED , &new->bflags); - ret =alloc_eb_folio_array (new, false); + ret =alloc_eb_folio_array (new,GFP_NOFS );if (ret) goto release_eb;
@@ -3138,7 +3136,7 @@ struct extent_buffer *alloc_dummy_extent_buffer(struct btrfs_fs_info *fs_info,if (!eb)return NULL ; - ret =alloc_eb_folio_array (eb, false); + ret =alloc_eb_folio_array (eb,GFP_NOFS );if (ret) goto release_eb;
@@ -3491,8 +3489,13 @@ struct extent_buffer *alloc_extent_buffer(struct btrfs_fs_info *fs_info, } reallocate: -/* Allocate all pages first. */ - ret =alloc_eb_folio_array (eb, true); +/* +* Allocate all pages first. These will be attached to +* btree_inode->i_mapping below (added to LRU, served by +* btree_migrate_folio), so request __GFP_MOVABLE so the +* page allocator places them in MOVABLE pageblocks. +*/ + ret =alloc_eb_folio_array (eb,GFP_NOFS | __GFP_NOFAIL | __GFP_MOVABLE);if (ret <0 ) {btrfs_free_folio_state (prealloc); goto out;
fs/btrfs/extent_io.h +2 -2
@@ -387,9 +387,9 @@ void btrfs_clear_buffer_dirty(struct btrfs_trans_handle *trans, struct extent_buffer *buf);int btrfs_alloc_page_array (unsigned int nr_pages,struct page **page_array, -bool nofail); +gfp_t gfp);int btrfs_alloc_folio_array (unsigned int nr_folios,unsigned int order, -struct folio **folio_array); +struct folio **folio_array,gfp_t gfp);#ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS bool find_lock_delalloc_range(struct inode *inode,
fs/btrfs/inode.c +1 -1
@@ -9659,7 +9659,7 @@ ssize_t btrfs_encoded_read_regular(struct kiocb *iocb, struct iov_iter *iter, pages = kzalloc_objs(struct page *, nr_pages, GFP_NOFS);if (!pages)return -ENOMEM ; - ret =btrfs_alloc_page_array (nr_pages, pages, false); + ret =btrfs_alloc_page_array (nr_pages, pages,GFP_NOFS );if (ret) { ret = -ENOMEM ; goto out;
fs/btrfs/ioctl.c +1 -1
@@ -4621,7 +4621,7 @@ static int btrfs_uring_read_extent(struct kiocb *iocb, struct iov_iter *iter, pages = kzalloc_objs(struct page *, nr_pages, GFP_NOFS);if (!pages)return -ENOMEM ; - ret =btrfs_alloc_page_array (nr_pages, pages,0 ); + ret =btrfs_alloc_page_array (nr_pages, pages,GFP_NOFS );if (ret) { ret = -ENOMEM ; goto out_fail;
fs/btrfs/raid56.c +3 -3
@@ -1123,7 +1123,7 @@ static int alloc_rbio_pages(struct btrfs_raid_bio *rbio) {int ret; - ret =btrfs_alloc_page_array (rbio->nr_pages, rbio->stripe_pages, false); + ret =btrfs_alloc_page_array (rbio->nr_pages, rbio->stripe_pages,GFP_NOFS );if (ret <0 )return ret;/* Mapping all sectors */
@@ -1138,7 +1138,7 @@ static int alloc_rbio_parity_pages(struct btrfs_raid_bio *rbio)int ret; ret =btrfs_alloc_page_array (rbio->nr_pages - data_pages, - rbio->stripe_pages + data_pages, false); + rbio->stripe_pages + data_pages,GFP_NOFS );if (ret <0 )return ret;
@@ -1732,7 +1732,7 @@ static int alloc_rbio_data_pages(struct btrfs_raid_bio *rbio)const int data_pages = rbio->nr_data * rbio->stripe_npages;int ret; - ret =btrfs_alloc_page_array (data_pages, rbio->stripe_pages, false); + ret =btrfs_alloc_page_array (data_pages, rbio->stripe_pages,GFP_NOFS );if (ret <0 )return ret;
fs/btrfs/relocation.c +1 -1
@@ -4038,7 +4038,7 @@ static int copy_remapped_data(struct btrfs_fs_info *fs_info, u64 old_addr,if (!pages)return -ENOMEM ; - ret =btrfs_alloc_page_array (nr_pages, pages,0 ); + ret =btrfs_alloc_page_array (nr_pages, pages,GFP_NOFS );if (ret) { ret = -ENOMEM ; goto end;
fs/btrfs/scrub.c +2 -1
@@ -369,7 +369,8 @@ static int init_scrub_stripe(struct btrfs_fs_info *fs_info,ASSERT (BTRFS_STRIPE_LEN >> min_folio_shift <=SCRUB_STRIPE_MAX_FOLIOS ); ret =btrfs_alloc_folio_array (BTRFS_STRIPE_LEN >> min_folio_shift, - fs_info->block_min_order, stripe->folios); + fs_info->block_min_order, stripe->folios, +GFP_NOFS );if (ret <0 ) goto error;
Signature · 1 line
2.53.0-Meta
Quoted 46 lines · Rik van Riel
Extent buffer pages allocated by
alloc_extent_buffer()are attached tobtree_inode->i_mapping(thebuffer_treepath), reach the LRU, and are served by thebtree_migrate_folioaops infs/btrfs/disk-io.c. They are migratable in practice once their owning extent buffer hits refs == 1, which happens naturally. The buddy allocator classifies them by GFP, however, and bareGFP_NOFSlands them inMIGRATE_UNMOVABLEpageblocks. The result: everybtree_inodepage we read in pins an unmovable pageblock from the page-superblock allocator's perspective, even though the page itself can be moved.Have each caller of
btrfs_alloc_page_array,btrfs_alloc_folio_array, andalloc_eb_folio_arraypass in the full GFP mask directly, instead of having the functions calculate it from boolean flags.The
alloc_extent_buffercall site passesGFP_NOFS|__GFP_NOFAIL|__GFP_MOVABLE. All other call sites pass plainGFP_NOFS.Three categories of caller stay on bare
GFP_NOFS, deliberately:-
alloc_dummy_extent_buffer/btrfs_clone_extent_buffer: the resulting eb isEXTENT_BUFFER_UNMAPPED,folio->mappingstays NULL, the folios never enter LRU, never getmigrate_folioaops. Tagging them__GFP_MOVABLEwould violate the page allocator's migrability contract and they would defeat compaction in MOVABLE pageblocks whereisolate_migratepages_blockskips non-LRU non-movable_ops pages outright.-
btrfs_alloc_page_arraycallers infs/btrfs/raid56.c(stripe pages),fs/btrfs/inode.c(encoded reads),fs/btrfs/ioctl.c(uring encoded reads),fs/btrfs/relocation.c(relocation buffers): same contract violation. raid56stripe_pagesadditionally persist in the stripe cache (RBIO_CACHE_SIZE=1024) well beyond a single I/O, so they are not transient enough to hand-wave the contract.-
btrfs_alloc_folio_arraycaller infs/btrfs/scrub.c(stripe folios): same -- stripe->folios[] are private buffers freed viafolio_putinrelease_scrub_stripe.This change targets the dominant fragmentation source observed on the page-superblock series: ~28 GB of
btree_inodepages parked across many tainted superpageblocks on a 250 GB test system with btrfs root, preventing 1 GiB hugepage allocation from those regions. With the movable hint, those pages now land in MOVABLE pageblocks where the existing background defragger drains them through the standardPB_has_movablegate, no LRU-sample fallback needed.
Quoted 8 lines
Cc Chris Mason <[email protected]>Cc David Sterba <[email protected]>Cc Boris Burkov <[email protected]>Signed-off-by Rik van Riel <[email protected]>Assisted-by Claude:claude-opus-4-6---
v2: pass the gfp mask directly to each function from the callers (thanks Boris)
Added to for-next, thanks.