btrfs: prevent direct reclaim during compressed readahead
cover btrfs: prevent direct reclaim during compressed readahead
We're finding that while under memory pressure, direct reclaim is kicking in during compressed readahead. This puts the associated task into D-state. Then shrink_lruvec() disables interrupts when acquiring the LRU lock. Under heavy pressure, reclaim can run long enough that the CPU becomes prone to CSD lock stalls since it cannot service incoming IPIs. Although the CSD lock stalls are the worst case scenario, we have found many more subtle occurrences of this latency on the order of seconds, over a minute in some cases.
Prevent direct reclaim during compressed readahead. This is achieved by using different GFP flags whenever the bio is marked for readahead. The flags are similar to GFP_NOFS but stripped of __GFP_DIRECT_RECLAIM. Also, __GFP_NOWARN is added since these allocations are allowed to fail. Demand reads still use full GFP_NOFS and will enter reclaim if needed.
There has been some previous work done to reduce the frequency of calling add_ra_bio_pages() [0]. This patch is complementary in that it reduces the latency associated with those calls.
JP Kobryn (Meta) (2):
btrfs: additional gfp api for allocating compressed folios
btrfs: prevent direct reclaim during compressed readahead
fs/btrfs/compression.c | 44 ++++++++++++++++++++++++++++++++++-------- fs/btrfs/compression.h | 1 + 2 files changed, 37 insertions(+), 8 deletions(-)
Signature · 1 line
2.52.0
PATCH 1/2 btrfs: additional gfp api for allocating compressed folios
btrfs_alloc_compr_folio() assumes all callers want GFP_NOFS. This is fine for most cases, however there are some call sites that would benefit from different flags. One such case is preventing direct reclaim from occurring during readahead allocations. With unbounded reclaim during this time, noticeable latency will occur under high memory pressure.
Provide an additional API that accepts one additional gfp_t parameter, giving callers flexibility over the characteristics of their allocation.
--- fs/btrfs/compression.c | 9 +++++++-- fs/btrfs/compression.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-)
fs/btrfs/compression.c +7 -2
@@ -180,7 +180,7 @@ static unsigned long btrfs_compr_pool_scan(struct shrinker *sh, struct shrink_co/* * Common wrappers for page allocation from compression wrappers */ -struct folio *btrfs_alloc_compr_folio (struct btrfs_fs_info *fs_info) +struct folio *btrfs_alloc_compr_folio_gfp (struct btrfs_fs_info *fs_info,gfp_t gfp) {struct folio *folio =NULL ;
@@ -200,7 +200,12 @@ struct folio *btrfs_alloc_compr_folio(struct btrfs_fs_info *fs_info)return folio; alloc: -return folio_alloc (GFP_NOFS , fs_info->block_min_order); +return folio_alloc (gfp, fs_info->block_min_order); +} + +struct folio *btrfs_alloc_compr_folio (struct btrfs_fs_info *fs_info) +{ +return btrfs_alloc_compr_folio_gfp (fs_info,GFP_NOFS ); }void btrfs_free_compr_folio (struct folio *folio)
fs/btrfs/compression.h +1 -0
@@ -99,6 +99,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio);int btrfs_compress_str2level (unsigned int type,const char *str,int *level_ret);struct folio *btrfs_alloc_compr_folio (struct btrfs_fs_info *fs_info); +struct folio *btrfs_alloc_compr_folio_gfp (struct btrfs_fs_info *fs_info,gfp_t gfp);void btrfs_free_compr_folio (struct folio *folio);struct workspace_manager {
Signature · 1 line
2.52.0
PATCH 2/2 btrfs: prevent direct reclaim during compressed readahead
In reply to JP Kobryn (Meta)Prevent direct reclaim during compressed readahead. This is achieved by passing specific GFP flags whenever the bio is marked for readahead. The flags are similar to GFP_NOFS but stripped of __GFP_DIRECT_RECLAIM. Also, __GFP_NOWARN is added since these allocations are allowed to fail. Demand reads still use full GFP_NOFS and will enter reclaim if needed.
btrfs_submit_compressed_read() now makes use of the new gfp_t API for allocations within. Since non-readahead code may call this function, the bio flags are inspected to determine whether direct reclaim should be restricted or not.
add_ra_bio_pages() gains a bool parameter which allows callers to specify if they want to allow direct reclaim or not. In either case, the NOWARN flag was added unconditionally since the allocations are speculative.
--- fs/btrfs/compression.c | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-)
fs/btrfs/compression.c +28 -5
@@ -372,7 +372,8 @@ struct compressed_bio *btrfs_alloc_compressed_write(struct btrfs_inode *inode, static noinline int add_ra_bio_pages(struct inode *inode,u64 compressed_end,struct compressed_bio *cb, -int *memstall,unsigned long *pflags) +int *memstall,unsigned long *pflags, +bool direct_reclaim) {struct btrfs_fs_info *fs_info =inode_to_fs_info (inode);pgoff_t end_index;
@@ -380,6 +381,7 @@ static noinline int add_ra_bio_pages(struct inode *inode,u64 cur = cb->orig_bbio->file_offset + orig_bio->bi_iter.bi_size;u64 isize =i_size_read (inode);int ret; +gfp_t constraint_gfp, cache_gfp;struct folio *folio;struct extent_map *em;struct address_space *mapping = inode->i_mapping;
@@ -409,6 +411,14 @@ static noinline int add_ra_bio_pages(struct inode *inode, end_index = (i_size_read (inode) -1 ) >>PAGE_SHIFT ; +if (!direct_reclaim) { + constraint_gfp = ~(__GFP_FS | __GFP_DIRECT_RECLAIM); + cache_gfp = (GFP_NOFS & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; + }else { + constraint_gfp = ~__GFP_FS; + cache_gfp =GFP_NOFS | __GFP_NOWARN; + } +while (cur < compressed_end) {pgoff_t page_end;pgoff_t pg_index = cur >>PAGE_SHIFT ;
@@ -438,12 +448,13 @@ static noinline int add_ra_bio_pages(struct inode *inode,continue ; } - folio =filemap_alloc_folio (mapping_gfp_constraint (mapping, ~__GFP_FS), + folio =filemap_alloc_folio (mapping_gfp_constraint (mapping, + constraint_gfp) | __GFP_NOWARN,0 ,NULL );if (!folio)break ; -if (filemap_add_folio (mapping, folio, pg_index,GFP_NOFS )) { +if (filemap_add_folio (mapping, folio, pg_index, cache_gfp)) {/* There is already a page, skip to page end */ cur +=folio_size (folio);folio_put (folio);
@@ -536,6 +547,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)unsigned int compressed_len;const u32 min_folio_size =btrfs_min_folio_size (fs_info);u64 file_offset = bbio->file_offset; +gfp_t gfp;u64 em_len;u64 em_start;struct extent_map *em;
@@ -543,6 +555,17 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)int memstall =0 ;int ret; +/* +* If this is a readahead bio, prevent direct reclaim. This is done to +* avoid stalling on speculative allocations when memory pressure is +* high. The demand fault will retry with GFP_NOFS and enter direct +* reclaim if needed. +*/ +if (bbio->bio.bi_opf &REQ_RAHEAD ) + gfp = (GFP_NOFS & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +else + gfp =GFP_NOFS ; +/* we need the actual starting offset of this extent in the file */ read_lock (&em_tree->lock); em =btrfs_lookup_extent_mapping (em_tree, file_offset, fs_info->sectorsize);
@@ -573,7 +596,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)struct folio *folio;u32 cur_len =min (compressed_len - i * min_folio_size, min_folio_size); - folio =btrfs_alloc_compr_folio (fs_info); + folio =btrfs_alloc_compr_folio_gfp (fs_info, gfp);if (!folio) { ret = -ENOMEM ; goto out_free_bio;
@@ -589,7 +612,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)ASSERT (cb->bbio.bio.bi_iter.bi_size == compressed_len);add_ra_bio_pages (&inode->vfs_inode, em_start + em_len, cb, &memstall, - &pflags); + &pflags, !(bbio->bio.bi_opf &REQ_RAHEAD )); cb->len = bbio->bio.bi_iter.bi_size; cb->bbio.bio.bi_iter.bi_sector = bbio->bio.bi_iter.bi_sector;
Signature · 1 line
2.52.0
Quoted 1 line · JP Kobryn (Meta)
Prevent direct reclaim during compressed readahead.
This completely fails to explain why you want that.
Quoted 5 lines · JP Kobryn (Meta)
Prevent direct reclaim during compressed readahead. This is achieved by passing specific GFP flags whenever the bio is marked for readahead. The flags are similar to
GFP_NOFSbut stripped of__GFP_DIRECT_RECLAIM. Also,__GFP_NOWARNis added since these allocations are allowed to fail. Demand reads still use fullGFP_NOFSand will enter reclaim if needed.
I believe it will be more convincing to explain why the current gfp flags is going to cause the problem you mentioned in the cover letter.
Quoted 9 lines
btrfs_submit_compressed_read()now makes use of the newgfp_tAPI for allocations within. Since non-readahead code may call this function, the bio flags are inspected to determine whether direct reclaim should be restricted or not.
add_ra_bio_pages()gains a bool parameter which allows callers to specify if they want to allow direct reclaim or not. In either case, the NOWARN flag was added unconditionally since the allocations are speculative.
After reading the code, I have a feeling that, we shouldn't act on the behalf of MM layer to add the next few folios into the page cache.
On the other hand, with the incoming large folios, we will completely skip the readahead for large folios.
I know this is not optimal as the next few folios may still belong to the same compressed extent and will cause re-read and re-decompression.
Thus I'm wondering, for your specific workload, will disabling compressed ra completely and fully rely on large folios help?
If the performance is acceptable, I'd prefer to disable compressed readahead completely and rely on large folios instead.
(Now I understand why other fses with compression support is completely relying on fixed IO size)
Thanks,
Qu
Quoted 103 lines
Signed-off-by JP Kobryn (Meta) <[email protected]>--- fs/btrfs/compression.c | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-)fs/btrfs/compression.c +28
-5@@ -372,7 +372,8 @@ struct compressed_bio *btrfs_alloc_compressed_write(struct btrfs_inode *inode, static noinline int add_ra_bio_pages(struct inode *inode,u64 compressed_end,struct compressed_bio *cb, -int *memstall,unsigned long *pflags) +int *memstall,unsigned long *pflags, +bool direct_reclaim) {struct btrfs_fs_info *fs_info =inode_to_fs_info (inode);pgoff_t end_index;@@ -380,6 +381,7 @@ static noinline int add_ra_bio_pages(struct inode *inode,u64 cur = cb->orig_bbio->file_offset + orig_bio->bi_iter.bi_size;u64 isize =i_size_read (inode);int ret; +gfp_t constraint_gfp, cache_gfp;struct folio *folio;struct extent_map *em;struct address_space *mapping = inode->i_mapping;@@ -409,6 +411,14 @@ static noinline int add_ra_bio_pages(struct inode *inode, end_index = (i_size_read (inode) -1 ) >>PAGE_SHIFT ; +if (!direct_reclaim) { + constraint_gfp = ~(__GFP_FS | __GFP_DIRECT_RECLAIM); + cache_gfp = (GFP_NOFS & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; + }else { + constraint_gfp = ~__GFP_FS; + cache_gfp =GFP_NOFS | __GFP_NOWARN; + } +while (cur < compressed_end) {pgoff_t page_end;pgoff_t pg_index = cur >>PAGE_SHIFT ;@@ -438,12 +448,13 @@ static noinline int add_ra_bio_pages(struct inode *inode,continue ; } - folio =filemap_alloc_folio (mapping_gfp_constraint (mapping, ~__GFP_FS), + folio =filemap_alloc_folio (mapping_gfp_constraint (mapping, + constraint_gfp) | __GFP_NOWARN,0 ,NULL );if (!folio)break ; -if (filemap_add_folio (mapping, folio, pg_index,GFP_NOFS )) { +if (filemap_add_folio (mapping, folio, pg_index, cache_gfp)) {/* There is already a page, skip to page end */ cur +=folio_size (folio);folio_put (folio);@@ -536,6 +547,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)unsigned int compressed_len;const u32 min_folio_size =btrfs_min_folio_size (fs_info);u64 file_offset = bbio->file_offset; +gfp_t gfp;u64 em_len;u64 em_start;struct extent_map *em;@@ -543,6 +555,17 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)int memstall =0 ;int ret; +/* +* If this is a readahead bio, prevent direct reclaim. This is done to +* avoid stalling on speculative allocations when memory pressure is +* high. The demand fault will retry with GFP_NOFS and enter direct +* reclaim if needed. +*/ +if (bbio->bio.bi_opf &REQ_RAHEAD ) + gfp = (GFP_NOFS & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +else + gfp =GFP_NOFS ; +/* we need the actual starting offset of this extent in the file */ read_lock (&em_tree->lock); em =btrfs_lookup_extent_mapping (em_tree, file_offset, fs_info->sectorsize);@@ -573,7 +596,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)struct folio *folio;u32 cur_len =min (compressed_len - i * min_folio_size, min_folio_size); - folio =btrfs_alloc_compr_folio (fs_info); + folio =btrfs_alloc_compr_folio_gfp (fs_info, gfp);if (!folio) { ret = -ENOMEM ; goto out_free_bio;@@ -589,7 +612,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)ASSERT (cb->bbio.bio.bi_iter.bi_size == compressed_len);add_ra_bio_pages (&inode->vfs_inode, em_start + em_len, cb, &memstall, - &pflags); + &pflags, !(bbio->bio.bi_opf &REQ_RAHEAD )); cb->len = bbio->bio.bi_iter.bi_size; cb->bbio.bio.bi_iter.bi_sector = bbio->bio.bi_iter.bi_sector;
Re: [PATCH 1/2] btrfs: additional gfp api for allocating compressed folios
In reply to JP Kobryn (Meta)Quoted 54 lines · JP Kobryn (Meta)
btrfs_alloc_compr_folio()assumes all callers wantGFP_NOFS. This is fine for most cases, however there are some call sites that would benefit from different flags. One such case is preventing direct reclaim from occurring during readahead allocations. With unbounded reclaim during this time, noticeable latency will occur under high memory pressure.Provide an additional API that accepts one additional
gfp_tparameter, giving callers flexibility over the characteristics of their allocation.Signed-off-by JP Kobryn (Meta) <[email protected]>--- fs/btrfs/compression.c | 9 +++++++-- fs/btrfs/compression.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-)fs/btrfs/compression.c +7
-2@@ -180,7 +180,7 @@ static unsigned long btrfs_compr_pool_scan(struct shrinker *sh, struct shrink_co/* * Common wrappers for page allocation from compression wrappers */ -struct folio *btrfs_alloc_compr_folio (struct btrfs_fs_info *fs_info) +struct folio *btrfs_alloc_compr_folio_gfp (struct btrfs_fs_info *fs_info,gfp_t gfp) {struct folio *folio =NULL ;@@ -200,7 +200,12 @@ struct folio *btrfs_alloc_compr_folio(struct btrfs_fs_info *fs_info)return folio; alloc: -return folio_alloc (GFP_NOFS , fs_info->block_min_order); +return folio_alloc (gfp, fs_info->block_min_order); +} + +struct folio *btrfs_alloc_compr_folio (struct btrfs_fs_info *fs_info) +{ +return btrfs_alloc_compr_folio_gfp (fs_info,GFP_NOFS ); }void btrfs_free_compr_folio (struct folio *folio)fs/btrfs/compression.h +1
-0@@ -99,6 +99,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio);int btrfs_compress_str2level (unsigned int type,const char *str,int *level_ret);struct folio *btrfs_alloc_compr_folio (struct btrfs_fs_info *fs_info); +struct folio *btrfs_alloc_compr_folio_gfp (struct btrfs_fs_info *fs_info,gfp_t gfp);void btrfs_free_compr_folio (struct folio *folio);struct workspace_manager {
Quoted 117 lines · JP Kobryn (Meta)
Prevent direct reclaim during compressed readahead. This is achieved by passing specific GFP flags whenever the bio is marked for readahead. The flags are similar to
GFP_NOFSbut stripped of__GFP_DIRECT_RECLAIM. Also,__GFP_NOWARNis added since these allocations are allowed to fail. Demand reads still use fullGFP_NOFSand will enter reclaim if needed.
btrfs_submit_compressed_read()now makes use of the newgfp_tAPI for allocations within. Since non-readahead code may call this function, the bio flags are inspected to determine whether direct reclaim should be restricted or not.
add_ra_bio_pages()gains a bool parameter which allows callers to specify if they want to allow direct reclaim or not. In either case, the NOWARN flag was added unconditionally since the allocations are speculative.Signed-off-by JP Kobryn (Meta) <[email protected]>--- fs/btrfs/compression.c | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-)fs/btrfs/compression.c +28
-5@@ -372,7 +372,8 @@ struct compressed_bio *btrfs_alloc_compressed_write(struct btrfs_inode *inode, static noinline int add_ra_bio_pages(struct inode *inode,u64 compressed_end,struct compressed_bio *cb, -int *memstall,unsigned long *pflags) +int *memstall,unsigned long *pflags, +bool direct_reclaim) {struct btrfs_fs_info *fs_info =inode_to_fs_info (inode);pgoff_t end_index;@@ -380,6 +381,7 @@ static noinline int add_ra_bio_pages(struct inode *inode,u64 cur = cb->orig_bbio->file_offset + orig_bio->bi_iter.bi_size;u64 isize =i_size_read (inode);int ret; +gfp_t constraint_gfp, cache_gfp;struct folio *folio;struct extent_map *em;struct address_space *mapping = inode->i_mapping;@@ -409,6 +411,14 @@ static noinline int add_ra_bio_pages(struct inode *inode, end_index = (i_size_read (inode) -1 ) >>PAGE_SHIFT ; +if (!direct_reclaim) { + constraint_gfp = ~(__GFP_FS | __GFP_DIRECT_RECLAIM); + cache_gfp = (GFP_NOFS & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; + }else { + constraint_gfp = ~__GFP_FS; + cache_gfp =GFP_NOFS | __GFP_NOWARN; + } +while (cur < compressed_end) {pgoff_t page_end;pgoff_t pg_index = cur >>PAGE_SHIFT ;@@ -438,12 +448,13 @@ static noinline int add_ra_bio_pages(struct inode *inode,continue ; } - folio =filemap_alloc_folio (mapping_gfp_constraint (mapping, ~__GFP_FS), + folio =filemap_alloc_folio (mapping_gfp_constraint (mapping, + constraint_gfp) | __GFP_NOWARN,0 ,NULL );if (!folio)break ; -if (filemap_add_folio (mapping, folio, pg_index,GFP_NOFS )) { +if (filemap_add_folio (mapping, folio, pg_index, cache_gfp)) {/* There is already a page, skip to page end */ cur +=folio_size (folio);folio_put (folio);@@ -536,6 +547,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)unsigned int compressed_len;const u32 min_folio_size =btrfs_min_folio_size (fs_info);u64 file_offset = bbio->file_offset; +gfp_t gfp;u64 em_len;u64 em_start;struct extent_map *em;@@ -543,6 +555,17 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)int memstall =0 ;int ret; +/* +* If this is a readahead bio, prevent direct reclaim. This is done to +* avoid stalling on speculative allocations when memory pressure is +* high. The demand fault will retry with GFP_NOFS and enter direct +* reclaim if needed. +*/ +if (bbio->bio.bi_opf &REQ_RAHEAD ) + gfp = (GFP_NOFS & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; +else + gfp =GFP_NOFS ; +/* we need the actual starting offset of this extent in the file */ read_lock (&em_tree->lock); em =btrfs_lookup_extent_mapping (em_tree, file_offset, fs_info->sectorsize);@@ -573,7 +596,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)struct folio *folio;u32 cur_len =min (compressed_len - i * min_folio_size, min_folio_size); - folio =btrfs_alloc_compr_folio (fs_info); + folio =btrfs_alloc_compr_folio_gfp (fs_info, gfp);if (!folio) { ret = -ENOMEM ; goto out_free_bio;@@ -589,7 +612,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio)ASSERT (cb->bbio.bio.bi_iter.bi_size == compressed_len);add_ra_bio_pages (&inode->vfs_inode, em_start + em_len, cb, &memstall, - &pflags); + &pflags, !(bbio->bio.bi_opf &REQ_RAHEAD )); cb->len = bbio->bio.bi_iter.bi_size; cb->bbio.bio.bi_iter.bi_sector = bbio->bio.bi_iter.bi_sector;
Quoted 17 lines · Qu Wenruo
Quoted 5 lines · JP Kobryn (Meta)
Prevent direct reclaim during compressed readahead. This is achieved by passing specific GFP flags whenever the bio is marked for readahead. The flags are similar to
GFP_NOFSbut stripped of__GFP_DIRECT_RECLAIM. Also,__GFP_NOWARNis added since these allocations are allowed to fail. Demand reads still use fullGFP_NOFSand will enter reclaim if needed.I believe it will be more convincing to explain why the current gfp flags is going to cause the problem you mentioned in the cover letter.
Quoted 6 lines
btrfs_submit_compressed_read()now makes use of the newgfp_tAPI for allocations within. Since non-readahead code may call this function, the bio flags are inspected to determine whether direct reclaim should be restricted or not.
Quoted 5 lines
Quoted 3 lines
add_ra_bio_pages()gains a bool parameter which allows callers to specify if they want to allow direct reclaim or not. In either case, the NOWARN flag was added unconditionally since the allocations are speculative.After reading the code, I have a feeling that, we shouldn't act on the behalf of MM layer to add the next few folios into the page cache.
Your idea might have merit, but this is a quick fix for a problem that JP has seen in production. Reworking the whole thing might be the ultimate solution, but that's much riskier, and will take more testing, than the proposed change.
Quoted 121 lines
On the other hand, with the incoming large folios, we will completely skip the readahead for large folios.
I know this is not optimal as the next few folios may still belong to the same compressed extent and will cause re-read and re-decompression.
Thus I'm wondering, for your specific workload, will disabling compressed ra completely and fully rely on large folios help?
If the performance is acceptable, I'd prefer to disable compressed readahead completely and rely on large folios instead.
(Now I understand why other fses with compression support is completely relying on fixed IO size)
Thanks,
QuQuoted 106 lines
Signed-off-by JP Kobryn (Meta) <[email protected]>---
fs/btrfs/compression.c| 33 ++++++++++++++++++++++++++++-----1 file changed, 28 insertions(+), 5 deletions(-)fs/btrfs/compression.c +2
-1@@ -372,7 +372,8 @@ struct compressed_bio *btrfs_alloc_compressed_write(struct btrfs_inode *inode, static noinline int add_ra_bio_pages(struct inode *inode, u64 compressed_end, struct compressed_bio *cb, - int *memstall, unsigned long *pflags) + int *memstall, unsigned long *pflags, + bool direct_reclaim) { struct btrfs_fs_info *fs_info = inode_to_fs_info(inode);
pgoff_tend_index;fs/btrfs/compression.c +1
-0@@ -380,6 +381,7 @@ static noinline int add_ra_bio_pages(struct inode *inode, u64 cur = cb->orig_bbio->file_offset + orig_bio->bi_iter.bi_size; u64 isize = i_size_read(inode); int ret; + gfp_t constraint_gfp, cache_gfp; struct folio *folio; struct extent_map *em;struct
address_space*mapping =inode->i_mapping;fs/btrfs/compression.c +12
-2@@ -409,6 +411,14 @@ static noinline int add_ra_bio_pages(struct inode *inode, end_index = (i_size_read(inode) - 1) >> PAGE_SHIFT; + if (!direct_reclaim) { + constraint_gfp = ~(__GFP_FS | __GFP_DIRECT_RECLAIM); + cache_gfp = (GFP_NOFS & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; + } else { + constraint_gfp = ~__GFP_FS; + cache_gfp = GFP_NOFS | __GFP_NOWARN; + } + while (cur < compressed_end) { pgoff_t page_end; pgoff_t pg_index = cur >> PAGE_SHIFT;@@ -438,12 +448,13 @@ static noinline int add_ra_bio_pages(structinode *inode, continue; } - folio = filemap_alloc_folio(mapping_gfp_constraint(mapping, ~__GFP_FS), + folio = filemap_alloc_folio(mapping_gfp_constraint(mapping, + constraint_gfp) | __GFP_NOWARN, 0, NULL); if (!folio) break; - if (filemap_add_folio(mapping, folio, pg_index, GFP_NOFS)) { + if (filemap_add_folio(mapping, folio, pg_index, cache_gfp)) { /* There is already a page, skip to page end */ cur += folio_size(folio); folio_put(folio);@@ -536,6 +547,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio) unsigned int compressed_len; const u32 min_folio_size = btrfs_min_folio_size(fs_info); u64 file_offset = bbio->file_offset; + gfp_t gfp; u64 em_len; u64 em_start;struct
extent_map*em;fs/btrfs/compression.c +2
-0@@ -543,6 +555,17 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio) int memstall = 0; int ret; + /* +* If this is a readahead bio, prevent direct reclaim. This is done to
+ * avoid stalling on speculative allocations when memory pressure is + * high. The demand fault will retry withGFP_NOFSand enter direct + * reclaim if needed.
+ */
+ if (bbio->bio.bi_opf&REQ_RAHEAD)
+ gfp = (GFP_NOFS& ~__GFP_DIRECT_RECLAIM) |__GFP_NOWARN; + else
+ gfp =GFP_NOFS;
+
/* we need the actual starting offset of this extent in the file */read_lock(&em_tree->lock);
em = btrfs_lookup_extent_mapping(em_tree,file_offset, fs_info-Quoted 1 line
sectorsize);
fs/btrfs/compression.c +2
-2@@ -573,7 +596,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio) struct folio *folio; u32 cur_len = min(compressed_len - i * min_folio_size, min_folio_size); - folio = btrfs_alloc_compr_folio(fs_info); + folio = btrfs_alloc_compr_folio_gfp(fs_info, gfp); if (!folio) { ret = -ENOMEM; goto out_free_bio;@@ -589,7 +612,7 @@ void btrfs_submit_compressed_read(struct btrfs_bio *bbio) ASSERT(cb->bbio.bio.bi_iter.bi_size == compressed_len); add_ra_bio_pages(&inode->vfs_inode, em_start + em_len, cb, &memstall, - &pflags); + &pflags, !(bbio->bio.bi_opf & REQ_RAHEAD)); cb->len = bbio->bio.bi_iter.bi_size; cb->bbio.bio.bi_iter.bi_sector = bbio->bio.bi_iter.bi_sector;
Quoted 9 lines · JP Kobryn (Meta)
We're finding that while under memory pressure, direct reclaim is kicking in during compressed readahead. This puts the associated task into D-state. Then
shrink_lruvec()disables interrupts when acquiring the LRU lock. Under heavy pressure, reclaim can run long enough that the CPU becomes prone to CSD lock stalls since it cannot service incoming IPIs. Although the CSD lock stalls are the worst case scenario, we have found many more subtle occurrences of this latency on the order of seconds, over a minute in some cases.
Quoted 5 lines
Prevent direct reclaim during compressed readahead. This is achieved by using different GFP flags whenever the bio is marked for readahead. The flags are similar to
GFP_NOFSbut stripped of__GFP_DIRECT_RECLAIM. Also,__GFP_NOWARNis added since these allocations are allowed to fail. Demand reads still use fullGFP_NOFSand will enter reclaim if needed.
This seems a sensible change to me. Read-ahead is speculative, so it's better for it to fail rather than cause problems elsewhere.
Quoted 14 lines
There has been some previous work done to reduce the frequency of calling
add_ra_bio_pages()[0]. This patch is complementary in that it reduces the latency associated with those calls.JP Kobryn (Meta) (2):
btrfs: additional gfp api for allocating compressed folios
btrfs: prevent direct reclaim during compressed readaheadfs/btrfs/compression.c | 44 ++++++++++++++++++++++++++++++++++-------- fs/btrfs/compression.h | 1 + 2 files changed, 37 insertions(+), 8 deletions(-)
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · JP Kobryn (Meta)
Prevent direct reclaim during compressed readahead.
This completely fails to explain why you want that.
I see that now.
Qu also had a good point about including info on why the current flags are an issue. Some of the cover letter text would help here. I'll bring some of that over and expand on it so we have the relevant details here.
Re: [PATCH 1/2] btrfs: additional gfp api for allocating compressed folios
In reply to JP Kobryn (Meta)Quoted 8 lines · JP Kobryn (Meta)
btrfs_alloc_compr_folio()assumes all callers wantGFP_NOFS. This is fine for most cases, however there are some call sites that would benefit from different flags. One such case is preventing direct reclaim from occurring during readahead allocations. With unbounded reclaim during this time, noticeable latency will occur under high memory pressure.Provide an additional API that accepts one additional
gfp_tparameter, giving callers flexibility over the characteristics of their allocation.
If you still need to set the gfp flags in v2, please drop this patch and extend btrfs_alloc_compr_folio(), the API is internal and we don't need it fine grained.