mm/filemap: reduce unnecessary xarray lookups
Subsystems (3)
cover mm/filemap: reduce unnecessary xarray lookups
From: Chi Zhiling <[email protected]>
This series optimizes xarray lookups in filemap by avoiding redundant iterations after obtaining the last needed folio. The boundary check is moved to before advancing the xarray iterator, eliminating unnecessary lookups and branches in the fast path. This reduces the overhead of filemap_get_read_batch() from 2.91% to 2.53% in 4k read tests.
This series is split from a previous patch series:
https://lore.kernel.org/linux-mm/[email protected]/
Chi Zhiling (2):
mm/filemap: reduce unnecessary xarray lookups when read cached pages mm/filemap: reduce unnecessary xarray lookups in filemap_get_folios_contig()
mm/filemap.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/2 mm/filemap: reduce unnecessary xarray lookups when read cached pages
From: Chi Zhiling <[email protected]>
When reading small amounts of data from the page cache, only a single folio is typically returned from filemap_read_get_batch(). In this case, calling xas_advance() or xas_next() after adding the folio to the batch is unnecessary and only introduces extra branches.
The same issue exists for large reads, where one additional xarray walk is always performed before termination.
Quit the loop once we get the last folio in the range, so the final redundant xarray advancement can be avoided.
The xas_next() does not update xa_index when xas->xa_node is set to XAS_RESTART, so the put and retry path would not update xa_index, hence the warning should therefore never trigger.
During the 4k reads test, the overhead of this function dropped from 2.91% to 2.53%.
--- mm/filemap.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
mm/filemap.c +6 -1
@@ -2467,11 +2467,14 @@ static void filemap_get_read_batch(struct address_space *mapping,XA_STATE (xas, &mapping->i_pages, index);struct folio *folio; +if (index > max) +return ; +rcu_read_lock ();for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ; -if (xas.xa_index > max ||xa_is_value (folio)) +if (xa_is_value (folio))break ;if (xa_is_sibling (folio))break ;
@@ -2488,6 +2491,8 @@ static void filemap_get_read_batch(struct address_space *mapping,if (folio_test_readahead (folio))break ;xas_advance (&xas,folio_next_index (folio) -1 ); +if (xas.xa_index >= max) +break ;continue ; put_folio:folio_put (folio);
Signature · 1 line
2.43.0
Re: [PATCH 1/2] mm/filemap: reduce unnecessary xarray lookups when read cached pages
Quoted 17 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
When reading small amounts of data from the page cache, only a single folio is typically returned from
filemap_read_get_batch(). In this case, callingxas_advance()orxas_next()after adding the folio to the batch is unnecessary and only introduces extra branches.The same issue exists for large reads, where one additional xarray walk is always performed before termination.
Quit the loop once we get the last folio in the range, so the final redundant xarray advancement can be avoided.
The
xas_next()does not updatexa_indexwhenxas->xa_nodeis set toXAS_RESTART, so the put and retry path would not updatexa_index, hence the warning should therefore never trigger.
Quoted 5 lines
During the 4k reads test, the overhead of this function dropped from 2.91% to 2.53%.
Suggested-by Matthew Wilcox (Oracle) <[email protected]>Signed-off-by Chi Zhiling <[email protected]>
Looks good. Feel free to add:
Honza
Quoted 36 lines
--- mm/filemap.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)mm/filemap.c +6
-1@@ -2467,11 +2467,14 @@ static void filemap_get_read_batch(struct address_space *mapping,XA_STATE (xas, &mapping->i_pages, index);struct folio *folio; +if (index > max) +return ; +rcu_read_lock ();for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ; -if (xas.xa_index > max ||xa_is_value (folio)) +if (xa_is_value (folio))break ;if (xa_is_sibling (folio))break ;@@ -2488,6 +2491,8 @@ static void filemap_get_read_batch(struct address_space *mapping,if (folio_test_readahead (folio))break ;xas_advance (&xas,folio_next_index (folio) -1 ); +if (xas.xa_index >= max) +break ;continue ; put_folio:folio_put (folio);Signature · 1 line
2.43.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH 2/2 mm/filemap: reduce unnecessary xarray lookups in filemap_get_folios_contig()
In reply to Chi ZhilingFrom: Chi Zhiling <[email protected]>
Apply the same optimization used in filemap_get_read_batch() by moving the boundary check from the loop condition to before xas_next(), avoiding an unnecessary xarray lookup and reducing branches in the fast path.
--- mm/filemap.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-)
mm/filemap.c +14 -15
@@ -2270,10 +2270,11 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,unsigned long nr;struct folio *folio; -rcu_read_lock (); +if (*start > end) +return 0 ; -for (folio =xas_load (&xas); folio && xas.xa_index <= end; - folio =xas_next (&xas)) { +rcu_read_lock (); +for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ;/*
@@ -2281,11 +2282,11 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,* No current caller is looking for DAX entries. */ if (xa_is_value (folio)) - goto update_start; +break ;/* If we landed in the middle of a THP, continue at its end. */ if (xa_is_sibling (folio)) - goto update_start; +break ;if (!folio_try_get (folio)) goto retry;
@@ -2293,29 +2294,27 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,if (unlikely (folio !=xas_reload (&xas))) goto put_folio; -if (!folio_batch_add (fbatch, folio)) { - *start =folio_next_index (folio); - goto out; - } +if (!folio_batch_add (fbatch, folio)) +break ; +xas_advance (&xas,folio_next_index (folio) -1 ); +if (xas.xa_index >= end) +break ;continue ; + put_folio:folio_put (folio); - retry:xas_reset (&xas); } +rcu_read_unlock (); -update_start: nr =folio_batch_count (fbatch); -if (nr) { folio = fbatch->folios[nr -1 ]; *start =folio_next_index (folio); } -out: -rcu_read_unlock (); -return folio_batch_count (fbatch); +return nr; }EXPORT_SYMBOL (filemap_get_folios_contig);
Signature · 1 line
2.43.0
Re: [PATCH 2/2] mm/filemap: reduce unnecessary xarray lookups in filemap_get_folios_contig()
Quoted 8 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Apply the same optimization used in
filemap_get_read_batch()by moving the boundary check from the loop condition to beforexas_next(), avoiding an unnecessary xarray lookup and reducing branches in the fast path.Signed-off-by Chi Zhiling <[email protected]>
Nice. Feel free to add:
Honza
Quoted 78 lines
--- mm/filemap.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-)mm/filemap.c +14
-15@@ -2270,10 +2270,11 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,unsigned long nr;struct folio *folio; -rcu_read_lock (); +if (*start > end) +return 0 ; -for (folio =xas_load (&xas); folio && xas.xa_index <= end; - folio =xas_next (&xas)) { +rcu_read_lock (); +for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ;/* @@ -2281,11 +2282,11 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,* No current caller is looking for DAX entries. */ if (xa_is_value (folio)) - goto update_start; +break ;/* If we landed in the middle of a THP, continue at its end. */ if (xa_is_sibling (folio)) - goto update_start; +break ;if (!folio_try_get (folio)) goto retry;@@ -2293,29 +2294,27 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,if (unlikely (folio !=xas_reload (&xas))) goto put_folio; -if (!folio_batch_add (fbatch, folio)) { - *start =folio_next_index (folio); - goto out; - } +if (!folio_batch_add (fbatch, folio)) +break ; +xas_advance (&xas,folio_next_index (folio) -1 ); +if (xas.xa_index >= end) +break ;continue ; + put_folio:folio_put (folio); - retry:xas_reset (&xas); } +rcu_read_unlock (); -update_start: nr =folio_batch_count (fbatch); -if (nr) { folio = fbatch->folios[nr -1 ]; *start =folio_next_index (folio); } -out: -rcu_read_unlock (); -return folio_batch_count (fbatch); +return nr; }EXPORT_SYMBOL (filemap_get_folios_contig);Signature · 1 line
2.43.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR