mm/shmem: optimize read with reduced xarray lookups and folio batching
Subsystems (7)
v2 cover mm/shmem: optimize read with reduced xarray lookups and folio batching
From: Chi Zhiling <[email protected]>
This series improves shmem read performance by implementing folio batching in the read path and reducing unnecessary xarray lookups.
Performance Results:
fio --ioengine=sync --rw=read --bs=$1 --size=1G --runtime=180 --time_based --group_reporting --name=seq_read_test --filename=testfile
| THP disabled in tmpfs | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | ---------------------- | ------------ | ----------------- | ----------- | | 1M + normal file | bw=11.5GiB/s | bw=12.7GiB/s | +10.4% | | 64k + normal file | bw=11.0GiB/s | bw=12.3GiB/s | +11.8% | | 4k + normal file | bw=3826MiB/s | bw=3849MiB/s | +0.6% | | 1M + fallocated file | bw=23.8GiB/s | bw=28.6GiB/s | +20.2% | | 64k + fallocated file | bw=22.5GiB/s | bw=27.3GiB/s | +21.3% | | 4k + fallocated file | bw=4655MiB/s | bw=4680MiB/s | +0.5% | | 1M + hole | bw=24.2GiB/s | bw=28.6GiB/s | +18.2% | | 64k + hole | bw=22.6GiB/s | bw=27.6GiB/s | +22.1% | | 4k + hole | bw=4652MiB/s | bw=4489MiB/s | -3.5% |
| THP enabled in tmpfs | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | --------------------- | ------------ | ----------------- | ----------- | | 1M + normal file | bw=13.7GiB/s | bw=13.9GiB/s | +1.4% | | 64k + normal file | bw=13.5GiB/s | bw=13.5GiB/s | +0.0% | | 4k + normal file | bw=3833MiB/s | bw=3859MiB/s | +0.7% | | 1M + fallocated file | bw=24.9GiB/s | bw=34.2GiB/s | +37.3% | | 64k + fallocated file | bw=23.0GiB/s | bw=31.4GiB/s | +36.5% | | 4k + fallocated file | bw=4710MiB/s | bw=4655MiB/s | -1.2% | | 1M + hole | bw=24.3GiB/s | bw=34.5GiB/s | +42.0% | | 64k + hole | bw=23.5GiB/s | bw=31.1GiB/s | +32.3% | | 4k + hole | bw=4690MiB/s | bw=4647MiB/s | -0.9% |
v1:
https://lore.kernel.org/linux-mm/[email protected]/#t
rfc:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Chi Zhiling (5):
mm/filemap: reduce unnecessary xarray lookups when read cached pages mm/filemap: reduce xarray lookups in filemap_get_folios_contig() mm/shmem: introduce copy_zero_to_iter() for large zeroing mm/shmem: remove page-copy fallback in shmem read path mm/shmem: optimize file read with folio batching
mm/filemap.c | 46 +++++++++++-------- mm/shmem.c | 126 +++++++++++++++++++++++++++++++++++---------------- 2 files changed, 113 insertions(+), 59 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/5 mm/filemap: reduce unnecessary xarray lookups when read cached pages
From: Chi Zhiling <[email protected]>
When reading small amounts of data from the page cache, only a single folio is typically returned from filemap_read_get_batch(). In this case, calling xas_advance() or xas_next() after adding the folio to the batch is unnecessary and only introduces extra branches.
The same issue exists for large reads, where one additional xarray walk is always performed before termination.
Quit the loop once we get the last folio in the range, so the final redundant xarray advancement can be avoided.
The xas_next() does not update xa_index when xas->xa_node is set to XAS_RESTART, so the put and retry path would not update xa_index, hence the warning should therefore never trigger.
During the 4k reads test, the overhead of this function dropped from 2.91% to 2.53%.
--- mm/filemap.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
mm/filemap.c +10 -2
@@ -2458,12 +2458,16 @@ static void filemap_get_read_batch(struct address_space *mapping, {XA_STATE (xas, &mapping->i_pages, index);struct folio *folio; +pgoff_t next; + +if (unlikely (index > max)) +return ;rcu_read_lock ();for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ; -if (xas.xa_index > max ||xa_is_value (folio)) +if (xa_is_value (folio) ||WARN_ON (xas.xa_index > max))break ;if (xa_is_sibling (folio))break ;
@@ -2479,7 +2483,11 @@ static void filemap_get_read_batch(struct address_space *mapping,break ;if (folio_test_readahead (folio))break ; -xas_advance (&xas,folio_next_index (folio) -1 ); + + next =folio_next_index (folio); +if (next > max) +break ; +xas_advance (&xas, next -1 );continue ; put_folio:folio_put (folio);
Signature · 1 line
2.43.0
Matthew Wilcox
maintainerRe: [PATCH v2 1/5] mm/filemap: reduce unnecessary xarray lookups when read cached pages
Quoted 6 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
When reading small amounts of data from the page cache, only a single folio is typically returned from
filemap_read_get_batch(). In this case, callingxas_advance()orxas_next()after adding the folio to the batch is unnecessary and only introduces extra branches.
I don't think xas_advance() is all that expensive. The expensive part is going to be calling xas_next() and realising that the folio now lies beyond the required range. I think you'll get just as much benefit from doing this:
mm/filemap.c +6 -1
@@ -2459,11 +2459,14 @@ static void filemap_get_read_batch(struct address_space *mapping,XA_STATE (xas, &mapping->i_pages, index);struct folio *folio; +if (index > max) +return ; +rcu_read_lock ();for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ; -if (xas.xa_index > max ||xa_is_value (folio)) +if (xa_is_value (folio))break ;if (xa_is_sibling (folio))break ;
@@ -2480,6 +2483,8 @@ static void filemap_get_read_batch(struct address_space *mapping,if (folio_test_readahead (folio))break ;xas_advance (&xas,folio_next_index (folio) -1 ); +if (xas.index >= max) +break ;continue ; put_folio:folio_put (folio);
Quoted 1 line
Re: [PATCH v2 1/5] mm/filemap: reduce unnecessary xarray lookups when read cached pages
Hi, Matthew
Quoted 42 lines · Matthew Wilcox
Quoted 6 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
When reading small amounts of data from the page cache, only a single folio is typically returned from
filemap_read_get_batch(). In this case, callingxas_advance()orxas_next()after adding the folio to the batch is unnecessary and only introduces extra branches.I don't think
xas_advance()is all that expensive. The expensive part is going to be callingxas_next()and realising that the folio now lies beyond the required range. I think you'll get just as much benefit from doing this:mm/filemap.c +6
-1@@ -2459,11 +2459,14 @@ static void filemap_get_read_batch(struct address_space *mapping,XA_STATE (xas, &mapping->i_pages, index);struct folio *folio; +if (index > max) +return ; +rcu_read_lock ();for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ; -if (xas.xa_index > max ||xa_is_value (folio)) +if (xa_is_value (folio))break ;if (xa_is_sibling (folio))break ;@@ -2480,6 +2483,8 @@ static void filemap_get_read_batch(struct address_space *mapping,if (folio_test_readahead (folio))break ;xas_advance (&xas,folio_next_index (folio) -1 ); +if (xas.index >= max) +break ;continue ; put_folio:folio_put (folio);
Quoted 1 line
Quoted 1 line
Looks good. I'll include it in the next version and make the same change in patch 2/5.
Thanks!
PATCH v2 2/5 mm/filemap: reduce xarray lookups in filemap_get_folios_contig()
In reply to Chi ZhilingFrom: Chi Zhiling <[email protected]>
Apply the same optimization used in filemap_get_read_batch() by moving the boundary check from the loop condition to before xas_advance(), avoiding an unnecessary xarray lookup and reducing branches in the fast path.
--- mm/filemap.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
mm/filemap.c +17 -17
@@ -2258,11 +2258,13 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,XA_STATE (xas, &mapping->i_pages, *start);unsigned long nr;struct folio *folio; +pgoff_t next; -rcu_read_lock (); +if (unlikely (*start > end)) +return 0 ; -for (folio =xas_load (&xas); folio && xas.xa_index <= end; - folio =xas_next (&xas)) { +rcu_read_lock (); +for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ;/*
@@ -2270,11 +2272,11 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,* No current caller is looking for DAX entries. */ if (xa_is_value (folio)) - goto update_start; +break ;/* If we landed in the middle of a THP, continue at its end. */ if (xa_is_sibling (folio)) - goto update_start; +break ;if (!folio_try_get (folio)) goto retry;
@@ -2282,30 +2284,28 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,if (unlikely (folio !=xas_reload (&xas))) goto put_folio; -if (!folio_batch_add (fbatch, folio)) { - nr =folio_nr_pages (folio); - *start = folio->index + nr; - goto out; - } -xas_advance (&xas,folio_next_index (folio) -1 ); +if (!folio_batch_add (fbatch, folio)) +break ; + + next =folio_next_index (folio); +if (next > end) +break ; +xas_advance (&xas, next -1 );continue ; + put_folio:folio_put (folio); - retry:xas_reset (&xas); } +rcu_read_unlock (); -update_start: nr =folio_batch_count (fbatch); -if (nr) { folio = fbatch->folios[nr -1 ]; *start =folio_next_index (folio); } -out: -rcu_read_unlock (); -return folio_batch_count (fbatch); +return nr; }EXPORT_SYMBOL (filemap_get_folios_contig);
Signature · 1 line
2.43.0
PATCH v2 3/5 mm/shmem: introduce copy_zero_to_iter() for large zeroing
In reply to Chi ZhilingFrom: Chi Zhiling <[email protected]>
This is a prep patch for shmem folio batching in the read path, where non-uptodate folios need to be handled in the main iteration loop. A large non-uptodate folio should be treated as a hole.
Currently, holes larger than PAGE_SIZE cannot be handled because ZERO_PAGE is limited to a single page. Add copy_zero_to_iter() as a wrapper to support copying larger zero ranges to the iterator.
--- mm/shmem.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
mm/shmem.c +15 -1
@@ -3339,6 +3339,20 @@ shmem_write_end(const struct kiocb *iocb, struct address_space *mapping,return copied; } +static size_t copy_zero_to_iter (size_t bytes,struct iov_iter *i) +{ +struct folio *zero =largest_zero_folio (); +unsigned long nr, ret, written =0 ; + +do { + nr =min (bytes - written,folio_size (zero)); + ret =copy_folio_to_iter (zero,0 , nr, i); + written += ret; + }while (written < bytes && ret == nr); + +return written; +} +static ssize_t shmem_file_read_iter (struct kiocb *iocb,struct iov_iter *to) {struct file *file = iocb->ki_filp;
@@ -3433,7 +3447,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)* clear_user() not so much, that it is noticeably * faster to copy the zero page instead of clearing. */ - ret =copy_page_to_iter (ZERO_PAGE (0 ), offset, nr, to); + ret =copy_zero_to_iter (nr, to); }else {/* * But submitting the same page twice in a row to
Signature · 1 line
2.43.0
Matthew Wilcox
maintainerRe: [PATCH v2 3/5] mm/shmem: introduce copy_zero_to_iter() for large zeroing
Quoted 3 lines · Chi Zhiling
Currently, holes larger than
PAGE_SIZEcannot be handled becauseZERO_PAGEis limited to a single page. Addcopy_zero_to_iter()as a wrapper to support copying larger zero ranges to the iterator.
I think Hugh put this optimisation in the wrong place, and you're perpetuating that ;-)
So perhaps we can start by moving this optimisation to lib/iov_iter.c? And then you can redo your optimisation on top of that.
lib/iov_iter.c +17 -1
@@ -451,7 +451,23 @@ static __always_inlinesize_t zero_to_user_iter (void __user *iter_to,size_t progress,size_t len,void *priv,void *priv2) { -return clear_user (iter_to, len); +/* +* it is noticeably faster to copy the zero page instead of +* calling clear_user(). Shame. +*/ +void *from =page_address (ZERO_PAGE (0 )); +size_t res =0 ; + +while (1 ) { +size_t n =min (len,PAGE_SIZE ); + n =copy_to_user_iter (iter_to, progress, n, from, priv2); + res += n; + len -= n; +if (!len || !n) +break ; + } + +return res; } static __always_inline
mm/shmem.c +0 -12
@@ -3427,19 +3427,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)else ret =copy_page_to_iter (page, offset, nr, to);folio_put (folio); - }else if (user_backed_iter (to)) { -/* -* Copy to user tends to be so well optimized, but -* clear_user() not so much, that it is noticeably -* faster to copy the zero page instead of clearing. -*/ - ret =copy_page_to_iter (ZERO_PAGE (0 ), offset, nr, to); }else { -/* -* But submitting the same page twice in a row to -* splice() - or others? - can result in confusion: -* so don't attempt that optimization on pipes etc. -*/ ret =iov_iter_zero (nr, to); }
Re: [PATCH v2 3/5] mm/shmem: introduce copy_zero_to_iter() for large zeroing
Quoted 10 lines · Matthew Wilcox
Quoted 3 lines · Chi Zhiling
Currently, holes larger than
PAGE_SIZEcannot be handled becauseZERO_PAGEis limited to a single page. Addcopy_zero_to_iter()as a wrapper to support copying larger zero ranges to the iterator.I think Hugh put this optimisation in the wrong place, and you're perpetuating that ;-)
So perhaps we can start by moving this optimisation to
lib/iov_iter.c? And then you can redo your optimisation on top of that.
Sure, it's good to see this optimization moved into another place :)
In that case, I'll assume iov_iter_zero() is already well optimized and drop copy_page_to_iter() in v3.
By the way, I assume the benefit of this optimization depends on the CPU architecture, right? If so, I wonder whether lib/iov_iter.c is the most appropriate place for it.
Quoted 13 lines
mm/shmem.c +0
-5@@ -3427,19 +3427,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)else ret =copy_page_to_iter (page, offset, nr, to);folio_put (folio); - }else if (user_backed_iter (to)) { -/* -* Copy to user tends to be so well optimized, but -* clear_user() not so much, that it is noticeably -* faster to copy the zero page instead of clearing.
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3432
Quoted 10 lines
mm/shmem.c +0
-7-*/ - ret =copy_page_to_iter (ZERO_PAGE (0 ), offset, nr, to); }else { -/* -* But submitting the same page twice in a row to -* splice() - or others? - can result in confusion: -* so don't attempt that optimization on pipes etc. -*/ ret =iov_iter_zero (nr, to); }
I'll include this change to v3.
Thanks,
Re: [PATCH v2 3/5] mm/shmem: introduce copy_zero_to_iter() for large zeroing
In reply to Matthew WilcoxQuoted 16 lines · Matthew Wilcox
Quoted 3 lines · Chi Zhiling
Currently, holes larger than
PAGE_SIZEcannot be handled becauseZERO_PAGEis limited to a single page. Addcopy_zero_to_iter()as a wrapper to support copying larger zero ranges to the iterator.I think Hugh put this optimisation in the wrong place, and you're perpetuating that ;-)
So perhaps we can start by moving this optimisation to
lib/iov_iter.c? And then you can redo your optimisation on top of that.lib/iov_iter.c +0
-0
lib/iov_iter.c static __always_inline · line 457
Quoted 8 lines
lib/iov_iter.c +4
-1@@ -451,7 +451,23 @@ static __always_inlinesize_t zero_to_user_iter (void __user *iter_to,size_t progress,size_t len,void *priv,void *priv2) { -return clear_user (iter_to, len); +/* +* it is noticeably faster to copy the zero page instead of +* calling clear_user(). Shame. +*/
This is a rather suspicious claim. If clear_user is indeed so terrible that it is faster to copy, the routine needs to get unfucked instead of the problem being worked around.
I can't speak for arm64 or other non-amd64 archs, maybe these are horrendeously broken.
On amd64 some archeology shows the following:
1. 0db7058e8e23e6bb ("x86/clear_user: Make it faster")
2022 vintage, replaces thoroughly terrible 8-byte per-iteration write with rep stos usage
2. 8c9b6a88b7e2f33c ("x86: improve on the non-rep 'clear_user' function")
inlines rep stosb at the callsite if the CPU has FSRS, otherwise fallsback to a new routine which does 64-byte writes per loop iteration.
FSRS is reasonably popular by now and chances are decent the test jig used by Chi has it.
For a size like 4096 bytes, the 64-byte loop will be slower than rep movsb and even rep stosq. This needs to be patched and maybe I'll get around to doing the needful(tm) in few days (it's not hard to write, but some care with testing is needed).
I could not be bothered to check how the workaround showed up, but it definitely needs to be removed as opposed to being perpetuated.
Re: [PATCH v2 3/5] mm/shmem: introduce copy_zero_to_iter() for large zeroing
Quoted 48 lines · Mateusz Guzik
Quoted 16 lines · Matthew Wilcox
Quoted 3 lines · Chi Zhiling
Currently, holes larger than
PAGE_SIZEcannot be handled becauseZERO_PAGEis limited to a single page. Addcopy_zero_to_iter()as a wrapper to support copying larger zero ranges to the iterator.I think Hugh put this optimisation in the wrong place, and you're perpetuating that ;-)
So perhaps we can start by moving this optimisation to
lib/iov_iter.c? And then you can redo your optimisation on top of that.lib/iov_iter.c +0
-0lib/iov_iter.c static __always_inline · line 457
Quoted 8 lines
lib/iov_iter.c +4
-1@@ -451,7 +451,23 @@ static __always_inlinesize_t zero_to_user_iter (void __user *iter_to,size_t progress,size_t len,void *priv,void *priv2) { -return clear_user (iter_to, len); +/* +* it is noticeably faster to copy the zero page instead of +* calling clear_user(). Shame. +*/ This is a rather suspicious claim. If
clear_useris indeed so terrible that it is faster to copy, the routine needs to get unfucked instead of the problem being worked around.I can't speak for arm64 or other non-amd64 archs, maybe these are horrendeously broken.
On amd64 some archeology shows the following:
1.0db7058e8e23e6bb("x86/clear_user: Make it faster")2022 vintage, replaces thoroughly terrible 8-byte per-iteration write with rep stos usage
2.
8c9b6a88b7e2f33c("x86: improve on the non-rep 'clear_user' function")inlines rep stosb at the callsite if the CPU has FSRS, otherwise fallsback to a new routine which does 64-byte writes per loop iteration.
FSRS is reasonably popular by now and chances are decent the test jig used by Chi has it.
Quoted 7 lines
For a size like 4096 bytes, the 64-byte loop will be slower than rep movsb and even rep stosq. This needs to be patched and maybe I'll get around to doing the needful(tm) in few days (it's not hard to write, but some care with testing is needed).
I could not be bothered to check how the workaround showed up, but it definitely needs to be removed as opposed to being perpetuated.
Maybe I was not clear enough, so here it is stated differently:
The routine was incredibly bad on amd64 prior to 2022, I presume the claim of bad performance predates the fixup. The current implementation for FSRS_enabled CPUs is fine and I guarantee it wont be slower than copying. For CPUs without FSRS the fallback is slower than it needs to be for sizes like 4096 bytes, that I'm going to fix soon(tm).
No matter what, there is no justification for issuing a copy from zero page just to avoid zeroing in place.
Matthew Wilcox
maintainerRe: [PATCH v2 3/5] mm/shmem: introduce copy_zero_to_iter() for large zeroing
In reply to Mateusz GuzikQuoted 9 lines · Mateusz Guzik
Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · Chi Zhiling
Currently, holes larger than
PAGE_SIZEcannot be handled becauseZERO_PAGEis limited to a single page. Addcopy_zero_to_iter()as a wrapper to support copying larger zero ranges to the iterator.I think Hugh put this optimisation in the wrong place, and you're perpetuating that ;-)
Quoted 6 lines
Quoted 2 lines
So perhaps we can start by moving this optimisation to
lib/iov_iter.c? And then you can redo your optimisation on top of that.This is a rather suspicious claim. If
clear_useris indeed so terrible that it is faster to copy, the routine needs to get unfucked instead of the problem being worked around.
Oh, I agree. Putting it in lib/iov_iter.c means more people will see it than if it's hidden in shmem.c.
Quoted 8 lines
I can't speak for arm64 or other non-amd64 archs, maybe these are horrendeously broken.
On amd64 some archeology shows the following:
1.0db7058e8e23e6bb("x86/clear_user: Make it faster")2022 vintage, replaces thoroughly terrible 8-byte per-iteration write with rep stos usage
That's a good candidate for fixing this problem. 56a8c8eb1eaf is from March 2022 and mentions the slowness of clear_user() on x86. So a commit from May 2022 might have fixed the problem without anyone going back to measure and remove the workaround.
Quoted 8 lines
2.
8c9b6a88b7e2f33c("x86: improve on the non-rep 'clear_user' function")inlines rep stosb at the callsite if the CPU has FSRS, otherwise fallsback to a new routine which does 64-byte writes per loop iteration.
FSRS is reasonably popular by now and chances are decent the test jig used by Chi has it.
Quoted 7 lines
For a size like 4096 bytes, the 64-byte loop will be slower than rep movsb and even rep stosq. This needs to be patched and maybe I'll get around to doing the needful(tm) in few days (it's not hard to write, but some care with testing is needed).
I could not be bothered to check how the workaround showed up, but it definitely needs to be removed as opposed to being perpetuated.
I'd be delighted if somebody tested just this patch. I'm not really set up for performance testing here.
mm/shmem.c +0 -12
@@ -3427,19 +3427,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)else ret =copy_page_to_iter (page, offset, nr, to);folio_put (folio); - }else if (user_backed_iter (to)) { -/* -* Copy to user tends to be so well optimized, but -* clear_user() not so much, that it is noticeably -* faster to copy the zero page instead of clearing. -*/ - ret =copy_page_to_iter (ZERO_PAGE (0 ), offset, nr, to); }else { -/* -* But submitting the same page twice in a row to -* splice() - or others? - can result in confusion: -* so don't attempt that optimization on pipes etc. -*/ ret =iov_iter_zero (nr, to); }
Re: [PATCH v2 3/5] mm/shmem: introduce copy_zero_to_iter() for large zeroing
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Mateusz Guzik
On Mon, Jun 01, 2026 at 02:22:04PM +0100, Matthew Wilcox wrote:
On amd64 some archeology shows the following:
1.0db7058e8e23e6bb("x86/clear_user: Make it faster")
Quoted 8 lines
Quoted 2 lines
2022 vintage, replaces thoroughly terrible 8-byte per-iteration write with rep stos usage
That's a good candidate for fixing this problem.
56a8c8eb1eafis from March 2022 and mentions the slowness ofclear_user()on x86. So a commit from May 2022 might have fixed the problem without anyone going back to measure and remove the workaround.
Ok in that case that's basically settled I think.
Quoted 17 lines
I'd be delighted if somebody tested just this patch. I'm not really set up for performance testing here.
mm/shmem.c +0
-6@@ -3427,19 +3427,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)else ret =copy_page_to_iter (page, offset, nr, to);folio_put (folio); - }else if (user_backed_iter (to)) { -/* -* Copy to user tends to be so well optimized, but -* clear_user() not so much, that it is noticeably -* faster to copy the zero page instead of clearing. -*/
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3433
Quoted 10 lines
mm/shmem.c +0
-6- ret =copy_page_to_iter (ZERO_PAGE (0 ), offset, nr, to); }else { -/* -* But submitting the same page twice in a row to -* splice() - or others? - can result in confusion: -* so don't attempt that optimization on pipes etc. -*/ ret =iov_iter_zero (nr, to); }
I'm not in position to test myself at the moment. But should someone be interested and try this on amd64, either make sure you have FSRS or in the worst case apply this hack which forces rep stosb:
(quoted diff) +0 -0
b/arch/x86/include/asm/uaccess_64.h
index 20de34cc9aa6..552d1b883579 100644
arch/x86/include/asm/uaccess_64.h +0 -0
@@ -189,8 +189,7 @@ static __always_inline __must_check unsigned long
__clear_user(void __user *addr
*/
asm volatile(
"1:\n\t"- ALTERNATIVE("rep stosb",
- "call rep_stos_alternative",ALT_NOT(X86_FEATURE_FSRS))
+ "rep stosb\n"
"2:\n"
_ASM_EXTABLE_UA(1b, 2b)
: "+c" (size), "+D" (addr), ASM_CALL_CONSTRAINTIt is always fine to use it from correctness pov. It would cause a slowdown for any size on old yellers predating Sandy Bridge (that's 14 years now I think?) or for small sizes on CPUs without FSRS. Given something like 4K I/O that's not a concern.
Re: [PATCH v2 3/5] mm/shmem: introduce copy_zero_to_iter() for large zeroing
In reply to Mateusz GuzikQuoted 45 lines · On Mon, 1 Jun 2026 17:02:01 +0200 Mateusz Guzik <[email protected]> wrote:
Quoted 16 lines · Matthew Wilcox
Quoted 3 lines · Chi Zhiling
Currently, holes larger than
PAGE_SIZEcannot be handled becauseZERO_PAGEis limited to a single page. Addcopy_zero_to_iter()as a wrapper to support copying larger zero ranges to the iterator.I think Hugh put this optimisation in the wrong place, and you're perpetuating that ;-)
So perhaps we can start by moving this optimisation to
lib/iov_iter.c? And then you can redo your optimisation on top of that.lib/iov_iter.c +0
-0lib/iov_iter.c static __always_inline · line 457
Quoted 8 lines
lib/iov_iter.c +4
-1@@ -451,7 +451,23 @@ static __always_inlinesize_t zero_to_user_iter (void __user *iter_to,size_t progress,size_t len,void *priv,void *priv2) { -return clear_user (iter_to, len); +/* +* it is noticeably faster to copy the zero page instead of +* calling clear_user(). Shame. +*/ This is a rather suspicious claim. If
clear_useris indeed so terrible that it is faster to copy, the routine needs to get unfucked instead of the problem being worked around.I can't speak for arm64 or other non-amd64 archs, maybe these are horrendeously broken.
On amd64 some archeology shows the following:
1.0db7058e8e23e6bb("x86/clear_user: Make it faster")2022 vintage, replaces thoroughly terrible 8-byte per-iteration write with rep stos usage
2.
8c9b6a88b7e2f33c("x86: improve on the non-rep 'clear_user' function")inlines rep stosb at the callsite if the CPU has FSRS, otherwise fallsback to a new routine which does 64-byte writes per loop iteration.
Quoted 7 lines
FSRS is reasonably popular by now and chances are decent the test jig used by Chi has it.
For a size like 4096 bytes, the 64-byte loop will be slower than rep movsb and even rep stosq. This needs to be patched and maybe I'll get around to doing the needful(tm) in few days (it's not hard to write, but some care with testing is needed).
I think Intel cpu from Sandy bridge onwards execute 'rep stosb' just as fast as 'rep stosq'.
(I'm sure I've done the measurements for 'rep movs' and stos ought to be similar.)
I suspect you get the same big gain (twice as fast) from an aligned destination (IIRC 64 bytes on later cpu).
(But I doubt it is worth the cost of aligning the destination.) The source alignment (for rep movs) make no difference at all.
The 'elephant in the room' is older zen cpu.
Some of those are no where near as fast as you might expect. (Look at the issues using 'rep movsb' for all copies.)
I can test a range of old Intel cpu, but not amd ones.
-- David
Quoted 4 lines
I could not be bothered to check how the workaround showed up, but it definitely needs to be removed as opposed to being perpetuated.
PATCH v2 4/5 mm/shmem: remove page-copy fallback in shmem read path
In reply to Chi ZhilingFrom: Chi Zhiling <[email protected]>
This is perp patch for folio batch support which removing the fallback to page-copy mode from shmem reads.
The existing hwpoison fallback would require fetching the same folio multiple times to read its each pages, which complicates the batching model. Instead, move hwpoison handling into copy_each_pages_to_iter(), allowing the error path to be centralized.
This simplifies the shmem fast read path and avoids special-case handling that conflicts with folio batching support.
--- mm/shmem.c | 63 +++++++++++++++++++++++++++++------------------------- 1 file changed, 34 insertions(+), 29 deletions(-)
mm/shmem.c +34 -29
@@ -3353,6 +3353,30 @@ static size_t copy_zero_to_iter(size_t bytes, struct iov_iter *i)return written; } +static size_t copy_pages_to_iter (struct folio *folio,size_t offset, +size_t bytes,struct iov_iter *i,int *error) +{ +struct page *page; +unsigned long off, nr, ret, written =0 ; + +do { + page =folio_page (folio, offset >>PAGE_SHIFT ); +if (PageHWPoison (page)) { + *error = -EIO ; +break ; + } + + off =offset_in_page (offset); + nr =min (PAGE_SIZE - off, bytes - written); + + ret =copy_page_to_iter (page, off, nr, i); + offset += ret; + written += ret; + }while (written < bytes && ret == nr); + +return written; +} +static ssize_t shmem_file_read_iter (struct kiocb *iocb,struct iov_iter *to) {struct file *file = iocb->ki_filp;
@@ -3365,10 +3389,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)for (;;) {struct folio *folio =NULL ; -struct page *page =NULL ;unsigned long nr, ret;loff_t end_offset, i_size =i_size_read (inode); -bool fallback_page_copy = false;size_t fsize;if (unlikely (iocb->ki_pos >= i_size))
@@ -3376,25 +3398,13 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) index = iocb->ki_pos >>PAGE_SHIFT ; error =shmem_get_folio (inode, index,0 , &folio,SGP_READ ); +if (folio) +folio_unlock (folio);if (error) {if (error == -EINVAL ) error =0 ;break ; } -if (folio) { -folio_unlock (folio); - - page =folio_file_page (folio, index); -if (PageHWPoison (page)) { -folio_put (folio); - error = -EIO ; -break ; - } - -if (folio_test_large (folio) && -folio_test_has_hwpoisoned (folio)) - fallback_page_copy = true; - }/* * We must evaluate after, since reads (unlike writes)
@@ -3406,12 +3416,9 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)folio_put (folio);break ; } - end_offset =min_t (loff_t, i_size, iocb->ki_pos + to->count); -if (folio &&likely (!fallback_page_copy)) - fsize =folio_size (folio); -else - fsize =PAGE_SIZE ; + fsize = folio ?folio_size (folio) :PAGE_SIZE ; offset = iocb->ki_pos & (fsize -1 ); + end_offset =min_t (loff_t, i_size, iocb->ki_pos + to->count); nr =min_t (loff_t, end_offset - iocb->ki_pos, fsize - offset);if (folio) {
@@ -3420,12 +3427,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)* virtual addresses, take care about potential aliasing * before reading the page on the kernel side. */ -if (mapping_writably_mapped (mapping)) { -if (likely (!fallback_page_copy)) -flush_dcache_folio (folio); -else -flush_dcache_page (page); - } +if (mapping_writably_mapped (mapping)) +flush_dcache_folio (folio);/* * Mark the folio accessed if we read the beginning.
@@ -3436,10 +3439,10 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)* Ok, we have the page, and it's up-to-date, so * now we can copy it to user space... */ -if (likely (!fallback_page_copy)) +if (likely (!folio_contain_hwpoisoned_page (folio))) ret =copy_folio_to_iter (folio, offset, nr, to);else - ret =copy_page_to_iter (page, offset, nr, to); + ret =copy_pages_to_iter (folio, offset, nr, to, &error);folio_put (folio); }else if (user_backed_iter (to)) {/*
@@ -3462,6 +3465,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)if (!iov_iter_count (to))break ; +if (unlikely (error)) +break ;if (ret < nr) { error = -EFAULT ;break ;
Signature · 1 line
2.43.0
Baolin Wang
reviewerRe: [PATCH v2 4/5] mm/shmem: remove page-copy fallback in shmem read path
Quoted 94 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
This is perp patch for folio batch support which removing the fallback to page-copy mode from shmem reads.
The existing hwpoison fallback would require fetching the same folio multiple times to read its each pages, which complicates the batching model. Instead, move hwpoison handling into
copy_each_pages_to_iter(), allowing the error path to be centralized.This simplifies the shmem fast read path and avoids special-case handling that conflicts with folio batching support.
Signed-off-by Chi Zhiling <[email protected]>--- mm/shmem.c | 63 +++++++++++++++++++++++++++++------------------------- 1 file changed, 34 insertions(+), 29 deletions(-)mm/shmem.c +26
-16@@ -3353,6 +3353,30 @@ static size_t copy_zero_to_iter(size_t bytes, struct iov_iter *i)return written; } +static size_t copy_pages_to_iter (struct folio *folio,size_t offset, +size_t bytes,struct iov_iter *i,int *error) +{ +struct page *page; +unsigned long off, nr, ret, written =0 ; + +do { + page =folio_page (folio, offset >>PAGE_SHIFT ); +if (PageHWPoison (page)) { + *error = -EIO ; +break ; + } + + off =offset_in_page (offset); + nr =min (PAGE_SIZE - off, bytes - written); + + ret =copy_page_to_iter (page, off, nr, i); + offset += ret; + written += ret; + }while (written < bytes && ret == nr); + +return written; +} +static ssize_t shmem_file_read_iter (struct kiocb *iocb,struct iov_iter *to) {struct file *file = iocb->ki_filp;@@ -3365,10 +3389,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)for (;;) {struct folio *folio =NULL ; -struct page *page =NULL ;unsigned long nr, ret;loff_t end_offset, i_size =i_size_read (inode); -bool fallback_page_copy = false;size_t fsize;if (unlikely (iocb->ki_pos >= i_size))@@ -3376,25 +3398,13 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) index = iocb->ki_pos >>PAGE_SHIFT ; error =shmem_get_folio (inode, index,0 , &folio,SGP_READ ); +if (folio) +folio_unlock (folio);if (error) {if (error == -EINVAL ) error =0 ;break ; } -if (folio) { -folio_unlock (folio); - - page =folio_file_page (folio, index); -if (PageHWPoison (page)) { -folio_put (folio); - error = -EIO ; -break ; - } - -if (folio_test_large (folio) && -folio_test_has_hwpoisoned (folio)) - fallback_page_copy = true; - }/* * We must evaluate after, since reads (unlike writes) @@ -3406,12 +3416,9 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)folio_put (folio);
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3421
Quoted 10 lines
mm/shmem.c +2
-5break ; } - end_offset =min_t (loff_t, i_size, iocb->ki_pos + to->count); -if (folio &&likely (!fallback_page_copy)) - fsize =folio_size (folio); -else - fsize =PAGE_SIZE ; + fsize = folio ?folio_size (folio) :PAGE_SIZE ; offset = iocb->ki_pos & (fsize -1 ); + end_offset =min_t (loff_t, i_size, iocb->ki_pos + to->count);
Do not add this unnecessary change.
Quoted 5 lines
mm/shmem.c +0
-0nr =min_t (loff_t, end_offset - iocb->ki_pos, fsize - offset);if (folio) {@@ -3420,12 +3427,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)* virtual addresses, take care about potential aliasing
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3431
Quoted 10 lines
mm/shmem.c +2
-6* before reading the page on the kernel side. */ -if (mapping_writably_mapped (mapping)) { -if (likely (!fallback_page_copy)) -flush_dcache_folio (folio); -else -flush_dcache_page (page); - } +if (mapping_writably_mapped (mapping)) +flush_dcache_folio (folio);
I'm not sure this is safe for all architectures (it's safe for arm64 since no actual data cache flush happens here), but for architectures that do perform an actual data cache flush, if the page has already been hwpoisoned, flushing the cache could trigger a RAS error?
I think you should move this into copy_pages_to_iter() for hwpoisoned folios.
Re: [PATCH v2 4/5] mm/shmem: remove page-copy fallback in shmem read path
Quoted 100 lines · Baolin Wang
Quoted 97 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
This is perp patch for folio batch support which removing the fallback to page-copy mode from shmem reads.
The existing hwpoison fallback would require fetching the same folio multiple times to read its each pages, which complicates the batching model. Instead, move hwpoison handling into
copy_each_pages_to_iter(), allowing the error path to be centralized.This simplifies the shmem fast read path and avoids special-case handling that conflicts with folio batching support.
Signed-off-by Chi Zhiling <[email protected]>---
mm/shmem.c| 63 +++++++++++++++++++++++++++++-------------------------1 file changed, 34 insertions(+), 29 deletions(-)mm/shmem.c +26
-18@@ -3353,6 +3353,30 @@ static size_t copy_zero_to_iter(size_t bytes, struct iov_iter *i) return written; } +static size_t copy_pages_to_iter (struct folio *folio,size_t offset, + size_t bytes, struct iov_iter *i, int *error) +{ + struct page *page; + unsigned long off, nr, ret, written = 0; + + do { + page = folio_page(folio, offset >> PAGE_SHIFT); + if (PageHWPoison(page)) { + *error = -EIO; + break; + } + + off = offset_in_page(offset); + nr = min(PAGE_SIZE - off, bytes - written); + + ret = copy_page_to_iter(page, off, nr, i); + offset += ret; + written += ret; + } while (written < bytes && ret == nr); + + return written; +} + static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) { struct file *file = iocb->ki_filp;@@ -3365,10 +3389,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) for (;;) { struct folio *folio = NULL; - struct page *page = NULL; unsigned long nr, ret; loff_t end_offset, i_size = i_size_read(inode); - bool fallback_page_copy = false; size_t fsize; if (unlikely(iocb->ki_pos >= i_size))@@ -3376,25 +3398,13 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) index = iocb->ki_pos >> PAGE_SHIFT; error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); + if (folio) + folio_unlock(folio); if (error) { if (error == -EINVAL) error = 0; break; } - if (folio) { - folio_unlock(folio); - - page = folio_file_page(folio, index); - if (PageHWPoison(page)) { - folio_put(folio); - error = -EIO; - break; - } - - if (folio_test_large(folio) && - folio_test_has_hwpoisoned(folio)) - fallback_page_copy = true; - } /** We must evaluate after, since reads (unlike writes) @@ -3406,12 +3416,9 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) folio_put(folio); break; } - end_offset = min_t(loff_t, i_size, iocb->ki_pos + to->count); - if (folio && likely(!fallback_page_copy))
Quoted 8 lines
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3421
Quoted 6 lines
mm/shmem.c +2
-3- fsize = folio_size(folio); - else - fsize = PAGE_SIZE; + fsize = folio ? folio_size(folio) : PAGE_SIZE; offset = iocb->ki_pos & (fsize - 1); + end_offset = min_t(loff_t, i_size, iocb->ki_pos + to->count);Do not add this unnecessary change.
Okay,
Quoted 19 lines
Quoted 6 lines
mm/shmem.c +0
-0nr = min_t(loff_t, end_offset - iocb->ki_pos, fsize - offset); if (folio) {@@ -3420,12 +3427,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) * virtual addresses, take care about potential aliasingmm/shmem.c static ssize_t shmem_file_read_iter(struct · line 3432
Quoted 10 lines
mm/shmem.c +2
-6* before reading the page on the kernel side. */ - if (mapping_writably_mapped(mapping)) { - if (likely(!fallback_page_copy)) - flush_dcache_folio(folio); - else - flush_dcache_page(page); - } + if (mapping_writably_mapped(mapping)) + flush_dcache_folio(folio);I'm not sure this is safe for all architectures (it's safe for arm64 since no actual data cache flush happens here), but for architectures that do perform an actual data cache flush, if the page has already been hwpoisoned, flushing the cache could trigger a RAS error?
Quoted 1 line
I think you should move this into
copy_pages_to_iter()for hwpoisoned folios.
Okay, will fix it, thanks!
PATCH v2 5/5 mm/shmem: optimize file read with folio batching
In reply to Chi ZhilingFrom: Chi Zhiling <[email protected]>
Optimize shmem file read by using filemap_get_folios_contig() to batch fetch contiguous folios from the page cache, reducing the overhead of repeated shmem_get_folio() calls.
This patch checks the uptodate flag without holding the folio lock, so it may observe a non-uptodate state on a locked folio that is still being initialized. This is safe because only zero-filled data can be copied to the user buffer in that scenario.
A non-uptodate folio in the swap cache cannot be added to the shmem page cache. This creates a semantic conflict, as shmem zeroes the folio out, but the swap cache would fill it by reading from the swap backing store.
--- mm/shmem.c | 57 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 15 deletions(-)
mm/shmem.c +42 -15
@@ -891,6 +891,14 @@ int shmem_add_to_page_cache(struct folio *folio,VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio);VM_BUG_ON_FOLIO (!folio_test_swapbacked (folio), folio); +/* +* Don't add a non-uptodate folio that is in swap cache to page +* cache, since shmem will zero it instead of reading from swap +* backing. +*/ +VM_BUG_ON_FOLIO (folio_test_swapcache (folio) && + !folio_test_uptodate (folio), folio); +folio_ref_add (folio, nr); folio->mapping = mapping; folio->index = index;
@@ -3382,11 +3390,13 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)struct file *file = iocb->ki_filp;struct inode *inode =file_inode (file);struct address_space *mapping = inode->i_mapping; -pgoff_t index; +struct folio_batch fbatch;unsigned long offset;int error =0 ;ssize_t retval =0 ; +folio_batch_init (&fbatch); +for (;;) {struct folio *folio =NULL ;unsigned long nr, ret;
@@ -3395,15 +3405,33 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)if (unlikely (iocb->ki_pos >= i_size))break ; +fetch: + folio =folio_batch_next (&fbatch); +if (!folio) { +pgoff_t start = iocb->ki_pos >>PAGE_SHIFT ; +pgoff_t end = (iocb->ki_pos + to->count -1 ) >>PAGE_SHIFT ; + +if (folio_batch_count (&fbatch)) { +for (int i =0 ; i <folio_batch_count (&fbatch); i++) +folio_put (fbatch.folios[i]); +folio_batch_reinit (&fbatch); + } - index = iocb->ki_pos >>PAGE_SHIFT ; - error =shmem_get_folio (inode, index,0 , &folio,SGP_READ ); -if (folio) -folio_unlock (folio); -if (error) { -if (error == -EINVAL ) - error =0 ; -break ; +filemap_get_folios_contig (inode->i_mapping, &start, end, &fbatch); +if (folio_batch_count (&fbatch)) + goto fetch; + + error =shmem_get_folio (inode, start,0 , &folio,SGP_READ ); +if (unlikely (error)) { +if (error == -EINVAL ) + error =0 ; +break ; + } +if (folio) { +folio_unlock (folio); +folio_batch_add (&fbatch, folio); + fbatch.i++; + } }/*
@@ -3411,17 +3439,15 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)* are called without i_rwsem protection against truncate */ i_size =i_size_read (inode); -if (unlikely (iocb->ki_pos >= i_size)) { -if (folio) -folio_put (folio); +if (unlikely (iocb->ki_pos >= i_size))break ; - } + fsize = folio ?folio_size (folio) :PAGE_SIZE ; offset = iocb->ki_pos & (fsize -1 ); end_offset =min_t (loff_t, i_size, iocb->ki_pos + to->count); nr =min_t (loff_t, end_offset - iocb->ki_pos, fsize - offset); -if (folio) { +if (folio &&folio_test_uptodate (folio)) {/* * If users can be writing to this page using arbitrary * virtual addresses, take care about potential aliasing
@@ -3443,7 +3469,6 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) ret =copy_folio_to_iter (folio, offset, nr, to); else ret =copy_pages_to_iter (folio, offset, nr, to, &error); -folio_put (folio); }else if (user_backed_iter (to)) {/* * Copy to user tends to be so well optimized, but
@@ -3474,6 +3499,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)cond_resched (); } +for (int i =0 ; i <folio_batch_count (&fbatch); i++) +folio_put (fbatch.folios[i]);file_accessed (file);return retval ? retval : error; }
Signature · 1 line
2.43.0
Matthew Wilcox
maintainerRe: [PATCH v2 5/5] mm/shmem: optimize file read with folio batching
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3418
Quoted 5 lines · Chi Zhiling
mm/shmem.c +5
-0+if (folio_batch_count (&fbatch)) { +for (int i =0 ; i <folio_batch_count (&fbatch); i++) +folio_put (fbatch.folios[i]); +folio_batch_reinit (&fbatch); + }
folios_put().
Review
Quoted 2 lines
(quoted diff) +2
-0+ for (int i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]);
Also folios_put().
Re: [PATCH v2 5/5] mm/shmem: optimize file read with folio batching
Quoted 9 lines · Matthew Wilcox
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3418
Quoted 5 lines · Chi Zhiling
mm/shmem.c +5
-0+if (folio_batch_count (&fbatch)) { +for (int i =0 ; i <folio_batch_count (&fbatch); i++) +folio_put (fbatch.folios[i]); +folio_batch_reinit (&fbatch); + }
folios_put().
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ for (int i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]);Also
folios_put().
Thanks, I'll replace it in v3.
Baolin Wang
reviewerRe: [PATCH v2 5/5] mm/shmem: optimize file read with folio batching
In reply to Chi ZhilingQuoted 25 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Optimize shmem file read by using
filemap_get_folios_contig()to batch fetch contiguous folios from the page cache, reducing the overhead of repeatedshmem_get_folio()calls.This patch checks the uptodate flag without holding the folio lock, so it may observe a non-uptodate state on a locked folio that is still being initialized. This is safe because only zero-filled data can be copied to the user buffer in that scenario.
A non-uptodate folio in the swap cache cannot be added to the shmem page cache. This creates a semantic conflict, as shmem zeroes the folio out, but the swap cache would fill it by reading from the swap backing store.
Signed-off-by Chi Zhiling <[email protected]>--- mm/shmem.c | 57 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 15 deletions(-)mm/shmem.c +0
-0
mm/shmem.c int shmem_add_to_page_cache(struct folio *folio, · line 900
Quoted 10 lines
mm/shmem.c +7
-0@@ -891,6 +891,14 @@ int shmem_add_to_page_cache(struct folio *folio,VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio);VM_BUG_ON_FOLIO (!folio_test_swapbacked (folio), folio); +/* +* Don't add a non-uptodate folio that is in swap cache to page +* cache, since shmem will zero it instead of reading from swap +* backing. +*/ +VM_BUG_ON_FOLIO (folio_test_swapcache (folio) && + !folio_test_uptodate (folio), folio);
It's impossible for a folio to be in both the swap cache and the shmem page cache. We can drop this.
Quoted 19 lines
mm/shmem.c +3
-1folio_ref_add (folio, nr); folio->mapping = mapping; folio->index = index;@@ -3382,11 +3390,13 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)struct file *file = iocb->ki_filp;struct inode *inode =file_inode (file);struct address_space *mapping = inode->i_mapping; -pgoff_t index; +struct folio_batch fbatch;unsigned long offset;int error =0 ;ssize_t retval =0 ; +folio_batch_init (&fbatch); +for (;;) {struct folio *folio =NULL ;unsigned long nr, ret;
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3412
Quoted 8 lines
mm/shmem.c +5
-0@@ -3395,15 +3405,33 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)if (unlikely (iocb->ki_pos >= i_size))break ; +fetch: + folio =folio_batch_next (&fbatch); +if (!folio) { +pgoff_t start = iocb->ki_pos >>PAGE_SHIFT ; +pgoff_t end = (iocb->ki_pos + to->count -1 ) >>PAGE_SHIFT ;
You should consider the inode size when calculating the 'end'. You can reuse the 'end_offset':
end_offset = min_t(loff_t, i_size, iocb->ki_pos + to->count);
then pass 'end_offset - 1' to filemap_get_folios_contig().
Quoted 71 lines
mm/shmem.c +26
-14+ +if (folio_batch_count (&fbatch)) { +for (int i =0 ; i <folio_batch_count (&fbatch); i++) +folio_put (fbatch.folios[i]); +folio_batch_reinit (&fbatch); + } - index = iocb->ki_pos >>PAGE_SHIFT ; - error =shmem_get_folio (inode, index,0 , &folio,SGP_READ ); -if (folio) -folio_unlock (folio); -if (error) { -if (error == -EINVAL ) - error =0 ; -break ; +filemap_get_folios_contig (inode->i_mapping, &start, end, &fbatch); +if (folio_batch_count (&fbatch)) + goto fetch; + + error =shmem_get_folio (inode, start,0 , &folio,SGP_READ ); +if (unlikely (error)) { +if (error == -EINVAL ) + error =0 ; +break ; + } +if (folio) { +folio_unlock (folio); +folio_batch_add (&fbatch, folio); + fbatch.i++; + } }/* @@ -3411,17 +3439,15 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)* are called without i_rwsem protection against truncate */ i_size =i_size_read (inode); -if (unlikely (iocb->ki_pos >= i_size)) { -if (folio) -folio_put (folio); +if (unlikely (iocb->ki_pos >= i_size))break ; - } + fsize = folio ?folio_size (folio) :PAGE_SIZE ; offset = iocb->ki_pos & (fsize -1 ); end_offset =min_t (loff_t, i_size, iocb->ki_pos + to->count); nr =min_t (loff_t, end_offset - iocb->ki_pos, fsize - offset); -if (folio) { +if (folio &&folio_test_uptodate (folio)) {/* * If users can be writing to this page using arbitrary * virtual addresses, take care about potential aliasing @@ -3443,7 +3469,6 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) ret =copy_folio_to_iter (folio, offset, nr, to); else ret =copy_pages_to_iter (folio, offset, nr, to, &error); -folio_put (folio); }else if (user_backed_iter (to)) {/* * Copy to user tends to be so well optimized, but @@ -3474,6 +3499,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)cond_resched (); } +for (int i =0 ; i <folio_batch_count (&fbatch); i++) +folio_put (fbatch.folios[i]);file_accessed (file);return retval ? retval : error; }
Re: [PATCH v2 5/5] mm/shmem: optimize file read with folio batching
Quoted 21 lines · Baolin Wang
Quoted 20 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Optimize shmem file read by using
filemap_get_folios_contig()to batch fetch contiguous folios from the page cache, reducing the overhead of repeatedshmem_get_folio()calls.This patch checks the uptodate flag without holding the folio lock, so it may observe a non-uptodate state on a locked folio that is still being initialized. This is safe because only zero-filled data can be copied to the user buffer in that scenario.
A non-uptodate folio in the swap cache cannot be added to the shmem page cache. This creates a semantic conflict, as shmem zeroes the folio out, but the swap cache would fill it by reading from the swap backing store.
Signed-off-by Chi Zhiling <[email protected]>---
mm/shmem.c| 57 ++++++++++++++++++++++++++++++++++++++++--------------1 file changed, 42 insertions(+), 15 deletions(-)
Quoted 16 lines
Quoted 5 lines
mm/shmem.c +0
-0mm/shmem.c int shmem_add_to_page_cache(struct folio *folio, · line 899
Quoted 9 lines
mm/shmem.c +7
-0@@ -891,6 +891,14 @@ int shmem_add_to_page_cache(struct folio *folio, VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_swapbacked(folio), folio); + /* + * Don't add a non-uptodate folio that is in swap cache to page + * cache, since shmem will zero it instead of reading from swap + * backing. + */ + VM_BUG_ON_FOLIO(folio_test_swapcache(folio) && + !folio_test_uptodate(folio), folio);It's impossible for a folio to be in both the swap cache and the shmem page cache. We can drop this.
Okay, I just want to set a reminder for others.
Quoted 31 lines
Quoted 20 lines
mm/shmem.c +3
-1folio_ref_add(folio, nr); folio->mapping = mapping; folio->index = index;@@ -3382,11 +3390,13 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) struct file *file = iocb->ki_filp; struct inode *inode = file_inode(file); struct address_space *mapping = inode->i_mapping; - pgoff_t index; + struct folio_batch fbatch; unsigned long offset; int error = 0; ssize_t retval = 0; + folio_batch_init(&fbatch); + for (;;) { struct folio *folio = NULL; unsigned long nr, ret;mm/shmem.c static ssize_t shmem_file_read_iter(struct · line 3412
Quoted 8 lines
mm/shmem.c +5
-0@@ -3395,15 +3405,33 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) if (unlikely(iocb->ki_pos >= i_size)) break; +fetch: + folio = folio_batch_next(&fbatch); + if (!folio) { + pgoff_t start = iocb->ki_pos >> PAGE_SHIFT; + pgoff_t end = (iocb->ki_pos + to->count - 1) >> PAGE_SHIFT;You should consider the inode size when calculating the 'end'. You can reuse the '
end_offset':
Quoted 3 lines
end_offset=min_t(loff_t, i_size, iocb->ki_pos + to->count);then pass '
end_offset- 1' tofilemap_get_folios_contig().
Okay, I will.
And I'm considering whether to reintroduce shmem_get_read_batch as a replacement for filemap_get_folios_contig.
The main reason is to be able to quickly skip over holes, so that we don't fall back when encountering holes during reads, thereby addressing the performance regression when reading holes.
Quoted 72 lines
Quoted 72 lines
mm/shmem.c +24
-13+ + if (folio_batch_count(&fbatch)) { + for (int i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]); + folio_batch_reinit(&fbatch); + } - index = iocb->ki_pos >> PAGE_SHIFT; - error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); - if (folio) - folio_unlock(folio); - if (error) { - if (error == -EINVAL) - error = 0; - break; + filemap_get_folios_contig(inode->i_mapping, &start, end, &fbatch); + if (folio_batch_count(&fbatch)) + goto fetch; + + error = shmem_get_folio(inode, start, 0, &folio, SGP_READ); + if (unlikely(error)) { + if (error == -EINVAL) + error = 0; + break; + } + if (folio) { + folio_unlock(folio); + folio_batch_add(&fbatch, folio); + fbatch.i++; + } } /*@@ -3411,17 +3439,15 @@ static ssize_t shmem_file_read_iter(structkiocb *iocb, struct iov_iter *to) * are called without i_rwsem protection against truncate */ i_size = i_size_read(inode); - if (unlikely(iocb->ki_pos >= i_size)) { - if (folio) - folio_put(folio); + if (unlikely(iocb->ki_pos >= i_size)) break; - } + fsize = folio ? folio_size(folio) : PAGE_SIZE; offset = iocb->ki_pos & (fsize - 1); end_offset = min_t(loff_t, i_size, iocb->ki_pos + to->count); nr = min_t(loff_t, end_offset - iocb->ki_pos, fsize - offset); - if (folio) { + if (folio && folio_test_uptodate(folio)) { /** If users can be writing to this page using arbitrary * virtual addresses, take care about potential aliasing @@ -3443,7 +3469,6 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) ret = copy_folio_to_iter(folio, offset, nr, to);else
ret =copy_pages_to_iter(folio, offset, nr, to, &error); -folio_put(folio);
} else if (user_backed_iter(to)) {
/*
* Copy to user tends to be so well optimized, butmm/shmem.c +2
-0@@ -3474,6 +3499,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) cond_resched(); } + for (int i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]); file_accessed(file); return retval ? retval : error; }
Quoted 17 lines · Chi Zhiling
Performance Results:
fio --ioengine=sync --rw=read --bs=$1 --size=1G --runtime=180 --time_based --group_reporting --name=seq_read_test --filename=testfile
| THP disabled in tmpfs | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | ---------------------- | ------------ | ----------------- | ----------- | | 1M + normal file | bw=11.5GiB/s | bw=12.7GiB/s | +10.4% | | 64k + normal file | bw=11.0GiB/s | bw=12.3GiB/s | +11.8% | | 4k + normal file | bw=3826MiB/s | bw=3849MiB/s | +0.6% | | 1M + fallocated file | bw=23.8GiB/s | bw=28.6GiB/s | +20.2% | | 64k + fallocated file | bw=22.5GiB/s | bw=27.3GiB/s | +21.3% | | 4k + fallocated file | bw=4655MiB/s | bw=4680MiB/s | +0.5% | | 1M + hole | bw=24.2GiB/s | bw=28.6GiB/s | +18.2% | | 64k + hole | bw=22.6GiB/s | bw=27.6GiB/s | +22.1% | | 4k + hole | bw=4652MiB/s | bw=4489MiB/s | -3.5% |
Quoted 12 lines
| THP enabled in tmpfs | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | --------------------- | ------------ | ----------------- | ----------- | | 1M + normal file | bw=13.7GiB/s | bw=13.9GiB/s | +1.4% | | 64k + normal file | bw=13.5GiB/s | bw=13.5GiB/s | +0.0% | | 4k + normal file | bw=3833MiB/s | bw=3859MiB/s | +0.7% | | 1M + fallocated file | bw=24.9GiB/s | bw=34.2GiB/s | +37.3% | | 64k + fallocated file | bw=23.0GiB/s | bw=31.4GiB/s | +36.5% | | 4k + fallocated file | bw=4710MiB/s | bw=4655MiB/s | -1.2% | | 1M + hole | bw=24.3GiB/s | bw=34.5GiB/s | +42.0% | | 64k + hole | bw=23.5GiB/s | bw=31.1GiB/s | +32.3% | | 4k + hole | bw=4690MiB/s | bw=4647MiB/s | -0.9% |
Apologies, due to my oversight, the tests involving hole were incorrect, the holes were not successfully created in the files during testing.
Below are the corrected results from a retest:
| THP disabled | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | ------------ | ------------ | ----------------- | ----------- | | 1M + hole | bw=27.3GiB/s | bw=23.4GiB/s | -14.3% | | 64k + hole | bw=27.3GiB/s | bw=23.3GiB/s | -14.7% | | 4k + hole | bw=4825MiB/s | bw=4624MiB/s | -4.2% |
| THP enabled | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | ----------- | ------------ | ----------------- | ----------- | | 1M + hole | bw=27.0GiB/s | bw=23.1GiB/s | -14.4% | | 64k + hole | bw=27.5GiB/s | bw=23.3GiB/s | -15.3% | | 4k + hole | bw=4777MiB/s | bw=4640MiB/s | -2.9% |
There is a noticeable performance drop when accessing holes, as every read triggers a fallback. I will address this in the next version.
Andrew Morton
maintainerQuoted 22 lines · On Mon, 1 Jun 2026 13:56:59 +0800 Chi Zhiling <[email protected]> wrote:
From: Chi Zhiling <[email protected]>
This series improves shmem read performance by implementing folio batching in the read path and reducing unnecessary xarray lookups.
Performance Results:
fio --ioengine=sync --rw=read --bs=$1 --size=1G --runtime=180 --time_based --group_reporting --name=seq_read_test --filename=testfile
| THP disabled in tmpfs | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | ---------------------- | ------------ | ----------------- | ----------- | | 1M + normal file | bw=11.5GiB/s | bw=12.7GiB/s | +10.4% | | 64k + normal file | bw=11.0GiB/s | bw=12.3GiB/s | +11.8% | | 4k + normal file | bw=3826MiB/s | bw=3849MiB/s | +0.6% | | 1M + fallocated file | bw=23.8GiB/s | bw=28.6GiB/s | +20.2% | | 64k + fallocated file | bw=22.5GiB/s | bw=27.3GiB/s | +21.3% | | 4k + fallocated file | bw=4655MiB/s | bw=4680MiB/s | +0.5% | | 1M + hole | bw=24.2GiB/s | bw=28.6GiB/s | +18.2% | | 64k + hole | bw=22.6GiB/s | bw=27.6GiB/s | +22.1% | | 4k + hole | bw=4652MiB/s | bw=4489MiB/s | -3.5% |
Quoted 12 lines
| THP enabled in tmpfs | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | --------------------- | ------------ | ----------------- | ----------- | | 1M + normal file | bw=13.7GiB/s | bw=13.9GiB/s | +1.4% | | 64k + normal file | bw=13.5GiB/s | bw=13.5GiB/s | +0.0% | | 4k + normal file | bw=3833MiB/s | bw=3859MiB/s | +0.7% | | 1M + fallocated file | bw=24.9GiB/s | bw=34.2GiB/s | +37.3% | | 64k + fallocated file | bw=23.0GiB/s | bw=31.4GiB/s | +36.5% | | 4k + fallocated file | bw=4710MiB/s | bw=4655MiB/s | -1.2% | | 1M + hole | bw=24.3GiB/s | bw=34.5GiB/s | +42.0% | | 64k + hole | bw=23.5GiB/s | bw=31.1GiB/s | +32.3% | | 4k + hole | bw=4690MiB/s | bw=4647MiB/s | -0.9% |
That looks nice.
Microbenchmarks are useful, but are you able to help us understand how much benefit our users might see in real-world workloads?
I'll take no action at this time - it's late in the cycle and reviewers have yet to participate.
AI review flagged a few possible issues, so please take a look:
https://sashiko.dev/#/patchset/[email protected]
Quoted 37 lines · Andrew Morton
Quoted 22 lines · On Mon, 1 Jun 2026 13:56:59 +0800 Chi Zhiling <[email protected]> wrote:
From: Chi Zhiling <[email protected]>
This series improves shmem read performance by implementing folio batching in the read path and reducing unnecessary xarray lookups.
Performance Results:
fio --ioengine=sync --rw=read --bs=$1 --size=1G --runtime=180 --time_based --group_reporting --name=seq_read_test --filename=testfile
| THP disabled in tmpfs | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | ---------------------- | ------------ | ----------------- | ----------- | | 1M + normal file | bw=11.5GiB/s | bw=12.7GiB/s | +10.4% | | 64k + normal file | bw=11.0GiB/s | bw=12.3GiB/s | +11.8% | | 4k + normal file | bw=3826MiB/s | bw=3849MiB/s | +0.6% | | 1M + fallocated file | bw=23.8GiB/s | bw=28.6GiB/s | +20.2% | | 64k + fallocated file | bw=22.5GiB/s | bw=27.3GiB/s | +21.3% | | 4k + fallocated file | bw=4655MiB/s | bw=4680MiB/s | +0.5% | | 1M + hole | bw=24.2GiB/s | bw=28.6GiB/s | +18.2% | | 64k + hole | bw=22.6GiB/s | bw=27.6GiB/s | +22.1% | | 4k + hole | bw=4652MiB/s | bw=4489MiB/s | -3.5% |Quoted 12 lines
| THP enabled in tmpfs | v7.1-rc5 | v7.1-rc5 + fbatch | Improvement | | --------------------- | ------------ | ----------------- | ----------- | | 1M + normal file | bw=13.7GiB/s | bw=13.9GiB/s | +1.4% | | 64k + normal file | bw=13.5GiB/s | bw=13.5GiB/s | +0.0% | | 4k + normal file | bw=3833MiB/s | bw=3859MiB/s | +0.7% | | 1M + fallocated file | bw=24.9GiB/s | bw=34.2GiB/s | +37.3% | | 64k + fallocated file | bw=23.0GiB/s | bw=31.4GiB/s | +36.5% | | 4k + fallocated file | bw=4710MiB/s | bw=4655MiB/s | -1.2% | | 1M + hole | bw=24.3GiB/s | bw=34.5GiB/s | +42.0% | | 64k + hole | bw=23.5GiB/s | bw=31.1GiB/s | +32.3% | | 4k + hole | bw=4690MiB/s | bw=4647MiB/s | -0.9% |
Quoted 4 lines
That looks nice.
Microbenchmarks are useful, but are you able to help us understand how much benefit our users might see in real-world workloads?
Hi, Andrew
I don't have real-world performance data yet. I'm working on this simply because the patch shows decent gains in microbenchmarks. Even with THP enabled, it can still reduce some unnecessary overhead.
Quoted 3 lines
I'll take no action at this time - it's late in the cycle and reviewers have yet to participate.
Yes, it's unlikely to land in 7.2, and I still need to resolve some performance regressions.
Quoted 3 lines
AI review flagged a few possible issues, so please take a look:
https://sashiko.dev/#/patchset/[email protected]
Okay, I will take a close look.
Thanks!