mm/shmem: optimize read with reduced xarray lookups and folio batching
Subsystems (5)
cover mm/shmem: optimize read with reduced xarray lookups and folio batching
From: Chi Zhiling <[email protected]>
This series improves shmem read performance by implementing folio batching in the read path and reducing unnecessary xarray lookups.
Changes since RFC
=================
The RFC version used xas_for_each() in shmem_get_read_batch(), which introduced about a 1% regression for 4K read workloads.
This v1 addresses the regression in patch 2 by switching to filemap_get_folios_contig() and optimizing it to avoid the extra xarray traversal overhead.
Performance Results
===================
Testing was performed with fio sequential read workloads:
fio --ioengine=sync --rw=read --size=1G --runtime=180
### THP Disabled - Normal Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 11.4GiB/s | 12.7GiB/s | +11.4% | | 64k | 11.2GiB/s | 12.2GiB/s | +8.9% | | 4k | 3809MiB/s | 3838MiB/s | +0.8% |
### THP Disabled - Fallocated Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 23.7GiB/s | 28.7GiB/s | +21.1% | | 64k | 22.6GiB/s | 27.0GiB/s | +19.5% | | 4k | 4668MiB/s | 4678MiB/s | +0.2% |
### THP Enabled - Normal Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 13.9GiB/s | 13.9GiB/s | 0% | | 64k | 13.4GiB/s | 13.4GiB/s | 0% | | 4k | 3818MiB/s | 3836MiB/s | +0.5% |
### THP Enabled - Fallocated Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 24.1GiB/s | 34.9GiB/s | +44.8% | | 64k | 22.9GiB/s | 31.3GiB/s | +36.7% | | 4k | 4721MiB/s | 4708MiB/s | -0.3% |
rfc:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Chi Zhiling (5):
mm/filemap: reduce unnecessary xarray lookups when read cached pages mm/filemap: reduce xarray lookups in filemap_get_folios_contig() mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ mm/shmem: introduce copy_zero_to_iter() for large zeroing mm/shmem: optimize file read with folio batching
include/linux/shmem_fs.h | 2 +- mm/filemap.c | 34 ++++++++++++------ mm/khugepaged.c | 2 +- mm/shmem.c | 75 +++++++++++++++++++++++++++++----------- 4 files changed, 79 insertions(+), 34 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/5 mm/filemap: reduce unnecessary xarray lookups when read cached pages
From: Chi Zhiling <[email protected]>
When reading small amounts of data from the page cache, only a single folio is typically returned from filemap_read_get_batch(). In this case, calling xas_advance() or xas_next() after adding the folio to the batch is unnecessary and only introduces extra branches.
The same issue exists for large reads, where one additional xarray walk is always performed before termination.
Move the boundary check to after the folio is added to the batch so the final redundant xarray advancement can be avoided. This significantly reduces the branch count in the read path.
xas_next() does not update xa_index when xas->xa_node is set to XAS_RESTART, so checking the boundary before updating xa_index is sufficient to keep the folio within range. The warning should therefore never trigger.
The branch count:
654.198 M/sec -> 646.444 M/sec
Performance counter stats for 'fio --ioengine=sync --rw=read --bs=4k --size=1G --runtime=300 --time_based --group_reporting --name=seq_read_test --filename=file':
before:
READ: bw=2697MiB/s (2828MB/s), 2697MiB/s-2697MiB/s (2828MB/s-2828MB/s), io=790GiB (848GB), run=300001-300001msec
245602051556 task-clock # 0.821 CPUs utilized
78467 context-switches # 319.488 /sec
40 cpu-migrations # 0.163 /sec
3388 page-faults # 13.795 /sec
758312319204 instructions # 0.74 insn per cycle
1025881497502 cycles # 4.177 GHz
160672383734 branches # 654.198 M/sec
361904512 branch-misses # 0.23% of all branchesafter:
READ: bw=2709MiB/s (2841MB/s), 2709MiB/s-2709MiB/s (2841MB/s-2841MB/s), io=794GiB (852GB), run=300000-300000msec
243985503670 task-clock # 0.812 CPUs utilized
79004 context-switches # 323.806 /sec
30 cpu-migrations # 0.123 /sec
3355 page-faults # 13.751 /sec
747830935069 instructions # 0.73 insn per cycle
1019609333322 cycles # 4.179 GHz
157722976668 branches # 646.444 M/sec
348984893 branch-misses # 0.22% of all branches--- mm/filemap.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
mm/filemap.c +10 -2
@@ -2458,12 +2458,16 @@ static void filemap_get_read_batch(struct address_space *mapping, {XA_STATE (xas, &mapping->i_pages, index);struct folio *folio; +pgoff_t next; + +if (unlikely (index > max)) +return ;rcu_read_lock ();for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ; -if (xas.xa_index > max ||xa_is_value (folio)) +if (xa_is_value (folio) ||WARN_ON (xas.xa_index > max))break ;if (xa_is_sibling (folio))break ;
@@ -2479,7 +2483,11 @@ static void filemap_get_read_batch(struct address_space *mapping,break ;if (folio_test_readahead (folio))break ; -xas_advance (&xas,folio_next_index (folio) -1 ); + + next =folio_next_index (folio); +if (next > max) +break ; +xas_advance (&xas, next -1 );continue ; put_folio:folio_put (folio);
Signature · 1 line
2.43.0
PATCH 2/5 mm/filemap: reduce xarray lookups in filemap_get_folios_contig()
In reply to Chi ZhilingFrom: Chi Zhiling <[email protected]>
Apply the same optimization used in filemap_get_read_batch() by moving the boundary check from the loop condition to before xas_advance(), avoiding an unnecessary xarray lookup and reducing branches in the fast path.
--- mm/filemap.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-)
mm/filemap.c +13 -9
@@ -2258,11 +2258,14 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,XA_STATE (xas, &mapping->i_pages, *start);unsigned long nr;struct folio *folio; +pgoff_t next; + +if (unlikely (*start > end)) +return 0 ;rcu_read_lock (); -for (folio =xas_load (&xas); folio && xas.xa_index <= end; - folio =xas_next (&xas)) { +for (folio =xas_load (&xas); folio; folio =xas_next (&xas)) {if (xas_retry (&xas, folio))continue ;/*
@@ -2270,11 +2273,11 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,* No current caller is looking for DAX entries. */ if (xa_is_value (folio)) - goto update_start; +break ;/* If we landed in the middle of a THP, continue at its end. */ if (xa_is_sibling (folio)) - goto update_start; +break ;if (!folio_try_get (folio)) goto retry;
@@ -2282,12 +2285,15 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,if (unlikely (folio !=xas_reload (&xas))) goto put_folio; + next =folio_next_index (folio);if (!folio_batch_add (fbatch, folio)) { - nr =folio_nr_pages (folio); - *start = folio->index + nr; + *start = next; goto out; } -xas_advance (&xas,folio_next_index (folio) -1 ); + +if (next > end) +break ; +xas_advance (&xas, next -1 );continue ; put_folio:folio_put (folio);
@@ -2296,9 +2302,7 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,xas_reset (&xas); } -update_start: nr =folio_batch_count (fbatch); -if (nr) { folio = fbatch->folios[nr -1 ]; *start =folio_next_index (folio);
Signature · 1 line
2.43.0
PATCH 3/5 mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ
In reply to Chi ZhilingFrom: Chi Zhiling <[email protected]>
Change SGP_NOALLOC to return 0 with NULL folio on hole, matching SGP_READ behavior. This simplifies the sgp_type handling by unifying hole semantics across these types.
Previously, SGP_NOALLOC returned -ENOENT on hole, while SGP_READ returned 0. This inconsistency required special handling in callers like khugepaged and userfaultfd.
After this change:
- khugepaged: behavior unchanged (checks both error and NULL folio)
- userfaultfd: behavior unchanged (both -ENOENT and NULL are converted to -EFAULT before returning to userspace)
--- include/linux/shmem_fs.h | 2 +- mm/khugepaged.c | 2 +- mm/shmem.c | 9 +++------ 3 files changed, 5 insertions(+), 8 deletions(-)
include/linux/shmem_fs.h +1 -1
@@ -165,7 +165,7 @@ extern unsigned long shmem_partial_swap_usage(struct address_space *mapping,/* Flag allocation requirements to shmem_get_folio */ enum sgp_type {SGP_READ ,/* don't exceed i_size, don't allocate page */ -SGP_NOALLOC ,/* similar, but fail on hole or use fallocated page */ +SGP_NOALLOC ,/* like SGP_READ, but accept fallocated page */ SGP_CACHE ,/* don't exceed i_size, may allocate page */ SGP_WRITE ,/* may exceed i_size, may allocate !Uptodate page */ SGP_FALLOC ,/* like SGP_WRITE, but make existing page Uptodate */
mm/khugepaged.c +1 -1
@@ -1950,7 +1950,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,xas_unlock_irq (&xas);/* swap in or instantiate fallocated page */ if (shmem_get_folio (mapping->host, index,0 , - &folio,SGP_NOALLOC )) { + &folio,SGP_NOALLOC ) || !folio) { result =SCAN_FAIL ; goto xa_unlocked; }
mm/shmem.c +3 -6
@@ -2524,14 +2524,11 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, }/* -* SGP_READ: succeed on hole, with NULL folio, letting caller zero. -* SGP_NOALLOC: fail on hole, with NULL folio, letting caller fail. +* SGP_READ/SGP_NOALLOC: succeed on hole, with NULL folio. */ *foliop =NULL ; -if (sgp ==SGP_READ ) +if (sgp <=SGP_NOALLOC )return 0 ; -if (sgp ==SGP_NOALLOC ) -return -ENOENT ;/* * Fast cache lookup and swap lookup did not find it: allocate.
@@ -2657,7 +2654,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* * When no folio is found, the behavior depends on @sgp: * - for SGP_READ, *@foliop is %NULL and 0 is returned -* - for SGP_NOALLOC, *@foliop is %NULL and -ENOENT is returned +* - for SGP_NOALLOC, *@foliop is %NULL and 0 is returned * - for all other flags a new folio is allocated, inserted into the * page cache and returned locked in @foliop. *
Signature · 1 line
2.43.0
Baolin Wang
reviewerRe: [PATCH v1 3/5] mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ
Quoted 9 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Change
SGP_NOALLOCto return 0 with NULL folio on hole, matchingSGP_READbehavior. This simplifies thesgp_typehandling by unifying hole semantics across these types.Previously,
SGP_NOALLOCreturned-ENOENTon hole, whileSGP_READreturned 0. This inconsistency required special handling in callers like khugepaged and userfaultfd.
This patch doesn't seem to be a performance optimization, and I'm not convinced.
Quoted 2 lines
After this change:
- khugepaged: behavior unchanged (checks both error and NULL folio)
But this adds an extra check to khugepaged, and I'm not sure it's worth it.
Quoted 2 lines
- userfaultfd: behavior unchanged (both
-ENOENTand NULL are converted to-EFAULTbefore returning to userspace)
No, this will break userfaultfd, cause userfaultfd currently returns an error code directly in this case (please update to the latest codebase.). See:
static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)
{
struct folio *folio;
int err;
err = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC);
if (err)
return ERR_PTR(err);
return folio;
}Quoted 63 lines
Signed-off-by Chi Zhiling <[email protected]>--- include/linux/shmem_fs.h | 2 +- mm/khugepaged.c | 2 +- mm/shmem.c | 9 +++------ 3 files changed, 5 insertions(+), 8 deletions(-)include/linux/shmem_fs.h +1
-1@@ -165,7 +165,7 @@ extern unsigned long shmem_partial_swap_usage(struct address_space *mapping,/* Flag allocation requirements to shmem_get_folio */ enum sgp_type {SGP_READ ,/* don't exceed i_size, don't allocate page */ -SGP_NOALLOC ,/* similar, but fail on hole or use fallocated page */ +SGP_NOALLOC ,/* like SGP_READ, but accept fallocated page */ SGP_CACHE ,/* don't exceed i_size, may allocate page */ SGP_WRITE ,/* may exceed i_size, may allocate !Uptodate page */ SGP_FALLOC ,/* like SGP_WRITE, but make existing page Uptodate */ mm/khugepaged.c +1
-1@@ -1950,7 +1950,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,xas_unlock_irq (&xas);/* swap in or instantiate fallocated page */ if (shmem_get_folio (mapping->host, index,0 , - &folio,SGP_NOALLOC )) { + &folio,SGP_NOALLOC ) || !folio) { result =SCAN_FAIL ; goto xa_unlocked; }mm/shmem.c +3
-6@@ -2524,14 +2524,11 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, }/* -* SGP_READ: succeed on hole, with NULL folio, letting caller zero. -* SGP_NOALLOC: fail on hole, with NULL folio, letting caller fail. +* SGP_READ/SGP_NOALLOC: succeed on hole, with NULL folio. */ *foliop =NULL ; -if (sgp ==SGP_READ ) +if (sgp <=SGP_NOALLOC )return 0 ; -if (sgp ==SGP_NOALLOC ) -return -ENOENT ;/* * Fast cache lookup and swap lookup did not find it: allocate. @@ -2657,7 +2654,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,* * When no folio is found, the behavior depends on @sgp: * - for SGP_READ, *@foliop is %NULL and 0 is returned -* - for SGP_NOALLOC, *@foliop is %NULL and -ENOENT is returned +* - for SGP_NOALLOC, *@foliop is %NULL and 0 is returned * - for all other flags a new folio is allocated, inserted into the * page cache and returned locked in @foliop. *
Re: [PATCH v1 3/5] mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ
Quoted 7 lines · Baolin Wang
Quoted 6 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Change
SGP_NOALLOCto return 0 with NULL folio on hole, matchingSGP_READbehavior. This simplifies thesgp_typehandling by unifying hole semantics across these types.
Quoted 5 lines
Quoted 3 lines
Previously,
SGP_NOALLOCreturned-ENOENTon hole, whileSGP_READreturned 0. This inconsistency required special handling in callers like khugepaged and userfaultfd.This patch doesn't seem to be a performance optimization, and I'm not convinced.
Hi, baolin
This is not an optimization patch, and it is not related to this patch series either.
Quoted 5 lines
Quoted 3 lines
After this change:
- khugepaged: behavior unchanged (checks both error and NULL folio)But this adds an extra check to khugepaged, and I'm not sure it's worth it.
You are right, I will drop this patch in v2.
Quoted 11 lines
Quoted 3 lines
- userfaultfd: behavior unchanged (both
-ENOENTand NULL are converted to-EFAULTbefore returning to userspace)No, this will break userfaultfd, cause userfaultfd currently returns an error code directly in this case (please update to the latest codebase.). See:
static struct folio *shmem_get_folio_noalloc(struct inode *inode,
pgoff_tpgoff) {
struct folio *folio;
int err;
Quoted 6 lines
err =
shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC); if (err)
returnERR_PTR(err);return folio;
}
The return value of shmem_get_folio_noalloc() will change, but it seems this does not affect the logic in mfill_atomic_pte_continue().
``` folio = ops->get_folio_noalloc(inode, pgoff); /* Our caller expects us to return -EFAULT if we failed to find folio */ if (IS_ERR_OR_NULL(folio)) return -EFAULT; ```
No matter whether get_folio_noalloc() returns an error or NULL, the error code is converted to -EFAULT here, and mfill_atomic_pte_continue() is currently the only caller of get_folio_noalloc().
Did I miss something?
Thanks,
Quoted 67 lines
Quoted 67 lines
Signed-off-by Chi Zhiling <[email protected]>---
include/linux/shmem_fs.h| 2 +-
mm/khugepaged.c | 2 +-
mm/shmem.c | 9 +++------3 files changed, 5 insertions(+), 8 deletions(-)include/linux/shmem_fs.h +0
-0@@ -165,7 +165,7 @@ extern unsigned long shmem_partial_swap_usage(struct address_space *mapping, /* Flag allocation requirements to shmem_get_folio */ enum sgp_type {SGP_READ, /* don't exceed
i_size, don't allocate page */ - SGP_NOALLOC, /* similar, but fail on hole or use fallocated page */
+ SGP_NOALLOC, /* likeSGP_READ, but accept fallocated page */ SGP_CACHE, /* don't exceedi_size, may allocate page */ SGP_WRITE, /* may exceedi_size, may allocate !Uptodate page */ SGP_FALLOC, /* likeSGP_WRITE, but make existing page Uptodate */mm/khugepaged.c +1
-1@@ -1950,7 +1950,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, xas_unlock_irq(&xas); /* swap in or instantiate fallocated page */ if (shmem_get_folio(mapping->host, index, 0, - &folio, SGP_NOALLOC)) { + &folio, SGP_NOALLOC) || !folio) { result = SCAN_FAIL; goto xa_unlocked;}
mm/shmem.c +3
-6@@ -2524,14 +2524,11 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, } /* - * SGP_READ: succeed on hole, with NULL folio, letting caller zero. - * SGP_NOALLOC: fail on hole, with NULL folio, letting caller fail. + * SGP_READ/SGP_NOALLOC: succeed on hole, with NULL folio. */ *foliop = NULL; - if (sgp == SGP_READ) + if (sgp <= SGP_NOALLOC) return 0; - if (sgp == SGP_NOALLOC) - return -ENOENT; /** Fast cache lookup and swap lookup did not find it: allocate. @@ -2657,7 +2654,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, * * When no folio is found, the behavior depends on @sgp: * - for SGP_READ, *@foliop is %NULL and 0 is returned - * - for SGP_NOALLOC, *@foliop is %NULL and -ENOENT is returned + * - for SGP_NOALLOC, *@foliop is %NULL and 0 is returned * - for all other flags a new folio is allocated, inserted into the * page cache and returned locked in @foliop.*
Baolin Wang
reviewerRe: [PATCH v1 3/5] mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ
Quoted 54 lines · Chi Zhiling
Quoted 7 lines · Baolin Wang
Quoted 6 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Change
SGP_NOALLOCto return 0 with NULL folio on hole, matchingSGP_READbehavior. This simplifies thesgp_typehandling by unifying hole semantics across these types.Quoted 5 lines
Quoted 3 lines
Previously,
SGP_NOALLOCreturned-ENOENTon hole, whileSGP_READreturned 0. This inconsistency required special handling in callers like khugepaged and userfaultfd.This patch doesn't seem to be a performance optimization, and I'm not convinced.
Hi, baolin
This is not an optimization patch, and it is not related to this patch series either.
Quoted 5 lines
Quoted 3 lines
After this change:
- khugepaged: behavior unchanged (checks both error and NULL folio)But this adds an extra check to khugepaged, and I'm not sure it's worth it.
You are right, I will drop this patch in v2.
Quoted 11 lines
Quoted 3 lines
- userfaultfd: behavior unchanged (both
-ENOENTand NULL are converted to-EFAULTbefore returning to userspace)No, this will break userfaultfd, cause userfaultfd currently returns an error code directly in this case (please update to the latest codebase.). See:
static struct folio *shmem_get_folio_noalloc(struct inode *inode,
pgoff_tpgoff) {
struct folio *folio;
int err;Quoted 6 lines
err =
shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC); if (err)
returnERR_PTR(err);return folio;
}The return value of
shmem_get_folio_noalloc()will change, but it seems this does not affect the logic inmfill_atomic_pte_continue().``` folio = ops->get_folio_noalloc(inode, pgoff); /* Our caller expects us to return -EFAULT if we failed to find folio */ if (IS_ERR_OR_NULL(folio)) return -EFAULT; ```
Quoted 3 lines
No matter whether
get_folio_noalloc()returns an error or NULL, the error code is converted to-EFAULThere, andmfill_atomic_pte_continue()is currently the only caller ofget_folio_noalloc().Did I miss something?
OK.
But I still wonder if it's worth changing the definition of a flag, especially when there's no obvious bug here and this is merely a cleanup. Yet this cleanup requires adding an extra check for khugepaged, which makes me feel it is not worth it. Unless Hugh prefers to change the meaning of this flag:).
PATCH 4/5 mm/shmem: introduce copy_zero_to_iter() for large zeroing
In reply to Chi ZhilingFrom: Chi Zhiling <[email protected]>
This is a prep patch for shmem folio batching in the read path, where non-uptodate folios need to be handled in the main iteration loop. A large non-uptodate folio should be treated as a hole.
Currently, holes larger than PAGE_SIZE cannot be handled because ZERO_PAGE is limited to a single page. Add copy_zero_to_iter() as a wrapper to support copying larger zero ranges to the iterator.
--- mm/shmem.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
mm/shmem.c +15 -1
@@ -3336,6 +3336,20 @@ shmem_write_end(const struct kiocb *iocb, struct address_space *mapping,return copied; } +static size_t copy_zero_to_iter (size_t bytes,struct iov_iter *i) +{ +struct folio *zero =largest_zero_folio (); +unsigned long nr, ret, written =0 ; + +do { + nr =min (bytes - written,folio_size (zero)); + ret =copy_folio_to_iter (zero,0 , nr, i); + written += ret; + }while (written < bytes && ret == nr); + +return written; +} +static ssize_t shmem_file_read_iter (struct kiocb *iocb,struct iov_iter *to) {struct file *file = iocb->ki_filp;
@@ -3430,7 +3444,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)* clear_user() not so much, that it is noticeably * faster to copy the zero page instead of clearing. */ - ret =copy_page_to_iter (ZERO_PAGE (0 ), offset, nr, to); + ret =copy_zero_to_iter (nr, to); }else {/* * But submitting the same page twice in a row to
Signature · 1 line
2.43.0
PATCH 5/5 mm/shmem: optimize file read with folio batching
In reply to Chi ZhilingFrom: Chi Zhiling <[email protected]>
Optimize shmem file read by using filemap_get_folios_contig() to batch fetch contiguous folios from the page cache, reducing the overhead of repeated shmem_get_folio() calls.
When the folio batch is exhausted, attempt to refill it with filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back to shmem_get_folio() to handle these cases individually.
Additionally:
- Defer folio_put() until the batch is exhausted or on exit
- Add folio_test_uptodate() check before copying to ensure data validity
--- mm/shmem.c | 50 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 36 insertions(+), 14 deletions(-)
mm/shmem.c +36 -14
@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)struct file *file = iocb->ki_filp;struct inode *inode =file_inode (file);struct address_space *mapping = inode->i_mapping; +struct folio_batch fbatch;pgoff_t index;unsigned long offset;int error =0 ;ssize_t retval =0 ; +folio_batch_init (&fbatch); +for (;;) {struct folio *folio =NULL ;struct page *page =NULL ;
@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)break ; index = iocb->ki_pos >>PAGE_SHIFT ; - error =shmem_get_folio (inode, index,0 , &folio,SGP_READ ); -if (error) { -if (error == -EINVAL ) - error =0 ; -break ; +fetch: + folio =folio_batch_next (&fbatch); +if (!folio) { +pgoff_t start = index; +pgoff_t end = (iocb->ki_pos + to->count -1 ) >>PAGE_SHIFT ; + +if (folio_batch_count (&fbatch)) { +for (int i =0 ; i <folio_batch_count (&fbatch); i++) +folio_put (fbatch.folios[i]); +folio_batch_reinit (&fbatch); + } + +filemap_get_folios_contig (inode->i_mapping, &start, end, &fbatch); +if (folio_batch_count (&fbatch)) + goto fetch; + + error =shmem_get_folio (inode, index,0 , &folio,SGP_READ ); +if (unlikely (error)) { +if (error == -EINVAL ) + error =0 ; +break ; + } +if (folio) { +folio_unlock (folio); +folio_batch_add (&fbatch, folio); + fbatch.i++; + } } -if (folio) { -folio_unlock (folio); +if (folio) { page =folio_file_page (folio, index);if (PageHWPoison (page)) { -folio_put (folio); error = -EIO ;break ; }
@@ -3398,11 +3421,9 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)* are called without i_rwsem protection against truncate */ i_size =i_size_read (inode); -if (unlikely (iocb->ki_pos >= i_size)) { -if (folio) -folio_put (folio); +if (unlikely (iocb->ki_pos >= i_size))break ; - } + end_offset =min_t (loff_t, i_size, iocb->ki_pos + to->count);if (folio &&likely (!fallback_page_copy)) fsize =folio_size (folio);
@@ -3411,7 +3432,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) offset = iocb->ki_pos & (fsize -1 ); nr =min_t (loff_t, end_offset - iocb->ki_pos, fsize - offset); -if (folio) { +if (folio &&folio_test_uptodate (folio)) {/* * If users can be writing to this page using arbitrary * virtual addresses, take care about potential aliasing
@@ -3437,7 +3458,6 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) ret =copy_folio_to_iter (folio, offset, nr, to); else ret =copy_page_to_iter (page, offset, nr, to); -folio_put (folio); }else if (user_backed_iter (to)) {/* * Copy to user tends to be so well optimized, but
@@ -3466,6 +3486,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)cond_resched (); } +for (int i =0 ; i <folio_batch_count (&fbatch); i++) +folio_put (fbatch.folios[i]);file_accessed (file);return retval ? retval : error; }
Signature · 1 line
2.43.0
Baolin Wang
reviewerRe: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching
Quoted 77 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Optimize shmem file read by using
filemap_get_folios_contig()to batch fetch contiguous folios from the page cache, reducing the overhead of repeatedshmem_get_folio()calls.When the folio batch is exhausted, attempt to refill it with
filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back toshmem_get_folio()to handle these cases individually.Additionally:
- Deferfolio_put()until the batch is exhausted or on exit
- Addfolio_test_uptodate()check before copying to ensure data validitySigned-off-by Chi Zhiling <[email protected]>--- mm/shmem.c | 50 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 36 insertions(+), 14 deletions(-)mm/shmem.c +30
-5@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)struct file *file = iocb->ki_filp;struct inode *inode =file_inode (file);struct address_space *mapping = inode->i_mapping; +struct folio_batch fbatch;pgoff_t index;unsigned long offset;int error =0 ;ssize_t retval =0 ; +folio_batch_init (&fbatch); +for (;;) {struct folio *folio =NULL ;struct page *page =NULL ;@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)break ; index = iocb->ki_pos >>PAGE_SHIFT ; - error =shmem_get_folio (inode, index,0 , &folio,SGP_READ ); -if (error) { -if (error == -EINVAL ) - error =0 ; -break ; +fetch: + folio =folio_batch_next (&fbatch); +if (!folio) { +pgoff_t start = index; +pgoff_t end = (iocb->ki_pos + to->count -1 ) >>PAGE_SHIFT ; + +if (folio_batch_count (&fbatch)) { +for (int i =0 ; i <folio_batch_count (&fbatch); i++) +folio_put (fbatch.folios[i]); +folio_batch_reinit (&fbatch); + } + +filemap_get_folios_contig (inode->i_mapping, &start, end, &fbatch); +if (folio_batch_count (&fbatch)) + goto fetch; + + error =shmem_get_folio (inode, index,0 , &folio,SGP_READ ); +if (unlikely (error)) { +if (error == -EINVAL ) + error =0 ; +break ; + } +if (folio) { +folio_unlock (folio); +folio_batch_add (&fbatch, folio); + fbatch.i++; + } }
mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3412
Quoted 10 lines
mm/shmem.c +1
-3-if (folio) { -folio_unlock (folio); +if (folio) { page =folio_file_page (folio, index);if (PageHWPoison (page)) { -folio_put (folio); error = -EIO ;break ; }
I haven't tested it yet, but I'm sure this will break the fallback_page_copy mode.
You're fetching from the batch at folio granularity here, but in fallback_page_copy mode, we still copy the data at page granularity.
Re: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching
Quoted 93 lines · Baolin Wang
Quoted 77 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Optimize shmem file read by using
filemap_get_folios_contig()to batch fetch contiguous folios from the page cache, reducing the overhead of repeatedshmem_get_folio()calls.When the folio batch is exhausted, attempt to refill it with
filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back toshmem_get_folio()to handle these cases individually.Additionally:
- Deferfolio_put()until the batch is exhausted or on exit
- Addfolio_test_uptodate()check before copying to ensure data validitySigned-off-by Chi Zhiling <[email protected]>---
mm/shmem.c| 50 ++++++++++++++++++++++++++++++++++++--------------1 file changed, 36 insertions(+), 14 deletions(-)mm/shmem.c +30
-5@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) struct file *file = iocb->ki_filp; struct inode *inode = file_inode(file); struct address_space *mapping = inode->i_mapping; + struct folio_batch fbatch; pgoff_t index; unsigned long offset; int error = 0; ssize_t retval = 0; + folio_batch_init(&fbatch); + for (;;) { struct folio *folio = NULL; struct page *page = NULL;@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) break; index = iocb->ki_pos >> PAGE_SHIFT; - error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); - if (error) { - if (error == -EINVAL) - error = 0; - break; +fetch: + folio = folio_batch_next(&fbatch); + if (!folio) { + pgoff_t start = index; + pgoff_t end = (iocb->ki_pos + to->count - 1) >> PAGE_SHIFT; + + if (folio_batch_count(&fbatch)) { + for (int i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]); + folio_batch_reinit(&fbatch); + } + + filemap_get_folios_contig(inode->i_mapping, &start, end, &fbatch); + if (folio_batch_count(&fbatch)) + goto fetch; + + error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); + if (unlikely(error)) { + if (error == -EINVAL) + error = 0; + break; + } + if (folio) { + folio_unlock(folio); + folio_batch_add(&fbatch, folio); + fbatch.i++; + }mm/shmem.c static ssize_t shmem_file_read_iter(struct · line 3412
Quoted 10 lines
mm/shmem.c +1
-3} - if (folio) { - folio_unlock(folio); + if (folio) { page = folio_file_page(folio, index); if (PageHWPoison(page)) { - folio_put(folio); error = -EIO; break; }I haven't tested it yet, but I'm sure this will break the
fallback_page_copymode.
Quoted 1 line
You're fetching from the batch at folio granularity here, but in
fallback_page_copymode, we still copy the data at page granularity.
Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.
Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?
~~~
Sashika:
Can this lockless check cause readers to receive spurious zeros instead of the actual data?
Since filemap_get_folios_contig() fetches locklessly, a folio that is currently being allocated and initialized, or swapped in by another thread, might not yet be marked uptodate.
By checking folio_test_uptodate() without holding the folio lock, this falls through to the zero-fill paths if the folio isn't uptodate yet. The original code avoided this by using shmem_get_folio() which waits for the folio lock and I/O completion.
~~~
Thanks,
Baolin Wang
reviewerRe: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching
Quoted 94 lines · Chi Zhiling
Quoted 93 lines · Baolin Wang
Quoted 77 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Optimize shmem file read by using
filemap_get_folios_contig()to batch fetch contiguous folios from the page cache, reducing the overhead of repeatedshmem_get_folio()calls.When the folio batch is exhausted, attempt to refill it with
filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back toshmem_get_folio()to handle these cases individually.Additionally:
- Deferfolio_put()until the batch is exhausted or on exit
- Addfolio_test_uptodate()check before copying to ensure data validitySigned-off-by Chi Zhiling <[email protected]>---
mm/shmem.c| 50 ++++++++++++++++++++++++++++++++++++--------------1 file changed, 36 insertions(+), 14 deletions(-)mm/shmem.c +30
-5@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) struct file *file = iocb->ki_filp; struct inode *inode = file_inode(file); struct address_space *mapping = inode->i_mapping; + struct folio_batch fbatch; pgoff_t index; unsigned long offset; int error = 0; ssize_t retval = 0; + folio_batch_init(&fbatch); + for (;;) { struct folio *folio = NULL; struct page *page = NULL;@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) break; index = iocb->ki_pos >> PAGE_SHIFT; - error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); - if (error) { - if (error == -EINVAL) - error = 0; - break; +fetch: + folio = folio_batch_next(&fbatch); + if (!folio) { + pgoff_t start = index; + pgoff_t end = (iocb->ki_pos + to->count - 1) >> PAGE_SHIFT; + + if (folio_batch_count(&fbatch)) { + for (int i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]); + folio_batch_reinit(&fbatch); + } + + filemap_get_folios_contig(inode->i_mapping, &start, end, &fbatch); + if (folio_batch_count(&fbatch)) + goto fetch; + + error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); + if (unlikely(error)) { + if (error == -EINVAL) + error = 0; + break; + } + if (folio) { + folio_unlock(folio); + folio_batch_add(&fbatch, folio); + fbatch.i++; + }mm/shmem.c static ssize_t shmem_file_read_iter(struct · line 3412
Quoted 10 lines
mm/shmem.c +1
-3} - if (folio) { - folio_unlock(folio); + if (folio) { page = folio_file_page(folio, index); if (PageHWPoison(page)) { - folio_put(folio); error = -EIO; break; }I haven't tested it yet, but I'm sure this will break the
fallback_page_copymode.
Quoted 3 lines
Quoted 1 line
You're fetching from the batch at folio granularity here, but in
fallback_page_copymode, we still copy the data at page granularity.Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.
Good.
Quoted 1 line
Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?
I'm not concerned about swap-in folios, since we already ensure the folio is uptodate when swapping out.
What I'm worried about is that this change moves the folio_test_uptodate() check outside of the folio lock, which could introduce some race? Of course, with the original logic, the folio state could also transition from !uptodate to uptodate after the folio is unlocked (I need more thought).
Another race condition that needs to be addressed is, when getting the folio locklessly via filemap_get_folios_contig(), there could be a concurrent race with swap-out (because you removed the folio->mapping check under folio lock). This could result in reading zero data when it shouldn't (the folio actually needs to be swapped in).
Re: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching
Quoted 105 lines · Baolin Wang
Quoted 96 lines · Chi Zhiling
Quoted 95 lines · Baolin Wang
Quoted 87 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Optimize shmem file read by using
filemap_get_folios_contig()to batch fetch contiguous folios from the page cache, reducing the overhead of repeatedshmem_get_folio()calls.When the folio batch is exhausted, attempt to refill it with
filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back toshmem_get_folio()to handle these cases individually.Additionally:
- Deferfolio_put()until the batch is exhausted or on exit
- Addfolio_test_uptodate()check before copying to ensure data validitySigned-off-by Chi Zhiling <[email protected]>---
mm/shmem.c| 50 ++++++++++++++++++++++++++++++++++++--------------1 file changed, 36 insertions(+), 14 deletions(-)mm/shmem.c +31
-8@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) struct file *file = iocb->ki_filp; struct inode *inode = file_inode(file); struct address_space *mapping = inode->i_mapping; + struct folio_batch fbatch; pgoff_t index; unsigned long offset; int error = 0; ssize_t retval = 0; + folio_batch_init(&fbatch); + for (;;) { struct folio *folio = NULL; struct page *page = NULL;@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) break; index = iocb->ki_pos >> PAGE_SHIFT; - error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); - if (error) { - if (error == -EINVAL) - error = 0; - break; +fetch: + folio = folio_batch_next(&fbatch); + if (!folio) { + pgoff_t start = index; + pgoff_t end = (iocb->ki_pos + to->count - 1) >>PAGE_SHIFT ; + + if (folio_batch_count(&fbatch)) { + for (int i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]); + folio_batch_reinit(&fbatch); + } + + filemap_get_folios_contig(inode->i_mapping, &start, end, &fbatch); + if (folio_batch_count(&fbatch)) + goto fetch; + + error = shmem_get_folio(inode, index, 0, &folio,SGP_READ ); + if (unlikely(error)) { + if (error == -EINVAL) + error = 0; + break; + } + if (folio) { + folio_unlock(folio); + folio_batch_add(&fbatch, folio); + fbatch.i++; + } } - if (folio) { - folio_unlock(folio); + if (folio) { page = folio_file_page(folio, index); if (PageHWPoison(page)) { - folio_put(folio); error = -EIO;Quoted 2 lines
break;
}I haven't tested it yet, but I'm sure this will break the
fallback_page_copymode.Quoted 3 lines
Quoted 1 line
You're fetching from the batch at folio granularity here, but in
fallback_page_copymode, we still copy the data at page granularity.Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.
Good.
Quoted 8 lines
Quoted 1 line
Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?
I'm not concerned about swap-in folios, since we already ensure the folio is uptodate when swapping out.
What I'm worried about is that this change moves the
folio_test_uptodate()check outside of the folio lock, which could introduce some race? Of course, with the original logic, the folio state could also transition from !uptodate to uptodate after the folio is unlocked (I need more thought).Another race condition that needs to be addressed is, when getting the folio locklessly via
filemap_get_folios_contig(), there could be a concurrent race with swap-out (because you removed thefolio->mappingcheck under folio lock). This could result in reading zero data when it shouldn't (the folio actually needs to be swapped in).
Sorry, I don't fully understand what you mean. It seems that swap-out does not clear the uptodate flag, and the folio cannot be freed during the read operation because its reference count is held. So a swapped-out folio should not be mistaken for a zero-filled folio. Right?
Baolin Wang
reviewerRe: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching
Quoted 108 lines · Chi Zhiling
Quoted 107 lines · Baolin Wang
Quoted 96 lines · Chi Zhiling
Quoted 95 lines · Baolin Wang
Quoted 87 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Optimize shmem file read by using
filemap_get_folios_contig()to batch fetch contiguous folios from the page cache, reducing the overhead of repeatedshmem_get_folio()calls.When the folio batch is exhausted, attempt to refill it with
filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back toshmem_get_folio()to handle these cases individually.Additionally:
- Deferfolio_put()until the batch is exhausted or on exit
- Addfolio_test_uptodate()check before copying to ensure data validitySigned-off-by Chi Zhiling <[email protected]>---
mm/shmem.c| 50 ++++++++++++++++++++++++++++++++++++--------------1 file changed, 36 insertions(+), 14 deletions(-)mm/shmem.c +31
-8@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) struct file *file = iocb->ki_filp; struct inode *inode = file_inode(file); struct address_space *mapping = inode->i_mapping; + struct folio_batch fbatch; pgoff_t index; unsigned long offset; int error = 0; ssize_t retval = 0; + folio_batch_init(&fbatch); + for (;;) { struct folio *folio = NULL; struct page *page = NULL;@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) break; index = iocb->ki_pos >> PAGE_SHIFT; - error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); - if (error) { - if (error == -EINVAL) - error = 0; - break; +fetch: + folio = folio_batch_next(&fbatch); + if (!folio) { + pgoff_t start = index; + pgoff_t end = (iocb->ki_pos + to->count - 1) >>PAGE_SHIFT ; + + if (folio_batch_count(&fbatch)) { + for (int i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]); + folio_batch_reinit(&fbatch); + } + + filemap_get_folios_contig(inode->i_mapping, &start, end, &fbatch); + if (folio_batch_count(&fbatch)) + goto fetch; + + error = shmem_get_folio(inode, index, 0, &folio,SGP_READ ); + if (unlikely(error)) { + if (error == -EINVAL) + error = 0; + break; + } + if (folio) { + folio_unlock(folio); + folio_batch_add(&fbatch, folio); + fbatch.i++; + } } - if (folio) { - folio_unlock(folio); + if (folio) { page = folio_file_page(folio, index); if (PageHWPoison(page)) { - folio_put(folio); error = -EIO;Quoted 2 lines
break;
}I haven't tested it yet, but I'm sure this will break the
fallback_page_copymode.Quoted 3 lines
Quoted 1 line
You're fetching from the batch at folio granularity here, but in
fallback_page_copymode, we still copy the data at page granularity.Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.
Good.
Quoted 1 line
Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?
Quoted 8 lines
Quoted 6 lines
I'm not concerned about swap-in folios, since we already ensure the folio is uptodate when swapping out.
What I'm worried about is that this change moves the
folio_test_uptodate()check outside of the folio lock, which could introduce some race? Of course, with the original logic, the folio state could also transition from !uptodate to uptodate after the folio is unlocked (I need more thought).Another race condition that needs to be addressed is, when getting the folio locklessly via
filemap_get_folios_contig(), there could be a concurrent race with swap-out (because you removed thefolio->mappingcheck under folio lock). This could result in reading zero data when it shouldn't (the folio actually needs to be swapped in).Sorry, I don't fully understand what you mean. It seems that swap-out does not clear the uptodate flag, and the folio cannot be freed during the read operation because its reference count is held. So a swapped-out folio should not be mistaken for a zero-filled folio. Right?
Sorry for the late reply. Ah, I missed the held refcnt, and now sounds reasonable.
Re: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching
Quoted 116 lines · Baolin Wang
Quoted 113 lines · Chi Zhiling
Quoted 112 lines · Baolin Wang
Quoted 96 lines · Chi Zhiling
Quoted 95 lines · Baolin Wang
Quoted 87 lines · Chi Zhiling
From: Chi Zhiling <[email protected]>
Optimize shmem file read by using
filemap_get_folios_contig()to batch fetch contiguous folios from the page cache, reducing the overhead of repeatedshmem_get_folio()calls.When the folio batch is exhausted, attempt to refill it with
filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back toshmem_get_folio()to handle these cases individually.Additionally:
- Deferfolio_put()until the batch is exhausted or on exit
- Addfolio_test_uptodate()check before copying to ensure data validitySigned-off-by Chi Zhiling <[email protected]>---
mm/shmem.c| 50 ++++++++++++++++++++++++++++++++++++--------------1 file changed, 36 insertions(+), 14 deletions(-)mm/shmem.c +31
-8@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) struct file *file = iocb->ki_filp; struct inode *inode = file_inode(file); struct address_space *mapping = inode->i_mapping; + struct folio_batch fbatch; pgoff_t index; unsigned long offset; int error = 0; ssize_t retval = 0; + folio_batch_init(&fbatch); + for (;;) { struct folio *folio = NULL; struct page *page = NULL;@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) break; index = iocb->ki_pos >> PAGE_SHIFT; - error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); - if (error) { - if (error == -EINVAL) - error = 0; - break; +fetch: + folio = folio_batch_next(&fbatch); + if (!folio) { + pgoff_t start = index; + pgoff_t end = (iocb->ki_pos + to->count - 1) >>PAGE_SHIFT ; + + if (folio_batch_count(&fbatch)) { + for (int i = 0; i < folio_batch_count(&fbatch); i++) + folio_put(fbatch.folios[i]); + folio_batch_reinit(&fbatch); + } + + filemap_get_folios_contig(inode->i_mapping, &start, end, &fbatch); + if (folio_batch_count(&fbatch)) + goto fetch; + + error = shmem_get_folio(inode, index, 0, &folio,SGP_READ ); + if (unlikely(error)) { + if (error == -EINVAL) + error = 0; + break; + } + if (folio) { + folio_unlock(folio); + folio_batch_add(&fbatch, folio); + fbatch.i++; + } } - if (folio) { - folio_unlock(folio); + if (folio) { page = folio_file_page(folio, index); if (PageHWPoison(page)) { - folio_put(folio); error = -EIO;Quoted 2 lines
break;
}I haven't tested it yet, but I'm sure this will break the
fallback_page_copymode.Quoted 3 lines
Quoted 1 line
You're fetching from the batch at folio granularity here, but in
fallback_page_copymode, we still copy the data at page granularity.Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.
Good.
Quoted 1 line
Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?
I'm not concerned about swap-in folios, since we already ensure the folio is uptodate when swapping out.
What I'm worried about is that this change moves the
folio_test_uptodate()check outside of the folio lock, which could introduce some race? Of course, with the original logic, the folio state could also transition from !uptodate to uptodate after the folio is unlocked (I need more thought).
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Another race condition that needs to be addressed is, when getting the folio locklessly via
filemap_get_folios_contig(), there could be a concurrent race with swap-out (because you removed the folio-Quoted 1 line
mapping check under folio lock). This could result in reading zero
data when it shouldn't (the folio actually needs to be swapped in).
Sorry, I don't fully understand what you mean. It seems that swap-out does not clear the uptodate flag, and the folio cannot be freed during the read operation because its reference count is held. So a swapped- out folio should not be mistaken for a zero-filled folio. Right?
Sorry for the late reply. Ah, I missed the held refcnt, and now sounds reasonable.
I've just sent out v2. Feel free to comment. :)
Andrew Morton
maintainerQuoted 5 lines · On Wed, 20 May 2026 18:15:33 +0800 Chi Zhiling <[email protected]> wrote:
From: Chi Zhiling <[email protected]>
This series improves shmem read performance by implementing folio batching in the read path and reducing unnecessary xarray lookups.
Thanks.
Quoted 32 lines
Performance Results
===================Testing was performed with fio sequential read workloads:
fio --ioengine=sync --rw=read --size=1G --runtime=180### THP Disabled - Normal Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 11.4GiB/s | 12.7GiB/s | +11.4% | | 64k | 11.2GiB/s | 12.2GiB/s | +8.9% | | 4k | 3809MiB/s | 3838MiB/s | +0.8% |### THP Disabled - Fallocated Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 23.7GiB/s | 28.7GiB/s | +21.1% | | 64k | 22.6GiB/s | 27.0GiB/s | +19.5% | | 4k | 4668MiB/s | 4678MiB/s | +0.2% |### THP Enabled - Normal Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 13.9GiB/s | 13.9GiB/s | 0% | | 64k | 13.4GiB/s | 13.4GiB/s | 0% | | 4k | 3818MiB/s | 3836MiB/s | +0.5% |
Quoted 7 lines
### THP Enabled - Fallocated Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 24.1GiB/s | 34.9GiB/s | +44.8% | | 64k | 22.9GiB/s | 31.3GiB/s | +36.7% | | 4k | 4721MiB/s | 4708MiB/s | -0.3% |
That looks nice.
AI review might have found a few things:
https://sashiko.dev/#/patchset/[email protected]
I'll skip the patchset for now (unreviewed v1!).
Quoted 50 lines · Andrew Morton
Quoted 5 lines · On Wed, 20 May 2026 18:15:33 +0800 Chi Zhiling <[email protected]> wrote:
From: Chi Zhiling <[email protected]>
This series improves shmem read performance by implementing folio batching in the read path and reducing unnecessary xarray lookups.
Thanks.
Quoted 32 lines
Performance Results
===================Testing was performed with fio sequential read workloads:
fio --ioengine=sync --rw=read --size=1G --runtime=180### THP Disabled - Normal Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 11.4GiB/s | 12.7GiB/s | +11.4% | | 64k | 11.2GiB/s | 12.2GiB/s | +8.9% | | 4k | 3809MiB/s | 3838MiB/s | +0.8% |### THP Disabled - Fallocated Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 23.7GiB/s | 28.7GiB/s | +21.1% | | 64k | 22.6GiB/s | 27.0GiB/s | +19.5% | | 4k | 4668MiB/s | 4678MiB/s | +0.2% |### THP Enabled - Normal Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 13.9GiB/s | 13.9GiB/s | 0% | | 64k | 13.4GiB/s | 13.4GiB/s | 0% | | 4k | 3818MiB/s | 3836MiB/s | +0.5% |Quoted 7 lines
### THP Enabled - Fallocated Files ###
| Block Size | Baseline | v1 | Improvement | | ---------- | --------- | --------- | ----------- | | 1M | 24.1GiB/s | 34.9GiB/s | +44.8% | | 64k | 22.9GiB/s | 31.3GiB/s | +36.7% | | 4k | 4721MiB/s | 4708MiB/s | -0.3% |
Quoted 6 lines
That looks nice.
AI review might have found a few things:
https://sashiko.dev/#/patchset/[email protected]I'll skip the patchset for now (unreviewed v1!).
Okay, I will fix those issues in v2.
Thanks!