Skip to contentSkip to conversation

mm/shmem: optimize read with reduced xarray lookups and folio batching

PATCH Superseded Chi Zhiling 3 participants 17 messages May 20 – Jun 1

Subsystems (5)
Memory management VFS Page cache THP +1 more subsystems

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover mm/shmem: optimize read with reduced xarray lookups and folio batching

From: Chi Zhiling <[email protected]>

This series improves shmem read performance by implementing folio batching in the read path and reducing unnecessary xarray lookups.

Changes since RFC
=================

The RFC version used xas_for_each() in shmem_get_read_batch(), which introduced about a 1% regression for 4K read workloads.

This v1 addresses the regression in patch 2 by switching to filemap_get_folios_contig() and optimizing it to avoid the extra xarray traversal overhead.

Performance Results
===================

Testing was performed with fio sequential read workloads:

  fio --ioengine=sync --rw=read --size=1G --runtime=180

### THP Disabled - Normal Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 11.4GiB/s | 12.7GiB/s | +11.4%      |
| 64k        | 11.2GiB/s | 12.2GiB/s | +8.9%       |
| 4k         | 3809MiB/s | 3838MiB/s | +0.8%       |

### THP Disabled - Fallocated Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 23.7GiB/s | 28.7GiB/s | +21.1%      |
| 64k        | 22.6GiB/s | 27.0GiB/s | +19.5%      |
| 4k         | 4668MiB/s | 4678MiB/s | +0.2%       |

### THP Enabled - Normal Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 13.9GiB/s | 13.9GiB/s | 0%          |
| 64k        | 13.4GiB/s | 13.4GiB/s | 0%          |
| 4k         | 3818MiB/s | 3836MiB/s | +0.5%       |

### THP Enabled - Fallocated Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 24.1GiB/s | 34.9GiB/s | +44.8%      |
| 64k        | 22.9GiB/s | 31.3GiB/s | +36.7%      |
| 4k         | 4721MiB/s | 4708MiB/s | -0.3%       |

rfc:
https://lore.kernel.org/linux-fsdevel/[email protected]/

Chi Zhiling (5):
mm/filemap: reduce unnecessary xarray lookups when read cached pages mm/filemap: reduce xarray lookups in filemap_get_folios_contig() mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ mm/shmem: introduce copy_zero_to_iter() for large zeroing mm/shmem: optimize file read with folio batching

 include/linux/shmem_fs.h |  2 +-
 mm/filemap.c             | 34 ++++++++++++------
 mm/khugepaged.c          |  2 +-
 mm/shmem.c               | 75 +++++++++++++++++++++++++++++-----------
 4 files changed, 79 insertions(+), 34 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/5 mm/filemap: reduce unnecessary xarray lookups when read cached pages

From: Chi Zhiling <[email protected]>

When reading small amounts of data from the page cache, only a single folio is typically returned from filemap_read_get_batch(). In this case, calling xas_advance() or xas_next() after adding the folio to the batch is unnecessary and only introduces extra branches.

The same issue exists for large reads, where one additional xarray walk is always performed before termination.

Move the boundary check to after the folio is added to the batch so the final redundant xarray advancement can be avoided. This significantly reduces the branch count in the read path.

xas_next() does not update xa_index when xas->xa_node is set to XAS_RESTART, so checking the boundary before updating xa_index is sufficient to keep the folio within range. The warning should therefore never trigger.

The branch count:
654.198 M/sec -> 646.444 M/sec

Performance counter stats for 'fio --ioengine=sync --rw=read --bs=4k --size=1G --runtime=300 --time_based --group_reporting --name=seq_read_test --filename=file':

before:
READ: bw=2697MiB/s (2828MB/s), 2697MiB/s-2697MiB/s (2828MB/s-2828MB/s), io=790GiB (848GB), run=300001-300001msec

      245602051556      task-clock                       #    0.821 CPUs utilized
             78467      context-switches                 #  319.488 /sec
                40      cpu-migrations                   #    0.163 /sec
              3388      page-faults                      #   13.795 /sec
      758312319204      instructions                     #    0.74  insn per cycle
     1025881497502      cycles                           #    4.177 GHz
      160672383734      branches                         #  654.198 M/sec
         361904512      branch-misses                    #    0.23% of all branches

after:
READ: bw=2709MiB/s (2841MB/s), 2709MiB/s-2709MiB/s (2841MB/s-2841MB/s), io=794GiB (852GB), run=300000-300000msec

      243985503670      task-clock                       #    0.812 CPUs utilized
             79004      context-switches                 #  323.806 /sec
                30      cpu-migrations                   #    0.123 /sec
              3355      page-faults                      #   13.751 /sec
      747830935069      instructions                     #    0.73  insn per cycle
     1019609333322      cycles                           #    4.179 GHz
      157722976668      branches                         #  646.444 M/sec
         348984893      branch-misses                    #    0.22% of all branches
---
 mm/filemap.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
mm/filemap.c +10 -2
index 4e636647100c..d54450e529bd 100644
@@ -2458,12 +2458,16 @@ static void filemap_get_read_batch(struct address_space *mapping,
 {
 	XA_STATE(xas, &mapping->i_pages, index);
 	struct folio *folio;
+	pgoff_t next;
+
+	if (unlikely(index > max))
+		return;
 
 	rcu_read_lock();
 	for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) {
 		if (xas_retry(&xas, folio))
 			continue;
-		if (xas.xa_index > max || xa_is_value(folio))
+		if (xa_is_value(folio) || WARN_ON(xas.xa_index > max))
 			break;
 		if (xa_is_sibling(folio))
 			break;
@@ -2479,7 +2483,11 @@ static void filemap_get_read_batch(struct address_space *mapping,
 			break;
 		if (folio_test_readahead(folio))
 			break;
-		xas_advance(&xas, folio_next_index(folio) - 1);
+
+		next = folio_next_index(folio);
+		if (next > max)
+			break;
+		xas_advance(&xas, next - 1);
 		continue;
 put_folio:
 		folio_put(folio);
Signature · 1 line
2.43.0
#2 lore raw

PATCH 2/5 mm/filemap: reduce xarray lookups in filemap_get_folios_contig()

In reply to Chi Zhiling

From: Chi Zhiling <[email protected]>

Apply the same optimization used in filemap_get_read_batch() by moving the boundary check from the loop condition to before xas_advance(), avoiding an unnecessary xarray lookup and reducing branches in the fast path.

---
 mm/filemap.c | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)
mm/filemap.c +13 -9
index d54450e529bd..5a5268a7c802 100644
@@ -2258,11 +2258,14 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,
 	XA_STATE(xas, &mapping->i_pages, *start);
 	unsigned long nr;
 	struct folio *folio;
+	pgoff_t next;
+
+	if (unlikely(*start > end))
+		return 0;
 
 	rcu_read_lock();
 
-	for (folio = xas_load(&xas); folio && xas.xa_index <= end;
-			folio = xas_next(&xas)) {
+	for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) {
 		if (xas_retry(&xas, folio))
 			continue;
 		/*
@@ -2270,11 +2273,11 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,
 		 * No current caller is looking for DAX entries.
 		 */
 		if (xa_is_value(folio))
-			goto update_start;
+			break;
 
 		/* If we landed in the middle of a THP, continue at its end. */
 		if (xa_is_sibling(folio))
-			goto update_start;
+			break;
 
 		if (!folio_try_get(folio))
 			goto retry;
@@ -2282,12 +2285,15 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,
 		if (unlikely(folio != xas_reload(&xas)))
 			goto put_folio;
 
+		next = folio_next_index(folio);
 		if (!folio_batch_add(fbatch, folio)) {
-			nr = folio_nr_pages(folio);
-			*start = folio->index + nr;
+			*start = next;
 			goto out;
 		}
-		xas_advance(&xas, folio_next_index(folio) - 1);
+
+		if (next > end)
+			break;
+		xas_advance(&xas, next - 1);
 		continue;
 put_folio:
 		folio_put(folio);
@@ -2296,9 +2302,7 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,
 		xas_reset(&xas);
 	}
 
-update_start:
 	nr = folio_batch_count(fbatch);
-
 	if (nr) {
 		folio = fbatch->folios[nr - 1];
 		*start = folio_next_index(folio);
Signature · 1 line
2.43.0
#3 lore raw

PATCH 3/5 mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ

In reply to Chi Zhiling

From: Chi Zhiling <[email protected]>

Change SGP_NOALLOC to return 0 with NULL folio on hole, matching SGP_READ behavior. This simplifies the sgp_type handling by unifying hole semantics across these types.

Previously, SGP_NOALLOC returned -ENOENT on hole, while SGP_READ returned 0. This inconsistency required special handling in callers like khugepaged and userfaultfd.

After this change:
- khugepaged: behavior unchanged (checks both error and NULL folio)
- userfaultfd: behavior unchanged (both -ENOENT and NULL are converted to -EFAULT before returning to userspace)

---
 include/linux/shmem_fs.h | 2 +-
 mm/khugepaged.c          | 2 +-
 mm/shmem.c               | 9 +++------
 3 files changed, 5 insertions(+), 8 deletions(-)
include/linux/shmem_fs.h +1 -1
index 93a0ba872ebe..d461713c095b 100644
@@ -165,7 +165,7 @@ extern unsigned long shmem_partial_swap_usage(struct address_space *mapping,
 /* Flag allocation requirements to shmem_get_folio */
 enum sgp_type {
 	SGP_READ,	/* don't exceed i_size, don't allocate page */
-	SGP_NOALLOC,	/* similar, but fail on hole or use fallocated page */
+	SGP_NOALLOC,	/* like SGP_READ, but accept fallocated page */
 	SGP_CACHE,	/* don't exceed i_size, may allocate page */
 	SGP_WRITE,	/* may exceed i_size, may allocate !Uptodate page */
 	SGP_FALLOC,	/* like SGP_WRITE, but make existing page Uptodate */
mm/khugepaged.c +1 -1
index b8452dbdb043..3309d1c094df 100644
@@ -1950,7 +1950,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
 				xas_unlock_irq(&xas);
 				/* swap in or instantiate fallocated page */
 				if (shmem_get_folio(mapping->host, index, 0,
-						&folio, SGP_NOALLOC)) {
+						&folio, SGP_NOALLOC) || !folio) {
 					result = SCAN_FAIL;
 					goto xa_unlocked;
 				}
mm/shmem.c +3 -6
index 3b5dc21b323c..458853c506ea 100644
@@ -2524,14 +2524,11 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 	}
 
 	/*
-	 * SGP_READ: succeed on hole, with NULL folio, letting caller zero.
-	 * SGP_NOALLOC: fail on hole, with NULL folio, letting caller fail.
+	 * SGP_READ/SGP_NOALLOC: succeed on hole, with NULL folio.
 	 */
 	*foliop = NULL;
-	if (sgp == SGP_READ)
+	if (sgp <= SGP_NOALLOC)
 		return 0;
-	if (sgp == SGP_NOALLOC)
-		return -ENOENT;
 
 	/*
 	 * Fast cache lookup and swap lookup did not find it: allocate.
@@ -2657,7 +2654,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
  *
  * When no folio is found, the behavior depends on @sgp:
  *  - for SGP_READ, *@foliop is %NULL and 0 is returned
- *  - for SGP_NOALLOC, *@foliop is %NULL and -ENOENT is returned
+ *  - for SGP_NOALLOC, *@foliop is %NULL and 0 is returned
  *  - for all other flags a new folio is allocated, inserted into the
  *    page cache and returned locked in @foliop.
  *
Signature · 1 line
2.43.0
reviewer
#4 lore raw

Re: [PATCH v1 3/5] mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ

Quoted 9 lines · Chi Zhiling

From: Chi Zhiling <[email protected]>

Change SGP_NOALLOC to return 0 with NULL folio on hole, matching SGP_READ behavior. This simplifies the sgp_type handling by unifying hole semantics across these types.

Previously, SGP_NOALLOC returned -ENOENT on hole, while SGP_READ returned 0. This inconsistency required special handling in callers like khugepaged and userfaultfd.

This patch doesn't seem to be a performance optimization, and I'm not convinced.

Quoted 2 lines

After this change:
- khugepaged: behavior unchanged (checks both error and NULL folio)

But this adds an extra check to khugepaged, and I'm not sure it's worth it.

Quoted 2 lines

- userfaultfd: behavior unchanged (both -ENOENT and NULL are converted to -EFAULT before returning to userspace)

No, this will break userfaultfd, cause userfaultfd currently returns an error code directly in this case (please update to the latest codebase.). See:

static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff)

{
	struct folio *folio;
	int err;

	err = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC);
	if (err)
		return ERR_PTR(err);

	return folio;
}
Quoted 63 lines
---
  include/linux/shmem_fs.h | 2 +-
  mm/khugepaged.c          | 2 +-
  mm/shmem.c               | 9 +++------
  3 files changed, 5 insertions(+), 8 deletions(-)
include/linux/shmem_fs.h +1 -1
index 93a0ba872ebe..d461713c095b 100644
@@ -165,7 +165,7 @@ extern unsigned long shmem_partial_swap_usage(struct address_space *mapping,
  /* Flag allocation requirements to shmem_get_folio */
  enum sgp_type {
  	SGP_READ,	/* don't exceed i_size, don't allocate page */
-	SGP_NOALLOC,	/* similar, but fail on hole or use fallocated page */
+	SGP_NOALLOC,	/* like SGP_READ, but accept fallocated page */
  	SGP_CACHE,	/* don't exceed i_size, may allocate page */
  	SGP_WRITE,	/* may exceed i_size, may allocate !Uptodate page */
  	SGP_FALLOC,	/* like SGP_WRITE, but make existing page Uptodate */
mm/khugepaged.c +1 -1
index b8452dbdb043..3309d1c094df 100644
@@ -1950,7 +1950,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
  				xas_unlock_irq(&xas);
  				/* swap in or instantiate fallocated page */
  				if (shmem_get_folio(mapping->host, index, 0,
-						&folio, SGP_NOALLOC)) {
+						&folio, SGP_NOALLOC) || !folio) {
  					result = SCAN_FAIL;
  					goto xa_unlocked;
  				}
mm/shmem.c +3 -6
index 3b5dc21b323c..458853c506ea 100644
@@ -2524,14 +2524,11 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
  	}
 
  	/*
-	 * SGP_READ: succeed on hole, with NULL folio, letting caller zero.
-	 * SGP_NOALLOC: fail on hole, with NULL folio, letting caller fail.
+	 * SGP_READ/SGP_NOALLOC: succeed on hole, with NULL folio.
  	 */
  	*foliop = NULL;
-	if (sgp == SGP_READ)
+	if (sgp <= SGP_NOALLOC)
  		return 0;
-	if (sgp == SGP_NOALLOC)
-		return -ENOENT;
 
  	/*
  	 * Fast cache lookup and swap lookup did not find it: allocate.
@@ -2657,7 +2654,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
   *
   * When no folio is found, the behavior depends on @sgp:
   *  - for SGP_READ, *@foliop is %NULL and 0 is returned
- *  - for SGP_NOALLOC, *@foliop is %NULL and -ENOENT is returned
+ *  - for SGP_NOALLOC, *@foliop is %NULL and 0 is returned
   *  - for all other flags a new folio is allocated, inserted into the
   *    page cache and returned locked in @foliop.
   *
#5 lore raw

Re: [PATCH v1 3/5] mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ

Quoted 7 lines · Baolin Wang
Quoted 6 lines · Chi Zhiling

From: Chi Zhiling <[email protected]>

Change SGP_NOALLOC to return 0 with NULL folio on hole, matching SGP_READ behavior. This simplifies the sgp_type handling by unifying hole semantics across these types.

Quoted 5 lines
Quoted 3 lines

Previously, SGP_NOALLOC returned -ENOENT on hole, while SGP_READ returned 0. This inconsistency required special handling in callers like khugepaged and userfaultfd.

This patch doesn't seem to be a performance optimization, and I'm not convinced.

Hi, baolin

This is not an optimization patch, and it is not related to this patch series either.

Quoted 5 lines
Quoted 3 lines

After this change:
- khugepaged: behavior unchanged (checks both error and NULL folio)

But this adds an extra check to khugepaged, and I'm not sure it's worth it.

You are right, I will drop this patch in v2.

Quoted 11 lines
Quoted 3 lines

- userfaultfd: behavior unchanged (both -ENOENT and NULL are converted to -EFAULT before returning to userspace)

No, this will break userfaultfd, cause userfaultfd currently returns an error code directly in this case (please update to the latest codebase.). See:

static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff) {
    struct folio *folio;
    int err;

Quoted 6 lines

    err = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC); if (err)
        return ERR_PTR(err);

    return folio;
}

The return value of shmem_get_folio_noalloc() will change, but it seems this does not affect the logic in mfill_atomic_pte_continue().

```
	folio = ops->get_folio_noalloc(inode, pgoff);
	/* Our caller expects us to return -EFAULT if we failed to find folio */
	if (IS_ERR_OR_NULL(folio))
		return -EFAULT;
```

No matter whether get_folio_noalloc() returns an error or NULL, the error code is converted to -EFAULT here, and mfill_atomic_pte_continue() is currently the only caller of get_folio_noalloc().

Did I miss something?

Thanks,

Quoted 67 lines
Quoted 67 lines

---
  include/linux/shmem_fs.h | 2 +-
  mm/khugepaged.c          | 2 +-
  mm/shmem.c               | 9 +++------

  3 files changed, 5 insertions(+), 8 deletions(-)
include/linux/shmem_fs.h +0 -0
index 93a0ba872ebe..d461713c095b 100644
@@ -165,7 +165,7 @@ extern unsigned long 
 shmem_partial_swap_usage(struct address_space *mapping,
   /* Flag allocation requirements to shmem_get_folio */
   enum sgp_type {

      SGP_READ,    /* don't exceed i_size, don't allocate page */ -    SGP_NOALLOC,    /* similar, but fail on hole or use fallocated page */
+    SGP_NOALLOC,    /* like SGP_READ, but accept fallocated page */ SGP_CACHE,    /* don't exceed i_size, may allocate page */ SGP_WRITE,    /* may exceed i_size, may allocate !Uptodate page */ SGP_FALLOC,    /* like SGP_WRITE, but make existing page Uptodate */

mm/khugepaged.c +1 -1
index b8452dbdb043..3309d1c094df 100644
@@ -1950,7 +1950,7 @@ static enum scan_result collapse_file(struct 
 mm_struct *mm, unsigned long addr,
                   xas_unlock_irq(&xas);
                   /* swap in or instantiate fallocated page */
                   if (shmem_get_folio(mapping->host, index, 0,
-                        &folio, SGP_NOALLOC)) {
+                        &folio, SGP_NOALLOC) || !folio) {
                       result = SCAN_FAIL;
                       goto xa_unlocked;

                  }

mm/shmem.c +3 -6
index 3b5dc21b323c..458853c506ea 100644
@@ -2524,14 +2524,11 @@ static int shmem_get_folio_gfp(struct inode 
 *inode, pgoff_t index,
       }
       /*
-     * SGP_READ: succeed on hole, with NULL folio, letting caller zero.
-     * SGP_NOALLOC: fail on hole, with NULL folio, letting caller fail.
+     * SGP_READ/SGP_NOALLOC: succeed on hole, with NULL folio.
        */
       *foliop = NULL;
-    if (sgp == SGP_READ)
+    if (sgp <= SGP_NOALLOC)
           return 0;
-    if (sgp == SGP_NOALLOC)
-        return -ENOENT;
       /*
        * Fast cache lookup and swap lookup did not find it: allocate.
@@ -2657,7 +2654,7 @@ static int shmem_get_folio_gfp(struct inode 
 *inode, pgoff_t index,
    *
    * When no folio is found, the behavior depends on @sgp:
    *  - for SGP_READ, *@foliop is %NULL and 0 is returned
- *  - for SGP_NOALLOC, *@foliop is %NULL and -ENOENT is returned
+ *  - for SGP_NOALLOC, *@foliop is %NULL and 0 is returned
    *  - for all other flags a new folio is allocated, inserted into the
    *    page cache and returned locked in @foliop.

   *

reviewer
#6 lore raw

Re: [PATCH v1 3/5] mm/shmem: make SGP_NOALLOC succeed on hole like SGP_READ

Quoted 54 lines · Chi Zhiling
Quoted 7 lines · Baolin Wang
Quoted 6 lines · Chi Zhiling

From: Chi Zhiling <[email protected]>

Change SGP_NOALLOC to return 0 with NULL folio on hole, matching SGP_READ behavior. This simplifies the sgp_type handling by unifying hole semantics across these types.

Quoted 5 lines
Quoted 3 lines

Previously, SGP_NOALLOC returned -ENOENT on hole, while SGP_READ returned 0. This inconsistency required special handling in callers like khugepaged and userfaultfd.

This patch doesn't seem to be a performance optimization, and I'm not convinced.

Hi, baolin

This is not an optimization patch, and it is not related to this patch series either.

Quoted 5 lines
Quoted 3 lines

After this change:
- khugepaged: behavior unchanged (checks both error and NULL folio)

But this adds an extra check to khugepaged, and I'm not sure it's worth it.

You are right, I will drop this patch in v2.

Quoted 11 lines
Quoted 3 lines

- userfaultfd: behavior unchanged (both -ENOENT and NULL are converted to -EFAULT before returning to userspace)

No, this will break userfaultfd, cause userfaultfd currently returns an error code directly in this case (please update to the latest codebase.). See:

static struct folio *shmem_get_folio_noalloc(struct inode *inode, pgoff_t pgoff) {
     struct folio *folio;
     int err;

Quoted 6 lines

     err = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC); if (err)
         return ERR_PTR(err);

     return folio;
}

The return value of shmem_get_folio_noalloc() will change, but it seems this does not affect the logic in mfill_atomic_pte_continue().

```
     folio = ops->get_folio_noalloc(inode, pgoff);
     /* Our caller expects us to return -EFAULT if we failed to find 
folio */
     if (IS_ERR_OR_NULL(folio))
         return -EFAULT;
```
Quoted 3 lines

No matter whether get_folio_noalloc() returns an error or NULL, the error code is converted to -EFAULT here, and mfill_atomic_pte_continue() is currently the only caller of get_folio_noalloc().

Did I miss something?

OK.

But I still wonder if it's worth changing the definition of a flag, especially when there's no obvious bug here and this is merely a cleanup. Yet this cleanup requires adding an extra check for khugepaged, which makes me feel it is not worth it. Unless Hugh prefers to change the meaning of this flag:).

#7 lore raw

PATCH 4/5 mm/shmem: introduce copy_zero_to_iter() for large zeroing

In reply to Chi Zhiling

From: Chi Zhiling <[email protected]>

This is a prep patch for shmem folio batching in the read path, where non-uptodate folios need to be handled in the main iteration loop. A large non-uptodate folio should be treated as a hole.

Currently, holes larger than PAGE_SIZE cannot be handled because ZERO_PAGE is limited to a single page. Add copy_zero_to_iter() as a wrapper to support copying larger zero ranges to the iterator.

---
 mm/shmem.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
mm/shmem.c +15 -1
index 458853c506ea..96ea5a4c0aff 100644
@@ -3336,6 +3336,20 @@ shmem_write_end(const struct kiocb *iocb, struct address_space *mapping,
 	return copied;
 }
 
+static size_t copy_zero_to_iter(size_t bytes, struct iov_iter *i)
+{
+	struct folio *zero = largest_zero_folio();
+	unsigned long nr, ret, written = 0;
+
+	do {
+		nr = min(bytes - written, folio_size(zero));
+		ret = copy_folio_to_iter(zero, 0, nr, i);
+		written += ret;
+	} while (written < bytes && ret == nr);
+
+	return written;
+}
+
 static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 {
 	struct file *file = iocb->ki_filp;
@@ -3430,7 +3444,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 			 * clear_user() not so much, that it is noticeably
 			 * faster to copy the zero page instead of clearing.
 			 */
-			ret = copy_page_to_iter(ZERO_PAGE(0), offset, nr, to);
+			ret = copy_zero_to_iter(nr, to);
 		} else {
 			/*
 			 * But submitting the same page twice in a row to
Signature · 1 line
2.43.0
#8 lore raw

PATCH 5/5 mm/shmem: optimize file read with folio batching

In reply to Chi Zhiling

From: Chi Zhiling <[email protected]>

Optimize shmem file read by using filemap_get_folios_contig() to batch fetch contiguous folios from the page cache, reducing the overhead of repeated shmem_get_folio() calls.

When the folio batch is exhausted, attempt to refill it with filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back to shmem_get_folio() to handle these cases individually.

Additionally:
- Defer folio_put() until the batch is exhausted or on exit
- Add folio_test_uptodate() check before copying to ensure data validity

---
 mm/shmem.c | 50 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 36 insertions(+), 14 deletions(-)
mm/shmem.c +36 -14
index 96ea5a4c0aff..e0eacc23cccd 100644
@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	struct file *file = iocb->ki_filp;
 	struct inode *inode = file_inode(file);
 	struct address_space *mapping = inode->i_mapping;
+	struct folio_batch fbatch;
 	pgoff_t index;
 	unsigned long offset;
 	int error = 0;
 	ssize_t retval = 0;
 
+	folio_batch_init(&fbatch);
+
 	for (;;) {
 		struct folio *folio = NULL;
 		struct page *page = NULL;
@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 			break;
 
 		index = iocb->ki_pos >> PAGE_SHIFT;
-		error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
-		if (error) {
-			if (error == -EINVAL)
-				error = 0;
-			break;
+fetch:
+		folio = folio_batch_next(&fbatch);
+		if (!folio) {
+			pgoff_t start = index;
+			pgoff_t end = (iocb->ki_pos + to->count - 1) >> PAGE_SHIFT;
+
+			if (folio_batch_count(&fbatch)) {
+				for (int i = 0; i < folio_batch_count(&fbatch); i++)
+					folio_put(fbatch.folios[i]);
+				folio_batch_reinit(&fbatch);
+			}
+
+			filemap_get_folios_contig(inode->i_mapping, &start, end, &fbatch);
+			if (folio_batch_count(&fbatch))
+				goto fetch;
+
+			error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
+			if (unlikely(error)) {
+				if (error == -EINVAL)
+					error = 0;
+				break;
+			}
+			if (folio) {
+				folio_unlock(folio);
+				folio_batch_add(&fbatch, folio);
+				fbatch.i++;
+			}
 		}
-		if (folio) {
-			folio_unlock(folio);
 
+		if (folio) {
 			page = folio_file_page(folio, index);
 			if (PageHWPoison(page)) {
-				folio_put(folio);
 				error = -EIO;
 				break;
 			}
@@ -3398,11 +3421,9 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		 * are called without i_rwsem protection against truncate
 		 */
 		i_size = i_size_read(inode);
-		if (unlikely(iocb->ki_pos >= i_size)) {
-			if (folio)
-				folio_put(folio);
+		if (unlikely(iocb->ki_pos >= i_size))
 			break;
-		}
+
 		end_offset = min_t(loff_t, i_size, iocb->ki_pos + to->count);
 		if (folio && likely(!fallback_page_copy))
 			fsize = folio_size(folio);
@@ -3411,7 +3432,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		offset = iocb->ki_pos & (fsize - 1);
 		nr = min_t(loff_t, end_offset - iocb->ki_pos, fsize - offset);
 
-		if (folio) {
+		if (folio && folio_test_uptodate(folio)) {
 			/*
 			 * If users can be writing to this page using arbitrary
 			 * virtual addresses, take care about potential aliasing
@@ -3437,7 +3458,6 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 				ret = copy_folio_to_iter(folio, offset, nr, to);
 			else
 				ret = copy_page_to_iter(page, offset, nr, to);
-			folio_put(folio);
 		} else if (user_backed_iter(to)) {
 			/*
 			 * Copy to user tends to be so well optimized, but
@@ -3466,6 +3486,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		cond_resched();
 	}
 
+	for (int i = 0; i < folio_batch_count(&fbatch); i++)
+		folio_put(fbatch.folios[i]);
 	file_accessed(file);
 	return retval ? retval : error;
 }
Signature · 1 line
2.43.0
reviewer
#9 lore raw

Re: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching

Quoted 77 lines · Chi Zhiling

From: Chi Zhiling <[email protected]>

Optimize shmem file read by using filemap_get_folios_contig() to batch fetch contiguous folios from the page cache, reducing the overhead of repeated shmem_get_folio() calls.

When the folio batch is exhausted, attempt to refill it with filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back to shmem_get_folio() to handle these cases individually.

Additionally:
- Defer folio_put() until the batch is exhausted or on exit
- Add folio_test_uptodate() check before copying to ensure data validity

---
  mm/shmem.c | 50 ++++++++++++++++++++++++++++++++++++--------------
  1 file changed, 36 insertions(+), 14 deletions(-)
mm/shmem.c +30 -5
index 96ea5a4c0aff..e0eacc23cccd 100644
@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
  	struct file *file = iocb->ki_filp;
  	struct inode *inode = file_inode(file);
  	struct address_space *mapping = inode->i_mapping;
+	struct folio_batch fbatch;
  	pgoff_t index;
  	unsigned long offset;
  	int error = 0;
  	ssize_t retval = 0;
 
+	folio_batch_init(&fbatch);
+
  	for (;;) {
  		struct folio *folio = NULL;
  		struct page *page = NULL;
@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
  			break;
 
  		index = iocb->ki_pos >> PAGE_SHIFT;
-		error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
-		if (error) {
-			if (error == -EINVAL)
-				error = 0;
-			break;
+fetch:
+		folio = folio_batch_next(&fbatch);
+		if (!folio) {
+			pgoff_t start = index;
+			pgoff_t end = (iocb->ki_pos + to->count - 1) >> PAGE_SHIFT;
+
+			if (folio_batch_count(&fbatch)) {
+				for (int i = 0; i < folio_batch_count(&fbatch); i++)
+					folio_put(fbatch.folios[i]);
+				folio_batch_reinit(&fbatch);
+			}
+
+			filemap_get_folios_contig(inode->i_mapping, &start, end, &fbatch);
+			if (folio_batch_count(&fbatch))
+				goto fetch;
+
+			error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
+			if (unlikely(error)) {
+				if (error == -EINVAL)
+					error = 0;
+				break;
+			}
+			if (folio) {
+				folio_unlock(folio);
+				folio_batch_add(&fbatch, folio);
+				fbatch.i++;
+			}
  		}

mm/shmem.c static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) · line 3412

Quoted 10 lines
mm/shmem.c +1 -3
-		if (folio) {
-			folio_unlock(folio);
 
+		if (folio) {
  			page = folio_file_page(folio, index);
  			if (PageHWPoison(page)) {
-				folio_put(folio);
  				error = -EIO;
  				break;
  			}

I haven't tested it yet, but I'm sure this will break the fallback_page_copy mode.

You're fetching from the batch at folio granularity here, but in fallback_page_copy mode, we still copy the data at page granularity.

#10 lore raw

Re: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching

Quoted 93 lines · Baolin Wang
Quoted 77 lines · Chi Zhiling

From: Chi Zhiling <[email protected]>

Optimize shmem file read by using filemap_get_folios_contig() to batch fetch contiguous folios from the page cache, reducing the overhead of repeated shmem_get_folio() calls.

When the folio batch is exhausted, attempt to refill it with filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back to shmem_get_folio() to handle these cases individually.

Additionally:
- Defer folio_put() until the batch is exhausted or on exit
- Add folio_test_uptodate() check before copying to ensure data validity

---
  mm/shmem.c | 50 ++++++++++++++++++++++++++++++++++++--------------

  1 file changed, 36 insertions(+), 14 deletions(-)
mm/shmem.c +30 -5
index 96ea5a4c0aff..e0eacc23cccd 100644
@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
       struct file *file = iocb->ki_filp;
       struct inode *inode = file_inode(file);
       struct address_space *mapping = inode->i_mapping;
+    struct folio_batch fbatch;
       pgoff_t index;
       unsigned long offset;
       int error = 0;
       ssize_t retval = 0;
+    folio_batch_init(&fbatch);
+
       for (;;) {
           struct folio *folio = NULL;
           struct page *page = NULL;
@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
               break;
           index = iocb->ki_pos >> PAGE_SHIFT;
-        error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
-        if (error) {
-            if (error == -EINVAL)
-                error = 0;
-            break;
+fetch:
+        folio = folio_batch_next(&fbatch);
+        if (!folio) {
+            pgoff_t start = index;
+            pgoff_t end = (iocb->ki_pos + to->count - 1) >> PAGE_SHIFT;
+
+            if (folio_batch_count(&fbatch)) {
+                for (int i = 0; i < folio_batch_count(&fbatch); i++)
+                    folio_put(fbatch.folios[i]);
+                folio_batch_reinit(&fbatch);
+            }
+
+            filemap_get_folios_contig(inode->i_mapping, &start, end, 
 &fbatch);
+            if (folio_batch_count(&fbatch))
+                goto fetch;
+
+            error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
+            if (unlikely(error)) {
+                if (error == -EINVAL)
+                    error = 0;
+                break;
+            }
+            if (folio) {
+                folio_unlock(folio);
+                folio_batch_add(&fbatch, folio);
+                fbatch.i++;
+            }

mm/shmem.c static ssize_t shmem_file_read_iter(struct · line 3412

Quoted 10 lines
mm/shmem.c +1 -3
           }
-        if (folio) {
-            folio_unlock(folio);
+        if (folio) {
               page = folio_file_page(folio, index);
               if (PageHWPoison(page)) {
-                folio_put(folio);
                   error = -EIO;
                   break;
               }

I haven't tested it yet, but I'm sure this will break the fallback_page_copy mode.

Quoted 1 line

You're fetching from the batch at folio granularity here, but in fallback_page_copy mode, we still copy the data at page granularity.

Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.

Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?

~~~
Sashika:

Can this lockless check cause readers to receive spurious zeros instead of the actual data?

Since filemap_get_folios_contig() fetches locklessly, a folio that is currently being allocated and initialized, or swapped in by another thread, might not yet be marked uptodate.

By checking folio_test_uptodate() without holding the folio lock, this falls through to the zero-fill paths if the folio isn't uptodate yet. The original code avoided this by using shmem_get_folio() which waits for the folio lock and I/O completion.
~~~

Thanks,

reviewer
#11 lore raw

Re: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching

Quoted 94 lines · Chi Zhiling
Quoted 93 lines · Baolin Wang
Quoted 77 lines · Chi Zhiling

From: Chi Zhiling <[email protected]>

Optimize shmem file read by using filemap_get_folios_contig() to batch fetch contiguous folios from the page cache, reducing the overhead of repeated shmem_get_folio() calls.

When the folio batch is exhausted, attempt to refill it with filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back to shmem_get_folio() to handle these cases individually.

Additionally:
- Defer folio_put() until the batch is exhausted or on exit
- Add folio_test_uptodate() check before copying to ensure data validity

---
  mm/shmem.c | 50 ++++++++++++++++++++++++++++++++++++--------------

  1 file changed, 36 insertions(+), 14 deletions(-)
mm/shmem.c +30 -5
index 96ea5a4c0aff..e0eacc23cccd 100644
@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
       struct file *file = iocb->ki_filp;
       struct inode *inode = file_inode(file);
       struct address_space *mapping = inode->i_mapping;
+    struct folio_batch fbatch;
       pgoff_t index;
       unsigned long offset;
       int error = 0;
       ssize_t retval = 0;
+    folio_batch_init(&fbatch);
+
       for (;;) {
           struct folio *folio = NULL;
           struct page *page = NULL;
@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
               break;
           index = iocb->ki_pos >> PAGE_SHIFT;
-        error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
-        if (error) {
-            if (error == -EINVAL)
-                error = 0;
-            break;
+fetch:
+        folio = folio_batch_next(&fbatch);
+        if (!folio) {
+            pgoff_t start = index;
+            pgoff_t end = (iocb->ki_pos + to->count - 1) >> PAGE_SHIFT;
+
+            if (folio_batch_count(&fbatch)) {
+                for (int i = 0; i < folio_batch_count(&fbatch); i++)
+                    folio_put(fbatch.folios[i]);
+                folio_batch_reinit(&fbatch);
+            }
+
+            filemap_get_folios_contig(inode->i_mapping, &start, end, 
 &fbatch);
+            if (folio_batch_count(&fbatch))
+                goto fetch;
+
+            error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
+            if (unlikely(error)) {
+                if (error == -EINVAL)
+                    error = 0;
+                break;
+            }
+            if (folio) {
+                folio_unlock(folio);
+                folio_batch_add(&fbatch, folio);
+                fbatch.i++;
+            }

mm/shmem.c static ssize_t shmem_file_read_iter(struct · line 3412

Quoted 10 lines
mm/shmem.c +1 -3
           }
-        if (folio) {
-            folio_unlock(folio);
+        if (folio) {
               page = folio_file_page(folio, index);
               if (PageHWPoison(page)) {
-                folio_put(folio);
                   error = -EIO;
                   break;
               }

I haven't tested it yet, but I'm sure this will break the fallback_page_copy mode.

Quoted 3 lines
Quoted 1 line

You're fetching from the batch at folio granularity here, but in fallback_page_copy mode, we still copy the data at page granularity.

Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.

Good.

Quoted 1 line

Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?

I'm not concerned about swap-in folios, since we already ensure the folio is uptodate when swapping out.

What I'm worried about is that this change moves the folio_test_uptodate() check outside of the folio lock, which could introduce some race? Of course, with the original logic, the folio state could also transition from !uptodate to uptodate after the folio is unlocked (I need more thought).

Another race condition that needs to be addressed is, when getting the folio locklessly via filemap_get_folios_contig(), there could be a concurrent race with swap-out (because you removed the folio->mapping check under folio lock). This could result in reading zero data when it shouldn't (the folio actually needs to be swapped in).

#12 lore raw

Re: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching

Quoted 105 lines · Baolin Wang
Quoted 96 lines · Chi Zhiling
Quoted 95 lines · Baolin Wang
Quoted 87 lines · Chi Zhiling

From: Chi Zhiling <[email protected]>

Optimize shmem file read by using filemap_get_folios_contig() to batch fetch contiguous folios from the page cache, reducing the overhead of repeated shmem_get_folio() calls.

When the folio batch is exhausted, attempt to refill it with filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back to shmem_get_folio() to handle these cases individually.

Additionally:
- Defer folio_put() until the batch is exhausted or on exit
- Add folio_test_uptodate() check before copying to ensure data validity

---
  mm/shmem.c | 50 ++++++++++++++++++++++++++++++++++++--------------

  1 file changed, 36 insertions(+), 14 deletions(-)
mm/shmem.c +31 -8
index 96ea5a4c0aff..e0eacc23cccd 100644
@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
       struct file *file = iocb->ki_filp;
       struct inode *inode = file_inode(file);
       struct address_space *mapping = inode->i_mapping;
+    struct folio_batch fbatch;
       pgoff_t index;
       unsigned long offset;
       int error = 0;
       ssize_t retval = 0;
+    folio_batch_init(&fbatch);
+
       for (;;) {
           struct folio *folio = NULL;
           struct page *page = NULL;
@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
               break;
           index = iocb->ki_pos >> PAGE_SHIFT;
-        error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
-        if (error) {
-            if (error == -EINVAL)
-                error = 0;
-            break;
+fetch:
+        folio = folio_batch_next(&fbatch);
+        if (!folio) {
+            pgoff_t start = index;
+            pgoff_t end = (iocb->ki_pos + to->count - 1) >> 
 PAGE_SHIFT;
+
+            if (folio_batch_count(&fbatch)) {
+                for (int i = 0; i < folio_batch_count(&fbatch); i++)
+                    folio_put(fbatch.folios[i]);
+                folio_batch_reinit(&fbatch);
+            }
+
+            filemap_get_folios_contig(inode->i_mapping, &start, 
 end, &fbatch);
+            if (folio_batch_count(&fbatch))
+                goto fetch;
+
+            error = shmem_get_folio(inode, index, 0, &folio, 
 SGP_READ);
+            if (unlikely(error)) {
+                if (error == -EINVAL)
+                    error = 0;
+                break;
+            }
+            if (folio) {
+                folio_unlock(folio);
+                folio_batch_add(&fbatch, folio);
+                fbatch.i++;
+            }
           }
-        if (folio) {
-            folio_unlock(folio);
+        if (folio) {
               page = folio_file_page(folio, index);
               if (PageHWPoison(page)) {
-                folio_put(folio);
                   error = -EIO;
Quoted 2 lines

                  break;
              }

I haven't tested it yet, but I'm sure this will break the fallback_page_copy mode.

Quoted 3 lines
Quoted 1 line

You're fetching from the batch at folio granularity here, but in fallback_page_copy mode, we still copy the data at page granularity.

Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.

Good.

Quoted 8 lines
Quoted 1 line

Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?

I'm not concerned about swap-in folios, since we already ensure the folio is uptodate when swapping out.

What I'm worried about is that this change moves the folio_test_uptodate() check outside of the folio lock, which could introduce some race? Of course, with the original logic, the folio state could also transition from !uptodate to uptodate after the folio is unlocked (I need more thought).

Another race condition that needs to be addressed is, when getting the folio locklessly via filemap_get_folios_contig(), there could be a concurrent race with swap-out (because you removed the folio->mapping check under folio lock). This could result in reading zero data when it shouldn't (the folio actually needs to be swapped in).

Sorry, I don't fully understand what you mean. It seems that swap-out does not clear the uptodate flag, and the folio cannot be freed during the read operation because its reference count is held. So a swapped-out folio should not be mistaken for a zero-filled folio. Right?

reviewer
#13 lore raw

Re: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching

Quoted 108 lines · Chi Zhiling
Quoted 107 lines · Baolin Wang
Quoted 96 lines · Chi Zhiling
Quoted 95 lines · Baolin Wang
Quoted 87 lines · Chi Zhiling

From: Chi Zhiling <[email protected]>

Optimize shmem file read by using filemap_get_folios_contig() to batch fetch contiguous folios from the page cache, reducing the overhead of repeated shmem_get_folio() calls.

When the folio batch is exhausted, attempt to refill it with filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back to shmem_get_folio() to handle these cases individually.

Additionally:
- Defer folio_put() until the batch is exhausted or on exit
- Add folio_test_uptodate() check before copying to ensure data validity

---
  mm/shmem.c | 50 ++++++++++++++++++++++++++++++++++++--------------

  1 file changed, 36 insertions(+), 14 deletions(-)
mm/shmem.c +31 -8
index 96ea5a4c0aff..e0eacc23cccd 100644
@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
       struct file *file = iocb->ki_filp;
       struct inode *inode = file_inode(file);
       struct address_space *mapping = inode->i_mapping;
+    struct folio_batch fbatch;
       pgoff_t index;
       unsigned long offset;
       int error = 0;
       ssize_t retval = 0;
+    folio_batch_init(&fbatch);
+
       for (;;) {
           struct folio *folio = NULL;
           struct page *page = NULL;
@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
               break;
           index = iocb->ki_pos >> PAGE_SHIFT;
-        error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
-        if (error) {
-            if (error == -EINVAL)
-                error = 0;
-            break;
+fetch:
+        folio = folio_batch_next(&fbatch);
+        if (!folio) {
+            pgoff_t start = index;
+            pgoff_t end = (iocb->ki_pos + to->count - 1) >> 
 PAGE_SHIFT;
+
+            if (folio_batch_count(&fbatch)) {
+                for (int i = 0; i < folio_batch_count(&fbatch); i++)
+                    folio_put(fbatch.folios[i]);
+                folio_batch_reinit(&fbatch);
+            }
+
+            filemap_get_folios_contig(inode->i_mapping, &start, 
 end, &fbatch);
+            if (folio_batch_count(&fbatch))
+                goto fetch;
+
+            error = shmem_get_folio(inode, index, 0, &folio, 
 SGP_READ);
+            if (unlikely(error)) {
+                if (error == -EINVAL)
+                    error = 0;
+                break;
+            }
+            if (folio) {
+                folio_unlock(folio);
+                folio_batch_add(&fbatch, folio);
+                fbatch.i++;
+            }
           }
-        if (folio) {
-            folio_unlock(folio);
+        if (folio) {
               page = folio_file_page(folio, index);
               if (PageHWPoison(page)) {
-                folio_put(folio);
                   error = -EIO;
Quoted 2 lines

                  break;
              }

I haven't tested it yet, but I'm sure this will break the fallback_page_copy mode.

Quoted 3 lines
Quoted 1 line

You're fetching from the batch at folio granularity here, but in fallback_page_copy mode, we still copy the data at page granularity.

Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.

Good.

Quoted 1 line

Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?

Quoted 8 lines
Quoted 6 lines

I'm not concerned about swap-in folios, since we already ensure the folio is uptodate when swapping out.

What I'm worried about is that this change moves the folio_test_uptodate() check outside of the folio lock, which could introduce some race? Of course, with the original logic, the folio state could also transition from !uptodate to uptodate after the folio is unlocked (I need more thought).

Another race condition that needs to be addressed is, when getting the folio locklessly via filemap_get_folios_contig(), there could be a concurrent race with swap-out (because you removed the folio->mapping check under folio lock). This could result in reading zero data when it shouldn't (the folio actually needs to be swapped in).

Sorry, I don't fully understand what you mean. It seems that swap-out does not clear the uptodate flag, and the folio cannot be freed during the read operation because its reference count is held. So a swapped-out folio should not be mistaken for a zero-filled folio. Right?

Sorry for the late reply. Ah, I missed the held refcnt, and now sounds reasonable.

#14 lore raw

Re: [PATCH v1 5/5] mm/shmem: optimize file read with folio batching

Quoted 116 lines · Baolin Wang
Quoted 113 lines · Chi Zhiling
Quoted 112 lines · Baolin Wang
Quoted 96 lines · Chi Zhiling
Quoted 95 lines · Baolin Wang
Quoted 87 lines · Chi Zhiling

From: Chi Zhiling <[email protected]>

Optimize shmem file read by using filemap_get_folios_contig() to batch fetch contiguous folios from the page cache, reducing the overhead of repeated shmem_get_folio() calls.

When the folio batch is exhausted, attempt to refill it with filemap_get_folios_contig(). If no folios are found (hole or swapped out pages), fall back to shmem_get_folio() to handle these cases individually.

Additionally:
- Defer folio_put() until the batch is exhausted or on exit
- Add folio_test_uptodate() check before copying to ensure data validity

---
  mm/shmem.c | 50 ++++++++++++++++++++++++++++++++++++--------------

  1 file changed, 36 insertions(+), 14 deletions(-)
mm/shmem.c +31 -8
index 96ea5a4c0aff..e0eacc23cccd 100644
@@ -3355,11 +3355,14 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
       struct file *file = iocb->ki_filp;
       struct inode *inode = file_inode(file);
       struct address_space *mapping = inode->i_mapping;
+    struct folio_batch fbatch;
       pgoff_t index;
       unsigned long offset;
       int error = 0;
       ssize_t retval = 0;
+    folio_batch_init(&fbatch);
+
       for (;;) {
           struct folio *folio = NULL;
           struct page *page = NULL;
@@ -3372,18 +3375,38 @@ static ssize_t shmem_file_read_iter(struct 
 kiocb *iocb, struct iov_iter *to)
               break;
           index = iocb->ki_pos >> PAGE_SHIFT;
-        error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
-        if (error) {
-            if (error == -EINVAL)
-                error = 0;
-            break;
+fetch:
+        folio = folio_batch_next(&fbatch);
+        if (!folio) {
+            pgoff_t start = index;
+            pgoff_t end = (iocb->ki_pos + to->count - 1) >> 
 PAGE_SHIFT;
+
+            if (folio_batch_count(&fbatch)) {
+                for (int i = 0; i < folio_batch_count(&fbatch); i++)
+                    folio_put(fbatch.folios[i]);
+                folio_batch_reinit(&fbatch);
+            }
+
+            filemap_get_folios_contig(inode->i_mapping, &start, 
 end, &fbatch);
+            if (folio_batch_count(&fbatch))
+                goto fetch;
+
+            error = shmem_get_folio(inode, index, 0, &folio, 
 SGP_READ);
+            if (unlikely(error)) {
+                if (error == -EINVAL)
+                    error = 0;
+                break;
+            }
+            if (folio) {
+                folio_unlock(folio);
+                folio_batch_add(&fbatch, folio);
+                fbatch.i++;
+            }
           }
-        if (folio) {
-            folio_unlock(folio);
+        if (folio) {
               page = folio_file_page(folio, index);
               if (PageHWPoison(page)) {
-                folio_put(folio);
                   error = -EIO;
Quoted 2 lines

                  break;
              }

I haven't tested it yet, but I'm sure this will break the fallback_page_copy mode.

Quoted 3 lines
Quoted 1 line

You're fetching from the batch at folio granularity here, but in fallback_page_copy mode, we still copy the data at page granularity.

Yes, that's a bug, Sashika has already reported it. I’ll fix it in the next version.

Good.

Quoted 1 line

Sashika also reported another potential issue below, but I think that is a misunderstanding, because a folio fetched from the swap cache is already marked Uptodate before being added to the page cache. What do you think?

I'm not concerned about swap-in folios, since we already ensure the folio is uptodate when swapping out.

What I'm worried about is that this change moves the folio_test_uptodate() check outside of the folio lock, which could introduce some race? Of course, with the original logic, the folio state could also transition from !uptodate to uptodate after the folio is unlocked (I need more thought).

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Another race condition that needs to be addressed is, when getting the folio locklessly via filemap_get_folios_contig(), there could be a concurrent race with swap-out (because you removed the folio-

Quoted 1 line

mapping check under folio lock). This could result in reading zero

data when it shouldn't (the folio actually needs to be swapped in).

Sorry, I don't fully understand what you mean. It seems that swap-out does not clear the uptodate flag, and the folio cannot be freed during the read operation because its reference count is held. So a swapped- out folio should not be mistaken for a zero-filled folio. Right?

Sorry for the late reply. Ah, I missed the held refcnt, and now sounds reasonable.

I've just sent out v2. Feel free to comment. :)

https://lore.kernel.org/linux-fsdevel/[email protected]/

maintainer
#15 lore raw
In reply to Chi Zhiling
Quoted 5 lines · On Wed, 20 May 2026 18:15:33 +0800 Chi Zhiling <[email protected]> wrote:

From: Chi Zhiling <[email protected]>

This series improves shmem read performance by implementing folio batching in the read path and reducing unnecessary xarray lookups.

Thanks.

Quoted 32 lines

Performance Results
===================

Testing was performed with fio sequential read workloads:

  fio --ioengine=sync --rw=read --size=1G --runtime=180

### THP Disabled - Normal Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 11.4GiB/s | 12.7GiB/s | +11.4%      |
| 64k        | 11.2GiB/s | 12.2GiB/s | +8.9%       |
| 4k         | 3809MiB/s | 3838MiB/s | +0.8%       |

### THP Disabled - Fallocated Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 23.7GiB/s | 28.7GiB/s | +21.1%      |
| 64k        | 22.6GiB/s | 27.0GiB/s | +19.5%      |
| 4k         | 4668MiB/s | 4678MiB/s | +0.2%       |

### THP Enabled - Normal Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 13.9GiB/s | 13.9GiB/s | 0%          |
| 64k        | 13.4GiB/s | 13.4GiB/s | 0%          |
| 4k         | 3818MiB/s | 3836MiB/s | +0.5%       |
Quoted 7 lines

### THP Enabled - Fallocated Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 24.1GiB/s | 34.9GiB/s | +44.8%      |
| 64k        | 22.9GiB/s | 31.3GiB/s | +36.7%      |
| 4k         | 4721MiB/s | 4708MiB/s | -0.3%       |

That looks nice.

AI review might have found a few things:

	https://sashiko.dev/#/patchset/[email protected]

I'll skip the patchset for now (unreviewed v1!).

#16 lore raw
Quoted 50 lines · Andrew Morton
Quoted 5 lines · On Wed, 20 May 2026 18:15:33 +0800 Chi Zhiling <[email protected]> wrote:

From: Chi Zhiling <[email protected]>

This series improves shmem read performance by implementing folio batching in the read path and reducing unnecessary xarray lookups.

Thanks.

Quoted 32 lines

Performance Results
===================

Testing was performed with fio sequential read workloads:

   fio --ioengine=sync --rw=read --size=1G --runtime=180

### THP Disabled - Normal Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 11.4GiB/s | 12.7GiB/s | +11.4%      |
| 64k        | 11.2GiB/s | 12.2GiB/s | +8.9%       |
| 4k         | 3809MiB/s | 3838MiB/s | +0.8%       |

### THP Disabled - Fallocated Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 23.7GiB/s | 28.7GiB/s | +21.1%      |
| 64k        | 22.6GiB/s | 27.0GiB/s | +19.5%      |
| 4k         | 4668MiB/s | 4678MiB/s | +0.2%       |

### THP Enabled - Normal Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 13.9GiB/s | 13.9GiB/s | 0%          |
| 64k        | 13.4GiB/s | 13.4GiB/s | 0%          |
| 4k         | 3818MiB/s | 3836MiB/s | +0.5%       |
Quoted 7 lines

### THP Enabled - Fallocated Files ###

| Block Size | Baseline  | v1        | Improvement |
| ---------- | --------- | --------- | ----------- |
| 1M         | 24.1GiB/s | 34.9GiB/s | +44.8%      |
| 64k        | 22.9GiB/s | 31.3GiB/s | +36.7%      |
| 4k         | 4721MiB/s | 4708MiB/s | -0.3%       |
Quoted 6 lines

That looks nice.

AI review might have found a few things:

	https://sashiko.dev/#/patchset/[email protected]

I'll skip the patchset for now (unreviewed v1!).

Okay, I will fix those issues in v2.

Thanks!