mm: read the page cache without folio references

Small reads of one file from many CPUs all take and drop a reference on the same folio, and its refcount bounces between the CPUs. This series lets filemap_read() and shmem copy from the page cache without taking a reference. It is Linus' RFC [1], reworked: a folio that such a reader may copy from is freed only after an RCU grace period.

Only the folios that reads keep coming back to get the flag,
PG_rcu_free: those that folio_mark_accessed() finds as accessed as the LRU records. Streaming reads and data that never gets hot are not deferred, so the memory held back stays small and most frees are not touched.

Who gains: programs that read the same few pages with read() from many threads and keep no cache of their own. Most programs that care about read performance cache hot data themselves and never get here; of the real workloads I tried, a kernel build, pgbench and SQLite do not change. A read still pays for the system call, so this does not replace a userspace cache. It does keep the page cache from collapsing under many readers, which may let some programs keep a smaller cache of their own, or none.

1 PG_rcu_free and the deferred free (64-bit, not PREEMPT_RT)
2-3 splice and fuse no longer reuse page cache folios that a
lockless reader may still copy from
4-8 NR_FREE_RCU_PAGES; an expedited grace period when a CPU's
deferred frees pile up and memory runs short; the waiting pages are freed before the allocator, alloc_contig_range() or memory offline give up, and when compaction empties a block
9 memory_failure() and soft offline deal with such pages
10-12 filemap_read() marks a folio accessed in one place and sets
PG_rcu_free on the folios that reads keep coming back to

  13     filemap_read() copies small folios without a reference
  14     ... and large folios, with one more lookup per folio
  15     no f_ra.prev_pos store for FMODE_RANDOM files after such a read
  16-17  shmem does the same
  18     a debugfs switch to turn the reads off

Numbers: 64-vCPU KVM guest on a Threadripper PRO 7975WX (Zen 4), no debug options, base v7.3, both kernels in the same session, 3 rounds. With the default mitigations every read here pays ~400 ns at the user/kernel boundary (TSA, SRSO), which holds one thread to ~1.8M reads/s; Linus' numbers with mitigations off are in [1].

16-byte pread() at offset 0, each thread with its own fd, M reads/s:

                                 base           this series
    ext4, 64 threads             17.1-18.1      68.7-71.0
    tmpfs, 64 threads            1.5-1.9        68.3-70.5
    block dev, 64 threads        14.7-14.9      28.3-28.9
    1 thread, any of them        1.6-1.8        1.7-1.9

Base tmpfs takes the folio lock on every read: 64 threads read as fast as one. The block device takes i_rwsem shared on every read (inode_lock_shared() in blkdev_read_iter()), and its count bounces in place of the refcount.

RocksDB readrandom on a hot range (db_bench, 100 keys, 4K blocks, no block cache; the threads share one fd per table file, opened with POSIX_FADV_RANDOM), 64 threads, M ops/s:

                                 base           this series
    ext4                         10.6-10.9      14.2-14.7
    XFS                          8.9            12.2-12.4

At 16 threads ext4 is within 5% of base, XFS at 0.89 of it by median, with the series spread over 5.7-6.5 against 6.2-6.4. 16-byte reads through one shared fd at 64 threads: 1.20x base at one offset, 0.97x on random pages, where the struct file's cache line takes over from the folio refcount.

With a block cache RocksDB reads the page cache no more: its HyperClockCache does 26.5M ops/s at 64 threads on either kernel, 1.8x the series without a cache. Its LRUCache does 3.5M on this hot range, below base without a cache.

Under memory pressure no load got an OOM kill. pgbench in an 8G guest runs at 0.92-0.97 of base. A stress load that keeps rereading and dropping 1G of hot data next to a memory hog runs at 0.88; it is there to push flagged folios through the deferred free.

Open:

- Patch 17 drops the folio lock from tmpfs reads. shmem_get_folio() held it only for the lookup, so a read was never atomic against a write or a hole punch; now a read also does not wait for one that holds the lock, as filemap_read() does not. Fallocated folios stay
hidden: they are not uptodate, and only uptodate folios get the flag. A race test of reads against fallocate, its undo, hole punch, swap, collapse and migration found no bad data. I would still like Hugh to double check.
- Patch 16 makes tmpfs reads store f_ra.prev_pos, as filemap_read() does. tmpfs ignores POSIX_FADV_RANDOM, so threads that read through one tmpfs fd cannot skip that store. Not measured.
- A deferred folio is uncharged from its memcg before it waits, as every page cache folio is. For a grace period it counts towards no cgroup.
- defrag_mode: a stress test that leaves free pages in a fixed pattern makes compaction fail and the allocator stall in direct reclaim, on base too. That is a compaction bug; [3] fixes it. On that layout the series stalls badly more often, 5 runs in 10 against 1, and I do not know why yet; the series with [3] is not measured. Patch 8 cuts compaction's scanning there, but not the stalls.
- The flag is a page flag, 64-bit only. On large configs it costs MGLRU one refs bit.
- On writably mapped files, architectures that implement flush_dcache_page() take the slow path.
- filemap_mark_hot() still tests uptodate. Willy pointed out that a filesystem clearing it under a reader is a filesystem bug; the test is cheap.

Changes since v1 [2]:

- Dropped v1 patch 12, which cleared the flag on the folios of an inode being evicted (suggested by Linus). Sashiko found an ABA: a reader holding a stale pointer can see the folio move to another inode, get the flag there, lose it when that inode is evicted, and be reused in the slot it looked at. Deleting or evicting a hot file defers its frees again. The f2fs fix it depended on is no longer needed.
- Replaced v1 patch 13, which stored f_ra.prev_pos only when it
changed: as Linus pointed out, it never stays the same in a real load, and the compare cost RocksDB 4-5%. Patch 15 instead skips the store for FMODE_RANDOM files after a read done without folio references.
- New patch 8: compaction frees the migrated folios that wait for a grace period before it checks whether it is done; async compaction does not.
- PAGE_RCU_FREE is off on PREEMPT_RT, where expedited grace periods are normal ones (patch 1).
- Patch 6: the drain before OOM frees to the buddy lists, counts what reclaim can still free, checks the watermark at order 0, and waits for expedites in flight (Sashiko).
- Patch 9: memory_failure() frees the waiting pages on every retry in get_any_page(), and waits for readers on the already-poisoned path (Sashiko).
- Patches 13-14: races with a split (Sashiko). An order-0 folio is copied only at its own index; a large folio's order is clamped and PG_has_hwpoisoned read without the PG_head assertion.
- Patch 13: the folio batch is declared in the slow-path loop, so stack auto-initialisation does not zero it on reads done without references.
- Patch 16: shmem stores f_ra.prev_pos as filemap_read() does.
- v1 carried Assisted-by trailers; they are gone, so every patch shows "!" in the range-diff below.

The series is also at:
https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git rcu-free/rfc-v2

[1] https://lore.kernel.org/all/CAHk-=wjLjgfwgSZ1RDk4YdNFcV0ntosMO5AmBoHD76Zb3HFY7Q@mail.gmail.com/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

Kiryl Shutsemau (Meta) (11):
fuse: don't move a folio with PG_rcu_free into a fuse file
mm: count the pages waiting for a grace period
mm: free the pages waiting for a grace period before OOM
mm: free the pages waiting for a grace period before isolation fails mm/compaction: free the migrated folios that wait for a grace period mm/memory-failure: handle folios with PG_rcu_free mm/filemap: read large folios without folio references mm/filemap: don't store f_ra.prev_pos for FMODE_RANDOM files mm/shmem: count accesses to a folio the way filemap_read() does mm/shmem: read without folio references
mm/filemap: add a debugfs switch for lockless page cache reads

Linus Torvalds (7):
mm: free PG_rcu_free folios only after an RCU grace period
splice: don't steal page cache folios
mm: expedite the grace period when a CPU has many deferred frees
mm: have folio_mark_accessed() return whether the folio is hot mm/filemap: mark a folio accessed in one place in filemap_read() mm/filemap: set PG_rcu_free on the folios that reads keep coming back
to
mm/filemap: read from the page cache without folio references

 fs/fuse/dev.c                  |   4 +
 fs/splice.c                    |  51 -----
 include/linux/mmzone.h         |   1 +
 include/linux/page-flags.h     |  21 ++
 include/linux/pagemap.h        |  12 ++
 include/linux/swap.h           |   2 +-
 include/linux/vm_event_item.h  |   2 +
 include/trace/events/mmflags.h |   7 +
 mm/Kconfig                     |  18 ++
 mm/compaction.c                |   8 +
 mm/filemap.c                   | 346 +++++++++++++++++++++++++++++++--
 mm/folio.c                     |  46 +++--
 mm/huge_memory.c               |   1 +
 mm/internal.h                  |   3 +
 mm/memory-failure.c            |  33 ++++
 mm/memory_hotplug.c            |   6 +
 mm/page_alloc.c                | 297 +++++++++++++++++++++++++++-
 mm/page_alloc.h                |   9 +
 mm/shmem.c                     |  20 +-
 mm/show_mem.c                  |   9 +-
 mm/vmstat.c                    |   6 +
 21 files changed, 808 insertions(+), 94 deletions(-)

Range-diff against v1:
1: 80d1ea6c758a ! 1: 4ef64163f1a6 mm: free PG_rcu_free folios only after an RCU grace period
@@ Commit message

The flag is 64-bit only (CONFIG_PAGE_RCU_FREE): page flags are scarce on 32-bit, and the contention this avoids is a problem of large
- machines.
+ machines. It is off on PREEMPT_RT, where expedited grace periods are
+ normal ones, so the waiting pages cannot be freed quickly when memory
+ runs short.

pgfree_rcu in /proc/vmstat counts the deferred pages. Nothing sets the flag yet.

- Assisted-by: LLM

- [kas: rework for flagging at read time, 64-bit only, shorter comments
- and changelog]
+ [kas: rework for flagging at read time, 64-bit only and not PREEMPT_RT,
+ shorter comments and changelog]

## include/linux/page-flags.h ##
@@ mm/Kconfig: config PCP_BATCH_SCALE_MAX
the maximum latency.

+config PAGE_RCU_FREE
-+ def_bool 64BIT
++ def_bool y
++ depends on 64BIT
++ depends on !PREEMPT_RT
+ help
+ Reads from the page cache can copy from folios that are read
+ again and again without taking a reference on them, so small
@@ mm/Kconfig: config PCP_BATCH_SCALE_MAX
+ folios becomes reusable only after an RCU grace period.
+
+ 64-bit only: it takes a page flag, and those are scarce on
-+ 32-bit.
++ 32-bit. Not on PREEMPT_RT, where expedited grace periods are
++ normal ones, so freeing the waiting pages on demand would stall.
+
config PHYS_ADDR_T_64BIT
def_bool 64BIT
2: 54ae7580bf73 ! 2: 1cfc33d7be6e splice: don't steal page cache folios
@@ Commit message
stealing fails. SPLICE_F_MOVE from the page cache into a fuse file or a virtio console now copies.

- Assisted-by: LLM

3: 7a4a1df99f88 ! 3: 6f1f2faa2ee2 fuse: don't move a folio with PG_rcu_free into a fuse file
@@ Commit message

Copy such a folio instead, as fuse does whenever it cannot take one.

- Assisted-by: LLM

## fs/fuse/dev.c ##
4: 226465c86d51 ! 4: b4b2afbb5dbf mm: count the pages waiting for a grace period
@@ Commit message
free_rcu in show_free_areas(). Preparation for freeing the waiting pages sooner when they hold back memory that is needed.

- Assisted-by: LLM

## include/linux/mmzone.h ##
5: 601a788a3c9d ! 5: 1114d57c872f mm: expedite the grace period when a CPU has many deferred frees
@@ Commit message
zone still has memory. When the readers that refill the range then run short, nothing is freed that would check again.

- With rcu_normal, as on PREEMPT_RT, the expedited grace period is a
- normal one.
+ With rcu_normal, the expedited grace period is a normal one.

pgfree_rcu_expedite in /proc/vmstat counts the expedite runs that freed something.

- Assisted-by: LLM

[kas: take both lists, expedite only when the memory matters, rewrote
the changelog]
6: d13fd5787f06 ! 6: f9063909ef1e mm: free the pages waiting for a grace period before OOM
@@ Commit message
grace period.

Before should_reclaim_retry() gives up, free the waiting pages if they
- would bring an eligible zone above its min watermark, and retry. That
- costs an expedited grace period, which is cheap next to the OOM killer.
- The retries are bounded per no-progress streak, so a drain that does not
- help still ends in an OOM kill.
+ and what reclaim can still free would bring an eligible zone above its
+ min watermark, and retry. The drain frees them straight to the buddy
+ lists, where the retry's watermark check sees them, and waits for the
+ expedites already in flight. That costs an expedited grace period,
+ which is cheap next to the OOM killer. The retries are bounded per
+ no-progress streak, so a drain that does not help still ends in an OOM
+ kill.

pgfree_rcu_drain in /proc/vmstat counts the drains that freed something.

- Assisted-by: LLM

## include/linux/vm_event_item.h ##
@@ include/linux/vm_event_item.h: enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSW
PGREUSE,

## mm/page_alloc.c ##
+@@ mm/page_alloc.c: static void rcu_free_take(struct rcu_free_pcpu *r, struct list_head *list)
+ r->nr_pending = 0;
+ }
+
+-static unsigned long rcu_free_pages(struct list_head *list)
++static unsigned long rcu_free_pages(struct list_head *list, bool direct)
+ {
+ struct folio_batch fbatch;
+ struct folio *folio, *next;
+@@ mm/page_alloc.c: static unsigned long rcu_free_pages(struct list_head *list)
+ nr += folio_nr_pages(folio);
+ mod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES,
+ -folio_nr_pages(folio));
+- if (!folio_batch_add(&fbatch, folio)) {
++ if (direct) {
++ __free_pages_ok(&folio->page, folio_order(folio),
++ FPI_NONE);
++ cond_resched();
++ } else if (!folio_batch_add(&fbatch, folio)) {
+ free_unref_folios(&fbatch);
+ cond_resched_tasks_rcu_qs();
+ }
+@@ mm/page_alloc.c: static unsigned long rcu_free_pages(struct list_head *list)
+ return nr;
+ }
+
+-/* Free what rcu_free_take() took: the grace period starts after that */ +-static unsigned long rcu_free_flush(struct list_head *list)
++/*
++ * Free what rcu_free_take() took: the grace period starts after that.
++ * @direct frees to the buddy lists instead of this CPU's pcp lists.
++ */
++static unsigned long rcu_free_flush(struct list_head *list, bool direct)
+ {
+ if (list_empty(list))
+ return 0;
+
+ synchronize_rcu_expedited();
+- return rcu_free_pages(list);
++ return rcu_free_pages(list, direct);
+ }
+
+ static void rcu_free_workfn(struct work_struct *work)
+@@ mm/page_alloc.c: static void rcu_free_workfn(struct work_struct *work)
+ rcu_free_start(r);
+ spin_unlock_irq(&r->lock);
+
+- rcu_free_pages(&done);
++ rcu_free_pages(&done, false);
+ }
+
+ static void rcu_free_expedite_workfn(struct work_struct *work)
+@@ mm/page_alloc.c: static void rcu_free_expedite_workfn(struct work_struct *work)
+ rcu_free_take(r, &list);
+ spin_unlock_irq(&r->lock);
+
+- if (rcu_free_flush(&list))
++ if (rcu_free_flush(&list, false))
+ count_vm_event(PGFREE_RCU_EXPEDITE);
+ }
+
@@ mm/page_alloc.c: static void rcu_free_defer(struct folio *folio)
queue_work(rcu_free_wq, &r->expedite);
}

+/*
-+ * Free what every CPU's lists hold. A concurrent drain holds the lock -+ * until it has freed what it took.
++ * Free what every CPU's lists hold, straight to the buddy lists: the ++ * callers want to allocate the pages next. A concurrent drain holds the ++ * lock until it has freed what it took.
+ */
+static void rcu_free_drain(void)
+{
@@ mm/page_alloc.c: static void rcu_free_defer(struct folio *folio)
+ rcu_free_take(r, &list);
+ spin_unlock_irq(&r->lock);
+ }
-+ if (rcu_free_flush(&list))
++ if (rcu_free_flush(&list, true))
+ count_vm_event(PGFREE_RCU_DRAIN);
++ /* An expedite in flight holds pages it took off the lists */
++ for_each_possible_cpu(cpu)
++ flush_work(&per_cpu_ptr(&rcu_free_pcpu, cpu)->expedite);
+ mutex_unlock(&lock);
+}
+
@@ mm/page_alloc.c: bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
}

+/*
-+ * Reclaim is about to give up: if the pages waiting for a grace period
-+ * would bring an eligible zone above its min watermark, free them and
-+ * retry. Not for a dying task, which frees memory by exiting.
++ * Reclaim is about to give up: if the pages waiting for a grace period,
++ * with what reclaim can still free, would bring an eligible zone above its
++ * min watermark, free them and retry. Not for a dying task, which frees
++ * memory by exiting.
+ */
+static bool rcu_free_wait(gfp_t gfp_mask, unsigned int order,
+ struct alloc_context *ac, int alloc_flags,
@@ mm/page_alloc.c: bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
+ if (!waiting)
+ continue;
+
-+ available = waiting +
++ available = waiting + zone_reclaimable_pages(zone) +
+ zone_page_state_snapshot(zone, NR_FREE_PAGES);
-+ if (__zone_watermark_ok(zone, order, min_wmark_pages(zone),
++ /*
++ * Count pages only: for order > 0 __zone_watermark_ok() also
++ * wants a free block of that order, and the waiting pages are
++ * not free yet.
++ */
++ if (__zone_watermark_ok(zone, 0, min_wmark_pages(zone),
+ ac->highest_zoneidx, alloc_flags,
+ available)) {
+ (*rcu_free_waits)++;
7: 61b797a643a3 ! 7: 1058885aa5a9 mm: free the pages waiting for a grace period before isolation fails

    @@ Commit message
         offline stuck on a busy page would otherwise start an expedited grace
         period on every retry.
     
    -    Assisted-by: LLM
         Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
     
      ## mm/memory_hotplug.c ##
    @@ mm/memory_hotplug.c: int offline_pages(unsigned long start_pfn, unsigned long nr
      		ret = test_pages_isolated(start_pfn, end_pfn,
      					  PB_ISOLATE_MODE_MEM_OFFLINE);
     +		/* What this pass migrated can wait for a grace period */
    -+		if (ret && migrated &&
    -+		    zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES))
    ++		if (ret && migrated && zone_rcu_free_waiting(zone))
     +			rcu_free_sync();
      
      	} while (ret);
    @@ mm/page_alloc.c: static void rcu_free_drain(void)
     +		return;
     +
     +	rcu_free_drain();
    -+	for_each_possible_cpu(cpu) {
    -+		struct rcu_free_pcpu *r = per_cpu_ptr(&rcu_free_pcpu, cpu);
    -+
    -+		flush_work(&r->rwork.work);
    -+		flush_work(&r->expedite);
    -+	}
    ++	for_each_possible_cpu(cpu)
    ++		flush_work(&per_cpu_ptr(&rcu_free_pcpu, cpu)->rwork.work);
     +}
     +
      static void __init rcu_free_init(void)
    @@ mm/page_alloc.c: int alloc_contig_frozen_range_noprof(unsigned long start, unsig
      	/* Make sure the range is really isolated. */
     -	if (test_pages_isolated(outer_start, end, mode)) {
     +	ret = test_pages_isolated(outer_start, end, mode);
    -+	if (ret && zone_page_state_snapshot(cc.zone, NR_FREE_RCU_PAGES)) {
    ++	if (ret && zone_rcu_free_waiting(cc.zone)) {
     +		rcu_free_sync();
     +
     +		/* A freed folio can merge with a free buddy below start */
    @@ mm/page_alloc.h: struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned in
      void free_frozen_pages(struct page *page, unsigned int order);
      void free_unref_folios(struct folio_batch *fbatch);
     +void rcu_free_sync(void);
    ++
    ++/* Whether @zone has pages waiting for a grace period */
    ++static inline bool zone_rcu_free_waiting(struct zone *zone)
    ++{
    ++	return IS_ENABLED(CONFIG_PAGE_RCU_FREE) &&
    ++	       zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES);
    ++}
      
      #ifdef CONFIG_NUMA
      struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);
 -:  ------------ >  8:  c1fe5ee6d2a9 mm/compaction: free the migrated folios that wait for a grace period
 8:  cbcc28707563 !  9:  58f670555c22 mm/memory-failure: handle folios with PG_rcu_free

@@ Commit message
later freed with the poisoned page in it, and that page can be handed out again.

- Free the waiting pages first when the page looks like one of them.
- memory_failure() has to do it before it sets PG_hwpoison. The free of a
- poisoned order-0 page drops the page instead of returning it to the
- buddy allocator, so a later drain would still leave the error ignored.
+ Free the waiting pages first when the page looks like one of them, and
+ again on each retry in get_any_page(): the page can be freed while it
+ retries. memory_failure() has to do it before it sets PG_hwpoison. The
+ free of a poisoned order-0 page drops the page instead of returning it
+ to the buddy allocator, so a later drain would still leave the error
+ ignored.

A folio with PG_rcu_free that is still in use may be copied by a reader that holds no reference. When memory_failure() succeeds, its caller can
@@ Commit message

Both callers hold mf_mutex and can wait for the grace period.

- Assisted-by: LLM

## mm/memory-failure.c ##
-@@ mm/memory-failure.c: static int get_hwpoison_page(struct page *p, unsigned long flags)
- return ret;
+@@ mm/memory-failure.c: static int __get_hwpoison_page(struct page *page, unsigned long flags)
+ return 0;
}

+/* A page waiting for a grace period is not in the buddy yet: free it */ +static void mf_free_rcu_pages(struct page *p)
+{
-+ if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
++ if (!zone_rcu_free_waiting(page_zone(p)))
+ return;
+ if (page_count(p))
+ return;
-+ if (!zone_page_state_snapshot(page_zone(p), NR_FREE_RCU_PAGES))
-+ return;
+ if (is_free_buddy_page(p))
+ return;
+
+ rcu_free_sync();
+}
+
- /*
- * The caller must guarantee the folio isn't large folio, except hugetlb.
- * try_to_unmap() can't handle it.
+ #define GET_PAGE_MAX_RETRY_NUM 3
+
+ static int get_any_page(struct page *p, unsigned long flags)
+@@ mm/memory-failure.c: static int get_any_page(struct page *p, unsigned long flags)
+ }
+
+ try_again:
++ mf_free_rcu_pages(p);
+ if (!count_increased) {
+ ret = __get_hwpoison_page(p, flags);
+ if (!ret) {
@@ mm/memory-failure.c: int memory_failure(unsigned long pfn, int flags)
if (res != -ENOENT)
goto unlock_mutex;
@@ mm/memory-failure.c: int memory_failure(unsigned long pfn, int flags)
if (TestSetPageHWPoison(p)) {
res = -EHWPOISON;
if (flags & MF_ACTION_REQUIRED)
+@@ mm/memory-failure.c: int memory_failure(unsigned long pfn, int flags)
+ if (flags & MF_COUNT_INCREASED)
+ put_page(p);
+ action_result(pfn, MF_MSG_ALREADY_POISONED, MF_FAILED);
++ /*
++ * The memory_failure() that set PG_hwpoison may have failed
++ * without waiting for lockless readers, and the caller may
++ * unmap the page once this returns 0.
++ */
++ if (!res && folio_test_rcu_free(page_folio(p)))
++ synchronize_rcu();
+ goto unlock_mutex;
+ }
+
@@ mm/memory-failure.c: int memory_failure(unsigned long pfn, int flags)

identify_page_state:
@@ mm/memory-failure.c: int memory_failure(unsigned long pfn, int flags)
mutex_unlock(&mf_mutex);
return res;
unlock_page:
-@@ mm/memory-failure.c: int soft_offline_page(unsigned long pfn, int flags)
-
- retry:
- get_online_mems();
-+ mf_free_rcu_pages(page);
- ret = get_hwpoison_page(page, flags | MF_SOFT_OFFLINE);
- put_online_mems();
-
9: e1e5b7cd3c3e ! 10: 46d88376b778 mm: have folio_mark_accessed() return whether the folio is hot
@@ Commit message
No caller uses the value yet. Preparation for reading the folios that
it calls hot without a folio reference.

- Assisted-by: LLM

10: fcd474ea7b1c ! 11: 3afb00ad5b68 mm/filemap: mark a folio accessed in one place in filemap_read()
@@ Commit message
No change in behaviour intended. Preparation for deciding in the
helper which folios are worth reading without a reference.

- Assisted-by: LLM

11: 585dd033a029 ! 12: abe088444b40 mm/filemap: set PG_rcu_free on the folios that reads keep coming back to
@@ Commit message

pgfree_rcu_mark in /proc/vmstat counts the pages that get the flag.

- Assisted-by: LLM

[kas: count the marks, AS_RCU_READ, 64-bit only, shorter comment,
rewrote the changelog]

12:  34d07f67849e <  -:  ------------ mm/filemap: don't defer the free of folios of an inode being evicted
13:  5bd0abaeaa08 <  -:  ------------ mm/filemap: store the end of a read in f_ra only when it changes
14:  e9ece1540b38 ! 13:  970b0c9bb8d4 mm/filemap: read from the page cache without folio references

@@ Commit message
Leave to the slow path a folio that is not uptodate, needs readahead or dropbehind, is not hot, or has a poisoned page. Leave large folios there too: looking one up for every page is slower than one reference
- for all of its pages. Copy at most 512 KiB per rcu_read_lock()
- section, and skip files that never had a folio flagged or that are
- writably mapped where flush_dcache_folio() is needed.
+ for all of its pages. A split clears PG_head before the new folios
+ replace the old one in their slots, so copy an order-0 folio only at its
+ own index. Copy at most 512 KiB per rcu_read_lock() section, and skip
+ files that never had a folio flagged or that are writably mapped where
+ flush_dcache_folio() is needed.
+
+ Declare the folio batch in the loop: with stack auto-initialisation,
+ the compiler zeroes it where it is declared, and the copy without
+ references does not use it.

- Assisted-by: LLM

[kas: i_size before the second look at the slot, poisoned folio check,
- large folios left to the slow path, 512 KiB sections, shorter comments
- and changelog]
+ large folios left to the slow path, 512 KiB sections, folio batch in
+ the loop, shorter comments and changelog]

## mm/filemap.c ##
@@ mm/filemap.c: static void filemap_end_dropbehind_read(struct folio *folio)
+ return false;
+ if (folio_test_readahead(folio) || folio_test_dropbehind(folio))
+ return false;
-+ /* memory_failure() waits only for readers that got past this */
-+ if (folio_contain_hwpoisoned_page(folio))
-+ return false;
+ /* One reference covers all of a large folio's pages */
+ if (folio_test_large(folio))
+ return false;
++ /* A split clears PG_head before it puts the new folios in their slots */
++ if (folio->index != index)
++ return false;
++ /* memory_failure() waits only for readers that got past this */
++ if (folio_test_hwpoison(folio))
++ return false;
+
+ /*
+ * The slow path does not call folio_mark_accessed() on the folio that
@@ mm/filemap.c: static void filemap_end_dropbehind_read(struct folio *folio)
* filemap_read - Read data from the page cache.
* @iocb: The iocb to read.
@@ mm/filemap.c: ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
+ struct file_ra_state *ra = &filp->f_ra;
+ struct address_space *mapping = filp->f_mapping;
+ struct inode *inode = mapping->host;
+- struct folio_batch fbatch;
+ int i, error = 0;
bool writably_mapped;
loff_t isize, end_offset;
loff_t last_pos = ra->prev_pos;
@@ mm/filemap.c: ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
if (unlikely(iocb->ki_pos < 0))
return -EINVAL;
@@ mm/filemap.c: ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
- iov_iter_truncate(iter, inode->i_sb->s_maxbytes - iocb->ki_pos);
- folio_batch_init(&fbatch);
+ return 0;

+ iov_iter_truncate(iter, inode->i_sb->s_maxbytes - iocb->ki_pos);
+- folio_batch_init(&fbatch);
++
+ fast_read = filemap_read_rcu(iocb, iter, last_pos);
+ if (fast_read) {
+ already_read += fast_read;
@@ mm/filemap.c: ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
+ if (!iov_iter_count(iter))
+ goto out;
+ }
-+
+
do {
++ struct folio_batch fbatch;
++
++ folio_batch_init(&fbatch);
cond_resched();

+ /*
@@ mm/filemap.c: ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
- folio_batch_init(&fbatch);
+ filemap_end_dropbehind_read(folio);
+ folio_put(folio);
+ }
+- folio_batch_init(&fbatch);
} while (iov_iter_count(iter) && iocb->ki_pos < isize && !error);

+out:
file_accessed(filp);
- /* Readers sharing the file would bounce the line for nothing */
- if (ra->prev_pos != last_pos)
+ ra->prev_pos = last_pos;
+ return already_read ? already_read : error;
15: c90a38e2e97a ! 14: 7eaf88828496 mm/filemap: read large folios without folio references
@@ Commit message
slot of the last page to copy holds it too, the pages in between are this folio's: copy them after one lookup. The folio's size is read without a reference and only picks the page to check; a split after the
- check copies PG_rcu_free to the new folios.
+ check copies PG_rcu_free to the new folios. A split can also change the
+ order and PG_head under the reader: clamp the order, and read
+ PG_has_hwpoisoned without folio_test_has_hwpoisoned(), which asserts
+ PG_head.

Copy page by page with _copy_to_iter(): the hardened usercopy check of copy_to_iter() reads the folio's order, which a split can change.

- Assisted-by: LLM

## mm/filemap.c ##
@@ mm/filemap.c: static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index,
- /* memory_failure() waits only for readers that got past this */
- if (folio_contain_hwpoisoned_page(folio))
+ return false;
+ if (folio_test_readahead(folio) || folio_test_dropbehind(folio))
return false;
- /* One reference covers all of a large folio's pages */
- if (folio_test_large(folio))
- return false;
+ /* A split clears PG_head before it puts the new folios in their slots */
+- if (folio->index != index)
++ if (!folio_test_large(folio) && folio->index != index)
+ return false;
+ /* memory_failure() waits only for readers that got past this */
+ if (folio_test_hwpoison(folio))
+ return false;
++ /* Not folio_test_has_hwpoisoned(): it asserts PG_head, a split clears it */
++ if (folio_test_large(folio) &&
++ test_bit(PG_has_hwpoisoned, &folio->_flags_1))
++ return false;

/*
* The slow path does not call folio_mark_accessed() on the folio that
@@ mm/filemap.c: static struct folio *filemap_read_rcu_lookup(struct xa_state *xas,
+ pgoff_t index, size_t offset, size_t bytes)
+{
+ pgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE);
++ unsigned int order;
+ pgoff_t next;
+
+ if (end - index > 1 && folio_test_large(folio)) {
-+ /* Read without a reference: it only picks the page to check */
-+ next = folio_next_index(folio);
++ /* A split can change it under us: it only picks the page to check */
++ order = min(folio_large_order(folio), MAX_PAGECACHE_ORDER);
++ next = folio->index + (1UL << order);
+ if (next > index && next < end)
+ end = next;
+ xas_set(xas, end - 1);

 -:  ------------ > 15:  1455ef101c5c mm/filemap: don't store f_ra.prev_pos for FMODE_RANDOM files
16:  c878dc794ca4 ! 16:  77e7c1ce6060 mm/shmem: count accesses to a folio the way filemap_read() does
    @@ Commit message
         marks a tmpfs folio accessed too, so a folio written in several pieces
         can get the flag on its first read.
     
    -    Assisted-by: LLM
         Signed-off-by: Kiryl Shutsemau (Meta) <[email protected]>
     
      ## mm/filemap.c ##
    @@ mm/shmem.c: static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_i
      	}
      
      	file_accessed(file);
    -+	/* Readers sharing the file would bounce the line for nothing */
    -+	if (file->f_ra.prev_pos != last_pos)
    -+		file->f_ra.prev_pos = last_pos;
    ++	file->f_ra.prev_pos = last_pos;
      	return retval ? retval : error;
      }

17: e472d4328d29 ! 17: 4fc2a5f445d0 mm/shmem: read without folio references
@@ Commit message
PG_reclaim, which drop_caches can leave set on dirty tmpfs folios. Do not let it end the copy there.

- Assisted-by: LLM

## mm/filemap.c ##
@@ mm/filemap.c: static void filemap_end_dropbehind_read(struct folio *folio)
+ /* shmem does no readahead: there the bit is PG_reclaim */
+ if (folio_test_readahead(folio) && !folio_test_swapbacked(folio))
return false;
- /* memory_failure() waits only for readers that got past this */
- if (folio_contain_hwpoisoned_page(folio))
+ /* A split clears PG_head before it puts the new folios in their slots */
+ if (!folio_test_large(folio) && folio->index != index)
@@ mm/filemap.c: static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,
* Copy from the page cache without folio references, in batches, and
* return how much was copied. The slow path reads the rest.
@@ mm/shmem.c: static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_i
-
+out:
file_accessed(file);
- /* Readers sharing the file would bounce the line for nothing */
- if (file->f_ra.prev_pos != last_pos)
+ file->f_ra.prev_pos = last_pos;
+ return retval ? retval : error;
18: b05c7c67fdf7 ! 18: 9a845ea67175 mm/filemap: add a debugfs switch for lockless page cache reads
@@ Commit message
Folios that have the flag keep it, and their free stays deferred: a read that tested the key before the switch may still be copying from them.

- Assisted-by: LLM

## mm/Kconfig ##
@@ mm/Kconfig: config PAGE_RCU_FREE
- 64-bit only: it takes a page flag, and those are scarce on
- 32-bit.
+ 32-bit. Not on PREEMPT_RT, where expedited grace periods are
+ normal ones, so freeing the waiting pages on demand would stall.

+ Writing 0 to lockless_page_cache_read in debugfs turns these
+ reads off.

base-commit: c03c1a14507dfaf8b856c2608f9ac216021fc222

2.54.0

From: Linus Torvalds <[email protected]>

Reading a page cache folio without taking a reference would keep CPUs that read the same file from bouncing its refcount. But without a reference the folio can be freed, and its memory reused, while the reader copies from it.

Add PG_rcu_free: the free of a folio with the flag waits for an RCU grace period. A reader that saw the flag under rcu_read_lock(), and then still found the folio where it looked, can read it until rcu_read_unlock(). A split copies the flag, as such a reader may still be reading the new folios.

The folios wait on per-CPU lists, one grace period in flight per CPU:
call_rcu() per folio would queue a callback for every folio of a truncated file. A work, which can reschedule, frees each batch. Reclaim depends on these frees, so the workqueue is WQ_MEM_RECLAIM.

The flag is 64-bit only (CONFIG_PAGE_RCU_FREE): page flags are scarce on 32-bit, and the contention this avoids is a problem of large machines. It is off on PREEMPT_RT, where expedited grace periods are normal ones, so the waiting pages cannot be freed quickly when memory runs short.

pgfree_rcu in /proc/vmstat counts the deferred pages. Nothing sets the flag yet.

[kas: rework for flagging at read time, 64-bit only and not PREEMPT_RT,
shorter comments and changelog]

---
 include/linux/page-flags.h     |  21 +++++++
 include/linux/vm_event_item.h  |   1 +
 include/trace/events/mmflags.h |   7 +++
 mm/Kconfig                     |  15 +++++
 mm/huge_memory.c               |   1 +
 mm/memory-failure.c            |   4 ++
 mm/page_alloc.c                | 105 +++++++++++++++++++++++++++++++++
 mm/vmstat.c                    |   1 +
 8 files changed, 155 insertions(+)

index 7a863572adce..a1e52a1f8a52 100644
@@ -111,6 +111,9 @@ enum pageflags {
 	PG_swapbacked,		/* Page is backed by RAM/swap */
 	PG_unevictable,		/* Page is "unevictable"  */
 	PG_dropbehind,		/* drop pages on IO completion */
+#ifdef CONFIG_PAGE_RCU_FREE
+	PG_rcu_free,		/* free only after an RCU grace period */
+#endif
 #ifdef CONFIG_MMU
 	PG_mlocked,		/* Page is vma mlocked */
 #endif

@@ -602,6 +605,24 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)
 	FOLIO_TEST_CLEAR_FLAG(dropbehind, FOLIO_HEAD_PAGE)
 	__FOLIO_SET_FLAG(dropbehind, FOLIO_HEAD_PAGE)
 
+/*
+ * A lockless lookup that sees PG_rcu_free and then still finds the folio
+ * where it looked may read it until rcu_read_unlock().  Set under a
+ * reference, cleared only by the free.  Not for hugetlb or
+ * free_pages_nolock().
+ */
+#ifdef CONFIG_PAGE_RCU_FREE
+FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+	FOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+	__FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+#define __PG_RCU_FREE (1UL << PG_rcu_free)
+#else
+FOLIO_TEST_FLAG_FALSE(rcu_free)
+	FOLIO_TEST_SET_FLAG_FALSE(rcu_free)
+	__FOLIO_CLEAR_FLAG_NOOP(rcu_free)
+#define __PG_RCU_FREE 0
+#endif
+
 #ifdef CONFIG_HIGHMEM
 /*
  * Must use a macro here due to header dependency issues. page_zone() is not

index 2628ccda076a..cbe849701653 100644
@@ -36,6 +36,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		FOR_ALL_ZONES(ALLOCSTALL)
 		FOR_ALL_ZONES(PGSCAN_SKIP)
 		PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE,
+		PGFREE_RCU,
 		PGFAULT, PGMAJFAULT,
 		PGLAZYFREED,
 		PGREUSE,

index 935893e5ea53..47c3411a75e8 100644
@@ -117,6 +117,12 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);
 #define IF_HAVE_PG_IDLE(_name)
 #endif
 
+#ifdef CONFIG_PAGE_RCU_FREE
+#define IF_HAVE_PG_RCU_FREE(_name) ,{1UL << PG_##_name, __stringify(_name)}
+#else
+#define IF_HAVE_PG_RCU_FREE(_name)
+#endif
+
 #ifdef CONFIG_ARCH_USES_PG_ARCH_2
 #define IF_HAVE_PG_ARCH_2(_name) ,{1UL << PG_##_name, __stringify(_name)}
 #else

@@ -152,6 +158,7 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);
 	DEF_PAGEFLAG_NAME(swapbacked),					\
 	DEF_PAGEFLAG_NAME(unevictable),					\
 	DEF_PAGEFLAG_NAME(dropbehind)					\
+IF_HAVE_PG_RCU_FREE(rcu_free)						\
 IF_HAVE_PG_MLOCK(mlocked)						\
 IF_HAVE_PG_HWPOISON(hwpoison)						\
 IF_HAVE_PG_IDLE(idle)							\

index 604c58199acb..6e5341b0bf64 100644
@@ -718,6 +718,21 @@ config PCP_BATCH_SCALE_MAX
 	  latency.  This option sets the upper limit of scale factor to limit
 	  the maximum latency.
 
+config PAGE_RCU_FREE
+	def_bool y
+	depends on 64BIT
+	depends on !PREEMPT_RT
+	help
+	  Reads from the page cache can copy from folios that are read
+	  again and again without taking a reference on them, so small
+	  reads of one file from many CPUs do not contend on the folio's
+	  reference count.  The cost is that memory freed from such
+	  folios becomes reusable only after an RCU grace period.
+
+	  64-bit only: it takes a page flag, and those are scarce on
+	  32-bit.  Not on PREEMPT_RT, where expedited grace periods are
+	  normal ones, so freeing the waiting pages on demand would stall.
+
 config PHYS_ADDR_T_64BIT
 	def_bool 64BIT
 

index 1e5d68acf62a..25082949a4ca 100644
@@ -3743,6 +3743,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
 #endif
 				 (1L << PG_dirty) |
 				 (1L << PG_dropbehind) |
+				 __PG_RCU_FREE |
 				 LRU_GEN_MASK | LRU_REFS_MASK));
 
 		new_folio->mapping = folio->mapping;

index a8b03e2920ba..7dde7c3c9a0c 100644
@@ -211,6 +211,10 @@ static bool page_handle_poison(struct page *page, bool hugepage_or_freepage, boo
 			return false;
 	}
 
+	/* The free of a hwpoisoned page is not deferred: wait for readers */
+	if (release && folio_test_rcu_free(page_folio(page)))
+		synchronize_rcu();
+
 	SetPageHWPoison(page);
 	if (release)
 		put_page(page);

index cb57714539fb..66ed017bde46 100644
@@ -1343,6 +1343,7 @@ const char * const vmstat_text[] = {
 	[I(PGACTIVATE)]				= "pgactivate",
 	[I(PGDEACTIVATE)]			= "pgdeactivate",
 	[I(PGLAZYFREE)]				= "pglazyfree",
+	[I(PGFREE_RCU)]				= "pgfree_rcu",
 
 	[I(PGFAULT)]				= "pgfault",
 	[I(PGMAJFAULT)]				= "pgmajfault",

2.54.0

Sure.

I will make the help describe what this patch does. The page cache is the only user, and both symbols would be def_bool without a prompt, so I would rather add the second one with a second user.

Does that work for you?

Makes sense. Something like the diff below, on top of this patch. Introducing split_copy_flags would be a preparatory patch, and this one would only add __PG_RCU_FREE to it.

Not all page cache folios are freed through __folio_put(). Reclaim frees them with free_unref_folios() directly (shrink_folio_list(), move_folios_to_lru()), and so do folios_put_refs() and the LRU batches. __free_pages_prepare() is the one place they all pass.

index a1e52a1f8a52..6d4e534c9ccc 100644
@@ -1185,6 +1185,18 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page)
 #define __PG_MLOCKED		0
 #endif
 
+#ifdef CONFIG_ARCH_USES_PG_ARCH_2
+#define __PG_ARCH_2		BIT(PG_arch_2)
+#else
+#define __PG_ARCH_2		0
+#endif
+
+#ifdef CONFIG_ARCH_USES_PG_ARCH_3
+#define __PG_ARCH_3		BIT(PG_arch_3)
+#else
+#define __PG_ARCH_3		0
+#endif
+
 /*
  * Flags checked when a page is freed.  Pages being freed should not have
  * these flags set.  If they are, there is a problem.

index 25082949a4ca..44a1aaa3f834 100644
@@ -3689,6 +3689,23 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
 	const bool handle_hwpoison = folio_test_has_hwpoisoned(folio) && new_order;
 	long new_nr_pages = 1 << new_order;
 	long nr_pages = 1 << old_order;
+	const unsigned long split_copy_flags =
+		BIT(PG_referenced) |
+		BIT(PG_swapbacked) |
+		BIT(PG_swapcache) |
+		__PG_MLOCKED |
+		BIT(PG_uptodate) |
+		BIT(PG_active) |
+		BIT(PG_workingset) |
+		BIT(PG_locked) |
+		BIT(PG_unevictable) |
+		__PG_ARCH_2 |
+		__PG_ARCH_3 |
+		BIT(PG_dirty) |
+		BIT(PG_dropbehind) |
+		__PG_RCU_FREE |
+		LRU_GEN_MASK |
+		LRU_REFS_MASK;
 	long i;
 
 	folio_clear_has_hwpoisoned(folio);

@@ -3725,26 +3742,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
 		 * PG_anon_exclusive (-> PG_mappedtodisk) for these here.
 		 */
 		new_folio->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
-		new_folio->flags.f |= (folio->flags.f &
-				((1L << PG_referenced) |
-				 (1L << PG_swapbacked) |
-				 (1L << PG_swapcache) |
-				 (1L << PG_mlocked) |
-				 (1L << PG_uptodate) |
-				 (1L << PG_active) |
-				 (1L << PG_workingset) |
-				 (1L << PG_locked) |
-				 (1L << PG_unevictable) |
-#ifdef CONFIG_ARCH_USES_PG_ARCH_2
-				 (1L << PG_arch_2) |
-#endif
-#ifdef CONFIG_ARCH_USES_PG_ARCH_3
-				 (1L << PG_arch_3) |
-#endif
-				 (1L << PG_dirty) |
-				 (1L << PG_dropbehind) |
-				 __PG_RCU_FREE |
-				 LRU_GEN_MASK | LRU_REFS_MASK));
+		new_folio->flags.f |= folio->flags.f & split_copy_flags;
 
 		new_folio->mapping = folio->mapping;
 		new_folio->index = folio->index + i;

  Kiryl Shutsemau / Kirill A. Shutemov

Can we just rework that code first to have some clear folio freeing path, IOW doing this in a folio-specific helper?

folios_put_refs() is just a specialized and batched variant of the __folio_put() freeing path in the end. So we can hook into these two easily.

For free_unref_folios() would need another hook or a cleaner rework.

Cheers,

David

I think you're making things worse rather than better.

The whole point was to hook into *one* place: the page freeing.

Trying to make it specific to one special freeing path only makes it fragile and less obvious, and results in anybody who then uses some other freeing model to miss the RCU thing.

It also means that suddenly nobody else can use the RCU path for their own uses.

Look at all the kmalloc RCU freeing: it's used all over the place. RCU has basically taken over the kernel. Every single performance-critical lock-free area of the kernel wants RCU freeing.

I think trying to corral it into some special case is the last thing we want to do. It only makes things less potentially useful and more fragile.

Linus

The current code deals with pages vs. folios in ways that already makes this code only work with folios even in the short term.

We'll be starting to let page_folio() fail soon on things that are not actual folios. PG_folio for that is already in the works.

Right, and I'd hook into the folio freeing path if it is folio-specific.

And we will need clean path for freeing folios either way as we are starting to decouple struct page from struct folio as envisioned by Willy.

If we actually manage to make this generic for different memdescs in the future, it can obviously stay on the generic path.

Let me wrap my head around how this would look when we decouple pages from folios, where pages might not have a ->lru anymore, or not even a refcount.

Cheers,

David

So for an example of a place that already does RCU freeing of pages - *not* folios - look at the TLB shootdown.

It's very round-about and hard to tell, but it's using

  tlb_remove_table_free
    call_rcu(&batch->rcu, tlb_remove_table_rcu);

and that then does

tlb_remove_table_rcu ->

    __tlb_remove_table_free ->
      __tlb_remove_table ->
        pagetable_dtor_free ->
          pagetable_free ->
            __pagetable_free ->
               __free_pages(page, compound_order(page))

(and I might have messed this up - this really is a rats nest of nested inline functions).

And in order to do this all, it sets up *another* page just to contain the batching (which is also RCU-free'd)

Now, the batching is probably a very good thing in that context, so I'm not actually suggesting this should be converted to some common RCU-freeing thing.

Rather, I'm pointing out that we have existing hacks in this area that have grown up over time and would have been a *lot* less hacky if we had had a proper RCU-freeing page allocator back when that whole TLB freeing code was written.

So I'm not saying "we should unify this". But I *am* saying "look, RCU freeing of pages is _already_ a thing, and we have complicated hacky things in place for it".

And they are complicated not because they necessarily should be complicated, but because we solved the TLB shootdown after we already had tons of complex architecture-specific code in place, and we basically ended up adding that RCU phase on top of that existing per-architecture complexity and abstraction.

*NOW* it would be actually fairly painful to try to convert the TLB shootdown to a new order. But originally, it would have been very nice if we had just had a "Oh, mark this page for RCU freeing".

(In all fairness, we also did it that way because some architectures don't need the RCU delay simply because they have some other hardware page walking infrastructure than the actual page tables the kernel builds up. Not that those architectures *matter* today, but some of that per-architecture inline function stuff is required to basically isolate these kinds of decisions from the MM code - that deep inline function indirection has a reason that is entirely independent of the freeing side).

Linus

Yes, I am well aware of that handling. Unfortunately. :)

Yeah, I recall some details about ->lru not being usable (as reused for other purposes by some architectures) and some things about multiple page tables residing in a single physical page on some architectures (at least powerpc), which makes using ptdesc / ->lru impossible as well, as we might only be freeing parts of a page instead of full pages.

And some page tables today might already be allocated from slab (I think at least sparc64 and m68k do that).

So that's one of the reason why tlb_remove_table() consumes a void * pointer instead of a page/ptdesc pointer. It's all linked to CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE IIRC.

See, I am not sure about that given the challenge with CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE. And apparently powerpc is using that, which is not quite the museum piece (compared to m68k and sparc64 :) )

In a perfect world without these corner cases that make our life hard, possibly.

Jup.

There are some dirty details when it comes to the upcoming page vs. folio split that I am still unclear about.

The page table example you mention above is another prime example of the challenges we'll face: page tables will have a separately allocated memdesc ("ptdesc"), just like folios will have their memdesc ("folio") separately allocated.

When RCU freeing, we'll really have to keep the allocated memdesc alive as well. RCU-freeing the page only is not sufficient.

There was even talk about allocating page tables from slab later, as long as we can adjust a "struct ptdesc" to it. Which would solve the problem of powerpc etc. that don't operate on full pages for page tables.

So I might be completely off, but what I imagine is that the memdescs will get RCU-freed (these are not pages), and once they are then getting freed, they will probably take the physical page with them.

And freeing memdescs through RCU will likely be memdesc specific.

I really don't know, Willy might know more about these plans.

But I am overloaded with all the Fixes, so my brain doesn't work quite right.

Cheers,

David

From: Linus Torvalds <[email protected]>

PG_rcu_free holds back the reuse of a folio's memory until a grace period after the folio is freed. A stolen page cache pipe buffer is reused without being freed: remove_mapping() takes the folio out of the page cache and the caller keeps it. fuse_try_move_folio() puts it into a fuse file, where writes change it under a reader that found it before the steal.

Drop ->try_steal from page_cache_pipe_buf_ops. fuse_try_move_folio() and virtio_console's pipe_to_sg() then copy, as they already do when stealing fails. SPLICE_F_MOVE from the page cache into a fuse file or a virtio console now copies.

---
 fs/splice.c | 51 ---------------------------------------------------
 1 file changed, 51 deletions(-)

index 9d8f63e2fd1a..0133c2348da0 100644
@@ -55,56 +55,6 @@ static noinline void pipe_clear_nowait(struct file *file)
 	} while (!try_cmpxchg(&file->f_mode, &fmode, fmode & ~FMODE_NOWAIT));
 }
 
-/*
- * Attempt to steal a page from a pipe buffer. This should perhaps go into
- * a vm helper function, it's already simplified quite a bit by the
- * addition of remove_mapping(). If success is returned, the caller may
- * attempt to reuse this page for another destination.
- */
-static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info *pipe,
-		struct pipe_buffer *buf)
-{
-	struct folio *folio = page_folio(buf->page);
-	struct address_space *mapping;
-
-	folio_lock(folio);
-
-	mapping = folio_mapping(folio);
-	if (mapping) {
-		WARN_ON(!folio_test_uptodate(folio));
-
-		/*
-		 * At least for ext2 with nobh option, we need to wait on
-		 * writeback completing on this folio, since we'll remove it
-		 * from the pagecache.  Otherwise truncate wont wait on the
-		 * folio, allowing the disk blocks to be reused by someone else
-		 * before we actually wrote our data to them. fs corruption
-		 * ensues.
-		 */
-		folio_wait_writeback(folio);
-
-		if (!filemap_release_folio(folio, GFP_KERNEL))
-			goto out_unlock;
-
-		/*
-		 * If we succeeded in removing the mapping, set LRU flag
-		 * and return good.
-		 */
-		if (remove_mapping(mapping, folio)) {
-			buf->flags |= PIPE_BUF_FLAG_LRU;
-			return true;
-		}
-	}
-
-	/*
-	 * Raced with truncate or failed to remove folio from current
-	 * address space, unlock and return failure.
-	 */
-out_unlock:
-	folio_unlock(folio);
-	return false;
-}
-
 static void page_cache_pipe_buf_release(struct pipe_inode_info *pipe,
 					struct pipe_buffer *buf)
 {

@@ -155,7 +105,6 @@ static int page_cache_pipe_buf_confirm(struct pipe_inode_info *pipe,
 const struct pipe_buf_operations page_cache_pipe_buf_ops = {
 	.confirm	= page_cache_pipe_buf_confirm,
 	.release	= page_cache_pipe_buf_release,
-	.try_steal	= page_cache_pipe_buf_try_steal,
 	.get		= generic_pipe_buf_get,
 };
 

2.54.0

Removing code is always nice, even independent of the other changes :)

The description doesn't really describe the impact of this change. I assume a slowdown of certain workloads (does anybody really care?), and temporary increased memory consumption (as we keep the pagecache page).

No idea how often stealing actually succeeds ... a bit of history why it was added in the first place and why we no longer care now would be nice.

Cheers,

David

SPLICE_F_MOVE has not moved page cache folios between files since 2007. See 485ddb4b9741 ("1/2 splice: dont steal"). The only users that still steal from a pipe are fuse, when a server splices into /dev/fuse with SPLICE_F_MOVE, and virtio_console's splice_write(). With this patch both copy, and the source file keeps its folio cached.

Miklos, Bernd, Amit: do you know of users that depend on the move? Would the copy hurt them?

  Kiryl Shutsemau / Kirill A. Shutemov

fuse_try_move_folio() takes the folio of a pipe buffer and puts it into a fuse file, where writes change it. A folio with PG_rcu_free may still be copied by a lockless page cache read that found it before it left the page cache.

Such a folio reaches fuse when a mapped file page is gifted with vmsplice(SPLICE_F_GIFT) and its file is then truncated. fuse_check_folio() refuses it only because the flag is not among those it allows, and with a dump_page().

Copy such a folio instead, as fuse does whenever it cannot take one.

---
 fs/fuse/dev.c | 4 ++++
 1 file changed, 4 insertions(+)

index 4fec31fc0b84..526929c7b3f1 100644
@@ -1152,6 +1152,10 @@ static int fuse_try_move_folio(struct fuse_copy_state *cs, struct folio **foliop
 
 	newfolio = page_folio(buf->page);
 
+	/* A lockless page cache read may still be copying from it */
+	if (folio_test_rcu_free(newfolio))
+		goto out_fallback_unlock;
+
 	folio_clear_uptodate(newfolio);
 	folio_clear_mappedtodisk(newfolio);
 

2.54.0

Please. That would be great.

Linus

.. and in a similar vein, we should get rid of splice doing the same entirely.

Kirly's patch to just remove page_cache_pipe_buf_try_steal() entirely (ok, it was originally mine, but Kiryl took ownership of that series:
"Tag, you're it") is worth doing, but I think worth doubling down on. I think we sh ould just get rid of 'try_steal' entirely. I don't think anybody really uses it. I think it exists in fuse and in the virtio console, and neither is worth it.

Removing pipe_buf_try_steal() would basically remove all of fuse_try_move_folio().

Conversely, removing fuse_try_move_folio() would make pipe_buf_try_steal() even less useful than it is now.

Just remove them both. Good riddance.

Linus

Yes, that would be very nice.

Cheers,

David

Agreed. I will post a separate series that removes ->try_steal and its users (fuse_try_move_folio(), virtio_console's pipe_to_sg()), with SPLICE_F_MOVE, SPLICE_F_GIFT and FUSE_SPLICE_MOVE left as no-ops, and drop patches 2-3 from this series on top of it.

  Kiryl Shutsemau / Kirill A. Shutemov

Deferred folios have refcount 0 and are not on the free lists, so no counter shows them. Memory that waits for a grace period is missing from the free count and from every LRU count.

Count it per zone in NR_FREE_RCU_PAGES: nr_free_rcu in /proc/vmstat, free_rcu in show_free_areas(). Preparation for freeing the waiting pages sooner when they hold back memory that is needed.

---
 include/linux/mmzone.h | 1 +
 mm/page_alloc.c        | 4 ++++
 mm/show_mem.c          | 9 ++++++---
 mm/vmstat.c            | 2 ++
 4 files changed, 13 insertions(+), 3 deletions(-)

index 94f9c3ff5416..3336b9e5ee99 100644
@@ -228,6 +228,7 @@ enum zone_stat_item {
 	NR_ZSPAGES,		/* allocated in zsmalloc */
 #endif
 	NR_FREE_CMA_PAGES,
+	NR_FREE_RCU_PAGES,	/* freed, waiting for an RCU grace period */
 #ifdef CONFIG_UNACCEPTED_MEMORY
 	NR_UNACCEPTED,
 #endif

index a371e92da273..15bb66e36304 100644
@@ -1348,6 +1348,8 @@ static void rcu_free_pages(struct list_head *list)
 
 	folio_batch_init(&fbatch);
 	list_for_each_entry_safe(folio, next, list, lru) {
+		mod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES,
+				    -folio_nr_pages(folio));
 		if (!folio_batch_add(&fbatch, folio)) {
 			free_unref_folios(&fbatch);
 			cond_resched_tasks_rcu_qs();

@@ -1375,9 +1377,11 @@ static void rcu_free_workfn(struct work_struct *work)
 static void rcu_free_defer(struct folio *folio)
 {
 	struct rcu_free_pcpu *r = raw_cpu_ptr(&rcu_free_pcpu);
+	struct zone *zone = folio_zone(folio);
 	unsigned long flags;
 
 	spin_lock_irqsave(&r->lock, flags);
+	__mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio));
 	list_add_tail(&folio->lru, &r->pending);
 	rcu_free_start(r);
 	spin_unlock_irqrestore(&r->lock, flags);

index b938cbcd774a..0d3bddbda108 100644
@@ -204,7 +204,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 		" mapped:%lu shmem:%lu pagetables:%lu\n"
 		" sec_pagetables:%lu bounce:%lu\n"
 		" kernel_misc_reclaimable:%lu\n"
-		" free:%lu free_pcp:%lu free_cma:%lu\n",
+		" free:%lu free_pcp:%lu free_cma:%lu free_rcu:%lu\n",
 		global_node_page_state(NR_ACTIVE_ANON),
 		global_node_page_state(NR_INACTIVE_ANON),
 		global_node_page_state(NR_ISOLATED_ANON),

@@ -224,7 +224,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 		global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE),
 		global_zone_page_state(NR_FREE_PAGES),
 		free_pcp,
-		global_zone_page_state(NR_FREE_CMA_PAGES));
+		global_zone_page_state(NR_FREE_CMA_PAGES),
+		global_zone_page_state(NR_FREE_RCU_PAGES));
 
 	for_each_online_pgdat(pgdat) {
 		if (show_mem_node_skip(filter, pgdat->node_id, nodemask))

@@ -325,6 +326,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 			" free_pcp:%lukB"
 			" local_pcp:%lukB"
 			" free_cma:%lukB"
+			" free_rcu:%lukB"
 			"\n",
 			zone->name,
 			K(zone_page_state(zone, NR_FREE_PAGES)),

@@ -351,7 +353,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 			0UL,
 			K(free_pcp),
 			K((unsigned long)this_cpu_read(zone->per_cpu_pageset->count)),
-			K(zone_page_state(zone, NR_FREE_CMA_PAGES)));
+			K(zone_page_state(zone, NR_FREE_CMA_PAGES)),
+			K(zone_page_state(zone, NR_FREE_RCU_PAGES)));
 		printk("lowmem_reserve[]:");
 		for (i = 0; i < MAX_NR_ZONES; i++)
 			printk(KERN_CONT " %ld", zone->lowmem_reserve[i]);

index 66ed017bde46..64068d73d6d2 100644
@@ -1212,6 +1212,7 @@ const char * const vmstat_text[] = {
 	[I(NR_ZSPAGES)]				= "nr_zspages",
 #endif
 	[I(NR_FREE_CMA_PAGES)]			= "nr_free_cma",
+	[I(NR_FREE_RCU_PAGES)]			= "nr_free_rcu",
 #ifdef CONFIG_UNACCEPTED_MEMORY
 	[I(NR_UNACCEPTED)]			= "nr_unaccepted",
 #endif

@@ -2023,6 +2024,7 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 		switch (i) {
 		case NR_ZONE_WRITE_PENDING:
 		case NR_FREE_CMA_PAGES:
+		case NR_FREE_RCU_PAGES:
 			continue;
 		}
 		val = atomic_long_read(&vm_zone_stat[i]);

2.54.0

Even if we switch to having this folio-specific internally, I think it will be the right approach to expose this as NR_FREE_RCU_PAGES. (not NR_*FOLIOS, for example, because we want the actual # pages)

Cheers,

David

From: Linus Torvalds <[email protected]>

A deferred folio waits one to two grace periods on its CPU's list, where the page allocator cannot use it. A CPU that frees many flagged folios at once, for example when a file is truncated, can hold a lot of memory there.

When more than 4 MiB is pending on a CPU, a work takes both of the CPU's lists and frees them after an expedited grace period. That grace period starts after the take, so it covers every reader that found one of the folios. The task that frees them does not wait.

An expedited grace period interrupts every CPU, so keep it for when the memory matters. That is when the zone is below its high watermark, or when the pages waiting in it are more than an eighth of its free pages. The watermark alone comes too late for a bulk free. A truncate or fadvise(POSIX_FADV_DONTNEED) of a hot range defers all of it while the zone still has memory. When the readers that refill the range then run short, nothing is freed that would check again.

With rcu_normal, the expedited grace period is a normal one.

pgfree_rcu_expedite in /proc/vmstat counts the expedite runs that freed something.

[kas: take both lists, expedite only when the memory matters, rewrote
the changelog]

---
 include/linux/vm_event_item.h |  2 +-
 mm/page_alloc.c               | 61 ++++++++++++++++++++++++++++++++++-
 mm/vmstat.c                   |  1 +
 3 files changed, 62 insertions(+), 2 deletions(-)

index cbe849701653..45527fa7d0c1 100644
@@ -36,7 +36,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		FOR_ALL_ZONES(ALLOCSTALL)
 		FOR_ALL_ZONES(PGSCAN_SKIP)
 		PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE,
-		PGFREE_RCU,
+		PGFREE_RCU, PGFREE_RCU_EXPEDITE,
 		PGFAULT, PGMAJFAULT,
 		PGLAZYFREED,
 		PGREUSE,

index 64068d73d6d2..aa7280b5e69c 100644
@@ -1345,6 +1345,7 @@ const char * const vmstat_text[] = {
 	[I(PGDEACTIVATE)]			= "pgdeactivate",
 	[I(PGLAZYFREE)]				= "pglazyfree",
 	[I(PGFREE_RCU)]				= "pgfree_rcu",
+	[I(PGFREE_RCU_EXPEDITE)]		= "pgfree_rcu_expedite",
 
 	[I(PGFAULT)]				= "pgfault",
 	[I(PGMAJFAULT)]				= "pgmajfault",

2.54.0

When reclaim fails, should_reclaim_retry() looks only at free and reclaimable pages. It can hand the allocation to the OOM killer while enough memory is about to come back from the pages that wait for a grace period.

Before should_reclaim_retry() gives up, free the waiting pages if they and what reclaim can still free would bring an eligible zone above its min watermark, and retry. The drain frees them straight to the buddy lists, where the retry's watermark check sees them, and waits for the expedites already in flight. That costs an expedited grace period, which is cheap next to the OOM killer. The retries are bounded per no-progress streak, so a drain that does not help still ends in an OOM kill.

pgfree_rcu_drain in /proc/vmstat counts the drains that freed something.

---
 include/linux/vm_event_item.h |   2 +-
 mm/page_alloc.c               | 119 ++++++++++++++++++++++++++++++----
 mm/vmstat.c                   |   1 +
 3 files changed, 110 insertions(+), 12 deletions(-)

index 45527fa7d0c1..7626a3e3a07a 100644
@@ -36,7 +36,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		FOR_ALL_ZONES(ALLOCSTALL)
 		FOR_ALL_ZONES(PGSCAN_SKIP)
 		PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE,
-		PGFREE_RCU, PGFREE_RCU_EXPEDITE,
+		PGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN,
 		PGFAULT, PGMAJFAULT,
 		PGLAZYFREED,
 		PGREUSE,

index aa7280b5e69c..d994de1fd739 100644
@@ -1346,6 +1346,7 @@ const char * const vmstat_text[] = {
 	[I(PGLAZYFREE)]				= "pglazyfree",
 	[I(PGFREE_RCU)]				= "pgfree_rcu",
 	[I(PGFREE_RCU_EXPEDITE)]		= "pgfree_rcu_expedite",
+	[I(PGFREE_RCU_DRAIN)]			= "pgfree_rcu_drain",
 
 	[I(PGFAULT)]				= "pgfault",
 	[I(PGMAJFAULT)]				= "pgmajfault",

2.54.0

alloc_contig_range() and offline_pages() migrate the folios out of a range and then check with test_pages_isolated() that every page is free. A migrated folio that had PG_rcu_free waits for a grace period after its free, so the check fails. alloc_contig_range() returns -EBUSY, and a CMA or gigantic page allocation can fail; offline_pages() retries until the grace period ends.

When the check fails and the zone has pages waiting, free them first. alloc_contig_range() then finds the start of the free range again, as the freed folio can merge with a free buddy below it. offline_pages() does so only after a pass that migrated folios: an offline stuck on a busy page would otherwise start an expedited grace period on every retry.

---
 mm/memory_hotplug.c |  6 ++++++
 mm/page_alloc.c     | 24 +++++++++++++++++++++++-
 mm/page_alloc.h     |  8 ++++++++
 3 files changed, 37 insertions(+), 1 deletion(-)

index 226ab9cb078a..c23fab652957 100644
@@ -1967,6 +1967,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 		.nid = NUMA_NO_NODE,
 	};
 	unsigned long flags;
+	bool migrated;
 	char *reason;
 	int ret;
 	unsigned long normal_pages = 0;

@@ -2050,6 +2051,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 
 	do {
 		pfn = start_pfn;
+		migrated = false;
 		do {
 			/*
 			 * Historically we always checked for any signal and

@@ -2071,6 +2073,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 				 * out
 				 */
 				do_migrate_range(pfn, end_pfn);
+				migrated = true;
 			}
 		} while (!ret);
 

@@ -2092,6 +2095,9 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 
 		ret = test_pages_isolated(start_pfn, end_pfn,
 					  PB_ISOLATE_MODE_MEM_OFFLINE);
+		/* What this pass migrated can wait for a grace period */
+		if (ret && migrated && zone_rcu_free_waiting(zone))
+			rcu_free_sync();
 
 	} while (ret);
 

index 9caaa57bf51d..eea287ab2a01 100644
@@ -1479,6 +1479,19 @@ static void rcu_free_drain(void)
 	mutex_unlock(&lock);
 }
 
+/* Free every waiting folio, including those a work took off the lists */
+void rcu_free_sync(void)
+{
+	int cpu;
+
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return;
+
+	rcu_free_drain();
+	for_each_possible_cpu(cpu)
+		flush_work(&per_cpu_ptr(&rcu_free_pcpu, cpu)->rwork.work);
+}
+
 static void __init rcu_free_init(void)
 {
 	int cpu;

@@ -7649,7 +7662,16 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
 	outer_start = find_large_buddy(start);
 
 	/* Make sure the range is really isolated. */
-	if (test_pages_isolated(outer_start, end, mode)) {
+	ret = test_pages_isolated(outer_start, end, mode);
+	if (ret && zone_rcu_free_waiting(cc.zone)) {
+		rcu_free_sync();
+
+		/* A freed folio can merge with a free buddy below start */
+		outer_start = find_large_buddy(start);
+		ret = test_pages_isolated(outer_start, end, mode);
+	}
+
+	if (ret) {
 		ret = -EBUSY;
 		goto done;
 	}

index b9259deddb59..764e3f43b3b9 100644
@@ -235,6 +235,14 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
 	alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))
 void free_frozen_pages(struct page *page, unsigned int order);
 void free_unref_folios(struct folio_batch *fbatch);
+void rcu_free_sync(void);
+
+/* Whether @zone has pages waiting for a grace period */
+static inline bool zone_rcu_free_waiting(struct zone *zone)
+{
+	return IS_ENABLED(CONFIG_PAGE_RCU_FREE) &&
+	       zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES);
+}
 
 #ifdef CONFIG_NUMA
 struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);

2.54.0

A block that compaction has migrated every folio out of becomes free once those folios are back in the buddy lists. A migrated folio that had PG_rcu_free comes back only after a grace period, so compaction finds the block still in use, migrates more blocks than it needs and fails more often. defrag_mode relies on compaction making whole free blocks, and there it shows up as allocation stalls.

When compaction moves past a block it migrated from and the zone has pages waiting, free them before it drains the per-cpu lists and checks whether it is done. The drain costs an expedited grace period, so async compaction, which must not block, skips it.

---
 mm/compaction.c | 8 ++++++++
 mm/page_alloc.c | 2 +-
 mm/page_alloc.h | 1 +
 3 files changed, 10 insertions(+), 1 deletion(-)

index a049415512c6..2acb87a3446d 100644
@@ -2760,6 +2760,14 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 				block_start_pfn(cc->migrate_pfn, cc->order);
 
 			if (last_migrated_pfn < current_block_start) {
+				/*
+				 * Folios migrated away from lockless readers
+				 * are freed after a grace period.  Async
+				 * compaction does not wait for it.
+				 */
+				if (cc->mode != MIGRATE_ASYNC &&
+				    zone_rcu_free_waiting(cc->zone))
+					rcu_free_drain();
 				lru_add_drain_cpu_zone(cc->zone);
 				/* No more flushing until we migrate again */
 				last_migrated_pfn = 0;

index eea287ab2a01..afd4343ad572 100644
@@ -1457,7 +1457,7 @@ static void rcu_free_defer(struct folio *folio)
  * callers want to allocate the pages next.  A concurrent drain holds the
  * lock until it has freed what it took.
  */
-static void rcu_free_drain(void)
+void rcu_free_drain(void)
 {
 	static DEFINE_MUTEX(lock);
 	LIST_HEAD(list);

index 764e3f43b3b9..4f5f06adbfdc 100644
@@ -235,6 +235,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
 	alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))
 void free_frozen_pages(struct page *page, unsigned int order);
 void free_unref_folios(struct folio_batch *fbatch);
+void rcu_free_drain(void);
 void rcu_free_sync(void);
 
 /* Whether @zone has pages waiting for a grace period */

2.54.0

A page whose free waits for an RCU grace period has refcount 0 and is neither on the LRU nor in the buddy allocator. memory_failure() and soft_offline_page() cannot handle such a page: get_any_page() retries a few times and gives up. Soft offline of the page fails, and a hard error is ignored. If the page is part of a large folio, the folio is later freed with the poisoned page in it, and that page can be handed out again.

Free the waiting pages first when the page looks like one of them, and again on each retry in get_any_page(): the page can be freed while it retries. memory_failure() has to do it before it sets PG_hwpoison. The free of a poisoned order-0 page drops the page instead of returning it to the buddy allocator, so a later drain would still leave the error ignored.

A folio with PG_rcu_free that is still in use may be copied by a reader that holds no reference. When memory_failure() succeeds, its caller can unmap the page from the direct map under such a reader. Wait for a grace period before returning success. By then the page has left the page cache, or is an order-0 folio with PG_hwpoison, which a reader that starts after the grace period has to check.

Wait at the end, not right after setting PG_hwpoison. A large folio is split later, and until the new folios are in the page cache, a reader can still copy the poisoned page as part of the large one.

Both callers hold mf_mutex and can wait for the grace period.

---
 mm/memory-failure.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

index 7dde7c3c9a0c..398463b2e82d 100644
@@ -1419,6 +1419,19 @@ static int __get_hwpoison_page(struct page *page, unsigned long flags)
 	return 0;
 }
 
+/* A page waiting for a grace period is not in the buddy yet: free it */
+static void mf_free_rcu_pages(struct page *p)
+{
+	if (!zone_rcu_free_waiting(page_zone(p)))
+		return;
+	if (page_count(p))
+		return;
+	if (is_free_buddy_page(p))
+		return;
+
+	rcu_free_sync();
+}
+
 #define GET_PAGE_MAX_RETRY_NUM 3
 
 static int get_any_page(struct page *p, unsigned long flags)

@@ -1443,6 +1456,7 @@ static int get_any_page(struct page *p, unsigned long flags)
 	}
 
 try_again:
+	mf_free_rcu_pages(p);
 	if (!count_increased) {
 		ret = __get_hwpoison_page(p, flags);
 		if (!ret) {

@@ -2440,6 +2454,7 @@ int memory_failure(unsigned long pfn, int flags)
 	if (res != -ENOENT)
 		goto unlock_mutex;
 
+	mf_free_rcu_pages(p);
 	if (TestSetPageHWPoison(p)) {
 		res = -EHWPOISON;
 		if (flags & MF_ACTION_REQUIRED)

@@ -2447,6 +2462,13 @@ int memory_failure(unsigned long pfn, int flags)
 		if (flags & MF_COUNT_INCREASED)
 			put_page(p);
 		action_result(pfn, MF_MSG_ALREADY_POISONED, MF_FAILED);
+		/*
+		 * The memory_failure() that set PG_hwpoison may have failed
+		 * without waiting for lockless readers, and the caller may
+		 * unmap the page once this returns 0.
+		 */
+		if (!res && folio_test_rcu_free(page_folio(p)))
+			synchronize_rcu();
 		goto unlock_mutex;
 	}
 

@@ -2612,6 +2634,13 @@ int memory_failure(unsigned long pfn, int flags)
 
 identify_page_state:
 	res = identify_page_state(pfn, p, page_flags);
+	/*
+	 * The caller may unmap the page once this returns 0.  Lockless
+	 * readers that started before the page got PG_hwpoison, or before a
+	 * split left it in a folio of its own, may still be copying it.
+	 */
+	if (!res && folio_test_rcu_free(folio))
+		synchronize_rcu();
 	mutex_unlock(&mf_mutex);
 	return res;
 unlock_page:

2.54.0

From: Linus Torvalds <[email protected]>

Each call of folio_mark_accessed() moves a folio one step up what the LRU records of its accesses. Once the folio is at the top, the call leaves it as it was: the folio is as hot as the LRU can tell.

Return whether the call found the folio there. On the classic LRU that is active, referenced and not idle. On the multi-gen LRU it is PG_referenced, all the refs bits and PG_workingset, which lru_gen_inc_refs() now reports.

An unevictable folio counts as hot once it is referenced, and on the multi-gen LRU from the first call: it stays in memory in any case. A dropbehind folio is not hot: it is dropped after the read.

No caller uses the value yet. Preparation for reading the folios that it calls hot without a folio reference.

---
 include/linux/swap.h |  2 +-
 mm/folio.c           | 46 ++++++++++++++++++++++++++++++--------------
 2 files changed, 33 insertions(+), 15 deletions(-)

index 5658a1634b85..67aa5271ae6c 100644
@@ -295,7 +295,7 @@ extern unsigned long totalreserve_pages;
 
 /* linux/mm/folio.c */
 void folio_add_lru(struct folio *folio);
-void folio_mark_accessed(struct folio *folio);
+bool folio_mark_accessed(struct folio *folio);
 void lru_add_drain_all(void);
 
 enum lru_cache_drained {

2.54.0

From: Linus Torvalds <[email protected]>

filemap_read() calls folio_mark_accessed() in two places. Before the copy loop it marks the first folio of the batch, unless the previous read through the file ended in that folio. In the loop it marks every later folio.

The test made for the first folio also says "mark" for every later one. The loop gets to a later folio only after the folio before it was copied to its end, so last_pos is at the later folio's first byte and pos_same_folio() is false.

Do both in one helper, filemap_read_accessed(), called for each folio in the loop. For the first folio the call moves below the test of end_offset against the folio's start, which is false for it: the folio holds ki_pos, and end_offset is beyond ki_pos.

No change in behaviour intended. Preparation for deciding in the helper which folios are worth reading without a reference.

---
 mm/filemap.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

index 00fd89cf6f55..3508eea44ad9 100644
@@ -2759,6 +2759,20 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio *folio)
 	return (pos1 >> shift == pos2 >> shift);
 }
 
+/*
+ * filemap_read() is about to copy from @folio at @pos.  @last_pos is where
+ * the previous read of this open file ended, or how far this one has got.
+ *
+ * Reading on in the folio that the last read ended in is not another access
+ * to it: a folio read in small pieces is marked accessed once.
+ */
+static void filemap_read_accessed(struct folio *folio, loff_t pos,
+				  loff_t last_pos)
+{
+	if (!pos_same_folio(pos, last_pos - 1, folio))
+		folio_mark_accessed(folio);
+}
+
 static void filemap_end_dropbehind_read(struct folio *folio)
 {
 	if (!folio_test_dropbehind(folio))

@@ -2844,14 +2858,6 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		 */
 		writably_mapped = mapping_writably_mapped(mapping);
 
-		/*
-		 * When a read accesses the same folio several times, only
-		 * mark it as accessed the first time.
-		 */
-		if (!pos_same_folio(iocb->ki_pos, last_pos - 1,
-				    fbatch.folios[0]))
-			folio_mark_accessed(fbatch.folios[0]);
-
 		for (i = 0; i < folio_batch_count(&fbatch); i++) {
 			struct folio *folio = fbatch.folios[i];
 			size_t fsize = folio_size(folio);

@@ -2862,8 +2868,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 
 			if (end_offset < folio_pos(folio))
 				break;
-			if (i > 0)
-				folio_mark_accessed(folio);
+			filemap_read_accessed(folio, iocb->ki_pos, last_pos);
 			/*
 			 * If users can be writing to this folio using arbitrary
 			 * virtual addresses, take care of potential aliasing

2.54.0

From: Linus Torvalds <[email protected]>

A folio with PG_rcu_free can be read without a reference, but its memory is held back for a grace period after it is freed. That trade is worth making only for folios that readers look up again and again. Flagging every page cache folio would also delay the free of streaming reads, truncated files and reclaimed cold data, which no reader comes back to.

Set the flag in filemap_read(), under the read's reference, when the folio turns out to be hot:

- folio_mark_accessed() finds it as accessed as the LRU records, or
- the read goes back to bytes that the previous read through the same open file has just read.

filemap_read() does not mark accessed the folio that the previous read ended in, so re-reads through one open file do not make a folio hotter. The second test is for that folio.

Leave alone a folio whose next read needs a reference anyway: not uptodate, readahead or dropbehind. Leave alone unevictable folios too. folio_mark_accessed() calls them hot after at most one access, so every ramfs folio that is read would get the flag. An initramfs deleted at switch_root would then have the free of all its folios deferred.

AS_RCU_READ records that an inode has had a folio flagged, so that a lockless read can skip the inodes that never had one.

pgfree_rcu_mark in /proc/vmstat counts the pages that get the flag.

[kas: count the marks, AS_RCU_READ, 64-bit only, shorter comment,
rewrote the changelog]

---
 include/linux/pagemap.h       | 12 +++++++++++
 include/linux/vm_event_item.h |  1 +
 mm/filemap.c                  | 40 +++++++++++++++++++++++++++++++++--
 mm/vmstat.c                   |  1 +
 4 files changed, 52 insertions(+), 2 deletions(-)

index 0adfa6605653..6c7f58803eee 100644
@@ -210,6 +210,7 @@ enum mapping_flags {
 	AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,
 	AS_KERNEL_FILE = 10,	/* mapping for a fake kernel file that shouldn't
 				   account usage to user cgroups */
+	AS_RCU_READ = 11,	/* a folio has had PG_rcu_free */
 	/* Bits 16-25 are used for FOLIO_ORDER */
 	AS_FOLIO_ORDER_BITS = 5,
 	AS_FOLIO_ORDER_MIN = 16,

@@ -269,6 +270,17 @@ static inline bool mapping_unevictable(const struct address_space *mapping)
 	return mapping && test_bit(AS_UNEVICTABLE, &mapping->flags);
 }
 
+static inline void mapping_set_rcu_read(struct address_space *mapping)
+{
+	if (!test_bit(AS_RCU_READ, &mapping->flags))
+		set_bit(AS_RCU_READ, &mapping->flags);
+}
+
+static inline bool mapping_rcu_read(const struct address_space *mapping)
+{
+	return test_bit(AS_RCU_READ, &mapping->flags);
+}
+
 static inline void mapping_set_exiting(struct address_space *mapping)
 {
 	set_bit(AS_EXITING, &mapping->flags);

index 7626a3e3a07a..c1a63f070aa7 100644
@@ -37,6 +37,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		FOR_ALL_ZONES(PGSCAN_SKIP)
 		PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE,
 		PGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN,
+		PGFREE_RCU_MARK,
 		PGFAULT, PGMAJFAULT,
 		PGLAZYFREED,
 		PGREUSE,

index 3508eea44ad9..b732ecaa78a9 100644
@@ -2759,18 +2759,54 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio *folio)
 	return (pos1 >> shift == pos2 >> shift);
 }
 
+/*
+ * Reads keep coming back to @folio: free it only after an RCU grace period
+ * from now on.  Not if its next read needs a reference anyway.  Not if it
+ * is unevictable: folio_mark_accessed() calls such a folio hot after at
+ * most one access, so every ramfs folio that is read would get the flag.
+ */
+static void filemap_mark_hot(struct folio *folio)
+{
+	struct address_space *mapping;
+
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return;
+	if (folio_test_rcu_free(folio) || !folio_test_uptodate(folio) ||
+	    folio_test_readahead(folio) || folio_test_dropbehind(folio) ||
+	    folio_test_unevictable(folio))
+		return;
+	if (folio_test_set_rcu_free(folio))
+		return;
+
+	count_vm_events(PGFREE_RCU_MARK, folio_nr_pages(folio));
+	/* NULL if the folio was truncated meanwhile */
+	mapping = READ_ONCE(folio->mapping);
+	if (mapping)
+		mapping_set_rcu_read(mapping);
+}
+
 /*
  * filemap_read() is about to copy from @folio at @pos.  @last_pos is where
  * the previous read of this open file ended, or how far this one has got.
  *
  * Reading on in the folio that the last read ended in is not another access
  * to it: a folio read in small pieces is marked accessed once.
+ *
+ * The folio is hot when folio_mark_accessed() finds it as accessed as the
+ * LRU records, or when this open file goes back to data that it has just
+ * read.
  */
 static void filemap_read_accessed(struct folio *folio, loff_t pos,
 				  loff_t last_pos)
 {
-	if (!pos_same_folio(pos, last_pos - 1, folio))
-		folio_mark_accessed(folio);
+	bool hot;
+
+	if (pos_same_folio(pos, last_pos - 1, folio))
+		hot = pos < last_pos;
+	else
+		hot = folio_mark_accessed(folio);
+	if (hot)
+		filemap_mark_hot(folio);
 }
 
 static void filemap_end_dropbehind_read(struct folio *folio)

index d994de1fd739..c27caad017e3 100644
@@ -1347,6 +1347,7 @@ const char * const vmstat_text[] = {
 	[I(PGFREE_RCU)]				= "pgfree_rcu",
 	[I(PGFREE_RCU_EXPEDITE)]		= "pgfree_rcu_expedite",
 	[I(PGFREE_RCU_DRAIN)]			= "pgfree_rcu_drain",
+	[I(PGFREE_RCU_MARK)]			= "pgfree_rcu_mark",
 
 	[I(PGFAULT)]				= "pgfault",
 	[I(PGMAJFAULT)]				= "pgmajfault",

2.54.0

From: Linus Torvalds <[email protected]>

Small reads of one file from many CPUs all take and drop a reference on the same folio, and its refcount bounces between the CPUs.

Before taking a reference, copy what can be copied under rcu_read_lock() from folios with PG_rcu_free, with page faults disabled. A flagged folio is not reused before rcu_read_unlock(), so a folio still in its slot after the flag test is this file's page until then. A folio that leaves the page cache during the copy reads as the file did when it left. Read i_size between the flag and the second look at the slot: a write that extends the file fills the slot before it moves i_size.

Leave to the slow path a folio that is not uptodate, needs readahead or dropbehind, is not hot, or has a poisoned page. Leave large folios there too: looking one up for every page is slower than one reference for all of its pages. A split clears PG_head before the new folios replace the old one in their slots, so copy an order-0 folio only at its own index. Copy at most 512 KiB per rcu_read_lock() section, and skip files that never had a folio flagged or that are writably mapped where flush_dcache_folio() is needed.

Declare the folio batch in the loop: with stack auto-initialisation, the compiler zeroes it where it is declared, and the copy without references does not use it.

[kas: i_size before the second look at the slot, poisoned folio check,
large folios left to the slow path, 512 KiB sections, folio batch in the loop, shorter comments and changelog]

---
 mm/filemap.c | 178 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 175 insertions(+), 3 deletions(-)

index b732ecaa78a9..e99705a66c1d 100644
@@ -2821,6 +2821,168 @@ static void filemap_end_dropbehind_read(struct folio *folio)
 	}
 }
 
+/*
+ * Whether filemap_read_rcu_batch() can copy from @folio, or has to leave it
+ * to the slow path, which holds a reference for readahead, dropbehind and
+ * folio_mark_accessed(), and deals with poisoned pages.  @last_index is the
+ * page the previous copy ended in.
+ */
+static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index,
+				      pgoff_t last_index)
+{
+	if (!folio_test_uptodate(folio))
+		return false;
+	if (folio_test_readahead(folio) || folio_test_dropbehind(folio))
+		return false;
+	/* One reference covers all of a large folio's pages */
+	if (folio_test_large(folio))
+		return false;
+	/* A split clears PG_head before it puts the new folios in their slots */
+	if (folio->index != index)
+		return false;
+	/* memory_failure() waits only for readers that got past this */
+	if (folio_test_hwpoison(folio))
+		return false;
+
+	/*
+	 * The slow path does not call folio_mark_accessed() on the folio that
+	 * the previous read ended in, so copying from it skips nothing.  The
+	 * folio's size can change without a reference: compare pages instead.
+	 */
+	if (index == last_index)
+		return true;
+
+	/* Otherwise only if folio_mark_accessed() would leave it as it is */
+	if (lru_gen_enabled()) {
+		unsigned long all = LRU_REFS_FLAGS | BIT(PG_workingset);
+
+		return folio_test_unevictable(folio) ||
+		       (READ_ONCE(folio->flags.f) & all) == all;
+	}
+	return folio_test_referenced(folio) && !folio_test_idle(folio) &&
+	       (folio_test_active(folio) || folio_test_unevictable(folio));
+}
+
+/*
+ * Return the folio at @pos if it is this file's page until rcu_read_unlock(),
+ * and read i_size into @isize.  A folio with PG_rcu_free is freed only after
+ * an RCU grace period.  Without a reference, the folio that xas_load()
+ * returned can be freed and its memory reused as another folio before the
+ * flag is tested, so look at the slot again after the flag.
+ */
+static struct folio *filemap_read_rcu_lookup(struct xa_state *xas,
+		struct inode *inode, loff_t pos, loff_t *isize)
+{
+	struct folio *folio;
+
+	xas_set(xas, pos >> PAGE_SHIFT);
+	folio = xas_load(xas);
+	if (xas_retry(xas, folio) || !folio || xa_is_value(folio))
+		return NULL;
+
+	/* Not folio_test_rcu_free(): it may not be a folio any more */
+	if (!(READ_ONCE(folio->flags.f) & __PG_RCU_FREE))
+		return NULL;
+	/* Read i_size and slot after the flag: it implies uptodate */
+	smp_rmb();
+	*isize = i_size_read(inode);
+	if (pos >= *isize)
+		return NULL;
+	/* i_size_read() is an acquire; a write fills the slot first */
+	if (xas_reload(xas) != folio)
+		return NULL;
+	/* Read the rest of the folio only once the slot holds it */
+	smp_rmb();
+	return folio;
+}
+
+/* Bytes one RCU read section copies at most: it holds up grace periods */
+#define FILEMAP_RCU_READ_MAX	SZ_512K
+
+/*
+ * Copy from folios with PG_rcu_free without taking references, at most
+ * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock().  Return true if that
+ * much was copied and the read is not done.  *last_index is the page the
+ * previous copy ended in, set as the slow path sets last_pos.
+ */
+static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,
+		pgoff_t *last_index, size_t *copied)
+{
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
+	struct inode *inode = mapping->host;
+	XA_STATE(xas, &mapping->i_pages, 0);
+	size_t budget = FILEMAP_RCU_READ_MAX;
+	bool more = false;
+
+	/* flush_dcache_folio() is not known to be safe without a reference */
+	if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE &&
+	    mapping_writably_mapped(mapping))
+		return false;
+
+	/* A user fault cannot sleep in the RCU read section: it ends the copy */
+	rcu_read_lock();
+	pagefault_disable();
+	while (iov_iter_count(iter)) {
+		loff_t pos = iocb->ki_pos;
+		pgoff_t index = pos >> PAGE_SHIFT;
+		size_t offset = offset_in_page(pos);
+		struct folio *folio;
+		size_t bytes, n;
+		loff_t isize;
+
+		if (!budget) {
+			more = true;
+			break;
+		}
+		folio = filemap_read_rcu_lookup(&xas, inode, pos, &isize);
+		if (!folio)
+			break;
+		if (!filemap_read_rcu_can_copy(folio, index, *last_index))
+			break;
+
+		bytes = min_t(loff_t, isize - pos, iov_iter_count(iter));
+		bytes = min3(bytes, budget, PAGE_SIZE - offset);
+		/* Not _nofault(): its usercopy check reads the folio's order */
+		n = copy_page_to_iter(folio_page(folio, index - folio->index),
+				      offset, bytes, iter);
+		*copied += n;
+		iocb->ki_pos += n;
+		budget -= n;
+		if (n)
+			*last_index = (pos + n - 1) >> PAGE_SHIFT;
+		if (n < bytes)
+			break;
+	}
+	pagefault_enable();
+	rcu_read_unlock();
+
+	return more;
+}
+
+/*
+ * Copy from the page cache without folio references, in batches, and
+ * return how much was copied.  The slow path reads the rest.
+ */
+static size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,
+		loff_t last_pos)
+{
+	/* The page the previous read ended in; ULONG_MAX if none */
+	pgoff_t last_index = (last_pos - 1) >> PAGE_SHIFT;
+	size_t copied = 0;
+
+	/* Not on an inode that never had a folio flagged */
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE) ||
+	    !mapping_rcu_read(iocb->ki_filp->f_mapping))
+		return 0;
+
+	while (filemap_read_rcu_batch(iocb, iter, &last_index, &copied)) {
+		if (fatal_signal_pending(current))
+			break;
+		cond_resched();
+	}
+	return copied;
+}
+
 /**
  * filemap_read - Read data from the page cache.
  * @iocb: The iocb to read.

@@ -2841,11 +3003,11 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 	struct file_ra_state *ra = &filp->f_ra;
 	struct address_space *mapping = filp->f_mapping;
 	struct inode *inode = mapping->host;
-	struct folio_batch fbatch;
 	int i, error = 0;
 	bool writably_mapped;
 	loff_t isize, end_offset;
 	loff_t last_pos = ra->prev_pos;
+	size_t fast_read;
 
 	if (unlikely(iocb->ki_pos < 0))
 		return -EINVAL;

@@ -2855,9 +3017,19 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		return 0;
 
 	iov_iter_truncate(iter, inode->i_sb->s_maxbytes - iocb->ki_pos);
-	folio_batch_init(&fbatch);
+
+	fast_read = filemap_read_rcu(iocb, iter, last_pos);
+	if (fast_read) {
+		already_read += fast_read;
+		last_pos = iocb->ki_pos;
+		if (!iov_iter_count(iter))
+			goto out;
+	}
 
 	do {
+		struct folio_batch fbatch;
+
+		folio_batch_init(&fbatch);
 		cond_resched();
 
 		/*

@@ -2931,9 +3103,9 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 			filemap_end_dropbehind_read(folio);
 			folio_put(folio);
 		}
-		folio_batch_init(&fbatch);
 	} while (iov_iter_count(iter) && iocb->ki_pos < isize && !error);
 
+out:
 	file_accessed(filp);
 	ra->prev_pos = last_pos;
 	return already_read ? already_read : error;

2.54.0

A large folio is in all the slots of its range or in none. Once the slot of the last page to copy holds it too, the pages in between are this folio's: copy them after one lookup. The folio's size is read without a reference and only picks the page to check; a split after the check copies PG_rcu_free to the new folios. A split can also change the order and PG_head under the reader: clamp the order, and read PG_has_hwpoisoned without folio_test_has_hwpoisoned(), which asserts PG_head.

Copy page by page with _copy_to_iter(): the hardened usercopy check of copy_to_iter() reads the folio's order, which a split can change.

---
 mm/filemap.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 64 insertions(+), 8 deletions(-)

index e99705a66c1d..182ae7d5db56 100644
@@ -2834,15 +2834,16 @@ static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index,
 		return false;
 	if (folio_test_readahead(folio) || folio_test_dropbehind(folio))
 		return false;
-	/* One reference covers all of a large folio's pages */
-	if (folio_test_large(folio))
-		return false;
 	/* A split clears PG_head before it puts the new folios in their slots */
-	if (folio->index != index)
+	if (!folio_test_large(folio) && folio->index != index)
 		return false;
 	/* memory_failure() waits only for readers that got past this */
 	if (folio_test_hwpoison(folio))
 		return false;
+	/* Not folio_test_has_hwpoisoned(): it asserts PG_head, a split clears it */
+	if (folio_test_large(folio) &&
+	    test_bit(PG_has_hwpoisoned, &folio->_flags_1))
+		return false;
 
 	/*
 	 * The slow path does not call folio_mark_accessed() on the folio that

@@ -2896,6 +2897,62 @@ static struct folio *filemap_read_rcu_lookup(struct xa_state *xas,
 	return folio;
 }
 
+/*
+ * How much of @bytes, starting @offset into page @index, can be copied from
+ * @folio.  A folio is in all the slots of its range or in none: when the slot
+ * of the last page holds it too, so do the slots in between, and the pages
+ * are this folio's.
+ */
+static size_t filemap_read_rcu_size(struct xa_state *xas, struct folio *folio,
+		pgoff_t index, size_t offset, size_t bytes)
+{
+	pgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE);
+	unsigned int order;
+	pgoff_t next;
+
+	if (end - index > 1 && folio_test_large(folio)) {
+		/* A split can change it under us: it only picks the page to check */
+		order = min(folio_large_order(folio), MAX_PAGECACHE_ORDER);
+		next = folio->index + (1UL << order);
+		if (next > index && next < end)
+			end = next;
+		xas_set(xas, end - 1);
+		/* A split after this copies PG_rcu_free to the new folios */
+		if (xas_load(xas) == folio)
+			return min(bytes, (end - index) * PAGE_SIZE - offset);
+	}
+	return min(bytes, PAGE_SIZE - offset);
+}
+
+/*
+ * Copy @bytes from @folio, starting @offset into its page at @index.  Page
+ * by page, as copy_page_to_iter() does, without its checks for each call.
+ * Not copy_to_iter(): its usercopy check reads the folio's order, which a
+ * split can change.
+ */
+static size_t filemap_read_rcu_copy(struct folio *folio, pgoff_t index,
+		size_t offset, size_t bytes, struct iov_iter *iter)
+{
+	struct page *page = folio_page(folio, index - folio->index);
+	size_t n = 0;
+
+	while (n < bytes) {
+		size_t len = min(bytes - n, PAGE_SIZE - offset);
+		size_t done;
+		void *kaddr;
+
+		kaddr = kmap_local_page(page);
+		done = _copy_to_iter(kaddr + offset, len, iter);
+		kunmap_local(kaddr);
+		n += done;
+		if (done < len)
+			break;
+		offset = 0;
+		page++;
+	}
+	return n;
+}
+
 /* Bytes one RCU read section copies at most: it holds up grace periods */
 #define FILEMAP_RCU_READ_MAX	SZ_512K
 

@@ -2941,10 +2998,9 @@ static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,
 			break;
 
 		bytes = min_t(loff_t, isize - pos, iov_iter_count(iter));
-		bytes = min3(bytes, budget, PAGE_SIZE - offset);
-		/* Not _nofault(): its usercopy check reads the folio's order */
-		n = copy_page_to_iter(folio_page(folio, index - folio->index),
-				      offset, bytes, iter);
+		bytes = filemap_read_rcu_size(&xas, folio, index, offset,
+					      min(bytes, budget));
+		n = filemap_read_rcu_copy(folio, index, offset, bytes, iter);
 		*copied += n;
 		iocb->ki_pos += n;
 		budget -= n;

2.54.0

filemap_read() writes the end of every read to file->f_ra.prev_pos, in the cache line that fdget() and fdput() write when threads share the file table.

Applications that read a file at random offsets from many threads, such as RocksDB, set POSIX_FADV_RANDOM on it. Readahead then never reads
prev_pos: page_cache_sync_ra() forces the read first.

Skip the store for such files after a read that filemap_read_rcu() did in full: without folio references, the file's line is what the threads contend on. Keep it on the slow path, which takes folio references:
skipping it there made reads slower in my testing.

filemap_read() compares the start of a read with prev_pos, to tell a read that goes on in the same folio from another access to it. On an FMODE_RANDOM file it can now be stale; with many threads on one file it was another thread's read already.

---
 mm/filemap.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

index 182ae7d5db56..5998de3d6697 100644
@@ -3078,8 +3078,13 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 	if (fast_read) {
 		already_read += fast_read;
 		last_pos = iocb->ki_pos;
-		if (!iov_iter_count(iter))
-			goto out;
+		if (!iov_iter_count(iter)) {
+			file_accessed(filp);
+			/* Readahead of a random file ignores it: skip the shared store */
+			if (!(filp->f_mode & FMODE_RANDOM))
+				ra->prev_pos = last_pos;
+			return already_read;
+		}
 	}
 
 	do {

@@ -3161,7 +3166,6 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		}
 	} while (iov_iter_count(iter) && iocb->ki_pos < isize && !error);
 
-out:
 	file_accessed(filp);
 	ra->prev_pos = last_pos;
 	return already_read ? already_read : error;

2.54.0

shmem_file_read_iter() marks a folio accessed only when a read starts at the beginning of the folio. Small reads that start anywhere else never count, so a folio that is read over and over at one offset ages as if it were never read.

Count accesses with filemap_read_accessed(), as filemap_read() does: a read marks the folio accessed unless it goes on in the folio that the previous read through the file ended in. Reading the same bytes again through one open file counts once, as for other files, where it used to count each time it started at the beginning of the folio. The end of the previous read is kept in f_ra.prev_pos, as for other files.

This also sets PG_rcu_free on the tmpfs folios that reads keep coming back to. On a shmem folio the PG_readahead bit is PG_reclaim, and shmem does no readahead, so do not let that bit keep the flag off. A write marks a tmpfs folio accessed too, so a folio written in several pieces can get the flag on its first read.

---
 mm/filemap.c  | 9 +++++----
 mm/internal.h | 1 +
 mm/shmem.c    | 9 ++++-----
 3 files changed, 10 insertions(+), 9 deletions(-)

index 5998de3d6697..65d414ad9d85 100644
@@ -2772,8 +2772,10 @@ static void filemap_mark_hot(struct folio *folio)
 	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
 		return;
 	if (folio_test_rcu_free(folio) || !folio_test_uptodate(folio) ||
-	    folio_test_readahead(folio) || folio_test_dropbehind(folio) ||
-	    folio_test_unevictable(folio))
+	    folio_test_dropbehind(folio) || folio_test_unevictable(folio))
+		return;
+	/* shmem does no readahead: there the bit is PG_reclaim */
+	if (folio_test_readahead(folio) && !folio_test_swapbacked(folio))
 		return;
 	if (folio_test_set_rcu_free(folio))
 		return;

@@ -2796,8 +2798,7 @@ static void filemap_mark_hot(struct folio *folio)
  * LRU records, or when this open file goes back to data that it has just
  * read.
  */
-static void filemap_read_accessed(struct folio *folio, loff_t pos,
-				  loff_t last_pos)
+void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos)
 {
 	bool hot;
 

index 38b1165212c9..18391dc50418 100644
@@ -1248,6 +1248,7 @@ struct migration_target_control {
  */
 size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,
 			      struct folio *folio, loff_t fpos, size_t size);
+void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos);
 
 static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
 {

index 848316eaa7f4..1c700fa9666d 100644
@@ -3309,6 +3309,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	struct file *file = iocb->ki_filp;
 	struct inode *inode = file_inode(file);
 	struct address_space *mapping = inode->i_mapping;
+	loff_t last_pos = file->f_ra.prev_pos;
 	pgoff_t index;
 	unsigned long offset;
 	int error = 0;

@@ -3378,11 +3379,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 					flush_dcache_page(page);
 			}
 
-			/*
-			 * Mark the folio accessed if we read the beginning.
-			 */
-			if (!offset)
-				folio_mark_accessed(folio);
+			filemap_read_accessed(folio, iocb->ki_pos, last_pos);
 			/*
 			 * Ok, we have the page, and it's up-to-date, so
 			 * now we can copy it to user space...

@@ -3410,6 +3407,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 
 		retval += ret;
 		iocb->ki_pos += ret;
+		last_pos = iocb->ki_pos;
 
 		if (!iov_iter_count(to))
 			break;

@@ -3421,6 +3419,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	}
 
 	file_accessed(file);
+	file->f_ra.prev_pos = last_pos;
 	return retval ? retval : error;
 }
 

2.54.0

shmem_file_read_iter() takes a reference on every folio it reads, so small reads of one tmpfs file from many CPUs bounce the folio's refcount.

Copy with filemap_read_rcu() first, and continue with the existing loop where it stops. Holes, swap entries and fallocated folios that are not uptodate all end the copy. So does a hwpoisoned page, which shmem keeps in the page cache and reports with -EIO.

This drops the folio lock as well as the reference. shmem_get_folio() held it only for the lookup, and the copy ran after folio_unlock(), so a read was never atomic against a write or a hole punch. Now a read also does not wait for one that holds the lock, as filemap_read() does not.

shmem does no readahead, and on its folios the PG_readahead bit is PG_reclaim, which drop_caches can leave set on dirty tmpfs folios. Do not let it end the copy there.

---
 mm/filemap.c  |  7 ++++---
 mm/internal.h |  2 ++
 mm/shmem.c    | 11 +++++++++--
 3 files changed, 15 insertions(+), 5 deletions(-)

index 65d414ad9d85..e148490db4da 100644
@@ -2831,9 +2831,10 @@ static void filemap_end_dropbehind_read(struct folio *folio)
 static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index,
 				      pgoff_t last_index)
 {
-	if (!folio_test_uptodate(folio))
+	if (!folio_test_uptodate(folio) || folio_test_dropbehind(folio))
 		return false;
-	if (folio_test_readahead(folio) || folio_test_dropbehind(folio))
+	/* shmem does no readahead: there the bit is PG_reclaim */
+	if (folio_test_readahead(folio) && !folio_test_swapbacked(folio))
 		return false;
 	/* A split clears PG_head before it puts the new folios in their slots */
 	if (!folio_test_large(folio) && folio->index != index)

@@ -3020,7 +3021,7 @@ static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,
  * Copy from the page cache without folio references, in batches, and
  * return how much was copied.  The slow path reads the rest.
  */
-static size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,
+size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,
 		loff_t last_pos)
 {
 	/* The page the previous read ended in; ULONG_MAX if none */

index 18391dc50418..ae9a4e511c82 100644
@@ -1249,6 +1249,8 @@ struct migration_target_control {
 size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,
 			      struct folio *folio, loff_t fpos, size_t size);
 void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos);
+size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,
+			loff_t last_pos);
 
 static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
 {

index 1c700fa9666d..8700b03a8a5e 100644
@@ -3313,7 +3313,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	pgoff_t index;
 	unsigned long offset;
 	int error = 0;
-	ssize_t retval = 0;
+	ssize_t retval;
+
+	retval = filemap_read_rcu(iocb, to, last_pos);
+	if (retval) {
+		last_pos = iocb->ki_pos;
+		if (!iov_iter_count(to))
+			goto out;
+	}
 
 	for (;;) {
 		struct folio *folio = NULL;