mm: enable lru cache for smaller large folios

For systems that primarily use smaller-order large folios, enabling the lru_cache can help reduce lock contention.

For higher-order large folios, the number of folios involved is likely to be smaller, making lock contention less significant.

This patch enables the lru_cache for large folios whose nr_pages is smaller than FOLIO_BATCH_SIZE. To avoid holding too many pages in the lru_cache, which could affect accounting and reclamation, we also limit the total number of pages in the cache to FOLIO_BATCH_SIZE.

To track the number of pages, this patch adds an unsigned short nr_pages field to struct folio_batch. It cannot overflow because the batch contains at most FOLIO_BATCH_SIZE folios, each of which has fewer than FOLIO_BATCH_SIZE pages.

For non-LRU caches, folio_batch only needs to track the number of folios, so nr_pages is left at zero.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 include/linux/folio_batch.h | 25 +++++++++++++++++++++++++
 mm/folio.c                  | 10 +++++++++-
 mm/internal.h               |  4 ++--
 3 files changed, 36 insertions(+), 3 deletions(-)

include/linux/folio_batch.h +1 -0

index b45946adc50b..ffc7de091fa3 100644
@@ -10,6 +10,7 @@
 #define _LINUX_FOLIO_BATCH_H
 
 #include <linux/types.h>
+#include <linux/mm.h>
 
 /* 31 pointers + header align the folio_batch structure to a power of two */
 #define FOLIO_BATCH_SIZE       31

include/linux/folio_batch.h +23 -0

        bool percpu_pvec_drained;
        struct folio *folios[FOLIO_BATCH_SIZE];
 };

@@ -42,6 +44,7 @@ static inline void folio_batch_init(struct folio_batch *fbatch)
 {
        fbatch->nr = 0;
        fbatch->i = 0;
+       fbatch->nr_pages = 0;
        fbatch->percpu_pvec_drained = false;
 }
 

@@ -49,6 +52,7 @@ static inline void folio_batch_reinit(struct folio_batch *fbatch)
 {
        fbatch->nr = 0;
        fbatch->i = 0;
+       fbatch->nr_pages = 0;
 }
 
 static inline unsigned int folio_batch_count(const struct folio_batch *fbatch)

@@ -78,6 +82,27 @@ static inline unsigned folio_batch_add(struct folio_batch *fbatch,
        return folio_batch_space(fbatch);
 }
 
+/**
+ * folio_batch_add_lru_cache() - Add a folio to a batch of lru_cache
+ * @fbatch: The folio batch.
+ * @folio: The folio to add.
+ *
+ * The folio is added to the end of the batch.
+ * The batch must have previously been initialised using folio_batch_init().
+ *
+ * Return: 0 if the lru_cache is filled with more than FOLIO_BATCH_SIZE
+ *         pages; otherwise, the number of available slots.
+ */
+static inline unsigned folio_batch_add_lru_cache(struct folio_batch *fbatch,
+               struct folio *folio)
+{
+       fbatch->folios[fbatch->nr++] = folio;
+       fbatch->nr_pages += (unsigned short)folio_nr_pages(folio);
+       if (fbatch->nr_pages > FOLIO_BATCH_SIZE)
+               return 0;
+       return folio_batch_space(fbatch);
+}
+
 /**
  * folio_batch_next - Return the next folio to process.
  * @fbatch: The folio batch being processed.

mm/folio.c +3 -1

index 59c477120b9a..e5820d7263e8 100644
@@ -219,7 +219,7 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
        else
                local_lock(&cpu_fbatches.lock);
 
-       if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
+       if (!folio_batch_add_lru_cache(this_cpu_ptr(fbatch), folio) ||
                        !folio_may_be_lru_cached(folio) || lru_cache_disabled())
                folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
 

@@ -981,6 +981,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
        int i, j;
        struct lruvec *lruvec = NULL;
        unsigned long flags = 0;
+       unsigned long nr_pages = 0;
 
        for (i = 0, j = 0; i < folios->nr; i++) {
                struct folio *folio = folios->folios[i];

@@ -1020,6 +1021,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
 
                if (j != i)
                        folios->folios[j] = folio;
+               nr_pages += folio_nr_pages(folio);
                j++;
        }
        if (lruvec)

Smaller large folios might now be present in the LRU cache. Use David's new lru_cache_drain_for_folio() helper to drain the LRU cache before splitting a folio, ensuring that the folio can be split successfully.

Also, we only perform the drain when it may actually help, assuming that the lru_cache holds an extra reference.

Signed-off-by Barry Song (Xiaomi) <[email protected]>

---
 mm/huge_memory.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

mm/huge_memory.c +6 -0

index ced400f72d43..263ef9b6949d 100644
@@ -4201,6 +4201,9 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
                if (shmem_mapping(mapping))
                        end = shmem_fallocend(mapping->host, end);
        }
+       if (folio_ref_count(folio) == folio_expected_ref_count(folio) + 1 +
+           folio_may_be_lru_cached(folio))
+               lru_cache_drain_for_folio(folio, 1, NULL);
 
        /*
         * Racy check if we can split the page, before unmap_folio() will

@@ -4325,6 +4328,9 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
        VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
        VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
 
+       if (folio_ref_count(folio) == folio_expected_ref_count(folio) + 1 +
+           folio_may_be_lru_cached(folio))
+               lru_cache_drain_for_folio(folio, 1, NULL);
        if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
                return -EAGAIN;
 

mm/huge_memory.c +0 -1

                /*
                 * For folios with private, split_huge_page_to_list_to_order()
                 * will try to drop it before split and then check if the folio
--

2.34.1

-RFC v3:
* Rather than hard-coding < COSTLY_ORDER to enable the lru_cache, allow the lru_cache for larger orders as long as the folio contains fewer than FOLIO_BATCH_LRU pages. Also limit the total number of

On Wed, Aug 19, 2026 at 06:59:00AM +0800, Barry Song (Xiaomi) wrote:
[...]

Quoted 5 lines

-RFC v3:
* Rather than hard-coding < COSTLY_ORDER to enable the lru_cache, allow the lru_cache for larger orders as long as the folio contains fewer than FOLIO_BATCH_LRU pages. Also limit the total number of

Tiny typo ... should FOLIO_BATCH_LRU be FOLIO_BATCH_SIZE here?

This patchset enables the per-CPU LRU cache for large folios with fewer than FOLIO_BATCH_SIZE (31) pages. It also limits each per-CPU LRU cache to at most FOLIO_BATCH_SIZE pages to avoid negatively affecting accounting and memory reclamation pressure.

This is particularly beneficial on systems that use relatively small large folios. For larger folios, the benefit is likely to be smaller because far fewer folios are expected to contend for the LRU cache.

Quoted 8 lines · Barry Song (Xiaomi)

This patchset enables the per-CPU LRU cache for large folios with fewer than FOLIO_BATCH_SIZE (31) pages. It also limits each per-CPU LRU cache to at most FOLIO_BATCH_SIZE pages to avoid negatively affecting accounting and memory reclamation pressure.

This is particularly beneficial on systems that use relatively small large folios. For larger folios, the benefit is likely to be smaller because far fewer folios are expected to contend for the LRU cache.

As raised, there is this problem with collect_longterm_unpinnable_folios()

(see
https://lore.kernel.org/r/[email protected]
)

whereby we don't know how many refs we actually hold. Certainly not 1.

We might have to wait for Hugh's cleanup to handle that cleanly (and avoid all the other LRU cache draining).

Quoted 13 lines · David Hildenbrand (Arm)

Quoted 8 lines · Barry Song (Xiaomi)

This patchset enables the per-CPU LRU cache for large folios with fewer than FOLIO_BATCH_SIZE (31) pages. It also limits each per-CPU LRU cache to at most FOLIO_BATCH_SIZE pages to avoid negatively affecting accounting and memory reclamation pressure.

This is particularly beneficial on systems that use relatively small large folios. For larger folios, the benefit is likely to be smaller because far fewer folios are expected to contend for the LRU cache.

As raised, there is this problem with collect_longterm_unpinnable_folios()

Quoted 8 lines

(see
https://lore.kernel.org/r/[email protected]
)

whereby we don't know how many refs we actually hold. Certainly not 1.

We might have to wait for Hugh's cleanup to handle that cleanly (and avoid all the other LRU cache draining).

Hi David,
Thanks for raising this.
Yes, I saw your comment and took a closer look at it. I think the best approach for now is to leave that part untouched until Hugh's patch lands? We might end up doing some extra draining in that case, but that's safe—any value greater than 1 might not be. We may just drain more than necessary?

Best Regards
Barry

Quoted 19 lines · David Hildenbrand (Arm)

Quoted 8 lines · Barry Song (Xiaomi)

This patchset enables the per-CPU LRU cache for large folios with fewer than FOLIO_BATCH_SIZE (31) pages. It also limits each per-CPU LRU cache to at most FOLIO_BATCH_SIZE pages to avoid negatively affecting accounting and memory reclamation pressure.

This is particularly beneficial on systems that use relatively small large folios. For larger folios, the benefit is likely to be smaller because far fewer folios are expected to contend for the LRU cache.

As raised, there is this problem with collect_longterm_unpinnable_folios()

(see
https://lore.kernel.org/r/[email protected]
)

whereby we don't know how many refs we actually hold. Certainly not 1.

Quoted 2 lines

We might have to wait for Hugh's cleanup to handle that cleanly (and avoid all the other LRU cache draining).

Hi David,
Thanks for raising this.
Yes, I saw your comment and took a closer look at it. I think the best approach for now is to leave that part untouched until Hugh's patch lands?

We might end up doing some extra draining in that case, but that's safe—any value greater than 1 might not be.

Quoted 21 lines · Barry Song

Quoted 19 lines · David Hildenbrand (Arm)

Quoted 8 lines · Barry Song (Xiaomi)

This patchset enables the per-CPU LRU cache for large folios with fewer than FOLIO_BATCH_SIZE (31) pages. It also limits each per-CPU LRU cache to at most FOLIO_BATCH_SIZE pages to avoid negatively affecting accounting and memory reclamation pressure.

This is particularly beneficial on systems that use relatively small large folios. For larger folios, the benefit is likely to be smaller because far fewer folios are expected to contend for the LRU cache.

As raised, there is this problem with collect_longterm_unpinnable_folios()

(see
https://lore.kernel.org/r/[email protected]
)

whereby we don't know how many refs we actually hold. Certainly not 1.

Quoted 8 lines

Quoted 2 lines

We might have to wait for Hugh's cleanup to handle that cleanly (and avoid all the other LRU cache draining).

Hi David,
Thanks for raising this.
Yes, I saw your comment and took a closer look at it. I think the best approach for now is to leave that part untouched until Hugh's patch lands?

I think we should not consider your patch set until Hugh either resolved it or we have a better way to handle draining. Building on top of this for large folios now just gets painful.

Quoted 3 lines

We might end up doing some extra draining in that case, but that's safe—any value greater than 1 might not be.

We'll unconditionally drain all LRU caches, which is precisely *not* what we want to do in the first place. :)

This patchset enables the per-CPU LRU cache for large folios with fewer than FOLIO_BATCH_SIZE (31) pages. It also limits each per-CPU LRU cache to at most FOLIO_BATCH_SIZE pages to avoid negatively affecting accounting and memory reclamation pressure.

This is particularly beneficial on systems that use relatively small large folios. For larger folios, the benefit is likely to be smaller because far fewer folios are expected to contend for the LRU cache.

* Use the following microbenchmark:

#include <pthread.h>
#include <sys/mman.h>
#include <string.h>

 #define NUM_THREADS     20
 #define MEM_SIZE        (16 * 1024 * 1024)
 #define LOOP_COUNT      1000

 void* thread_worker(void* arg) {
        void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
                        MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

        for (int i = 0; i < LOOP_COUNT; i++) {
                for(int j = 0; j < MEM_SIZE; j += 4096)
                        *(unsigned char *)(addr + j) = 0x55;
                madvise(addr, MEM_SIZE, MADV_DONTNEED);
        }

        munmap(addr, MEM_SIZE);
        pthread_exit(NULL);

}

 int main() {
        pthread_t threads[NUM_THREADS];

        for (long t = 0; t < NUM_THREADS; t++) {
                pthread_create(&threads[t], NULL, thread_worker, (void*)t);
        }

        for (int t = 0; t < NUM_THREADS; t++) {
                pthread_join(threads[t], NULL);
        }

        return 0;

}

w/o patch:

$ time ./a.out 

real 0m17.760s
user 0m8.207s
sys 5m46.067s

perf lock report:

                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                       21414857   21414857     30.07 us     10.73 m     101.94 us       880 ns 
                          42890      42890     11.29 us    484.21 ms    123.08 us       923 ns 
                           1678       1678      1.57 us      2.63 ms      5.77 us       985 ns 
                             52         52      1.46 us     75.98 us      2.69 us      1.07 us 
                             18         18     17.82 us    320.69 us     58.92 us      1.62 us 
                             18         18      2.36 ms     42.52 ms      5.31 ms      1.47 us 
           rcu_state         12         12      1.98 us     23.78 us      2.62 us      1.47 us 
           rcu_state          9          9      1.72 us     15.49 us      2.27 us      1.34 us 
                              2          2      2.42 us      4.83 us      2.48 us      2.36 us 

w/ patch:

$ time ./a.out 

real 0m16.292s
user 0m8.587s
sys 5m13.787s

perf lock report

                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                        2641286    2641286     46.87 us      2.06 m     107.92 us      1.01 us 
                         235275     235275     10.57 us      2.49 s     115.32 us      1.02 us 
           rcu_state       1982       1982      7.58 us     15.02 ms     31.72 us      1.26 us 
           rcu_state       1929       1929      7.42 us     14.32 ms     30.50 us      1.43 us 
                             86         86      3.61 us    310.19 us     13.89 us      1.15 us 
                             20         20     19.70 us    394.03 us    108.07 us      2.15 us 
       tasklist_lock          1          1      2.13 us      2.13 us      2.13 us      2.13 us 

* Build the kernel in a 1 GiB memcg by -j20 with zRAM configured as swap:

w/o patch:

Perf lock report:

                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                         782337     782337     17.61 us     13.78 s     402.22 us       896 ns 
                          55459      55459     19.48 us      1.08 s     117.59 us      1.01 us 
                           7826       7826      8.01 us     62.68 ms     19.03 us       887 ns 
                           5324       5324      7.83 us     41.68 ms     37.59 us      1.11 us 
           rcu_state       5144       5144      6.27 us     32.23 ms     25.17 us      1.58 us 
           rcu_state       5142       5142      6.35 us     32.67 ms     30.55 us      1.48 us 
                           3855       3855      8.74 us     33.68 ms     42.32 us       996 ns 
                           2770       2770      9.22 us     25.55 ms     27.83 us       914 ns 
                           2342       2342      5.88 us     13.77 ms    318.75 us      1.00 us 

time:

*** Executing round 0 ***

real    1m46.847s
user    25m10.848s
sys     2m57.282s

*** Executing round 1 ***

real    1m46.423s
user    25m10.072s
sys     2m54.348s

*** Executing round 2 ***

real    1m46.308s
user    25m13.800s
sys     2m58.963s

*** Executing round 3 ***

real    1m46.155s
user    25m18.079s
sys     2m59.721s

*** Executing round 4 ***

real    1m45.980s
user    25m15.493s
sys     2m56.959s

w/ patch:

perf lock report

                Name   acquired  contended     avg wait   total wait     max wait     min wait 

                         202647     202647     34.27 us      6.94 s     467.82 us      1.18 us 
                          51819      51819     16.26 us    842.46 ms    245.55 us       885 ns 
     inode_hash_lock      15169      15169      8.58 us    130.21 ms     23.23 us      1.04 us 
                           5306       5306      7.54 us     40.03 ms     31.53 us      1.17 us 
           rcu_state       4945       4945      6.97 us     34.47 ms     30.00 us      1.76 us 
           rcu_state       4899       4899      7.03 us     34.42 ms     24.65 us      1.42 us 
                           3923       3923      8.56 us     33.57 ms     27.51 us      1.08 us 
                           2212       2212      5.23 us     11.56 ms    222.79 us       965 ns 
                           1412       1412      6.35 us      8.97 ms     23.09 us      1.77 us 

time:

*** Executing round 0 ***

real 1m46.463s
user 25m17.448s
sys 2m49.524s

*** Executing round 1 ***

real 1m46.274s
user 25m12.178s
sys 2m53.522s

*** Executing round 2 ***

real 1m46.362s
user 25m13.115s
sys 2m53.005s

*** Executing round 3 ***

real 1m46.036s
user 25m17.627s
sys 2m53.477s

*** Executing round 4 ***

real 1m46.329s
user 25m15.130s
sys 2m51.508s

(quoted diff) +0 -2

-RFC v3:
  * Rather than hard-coding `< COSTLY_ORDER` to enable the lru_cache,
    allow the lru_cache for larger orders as long as the folio contains
    fewer than `FOLIO_BATCH_LRU` pages. Also limit the total number of
    pages held in the lru_cache. Hugh may prefer this approach as well.
  * Clean up the comments and `if` conditions in "mm: improve large folio
    reuse for LRU-cached folios" based on David's feedback. Thanks!
  * Properly drain the lru_cache when splitting folios. Thanks to Sashiko
    and David!
 
-RFC v2:
 * Make __wp_can_reuse_large_anon_folio() aware of LRU-cached large
   folios. As David pointed out, it currently does not account for
   large folios residing in the per-CPU LRU cache.
   https://lore.kernel.org/all/[email protected]/

Barry Song (Xiaomi) (4):
mm: allow smaller large folios to use lru_cache
mm: improve large folio reuse for LRU-cached folios
mm: drain LRU cache if necessary for splitting large folios
mm: batch lru_cache draining in deferred_split_scan

 include/linux/folio_batch.h | 25 +++++++++++++++++++++++++
 mm/folio.c                  | 10 +++++++++-
 mm/huge_memory.c            | 23 +++++++++++++++++++----
 mm/internal.h               |  4 ++--
 mm/memory.c                 |  6 ++++++
 5 files changed, 61 insertions(+), 7 deletions(-)

Thanks for this series.
I tested this against a Graph500 regression with 16K mTHP.
The series removes the regression and reduces folio_lruvec_lock_irqsave contention to approximately the 4K rate.

Tested-by Shivank Garg <[email protected]>

System and Workload
===================

AMD EPYC Zen 5 System, 2 sockets, 160 cores/socket, SMT on -> 320 cores / 640 threads
2 NUMA nodes, ~512 GB/node
node 0 CPUs: 0-159,320-479

Base: v7.3-rc2+ (50d05c7c76c9)
Runtime: performance governor, preempt=full (lazy)

Workload: Graph500, Scale 27, edgefactor 16
Metric: median_time in ms (lower is better)

Bind to node 0 with numactl -C 0-159,320-479 -m 0

Performance
===========

                  Base       Patched    delta
4K                109.60     109.11     -0.4%

mTHP-16K 144.10 99.00 -31.3%
mTHP-64K 104.75 104.25 -0.5%

mTHP-16K was ~31% slower than THP-never, due to lock contentions. With your series, mTHP-16 performs ~9% faster than THP-never.

The dominant folio_lruvec_lock_irqsave entries were:

Base Patched
contended wait contended wait

4K               834603   2.41 m         844193   2.61 m
mTHP-16K        7008834   28.88 m        840695   4.87 m
mTHP-64K         188144   752 ms         40874    93.46 ms

On Fri, Sep 11, 2026 at 11:39 PM Garg, Shivank <[email protected]> wrote:
[...]

Quoted 8 lines

Thanks for this series.
I tested this against a Graph500 regression with 16K mTHP.
The series removes the regression and reduces folio_lruvec_lock_irqsave contention to approximately the 4K rate.

Tested-by Shivank Garg <[email protected]>

Thanks very much, Shivank. I’m very happy to see that mTHP LRUCACHE not only resolves your regression but also improves the performance of your case.

As discussed with David, this series will be respun after Hugh’s work[1] is merged.

In the meantime, I’d really appreciate it if you could test an updated version that I haven’t sent out before:

https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=mthp_lrucache

It includes some modifications based on Sashiko’s review comments on the version you tested. I’d like to know whether it still provides the same performance gains you saw with that version.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 39 lines

System and Workload
===================

AMD EPYC Zen 5 System, 2 sockets, 160 cores/socket, SMT on -> 320 cores / 640 threads
2 NUMA nodes, ~512 GB/node
node 0 CPUs: 0-159,320-479

Base: v7.3-rc2+ (50d05c7c76c9)
Runtime: performance governor, preempt=full (lazy)

Workload: Graph500, Scale 27, edgefactor 16
Metric: median_time in ms (lower is better)

Bind to node 0 with numactl -C 0-159,320-479 -m 0

Performance
===========

                  Base       Patched    delta
4K                109.60     109.11     -0.4%

mTHP-16K 144.10 99.00 -31.3%
mTHP-64K 104.75 104.25 -0.5%

mTHP-16K was ~31% slower than THP-never, due to lock contentions. With your series, mTHP-16 performs ~9% faster than THP-never.

The dominant folio_lruvec_lock_irqsave entries were:

Base Patched
contended wait contended wait

4K               834603   2.41 m         844193   2.61 m
mTHP-16K        7008834   28.88 m        840695   4.87 m
mTHP-64K         188144   752 ms         40874    93.46 ms