mm: allow smaller large folios to use lru_cache
Subsystems (3)
Barry Song (Xiaomi)
reviewerRFC PATCH 1/1 mm: allow smaller large folios to use lru_cache
For a system which primarily uses smaller orders for large folios, it could be beneficial to enable lru_cache to avoid lock contention.
For large folios with higher orders, the number of folios involved would be small, so the contention is less significant.
The following small program runs on a system with 16 KiB mTHP enabled.
#include <pthread.h>
#include <sys/mman.h>
#include <string.h>
#define NUM_THREADS 20 #define MEM_SIZE (16 * 1024 * 1024) #define LOOP_COUNT 1000
void* thread_worker(void* arg) {
void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
for (int i = 0; i < LOOP_COUNT; i++) {
memset(addr, 0x55, MEM_SIZE);
madvise(addr, MEM_SIZE, MADV_DONTNEED);
}
munmap(addr, MEM_SIZE);
pthread_exit(NULL);}
int main() {
pthread_t threads[NUM_THREADS];
for (long t = 0; t < NUM_THREADS; t++) {
pthread_create(&threads[t], NULL, thread_worker, (void*)t);
}
for (int t = 0; t < NUM_THREADS; t++) {
pthread_join(threads[t], NULL);
}
return 0;}
Before patch:
root@barry-desktop:/home/barry# time ./a.out
real 0m20.233s
user 0m21.739s
sys 6m21.143s
Perf lock report:
Name acquired contended avg wait total wait max wait min wait
21158662 21158662 29.20 us 10.30 m 67.83 us 884 ns
13547 13547 14.68 us 198.93 ms 135.39 us 982 ns
rcu_state 41 41 1.83 us 75.21 us 3.46 us 1.27 us
rcu_state 34 34 1.83 us 62.09 us 2.60 us 1.16 us
5 5 21.31 us 106.57 us 25.77 us 18.17 us
5 5 3.01 us 15.07 us 5.19 us 1.53 us
3 3 6.42 us 19.26 us 9.63 us 4.22 us
3 3 4.20 us 12.61 us 7.45 us 2.10 us
2 2 5.03 us 10.06 us 7.88 us 2.18 us
2 2 2.79 us 5.59 us 3.34 us 2.25 us
1 1 4.70 us 4.70 us 4.70 us 4.70 us
1 1 9.77 us 9.77 us 9.77 us 9.77 usAfter patch:
root@barry-desktop:/home/barry# time ./a.out
real 0m17.796s
user 0m24.962s
sys 5m28.774s
Perf lock report:
Name acquired contended avg wait total wait max wait min wait
1407013 1407013 25.44 us 35.80 s 127.83 us 972 ns
845113 845113 47.96 us 40.54 s 152.86 us 1.21 us
rcu_state 1920 1920 5.87 us 11.27 ms 25.02 us 1.76 us
rcu_state 1889 1889 5.80 us 10.96 ms 30.35 us 1.61 us
140 140 3.97 us 555.52 us 12.51 us 1.20 us
10 10 61.34 us 613.42 us 96.90 us 44.48 us
8 8 24.03 us 192.26 us 37.64 us 6.95 us
8 8 12.32 us 98.56 us 25.55 us 4.16 usBoth system time and lock contention are noticeably reduced.
--- include/linux/swap.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/swap.h +2 -2
@@ -322,9 +322,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)/* * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting. * Holding small numbers of low-order mTHP folios in per-CPU LRU cache -* will be sensible, but nobody has implemented and tested that yet. +* will be sensible. */ -return !folio_test_large (folio); +return folio_order (folio) <PAGE_ALLOC_COSTLY_ORDER ; }extern atomic_t lru_disable_count;
Signature · 1 line
2.39.3 (Apple Git-146)
David Hildenbrand (Arm)
maintainerQuoted 108 lines · Barry Song (Xiaomi)
For a system which primarily uses smaller orders for large folios, it could be beneficial to enable
lru_cacheto avoid lock contention.For large folios with higher orders, the number of folios involved would be small, so the contention is less significant.
The following small program runs on a system with 16 KiB mTHP enabled.
#include <
pthread.h>
#include <sys/mman.h>
#include <string.h>#define NUM_THREADS 20 #define MEM_SIZE (16 * 1024 * 1024) #define LOOP_COUNT 1000void* thread_worker(void* arg) { void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); for (int i = 0; i < LOOP_COUNT; i++) { memset(addr, 0x55, MEM_SIZE); madvise(addr, MEM_SIZE, MADV_DONTNEED); } munmap(addr, MEM_SIZE); pthread_exit(NULL);}
int main() { pthread_t threads[NUM_THREADS]; for (long t = 0; t < NUM_THREADS; t++) { pthread_create(&threads[t], NULL, thread_worker, (void*)t); } for (int t = 0; t < NUM_THREADS; t++) { pthread_join(threads[t], NULL); } return 0;}
Before patch:
root@barry-desktop:/home/barry# time ./a.outreal 0m20.233s
user 0m21.739s
sys 6m21.143sPerf lock report:
Name acquired contended avg wait total wait max wait min wait 21158662 21158662 29.20 us 10.30 m 67.83 us 884 ns 13547 13547 14.68 us 198.93 ms 135.39 us 982 ns rcu_state 41 41 1.83 us 75.21 us 3.46 us 1.27 us rcu_state 34 34 1.83 us 62.09 us 2.60 us 1.16 us 5 5 21.31 us 106.57 us 25.77 us 18.17 us 5 5 3.01 us 15.07 us 5.19 us 1.53 us 3 3 6.42 us 19.26 us 9.63 us 4.22 us 3 3 4.20 us 12.61 us 7.45 us 2.10 us 2 2 5.03 us 10.06 us 7.88 us 2.18 us 2 2 2.79 us 5.59 us 3.34 us 2.25 us 1 1 4.70 us 4.70 us 4.70 us 4.70 us 1 1 9.77 us 9.77 us 9.77 us 9.77 usAfter patch:
root@barry-desktop:/home/barry# time ./a.outreal 0m17.796s
user 0m24.962s
sys 5m28.774sPerf lock report:
Name acquired contended avg wait total wait max wait min wait 1407013 1407013 25.44 us 35.80 s 127.83 us 972 ns 845113 845113 47.96 us 40.54 s 152.86 us 1.21 us rcu_state 1920 1920 5.87 us 11.27 ms 25.02 us 1.76 us rcu_state 1889 1889 5.80 us 10.96 ms 30.35 us 1.61 us 140 140 3.97 us 555.52 us 12.51 us 1.20 us 10 10 61.34 us 613.42 us 96.90 us 44.48 us 8 8 24.03 us 192.26 us 37.64 us 6.95 us 8 8 12.32 us 98.56 us 25.55 us 4.16 usBoth system time and lock contention are noticeably reduced.
Cc David Hildenbrand <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Zi Yan <[email protected]>Cc Baolin Wang <[email protected]>Cc Liam R. Howlett <[email protected]>Cc Nico Pache <[email protected]>Cc Ryan Roberts <[email protected]>Cc Dev Jain <[email protected]>Cc Lance Yang <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>--- include/linux/swap.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)include/linux/swap.h +0
-0@@ -322,9 +322,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)/*
include/linux/swap.h static inline bool folio_may_be_lru_cached(struct folio *folio) · line 330
Quoted 10 lines
include/linux/swap.h +2
-2* Holding PMD-sized folios in per-CPU LRU cache unbalances accounting. * Holding small numbers of low-order mTHP folios in per-CPU LRU cache -* will be sensible, but nobody has implemented and tested that yet. +* will be sensible. */ -return !folio_test_large (folio); +return folio_order (folio) <PAGE_ALLOC_COSTLY_ORDER ; }extern atomic_t lru_disable_count;
Not that __wp_can_reuse_large_anon_folio() currently assumes that large folios are not in the LRU cache.
Signature · 3 lines
Cheers, David
Barry Song
reviewerOn Thu, Jul 2, 2026 at 9:00 PM David Hildenbrand (Arm) <[email protected]> wrote:
[...]
Quoted 15 lines
Quoted 15 lines
include/linux/swap.h +2
-2@@ -322,9 +322,9 @@ static inline bool folio_may_be_lru_cached(struct folio *folio)/* * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting. * Holding small numbers of low-order mTHP folios in per-CPU LRU cache -* will be sensible, but nobody has implemented and tested that yet. +* will be sensible. */ -return !folio_test_large (folio); +return folio_order (folio) <PAGE_ALLOC_COSTLY_ORDER ; }
Quoted 4 lines
include/linux/swap.h static inline bool folio_may_be_lru_cached(struct folio *folio) · line 330
Quoted 1 line
include/linux/swap.h +0
-0extern atomic_t lru_disable_count;Not that
__wp_can_reuse_large_anon_folio()currently assumes that large folios are not in the LRU cache.
Thanks for pointing this out. I assume we could do something similar to wp_can_reuse_anon_folio(). It won't always help drain lru_cache, but it could help in some cases?
mm/memory.c +0 -0
@@ -4143,6 +4143,12 @@ static bool
__wp_can_reuse_large_anon_folio(struct folio *folio,
folio_unlock(folio);
}+ if (folio_may_be_lru_cached(folio) && !folio_test_lru(folio)) {
+ if (folio_ref_count(folio) > folio_large_mapcount(folio) + 1)
+ return false;
+ lru_add_drain();
+ }
+
if (folio_large_mapcount(folio) != folio_ref_count(folio))
return false;Best Regards
Barry