RFC mm: enable lru cache for smaller large folios
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v3 RFC Latest4 patches 12 repliesIn review
- v2 RFC2 patches 7 repliesSuperseded
- v1 RFC1 patch 2 repliesSuperseded
Patches in v3 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/4 | mm: allow smaller large folios to use lru_cacheinclude/ mm/ mm/ | +36 | 1 reply | |
| 2/4 | mm: improve large folio reuse for LRU-cached foliosmm/ | +6 | 0 replies | |
| 3/4 | mm: drain LRU cache if necessary for splitting large foliosmm/ | +10 | 1 reply | |
| 4/4 | mm: batch lru_cache draining in deferred_split_scanmm/ | +10 | 0 replies |
Reviews trailers per patch
None of the 4 patches has a Reviewed-by or Acked-by.
| Patch | Shivank Garg |
|---|---|
| Series | T-b |
| 1/4 | |
| 2/4 | |
| 3/4 | |
| 4/4 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [RFC PATCH v3 0/4] mm: enable lru cache for smaller large folios
This patchset enables the per-CPU LRU cache for large folios with fewer than FOLIO_BATCH_SIZE (31) pages. It also limits each per-CPU LRU cache to at most FOLIO_BATCH_SIZE pages to avoid negatively affecting accounting and memory reclamation pressure.
This is particularly beneficial on systems that use relatively small large folios. For larger folios, the benefit is likely to be smaller because far fewer folios are expected to contend for the LRU cache.
* Use the following microbenchmark:
#include <pthread.h>
#include <sys/mman.h>
#include <string.h>
#define NUM_THREADS 20 #define MEM_SIZE (16 * 1024 * 1024) #define LOOP_COUNT 1000
void* thread_worker(void* arg) {
void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
for (int i = 0; i < LOOP_COUNT; i++) {
for(int j = 0; j < MEM_SIZE; j += 4096)
*(unsigned char *)(addr + j) = 0x55;
madvise(addr, MEM_SIZE, MADV_DONTNEED);
}
munmap(addr, MEM_SIZE);
pthread_exit(NULL);}
int main() {
pthread_t threads[NUM_THREADS];
for (long t = 0; t < NUM_THREADS; t++) {
pthread_create(&threads[t], NULL, thread_worker, (void*)t);
}
for (int t = 0; t < NUM_THREADS; t++) {
pthread_join(threads[t], NULL);
}
return 0;}
w/o patch:
$ time ./a.out
real 0m17.760s
user 0m8.207s
sys 5m46.067s
perf lock report:
Name acquired contended avg wait total wait max wait min wait
21414857 21414857 30.07 us 10.73 m 101.94 us 880 ns
42890 42890 11.29 us 484.21 ms 123.08 us 923 ns
1678 1678 1.57 us 2.63 ms 5.77 us 985 ns
52 52 1.46 us 75.98 us 2.69 us 1.07 us
18 18 17.82 us 320.69 us 58.92 us 1.62 us
18 18 2.36 ms 42.52 ms 5.31 ms 1.47 us
rcu_state 12 12 1.98 us 23.78 us 2.62 us 1.47 us
rcu_state 9 9 1.72 us 15.49 us 2.27 us 1.34 us
2 2 2.42 us 4.83 us 2.48 us 2.36 us w/ patch:
$ time ./a.out
real 0m16.292s
user 0m8.587s
sys 5m13.787s
perf lock report
Name acquired contended avg wait total wait max wait min wait
2641286 2641286 46.87 us 2.06 m 107.92 us 1.01 us
235275 235275 10.57 us 2.49 s 115.32 us 1.02 us
rcu_state 1982 1982 7.58 us 15.02 ms 31.72 us 1.26 us
rcu_state 1929 1929 7.42 us 14.32 ms 30.50 us 1.43 us
86 86 3.61 us 310.19 us 13.89 us 1.15 us
20 20 19.70 us 394.03 us 108.07 us 2.15 us
tasklist_lock 1 1 2.13 us 2.13 us 2.13 us 2.13 us * Build the kernel in a 1 GiB memcg by -j20 with zRAM configured as swap:
w/o patch:
Perf lock report:
Name acquired contended avg wait total wait max wait min wait
782337 782337 17.61 us 13.78 s 402.22 us 896 ns
55459 55459 19.48 us 1.08 s 117.59 us 1.01 us
7826 7826 8.01 us 62.68 ms 19.03 us 887 ns
5324 5324 7.83 us 41.68 ms 37.59 us 1.11 us
rcu_state 5144 5144 6.27 us 32.23 ms 25.17 us 1.58 us
rcu_state 5142 5142 6.35 us 32.67 ms 30.55 us 1.48 us
3855 3855 8.74 us 33.68 ms 42.32 us 996 ns
2770 2770 9.22 us 25.55 ms 27.83 us 914 ns
2342 2342 5.88 us 13.77 ms 318.75 us 1.00 us time:
*** Executing round 0 ***
real 1m46.847s user 25m10.848s sys 2m57.282s
*** Executing round 1 ***
real 1m46.423s user 25m10.072s sys 2m54.348s
*** Executing round 2 ***
real 1m46.308s user 25m13.800s sys 2m58.963s
*** Executing round 3 ***
real 1m46.155s user 25m18.079s sys 2m59.721s
*** Executing round 4 ***
real 1m45.980s user 25m15.493s sys 2m56.959s
w/ patch:
perf lock report
Name acquired contended avg wait total wait max wait min wait
202647 202647 34.27 us 6.94 s 467.82 us 1.18 us
51819 51819 16.26 us 842.46 ms 245.55 us 885 ns
inode_hash_lock 15169 15169 8.58 us 130.21 ms 23.23 us 1.04 us
5306 5306 7.54 us 40.03 ms 31.53 us 1.17 us
rcu_state 4945 4945 6.97 us 34.47 ms 30.00 us 1.76 us
rcu_state 4899 4899 7.03 us 34.42 ms 24.65 us 1.42 us
3923 3923 8.56 us 33.57 ms 27.51 us 1.08 us
2212 2212 5.23 us 11.56 ms 222.79 us 965 ns
1412 1412 6.35 us 8.97 ms 23.09 us 1.77 us time:
*** Executing round 0 ***
real 1m46.463s
user 25m17.448s
sys 2m49.524s
*** Executing round 1 ***
real 1m46.274s
user 25m12.178s
sys 2m53.522s
*** Executing round 2 ***
real 1m46.362s
user 25m13.115s
sys 2m53.005s
*** Executing round 3 ***
real 1m46.036s
user 25m17.627s
sys 2m53.477s
*** Executing round 4 ***
real 1m46.329s
user 25m15.130s
sys 2m51.508s
-RFC v3:
* Rather than hard-coding < COSTLY_ORDER to enable the lru_cache, allow the lru_cache for larger orders as long as the folio contains fewer than FOLIO_BATCH_LRU pages. Also limit the total number of pages held in the lru_cache. Hugh may prefer this approach as well.
* Clean up the comments and if conditions in "mm: improve large folio reuse for LRU-cached folios" based on David's feedback. Thanks!
* Properly drain the lru_cache when splitting folios. Thanks to Sashiko and David!
-RFC v2:
* Make __wp_can_reuse_large_anon_folio() aware of LRU-cached large folios. As David pointed out, it currently does not account for large folios residing in the per-CPU LRU cache.
https://lore.kernel.org/all/[email protected]/
Barry Song (Xiaomi) (4):
mm: allow smaller large folios to use lru_cache
mm: improve large folio reuse for LRU-cached folios
mm: drain LRU cache if necessary for splitting large folios
mm: batch lru_cache draining in deferred_split_scan
include/linux/folio_batch.h | 25 +++++++++++++++++++++++++ mm/folio.c | 10 +++++++++- mm/huge_memory.c | 23 +++++++++++++++++++---- mm/internal.h | 4 ++-- mm/memory.c | 6 ++++++ 5 files changed, 61 insertions(+), 7 deletions(-)
Signature · 1 line
2.34.1