Refault distance checking for MGLRU
Subsystems (5)
cover Refault distance checking for MGLRU
From: Kairui Song <[email protected]>
Hi, linux-mm
I noticed MGLRU not working very well on certain workflows, which is observed on some instances on some heavily stressed machines.
I found this was related to refault distance detection, when the file page workingset size exceeds total memory, and the access distance (the left-shift time of a page before it gets activated, considering LRU starts from right) of file pages also larger than total memory. All file pages are stuck on the oldest generation and getting read-in then evicted permutably, few get activated and stay in memory.
This series tries to fix this problem by rework the refault distance detection to better fit MGLRU, and also tries to use a unified algorithm for both MGLRU and Inactive/Active LRU.
Patch 1/4 reworked the refault distance detection model for Inactive/Active LRU.
Patch 2/4 and 3/4 are simplification and prepare.
Patch 4/4 applies the modified refault distance detection for MGLRU.
Following benchmark showed 5x improvement:
To simulate the workflow, I setup a 3-replicated mongodb cluster using docker, each in a standalone cgroup, set to use 5 gb of cache and 10g of oplog, on a 32G VM. The benchmark is done using
https://github.com/apavlo/py-tpcc.git, modified to run STOCK_LEVEL query only, for simulating slow query and get a stable result.
Before the patch (with 10G swap, the result won't change whether swap is on or not):
$ tpcc.py --config=mongodb.config mongodb --duration=900 --warehouses=500 --clients=30
==================================================================
Execution Results after 904 seconds
------------------------------------------------------------------
Executed Time (µs) Rate
STOCK_LEVEL 503 27150226136.4 0.02 txn/s
------------------------------------------------------------------
TOTAL 503 27150226136.4 0.02 txn/s$ cat /proc/vmstat | grep working workingset_nodes 53391 workingset_refault_anon 0 workingset_refault_file 23856735 workingset_activate_anon 0 workingset_activate_file 23845737 workingset_restore_anon 0 workingset_restore_file 18280692 workingset_nodereclaim 1024
$ free -m
total used free shared buff/cache available
Mem: 31837 6752 379 23 24706 24607
Swap: 10239 0 10239After the patch (with 10G swap on same disk, similar result using ZRAM):
$ tpcc.py --config=mongodb.config mongodb --duration=900 --warehouses=500 --clients=30
==================================================================
Execution Results after 903 seconds
------------------------------------------------------------------
Executed Time (µs) Rate
STOCK_LEVEL 2575 27094953498.8 0.10 txn/s
------------------------------------------------------------------
TOTAL 2575 27094953498.8 0.10 txn/s$ cat /proc/vmstat | grep working workingset_nodes 78249 workingset_refault_anon 10139 workingset_refault_file 23001863 workingset_activate_anon 7238 workingset_activate_file 6718032 workingset_restore_anon 7432 workingset_restore_file 6719406 workingset_nodereclaim 9747
$ free -m
total used free shared buff/cache available
Mem: 31837 7376 320 3 24140 24014
Swap: 10239 1662 8577The performance is 5x times better than before, and the idle anon pages now can get swapped out as expected. Testing with lower stress also shows a improvement.
I also checked the benchmark with memtier/memcached and fio, using similar setup as in commit ac35a4902374 but scaled down to fit in my test environment:
memtier test (with 16G ramdisk as swap and 2G cgroup limit):
memcached -u nobody -m 16384 -s /tmp/memcached.socket -a 0766 \
-t 12 -B binary &
memtier_benchmark -S /tmp/memcached.socket -P memcache_binary -n allkeys\
--key-minimum=1 --key-maximum=24000000 --key-pattern=P:P -c 1 \
-t 12 --ratio 1:0 --pipeline 8 -d 2000 -x 6
fio test (with 16G ramdisk on /mnt and 4G cgroup limit):
fio -name=refault --numjobs=12 --directory=/mnt --size=1024m \
--buffered=1 --ioengine=io_uring --iodepth=128 \
--iodepth_batch_submit=32 --iodepth_batch_complete=32 \
--rw=randread --random_distribution=random --norandommap \
--time_based --ramp_time=5m --runtime=5m --group_reportingBefore this patch:
memcached:
Ops/sec Hits/sec Misses/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
Best 52832.79 0.00 0.00 1.82042 1.70300 4.54300 6.27100 105641.69 Worst 46613.56 0.00 0.00 2.05686 1.77500 7.80700 11.83900 93206.05 Avg (6x) 51024.85 0.00 0.00 1.88506 1.73500 5.43900 9.47100 102026.64
fio:
read: IOPS=2211k, BW=8637MiB/s (9056MB/s)(2530GiB/300001msec)
After this patch:
memcached:
Ops/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
Best 54218.92 1.76930 1.65500 4.41500 6.27100 108413.34 Worst 47640.13 2.01495 1.74300 7.64700 11.64700 95258.72 Avg (6x) 51408.33 1.86988 1.71900 5.43900 9.34300 102793.42
fio:
read: IOPS=2166k, BW=8462MiB/s (8873MB/s)(2479GiB/300001msec)
memcached looks ok but there is a %2 performance drop for FIO test, and after some profiling this is mainly caused by the extra atomic operations and new functions, there seems to be no LRU accuracy drop.
Sending this as RFC as I'm not entirely sure if this is the right way to fix this issue, of if this is a generic issue or considered more of a misconfiguration. Any suggetions about how should I test it is welcomed.
Kairui Song (4):
workingset: simplify and use a more intuitive model
workingset: simplify lru_gen_test_recent
lru_gen: convert avg_total and avg_refaulted to atomic workingset, lru_gen: apply refault-distance based re-activation
include/linux/mmzone.h | 4 +- include/linux/swap.h | 2 - mm/swap.c | 1 - mm/vmscan.c | 18 ++- mm/workingset.c | 315 ++++++++++++++++++++++------------------- 5 files changed, 179 insertions(+), 161 deletions(-)
Signature · 1 line
2.41.0
RFC PATCH 1/4 workingset: simplify and use a more intuitive model
From: Kairui Song <[email protected]>
This basically removed workingset_activation and reduced calls to workingset_age_nonresident.
The idea behind this change is a new way to calculate the refault distance which seems working fine in most cases and fits for MGLRU (in later commits).
Current refault distance is based on two assumptions:
1. Activation of an inactive page will left-shift LRU pages (consider LRU starts from right).
2. Eviction of an inactive page will left-shift LRU pages.
Assumption 2 is correct, but assumption 1 is not always true, the activated page could be anywhere in the LRU list, it only left-shift the pages on its right. And one page can get activate/deactivated for multiple times.
And MGLRU doesn't fit with this model, since there are multiple gens, and pages are getting aged and activated constantly upon gen growth.
So instead we introduce a new idea here, "Shadow LRU Position". Simply consider the evicted pages are still in memory, each has an eviction sequence like before. Let the nonresistence_age be NA and get increased for each eviction, so the "Shadow LRU Position" of one evicted page will be:
SP = ((NA's reading @ eviction) - (NA's reading @ current))
+---------------------------------------+==========+========+
| * Shadow LRU O O O | INACTIVE | ACTIVE |
+----+----------------------------------+==========+========+
| |
+----------------------------------+
| SP
refault page O -> Hole left by previously refaulted page* -> The page corresponding to SP
Now since SP simply stands for how much currently workflow could push a page out of current memory, which also means if the page started on INACTIVE part, it *may* get re-activated if it right shift SP slots into the ACTIVE list and still doesn't go exceed total memory, which is:
SP + NR_INACTIVE < NR_INACTIVE + NR_ACTIVE
Which can be simplified to:
SP < NR_ACTIVE
And since this is only an estimation, based on several hypothesis and it actually violates the normal routine of LRU when LRU is working well, so throttle this by two factor:
1. Previously refaulted pages may leave "holes" on the shadow LRU and decrease re-active rate for distant shadow pages.
2. When the ACTIVE part of LRU is long enough, chanllaging them by activating one-time faulted inactive page may not be a good idea so throttle it by the ratio of ACTIVE/INACTIVE.
Combined all above, we have:
Upon refault:
- If ACTIVE LRU is low, check if SP < NR_ACTIVE to check for re-activation.
- If ACTIVE LRU is high, check if
SP < min(NR_ACTIVE, NR_INACTIVE) / (exponential ratio of ACTIVE / INACTIVE).
This is simpler than before since no longer need to do lruvec operations when activating a page, and so far, a few benchmarks shows a fair result.
Using memtier and fio test from commit ac35a4902374 but scaled down to fit in my test environment, and some other test results:
memtier test (with 16G ramdisk as swap and 2G cgroup limit):
memcached -u nobody -m 16384 -s /tmp/memcached.socket \
-a 0700 -t 12 -B binary &
memtier_benchmark -S /tmp/memcached.socket -P memcache_binary -n allkeys\
--key-minimum=1 --key-maximum=24000000 --key-pattern=P:P -c 1 \
-t 12 --ratio 1:0 --pipeline 8 -d 2000 -x 6
fio test (with 16G ramdisk on /mnt and 4G cgroup limit):
fio -name=refault --numjobs=12 --directory=/mnt --size=1024m \
--buffered=1 --ioengine=io_uring --iodepth=128 \
--iodepth_batch_submit=32 --iodepth_batch_complete=32 \
--rw=randread --random_distribution=random --norandommap \
--time_based --ramp_time=5m --runtime=5m --group_reportingPgbench setup using phronix-test-suite with scale 1000 and 50 clients on a 5G VM.
Linux Kernel compliation test done with defconfig on a 2G VM.
Before:
memcached: 48157.04 ops/s
read: IOPS=2003k, BW=7823MiB/s (8203MB/s)(2292GiB/300001msec)
pgbench: 5845 qps
build-linux: 247.063
After:
memcached: 49144.55 ops/s
read: IOPS=2005k, BW=7832MiB/s (8212MB/s)(2294GiB/300002msec)
pgbench: 5832 qps
build-linux: 247.302
--- include/linux/swap.h | 2 - mm/swap.c | 1 - mm/vmscan.c | 2 - mm/workingset.c | 217 +++++++++++++++++++++---------------------- 4 files changed, 108 insertions(+), 114 deletions(-)
include/linux/swap.h +0 -2
@@ -350,10 +350,8 @@ static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry)/* linux/mm/workingset.c */ bool workingset_test_recent (void *shadow,bool file,bool *workingset); -void workingset_age_nonresident (struct lruvec *lruvec,unsigned long nr_pages);void *workingset_eviction (struct folio *folio,struct mem_cgroup *target_memcg);void workingset_refault (struct folio *folio,void *shadow); -void workingset_activation (struct folio *folio);/* Only track the nodes of mappings with shadow entries */ void workingset_update_node (struct xa_node *node);
mm/swap.c +0 -1
@@ -482,7 +482,6 @@ void folio_mark_accessed(struct folio *folio)else __lru_cache_activate_folio (folio);folio_clear_referenced (folio); -workingset_activation (folio); }if (folio_test_idle (folio))folio_clear_idle (folio);
mm/vmscan.c +0 -2
@@ -2539,8 +2539,6 @@ static unsigned int move_folios_to_lru(struct lruvec *lruvec,lruvec_add_folio (lruvec, folio); nr_pages =folio_nr_pages (folio); nr_moved += nr_pages; -if (folio_test_active (folio)) -workingset_age_nonresident (lruvec, nr_pages); }/*
mm/workingset.c +108 -109
292 lines not on this page · raw message
Signature · 1 line
2.41.0
RFC PATCH 2/4 workingset: simplify lru_gen_test_recent
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Simplify the code, move some common path into its caller, prepare for following commits.
--- mm/workingset.c | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-)
mm/workingset.c +13 -17
@@ -357,42 +357,38 @@ static void *lru_gen_eviction(struct folio *folio)* Tests if the shadow entry is for a folio that was recently evicted. * Fills in @lruvec, @token, @workingset with the values unpacked from shadow. */ -static bool lru_gen_test_recent (void *shadow,bool file,struct lruvec **lruvec, -unsigned long *token,bool *workingset) +static bool lru_gen_test_recent (struct lruvec *lruvec,bool file, +unsigned long token) { -int memcg_id;unsigned long min_seq; -struct mem_cgroup *memcg; -struct pglist_data *pgdat; -unpack_shadow (shadow, &memcg_id, &pgdat, token, workingset); - - memcg =mem_cgroup_from_id (memcg_id); - *lruvec =mem_cgroup_lruvec (memcg, pgdat); - - min_seq =READ_ONCE ((*lruvec)->lrugen.min_seq[file]); -return (*token >>LRU_REFS_WIDTH ) == (min_seq & (EVICTION_MASK >>LRU_REFS_WIDTH )); + min_seq =READ_ONCE (lruvec->lrugen.min_seq[file]); +return (token >>LRU_REFS_WIDTH ) == (min_seq & (EVICTION_MASK >>LRU_REFS_WIDTH )); }static void lru_gen_refault (struct folio *folio,void *shadow) { +int memcgid;bool recent; -int hist, tier, refs;bool workingset;unsigned long token; +int hist, tier, refs;struct lruvec *lruvec; +struct pglist_data *pgdat;struct lru_gen_folio *lrugen;int type =folio_is_file_lru (folio);int delta =folio_nr_pages (folio);rcu_read_lock (); - recent =lru_gen_test_recent (shadow, type, &lruvec, &token, &workingset); +unpack_shadow (shadow, &memcgid, &pgdat, &token, &workingset); + lruvec =mem_cgroup_lruvec (mem_cgroup_from_id (memcgid), pgdat);if (lruvec !=folio_lruvec (folio)) goto unlock;mod_lruvec_state (lruvec,WORKINGSET_REFAULT_BASE + type, delta); + recent =lru_gen_test_recent (lruvec, type, token);if (!recent) goto unlock;
@@ -492,9 +488,6 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset)struct pglist_data *pgdat;unsigned long eviction; -if (lru_gen_enabled ()) -return lru_gen_test_recent (shadow, file, &eviction_lruvec, &eviction, workingset); -unpack_shadow (shadow, &memcgid, &pgdat, &eviction, workingset);/*
@@ -518,6 +511,9 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset)return false; eviction_lruvec =mem_cgroup_lruvec (eviction_memcg, pgdat); +if (lru_gen_enabled ()) +return lru_gen_test_recent (eviction_lruvec, file, eviction); +return lru_refault (eviction_memcg, eviction_lruvec, eviction,EVICTION_BITS , bucket_order); }
Signature · 1 line
2.41.0
kernel test robot
build botRe: [RFC PATCH 2/4] workingset: simplify lru_gen_test_recent
kernel test robot noticed the following build errors:
Hi Kairui,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.5-rc3 next-20230725] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Kairui-Song/workingset-simplify-and-use-a-more-intuitive-model/20230726-025946 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20230725185733.43929-3-ryncsn%40gmail.com patch subject: [RFC PATCH 2/4] workingset: simplify lru_gen_test_recent config: microblaze-randconfig-r012-20230725 (https://download.01.org/0day-ci/archive/20230726/[email protected]/config) compiler: microblaze-linux-gcc (GCC) 12.3.0 reproduce: (https://download.01.org/0day-ci/archive/20230726/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All error/warnings (new ones prefixed by >>):
mm/workingset.c: In function 'workingset_test_recent':
>> mm/workingset.c:515:67: warning: passing argument 3 of 'lru_gen_test_recent' makes pointer from integer without a cast [-Wint-conversion]
515 | return lru_gen_test_recent(eviction_lruvec, file, eviction);
| ^~~~~~~~
| |
| long unsigned int
mm/workingset.c:427:74: note: expected 'struct lruvec **' but argument is of type 'long unsigned int'
427 | static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec,
| ~~~~~~~~~~~~~~~~^~~~~~
>> mm/workingset.c:515:24: error: too few arguments to function 'lru_gen_test_recent'
515 | return lru_gen_test_recent(eviction_lruvec, file, eviction);
| ^~~~~~~~~~~~~~~~~~~ mm/workingset.c:427:13: note: declared here
427 | static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec,
| ^~~~~~~~~~~~~~~~~~~
vim +/lru_gen_test_recent +515 mm/workingset.c
471
472 /**
473 * workingset_test_recent - tests if the shadow entry is for a folio that was
474 * recently evicted. Also fills in @workingset with the value unpacked from
475 * shadow.
476 * @shadow: the shadow entry to be tested.
477 * @file: whether the corresponding folio is from the file lru.
478 * @workingset: where the workingset value unpacked from shadow should
479 * be stored.
480 *
481 * Return: true if the shadow is for a recently evicted folio; false otherwise.
482 */
483 bool workingset_test_recent(void *shadow, bool file, bool *workingset)
484 {
485 struct mem_cgroup *eviction_memcg;
486 struct lruvec *eviction_lruvec;
487 int memcgid;
488 struct pglist_data *pgdat;
489 unsigned long eviction;
490
491 unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset);
492
493 /*
494 * Look up the memcg associated with the stored ID. It might
495 * have been deleted since the folio's eviction.
496 *
497 * Note that in rare events the ID could have been recycled
498 * for a new cgroup that refaults a shared folio. This is
499 * impossible to tell from the available data. However, this
500 * should be a rare and limited disturbance, and activations
501 * are always speculative anyway. Ultimately, it's the aging
502 * algorithm's job to shake out the minimum access frequency
503 * for the active cache.
504 *
505 * XXX: On !CONFIG_MEMCG, this will always return NULL; it
506 * would be better if the root_mem_cgroup existed in all
507 * configurations instead.
508 */
509 eviction_memcg = mem_cgroup_from_id(memcgid);
510 if (!mem_cgroup_disabled() && !eviction_memcg)
511 return false;
512 eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat);
513
514 if (lru_gen_enabled())
> 515 return lru_gen_test_recent(eviction_lruvec, file, eviction);
516
517 return lru_refault(eviction_memcg, eviction_lruvec, eviction,
518 EVICTION_BITS, bucket_order);
519 }
520
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RFC PATCH 3/4 lru_gen: convert avg_total and avg_refaulted to atomic
In reply to Kairui SongFrom: Kairui Song <[email protected]>
No feature change, prepare for later patch.
--- include/linux/mmzone.h | 4 ++-- mm/vmscan.c | 16 ++++++++-------- 2 files changed, 10 insertions(+), 10 deletions(-)
include/linux/mmzone.h +2 -2
@@ -425,9 +425,9 @@ struct lru_gen_folio {/* the multi-gen LRU sizes, eventually consistent */ long nr_pages[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ];/* the exponential moving average of refaulted */ -unsigned long avg_refaulted[ANON_AND_FILE ][MAX_NR_TIERS ]; +atomic_long_t avg_refaulted[ANON_AND_FILE ][MAX_NR_TIERS ];/* the exponential moving average of evicted+protected */ -unsigned long avg_total[ANON_AND_FILE ][MAX_NR_TIERS ]; +atomic_long_t avg_total[ANON_AND_FILE ][MAX_NR_TIERS ];/* the first tier doesn't need protection, hence the minus one */ unsigned long protected[NR_HIST_GENS ][ANON_AND_FILE ][MAX_NR_TIERS -1 ];/* can be modified without holding the LRU lock */
mm/vmscan.c +8 -8
@@ -3705,9 +3705,9 @@ static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int gain,struct lru_gen_folio *lrugen = &lruvec->lrugen;int hist =lru_hist_from_seq (lrugen->min_seq[type]); - pos->refaulted = lrugen->avg_refaulted[type][tier] + + pos->refaulted =atomic_long_read (&lrugen->avg_refaulted[type][tier]) +atomic_long_read (&lrugen->refaulted[hist][type][tier]); - pos->total = lrugen->avg_total[type][tier] + + pos->total =atomic_long_read (&lrugen->avg_total[type][tier]) +atomic_long_read (&lrugen->evicted[hist][type][tier]);if (tier) pos->total += lrugen->protected[hist][type][tier -1 ];
@@ -3732,15 +3732,15 @@ static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover)if (carryover) {unsigned long sum; - sum = lrugen->avg_refaulted[type][tier] + + sum =atomic_long_read (&lrugen->avg_refaulted[type][tier]) +atomic_long_read (&lrugen->refaulted[hist][type][tier]); -WRITE_ONCE (lrugen->avg_refaulted[type][tier], sum /2 ); +atomic_long_set (&lrugen->avg_refaulted[type][tier], sum /2 ); - sum = lrugen->avg_total[type][tier] + + sum =atomic_long_read (&lrugen->avg_total[type][tier]) +atomic_long_read (&lrugen->evicted[hist][type][tier]);if (tier) sum += lrugen->protected[hist][type][tier -1 ]; -WRITE_ONCE (lrugen->avg_total[type][tier], sum /2 ); +atomic_long_set (&lrugen->avg_total[type][tier], sum /2 ); }if (clear) {
@@ -5869,8 +5869,8 @@ static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruvec,if (seq == max_seq) { s ="RT " ; - n[0 ] =READ_ONCE (lrugen->avg_refaulted[type][tier]); - n[1 ] =READ_ONCE (lrugen->avg_total[type][tier]); + n[0 ] =atomic_long_read (&lrugen->avg_refaulted[type][tier]); + n[1 ] =atomic_long_read (&lrugen->avg_total[type][tier]); }else if (seq == min_seq[type] ||NR_HIST_GENS >1 ) { s ="rep" ; n[0 ] =atomic_long_read (&lrugen->refaulted[hist][type][tier]);
Signature · 1 line
2.41.0
RFC PATCH 4/4 workingset, lru_gen: apply refault-distance based re-activation
In reply to Kairui SongFrom: Kairui Song <[email protected]>
I noticed MGLRU not working very well on certain workflows, which is observed on some heavily stressed databases. That is when the file page workingset size exceeds total memory, and the access distance (the left-shift time of a page before it gets activated, considering LRU starts from right) of file pages also larger than total memory. All file pages are stuck on the oldest generation and getting read-in then evicted permutably. Despite anon pages being idle, they never get aged. PID controller didn't kickin until there are some minor access pattern changes. And file pages are not promoted or reused.
Even though the memory can't cover the whole workingset, the refault-distance based re-activation can help hold part of the workingset in-memory to help reduce the IO workload significantly.
So apply it for MGLRU as well. The updated refault-distance model fits well for MGLRU in most cases, if we just consider the last two generation as the inactive LRU and the first two generations as active LRU.
Some minor tinkering is done to fit the logic better, also make the refault-distance contributed to page tiering and PID refault detection of MGLRU:
- If a tier-0 page have a qualified refault-distance, just promote it to higher tier, send it to second oldest gen.
- If a tier >= 1 page have a qualified refault-distance, mark it as active and send it to youngest gen.
- Increase the reference of every page that have a qualified refault-distance and increase the PID countroled refault rate of the updated tier.
Following benchmark showed a major improvement.
To simulate the workflow, I setup a 3-replicated mongodb cluster using docker, each in a standalone cgroup, set to use 5 gb of cache and 10g of oplog, on a 32G VM. The benchmark is done using
https://github.com/apavlo/py-tpcc.git, modified to run STOCK_LEVEL query only, for simulating slow query and get a stable result.
Before the patch (with 10G swap, the result won't change whether swap is on or not):
$ tpcc.py --config=mongodb.config mongodb --duration=900 --warehouses=500 --clients=30
==================================================================
Execution Results after 904 seconds
------------------------------------------------------------------
Executed Time (µs) Rate
STOCK_LEVEL 503 27150226136.4 0.02 txn/s
------------------------------------------------------------------
TOTAL 503 27150226136.4 0.02 txn/s$ cat /proc/vmstat | grep working workingset_nodes 53391 workingset_refault_anon 0 workingset_refault_file 23856735 workingset_activate_anon 0 workingset_activate_file 23845737 workingset_restore_anon 0 workingset_restore_file 18280692 workingset_nodereclaim 1024
$ free -m
total used free shared buff/cache available
Mem: 31837 6752 379 23 24706 24607
Swap: 10239 0 10239After the patch (with 10G swap on same disk, similar result using ZRAM):
$ tpcc.py --config=mongodb.config mongodb --duration=900 --warehouses=500 --clients=30
==================================================================
Execution Results after 903 seconds
------------------------------------------------------------------
Executed Time (µs) Rate
STOCK_LEVEL 2575 27094953498.8 0.10 txn/s
------------------------------------------------------------------
TOTAL 2575 27094953498.8 0.10 txn/s$ cat /proc/vmstat | grep working workingset_nodes 78249 workingset_refault_anon 10139 workingset_refault_file 23001863 workingset_activate_anon 7238 workingset_activate_file 6718032 workingset_restore_anon 7432 workingset_restore_file 6719406 workingset_nodereclaim 9747
$ free -m
total used free shared buff/cache available
Mem: 31837 7376 320 3 24140 24014
Swap: 10239 1662 8577The performance is 5x times better than before, and the idle anon pages now can get swapped out as expected. The result is also better with lower test stress, testing with lower stress also shows a improvement.
I also checked the benchmark with memtier/memcached and fio, using similar setup as in commit ac35a4902374 but scaled down to fit in my test environment:
memcached test (with 16G ramdisk as swap and 2G cgroup limit):
memcached -u nobody -m 16384 -s /tmp/memcached.socket -a 0766 \
-t 12 -B binary &
memtier_benchmark -S /tmp/memcached.socket -P memcache_binary -n allkeys\
--key-minimum=1 --key-maximum=24000000 --key-pattern=P:P -c 1 \
-t 12 --ratio 1:0 --pipeline 8 -d 2000 -x 6
fio test (with 16G ramdisk on /mnt and 4G cgroup limit):
fio -name=refault --numjobs=12 --directory=/mnt --size=1024m \
--buffered=1 --ioengine=io_uring --iodepth=128 \
--iodepth_batch_submit=32 --iodepth_batch_complete=32 \
--rw=randread --random_distribution=random --norandommap \
--time_based --ramp_time=5m --runtime=5m --group_reportingBefore this patch:
memcached read:
Ops/sec Hits/sec Misses/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
Best 52832.79 0.00 0.00 1.82042 1.70300 4.54300 6.27100 105641.69 Worst 46613.56 0.00 0.00 2.05686 1.77500 7.80700 11.83900 93206.05 Avg (6x) 51024.85 0.00 0.00 1.88506 1.73500 5.43900 9.47100 102026.64
fio:
read: IOPS=2211k, BW=8637MiB/s (9056MB/s)(2530GiB/300001msec)
After this patch:
memcached read:
Ops/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
Best 54218.92 1.76930 1.65500 4.41500 6.27100 108413.34 Worst 47640.13 2.01495 1.74300 7.64700 11.64700 95258.72 Avg (6x) 51408.33 1.86988 1.71900 5.43900 9.34300 102793.42
fio:
read: IOPS=2166k, BW=8462MiB/s (8873MB/s)(2479GiB/300001msec)
memcached looks ok but there is a %2 performance drop for FIO test, and after some profiling this is mainly caused by the extra atomic operations and new functions, there seems to be no LRU accuracy drop.
--- mm/workingset.c | 74 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 51 insertions(+), 23 deletions(-)
mm/workingset.c +51 -23
@@ -185,6 +185,7 @@ MEM_CGROUP_ID_SHIFT)#define EVICTION_BITS (BITS_PER_LONG - (EVICTION_SHIFT))#define EVICTION_MASK (~0UL >> EVICTION_SHIFT) +#define LRU_GEN_EVICTION_BITS (EVICTION_BITS - LRU_REFS_WIDTH - LRU_GEN_WIDTH)/* * Eviction timestamps need to be able to cover the full range of
@@ -195,6 +196,7 @@* evictions into coarser buckets by shaving off lower timestamp bits. */ static unsigned int bucket_order__read_mostly ; +static unsigned int lru_gen_bucket_order__read_mostly ;static void *pack_shadow (int memcgid,pg_data_t *pgdat,unsigned long eviction,bool workingset)
@@ -345,10 +347,14 @@ static void *lru_gen_eviction(struct folio *folio) lruvec =mem_cgroup_lruvec (memcg, pgdat); lrugen = &lruvec->lrugen; min_seq =READ_ONCE (lrugen->min_seq[type]); + token = (min_seq <<LRU_REFS_WIDTH ) |max (refs -1 ,0 ); + token <<=LRU_GEN_EVICTION_BITS ; + token |=lru_eviction (lruvec,LRU_GEN_EVICTION_BITS , lru_gen_bucket_order); hist =lru_hist_from_seq (min_seq);atomic_long_add (delta, &lrugen->evicted[hist][type][tier]); +workingset_age_nonresident (lruvec,folio_nr_pages (folio));return pack_shadow (mem_cgroup_id (memcg), pgdat, token, refs); }
@@ -363,44 +369,55 @@ static bool lru_gen_test_recent(struct lruvec *lruvec, bool file,unsigned long min_seq; min_seq =READ_ONCE (lruvec->lrugen.min_seq[file]); + token >>=LRU_GEN_EVICTION_BITS ;return (token >>LRU_REFS_WIDTH ) == (min_seq & (EVICTION_MASK >>LRU_REFS_WIDTH )); }static void lru_gen_refault (struct folio *folio,void *shadow) {int memcgid; -bool recent; +bool refault;bool workingset;unsigned long token; +bool recent = false; +int refault_tier =0 ;int hist, tier, refs;struct lruvec *lruvec; +struct mem_cgroup *memcg;struct pglist_data *pgdat;struct lru_gen_folio *lrugen;int type =folio_is_file_lru (folio);int delta =folio_nr_pages (folio); -rcu_read_lock (); -unpack_shadow (shadow, &memcgid, &pgdat, &token, &workingset); - lruvec =mem_cgroup_lruvec (mem_cgroup_from_id (memcgid), pgdat); -if (lruvec !=folio_lruvec (folio)) - goto unlock; + memcg =mem_cgroup_from_id (memcgid); + lruvec =mem_cgroup_lruvec (memcg, pgdat); +/* memcg can be NULL, go through lruvec */ + memcg =lruvec_memcg (lruvec);mod_lruvec_state (lruvec,WORKINGSET_REFAULT_BASE + type, delta); - - recent =lru_gen_test_recent (lruvec, type, token); -if (!recent) - goto unlock; + refault =lru_refault (memcg, lruvec, token,LRU_GEN_EVICTION_BITS , + lru_gen_bucket_order); +if (lruvec ==folio_lruvec (folio)) + recent =lru_gen_test_recent (lruvec, type, token); +if (!recent && !refault) +return ; lrugen = &lruvec->lrugen; - hist =lru_hist_from_seq (READ_ONCE (lrugen->min_seq[type]));/* see the comment in folio_lru_refs() */ + token >>=LRU_GEN_EVICTION_BITS ; refs = (token & (BIT (LRU_REFS_WIDTH ) -1 )) + workingset; tier =lru_tier_from_refs (refs); - -atomic_long_add (delta, &lrugen->refaulted[hist][type][tier]); -mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); + refault_tier = tier; + +if (refault) { +if (refs) +folio_set_active (folio); +if (refs !=BIT (LRU_REFS_WIDTH )) + refault_tier =lru_tier_from_refs (refs +1 ); +mod_lruvec_state (lruvec,WORKINGSET_ACTIVATE_BASE + type, delta); + }/* * Count the following two cases as stalls:
@@ -409,12 +426,17 @@ static void lru_gen_refault(struct folio *folio, void *shadow)* 2. For pages accessed multiple times through file descriptors, * numbers of accesses might have been out of the range. */ -if (lru_gen_in_fault () || refs ==BIT (LRU_REFS_WIDTH )) { +if (refault ||lru_gen_in_fault () || refs ==BIT (LRU_REFS_WIDTH )) {folio_set_workingset (folio);mod_lruvec_state (lruvec,WORKINGSET_RESTORE_BASE + type, delta); } -unlock: - rcu_read_unlock(); + +if (recent && refault_tier == tier) { +atomic_long_add (delta, &lrugen->refaulted[hist][type][tier]); + }else { +atomic_long_add (delta, &lrugen->avg_total[type][refault_tier]); +atomic_long_add (delta, &lrugen->avg_refaulted[type][refault_tier]); + } }#else /* !CONFIG_LRU_GEN */
@@ -536,16 +558,15 @@ void workingset_refault(struct folio *folio, void *shadow)bool workingset;long nr; -if (lru_gen_enabled ()) { -lru_gen_refault (folio, shadow); -return ; - } -/* Flush stats (and potentially sleep) before holding RCU read lock */ mem_cgroup_flush_stats_ratelimited (); -rcu_read_lock (); +if (lru_gen_enabled ()) { +lru_gen_refault (folio, shadow); + goto out; + } +/* * The activation decision for this folio is made at the level * where the eviction occurred, as that is where the LRU order
@@ -791,6 +812,13 @@ static int __init workingset_init(void)pr_info ("workingset: timestamp_bits=%d max_order=%d bucket_order=%u\n" ,EVICTION_BITS , max_order, bucket_order); +#ifdef CONFIG_LRU_GEN +if (max_order >LRU_GEN_EVICTION_BITS ) + lru_gen_bucket_order = max_order -LRU_GEN_EVICTION_BITS ; +pr_info ("workingset: lru_gen_timestamp_bits=%d lru_gen_bucket_order=%u\n" , +LRU_GEN_EVICTION_BITS , lru_gen_bucket_order); +#endif + ret =prealloc_shrinker (&workingset_shadow_shrinker,"mm-shadow" );if (ret) goto err;
Signature · 1 line
2.41.0
kernel test robot
build botRe: [RFC PATCH 4/4] workingset, lru_gen: apply refault-distance based re-activation
kernel test robot noticed the following build warnings:
Hi Kairui,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on linus/master v6.5-rc3 next-20230725] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Kairui-Song/workingset-simplify-and-use-a-more-intuitive-model/20230726-025946 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20230725185733.43929-5-ryncsn%40gmail.com patch subject: [RFC PATCH 4/4] workingset, lru_gen: apply refault-distance based re-activation config: i386-randconfig-i012-20230725 (https://download.01.org/0day-ci/archive/20230726/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce: (https://download.01.org/0day-ci/archive/20230726/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
mm/workingset.c: In function 'workingset_test_recent':
mm/workingset.c:537:67: warning: passing argument 3 of 'lru_gen_test_recent' makes pointer from integer without a cast [-Wint-conversion]
537 | return lru_gen_test_recent(eviction_lruvec, file, eviction);
| ^~~~~~~~
| |
| long unsigned int
mm/workingset.c:449:74: note: expected 'struct lruvec **' but argument is of type 'long unsigned int'
449 | static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec,
| ~~~~~~~~~~~~~~~~^~~~~~
mm/workingset.c:537:24: error: too few arguments to function 'lru_gen_test_recent'
537 | return lru_gen_test_recent(eviction_lruvec, file, eviction);
| ^~~~~~~~~~~~~~~~~~~
mm/workingset.c:449:13: note: declared here
449 | static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec,
| ^~~~~~~~~~~~~~~~~~~
mm/workingset.c: At top level:
>> mm/workingset.c:199:21: warning: 'lru_gen_bucket_order' defined but not used [-Wunused-variable]
199 | static unsigned int lru_gen_bucket_order __read_mostly;
| ^~~~~~~~~~~~~~~~~~~~vim +/lru_gen_bucket_order +199 mm/workingset.c
189
190 /*
191 * Eviction timestamps need to be able to cover the full range of
192 * actionable refaults. However, bits are tight in the xarray
193 * entry, and after storing the identifier for the lruvec there might
194 * not be enough left to represent every single actionable refault. In
195 * that case, we have to sacrifice granularity for distance, and group
196 * evictions into coarser buckets by shaving off lower timestamp bits.
197 */
198 static unsigned int bucket_order __read_mostly;
> 199 static unsigned int lru_gen_bucket_order __read_mostly;
200
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki