Skip to contentSkip to conversation

mm/lru_gen: Move some code around so that next patch is simpler

PATCH In review Aneesh Kumar K.V 4 participants 14 messages Jun 13, 2023 – Jun 27, 2023

Subsystems (10)
Memory management x86 MM core arm64 +6 more subsystems
lore
Share & tools
lore
#0 lore raw

PATCH 1/3 mm/lru_gen: Move some code around so that next patch is simpler

Move lrur_gen_add_folio to .c. We will support arch specific mapping of page access count to generation in a later patch and will use that when adding folio to lruvec. This move enables that.

No functional change in this patch.

---
 include/linux/mm_inline.h |  47 +----------
 mm/vmscan.c               | 172 ++++++++++++++++++++++++--------------
 2 files changed, 110 insertions(+), 109 deletions(-)
include/linux/mm_inline.h +1 -46
index 0e1d239a882c..2a86dc4d96ab 100644
@@ -217,52 +217,7 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
 	VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen));
 }
 
-static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
-{
-	unsigned long seq;
-	unsigned long flags;
-	int gen = folio_lru_gen(folio);
-	int type = folio_is_file_lru(folio);
-	int zone = folio_zonenum(folio);
-	struct lru_gen_folio *lrugen = &lruvec->lrugen;
-
-	VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
-
-	if (folio_test_unevictable(folio) || !lrugen->enabled)
-		return false;
-	/*
-	 * There are three common cases for this page:
-	 * 1. If it's hot, e.g., freshly faulted in or previously hot and
-	 *    migrated, add it to the youngest generation.
-	 * 2. If it's cold but can't be evicted immediately, i.e., an anon page
-	 *    not in swapcache or a dirty page pending writeback, add it to the
-	 *    second oldest generation.
-	 * 3. Everything else (clean, cold) is added to the oldest generation.
-	 */
-	if (folio_test_active(folio))
-		seq = lrugen->max_seq;
-	else if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) ||
-		 (folio_test_reclaim(folio) &&
-		  (folio_test_dirty(folio) || folio_test_writeback(folio))))
-		seq = lrugen->min_seq[type] + 1;
-	else
-		seq = lrugen->min_seq[type];
-
-	gen = lru_gen_from_seq(seq);
-	flags = (gen + 1UL) << LRU_GEN_PGOFF;
-	/* see the comment on MIN_NR_GENS about PG_active */
-	set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags);
-
-	lru_gen_update_size(lruvec, folio, -1, gen);
-	/* for folio_rotate_reclaimable() */
-	if (reclaiming)
-		list_add_tail(&folio->lru, &lrugen->folios[gen][type][zone]);
-	else
-		list_add(&folio->lru, &lrugen->folios[gen][type][zone]);
-
-	return true;
-}
-
+bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming);
 static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
 {
 	unsigned long flags;
mm/vmscan.c +109 -63

202 lines not on this page · raw message

Signature · 1 line
2.40.1
#1 lore raw

PATCH 2/3 mm/lru_gen: lru_gen_look_around simplification

To store generation details in folio_flags we need lru_gen_mm_walk structure in which we batch the nr_pages update. A follow-up patch wants to avoid compiling all the lru_gen mm walk-related code on architectures that don't support it. Split out the look-around generation update by marking folio active into a separate helper which will be used in that case.

---
 mm/vmscan.c | 57 +++++++++++++++++++++++++++++++++++------------------
 1 file changed, 38 insertions(+), 19 deletions(-)
mm/vmscan.c +38 -19
index edfe073b475e..f277beba556c 100644
@@ -4619,6 +4619,39 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)
  *                          rmap/PT walk feedback
  ******************************************************************************/
 
+static void __look_around_gen_update(struct folio *folio, int new_gen)
+{
+	int old_gen;
+
+	old_gen = folio_lru_gen(folio);
+	if (old_gen < 0)
+		folio_set_referenced(folio);
+	else if (old_gen != new_gen)
+		folio_activate(folio);
+}
+
+static inline bool current_reclaim_state_can_swap(void)
+{
+	if (current->reclaim_state)
+		return current->reclaim_state->mm_walk->can_swap;
+	return true;
+}
+
+static void look_around_gen_update(struct folio *folio, int new_gen)
+{
+	int old_gen;
+	struct lru_gen_mm_walk *walk;
+
+	walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL;
+	if (walk) {
+		old_gen = folio_update_gen(folio, new_gen);
+		if (old_gen >= 0 && old_gen != new_gen)
+			update_batch_size(walk, folio, old_gen, new_gen);
+		return;
+	}
+	return __look_around_gen_update(folio, new_gen);
+}
+
 /*
  * This function exploits spatial locality when shrink_folio_list() walks the
  * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. If
@@ -4631,7 +4664,6 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 	int i;
 	unsigned long start;
 	unsigned long end;
-	struct lru_gen_mm_walk *walk;
 	int young = 0;
 	pte_t *pte = pvmw->pte;
 	unsigned long addr = pvmw->address;
@@ -4640,7 +4672,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 	struct pglist_data *pgdat = folio_pgdat(folio);
 	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
 	DEFINE_MAX_SEQ(lruvec);
-	int old_gen, new_gen = lru_gen_from_seq(max_seq);
+	int new_gen = lru_gen_from_seq(max_seq);
 
 	lockdep_assert_held(pvmw->ptl);
 	VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio);
@@ -4648,9 +4680,6 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 	if (spin_is_contended(pvmw->ptl))
 		return;
 
-	/* avoid taking the LRU lock under the PTL when possible */
-	walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL;
-
 	start = max(addr & PMD_MASK, pvmw->vma->vm_start);
 	end = min(addr | ~PMD_MASK, pvmw->vma->vm_end - 1) + 1;
 
@@ -4683,7 +4712,9 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 		if (!pte_young(pte[i]))
 			continue;
 
-		folio = get_pfn_folio(pfn, memcg, pgdat, !walk || walk->can_swap);
+		folio = get_pfn_folio(pfn, memcg, pgdat,
+				      current_reclaim_state_can_swap());
+
 		if (!folio)
 			continue;
 
@@ -4697,19 +4728,7 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
 		      !folio_test_swapcache(folio)))
 			folio_mark_dirty(folio);
 
-		if (walk) {
-			old_gen = folio_update_gen(folio, new_gen);
-			if (old_gen >= 0 && old_gen != new_gen)
-				update_batch_size(walk, folio, old_gen, new_gen);
-
-			continue;
-		}
-
-		old_gen = folio_lru_gen(folio);
-		if (old_gen < 0)
-			folio_set_referenced(folio);
-		else if (old_gen != new_gen)
-			folio_activate(folio);
+		look_around_gen_update(folio, new_gen);
 	}
 
 	arch_leave_lazy_mmu_mode();
Signature · 1 line
2.40.1
#2 lore raw

PATCH 3/3 mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

In reply to Aneesh Kumar K.V

Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture.

No preformance change observed with mongodb ycsb test:

Patch details Throughput(Ops/sec)
without patch 93278
With patch 93400

Without patch:

$ size mm/vmscan.o
   text    data     bss     dec     hex filename
 112102   42721      40  154863   25cef mm/vmscan.o

With patch

$ size  mm/vmscan.o
   text    data     bss     dec     hex filename
 105430   41333      24  146787   23d63 mm/vmscan.o
---
 arch/Kconfig               |   3 +
 arch/arm64/Kconfig         |   1 +
 arch/x86/Kconfig           |   1 +
 include/linux/memcontrol.h |   2 +-
 include/linux/mm_types.h   |   8 +--
 include/linux/mmzone.h     |   8 +++
 include/linux/swap.h       |   2 +-
 kernel/fork.c              |   2 +-
 mm/memcontrol.c            |   2 +-
 mm/vmscan.c                | 128 +++++++++++++++++++++++++++++++++----
 10 files changed, 137 insertions(+), 20 deletions(-)
arch/Kconfig +3 -0
index 205fd23e0cad..5cdd98731298 100644
@@ -1458,6 +1458,9 @@ config DYNAMIC_SIGFRAME
 config HAVE_ARCH_NODE_DEV_GROUP
 	bool
 
+config LRU_TASK_PAGE_AGING
+	bool
+
 config ARCH_HAS_NONLEAF_PMD_YOUNG
 	bool
 	help
arch/arm64/Kconfig +1 -0
index b1201d25a8a4..e0994fb3504b 100644
@@ -225,6 +225,7 @@ config ARM64
 	select IRQ_DOMAIN
 	select IRQ_FORCED_THREADING
 	select KASAN_VMALLOC if KASAN
+	select LRU_TASK_PAGE_AGING		if LRU_GEN
 	select MODULES_USE_ELF_RELA
 	select NEED_DMA_MAP_STATE
 	select NEED_SG_DMA_LENGTH
arch/x86/Kconfig +1 -0
index 53bab123a8ee..bde9e6f33b22 100644
@@ -276,6 +276,7 @@ config X86
 	select HAVE_GENERIC_VDSO
 	select HOTPLUG_SMT			if SMP
 	select IRQ_FORCED_THREADING
+	select LRU_TASK_PAGE_AGING		if LRU_GEN
 	select NEED_PER_CPU_EMBED_FIRST_CHUNK
 	select NEED_PER_CPU_PAGE_FIRST_CHUNK
 	select NEED_SG_DMA_LENGTH
include/linux/memcontrol.h +1 -1
index 43d4ec8445d4..ea5d1d7bfb8b 100644
@@ -321,7 +321,7 @@ struct mem_cgroup {
 	struct deferred_split deferred_split_queue;
 #endif
 
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 	/* per-memcg mm_struct list */
 	struct lru_gen_mm_list mm_list;
 #endif
include/linux/mm_types.h +4 -4
index 306a3d1a0fa6..f90a4860a792 100644
@@ -786,7 +786,7 @@ struct mm_struct {
 		 */
 		unsigned long ksm_rmap_items;
 #endif
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 		struct {
 			/* this mm_struct is on lru_gen_mm_list */
 			struct list_head list;
@@ -801,7 +801,7 @@ struct mm_struct {
 			struct mem_cgroup *memcg;
 #endif
 		} lru_gen;
-#endif /* CONFIG_LRU_GEN */
+#endif /* CONFIG_LRU_TASK_PAGE_AGING */
 	} __randomize_layout;
 
 	/*
@@ -830,7 +830,7 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm)
 	return (struct cpumask *)&mm->cpu_bitmap;
 }
 
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 
 struct lru_gen_mm_list {
 	/* mm_struct list for page table walkers */
@@ -864,7 +864,7 @@ static inline void lru_gen_use_mm(struct mm_struct *mm)
 	WRITE_ONCE(mm->lru_gen.bitmap, -1);
 }
 
-#else /* !CONFIG_LRU_GEN */
+#else /* !CONFIG_LRU_TASK_PAGE_AGING */
 
 static inline void lru_gen_add_mm(struct mm_struct *mm)
 {
include/linux/mmzone.h +8 -0
index a4889c9d4055..b35698148d3c 100644
@@ -427,6 +427,7 @@ struct lru_gen_folio {
 #endif
 };
 
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 enum {
 	MM_LEAF_TOTAL,		/* total leaf entries */
 	MM_LEAF_OLD,		/* old leaf entries */
@@ -469,6 +470,7 @@ struct lru_gen_mm_walk {
 	bool can_swap;
 	bool force_scan;
 };
+#endif
 
 void lru_gen_init_lruvec(struct lruvec *lruvec);
 void lru_gen_look_around(struct page_vma_mapped_walk *pvmw);
@@ -613,8 +615,12 @@ struct lruvec {
 #ifdef CONFIG_LRU_GEN
 	/* evictable pages divided into generations */
 	struct lru_gen_folio		lrugen;
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 	/* to concurrently iterate lru_gen_mm_list */
 	struct lru_gen_mm_state		mm_state;
+#else
+	bool seq_update_progress;
+#endif
 #endif
 #ifdef CONFIG_MEMCG
 	struct pglist_data *pgdat;
@@ -1354,8 +1360,10 @@ typedef struct pglist_data {
 	unsigned long		flags;
 
 #ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 	/* kswap mm walk data */
 	struct lru_gen_mm_walk mm_walk;
+#endif
 	/* lru_gen_folio list */
 	struct lru_gen_memcg memcg_lru;
 #endif
include/linux/swap.h +1 -1
index 3c69cb653cb9..ce09b1e44275 100644
@@ -155,7 +155,7 @@ union swap_header {
 struct reclaim_state {
 	/* pages reclaimed outside of LRU-based reclaim */
 	unsigned long reclaimed;
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 	/* per-thread mm walk data */
 	struct lru_gen_mm_walk *mm_walk;
 #endif
kernel/fork.c +1 -1
index ed4e01daccaa..2c9e21e39f84 100644
@@ -2939,7 +2939,7 @@ pid_t kernel_clone(struct kernel_clone_args *args)
 		get_task_struct(p);
 	}
 
-	if (IS_ENABLED(CONFIG_LRU_GEN) && !(clone_flags & CLONE_VM)) {
+	if (IS_ENABLED(CONFIG_LRU_TASK_PAGE_AGING) && !(clone_flags & CLONE_VM)) {
 		/* lock the task to synchronize with memcg migration */
 		task_lock(p);
 		lru_gen_add_mm(p->mm);
mm/memcontrol.c +1 -1
index 959d6a27e23d..d8fe30d880c6 100644
@@ -6404,7 +6404,7 @@ static void mem_cgroup_move_task(void)
 }
 #endif
 
-#ifdef CONFIG_LRU_GEN
+#ifdef CONFIG_LRU_TASK_PAGE_AGING
 static void mem_cgroup_attach(struct cgroup_taskset *tset)
 {
 	struct task_struct *task;
mm/vmscan.c +116 -12

280 lines not on this page · raw message

Signature · 1 line
2.40.1
#3 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

mm/vmscan.c static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, · line 4536

Quoted 6 lines · Aneesh Kumar K.V
mm/vmscan.c +1 -1
@@ -4498,7 +4533,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
 		goto done;
 	}
 
-	walk = set_mm_walk(NULL, true);
+	walk = (struct lru_gen_mm_walk *)set_mm_walk(NULL, true);

This isn't C++.

#4 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

Quoted 10 lines · Matthew Wilcox

mm/vmscan.c static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, · line 4536

Quoted 6 lines · Aneesh Kumar K.V
mm/vmscan.c +1 -1
@@ -4498,7 +4533,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
 		goto done;
 	}
 
-	walk = set_mm_walk(NULL, true);
+	walk = (struct lru_gen_mm_walk *)set_mm_walk(NULL, true);

This isn't C++.

We have similar pattern for things like kmalloc()? I understand the desire to have functions return the correct type. But the amount of code that we are able to avoid with this patch for certain architecture is really large.

-aneesh

#5 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

Quoted 6 lines · Aneesh Kumar K V
Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · Aneesh Kumar K.V
mm/vmscan.c +0 -0
@@ -4498,7 +4533,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
 		goto done;
 	}
 
Quoted 7 lines
Quoted 5 lines

mm/vmscan.c static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, · line 4536

Quoted 2 lines
mm/vmscan.c +1 -1
-	walk = set_mm_walk(NULL, true);
+	walk = (struct lru_gen_mm_walk *)set_mm_walk(NULL, true);

This isn't C++.

We have similar pattern for things like kmalloc()?

No. No, we don't. Nobody does that. Perhaps some really crappy code in staging. DO NOT USE CASTS.

Quoted 3 lines

I understand the desire to have functions return
the correct type. But the amount of code that we are able to avoid with this patch for certain architecture is really large.

There's probably a better way to do what you're trying to do, but the simple fact remains that the cast you added is needed in C++ and not in C. Linux is not written in C++. Do not add the cast.

#6 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

Quoted 18 lines · Matthew Wilcox
Quoted 6 lines · Aneesh Kumar K V
Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · Aneesh Kumar K.V
mm/vmscan.c +0 -0
@@ -4498,7 +4533,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
 		goto done;
 	}
 
Quoted 7 lines
Quoted 5 lines

mm/vmscan.c static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, · line 4536

Quoted 2 lines
mm/vmscan.c +1 -1
-	walk = set_mm_walk(NULL, true);
+	walk = (struct lru_gen_mm_walk *)set_mm_walk(NULL, true);

This isn't C++.

We have similar pattern for things like kmalloc()?

No. No, we don't. Nobody does that. Perhaps some really crappy code in staging. DO NOT USE CASTS.

Quoted 7 lines
Quoted 3 lines

I understand the desire to have functions return
the correct type. But the amount of code that we are able to avoid with this patch for certain architecture is really large.

There's probably a better way to do what you're trying to do, but the simple fact remains that the cast you added is needed in C++ and not in C. Linux is not written in C++. Do not add the cast.

What I want is to allow the usage of set_mm_walk() such that i don't need to have that #ifdef throughout the code. I also want to keep the definition of struct lru_gen_mm_walk {} within that #ifdef as below.

#ifdef CONFIG_LRU_TASK_PAGE_AGING

struct lru_gen_mm_walk {
	/* the lruvec under reclaim */
	...
};

static void *set_mm_walk(struct pglist_data *pgdat, bool force_alloc)

{
	struct lru_gen_mm_walk *walk = current->reclaim_state->mm_walk;
	...
	return walk;
}

#else

static inline void *set_mm_walk(struct pglist_data *pgdat, bool force_alloc)

{
	return NULL;
}

#endif

#7 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

In reply to Aneesh Kumar K.V
kernel test robot noticed the following build warnings:

Hi Aneesh,

kernel test robot noticed the following build warnings:

[auto build test WARNING on arm64/for-next/core]
[also build test WARNING on linus/master tip/x86/core v6.4-rc7 next-20230620] [cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Aneesh-Kumar-K-V/mm-lru_gen-lru_gen_look_around-simplification/20230613-200408
base:   https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core
patch link:    https://lore.kernel.org/r/20230613120047.149573-3-aneesh.kumar%40linux.ibm.com
patch subject: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported
config: alpha-allmodconfig (https://download.01.org/0day-ci/archive/20230621/[email protected]/config)
compiler: alpha-linux-gcc (GCC) 12.3.0
reproduce: (https://download.01.org/0day-ci/archive/20230621/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> mm/vmscan.c:4564:6: warning: no previous prototype for '__try_to_inc_max_seq' [-Wmissing-prototypes]

4564 | bool __try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,
| ^~~~~~~~~~~~~~~~~~~~

vim +/__try_to_inc_max_seq +4564 mm/vmscan.c

4559
4560 /*
4561 * inc_max_seq can drop the lru_lock in between. So use a waitqueue seq_update_progress
4562 * to allow concurrent access.
4563 */

> 4564	bool __try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq,

4565 bool can_swap, bool force_scan)
4566 {
4567 bool success = false;
4568 struct lru_gen_folio *lrugen = &lruvec->lrugen;
4569
4570 VM_WARN_ON_ONCE(max_seq > READ_ONCE(lrugen->max_seq));
4571
4572 /* see the comment in iterate_mm_list() */
4573 if (lruvec->seq_update_progress)
4574 success = false;
4575 else {
4576 spin_lock_irq(&lruvec->lru_lock);
4577
4578 if (max_seq != lrugen->max_seq)
4579 goto done;
4580
4581 if (lruvec->seq_update_progress)
4582 goto done;
4583
4584 success = true;
4585 lruvec->seq_update_progress = true;
4586 done:
4587 spin_unlock_irq(&lruvec->lru_lock);
4588 }
4589
4590 if (success)
4591 inc_max_seq(lruvec, can_swap, force_scan);
4592
4593 return success;
4594 }
4595

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#8 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

In reply to Aneesh Kumar K.V

Hi Yu Zhao,

Quoted 16 lines · Aneesh Kumar K.V

Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture.

No preformance change observed with mongodb ycsb test:

Patch details Throughput(Ops/sec)
without patch 93278
With patch 93400

Without patch:

$ size mm/vmscan.o
   text    data     bss     dec     hex filename
 112102   42721      40  154863   25cef mm/vmscan.o
Quoted 6 lines

With patch

$ size  mm/vmscan.o
   text    data     bss     dec     hex filename
 105430   41333      24  146787   23d63 mm/vmscan.o

Any feedback on this patch? Can we look at merging this change?

-aneesh

#9 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

Quoted 21 lines · Aneesh Kumar K.V

Hi Yu Zhao,

Quoted 16 lines · Aneesh Kumar K.V

Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture.

No preformance change observed with mongodb ycsb test:

Patch details Throughput(Ops/sec)
without patch 93278
With patch 93400

Without patch:

$ size mm/vmscan.o
   text    data     bss     dec     hex filename
 112102   42721      40  154863   25cef mm/vmscan.o
Quoted 8 lines
Quoted 6 lines

With patch

$ size  mm/vmscan.o
   text    data     bss     dec     hex filename
 105430   41333      24  146787   23d63 mm/vmscan.o

Any feedback on this patch? Can we look at merging this change?

Just want to make sure I fully understand the motivation: are there any other end goals besides reducing the footprint mentioned above? E.g., preparing for HCA, etc. (My current understanding is that HCA shouldn't care about it, since it's already runtime disabled if HCA doesn't want to use it.)

Also as explained offline, solely relying on folio_activate() in lru_gen_look_around() can cause a measure regression on powerpc, because
1. PAGEVEC_SIZE is 15 whereas pglist_data->mm_walk.batched is virtually unlimited.
2. Once folio_activate() reaches that limit, it takes the LRU lock on top of the PTL, which can be shared by multiple page tables on powerpc.

In fact, I think we try the opposite direction first, before arriving at any conclusions, i.e.,

    #define arch_has_hw_pte_young() radix_enabled()

on powerpc. This might benefit platforms with the A-bit but not HCA, e.g., POWER9. I just ran a quick test (memcached/memtier I previously shared with you) and it showed far less PTL contention in kswapd. I'm attaching the flamegraphs for you to analyze. Could you try some benchmarks with the above change on your end as well?

Thanks.

#10 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

Quoted 30 lines · Yu Zhao
Quoted 28 lines · Aneesh Kumar K.V

Hi Yu Zhao,

Quoted 22 lines · Aneesh Kumar K.V

Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture.

No preformance change observed with mongodb ycsb test:

Patch details Throughput(Ops/sec)
without patch 93278
With patch 93400

Without patch:

$ size mm/vmscan.o
   text    data     bss     dec     hex filename
 112102   42721      40  154863   25cef mm/vmscan.o

With patch

$ size  mm/vmscan.o
   text    data     bss     dec     hex filename
 105430   41333      24  146787   23d63 mm/vmscan.o
Quoted 8 lines
Quoted 1 line

Any feedback on this patch? Can we look at merging this change?

Just want to make sure I fully understand the motivation: are there any other end goals besides reducing the footprint mentioned above? E.g., preparing for HCA, etc. (My current understanding is that HCA shouldn't care about it, since it's already runtime disabled if HCA doesn't want to use it.)

My goal with this change was to remove all those dead code from getting complied in for ppc64.

Quoted 9 lines

Also as explained offline, solely relying on folio_activate() in lru_gen_look_around() can cause a measure regression on powerpc, because
1. PAGEVEC_SIZE is 15 whereas pglist_data->mm_walk.batched is virtually unlimited.
2. Once folio_activate() reaches that limit, it takes the LRU lock on top of the PTL, which can be shared by multiple page tables on powerpc.

Quoted 3 lines

In fact, I think we try the opposite direction first, before arriving at any conclusions, i.e.,

    #define arch_has_hw_pte_young() radix_enabled()

The reason it is disabled on powerpc was that a reference bit update takes a pagefault on powerpc irrespective of the translation mode.

Quoted 6 lines

on powerpc. This might benefit platforms with the A-bit but not HCA, e.g., POWER9. I just ran a quick test (memcached/memtier I previously shared with you) and it showed far less PTL contention in kswapd. I'm attaching the flamegraphs for you to analyze. Could you try some benchmarks with the above change on your end as well?

The ptl lock is a valid concern even though i didn't observe the contention increasing with the change. I will rerun the test to verify. We have possibly two options here

1) Delay the lruvec->nr_pages update until the sort phase. But as you explained earlier, that can impact should_run_aging().

2) Add another batching mechanism similar to pglist_data->mm_walk which can be used on architecture that don't support hardware update of access/reference bit to be used only by lru_gen_look_around()

-aneesh

#11 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

Quoted 41 lines · Aneesh Kumar K V
Quoted 30 lines · Yu Zhao
Quoted 28 lines · Aneesh Kumar K.V

Hi Yu Zhao,

Quoted 22 lines · Aneesh Kumar K.V

Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture.

No preformance change observed with mongodb ycsb test:

Patch details Throughput(Ops/sec)
without patch 93278
With patch 93400

Without patch:

$ size mm/vmscan.o
   text    data     bss     dec     hex filename
 112102   42721      40  154863   25cef mm/vmscan.o

With patch

$ size  mm/vmscan.o
   text    data     bss     dec     hex filename
 105430   41333      24  146787   23d63 mm/vmscan.o
Quoted 8 lines
Quoted 1 line

Any feedback on this patch? Can we look at merging this change?

Just want to make sure I fully understand the motivation: are there any other end goals besides reducing the footprint mentioned above? E.g., preparing for HCA, etc. (My current understanding is that HCA shouldn't care about it, since it's already runtime disabled if HCA doesn't want to use it.)

Quoted 2 lines

My goal with this change was to remove all those dead code from getting complied in for ppc64.

I see. But the first thing (lru_gen_add_folio()) you moved has nothing to do with this goal, because it's still compiled after the entire series.

Quoted 9 lines
Quoted 9 lines

Also as explained offline, solely relying on folio_activate() in lru_gen_look_around() can cause a measure regression on powerpc, because
1. PAGEVEC_SIZE is 15 whereas pglist_data->mm_walk.batched is virtually unlimited.
2. Once folio_activate() reaches that limit, it takes the LRU lock on top of the PTL, which can be shared by multiple page tables on powerpc.

Quoted 6 lines
Quoted 3 lines

In fact, I think we try the opposite direction first, before arriving at any conclusions, i.e.,

    #define arch_has_hw_pte_young() radix_enabled()

The reason it is disabled on powerpc was that a reference bit update takes a pagefault on powerpc irrespective of the translation mode.

This is not true.

From "IBM POWER9 Processor User Manual":
https://openpowerfoundation.org/resources/ibmpower9usermanual/

4.10.14 Reference and Change Bits
...
When performing HPT translation, the hardware performs the R and C
bit updates nonatomically.
...

The radix case is more complex, and I'll leave it to you to interpret what it means:

From "Power ISA Version 3.0 B":
https://openpowerfoundation.org/specifications/isa/

5.7.12 Reference and Change Recording
...
For Radix Tree translation, the Reference and Change bits are set atomically.
...

Quoted 10 lines
Quoted 6 lines

on powerpc. This might benefit platforms with the A-bit but not HCA, e.g., POWER9. I just ran a quick test (memcached/memtier I previously shared with you) and it showed far less PTL contention in kswapd. I'm attaching the flamegraphs for you to analyze. Could you try some benchmarks with the above change on your end as well?

The ptl lock is a valid concern even though i didn't observe the contention increasing with the change. I will rerun the test to verify. We have possibly two options here

Quoted 6 lines

1) Delay the lruvec->nr_pages update until the sort phase. But as you explained earlier, that can impact should_run_aging().

2) Add another batching mechanism similar to pglist_data->mm_walk which can be used on architecture that don't support hardware update of access/reference bit to be used only by lru_gen_look_around()

Sounds good. Thanks.

#12 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

Quoted 43 lines · Yu Zhao
Quoted 41 lines · Aneesh Kumar K V
Quoted 38 lines · Yu Zhao
Quoted 21 lines · Aneesh Kumar K.V

Hi Yu Zhao,

Quoted 16 lines · Aneesh Kumar K.V

Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture.

No preformance change observed with mongodb ycsb test:

Patch details Throughput(Ops/sec)
without patch 93278
With patch 93400

Without patch:

$ size mm/vmscan.o
   text    data     bss     dec     hex filename
 112102   42721      40  154863   25cef mm/vmscan.o
Quoted 8 lines
Quoted 6 lines

With patch

$ size  mm/vmscan.o
   text    data     bss     dec     hex filename
 105430   41333      24  146787   23d63 mm/vmscan.o

Any feedback on this patch? Can we look at merging this change?

Just want to make sure I fully understand the motivation: are there any other end goals besides reducing the footprint mentioned above? E.g., preparing for HCA, etc. (My current understanding is that HCA shouldn't care about it, since it's already runtime disabled if HCA doesn't want to use it.)

Quoted 7 lines
Quoted 2 lines

My goal with this change was to remove all those dead code from getting complied in for ppc64.

I see. But the first thing (lru_gen_add_folio()) you moved has nothing to do with this goal, because it's still compiled after the entire series.

Sure. will drop that change.

Quoted 30 lines
Quoted 9 lines
Quoted 9 lines

Also as explained offline, solely relying on folio_activate() in lru_gen_look_around() can cause a measure regression on powerpc, because
1. PAGEVEC_SIZE is 15 whereas pglist_data->mm_walk.batched is virtually unlimited.
2. Once folio_activate() reaches that limit, it takes the LRU lock on top of the PTL, which can be shared by multiple page tables on powerpc.

Quoted 6 lines
Quoted 3 lines

In fact, I think we try the opposite direction first, before arriving at any conclusions, i.e.,

    #define arch_has_hw_pte_young() radix_enabled()

The reason it is disabled on powerpc was that a reference bit update takes a pagefault on powerpc irrespective of the translation mode.

This is not true.

From "IBM POWER9 Processor User Manual":
https://openpowerfoundation.org/resources/ibmpower9usermanual/

4.10.14 Reference and Change Bits
...
When performing HPT translation, the hardware performs the R and C
bit updates nonatomically.
...

The radix case is more complex, and I'll leave it to you to interpret what it means:

Quoted 8 lines

From "Power ISA Version 3.0 B":
https://openpowerfoundation.org/specifications/isa/

5.7.12 Reference and Change Recording
...
For Radix Tree translation, the Reference and Change bits are set atomically.
...

it is atomic in that software use ldarx/stdcx to update these bits. Hardware/core won't update this directly even though Nest can update this directly without taking a fault. So for all purpose we can assume that on radix R/C bit is updated by page fault handler.

Generic page table update sequence are slightly different with hash translation in that some page table field updates requires marking the page table entry invalid.

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

on powerpc. This might benefit platforms with the A-bit but not HCA, e.g., POWER9. I just ran a quick test (memcached/memtier I previously shared with you) and it showed far less PTL contention in kswapd. I'm attaching the flamegraphs for you to analyze. Could you try some benchmarks with the above change on your end as well?

The ptl lock is a valid concern even though i didn't observe the contention increasing with the change. I will rerun the test to verify. We have possibly two options here

Quoted 8 lines
Quoted 6 lines

1) Delay the lruvec->nr_pages update until the sort phase. But as you explained earlier, that can impact should_run_aging().

2) Add another batching mechanism similar to pglist_data->mm_walk which can be used on architecture that don't support hardware update of access/reference bit to be used only by lru_gen_look_around()

Sounds good. Thanks.

I will go ahead working on the approach 2 I outlined above?

-aneesh

#13 lore raw

Re: [PATCH 3/3] mm/lru_gen: Don't build multi-gen LRU page table walk code on architecture not supported

Quoted 94 lines · Aneesh Kumar K V
Quoted 43 lines · Yu Zhao
Quoted 41 lines · Aneesh Kumar K V
Quoted 38 lines · Yu Zhao
Quoted 21 lines · Aneesh Kumar K.V

Hi Yu Zhao,

Quoted 16 lines · Aneesh Kumar K.V

Not all architecture supports hardware atomic updates of access bits. On such an arch, we don't use page table walk to classify pages into generations. Add a kernel config option and remove adding all the page table walk code on such architecture.

No preformance change observed with mongodb ycsb test:

Patch details Throughput(Ops/sec)
without patch 93278
With patch 93400

Without patch:

$ size mm/vmscan.o
   text    data     bss     dec     hex filename
 112102   42721      40  154863   25cef mm/vmscan.o
Quoted 8 lines
Quoted 6 lines

With patch

$ size  mm/vmscan.o
   text    data     bss     dec     hex filename
 105430   41333      24  146787   23d63 mm/vmscan.o

Any feedback on this patch? Can we look at merging this change?

Just want to make sure I fully understand the motivation: are there any other end goals besides reducing the footprint mentioned above? E.g., preparing for HCA, etc. (My current understanding is that HCA shouldn't care about it, since it's already runtime disabled if HCA doesn't want to use it.)

Quoted 7 lines
Quoted 2 lines

My goal with this change was to remove all those dead code from getting complied in for ppc64.

I see. But the first thing (lru_gen_add_folio()) you moved has nothing to do with this goal, because it's still compiled after the entire series.

Sure. will drop that change.

Quoted 30 lines
Quoted 9 lines
Quoted 9 lines

Also as explained offline, solely relying on folio_activate() in lru_gen_look_around() can cause a measure regression on powerpc, because
1. PAGEVEC_SIZE is 15 whereas pglist_data->mm_walk.batched is virtually unlimited.
2. Once folio_activate() reaches that limit, it takes the LRU lock on top of the PTL, which can be shared by multiple page tables on powerpc.

Quoted 6 lines
Quoted 3 lines

In fact, I think we try the opposite direction first, before arriving at any conclusions, i.e.,

    #define arch_has_hw_pte_young() radix_enabled()

The reason it is disabled on powerpc was that a reference bit update takes a pagefault on powerpc irrespective of the translation mode.

This is not true.

From "IBM POWER9 Processor User Manual":
https://openpowerfoundation.org/resources/ibmpower9usermanual/

4.10.14 Reference and Change Bits
...
When performing HPT translation, the hardware performs the R and C
bit updates nonatomically.
...

The radix case is more complex, and I'll leave it to you to interpret what it means:

Quoted 8 lines

From "Power ISA Version 3.0 B":
https://openpowerfoundation.org/specifications/isa/

5.7.12 Reference and Change Recording
...
For Radix Tree translation, the Reference and Change bits are set atomically.
...

Quoted 3 lines

it is atomic in that software use ldarx/stdcx to update these bits. Hardware/core won't update this directly even though Nest can update this directly without taking a fault. So for all purpose we can assume that on radix R/C bit is updated by page fault handler.

Thanks. To me, it sounds like stating a function provided by h/w, not a requirement for s/w. (IMO, the latter would be something like "software must/should set the bits atomically.) But I'll take your word for it.