Skip to contentSkip to conversation

Reduce TLB flushes under some specific conditions

RFC Superseded Byungchul Park 1 participant 7 messages Aug 17, 2023

Subsystems (10)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Byungchul Park

author reviewer
#0 lore raw

v2 cover Reduce TLB flushes under some specific conditions

Hi,

While I'm working with CXL memory, I have been facing migraion overhead esp. TLB shootdown on promotion or demotion between different tiers. Yeah.. most TLB shootdowns on migration through hinting fault can be avoided thanks to Huang Ying's work, commit 4d4b6d66db ("mm,unmap: avoid flushing TLB in batch if PTE is inaccessible").

However, it's only for ones using hinting fault. I thought it'd be much better if we have a general mechanism to reduce # of TLB flushes that we can apply to any type of migration. I tried it only for tiering migration for now tho.

I'm suggesting a mechanism to reduce TLB flushes by keeping source and destination of folios participated in the migrations until all TLB flushes required are done, only if those folios are not mapped with write permission PTE entries at all.

I saw the number of TLB full flush reduced over 50% and the performance a little bit improved but not that big with the workload I tested with, XSBench. However, I believe that it would help more with other ones or any real ones. It'd be appreciated to tell me if I'm missing something.

Byungchul

---

Changes from RFC:

1. Fix a bug triggered when a destination folio at the previous migration becomes a source folio at the next migration, before the folio gets handled properly so that the folio can play with another migration. There was inconsistency in the folio's state. Fixed it.

2. Split the patch set into more pieces so that the folks can review better. (Feedbacked by Nadav Amit)

3. Fix a wrong usage of barrier e.g. smp_mb__after_atomic(). (Feedbacked by Nadav Amit)

4. Tried to add sufficient comments to explain the patch set better. (Feedbacked by Nadav Amit)

Byungchul Park (6):
mm/rmap: Recognize non-writable TLB entries during TLB batch flush
mm: Defer TLB flush by keeping both src and dst folios at migration mm, migrc: Skip TLB flushes at the CPUs that already have been done mm, migrc: Ajust __zone_watermark_ok() with the amount of pending
folios
mm, migrc: Add a sysctl knob to enable/disable MIGRC mechanism mm, migrc: Implement internal allocator to minimize impact onto vm

 arch/x86/include/asm/tlbflush.h |   9 +
 arch/x86/mm/tlb.c               |  67 ++++++
 include/linux/mm.h              |  30 +++
 include/linux/mm_types.h        |  47 ++++
 include/linux/mm_types_task.h   |   4 +-
 include/linux/mmzone.h          |   6 +
 include/linux/sched.h           |   5 +
 init/Kconfig                    |  13 ++
 mm/internal.h                   |  14 ++
 mm/memory.c                     |  17 +-
 mm/migrate.c                    | 381 +++++++++++++++++++++++++++++++-
 mm/mm_init.c                    |   1 +
 mm/page_alloc.c                 |  19 ++
 mm/rmap.c                       | 133 ++++++++++-
 14 files changed, 734 insertions(+), 12 deletions(-)
Signature · 1 line
2.17.1
author reviewer
#1 lore raw

RFC v2 1/6 mm/rmap: Recognize non-writable TLB entries during TLB batch flush

Functionally, no change. This is a preparation for CONFIG_MIGRC that requires to recognize non-writable TLB entries and makes use of them to batch more aggressively or even skip TLB flushes.

While at it, changed struct tlbflush_unmap's ->flush_required(boolean) to ->nr_flush_required(int) in order to take into account not only whether it has been requested or not, but also the exact number of the requests. That will be used in CONFIG_MIGRC implementation.

---
 arch/x86/include/asm/tlbflush.h |  2 ++
 arch/x86/mm/tlb.c               |  7 +++++++
 include/linux/mm_types_task.h   |  4 ++--
 include/linux/sched.h           |  1 +
 mm/internal.h                   |  4 ++++
 mm/rmap.c                       | 29 ++++++++++++++++++++++++-----
 6 files changed, 40 insertions(+), 7 deletions(-)
arch/x86/include/asm/tlbflush.h +2 -0
index 75bfaa421030..63504cde364b 100644
@@ -279,6 +279,8 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch,
 }
 
 extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch);
+extern void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst,
+			       struct arch_tlbflush_unmap_batch *bsrc);
 
 static inline bool pte_flags_need_flush(unsigned long oldflags,
 					unsigned long newflags,
arch/x86/mm/tlb.c +7 -0
index 267acf27480a..69d145f1fff1 100644
@@ -1265,6 +1265,13 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 	put_cpu();
 }
 
+void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst,
+			struct arch_tlbflush_unmap_batch *bsrc)
+{
+	cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask);
+	cpumask_clear(&bsrc->cpumask);
+}
+
 /*
  * Blindly accessing user memory from NMI context can be dangerous
  * if we're in the middle of switching the current user task or
include/linux/mm_types_task.h +2 -2
index 5414b5c6a103..6f3bb757eb46 100644
@@ -59,8 +59,8 @@ struct tlbflush_unmap_batch {
 	 */
 	struct arch_tlbflush_unmap_batch arch;
 
-	/* True if a flush is needed. */
-	bool flush_required;
+	/* The number of flush requested. */
+	int nr_flush_required;
 
 	/*
 	 * If true then the PTE was dirty when unmapped. The entry must be
include/linux/sched.h +1 -0
index eed5d65b8d1f..2232b2cdfce8 100644
@@ -1322,6 +1322,7 @@ struct task_struct {
 #endif
 
 	struct tlbflush_unmap_batch	tlb_ubc;
+	struct tlbflush_unmap_batch	tlb_ubc_nowr;
 
 	/* Cache last used pipe for splice(): */
 	struct pipe_inode_info		*splice_pipe;
mm/internal.h +4 -0
index 68410c6d97ac..b90d516ad41f 100644
@@ -840,6 +840,7 @@ extern struct workqueue_struct *mm_percpu_wq;
 void try_to_unmap_flush(void);
 void try_to_unmap_flush_dirty(void);
 void flush_tlb_batched_pending(struct mm_struct *mm);
+void fold_ubc_nowr(void);
 #else
 static inline void try_to_unmap_flush(void)
 {
@@ -850,6 +851,9 @@ static inline void try_to_unmap_flush_dirty(void)
 static inline void flush_tlb_batched_pending(struct mm_struct *mm)
 {
 }
+static inline void fold_ubc_nowr(void)
+{
+}
 #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */
 
 extern const struct trace_print_flags pageflag_names[];
mm/rmap.c +24 -5
index 19392e090bec..d18460a48485 100644
@@ -605,6 +605,22 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio,
 }
 
 #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
+
+void fold_ubc_nowr(void)
+{
+	struct tlbflush_unmap_batch *tlb_ubc = &current->tlb_ubc;
+	struct tlbflush_unmap_batch *tlb_ubc_nowr = &current->tlb_ubc_nowr;
+
+	if (!tlb_ubc_nowr->nr_flush_required)
+		return;
+
+	arch_tlbbatch_fold(&tlb_ubc->arch, &tlb_ubc_nowr->arch);
+	tlb_ubc->writable = tlb_ubc->writable || tlb_ubc_nowr->writable;
+	tlb_ubc->nr_flush_required += tlb_ubc_nowr->nr_flush_required;
+	tlb_ubc_nowr->nr_flush_required = 0;
+	tlb_ubc_nowr->writable = false;
+}
+
 /*
  * Flush TLB entries for recently unmapped pages from remote CPUs. It is
  * important if a PTE was dirty when it was unmapped that it's flushed
@@ -615,11 +631,12 @@ void try_to_unmap_flush(void)
 {
 	struct tlbflush_unmap_batch *tlb_ubc = &current->tlb_ubc;
 
-	if (!tlb_ubc->flush_required)
+	fold_ubc_nowr();
+	if (!tlb_ubc->nr_flush_required)
 		return;
 
 	arch_tlbbatch_flush(&tlb_ubc->arch);
-	tlb_ubc->flush_required = false;
+	tlb_ubc->nr_flush_required = 0;
 	tlb_ubc->writable = false;
 }
 
@@ -627,8 +644,9 @@ void try_to_unmap_flush(void)
 void try_to_unmap_flush_dirty(void)
 {
 	struct tlbflush_unmap_batch *tlb_ubc = &current->tlb_ubc;
+	struct tlbflush_unmap_batch *tlb_ubc_nowr = &current->tlb_ubc_nowr;
 
-	if (tlb_ubc->writable)
+	if (tlb_ubc->writable || tlb_ubc_nowr->writable)
 		try_to_unmap_flush();
 }
 
@@ -644,15 +662,16 @@ void try_to_unmap_flush_dirty(void)
 
 static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval)
 {
-	struct tlbflush_unmap_batch *tlb_ubc = &current->tlb_ubc;
+	struct tlbflush_unmap_batch *tlb_ubc;
 	int batch;
 	bool writable = pte_dirty(pteval);
 
 	if (!pte_accessible(mm, pteval))
 		return;
 
+	tlb_ubc = pte_write(pteval) ? &current->tlb_ubc : &current->tlb_ubc_nowr;
 	arch_tlbbatch_add_mm(&tlb_ubc->arch, mm);
-	tlb_ubc->flush_required = true;
+	tlb_ubc->nr_flush_required += 1;
 
 	/*
 	 * Ensure compiler does not re-order the setting of tlb_flush_batched
Signature · 1 line
2.17.1
author reviewer
#2 lore raw

RFC v2 2/6 mm: Defer TLB flush by keeping both src and dst folios at migration

In reply to Byungchul Park

Implementation of CONFIG_MIGRC that stands for 'Migration Read Copy'. We always face the migration overhead at either promotion or demotion, while working with tiered memory e.g. CXL memory and found out TLB shootdown is a quite big one that is needed to get rid of if possible.

Fortunately, TLB flush can be defered or even skipped if both source and destination of folios during migration are kept until all TLB flushes required will have been done, of course, only if the target PTE entries have read only permission, more precisely speaking, don't have write permission. Otherwise, no doubt the folio might get messed up.

To achieve that:

1. For the folios that map only to non-writable TLB entries, prevent TLB flush at migration by keeping both source and destination folios, which will be handled later at a better time.

2. When any non-writable TLB entry changes to writable e.g. through fault handler, give up CONFIG_MIGRC mechanism so as to perform TLB flush required right away.

The measurement result:

Architecture - x86_64
QEMU - kvm enabled, host cpu, 2nodes {(4cpus, 2GB), (cpuless, 6GB)} Linux Kernel - v6.4, numa balancing tiering on, demotion enabled Benchmark - XSBench with no parameter changed

run 'perf stat' using events:
1) itlb.itlb_flush
2) tlb_flush.dtlb_thread
3) tlb_flush.stlb_any

run 'cat /proc/vmstat' and pick up:
1) pgdemote_kswapd
2) numa_pages_migrated
3) pgmigrate_success
4) nr_tlb_remote_flush
5) nr_tlb_remote_flush_received
6) nr_tlb_local_flush_all
7) nr_tlb_local_flush_one

BEFORE - mainline v6.4
------------------------------------------

   $ perf stat -e itlb.itlb_flush,tlb_flush.dtlb_thread,tlb_flush.stlb_any ./XSBench

Performance counter stats for './XSBench':

      426856       itlb.itlb_flush
      6900414      tlb_flush.dtlb_thread
      7303137      tlb_flush.stlb_any

33.500486566 seconds time elapsed
92.852128000 seconds user
10.526718000 seconds sys

   $ cat /proc/vmstat

...
pgdemote_kswapd 1052596
numa_pages_migrated 1052359
pgmigrate_success 2161846
nr_tlb_remote_flush 72370
nr_tlb_remote_flush_received 213711
nr_tlb_local_flush_all 3385
nr_tlb_local_flush_one 198679
...

AFTER - mainline v6.4 + CONFIG_MIGRC
------------------------------------------

   $ perf stat -e itlb.itlb_flush,tlb_flush.dtlb_thread,tlb_flush.stlb_any ./XSBench

Performance counter stats for './XSBench':

      179537       itlb.itlb_flush
      6131135      tlb_flush.dtlb_thread
      6920979      tlb_flush.stlb_any

30.396700625 seconds time elapsed
80.331252000 seconds user
10.303761000 seconds sys

   $ cat /proc/vmstat

...
pgdemote_kswapd 1044602
numa_pages_migrated 1044202
pgmigrate_success 2157808
nr_tlb_remote_flush 30453
nr_tlb_remote_flush_received 88840
nr_tlb_local_flush_all 3039
nr_tlb_local_flush_one 198875
...

---
 arch/x86/include/asm/tlbflush.h |   1 +
 arch/x86/mm/tlb.c               |   5 +
 include/linux/mm.h              |  28 +++
 include/linux/mm_types.h        |  47 +++++
 include/linux/mmzone.h          |   3 +
 include/linux/sched.h           |   4 +
 init/Kconfig                    |  13 ++
 mm/internal.h                   |  10 ++
 mm/memory.c                     |  17 +-
 mm/migrate.c                    | 296 +++++++++++++++++++++++++++++++-
 mm/mm_init.c                    |   1 +
 mm/page_alloc.c                 |   3 +
 mm/rmap.c                       | 103 +++++++++++
 13 files changed, 526 insertions(+), 5 deletions(-)
arch/x86/include/asm/tlbflush.h +1 -0
index 63504cde364b..752d72ea209b 100644
@@ -279,6 +279,7 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch,
 }
 
 extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch);
+extern void arch_tlbbatch_clean(struct arch_tlbflush_unmap_batch *batch);
 extern void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst,
 			       struct arch_tlbflush_unmap_batch *bsrc);
 
arch/x86/mm/tlb.c +5 -0
index 69d145f1fff1..2dabf0f340fb 100644
@@ -1265,6 +1265,11 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 	put_cpu();
 }
 
+void arch_tlbbatch_clean(struct arch_tlbflush_unmap_batch *batch)
+{
+	cpumask_clear(&batch->cpumask);
+}
+
 void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst,
 			struct arch_tlbflush_unmap_batch *bsrc)
 {
include/linux/mm.h +28 -0
index 27ce77080c79..1ceec7f3591e 100644
@@ -3816,4 +3816,32 @@ madvise_set_anon_name(struct mm_struct *mm, unsigned long start,
 }
 #endif
 
+#ifdef CONFIG_MIGRC
+void migrc_init_page(struct page *p);
+bool migrc_pending(struct folio *f);
+void migrc_shrink(struct llist_head *h);
+void migrc_req_start(void);
+void migrc_req_end(void);
+bool migrc_req_processing(void);
+bool migrc_try_flush_free_folios(void);
+void migrc_try_flush_free_folios_dirty(void);
+struct migrc_req *fold_ubc_nowr_to_migrc(void);
+void free_migrc_req(struct migrc_req *req);
+
+extern atomic_t migrc_gen;
+extern struct llist_head migrc_reqs;
+extern struct llist_head migrc_reqs_dirty;
+#else
+static inline void migrc_init_page(struct page *p) {}
+static inline bool migrc_pending(struct folio *f) { return false; }
+static inline void migrc_shrink(struct llist_head *h) {}
+static inline void migrc_req_start(void) {}
+static inline void migrc_req_end(void) {}
+static inline bool migrc_req_processing(void) { return false; }
+static inline bool migrc_try_flush_free_folios(void) { return false; }
+static inline void migrc_try_flush_free_folios_dirty(void) {}
+static inline struct migrc_req *fold_ubc_nowr_to_migrc(void) { return NULL; }
+static inline void free_migrc_req(struct migrc_req *req) {}
+#endif
+
 #endif /* _LINUX_MM_H */
include/linux/mm_types.h +47 -0

57 lines not on this page · raw message

include/linux/mmzone.h +3 -0
index a4889c9d4055..6d645beaf7a6 100644
@@ -1371,6 +1371,9 @@ typedef struct pglist_data {
 #ifdef CONFIG_MEMORY_FAILURE
 	struct memory_failure_stats mf_stats;
 #endif
+#ifdef CONFIG_MIGRC
+	atomic_t migrc_pending_nr;
+#endif
 } pg_data_t;
 
 #define node_present_pages(nid)	(NODE_DATA(nid)->node_present_pages)
include/linux/sched.h +4 -0
index 2232b2cdfce8..d0a46089959d 100644
@@ -1323,6 +1323,10 @@ struct task_struct {
 
 	struct tlbflush_unmap_batch	tlb_ubc;
 	struct tlbflush_unmap_batch	tlb_ubc_nowr;
+#ifdef CONFIG_MIGRC
+	struct migrc_req		*mreq;
+	struct migrc_req		*mreq_dirty;
+#endif
 
 	/* Cache last used pipe for splice(): */
 	struct pipe_inode_info		*splice_pipe;
init/Kconfig +13 -0
index 32c24950c4ce..9f9d0f7e15d2 100644
@@ -907,6 +907,19 @@ config NUMA_BALANCING_DEFAULT_ENABLED
 	  If set, automatic NUMA balancing will be enabled if running on a NUMA
 	  machine.
 
+config MIGRC
+	bool "Deferring TLB flush by keeping read copies on migration"
+	depends on ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
+	depends on NUMA_BALANCING
+	default n
+	help
+	  TLB flush is necessary when PTE changes by migration. However,
+	  TLB flush can be deferred if both copies of the src page and
+	  the dst page are kept until TLB flush if they are non-writable.
+	  System performance will be improved especially in case that
+	  promotion and demotion types of migrations are heavily
+	  happening.
+
 menuconfig CGROUPS
 	bool "Control Group support"
 	select KERNFS
mm/internal.h +10 -0
index b90d516ad41f..a8e3168614d6 100644
@@ -841,6 +841,8 @@ void try_to_unmap_flush(void);
 void try_to_unmap_flush_dirty(void);
 void flush_tlb_batched_pending(struct mm_struct *mm);
 void fold_ubc_nowr(void);
+int nr_flush_required(void);
+int nr_flush_required_nowr(void);
 #else
 static inline void try_to_unmap_flush(void)
 {
@@ -854,6 +856,14 @@ static inline void flush_tlb_batched_pending(struct mm_struct *mm)
 static inline void fold_ubc_nowr(void)
 {
 }
+static inline int nr_flush_required(void)
+{
+	return 0;
+}
+static inline int nr_flush_required_nowr(void)
+{
+	return 0;
+}
 #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */
 
 extern const struct trace_print_flags pageflag_names[];
mm/memory.c +14 -3
index f69fbc251198..066b7d5b7217 100644
@@ -3345,6 +3345,20 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
 
 	vmf->page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
 
+	if (vmf->page)
+		folio = page_folio(vmf->page);
+
+	/*
+	 * This folio has its read copy to prevent inconsistency while
+	 * deferring TLB flushes. However, the problem might arise if
+	 * it's going to become writable.
+	 *
+	 * To prevent it, give up the deferring TLB flushes and perform
+	 * TLB flush right away.
+	 */
+	if (folio && migrc_pending(folio))
+		migrc_try_flush_free_folios();
+
 	/*
 	 * Shared mapping: we are guaranteed to have VM_WRITE and
 	 * FAULT_FLAG_WRITE set at this point.
@@ -3362,9 +3376,6 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
 		return wp_page_shared(vmf);
 	}
 
-	if (vmf->page)
-		folio = page_folio(vmf->page);
-
 	/*
 	 * Private mapping: create an exclusive anonymous page copy if reuse
 	 * is impossible. We might miss VM_WRITE for FOLL_FORCE handling.
mm/migrate.c +294 -2

366 lines not on this page · raw message

mm/mm_init.c +1 -0
index 7f7f9c677854..87cbddc7d780 100644
@@ -558,6 +558,7 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn,
 	page_mapcount_reset(page);
 	page_cpupid_reset_last(page);
 	page_kasan_tag_reset(page);
+	migrc_init_page(page);
 
 	INIT_LIST_HEAD(&page->lru);
 #ifdef WANT_PAGE_VIRTUAL
mm/page_alloc.c +3 -0
index 47421bedc12b..c51cbdb45d86 100644
@@ -1730,6 +1730,9 @@ inline void post_alloc_hook(struct page *page, unsigned int order,
 
 	set_page_owner(page, order, gfp_flags);
 	page_table_check_alloc(page, order);
+
+	for (i = 0; i != 1 << order; ++i)
+		migrc_init_page(page + i);
 }
 
 static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
mm/rmap.c +103 -0

121 lines not on this page · raw message

Signature · 1 line
2.17.1
author reviewer
#3 lore raw

RFC v2 3/6 mm, migrc: Skip TLB flushes at the CPUs that already have been done

In reply to Byungchul Park

TLB flushes can be skipped if TLB flushes requested have been done by any reason, which doesn't have to be done from migrations. It can be tracked by keeping timestamp(= migrc_gen) when it's requested and when it's triggered.

---
 arch/x86/include/asm/tlbflush.h |  6 ++++
 arch/x86/mm/tlb.c               | 55 +++++++++++++++++++++++++++++++++
 mm/migrate.c                    | 10 ++++++
 mm/rmap.c                       |  1 +
 4 files changed, 72 insertions(+)
arch/x86/include/asm/tlbflush.h +6 -0
index 752d72ea209b..da987c15049e 100644
@@ -283,6 +283,12 @@ extern void arch_tlbbatch_clean(struct arch_tlbflush_unmap_batch *batch);
 extern void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst,
 			       struct arch_tlbflush_unmap_batch *bsrc);
 
+#ifdef CONFIG_MIGRC
+extern void arch_migrc_adj(struct arch_tlbflush_unmap_batch *batch, int gen);
+#else
+static inline void arch_migrc_adj(struct arch_tlbflush_unmap_batch *batch, int gen) {}
+#endif
+
 static inline bool pte_flags_need_flush(unsigned long oldflags,
 					unsigned long newflags,
 					bool ignore_access)
arch/x86/mm/tlb.c +55 -0

70 lines not on this page · raw message

mm/migrate.c +10 -0
index f9446f5b312a..c7b72d275b2a 100644
@@ -2053,6 +2053,16 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 	stats->nr_thp_failed += thp_retry;
 	stats->nr_failed_pages += nr_retry_pages;
 move:
+	/*
+	 * Should be prior to try_to_unmap_flush() so that
+	 * migrc_try_flush_free_folios() that will be called later
+	 * can take benefit from the TLB flushes in try_to_unmap_flush().
+	 *
+	 * migrc_req_end() will store the timestamp for pending, and
+	 * TLB flushes will also store the timestamp for TLB flush so
+	 * that unnecessary TLB flushes can be skipped using the time
+	 * information.
+	 */
 	if (migrc_cond1)
 		migrc_req_end();
 
mm/rmap.c +1 -0
index 0652d25206ee..2ae1b1324f84 100644
@@ -627,6 +627,7 @@ static bool __migrc_try_flush_free_folios(struct llist_head *h)
 	llist_for_each_entry_safe(req, req2, reqs, llnode) {
 		struct llist_node *n;
 
+		arch_migrc_adj(&req->arch, req->gen);
 		arch_tlbbatch_fold(&arch, &req->arch);
 
 		n = llist_del_all(&req->pages);
Signature · 1 line
2.17.1
author reviewer
#4 lore raw

RFC v2 4/6 mm, migrc: Ajust __zone_watermark_ok() with the amount of pending folios

In reply to Byungchul Park

CONFIG_MIGRC duplicates folios participated in migration to avoid TLB flushes and provide a consistent view to CPUs that are still caching its old mapping in TLB. However, the duplicated folios can be freed and available right away through appropreate TLB flushes if needed.

Adjusted watermark check routine, __zone_watermark_ok(), with the number of duplicated folios and made it perform TLB flushes and free the duplicated folios if page allocation routine is in trouble due to memory pressure, even more aggresively for high order allocation.

---
 include/linux/mm.h     |  2 ++
 include/linux/mmzone.h |  3 +++
 mm/migrate.c           | 12 ++++++++++++
 mm/page_alloc.c        | 16 ++++++++++++++++
 4 files changed, 33 insertions(+)
include/linux/mm.h +2 -0
index 1ceec7f3591e..9df393074e6a 100644
@@ -3827,6 +3827,7 @@ bool migrc_try_flush_free_folios(void);
 void migrc_try_flush_free_folios_dirty(void);
 struct migrc_req *fold_ubc_nowr_to_migrc(void);
 void free_migrc_req(struct migrc_req *req);
+int migrc_pending_nr_in_zone(struct zone *z);
 
 extern atomic_t migrc_gen;
 extern struct llist_head migrc_reqs;
@@ -3842,6 +3843,7 @@ static inline bool migrc_try_flush_free_folios(void) { return false; }
 static inline void migrc_try_flush_free_folios_dirty(void) {}
 static inline struct migrc_req *fold_ubc_nowr_to_migrc(void) { return NULL; }
 static inline void free_migrc_req(struct migrc_req *req) {}
+static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; }
 #endif
 
 #endif /* _LINUX_MM_H */
include/linux/mmzone.h +3 -0
index 6d645beaf7a6..1ec79bb63ba7 100644
@@ -958,6 +958,9 @@ struct zone {
 	/* Zone statistics */
 	atomic_long_t		vm_stat[NR_VM_ZONE_STAT_ITEMS];
 	atomic_long_t		vm_numa_event[NR_VM_NUMA_EVENT_ITEMS];
+#ifdef CONFIG_MIGRC
+	atomic_t		migrc_pending_nr;
+#endif
 } ____cacheline_internodealigned_in_smp;
 
 enum pgdat_flags {
mm/migrate.c +12 -0
index c7b72d275b2a..badef3d89c6c 100644
@@ -117,9 +117,12 @@ void migrc_shrink(struct llist_head *h)
 	llist_for_each_entry_safe(p, p2, n, migrc_node) {
 		if (p->migrc_state == MIGRC_SRC_PENDING) {
 			struct pglist_data *node;
+			struct zone *zone;
 
 			node = NODE_DATA(page_to_nid(p));
+			zone = page_zone(p);
 			atomic_dec(&node->migrc_pending_nr);
+			atomic_dec(&zone->migrc_pending_nr);
 		}
 
 		if (WARN_ON(!migrc_pending(page_folio(p))))
@@ -172,6 +175,7 @@ static void migrc_expand_req(struct folio *fsrc, struct folio *fdst)
 {
 	struct migrc_req *req;
 	struct pglist_data *node;
+	struct zone *zone;
 
 	req = fold_ubc_nowr_to_migrc();
 	if (!req)
@@ -190,7 +194,9 @@ static void migrc_expand_req(struct folio *fsrc, struct folio *fdst)
 		req->last = &fsrc->page.migrc_node;
 
 	node = NODE_DATA(folio_nid(fsrc));
+	zone = page_zone(&fsrc->page);
 	atomic_inc(&node->migrc_pending_nr);
+	atomic_inc(&zone->migrc_pending_nr);
 
 	if (migrc_is_full(folio_nid(fsrc)))
 		migrc_try_flush_free_folios();
@@ -275,6 +281,12 @@ bool migrc_req_processing(void)
 {
 	return current->mreq && current->mreq_dirty;
 }
+
+int migrc_pending_nr_in_zone(struct zone *z)
+{
+	return atomic_read(&z->migrc_pending_nr);
+
+}
 #else
 static inline bool migrc_src_pending(struct folio *f) { return false; }
 static inline bool migrc_dst_pending(struct folio *f) { return false; }
mm/page_alloc.c +16 -0
index c51cbdb45d86..9f791c0fa15d 100644
@@ -3179,6 +3179,11 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
 	long min = mark;
 	int o;
 
+	/*
+	 * There are pages that can be freed by migrc_try_flush_free_folios().
+	 */
+	free_pages += migrc_pending_nr_in_zone(z);
+
 	/* free_pages may go negative - that's OK */
 	free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags);
 
@@ -4257,6 +4262,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	unsigned int zonelist_iter_cookie;
 	int reserve_flags;
 
+	migrc_try_flush_free_folios();
 restart:
 	compaction_retries = 0;
 	no_progress_loops = 0;
@@ -4772,6 +4778,16 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid,
 	if (likely(page))
 		goto out;
 
+	if (order && migrc_try_flush_free_folios()) {
+		/*
+		 * Try again after freeing migrc's pending pages in case
+		 * of high order allocation.
+		 */
+		page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac);
+		if (likely(page))
+			goto out;
+	}
+
 	alloc_gfp = gfp;
 	ac.spread_dirty_pages = false;
 
Signature · 1 line
2.17.1
author reviewer
#5 lore raw

RFC v2 5/6 mm, migrc: Add a sysctl knob to enable/disable MIGRC mechanism

In reply to Byungchul Park

Add a sysctl knob, '/proc/sys/vm/migrc_enable' to switch on/off migrc.

---
 mm/migrate.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 46 insertions(+), 2 deletions(-)
mm/migrate.c +46 -2
index badef3d89c6c..c57536a0b2a6 100644
@@ -59,6 +59,48 @@
 #include "internal.h"
 
 #ifdef CONFIG_MIGRC
+static int sysctl_migrc_enable = 1;
+#ifdef CONFIG_SYSCTL
+static int sysctl_migrc_enable_handler(struct ctl_table *table, int write,
+		void *buffer, size_t *lenp, loff_t *ppos)
+{
+	struct ctl_table t;
+	int err;
+	int enabled = sysctl_migrc_enable;
+
+	if (write && !capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	t = *table;
+	t.data = &enabled;
+	err = proc_dointvec_minmax(&t, write, buffer, lenp, ppos);
+	if (err < 0)
+		return err;
+	if (write)
+		sysctl_migrc_enable = enabled;
+	return err;
+}
+
+static struct ctl_table migrc_sysctls[] = {
+	{
+		.procname	= "migrc_enable",
+		.data		= NULL, /* filled in by handler */
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= sysctl_migrc_enable_handler,
+		.extra1         = SYSCTL_ZERO,
+		.extra2         = SYSCTL_ONE,
+	},
+	{}
+};
+
+static int __init migrc_sysctl_init(void)
+{
+	register_sysctl_init("vm", migrc_sysctls);
+	return 0;
+}
+late_initcall(migrc_sysctl_init);
+#endif
 
 /*
  * TODO: Yeah, it's a non-sense magic number. This simple value manages
@@ -288,6 +330,7 @@ int migrc_pending_nr_in_zone(struct zone *z)
 
 }
 #else
+static const int sysctl_migrc_enable;
 static inline bool migrc_src_pending(struct folio *f) { return false; }
 static inline bool migrc_dst_pending(struct folio *f) { return false; }
 static inline bool migrc_is_full(int nid) { return true; }
@@ -1878,8 +1921,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
 			!list_empty(from) && !list_is_singular(from));
 
-	migrc_cond1 = (reason == MR_DEMOTION && current_is_kswapd()) ||
-		      (reason == MR_NUMA_MISPLACED);
+	migrc_cond1 = sysctl_migrc_enable &&
+		      ((reason == MR_DEMOTION && current_is_kswapd()) ||
+		      (reason == MR_NUMA_MISPLACED));
 
 	if (migrc_cond1)
 		migrc_req_start();
Signature · 1 line
2.17.1
author reviewer
#6 lore raw

RFC v2 6/6 mm, migrc: Implement internal allocator to minimize impact onto vm

In reply to Byungchul Park

(Not sure if this patch works meaningfully. Ignore if not.)

---
 mm/migrate.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
mm/migrate.c +21 -2
index c57536a0b2a6..6b5113d5a1e2 100644
@@ -122,14 +122,33 @@ enum {
 	MIGRC_DST_PENDING,
 };
 
+#define MAX_MIGRC_REQ_NR	4096
+static struct migrc_req migrc_req_pool_static[MAX_MIGRC_REQ_NR];
+static atomic_t migrc_req_pool_idx = ATOMIC_INIT(-1);
+static LLIST_HEAD(migrc_req_pool_llist);
+static DEFINE_SPINLOCK(migrc_req_pool_lock);
+
 static struct migrc_req *alloc_migrc_req(void)
 {
-	return kmalloc(sizeof(struct migrc_req), GFP_KERNEL);
+	int idx = atomic_read(&migrc_req_pool_idx);
+	struct llist_node *n;
+
+	if (idx < MAX_MIGRC_REQ_NR - 1) {
+		idx = atomic_inc_return(&migrc_req_pool_idx);
+		if (idx < MAX_MIGRC_REQ_NR)
+			return migrc_req_pool_static + idx;
+	}
+
+	spin_lock(&migrc_req_pool_lock);
+	n = llist_del_first(&migrc_req_pool_llist);
+	spin_unlock(&migrc_req_pool_lock);
+
+	return n ? llist_entry(n, struct migrc_req, llnode) : NULL;
 }
 
 void free_migrc_req(struct migrc_req *req)
 {
-	kfree(req);
+	llist_add(&req->llnode, &migrc_req_pool_llist);
 }
 
 static bool migrc_is_full(int nid)
Signature · 1 line
2.17.1