Skip to contentSkip to conversation

Reduce NUMA balance caused TLB-shootdowns in a VM

RFC PATCH In review Yan Zhao 10 participants 50 messages Aug 10, 2023 – Sep 4, 2023

Subsystems (7)
Memory management x86 KVM MM core +3 more subsystems
lore
Share & tools
lore

Yan Zhao

author
#0 lore raw

v2 cover Reduce NUMA balance caused TLB-shootdowns in a VM

This is an RFC series trying to fix the issue of unnecessary NUMA protection and TLB-shootdowns found in VMs with assigned devices or VFIO mediated devices during NUMA balance.

For VMs with assigned devices or VFIO mediated devices, all or part of guest memory are pinned for long-term.

Auto NUMA balancing will periodically selects VMAs of a process and change protections to PROT_NONE even though some or all pages in the selected ranges are long-term pinned for DMAs, which is true for VMs with assigned devices or VFIO mediated devices.

Though this will not cause real problem because NUMA migration will ultimately reject migration of those kind of pages and restore those PROT_NONE PTEs, it causes KVM's secondary MMU to be zapped periodically with equal SPTEs finally faulted back, wasting CPU cycles and generating unnecessary TLB-shootdowns.

This series first introduces a new flag MMU_NOTIFIER_RANGE_NUMA in patch 1 to work with mmu notifier event type MMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.

Patch 2 skips setting PROT_NONE to long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.

Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be PROT_NONE protected.

Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.

Changelog:
RFC v1 --> v2:
1. added patch 3-4 to introduce a new callback .numa_protect()
2. Rather than have KVM duplicate logic to check if a page is pinned for long-term, let KVM depend on the new callback .numa_protect() to do the page unmap in secondary MMU for NUMA migration purpose.

RFC v1:
https://lore.kernel.org/all/[email protected]/

Yan Zhao (5):
mm/mmu_notifier: introduce a new mmu notifier flag
MMU_NOTIFIER_RANGE_NUMA
mm: don't set PROT_NONE to maybe-dma-pinned pages for NUMA-migrate
purpose
mm/mmu_notifier: introduce a new callback .numa_protect mm/autonuma: call .numa_protect() when page is protected for NUMA
migrate
KVM: Unmap pages only when it's indeed protected for NUMA migration

 include/linux/mmu_notifier.h | 16 ++++++++++++++++
 mm/huge_memory.c             |  6 ++++++
 mm/mmu_notifier.c            | 18 ++++++++++++++++++
 mm/mprotect.c                | 10 +++++++++-
 virt/kvm/kvm_main.c          | 25 ++++++++++++++++++++++---
 5 files changed, 71 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.1
author
#1 lore raw

RFC PATCH v2 1/5 mm/mmu_notifier: introduce a new mmu notifier flag MMU_NOTIFIER_RANGE_NUMA

Introduce a new mmu notifier flag MMU_NOTIFIER_RANGE_NUMA to indicate the notification of MMU_NOTIFY_PROTECTION_VMA is for NUMA balance purpose specifically.

So that, the subscriber of mmu notifier, like KVM, can recognize this type of notification and do numa protection specific operations in the handler.

---
 include/linux/mmu_notifier.h | 1 +
 mm/mprotect.c                | 4 +++-
 2 files changed, 4 insertions(+), 1 deletion(-)
include/linux/mmu_notifier.h +1 -0
index 64a3e051c3c4..a6dc829a4bce 100644
@@ -60,6 +60,7 @@ enum mmu_notifier_event {
 };
 
 #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)
+#define MMU_NOTIFIER_RANGE_NUMA (1 << 1)
 
 struct mmu_notifier_ops {
 	/*
mm/mprotect.c +3 -1
index 6f658d483704..cb99a7d66467 100644
@@ -381,7 +381,9 @@ static inline long change_pmd_range(struct mmu_gather *tlb,
 		/* invoke the mmu notifier if the pmd is populated */
 		if (!range.start) {
 			mmu_notifier_range_init(&range,
-				MMU_NOTIFY_PROTECTION_VMA, 0,
+				MMU_NOTIFY_PROTECTION_VMA,
+				cp_flags & MM_CP_PROT_NUMA ?
+				MMU_NOTIFIER_RANGE_NUMA : 0,
 				vma->vm_mm, addr, end);
 			mmu_notifier_invalidate_range_start(&range);
 		}
Signature · 1 line
2.17.1
author
#2 lore raw

RFC PATCH v2 2/5 mm: don't set PROT_NONE to maybe-dma-pinned pages for NUMA-migrate purpose

In reply to Yan Zhao

Don't set PROT_NONE for exclusive anonymas and maybe-dma-pinned pages for NUMA migration purpose.

For exclusive anonymas and page_maybe_dma_pinned() pages, NUMA-migration will eventually drop migration of those pages in try_to_migrate_one(). (i.e. after -EBUSY returned in page_try_share_anon_rmap()).

So, skip setting PROT_NONE to those kind of pages earlier in change_protection_range() phase to avoid later futile page faults, detections, and restoration to original PTEs/PMDs.

---
 mm/huge_memory.c | 5 +++++
 mm/mprotect.c    | 5 +++++
 2 files changed, 10 insertions(+)
mm/huge_memory.c +5 -0
index eb3678360b97..a71cf686e3b2 100644
@@ -1875,6 +1875,11 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 			goto unlock;
 
 		page = pmd_page(*pmd);
+
+		if (PageAnon(page) && PageAnonExclusive(page) &&
+		    page_maybe_dma_pinned(page))
+			goto unlock;
+
 		toptier = node_is_toptier(page_to_nid(page));
 		/*
 		 * Skip scanning top tier node if normal numa
mm/mprotect.c +5 -0
index cb99a7d66467..a1f63df34b86 100644
@@ -146,6 +146,11 @@ static long change_pte_range(struct mmu_gather *tlb,
 				nid = page_to_nid(page);
 				if (target_node == nid)
 					continue;
+
+				if (PageAnon(page) && PageAnonExclusive(page) &&
+				    page_maybe_dma_pinned(page))
+					continue;
+
 				toptier = node_is_toptier(nid);
 
 				/*
Signature · 1 line
2.17.1
author
#3 lore raw

RFC PATCH v2 3/5 mm/mmu_notifier: introduce a new callback .numa_protect

In reply to Yan Zhao

This .numa_protect callback is called when PROT_NONE is set for sure on a PTE or a huge PMD for numa migration purpose.

With this callback, subscriber of mmu notifier, (e.g. KVM), can unmap NUMA migration protected pages only in the handler, rather than unmap a wider range containing pages that are obvious none-NUMA-migratble.

---
 include/linux/mmu_notifier.h | 15 +++++++++++++++
 mm/mmu_notifier.c            | 18 ++++++++++++++++++
 2 files changed, 33 insertions(+)
include/linux/mmu_notifier.h +15 -0
index a6dc829a4bce..a173db83b071 100644
@@ -132,6 +132,10 @@ struct mmu_notifier_ops {
 			   unsigned long address,
 			   pte_t pte);
 
+	void (*numa_protect)(struct mmu_notifier *subscription,
+			     struct mm_struct *mm,
+			     unsigned long start,
+			     unsigned long end);
 	/*
 	 * invalidate_range_start() and invalidate_range_end() must be
 	 * paired and are called only when the mmap_lock and/or the
@@ -395,6 +399,9 @@ extern int __mmu_notifier_test_young(struct mm_struct *mm,
 				     unsigned long address);
 extern void __mmu_notifier_change_pte(struct mm_struct *mm,
 				      unsigned long address, pte_t pte);
+extern void __mmu_notifier_numa_protect(struct mm_struct *mm,
+					unsigned long start,
+					unsigned long end);
 extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r);
 extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r,
 				  bool only_end);
@@ -448,6 +455,14 @@ static inline void mmu_notifier_change_pte(struct mm_struct *mm,
 		__mmu_notifier_change_pte(mm, address, pte);
 }
 
+static inline void mmu_notifier_numa_protect(struct mm_struct *mm,
+					     unsigned long start,
+					     unsigned long end)
+{
+	if (mm_has_notifiers(mm))
+		__mmu_notifier_numa_protect(mm, start, end);
+}
+
 static inline void
 mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range)
 {
mm/mmu_notifier.c +18 -0
index 50c0dde1354f..fc96fbd46e1d 100644
@@ -382,6 +382,24 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,
 	return young;
 }
 
+void __mmu_notifier_numa_protect(struct mm_struct *mm,
+				 unsigned long start,
+				 unsigned long end)
+{
+	struct mmu_notifier *subscription;
+	int id;
+
+	id = srcu_read_lock(&srcu);
+	hlist_for_each_entry_rcu(subscription,
+				 &mm->notifier_subscriptions->list, hlist,
+				 srcu_read_lock_held(&srcu)) {
+		if (subscription->ops->numa_protect)
+			subscription->ops->numa_protect(subscription, mm, start,
+							end);
+	}
+	srcu_read_unlock(&srcu, id);
+}
+
 int __mmu_notifier_clear_young(struct mm_struct *mm,
 			       unsigned long start,
 			       unsigned long end)
Signature · 1 line
2.17.1
author
#4 lore raw

RFC PATCH v2 4/5 mm/autonuma: call .numa_protect() when page is protected for NUMA migrate

In reply to Yan Zhao

Call mmu notifier's callback .numa_protect() in change_pmd_range() when a page is ensured to be protected by PROT_NONE for NUMA migration purpose.

---
 mm/huge_memory.c | 1 +
 mm/mprotect.c    | 1 +
 2 files changed, 2 insertions(+)
mm/huge_memory.c +1 -0
index a71cf686e3b2..8ae56507da12 100644
@@ -1892,6 +1892,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
 		    !toptier)
 			xchg_page_access_time(page, jiffies_to_msecs(jiffies));
+		mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PMD_SIZE);
 	}
 	/*
 	 * In case prot_numa, we are under mmap_read_lock(mm). It's critical
mm/mprotect.c +1 -0
index a1f63df34b86..c401814b2992 100644
@@ -164,6 +164,7 @@ static long change_pte_range(struct mmu_gather *tlb,
 				    !toptier)
 					xchg_page_access_time(page,
 						jiffies_to_msecs(jiffies));
+				mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PAGE_SIZE);
 			}
 
 			oldpte = ptep_modify_prot_start(vma, addr, pte);
Signature · 1 line
2.17.1
#5 lore raw

Re: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate

kernel test robot noticed the following build errors:

Hi Yan,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on kvm/queue]
[also build test ERROR on mst-vhost/linux-next linus/master v6.5-rc5 next-20230809] [cannot apply to akpm-mm/mm-everything kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Yan-Zhao/mm-mmu_notifier-introduce-a-new-mmu-notifier-flag-MMU_NOTIFIER_RANGE_NUMA/20230810-172955
base:   https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue
patch link:    https://lore.kernel.org/r/20230810090048.26184-1-yan.y.zhao%40intel.com
patch subject: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate
config: um-allnoconfig (https://download.01.org/0day-ci/archive/20230810/[email protected]/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a)
reproduce: (https://download.01.org/0day-ci/archive/20230810/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from mm/mprotect.c:13:
In file included from include/linux/hugetlb.h:10:
In file included from include/linux/cgroup.h:26:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:547:31: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]

     547 |         val = __raw_readb(PCI_IOBASE + addr);
         |                           ~~~~~~~~~~ ^
   include/asm-generic/io.h:560:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     560 |         val = __le16_to_cpu((__le16 __force)__raw_readw(PCI_IOBASE + addr));
         |                                                         ~~~~~~~~~~ ^
   include/uapi/linux/byteorder/little_endian.h:37:51: note: expanded from macro '__le16_to_cpu'
      37 | #define __le16_to_cpu(x) ((__force __u16)(__le16)(x))
         |                                                   ^
   In file included from mm/mprotect.c:13:
   In file included from include/linux/hugetlb.h:10:
   In file included from include/linux/cgroup.h:26:
   In file included from include/linux/kernel_stat.h:9:
   In file included from include/linux/interrupt.h:11:
   In file included from include/linux/hardirq.h:11:
   In file included from arch/um/include/asm/hardirq.h:5:
   In file included from include/asm-generic/hardirq.h:17:
   In file included from include/linux/irq.h:20:
   In file included from include/linux/io.h:13:
   In file included from arch/um/include/asm/io.h:24:
   include/asm-generic/io.h:573:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     573 |         val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
         |                                                         ~~~~~~~~~~ ^
   include/uapi/linux/byteorder/little_endian.h:35:51: note: expanded from macro '__le32_to_cpu'
      35 | #define __le32_to_cpu(x) ((__force __u32)(__le32)(x))
         |                                                   ^
   In file included from mm/mprotect.c:13:
   In file included from include/linux/hugetlb.h:10:
   In file included from include/linux/cgroup.h:26:
   In file included from include/linux/kernel_stat.h:9:
   In file included from include/linux/interrupt.h:11:
   In file included from include/linux/hardirq.h:11:
   In file included from arch/um/include/asm/hardirq.h:5:
   In file included from include/asm-generic/hardirq.h:17:
   In file included from include/linux/irq.h:20:
   In file included from include/linux/io.h:13:
   In file included from arch/um/include/asm/io.h:24:
   include/asm-generic/io.h:584:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     584 |         __raw_writeb(value, PCI_IOBASE + addr);
         |                             ~~~~~~~~~~ ^
   include/asm-generic/io.h:594:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     594 |         __raw_writew((u16 __force)cpu_to_le16(value), PCI_IOBASE + addr);
         |                                                       ~~~~~~~~~~ ^
   include/asm-generic/io.h:604:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     604 |         __raw_writel((u32 __force)cpu_to_le32(value), PCI_IOBASE + addr);
         |                                                       ~~~~~~~~~~ ^
   include/asm-generic/io.h:692:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     692 |         readsb(PCI_IOBASE + addr, buffer, count);
         |                ~~~~~~~~~~ ^
   include/asm-generic/io.h:700:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     700 |         readsw(PCI_IOBASE + addr, buffer, count);
         |                ~~~~~~~~~~ ^
   include/asm-generic/io.h:708:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     708 |         readsl(PCI_IOBASE + addr, buffer, count);
         |                ~~~~~~~~~~ ^
   include/asm-generic/io.h:717:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     717 |         writesb(PCI_IOBASE + addr, buffer, count);
         |                 ~~~~~~~~~~ ^
   include/asm-generic/io.h:726:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     726 |         writesw(PCI_IOBASE + addr, buffer, count);
         |                 ~~~~~~~~~~ ^
   include/asm-generic/io.h:735:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
     735 |         writesl(PCI_IOBASE + addr, buffer, count);
         |                 ~~~~~~~~~~ ^
>> mm/mprotect.c:167:5: error: call to undeclared function 'mmu_notifier_numa_protect'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     167 |                                 mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PAGE_SIZE);
         |                                 ^

12 warnings and 1 error generated.

vim +/mmu_notifier_numa_protect +167 mm/mprotect.c

    82	
    83	static long change_pte_range(struct mmu_gather *tlb,
    84			struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr,
    85			unsigned long end, pgprot_t newprot, unsigned long cp_flags)
    86	{
    87		pte_t *pte, oldpte;
    88		spinlock_t *ptl;
    89		long pages = 0;
    90		int target_node = NUMA_NO_NODE;
    91		bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
    92		bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
    93		bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
    94	
    95		tlb_change_page_size(tlb, PAGE_SIZE);
    96		pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
    97		if (!pte)
    98			return -EAGAIN;
    99	
   100		/* Get target node for single threaded private VMAs */
   101		if (prot_numa && !(vma->vm_flags & VM_SHARED) &&
   102		    atomic_read(&vma->vm_mm->mm_users) == 1)
   103			target_node = numa_node_id();
   104	
   105		flush_tlb_batched_pending(vma->vm_mm);
   106		arch_enter_lazy_mmu_mode();
   107		do {
   108			oldpte = ptep_get(pte);
   109			if (pte_present(oldpte)) {
   110				pte_t ptent;
   111	
   112				/*
   113				 * Avoid trapping faults against the zero or KSM
   114				 * pages. See similar comment in change_huge_pmd.
   115				 */
   116				if (prot_numa) {
   117					struct page *page;
   118					int nid;
   119					bool toptier;
   120	
   121					/* Avoid TLB flush if possible */
   122					if (pte_protnone(oldpte))
   123						continue;
   124	
   125					page = vm_normal_page(vma, addr, oldpte);
   126					if (!page || is_zone_device_page(page) || PageKsm(page))
   127						continue;
   128	
   129					/* Also skip shared copy-on-write pages */
   130					if (is_cow_mapping(vma->vm_flags) &&
   131					    page_count(page) != 1)
   132						continue;
   133	
   134					/*
   135					 * While migration can move some dirty pages,
   136					 * it cannot move them all from MIGRATE_ASYNC
   137					 * context.
   138					 */
   139					if (page_is_file_lru(page) && PageDirty(page))
   140						continue;
   141	
   142					/*
   143					 * Don't mess with PTEs if page is already on the node
   144					 * a single-threaded process is running on.
   145					 */
   146					nid = page_to_nid(page);
   147					if (target_node == nid)
   148						continue;
   149	
   150					if (PageAnon(page) && PageAnonExclusive(page) &&
   151					    page_maybe_dma_pinned(page))
   152						continue;
   153	
   154					toptier = node_is_toptier(nid);
   155	
   156					/*
   157					 * Skip scanning top tier node if normal numa
   158					 * balancing is disabled
   159					 */
   160					if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
   161					    toptier)
   162						continue;
   163					if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
   164					    !toptier)
   165						xchg_page_access_time(page,
   166							jiffies_to_msecs(jiffies));
 > 167					mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PAGE_SIZE);

168 }
169
170 oldpte = ptep_modify_prot_start(vma, addr, pte);
171 ptent = pte_modify(oldpte, newprot);
172
173 if (uffd_wp)
174 ptent = pte_mkuffd_wp(ptent);
175 else if (uffd_wp_resolve)
176 ptent = pte_clear_uffd_wp(ptent);
177
178 /*
179 * In some writable, shared mappings, we might want
180 * to catch actual write access -- see
181 * vma_wants_writenotify().
182 *
183 * In all writable, private mappings, we have to
184 * properly handle COW.
185 *
186 * In both cases, we can sometimes still change PTEs
187 * writable and avoid the write-fault handler, for
188 * example, if a PTE is already dirty and no other
189 * COW or special handling is required.
190 */
191 if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
192 !pte_write(ptent) &&
193 can_change_pte_writable(vma, addr, ptent))
194 ptent = pte_mkwrite(ptent);
195
196 ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent);
197 if (pte_needs_flush(oldpte, ptent))
198 tlb_flush_pte_range(tlb, addr, PAGE_SIZE);
199 pages++;
200 } else if (is_swap_pte(oldpte)) {
201 swp_entry_t entry = pte_to_swp_entry(oldpte);
202 pte_t newpte;
203
204 if (is_writable_migration_entry(entry)) {
205 struct page *page = pfn_swap_entry_to_page(entry);
206
207 /*
208 * A protection check is difficult so
209 * just be safe and disable write
210 */
211 if (PageAnon(page))
212 entry = make_readable_exclusive_migration_entry(
213 swp_offset(entry));
214 else
215 entry = make_readable_migration_entry(swp_offset(entry));
216 newpte = swp_entry_to_pte(entry);
217 if (pte_swp_soft_dirty(oldpte))
218 newpte = pte_swp_mksoft_dirty(newpte);
219 } else if (is_writable_device_private_entry(entry)) {
220 /*
221 * We do not preserve soft-dirtiness. See
222 * copy_one_pte() for explanation.
223 */
224 entry = make_readable_device_private_entry(
225 swp_offset(entry));
226 newpte = swp_entry_to_pte(entry);
227 if (pte_swp_uffd_wp(oldpte))
228 newpte = pte_swp_mkuffd_wp(newpte);
229 } else if (is_writable_device_exclusive_entry(entry)) {
230 entry = make_readable_device_exclusive_entry(
231 swp_offset(entry));
232 newpte = swp_entry_to_pte(entry);
233 if (pte_swp_soft_dirty(oldpte))
234 newpte = pte_swp_mksoft_dirty(newpte);
235 if (pte_swp_uffd_wp(oldpte))
236 newpte = pte_swp_mkuffd_wp(newpte);
237 } else if (is_pte_marker_entry(entry)) {
238 /*
239 * Ignore swapin errors unconditionally,
240 * because any access should sigbus anyway.
241 */
242 if (is_swapin_error_entry(entry))
243 continue;
244 /*
245 * If this is uffd-wp pte marker and we'd like
246 * to unprotect it, drop it; the next page
247 * fault will trigger without uffd trapping.
248 */
249 if (uffd_wp_resolve) {
250 pte_clear(vma->vm_mm, addr, pte);
251 pages++;
252 }
253 continue;
254 } else {
255 newpte = oldpte;
256 }
257
258 if (uffd_wp)
259 newpte = pte_swp_mkuffd_wp(newpte);
260 else if (uffd_wp_resolve)
261 newpte = pte_swp_clear_uffd_wp(newpte);
262
263 if (!pte_same(oldpte, newpte)) {
264 set_pte_at(vma->vm_mm, addr, pte, newpte);
265 pages++;
266 }
267 } else {
268 /* It must be an none page, or what else?.. */
269 WARN_ON_ONCE(!pte_none(oldpte));
270
271 /*
272 * Nobody plays with any none ptes besides
273 * userfaultfd when applying the protections.
274 */
275 if (likely(!uffd_wp))
276 continue;
277
278 if (userfaultfd_wp_use_markers(vma)) {
279 /*
280 * For file-backed mem, we need to be able to
281 * wr-protect a none pte, because even if the
282 * pte is none, the page/swap cache could
283 * exist. Doing that by install a marker.
284 */
285 set_pte_at(vma->vm_mm, addr, pte,
286 make_pte_marker(PTE_MARKER_UFFD_WP));
287 pages++;
288 }
289 }
290 } while (pte++, addr += PAGE_SIZE, addr != end);
291 arch_leave_lazy_mmu_mode();
292 pte_unmap_unlock(pte - 1, ptl);
293
294 return pages;
295 }
296

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#6 lore raw

Re: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate

In reply to Yan Zhao
kernel test robot noticed the following build errors:

Hi Yan,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on kvm/queue]
[also build test ERROR on mst-vhost/linux-next linus/master] [cannot apply to akpm-mm/mm-everything kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Yan-Zhao/mm-mmu_notifier-introduce-a-new-mmu-notifier-flag-MMU_NOTIFIER_RANGE_NUMA/20230810-172955
base:   https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue
patch link:    https://lore.kernel.org/r/20230810090048.26184-1-yan.y.zhao%40intel.com
patch subject: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate
config: x86_64-buildonly-randconfig-r003-20230809 (https://download.01.org/0day-ci/archive/20230810/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce: (https://download.01.org/0day-ci/archive/20230810/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

mm/huge_memory.c: In function 'change_huge_pmd':

>> mm/huge_memory.c:1895:17: error: implicit declaration of function 'mmu_notifier_numa_protect' [-Werror=implicit-function-declaration]
    1895 |                 mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PMD_SIZE);
         |                 ^~~~~~~~~~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +/mmu_notifier_numa_protect +1895 mm/huge_memory.c

1862
1863 if (prot_numa) {
1864 struct page *page;
1865 bool toptier;
1866 /*
1867 * Avoid trapping faults against the zero page. The read-only
1868 * data is likely to be read-cached on the local CPU and
1869 * local/remote hits to the zero page are not interesting.
1870 */
1871 if (is_huge_zero_pmd(*pmd))
1872 goto unlock;
1873
1874 if (pmd_protnone(*pmd))
1875 goto unlock;
1876
1877 page = pmd_page(*pmd);
1878
1879 if (PageAnon(page) && PageAnonExclusive(page) &&
1880 page_maybe_dma_pinned(page))
1881 goto unlock;
1882
1883 toptier = node_is_toptier(page_to_nid(page));
1884 /*
1885 * Skip scanning top tier node if normal numa
1886 * balancing is disabled
1887 */
1888 if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
1889 toptier)
1890 goto unlock;
1891
1892 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
1893 !toptier)
1894 xchg_page_access_time(page, jiffies_to_msecs(jiffies));

> 1895			mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PMD_SIZE);

1896 }
1897 /*
1898 * In case prot_numa, we are under mmap_read_lock(mm). It's critical
1899 * to not clear pmd intermittently to avoid race with MADV_DONTNEED
1900 * which is also under mmap_read_lock(mm):
1901 *
1902 * CPU0: CPU1:
1903 * change_huge_pmd(prot_numa=1)
1904 * pmdp_huge_get_and_clear_notify()
1905 * madvise_dontneed()
1906 * zap_pmd_range()
1907 * pmd_trans_huge(*pmd) == 0 (without ptl)
1908 * // skip the pmd
1909 * set_pmd_at();
1910 * // pmd is re-established
1911 *
1912 * The race makes MADV_DONTNEED miss the huge pmd and don't clear it
1913 * which may break userspace.
1914 *
1915 * pmdp_invalidate_ad() is required to make sure we don't miss
1916 * dirty/young flags set by hardware.
1917 */
1918 oldpmd = pmdp_invalidate_ad(vma, addr, pmd);
1919
1920 entry = pmd_modify(oldpmd, newprot);
1921 if (uffd_wp)
1922 entry = pmd_mkuffd_wp(entry);
1923 else if (uffd_wp_resolve)
1924 /*
1925 * Leave the write bit to be handled by PF interrupt
1926 * handler, then things like COW could be properly
1927 * handled.
1928 */
1929 entry = pmd_clear_uffd_wp(entry);
1930
1931 /* See change_pte_range(). */
1932 if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pmd_write(entry) &&
1933 can_change_pmd_writable(vma, addr, entry))
1934 entry = pmd_mkwrite(entry);
1935
1936 ret = HPAGE_PMD_NR;
1937 set_pmd_at(mm, addr, pmd, entry);
1938
1939 if (huge_pmd_needs_flush(oldpmd, entry))
1940 tlb_flush_pmd_range(tlb, addr, HPAGE_PMD_SIZE);
1941 unlock:
1942 spin_unlock(ptl);
1943 return ret;
1944 }
1945

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#7 lore raw

Re: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate

In reply to Yan Zhao
Quoted 5 lines · Yan Zhao

Call mmu notifier's callback .numa_protect() in change_pmd_range() when a page is ensured to be protected by PROT_NONE for NUMA migration purpose.

Consider squashing with the previous patch. It’s better to see the user (caller) with the new functionality.

It would be useful to describe what the expected course of action that numa_protect callback should take.

author
#8 lore raw

Re: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate

Quoted 11 lines · Nadav Amit
Quoted 5 lines · Yan Zhao

Call mmu notifier's callback .numa_protect() in change_pmd_range() when a page is ensured to be protected by PROT_NONE for NUMA migration purpose.

Consider squashing with the previous patch. It’s better to see the user (caller) with the new functionality.

It would be useful to describe what the expected course of action that numa_protect callback should take.

Thanks! I'll do in this way when I prepare patches in future :)

author
#9 lore raw

RFC PATCH v2 5/5 KVM: Unmap pages only when it's indeed protected for NUMA migration

In reply to Yan Zhao

Register to .numa_protect() callback in mmu notifier so that KVM can get acurate information about when a page is PROT_NONE protected in primary MMU and unmap it in secondary MMU accordingly.

In KVM's .invalidate_range_start() handler, if the event is to notify that the range may be protected to PROT_NONE for NUMA migration purpose, don't do the unmapping in secondary MMU. Hold on until.numa_protect() comes.

---
 virt/kvm/kvm_main.c | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)
virt/kvm/kvm_main.c +22 -3
index dfbaafbe3a00..907444a1761b 100644
@@ -711,6 +711,20 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
 	kvm_handle_hva_range(mn, address, address + 1, pte, kvm_change_spte_gfn);
 }
 
+static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn,
+					  struct mm_struct *mm,
+					  unsigned long start,
+					  unsigned long end)
+{
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
+
+	WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count));
+	if (!READ_ONCE(kvm->mmu_invalidate_in_progress))
+		return;
+
+	kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range);
+}
+
 void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
 			      unsigned long end)
 {
@@ -744,14 +758,18 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 					const struct mmu_notifier_range *range)
 {
 	struct kvm *kvm = mmu_notifier_to_kvm(mn);
+	bool is_numa = (range->event == MMU_NOTIFY_PROTECTION_VMA) &&
+		       (range->flags & MMU_NOTIFIER_RANGE_NUMA);
 	const struct kvm_hva_range hva_range = {
 		.start		= range->start,
 		.end		= range->end,
 		.pte		= __pte(0),
-		.handler	= kvm_unmap_gfn_range,
+		.handler	= !is_numa ? kvm_unmap_gfn_range :
+				  (void *)kvm_null_fn,
 		.on_lock	= kvm_mmu_invalidate_begin,
-		.on_unlock	= kvm_arch_guest_memory_reclaimed,
-		.flush_on_ret	= true,
+		.on_unlock	= !is_numa ? kvm_arch_guest_memory_reclaimed :
+				  (void *)kvm_null_fn,
+		.flush_on_ret	= !is_numa ? true : false,
 		.may_block	= mmu_notifier_range_blockable(range),
 	};
 
@@ -899,6 +917,7 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = {
 	.clear_young		= kvm_mmu_notifier_clear_young,
 	.test_young		= kvm_mmu_notifier_test_young,
 	.change_pte		= kvm_mmu_notifier_change_pte,
+	.numa_protect		= kvm_mmu_notifier_numa_protect,
 	.release		= kvm_mmu_notifier_release,
 };
 
Signature · 1 line
2.17.1
#10 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 25 lines · Yan Zhao

Register to .numa_protect() callback in mmu notifier so that KVM can get acurate information about when a page is PROT_NONE protected in primary MMU and unmap it in secondary MMU accordingly.

In KVM's .invalidate_range_start() handler, if the event is to notify that the range may be protected to PROT_NONE for NUMA migration purpose, don't do the unmapping in secondary MMU. Hold on until.numa_protect() comes.

---
 virt/kvm/kvm_main.c | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)
virt/kvm/kvm_main.c +3 -0
index dfbaafbe3a00..907444a1761b 100644
@@ -711,6 +711,20 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
 	kvm_handle_hva_range(mn, address, address + 1, pte, kvm_change_spte_gfn);
 }
 
+static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn,
+					  struct mm_struct *mm,
+					  unsigned long start,

virt/kvm/kvm_main.c static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, · line 726

Quoted 10 lines
virt/kvm/kvm_main.c +10 -0
+					  unsigned long end)
+{
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
+
+	WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count));
+	if (!READ_ONCE(kvm->mmu_invalidate_in_progress))
+		return;
+
+	kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range);
+}

numa balance will scan wide memory range, and there will be one time ipi notification with kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.

however numa balance notification, pmd table of vm maybe needs not be freed in kvm_unmap_gfn_range.

Regards
Bibo Mao

Quoted 34 lines
virt/kvm/kvm_main.c +9 -3
+
 void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start,
 			      unsigned long end)
 {
@@ -744,14 +758,18 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 					const struct mmu_notifier_range *range)
 {
 	struct kvm *kvm = mmu_notifier_to_kvm(mn);
+	bool is_numa = (range->event == MMU_NOTIFY_PROTECTION_VMA) &&
+		       (range->flags & MMU_NOTIFIER_RANGE_NUMA);
 	const struct kvm_hva_range hva_range = {
 		.start		= range->start,
 		.end		= range->end,
 		.pte		= __pte(0),
-		.handler	= kvm_unmap_gfn_range,
+		.handler	= !is_numa ? kvm_unmap_gfn_range :
+				  (void *)kvm_null_fn,
 		.on_lock	= kvm_mmu_invalidate_begin,
-		.on_unlock	= kvm_arch_guest_memory_reclaimed,
-		.flush_on_ret	= true,
+		.on_unlock	= !is_numa ? kvm_arch_guest_memory_reclaimed :
+				  (void *)kvm_null_fn,
+		.flush_on_ret	= !is_numa ? true : false,
 		.may_block	= mmu_notifier_range_blockable(range),
 	};
 
@@ -899,6 +917,7 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = {
 	.clear_young		= kvm_mmu_notifier_clear_young,
 	.test_young		= kvm_mmu_notifier_test_young,
 	.change_pte		= kvm_mmu_notifier_change_pte,
+	.numa_protect		= kvm_mmu_notifier_numa_protect,
 	.release		= kvm_mmu_notifier_release,
 };
 
author
#11 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 6 lines
Quoted 6 lines
virt/kvm/kvm_main.c +6 -0
+static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn,
+					  struct mm_struct *mm,
+					  unsigned long start,
+					  unsigned long end)
+{
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
Quoted 8 lines

virt/kvm/kvm_main.c static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, · line 726

Quoted 7 lines
virt/kvm/kvm_main.c +7 -0
+
+	WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count));
+	if (!READ_ONCE(kvm->mmu_invalidate_in_progress))
+		return;
+
+	kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range);
+}

numa balance will scan wide memory range, and there will be one time

Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.

Quoted 2 lines

ipi notification with kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.

Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.

For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.

The reason is below:

Before this series, kvm_unmap_gfn_range() is called for once for a 2M range.
After this series, kvm_unmap_gfn_range() is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.

Though kvm_unmap_gfn_range() is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (in tdp_mmu_iter_cond_resched()) if the pages are mapped in 4K in secondary MMU.

With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.

Below is my testing data on a VM without assigned devices:
The data is an average of 10 times guest boot-up.

    data           | numa balancing caused  | numa balancing caused    
  on average       | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         35             |     8625                 
after  this series |      10037             |     4610                 

For a single guest bootup,

                   | numa balancing caused  | numa balancing caused    
    best  data     | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         28             |       13                  
after  this series |        406             |      195                  
                   | numa balancing caused  | numa balancing caused    
   worst  data     | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         44             |    43920               
after  this series |      17352             |     8668                 
Quoted 4 lines

however numa balance notification, pmd table of vm maybe needs not be freed in kvm_unmap_gfn_range.

#12 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 14 lines · Yan Zhao
Quoted 6 lines
Quoted 6 lines
virt/kvm/kvm_main.c +6 -0
+static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn,
+					  struct mm_struct *mm,
+					  unsigned long start,
+					  unsigned long end)
+{
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
Quoted 8 lines

virt/kvm/kvm_main.c static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, · line 726

Quoted 7 lines
virt/kvm/kvm_main.c +7 -0
+
+	WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count));
+	if (!READ_ONCE(kvm->mmu_invalidate_in_progress))
+		return;
+
+	kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range);
+}

numa balance will scan wide memory range, and there will be one time

Quoted 2 lines

Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.

yes, range is huge page size when changing numa protection during numa scanning.

Quoted 20 lines
Quoted 3 lines

ipi notification with kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.

Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.

For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.

The reason is below:

Before this series, kvm_unmap_gfn_range() is called for once for a 2M range.
After this series, kvm_unmap_gfn_range() is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.

Quoted 3 lines

Though kvm_unmap_gfn_range() is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (in tdp_mmu_iter_cond_resched())

I do not know much about x86, does this happen always or only need reschedule from code? so that there will be many times of tlb IPIs in only once function call about kvm_unmap_gfn_range.

Quoted 6 lines

if the pages are mapped in 4K in secondary MMU.

With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.

No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.

Quoted 9 lines

Below is my testing data on a VM without assigned devices:
The data is an average of 10 times guest boot-up.

    data           | numa balancing caused  | numa balancing caused    
  on average       | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         35             |     8625                 
after  this series |      10037             |     4610   

just be cautious, before the series there are 8625/35 = 246 IPI tlb flush ops during one time kvm_unmap_gfn_range, is that x86 specific or generic?

By the way are primary mmu and secondary mmu both 4K small page size "on average"?

Regards
Bibo Mao

Quoted 21 lines

For a single guest bootup,

                   | numa balancing caused  | numa balancing caused    
    best  data     | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         28             |       13                  
after  this series |        406             |      195                  
                   | numa balancing caused  | numa balancing caused    
   worst  data     | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         44             |    43920               
after  this series |      17352             |     8668                 
Quoted 4 lines

however numa balance notification, pmd table of vm maybe needs not be freed in kvm_unmap_gfn_range.

author
#13 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 41 lines · bibo mao
Quoted 14 lines · Yan Zhao
Quoted 6 lines
Quoted 6 lines
virt/kvm/kvm_main.c +6 -0
+static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn,
+					  struct mm_struct *mm,
+					  unsigned long start,
+					  unsigned long end)
+{
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
Quoted 8 lines

virt/kvm/kvm_main.c static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, · line 726

Quoted 7 lines
virt/kvm/kvm_main.c +7 -0
+
+	WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count));
+	if (!READ_ONCE(kvm->mmu_invalidate_in_progress))
+		return;
+
+	kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range);
+}

numa balance will scan wide memory range, and there will be one time

Quoted 2 lines

Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.

yes, range is huge page size when changing numa protection during numa scanning.

Quoted 20 lines
Quoted 3 lines

ipi notification with kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.

Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.

For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.

The reason is below:

Before this series, kvm_unmap_gfn_range() is called for once for a 2M range.
After this series, kvm_unmap_gfn_range() is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.

Quoted 4 lines
Quoted 3 lines

Though kvm_unmap_gfn_range() is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (in tdp_mmu_iter_cond_resched())

I do not know much about x86, does this happen always or only need reschedule

Ah, sorry, I missed platforms other than x86. Maybe there will be a big difference in other platforms.

Quoted 1 line

from code? so that there will be many times of tlb IPIs in only once function

Only when MMU lock is contended. But it's not seldom because of the contention in TDP page fault.

Quoted 10 lines

call about kvm_unmap_gfn_range.

Quoted 6 lines

if the pages are mapped in 4K in secondary MMU.

With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.

No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.

The .numa_protect() is called in patch 4 only when PROT_NONE is set to the page.

Quoted 5 lines
Quoted 5 lines

Below is my testing data on a VM without assigned devices:
The data is an average of 10 times guest boot-up.

Quoted 7 lines
Quoted 5 lines
    data           | numa balancing caused  | numa balancing caused    
  on average       | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() 
-------------------|------------------------|--------------------------
before this series |         35             |     8625                 
after  this series |      10037             |     4610   

just be cautious, before the series there are 8625/35 = 246 IPI tlb flush ops during one time kvm_unmap_gfn_range, is that x86 specific or generic?

Only on x86. Didn't test on other platforms.

Quoted 2 lines

By the way are primary mmu and secondary mmu both 4K small page size "on average"?

No. 4K and 2M combined in both.

#14 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 28 lines · Yan Zhao
Quoted 27 lines · bibo mao
Quoted 14 lines · Yan Zhao
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6 -0
+static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn,
+					  struct mm_struct *mm,
+					  unsigned long start,
+					  unsigned long end)
+{
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
Quoted 8 lines

Review

Quoted 7 lines
(quoted diff) +7 -0
+
+	WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count));
+	if (!READ_ONCE(kvm->mmu_invalidate_in_progress))
+		return;
+
+	kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range);
+}

numa balance will scan wide memory range, and there will be one time

Quoted 2 lines

Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.

yes, range is huge page size when changing numa protection during numa scanning.

Quoted 7 lines
Quoted 3 lines

ipi notification with kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.

Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.

The reason is below:

Before this series, kvm_unmap_gfn_range() is called for once for a 2M range.

No, it's potentially called once per 1GiB range. change_pmd_range() invokes the mmu_notifier with addr+end, where "end" is the end of the range covered by the PUD, not the the end of the current PMD. So the worst case scenario would be a 256k increase. Of course, if you have to migrate an entire 1GiB chunk of memory then you likely have bigger problems, but still.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

After this series, kvm_unmap_gfn_range() is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.

Though kvm_unmap_gfn_range() is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (in tdp_mmu_iter_cond_resched())

Quoted 7 lines
Quoted 1 line

I do not know much about x86, does this happen always or only need reschedule

Ah, sorry, I missed platforms other than x86. Maybe there will be a big difference in other platforms.

Quoted 1 line

from code? so that there will be many times of tlb IPIs in only once function

Only when MMU lock is contended. But it's not seldom because of the contention in TDP page fault.

No? I don't see how mmu_lock contention would affect the number of calls to kvm_flush_remote_tlbs(). If vCPUs are running and not generating faults, i.e. not trying to access the range in question, then ever zap will generate a remote TLB flush and thus send IPIs to all running vCPUs.

Quoted 4 lines
Quoted 4 lines

call about kvm_unmap_gfn_range.

Quoted 2 lines

if the pages are mapped in 4K in secondary MMU.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.

No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.

The .numa_protect() is called in patch 4 only when PROT_NONE is set to the page.

I'm strongly opposed to adding MMU_NOTIFIER_RANGE_NUMA. It's too much of a one-off, and losing the batching of invalidations makes me nervous. As Bibo points out, the behavior will vary based on the workload, VM configuration, etc.

There's also a *very* subtle change, in that the notification will be sent while holding the PMD/PTE lock. Taking KVM's mmu_lock under that is *probably* ok, but I'm not exactly 100% confident on that. And the only reason there isn't a more obvious bug is because kvm_handle_hva_range() sets may_block to false, e.g. KVM won't yield if there's mmu_lock contention.

However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.

Rather than muck with the notification types and add a one-off for NUMA, just defer the notification until a present PMD/PTE is actually going to be modified. It's not the prettiest, but other than the locking, I don't think has any chance of regressing other workloads/configurations.

Note, I'm assuming secondary MMUs aren't allowed to map swap entries...

Compile tested only.

From: Sean Christopherson <[email protected]>
Date: Fri, 11 Aug 2023 10:03:36 -0700
Subject: [PATCH] tmp

---
 include/linux/huge_mm.h |  4 +++-
 include/linux/mm.h      |  3 +++
 mm/huge_memory.c        |  5 ++++-
 mm/mprotect.c           | 47 +++++++++++++++++++++++++++++------------
 4 files changed, 44 insertions(+), 15 deletions(-)
include/linux/huge_mm.h +3 -1
index 20284387b841..b88316adaaad 100644
@@ -7,6 +7,8 @@
 
 #include <linux/fs.h> /* only for vma_is_dax() */
 
+struct mmu_notifier_range;
+
 vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf);
 int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		  pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
@@ -38,7 +40,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 		   unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd);
 int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		    pmd_t *pmd, unsigned long addr, pgprot_t newprot,
-		    unsigned long cp_flags);
+		    unsigned long cp_flags, struct mmu_notifier_range *range);
 
 vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write);
 vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write);
include/linux/mm.h +3 -0
index 2dd73e4f3d8e..284f61cf9c37 100644
@@ -2478,6 +2478,9 @@ static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma
 	return !!(vma->vm_flags & VM_WRITE);
 
 }
+
+void change_pmd_range_notify_secondary_mmus(unsigned long addr,
+					    struct mmu_notifier_range *range);
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
 extern long change_protection(struct mmu_gather *tlb,
mm/huge_memory.c +4 -1
index a71cf686e3b2..47c7712b163e 100644
@@ -1808,7 +1808,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
  */
 int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		    pmd_t *pmd, unsigned long addr, pgprot_t newprot,
-		    unsigned long cp_flags)
+		    unsigned long cp_flags, struct mmu_notifier_range *range)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	spinlock_t *ptl;
@@ -1893,6 +1893,9 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		    !toptier)
 			xchg_page_access_time(page, jiffies_to_msecs(jiffies));
 	}
+
+	change_pmd_range_notify_secondary_mmus(addr, range);
+
 	/*
 	 * In case prot_numa, we are under mmap_read_lock(mm). It's critical
 	 * to not clear pmd intermittently to avoid race with MADV_DONTNEED
mm/mprotect.c +34 -13

95 lines not on this page · raw message

base-commit: 1f40f634009556c119974cafce4c7b2f9b8c58ad

#15 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 3 lines

However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.

No, this is not legal.

mm/mprotect.c pgtable_populate_needed(struct vm_area_struct *vma, unsigned long cp_flags) · line 372

Quoted 10 lines
mm/mprotect.c +10 -0
+void change_pmd_range_notify_secondary_mmus(unsigned long addr,
+					    struct mmu_notifier_range *range)
+{
+	if (range->start)
+		return;
+
+	VM_WARN_ON(addr >= range->end);
+	range->start = addr;
+	mmu_notifier_invalidate_range_start_nonblock(range);
+}

The 'nonblock' entry point is advisory, if it fails the caller must not change the mm.

Jason

author
#16 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

In reply to Sean Christopherson
Quoted 35 lines · Sean Christopherson
Quoted 34 lines · Yan Zhao
Quoted 33 lines · bibo mao
Quoted 14 lines · Yan Zhao
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6 -0
+static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn,
+					  struct mm_struct *mm,
+					  unsigned long start,
+					  unsigned long end)
+{
+	struct kvm *kvm = mmu_notifier_to_kvm(mn);
Quoted 8 lines

Review

Quoted 7 lines
(quoted diff) +7 -0
+
+	WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count));
+	if (!READ_ONCE(kvm->mmu_invalidate_in_progress))
+		return;
+
+	kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range);
+}

numa balance will scan wide memory range, and there will be one time

Quoted 2 lines

Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.

yes, range is huge page size when changing numa protection during numa scanning.

Quoted 13 lines
Quoted 3 lines

ipi notification with kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.

Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.

For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.

The reason is below:

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Before this series, kvm_unmap_gfn_range() is called for once for a 2M range.

No, it's potentially called once per 1GiB range. change_pmd_range() invokes the mmu_notifier with addr+end, where "end" is the end of the range covered by the PUD, not the the end of the current PMD. So the worst case scenario would be a 256k increase. Of course, if you have to migrate an entire 1GiB chunk of memory then you likely have bigger problems, but still.

Yes, thanks for pointing it out.
I realized it after re-reading the code and re-checking the log message. This wider range also explained the collected worst data:
44 kvm_unmap_gfn_range() caused 43920 kvm_flush_remote_tlbs()requests. i.e. 998 remote tlb flush requests per kvm_unmap_gfn_range().

Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
Quoted 9 lines

After this series, kvm_unmap_gfn_range() is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.

Though kvm_unmap_gfn_range() is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (in tdp_mmu_iter_cond_resched())

I do not know much about x86, does this happen always or only need reschedule

Ah, sorry, I missed platforms other than x86. Maybe there will be a big difference in other platforms.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

from code? so that there will be many times of tlb IPIs in only once function

Only when MMU lock is contended. But it's not seldom because of the contention in TDP page fault.

No? I don't see how mmu_lock contention would affect the number of calls to kvm_flush_remote_tlbs(). If vCPUs are running and not generating faults, i.e. not trying to access the range in question, then ever zap will generate a remote TLB flush and thus send IPIs to all running vCPUs.

In tdp_mmu_zap_leafs() for kvm_unmap_gfn_range(), the flow is like this:

1. Check necessity of mmu_lock reschedule
1.a -- if yes,
1.a.1 do kvm_flush_remote_tlbs() if flush is true. 1.a.2 reschedule of mmu_lock
1.a.3 goto 2.
1.b -- if no, goto 2
2. Zap present leaf entry, and set flush to be true
3. Get next gfn and go to to 1

With a wide range to .invalidate_range_start()/end(), it's easy to find rwlock_needbreak(&kvm->mmu_lock) to be true (goes to 1.a and 1.a.1) And in tdp_mmu_zap_leafs(), before rescheduling of mmu_lock, tlb flush request is performed. (1.a.1)

Take a real case for example,
NUMA balancing requests KVM to zap GFN range from 0xb4000 to 0xb9800. Then when KVM zaps GFN 0xb807b, it will finds mmu_lock needs break because vCPU is faulting GFN 0xb804c.
And vCPUs fill constantly retry faulting 0xb804c for 3298 times until .invalidate_range_end().
Then for this zap of GFN range from 0xb4000 - 0xb9800,
vCPUs retry fault of
GFN 0xb804c for 3298 times,
GFN 0xb8074 for 3161 times,
GFN 0xb81ce for 15190 times,
and the accesses of the 3 GFNs cause 3209 times of kvm_flush_remote_tlbs() for one kvm_unmap_gfn_range() because flush requests are not batched. (this range is mapped in both 2M and 4K in secondary MMU).

Without the contending of mmu_lock, tdp_mmu_zap_leafs() just combines all flush requests and leaves 1 kvm_flush_remote_tlbs() to be called after it returns.

In my test scenario, which is VM boot-up, this kind of contention is frequent.
Here's the 10 times data for a VM boot-up collected previously.

       |      count of       |       count of          |
       | kvm_unmap_gfn_range | kvm_flush_remote_tlbs() |
-------|---------------------|-------------------------|
   1   |         38          |           14            |
   2   |         28          |         5191            |
   3   |         36          |        13398            |
   4   |         44          |        43920            |
   5   |         28          |           14            |
   6   |         36          |        10803            |
   7   |         38          |          892            |
   8   |         32          |         5905            |
   9   |         32          |           13            |
  10   |         38          |         6096            |
-------|---------------------|-------------------------|
average|         35          |         8625            |

I wonder if we could loose the frequency to check for rescheduling in tdp_mmu_iter_cond_resched() if the zap range is wide, e.g.

if (iter->next_last_level_gfn ==

    iter->yielded_gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M))
	return false; 
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines

call about kvm_unmap_gfn_range.

Quoted 2 lines

if the pages are mapped in 4K in secondary MMU.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.

No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.

The .numa_protect() is called in patch 4 only when PROT_NONE is set to the page.

Quoted 7 lines

I'm strongly opposed to adding MMU_NOTIFIER_RANGE_NUMA. It's too much of a one-off, and losing the batching of invalidations makes me nervous. As Bibo points out, the behavior will vary based on the workload, VM configuration, etc.

There's also a *very* subtle change, in that the notification will be sent while holding the PMD/PTE lock. Taking KVM's mmu_lock under that is *probably* ok, but I'm not exactly 100% confident on that. And the only reason there isn't a more

MMU lock is a rwlock, which is a variant of spinlock.
So, I think taking it within PMD/PTE lock is ok.
Actually we have already done that during the .change_pte() notification, where

kvm_mmu_notifier_change_pte() takes KVM mmu_lock for write,
while PTE lock is held while sending set_pte_at_notify() --> .change_pte() handlers

Quoted 2 lines

obvious bug is because kvm_handle_hva_range() sets may_block to false, e.g. KVM won't yield if there's mmu_lock contention.

Yes, KVM won't yield and reschedule of KVM mmu_lock, so it's fine.

Quoted 4 lines

However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.

Quoted 8 lines

Rather than muck with the notification types and add a one-off for NUMA, just defer the notification until a present PMD/PTE is actually going to be modified. It's not the prettiest, but other than the locking, I don't think has any chance of regressing other workloads/configurations.

Note, I'm assuming secondary MMUs aren't allowed to map swap entries...

Compile tested only.

I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().

Quoted 102 lines

From: Sean Christopherson <[email protected]>
Date: Fri, 11 Aug 2023 10:03:36 -0700
Subject: [PATCH] tmp

---
 include/linux/huge_mm.h |  4 +++-
 include/linux/mm.h      |  3 +++
 mm/huge_memory.c        |  5 ++++-
 mm/mprotect.c           | 47 +++++++++++++++++++++++++++++------------
 4 files changed, 44 insertions(+), 15 deletions(-)
include/linux/huge_mm.h +3 -1
index 20284387b841..b88316adaaad 100644
@@ -7,6 +7,8 @@
 
 #include <linux/fs.h> /* only for vma_is_dax() */
 
+struct mmu_notifier_range;
+
 vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf);
 int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		  pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
@@ -38,7 +40,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 		   unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd);
 int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		    pmd_t *pmd, unsigned long addr, pgprot_t newprot,
-		    unsigned long cp_flags);
+		    unsigned long cp_flags, struct mmu_notifier_range *range);
 
 vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write);
 vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write);
include/linux/mm.h +3 -0
index 2dd73e4f3d8e..284f61cf9c37 100644
@@ -2478,6 +2478,9 @@ static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma
 	return !!(vma->vm_flags & VM_WRITE);
 
 }
+
+void change_pmd_range_notify_secondary_mmus(unsigned long addr,
+					    struct mmu_notifier_range *range);
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
 extern long change_protection(struct mmu_gather *tlb,
mm/huge_memory.c +4 -1
index a71cf686e3b2..47c7712b163e 100644
@@ -1808,7 +1808,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
  */
 int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		    pmd_t *pmd, unsigned long addr, pgprot_t newprot,
-		    unsigned long cp_flags)
+		    unsigned long cp_flags, struct mmu_notifier_range *range)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	spinlock_t *ptl;
@@ -1893,6 +1893,9 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		    !toptier)
 			xchg_page_access_time(page, jiffies_to_msecs(jiffies));
 	}
+
+	change_pmd_range_notify_secondary_mmus(addr, range);
+
 	/*
 	 * In case prot_numa, we are under mmap_read_lock(mm). It's critical
 	 * to not clear pmd intermittently to avoid race with MADV_DONTNEED
mm/mprotect.c +6 -1
index d1a809167f49..f5844adbe0cb 100644
@@ -82,7 +82,8 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 
 static long change_pte_range(struct mmu_gather *tlb,
 		struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr,
-		unsigned long end, pgprot_t newprot, unsigned long cp_flags)
+		unsigned long end, pgprot_t newprot, unsigned long cp_flags,
+		struct mmu_notifier_range *range)
 {
 	pte_t *pte, oldpte;
 	spinlock_t *ptl;
@@ -164,8 +165,12 @@ static long change_pte_range(struct mmu_gather *tlb,
 				    !toptier)
 					xchg_page_access_time(page,
 						jiffies_to_msecs(jiffies));
+
+
 			}
 
+			change_pmd_range_notify_secondary_mmus(addr, range);
+
 			oldpte = ptep_modify_prot_start(vma, addr, pte);
 			ptent = pte_modify(oldpte, newprot);
 
@@ -355,6 +360,17 @@ pgtable_populate_needed(struct vm_area_struct *vma, unsigned long cp_flags)
 		err;							\
 	})

mm/mprotect.c pgtable_populate_needed(struct vm_area_struct *vma, unsigned long cp_flags) · line 371

Quoted 10 lines
mm/mprotect.c +9 -0
 
+void change_pmd_range_notify_secondary_mmus(unsigned long addr,
+					    struct mmu_notifier_range *range)
+{
+	if (range->start)
+		return;
+
+	VM_WARN_ON(addr >= range->end);
+	range->start = addr;
+	mmu_notifier_invalidate_range_start_nonblock(range);

This will cause range from addr to end to be invalidated, which may include pinned ranges.

Quoted 72 lines
mm/mprotect.c +19 -12
+}
+
 static inline long change_pmd_range(struct mmu_gather *tlb,
 		struct vm_area_struct *vma, pud_t *pud, unsigned long addr,
 		unsigned long end, pgprot_t newprot, unsigned long cp_flags)
@@ -365,7 +381,14 @@ static inline long change_pmd_range(struct mmu_gather *tlb,
 	unsigned long nr_huge_updates = 0;
 	struct mmu_notifier_range range;
 
-	range.start = 0;
+	/*
+	 * Defer invalidation of secondary MMUs until a PMD/PTE change is
+	 * imminent, e.g. NUMA balancing in particular can "fail" for certain
+	 * types of mappings.  Initialize range.start to '0' and use it to
+	 * track whether or not the invalidation notification has been set.
+	 */
+	mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0,
+				vma->vm_mm, 0, end);
 
 	pmd = pmd_offset(pud, addr);
 	do {
@@ -383,18 +406,16 @@ static inline long change_pmd_range(struct mmu_gather *tlb,
 		if (pmd_none(*pmd))
 			goto next;
 
-		/* invoke the mmu notifier if the pmd is populated */
-		if (!range.start) {
-			mmu_notifier_range_init(&range,
-				MMU_NOTIFY_PROTECTION_VMA, 0,
-				vma->vm_mm, addr, end);
-			mmu_notifier_invalidate_range_start(&range);
-		}
-
 		_pmd = pmdp_get_lockless(pmd);
 		if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) {
 			if ((next - addr != HPAGE_PMD_SIZE) ||
 			    pgtable_split_needed(vma, cp_flags)) {
+				/*
+				 * FIXME: __split_huge_pmd() performs its own
+				 * mmu_notifier invalidation prior to clearing
+				 * the PMD, ideally all invalidations for the
+				 * range would be batched.
+				 */
 				__split_huge_pmd(vma, pmd, addr, false, NULL);
 				/*
 				 * For file-backed, the pmd could have been
@@ -407,8 +428,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb,
 					break;
 				}
 			} else {
-				ret = change_huge_pmd(tlb, vma, pmd,
-						addr, newprot, cp_flags);
+				ret = change_huge_pmd(tlb, vma, pmd, addr,
+						      newprot, cp_flags, &range);
 				if (ret) {
 					if (ret == HPAGE_PMD_NR) {
 						pages += HPAGE_PMD_NR;
@@ -423,7 +444,7 @@ static inline long change_pmd_range(struct mmu_gather *tlb,
 		}
 
 		ret = change_pte_range(tlb, vma, pmd, addr, next, newprot,
-				       cp_flags);
+				       cp_flags, &range);
 		if (ret < 0)
 			goto again;
 		pages += ret;

base-commit: 1f40f634009556c119974cafce4c7b2f9b8c58ad

author
#17 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 6 lines · Yan Zhao

I wonder if we could loose the frequency to check for rescheduling in tdp_mmu_iter_cond_resched() if the zap range is wide, e.g.

if (iter->next_last_level_gfn ==

    iter->yielded_gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M))
	return false;

Correct:

@@ -712,7 +713,8 @@ static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm,

        WARN_ON(iter->yielded);

        /* Ensure forward progress has been made before yielding. */
-       if (iter->next_last_level_gfn == iter->yielded_gfn)
+       if (iter->next_last_level_gfn >= iter->yielded_gfn &&
+          iter->next_last_level_gfn < iter->yielded_gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M))
                return false;

        if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {

This can reduce kvm_flush_remote_tlbs() a lot in one kvm_unmap_gfn_range() in KVM x86 TDP MMU.

#18 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

In reply to Yan Zhao
Quoted 17 lines · Yan Zhao
Quoted 16 lines · Sean Christopherson
Quoted 15 lines
Quoted 9 lines
Quoted 8 lines

After this series, kvm_unmap_gfn_range() is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.

Though kvm_unmap_gfn_range() is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (in tdp_mmu_iter_cond_resched())

I do not know much about x86, does this happen always or only need reschedule

Ah, sorry, I missed platforms other than x86. Maybe there will be a big difference in other platforms.

Quoted 1 line

from code? so that there will be many times of tlb IPIs in only once function

Only when MMU lock is contended. But it's not seldom because of the contention in TDP page fault.

Quoted 7 lines
Quoted 4 lines

No? I don't see how mmu_lock contention would affect the number of calls to kvm_flush_remote_tlbs(). If vCPUs are running and not generating faults, i.e. not trying to access the range in question, then ever zap will generate a remote TLB flush and thus send IPIs to all running vCPUs.

In tdp_mmu_zap_leafs() for kvm_unmap_gfn_range(), the flow is like this:

1. Check necessity of mmu_lock reschedule

Ah, you're running a preemptible kernel.

Quoted 29 lines

1.a -- if yes,
1.a.1 do kvm_flush_remote_tlbs() if flush is true. 1.a.2 reschedule of mmu_lock
1.a.3 goto 2.
1.b -- if no, goto 2
2. Zap present leaf entry, and set flush to be true
3. Get next gfn and go to to 1

With a wide range to .invalidate_range_start()/end(), it's easy to find rwlock_needbreak(&kvm->mmu_lock) to be true (goes to 1.a and 1.a.1) And in tdp_mmu_zap_leafs(), before rescheduling of mmu_lock, tlb flush request is performed. (1.a.1)

Take a real case for example,
NUMA balancing requests KVM to zap GFN range from 0xb4000 to 0xb9800. Then when KVM zaps GFN 0xb807b, it will finds mmu_lock needs break because vCPU is faulting GFN 0xb804c.
And vCPUs fill constantly retry faulting 0xb804c for 3298 times until .invalidate_range_end().
Then for this zap of GFN range from 0xb4000 - 0xb9800,
vCPUs retry fault of
GFN 0xb804c for 3298 times,
GFN 0xb8074 for 3161 times,
GFN 0xb81ce for 15190 times,
and the accesses of the 3 GFNs cause 3209 times of kvm_flush_remote_tlbs() for one kvm_unmap_gfn_range() because flush requests are not batched. (this range is mapped in both 2M and 4K in secondary MMU).

Quoted 7 lines

Without the contending of mmu_lock, tdp_mmu_zap_leafs() just combines all flush requests and leaves 1 kvm_flush_remote_tlbs() to be called after it returns.

In my test scenario, which is VM boot-up, this kind of contention is frequent.

Hmm, I think your test scenario is a bit of an outlier. Running a preemptible kernel on a multi-socket system (presumably your system is multi-socket if you have NUMA) is a bit odd.

Not that that invalidates the test result, but I would be quite surprised if there are production use cases for NUMA+KVM+preemptible. Though maybe I'm underestimating how much KVM is used on workstations?

Quoted 18 lines

Here's the 10 times data for a VM boot-up collected previously.

       |      count of       |       count of          |
       | kvm_unmap_gfn_range | kvm_flush_remote_tlbs() |
-------|---------------------|-------------------------|
   1   |         38          |           14            |
   2   |         28          |         5191            |
   3   |         36          |        13398            |
   4   |         44          |        43920            |
   5   |         28          |           14            |
   6   |         36          |        10803            |
   7   |         38          |          892            |
   8   |         32          |         5905            |
   9   |         32          |           13            |
  10   |         38          |         6096            |
-------|---------------------|-------------------------|
average|         35          |         8625            |
Quoted 6 lines

I wonder if we could loose the frequency to check for rescheduling in tdp_mmu_iter_cond_resched() if the zap range is wide, e.g.

if (iter->next_last_level_gfn ==

    iter->yielded_gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M))
	return false;

Hrm, no. We'd want to repeat that logic for the shadow MMU, and it could regress other scenarios, e.g. if a vCPU is trying to fault-in an address that isn't part of the invalidation, then yielding asap is desirable.

Unless I'm missing something, the easiest solution is to check for an invalidation *before* acquiring mmu_lock. The check would be prone to false negatives and false positives, so KVM would need to recheck after acquiring mmu_lock, but the check itself is super cheap relative to the overall cost of the page fault.

Compile tested patch at the bottom. I'll test and post formally (probably with similar treatment for other architectures).

Quoted 20 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines

call about kvm_unmap_gfn_range.

Quoted 2 lines

if the pages are mapped in 4K in secondary MMU.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.

No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.

The .numa_protect() is called in patch 4 only when PROT_NONE is set to the page.

Quoted 7 lines

I'm strongly opposed to adding MMU_NOTIFIER_RANGE_NUMA. It's too much of a one-off, and losing the batching of invalidations makes me nervous. As Bibo points out, the behavior will vary based on the workload, VM configuration, etc.

There's also a *very* subtle change, in that the notification will be sent while holding the PMD/PTE lock. Taking KVM's mmu_lock under that is *probably* ok, but I'm not exactly 100% confident on that. And the only reason there isn't a more

Quoted 6 lines

MMU lock is a rwlock, which is a variant of spinlock.
So, I think taking it within PMD/PTE lock is ok.
Actually we have already done that during the .change_pte() notification, where

kvm_mmu_notifier_change_pte() takes KVM mmu_lock for write,
while PTE lock is held while sending set_pte_at_notify() --> .change_pte() handlers

.change_pte() gets away with running under PMD/PTE lock because (a) it's not allowed to fail and (b) KVM is the only secondary MMU that hooks .change_pte() and KVM doesn't use a sleepable lock.

As Jason pointed out, mmu_notifier_invalidate_range_start_nonblock() can fail because some secondary MMUs use mutexes or r/w semaphores to handle mmu_notifier events. For NUMA balancing, canceling the protection change might be ok, but for everything else, failure is not an option. So unfortunately, my idea won't work as-is.

We might get away with deferring just the change_prot_numa() case, but I don't think that's worth the mess/complexity.

I would much rather tell userspace to disable migrate-on-fault for KVM guest mappings (mbind() should work?) for these types of setups, or to disable NUMA balancing entirely. If the user really cares about performance of their VM(s), vCPUs should be affined to a single NUMA node (or core, or pinned 1:1), and if the VM spans multiple nodes, a virtual NUMA topology should be given to the guest. At that point, NUMA balancing is likely to do more harm than good.

Quoted 13 lines
Quoted 2 lines

obvious bug is because kvm_handle_hva_range() sets may_block to false, e.g. KVM won't yield if there's mmu_lock contention.

Yes, KVM won't yield and reschedule of KVM mmu_lock, so it's fine.

Quoted 9 lines

However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.

Rather than muck with the notification types and add a one-off for NUMA, just defer the notification until a present PMD/PTE is actually going to be modified. It's not the prettiest, but other than the locking, I don't think has any chance of regressing other workloads/configurations.

Quoted 6 lines
Quoted 3 lines

Note, I'm assuming secondary MMUs aren't allowed to map swap entries...

Compile tested only.

I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().

It pairs with existing call to mmu_notifier_invalidate_range_end() in change_pmd_range():

	if (range.start)
		mmu_notifier_invalidate_range_end(&range);

--
From: Sean Christopherson <[email protected]>
Date: Mon, 14 Aug 2023 08:59:12 -0700
Subject: [PATCH] KVM: x86/mmu: Retry fault before acquiring mmu_lock if
mapping is changing

Retry page faults without acquiring mmu_lock if the resolve hva is covered by an active invalidation. Contending for mmu_lock is especially problematic on preemptible kernels as the invalidation task will yield the lock (see rwlock_needbreak()), delay the in-progress invalidation, and ultimately increase the latency of resolving the page fault. And in the worst case scenario, yielding will be accompanied by a remote TLB flush, e.g. if the invalidation covers a large range of memory and vCPUs are accessing addresses that were already zapped.

Alternatively, the yielding issue could be mitigated by teaching KVM's MMU iterators to perform more work before yielding, but that wouldn't solve the lock contention and would negatively affect scenarios where a vCPU is trying to fault in an address that is NOT covered by the in-progress invalidation.

---
 arch/x86/kvm/mmu/mmu.c   | 3 +++
 include/linux/kvm_host.h | 8 +++++++-
 2 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +3 -0
index 9e4cd8b4a202..f29718a16211 100644
@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
 	if (unlikely(!fault->slot))
 		return kvm_handle_noslot_fault(vcpu, fault, access);
 
+	if (mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva))
+		return RET_PF_RETRY;
+
 	return RET_PF_CONTINUE;
 }
 
include/linux/kvm_host.h +7 -1
index cb86108c624d..f41d4fe61a06 100644
@@ -1960,7 +1960,6 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
 					   unsigned long mmu_seq,
 					   unsigned long hva)
 {
-	lockdep_assert_held(&kvm->mmu_lock);
 	/*
 	 * If mmu_invalidate_in_progress is non-zero, then the range maintained
 	 * by kvm_mmu_notifier_invalidate_range_start contains all addresses
@@ -1971,6 +1970,13 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
 	    hva >= kvm->mmu_invalidate_range_start &&
 	    hva < kvm->mmu_invalidate_range_end)
 		return 1;
+
+	/*
+	 * Note the lack of a memory barrier!  The caller *must* hold mmu_lock
+	 * to avoid false negatives and/or false positives (less concerning).
+	 * Holding mmu_lock is not mandatory though, e.g. to allow pre-checking
+	 * for an in-progress invalidation to avoiding contending mmu_lock.
+	 */
 	if (kvm->mmu_invalidate_seq != mmu_seq)
 		return 1;
 	return 0;

base-commit: 5bc7f472423f95a3f5c73b0b56c47e57d8833efc

author
#19 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 11 lines · Sean Christopherson
Quoted 11 lines
Quoted 7 lines

I'm strongly opposed to adding MMU_NOTIFIER_RANGE_NUMA. It's too much of a one-off, and losing the batching of invalidations makes me nervous. As Bibo points out, the behavior will vary based on the workload, VM configuration, etc.

There's also a *very* subtle change, in that the notification will be sent while holding the PMD/PTE lock. Taking KVM's mmu_lock under that is *probably* ok, but I'm not exactly 100% confident on that. And the only reason there isn't a more

MMU lock is a rwlock, which is a variant of spinlock.
So, I think taking it within PMD/PTE lock is ok.
Actually we have already done that during the .change_pte() notification, where

Quoted 6 lines
Quoted 2 lines

kvm_mmu_notifier_change_pte() takes KVM mmu_lock for write,
while PTE lock is held while sending set_pte_at_notify() --> .change_pte() handlers

.change_pte() gets away with running under PMD/PTE lock because (a) it's not allowed to fail and (b) KVM is the only secondary MMU that hooks .change_pte() and KVM doesn't use a sleepable lock.

.numa_protect() in patch 4 is also sent when it's not allowed to fail and there's no sleepable lock in KVM's handler :)

Quoted 8 lines

As Jason pointed out, mmu_notifier_invalidate_range_start_nonblock() can fail because some secondary MMUs use mutexes or r/w semaphores to handle mmu_notifier events. For NUMA balancing, canceling the protection change might be ok, but for everything else, failure is not an option. So unfortunately, my idea won't work as-is.

We might get away with deferring just the change_prot_numa() case, but I don't think that's worth the mess/complexity.

Yes, I also think deferring mmu_notifier_invalidate_range_start() is not working. One possible approach is to send successful .numa_protect() in batch. But I admit it will introduce complexity.

Quoted 25 lines

I would much rather tell userspace to disable migrate-on-fault for KVM guest mappings (mbind() should work?) for these types of setups, or to disable NUMA balancing entirely. If the user really cares about performance of their VM(s), vCPUs should be affined to a single NUMA node (or core, or pinned 1:1), and if the VM spans multiple nodes, a virtual NUMA topology should be given to the guest. At that point, NUMA balancing is likely to do more harm than good.

Quoted 17 lines
Quoted 2 lines

obvious bug is because kvm_handle_hva_range() sets may_block to false, e.g. KVM won't yield if there's mmu_lock contention.

Yes, KVM won't yield and reschedule of KVM mmu_lock, so it's fine.

Quoted 12 lines

However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.

Rather than muck with the notification types and add a one-off for NUMA, just defer the notification until a present PMD/PTE is actually going to be modified. It's not the prettiest, but other than the locking, I don't think has any chance of regressing other workloads/configurations.

Note, I'm assuming secondary MMUs aren't allowed to map swap entries...

Compile tested only.

Quoted 7 lines
Quoted 2 lines

I don't find a matching end to each mmu_notifier_invalidate_range_start_nonblock().

It pairs with existing call to mmu_notifier_invalidate_range_end() in change_pmd_range():

	if (range.start)
		mmu_notifier_invalidate_range_end(&range);

No, It doesn't work for mmu_notifier_invalidate_range_start() sent in change_pte_range(), if we only want the range to include pages successfully set to PROT_NONE.

Quoted 34 lines

--
From: Sean Christopherson <[email protected]>
Date: Mon, 14 Aug 2023 08:59:12 -0700
Subject: [PATCH] KVM: x86/mmu: Retry fault before acquiring mmu_lock if
mapping is changing

Retry page faults without acquiring mmu_lock if the resolve hva is covered by an active invalidation. Contending for mmu_lock is especially problematic on preemptible kernels as the invalidation task will yield the lock (see rwlock_needbreak()), delay the in-progress invalidation, and ultimately increase the latency of resolving the page fault. And in the worst case scenario, yielding will be accompanied by a remote TLB flush, e.g. if the invalidation covers a large range of memory and vCPUs are accessing addresses that were already zapped.

Alternatively, the yielding issue could be mitigated by teaching KVM's MMU iterators to perform more work before yielding, but that wouldn't solve the lock contention and would negatively affect scenarios where a vCPU is trying to fault in an address that is NOT covered by the in-progress invalidation.

---
 arch/x86/kvm/mmu/mmu.c   | 3 +++
 include/linux/kvm_host.h | 8 +++++++-
 2 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +0 -0
index 9e4cd8b4a202..f29718a16211 100644

arch/x86/kvm/mmu/mmu.c static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · line 4350

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
 	if (unlikely(!fault->slot))
 		return kvm_handle_noslot_fault(vcpu, fault, access);
 
+	if (mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva))
+		return RET_PF_RETRY;
+

This can effectively reduce the remote flush IPIs a lot!
One Nit is that, maybe rmb() or READ_ONCE() is required for kvm->mmu_invalidate_range_start and kvm->mmu_invalidate_range_end.
Otherwise, I'm somewhat worried about constant false positive and retry.

Quoted 33 lines
arch/x86/kvm/mmu/mmu.c +0 -0
 	return RET_PF_CONTINUE;
 }
 
include/linux/kvm_host.h +7 -1
index cb86108c624d..f41d4fe61a06 100644
@@ -1960,7 +1960,6 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
 					   unsigned long mmu_seq,
 					   unsigned long hva)
 {
-	lockdep_assert_held(&kvm->mmu_lock);
 	/*
 	 * If mmu_invalidate_in_progress is non-zero, then the range maintained
 	 * by kvm_mmu_notifier_invalidate_range_start contains all addresses
@@ -1971,6 +1970,13 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,
 	    hva >= kvm->mmu_invalidate_range_start &&
 	    hva < kvm->mmu_invalidate_range_end)
 		return 1;
+
+	/*
+	 * Note the lack of a memory barrier!  The caller *must* hold mmu_lock
+	 * to avoid false negatives and/or false positives (less concerning).
+	 * Holding mmu_lock is not mandatory though, e.g. to allow pre-checking
+	 * for an in-progress invalidation to avoiding contending mmu_lock.
+	 */
 	if (kvm->mmu_invalidate_seq != mmu_seq)
 		return 1;
 	return 0;

base-commit: 5bc7f472423f95a3f5c73b0b56c47e57d8833efc

#20 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines

Note, I'm assuming secondary MMUs aren't allowed to map swap entries...

Compile tested only.

I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().

Quoted 6 lines
Quoted 4 lines

It pairs with existing call to mmu_notifier_invalidate_range_end() in change_pmd_range():

	if (range.start)
		mmu_notifier_invalidate_range_end(&range);

No, It doesn't work for mmu_notifier_invalidate_range_start() sent in change_pte_range(), if we only want the range to include pages successfully set to PROT_NONE.

Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the start() call still used the original @end.

The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.

Quoted 8 lines
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +0 -0
index 9e4cd8b4a202..f29718a16211 100644
@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
 	if (unlikely(!fault->slot))
 		return kvm_handle_noslot_fault(vcpu, fault, access);
 
Quoted 7 lines

arch/x86/kvm/mmu/mmu.c static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · line 4350

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3 -0
+	if (mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva))
+		return RET_PF_RETRY;
+

This can effectively reduce the remote flush IPIs a lot!
One Nit is that, maybe rmb() or READ_ONCE() is required for kvm->mmu_invalidate_range_start and kvm->mmu_invalidate_range_end.
Otherwise, I'm somewhat worried about constant false positive and retry.

If anything, this needs a READ_ONCE() on mmu_invalidate_in_progress. The ranges aren't touched when when mmu_invalidate_in_progress goes to zero, so ensuring they are reloaded wouldn't do anything. The key to making forward progress is seeing that there is no in-progress invalidation.

I did consider adding said READ_ONCE(), but practically speaking, constant false positives are impossible. KVM will re-enter the guest when retrying, and there is zero chance of the compiler avoiding reloads across VM-Enter+VM-Exit.

I suppose in theory we might someday differentiate between "retry because a different vCPU may have fixed the fault" and "retry because there's an in-progress invalidation", and not bother re-entering the guest for the latter, e.g. have it try to yield instead.

All that said, READ_ONCE() on mmu_invalidate_in_progress should effectively be a nop, so it wouldn't hurt to be paranoid in this case.

Hmm, at that point, it probably makes sense to add a READ_ONCE() for mmu_invalidate_seq too, e.g. so that a sufficiently clever compiler doesn't completely optimize away the check. Losing the check wouldn't be problematic (false negatives are fine, especially on that particular check), but the generated code would *look* buggy.

#21 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 16 lines · Sean Christopherson
Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines

Note, I'm assuming secondary MMUs aren't allowed to map swap entries...

Compile tested only.

I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().

Quoted 6 lines
Quoted 4 lines

It pairs with existing call to mmu_notifier_invalidate_range_end() in change_pmd_range():

	if (range.start)
		mmu_notifier_invalidate_range_end(&range);

No, It doesn't work for mmu_notifier_invalidate_range_start() sent in change_pte_range(), if we only want the range to include pages successfully set to PROT_NONE.

Quoted 8 lines

Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the start() call still used the original @end.

The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.

For numa-balance scenery, can original page still be used by application even if pte is changed with PROT_NONE? If it can be used, maybe we can zap shadow mmu and flush tlb in notification mmu_notifier_invalidate_range_end with precised range, the range can be cross-range between range mmu_gather and mmu_notifier_range.

Regards
Bibo Mao

Quoted 38 lines
Quoted 9 lines
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +0 -0
index 9e4cd8b4a202..f29718a16211 100644
@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
 	if (unlikely(!fault->slot))
 		return kvm_handle_noslot_fault(vcpu, fault, access);
 
Quoted 7 lines

arch/x86/kvm/mmu/mmu.c static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · line 4350

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3 -0
+	if (mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva))
+		return RET_PF_RETRY;
+

This can effectively reduce the remote flush IPIs a lot!
One Nit is that, maybe rmb() or READ_ONCE() is required for kvm->mmu_invalidate_range_start and kvm->mmu_invalidate_range_end.
Otherwise, I'm somewhat worried about constant false positive and retry.

If anything, this needs a READ_ONCE() on mmu_invalidate_in_progress. The ranges aren't touched when when mmu_invalidate_in_progress goes to zero, so ensuring they are reloaded wouldn't do anything. The key to making forward progress is seeing that there is no in-progress invalidation.

I did consider adding said READ_ONCE(), but practically speaking, constant false positives are impossible. KVM will re-enter the guest when retrying, and there is zero chance of the compiler avoiding reloads across VM-Enter+VM-Exit.

I suppose in theory we might someday differentiate between "retry because a different vCPU may have fixed the fault" and "retry because there's an in-progress invalidation", and not bother re-entering the guest for the latter, e.g. have it try to yield instead.

All that said, READ_ONCE() on mmu_invalidate_in_progress should effectively be a nop, so it wouldn't hurt to be paranoid in this case.

Hmm, at that point, it probably makes sense to add a READ_ONCE() for mmu_invalidate_seq too, e.g. so that a sufficiently clever compiler doesn't completely optimize away the check. Losing the check wouldn't be problematic (false negatives are fine, especially on that particular check), but the generated code would *look* buggy.

#22 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 24 lines · bibo mao
Quoted 21 lines · Sean Christopherson
Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines

Note, I'm assuming secondary MMUs aren't allowed to map swap entries...

Compile tested only.

I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().

Quoted 6 lines
Quoted 4 lines

It pairs with existing call to mmu_notifier_invalidate_range_end() in change_pmd_range():

	if (range.start)
		mmu_notifier_invalidate_range_end(&range);

No, It doesn't work for mmu_notifier_invalidate_range_start() sent in change_pte_range(), if we only want the range to include pages successfully set to PROT_NONE.

Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the start() call still used the original @end.

Quoted 5 lines
Quoted 3 lines

The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.

For numa-balance scenery, can original page still be used by application even if pte is changed with PROT_NONE? If it can be used, maybe we can zap shadow mmu and flush tlb

Since there is kvm_mmu_notifier_change_pte notification when numa page is replaced with new page, my meaning that can original page still be used by application even if pte is changed with PROT_NONE and before replaced with new page?

And for primary mmu, tlb is flushed after pte is changed with PROT_NONE and after mmu_notifier_invalidate_range_end notification for secondary mmu.

Regards
Bibo Mao

Quoted 43 lines

in notification mmu_notifier_invalidate_range_end with precised range, the range can be cross-range between range mmu_gather and mmu_notifier_range.

Regards
Bibo Mao

Quoted 38 lines
Quoted 9 lines
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +0 -0
index 9e4cd8b4a202..f29718a16211 100644
@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
 	if (unlikely(!fault->slot))
 		return kvm_handle_noslot_fault(vcpu, fault, access);
 
Quoted 7 lines

arch/x86/kvm/mmu/mmu.c static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · line 4350

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3 -0
+	if (mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva))
+		return RET_PF_RETRY;
+

This can effectively reduce the remote flush IPIs a lot!
One Nit is that, maybe rmb() or READ_ONCE() is required for kvm->mmu_invalidate_range_start and kvm->mmu_invalidate_range_end.
Otherwise, I'm somewhat worried about constant false positive and retry.

If anything, this needs a READ_ONCE() on mmu_invalidate_in_progress. The ranges aren't touched when when mmu_invalidate_in_progress goes to zero, so ensuring they are reloaded wouldn't do anything. The key to making forward progress is seeing that there is no in-progress invalidation.

I did consider adding said READ_ONCE(), but practically speaking, constant false positives are impossible. KVM will re-enter the guest when retrying, and there is zero chance of the compiler avoiding reloads across VM-Enter+VM-Exit.

I suppose in theory we might someday differentiate between "retry because a different vCPU may have fixed the fault" and "retry because there's an in-progress invalidation", and not bother re-entering the guest for the latter, e.g. have it try to yield instead.

All that said, READ_ONCE() on mmu_invalidate_in_progress should effectively be a nop, so it wouldn't hurt to be paranoid in this case.

Hmm, at that point, it probably makes sense to add a READ_ONCE() for mmu_invalidate_seq too, e.g. so that a sufficiently clever compiler doesn't completely optimize away the check. Losing the check wouldn't be problematic (false negatives are fine, especially on that particular check), but the generated code would *look* buggy.

author
#23 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 27 lines · bibo mao
Quoted 24 lines · bibo mao
Quoted 21 lines · Sean Christopherson
Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines

Note, I'm assuming secondary MMUs aren't allowed to map swap entries...

Compile tested only.

I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().

Quoted 6 lines
Quoted 4 lines

It pairs with existing call to mmu_notifier_invalidate_range_end() in change_pmd_range():

	if (range.start)
		mmu_notifier_invalidate_range_end(&range);

No, It doesn't work for mmu_notifier_invalidate_range_start() sent in change_pte_range(), if we only want the range to include pages successfully set to PROT_NONE.

Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the start() call still used the original @end.

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.

For numa-balance scenery, can original page still be used by application even if pte is changed with PROT_NONE? If it can be used, maybe we can zap shadow mmu and flush tlb

For GUPs that does not honor FOLL_HONOR_NUMA_FAULT, yes,

See https://lore.kernel.org/all/[email protected]/

Quoted 3 lines

Since there is kvm_mmu_notifier_change_pte notification when numa page is replaced with new page, my meaning that can original page still be used by application even if pte is changed with PROT_NONE and before replaced with new page?

It's not .change_pte() notification, which is sent when COW. The do_numa_page()/do_huge_pmd_numa_page() will try to unmap old page protected with PROT_NONE, and if every check passes, a separate .invalidate_range_start()/end() with event type MMU_NOTIFY_CLEAR will be sent.

So, I think KVM (though it honors FOLL_HONOR_NUMA_FAULT), can safely keep mapping maybe-dma pages until MMU_NOTIFY_CLEAR is sent. (this approach is implemented in RFC v1
https://lore.kernel.org/all/[email protected]/)

Quoted 7 lines

And for primary mmu, tlb is flushed after pte is changed with PROT_NONE and after mmu_notifier_invalidate_range_end notification for secondary mmu. Regards
Bibo Mao

Quoted 1 line
Quoted 1 line

in notification mmu_notifier_invalidate_range_end with precised range, the range can

But I don't think flush tlb only in the .invalidate_range_end() in secondary MMU is a good idea.
Flush must be done before kvm->mmu_lock is unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.

Quoted 1 line
Quoted 1 line
Quoted 1 line

be cross-range between range mmu_gather and mmu_notifier_range.

#24 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 37 lines · Yan Zhao
Quoted 27 lines · bibo mao
Quoted 24 lines · bibo mao
Quoted 21 lines · Sean Christopherson
Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines

Note, I'm assuming secondary MMUs aren't allowed to map swap entries...

Compile tested only.

I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().

Quoted 6 lines
Quoted 4 lines

It pairs with existing call to mmu_notifier_invalidate_range_end() in change_pmd_range():

	if (range.start)
		mmu_notifier_invalidate_range_end(&range);

No, It doesn't work for mmu_notifier_invalidate_range_start() sent in change_pte_range(), if we only want the range to include pages successfully set to PROT_NONE.

Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the start() call still used the original @end.

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.

For numa-balance scenery, can original page still be used by application even if pte is changed with PROT_NONE? If it can be used, maybe we can zap shadow mmu and flush tlb

For GUPs that does not honor FOLL_HONOR_NUMA_FAULT, yes,

See https://lore.kernel.org/all/[email protected]/

Quoted 8 lines
Quoted 3 lines

Since there is kvm_mmu_notifier_change_pte notification when numa page is replaced with new page, my meaning that can original page still be used by application even if pte is changed with PROT_NONE and before replaced with new page?

It's not .change_pte() notification, which is sent when COW. The do_numa_page()/do_huge_pmd_numa_page() will try to unmap old page protected with PROT_NONE, and if every check passes, a separate .invalidate_range_start()/end() with event type MMU_NOTIFY_CLEAR will be sent.

yes, you are right. change_pte() notification will be will called when migrate_vma_pages, I messed it with numa page migration. However invalidate_range_start()/end() with event type MMU_NOTIFY_CLEAR will be sent also when new page is replaced.

Quoted 7 lines

So, I think KVM (though it honors FOLL_HONOR_NUMA_FAULT), can safely keep mapping maybe-dma pages until MMU_NOTIFY_CLEAR is sent. (this approach is implemented in RFC v1
https://lore.kernel.org/all/[email protected]/)

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

And for primary mmu, tlb is flushed after pte is changed with PROT_NONE and after mmu_notifier_invalidate_range_end notification for secondary mmu. Regards
Bibo Mao

Quoted 1 line
Quoted 1 line

in notification mmu_notifier_invalidate_range_end with precised range, the range can

But I don't think flush tlb only in the .invalidate_range_end() in secondary MMU is a good idea.

I have no good idea, and it beyond my ability to modify kvm framework now :(

Quoted 3 lines

Flush must be done before kvm->mmu_lock is unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.

Since tlb flush is delayed after all pte entries are cleared, and currently there is no tlb flush range supported for secondary mmu. I do know why there is confusion before or after kvm->mmu_lock.

Regards
Bibo Mao

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

be cross-range between range mmu_gather and mmu_notifier_range.

author
#25 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 6 lines · bibo mao
Quoted 3 lines

Flush must be done before kvm->mmu_lock is unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.

Since tlb flush is delayed after all pte entries are cleared, and currently there is no tlb flush range supported for secondary mmu. I do know why there is confusion before or after kvm->mmu_lock.

Oh, do you mean only do kvm_unmap_gfn_range() in .invalidate_range_end()? Then check if PROT_NONE is set in primary MMU before unmap? Looks like a good idea, I need to check if it's feasible. Thanks!

#26 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 9 lines · Yan Zhao
Quoted 6 lines · bibo mao
Quoted 3 lines

Flush must be done before kvm->mmu_lock is unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.

Since tlb flush is delayed after all pte entries are cleared, and currently there is no tlb flush range supported for secondary mmu. I do know why there is confusion before or after kvm->mmu_lock.

Oh, do you mean only do kvm_unmap_gfn_range() in .invalidate_range_end()?

yes, it is just sketchy thought for numa balance scenery, do kvm_unmap_gfn_range() in invalidate_range_end rather than invalidate_range_start.

Quoted 5 lines

Then check if PROT_NONE is set in primary MMU before unmap? Looks like a good idea, I need to check if it's feasible. Thanks!

#27 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

Quoted 7 lines · bibo mao
Quoted 4 lines · Yan Zhao
Quoted 3 lines · bibo mao
Quoted 3 lines

Flush must be done before kvm->mmu_lock is unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Since tlb flush is delayed after all pte entries are cleared, and currently there is no tlb flush range supported for secondary mmu. I do know why there is confusion before or after kvm->mmu_lock.

Oh, do you mean only do kvm_unmap_gfn_range() in .invalidate_range_end()?

yes, it is just sketchy thought for numa balance scenery, do kvm_unmap_gfn_range() in invalidate_range_end rather than invalidate_range_start.

That is not an option, it's a direction violation of the mmu_notifier contract. Secondary MMUs must drop all references before returning from invalidate_range_start().

#28 lore raw

Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration

In reply to Yan Zhao
kernel test robot noticed the following build warnings:

Hi Yan,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on kvm/queue]
[also build test WARNING on mst-vhost/linux-next linus/master v6.5-rc5 next-20230809] [cannot apply to akpm-mm/mm-everything kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Yan-Zhao/mm-mmu_notifier-introduce-a-new-mmu-notifier-flag-MMU_NOTIFIER_RANGE_NUMA/20230810-172955
base:   https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue
patch link:    https://lore.kernel.org/r/20230810090218.26244-1-yan.y.zhao%40intel.com
patch subject: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
config: riscv-randconfig-r023-20230810 (https://download.01.org/0day-ci/archive/20230810/[email protected]/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a)
reproduce: (https://download.01.org/0day-ci/archive/20230810/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> arch/riscv/kvm/../../../virt/kvm/kvm_main.c:767:23: warning: pointer type mismatch ('bool (*)(struct kvm *, struct kvm_gfn_range *)' (aka '_Bool (*)(struct kvm *, struct kvm_gfn_range *)') and 'void *') [-Wpointer-type-mismatch]
     767 |                 .handler        = !is_numa ? kvm_unmap_gfn_range :
         |                                            ^ ~~~~~~~~~~~~~~~~~~~
     768 |                                   (void *)kvm_null_fn,
         |                                   ~~~~~~~~~~~~~~~~~~~
>> arch/riscv/kvm/../../../virt/kvm/kvm_main.c:770:25: warning: pointer type mismatch ('void (*)(struct kvm *)' and 'void *') [-Wpointer-type-mismatch]
     770 |                 .on_unlock      = !is_numa ? kvm_arch_guest_memory_reclaimed :
         |                                            ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     771 |                                   (void *)kvm_null_fn,
         |                                   ~~~~~~~~~~~~~~~~~~~

2 warnings generated.

vim +767 arch/riscv/kvm/../../../virt/kvm/kvm_main.c

756
757 static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
758 const struct mmu_notifier_range *range)
759 {
760 struct kvm *kvm = mmu_notifier_to_kvm(mn);
761 bool is_numa = (range->event == MMU_NOTIFY_PROTECTION_VMA) &&
762 (range->flags & MMU_NOTIFIER_RANGE_NUMA);
763 const struct kvm_hva_range hva_range = {
764 .start = range->start,
765 .end = range->end,
766 .pte = __pte(0),

 > 767			.handler	= !is_numa ? kvm_unmap_gfn_range :
   768					  (void *)kvm_null_fn,
   769			.on_lock	= kvm_mmu_invalidate_begin,
 > 770			.on_unlock	= !is_numa ? kvm_arch_guest_memory_reclaimed :

771 (void *)kvm_null_fn,
772 .flush_on_ret = !is_numa ? true : false,
773 .may_block = mmu_notifier_range_blockable(range),
774 };
775
776 trace_kvm_unmap_hva_range(range->start, range->end);
777
778 /*
779 * Prevent memslot modification between range_start() and range_end()
780 * so that conditionally locking provides the same result in both
781 * functions. Without that guarantee, the mmu_invalidate_in_progress
782 * adjustments will be imbalanced.
783 *
784 * Pairs with the decrement in range_end().
785 */
786 spin_lock(&kvm->mn_invalidate_lock);
787 kvm->mn_active_invalidate_count++;
788 spin_unlock(&kvm->mn_invalidate_lock);
789
790 /*
791 * Invalidate pfn caches _before_ invalidating the secondary MMUs, i.e.
792 * before acquiring mmu_lock, to avoid holding mmu_lock while acquiring
793 * each cache's lock. There are relatively few caches in existence at
794 * any given time, and the caches themselves can check for hva overlap,
795 * i.e. don't need to rely on memslot overlap checks for performance.
796 * Because this runs without holding mmu_lock, the pfn caches must use
797 * mn_active_invalidate_count (see above) instead of
798 * mmu_invalidate_in_progress.
799 */
800 gfn_to_pfn_cache_invalidate_start(kvm, range->start, range->end,
801 hva_range.may_block);
802
803 __kvm_handle_hva_range(kvm, &hva_range);
804
805 return 0;
806 }
807

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#29 lore raw
In reply to Yan Zhao
Quoted 27 lines · Yan Zhao

This is an RFC series trying to fix the issue of unnecessary NUMA protection and TLB-shootdowns found in VMs with assigned devices or VFIO mediated devices during NUMA balance.

For VMs with assigned devices or VFIO mediated devices, all or part of guest memory are pinned for long-term.

Auto NUMA balancing will periodically selects VMAs of a process and change protections to PROT_NONE even though some or all pages in the selected ranges are long-term pinned for DMAs, which is true for VMs with assigned devices or VFIO mediated devices.

Though this will not cause real problem because NUMA migration will ultimately reject migration of those kind of pages and restore those PROT_NONE PTEs, it causes KVM's secondary MMU to be zapped periodically with equal SPTEs finally faulted back, wasting CPU cycles and generating unnecessary TLB-shootdowns.

This series first introduces a new flag MMU_NOTIFIER_RANGE_NUMA in patch 1 to work with mmu notifier event type MMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.

Patch 2 skips setting PROT_NONE to long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.

Quoted 7 lines

Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be PROT_NONE protected.

Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.

Why do we need all that, when we should simply not be applying PROT_NONE to pinned pages?

In change_pte_range() we already have:

if (is_cow_mapping(vma->vm_flags) &&

    page_count(page) != 1)

Which includes both, shared and pinned pages.

Staring at page #2, are we still missing something similar for THPs?

Why is that MMU notifier thingy and touching KVM code required?

Signature · 3 lines
Cheers,

David / dhildenb
author
#30 lore raw
Quoted 20 lines · David Hildenbrand
Quoted 9 lines

This series first introduces a new flag MMU_NOTIFIER_RANGE_NUMA in patch 1 to work with mmu notifier event type MMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.

Patch 2 skips setting PROT_NONE to long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.

Quoted 7 lines

Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be PROT_NONE protected.

Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.

Why do we need all that, when we should simply not be applying PROT_NONE to pinned pages?

Quoted 6 lines

In change_pte_range() we already have:

if (is_cow_mapping(vma->vm_flags) &&

    page_count(page) != 1)

Which includes both, shared and pinned pages.

Ah, right, currently in my side, I don't see any pinned pages are outside of this condition.
But I have a question regarding to is_cow_mapping(vma->vm_flags), do we need to allow pinned pages in !is_cow_mapping(vma->vm_flags)?

Quoted 1 line

Staring at page #2, are we still missing something similar for THPs?

Yes.

Quoted 1 line

Why is that MMU notifier thingy and touching KVM code required?

Because NUMA balancing code will firstly send .invalidate_range_start() with event type MMU_NOTIFY_PROTECTION_VMA to KVM in change_pmd_range() unconditionally, before it goes down into change_pte_range() and change_huge_pmd() to check each page count and apply PROT_NONE.

Then current KVM will unmap all notified pages from secondary MMU in .invalidate_range_start(), which could include pages that finally not set to PROT_NONE in primary MMU.

For VMs with pass-through devices, though all guest pages are pinned, KVM still periodically unmap pages in response to the .invalidate_range_start() notification from auto NUMA balancing, which is a waste.

So, if there's a new callback sent when pages is set to PROT_NONE for NUMA migrate only, KVM can unmap only those pages.
As KVM still needs to unmap pages for other type of event in its handler of .invalidate_range_start() (.i.e. kvm_mmu_notifier_invalidate_range_start()), and MMU_NOTIFY_PROTECTION_VMA also include other reasons, so patch 1 added a range flag to help KVM not to do a blind unmap in .invalidate_range_start(), but do it in the new .numa_protect() handler.

Quoted 7 lines
Signature · 4 lines
-- 
Cheers,

David / dhildenb
#31 lore raw
Quoted 23 lines · Yan Zhao
Quoted 22 lines · David Hildenbrand
Quoted 9 lines

This series first introduces a new flag MMU_NOTIFIER_RANGE_NUMA in patch 1 to work with mmu notifier event type MMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.

Patch 2 skips setting PROT_NONE to long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.

Quoted 7 lines

Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be PROT_NONE protected.

Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.

Why do we need all that, when we should simply not be applying PROT_NONE to pinned pages?

In change_pte_range() we already have:

Quoted 8 lines
Quoted 4 lines

if (is_cow_mapping(vma->vm_flags) &&

     page_count(page) != 1)

Which includes both, shared and pinned pages.

Ah, right, currently in my side, I don't see any pinned pages are outside of this condition.
But I have a question regarding to is_cow_mapping(vma->vm_flags), do we need to allow pinned pages in !is_cow_mapping(vma->vm_flags)?

One issue is that folio_maybe_pinned...() ... is unreliable as soon as your page is mapped more than 1024 times.

One might argue that we also want to exclude pages that are mapped that often. That might possibly work.

Quoted 9 lines
Quoted 2 lines

Staring at page #2, are we still missing something similar for THPs?

Yes.

Quoted 1 line

Why is that MMU notifier thingy and touching KVM code required?

Because NUMA balancing code will firstly send .invalidate_range_start() with event type MMU_NOTIFY_PROTECTION_VMA to KVM in change_pmd_range() unconditionally, before it goes down into change_pte_range() and change_huge_pmd() to check each page count and apply PROT_NONE.

Ah, okay I see, thanks. That's indeed unfortunate.

Quoted 9 lines

Then current KVM will unmap all notified pages from secondary MMU in .invalidate_range_start(), which could include pages that finally not set to PROT_NONE in primary MMU.

For VMs with pass-through devices, though all guest pages are pinned, KVM still periodically unmap pages in response to the .invalidate_range_start() notification from auto NUMA balancing, which is a waste.

Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are similar issues with GPU memory: NUMA hinting is actually counter-productive and they end up disabling it.

Signature · 3 lines
Cheers,

David / dhildenb
#32 lore raw

On 8/11/23 10:25, David Hildenbrand wrote:
...

Quoted 3 lines

One issue is that folio_maybe_pinned...() ... is unreliable as soon as your page is mapped more than 1024 times.

One might argue that we also want to exclude pages that are mapped that often. That might possibly work.

Yes.

Quoted 11 lines
Quoted 9 lines
Quoted 2 lines

Staring at page #2, are we still missing something similar for THPs?

Yes.

Quoted 1 line

Why is that MMU notifier thingy and touching KVM code required?

Because NUMA balancing code will firstly send .invalidate_range_start() with event type MMU_NOTIFY_PROTECTION_VMA to KVM in change_pmd_range() unconditionally, before it goes down into change_pte_range() and change_huge_pmd() to check each page count and apply PROT_NONE.

Ah, okay I see, thanks. That's indeed unfortunate.

Sigh. All this difficulty reminds me that this mechanism was created in the early days of NUMA. I wonder sometimes lately whether the cost, in complexity and CPU time, is still worth it on today's hardware.

But of course I am deeply biased, so don't take that too seriously. See below. :)

Quoted 6 lines
Quoted 6 lines

Then current KVM will unmap all notified pages from secondary MMU in .invalidate_range_start(), which could include pages that finally not set to PROT_NONE in primary MMU.

Quoted 6 lines
Quoted 4 lines

For VMs with pass-through devices, though all guest pages are pinned, KVM still periodically unmap pages in response to the .invalidate_range_start() notification from auto NUMA balancing, which is a waste.

Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are

similar issues with GPU memory:  NUMA hinting is actually counter-productive and they end up disabling it.

Quoted 1 line

Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#33 lore raw
Quoted 8 lines
Quoted 1 line

Ah, okay I see, thanks. That's indeed unfortunate.

Sigh. All this difficulty reminds me that this mechanism was created in the early days of NUMA. I wonder sometimes lately whether the cost, in complexity and CPU time, is still worth it on today's hardware.

But of course I am deeply biased, so don't take that too seriously. See below. :)

:)

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines

Then current KVM will unmap all notified pages from secondary MMU in .invalidate_range_start(), which could include pages that finally not set to PROT_NONE in primary MMU.

Quoted 6 lines
Quoted 4 lines

For VMs with pass-through devices, though all guest pages are pinned, KVM still periodically unmap pages in response to the .invalidate_range_start() notification from auto NUMA balancing, which is a waste.

Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are

similar issues with GPU memory:  NUMA hinting is actually counter-productive and they end up disabling it.

Quoted 1 line
Quoted 6 lines

Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.

Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?

MMF_HAS_PINNED might be too restrictive. MMF_HAS_PINNED_LONGTERM might be better, but with things like iouring still too restrictive eventually.

I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.

CCing Mel.

Signature · 3 lines
Cheers,

David / dhildenb
#34 lore raw

On 8/11/23 11:39, David Hildenbrand wrote:
...

Quoted 11 lines
Quoted 10 lines
Quoted 1 line

Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are

similar issues with GPU memory:  NUMA hinting is actually counter-productive and they end up disabling it.

Quoted 1 line

Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.

Quoted 8 lines

Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?

MMF_HAS_PINNED might be too restrictive. MMF_HAS_PINNED_LONGTERM might be better, but with things like iouring still too restrictive eventually.

I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.

CCing Mel.

Let's discern between page pinning situations, and HMM-style situations. Page pinning of CPU memory is unnecessary when setting up for using that memory by modern GPUs or accelerators, because the latter can handle replayable page faults. So for such cases, the pages are in use by a GPU or accelerator, but unpinned.

The performance problem occurs because for those pages, the NUMA balancing causes unmapping, which generates callbacks to the device driver, which dutifully unmaps the pages from the GPU or accelerator, even if the GPU might be busy using those pages. The device promptly causes a device page fault, and the driver then re-establishes the device page table mapping, which is good until the next round of unmapping from the NUMA balancer.

hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the memremap_pages() region, come to think of it. That seems relatively easy and clean at first glance anyway.

For other regions (allocated by the device driver), a per-VMA flag seems about right: VM_NO_NUMA_BALANCING ?

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
author
#35 lore raw
Quoted 36 lines · John Hubbard

On 8/11/23 11:39, David Hildenbrand wrote:
...

Quoted 11 lines
Quoted 10 lines
Quoted 1 line

Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are

similar issues with GPU memory:  NUMA hinting is actually counter-productive and they end up disabling it.

Quoted 1 line

Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.

Quoted 8 lines

Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?

MMF_HAS_PINNED might be too restrictive. MMF_HAS_PINNED_LONGTERM might be better, but with things like iouring still too restrictive eventually.

I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.

CCing Mel.

Let's discern between page pinning situations, and HMM-style situations. Page pinning of CPU memory is unnecessary when setting up for using that memory by modern GPUs or accelerators, because the latter can handle replayable page faults. So for such cases, the pages are in use by a GPU or accelerator, but unpinned.

The performance problem occurs because for those pages, the NUMA balancing causes unmapping, which generates callbacks to the device driver, which dutifully unmaps the pages from the GPU or accelerator, even if the GPU might be busy using those pages. The device promptly causes a device page fault, and the driver then re-establishes the device page table mapping, which is good until the next round of unmapping from the NUMA balancer.

Quoted 8 lines

hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the memremap_pages() region, come to think of it. That seems relatively easy and clean at first glance anyway.

For other regions (allocated by the device driver), a per-VMA flag seems about right: VM_NO_NUMA_BALANCING ?

Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard in mmap() in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?

#36 lore raw

On 8/14/23 02:09, Yan Zhao wrote:
...

Quoted 8 lines
Quoted 8 lines

hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the memremap_pages() region, come to think of it. That seems relatively easy and clean at first glance anyway.

For other regions (allocated by the device driver), a per-VMA flag seems about right: VM_NO_NUMA_BALANCING ?

Quoted 7 lines

Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard in mmap() in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?

I'm hoping someone who thinks about VMs and VFIO often can chime in.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#37 lore raw
Quoted 18 lines · John Hubbard

On 8/14/23 02:09, Yan Zhao wrote:
...

Quoted 8 lines
Quoted 8 lines

hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the memremap_pages() region, come to think of it. That seems relatively easy and clean at first glance anyway.

For other regions (allocated by the device driver), a per-VMA flag seems about right: VM_NO_NUMA_BALANCING ?

Quoted 7 lines

Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard in mmap() in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?

Quoted 1 line

I'm hoping someone who thinks about VMs and VFIO often can chime in.

At least QEMU could just set it on the applicable VMAs (as said by Yuan Yao, using madvise).

BUT, I do wonder what value there would be for autonuma to still be active for the remainder of the hypervisor. If there is none, a prctl() would be better.

We already do have a mechanism in QEMU to get notified when longterm-pinning in the kernel might happen (and, therefore, MADV_DONTNEED must not be used):
* ram_block_discard_disable()
* ram_block_uncoordinated_discard_disable()

Signature · 3 lines
Cheers,

David / dhildenb
author
#38 lore raw
Quoted 19 lines · David Hildenbrand
Quoted 18 lines · John Hubbard

On 8/14/23 02:09, Yan Zhao wrote:
...

Quoted 15 lines
Quoted 8 lines

hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the memremap_pages() region, come to think of it. That seems relatively easy and clean at first glance anyway.

For other regions (allocated by the device driver), a per-VMA flag seems about right: VM_NO_NUMA_BALANCING ?

Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard in mmap() in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?

Quoted 6 lines
Quoted 1 line

I'm hoping someone who thinks about VMs and VFIO often can chime in.

At least QEMU could just set it on the applicable VMAs (as said by Yuan Yao, using madvise).

BUT, I do wonder what value there would be for autonuma to still be active

Currently MADV_* is up to 25

	#define MADV_COLLAPSE   25,

while madvise behavior is of type "int". So it's ok.

But vma->vm_flags is of "unsigned long", so it's full at least on 32bit platform.

Quoted 2 lines

for the remainder of the hypervisor. If there is none, a prctl() would be better.

Add a new field in "struct vma_numab_state" in vma, and use prctl() to update this field?

e.g.

struct vma_numab_state {
        unsigned long next_scan;
        unsigned long next_pid_reset;
        unsigned long access_pids[2];
	bool no_scan;
};
Quoted 5 lines

We already do have a mechanism in QEMU to get notified when longterm-pinning in the kernel might happen (and, therefore, MADV_DONTNEED must not be used):
* ram_block_discard_disable()
* ram_block_uncoordinated_discard_disable()

Looks this ram_block_discard allow/disallow state is global rather than per-VMA in QEMU.
So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and it's up to the user space to choose between per-VMA and complex way or global and simpler way?

#39 lore raw
Quoted 25 lines · Yan Zhao
Quoted 24 lines · David Hildenbrand
Quoted 18 lines · John Hubbard

On 8/14/23 02:09, Yan Zhao wrote:
...

Quoted 8 lines
Quoted 8 lines

hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the memremap_pages() region, come to think of it. That seems relatively easy and clean at first glance anyway.

For other regions (allocated by the device driver), a per-VMA flag seems about right: VM_NO_NUMA_BALANCING ?

Quoted 7 lines

Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard in mmap() in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?

Quoted 1 line

I'm hoping someone who thinks about VMs and VFIO often can chime in.

At least QEMU could just set it on the applicable VMAs (as said by Yuan Yao, using madvise).

Quoted 6 lines
Quoted 1 line

BUT, I do wonder what value there would be for autonuma to still be active

Currently MADV_* is up to 25

	#define MADV_COLLAPSE   25,

while madvise behavior is of type "int". So it's ok.

But vma->vm_flags is of "unsigned long", so it's full at least on 32bit platform.

I remember there were discussions to increase it also for 32bit. If that's required, we might want to go down that path.

But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...

Quoted 5 lines
Quoted 3 lines

for the remainder of the hypervisor. If there is none, a prctl() would be better.

Add a new field in "struct vma_numab_state" in vma, and use prctl() to update this field?

Rather a global toggle per MM, no need to update individual VMAs -- if we go down that prctl() path.

No need to consume more memory for VMAs.

[...]

Quoted 6 lines
Quoted 4 lines

We already do have a mechanism in QEMU to get notified when longterm-pinning in the kernel might happen (and, therefore, MADV_DONTNEED must not be used):
* ram_block_discard_disable()
* ram_block_uncoordinated_discard_disable()

Looks this ram_block_discard allow/disallow state is global rather than per-VMA in QEMU.

Yes. Once you transition into "discard of any kind disabled", you can go over all guest memory VMAs (RAMBlock) and issue an madvise() for them. (or alternatively, do the prctl() once )

We'll also have to handle new guest memory being created afterwards, but that is easy.

Once we transition to "no discarding disabled", you can go over all guest memory VMAs (RAMBlock) and issue an madvise() for them again (or alternatively, do the prctl() once).

Quoted 3 lines

So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and it's up to the user space to choose between per-VMA and complex way or global and simpler way?

QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.

Signature · 3 lines
Cheers,

David / dhildenb
#40 lore raw
Quoted 1 line · David Hildenbrand

But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...

Probably not!

...

Quoted 5 lines
Quoted 3 lines

So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and it's up to the user space to choose between per-VMA and complex way or global and simpler way?

QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.

From our experience with compute on GPUs, a per-mm setting would suffice. No need to go all the way to VMA granularity.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
author
#41 lore raw
Quoted 12 lines · John Hubbard
Quoted 2 lines · David Hildenbrand

But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...

Probably not!

...

Quoted 5 lines
Quoted 4 lines

So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and
it's up to the user space to choose between per-VMA and complex way or global and simpler way?

Quoted 6 lines
Quoted 2 lines

QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.

From our experience with compute on GPUs, a per-mm setting would suffice. No need to go all the way to VMA granularity.

After an offline internal discussion, we think a per-mm setting is also enough for device passthrough in VMs.

BTW, if we want a per-VMA flag, compared to VM_NO_NUMA_BALANCING, do you think it's of any value to providing a flag like VM_MAYDMA? Auto NUMA balancing or other components can decide how to use it by themselves.

#42 lore raw
Quoted 19 lines · Yan Zhao
Quoted 12 lines · John Hubbard
Quoted 2 lines · David Hildenbrand

But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...

Probably not!

...

Quoted 5 lines
Quoted 4 lines

So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and
it's up to the user space to choose between per-VMA and complex way or global and simpler way?

Quoted 6 lines
Quoted 2 lines

QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.

From our experience with compute on GPUs, a per-mm setting would suffice.
No need to go all the way to VMA granularity.

Quoted 7 lines

After an offline internal discussion, we think a per-mm setting is also enough for device passthrough in VMs.

BTW, if we want a per-VMA flag, compared to VM_NO_NUMA_BALANCING, do you think it's of any value to providing a flag like VM_MAYDMA? Auto NUMA balancing or other components can decide how to use it by themselves.

Short-lived DMA is not really the problem. The problem is long-term pinning.

There was a discussion about letting user space similarly hint that long-term pinning might/will happen.

Because when long-term pinning a page we have to make sure to migrate it off of ZONE_MOVABLE / MIGRATE_CMA.

But the kernel prefers to place pages there.

So with vfio in QEMU, we might preallocate memory for the guest and place it on ZONE_MOVABLE/MIGRATE_CMA, just so long-term pinning has to migrate all these fresh pages out of these areas again.

So letting the kernel know about that in this context might also help.

Signature · 3 lines
Cheers,

David / dhildenb
author
#43 lore raw
Quoted 36 lines · David Hildenbrand
Quoted 19 lines · Yan Zhao
Quoted 12 lines · John Hubbard
Quoted 2 lines · David Hildenbrand

But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...

Probably not!

...

Quoted 5 lines
Quoted 4 lines

So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and
it's up to the user space to choose between per-VMA and complex way or global and simpler way?

Quoted 6 lines
Quoted 2 lines

QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.

From our experience with compute on GPUs, a per-mm setting would suffice.
No need to go all the way to VMA granularity.

Quoted 7 lines

After an offline internal discussion, we think a per-mm setting is also enough for device passthrough in VMs.

BTW, if we want a per-VMA flag, compared to VM_NO_NUMA_BALANCING, do you think it's of any value to providing a flag like VM_MAYDMA? Auto NUMA balancing or other components can decide how to use it by themselves.

Short-lived DMA is not really the problem. The problem is long-term pinning.

There was a discussion about letting user space similarly hint that long-term pinning might/will happen.

Because when long-term pinning a page we have to make sure to migrate it off of ZONE_MOVABLE / MIGRATE_CMA.

Quoted 8 lines

But the kernel prefers to place pages there.

So with vfio in QEMU, we might preallocate memory for the guest and place it on ZONE_MOVABLE/MIGRATE_CMA, just so long-term pinning has to migrate all these fresh pages out of these areas again.

So letting the kernel know about that in this context might also help.

Thanks! Glad to know it :)
But consider for GPUs case as what John mentioned, since the memory is not even pinned, maybe they still need flag VM_NO_NUMA_BALANCING ? For VMs, we hint VM_NO_NUMA_BALANCING for passthrough devices supporting IO page fault (so no need to pin), and VM_MAYLONGTERMDMA to avoid misplace and migration.

Is that good?
Or do you think just a per-mm flag like MMF_NO_NUMA is good enough for now?

#44 lore raw

On 8/17/23 17:13, Yan Zhao wrote:
...

Quoted 10 lines

But consider for GPUs case as what John mentioned, since the memory is not even pinned, maybe they still need flag VM_NO_NUMA_BALANCING ? For VMs, we hint VM_NO_NUMA_BALANCING for passthrough devices supporting IO page fault (so no need to pin), and VM_MAYLONGTERMDMA to avoid misplace and migration.

Is that good?
Or do you think just a per-mm flag like MMF_NO_NUMA is good enough for now?

So far, a per-mm setting seems like it would suffice. However, it is also true that new hardware is getting really creative and large, to the point that it's not inconceivable that a process might actually want to let NUMA balancing run in part of its mm, but turn it off to allow fault-able device access to another part of the mm.

We aren't seeing that yet, but on the other hand, that may be simply because there is no practical way to set that up and see how well it works.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
author
#45 lore raw
Quoted 19 lines · John Hubbard

On 8/17/23 17:13, Yan Zhao wrote:
...

Quoted 10 lines

But consider for GPUs case as what John mentioned, since the memory is not even pinned, maybe they still need flag VM_NO_NUMA_BALANCING ? For VMs, we hint VM_NO_NUMA_BALANCING for passthrough devices supporting IO page fault (so no need to pin), and VM_MAYLONGTERMDMA to avoid misplace and migration.

Is that good?
Or do you think just a per-mm flag like MMF_NO_NUMA is good enough for now?

So far, a per-mm setting seems like it would suffice. However, it is also true that new hardware is getting really creative and large, to the point that it's not inconceivable that a process might actually want to let NUMA balancing run in part of its mm, but turn it off to allow fault-able device access to another part of the mm.

Quoted 5 lines

We aren't seeing that yet, but on the other hand, that may be simply because there is no practical way to set that up and see how well it works.

Hi guys,
Thanks a lot for your review and suggestions!
I'll firstly try to add a per-mm flag to fix this problem later (but maybe not very soon)

Thanks
Yan

#46 lore raw
In reply to Yan Zhao
Quoted 45 lines · Yan Zhao
Quoted 36 lines · John Hubbard

On 8/11/23 11:39, David Hildenbrand wrote:
...

Quoted 11 lines
Quoted 10 lines
Quoted 1 line

Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are

similar issues with GPU memory:  NUMA hinting is actually counter-productive and they end up disabling it.

Quoted 1 line

Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.

Quoted 8 lines

Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?

MMF_HAS_PINNED might be too restrictive. MMF_HAS_PINNED_LONGTERM might be better, but with things like iouring still too restrictive eventually.

I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.

CCing Mel.

Let's discern between page pinning situations, and HMM-style situations. Page pinning of CPU memory is unnecessary when setting up for using that memory by modern GPUs or accelerators, because the latter can handle replayable page faults. So for such cases, the pages are in use by a GPU or accelerator, but unpinned.

The performance problem occurs because for those pages, the NUMA balancing causes unmapping, which generates callbacks to the device driver, which dutifully unmaps the pages from the GPU or accelerator, even if the GPU might be busy using those pages. The device promptly causes a device page fault, and the driver then re-establishes the device page table mapping, which is good until the next round of unmapping from the NUMA balancer.

Quoted 8 lines

hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the memremap_pages() region, come to think of it. That seems relatively easy and clean at first glance anyway.

For other regions (allocated by the device driver), a per-VMA flag seems about right: VM_NO_NUMA_BALANCING ?

Quoted 6 lines

Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard in mmap() in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?

Emm... syscall madvise() in my mind, it does things like change flags on VMA, e.g madvise(MADV_DONTFORK) adds VM_DONTCOPY to the VMA.

Quoted 2 lines
author
#47 lore raw
Quoted 53 lines · Yuan Yao
Quoted 45 lines · Yan Zhao
Quoted 36 lines · John Hubbard

On 8/11/23 11:39, David Hildenbrand wrote:
...

Quoted 11 lines
Quoted 10 lines
Quoted 1 line

Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are

similar issues with GPU memory:  NUMA hinting is actually counter-productive and they end up disabling it.

Quoted 1 line

Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.

Quoted 8 lines

Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?

MMF_HAS_PINNED might be too restrictive. MMF_HAS_PINNED_LONGTERM might be better, but with things like iouring still too restrictive eventually.

I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.

CCing Mel.

Let's discern between page pinning situations, and HMM-style situations. Page pinning of CPU memory is unnecessary when setting up for using that memory by modern GPUs or accelerators, because the latter can handle replayable page faults. So for such cases, the pages are in use by a GPU or accelerator, but unpinned.

The performance problem occurs because for those pages, the NUMA balancing causes unmapping, which generates callbacks to the device driver, which dutifully unmaps the pages from the GPU or accelerator, even if the GPU might be busy using those pages. The device promptly causes a device page fault, and the driver then re-establishes the device page table mapping, which is good until the next round of unmapping from the NUMA balancer.

Quoted 8 lines

hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the memremap_pages() region, come to think of it. That seems relatively easy and clean at first glance anyway.

For other regions (allocated by the device driver), a per-VMA flag seems about right: VM_NO_NUMA_BALANCING ?

Quoted 6 lines

Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard in mmap() in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?

Quoted 2 lines

Emm... syscall madvise() in my mind, it does things like change flags on VMA, e.g madvise(MADV_DONTFORK) adds VM_DONTCOPY to the VMA.

Yes, madvise() might work.
And setting this flag might be an easy decision, while unsetting it might be hard unless some counters introduced.

#48 lore raw
In reply to Yan Zhao
Quoted 12 lines · Yan Zhao

This is an RFC series trying to fix the issue of unnecessary NUMA protection and TLB-shootdowns found in VMs with assigned devices or VFIO mediated devices during NUMA balance.

For VMs with assigned devices or VFIO mediated devices, all or part of guest memory are pinned for long-term.

Auto NUMA balancing will periodically selects VMAs of a process and change protections to PROT_NONE even though some or all pages in the selected ranges are long-term pinned for DMAs, which is true for VMs with assigned devices or VFIO mediated devices.

Quoted 5 lines

Though this will not cause real problem because NUMA migration will ultimately reject migration of those kind of pages and restore those PROT_NONE PTEs, it causes KVM's secondary MMU to be zapped periodically with equal SPTEs finally faulted back, wasting CPU cycles and generating unnecessary TLB-shootdowns.

In my understanding, NUMA balancing also moves tasks closer to the memory they are accessing. Can this still work with this series applied?

Quoted 48 lines

This series first introduces a new flag MMU_NOTIFIER_RANGE_NUMA in patch 1 to work with mmu notifier event type MMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.

Patch 2 skips setting PROT_NONE to long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.

Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be PROT_NONE protected.

Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.

Changelog:
RFC v1 --> v2:
1. added patch 3-4 to introduce a new callback .numa_protect()
2. Rather than have KVM duplicate logic to check if a page is pinned for long-term, let KVM depend on the new callback .numa_protect() to do the page unmap in secondary MMU for NUMA migration purpose.

RFC v1:
https://lore.kernel.org/all/[email protected]/

Yan Zhao (5):
mm/mmu_notifier: introduce a new mmu notifier flag
MMU_NOTIFIER_RANGE_NUMA
mm: don't set PROT_NONE to maybe-dma-pinned pages for NUMA-migrate
purpose
mm/mmu_notifier: introduce a new callback .numa_protect mm/autonuma: call .numa_protect() when page is protected for NUMA
migrate
KVM: Unmap pages only when it's indeed protected for NUMA migration

include/linux/mmu_notifier.h | 16 ++++++++++++++++
mm/huge_memory.c             |  6 ++++++
mm/mmu_notifier.c            | 18 ++++++++++++++++++
mm/mprotect.c                | 10 +++++++++-
virt/kvm/kvm_main.c          | 25 ++++++++++++++++++++++---
5 files changed, 71 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.1
author
#49 lore raw
Quoted 13 lines · Chao Gao
Quoted 12 lines · Yan Zhao

This is an RFC series trying to fix the issue of unnecessary NUMA protection and TLB-shootdowns found in VMs with assigned devices or VFIO mediated devices during NUMA balance.

For VMs with assigned devices or VFIO mediated devices, all or part of guest memory are pinned for long-term.

Auto NUMA balancing will periodically selects VMAs of a process and change protections to PROT_NONE even though some or all pages in the selected ranges are long-term pinned for DMAs, which is true for VMs with assigned devices or VFIO mediated devices.

Quoted 9 lines
Quoted 5 lines

Though this will not cause real problem because NUMA migration will ultimately reject migration of those kind of pages and restore those PROT_NONE PTEs, it causes KVM's secondary MMU to be zapped periodically with equal SPTEs finally faulted back, wasting CPU cycles and generating unnecessary TLB-shootdowns.

In my understanding, NUMA balancing also moves tasks closer to the memory they are accessing. Can this still work with this series applied?

For pages protected with PROT_NONE in primary MMU in scanning phase, yes; For pages not set to PROT_NONE, no.
Because looks this task_numa_migrate() is only triggered in next page fault when PROT_NONE and accessible VMA is found.