Reduce NUMA balance caused TLB-shootdowns in a VM
Subsystems (7)
v2 cover Reduce NUMA balance caused TLB-shootdowns in a VM
This is an RFC series trying to fix the issue of unnecessary NUMA protection and TLB-shootdowns found in VMs with assigned devices or VFIO mediated devices during NUMA balance.
For VMs with assigned devices or VFIO mediated devices, all or part of guest memory are pinned for long-term.
Auto NUMA balancing will periodically selects VMAs of a process and change protections to PROT_NONE even though some or all pages in the selected ranges are long-term pinned for DMAs, which is true for VMs with assigned devices or VFIO mediated devices.
Though this will not cause real problem because NUMA migration will ultimately reject migration of those kind of pages and restore those PROT_NONE PTEs, it causes KVM's secondary MMU to be zapped periodically with equal SPTEs finally faulted back, wasting CPU cycles and generating unnecessary TLB-shootdowns.
This series first introduces a new flag MMU_NOTIFIER_RANGE_NUMA in patch 1 to work with mmu notifier event type MMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.
Patch 2 skips setting PROT_NONE to long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.
Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be PROT_NONE protected.
Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.
Changelog:
RFC v1 --> v2:
1. added patch 3-4 to introduce a new callback .numa_protect()
2. Rather than have KVM duplicate logic to check if a page is pinned for long-term, let KVM depend on the new callback .numa_protect() to do the page unmap in secondary MMU for NUMA migration purpose.
RFC v1:
https://lore.kernel.org/all/[email protected]/
Yan Zhao (5):
mm/mmu_notifier: introduce a new mmu notifier flag
MMU_NOTIFIER_RANGE_NUMA
mm: don't set PROT_NONE to maybe-dma-pinned pages for NUMA-migrate
purpose
mm/mmu_notifier: introduce a new callback .numa_protect mm/autonuma: call .numa_protect() when page is protected for NUMA
migrate
KVM: Unmap pages only when it's indeed protected for NUMA migration
include/linux/mmu_notifier.h | 16 ++++++++++++++++ mm/huge_memory.c | 6 ++++++ mm/mmu_notifier.c | 18 ++++++++++++++++++ mm/mprotect.c | 10 +++++++++- virt/kvm/kvm_main.c | 25 ++++++++++++++++++++++--- 5 files changed, 71 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.1
RFC PATCH v2 1/5 mm/mmu_notifier: introduce a new mmu notifier flag MMU_NOTIFIER_RANGE_NUMA
Introduce a new mmu notifier flag MMU_NOTIFIER_RANGE_NUMA to indicate the notification of MMU_NOTIFY_PROTECTION_VMA is for NUMA balance purpose specifically.
So that, the subscriber of mmu notifier, like KVM, can recognize this type of notification and do numa protection specific operations in the handler.
--- include/linux/mmu_notifier.h | 1 + mm/mprotect.c | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-)
include/linux/mmu_notifier.h +1 -0
@@ -60,6 +60,7 @@ enum mmu_notifier_event { };#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_RANGE_NUMA (1 << 1)struct mmu_notifier_ops {/*
mm/mprotect.c +3 -1
@@ -381,7 +381,9 @@ static inline long change_pmd_range(struct mmu_gather *tlb,/* invoke the mmu notifier if the pmd is populated */ if (!range.start) {mmu_notifier_range_init (&range, -MMU_NOTIFY_PROTECTION_VMA ,0 , +MMU_NOTIFY_PROTECTION_VMA , + cp_flags &MM_CP_PROT_NUMA ? +MMU_NOTIFIER_RANGE_NUMA :0 , vma->vm_mm, addr, end);mmu_notifier_invalidate_range_start (&range); }
Signature · 1 line
2.17.1
RFC PATCH v2 2/5 mm: don't set PROT_NONE to maybe-dma-pinned pages for NUMA-migrate purpose
In reply to Yan ZhaoDon't set PROT_NONE for exclusive anonymas and maybe-dma-pinned pages for NUMA migration purpose.
For exclusive anonymas and page_maybe_dma_pinned() pages, NUMA-migration will eventually drop migration of those pages in try_to_migrate_one(). (i.e. after -EBUSY returned in page_try_share_anon_rmap()).
So, skip setting PROT_NONE to those kind of pages earlier in change_protection_range() phase to avoid later futile page faults, detections, and restoration to original PTEs/PMDs.
--- mm/huge_memory.c | 5 +++++ mm/mprotect.c | 5 +++++ 2 files changed, 10 insertions(+)
mm/huge_memory.c +5 -0
@@ -1875,6 +1875,11 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, goto unlock; page =pmd_page (*pmd); + +if (PageAnon (page) &&PageAnonExclusive (page) && +page_maybe_dma_pinned (page)) + goto unlock; + toptier =node_is_toptier (page_to_nid (page));/* * Skip scanning top tier node if normal numa
mm/mprotect.c +5 -0
@@ -146,6 +146,11 @@ static long change_pte_range(struct mmu_gather *tlb, nid =page_to_nid (page);if (target_node == nid)continue ; + +if (PageAnon (page) &&PageAnonExclusive (page) && +page_maybe_dma_pinned (page)) +continue ; + toptier =node_is_toptier (nid);/*
Signature · 1 line
2.17.1
RFC PATCH v2 3/5 mm/mmu_notifier: introduce a new callback .numa_protect
In reply to Yan ZhaoThis .numa_protect callback is called when PROT_NONE is set for sure on a PTE or a huge PMD for numa migration purpose.
With this callback, subscriber of mmu notifier, (e.g. KVM), can unmap NUMA migration protected pages only in the handler, rather than unmap a wider range containing pages that are obvious none-NUMA-migratble.
--- include/linux/mmu_notifier.h | 15 +++++++++++++++ mm/mmu_notifier.c | 18 ++++++++++++++++++ 2 files changed, 33 insertions(+)
include/linux/mmu_notifier.h +15 -0
@@ -132,6 +132,10 @@ struct mmu_notifier_ops {unsigned long address, pte_t pte); +void (*numa_protect)(struct mmu_notifier *subscription, +struct mm_struct *mm, +unsigned long start, +unsigned long end);/* * invalidate_range_start() and invalidate_range_end() must be * paired and are called only when the mmap_lock and/or the
@@ -395,6 +399,9 @@ extern int __mmu_notifier_test_young(struct mm_struct *mm, unsigned long address);extern void __mmu_notifier_change_pte (struct mm_struct *mm,unsigned long address,pte_t pte); +extern void __mmu_notifier_numa_protect (struct mm_struct *mm, +unsigned long start, +unsigned long end);extern int __mmu_notifier_invalidate_range_start (struct mmu_notifier_range *r);extern void __mmu_notifier_invalidate_range_end (struct mmu_notifier_range *r,bool only_end);
@@ -448,6 +455,14 @@ static inline void mmu_notifier_change_pte(struct mm_struct *mm,__mmu_notifier_change_pte (mm, address, pte); } +static inline void mmu_notifier_numa_protect (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +if (mm_has_notifiers (mm)) +__mmu_notifier_numa_protect (mm, start, end); +} +static inline void mmu_notifier_invalidate_range_start (struct mmu_notifier_range *range) {
mm/mmu_notifier.c +18 -0
@@ -382,6 +382,24 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm,return young; } +void __mmu_notifier_numa_protect (struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +struct mmu_notifier *subscription; +int id; + + id =srcu_read_lock (&srcu); +hlist_for_each_entry_rcu (subscription, + &mm->notifier_subscriptions->list, hlist, +srcu_read_lock_held (&srcu)) { +if (subscription->ops->numa_protect) + subscription->ops->numa_protect (subscription, mm, start, + end); + } +srcu_read_unlock (&srcu, id); +} +int __mmu_notifier_clear_young (struct mm_struct *mm,unsigned long start,unsigned long end)
Signature · 1 line
2.17.1
RFC PATCH v2 4/5 mm/autonuma: call .numa_protect() when page is protected for NUMA migrate
In reply to Yan ZhaoCall mmu notifier's callback .numa_protect() in change_pmd_range() when a page is ensured to be protected by PROT_NONE for NUMA migration purpose.
--- mm/huge_memory.c | 1 + mm/mprotect.c | 1 + 2 files changed, 2 insertions(+)
mm/huge_memory.c +1 -0
@@ -1892,6 +1892,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,if (sysctl_numa_balancing_mode &NUMA_BALANCING_MEMORY_TIERING && !toptier)xchg_page_access_time (page,jiffies_to_msecs (jiffies)); +mmu_notifier_numa_protect (vma->vm_mm, addr, addr +PMD_SIZE ); }/* * In case prot_numa, we are under mmap_read_lock(mm). It's critical
mm/mprotect.c +1 -0
@@ -164,6 +164,7 @@ static long change_pte_range(struct mmu_gather *tlb, !toptier)xchg_page_access_time (page,jiffies_to_msecs (jiffies)); +mmu_notifier_numa_protect (vma->vm_mm, addr, addr +PAGE_SIZE ); } oldpte =ptep_modify_prot_start (vma, addr, pte);
Signature · 1 line
2.17.1
kernel test robot
build botRe: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate
kernel test robot noticed the following build errors:
Hi Yan,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on kvm/queue]
[also build test ERROR on mst-vhost/linux-next linus/master v6.5-rc5 next-20230809] [cannot apply to akpm-mm/mm-everything kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Yan-Zhao/mm-mmu_notifier-introduce-a-new-mmu-notifier-flag-MMU_NOTIFIER_RANGE_NUMA/20230810-172955 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue patch link: https://lore.kernel.org/r/20230810090048.26184-1-yan.y.zhao%40intel.com patch subject: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate config: um-allnoconfig (https://download.01.org/0day-ci/archive/20230810/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a) reproduce: (https://download.01.org/0day-ci/archive/20230810/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from mm/mprotect.c:13:
In file included from include/linux/hugetlb.h:10:
In file included from include/linux/cgroup.h:26:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:547:31: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
547 | val = __raw_readb(PCI_IOBASE + addr);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:560:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
560 | val = __le16_to_cpu((__le16 __force)__raw_readw(PCI_IOBASE + addr));
| ~~~~~~~~~~ ^
include/uapi/linux/byteorder/little_endian.h:37:51: note: expanded from macro '__le16_to_cpu'
37 | #define __le16_to_cpu(x) ((__force __u16)(__le16)(x))
| ^
In file included from mm/mprotect.c:13:
In file included from include/linux/hugetlb.h:10:
In file included from include/linux/cgroup.h:26:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:573:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
573 | val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
| ~~~~~~~~~~ ^
include/uapi/linux/byteorder/little_endian.h:35:51: note: expanded from macro '__le32_to_cpu'
35 | #define __le32_to_cpu(x) ((__force __u32)(__le32)(x))
| ^
In file included from mm/mprotect.c:13:
In file included from include/linux/hugetlb.h:10:
In file included from include/linux/cgroup.h:26:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:584:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
584 | __raw_writeb(value, PCI_IOBASE + addr);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:594:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
594 | __raw_writew((u16 __force)cpu_to_le16(value), PCI_IOBASE + addr);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:604:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
604 | __raw_writel((u32 __force)cpu_to_le32(value), PCI_IOBASE + addr);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:692:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
692 | readsb(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:700:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
700 | readsw(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:708:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
708 | readsl(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:717:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
717 | writesb(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:726:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
726 | writesw(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:735:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
735 | writesl(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
>> mm/mprotect.c:167:5: error: call to undeclared function 'mmu_notifier_numa_protect'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
167 | mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PAGE_SIZE);
| ^12 warnings and 1 error generated.
vim +/mmu_notifier_numa_protect +167 mm/mprotect.c
82
83 static long change_pte_range(struct mmu_gather *tlb,
84 struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr,
85 unsigned long end, pgprot_t newprot, unsigned long cp_flags)
86 {
87 pte_t *pte, oldpte;
88 spinlock_t *ptl;
89 long pages = 0;
90 int target_node = NUMA_NO_NODE;
91 bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
92 bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
93 bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
94
95 tlb_change_page_size(tlb, PAGE_SIZE);
96 pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
97 if (!pte)
98 return -EAGAIN;
99
100 /* Get target node for single threaded private VMAs */
101 if (prot_numa && !(vma->vm_flags & VM_SHARED) &&
102 atomic_read(&vma->vm_mm->mm_users) == 1)
103 target_node = numa_node_id();
104
105 flush_tlb_batched_pending(vma->vm_mm);
106 arch_enter_lazy_mmu_mode();
107 do {
108 oldpte = ptep_get(pte);
109 if (pte_present(oldpte)) {
110 pte_t ptent;
111
112 /*
113 * Avoid trapping faults against the zero or KSM
114 * pages. See similar comment in change_huge_pmd.
115 */
116 if (prot_numa) {
117 struct page *page;
118 int nid;
119 bool toptier;
120
121 /* Avoid TLB flush if possible */
122 if (pte_protnone(oldpte))
123 continue;
124
125 page = vm_normal_page(vma, addr, oldpte);
126 if (!page || is_zone_device_page(page) || PageKsm(page))
127 continue;
128
129 /* Also skip shared copy-on-write pages */
130 if (is_cow_mapping(vma->vm_flags) &&
131 page_count(page) != 1)
132 continue;
133
134 /*
135 * While migration can move some dirty pages,
136 * it cannot move them all from MIGRATE_ASYNC
137 * context.
138 */
139 if (page_is_file_lru(page) && PageDirty(page))
140 continue;
141
142 /*
143 * Don't mess with PTEs if page is already on the node
144 * a single-threaded process is running on.
145 */
146 nid = page_to_nid(page);
147 if (target_node == nid)
148 continue;
149
150 if (PageAnon(page) && PageAnonExclusive(page) &&
151 page_maybe_dma_pinned(page))
152 continue;
153
154 toptier = node_is_toptier(nid);
155
156 /*
157 * Skip scanning top tier node if normal numa
158 * balancing is disabled
159 */
160 if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
161 toptier)
162 continue;
163 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
164 !toptier)
165 xchg_page_access_time(page,
166 jiffies_to_msecs(jiffies));
> 167 mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PAGE_SIZE); 168 }
169
170 oldpte = ptep_modify_prot_start(vma, addr, pte);
171 ptent = pte_modify(oldpte, newprot);
172
173 if (uffd_wp)
174 ptent = pte_mkuffd_wp(ptent);
175 else if (uffd_wp_resolve)
176 ptent = pte_clear_uffd_wp(ptent);
177
178 /*
179 * In some writable, shared mappings, we might want
180 * to catch actual write access -- see
181 * vma_wants_writenotify().
182 *
183 * In all writable, private mappings, we have to
184 * properly handle COW.
185 *
186 * In both cases, we can sometimes still change PTEs
187 * writable and avoid the write-fault handler, for
188 * example, if a PTE is already dirty and no other
189 * COW or special handling is required.
190 */
191 if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
192 !pte_write(ptent) &&
193 can_change_pte_writable(vma, addr, ptent))
194 ptent = pte_mkwrite(ptent);
195
196 ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent);
197 if (pte_needs_flush(oldpte, ptent))
198 tlb_flush_pte_range(tlb, addr, PAGE_SIZE);
199 pages++;
200 } else if (is_swap_pte(oldpte)) {
201 swp_entry_t entry = pte_to_swp_entry(oldpte);
202 pte_t newpte;
203
204 if (is_writable_migration_entry(entry)) {
205 struct page *page = pfn_swap_entry_to_page(entry);
206
207 /*
208 * A protection check is difficult so
209 * just be safe and disable write
210 */
211 if (PageAnon(page))
212 entry = make_readable_exclusive_migration_entry(
213 swp_offset(entry));
214 else
215 entry = make_readable_migration_entry(swp_offset(entry));
216 newpte = swp_entry_to_pte(entry);
217 if (pte_swp_soft_dirty(oldpte))
218 newpte = pte_swp_mksoft_dirty(newpte);
219 } else if (is_writable_device_private_entry(entry)) {
220 /*
221 * We do not preserve soft-dirtiness. See
222 * copy_one_pte() for explanation.
223 */
224 entry = make_readable_device_private_entry(
225 swp_offset(entry));
226 newpte = swp_entry_to_pte(entry);
227 if (pte_swp_uffd_wp(oldpte))
228 newpte = pte_swp_mkuffd_wp(newpte);
229 } else if (is_writable_device_exclusive_entry(entry)) {
230 entry = make_readable_device_exclusive_entry(
231 swp_offset(entry));
232 newpte = swp_entry_to_pte(entry);
233 if (pte_swp_soft_dirty(oldpte))
234 newpte = pte_swp_mksoft_dirty(newpte);
235 if (pte_swp_uffd_wp(oldpte))
236 newpte = pte_swp_mkuffd_wp(newpte);
237 } else if (is_pte_marker_entry(entry)) {
238 /*
239 * Ignore swapin errors unconditionally,
240 * because any access should sigbus anyway.
241 */
242 if (is_swapin_error_entry(entry))
243 continue;
244 /*
245 * If this is uffd-wp pte marker and we'd like
246 * to unprotect it, drop it; the next page
247 * fault will trigger without uffd trapping.
248 */
249 if (uffd_wp_resolve) {
250 pte_clear(vma->vm_mm, addr, pte);
251 pages++;
252 }
253 continue;
254 } else {
255 newpte = oldpte;
256 }
257
258 if (uffd_wp)
259 newpte = pte_swp_mkuffd_wp(newpte);
260 else if (uffd_wp_resolve)
261 newpte = pte_swp_clear_uffd_wp(newpte);
262
263 if (!pte_same(oldpte, newpte)) {
264 set_pte_at(vma->vm_mm, addr, pte, newpte);
265 pages++;
266 }
267 } else {
268 /* It must be an none page, or what else?.. */
269 WARN_ON_ONCE(!pte_none(oldpte));
270
271 /*
272 * Nobody plays with any none ptes besides
273 * userfaultfd when applying the protections.
274 */
275 if (likely(!uffd_wp))
276 continue;
277
278 if (userfaultfd_wp_use_markers(vma)) {
279 /*
280 * For file-backed mem, we need to be able to
281 * wr-protect a none pte, because even if the
282 * pte is none, the page/swap cache could
283 * exist. Doing that by install a marker.
284 */
285 set_pte_at(vma->vm_mm, addr, pte,
286 make_pte_marker(PTE_MARKER_UFFD_WP));
287 pages++;
288 }
289 }
290 } while (pte++, addr += PAGE_SIZE, addr != end);
291 arch_leave_lazy_mmu_mode();
292 pte_unmap_unlock(pte - 1, ptl);
293
294 return pages;
295 }
296
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate
In reply to Yan Zhaokernel test robot noticed the following build errors:
Hi Yan,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on kvm/queue]
[also build test ERROR on mst-vhost/linux-next linus/master] [cannot apply to akpm-mm/mm-everything kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Yan-Zhao/mm-mmu_notifier-introduce-a-new-mmu-notifier-flag-MMU_NOTIFIER_RANGE_NUMA/20230810-172955 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue patch link: https://lore.kernel.org/r/20230810090048.26184-1-yan.y.zhao%40intel.com patch subject: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate config: x86_64-buildonly-randconfig-r003-20230809 (https://download.01.org/0day-ci/archive/20230810/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce: (https://download.01.org/0day-ci/archive/20230810/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
mm/huge_memory.c: In function 'change_huge_pmd':
>> mm/huge_memory.c:1895:17: error: implicit declaration of function 'mmu_notifier_numa_protect' [-Werror=implicit-function-declaration]
1895 | mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PMD_SIZE);
| ^~~~~~~~~~~~~~~~~~~~~~~~~cc1: some warnings being treated as errors
vim +/mmu_notifier_numa_protect +1895 mm/huge_memory.c
1862
1863 if (prot_numa) {
1864 struct page *page;
1865 bool toptier;
1866 /*
1867 * Avoid trapping faults against the zero page. The read-only
1868 * data is likely to be read-cached on the local CPU and
1869 * local/remote hits to the zero page are not interesting.
1870 */
1871 if (is_huge_zero_pmd(*pmd))
1872 goto unlock;
1873
1874 if (pmd_protnone(*pmd))
1875 goto unlock;
1876
1877 page = pmd_page(*pmd);
1878
1879 if (PageAnon(page) && PageAnonExclusive(page) &&
1880 page_maybe_dma_pinned(page))
1881 goto unlock;
1882
1883 toptier = node_is_toptier(page_to_nid(page));
1884 /*
1885 * Skip scanning top tier node if normal numa
1886 * balancing is disabled
1887 */
1888 if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
1889 toptier)
1890 goto unlock;
1891
1892 if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING &&
1893 !toptier)
1894 xchg_page_access_time(page, jiffies_to_msecs(jiffies));
> 1895 mmu_notifier_numa_protect(vma->vm_mm, addr, addr + PMD_SIZE);
1896 }
1897 /*
1898 * In case prot_numa, we are under mmap_read_lock(mm). It's critical
1899 * to not clear pmd intermittently to avoid race with MADV_DONTNEED
1900 * which is also under mmap_read_lock(mm):
1901 *
1902 * CPU0: CPU1:
1903 * change_huge_pmd(prot_numa=1)
1904 * pmdp_huge_get_and_clear_notify()
1905 * madvise_dontneed()
1906 * zap_pmd_range()
1907 * pmd_trans_huge(*pmd) == 0 (without ptl)
1908 * // skip the pmd
1909 * set_pmd_at();
1910 * // pmd is re-established
1911 *
1912 * The race makes MADV_DONTNEED miss the huge pmd and don't clear it
1913 * which may break userspace.
1914 *
1915 * pmdp_invalidate_ad() is required to make sure we don't miss
1916 * dirty/young flags set by hardware.
1917 */
1918 oldpmd = pmdp_invalidate_ad(vma, addr, pmd);
1919
1920 entry = pmd_modify(oldpmd, newprot);
1921 if (uffd_wp)
1922 entry = pmd_mkuffd_wp(entry);
1923 else if (uffd_wp_resolve)
1924 /*
1925 * Leave the write bit to be handled by PF interrupt
1926 * handler, then things like COW could be properly
1927 * handled.
1928 */
1929 entry = pmd_clear_uffd_wp(entry);
1930
1931 /* See change_pte_range(). */
1932 if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pmd_write(entry) &&
1933 can_change_pmd_writable(vma, addr, entry))
1934 entry = pmd_mkwrite(entry);
1935
1936 ret = HPAGE_PMD_NR;
1937 set_pmd_at(mm, addr, pmd, entry);
1938
1939 if (huge_pmd_needs_flush(oldpmd, entry))
1940 tlb_flush_pmd_range(tlb, addr, HPAGE_PMD_SIZE);
1941 unlock:
1942 spin_unlock(ptl);
1943 return ret;
1944 }
1945
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate
In reply to Yan ZhaoQuoted 5 lines · Yan Zhao
Call mmu notifier's callback .numa_protect() in
change_pmd_range()when a page is ensured to be protected byPROT_NONEfor NUMA migration purpose.
Consider squashing with the previous patch. It’s better to see the user (caller) with the new functionality.
It would be useful to describe what the expected course of action that numa_protect callback should take.
Re: [RFC PATCH v2 4/5] mm/autonuma: call .numa_protect() when page is protected for NUMA migrate
Quoted 11 lines · Nadav Amit
Quoted 5 lines · Yan Zhao
Call mmu notifier's callback .numa_protect() in
change_pmd_range()when a page is ensured to be protected byPROT_NONEfor NUMA migration purpose.Consider squashing with the previous patch. It’s better to see the user (caller) with the new functionality.
It would be useful to describe what the expected course of action that
numa_protectcallback should take.
Thanks! I'll do in this way when I prepare patches in future :)
RFC PATCH v2 5/5 KVM: Unmap pages only when it's indeed protected for NUMA migration
In reply to Yan ZhaoRegister to .numa_protect() callback in mmu notifier so that KVM can get acurate information about when a page is PROT_NONE protected in primary MMU and unmap it in secondary MMU accordingly.
In KVM's .invalidate_range_start() handler, if the event is to notify that the range may be protected to PROT_NONE for NUMA migration purpose, don't do the unmapping in secondary MMU. Hold on until.numa_protect() comes.
--- virt/kvm/kvm_main.c | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-)
virt/kvm/kvm_main.c +22 -3
@@ -711,6 +711,20 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_change_spte_gfn); } +static void kvm_mmu_notifier_numa_protect (struct mmu_notifier *mn, +struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +struct kvm *kvm =mmu_notifier_to_kvm (mn); + +WARN_ON_ONCE (!READ_ONCE (kvm->mn_active_invalidate_count)); +if (!READ_ONCE (kvm->mmu_invalidate_in_progress)) +return ; + +kvm_handle_hva_range (mn, start, end,__pte (0 ), kvm_unmap_gfn_range); +} +void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start,unsigned long end) {
@@ -744,14 +758,18 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, const struct mmu_notifier_range *range) {struct kvm *kvm =mmu_notifier_to_kvm (mn); +bool is_numa = (range->event ==MMU_NOTIFY_PROTECTION_VMA ) && + (range->flags &MMU_NOTIFIER_RANGE_NUMA );const struct kvm_hva_range hva_range = { .start = range->start, .end = range->end, .pte =__pte (0 ), - .handler = kvm_unmap_gfn_range, + .handler = !is_numa ? kvm_unmap_gfn_range : + (void *)kvm_null_fn, .on_lock = kvm_mmu_invalidate_begin, - .on_unlock = kvm_arch_guest_memory_reclaimed, - .flush_on_ret = true, + .on_unlock = !is_numa ? kvm_arch_guest_memory_reclaimed : + (void *)kvm_null_fn, + .flush_on_ret = !is_numa ? true : false, .may_block =mmu_notifier_range_blockable (range), };
@@ -899,6 +917,7 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { .clear_young = kvm_mmu_notifier_clear_young, .test_young = kvm_mmu_notifier_test_young, .change_pte = kvm_mmu_notifier_change_pte, + .numa_protect = kvm_mmu_notifier_numa_protect, .release = kvm_mmu_notifier_release, };
Signature · 1 line
2.17.1
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 25 lines · Yan Zhao
Register to .numa_protect() callback in mmu notifier so that KVM can get acurate information about when a page is
PROT_NONEprotected in primary MMU and unmap it in secondary MMU accordingly.In KVM's .invalidate_range_start() handler, if the event is to notify that the range may be protected to
PROT_NONEfor NUMA migration purpose, don't do the unmapping in secondary MMU. Hold onuntil.numa_protect()comes.Signed-off-by Yan Zhao <[email protected]>--- virt/kvm/kvm_main.c | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-)virt/kvm/kvm_main.c +3
-0@@ -711,6 +711,20 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,kvm_handle_hva_range (mn, address, address +1 , pte, kvm_change_spte_gfn); } +static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start,
virt/kvm/kvm_main.c static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, · line 726
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+ unsigned long end) +{ +struct kvm *kvm =mmu_notifier_to_kvm (mn); + +WARN_ON_ONCE (!READ_ONCE (kvm->mn_active_invalidate_count)); +if (!READ_ONCE (kvm->mmu_invalidate_in_progress)) +return ; + +kvm_handle_hva_range (mn, start, end,__pte (0 ), kvm_unmap_gfn_range); +}
numa balance will scan wide memory range, and there will be one time ipi notification with kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.
however numa balance notification, pmd table of vm maybe needs not be freed in kvm_unmap_gfn_range.
Regards
Bibo Mao
Quoted 34 lines
virt/kvm/kvm_main.c +9
-3+void kvm_mmu_invalidate_begin (struct kvm *kvm,unsigned long start,unsigned long end) {@@ -744,14 +758,18 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, const struct mmu_notifier_range *range) {struct kvm *kvm =mmu_notifier_to_kvm (mn); +bool is_numa = (range->event ==MMU_NOTIFY_PROTECTION_VMA ) && + (range->flags &MMU_NOTIFIER_RANGE_NUMA );const struct kvm_hva_range hva_range = { .start = range->start, .end = range->end, .pte =__pte (0 ), - .handler = kvm_unmap_gfn_range, + .handler = !is_numa ? kvm_unmap_gfn_range : + (void *)kvm_null_fn, .on_lock = kvm_mmu_invalidate_begin, - .on_unlock = kvm_arch_guest_memory_reclaimed, - .flush_on_ret = true, + .on_unlock = !is_numa ? kvm_arch_guest_memory_reclaimed : + (void *)kvm_null_fn, + .flush_on_ret = !is_numa ? true : false, .may_block =mmu_notifier_range_blockable (range), };@@ -899,6 +917,7 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { .clear_young = kvm_mmu_notifier_clear_young, .test_young = kvm_mmu_notifier_test_young, .change_pte = kvm_mmu_notifier_change_pte, + .numa_protect = kvm_mmu_notifier_numa_protect, .release = kvm_mmu_notifier_release, };
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 6 lines
Quoted 6 lines
virt/kvm/kvm_main.c +6
-0+static void kvm_mmu_notifier_numa_protect (struct mmu_notifier *mn, +struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +struct kvm *kvm =mmu_notifier_to_kvm (mn);
Quoted 8 lines
virt/kvm/kvm_main.c static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, · line 726
Quoted 7 lines
virt/kvm/kvm_main.c +7
-0+ +WARN_ON_ONCE (!READ_ONCE (kvm->mn_active_invalidate_count)); +if (!READ_ONCE (kvm->mmu_invalidate_in_progress)) +return ; + +kvm_handle_hva_range (mn, start, end,__pte (0 ), kvm_unmap_gfn_range); +}numa balance will scan wide memory range, and there will be one time
Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.
Quoted 2 lines
ipi notification with
kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.
Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.
For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.
The reason is below:
Before this series, kvm_unmap_gfn_range() is called for once for a 2M range.
After this series, kvm_unmap_gfn_range() is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.
Though kvm_unmap_gfn_range() is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (in tdp_mmu_iter_cond_resched()) if the pages are mapped in 4K in secondary MMU.
With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.
Below is my testing data on a VM without assigned devices:
The data is an average of 10 times guest boot-up.
data | numa balancing caused | numa balancing caused on average | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() -------------------|------------------------|-------------------------- before this series | 35 | 8625 after this series | 10037 | 4610
For a single guest bootup,
| numa balancing caused | numa balancing caused
best data | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs()
-------------------|------------------------|--------------------------
before this series | 28 | 13
after this series | 406 | 195 | numa balancing caused | numa balancing caused worst data | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() -------------------|------------------------|-------------------------- before this series | 44 | 43920 after this series | 17352 | 8668
Quoted 4 lines
however numa balance notification, pmd table of vm maybe needs not be freed in
kvm_unmap_gfn_range.
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 14 lines · Yan Zhao
Quoted 6 lines
Quoted 6 lines
virt/kvm/kvm_main.c +6
-0+static void kvm_mmu_notifier_numa_protect (struct mmu_notifier *mn, +struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +struct kvm *kvm =mmu_notifier_to_kvm (mn);Quoted 8 lines
virt/kvm/kvm_main.c static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, · line 726
Quoted 7 lines
virt/kvm/kvm_main.c +7
-0+ +WARN_ON_ONCE (!READ_ONCE (kvm->mn_active_invalidate_count)); +if (!READ_ONCE (kvm->mmu_invalidate_in_progress)) +return ; + +kvm_handle_hva_range (mn, start, end,__pte (0 ), kvm_unmap_gfn_range); +}numa balance will scan wide memory range, and there will be one time
Quoted 2 lines
Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.
yes, range is huge page size when changing numa protection during numa scanning.
Quoted 20 lines
Quoted 3 lines
ipi notification with
kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.
For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.The reason is below:
Before this series,
kvm_unmap_gfn_range()is called for once for a 2M range.
After this series,kvm_unmap_gfn_range()is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.
Quoted 3 lines
Though
kvm_unmap_gfn_range()is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (intdp_mmu_iter_cond_resched())
I do not know much about x86, does this happen always or only need reschedule from code? so that there will be many times of tlb IPIs in only once function call about kvm_unmap_gfn_range.
Quoted 6 lines
if the pages are mapped in 4K in secondary MMU.
With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.
No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.
Quoted 9 lines
Below is my testing data on a VM without assigned devices:
The data is an average of 10 times guest boot-up.data | numa balancing caused | numa balancing caused on average | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() -------------------|------------------------|-------------------------- before this series | 35 | 8625 after this series | 10037 | 4610
just be cautious, before the series there are 8625/35 = 246 IPI tlb flush ops during one time kvm_unmap_gfn_range, is that x86 specific or generic?
By the way are primary mmu and secondary mmu both 4K small page size "on average"?
Regards
Bibo Mao
Quoted 21 lines
For a single guest bootup,
| numa balancing caused | numa balancing caused best data | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() -------------------|------------------------|-------------------------- before this series | 28 | 13 after this series | 406 | 195| numa balancing caused | numa balancing caused worst data | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() -------------------|------------------------|-------------------------- before this series | 44 | 43920 after this series | 17352 | 8668Quoted 4 lines
however numa balance notification, pmd table of vm maybe needs not be freed in
kvm_unmap_gfn_range.
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 41 lines · bibo mao
Quoted 14 lines · Yan Zhao
Quoted 6 lines
Quoted 6 lines
virt/kvm/kvm_main.c +6
-0+static void kvm_mmu_notifier_numa_protect (struct mmu_notifier *mn, +struct mm_struct *mm, +unsigned long start, +unsigned long end) +{ +struct kvm *kvm =mmu_notifier_to_kvm (mn);Quoted 8 lines
virt/kvm/kvm_main.c static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, · line 726
Quoted 7 lines
virt/kvm/kvm_main.c +7
-0+ +WARN_ON_ONCE (!READ_ONCE (kvm->mn_active_invalidate_count)); +if (!READ_ONCE (kvm->mmu_invalidate_in_progress)) +return ; + +kvm_handle_hva_range (mn, start, end,__pte (0 ), kvm_unmap_gfn_range); +}numa balance will scan wide memory range, and there will be one time
Quoted 2 lines
Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.
yes, range is huge page size when changing numa protection during numa scanning.
Quoted 20 lines
Quoted 3 lines
ipi notification with
kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.
For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.The reason is below:
Before this series,
kvm_unmap_gfn_range()is called for once for a 2M range.
After this series,kvm_unmap_gfn_range()is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.
Quoted 4 lines
Quoted 3 lines
Though
kvm_unmap_gfn_range()is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (intdp_mmu_iter_cond_resched())I do not know much about x86, does this happen always or only need reschedule
Ah, sorry, I missed platforms other than x86. Maybe there will be a big difference in other platforms.
Quoted 1 line
from code? so that there will be many times of tlb IPIs in only once function
Only when MMU lock is contended. But it's not seldom because of the contention in TDP page fault.
Quoted 10 lines
call about
kvm_unmap_gfn_range.Quoted 6 lines
if the pages are mapped in 4K in secondary MMU.
With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.
The .numa_protect() is called in patch 4 only when PROT_NONE is set to the page.
Quoted 5 lines
Quoted 5 lines
Below is my testing data on a VM without assigned devices:
The data is an average of 10 times guest boot-up.
Quoted 7 lines
Quoted 5 lines
data | numa balancing caused | numa balancing caused on average | #kvm_unmap_gfn_range() | #kvm_flush_remote_tlbs() -------------------|------------------------|-------------------------- before this series | 35 | 8625 after this series | 10037 | 4610just be cautious, before the series there are 8625/35 = 246 IPI tlb flush ops during one time
kvm_unmap_gfn_range, is that x86 specific or generic?
Only on x86. Didn't test on other platforms.
Quoted 2 lines
By the way are primary mmu and secondary mmu both 4K small page size "on average"?
No. 4K and 2M combined in both.
Sean Christopherson
maintainerRe: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 28 lines · Yan Zhao
Quoted 27 lines · bibo mao
Quoted 14 lines · Yan Zhao
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6
-0+static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + struct kvm *kvm = mmu_notifier_to_kvm(mn);Quoted 8 lines
Review
Quoted 7 lines
(quoted diff) +7
-0+ + WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count)); + if (!READ_ONCE(kvm->mmu_invalidate_in_progress)) + return; + + kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range); +}numa balance will scan wide memory range, and there will be one time
Quoted 2 lines
Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.
yes, range is huge page size when changing numa protection during numa scanning.
Quoted 7 lines
Quoted 3 lines
ipi notification with
kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.The reason is below:
Before this series,
kvm_unmap_gfn_range()is called for once for a 2M range.
No, it's potentially called once per 1GiB range. change_pmd_range() invokes the mmu_notifier with addr+end, where "end" is the end of the range covered by the PUD, not the the end of the current PMD. So the worst case scenario would be a 256k increase. Of course, if you have to migrate an entire 1GiB chunk of memory then you likely have bigger problems, but still.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
After this series,
kvm_unmap_gfn_range()is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.Though
kvm_unmap_gfn_range()is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (intdp_mmu_iter_cond_resched())
Quoted 7 lines
Quoted 1 line
I do not know much about x86, does this happen always or only need reschedule
Ah, sorry, I missed platforms other than x86. Maybe there will be a big difference in other platforms.
Quoted 1 line
from code? so that there will be many times of tlb IPIs in only once function
Only when MMU lock is contended. But it's not seldom because of the contention in TDP page fault.
No? I don't see how mmu_lock contention would affect the number of calls to kvm_flush_remote_tlbs(). If vCPUs are running and not generating faults, i.e. not trying to access the range in question, then ever zap will generate a remote TLB flush and thus send IPIs to all running vCPUs.
Quoted 4 lines
Quoted 4 lines
call about
kvm_unmap_gfn_range.Quoted 2 lines
if the pages are mapped in 4K in secondary MMU.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.
The .numa_protect() is called in patch 4 only when
PROT_NONEis set to the page.
I'm strongly opposed to adding MMU_NOTIFIER_RANGE_NUMA. It's too much of a one-off, and losing the batching of invalidations makes me nervous. As Bibo points out, the behavior will vary based on the workload, VM configuration, etc.
There's also a *very* subtle change, in that the notification will be sent while holding the PMD/PTE lock. Taking KVM's mmu_lock under that is *probably* ok, but I'm not exactly 100% confident on that. And the only reason there isn't a more obvious bug is because kvm_handle_hva_range() sets may_block to false, e.g. KVM won't yield if there's mmu_lock contention.
However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.
Rather than muck with the notification types and add a one-off for NUMA, just defer the notification until a present PMD/PTE is actually going to be modified. It's not the prettiest, but other than the locking, I don't think has any chance of regressing other workloads/configurations.
Note, I'm assuming secondary MMUs aren't allowed to map swap entries...
Compile tested only.
From: Sean Christopherson <[email protected]>
Date: Fri, 11 Aug 2023 10:03:36 -0700
Subject: [PATCH] tmp
--- include/linux/huge_mm.h | 4 +++- include/linux/mm.h | 3 +++ mm/huge_memory.c | 5 ++++- mm/mprotect.c | 47 +++++++++++++++++++++++++++++------------ 4 files changed, 44 insertions(+), 15 deletions(-)
include/linux/huge_mm.h +3 -1
@@ -7,6 +7,8 @@#include <linux/fs.h> /* only for vma_is_dax() */ +struct mmu_notifier_range ; +vm_fault_t do_huge_pmd_anonymous_page (struct vm_fault *vmf); int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
@@ -38,7 +40,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd);int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot, -unsigned long cp_flags); +unsigned long cp_flags,struct mmu_notifier_range *range);vm_fault_t vmf_insert_pfn_pmd (struct vm_fault *vmf,pfn_t pfn,bool write);vm_fault_t vmf_insert_pfn_pud (struct vm_fault *vmf,pfn_t pfn,bool write);
include/linux/mm.h +3 -0
@@ -2478,6 +2478,9 @@ static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vmareturn !!(vma->vm_flags &VM_WRITE ); } + +void change_pmd_range_notify_secondary_mmus (unsigned long addr, +struct mmu_notifier_range *range);bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr,pte_t pte); extern long change_protection(struct mmu_gather *tlb,
mm/huge_memory.c +4 -1
@@ -1808,7 +1808,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,*/ int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot, -unsigned long cp_flags) +unsigned long cp_flags,struct mmu_notifier_range *range) {struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl;
@@ -1893,6 +1893,9 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, !toptier)xchg_page_access_time (page,jiffies_to_msecs (jiffies)); } + +change_pmd_range_notify_secondary_mmus (addr, range); +/* * In case prot_numa, we are under mmap_read_lock(mm). It's critical * to not clear pmd intermittently to avoid race with MADV_DONTNEED
mm/mprotect.c +34 -13
95 lines not on this page · raw message
base-commit: 1f40f634009556c119974cafce4c7b2f9b8c58ad
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 3 lines
However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.
No, this is not legal.
mm/mprotect.c pgtable_populate_needed(struct vm_area_struct *vma, unsigned long cp_flags) · line 372
Quoted 10 lines
mm/mprotect.c +10
-0+void change_pmd_range_notify_secondary_mmus (unsigned long addr, +struct mmu_notifier_range *range) +{ +if (range->start) +return ; + +VM_WARN_ON (addr >= range->end); + range->start = addr; +mmu_notifier_invalidate_range_start_nonblock (range); +}
The 'nonblock' entry point is advisory, if it fails the caller must not change the mm.
Jason
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
In reply to Sean ChristophersonQuoted 35 lines · Sean Christopherson
Quoted 34 lines · Yan Zhao
Quoted 33 lines · bibo mao
Quoted 14 lines · Yan Zhao
Quoted 6 lines
Quoted 6 lines
(quoted diff) +6
-0+static void kvm_mmu_notifier_numa_protect(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + struct kvm *kvm = mmu_notifier_to_kvm(mn);Quoted 8 lines
Review
Quoted 7 lines
(quoted diff) +7
-0+ + WARN_ON_ONCE(!READ_ONCE(kvm->mn_active_invalidate_count)); + if (!READ_ONCE(kvm->mmu_invalidate_in_progress)) + return; + + kvm_handle_hva_range(mn, start, end, __pte(0), kvm_unmap_gfn_range); +}numa balance will scan wide memory range, and there will be one time
Quoted 2 lines
Though scanning memory range is wide, .invalidate_range_start() is sent for each 2M range.
yes, range is huge page size when changing numa protection during numa scanning.
Quoted 13 lines
Quoted 3 lines
ipi notification with
kvm_flush_remote_tlbs. With page level notification, it may bring out lots of flush remote tlb ipi notification.Hmm, for VMs with assigned devices, apparently, the flush remote tlb IPIs will be reduced to 0 with this series.
For VMs without assigned devices or mdev devices, I was previously also worried about that there might be more IPIs.
But with current test data, there's no more remote tlb IPIs on average.The reason is below:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Before this series,
kvm_unmap_gfn_range()is called for once for a 2M range.No, it's potentially called once per 1GiB range.
change_pmd_range()invokes themmu_notifierwith addr+end, where "end" is the end of the range covered by the PUD, not the the end of the current PMD. So the worst case scenario would be a 256k increase. Of course, if you have to migrate an entire 1GiB chunk of memory then you likely have bigger problems, but still.
Yes, thanks for pointing it out.
I realized it after re-reading the code and re-checking the log message. This wider range also explained the collected worst data:
44 kvm_unmap_gfn_range() caused 43920 kvm_flush_remote_tlbs()requests. i.e. 998 remote tlb flush requests per kvm_unmap_gfn_range().
Quoted 13 lines
Quoted 13 lines
Quoted 10 lines
Quoted 9 lines
After this series,
kvm_unmap_gfn_range()is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.Though
kvm_unmap_gfn_range()is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (intdp_mmu_iter_cond_resched())I do not know much about x86, does this happen always or only need reschedule
Ah, sorry, I missed platforms other than x86. Maybe there will be a big difference in other platforms.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
from code? so that there will be many times of tlb IPIs in only once function
Only when MMU lock is contended. But it's not seldom because of the contention in TDP page fault.
No? I don't see how
mmu_lockcontention would affect the number of calls tokvm_flush_remote_tlbs(). If vCPUs are running and not generating faults, i.e. not trying to access the range in question, then ever zap will generate a remote TLB flush and thus send IPIs to all running vCPUs.
In tdp_mmu_zap_leafs() for kvm_unmap_gfn_range(), the flow is like this:
1. Check necessity of mmu_lock reschedule
1.a -- if yes,
1.a.1 do kvm_flush_remote_tlbs() if flush is true. 1.a.2 reschedule of mmu_lock
1.a.3 goto 2.
1.b -- if no, goto 2
2. Zap present leaf entry, and set flush to be true
3. Get next gfn and go to to 1
With a wide range to .invalidate_range_start()/end(), it's easy to find rwlock_needbreak(&kvm->mmu_lock) to be true (goes to 1.a and 1.a.1) And in tdp_mmu_zap_leafs(), before rescheduling of mmu_lock, tlb flush request is performed. (1.a.1)
Take a real case for example,
NUMA balancing requests KVM to zap GFN range from 0xb4000 to 0xb9800. Then when KVM zaps GFN 0xb807b, it will finds mmu_lock needs break because vCPU is faulting GFN 0xb804c.
And vCPUs fill constantly retry faulting 0xb804c for 3298 times until .invalidate_range_end().
Then for this zap of GFN range from 0xb4000 - 0xb9800,
vCPUs retry fault of
GFN 0xb804c for 3298 times,
GFN 0xb8074 for 3161 times,
GFN 0xb81ce for 15190 times,
and the accesses of the 3 GFNs cause 3209 times of kvm_flush_remote_tlbs() for one kvm_unmap_gfn_range() because flush requests are not batched. (this range is mapped in both 2M and 4K in secondary MMU).
Without the contending of mmu_lock, tdp_mmu_zap_leafs() just combines all flush requests and leaves 1 kvm_flush_remote_tlbs() to be called after it returns.
In my test scenario, which is VM boot-up, this kind of contention is frequent.
Here's the 10 times data for a VM boot-up collected previously.
| count of | count of |
| kvm_unmap_gfn_range | kvm_flush_remote_tlbs() |
-------|---------------------|-------------------------|
1 | 38 | 14 |
2 | 28 | 5191 |
3 | 36 | 13398 |
4 | 44 | 43920 |
5 | 28 | 14 |
6 | 36 | 10803 |
7 | 38 | 892 |
8 | 32 | 5905 |
9 | 32 | 13 |
10 | 38 | 6096 |
-------|---------------------|-------------------------|
average| 35 | 8625 |I wonder if we could loose the frequency to check for rescheduling in tdp_mmu_iter_cond_resched() if the zap range is wide, e.g.
if (iter->next_last_level_gfn ==
iter->yielded_gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M)) return false;
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
call about
kvm_unmap_gfn_range.Quoted 2 lines
if the pages are mapped in 4K in secondary MMU.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.
The .numa_protect() is called in patch 4 only when
PROT_NONEis set to the page.
Quoted 7 lines
I'm strongly opposed to adding
MMU_NOTIFIER_RANGE_NUMA. It's too much of a one-off, and losing the batching of invalidations makes me nervous. As Bibo points out, the behavior will vary based on the workload, VM configuration, etc.There's also a *very* subtle change, in that the notification will be sent while holding the PMD/PTE lock. Taking KVM's
mmu_lockunder that is *probably* ok, but I'm not exactly 100% confident on that. And the only reason there isn't a more
MMU lock is a rwlock, which is a variant of spinlock.
So, I think taking it within PMD/PTE lock is ok.
Actually we have already done that during the .change_pte() notification, where
kvm_mmu_notifier_change_pte() takes KVM mmu_lock for write,
while PTE lock is held while sending set_pte_at_notify() --> .change_pte() handlers
Quoted 2 lines
obvious bug is because
kvm_handle_hva_range()setsmay_blockto false, e.g. KVM won't yield if there'smmu_lockcontention.
Yes, KVM won't yield and reschedule of KVM mmu_lock, so it's fine.
Quoted 4 lines
However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.
Quoted 8 lines
Rather than muck with the notification types and add a one-off for NUMA, just defer the notification until a present PMD/PTE is actually going to be modified. It's not the prettiest, but other than the locking, I don't think has any chance of regressing other workloads/configurations.
Note, I'm assuming secondary MMUs aren't allowed to map swap entries...
Compile tested only.
I don't find a matching end to eachmmu_notifier_invalidate_range_start_nonblock().
Quoted 102 lines
From: Sean Christopherson <[email protected]>
Date: Fri, 11 Aug 2023 10:03:36 -0700
Subject: [PATCH] tmpSigned-off-by Sean Christopherson <[email protected]>--- include/linux/huge_mm.h | 4 +++- include/linux/mm.h | 3 +++ mm/huge_memory.c | 5 ++++- mm/mprotect.c | 47 +++++++++++++++++++++++++++++------------ 4 files changed, 44 insertions(+), 15 deletions(-)include/linux/huge_mm.h +3
-1@@ -7,6 +7,8 @@#include <linux/fs.h> /* only for vma_is_dax() */ +struct mmu_notifier_range ; +vm_fault_t do_huge_pmd_anonymous_page (struct vm_fault *vmf); int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,@@ -38,7 +40,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd);int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot, -unsigned long cp_flags); +unsigned long cp_flags,struct mmu_notifier_range *range);vm_fault_t vmf_insert_pfn_pmd (struct vm_fault *vmf,pfn_t pfn,bool write);vm_fault_t vmf_insert_pfn_pud (struct vm_fault *vmf,pfn_t pfn,bool write);include/linux/mm.h +3
-0@@ -2478,6 +2478,9 @@ static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vmareturn !!(vma->vm_flags &VM_WRITE ); } + +void change_pmd_range_notify_secondary_mmus (unsigned long addr, +struct mmu_notifier_range *range);bool can_change_pte_writable (struct vm_area_struct *vma,unsigned long addr,pte_t pte); extern long change_protection(struct mmu_gather *tlb,mm/huge_memory.c +4
-1@@ -1808,7 +1808,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,*/ int change_huge_pmd (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr,pgprot_t newprot, -unsigned long cp_flags) +unsigned long cp_flags,struct mmu_notifier_range *range) {struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl;@@ -1893,6 +1893,9 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, !toptier)xchg_page_access_time (page,jiffies_to_msecs (jiffies)); } + +change_pmd_range_notify_secondary_mmus (addr, range); +/* * In case prot_numa, we are under mmap_read_lock(mm). It's critical * to not clear pmd intermittently to avoid race with MADV_DONTNEED mm/mprotect.c +6
-1@@ -82,7 +82,8 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,static long change_pte_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pmd_t *pmd,unsigned long addr, -unsigned long end,pgprot_t newprot,unsigned long cp_flags) +unsigned long end,pgprot_t newprot,unsigned long cp_flags, +struct mmu_notifier_range *range) {pte_t *pte, oldpte;spinlock_t *ptl;@@ -164,8 +165,12 @@ static long change_pte_range(struct mmu_gather *tlb, !toptier)xchg_page_access_time (page,jiffies_to_msecs (jiffies)); + + } +change_pmd_range_notify_secondary_mmus (addr, range); + oldpte =ptep_modify_prot_start (vma, addr, pte); ptent =pte_modify (oldpte, newprot);@@ -355,6 +360,17 @@ pgtable_populate_needed(struct vm_area_struct *vma, unsigned long cp_flags) err; \ })
mm/mprotect.c pgtable_populate_needed(struct vm_area_struct *vma, unsigned long cp_flags) · line 371
Quoted 10 lines
mm/mprotect.c +9
-0+void change_pmd_range_notify_secondary_mmus (unsigned long addr, +struct mmu_notifier_range *range) +{ +if (range->start) +return ; + +VM_WARN_ON (addr >= range->end); + range->start = addr; +mmu_notifier_invalidate_range_start_nonblock (range);
This will cause range from addr to end to be invalidated, which may include pinned ranges.
Quoted 72 lines
mm/mprotect.c +19
-12+} +static inline long change_pmd_range (struct mmu_gather *tlb,struct vm_area_struct *vma,pud_t *pud,unsigned long addr,unsigned long end,pgprot_t newprot,unsigned long cp_flags)@@ -365,7 +381,14 @@ static inline long change_pmd_range(struct mmu_gather *tlb,unsigned long nr_huge_updates =0 ;struct mmu_notifier_range range; - range.start =0 ; +/* +* Defer invalidation of secondary MMUs until a PMD/PTE change is +* imminent, e.g. NUMA balancing in particular can "fail" for certain +* types of mappings. Initialize range.start to '0' and use it to +* track whether or not the invalidation notification has been set. +*/ +mmu_notifier_range_init (&range,MMU_NOTIFY_PROTECTION_VMA ,0 , + vma->vm_mm,0 , end); pmd =pmd_offset (pud, addr); do {@@ -383,18 +406,16 @@ static inline long change_pmd_range(struct mmu_gather *tlb,if (pmd_none (*pmd)) goto next; -/* invoke the mmu notifier if the pmd is populated */ -if (!range.start) { -mmu_notifier_range_init (&range, -MMU_NOTIFY_PROTECTION_VMA ,0 , - vma->vm_mm, addr, end); -mmu_notifier_invalidate_range_start (&range); - } - _pmd =pmdp_get_lockless (pmd);if (is_swap_pmd (_pmd) ||pmd_trans_huge (_pmd) ||pmd_devmap (_pmd)) {if ((next - addr !=HPAGE_PMD_SIZE ) ||pgtable_split_needed (vma, cp_flags)) { +/* +* FIXME: __split_huge_pmd() performs its own +* mmu_notifier invalidation prior to clearing +* the PMD, ideally all invalidations for the +* range would be batched. +*/ __split_huge_pmd (vma, pmd, addr, false,NULL );/* * For file-backed, the pmd could have been @@ -407,8 +428,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb,break ; } }else { - ret =change_huge_pmd (tlb, vma, pmd, - addr, newprot, cp_flags); + ret =change_huge_pmd (tlb, vma, pmd, addr, + newprot, cp_flags, &range);if (ret) {if (ret ==HPAGE_PMD_NR ) { pages +=HPAGE_PMD_NR ;@@ -423,7 +444,7 @@ static inline long change_pmd_range(struct mmu_gather *tlb, } ret =change_pte_range (tlb, vma, pmd, addr, next, newprot, - cp_flags); + cp_flags, &range);if (ret <0 ) goto again; pages += ret;base-commit:
1f40f634009556c119974cafce4c7b2f9b8c58ad
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 6 lines · Yan Zhao
I wonder if we could loose the frequency to check for rescheduling in
tdp_mmu_iter_cond_resched()if the zap range is wide, e.g.if (
iter->next_last_level_gfn==iter->yielded_gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M)) return false;
Correct:
@@ -712,7 +713,8 @@ static inline bool __must_check tdp_mmu_iter_cond_resched(struct kvm *kvm,
WARN_ON(iter->yielded);
/* Ensure forward progress has been made before yielding. */
- if (iter->next_last_level_gfn == iter->yielded_gfn)
+ if (iter->next_last_level_gfn >= iter->yielded_gfn &&
+ iter->next_last_level_gfn < iter->yielded_gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M))
return false;
if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) {This can reduce kvm_flush_remote_tlbs() a lot in one kvm_unmap_gfn_range() in KVM x86 TDP MMU.
Sean Christopherson
maintainerRe: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
In reply to Yan ZhaoQuoted 17 lines · Yan Zhao
Quoted 16 lines · Sean Christopherson
Quoted 15 lines
Quoted 9 lines
Quoted 8 lines
After this series,
kvm_unmap_gfn_range()is called for once if the 2M is mapped to a huge page in primary MMU, and called for at most 512 times if mapped to 4K pages in primary MMU.Though
kvm_unmap_gfn_range()is only called once before this series, as the range is blockable, when there're contentions, remote tlb IPIs can be sent page by page in 4K granularity (intdp_mmu_iter_cond_resched())I do not know much about x86, does this happen always or only need reschedule
Ah, sorry, I missed platforms other than x86. Maybe there will be a big difference in other platforms.
Quoted 1 line
from code? so that there will be many times of tlb IPIs in only once function
Only when MMU lock is contended. But it's not seldom because of the contention in TDP page fault.
Quoted 7 lines
Quoted 4 lines
No? I don't see how
mmu_lockcontention would affect the number of calls tokvm_flush_remote_tlbs(). If vCPUs are running and not generating faults, i.e. not trying to access the range in question, then ever zap will generate a remote TLB flush and thus send IPIs to all running vCPUs.In
tdp_mmu_zap_leafs()forkvm_unmap_gfn_range(), the flow is like this:1. Check necessity of
mmu_lockreschedule
Ah, you're running a preemptible kernel.
Quoted 29 lines
1.a -- if yes,
1.a.1 dokvm_flush_remote_tlbs()if flush is true. 1.a.2 reschedule ofmmu_lock
1.a.3 goto 2.
1.b -- if no, goto 2
2. Zap present leaf entry, and set flush to be true
3. Get next gfn and go to to 1With a wide range to .invalidate_range_start()/end(), it's easy to find
rwlock_needbreak(&kvm->mmu_lock)to be true (goes to 1.a and 1.a.1) And intdp_mmu_zap_leafs(), before rescheduling ofmmu_lock, tlb flush request is performed. (1.a.1)Take a real case for example,
NUMA balancing requests KVM to zap GFN range from0xb4000to0xb9800. Then when KVM zaps GFN0xb807b, it will findsmmu_lockneeds break because vCPU is faulting GFN0xb804c.
And vCPUs fill constantly retry faulting0xb804cfor 3298 times until .invalidate_range_end().
Then for this zap of GFN range from0xb4000-0xb9800,
vCPUs retry fault of
GFN0xb804cfor 3298 times,
GFN0xb8074for 3161 times,
GFN0xb81cefor 15190 times,
and the accesses of the 3 GFNs cause 3209 times ofkvm_flush_remote_tlbs()for onekvm_unmap_gfn_range()because flush requests are not batched. (this range is mapped in both 2M and 4K in secondary MMU).
Quoted 7 lines
Without the contending of
mmu_lock,tdp_mmu_zap_leafs()just combines all flush requests and leaves 1kvm_flush_remote_tlbs()to be called after it returns.In my test scenario, which is VM boot-up, this kind of contention is frequent.
Hmm, I think your test scenario is a bit of an outlier. Running a preemptible kernel on a multi-socket system (presumably your system is multi-socket if you have NUMA) is a bit odd.
Not that that invalidates the test result, but I would be quite surprised if there are production use cases for NUMA+KVM+preemptible. Though maybe I'm underestimating how much KVM is used on workstations?
Quoted 18 lines
Here's the 10 times data for a VM boot-up collected previously.
| count of | count of | | kvm_unmap_gfn_range | kvm_flush_remote_tlbs() | -------|---------------------|-------------------------| 1 | 38 | 14 | 2 | 28 | 5191 | 3 | 36 | 13398 | 4 | 44 | 43920 | 5 | 28 | 14 | 6 | 36 | 10803 | 7 | 38 | 892 | 8 | 32 | 5905 | 9 | 32 | 13 | 10 | 38 | 6096 | -------|---------------------|-------------------------| average| 35 | 8625 |
Quoted 6 lines
I wonder if we could loose the frequency to check for rescheduling in
tdp_mmu_iter_cond_resched()if the zap range is wide, e.g.if (
iter->next_last_level_gfn==iter->yielded_gfn + KVM_PAGES_PER_HPAGE(PG_LEVEL_2M)) return false;
Hrm, no. We'd want to repeat that logic for the shadow MMU, and it could regress other scenarios, e.g. if a vCPU is trying to fault-in an address that isn't part of the invalidation, then yielding asap is desirable.
Unless I'm missing something, the easiest solution is to check for an invalidation *before* acquiring mmu_lock. The check would be prone to false negatives and false positives, so KVM would need to recheck after acquiring mmu_lock, but the check itself is super cheap relative to the overall cost of the page fault.
Compile tested patch at the bottom. I'll test and post formally (probably with similar treatment for other architectures).
Quoted 20 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
call about
kvm_unmap_gfn_range.Quoted 2 lines
if the pages are mapped in 4K in secondary MMU.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
With this series, on the other hand, .numa_protect() sets range to be unblockable. So there could be less remote tlb IPIs when a 2M range is mapped into small PTEs in secondary MMU.
Besides, .numa_protect() is not sent for all pages in a given 2M range.No, .numa_protect() is not sent for all pages. It depends on the workload, whether the page is accessed for different cpu threads cross-nodes.
The .numa_protect() is called in patch 4 only when
PROT_NONEis set to the page.Quoted 7 lines
I'm strongly opposed to adding
MMU_NOTIFIER_RANGE_NUMA. It's too much of a one-off, and losing the batching of invalidations makes me nervous. As Bibo points out, the behavior will vary based on the workload, VM configuration, etc.There's also a *very* subtle change, in that the notification will be sent while holding the PMD/PTE lock. Taking KVM's
mmu_lockunder that is *probably* ok, but I'm not exactly 100% confident on that. And the only reason there isn't a more
Quoted 6 lines
MMU lock is a rwlock, which is a variant of spinlock.
So, I think taking it within PMD/PTE lock is ok.
Actually we have already done that during the .change_pte() notification, where
kvm_mmu_notifier_change_pte()takes KVMmmu_lockfor write,
while PTE lock is held while sendingset_pte_at_notify()--> .change_pte() handlers
.change_pte() gets away with running under PMD/PTE lock because (a) it's not allowed to fail and (b) KVM is the only secondary MMU that hooks .change_pte() and KVM doesn't use a sleepable lock.
As Jason pointed out, mmu_notifier_invalidate_range_start_nonblock() can fail because some secondary MMUs use mutexes or r/w semaphores to handle mmu_notifier events. For NUMA balancing, canceling the protection change might be ok, but for everything else, failure is not an option. So unfortunately, my idea won't work as-is.
We might get away with deferring just the change_prot_numa() case, but I don't think that's worth the mess/complexity.
I would much rather tell userspace to disable migrate-on-fault for KVM guest mappings (mbind() should work?) for these types of setups, or to disable NUMA balancing entirely. If the user really cares about performance of their VM(s), vCPUs should be affined to a single NUMA node (or core, or pinned 1:1), and if the VM spans multiple nodes, a virtual NUMA topology should be given to the guest. At that point, NUMA balancing is likely to do more harm than good.
Quoted 13 lines
Quoted 2 lines
obvious bug is because
kvm_handle_hva_range()setsmay_blockto false, e.g. KVM won't yield if there'smmu_lockcontention.Yes, KVM won't yield and reschedule of KVM
mmu_lock, so it's fine.Quoted 9 lines
However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.
Rather than muck with the notification types and add a one-off for NUMA, just defer the notification until a present PMD/PTE is actually going to be modified. It's not the prettiest, but other than the locking, I don't think has any chance of regressing other workloads/configurations.
Quoted 6 lines
Quoted 3 lines
Note, I'm assuming secondary MMUs aren't allowed to map swap entries...
Compile tested only.
I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().
It pairs with existing call to mmu_notifier_invalidate_range_end() in change_pmd_range():
if (range.start) mmu_notifier_invalidate_range_end(&range);
--
From: Sean Christopherson <[email protected]>
Date: Mon, 14 Aug 2023 08:59:12 -0700
Subject: [PATCH] KVM: x86/mmu: Retry fault before acquiring mmu_lock if
mapping is changing
Retry page faults without acquiring mmu_lock if the resolve hva is covered by an active invalidation. Contending for mmu_lock is especially problematic on preemptible kernels as the invalidation task will yield the lock (see rwlock_needbreak()), delay the in-progress invalidation, and ultimately increase the latency of resolving the page fault. And in the worst case scenario, yielding will be accompanied by a remote TLB flush, e.g. if the invalidation covers a large range of memory and vCPUs are accessing addresses that were already zapped.
Alternatively, the yielding issue could be mitigated by teaching KVM's MMU iterators to perform more work before yielding, but that wouldn't solve the lock contention and would negatively affect scenarios where a vCPU is trying to fault in an address that is NOT covered by the in-progress invalidation.
--- arch/x86/kvm/mmu/mmu.c | 3 +++ include/linux/kvm_host.h | 8 +++++++- 2 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,if (unlikely (!fault->slot))return kvm_handle_noslot_fault (vcpu, fault, access); +if (mmu_invalidate_retry_hva (vcpu->kvm, fault->mmu_seq, fault->hva)) +return RET_PF_RETRY ; +return RET_PF_CONTINUE ; }
include/linux/kvm_host.h +7 -1
@@ -1960,7 +1960,6 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,unsigned long mmu_seq, unsigned long hva) { - lockdep_assert_held(&kvm->mmu_lock);/* * If mmu_invalidate_in_progress is non-zero, then the range maintained * by kvm_mmu_notifier_invalidate_range_start contains all addresses
@@ -1971,6 +1970,13 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm, hva >= kvm->mmu_invalidate_range_start && hva < kvm->mmu_invalidate_range_end)return 1 ; + +/* +* Note the lack of a memory barrier! The caller *must* hold mmu_lock +* to avoid false negatives and/or false positives (less concerning). +* Holding mmu_lock is not mandatory though, e.g. to allow pre-checking +* for an in-progress invalidation to avoiding contending mmu_lock. +*/ if (kvm->mmu_invalidate_seq != mmu_seq)return 1 ;return 0 ;
base-commit: 5bc7f472423f95a3f5c73b0b56c47e57d8833efc
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 11 lines · Sean Christopherson
Quoted 11 lines
Quoted 7 lines
I'm strongly opposed to adding
MMU_NOTIFIER_RANGE_NUMA. It's too much of a one-off, and losing the batching of invalidations makes me nervous. As Bibo points out, the behavior will vary based on the workload, VM configuration, etc.There's also a *very* subtle change, in that the notification will be sent while holding the PMD/PTE lock. Taking KVM's
mmu_lockunder that is *probably* ok, but I'm not exactly 100% confident on that. And the only reason there isn't a moreMMU lock is a rwlock, which is a variant of spinlock.
So, I think taking it within PMD/PTE lock is ok.
Actually we have already done that during the .change_pte() notification, where
Quoted 6 lines
Quoted 2 lines
kvm_mmu_notifier_change_pte()takes KVMmmu_lockfor write,
while PTE lock is held while sendingset_pte_at_notify()--> .change_pte() handlers.change_pte() gets away with running under PMD/PTE lock because (a) it's not allowed to fail and (b) KVM is the only secondary MMU that hooks .change_pte() and KVM doesn't use a sleepable lock.
.numa_protect() in patch 4 is also sent when it's not allowed to fail and there's no sleepable lock in KVM's handler :)
Quoted 8 lines
As Jason pointed out,
mmu_notifier_invalidate_range_start_nonblock()can fail because some secondary MMUs use mutexes or r/w semaphores to handlemmu_notifierevents. For NUMA balancing, canceling the protection change might be ok, but for everything else, failure is not an option. So unfortunately, my idea won't work as-is.We might get away with deferring just the
change_prot_numa()case, but I don't think that's worth the mess/complexity.
Yes, I also think deferring mmu_notifier_invalidate_range_start() is not working. One possible approach is to send successful .numa_protect() in batch. But I admit it will introduce complexity.
Quoted 25 lines
I would much rather tell userspace to disable migrate-on-fault for KVM guest mappings (
mbind()should work?) for these types of setups, or to disable NUMA balancing entirely. If the user really cares about performance of their VM(s), vCPUs should be affined to a single NUMA node (or core, or pinned 1:1), and if the VM spans multiple nodes, a virtual NUMA topology should be given to the guest. At that point, NUMA balancing is likely to do more harm than good.Quoted 17 lines
Quoted 2 lines
obvious bug is because
kvm_handle_hva_range()setsmay_blockto false, e.g. KVM won't yield if there'smmu_lockcontention.Yes, KVM won't yield and reschedule of KVM
mmu_lock, so it's fine.Quoted 12 lines
However, *if* it's ok to invoke MMU notifiers while holding PMD/PTE locks, then I think we can achieve what you want without losing batching, and without changing secondary MMUs.
Rather than muck with the notification types and add a one-off for NUMA, just defer the notification until a present PMD/PTE is actually going to be modified. It's not the prettiest, but other than the locking, I don't think has any chance of regressing other workloads/configurations.
Note, I'm assuming secondary MMUs aren't allowed to map swap entries...
Compile tested only.
Quoted 7 lines
Quoted 2 lines
I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().It pairs with existing call to
mmu_notifier_invalidate_range_end()inchange_pmd_range():if (range.start) mmu_notifier_invalidate_range_end(&range);
No, It doesn't work for mmu_notifier_invalidate_range_start() sent in change_pte_range(), if we only want the range to include pages successfully set to PROT_NONE.
Quoted 34 lines
--
From: Sean Christopherson <[email protected]>
Date: Mon, 14 Aug 2023 08:59:12 -0700
Subject: [PATCH] KVM: x86/mmu: Retry fault before acquiringmmu_lockif
mapping is changingRetry page faults without acquiring
mmu_lockif the resolve hva is covered by an active invalidation. Contending formmu_lockis especially problematic on preemptible kernels as the invalidation task will yield the lock (seerwlock_needbreak()), delay the in-progress invalidation, and ultimately increase the latency of resolving the page fault. And in the worst case scenario, yielding will be accompanied by a remote TLB flush, e.g. if the invalidation covers a large range of memory and vCPUs are accessing addresses that were already zapped.Alternatively, the yielding issue could be mitigated by teaching KVM's MMU iterators to perform more work before yielding, but that wouldn't solve the lock contention and would negatively affect scenarios where a vCPU is trying to fault in an address that is NOT covered by the in-progress invalidation.
Reported-by Yan Zhao <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 3 +++ include/linux/kvm_host.h | 8 +++++++- 2 files changed, 10 insertions(+), 1 deletion(-)arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · line 4350
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +3
-0@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,if (unlikely (!fault->slot))return kvm_handle_noslot_fault (vcpu, fault, access); +if (mmu_invalidate_retry_hva (vcpu->kvm, fault->mmu_seq, fault->hva)) +return RET_PF_RETRY ; +
This can effectively reduce the remote flush IPIs a lot!
One Nit is that, maybe rmb() or READ_ONCE() is required for kvm->mmu_invalidate_range_start and kvm->mmu_invalidate_range_end.
Otherwise, I'm somewhat worried about constant false positive and retry.
Quoted 33 lines
arch/x86/kvm/mmu/mmu.c +0
-0return RET_PF_CONTINUE ; }include/linux/kvm_host.h +7
-1@@ -1960,7 +1960,6 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm,unsigned long mmu_seq, unsigned long hva) { - lockdep_assert_held(&kvm->mmu_lock);/* * If mmu_invalidate_in_progress is non-zero, then the range maintained * by kvm_mmu_notifier_invalidate_range_start contains all addresses @@ -1971,6 +1970,13 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm, hva >= kvm->mmu_invalidate_range_start && hva < kvm->mmu_invalidate_range_end)return 1 ; + +/* +* Note the lack of a memory barrier! The caller *must* hold mmu_lock +* to avoid false negatives and/or false positives (less concerning). +* Holding mmu_lock is not mandatory though, e.g. to allow pre-checking +* for an in-progress invalidation to avoiding contending mmu_lock. +*/ if (kvm->mmu_invalidate_seq != mmu_seq)return 1 ;return 0 ;base-commit:
5bc7f472423f95a3f5c73b0b56c47e57d8833efc
Sean Christopherson
maintainerRe: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines
Note, I'm assuming secondary MMUs aren't allowed to map swap entries...
Compile tested only.
I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().
Quoted 6 lines
Quoted 4 lines
It pairs with existing call to
mmu_notifier_invalidate_range_end()inchange_pmd_range():if (range.start) mmu_notifier_invalidate_range_end(&range);No, It doesn't work for
mmu_notifier_invalidate_range_start()sent inchange_pte_range(), if we only want the range to include pages successfully set toPROT_NONE.
Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the start() call still used the original @end.
The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.
Quoted 8 lines
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +0
-0@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,if (unlikely (!fault->slot))return kvm_handle_noslot_fault (vcpu, fault, access);
Quoted 7 lines
arch/x86/kvm/mmu/mmu.c static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · line 4350
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+if (mmu_invalidate_retry_hva (vcpu->kvm, fault->mmu_seq, fault->hva)) +return RET_PF_RETRY ; +This can effectively reduce the remote flush IPIs a lot!
One Nit is that, maybermb()orREAD_ONCE()is required forkvm->mmu_invalidate_range_startandkvm->mmu_invalidate_range_end.
Otherwise, I'm somewhat worried about constant false positive and retry.
If anything, this needs a READ_ONCE() on mmu_invalidate_in_progress. The ranges aren't touched when when mmu_invalidate_in_progress goes to zero, so ensuring they are reloaded wouldn't do anything. The key to making forward progress is seeing that there is no in-progress invalidation.
I did consider adding said READ_ONCE(), but practically speaking, constant false positives are impossible. KVM will re-enter the guest when retrying, and there is zero chance of the compiler avoiding reloads across VM-Enter+VM-Exit.
I suppose in theory we might someday differentiate between "retry because a different vCPU may have fixed the fault" and "retry because there's an in-progress invalidation", and not bother re-entering the guest for the latter, e.g. have it try to yield instead.
All that said, READ_ONCE() on mmu_invalidate_in_progress should effectively be a nop, so it wouldn't hurt to be paranoid in this case.
Hmm, at that point, it probably makes sense to add a READ_ONCE() for mmu_invalidate_seq too, e.g. so that a sufficiently clever compiler doesn't completely optimize away the check. Losing the check wouldn't be problematic (false negatives are fine, especially on that particular check), but the generated code would *look* buggy.
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 16 lines · Sean Christopherson
Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines
Note, I'm assuming secondary MMUs aren't allowed to map swap entries...
Compile tested only.
I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().Quoted 6 lines
Quoted 4 lines
It pairs with existing call to
mmu_notifier_invalidate_range_end()inchange_pmd_range():if (range.start) mmu_notifier_invalidate_range_end(&range);No, It doesn't work for
mmu_notifier_invalidate_range_start()sent inchange_pte_range(), if we only want the range to include pages successfully set toPROT_NONE.
Quoted 8 lines
Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the
start()call still used the original @end.The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.
For numa-balance scenery, can original page still be used by application even if pte is changed with PROT_NONE? If it can be used, maybe we can zap shadow mmu and flush tlb in notification mmu_notifier_invalidate_range_end with precised range, the range can be cross-range between range mmu_gather and mmu_notifier_range.
Regards
Bibo Mao
Quoted 38 lines
Quoted 9 lines
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +0
-0@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,if (unlikely (!fault->slot))return kvm_handle_noslot_fault (vcpu, fault, access);Quoted 7 lines
arch/x86/kvm/mmu/mmu.c static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · line 4350
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+if (mmu_invalidate_retry_hva (vcpu->kvm, fault->mmu_seq, fault->hva)) +return RET_PF_RETRY ; +This can effectively reduce the remote flush IPIs a lot!
One Nit is that, maybermb()orREAD_ONCE()is required forkvm->mmu_invalidate_range_startandkvm->mmu_invalidate_range_end.
Otherwise, I'm somewhat worried about constant false positive and retry.If anything, this needs a
READ_ONCE()onmmu_invalidate_in_progress. The ranges aren't touched when whenmmu_invalidate_in_progressgoes to zero, so ensuring they are reloaded wouldn't do anything. The key to making forward progress is seeing that there is no in-progress invalidation.I did consider adding said
READ_ONCE(), but practically speaking, constant false positives are impossible. KVM will re-enter the guest when retrying, and there is zero chance of the compiler avoiding reloads across VM-Enter+VM-Exit.I suppose in theory we might someday differentiate between "retry because a different vCPU may have fixed the fault" and "retry because there's an in-progress invalidation", and not bother re-entering the guest for the latter, e.g. have it try to yield instead.
All that said,
READ_ONCE()onmmu_invalidate_in_progressshould effectively be a nop, so it wouldn't hurt to be paranoid in this case.Hmm, at that point, it probably makes sense to add a
READ_ONCE()formmu_invalidate_seqtoo, e.g. so that a sufficiently clever compiler doesn't completely optimize away the check. Losing the check wouldn't be problematic (false negatives are fine, especially on that particular check), but the generated code would *look* buggy.
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 24 lines · bibo mao
Quoted 21 lines · Sean Christopherson
Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines
Note, I'm assuming secondary MMUs aren't allowed to map swap entries...
Compile tested only.
I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().Quoted 6 lines
Quoted 4 lines
It pairs with existing call to
mmu_notifier_invalidate_range_end()inchange_pmd_range():if (range.start) mmu_notifier_invalidate_range_end(&range);No, It doesn't work for
mmu_notifier_invalidate_range_start()sent inchange_pte_range(), if we only want the range to include pages successfully set toPROT_NONE.Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the
start()call still used the original @end.
Quoted 5 lines
Quoted 3 lines
The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.
For numa-balance scenery, can original page still be used by application even if pte is changed with
PROT_NONE? If it can be used, maybe we can zap shadow mmu and flush tlb
Since there is kvm_mmu_notifier_change_pte notification when numa page is replaced with new page, my meaning that can original page still be used by application even if pte is changed with PROT_NONE and before replaced with new page?
And for primary mmu, tlb is flushed after pte is changed with PROT_NONE and after mmu_notifier_invalidate_range_end notification for secondary mmu.
Regards
Bibo Mao
Quoted 43 lines
in notification
mmu_notifier_invalidate_range_endwith precised range, the range can be cross-range between rangemmu_gatherandmmu_notifier_range.Regards
Bibo MaoQuoted 38 lines
Quoted 9 lines
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +0
-0@@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,if (unlikely (!fault->slot))return kvm_handle_noslot_fault (vcpu, fault, access);Quoted 7 lines
arch/x86/kvm/mmu/mmu.c static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, · line 4350
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3
-0+if (mmu_invalidate_retry_hva (vcpu->kvm, fault->mmu_seq, fault->hva)) +return RET_PF_RETRY ; +This can effectively reduce the remote flush IPIs a lot!
One Nit is that, maybermb()orREAD_ONCE()is required forkvm->mmu_invalidate_range_startandkvm->mmu_invalidate_range_end.
Otherwise, I'm somewhat worried about constant false positive and retry.If anything, this needs a
READ_ONCE()onmmu_invalidate_in_progress. The ranges aren't touched when whenmmu_invalidate_in_progressgoes to zero, so ensuring they are reloaded wouldn't do anything. The key to making forward progress is seeing that there is no in-progress invalidation.I did consider adding said
READ_ONCE(), but practically speaking, constant false positives are impossible. KVM will re-enter the guest when retrying, and there is zero chance of the compiler avoiding reloads across VM-Enter+VM-Exit.I suppose in theory we might someday differentiate between "retry because a different vCPU may have fixed the fault" and "retry because there's an in-progress invalidation", and not bother re-entering the guest for the latter, e.g. have it try to yield instead.
All that said,
READ_ONCE()onmmu_invalidate_in_progressshould effectively be a nop, so it wouldn't hurt to be paranoid in this case.Hmm, at that point, it probably makes sense to add a
READ_ONCE()formmu_invalidate_seqtoo, e.g. so that a sufficiently clever compiler doesn't completely optimize away the check. Losing the check wouldn't be problematic (false negatives are fine, especially on that particular check), but the generated code would *look* buggy.
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 27 lines · bibo mao
Quoted 24 lines · bibo mao
Quoted 21 lines · Sean Christopherson
Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines
Note, I'm assuming secondary MMUs aren't allowed to map swap entries...
Compile tested only.
I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().Quoted 6 lines
Quoted 4 lines
It pairs with existing call to
mmu_notifier_invalidate_range_end()inchange_pmd_range():if (range.start) mmu_notifier_invalidate_range_end(&range);No, It doesn't work for
mmu_notifier_invalidate_range_start()sent inchange_pte_range(), if we only want the range to include pages successfully set toPROT_NONE.Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the
start()call still used the original @end.
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.
For numa-balance scenery, can original page still be used by application even if pte is changed with
PROT_NONE? If it can be used, maybe we can zap shadow mmu and flush tlb
For GUPs that does not honor FOLL_HONOR_NUMA_FAULT, yes,
See https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
Since there is
kvm_mmu_notifier_change_ptenotification when numa page is replaced with new page, my meaning that can original page still be used by application even if pte is changed withPROT_NONEand before replaced with new page?
It's not .change_pte() notification, which is sent when COW. The do_numa_page()/do_huge_pmd_numa_page() will try to unmap old page protected with PROT_NONE, and if every check passes, a separate .invalidate_range_start()/end() with event type MMU_NOTIFY_CLEAR will be sent.
So, I think KVM (though it honors FOLL_HONOR_NUMA_FAULT), can safely keep mapping maybe-dma pages until MMU_NOTIFY_CLEAR is sent. (this approach is implemented in RFC v1
https://lore.kernel.org/all/[email protected]/)
Quoted 7 lines
And for primary mmu, tlb is flushed after pte is changed with
PROT_NONEand aftermmu_notifier_invalidate_range_endnotification for secondary mmu. Regards
Bibo MaoQuoted 1 line
Quoted 1 line
in notification
mmu_notifier_invalidate_range_endwith precised range, the range can
But I don't think flush tlb only in the .invalidate_range_end() in secondary MMU is a good idea.
Flush must be done before kvm->mmu_lock is unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.
Quoted 1 line
Quoted 1 line
Quoted 1 line
be cross-range between range
mmu_gatherandmmu_notifier_range.
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 37 lines · Yan Zhao
Quoted 27 lines · bibo mao
Quoted 24 lines · bibo mao
Quoted 21 lines · Sean Christopherson
Quoted 8 lines · Yan Zhao
Quoted 7 lines · Sean Christopherson
Quoted 6 lines
Quoted 3 lines
Note, I'm assuming secondary MMUs aren't allowed to map swap entries...
Compile tested only.
I don't find a matching end to each
mmu_notifier_invalidate_range_start_nonblock().Quoted 6 lines
Quoted 4 lines
It pairs with existing call to
mmu_notifier_invalidate_range_end()inchange_pmd_range():if (range.start) mmu_notifier_invalidate_range_end(&range);No, It doesn't work for
mmu_notifier_invalidate_range_start()sent inchange_pte_range(), if we only want the range to include pages successfully set toPROT_NONE.Precise invalidation was a non-goal for my hack-a-patch. The intent was purely to defer invalidation until it was actually needed, but still perform only a single notification so as to batch the TLB flushes, e.g. the
start()call still used the original @end.Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
The idea was to play nice with the scenario where nothing in a VMA could be migrated. It was complete untested though, so it may not have actually done anything to reduce the number of pointless invalidations.
For numa-balance scenery, can original page still be used by application even if pte is changed with
PROT_NONE? If it can be used, maybe we can zap shadow mmu and flush tlbFor GUPs that does not honor
FOLL_HONOR_NUMA_FAULT, yes,
Quoted 8 lines
Quoted 3 lines
Since there is
kvm_mmu_notifier_change_ptenotification when numa page is replaced with new page, my meaning that can original page still be used by application even if pte is changed withPROT_NONEand before replaced with new page?It's not .change_pte() notification, which is sent when COW. The do_numa_page()/do_huge_pmd_numa_page() will try to unmap old page protected with
PROT_NONE, and if every check passes, a separate .invalidate_range_start()/end() with event typeMMU_NOTIFY_CLEARwill be sent.
yes, you are right. change_pte() notification will be will called when migrate_vma_pages, I messed it with numa page migration. However invalidate_range_start()/end() with event type MMU_NOTIFY_CLEAR will be sent also when new page is replaced.
Quoted 7 lines
So, I think KVM (though it honors
FOLL_HONOR_NUMA_FAULT), can safely keep mapping maybe-dma pages untilMMU_NOTIFY_CLEARis sent. (this approach is implemented in RFC v1
https://lore.kernel.org/all/[email protected]/)Quoted 1 line
Quoted 8 lines
Quoted 6 lines
And for primary mmu, tlb is flushed after pte is changed with
PROT_NONEand aftermmu_notifier_invalidate_range_endnotification for secondary mmu. Regards
Bibo MaoQuoted 1 line
Quoted 1 line
in notification
mmu_notifier_invalidate_range_endwith precised range, the range canBut I don't think flush tlb only in the .invalidate_range_end() in secondary MMU is a good idea.
I have no good idea, and it beyond my ability to modify kvm framework now :(
Quoted 3 lines
Flush must be done before
kvm->mmu_lockis unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.
Since tlb flush is delayed after all pte entries are cleared, and currently there is no tlb flush range supported for secondary mmu. I do know why there is confusion before or after kvm->mmu_lock.
Regards
Bibo Mao
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
be cross-range between range
mmu_gatherandmmu_notifier_range.
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 6 lines · bibo mao
Quoted 3 lines
Flush must be done before
kvm->mmu_lockis unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.Since tlb flush is delayed after all pte entries are cleared, and currently there is no tlb flush range supported for secondary mmu. I do know why there is confusion before or after
kvm->mmu_lock.
Oh, do you mean only do kvm_unmap_gfn_range() in .invalidate_range_end()? Then check if PROT_NONE is set in primary MMU before unmap? Looks like a good idea, I need to check if it's feasible. Thanks!
Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 9 lines · Yan Zhao
Quoted 6 lines · bibo mao
Quoted 3 lines
Flush must be done before
kvm->mmu_lockis unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.Since tlb flush is delayed after all pte entries are cleared, and currently there is no tlb flush range supported for secondary mmu. I do know why there is confusion before or after
kvm->mmu_lock.Oh, do you mean only do
kvm_unmap_gfn_range()in .invalidate_range_end()?
yes, it is just sketchy thought for numa balance scenery, do kvm_unmap_gfn_range() in invalidate_range_end rather than invalidate_range_start.
Quoted 5 lines
Then check if
PROT_NONEis set in primary MMU before unmap? Looks like a good idea, I need to check if it's feasible. Thanks!
Sean Christopherson
maintainerRe: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
Quoted 7 lines · bibo mao
Quoted 4 lines · Yan Zhao
Quoted 3 lines · bibo mao
Quoted 3 lines
Flush must be done before
kvm->mmu_lockis unlocked, otherwise, confusion will be caused when multiple threads trying to update the secondary MMU.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Since tlb flush is delayed after all pte entries are cleared, and currently there is no tlb flush range supported for secondary mmu. I do know why there is confusion before or after
kvm->mmu_lock.Oh, do you mean only do
kvm_unmap_gfn_range()in .invalidate_range_end()?yes, it is just sketchy thought for numa balance scenery, do
kvm_unmap_gfn_range()ininvalidate_range_endrather thaninvalidate_range_start.
That is not an option, it's a direction violation of the mmu_notifier contract. Secondary MMUs must drop all references before returning from invalidate_range_start().
kernel test robot
build botRe: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration
In reply to Yan Zhaokernel test robot noticed the following build warnings:
Hi Yan,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on kvm/queue]
[also build test WARNING on mst-vhost/linux-next linus/master v6.5-rc5 next-20230809] [cannot apply to akpm-mm/mm-everything kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Yan-Zhao/mm-mmu_notifier-introduce-a-new-mmu-notifier-flag-MMU_NOTIFIER_RANGE_NUMA/20230810-172955 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue patch link: https://lore.kernel.org/r/20230810090218.26244-1-yan.y.zhao%40intel.com patch subject: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration config: riscv-randconfig-r023-20230810 (https://download.01.org/0day-ci/archive/20230810/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a) reproduce: (https://download.01.org/0day-ci/archive/20230810/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> arch/riscv/kvm/../../../virt/kvm/kvm_main.c:767:23: warning: pointer type mismatch ('bool (*)(struct kvm *, struct kvm_gfn_range *)' (aka '_Bool (*)(struct kvm *, struct kvm_gfn_range *)') and 'void *') [-Wpointer-type-mismatch]
767 | .handler = !is_numa ? kvm_unmap_gfn_range :
| ^ ~~~~~~~~~~~~~~~~~~~
768 | (void *)kvm_null_fn,
| ~~~~~~~~~~~~~~~~~~~
>> arch/riscv/kvm/../../../virt/kvm/kvm_main.c:770:25: warning: pointer type mismatch ('void (*)(struct kvm *)' and 'void *') [-Wpointer-type-mismatch]
770 | .on_unlock = !is_numa ? kvm_arch_guest_memory_reclaimed :
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
771 | (void *)kvm_null_fn,
| ~~~~~~~~~~~~~~~~~~~2 warnings generated.
vim +767 arch/riscv/kvm/../../../virt/kvm/kvm_main.c
756
757 static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
758 const struct mmu_notifier_range *range)
759 {
760 struct kvm *kvm = mmu_notifier_to_kvm(mn);
761 bool is_numa = (range->event == MMU_NOTIFY_PROTECTION_VMA) &&
762 (range->flags & MMU_NOTIFIER_RANGE_NUMA);
763 const struct kvm_hva_range hva_range = {
764 .start = range->start,
765 .end = range->end,
766 .pte = __pte(0),
> 767 .handler = !is_numa ? kvm_unmap_gfn_range : 768 (void *)kvm_null_fn, 769 .on_lock = kvm_mmu_invalidate_begin, > 770 .on_unlock = !is_numa ? kvm_arch_guest_memory_reclaimed :
771 (void *)kvm_null_fn,
772 .flush_on_ret = !is_numa ? true : false,
773 .may_block = mmu_notifier_range_blockable(range),
774 };
775
776 trace_kvm_unmap_hva_range(range->start, range->end);
777
778 /*
779 * Prevent memslot modification between range_start() and range_end()
780 * so that conditionally locking provides the same result in both
781 * functions. Without that guarantee, the mmu_invalidate_in_progress
782 * adjustments will be imbalanced.
783 *
784 * Pairs with the decrement in range_end().
785 */
786 spin_lock(&kvm->mn_invalidate_lock);
787 kvm->mn_active_invalidate_count++;
788 spin_unlock(&kvm->mn_invalidate_lock);
789
790 /*
791 * Invalidate pfn caches _before_ invalidating the secondary MMUs, i.e.
792 * before acquiring mmu_lock, to avoid holding mmu_lock while acquiring
793 * each cache's lock. There are relatively few caches in existence at
794 * any given time, and the caches themselves can check for hva overlap,
795 * i.e. don't need to rely on memslot overlap checks for performance.
796 * Because this runs without holding mmu_lock, the pfn caches must use
797 * mn_active_invalidate_count (see above) instead of
798 * mmu_invalidate_in_progress.
799 */
800 gfn_to_pfn_cache_invalidate_start(kvm, range->start, range->end,
801 hva_range.may_block);
802
803 __kvm_handle_hva_range(kvm, &hva_range);
804
805 return 0;
806 }
807
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Quoted 27 lines · Yan Zhao
This is an RFC series trying to fix the issue of unnecessary NUMA protection and TLB-shootdowns found in VMs with assigned devices or VFIO mediated devices during NUMA balance.
For VMs with assigned devices or VFIO mediated devices, all or part of guest memory are pinned for long-term.
Auto NUMA balancing will periodically selects VMAs of a process and change protections to
PROT_NONEeven though some or all pages in the selected ranges are long-term pinned for DMAs, which is true for VMs with assigned devices or VFIO mediated devices.Though this will not cause real problem because NUMA migration will ultimately reject migration of those kind of pages and restore those
PROT_NONEPTEs, it causes KVM's secondary MMU to be zapped periodically with equal SPTEs finally faulted back, wasting CPU cycles and generating unnecessary TLB-shootdowns.This series first introduces a new flag
MMU_NOTIFIER_RANGE_NUMAin patch 1 to work with mmu notifier event typeMMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.Patch 2 skips setting
PROT_NONEto long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.
Quoted 7 lines
Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be
PROT_NONEprotected.Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.
Why do we need all that, when we should simply not be applying PROT_NONE to pinned pages?
In change_pte_range() we already have:
if (is_cow_mapping(vma->vm_flags) &&
page_count(page) != 1)
Which includes both, shared and pinned pages.
Staring at page #2, are we still missing something similar for THPs?
Why is that MMU notifier thingy and touching KVM code required?
Signature · 3 lines
Cheers, David / dhildenb
Quoted 20 lines · David Hildenbrand
Quoted 9 lines
This series first introduces a new flag
MMU_NOTIFIER_RANGE_NUMAin patch 1 to work with mmu notifier event typeMMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.Patch 2 skips setting
PROT_NONEto long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.Quoted 7 lines
Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be
PROT_NONEprotected.Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.
Why do we need all that, when we should simply not be applying
PROT_NONEto pinned pages?
Quoted 6 lines
In
change_pte_range()we already have:if (
is_cow_mapping(vma->vm_flags)&&page_count(page) != 1)Which includes both, shared and pinned pages.
Ah, right, currently in my side, I don't see any pinned pages are outside of this condition.
But I have a question regarding to is_cow_mapping(vma->vm_flags), do we need to allow pinned pages in !is_cow_mapping(vma->vm_flags)?
Quoted 1 line
Staring at page #2, are we still missing something similar for THPs?
Yes.
Quoted 1 line
Why is that MMU notifier thingy and touching KVM code required?
Because NUMA balancing code will firstly send .invalidate_range_start() with event type MMU_NOTIFY_PROTECTION_VMA to KVM in change_pmd_range() unconditionally, before it goes down into change_pte_range() and change_huge_pmd() to check each page count and apply PROT_NONE.
Then current KVM will unmap all notified pages from secondary MMU in .invalidate_range_start(), which could include pages that finally not set to PROT_NONE in primary MMU.
For VMs with pass-through devices, though all guest pages are pinned, KVM still periodically unmap pages in response to the .invalidate_range_start() notification from auto NUMA balancing, which is a waste.
So, if there's a new callback sent when pages is set to PROT_NONE for NUMA migrate only, KVM can unmap only those pages.
As KVM still needs to unmap pages for other type of event in its handler of .invalidate_range_start() (.i.e. kvm_mmu_notifier_invalidate_range_start()), and MMU_NOTIFY_PROTECTION_VMA also include other reasons, so patch 1 added a range flag to help KVM not to do a blind unmap in .invalidate_range_start(), but do it in the new .numa_protect() handler.
Quoted 7 lines
Signature · 4 lines
-- Cheers, David / dhildenb
Quoted 23 lines · Yan Zhao
Quoted 22 lines · David Hildenbrand
Quoted 9 lines
This series first introduces a new flag
MMU_NOTIFIER_RANGE_NUMAin patch 1 to work with mmu notifier event typeMMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.Patch 2 skips setting
PROT_NONEto long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.Quoted 7 lines
Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be
PROT_NONEprotected.Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.
Why do we need all that, when we should simply not be applying
PROT_NONEto pinned pages?In
change_pte_range()we already have:
Quoted 8 lines
Quoted 4 lines
if (
is_cow_mapping(vma->vm_flags)&&page_count(page) != 1)Which includes both, shared and pinned pages.
Ah, right, currently in my side, I don't see any pinned pages are outside of this condition.
But I have a question regarding tois_cow_mapping(vma->vm_flags), do we need to allow pinned pages in !is_cow_mapping(vma->vm_flags)?
One issue is that folio_maybe_pinned...() ... is unreliable as soon as your page is mapped more than 1024 times.
One might argue that we also want to exclude pages that are mapped that often. That might possibly work.
Quoted 9 lines
Quoted 2 lines
Staring at page #2, are we still missing something similar for THPs?
Yes.
Quoted 1 line
Why is that MMU notifier thingy and touching KVM code required?
Because NUMA balancing code will firstly send .invalidate_range_start() with event type
MMU_NOTIFY_PROTECTION_VMAto KVM inchange_pmd_range()unconditionally, before it goes down intochange_pte_range()andchange_huge_pmd()to check each page count and applyPROT_NONE.
Ah, okay I see, thanks. That's indeed unfortunate.
Quoted 9 lines
Then current KVM will unmap all notified pages from secondary MMU in .invalidate_range_start(), which could include pages that finally not set to
PROT_NONEin primary MMU.For VMs with pass-through devices, though all guest pages are pinned, KVM still periodically unmap pages in response to the .invalidate_range_start() notification from auto NUMA balancing, which is a waste.
Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are similar issues with GPU memory: NUMA hinting is actually counter-productive and they end up disabling it.
Signature · 3 lines
Cheers, David / dhildenb
On 8/11/23 10:25, David Hildenbrand wrote:
...
Quoted 3 lines
One issue is that folio_maybe_pinned...() ... is unreliable as soon as your page is mapped more than 1024 times.
One might argue that we also want to exclude pages that are mapped that often. That might possibly work.
Yes.
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines
Staring at page #2, are we still missing something similar for THPs?
Yes.
Quoted 1 line
Why is that MMU notifier thingy and touching KVM code required?
Because NUMA balancing code will firstly send .invalidate_range_start() with event type
MMU_NOTIFY_PROTECTION_VMAto KVM inchange_pmd_range()unconditionally, before it goes down intochange_pte_range()andchange_huge_pmd()to check each page count and applyPROT_NONE.Ah, okay I see, thanks. That's indeed unfortunate.
Sigh. All this difficulty reminds me that this mechanism was created in the early days of NUMA. I wonder sometimes lately whether the cost, in complexity and CPU time, is still worth it on today's hardware.
But of course I am deeply biased, so don't take that too seriously. See below. :)
Quoted 6 lines
Quoted 6 lines
Then current KVM will unmap all notified pages from secondary MMU in .invalidate_range_start(), which could include pages that finally not set to
PROT_NONEin primary MMU.
Quoted 6 lines
Quoted 4 lines
For VMs with pass-through devices, though all guest pages are pinned, KVM still periodically unmap pages in response to the .invalidate_range_start() notification from auto NUMA balancing, which is a waste.
Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are
similar issues with GPU memory: NUMA hinting is actually counter-productive and they end up disabling it.
Quoted 1 line
Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 8 lines
Quoted 1 line
Ah, okay I see, thanks. That's indeed unfortunate.
Sigh. All this difficulty reminds me that this mechanism was created in the early days of NUMA. I wonder sometimes lately whether the cost, in complexity and CPU time, is still worth it on today's hardware.
But of course I am deeply biased, so don't take that too seriously. See below. :)
:)
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Then current KVM will unmap all notified pages from secondary MMU in .invalidate_range_start(), which could include pages that finally not set to
PROT_NONEin primary MMU.Quoted 6 lines
Quoted 4 lines
For VMs with pass-through devices, though all guest pages are pinned, KVM still periodically unmap pages in response to the .invalidate_range_start() notification from auto NUMA balancing, which is a waste.
Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are
similar issues with GPU memory: NUMA hinting is actually counter-productive and they end up disabling it.
Quoted 1 line
Quoted 6 lines
Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.
Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?
MMF_HAS_PINNED might be too restrictive. MMF_HAS_PINNED_LONGTERM might be better, but with things like iouring still too restrictive eventually.
I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.
CCing Mel.
Signature · 3 lines
Cheers, David / dhildenb
On 8/11/23 11:39, David Hildenbrand wrote:
...
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are
similar issues with GPU memory: NUMA hinting is actually counter-productive and they end up disabling it.
Quoted 1 line
Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.
Quoted 8 lines
Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?
MMF_HAS_PINNEDmight be too restrictive.MMF_HAS_PINNED_LONGTERMmight be better, but with things like iouring still too restrictive eventually.I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.
CCing Mel.
Let's discern between page pinning situations, and HMM-style situations. Page pinning of CPU memory is unnecessary when setting up for using that memory by modern GPUs or accelerators, because the latter can handle replayable page faults. So for such cases, the pages are in use by a GPU or accelerator, but unpinned.
The performance problem occurs because for those pages, the NUMA balancing causes unmapping, which generates callbacks to the device driver, which dutifully unmaps the pages from the GPU or accelerator, even if the GPU might be busy using those pages. The device promptly causes a device page fault, and the driver then re-establishes the device page table mapping, which is good until the next round of unmapping from the NUMA balancer.
hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the memremap_pages() region, come to think of it. That seems relatively easy and clean at first glance anyway.
For other regions (allocated by the device driver), a per-VMA flag seems about right: VM_NO_NUMA_BALANCING ?
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 36 lines · John Hubbard
On 8/11/23 11:39, David Hildenbrand wrote:
...Quoted 11 lines
Quoted 10 lines
Quoted 1 line
Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are
similar issues with GPU memory: NUMA hinting is actually counter-productive and they end up disabling it.
Quoted 1 line
Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.
Quoted 8 lines
Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?
MMF_HAS_PINNEDmight be too restrictive.MMF_HAS_PINNED_LONGTERMmight be better, but with things like iouring still too restrictive eventually.I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.
CCing Mel.
Let's discern between page pinning situations, and HMM-style situations. Page pinning of CPU memory is unnecessary when setting up for using that memory by modern GPUs or accelerators, because the latter can handle replayable page faults. So for such cases, the pages are in use by a GPU or accelerator, but unpinned.
The performance problem occurs because for those pages, the NUMA balancing causes unmapping, which generates callbacks to the device driver, which dutifully unmaps the pages from the GPU or accelerator, even if the GPU might be busy using those pages. The device promptly causes a device page fault, and the driver then re-establishes the device page table mapping, which is good until the next round of unmapping from the NUMA balancer.
Quoted 8 lines
hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the
memremap_pages()region, come to think of it. That seems relatively easy and clean at first glance anyway.For other regions (allocated by the device driver), a per-VMA flag seems about right:
VM_NO_NUMA_BALANCING?
Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard in mmap() in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?
On 8/14/23 02:09, Yan Zhao wrote:
...
Quoted 8 lines
Quoted 8 lines
hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the
memremap_pages()region, come to think of it. That seems relatively easy and clean at first glance anyway.For other regions (allocated by the device driver), a per-VMA flag seems about right:
VM_NO_NUMA_BALANCING?
Quoted 7 lines
Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard inmmap()in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?
I'm hoping someone who thinks about VMs and VFIO often can chime in.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 18 lines · John Hubbard
On 8/14/23 02:09, Yan Zhao wrote:
...Quoted 8 lines
Quoted 8 lines
hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the
memremap_pages()region, come to think of it. That seems relatively easy and clean at first glance anyway.For other regions (allocated by the device driver), a per-VMA flag seems about right:
VM_NO_NUMA_BALANCING?Quoted 7 lines
Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard inmmap()in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?
Quoted 1 line
I'm hoping someone who thinks about VMs and VFIO often can chime in.
At least QEMU could just set it on the applicable VMAs (as said by Yuan Yao, using madvise).
BUT, I do wonder what value there would be for autonuma to still be active for the remainder of the hypervisor. If there is none, a prctl() would be better.
We already do have a mechanism in QEMU to get notified when longterm-pinning in the kernel might happen (and, therefore, MADV_DONTNEED must not be used):
* ram_block_discard_disable()
* ram_block_uncoordinated_discard_disable()
Signature · 3 lines
Cheers, David / dhildenb
Quoted 19 lines · David Hildenbrand
Quoted 18 lines · John Hubbard
On 8/14/23 02:09, Yan Zhao wrote:
...Quoted 15 lines
Quoted 8 lines
hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the
memremap_pages()region, come to think of it. That seems relatively easy and clean at first glance anyway.For other regions (allocated by the device driver), a per-VMA flag seems about right:
VM_NO_NUMA_BALANCING?Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard inmmap()in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?
Quoted 6 lines
Quoted 1 line
I'm hoping someone who thinks about VMs and VFIO often can chime in.
At least QEMU could just set it on the applicable VMAs (as said by Yuan Yao, using madvise).
BUT, I do wonder what value there would be for autonuma to still be active
Currently MADV_* is up to 25
#define MADV_COLLAPSE 25,
while madvise behavior is of type "int". So it's ok.
But vma->vm_flags is of "unsigned long", so it's full at least on 32bit platform.
Quoted 2 lines
for the remainder of the hypervisor. If there is none, a
prctl()would be better.
Add a new field in "struct vma_numab_state" in vma, and use prctl() to update this field?
e.g.
struct vma_numab_state {
unsigned long next_scan;
unsigned long next_pid_reset;
unsigned long access_pids[2];
bool no_scan;
};Quoted 5 lines
We already do have a mechanism in QEMU to get notified when longterm-pinning in the kernel might happen (and, therefore,
MADV_DONTNEEDmust not be used):
*ram_block_discard_disable()
*ram_block_uncoordinated_discard_disable()
Looks this ram_block_discard allow/disallow state is global rather than per-VMA in QEMU.
So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and it's up to the user space to choose between per-VMA and complex way or global and simpler way?
Quoted 25 lines · Yan Zhao
Quoted 24 lines · David Hildenbrand
Quoted 18 lines · John Hubbard
On 8/14/23 02:09, Yan Zhao wrote:
...Quoted 8 lines
Quoted 8 lines
hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the
memremap_pages()region, come to think of it. That seems relatively easy and clean at first glance anyway.For other regions (allocated by the device driver), a per-VMA flag seems about right:
VM_NO_NUMA_BALANCING?Quoted 7 lines
Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard inmmap()in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?Quoted 1 line
I'm hoping someone who thinks about VMs and VFIO often can chime in.
At least QEMU could just set it on the applicable VMAs (as said by Yuan Yao, using madvise).
Quoted 6 lines
Quoted 1 line
BUT, I do wonder what value there would be for autonuma to still be active
Currently MADV_* is up to 25
#define MADV_COLLAPSE 25,while madvise behavior is of type "int". So it's ok.
But
vma->vm_flagsis of "unsigned long", so it's full at least on 32bit platform.
I remember there were discussions to increase it also for 32bit. If that's required, we might want to go down that path.
But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...
Quoted 5 lines
Quoted 3 lines
for the remainder of the hypervisor. If there is none, a
prctl()would be better.Add a new field in "struct
vma_numab_state" in vma, and useprctl()to update this field?
Rather a global toggle per MM, no need to update individual VMAs -- if we go down that prctl() path.
No need to consume more memory for VMAs.
[...]
Quoted 6 lines
Quoted 4 lines
We already do have a mechanism in QEMU to get notified when longterm-pinning in the kernel might happen (and, therefore,
MADV_DONTNEEDmust not be used):
*ram_block_discard_disable()
*ram_block_uncoordinated_discard_disable()Looks this
ram_block_discardallow/disallow state is global rather than per-VMA in QEMU.
Yes. Once you transition into "discard of any kind disabled", you can go over all guest memory VMAs (RAMBlock) and issue an madvise() for them. (or alternatively, do the prctl() once )
We'll also have to handle new guest memory being created afterwards, but that is easy.
Once we transition to "no discarding disabled", you can go over all guest memory VMAs (RAMBlock) and issue an madvise() for them again (or alternatively, do the prctl() once).
Quoted 3 lines
So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and it's up to the user space to choose between per-VMA and complex way or global and simpler way?
QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 1 line · David Hildenbrand
But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...
Probably not!
...
Quoted 5 lines
Quoted 3 lines
So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and it's up to the user space to choose between per-VMA and complex way or global and simpler way?
QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.
From our experience with compute on GPUs, a per-mm setting would suffice. No need to go all the way to VMA granularity.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 12 lines · John Hubbard
Quoted 2 lines · David Hildenbrand
But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...
Probably not!
...
Quoted 5 lines
Quoted 4 lines
So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and
it's up to the user space to choose between per-VMA and complex way or global and simpler way?
Quoted 6 lines
Quoted 2 lines
QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.
From our experience with compute on GPUs, a per-mm setting would suffice. No need to go all the way to VMA granularity.
After an offline internal discussion, we think a per-mm setting is also enough for device passthrough in VMs.
BTW, if we want a per-VMA flag, compared to VM_NO_NUMA_BALANCING, do you think it's of any value to providing a flag like VM_MAYDMA? Auto NUMA balancing or other components can decide how to use it by themselves.
Quoted 19 lines · Yan Zhao
Quoted 12 lines · John Hubbard
Quoted 2 lines · David Hildenbrand
But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...
Probably not!
...
Quoted 5 lines
Quoted 4 lines
So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and
it's up to the user space to choose between per-VMA and complex way or global and simpler way?Quoted 6 lines
Quoted 2 lines
QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.
From our experience with compute on GPUs, a per-mm setting would suffice.
No need to go all the way to VMA granularity.
Quoted 7 lines
After an offline internal discussion, we think a per-mm setting is also enough for device passthrough in VMs.
BTW, if we want a per-VMA flag, compared to
VM_NO_NUMA_BALANCING, do you think it's of any value to providing a flag likeVM_MAYDMA? Auto NUMA balancing or other components can decide how to use it by themselves.
Short-lived DMA is not really the problem. The problem is long-term pinning.
There was a discussion about letting user space similarly hint that long-term pinning might/will happen.
Because when long-term pinning a page we have to make sure to migrate it off of ZONE_MOVABLE / MIGRATE_CMA.
But the kernel prefers to place pages there.
So with vfio in QEMU, we might preallocate memory for the guest and place it on ZONE_MOVABLE/MIGRATE_CMA, just so long-term pinning has to migrate all these fresh pages out of these areas again.
So letting the kernel know about that in this context might also help.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 36 lines · David Hildenbrand
Quoted 19 lines · Yan Zhao
Quoted 12 lines · John Hubbard
Quoted 2 lines · David Hildenbrand
But do 32bit architectures even care about NUMA hinting? If not, just ignore them ...
Probably not!
...
Quoted 5 lines
Quoted 4 lines
So, do you mean that let kernel provide a per-VMA allow/disallow mechanism, and
it's up to the user space to choose between per-VMA and complex way or global and simpler way?Quoted 6 lines
Quoted 2 lines
QEMU could do either way. The question would be if a per-vma settings makes sense for NUMA hinting.
From our experience with compute on GPUs, a per-mm setting would suffice.
No need to go all the way to VMA granularity.Quoted 7 lines
After an offline internal discussion, we think a per-mm setting is also enough for device passthrough in VMs.
BTW, if we want a per-VMA flag, compared to
VM_NO_NUMA_BALANCING, do you think it's of any value to providing a flag likeVM_MAYDMA? Auto NUMA balancing or other components can decide how to use it by themselves.Short-lived DMA is not really the problem. The problem is long-term pinning.
There was a discussion about letting user space similarly hint that long-term pinning might/will happen.
Because when long-term pinning a page we have to make sure to migrate it off of
ZONE_MOVABLE/MIGRATE_CMA.
Quoted 8 lines
But the kernel prefers to place pages there.
So with vfio in QEMU, we might preallocate memory for the guest and place it on ZONE_MOVABLE/MIGRATE_CMA, just so long-term pinning has to migrate all these fresh pages out of these areas again.
So letting the kernel know about that in this context might also help.
Thanks! Glad to know it :)
But consider for GPUs case as what John mentioned, since the memory is not even pinned, maybe they still need flag VM_NO_NUMA_BALANCING ? For VMs, we hint VM_NO_NUMA_BALANCING for passthrough devices supporting IO page fault (so no need to pin), and VM_MAYLONGTERMDMA to avoid misplace and migration.
Is that good?
Or do you think just a per-mm flag like MMF_NO_NUMA is good enough for now?
On 8/17/23 17:13, Yan Zhao wrote:
...
Quoted 10 lines
But consider for GPUs case as what John mentioned, since the memory is not even pinned, maybe they still need flag
VM_NO_NUMA_BALANCING? For VMs, we hintVM_NO_NUMA_BALANCINGfor passthrough devices supporting IO page fault (so no need to pin), andVM_MAYLONGTERMDMAto avoid misplace and migration.Is that good?
Or do you think just a per-mm flag likeMMF_NO_NUMAis good enough for now?
So far, a per-mm setting seems like it would suffice. However, it is also true that new hardware is getting really creative and large, to the point that it's not inconceivable that a process might actually want to let NUMA balancing run in part of its mm, but turn it off to allow fault-able device access to another part of the mm.
We aren't seeing that yet, but on the other hand, that may be simply because there is no practical way to set that up and see how well it works.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 19 lines · John Hubbard
On 8/17/23 17:13, Yan Zhao wrote:
...Quoted 10 lines
But consider for GPUs case as what John mentioned, since the memory is not even pinned, maybe they still need flag
VM_NO_NUMA_BALANCING? For VMs, we hintVM_NO_NUMA_BALANCINGfor passthrough devices supporting IO page fault (so no need to pin), andVM_MAYLONGTERMDMAto avoid misplace and migration.Is that good?
Or do you think just a per-mm flag likeMMF_NO_NUMAis good enough for now?So far, a per-mm setting seems like it would suffice. However, it is also true that new hardware is getting really creative and large, to the point that it's not inconceivable that a process might actually want to let NUMA balancing run in part of its mm, but turn it off to allow fault-able device access to another part of the mm.
Quoted 5 lines
We aren't seeing that yet, but on the other hand, that may be simply because there is no practical way to set that up and see how well it works.
Hi guys,
Thanks a lot for your review and suggestions!
I'll firstly try to add a per-mm flag to fix this problem later (but maybe not very soon)
Thanks
Yan
Quoted 45 lines · Yan Zhao
Quoted 36 lines · John Hubbard
On 8/11/23 11:39, David Hildenbrand wrote:
...Quoted 11 lines
Quoted 10 lines
Quoted 1 line
Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are
similar issues with GPU memory: NUMA hinting is actually counter-productive and they end up disabling it.
Quoted 1 line
Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.
Quoted 8 lines
Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?
MMF_HAS_PINNEDmight be too restrictive.MMF_HAS_PINNED_LONGTERMmight be better, but with things like iouring still too restrictive eventually.I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.
CCing Mel.
Let's discern between page pinning situations, and HMM-style situations. Page pinning of CPU memory is unnecessary when setting up for using that memory by modern GPUs or accelerators, because the latter can handle replayable page faults. So for such cases, the pages are in use by a GPU or accelerator, but unpinned.
The performance problem occurs because for those pages, the NUMA balancing causes unmapping, which generates callbacks to the device driver, which dutifully unmaps the pages from the GPU or accelerator, even if the GPU might be busy using those pages. The device promptly causes a device page fault, and the driver then re-establishes the device page table mapping, which is good until the next round of unmapping from the NUMA balancer.
Quoted 8 lines
hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the
memremap_pages()region, come to think of it. That seems relatively easy and clean at first glance anyway.For other regions (allocated by the device driver), a per-VMA flag seems about right:
VM_NO_NUMA_BALANCING?
Quoted 6 lines
Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard inmmap()in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?
Emm... syscall madvise() in my mind, it does things like change flags on VMA, e.g madvise(MADV_DONTFORK) adds VM_DONTCOPY to the VMA.
Quoted 2 lines
Quoted 53 lines · Yuan Yao
Quoted 45 lines · Yan Zhao
Quoted 36 lines · John Hubbard
On 8/11/23 11:39, David Hildenbrand wrote:
...Quoted 11 lines
Quoted 10 lines
Quoted 1 line
Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are
similar issues with GPU memory: NUMA hinting is actually counter-productive and they end up disabling it.
Quoted 1 line
Yes, NUMA balancing is incredibly harmful to performance, for GPU and accelerators that map memory...and VMs as well, it seems. Basically, anything that has its own processors and page tables needs to be left strictly alone by NUMA balancing. Because the kernel is (still, even today) unaware of what those processors are doing, and so it has no way to do productive NUMA balancing.
Quoted 8 lines
Is there any existing way we could handle that better on a per-VMA level, or on the process level? Any magic toggles?
MMF_HAS_PINNEDmight be too restrictive.MMF_HAS_PINNED_LONGTERMmight be better, but with things like iouring still too restrictive eventually.I recall that setting a mempolicy could prevent auto-numa from getting active, but that might be undesired.
CCing Mel.
Let's discern between page pinning situations, and HMM-style situations. Page pinning of CPU memory is unnecessary when setting up for using that memory by modern GPUs or accelerators, because the latter can handle replayable page faults. So for such cases, the pages are in use by a GPU or accelerator, but unpinned.
The performance problem occurs because for those pages, the NUMA balancing causes unmapping, which generates callbacks to the device driver, which dutifully unmaps the pages from the GPU or accelerator, even if the GPU might be busy using those pages. The device promptly causes a device page fault, and the driver then re-establishes the device page table mapping, which is good until the next round of unmapping from the NUMA balancer.
Quoted 8 lines
hmm_range_fault()-based memory management in particular might benefit from having NUMA balancing disabled entirely for the
memremap_pages()region, come to think of it. That seems relatively easy and clean at first glance anyway.For other regions (allocated by the device driver), a per-VMA flag seems about right:
VM_NO_NUMA_BALANCING?Quoted 6 lines
Thanks a lot for those good suggestions!
For VMs, when could a per-VMA flag be set?
Might be hard inmmap()in QEMU because a VMA may not be used for DMA until after it's mapped into VFIO.
Then, should VFIO set this flag on after it maps a range?
Could this flag be unset after device hot-unplug?
Quoted 2 lines
Emm... syscall
madvise()in my mind, it does things like change flags on VMA, e.gmadvise(MADV_DONTFORK)addsVM_DONTCOPYto the VMA.
Yes, madvise() might work.
And setting this flag might be an easy decision, while unsetting it might be hard unless some counters introduced.
Quoted 12 lines · Yan Zhao
This is an RFC series trying to fix the issue of unnecessary NUMA protection and TLB-shootdowns found in VMs with assigned devices or VFIO mediated devices during NUMA balance.
For VMs with assigned devices or VFIO mediated devices, all or part of guest memory are pinned for long-term.
Auto NUMA balancing will periodically selects VMAs of a process and change protections to
PROT_NONEeven though some or all pages in the selected ranges are long-term pinned for DMAs, which is true for VMs with assigned devices or VFIO mediated devices.
Quoted 5 lines
Though this will not cause real problem because NUMA migration will ultimately reject migration of those kind of pages and restore those
PROT_NONEPTEs, it causes KVM's secondary MMU to be zapped periodically with equal SPTEs finally faulted back, wasting CPU cycles and generating unnecessary TLB-shootdowns.
In my understanding, NUMA balancing also moves tasks closer to the memory they are accessing. Can this still work with this series applied?
Quoted 48 lines
This series first introduces a new flag
MMU_NOTIFIER_RANGE_NUMAin patch 1 to work with mmu notifier event typeMMU_NOTIFY_PROTECTION_VMA, so that the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation event is sent for NUMA migration purpose in specific.Patch 2 skips setting
PROT_NONEto long-term pinned pages in the primary MMU to avoid NUMA protection introduced page faults and restoration of old huge PMDs/PTEs in primary MMU.Patch 3 introduces a new mmu notifier callback .numa_protect(), which will be called in patch 4 when a page is ensured to be
PROT_NONEprotected.Then in patch 5, KVM can recognize a .invalidate_range_start() notification is for NUMA balancing specific and do not do the page unmap in secondary MMU until .numa_protect() comes.
Changelog:
RFC v1 --> v2:
1. added patch 3-4 to introduce a new callback .numa_protect()
2. Rather than have KVM duplicate logic to check if a page is pinned for long-term, let KVM depend on the new callback .numa_protect() to do the page unmap in secondary MMU for NUMA migration purpose.RFC v1:
https://lore.kernel.org/all/[email protected]/Yan Zhao (5):
mm/mmu_notifier: introduce a new mmu notifier flag
MMU_NOTIFIER_RANGE_NUMA
mm: don't setPROT_NONEto maybe-dma-pinned pages for NUMA-migrate
purpose
mm/mmu_notifier: introduce a new callback .numa_protect mm/autonuma: call .numa_protect() when page is protected for NUMA
migrate
KVM: Unmap pages only when it's indeed protected for NUMA migrationinclude/linux/mmu_notifier.h | 16 ++++++++++++++++ mm/huge_memory.c | 6 ++++++ mm/mmu_notifier.c | 18 ++++++++++++++++++ mm/mprotect.c | 10 +++++++++- virt/kvm/kvm_main.c | 25 ++++++++++++++++++++++--- 5 files changed, 71 insertions(+), 4 deletions(-)Signature · 1 line
2.17.1
Quoted 13 lines · Chao Gao
Quoted 12 lines · Yan Zhao
This is an RFC series trying to fix the issue of unnecessary NUMA protection and TLB-shootdowns found in VMs with assigned devices or VFIO mediated devices during NUMA balance.
For VMs with assigned devices or VFIO mediated devices, all or part of guest memory are pinned for long-term.
Auto NUMA balancing will periodically selects VMAs of a process and change protections to
PROT_NONEeven though some or all pages in the selected ranges are long-term pinned for DMAs, which is true for VMs with assigned devices or VFIO mediated devices.
Quoted 9 lines
Quoted 5 lines
Though this will not cause real problem because NUMA migration will ultimately reject migration of those kind of pages and restore those
PROT_NONEPTEs, it causes KVM's secondary MMU to be zapped periodically with equal SPTEs finally faulted back, wasting CPU cycles and generating unnecessary TLB-shootdowns.In my understanding, NUMA balancing also moves tasks closer to the memory they are accessing. Can this still work with this series applied?
For pages protected with PROT_NONE in primary MMU in scanning phase, yes; For pages not set to PROT_NONE, no.
Because looks this task_numa_migrate() is only triggered in next page fault when PROT_NONE and accessible VMA is found.