Skip to contentSkip to conversation

Based on latest mm-unstable (85b44c25cd1e).

PATCH In review James Houghton 6 participants 126 messages Jan 5, 2023 – Feb 9, 2023 Page 2 of 2

Subsystems (31)
lore
Share & tools
lore
#25 lore raw

PATCH 17/46 hugetlb: make unmapping compatible with high-granularity mappings

In reply to James Houghton

Enlighten __unmap_hugepage_range to deal with high-granularity mappings. This doesn't change its API; it still must be called with hugepage alignment, but it will correctly unmap hugepages that have been mapped at high granularity.

The rules for mapcount and refcount here are:
1. Refcount and mapcount are tracked on the head page.
2. Each page table mapping into some of an hpage will increase that hpage's mapcount and refcount by 1.

Eventually, functionality here can be expanded to allow users to call MADV_DONTNEED on PAGE_SIZE-aligned sections of a hugepage, but that is not done here.

---
 include/asm-generic/tlb.h |  6 ++--
 mm/hugetlb.c              | 74 ++++++++++++++++++++++++---------------
 2 files changed, 48 insertions(+), 32 deletions(-)
include/asm-generic/tlb.h +3 -3
index b46617207c93..31267471760e 100644
@@ -598,9 +598,9 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb,
 		__tlb_remove_tlb_entry(tlb, ptep, address);	\
 	} while (0)
 
-#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address)	\
+#define tlb_remove_huge_tlb_entry(tlb, hpte, address)	\
 	do {							\
-		unsigned long _sz = huge_page_size(h);		\
+		unsigned long _sz = hugetlb_pte_size(&hpte);	\
 		if (_sz >= P4D_SIZE)				\
 			tlb_flush_p4d_range(tlb, address, _sz);	\
 		else if (_sz >= PUD_SIZE)			\
@@ -609,7 +609,7 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb,
 			tlb_flush_pmd_range(tlb, address, _sz);	\
 		else						\
 			tlb_flush_pte_range(tlb, address, _sz);	\
-		__tlb_remove_tlb_entry(tlb, ptep, address);	\
+		__tlb_remove_tlb_entry(tlb, hpte.ptep, address);\
 	} while (0)
 
 /**
mm/hugetlb.c +45 -29

139 lines not on this page · raw message

Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#26 lore raw

PATCH 18/46 hugetlb: add HGM support for hugetlb_change_protection

In reply to James Houghton

The main change here is to do a high-granularity walk and pulling the shift from the walk (not from the hstate).

---
 mm/hugetlb.c | 59 +++++++++++++++++++++++++++++++++-------------------
 1 file changed, 38 insertions(+), 21 deletions(-)
mm/hugetlb.c +38 -21
index dfd6c1491ac3..73672d806172 100644
@@ -6798,15 +6798,15 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long start = address;
-	pte_t *ptep;
 	pte_t pte;
 	struct hstate *h = hstate_vma(vma);
-	unsigned long pages = 0, psize = huge_page_size(h);
+	unsigned long base_pages = 0, psize = huge_page_size(h);
 	bool shared_pmd = false;
 	struct mmu_notifier_range range;
 	unsigned long last_addr_mask;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
 	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
+	struct hugetlb_pte hpte;
 
 	/*
 	 * In the case of shared PMDs, the area to flush could be beyond
@@ -6824,28 +6824,30 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 	hugetlb_vma_lock_write(vma);
 	i_mmap_lock_write(vma->vm_file->f_mapping);
 	last_addr_mask = hugetlb_mask_last_page(h);
-	for (; address < end; address += psize) {
+	while (address < end) {
 		spinlock_t *ptl;
-		ptep = hugetlb_walk(vma, address, psize);
-		if (!ptep) {
-			address |= last_addr_mask;
+
+		if (hugetlb_full_walk(&hpte, vma, address)) {
+			address = (address | last_addr_mask) + psize;
 			continue;
 		}
-		ptl = huge_pte_lock(h, mm, ptep);
-		if (huge_pmd_unshare(mm, vma, address, ptep)) {
+
+		ptl = hugetlb_pte_lock(&hpte);
+		if (hugetlb_pte_size(&hpte) == psize &&
+		    huge_pmd_unshare(mm, vma, address, hpte.ptep)) {
 			/*
 			 * When uffd-wp is enabled on the vma, unshare
 			 * shouldn't happen at all.  Warn about it if it
 			 * happened due to some reason.
 			 */
 			WARN_ON_ONCE(uffd_wp || uffd_wp_resolve);
-			pages++;
+			base_pages += psize / PAGE_SIZE;
 			spin_unlock(ptl);
 			shared_pmd = true;
-			address |= last_addr_mask;
+			address = (address | last_addr_mask) + psize;
 			continue;
 		}
-		pte = huge_ptep_get(ptep);
+		pte = huge_ptep_get(hpte.ptep);
 		if (unlikely(is_hugetlb_entry_hwpoisoned(pte))) {
 			/* Nothing to do. */
 		} else if (unlikely(is_hugetlb_entry_migration(pte))) {
@@ -6861,7 +6863,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 					entry = make_readable_migration_entry(
 								swp_offset(entry));
 				newpte = swp_entry_to_pte(entry);
-				pages++;
+				base_pages += hugetlb_pte_size(&hpte) / PAGE_SIZE;
 			}
 
 			if (uffd_wp)
@@ -6869,34 +6871,49 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 			else if (uffd_wp_resolve)
 				newpte = pte_swp_clear_uffd_wp(newpte);
 			if (!pte_same(pte, newpte))
-				set_huge_pte_at(mm, address, ptep, newpte);
+				set_huge_pte_at(mm, address, hpte.ptep, newpte);
 		} else if (unlikely(is_pte_marker(pte))) {
 			/* No other markers apply for now. */
 			WARN_ON_ONCE(!pte_marker_uffd_wp(pte));
 			if (uffd_wp_resolve)
 				/* Safe to modify directly (non-present->none). */
-				huge_pte_clear(mm, address, ptep, psize);
+				huge_pte_clear(mm, address, hpte.ptep,
+						hugetlb_pte_size(&hpte));
 		} else if (!huge_pte_none(pte)) {
 			pte_t old_pte;
-			unsigned int shift = huge_page_shift(hstate_vma(vma));
+			unsigned int shift = hpte.shift;
 
-			old_pte = huge_ptep_modify_prot_start(vma, address, ptep);
+			if (unlikely(!hugetlb_pte_present_leaf(&hpte, pte))) {
+				/*
+				 * Someone split the PTE from under us, so retry
+				 * the walk,
+				 */
+				spin_unlock(ptl);
+				continue;
+			}
+
+			old_pte = huge_ptep_modify_prot_start(
+					vma, address, hpte.ptep);
 			pte = huge_pte_modify(old_pte, newprot);
-			pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
+			pte = arch_make_huge_pte(
+					pte, shift, vma->vm_flags);
 			if (uffd_wp)
 				pte = huge_pte_mkuffd_wp(pte);
 			else if (uffd_wp_resolve)
 				pte = huge_pte_clear_uffd_wp(pte);
-			huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte);
-			pages++;
+			huge_ptep_modify_prot_commit(
+					vma, address, hpte.ptep,
+					old_pte, pte);
+			base_pages += hugetlb_pte_size(&hpte) / PAGE_SIZE;
 		} else {
 			/* None pte */
 			if (unlikely(uffd_wp))
 				/* Safe to modify directly (none->non-present). */
-				set_huge_pte_at(mm, address, ptep,
+				set_huge_pte_at(mm, address, hpte.ptep,
 						make_pte_marker(PTE_MARKER_UFFD_WP));
 		}
 		spin_unlock(ptl);
+		address += hugetlb_pte_size(&hpte);
 	}
 	/*
 	 * Must flush TLB before releasing i_mmap_rwsem: x86's huge_pmd_unshare
@@ -6919,7 +6936,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
 	hugetlb_vma_unlock_write(vma);
 	mmu_notifier_invalidate_range_end(&range);
 
-	return pages << h->order;
+	return base_pages;
 }
 
 /* Return true if reservation was successful, false otherwise.  */
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#27 lore raw

PATCH 19/46 hugetlb: add HGM support for follow_hugetlb_page

In reply to James Houghton

This enables high-granularity mapping support in GUP.

In case it is confusing, pfn_offset is the offset (in PAGE_SIZE units) that vaddr points to within the subpage that hpte points to.

---
 mm/hugetlb.c | 59 ++++++++++++++++++++++++++++++++--------------------
 1 file changed, 37 insertions(+), 22 deletions(-)
mm/hugetlb.c +37 -22
index 73672d806172..30fea414d9ee 100644
@@ -6532,11 +6532,9 @@ static void record_subpages_vmas(struct page *page, struct vm_area_struct *vma,
 }
 
 static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma,
-					       unsigned int flags, pte_t *pte,
+					       unsigned int flags, pte_t pteval,
 					       bool *unshare)
 {
-	pte_t pteval = huge_ptep_get(pte);
-
 	*unshare = false;
 	if (is_swap_pte(pteval))
 		return true;
@@ -6611,11 +6609,13 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	int err = -EFAULT, refs;
 
 	while (vaddr < vma->vm_end && remainder) {
-		pte_t *pte;
+		pte_t *ptep, pte;
 		spinlock_t *ptl = NULL;
 		bool unshare = false;
 		int absent;
-		struct page *page;
+		unsigned long pages_per_hpte;
+		struct page *page, *subpage;
+		struct hugetlb_pte hpte;
 
 		/*
 		 * If we have a pending SIGKILL, don't keep faulting pages and
@@ -6632,13 +6632,19 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		 * each hugepage.  We have to make sure we get the
 		 * first, for the page indexing below to work.
 		 *
-		 * Note that page table lock is not held when pte is null.
+		 * hugetlb_full_walk will mask the address appropriately.
+		 *
+		 * Note that page table lock is not held when ptep is null.
 		 */
-		pte = hugetlb_walk(vma, vaddr & huge_page_mask(h),
-				   huge_page_size(h));
-		if (pte)
-			ptl = huge_pte_lock(h, mm, pte);
-		absent = !pte || huge_pte_none(huge_ptep_get(pte));
+		if (hugetlb_full_walk(&hpte, vma, vaddr)) {
+			ptep = NULL;
+			absent = true;
+		} else {
+			ptl = hugetlb_pte_lock(&hpte);
+			ptep = hpte.ptep;
+			pte = huge_ptep_get(ptep);
+			absent = huge_pte_none(pte);
+		}
 
 		/*
 		 * When coredumping, it suits get_dump_page if we just return
@@ -6649,13 +6655,20 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		 */
 		if (absent && (flags & FOLL_DUMP) &&
 		    !hugetlbfs_pagecache_present(h, vma, vaddr)) {
-			if (pte)
+			if (ptep)
 				spin_unlock(ptl);
 			hugetlb_vma_unlock_read(vma);
 			remainder = 0;
 			break;
 		}
 
+		if (!absent && pte_present(pte) &&
+				!hugetlb_pte_present_leaf(&hpte, pte)) {
+			/* We raced with someone splitting the PTE, so retry. */
+			spin_unlock(ptl);
+			continue;
+		}
+
 		/*
 		 * We need call hugetlb_fault for both hugepages under migration
 		 * (in which case hugetlb_fault waits for the migration,) and
@@ -6671,7 +6684,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 			vm_fault_t ret;
 			unsigned int fault_flags = 0;
 
-			if (pte)
+			if (ptep)
 				spin_unlock(ptl);
 			hugetlb_vma_unlock_read(vma);
 
@@ -6720,8 +6733,10 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 			continue;
 		}
 
-		pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT;
-		page = pte_page(huge_ptep_get(pte));
+		pfn_offset = (vaddr & ~hugetlb_pte_mask(&hpte)) >> PAGE_SHIFT;
+		subpage = pte_page(pte);
+		pages_per_hpte = hugetlb_pte_size(&hpte) / PAGE_SIZE;
+		page = compound_head(subpage);
 
 		VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) &&
 			       !PageAnonExclusive(page), page);
@@ -6731,22 +6746,22 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		 * and skip the same_page loop below.
 		 */
 		if (!pages && !vmas && !pfn_offset &&
-		    (vaddr + huge_page_size(h) < vma->vm_end) &&
-		    (remainder >= pages_per_huge_page(h))) {
-			vaddr += huge_page_size(h);
-			remainder -= pages_per_huge_page(h);
-			i += pages_per_huge_page(h);
+		    (vaddr + pages_per_hpte < vma->vm_end) &&
+		    (remainder >= pages_per_hpte)) {
+			vaddr += pages_per_hpte;
+			remainder -= pages_per_hpte;
+			i += pages_per_hpte;
 			spin_unlock(ptl);
 			hugetlb_vma_unlock_read(vma);
 			continue;
 		}
 
 		/* vaddr may not be aligned to PAGE_SIZE */
-		refs = min3(pages_per_huge_page(h) - pfn_offset, remainder,
+		refs = min3(pages_per_hpte - pfn_offset, remainder,
 		    (vma->vm_end - ALIGN_DOWN(vaddr, PAGE_SIZE)) >> PAGE_SHIFT);
 
 		if (pages || vmas)
-			record_subpages_vmas(nth_page(page, pfn_offset),
+			record_subpages_vmas(nth_page(subpage, pfn_offset),
 					     vma, refs,
 					     likely(pages) ? pages + i : NULL,
 					     vmas ? vmas + i : NULL);
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
reviewer
#28 lore raw

Re: [PATCH 19/46] hugetlb: add HGM support for follow_hugetlb_page

Quoted 6 lines · James Houghton
mm/hugetlb.c +1 -1
@@ -6649,13 +6655,20 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		 */
 		if (absent && (flags & FOLL_DUMP) &&
 		    !hugetlbfs_pagecache_present(h, vma, vaddr)) {
-			if (pte)
+			if (ptep)

mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 6668

Quoted 10 lines
mm/hugetlb.c +4 -0
 				spin_unlock(ptl);
 			hugetlb_vma_unlock_read(vma);
 			remainder = 0;
 			break;
 		}
 
+		if (!absent && pte_present(pte) &&
+				!hugetlb_pte_present_leaf(&hpte, pte)) {
+			/* We raced with someone splitting the PTE, so retry. */
+			spin_unlock(ptl);

vma unlock missing here.

Quoted 2 lines
mm/hugetlb.c +2 -0
+			continue;
+		}
Signature · 1 line
Peter Xu
reviewer
#29 lore raw

Re: [PATCH 19/46] hugetlb: add HGM support for follow_hugetlb_page

In reply to James Houghton

mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 6751

Quoted 12 lines · James Houghton
mm/hugetlb.c +3 -5
@@ -6731,22 +6746,22 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 		 * and skip the same_page loop below.
 		 */
 		if (!pages && !vmas && !pfn_offset &&
-		    (vaddr + huge_page_size(h) < vma->vm_end) &&
-		    (remainder >= pages_per_huge_page(h))) {
-			vaddr += huge_page_size(h);
-			remainder -= pages_per_huge_page(h);
-			i += pages_per_huge_page(h);
+		    (vaddr + pages_per_hpte < vma->vm_end) &&
+		    (remainder >= pages_per_hpte)) {
+			vaddr += pages_per_hpte;

This silently breaks hugetlb GUP.. should be

                        vaddr += hugetlb_pte_size(&hpte);

It caused misterious MISSING events when I'm playing with this tree, and I'm surprised it rooted here. So far the most time consuming one. :)

Quoted 6 lines
mm/hugetlb.c +2 -0
+			remainder -= pages_per_hpte;
+			i += pages_per_hpte;
 			spin_unlock(ptl);
 			hugetlb_vma_unlock_read(vma);
 			continue;
 		}
Signature · 1 line
Peter Xu
#30 lore raw

Re: [PATCH 19/46] hugetlb: add HGM support for follow_hugetlb_page

Quoted 17 lines · Peter Xu

mm/hugetlb.c long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, · line 6751

Quoted 12 lines · James Houghton
mm/hugetlb.c +3 -5
@@ -6731,22 +6746,22 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
               * and skip the same_page loop below.
               */
              if (!pages && !vmas && !pfn_offset &&
-                 (vaddr + huge_page_size(h) < vma->vm_end) &&
-                 (remainder >= pages_per_huge_page(h))) {
-                     vaddr += huge_page_size(h);
-                     remainder -= pages_per_huge_page(h);
-                     i += pages_per_huge_page(h);
+                 (vaddr + pages_per_hpte < vma->vm_end) &&
+                 (remainder >= pages_per_hpte)) {
+                     vaddr += pages_per_hpte;

This silently breaks hugetlb GUP.. should be

Quoted 4 lines
                        vaddr += hugetlb_pte_size(&hpte);

It caused misterious MISSING events when I'm playing with this tree, and I'm surprised it rooted here. So far the most time consuming one. :)

Thanks Peter!! And the vaddr + pages_per_hpte < vma->vm_end should be vaddr + hugetlb_pte_size(&hpte) < vma->vm_end too.

#31 lore raw

PATCH 20/46 hugetlb: add HGM support for hugetlb_follow_page_mask

In reply to James Houghton

The change here is very simple: do a high-granularity walk.

---
 mm/hugetlb.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)
mm/hugetlb.c +17 -7
index 30fea414d9ee..718572444a73 100644
@@ -6553,11 +6553,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,
 				unsigned long address, unsigned int flags)
 {
 	struct hstate *h = hstate_vma(vma);
-	struct mm_struct *mm = vma->vm_mm;
-	unsigned long haddr = address & huge_page_mask(h);
 	struct page *page = NULL;
 	spinlock_t *ptl;
-	pte_t *pte, entry;
+	pte_t entry;
+	struct hugetlb_pte hpte;
 
 	/*
 	 * FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via
@@ -6567,13 +6566,24 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma,
 		return NULL;
 
 	hugetlb_vma_lock_read(vma);
-	pte = hugetlb_walk(vma, haddr, huge_page_size(h));
-	if (!pte)
+
+	if (hugetlb_full_walk(&hpte, vma, address))
 		goto out_unlock;
 
-	ptl = huge_pte_lock(h, mm, pte);
-	entry = huge_ptep_get(pte);
+retry:
+	ptl = hugetlb_pte_lock(&hpte);
+	entry = huge_ptep_get(hpte.ptep);
 	if (pte_present(entry)) {
+		if (unlikely(!hugetlb_pte_present_leaf(&hpte, entry))) {
+			/*
+			 * We raced with someone splitting from under us.
+			 * Keep walking to get to the real leaf.
+			 */
+			spin_unlock(ptl);
+			hugetlb_full_walk_continue(&hpte, vma, address);
+			goto retry;
+		}
+
 		page = pte_page(entry) +
 				((address & ~huge_page_mask(h)) >> PAGE_SHIFT);
 		/*
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#32 lore raw

PATCH 21/46 hugetlb: use struct hugetlb_pte for walk_hugetlb_range

In reply to James Houghton

The main change in this commit is to walk_hugetlb_range to support walking HGM mappings, but all walk_hugetlb_range callers must be updated to use the new API and take the correct action.

Listing all the changes to the callers:

For s390 changes, we simply ignore HGM PTEs (we don't support s390 yet).

For smaps, shared_hugetlb (and private_hugetlb, although private mappings don't support HGM) may now not be divisible by the hugepage size. The appropriate changes have been made to support analyzing HGM PTEs.

For pagemap, we ignore non-leaf PTEs by treating that as if they were none PTEs. We can only end up with non-leaf PTEs if they had just been updated from a none PTE.

For show_numa_map, the challenge is that, if any of a hugepage is mapped, we have to count that entire page exactly once, as the results are given in units of hugepages. To support HGM mappings, we keep track of the last page that we looked it. If the hugepage we are currently looking at is the same as the last one, then we must be looking at an HGM-mapped page that has been mapped at high-granularity, and we've already accounted for it.

For DAMON, we treat non-leaf PTEs as if they were blank, for the same reason as pagemap.

For hwpoison, we proactively update the logic to support the case when hpte is pointing to a subpage within the poisoned hugepage.

For queue_pages_hugetlb/migration, we ignore all HGM-enabled VMAs for now.

For mincore, we ignore non-leaf PTEs for the same reason as pagemap.

For mprotect/prot_none_hugetlb_entry, we retry the walk when we get a non-leaf PTE.

---
 arch/s390/mm/gmap.c      | 20 ++++++++--
 fs/proc/task_mmu.c       | 83 +++++++++++++++++++++++++++++-----------
 include/linux/pagewalk.h | 10 +++--
 mm/damon/vaddr.c         | 42 +++++++++++++-------
 mm/hmm.c                 | 20 +++++++---
 mm/memory-failure.c      | 17 ++++----
 mm/mempolicy.c           | 12 ++++--
 mm/mincore.c             | 17 ++++++--
 mm/mprotect.c            | 18 ++++++---
 mm/pagewalk.c            | 20 +++++-----
 10 files changed, 180 insertions(+), 79 deletions(-)
arch/s390/mm/gmap.c +16 -4
index 74e1d873dce0..284466bf4f25 100644
@@ -2626,13 +2626,25 @@ static int __s390_enable_skey_pmd(pmd_t *pmd, unsigned long addr,
 	return 0;
 }
 
-static int __s390_enable_skey_hugetlb(pte_t *pte, unsigned long addr,
-				      unsigned long hmask, unsigned long next,
+static int __s390_enable_skey_hugetlb(struct hugetlb_pte *hpte,
+				      unsigned long addr,
 				      struct mm_walk *walk)
 {
-	pmd_t *pmd = (pmd_t *)pte;
+	struct hstate *h = hstate_vma(walk->vma);
+	pmd_t *pmd;
 	unsigned long start, end;
-	struct page *page = pmd_page(*pmd);
+	struct page *page;
+
+	if (huge_page_size(h) != hugetlb_pte_size(hpte))
+		/* Ignore high-granularity PTEs. */
+		return 0;
+
+	if (!pte_present(huge_ptep_get(hpte->ptep)))
+		/* Ignore non-present PTEs. */
+		return 0;
+
+	pmd = (pmd_t *)hpte->ptep;
+	page = pmd_page(*pmd);
 
 	/*
 	 * The write check makes sure we do not set a key on shared
fs/proc/task_mmu.c +61 -22

167 lines not on this page · raw message

include/linux/pagewalk.h +7 -3
index 27a6df448ee5..f4bddad615c2 100644
@@ -3,6 +3,7 @@
 #define _LINUX_PAGEWALK_H
 
 #include <linux/mm.h>
+#include <linux/hugetlb.h>
 
 struct mm_walk;
 
@@ -31,6 +32,10 @@ struct mm_walk;
  *			ptl after dropping the vma lock, or else revalidate
  *			those items after re-acquiring the vma lock and before
  *			accessing them.
+ *			In the presence of high-granularity hugetlb entries,
+ *			@hugetlb_entry is called only for leaf-level entries
+ *			(hstate-level entries are ignored if they are not
+ *			leaves).
  * @test_walk:		caller specific callback function to determine whether
  *			we walk over the current vma or not. Returning 0 means
  *			"do page table walk over the current vma", returning
@@ -58,9 +63,8 @@ struct mm_walk_ops {
 			 unsigned long next, struct mm_walk *walk);
 	int (*pte_hole)(unsigned long addr, unsigned long next,
 			int depth, struct mm_walk *walk);
-	int (*hugetlb_entry)(pte_t *pte, unsigned long hmask,
-			     unsigned long addr, unsigned long next,
-			     struct mm_walk *walk);
+	int (*hugetlb_entry)(struct hugetlb_pte *hpte,
+			     unsigned long addr, struct mm_walk *walk);
 	int (*test_walk)(unsigned long addr, unsigned long next,
 			struct mm_walk *walk);
 	int (*pre_vma)(unsigned long start, unsigned long end,
mm/damon/vaddr.c +28 -14

90 lines not on this page · raw message

mm/hmm.c +14 -6
index 6a151c09de5e..d3e40cfdd4cb 100644
@@ -468,8 +468,8 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end,
 #endif
 
 #ifdef CONFIG_HUGETLB_PAGE
-static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask,
-				      unsigned long start, unsigned long end,
+static int hmm_vma_walk_hugetlb_entry(struct hugetlb_pte *hpte,
+				      unsigned long start,
 				      struct mm_walk *walk)
 {
 	unsigned long addr = start, i, pfn;
@@ -479,16 +479,24 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask,
 	unsigned int required_fault;
 	unsigned long pfn_req_flags;
 	unsigned long cpu_flags;
+	unsigned long hmask = hugetlb_pte_mask(hpte);
+	unsigned int order = hpte->shift - PAGE_SHIFT;
+	unsigned long end = start + hugetlb_pte_size(hpte);
 	spinlock_t *ptl;
 	pte_t entry;
 
-	ptl = huge_pte_lock(hstate_vma(vma), walk->mm, pte);
-	entry = huge_ptep_get(pte);
+	ptl = hugetlb_pte_lock(hpte);
+	entry = huge_ptep_get(hpte->ptep);
+
+	if (!hugetlb_pte_present_leaf(hpte, entry)) {
+		spin_unlock(ptl);
+		return -EAGAIN;
+	}
 
 	i = (start - range->start) >> PAGE_SHIFT;
 	pfn_req_flags = range->hmm_pfns[i];
 	cpu_flags = pte_to_hmm_pfn_flags(range, entry) |
-		    hmm_pfn_flags_order(huge_page_order(hstate_vma(vma)));
+		    hmm_pfn_flags_order(order);
 	required_fault =
 		hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, cpu_flags);
 	if (required_fault) {
@@ -605,7 +613,7 @@ int hmm_range_fault(struct hmm_range *range)
 		 * in pfns. All entries < last in the pfn array are set to their
 		 * output, and all >= are still at their input values.
 		 */
-	} while (ret == -EBUSY);
+	} while (ret == -EBUSY || ret == -EAGAIN);
 	return ret;
 }
 EXPORT_SYMBOL(hmm_range_fault);
mm/memory-failure.c +9 -8
index c77a9e37e27e..e7e56298d305 100644
@@ -641,6 +641,7 @@ static int check_hwpoisoned_entry(pte_t pte, unsigned long addr, short shift,
 				unsigned long poisoned_pfn, struct to_kill *tk)
 {
 	unsigned long pfn = 0;
+	unsigned long base_pages_poisoned = (1UL << shift) / PAGE_SIZE;
 
 	if (pte_present(pte)) {
 		pfn = pte_pfn(pte);
@@ -651,7 +652,8 @@ static int check_hwpoisoned_entry(pte_t pte, unsigned long addr, short shift,
 			pfn = swp_offset_pfn(swp);
 	}
 
-	if (!pfn || pfn != poisoned_pfn)
+	if (!pfn || pfn < poisoned_pfn ||
+			pfn >= poisoned_pfn + base_pages_poisoned)
 		return 0;
 
 	set_to_kill(tk, addr, shift);
@@ -717,16 +719,15 @@ static int hwpoison_pte_range(pmd_t *pmdp, unsigned long addr,
 }
 
 #ifdef CONFIG_HUGETLB_PAGE
-static int hwpoison_hugetlb_range(pte_t *ptep, unsigned long hmask,
-			    unsigned long addr, unsigned long end,
-			    struct mm_walk *walk)
+static int hwpoison_hugetlb_range(struct hugetlb_pte *hpte,
+				  unsigned long addr,
+				  struct mm_walk *walk)
 {
 	struct hwp_walk *hwp = walk->private;
-	pte_t pte = huge_ptep_get(ptep);
-	struct hstate *h = hstate_vma(walk->vma);
+	pte_t pte = huge_ptep_get(hpte->ptep);
 
-	return check_hwpoisoned_entry(pte, addr, huge_page_shift(h),
-				      hwp->pfn, &hwp->tk);
+	return check_hwpoisoned_entry(pte, addr & hugetlb_pte_mask(hpte),
+			hpte->shift, hwp->pfn, &hwp->tk);
 }
 #else
 #define hwpoison_hugetlb_range	NULL
mm/mempolicy.c +8 -4
index d3558248a0f0..e5859ed34e90 100644
@@ -558,8 +558,8 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,
 	return addr != end ? -EIO : 0;
 }
 
-static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask,
-			       unsigned long addr, unsigned long end,
+static int queue_pages_hugetlb(struct hugetlb_pte *hpte,
+			       unsigned long addr,
 			       struct mm_walk *walk)
 {
 	int ret = 0;
@@ -570,8 +570,12 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask,
 	spinlock_t *ptl;
 	pte_t entry;
 
-	ptl = huge_pte_lock(hstate_vma(walk->vma), walk->mm, pte);
-	entry = huge_ptep_get(pte);
+	/* We don't migrate high-granularity HugeTLB mappings for now. */
+	if (hugetlb_hgm_enabled(walk->vma))
+		return -EINVAL;
+
+	ptl = hugetlb_pte_lock(hpte);
+	entry = huge_ptep_get(hpte->ptep);
 	if (!pte_present(entry))
 		goto unlock;
 	page = pte_page(entry);
mm/mincore.c +14 -3
index a085a2aeabd8..0894965b3944 100644
@@ -22,18 +22,29 @@
 #include <linux/uaccess.h>
 #include "swap.h"
 
-static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,
-			unsigned long end, struct mm_walk *walk)
+static int mincore_hugetlb(struct hugetlb_pte *hpte, unsigned long addr,
+			   struct mm_walk *walk)
 {
 #ifdef CONFIG_HUGETLB_PAGE
 	unsigned char present;
+	unsigned long end = addr + hugetlb_pte_size(hpte);
 	unsigned char *vec = walk->private;
+	pte_t pte = huge_ptep_get(hpte->ptep);
 
 	/*
 	 * Hugepages under user process are always in RAM and never
 	 * swapped out, but theoretically it needs to be checked.
 	 */
-	present = pte && !huge_pte_none(huge_ptep_get(pte));
+	present = !huge_pte_none(pte);
+
+	/*
+	 * If the pte is present but not a leaf, we raced with someone
+	 * splitting it. For someone to have split it, it must have been
+	 * huge_pte_none before, so treat it as such.
+	 */
+	if (pte_present(pte) && !hugetlb_pte_present_leaf(hpte, pte))
+		present = false;
+
 	for (; addr != end; vec++, addr += PAGE_SIZE)
 		*vec = present;
 	walk->private = vec;
mm/mprotect.c +12 -6
index 71358e45a742..62d8c5f7bc92 100644
@@ -543,12 +543,16 @@ static int prot_none_pte_entry(pte_t *pte, unsigned long addr,
 		0 : -EACCES;
 }
 
-static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask,
-				   unsigned long addr, unsigned long next,
+static int prot_none_hugetlb_entry(struct hugetlb_pte *hpte,
+				   unsigned long addr,
 				   struct mm_walk *walk)
 {
-	return pfn_modify_allowed(pte_pfn(*pte), *(pgprot_t *)(walk->private)) ?
-		0 : -EACCES;
+	pte_t pte = huge_ptep_get(hpte->ptep);
+
+	if (!hugetlb_pte_present_leaf(hpte, pte))
+		return -EAGAIN;
+	return pfn_modify_allowed(pte_pfn(pte),
+			*(pgprot_t *)(walk->private)) ? 0 : -EACCES;
 }
 
 static int prot_none_test(unsigned long addr, unsigned long next,
@@ -591,8 +595,10 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	    (newflags & VM_ACCESS_FLAGS) == 0) {
 		pgprot_t new_pgprot = vm_get_page_prot(newflags);
 
-		error = walk_page_range(current->mm, start, end,
-				&prot_none_walk_ops, &new_pgprot);
+		do {
+			error = walk_page_range(current->mm, start, end,
+					&prot_none_walk_ops, &new_pgprot);
+		} while (error == -EAGAIN);
 		if (error)
 			return error;
 	}
mm/pagewalk.c +11 -9
index cb23f8a15c13..05ce242f8b7e 100644
@@ -3,6 +3,7 @@
 #include <linux/highmem.h>
 #include <linux/sched.h>
 #include <linux/hugetlb.h>
+#include <linux/minmax.h>
 
 /*
  * We want to know the real level where a entry is located ignoring any
@@ -296,20 +297,21 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,
 	struct vm_area_struct *vma = walk->vma;
 	struct hstate *h = hstate_vma(vma);
 	unsigned long next;
-	unsigned long hmask = huge_page_mask(h);
-	unsigned long sz = huge_page_size(h);
-	pte_t *pte;
 	const struct mm_walk_ops *ops = walk->ops;
 	int err = 0;
+	struct hugetlb_pte hpte;
 
 	hugetlb_vma_lock_read(vma);
 	do {
-		next = hugetlb_entry_end(h, addr, end);
-		pte = hugetlb_walk(vma, addr & hmask, sz);
-		if (pte)
-			err = ops->hugetlb_entry(pte, hmask, addr, next, walk);
-		else if (ops->pte_hole)
-			err = ops->pte_hole(addr, next, -1, walk);
+		if (hugetlb_full_walk(&hpte, vma, addr)) {
+			next = hugetlb_entry_end(h, addr, end);
+			if (ops->pte_hole)
+				err = ops->pte_hole(addr, next, -1, walk);
+		} else {
+			err = ops->hugetlb_entry(
+					&hpte, addr, walk);
+			next = min(addr + hugetlb_pte_size(&hpte), end);
+		}
 		if (err)
 			break;
 	} while (addr = next, addr != end);
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
reviewer
#33 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

mm/damon/vaddr.c static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, · line 338

Quoted 8 lines · James Houghton
mm/damon/vaddr.c +3 -2
-static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
+static void damon_hugetlb_mkold(struct hugetlb_pte *hpte, pte_t entry,
+				struct mm_struct *mm,
 				struct vm_area_struct *vma, unsigned long addr)
 {
 	bool referenced = false;
-	pte_t entry = huge_ptep_get(pte);
+	pte_t entry = huge_ptep_get(hpte->ptep);

My compiler throws me:

mm/damon/vaddr.c: In function ‘damon_hugetlb_mkold’:
mm/damon/vaddr.c:338:15: error: ‘entry’ redeclared as different kind of symbol

  338 |         pte_t entry = huge_ptep_get(hpte->ptep);
      |               ^~~~~   

I guess this line can just be dropped.

Quoted 3 lines
mm/damon/vaddr.c +0 -0
 	struct folio *folio = pfn_folio(pte_pfn(entry));
 
 	folio_get(folio);
Signature · 1 line
Peter Xu
reviewer
#34 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

In reply to James Houghton

James,

fs/proc/task_mmu.c static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, · line 768

Quoted 11 lines · James Houghton
fs/proc/task_mmu.c +2 -2
@@ -751,9 +761,9 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
 		int mapcount = page_mapcount(page);
 
 		if (mapcount >= 2)
-			mss->shared_hugetlb += huge_page_size(hstate_vma(vma));
+			mss->shared_hugetlb += hugetlb_pte_size(hpte);
 		else
-			mss->private_hugetlb += huge_page_size(hstate_vma(vma));
+			mss->private_hugetlb += hugetlb_pte_size(hpte);
 	}
 	return 0;

One thing interesting I found with hgm right now is mostly everything will be counted as "shared" here, I think it's because mapcount is accounted always to the huge page even if mapped in smaller sizes, so page_mapcount() to a small page should be huge too because the head page mapcount should be huge. I'm curious the reasons behind the mapcount decision.

For example, would that risk overflow with head_compound_mapcount? One 1G page mapping all 4K takes 0.25M counts, while the limit should be 2G for atomic_t. Looks like it's possible.

Btw, are the small page* pointers still needed in the latest HGM design? Is there code taking care of disabling of hugetlb vmemmap optimization for HGM? Or maybe it's not needed anymore for the current design?

Thanks,

Signature · 1 line
Peter Xu
#35 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 22 lines · Peter Xu

James,

fs/proc/task_mmu.c static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, · line 768

Quoted 11 lines · James Houghton
fs/proc/task_mmu.c +2 -2
@@ -751,9 +761,9 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
              int mapcount = page_mapcount(page);
 
              if (mapcount >= 2)
-                     mss->shared_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->shared_hugetlb += hugetlb_pte_size(hpte);
              else
-                     mss->private_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->private_hugetlb += hugetlb_pte_size(hpte);
      }
      return 0;

One thing interesting I found with hgm right now is mostly everything will be counted as "shared" here, I think it's because mapcount is accounted always to the huge page even if mapped in smaller sizes, so page_mapcount() to a small page should be huge too because the head page mapcount should be huge. I'm curious the reasons behind the mapcount decision.

Quoted 3 lines

For example, would that risk overflow with head_compound_mapcount? One 1G page mapping all 4K takes 0.25M counts, while the limit should be 2G for atomic_t. Looks like it's possible.

The original mapcount approach was "if the hstate-level PTE is present, increment the compound_mapcount by 1" (basically "if any of the hugepage is mapped, increment the compound_mapcount by 1"), but this was painful to implement, so I changed it to what it is now (each new PT mapping increments the compound_mapcount by 1). I think you're right, there is absolutely an overflow risk. :( I'm not sure what the best solution is. I could just go back to the old approach.

Regarding when things are accounted in private_hugetlb vs. shared_hugetlb, HGM shouldn't change that, because it only applies to shared mappings (right now anyway). It seems like "private_hugetlb" can include cases where the page is shared but has only one mapping, in which case HGM will change it from "private" to "shared".

Quoted 4 lines

Btw, are the small page* pointers still needed in the latest HGM design? Is there code taking care of disabling of hugetlb vmemmap optimization for HGM? Or maybe it's not needed anymore for the current design?

The hugetlb vmemmap optimization can still be used with HGM, so there is no code to disable it. We don't need small page* pointers either, except for when we're dealing with mapping subpages, like in hugetlb_no_page. Everything else can handle the hugetlb page as a folio.

I hope we can keep compatibility with the vmemmap optimization while solving the mapcount overflow risk.

Thanks Peter.
- James

reviewer
#36 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 23 lines · James Houghton
Quoted 22 lines · Peter Xu

James,

fs/proc/task_mmu.c static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, · line 768

Quoted 11 lines · James Houghton
fs/proc/task_mmu.c +2 -2
@@ -751,9 +761,9 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
              int mapcount = page_mapcount(page);
 
              if (mapcount >= 2)
-                     mss->shared_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->shared_hugetlb += hugetlb_pte_size(hpte);
              else
-                     mss->private_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->private_hugetlb += hugetlb_pte_size(hpte);
      }
      return 0;

One thing interesting I found with hgm right now is mostly everything will be counted as "shared" here, I think it's because mapcount is accounted always to the huge page even if mapped in smaller sizes, so page_mapcount() to a small page should be huge too because the head page mapcount should be huge. I'm curious the reasons behind the mapcount decision.

Quoted 8 lines
Quoted 3 lines

For example, would that risk overflow with head_compound_mapcount? One 1G page mapping all 4K takes 0.25M counts, while the limit should be 2G for atomic_t. Looks like it's possible.

The original mapcount approach was "if the hstate-level PTE is present, increment the compound_mapcount by 1" (basically "if any of the hugepage is mapped, increment the compound_mapcount by 1"), but this was painful to implement,

Any more info here on why it was painful? What is the major blocker?

Quoted 4 lines

so I changed it to what it is now (each new PT mapping increments the compound_mapcount by 1). I think you're right, there is absolutely an overflow risk. :( I'm not sure what the best solution is. I could just go back to the old approach.

No rush on that; let's discuss it thoroughly before doing anything. We have more context than when it was discussed initially in the calls, so maybe a good time to revisit.

Quoted 6 lines

Regarding when things are accounted in private_hugetlb vs. shared_hugetlb, HGM shouldn't change that, because it only applies to shared mappings (right now anyway). It seems like "private_hugetlb" can include cases where the page is shared but has only one mapping, in which case HGM will change it from "private" to "shared".

The two fields are not defined against VM_SHARED, it seems. At least not with current code base.

Let me quote the code again just to be clear:

		int mapcount = page_mapcount(page);    <------------- [1]

		if (mapcount >= 2)
			mss->shared_hugetlb += hugetlb_pte_size(hpte);
		else
			mss->private_hugetlb += hugetlb_pte_size(hpte);

		smaps_hugetlb_hgm_account(mss, hpte);

So that information (for some reason) is only relevant to how many mapcount is there. If we have one 1G page and only two 4K mapped, with the existing logic we should see 8K private_hugetlb while in fact I think it should be 8K shared_hugetlb due to page_mapcount() taking account of both 4K mappings (as they all goes back to the head).

I have no idea whether violating that will be a problem or not, I suppose at least it needs justification if it will be violated, or hopefully it can be kept as-is.

Quoted 6 lines
Quoted 5 lines

Btw, are the small page* pointers still needed in the latest HGM design? Is there code taking care of disabling of hugetlb vmemmap optimization for HGM? Or maybe it's not needed anymore for the current design?

Quoted 8 lines

The hugetlb vmemmap optimization can still be used with HGM, so there is no code to disable it. We don't need small page* pointers either, except for when we're dealing with mapping subpages, like in hugetlb_no_page. Everything else can handle the hugetlb page as a folio.

I hope we can keep compatibility with the vmemmap optimization while solving the mapcount overflow risk.

Yeh that'll be perfect if it works. But afaiu even with your current design (ignoring all the issues on either smaps accounting or overflow risks), we already referenced the small pages, aren't we? See:

static inline int page_mapcount(struct page *page)

{
	int mapcount = atomic_read(&page->_mapcount) + 1;  <-------- here

	if (likely(!PageCompound(page)))
		return mapcount;
	page = compound_head(page);
	return head_compound_mapcount(page) + mapcount;
}

Even if we assume small page->_mapcount should always be zero in this case, we may need to take special care of hugetlb pages in page_mapcount() to not reference it at all. Or I think it's reading random values and some days it can be non-zero.

The other approach is probably using the thp approach. After Hugh's rework on the thp accounting I assumed it would be even cleaner (at least no DoubleMap complexity anymore.. even though I can't say I fully digested the whole history of that). It's all about what's the major challenges of using the same approach there with thp. You may have more knowledge on that aspect so I'd be willing to know.

Thanks,

Signature · 1 line
Peter Xu
#37 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 29 lines · Peter Xu
Quoted 27 lines · James Houghton
Quoted 25 lines · Peter Xu

James,

fs/proc/task_mmu.c static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, · line 768

Quoted 11 lines · James Houghton
fs/proc/task_mmu.c +2 -2
@@ -751,9 +761,9 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
              int mapcount = page_mapcount(page);
 
              if (mapcount >= 2)
-                     mss->shared_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->shared_hugetlb += hugetlb_pte_size(hpte);
              else
-                     mss->private_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->private_hugetlb += hugetlb_pte_size(hpte);
      }
      return 0;

One thing interesting I found with hgm right now is mostly everything will be counted as "shared" here, I think it's because mapcount is accounted always to the huge page even if mapped in smaller sizes, so page_mapcount() to a small page should be huge too because the head page mapcount should be huge. I'm curious the reasons behind the mapcount decision.

For example, would that risk overflow with head_compound_mapcount? One 1G page mapping all 4K takes 0.25M counts, while the limit should be 2G for atomic_t. Looks like it's possible.

Quoted 6 lines
Quoted 4 lines

The original mapcount approach was "if the hstate-level PTE is present, increment the compound_mapcount by 1" (basically "if any of the hugepage is mapped, increment the compound_mapcount by 1"), but this was painful to implement,

Any more info here on why it was painful? What is the major blocker?

The original approach was implemented in RFC v1, but the implementation was broken: the way refcount was handled was wrong; it was incremented once for each new page table mapping. (How? find_lock_page(), called once per hugetlb_no_page/UFFDIO_CONTINUE would increment refcount and we wouldn't drop it, and in __unmap_hugepage_range(), the mmu_gather bits would decrement the refcount once per mapping.)

At the time, I figured the complexity of handling mapcount AND refcount correctly in the original approach would be quite complex, so I switched to the new one.

1. In places that already change the mapcount, check that we're installing the hstate-level PTE, not a high-granularity PTE. Adjust mapcount AND refcount appropriately.
2. In the HGM walking bits, to the caller if we made the hstate-level PTE present. (hugetlb_[pmd,pte]_alloc is the source of truth.) Need to keep track of this until we figure out which page we're allocating PTEs for, then change mapcount/refcount appropriately.
3. In unmapping bits, change mmu_gather/tlb bits to drop refcount only once per hugepage. (This is probably the hardest of these three things to get right.)

Quoted 37 lines
Quoted 5 lines

so I changed it to what it is now (each new PT mapping increments the compound_mapcount by 1). I think you're right, there is absolutely an overflow risk. :( I'm not sure what the best solution is. I could just go back to the old approach.

No rush on that; let's discuss it thoroughly before doing anything. We have more context than when it was discussed initially in the calls, so maybe a good time to revisit.

Quoted 6 lines

Regarding when things are accounted in private_hugetlb vs. shared_hugetlb, HGM shouldn't change that, because it only applies to shared mappings (right now anyway). It seems like "private_hugetlb" can include cases where the page is shared but has only one mapping, in which case HGM will change it from "private" to "shared".

The two fields are not defined against VM_SHARED, it seems. At least not with current code base.

Let me quote the code again just to be clear:

                int mapcount = page_mapcount(page);    <------------- [1]

                if (mapcount >= 2)
                        mss->shared_hugetlb += hugetlb_pte_size(hpte);
                else
                        mss->private_hugetlb += hugetlb_pte_size(hpte);

                smaps_hugetlb_hgm_account(mss, hpte);

So that information (for some reason) is only relevant to how many mapcount is there. If we have one 1G page and only two 4K mapped, with the existing logic we should see 8K private_hugetlb while in fact I think it should be 8K shared_hugetlb due to page_mapcount() taking account of both 4K mappings (as they all goes back to the head).

Quoted 3 lines

I have no idea whether violating that will be a problem or not, I suppose at least it needs justification if it will be violated, or hopefully it can be kept as-is.

Agreed that this is a problem. I'm not sure what should be done here. It seems like the current upstream implementation is incorrect (surely MAP_SHARED with only one mapping should still be shared_hugetlb not private_hugetlb); the check should really be if (vma->vm_flags & VM_MAYSHARE) instead of mapcount >= 2. If that change can be taken, we don't have a problem here.

Quoted 30 lines
Quoted 7 lines
Quoted 6 lines

Btw, are the small page* pointers still needed in the latest HGM design? Is there code taking care of disabling of hugetlb vmemmap optimization for HGM? Or maybe it's not needed anymore for the current design?

Quoted 8 lines

The hugetlb vmemmap optimization can still be used with HGM, so there is no code to disable it. We don't need small page* pointers either, except for when we're dealing with mapping subpages, like in hugetlb_no_page. Everything else can handle the hugetlb page as a folio.

I hope we can keep compatibility with the vmemmap optimization while solving the mapcount overflow risk.

Yeh that'll be perfect if it works. But afaiu even with your current design (ignoring all the issues on either smaps accounting or overflow risks), we already referenced the small pages, aren't we? See:

static inline int page_mapcount(struct page *page)

{
        int mapcount = atomic_read(&page->_mapcount) + 1;  <-------- here

        if (likely(!PageCompound(page)))
                return mapcount;
        page = compound_head(page);
        return head_compound_mapcount(page) + mapcount;
}
Quoted 4 lines

Even if we assume small page->_mapcount should always be zero in this case, we may need to take special care of hugetlb pages in page_mapcount() to not reference it at all. Or I think it's reading random values and some days it can be non-zero.

IIUC, it's ok to read from all the hugetlb subpage structs, you just can't *write* to them (except the first few). The first page of page structs is mapped RW; all the others are mapped RO to a single physical page.

Quoted 7 lines

The other approach is probably using the thp approach. After Hugh's rework on the thp accounting I assumed it would be even cleaner (at least no DoubleMap complexity anymore.. even though I can't say I fully digested the whole history of that). It's all about what's the major challenges of using the same approach there with thp. You may have more knowledge on that aspect so I'd be willing to know.

I need to take a closer look at Hugh's approach to see if we can do it the same way. (I wonder if the 1G THP series has some ideas too.)

A really simple solution could be just to prevent userspace from doing MADV_SPLIT (or, if we are enabling HGM due to hwpoison, ignore the poison) if it could result in a mapcount overflow. For 1G pages, userspace would need 8192 mappings to overflow mapcount/refcount.

#38 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 11 lines

The original approach was implemented in RFC v1, but the implementation was broken: the way refcount was handled was wrong; it was incremented once for each new page table mapping. (How? find_lock_page(), called once per hugetlb_no_page/UFFDIO_CONTINUE would increment refcount and we wouldn't drop it, and in __unmap_hugepage_range(), the mmu_gather bits would decrement the refcount once per mapping.)

At the time, I figured the complexity of handling mapcount AND refcount correctly in the original approach would be quite complex, so I switched to the new one.

Sorry I didn't make this clear... the following steps are how we could correctly implement the original approach.

Quoted 10 lines

1. In places that already change the mapcount, check that we're installing the hstate-level PTE, not a high-granularity PTE. Adjust mapcount AND refcount appropriately.
2. In the HGM walking bits, to the caller if we made the hstate-level PTE present. (hugetlb_[pmd,pte]_alloc is the source of truth.) Need to keep track of this until we figure out which page we're allocating PTEs for, then change mapcount/refcount appropriately.
3. In unmapping bits, change mmu_gather/tlb bits to drop refcount only once per hugepage. (This is probably the hardest of these three things to get right.)

reviewer
#39 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

In reply to James Houghton
Quoted 37 lines · James Houghton
Quoted 29 lines · Peter Xu
Quoted 27 lines · James Houghton
Quoted 25 lines · Peter Xu

James,

fs/proc/task_mmu.c static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, · line 768

Quoted 11 lines · James Houghton
fs/proc/task_mmu.c +2 -2
@@ -751,9 +761,9 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
              int mapcount = page_mapcount(page);
 
              if (mapcount >= 2)
-                     mss->shared_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->shared_hugetlb += hugetlb_pte_size(hpte);
              else
-                     mss->private_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->private_hugetlb += hugetlb_pte_size(hpte);
      }
      return 0;

One thing interesting I found with hgm right now is mostly everything will be counted as "shared" here, I think it's because mapcount is accounted always to the huge page even if mapped in smaller sizes, so page_mapcount() to a small page should be huge too because the head page mapcount should be huge. I'm curious the reasons behind the mapcount decision.

For example, would that risk overflow with head_compound_mapcount? One 1G page mapping all 4K takes 0.25M counts, while the limit should be 2G for atomic_t. Looks like it's possible.

Quoted 6 lines
Quoted 4 lines

The original mapcount approach was "if the hstate-level PTE is present, increment the compound_mapcount by 1" (basically "if any of the hugepage is mapped, increment the compound_mapcount by 1"), but this was painful to implement,

Any more info here on why it was painful? What is the major blocker?

Quoted 7 lines

The original approach was implemented in RFC v1, but the implementation was broken: the way refcount was handled was wrong; it was incremented once for each new page table mapping. (How? find_lock_page(), called once per hugetlb_no_page/UFFDIO_CONTINUE would increment refcount and we wouldn't drop it, and in __unmap_hugepage_range(), the mmu_gather bits would decrement the refcount once per mapping.)

I'm not sure I fully get the point here, but perhaps it's mostly about how hugetlb page cache is managed (in hstate size not PAGE_SIZE)?

static inline struct page *folio_file_page(struct folio *folio, pgoff_t index)

{
	/* HugeTLBfs indexes the page cache in units of hpage_size */
	if (folio_test_hugetlb(folio))
		return &folio->page;
	return folio_page(folio, index & (folio_nr_pages(folio) - 1));
}

I haven't thought through on that either. Is it possible that we switche the pgcache layout to be in PAGE_SIZE granule too when HGM enabled (e.g. a simple scheme is we can fail MADV_SPLIT if hugetlb pgcache contains any page already).

If keep using the same pgcache scheme (hpage_size stepped indexes), find_lock_page() will also easily content on the head page lock, so we may not be able to handle concurrent page faults on small mappings on the same page as efficient as thp.

Quoted 57 lines

At the time, I figured the complexity of handling mapcount AND refcount correctly in the original approach would be quite complex, so I switched to the new one.

1. In places that already change the mapcount, check that we're installing the hstate-level PTE, not a high-granularity PTE. Adjust mapcount AND refcount appropriately.
2. In the HGM walking bits, to the caller if we made the hstate-level PTE present. (hugetlb_[pmd,pte]_alloc is the source of truth.) Need to keep track of this until we figure out which page we're allocating PTEs for, then change mapcount/refcount appropriately.
3. In unmapping bits, change mmu_gather/tlb bits to drop refcount only once per hugepage. (This is probably the hardest of these three things to get right.)

Quoted 37 lines
Quoted 5 lines

so I changed it to what it is now (each new PT mapping increments the compound_mapcount by 1). I think you're right, there is absolutely an overflow risk. :( I'm not sure what the best solution is. I could just go back to the old approach.

No rush on that; let's discuss it thoroughly before doing anything. We have more context than when it was discussed initially in the calls, so maybe a good time to revisit.

Quoted 6 lines

Regarding when things are accounted in private_hugetlb vs. shared_hugetlb, HGM shouldn't change that, because it only applies to shared mappings (right now anyway). It seems like "private_hugetlb" can include cases where the page is shared but has only one mapping, in which case HGM will change it from "private" to "shared".

The two fields are not defined against VM_SHARED, it seems. At least not with current code base.

Let me quote the code again just to be clear:

                int mapcount = page_mapcount(page);    <------------- [1]

                if (mapcount >= 2)
                        mss->shared_hugetlb += hugetlb_pte_size(hpte);
                else
                        mss->private_hugetlb += hugetlb_pte_size(hpte);

                smaps_hugetlb_hgm_account(mss, hpte);

So that information (for some reason) is only relevant to how many mapcount is there. If we have one 1G page and only two 4K mapped, with the existing logic we should see 8K private_hugetlb while in fact I think it should be 8K shared_hugetlb due to page_mapcount() taking account of both 4K mappings (as they all goes back to the head).

Quoted 3 lines

I have no idea whether violating that will be a problem or not, I suppose at least it needs justification if it will be violated, or hopefully it can be kept as-is.

Quoted 6 lines

Agreed that this is a problem. I'm not sure what should be done here. It seems like the current upstream implementation is incorrect (surely MAP_SHARED with only one mapping should still be shared_hugetlb not private_hugetlb); the check should really be if (vma->vm_flags & VM_MAYSHARE) instead of mapcount >= 2. If that change can be taken, we don't have a problem here.

Naoya added relevant code. Not sure whether he'll chim in.

commit 25ee01a2fca02dfb5a3ce316e77910c468108199

Author: Naoya Horiguchi <[email protected]>
Date:   Thu Nov 5 18:47:11 2015 -0800

mm: hugetlb: proc: add hugetlb-related fields to /proc/PID/smaps

In all cases, it'll still be a slight ABI change, so worth considering the effects to existing users.

Quoted 36 lines
Quoted 31 lines
Quoted 8 lines
Quoted 7 lines

Btw, are the small page* pointers still needed in the latest HGM design? Is there code taking care of disabling of hugetlb vmemmap optimization for HGM? Or maybe it's not needed anymore for the current design?

Quoted 8 lines

The hugetlb vmemmap optimization can still be used with HGM, so there is no code to disable it. We don't need small page* pointers either, except for when we're dealing with mapping subpages, like in hugetlb_no_page. Everything else can handle the hugetlb page as a folio.

I hope we can keep compatibility with the vmemmap optimization while solving the mapcount overflow risk.

Yeh that'll be perfect if it works. But afaiu even with your current design (ignoring all the issues on either smaps accounting or overflow risks), we already referenced the small pages, aren't we? See:

static inline int page_mapcount(struct page *page)

{
        int mapcount = atomic_read(&page->_mapcount) + 1;  <-------- here

        if (likely(!PageCompound(page)))
                return mapcount;
        page = compound_head(page);
        return head_compound_mapcount(page) + mapcount;
}
Quoted 4 lines

Even if we assume small page->_mapcount should always be zero in this case, we may need to take special care of hugetlb pages in page_mapcount() to not reference it at all. Or I think it's reading random values and some days it can be non-zero.

Quoted 4 lines

IIUC, it's ok to read from all the hugetlb subpage structs, you just can't *write* to them (except the first few). The first page of page structs is mapped RW; all the others are mapped RO to a single physical page.

I'm not familiar with vmemmap work, but I saw this:

/*
* Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end)
* to the page which @vmemmap_reuse is mapped to, then free the pages
* which the range [@vmemmap_start, @vmemmap_end] is mapped to. */
if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse))

It seems it'll reuse the 1st page of the huge page* rather than backing the rest vmemmap with zero pages. Would that be a problem (as I think some small page* can actually points to the e.g. head page* if referenced)?

Quoted 11 lines
Quoted 8 lines

The other approach is probably using the thp approach. After Hugh's rework on the thp accounting I assumed it would be even cleaner (at least no DoubleMap complexity anymore.. even though I can't say I fully digested the whole history of that). It's all about what's the major challenges of using the same approach there with thp. You may have more knowledge on that aspect so I'd be willing to know.

I need to take a closer look at Hugh's approach to see if we can do it the same way. (I wonder if the 1G THP series has some ideas too.)

https://lore.kernel.org/all/[email protected]/

It's already in the mainline. I think it's mostly internally implemented under the rmap code APIs. For the HGM effort, I'd start with simply passing around compound=false when using the rmap APIs, and see what'll start to fail.

Quoted 5 lines

A really simple solution could be just to prevent userspace from doing MADV_SPLIT (or, if we are enabling HGM due to hwpoison, ignore the poison) if it could result in a mapcount overflow. For 1G pages, userspace would need 8192 mappings to overflow mapcount/refcount.

I'm not sure you can calculate it; consider fork()s along the way when pmd sharing disabled, or whatever reason when the 1G pages mapped at multiple places with more than one mmap()s.

Thanks,

Signature · 1 line
Peter Xu
#40 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 63 lines · Peter Xu
Quoted 37 lines · James Houghton
Quoted 29 lines · Peter Xu
Quoted 27 lines · James Houghton
Quoted 25 lines · Peter Xu

James,

fs/proc/task_mmu.c static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, · line 768

Quoted 11 lines · James Houghton
fs/proc/task_mmu.c +2 -2
@@ -751,9 +761,9 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
              int mapcount = page_mapcount(page);
 
              if (mapcount >= 2)
-                     mss->shared_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->shared_hugetlb += hugetlb_pte_size(hpte);
              else
-                     mss->private_hugetlb += huge_page_size(hstate_vma(vma));
+                     mss->private_hugetlb += hugetlb_pte_size(hpte);
      }
      return 0;

One thing interesting I found with hgm right now is mostly everything will be counted as "shared" here, I think it's because mapcount is accounted always to the huge page even if mapped in smaller sizes, so page_mapcount() to a small page should be huge too because the head page mapcount should be huge. I'm curious the reasons behind the mapcount decision.

For example, would that risk overflow with head_compound_mapcount? One 1G page mapping all 4K takes 0.25M counts, while the limit should be 2G for atomic_t. Looks like it's possible.

Quoted 6 lines
Quoted 4 lines

The original mapcount approach was "if the hstate-level PTE is present, increment the compound_mapcount by 1" (basically "if any of the hugepage is mapped, increment the compound_mapcount by 1"), but this was painful to implement,

Any more info here on why it was painful? What is the major blocker?

Quoted 7 lines

The original approach was implemented in RFC v1, but the implementation was broken: the way refcount was handled was wrong; it was incremented once for each new page table mapping. (How? find_lock_page(), called once per hugetlb_no_page/UFFDIO_CONTINUE would increment refcount and we wouldn't drop it, and in __unmap_hugepage_range(), the mmu_gather bits would decrement the refcount once per mapping.)

I'm not sure I fully get the point here, but perhaps it's mostly about how hugetlb page cache is managed (in hstate size not PAGE_SIZE)?

static inline struct page *folio_file_page(struct folio *folio, pgoff_t index)

{
        /* HugeTLBfs indexes the page cache in units of hpage_size */
        if (folio_test_hugetlb(folio))
                return &folio->page;
        return folio_page(folio, index & (folio_nr_pages(folio) - 1));
}

I haven't thought through on that either. Is it possible that we switche the pgcache layout to be in PAGE_SIZE granule too when HGM enabled (e.g. a simple scheme is we can fail MADV_SPLIT if hugetlb pgcache contains any page already).

Quoted 4 lines

If keep using the same pgcache scheme (hpage_size stepped indexes), find_lock_page() will also easily content on the head page lock, so we may not be able to handle concurrent page faults on small mappings on the same page as efficient as thp.

We keep the pgcache scheme the same: hpage_size stepped indexes. The refcount and mapcount are both stored in the head page. The problems with the implementation in RFC v1 were (1) refcount became much much larger than mapcount, and (2) refcount would have the same overflow problem we're discussing.

You're right -- find_lock_page() in hugetlb_no_page() and hugetlb_mcopy_atomic_pte() will contend on the head page lock, but I think it's possible to improve things here (i.e., replace them with find_get_page() somehow).

Quoted 126 lines
Quoted 58 lines

At the time, I figured the complexity of handling mapcount AND refcount correctly in the original approach would be quite complex, so I switched to the new one.

1. In places that already change the mapcount, check that we're installing the hstate-level PTE, not a high-granularity PTE. Adjust mapcount AND refcount appropriately.
2. In the HGM walking bits, to the caller if we made the hstate-level PTE present. (hugetlb_[pmd,pte]_alloc is the source of truth.) Need to keep track of this until we figure out which page we're allocating PTEs for, then change mapcount/refcount appropriately.
3. In unmapping bits, change mmu_gather/tlb bits to drop refcount only once per hugepage. (This is probably the hardest of these three things to get right.)

Quoted 37 lines
Quoted 5 lines

so I changed it to what it is now (each new PT mapping increments the compound_mapcount by 1). I think you're right, there is absolutely an overflow risk. :( I'm not sure what the best solution is. I could just go back to the old approach.

No rush on that; let's discuss it thoroughly before doing anything. We have more context than when it was discussed initially in the calls, so maybe a good time to revisit.

Quoted 6 lines

Regarding when things are accounted in private_hugetlb vs. shared_hugetlb, HGM shouldn't change that, because it only applies to shared mappings (right now anyway). It seems like "private_hugetlb" can include cases where the page is shared but has only one mapping, in which case HGM will change it from "private" to "shared".

The two fields are not defined against VM_SHARED, it seems. At least not with current code base.

Let me quote the code again just to be clear:

                int mapcount = page_mapcount(page);    <------------- [1]

                if (mapcount >= 2)
                        mss->shared_hugetlb += hugetlb_pte_size(hpte);
                else
                        mss->private_hugetlb += hugetlb_pte_size(hpte);

                smaps_hugetlb_hgm_account(mss, hpte);

So that information (for some reason) is only relevant to how many mapcount is there. If we have one 1G page and only two 4K mapped, with the existing logic we should see 8K private_hugetlb while in fact I think it should be 8K shared_hugetlb due to page_mapcount() taking account of both 4K mappings (as they all goes back to the head).

Quoted 3 lines

I have no idea whether violating that will be a problem or not, I suppose at least it needs justification if it will be violated, or hopefully it can be kept as-is.

Quoted 6 lines

Agreed that this is a problem. I'm not sure what should be done here. It seems like the current upstream implementation is incorrect (surely MAP_SHARED with only one mapping should still be shared_hugetlb not private_hugetlb); the check should really be if (vma->vm_flags & VM_MAYSHARE) instead of mapcount >= 2. If that change can be taken, we don't have a problem here.

Naoya added relevant code. Not sure whether he'll chim in.

commit 25ee01a2fca02dfb5a3ce316e77910c468108199

Author: Naoya Horiguchi <[email protected]>
Date:   Thu Nov 5 18:47:11 2015 -0800

mm: hugetlb: proc: add hugetlb-related fields to /proc/PID/smaps

In all cases, it'll still be a slight ABI change, so worth considering the effects to existing users.

Quoted 36 lines
Quoted 31 lines
Quoted 8 lines
Quoted 7 lines

Btw, are the small page* pointers still needed in the latest HGM design? Is there code taking care of disabling of hugetlb vmemmap optimization for HGM? Or maybe it's not needed anymore for the current design?

Quoted 8 lines

The hugetlb vmemmap optimization can still be used with HGM, so there is no code to disable it. We don't need small page* pointers either, except for when we're dealing with mapping subpages, like in hugetlb_no_page. Everything else can handle the hugetlb page as a folio.

I hope we can keep compatibility with the vmemmap optimization while solving the mapcount overflow risk.

Yeh that'll be perfect if it works. But afaiu even with your current design (ignoring all the issues on either smaps accounting or overflow risks), we already referenced the small pages, aren't we? See:

static inline int page_mapcount(struct page *page)

{
        int mapcount = atomic_read(&page->_mapcount) + 1;  <-------- here

        if (likely(!PageCompound(page)))
                return mapcount;
        page = compound_head(page);
        return head_compound_mapcount(page) + mapcount;
}
Quoted 4 lines

Even if we assume small page->_mapcount should always be zero in this case, we may need to take special care of hugetlb pages in page_mapcount() to not reference it at all. Or I think it's reading random values and some days it can be non-zero.

Quoted 4 lines

IIUC, it's ok to read from all the hugetlb subpage structs, you just can't *write* to them (except the first few). The first page of page structs is mapped RW; all the others are mapped RO to a single physical page.

I'm not familiar with vmemmap work, but I saw this:

/*
* Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end)
* to the page which @vmemmap_reuse is mapped to, then free the pages
* which the range [@vmemmap_start, @vmemmap_end] is mapped to. */
if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse))

Quoted 3 lines

It seems it'll reuse the 1st page of the huge page* rather than backing the rest vmemmap with zero pages. Would that be a problem (as I think some small page* can actually points to the e.g. head page* if referenced)?

It shouldn't be a problem if we don't use _mapcount and always use compound_mapcount, which is how mapcount is handled in this version of this series.

Quoted 13 lines
Quoted 10 lines
Quoted 9 lines

The other approach is probably using the thp approach. After Hugh's rework on the thp accounting I assumed it would be even cleaner (at least no DoubleMap complexity anymore.. even though I can't say I fully digested the whole history of that). It's all about what's the major challenges of using the same approach there with thp. You may have more knowledge on that aspect so I'd be willing to know.

Quoted 2 lines

I need to take a closer look at Hugh's approach to see if we can do it the same way. (I wonder if the 1G THP series has some ideas too.)

Quoted 6 lines

https://lore.kernel.org/all/[email protected]/

It's already in the mainline. I think it's mostly internally implemented under the rmap code APIs. For the HGM effort, I'd start with simply passing around compound=false when using the rmap APIs, and see what'll start to fail.

I'll look into it, but doing it this way will use _mapcount, so we won't be able to use the vmemmap optimization. I think even if we do use Hugh's approach, refcount is still being kept on the head page, so there's still an overflow risk there (but maybe I am misunderstanding).

Quoted 10 lines
Quoted 6 lines

A really simple solution could be just to prevent userspace from doing MADV_SPLIT (or, if we are enabling HGM due to hwpoison, ignore the poison) if it could result in a mapcount overflow. For 1G pages, userspace would need 8192 mappings to overflow mapcount/refcount.

I'm not sure you can calculate it; consider fork()s along the way when pmd sharing disabled, or whatever reason when the 1G pages mapped at multiple places with more than one mmap()s.

Yeah you're right. :(

reviewer
#41 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 5 lines · James Houghton

I'll look into it, but doing it this way will use _mapcount, so we won't be able to use the vmemmap optimization. I think even if we do use Hugh's approach, refcount is still being kept on the head page, so there's still an overflow risk there (but maybe I am misunderstanding).

Could you remind me what's the issue if using refcount on the small pages rather than the head (assuming vmemmap still can be disabled)?

Thanks,

Signature · 1 line
Peter Xu
#42 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 9 lines · Peter Xu
Quoted 5 lines · James Houghton

I'll look into it, but doing it this way will use _mapcount, so we won't be able to use the vmemmap optimization. I think even if we do use Hugh's approach, refcount is still being kept on the head page, so there's still an overflow risk there (but maybe I am misunderstanding).

Could you remind me what's the issue if using refcount on the small pages rather than the head (assuming vmemmap still can be disabled)?

The THP-way of doing things is refcounting on the head page. All folios use a single refcount on the head.

There has to be a pretty good reason to do it differently.

Signature · 3 lines
Thanks,

David / dhildenb
#43 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Peter Xu
Quoted 5 lines · James Houghton

I'll look into it, but doing it this way will use _mapcount, so we won't be able to use the vmemmap optimization. I think even if we do use Hugh's approach, refcount is still being kept on the head page, so there's still an overflow risk there (but maybe I am misunderstanding).

Quoted 7 lines
Quoted 2 lines

Could you remind me what's the issue if using refcount on the small pages rather than the head (assuming vmemmap still can be disabled)?

The THP-way of doing things is refcounting on the head page. All folios use a single refcount on the head.

There has to be a pretty good reason to do it differently.

Peter and I have discussed this a lot offline. There are two main problems here:

1. Refcount overflow

Refcount is always kept on the head page (before and after this series). IIUC, this means that if THPs could be 1G in size, they too would be susceptible to the same potential overflow. How easy is the overflow? [1]

To deal with this, the best solution we've been able to come up with is to check if refcount is > INT_MAX/2 (similar to try_get_page()), and if it is, stop the operation (UFFDIO_CONTINUE or a page fault) from proceeding. In the UFFDIO_CONTINUE case, return ENOMEM. In the page fault cause, return VM_FAULT_SIGBUS (not VM_FAULT_OOM; we don't want to kill a random process).

(So David, I think this answers your question. Refcount should be handled just like THPs.)

2. page_mapcount() API differences

In this series, page_mapcount() returns the total number of page table references for the compound page. For example, if you have a PTE-mapped 2M page (with no other mappings), page_mapcount() for each 4K page will be 512. This is not the same as a THP: page_mapcount() would return 1 for each page. Because of the difference in page_mapcount(), we have 4 problems:

i. Smaps uses page_mapcount() >= 2 to determine if hugetlb memory is "private_hugetlb" or "shared_hugetlb".
ii. Migration with MPOL_MF_MOVE will check page_mapcount() to see if the hugepage is shared or not. Pages that would otherwise be migrated now require MPOL_MF_MOVE_ALL to be migrated.
[Really both of the above are checking how many VMAs are mapping our hugepage.] iii. CoW. This isn't a problem right now because CoW is only possible with MAP_PRIVATE VMAs and HGM can only be enabled for MAP_SHARED VMAs. iv. The hwpoison handling code will check if it successfully unmapped the poisoned page. This isn't a problem right now, as hwpoison will unmap all the mappings for the hugepage, not just the 4K where the poison was found.

Doing it this way allows HGM to remain compatible with the hugetlb vmemmap optimization. None of the above problems strike me as particularly major, but it's unclear to me how important it is to have page_mapcount() have a consistent meaning for hugetlb vs non-hugetlb.

The other way page_mapcount() (let's say the "THP-like way") could be done is like this: increment compound mapcount if we're mapping a hugetlb page normally (e.g., 1G page with a PUD). If we're mapping at high-granularity, increment the mapcount for each 4K page that is getting mapped (e.g., PMD within a 1G page: increment the mapcount for the 512 pages that are now mapped). This yields the same page_mapcount() API we had before, but we lose the hugetlb vmemmap optimization.

We could introduce an API like hugetlb_vma_mapcount() that would, for hugetlb, give us the number of VMAs that map a hugepage, but I don't think people would like this.

I'm curious what others think (Mike, Matthew?). I'm guessing the THP-like way is probably what most people would want, though it would be a real shame to lose the vmemmap optimization.

- James

[1] INT_MAX is 2^31. We increment the refcount once for each 4K mapping in 1G: that's 512 * 512 (2^18). That means we only need 8192 (2^13) VMAs for the same 1G page to overflow refcount. I think it's safe to say that if userspace is doing this, they are attempting to overflow refcount.

#44 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 18 lines · James Houghton
Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Peter Xu
Quoted 5 lines · James Houghton

I'll look into it, but doing it this way will use _mapcount, so we won't be able to use the vmemmap optimization. I think even if we do use Hugh's approach, refcount is still being kept on the head page, so there's still an overflow risk there (but maybe I am misunderstanding).

Quoted 7 lines
Quoted 2 lines

Could you remind me what's the issue if using refcount on the small pages rather than the head (assuming vmemmap still can be disabled)?

The THP-way of doing things is refcounting on the head page. All folios use a single refcount on the head.

There has to be a pretty good reason to do it differently.

Quoted 8 lines

Peter and I have discussed this a lot offline. There are two main problems here:

1. Refcount overflow

Refcount is always kept on the head page (before and after this series). IIUC, this means that if THPs could be 1G in size, they too would be susceptible to the same potential overflow. How easy is the overflow? [1]

Right. You'd need 8k VMAs. With 2 MiB THP you'd need 4096k VMAs. So ~64 processes with 64k VMAs. Not impossible to achieve if one really wants to break the system ...

Side note: a long long time ago, we used to have sub-page refcounts for THP. IIRC, that was even before we had sub-page mapcounts and was used to make COW decisions.

Quoted 7 lines

To deal with this, the best solution we've been able to come up with is to check if refcount is > INT_MAX/2 (similar to try_get_page()), and if it is, stop the operation (UFFDIO_CONTINUE or a page fault) from proceeding. In the UFFDIO_CONTINUE case, return ENOMEM. In the page fault cause, return VM_FAULT_SIGBUS (not VM_FAULT_OOM; we don't want to kill a random process).

You'd have to also make sure that fork() won't do the same. At least with uffd-wp, Peter also added page table copying during fork() for MAP_SHARED mappings, which would have to be handled.

Of course, one can just disallow fork() with any HGM right from the start and keep it all simpler to not open up a can of worms there.

Is it reasonable, to have more than one (or a handful) of VMAs mapping a huge page via a HGM? Restricting it to a single one, would make handling
much easier.

If there is ever demand for more HGM mappings, that whole problem (and complexity) could be dealt with later. ... but I assume it will already be a requirement for VMs (e.g., under QEMU) that share memory with other processes (virtiofsd and friends?)

Quoted 4 lines

(So David, I think this answers your question. Refcount should be handled just like THPs.)

Quoted 8 lines

2. page_mapcount() API differences

In this series, page_mapcount() returns the total number of page table references for the compound page. For example, if you have a PTE-mapped 2M page (with no other mappings), page_mapcount() for each 4K page will be 512. This is not the same as a THP: page_mapcount() would return 1 for each page. Because of the difference in page_mapcount(), we have 4 problems:

IMHO, it would actually be great to just be able to remove the sub-page mapcounts for THP and make it all simpler.

Right now, the sub-page mapcount is mostly required for making COW decisions, but only for accounting purposes IIRC (NR_ANON_THPS, NR_SHMEM_PMDMAPPED, NR_FILE_PMDMAPPED) and mlock handling IIRC. See page_remove_rmap().

If we can avoid that complexity right from the start for hugetlb, great, ..

Quoted 14 lines

i. Smaps uses page_mapcount() >= 2 to determine if hugetlb memory is "private_hugetlb" or "shared_hugetlb".
ii. Migration with MPOL_MF_MOVE will check page_mapcount() to see if the hugepage is shared or not. Pages that would otherwise be migrated now require MPOL_MF_MOVE_ALL to be migrated.
[Really both of the above are checking how many VMAs are mapping our hugepage.] iii. CoW. This isn't a problem right now because CoW is only possible with MAP_PRIVATE VMAs and HGM can only be enabled for MAP_SHARED VMAs. iv. The hwpoison handling code will check if it successfully unmapped the poisoned page. This isn't a problem right now, as hwpoison will unmap all the mappings for the hugepage, not just the 4K where the poison was found.

Quoted 4 lines

Doing it this way allows HGM to remain compatible with the hugetlb vmemmap optimization. None of the above problems strike me as particularly major, but it's unclear to me how important it is to have page_mapcount() have a consistent meaning for hugetlb vs non-hugetlb.

See below, maybe we should tackle HGM from a different direction.

Quoted 10 lines

The other way page_mapcount() (let's say the "THP-like way") could be done is like this: increment compound mapcount if we're mapping a hugetlb page normally (e.g., 1G page with a PUD). If we're mapping at high-granularity, increment the mapcount for each 4K page that is getting mapped (e.g., PMD within a 1G page: increment the mapcount for the 512 pages that are now mapped). This yields the same page_mapcount() API we had before, but we lose the hugetlb vmemmap optimization.

Quoted 7 lines

We could introduce an API like hugetlb_vma_mapcount() that would, for hugetlb, give us the number of VMAs that map a hugepage, but I don't think people would like this.

I'm curious what others think (Mike, Matthew?). I'm guessing the THP-like way is probably what most people would want, though it would be a real shame to lose the vmemmap optimization.

Heh, not me :) Having a single mapcount is certainly much cleaner. ... and if we're dealing with refcount overflows already, mapcount overflows are not an issue.

I wonder if the following crazy idea has already been discussed: treat the whole mapping as a single large logical mapping. One reference and one mapping, no matter how the individual parts are mapped into the assigned page table sub-tree.

Because for hugetlb with MAP_SHARED, we know that the complete assigned sub-tree of page tables can only map the given hugetlb page, no fragments of something else. That's very different to THP in private mappings ...

So as soon as the first piece gets mapped, we increment refcount+mapcount. Other pieces in the same subtree don't do that.

Once the last piece is unmapped (or simpler: once the complete subtree of page tables is gone), we decrement refcount+mapcount. Might require some brain power to do this tracking, but I wouldn't call it impossible right from the start.

Would such a design violate other design aspects that are important?

Signature · 3 lines
Thanks,

David / dhildenb
reviewer
#45 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 44 lines · David Hildenbrand
Quoted 18 lines · James Houghton
Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Peter Xu
Quoted 5 lines · James Houghton

I'll look into it, but doing it this way will use _mapcount, so we won't be able to use the vmemmap optimization. I think even if we do use Hugh's approach, refcount is still being kept on the head page, so there's still an overflow risk there (but maybe I am misunderstanding).

Quoted 7 lines
Quoted 2 lines

Could you remind me what's the issue if using refcount on the small pages rather than the head (assuming vmemmap still can be disabled)?

The THP-way of doing things is refcounting on the head page. All folios use a single refcount on the head.

There has to be a pretty good reason to do it differently.

Quoted 8 lines

Peter and I have discussed this a lot offline. There are two main problems here:

1. Refcount overflow

Refcount is always kept on the head page (before and after this series). IIUC, this means that if THPs could be 1G in size, they too would be susceptible to the same potential overflow. How easy is the overflow? [1]

Right. You'd need 8k VMAs. With 2 MiB THP you'd need 4096k VMAs. So ~64 processes with 64k VMAs. Not impossible to achieve if one really wants to break the system ...

Side note: a long long time ago, we used to have sub-page refcounts for THP. IIRC, that was even before we had sub-page mapcounts and was used to make COW decisions.

Quoted 7 lines

To deal with this, the best solution we've been able to come up with is to check if refcount is > INT_MAX/2 (similar to try_get_page()), and if it is, stop the operation (UFFDIO_CONTINUE or a page fault) from proceeding. In the UFFDIO_CONTINUE case, return ENOMEM. In the page fault cause, return VM_FAULT_SIGBUS (not VM_FAULT_OOM; we don't want to kill a random process).

Quoted 3 lines

You'd have to also make sure that fork() won't do the same. At least with uffd-wp, Peter also added page table copying during fork() for MAP_SHARED mappings, which would have to be handled.

If we want such a check to make a real difference, IIUC we may want to consider having similar check in:

page_ref_add
page_ref_inc
page_ref_inc_return
page_ref_add_unless

But it's unfortunate that mostly all the callers to these functions (especially the first two) do not have a retval yet at all. Considering the low possibility so far on having it overflow, maybe it can also be done for later (and I think checking negative as try_get_page would suffice too).

Quoted 4 lines

Of course, one can just disallow fork() with any HGM right from the start and keep it all simpler to not open up a can of worms there.

Quoted 8 lines

Is it reasonable, to have more than one (or a handful) of VMAs mapping a huge page via a HGM? Restricting it to a single one, would make handling much easier.

If there is ever demand for more HGM mappings, that whole problem (and complexity) could be dealt with later. ... but I assume it will already be a requirement for VMs (e.g., under QEMU) that share memory with other processes (virtiofsd and friends?)

Yes, any form of multi-proc QEMU will need that for supporting HGM postcopy.

Quoted 81 lines
Quoted 6 lines

(So David, I think this answers your question. Refcount should be handled just like THPs.)

Quoted 8 lines

2. page_mapcount() API differences

In this series, page_mapcount() returns the total number of page table references for the compound page. For example, if you have a PTE-mapped 2M page (with no other mappings), page_mapcount() for each 4K page will be 512. This is not the same as a THP: page_mapcount() would return 1 for each page. Because of the difference in page_mapcount(), we have 4 problems:

IMHO, it would actually be great to just be able to remove the sub-page mapcounts for THP and make it all simpler.

Right now, the sub-page mapcount is mostly required for making COW decisions, but only for accounting purposes IIRC (NR_ANON_THPS, NR_SHMEM_PMDMAPPED, NR_FILE_PMDMAPPED) and mlock handling IIRC. See page_remove_rmap().

If we can avoid that complexity right from the start for hugetlb, great, ..

Quoted 14 lines

i. Smaps uses page_mapcount() >= 2 to determine if hugetlb memory is "private_hugetlb" or "shared_hugetlb".
ii. Migration with MPOL_MF_MOVE will check page_mapcount() to see if the hugepage is shared or not. Pages that would otherwise be migrated now require MPOL_MF_MOVE_ALL to be migrated.
[Really both of the above are checking how many VMAs are mapping our hugepage.] iii. CoW. This isn't a problem right now because CoW is only possible with MAP_PRIVATE VMAs and HGM can only be enabled for MAP_SHARED VMAs. iv. The hwpoison handling code will check if it successfully unmapped the poisoned page. This isn't a problem right now, as hwpoison will unmap all the mappings for the hugepage, not just the 4K where the poison was found.

Quoted 4 lines

Doing it this way allows HGM to remain compatible with the hugetlb vmemmap optimization. None of the above problems strike me as particularly major, but it's unclear to me how important it is to have page_mapcount() have a consistent meaning for hugetlb vs non-hugetlb.

See below, maybe we should tackle HGM from a different direction.

Quoted 10 lines

The other way page_mapcount() (let's say the "THP-like way") could be done is like this: increment compound mapcount if we're mapping a hugetlb page normally (e.g., 1G page with a PUD). If we're mapping at high-granularity, increment the mapcount for each 4K page that is getting mapped (e.g., PMD within a 1G page: increment the mapcount for the 512 pages that are now mapped). This yields the same page_mapcount() API we had before, but we lose the hugetlb vmemmap optimization.

Quoted 7 lines

We could introduce an API like hugetlb_vma_mapcount() that would, for hugetlb, give us the number of VMAs that map a hugepage, but I don't think people would like this.

I'm curious what others think (Mike, Matthew?). I'm guessing the THP-like way is probably what most people would want, though it would be a real shame to lose the vmemmap optimization.

Heh, not me :) Having a single mapcount is certainly much cleaner. ... and if we're dealing with refcount overflows already, mapcount overflows are not an issue.

I wonder if the following crazy idea has already been discussed: treat the whole mapping as a single large logical mapping. One reference and one mapping, no matter how the individual parts are mapped into the assigned page table sub-tree.

Because for hugetlb with MAP_SHARED, we know that the complete assigned sub-tree of page tables can only map the given hugetlb page, no fragments of something else. That's very different to THP in private mappings ...

So as soon as the first piece gets mapped, we increment refcount+mapcount. Other pieces in the same subtree don't do that.

Quoted 6 lines

Once the last piece is unmapped (or simpler: once the complete subtree of page tables is gone), we decrement refcount+mapcount. Might require some brain power to do this tracking, but I wouldn't call it impossible right from the start.

Would such a design violate other design aspects that are important?

The question is how to maintaining above information.

It needs to be per-map (so one page mapped multiple times can be accounted differently), and per-page (so one mapping/vma can contain multiple pages). So far I think that's exactly the pgtable. If we can squeeze information into the pgtable it'll work out, but definitely not trivial. Or we can maintain seperate allocates for such information, but that can be extra overheads too.

So far I'd still consider going with reusing thp mapcounts, which will mostly be what James mentioned above. The only difference is I'm not sure whether we should allow mapping e.g. 2M ranges for 1G pages. THP mapcount doesn't have intermediate layer to maintain mapcount information like 2M, so to me it's easier we start with only mapping either the hpage size or PAGE_SIZE, not any intermediate size allowed.

Having intermediate size mapping allowed can at least be error prone to me. One example is if some pgtable walker found a 2M page, it may easily fetch the PFN out of it, assuming it's a compound page and it should satisfy PageHead(pfn)==true but it'll start to break here, because the 2M PFN will only be a small page pfn for the 1G huge page in this case.

To me, intermediate sized mappings are good to have but not required to resolve HGM problems, at least so far. Said that, I'm fine with looking at what it'll look like if James would like to keep persuing that direction.

Thanks,

Signature · 1 line
Peter Xu
#46 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

To deal with this, the best solution we've been able to come up with is to check if refcount is > INT_MAX/2 (similar to try_get_page()), and if it is, stop the operation (UFFDIO_CONTINUE or a page fault) from proceeding. In the UFFDIO_CONTINUE case, return ENOMEM. In the page fault cause, return VM_FAULT_SIGBUS (not VM_FAULT_OOM; we don't want to kill a random process).

You'd have to also make sure that fork() won't do the same. At least with uffd-wp, Peter also added page table copying during fork() for MAP_SHARED mappings, which would have to be handled.

Indeed, thank you! copy_hugetlb_page_range() (and therefore fork() in some cases) would need this check too.

Quoted 9 lines

If we want such a check to make a real difference, IIUC we may want to consider having similar check in:

page_ref_add
page_ref_inc
page_ref_inc_return
page_ref_add_unless

Quoted 4 lines

But it's unfortunate that mostly all the callers to these functions (especially the first two) do not have a retval yet at all. Considering the low possibility so far on having it overflow, maybe it can also be done for later (and I think checking negative as try_get_page would suffice too).

I think as long as we annotate the few cases that userspace can exploit to overflow refcount, we should be ok. I think this was the same idea with try_get_page(): it is supposed to be used in places that userspace could reasonably exploit to overflow refcount.

Quoted 9 lines
Quoted 9 lines

Of course, one can just disallow fork() with any HGM right from the start and keep it all simpler to not open up a can of worms there.

Is it reasonable, to have more than one (or a handful) of VMAs mapping a huge page via a HGM? Restricting it to a single one, would make handling much easier.

Quoted 7 lines
Quoted 4 lines

If there is ever demand for more HGM mappings, that whole problem (and complexity) could be dealt with later. ... but I assume it will already be a requirement for VMs (e.g., under QEMU) that share memory with other processes (virtiofsd and friends?)

Yes, any form of multi-proc QEMU will need that for supporting HGM postcopy.

+1. Disallowing fork() entirely is quite a restrictive limitation.

[snip]

Quoted 21 lines
Quoted 21 lines
Quoted 3 lines

I'm curious what others think (Mike, Matthew?). I'm guessing the THP-like way is probably what most people would want, though it would be a real shame to lose the vmemmap optimization.

Heh, not me :) Having a single mapcount is certainly much cleaner. ... and if we're dealing with refcount overflows already, mapcount overflows are not an issue.

I wonder if the following crazy idea has already been discussed: treat the whole mapping as a single large logical mapping. One reference and one mapping, no matter how the individual parts are mapped into the assigned page table sub-tree.

Because for hugetlb with MAP_SHARED, we know that the complete assigned sub-tree of page tables can only map the given hugetlb page, no fragments of something else. That's very different to THP in private mappings ...

So as soon as the first piece gets mapped, we increment refcount+mapcount. Other pieces in the same subtree don't do that.

Quoted 6 lines
Quoted 6 lines

Once the last piece is unmapped (or simpler: once the complete subtree of page tables is gone), we decrement refcount+mapcount. Might require some brain power to do this tracking, but I wouldn't call it impossible right from the start.

Would such a design violate other design aspects that are important?

This is actually how mapcount was treated in HGM RFC v1 (though not refcount); it is doable for both [2].

One caveat here: if a page is unmapped in small pieces, it is difficult to know if the page is legitimately completely unmapped (we would have to check all the PTEs in the page table). In RFC v1, I sidestepped this caveat by saying that "page_mapcount() is incremented if the hstate-level PTE is present". A single unmap on the whole hugepage will clear the hstate-level PTE, thus decrementing the mapcount.

On a related note, there still exists an (albeit minor) API difference vs. THPs: a piece of a page that is legitimately unmapped can still have a positive page_mapcount().

Given that this approach allows us to retain the hugetlb vmemmap optimization (and it wouldn't require a horrible amount of complexity), I prefer this approach over the THP-like approach.

Quoted 9 lines

The question is how to maintaining above information.

It needs to be per-map (so one page mapped multiple times can be accounted differently), and per-page (so one mapping/vma can contain multiple pages). So far I think that's exactly the pgtable. If we can squeeze information into the pgtable it'll work out, but definitely not trivial. Or we can maintain seperate allocates for such information, but that can be extra overheads too.

I don't think we necessarily need to check the page table if we allow for the limitations stated above.

Quoted 8 lines

So far I'd still consider going with reusing thp mapcounts, which will mostly be what James mentioned above. The only difference is I'm not sure whether we should allow mapping e.g. 2M ranges for 1G pages. THP mapcount doesn't have intermediate layer to maintain mapcount information like 2M, so to me it's easier we start with only mapping either the hpage size or PAGE_SIZE, not any intermediate size allowed.

Quoted 5 lines

Having intermediate size mapping allowed can at least be error prone to me. One example is if some pgtable walker found a 2M page, it may easily fetch the PFN out of it, assuming it's a compound page and it should satisfy PageHead(pfn)==true but it'll start to break here, because the 2M PFN will only be a small page pfn for the 1G huge page in this case.

I assume you mean PageHuge(page). There are cases where assumptions are made about hugetlb pages and VMAs; they need to be corrected. It sounds like you're really talking about errors wrt missing a compound_head()/page_folio(), but it can be even more general than that. For example, the arm64 KVM MMU assumes that hugetlb HGM doesn't exist, and so it needs to be fixed before HGM can be enabled for arm64.

If a page is HGM-mapped, I think it's more error-prone to somehow make PageHuge() come back with false. So the only solution I see here is careful auditing and testing.

I don't really see how allow/disallowing intermediate sizes changes this problem either.

Quoted 4 lines

To me, intermediate sized mappings are good to have but not required to resolve HGM problems, at least so far. Said that, I'm fine with looking at what it'll look like if James would like to keep persuing that direction.

I've mentioned this to Peter already, but I don't think discarding intermediate mapping levels really reduces complexity all that much.

[2]: Using the names of functions as of v1 (Peter has since suggested a name change): hugetlb_alloc_{pte,pmd} will tell us if we really allocated the PTE. That info can be passed up through hugetlb_walk_step()=>hugetlb_hgm_walk() (where we only care if the *hstate-level* PTE got allocated) and hugetlb_full_walk*() for the caller to determine if refcount/mapcount should be incremented.

Thank you both for your thoughts so far. :)

- James

#47 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 18 lines
Quoted 6 lines
Quoted 6 lines

Once the last piece is unmapped (or simpler: once the complete subtree of page tables is gone), we decrement refcount+mapcount. Might require some brain power to do this tracking, but I wouldn't call it impossible right from the start.

Would such a design violate other design aspects that are important?

This is actually how mapcount was treated in HGM RFC v1 (though not refcount); it is doable for both [2].

One caveat here: if a page is unmapped in small pieces, it is difficult to know if the page is legitimately completely unmapped (we would have to check all the PTEs in the page table). In RFC v1, I sidestepped this caveat by saying that "page_mapcount() is incremented if the hstate-level PTE is present". A single unmap on the whole hugepage will clear the hstate-level PTE, thus decrementing the mapcount.

Quoted 7 lines

On a related note, there still exists an (albeit minor) API difference vs. THPs: a piece of a page that is legitimately unmapped can still have a positive page_mapcount().

Given that this approach allows us to retain the hugetlb vmemmap optimization (and it wouldn't require a horrible amount of complexity), I prefer this approach over the THP-like approach.

If we can store (directly/indirectly) metadata in the highest pgtable that HGM-maps a hugetlb page, I guess what would be reasonable:

* hugetlb page pointer
* mapped size

Whenever mapping/unmapping sub-parts, we'd have to update that information.

Once "mapped size" dropped to 0, we know that the hugetlb page was completely unmapped and we can drop the refcount+mapcount, clear metadata (including hugetlb page pointer) [+ remove the page tables?].

Similarly, once "mapped size" corresponds to the hugetlb size, we can immediately spot that everything is mapped.

Again, just a high-level idea.

Signature · 3 lines
Thanks,

David / dhildenb
#48 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

In reply to James Houghton
Quoted 79 lines · James Houghton
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

To deal with this, the best solution we've been able to come up with is to check if refcount is > INT_MAX/2 (similar to try_get_page()), and if it is, stop the operation (UFFDIO_CONTINUE or a page fault) from proceeding. In the UFFDIO_CONTINUE case, return ENOMEM. In the page fault cause, return VM_FAULT_SIGBUS (not VM_FAULT_OOM; we don't want to kill a random process).

You'd have to also make sure that fork() won't do the same. At least with uffd-wp, Peter also added page table copying during fork() for MAP_SHARED mappings, which would have to be handled.

Indeed, thank you! copy_hugetlb_page_range() (and therefore fork() in some cases) would need this check too.

Quoted 9 lines

If we want such a check to make a real difference, IIUC we may want to consider having similar check in:

page_ref_add
page_ref_inc
page_ref_inc_return
page_ref_add_unless

Quoted 4 lines

But it's unfortunate that mostly all the callers to these functions (especially the first two) do not have a retval yet at all. Considering the low possibility so far on having it overflow, maybe it can also be done for later (and I think checking negative as try_get_page would suffice too).

I think as long as we annotate the few cases that userspace can exploit to overflow refcount, we should be ok. I think this was the same idea with try_get_page(): it is supposed to be used in places that userspace could reasonably exploit to overflow refcount.

Quoted 9 lines
Quoted 9 lines

Of course, one can just disallow fork() with any HGM right from the start and keep it all simpler to not open up a can of worms there.

Is it reasonable, to have more than one (or a handful) of VMAs mapping a huge page via a HGM? Restricting it to a single one, would make handling much easier.

Quoted 7 lines
Quoted 4 lines

If there is ever demand for more HGM mappings, that whole problem (and complexity) could be dealt with later. ... but I assume it will already be a requirement for VMs (e.g., under QEMU) that share memory with other processes (virtiofsd and friends?)

Yes, any form of multi-proc QEMU will need that for supporting HGM postcopy.

+1. Disallowing fork() entirely is quite a restrictive limitation.

[snip]

Quoted 26 lines
Quoted 26 lines
Quoted 3 lines

I'm curious what others think (Mike, Matthew?). I'm guessing the THP-like way is probably what most people would want, though it would be a real shame to lose the vmemmap optimization.

Heh, not me :) Having a single mapcount is certainly much cleaner. ... and if we're dealing with refcount overflows already, mapcount overflows are not an issue.

I wonder if the following crazy idea has already been discussed: treat the whole mapping as a single large logical mapping. One reference and one mapping, no matter how the individual parts are mapped into the assigned page table sub-tree.

Because for hugetlb with MAP_SHARED, we know that the complete assigned sub-tree of page tables can only map the given hugetlb page, no fragments of something else. That's very different to THP in private mappings ...

So as soon as the first piece gets mapped, we increment refcount+mapcount. Other pieces in the same subtree don't do that.

Once the last piece is unmapped (or simpler: once the complete subtree of page tables is gone), we decrement refcount+mapcount. Might require some brain power to do this tracking, but I wouldn't call it impossible right from the start.

Quoted 4 lines
Quoted 1 line
Quoted 1 line

Would such a design violate other design aspects that are important?

This is actually how mapcount was treated in HGM RFC v1 (though not refcount); it is doable for both [2].

My apologies for being late to the party :)

When Peter first brought up the issue with ref/map_count overflows I was thinking that we should use a scheme like David describes above. As James points out, this was the approach taken in the first RFC.

Quoted 3 lines

One caveat here: if a page is unmapped in small pieces, it is difficult to know if the page is legitimately completely unmapped (we would have to check all the PTEs in the page table).

Are we allowing unmapping of small (non-huge page sized) areas with HGM? We must be if you are concerned with it. What API would cause this? I just do not remember this discussion.

Quoted 6 lines

would have to check all the PTEs in the page table). In RFC v1, I sidestepped this caveat by saying that "page_mapcount() is incremented if the hstate-level PTE is present". A single unmap on the whole hugepage will clear the hstate-level PTE, thus decrementing the mapcount.

Quoted 7 lines

On a related note, there still exists an (albeit minor) API difference vs. THPs: a piece of a page that is legitimately unmapped can still have a positive page_mapcount().

Given that this approach allows us to retain the hugetlb vmemmap optimization (and it wouldn't require a horrible amount of complexity), I prefer this approach over the THP-like approach.

Me too.

Quoted 10 lines
Quoted 9 lines

The question is how to maintaining above information.

It needs to be per-map (so one page mapped multiple times can be accounted differently), and per-page (so one mapping/vma can contain multiple pages). So far I think that's exactly the pgtable. If we can squeeze information into the pgtable it'll work out, but definitely not trivial. Or we can maintain seperate allocates for such information, but that can be extra overheads too.

Quoted 3 lines

I don't think we necessarily need to check the page table if we allow for the limitations stated above.

When I was thinking about this I was a bit concerned about having enough information to know exactly when to inc or dec counts. I was actually worried about knowing to do the increment. I don't recall how it was done in the first RFC, but from a high level it would need to be done when the first hstate level PTE is allocated/added to the page table. Right? My concern was with all the places where we could 'error out' after allocating the PTE, but before initializing it. I was just thinking that we might need to scan the page table or keep metadata for better or easier accounting.

I think Peter mentioned it elsewhere, we should come up with a workable scheme for HGM ref/map counting. This can be done somewhat independently.

Signature · 1 line
Mike Kravetz
#49 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 28 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

I wonder if the following crazy idea has already been discussed: treat the whole mapping as a single large logical mapping. One reference and one mapping, no matter how the individual parts are mapped into the assigned page table sub-tree.

Because for hugetlb with MAP_SHARED, we know that the complete assigned sub-tree of page tables can only map the given hugetlb page, no fragments of something else. That's very different to THP in private mappings ...

So as soon as the first piece gets mapped, we increment refcount+mapcount. Other pieces in the same subtree don't do that.

Once the last piece is unmapped (or simpler: once the complete subtree of page tables is gone), we decrement refcount+mapcount. Might require some brain power to do this tracking, but I wouldn't call it impossible right from the start.

Quoted 4 lines
Quoted 1 line
Quoted 1 line

Would such a design violate other design aspects that are important?

This is actually how mapcount was treated in HGM RFC v1 (though not refcount); it is doable for both [2].

My apologies for being late to the party :)

When Peter first brought up the issue with ref/map_count overflows I was thinking that we should use a scheme like David describes above. As James points out, this was the approach taken in the first RFC.

Quoted 7 lines
Quoted 3 lines

One caveat here: if a page is unmapped in small pieces, it is difficult to know if the page is legitimately completely unmapped (we would have to check all the PTEs in the page table).

Are we allowing unmapping of small (non-huge page sized) areas with HGM? We must be if you are concerned with it. What API would cause this? I just do not remember this discussion.

There was some discussion about allowing MADV_DONTNEED on less-than-hugepage pieces [3] (it actually motivated the switch from UFFD_FEATURE_MINOR_HUGETLBFS_HGM to MADV_SPLIT). It isn't implemented in this series, but it could be implemented in the future.

In the more immediate future, we want hwpoison to unmap at 4K, so MADV_HWPOISON would be a mechanism that userspace is granted to do this.

Quoted 31 lines
Quoted 7 lines

would have to check all the PTEs in the page table). In RFC v1, I sidestepped this caveat by saying that "page_mapcount() is incremented if the hstate-level PTE is present". A single unmap on the whole hugepage will clear the hstate-level PTE, thus decrementing the mapcount.

Quoted 7 lines

On a related note, there still exists an (albeit minor) API difference vs. THPs: a piece of a page that is legitimately unmapped can still have a positive page_mapcount().

Given that this approach allows us to retain the hugetlb vmemmap optimization (and it wouldn't require a horrible amount of complexity), I prefer this approach over the THP-like approach.

Me too.

Quoted 10 lines
Quoted 9 lines

The question is how to maintaining above information.

It needs to be per-map (so one page mapped multiple times can be accounted differently), and per-page (so one mapping/vma can contain multiple pages). So far I think that's exactly the pgtable. If we can squeeze information into the pgtable it'll work out, but definitely not trivial. Or we can maintain seperate allocates for such information, but that can be extra overheads too.

Quoted 3 lines

I don't think we necessarily need to check the page table if we allow for the limitations stated above.

Quoted 9 lines

When I was thinking about this I was a bit concerned about having enough information to know exactly when to inc or dec counts. I was actually worried about knowing to do the increment. I don't recall how it was done in the first RFC, but from a high level it would need to be done when the first hstate level PTE is allocated/added to the page table. Right? My concern was with all the places where we could 'error out' after allocating the PTE, but before initializing it. I was just thinking that we might need to scan the page table or keep metadata for better or easier accounting.

The only two places where we can *create* a high-granularity page table are: __mcopy_atomic_hugetlb (UFFDIO_CONTINUE) and copy_hugetlb_page_range. RFC v1 did not properly deal with the cases where we error out. To correctly handle these cases, we basically have to do the pagecache lookup before touching the page table.

1. For __mcopy_atomic_hugetlb, we can lookup the page before doing the PT walk/alloc. If PT walk tells us to inc the page ref/mapcount, we do so immediately. We can easily pass the page into hugetlb_mcopy_atomic_pte() (via 'pagep') .

2. For copy_hugetlb_page_range() for VM_MAYSHARE, we can also do the lookup before we do the page table walk. I'm not sure how to support non-shared HGM mappings with this scheme (in this series, we also don't support non-shared; we return -EINVAL).
NB: The only case where high-granularity mappings for !VM_MAYSHARE VMAs would come up is as a result of hwpoison.

So we can avoid keeping additional metadata for what this series is trying to accomplish, but if the above isn't acceptable, then I/we can try to come up with a scheme that would be acceptable.

There is also the possibility that the scheme implemented in this version of the series is acceptable (i.e., the page_mapcount() API difference, which results in slightly modified page migration behavior and smaps output, is ok... assuming we have the refcount overflow check).

Quoted 3 lines

I think Peter mentioned it elsewhere, we should come up with a workable scheme for HGM ref/map counting. This can be done somewhat independently.

FWIW, what makes the most sense to me right now is to implement the THP-like scheme and mark HGM as mutually exclusive with the vmemmap optimization. We can later come up with a scheme that lets us retain compatibility. (Is that what you mean by "this can be done somewhat independently", Mike?)

[3]: https://lore.kernel.org/linux-mm/[email protected]/T/#m9a1090108b61d32c04b68a1f3f2577644823a999

- James

#50 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 32 lines · James Houghton
Quoted 32 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

I wonder if the following crazy idea has already been discussed: treat the whole mapping as a single large logical mapping. One reference and one mapping, no matter how the individual parts are mapped into the assigned page table sub-tree.

Because for hugetlb with MAP_SHARED, we know that the complete assigned sub-tree of page tables can only map the given hugetlb page, no fragments of something else. That's very different to THP in private mappings ...

So as soon as the first piece gets mapped, we increment refcount+mapcount. Other pieces in the same subtree don't do that.

Once the last piece is unmapped (or simpler: once the complete subtree of page tables is gone), we decrement refcount+mapcount. Might require some brain power to do this tracking, but I wouldn't call it impossible right from the start.

Quoted 4 lines
Quoted 1 line
Quoted 1 line

Would such a design violate other design aspects that are important?

This is actually how mapcount was treated in HGM RFC v1 (though not refcount); it is doable for both [2].

My apologies for being late to the party :)

When Peter first brought up the issue with ref/map_count overflows I was thinking that we should use a scheme like David describes above. As James points out, this was the approach taken in the first RFC.

Quoted 3 lines

One caveat here: if a page is unmapped in small pieces, it is difficult to know if the page is legitimately completely unmapped (we would have to check all the PTEs in the page table).

Quoted 8 lines
Quoted 3 lines

Are we allowing unmapping of small (non-huge page sized) areas with HGM? We must be if you are concerned with it. What API would cause this? I just do not remember this discussion.

There was some discussion about allowing MADV_DONTNEED on less-than-hugepage pieces [3] (it actually motivated the switch from UFFD_FEATURE_MINOR_HUGETLBFS_HGM to MADV_SPLIT). It isn't implemented in this series, but it could be implemented in the future.

OK, so we do not actually create HGM mappings until a uffd operation is done at a less than huge page size granularity. MADV_SPLIT just says that HGM mappings are 'possible' for this vma. Hopefully, my understanding is correct.

I was concerned about things like the page fault path, but in that case we have already 'entered HGM mode' via a uffd operation.

Both David and Peter have asked whether eliminating intermediate mapping levels would be a simplification. I trust your response that it would not help much in the current design/implementation. But, it did get me thinking about something else.

Perhaps we have discussed this before, and perhaps it does not meet all user needs, but one way possibly simplify this is:

- 'Enable HGM' via MADV_SPLIT. Must be done at huge page (hstate) granularity.
- MADV_SPLIT implicitly unmaps everything with in the range.
- MADV_SPLIT says all mappings for this vma will now be done at a base (4K) page size granularity. vma would be marked some way.
- I think this eliminates the need for hugetlb_pte's as we KNOW the mapping size.
- We still use huge pages to back 4K mappings, and we still have to deal with the ref/map_count issues.
- Code touching hugetlb page tables would KNOW the mapping size up front.

Again, apologies if we talked about and previously dismissed this type of approach.

Quoted 28 lines
Quoted 9 lines

When I was thinking about this I was a bit concerned about having enough information to know exactly when to inc or dec counts. I was actually worried about knowing to do the increment. I don't recall how it was done in the first RFC, but from a high level it would need to be done when the first hstate level PTE is allocated/added to the page table. Right? My concern was with all the places where we could 'error out' after allocating the PTE, but before initializing it. I was just thinking that we might need to scan the page table or keep metadata for better or easier accounting.

The only two places where we can *create* a high-granularity page table are: __mcopy_atomic_hugetlb (UFFDIO_CONTINUE) and copy_hugetlb_page_range. RFC v1 did not properly deal with the cases where we error out. To correctly handle these cases, we basically have to do the pagecache lookup before touching the page table.

1. For __mcopy_atomic_hugetlb, we can lookup the page before doing the PT walk/alloc. If PT walk tells us to inc the page ref/mapcount, we do so immediately. We can easily pass the page into hugetlb_mcopy_atomic_pte() (via 'pagep') .

2. For copy_hugetlb_page_range() for VM_MAYSHARE, we can also do the lookup before we do the page table walk. I'm not sure how to support non-shared HGM mappings with this scheme (in this series, we also don't support non-shared; we return -EINVAL).
NB: The only case where high-granularity mappings for !VM_MAYSHARE VMAs would come up is as a result of hwpoison.

Quoted 3 lines

So we can avoid keeping additional metadata for what this series is trying to accomplish, but if the above isn't acceptable, then I/we can try to come up with a scheme that would be acceptable.

Ok, I was thinking we had to deal with other code paths such as page fault. But, now I understand that is not the case with this design.

Quoted 7 lines

There is also the possibility that the scheme implemented in this version of the series is acceptable (i.e., the page_mapcount() API difference, which results in slightly modified page migration behavior and smaps output, is ok... assuming we have the refcount overflow check).

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

I think Peter mentioned it elsewhere, we should come up with a workable scheme for HGM ref/map counting. This can be done somewhat independently.

FWIW, what makes the most sense to me right now is to implement the THP-like scheme and mark HGM as mutually exclusive with the vmemmap optimization. We can later come up with a scheme that lets us retain compatibility. (Is that what you mean by "this can be done somewhat independently", Mike?)

Sort of, I was only saying that getting the ref/map counting right seems like a task than can be independently worked. Using the THP-like scheme is good.

Signature · 1 line
Mike Kravetz
#51 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 43 lines · Mike Kravetz
Quoted 32 lines · James Houghton
Quoted 32 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

I wonder if the following crazy idea has already been discussed: treat the whole mapping as a single large logical mapping. One reference and one mapping, no matter how the individual parts are mapped into the assigned page table sub-tree.

Because for hugetlb with MAP_SHARED, we know that the complete assigned sub-tree of page tables can only map the given hugetlb page, no fragments of something else. That's very different to THP in private mappings ...

So as soon as the first piece gets mapped, we increment refcount+mapcount. Other pieces in the same subtree don't do that.

Once the last piece is unmapped (or simpler: once the complete subtree of page tables is gone), we decrement refcount+mapcount. Might require some brain power to do this tracking, but I wouldn't call it impossible right from the start.

Quoted 4 lines
Quoted 1 line
Quoted 1 line

Would such a design violate other design aspects that are important?

This is actually how mapcount was treated in HGM RFC v1 (though not refcount); it is doable for both [2].

My apologies for being late to the party :)

When Peter first brought up the issue with ref/map_count overflows I was thinking that we should use a scheme like David describes above. As James points out, this was the approach taken in the first RFC.

Quoted 3 lines

One caveat here: if a page is unmapped in small pieces, it is difficult to know if the page is legitimately completely unmapped (we would have to check all the PTEs in the page table).

Quoted 8 lines
Quoted 3 lines

Are we allowing unmapping of small (non-huge page sized) areas with HGM? We must be if you are concerned with it. What API would cause this? I just do not remember this discussion.

There was some discussion about allowing MADV_DONTNEED on less-than-hugepage pieces [3] (it actually motivated the switch from UFFD_FEATURE_MINOR_HUGETLBFS_HGM to MADV_SPLIT). It isn't implemented in this series, but it could be implemented in the future.

Quoted 4 lines

OK, so we do not actually create HGM mappings until a uffd operation is done at a less than huge page size granularity. MADV_SPLIT just says that HGM mappings are 'possible' for this vma. Hopefully, my understanding is correct.

Right, that's the current meaning of MADV_SPLIT for hugetlb.

Quoted 22 lines

I was concerned about things like the page fault path, but in that case we have already 'entered HGM mode' via a uffd operation.

Both David and Peter have asked whether eliminating intermediate mapping levels would be a simplification. I trust your response that it would not help much in the current design/implementation. But, it did get me thinking about something else.

Perhaps we have discussed this before, and perhaps it does not meet all user needs, but one way possibly simplify this is:

- 'Enable HGM' via MADV_SPLIT. Must be done at huge page (hstate) granularity.
- MADV_SPLIT implicitly unmaps everything with in the range.
- MADV_SPLIT says all mappings for this vma will now be done at a base (4K) page size granularity. vma would be marked some way.
- I think this eliminates the need for hugetlb_pte's as we KNOW the mapping size.
- We still use huge pages to back 4K mappings, and we still have to deal with the ref/map_count issues.
- Code touching hugetlb page tables would KNOW the mapping size up front.

Quoted 2 lines

Again, apologies if we talked about and previously dismissed this type of approach.

I think Peter was the one who originally suggested an approach like this, and it meets my needs. However, I still think the way that things are currently implemented is the right way to go.

Assuming we want decent performance, we can't get away with the same strategy of just passing pte_t*s everywhere. The PTL for a 4K PTE should be based on the PMD above the PTE, so we need to either pass around the PMD above our PTE, or we need to pass around the PTL. This is something that hugetlb_pte does for us, so, in some sense, even going with this simpler approach, we still need a hugetlb_pte-like construct.

Although most of the other complexity that HGM introduces would have to be introduced either way (like having to deal with putting compound_head()/page_folio() in more places and doing some per-architecture updates), there are some complexities that the simpler approach avoids:

- We avoid problems related to compound PTEs (the problem being: two threads racing to populate a contiguous and non-contiguous PTE that take up the same space could lead to user-detectable incorrect behavior. This isn't hard to fix; it will be when I send the arm64 patches up.)

- We don't need to check if PTEs get split from under us in PT walks. (In a lot of cases, the appropriate action is just to treat the PTE as if it were pte_none().) In the same vein, we don't need hugetlb_pte_present_leaf() at all, because PTEs we find will always be leaves.

- We don't have to deal with sorting hstates or implementing for_each_hgm_shift()/hugetlb_alloc_largest_pte().

None of these complexities are particularly major in my opinion.

This might seem kind of contrived, but let's say you have a VM with 1T of memory, and you find 100 memory errors all in different 1G pages over the life of this VM (years, potentially). Having 10% of your memory be 4K-mapped is definitely worse than having 10% be 2M-mapped (lost performance and increased memory overhead). There might be other cases in the future where being able to have intermediate mapping sizes could be helpful.

Quoted 51 lines
Quoted 28 lines
Quoted 9 lines

When I was thinking about this I was a bit concerned about having enough information to know exactly when to inc or dec counts. I was actually worried about knowing to do the increment. I don't recall how it was done in the first RFC, but from a high level it would need to be done when the first hstate level PTE is allocated/added to the page table. Right? My concern was with all the places where we could 'error out' after allocating the PTE, but before initializing it. I was just thinking that we might need to scan the page table or keep metadata for better or easier accounting.

The only two places where we can *create* a high-granularity page table are: __mcopy_atomic_hugetlb (UFFDIO_CONTINUE) and copy_hugetlb_page_range. RFC v1 did not properly deal with the cases where we error out. To correctly handle these cases, we basically have to do the pagecache lookup before touching the page table.

1. For __mcopy_atomic_hugetlb, we can lookup the page before doing the PT walk/alloc. If PT walk tells us to inc the page ref/mapcount, we do so immediately. We can easily pass the page into hugetlb_mcopy_atomic_pte() (via 'pagep') .

2. For copy_hugetlb_page_range() for VM_MAYSHARE, we can also do the lookup before we do the page table walk. I'm not sure how to support non-shared HGM mappings with this scheme (in this series, we also don't support non-shared; we return -EINVAL).
NB: The only case where high-granularity mappings for !VM_MAYSHARE VMAs would come up is as a result of hwpoison.

Quoted 3 lines

So we can avoid keeping additional metadata for what this series is trying to accomplish, but if the above isn't acceptable, then I/we can try to come up with a scheme that would be acceptable.

Ok, I was thinking we had to deal with other code paths such as page fault. But, now I understand that is not the case with this design.

Quoted 7 lines

There is also the possibility that the scheme implemented in this version of the series is acceptable (i.e., the page_mapcount() API difference, which results in slightly modified page migration behavior and smaps output, is ok... assuming we have the refcount overflow check).

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

I think Peter mentioned it elsewhere, we should come up with a workable scheme for HGM ref/map counting. This can be done somewhat independently.

FWIW, what makes the most sense to me right now is to implement the THP-like scheme and mark HGM as mutually exclusive with the vmemmap optimization. We can later come up with a scheme that lets us retain compatibility. (Is that what you mean by "this can be done somewhat independently", Mike?)

Quoted 3 lines

Sort of, I was only saying that getting the ref/map counting right seems like a task than can be independently worked. Using the THP-like scheme is good.

Ok! So if you're ok with the intermediate mapping sizes, it sounds like I should go ahead and implement the THP-like scheme.

- James

reviewer
#52 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 5 lines · James Houghton

- We avoid problems related to compound PTEs (the problem being: two threads racing to populate a contiguous and non-contiguous PTE that take up the same space could lead to user-detectable incorrect behavior. This isn't hard to fix; it will be when I send the arm64 patches up.)

Could you elaborate this one a bit more?

Quoted 7 lines

This might seem kind of contrived, but let's say you have a VM with 1T of memory, and you find 100 memory errors all in different 1G pages over the life of this VM (years, potentially). Having 10% of your memory be 4K-mapped is definitely worse than having 10% be 2M-mapped (lost performance and increased memory overhead). There might be other cases in the future where being able to have intermediate mapping sizes could be helpful.

This is not the norm, or is it? How the possibility of bad pages can distribute over hosts over years? This can definitely affect how we should target the intermediate level mappings.

Thanks,

Signature · 1 line
Peter Xu
#53 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 9 lines · Peter Xu
Quoted 5 lines · James Houghton

- We avoid problems related to compound PTEs (the problem being: two threads racing to populate a contiguous and non-contiguous PTE that take up the same space could lead to user-detectable incorrect behavior. This isn't hard to fix; it will be when I send the arm64 patches up.)

Could you elaborate this one a bit more?

In hugetlb_mcopy_atomic_pte(), we check that the PTE we're about to overwrite is pte_none() before overwriting it. For contiguous PTEs, this only checks the first PTE in the bunch.

If someone came around and populated one of the PTEs that lied in the middle of a potentially contiguous group of PTEs, we could end up overwriting that PTE if we later UFFDIO_CONTINUEd in such a way to create a contiguous PTE.

We would expect to get EEXIST here, but in this case the operation would succeed. To fix this, we can just check that ALL the PTEs in the contiguous bunch have the value that we're expecting, not just the first one.

hugetlb_no_page() has the same problem, but it's not immediately clear to me how it would result in incorrect behavior.

Quoted 9 lines
Quoted 8 lines

This might seem kind of contrived, but let's say you have a VM with 1T of memory, and you find 100 memory errors all in different 1G pages over the life of this VM (years, potentially). Having 10% of your memory be 4K-mapped is definitely worse than having 10% be 2M-mapped (lost performance and increased memory overhead). There might be other cases in the future where being able to have intermediate mapping sizes could be helpful.

Quoted 3 lines

This is not the norm, or is it? How the possibility of bad pages can distribute over hosts over years? This can definitely affect how we should target the intermediate level mappings.

I can't really speak for norms generally, but I can try to speak for Google Cloud. Google Cloud hasn't had memory error virtualization for very long (only about a year), but we've seen cases where VMs can pick up several memory errors over a few days/weeks. IMO, 100 errors in separate 1G pages over a few years isn't completely nonsensical, especially if the memory that you're using isn't so reliable or was damaged in shipping (like if it was flown over the poles or something!).

Now there is the concern about how an application would handle it. In a VMM's case, we can virtualize the error for the guest. In the guest, it's possible that a good chunk of the errors lie in unused pages and so can be easily marked as poisoned. It's possible that recovery is much more difficult. It's not unreasonable for an application to recover from a lot of memory errors.

- James

#54 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

In reply to James Houghton
Quoted 39 lines · James Houghton
Quoted 6 lines · Mike Kravetz

On 01/19/23 08:57, James Houghton wrote:

OK, so we do not actually create HGM mappings until a uffd operation is done at a less than huge page size granularity. MADV_SPLIT just says that HGM mappings are 'possible' for this vma. Hopefully, my understanding is correct.

Right, that's the current meaning of MADV_SPLIT for hugetlb.

Quoted 22 lines

I was concerned about things like the page fault path, but in that case we have already 'entered HGM mode' via a uffd operation.

Both David and Peter have asked whether eliminating intermediate mapping levels would be a simplification. I trust your response that it would not help much in the current design/implementation. But, it did get me thinking about something else.

Perhaps we have discussed this before, and perhaps it does not meet all user needs, but one way possibly simplify this is:

- 'Enable HGM' via MADV_SPLIT. Must be done at huge page (hstate) granularity.
- MADV_SPLIT implicitly unmaps everything with in the range.
- MADV_SPLIT says all mappings for this vma will now be done at a base (4K) page size granularity. vma would be marked some way.
- I think this eliminates the need for hugetlb_pte's as we KNOW the mapping size.
- We still use huge pages to back 4K mappings, and we still have to deal with the ref/map_count issues.
- Code touching hugetlb page tables would KNOW the mapping size up front.

Quoted 2 lines

Again, apologies if we talked about and previously dismissed this type of approach.

I think Peter was the one who originally suggested an approach like this, and it meets my needs. However, I still think the way that things are currently implemented is the right way to go.

Quoted 7 lines

Assuming we want decent performance, we can't get away with the same strategy of just passing pte_t*s everywhere. The PTL for a 4K PTE should be based on the PMD above the PTE, so we need to either pass around the PMD above our PTE, or we need to pass around the PTL. This is something that hugetlb_pte does for us, so, in some sense, even going with this simpler approach, we still need a hugetlb_pte-like construct.

Agree there is this performance hit. However, the 'simplest' approach would be to just use the page table lock as is done by default for 4K PTEs.

I do not know much about the (primary) live migration use case. My guess is that page table lock contention may be an issue? In this use case, HGM is only enabled for the duration the live migration operation, then a MADV_COLLAPSE is performed. If contention is likely to be an issue during this time, then yes we would need to pass around with something like hugetlb_pte.

Quoted 18 lines

Although most of the other complexity that HGM introduces would have to be introduced either way (like having to deal with putting compound_head()/page_folio() in more places and doing some per-architecture updates), there are some complexities that the simpler approach avoids:

- We avoid problems related to compound PTEs (the problem being: two threads racing to populate a contiguous and non-contiguous PTE that take up the same space could lead to user-detectable incorrect behavior. This isn't hard to fix; it will be when I send the arm64 patches up.)

- We don't need to check if PTEs get split from under us in PT walks. (In a lot of cases, the appropriate action is just to treat the PTE as if it were pte_none().) In the same vein, we don't need hugetlb_pte_present_leaf() at all, because PTEs we find will always be leaves.

Quoted 4 lines

- We don't have to deal with sorting hstates or implementing for_each_hgm_shift()/hugetlb_alloc_largest_pte().

None of these complexities are particularly major in my opinion.

Perhaps not. I was just thinking about the overall complexity of the hugetlb code after HGM. Currently, it is 'relatively simple' with fixed huge page sizes. IMO, much simpler than THP with two possible mapping sizes. With HGM and intermediate mapping sizes, it seems things could get more complicated than THP. Perhaps it is just me. I am just too familiar with the current code and a bit anxious about added complexity. But, I felt the same about vmemmap optimizations. :)

Quoted 7 lines

This might seem kind of contrived, but let's say you have a VM with 1T of memory, and you find 100 memory errors all in different 1G pages over the life of this VM (years, potentially). Having 10% of your memory be 4K-mapped is definitely worse than having 10% be 2M-mapped (lost performance and increased memory overhead). There might be other cases in the future where being able to have intermediate mapping sizes could be helpful.

That may be a bit contrived. We know memory error handling is a future use case, but I believe there is work outside of HGM than needs to be done to handle such situations. For example, HGM will allow the 1G mapping to isolate the 4K page with error. This prevents errors if you fault almost anywhere within the 1G page. But, there still remains the possibility of accessing that 4K page page with error. IMO, it will require user space/application intervention to address this as only the application knows about the potentially lost data. This is still something that needs to be designed. It would then makes sense for the application to also determine how it wants to proceed WRT mapping the 1G area. Perhaps they will want (and there will exist a mechanism) to migrate the data to a new 1G page without error.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

I think Peter mentioned it elsewhere, we should come up with a workable scheme for HGM ref/map counting. This can be done somewhat independently.

FWIW, what makes the most sense to me right now is to implement the THP-like scheme and mark HGM as mutually exclusive with the vmemmap optimization. We can later come up with a scheme that lets us retain compatibility. (Is that what you mean by "this can be done somewhat independently", Mike?)

Quoted 6 lines
Quoted 3 lines

Sort of, I was only saying that getting the ref/map counting right seems like a task than can be independently worked. Using the THP-like scheme is good.

Ok! So if you're ok with the intermediate mapping sizes, it sounds like I should go ahead and implement the THP-like scheme.

Yes, I am OK with it. Just expressed a bit of concern above.

Signature · 1 line
Mike Kravetz
reviewer
#55 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 6 lines · Mike Kravetz

I do not know much about the (primary) live migration use case. My guess is that page table lock contention may be an issue? In this use case, HGM is only enabled for the duration the live migration operation, then a MADV_COLLAPSE is performed. If contention is likely to be an issue during this time, then yes we would need to pass around with something like hugetlb_pte.

I'm not aware of any such contention issue. IMHO the migration problem is majorly about being too slow transferring a page being so large. Shrinking the page size should resolve the major problem already here IIUC.

AFAIU 4K-only solution should only reduce any lock contention because locks will always be pte-level if VM_HUGETLB_HGM set. When walking and creating the intermediate pgtable entries we can use atomic ops just like generic mm, so no lock needed at all. With uncertainty on the size of mappings, we'll need to take any of the multiple layers of locks.

[...]

Quoted 7 lines
Quoted 1 line

None of these complexities are particularly major in my opinion.

Perhaps not. I was just thinking about the overall complexity of the hugetlb code after HGM. Currently, it is 'relatively simple' with fixed huge page sizes. IMO, much simpler than THP with two possible mapping sizes. With HGM and intermediate mapping sizes, it seems things could get more complicated than THP. Perhaps it is just me.

Please count me in. :) I'm just still happy to see what it'll look like if James think having that complexity doesn't greatly affect the whole design.

Thanks,

Signature · 1 line
Peter Xu
#56 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 9 lines · Peter Xu
Quoted 6 lines · Mike Kravetz

I do not know much about the (primary) live migration use case. My guess is that page table lock contention may be an issue? In this use case, HGM is only enabled for the duration the live migration operation, then a MADV_COLLAPSE is performed. If contention is likely to be an issue during this time, then yes we would need to pass around with something like hugetlb_pte.

Quoted 3 lines

I'm not aware of any such contention issue. IMHO the migration problem is majorly about being too slow transferring a page being so large. Shrinking the page size should resolve the major problem already here IIUC.

This will be problematic if you scale up VMs to be quite large. Google upstreamed the "TDP MMU" for KVM/x86 that removed the need to take the MMU lock for writing in the EPT violation path. We found that this change is required for VMs >200 or so vCPUs to consistently avoid CPU soft lockups in the guest.

Requiring each UFFDIO_CONTINUE (in the post-copy path) to serialize on the same PTL would be problematic in the same way.

Quoted 7 lines

AFAIU 4K-only solution should only reduce any lock contention because locks will always be pte-level if VM_HUGETLB_HGM set. When walking and creating the intermediate pgtable entries we can use atomic ops just like generic mm, so no lock needed at all. With uncertainty on the size of mappings, we'll need to take any of the multiple layers of locks.

Other than taking the HugeTLB VMA lock for reading, walking/allocating page tables won't need any additional locking.

We take the PTL to allocate the next level down, but so does generic mm (look at __pud_alloc, __pmd_alloc for example). Maybe I am misunderstanding.

- James

reviewer
#57 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 10 lines · James Houghton
Quoted 9 lines · Peter Xu
Quoted 6 lines · Mike Kravetz

I do not know much about the (primary) live migration use case. My guess is that page table lock contention may be an issue? In this use case, HGM is only enabled for the duration the live migration operation, then a MADV_COLLAPSE is performed. If contention is likely to be an issue during this time, then yes we would need to pass around with something like hugetlb_pte.

Quoted 5 lines
Quoted 3 lines

I'm not aware of any such contention issue. IMHO the migration problem is majorly about being too slow transferring a page being so large. Shrinking the page size should resolve the major problem already here IIUC.

This will be problematic if you scale up VMs to be quite large.

Do you mean that for the postcopy use case one can leverage e.g. 2M mappings (over 1G) to avoid lock contentions when VM is large? I agree it should be more efficient than having 512 4K page installed, but I think it'll make the page fault resolution slower too if some thead is only looking for a 4k portion of it.

Quoted 4 lines

Google upstreamed the "TDP MMU" for KVM/x86 that removed the need to take the MMU lock for writing in the EPT violation path. We found that this change is required for VMs >200 or so vCPUs to consistently avoid CPU soft lockups in the guest.

After the kvm mmu rwlock convertion, it'll allow concurrent page faults even if only 4K pages are used, so it seems not directly relevant to what we're discussing here, no?

Quoted 3 lines

Requiring each UFFDIO_CONTINUE (in the post-copy path) to serialize on the same PTL would be problematic in the same way.

Pte-level pgtable lock only covers 2M range, so I think it depends on which is the address that the vcpu is faulted on? IIUC the major case should be that the faulted threads are not falling upon the same 2M range.

Quoted 11 lines
Quoted 8 lines

AFAIU 4K-only solution should only reduce any lock contention because locks will always be pte-level if VM_HUGETLB_HGM set. When walking and creating the intermediate pgtable entries we can use atomic ops just like generic mm, so no lock needed at all. With uncertainty on the size of mappings, we'll need to take any of the multiple layers of locks.

Other than taking the HugeTLB VMA lock for reading, walking/allocating page tables won't need any additional locking.

Actually when revisiting the locks I'm getting a bit confused on whether the vma lock is needed if pmd sharing is anyway forbidden for HGM. I raised a question in the other patch of MADV_COLLAPSE, maybe they're related questions so we can keep it there.

Quoted 4 lines

We take the PTL to allocate the next level down, but so does generic mm (look at __pud_alloc, __pmd_alloc for example). Maybe I am misunderstanding.

Sorry you're right, please ignore that. I don't know why I had that impression that spinlocks are not needed in that process.

Actually I am also curious why atomics won't work (by holding mmap read lock, then do cmpxchg(old_entry=0, new_entry) upon the pgtable entries). I think it's possible I just missed something else.

Signature · 1 line
Peter Xu
#58 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 16 lines · Peter Xu
Quoted 14 lines · James Houghton
Quoted 12 lines · Peter Xu
Quoted 6 lines · Mike Kravetz

I do not know much about the (primary) live migration use case. My guess is that page table lock contention may be an issue? In this use case, HGM is only enabled for the duration the live migration operation, then a MADV_COLLAPSE is performed. If contention is likely to be an issue during this time, then yes we would need to pass around with something like hugetlb_pte.

I'm not aware of any such contention issue. IMHO the migration problem is majorly about being too slow transferring a page being so large. Shrinking the page size should resolve the major problem already here IIUC.

Quoted 7 lines
Quoted 1 line

This will be problematic if you scale up VMs to be quite large.

Do you mean that for the postcopy use case one can leverage e.g. 2M mappings (over 1G) to avoid lock contentions when VM is large I agree it should be more efficient than having 512 4K page installed, but I think it'll make the page fault resolution slower too if some thead is only looking for a 4k portion of it.

No, that's not what I meant. Sorry. If you can use the PTL that is normally used for 4K PTEs, then you're right, there is no contention problem. However, this PTL is determined by the value of the PMD, so you need a pointer to the PMD to determine what the PTL should be (or a pointer to the PTL itself).

In hugetlb, we only ever pass around the PTE pointer, and we rely on huge_pte_lockptr() to find the PTL for us (and it does so appropriately for everything except 4K PTEs). We would need to add the complexity of passing around a PMD or PTL everywhere, and that's what hugetlb_pte does for us. So that complexity is basically unavoidable, unless you're ok with 4K PTEs with taking mm->page_table_lock (I'm not).

Quoted 9 lines
Quoted 5 lines

Google upstreamed the "TDP MMU" for KVM/x86 that removed the need to take the MMU lock for writing in the EPT violation path. We found that this change is required for VMs >200 or so vCPUs to consistently avoid CPU soft lockups in the guest.

After the kvm mmu rwlock convertion, it'll allow concurrent page faults even if only 4K pages are used, so it seems not directly relevant to what we're discussing here, no?

Right. I was just bringing it up to say that if 4K PTLs were mm->page_table_lock, we would have a problem.

Quoted 8 lines
Quoted 4 lines

Requiring each UFFDIO_CONTINUE (in the post-copy path) to serialize on the same PTL would be problematic in the same way.

Pte-level pgtable lock only covers 2M range, so I think it depends on which is the address that the vcpu is faulted on? IIUC the major case should be that the faulted threads are not falling upon the same 2M range.

Right. I think my comment should make more sense with the above clarification.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

AFAIU 4K-only solution should only reduce any lock contention because locks will always be pte-level if VM_HUGETLB_HGM set. When walking and creating the intermediate pgtable entries we can use atomic ops just like generic mm, so no lock needed at all. With uncertainty on the size of mappings, we'll need to take any of the multiple layers of locks.

Quoted 7 lines
Quoted 2 lines

Other than taking the HugeTLB VMA lock for reading, walking/allocating page tables won't need any additional locking.

Actually when revisiting the locks I'm getting a bit confused on whether the vma lock is needed if pmd sharing is anyway forbidden for HGM. I raised a question in the other patch of MADV_COLLAPSE, maybe they're related questions so we can keep it there.

We can discuss there. :) I take both the VMA lock and mapping lock so that it can stay in sync with huge_pmd_unshare(), and so HGM walks have the same synchronization as regular hugetlb PT walks.

Quoted 6 lines
Quoted 5 lines

We take the PTL to allocate the next level down, but so does generic mm (look at __pud_alloc, __pmd_alloc for example). Maybe I am misunderstanding.

Quoted 6 lines

Sorry you're right, please ignore that. I don't know why I had that impression that spinlocks are not needed in that process.

Actually I am also curious why atomics won't work (by holding mmap read lock, then do cmpxchg(old_entry=0, new_entry) upon the pgtable entries). I think it's possible I just missed something else.

I think there are cases where we need to make sure the value of a PTE isn't going to change from under us while we're doing some kind of other operation, and so a compare-and-swap won't really be what we need.

reviewer
#59 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 45 lines · James Houghton
Quoted 16 lines · Peter Xu
Quoted 14 lines · James Houghton
Quoted 12 lines · Peter Xu
Quoted 6 lines · Mike Kravetz

I do not know much about the (primary) live migration use case. My guess is that page table lock contention may be an issue? In this use case, HGM is only enabled for the duration the live migration operation, then a MADV_COLLAPSE is performed. If contention is likely to be an issue during this time, then yes we would need to pass around with something like hugetlb_pte.

I'm not aware of any such contention issue. IMHO the migration problem is majorly about being too slow transferring a page being so large. Shrinking the page size should resolve the major problem already here IIUC.

Quoted 7 lines
Quoted 1 line

This will be problematic if you scale up VMs to be quite large.

Do you mean that for the postcopy use case one can leverage e.g. 2M mappings (over 1G) to avoid lock contentions when VM is large I agree it should be more efficient than having 512 4K page installed, but I think it'll make the page fault resolution slower too if some thead is only looking for a 4k portion of it.

No, that's not what I meant. Sorry. If you can use the PTL that is normally used for 4K PTEs, then you're right, there is no contention problem. However, this PTL is determined by the value of the PMD, so you need a pointer to the PMD to determine what the PTL should be (or a pointer to the PTL itself).

In hugetlb, we only ever pass around the PTE pointer, and we rely on huge_pte_lockptr() to find the PTL for us (and it does so appropriately for everything except 4K PTEs). We would need to add the complexity of passing around a PMD or PTL everywhere, and that's what hugetlb_pte does for us. So that complexity is basically unavoidable, unless you're ok with 4K PTEs with taking mm->page_table_lock (I'm not).

Quoted 6 lines
Quoted 5 lines

Google upstreamed the "TDP MMU" for KVM/x86 that removed the need to take the MMU lock for writing in the EPT violation path. We found that this change is required for VMs >200 or so vCPUs to consistently avoid CPU soft lockups in the guest.

Quoted 6 lines
Quoted 3 lines

After the kvm mmu rwlock convertion, it'll allow concurrent page faults even if only 4K pages are used, so it seems not directly relevant to what we're discussing here, no?

Right. I was just bringing it up to say that if 4K PTLs were mm->page_table_lock, we would have a problem.

Ah I see what you meant. We definitely don't want to use the page_table_lock for sure.

So if it's about keeping hugetlb_pte I'm fine with it, no matter what the final version will look like.

Quoted 25 lines
Quoted 9 lines
Quoted 5 lines

Requiring each UFFDIO_CONTINUE (in the post-copy path) to serialize on the same PTL would be problematic in the same way.

Pte-level pgtable lock only covers 2M range, so I think it depends on which is the address that the vcpu is faulted on? IIUC the major case should be that the faulted threads are not falling upon the same 2M range.

Right. I think my comment should make more sense with the above clarification.

Quoted 13 lines
Quoted 12 lines
Quoted 9 lines

AFAIU 4K-only solution should only reduce any lock contention because locks will always be pte-level if VM_HUGETLB_HGM set. When walking and creating the intermediate pgtable entries we can use atomic ops just like generic mm, so no lock needed at all. With uncertainty on the size of mappings, we'll need to take any of the multiple layers of locks.

Other than taking the HugeTLB VMA lock for reading, walking/allocating page tables won't need any additional locking.

Quoted 8 lines
Quoted 4 lines

Actually when revisiting the locks I'm getting a bit confused on whether the vma lock is needed if pmd sharing is anyway forbidden for HGM. I raised a question in the other patch of MADV_COLLAPSE, maybe they're related questions so we can keep it there.

We can discuss there. :) I take both the VMA lock and mapping lock so that it can stay in sync with huge_pmd_unshare(), and so HGM walks have the same synchronization as regular hugetlb PT walks.

Sure. :)

Now after a 2nd thought I don't think it's unsafe to take the vma write lock here, especially for VM_SHARED. I can't think of anything that will go wrong. It's because we need the vma lock anywhere we'll be walking the pgtables when having mmap_sem read I think, being afraid of having pmd sharing being possible.

But I'm not sure whether this is the cleanest way to do it.

IMHO the major special part of hugetlb comparing to generic mm on pgtable thread safety. I worry that complicating this lock can potentially make the hugetlb code even more specific, which is not good for the long term if we still have a hope of merging more hugetlb codes with the generic paths.

Here since pmd sharing is impossible for HGM, the original vma lock is not needed here. Meanwhile, what we want to guard is the pgtable walkers. They're logically being protected by either mmap lock or the mapping lock (for rmap walkers). Fast-gup is another thing but so far I think it's all safe when you're following the mmu gather facilities.

Somehow I had a feeling that the hugetlb vma lock (along with the pgtable sharing explorations in the hugetlb world keeping going..) can keep evolving in the future, and it should be helpful to keep its semantics simple too.

So to summarize: I wonder whether we can use mmap write lock and i_mmap_rwsem write lock to protect collapsing for hugetlb, just like what we do with THP collapsing (after Jann's fix).

madvise_need_mmap_write() is not easily feasible because it's before the vma scanning so we can't take conditional write lock only for hugetlb, but that's the next question to ask only if we can reach a consensus on the lock scheme first for HGM in general.

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

We take the PTL to allocate the next level down, but so does generic mm (look at __pud_alloc, __pmd_alloc for example). Maybe I am misunderstanding.

Sorry you're right, please ignore that. I don't know why I had that impression that spinlocks are not needed in that process.

Quoted 8 lines
Quoted 3 lines

Actually I am also curious why atomics won't work (by holding mmap read lock, then do cmpxchg(old_entry=0, new_entry) upon the pgtable entries). I think it's possible I just missed something else.

I think there are cases where we need to make sure the value of a PTE isn't going to change from under us while we're doing some kind of other operation, and so a compare-and-swap won't really be what we need.

Currently the pgtable spinlock is only taken during populating the pgtables. If it can happen, then it can happen too right after we release the spinlock in e.g. __pmd_alloc().

One thing I can think of is we need more things done rather than the pgtable entry installations so atomics will stop working if so. E.g. on x86 we have paravirt_alloc_pmd(). But I'm not sure whether that's the only reason.

Signature · 1 line
Peter Xu
#60 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

In reply to Peter Xu
Quoted 5 lines · Peter Xu

Actually when revisiting the locks I'm getting a bit confused on whether the vma lock is needed if pmd sharing is anyway forbidden for HGM. I raised a question in the other patch of MADV_COLLAPSE, maybe they're related questions so we can keep it there.

I can quickly answer that. Yes. The vma lock is also being used for fault/truncation synchronization. Commit e700898fa075 make sure it is even used on architectures that do not support PMD sharing.

I had come up with a rather ugly method of using the fault mutex for fault/truncation synchronization, but using the vma lock was more elegant.

Signature · 1 line
Mike Kravetz
reviewer
#61 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Hi, Mike,

Quoted 7 lines · Mike Kravetz
Quoted 5 lines · Peter Xu

Actually when revisiting the locks I'm getting a bit confused on whether the vma lock is needed if pmd sharing is anyway forbidden for HGM. I raised a question in the other patch of MADV_COLLAPSE, maybe they're related questions so we can keep it there.

Quoted 7 lines

I can quickly answer that. Yes. The vma lock is also being used for fault/truncation synchronization. Commit e700898fa075 make sure it is even used on architectures that do not support PMD sharing.

I had come up with a rather ugly method of using the fault mutex for fault/truncation synchronization, but using the vma lock was more elegant.

Thanks for answering, I'll need to read some more on truncation later. Before that, since COLLAPSE will already require the i_mmap_rwsem write lock already, does it mean it is naturally race-free against truncation even without vma lock?

Signature · 1 line
Peter Xu
#62 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 14 lines · Peter Xu

Hi, Mike,

Quoted 11 lines · Mike Kravetz
Quoted 5 lines · Peter Xu

Actually when revisiting the locks I'm getting a bit confused on whether the vma lock is needed if pmd sharing is anyway forbidden for HGM. I raised a question in the other patch of MADV_COLLAPSE, maybe they're related questions so we can keep it there.

I can quickly answer that. Yes. The vma lock is also being used for fault/truncation synchronization. Commit e700898fa075 make sure it is even used on architectures that do not support PMD sharing.

Quoted 8 lines
Quoted 3 lines

I had come up with a rather ugly method of using the fault mutex for fault/truncation synchronization, but using the vma lock was more elegant.

Thanks for answering, I'll need to read some more on truncation later. Before that, since COLLAPSE will already require the i_mmap_rwsem write lock already, does it mean it is naturally race-free against truncation even without vma lock?

Yes, and thanks for making me take a closer look at COLLAPSE. :)

Signature · 1 line
Mike Kravetz
#63 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

In reply to James Houghton
Quoted 10 lines · James Houghton
Quoted 8 lines · Mike Kravetz
Quoted 5 lines · James Houghton

FWIW, what makes the most sense to me right now is to implement the THP-like scheme and mark HGM as mutually exclusive with the vmemmap optimization. We can later come up with a scheme that lets us retain compatibility. (Is that what you mean by "this can be done somewhat independently", Mike?)

Quoted 6 lines
Quoted 3 lines

Sort of, I was only saying that getting the ref/map counting right seems like a task than can be independently worked. Using the THP-like scheme is good.

Ok! So if you're ok with the intermediate mapping sizes, it sounds like I should go ahead and implement the THP-like scheme.

It turns out that the THP-like scheme significantly slows down
MADV_COLLAPSE: decrementing the mapcounts for the 4K subpages becomes the vast majority of the time spent in MADV_COLLAPSE when collapsing 1G mappings. It is doing 262k atomic decrements, so this makes sense.

This is only really a problem because this is done between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), so KVM won't allow vCPUs to access any of the 1G page while we're doing this (and it can take like ~1 second for each 1G, at least on the x86 server I was testing on).

- James

reviewer
#64 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

James,

Quoted 10 lines · James Houghton

It turns out that the THP-like scheme significantly slows down
MADV_COLLAPSE: decrementing the mapcounts for the 4K subpages becomes the vast majority of the time spent in MADV_COLLAPSE when collapsing 1G mappings. It is doing 262k atomic decrements, so this makes sense.

This is only really a problem because this is done between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), so KVM won't allow vCPUs to access any of the 1G page while we're doing this (and it can take like ~1 second for each 1G, at least on the x86 server I was testing on).

Did you try to measure the time, or it's a quick observation from perf?

IIRC I used to measure some atomic ops, it is not as drastic as I thought. But maybe it depends on many things.

I'm curious how the 1sec is provisioned between the procedures. E.g., I would expect mmu_notifier_invalidate_range_start() to also take some time too as it should walk the smally mapped EPT pgtables.

Since we'll still keep the intermediate levels around - from application POV, one other thing to remedy this is further shrink the size of COLLAPSE so potentially for a very large page we can start with building 2M layers. But then collapse will need to be run at least two rounds.

Signature · 1 line
Peter Xu
#65 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 9 lines · Peter Xu

James,

Quoted 5 lines · James Houghton

It turns out that the THP-like scheme significantly slows down
MADV_COLLAPSE: decrementing the mapcounts for the 4K subpages becomes the vast majority of the time spent in MADV_COLLAPSE when collapsing 1G mappings. It is doing 262k atomic decrements, so this makes sense.

Quoted 7 lines
Quoted 5 lines

This is only really a problem because this is done between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), so KVM won't allow vCPUs to access any of the 1G page while we're doing this (and it can take like ~1 second for each 1G, at least on the x86 server I was testing on).

Did you try to measure the time, or it's a quick observation from perf?

I put some ktime_get()s in.

Quoted 7 lines

IIRC I used to measure some atomic ops, it is not as drastic as I thought. But maybe it depends on many things.

I'm curious how the 1sec is provisioned between the procedures. E.g., I would expect mmu_notifier_invalidate_range_start() to also take some time too as it should walk the smally mapped EPT pgtables.

Somehow this doesn't take all that long (only like 10-30ms when collapsing from 4K -> 1G) compared to hugetlb_collapse().

Quoted 5 lines

Since we'll still keep the intermediate levels around - from application POV, one other thing to remedy this is further shrink the size of COLLAPSE so potentially for a very large page we can start with building 2M layers. But then collapse will need to be run at least two rounds.

That's exactly what I thought to do. :) I realized, too, that this is actually how userspace *should* collapse things to avoid holding up vCPUs too long. I think this is a good reason to keep intermediate page sizes.

When collapsing 4K -> 1G, the mapcount scheme doesn't actually make a huge difference: the THP-like scheme is about 30% slower overall.

When collapsing 4K -> 2M -> 1G, the mapcount scheme makes a HUGE difference. For the THP-like scheme, collapsing 4K -> 2M requires decrementing and then re-incrementing subpage->_mapcount, and then from 2M -> 1G, we have to decrement all 262k subpages->_mapcount. For the head-only scheme, for each 2M in the 4K -> 2M collapse, we decrement the compound_mapcount 512 times (once per PTE), then increment it once. And then for 2M -> 1G, for each 1G, we decrement mapcount again by 512 (once per PMD), incrementing it once.

The mapcount decrements are about on par with how long it takes to do other things, like updating page tables. The main problem is, with the THP-like scheme (implemented like this [1]), there isn't a way to avoid the 262k decrements when collapsing 1G. So if we want MADV_COLLAPSE to be fast and we want a THP-like page_mapcount() API, then I think something more clever needs to be implemented.

[1]: https://github.com/48ca/linux/blob/hgmv2-jan24/mm/hugetlb.c#L127-L178

- James

reviewer
#66 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 24 lines · James Houghton
Quoted 9 lines · Peter Xu

James,

Quoted 5 lines · James Houghton

It turns out that the THP-like scheme significantly slows down
MADV_COLLAPSE: decrementing the mapcounts for the 4K subpages becomes the vast majority of the time spent in MADV_COLLAPSE when collapsing 1G mappings. It is doing 262k atomic decrements, so this makes sense.

Quoted 7 lines
Quoted 5 lines

This is only really a problem because this is done between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), so KVM won't allow vCPUs to access any of the 1G page while we're doing this (and it can take like ~1 second for each 1G, at least on the x86 server I was testing on).

Did you try to measure the time, or it's a quick observation from perf?

I put some ktime_get()s in.

Quoted 4 lines

IIRC I used to measure some atomic ops, it is not as drastic as I thought. But maybe it depends on many things.

Quoted 6 lines
Quoted 3 lines

I'm curious how the 1sec is provisioned between the procedures. E.g., I would expect mmu_notifier_invalidate_range_start() to also take some time too as it should walk the smally mapped EPT pgtables.

Somehow this doesn't take all that long (only like 10-30ms when collapsing from 4K -> 1G) compared to hugetlb_collapse().

Did you populate as much the EPT pgtable when measuring this?

IIUC this number should be pretty much relevant to how many pages are shadowed to the kvm pgtables. If the EPT table is mostly empty it should be super fast, but OTOH it can be much slower if when it's populated, because tdp mmu should need to handle the pgtable leaves one by one.

E.g. it should be fully populated if you have a program busy dirtying most of the guest pages during test migration.

Write op should be the worst here case since it'll require the atomic op being applied; see kvm_tdp_mmu_write_spte().

Quoted 15 lines
Quoted 6 lines

Since we'll still keep the intermediate levels around - from application POV, one other thing to remedy this is further shrink the size of COLLAPSE so potentially for a very large page we can start with building 2M layers. But then collapse will need to be run at least two rounds.

That's exactly what I thought to do. :) I realized, too, that this is actually how userspace *should* collapse things to avoid holding up vCPUs too long. I think this is a good reason to keep intermediate page sizes.

When collapsing 4K -> 1G, the mapcount scheme doesn't actually make a huge difference: the THP-like scheme is about 30% slower overall.

Quoted 8 lines

When collapsing 4K -> 2M -> 1G, the mapcount scheme makes a HUGE difference. For the THP-like scheme, collapsing 4K -> 2M requires decrementing and then re-incrementing subpage->_mapcount, and then from 2M -> 1G, we have to decrement all 262k subpages->_mapcount. For the head-only scheme, for each 2M in the 4K -> 2M collapse, we decrement the compound_mapcount 512 times (once per PTE), then increment it once. And then for 2M -> 1G, for each 1G, we decrement mapcount again by 512 (once per PMD), incrementing it once.

Did you have quantified numbers (with your ktime treak) to compare these? If we want to go the other route, I think these will be materials to justify any other approach on mapcount handling.

Quoted 9 lines

The mapcount decrements are about on par with how long it takes to do other things, like updating page tables. The main problem is, with the THP-like scheme (implemented like this [1]), there isn't a way to avoid the 262k decrements when collapsing 1G. So if we want MADV_COLLAPSE to be fast and we want a THP-like page_mapcount() API, then I think something more clever needs to be implemented.

[1]: https://github.com/48ca/linux/blob/hgmv2-jan24/mm/hugetlb.c#L127-L178

I believe the whole goal of HGM is trying to face the same challenge if we'll allow 1G THP exist and being able to split for anon.

I don't remember whether we discussed below, maybe we did? Anyway...

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Again, in all cases I think some solid measurements would definitely be helpful (as commented above) to see how much overhead will there be and whether that'll start to become a problem at least for the current motivations of the whole HGM idea.

Thanks,

Signature · 1 line
Peter Xu
#67 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 35 lines · Peter Xu
Quoted 24 lines · James Houghton
Quoted 9 lines · Peter Xu

James,

Quoted 5 lines · James Houghton

It turns out that the THP-like scheme significantly slows down
MADV_COLLAPSE: decrementing the mapcounts for the 4K subpages becomes the vast majority of the time spent in MADV_COLLAPSE when collapsing 1G mappings. It is doing 262k atomic decrements, so this makes sense.

Quoted 7 lines
Quoted 5 lines

This is only really a problem because this is done between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), so KVM won't allow vCPUs to access any of the 1G page while we're doing this (and it can take like ~1 second for each 1G, at least on the x86 server I was testing on).

Did you try to measure the time, or it's a quick observation from perf?

I put some ktime_get()s in.

Quoted 4 lines

IIRC I used to measure some atomic ops, it is not as drastic as I thought. But maybe it depends on many things.

Quoted 6 lines
Quoted 3 lines

I'm curious how the 1sec is provisioned between the procedures. E.g., I would expect mmu_notifier_invalidate_range_start() to also take some time too as it should walk the smally mapped EPT pgtables.

Somehow this doesn't take all that long (only like 10-30ms when collapsing from 4K -> 1G) compared to hugetlb_collapse().

Did you populate as much the EPT pgtable when measuring this?

Quoted 7 lines

IIUC this number should be pretty much relevant to how many pages are shadowed to the kvm pgtables. If the EPT table is mostly empty it should be super fast, but OTOH it can be much slower if when it's populated, because tdp mmu should need to handle the pgtable leaves one by one.

E.g. it should be fully populated if you have a program busy dirtying most of the guest pages during test migration.

That's what I was doing. I was running a workload in the guest that just writes 8 bytes to a page and jumps ahead a few pages on all vCPUs, touching most of its memory.

But there is more to understand; I'll collect more results. I'm not sure why the EPT can be unmapped/collapsed so quickly.

Quoted 28 lines

Write op should be the worst here case since it'll require the atomic op being applied; see kvm_tdp_mmu_write_spte().

Quoted 15 lines
Quoted 6 lines

Since we'll still keep the intermediate levels around - from application POV, one other thing to remedy this is further shrink the size of COLLAPSE so potentially for a very large page we can start with building 2M layers. But then collapse will need to be run at least two rounds.

That's exactly what I thought to do. :) I realized, too, that this is actually how userspace *should* collapse things to avoid holding up vCPUs too long. I think this is a good reason to keep intermediate page sizes.

When collapsing 4K -> 1G, the mapcount scheme doesn't actually make a huge difference: the THP-like scheme is about 30% slower overall.

Quoted 8 lines

When collapsing 4K -> 2M -> 1G, the mapcount scheme makes a HUGE difference. For the THP-like scheme, collapsing 4K -> 2M requires decrementing and then re-incrementing subpage->_mapcount, and then from 2M -> 1G, we have to decrement all 262k subpages->_mapcount. For the head-only scheme, for each 2M in the 4K -> 2M collapse, we decrement the compound_mapcount 512 times (once per PTE), then increment it once. And then for 2M -> 1G, for each 1G, we decrement mapcount again by 512 (once per PMD), incrementing it once.

Quoted 3 lines

Did you have quantified numbers (with your ktime treak) to compare these? If we want to go the other route, I think these will be materials to justify any other approach on mapcount handling.

Ok, I can do that. GIve me a couple days to collect more results and organize them in a helpful way.

(If it's helpful at all, here are some results I collected last week:
[2]. Please ignore it if it's not helpful.)

Quoted 14 lines
Quoted 10 lines

The mapcount decrements are about on par with how long it takes to do other things, like updating page tables. The main problem is, with the THP-like scheme (implemented like this [1]), there isn't a way to avoid the 262k decrements when collapsing 1G. So if we want MADV_COLLAPSE to be fast and we want a THP-like page_mapcount() API, then I think something more clever needs to be implemented.

[1]: https://github.com/48ca/linux/blob/hgmv2-jan24/mm/hugetlb.c#L127-L178

I believe the whole goal of HGM is trying to face the same challenge if we'll allow 1G THP exist and being able to split for anon.

Quoted 8 lines

I don't remember whether we discussed below, maybe we did? Anyway...

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

Quoted 11 lines

Again, in all cases I think some solid measurements would definitely be helpful (as commented above) to see how much overhead will there be and whether that'll start to become a problem at least for the current motivations of the whole HGM idea.

Thanks,

--
Peter Xu

Thanks, Peter!

[2]: https://pastebin.com/raw/DVfNFi2m

- James

reviewer
#68 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 44 lines · James Houghton
Quoted 35 lines · Peter Xu
Quoted 24 lines · James Houghton
Quoted 9 lines · Peter Xu

James,

Quoted 5 lines · James Houghton

It turns out that the THP-like scheme significantly slows down
MADV_COLLAPSE: decrementing the mapcounts for the 4K subpages becomes the vast majority of the time spent in MADV_COLLAPSE when collapsing 1G mappings. It is doing 262k atomic decrements, so this makes sense.

Quoted 7 lines
Quoted 5 lines

This is only really a problem because this is done between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), so KVM won't allow vCPUs to access any of the 1G page while we're doing this (and it can take like ~1 second for each 1G, at least on the x86 server I was testing on).

Did you try to measure the time, or it's a quick observation from perf?

I put some ktime_get()s in.

Quoted 4 lines

IIRC I used to measure some atomic ops, it is not as drastic as I thought. But maybe it depends on many things.

Quoted 6 lines
Quoted 3 lines

I'm curious how the 1sec is provisioned between the procedures. E.g., I would expect mmu_notifier_invalidate_range_start() to also take some time too as it should walk the smally mapped EPT pgtables.

Somehow this doesn't take all that long (only like 10-30ms when collapsing from 4K -> 1G) compared to hugetlb_collapse().

Did you populate as much the EPT pgtable when measuring this?

Quoted 7 lines

IIUC this number should be pretty much relevant to how many pages are shadowed to the kvm pgtables. If the EPT table is mostly empty it should be super fast, but OTOH it can be much slower if when it's populated, because tdp mmu should need to handle the pgtable leaves one by one.

E.g. it should be fully populated if you have a program busy dirtying most of the guest pages during test migration.

Quoted 6 lines

That's what I was doing. I was running a workload in the guest that just writes 8 bytes to a page and jumps ahead a few pages on all vCPUs, touching most of its memory.

But there is more to understand; I'll collect more results. I'm not sure why the EPT can be unmapped/collapsed so quickly.

Maybe something smart done by the hypervisor?

Quoted 33 lines
Quoted 29 lines

Write op should be the worst here case since it'll require the atomic op being applied; see kvm_tdp_mmu_write_spte().

Quoted 15 lines
Quoted 6 lines

Since we'll still keep the intermediate levels around - from application POV, one other thing to remedy this is further shrink the size of COLLAPSE so potentially for a very large page we can start with building 2M layers. But then collapse will need to be run at least two rounds.

That's exactly what I thought to do. :) I realized, too, that this is actually how userspace *should* collapse things to avoid holding up vCPUs too long. I think this is a good reason to keep intermediate page sizes.

When collapsing 4K -> 1G, the mapcount scheme doesn't actually make a huge difference: the THP-like scheme is about 30% slower overall.

Quoted 8 lines

When collapsing 4K -> 2M -> 1G, the mapcount scheme makes a HUGE difference. For the THP-like scheme, collapsing 4K -> 2M requires decrementing and then re-incrementing subpage->_mapcount, and then from 2M -> 1G, we have to decrement all 262k subpages->_mapcount. For the head-only scheme, for each 2M in the 4K -> 2M collapse, we decrement the compound_mapcount 512 times (once per PTE), then increment it once. And then for 2M -> 1G, for each 1G, we decrement mapcount again by 512 (once per PMD), incrementing it once.

Quoted 3 lines

Did you have quantified numbers (with your ktime treak) to compare these? If we want to go the other route, I think these will be materials to justify any other approach on mapcount handling.

Quoted 5 lines

Ok, I can do that. GIve me a couple days to collect more results and organize them in a helpful way.

(If it's helpful at all, here are some results I collected last week:
[2]. Please ignore it if it's not helpful.)

It's helpful already at least to me, thanks. Yes the change is drastic.

Quoted 24 lines
Quoted 15 lines
Quoted 11 lines

The mapcount decrements are about on par with how long it takes to do other things, like updating page tables. The main problem is, with the THP-like scheme (implemented like this [1]), there isn't a way to avoid the 262k decrements when collapsing 1G. So if we want MADV_COLLAPSE to be fast and we want a THP-like page_mapcount() API, then I think something more clever needs to be implemented.

[1]: https://github.com/48ca/linux/blob/hgmv2-jan24/mm/hugetlb.c#L127-L178

I believe the whole goal of HGM is trying to face the same challenge if we'll allow 1G THP exist and being able to split for anon.

Quoted 8 lines

I don't remember whether we discussed below, maybe we did? Anyway...

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Quoted 6 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Signature · 1 line
Peter Xu
#69 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 46 lines · Peter Xu
Quoted 44 lines · James Houghton
Quoted 42 lines · Peter Xu
Quoted 24 lines · James Houghton
Quoted 9 lines · Peter Xu

James,

Quoted 5 lines · James Houghton

It turns out that the THP-like scheme significantly slows down
MADV_COLLAPSE: decrementing the mapcounts for the 4K subpages becomes the vast majority of the time spent in MADV_COLLAPSE when collapsing 1G mappings. It is doing 262k atomic decrements, so this makes sense.

Quoted 7 lines
Quoted 5 lines

This is only really a problem because this is done between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), so KVM won't allow vCPUs to access any of the 1G page while we're doing this (and it can take like ~1 second for each 1G, at least on the x86 server I was testing on).

Did you try to measure the time, or it's a quick observation from perf?

I put some ktime_get()s in.

Quoted 4 lines

IIRC I used to measure some atomic ops, it is not as drastic as I thought. But maybe it depends on many things.

Quoted 6 lines
Quoted 3 lines

I'm curious how the 1sec is provisioned between the procedures. E.g., I would expect mmu_notifier_invalidate_range_start() to also take some time too as it should walk the smally mapped EPT pgtables.

Somehow this doesn't take all that long (only like 10-30ms when collapsing from 4K -> 1G) compared to hugetlb_collapse().

Did you populate as much the EPT pgtable when measuring this?

IIUC this number should be pretty much relevant to how many pages are shadowed to the kvm pgtables. If the EPT table is mostly empty it should be super fast, but OTOH it can be much slower if when it's populated, because tdp mmu should need to handle the pgtable leaves one by one.

E.g. it should be fully populated if you have a program busy dirtying most of the guest pages during test migration.

Quoted 8 lines
Quoted 6 lines

That's what I was doing. I was running a workload in the guest that just writes 8 bytes to a page and jumps ahead a few pages on all vCPUs, touching most of its memory.

But there is more to understand; I'll collect more results. I'm not sure why the EPT can be unmapped/collapsed so quickly.

Maybe something smart done by the hypervisor?

Doing a little bit more digging, it looks like the invalidate_range_start notifier clears the sptes, and then later on (on the next EPT violation), the page tables are freed. I still need to look at how they end up being so much faster still, but I thought that was interesting.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines

Write op should be the worst here case since it'll require the atomic op being applied; see kvm_tdp_mmu_write_spte().

Quoted 15 lines
Quoted 6 lines

Since we'll still keep the intermediate levels around - from application POV, one other thing to remedy this is further shrink the size of COLLAPSE so potentially for a very large page we can start with building 2M layers. But then collapse will need to be run at least two rounds.

That's exactly what I thought to do. :) I realized, too, that this is actually how userspace *should* collapse things to avoid holding up vCPUs too long. I think this is a good reason to keep intermediate page sizes.

When collapsing 4K -> 1G, the mapcount scheme doesn't actually make a huge difference: the THP-like scheme is about 30% slower overall.

Quoted 8 lines

When collapsing 4K -> 2M -> 1G, the mapcount scheme makes a HUGE difference. For the THP-like scheme, collapsing 4K -> 2M requires decrementing and then re-incrementing subpage->_mapcount, and then from 2M -> 1G, we have to decrement all 262k subpages->_mapcount. For the head-only scheme, for each 2M in the 4K -> 2M collapse, we decrement the compound_mapcount 512 times (once per PTE), then increment it once. And then for 2M -> 1G, for each 1G, we decrement mapcount again by 512 (once per PMD), incrementing it once.

Did you have quantified numbers (with your ktime treak) to compare these? If we want to go the other route, I think these will be materials to justify any other approach on mapcount handling.

Quoted 7 lines
Quoted 5 lines

Ok, I can do that. GIve me a couple days to collect more results and organize them in a helpful way.

(If it's helpful at all, here are some results I collected last week:
[2]. Please ignore it if it's not helpful.)

It's helpful already at least to me, thanks. Yes the change is drastic.

That data only contains THP-like mapcount performance, no performance for the head-only way. But the head-only scheme makes the 2M -> 1G very good ("56" comes down to about the same everything else, instead of being ~100-500x bigger).

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 4 lines
Quoted 8 lines

The mapcount decrements are about on par with how long it takes to do other things, like updating page tables. The main problem is, with the THP-like scheme (implemented like this [1]), there isn't a way to avoid the 262k decrements when collapsing 1G. So if we want MADV_COLLAPSE to be fast and we want a THP-like page_mapcount() API, then I think something more clever needs to be implemented.

[1]: https://github.com/48ca/linux/blob/hgmv2-jan24/mm/hugetlb.c#L127-L178

I believe the whole goal of HGM is trying to face the same challenge if we'll allow 1G THP exist and being able to split for anon.

I don't remember whether we discussed below, maybe we did? Anyway...

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 11 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

I don't fully understand what you mean here.

Quoted 10 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

reviewer
#70 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 102 lines · James Houghton
Quoted 46 lines · Peter Xu
Quoted 44 lines · James Houghton
Quoted 42 lines · Peter Xu
Quoted 24 lines · James Houghton
Quoted 9 lines · Peter Xu

James,

Quoted 5 lines · James Houghton

It turns out that the THP-like scheme significantly slows down
MADV_COLLAPSE: decrementing the mapcounts for the 4K subpages becomes the vast majority of the time spent in MADV_COLLAPSE when collapsing 1G mappings. It is doing 262k atomic decrements, so this makes sense.

Quoted 7 lines
Quoted 5 lines

This is only really a problem because this is done between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), so KVM won't allow vCPUs to access any of the 1G page while we're doing this (and it can take like ~1 second for each 1G, at least on the x86 server I was testing on).

Did you try to measure the time, or it's a quick observation from perf?

I put some ktime_get()s in.

Quoted 4 lines

IIRC I used to measure some atomic ops, it is not as drastic as I thought. But maybe it depends on many things.

Quoted 6 lines
Quoted 3 lines

I'm curious how the 1sec is provisioned between the procedures. E.g., I would expect mmu_notifier_invalidate_range_start() to also take some time too as it should walk the smally mapped EPT pgtables.

Somehow this doesn't take all that long (only like 10-30ms when collapsing from 4K -> 1G) compared to hugetlb_collapse().

Did you populate as much the EPT pgtable when measuring this?

IIUC this number should be pretty much relevant to how many pages are shadowed to the kvm pgtables. If the EPT table is mostly empty it should be super fast, but OTOH it can be much slower if when it's populated, because tdp mmu should need to handle the pgtable leaves one by one.

E.g. it should be fully populated if you have a program busy dirtying most of the guest pages during test migration.

Quoted 8 lines
Quoted 6 lines

That's what I was doing. I was running a workload in the guest that just writes 8 bytes to a page and jumps ahead a few pages on all vCPUs, touching most of its memory.

But there is more to understand; I'll collect more results. I'm not sure why the EPT can be unmapped/collapsed so quickly.

Maybe something smart done by the hypervisor?

Doing a little bit more digging, it looks like the invalidate_range_start notifier clears the sptes, and then later on (on the next EPT violation), the page tables are freed. I still need to look at how they end up being so much faster still, but I thought that was interesting.

Quoted 40 lines
Quoted 39 lines
Quoted 30 lines

Write op should be the worst here case since it'll require the atomic op being applied; see kvm_tdp_mmu_write_spte().

Quoted 15 lines
Quoted 6 lines

Since we'll still keep the intermediate levels around - from application POV, one other thing to remedy this is further shrink the size of COLLAPSE so potentially for a very large page we can start with building 2M layers. But then collapse will need to be run at least two rounds.

That's exactly what I thought to do. :) I realized, too, that this is actually how userspace *should* collapse things to avoid holding up vCPUs too long. I think this is a good reason to keep intermediate page sizes.

When collapsing 4K -> 1G, the mapcount scheme doesn't actually make a huge difference: the THP-like scheme is about 30% slower overall.

Quoted 8 lines

When collapsing 4K -> 2M -> 1G, the mapcount scheme makes a HUGE difference. For the THP-like scheme, collapsing 4K -> 2M requires decrementing and then re-incrementing subpage->_mapcount, and then from 2M -> 1G, we have to decrement all 262k subpages->_mapcount. For the head-only scheme, for each 2M in the 4K -> 2M collapse, we decrement the compound_mapcount 512 times (once per PTE), then increment it once. And then for 2M -> 1G, for each 1G, we decrement mapcount again by 512 (once per PMD), incrementing it once.

Quoted 3 lines

Did you have quantified numbers (with your ktime treak) to compare these? If we want to go the other route, I think these will be materials to justify any other approach on mapcount handling.

Ok, I can do that. GIve me a couple days to collect more results and organize them in a helpful way.

(If it's helpful at all, here are some results I collected last week:
[2]. Please ignore it if it's not helpful.)

Quoted 6 lines
Quoted 1 line

It's helpful already at least to me, thanks. Yes the change is drastic.

That data only contains THP-like mapcount performance, no performance for the head-only way. But the head-only scheme makes the 2M -> 1G very good ("56" comes down to about the same everything else, instead of being ~100-500x bigger).

Oops, I think I misread those. Yeah please keep sharing information if you come up with any.

Quoted 54 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 5 lines
Quoted 8 lines

The mapcount decrements are about on par with how long it takes to do other things, like updating page tables. The main problem is, with the THP-like scheme (implemented like this [1]), there isn't a way to avoid the 262k decrements when collapsing 1G. So if we want MADV_COLLAPSE to be fast and we want a THP-like page_mapcount() API, then I think something more clever needs to be implemented.

[1]: https://github.com/48ca/linux/blob/hgmv2-jan24/mm/hugetlb.c#L127-L178

I believe the whole goal of HGM is trying to face the same challenge if we'll allow 1G THP exist and being able to split for anon.

I don't remember whether we discussed below, maybe we did? Anyway...

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 8 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

Quoted 5 lines
Quoted 3 lines

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

I don't fully understand what you mean here.

I meant the rmap_walk pairing with folio_referenced_one() will walk all the leaves for the folio, big or small. I think that will match the number with what got returned from folio_mapcount().

Quoted 12 lines
Quoted 11 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Quoted 5 lines

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

I didn't quite follow - isn't that already doing so?

folio_mapcount() is total_compound_mapcount() here, IIUC it is an accumulated value of all possible PTEs or PMDs being mapped as long as it's all or part of the folio being mapped.

Signature · 1 line
Peter Xu
#71 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 8 lines · Peter Xu
Quoted 6 lines · James Houghton
Quoted 5 lines · Peter Xu
Quoted 3 lines · James Houghton
Quoted 2 lines · Peter Xu

On Fri, Jan 27, 2023 at 01:02:02PM -0800, James Houghton wrote:

[snip]

Quoted 39 lines
Quoted 39 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 11 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

Quoted 5 lines
Quoted 1 line

I don't fully understand what you mean here.

I meant the rmap_walk pairing with folio_referenced_one() will walk all the leaves for the folio, big or small. I think that will match the number with what got returned from folio_mapcount().

See below.

Quoted 19 lines
Quoted 13 lines
Quoted 9 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

Quoted 3 lines

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Quoted 5 lines

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

Quoted 5 lines

I didn't quite follow - isn't that already doing so?

folio_mapcount() is total_compound_mapcount() here, IIUC it is an accumulated value of all possible PTEs or PMDs being mapped as long as it's all or part of the folio being mapped.

We've talked about 3 ways of handling mapcount:

1. The RFC v2 way, which is head-only, and we increment the compound mapcount for each PT mapping we have. So a PTE-mapped 2M page, compound_mapcount=512, subpage->_mapcount=0 (ignoring the -1 bias).
2. The THP-like way. If we are fully mapping the hugetlb page with the hstate-level PTE, we increment the compound mapcount, otherwise we increment subpage->_mapcount.
3. The RFC v1 way (the way you have suggested above), which is head-only, and we increment the compound mapcount if the hstate-level PTE is made present.

With #1 and #2, there is no concern with folio_mapcount(). But with #3, folio_mapcount() for a PTE-mapped 2M page mapped in a single VMA would yield 1 instead of 512 (right?). That's what I mean.

#1 has problems wrt smaps and migration (though there were other problems with those anyway that Mike has fixed), and #2 makes MADV_COLLAPSE slow to the point of being unusable for some applications.

It seems like the least bad option is #1, but maybe we should have a face-to-face discussion about it? I'm still collecting some more performance numbers.

- James

#72 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 94 lines · James Houghton
Quoted 8 lines · Peter Xu
Quoted 6 lines · James Houghton
Quoted 5 lines · Peter Xu
Quoted 3 lines · James Houghton
Quoted 2 lines · Peter Xu

On Fri, Jan 27, 2023 at 01:02:02PM -0800, James Houghton wrote:

[snip]

Quoted 39 lines
Quoted 39 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 11 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

Quoted 5 lines
Quoted 1 line

I don't fully understand what you mean here.

I meant the rmap_walk pairing with folio_referenced_one() will walk all the leaves for the folio, big or small. I think that will match the number with what got returned from folio_mapcount().

See below.

Quoted 19 lines
Quoted 13 lines
Quoted 9 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

Quoted 3 lines

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Quoted 5 lines

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

Quoted 5 lines

I didn't quite follow - isn't that already doing so?

folio_mapcount() is total_compound_mapcount() here, IIUC it is an accumulated value of all possible PTEs or PMDs being mapped as long as it's all or part of the folio being mapped.

We've talked about 3 ways of handling mapcount:

1. The RFC v2 way, which is head-only, and we increment the compound mapcount for each PT mapping we have. So a PTE-mapped 2M page, compound_mapcount=512, subpage->_mapcount=0 (ignoring the -1 bias).
2. The THP-like way. If we are fully mapping the hugetlb page with the hstate-level PTE, we increment the compound mapcount, otherwise we increment subpage->_mapcount.
3. The RFC v1 way (the way you have suggested above), which is head-only, and we increment the compound mapcount if the hstate-level PTE is made present.

Quoted 3 lines

With #1 and #2, there is no concern with folio_mapcount(). But with #3, folio_mapcount() for a PTE-mapped 2M page mapped in a single VMA would yield 1 instead of 512 (right?). That's what I mean.

My 2 cents:

The mapcount is primarily used (in hugetlb context) to

(a) Detect if a page might be shared. mapcount > 1 implies that two independent page table hierarchies are mapping the page. We care about mapcount == 1 vs. mapcount != 1.

(b) Detect if unmapping was sucessfull. We care about mapcount == 0 vs. mapcount != 0.

For hugetlb, I don't see why we should care about the subpage mapcount at all.

For (a) it's even good to count "somehow mapped into a single page table structure" as "mapcount == 1" For (b), we don't care as long as "still mapped" implies "mapcount != 0".

Signature · 3 lines
Thanks,

David / dhildenb
#73 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 108 lines · David Hildenbrand
Quoted 94 lines · James Houghton
Quoted 8 lines · Peter Xu
Quoted 6 lines · James Houghton
Quoted 5 lines · Peter Xu
Quoted 3 lines · James Houghton
Quoted 2 lines · Peter Xu

On Fri, Jan 27, 2023 at 01:02:02PM -0800, James Houghton wrote:

[snip]

Quoted 39 lines
Quoted 39 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 11 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

Quoted 5 lines
Quoted 1 line

I don't fully understand what you mean here.

I meant the rmap_walk pairing with folio_referenced_one() will walk all the leaves for the folio, big or small. I think that will match the number with what got returned from folio_mapcount().

See below.

Quoted 19 lines
Quoted 13 lines
Quoted 9 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

Quoted 3 lines

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Quoted 5 lines

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

Quoted 5 lines

I didn't quite follow - isn't that already doing so?

folio_mapcount() is total_compound_mapcount() here, IIUC it is an accumulated value of all possible PTEs or PMDs being mapped as long as it's all or part of the folio being mapped.

We've talked about 3 ways of handling mapcount:

1. The RFC v2 way, which is head-only, and we increment the compound mapcount for each PT mapping we have. So a PTE-mapped 2M page, compound_mapcount=512, subpage->_mapcount=0 (ignoring the -1 bias).
2. The THP-like way. If we are fully mapping the hugetlb page with the hstate-level PTE, we increment the compound mapcount, otherwise we increment subpage->_mapcount.
3. The RFC v1 way (the way you have suggested above), which is head-only, and we increment the compound mapcount if the hstate-level PTE is made present.

Quoted 3 lines

With #1 and #2, there is no concern with folio_mapcount(). But with #3, folio_mapcount() for a PTE-mapped 2M page mapped in a single VMA would yield 1 instead of 512 (right?). That's what I mean.

My 2 cents:

The mapcount is primarily used (in hugetlb context) to

(a) Detect if a page might be shared. mapcount > 1 implies that two independent page table hierarchies are mapping the page. We care about mapcount == 1 vs. mapcount != 1.

Quoted 5 lines

(b) Detect if unmapping was sucessfull. We care about mapcount == 0 vs. mapcount != 0.

For hugetlb, I don't see why we should care about the subpage mapcount at all.

Agreed -- it shouldn't really matter all that much.

Quoted 4 lines

For (a) it's even good to count "somehow mapped into a single page table structure" as "mapcount == 1" For (b), we don't care as long as "still mapped" implies "mapcount != 0".

Thanks for your thoughts, David. So it sounds like you're still squarely in the #3 camp. :)

#74 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 31 lines
Quoted 22 lines
Quoted 10 lines

1. The RFC v2 way, which is head-only, and we increment the compound mapcount for each PT mapping we have. So a PTE-mapped 2M page, compound_mapcount=512, subpage->_mapcount=0 (ignoring the -1 bias).
2. The THP-like way. If we are fully mapping the hugetlb page with the hstate-level PTE, we increment the compound mapcount, otherwise we increment subpage->_mapcount.
3. The RFC v1 way (the way you have suggested above), which is head-only, and we increment the compound mapcount if the hstate-level PTE is made present.

Quoted 3 lines

With #1 and #2, there is no concern with folio_mapcount(). But with #3, folio_mapcount() for a PTE-mapped 2M page mapped in a single VMA would yield 1 instead of 512 (right?). That's what I mean.

My 2 cents:

The mapcount is primarily used (in hugetlb context) to

(a) Detect if a page might be shared. mapcount > 1 implies that two independent page table hierarchies are mapping the page. We care about mapcount == 1 vs. mapcount != 1.

Quoted 5 lines

(b) Detect if unmapping was sucessfull. We care about mapcount == 0 vs. mapcount != 0.

For hugetlb, I don't see why we should care about the subpage mapcount at all.

Agreed -- it shouldn't really matter all that much.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

For (a) it's even good to count "somehow mapped into a single page table structure" as "mapcount == 1" For (b), we don't care as long as "still mapped" implies "mapcount != 0".

Thanks for your thoughts, David. So it sounds like you're still squarely in the #3 camp. :)

Well, yes. As long as we can get it implemented in a clean way ... :)

Signature · 3 lines
Thanks,

David / dhildenb
reviewer
#75 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

In reply to James Houghton
Quoted 84 lines · James Houghton
Quoted 8 lines · Peter Xu
Quoted 6 lines · James Houghton
Quoted 5 lines · Peter Xu
Quoted 3 lines · James Houghton
Quoted 2 lines · Peter Xu

On Fri, Jan 27, 2023 at 01:02:02PM -0800, James Houghton wrote:

[snip]

Quoted 39 lines
Quoted 39 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 11 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

Quoted 5 lines
Quoted 1 line

I don't fully understand what you mean here.

I meant the rmap_walk pairing with folio_referenced_one() will walk all the leaves for the folio, big or small. I think that will match the number with what got returned from folio_mapcount().

See below.

Quoted 19 lines
Quoted 13 lines
Quoted 9 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

Quoted 3 lines

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Quoted 5 lines

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

Quoted 5 lines

I didn't quite follow - isn't that already doing so?

folio_mapcount() is total_compound_mapcount() here, IIUC it is an accumulated value of all possible PTEs or PMDs being mapped as long as it's all or part of the folio being mapped.

We've talked about 3 ways of handling mapcount:

Quoted 9 lines

1. The RFC v2 way, which is head-only, and we increment the compound mapcount for each PT mapping we have. So a PTE-mapped 2M page, compound_mapcount=512, subpage->_mapcount=0 (ignoring the -1 bias).
2. The THP-like way. If we are fully mapping the hugetlb page with the hstate-level PTE, we increment the compound mapcount, otherwise we increment subpage->_mapcount.
3. The RFC v1 way (the way you have suggested above), which is head-only, and we increment the compound mapcount if the hstate-level PTE is made present.

Oh that's where it come from! It took quite some months going through all these, I can hardly remember the details.

Quoted 9 lines

With #1 and #2, there is no concern with folio_mapcount(). But with #3, folio_mapcount() for a PTE-mapped 2M page mapped in a single VMA would yield 1 instead of 512 (right?). That's what I mean.

#1 has problems wrt smaps and migration (though there were other problems with those anyway that Mike has fixed), and #2 makes MADV_COLLAPSE slow to the point of being unusable for some applications.

Ah so you're talking about after HGM being applied.. while I was only talking about THPs.

If to apply the logic here with idea 3), the worst case is we'll need to have special care of HGM hugetlb in folio_referenced_one(), so the default page_vma_mapped_walk() may not apply anymore - the resource is always in hstate sized, so counting small ptes do not help too - we can just walk until the hstate entry and do referenced++ if it's not none, at the entrance of folio_referenced_one().

But I'm not sure whether that'll be necessary at all, as I'm not sure whether that path can be triggered at all in any form (where from the top it should always be shrink_page_list()). In that sense maybe we can also consider adding a WARN_ON_ONCE() in folio_referenced() where it is a hugetlb page that got passed in? Meanwhile, adding a TODO comment explaining that current walk won't work easily for HGM only, so when it will be applicable to hugetlb we need to rework?

I confess that's not pretty, though. But that'll make 3) with no major defect from function-wise.

Side note: did we finish folio conversion on hugetlb at all? I think at least we need some helper like folio_test_huge(). It seems still missing. Maybe it's another clue that hugetlb is not important to folio_referenced() because it's already fully converted?

Quoted 4 lines

It seems like the least bad option is #1, but maybe we should have a face-to-face discussion about it? I'm still collecting some more performance numbers.

Let's see how it goes..

Thanks,

Signature · 1 line
Peter Xu
#76 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 127 lines · Peter Xu
Quoted 84 lines · James Houghton
Quoted 8 lines · Peter Xu
Quoted 6 lines · James Houghton
Quoted 5 lines · Peter Xu
Quoted 3 lines · James Houghton
Quoted 2 lines · Peter Xu

On Fri, Jan 27, 2023 at 01:02:02PM -0800, James Houghton wrote:

[snip]

Quoted 39 lines
Quoted 39 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 11 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

Quoted 5 lines
Quoted 1 line

I don't fully understand what you mean here.

I meant the rmap_walk pairing with folio_referenced_one() will walk all the leaves for the folio, big or small. I think that will match the number with what got returned from folio_mapcount().

See below.

Quoted 19 lines
Quoted 13 lines
Quoted 9 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

Quoted 3 lines

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Quoted 5 lines

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

Quoted 5 lines

I didn't quite follow - isn't that already doing so?

folio_mapcount() is total_compound_mapcount() here, IIUC it is an accumulated value of all possible PTEs or PMDs being mapped as long as it's all or part of the folio being mapped.

We've talked about 3 ways of handling mapcount:

Quoted 9 lines

1. The RFC v2 way, which is head-only, and we increment the compound mapcount for each PT mapping we have. So a PTE-mapped 2M page, compound_mapcount=512, subpage->_mapcount=0 (ignoring the -1 bias).
2. The THP-like way. If we are fully mapping the hugetlb page with the hstate-level PTE, we increment the compound mapcount, otherwise we increment subpage->_mapcount.
3. The RFC v1 way (the way you have suggested above), which is head-only, and we increment the compound mapcount if the hstate-level PTE is made present.

Oh that's where it come from! It took quite some months going through all these, I can hardly remember the details.

Quoted 9 lines

With #1 and #2, there is no concern with folio_mapcount(). But with #3, folio_mapcount() for a PTE-mapped 2M page mapped in a single VMA would yield 1 instead of 512 (right?). That's what I mean.

#1 has problems wrt smaps and migration (though there were other problems with those anyway that Mike has fixed), and #2 makes MADV_COLLAPSE slow to the point of being unusable for some applications.

Ah so you're talking about after HGM being applied.. while I was only talking about THPs.

If to apply the logic here with idea 3), the worst case is we'll need to have special care of HGM hugetlb in folio_referenced_one(), so the default page_vma_mapped_walk() may not apply anymore - the resource is always in hstate sized, so counting small ptes do not help too - we can just walk until the hstate entry and do referenced++ if it's not none, at the entrance of folio_referenced_one().

But I'm not sure whether that'll be necessary at all, as I'm not sure whether that path can be triggered at all in any form (where from the top it should always be shrink_page_list()). In that sense maybe we can also consider adding a WARN_ON_ONCE() in folio_referenced() where it is a hugetlb page that got passed in? Meanwhile, adding a TODO comment explaining that current walk won't work easily for HGM only, so when it will be applicable to hugetlb we need to rework?

Quoted 2 lines

I confess that's not pretty, though. But that'll make 3) with no major defect from function-wise.

Another potential idea would be to add something like page_vmacount(). For non-HugeTLB pages, page_vmacount() == page_mapcount(). Then for HugeTLB pages, we could keep a separate count (in one of the tail pages, I guess). And then in the places that matter (so smaps, migration, and maybe CoW and hwpoison), potentially change their calls to page_vmacount() instead of page_mapcount().

Then to implement page_vmacount(), we do the RFC v1 mapcount approach (but like.... correctly this time). And then for page_mapcount(), we do the RFC v2 mapcount approach (head-only, once per PTE).

Then we fix folio_referenced() without needing to special-case it for HugeTLB. :) Or we could just special-case it. *shrug*

Does that sound reasonable? We still have the problem where a series of partially unmaps could leave page_vmacount() incremented, but I don't think that's a big problem.

Quoted 5 lines

Side note: did we finish folio conversion on hugetlb at all? I think at least we need some helper like folio_test_huge(). It seems still missing. Maybe it's another clue that hugetlb is not important to folio_referenced() because it's already fully converted?

I'm not sure. A lot of work was done very pretty recently, so I bet there's probably some work left to do.

reviewer
#77 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 142 lines · James Houghton
Quoted 127 lines · Peter Xu
Quoted 84 lines · James Houghton
Quoted 8 lines · Peter Xu
Quoted 6 lines · James Houghton
Quoted 5 lines · Peter Xu
Quoted 3 lines · James Houghton
Quoted 2 lines · Peter Xu

On Fri, Jan 27, 2023 at 01:02:02PM -0800, James Houghton wrote:

[snip]

Quoted 39 lines
Quoted 39 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 11 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

Quoted 5 lines
Quoted 1 line

I don't fully understand what you mean here.

I meant the rmap_walk pairing with folio_referenced_one() will walk all the leaves for the folio, big or small. I think that will match the number with what got returned from folio_mapcount().

See below.

Quoted 19 lines
Quoted 13 lines
Quoted 9 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

Quoted 3 lines

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Quoted 5 lines

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

Quoted 5 lines

I didn't quite follow - isn't that already doing so?

folio_mapcount() is total_compound_mapcount() here, IIUC it is an accumulated value of all possible PTEs or PMDs being mapped as long as it's all or part of the folio being mapped.

We've talked about 3 ways of handling mapcount:

Quoted 9 lines

1. The RFC v2 way, which is head-only, and we increment the compound mapcount for each PT mapping we have. So a PTE-mapped 2M page, compound_mapcount=512, subpage->_mapcount=0 (ignoring the -1 bias).
2. The THP-like way. If we are fully mapping the hugetlb page with the hstate-level PTE, we increment the compound mapcount, otherwise we increment subpage->_mapcount.
3. The RFC v1 way (the way you have suggested above), which is head-only, and we increment the compound mapcount if the hstate-level PTE is made present.

Oh that's where it come from! It took quite some months going through all these, I can hardly remember the details.

Quoted 9 lines

With #1 and #2, there is no concern with folio_mapcount(). But with #3, folio_mapcount() for a PTE-mapped 2M page mapped in a single VMA would yield 1 instead of 512 (right?). That's what I mean.

#1 has problems wrt smaps and migration (though there were other problems with those anyway that Mike has fixed), and #2 makes MADV_COLLAPSE slow to the point of being unusable for some applications.

Ah so you're talking about after HGM being applied.. while I was only talking about THPs.

If to apply the logic here with idea 3), the worst case is we'll need to have special care of HGM hugetlb in folio_referenced_one(), so the default page_vma_mapped_walk() may not apply anymore - the resource is always in hstate sized, so counting small ptes do not help too - we can just walk until the hstate entry and do referenced++ if it's not none, at the entrance of folio_referenced_one().

But I'm not sure whether that'll be necessary at all, as I'm not sure whether that path can be triggered at all in any form (where from the top it should always be shrink_page_list()). In that sense maybe we can also consider adding a WARN_ON_ONCE() in folio_referenced() where it is a hugetlb page that got passed in? Meanwhile, adding a TODO comment explaining that current walk won't work easily for HGM only, so when it will be applicable to hugetlb we need to rework?

Quoted 2 lines

I confess that's not pretty, though. But that'll make 3) with no major defect from function-wise.

Another potential idea would be to add something like page_vmacount(). For non-HugeTLB pages, page_vmacount() == page_mapcount(). Then for HugeTLB pages, we could keep a separate count (in one of the tail pages, I guess). And then in the places that matter (so smaps, migration, and maybe CoW and hwpoison), potentially change their calls to page_vmacount() instead of page_mapcount().

Then to implement page_vmacount(), we do the RFC v1 mapcount approach (but like.... correctly this time). And then for page_mapcount(), we do the RFC v2 mapcount approach (head-only, once per PTE).

Quoted 6 lines

Then we fix folio_referenced() without needing to special-case it for HugeTLB. :) Or we could just special-case it. *shrug*

Does that sound reasonable? We still have the problem where a series of partially unmaps could leave page_vmacount() incremented, but I don't think that's a big problem.

I'm afraid someone will stop you from introducing yet another definition of mapcount, where others are trying to remove it. :)

Or, can we just drop folio_referenced_arg.mapcount? We need to keep:

	if (!pra.mapcount)
		return 0;

By replacing it with folio_mapcount which is definitely something worthwhile, but what about the rest?

If it can be dropped, afaict it'll naturally work with HGM again.

IIUC that's an optimization where we want to stop the rmap walk as long as we found all the pages, however (1) IIUC it's not required to function, and
(2) it's not guaranteed to work as solid anyway.. As we've discussed
before: right after it reads mapcount (but before taking the page lock), the mapcount can get decreased by 1, then it'll still need to loop over all the vmas just to find that there's one "misterious" mapcount lost.

Personally I really have no idea on how much that optimization can help.

Signature · 1 line
Peter Xu
#78 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 164 lines · Peter Xu
Quoted 142 lines · James Houghton
Quoted 127 lines · Peter Xu
Quoted 84 lines · James Houghton
Quoted 8 lines · Peter Xu
Quoted 6 lines · James Houghton
Quoted 5 lines · Peter Xu
Quoted 3 lines · James Houghton
Quoted 2 lines · Peter Xu

On Fri, Jan 27, 2023 at 01:02:02PM -0800, James Houghton wrote:

[snip]

Quoted 39 lines
Quoted 39 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 11 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

Quoted 5 lines
Quoted 1 line

I don't fully understand what you mean here.

I meant the rmap_walk pairing with folio_referenced_one() will walk all the leaves for the folio, big or small. I think that will match the number with what got returned from folio_mapcount().

See below.

Quoted 19 lines
Quoted 13 lines
Quoted 9 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

Quoted 3 lines

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Quoted 5 lines

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

Quoted 5 lines

I didn't quite follow - isn't that already doing so?

folio_mapcount() is total_compound_mapcount() here, IIUC it is an accumulated value of all possible PTEs or PMDs being mapped as long as it's all or part of the folio being mapped.

We've talked about 3 ways of handling mapcount:

Quoted 9 lines

1. The RFC v2 way, which is head-only, and we increment the compound mapcount for each PT mapping we have. So a PTE-mapped 2M page, compound_mapcount=512, subpage->_mapcount=0 (ignoring the -1 bias).
2. The THP-like way. If we are fully mapping the hugetlb page with the hstate-level PTE, we increment the compound mapcount, otherwise we increment subpage->_mapcount.
3. The RFC v1 way (the way you have suggested above), which is head-only, and we increment the compound mapcount if the hstate-level PTE is made present.

Oh that's where it come from! It took quite some months going through all these, I can hardly remember the details.

Quoted 9 lines

With #1 and #2, there is no concern with folio_mapcount(). But with #3, folio_mapcount() for a PTE-mapped 2M page mapped in a single VMA would yield 1 instead of 512 (right?). That's what I mean.

#1 has problems wrt smaps and migration (though there were other problems with those anyway that Mike has fixed), and #2 makes MADV_COLLAPSE slow to the point of being unusable for some applications.

Ah so you're talking about after HGM being applied.. while I was only talking about THPs.

If to apply the logic here with idea 3), the worst case is we'll need to have special care of HGM hugetlb in folio_referenced_one(), so the default page_vma_mapped_walk() may not apply anymore - the resource is always in hstate sized, so counting small ptes do not help too - we can just walk until the hstate entry and do referenced++ if it's not none, at the entrance of folio_referenced_one().

But I'm not sure whether that'll be necessary at all, as I'm not sure whether that path can be triggered at all in any form (where from the top it should always be shrink_page_list()). In that sense maybe we can also consider adding a WARN_ON_ONCE() in folio_referenced() where it is a hugetlb page that got passed in? Meanwhile, adding a TODO comment explaining that current walk won't work easily for HGM only, so when it will be applicable to hugetlb we need to rework?

Quoted 2 lines

I confess that's not pretty, though. But that'll make 3) with no major defect from function-wise.

Another potential idea would be to add something like page_vmacount(). For non-HugeTLB pages, page_vmacount() == page_mapcount(). Then for HugeTLB pages, we could keep a separate count (in one of the tail pages, I guess). And then in the places that matter (so smaps, migration, and maybe CoW and hwpoison), potentially change their calls to page_vmacount() instead of page_mapcount().

Then to implement page_vmacount(), we do the RFC v1 mapcount approach (but like.... correctly this time). And then for page_mapcount(), we do the RFC v2 mapcount approach (head-only, once per PTE).

Quoted 6 lines

Then we fix folio_referenced() without needing to special-case it for HugeTLB. :) Or we could just special-case it. *shrug*

Does that sound reasonable? We still have the problem where a series of partially unmaps could leave page_vmacount() incremented, but I don't think that's a big problem.

I'm afraid someone will stop you from introducing yet another definition of mapcount, where others are trying to remove it. :)

Or, can we just drop folio_referenced_arg.mapcount? We need to keep:

        if (!pra.mapcount)
                return 0;

By replacing it with folio_mapcount which is definitely something worthwhile, but what about the rest?

If it can be dropped, afaict it'll naturally work with HGM again.

Quoted 8 lines

IIUC that's an optimization where we want to stop the rmap walk as long as we found all the pages, however (1) IIUC it's not required to function, and
(2) it's not guaranteed to work as solid anyway.. As we've discussed
before: right after it reads mapcount (but before taking the page lock), the mapcount can get decreased by 1, then it'll still need to loop over all the vmas just to find that there's one "misterious" mapcount lost.

Personally I really have no idea on how much that optimization can help.

Ok, yeah, I think pra.mapcount can be removed too. (And we replace !pra.mapcount with !folio_mapcount().)

I don't see any other existing users of folio_mapcount() and total_mapcount() that are problematic. We do need to make sure to keep refcount and mapcount in sync though; it can be done.

So I'll compare this "RFC v1" way with the THP-like way and get you a performance comparison.

- James

#79 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 178 lines · James Houghton
Quoted 164 lines · Peter Xu
Quoted 142 lines · James Houghton
Quoted 127 lines · Peter Xu
Quoted 84 lines · James Houghton
Quoted 8 lines · Peter Xu
Quoted 6 lines · James Houghton
Quoted 5 lines · Peter Xu
Quoted 3 lines · James Houghton
Quoted 2 lines · Peter Xu

On Fri, Jan 27, 2023 at 01:02:02PM -0800, James Houghton wrote:

[snip]

Quoted 39 lines
Quoted 39 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Another way to not use thp mapcount, nor break smaps and similar calls to page_mapcount() on small page, is to only increase the hpage mapcount only when hstate pXd (in case of 1G it's PUD) entry being populated (no matter as leaf or a non-leaf), and the mapcount can be decreased when the pXd entry is removed (for leaf, it's the same as for now; for HGM, it's when freeing pgtable of the PUD entry).

Quoted 6 lines
Quoted 2 lines

Right, and this is doable. Also it seems like this is pretty close to the direction Matthew Wilcox wants to go with THPs.

I may not be familiar with it, do you mean this one?

https://lore.kernel.org/all/Y9Afwds%[email protected]/

Yep that's it.

Quoted 7 lines

For hugetlb I think it should be easier to maintain rather than any-sized folios, because there's the pgtable non-leaf entry to track rmap information and the folio size being static to hpage size.

It'll be different to folios where it can be random sized pages chunk, so it needs to be managed by batching the ptes when install/zap.

Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes.

Quoted 11 lines
Quoted 7 lines

Something I noticed though, from the implementation of folio_referenced()/folio_referenced_one(), is that folio_mapcount() ought to report the total number of PTEs that are pointing on the page (or the number of times page_vma_mapped_walk returns true). FWIW, folio_referenced() is never called for hugetlb folios.

FWIU folio_mapcount is the thing it needs for now to do the rmap walks - it'll walk every leaf page being mapped, big or small, so IIUC that number should match with what it expects to see later, more or less.

Quoted 5 lines
Quoted 1 line

I don't fully understand what you mean here.

I meant the rmap_walk pairing with folio_referenced_one() will walk all the leaves for the folio, big or small. I think that will match the number with what got returned from folio_mapcount().

See below.

Quoted 19 lines
Quoted 13 lines
Quoted 9 lines

But I agree the mapcount/referenced value itself is debatable to me, just like what you raised in the other thread on page migration. Meanwhile, I am not certain whether the mapcount is accurate either because AFAICT the mapcount can be modified if e.g. new page mapping established as long as before taking the page lock later in folio_referenced().

Quoted 3 lines

It's just that I don't see any severe issue either due to any of above, as long as that information is only used as a hint for next steps, e.g., to swap which page out.

Quoted 5 lines

I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?

Quoted 5 lines

I didn't quite follow - isn't that already doing so?

folio_mapcount() is total_compound_mapcount() here, IIUC it is an accumulated value of all possible PTEs or PMDs being mapped as long as it's all or part of the folio being mapped.

We've talked about 3 ways of handling mapcount:

Quoted 9 lines

1. The RFC v2 way, which is head-only, and we increment the compound mapcount for each PT mapping we have. So a PTE-mapped 2M page, compound_mapcount=512, subpage->_mapcount=0 (ignoring the -1 bias).
2. The THP-like way. If we are fully mapping the hugetlb page with the hstate-level PTE, we increment the compound mapcount, otherwise we increment subpage->_mapcount.
3. The RFC v1 way (the way you have suggested above), which is head-only, and we increment the compound mapcount if the hstate-level PTE is made present.

Oh that's where it come from! It took quite some months going through all these, I can hardly remember the details.

Quoted 9 lines

With #1 and #2, there is no concern with folio_mapcount(). But with #3, folio_mapcount() for a PTE-mapped 2M page mapped in a single VMA would yield 1 instead of 512 (right?). That's what I mean.

#1 has problems wrt smaps and migration (though there were other problems with those anyway that Mike has fixed), and #2 makes MADV_COLLAPSE slow to the point of being unusable for some applications.

Ah so you're talking about after HGM being applied.. while I was only talking about THPs.

If to apply the logic here with idea 3), the worst case is we'll need to have special care of HGM hugetlb in folio_referenced_one(), so the default page_vma_mapped_walk() may not apply anymore - the resource is always in hstate sized, so counting small ptes do not help too - we can just walk until the hstate entry and do referenced++ if it's not none, at the entrance of folio_referenced_one().

But I'm not sure whether that'll be necessary at all, as I'm not sure whether that path can be triggered at all in any form (where from the top it should always be shrink_page_list()). In that sense maybe we can also consider adding a WARN_ON_ONCE() in folio_referenced() where it is a hugetlb page that got passed in? Meanwhile, adding a TODO comment explaining that current walk won't work easily for HGM only, so when it will be applicable to hugetlb we need to rework?

Quoted 2 lines

I confess that's not pretty, though. But that'll make 3) with no major defect from function-wise.

Another potential idea would be to add something like page_vmacount(). For non-HugeTLB pages, page_vmacount() == page_mapcount(). Then for HugeTLB pages, we could keep a separate count (in one of the tail pages, I guess). And then in the places that matter (so smaps, migration, and maybe CoW and hwpoison), potentially change their calls to page_vmacount() instead of page_mapcount().

Then to implement page_vmacount(), we do the RFC v1 mapcount approach (but like.... correctly this time). And then for page_mapcount(), we do the RFC v2 mapcount approach (head-only, once per PTE).

Quoted 6 lines

Then we fix folio_referenced() without needing to special-case it for HugeTLB. :) Or we could just special-case it. *shrug*

Does that sound reasonable? We still have the problem where a series of partially unmaps could leave page_vmacount() incremented, but I don't think that's a big problem.

I'm afraid someone will stop you from introducing yet another definition of mapcount, where others are trying to remove it. :)

Or, can we just drop folio_referenced_arg.mapcount? We need to keep:

        if (!pra.mapcount)
                return 0;

By replacing it with folio_mapcount which is definitely something worthwhile, but what about the rest?

If it can be dropped, afaict it'll naturally work with HGM again.

Quoted 8 lines

IIUC that's an optimization where we want to stop the rmap walk as long as we found all the pages, however (1) IIUC it's not required to function, and
(2) it's not guaranteed to work as solid anyway.. As we've discussed
before: right after it reads mapcount (but before taking the page lock), the mapcount can get decreased by 1, then it'll still need to loop over all the vmas just to find that there's one "misterious" mapcount lost.

Personally I really have no idea on how much that optimization can help.

Ok, yeah, I think pra.mapcount can be removed too. (And we replace !pra.mapcount with !folio_mapcount().)

Quoted 6 lines

I don't see any other existing users of folio_mapcount() and total_mapcount() that are problematic. We do need to make sure to keep refcount and mapcount in sync though; it can be done.

So I'll compare this "RFC v1" way with the THP-like way and get you a performance comparison.

Here is the result: [1] (sorry it took a little while heh). The implementation of the "RFC v1" way is pretty horrible[2] (and this implementation probably has bugs anyway; it doesn't account for the folio_referenced() problem).

Matthew is trying to solve the same problem with THPs right now: [3]. I haven't figured out how we can apply Matthews's approach to HGM right now, but there probably is a way. (If we left the mapcount increment bits in the same place, we couldn't just check the hstate-level PTE; it would have already been made present.)

We could:
- use the THP-like way and tolerate ~1 second collapses
- use the (non-RFC) v1 way and tolerate the migration/smaps differences
- use the RFC v1 way and tolerate the complicated mapcount accounting
- flesh out [3] and see if it can be applied to HGM nicely

I'm happy to go with any of these approaches.

[1]: https://pastebin.com/raw/hJzFJHiD
[2]: https://github.com/48ca/linux/commit/4495f16a09b660aff44b3edcc125aa3a3df85976
[3]: https://lore.kernel.org/linux-mm/[email protected]/

- James

#80 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 5 lines

Here is the result: [1] (sorry it took a little while heh). The implementation of the "RFC v1" way is pretty horrible[2] (and this implementation probably has bugs anyway; it doesn't account for the folio_referenced() problem).

Quoted 8 lines

Matthew is trying to solve the same problem with THPs right now: [3]. I haven't figured out how we can apply Matthews's approach to HGM right now, but there probably is a way. (If we left the mapcount increment bits in the same place, we couldn't just check the hstate-level PTE; it would have already been made present.)

We could:
- use the THP-like way and tolerate ~1 second collapses

Another thought here. We don't necessarily *need* to collapse the page table mappings in between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), as the pfns aren't changing, we aren't punching any holes, and we aren't changing permission bits. If we had an MMU notifier that simply informed KVM that we collapsed the page tables *after* we finished collapsing, then it would be ok for hugetlb_collapse() to be slow.

If this MMU notifier is something that makes sense, it probably applies to MADV_COLLAPSE for THPs as well.

Quoted 9 lines

- use the (non-RFC) v1 way and tolerate the migration/smaps differences
- use the RFC v1 way and tolerate the complicated mapcount accounting
- flesh out [3] and see if it can be applied to HGM nicely

I'm happy to go with any of these approaches.

[1]: https://pastebin.com/raw/hJzFJHiD
[2]: https://github.com/48ca/linux/commit/4495f16a09b660aff44b3edcc125aa3a3df85976
[3]: https://lore.kernel.org/linux-mm/[email protected]/

- James

reviewer
#81 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

James,

Quoted 1 line · James Houghton
Quoted 1 line

Here is the result: [1] (sorry it took a little while heh). The

Thanks. From what I can tell, that number shows that it'll be great we start with your rfcv1 mapcount approach, which mimics what's proposed by Matthew for generic folio.

Quoted 1 line
Quoted 1 line

implementation of the "RFC v1" way is pretty horrible[2] (and this

Any more information on why it's horrible? :)

A quick comment is I'm wondering whether that "whether we should boost the mapcount" value can be hidden in hugetlb_pte* so you don't need to pass over a lot of bool* deep into the hgm walk routines.

Quoted 2 lines
Quoted 2 lines

implementation probably has bugs anyway; it doesn't account for the folio_referenced() problem).

I thought we reached a consensus on the resolution, by a proposal to remove folio_referenced_arg.mapcount. Is it not working for some reason?

Quoted 6 lines
Quoted 6 lines

Matthew is trying to solve the same problem with THPs right now: [3]. I haven't figured out how we can apply Matthews's approach to HGM right now, but there probably is a way. (If we left the mapcount increment bits in the same place, we couldn't just check the hstate-level PTE; it would have already been made present.)

I'm just worried that (1) this may add yet another dependency to your work which is still during discussion phase, and (2) whether the folio approach is easily applicable here, e.g., we may not want to populate all the ptes for hugetlb HGMs by default.

Quoted 11 lines
Quoted 3 lines

We could:
- use the THP-like way and tolerate ~1 second collapses

Another thought here. We don't necessarily *need* to collapse the page table mappings in between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), as the pfns aren't changing, we aren't punching any holes, and we aren't changing permission bits. If we had an MMU notifier that simply informed KVM that we collapsed the page tables *after* we finished collapsing, then it would be ok for hugetlb_collapse() to be slow.

That's a great point! It'll definitely apply to either approach.

Quoted 3 lines

If this MMU notifier is something that makes sense, it probably applies to MADV_COLLAPSE for THPs as well.

THPs are definitely different, mmu notifiers should be required there, afaict. Isn't that what the current code does?

See collapse_and_free_pmd() for shmem and collapse_huge_page() for anon.

Quoted 14 lines
Quoted 11 lines

- use the (non-RFC) v1 way and tolerate the migration/smaps differences
- use the RFC v1 way and tolerate the complicated mapcount accounting
- flesh out [3] and see if it can be applied to HGM nicely

I'm happy to go with any of these approaches.

[1]: https://pastebin.com/raw/hJzFJHiD
[2]: https://github.com/48ca/linux/commit/4495f16a09b660aff44b3edcc125aa3a3df85976
[3]: https://lore.kernel.org/linux-mm/[email protected]/

- James

Signature · 1 line
Peter Xu
#82 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 9 lines · Peter Xu

James,

Quoted 1 line · James Houghton
Quoted 1 line

Here is the result: [1] (sorry it took a little while heh). The

Thanks. From what I can tell, that number shows that it'll be great we start with your rfcv1 mapcount approach, which mimics what's proposed by Matthew for generic folio.

Do you think the RFC v1 way is better than doing the THP-like way *with the additional MMU notifier*?

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

implementation of the "RFC v1" way is pretty horrible[2] (and this

Any more information on why it's horrible? :)

I figured the code would speak for itself, heh. It's quite complicated.

I really didn't like:
1. The 'inc' business in copy_hugetlb_page_range.
2. How/where I call put_page()/folio_put() to keep the refcount and mapcount synced up.
3. Having to check the page cache in UFFDIO_CONTINUE.

Quoted 4 lines

A quick comment is I'm wondering whether that "whether we should boost the mapcount" value can be hidden in hugetlb_pte* so you don't need to pass over a lot of bool* deep into the hgm walk routines.

Oh yeah, that's a great idea.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

implementation probably has bugs anyway; it doesn't account for the folio_referenced() problem).

I thought we reached a consensus on the resolution, by a proposal to remove folio_referenced_arg.mapcount. Is it not working for some reason?

I think that works, I just didn't bother here. I just wanted to show you approximately what it would look like to implement the RFC v1 approach.

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

Matthew is trying to solve the same problem with THPs right now: [3]. I haven't figured out how we can apply Matthews's approach to HGM right now, but there probably is a way. (If we left the mapcount increment bits in the same place, we couldn't just check the hstate-level PTE; it would have already been made present.)

Quoted 4 lines

I'm just worried that (1) this may add yet another dependency to your work which is still during discussion phase, and (2) whether the folio approach is easily applicable here, e.g., we may not want to populate all the ptes for hugetlb HGMs by default.

That's true. I definitely don't want to wait for this either. It seems like Matthew's approach won't work very well for us -- when doing a lot of high-granularity UFFDIO_CONTINUEs on a 1G page, checking all the PTEs to see if any of them are mapped would get really slow.

Quoted 16 lines
Quoted 5 lines
Quoted 4 lines

We could:
- use the THP-like way and tolerate ~1 second collapses

Quoted 7 lines

Another thought here. We don't necessarily *need* to collapse the page table mappings in between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), as the pfns aren't changing, we aren't punching any holes, and we aren't changing permission bits. If we had an MMU notifier that simply informed KVM that we collapsed the page tables *after* we finished collapsing, then it would be ok for hugetlb_collapse() to be slow.

That's a great point! It'll definitely apply to either approach.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

If this MMU notifier is something that makes sense, it probably applies to MADV_COLLAPSE for THPs as well.

THPs are definitely different, mmu notifiers should be required there, afaict. Isn't that what the current code does?

See collapse_and_free_pmd() for shmem and collapse_huge_page() for anon.

Oh, yes, of course, MADV_COLLAPSE can actually move things around and properly make THPs. Thanks. But it would apply if we were only collapsing PTE-mapped THPs, I think?

- James

reviewer
#83 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 5 lines · James Houghton
Quoted 4 lines · Peter Xu

James,

On Tue, Feb 07, 2023 at 02:46:04PM -0800, James Houghton wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line

Here is the result: [1] (sorry it took a little while heh). The

Thanks. From what I can tell, that number shows that it'll be great we start with your rfcv1 mapcount approach, which mimics what's proposed by Matthew for generic folio.

Do you think the RFC v1 way is better than doing the THP-like way *with the additional MMU notifier*?

What's the additional MMU notifier you're referring?

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

implementation of the "RFC v1" way is pretty horrible[2] (and this

Any more information on why it's horrible? :)

Quoted 7 lines

I figured the code would speak for itself, heh. It's quite complicated.

I really didn't like:
1. The 'inc' business in copy_hugetlb_page_range.
2. How/where I call put_page()/folio_put() to keep the refcount and mapcount synced up.
3. Having to check the page cache in UFFDIO_CONTINUE.

I think the complexity is one thing which I'm fine with so far. However when I think again about the things behind that complexity, I noticed there may be at least one flaw that may not be trivial to work around.

It's about truncation. The problem is now we use the pgtable entry to represent the mapcount, but the pgtable entry cannot be zapped easily, unless vma unmapped or collapsed.

It means e.g. truncate_inode_folio() may stop working for hugetlb (of course, with page lock held). The mappings will be removed for real, but not the mapcount for HGM anymore, because unmap_mapping_folio() only zaps the pgtable leaves, not the ones that we used to account for mapcounts.

So the kernel may see weird things, like mapcount>0 after truncate_inode_folio() being finished completely.

For HGM to do the right thing, we may want to also remove the non-leaf entries when truncating or doing similar things like a rmap walk to drop any mappings for a page/folio. Though that's not doable for now because the locks that truncate_inode_folio() is weaker than what we need to free the pgtable non-leaf entries - we'll need mmap write lock for that, the same as when we unmap or collapse.

Matthew's design doesn't have such issue if the ptes need to be populated, because mapcount is still with the leaves; not the case for us here.

If that's the case, _maybe_ we still need to start with the stupid but working approach of subpage mapcounts.

[...]

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines

Matthew is trying to solve the same problem with THPs right now: [3]. I haven't figured out how we can apply Matthews's approach to HGM right now, but there probably is a way. (If we left the mapcount increment bits in the same place, we couldn't just check the hstate-level PTE; it would have already been made present.)

I'm just worried that (1) this may add yet another dependency to your work which is still during discussion phase, and (2) whether the folio approach is easily applicable here, e.g., we may not want to populate all the ptes for hugetlb HGMs by default.

Quoted 4 lines

That's true. I definitely don't want to wait for this either. It seems like Matthew's approach won't work very well for us -- when doing a lot of high-granularity UFFDIO_CONTINUEs on a 1G page, checking all the PTEs to see if any of them are mapped would get really slow.

I think it'll be a common problem to userfaultfd when it comes, e.g., userfaultfd by design is PAGE_SIZE based so far. It needs page size granule on pgtable manipulations, unless we extend the userfaultfd protocol to support folios, iiuc.

Quoted 20 lines
Quoted 20 lines
Quoted 6 lines
Quoted 5 lines

We could:
- use the THP-like way and tolerate ~1 second collapses

Quoted 7 lines

Another thought here. We don't necessarily *need* to collapse the page table mappings in between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), as the pfns aren't changing, we aren't punching any holes, and we aren't changing permission bits. If we had an MMU notifier that simply informed KVM that we collapsed the page tables *after* we finished collapsing, then it would be ok for hugetlb_collapse() to be slow.

That's a great point! It'll definitely apply to either approach.

Quoted 3 lines

If this MMU notifier is something that makes sense, it probably applies to MADV_COLLAPSE for THPs as well.

Quoted 8 lines
Quoted 4 lines

THPs are definitely different, mmu notifiers should be required there, afaict. Isn't that what the current code does?

See collapse_and_free_pmd() for shmem and collapse_huge_page() for anon.

Oh, yes, of course, MADV_COLLAPSE can actually move things around and properly make THPs. Thanks. But it would apply if we were only collapsing PTE-mapped THPs, I think?

Yes it applies I think. And if I'm not wrong it's also doing so. :) See collapse_pte_mapped_thp().

While for anon we always allocate a new page, hence not applicable.

Signature · 1 line
Peter Xu
#84 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 9 lines · Peter Xu
Quoted 7 lines · James Houghton
Quoted 6 lines · Peter Xu

James,

Quoted 1 line · James Houghton
Quoted 1 line

Here is the result: [1] (sorry it took a little while heh). The

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Thanks. From what I can tell, that number shows that it'll be great we start with your rfcv1 mapcount approach, which mimics what's proposed by Matthew for generic folio.

Do you think the RFC v1 way is better than doing the THP-like way *with the additional MMU notifier*?

What's the additional MMU notifier you're referring?

An MMU notifier that informs KVM that a collapse has happened without having to invalidate_range_start() and invalidate_range_end(), the one you're replying to lower down in the email. :) [ see below... ]

Quoted 38 lines
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

implementation of the "RFC v1" way is pretty horrible[2] (and this

Any more information on why it's horrible? :)

Quoted 7 lines

I figured the code would speak for itself, heh. It's quite complicated.

I really didn't like:
1. The 'inc' business in copy_hugetlb_page_range.
2. How/where I call put_page()/folio_put() to keep the refcount and mapcount synced up.
3. Having to check the page cache in UFFDIO_CONTINUE.

I think the complexity is one thing which I'm fine with so far. However when I think again about the things behind that complexity, I noticed there may be at least one flaw that may not be trivial to work around.

It's about truncation. The problem is now we use the pgtable entry to represent the mapcount, but the pgtable entry cannot be zapped easily, unless vma unmapped or collapsed.

It means e.g. truncate_inode_folio() may stop working for hugetlb (of course, with page lock held). The mappings will be removed for real, but not the mapcount for HGM anymore, because unmap_mapping_folio() only zaps the pgtable leaves, not the ones that we used to account for mapcounts.

So the kernel may see weird things, like mapcount>0 after truncate_inode_folio() being finished completely.

For HGM to do the right thing, we may want to also remove the non-leaf entries when truncating or doing similar things like a rmap walk to drop any mappings for a page/folio. Though that's not doable for now because the locks that truncate_inode_folio() is weaker than what we need to free the pgtable non-leaf entries - we'll need mmap write lock for that, the same as when we unmap or collapse.

Quoted 5 lines

Matthew's design doesn't have such issue if the ptes need to be populated, because mapcount is still with the leaves; not the case for us here.

If that's the case, _maybe_ we still need to start with the stupid but working approach of subpage mapcounts.

Good point. I can't immediately think of a solution. I would prefer to go with the subpage mapcount approach to simplify HGM for now; optimizing mapcount for HugeTLB can then be handled separately. If you're ok with this, I'll go ahead and send v2.

One way that might be possible: using the PAGE_SPECIAL bit on the hstate-level PTE to indicate if mapcount has been incremented or not (if the PTE is pointing to page tables). As far as I can tell, PAGE_SPECIAL doesn't carry any meaning for HugeTLB PTEs, but we would need to be careful with existing PTE examination code as to not misinterpret these PTEs.

Quoted 19 lines

[...]

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines

Matthew is trying to solve the same problem with THPs right now: [3]. I haven't figured out how we can apply Matthews's approach to HGM right now, but there probably is a way. (If we left the mapcount increment bits in the same place, we couldn't just check the hstate-level PTE; it would have already been made present.)

I'm just worried that (1) this may add yet another dependency to your work which is still during discussion phase, and (2) whether the folio approach is easily applicable here, e.g., we may not want to populate all the ptes for hugetlb HGMs by default.

Quoted 4 lines

That's true. I definitely don't want to wait for this either. It seems like Matthew's approach won't work very well for us -- when doing a lot of high-granularity UFFDIO_CONTINUEs on a 1G page, checking all the PTEs to see if any of them are mapped would get really slow.

Quoted 18 lines

I think it'll be a common problem to userfaultfd when it comes, e.g., userfaultfd by design is PAGE_SIZE based so far. It needs page size granule on pgtable manipulations, unless we extend the userfaultfd protocol to support folios, iiuc.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines

We could:
- use the THP-like way and tolerate ~1 second collapses

Another thought here. We don't necessarily *need* to collapse the page table mappings in between mmu_notifier_invalidate_range_start() and mmu_notifier_invalidate_range_end(), as the pfns aren't changing, we aren't punching any holes, and we aren't changing permission bits. If we had an MMU notifier that simply informed KVM that we collapsed the page tables *after* we finished collapsing, then it would be ok for hugetlb_collapse() to be slow.

[ from above... ] This MMU notifier. :)

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines

That's a great point! It'll definitely apply to either approach.

Quoted 3 lines

If this MMU notifier is something that makes sense, it probably applies to MADV_COLLAPSE for THPs as well.

Quoted 8 lines
Quoted 4 lines

THPs are definitely different, mmu notifiers should be required there, afaict. Isn't that what the current code does?

See collapse_and_free_pmd() for shmem and collapse_huge_page() for anon.

Oh, yes, of course, MADV_COLLAPSE can actually move things around and properly make THPs. Thanks. But it would apply if we were only collapsing PTE-mapped THPs, I think?

Quoted 7 lines

Yes it applies I think. And if I'm not wrong it's also doing so. :) See collapse_pte_mapped_thp().

While for anon we always allocate a new page, hence not applicable.

--
Peter Xu

Thanks Peter!
- James

reviewer
#85 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 14 lines · James Houghton
Quoted 13 lines · Peter Xu
Quoted 11 lines · James Houghton
Quoted 9 lines · Peter Xu

James,

Quoted 1 line · James Houghton
Quoted 1 line

Here is the result: [1] (sorry it took a little while heh). The

Thanks. From what I can tell, that number shows that it'll be great we start with your rfcv1 mapcount approach, which mimics what's proposed by Matthew for generic folio.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Do you think the RFC v1 way is better than doing the THP-like way *with the additional MMU notifier*?

What's the additional MMU notifier you're referring?

An MMU notifier that informs KVM that a collapse has happened without having to invalidate_range_start() and invalidate_range_end(), the one you're replying to lower down in the email. :) [ see below... ]

Isn't that something that is needed no matter what mapcount approach we'll go for? Did I miss something?

Quoted 42 lines
Quoted 42 lines
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

implementation of the "RFC v1" way is pretty horrible[2] (and this

Any more information on why it's horrible? :)

Quoted 7 lines

I figured the code would speak for itself, heh. It's quite complicated.

I really didn't like:
1. The 'inc' business in copy_hugetlb_page_range.
2. How/where I call put_page()/folio_put() to keep the refcount and mapcount synced up.
3. Having to check the page cache in UFFDIO_CONTINUE.

I think the complexity is one thing which I'm fine with so far. However when I think again about the things behind that complexity, I noticed there may be at least one flaw that may not be trivial to work around.

It's about truncation. The problem is now we use the pgtable entry to represent the mapcount, but the pgtable entry cannot be zapped easily, unless vma unmapped or collapsed.

It means e.g. truncate_inode_folio() may stop working for hugetlb (of course, with page lock held). The mappings will be removed for real, but not the mapcount for HGM anymore, because unmap_mapping_folio() only zaps the pgtable leaves, not the ones that we used to account for mapcounts.

So the kernel may see weird things, like mapcount>0 after truncate_inode_folio() being finished completely.

For HGM to do the right thing, we may want to also remove the non-leaf entries when truncating or doing similar things like a rmap walk to drop any mappings for a page/folio. Though that's not doable for now because the locks that truncate_inode_folio() is weaker than what we need to free the pgtable non-leaf entries - we'll need mmap write lock for that, the same as when we unmap or collapse.

Matthew's design doesn't have such issue if the ptes need to be populated, because mapcount is still with the leaves; not the case for us here.

Quoted 7 lines
Quoted 2 lines

If that's the case, _maybe_ we still need to start with the stupid but working approach of subpage mapcounts.

Good point. I can't immediately think of a solution. I would prefer to go with the subpage mapcount approach to simplify HGM for now; optimizing mapcount for HugeTLB can then be handled separately. If you're ok with this, I'll go ahead and send v2.

I'm okay with it, but I suggest wait for at least another one day or two to see whether Mike or others have any comments.

Quoted 7 lines

One way that might be possible: using the PAGE_SPECIAL bit on the hstate-level PTE to indicate if mapcount has been incremented or not (if the PTE is pointing to page tables). As far as I can tell, PAGE_SPECIAL doesn't carry any meaning for HugeTLB PTEs, but we would need to be careful with existing PTE examination code as to not misinterpret these PTEs.

This is an interesting idea. :) Yes I don't see it being used at all in any pgtable non-leaves.

Then it's about how to let the zap code know when to remove the special bit, hence the mapcount, because not all of them should.

Maybe it can be passed over as a new zap_flags_t bit?

Thanks,

Signature · 1 line
Peter Xu
#86 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 19 lines · Peter Xu
Quoted 17 lines · James Houghton
Quoted 16 lines · Peter Xu
Quoted 13 lines · James Houghton
Quoted 9 lines · Peter Xu

James,

Quoted 1 line · James Houghton
Quoted 1 line

Here is the result: [1] (sorry it took a little while heh). The

Thanks. From what I can tell, that number shows that it'll be great we start with your rfcv1 mapcount approach, which mimics what's proposed by Matthew for generic folio.

Do you think the RFC v1 way is better than doing the THP-like way *with the additional MMU notifier*?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

What's the additional MMU notifier you're referring?

An MMU notifier that informs KVM that a collapse has happened without having to invalidate_range_start() and invalidate_range_end(), the one you're replying to lower down in the email. :) [ see below... ]

Isn't that something that is needed no matter what mapcount approach we'll go for? Did I miss something?

It's not really needed for anything, but it could be an optimization for both approaches. However, for the subpage-mapcount approach, it would have a *huge* impact. That's what I mean.

Quoted 46 lines
Quoted 46 lines
Quoted 45 lines
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

implementation of the "RFC v1" way is pretty horrible[2] (and this

Any more information on why it's horrible? :)

Quoted 7 lines

I figured the code would speak for itself, heh. It's quite complicated.

I really didn't like:
1. The 'inc' business in copy_hugetlb_page_range.
2. How/where I call put_page()/folio_put() to keep the refcount and mapcount synced up.
3. Having to check the page cache in UFFDIO_CONTINUE.

I think the complexity is one thing which I'm fine with so far. However when I think again about the things behind that complexity, I noticed there may be at least one flaw that may not be trivial to work around.

It's about truncation. The problem is now we use the pgtable entry to represent the mapcount, but the pgtable entry cannot be zapped easily, unless vma unmapped or collapsed.

It means e.g. truncate_inode_folio() may stop working for hugetlb (of course, with page lock held). The mappings will be removed for real, but not the mapcount for HGM anymore, because unmap_mapping_folio() only zaps the pgtable leaves, not the ones that we used to account for mapcounts.

So the kernel may see weird things, like mapcount>0 after truncate_inode_folio() being finished completely.

For HGM to do the right thing, we may want to also remove the non-leaf entries when truncating or doing similar things like a rmap walk to drop any mappings for a page/folio. Though that's not doable for now because the locks that truncate_inode_folio() is weaker than what we need to free the pgtable non-leaf entries - we'll need mmap write lock for that, the same as when we unmap or collapse.

Matthew's design doesn't have such issue if the ptes need to be populated, because mapcount is still with the leaves; not the case for us here.

If that's the case, _maybe_ we still need to start with the stupid but working approach of subpage mapcounts.

Quoted 7 lines
Quoted 4 lines

Good point. I can't immediately think of a solution. I would prefer to go with the subpage mapcount approach to simplify HGM for now; optimizing mapcount for HugeTLB can then be handled separately. If you're ok with this, I'll go ahead and send v2.

I'm okay with it, but I suggest wait for at least another one day or two to see whether Mike or others have any comments.

Ok. :)

Quoted 9 lines
Quoted 8 lines

One way that might be possible: using the PAGE_SPECIAL bit on the hstate-level PTE to indicate if mapcount has been incremented or not (if the PTE is pointing to page tables). As far as I can tell, PAGE_SPECIAL doesn't carry any meaning for HugeTLB PTEs, but we would need to be careful with existing PTE examination code as to not misinterpret these PTEs.

Quoted 7 lines

This is an interesting idea. :) Yes I don't see it being used at all in any pgtable non-leaves.

Then it's about how to let the zap code know when to remove the special bit, hence the mapcount, because not all of them should.

Maybe it can be passed over as a new zap_flags_t bit?

Here[1] is one way it could be done (it doesn't work 100% correctly, it's just approximately what we could do). Basically we pass in the entire range that we are unmapping ("floor" and "ceil"), and if hugetlb_remove_rmap finds that we're doing the final removal of a page that we are entirely unmapping (i.e., floor <= addr & huge_page_mask(h)). Having a zap flag would probably work too.

I think something like [1] ought to go in its own series. :)

[1]: https://github.com/48ca/linux/commit/de884eaaadf61b8dcfb1defd99bbf487667e46f4

- James

reviewer
#87 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

Quoted 26 lines · James Houghton
Quoted 25 lines · Peter Xu
Quoted 22 lines · James Houghton
Quoted 9 lines · Peter Xu
Quoted 7 lines · James Houghton
Quoted 6 lines · Peter Xu

James,

Quoted 1 line · James Houghton
Quoted 1 line

Here is the result: [1] (sorry it took a little while heh). The

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Thanks. From what I can tell, that number shows that it'll be great we start with your rfcv1 mapcount approach, which mimics what's proposed by Matthew for generic folio.

Do you think the RFC v1 way is better than doing the THP-like way *with the additional MMU notifier*?

What's the additional MMU notifier you're referring?

An MMU notifier that informs KVM that a collapse has happened without having to invalidate_range_start() and invalidate_range_end(), the one you're replying to lower down in the email. :) [ see below... ]

Quoted 6 lines
Quoted 2 lines

Isn't that something that is needed no matter what mapcount approach we'll go for? Did I miss something?

It's not really needed for anything, but it could be an optimization for both approaches. However, for the subpage-mapcount approach, it would have a *huge* impact. That's what I mean.

Ah, okay.

Quoted 72 lines
Quoted 47 lines
Quoted 47 lines
Quoted 46 lines
Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines

implementation of the "RFC v1" way is pretty horrible[2] (and this

Any more information on why it's horrible? :)

Quoted 7 lines

I figured the code would speak for itself, heh. It's quite complicated.

I really didn't like:
1. The 'inc' business in copy_hugetlb_page_range.
2. How/where I call put_page()/folio_put() to keep the refcount and mapcount synced up.
3. Having to check the page cache in UFFDIO_CONTINUE.

I think the complexity is one thing which I'm fine with so far. However when I think again about the things behind that complexity, I noticed there may be at least one flaw that may not be trivial to work around.

It's about truncation. The problem is now we use the pgtable entry to represent the mapcount, but the pgtable entry cannot be zapped easily, unless vma unmapped or collapsed.

It means e.g. truncate_inode_folio() may stop working for hugetlb (of course, with page lock held). The mappings will be removed for real, but not the mapcount for HGM anymore, because unmap_mapping_folio() only zaps the pgtable leaves, not the ones that we used to account for mapcounts.

So the kernel may see weird things, like mapcount>0 after truncate_inode_folio() being finished completely.

For HGM to do the right thing, we may want to also remove the non-leaf entries when truncating or doing similar things like a rmap walk to drop any mappings for a page/folio. Though that's not doable for now because the locks that truncate_inode_folio() is weaker than what we need to free the pgtable non-leaf entries - we'll need mmap write lock for that, the same as when we unmap or collapse.

Matthew's design doesn't have such issue if the ptes need to be populated, because mapcount is still with the leaves; not the case for us here.

If that's the case, _maybe_ we still need to start with the stupid but working approach of subpage mapcounts.

Quoted 7 lines
Quoted 4 lines

Good point. I can't immediately think of a solution. I would prefer to go with the subpage mapcount approach to simplify HGM for now; optimizing mapcount for HugeTLB can then be handled separately. If you're ok with this, I'll go ahead and send v2.

I'm okay with it, but I suggest wait for at least another one day or two to see whether Mike or others have any comments.

Ok. :)

Quoted 15 lines
Quoted 8 lines

One way that might be possible: using the PAGE_SPECIAL bit on the hstate-level PTE to indicate if mapcount has been incremented or not (if the PTE is pointing to page tables). As far as I can tell, PAGE_SPECIAL doesn't carry any meaning for HugeTLB PTEs, but we would need to be careful with existing PTE examination code as to not misinterpret these PTEs.

This is an interesting idea. :) Yes I don't see it being used at all in any pgtable non-leaves.

Then it's about how to let the zap code know when to remove the special bit, hence the mapcount, because not all of them should.

Quoted 8 lines
Quoted 1 line

Maybe it can be passed over as a new zap_flags_t bit?

Here[1] is one way it could be done (it doesn't work 100% correctly, it's just approximately what we could do). Basically we pass in the entire range that we are unmapping ("floor" and "ceil"), and if hugetlb_remove_rmap finds that we're doing the final removal of a page that we are entirely unmapping (i.e., floor <= addr & huge_page_mask(h)). Having a zap flag would probably work too.

Yeah maybe flags are not needed at all. I had a quick glance, looks good in general.

I think the trick is when it's not unmapped in a single shot. Consider someone zaps the first half of HGM-mapped hpage then the other half. The range may not always tell the whole story so rmap might be left over in some cases.

But maybe it is not a big deal. The only thing I think of so far is the partial DONTNEED. but I think maybe it's fine to leave it there until another more serious request to either truncate or unmap it. At least all rmap walks should work as expected.

Quoted 4 lines

I think something like [1] ought to go in its own series. :)

[1]: https://github.com/48ca/linux/commit/de884eaaadf61b8dcfb1defd99bbf487667e46f4

Yes I agree it can be worked on top.

Signature · 1 line
Peter Xu
#88 lore raw

Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range

In reply to Peter Xu
Quoted 159 lines · Peter Xu
Quoted 44 lines · David Hildenbrand
Quoted 18 lines · James Houghton
Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Peter Xu
Quoted 5 lines · James Houghton

I'll look into it, but doing it this way will use _mapcount, so we won't be able to use the vmemmap optimization. I think even if we do use Hugh's approach, refcount is still being kept on the head page, so there's still an overflow risk there (but maybe I am misunderstanding).

Quoted 7 lines
Quoted 2 lines

Could you remind me what's the issue if using refcount on the small pages rather than the head (assuming vmemmap still can be disabled)?

The THP-way of doing things is refcounting on the head page. All folios use a single refcount on the head.

There has to be a pretty good reason to do it differently.

Quoted 8 lines

Peter and I have discussed this a lot offline. There are two main problems here:

1. Refcount overflow

Refcount is always kept on the head page (before and after this series). IIUC, this means that if THPs could be 1G in size, they too would be susceptible to the same potential overflow. How easy is the overflow? [1]

Right. You'd need 8k VMAs. With 2 MiB THP you'd need 4096k VMAs. So ~64 processes with 64k VMAs. Not impossible to achieve if one really wants to break the system ...

Side note: a long long time ago, we used to have sub-page refcounts for THP. IIRC, that was even before we had sub-page mapcounts and was used to make COW decisions.

Quoted 7 lines

To deal with this, the best solution we've been able to come up with is to check if refcount is > INT_MAX/2 (similar to try_get_page()), and if it is, stop the operation (UFFDIO_CONTINUE or a page fault) from proceeding. In the UFFDIO_CONTINUE case, return ENOMEM. In the page fault cause, return VM_FAULT_SIGBUS (not VM_FAULT_OOM; we don't want to kill a random process).

Quoted 3 lines

You'd have to also make sure that fork() won't do the same. At least with uffd-wp, Peter also added page table copying during fork() for MAP_SHARED mappings, which would have to be handled.

If we want such a check to make a real difference, IIUC we may want to consider having similar check in:

page_ref_add
page_ref_inc
page_ref_inc_return
page_ref_add_unless

But it's unfortunate that mostly all the callers to these functions (especially the first two) do not have a retval yet at all. Considering the low possibility so far on having it overflow, maybe it can also be done for later (and I think checking negative as try_get_page would suffice too).

Quoted 4 lines

Of course, one can just disallow fork() with any HGM right from the start and keep it all simpler to not open up a can of worms there.

Quoted 8 lines

Is it reasonable, to have more than one (or a handful) of VMAs mapping a huge page via a HGM? Restricting it to a single one, would make handling much easier.

If there is ever demand for more HGM mappings, that whole problem (and complexity) could be dealt with later. ... but I assume it will already be a requirement for VMs (e.g., under QEMU) that share memory with other processes (virtiofsd and friends?)

Yes, any form of multi-proc QEMU will need that for supporting HGM postcopy.

Quoted 81 lines
Quoted 6 lines

(So David, I think this answers your question. Refcount should be handled just like THPs.)

Quoted 8 lines

2. page_mapcount() API differences

In this series, page_mapcount() returns the total number of page table references for the compound page. For example, if you have a PTE-mapped 2M page (with no other mappings), page_mapcount() for each 4K page will be 512. This is not the same as a THP: page_mapcount() would return 1 for each page. Because of the difference in page_mapcount(), we have 4 problems:

IMHO, it would actually be great to just be able to remove the sub-page mapcounts for THP and make it all simpler.

Right now, the sub-page mapcount is mostly required for making COW decisions, but only for accounting purposes IIRC (NR_ANON_THPS, NR_SHMEM_PMDMAPPED, NR_FILE_PMDMAPPED) and mlock handling IIRC. See page_remove_rmap().

If we can avoid that complexity right from the start for hugetlb, great, ..

Quoted 14 lines

i. Smaps uses page_mapcount() >= 2 to determine if hugetlb memory is "private_hugetlb" or "shared_hugetlb".
ii. Migration with MPOL_MF_MOVE will check page_mapcount() to see if the hugepage is shared or not. Pages that would otherwise be migrated now require MPOL_MF_MOVE_ALL to be migrated.
[Really both of the above are checking how many VMAs are mapping our hugepage.] iii. CoW. This isn't a problem right now because CoW is only possible with MAP_PRIVATE VMAs and HGM can only be enabled for MAP_SHARED VMAs. iv. The hwpoison handling code will check if it successfully unmapped the poisoned page. This isn't a problem right now, as hwpoison will unmap all the mappings for the hugepage, not just the 4K where the poison was found.

Quoted 4 lines

Doing it this way allows HGM to remain compatible with the hugetlb vmemmap optimization. None of the above problems strike me as particularly major, but it's unclear to me how important it is to have page_mapcount() have a consistent meaning for hugetlb vs non-hugetlb.

See below, maybe we should tackle HGM from a different direction.

Quoted 10 lines

The other way page_mapcount() (let's say the "THP-like way") could be done is like this: increment compound mapcount if we're mapping a hugetlb page normally (e.g., 1G page with a PUD). If we're mapping at high-granularity, increment the mapcount for each 4K page that is getting mapped (e.g., PMD within a 1G page: increment the mapcount for the 512 pages that are now mapped). This yields the same page_mapcount() API we had before, but we lose the hugetlb vmemmap optimization.

Quoted 7 lines

We could introduce an API like hugetlb_vma_mapcount() that would, for hugetlb, give us the number of VMAs that map a hugepage, but I don't think people would like this.

I'm curious what others think (Mike, Matthew?). I'm guessing the THP-like way is probably what most people would want, though it would be a real shame to lose the vmemmap optimization.

Heh, not me :) Having a single mapcount is certainly much cleaner. ... and if we're dealing with refcount overflows already, mapcount overflows are not an issue.

I wonder if the following crazy idea has already been discussed: treat the whole mapping as a single large logical mapping. One reference and one mapping, no matter how the individual parts are mapped into the assigned page table sub-tree.

Because for hugetlb with MAP_SHARED, we know that the complete assigned sub-tree of page tables can only map the given hugetlb page, no fragments of something else. That's very different to THP in private mappings ...

So as soon as the first piece gets mapped, we increment refcount+mapcount. Other pieces in the same subtree don't do that.

Quoted 8 lines
Quoted 6 lines

Once the last piece is unmapped (or simpler: once the complete subtree of page tables is gone), we decrement refcount+mapcount. Might require some brain power to do this tracking, but I wouldn't call it impossible right from the start.

Would such a design violate other design aspects that are important?

The question is how to maintaining above information.

Right.

Quoted 7 lines

It needs to be per-map (so one page mapped multiple times can be accounted differently), and per-page (so one mapping/vma can contain multiple pages). So far I think that's exactly the pgtable. If we can squeeze information into the pgtable it'll work out, but definitely not trivial. Or we can maintain seperate allocates for such information, but that can be extra overheads too.

If there is no sub-pgtable level, there is certainly no HGM. If there is a sub-pgtable, we can store that information in that pgtable memmap most probably. Maybe simply a pointer to the hugetlb page. As long the pointer is there, we increment the mapcount/refcount.

Either directly, or via some additional metadata. Metadata should be small and most probably "noting relevant in size" compared to the actual 1 GiB page or the 2 MiB+ of page tables to cover 1 GiB.

We could even teach most pgtable walkers to just assume that "logically" there is simply a hugtlb page mapped, without traversing the actual sub-pgtables. IIUC, only pgtable walkers that actually want to access page content (page faults, pinning) or change PTEs (mprotect, uffd) would really care. Maybe stuff like smaps could just say "well, there is a hugetlb page mapped" and continue. Just a thought.

Quoted 14 lines

So far I'd still consider going with reusing thp mapcounts, which will mostly be what James mentioned above. The only difference is I'm not sure whether we should allow mapping e.g. 2M ranges for 1G pages. THP mapcount doesn't have intermediate layer to maintain mapcount information like 2M, so to me it's easier we start with only mapping either the hpage size or PAGE_SIZE, not any intermediate size allowed.

Having intermediate size mapping allowed can at least be error prone to me. One example is if some pgtable walker found a 2M page, it may easily fetch the PFN out of it, assuming it's a compound page and it should satisfy PageHead(pfn)==true but it'll start to break here, because the 2M PFN will only be a small page pfn for the 1G huge page in this case.

Quoted 3 lines

To me, intermediate sized mappings are good to have but not required to resolve HGM problems, at least so far. Said that, I'm fine with looking at what it'll look like if James would like to keep persuing that direction.

Yeah, was just an idea from my side to avoid most of the refcount and mapcount issues -- in theory :)

Let me think about that all a bit more ...

Signature · 3 lines
Thanks,

David / dhildenb
#89 lore raw

PATCH 22/46 mm: rmap: provide pte_order in page_vma_mapped_walk

In reply to James Houghton

page_vma_mapped_walk callers will need this information to know how HugeTLB pages are mapped. pte_order only applies if pte is not NULL.

---
 include/linux/rmap.h | 1 +
 mm/page_vma_mapped.c | 3 +++
 2 files changed, 4 insertions(+)
include/linux/rmap.h +1 -0
index bd3504d11b15..e0557ede2951 100644
@@ -378,6 +378,7 @@ struct page_vma_mapped_walk {
 	pmd_t *pmd;
 	pte_t *pte;
 	spinlock_t *ptl;
+	unsigned int pte_order;
 	unsigned int flags;
 };
 
mm/page_vma_mapped.c +3 -0
index 4e448cfbc6ef..08295b122ad6 100644
@@ -16,6 +16,7 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw)
 static bool map_pte(struct page_vma_mapped_walk *pvmw)
 {
 	pvmw->pte = pte_offset_map(pvmw->pmd, pvmw->address);
+	pvmw->pte_order = 0;
 	if (!(pvmw->flags & PVMW_SYNC)) {
 		if (pvmw->flags & PVMW_MIGRATION) {
 			if (!is_swap_pte(*pvmw->pte))
@@ -177,6 +178,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
 		if (!pvmw->pte)
 			return false;
 
+		pvmw->pte_order = huge_page_order(hstate);
 		pvmw->ptl = huge_pte_lock(hstate, mm, pvmw->pte);
 		if (!check_pte(pvmw))
 			return not_found(pvmw);
@@ -272,6 +274,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
 				}
 				pte_unmap(pvmw->pte);
 				pvmw->pte = NULL;
+				pvmw->pte_order = 0;
 				goto restart;
 			}
 			pvmw->pte++;
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#90 lore raw

PATCH 23/46 mm: rmap: make page_vma_mapped_walk callers use pte_order

In reply to James Houghton

This also updates the callers' hugetlb mapcounting code to handle mapcount properly for subpage-mapped hugetlb pages.

---
 mm/migrate.c |  2 +-
 mm/rmap.c    | 17 +++++++++++++----
 2 files changed, 14 insertions(+), 5 deletions(-)
mm/migrate.c +1 -1
index 832f639fc49a..0062689f4878 100644
@@ -244,7 +244,7 @@ static bool remove_migration_pte(struct folio *folio,
 
 #ifdef CONFIG_HUGETLB_PAGE
 		if (folio_test_hugetlb(folio)) {
-			unsigned int shift = huge_page_shift(hstate_vma(vma));
+			unsigned int shift = pvmw.pte_order + PAGE_SHIFT;
 
 			pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
 			if (folio_test_anon(folio))
mm/rmap.c +13 -4
index 8a24b90d9531..ff7e6c770b0a 100644
@@ -1608,7 +1608,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		if (PageHWPoison(subpage) && !(flags & TTU_IGNORE_HWPOISON)) {
 			pteval = swp_entry_to_pte(make_hwpoison_entry(subpage));
 			if (folio_test_hugetlb(folio)) {
-				hugetlb_count_sub(folio_nr_pages(folio), mm);
+				hugetlb_count_sub(1UL << pvmw.pte_order, mm);
 				set_huge_pte_at(mm, address, pvmw.pte, pteval);
 			} else {
 				dec_mm_counter(mm, mm_counter(&folio->page));
@@ -1767,7 +1767,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		 *
 		 * See Documentation/mm/mmu_notifier.rst
 		 */
-		page_remove_rmap(subpage, vma, folio_test_hugetlb(folio));
+		if (folio_test_hugetlb(folio))
+			page_remove_rmap(&folio->page, vma, true);
+		else
+			page_remove_rmap(subpage, vma, false);
+
 		if (vma->vm_flags & VM_LOCKED)
 			mlock_page_drain_local();
 		folio_put(folio);
@@ -2030,7 +2034,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 		} else if (PageHWPoison(subpage)) {
 			pteval = swp_entry_to_pte(make_hwpoison_entry(subpage));
 			if (folio_test_hugetlb(folio)) {
-				hugetlb_count_sub(folio_nr_pages(folio), mm);
+				hugetlb_count_sub(1L << pvmw.pte_order, mm);
 				set_huge_pte_at(mm, address, pvmw.pte, pteval);
 			} else {
 				dec_mm_counter(mm, mm_counter(&folio->page));
@@ -2122,7 +2126,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 		 *
 		 * See Documentation/mm/mmu_notifier.rst
 		 */
-		page_remove_rmap(subpage, vma, folio_test_hugetlb(folio));
+		if (folio_test_hugetlb(folio))
+			page_remove_rmap(&folio->page, vma, true);
+		else
+			page_remove_rmap(subpage, vma, false);
 		if (vma->vm_flags & VM_LOCKED)
 			mlock_page_drain_local();
 		folio_put(folio);
@@ -2206,6 +2213,8 @@ static bool page_make_device_exclusive_one(struct folio *folio,
 				      args->owner);
 	mmu_notifier_invalidate_range_start(&range);
 
+	VM_BUG_ON_FOLIO(folio_test_hugetlb(folio), folio);
+
 	while (page_vma_mapped_walk(&pvmw)) {
 		/* Unexpected PMD-mapped THP? */
 		VM_BUG_ON_FOLIO(!pvmw.pte, folio);
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#91 lore raw

PATCH 24/46 rmap: update hugetlb lock comment for HGM

In reply to James Houghton

The VMA lock is used to prevent high-granularity HugeTLB mappings from being collapsed while other threads are doing high-granularity page table walks.

---
 include/linux/hugetlb.h | 12 ++++++++++++
 mm/rmap.c               |  3 ++-
 2 files changed, 14 insertions(+), 1 deletion(-)
include/linux/hugetlb.h +12 -0
index b7cf45535d64..daf993fdbc38 100644
@@ -156,6 +156,18 @@ struct file_region {
 #endif
 };
 
+/*
+ * The HugeTLB VMA lock is used to synchronize HugeTLB page table walks.
+ * Right now, it is only used for VM_SHARED mappings.
+ * - The read lock is held when we want to stabilize mappings (prevent PMD
+ *   unsharing or MADV_COLLAPSE for high-granularity mappings).
+ * - The write lock is held when we want to free mappings (PMD unsharing and
+ *   MADV_COLLAPSE for high-granularity mappings).
+ *
+ * Note: For PMD unsharing and MADV_COLLAPSE, the i_mmap_rwsem is held for
+ * writing as well, so page table walkers will also be safe if they hold
+ * i_mmap_rwsem for at least reading. See hugetlb_walk() for more information.
+ */
 struct hugetlb_vma_lock {
 	struct kref refs;
 	struct rw_semaphore rw_sema;
mm/rmap.c +2 -1
index ff7e6c770b0a..076ea77010e5 100644
@@ -47,7 +47,8 @@
  *
  * hugetlbfs PageHuge() take locks in this order:
  *   hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
- *     vma_lock (hugetlb specific lock for pmd_sharing)
+ *     vma_lock (hugetlb specific lock for pmd_sharing and high-granularity
+ *               mapping)
  *       mapping->i_mmap_rwsem (also used for hugetlb pmd sharing)
  *         page->flags PG_locked (lock_page)
  */
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#92 lore raw

PATCH 25/46 hugetlb: update page_vma_mapped to do high-granularity walks

In reply to James Houghton

Update the HugeTLB logic to look a lot more like the PTE-mapped THP logic. When a user calls us in a loop, we will update pvmw->address to walk to each page table entry that could possibly map the hugepage containing pvmw->pfn.

Make use of the new pte_order so callers know what size PTE they're getting.

The !pte failure case is changed to call not_found() instead of just returning false. This should be a no-op, but if somehow the hstate-level PTE were deallocated between iterations, not_found() should be called to drop locks.

---
 mm/page_vma_mapped.c | 59 +++++++++++++++++++++++++++++++-------------
 1 file changed, 42 insertions(+), 17 deletions(-)
mm/page_vma_mapped.c +42 -17
index 08295b122ad6..03e8a4987272 100644
@@ -133,7 +133,8 @@ static void step_forward(struct page_vma_mapped_walk *pvmw, unsigned long size)
  *
  * Returns true if the page is mapped in the vma. @pvmw->pmd and @pvmw->pte point
  * to relevant page table entries. @pvmw->ptl is locked. @pvmw->address is
- * adjusted if needed (for PTE-mapped THPs).
+ * adjusted if needed (for PTE-mapped THPs and high-granularity-mapped HugeTLB
+ * pages).
  *
  * If @pvmw->pmd is set but @pvmw->pte is not, you have found PMD-mapped page
  * (usually THP). For PTE-mapped THP, you should run page_vma_mapped_walk() in
@@ -165,23 +166,47 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
 
 	if (unlikely(is_vm_hugetlb_page(vma))) {
 		struct hstate *hstate = hstate_vma(vma);
-		unsigned long size = huge_page_size(hstate);
-		/* The only possible mapping was handled on last iteration */
-		if (pvmw->pte)
-			return not_found(pvmw);
-		/*
-		 * All callers that get here will already hold the
-		 * i_mmap_rwsem.  Therefore, no additional locks need to be
-		 * taken before calling hugetlb_walk().
-		 */
-		pvmw->pte = hugetlb_walk(vma, pvmw->address, size);
-		if (!pvmw->pte)
-			return false;
+		struct hugetlb_pte hpte;
+		pte_t pteval;
+
+		end = (pvmw->address & huge_page_mask(hstate)) +
+			huge_page_size(hstate);
+
+		do {
+			if (pvmw->pte) {
+				if (pvmw->ptl)
+					spin_unlock(pvmw->ptl);
+				pvmw->ptl = NULL;
+				pvmw->address += PAGE_SIZE << pvmw->pte_order;
+				if (pvmw->address >= end)
+					return not_found(pvmw);
+			}
 
-		pvmw->pte_order = huge_page_order(hstate);
-		pvmw->ptl = huge_pte_lock(hstate, mm, pvmw->pte);
-		if (!check_pte(pvmw))
-			return not_found(pvmw);
+			/*
+			 * All callers that get here will already hold the
+			 * i_mmap_rwsem. Therefore, no additional locks need to
+			 * be taken before calling hugetlb_walk().
+			 */
+			if (hugetlb_full_walk(&hpte, vma, pvmw->address))
+				return not_found(pvmw);
+
+retry:
+			pvmw->pte = hpte.ptep;
+			pvmw->pte_order = hpte.shift - PAGE_SHIFT;
+			pvmw->ptl = hugetlb_pte_lock(&hpte);
+			pteval = huge_ptep_get(hpte.ptep);
+			if (pte_present(pteval) && !hugetlb_pte_present_leaf(
+						&hpte, pteval)) {
+				/*
+				 * Someone split from under us, so keep
+				 * walking.
+				 */
+				spin_unlock(pvmw->ptl);
+				hugetlb_full_walk_continue(&hpte, vma,
+						pvmw->address);
+				goto retry;
+			}
+		} while (!check_pte(pvmw));
 		return true;
 	}
 
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#93 lore raw

PATCH 26/46 hugetlb: add HGM support for copy_hugetlb_page_range

In reply to James Houghton

This allows fork() to work with high-granularity mappings. The page table structure is copied such that partially mapped regions will remain partially mapped in the same way for the new process.

A page's reference count is incremented for *each* portion of it that is mapped in the page table. For example, if you have a PMD-mapped 1G page, the reference count and mapcount will be incremented by 512.

---
 mm/hugetlb.c | 75 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 50 insertions(+), 25 deletions(-)
mm/hugetlb.c +50 -25
index 718572444a73..21a5116f509b 100644
@@ -5106,7 +5106,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 			    struct vm_area_struct *src_vma)
 {
 	pte_t *src_pte, *dst_pte, entry;
-	struct page *ptepage;
+	struct hugetlb_pte src_hpte, dst_hpte;
+	struct page *ptepage, *hpage;
 	unsigned long addr;
 	bool cow = is_cow_mapping(src_vma->vm_flags);
 	struct hstate *h = hstate_vma(src_vma);
@@ -5126,26 +5127,34 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 	} else {
 		/*
 		 * For shared mappings the vma lock must be held before
-		 * calling hugetlb_walk() in the src vma. Otherwise, the
-		 * returned ptep could go away if part of a shared pmd and
-		 * another thread calls huge_pmd_unshare.
+		 * calling hugetlb_full_walk() in the src vma. Otherwise, the
+		 * returned hpte could go away if
+		 *  - part of a shared pmd and another thread calls
+		 *  - huge_pmd_unshare, or
+		 *  - another thread collapses a high-granularity mapping.
 		 */
 		hugetlb_vma_lock_read(src_vma);
 	}
 
 	last_addr_mask = hugetlb_mask_last_page(h);
-	for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) {
+	addr = src_vma->vm_start;
+	while (addr < src_vma->vm_end) {
 		spinlock_t *src_ptl, *dst_ptl;
-		src_pte = hugetlb_walk(src_vma, addr, sz);
-		if (!src_pte) {
-			addr |= last_addr_mask;
+		unsigned long hpte_sz;
+
+		if (hugetlb_full_walk(&src_hpte, src_vma, addr)) {
+			addr = (addr | last_addr_mask) + sz;
 			continue;
 		}
-		dst_pte = huge_pte_alloc(dst, dst_vma, addr, sz);
-		if (!dst_pte) {
-			ret = -ENOMEM;
+		ret = hugetlb_full_walk_alloc(&dst_hpte, dst_vma, addr,
+				hugetlb_pte_size(&src_hpte));
+		if (ret)
 			break;
-		}
+
+		src_pte = src_hpte.ptep;
+		dst_pte = dst_hpte.ptep;
+
+		hpte_sz = hugetlb_pte_size(&src_hpte);
 
 		/*
 		 * If the pagetables are shared don't copy or take references.
@@ -5155,13 +5164,14 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 		 * another vma. So page_count of ptep page is checked instead
 		 * to reliably determine whether pte is shared.
 		 */
-		if (page_count(virt_to_page(dst_pte)) > 1) {
-			addr |= last_addr_mask;
+		if (hugetlb_pte_size(&dst_hpte) == sz &&
+		    page_count(virt_to_page(dst_pte)) > 1) {
+			addr = (addr | last_addr_mask) + sz;
 			continue;
 		}
 
-		dst_ptl = huge_pte_lock(h, dst, dst_pte);
-		src_ptl = huge_pte_lockptr(huge_page_shift(h), src, src_pte);
+		dst_ptl = hugetlb_pte_lock(&dst_hpte);
+		src_ptl = hugetlb_pte_lockptr(&src_hpte);
 		spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
 		entry = huge_ptep_get(src_pte);
 again:
@@ -5205,10 +5215,15 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 			 */
 			if (userfaultfd_wp(dst_vma))
 				set_huge_pte_at(dst, addr, dst_pte, entry);
+		} else if (!hugetlb_pte_present_leaf(&src_hpte, entry)) {
+			/* Retry the walk. */
+			spin_unlock(src_ptl);
+			spin_unlock(dst_ptl);
+			continue;
 		} else {
-			entry = huge_ptep_get(src_pte);
 			ptepage = pte_page(entry);
-			get_page(ptepage);
+			hpage = compound_head(ptepage);
+			get_page(hpage);
 
 			/*
 			 * Failing to duplicate the anon rmap is a rare case
@@ -5220,25 +5235,31 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 			 * need to be without the pgtable locks since we could
 			 * sleep during the process.
 			 */
-			if (!PageAnon(ptepage)) {
-				page_dup_file_rmap(ptepage, true);
-			} else if (page_try_dup_anon_rmap(ptepage, true,
+			if (!PageAnon(hpage)) {
+				page_dup_file_rmap(hpage, true);
+			} else if (page_try_dup_anon_rmap(hpage, true,
 							  src_vma)) {
 				pte_t src_pte_old = entry;
 				struct page *new;
 
+				if (hugetlb_pte_size(&src_hpte) != sz) {
+					put_page(hpage);
+					ret = -EINVAL;
+					break;
+				}
+
 				spin_unlock(src_ptl);
 				spin_unlock(dst_ptl);
 				/* Do not use reserve as it's private owned */
 				new = alloc_huge_page(dst_vma, addr, 1);
 				if (IS_ERR(new)) {
-					put_page(ptepage);
+					put_page(hpage);
 					ret = PTR_ERR(new);
 					break;
 				}
-				copy_user_huge_page(new, ptepage, addr, dst_vma,
+				copy_user_huge_page(new, hpage, addr, dst_vma,
 						    npages);
-				put_page(ptepage);
+				put_page(hpage);
 
 				/* Install the new huge page if src pte stable */
 				dst_ptl = huge_pte_lock(h, dst, dst_pte);
@@ -5256,6 +5277,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 				hugetlb_install_page(dst_vma, dst_pte, addr, new);
 				spin_unlock(src_ptl);
 				spin_unlock(dst_ptl);
+				addr += hugetlb_pte_size(&src_hpte);
 				continue;
 			}
 
@@ -5272,10 +5294,13 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 			}
 
 			set_huge_pte_at(dst, addr, dst_pte, entry);
-			hugetlb_count_add(npages, dst);
+			hugetlb_count_add(
+					hugetlb_pte_size(&dst_hpte) / PAGE_SIZE,
+					dst);
 		}
 		spin_unlock(src_ptl);
 		spin_unlock(dst_ptl);
+		addr += hugetlb_pte_size(&src_hpte);
 	}
 
 	if (cow) {
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#94 lore raw

PATCH 27/46 hugetlb: add HGM support for move_hugetlb_page_tables

In reply to James Houghton

This is very similar to the support that was added to copy_hugetlb_page_range. We simply do a high-granularity walk now, and most of the rest of the code stays the same.

---
 mm/hugetlb.c | 47 +++++++++++++++++++++++++++--------------------
 1 file changed, 27 insertions(+), 20 deletions(-)
mm/hugetlb.c +27 -20
index 21a5116f509b..582d14a206b5 100644
@@ -5313,16 +5313,16 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 	return ret;
 }
 
-static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
-			  unsigned long new_addr, pte_t *src_pte, pte_t *dst_pte)
+static void move_hugetlb_pte(struct vm_area_struct *vma, unsigned long old_addr,
+			     unsigned long new_addr, struct hugetlb_pte *src_hpte,
+			     struct hugetlb_pte *dst_hpte)
 {
-	struct hstate *h = hstate_vma(vma);
 	struct mm_struct *mm = vma->vm_mm;
 	spinlock_t *src_ptl, *dst_ptl;
 	pte_t pte;
 
-	dst_ptl = huge_pte_lock(h, mm, dst_pte);
-	src_ptl = huge_pte_lockptr(huge_page_shift(h), mm, src_pte);
+	dst_ptl = hugetlb_pte_lock(dst_hpte);
+	src_ptl = hugetlb_pte_lockptr(src_hpte);
 
 	/*
 	 * We don't have to worry about the ordering of src and dst ptlocks
@@ -5331,8 +5331,8 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
 	if (src_ptl != dst_ptl)
 		spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
 
-	pte = huge_ptep_get_and_clear(mm, old_addr, src_pte);
-	set_huge_pte_at(mm, new_addr, dst_pte, pte);
+	pte = huge_ptep_get_and_clear(mm, old_addr, src_hpte->ptep);
+	set_huge_pte_at(mm, new_addr, dst_hpte->ptep, pte);
 
 	if (src_ptl != dst_ptl)
 		spin_unlock(src_ptl);
@@ -5350,9 +5350,9 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long old_end = old_addr + len;
 	unsigned long last_addr_mask;
-	pte_t *src_pte, *dst_pte;
 	struct mmu_notifier_range range;
 	bool shared_pmd = false;
+	struct hugetlb_pte src_hpte, dst_hpte;
 
 	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, old_addr,
 				old_end);
@@ -5368,28 +5368,35 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,
 	/* Prevent race with file truncation */
 	hugetlb_vma_lock_write(vma);
 	i_mmap_lock_write(mapping);
-	for (; old_addr < old_end; old_addr += sz, new_addr += sz) {
-		src_pte = hugetlb_walk(vma, old_addr, sz);
-		if (!src_pte) {
-			old_addr |= last_addr_mask;
-			new_addr |= last_addr_mask;
+	while (old_addr < old_end) {
+		if (hugetlb_full_walk(&src_hpte, vma, old_addr)) {
+			/* The hstate-level PTE wasn't allocated. */
+			old_addr = (old_addr | last_addr_mask) + sz;
+			new_addr = (new_addr | last_addr_mask) + sz;
 			continue;
 		}
-		if (huge_pte_none(huge_ptep_get(src_pte)))
+
+		if (huge_pte_none(huge_ptep_get(src_hpte.ptep))) {
+			old_addr += hugetlb_pte_size(&src_hpte);
+			new_addr += hugetlb_pte_size(&src_hpte);
 			continue;
+		}
 
-		if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) {
+		if (hugetlb_pte_size(&src_hpte) == sz &&
+		    huge_pmd_unshare(mm, vma, old_addr, src_hpte.ptep)) {
 			shared_pmd = true;
-			old_addr |= last_addr_mask;
-			new_addr |= last_addr_mask;
+			old_addr = (old_addr | last_addr_mask) + sz;
+			new_addr = (new_addr | last_addr_mask) + sz;
 			continue;
 		}
 
-		dst_pte = huge_pte_alloc(mm, new_vma, new_addr, sz);
-		if (!dst_pte)
+		if (hugetlb_full_walk_alloc(&dst_hpte, new_vma, new_addr,
+					hugetlb_pte_size(&src_hpte)))
 			break;
 
-		move_huge_pte(vma, old_addr, new_addr, src_pte, dst_pte);
+		move_hugetlb_pte(vma, old_addr, new_addr, &src_hpte, &dst_hpte);
+		old_addr += hugetlb_pte_size(&src_hpte);
+		new_addr += hugetlb_pte_size(&src_hpte);
 	}
 
 	if (shared_pmd)
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#95 lore raw

PATCH 28/46 hugetlb: add HGM support for hugetlb_fault and hugetlb_no_page

In reply to James Houghton

Update the page fault handler to support high-granularity page faults. While handling a page fault on a partially-mapped HugeTLB page, if the PTE we find with hugetlb_pte_walk is none, then we will replace it with a leaf-level PTE to map the page. To give some examples:
1. For a completely unmapped 1G page, it will be mapped with a 1G PUD.
2. For a 1G page that has its first 512M mapped, any faults on the unmapped sections will result in 2M PMDs mapping each unmapped 2M section.
3. For a 1G page that has only its first 4K mapped, a page fault on its second 4K section will get a 4K PTE to map it.

Unless high-granularity mappings are created via UFFDIO_CONTINUE, it is impossible for hugetlb_fault to create high-granularity mappings.

This commit does not handle hugetlb_wp right now, and it doesn't handle HugeTLB page migration and swap entries.

The BUG_ON in huge_pte_alloc is removed, as it is not longer valid when HGM is possible. HGM can be disabled if the VMA lock cannot be allocated after a VMA is split, yet high-granularity mappings may still exist.

---
 mm/hugetlb.c | 115 ++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 81 insertions(+), 34 deletions(-)
mm/hugetlb.c +81 -34
index 582d14a206b5..8e690a22456a 100644
@@ -117,6 +117,18 @@ enum hugetlb_level hpage_size_to_level(unsigned long sz)
 	return HUGETLB_LEVEL_PGD;
 }
 
+/*
+ * Find the subpage that corresponds to `addr` in `hpage`.
+ */
+static struct page *hugetlb_find_subpage(struct hstate *h, struct page *hpage,
+				 unsigned long addr)
+{
+	size_t idx = (addr & ~huge_page_mask(h))/PAGE_SIZE;
+
+	BUG_ON(idx >= pages_per_huge_page(h));
+	return &hpage[idx];
+}
+
 static inline bool subpool_is_free(struct hugepage_subpool *spool)
 {
 	if (spool->count)
@@ -5926,14 +5938,14 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma,
  * Recheck pte with pgtable lock.  Returns true if pte didn't change, or
  * false if pte changed or is changing.
  */
-static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm,
-			       pte_t *ptep, pte_t old_pte)
+static bool hugetlb_pte_stable(struct hstate *h, struct hugetlb_pte *hpte,
+			       pte_t old_pte)
 {
 	spinlock_t *ptl;
 	bool same;
 
-	ptl = huge_pte_lock(h, mm, ptep);
-	same = pte_same(huge_ptep_get(ptep), old_pte);
+	ptl = hugetlb_pte_lock(hpte);
+	same = pte_same(huge_ptep_get(hpte->ptep), old_pte);
 	spin_unlock(ptl);
 
 	return same;
@@ -5942,17 +5954,18 @@ static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm,
 static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
 			struct vm_area_struct *vma,
 			struct address_space *mapping, pgoff_t idx,
-			unsigned long address, pte_t *ptep,
+			unsigned long address, struct hugetlb_pte *hpte,
 			pte_t old_pte, unsigned int flags)
 {
 	struct hstate *h = hstate_vma(vma);
 	vm_fault_t ret = VM_FAULT_SIGBUS;
 	int anon_rmap = 0;
 	unsigned long size;
-	struct page *page;
+	struct page *page, *subpage;
 	pte_t new_pte;
 	spinlock_t *ptl;
 	unsigned long haddr = address & huge_page_mask(h);
+	unsigned long haddr_hgm = address & hugetlb_pte_mask(hpte);
 	bool new_page, new_pagecache_page = false;
 	u32 hash = hugetlb_fault_mutex_hash(mapping, idx);
 
@@ -5997,7 +6010,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
 			 * never happen on the page after UFFDIO_COPY has
 			 * correctly installed the page and returned.
 			 */
-			if (!hugetlb_pte_stable(h, mm, ptep, old_pte)) {
+			if (!hugetlb_pte_stable(h, hpte, old_pte)) {
 				ret = 0;
 				goto out;
 			}
@@ -6021,7 +6034,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
 			 * here.  Before returning error, get ptl and make
 			 * sure there really is no pte entry.
 			 */
-			if (hugetlb_pte_stable(h, mm, ptep, old_pte))
+			if (hugetlb_pte_stable(h, hpte, old_pte))
 				ret = vmf_error(PTR_ERR(page));
 			else
 				ret = 0;
@@ -6071,7 +6084,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
 			unlock_page(page);
 			put_page(page);
 			/* See comment in userfaultfd_missing() block above */
-			if (!hugetlb_pte_stable(h, mm, ptep, old_pte)) {
+			if (!hugetlb_pte_stable(h, hpte, old_pte)) {
 				ret = 0;
 				goto out;
 			}
@@ -6096,30 +6109,43 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
 		vma_end_reservation(h, vma, haddr);
 	}
 
-	ptl = huge_pte_lock(h, mm, ptep);
+	ptl = hugetlb_pte_lock(hpte);
 	ret = 0;
-	/* If pte changed from under us, retry */
-	if (!pte_same(huge_ptep_get(ptep), old_pte))
+	/*
+	 * If pte changed from under us, retry.
+	 *
+	 * When dealing with high-granularity-mapped PTEs, it's possible that
+	 * a non-contiguous PTE within our contiguous PTE group gets populated,
+	 * in which case, we need to retry here. This is NOT caught here, and
+	 * will need to be addressed when HGM is supported for architectures
+	 * that support contiguous PTEs.
+	 */
+	if (!pte_same(huge_ptep_get(hpte->ptep), old_pte))
 		goto backout;
 
 	if (anon_rmap)
 		hugepage_add_new_anon_rmap(page, vma, haddr);
 	else
 		page_dup_file_rmap(page, true);
-	new_pte = make_huge_pte(vma, page, ((vma->vm_flags & VM_WRITE)
-				&& (vma->vm_flags & VM_SHARED)));
+
+	subpage = hugetlb_find_subpage(h, page, haddr_hgm);
+	new_pte = make_huge_pte_with_shift(vma, subpage,
+			((vma->vm_flags & VM_WRITE)
+			 && (vma->vm_flags & VM_SHARED)),
+			hpte->shift);
 	/*
 	 * If this pte was previously wr-protected, keep it wr-protected even
 	 * if populated.
 	 */
 	if (unlikely(pte_marker_uffd_wp(old_pte)))
 		new_pte = huge_pte_mkuffd_wp(new_pte);
-	set_huge_pte_at(mm, haddr, ptep, new_pte);
+	set_huge_pte_at(mm, haddr_hgm, hpte->ptep, new_pte);
 
-	hugetlb_count_add(pages_per_huge_page(h), mm);
+	hugetlb_count_add(hugetlb_pte_size(hpte) / PAGE_SIZE, mm);
 	if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) {
+		WARN_ON_ONCE(hugetlb_pte_size(hpte) != huge_page_size(h));
 		/* Optimization, do the COW without a second fault */
-		ret = hugetlb_wp(mm, vma, address, ptep, flags, page, ptl);
+		ret = hugetlb_wp(mm, vma, address, hpte->ptep, flags, page, ptl);
 	}
 
 	spin_unlock(ptl);
@@ -6176,17 +6202,20 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx)
 vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 			unsigned long address, unsigned int flags)
 {
-	pte_t *ptep, entry;
+	pte_t entry;
 	spinlock_t *ptl;
 	vm_fault_t ret;
 	u32 hash;
 	pgoff_t idx;
 	struct page *page = NULL;
+	struct page *subpage = NULL;
 	struct page *pagecache_page = NULL;
 	struct hstate *h = hstate_vma(vma);
 	struct address_space *mapping;
 	int need_wait_lock = 0;
 	unsigned long haddr = address & huge_page_mask(h);
+	unsigned long haddr_hgm;
+	struct hugetlb_pte hpte;
 
 	/*
 	 * Serialize hugepage allocation and instantiation, so that we don't
@@ -6200,26 +6229,26 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 
 	/*
 	 * Acquire vma lock before calling huge_pte_alloc and hold
-	 * until finished with ptep.  This prevents huge_pmd_unshare from
-	 * being called elsewhere and making the ptep no longer valid.
+	 * until finished with hpte.  This prevents huge_pmd_unshare from
+	 * being called elsewhere and making the hpte no longer valid.
 	 */
 	hugetlb_vma_lock_read(vma);
-	ptep = huge_pte_alloc(mm, vma, haddr, huge_page_size(h));
-	if (!ptep) {
+	if (hugetlb_full_walk_alloc(&hpte, vma, address, 0)) {
 		hugetlb_vma_unlock_read(vma);
 		mutex_unlock(&hugetlb_fault_mutex_table[hash]);
 		return VM_FAULT_OOM;
 	}
 
-	entry = huge_ptep_get(ptep);
+	entry = huge_ptep_get(hpte.ptep);
 	/* PTE markers should be handled the same way as none pte */
-	if (huge_pte_none_mostly(entry))
+	if (huge_pte_none_mostly(entry)) {
 		/*
 		 * hugetlb_no_page will drop vma lock and hugetlb fault
 		 * mutex internally, which make us return immediately.
 		 */
-		return hugetlb_no_page(mm, vma, mapping, idx, address, ptep,
+		return hugetlb_no_page(mm, vma, mapping, idx, address, &hpte,
 				      entry, flags);
+	}
 
 	ret = 0;
 
@@ -6240,7 +6269,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 			 * be released there.
 			 */
 			mutex_unlock(&hugetlb_fault_mutex_table[hash]);
-			migration_entry_wait_huge(vma, ptep);
+			migration_entry_wait_huge(vma, hpte.ptep);
 			return 0;
 		} else if (unlikely(is_hugetlb_entry_hwpoisoned(entry)))
 			ret = VM_FAULT_HWPOISON_LARGE |
@@ -6248,6 +6277,10 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 		goto out_mutex;
 	}
 
+	if (!hugetlb_pte_present_leaf(&hpte, entry))
+		/* We raced with someone splitting the entry. */
+		goto out_mutex;
+
 	/*
 	 * If we are going to COW/unshare the mapping later, we examine the
 	 * pending reservations for this page now. This will ensure that any
@@ -6267,14 +6300,17 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 		pagecache_page = find_lock_page(mapping, idx);
 	}
 
-	ptl = huge_pte_lock(h, mm, ptep);
+	ptl = hugetlb_pte_lock(&hpte);
 
 	/* Check for a racing update before calling hugetlb_wp() */
-	if (unlikely(!pte_same(entry, huge_ptep_get(ptep))))
+	if (unlikely(!pte_same(entry, huge_ptep_get(hpte.ptep))))
 		goto out_ptl;
 
+	/* haddr_hgm is the base address of the region that hpte maps. */
+	haddr_hgm = address & hugetlb_pte_mask(&hpte);
+
 	/* Handle userfault-wp first, before trying to lock more pages */
-	if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) &&
+	if (userfaultfd_wp(vma) && huge_pte_uffd_wp(entry) &&
 	    (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) {
 		struct vm_fault vmf = {
 			.vma = vma,
@@ -6298,7 +6334,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 	 * pagecache_page, so here we need take the former one
 	 * when page != pagecache_page or !pagecache_page.
 	 */
-	page = pte_page(entry);
+	subpage = pte_page(entry);
+	page = compound_head(subpage);
 	if (page != pagecache_page)
 		if (!trylock_page(page)) {
 			need_wait_lock = 1;
@@ -6309,7 +6346,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 
 	if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) {
 		if (!huge_pte_write(entry)) {
-			ret = hugetlb_wp(mm, vma, address, ptep, flags,
+			WARN_ON_ONCE(hugetlb_pte_size(&hpte) !=
+					huge_page_size(h));
+			ret = hugetlb_wp(mm, vma, address, hpte.ptep, flags,
 					 pagecache_page, ptl);
 			goto out_put_page;
 		} else if (likely(flags & FAULT_FLAG_WRITE)) {
@@ -6317,9 +6356,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 		}
 	}
 	entry = pte_mkyoung(entry);
-	if (huge_ptep_set_access_flags(vma, haddr, ptep, entry,
+	if (huge_ptep_set_access_flags(vma, haddr_hgm, hpte.ptep, entry,
 						flags & FAULT_FLAG_WRITE))
-		update_mmu_cache(vma, haddr, ptep);
+		update_mmu_cache(vma, haddr_hgm, hpte.ptep);
 out_put_page:
 	if (page != pagecache_page)
 		unlock_page(page);
@@ -7523,6 +7562,9 @@ int hugetlb_full_walk(struct hugetlb_pte *hpte,
 /*
  * hugetlb_full_walk_alloc - do a high-granularity walk, potentially allocate
  *	new PTEs.
+ *
+ * If @target_sz is 0, then only attempt to allocate the hstate-level PTE, and
+ * walk as far as we can go.
  */
 int hugetlb_full_walk_alloc(struct hugetlb_pte *hpte,
 				   struct vm_area_struct *vma,
@@ -7541,6 +7583,12 @@ int hugetlb_full_walk_alloc(struct hugetlb_pte *hpte,
 	if (!ptep)
 		return -ENOMEM;
 
+	if (!target_sz) {
+		WARN_ON_ONCE(hugetlb_hgm_walk_uninit(hpte, ptep, vma, addr,
+					PAGE_SIZE, false));
+		return 0;
+	}
+
 	return hugetlb_hgm_walk_uninit(hpte, ptep, vma, addr, target_sz, true);
 }
 
@@ -7569,7 +7617,6 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 				pte = (pte_t *)pmd_alloc(mm, pud, addr);
 		}
 	}
-	BUG_ON(pte && pte_present(*pte) && !pte_huge(*pte));
 
 	return pte;
 }
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#96 lore raw

PATCH 29/46 rmap: in try_to_{migrate,unmap}_one, check head page for page flags

In reply to James Houghton

The main complication here is that HugeTLB pages have their poison status stored in the head page as the HWPoison page flag. Because HugeTLB high-granularity mapping can create PTEs that point to subpages instead of always the head of a hugepage, we need to check the compound_head for page flags.

---
 mm/rmap.c | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)
mm/rmap.c +26 -8
index 076ea77010e5..a6004d6b0415 100644
@@ -1456,10 +1456,11 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 	struct mm_struct *mm = vma->vm_mm;
 	DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0);
 	pte_t pteval;
-	struct page *subpage;
+	struct page *subpage, *page_flags_page;
 	bool anon_exclusive, ret = true;
 	struct mmu_notifier_range range;
 	enum ttu_flags flags = (enum ttu_flags)(long)arg;
+	bool page_poisoned;
 
 	/*
 	 * When racing against e.g. zap_pte_range() on another cpu,
@@ -1512,9 +1513,17 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 
 		subpage = folio_page(folio,
 					pte_pfn(*pvmw.pte) - folio_pfn(folio));
+		/*
+		 * We check the page flags of HugeTLB pages by checking the
+		 * head page.
+		 */
+		page_flags_page = folio_test_hugetlb(folio)
+			? &folio->page
+			: subpage;
+		page_poisoned = PageHWPoison(page_flags_page);
 		address = pvmw.address;
 		anon_exclusive = folio_test_anon(folio) &&
-				 PageAnonExclusive(subpage);
+				 PageAnonExclusive(page_flags_page);
 
 		if (folio_test_hugetlb(folio)) {
 			bool anon = folio_test_anon(folio);
@@ -1523,7 +1532,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 			 * The try_to_unmap() is only passed a hugetlb page
 			 * in the case where the hugetlb page is poisoned.
 			 */
-			VM_BUG_ON_PAGE(!PageHWPoison(subpage), subpage);
+			VM_BUG_ON_FOLIO(!page_poisoned, folio);
 			/*
 			 * huge_pmd_unshare may unmap an entire PMD page.
 			 * There is no way of knowing exactly which PMDs may
@@ -1606,7 +1615,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		/* Update high watermark before we lower rss */
 		update_hiwater_rss(mm);
 
-		if (PageHWPoison(subpage) && !(flags & TTU_IGNORE_HWPOISON)) {
+		if (page_poisoned && !(flags & TTU_IGNORE_HWPOISON)) {
 			pteval = swp_entry_to_pte(make_hwpoison_entry(subpage));
 			if (folio_test_hugetlb(folio)) {
 				hugetlb_count_sub(1UL << pvmw.pte_order, mm);
@@ -1632,7 +1641,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 			mmu_notifier_invalidate_range(mm, address,
 						      address + PAGE_SIZE);
 		} else if (folio_test_anon(folio)) {
-			swp_entry_t entry = { .val = page_private(subpage) };
+			swp_entry_t entry = {
+				.val = page_private(page_flags_page)
+			};
 			pte_t swp_pte;
 			/*
 			 * Store the swap location in the pte.
@@ -1831,7 +1842,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 	struct mm_struct *mm = vma->vm_mm;
 	DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0);
 	pte_t pteval;
-	struct page *subpage;
+	struct page *subpage, *page_flags_page;
 	bool anon_exclusive, ret = true;
 	struct mmu_notifier_range range;
 	enum ttu_flags flags = (enum ttu_flags)(long)arg;
@@ -1911,9 +1922,16 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 			subpage = folio_page(folio,
 					pte_pfn(*pvmw.pte) - folio_pfn(folio));
 		}
+		/*
+		 * We check the page flags of HugeTLB pages by checking the
+		 * head page.
+		 */
+		page_flags_page = folio_test_hugetlb(folio)
+			? &folio->page
+			: subpage;
 		address = pvmw.address;
 		anon_exclusive = folio_test_anon(folio) &&
-				 PageAnonExclusive(subpage);
+				 PageAnonExclusive(page_flags_page);
 
 		if (folio_test_hugetlb(folio)) {
 			bool anon = folio_test_anon(folio);
@@ -2032,7 +2050,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 			 * No need to invalidate here it will synchronize on
 			 * against the special swap migration pte.
 			 */
-		} else if (PageHWPoison(subpage)) {
+		} else if (PageHWPoison(page_flags_page)) {
 			pteval = swp_entry_to_pte(make_hwpoison_entry(subpage));
 			if (folio_test_hugetlb(folio)) {
 				hugetlb_count_sub(1L << pvmw.pte_order, mm);
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#97 lore raw

PATCH 30/46 hugetlb: add high-granularity migration support

In reply to James Houghton

To prevent queueing a hugepage for migration multiple times, we use last_page to keep track of the last page we saw in queue_pages_hugetlb, and if the page we're looking at is last_page, then we skip it.

For the non-hugetlb cases, last_page, although unused, is still updated so that it has a consistent meaning with the hugetlb case.

This commit adds a check in hugetlb_fault for high-granularity migration PTEs.

---
 include/linux/swapops.h |  8 ++++++--
 mm/hugetlb.c            |  2 +-
 mm/mempolicy.c          | 24 +++++++++++++++++++-----
 mm/migrate.c            | 18 ++++++++++--------
 4 files changed, 36 insertions(+), 16 deletions(-)
include/linux/swapops.h +6 -2
index 3a451b7afcb3..6ef80763e629 100644
@@ -68,6 +68,8 @@
 
 static inline bool is_pfn_swap_entry(swp_entry_t entry);
 
+struct hugetlb_pte;
+
 /* Clear all flags but only keep swp_entry_t related information */
 static inline pte_t pte_swp_clear_flags(pte_t pte)
 {
@@ -339,7 +341,8 @@ extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd,
 #ifdef CONFIG_HUGETLB_PAGE
 extern void __migration_entry_wait_huge(struct vm_area_struct *vma,
 					pte_t *ptep, spinlock_t *ptl);
-extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte);
+extern void migration_entry_wait_huge(struct vm_area_struct *vma,
+					struct hugetlb_pte *hpte);
 #endif	/* CONFIG_HUGETLB_PAGE */
 #else  /* CONFIG_MIGRATION */
 static inline swp_entry_t make_readable_migration_entry(pgoff_t offset)
@@ -369,7 +372,8 @@ static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd,
 #ifdef CONFIG_HUGETLB_PAGE
 static inline void __migration_entry_wait_huge(struct vm_area_struct *vma,
 					       pte_t *ptep, spinlock_t *ptl) { }
-static inline void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { }
+static inline void migration_entry_wait_huge(struct vm_area_struct *vma,
+						struct hugetlb_pte *hpte) { }
 #endif	/* CONFIG_HUGETLB_PAGE */
 static inline int is_writable_migration_entry(swp_entry_t entry)
 {
mm/hugetlb.c +1 -1
index 8e690a22456a..2fb95ecafc63 100644
@@ -6269,7 +6269,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 			 * be released there.
 			 */
 			mutex_unlock(&hugetlb_fault_mutex_table[hash]);
-			migration_entry_wait_huge(vma, hpte.ptep);
+			migration_entry_wait_huge(vma, &hpte);
 			return 0;
 		} else if (unlikely(is_hugetlb_entry_hwpoisoned(entry)))
 			ret = VM_FAULT_HWPOISON_LARGE |
mm/mempolicy.c +19 -5
index e5859ed34e90..6c4c3c923fa2 100644
@@ -424,6 +424,7 @@ struct queue_pages {
 	unsigned long start;
 	unsigned long end;
 	struct vm_area_struct *first;
+	struct page *last_page;
 };
 
 /*
@@ -475,6 +476,7 @@ static int queue_pages_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr,
 	flags = qp->flags;
 	/* go to thp migration */
 	if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
+		qp->last_page = page;
 		if (!vma_migratable(walk->vma) ||
 		    migrate_page_add(page, qp->pagelist, flags)) {
 			ret = 1;
@@ -532,6 +534,7 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,
 			continue;
 		if (!queue_pages_required(page, qp))
 			continue;
+
 		if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
 			/* MPOL_MF_STRICT must be specified if we get here */
 			if (!vma_migratable(vma)) {
@@ -539,6 +542,8 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,
 				break;
 			}
 
+			qp->last_page = page;
+
 			/*
 			 * Do not abort immediately since there may be
 			 * temporary off LRU pages in the range.  Still
@@ -570,15 +575,22 @@ static int queue_pages_hugetlb(struct hugetlb_pte *hpte,
 	spinlock_t *ptl;
 	pte_t entry;
 
-	/* We don't migrate high-granularity HugeTLB mappings for now. */
-	if (hugetlb_hgm_enabled(walk->vma))
-		return -EINVAL;
-
 	ptl = hugetlb_pte_lock(hpte);
 	entry = huge_ptep_get(hpte->ptep);
 	if (!pte_present(entry))
 		goto unlock;
-	page = pte_page(entry);
+
+	if (!hugetlb_pte_present_leaf(hpte, entry)) {
+		ret = -EAGAIN;
+		goto unlock;
+	}
+
+	page = compound_head(pte_page(entry));
+
+	/* We already queued this page with another high-granularity PTE. */
+	if (page == qp->last_page)
+		goto unlock;
+
 	if (!queue_pages_required(page, qp))
 		goto unlock;
 
@@ -605,6 +617,7 @@ static int queue_pages_hugetlb(struct hugetlb_pte *hpte,
 	/* With MPOL_MF_MOVE, we migrate only unshared hugepage. */
 	if (flags & (MPOL_MF_MOVE_ALL) ||
 	    (flags & MPOL_MF_MOVE && page_mapcount(page) == 1)) {
+		qp->last_page = page;
 		if (isolate_hugetlb(page, qp->pagelist) &&
 			(flags & MPOL_MF_STRICT))
 			/*
@@ -739,6 +752,7 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end,
 		.start = start,
 		.end = end,
 		.first = NULL,
+		.last_page = NULL,
 	};
 
 	err = walk_page_range(mm, start, end, &queue_pages_walk_ops, &qp);
mm/migrate.c +10 -8
index 0062689f4878..c30647b75459 100644
@@ -195,6 +195,9 @@ static bool remove_migration_pte(struct folio *folio,
 		/* pgoff is invalid for ksm pages, but they are never large */
 		if (folio_test_large(folio) && !folio_test_hugetlb(folio))
 			idx = linear_page_index(vma, pvmw.address) - pvmw.pgoff;
+		else if (folio_test_hugetlb(folio))
+			idx = (pvmw.address & ~huge_page_mask(hstate_vma(vma)))/
+				PAGE_SIZE;
 		new = folio_page(folio, idx);
 
 #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
@@ -244,14 +247,15 @@ static bool remove_migration_pte(struct folio *folio,
 
 #ifdef CONFIG_HUGETLB_PAGE
 		if (folio_test_hugetlb(folio)) {
+			struct page *hpage = folio_page(folio, 0);
 			unsigned int shift = pvmw.pte_order + PAGE_SHIFT;
 
 			pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
 			if (folio_test_anon(folio))
-				hugepage_add_anon_rmap(new, vma, pvmw.address,
+				hugepage_add_anon_rmap(hpage, vma, pvmw.address,
 						       rmap_flags);
 			else
-				page_dup_file_rmap(new, true);
+				page_dup_file_rmap(hpage, true);
 			set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte);
 		} else
 #endif
@@ -267,7 +271,7 @@ static bool remove_migration_pte(struct folio *folio,
 			mlock_page_drain_local();
 
 		trace_remove_migration_pte(pvmw.address, pte_val(pte),
-					   compound_order(new));
+					   pvmw.pte_order);
 
 		/* No need to invalidate - it was non-present before */
 		update_mmu_cache(vma, pvmw.address, pvmw.pte);
@@ -358,12 +362,10 @@ void __migration_entry_wait_huge(struct vm_area_struct *vma,
 	}
 }
 
-void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte)
+void migration_entry_wait_huge(struct vm_area_struct *vma,
+				struct hugetlb_pte *hpte)
 {
-	spinlock_t *ptl = huge_pte_lockptr(huge_page_shift(hstate_vma(vma)),
-					   vma->vm_mm, pte);
-
-	__migration_entry_wait_huge(vma, pte, ptl);
+	__migration_entry_wait_huge(vma, hpte->ptep, hpte->ptl);
 }
 #endif
 
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#98 lore raw

PATCH 31/46 hugetlb: sort hstates in hugetlb_init_hstates

In reply to James Houghton

When using HugeTLB high-granularity mapping, we need to go through the supported hugepage sizes in decreasing order so that we pick the largest size that works. Consider the case where we're faulting in a 1G hugepage for the first time: we want hugetlb_fault/hugetlb_no_page to map it with a PUD. By going through the sizes in decreasing order, we will find that PUD_SIZE works before finding out that PMD_SIZE or PAGE_SIZE work too.

This commit also changes bootmem hugepages from storing hstate pointers directly to storing the hstate sizes. The hstate pointers used for boot-time-allocated hugepages become invalid after we sort the hstates. gather_bootmem_prealloc, called after the hstates have been sorted, now converts the size to the correct hstate.

---
 include/linux/hugetlb.h |  2 +-
 mm/hugetlb.c            | 49 ++++++++++++++++++++++++++++++++---------
 2 files changed, 40 insertions(+), 11 deletions(-)
include/linux/hugetlb.h +1 -1
index daf993fdbc38..8a664a9dd0a8 100644
@@ -789,7 +789,7 @@ struct hstate {
 
 struct huge_bootmem_page {
 	struct list_head list;
-	struct hstate *hstate;
+	unsigned long hstate_sz;
 };
 
 int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list);
mm/hugetlb.c +39 -10

87 lines not on this page · raw message

Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#99 lore raw

PATCH 32/46 hugetlb: add for_each_hgm_shift

In reply to James Houghton

This is a helper macro to loop through all the usable page sizes for a high-granularity-enabled HugeTLB VMA. Given the VMA's hstate, it will loop, in descending order, through the page sizes that HugeTLB supports for this architecture. It always includes PAGE_SIZE.

This is done by looping through the hstates; however, there is no hstate for PAGE_SIZE. To handle this case, the loop intentionally goes out of bounds, and the out-of-bounds pointer is mapped to PAGE_SIZE.

---
 mm/hugetlb.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
mm/hugetlb.c +18 -0
index 1e9e149587b3..1eef6968b1fa 100644
@@ -7780,6 +7780,24 @@ bool hugetlb_hgm_enabled(struct vm_area_struct *vma)
 {
 	return vma && (vma->vm_flags & VM_HUGETLB_HGM);
 }
+/* Should only be used by the for_each_hgm_shift macro. */
+static unsigned int __shift_for_hstate(struct hstate *h)
+{
+	/* If h is out of bounds, we have reached the end, so give PAGE_SIZE */
+	if (h >= &hstates[hugetlb_max_hstate])
+		return PAGE_SHIFT;
+	return huge_page_shift(h);
+}
+
+/*
+ * Intentionally go out of bounds. An out-of-bounds hstate will be converted to
+ * PAGE_SIZE.
+ */
+#define for_each_hgm_shift(hstate, tmp_h, shift) \
+	for ((tmp_h) = hstate; (shift) = __shift_for_hstate(tmp_h), \
+			       (tmp_h) <= &hstates[hugetlb_max_hstate]; \
+			       (tmp_h)++)
+
 #endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */
 
 /*
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#100 lore raw

PATCH 33/46 hugetlb: userfaultfd: add support for high-granularity UFFDIO_CONTINUE

In reply to James Houghton

Changes here are similar to the changes made for hugetlb_no_page.

Pass vmf->real_address to userfaultfd_huge_must_wait because vmf->address may be rounded down to the hugepage size, and a high-granularity page table walk would look up the wrong PTE. Also change the call to userfaultfd_must_wait in the same way for consistency.

This commit introduces hugetlb_alloc_largest_pte which is used to find the appropriate PTE size to map pages with UFFDIO_CONTINUE.

When MADV_SPLIT is provided, page fault events will report PAGE_SIZE-aligned address instead of huge_page_size(h)-aligned addresses, regardless of if UFFD_FEATURE_EXACT_ADDRESS is used.

---
 fs/userfaultfd.c        | 14 +++----
 include/linux/hugetlb.h | 18 ++++++++-
 mm/hugetlb.c            | 85 +++++++++++++++++++++++++++++++++--------
 mm/userfaultfd.c        | 40 +++++++++++--------
 4 files changed, 119 insertions(+), 38 deletions(-)
fs/userfaultfd.c +7 -7
index 15a5bf765d43..940ff63096a9 100644
@@ -252,17 +252,17 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
 					 unsigned long flags,
 					 unsigned long reason)
 {
-	pte_t *ptep, pte;
+	pte_t pte;
 	bool ret = true;
+	struct hugetlb_pte hpte;
 
 	mmap_assert_locked(ctx->mm);
 
-	ptep = hugetlb_walk(vma, address, vma_mmu_pagesize(vma));
-	if (!ptep)
+	if (hugetlb_full_walk(&hpte, vma, address))
 		goto out;
 
 	ret = false;
-	pte = huge_ptep_get(ptep);
+	pte = huge_ptep_get(hpte.ptep);
 
 	/*
 	 * Lockless access: we're in a wait_event so it's ok if it
@@ -531,11 +531,11 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
 	spin_unlock_irq(&ctx->fault_pending_wqh.lock);
 
 	if (!is_vm_hugetlb_page(vma))
-		must_wait = userfaultfd_must_wait(ctx, vmf->address, vmf->flags,
-						  reason);
+		must_wait = userfaultfd_must_wait(ctx, vmf->real_address,
+						  vmf->flags, reason);
 	else
 		must_wait = userfaultfd_huge_must_wait(ctx, vma,
-						       vmf->address,
+						       vmf->real_address,
 						       vmf->flags, reason);
 	if (is_vm_hugetlb_page(vma))
 		hugetlb_vma_unlock_read(vma);
include/linux/hugetlb.h +17 -1
index 8a664a9dd0a8..c8524ac49b24 100644
@@ -224,7 +224,8 @@ unsigned long hugetlb_total_pages(void);
 vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 			unsigned long address, unsigned int flags);
 #ifdef CONFIG_USERFAULTFD
-int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte,
+int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,
+				struct hugetlb_pte *dst_hpte,
 				struct vm_area_struct *dst_vma,
 				unsigned long dst_addr,
 				unsigned long src_addr,
@@ -1292,16 +1293,31 @@ static inline enum hugetlb_level hpage_size_to_level(unsigned long sz)
 
 #ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING
 bool hugetlb_hgm_enabled(struct vm_area_struct *vma);
+bool hugetlb_hgm_advised(struct vm_area_struct *vma);
 bool hugetlb_hgm_eligible(struct vm_area_struct *vma);
+int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm,
+			      struct vm_area_struct *vma, unsigned long start,
+			      unsigned long end);
 #else
 static inline bool hugetlb_hgm_enabled(struct vm_area_struct *vma)
 {
 	return false;
 }
+static inline bool hugetlb_hgm_advised(struct vm_area_struct *vma)
+{
+	return false;
+}
 static inline bool hugetlb_hgm_eligible(struct vm_area_struct *vma)
 {
 	return false;
 }
+static inline
+int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm,
+			      struct vm_area_struct *vma, unsigned long start,
+			      unsigned long end)
+{
+	return -EINVAL;
+}
 #endif
 
 static inline spinlock_t *huge_pte_lock(struct hstate *h,
mm/hugetlb.c +70 -15

180 lines not on this page · raw message

mm/userfaultfd.c +25 -15
index 65ad172add27..2b233d31be24 100644
@@ -320,14 +320,16 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
 {
 	int vm_shared = dst_vma->vm_flags & VM_SHARED;
 	ssize_t err;
-	pte_t *dst_pte;
 	unsigned long src_addr, dst_addr;
 	long copied;
 	struct page *page;
-	unsigned long vma_hpagesize;
+	unsigned long vma_hpagesize, target_pagesize;
 	pgoff_t idx;
 	u32 hash;
 	struct address_space *mapping;
+	bool use_hgm = hugetlb_hgm_advised(dst_vma) &&
+		mode == MCOPY_ATOMIC_CONTINUE;
+	struct hstate *h = hstate_vma(dst_vma);
 
 	/*
 	 * There is no default zero huge page for all huge page sizes as
@@ -345,12 +347,13 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
 	copied = 0;
 	page = NULL;
 	vma_hpagesize = vma_kernel_pagesize(dst_vma);
+	target_pagesize = use_hgm ? PAGE_SIZE : vma_hpagesize;
 
 	/*
-	 * Validate alignment based on huge page size
+	 * Validate alignment based on the targeted page size.
 	 */
 	err = -EINVAL;
-	if (dst_start & (vma_hpagesize - 1) || len & (vma_hpagesize - 1))
+	if (dst_start & (target_pagesize - 1) || len & (target_pagesize - 1))
 		goto out_unlock;
 
 retry:
@@ -381,13 +384,14 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
 	}
 
 	while (src_addr < src_start + len) {
+		struct hugetlb_pte hpte;
 		BUG_ON(dst_addr >= dst_start + len);
 
 		/*
 		 * Serialize via vma_lock and hugetlb_fault_mutex.
-		 * vma_lock ensures the dst_pte remains valid even
-		 * in the case of shared pmds.  fault mutex prevents
-		 * races with other faulting threads.
+		 * vma_lock ensures the hpte.ptep remains valid even
+		 * in the case of shared pmds and page table collapsing.
+		 * fault mutex prevents races with other faulting threads.
 		 */
 		idx = linear_page_index(dst_vma, dst_addr);
 		mapping = dst_vma->vm_file->f_mapping;
@@ -395,23 +399,28 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
 		mutex_lock(&hugetlb_fault_mutex_table[hash]);
 		hugetlb_vma_lock_read(dst_vma);
 
-		err = -ENOMEM;
-		dst_pte = huge_pte_alloc(dst_mm, dst_vma, dst_addr, vma_hpagesize);
-		if (!dst_pte) {
+		if (use_hgm)
+			err = hugetlb_alloc_largest_pte(&hpte, dst_mm, dst_vma,
+							dst_addr,
+							dst_start + len);
+		else
+			err = hugetlb_full_walk_alloc(&hpte, dst_vma, dst_addr,
+						      vma_hpagesize);
+		if (err) {
 			hugetlb_vma_unlock_read(dst_vma);
 			mutex_unlock(&hugetlb_fault_mutex_table[hash]);
 			goto out_unlock;
 		}
 
 		if (mode != MCOPY_ATOMIC_CONTINUE &&
-		    !huge_pte_none_mostly(huge_ptep_get(dst_pte))) {
+		    !huge_pte_none_mostly(huge_ptep_get(hpte.ptep))) {
 			err = -EEXIST;
 			hugetlb_vma_unlock_read(dst_vma);
 			mutex_unlock(&hugetlb_fault_mutex_table[hash]);
 			goto out_unlock;
 		}
 
-		err = hugetlb_mcopy_atomic_pte(dst_mm, dst_pte, dst_vma,
+		err = hugetlb_mcopy_atomic_pte(dst_mm, &hpte, dst_vma,
 					       dst_addr, src_addr, mode, &page,
 					       wp_copy);
 
@@ -423,6 +432,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
 		if (unlikely(err == -ENOENT)) {
 			mmap_read_unlock(dst_mm);
 			BUG_ON(!page);
+			WARN_ON_ONCE(hpte.shift != huge_page_shift(h));
 
 			err = copy_huge_page_from_user(page,
 						(const void __user *)src_addr,
@@ -440,9 +450,9 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
 			BUG_ON(page);
 
 		if (!err) {
-			dst_addr += vma_hpagesize;
-			src_addr += vma_hpagesize;
-			copied += vma_hpagesize;
+			dst_addr += hugetlb_pte_size(&hpte);
+			src_addr += hugetlb_pte_size(&hpte);
+			copied += hugetlb_pte_size(&hpte);
 
 			if (fatal_signal_pending(current))
 				err = -EINTR;
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#101 lore raw

PATCH 34/46 hugetlb: userfaultfd: when using MADV_SPLIT, round addresses to PAGE_SIZE

In reply to James Houghton

MADV_SPLIT enables HugeTLB HGM which allows for UFFDIO_CONTINUE in PAGE_SIZE chunks. If a huge-page-aligned address were to be provided, userspace would be completely unable to take advantage of HGM. That would then require userspace to know to provide UFFD_FEATURE_EXACT_ADDRESS.

This patch would make it harder to make a mistake. Instead of requiring userspace to provide UFFD_FEATURE_EXACT_ADDRESS, always provide a usable address.

---
 mm/hugetlb.c | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
mm/hugetlb.c +15 -16
index 5af6db52f34e..5b6215e03fe1 100644
@@ -5936,28 +5936,27 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma,
 						  unsigned long addr,
 						  unsigned long reason)
 {
+	u32 hash;
+	struct vm_fault vmf;
+
 	/*
 	 * Don't use the hpage-aligned address if the user has explicitly
 	 * enabled HGM.
 	 */
 	if (hugetlb_hgm_advised(vma) && reason == VM_UFFD_MINOR)
-		haddr = address & PAGE_MASK;
-
-	u32 hash;
-	struct vm_fault vmf = {
-		.vma = vma,
-		.address = haddr,
-		.real_address = addr,
-		.flags = flags,
+		haddr = addr & PAGE_MASK;
 
-		/*
-		 * Hard to debug if it ends up being
-		 * used by a callee that assumes
-		 * something about the other
-		 * uninitialized fields... same as in
-		 * memory.c
-		 */
-	};
+	vmf.vma = vma;
+	vmf.address = haddr;
+	vmf.real_address = addr;
+	vmf.flags = flags;
+	/*
+	 * Hard to debug if it ends up being
+	 * used by a callee that assumes
+	 * something about the other
+	 * uninitialized fields... same as in
+	 * memory.c
+	 */
 
 	/*
 	 * vma_lock and hugetlb_fault_mutex must be dropped before handling
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
reviewer
#102 lore raw

Re: [PATCH 34/46] hugetlb: userfaultfd: when using MADV_SPLIT, round addresses to PAGE_SIZE

Quoted 43 lines · James Houghton

MADV_SPLIT enables HugeTLB HGM which allows for UFFDIO_CONTINUE in PAGE_SIZE chunks. If a huge-page-aligned address were to be provided, userspace would be completely unable to take advantage of HGM. That would then require userspace to know to provide UFFD_FEATURE_EXACT_ADDRESS.

This patch would make it harder to make a mistake. Instead of requiring userspace to provide UFFD_FEATURE_EXACT_ADDRESS, always provide a usable address.

---
 mm/hugetlb.c | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
mm/hugetlb.c +4 -10
index 5af6db52f34e..5b6215e03fe1 100644
@@ -5936,28 +5936,27 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma,
 						  unsigned long addr,
 						  unsigned long reason)
 {
+	u32 hash;
+	struct vm_fault vmf;
+
 	/*
 	 * Don't use the hpage-aligned address if the user has explicitly
 	 * enabled HGM.
 	 */
 	if (hugetlb_hgm_advised(vma) && reason == VM_UFFD_MINOR)
-		haddr = address & PAGE_MASK;
-
-	u32 hash;
-	struct vm_fault vmf = {
-		.vma = vma,
-		.address = haddr,
-		.real_address = addr,
-		.flags = flags,
+		haddr = addr & PAGE_MASK;
 
-		/*
-		 * Hard to debug if it ends up being

mm/hugetlb.c static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma, · line 5952

Quoted 10 lines
mm/hugetlb.c +4 -6
-		 * used by a callee that assumes
-		 * something about the other
-		 * uninitialized fields... same as in
-		 * memory.c
-		 */
-	};
+	vmf.vma = vma;
+	vmf.address = haddr;
+	vmf.real_address = addr;
+	vmf.flags = flags;

Const fields here:

mm/hugetlb.c: In function ‘hugetlb_handle_userfault’:
mm/hugetlb.c:5961:17: error: assignment of member ‘vma’ in read-only object
5961 | vmf.vma = vma;
| ^
mm/hugetlb.c:5962:21: error: assignment of member ‘address’ in read-only object
5962 | vmf.address = haddr;
| ^
mm/hugetlb.c:5963:26: error: assignment of member ‘real_address’ in read-only object
5963 | vmf.real_address = addr;

mm/hugetlb.c static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma, · line 5959

Quoted 7 lines
mm/hugetlb.c +7 -0
+	/*
+	 * Hard to debug if it ends up being
+	 * used by a callee that assumes
+	 * something about the other
+	 * uninitialized fields... same as in
+	 * memory.c
+	 */

PS: I think we can drop this along the way.

Quoted 6 lines
mm/hugetlb.c +0 -0
 	/*
 	 * vma_lock and hugetlb_fault_mutex must be dropped before handling
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
Signature · 1 line
Peter Xu
#103 lore raw

Re: [PATCH 34/46] hugetlb: userfaultfd: when using MADV_SPLIT, round addresses to PAGE_SIZE

Quoted 58 lines · Peter Xu
Quoted 43 lines · James Houghton

MADV_SPLIT enables HugeTLB HGM which allows for UFFDIO_CONTINUE in PAGE_SIZE chunks. If a huge-page-aligned address were to be provided, userspace would be completely unable to take advantage of HGM. That would then require userspace to know to provide UFFD_FEATURE_EXACT_ADDRESS.

This patch would make it harder to make a mistake. Instead of requiring userspace to provide UFFD_FEATURE_EXACT_ADDRESS, always provide a usable address.

---
 mm/hugetlb.c | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
mm/hugetlb.c +4 -10
index 5af6db52f34e..5b6215e03fe1 100644
@@ -5936,28 +5936,27 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma,
                                                unsigned long addr,
                                                unsigned long reason)
 {
+     u32 hash;
+     struct vm_fault vmf;
+
      /*
       * Don't use the hpage-aligned address if the user has explicitly
       * enabled HGM.
       */
      if (hugetlb_hgm_advised(vma) && reason == VM_UFFD_MINOR)
-             haddr = address & PAGE_MASK;
-
-     u32 hash;
-     struct vm_fault vmf = {
-             .vma = vma,
-             .address = haddr,
-             .real_address = addr,
-             .flags = flags,
+             haddr = addr & PAGE_MASK;
 
-             /*
-              * Hard to debug if it ends up being

mm/hugetlb.c static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma, · line 5952

Quoted 10 lines
mm/hugetlb.c +4 -6
-              * used by a callee that assumes
-              * something about the other
-              * uninitialized fields... same as in
-              * memory.c
-              */
-     };
+     vmf.vma = vma;
+     vmf.address = haddr;
+     vmf.real_address = addr;
+     vmf.flags = flags;

Const fields here:

Quoted 9 lines

mm/hugetlb.c: In function ‘hugetlb_handle_userfault’:
mm/hugetlb.c:5961:17: error: assignment of member ‘vma’ in read-only object
5961 | vmf.vma = vma;
| ^
mm/hugetlb.c:5962:21: error: assignment of member ‘address’ in read-only object
5962 | vmf.address = haddr;
| ^
mm/hugetlb.c:5963:26: error: assignment of member ‘real_address’ in read-only object
5963 | vmf.real_address = addr;

Thanks Peter for this and your other findings. Not sure why my compiler (clang) let me do this. :/ Will send a v2 soon with this + the other problems fixed.

#104 lore raw

PATCH 35/46 hugetlb: add MADV_COLLAPSE for hugetlb

In reply to James Houghton

This is a necessary extension to the UFFDIO_CONTINUE changes. When userspace finishes mapping an entire hugepage with UFFDIO_CONTINUE, the kernel has no mechanism to automatically collapse the page table to map the whole hugepage normally. We require userspace to inform us that they would like the mapping to be collapsed; they do this with MADV_COLLAPSE.

If userspace has not mapped all of a hugepage with UFFDIO_CONTINUE, but only some, hugetlb_collapse will cause the requested range to be mapped as if it were UFFDIO_CONTINUE'd already. The effects of any UFFDIO_WRITEPROTECT calls may be undone by a call to MADV_COLLAPSE for intersecting address ranges.

This commit is co-opting the same madvise mode that has been introduced to synchronously collapse THPs. The function that does THP collapsing has been renamed to madvise_collapse_thp.

As with the rest of the high-granularity mapping support, MADV_COLLAPSE is only supported for shared VMAs right now.

MADV_COLLAPSE has the same synchronization as huge_pmd_unshare.

---
 include/linux/huge_mm.h |  12 +--
 include/linux/hugetlb.h |   8 ++
 mm/hugetlb.c            | 164 ++++++++++++++++++++++++++++++++++++++++
 mm/khugepaged.c         |   4 +-
 mm/madvise.c            |  18 ++++-
 5 files changed, 197 insertions(+), 9 deletions(-)
include/linux/huge_mm.h +6 -6
index a1341fdcf666..5d1e3c980f74 100644
@@ -218,9 +218,9 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud,
 
 int hugepage_madvise(struct vm_area_struct *vma, unsigned long *vm_flags,
 		     int advice);
-int madvise_collapse(struct vm_area_struct *vma,
-		     struct vm_area_struct **prev,
-		     unsigned long start, unsigned long end);
+int madvise_collapse_thp(struct vm_area_struct *vma,
+			 struct vm_area_struct **prev,
+			 unsigned long start, unsigned long end);
 void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start,
 			   unsigned long end, long adjust_next);
 spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma);
@@ -367,9 +367,9 @@ static inline int hugepage_madvise(struct vm_area_struct *vma,
 	return -EINVAL;
 }
 
-static inline int madvise_collapse(struct vm_area_struct *vma,
-				   struct vm_area_struct **prev,
-				   unsigned long start, unsigned long end)
+static inline int madvise_collapse_thp(struct vm_area_struct *vma,
+				       struct vm_area_struct **prev,
+				       unsigned long start, unsigned long end)
 {
 	return -EINVAL;
 }
include/linux/hugetlb.h +8 -0
index c8524ac49b24..e1baf939afb6 100644
@@ -1298,6 +1298,8 @@ bool hugetlb_hgm_eligible(struct vm_area_struct *vma);
 int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm,
 			      struct vm_area_struct *vma, unsigned long start,
 			      unsigned long end);
+int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma,
+		     unsigned long start, unsigned long end);
 #else
 static inline bool hugetlb_hgm_enabled(struct vm_area_struct *vma)
 {
@@ -1318,6 +1320,12 @@ int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm,
 {
 	return -EINVAL;
 }
+static inline
+int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma,
+		     unsigned long start, unsigned long end)
+{
+	return -EINVAL;
+}
 #endif
 
 static inline spinlock_t *huge_pte_lock(struct hstate *h,
mm/hugetlb.c +164 -0

170 lines not on this page · raw message

mm/khugepaged.c +2 -2
index e1c7c1f357ef..cbeb7f00f1bf 100644
@@ -2718,8 +2718,8 @@ static int madvise_collapse_errno(enum scan_result r)
 	}
 }
 
-int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev,
-		     unsigned long start, unsigned long end)
+int madvise_collapse_thp(struct vm_area_struct *vma, struct vm_area_struct **prev,
+			 unsigned long start, unsigned long end)
 {
 	struct collapse_control *cc;
 	struct mm_struct *mm = vma->vm_mm;
mm/madvise.c +17 -1
index 04ee28992e52..fec47e9f845b 100644
@@ -1029,6 +1029,18 @@ static int madvise_split(struct vm_area_struct *vma,
 	return 0;
 }
 
+static int madvise_collapse(struct vm_area_struct *vma,
+			    struct vm_area_struct **prev,
+			    unsigned long start, unsigned long end)
+{
+	if (is_vm_hugetlb_page(vma)) {
+		*prev = vma;
+		return hugetlb_collapse(vma->vm_mm, vma, start, end);
+	}
+
+	return madvise_collapse_thp(vma, prev, start, end);
+}
+
 /*
  * Apply an madvise behavior to a region of a vma.  madvise_update_vma
  * will handle splitting a vm area into separate areas, each area with its own
@@ -1205,6 +1217,9 @@ madvise_behavior_valid(int behavior)
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	case MADV_HUGEPAGE:
 	case MADV_NOHUGEPAGE:
+#endif
+#if defined(CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING) || \
+		defined(CONFIG_TRANSPARENT_HUGEPAGE)
 	case MADV_COLLAPSE:
 #endif
 #ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING
@@ -1398,7 +1413,8 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start,
  *  MADV_NOHUGEPAGE - mark the given range as not worth being backed by
  *		transparent huge pages so the existing pages will not be
  *		coalesced into THP and new pages will not be allocated as THP.
- *  MADV_COLLAPSE - synchronously coalesce pages into new THP.
+ *  MADV_COLLAPSE - synchronously coalesce pages into new THP, or, for HugeTLB
+ *		pages, collapse the mapping.
  *  MADV_DONTDUMP - the application wants to prevent pages in the given range
  *		from being included in its core dump.
  *  MADV_DODUMP - cancel MADV_DONTDUMP: no longer exclude from core dump.
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#105 lore raw

Re: [PATCH 35/46] hugetlb: add MADV_COLLAPSE for hugetlb

mm/hugetlb.c int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, · line 8008

Quoted 6 lines
mm/hugetlb.c +6 -0
+
+               if (curr < end) {
+                       /* Don't hold the VMA lock for too long. */
+                       hugetlb_vma_unlock_write(vma);
+                       cond_resched();
+                       hugetlb_vma_lock_write(vma);

I need to drop/reacquire the mapping lock here too (missed this when I added the bits to grab the mapping lock in this function).

reviewer
#106 lore raw

Re: [PATCH 35/46] hugetlb: add MADV_COLLAPSE for hugetlb

In reply to James Houghton
Quoted 238 lines · James Houghton

This is a necessary extension to the UFFDIO_CONTINUE changes. When userspace finishes mapping an entire hugepage with UFFDIO_CONTINUE, the kernel has no mechanism to automatically collapse the page table to map the whole hugepage normally. We require userspace to inform us that they would like the mapping to be collapsed; they do this with MADV_COLLAPSE.

If userspace has not mapped all of a hugepage with UFFDIO_CONTINUE, but only some, hugetlb_collapse will cause the requested range to be mapped as if it were UFFDIO_CONTINUE'd already. The effects of any UFFDIO_WRITEPROTECT calls may be undone by a call to MADV_COLLAPSE for intersecting address ranges.

This commit is co-opting the same madvise mode that has been introduced to synchronously collapse THPs. The function that does THP collapsing has been renamed to madvise_collapse_thp.

As with the rest of the high-granularity mapping support, MADV_COLLAPSE is only supported for shared VMAs right now.

MADV_COLLAPSE has the same synchronization as huge_pmd_unshare.

---
 include/linux/huge_mm.h |  12 +--
 include/linux/hugetlb.h |   8 ++
 mm/hugetlb.c            | 164 ++++++++++++++++++++++++++++++++++++++++
 mm/khugepaged.c         |   4 +-
 mm/madvise.c            |  18 ++++-
 5 files changed, 197 insertions(+), 9 deletions(-)
include/linux/huge_mm.h +6 -6
index a1341fdcf666..5d1e3c980f74 100644
@@ -218,9 +218,9 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud,
 
 int hugepage_madvise(struct vm_area_struct *vma, unsigned long *vm_flags,
 		     int advice);
-int madvise_collapse(struct vm_area_struct *vma,
-		     struct vm_area_struct **prev,
-		     unsigned long start, unsigned long end);
+int madvise_collapse_thp(struct vm_area_struct *vma,
+			 struct vm_area_struct **prev,
+			 unsigned long start, unsigned long end);
 void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start,
 			   unsigned long end, long adjust_next);
 spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma);
@@ -367,9 +367,9 @@ static inline int hugepage_madvise(struct vm_area_struct *vma,
 	return -EINVAL;
 }
 
-static inline int madvise_collapse(struct vm_area_struct *vma,
-				   struct vm_area_struct **prev,
-				   unsigned long start, unsigned long end)
+static inline int madvise_collapse_thp(struct vm_area_struct *vma,
+				       struct vm_area_struct **prev,
+				       unsigned long start, unsigned long end)
 {
 	return -EINVAL;
 }
include/linux/hugetlb.h +8 -0
index c8524ac49b24..e1baf939afb6 100644
@@ -1298,6 +1298,8 @@ bool hugetlb_hgm_eligible(struct vm_area_struct *vma);
 int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm,
 			      struct vm_area_struct *vma, unsigned long start,
 			      unsigned long end);
+int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma,
+		     unsigned long start, unsigned long end);
 #else
 static inline bool hugetlb_hgm_enabled(struct vm_area_struct *vma)
 {
@@ -1318,6 +1320,12 @@ int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm,
 {
 	return -EINVAL;
 }
+static inline
+int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma,
+		     unsigned long start, unsigned long end)
+{
+	return -EINVAL;
+}
 #endif
 
 static inline spinlock_t *huge_pte_lock(struct hstate *h,
mm/hugetlb.c +144 -0
index 5b6215e03fe1..388c46c7e77a 100644
@@ -7852,6 +7852,170 @@ int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm,
 	return 0;
 }
 
+static bool hugetlb_hgm_collapsable(struct vm_area_struct *vma)
+{
+	if (!hugetlb_hgm_eligible(vma))
+		return false;
+	if (!vma->vm_private_data)	/* vma lock required for collapsing */
+		return false;
+	return true;
+}
+
+/*
+ * Collapse the address range from @start to @end to be mapped optimally.
+ *
+ * This is only valid for shared mappings. The main use case for this function
+ * is following UFFDIO_CONTINUE. If a user UFFDIO_CONTINUEs an entire hugepage
+ * by calling UFFDIO_CONTINUE once for each 4K region, the kernel doesn't know
+ * to collapse the mapping after the final UFFDIO_CONTINUE. Instead, we leave
+ * it up to userspace to tell us to do so, via MADV_COLLAPSE.
+ *
+ * Any holes in the mapping will be filled. If there is no page in the
+ * pagecache for a region we're collapsing, the PTEs will be cleared.
+ *
+ * If high-granularity PTEs are uffd-wp markers, those markers will be dropped.
+ */
+int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma,
+			    unsigned long start, unsigned long end)
+{
+	struct hstate *h = hstate_vma(vma);
+	struct address_space *mapping = vma->vm_file->f_mapping;
+	struct mmu_notifier_range range;
+	struct mmu_gather tlb;
+	unsigned long curr = start;
+	int ret = 0;
+	struct page *hpage, *subpage;
+	pgoff_t idx;
+	bool writable = vma->vm_flags & VM_WRITE;
+	bool shared = vma->vm_flags & VM_SHARED;
+	struct hugetlb_pte hpte;
+	pte_t entry;
+
+	/*
+	 * This is only supported for shared VMAs, because we need to look up
+	 * the page to use for any PTEs we end up creating.
+	 */
+	if (!shared)
+		return -EINVAL;
+
+	/* If HGM is not enabled, there is nothing to collapse. */
+	if (!hugetlb_hgm_enabled(vma))
+		return 0;
+
+	/*
+	 * We lost the VMA lock after splitting, so we can't safely collapse.
+	 * We could improve this in the future (like take the mmap_lock for
+	 * writing and try again), but for now just fail with ENOMEM.
+	 */
+	if (unlikely(!hugetlb_hgm_collapsable(vma)))
+		return -ENOMEM;
+
+	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm,
+				start, end);
+	mmu_notifier_invalidate_range_start(&range);
+	tlb_gather_mmu(&tlb, mm);
+
+	/*
+	 * Grab the VMA lock and mapping sem for writing. This will prevent
+	 * concurrent high-granularity page table walks, so that we can safely
+	 * collapse and free page tables.
+	 *
+	 * This is the same locking that huge_pmd_unshare requires.
+	 */
+	hugetlb_vma_lock_write(vma);
+	i_mmap_lock_write(vma->vm_file->f_mapping);
+
+	while (curr < end) {
+		ret = hugetlb_alloc_largest_pte(&hpte, mm, vma, curr, end);
+		if (ret)
+			goto out;
+
+		entry = huge_ptep_get(hpte.ptep);
+
+		/*
+		 * There is no work to do if the PTE doesn't point to page
+		 * tables.
+		 */
+		if (!pte_present(entry))
+			goto next_hpte;
+		if (hugetlb_pte_present_leaf(&hpte, entry))
+			goto next_hpte;
+
+		idx = vma_hugecache_offset(h, vma, curr);
+		hpage = find_get_page(mapping, idx);
+
+		if (hpage && !HPageMigratable(hpage)) {
+			/*
+			 * Don't collapse a mapping to a page that is pending
+			 * a migration. Migration swap entries may have placed
+			 * in the page table.
+			 */
+			ret = -EBUSY;
+			put_page(hpage);
+			goto out;
+		}
+
+		if (hpage && PageHWPoison(hpage)) {
+			/*
+			 * Don't collapse a mapping to a page that is
+			 * hwpoisoned.
+			 */
+			ret = -EHWPOISON;
+			put_page(hpage);
+			/*
+			 * By setting ret to -EHWPOISON, if nothing else
+			 * happens, we will tell userspace that we couldn't
+			 * fully collapse everything due to poison.
+			 *
+			 * Skip this page, and continue to collapse the rest
+			 * of the mapping.
+			 */
+			curr = (curr & huge_page_mask(h)) + huge_page_size(h);
+			continue;
+		}
+
+		/*
+		 * Clear all the PTEs, and drop ref/mapcounts
+		 * (on tlb_finish_mmu).
+		 */
+		__unmap_hugepage_range(&tlb, vma, curr,
+			curr + hugetlb_pte_size(&hpte),
+			NULL,
+			ZAP_FLAG_DROP_MARKER);
+		/* Free the PTEs. */
+		hugetlb_free_pgd_range(&tlb,
+				curr, curr + hugetlb_pte_size(&hpte),
+				curr, curr + hugetlb_pte_size(&hpte));
+		if (!hpage) {
+			huge_pte_clear(mm, curr, hpte.ptep,
+					hugetlb_pte_size(&hpte));
+			goto next_hpte;
+		}
+
+		page_dup_file_rmap(hpage, true);
+
+		subpage = hugetlb_find_subpage(h, hpage, curr);
+		entry = make_huge_pte_with_shift(vma, subpage,

mm/hugetlb.c int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, · line 8008

Quoted 10 lines
mm/hugetlb.c +10 -0
+						 writable, hpte.shift);
+		set_huge_pte_at(mm, curr, hpte.ptep, entry);
+next_hpte:
+		curr += hugetlb_pte_size(&hpte);
+
+		if (curr < end) {
+			/* Don't hold the VMA lock for too long. */
+			hugetlb_vma_unlock_write(vma);
+			cond_resched();
+			hugetlb_vma_lock_write(vma);

The intention is good here but IIUC this will cause vma lock to be taken after the i_mmap_rwsem, which can cause circular deadlocks. If to do this properly we'll need to also release the i_mmap_rwsem.

However it may make the resched() logic over complicated, meanwhile for 2M huge pages I think this will be called for each 2M range which can be too fine grained, so it looks like the "cur < end" check is a bit too aggresive.

The other thing is I noticed that the long period of mmu notifier invalidate between start -> end will (in reallife VM context) causing vcpu threads spinning.

I _think_ it's because is_page_fault_stale() (when during a vmexit following a kvm page fault) always reports true during the long procedure of MADV_COLLAPSE if to be called upon a large range, so even if we release both locks here it may not tremedously on the VM migration use case because of the long-standing mmu notifier invalidation procedure.

To summarize.. I think a simpler start version of hugetlb MADV_COLLAPSE can drop this "if" block, and let the userapp decide the step size of COLLAPSE?

Quoted 11 lines
mm/hugetlb.c +10 -0
+		}
+	}
+out:
+	i_mmap_unlock_write(vma->vm_file->f_mapping);
+	hugetlb_vma_unlock_write(vma);
+	tlb_finish_mmu(&tlb);
+	mmu_notifier_invalidate_range_end(&range);
+	return ret;
+}
+
 #endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */
Signature · 1 line
Peter Xu
#107 lore raw

Re: [PATCH 35/46] hugetlb: add MADV_COLLAPSE for hugetlb

Quoted 9 lines

mm/hugetlb.c int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, · line 8008

Quoted 5 lines
mm/hugetlb.c +5 -0
+             if (curr < end) {
+                     /* Don't hold the VMA lock for too long. */
+                     hugetlb_vma_unlock_write(vma);
+                     cond_resched();
+                     hugetlb_vma_lock_write(vma);

The intention is good here but IIUC this will cause vma lock to be taken after the i_mmap_rwsem, which can cause circular deadlocks. If to do this properly we'll need to also release the i_mmap_rwsem.

Sorry if you spent a long time debugging this! I sent a reply a week ago about this too.

Quoted 9 lines

However it may make the resched() logic over complicated, meanwhile for 2M huge pages I think this will be called for each 2M range which can be too fine grained, so it looks like the "cur < end" check is a bit too aggresive.

The other thing is I noticed that the long period of mmu notifier invalidate between start -> end will (in reallife VM context) causing vcpu threads spinning.

Quoted 5 lines

I _think_ it's because is_page_fault_stale() (when during a vmexit following a kvm page fault) always reports true during the long procedure of MADV_COLLAPSE if to be called upon a large range, so even if we release both locks here it may not tremedously on the VM migration use case because of the long-standing mmu notifier invalidation procedure.

Oh... indeed. Thanks for pointing that out.

Quoted 3 lines

To summarize.. I think a simpler start version of hugetlb MADV_COLLAPSE can drop this "if" block, and let the userapp decide the step size of COLLAPSE?

I'll drop this resched logic. Thanks Peter.

reviewer
#108 lore raw

Re: [PATCH 35/46] hugetlb: add MADV_COLLAPSE for hugetlb

        On Tue, Jan 17, 2023 at 01:38:24PM -0800, James Houghton wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
mm/hugetlb.c +5 -0
+             if (curr < end) {
+                     /* Don't hold the VMA lock for too long. */
+                     hugetlb_vma_unlock_write(vma);
+                     cond_resched();
+                     hugetlb_vma_lock_write(vma);
Quoted 6 lines
Quoted 3 lines

The intention is good here but IIUC this will cause vma lock to be taken after the i_mmap_rwsem, which can cause circular deadlocks. If to do this properly we'll need to also release the i_mmap_rwsem.

Sorry if you spent a long time debugging this! I sent a reply a week ago about this too.

Oops, yes, I somehow missed that one. No worry - it's reported by lockdep. :)

Quoted 16 lines
Quoted 10 lines

However it may make the resched() logic over complicated, meanwhile for 2M huge pages I think this will be called for each 2M range which can be too fine grained, so it looks like the "cur < end" check is a bit too aggresive.

The other thing is I noticed that the long period of mmu notifier invalidate between start -> end will (in reallife VM context) causing vcpu threads spinning.

Quoted 5 lines

I _think_ it's because is_page_fault_stale() (when during a vmexit following a kvm page fault) always reports true during the long procedure of MADV_COLLAPSE if to be called upon a large range, so even if we release both locks here it may not tremedously on the VM migration use case because of the long-standing mmu notifier invalidation procedure.

Quoted 7 lines

Oh... indeed. Thanks for pointing that out.

Quoted 3 lines

To summarize.. I think a simpler start version of hugetlb MADV_COLLAPSE can drop this "if" block, and let the userapp decide the step size of COLLAPSE?

I'll drop this resched logic. Thanks Peter.

Sounds good, thanks.

Signature · 1 line
Peter Xu
reviewer
#109 lore raw

Re: [PATCH 35/46] hugetlb: add MADV_COLLAPSE for hugetlb

In reply to James Houghton

mm/hugetlb.c int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, · line 7926

Quoted 9 lines · James Houghton
mm/hugetlb.c +9 -0
+	/*
+	 * Grab the VMA lock and mapping sem for writing. This will prevent
+	 * concurrent high-granularity page table walks, so that we can safely
+	 * collapse and free page tables.
+	 *
+	 * This is the same locking that huge_pmd_unshare requires.
+	 */
+	hugetlb_vma_lock_write(vma);
+	i_mmap_lock_write(vma->vm_file->f_mapping);

One thing I just noticed - do we need the mmap write lock here? I don't quickly see what stops another thread from having the mmap read and walking upon the pgtables being collapsed.

Thanks,

Signature · 1 line
Peter Xu
#110 lore raw

Re: [PATCH 35/46] hugetlb: add MADV_COLLAPSE for hugetlb

Quoted 12 lines · Peter Xu

mm/hugetlb.c int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, · line 7926

Quoted 9 lines · James Houghton
mm/hugetlb.c +9 -0
+     /*
+      * Grab the VMA lock and mapping sem for writing. This will prevent
+      * concurrent high-granularity page table walks, so that we can safely
+      * collapse and free page tables.
+      *
+      * This is the same locking that huge_pmd_unshare requires.
+      */
+     hugetlb_vma_lock_write(vma);
+     i_mmap_lock_write(vma->vm_file->f_mapping);
Quoted 3 lines

One thing I just noticed - do we need the mmap write lock here? I don't quickly see what stops another thread from having the mmap read and walking upon the pgtables being collapsed.

Maybe. Does huge_pmd_unshare() have the same problem?

#111 lore raw

PATCH 36/46 hugetlb: remove huge_pte_lock and huge_pte_lockptr

In reply to James Houghton

They are replaced with hugetlb_pte_lock{,ptr}. All callers that haven't already been replaced don't get called when using HGM, so we handle them by populating hugetlb_ptes with the standard, hstate-sized huge PTEs.

---
 arch/powerpc/mm/pgtable.c |  7 +++++--
 include/linux/hugetlb.h   | 42 +++++++++++++++------------------------
 mm/hugetlb.c              | 22 +++++++++++++-------
 3 files changed, 36 insertions(+), 35 deletions(-)
arch/powerpc/mm/pgtable.c +5 -2
index 035a0df47af0..e20d6aa9a2a6 100644
@@ -258,11 +258,14 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
 
 #ifdef CONFIG_PPC_BOOK3S_64
 		struct hstate *h = hstate_vma(vma);
+		struct hugetlb_pte hpte;
 
 		psize = hstate_get_psize(h);
 #ifdef CONFIG_DEBUG_VM
-		assert_spin_locked(huge_pte_lockptr(huge_page_shift(h),
-						    vma->vm_mm, ptep));
+		/* HGM is not supported for powerpc yet. */
+		hugetlb_pte_populate(&hpte, ptep, huge_page_shift(h),
+				hpage_size_to_level(psize));
+		assert_spin_locked(hpte.ptl);
 #endif
 
 #else
include/linux/hugetlb.h +16 -26

66 lines not on this page · raw message

mm/hugetlb.c +15 -7
index 388c46c7e77a..d71adc03138d 100644
@@ -5303,9 +5303,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 				put_page(hpage);
 
 				/* Install the new huge page if src pte stable */
-				dst_ptl = huge_pte_lock(h, dst, dst_pte);
-				src_ptl = huge_pte_lockptr(huge_page_shift(h),
-							   src, src_pte);
+				dst_ptl = hugetlb_pte_lock(&dst_hpte);
+				src_ptl = hugetlb_pte_lockptr(&src_hpte);
 				spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
 				entry = huge_ptep_get(src_pte);
 				if (!pte_same(src_pte_old, entry)) {
@@ -7383,7 +7382,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
 	unsigned long saddr;
 	pte_t *spte = NULL;
 	pte_t *pte;
-	spinlock_t *ptl;
+	struct hugetlb_pte hpte;
+	struct hstate *shstate;
 
 	i_mmap_lock_read(mapping);
 	vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) {
@@ -7404,7 +7404,11 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
 	if (!spte)
 		goto out;
 
-	ptl = huge_pte_lock(hstate_vma(vma), mm, spte);
+	shstate = hstate_vma(svma);
+
+	hugetlb_pte_populate(mm, &hpte, spte, huge_page_shift(shstate),
+			hpage_size_to_level(huge_page_size(shstate)));
+	spin_lock(hpte.ptl);
 	if (pud_none(*pud)) {
 		pud_populate(mm, pud,
 				(pmd_t *)((unsigned long)spte & PAGE_MASK));
@@ -7412,7 +7416,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
 	} else {
 		put_page(virt_to_page(spte));
 	}
-	spin_unlock(ptl);
+	spin_unlock(hpte.ptl);
 out:
 	pte = (pte_t *)pmd_alloc(mm, pud, addr);
 	i_mmap_unlock_read(mapping);
@@ -8132,6 +8136,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
 	unsigned long address, start, end;
 	spinlock_t *ptl;
 	pte_t *ptep;
+	struct hugetlb_pte hpte;
 
 	if (!(vma->vm_flags & VM_MAYSHARE))
 		return;
@@ -8156,7 +8161,10 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
 		ptep = hugetlb_walk(vma, address, sz);
 		if (!ptep)
 			continue;
-		ptl = huge_pte_lock(h, mm, ptep);
+
+		hugetlb_pte_populate(mm, &hpte, ptep, huge_page_shift(h),
+				     hpage_size_to_level(sz));
+		ptl = hugetlb_pte_lock(&hpte);
 		huge_pmd_unshare(mm, vma, address, ptep);
 		spin_unlock(ptl);
 	}
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#112 lore raw

PATCH 37/46 hugetlb: replace make_huge_pte with make_huge_pte_with_shift

In reply to James Houghton

This removes the old definition of make_huge_pte, where now we always require the shift to be explicitly given. All callsites are cleaned up.

---
 mm/hugetlb.c | 31 ++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)
mm/hugetlb.c +12 -19
index d71adc03138d..10a323e6bd9c 100644
@@ -5069,9 +5069,9 @@ const struct vm_operations_struct hugetlb_vm_ops = {
 	.pagesize = hugetlb_vm_op_pagesize,
 };
 
-static pte_t make_huge_pte_with_shift(struct vm_area_struct *vma,
-				      struct page *page, int writable,
-				      int shift)
+static pte_t make_huge_pte(struct vm_area_struct *vma,
+			   struct page *page, int writable,
+			   int shift)
 {
 	pte_t entry;
 
@@ -5087,14 +5087,6 @@ static pte_t make_huge_pte_with_shift(struct vm_area_struct *vma,
 	return entry;
 }
 
-static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page,
-			   int writable)
-{
-	unsigned int shift = huge_page_shift(hstate_vma(vma));
-
-	return make_huge_pte_with_shift(vma, page, writable, shift);
-}
-
 static void set_huge_ptep_writable(struct vm_area_struct *vma,
 				   unsigned long address, pte_t *ptep)
 {
@@ -5135,10 +5127,12 @@ static void
 hugetlb_install_page(struct vm_area_struct *vma, pte_t *ptep, unsigned long addr,
 		     struct page *new_page)
 {
+	struct hstate *h = hstate_vma(vma);
 	__SetPageUptodate(new_page);
 	hugepage_add_new_anon_rmap(new_page, vma, addr);
-	set_huge_pte_at(vma->vm_mm, addr, ptep, make_huge_pte(vma, new_page, 1));
-	hugetlb_count_add(pages_per_huge_page(hstate_vma(vma)), vma->vm_mm);
+	set_huge_pte_at(vma->vm_mm, addr, ptep, make_huge_pte(vma, new_page, 1,
+				huge_page_shift(h)));
+	hugetlb_count_add(pages_per_huge_page(h), vma->vm_mm);
 	SetHPageMigratable(new_page);
 }
 
@@ -5854,7 +5848,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,
 		page_remove_rmap(old_page, vma, true);
 		hugepage_add_new_anon_rmap(new_page, vma, haddr);
 		set_huge_pte_at(mm, haddr, ptep,
-				make_huge_pte(vma, new_page, !unshare));
+				make_huge_pte(vma, new_page, !unshare,
+					huge_page_shift(h)));
 		SetHPageMigratable(new_page);
 		/* Make the old page be freed below */
 		new_page = old_page;
@@ -6163,7 +6158,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
 		page_dup_file_rmap(page, true);
 
 	subpage = hugetlb_find_subpage(h, page, haddr_hgm);
-	new_pte = make_huge_pte_with_shift(vma, subpage,
+	new_pte = make_huge_pte(vma, subpage,
 			((vma->vm_flags & VM_WRITE)
 			 && (vma->vm_flags & VM_SHARED)),
 			hpte->shift);
@@ -6585,8 +6580,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,
 
 	subpage = hugetlb_find_subpage(h, page, dst_addr);
 
-	_dst_pte = make_huge_pte_with_shift(dst_vma, subpage, writable,
-			dst_hpte->shift);
+	_dst_pte = make_huge_pte(dst_vma, subpage, writable, dst_hpte->shift);
 	/*
 	 * Always mark UFFDIO_COPY page dirty; note that this may not be
 	 * extremely important for hugetlbfs for now since swapping is not
@@ -7999,8 +7993,7 @@ int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma,
 		page_dup_file_rmap(hpage, true);
 
 		subpage = hugetlb_find_subpage(h, hpage, curr);
-		entry = make_huge_pte_with_shift(vma, subpage,
-						 writable, hpte.shift);
+		entry = make_huge_pte(vma, subpage, writable, hpte.shift);
 		set_huge_pte_at(mm, curr, hpte.ptep, entry);
 next_hpte:
 		curr += hugetlb_pte_size(&hpte);
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#113 lore raw

PATCH 38/46 mm: smaps: add stats for HugeTLB mapping size

In reply to James Houghton

When the kernel is compiled with HUGETLB_HIGH_GRANULARITY_MAPPING, smaps may provide HugetlbPudMapped, HugetlbPmdMapped, and HugetlbPteMapped. Levels that are folded will not be outputted.

---
 fs/proc/task_mmu.c | 101 +++++++++++++++++++++++++++++++++------------
 1 file changed, 75 insertions(+), 26 deletions(-)
fs/proc/task_mmu.c +75 -26
index c353cab11eee..af31c4d314d2 100644
@@ -412,6 +412,15 @@ struct mem_size_stats {
 	unsigned long swap;
 	unsigned long shared_hugetlb;
 	unsigned long private_hugetlb;
+#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING
+#ifndef __PAGETABLE_PUD_FOLDED
+	unsigned long hugetlb_pud_mapped;
+#endif
+#ifndef __PAGETABLE_PMD_FOLDED
+	unsigned long hugetlb_pmd_mapped;
+#endif
+	unsigned long hugetlb_pte_mapped;
+#endif
 	u64 pss;
 	u64 pss_anon;
 	u64 pss_file;
@@ -731,6 +740,35 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma)
 }
 
 #ifdef CONFIG_HUGETLB_PAGE
+
+static void smaps_hugetlb_hgm_account(struct mem_size_stats *mss,
+		struct hugetlb_pte *hpte)
+{
+#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING
+	unsigned long size = hugetlb_pte_size(hpte);
+
+	switch (hpte->level) {
+#ifndef __PAGETABLE_PUD_FOLDED
+	case HUGETLB_LEVEL_PUD:
+		mss->hugetlb_pud_mapped += size;
+		break;
+#endif
+#ifndef __PAGETABLE_PMD_FOLDED
+	case HUGETLB_LEVEL_PMD:
+		mss->hugetlb_pmd_mapped += size;
+		break;
+#endif
+	case HUGETLB_LEVEL_PTE:
+		mss->hugetlb_pte_mapped += size;
+		break;
+	default:
+		break;
+	}
+#else
+	return;
+#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */
+}
+
 static int smaps_hugetlb_range(struct hugetlb_pte *hpte,
 				unsigned long addr,
 				struct mm_walk *walk)
@@ -764,6 +802,8 @@ static int smaps_hugetlb_range(struct hugetlb_pte *hpte,
 			mss->shared_hugetlb += hugetlb_pte_size(hpte);
 		else
 			mss->private_hugetlb += hugetlb_pte_size(hpte);
+
+		smaps_hugetlb_hgm_account(mss, hpte);
 	}
 	return 0;
 }
@@ -833,38 +873,47 @@ static void smap_gather_stats(struct vm_area_struct *vma,
 static void __show_smap(struct seq_file *m, const struct mem_size_stats *mss,
 	bool rollup_mode)
 {
-	SEQ_PUT_DEC("Rss:            ", mss->resident);
-	SEQ_PUT_DEC(" kB\nPss:            ", mss->pss >> PSS_SHIFT);
-	SEQ_PUT_DEC(" kB\nPss_Dirty:      ", mss->pss_dirty >> PSS_SHIFT);
+	SEQ_PUT_DEC("Rss:              ", mss->resident);
+	SEQ_PUT_DEC(" kB\nPss:              ", mss->pss >> PSS_SHIFT);
+	SEQ_PUT_DEC(" kB\nPss_Dirty:        ", mss->pss_dirty >> PSS_SHIFT);
 	if (rollup_mode) {
 		/*
 		 * These are meaningful only for smaps_rollup, otherwise two of
 		 * them are zero, and the other one is the same as Pss.
 		 */
-		SEQ_PUT_DEC(" kB\nPss_Anon:       ",
+		SEQ_PUT_DEC(" kB\nPss_Anon:         ",
 			mss->pss_anon >> PSS_SHIFT);
-		SEQ_PUT_DEC(" kB\nPss_File:       ",
+		SEQ_PUT_DEC(" kB\nPss_File:         ",
 			mss->pss_file >> PSS_SHIFT);
-		SEQ_PUT_DEC(" kB\nPss_Shmem:      ",
+		SEQ_PUT_DEC(" kB\nPss_Shmem:        ",
 			mss->pss_shmem >> PSS_SHIFT);
 	}
-	SEQ_PUT_DEC(" kB\nShared_Clean:   ", mss->shared_clean);
-	SEQ_PUT_DEC(" kB\nShared_Dirty:   ", mss->shared_dirty);
-	SEQ_PUT_DEC(" kB\nPrivate_Clean:  ", mss->private_clean);
-	SEQ_PUT_DEC(" kB\nPrivate_Dirty:  ", mss->private_dirty);
-	SEQ_PUT_DEC(" kB\nReferenced:     ", mss->referenced);
-	SEQ_PUT_DEC(" kB\nAnonymous:      ", mss->anonymous);
-	SEQ_PUT_DEC(" kB\nLazyFree:       ", mss->lazyfree);
-	SEQ_PUT_DEC(" kB\nAnonHugePages:  ", mss->anonymous_thp);
-	SEQ_PUT_DEC(" kB\nShmemPmdMapped: ", mss->shmem_thp);
-	SEQ_PUT_DEC(" kB\nFilePmdMapped:  ", mss->file_thp);
-	SEQ_PUT_DEC(" kB\nShared_Hugetlb: ", mss->shared_hugetlb);
-	seq_put_decimal_ull_width(m, " kB\nPrivate_Hugetlb: ",
+	SEQ_PUT_DEC(" kB\nShared_Clean:     ", mss->shared_clean);
+	SEQ_PUT_DEC(" kB\nShared_Dirty:     ", mss->shared_dirty);
+	SEQ_PUT_DEC(" kB\nPrivate_Clean:    ", mss->private_clean);
+	SEQ_PUT_DEC(" kB\nPrivate_Dirty:    ", mss->private_dirty);
+	SEQ_PUT_DEC(" kB\nReferenced:       ", mss->referenced);
+	SEQ_PUT_DEC(" kB\nAnonymous:        ", mss->anonymous);
+	SEQ_PUT_DEC(" kB\nLazyFree:         ", mss->lazyfree);
+	SEQ_PUT_DEC(" kB\nAnonHugePages:    ", mss->anonymous_thp);
+	SEQ_PUT_DEC(" kB\nShmemPmdMapped:   ", mss->shmem_thp);
+	SEQ_PUT_DEC(" kB\nFilePmdMapped:    ", mss->file_thp);
+	SEQ_PUT_DEC(" kB\nShared_Hugetlb:   ", mss->shared_hugetlb);
+	seq_put_decimal_ull_width(m, " kB\nPrivate_Hugetlb:   ",
 				  mss->private_hugetlb >> 10, 7);
-	SEQ_PUT_DEC(" kB\nSwap:           ", mss->swap);
-	SEQ_PUT_DEC(" kB\nSwapPss:        ",
+#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING
+#ifndef __PAGETABLE_PUD_FOLDED
+	SEQ_PUT_DEC(" kB\nHugetlbPudMapped: ", mss->hugetlb_pud_mapped);
+#endif
+#ifndef __PAGETABLE_PMD_FOLDED
+	SEQ_PUT_DEC(" kB\nHugetlbPmdMapped: ", mss->hugetlb_pmd_mapped);
+#endif
+	SEQ_PUT_DEC(" kB\nHugetlbPteMapped: ", mss->hugetlb_pte_mapped);
+#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */
+	SEQ_PUT_DEC(" kB\nSwap:             ", mss->swap);
+	SEQ_PUT_DEC(" kB\nSwapPss:          ",
 					mss->swap_pss >> PSS_SHIFT);
-	SEQ_PUT_DEC(" kB\nLocked:         ",
+	SEQ_PUT_DEC(" kB\nLocked:           ",
 					mss->pss_locked >> PSS_SHIFT);
 	seq_puts(m, " kB\n");
 }
@@ -880,18 +929,18 @@ static int show_smap(struct seq_file *m, void *v)
 
 	show_map_vma(m, vma);
 
-	SEQ_PUT_DEC("Size:           ", vma->vm_end - vma->vm_start);
-	SEQ_PUT_DEC(" kB\nKernelPageSize: ", vma_kernel_pagesize(vma));
-	SEQ_PUT_DEC(" kB\nMMUPageSize:    ", vma_mmu_pagesize(vma));
+	SEQ_PUT_DEC("Size:             ", vma->vm_end - vma->vm_start);
+	SEQ_PUT_DEC(" kB\nKernelPageSize:   ", vma_kernel_pagesize(vma));
+	SEQ_PUT_DEC(" kB\nMMUPageSize:      ", vma_mmu_pagesize(vma));
 	seq_puts(m, " kB\n");
 
 	__show_smap(m, &mss, false);
 
-	seq_printf(m, "THPeligible:    %d\n",
+	seq_printf(m, "THPeligible:      %d\n",
 		   hugepage_vma_check(vma, vma->vm_flags, true, false, true));
 
 	if (arch_pkeys_enabled())
-		seq_printf(m, "ProtectionKey:  %8u\n", vma_pkey(vma));
+		seq_printf(m, "ProtectionKey:    %8u\n", vma_pkey(vma));
 	show_smap_vma_flags(m, vma);
 
 	return 0;
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#114 lore raw

PATCH 39/46 hugetlb: x86: enable high-granularity mapping

In reply to James Houghton

Now that HGM is fully supported for GENERAL_HUGETLB, x86 can enable it. The x86 KVM MMU already properly handles HugeTLB HGM pages (it does a page table walk to determine which size to use in the second-stage page table instead of, for example, checking vma_mmu_pagesize, like arm64 does).

We could also enable HugeTLB HGM for arm (32-bit) at this point, as it also uses GENERAL_HUGETLB and I don't see anything else that is needed for it. However, I haven't tested on arm at all, so I won't enable it.

---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
index 3604074a878b..3d08cd45549c 100644
@@ -126,6 +126,7 @@ config X86
 	select ARCH_WANT_GENERAL_HUGETLB
 	select ARCH_WANT_HUGE_PMD_SHARE
 	select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP	if X86_64
+	select ARCH_WANT_HUGETLB_HIGH_GRANULARITY_MAPPING
 	select ARCH_WANT_LD_ORPHAN_WARN
 	select ARCH_WANTS_THP_SWAP		if X86_64
 	select ARCH_HAS_PARANOID_L1D_FLUSH
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#115 lore raw

Re: [PATCH 39/46] hugetlb: x86: enable high-granularity mapping

Quoted 10 lines · James Houghton

Now that HGM is fully supported for GENERAL_HUGETLB, x86 can enable it. The x86 KVM MMU already properly handles HugeTLB HGM pages (it does a page table walk to determine which size to use in the second-stage page table instead of, for example, checking vma_mmu_pagesize, like arm64 does).

We could also enable HugeTLB HGM for arm (32-bit) at this point, as it also uses GENERAL_HUGETLB and I don't see anything else that is needed for it. However, I haven't tested on arm at all, so I won't enable it.

Given that we are using a high bit for VM_HUGETLB_HGM, we can only support 64-bit architectures. Userfaultfd minor faults is limited to 64-bit architectures for the same reason: VM_UFFD_MINOR uses a bit.

Quoted 11 lines
---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/x86/Kconfig +0 -0
index 3604074a878b..3d08cd45549c 100644

arch/x86/Kconfig config X86 · line 129

Quoted 4 lines
arch/x86/Kconfig +1 -0
@@ -126,6 +126,7 @@ config X86
        select ARCH_WANT_GENERAL_HUGETLB
        select ARCH_WANT_HUGE_PMD_SHARE
        select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP  if X86_64
+       select ARCH_WANT_HUGETLB_HIGH_GRANULARITY_MAPPING

This needs if X86_64 at the end. Will be corrected for v2.

Quoted 6 lines
arch/x86/Kconfig +0 -1
        select ARCH_WANT_LD_ORPHAN_WARN
        select ARCH_WANTS_THP_SWAP              if X86_64
        select ARCH_HAS_PARANOID_L1D_FLUSH
--

2.39.0.314.g84b9a713c41-goog

#116 lore raw

PATCH 40/46 docs: hugetlb: update hugetlb and userfaultfd admin-guides with HGM info

In reply to James Houghton

This includes information about how UFFD_FEATURE_MINOR_HUGETLBFS_HGM should be used and when MADV_COLLAPSE should be used with it.

---
 Documentation/admin-guide/mm/hugetlbpage.rst |  4 ++++
 Documentation/admin-guide/mm/userfaultfd.rst | 16 +++++++++++++++-
 2 files changed, 19 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/hugetlbpage.rst +4 -0
index 19f27c0d92e0..ca7db15ae768 100644
@@ -454,6 +454,10 @@ errno set to EINVAL or exclude hugetlb pages that extend beyond the length if
 not hugepage aligned.  For example, munmap(2) will fail if memory is backed by
 a hugetlb page and the length is smaller than the hugepage size.
 
+It is possible for users to map HugeTLB pages at a higher granularity than
+normal using HugeTLB high-granularity mapping (HGM). For example, when using 1G
+pages on x86, a user could map that page with 4K PTEs, 2M PMDs, a combination of
+the two. See Documentation/admin-guide/mm/userfaultfd.rst.
 
 Examples
 ========
Documentation/admin-guide/mm/userfaultfd.rst +15 -1
index 83f31919ebb3..19877aaad61b 100644
@@ -115,6 +115,14 @@ events, except page fault notifications, may be generated:
   areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating
   support for shmem virtual memory areas.
 
+- ``UFFD_FEATURE_MINOR_HUGETLBFS_HGM`` indicates that the kernel supports
+  small-page-aligned regions for ``UFFDIO_CONTINUE`` in HugeTLB-backed
+  virtual memory areas. ``UFFD_FEATURE_MINOR_HUGETLBFS_HGM`` and
+  ``UFFD_FEATURE_EXACT_ADDRESS`` must both be specified explicitly to enable
+  this behavior. If ``UFFD_FEATURE_MINOR_HUGETLBFS_HGM`` is specified but
+  ``UFFD_FEATURE_EXACT_ADDRESS`` is not, then ``UFFDIO_API`` will fail with
+  ``EINVAL``.
+
 The userland application should set the feature flags it intends to use
 when invoking the ``UFFDIO_API`` ioctl, to request that those features be
 enabled if supported.
@@ -169,7 +177,13 @@ like to do to resolve it:
   the page cache). Userspace has the option of modifying the page's
   contents before resolving the fault. Once the contents are correct
   (modified or not), userspace asks the kernel to map the page and let the
-  faulting thread continue with ``UFFDIO_CONTINUE``.
+  faulting thread continue with ``UFFDIO_CONTINUE``. If this is done at the
+  base-page size in a transparent-hugepage-eligible VMA or in a HugeTLB VMA
+  (requires ``UFFD_FEATURE_MINOR_HUGETLBFS_HGM``), then userspace may want to
+  use ``MADV_COLLAPSE`` when a hugepage is fully populated to inform the kernel
+  that it may be able to collapse the mapping. ``MADV_COLLAPSE`` will may undo
+  the effect of any ``UFFDIO_WRITEPROTECT`` calls on the collapsed address
+  range.
 
 Notes:
 
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#117 lore raw

PATCH 41/46 docs: proc: include information about HugeTLB HGM

In reply to James Houghton

This includes the updates that have been made to smaps, specifically, the addition of Hugetlb[Pud,Pmd,Pte]Mapped.

---
 Documentation/filesystems/proc.rst | 56 +++++++++++++++++-------------
 1 file changed, 32 insertions(+), 24 deletions(-)
Documentation/filesystems/proc.rst +32 -24
index e224b6d5b642..1fbb1310cea1 100644
@@ -447,29 +447,32 @@ Memory Area, or VMA) there is a series of lines such as the following::
 
     08048000-080bc000 r-xp 00000000 03:02 13130      /bin/bash
 
-    Size:               1084 kB
-    KernelPageSize:        4 kB
-    MMUPageSize:           4 kB
-    Rss:                 892 kB
-    Pss:                 374 kB
-    Pss_Dirty:             0 kB
-    Shared_Clean:        892 kB
-    Shared_Dirty:          0 kB
-    Private_Clean:         0 kB
-    Private_Dirty:         0 kB
-    Referenced:          892 kB
-    Anonymous:             0 kB
-    LazyFree:              0 kB
-    AnonHugePages:         0 kB
-    ShmemPmdMapped:        0 kB
-    Shared_Hugetlb:        0 kB
-    Private_Hugetlb:       0 kB
-    Swap:                  0 kB
-    SwapPss:               0 kB
-    KernelPageSize:        4 kB
-    MMUPageSize:           4 kB
-    Locked:                0 kB
-    THPeligible:           0
+    Size:                 1084 kB
+    KernelPageSize:          4 kB
+    MMUPageSize:             4 kB
+    Rss:                   892 kB
+    Pss:                   374 kB
+    Pss_Dirty:               0 kB
+    Shared_Clean:          892 kB
+    Shared_Dirty:            0 kB
+    Private_Clean:           0 kB
+    Private_Dirty:           0 kB
+    Referenced:            892 kB
+    Anonymous:               0 kB
+    LazyFree:                0 kB
+    AnonHugePages:           0 kB
+    ShmemPmdMapped:          0 kB
+    Shared_Hugetlb:          0 kB
+    Private_Hugetlb:         0 kB
+    HugetlbPudMapped:        0 kB
+    HugetlbPmdMapped:        0 kB
+    HugetlbPteMapped:        0 kB
+    Swap:                    0 kB
+    SwapPss:                 0 kB
+    KernelPageSize:          4 kB
+    MMUPageSize:             4 kB
+    Locked:                  0 kB
+    THPeligible:             0
     VmFlags: rd ex mr mw me dw
 
 The first of these lines shows the same information as is displayed for the
@@ -510,10 +513,15 @@ implementation. If this is not desirable please file a bug report.
 "ShmemPmdMapped" shows the ammount of shared (shmem/tmpfs) memory backed by
 huge pages.
 
-"Shared_Hugetlb" and "Private_Hugetlb" show the ammounts of memory backed by
+"Shared_Hugetlb" and "Private_Hugetlb" show the amounts of memory backed by
 hugetlbfs page which is *not* counted in "RSS" or "PSS" field for historical
 reasons. And these are not included in {Shared,Private}_{Clean,Dirty} field.
 
+If the kernel was compiled with ``CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING``,
+"HugetlbPudMapped", "HugetlbPmdMapped", and "HugetlbPteMapped" will appear and
+show the amount of HugeTLB memory mapped with PUDs, PMDs, and PTEs respectively.
+See Documentation/admin-guide/mm/hugetlbpage.rst.
+
 "Swap" shows how much would-be-anonymous memory is also used, but out on swap.
 
 For shmem mappings, "Swap" includes also the size of the mapped (and not
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#118 lore raw

PATCH 42/46 selftests/vm: add HugeTLB HGM to userfaultfd selftest

In reply to James Houghton

This test case behaves similarly to the regular shared HugeTLB configuration, except that it uses 4K instead of hugepages, and that we ignore the UFFDIO_COPY tests, as UFFDIO_CONTINUE is the only ioctl that supports PAGE_SIZE-aligned regions.

This doesn't test MADV_COLLAPSE. Other tests are added later to exercise MADV_COLLAPSE.

---
 tools/testing/selftests/vm/userfaultfd.c | 84 +++++++++++++++++++-----
 1 file changed, 69 insertions(+), 15 deletions(-)
tools/testing/selftests/vm/userfaultfd.c +69 -15
index 7f22844ed704..681c5c5f863b 100644
@@ -73,9 +73,10 @@ static unsigned long nr_cpus, nr_pages, nr_pages_per_cpu, page_size, hpage_size;
 #define BOUNCE_POLL		(1<<3)
 static int bounces;
 
-#define TEST_ANON	1
-#define TEST_HUGETLB	2
-#define TEST_SHMEM	3
+#define TEST_ANON		1
+#define TEST_HUGETLB		2
+#define TEST_HUGETLB_HGM	3
+#define TEST_SHMEM		4
 static int test_type;
 
 #define UFFD_FLAGS	(O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY)
@@ -93,6 +94,8 @@ static volatile bool test_uffdio_zeropage_eexist = true;
 static bool test_uffdio_wp = true;
 /* Whether to test uffd minor faults */
 static bool test_uffdio_minor = false;
+static bool test_uffdio_copy = true;
+
 static bool map_shared;
 static int mem_fd;
 static unsigned long long *count_verify;
@@ -151,7 +154,7 @@ static void usage(void)
 	fprintf(stderr, "\nUsage: ./userfaultfd <test type> <MiB> <bounces> "
 		"[hugetlbfs_file]\n\n");
 	fprintf(stderr, "Supported <test type>: anon, hugetlb, "
-		"hugetlb_shared, shmem\n\n");
+		"hugetlb_shared, hugetlb_shared_hgm, shmem\n\n");
 	fprintf(stderr, "'Test mods' can be joined to the test type string with a ':'. "
 		"Supported mods:\n");
 	fprintf(stderr, "\tsyscall - Use userfaultfd(2) (default)\n");
@@ -167,6 +170,11 @@ static void usage(void)
 	exit(1);
 }
 
+static bool test_is_hugetlb(void)
+{
+	return test_type == TEST_HUGETLB || test_type == TEST_HUGETLB_HGM;
+}
+
 #define _err(fmt, ...)						\
 	do {							\
 		int ret = errno;				\
@@ -381,7 +389,7 @@ static struct uffd_test_ops *uffd_test_ops;
 
 static inline uint64_t uffd_minor_feature(void)
 {
-	if (test_type == TEST_HUGETLB && map_shared)
+	if (test_is_hugetlb() && map_shared)
 		return UFFD_FEATURE_MINOR_HUGETLBFS;
 	else if (test_type == TEST_SHMEM)
 		return UFFD_FEATURE_MINOR_SHMEM;
@@ -393,7 +401,7 @@ static uint64_t get_expected_ioctls(uint64_t mode)
 {
 	uint64_t ioctls = UFFD_API_RANGE_IOCTLS;
 
-	if (test_type == TEST_HUGETLB)
+	if (test_is_hugetlb())
 		ioctls &= ~(1 << _UFFDIO_ZEROPAGE);
 
 	if (!((mode & UFFDIO_REGISTER_MODE_WP) && test_uffdio_wp))
@@ -500,13 +508,16 @@ static void uffd_test_ctx_clear(void)
 static void uffd_test_ctx_init(uint64_t features)
 {
 	unsigned long nr, cpu;
+	uint64_t enabled_features = features;
 
 	uffd_test_ctx_clear();
 
 	uffd_test_ops->allocate_area((void **)&area_src, true);
 	uffd_test_ops->allocate_area((void **)&area_dst, false);
 
-	userfaultfd_open(&features);
+	userfaultfd_open(&enabled_features);
+	if ((enabled_features & features) != features)
+		err("couldn't enable all features");
 
 	count_verify = malloc(nr_pages * sizeof(unsigned long long));
 	if (!count_verify)
@@ -726,13 +737,16 @@ static void uffd_handle_page_fault(struct uffd_msg *msg,
 				   struct uffd_stats *stats)
 {
 	unsigned long offset;
+	unsigned long address;
 
 	if (msg->event != UFFD_EVENT_PAGEFAULT)
 		err("unexpected msg event %u", msg->event);
 
+	address = msg->arg.pagefault.address;
+
 	if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP) {
 		/* Write protect page faults */
-		wp_range(uffd, msg->arg.pagefault.address, page_size, false);
+		wp_range(uffd, address, page_size, false);
 		stats->wp_faults++;
 	} else if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR) {
 		uint8_t *area;
@@ -751,11 +765,10 @@ static void uffd_handle_page_fault(struct uffd_msg *msg,
 		 */
 
 		area = (uint8_t *)(area_dst +
-				   ((char *)msg->arg.pagefault.address -
-				    area_dst_alias));
+				   ((char *)address - area_dst_alias));
 		for (b = 0; b < page_size; ++b)
 			area[b] = ~area[b];
-		continue_range(uffd, msg->arg.pagefault.address, page_size);
+		continue_range(uffd, address, page_size);
 		stats->minor_faults++;
 	} else {
 		/*
@@ -782,7 +795,7 @@ static void uffd_handle_page_fault(struct uffd_msg *msg,
 		if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WRITE)
 			err("unexpected write fault");
 
-		offset = (char *)(unsigned long)msg->arg.pagefault.address - area_dst;
+		offset = (char *)address - area_dst;
 		offset &= ~(page_size-1);
 
 		if (copy_page(uffd, offset))
@@ -1192,6 +1205,12 @@ static int userfaultfd_events_test(void)
 	char c;
 	struct uffd_stats stats = { 0 };
 
+	if (!test_uffdio_copy) {
+		printf("Skipping userfaultfd events test "
+			"(test_uffdio_copy=false)\n");
+		return 0;
+	}
+
 	printf("testing events (fork, remap, remove): ");
 	fflush(stdout);
 
@@ -1245,6 +1264,12 @@ static int userfaultfd_sig_test(void)
 	char c;
 	struct uffd_stats stats = { 0 };
 
+	if (!test_uffdio_copy) {
+		printf("Skipping userfaultfd signal test "
+			"(test_uffdio_copy=false)\n");
+		return 0;
+	}
+
 	printf("testing signal delivery: ");
 	fflush(stdout);
 
@@ -1329,6 +1354,11 @@ static int userfaultfd_minor_test(void)
 
 	uffd_test_ctx_init(uffd_minor_feature());
 
+	if (test_type == TEST_HUGETLB_HGM)
+		/* Enable high-granularity userfaultfd ioctls for HugeTLB */
+		if (madvise(area_dst_alias, nr_pages * page_size, MADV_SPLIT))
+			err("MADV_SPLIT failed");
+
 	uffdio_register.range.start = (unsigned long)area_dst_alias;
 	uffdio_register.range.len = nr_pages * page_size;
 	uffdio_register.mode = UFFDIO_REGISTER_MODE_MINOR;
@@ -1538,6 +1568,12 @@ static int userfaultfd_stress(void)
 	pthread_attr_init(&attr);
 	pthread_attr_setstacksize(&attr, 16*1024*1024);
 
+	if (!test_uffdio_copy) {
+		printf("Skipping userfaultfd stress test "
+			"(test_uffdio_copy=false)\n");
+		bounces = 0;
+	}
+
 	while (bounces--) {
 		printf("bounces: %d, mode:", bounces);
 		if (bounces & BOUNCE_RANDOM)
@@ -1696,6 +1732,16 @@ static void set_test_type(const char *type)
 		uffd_test_ops = &hugetlb_uffd_test_ops;
 		/* Minor faults require shared hugetlb; only enable here. */
 		test_uffdio_minor = true;
+	} else if (!strcmp(type, "hugetlb_shared_hgm")) {
+		map_shared = true;
+		test_type = TEST_HUGETLB_HGM;
+		uffd_test_ops = &hugetlb_uffd_test_ops;
+		/*
+		 * HugeTLB HGM only changes UFFDIO_CONTINUE, so don't test
+		 * UFFDIO_COPY.
+		 */
+		test_uffdio_minor = true;
+		test_uffdio_copy = false;
 	} else if (!strcmp(type, "shmem")) {
 		map_shared = true;
 		test_type = TEST_SHMEM;
@@ -1731,6 +1777,7 @@ static void parse_test_type_arg(const char *raw_type)
 		err("Unsupported test: %s", raw_type);
 
 	if (test_type == TEST_HUGETLB)
+		/* TEST_HUGETLB_HGM gets small pages. */
 		page_size = hpage_size;
 	else
 		page_size = sysconf(_SC_PAGE_SIZE);
@@ -1813,22 +1860,29 @@ int main(int argc, char **argv)
 		nr_cpus = x < y ? x : y;
 	}
 	nr_pages_per_cpu = bytes / page_size / nr_cpus;
+	if (test_type == TEST_HUGETLB_HGM)
+		/*
+		 * `page_size` refers to the page_size we can use in
+		 * UFFDIO_CONTINUE. We still need nr_pages to be appropriately
+		 * aligned, so align it here.
+		 */
+		nr_pages_per_cpu -= nr_pages_per_cpu % (hpage_size / page_size);
 	if (!nr_pages_per_cpu) {
 		_err("invalid MiB");
 		usage();
 	}
+	nr_pages = nr_pages_per_cpu * nr_cpus;
 
 	bounces = atoi(argv[3]);
 	if (bounces <= 0) {
 		_err("invalid bounces");
 		usage();
 	}
-	nr_pages = nr_pages_per_cpu * nr_cpus;
 
-	if (test_type == TEST_SHMEM || test_type == TEST_HUGETLB) {
+	if (test_type == TEST_SHMEM || test_is_hugetlb()) {
 		unsigned int memfd_flags = 0;
 
-		if (test_type == TEST_HUGETLB)
+		if (test_is_hugetlb())
 			memfd_flags = MFD_HUGETLB;
 		mem_fd = memfd_create(argv[0], memfd_flags);
 		if (mem_fd < 0)
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#119 lore raw

PATCH 43/46 selftests/kvm: add HugeTLB HGM to KVM demand paging selftest

In reply to James Houghton

This test exercises the GUP paths for HGM. MADV_COLLAPSE is not tested.

---
 tools/testing/selftests/kvm/demand_paging_test.c   |  2 +-
 tools/testing/selftests/kvm/include/test_util.h    |  2 ++
 .../selftests/kvm/include/userfaultfd_util.h       |  6 +++---
 tools/testing/selftests/kvm/lib/kvm_util.c         |  2 +-
 tools/testing/selftests/kvm/lib/test_util.c        | 14 ++++++++++++++
 tools/testing/selftests/kvm/lib/userfaultfd_util.c | 14 +++++++++++---
 6 files changed, 32 insertions(+), 8 deletions(-)
tools/testing/selftests/kvm/demand_paging_test.c +1 -1
index b0e1fc4de9e2..e534f9c927bf 100644
@@ -170,7 +170,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)
 			uffd_descs[i] = uffd_setup_demand_paging(
 				p->uffd_mode, p->uffd_delay, vcpu_hva,
 				vcpu_args->pages * memstress_args.guest_page_size,
-				&handle_uffd_page_request);
+				p->src_type, &handle_uffd_page_request);
 		}
 	}
 
tools/testing/selftests/kvm/include/test_util.h +2 -0
index 80d6416f3012..a2106c19a614 100644
@@ -103,6 +103,7 @@ enum vm_mem_backing_src_type {
 	VM_MEM_SRC_ANONYMOUS_HUGETLB_16GB,
 	VM_MEM_SRC_SHMEM,
 	VM_MEM_SRC_SHARED_HUGETLB,
+	VM_MEM_SRC_SHARED_HUGETLB_HGM,
 	NUM_SRC_TYPES,
 };
 
@@ -121,6 +122,7 @@ size_t get_def_hugetlb_pagesz(void);
 const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(uint32_t i);
 size_t get_backing_src_pagesz(uint32_t i);
 bool is_backing_src_hugetlb(uint32_t i);
+bool is_backing_src_shared_hugetlb(enum vm_mem_backing_src_type src_type);
 void backing_src_help(const char *flag);
 enum vm_mem_backing_src_type parse_backing_src_type(const char *type_name);
 long get_run_delay(void);
tools/testing/selftests/kvm/include/userfaultfd_util.h +3 -3
index 877449c34592..d91528a58245 100644
@@ -26,9 +26,9 @@ struct uffd_desc {
 	pthread_t thread;
 };
 
-struct uffd_desc *uffd_setup_demand_paging(int uffd_mode, useconds_t delay,
-					   void *hva, uint64_t len,
-					   uffd_handler_t handler);
+struct uffd_desc *uffd_setup_demand_paging(
+		int uffd_mode, useconds_t delay, void *hva, uint64_t len,
+		enum vm_mem_backing_src_type src_type, uffd_handler_t handler);
 
 void uffd_stop_demand_paging(struct uffd_desc *uffd);
 
tools/testing/selftests/kvm/lib/kvm_util.c +1 -1
index c88c3ace16d2..67e7223f054b 100644
@@ -972,7 +972,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm,
 	region->fd = -1;
 	if (backing_src_is_shared(src_type))
 		region->fd = kvm_memfd_alloc(region->mmap_size,
-					     src_type == VM_MEM_SRC_SHARED_HUGETLB);
+				is_backing_src_shared_hugetlb(src_type));
 
 	region->mmap_start = mmap(NULL, region->mmap_size,
 				  PROT_READ | PROT_WRITE,
tools/testing/selftests/kvm/lib/test_util.c +14 -0
index 5c22fa4c2825..712a0878932e 100644
@@ -271,6 +271,13 @@ const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(uint32_t i)
 			 */
 			.flag = MAP_SHARED,
 		},
+		[VM_MEM_SRC_SHARED_HUGETLB_HGM] = {
+			/*
+			 * Identical to shared_hugetlb except for the name.
+			 */
+			.name = "shared_hugetlb_hgm",
+			.flag = MAP_SHARED,
+		},
 	};
 	_Static_assert(ARRAY_SIZE(aliases) == NUM_SRC_TYPES,
 		       "Missing new backing src types?");
@@ -289,6 +296,7 @@ size_t get_backing_src_pagesz(uint32_t i)
 	switch (i) {
 	case VM_MEM_SRC_ANONYMOUS:
 	case VM_MEM_SRC_SHMEM:
+	case VM_MEM_SRC_SHARED_HUGETLB_HGM:
 		return getpagesize();
 	case VM_MEM_SRC_ANONYMOUS_THP:
 		return get_trans_hugepagesz();
@@ -305,6 +313,12 @@ bool is_backing_src_hugetlb(uint32_t i)
 	return !!(vm_mem_backing_src_alias(i)->flag & MAP_HUGETLB);
 }
 
+bool is_backing_src_shared_hugetlb(enum vm_mem_backing_src_type src_type)
+{
+	return src_type == VM_MEM_SRC_SHARED_HUGETLB ||
+		src_type == VM_MEM_SRC_SHARED_HUGETLB_HGM;
+}
+
 static void print_available_backing_src_types(const char *prefix)
 {
 	int i;
tools/testing/selftests/kvm/lib/userfaultfd_util.c +11 -3
index 92cef20902f1..3c7178d6c4f4 100644
@@ -25,6 +25,10 @@
 
 #ifdef __NR_userfaultfd
 
+#ifndef MADV_SPLIT
+#define MADV_SPLIT 26
+#endif
+
 static void *uffd_handler_thread_fn(void *arg)
 {
 	struct uffd_desc *uffd_desc = (struct uffd_desc *)arg;
@@ -108,9 +112,9 @@ static void *uffd_handler_thread_fn(void *arg)
 	return NULL;
 }
 
-struct uffd_desc *uffd_setup_demand_paging(int uffd_mode, useconds_t delay,
-					   void *hva, uint64_t len,
-					   uffd_handler_t handler)
+struct uffd_desc *uffd_setup_demand_paging(
+		int uffd_mode, useconds_t delay, void *hva, uint64_t len,
+		enum vm_mem_backing_src_type src_type, uffd_handler_t handler)
 {
 	struct uffd_desc *uffd_desc;
 	bool is_minor = (uffd_mode == UFFDIO_REGISTER_MODE_MINOR);
@@ -140,6 +144,10 @@ struct uffd_desc *uffd_setup_demand_paging(int uffd_mode, useconds_t delay,
 		    "ioctl UFFDIO_API failed: %" PRIu64,
 		    (uint64_t)uffdio_api.api);
 
+	if (src_type == VM_MEM_SRC_SHARED_HUGETLB_HGM)
+		TEST_ASSERT(!madvise(hva, len, MADV_SPLIT),
+				"Could not enable HGM");
+
 	uffdio_register.range.start = (uint64_t)hva;
 	uffdio_register.range.len = len;
 	uffdio_register.mode = uffd_mode;
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#120 lore raw

PATCH 44/46 selftests/vm: add anon and shared hugetlb to migration test

In reply to James Houghton

Shared HugeTLB mappings are migrated best-effort. Sometimes, due to being unable to grab the VMA lock for writing, migration may just randomly fail. To allow for that, we allow retries.

---
 tools/testing/selftests/vm/migration.c | 83 ++++++++++++++++++++++++--
 1 file changed, 79 insertions(+), 4 deletions(-)
tools/testing/selftests/vm/migration.c +79 -4
index 1cec8425e3ca..21577a84d7e4 100644
@@ -13,6 +13,7 @@
 #include <sys/types.h>
 #include <signal.h>
 #include <time.h>
+#include <sys/statfs.h>
 
 #define TWOMEG (2<<20)
 #define RUNTIME (60)
@@ -59,11 +60,12 @@ FIXTURE_TEARDOWN(migration)
 	free(self->pids);
 }
 
-int migrate(uint64_t *ptr, int n1, int n2)
+int migrate(uint64_t *ptr, int n1, int n2, int retries)
 {
 	int ret, tmp;
 	int status = 0;
 	struct timespec ts1, ts2;
+	int failed = 0;
 
 	if (clock_gettime(CLOCK_MONOTONIC, &ts1))
 		return -1;
@@ -78,6 +80,9 @@ int migrate(uint64_t *ptr, int n1, int n2)
 		ret = move_pages(0, 1, (void **) &ptr, &n2, &status,
 				MPOL_MF_MOVE_ALL);
 		if (ret) {
+			if (++failed < retries)
+				continue;
+
 			if (ret > 0)
 				printf("Didn't migrate %d pages\n", ret);
 			else
@@ -88,6 +93,7 @@ int migrate(uint64_t *ptr, int n1, int n2)
 		tmp = n2;
 		n2 = n1;
 		n1 = tmp;
+		failed = 0;
 	}
 
 	return 0;
@@ -128,7 +134,7 @@ TEST_F_TIMEOUT(migration, private_anon, 2*RUNTIME)
 		if (pthread_create(&self->threads[i], NULL, access_mem, ptr))
 			perror("Couldn't create thread");
 
-	ASSERT_EQ(migrate(ptr, self->n1, self->n2), 0);
+	ASSERT_EQ(migrate(ptr, self->n1, self->n2, 1), 0);
 	for (i = 0; i < self->nthreads - 1; i++)
 		ASSERT_EQ(pthread_cancel(self->threads[i]), 0);
 }
@@ -158,7 +164,7 @@ TEST_F_TIMEOUT(migration, shared_anon, 2*RUNTIME)
 			self->pids[i] = pid;
 	}
 
-	ASSERT_EQ(migrate(ptr, self->n1, self->n2), 0);
+	ASSERT_EQ(migrate(ptr, self->n1, self->n2, 1), 0);
 	for (i = 0; i < self->nthreads - 1; i++)
 		ASSERT_EQ(kill(self->pids[i], SIGTERM), 0);
 }
@@ -185,9 +191,78 @@ TEST_F_TIMEOUT(migration, private_anon_thp, 2*RUNTIME)
 		if (pthread_create(&self->threads[i], NULL, access_mem, ptr))
 			perror("Couldn't create thread");
 
-	ASSERT_EQ(migrate(ptr, self->n1, self->n2), 0);
+	ASSERT_EQ(migrate(ptr, self->n1, self->n2, 1), 0);
+	for (i = 0; i < self->nthreads - 1; i++)
+		ASSERT_EQ(pthread_cancel(self->threads[i]), 0);
+}
+
+/*
+ * Tests the anon hugetlb migration entry paths.
+ */
+TEST_F_TIMEOUT(migration, private_anon_hugetlb, 2*RUNTIME)
+{
+	uint64_t *ptr;
+	int i;
+
+	if (self->nthreads < 2 || self->n1 < 0 || self->n2 < 0)
+		SKIP(return, "Not enough threads or NUMA nodes available");
+
+	ptr = mmap(NULL, TWOMEG, PROT_READ | PROT_WRITE,
+		MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
+	if (ptr == MAP_FAILED)
+		SKIP(return, "Could not allocate hugetlb pages");
+
+	memset(ptr, 0xde, TWOMEG);
+	for (i = 0; i < self->nthreads - 1; i++)
+		if (pthread_create(&self->threads[i], NULL, access_mem, ptr))
+			perror("Couldn't create thread");
+
+	ASSERT_EQ(migrate(ptr, self->n1, self->n2, 1), 0);
 	for (i = 0; i < self->nthreads - 1; i++)
 		ASSERT_EQ(pthread_cancel(self->threads[i]), 0);
 }
 
+/*
+ * Tests the shared hugetlb migration entry paths.
+ */
+TEST_F_TIMEOUT(migration, shared_hugetlb, 2*RUNTIME)
+{
+	uint64_t *ptr;
+	int i;
+	int fd;
+	unsigned long sz;
+	struct statfs filestat;
+
+	if (self->nthreads < 2 || self->n1 < 0 || self->n2 < 0)
+		SKIP(return, "Not enough threads or NUMA nodes available");
+
+	fd = memfd_create("tmp_hugetlb", MFD_HUGETLB);
+	if (fd < 0)
+		SKIP(return, "Couldn't create hugetlb memfd");
+
+	if (fstatfs(fd, &filestat) < 0)
+		SKIP(return, "Couldn't fstatfs hugetlb file");
+
+	sz = filestat.f_bsize;
+
+	if (ftruncate(fd, sz))
+		SKIP(return, "Couldn't allocate hugetlb pages");
+	ptr = mmap(NULL, sz, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+	if (ptr == MAP_FAILED)
+		SKIP(return, "Could not map hugetlb pages");
+
+	memset(ptr, 0xde, sz);
+	for (i = 0; i < self->nthreads - 1; i++)
+		if (pthread_create(&self->threads[i], NULL, access_mem, ptr))
+			perror("Couldn't create thread");
+
+	ASSERT_EQ(migrate(ptr, self->n1, self->n2, 10), 0);
+	for (i = 0; i < self->nthreads - 1; i++) {
+		ASSERT_EQ(pthread_cancel(self->threads[i]), 0);
+		pthread_join(self->threads[i], NULL);
+	}
+	ftruncate(fd, 0);
+	close(fd);
+}
+
 TEST_HARNESS_MAIN
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#121 lore raw

PATCH 45/46 selftests/vm: add hugetlb HGM test to migration selftest

In reply to James Houghton

This is mostly the same as the shared HugeTLB case, but instead of mapping the page with a regular page fault, we map it with lots of UFFDIO_CONTINUE operations. We also verify that the contents haven't changed after the migration, which would be the case if the post-migration PTEs pointed to the wrong page.

---
 tools/testing/selftests/vm/migration.c | 146 +++++++++++++++++++++++++
 1 file changed, 146 insertions(+)
tools/testing/selftests/vm/migration.c +146 -0
index 21577a84d7e4..1fb3607accab 100644
@@ -14,12 +14,21 @@
 #include <signal.h>
 #include <time.h>
 #include <sys/statfs.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
+#include <linux/userfaultfd.h>
+#include <sys/syscall.h>
+#include <fcntl.h>
 
 #define TWOMEG (2<<20)
 #define RUNTIME (60)
 
 #define ALIGN(x, a) (((x) + (a - 1)) & (~((a) - 1)))
 
+#ifndef MADV_SPLIT
+#define MADV_SPLIT 26
+#endif
+
 FIXTURE(migration)
 {
 	pthread_t *threads;
@@ -265,4 +274,141 @@ TEST_F_TIMEOUT(migration, shared_hugetlb, 2*RUNTIME)
 	close(fd);
 }
 
+#ifdef __NR_userfaultfd
+static int map_at_high_granularity(char *mem, size_t length)
+{
+	int i;
+	int ret;
+	int uffd = syscall(__NR_userfaultfd, 0);
+	struct uffdio_api api;
+	struct uffdio_register reg;
+	int pagesize = getpagesize();
+
+	if (uffd < 0) {
+		perror("couldn't create uffd");
+		return uffd;
+	}
+
+	api.api = UFFD_API;
+	api.features = 0;
+
+	ret = ioctl(uffd, UFFDIO_API, &api);
+	if (ret || api.api != UFFD_API) {
+		perror("UFFDIO_API failed");
+		goto out;
+	}
+
+	if (madvise(mem, length, MADV_SPLIT) == -1) {
+		perror("MADV_SPLIT failed");
+		goto out;
+	}
+
+	reg.range.start = (unsigned long)mem;
+	reg.range.len = length;
+
+	reg.mode = UFFDIO_REGISTER_MODE_MISSING | UFFDIO_REGISTER_MODE_MINOR;
+
+	ret = ioctl(uffd, UFFDIO_REGISTER, &reg);
+	if (ret) {
+		perror("UFFDIO_REGISTER failed");
+		goto out;
+	}
+
+	/* UFFDIO_CONTINUE each 4K segment of the 2M page. */
+	for (i = 0; i < length/pagesize; ++i) {
+		struct uffdio_continue cont;
+
+		cont.range.start = (unsigned long long)mem + i * pagesize;
+		cont.range.len = pagesize;
+		cont.mode = 0;
+		ret = ioctl(uffd, UFFDIO_CONTINUE, &cont);
+		if (ret) {
+			fprintf(stderr, "UFFDIO_CONTINUE failed "
+					"for %llx -> %llx: %d\n",
+					cont.range.start,
+					cont.range.start + cont.range.len,
+					errno);
+			goto out;
+		}
+	}
+	ret = 0;
+out:
+	close(uffd);
+	return ret;
+}
+#else
+static int map_at_high_granularity(char *mem, size_t length)
+{
+	fprintf(stderr, "Userfaultfd missing\n");
+	return -1;
+}
+#endif /* __NR_userfaultfd */
+
+/*
+ * Tests the high-granularity hugetlb migration entry paths.
+ */
+TEST_F_TIMEOUT(migration, shared_hugetlb_hgm, 2*RUNTIME)
+{
+	uint64_t *ptr;
+	int i;
+	int fd;
+	unsigned long sz;
+	struct statfs filestat;
+
+	if (self->nthreads < 2 || self->n1 < 0 || self->n2 < 0)
+		SKIP(return, "Not enough threads or NUMA nodes available");
+
+	fd = memfd_create("tmp_hugetlb", MFD_HUGETLB);
+	if (fd < 0)
+		SKIP(return, "Couldn't create hugetlb memfd");
+
+	if (fstatfs(fd, &filestat) < 0)
+		SKIP(return, "Couldn't fstatfs hugetlb file");
+
+	sz = filestat.f_bsize;
+
+	if (ftruncate(fd, sz))
+		SKIP(return, "Couldn't allocate hugetlb pages");
+
+	if (fallocate(fd, 0, 0, sz) < 0) {
+		perror("fallocate failed");
+		SKIP(return, "fallocate failed");
+	}
+
+	ptr = mmap(NULL, sz, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+	if (ptr == MAP_FAILED)
+		SKIP(return, "Could not allocate hugetlb pages");
+
+	/*
+	 * We have to map_at_high_granularity before we memset, otherwise
+	 * memset will map everything at the hugepage size.
+	 */
+	if (map_at_high_granularity((char *)ptr, sz) < 0)
+		SKIP(return, "Could not map HugeTLB range at high granularity");
+
+	/* Populate the page we're migrating. */
+	for (i = 0; i < sz/sizeof(*ptr); ++i)
+		ptr[i] = i;
+
+	for (i = 0; i < self->nthreads - 1; i++)
+		if (pthread_create(&self->threads[i], NULL, access_mem, ptr))
+			perror("Couldn't create thread");
+
+	ASSERT_EQ(migrate(ptr, self->n1, self->n2, 10), 0);
+	for (i = 0; i < self->nthreads - 1; i++) {
+		ASSERT_EQ(pthread_cancel(self->threads[i]), 0);
+		pthread_join(self->threads[i], NULL);
+	}
+
+	/* Check that the contents didnt' change. */
+	for (i = 0; i < sz/sizeof(*ptr); ++i) {
+		ASSERT_EQ(ptr[i], i);
+		if (ptr[i] != i)
+			break;
+	}
+
+	ftruncate(fd, 0);
+	close(fd);
+}
+
 TEST_HARNESS_MAIN
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#122 lore raw

PATCH 46/46 selftests/vm: add HGM UFFDIO_CONTINUE and hwpoison tests

In reply to James Houghton

This tests that high-granularity CONTINUEs at all sizes work (exercising contiguous PTE sizes for arm64, when support is added). This also tests that collapse works and hwpoison works correctly (although we aren't yet testing high-granularity poison).

This test uses UFFD_FEATURE_EVENT_FORK + UFFD_REGISTER_MODE_WP to force the kernel to copy page tables on fork(), exercising the changes to copy_hugetlb_page_range().

---
 tools/testing/selftests/vm/Makefile      |   1 +
 tools/testing/selftests/vm/hugetlb-hgm.c | 455 +++++++++++++++++++++++
 2 files changed, 456 insertions(+)
 create mode 100644 tools/testing/selftests/vm/hugetlb-hgm.c
tools/testing/selftests/vm/Makefile +1 -0
index 89c14e41bd43..4aa4ca75a471 100644
@@ -32,6 +32,7 @@ TEST_GEN_FILES += compaction_test
 TEST_GEN_FILES += gup_test
 TEST_GEN_FILES += hmm-tests
 TEST_GEN_FILES += hugetlb-madvise
+TEST_GEN_FILES += hugetlb-hgm
 TEST_GEN_FILES += hugepage-mmap
 TEST_GEN_FILES += hugepage-mremap
 TEST_GEN_FILES += hugepage-shm
tools/testing/selftests/vm/hugetlb-hgm.c new file +455 -0

455 lines not on this page · raw message

Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#123 lore raw
In reply to James Houghton
Quoted 6 lines · James Houghton

This series introduces the concept of HugeTLB high-granularity mapping (HGM). This series teaches HugeTLB how to map HugeTLB pages at high-granularity, similar to how THPs can be PTE-mapped.

Support for HGM in this series is for MAP_SHARED VMAs on x86 only. Other architectures and (some) support for MAP_PRIVATE will come later.

Why even care about the complexity of COW-sharable anon pages? TBH, I'd just limit this to MAP_SHARED and call it a day. Sure, we can come up with use cases for everything (snapshotting VMs using fork while also support optimized postcopy), but I think this would need some real justification for the added complexity and possible (likely!) issues.

Signature · 3 lines
Thanks,

David / dhildenb
#124 lore raw
Quoted 5 lines · David Hildenbrand
Quoted 4 lines · James Houghton

This series introduces the concept of HugeTLB high-granularity mapping (HGM). This series teaches HugeTLB how to map HugeTLB pages at high-granularity, similar to how THPs can be PTE-mapped.

Quoted 8 lines
Quoted 2 lines

Support for HGM in this series is for MAP_SHARED VMAs on x86 only. Other architectures and (some) support for MAP_PRIVATE will come later.

Why even care about the complexity of COW-sharable anon pages? TBH, I'd just limit this to MAP_SHARED and call it a day. Sure, we can come up with use cases for everything (snapshotting VMs using fork while also support optimized postcopy), but I think this would need some real justification for the added complexity and possible (likely!) issues.

I believe the primary use case driving this beyond MAP_SHARED would be poisoning due to memory errors. Extending HGM seems to be the most elegant way to start providing better support for this.

Signature · 1 line
Mike Kravetz
#125 lore raw
Quoted 15 lines · Mike Kravetz
Quoted 5 lines · David Hildenbrand
Quoted 4 lines · James Houghton

This series introduces the concept of HugeTLB high-granularity mapping (HGM). This series teaches HugeTLB how to map HugeTLB pages at high-granularity, similar to how THPs can be PTE-mapped.

Quoted 8 lines
Quoted 2 lines

Support for HGM in this series is for MAP_SHARED VMAs on x86 only. Other architectures and (some) support for MAP_PRIVATE will come later.

Why even care about the complexity of COW-sharable anon pages? TBH, I'd just limit this to MAP_SHARED and call it a day. Sure, we can come up with use cases for everything (snapshotting VMs using fork while also support optimized postcopy), but I think this would need some real justification for the added complexity and possible (likely!) issues.

Quoted 3 lines

I believe the primary use case driving this beyond MAP_SHARED would be poisoning due to memory errors. Extending HGM seems to be the most elegant way to start providing better support for this.

Good point. Although I wonder if in practice, most applicable users either already are, or should switch to, using MAP_SHARED hugetlb.

Signature · 3 lines
Thanks,

David / dhildenb