Skip to contentSkip to conversation

Fixes for contiguous hugetlb

PATCH New Steve Capper 2 participants 4 messages Sep 21, 2018 – Sep 24, 2018 arm64

lore
Share & tools
lore
#0 lore raw

v2 cover Fixes for contiguous hugetlb

This patch series addresses a couple of issues found with contiguous hugetlb.

1) It is possible for old hugetlb's to be created by remove_migration_pte, the first patch adjusts the code to capture any young ptes that could have been created by hardware dirty bit management.

2) huge_ptep_set_access_flags has been changed in the second patch to prevent it from unnecessarily invalidating contiguous hugetlb mappings (which also stops it from potentially evoking back-to-back page faults).

These patches have not yet been tested on hardware.

Steve Capper (2):
arm64: hugetlb: Fix handling of young ptes
arm64: hugetlb: Avoid unnecessary clearing in
huge_ptep_set_access_flags

 arch/arm64/mm/hugetlbpage.c | 50 ++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 43 insertions(+), 7 deletions(-)
Signature · 1 line
2.11.0
#1 lore raw

PATCH v2 1/2 arm64: hugetlb: Fix handling of young ptes

In the contiguous bit hugetlb break-before-make code we assume that all hugetlb pages are young.

In fact, remove_migration_pte is able to place an old hugetlb pte so this assumption is not valid.

This patch fixes the contiguous hugetlb scanning code to preserve young ptes.

Fixes d8bdcff28764 ("arm64: hugetlb: Add break-before-make logic for contiguous entries")
---
 arch/arm64/mm/hugetlbpage.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
arch/arm64/mm/hugetlbpage.c +9 -3
index 192b3ba07075..f85be2f8b140 100644
@@ -117,11 +117,14 @@ static pte_t get_clear_flush(struct mm_struct *mm,
 
 		/*
 		 * If HW_AFDBM is enabled, then the HW could turn on
-		 * the dirty bit for any page in the set, so check
-		 * them all.  All hugetlb entries are already young.
+		 * the dirty or accessed bit for any page in the set,
+		 * so check them all.
 		 */
 		if (pte_dirty(pte))
 			orig_pte = pte_mkdirty(orig_pte);
+
+		if (pte_young(pte))
+			orig_pte = pte_mkyoung(orig_pte);
 	}
 
 	if (valid) {
@@ -340,10 +343,13 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
 	if (!pte_same(orig_pte, pte))
 		changed = 1;
 
-	/* Make sure we don't lose the dirty state */
+	/* Make sure we don't lose the dirty or young state */
 	if (pte_dirty(orig_pte))
 		pte = pte_mkdirty(pte);
 
+	if (pte_young(orig_pte))
+		pte = pte_mkyoung(pte);
+
 	hugeprot = pte_pgprot(pte);
 	for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn)
 		set_pte_at(vma->vm_mm, addr, ptep, pfn_pte(pfn, hugeprot));
Signature · 1 line
2.11.0
#2 lore raw

PATCH v2 2/2 arm64: hugetlb: Avoid unnecessary clearing in huge_ptep_set_access_flags

In reply to Steve Capper

For contiguous hugetlb, huge_ptep_set_access_flags performs a get_clear_flush (which then flushes the TLBs) even when no change of ptes is necessary.

Unfortunately, this behaviour can lead to back-to-back page faults being generated when running with multiple threads that access the same contiguous huge page.

Thread 1 | Thread 2
-----------------------------+------------------------------

hugetlb_fault                |
huge_ptep_set_access_flags   |
  -> invalidate pte range    | hugetlb_fault
continue processing          | wait for hugetlb_fault_mutex
release mutex and return     | huge_ptep_set_access_flags
                             |   -> invalidate pte range

hugetlb_fault
...

This patch changes huge_ptep_set_access_flags s.t. we first read the contiguous range of ptes (whilst preserving dirty information); the pte range is only then invalidated where necessary and this prevents further spurious page faults.

Fixes d8bdcff28764 ("arm64: hugetlb: Add break-before-make logic for contiguous entries")

---
Changed in v2: we just check dirty, young for the contiguous range and writable for the first pte in the range.

---
 arch/arm64/mm/hugetlbpage.c | 38 ++++++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)
arch/arm64/mm/hugetlbpage.c +34 -4
index f85be2f8b140..f58ea503ad01 100644
@@ -323,11 +323,40 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
 	return get_clear_flush(mm, addr, ptep, pgsize, ncontig);
 }
 
+/*
+ * huge_ptep_set_access_flags will update access flags (dirty, accesssed)
+ * and write permission.
+ *
+ * For a contiguous huge pte range we need to check whether or not write
+ * permission has to change only on the first pte in the set. Then for
+ * all the contiguous ptes we need to check whether or not there is a
+ * discrepancy between dirty or young.
+ */
+static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig)
+{
+	int i;
+
+	if (pte_write(pte) != pte_write(huge_ptep_get(ptep)))
+		return 1;
+
+	for (i = 0; i < ncontig; i++) {
+		pte_t orig_pte = huge_ptep_get(ptep + i);
+
+		if (pte_dirty(pte) != pte_dirty(orig_pte))
+			return 1;
+
+		if (pte_young(pte) != pte_young(orig_pte))
+			return 1;
+	}
+
+	return 0;
+}
+
 int huge_ptep_set_access_flags(struct vm_area_struct *vma,
 			       unsigned long addr, pte_t *ptep,
 			       pte_t pte, int dirty)
 {
-	int ncontig, i, changed = 0;
+	int ncontig, i;
 	size_t pgsize = 0;
 	unsigned long pfn = pte_pfn(pte), dpfn;
 	pgprot_t hugeprot;
@@ -339,9 +368,10 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
 	ncontig = find_num_contig(vma->vm_mm, addr, ptep, &pgsize);
 	dpfn = pgsize >> PAGE_SHIFT;
 
+	if (!__cont_access_flags_changed(ptep, pte, ncontig))
+		return 0;
+
 	orig_pte = get_clear_flush(vma->vm_mm, addr, ptep, pgsize, ncontig);
-	if (!pte_same(orig_pte, pte))
-		changed = 1;
 
 	/* Make sure we don't lose the dirty or young state */
 	if (pte_dirty(orig_pte))
@@ -354,7 +384,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
 	for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn)
 		set_pte_at(vma->vm_mm, addr, ptep, pfn_pte(pfn, hugeprot));
 
-	return changed;
+	return 1;
 }
 
 void huge_ptep_set_wrprotect(struct mm_struct *mm,
Signature · 1 line
2.11.0
#3 lore raw

v2 cover Fixes for contiguous hugetlb

In reply to Steve Capper
Quoted 9 lines · Steve Capper

This patch series addresses a couple of issues found with contiguous hugetlb.

1) It is possible for old hugetlb's to be created by remove_migration_pte, the first patch adjusts the code to capture any young ptes that could have been created by hardware dirty bit management.

2) huge_ptep_set_access_flags has been changed in the second patch to prevent it from unnecessarily invalidating contiguous hugetlb mappings (which also stops it from potentially evoking back-to-back page faults).

Thanks Steve, these look good to me. I'll pick them up as fixes.

Will