Skip to contentSkip to conversation

riscv: mm: Implement pmdp_collapse_flush for THP

PATCH Superseded Mayuresh Chitale 4 participants 8 messages Jan 25, 2023 – Jan 30, 2023 RISC-V

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 riscv: mm: Implement pmdp_collapse_flush for THP

When THP is enabled, 4K pages are collapsed into a single huge page using the generic pmdp_collapse_flush() which will further use flush_tlb_range() to shoot-down stale TLB entries. Unfortunately, the generic pmdp_collapse_flush() only invalidates cached leaf PTEs using address specific SFENCEs which results in repetitive (or unpredictable) page faults on RISC-V implementations which cache non-leaf PTEs.

Provide a RISC-V specific pmdp_collapse_flush() which ensures both cached leaf and non-leaf PTEs are invalidated by using non-address specific SFENCEs as recommended by the RISC-V privileged specification.

Fixes e88b333142e4 ("riscv: mm: add THP support on 64-bit")
---
 arch/riscv/include/asm/pgtable.h | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/include/asm/pgtable.h +24 -0
index 4eba9a98d0e3..6d948dec6020 100644
@@ -721,6 +721,30 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
 	return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd)));
 }
+
+#define pmdp_collapse_flush pmdp_collapse_flush
+static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
+					unsigned long address, pmd_t *pmdp)
+{
+	pmd_t pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
+
+	/*
+	 * When leaf PTE enteries (regular pages) are collapsed into a leaf
+	 * PMD entry (huge page), a valid non-leaf PTE is converted into a
+	 * valid leaf PTE at the level 1 page table. The RISC-V privileged v1.12
+	 * specification allows implementations to cache valid non-leaf PTEs,
+	 * but the section "4.2.1 Supervisor Memory-Management Fence
+	 * Instruction" recommends the following:
+	 * "If software modifies a non-leaf PTE, it should execute SFENCE.VMA
+	 * with rs1=x0. If any PTE along the traversal path had its G bit set,
+	 * rs2 must be x0; otherwise, rs2 should be set to the ASID for which
+	 * the translation is being modified."
+	 * Based on the above recommendation, we should do full flush whenever
+	 * leaf PTE entries are collapsed into a leaf PMD entry.
+	 */
+	flush_tlb_mm(vma->vm_mm);
+	return pmd;
+}
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 
 /*
Signature · 7 lines
2.34.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#1 lore raw
Quoted 23 lines · Mayuresh Chitale

When THP is enabled, 4K pages are collapsed into a single huge page using the generic pmdp_collapse_flush() which will further use flush_tlb_range() to shoot-down stale TLB entries. Unfortunately, the generic pmdp_collapse_flush() only invalidates cached leaf PTEs using address specific SFENCEs which results in repetitive (or unpredictable) page faults on RISC-V implementations which cache non-leaf PTEs.

Provide a RISC-V specific pmdp_collapse_flush() which ensures both cached leaf and non-leaf PTEs are invalidated by using non-address specific SFENCEs as recommended by the RISC-V privileged specification.

Fixes e88b333142e4 ("riscv: mm: add THP support on 64-bit")
---
 arch/riscv/include/asm/pgtable.h | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/include/asm/pgtable.h +0 -0
index 4eba9a98d0e3..6d948dec6020 100644

arch/riscv/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 729

Quoted 9 lines
arch/riscv/include/asm/pgtable.h +6 -0
@@ -721,6 +721,30 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
 	return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd)));
 }
+
+#define pmdp_collapse_flush pmdp_collapse_flush
+static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
+					unsigned long address, pmd_t *pmdp)
+{
+	pmd_t pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);

The generic version of this function has a couple sanity checks for kernels compiled with CONFIG_DEBUG_VM. Shouldn't we duplicate those?

Thanks,
drew

Quoted 24 lines
arch/riscv/include/asm/pgtable.h +18 -0
+
+	/*
+	 * When leaf PTE enteries (regular pages) are collapsed into a leaf
+	 * PMD entry (huge page), a valid non-leaf PTE is converted into a
+	 * valid leaf PTE at the level 1 page table. The RISC-V privileged v1.12
+	 * specification allows implementations to cache valid non-leaf PTEs,
+	 * but the section "4.2.1 Supervisor Memory-Management Fence
+	 * Instruction" recommends the following:
+	 * "If software modifies a non-leaf PTE, it should execute SFENCE.VMA
+	 * with rs1=x0. If any PTE along the traversal path had its G bit set,
+	 * rs2 must be x0; otherwise, rs2 should be set to the ASID for which
+	 * the translation is being modified."
+	 * Based on the above recommendation, we should do full flush whenever
+	 * leaf PTE entries are collapsed into a leaf PMD entry.
+	 */
+	flush_tlb_mm(vma->vm_mm);
+	return pmd;
+}
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 
 /*
Signature · 1 line
2.34.1

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#2 lore raw
Quoted 35 lines · Andrew Jones
Quoted 34 lines · Mayuresh Chitale

When THP is enabled, 4K pages are collapsed into a single huge page using the generic pmdp_collapse_flush() which will further use flush_tlb_range() to shoot-down stale TLB entries. Unfortunately,
the generic pmdp_collapse_flush() only invalidates cached leaf PTEs using address specific SFENCEs which results in repetitive (or unpredictable) page faults on RISC-V implementations which cache non-leaf PTEs.

Provide a RISC-V specific pmdp_collapse_flush() which ensures both cached leaf and non-leaf PTEs are invalidated by using non-address specific SFENCEs as recommended by the RISC-V privileged specification.

Fixes e88b333142e4 ("riscv: mm: add THP support on 64-bit")
---
 arch/riscv/include/asm/pgtable.h | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/include/asm/pgtable.h +3 -0
index 4eba9a98d0e3..6d948dec6020 100644
@@ -721,6 +721,30 @@ static inline pmd_t pmdp_establish(struct
 vm_area_struct *vma,
 	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
 	return __pmd(atomic_long_xchg((atomic_long_t *)pmdp,
 pmd_val(pmd)));
 }
+
+#define pmdp_collapse_flush pmdp_collapse_flush
+static inline pmd_t pmdp_collapse_flush(struct vm_area_struct
Quoted 8 lines

arch/riscv/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 729

Quoted 5 lines
arch/riscv/include/asm/pgtable.h +1 -0
 *vma,
+					unsigned long address, pmd_t

*pmdp)

arch/riscv/include/asm/pgtable.h +2 -0
+{
+	pmd_t pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);

The generic version of this function has a couple sanity checks for kernels compiled with CONFIG_DEBUG_VM. Shouldn't we duplicate those?

Ok. I will add those checks too.

Quoted 32 lines

Thanks,
drew

Quoted 28 lines
arch/riscv/include/asm/pgtable.h +3 -0
+
+	/*
+	 * When leaf PTE enteries (regular pages) are collapsed into a

leaf

arch/riscv/include/asm/pgtable.h +1 -0
+	 * PMD entry (huge page), a valid non-leaf PTE is converted

into a

arch/riscv/include/asm/pgtable.h +1 -0
+	 * valid leaf PTE at the level 1 page table. The RISC-V

privileged v1.12

arch/riscv/include/asm/pgtable.h +1 -0
+	 * specification allows implementations to cache valid non-leaf 

PTEs,

arch/riscv/include/asm/pgtable.h +3 -0
+	 * but the section "4.2.1 Supervisor Memory-Management Fence
+	 * Instruction" recommends the following:
+	 * "If software modifies a non-leaf PTE, it should execute

SFENCE.VMA

arch/riscv/include/asm/pgtable.h +1 -0
+	 * with rs1=x0. If any PTE along the traversal path had its G

bit set,

arch/riscv/include/asm/pgtable.h +1 -0
+	 * rs2 must be x0; otherwise, rs2 should be set to the ASID for

which

arch/riscv/include/asm/pgtable.h +2 -0
+	 * the translation is being modified."
+	 * Based on the above recommendation, we should do full flush

whenever

arch/riscv/include/asm/pgtable.h +5 -0
+	 * leaf PTE entries are collapsed into a leaf PMD entry.
+	 */
+	flush_tlb_mm(vma->vm_mm);
+	return pmd;
+}
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
Quoted 9 lines
Quoted 4 lines

/*

Signature · 1 line
2.34.1

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#3 lore raw
In reply to Mayuresh Chitale

Hi Mayuresh,

Quoted 7 lines · Mayuresh Chitale

When THP is enabled, 4K pages are collapsed into a single huge page using the generic pmdp_collapse_flush() which will further use flush_tlb_range() to shoot-down stale TLB entries. Unfortunately, the generic pmdp_collapse_flush() only invalidates cached leaf PTEs using address specific SFENCEs which results in repetitive (or unpredictable) page faults on RISC-V implementations which cache non-leaf PTEs.

That's interesting! I'm wondering if the same issue will happen if a user maps 4K, unmaps it and at the same address maps a 2MB hugepage: I'm not sure the mm code would correctly flush the non-leaf PTE when unmapping the 4KB page. In that case, your patch only fixes the THP usecase and maybe we should try to catch this non-leaf -> leaf upgrade at some lower level page table functions, what do you think?

Alex

Quoted 35 lines

Provide a RISC-V specific pmdp_collapse_flush() which ensures both cached leaf and non-leaf PTEs are invalidated by using non-address specific SFENCEs as recommended by the RISC-V privileged specification.

Fixes e88b333142e4 ("riscv: mm: add THP support on 64-bit")
---
  arch/riscv/include/asm/pgtable.h | 24 ++++++++++++++++++++++++
  1 file changed, 24 insertions(+)
arch/riscv/include/asm/pgtable.h +17 -0
index 4eba9a98d0e3..6d948dec6020 100644
@@ -721,6 +721,30 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
  	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
  	return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd)));
  }
+
+#define pmdp_collapse_flush pmdp_collapse_flush
+static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
+					unsigned long address, pmd_t *pmdp)
+{
+	pmd_t pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
+
+	/*
+	 * When leaf PTE enteries (regular pages) are collapsed into a leaf
+	 * PMD entry (huge page), a valid non-leaf PTE is converted into a
+	 * valid leaf PTE at the level 1 page table. The RISC-V privileged v1.12
+	 * specification allows implementations to cache valid non-leaf PTEs,
+	 * but the section "4.2.1 Supervisor Memory-Management Fence
+	 * Instruction" recommends the following:
+	 * "If software modifies a non-leaf PTE, it should execute SFENCE.VMA
+	 * with rs1=x0. If any PTE along the traversal path had its G bit set,
+	 * rs2 must be x0; otherwise, rs2 should be set to the ASID for which

arch/riscv/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 750

Quoted 10 lines
arch/riscv/include/asm/pgtable.h +7 -0
+	 * the translation is being modified."
+	 * Based on the above recommendation, we should do full flush whenever
+	 * leaf PTE entries are collapsed into a leaf PMD entry.
+	 */
+	flush_tlb_mm(vma->vm_mm);
+	return pmd;
+}
  #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 
  /*

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#4 lore raw
Quoted 13 lines · Alexandre Ghiti

Hi Mayuresh,

Quoted 7 lines · Mayuresh Chitale

When THP is enabled, 4K pages are collapsed into a single huge page using the generic pmdp_collapse_flush() which will further use flush_tlb_range() to shoot-down stale TLB entries. Unfortunately, the generic pmdp_collapse_flush() only invalidates cached leaf PTEs using address specific SFENCEs which results in repetitive (or unpredictable) page faults on RISC-V implementations which cache non-leaf PTEs.

Quoted 6 lines

That's interesting! I'm wondering if the same issue will happen if a user maps 4K, unmaps it and at the same address maps a 2MB hugepage: I'm not sure the mm code would correctly flush the non-leaf PTE when unmapping the 4KB page. In that case, your patch only fixes the THP usecase and maybe we should try to catch this non-leaf -> leaf upgrade at some lower level page table functions, what do you think?

This issue can happen whenever existing/valid non-leaf PTE is modified.

We hit this issue in the THP case but we can also hit this issue in other scenarios where the page table programming pattern is similar.

Regards,
Anup

Quoted 48 lines

Alex

Quoted 44 lines

Provide a RISC-V specific pmdp_collapse_flush() which ensures both cached leaf and non-leaf PTEs are invalidated by using non-address specific SFENCEs as recommended by the RISC-V privileged specification.

Fixes e88b333142e4 ("riscv: mm: add THP support on 64-bit")
---
  arch/riscv/include/asm/pgtable.h | 24 ++++++++++++++++++++++++
  1 file changed, 24 insertions(+)
arch/riscv/include/asm/pgtable.h +24 -0
index 4eba9a98d0e3..6d948dec6020 100644
@@ -721,6 +721,30 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
      page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
      return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd)));
  }
+
+#define pmdp_collapse_flush pmdp_collapse_flush
+static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
+                                     unsigned long address, pmd_t *pmdp)
+{
+     pmd_t pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
+
+     /*
+      * When leaf PTE enteries (regular pages) are collapsed into a leaf
+      * PMD entry (huge page), a valid non-leaf PTE is converted into a
+      * valid leaf PTE at the level 1 page table. The RISC-V privileged v1.12
+      * specification allows implementations to cache valid non-leaf PTEs,
+      * but the section "4.2.1 Supervisor Memory-Management Fence
+      * Instruction" recommends the following:
+      * "If software modifies a non-leaf PTE, it should execute SFENCE.VMA
+      * with rs1=x0. If any PTE along the traversal path had its G bit set,
+      * rs2 must be x0; otherwise, rs2 should be set to the ASID for which
+      * the translation is being modified."
+      * Based on the above recommendation, we should do full flush whenever
+      * leaf PTE entries are collapsed into a leaf PMD entry.
+      */
+     flush_tlb_mm(vma->vm_mm);
+     return pmd;
+}
  #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 
Quoted 1 line

arch/riscv/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 750

Quoted 1 line
arch/riscv/include/asm/pgtable.h +0 -0
  /*

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#5 lore raw
Quoted 18 lines · Anup Patel
Quoted 11 lines · Alexandre Ghiti

Hi Mayuresh,

Quoted 7 lines · Mayuresh Chitale

When THP is enabled, 4K pages are collapsed into a single huge page using the generic pmdp_collapse_flush() which will further use flush_tlb_range() to shoot-down stale TLB entries. Unfortunately, the generic pmdp_collapse_flush() only invalidates cached leaf PTEs using address specific SFENCEs which results in repetitive (or unpredictable) page faults on RISC-V implementations which cache non-leaf PTEs.

Quoted 6 lines

That's interesting! I'm wondering if the same issue will happen if a user maps 4K, unmaps it and at the same address maps a 2MB hugepage: I'm not sure the mm code would correctly flush the non-leaf PTE when unmapping the 4KB page. In that case, your patch only fixes the THP usecase and maybe we should try to catch this non-leaf -> leaf upgrade at some lower level page table functions, what do you think?

Quoted 4 lines

This issue can happen whenever existing/valid non-leaf PTE is modified.

We hit this issue in the THP case but we can also hit this issue in other scenarios where the page table programming pattern is similar.

Then what about trying to get all those cases at once? We can easily catch those spurious page faults as the pte would be valid: we would just have to flush_tlb_mm at the first page fault, if any.

Quoted 51 lines

Regards,
Anup

Quoted 47 lines

Alex

Quoted 44 lines

Provide a RISC-V specific pmdp_collapse_flush() which ensures both cached leaf and non-leaf PTEs are invalidated by using non-address specific SFENCEs as recommended by the RISC-V privileged specification.

Fixes e88b333142e4 ("riscv: mm: add THP support on 64-bit")
---
   arch/riscv/include/asm/pgtable.h | 24 ++++++++++++++++++++++++
   1 file changed, 24 insertions(+)
arch/riscv/include/asm/pgtable.h +24 -0
index 4eba9a98d0e3..6d948dec6020 100644
@@ -721,6 +721,30 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
       page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
       return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd)));
   }
+
+#define pmdp_collapse_flush pmdp_collapse_flush
+static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
+                                     unsigned long address, pmd_t *pmdp)
+{
+     pmd_t pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
+
+     /*
+      * When leaf PTE enteries (regular pages) are collapsed into a leaf
+      * PMD entry (huge page), a valid non-leaf PTE is converted into a
+      * valid leaf PTE at the level 1 page table. The RISC-V privileged v1.12
+      * specification allows implementations to cache valid non-leaf PTEs,
+      * but the section "4.2.1 Supervisor Memory-Management Fence
+      * Instruction" recommends the following:
+      * "If software modifies a non-leaf PTE, it should execute SFENCE.VMA
+      * with rs1=x0. If any PTE along the traversal path had its G bit set,
+      * rs2 must be x0; otherwise, rs2 should be set to the ASID for which
+      * the translation is being modified."
+      * Based on the above recommendation, we should do full flush whenever
+      * leaf PTE entries are collapsed into a leaf PMD entry.
+      */
+     flush_tlb_mm(vma->vm_mm);
+     return pmd;
+}
   #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 
Quoted 5 lines
Quoted 1 line

arch/riscv/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 750

Quoted 1 line
arch/riscv/include/asm/pgtable.h +0 -0
   /*

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#6 lore raw
Quoted 30 lines · Alexandre Ghiti
Quoted 29 lines · Anup Patel
Quoted 14 lines · Alexandre Ghiti

Hi Mayuresh,

Quoted 10 lines · Mayuresh Chitale

When THP is enabled, 4K pages are collapsed into a single huge page using the generic pmdp_collapse_flush() which will further use flush_tlb_range() to shoot-down stale TLB entries. Unfortunately,
the generic pmdp_collapse_flush() only invalidates cached leaf PTEs
using address specific SFENCEs which results in repetitive (or unpredictable) page faults on RISC-V implementations which cache
non-leaf PTEs.

Quoted 10 lines

That's interesting! I'm wondering if the same issue will happen if a
user maps 4K, unmaps it and at the same address maps a 2MB
hugepage: I'm
not sure the mm code would correctly flush the non-leaf PTE when unmapping the 4KB page. In that case, your patch only fixes the THP
usecase and maybe we should try to catch this non-leaf -> leaf upgrade
at some lower level page table functions, what do you think?

This issue can happen whenever existing/valid non-leaf PTE is modified.

Quoted 7 lines
Quoted 3 lines

We hit this issue in the THP case but we can also hit this issue in other
scenarios where the page table programming pattern is similar.

Then what about trying to get all those cases at once? We can easily catch those spurious page faults as the pte would be valid: we would just have to flush_tlb_mm at the first page fault, if any.

IMO, we can consider fixing those cases in a separate patch.

Quoted 78 lines
Quoted 72 lines

Regards,
Anup

Quoted 67 lines

Alex

Quoted 64 lines

Provide a RISC-V specific pmdp_collapse_flush() which ensures both
cached leaf and non-leaf PTEs are invalidated by using non- address
specific SFENCEs as recommended by the RISC-V privileged specification.

Fixes e88b333142e4 ("riscv: mm: add THP support on 64-bit")
---
   arch/riscv/include/asm/pgtable.h | 24
++++++++++++++++++++++++
   1 file changed, 24 insertions(+)
arch/riscv/include/asm/pgtable.h +3 -0
index 4eba9a98d0e3..6d948dec6020 100644
@@ -721,6 +721,30 @@ static inline pmd_t pmdp_establish(struct
 vm_area_struct *vma,
       page_table_check_pmd_set(vma->vm_mm, address, pmdp,
 pmd);
       return __pmd(atomic_long_xchg((atomic_long_t *)pmdp,
 pmd_val(pmd)));
   }
+
+#define pmdp_collapse_flush pmdp_collapse_flush
+static inline pmd_t pmdp_collapse_flush(struct vm_area_struct

*vma,
+ unsigned long address,
pmd_t *pmdp)

arch/riscv/include/asm/pgtable.h +2 -0
+{
+     pmd_t pmd = pmdp_huge_get_and_clear(vma->vm_mm, address,

pmdp);

arch/riscv/include/asm/pgtable.h +3 -0
+
+     /*
+      * When leaf PTE enteries (regular pages) are collapsed

into a leaf
+ * PMD entry (huge page), a valid non-leaf PTE is
converted into a
+ * valid leaf PTE at the level 1 page table. The RISC-V
privileged v1.12
+ * specification allows implementations to cache valid
non-leaf PTEs,
+ * but the section "4.2.1 Supervisor Memory-Management
Fence

arch/riscv/include/asm/pgtable.h +2 -0
+      * Instruction" recommends the following:
+      * "If software modifies a non-leaf PTE, it should

execute SFENCE.VMA
+ * with rs1=x0. If any PTE along the traversal path had
its G bit set,
+ * rs2 must be x0; otherwise, rs2 should be set to the
ASID for which

arch/riscv/include/asm/pgtable.h +2 -0
+      * the translation is being modified."
+      * Based on the above recommendation, we should do full

flush whenever

arch/riscv/include/asm/pgtable.h +5 -0
+      * leaf PTE entries are collapsed into a leaf PMD entry.
+      */
+     flush_tlb_mm(vma->vm_mm);
+     return pmd;
+}
   #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
Quoted 5 lines
Quoted 1 line
Quoted 1 line
   /*

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

Quoted 4 lines

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#7 lore raw
In reply to Alexandre Ghiti
Quoted 12 lines · Alexandre Ghiti

Hi Mayuresh,

Quoted 8 lines · Mayuresh Chitale

When THP is enabled, 4K pages are collapsed into a single huge page using the generic pmdp_collapse_flush() which will further use flush_tlb_range() to shoot-down stale TLB entries. Unfortunately,
the generic pmdp_collapse_flush() only invalidates cached leaf PTEs using address specific SFENCEs which results in repetitive (or unpredictable) page faults on RISC-V implementations which cache non-leaf PTEs.

Quoted 8 lines

That's interesting! I'm wondering if the same issue will happen if a user maps 4K, unmaps it and at the same address maps a 2MB hugepage:
I'm
not sure the mm code would correctly flush the non-leaf PTE when unmapping the 4KB page. In that case, your patch only fixes the THP usecase and maybe we should try to catch this non-leaf -> leaf upgrade
at some lower level page table functions, what do you think?

I will look into it but I dont know how to reproduce the issue without the THP use case. It would be great if you could share the test case or test code to reproduce it.

Quoted 62 lines

Alex

Quoted 58 lines

Provide a RISC-V specific pmdp_collapse_flush() which ensures both cached leaf and non-leaf PTEs are invalidated by using non-address specific SFENCEs as recommended by the RISC-V privileged specification.

Fixes e88b333142e4 ("riscv: mm: add THP support on 64-bit")
---
  arch/riscv/include/asm/pgtable.h | 24 ++++++++++++++++++++++++
  1 file changed, 24 insertions(+)
arch/riscv/include/asm/pgtable.h +4 -0
index 4eba9a98d0e3..6d948dec6020 100644
@@ -721,6 +721,30 @@ static inline pmd_t pmdp_establish(struct
 vm_area_struct *vma,
  	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
  	return __pmd(atomic_long_xchg((atomic_long_t *)pmdp,
 pmd_val(pmd)));
  }
+
+#define pmdp_collapse_flush pmdp_collapse_flush
+static inline pmd_t pmdp_collapse_flush(struct vm_area_struct
 *vma,
+					unsigned long address, pmd_t

*pmdp)

arch/riscv/include/asm/pgtable.h +5 -0
+{
+	pmd_t pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
+
+	/*
+	 * When leaf PTE enteries (regular pages) are collapsed into a

leaf

arch/riscv/include/asm/pgtable.h +1 -0
+	 * PMD entry (huge page), a valid non-leaf PTE is converted

into a

arch/riscv/include/asm/pgtable.h +1 -0
+	 * valid leaf PTE at the level 1 page table. The RISC-V

privileged v1.12

arch/riscv/include/asm/pgtable.h +1 -0
+	 * specification allows implementations to cache valid non-leaf 

PTEs,

arch/riscv/include/asm/pgtable.h +3 -0
+	 * but the section "4.2.1 Supervisor Memory-Management Fence
+	 * Instruction" recommends the following:
+	 * "If software modifies a non-leaf PTE, it should execute

SFENCE.VMA

arch/riscv/include/asm/pgtable.h +1 -0
+	 * with rs1=x0. If any PTE along the traversal path had its G

bit set,

arch/riscv/include/asm/pgtable.h +1 -0
+	 * rs2 must be x0; otherwise, rs2 should be set to the ASID for

which

arch/riscv/include/asm/pgtable.h +2 -0
+	 * the translation is being modified."
+	 * Based on the above recommendation, we should do full flush

whenever

arch/riscv/include/asm/pgtable.h +5 -0
+	 * leaf PTE entries are collapsed into a leaf PMD entry.
+	 */
+	flush_tlb_mm(vma->vm_mm);
+	return pmd;
+}
  #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
Quoted 6 lines
Quoted 1 line

/*

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv