Skip to contentSkip to conversation

riscv: support fast gup

PATCH Applied Jisheng Zhang 5 participants 19 messages Dec 19, 2023 – Feb 7, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover riscv: support fast gup

This series adds fast gup support to riscv.

The First patch fixes a bug in __p*d_free_tlb(). Per the riscv privileged spec, if non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must.

The 2nd patch is a preparation patch.

The last two patches do the real work:
In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.

riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the riscv_ipi_for_rfence is true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually the riscv_ipi_for_rfence is false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h for more details.

This patch enables MMU_GATHER_RCU_TABLE_FREE, then use

*tlb_remove_page_ptdesc() for those platforms which use IPI to perform TLB shootdown;

*tlb_remove_ptdesc() for those platforms which use SBI to perform TLB shootdown;

Both case mean that disabling interrupts will block the free and protect the fast gup page walker.

So after the 3rd patch, everything is well prepared, let's select HAVE_FAST_GUP if MMU.

Jisheng Zhang (4):
riscv: tlb: fix __p*d_free_tlb()
riscv: tlb: convert __p*d_free_tlb() to inline functions
riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU
riscv: enable HAVE_FAST_GUP if MMU

 arch/riscv/Kconfig               |  2 ++
 arch/riscv/include/asm/pgalloc.h | 53 +++++++++++++++++++++++++++-----
 arch/riscv/include/asm/pgtable.h |  6 ++++
 arch/riscv/include/asm/tlb.h     | 18 +++++++++++
 4 files changed, 71 insertions(+), 8 deletions(-)
Signature · 1 line
2.40.0
#1 lore raw

PATCH 1/4 riscv: tlb: fix __p*d_free_tlb()

If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.

---
 arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
arch/riscv/include/asm/pgalloc.h +17 -3
index d169a4f41a2e..a12fb83fa1f5 100644
@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
 		__pud_free(mm, pud);
 }
 
-#define __pud_free_tlb(tlb, pud, addr)  pud_free((tlb)->mm, pud)
+#define __pud_free_tlb(tlb, pud, addr)					\
+do {									\
+	if (pgtable_l4_enabled) {					\
+		pagetable_pud_dtor(virt_to_ptdesc(pud));		\
+		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud));	\
+	}								\
+} while (0)
 
 #define p4d_alloc_one p4d_alloc_one
 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
 		__p4d_free(mm, p4d);
 }
 
-#define __p4d_free_tlb(tlb, p4d, addr)  p4d_free((tlb)->mm, p4d)
+#define __p4d_free_tlb(tlb, p4d, addr)					\
+do {									\
+	if (pgtable_l5_enabled)						\
+		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d));	\
+} while (0)
 #endif /* __PAGETABLE_PMD_FOLDED */
 
 static inline void sync_kernel_mappings(pgd_t *pgd)
@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 
 #ifndef __PAGETABLE_PMD_FOLDED
 
-#define __pmd_free_tlb(tlb, pmd, addr)  pmd_free((tlb)->mm, pmd)
+#define __pmd_free_tlb(tlb, pmd, addr)				\
+do {								\
+	pagetable_pmd_dtor(virt_to_ptdesc(pmd));		\
+	tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd));	\
+} while (0)
 
 #endif /* __PAGETABLE_PMD_FOLDED */
 
Signature · 1 line
2.40.0
#2 lore raw

Re: [PATCH 1/4] riscv: tlb: fix __p*d_free_tlb()

Hi Jisheng,

Quoted 15 lines · Jisheng Zhang

If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.

---
  arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++---
  1 file changed, 17 insertions(+), 3 deletions(-)
arch/riscv/include/asm/pgalloc.h +0 -0
index d169a4f41a2e..a12fb83fa1f5 100644

arch/riscv/include/asm/pgalloc.h static inline void pud_free(struct mm_struct *mm, pud_t *pud) · line 102

Quoted 9 lines
arch/riscv/include/asm/pgalloc.h +5 -1
@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
  		__pud_free(mm, pud);
  }
 
-#define __pud_free_tlb(tlb, pud, addr)  pud_free((tlb)->mm, pud)
+#define __pud_free_tlb(tlb, pud, addr)					\
+do {									\
+	if (pgtable_l4_enabled) {					\
+		pagetable_pud_dtor(virt_to_ptdesc(pud));		\
+		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud));	\

The specification indeed states that an sfence.vma must be emitted after a page directory modification. Your change is not enough though since eventually tlb_flush() is called and in this function we should add:

if (tlb->freed_tables)
    tlb_flush_mm();

otherwise we are not guaranteed that a "global" sfence.vma is called.

Would you be able to benchmark this change and see the performance impact?

Thanks,

Alex

Quoted 31 lines
arch/riscv/include/asm/pgalloc.h +12 -2
+	}								\
+} while (0)
 
  #define p4d_alloc_one p4d_alloc_one
  static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
  		__p4d_free(mm, p4d);
  }
 
-#define __p4d_free_tlb(tlb, p4d, addr)  p4d_free((tlb)->mm, p4d)
+#define __p4d_free_tlb(tlb, p4d, addr)					\
+do {									\
+	if (pgtable_l5_enabled)						\
+		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d));	\
+} while (0)
  #endif /* __PAGETABLE_PMD_FOLDED */
 
  static inline void sync_kernel_mappings(pgd_t *pgd)
@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 
  #ifndef __PAGETABLE_PMD_FOLDED
 
-#define __pmd_free_tlb(tlb, pmd, addr)  pmd_free((tlb)->mm, pmd)
+#define __pmd_free_tlb(tlb, pmd, addr)				\
+do {								\
+	pagetable_pmd_dtor(virt_to_ptdesc(pmd));		\
+	tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd));	\
+} while (0)
 
  #endif /* __PAGETABLE_PMD_FOLDED */
 
reviewer Reviewed-by
#3 lore raw

Re: [PATCH 1/4] riscv: tlb: fix __p*d_free_tlb()

Hi Jisheng,

Quoted 31 lines · Alexandre Ghiti

Hi Jisheng,

Quoted 15 lines · Jisheng Zhang

If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.

---
  arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++---

  1 file changed, 17 insertions(+), 3 deletions(-)
arch/riscv/include/asm/pgalloc.h +0 -0
index d169a4f41a2e..a12fb83fa1f5 100644

arch/riscv/include/asm/pgalloc.h static inline void pud_free(struct mm_struct *mm, · line 103

Quoted 9 lines
arch/riscv/include/asm/pgalloc.h +5 -0
@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, 
 pud_t *pud)
           __pud_free(mm, pud);
   }
   -#define __pud_free_tlb(tlb, pud, addr) pud_free((tlb)->mm, pud)
+#define __pud_free_tlb(tlb, pud, addr)                    \
+do {                                    \
+    if (pgtable_l4_enabled) {                    \
+        pagetable_pud_dtor(virt_to_ptdesc(pud));        \
+        tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \

The specification indeed states that an sfence.vma must be emitted after a page directory modification. Your change is not enough though since eventually tlb_flush() is called and in this function we should add:

Quoted 2 lines

if (tlb->freed_tables)
    tlb_flush_mm();

I sent a patch for that here https://lore.kernel.org/linux-riscv/[email protected]/

You can add:

Thanks,

Alex

Quoted 37 lines

otherwise we are not guaranteed that a "global" sfence.vma is called.

Would you be able to benchmark this change and see the performance impact?

Thanks,

Alex

Quoted 26 lines
arch/riscv/include/asm/pgalloc.h +12 -0
+    }                                \
+} while (0)
     #define p4d_alloc_one p4d_alloc_one
   static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct 
 *mm, p4d_t *p4d)
           __p4d_free(mm, p4d);
   }
   -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d)
+#define __p4d_free_tlb(tlb, p4d, addr)                    \
+do {                                    \
+    if (pgtable_l5_enabled)                        \
+        tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \
+} while (0)
   #endif /* __PAGETABLE_PMD_FOLDED */
     static inline void sync_kernel_mappings(pgd_t *pgd)
@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct 
 *mm)
     #ifndef __PAGETABLE_PMD_FOLDED
   -#define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd)
+#define __pmd_free_tlb(tlb, pmd, addr)                \
+do {                                \
+    pagetable_pmd_dtor(virt_to_ptdesc(pmd));        \
+    tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd));    \
+} while (0)
     #endif /* __PAGETABLE_PMD_FOLDED */
#4 lore raw

Re: [PATCH 1/4] riscv: tlb: fix __p*d_free_tlb()

In reply to Jisheng Zhang
Quoted 4 lines · Jisheng Zhang

If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.

And since this is a fix, it would be worth trying to add a Fixes tag here. Not easy I agree because it fixes several commits (I have 07037db5d479f,  e8a62cc26ddf5, d10efa21a9374 and c5e9b2c2ae822 if you implement tlb_flush() as I suggested).

So I would add the latest commit as the Fixes commit (which would be c5e9b2c2ae822), and then I'd send a patch to stable for each commit with the right Fixes tag...@Conor: let me know if you have a simpler idea or if this is wrong.

Thanks,

Alex

Quoted 50 lines
---
  arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++---
  1 file changed, 17 insertions(+), 3 deletions(-)
arch/riscv/include/asm/pgalloc.h +17 -3
index d169a4f41a2e..a12fb83fa1f5 100644
@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
  		__pud_free(mm, pud);
  }
 
-#define __pud_free_tlb(tlb, pud, addr)  pud_free((tlb)->mm, pud)
+#define __pud_free_tlb(tlb, pud, addr)					\
+do {									\
+	if (pgtable_l4_enabled) {					\
+		pagetable_pud_dtor(virt_to_ptdesc(pud));		\
+		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud));	\
+	}								\
+} while (0)
 
  #define p4d_alloc_one p4d_alloc_one
  static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
  		__p4d_free(mm, p4d);
  }
 
-#define __p4d_free_tlb(tlb, p4d, addr)  p4d_free((tlb)->mm, p4d)
+#define __p4d_free_tlb(tlb, p4d, addr)					\
+do {									\
+	if (pgtable_l5_enabled)						\
+		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d));	\
+} while (0)
  #endif /* __PAGETABLE_PMD_FOLDED */
 
  static inline void sync_kernel_mappings(pgd_t *pgd)
@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 
  #ifndef __PAGETABLE_PMD_FOLDED
 
-#define __pmd_free_tlb(tlb, pmd, addr)  pmd_free((tlb)->mm, pmd)
+#define __pmd_free_tlb(tlb, pmd, addr)				\
+do {								\
+	pagetable_pmd_dtor(virt_to_ptdesc(pmd));		\
+	tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd));	\
+} while (0)
 
  #endif /* __PAGETABLE_PMD_FOLDED */
 
maintainer
#5 lore raw

Re: [PATCH 1/4] riscv: tlb: fix __p*d_free_tlb()

Quoted 12 lines · On Thu, 04 Jan 2024 02:55:40 PST (-0800), [email protected] wrote:
Quoted 4 lines · Jisheng Zhang

If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.

And since this is a fix, it would be worth trying to add a Fixes tag here. Not easy I agree because it fixes several commits (I have 07037db5d479f,  e8a62cc26ddf5, d10efa21a9374 and c5e9b2c2ae822 if you implement tlb_flush() as I suggested).

Quoted 4 lines

So I would add the latest commit as the Fixes commit (which would be c5e9b2c2ae822), and then I'd send a patch to stable for each commit with the right Fixes tag...@Conor: let me know if you have a simpler idea or if this is wrong.

I just went with

Fixes c5e9b2c2ae82 ("riscv: Improve tlb_flush()")

hopefully that's fine. It's still getting tested, it's batched up with some other stuff and I managed to find a bad merge so it might take a bit...

Quoted 56 lines

Thanks,

Alex

Quoted 50 lines
---
  arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++---
  1 file changed, 17 insertions(+), 3 deletions(-)
arch/riscv/include/asm/pgalloc.h +17 -3
index d169a4f41a2e..a12fb83fa1f5 100644
@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
  		__pud_free(mm, pud);
  }
 
-#define __pud_free_tlb(tlb, pud, addr)  pud_free((tlb)->mm, pud)
+#define __pud_free_tlb(tlb, pud, addr)					\
+do {									\
+	if (pgtable_l4_enabled) {					\
+		pagetable_pud_dtor(virt_to_ptdesc(pud));		\
+		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud));	\
+	}								\
+} while (0)
 
  #define p4d_alloc_one p4d_alloc_one
  static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
  		__p4d_free(mm, p4d);
  }
 
-#define __p4d_free_tlb(tlb, p4d, addr)  p4d_free((tlb)->mm, p4d)
+#define __p4d_free_tlb(tlb, p4d, addr)					\
+do {									\
+	if (pgtable_l5_enabled)						\
+		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d));	\
+} while (0)
  #endif /* __PAGETABLE_PMD_FOLDED */
 
  static inline void sync_kernel_mappings(pgd_t *pgd)
@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 
  #ifndef __PAGETABLE_PMD_FOLDED
 
-#define __pmd_free_tlb(tlb, pmd, addr)  pmd_free((tlb)->mm, pmd)
+#define __pmd_free_tlb(tlb, pmd, addr)				\
+do {								\
+	pagetable_pmd_dtor(virt_to_ptdesc(pmd));		\
+	tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd));	\
+} while (0)
 
  #endif /* __PAGETABLE_PMD_FOLDED */
 
#6 lore raw

PATCH 2/4 riscv: tlb: convert __p*d_free_tlb() to inline functions

In reply to Jisheng Zhang

This is to prepare for enabling MMU_GATHER_RCU_TABLE_FREE. No functionality changes.

---
 arch/riscv/include/asm/pgalloc.h | 54 +++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 22 deletions(-)
arch/riscv/include/asm/pgalloc.h +32 -22
index a12fb83fa1f5..3c5e3bd15f46 100644
@@ -95,13 +95,16 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
 		__pud_free(mm, pud);
 }
 
-#define __pud_free_tlb(tlb, pud, addr)					\
-do {									\
-	if (pgtable_l4_enabled) {					\
-		pagetable_pud_dtor(virt_to_ptdesc(pud));		\
-		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud));	\
-	}								\
-} while (0)
+static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
+				  unsigned long addr)
+{
+	if (pgtable_l4_enabled) {
+		struct ptdesc *ptdesc = virt_to_ptdesc(pud);
+
+		pagetable_pud_dtor(ptdesc);
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	}
+}
 
 #define p4d_alloc_one p4d_alloc_one
 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -130,11 +133,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
 		__p4d_free(mm, p4d);
 }
 
-#define __p4d_free_tlb(tlb, p4d, addr)					\
-do {									\
-	if (pgtable_l5_enabled)						\
-		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d));	\
-} while (0)
+static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
+				  unsigned long addr)
+{
+	if (pgtable_l5_enabled)
+		tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+}
 #endif /* __PAGETABLE_PMD_FOLDED */
 
 static inline void sync_kernel_mappings(pgd_t *pgd)
@@ -159,19 +163,25 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 
 #ifndef __PAGETABLE_PMD_FOLDED
 
-#define __pmd_free_tlb(tlb, pmd, addr)				\
-do {								\
-	pagetable_pmd_dtor(virt_to_ptdesc(pmd));		\
-	tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd));	\
-} while (0)
+static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
+				  unsigned long addr)
+{
+	struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
+
+	pagetable_pmd_dtor(ptdesc);
+	tlb_remove_page_ptdesc(tlb, ptdesc);
+}
 
 #endif /* __PAGETABLE_PMD_FOLDED */
 
-#define __pte_free_tlb(tlb, pte, buf)			\
-do {							\
-	pagetable_pte_dtor(page_ptdesc(pte));		\
-	tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\
-} while (0)
+static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
+				  unsigned long addr)
+{
+	struct ptdesc *ptdesc = page_ptdesc(pte);
+
+	pagetable_pte_dtor(ptdesc);
+	tlb_remove_page_ptdesc(tlb, ptdesc);
+}
 #endif /* CONFIG_MMU */
 
 #endif /* _ASM_RISCV_PGALLOC_H */
Signature · 1 line
2.40.0
#7 lore raw

Re: [External] [PATCH 2/4] riscv: tlb: convert __p*d_free_tlb() to inline functions

Hi Jisheng,

Quoted 92 lines · Jisheng Zhang

This is to prepare for enabling MMU_GATHER_RCU_TABLE_FREE. No functionality changes.

---
 arch/riscv/include/asm/pgalloc.h | 54 +++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 22 deletions(-)
arch/riscv/include/asm/pgalloc.h +32 -23
index a12fb83fa1f5..3c5e3bd15f46 100644
@@ -95,13 +95,16 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
                __pud_free(mm, pud);
 }
 
-#define __pud_free_tlb(tlb, pud, addr)                                 \
-do {                                                                   \
-       if (pgtable_l4_enabled) {                                       \
-               pagetable_pud_dtor(virt_to_ptdesc(pud));                \
-               tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud));     \
-       }                                                               \
-} while (0)
+static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
+                                 unsigned long addr)
+{
+       if (pgtable_l4_enabled) {
+               struct ptdesc *ptdesc = virt_to_ptdesc(pud);
+
+               pagetable_pud_dtor(ptdesc);
+               tlb_remove_page_ptdesc(tlb, ptdesc);
+       }
+}
 
 #define p4d_alloc_one p4d_alloc_one
 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -130,11 +133,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
                __p4d_free(mm, p4d);
 }
 
-#define __p4d_free_tlb(tlb, p4d, addr)                                 \
-do {                                                                   \
-       if (pgtable_l5_enabled)                                         \
-               tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d));     \
-} while (0)
+static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
+                                 unsigned long addr)
+{
+       if (pgtable_l5_enabled)
+               tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+}
 #endif /* __PAGETABLE_PMD_FOLDED */
 
 static inline void sync_kernel_mappings(pgd_t *pgd)
@@ -159,19 +163,25 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 
 #ifndef __PAGETABLE_PMD_FOLDED
 
-#define __pmd_free_tlb(tlb, pmd, addr)                         \
-do {                                                           \
-       pagetable_pmd_dtor(virt_to_ptdesc(pmd));                \
-       tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd));     \
-} while (0)
+static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
+                                 unsigned long addr)
+{
+       struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
+
+       pagetable_pmd_dtor(ptdesc);
+       tlb_remove_page_ptdesc(tlb, ptdesc);
+}
 
 #endif /* __PAGETABLE_PMD_FOLDED */
 
-#define __pte_free_tlb(tlb, pte, buf)                  \
-do {                                                   \
-       pagetable_pte_dtor(page_ptdesc(pte));           \
-       tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\
-} while (0)
+static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
+                                 unsigned long addr)
+{
+       struct ptdesc *ptdesc = page_ptdesc(pte);
+
+       pagetable_pte_dtor(ptdesc);
+       tlb_remove_page_ptdesc(tlb, ptdesc);
+}
 #endif /* CONFIG_MMU */
 
 #endif /* _ASM_RISCV_PGALLOC_H */
--
Quoted 2 lines

2.40.0

Why is it necessary to convert to inline functions?

Thanks,
Yunhui

#8 lore raw

Re: [External] [PATCH 2/4] riscv: tlb: convert __p*d_free_tlb() to inline functions

Quoted 1 line · yunhui cui

Hi Jisheng,

Hi,

Quoted 92 lines · Jisheng Zhang
Quoted 90 lines

This is to prepare for enabling MMU_GATHER_RCU_TABLE_FREE. No functionality changes.

---
 arch/riscv/include/asm/pgalloc.h | 54 +++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 22 deletions(-)
arch/riscv/include/asm/pgalloc.h +32 -22
index a12fb83fa1f5..3c5e3bd15f46 100644
@@ -95,13 +95,16 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
                __pud_free(mm, pud);
 }
 
-#define __pud_free_tlb(tlb, pud, addr)                                 \
-do {                                                                   \
-       if (pgtable_l4_enabled) {                                       \
-               pagetable_pud_dtor(virt_to_ptdesc(pud));                \
-               tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud));     \
-       }                                                               \
-} while (0)
+static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
+                                 unsigned long addr)
+{
+       if (pgtable_l4_enabled) {
+               struct ptdesc *ptdesc = virt_to_ptdesc(pud);
+
+               pagetable_pud_dtor(ptdesc);
+               tlb_remove_page_ptdesc(tlb, ptdesc);
+       }
+}
 
 #define p4d_alloc_one p4d_alloc_one
 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -130,11 +133,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
                __p4d_free(mm, p4d);
 }
 
-#define __p4d_free_tlb(tlb, p4d, addr)                                 \
-do {                                                                   \
-       if (pgtable_l5_enabled)                                         \
-               tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d));     \
-} while (0)
+static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
+                                 unsigned long addr)
+{
+       if (pgtable_l5_enabled)
+               tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+}
 #endif /* __PAGETABLE_PMD_FOLDED */
 
 static inline void sync_kernel_mappings(pgd_t *pgd)
@@ -159,19 +163,25 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 
 #ifndef __PAGETABLE_PMD_FOLDED
 
-#define __pmd_free_tlb(tlb, pmd, addr)                         \
-do {                                                           \
-       pagetable_pmd_dtor(virt_to_ptdesc(pmd));                \
-       tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd));     \
-} while (0)
+static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
+                                 unsigned long addr)
+{
+       struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
+
+       pagetable_pmd_dtor(ptdesc);
+       tlb_remove_page_ptdesc(tlb, ptdesc);
+}
 
 #endif /* __PAGETABLE_PMD_FOLDED */
 
-#define __pte_free_tlb(tlb, pte, buf)                  \
-do {                                                   \
-       pagetable_pte_dtor(page_ptdesc(pte));           \
-       tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\
-} while (0)
+static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
+                                 unsigned long addr)
+{
+       struct ptdesc *ptdesc = page_ptdesc(pte);
+
+       pagetable_pte_dtor(ptdesc);
+       tlb_remove_page_ptdesc(tlb, ptdesc);
+}
 #endif /* CONFIG_MMU */
 
Quoted 6 lines
Quoted 4 lines
arch/riscv/include/asm/pgalloc.h +0 -1
 #endif /* _ASM_RISCV_PGALLOC_H */
--

2.40.0

Why is it necessary to convert to inline functions?

Hmm, it's not necessary but a plus, the inline version's readability and maintainability is better than macros

Regards

reviewer Reviewed-by on v1 2/4
#9 lore raw

Re: [PATCH 2/4] riscv: tlb: convert __p*d_free_tlb() to inline functions

In reply to Jisheng Zhang
Quoted 80 lines · Jisheng Zhang

This is to prepare for enabling MMU_GATHER_RCU_TABLE_FREE. No functionality changes.

---
  arch/riscv/include/asm/pgalloc.h | 54 +++++++++++++++++++-------------
  1 file changed, 32 insertions(+), 22 deletions(-)
arch/riscv/include/asm/pgalloc.h +25 -22
index a12fb83fa1f5..3c5e3bd15f46 100644
@@ -95,13 +95,16 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)
  		__pud_free(mm, pud);
  }
 
-#define __pud_free_tlb(tlb, pud, addr)					\
-do {									\
-	if (pgtable_l4_enabled) {					\
-		pagetable_pud_dtor(virt_to_ptdesc(pud));		\
-		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud));	\
-	}								\
-} while (0)
+static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
+				  unsigned long addr)
+{
+	if (pgtable_l4_enabled) {
+		struct ptdesc *ptdesc = virt_to_ptdesc(pud);
+
+		pagetable_pud_dtor(ptdesc);
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	}
+}
 
  #define p4d_alloc_one p4d_alloc_one
  static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -130,11 +133,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
  		__p4d_free(mm, p4d);
  }
 
-#define __p4d_free_tlb(tlb, p4d, addr)					\
-do {									\
-	if (pgtable_l5_enabled)						\
-		tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d));	\
-} while (0)
+static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
+				  unsigned long addr)
+{
+	if (pgtable_l5_enabled)
+		tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+}
  #endif /* __PAGETABLE_PMD_FOLDED */
 
  static inline void sync_kernel_mappings(pgd_t *pgd)
@@ -159,19 +163,25 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)
 
  #ifndef __PAGETABLE_PMD_FOLDED
 
-#define __pmd_free_tlb(tlb, pmd, addr)				\
-do {								\
-	pagetable_pmd_dtor(virt_to_ptdesc(pmd));		\
-	tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd));	\
-} while (0)
+static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
+				  unsigned long addr)
+{
+	struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
+
+	pagetable_pmd_dtor(ptdesc);
+	tlb_remove_page_ptdesc(tlb, ptdesc);
+}
 
  #endif /* __PAGETABLE_PMD_FOLDED */
 
-#define __pte_free_tlb(tlb, pte, buf)			\
-do {							\
-	pagetable_pte_dtor(page_ptdesc(pte));		\
-	tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\
-} while (0)
+static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,

arch/riscv/include/asm/pgalloc.h static inline pgd_t *pgd_alloc(struct mm_struct *mm) · line 187

Quoted 10 lines
arch/riscv/include/asm/pgalloc.h +7 -0
+				  unsigned long addr)
+{
+	struct ptdesc *ptdesc = page_ptdesc(pte);
+
+	pagetable_pte_dtor(ptdesc);
+	tlb_remove_page_ptdesc(tlb, ptdesc);
+}
  #endif /* CONFIG_MMU */
 
  #endif /* _ASM_RISCV_PGALLOC_H */

You can add:

Thanks,

Alex

#10 lore raw

PATCH 3/4 riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU

In reply to Jisheng Zhang

In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.

riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the riscv_ipi_for_rfence is true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually the riscv_ipi_for_rfence is false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h for more details.

This patch enables MMU_GATHER_RCU_TABLE_FREE, then use

*tlb_remove_page_ptdesc() for those platforms which use IPI to perform TLB shootdown;

*tlb_remove_ptdesc() for those platforms which use SBI to perform TLB shootdown;

Both case mean that disabling interrupts will block the free and protect the fast gup page walker.

---
 arch/riscv/Kconfig               |  1 +
 arch/riscv/include/asm/pgalloc.h | 23 ++++++++++++++++++-----
 arch/riscv/include/asm/tlb.h     | 18 ++++++++++++++++++
 3 files changed, 37 insertions(+), 5 deletions(-)
arch/riscv/Kconfig +1 -0
index 24c1799e2ec4..d3555173d9f4 100644
@@ -147,6 +147,7 @@ config RISCV
 	select IRQ_FORCED_THREADING
 	select KASAN_VMALLOC if KASAN
 	select LOCK_MM_AND_FIND_VMA
+	select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU
 	select MODULES_USE_ELF_RELA if MODULES
 	select MODULE_SECTIONS if MODULES
 	select OF
arch/riscv/include/asm/pgalloc.h +18 -5
index 3c5e3bd15f46..deaf971253a2 100644
@@ -102,7 +102,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
 		struct ptdesc *ptdesc = virt_to_ptdesc(pud);
 
 		pagetable_pud_dtor(ptdesc);
-		tlb_remove_page_ptdesc(tlb, ptdesc);
+		if (riscv_use_ipi_for_rfence())
+			tlb_remove_page_ptdesc(tlb, ptdesc);
+		else
+			tlb_remove_ptdesc(tlb, ptdesc);
 	}
 }
 
@@ -136,8 +139,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
 static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
 				  unsigned long addr)
 {
-	if (pgtable_l5_enabled)
-		tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+	if (pgtable_l5_enabled) {
+		if (riscv_use_ipi_for_rfence())
+			tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+		else
+			tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d));
+	}
 }
 #endif /* __PAGETABLE_PMD_FOLDED */
 
@@ -169,7 +176,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
 	struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
 
 	pagetable_pmd_dtor(ptdesc);
-	tlb_remove_page_ptdesc(tlb, ptdesc);
+	if (riscv_use_ipi_for_rfence())
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	else
+		tlb_remove_ptdesc(tlb, ptdesc);
 }
 
 #endif /* __PAGETABLE_PMD_FOLDED */
@@ -180,7 +190,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
 	struct ptdesc *ptdesc = page_ptdesc(pte);
 
 	pagetable_pte_dtor(ptdesc);
-	tlb_remove_page_ptdesc(tlb, ptdesc);
+	if (riscv_use_ipi_for_rfence())
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	else
+		tlb_remove_ptdesc(tlb, ptdesc);
 }
 #endif /* CONFIG_MMU */
 
arch/riscv/include/asm/tlb.h +18 -0
index 1eb5682b2af6..a0b8b853503f 100644
@@ -10,6 +10,24 @@ struct mmu_gather;
 
 static void tlb_flush(struct mmu_gather *tlb);
 
+#ifdef CONFIG_MMU
+#include <linux/swap.h>
+
+/*
+ * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to
+ * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use
+ * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this
+ * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the
+ * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h
+ * for more details.
+ */
+static inline void __tlb_remove_table(void *table)
+{
+	free_page_and_swap_cache(table);
+}
+
+#endif /* CONFIG_MMU */
+
 #define tlb_flush tlb_flush
 #include <asm-generic/tlb.h>
 
Signature · 1 line
2.40.0
#11 lore raw

Re: [PATCH 3/4] riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU

Quoted 14 lines · Jisheng Zhang

In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.

riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the riscv_ipi_for_rfence is true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually the riscv_ipi_for_rfence is false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h for more details.

Quoted 7 lines

This patch enables MMU_GATHER_RCU_TABLE_FREE, then use

*tlb_remove_page_ptdesc() for those platforms which use IPI to perform TLB shootdown;

*tlb_remove_ptdesc() for those platforms which use SBI to perform TLB shootdown;

Can you elaborate a bit more on what those functions do differently and why we need to differentiate IPI vs SBI TLB shootdown? I don't understand this.

Thanks,

Alex

Quoted 106 lines

Both case mean that disabling interrupts will block the free and protect the fast gup page walker.

---
  arch/riscv/Kconfig               |  1 +
  arch/riscv/include/asm/pgalloc.h | 23 ++++++++++++++++++-----
  arch/riscv/include/asm/tlb.h     | 18 ++++++++++++++++++
  3 files changed, 37 insertions(+), 5 deletions(-)
arch/riscv/Kconfig +1 -0
index 24c1799e2ec4..d3555173d9f4 100644
@@ -147,6 +147,7 @@ config RISCV
  	select IRQ_FORCED_THREADING
  	select KASAN_VMALLOC if KASAN
  	select LOCK_MM_AND_FIND_VMA
+	select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU
  	select MODULES_USE_ELF_RELA if MODULES
  	select MODULE_SECTIONS if MODULES
  	select OF
arch/riscv/include/asm/pgalloc.h +18 -5
index 3c5e3bd15f46..deaf971253a2 100644
@@ -102,7 +102,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
  		struct ptdesc *ptdesc = virt_to_ptdesc(pud);
 
  		pagetable_pud_dtor(ptdesc);
-		tlb_remove_page_ptdesc(tlb, ptdesc);
+		if (riscv_use_ipi_for_rfence())
+			tlb_remove_page_ptdesc(tlb, ptdesc);
+		else
+			tlb_remove_ptdesc(tlb, ptdesc);
  	}
  }
 
@@ -136,8 +139,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
  static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
  				  unsigned long addr)
  {
-	if (pgtable_l5_enabled)
-		tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+	if (pgtable_l5_enabled) {
+		if (riscv_use_ipi_for_rfence())
+			tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+		else
+			tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d));
+	}
  }
  #endif /* __PAGETABLE_PMD_FOLDED */
 
@@ -169,7 +176,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
  	struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
 
  	pagetable_pmd_dtor(ptdesc);
-	tlb_remove_page_ptdesc(tlb, ptdesc);
+	if (riscv_use_ipi_for_rfence())
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	else
+		tlb_remove_ptdesc(tlb, ptdesc);
  }
 
  #endif /* __PAGETABLE_PMD_FOLDED */
@@ -180,7 +190,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
  	struct ptdesc *ptdesc = page_ptdesc(pte);
 
  	pagetable_pte_dtor(ptdesc);
-	tlb_remove_page_ptdesc(tlb, ptdesc);
+	if (riscv_use_ipi_for_rfence())
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	else
+		tlb_remove_ptdesc(tlb, ptdesc);
  }
  #endif /* CONFIG_MMU */
 
arch/riscv/include/asm/tlb.h +18 -0
index 1eb5682b2af6..a0b8b853503f 100644
@@ -10,6 +10,24 @@ struct mmu_gather;
 
  static void tlb_flush(struct mmu_gather *tlb);
 
+#ifdef CONFIG_MMU
+#include <linux/swap.h>
+
+/*
+ * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to
+ * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use
+ * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this
+ * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the
+ * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h
+ * for more details.
+ */
+static inline void __tlb_remove_table(void *table)
+{
+	free_page_and_swap_cache(table);
+}
+
+#endif /* CONFIG_MMU */
+
  #define tlb_flush tlb_flush
  #include <asm-generic/tlb.h>
 
#12 lore raw

Re: [PATCH 3/4] riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU

Quoted 20 lines · Alexandre Ghiti
Quoted 19 lines · Jisheng Zhang

In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.

riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the riscv_ipi_for_rfence is true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually the riscv_ipi_for_rfence is false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h for more details.

This patch enables MMU_GATHER_RCU_TABLE_FREE, then use

*tlb_remove_page_ptdesc() for those platforms which use IPI to perform TLB shootdown;

Quoted 6 lines
Quoted 2 lines

*tlb_remove_ptdesc() for those platforms which use SBI to perform TLB shootdown;

Can you elaborate a bit more on what those functions do differently and why we need to differentiate IPI vs SBI TLB shootdown? I don't understand this.

Hi Alex,

If IPI, the local_irq_save in lockless_pages_from_mm() of fast gup code path will block page table pages from being freed, I think the comments there is execellent.

If SBI, the local_irq_save in lockless_pages_from_mm() can't acchieve the goal however. Because local_irq_save() only disable S-privilege IPI irq, it can't disable M-privilege's, which the SBI implementation use to shootdown TLB entry. So we need MMU_GATHER_RCU_TABLE_FREE helper for SBI case.

Thanks

Quoted 97 lines
Quoted 97 lines

Both case mean that disabling interrupts will block the free and protect the fast gup page walker.

---
  arch/riscv/Kconfig               |  1 +
  arch/riscv/include/asm/pgalloc.h | 23 ++++++++++++++++++-----
  arch/riscv/include/asm/tlb.h     | 18 ++++++++++++++++++
  3 files changed, 37 insertions(+), 5 deletions(-)
arch/riscv/Kconfig +1 -0
index 24c1799e2ec4..d3555173d9f4 100644
@@ -147,6 +147,7 @@ config RISCV
  	select IRQ_FORCED_THREADING
  	select KASAN_VMALLOC if KASAN
  	select LOCK_MM_AND_FIND_VMA
+	select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU
  	select MODULES_USE_ELF_RELA if MODULES
  	select MODULE_SECTIONS if MODULES
  	select OF
arch/riscv/include/asm/pgalloc.h +18 -5
index 3c5e3bd15f46..deaf971253a2 100644
@@ -102,7 +102,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
  		struct ptdesc *ptdesc = virt_to_ptdesc(pud);
  		pagetable_pud_dtor(ptdesc);
-		tlb_remove_page_ptdesc(tlb, ptdesc);
+		if (riscv_use_ipi_for_rfence())
+			tlb_remove_page_ptdesc(tlb, ptdesc);
+		else
+			tlb_remove_ptdesc(tlb, ptdesc);
  	}
  }
@@ -136,8 +139,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
  static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
  				  unsigned long addr)
  {
-	if (pgtable_l5_enabled)
-		tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+	if (pgtable_l5_enabled) {
+		if (riscv_use_ipi_for_rfence())
+			tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+		else
+			tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d));
+	}
  }
  #endif /* __PAGETABLE_PMD_FOLDED */
@@ -169,7 +176,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
  	struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
  	pagetable_pmd_dtor(ptdesc);
-	tlb_remove_page_ptdesc(tlb, ptdesc);
+	if (riscv_use_ipi_for_rfence())
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	else
+		tlb_remove_ptdesc(tlb, ptdesc);
  }
  #endif /* __PAGETABLE_PMD_FOLDED */
@@ -180,7 +190,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
  	struct ptdesc *ptdesc = page_ptdesc(pte);
  	pagetable_pte_dtor(ptdesc);
-	tlb_remove_page_ptdesc(tlb, ptdesc);
+	if (riscv_use_ipi_for_rfence())
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	else
+		tlb_remove_ptdesc(tlb, ptdesc);
  }
  #endif /* CONFIG_MMU */
arch/riscv/include/asm/tlb.h +18 -0
index 1eb5682b2af6..a0b8b853503f 100644
@@ -10,6 +10,24 @@ struct mmu_gather;
  static void tlb_flush(struct mmu_gather *tlb);
+#ifdef CONFIG_MMU
+#include <linux/swap.h>
+
+/*
+ * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to
+ * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use
+ * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this
+ * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the
+ * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h
+ * for more details.
+ */
+static inline void __tlb_remove_table(void *table)
+{
+	free_page_and_swap_cache(table);
+}
+
+#endif /* CONFIG_MMU */
+
  #define tlb_flush tlb_flush
  #include <asm-generic/tlb.h>
reviewer Reviewed-by
#13 lore raw

Re: [PATCH 3/4] riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU

Hi Jisheng,

Quoted 32 lines · Jisheng Zhang
Quoted 17 lines · Alexandre Ghiti
Quoted 16 lines · Jisheng Zhang

In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.

riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the riscv_ipi_for_rfence is true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually the riscv_ipi_for_rfence is false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h for more details.

This patch enables MMU_GATHER_RCU_TABLE_FREE, then use

Quoted 8 lines
Quoted 5 lines

*tlb_remove_page_ptdesc() for those platforms which use IPI to perform TLB shootdown;

*tlb_remove_ptdesc() for those platforms which use SBI to perform TLB shootdown;

Can you elaborate a bit more on what those functions do differently and why we need to differentiate IPI vs SBI TLB shootdown? I don't understand this.

Hi Alex,

If IPI, the local_irq_save in lockless_pages_from_mm() of fast gup code path will block page table pages from being freed, I think the comments there is execellent.

Quoted 5 lines

If SBI, the local_irq_save in lockless_pages_from_mm() can't acchieve the goal however. Because local_irq_save() only disable S-privilege IPI irq, it can't disable M-privilege's, which the SBI implementation use to shootdown TLB entry. So we need MMU_GATHER_RCU_TABLE_FREE helper for SBI case.

Ok, I get it now, can you add the following link to your commit description https://elixir.bootlin.com/linux/v6.6/source/mm/mmu_gather.c#L162 ? It describes the problem very clearly.

Quoted 79 lines

Thanks

Quoted 77 lines
Quoted 77 lines

Both case mean that disabling interrupts will block the free and protect the fast gup page walker.

---
   arch/riscv/Kconfig               |  1 +
   arch/riscv/include/asm/pgalloc.h | 23 ++++++++++++++++++-----
   arch/riscv/include/asm/tlb.h     | 18 ++++++++++++++++++
   3 files changed, 37 insertions(+), 5 deletions(-)
arch/riscv/Kconfig +1 -0
index 24c1799e2ec4..d3555173d9f4 100644
@@ -147,6 +147,7 @@ config RISCV
   	select IRQ_FORCED_THREADING
   	select KASAN_VMALLOC if KASAN
   	select LOCK_MM_AND_FIND_VMA
+	select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU
   	select MODULES_USE_ELF_RELA if MODULES
   	select MODULE_SECTIONS if MODULES
   	select OF
arch/riscv/include/asm/pgalloc.h +18 -5
index 3c5e3bd15f46..deaf971253a2 100644
@@ -102,7 +102,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
   		struct ptdesc *ptdesc = virt_to_ptdesc(pud);
   		pagetable_pud_dtor(ptdesc);
-		tlb_remove_page_ptdesc(tlb, ptdesc);
+		if (riscv_use_ipi_for_rfence())
+			tlb_remove_page_ptdesc(tlb, ptdesc);
+		else
+			tlb_remove_ptdesc(tlb, ptdesc);
   	}
   }
@@ -136,8 +139,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
   static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
   				  unsigned long addr)
   {
-	if (pgtable_l5_enabled)
-		tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+	if (pgtable_l5_enabled) {
+		if (riscv_use_ipi_for_rfence())
+			tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d));
+		else
+			tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d));
+	}
   }
   #endif /* __PAGETABLE_PMD_FOLDED */
@@ -169,7 +176,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
   	struct ptdesc *ptdesc = virt_to_ptdesc(pmd);
   	pagetable_pmd_dtor(ptdesc);
-	tlb_remove_page_ptdesc(tlb, ptdesc);
+	if (riscv_use_ipi_for_rfence())
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	else
+		tlb_remove_ptdesc(tlb, ptdesc);
   }
   #endif /* __PAGETABLE_PMD_FOLDED */
@@ -180,7 +190,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
   	struct ptdesc *ptdesc = page_ptdesc(pte);
   	pagetable_pte_dtor(ptdesc);
-	tlb_remove_page_ptdesc(tlb, ptdesc);
+	if (riscv_use_ipi_for_rfence())
+		tlb_remove_page_ptdesc(tlb, ptdesc);
+	else
+		tlb_remove_ptdesc(tlb, ptdesc);
   }
   #endif /* CONFIG_MMU */
arch/riscv/include/asm/tlb.h +1 -0
index 1eb5682b2af6..a0b8b853503f 100644
@@ -10,6 +10,24 @@ struct mmu_gather;
   static void tlb_flush(struct mmu_gather *tlb);
+#ifdef CONFIG_MMU
Quoted 8 lines
Quoted 8 lines

arch/riscv/include/asm/tlb.h struct mmu_gather; · line 21

Quoted 8 lines
arch/riscv/include/asm/tlb.h +8 -0
+#include <linux/swap.h>
+
+/*
+ * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to
+ * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use
+ * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this
+ * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the
+ * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h

A direct link would be better, I did not find the comment you mention here and even though, it can still move around later.

And then you can add:

Thanks!

Alex

Quoted 15 lines
Quoted 11 lines

arch/riscv/include/asm/tlb.h struct mmu_gather; · line 32

Quoted 11 lines
arch/riscv/include/asm/tlb.h +9 -0
+ * for more details.
+ */
+static inline void __tlb_remove_table(void *table)
+{
+	free_page_and_swap_cache(table);
+}
+
+#endif /* CONFIG_MMU */
+
   #define tlb_flush tlb_flush
   #include <asm-generic/tlb.h>

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#14 lore raw

PATCH 4/4 riscv: enable HAVE_FAST_GUP if MMU

In reply to Jisheng Zhang

Activate the fast gup for riscv mmu platforms. Here are some GUP_FAST_BENCHMARK performance numbers:

Before the patch:
GUP_FAST_BENCHMARK: Time: get:53203 put:5085 us

After the patch:
GUP_FAST_BENCHMARK: Time: get:17711 put:5060 us

The get time is reduced by 66.7%! IOW, 3x get speed!

---
 arch/riscv/Kconfig               | 1 +
 arch/riscv/include/asm/pgtable.h | 6 ++++++
 2 files changed, 7 insertions(+)
arch/riscv/Kconfig +1 -0
index d3555173d9f4..04df9920282d 100644
@@ -119,6 +119,7 @@ config RISCV
 	select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER
 	select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION
 	select HAVE_EBPF_JIT if MMU
+	select HAVE_FAST_GUP if MMU
 	select HAVE_FUNCTION_ARG_ACCESS_API
 	select HAVE_FUNCTION_ERROR_INJECTION
 	select HAVE_GCC_PLUGINS
arch/riscv/include/asm/pgtable.h +6 -0
index ab00235b018f..c6eb214139e6 100644
@@ -673,6 +673,12 @@ static inline int pmd_write(pmd_t pmd)
 	return pte_write(pmd_pte(pmd));
 }
 
+#define pud_write pud_write
+static inline int pud_write(pud_t pud)
+{
+	return pte_write(pud_pte(pud));
+}
+
 static inline int pmd_dirty(pmd_t pmd)
 {
 	return pte_dirty(pmd_pte(pmd));
Signature · 1 line
2.40.0
#15 lore raw

Re: [PATCH 4/4] riscv: enable HAVE_FAST_GUP if MMU

Quoted 8 lines · Jisheng Zhang

Activate the fast gup for riscv mmu platforms. Here are some GUP_FAST_BENCHMARK performance numbers:

Before the patch:
GUP_FAST_BENCHMARK: Time: get:53203 put:5085 us

After the patch:
GUP_FAST_BENCHMARK: Time: get:17711 put:5060 us

On which platform did you run this benchmark?

Quoted 2 lines

The get time is reduced by 66.7%! IOW, 3x get speed!

Well done!

Thanks,

Alex

Quoted 36 lines
---
  arch/riscv/Kconfig               | 1 +
  arch/riscv/include/asm/pgtable.h | 6 ++++++
  2 files changed, 7 insertions(+)
arch/riscv/Kconfig +1 -0
index d3555173d9f4..04df9920282d 100644
@@ -119,6 +119,7 @@ config RISCV
  	select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER
  	select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION
  	select HAVE_EBPF_JIT if MMU
+	select HAVE_FAST_GUP if MMU
  	select HAVE_FUNCTION_ARG_ACCESS_API
  	select HAVE_FUNCTION_ERROR_INJECTION
  	select HAVE_GCC_PLUGINS
arch/riscv/include/asm/pgtable.h +6 -0
index ab00235b018f..c6eb214139e6 100644
@@ -673,6 +673,12 @@ static inline int pmd_write(pmd_t pmd)
  	return pte_write(pmd_pte(pmd));
  }
 
+#define pud_write pud_write
+static inline int pud_write(pud_t pud)
+{
+	return pte_write(pud_pte(pud));
+}
+
  static inline int pmd_dirty(pmd_t pmd)
  {
  	return pte_dirty(pmd_pte(pmd));
#16 lore raw

Re: [PATCH 4/4] riscv: enable HAVE_FAST_GUP if MMU

Quoted 4 lines · Alexandre Ghiti
Quoted 3 lines · Jisheng Zhang

Activate the fast gup for riscv mmu platforms. Here are some GUP_FAST_BENCHMARK performance numbers:

Quoted 8 lines
Quoted 5 lines

Before the patch:
GUP_FAST_BENCHMARK: Time: get:53203 put:5085 us

After the patch:
GUP_FAST_BENCHMARK: Time: get:17711 put:5060 us

On which platform did you run this benchmark?

T-HEAD th1520(cpufreq isn't enabled since the clk/pll isn't upstreamed, so cpu is running at the default freq set by u-boot)

Quoted 48 lines
Quoted 4 lines

The get time is reduced by 66.7%! IOW, 3x get speed!

Well done!

Thanks,

Alex

Quoted 35 lines
---
  arch/riscv/Kconfig               | 1 +
  arch/riscv/include/asm/pgtable.h | 6 ++++++
  2 files changed, 7 insertions(+)
arch/riscv/Kconfig +1 -0
index d3555173d9f4..04df9920282d 100644
@@ -119,6 +119,7 @@ config RISCV
  	select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER
  	select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION
  	select HAVE_EBPF_JIT if MMU
+	select HAVE_FAST_GUP if MMU
  	select HAVE_FUNCTION_ARG_ACCESS_API
  	select HAVE_FUNCTION_ERROR_INJECTION
  	select HAVE_GCC_PLUGINS
arch/riscv/include/asm/pgtable.h +6 -0
index ab00235b018f..c6eb214139e6 100644
@@ -673,6 +673,12 @@ static inline int pmd_write(pmd_t pmd)
  	return pte_write(pmd_pte(pmd));
  }
+#define pud_write pud_write
+static inline int pud_write(pud_t pud)
+{
+	return pte_write(pud_pte(pud));
+}
+
  static inline int pmd_dirty(pmd_t pmd)
  {
  	return pte_dirty(pmd_pte(pmd));
reviewer Reviewed-by
#17 lore raw

Re: [PATCH 4/4] riscv: enable HAVE_FAST_GUP if MMU

Quoted 50 lines · Jisheng Zhang
Quoted 11 lines · Alexandre Ghiti
Quoted 8 lines · Jisheng Zhang

Activate the fast gup for riscv mmu platforms. Here are some GUP_FAST_BENCHMARK performance numbers:

Before the patch:
GUP_FAST_BENCHMARK: Time: get:53203 put:5085 us

After the patch:
GUP_FAST_BENCHMARK: Time: get:17711 put:5060 us

On which platform did you run this benchmark?

T-HEAD th1520(cpufreq isn't enabled since the clk/pll isn't upstreamed, so cpu is running at the default freq set by u-boot)

Quoted 36 lines
Quoted 2 lines

The get time is reduced by 66.7%! IOW, 3x get speed!

Well done!

Thanks,

Alex

Quoted 26 lines
---
   arch/riscv/Kconfig               | 1 +
   arch/riscv/include/asm/pgtable.h | 6 ++++++
   2 files changed, 7 insertions(+)
arch/riscv/Kconfig +1 -0
index d3555173d9f4..04df9920282d 100644
@@ -119,6 +119,7 @@ config RISCV
   	select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER
   	select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION
   	select HAVE_EBPF_JIT if MMU
+	select HAVE_FAST_GUP if MMU
   	select HAVE_FUNCTION_ARG_ACCESS_API
   	select HAVE_FUNCTION_ERROR_INJECTION
   	select HAVE_GCC_PLUGINS
arch/riscv/include/asm/pgtable.h +1 -0
index ab00235b018f..c6eb214139e6 100644
@@ -673,6 +673,12 @@ static inline int pmd_write(pmd_t pmd)
   	return pte_write(pmd_pte(pmd));
   }
+#define pud_write pud_write
Quoted 8 lines
Quoted 8 lines

arch/riscv/include/asm/pgtable.h static inline int pmd_write(pmd_t pmd) · line 684

Quoted 8 lines
arch/riscv/include/asm/pgtable.h +5 -0
+static inline int pud_write(pud_t pud)
+{
+	return pte_write(pud_pte(pud));
+}
+
   static inline int pmd_dirty(pmd_t pmd)
   {
   	return pte_dirty(pmd_pte(pmd));

Thanks, you can add:

Thanks,

Alex

Quoted 4 lines

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#18 lore raw
In reply to Jisheng Zhang
This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:

Hello:

This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:

Quoted 9 lines · On Wed, 20 Dec 2023 01:50:42 +0800 you wrote:

This series adds fast gup support to riscv.

The First patch fixes a bug in __p*d_free_tlb(). Per the riscv privileged spec, if non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must.

The 2nd patch is a preparation patch.

[...]

Here is the summary with links:
- [1/4] riscv: tlb: fix __p*d_free_tlb()

    https://git.kernel.org/riscv/c/8246601a7d39
  - [2/4] riscv: tlb: convert __p*d_free_tlb() to inline functions
    https://git.kernel.org/riscv/c/40d1bb92a493
  - [3/4] riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU
    https://git.kernel.org/riscv/c/69be3fb111e7
  - [4/4] riscv: enable HAVE_FAST_GUP if MMU
    https://git.kernel.org/riscv/c/3f910b7a522e

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html