riscv: support fast gup
Subsystems (4)
cover riscv: support fast gup
This series adds fast gup support to riscv.
The First patch fixes a bug in __p*d_free_tlb(). Per the riscv privileged spec, if non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must.
The 2nd patch is a preparation patch.
The last two patches do the real work:
In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.
riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the riscv_ipi_for_rfence is true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually the riscv_ipi_for_rfence is false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h for more details.
This patch enables MMU_GATHER_RCU_TABLE_FREE, then use
*tlb_remove_page_ptdesc() for those platforms which use IPI to perform TLB shootdown;
*tlb_remove_ptdesc() for those platforms which use SBI to perform TLB shootdown;
Both case mean that disabling interrupts will block the free and protect the fast gup page walker.
So after the 3rd patch, everything is well prepared, let's select HAVE_FAST_GUP if MMU.
Jisheng Zhang (4):
riscv: tlb: fix __p*d_free_tlb()
riscv: tlb: convert __p*d_free_tlb() to inline functions
riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU
riscv: enable HAVE_FAST_GUP if MMU
arch/riscv/Kconfig | 2 ++ arch/riscv/include/asm/pgalloc.h | 53 +++++++++++++++++++++++++++----- arch/riscv/include/asm/pgtable.h | 6 ++++ arch/riscv/include/asm/tlb.h | 18 +++++++++++ 4 files changed, 71 insertions(+), 8 deletions(-)
Signature · 1 line
2.40.0
PATCH 1/4 riscv: tlb: fix __p*d_free_tlb()
If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.
--- arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
arch/riscv/include/asm/pgalloc.h +17 -3
@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)__pud_free (mm, pud); } -#define __pud_free_tlb (tlb, pud, addr) pud_free((tlb)->mm, pud) +#define __pud_free_tlb (tlb, pud, addr) \ +do { \ + if (pgtable_l4_enabled) { \ + pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ + } \ +} while (0)#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr)
@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb (tlb, p4d, addr) p4d_free((tlb)->mm, p4d) +#define __p4d_free_tlb (tlb, p4d, addr) \ +do { \ + if (pgtable_l5_enabled) \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ +} while (0)#endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings (pgd_t *pgd)
@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)#ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb (tlb, pmd, addr) pmd_free((tlb)->mm, pmd) +#define __pmd_free_tlb (tlb, pmd, addr) \ +do { \ + pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ +} while (0)#endif /* __PAGETABLE_PMD_FOLDED */
Signature · 1 line
2.40.0
Alexandre Ghiti
reviewerRe: [PATCH 1/4] riscv: tlb: fix __p*d_free_tlb()
Hi Jisheng,
Quoted 15 lines · Jisheng Zhang
If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.
Signed-off-by Jisheng Zhang<[email protected]>--- arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)arch/riscv/include/asm/pgalloc.h +0
-0
arch/riscv/include/asm/pgalloc.h static inline void pud_free(struct mm_struct *mm, pud_t *pud) · line 102
Quoted 9 lines
arch/riscv/include/asm/pgalloc.h +5
-1@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)__pud_free (mm, pud); } -#define __pud_free_tlb (tlb, pud, addr) pud_free((tlb)->mm, pud) +#define __pud_free_tlb (tlb, pud, addr) \ +do { \ + if (pgtable_l4_enabled) { \ + pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \
The specification indeed states that an sfence.vma must be emitted after a page directory modification. Your change is not enough though since eventually tlb_flush() is called and in this function we should add:
if (tlb->freed_tables)
tlb_flush_mm();
otherwise we are not guaranteed that a "global" sfence.vma is called.
Would you be able to benchmark this change and see the performance impact?
Thanks,
Alex
Quoted 31 lines
arch/riscv/include/asm/pgalloc.h +12
-2+ } \ +} while (0)#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr)@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb (tlb, p4d, addr) p4d_free((tlb)->mm, p4d) +#define __p4d_free_tlb (tlb, p4d, addr) \ +do { \ + if (pgtable_l5_enabled) \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ +} while (0)#endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings (pgd_t *pgd)@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)#ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb (tlb, pmd, addr) pmd_free((tlb)->mm, pmd) +#define __pmd_free_tlb (tlb, pmd, addr) \ +do { \ + pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ +} while (0)#endif /* __PAGETABLE_PMD_FOLDED */
Re: [PATCH 1/4] riscv: tlb: fix __p*d_free_tlb()
Hi Jisheng,
Quoted 31 lines · Alexandre Ghiti
Hi Jisheng,
Quoted 15 lines · Jisheng Zhang
If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.
Signed-off-by Jisheng Zhang<[email protected]>---
arch/riscv/include/asm/pgalloc.h| 20 +++++++++++++++++---1 file changed, 17 insertions(+), 3 deletions(-)arch/riscv/include/asm/pgalloc.h +0
-0arch/riscv/include/asm/pgalloc.h static inline void pud_free(struct mm_struct *mm, · line 103
Quoted 9 lines
arch/riscv/include/asm/pgalloc.h +5
-0@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) __pud_free(mm, pud); } -#define __pud_free_tlb(tlb, pud, addr) pud_free((tlb)->mm, pud) +#define __pud_free_tlb (tlb, pud, addr) \ +do { \ + if (pgtable_l4_enabled) { \ + pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \The specification indeed states that an sfence.vma must be emitted after a page directory modification. Your change is not enough though since eventually
tlb_flush()is called and in this function we should add:
Quoted 2 lines
if (
tlb->freed_tables)
tlb_flush_mm();
I sent a patch for that here https://lore.kernel.org/linux-riscv/[email protected]/
You can add:
Thanks,
Alex
Quoted 37 lines
otherwise we are not guaranteed that a "global" sfence.vma is called.
Would you be able to benchmark this change and see the performance impact?
Thanks,
Alex
Quoted 26 lines
arch/riscv/include/asm/pgalloc.h +12
-0+ } \ +} while (0) #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) +#define __p4d_free_tlb (tlb, p4d, addr) \ +do { \ + if (pgtable_l5_enabled) \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ +} while (0) #endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings(pgd_t *pgd)@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd) +#define __pmd_free_tlb (tlb, pmd, addr) \ +do { \ + pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ +} while (0) #endif /* __PAGETABLE_PMD_FOLDED */
Alexandre Ghiti
reviewerRe: [PATCH 1/4] riscv: tlb: fix __p*d_free_tlb()
In reply to Jisheng ZhangQuoted 4 lines · Jisheng Zhang
If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.
And since this is a fix, it would be worth trying to add a Fixes tag here. Not easy I agree because it fixes several commits (I have 07037db5d479f, e8a62cc26ddf5, d10efa21a9374 and c5e9b2c2ae822 if you implement tlb_flush() as I suggested).
So I would add the latest commit as the Fixes commit (which would be c5e9b2c2ae822), and then I'd send a patch to stable for each commit with the right Fixes tag...@Conor: let me know if you have a simpler idea or if this is wrong.
Thanks,
Alex
Quoted 50 lines
Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)arch/riscv/include/asm/pgalloc.h +17
-3@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)__pud_free (mm, pud); } -#define __pud_free_tlb (tlb, pud, addr) pud_free((tlb)->mm, pud) +#define __pud_free_tlb (tlb, pud, addr) \ +do { \ + if (pgtable_l4_enabled) { \ + pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ + } \ +} while (0)#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr)@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb (tlb, p4d, addr) p4d_free((tlb)->mm, p4d) +#define __p4d_free_tlb (tlb, p4d, addr) \ +do { \ + if (pgtable_l5_enabled) \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ +} while (0)#endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings (pgd_t *pgd)@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)#ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb (tlb, pmd, addr) pmd_free((tlb)->mm, pmd) +#define __pmd_free_tlb (tlb, pmd, addr) \ +do { \ + pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ +} while (0)#endif /* __PAGETABLE_PMD_FOLDED */
Palmer Dabbelt
maintainerRe: [PATCH 1/4] riscv: tlb: fix __p*d_free_tlb()
Quoted 12 lines · On Thu, 04 Jan 2024 02:55:40 PST (-0800), [email protected] wrote:
Quoted 4 lines · Jisheng Zhang
If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory.
And since this is a fix, it would be worth trying to add a Fixes tag here. Not easy I agree because it fixes several commits (I have 07037db5d479f,
e8a62cc26ddf5,d10efa21a9374andc5e9b2c2ae822if you implementtlb_flush()as I suggested).
Quoted 4 lines
So I would add the latest commit as the Fixes commit (which would be
c5e9b2c2ae822), and then I'd send a patch to stable for each commit with the right Fixes tag...@Conor: let me know if you have a simpler idea or if this is wrong.
I just went with
hopefully that's fine. It's still getting tested, it's batched up with some other stuff and I managed to find a bad merge so it might take a bit...
Quoted 56 lines
Thanks,
Alex
Quoted 50 lines
Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)arch/riscv/include/asm/pgalloc.h +17
-3@@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)__pud_free (mm, pud); } -#define __pud_free_tlb (tlb, pud, addr) pud_free((tlb)->mm, pud) +#define __pud_free_tlb (tlb, pud, addr) \ +do { \ + if (pgtable_l4_enabled) { \ + pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ + } \ +} while (0)#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr)@@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb (tlb, p4d, addr) p4d_free((tlb)->mm, p4d) +#define __p4d_free_tlb (tlb, p4d, addr) \ +do { \ + if (pgtable_l5_enabled) \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ +} while (0)#endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings (pgd_t *pgd)@@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)#ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb (tlb, pmd, addr) pmd_free((tlb)->mm, pmd) +#define __pmd_free_tlb (tlb, pmd, addr) \ +do { \ + pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ +} while (0)#endif /* __PAGETABLE_PMD_FOLDED */
PATCH 2/4 riscv: tlb: convert __p*d_free_tlb() to inline functions
In reply to Jisheng ZhangThis is to prepare for enabling MMU_GATHER_RCU_TABLE_FREE. No functionality changes.
--- arch/riscv/include/asm/pgalloc.h | 54 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 22 deletions(-)
arch/riscv/include/asm/pgalloc.h +32 -22
@@ -95,13 +95,16 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)__pud_free (mm, pud); } -#define __pud_free_tlb (tlb, pud, addr) \ -do { \ - if (pgtable_l4_enabled) { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ - } \ -} while (0) +static inline void __pud_free_tlb (struct mmu_gather *tlb,pud_t *pud, +unsigned long addr) +{ +if (pgtable_l4_enabled) { +struct ptdesc *ptdesc =virt_to_ptdesc (pud); + +pagetable_pud_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); + } +}#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr)
@@ -130,11 +133,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb (tlb, p4d, addr) \ -do { \ - if (pgtable_l5_enabled) \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ -} while (0) +static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d, +unsigned long addr) +{ +if (pgtable_l5_enabled) +tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +}#endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings (pgd_t *pgd)
@@ -159,19 +163,25 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)#ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb (tlb, pmd, addr) \ -do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ -} while (0) +static inline void __pmd_free_tlb (struct mmu_gather *tlb,pmd_t *pmd, +unsigned long addr) +{ +struct ptdesc *ptdesc =virt_to_ptdesc (pmd); + +pagetable_pmd_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); +}#endif /* __PAGETABLE_PMD_FOLDED */ -#define __pte_free_tlb (tlb, pte, buf) \ -do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\ -} while (0) +static inline void __pte_free_tlb (struct mmu_gather *tlb,pgtable_t pte, +unsigned long addr) +{ +struct ptdesc *ptdesc =page_ptdesc (pte); + +pagetable_pte_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); +}#endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_PGALLOC_H */
Signature · 1 line
2.40.0
Re: [External] [PATCH 2/4] riscv: tlb: convert __p*d_free_tlb() to inline functions
Hi Jisheng,
Quoted 92 lines · Jisheng Zhang
This is to prepare for enabling
MMU_GATHER_RCU_TABLE_FREE. No functionality changes.Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/include/asm/pgalloc.h | 54 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 22 deletions(-)arch/riscv/include/asm/pgalloc.h +32
-23@@ -95,13 +95,16 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)__pud_free (mm, pud); } -#define __pud_free_tlb (tlb, pud, addr) \ -do { \ - if (pgtable_l4_enabled) { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ - } \ -} while (0) +static inline void __pud_free_tlb (struct mmu_gather *tlb,pud_t *pud, +unsigned long addr) +{ +if (pgtable_l4_enabled) { +struct ptdesc *ptdesc =virt_to_ptdesc (pud); + +pagetable_pud_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); + } +}#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr)@@ -130,11 +133,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb (tlb, p4d, addr) \ -do { \ - if (pgtable_l5_enabled) \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ -} while (0) +static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d, +unsigned long addr) +{ +if (pgtable_l5_enabled) +tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +}#endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings (pgd_t *pgd)@@ -159,19 +163,25 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)#ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb (tlb, pmd, addr) \ -do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ -} while (0) +static inline void __pmd_free_tlb (struct mmu_gather *tlb,pmd_t *pmd, +unsigned long addr) +{ +struct ptdesc *ptdesc =virt_to_ptdesc (pmd); + +pagetable_pmd_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); +}#endif /* __PAGETABLE_PMD_FOLDED */ -#define __pte_free_tlb (tlb, pte, buf) \ -do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\ -} while (0) +static inline void __pte_free_tlb (struct mmu_gather *tlb,pgtable_t pte, +unsigned long addr) +{ +struct ptdesc *ptdesc =page_ptdesc (pte); + +pagetable_pte_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); +}#endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_PGALLOC_H */ --
Quoted 2 lines
2.40.0
Why is it necessary to convert to inline functions?
Thanks,
Yunhui
Re: [External] [PATCH 2/4] riscv: tlb: convert __p*d_free_tlb() to inline functions
Quoted 1 line · yunhui cui
Hi Jisheng,
Hi,
Quoted 92 lines · Jisheng Zhang
Quoted 90 lines
This is to prepare for enabling
MMU_GATHER_RCU_TABLE_FREE. No functionality changes.Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/include/asm/pgalloc.h | 54 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 22 deletions(-)arch/riscv/include/asm/pgalloc.h +32
-22@@ -95,13 +95,16 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)__pud_free (mm, pud); } -#define __pud_free_tlb (tlb, pud, addr) \ -do { \ - if (pgtable_l4_enabled) { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ - } \ -} while (0) +static inline void __pud_free_tlb (struct mmu_gather *tlb,pud_t *pud, +unsigned long addr) +{ +if (pgtable_l4_enabled) { +struct ptdesc *ptdesc =virt_to_ptdesc (pud); + +pagetable_pud_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); + } +}#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr)@@ -130,11 +133,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb (tlb, p4d, addr) \ -do { \ - if (pgtable_l5_enabled) \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ -} while (0) +static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d, +unsigned long addr) +{ +if (pgtable_l5_enabled) +tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +}#endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings (pgd_t *pgd)@@ -159,19 +163,25 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)#ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb (tlb, pmd, addr) \ -do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ -} while (0) +static inline void __pmd_free_tlb (struct mmu_gather *tlb,pmd_t *pmd, +unsigned long addr) +{ +struct ptdesc *ptdesc =virt_to_ptdesc (pmd); + +pagetable_pmd_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); +}#endif /* __PAGETABLE_PMD_FOLDED */ -#define __pte_free_tlb (tlb, pte, buf) \ -do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\ -} while (0) +static inline void __pte_free_tlb (struct mmu_gather *tlb,pgtable_t pte, +unsigned long addr) +{ +struct ptdesc *ptdesc =page_ptdesc (pte); + +pagetable_pte_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); +}#endif /* CONFIG_MMU */
Quoted 6 lines
Quoted 4 lines
arch/riscv/include/asm/pgalloc.h +0
-1#endif /* _ASM_RISCV_PGALLOC_H */ --2.40.0
Why is it necessary to convert to inline functions?
Hmm, it's not necessary but a plus, the inline version's readability and maintainability is better than macros
Regards
Re: [PATCH 2/4] riscv: tlb: convert __p*d_free_tlb() to inline functions
In reply to Jisheng ZhangQuoted 80 lines · Jisheng Zhang
This is to prepare for enabling
MMU_GATHER_RCU_TABLE_FREE. No functionality changes.Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/include/asm/pgalloc.h | 54 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 22 deletions(-)arch/riscv/include/asm/pgalloc.h +25
-22@@ -95,13 +95,16 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)__pud_free (mm, pud); } -#define __pud_free_tlb (tlb, pud, addr) \ -do { \ - if (pgtable_l4_enabled) { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ - } \ -} while (0) +static inline void __pud_free_tlb (struct mmu_gather *tlb,pud_t *pud, +unsigned long addr) +{ +if (pgtable_l4_enabled) { +struct ptdesc *ptdesc =virt_to_ptdesc (pud); + +pagetable_pud_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); + } +}#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr)@@ -130,11 +133,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb (tlb, p4d, addr) \ -do { \ - if (pgtable_l5_enabled) \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ -} while (0) +static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d, +unsigned long addr) +{ +if (pgtable_l5_enabled) +tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +}#endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings (pgd_t *pgd)@@ -159,19 +163,25 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)#ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb (tlb, pmd, addr) \ -do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ -} while (0) +static inline void __pmd_free_tlb (struct mmu_gather *tlb,pmd_t *pmd, +unsigned long addr) +{ +struct ptdesc *ptdesc =virt_to_ptdesc (pmd); + +pagetable_pmd_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); +}#endif /* __PAGETABLE_PMD_FOLDED */ -#define __pte_free_tlb (tlb, pte, buf) \ -do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\ -} while (0) +static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
arch/riscv/include/asm/pgalloc.h static inline pgd_t *pgd_alloc(struct mm_struct *mm) · line 187
Quoted 10 lines
arch/riscv/include/asm/pgalloc.h +7
-0+ unsigned long addr) +{ +struct ptdesc *ptdesc =page_ptdesc (pte); + +pagetable_pte_dtor (ptdesc); +tlb_remove_page_ptdesc (tlb, ptdesc); +}#endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_PGALLOC_H */
You can add:
Thanks,
Alex
PATCH 3/4 riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU
In reply to Jisheng ZhangIn order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.
riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the riscv_ipi_for_rfence is true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually the riscv_ipi_for_rfence is false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h for more details.
This patch enables MMU_GATHER_RCU_TABLE_FREE, then use
*tlb_remove_page_ptdesc() for those platforms which use IPI to perform TLB shootdown;
*tlb_remove_ptdesc() for those platforms which use SBI to perform TLB shootdown;
Both case mean that disabling interrupts will block the free and protect the fast gup page walker.
--- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgalloc.h | 23 ++++++++++++++++++----- arch/riscv/include/asm/tlb.h | 18 ++++++++++++++++++ 3 files changed, 37 insertions(+), 5 deletions(-)
arch/riscv/Kconfig +1 -0
@@ -147,6 +147,7 @@ config RISCV select IRQ_FORCED_THREADING select KASAN_VMALLOC if KASAN select LOCK_MM_AND_FIND_VMA + select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU select MODULES_USE_ELF_RELA if MODULES select MODULE_SECTIONS if MODULES select OF
arch/riscv/include/asm/pgalloc.h +18 -5
@@ -102,7 +102,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,struct ptdesc *ptdesc =virt_to_ptdesc (pud);pagetable_pud_dtor (ptdesc); -tlb_remove_page_ptdesc (tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else +tlb_remove_ptdesc (tlb, ptdesc); } }
@@ -136,8 +139,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr) { -if (pgtable_l5_enabled) -tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +if (pgtable_l5_enabled) { +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +else +tlb_remove_ptdesc (tlb,virt_to_ptdesc (p4d)); + } }#endif /* __PAGETABLE_PMD_FOLDED */
@@ -169,7 +176,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,struct ptdesc *ptdesc =virt_to_ptdesc (pmd);pagetable_pmd_dtor (ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else + tlb_remove_ptdesc(tlb, ptdesc); }#endif /* __PAGETABLE_PMD_FOLDED */
@@ -180,7 +190,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,struct ptdesc *ptdesc =page_ptdesc (pte);pagetable_pte_dtor (ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else + tlb_remove_ptdesc(tlb, ptdesc); }#endif /* CONFIG_MMU */
arch/riscv/include/asm/tlb.h +18 -0
@@ -10,6 +10,24 @@ struct mmu_gather;static void tlb_flush (struct mmu_gather *tlb); +#ifdef CONFIG_MMU +#include <linux/swap.h> + +/* +* While riscv platforms with riscv_ipi_for_rfence as true require an IPI to +* perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use +* SBI to perform TLB shootdown. To keep software pagetable walkers safe in this +* case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the +* comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h +* for more details. +*/ +static inline void __tlb_remove_table (void *table) +{ +free_page_and_swap_cache (table); +} + +#endif /* CONFIG_MMU */ +#define tlb_flush tlb_flush#include <asm-generic/tlb.h>
Signature · 1 line
2.40.0
Alexandre Ghiti
reviewerRe: [PATCH 3/4] riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU
Quoted 14 lines · Jisheng Zhang
In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.
riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the
riscv_ipi_for_rfenceis true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually theriscv_ipi_for_rfenceis false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdefCONFIG_MMU_GATHER_RCU_TABLE_FREE' ininclude/asm-generic/tlb.hfor more details.
Quoted 7 lines
This patch enables
MMU_GATHER_RCU_TABLE_FREE, then use
*tlb_remove_page_ptdesc()for those platforms which use IPI to perform TLB shootdown;
*tlb_remove_ptdesc()for those platforms which use SBI to perform TLB shootdown;
Can you elaborate a bit more on what those functions do differently and why we need to differentiate IPI vs SBI TLB shootdown? I don't understand this.
Thanks,
Alex
Quoted 106 lines
Both case mean that disabling interrupts will block the free and protect the fast gup page walker.
Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgalloc.h | 23 ++++++++++++++++++----- arch/riscv/include/asm/tlb.h | 18 ++++++++++++++++++ 3 files changed, 37 insertions(+), 5 deletions(-)arch/riscv/Kconfig +1
-0@@ -147,6 +147,7 @@ config RISCV select IRQ_FORCED_THREADING select KASAN_VMALLOC if KASAN select LOCK_MM_AND_FIND_VMA + select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU select MODULES_USE_ELF_RELA if MODULES select MODULE_SECTIONS if MODULES select OFarch/riscv/include/asm/pgalloc.h +18
-5@@ -102,7 +102,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,struct ptdesc *ptdesc =virt_to_ptdesc (pud);pagetable_pud_dtor (ptdesc); -tlb_remove_page_ptdesc (tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else +tlb_remove_ptdesc (tlb, ptdesc); } }@@ -136,8 +139,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr) { -if (pgtable_l5_enabled) -tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +if (pgtable_l5_enabled) { +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +else +tlb_remove_ptdesc (tlb,virt_to_ptdesc (p4d)); + } }#endif /* __PAGETABLE_PMD_FOLDED */ @@ -169,7 +176,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,struct ptdesc *ptdesc =virt_to_ptdesc (pmd);pagetable_pmd_dtor (ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else + tlb_remove_ptdesc(tlb, ptdesc); }#endif /* __PAGETABLE_PMD_FOLDED */ @@ -180,7 +190,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,struct ptdesc *ptdesc =page_ptdesc (pte);pagetable_pte_dtor (ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else + tlb_remove_ptdesc(tlb, ptdesc); }#endif /* CONFIG_MMU */ arch/riscv/include/asm/tlb.h +18
-0@@ -10,6 +10,24 @@ struct mmu_gather;static void tlb_flush (struct mmu_gather *tlb); +#ifdef CONFIG_MMU +#include <linux/swap.h> + +/* +* While riscv platforms with riscv_ipi_for_rfence as true require an IPI to +* perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use +* SBI to perform TLB shootdown. To keep software pagetable walkers safe in this +* case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the +* comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h +* for more details. +*/ +static inline void __tlb_remove_table (void *table) +{ +free_page_and_swap_cache (table); +} + +#endif /* CONFIG_MMU */ +#define tlb_flush tlb_flush#include <asm-generic/tlb.h>
Re: [PATCH 3/4] riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU
Quoted 20 lines · Alexandre Ghiti
Quoted 19 lines · Jisheng Zhang
In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.
riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the
riscv_ipi_for_rfenceis true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually theriscv_ipi_for_rfenceis false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdefCONFIG_MMU_GATHER_RCU_TABLE_FREE' ininclude/asm-generic/tlb.hfor more details.This patch enables
MMU_GATHER_RCU_TABLE_FREE, then use
*tlb_remove_page_ptdesc()for those platforms which use IPI to perform TLB shootdown;
Quoted 6 lines
Quoted 2 lines
*tlb_remove_ptdesc()for those platforms which use SBI to perform TLB shootdown;Can you elaborate a bit more on what those functions do differently and why we need to differentiate IPI vs SBI TLB shootdown? I don't understand this.
Hi Alex,
If IPI, the local_irq_save in lockless_pages_from_mm() of fast gup code path will block page table pages from being freed, I think the comments there is execellent.
If SBI, the local_irq_save in lockless_pages_from_mm() can't acchieve the goal however. Because local_irq_save() only disable S-privilege IPI irq, it can't disable M-privilege's, which the SBI implementation use to shootdown TLB entry. So we need MMU_GATHER_RCU_TABLE_FREE helper for SBI case.
Thanks
Quoted 97 lines
Quoted 97 lines
Both case mean that disabling interrupts will block the free and protect the fast gup page walker.
Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgalloc.h | 23 ++++++++++++++++++----- arch/riscv/include/asm/tlb.h | 18 ++++++++++++++++++ 3 files changed, 37 insertions(+), 5 deletions(-)arch/riscv/Kconfig +1
-0@@ -147,6 +147,7 @@ config RISCV select IRQ_FORCED_THREADING select KASAN_VMALLOC if KASAN select LOCK_MM_AND_FIND_VMA + select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU select MODULES_USE_ELF_RELA if MODULES select MODULE_SECTIONS if MODULES select OFarch/riscv/include/asm/pgalloc.h +18
-5@@ -102,7 +102,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,struct ptdesc *ptdesc =virt_to_ptdesc (pud);pagetable_pud_dtor (ptdesc); -tlb_remove_page_ptdesc (tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else +tlb_remove_ptdesc (tlb, ptdesc); } }@@ -136,8 +139,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr) { -if (pgtable_l5_enabled) -tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +if (pgtable_l5_enabled) { +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +else +tlb_remove_ptdesc (tlb,virt_to_ptdesc (p4d)); + } }#endif /* __PAGETABLE_PMD_FOLDED */ @@ -169,7 +176,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,struct ptdesc *ptdesc =virt_to_ptdesc (pmd);pagetable_pmd_dtor (ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else + tlb_remove_ptdesc(tlb, ptdesc); }#endif /* __PAGETABLE_PMD_FOLDED */ @@ -180,7 +190,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,struct ptdesc *ptdesc =page_ptdesc (pte);pagetable_pte_dtor (ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else + tlb_remove_ptdesc(tlb, ptdesc); }#endif /* CONFIG_MMU */ arch/riscv/include/asm/tlb.h +18
-0@@ -10,6 +10,24 @@ struct mmu_gather;static void tlb_flush (struct mmu_gather *tlb); +#ifdef CONFIG_MMU +#include <linux/swap.h> + +/* +* While riscv platforms with riscv_ipi_for_rfence as true require an IPI to +* perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use +* SBI to perform TLB shootdown. To keep software pagetable walkers safe in this +* case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the +* comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h +* for more details. +*/ +static inline void __tlb_remove_table (void *table) +{ +free_page_and_swap_cache (table); +} + +#endif /* CONFIG_MMU */ +#define tlb_flush tlb_flush#include <asm-generic/tlb.h>
Re: [PATCH 3/4] riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU
Hi Jisheng,
Quoted 32 lines · Jisheng Zhang
Quoted 17 lines · Alexandre Ghiti
Quoted 16 lines · Jisheng Zhang
In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it.
riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the
riscv_ipi_for_rfenceis true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually theriscv_ipi_for_rfenceis false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdefCONFIG_MMU_GATHER_RCU_TABLE_FREE' ininclude/asm-generic/tlb.hfor more details.This patch enables
MMU_GATHER_RCU_TABLE_FREE, then useQuoted 8 lines
Quoted 5 lines
*tlb_remove_page_ptdesc()for those platforms which use IPI to perform TLB shootdown;
*tlb_remove_ptdesc()for those platforms which use SBI to perform TLB shootdown;Can you elaborate a bit more on what those functions do differently and why we need to differentiate IPI vs SBI TLB shootdown? I don't understand this.
Hi Alex,
If IPI, the
local_irq_saveinlockless_pages_from_mm()of fast gup code path will block page table pages from being freed, I think the comments there is execellent.
Quoted 5 lines
If SBI, the
local_irq_saveinlockless_pages_from_mm()can't acchieve the goal however. Becauselocal_irq_save()only disable S-privilege IPI irq, it can't disable M-privilege's, which the SBI implementation use to shootdown TLB entry. So we needMMU_GATHER_RCU_TABLE_FREEhelper for SBI case.
Ok, I get it now, can you add the following link to your commit description https://elixir.bootlin.com/linux/v6.6/source/mm/mmu_gather.c#L162 ? It describes the problem very clearly.
Quoted 79 lines
Thanks
Quoted 77 lines
Quoted 77 lines
Both case mean that disabling interrupts will block the free and protect the fast gup page walker.
Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgalloc.h | 23 ++++++++++++++++++----- arch/riscv/include/asm/tlb.h | 18 ++++++++++++++++++ 3 files changed, 37 insertions(+), 5 deletions(-)arch/riscv/Kconfig +1
-0@@ -147,6 +147,7 @@ config RISCV select IRQ_FORCED_THREADING select KASAN_VMALLOC if KASAN select LOCK_MM_AND_FIND_VMA + select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU select MODULES_USE_ELF_RELA if MODULES select MODULE_SECTIONS if MODULES select OFarch/riscv/include/asm/pgalloc.h +18
-5@@ -102,7 +102,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,struct ptdesc *ptdesc =virt_to_ptdesc (pud);pagetable_pud_dtor (ptdesc); -tlb_remove_page_ptdesc (tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else +tlb_remove_ptdesc (tlb, ptdesc); } }@@ -136,8 +139,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr) { -if (pgtable_l5_enabled) -tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +if (pgtable_l5_enabled) { +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb,virt_to_ptdesc (p4d)); +else +tlb_remove_ptdesc (tlb,virt_to_ptdesc (p4d)); + } }#endif /* __PAGETABLE_PMD_FOLDED */ @@ -169,7 +176,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,struct ptdesc *ptdesc =virt_to_ptdesc (pmd);pagetable_pmd_dtor (ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else + tlb_remove_ptdesc(tlb, ptdesc); }#endif /* __PAGETABLE_PMD_FOLDED */ @@ -180,7 +190,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,struct ptdesc *ptdesc =page_ptdesc (pte);pagetable_pte_dtor (ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); +if (riscv_use_ipi_for_rfence ()) +tlb_remove_page_ptdesc (tlb, ptdesc); +else + tlb_remove_ptdesc(tlb, ptdesc); }#endif /* CONFIG_MMU */ arch/riscv/include/asm/tlb.h +1
-0@@ -10,6 +10,24 @@ struct mmu_gather;static void tlb_flush (struct mmu_gather *tlb); +#ifdef CONFIG_MMU
Quoted 8 lines
Quoted 8 lines
arch/riscv/include/asm/tlb.h struct mmu_gather; · line 21
Quoted 8 lines
arch/riscv/include/asm/tlb.h +8
-0+#include <linux/swap.h> + +/* +* While riscv platforms with riscv_ipi_for_rfence as true require an IPI to +* perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use +* SBI to perform TLB shootdown. To keep software pagetable walkers safe in this +* case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the +* comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h
A direct link would be better, I did not find the comment you mention here and even though, it can still move around later.
And then you can add:
Thanks!
Alex
Quoted 15 lines
Quoted 11 lines
arch/riscv/include/asm/tlb.h struct mmu_gather; · line 32
Quoted 11 lines
arch/riscv/include/asm/tlb.h +9
-0+* for more details. +*/ +static inline void __tlb_remove_table (void *table) +{ +free_page_and_swap_cache (table); +} + +#endif /* CONFIG_MMU */ +#define tlb_flush tlb_flush#include <asm-generic/tlb.h> _______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH 4/4 riscv: enable HAVE_FAST_GUP if MMU
In reply to Jisheng ZhangActivate the fast gup for riscv mmu platforms. Here are some GUP_FAST_BENCHMARK performance numbers:
Before the patch:GUP_FAST_BENCHMARK: Time: get:53203 put:5085 us
After the patch:GUP_FAST_BENCHMARK: Time: get:17711 put:5060 us
The get time is reduced by 66.7%! IOW, 3x get speed!
--- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgtable.h | 6 ++++++ 2 files changed, 7 insertions(+)
arch/riscv/Kconfig +1 -0
@@ -119,6 +119,7 @@ config RISCV select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION select HAVE_EBPF_JIT if MMU + select HAVE_FAST_GUP if MMU select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUNCTION_ERROR_INJECTION select HAVE_GCC_PLUGINS
arch/riscv/include/asm/pgtable.h +6 -0
@@ -673,6 +673,12 @@ static inline int pmd_write(pmd_t pmd)return pte_write (pmd_pte (pmd)); } +#define pud_write pud_write +static inline int pud_write (pud_t pud) +{ +return pte_write (pud_pte (pud)); +} +static inline int pmd_dirty (pmd_t pmd) {return pte_dirty (pmd_pte (pmd));
Signature · 1 line
2.40.0
Alexandre Ghiti
reviewerRe: [PATCH 4/4] riscv: enable HAVE_FAST_GUP if MMU
Quoted 8 lines · Jisheng Zhang
Activate the fast gup for riscv mmu platforms. Here are some
GUP_FAST_BENCHMARKperformance numbers:Before the patch:
GUP_FAST_BENCHMARK: Time: get:53203 put:5085 usAfter the patch:
GUP_FAST_BENCHMARK: Time: get:17711 put:5060 us
On which platform did you run this benchmark?
Quoted 2 lines
The get time is reduced by 66.7%! IOW, 3x get speed!
Well done!
Thanks,
Alex
Quoted 36 lines
Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgtable.h | 6 ++++++ 2 files changed, 7 insertions(+)arch/riscv/Kconfig +1
-0@@ -119,6 +119,7 @@ config RISCV select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION select HAVE_EBPF_JIT if MMU + select HAVE_FAST_GUP if MMU select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUNCTION_ERROR_INJECTION select HAVE_GCC_PLUGINSarch/riscv/include/asm/pgtable.h +6
-0@@ -673,6 +673,12 @@ static inline int pmd_write(pmd_t pmd)return pte_write (pmd_pte (pmd)); } +#define pud_write pud_write +static inline int pud_write (pud_t pud) +{ +return pte_write (pud_pte (pud)); +} +static inline int pmd_dirty (pmd_t pmd) {return pte_dirty (pmd_pte (pmd));
Re: [PATCH 4/4] riscv: enable HAVE_FAST_GUP if MMU
Quoted 4 lines · Alexandre Ghiti
Quoted 3 lines · Jisheng Zhang
Activate the fast gup for riscv mmu platforms. Here are some
GUP_FAST_BENCHMARKperformance numbers:
Quoted 8 lines
Quoted 5 lines
Before the patch:
GUP_FAST_BENCHMARK: Time: get:53203 put:5085 usAfter the patch:
GUP_FAST_BENCHMARK: Time: get:17711 put:5060 usOn which platform did you run this benchmark?
T-HEAD th1520(cpufreq isn't enabled since the clk/pll isn't upstreamed, so cpu is running at the default freq set by u-boot)
Quoted 48 lines
Quoted 4 lines
The get time is reduced by 66.7%! IOW, 3x get speed!
Well done!
Thanks,
Alex
Quoted 35 lines
Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgtable.h | 6 ++++++ 2 files changed, 7 insertions(+)arch/riscv/Kconfig +1
-0@@ -119,6 +119,7 @@ config RISCV select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION select HAVE_EBPF_JIT if MMU + select HAVE_FAST_GUP if MMU select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUNCTION_ERROR_INJECTION select HAVE_GCC_PLUGINSarch/riscv/include/asm/pgtable.h +6
-0@@ -673,6 +673,12 @@ static inline int pmd_write(pmd_t pmd)return pte_write (pmd_pte (pmd)); } +#define pud_write pud_write +static inline int pud_write (pud_t pud) +{ +return pte_write (pud_pte (pud)); +} +static inline int pmd_dirty (pmd_t pmd) {return pte_dirty (pmd_pte (pmd));
Re: [PATCH 4/4] riscv: enable HAVE_FAST_GUP if MMU
Quoted 50 lines · Jisheng Zhang
Quoted 11 lines · Alexandre Ghiti
Quoted 8 lines · Jisheng Zhang
Activate the fast gup for riscv mmu platforms. Here are some
GUP_FAST_BENCHMARKperformance numbers:Before the patch:
GUP_FAST_BENCHMARK: Time: get:53203 put:5085 usAfter the patch:
GUP_FAST_BENCHMARK: Time: get:17711 put:5060 usOn which platform did you run this benchmark?
T-HEAD th1520(cpufreq isn't enabled since the clk/pll isn't upstreamed, so cpu is running at the default freq set by u-boot)
Quoted 36 lines
Quoted 2 lines
The get time is reduced by 66.7%! IOW, 3x get speed!
Well done!
Thanks,
Alex
Quoted 26 lines
Signed-off-by Jisheng Zhang <[email protected]>--- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgtable.h | 6 ++++++ 2 files changed, 7 insertions(+)arch/riscv/Kconfig +1
-0@@ -119,6 +119,7 @@ config RISCV select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION select HAVE_EBPF_JIT if MMU + select HAVE_FAST_GUP if MMU select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUNCTION_ERROR_INJECTION select HAVE_GCC_PLUGINSarch/riscv/include/asm/pgtable.h +1
-0@@ -673,6 +673,12 @@ static inline int pmd_write(pmd_t pmd)return pte_write (pmd_pte (pmd)); } +#define pud_write pud_write
Quoted 8 lines
Quoted 8 lines
arch/riscv/include/asm/pgtable.h static inline int pmd_write(pmd_t pmd) · line 684
Quoted 8 lines
arch/riscv/include/asm/pgtable.h +5
-0+static inline int pud_write (pud_t pud) +{ +return pte_write (pud_pte (pud)); +} +static inline int pmd_dirty (pmd_t pmd) {return pte_dirty (pmd_pte (pmd));
Thanks, you can add:
Thanks,
Alex
Quoted 4 lines
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:
Hello:
This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:
Quoted 9 lines · On Wed, 20 Dec 2023 01:50:42 +0800 you wrote:
This series adds fast gup support to riscv.
The First patch fixes a bug in __p*d_free_tlb(). Per the riscv privileged spec, if non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must.
The 2nd patch is a preparation patch.
[...]
Here is the summary with links:
- [1/4] riscv: tlb: fix __p*d_free_tlb()
https://git.kernel.org/riscv/c/8246601a7d39 - [2/4] riscv: tlb: convert __p*d_free_tlb() to inline functions https://git.kernel.org/riscv/c/40d1bb92a493 - [3/4] riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU https://git.kernel.org/riscv/c/69be3fb111e7 - [4/4] riscv: enable HAVE_FAST_GUP if MMU https://git.kernel.org/riscv/c/3f910b7a522e
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html