move pagetable_*_dtor() to __tlb_remove_table()
Subsystems (26)
v5 cover move pagetable_*_dtor() to __tlb_remove_table()
Changes in v5:
- cancel the move of p4d_free_tlb()'s location in [PATCH v4 06/15] (Alexander Gordeev)
- fix the missing pagetable_dtor() in [PATCH v4 08/15] (Kevin Brodsky)
- change the subject and description in [PATCH v4 12/15]
(Alexander Gordeev)
- remove the redundant __HAVE_ARCH_TLB_REMOVE_TABLE definition in [PATCH v4 13/15] (Andreas Larsson)
- add "mm: pgtable: completely move pagetable_dtor() to generic tlb_remove_table()" (Kevin Brodsky)
- add "x86: pgtable: convert __tlb_remove_table() to use struct ptdesc"
- collect Acked-bys and Reviewed-bys
Changes in v4:
- remove [PATCH v3 15/17] and [PATCH v3 16/17] (Mike Rapoport) (the tlb_remove_page_ptdesc() and tlb_remove_ptdesc() are intermediate products of the project: https://kernelnewbies.org/MatthewWilcox/Memdescs,
so keep them)
- collect Acked-by
Changes in v3:
- take patch #5 and #6 from Kevin Brodsky's patch series below.
- separate the statistics part from [PATCH v2 02/15] as [PATCH v3 04/17], and replace the rest part with Kevin Brodsky's patch #6
(Alexander Gordeev and Kevin Brodsky)
- change the commit message of [PATCH v2 10/15] and [PATCH v2 11/15] (Alexander Gordeev)
- fix the bug introduced by [PATCH v2 11/15]
(Peter Zijlstra)
- rebase onto the next-20241220
Changes in v2:
- add [PATCH v2 13|14|15/15] (suggested by Peter Zijlstra)
- add Originally-bys and Suggested-bys
- rebase onto the next-20241218
Hi all,
As proposed [1] by Peter Zijlstra below, this patch series aims to move pagetable_*_dtor() into __tlb_remove_table(). This will cleanup pagetable_*_dtor() a bit and more gracefully fix the UAF issue [2] reported by syzbot.
``` Notably: - s390 pud isn't calling the existing pagetable_pud_[cd]tor() - none of the p4d things have pagetable_p4d_[cd]tor() (x86,arm64,s390,riscv) and they have inconsistent accounting - while much of the _ctor calls are in generic code, many of the _dtor calls are in arch code for hysterial raisins, this could easily be fixed - if we fix ptlock_free() to handle NULL, then all the _dtor() functions can use it, and we can observe they're all identical and can be folded after all that cleanup, you can move the _dtor from *_free_tlb() into tlb_remove_table() -- which for the above case, would then have it called from __tlb_remove_table_free(). ```
And hi Andrew, I developed the code based on the latest linux-next, so I reverted the "mm: pgtable: make ptlock be freed by RCU" first. Once the review of this patch series is completed, the "mm: pgtable: make ptlock be freed by RCU" can be dropped directly from mm tree, and this revert patch will not be needed.
This series is based on next-20241220. And I tested this patch series on x86 and only cross-compiled it on arm, arm64, powerpc, riscv, s390 and sparc.
Comments and suggestions are welcome!
Thanks,
Qi
[1]. https://lore.kernel.org/all/[email protected]/
[2]. https://lore.kernel.org/all/[email protected]/
Kevin Brodsky (2):
riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one
asm-generic: pgalloc: Provide generic p4d_{alloc_one,free}
Qi Zheng (15):
Revert "mm: pgtable: make ptlock be freed by RCU"
mm: pgtable: add statistics for P4D level page table
arm64: pgtable: use mmu gather to free p4d level page table
s390: pgtable: add statistics for PUD and P4D level page table
mm: pgtable: introduce pagetable_dtor()
arm: pgtable: move pagetable_dtor() to __tlb_remove_table()
arm64: pgtable: move pagetable_dtor() to __tlb_remove_table()
riscv: pgtable: move pagetable_dtor() to __tlb_remove_table()
x86: pgtable: convert __tlb_remove_table() to use struct ptdesc
x86: pgtable: move pagetable_dtor() to __tlb_remove_table()
s390: pgtable: consolidate PxD and PTE TLB free paths
mm: pgtable: introduce generic __tlb_remove_table()
mm: pgtable: completely move pagetable_dtor() to generic
tlb_remove_table()
mm: pgtable: move __tlb_remove_table_one() in x86 to generic file
mm: pgtable: introduce generic pagetable_dtor_free()
Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 10 ---- arch/arm64/include/asm/pgalloc.h | 18 ------ arch/arm64/include/asm/tlb.h | 21 ++++--- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/include/asm/tlb.h | 1 + arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 69 +++++----------------- arch/riscv/include/asm/tlb.h | 18 ------ arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 31 +++++++--- arch/s390/include/asm/tlb.h | 10 ++-- arch/s390/mm/pgalloc.c | 23 +------- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/include/asm/tlb_64.h | 1 + arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +- arch/x86/include/asm/pgalloc.h | 18 ------ arch/x86/include/asm/tlb.h | 33 ----------- arch/x86/kernel/paravirt.c | 5 +- arch/x86/mm/pgtable.c | 23 ++++---- include/asm-generic/pgalloc.h | 55 +++++++++++++++-- include/asm-generic/tlb.h | 24 ++++++-- include/linux/mm.h | 50 ++++++---------- include/linux/mm_types.h | 9 +-- mm/memory.c | 23 +++----- mm/mmu_gather.c | 20 ++++++- 38 files changed, 211 insertions(+), 299 deletions(-)
Signature · 1 line
2.20.1
PATCH v5 1/17 Revert "mm: pgtable: make ptlock be freed by RCU"
This reverts commit 2f3443770437e49abc39af26962d293851cbab6d.
--- include/linux/mm.h | 2 +- include/linux/mm_types.h | 9 +-------- mm/memory.c | 22 ++++++---------------- 3 files changed, 8 insertions(+), 25 deletions(-)
include/linux/mm.h +1 -1
@@ -2925,7 +2925,7 @@ void ptlock_free(struct ptdesc *ptdesc);static inline spinlock_t *ptlock_ptr (struct ptdesc *ptdesc) { -return &(ptdesc->ptl->ptl); +return ptdesc->ptl; }#else /* ALLOC_SPLIT_PTLOCKS */ static inline void ptlock_cache_init (void )
include/linux/mm_types.h +1 -8
@@ -434,13 +434,6 @@ FOLIO_MATCH(flags, _flags_2a);FOLIO_MATCH (compound_head, _head_2a);#undef FOLIO_MATCH -#if ALLOC_SPLIT_PTLOCKS -struct pt_lock { -spinlock_t ptl; -struct rcu_head rcu; -}; -#endif -/** * struct ptdesc - Memory descriptor for page tables. * @__page_flags: Same as page flags. Powerpc only.
@@ -489,7 +482,7 @@ struct ptdesc {union {unsigned long _pt_pad_2;#if ALLOC_SPLIT_PTLOCKS -struct pt_lock *ptl; +spinlock_t *ptl;#else spinlock_t ptl;#endif
mm/memory.c +6 -16
@@ -7034,34 +7034,24 @@ static struct kmem_cache *page_ptl_cachep;void __init ptlock_cache_init (void ) { - page_ptl_cachep =kmem_cache_create ("page->ptl" ,sizeof (struct pt_lock ),0 , + page_ptl_cachep =kmem_cache_create ("page->ptl" ,sizeof (spinlock_t),0 ,SLAB_PANIC ,NULL ); }bool ptlock_alloc (struct ptdesc *ptdesc) { -struct pt_lock *pt_lock; +spinlock_t *ptl; - pt_lock =kmem_cache_alloc (page_ptl_cachep,GFP_KERNEL ); -if (!pt_lock) + ptl =kmem_cache_alloc (page_ptl_cachep,GFP_KERNEL ); +if (!ptl)return false; - ptdesc->ptl = pt_lock; + ptdesc->ptl = ptl;return true; } -static void ptlock_free_rcu (struct rcu_head *head) -{ -struct pt_lock *pt_lock; - - pt_lock = container_of(head, struct pt_lock, rcu); -kmem_cache_free (page_ptl_cachep, pt_lock); -} -void ptlock_free (struct ptdesc *ptdesc) { -struct pt_lock *pt_lock = ptdesc->ptl; - -call_rcu (&pt_lock->rcu, ptlock_free_rcu); +kmem_cache_free (page_ptl_cachep, ptdesc->ptl); }#endif
Signature · 1 line
2.20.1
PATCH v5 2/17 riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one
In reply to Qi ZhengFrom: Kevin Brodsky <[email protected]>
{pmd,pud,p4d}_alloc_one() is never called if the corresponding page table level is folded, as {pmd,pud,p4d}_alloc() already does the required check. We can therefore remove the runtime page table level checks in {pud,p4d}_alloc_one. The PUD helper becomes equivalent to the generic version, so we remove it altogether.
This is consistent with the way arm64 and x86 handle this situation (runtime check in p4d_free() only).
--- arch/riscv/include/asm/pgalloc.h | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-)
arch/riscv/include/asm/pgalloc.h +4 -18
@@ -12,7 +12,6 @@#include <asm/tlb.h> #ifdef CONFIG_MMU -#define __HAVE_ARCH_PUD_ALLOC_ONE#define __HAVE_ARCH_PUD_FREE#include <asm-generic/pgalloc.h>
@@ -88,15 +87,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, } } -#define pud_alloc_one pud_alloc_one -static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr) -{ -if (pgtable_l4_enabled) -return __pud_alloc_one (mm, addr); - -return NULL ; -} -#define pud_free pud_freestatic inline void pud_free (struct mm_struct *mm,pud_t *pud) {
@@ -118,15 +108,11 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr) { -if (pgtable_l5_enabled) { -gfp_t gfp =GFP_PGTABLE_USER ; - -if (mm == &init_mm) - gfp =GFP_PGTABLE_KERNEL ; -return (p4d_t *)get_zeroed_page (gfp); - } +gfp_t gfp =GFP_PGTABLE_USER ; -return NULL ; +if (mm == &init_mm) + gfp =GFP_PGTABLE_KERNEL ; +return (p4d_t *)get_zeroed_page (gfp); }static inline void __p4d_free (struct mm_struct *mm,p4d_t *p4d)
Signature · 1 line
2.20.1
PATCH v5 3/17 asm-generic: pgalloc: Provide generic p4d_{alloc_one,free}
In reply to Qi ZhengFrom: Kevin Brodsky <[email protected]>
Four architectures currently implement 5-level pgtables: arm64, riscv, x86 and s390. The first three have essentially the same implementation for p4d_alloc_one() and p4d_free(), so we've got an opportunity to reduce duplication like at the lower levels.
Provide a generic version of p4d_alloc_one() and p4d_free(), and make use of it on those architectures.
Their implementation is the same as at PUD level, except that p4d_free() performs a runtime check by calling mm_p4d_folded(). 5-level pgtables depend on a runtime-detected hardware feature on all supported architectures, so we might as well include this check in the generic implementation. No runtime check is required in p4d_alloc_one() as the top-level p4d_alloc() already does the required check.
--- arch/arm64/include/asm/pgalloc.h | 17 ------------ arch/riscv/include/asm/pgalloc.h | 23 ---------------- arch/x86/include/asm/pgalloc.h | 18 ------------- include/asm-generic/pgalloc.h | 45 ++++++++++++++++++++++++++++++++ 4 files changed, 45 insertions(+), 58 deletions(-)
arch/arm64/include/asm/pgalloc.h +0 -17
@@ -85,23 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -static inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr) -{ -gfp_t gfp =GFP_PGTABLE_USER ; - -if (mm == &init_mm) - gfp =GFP_PGTABLE_KERNEL ; -return (p4d_t *)get_zeroed_page (gfp); -} - -static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) -{ -if (!pgtable_l5_enabled ()) -return ; -BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 )); -free_page ((unsigned long )p4d); -} -#define __p4d_free_tlb (tlb, p4d, addr) p4d_free((tlb)->mm, p4d)#else static inline void __pgd_populate (pgd_t *pgdp,phys_addr_t p4dp,pgdval_t prot)
arch/riscv/include/asm/pgalloc.h +0 -23
@@ -105,29 +105,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, } } -#define p4d_alloc_one p4d_alloc_one -static inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr) -{ -gfp_t gfp =GFP_PGTABLE_USER ; - -if (mm == &init_mm) - gfp =GFP_PGTABLE_KERNEL ; -return (p4d_t *)get_zeroed_page (gfp); -} - -static inline void __p4d_free (struct mm_struct *mm,p4d_t *p4d) -{ -BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 )); -free_page ((unsigned long )p4d); -} - -#define p4d_free p4d_free -static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) -{ -if (pgtable_l5_enabled) -__p4d_free (mm, p4d); -} -static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr) {
arch/x86/include/asm/pgalloc.h +0 -18
@@ -147,24 +147,6 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4set_pgd_safe (pgd,__pgd (_PAGE_TABLE |__pa (p4d))); } -static inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr) -{ -gfp_t gfp =GFP_KERNEL_ACCOUNT ; - -if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; -return (p4d_t *)get_zeroed_page (gfp); -} - -static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) -{ -if (!pgtable_l5_enabled ()) -return ; - -BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 )); -free_page ((unsigned long )p4d); -} -extern void ___p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d); static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
include/asm-generic/pgalloc.h +45 -0
51 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH v5 03/17] asm-generic: pgalloc: Provide generic p4d_{alloc_one,free}
Quoted 21 lines · Qi Zheng
From: Kevin Brodsky <[email protected]>
Four architectures currently implement 5-level pgtables: arm64, riscv, x86 and s390. The first three have essentially the same implementation for
p4d_alloc_one()andp4d_free(), so we've got an opportunity to reduce duplication like at the lower levels.Provide a generic version of
p4d_alloc_one()andp4d_free(), and make use of it on those architectures.Their implementation is the same as at PUD level, except that
p4d_free()performs a runtime check by callingmm_p4d_folded(). 5-level pgtables depend on a runtime-detected hardware feature on all supported architectures, so we might as well include this check in the generic implementation. No runtime check is required inp4d_alloc_one()as the top-levelp4d_alloc()already does the required check.Signed-off-by Kevin Brodsky <[email protected]>Acked-by Dave Hansen <[email protected]>Signed-off-by Qi Zheng <[email protected]>
Quoted 6 lines
--- arch/arm64/include/asm/pgalloc.h | 17 ------------ arch/riscv/include/asm/pgalloc.h | 23 ---------------- arch/x86/include/asm/pgalloc.h | 18 ------------- include/asm-generic/pgalloc.h | 45 ++++++++++++++++++++++++++++++++ 4 files changed, 45 insertions(+), 58 deletions(-)
PATCH v5 4/17 mm: pgtable: add statistics for P4D level page table
In reply to Qi ZhengLike other levels of page tables, add statistics for P4D level page table.
--- arch/riscv/include/asm/pgalloc.h | 6 +++++- arch/x86/mm/pgtable.c | 3 +++ include/asm-generic/pgalloc.h | 2 ++ include/linux/mm.h | 16 ++++++++++++++++ 4 files changed, 26 insertions(+), 1 deletion(-)
arch/riscv/include/asm/pgalloc.h +5 -1
@@ -108,8 +108,12 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr) { -if (pgtable_l5_enabled) +if (pgtable_l5_enabled) { +struct ptdesc *ptdesc =virt_to_ptdesc (p4d); + +pagetable_p4d_dtor (ptdesc);riscv_tlb_remove_ptdesc (tlb,virt_to_ptdesc (p4d)); + } }#endif /* __PAGETABLE_PMD_FOLDED */
arch/x86/mm/pgtable.c +3 -0
@@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)#if CONFIG_PGTABLE_LEVELS >4 void ___p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d) { +struct ptdesc *ptdesc =virt_to_ptdesc (p4d); + +pagetable_p4d_dtor (ptdesc);paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (p4d)); }
include/asm-generic/pgalloc.h +2 -0
@@ -230,6 +230,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned longif (!ptdesc)return NULL ; +pagetable_p4d_ctor (ptdesc);return ptdesc_address (ptdesc); }#define __p4d_alloc_one (...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__))
@@ -247,6 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d)struct ptdesc *ptdesc =virt_to_ptdesc (p4d);BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 )); +pagetable_p4d_dtor (ptdesc);pagetable_free (ptdesc); }
include/linux/mm.h +16 -0
@@ -3175,6 +3175,22 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc)lruvec_stat_sub_folio (folio,NR_PAGETABLE ); } +static inline void pagetable_p4d_ctor (struct ptdesc *ptdesc) +{ +struct folio *folio =ptdesc_folio (ptdesc); + +__folio_set_pgtable (folio); +lruvec_stat_add_folio (folio,NR_PAGETABLE ); +} + +static inline void pagetable_p4d_dtor (struct ptdesc *ptdesc) +{ +struct folio *folio =ptdesc_folio (ptdesc); + +__folio_clear_pgtable (folio); +lruvec_stat_sub_folio (folio,NR_PAGETABLE ); +} +extern void __init pagecache_init (void );extern void free_initmem (void );
Signature · 1 line
2.20.1
PATCH v5 5/17 arm64: pgtable: use mmu gather to free p4d level page table
In reply to Qi ZhengLike other levels of page tables, also use mmu gather mechanism to free p4d level page table.
--- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)
arch/arm64/include/asm/pgalloc.h +0 -1
@@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -#define __p4d_free_tlb (tlb, p4d, addr) p4d_free((tlb)->mm, p4d)#else static inline void __pgd_populate (pgd_t *pgdp,phys_addr_t p4dp,pgdval_t prot) {
arch/arm64/include/asm/tlb.h +14 -0
@@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, }#endif +#if CONFIG_PGTABLE_LEVELS >4 +static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4dp, +unsigned long addr) +{ +struct ptdesc *ptdesc =virt_to_ptdesc (p4dp); + +if (!pgtable_l5_enabled ()) +return ; + +pagetable_p4d_dtor (ptdesc); +tlb_remove_ptdesc (tlb, ptdesc); +} +#endif +#endif
Signature · 1 line
2.20.1
Will Deacon
maintainerRe: [PATCH v5 05/17] arm64: pgtable: use mmu gather to free p4d level page table
Quoted 34 lines · Qi Zheng
Like other levels of page tables, also use mmu gather mechanism to free p4d level page table.
Signed-off-by Qi Zheng <[email protected]>Originally-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Kevin Brodsky <[email protected]>--- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)arch/arm64/include/asm/pgalloc.h +0
-1@@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -#define __p4d_free_tlb (tlb, p4d, addr) p4d_free((tlb)->mm, p4d)#else static inline void __pgd_populate (pgd_t *pgdp,phys_addr_t p4dp,pgdval_t prot) {arch/arm64/include/asm/tlb.h +2
-0@@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, }#endif +#if CONFIG_PGTABLE_LEVELS >4 +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp,
arch/arm64/include/asm/tlb.h static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, · line 125
Quoted 10 lines
arch/arm64/include/asm/tlb.h +10
-0+ unsigned long addr) +{ +struct ptdesc *ptdesc =virt_to_ptdesc (p4dp); + +if (!pgtable_l5_enabled ()) +return ; + +pagetable_p4d_dtor (ptdesc); +tlb_remove_ptdesc (tlb, ptdesc); +}
Should we update p4d_free() to call the destructor, too? It looks like it just does free_page() atm.
Will
Re: [PATCH v5 05/17] arm64: pgtable: use mmu gather to free p4d level page table
Hi Will,
Quoted 33 lines · Will Deacon
Quoted 32 lines · Qi Zheng
Like other levels of page tables, also use mmu gather mechanism to free p4d level page table.
Signed-off-by Qi Zheng <[email protected]>Originally-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Kevin Brodsky <[email protected]>--- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)arch/arm64/include/asm/pgalloc.h +0
-1@@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -#define __p4d_free_tlb (tlb, p4d, addr) p4d_free((tlb)->mm, p4d)#else static inline void __pgd_populate (pgd_t *pgdp,phys_addr_t p4dp,pgdval_t prot) {arch/arm64/include/asm/tlb.h +0
-0@@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, }#endif
Quoted 15 lines
arch/arm64/include/asm/tlb.h static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, · line 125
Quoted 12 lines
arch/arm64/include/asm/tlb.h +12
-0+#if CONFIG_PGTABLE_LEVELS >4 +static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4dp, +unsigned long addr) +{ +struct ptdesc *ptdesc =virt_to_ptdesc (p4dp); + +if (!pgtable_l5_enabled ()) +return ; + +pagetable_p4d_dtor (ptdesc); +tlb_remove_ptdesc (tlb, ptdesc); +}Should we update
p4d_free()to call the destructor, too? It looks like it just doesfree_page()atm.
The patch #3 introduces the generic p4d_free() and lets arm64 to use it. The patch #4 adds the destructor to generic p4d_free(). So IIUC, there is no problem here.
Thanks!
Quoted 2 lines
Will
Re: [PATCH v5 05/17] arm64: pgtable: use mmu gather to free p4d level page table
Quoted 50 lines · Qi Zheng
Hi Will,
Quoted 38 lines · Will Deacon
Quoted 37 lines · Qi Zheng
Like other levels of page tables, also use mmu gather mechanism to free p4d level page table.
Signed-off-by Qi Zheng <[email protected]>Originally-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Kevin Brodsky <[email protected]>--- arch/arm64/include/asm/pgalloc.h | 1 - arch/arm64/include/asm/tlb.h | 14 ++++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)arch/arm64/include/asm/pgalloc.h +0
-1@@ -85,7 +85,6 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) __pgd_populate(pgdp, __pa(p4dp), pgdval); } -#define __p4d_free_tlb (tlb, p4d, addr) p4d_free((tlb)->mm, p4d)#else static inline void __pgd_populate (pgd_t *pgdp,phys_addr_t p4dp,pgdval_t prot) {arch/arm64/include/asm/tlb.h +7
-0@@ -111,4 +111,18 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, }#endif +#if CONFIG_PGTABLE_LEVELS >4 +static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4dp, +unsigned long addr) +{ +struct ptdesc *ptdesc =virt_to_ptdesc (p4dp); + +if (!pgtable_l5_enabled ())Quoted 8 lines
arch/arm64/include/asm/tlb.h static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, · line 125
Quoted 5 lines
arch/arm64/include/asm/tlb.h +5
-0+return ; + +pagetable_p4d_dtor (ptdesc); +tlb_remove_ptdesc (tlb, ptdesc); +}Should we update
p4d_free()to call the destructor, too? It looks like it just doesfree_page()atm.
Quoted 3 lines
The patch #3 introduces the generic
p4d_free()and lets arm64 to use it. The patch #4 adds the destructor to genericp4d_free(). So IIUC, there is no problem here.
Sorry, I missed that. In which case:
Will
PATCH v5 6/17 s390: pgtable: add statistics for PUD and P4D level page table
In reply to Qi ZhengLike PMD and PTE level page table, also add statistics for PUD and P4D page table.
--- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++++-------- arch/s390/include/asm/tlb.h | 2 ++ 2 files changed, 23 insertions(+), 8 deletions(-)
arch/s390/include/asm/pgalloc.h +21 -8
@@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) {unsigned long *table =crst_table_alloc (mm); -if (table) -crst_table_init (table, _REGION2_ENTRY_EMPTY); +if (!table) +return NULL ; +crst_table_init (table, _REGION2_ENTRY_EMPTY); +pagetable_p4d_ctor (virt_to_ptdesc (table)); +return (p4d_t *) table; }static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) { -if (!mm_p4d_folded (mm)) -crst_table_free (mm, (unsigned long *) p4d); +if (mm_p4d_folded (mm)) +return ; + +pagetable_p4d_dtor (virt_to_ptdesc (p4d)); +crst_table_free (mm, (unsigned long *) p4d); }static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long address) {unsigned long *table =crst_table_alloc (mm); -if (table) -crst_table_init (table, _REGION3_ENTRY_EMPTY); + +if (!table) +return NULL ; +crst_table_init (table, _REGION3_ENTRY_EMPTY); +pagetable_pud_ctor (virt_to_ptdesc (table)); +return (pud_t *) table; }static inline void pud_free (struct mm_struct *mm,pud_t *pud) { -if (!mm_pud_folded (mm)) -crst_table_free (mm, (unsigned long *) pud); +if (mm_pud_folded (mm)) +return ; + +pagetable_pud_dtor (virt_to_ptdesc (pud)); +crst_table_free (mm, (unsigned long *) pud); }static inline pmd_t *pmd_alloc_one (struct mm_struct *mm,unsigned long vmaddr)
arch/s390/include/asm/tlb.h +2 -0
@@ -122,6 +122,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, {if (mm_p4d_folded (tlb->mm))return ; +pagetable_p4d_dtor (virt_to_ptdesc (p4d));__tlb_adjust_range (tlb, address,PAGE_SIZE ); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ;
@@ -140,6 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, {if (mm_pud_folded (tlb->mm))return ; +pagetable_pud_dtor (virt_to_ptdesc (pud)); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ; tlb->cleared_p4ds =1 ;
Signature · 1 line
2.20.1
Re: [PATCH v5 06/17] s390: pgtable: add statistics for PUD and P4D level page table
Quoted 80 lines · Qi Zheng
Like PMD and PTE level page table, also add statistics for PUD and P4D page table.
Signed-off-by Qi Zheng <[email protected]>Suggested-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Kevin Brodsky <[email protected]>--- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++++-------- arch/s390/include/asm/tlb.h | 2 ++ 2 files changed, 23 insertions(+), 8 deletions(-)arch/s390/include/asm/pgalloc.h +21
-8@@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) {unsigned long *table =crst_table_alloc (mm); -if (table) -crst_table_init (table, _REGION2_ENTRY_EMPTY); +if (!table) +return NULL ; +crst_table_init (table, _REGION2_ENTRY_EMPTY); +pagetable_p4d_ctor (virt_to_ptdesc (table)); +return (p4d_t *) table; }static inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) { -if (!mm_p4d_folded (mm)) -crst_table_free (mm, (unsigned long *) p4d); +if (mm_p4d_folded (mm)) +return ; + +pagetable_p4d_dtor (virt_to_ptdesc (p4d)); +crst_table_free (mm, (unsigned long *) p4d); }static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long address) {unsigned long *table =crst_table_alloc (mm); -if (table) -crst_table_init (table, _REGION3_ENTRY_EMPTY); + +if (!table) +return NULL ; +crst_table_init (table, _REGION3_ENTRY_EMPTY); +pagetable_pud_ctor (virt_to_ptdesc (table)); +return (pud_t *) table; }static inline void pud_free (struct mm_struct *mm,pud_t *pud) { -if (!mm_pud_folded (mm)) -crst_table_free (mm, (unsigned long *) pud); +if (mm_pud_folded (mm)) +return ; + +pagetable_pud_dtor (virt_to_ptdesc (pud)); +crst_table_free (mm, (unsigned long *) pud); }static inline pmd_t *pmd_alloc_one (struct mm_struct *mm,unsigned long vmaddr)arch/s390/include/asm/tlb.h +1
-0@@ -122,6 +122,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, {if (mm_p4d_folded (tlb->mm))return ; +pagetable_p4d_dtor (virt_to_ptdesc (p4d));__tlb_adjust_range (tlb, address,PAGE_SIZE ); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ;
arch/s390/include/asm/tlb.h static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, · line 147
Quoted 7 lines
arch/s390/include/asm/tlb.h +1
-0@@ -140,6 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, {if (mm_pud_folded (tlb->mm))return ; +pagetable_pud_dtor (virt_to_ptdesc (pud)); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ; tlb->cleared_p4ds =1 ;
Thanks!
PATCH v5 7/17 mm: pgtable: introduce pagetable_dtor()
In reply to Qi ZhengThe pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make ptlock_free() handle the case where ptdesc->ptl is NULL and remove VM_BUG_ON_PAGE() from pmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introduce pagetable_dtor() to do this.
Later, pagetable_dtor() will be moved to tlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU.
--- Documentation/mm/split_page_table_lock.rst | 4 +- arch/arm/include/asm/tlb.h | 4 +- arch/arm64/include/asm/tlb.h | 8 ++-- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/mmu_context.c | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/pgtable-frag.c | 4 +- arch/riscv/include/asm/pgalloc.h | 8 ++-- arch/riscv/mm/init.c | 4 +- arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/srmmu.c | 2 +- arch/um/include/asm/pgalloc.h | 6 +-- arch/x86/mm/pgtable.c | 12 ++--- include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-)
Documentation/mm/split_page_table_lock.rst +2 -2
@@ -62,7 +62,7 @@ Support of split page table lock by an architecture =================================================== There's no need in special enabling of PTE split page table lock: everything -required is done by pagetable_pte_ctor() and pagetable_pte_dtor(), which +required is done by pagetable_pte_ctor() and pagetable_dtor(), which must be called on PTE table allocation / freeing. Make sure the architecture doesn't use slab allocator for page table
@@ -73,7 +73,7 @@ PMD split lock only makes sense if you have more than two page table levels. PMD split lock enabling requires pagetable_pmd_ctor() call on PMD table -allocation and pagetable_pmd_dtor() on freeing. +allocation and pagetable_dtor() on freeing. Allocation usually happens in pmd_alloc_one(), freeing in pmd_free() and pmd_free_tlb(), but make sure you cover all PMD table allocation / freeing
arch/arm/include/asm/tlb.h +2 -2
@@ -41,7 +41,7 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) {struct ptdesc *ptdesc =page_ptdesc (pte); -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);#ifndef CONFIG_ARM_LPAE /*
@@ -61,7 +61,7 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr)#ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =virt_to_ptdesc (pmdp); -pagetable_pmd_dtor (ptdesc); +pagetable_dtor (ptdesc);tlb_remove_ptdesc (tlb, ptdesc);#endif }
arch/arm64/include/asm/tlb.h +4 -4
@@ -82,7 +82,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, {struct ptdesc *ptdesc =page_ptdesc (pte); -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);tlb_remove_ptdesc (tlb, ptdesc); }
@@ -92,7 +92,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, {struct ptdesc *ptdesc =virt_to_ptdesc (pmdp); -pagetable_pmd_dtor (ptdesc); +pagetable_dtor (ptdesc);tlb_remove_ptdesc (tlb, ptdesc); }#endif
@@ -106,7 +106,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,if (!pgtable_l4_enabled ())return ; -pagetable_pud_dtor (ptdesc); +pagetable_dtor (ptdesc); tlb_remove_ptdesc(tlb, ptdesc); }#endif
@@ -120,7 +120,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp,if (!pgtable_l5_enabled ())return ; -pagetable_p4d_dtor (ptdesc); +pagetable_dtor (ptdesc); tlb_remove_ptdesc(tlb, ptdesc); }#endif
arch/csky/include/asm/pgalloc.h +1 -1
@@ -63,7 +63,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)#define __pte_free_tlb (tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc(tlb, page_ptdesc(pte)); \ } while (0)
arch/hexagon/include/asm/pgalloc.h +1 -1
@@ -89,7 +89,7 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd,#define __pte_free_tlb (tlb, pte, addr) \ do { \ - pagetable_pte_dtor((page_ptdesc(pte))); \ + pagetable_dtor((page_ptdesc(pte))); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0)
arch/loongarch/include/asm/pgalloc.h +1 -1
@@ -57,7 +57,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm)#define __pte_free_tlb (tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0)
arch/m68k/include/asm/mcf_pgalloc.h +2 -2
@@ -37,7 +37,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtable, {struct ptdesc *ptdesc =virt_to_ptdesc (pgtable); -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); }
@@ -61,7 +61,7 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pgtable) {struct ptdesc *ptdesc =virt_to_ptdesc (pgtable); -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); }
arch/m68k/include/asm/sun3_pgalloc.h +1 -1
@@ -19,7 +19,7 @@ extern const char bad_pmd_string[];#define __pte_free_tlb (tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0)
arch/m68k/mm/motorola.c +1 -1
@@ -201,7 +201,7 @@ int free_pointer_table(void *table, int type)list_del (dp);mmu_page_dtor ((void *)page);if (type ==TABLE_PTE ) -pagetable_pte_dtor (virt_to_ptdesc ((void *)page)); +pagetable_dtor (virt_to_ptdesc ((void *)page));free_page (page);return 1 ; }else if (ptable_list[type].next != dp) {
arch/mips/include/asm/pgalloc.h +1 -1
@@ -56,7 +56,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)#define __pte_free_tlb (tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0)
arch/nios2/include/asm/pgalloc.h +1 -1
@@ -30,7 +30,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm);#define __pte_free_tlb (tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0)
arch/openrisc/include/asm/pgalloc.h +1 -1
@@ -68,7 +68,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm);#define __pte_free_tlb (tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0)
arch/powerpc/mm/book3s64/mmu_context.c +1 -1
@@ -253,7 +253,7 @@ static void pmd_frag_destroy(void *pmd_frag) count = ((unsigned long )pmd_frag & ~PAGE_MASK ) >>PMD_FRAG_SIZE_SHIFT ;/* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test (PMD_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { -pagetable_pmd_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); } }
arch/powerpc/mm/book3s64/pgtable.c +1 -1
@@ -477,7 +477,7 @@ void pmd_fragment_free(unsigned long *pmd)BUG_ON (atomic_read (&ptdesc->pt_frag_refcount) <=0 );if (atomic_dec_and_test (&ptdesc->pt_frag_refcount)) { -pagetable_pmd_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); } }
arch/powerpc/mm/pgtable-frag.c +2 -2
@@ -25,7 +25,7 @@ void pte_frag_destroy(void *pte_frag) count = ((unsigned long )pte_frag & ~PAGE_MASK ) >>PTE_FRAG_SIZE_SHIFT ;/* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test (PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); } }
@@ -111,7 +111,7 @@ static void pte_free_now(struct rcu_head *head)struct ptdesc *ptdesc; ptdesc = container_of(head, struct ptdesc, pt_rcu_head); -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); }
arch/riscv/include/asm/pgalloc.h +4 -4
@@ -100,7 +100,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,if (pgtable_l4_enabled) {struct ptdesc *ptdesc =virt_to_ptdesc (pud); -pagetable_pud_dtor (ptdesc); +pagetable_dtor (ptdesc);riscv_tlb_remove_ptdesc (tlb, ptdesc); } }
@@ -111,7 +111,7 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,if (pgtable_l5_enabled) {struct ptdesc *ptdesc =virt_to_ptdesc (p4d); -pagetable_p4d_dtor (ptdesc); +pagetable_dtor (ptdesc);riscv_tlb_remove_ptdesc (tlb,virt_to_ptdesc (p4d)); } }
@@ -144,7 +144,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, {struct ptdesc *ptdesc =virt_to_ptdesc (pmd); -pagetable_pmd_dtor (ptdesc); +pagetable_dtor (ptdesc);riscv_tlb_remove_ptdesc (tlb, ptdesc); }
@@ -155,7 +155,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, {struct ptdesc *ptdesc =page_ptdesc (pte); -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);riscv_tlb_remove_ptdesc (tlb, ptdesc); }#endif /* CONFIG_MMU */
arch/riscv/mm/init.c +2 -2
@@ -1558,7 +1558,7 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)return ; } -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);if (PageReserved (page))free_reserved_page (page);else
@@ -1580,7 +1580,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, bool is_vmemm }if (!is_vmemmap) -pagetable_pmd_dtor (ptdesc); +pagetable_dtor (ptdesc);if (PageReserved (page))free_reserved_page (page);else
arch/s390/include/asm/pgalloc.h +3 -3
@@ -66,7 +66,7 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)if (mm_p4d_folded (mm))return ; -pagetable_p4d_dtor (virt_to_ptdesc (p4d)); +pagetable_dtor (virt_to_ptdesc (p4d));crst_table_free (mm, (unsigned long *) p4d); }
@@ -87,7 +87,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)if (mm_pud_folded (mm))return ; -pagetable_pud_dtor (virt_to_ptdesc (pud)); +pagetable_dtor (virt_to_ptdesc (pud));crst_table_free (mm, (unsigned long *) pud); }
@@ -109,7 +109,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) {if (mm_pmd_folded (mm))return ; -pagetable_pmd_dtor (virt_to_ptdesc (pmd)); +pagetable_dtor (virt_to_ptdesc (pmd));crst_table_free (mm, (unsigned long *) pmd); }
arch/s390/include/asm/tlb.h +3 -3
@@ -102,7 +102,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, {if (mm_pmd_folded (tlb->mm))return ; -pagetable_pmd_dtor (virt_to_ptdesc (pmd)); +pagetable_dtor (virt_to_ptdesc (pmd));__tlb_adjust_range (tlb, address,PAGE_SIZE ); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ;
@@ -122,7 +122,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, {if (mm_p4d_folded (tlb->mm))return ; -pagetable_p4d_dtor (virt_to_ptdesc (p4d)); +pagetable_dtor (virt_to_ptdesc (p4d));__tlb_adjust_range (tlb, address,PAGE_SIZE ); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ;
@@ -141,7 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, {if (mm_pud_folded (tlb->mm))return ; -pagetable_pud_dtor (virt_to_ptdesc (pud)); +pagetable_dtor (virt_to_ptdesc (pud)); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ; tlb->cleared_p4ds =1 ;
arch/s390/mm/pgalloc.c +1 -1
@@ -182,7 +182,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm)static void pagetable_pte_dtor_free (struct ptdesc *ptdesc) { -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); }
arch/sh/include/asm/pgalloc.h +1 -1
@@ -34,7 +34,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,#define __pte_free_tlb (tlb, pte, addr) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0)
arch/sparc/mm/init_64.c +1 -1
@@ -2915,7 +2915,7 @@ static void __pte_free(pgtable_t pte) {struct ptdesc *ptdesc =virt_to_ptdesc (pte); -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); }
arch/sparc/mm/srmmu.c +1 -1
@@ -372,7 +372,7 @@ void pte_free(struct mm_struct *mm, pgtable_t ptep) page =pfn_to_page (__nocache_pa ((unsigned long )ptep) >>PAGE_SHIFT );spin_lock (&mm->page_table_lock);if (page_ref_dec_return (page) ==1 ) -pagetable_pte_dtor (page_ptdesc (page)); +pagetable_dtor (page_ptdesc (page));spin_unlock (&mm->page_table_lock);srmmu_free_nocache (ptep,SRMMU_PTE_TABLE_SIZE );
arch/um/include/asm/pgalloc.h +3 -3
@@ -27,7 +27,7 @@ extern pgd_t *pgd_alloc(struct mm_struct *);#define __pte_free_tlb (tlb, pte, address) \ do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ + pagetable_dtor(page_ptdesc(pte)); \ tlb_remove_page_ptdesc((tlb), (page_ptdesc(pte))); \ } while (0)
@@ -35,7 +35,7 @@ do { \#define __pmd_free_tlb (tlb, pmd, address) \ do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + pagetable_dtor(virt_to_ptdesc(pmd)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ } while (0)
@@ -43,7 +43,7 @@ do { \#define __pud_free_tlb (tlb, pud, address) \ do { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + pagetable_dtor(virt_to_ptdesc(pud)); \ tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ } while (0)
arch/x86/mm/pgtable.c +6 -6
@@ -60,7 +60,7 @@ early_param("userpte", setup_userpte);void ___pte_free_tlb (struct mmu_gather *tlb,struct page *pte) { -pagetable_pte_dtor (page_ptdesc (pte)); +pagetable_dtor (page_ptdesc (pte));paravirt_release_pte (page_to_pfn (pte));paravirt_tlb_remove_table (tlb, pte); }
@@ -77,7 +77,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)#ifdef CONFIG_X86_PAE tlb->need_flush_all =1 ;#endif -pagetable_pmd_dtor (ptdesc); +pagetable_dtor (ptdesc);paravirt_tlb_remove_table (tlb,ptdesc_page (ptdesc)); }
@@ -86,7 +86,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) {struct ptdesc *ptdesc =virt_to_ptdesc (pud); -pagetable_pud_dtor (ptdesc); +pagetable_dtor (ptdesc);paravirt_release_pud (__pa (pud) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (pud)); }
@@ -96,7 +96,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) {struct ptdesc *ptdesc =virt_to_ptdesc (p4d); -pagetable_p4d_dtor (ptdesc); +pagetable_dtor (ptdesc);paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (p4d)); }
@@ -233,7 +233,7 @@ static void free_pmds(struct mm_struct *mm, pmd_t *pmds[], int count)if (pmds[i]) { ptdesc =virt_to_ptdesc (pmds[i]); -pagetable_pmd_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc);mm_dec_nr_pmds (mm); }
@@ -867,7 +867,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr)free_page ((unsigned long )pmd_sv); -pagetable_pmd_dtor (virt_to_ptdesc (pmd)); +pagetable_dtor (virt_to_ptdesc (pmd));free_page ((unsigned long )pmd);return 1 ;
include/asm-generic/pgalloc.h +4 -4
@@ -109,7 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) {struct ptdesc *ptdesc =page_ptdesc (pte_page); -pagetable_pte_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); }
@@ -153,7 +153,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)struct ptdesc *ptdesc =virt_to_ptdesc (pmd);BUG_ON ((unsigned long )pmd & (PAGE_SIZE -1 )); -pagetable_pmd_dtor (ptdesc); +pagetable_dtor (ptdesc); pagetable_free(ptdesc); }#endif
@@ -202,7 +202,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud)struct ptdesc *ptdesc =virt_to_ptdesc (pud);BUG_ON ((unsigned long )pud & (PAGE_SIZE -1 )); -pagetable_pud_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); }
@@ -248,7 +248,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d)struct ptdesc *ptdesc =virt_to_ptdesc (p4d);BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 )); -pagetable_p4d_dtor (ptdesc); +pagetable_dtor (ptdesc);pagetable_free (ptdesc); }
include/linux/mm.h +9 -43
94 lines not on this page · raw message
mm/memory.c +2 -1
@@ -7051,7 +7051,8 @@ bool ptlock_alloc(struct ptdesc *ptdesc)void ptlock_free (struct ptdesc *ptdesc) { -kmem_cache_free (page_ptl_cachep, ptdesc->ptl); +if (ptdesc->ptl) +kmem_cache_free (page_ptl_cachep, ptdesc->ptl); }#endif
Signature · 1 line
2.20.1
Re: [PATCH v5 07/17] mm: pgtable: introduce pagetable_dtor()
Quoted 15 lines · Qi Zheng
The pagetable_p*_dtor() are exactly the same except for the handling of ptlock. If we make
ptlock_free()handle the case whereptdesc->ptlis NULL and removeVM_BUG_ON_PAGE()frompmd_ptlock_free(), we can unify pagetable_p*_dtor() into one function. Let's introducepagetable_dtor()to do this.Later,
pagetable_dtor()will be moved totlb_remove_ptdesc(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU.Signed-off-by Qi Zheng <[email protected]>Originally-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Kevin Brodsky <[email protected]>---
...
Quoted 3 lines
arch/s390/include/asm/pgalloc.h | 6 +-- arch/s390/include/asm/tlb.h | 6 +-- arch/s390/mm/pgalloc.c | 2 +-
...
Quoted 4 lines
include/asm-generic/pgalloc.h | 8 ++-- include/linux/mm.h | 52 ++++------------------ mm/memory.c | 3 +- 28 files changed, 62 insertions(+), 95 deletions(-)
...
For s390:
Thanks!
PATCH v5 8/17 arm: pgtable: move pagetable_dtor() to __tlb_remove_table()
In reply to Qi ZhengMove pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU.
Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages.
--- arch/arm/include/asm/tlb.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
arch/arm/include/asm/tlb.h +6 -3
@@ -26,12 +26,14 @@#else /* !CONFIG_MMU */ -#include <linux/swap.h> #include <asm/tlbflush.h> static inline void __tlb_remove_table (void *_table) { -free_page_and_swap_cache ((struct page *)_table); +struct ptdesc *ptdesc = (struct ptdesc *)_table; + +pagetable_dtor (ptdesc); +pagetable_free (ptdesc); }#include <asm-generic/tlb.h>
@@ -41,7 +43,9 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) {struct ptdesc *ptdesc =page_ptdesc (pte); +#ifndef CONFIG_MMU_GATHER_TABLE_FREE pagetable_dtor (ptdesc); +#endif #ifndef CONFIG_ARM_LPAE /*
@@ -61,7 +65,6 @@ __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr)#ifdef CONFIG_ARM_LPAE struct ptdesc *ptdesc =virt_to_ptdesc (pmdp); -pagetable_dtor (ptdesc);tlb_remove_ptdesc (tlb, ptdesc);#endif }
Signature · 1 line
2.20.1
PATCH v5 9/17 arm64: pgtable: move pagetable_dtor() to __tlb_remove_table()
In reply to Qi ZhengMove pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU.
Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages.
--- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
arch/arm64/include/asm/tlb.h +4 -6
@@ -9,11 +9,13 @@#define __ASM_TLB_H#include <linux/pagemap.h> -#include <linux/swap.h> static inline void __tlb_remove_table (void *_table) { -free_page_and_swap_cache ((struct page *)_table); +struct ptdesc *ptdesc = (struct ptdesc *)_table; + +pagetable_dtor (ptdesc); +pagetable_free (ptdesc); }#define tlb_flush tlb_flush
@@ -82,7 +84,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, {struct ptdesc *ptdesc =page_ptdesc (pte); -pagetable_dtor (ptdesc);tlb_remove_ptdesc (tlb, ptdesc); }
@@ -92,7 +93,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, {struct ptdesc *ptdesc =virt_to_ptdesc (pmdp); -pagetable_dtor (ptdesc);tlb_remove_ptdesc (tlb, ptdesc); }#endif
@@ -106,7 +106,6 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,if (!pgtable_l4_enabled ())return ; -pagetable_dtor (ptdesc); tlb_remove_ptdesc(tlb, ptdesc); }#endif
@@ -120,7 +119,6 @@ static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4dp,if (!pgtable_l5_enabled ())return ; -pagetable_dtor (ptdesc); tlb_remove_ptdesc(tlb, ptdesc); }#endif
Signature · 1 line
2.20.1
Re: [PATCH v5 09/17] arm64: pgtable: move pagetable_dtor() to __tlb_remove_table()
Quoted 8 lines · Qi Zheng
Move
pagetable_dtor()to__tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU.Page tables shouldn't have swap cache, so use
pagetable_free()instead offree_page_and_swap_cache()to free page table pages.
Quoted 7 lines
Signed-off-by Qi Zheng <[email protected]>Suggested-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Kevin Brodsky <[email protected]>--- arch/arm64/include/asm/tlb.h | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
Will
PATCH v5 10/17 riscv: pgtable: move pagetable_dtor() to __tlb_remove_table()
In reply to Qi ZhengMove pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU.
Page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages.
By the way, move the comment above __tlb_remove_table() to riscv_tlb_remove_ptdesc(), it will be more appropriate.
--- arch/riscv/include/asm/pgalloc.h | 38 ++++++++++++++------------------ arch/riscv/include/asm/tlb.h | 14 ++++-------- 2 files changed, 21 insertions(+), 31 deletions(-)
arch/riscv/include/asm/pgalloc.h +17 -21
@@ -15,12 +15,22 @@#define __HAVE_ARCH_PUD_FREE#include <asm-generic/pgalloc.h> +/* +* While riscv platforms with riscv_ipi_for_rfence as true require an IPI to +* perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use +* SBI to perform TLB shootdown. To keep software pagetable walkers safe in this +* case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the +* comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h +* for more details. +*/ static inline void riscv_tlb_remove_ptdesc (struct mmu_gather *tlb,void *pt) { -if (riscv_use_sbi_for_rfence ()) +if (riscv_use_sbi_for_rfence ()) {tlb_remove_ptdesc (tlb, pt); -else + }else { +pagetable_dtor (pt);tlb_remove_page_ptdesc (tlb, pt); + } } static inline void pmd_populate_kernel(struct mm_struct *mm,
@@ -97,23 +107,15 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)static inline void __pud_free_tlb (struct mmu_gather *tlb,pud_t *pud,unsigned long addr) { -if (pgtable_l4_enabled) { -struct ptdesc *ptdesc =virt_to_ptdesc (pud); - -pagetable_dtor (ptdesc); -riscv_tlb_remove_ptdesc (tlb, ptdesc); - } +if (pgtable_l4_enabled) +riscv_tlb_remove_ptdesc (tlb,virt_to_ptdesc (pud)); }static inline void __p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr) { -if (pgtable_l5_enabled) { -struct ptdesc *ptdesc =virt_to_ptdesc (p4d); - -pagetable_dtor (ptdesc); +if (pgtable_l5_enabled)riscv_tlb_remove_ptdesc (tlb,virt_to_ptdesc (p4d)); - } }#endif /* __PAGETABLE_PMD_FOLDED */
@@ -142,10 +144,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)static inline void __pmd_free_tlb (struct mmu_gather *tlb,pmd_t *pmd,unsigned long addr) { -struct ptdesc *ptdesc =virt_to_ptdesc (pmd); - -pagetable_dtor (ptdesc); -riscv_tlb_remove_ptdesc (tlb, ptdesc); +riscv_tlb_remove_ptdesc (tlb,virt_to_ptdesc (pmd)); }#endif /* __PAGETABLE_PMD_FOLDED */
@@ -153,10 +152,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,static inline void __pte_free_tlb (struct mmu_gather *tlb,pgtable_t pte,unsigned long addr) { -struct ptdesc *ptdesc =page_ptdesc (pte); - -pagetable_dtor (ptdesc); -riscv_tlb_remove_ptdesc (tlb, ptdesc); +riscv_tlb_remove_ptdesc (tlb,page_ptdesc (pte)); }#endif /* CONFIG_MMU */
arch/riscv/include/asm/tlb.h +4 -10
@@ -11,19 +11,13 @@ struct mmu_gather;static void tlb_flush (struct mmu_gather *tlb);#ifdef CONFIG_MMU -#include <linux/swap.h> -/* -* While riscv platforms with riscv_ipi_for_rfence as true require an IPI to -* perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use -* SBI to perform TLB shootdown. To keep software pagetable walkers safe in this -* case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the -* comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h -* for more details. -*/ static inline void __tlb_remove_table (void *table) { -free_page_and_swap_cache (table); +struct ptdesc *ptdesc = (struct ptdesc *)table; + +pagetable_dtor (ptdesc); +pagetable_free (ptdesc); }#endif /* CONFIG_MMU */
Signature · 1 line
2.20.1
PATCH v5 11/17 x86: pgtable: convert __tlb_remove_table() to use struct ptdesc
In reply to Qi ZhengConvert __tlb_remove_table() to use struct ptdesc, which will help to move pagetable_dtor() to __tlb_remove_table().
And page tables shouldn't have swap cache, so use pagetable_free() instead of free_page_and_swap_cache() to free page table pages.
--- arch/x86/include/asm/tlb.h | 16 +++++++++------- arch/x86/kernel/paravirt.c | 4 +++- arch/x86/mm/pgtable.c | 12 +++++++----- 3 files changed, 19 insertions(+), 13 deletions(-)
arch/x86/include/asm/tlb.h +9 -7
@@ -31,24 +31,26 @@ static inline void tlb_flush(struct mmu_gather *tlb)*/ static inline void __tlb_remove_table (void *table) { -free_page_and_swap_cache (table); +struct ptdesc *ptdesc = (struct ptdesc *)table; + +pagetable_free (ptdesc); }#ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu (struct rcu_head *head) { -struct page *page; +struct ptdesc *ptdesc; - page = container_of(head, struct page, rcu_head); -put_page (page); + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); +__tlb_remove_table (ptdesc); }static inline void __tlb_remove_table_one (void *table) { -struct page *page; +struct ptdesc *ptdesc; - page = table; -call_rcu (&page->rcu_head, __tlb_remove_table_one_rcu); + ptdesc = table; +call_rcu (&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); }#define __tlb_remove_table_one __tlb_remove_table_one#endif /* CONFIG_PT_RECLAIM */
arch/x86/kernel/paravirt.c +3 -1
@@ -62,7 +62,9 @@ void __init native_pv_lock_init(void)#ifndef CONFIG_PT_RECLAIM static void native_tlb_remove_table (struct mmu_gather *tlb,void *table) { -tlb_remove_page (tlb, table); +struct ptdesc *ptdesc = (struct ptdesc *)table; + +tlb_remove_page (tlb,ptdesc_page (ptdesc)); }#else static void native_tlb_remove_table (struct mmu_gather *tlb,void *table)
arch/x86/mm/pgtable.c +7 -5
@@ -23,7 +23,9 @@ EXPORT_SYMBOL(physical_mask);static inline void paravirt_tlb_remove_table (struct mmu_gather *tlb,void *table) { -tlb_remove_page (tlb, table); +struct ptdesc *ptdesc = (struct ptdesc *)table; + +tlb_remove_page (tlb,ptdesc_page (ptdesc)); }#else static inline
@@ -62,7 +64,7 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) {pagetable_dtor (page_ptdesc (pte));paravirt_release_pte (page_to_pfn (pte)); -paravirt_tlb_remove_table (tlb, pte); +paravirt_tlb_remove_table (tlb,page_ptdesc (pte)); }#if CONFIG_PGTABLE_LEVELS >2
@@ -78,7 +80,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) tlb->need_flush_all =1 ;#endif pagetable_dtor (ptdesc); -paravirt_tlb_remove_table (tlb,ptdesc_page (ptdesc)); +paravirt_tlb_remove_table (tlb, ptdesc); }#if CONFIG_PGTABLE_LEVELS >3
@@ -88,7 +90,7 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)pagetable_dtor (ptdesc);paravirt_release_pud (__pa (pud) >>PAGE_SHIFT ); -paravirt_tlb_remove_table (tlb,virt_to_page (pud)); +paravirt_tlb_remove_table (tlb, ptdesc); }#if CONFIG_PGTABLE_LEVELS >4
@@ -98,7 +100,7 @@ void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d)pagetable_dtor (ptdesc);paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT ); -paravirt_tlb_remove_table (tlb,virt_to_page (p4d)); +paravirt_tlb_remove_table (tlb, ptdesc); }#endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */
Signature · 1 line
2.20.1
Re: [PATCH v5 11/17] x86: pgtable: convert __tlb_remove_table() to use struct ptdesc
Quoted 7 lines · Qi Zheng
Convert
__tlb_remove_table()to use struct ptdesc, which will help to movepagetable_dtor()to__tlb_remove_table().And page tables shouldn't have swap cache, so use
pagetable_free()instead offree_page_and_swap_cache()to free page table pages.Signed-off-by Qi Zheng <[email protected]>
Definitely a good idea to have split patch 11 from v4.
- Kevin
PATCH v5 12/17 x86: pgtable: move pagetable_dtor() to __tlb_remove_table()
In reply to Qi ZhengMove pagetable_dtor() to __tlb_remove_table(), so that ptlock and page table pages can be freed together (regardless of whether RCU is used). This prevents the use-after-free problem where the ptlock is freed immediately but the page table pages is freed later via RCU.
--- arch/x86/include/asm/tlb.h | 1 + arch/x86/kernel/paravirt.c | 1 + arch/x86/mm/pgtable.c | 16 ++++------------ 3 files changed, 6 insertions(+), 12 deletions(-)
arch/x86/include/asm/tlb.h +1 -0
@@ -33,6 +33,7 @@ static inline void __tlb_remove_table(void *table) {struct ptdesc *ptdesc = (struct ptdesc *)table; +pagetable_dtor (ptdesc);pagetable_free (ptdesc); }
arch/x86/kernel/paravirt.c +1 -0
@@ -64,6 +64,7 @@ static void native_tlb_remove_table(struct mmu_gather *tlb, void *table) {struct ptdesc *ptdesc = (struct ptdesc *)table; +pagetable_dtor (ptdesc);tlb_remove_page (tlb,ptdesc_page (ptdesc)); }#else
arch/x86/mm/pgtable.c +4 -12
@@ -25,6 +25,7 @@ void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) {struct ptdesc *ptdesc = (struct ptdesc *)table; +pagetable_dtor (ptdesc);tlb_remove_page (tlb,ptdesc_page (ptdesc)); }#else
@@ -62,7 +63,6 @@ early_param("userpte", setup_userpte);void ___pte_free_tlb (struct mmu_gather *tlb,struct page *pte) { -pagetable_dtor (page_ptdesc (pte));paravirt_release_pte (page_to_pfn (pte));paravirt_tlb_remove_table (tlb,page_ptdesc (pte)); }
@@ -70,7 +70,6 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte)#if CONFIG_PGTABLE_LEVELS >2 void ___pmd_free_tlb (struct mmu_gather *tlb,pmd_t *pmd) { -struct ptdesc *ptdesc =virt_to_ptdesc (pmd);paravirt_release_pmd (__pa (pmd) >>PAGE_SHIFT );/* * NOTE! For PAE, any changes to the top page-directory-pointer-table
@@ -79,28 +78,21 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)#ifdef CONFIG_X86_PAE tlb->need_flush_all =1 ;#endif -pagetable_dtor (ptdesc); -paravirt_tlb_remove_table (tlb, ptdesc); +paravirt_tlb_remove_table (tlb,virt_to_ptdesc (pmd)); }#if CONFIG_PGTABLE_LEVELS >3 void ___pud_free_tlb (struct mmu_gather *tlb,pud_t *pud) { -struct ptdesc *ptdesc =virt_to_ptdesc (pud); - -pagetable_dtor (ptdesc);paravirt_release_pud (__pa (pud) >>PAGE_SHIFT ); -paravirt_tlb_remove_table (tlb, ptdesc); +paravirt_tlb_remove_table (tlb,virt_to_ptdesc (pud)); }#if CONFIG_PGTABLE_LEVELS >4 void ___p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d) { -struct ptdesc *ptdesc =virt_to_ptdesc (p4d); - -pagetable_dtor (ptdesc);paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT ); -paravirt_tlb_remove_table (tlb, ptdesc); +paravirt_tlb_remove_table (tlb,virt_to_ptdesc (p4d)); }#endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */
Signature · 1 line
2.20.1
PATCH v5 13/17 s390: pgtable: consolidate PxD and PTE TLB free paths
In reply to Qi ZhengCall pagetable_dtor() for PMD|PUD|P4D tables just before ptdesc is freed - same as it is done for PTE tables. That allows consolidating TLB free paths for all table types.
--- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-)
arch/s390/include/asm/tlb.h +0 -3
@@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, {if (mm_pmd_folded (tlb->mm))return ; -pagetable_dtor (virt_to_ptdesc (pmd));__tlb_adjust_range (tlb, address,PAGE_SIZE ); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ;
@@ -122,7 +121,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, {if (mm_p4d_folded (tlb->mm))return ; -pagetable_dtor (virt_to_ptdesc (p4d));__tlb_adjust_range (tlb, address,PAGE_SIZE ); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ;
@@ -141,7 +139,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, {if (mm_pud_folded (tlb->mm))return ; -pagetable_dtor (virt_to_ptdesc (pud)); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ; tlb->cleared_p4ds =1 ;
arch/s390/mm/pgalloc.c +4 -10
@@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm)return table; } -static void pagetable_pte_dtor_free (struct ptdesc *ptdesc) +static void pagetable_dtor_free (struct ptdesc *ptdesc) {pagetable_dtor (ptdesc);pagetable_free (ptdesc);
@@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) {struct ptdesc *ptdesc =virt_to_ptdesc (table); -pagetable_pte_dtor_free (ptdesc); +pagetable_dtor_free (ptdesc); }void __tlb_remove_table (void *table) {struct ptdesc *ptdesc =virt_to_ptdesc (table); -struct page *page =ptdesc_page (ptdesc); -if (compound_order (page) ==CRST_ALLOC_ORDER ) { -/* pmd, pud, or p4d */ -pagetable_free (ptdesc); -return ; - } -pagetable_pte_dtor_free (ptdesc); +pagetable_dtor_free (ptdesc); }#ifdef CONFIG_TRANSPARENT_HUGEPAGE
@@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); -pagetable_pte_dtor_free (ptdesc); +pagetable_dtor_free (ptdesc); }void pte_free_defer (struct mm_struct *mm,pgtable_t pgtable)
Signature · 1 line
2.20.1
Re: [PATCH v5 13/17] s390: pgtable: consolidate PxD and PTE TLB free paths
Quoted 78 lines · Qi Zheng
Call
pagetable_dtor()for PMD|PUD|P4D tables just before ptdesc is freed - same as it is done for PTE tables. That allows consolidating TLB free paths for all table types.Signed-off-by Qi Zheng <[email protected]>Suggested-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Kevin Brodsky <[email protected]>--- arch/s390/include/asm/tlb.h | 3 --- arch/s390/mm/pgalloc.c | 14 ++++---------- 2 files changed, 4 insertions(+), 13 deletions(-)arch/s390/include/asm/tlb.h +0
-3@@ -102,7 +102,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, {if (mm_pmd_folded (tlb->mm))return ; -pagetable_dtor (virt_to_ptdesc (pmd));__tlb_adjust_range (tlb, address,PAGE_SIZE ); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ;@@ -122,7 +121,6 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, {if (mm_p4d_folded (tlb->mm))return ; -pagetable_dtor (virt_to_ptdesc (p4d));__tlb_adjust_range (tlb, address,PAGE_SIZE ); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ;@@ -141,7 +139,6 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, {if (mm_pud_folded (tlb->mm))return ; -pagetable_dtor (virt_to_ptdesc (pud)); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ; tlb->cleared_p4ds =1 ;arch/s390/mm/pgalloc.c +3
-9@@ -180,7 +180,7 @@ unsigned long *page_table_alloc(struct mm_struct *mm)return table; } -static void pagetable_pte_dtor_free (struct ptdesc *ptdesc) +static void pagetable_dtor_free (struct ptdesc *ptdesc) {pagetable_dtor (ptdesc);pagetable_free (ptdesc);@@ -190,20 +190,14 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) {struct ptdesc *ptdesc =virt_to_ptdesc (table); -pagetable_pte_dtor_free (ptdesc); +pagetable_dtor_free (ptdesc); }void __tlb_remove_table (void *table) {struct ptdesc *ptdesc =virt_to_ptdesc (table); -struct page *page =ptdesc_page (ptdesc); -if (compound_order (page) ==CRST_ALLOC_ORDER ) { -/* pmd, pud, or p4d */ -pagetable_free (ptdesc); -return ; - } -pagetable_pte_dtor_free (ptdesc); +pagetable_dtor_free (ptdesc); }#ifdef CONFIG_TRANSPARENT_HUGEPAGE
arch/s390/mm/pgalloc.c static void pte_free_now(struct rcu_head *head) · line 211
Quoted 8 lines
arch/s390/mm/pgalloc.c +1
-1@@ -211,7 +205,7 @@ static void pte_free_now(struct rcu_head *head) { struct ptdesc *ptdesc = container_of(head, struct ptdesc, pt_rcu_head); -pagetable_pte_dtor_free (ptdesc); +pagetable_dtor_free (ptdesc); }void pte_free_defer (struct mm_struct *mm,pgtable_t pgtable)
Thanks!
PATCH v5 14/17 mm: pgtable: introduce generic __tlb_remove_table()
In reply to Qi ZhengSeveral architectures (arm, arm64, riscv and x86) define exactly the same __tlb_remove_table(), just introduce generic __tlb_remove_table() to eliminate these duplications.
The s390 __tlb_remove_table() is nearly the same, so also make s390 __tlb_remove_table() version generic.
--- arch/arm/include/asm/tlb.h | 9 --------- arch/arm64/include/asm/tlb.h | 7 ------- arch/powerpc/include/asm/tlb.h | 1 + arch/riscv/include/asm/tlb.h | 12 ------------ arch/s390/include/asm/tlb.h | 9 ++++----- arch/s390/mm/pgalloc.c | 7 ------- arch/sparc/include/asm/tlb_64.h | 1 + arch/x86/include/asm/tlb.h | 17 ----------------- include/asm-generic/tlb.h | 15 +++++++++++++-- 9 files changed, 19 insertions(+), 59 deletions(-)
arch/arm/include/asm/tlb.h +0 -9
@@ -27,15 +27,6 @@#else /* !CONFIG_MMU */ #include <asm/tlbflush.h> - -static inline void __tlb_remove_table (void *_table) -{ -struct ptdesc *ptdesc = (struct ptdesc *)_table; - -pagetable_dtor (ptdesc); -pagetable_free (ptdesc); -} -#include <asm-generic/tlb.h> static inline void
arch/arm64/include/asm/tlb.h +0 -7
@@ -10,13 +10,6 @@#include <linux/pagemap.h> -static inline void __tlb_remove_table (void *_table) -{ -struct ptdesc *ptdesc = (struct ptdesc *)_table; - -pagetable_dtor (ptdesc); -pagetable_free (ptdesc); -}#define tlb_flush tlb_flushstatic void tlb_flush (struct mmu_gather *tlb);
arch/powerpc/include/asm/tlb.h +1 -0
@@ -37,6 +37,7 @@ extern void tlb_flush(struct mmu_gather *tlb);*/ #define tlb_needs_table_invalidate () radix_enabled() +#define __HAVE_ARCH_TLB_REMOVE_TABLE/* Get the generic bits... */ #include <asm-generic/tlb.h>
arch/riscv/include/asm/tlb.h +0 -12
@@ -10,18 +10,6 @@ struct mmu_gather;static void tlb_flush (struct mmu_gather *tlb); -#ifdef CONFIG_MMU - -static inline void __tlb_remove_table (void *table) -{ -struct ptdesc *ptdesc = (struct ptdesc *)table; - -pagetable_dtor (ptdesc); -pagetable_free (ptdesc); -} - -#endif /* CONFIG_MMU */ -#define tlb_flush tlb_flush#include <asm-generic/tlb.h>
arch/s390/include/asm/tlb.h +4 -5
@@ -22,7 +22,6 @@* Pages used for the page tables is a different story. FIXME: more */ -void __tlb_remove_table (void *_table);static inline void tlb_flush (struct mmu_gather *tlb);static inline bool __tlb_remove_page_size (struct mmu_gather *tlb,struct page *page,bool delay_rmap,int page_size);
@@ -87,7 +86,7 @@ static inline void pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, tlb->cleared_pmds =1 ;if (mm_alloc_pgste (tlb->mm))gmap_unlink (tlb->mm, (unsigned long *)pte, address); -tlb_remove_ptdesc (tlb, pte); +tlb_remove_ptdesc (tlb,virt_to_ptdesc (pte)); }/*
@@ -106,7 +105,7 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ; tlb->cleared_puds =1 ; -tlb_remove_ptdesc (tlb, pmd); +tlb_remove_ptdesc (tlb,virt_to_ptdesc (pmd)); }/*
@@ -124,7 +123,7 @@ static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,__tlb_adjust_range (tlb, address,PAGE_SIZE ); tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ; -tlb_remove_ptdesc (tlb, p4d); +tlb_remove_ptdesc (tlb,virt_to_ptdesc (p4d)); }/*
@@ -142,7 +141,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, tlb->mm->context.flush_mm =1 ; tlb->freed_tables =1 ; tlb->cleared_p4ds =1 ; -tlb_remove_ptdesc (tlb, pud); +tlb_remove_ptdesc (tlb,virt_to_ptdesc (pud)); }
arch/s390/mm/pgalloc.c +0 -7
@@ -193,13 +193,6 @@ void page_table_free(struct mm_struct *mm, unsigned long *table)pagetable_dtor_free (ptdesc); } -void __tlb_remove_table (void *table) -{ -struct ptdesc *ptdesc =virt_to_ptdesc (table); - -pagetable_dtor_free (ptdesc); -} -#ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pte_free_now (struct rcu_head *head) {
arch/sparc/include/asm/tlb_64.h +1 -0
@@ -33,6 +33,7 @@ void flush_tlb_pending(void);#define tlb_needs_table_invalidate () (false)#endif +#define __HAVE_ARCH_TLB_REMOVE_TABLE#include <asm-generic/tlb.h> #endif /* _SPARC64_TLB_H */
arch/x86/include/asm/tlb.h +0 -17
@@ -20,23 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb)flush_tlb_mm_range (tlb->mm, start, end, stride_shift, tlb->freed_tables); } -/* -* While x86 architecture in general requires an IPI to perform TLB -* shootdown, enablement code for several hypervisors overrides -* .flush_tlb_others hook in pv_mmu_ops and implements it by issuing -* a hypercall. To keep software pagetable walkers safe in this case we -* switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment -* below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h -* for more details. -*/ -static inline void __tlb_remove_table (void *table) -{ -struct ptdesc *ptdesc = (struct ptdesc *)table; - -pagetable_dtor (ptdesc); -pagetable_free (ptdesc); -} -#ifdef CONFIG_PT_RECLAIM static inline void __tlb_remove_table_one_rcu (struct rcu_head *head) {
include/asm-generic/tlb.h +13 -2
@@ -153,8 +153,9 @@* * Useful if your architecture has non-page page directories. * -* When used, an architecture is expected to provide __tlb_remove_table() -* which does the actual freeing of these pages. +* When used, an architecture is expected to provide __tlb_remove_table() or +* use the generic __tlb_remove_table(), which does the actual freeing of these +* pages. * * MMU_GATHER_RCU_TABLE_FREE *
@@ -207,6 +208,16 @@ struct mmu_table_batch {#define MAX_TABLE_BATCH \ ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) +#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE +static inline void __tlb_remove_table (void *table) +{ +struct ptdesc *ptdesc = (struct ptdesc *)table; + +pagetable_dtor (ptdesc); +pagetable_free (ptdesc); +} +#endif +extern void tlb_remove_table (struct mmu_gather *tlb,void *table);#else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */
Signature · 1 line
2.20.1
Re: [PATCH v5 14/17] mm: pgtable: introduce generic __tlb_remove_table()
Quoted 11 lines · Qi Zheng
Several architectures (arm, arm64, riscv and x86) define exactly the same
__tlb_remove_table(), just introduce generic__tlb_remove_table()to eliminate these duplications.The s390
__tlb_remove_table()is nearly the same, so also make s390__tlb_remove_table()version generic.Signed-off-by Qi Zheng <[email protected]>Reviewed-by Kevin Brodsky <[email protected]>
PATCH v5 15/17 mm: pgtable: completely move pagetable_dtor() to generic tlb_remove_table()
In reply to Qi ZhengFor the generic tlb_remove_table(), it is implemented in the following two forms:
1) CONFIG_MMU_GATHER_TABLE_FREE is enabled
tlb_remove_table
--> generic __tlb_remove_table()
2) CONFIG_MMU_GATHER_TABLE_FREE is disabled
tlb_remove_table
--> tlb_remove_page
For case 1), the pagetable_dtor() has already been moved to generic __tlb_remove_table().
For case 2), now only arm will call tlb_remove_table()/tlb_remove_ptdesc() when CONFIG_MMU_GATHER_TABLE_FREE is disabled. Let's move pagetable_dtor() completely to generic tlb_remove_table(), so that the architectures can follow more easily.
--- arch/arm/include/asm/tlb.h | 4 ---- include/asm-generic/tlb.h | 10 ++++++++-- 2 files changed, 8 insertions(+), 6 deletions(-)
arch/arm/include/asm/tlb.h +0 -4
@@ -34,10 +34,6 @@ __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) {struct ptdesc *ptdesc =page_ptdesc (pte); -#ifndef CONFIG_MMU_GATHER_TABLE_FREE -pagetable_dtor (ptdesc); -#endif -#ifndef CONFIG_ARM_LPAE /* * With the classic ARM MMU, a pte page has two corresponding pmd
include/asm-generic/tlb.h +8 -2
@@ -220,14 +220,20 @@ static inline void __tlb_remove_table(void *table)extern void tlb_remove_table (struct mmu_gather *tlb,void *table); -#else /* !CONFIG_MMU_GATHER_HAVE_TABLE_FREE */ +#else /* !CONFIG_MMU_GATHER_TABLE_FREE */ +static inline void tlb_remove_page (struct mmu_gather *tlb,struct page *page);/* * Without MMU_GATHER_TABLE_FREE the architecture is assumed to have page based * page directories and we can use the normal page batching to free them. */ -#define tlb_remove_table (tlb, page) tlb_remove_page((tlb), (page)) +static inline void tlb_remove_table (struct mmu_gather *tlb,void *table) +{ +struct page *page = (struct page *)table; +pagetable_dtor (page_ptdesc (page)); +tlb_remove_page (tlb, page); +}#endif /* CONFIG_MMU_GATHER_TABLE_FREE */ #ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
Signature · 1 line
2.20.1
PATCH v5 16/17 mm: pgtable: move __tlb_remove_table_one() in x86 to generic file
In reply to Qi ZhengThe __tlb_remove_table_one() in x86 does not contain architecture-specific content, so move it to the generic file.
--- arch/x86/include/asm/tlb.h | 19 ------------------- mm/mmu_gather.c | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 21 deletions(-)
arch/x86/include/asm/tlb.h +0 -19
@@ -20,25 +20,6 @@ static inline void tlb_flush(struct mmu_gather *tlb)flush_tlb_mm_range (tlb->mm, start, end, stride_shift, tlb->freed_tables); } -#ifdef CONFIG_PT_RECLAIM -static inline void __tlb_remove_table_one_rcu (struct rcu_head *head) -{ -struct ptdesc *ptdesc; - - ptdesc = container_of(head, struct ptdesc, pt_rcu_head); -__tlb_remove_table (ptdesc); -} - -static inline void __tlb_remove_table_one (void *table) -{ -struct ptdesc *ptdesc; - - ptdesc = table; -call_rcu (&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); -} -#define __tlb_remove_table_one __tlb_remove_table_one -#endif /* CONFIG_PT_RECLAIM */ -static inline void invlpg (unsigned long addr) { asmvolatile ("invlpg (%0)" ::"r" (addr) :"memory" );
mm/mmu_gather.c +18 -2
@@ -311,13 +311,29 @@ static inline void tlb_table_invalidate(struct mmu_gather *tlb) } } -#ifndef __tlb_remove_table_one +#ifdef CONFIG_PT_RECLAIM +static inline void __tlb_remove_table_one_rcu (struct rcu_head *head) +{ +struct ptdesc *ptdesc; + + ptdesc = container_of(head, struct ptdesc, pt_rcu_head); +__tlb_remove_table (ptdesc); +} + +static inline void __tlb_remove_table_one (void *table) +{ +struct ptdesc *ptdesc; + + ptdesc = table; +call_rcu (&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu); +} +#else static inline void __tlb_remove_table_one (void *table) {tlb_remove_table_sync_one ();__tlb_remove_table (table); } -#endif +#endif /* CONFIG_PT_RECLAIM */ static void tlb_remove_table_one (void *table) {
Signature · 1 line
2.20.1
PATCH v5 17/17 mm: pgtable: introduce generic pagetable_dtor_free()
In reply to Qi ZhengThe pte_free(), pmd_free(), __pud_free() and __p4d_free() in asm-generic/pgalloc.h and the generic __tlb_remove_table() are basically the same, so let's introduce pagetable_dtor_free() to deduplicate them.
In addition, the pagetable_dtor_free() in s390 does the same thing, so let's s390 also calls generic pagetable_dtor_free().
--- arch/s390/mm/pgalloc.c | 6 ------ include/asm-generic/pgalloc.h | 12 ++++-------- include/asm-generic/tlb.h | 3 +-- include/linux/mm.h | 6 ++++++ 4 files changed, 11 insertions(+), 16 deletions(-)
arch/s390/mm/pgalloc.c +0 -6
@@ -180,12 +180,6 @@ unsigned long *page_table_alloc(struct mm_struct *mm)return table; } -static void pagetable_dtor_free (struct ptdesc *ptdesc) -{ -pagetable_dtor (ptdesc); -pagetable_free (ptdesc); -} -void page_table_free (struct mm_struct *mm,unsigned long *table) {struct ptdesc *ptdesc =virt_to_ptdesc (table);
include/asm-generic/pgalloc.h +4 -8
@@ -109,8 +109,7 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte_page) {struct ptdesc *ptdesc =page_ptdesc (pte_page); -pagetable_dtor (ptdesc); -pagetable_free (ptdesc); +pagetable_dtor_free (ptdesc); }
@@ -153,8 +152,7 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)struct ptdesc *ptdesc =virt_to_ptdesc (pmd);BUG_ON ((unsigned long )pmd & (PAGE_SIZE -1 )); -pagetable_dtor (ptdesc); - pagetable_free(ptdesc); + pagetable_dtor_free(ptdesc); }#endif
@@ -202,8 +200,7 @@ static inline void __pud_free(struct mm_struct *mm, pud_t *pud)struct ptdesc *ptdesc =virt_to_ptdesc (pud);BUG_ON ((unsigned long )pud & (PAGE_SIZE -1 )); -pagetable_dtor (ptdesc); -pagetable_free (ptdesc); +pagetable_dtor_free (ptdesc); }#ifndef __HAVE_ARCH_PUD_FREE
@@ -248,8 +245,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d)struct ptdesc *ptdesc =virt_to_ptdesc (p4d);BUG_ON ((unsigned long )p4d & (PAGE_SIZE -1 )); -pagetable_dtor (ptdesc); -pagetable_free (ptdesc); +pagetable_dtor_free (ptdesc); }#ifndef __HAVE_ARCH_P4D_FREE
include/asm-generic/tlb.h +1 -2
@@ -213,8 +213,7 @@ static inline void __tlb_remove_table(void *table) {struct ptdesc *ptdesc = (struct ptdesc *)table; -pagetable_dtor (ptdesc); -pagetable_free (ptdesc); +pagetable_dtor_free (ptdesc); }#endif
include/linux/mm.h +6 -0
@@ -3001,6 +3001,12 @@ static inline void pagetable_dtor(struct ptdesc *ptdesc)lruvec_stat_sub_folio (folio,NR_PAGETABLE ); } +static inline void pagetable_dtor_free (struct ptdesc *ptdesc) +{ +pagetable_dtor (ptdesc); +pagetable_free (ptdesc); +} +static inline bool pagetable_pte_ctor (struct ptdesc *ptdesc) {struct folio *folio =ptdesc_folio (ptdesc);
Signature · 1 line
2.20.1