mm: Get rid of vmalloc_sync_(un)mappings()
Subsystems (11)
v3 cover mm: Get rid of vmalloc_sync_(un)mappings()
Hi,
here is the updated version of this series with these changes:
- Removed sync_current_stack_to_mm() too.
- Added Acked-by's from Andy Lutomirski
The previous versions can be found here:
v1: https://lore.kernel.org/lkml/[email protected]/ v2: https://lore.kernel.org/lkml/[email protected]/
The cover-letter of v1 has more details on the motivation for this patch-set.
Please review.
Regards,
Joerg
Joerg Roedel (7):
mm: Add functions to track page directory modifications mm/vmalloc: Track which page-table levels were modified mm/ioremap: Track which page-table levels were modified x86/mm/64: Implement arch_sync_kernel_mappings() x86/mm/32: Implement arch_sync_kernel_mappings()
mm: Remove vmalloc_sync_(un)mappings()
x86/mm: Remove vmalloc faulting
arch/x86/include/asm/pgtable-2level_types.h | 2 + arch/x86/include/asm/pgtable-3level_types.h | 2 + arch/x86/include/asm/pgtable_64_types.h | 2 + arch/x86/include/asm/switch_to.h | 23 --- arch/x86/kernel/setup_percpu.c | 6 +- arch/x86/mm/fault.c | 176 +------------------- arch/x86/mm/init_64.c | 5 + arch/x86/mm/pti.c | 8 +- arch/x86/mm/tlb.c | 37 ---- drivers/acpi/apei/ghes.c | 6 - include/asm-generic/5level-fixup.h | 5 +- include/asm-generic/pgtable.h | 23 +++ include/linux/mm.h | 46 +++++ include/linux/vmalloc.h | 18 +- kernel/notifier.c | 1 - kernel/trace/trace.c | 12 -- lib/ioremap.c | 46 +++-- mm/nommu.c | 12 -- mm/vmalloc.c | 109 +++++++----- 19 files changed, 204 insertions(+), 335 deletions(-)
Signature · 1 line
2.17.1
PATCH v3 1/7 mm: Add functions to track page directory modifications
From: Joerg Roedel <[email protected]>
Add page-table allocation functions which will keep track of changed directory entries. They are needed for new PGD, P4D, PUD, and PMD entries and will be used in vmalloc and ioremap code to decide whether any changes in the kernel mappings need to be synchronized between page-tables in the system.
--- include/asm-generic/5level-fixup.h | 5 ++-- include/asm-generic/pgtable.h | 23 +++++++++++++++ include/linux/mm.h | 46 ++++++++++++++++++++++++++++++ 3 files changed, 72 insertions(+), 2 deletions(-)
include/asm-generic/5level-fixup.h +3 -2
@@ -17,8 +17,9 @@ ((unlikely (pgd_none (*(p4d))) &&__pud_alloc (mm, p4d, address)) ? \ NULL : pud_offset(p4d, address)) -#define p4d_alloc (mm, pgd, address) (pgd) -#define p4d_offset (pgd, start) (pgd) +#define p4d_alloc (mm, pgd, address) (pgd) +#define p4d_alloc_track (mm, pgd, address, mask) (pgd) +#define p4d_offset (pgd, start) (pgd)#ifndef __ASSEMBLY__static inline int p4d_none (p4d_t p4d)
include/asm-generic/pgtable.h +23 -0
@@ -1209,6 +1209,29 @@ static inline bool arch_has_pfn_modify_check(void)# define PAGE_KERNEL_EXEC PAGE_KERNEL#endif +/* +* Page Table Modification bits for pgtbl_mod_mask. +* +* These are used by the p?d_alloc_track*() set of functions an in the generic +* vmalloc/ioremap code to track at which page-table levels entries have been +* modified. Based on that the code can better decide when vmalloc and ioremap +* mapping changes need to be synchronized to other page-tables in the system. +*/ +#define __PGTBL_PGD_MODIFIED 0 +#define __PGTBL_P4D_MODIFIED 1 +#define __PGTBL_PUD_MODIFIED 2 +#define __PGTBL_PMD_MODIFIED 3 +#define __PGTBL_PTE_MODIFIED 4 + +#define PGTBL_PGD_MODIFIED BIT(__PGTBL_PGD_MODIFIED) +#define PGTBL_P4D_MODIFIED BIT(__PGTBL_P4D_MODIFIED) +#define PGTBL_PUD_MODIFIED BIT(__PGTBL_PUD_MODIFIED) +#define PGTBL_PMD_MODIFIED BIT(__PGTBL_PMD_MODIFIED) +#define PGTBL_PTE_MODIFIED BIT(__PGTBL_PTE_MODIFIED) + +/* Page-Table Modification Mask */ +typedef unsigned int pgtbl_mod_mask ; +#endif /* !__ASSEMBLY__ */ #ifndef io_remap_pfn_range
include/linux/mm.h +46 -0
65 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v3 1/7] mm: Add functions to track page directory modifications
Hi Joerg,
Quoted 27 lines · Joerg Roedel
include/linux/mm.h +17
-0@@ -2078,13 +2078,54 @@ static inline pud_t *pud_alloc(struct mm_struct *mm, p4d_t *p4d,return (unlikely (p4d_none (*p4d)) &&__pud_alloc (mm, p4d, address)) ?NULL :pud_offset (p4d, address); } + +static inline p4d_t *p4d_alloc_track (struct mm_struct *mm,pgd_t *pgd, +unsigned long address, +pgtbl_mod_mask *mod_mask) + +{ +if (unlikely (pgd_none (*pgd))) { +if (__p4d_alloc (mm, pgd, address)) +return NULL ; + *mod_mask |=PGTBL_PGD_MODIFIED ; + } + +return p4d_offset (pgd, address); +} +#endif /* !__ARCH_HAS_5LEVEL_HACK */ +static inline pud_t *pud_alloc_track(struct mm_struct *mm, p4d_t *p4d, + unsigned long address,
include/linux/mm.h static inline pud_t *pud_alloc(struct mm_struct *mm, p4d_t *p4d, · line 2109
Quoted 10 lines
include/linux/mm.h +10
-0+ pgtbl_mod_mask *mod_mask) +{ +if (unlikely (p4d_none (*p4d))) { +if (__pud_alloc (mm, p4d, address)) +return NULL ; + *mod_mask |=PGTBL_P4D_MODIFIED ; + } + +return pud_offset (p4d, address); +}
This patch causes a kernel panic on arm64 (and possibly powerpc, I haven't tried). arm64 still uses the 5level-fixup.h and pud_alloc() checks for the empty p4d with pgd_none() instead of p4d_none().
The patch below fixes it:
-----------------------8<----------------------------------- From 8fb5f4c1e0983bb501c8ec71f9bb9dd344b80e87 Mon Sep 17 00:00:00 2001
From: Catalin Marinas <[email protected]>
Date: Fri, 5 Jun 2020 10:56:03 +0100
Subject: [PATCH] mm: Use pgd_none() in pud_alloc_track() with
__ARCH_HAS_5LEVEL_HACK
Commit d8626138009b ("mm: add functions to track page directory modifications") introduced the pud_alloc_track() function checking for an empty p4d using p4d_none(). However, when __ARCH_HAS_5LEVEL_HACK is defined, the pud_alloc() counterpart checks for an empty p4d using pgd_none(). Since p4d_none() is always 0 in this case, no pud would be allocated and the kernel panics during boot on arm64 (at least).
Until all architectures are moved away from the 5level-fixup.h, define a pud_alloc_track() that matches the __ARCH_HAS_5LEVEL_HACK pud_alloc().
--- include/linux/mm.h | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
include/linux/mm.h +17 -2
@@ -2106,8 +2106,6 @@ static inline p4d_t *p4d_alloc_track(struct mm_struct *mm, pgd_t *pgd,return p4d_offset (pgd, address); } -#endif /* !__ARCH_HAS_5LEVEL_HACK */ -static inline pud_t *pud_alloc_track (struct mm_struct *mm,p4d_t *p4d,unsigned long address,pgtbl_mod_mask *mod_mask)
@@ -2121,6 +2119,23 @@ static inline pud_t *pud_alloc_track(struct mm_struct *mm, p4d_t *p4d,return pud_offset (p4d, address); } +#else /* __ARCH_HAS_5LEVEL_HACK */ + +static inline pud_t *pud_alloc_track (struct mm_struct *mm,p4d_t *p4d, +unsigned long address, +pgtbl_mod_mask *mod_mask) +{ +if (unlikely (pgd_none (*p4d))) { +if (__pud_alloc (mm, p4d, address)) +return NULL ; + *mod_mask |=PGTBL_P4D_MODIFIED ; + } + +return pud_offset (p4d, address); +} + +#endif /* !__ARCH_HAS_5LEVEL_HACK */ +static inline pmd_t *pmd_alloc (struct mm_struct *mm,pud_t *pud,unsigned long address) { return (unlikely(pud_none(*pud)) && __pmd_alloc(mm, pud, address))?
Re: [PATCH v3 1/7] mm: Add functions to track page directory modifications
Quoted 3 lines · Catalin Marinas
This patch causes a kernel panic on arm64 (and possibly powerpc, I haven't tried). arm64 still uses the
5level-fixup.handpud_alloc()checks for the empty p4d withpgd_none()instead ofp4d_none().
Ah, should have checked the list first
(https://lore.kernel.org/linux-mm/[email protected]/).
Please ignore my patch then.
Signature · 1 line
Catalin
Andrew Morton
maintainerRe: [PATCH v3 1/7] mm: Add functions to track page directory modifications
Quoted 8 lines · On Fri, 5 Jun 2020 12:46:55 +0100 Catalin Marinas <[email protected]>…
Quoted 3 lines · Catalin Marinas
This patch causes a kernel panic on arm64 (and possibly powerpc, I haven't tried). arm64 still uses the
5level-fixup.handpud_alloc()checks for the empty p4d withpgd_none()instead ofp4d_none().Ah, should have checked the list first
(https://lore.kernel.org/linux-mm/[email protected]/).
Sorry about that.
Can you confirm that the merge of Mike's 5level_hack zappage has fixed the arm64 wontboot?
Re: [PATCH v3 1/7] mm: Add functions to track page directory modifications
Quoted 11 lines · Andrew Morton
Quoted 7 lines · On Fri, 5 Jun 2020 12:46:55 +0100 Catalin Marinas <[email protected]>…
Quoted 3 lines · Catalin Marinas
This patch causes a kernel panic on arm64 (and possibly powerpc, I haven't tried). arm64 still uses the
5level-fixup.handpud_alloc()checks for the empty p4d withpgd_none()instead ofp4d_none().Ah, should have checked the list first
(https://lore.kernel.org/linux-mm/[email protected]/).Can you confirm that the merge of Mike's 5level_hack zappage has fixed the arm64 wontboot?
Yes, it has. Mainline is booting again on arm64.
Thanks.
Signature · 1 line
Catalin
PATCH v3 2/7 mm/vmalloc: Track which page-table levels were modified
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Track at which levels in the page-table entries were modified by vmap/vunmap. After the page-table has been modified, use that information do decide whether the new arch_sync_kernel_mappings() needs to be called.
--- include/linux/vmalloc.h | 16 ++++++++ mm/vmalloc.c | 88 ++++++++++++++++++++++++++++++----------- 2 files changed, 80 insertions(+), 24 deletions(-)
include/linux/vmalloc.h +16 -0
@@ -144,6 +144,22 @@ extern int remap_vmalloc_range(struct vm_area_struct *vma, void *addr,void vmalloc_sync_mappings (void );void vmalloc_sync_unmappings (void ); +/* +* Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED values +* and let generic vmalloc and ioremap code know when arch_sync_kernel_mappings() +* needs to be called. +*/ +#ifndef ARCH_PAGE_TABLE_SYNC_MASK +#define ARCH_PAGE_TABLE_SYNC_MASK 0 +#endif + +/* +* There is no default implementation for arch_sync_kernel_mappings(). It is +* relied upon the compiler to optimize calls out if ARCH_PAGE_TABLE_SYNC_MASK +* is 0. +*/ +void arch_sync_kernel_mappings (unsigned long start,unsigned long end); +/* * Lowlevel-APIs (not for driver use!) */
mm/vmalloc.c +64 -24
222 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v3 2/7] mm/vmalloc: Track which page-table levels were modified
Quoted 4 lines · On Fri, 15 May 2020 16:00:18 +0200 Joerg Roedel <[email protected]> wrote:
Track at which levels in the page-table entries were modified by vmap/vunmap. After the page-table has been modified, use that information do decide whether the new
arch_sync_kernel_mappings()needs to be called.
Lots of collisions here with Christoph's "decruft the vmalloc API" series
(http://lkml.kernel.org/r/[email protected]).
I attempted to fix things up.
unmap_kernel_range_noflush() needed to be redone.
map_kernel_range_noflush() might need the arch_sync_kernel_mappings() call?
From: Joerg Roedel <[email protected]>
Subject: mm/vmalloc: Track which page-table levels were modified
Track at which levels in the page-table entries were modified by vmap/vunmap. After the page-table has been modified, use that information do decide whether the new arch_sync_kernel_mappings() needs to be called.
---
include/linux/vmalloc.h | 16 ++++++ mm/vmalloc.c | 91 +++++++++++++++++++++++++++----------- 2 files changed, 81 insertions(+), 26 deletions(-)
include/linux/vmalloc.h renamed from include/linux/vmalloc.h~mm-vmalloc-track-which-page-table-levels-were-modified +16 -0
@@ -134,6 +134,22 @@ void vmalloc_sync_mappings(void);void vmalloc_sync_unmappings (void );/* +* Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED values +* and let generic vmalloc and ioremap code know when arch_sync_kernel_mappings() +* needs to be called. +*/ +#ifndef ARCH_PAGE_TABLE_SYNC_MASK +#define ARCH_PAGE_TABLE_SYNC_MASK 0 +#endif + +/* +* There is no default implementation for arch_sync_kernel_mappings(). It is +* relied upon the compiler to optimize calls out if ARCH_PAGE_TABLE_SYNC_MASK +* is 0. +*/ +void arch_sync_kernel_mappings (unsigned long start,unsigned long end); + +/* * Lowlevel-APIs (not for driver use!) */
mm/vmalloc.c renamed from mm/vmalloc.c~mm-vmalloc-track-which-page-table-levels-were-modified +65 -26
229 lines not on this page · raw message
_
Re: [PATCH v3 2/7] mm/vmalloc: Track which page-table levels were modified
Hi Andrew,
Quoted 9 lines · Andrew Morton
On Fri, 15 May 2020 16:00:18 +0200 Joerg Roedel <[email protected]> wrote:
Lots of collisions here with Christoph's "decruft the vmalloc API" series
(http://lkml.kernel.org/r/[email protected]).I attempted to fix things up.
unmap_kernel_range_noflush()needed to be redone.
map_kernel_range_noflush()might need thearch_sync_kernel_mappings()call?
Yes, map_kernel_range_noflush() needs the arch_sync_kernel_mappings() call as well.
Regards,
Joerg
Andrew Morton
maintainerRe: [PATCH v3 2/7] mm/vmalloc: Track which page-table levels were modified
Quoted 7 lines · On Sat, 16 May 2020 14:56:41 +0200 Joerg Roedel <[email protected]> wrote:
Hi Andrew,
Quoted 4 lines · Andrew Morton
On Fri, 15 May 2020 16:00:18 +0200 Joerg Roedel <[email protected]> wrote:
Lots of collisions here with Christoph's "decruft the vmalloc API" series
(http://lkml.kernel.org/r/[email protected]).
Quoted 9 lines
Quoted 5 lines
I attempted to fix things up.
unmap_kernel_range_noflush()needed to be redone.
map_kernel_range_noflush()might need thearch_sync_kernel_mappings()call?Yes,
map_kernel_range_noflush()needs thearch_sync_kernel_mappings()call as well.
This?
mm/vmalloc.c renamed from mm/vmalloc.c~mm-vmalloc-track-which-page-table-levels-were-modified-fix +3 -0
@@ -309,6 +309,9 @@ int map_kernel_range_noflush(unsigned loreturn err; }while (pgd++, addr = next, addr != end); +if (mask &ARCH_PAGE_TABLE_SYNC_MASK ) +arch_sync_kernel_mappings (start, end); +return 0 ; }
It would be nice to get all this (ie, linux-next) retested before we send it upstream, please.
Re: [PATCH v3 2/7] mm/vmalloc: Track which page-table levels were modified
mm/vmalloc.c int map_kernel_range_noflush(unsigned lo · line 316
Quoted 12 lines · Andrew Morton
On Sat, 16 May 2020 14:56:41 +0200 Joerg Roedel <[email protected]> wrote:
mm/vmalloc.c renamed from mm/vmalloc.c~mm-vmalloc-track-which-page-table-levels-were-modified-fix +3
-0@@ -309,6 +309,9 @@ int map_kernel_range_noflush(unsigned loreturn err; }while (pgd++, addr = next, addr != end); +if (mask &ARCH_PAGE_TABLE_SYNC_MASK ) +arch_sync_kernel_mappings (start, end); +return 0 ; }
Yes, this is the right call.
Quoted 2 lines
It would be nice to get all this (ie, linux-next) retested before we send it upstream, please.
Will do and report back.
Thanks,
Joerg
Re: [PATCH v3 2/7] mm/vmalloc: Track which page-table levels were modified
In reply to Andrew MortonQuoted 2 lines · Andrew Morton
It would be nice to get all this (ie, linux-next) retested before we send it upstream, please.
Tested these patches as in next-20200522, with three 2, 3, and 4-level paging. On 4-level (aka 64-bit) I also re-ran the Stevens trace-cmd reproducer, no issues found and the trace-cmd problem is still fixed.
Thanks,
Joerg
PATCH v3 3/7 mm/ioremap: Track which page-table levels were modified
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Track at which levels in the page-table entries were modified by ioremap_page_range(). After the page-table has been modified, use that information do decide whether the new arch_sync_kernel_mappings() needs to be called. The iounmap path re-uses vunmap(), which has already been taken care of.
--- lib/ioremap.c | 46 +++++++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 15 deletions(-)
lib/ioremap.c +31 -15
@@ -61,13 +61,14 @@ static inline int ioremap_pmd_enabled(void) { return 0; }#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ static int ioremap_pte_range (pmd_t *pmd,unsigned long addr, -unsigned long end,phys_addr_t phys_addr,pgprot_t prot) +unsigned long end,phys_addr_t phys_addr,pgprot_t prot, +pgtbl_mod_mask *mask) {pte_t *pte;u64 pfn; pfn = phys_addr >>PAGE_SHIFT ; - pte =pte_alloc_kernel (pmd, addr); + pte =pte_alloc_kernel_track (pmd, addr, mask);if (!pte)return -ENOMEM ; do {
@@ -75,6 +76,7 @@ static int ioremap_pte_range(pmd_t *pmd, unsigned long addr,set_pte_at (&init_mm, addr, pte,pfn_pte (pfn, prot)); pfn++; }while (pte++, addr +=PAGE_SIZE , addr != end); + *mask |=PGTBL_PTE_MODIFIED ;return 0 ; }
@@ -101,21 +103,24 @@ static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, }static inline int ioremap_pmd_range (pud_t *pud,unsigned long addr, -unsigned long end,phys_addr_t phys_addr,pgprot_t prot) +unsigned long end,phys_addr_t phys_addr,pgprot_t prot, +pgtbl_mod_mask *mask) {pmd_t *pmd;unsigned long next; - pmd =pmd_alloc (&init_mm, pud, addr); + pmd =pmd_alloc_track (&init_mm, pud, addr, mask);if (!pmd)return -ENOMEM ;do { next =pmd_addr_end (addr, end); -if (ioremap_try_huge_pmd (pmd, addr, next, phys_addr, prot)) +if (ioremap_try_huge_pmd (pmd, addr, next, phys_addr, prot)) { + *mask |=PGTBL_PMD_MODIFIED ;continue ; + } -if (ioremap_pte_range (pmd, addr, next, phys_addr, prot)) +if (ioremap_pte_range (pmd, addr, next, phys_addr, prot, mask))return -ENOMEM ; }while (pmd++, phys_addr += (next - addr), addr = next, addr != end);return 0 ;
@@ -144,21 +149,24 @@ static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, }static inline int ioremap_pud_range (p4d_t *p4d,unsigned long addr, -unsigned long end,phys_addr_t phys_addr,pgprot_t prot) +unsigned long end,phys_addr_t phys_addr,pgprot_t prot, +pgtbl_mod_mask *mask) {pud_t *pud;unsigned long next; - pud =pud_alloc (&init_mm, p4d, addr); + pud =pud_alloc_track (&init_mm, p4d, addr, mask);if (!pud)return -ENOMEM ;do { next =pud_addr_end (addr, end); -if (ioremap_try_huge_pud (pud, addr, next, phys_addr, prot)) +if (ioremap_try_huge_pud (pud, addr, next, phys_addr, prot)) { + *mask |=PGTBL_PUD_MODIFIED ;continue ; + } -if (ioremap_pmd_range (pud, addr, next, phys_addr, prot)) +if (ioremap_pmd_range (pud, addr, next, phys_addr, prot, mask))return -ENOMEM ; }while (pud++, phys_addr += (next - addr), addr = next, addr != end);return 0 ;
@@ -187,21 +195,24 @@ static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, }static inline int ioremap_p4d_range (pgd_t *pgd,unsigned long addr, -unsigned long end,phys_addr_t phys_addr,pgprot_t prot) +unsigned long end,phys_addr_t phys_addr,pgprot_t prot, +pgtbl_mod_mask *mask) {p4d_t *p4d;unsigned long next; - p4d =p4d_alloc (&init_mm, pgd, addr); + p4d =p4d_alloc_track (&init_mm, pgd, addr, mask);if (!p4d)return -ENOMEM ;do { next =p4d_addr_end (addr, end); -if (ioremap_try_huge_p4d (p4d, addr, next, phys_addr, prot)) +if (ioremap_try_huge_p4d (p4d, addr, next, phys_addr, prot)) { + *mask |=PGTBL_P4D_MODIFIED ;continue ; + } -if (ioremap_pud_range (p4d, addr, next, phys_addr, prot)) +if (ioremap_pud_range (p4d, addr, next, phys_addr, prot, mask))return -ENOMEM ; }while (p4d++, phys_addr += (next - addr), addr = next, addr != end);return 0 ;
@@ -214,6 +225,7 @@ int ioremap_page_range(unsigned long addr,unsigned long start;unsigned long next;int err; +pgtbl_mod_mask mask =0 ;might_sleep ();BUG_ON (addr >= end);
@@ -222,13 +234,17 @@ int ioremap_page_range(unsigned long addr, pgd =pgd_offset_k (addr);do { next =pgd_addr_end (addr, end); - err =ioremap_p4d_range (pgd, addr, next, phys_addr, prot); + err =ioremap_p4d_range (pgd, addr, next, phys_addr, prot, + &mask);if (err)break ; }while (pgd++, phys_addr += (next - addr), addr = next, addr != end);flush_cache_vmap (start, end); +if (mask &ARCH_PAGE_TABLE_SYNC_MASK ) +arch_sync_kernel_mappings (start, end); +return err; }
Signature · 1 line
2.17.1
PATCH v3 4/7 x86/mm/64: Implement arch_sync_kernel_mappings()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Implement the function to sync changes in vmalloc and ioremap ranges to all page-tables.
--- arch/x86/include/asm/pgtable_64_types.h | 2 ++ arch/x86/mm/init_64.c | 5 +++++ 2 files changed, 7 insertions(+)
arch/x86/include/asm/pgtable_64_types.h +2 -0
@@ -159,4 +159,6 @@ extern unsigned int ptrs_per_p4d;#define PGD_KERNEL_START ((PAGE_SIZE / 2) / sizeof(pgd_t)) +#define ARCH_PAGE_TABLE_SYNC_MASK (pgtable_l5_enabled() ? PGTBL_PGD_MODIFIED : PGTBL_P4D_MODIFIED) +#endif /* _ASM_X86_PGTABLE_64_DEFS_H */
arch/x86/mm/init_64.c +5 -0
@@ -217,6 +217,11 @@ void sync_global_pgds(unsigned long start, unsigned long end)sync_global_pgds_l4 (start, end); } +void arch_sync_kernel_mappings (unsigned long start,unsigned long end) +{ +sync_global_pgds (start, end); +} +/* * NOTE: This function is marked __ref because it calls __init function * (alloc_bootmem_pages). It's safe to do it ONLY when after_bootmem == 0.
Signature · 1 line
2.17.1
PATCH v3 5/7 x86/mm/32: Implement arch_sync_kernel_mappings()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Implement the function to sync changes in vmalloc and ioremap ranges to all page-tables.
--- arch/x86/include/asm/pgtable-2level_types.h | 2 ++ arch/x86/include/asm/pgtable-3level_types.h | 2 ++ arch/x86/mm/fault.c | 25 +++++++++++++-------- 3 files changed, 20 insertions(+), 9 deletions(-)
arch/x86/include/asm/pgtable-2level_types.h +2 -0
@@ -20,6 +20,8 @@ typedef union {#define SHARED_KERNEL_PMD 0 +#define ARCH_PAGE_TABLE_SYNC_MASK PGTBL_PMD_MODIFIED +/* * traditional i386 two-level paging structure: */
arch/x86/include/asm/pgtable-3level_types.h +2 -0
@@ -27,6 +27,8 @@ typedef union {#define SHARED_KERNEL_PMD (!static_cpu_has(X86_FEATURE_PTI))#endif +#define ARCH_PAGE_TABLE_SYNC_MASK (SHARED_KERNEL_PMD ? 0 : PGTBL_PMD_MODIFIED) +/* * PGDIR_SHIFT determines what a top-level page table entry can map */
arch/x86/mm/fault.c +16 -9
@@ -190,16 +190,13 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address)return pmd_k; } -static void vmalloc_sync (void ) +void arch_sync_kernel_mappings (unsigned long start,unsigned long end) { -unsigned long address; - -if (SHARED_KERNEL_PMD ) -return ; +unsigned long addr; -for (address =VMALLOC_START &PMD_MASK ; - address >=TASK_SIZE_MAX && address <VMALLOC_END ; - address +=PMD_SIZE ) { +for (addr = start &PMD_MASK ; + addr >=TASK_SIZE_MAX && addr <VMALLOC_END ; + addr +=PMD_SIZE ) {struct page *page;spin_lock (&pgd_lock);
@@ -210,13 +207,23 @@ static void vmalloc_sync(void) pgt_lock = &pgd_page_get_mm (page)->page_table_lock;spin_lock (pgt_lock); -vmalloc_sync_one (page_address (page), address); +vmalloc_sync_one (page_address (page), addr);spin_unlock (pgt_lock); }spin_unlock (&pgd_lock); } } +static void vmalloc_sync (void ) +{ +unsigned long address; + +if (SHARED_KERNEL_PMD ) +return ; + +arch_sync_kernel_mappings (VMALLOC_START ,VMALLOC_END ); +} +void vmalloc_sync_mappings (void ) {vmalloc_sync ();
Signature · 1 line
2.17.1
PATCH v3 6/7 mm: Remove vmalloc_sync_(un)mappings()
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
These functions are not needed anymore because the vmalloc and ioremap mappings are now synchronized when they are created or teared down.
Remove all callers and function definitions.
--- arch/x86/mm/fault.c | 37 ------------------------------------- drivers/acpi/apei/ghes.c | 6 ------ include/linux/vmalloc.h | 2 -- kernel/notifier.c | 1 - kernel/trace/trace.c | 12 ------------ mm/nommu.c | 12 ------------ mm/vmalloc.c | 21 --------------------- 7 files changed, 91 deletions(-)
arch/x86/mm/fault.c +0 -37
@@ -214,26 +214,6 @@ void arch_sync_kernel_mappings(unsigned long start, unsigned long end) } } -static void vmalloc_sync (void ) -{ -unsigned long address; - -if (SHARED_KERNEL_PMD ) -return ; - -arch_sync_kernel_mappings (VMALLOC_START ,VMALLOC_END ); -} - -void vmalloc_sync_mappings (void ) -{ -vmalloc_sync (); -} - -void vmalloc_sync_unmappings (void ) -{ -vmalloc_sync (); -} -/* * 32-bit: *
@@ -336,23 +316,6 @@ static void dump_pagetable(unsigned long address)#else /* CONFIG_X86_64: */ -void vmalloc_sync_mappings (void ) -{ -/* -* 64-bit mappings might allocate new p4d/pud pages -* that need to be propagated to all tasks' PGDs. -*/ -sync_global_pgds (VMALLOC_START &PGDIR_MASK ,VMALLOC_END ); -} - -void vmalloc_sync_unmappings (void ) -{ -/* -* Unmappings never allocate or free p4d/pud pages. -* No work is required here. -*/ -} -/* * 64-bit: *
drivers/acpi/apei/ghes.c +0 -6
@@ -167,12 +167,6 @@ int ghes_estatus_pool_init(int num_ghes)if (!addr) goto err_pool_alloc; -/* -* New allocation must be visible in all pgd before it can be found by -* an NMI allocating from the pool. -*/ -vmalloc_sync_mappings (); - rc =gen_pool_add (ghes_estatus_pool, addr,PAGE_ALIGN (len),-1 );if (rc) goto err_pool_add;
include/linux/vmalloc.h +0 -2
@@ -141,8 +141,6 @@ extern int remap_vmalloc_range_partial(struct vm_area_struct *vma,extern int remap_vmalloc_range (struct vm_area_struct *vma,void *addr,unsigned long pgoff); -void vmalloc_sync_mappings (void ); -void vmalloc_sync_unmappings (void );/* * Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED values
kernel/notifier.c +0 -1
@@ -519,7 +519,6 @@ NOKPROBE_SYMBOL(notify_die);int register_die_notifier (struct notifier_block *nb) { -vmalloc_sync_mappings ();return atomic_notifier_chain_register (&die_chain, nb); }EXPORT_SYMBOL_GPL (register_die_notifier);
kernel/trace/trace.c +0 -12
@@ -8527,18 +8527,6 @@ static int allocate_trace_buffers(struct trace_array *tr, int size) allocate_snapshot = false;#endif -/* -* Because of some magic with the way alloc_percpu() works on -* x86_64, we need to synchronize the pgd of all the tables, -* otherwise the trace events that happen in x86_64 page fault -* handlers can't cope with accessing the chance that a -* alloc_percpu()'d memory might be touched in the page fault trace -* event. Oh, and we need to audit all other alloc_percpu() and vmalloc() -* calls in tracing, because something might get triggered within a -* page fault trace event! -*/ -vmalloc_sync_mappings (); -return 0 ; }
mm/nommu.c +0 -12
@@ -369,18 +369,6 @@ void vm_unmap_aliases(void) }EXPORT_SYMBOL_GPL (vm_unmap_aliases); -/* -* Implement a stub for vmalloc_sync_[un]mapping() if the architecture -* chose not to have one. -*/ -void __weak vmalloc_sync_mappings (void ) -{ -} - -void __weak vmalloc_sync_unmappings (void ) -{ -} -struct vm_struct *alloc_vm_area (size_t size,pte_t **ptes) {BUG ();
mm/vmalloc.c +0 -21
@@ -1332,12 +1332,6 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end)if (unlikely (valist ==NULL ))return false; -/* -* First make sure the mappings are removed from all page-tables -* before they are freed. -*/ -vmalloc_sync_unmappings (); -/* * TODO: to calculate a flush range without looping. * The list can be up to lazy_max_pages() elements.
@@ -3177,21 +3171,6 @@ int remap_vmalloc_range(struct vm_area_struct *vma, void *addr, }EXPORT_SYMBOL (remap_vmalloc_range); -/* -* Implement stubs for vmalloc_sync_[un]mappings () if the architecture chose -* not to have one. -* -* The purpose of this function is to make sure the vmalloc area -* mappings are identical in all page-tables in the system. -*/ -void __weak vmalloc_sync_mappings (void ) -{ -} - -void __weak vmalloc_sync_unmappings (void ) -{ -} -static int f (pte_t *pte,unsigned long addr,void *data) {pte_t ***p = data;
Signature · 1 line
2.17.1
PATCH v3 7/7 x86/mm: Remove vmalloc faulting
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Remove fault handling on vmalloc areas, as the vmalloc code now takes care of synchronizing changes to all page-tables in the system.
--- arch/x86/include/asm/switch_to.h | 23 ------ arch/x86/kernel/setup_percpu.c | 6 +- arch/x86/mm/fault.c | 134 ------------------------------- arch/x86/mm/pti.c | 8 +- arch/x86/mm/tlb.c | 37 --------- 5 files changed, 4 insertions(+), 204 deletions(-)
arch/x86/include/asm/switch_to.h +0 -23
@@ -12,27 +12,6 @@ struct task_struct *__switch_to_asm(struct task_struct *prev, __visible struct task_struct *__switch_to(struct task_struct *prev,struct task_struct *next); -/* This runs runs on the previous thread's stack. */ -static inline void prepare_switch_to (struct task_struct *next) -{ -#ifdef CONFIG_VMAP_STACK -/* -* If we switch to a stack that has a top-level paging entry -* that is not present in the current mm, the resulting #PF will -* will be promoted to a double-fault and we'll panic. Probe -* the new stack now so that vmalloc_fault can fix up the page -* tables if needed. This can only happen if we use a stack -* in vmap space. -* -* We assume that the stack is aligned so that it never spans -* more than one top-level paging entry. -* -* To minimize cache pollution, just follow the stack pointer. -*/ -READ_ONCE (*(unsigned char *)next->thread.sp); -#endif -} - asmlinkage void ret_from_fork(void);/*
@@ -67,8 +46,6 @@ struct fork_frame {#define switch_to (prev, next, last) \ do { \ - prepare_switch_to(next); \ - \ ((last) = __switch_to_asm((prev), (next))); \ } while (0)
arch/x86/kernel/setup_percpu.c +3 -3
@@ -287,9 +287,9 @@ void __init setup_per_cpu_areas(void)/* * Sync back kernel address range again. We already did this in * setup_arch(), but percpu data also needs to be available in -* the smpboot asm. We can't reliably pick up percpu mappings -* using vmalloc_fault(), because exception dispatch needs -* percpu data. +* the smpboot asm and arch_sync_kernel_mappings() doesn't sync to +* swapper_pg_dir on 32-bit. The per-cpu mappings need to be available +* there too. * * FIXME: Can the later sync in setup_cpu_entry_areas() replace * this call?
arch/x86/mm/fault.c +0 -134
152 lines not on this page · raw message
arch/x86/mm/pti.c +1 -7
@@ -448,13 +448,7 @@ static void __init pti_clone_user_shared(void)* the sp1 and sp2 slots. * * This is done for all possible CPUs during boot to ensure -* that it's propagated to all mms. If we were to add one of -* these mappings during CPU hotplug, we would need to take -* some measure to make sure that every mm that subsequently -* ran on that CPU would have the relevant PGD entry in its -* pagetables. The usual vmalloc_fault() mechanism would not -* work for page faults taken in entry_SYSCALL_64 before RSP -* is set up. +* that it's propagated to all mms. */ unsigned long va = (unsigned long )&per_cpu (cpu_tss_rw, cpu);
arch/x86/mm/tlb.c +0 -37
@@ -161,34 +161,6 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next,local_irq_restore (flags); } -static void sync_current_stack_to_mm (struct mm_struct *mm) -{ -unsigned long sp = current_stack_pointer; -pgd_t *pgd =pgd_offset (mm, sp); - -if (pgtable_l5_enabled ()) { -if (unlikely (pgd_none (*pgd))) { -pgd_t *pgd_ref =pgd_offset_k (sp); - -set_pgd (pgd, *pgd_ref); - } - }else { -/* -* "pgd" is faked. The top level entries are "p4d"s, so sync -* the p4d. This compiles to approximately the same code as -* the 5-level case. -*/ -p4d_t *p4d =p4d_offset (pgd, sp); - -if (unlikely (p4d_none (*p4d))) { -pgd_t *pgd_ref =pgd_offset_k (sp); -p4d_t *p4d_ref =p4d_offset (pgd_ref, sp); - -set_p4d (p4d, *p4d_ref); - } - } -} -static inline unsigned long mm_mangle_tif_spec_ib (struct task_struct *next) {unsigned long next_tif =task_thread_info (next)->flags;
@@ -377,15 +349,6 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,*/ cond_ibpb (tsk); -if (IS_ENABLED (CONFIG_VMAP_STACK )) { -/* -* If our current stack is in vmalloc space and isn't -* mapped in the new pgd, we'll double-fault. Forcibly -* map it. -*/ -sync_current_stack_to_mm (next); - } -/* * Stop remote flushes for the previous mm. * Skip kernel threads; we never send init_mm TLB flushing IPIs,
Signature · 1 line
2.17.1
Quoted 21 lines · Joerg Roedel
Joerg Roedel (7):
mm: Add functions to track page directory modifications mm/vmalloc: Track which page-table levels were modified mm/ioremap: Track which page-table levels were modified x86/mm/64: Implementarch_sync_kernel_mappings()x86/mm/32: Implementarch_sync_kernel_mappings()
mm: Remove vmalloc_sync_(un)mappings()
x86/mm: Remove vmalloc faultingarch/x86/include/asm/pgtable-2level_types.h | 2 + arch/x86/include/asm/pgtable-3level_types.h | 2 + arch/x86/include/asm/pgtable_64_types.h | 2 + arch/x86/include/asm/switch_to.h | 23 --- arch/x86/kernel/setup_percpu.c | 6 +- arch/x86/mm/fault.c | 176 +------------------- arch/x86/mm/init_64.c | 5 + arch/x86/mm/pti.c | 8 +- arch/x86/mm/tlb.c | 37 ---- drivers/acpi/apei/ghes.c | 6 - include/asm-generic/5level-fixup.h | 5 +- include/asm-generic/pgtable.h | 23 +++
Quoted 8 lines
include/linux/mm.h | 46 +++++ include/linux/vmalloc.h | 18 +- kernel/notifier.c | 1 - kernel/trace/trace.c | 12 -- lib/ioremap.c | 46 +++-- mm/nommu.c | 12 -- mm/vmalloc.c | 109 +++++++----- 19 files changed, 204 insertions(+), 335 deletions(-)
I'm thinking this improves the status-quo, so:
Like Andy, I think I'd like x86_64 to pre-populate, but that can easily be done on top and should not hold this back.