x86/mm/64: Remove vmalloc/ioremap pgtable synchronization
Subsystems (3)
v2 cover x86/mm/64: Remove vmalloc/ioremap pgtable synchronization
Hi,
here is a small patch-set to implement pre-allocation of the P4D/PUD pages in the vmalloc/ioremap area of the address space. This allows to remove the synchronization of these mappings for x86-64 and unexporting the sync_global_pgds() function.
Please review.
Thanks,
Joerg
Changes to v1:
- Made failure at pre-allocation stage fatal
- As a consequence removed the arch_sync_kernel_mappings() implementation for x86-64
- Unexported sync_global_pgds()
- Removed some left-over debug code from the first patch
Joerg Roedel (3):
x86/mm/64: Pre-allocate p4d/pud pages for vmalloc area x86/mm/64: Do not sync vmalloc/ioremap mappings x86/mm/64: Make sync_global_pgds() static
arch/x86/include/asm/pgtable_64.h | 2 - arch/x86/include/asm/pgtable_64_types.h | 2 - arch/x86/mm/init_64.c | 59 ++++++++++++++++++++++--- 3 files changed, 53 insertions(+), 10 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/3 x86/mm/64: Pre-allocate p4d/pud pages for vmalloc area
From: Joerg Roedel <[email protected]>
Pre-allocate the page-table pages for the vmalloc area at the level which needs synchronization on x86. This is P4D for 5-level and PUD for 4-level paging.
Doing this at boot makes sure all page-tables in the system have these pages already and do not need to be synchronized at runtime. The runtime synchronizatin takes the pgd_lock and iterates over all page-tables in the system, so it can take quite long and is better avoided.
--- arch/x86/mm/init_64.c | 52 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)
arch/x86/mm/init_64.c +52 -0
@@ -1238,6 +1238,56 @@ static void __init register_page_bootmem_info(void)#endif } +/* +* Pre-allocates page-table pages for the vmalloc area in the kernel page-table. +* Only the level which needs to be synchronized between all page-tables is +* allocated because the synchronization can be expensive. +*/ +static void __init preallocate_vmalloc_pages (void ) +{ +unsigned long addr; +const char *lvl; + +for (addr =VMALLOC_START ; addr <=VMALLOC_END ; addr =ALIGN (addr +1 ,PGDIR_SIZE )) { +pgd_t *pgd =pgd_offset_k (addr); +p4d_t *p4d; +pud_t *pud; + + p4d =p4d_offset (pgd, addr); +if (p4d_none (*p4d)) { +/* Can only happen with 5-level paging */ + p4d =p4d_alloc (&init_mm, pgd, addr); +if (!p4d) { + lvl ="p4d" ; + goto failed; + } + } + +if (pgtable_l5_enabled ()) +continue ; + + pud =pud_offset (p4d, addr); +if (pud_none (*pud)) { +/* Ends up here only with 4-level paging */ + pud =pud_alloc (&init_mm, p4d, addr); +if (!pud) { + lvl ="pud" ; + goto failed; + } + } + } + +return ; + +failed: + +/* +* The pages have to be there now or they will be missing in +* process page-tables later. +*/ +panic ("Failed to pre-allocate %s pages for vmalloc area\n" , lvl); +} +void __init mem_init (void ) {pci_iommu_alloc ();
@@ -1261,6 +1311,8 @@ void __init mem_init(void)if (get_gate_vma (&init_mm))kclist_add (&kcore_vsyscall, (void *)VSYSCALL_ADDR ,PAGE_SIZE ,KCORE_USER ); +preallocate_vmalloc_pages (); +mem_init_print_info (NULL ); }
Signature · 1 line
2.17.1
PATCH v2 2/3 x86/mm/64: Do not sync vmalloc/ioremap mappings
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
Remove the code to sync the vmalloc and ioremap ranges for x86-64. The page-table pages are all pre-allocated now so that synchronizing the top-level page happens at page-table creation.
--- arch/x86/include/asm/pgtable_64_types.h | 2 -- arch/x86/mm/init_64.c | 5 ----- 2 files changed, 7 deletions(-)
arch/x86/include/asm/pgtable_64_types.h +0 -2
@@ -159,6 +159,4 @@ extern unsigned int ptrs_per_p4d;#define PGD_KERNEL_START ((PAGE_SIZE / 2) / sizeof(pgd_t)) -#define ARCH_PAGE_TABLE_SYNC_MASK (pgtable_l5_enabled() ? PGTBL_PGD_MODIFIED : PGTBL_P4D_MODIFIED) -#endif /* _ASM_X86_PGTABLE_64_DEFS_H */
arch/x86/mm/init_64.c +0 -5
@@ -217,11 +217,6 @@ void sync_global_pgds(unsigned long start, unsigned long end)sync_global_pgds_l4 (start, end); } -void arch_sync_kernel_mappings (unsigned long start,unsigned long end) -{ -sync_global_pgds (start, end); -} -/* * NOTE: This function is marked __ref because it calls __init function * (alloc_bootmem_pages). It's safe to do it ONLY when after_bootmem == 0.
Signature · 1 line
2.17.1
PATCH v2 3/3 x86/mm/64: Make sync_global_pgds() static
In reply to Joerg RoedelFrom: Joerg Roedel <[email protected]>
The function is only called from within init_64.c by now and can be static. Also remove it from pgtable_64.h.
--- arch/x86/include/asm/pgtable_64.h | 2 -- arch/x86/mm/init_64.c | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-)
arch/x86/include/asm/pgtable_64.h +0 -2
@@ -168,8 +168,6 @@ static inline void native_pgd_clear(pgd_t *pgd)native_set_pgd (pgd,native_make_pgd (0 )); } -extern void sync_global_pgds (unsigned long start,unsigned long end); -/* * Conversion functions: convert a page and protection to a page entry, * and a page entry and page directory to the page they refer to.
arch/x86/mm/init_64.c +1 -1
@@ -209,7 +209,7 @@ static void sync_global_pgds_l4(unsigned long start, unsigned long end)* When memory was added make sure all the processes MM have * suitable PGD entries in the local PGD level page. */ -void sync_global_pgds (unsigned long start,unsigned long end) +static void sync_global_pgds (unsigned long start,unsigned long end) {if (pgtable_l5_enabled ())sync_global_pgds_l5 (start, end);
Signature · 1 line
2.17.1