x86/mm: Pre-allocate p4d/pud pages for vmalloc area
Subsystems (3)
PATCH 1/1 x86/mm: Pre-allocate p4d/pud pages for vmalloc area
From: Joerg Roedel <[email protected]>
Pre-allocate the page-table pages for the vmalloc area at the level which needs synchronization on x86. This is P4D for 5-level and PUD for 4-level paging.
Doing this at boot makes sure all page-tables in the system have these pages already and do not need to be synchronized at runtime. The runtime synchronizatin takes the pgd_lock and iterates over all page-tables in the system, so it can take quite long and is better avoided.
--- arch/x86/mm/init_64.c | 55 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
arch/x86/mm/init_64.c +55 -0
@@ -1238,6 +1238,59 @@ static void __init register_page_bootmem_info(void)#endif } +/* +* Pre-allocates page-table pages for the vmalloc area in the kernel page-table. +* Only the level which needs to be synchronized between all page-tables is +* allocated because the synchronization can be expensive. +*/ +static void __init preallocate_vmalloc_pages (void ) +{ +unsigned long addr; +const char *lvl; +int count =0 ; + +for (addr =VMALLOC_START ; addr <=VMALLOC_END ; addr =ALIGN (addr +1 ,PGDIR_SIZE )) { +pgd_t *pgd =pgd_offset_k (addr); +p4d_t *p4d; +pud_t *pud; + + p4d =p4d_offset (pgd, addr); +if (p4d_none (*p4d)) { +/* Can only happen with 5-level paging */ + p4d =p4d_alloc (&init_mm, pgd, addr); +if (!p4d) { + lvl ="p4d" ; + goto failed; + } + count +=1 ; + } + +if (pgtable_l5_enabled ()) +continue ; + + pud =pud_offset (p4d, addr); +if (pud_none (*pud)) { +/* Ends up here only with 4-level paging */ + pud =pud_alloc (&init_mm, p4d, addr); +if (!pud) { + lvl ="pud" ; + goto failed; + } + count +=1 ; + } + } + +return ; + +failed: + +/* +* A failure here is not fatal - If the pages can be allocated later it +* will be synchronized to other page-tables. +*/ +pr_err ("Failed to pre-allocate %s pages for vmalloc area\n" , lvl); +} +void __init mem_init (void ) {pci_iommu_alloc ();
@@ -1261,6 +1314,8 @@ void __init mem_init(void)if (get_gate_vma (&init_mm))kclist_add (&kcore_vsyscall, (void *)VSYSCALL_ADDR ,PAGE_SIZE ,KCORE_USER ); +preallocate_vmalloc_pages (); +mem_init_print_info (NULL ); }
Signature · 1 line
2.27.0
Peter Zijlstra
maintainerQuoted 6 lines · Joerg Roedel
From: Joerg Roedel <[email protected]>
Pre-allocate the page-table pages for the vmalloc area at the level which needs synchronization on x86. This is P4D for 5-level and PUD for 4-level paging.
Quoted 8 lines
Doing this at boot makes sure all page-tables in the system have these pages already and do not need to be synchronized at runtime. The runtime synchronizatin takes the
pgd_lockand iterates over all page-tables in the system, so it can take quite long and is better avoided.Signed-off-by Joerg Roedel <[email protected]>---
Can't we now remove arch_sync_kernel_mappings() from this same file?
Quoted 1 line · Peter Zijlstra
Can't we now remove
arch_sync_kernel_mappings()from this same file?
Only if we panic on allocation failure.
Joerg
Peter Zijlstra
maintainerQuoted 4 lines · Joerg Roedel
Quoted 1 line · Peter Zijlstra
Can't we now remove
arch_sync_kernel_mappings()from this same file?Only if we panic on allocation failure.
I think we do that in plenty places already, so sure ;-)
Peter Zijlstra
maintainerQuoted 7 lines · Peter Zijlstra
Quoted 4 lines · Joerg Roedel
Quoted 1 line · Peter Zijlstra
Can't we now remove
arch_sync_kernel_mappings()from this same file?Only if we panic on allocation failure.
I think we do that in plenty places already, so sure ;-)
That is, this is boot time only, right? clone() would return -ENOMEM, as it's part of the normal page-table copy.
Quoted 2 lines · Peter Zijlstra
That is, this is boot time only, right?
clone()would return-ENOMEM, as it's part of the normal page-table copy.
Yes, the pre-allocation happens shortly after the buddy allocator took over from bootmem. I don't quite get what clone() has to do with it.
Joerg