4level support for x86_64
PATCH 17/17 4level support for x86_64
4level support for x86_64
This patch extends x86-64 to use a true 4level VM. This extends all user address spaces to 47bits (128TB)
fork/exec performance in lmbench is down by a few percent because they have to walk more page tables right now. There are plans to recovers this with future optimizations.
Signed-off-by Andi Kleen <[email protected]>
Documentation/x86_64/mm.txt +21 -145
@@ -1,148 +1,24 @@ -The paging design used on the x86-64 linux kernel port in 2.4.x provides: -o per process virtual address space limit of 512 Gigabytes -o top of userspace stack located at address 0x0000007fffffffff -o PAGE_OFFSET = 0xffff800000000000 -o start of the kernel = 0xffffffff800000000 -o global RAM per system 2^64-PAGE_OFFSET-sizeof(kernel) = 128 Terabytes - 2 Gigabytes -o no need of any common code change -o no need to use highmem to handle the 128 Terabytes of RAM +<previous description obsolete, deleted> + +Virtual memory map with 4 level page tables: + +0000000000000000 - 00007fffffffffff (=47bits) user space, different per mm +hole caused by [48:63] sign extension +ffff800000000000 - ffff80ffffffffff (=40bits) guard hole +ffff810000000000 - ffffc0ffffffffff (=46bits) direct mapping of phys. memory +ffffc10000000000 - ffffc1ffffffffff (=40bits) hole +ffffc20000000000 - ffffe1ffffffffff (=45bits) vmalloc/ioremap space +... unused hole ... +ffffffff80000000 - ffffffff82800000 (=40MB) kernel text mapping, from phys 0 +... unused hole ... +ffffffff88000000 - fffffffffff00000 (=1919MB) module mapping space + +vmalloc space is lazily synchronized into the different PML4 pages of +the processes using the page fault handler, with init_level4_pgt as +reference. -Description: +Current X86-64 implementations only support 40 bit of address space, +but we support upto 46bits. This expands into MBZ space in the page tables. - Userspace is able to modify and it sees only the 3rd/2nd/1st level - pagetables (pgd_offset() implicitly walks the 1st slot of the 4th - level pagetable and it returns an entry into the 3rd level pagetable). - This is where the per-process 512 Gigabytes limit cames from. - - The common code pgd is the PDPE, the pmd is the PDE, the - pte is the PTE. The PML4E remains invisible to the common - code. - - The kernel uses all the first 47 bits of the negative half - of the virtual address space to build the direct mapping using - 2 Mbytes page size. The kernel virtual addresses have bit number - 47 always set to 1 (and in turn also bits 48-63 are set to 1 too, - due the sign extension). This is where the 128 Terabytes - 2 Gigabytes global - limit of RAM cames from. - - Since the per-process limit is 512 Gigabytes (due to kernel common - code 3 level pagetable limitation), the higher virtual address mapped - into userspace is 0x7fffffffff and it makes sense to use it - as the top of the userspace stack to allow the stack to grow as - much as possible. - - Setting the PAGE_OFFSET to 2^39 (after the last userspace - virtual address) wouldn't make much difference compared to - setting PAGE_OFFSET to 0xffff800000000000 because we have an - hole into the virtual address space. The last byte mapped by the - 255th slot in the 4th level pagetable is at virtual address - 0x00007fffffffffff and the first byte mapped by the 256th slot in the - 4th level pagetable is at address 0xffff800000000000. Due to this - hole we can't trivially build a direct mapping across all the - 512 slots of the 4th level pagetable, so we simply use only the - second (negative) half of the 4th level pagetable for that purpose - (that provides us 128 Terabytes of contigous virtual addresses). - Strictly speaking we could build a direct mapping also across the hole - using some DISCONTIGMEM trick, but we don't need such a large - direct mapping right now. - -Future: - - During 2.5.x we can break the 512 Gigabytes per-process limit - possibly by removing from the common code any knowledge about the - architectural dependent physical layout of the virtual to physical - mapping. - - Once the 512 Gigabytes limit will be removed the kernel stack will - be moved (most probably to virtual address 0x00007fffffffffff). - Nothing will break in userspace due that move, as nothing breaks - in IA32 compiling the kernel with CONFIG_2G. - -Linus agreed on not breaking common code and to live with the 512 Gigabytes -per-process limitation for the 2.4.x timeframe and he has given me and Andi -some very useful hints... (thanks! :) - -Thanks also to H. Peter Anvin for his interesting and useful suggestions on -the x86-64-discuss lists! - -Other memory management related issues follows: - -PAGE_SIZE: - - If somebody is wondering why these days we still have a so small - 4k pagesize (16 or 32 kbytes would be much better for performance - of course), the PAGE_SIZE have to remain 4k for 32bit apps to - provide 100% backwards compatible IA32 API (we can't allow silent - fs corruption or as best a loss of coherency with the page cache - by allocating MAP_SHARED areas in MAP_ANONYMOUS memory with a - do_mmap_fake). I think it could be possible to have a dynamic page - size between 32bit and 64bit apps but it would need extremely - intrusive changes in the common code as first for page cache and - we sure don't want to depend on them right now even if the - hardware would support that. - -PAGETABLE SIZE: - - In turn we can't afford to have pagetables larger than 4k because - we could not be able to allocate them due physical memory - fragmentation, and failing to allocate the kernel stack is a minor - issue compared to failing the allocation of a pagetable. If we - fail the allocation of a pagetable the only thing we can do is to - sched_yield polling the freelist (deadlock prone) or to segfault - the task (not even the sighandler would be sure to run). - -KERNEL STACK: - - 1st stage: - - The kernel stack will be at first allocated with an order 2 allocation - (16k) (the utilization of the stack for a 64bit platform really - isn't exactly the double of a 32bit platform because the local - variables may not be all 64bit wide, but not much less). This will - make things even worse than they are right now on IA32 with - respect of failing fork/clone due memory fragmentation. - - 2nd stage: - - We'll benchmark if reserving one register as task_struct - pointer will improve performance of the kernel (instead of - recalculating the task_struct pointer starting from the stack - pointer each time). My guess is that recalculating will be faster - but it worth a try. - - If reserving one register for the task_struct pointer - will be faster we can as well split task_struct and kernel - stack. task_struct can be a slab allocation or a - PAGE_SIZEd allocation, and the kernel stack can then be - allocated in a order 1 allocation. Really this is risky, - since 8k on a 64bit platform is going to be less than 7k - on a 32bit platform but we could try it out. This would - reduce the fragmentation problem of an order of magnitude - making it equal to the current IA32. - - We must also consider the x86-64 seems to provide in hardware a - per-irq stack that could allow us to remove the irq handler - footprint from the regular per-process-stack, so it could allow - us to live with a smaller kernel stack compared to the other - linux architectures. - - 3rd stage: - - Before going into production if we still have the order 2 - allocation we can add a sysctl that allows the kernel stack to be - allocated with vmalloc during memory fragmentation. This have to - remain turned off during benchmarks :) but it should be ok in real - life. - -Order of PAGE_CACHE_SIZE and other allocations: - - On the long run we can increase the PAGE_CACHE_SIZE to be - an order 2 allocations and also the slab/buffercache etc.ec.. - could be all done with order 2 allocations. To make the above - to work we should change lots of common code thus it can be done - only once the basic port will be in a production state. Having - a working PAGE_CACHE_SIZE would be a benefit also for - IA32 and other architectures of course. - -Andrea <[email protected]> SuSE +-Andi Kleen, Jul 2004
arch/x86_64/ia32/syscall32.c +19 -12
@@ -40,23 +40,30 @@ static int use_sysenter __initdata = -1;*/ int __map_syscall32 (struct mm_struct *mm,unsigned long address) { +pml4_t *pml4; +pgd_t *pgd;pte_t *pte;pmd_t *pmd; -int err =0 ; +int err = -ENOMEM ;spin_lock (&mm->page_table_lock); - pmd =pmd_alloc (mm,pgd_offset (mm, address), address); -if (pmd && (pte =pte_alloc_map (mm, pmd, address)) !=NULL ) { -if (pte_none (*pte)) { -set_pte (pte, -mk_pte (virt_to_page (syscall32_page), -PAGE_KERNEL_VSYSCALL )); + pml4 =pml4_offset (mm, address); + pgd =pgd_alloc (mm, pml4, address); +if (pgd) { + pmd =pmd_alloc (mm, pgd, address); +if (pmd && (pte =pte_alloc_map (mm, pmd, address)) !=NULL ) { +if (pte_none (*pte)) { +set_pte (pte, +mk_pte (virt_to_page (syscall32_page), +PAGE_KERNEL_VSYSCALL )); + } +/* Flush only the local CPU. Other CPUs taking a fault +will just end up here again +This probably not needed and just paranoia. */ +__flush_tlb_one (address); + err =0 ; } -/* Flush only the local CPU. Other CPUs taking a fault -will just end up here again */ -__flush_tlb_one (address); - }else - err = -ENOMEM ; + }spin_unlock (&mm->page_table_lock);return err; }
arch/x86_64/kernel/acpi/sleep.c +6 -2
@@ -61,9 +61,13 @@ extern char wakeup_start, wakeup_end;extern unsigned long FASTCALL (acpi_copy_wakeup_routine (unsigned long )); +static pml4_t low_ptr; +static void init_low_mapping (void ) { - cpu_pda[0 ].level4_pgt[0 ] = cpu_pda[0 ].level4_pgt[pml4_index (PAGE_OFFSET )]; +pml4_t *slot0 =pml4_offset (current->mm,0UL ); + low_ptr = *slot0; +set_pml4 (slot0, *pml4_offset (current->mm,PAGE_OFFSET ));flush_tlb_all (); }
@@ -97,7 +101,7 @@ int acpi_save_state_disk (void)*/ void acpi_restore_state_mem (void ) { - cpu_pda[0 ].level4_pgt[0 ] =0 ; +set_pml4 (pml4_offset (current->mm,0UL ), low_ptr);flush_tlb_all (); }
arch/x86_64/kernel/head.S +0 -1
@@ -213,7 +213,6 @@ ENTRY(init_level4_pgt) .quad 0x0000000000103007 /* -> level3_kernel_pgt */ .org 0x2000 -/* Kernel does not "know" about 4-th level of page tables. */ ENTRY(level3_ident_pgt) .quad 0x0000000000104007 .fill 511,8,0
arch/x86_64/kernel/init_task.c +0 -2
@@ -47,5 +47,3 @@ EXPORT_SYMBOL(init_task); DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp;#define ALIGN_TO_4K __attribute__((section(".data.init_task"))) - -pgd_t boot_vmalloc_pgt[512] ALIGN_TO_4K;
arch/x86_64/kernel/setup64.c +1 -10
@@ -100,7 +100,6 @@ void __init setup_per_cpu_areas(void)void pda_init (int cpu) { -pml4_t *level4;struct x8664_pda *pda = &cpu_pda[cpu];/* Setup up data that may be needed in __get_free_pages early */
@@ -119,22 +118,14 @@ void pda_init(int cpu)/* others are initialized in smpboot.c */ pda->pcurrent = &init_task; pda->irqstackptr = boot_cpu_stack; - level4 = init_level4_pgt; }else { - level4 = (pml4_t *)__get_free_pages (GFP_ATOMIC ,0 ); -if (!level4) -panic ("Cannot allocate top level page for cpu %d" , cpu); pda->irqstackptr = (char *)__get_free_pages (GFP_ATOMIC ,IRQSTACK_ORDER );if (!pda->irqstackptr)panic ("cannot allocate irqstack for cpu %d" , cpu); } - pda->level4_pgt = (unsigned long *)level4; -if (level4 != init_level4_pgt) -memcpy (level4, &init_level4_pgt,PAGE_SIZE ); -set_pml4 (level4 +510 ,mk_kernel_pml4 (__pa_symbol (boot_vmalloc_pgt))); - asmvolatile ("movq %0,%%cr3" ::"r" (__pa (level4))); + asmvolatile ("movq %0,%%cr3" ::"r" (__pa_symbol (&init_level4_pgt))); pda->irqstackptr +=IRQSTACKSIZE -64 ; }
arch/x86_64/mm/fault.c +53 -34
109 lines not on this page · raw message
arch/x86_64/mm/init.c +17 -26
81 lines not on this page · raw message
arch/x86_64/mm/ioremap.c +31 -8
@@ -67,31 +67,54 @@ static inline int remap_area_pmd(pmd_t *return 0 ; } +static inline int remap_area_pgd (pgd_t * pgd,unsigned long address,unsigned long size, +unsigned long phys_addr,unsigned long flags) +{ +unsigned long end; + + address &= ~PML4_MASK ; + end = address + size; +if (end >PML4_SIZE ) + end =PML4_SIZE ; + phys_addr -= address; +if (address >= end) +BUG (); +do { +pmd_t * pmd =pmd_alloc (&init_mm, pgd, address); +if (!pmd) +return -ENOMEM ; +remap_area_pmd (pmd, address, end - address, address + phys_addr, flags); + address = (address +PGDIR_SIZE ) &PGDIR_MASK ; + pmd++; + }while (address && (address < end)); +return 0 ; +} +static int remap_area_pages (unsigned long address,unsigned long phys_addr,unsigned long size,unsigned long flags) {int error; -pgd_t * dir; +pml4_t *pml4;unsigned long end = address + size; phys_addr -= address; - dir =pgd_offset_k (address); + pml4 =pml4_offset_k (address);flush_cache_all ();if (address >= end)BUG ();spin_lock (&init_mm.page_table_lock);do { -pmd_t *pmd; - pmd =pmd_alloc (&init_mm, dir, address); +pgd_t *pgd; + pgd =pgd_alloc (&init_mm, pml4, address); error = -ENOMEM ; -if (!pmd) +if (!pgd)break ; -if (remap_area_pmd (pmd, address, end - address, +if (remap_area_pgd (pgd, address, end - address, phys_addr + address, flags))break ; error =0 ; - address = (address +PGDIR_SIZE ) &PGDIR_MASK ; - dir++; + address = (address +PML4_SIZE ) &PML4_MASK ; + pml4++; }while (address && (address < end));spin_unlock (&init_mm.page_table_lock);flush_tlb_all ();
arch/x86_64/mm/pageattr.c +20 -12
@@ -16,10 +16,14 @@static inline pte_t *lookup_address (unsigned long address) { -pgd_t *pgd =pgd_offset_k (address); +pml4_t *pml4 =pml4_offset_k (address); +pgd_t *pgd;pmd_t *pmd;pte_t *pte; -if (!pgd || !pgd_present (*pgd)) +if (pml4_none (*pml4)) +return NULL ; + pgd =pml4_pgd_offset (pml4, address); +if (!pgd_present (*pgd))return NULL ; pmd =pmd_offset (pgd, address);if (!pmd_present (*pmd))
@@ -98,16 +102,20 @@ static inline void save_page(unsigned lo*/ static void revert_page (unsigned long address,pgprot_t ref_prot) { -pgd_t *pgd; -pmd_t *pmd; -pte_t large_pte; - - pgd =pgd_offset_k (address); - pmd =pmd_offset (pgd, address); -BUG_ON (pmd_val (*pmd) & _PAGE_PSE); -pgprot_val (ref_prot) |= _PAGE_PSE; - large_pte =mk_pte_phys (__pa (address) &LARGE_PAGE_MASK , ref_prot); -set_pte ((pte_t *)pmd, large_pte); +pml4_t *pml4; +pgd_t *pgd; +pmd_t *pmd; +pte_t large_pte; + + pml4 =pml4_offset_k (address); +BUG_ON (pml4_none (*pml4)); + pgd =pml4_pgd_offset (pml4,address); +BUG_ON (pgd_none (*pgd)); + pmd =pmd_offset (pgd, address); +BUG_ON (pmd_val (*pmd) & _PAGE_PSE); +pgprot_val (ref_prot) |= _PAGE_PSE; + large_pte =mk_pte_phys (__pa (address) &LARGE_PAGE_MASK , ref_prot); +set_pte ((pte_t *)pmd, large_pte); } static int
include/asm-x86_64/e820.h +0 -3
@@ -26,9 +26,6 @@#define LOWMEMSIZE () (0x9f000) -#define MAXMEM (120UL * 1024 * 1024 * 1024 * 1024)/* 120TB */ - - #ifndef __ASSEMBLY__ struct e820entry { u64 addr; /* start of memory segment */
include/asm-x86_64/mmu_context.h +1 -4
@@ -40,10 +40,7 @@ static inline void switch_mm(struct mm_swrite_pda (active_mm, next);#endif set_bit (cpu, &next->cpu_vm_mask); -/* Re-load page tables */ - *read_pda (level4_pgt) =__pa (next->pgd) | _PAGE_TABLE; -__flush_tlb (); - + asmvolatile ("movq %0,%%cr3" ::"r" (__pa (next->pml4)) :"memory" );if (unlikely (next->context.ldt != prev->context.ldt))load_LDT_nolock (&next->context, cpu); }
include/asm-x86_64/page.h +3 -3
@@ -67,19 +67,19 @@ extern unsigned long vm_force_exec32;#define __START_KERNEL 0xffffffff80100000UL#define __START_KERNEL_map 0xffffffff80000000UL -#define __PAGE_OFFSET 0x0000010000000000UL/* 1 << 40 */ +#define __PAGE_OFFSET 0xffff810000000000UL#else #define __START_KERNEL 0xffffffff80100000#define __START_KERNEL_map 0xffffffff80000000 -#define __PAGE_OFFSET 0x0000010000000000/* 1 << 40 */ +#define __PAGE_OFFSET 0xffff810000000000#endif /* !__ASSEMBLY__ */ /* to align the pointer to the (next) page boundary */ #define PAGE_ALIGN (addr) (((addr)+PAGE_SIZE-1)&PAGE_MASK)/* See Documentation/x86_64/mm.txt for a description of the memory map. */ -#define __PHYSICAL_MASK_SHIFT 40 +#define __PHYSICAL_MASK_SHIFT 46#define __PHYSICAL_MASK ((1UL << __PHYSICAL_MASK_SHIFT) - 1)#define __VIRTUAL_MASK_SHIFT 48#define __VIRTUAL_MASK ((1UL << __VIRTUAL_MASK_SHIFT) - 1)
include/asm-x86_64/pda.h +0 -1
@@ -17,7 +17,6 @@ struct x8664_pda {int irqcount;/* Irq nesting counter. Starts with -1 */ int cpunumber;/* Logical CPU number */ char *irqstackptr;/* top of irqstack */ -unsigned long volatile *level4_pgt;/* Per CPU top level page table */ unsigned int __softirq_pending;unsigned int __nmi_count;/* number of NMI on this CPUs */ struct mm_struct *active_mm;
include/asm-x86_64/pgalloc.h +29 -1
@@ -11,6 +11,8 @@ set_pmd(pmd, __pmd(_PAGE_TABLE | __pa(pte)))#define pgd_populate (mm, pgd, pmd) \ set_pgd(pgd, __pgd(_PAGE_TABLE | __pa(pmd))) +#define pml4_populate (mm, pml4, pgd) \ + set_pml4(pml4, __pml4(_PAGE_TABLE | __pa(pgd)))static inline void pmd_populate (struct mm_struct *mm,pmd_t *pmd,struct page *pte) {
@@ -33,7 +35,7 @@ static inline pmd_t *pmd_alloc_one (strureturn (pmd_t *)get_zeroed_page (GFP_KERNEL |__GFP_REPEAT); } -static inline pgd_t *pgd_alloc (struct mm_struct *mm) +static inline pgd_t *pgd_alloc_one (struct mm_struct *mm,unsigned long addr) {return (pgd_t *)get_zeroed_page (GFP_KERNEL |__GFP_REPEAT); }
@@ -44,6 +46,31 @@ static inline void pgd_free (pgd_t *pgd)free_page ((unsigned long )pgd); } +static inline pml4_t *pml4_alloc (struct mm_struct *mm) +{ +unsigned boundary; +pml4_t *pml4 = (pml4_t *)__get_free_page (GFP_KERNEL |__GFP_REPEAT); +if (!pml4) +return NULL ; +/* +* Copy kernel pointers in from init. +* Could keep a freelist or slab cache of those because the kernel +* part never changes. +*/ + boundary =pml4_index (__PAGE_OFFSET); +memset (pml4,0 , boundary *sizeof (pml4_t)); +memcpy (pml4 + boundary, + init_level4_pgt + boundary, + (PTRS_PER_PML4 - boundary) *sizeof (pml4_t)); +return pml4; +} + +static inline void pml4_free (pml4_t *pml4) +{ +BUG_ON ((unsigned long )pml4 & (PAGE_SIZE -1 )); +free_page ((unsigned long )pml4); +} +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address) {return (pte_t *)get_zeroed_page (GFP_KERNEL |__GFP_REPEAT);
@@ -73,5 +100,6 @@ extern inline void pte_free(struct page#define __pte_free_tlb (tlb,pte) tlb_remove_page((tlb),(pte))#define __pmd_free_tlb (tlb,x) pmd_free(x) +#define __pgd_free_tlb (tlb,x) pgd_free(x)#endif /* _X86_64_PGALLOC_H */
include/asm-x86_64/pgtable.h +33 -55
155 lines not on this page · raw message
include/asm-x86_64/processor.h +2 -2
@@ -165,9 +165,9 @@ static inline void clear_in_cr4 (unsigne/* -* User space process size: 512GB - 1GB (default). +* User space process size. 47bits. */ -#define TASK_SIZE (0x0000007fc0000000UL) +#define TASK_SIZE 0x800000000000/* This decides where the kernel will search for a free chunk of vm * space during mmap's.