Skip to contentSkip to conversation

4level support for x86_64

PATCH In review Andreas Kleen 1 participant 1 message Oct 25, 2004 Documentation

lore
Share & tools
lore
#0 lore raw

PATCH 17/17 4level support for x86_64

4level support for x86_64

This patch extends x86-64 to use a true 4level VM. This extends all user address spaces to 47bits (128TB)

fork/exec performance in lmbench is down by a few percent because they have to walk more page tables right now. There are plans to recovers this with future optimizations.

Documentation/x86_64/mm.txt +21 -145
diff -urpN -X ../KDIFX linux-2.6.10rc1/Documentation/x86_64/mm.txt linux-2.6.10rc1-4level/Documentation/x86_64/mm.txt
@@ -1,148 +1,24 @@
-The paging design used on the x86-64 linux kernel port in 2.4.x provides:
 
-o	per process virtual address space limit of 512 Gigabytes
-o	top of userspace stack located at address 0x0000007fffffffff
-o	PAGE_OFFSET = 0xffff800000000000
-o	start of the kernel = 0xffffffff800000000
-o	global RAM per system 2^64-PAGE_OFFSET-sizeof(kernel) = 128 Terabytes - 2 Gigabytes
-o	no need of any common code change
-o	no need to use highmem to handle the 128 Terabytes of RAM
+<previous description obsolete, deleted> 
+ 
+Virtual memory map with 4 level page tables: 
+
+0000000000000000 - 00007fffffffffff (=47bits) user space, different per mm
+hole caused by [48:63] sign extension
+ffff800000000000 - ffff80ffffffffff (=40bits) guard hole
+ffff810000000000 - ffffc0ffffffffff (=46bits) direct mapping of phys. memory
+ffffc10000000000 - ffffc1ffffffffff (=40bits) hole
+ffffc20000000000 - ffffe1ffffffffff (=45bits) vmalloc/ioremap space
+... unused hole ... 
+ffffffff80000000 - ffffffff82800000 (=40MB)   kernel text mapping, from phys 0
+... unused hole ...
+ffffffff88000000 - fffffffffff00000 (=1919MB) module mapping space
+
+vmalloc space is lazily synchronized into the different PML4 pages of
+the processes using the page fault handler, with init_level4_pgt as
+reference.
 
-Description:
+Current X86-64 implementations only support 40 bit of address space,
+but we support upto 46bits. This expands into MBZ space in the page tables.
 
-	Userspace is able to modify and it sees only the 3rd/2nd/1st level
-	pagetables (pgd_offset() implicitly walks the 1st slot of the 4th
-	level pagetable and it returns an entry into the 3rd level pagetable).
-	This is where the per-process 512 Gigabytes limit cames from.
-
-	The common code pgd is the PDPE, the pmd is the PDE, the
-	pte is the PTE. The PML4E remains invisible to the common
-	code.
-
-	The kernel uses all the first 47 bits of the negative half
-	of the virtual address space to build the direct mapping using
-	2 Mbytes page size. The kernel virtual	addresses have bit number
-	47 always set to 1 (and in turn also bits 48-63 are set to 1 too,
-	due the sign extension). This is where the 128 Terabytes - 2 Gigabytes global
-	limit of RAM cames from.
-
-	Since the per-process limit is 512 Gigabytes (due to kernel common
-	code 3 level pagetable limitation), the higher virtual address mapped
-	into userspace is 0x7fffffffff and it makes sense to use it
-	as the top of the userspace stack to allow the stack to grow as
-	much as possible.
-
-	Setting the PAGE_OFFSET to 2^39 (after the last userspace
-	virtual address) wouldn't make much difference compared to
-	setting PAGE_OFFSET to 0xffff800000000000 because we have an
-	hole into the virtual address space. The last byte mapped by the
-	255th slot in the 4th level pagetable is at virtual address
-	0x00007fffffffffff and the first byte mapped by the 256th slot in the
-	4th level pagetable is at address 0xffff800000000000. Due to this
-	hole we can't trivially build a direct mapping across all the
-	512 slots of the 4th level pagetable, so we simply use only the
-	second (negative) half of the 4th level pagetable for that purpose
-	(that provides us 128 Terabytes of contigous virtual addresses).
-	Strictly speaking we could build a direct mapping also across the hole
-	using some DISCONTIGMEM trick, but we don't need such a large
-	direct mapping right now.
-
-Future:
-
-	During 2.5.x we can break the 512 Gigabytes per-process limit
-	possibly by removing from the common code any knowledge about the
-	architectural dependent physical layout of the virtual to physical
-	mapping.
-
-	Once the 512 Gigabytes limit will be removed the kernel stack will
-	be moved (most probably to virtual address 0x00007fffffffffff).
-	Nothing	will break in userspace due that move, as nothing breaks
-	in IA32 compiling the kernel with CONFIG_2G.
-
-Linus agreed on not breaking common code and to live with the 512 Gigabytes
-per-process limitation for the 2.4.x timeframe and he has given me and Andi
-some very useful hints... (thanks! :)
-
-Thanks also to H. Peter Anvin for his interesting and useful suggestions on
-the x86-64-discuss lists!
-
-Other memory management related issues follows:
-
-PAGE_SIZE:
-
-	If somebody is wondering why these days we still have a so small
-	4k pagesize (16 or 32 kbytes would be much better for performance
-	of course), the PAGE_SIZE have to remain 4k for 32bit apps to
-	provide 100% backwards compatible IA32 API (we can't allow silent
-	fs corruption or as best a loss of coherency with the page cache
-	by allocating MAP_SHARED areas in MAP_ANONYMOUS memory with a
-	do_mmap_fake). I think it could be possible to have a dynamic page
-	size between 32bit and 64bit apps but it would need extremely
-	intrusive changes in the common code as first for page cache and
-	we sure don't want to depend on them right now even if the
-	hardware would support that.
-
-PAGETABLE SIZE:
-
-	In turn we can't afford to have pagetables larger than 4k because
-	we could not be able to allocate them due physical memory
-	fragmentation, and failing to allocate the kernel stack is a minor
-	issue compared to failing the allocation of a pagetable. If we
-	fail the allocation of a pagetable the only thing we can do is to
-	sched_yield polling the freelist (deadlock prone) or to segfault
-	the task (not even the sighandler would be sure to run).
-
-KERNEL STACK:
-
-	1st stage:
-
-	The kernel stack will be at first allocated with an order 2 allocation
-	(16k) (the utilization of the stack for a 64bit platform really
-	isn't exactly the double of a 32bit platform because the local
-	variables may not be all 64bit wide, but not much less). This will
-	make things even worse than they are right now on IA32 with
-	respect of failing fork/clone due memory fragmentation.
-
-	2nd stage:
-
-	We'll benchmark if reserving one register as task_struct
-	pointer will improve performance of the kernel (instead of
-	recalculating the task_struct pointer starting from the stack
-	pointer each time). My guess is that recalculating will be faster
-	but it worth a try.
-
-		If reserving one register for the task_struct pointer
-		will be faster we can as well split task_struct and kernel
-		stack. task_struct can be a slab allocation or a
-		PAGE_SIZEd allocation, and the kernel stack can then be
-		allocated in a order 1 allocation. Really this is risky,
-		since 8k on a 64bit platform is going to be less than 7k
-		on a 32bit platform but we could try it out. This would
-		reduce the fragmentation problem of an order of magnitude
-		making it equal to the current IA32.
-
-		We must also consider the x86-64 seems to provide in hardware a
-		per-irq stack that could allow us to remove the irq handler
-		footprint from the regular per-process-stack, so it could allow
-		us to live with a smaller kernel stack compared to the other
-		linux architectures.
-
-	3rd stage:
-
-	Before going into production if we still have the order 2
-	allocation we can add a sysctl that allows the kernel stack to be
-	allocated with vmalloc during memory fragmentation. This have to
-	remain turned off during benchmarks :) but it should be ok in real
-	life.
-
-Order of PAGE_CACHE_SIZE and other allocations:
-
-	On the long run we can increase the PAGE_CACHE_SIZE to be
-	an order 2 allocations and also the slab/buffercache etc.ec..
-	could be all done with order 2 allocations. To make the above
-	to work we should change lots of common code thus it can be done
-	only once the basic port will be in a production state. Having
-	a working PAGE_CACHE_SIZE would be a benefit also for
-	IA32 and other architectures of course.
-
-Andrea <[email protected]> SuSE
+-Andi Kleen, Jul 2004
arch/x86_64/ia32/syscall32.c +19 -12
diff -urpN -X ../KDIFX linux-2.6.10rc1/arch/x86_64/ia32/syscall32.c linux-2.6.10rc1-4level/arch/x86_64/ia32/syscall32.c
@@ -40,23 +40,30 @@ static int use_sysenter __initdata = -1;
  */
 int __map_syscall32(struct mm_struct *mm, unsigned long address)
 { 
+	pml4_t *pml4;
+	pgd_t *pgd;
 	pte_t *pte;
 	pmd_t *pmd;
-	int err = 0;
+	int err = -ENOMEM;
 
 	spin_lock(&mm->page_table_lock); 
-	pmd = pmd_alloc(mm, pgd_offset(mm, address), address); 
-	if (pmd && (pte = pte_alloc_map(mm, pmd, address)) != NULL) { 
-		if (pte_none(*pte)) { 
-			set_pte(pte, 
-				mk_pte(virt_to_page(syscall32_page), 
-				       PAGE_KERNEL_VSYSCALL)); 
+ 	pml4 = pml4_offset(mm, address); 
+ 	pgd = pgd_alloc(mm, pml4, address);
+ 	if (pgd) { 
+ 		pmd = pmd_alloc(mm, pgd, address); 
+ 		if (pmd && (pte = pte_alloc_map(mm, pmd, address)) != NULL) { 
+ 			if (pte_none(*pte)) { 
+ 				set_pte(pte, 
+ 					mk_pte(virt_to_page(syscall32_page), 
+ 					       PAGE_KERNEL_VSYSCALL)); 
+ 			}
+ 			/* Flush only the local CPU. Other CPUs taking a fault
+ 			   will just end up here again
+			   This probably not needed and just paranoia. */
+ 			__flush_tlb_one(address); 
+ 			err = 0;
 		}
-		/* Flush only the local CPU. Other CPUs taking a fault
-		   will just end up here again */
-		__flush_tlb_one(address); 
-	} else
-		err = -ENOMEM; 
+	}
 	spin_unlock(&mm->page_table_lock);
 	return err;
 }
arch/x86_64/kernel/acpi/sleep.c +6 -2
diff -urpN -X ../KDIFX linux-2.6.10rc1/arch/x86_64/kernel/acpi/sleep.c linux-2.6.10rc1-4level/arch/x86_64/kernel/acpi/sleep.c
@@ -61,9 +61,13 @@ extern char wakeup_start, wakeup_end;
 
 extern unsigned long FASTCALL(acpi_copy_wakeup_routine(unsigned long));
 
+static pml4_t low_ptr; 
+
 static void init_low_mapping(void)
 {
-	cpu_pda[0].level4_pgt[0] = cpu_pda[0].level4_pgt[pml4_index(PAGE_OFFSET)];
+	pml4_t *slot0 = pml4_offset(current->mm, 0UL);
+	low_ptr = *slot0;
+	set_pml4(slot0, *pml4_offset(current->mm, PAGE_OFFSET)); 
 	flush_tlb_all();
 }
 
@@ -97,7 +101,7 @@ int acpi_save_state_disk (void)
  */
 void acpi_restore_state_mem (void)
 {
-	cpu_pda[0].level4_pgt[0] = 0;
+	set_pml4(pml4_offset(current->mm, 0UL), low_ptr); 
 	flush_tlb_all();
 }
 
arch/x86_64/kernel/head.S +0 -1
diff -urpN -X ../KDIFX linux-2.6.10rc1/arch/x86_64/kernel/head.S linux-2.6.10rc1-4level/arch/x86_64/kernel/head.S
@@ -213,7 +213,6 @@ ENTRY(init_level4_pgt)
 	.quad	0x0000000000103007		/* -> level3_kernel_pgt */
 
 .org 0x2000
-/* Kernel does not "know" about 4-th level of page tables. */
 ENTRY(level3_ident_pgt)
 	.quad	0x0000000000104007
 	.fill	511,8,0
arch/x86_64/kernel/init_task.c +0 -2
diff -urpN -X ../KDIFX linux-2.6.10rc1/arch/x86_64/kernel/init_task.c linux-2.6.10rc1-4level/arch/x86_64/kernel/init_task.c
@@ -47,5 +47,3 @@ EXPORT_SYMBOL(init_task);
 DEFINE_PER_CPU(struct tss_struct, init_tss) ____cacheline_maxaligned_in_smp;
 
 #define ALIGN_TO_4K __attribute__((section(".data.init_task")))
-
-pgd_t boot_vmalloc_pgt[512]  ALIGN_TO_4K;
arch/x86_64/kernel/setup64.c +1 -10
diff -urpN -X ../KDIFX linux-2.6.10rc1/arch/x86_64/kernel/setup64.c linux-2.6.10rc1-4level/arch/x86_64/kernel/setup64.c
@@ -100,7 +100,6 @@ void __init setup_per_cpu_areas(void)
 
 void pda_init(int cpu)
 { 
-        pml4_t *level4;
 	struct x8664_pda *pda = &cpu_pda[cpu];
 
 	/* Setup up data that may be needed in __get_free_pages early */
@@ -119,22 +118,14 @@ void pda_init(int cpu)
 		/* others are initialized in smpboot.c */
 		pda->pcurrent = &init_task;
 		pda->irqstackptr = boot_cpu_stack; 
-		level4 = init_level4_pgt; 
 	} else {
-		level4 = (pml4_t *)__get_free_pages(GFP_ATOMIC, 0); 
-		if (!level4) 
-			panic("Cannot allocate top level page for cpu %d", cpu); 
 		pda->irqstackptr = (char *)
 			__get_free_pages(GFP_ATOMIC, IRQSTACK_ORDER);
 		if (!pda->irqstackptr)
 			panic("cannot allocate irqstack for cpu %d", cpu); 
 	}
 
-	pda->level4_pgt = (unsigned long *)level4; 
-	if (level4 != init_level4_pgt)
-		memcpy(level4, &init_level4_pgt, PAGE_SIZE); 
-	set_pml4(level4 + 510, mk_kernel_pml4(__pa_symbol(boot_vmalloc_pgt)));
-	asm volatile("movq %0,%%cr3" :: "r" (__pa(level4))); 
+	asm volatile("movq %0,%%cr3" :: "r" (__pa_symbol(&init_level4_pgt)));
 
 	pda->irqstackptr += IRQSTACKSIZE-64;
 } 
arch/x86_64/mm/fault.c +53 -34

109 lines not on this page · raw message

arch/x86_64/mm/init.c +17 -26

81 lines not on this page · raw message

arch/x86_64/mm/ioremap.c +31 -8
diff -urpN -X ../KDIFX linux-2.6.10rc1/arch/x86_64/mm/ioremap.c linux-2.6.10rc1-4level/arch/x86_64/mm/ioremap.c
@@ -67,31 +67,54 @@ static inline int remap_area_pmd(pmd_t *
 	return 0;
 }
 
+static inline int remap_area_pgd(pgd_t * pgd, unsigned long address, unsigned long size,
+	unsigned long phys_addr, unsigned long flags)
+{
+	unsigned long end;
+
+	address &= ~PML4_MASK;
+	end = address + size;
+	if (end > PML4_SIZE)
+		end = PML4_SIZE;
+	phys_addr -= address;
+	if (address >= end)
+		BUG();
+	do {
+		pmd_t * pmd = pmd_alloc(&init_mm, pgd, address);
+		if (!pmd)
+			return -ENOMEM;
+		remap_area_pmd(pmd, address, end - address, address + phys_addr, flags);
+		address = (address + PGDIR_SIZE) & PGDIR_MASK;
+		pmd++;
+	} while (address && (address < end));
+	return 0;
+}
+
 static int remap_area_pages(unsigned long address, unsigned long phys_addr,
 				 unsigned long size, unsigned long flags)
 {
 	int error;
-	pgd_t * dir;
+	pml4_t *pml4;
 	unsigned long end = address + size;
 
 	phys_addr -= address;
-	dir = pgd_offset_k(address);
+	pml4 = pml4_offset_k(address); 
 	flush_cache_all();
 	if (address >= end)
 		BUG();
 	spin_lock(&init_mm.page_table_lock);
 	do {
-		pmd_t *pmd;
-		pmd = pmd_alloc(&init_mm, dir, address);
+		pgd_t *pgd;
+		pgd = pgd_alloc(&init_mm, pml4, address);
 		error = -ENOMEM;
-		if (!pmd)
+		if (!pgd)
 			break;
-		if (remap_area_pmd(pmd, address, end - address,
+		if (remap_area_pgd(pgd, address, end - address,
 					 phys_addr + address, flags))
 			break;
 		error = 0;
-		address = (address + PGDIR_SIZE) & PGDIR_MASK;
-		dir++;
+		address = (address + PML4_SIZE) & PML4_MASK;
+		pml4++;
 	} while (address && (address < end));
 	spin_unlock(&init_mm.page_table_lock);
 	flush_tlb_all();
arch/x86_64/mm/pageattr.c +20 -12
diff -urpN -X ../KDIFX linux-2.6.10rc1/arch/x86_64/mm/pageattr.c linux-2.6.10rc1-4level/arch/x86_64/mm/pageattr.c
@@ -16,10 +16,14 @@
 
 static inline pte_t *lookup_address(unsigned long address) 
 { 
-	pgd_t *pgd = pgd_offset_k(address); 
+	pml4_t *pml4 = pml4_offset_k(address);
+	pgd_t *pgd;
 	pmd_t *pmd;
 	pte_t *pte;
-	if (!pgd || !pgd_present(*pgd))
+	if (pml4_none(*pml4))
+		return NULL; 
+	pgd = pml4_pgd_offset(pml4, address);
+	if (!pgd_present(*pgd))
 		return NULL; 
 	pmd = pmd_offset(pgd, address); 	       
 	if (!pmd_present(*pmd))
@@ -98,16 +102,20 @@ static inline void save_page(unsigned lo
  */
 static void revert_page(unsigned long address, pgprot_t ref_prot)
 {
-       pgd_t *pgd;
-       pmd_t *pmd; 
-       pte_t large_pte; 
-       
-       pgd = pgd_offset_k(address); 
-       pmd = pmd_offset(pgd, address);
-       BUG_ON(pmd_val(*pmd) & _PAGE_PSE); 
-       pgprot_val(ref_prot) |= _PAGE_PSE;
-       large_pte = mk_pte_phys(__pa(address) & LARGE_PAGE_MASK, ref_prot);
-       set_pte((pte_t *)pmd, large_pte);
+	pml4_t *pml4; 
+	pgd_t *pgd;
+	pmd_t *pmd; 
+	pte_t large_pte; 
+	
+	pml4 = pml4_offset_k(address);
+	BUG_ON(pml4_none(*pml4));
+	pgd = pml4_pgd_offset(pml4,address); 
+	BUG_ON(pgd_none(*pgd)); 
+	pmd = pmd_offset(pgd, address);
+	BUG_ON(pmd_val(*pmd) & _PAGE_PSE); 
+	pgprot_val(ref_prot) |= _PAGE_PSE;
+	large_pte = mk_pte_phys(__pa(address) & LARGE_PAGE_MASK, ref_prot);
+	set_pte((pte_t *)pmd, large_pte);
 }      
 
 static int
include/asm-x86_64/e820.h +0 -3
diff -urpN -X ../KDIFX linux-2.6.10rc1/include/asm-x86_64/e820.h linux-2.6.10rc1-4level/include/asm-x86_64/e820.h
@@ -26,9 +26,6 @@
 
 #define LOWMEMSIZE()	(0x9f000)
 
-#define MAXMEM		(120UL * 1024 * 1024 * 1024 * 1024)  /* 120TB */ 
-
-
 #ifndef __ASSEMBLY__
 struct e820entry {
 	u64 addr;	/* start of memory segment */
include/asm-x86_64/mmu_context.h +1 -4
diff -urpN -X ../KDIFX linux-2.6.10rc1/include/asm-x86_64/mmu_context.h linux-2.6.10rc1-4level/include/asm-x86_64/mmu_context.h
@@ -40,10 +40,7 @@ static inline void switch_mm(struct mm_s
 		write_pda(active_mm, next);
 #endif
 		set_bit(cpu, &next->cpu_vm_mask);
-		/* Re-load page tables */
-		*read_pda(level4_pgt) = __pa(next->pgd) | _PAGE_TABLE;
-		__flush_tlb();
-
+		asm volatile("movq %0,%%cr3" :: "r" (__pa(next->pml4)) : "memory");
 		if (unlikely(next->context.ldt != prev->context.ldt)) 
 			load_LDT_nolock(&next->context, cpu);
 	}
include/asm-x86_64/page.h +3 -3
diff -urpN -X ../KDIFX linux-2.6.10rc1/include/asm-x86_64/page.h linux-2.6.10rc1-4level/include/asm-x86_64/page.h
@@ -67,19 +67,19 @@ extern unsigned long vm_force_exec32;
 
 #define __START_KERNEL		0xffffffff80100000UL
 #define __START_KERNEL_map	0xffffffff80000000UL
-#define __PAGE_OFFSET           0x0000010000000000UL	/* 1 << 40 */
+#define __PAGE_OFFSET           0xffff810000000000UL
 
 #else
 #define __START_KERNEL		0xffffffff80100000
 #define __START_KERNEL_map	0xffffffff80000000
-#define __PAGE_OFFSET           0x0000010000000000	/* 1 << 40 */
+#define __PAGE_OFFSET           0xffff810000000000
 #endif /* !__ASSEMBLY__ */
 
 /* to align the pointer to the (next) page boundary */
 #define PAGE_ALIGN(addr)	(((addr)+PAGE_SIZE-1)&PAGE_MASK)
 
 /* See Documentation/x86_64/mm.txt for a description of the memory map. */
-#define __PHYSICAL_MASK_SHIFT	40
+#define __PHYSICAL_MASK_SHIFT	46
 #define __PHYSICAL_MASK		((1UL << __PHYSICAL_MASK_SHIFT) - 1)
 #define __VIRTUAL_MASK_SHIFT	48
 #define __VIRTUAL_MASK		((1UL << __VIRTUAL_MASK_SHIFT) - 1)
include/asm-x86_64/pda.h +0 -1
diff -urpN -X ../KDIFX linux-2.6.10rc1/include/asm-x86_64/pda.h linux-2.6.10rc1-4level/include/asm-x86_64/pda.h
@@ -17,7 +17,6 @@ struct x8664_pda {
         int irqcount;		    /* Irq nesting counter. Starts with -1 */  	
 	int cpunumber;		    /* Logical CPU number */
 	char *irqstackptr;	/* top of irqstack */
-	unsigned long volatile *level4_pgt; /* Per CPU top level page table */
 	unsigned int __softirq_pending;
 	unsigned int __nmi_count;	/* number of NMI on this CPUs */
 	struct mm_struct *active_mm;
include/asm-x86_64/pgalloc.h +29 -1
diff -urpN -X ../KDIFX linux-2.6.10rc1/include/asm-x86_64/pgalloc.h linux-2.6.10rc1-4level/include/asm-x86_64/pgalloc.h
@@ -11,6 +11,8 @@
 		set_pmd(pmd, __pmd(_PAGE_TABLE | __pa(pte)))
 #define pgd_populate(mm, pgd, pmd) \
 		set_pgd(pgd, __pgd(_PAGE_TABLE | __pa(pmd)))
+#define pml4_populate(mm, pml4, pgd) \
+		set_pml4(pml4, __pml4(_PAGE_TABLE | __pa(pgd)))
 
 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, struct page *pte)
 {
@@ -33,7 +35,7 @@ static inline pmd_t *pmd_alloc_one (stru
 	return (pmd_t *)get_zeroed_page(GFP_KERNEL|__GFP_REPEAT);
 }
 
-static inline pgd_t *pgd_alloc (struct mm_struct *mm)
+static inline pgd_t *pgd_alloc_one(struct mm_struct *mm, unsigned long addr)
 {
 	return (pgd_t *)get_zeroed_page(GFP_KERNEL|__GFP_REPEAT);
 }
@@ -44,6 +46,31 @@ static inline void pgd_free (pgd_t *pgd)
 	free_page((unsigned long)pgd);
 }
 
+static inline pml4_t *pml4_alloc(struct mm_struct *mm) 
+{ 
+	unsigned boundary;
+	pml4_t *pml4 = (pml4_t *)__get_free_page(GFP_KERNEL|__GFP_REPEAT);
+	if (!pml4)
+		return NULL;
+	/*
+	 * Copy kernel pointers in from init.
+	 * Could keep a freelist or slab cache of those because the kernel 
+	 * part never changes.
+	 */
+	boundary = pml4_index(__PAGE_OFFSET);
+	memset(pml4, 0, boundary * sizeof(pml4_t)); 
+	memcpy(pml4 + boundary, 
+	       init_level4_pgt + boundary,
+	       (PTRS_PER_PML4 - boundary) * sizeof(pml4_t));
+	return pml4; 
+} 
+
+static inline void pml4_free(pml4_t *pml4)
+{ 
+	BUG_ON((unsigned long)pml4 & (PAGE_SIZE-1)); 
+	free_page((unsigned long)pml4);
+} 
+
 static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
 {
 	return (pte_t *)get_zeroed_page(GFP_KERNEL|__GFP_REPEAT);
@@ -73,5 +100,6 @@ extern inline void pte_free(struct page 
 
 #define __pte_free_tlb(tlb,pte) tlb_remove_page((tlb),(pte))
 #define __pmd_free_tlb(tlb,x)   pmd_free(x)
+#define __pgd_free_tlb(tlb,x)   pgd_free(x)
 
 #endif /* _X86_64_PGALLOC_H */
include/asm-x86_64/pgtable.h +33 -55

155 lines not on this page · raw message

include/asm-x86_64/processor.h +2 -2
diff -urpN -X ../KDIFX linux-2.6.10rc1/include/asm-x86_64/processor.h linux-2.6.10rc1-4level/include/asm-x86_64/processor.h
@@ -165,9 +165,9 @@ static inline void clear_in_cr4 (unsigne
 
 
 /*
- * User space process size: 512GB - 1GB (default).
+ * User space process size. 47bits. 
  */
-#define TASK_SIZE	(0x0000007fc0000000UL)
+#define TASK_SIZE	0x800000000000
 
 /* This decides where the kernel will search for a free chunk of vm
  * space during mmap's.