Increased address space for 64 bit
Subsystems (2)
v8 cover Increased address space for 64 bit
From: Benjamin Berg <[email protected]>
The new version of the patchset uses execveat on a memfd instead of cloning twice to disable rseq. This should be much more robust going forward as it will also avoid issues with other new features like mseal.
This patchset fixes a few bugs, adds a new method of discovering the host task size and finally adds four level page table support. All of this means the userspace TASK_SIZE is much larger and in turns permits userspace applications that need a lot of virtual addresses to work fine.
One such application is ASAN which uses a fixed address in memory that would otherwise not be addressable.
v8:
* Make changes suggested by Johannes Berg
v7:
* Plenty of changes to fix 32 bit and improve the logic
v6:
* Apply fixes pointed out by Tiwei Bie
* Add temporary file fallback as memfd is not always supported
v5:
* Use execveat with memfd instead of double clone
v4:
* Do not use WNOHANG in wait for CLONE_VFORK
v3:
* Undo incorrect change in child wait loop
v2:
* Improved double clone logic using CLONE_VFORK
* Kconfig fixes pointed out by Tiwei Bie
Benjamin Berg (7):
um: Add generic stub_syscall1 function
um: use execveat to create userspace MMs
um: Fix stub_start address calculation
um: Limit TASK_SIZE to the addressable range
um: Discover host_task_size from envp
um: clear all memory in new userspace processes
um: Add 4 level page table support
arch/um/Kconfig | 1 + arch/um/include/asm/page.h | 14 +- arch/um/include/asm/pgalloc.h | 11 +- arch/um/include/asm/pgtable-4level.h | 119 ++++++++++++++++ arch/um/include/asm/pgtable.h | 6 +- arch/um/include/shared/as-layout.h | 2 +- arch/um/include/shared/os.h | 2 +- arch/um/include/shared/skas/stub-data.h | 11 ++ arch/um/kernel/mem.c | 17 ++- arch/um/kernel/skas/.gitignore | 2 + arch/um/kernel/skas/Makefile | 33 ++++- arch/um/kernel/skas/mmu.c | 25 +--- arch/um/kernel/skas/stub_exe.c | 88 ++++++++++++ arch/um/kernel/skas/stub_exe_embed.S | 11 ++ arch/um/kernel/um_arch.c | 14 +- arch/um/os-Linux/main.c | 9 +- arch/um/os-Linux/mem.c | 2 +- arch/um/os-Linux/skas/process.c | 181 ++++++++++++++++-------- arch/x86/um/Kconfig | 38 +++-- arch/x86/um/os-Linux/task_size.c | 152 ++------------------ arch/x86/um/shared/sysdep/stub_64.h | 11 ++ 21 files changed, 501 insertions(+), 248 deletions(-) create mode 100644 arch/um/include/asm/pgtable-4level.h create mode 100644 arch/um/kernel/skas/.gitignore create mode 100644 arch/um/kernel/skas/stub_exe.c create mode 100644 arch/um/kernel/skas/stub_exe_embed.S
Signature · 1 line
2.45.2
PATCH v8 1/7 um: Add generic stub_syscall1 function
From: Benjamin Berg <[email protected]>
The 64bit version did not have a stub_syscall1 function yet. Add it as it will be useful to implement a static binary for stub loading.
--- arch/x86/um/shared/sysdep/stub_64.h | 11 +++++++++++ 1 file changed, 11 insertions(+)
arch/x86/um/shared/sysdep/stub_64.h +11 -0
@@ -28,6 +28,17 @@ static __always_inline long stub_syscall0(long syscall)return ret; } +static __always_inline long stub_syscall1 (long syscall,long arg1) +{ +long ret; + + __asm__ volatile (__syscall + : "=a" (ret) + : "0" (syscall), "D" (arg1) : __syscall_clobber ); + +return ret; +} +static __always_inline long stub_syscall2 (long syscall,long arg1,long arg2) {long ret;
Signature · 1 line
2.45.2
PATCH v8 2/7 um: use execveat to create userspace MMs
In reply to Benjamin BergFrom: Benjamin Berg <[email protected]>
Using clone will not undo features that have been enabled by libc. An example of this already happening is rseq, which could cause the kernel to read/write memory of the userspace process. In the future the standard library might also use mseal by default to protect itself, which would also thwart our attempts at unmapping everything.
Solve all this by taking a step back and doing an execve into a tiny static binary that sets up the minimal environment required for the stub without using any standard library. That way we have a clean execution environment that is fully under the control of UML.
Note that this changes things a bit as the FDs are not anymore shared with the kernel. Instead, we explicitly share the FDs for the physical memory and all existing iomem regions. Doing this is fine, as iomem regions cannot be added at runtime.
---
v8:
- Make changes suggested by Johannes Berg
v7:
- Rename stub_elf to stub_exe
- Move into architecture independent directory
- Fix 32 bit issues
- Improve tempfile logic
- Other cleanups
v6:
- Apply fixes pointed out by Tiwei Bie
- Add temporary file fallback as memfd is not always supported
--- arch/um/include/shared/skas/stub-data.h | 11 ++ arch/um/kernel/skas/.gitignore | 2 + arch/um/kernel/skas/Makefile | 33 ++++- arch/um/kernel/skas/stub_exe.c | 88 ++++++++++++ arch/um/kernel/skas/stub_exe_embed.S | 11 ++ arch/um/os-Linux/mem.c | 2 +- arch/um/os-Linux/skas/process.c | 181 ++++++++++++++++-------- 7 files changed, 269 insertions(+), 59 deletions(-) create mode 100644 arch/um/kernel/skas/.gitignore create mode 100644 arch/um/kernel/skas/stub_exe.c create mode 100644 arch/um/kernel/skas/stub_exe_embed.S
arch/um/include/shared/skas/stub-data.h +11 -0
@@ -12,6 +12,17 @@#include <as-layout.h> #include <sysdep/tls.h> +struct stub_init_data { +unsigned long stub_start; + +int stub_code_fd; +unsigned long stub_code_offset; +int stub_data_fd; +unsigned long stub_data_offset; + +unsigned long segv_handler; +}; +#define STUB_NEXT_SYSCALL (s) \ ((struct stub_syscall *) (((unsigned long) s) + (s)->cmd_len))
arch/um/kernel/skas/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +stub_exe +stub_exe.dbg
arch/um/kernel/skas/Makefile +31 -2
@@ -3,14 +3,43 @@ # Copyright (C) 2002 - 2007 Jeff Dike (jdike@{addtoit,linux.intel}.com) # -obj-y := stub.o mmu.o process.o syscall.o uaccess.o +obj-y := stub.o mmu.o process.o syscall.o uaccess.o \ + stub_exe_embed.o + +# Stub executable + +stub_exe_objs-y := stub_exe.o + +stub_exe_objs := $(foreach F,$(stub_exe_objs-y),$(obj)/$F) + +# Object file containing the ELF executable +$(obj)/stub_exe_embed.o: $(src)/stub_exe_embed.S $(obj)/stub_exe + +$(obj)/stub_exe.dbg: $(stub_exe_objs) FORCE + $(call if_changed,stub_exe) + +$(obj)/stub_exe: OBJCOPYFLAGS := -S +$(obj)/stub_exe: $(obj)/stub_exe.dbg FORCE + $(call if_changed,objcopy) + +quiet_cmd_stub_exe = STUB_EXE $@ + cmd_stub_exe = $(CC) -nostdlib -o $@ \ + $(KBUILD_CFLAGS) $(STUB_EXE_LDFLAGS) \ + $(filter %.o,$^) + +STUB_EXE_LDFLAGS = -n -static + +targets += stub_exe.dbg stub_exe $(stub_exe_objs-y) + +# end # stub.o is in the stub, so it can't be built with profiling # GCC hardened also auto-enables -fpic, but we need %ebx so it can't work -> # disable it CFLAGS_stub.o := $(CFLAGS_NO_HARDENING) -UNPROFILE_OBJS := stub.o +CFLAGS_stub_exe.o := $(CFLAGS_NO_HARDENING) +UNPROFILE_OBJS := stub.o stub_exe.o KCOV_INSTRUMENT := n include $(srctree)/arch/um/scripts/Makefile.rules
arch/um/kernel/skas/stub_exe.c new file +88 -0
88 lines not on this page · raw message
arch/um/kernel/skas/stub_exe_embed.S new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/init.h> +#include <linux/linkage.h> + +__INITDATA + +SYM_DATA_START(stub_exe_start) + .incbin "arch/um/kernel/skas/stub_exe" +SYM_DATA_END_LABEL(stub_exe_start, SYM_L_GLOBAL, stub_exe_end) + +__FINIT
arch/um/os-Linux/mem.c +1 -1
@@ -42,7 +42,7 @@ void kasan_map_memory(void *start, size_t len) }/* Set by make_tempfile() during early boot. */ -static char *tempdir =NULL ; +char *tempdir =NULL ;/* Check if dir is on tmpfs. Return 0 if yes, -1 if no or error. */ static int __init check_tmpfs (const char *dir)
arch/um/os-Linux/skas/process.c +125 -56
219 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [PATCH v8 2/7] um: use execveat to create userspace MMs
On 2024/7/5 03:05, Benjamin Berg wrote:
[...]
Quoted 32 lines
static int
userspace_tramp(void *stack)
{(quoted diff) +10
-19- struct sigaction sa; - void *addr; - int fd; + char *const argv[] = { "uml-userspace", NULL }; + int pipe_fds[2]; unsigned long long offset; - unsigned long segv_handler = STUB_CODE + - (unsigned long) stub_segv_handler - - (unsigned long) __syscall_stub_start; - - ptrace(PTRACE_TRACEME, 0, 0, 0); - - signal(SIGTERM, SIG_DFL); - signal(SIGWINCH, SIG_IGN); - - fd = phys_mapping(uml_to_phys(__syscall_stub_start), &offset); - addr = mmap64((void *) STUB_CODE, UM_KERN_PAGE_SIZE, - PROT_EXEC, MAP_FIXED | MAP_PRIVATE, fd, offset); - if (addr == MAP_FAILED) { - os_info("mapping mmap stub at 0x%lx failed, errno = %d\n", - STUB_CODE, errno); - exit(1); + struct stub_init_data init_data = { + .stub_start = STUB_START, + .segv_handler = STUB_CODE + + (unsigned long) stub_segv_handler - + (unsigned long) __syscall_stub_start, + }; + struct iomem_region *iomem; + int ret;
Review
Quoted 10 lines
(quoted diff) +10
-0+ + init_data.stub_code_fd = phys_mapping(uml_to_phys(__syscall_stub_start), + &offset); + init_data.stub_code_offset = MMAP_OFFSET(offset); + + init_data.stub_data_fd = phys_mapping(uml_to_phys(stack), &offset); + init_data.stub_data_offset = MMAP_OFFSET(offset); + + /* Set CLOEXEC on all FDs and then unset on all memory related FDs */ + close_range(0, ~0U, CLOSE_RANGE_CLOEXEC);
I got a link error on Ubuntu 22.04.4 (glibc-2.35) when STATIC_LINK is enabled:
/usr/bin/ld:
/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/libc.a(close_range.o):
in function `__close_range':
(.text+0x0): multiple definition of `__close_range'; fs/file.o:/home/test/linux/fs/file.c:741: first defined here
We probably need something like this:
arch/um/Makefile +2 -1
@@ -61,7 +61,8 @@ KBUILD_CFLAGS += $(CFLAGS) $(CFLAGS-y) -D__arch_um__ \ $(ARCH_INCLUDE) $(MODE_INCLUDE) -Dvmap=kernel_vmap \ -Dlongjmp=kernel_longjmp -Dsetjmp=kernel_setjmp \ -Din6addr_loopback=kernel_in6addr_loopback \ - -Din6addr_any=kernel_in6addr_any -Dstrrchr=kernel_strrchr + -Din6addr_any=kernel_in6addr_any -Dstrrchr=kernel_strrchr \ + -D__close_range=kernel__close_range KBUILD_RUSTFLAGS += -Crelocation-model=pie
Regards,
Tiwei
PATCH v8 3/7 um: Fix stub_start address calculation
In reply to Benjamin BergFrom: Benjamin Berg <[email protected]>
The calculation was wrong as it only subtracted one and then rounded down for alignment. However, this is incorrect if host_task_size is not already aligned.
This probably worked fine because on 64 bit the host_task_size is bigger than returned by os_get_top_address.
--- arch/um/kernel/um_arch.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/um/kernel/um_arch.c +2 -1
@@ -328,7 +328,8 @@ int __init linux_main(int argc, char **argv)/* reserve a few pages for the stubs (taking care of data alignment) */ /* align the data portion */ BUILD_BUG_ON (!is_power_of_2 (STUB_DATA_PAGES )); - stub_start = (host_task_size -1 ) & ~(STUB_DATA_PAGES *PAGE_SIZE -1 ); + stub_start = (host_task_size -STUB_DATA_PAGES *PAGE_SIZE ) & + ~(STUB_DATA_PAGES *PAGE_SIZE -1 );/* another page for the code portion */ stub_start -=PAGE_SIZE ; host_task_size = stub_start;
Signature · 1 line
2.45.2
PATCH v8 4/7 um: Limit TASK_SIZE to the addressable range
In reply to Benjamin BergFrom: Benjamin Berg <[email protected]>
We may have a TASK_SIZE from the host that is bigger than UML is able to address with a three-level pagetable. Guard against that by clipping the maximum TASK_SIZE to the maximum addressable area.
---
v7: Fix integer overflow on 32 bit with 3-level page tables
--- arch/um/kernel/um_arch.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/um/kernel/um_arch.c +6 -1
@@ -334,11 +334,16 @@ int __init linux_main(int argc, char **argv) stub_start -=PAGE_SIZE ; host_task_size = stub_start; +/* Limit TASK_SIZE to what is addressable by the page table */ + task_size = host_task_size; +if (task_size > (unsigned long long )PTRS_PER_PGD *PGDIR_SIZE ) + task_size =PTRS_PER_PGD *PGDIR_SIZE ; +/* * TASK_SIZE needs to be PGDIR_SIZE aligned or else exit_mmap craps * out */ - task_size = host_task_size &PGDIR_MASK ; + task_size = task_size &PGDIR_MASK ;/* OS sanity checks that need to happen before the kernel runs */ os_early_checks ();
Signature · 1 line
2.45.2
PATCH v8 5/7 um: Discover host_task_size from envp
In reply to Benjamin BergFrom: Benjamin Berg <[email protected]>
When loading the UML binary, the host kernel will place the stack at the highest possible address. It will then map the program name and environment variables onto the start of the stack.
As such, an easy way to figure out the host_task_size is to use the highest pointer to an environment variable as a reference.
Ensure that this works by disabling address layout randomization and re-executing UML in case it was enabled.
This increases the available TASK_SIZE for 64 bit UML considerably.
---
v7: Also use the same logic on 32bit
--- arch/um/include/shared/as-layout.h | 2 +- arch/um/include/shared/os.h | 2 +- arch/um/kernel/um_arch.c | 4 +- arch/um/os-Linux/main.c | 9 +- arch/x86/um/os-Linux/task_size.c | 152 ++--------------------------- 5 files changed, 22 insertions(+), 147 deletions(-)
arch/um/include/shared/as-layout.h +1 -1
@@ -48,7 +48,7 @@ extern unsigned long brk_start;extern unsigned long host_task_size;extern unsigned long stub_start; -extern int linux_main (int argc,char **argv); +extern int linux_main (int argc,char **argv,char **envp);extern void uml_finishsetup (void );struct siginfo ;
arch/um/include/shared/os.h +1 -1
@@ -330,7 +330,7 @@ extern int __ignore_sigio_fd(int fd);extern int get_pty (void );/* sys-$ARCH/task_size.c */ -extern unsigned long os_get_top_address (void ); +extern unsigned long os_get_top_address (char **envp);long syscall (long number, ...);
arch/um/kernel/um_arch.c +2 -2
@@ -302,7 +302,7 @@ static void parse_cache_line(char *line) } } -int __init linux_main (int argc,char **argv) +int __init linux_main (int argc,char **argv,char **envp) {unsigned long avail, diff;unsigned long virtmem_size, max_physmem;
@@ -324,7 +324,7 @@ int __init linux_main(int argc, char **argv)if (have_console ==0 )add_arg (DEFAULT_COMMAND_LINE_CONSOLE ); - host_task_size =os_get_top_address (); + host_task_size =os_get_top_address (envp);/* reserve a few pages for the stubs (taking care of data alignment) */ /* align the data portion */ BUILD_BUG_ON (!is_power_of_2 (STUB_DATA_PAGES ));
arch/um/os-Linux/main.c +8 -1
@@ -11,6 +11,7 @@#include <signal.h> #include <string.h> #include <sys/resource.h> +#include <sys/personality.h> #include <as-layout.h> #include <init.h> #include <kern_util.h>
@@ -108,6 +109,12 @@ int __init main(int argc, char **argv, char **envp)char **new_argv;int ret, i, err; +/* Disable randomization and re-exec if it was changed successfully */ + ret =personality (PER_LINUX |ADDR_NO_RANDOMIZE ); +if (ret >=0 && (ret & (PER_LINUX |ADDR_NO_RANDOMIZE )) != + (PER_LINUX |ADDR_NO_RANDOMIZE )) +execve ("/proc/self/exe" , argv, envp); +set_stklim ();setup_env_path ();
@@ -140,7 +147,7 @@ int __init main(int argc, char **argv, char **envp)#endif change_sig (SIGPIPE ,0 ); - ret =linux_main (argc, argv); + ret =linux_main (argc, argv, envp);/* * Disable SIGPROF - I have no idea why libc doesn't do this or turn
arch/x86/um/os-Linux/task_size.c +10 -142
161 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [PATCH v8 5/7] um: Discover host_task_size from envp
On 2024/7/5 03:05, Benjamin Berg wrote:
[...]
arch/x86/um/os-Linux/task_size.c line 2
Quoted 12 lines
arch/x86/um/os-Linux/task_size.c +0
-5@@ -1,151 +1,19 @@// SPDX-License-Identifier: GPL-2.0 -#include <stdio.h> -#include <stdlib.h> -#include <signal.h> -#include <sys/mman.h> -#include <longjmp.h>
Tiny nit: we need to include os.h to avoid below warning:
arch/x86/um/os-Linux/task_size.c:3:15: warning: no previous prototype for ‘os_get_top_address’ [-Wmissing-prototypes]
3 | unsigned long os_get_top_address(char **envp)
| ^~~~~~~~~~~~~~~~~~
Regards,
Tiwei
Quoted 154 lines
arch/x86/um/os-Linux/task_size.c +10
-137-#ifdef __i386__ - -static jmp_buf buf; - -static void segfault (int sig) -{ -longjmp (buf,1 ); -} - -static int page_ok (unsigned long page) -{ -unsigned long *address = (unsigned long *) (page <<UM_KERN_PAGE_SHIFT ); -unsigned long n = ~0UL ; -void *mapped =NULL ; -int ok =0 ; - -/* -* First see if the page is readable. If it is, it may still -* be a VDSO, so we go on to see if it's writable. If not -* then try mapping memory there. If that fails, then we're -* still in the kernel area. As a sanity check, we'll fail if -* the mmap succeeds, but gives us an address different from -* what we wanted. -*/ -if (setjmp (buf) ==0 ) - n = *address; -else { - mapped =mmap (address,UM_KERN_PAGE_SIZE , -PROT_READ |PROT_WRITE , -MAP_FIXED |MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); -if (mapped ==MAP_FAILED ) -return 0 ; -if (mapped != address) - goto out; - } - -/* -* Now, is it writeable? If so, then we're in user address -* space. If not, then try mprotecting it and try the write -* again. -*/ -if (setjmp (buf) ==0 ) { - *address = n; - ok =1 ; - goto out; - }else if (mprotect (address,UM_KERN_PAGE_SIZE , -PROT_READ |PROT_WRITE ) !=0 ) - goto out; - -if (setjmp (buf) ==0 ) { - *address = n; - ok =1 ; - } - - out: -if (mapped !=NULL ) -munmap (mapped,UM_KERN_PAGE_SIZE ); -return ok; -} - -unsigned long os_get_top_address (void ) +unsigned long os_get_top_address (char **envp) { -struct sigaction sa, old; -unsigned long bottom =0 ; -/* -* A 32-bit UML on a 64-bit host gets confused about the VDSO at -* 0xffffe000. It is mapped, is readable, can be reprotected writeable -* and written. However, exec discovers later that it can't be -* unmapped. So, just set the highest address to be checked to just -* below it. This might waste some address space on 4G/4G 32-bit -* hosts, but shouldn't hurt otherwise. -*/ -unsigned long top =0xffffd000 >>UM_KERN_PAGE_SHIFT ; -unsigned long test, original; +unsigned long top_addr = (unsigned long ) &top_addr; +int i; -printf ("Locating the bottom of the address space ... " ); -fflush (stdout); - -/* -* We're going to be longjmping out of the signal handler, so -* SA_DEFER needs to be set. -*/ - sa.sa_handler = segfault; -sigemptyset (&sa.sa_mask); - sa.sa_flags =SA_NODEFER ; -if (sigaction (SIGSEGV , &sa, &old)) { -perror ("os_get_top_address" ); -exit (1 ); - } - -/* Manually scan the address space, bottom-up, until we find -* the first valid page (or run out of them). -*/ -for (bottom =0 ; bottom < top; bottom++) { -if (page_ok (bottom)) -break ; - } - -/* If we've got this far, we ran out of pages. */ -if (bottom == top) { -fprintf (stderr,"Unable to determine bottom of address " -"space.\n" ); -exit (1 ); - } - -printf ("0x%lx\n" , bottom <<UM_KERN_PAGE_SHIFT ); -printf ("Locating the top of the address space ... " ); -fflush (stdout); - - original = bottom; - -/* This could happen with a 4G/4G split */ -if (page_ok (top)) - goto out; - -do { - test = bottom + (top - bottom) /2 ; -if (page_ok (test)) - bottom = test; -else - top = test; - }while (top - bottom >1 ); - -out: -/* Restore the old SIGSEGV handling */ -if (sigaction (SIGSEGV , &old,NULL )) { -perror ("os_get_top_address" ); -exit (1 ); +/* The earliest variable should be after the program name in ELF */ +for (i =0 ; envp[i]; i++) { +if ((unsigned long ) envp[i] > top_addr) + top_addr = (unsigned long ) envp[i]; } - top <<=UM_KERN_PAGE_SHIFT ; -printf ("0x%lx\n" , top); -return top; -} - -#else + top_addr &= ~(UM_KERN_PAGE_SIZE -1 ); + top_addr +=UM_KERN_PAGE_SIZE ; -unsigned long os_get_top_address (void ) -{ -/* The old value of CONFIG_TOP_ADDR */ -return 0x7fc0002000 ; +return top_addr; } -#endif
PATCH v8 6/7 um: clear all memory in new userspace processes
In reply to Benjamin BergFrom: Benjamin Berg <[email protected]>
With the change to use execve() we can now safely clear the memory up to STUB_START as rseq will not be trying to use memory in that region. Also, on 64 bit the previous changes should mean that there is no usable memory range above the stub.
Make the change and remove the comment as it is not needed anymore.
--- arch/um/kernel/skas/mmu.c | 25 ++----------------------- 1 file changed, 2 insertions(+), 23 deletions(-)
arch/um/kernel/skas/mmu.c +2 -23
@@ -40,29 +40,8 @@ int init_new_context(struct task_struct *task, struct mm_struct *mm) goto out_free; } -/* -* Ensure the new MM is clean and nothing unwanted is mapped. -* -* TODO: We should clear the memory up to STUB_START to ensure there is -* nothing mapped there, i.e. we (currently) have: -* -* |- user memory -|- unused -|- stub -|- unused -| -* ^ TASK_SIZE ^ STUB_START -* -* Meaning we have two unused areas where we may still have valid -* mappings from our internal clone(). That isn't really a problem as -* userspace is not going to access them, but it is definitely not -* correct. -* -* However, we are "lucky" and if rseq is configured, then on 32 bit -* it will fall into the first empty range while on 64 bit it is going -* to use an anonymous mapping in the second range. As such, things -* continue to work for now as long as we don't start unmapping these -* areas. -* -* Change this to STUB_START once we have a clean userspace. -*/ -unmap (new_id,0 ,TASK_SIZE ); +/* Ensure the new MM is clean and nothing unwanted is mapped */ +unmap (new_id,0 ,STUB_START );return 0 ;
Signature · 1 line
2.45.2
PATCH v8 7/7 um: Add 4 level page table support
In reply to Benjamin BergFrom: Benjamin Berg <[email protected]>
The larger memory space is useful to support more applications inside UML. One example for this is ASAN instrumentation of userspace applications which requires addresses that would otherwise not be available.
---
v7:
- Reword options and fix documentation of x86-64 default
v2:
- Do not hide option behind the EXPERT flag
- Fix typo in new "Two-level pagetables" option
--- arch/um/Kconfig | 1 + arch/um/include/asm/page.h | 14 +++- arch/um/include/asm/pgalloc.h | 11 ++- arch/um/include/asm/pgtable-4level.h | 119 +++++++++++++++++++++++++++ arch/um/include/asm/pgtable.h | 6 +- arch/um/kernel/mem.c | 17 +++- arch/x86/um/Kconfig | 38 ++++++--- 7 files changed, 189 insertions(+), 17 deletions(-) create mode 100644 arch/um/include/asm/pgtable-4level.h
arch/um/Kconfig +1 -0
@@ -210,6 +210,7 @@ config MMAPPER config PGTABLE_LEVELS int + default 4 if 4_LEVEL_PGTABLES default 3 if 3_LEVEL_PGTABLES default 2
arch/um/include/asm/page.h +11 -3
@@ -57,14 +57,22 @@ typedef unsigned long long phys_t;typedef struct {unsigned long pte; }pte_t ;typedef struct {unsigned long pgd; }pgd_t ; -#ifdef CONFIG_3_LEVEL_PGTABLES +#if CONFIG_PGTABLE_LEVELS >2 +typedef struct {unsigned long pmd; }pmd_t ;#define pmd_val (x) ((x).pmd)#define __pmd (x) ((pmd_t) { (x) } ) -#endif -#define pte_val (x) ((x).pte) +#if CONFIG_PGTABLE_LEVELS >3 +typedef struct {unsigned long pud; }pud_t ; +#define pud_val (x) ((x).pud) +#define __pud (x) ((pud_t) { (x) } ) + +#endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#endif /* CONFIG_PGTABLE_LEVELS > 2 */ + +#define pte_val (x) ((x).pte)#define pte_get_bits (p, bits) ((p).pte & (bits))#define pte_set_bits (p, bits) ((p).pte |= (bits))
arch/um/include/asm/pgalloc.h +10 -1
@@ -31,7 +31,7 @@ do { \tlb_remove_page_ptdesc ((tlb), (page_ptdesc (pte))); \ } while (0) -#ifdef CONFIG_3_LEVEL_PGTABLES +#if CONFIG_PGTABLE_LEVELS >2 #define __pmd_free_tlb (tlb, pmd, address) \ do { \
@@ -39,6 +39,15 @@ do { \tlb_remove_page_ptdesc ((tlb),virt_to_ptdesc (pmd)); \ } while (0) +#if CONFIG_PGTABLE_LEVELS >3 + +#define __pud_free_tlb (tlb, pud, address) \ +do { \ + pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ +} while (0) + +#endif #endif #endif
arch/um/include/asm/pgtable-4level.h new file +119 -0
119 lines not on this page · raw message
arch/um/include/asm/pgtable.h +4 -2
@@ -24,9 +24,11 @@/* We borrow bit 10 to store the exclusive marker in swap PTEs. */ #define _PAGE_SWP_EXCLUSIVE 0x400 -#ifdef CONFIG_3_LEVEL_PGTABLES +#if CONFIG_PGTABLE_LEVELS ==4 +#include <asm/pgtable-4level.h> +#elif CONFIG_PGTABLE_LEVELS ==3 #include <asm/pgtable-3level.h> -#else +#elif CONFIG_PGTABLE_LEVELS ==2 #include <asm/pgtable-2level.h> #endif
arch/um/kernel/mem.c +16 -1
@@ -98,7 +98,7 @@ static void __init one_page_table_init(pmd_t *pmd)static void __init one_md_table_init (pud_t *pud) { -#ifdef CONFIG_3_LEVEL_PGTABLES +#if CONFIG_PGTABLE_LEVELS >2 pmd_t *pmd_table = (pmd_t *)memblock_alloc_low (PAGE_SIZE ,PAGE_SIZE );if (!pmd_table) panic("%s: Failed to allocate %lu bytes align=%lx\n",
@@ -109,6 +109,19 @@ static void __init one_md_table_init(pud_t *pud)#endif } +static void __init one_ud_table_init (p4d_t *p4d) +{ +#if CONFIG_PGTABLE_LEVELS >3 +pud_t *pud_table = (pud_t *)memblock_alloc_low (PAGE_SIZE ,PAGE_SIZE ); +if (!pud_table) +panic ("%s: Failed to allocate %lu bytes align=%lx\n" , + __func__,PAGE_SIZE ,PAGE_SIZE ); + +set_p4d (p4d,__p4d (_KERNPG_TABLE + (unsigned long )__pa (pud_table))); +BUG_ON (pud_table !=pud_offset (p4d,0 )); +#endif +} +static void __init fixrange_init (unsigned long start,unsigned long end,pgd_t *pgd_base) {
@@ -126,6 +139,8 @@ static void __init fixrange_init(unsigned long start, unsigned long end,for ( ; (i <PTRS_PER_PGD ) && (vaddr < end); pgd++, i++) { p4d =p4d_offset (pgd, vaddr); +if (p4d_none (*p4d)) +one_ud_table_init (p4d); pud =pud_offset (p4d, vaddr);if (pud_none (*pud))one_md_table_init (pud);
arch/x86/um/Kconfig +28 -10
@@ -28,16 +28,34 @@ config X86_64 def_bool 64BIT select MODULES_USE_ELF_RELA -config 3_LEVEL_PGTABLES - bool "Three-level pagetables" if !64BIT - default 64BIT - help - Three-level pagetables will let UML have more than 4G of physical - memory. All the memory that can't be mapped directly will be treated - as high memory. - - However, this it experimental on 32-bit architectures, so if unsure say - N (on x86-64 it's automatically enabled, instead, as it's safe there). +choice + prompt "Pagetable levels" + default 2_LEVEL_PGTABLES if !64BIT + default 4_LEVEL_PGTABLES if 64BIT + + config 2_LEVEL_PGTABLES + bool "Two-level pagetables" if !64BIT + depends on !64BIT + help + Two-level page table for 32-bit architectures. + + config 3_LEVEL_PGTABLES + bool "Three-level pagetables" if 64BIT || (!64BIT && EXPERT) + help + Three-level pagetables will let UML have more than 4G of + physical memory. All the memory that can't be mapped + directly will be treated as high memory. + + However, this it experimental on 32-bit architectures, so if + unsure say N + + config 4_LEVEL_PGTABLES + bool "Four-level pagetables" if 64BIT + depends on 64BIT + help + Four-level pagetables, results in a bigger address space + which can be useful for some applications (e.g. ASAN). +endchoice config ARCH_HAS_SC_SIGNALS def_bool !64BIT
Signature · 1 line
2.45.2