Increased address space for 64 bit
Subsystems (2)
v4 cover Increased address space for 64 bit
From: Benjamin Berg <[email protected]>
Sorry for the noise, but I noticed that v3 has an issue on some host kernels, so I changed the waitpid back to how it was before CLONE_VFORK.
This patchset fixes a few bugs, adds a new method of discovering the host task size and finally adds four level page table support. All of this means the userspace TASK_SIZE is much larger and in turns permits userspace applications that need a lot of virtual addresses to work fine.
One such application is ASAN which uses a fixed address in memory that would otherwise not be addressable.
v2:
* Improved double clone logic using CLONE_VFORK
* Kconfig fixes pointed out by Tiwei Bie
v3:
* Undo incorrect change in child wait loop
v4:
* Do not use WNOHANG in wait for CLONE_VFORK
Benjamin Berg (5):
um: Fix stub_start address calculation
um: Limit TASK_SIZE to the addressable range
um: Do a double clone to disable rseq
um: Discover host_task_size from envp
um: Add 4 level page table support
arch/um/Kconfig | 1 + arch/um/include/asm/page.h | 14 +++- arch/um/include/asm/pgalloc.h | 11 ++- arch/um/include/asm/pgtable-4level.h | 119 +++++++++++++++++++++++++++ arch/um/include/asm/pgtable.h | 6 +- arch/um/include/shared/as-layout.h | 2 +- arch/um/include/shared/os.h | 2 +- arch/um/kernel/mem.c | 17 +++- arch/um/kernel/um_arch.c | 14 +++- arch/um/os-Linux/main.c | 9 +- arch/um/os-Linux/skas/process.c | 53 +++++++++++- arch/x86/um/Kconfig | 38 ++++++--- arch/x86/um/os-Linux/task_size.c | 19 ++++- 13 files changed, 273 insertions(+), 32 deletions(-) create mode 100644 arch/um/include/asm/pgtable-4level.h
Signature · 1 line
2.45.1
PATCH v4 1/5 um: Fix stub_start address calculation
From: Benjamin Berg <[email protected]>
The calculation was wrong as it only subtracted one and then rounded down for alignment. However, this is incorrect if host_task_size is not already aligned.
This probably worked fine because on 64 bit the host_task_size is bigger than returned by os_get_top_address.
--- arch/um/kernel/um_arch.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/um/kernel/um_arch.c +2 -1
@@ -331,7 +331,8 @@ int __init linux_main(int argc, char **argv)/* reserve a few pages for the stubs (taking care of data alignment) */ /* align the data portion */ BUILD_BUG_ON (!is_power_of_2 (STUB_DATA_PAGES )); - stub_start = (host_task_size -1 ) & ~(STUB_DATA_PAGES *PAGE_SIZE -1 ); + stub_start = (host_task_size -STUB_DATA_PAGES *PAGE_SIZE ) & + ~(STUB_DATA_PAGES *PAGE_SIZE -1 );/* another page for the code portion */ stub_start -=PAGE_SIZE ; host_task_size = stub_start;
Signature · 1 line
2.45.1
PATCH v4 2/5 um: Limit TASK_SIZE to the addressable range
In reply to [email protected]From: Benjamin Berg <[email protected]>
We may have a TASK_SIZE from the host that is bigger than UML is able to address with a three-level pagetable. Guard against that by clipping the maximum TASK_SIZE to the maximum addressable area.
--- arch/um/kernel/um_arch.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/um/kernel/um_arch.c +6 -1
@@ -337,11 +337,16 @@ int __init linux_main(int argc, char **argv) stub_start -=PAGE_SIZE ; host_task_size = stub_start; +/* Limit TASK_SIZE to what is addressable by the page table */ + task_size = host_task_size; +if (task_size >PTRS_PER_PGD *PGDIR_SIZE ) + task_size =PTRS_PER_PGD *PGDIR_SIZE ; +/* * TASK_SIZE needs to be PGDIR_SIZE aligned or else exit_mmap craps * out */ - task_size = host_task_size &PGDIR_MASK ; + task_size = task_size &PGDIR_MASK ;/* OS sanity checks that need to happen before the kernel runs */ os_early_checks ();
Signature · 1 line
2.45.1
PATCH v4 3/5 um: Do a double clone to disable rseq
In reply to [email protected]From: Benjamin Berg <[email protected]>
Newer glibc versions are enabling rseq support by default. This remains enabled in the cloned child process, potentially causing the host kernel to write/read memory in the child.
It appears that this was purely not an issue because the used memory area happened to be above TASK_SIZE and remains mapped.
Note that a better approach would be to exec a small static binary that does not link with other libraries. Using a memfd and execveat the binary could be embedded into UML itself and it would result in an entirely clean execution environment for userspace.
---
v2: Improved clone logic using CLONE_VFORK
v3: Undo incorrect change in child wait loop
v4: Do not use WNOHANG in wait for CLONE_VFORK, this seems to fail on
older host kernel versions.
--- arch/um/os-Linux/skas/process.c | 53 ++++++++++++++++++++++++++++++--- 1 file changed, 49 insertions(+), 4 deletions(-)
arch/um/os-Linux/skas/process.c +49 -4
@@ -255,6 +255,32 @@ static int userspace_tramp(void *stack)int userspace_pid[NR_CPUS ];int kill_userspace_mm[NR_CPUS ]; +struct tramp_data { +int pid; +void *clone_sp; +void *stack; +}; + +static int userspace_tramp_clone_vm (void *data) +{ +struct tramp_data *tramp_data = data; + +/* +* At this point we are still in the same VM as the parent, but rseq +* has been disabled for this process. +* Continue with the clone into the new userspace process, the kernel +* continues as soon as this process quits (CLONE_VFORK). +*/ + + tramp_data->pid =clone (userspace_tramp, tramp_data->clone_sp, +CLONE_PARENT |CLONE_FILES |SIGCHLD , + tramp_data->stack); +if (tramp_data->pid <0 ) + tramp_data->pid = -errno; + +exit (0 ); +} +/** * start_userspace() - prepare a new userspace process * @stub_stack: pointer to the stub stack.
@@ -268,9 +294,10 @@ int kill_userspace_mm[NR_CPUS];*/ int start_userspace (unsigned long stub_stack) { +struct tramp_data tramp_data;void *stack;unsigned long sp; -int pid, status, n, flags, err; +int pid, status, n, err;/* setup a temporary stack page */ stack = mmap(NULL, UM_KERN_PAGE_SIZE,
@@ -286,10 +313,13 @@ int start_userspace(unsigned long stub_stack)/* set stack pointer to the end of the stack page, so it can grow downwards */ sp = (unsigned long )stack +UM_KERN_PAGE_SIZE ; - flags =CLONE_FILES |SIGCHLD ; + tramp_data.stack = (void *) stub_stack; + tramp_data.clone_sp = (void *) sp; + tramp_data.pid = -EINVAL ; -/* clone into new userspace process */ - pid =clone (userspace_tramp, (void *) sp, flags, (void *) stub_stack); +/* first stage CLONE_VM clone using VFORK and no signal notification */ + pid =clone (userspace_tramp_clone_vm, (void *) sp, +CLONE_VM |CLONE_FILES |CLONE_VFORK , &tramp_data);if (pid <0 ) { err = -errno; printk(UM_KERN_ERR "%s : clone failed, errno = %d\n",
@@ -297,6 +327,21 @@ int start_userspace(unsigned long stub_stack)return err; } +CATCH_EINTR (n =waitpid (pid, &status,WUNTRACED | __WALL)); +if (n <0 || !WIFEXITED (status) ||WEXITSTATUS (status)) { + err = -errno; +printk (UM_KERN_ERR "%s : wait failed, errno = %d, status = %d\n" , + __func__, n <0 ? errno :0 , status); + goto out_kill; + } + + pid = tramp_data.pid; +if (pid <0 ) { +printk (UM_KERN_ERR "%s : second clone failed, errno = %d\n" , + __func__, -pid); +return pid; + } + do { CATCH_EINTR(n = waitpid(pid, &status, WUNTRACED | __WALL)); if (n < 0) {
Signature · 1 line
2.45.1
PATCH v4 4/5 um: Discover host_task_size from envp
In reply to [email protected]From: Benjamin Berg <[email protected]>
When loading the UML binary, the host kernel will place the stack at the highest possible address. It will then map the program name and environment variables onto the start of the stack.
As such, an easy way to figure out the host_task_size is to use the highest pointer to an environment variable as a reference.
Ensure that this works by disabling address layout randomization and re-executing UML in case it was enabled.
This increases the available TASK_SIZE for 64 bit UML considerably.
--- arch/um/include/shared/as-layout.h | 2 +- arch/um/include/shared/os.h | 2 +- arch/um/kernel/um_arch.c | 4 ++-- arch/um/os-Linux/main.c | 9 ++++++++- arch/x86/um/os-Linux/task_size.c | 19 +++++++++++++++---- 5 files changed, 27 insertions(+), 9 deletions(-)
arch/um/include/shared/as-layout.h +1 -1
@@ -48,7 +48,7 @@ extern unsigned long brk_start;extern unsigned long host_task_size;extern unsigned long stub_start; -extern int linux_main (int argc,char **argv); +extern int linux_main (int argc,char **argv,char **envp);extern void uml_finishsetup (void );struct siginfo ;
arch/um/include/shared/os.h +1 -1
@@ -327,7 +327,7 @@ extern int __ignore_sigio_fd(int fd);extern int get_pty (void );/* sys-$ARCH/task_size.c */ -extern unsigned long os_get_top_address (void ); +extern unsigned long os_get_top_address (char **envp);long syscall (long number, ...);
arch/um/kernel/um_arch.c +2 -2
@@ -305,7 +305,7 @@ static void parse_cache_line(char *line) } } -int __init linux_main (int argc,char **argv) +int __init linux_main (int argc,char **argv,char **envp) {unsigned long avail, diff;unsigned long virtmem_size, max_physmem;
@@ -327,7 +327,7 @@ int __init linux_main(int argc, char **argv)if (have_console ==0 )add_arg (DEFAULT_COMMAND_LINE_CONSOLE ); - host_task_size =os_get_top_address (); + host_task_size =os_get_top_address (envp);/* reserve a few pages for the stubs (taking care of data alignment) */ /* align the data portion */ BUILD_BUG_ON (!is_power_of_2 (STUB_DATA_PAGES ));
arch/um/os-Linux/main.c +8 -1
@@ -11,6 +11,7 @@#include <signal.h> #include <string.h> #include <sys/resource.h> +#include <sys/personality.h> #include <as-layout.h> #include <init.h> #include <kern_util.h>
@@ -108,6 +109,12 @@ int __init main(int argc, char **argv, char **envp)char **new_argv;int ret, i, err; +/* Disable randomization and re-exec if it was changed successfully */ + ret =personality (PER_LINUX |ADDR_NO_RANDOMIZE ); +if (ret >=0 && (ret & (PER_LINUX |ADDR_NO_RANDOMIZE )) != + (PER_LINUX |ADDR_NO_RANDOMIZE )) +execve ("/proc/self/exe" , argv, envp); +set_stklim ();setup_env_path ();
@@ -140,7 +147,7 @@ int __init main(int argc, char **argv, char **envp)#endif change_sig (SIGPIPE ,0 ); - ret =linux_main (argc, argv); + ret =linux_main (argc, argv, envp);/* * Disable SIGPROF - I have no idea why libc doesn't do this or turn
arch/x86/um/os-Linux/task_size.c +15 -4
@@ -65,7 +65,7 @@ static int page_ok(unsigned long page)return ok; } -unsigned long os_get_top_address (void ) +unsigned long os_get_top_address (char **envp) {struct sigaction sa, old;unsigned long bottom =0 ;
@@ -142,10 +142,21 @@ unsigned long os_get_top_address(void)#else -unsigned long os_get_top_address (void ) +unsigned long os_get_top_address (char **envp) { -/* The old value of CONFIG_TOP_ADDR */ -return 0x7fc0002000 ; +unsigned long top_addr = (unsigned long ) &top_addr; +int i; + +/* The earliest variable should be after the program name in ELF */ +for (i =0 ; envp[i]; i++) { +if ((unsigned long ) envp[i] > top_addr) + top_addr = (unsigned long ) envp[i]; + } + + top_addr &= ~(UM_KERN_PAGE_SIZE -1 ); + top_addr +=UM_KERN_PAGE_SIZE ; + +return top_addr; }#endif
Signature · 1 line
2.45.1
PATCH v4 5/5 um: Add 4 level page table support
In reply to [email protected]From: Benjamin Berg <[email protected]>
The larger memory space is useful to support more applications inside UML. One example for this is ASAN instrumentation of userspace applications which requires addresses that would otherwise not be available.
---
v2:
- Do not hide option behind the EXPERT flag
- Fix typo in new "Two-level pagetables" option
--- arch/um/Kconfig | 1 + arch/um/include/asm/page.h | 14 +++- arch/um/include/asm/pgalloc.h | 11 ++- arch/um/include/asm/pgtable-4level.h | 119 +++++++++++++++++++++++++++ arch/um/include/asm/pgtable.h | 6 +- arch/um/kernel/mem.c | 17 +++- arch/x86/um/Kconfig | 38 ++++++--- 7 files changed, 189 insertions(+), 17 deletions(-) create mode 100644 arch/um/include/asm/pgtable-4level.h
arch/um/Kconfig +1 -0
@@ -208,6 +208,7 @@ config MMAPPER config PGTABLE_LEVELS int + default 4 if 4_LEVEL_PGTABLES default 3 if 3_LEVEL_PGTABLES default 2
arch/um/include/asm/page.h +11 -3
@@ -57,14 +57,22 @@ typedef unsigned long long phys_t;typedef struct {unsigned long pte; }pte_t ;typedef struct {unsigned long pgd; }pgd_t ; -#ifdef CONFIG_3_LEVEL_PGTABLES +#if CONFIG_PGTABLE_LEVELS >2 +typedef struct {unsigned long pmd; }pmd_t ;#define pmd_val (x) ((x).pmd)#define __pmd (x) ((pmd_t) { (x) } ) -#endif -#define pte_val (x) ((x).pte) +#if CONFIG_PGTABLE_LEVELS >3 +typedef struct {unsigned long pud; }pud_t ; +#define pud_val (x) ((x).pud) +#define __pud (x) ((pud_t) { (x) } ) + +#endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#endif /* CONFIG_PGTABLE_LEVELS > 2 */ + +#define pte_val (x) ((x).pte)#define pte_get_bits (p, bits) ((p).pte & (bits))#define pte_set_bits (p, bits) ((p).pte |= (bits))
arch/um/include/asm/pgalloc.h +10 -1
@@ -31,7 +31,7 @@ do { \tlb_remove_page_ptdesc ((tlb), (page_ptdesc (pte))); \ } while (0) -#ifdef CONFIG_3_LEVEL_PGTABLES +#if CONFIG_PGTABLE_LEVELS >2 #define __pmd_free_tlb (tlb, pmd, address) \ do { \
@@ -39,6 +39,15 @@ do { \tlb_remove_page_ptdesc ((tlb),virt_to_ptdesc (pmd)); \ } while (0) +#if CONFIG_PGTABLE_LEVELS >3 + +#define __pud_free_tlb (tlb, pud, address) \ +do { \ + pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ +} while (0) + +#endif #endif #endif
arch/um/include/asm/pgtable-4level.h new file +119 -0
119 lines not on this page · raw message
arch/um/include/asm/pgtable.h +4 -2
@@ -24,9 +24,11 @@/* We borrow bit 10 to store the exclusive marker in swap PTEs. */ #define _PAGE_SWP_EXCLUSIVE 0x400 -#ifdef CONFIG_3_LEVEL_PGTABLES +#if CONFIG_PGTABLE_LEVELS ==4 +#include <asm/pgtable-4level.h> +#elif CONFIG_PGTABLE_LEVELS ==3 #include <asm/pgtable-3level.h> -#else +#elif CONFIG_PGTABLE_LEVELS ==2 #include <asm/pgtable-2level.h> #endif
arch/um/kernel/mem.c +16 -1
@@ -99,7 +99,7 @@ static void __init one_page_table_init(pmd_t *pmd)static void __init one_md_table_init (pud_t *pud) { -#ifdef CONFIG_3_LEVEL_PGTABLES +#if CONFIG_PGTABLE_LEVELS >2 pmd_t *pmd_table = (pmd_t *)memblock_alloc_low (PAGE_SIZE ,PAGE_SIZE );if (!pmd_table) panic("%s: Failed to allocate %lu bytes align=%lx\n",
@@ -110,6 +110,19 @@ static void __init one_md_table_init(pud_t *pud)#endif } +static void __init one_ud_table_init (p4d_t *p4d) +{ +#if CONFIG_PGTABLE_LEVELS >3 +pud_t *pud_table = (pud_t *)memblock_alloc_low (PAGE_SIZE ,PAGE_SIZE ); +if (!pud_table) +panic ("%s: Failed to allocate %lu bytes align=%lx\n" , + __func__,PAGE_SIZE ,PAGE_SIZE ); + +set_p4d (p4d,__p4d (_KERNPG_TABLE + (unsigned long )__pa (pud_table))); +BUG_ON (pud_table !=pud_offset (p4d,0 )); +#endif +} +static void __init fixrange_init (unsigned long start,unsigned long end,pgd_t *pgd_base) {
@@ -127,6 +140,8 @@ static void __init fixrange_init(unsigned long start, unsigned long end,for ( ; (i <PTRS_PER_PGD ) && (vaddr < end); pgd++, i++) { p4d =p4d_offset (pgd, vaddr); +if (p4d_none (*p4d)) +one_ud_table_init (p4d); pud =pud_offset (p4d, vaddr);if (pud_none (*pud))one_md_table_init (pud);
arch/x86/um/Kconfig +28 -10
@@ -28,16 +28,34 @@ config X86_64 def_bool 64BIT select MODULES_USE_ELF_RELA -config 3_LEVEL_PGTABLES - bool "Three-level pagetables" if !64BIT - default 64BIT - help - Three-level pagetables will let UML have more than 4G of physical - memory. All the memory that can't be mapped directly will be treated - as high memory. - - However, this it experimental on 32-bit architectures, so if unsure say - N (on x86-64 it's automatically enabled, instead, as it's safe there). +choice + prompt "Pagetable levels" + default 2_LEVEL_PGTABLES if !64BIT + default 4_LEVEL_PGTABLES if 64BIT + + config 2_LEVEL_PGTABLES + bool "Two-level pagetables" if !64BIT + depends on !64BIT + help + Two-level page table for 32-bit architectures. + + config 3_LEVEL_PGTABLES + bool "Three-level pagetables" if 64BIT + help + Three-level pagetables will let UML have more than 4G of physical + memory. All the memory that can't be mapped directly will be treated + as high memory. + + However, this it experimental on 32-bit architectures, so if unsure say + N (on x86-64 it's automatically enabled, instead, as it's safe there). + + config 4_LEVEL_PGTABLES + bool "Four-level pagetables" if 64BIT + depends on 64BIT + help + Four-level pagetables, gives a bigger address space which can be + useful for some applications (e.g. ASAN). +endchoice config ARCH_HAS_SC_SIGNALS def_bool !64BIT
Signature · 1 line
2.45.1