Cover a guard gap corner case
Subsystems (26)
v3 cover Cover a guard gap corner case
Hi,
For v3, the change is in the struct vm_unmapped_area_info zeroing patches. Per discussion[0], they are switched to a method of intializing the struct at the callers that also doesn't leave useless statements as cleanup, but is a bit easier to manually inspect for bugs. The arch's that acked the old versions are left separate. What's left after that happens in a treewide change.
It seems like a more straightforward change now, but I would still appreciate if anyone can double check the treewide change.
Also, rebase to v6.8.
[0] https://lore.kernel.org/lkml/[email protected]/
v2:
https://lore.kernel.org/lkml/[email protected]/
v1:
https://lore.kernel.org/lkml/[email protected]/
=======
In working on x86’s shadow stack feature, I came across some limitations around the kernel’s handling of guard gaps. AFAICT these limitations are not too important for the traditional stack usage of guard gaps, but have bigger impact on shadow stack’s usage. And now in addition to x86, we have two other architectures implementing shadow stack like features that plan to use guard gaps. I wanted to see about addressing them, but I have not worked on mmap() placement related code before, so would greatly appreciate if people could take a look and point me in the right direction.
The nature of the limitations of concern is as follows. In order to ensure guard gaps between mappings, mmap() would need to consider two things:
1. That the new mapping isn’t placed in an any existing mapping’s guard gap.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps
Currently mmap never considers (2), and (1) is not considered in some situations.
When not passing an address hint, or passing one without MAP_FIXED_NOREPLACE, (1) is enforced. With MAP_FIXED_NOREPLACE, (1) is not enforced. With MAP_FIXED, (1) is not considered, but this seems to be expected since MAP_FIXED can already clobber existing mappings. For MAP_FIXED_NOREPLACE I would have guessed it should respect the guard gaps of existing mappings, but it is probably a little ambiguous.
In this series I just tried to add enforcement of (2) for the normal (no address hint) case and only for the newer shadow stack memory (not stacks). The reason is that with the no-address-hint situation, landing next to a guard gap could come up naturally and so be more influencable by attackers such that two shadow stacks could be adjacent without a guard gap. Where as the address-hint scenarios would require more control - being able to call mmap() with specific arguments. As for why not just fix the other corner cases anyway, I thought it might have some greater possibility of affecting existing apps.
Thanks,
Rick
Rick Edgecombe (12):
mm: Switch mm->get_unmapped_area() to a flag
mm: Introduce arch_get_unmapped_area_vmflags()
mm: Use get_unmapped_area_vmflags()
thp: Add thp_get_unmapped_area_vmflags()
csky: Use initializer for struct vm_unmapped_area_info
parisc: Use initializer for struct vm_unmapped_area_info
powerpc: Use initializer for struct vm_unmapped_area_info
treewide: Use initializer for struct vm_unmapped_area_info
mm: Take placement mappings gap into account x86/mm: Implement HAVE_ARCH_UNMAPPED_AREA_VMFLAGS x86/mm: Care about shadow stack guard gap during placement selftests/x86: Add placement guard gap test for shstk
arch/alpha/kernel/osf_sys.c | 5 +- arch/arc/mm/mmap.c | 4 +- arch/arm/mm/mmap.c | 5 +- arch/csky/abiv1/mmap.c | 12 +- arch/loongarch/mm/mmap.c | 3 +- arch/mips/mm/mmap.c | 3 +- arch/parisc/kernel/sys_parisc.c | 6 +- arch/powerpc/mm/book3s64/slice.c | 23 ++-- arch/s390/mm/hugetlbpage.c | 9 +- arch/s390/mm/mmap.c | 15 +-- arch/sh/mm/mmap.c | 5 +- arch/sparc/kernel/sys_sparc_32.c | 3 +- arch/sparc/kernel/sys_sparc_64.c | 20 ++-- arch/sparc/mm/hugetlbpage.c | 9 +- arch/x86/include/asm/pgtable_64.h | 1 + arch/x86/kernel/cpu/sgx/driver.c | 2 +- arch/x86/kernel/sys_x86_64.c | 42 +++++-- arch/x86/mm/hugetlbpage.c | 9 +- arch/x86/mm/mmap.c | 4 +- drivers/char/mem.c | 2 +- drivers/dax/device.c | 6 +- fs/hugetlbfs/inode.c | 11 +- fs/proc/inode.c | 15 +-- fs/ramfs/file-mmu.c | 2 +- include/linux/huge_mm.h | 11 ++ include/linux/mm.h | 12 +- include/linux/mm_types.h | 6 +- include/linux/sched/coredump.h | 5 +- include/linux/sched/mm.h | 22 ++++ io_uring/io_uring.c | 2 +- mm/debug.c | 6 - mm/huge_memory.c | 26 +++-- mm/mmap.c | 106 +++++++++++++----- mm/shmem.c | 11 +- mm/util.c | 6 +- .../testing/selftests/x86/test_shadow_stack.c | 67 ++++++++++- 36 files changed, 319 insertions(+), 177 deletions(-)
Signature · 1 line
2.34.1
PATCH v3 1/12 mm: Switch mm->get_unmapped_area() to a flag
The mm_struct contains a function pointer *get_unmapped_area(), which is set to either arch_get_unmapped_area() or arch_get_unmapped_area_topdown() during the initialization of the mm.
Since the function pointer only ever points to two functions that are named the same across all arch's, a function pointer is not really required. In addition future changes will want to add versions of the functions that take additional arguments. So to save a pointers worth of bytes in mm_struct, and prevent adding additional function pointers to mm_struct in future changes, remove it and keep the information about which get_unmapped_area() to use in a flag.
Add the new flag to MMF_INIT_MASK so it doesn't get clobbered on fork by mmf_init_flags(). Most MM flags get clobbered on fork. In the pre-existing behavior mm->get_unmapped_area() would get copied to the new mm in dup_mm(), so not clobbering the flag preserves the existing behavior around inheriting the topdown-ness.
Introduce a helper, mm_get_unmapped_area(), to easily convert code that refers to the old function pointer to instead select and call either arch_get_unmapped_area() or arch_get_unmapped_area_topdown() based on the flag. Then drop the mm->get_unmapped_area() function pointer. Leave the get_unmapped_area() pointer in struct file_operations alone. The main purpose of this change is to reorganize in preparation for future changes, but it also converts the calls of mm->get_unmapped_area() from indirect branches into a direct ones.
The stress-ng bigheap benchmark calls realloc a lot, which calls through get_unmapped_area() in the kernel. On x86, the change yielded a ~1% improvement there on a retpoline config.
In testing a few x86 configs, removing the pointer unfortunately didn't result in any actual size reductions in the compiled layout of mm_struct. But depending on compiler or arch alignment requirements, the change could shrink the size of mm_struct.
---
v3:
- Fix comment that still referred to mm->get_unmapped_area()
- Resolve trivial rebase conflicts with "mm: thp_get_unmapped_area must honour topdown preference"
- Spelling fix in log
v2:
- Fix comment on MMF_TOPDOWN (Kirill, rppt)
- Move MMF_TOPDOWN to actually unused bit
- Add MMF_TOPDOWN to MMF_INIT_MASK so it doesn't get clobbered on fork, and result in the children using the search up path.
- New lower performance results after above bug fix
- Add Reviews and Acks
--- arch/s390/mm/hugetlbpage.c | 2 +- arch/s390/mm/mmap.c | 4 ++-- arch/sparc/kernel/sys_sparc_64.c | 15 ++++++--------- arch/sparc/mm/hugetlbpage.c | 2 +- arch/x86/kernel/cpu/sgx/driver.c | 2 +- arch/x86/mm/hugetlbpage.c | 2 +- arch/x86/mm/mmap.c | 4 ++-- drivers/char/mem.c | 2 +- drivers/dax/device.c | 6 +++--- fs/hugetlbfs/inode.c | 4 ++-- fs/proc/inode.c | 15 ++++++++------- fs/ramfs/file-mmu.c | 2 +- include/linux/mm_types.h | 6 +----- include/linux/sched/coredump.h | 5 ++++- include/linux/sched/mm.h | 5 +++++ io_uring/io_uring.c | 2 +- mm/debug.c | 6 ------ mm/huge_memory.c | 9 ++++----- mm/mmap.c | 21 ++++++++++++++++++--- mm/shmem.c | 11 +++++------ mm/util.c | 6 +++--- 21 files changed, 70 insertions(+), 61 deletions(-)
arch/s390/mm/hugetlbpage.c +1 -1
@@ -328,7 +328,7 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, goto check_asce_limit; } -if (mm->get_unmapped_area == arch_get_unmapped_area) +if (!test_bit (MMF_TOPDOWN , &mm->flags)) addr =hugetlb_get_unmapped_area_bottomup (file, addr, len, pgoff, flags);else
arch/s390/mm/mmap.c +2 -2
@@ -182,10 +182,10 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack)*/ if (mmap_is_legacy (rlim_stack)) { mm->mmap_base =mmap_base_legacy (random_factor); - mm->get_unmapped_area = arch_get_unmapped_area; +clear_bit (MMF_TOPDOWN , &mm->flags); }else { mm->mmap_base =mmap_base (random_factor, rlim_stack); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; +set_bit (MMF_TOPDOWN , &mm->flags); } }
arch/sparc/kernel/sys_sparc_64.c +6 -9
@@ -218,14 +218,10 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,unsigned long get_fb_unmapped_area (struct file *filp,unsigned long orig_addr,unsigned long len,unsigned long pgoff,unsigned long flags) {unsigned long align_goal, addr = -ENOMEM ; -unsigned long (*get_area)(struct file *,unsigned long , -unsigned long ,unsigned long ,unsigned long ); - - get_area = current->mm->get_unmapped_area;if (flags &MAP_FIXED ) {/* Ok, don't mess with it. */ -return get_area (NULL , orig_addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm,NULL , orig_addr, len, pgoff, flags); } flags &= ~MAP_SHARED ;
@@ -238,7 +234,8 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u align_goal = (64UL *1024 ); do { - addr = get_area(NULL, orig_addr, len + (align_goal - PAGE_SIZE), pgoff, flags); + addr = mm_get_unmapped_area(current->mm, NULL, orig_addr, + len + (align_goal - PAGE_SIZE), pgoff, flags); if (!(addr & ~PAGE_MASK)) { addr = (addr + (align_goal - 1UL)) & ~(align_goal - 1UL); break;
@@ -256,7 +253,7 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u* be obtained. */ if (addr & ~PAGE_MASK ) - addr =get_area (NULL , orig_addr, len, pgoff, flags); + addr =mm_get_unmapped_area (current->mm,NULL , orig_addr, len, pgoff, flags);return addr; }
@@ -292,7 +289,7 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) gap == RLIM_INFINITY || sysctl_legacy_va_layout) { mm->mmap_base = TASK_UNMAPPED_BASE + random_factor; - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } else {/* We know it's 32-bit */ unsigned long task_size =STACK_TOP32 ;
@@ -303,7 +300,7 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) gap = (task_size /6 *5 ); mm->mmap_base =PAGE_ALIGN (task_size - gap - random_factor); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; +set_bit (MMF_TOPDOWN , &mm->flags); } }
arch/sparc/mm/hugetlbpage.c +1 -1
@@ -123,7 +123,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, (!vma || addr + len <= vm_start_gap(vma)))return addr; } -if (mm->get_unmapped_area == arch_get_unmapped_area) +if (!test_bit (MMF_TOPDOWN , &mm->flags))return hugetlb_get_unmapped_area_bottomup (file, addr, len, pgoff, flags);else
arch/x86/kernel/cpu/sgx/driver.c +1 -1
@@ -113,7 +113,7 @@ static unsigned long sgx_get_unmapped_area(struct file *file,if (flags &MAP_FIXED )return addr; -return current->mm->get_unmapped_area (file, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, file, addr, len, pgoff, flags); }#ifdef CONFIG_COMPAT
arch/x86/mm/hugetlbpage.c +1 -1
@@ -141,7 +141,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, } get_unmapped_area: -if (mm->get_unmapped_area == arch_get_unmapped_area) +if (!test_bit (MMF_TOPDOWN , &mm->flags))return hugetlb_get_unmapped_area_bottomup (file, addr, len, pgoff, flags);else
arch/x86/mm/mmap.c +2 -2
@@ -129,9 +129,9 @@ static void arch_pick_mmap_base(unsigned long *base, unsigned long *legacy_base,void arch_pick_mmap_layout (struct mm_struct *mm,struct rlimit *rlim_stack) {if (mmap_is_legacy ()) - mm->get_unmapped_area = arch_get_unmapped_area; +clear_bit (MMF_TOPDOWN , &mm->flags);else - mm->get_unmapped_area = arch_get_unmapped_area_topdown; +set_bit (MMF_TOPDOWN , &mm->flags); arch_pick_mmap_base(&mm->mmap_base, &mm->mmap_legacy_base, arch_rnd(mmap64_rnd_bits), task_size_64bit(0),
drivers/char/mem.c +1 -1
@@ -544,7 +544,7 @@ static unsigned long get_unmapped_area_zero(struct file *file, }/* Otherwise flags & MAP_PRIVATE: with no shmem object beneath it */ -return current->mm->get_unmapped_area (file, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, file, addr, len, pgoff, flags);#else return -ENOSYS ;#endif
drivers/dax/device.c +3 -3
@@ -329,14 +329,14 @@ static unsigned long dax_get_unmapped_area(struct file *filp,if ((off + len_align) < off) goto out; - addr_align = current->mm->get_unmapped_area (filp, addr, len_align, - pgoff, flags); + addr_align =mm_get_unmapped_area (current->mm, filp, addr, len_align, + pgoff, flags);if (!IS_ERR_VALUE (addr_align)) { addr_align += (off - addr_align) & (align -1 );return addr_align; } out: -return current->mm->get_unmapped_area (filp, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, filp, addr, len, pgoff, flags); }static const struct address_space_operations dev_dax_aops = {
fs/hugetlbfs/inode.c +2 -2
@@ -249,11 +249,11 @@ generic_hugetlb_get_unmapped_area(struct file *file, unsigned long addr, }/* -* Use mm->get_unmapped_area value as a hint to use topdown routine. +* Use MMF_TOPDOWN flag as a hint to use topdown routine. * If architectures have special needs, they should define their own * version of hugetlb_get_unmapped_area. */ -if (mm->get_unmapped_area == arch_get_unmapped_area_topdown) +if (test_bit (MMF_TOPDOWN , &mm->flags))return hugetlb_get_unmapped_area_topdown (file, addr, len, pgoff, flags); return hugetlb_get_unmapped_area_bottomup(file, addr, len,
fs/proc/inode.c +8 -7
@@ -451,15 +451,16 @@ pde_get_unmapped_area(struct proc_dir_entry *pde, struct file *file, unsigned lo unsigned long len, unsigned long pgoff, unsigned long flags) { - typeof_member(struct proc_ops, proc_get_unmapped_area) get_area; - - get_area = pde->proc_ops->proc_get_unmapped_area; + if (pde->proc_ops->proc_get_unmapped_area) + return pde->proc_ops->proc_get_unmapped_area(file, orig_addr, + len, pgoff, + flags);#ifdef CONFIG_MMU -if (!get_area) - get_area = current->mm->get_unmapped_area; + else + return mm_get_unmapped_area(current->mm, file, orig_addr, + len, pgoff, flags);#endif -if (get_area) -return get_area (file, orig_addr, len, pgoff, flags); +return orig_addr; }
fs/ramfs/file-mmu.c +1 -1
@@ -35,7 +35,7 @@ static unsigned long ramfs_mmu_get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); } const struct file_operations ramfs_file_operations = {
include/linux/mm_types.h +1 -5
@@ -749,11 +749,7 @@ struct mm_struct { }____cacheline_aligned_in_smp ;struct maple_tree mm_mt; -#ifdef CONFIG_MMU -unsigned long (*get_unmapped_area) (struct file *filp, -unsigned long addr,unsigned long len, -unsigned long pgoff,unsigned long flags); -#endif +unsigned long mmap_base;/* base of mmap area */ unsigned long mmap_legacy_base;/* base of mmap area in bottom-up allocations */ #ifdef CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES
include/linux/sched/coredump.h +4 -1
@@ -92,9 +92,12 @@ static inline int get_dumpable(struct mm_struct *mm)#define MMF_VM_MERGE_ANY 30#define MMF_VM_MERGE_ANY_MASK (1 << MMF_VM_MERGE_ANY) +#define MMF_TOPDOWN 31/* mm searches top down by default */ +#define MMF_TOPDOWN_MASK (1 << MMF_TOPDOWN) +#define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ - MMF_VM_MERGE_ANY_MASK) + MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK)static inline unsigned long mmf_init_flags (unsigned long flags) {
include/linux/sched/mm.h +5 -0
@@ -8,6 +8,7 @@#include <linux/mm_types.h> #include <linux/gfp.h> #include <linux/sync_core.h> +#include <linux/sched/coredump.h> /* * Routines for handling mm_structs
@@ -186,6 +187,10 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +unsigned long mm_get_unmapped_area (struct mm_struct *mm,struct file *filp, +unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags); + unsigned long generic_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff,
io_uring/io_uring.c +1 -1
@@ -3513,7 +3513,7 @@ static unsigned long io_uring_mmu_get_unmapped_area(struct file *filp,#else addr =0UL ;#endif -return current->mm->get_unmapped_area (filp, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, filp, addr, len, pgoff, flags); }#else /* !CONFIG_MMU */
mm/debug.c +0 -6
@@ -162,9 +162,6 @@ EXPORT_SYMBOL(dump_vma);void dump_mm (const struct mm_struct *mm) {pr_emerg ("mm %px task_size %lu\n" -#ifdef CONFIG_MMU - "get_unmapped_area %px\n" -#endif"mmap_base %lu mmap_legacy_base %lu\n" "pgd %px mm_users %d mm_count %d pgtables_bytes %lu map_count %d\n" "hiwater_rss %lx hiwater_vm %lx total_vm %lx locked_vm %lx\n"
@@ -190,9 +187,6 @@ void dump_mm(const struct mm_struct *mm)"def_flags: %#lx(%pGv)\n" , mm, mm->task_size, -#ifdef CONFIG_MMU - mm->get_unmapped_area, -#endif mm->mmap_base, mm->mmap_legacy_base, mm->pgd,atomic_read (&mm->mm_users), atomic_read(&mm->mm_count),
mm/huge_memory.c +4 -5
@@ -822,8 +822,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp,if (len_pad < len || (off + len_pad) < off)return 0 ; - ret = current->mm->get_unmapped_area (filp, addr, len_pad, - off >>PAGE_SHIFT , flags); + ret =mm_get_unmapped_area (current->mm, filp, addr, len_pad, + off >>PAGE_SHIFT , flags);/* * The failure might be due to length padding. The caller will retry
@@ -841,8 +841,7 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, off_sub = (off - ret) & (size -1 ); -if (current->mm->get_unmapped_area == arch_get_unmapped_area_topdown && - !off_sub) +if (test_bit (MMF_TOPDOWN , ¤t->mm->flags) && !off_sub)return ret + size; ret += off_sub;
@@ -859,7 +858,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,if (ret)return ret; -return current->mm->get_unmapped_area (filp, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, filp, addr, len, pgoff, flags); }EXPORT_SYMBOL_GPL (thp_get_unmapped_area);
mm/mmap.c +18 -3
@@ -1815,7 +1815,8 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) {unsigned long (*get_area)(struct file *,unsigned long , -unsigned long ,unsigned long ,unsigned long ); +unsigned long ,unsigned long ,unsigned long ) + =NULL ;unsigned long error =arch_mmap_check (addr, len, flags);if (error)
@@ -1825,7 +1826,6 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,if (len >TASK_SIZE )return -ENOMEM ; - get_area = current->mm->get_unmapped_area;if (file) {if (file->f_op->get_unmapped_area) get_area = file->f_op->get_unmapped_area;
@@ -1844,7 +1844,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,if (!file) pgoff =0 ; - addr =get_area (file, addr, len, pgoff, flags); +if (get_area) + addr =get_area (file, addr, len, pgoff, flags); +else + addr =mm_get_unmapped_area (current->mm, file, addr, len, + pgoff, flags);if (IS_ERR_VALUE (addr))return addr;
@@ -1859,6 +1863,17 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,EXPORT_SYMBOL (get_unmapped_area); +unsigned long +mm_get_unmapped_area (struct mm_struct *mm,struct file *file, +unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags) +{ +if (test_bit (MMF_TOPDOWN , &mm->flags)) +return arch_get_unmapped_area_topdown (file, addr, len, pgoff, flags); +return arch_get_unmapped_area (file, addr, len, pgoff, flags); +} +EXPORT_SYMBOL (mm_get_unmapped_area); +/** * find_vma_intersection() - Look up the first VMA which intersects the interval * @mm: The process address space.
mm/shmem.c +5 -6
@@ -2240,8 +2240,6 @@ unsigned long shmem_get_unmapped_area(struct file *file, unsigned long uaddr, unsigned long len, unsigned long pgoff, unsigned long flags) { - unsigned long (*get_area)(struct file *, -unsigned long ,unsigned long ,unsigned long ,unsigned long ); unsigned long addr;unsigned long offset;unsigned long inflated_len;
@@ -2251,8 +2249,8 @@ unsigned long shmem_get_unmapped_area(struct file *file,if (len >TASK_SIZE )return -ENOMEM ; - get_area = current->mm->get_unmapped_area; - addr =get_area (file, uaddr, len, pgoff, flags); + addr =mm_get_unmapped_area (current->mm, file, uaddr, len, pgoff, + flags);if (!IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ))return addr;
@@ -2309,7 +2307,8 @@ unsigned long shmem_get_unmapped_area(struct file *file,if (inflated_len < len)return addr; - inflated_addr =get_area (NULL , uaddr, inflated_len,0 , flags); + inflated_addr =mm_get_unmapped_area (current->mm,NULL , uaddr, + inflated_len,0 , flags);if (IS_ERR_VALUE (inflated_addr))return addr;if (inflated_addr & ~PAGE_MASK )
@@ -4755,7 +4754,7 @@ unsigned long shmem_get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); }#endif
mm/util.c +3 -3
@@ -452,17 +452,17 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack)if (mmap_is_legacy (rlim_stack)) { mm->mmap_base =TASK_UNMAPPED_BASE + random_factor; - mm->get_unmapped_area = arch_get_unmapped_area; +clear_bit (MMF_TOPDOWN , &mm->flags); }else { mm->mmap_base =mmap_base (random_factor, rlim_stack); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; +set_bit (MMF_TOPDOWN , &mm->flags); } }#elif defined(CONFIG_MMU) && !defined(HAVE_ARCH_PICK_MMAP_LAYOUT)void arch_pick_mmap_layout (struct mm_struct *mm,struct rlimit *rlim_stack) { mm->mmap_base =TASK_UNMAPPED_BASE ; - mm->get_unmapped_area = arch_get_unmapped_area; +clear_bit (MMF_TOPDOWN , &mm->flags); }#endif
Signature · 1 line
2.34.1
Re: [PATCH v3 01/12] mm: Switch mm->get_unmapped_area() to a flag
Le 12/03/2024 à 23:28, Rick Edgecombe a écrit :
Quoted 27 lines
The
mm_structcontains a function pointer*get_unmapped_area(), which is set to eitherarch_get_unmapped_area()orarch_get_unmapped_area_topdown()during the initialization of the mm.Since the function pointer only ever points to two functions that are named the same across all arch's, a function pointer is not really required. In addition future changes will want to add versions of the functions that take additional arguments. So to save a pointers worth of bytes in
mm_struct, and prevent adding additional function pointers tomm_structin future changes, remove it and keep the information about whichget_unmapped_area()to use in a flag.Add the new flag to
MMF_INIT_MASKso it doesn't get clobbered on fork bymmf_init_flags(). Most MM flags get clobbered on fork. In the pre-existing behaviormm->get_unmapped_area()would get copied to the new mm indup_mm(), so not clobbering the flag preserves the existing behavior around inheriting the topdown-ness.Introduce a helper,
mm_get_unmapped_area(), to easily convert code that refers to the old function pointer to instead select and call eitherarch_get_unmapped_area()orarch_get_unmapped_area_topdown()based on the flag. Then drop themm->get_unmapped_area()function pointer. Leave theget_unmapped_area()pointer in structfile_operationsalone. The main purpose of this change is to reorganize in preparation for future changes, but it also converts the calls ofmm->get_unmapped_area()from indirect branches into a direct ones.
Quoted 8 lines
The stress-ng bigheap benchmark calls realloc a lot, which calls through
get_unmapped_area()in the kernel. On x86, the change yielded a ~1% improvement there on a retpoline config.In testing a few x86 configs, removing the pointer unfortunately didn't result in any actual size reductions in the compiled layout of
mm_struct. But depending on compiler or arch alignment requirements, the change could shrink the size ofmm_struct.
This patch is quite big and un-easy to follow. Would be worth splitting in several patches if possible. Some of the changes seem to go further than just switching mm->get_unmapped_area() to a flag.
First patch could add the new flag and necessary helpers, then following patches could convert sub-systems one by one then last patch would remove mm->get_unmapped_area() once all users are converted.
Quoted 255 lines
Signed-off-by Rick Edgecombe <[email protected]>Acked-by Dave Hansen <[email protected]>Acked-by Liam R. Howlett <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>---
v3:
- Fix comment that still referred tomm->get_unmapped_area()
- Resolve trivial rebase conflicts with "mm:thp_get_unmapped_areamust honour topdown preference"
- Spelling fix in logv2:
- Fix comment onMMF_TOPDOWN(Kirill, rppt)
- MoveMMF_TOPDOWNto actually unused bit
- AddMMF_TOPDOWNtoMMF_INIT_MASKso it doesn't get clobbered on fork, and result in the children using the search up path.
- New lower performance results after above bug fix
- Add Reviews and Acks--- arch/s390/mm/hugetlbpage.c | 2 +- arch/s390/mm/mmap.c | 4 ++-- arch/sparc/kernel/sys_sparc_64.c | 15 ++++++--------- arch/sparc/mm/hugetlbpage.c | 2 +- arch/x86/kernel/cpu/sgx/driver.c | 2 +- arch/x86/mm/hugetlbpage.c | 2 +- arch/x86/mm/mmap.c | 4 ++-- drivers/char/mem.c | 2 +- drivers/dax/device.c | 6 +++--- fs/hugetlbfs/inode.c | 4 ++-- fs/proc/inode.c | 15 ++++++++------- fs/ramfs/file-mmu.c | 2 +- include/linux/mm_types.h | 6 +----- include/linux/sched/coredump.h | 5 ++++- include/linux/sched/mm.h | 5 +++++ io_uring/io_uring.c | 2 +- mm/debug.c | 6 ------ mm/huge_memory.c | 9 ++++----- mm/mmap.c | 21 ++++++++++++++++++--- mm/shmem.c | 11 +++++------ mm/util.c | 6 +++--- 21 files changed, 70 insertions(+), 61 deletions(-)arch/s390/mm/hugetlbpage.c +1
-1@@ -328,7 +328,7 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, goto check_asce_limit; } -if (mm->get_unmapped_area == arch_get_unmapped_area) +if (!test_bit (MMF_TOPDOWN , &mm->flags)) addr =hugetlb_get_unmapped_area_bottomup (file, addr, len, pgoff, flags);else arch/s390/mm/mmap.c +2
-2@@ -182,10 +182,10 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack)*/ if (mmap_is_legacy (rlim_stack)) { mm->mmap_base =mmap_base_legacy (random_factor); - mm->get_unmapped_area = arch_get_unmapped_area; +clear_bit (MMF_TOPDOWN , &mm->flags); }else { mm->mmap_base =mmap_base (random_factor, rlim_stack); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; +set_bit (MMF_TOPDOWN , &mm->flags); } }arch/sparc/kernel/sys_sparc_64.c +6
-9@@ -218,14 +218,10 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,unsigned long get_fb_unmapped_area (struct file *filp,unsigned long orig_addr,unsigned long len,unsigned long pgoff,unsigned long flags) {unsigned long align_goal, addr = -ENOMEM ; -unsigned long (*get_area)(struct file *,unsigned long , -unsigned long ,unsigned long ,unsigned long ); - - get_area = current->mm->get_unmapped_area;if (flags &MAP_FIXED ) {/* Ok, don't mess with it. */ -return get_area (NULL , orig_addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm,NULL , orig_addr, len, pgoff, flags); } flags &= ~MAP_SHARED ;@@ -238,7 +234,8 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u align_goal = (64UL *1024 ); do { - addr = get_area(NULL, orig_addr, len + (align_goal - PAGE_SIZE), pgoff, flags); + addr = mm_get_unmapped_area(current->mm, NULL, orig_addr, + len + (align_goal - PAGE_SIZE), pgoff, flags); if (!(addr & ~PAGE_MASK)) { addr = (addr + (align_goal - 1UL)) & ~(align_goal - 1UL); break;@@ -256,7 +253,7 @@ unsigned long get_fb_unmapped_area(struct file *filp, unsigned long orig_addr, u* be obtained. */ if (addr & ~PAGE_MASK ) - addr =get_area (NULL , orig_addr, len, pgoff, flags); + addr =mm_get_unmapped_area (current->mm,NULL , orig_addr, len, pgoff, flags);return addr; }@@ -292,7 +289,7 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) gap == RLIM_INFINITY || sysctl_legacy_va_layout) { mm->mmap_base = TASK_UNMAPPED_BASE + random_factor; - mm->get_unmapped_area = arch_get_unmapped_area; + clear_bit(MMF_TOPDOWN, &mm->flags); } else {/* We know it's 32-bit */ unsigned long task_size =STACK_TOP32 ;@@ -303,7 +300,7 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) gap = (task_size /6 *5 ); mm->mmap_base =PAGE_ALIGN (task_size - gap - random_factor); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; +set_bit (MMF_TOPDOWN , &mm->flags); } }arch/sparc/mm/hugetlbpage.c +1
-1@@ -123,7 +123,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, (!vma || addr + len <= vm_start_gap(vma)))return addr; } -if (mm->get_unmapped_area == arch_get_unmapped_area) +if (!test_bit (MMF_TOPDOWN , &mm->flags))return hugetlb_get_unmapped_area_bottomup (file, addr, len, pgoff, flags);else arch/x86/kernel/cpu/sgx/driver.c +1
-1@@ -113,7 +113,7 @@ static unsigned long sgx_get_unmapped_area(struct file *file,if (flags &MAP_FIXED )return addr; -return current->mm->get_unmapped_area (file, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, file, addr, len, pgoff, flags); }#ifdef CONFIG_COMPAT arch/x86/mm/hugetlbpage.c +1
-1@@ -141,7 +141,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, } get_unmapped_area: -if (mm->get_unmapped_area == arch_get_unmapped_area) +if (!test_bit (MMF_TOPDOWN , &mm->flags))return hugetlb_get_unmapped_area_bottomup (file, addr, len, pgoff, flags);else arch/x86/mm/mmap.c +2
-2@@ -129,9 +129,9 @@ static void arch_pick_mmap_base(unsigned long *base, unsigned long *legacy_base,void arch_pick_mmap_layout (struct mm_struct *mm,struct rlimit *rlim_stack) {if (mmap_is_legacy ()) - mm->get_unmapped_area = arch_get_unmapped_area; +clear_bit (MMF_TOPDOWN , &mm->flags);else - mm->get_unmapped_area = arch_get_unmapped_area_topdown; +set_bit (MMF_TOPDOWN , &mm->flags); arch_pick_mmap_base(&mm->mmap_base, &mm->mmap_legacy_base, arch_rnd(mmap64_rnd_bits), task_size_64bit(0),drivers/char/mem.c +1
-1@@ -544,7 +544,7 @@ static unsigned long get_unmapped_area_zero(struct file *file, }/* Otherwise flags & MAP_PRIVATE: with no shmem object beneath it */ -return current->mm->get_unmapped_area (file, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, file, addr, len, pgoff, flags);#else return -ENOSYS ;#endif drivers/dax/device.c +3
-3@@ -329,14 +329,14 @@ static unsigned long dax_get_unmapped_area(struct file *filp,if ((off + len_align) < off) goto out; - addr_align = current->mm->get_unmapped_area (filp, addr, len_align, - pgoff, flags); + addr_align =mm_get_unmapped_area (current->mm, filp, addr, len_align, + pgoff, flags);if (!IS_ERR_VALUE (addr_align)) { addr_align += (off - addr_align) & (align -1 );return addr_align; } out: -return current->mm->get_unmapped_area (filp, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, filp, addr, len, pgoff, flags); }static const struct address_space_operations dev_dax_aops = {fs/hugetlbfs/inode.c +2
-2@@ -249,11 +249,11 @@ generic_hugetlb_get_unmapped_area(struct file *file, unsigned long addr, }/* -* Use mm->get_unmapped_area value as a hint to use topdown routine. +* Use MMF_TOPDOWN flag as a hint to use topdown routine. * If architectures have special needs, they should define their own * version of hugetlb_get_unmapped_area. */ -if (mm->get_unmapped_area == arch_get_unmapped_area_topdown) +if (test_bit (MMF_TOPDOWN , &mm->flags))return hugetlb_get_unmapped_area_topdown (file, addr, len, pgoff, flags); return hugetlb_get_unmapped_area_bottomup(file, addr, len,fs/proc/inode.c +4
-4@@ -451,15 +451,16 @@ pde_get_unmapped_area(struct proc_dir_entry *pde, struct file *file, unsigned lo unsigned long len, unsigned long pgoff, unsigned long flags) { - typeof_member(struct proc_ops, proc_get_unmapped_area) get_area; - - get_area = pde->proc_ops->proc_get_unmapped_area; + if (pde->proc_ops->proc_get_unmapped_area) + return pde->proc_ops->proc_get_unmapped_area(file, orig_addr, + len, pgoff, + flags);#ifdef CONFIG_MMU -if (!get_area)
fs/proc/inode.c pde_get_unmapped_area(struct proc_dir_entry *pde, struct file *file, unsigned lo · line 465
Quoted 10 lines
fs/proc/inode.c +4
-3- get_area = current->mm->get_unmapped_area; + else + return mm_get_unmapped_area(current->mm, file, orig_addr, + len, pgoff, flags);#endif -if (get_area) -return get_area (file, orig_addr, len, pgoff, flags); +return orig_addr; }
The change looks unclear at first look. Ok after looking a second time it seems to simplify things, but would be better as a separate patch. Don't know.
Quoted 181 lines
fs/ramfs/file-mmu.c +1
-1@@ -35,7 +35,7 @@ static unsigned long ramfs_mmu_get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); } const struct file_operations ramfs_file_operations = {include/linux/mm_types.h +1
-5@@ -749,11 +749,7 @@ struct mm_struct { }____cacheline_aligned_in_smp ;struct maple_tree mm_mt; -#ifdef CONFIG_MMU -unsigned long (*get_unmapped_area) (struct file *filp, -unsigned long addr,unsigned long len, -unsigned long pgoff,unsigned long flags); -#endif +unsigned long mmap_base;/* base of mmap area */ unsigned long mmap_legacy_base;/* base of mmap area in bottom-up allocations */ #ifdef CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES include/linux/sched/coredump.h +4
-1@@ -92,9 +92,12 @@ static inline int get_dumpable(struct mm_struct *mm)#define MMF_VM_MERGE_ANY 30#define MMF_VM_MERGE_ANY_MASK (1 << MMF_VM_MERGE_ANY) +#define MMF_TOPDOWN 31/* mm searches top down by default */ +#define MMF_TOPDOWN_MASK (1 << MMF_TOPDOWN) +#define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ - MMF_VM_MERGE_ANY_MASK) + MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK)static inline unsigned long mmf_init_flags (unsigned long flags) {include/linux/sched/mm.h +5
-0@@ -8,6 +8,7 @@#include <linux/mm_types.h> #include <linux/gfp.h> #include <linux/sync_core.h> +#include <linux/sched/coredump.h> /* * Routines for handling mm_structs @@ -186,6 +187,10 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +unsigned long mm_get_unmapped_area (struct mm_struct *mm,struct file *filp, +unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags); + unsigned long generic_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff,io_uring/io_uring.c +1
-1@@ -3513,7 +3513,7 @@ static unsigned long io_uring_mmu_get_unmapped_area(struct file *filp,#else addr =0UL ;#endif -return current->mm->get_unmapped_area (filp, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, filp, addr, len, pgoff, flags); }#else /* !CONFIG_MMU */ mm/debug.c +0
-6@@ -162,9 +162,6 @@ EXPORT_SYMBOL(dump_vma);void dump_mm (const struct mm_struct *mm) {pr_emerg ("mm %px task_size %lu\n" -#ifdef CONFIG_MMU - "get_unmapped_area %px\n" -#endif"mmap_base %lu mmap_legacy_base %lu\n" "pgd %px mm_users %d mm_count %d pgtables_bytes %lu map_count %d\n" "hiwater_rss %lx hiwater_vm %lx total_vm %lx locked_vm %lx\n"@@ -190,9 +187,6 @@ void dump_mm(const struct mm_struct *mm)"def_flags: %#lx(%pGv)\n" , mm, mm->task_size, -#ifdef CONFIG_MMU - mm->get_unmapped_area, -#endif mm->mmap_base, mm->mmap_legacy_base, mm->pgd,atomic_read (&mm->mm_users), atomic_read(&mm->mm_count),mm/huge_memory.c +4
-5@@ -822,8 +822,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp,if (len_pad < len || (off + len_pad) < off)return 0 ; - ret = current->mm->get_unmapped_area (filp, addr, len_pad, - off >>PAGE_SHIFT , flags); + ret =mm_get_unmapped_area (current->mm, filp, addr, len_pad, + off >>PAGE_SHIFT , flags);/* * The failure might be due to length padding. The caller will retry @@ -841,8 +841,7 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, off_sub = (off - ret) & (size -1 ); -if (current->mm->get_unmapped_area == arch_get_unmapped_area_topdown && - !off_sub) +if (test_bit (MMF_TOPDOWN , ¤t->mm->flags) && !off_sub)return ret + size; ret += off_sub;@@ -859,7 +858,7 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,if (ret)return ret; -return current->mm->get_unmapped_area (filp, addr, len, pgoff, flags); +return mm_get_unmapped_area (current->mm, filp, addr, len, pgoff, flags); }EXPORT_SYMBOL_GPL (thp_get_unmapped_area);mm/mmap.c +7
-3@@ -1815,7 +1815,8 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) {unsigned long (*get_area)(struct file *,unsigned long , -unsigned long ,unsigned long ,unsigned long ); +unsigned long ,unsigned long ,unsigned long ) + =NULL ;unsigned long error =arch_mmap_check (addr, len, flags);if (error)@@ -1825,7 +1826,6 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,if (len >TASK_SIZE )return -ENOMEM ; - get_area = current->mm->get_unmapped_area;if (file) {if (file->f_op->get_unmapped_area) get_area = file->f_op->get_unmapped_area;@@ -1844,7 +1844,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,if (!file) pgoff =0 ; - addr =get_area (file, addr, len, pgoff, flags); +if (get_area) + addr =get_area (file, addr, len, pgoff, flags); +else + addr =mm_get_unmapped_area (current->mm, file, addr, len, + pgoff, flags);if (IS_ERR_VALUE (addr))return addr;@@ -1859,6 +1863,17 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,EXPORT_SYMBOL (get_unmapped_area);
mm/mmap.c get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, · line 1874
Quoted 10 lines
mm/mmap.c +9
-0+unsigned long +mm_get_unmapped_area (struct mm_struct *mm,struct file *file, +unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags) +{ +if (test_bit (MMF_TOPDOWN , &mm->flags)) +return arch_get_unmapped_area_topdown (file, addr, len, pgoff, flags); +return arch_get_unmapped_area (file, addr, len, pgoff, flags); +}
This function seems quite simple, wouldn't it be better to make it a static inline ?
Quoted 73 lines
mm/mmap.c +2
-0+EXPORT_SYMBOL (mm_get_unmapped_area); +/** * find_vma_intersection() - Look up the first VMA which intersects the interval * @mm: The process address space. mm/shmem.c +5
-6@@ -2240,8 +2240,6 @@ unsigned long shmem_get_unmapped_area(struct file *file, unsigned long uaddr, unsigned long len, unsigned long pgoff, unsigned long flags) { - unsigned long (*get_area)(struct file *, -unsigned long ,unsigned long ,unsigned long ,unsigned long ); unsigned long addr;unsigned long offset;unsigned long inflated_len;@@ -2251,8 +2249,8 @@ unsigned long shmem_get_unmapped_area(struct file *file,if (len >TASK_SIZE )return -ENOMEM ; - get_area = current->mm->get_unmapped_area; - addr =get_area (file, uaddr, len, pgoff, flags); + addr =mm_get_unmapped_area (current->mm, file, uaddr, len, pgoff, + flags);if (!IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ))return addr;@@ -2309,7 +2307,8 @@ unsigned long shmem_get_unmapped_area(struct file *file,if (inflated_len < len)return addr; - inflated_addr =get_area (NULL , uaddr, inflated_len,0 , flags); + inflated_addr =mm_get_unmapped_area (current->mm,NULL , uaddr, + inflated_len,0 , flags);if (IS_ERR_VALUE (inflated_addr))return addr;if (inflated_addr & ~PAGE_MASK )@@ -4755,7 +4754,7 @@ unsigned long shmem_get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { - return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); + return mm_get_unmapped_area(current->mm, file, addr, len, pgoff, flags); }#endif mm/util.c +3
-3@@ -452,17 +452,17 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack)if (mmap_is_legacy (rlim_stack)) { mm->mmap_base =TASK_UNMAPPED_BASE + random_factor; - mm->get_unmapped_area = arch_get_unmapped_area; +clear_bit (MMF_TOPDOWN , &mm->flags); }else { mm->mmap_base =mmap_base (random_factor, rlim_stack); - mm->get_unmapped_area = arch_get_unmapped_area_topdown; +set_bit (MMF_TOPDOWN , &mm->flags); } }#elif defined(CONFIG_MMU) && !defined(HAVE_ARCH_PICK_MMAP_LAYOUT)void arch_pick_mmap_layout (struct mm_struct *mm,struct rlimit *rlim_stack) { mm->mmap_base =TASK_UNMAPPED_BASE ; - mm->get_unmapped_area = arch_get_unmapped_area; +clear_bit (MMF_TOPDOWN , &mm->flags); }#endif
Re: [PATCH v3 01/12] mm: Switch mm->get_unmapped_area() to a flag
Quoted 10 lines · Christophe Leroy
This patch is quite big and un-easy to follow. Would be worth splitting
in several patches if possible. Some of the changes seem to go further
than just switchingmm->get_unmapped_area()to a flag.First patch could add the new flag and necessary helpers, then following
patches could convert sub-systems one by one then last patch would removemm->get_unmapped_area()once all users are converted.
So you are saying to do the tracking in both the new flag and mm-
Quoted 1 line
get_unmapped_area()during the conversion process and then remove the
pointer at the end? I guess it could be broken out, but most of the conversions are trivial one line changes. Hmm, I'm not sure.
[snip]
Quoted 16 lines
Quoted 15 lines
#ifdef
CONFIG_MMU
- if (!get_area)
- get_area =current->mm->get_unmapped_area; + else
+ return mm_get_unmapped_area(current->mm, file,orig_addr,
+ len, pgoff, flags); #endif
- if (get_area)
- returnget_area(file, orig_addr, len, pgoff, flags);
+
returnorig_addr;
}
Quoted 4 lines
The change looks unclear at first look. Ok after looking a second time
it seems to simplify things, but would be better as a separate patch. Don't know.
Hmm. I think the only way to do it in smaller chunks is to do both methods of tracking the direction during the conversion process. And then the smaller pieces would be really small. So it would probably help for changes like this, but otherwise would generate a lot of patches with small changes.
The steps are basically:
1. Introduce flag and helpers
2. convert arch's to use it one by one
3. convert callers to use mm_get_unmapped_area() one by one
4. remove setting get_unmapped_area in each arch
5. remove get_unmapped_area
Step 3 is where the few non-oneline changes would be, but most would still be one liners. 1, 2, 4 and 5 seem simpler as a tree wide patch because of the one line changes.
I don't know any other variations are a ton simpler. Hopefully others will weigh in.
[snip]
Quoted 7 lines
Quoted 7 lines
mm/mmap.c +6
-0+unsigned long +mm_get_unmapped_area (struct mm_struct *mm,struct file *file, + unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + if (test_bit(MMF_TOPDOWN, &mm->flags))
Quoted 8 lines
Quoted 5 lines
+ return
arch_get_unmapped_area_topdown(file, addr, len, pgoff, flags);
+ returnarch_get_unmapped_area(file, addr, len, pgoff, flags);
+}This function seems quite simple, wouldn't it be better to make it a static inline ?
Then all of the arch_get_unmapped_area() and arch_get_unmapped_area_topdown() would need to be exported. I think it is better to only export the higher level functions.
Re: [PATCH v3 01/12] mm: Switch mm->get_unmapped_area() to a flag
Le 13/03/2024 à 15:48, Edgecombe, Rick P a écrit :
Quoted 39 lines · Christophe Leroy
Quoted 10 lines
This patch is quite big and un-easy to follow. Would be worth splitting
in several patches if possible. Some of the changes seem to go further
than just switchingmm->get_unmapped_area()to a flag.First patch could add the new flag and necessary helpers, then following
patches could convert sub-systems one by one then last patch would removemm->get_unmapped_area()once all users are converted.So you are saying to do the tracking in both the new flag and mm-
Quoted 1 line
get_unmapped_area()during the conversion process and then remove thepointer at the end? I guess it could be broken out, but most of the conversions are trivial one line changes. Hmm, I'm not sure.
[snip]
Quoted 16 lines
Quoted 15 lines
#ifdef
CONFIG_MMU
- if (!get_area)
- get_area =current->mm->get_unmapped_area; + else
+ return mm_get_unmapped_area(current->mm, file,orig_addr,
+ len, pgoff, flags);
#endif
- if (get_area)
- returnget_area(file, orig_addr, len, pgoff, flags);
+
returnorig_addr;
}Quoted 4 lines
The change looks unclear at first look. Ok after looking a second time
it seems to simplify things, but would be better as a separate patch. Don't know.
Quoted 5 lines
Hmm. I think the only way to do it in smaller chunks is to do both methods of tracking the direction during the conversion process. And then the smaller pieces would be really small. So it would probably help for changes like this, but otherwise would generate a lot of patches with small changes.
Yes. Maybe the best would be to have a preparation patch to churn this function a bit so that when it comes to the conservion it is trivial.
Something like:
if (pde->proc_ops->proc_get_unmapped_area) return pde->proc_ops->proc_get_unmapped_area(file, orig_addr, len,
pgoff, flags);
#ifdef CONFIG_MMU return current->mm->get_unmapped_area(file, orig_addr, len, pgoff, flags); #endif return orig_addr;
Note that a length of 100 chars is now tolerated when it eases reading so you should avoid those 3 lines.
And the else inside #ifdef CONFIG_MMU is not needed because 'if' has returned.
Quoted 11 lines
The steps are basically:
1. Introduce flag and helpers
2. convert arch's to use it one by one
3. convert callers to usemm_get_unmapped_area()one by one
4. remove settingget_unmapped_areain each arch
5. removeget_unmapped_areaStep 3 is where the few non-oneline changes would be, but most would still be one liners. 1, 2, 4 and 5 seem simpler as a tree wide patch because of the one line changes.
I missed the setting of get_unmapped_area by each arch, you are right it might be complicated at the end.
Quoted 23 lines
I don't know any other variations are a ton simpler. Hopefully others will weigh in.
[snip]
Quoted 15 lines
Quoted 8 lines
mm/mmap.c +7
-0+unsigned long +mm_get_unmapped_area (struct mm_struct *mm,struct file *file, + unsigned long addr, unsigned long len, + unsigned long pgoff, unsigned long flags) +{ + if (test_bit(MMF_TOPDOWN, &mm->flags)) + return arch_get_unmapped_area_topdown(file, addr,Quoted 4 lines
len, pgoff, flags);
+ returnarch_get_unmapped_area(file, addr, len, pgoff, flags);
+}This function seems quite simple, wouldn't it be better to make it a static inline ?
Quoted 3 lines
Then all of the
arch_get_unmapped_area()andarch_get_unmapped_area_topdown()would need to be exported. I think it is better to only export the higher level functions.
Right.
PATCH v3 2/12 mm: Introduce arch_get_unmapped_area_vmflags()
In reply to Rick EdgecombeWhen memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.
The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.
In order to take the start gap into account, the maple tree search needs to know the size of start gap the new mapping will need. The call chain from do_mmap() to the actual maple tree search looks like this:
do_mmap(size, vm_flags, map_flags, ..)
mm/mmap.c:get_unmapped_area(size, map_flags, ...) arch_get_unmapped_area(size, map_flags, ...) vm_unmapped_area(struct vm_unmapped_area_info)
One option would be to add another MAP_ flag to mean a one page start gap (as is for shadow stack), but this consumes a flag unnecessarily. Another option could be to simply increase the size passed in do_mmap() by the start gap size, and adjust after the fact, but this will interfere with the alignment requirements passed in struct vm_unmapped_area_info, and unknown to mmap.c. Instead, introduce variants of arch_get_unmapped_area/_topdown() that take vm_flags. In future changes, these variants can be used in mmap.c:get_unmapped_area() to allow the vm_flags to be passed through to vm_unmapped_area(), while preserving the normal arch_get_unmapped_area/_topdown() for the existing callers.
--- include/linux/sched/mm.h | 17 +++++++++++++++++ mm/mmap.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+)
include/linux/sched/mm.h +17 -0
@@ -191,6 +191,23 @@ unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +extern unsigned long +arch_get_unmapped_area_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t vm_flags); +extern unsigned long +arch_get_unmapped_area_topdown_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t ); + +unsigned long mm_get_unmapped_area_vmflags (struct mm_struct *mm, +struct file *filp, +unsigned long addr, +unsigned long len, +unsigned long pgoff, +unsigned long flags, +vm_flags_t vm_flags); + unsigned long generic_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff,
mm/mmap.c +28 -0
@@ -1810,6 +1810,34 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr, }#endif +#ifndef HAVE_ARCH_UNMAPPED_AREA_VMFLAGS +extern unsigned long +arch_get_unmapped_area_vmflags (struct file *filp,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags,vm_flags_t vm_flags) +{ +return arch_get_unmapped_area (filp, addr, len, pgoff, flags); +} + +extern unsigned long +arch_get_unmapped_area_topdown_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t vm_flags) +{ +return arch_get_unmapped_area_topdown (filp, addr, len, pgoff, flags); +} +#endif + +unsigned long mm_get_unmapped_area_vmflags (struct mm_struct *mm,struct file *filp, +unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags, +vm_flags_t vm_flags) +{ +if (test_bit (MMF_TOPDOWN , &mm->flags)) +return arch_get_unmapped_area_topdown_vmflags (filp, addr, len, pgoff, + flags, vm_flags); +return arch_get_unmapped_area_vmflags (filp, addr, len, pgoff, flags, vm_flags); +} +unsigned long get_unmapped_area (struct file *file,unsigned long addr,unsigned long len,unsigned long pgoff,unsigned long flags)
Signature · 1 line
2.34.1
Re: [PATCH v3 02/12] mm: Introduce arch_get_unmapped_area_vmflags()
Le 12/03/2024 à 23:28, Rick Edgecombe a écrit :
Quoted 48 lines
When memory is being placed,
mmap()will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK,VM_GROWSUPandVM_GROWSDOWN). In order to ensure guard gaps between mappings,mmap()needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.The long standing behavior of
mmap()is to ensure 1, but not take any care around 2. So for example, if there is aPAGE_SIZEfree area, and ammap()with aPAGE_SIZEsize, and a type that has a guard gap is being placed,mmap()may place the shadow stack in thePAGE_SIZEfree area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.In order to take the start gap into account, the maple tree search needs to know the size of start gap the new mapping will need. The call chain from
do_mmap()to the actual maple tree search looks like this:
do_mmap(size, vm_flags, map_flags, ..)mm/mmap.c:get_unmapped_area(size, map_flags, ...) arch_get_unmapped_area(size, map_flags, ...) vm_unmapped_area(struct vm_unmapped_area_info)One option would be to add another
MAP_flag to mean a one page start gap (as is for shadow stack), but this consumes a flag unnecessarily. Another option could be to simply increase the size passed indo_mmap()by the start gap size, and adjust after the fact, but this will interfere with the alignment requirements passed in structvm_unmapped_area_info, and unknown tommap.c. Instead, introduce variants of arch_get_unmapped_area/_topdown() that takevm_flags. In future changes, these variants can be used in mmap.c:get_unmapped_area() to allow thevm_flagsto be passed through tovm_unmapped_area(), while preserving the normal arch_get_unmapped_area/_topdown() for the existing callers.Signed-off-by Rick Edgecombe <[email protected]>--- include/linux/sched/mm.h | 17 +++++++++++++++++ mm/mmap.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+)include/linux/sched/mm.h +0
-0@@ -191,6 +191,23 @@ unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *filp, unsigned long addr, unsigned long len,
include/linux/sched/mm.h unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *filp, · line 201
Quoted 10 lines
include/linux/sched/mm.h +8
-0unsigned long pgoff, unsigned long flags); +extern unsigned long +arch_get_unmapped_area_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t vm_flags); +extern unsigned long +arch_get_unmapped_area_topdown_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t );
Please, no new "extern", that's pointless for function prototypes and outdate coding style. Checkpatch --strict is likely protesting about it.
Quoted 17 lines
include/linux/sched/mm.h +9
-0+ +unsigned long mm_get_unmapped_area_vmflags (struct mm_struct *mm, +struct file *filp, +unsigned long addr, +unsigned long len, +unsigned long pgoff, +unsigned long flags, +vm_flags_t vm_flags); + unsigned long generic_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff,mm/mmap.c +0
-0
mm/mmap.c arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr, · line 1819
Quoted 10 lines
mm/mmap.c +7
-0@@ -1810,6 +1810,34 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr, }#endif +#ifndef HAVE_ARCH_UNMAPPED_AREA_VMFLAGS +extern unsigned long +arch_get_unmapped_area_vmflags (struct file *filp,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags,vm_flags_t vm_flags) +{ +return arch_get_unmapped_area (filp, addr, len, pgoff, flags); +}
Same, what is the point with that "extern" keyword in function definition ?
Quoted 24 lines
mm/mmap.c +21
-0+ +extern unsigned long +arch_get_unmapped_area_topdown_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t vm_flags) +{ +return arch_get_unmapped_area_topdown (filp, addr, len, pgoff, flags); +} +#endif + +unsigned long mm_get_unmapped_area_vmflags (struct mm_struct *mm,struct file *filp, +unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags, +vm_flags_t vm_flags) +{ +if (test_bit (MMF_TOPDOWN , &mm->flags)) +return arch_get_unmapped_area_topdown_vmflags (filp, addr, len, pgoff, + flags, vm_flags); +return arch_get_unmapped_area_vmflags (filp, addr, len, pgoff, flags, vm_flags); +} +unsigned long get_unmapped_area (struct file *file,unsigned long addr,unsigned long len,unsigned long pgoff,unsigned long flags)
Re: [PATCH v3 02/12] mm: Introduce arch_get_unmapped_area_vmflags()
Quoted 4 lines · Christophe Leroy
Please, no new "extern", that's pointless for function prototypes and outdate coding style. Checkpatch --strict is likely protesting about it.
I was trying to match the style of the surrounding code in the headers. Previous feedback was to not fix some alignment as well.
But the extern in the mmap.c file was just a mistake. I can remove the externs.
PATCH v3 3/12 mm: Use get_unmapped_area_vmflags()
In reply to Rick EdgecombeWhen memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.
The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.
Use mm_get_unmapped_area_vmflags() in the do_mmap() so future changes can cause shadow stack mappings to be placed with a guard gap. Also use the THP variant that takes vm_flags, such that THP shadow stack can get the same treatment. Adjust the vm_flags calculation to happen earlier so that the vm_flags can be passed into __get_unmapped_area().
---
v2:
- Make get_unmapped_area() a static inline (Kirill)
--- include/linux/mm.h | 11 ++++++++++- mm/mmap.c | 34 ++++++++++++++++------------------ 2 files changed, 26 insertions(+), 19 deletions(-)
include/linux/mm.h +10 -1
@@ -3363,7 +3363,16 @@ extern int install_special_mapping(struct mm_struct *mm,unsigned long randomize_stack_top (unsigned long stack_top);unsigned long randomize_page (unsigned long start,unsigned long range); -extern unsigned long get_unmapped_area (struct file *,unsigned long ,unsigned long ,unsigned long ,unsigned long ); +unsigned long +__get_unmapped_area (struct file *file,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags,vm_flags_t vm_flags); + +static inline unsigned long +get_unmapped_area (struct file *file,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags) +{ +return __get_unmapped_area (file, addr, len, pgoff, flags,0 ); +} extern unsigned long mmap_region(struct file *file, unsigned long addr, unsigned long len, vm_flags_t vm_flags, unsigned long pgoff,
mm/mmap.c +16 -18
@@ -1257,18 +1257,6 @@ unsigned long do_mmap(struct file *file, unsigned long addr,if (mm->map_count > sysctl_max_map_count)return -ENOMEM ; -/* Obtain the address to map to. we verify (or select) it and ensure -* that it represents a valid section of the address space. -*/ - addr =get_unmapped_area (file, addr, len, pgoff, flags); -if (IS_ERR_VALUE (addr)) -return addr; - -if (flags &MAP_FIXED_NOREPLACE ) { -if (find_vma_intersection (mm, addr, addr + len)) -return -EEXIST ; - } -if (prot ==PROT_EXEC ) { pkey =execute_only_pkey (mm);if (pkey <0 )
@@ -1282,6 +1270,18 @@ unsigned long do_mmap(struct file *file, unsigned long addr, vm_flags |=calc_vm_prot_bits (prot, pkey) |calc_vm_flag_bits (flags) | mm->def_flags |VM_MAYREAD |VM_MAYWRITE |VM_MAYEXEC ; +/* Obtain the address to map to. we verify (or select) it and ensure +* that it represents a valid section of the address space. +*/ + addr =__get_unmapped_area (file, addr, len, pgoff, flags, vm_flags); +if (IS_ERR_VALUE (addr)) +return addr; + +if (flags &MAP_FIXED_NOREPLACE ) { +if (find_vma_intersection (mm, addr, addr + len)) +return -EEXIST ; + } +if (flags &MAP_LOCKED )if (!can_do_mlock ())return -EPERM ;
@@ -1839,8 +1839,8 @@ unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm, struct file *fi }unsigned long -get_unmapped_area (struct file *file,unsigned long addr,unsigned long len, -unsigned long pgoff,unsigned long flags) +__get_unmapped_area (struct file *file,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags,vm_flags_t vm_flags) {unsigned long (*get_area)(struct file *,unsigned long ,unsigned long ,unsigned long ,unsigned long )
@@ -1875,8 +1875,8 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,if (get_area) addr =get_area (file, addr, len, pgoff, flags);else - addr =mm_get_unmapped_area (current->mm, file, addr, len, - pgoff, flags); + addr =mm_get_unmapped_area_vmflags (current->mm, file, addr, len, + pgoff, flags, vm_flags);if (IS_ERR_VALUE (addr))return addr;
@@ -1889,8 +1889,6 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,return error ? error : addr; } -EXPORT_SYMBOL (get_unmapped_area); - unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *file, unsigned long addr, unsigned long len,
Signature · 1 line
2.34.1
Re: [PATCH v3 03/12] mm: Use get_unmapped_area_vmflags()
Le 12/03/2024 à 23:28, Rick Edgecombe a écrit :
Quoted 107 lines
When memory is being placed,
mmap()will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK,VM_GROWSUPandVM_GROWSDOWN). In order to ensure guard gaps between mappings,mmap()needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.The long standing behavior of
mmap()is to ensure 1, but not take any care around 2. So for example, if there is aPAGE_SIZEfree area, and ammap()with aPAGE_SIZEsize, and a type that has a guard gap is being placed,mmap()may place the shadow stack in thePAGE_SIZEfree area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.Use
mm_get_unmapped_area_vmflags()in thedo_mmap()so future changes can cause shadow stack mappings to be placed with a guard gap. Also use the THP variant that takesvm_flags, such that THP shadow stack can get the same treatment. Adjust thevm_flagscalculation to happen earlier so that thevm_flagscan be passed into__get_unmapped_area().Signed-off-by Rick Edgecombe <[email protected]>---
v2:
- Makeget_unmapped_area()a static inline (Kirill)--- include/linux/mm.h | 11 ++++++++++- mm/mmap.c | 34 ++++++++++++++++------------------ 2 files changed, 26 insertions(+), 19 deletions(-)include/linux/mm.h +10
-1@@ -3363,7 +3363,16 @@ extern int install_special_mapping(struct mm_struct *mm,unsigned long randomize_stack_top (unsigned long stack_top);unsigned long randomize_page (unsigned long start,unsigned long range); -extern unsigned long get_unmapped_area (struct file *,unsigned long ,unsigned long ,unsigned long ,unsigned long ); +unsigned long +__get_unmapped_area (struct file *file,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags,vm_flags_t vm_flags); + +static inline unsigned long +get_unmapped_area (struct file *file,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags) +{ +return __get_unmapped_area (file, addr, len, pgoff, flags,0 ); +} extern unsigned long mmap_region(struct file *file, unsigned long addr, unsigned long len, vm_flags_t vm_flags, unsigned long pgoff,mm/mmap.c +14
-14@@ -1257,18 +1257,6 @@ unsigned long do_mmap(struct file *file, unsigned long addr,if (mm->map_count > sysctl_max_map_count)return -ENOMEM ; -/* Obtain the address to map to. we verify (or select) it and ensure -* that it represents a valid section of the address space. -*/ - addr =get_unmapped_area (file, addr, len, pgoff, flags); -if (IS_ERR_VALUE (addr)) -return addr; - -if (flags &MAP_FIXED_NOREPLACE ) { -if (find_vma_intersection (mm, addr, addr + len)) -return -EEXIST ; - } -if (prot ==PROT_EXEC ) { pkey =execute_only_pkey (mm);if (pkey <0 )@@ -1282,6 +1270,18 @@ unsigned long do_mmap(struct file *file, unsigned long addr, vm_flags |=calc_vm_prot_bits (prot, pkey) |calc_vm_flag_bits (flags) | mm->def_flags |VM_MAYREAD |VM_MAYWRITE |VM_MAYEXEC ; +/* Obtain the address to map to. we verify (or select) it and ensure +* that it represents a valid section of the address space. +*/ + addr =__get_unmapped_area (file, addr, len, pgoff, flags, vm_flags); +if (IS_ERR_VALUE (addr)) +return addr; + +if (flags &MAP_FIXED_NOREPLACE ) { +if (find_vma_intersection (mm, addr, addr + len)) +return -EEXIST ; + } +if (flags &MAP_LOCKED )if (!can_do_mlock ())return -EPERM ;@@ -1839,8 +1839,8 @@ unsigned long mm_get_unmapped_area_vmflags(struct mm_struct *mm, struct file *fi }unsigned long -get_unmapped_area (struct file *file,unsigned long addr,unsigned long len, -unsigned long pgoff,unsigned long flags) +__get_unmapped_area (struct file *file,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags,vm_flags_t vm_flags) {unsigned long (*get_area)(struct file *,unsigned long ,unsigned long ,unsigned long ,unsigned long )
mm/mmap.c get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, · line 1876
Quoted 2 lines
mm/mmap.c +0
-0@@ -1875,8 +1875,8 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,if (get_area) addr =get_area (file, addr, len, pgoff, flags);
What about get_area(), what happens to vm_flags in case that function is used ?
Quoted 9 lines
mm/mmap.c +2
-2else - addr =mm_get_unmapped_area (current->mm, file, addr, len, - pgoff, flags); + addr =mm_get_unmapped_area_vmflags (current->mm, file, addr, len, + pgoff, flags, vm_flags);if (IS_ERR_VALUE (addr))return addr;
mm/mmap.c get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, · line 1893
Quoted 5 lines
mm/mmap.c +0
-2@@ -1889,8 +1889,6 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,return error ? error : addr; } -EXPORT_SYMBOL (get_unmapped_area); -
Don't you have to export __get_unmapped_area() so that the static inline get_unmapped_area() used in a module have access to __get_unmapped_area() ?
If it was pointless to export get_unmapped_area(), its removal should be a separate patch.
Quoted 3 lines
mm/mmap.c +0
-0unsigned long mm_get_unmapped_area(struct mm_struct *mm, struct file *file, unsigned long addr, unsigned long len,
Re: [PATCH v3 03/12] mm: Use get_unmapped_area_vmflags()
Quoted 4 lines · Christophe Leroy
What about
get_area(), what happens tovm_flagsin case that function is
used ?
Shadow stack can only be private and anonymous memory. So this way is to avoid unnecessary plumbing. And that plumbing becomes quite extensive when you start intersecting with the f_ops->get_unmapped_area(), so its pretty fortunate.
Christophe Leroy
Reviewed-byRe: [PATCH v3 03/12] mm: Use get_unmapped_area_vmflags()
Le 13/03/2024 à 15:55, Edgecombe, Rick P a écrit :
Quoted 10 lines · Christophe Leroy
Quoted 4 lines
What about
get_area(), what happens tovm_flagsin case that function is
used ?Shadow stack can only be private and anonymous memory. So this way is to avoid unnecessary plumbing. And that plumbing becomes quite extensive when you start intersecting with the
f_ops->get_unmapped_area(), so its pretty fortunate.
Ok,
Re: [PATCH v3 03/12] mm: Use get_unmapped_area_vmflags()
In reply to Christophe LeroyQuoted 11 lines · Christophe Leroy
mm/mmap.c get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, · line 1893
Quoted 2 lines
mm/mmap.c +0
-2-EXPORT_SYMBOL (get_unmapped_area); -Don't you have to export
__get_unmapped_area()so that the static inlineget_unmapped_area()used in a module have access to__get_unmapped_area()?If it was pointless to export
get_unmapped_area(), its removal should be
a separate patch.
Yes, it seems to have been pointless. I can split it out. Thanks.
PATCH v3 4/12 thp: Add thp_get_unmapped_area_vmflags()
In reply to Rick EdgecombeWhen memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.
The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.
Add a THP implementations of the vm_flags variant of get_unmapped_area(). Future changes will call this from mmap.c in the do_mmap() path to allow shadow stacks to be placed with consideration taken for the start guard gap. Shadow stack memory is always private and anonymous and so special guard gap logic is not needed in a lot of caseis, but it can be mapped by THP, so needs to be handled.
--- include/linux/huge_mm.h | 11 +++++++++++ mm/huge_memory.c | 23 ++++++++++++++++------- mm/mmap.c | 12 +++++++----- 3 files changed, 34 insertions(+), 12 deletions(-)
include/linux/huge_mm.h +11 -0
@@ -262,6 +262,9 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,unsigned long thp_get_unmapped_area (struct file *filp,unsigned long addr,unsigned long len,unsigned long pgoff,unsigned long flags); +unsigned long thp_get_unmapped_area_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff,unsigned long flags, +vm_flags_t vm_flags);void folio_prep_large_rmappable (struct folio *folio);bool can_split_folio (struct folio *folio,int *pextra_pins);
@@ -416,6 +419,14 @@ static inline void folio_prep_large_rmappable(struct folio *folio) {}#define thp_get_unmapped_area NULL +static inline unsigned long +thp_get_unmapped_area_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t vm_flags) +{ +return 0 ; +} +static inline bool can_split_folio (struct folio *folio,int *pextra_pins) {
mm/huge_memory.c +16 -7
@@ -806,7 +806,8 @@ static inline bool is_transparent_hugepage(struct folio *folio)static unsigned long __thp_get_unmapped_area (struct file *filp,unsigned long addr,unsigned long len, -loff_t off,unsigned long flags,unsigned long size) +loff_t off,unsigned long flags,unsigned long size, +vm_flags_t vm_flags) {loff_t off_end = off + len;loff_t off_align =round_up (off, size);
@@ -822,8 +823,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp,if (len_pad < len || (off + len_pad) < off)return 0 ; - ret =mm_get_unmapped_area (current->mm, filp, addr, len_pad, - off >>PAGE_SHIFT , flags); + ret =mm_get_unmapped_area_vmflags (current->mm, filp, addr, len_pad, + off >>PAGE_SHIFT , flags, vm_flags);/* * The failure might be due to length padding. The caller will retry
@@ -848,17 +849,25 @@ static unsigned long __thp_get_unmapped_area(struct file *filp,return ret; } -unsigned long thp_get_unmapped_area (struct file *filp,unsigned long addr, -unsigned long len,unsigned long pgoff,unsigned long flags) +unsigned long thp_get_unmapped_area_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff,unsigned long flags, +vm_flags_t vm_flags) {unsigned long ret;loff_t off = (loff_t )pgoff <<PAGE_SHIFT ; - ret =__thp_get_unmapped_area (filp, addr, len, off, flags,PMD_SIZE ); + ret =__thp_get_unmapped_area (filp, addr, len, off, flags,PMD_SIZE , vm_flags);if (ret)return ret; -return mm_get_unmapped_area (current->mm, filp, addr, len, pgoff, flags); +return mm_get_unmapped_area_vmflags (current->mm, filp, addr, len, pgoff, flags, + vm_flags); +} + +unsigned long thp_get_unmapped_area (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff,unsigned long flags) +{ +return thp_get_unmapped_area_vmflags (filp, addr, len, pgoff, flags,0 ); }EXPORT_SYMBOL_GPL (thp_get_unmapped_area);
mm/mmap.c +7 -5
@@ -1863,20 +1863,22 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,* so use shmem's get_unmapped_area in case it can be huge. */ get_area = shmem_get_unmapped_area; - }else if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { -/* Ensures that larger anonymous mappings are THP aligned. */ - get_area = thp_get_unmapped_area; }/* Always treat pgoff as zero for anonymous memory. */ if (!file) pgoff =0 ; -if (get_area) +if (get_area) { addr =get_area (file, addr, len, pgoff, flags); -else + }else if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { +/* Ensures that larger anonymous mappings are THP aligned. */ + addr =thp_get_unmapped_area_vmflags (file, addr, len, + pgoff, flags, vm_flags); + }else { addr =mm_get_unmapped_area_vmflags (current->mm, file, addr, len, pgoff, flags, vm_flags); + }if (IS_ERR_VALUE (addr))return addr;
Signature · 1 line
2.34.1
Re: [PATCH v3 04/12] thp: Add thp_get_unmapped_area_vmflags()
Le 12/03/2024 à 23:28, Rick Edgecombe a écrit :
Quoted 16 lines
When memory is being placed,
mmap()will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK,VM_GROWSUPandVM_GROWSDOWN). In order to ensure guard gaps between mappings,mmap()needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.The long standing behavior of
mmap()is to ensure 1, but not take any care around 2. So for example, if there is aPAGE_SIZEfree area, and ammap()with aPAGE_SIZEsize, and a type that has a guard gap is being placed,mmap()may place the shadow stack in thePAGE_SIZEfree area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.
Quoted 8 lines
Add a THP implementations of the
vm_flagsvariant ofget_unmapped_area(). Future changes will call this frommmap.cin thedo_mmap()path to allow shadow stacks to be placed with consideration taken for the start guard gap. Shadow stack memory is always private and anonymous and so special guard gap logic is not needed in a lot of caseis, but it can be mapped by THP, so needs to be handled.Signed-off-by Rick Edgecombe <[email protected]>
Quoted 122 lines
--- include/linux/huge_mm.h | 11 +++++++++++ mm/huge_memory.c | 23 ++++++++++++++++------- mm/mmap.c | 12 +++++++----- 3 files changed, 34 insertions(+), 12 deletions(-)include/linux/huge_mm.h +11
-0@@ -262,6 +262,9 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,unsigned long thp_get_unmapped_area (struct file *filp,unsigned long addr,unsigned long len,unsigned long pgoff,unsigned long flags); +unsigned long thp_get_unmapped_area_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff,unsigned long flags, +vm_flags_t vm_flags);void folio_prep_large_rmappable (struct folio *folio);bool can_split_folio (struct folio *folio,int *pextra_pins);@@ -416,6 +419,14 @@ static inline void folio_prep_large_rmappable(struct folio *folio) {}#define thp_get_unmapped_area NULL +static inline unsigned long +thp_get_unmapped_area_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t vm_flags) +{ +return 0 ; +} +static inline bool can_split_folio (struct folio *folio,int *pextra_pins) {mm/huge_memory.c +16
-7@@ -806,7 +806,8 @@ static inline bool is_transparent_hugepage(struct folio *folio)static unsigned long __thp_get_unmapped_area (struct file *filp,unsigned long addr,unsigned long len, -loff_t off,unsigned long flags,unsigned long size) +loff_t off,unsigned long flags,unsigned long size, +vm_flags_t vm_flags) {loff_t off_end = off + len;loff_t off_align =round_up (off, size);@@ -822,8 +823,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp,if (len_pad < len || (off + len_pad) < off)return 0 ; - ret =mm_get_unmapped_area (current->mm, filp, addr, len_pad, - off >>PAGE_SHIFT , flags); + ret =mm_get_unmapped_area_vmflags (current->mm, filp, addr, len_pad, + off >>PAGE_SHIFT , flags, vm_flags);/* * The failure might be due to length padding. The caller will retry @@ -848,17 +849,25 @@ static unsigned long __thp_get_unmapped_area(struct file *filp,return ret; } -unsigned long thp_get_unmapped_area (struct file *filp,unsigned long addr, -unsigned long len,unsigned long pgoff,unsigned long flags) +unsigned long thp_get_unmapped_area_vmflags (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff,unsigned long flags, +vm_flags_t vm_flags) {unsigned long ret;loff_t off = (loff_t )pgoff <<PAGE_SHIFT ; - ret =__thp_get_unmapped_area (filp, addr, len, off, flags,PMD_SIZE ); + ret =__thp_get_unmapped_area (filp, addr, len, off, flags,PMD_SIZE , vm_flags);if (ret)return ret; -return mm_get_unmapped_area (current->mm, filp, addr, len, pgoff, flags); +return mm_get_unmapped_area_vmflags (current->mm, filp, addr, len, pgoff, flags, + vm_flags); +} + +unsigned long thp_get_unmapped_area (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff,unsigned long flags) +{ +return thp_get_unmapped_area_vmflags (filp, addr, len, pgoff, flags,0 ); }EXPORT_SYMBOL_GPL (thp_get_unmapped_area);mm/mmap.c +7
-5@@ -1863,20 +1863,22 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len,* so use shmem's get_unmapped_area in case it can be huge. */ get_area = shmem_get_unmapped_area; - }else if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { -/* Ensures that larger anonymous mappings are THP aligned. */ - get_area = thp_get_unmapped_area; }/* Always treat pgoff as zero for anonymous memory. */ if (!file) pgoff =0 ; -if (get_area) +if (get_area) { addr =get_area (file, addr, len, pgoff, flags); -else + }else if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { +/* Ensures that larger anonymous mappings are THP aligned. */ + addr =thp_get_unmapped_area_vmflags (file, addr, len, + pgoff, flags, vm_flags); + }else { addr =mm_get_unmapped_area_vmflags (current->mm, file, addr, len, pgoff, flags, vm_flags); + }if (IS_ERR_VALUE (addr))return addr;
PATCH v3 5/12 csky: Use initializer for struct vm_unmapped_area_info
In reply to Rick EdgecombeFuture changes will need to add a new member to struct vm_unmapped_area_info. This would cause trouble for any call site that doesn't initialize the struct. Currently every caller sets each member manually, so if new members are added they will be uninitialized and the core code parsing the struct will see garbage in the new member.
It could be possible to initialize the new member manually to 0 at each call site. This and a couple other options were discussed, and a working consensus (see links) was that in general the best way to accomplish this would be via static initialization with designated member initiators. Having some struct vm_unmapped_area_info instances not zero initialized will put those sites at risk of feeding garbage into vm_unmapped_area() if the convention is to zero initialize the struct and any new member addition misses a call site that initializes each member manually.
It could be possible to leave the code mostly untouched, and just change the line:
struct vm_unmapped_area_info info
to:
struct vm_unmapped_area_info info = {};
However, that would leave cleanup for the members that are manually set to zero, as it would no longer be required.
So to be reduce the chance of bugs via uninitialized members, instead simply continue the process to initialize the struct this way tree wide. This will zero any unspecified members. Move the member initializers to the struct declaration when they are known at that time. Leave the members out that were manually initialized to zero, as this would be redundant for designated initializers.
---
v3:
- Fixed spelling errors in log
- Be consistent about field vs member in log
Hi,
This patch was split and refactored out of a tree-wide change [0] to just zero-init each struct vm_unmapped_area_info. The overall goal of the series is to help shadow stack guard gaps. Currently, there is only one arch with shadow stacks, but two more are in progress. It is compile tested only.
There was further discussion that this method of initializing the structs while nice in some ways has a greater risk of introducing bugs in some of the more complicated callers. Since this version was reviewed my arch maintainers already, leave it as was already acknowledged.
Thanks,
Rick
[0] https://lore.kernel.org/lkml/[email protected]/
--- arch/csky/abiv1/mmap.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
arch/csky/abiv1/mmap.c +6 -6
@@ -28,7 +28,12 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,struct mm_struct *mm = current->mm;struct vm_area_struct *vma;int do_align =0 ; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = { + .length = len, + .low_limit = mm->mmap_base, + .high_limit =TASK_SIZE , + .align_offset = pgoff <<PAGE_SHIFT + };/* * We only need to do colour alignment if either the I or D
@@ -61,11 +66,6 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,return addr; } - info.flags =0 ; - info.length = len; - info.low_limit = mm->mmap_base; - info.high_limit =TASK_SIZE ; info.align_mask = do_align ? (PAGE_MASK & (SHMLBA -1 )) :0 ; - info.align_offset = pgoff <<PAGE_SHIFT ;return vm_unmapped_area (&info); }
Signature · 1 line
2.34.1
Re: [PATCH v3 05/12] csky: Use initializer for struct vm_unmapped_area_info
Le 12/03/2024 à 23:28, Rick Edgecombe a écrit :
Quoted 31 lines
Future changes will need to add a new member to struct
vm_unmapped_area_info. This would cause trouble for any call site that doesn't initialize the struct. Currently every caller sets each member manually, so if new members are added they will be uninitialized and the core code parsing the struct will see garbage in the new member.It could be possible to initialize the new member manually to 0 at each call site. This and a couple other options were discussed, and a working consensus (see links) was that in general the best way to accomplish this would be via static initialization with designated member initiators. Having some struct
vm_unmapped_area_infoinstances not zero initialized will put those sites at risk of feeding garbage intovm_unmapped_area()if the convention is to zero initialize the struct and any new member addition misses a call site that initializes each member manually.It could be possible to leave the code mostly untouched, and just change the line:
structvm_unmapped_area_infoinfo
to:
structvm_unmapped_area_infoinfo = {};However, that would leave cleanup for the members that are manually set to zero, as it would no longer be required.
So to be reduce the chance of bugs via uninitialized members, instead simply continue the process to initialize the struct this way tree wide. This will zero any unspecified members. Move the member initializers to the struct declaration when they are known at that time. Leave the members out that were manually initialized to zero, as this would be redundant for designated initializers.
Quoted 2 lines
Signed-off-by Rick Edgecombe <[email protected]>Reviewed-by Guo Ren <[email protected]>
Quoted 36 lines
Cc Guo Ren <[email protected]>---
v3:
- Fixed spelling errors in log
- Be consistent about field vs member in logHi,
This patch was split and refactored out of a tree-wide change [0] to just zero-init each struct
vm_unmapped_area_info. The overall goal of the series is to help shadow stack guard gaps. Currently, there is only one arch with shadow stacks, but two more are in progress. It is compile tested only.There was further discussion that this method of initializing the structs while nice in some ways has a greater risk of introducing bugs in some of the more complicated callers. Since this version was reviewed my arch maintainers already, leave it as was already acknowledged.
Thanks,
Rick
[0] https://lore.kernel.org/lkml/[email protected]/
--- arch/csky/abiv1/mmap.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)arch/csky/abiv1/mmap.c +0
-0
arch/csky/abiv1/mmap.c arch_get_unmapped_area(struct file *filp, unsigned long addr, · line 31
Quoted 5 lines
arch/csky/abiv1/mmap.c +1
-1@@ -28,7 +28,12 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,struct mm_struct *mm = current->mm;struct vm_area_struct *vma;int do_align =0 ; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {
I see you didn't had .flags = 0, which is good. Wondering why you didn't do the same for powerpc.
arch/csky/abiv1/mmap.c arch_get_unmapped_area(struct file *filp, unsigned long addr, · line 35
Quoted 4 lines
arch/csky/abiv1/mmap.c +4
-0+ .length = len, + .low_limit = mm->mmap_base, + .high_limit =TASK_SIZE , + .align_offset = pgoff << PAGE_SHIFT
Usually we leave the comma at the end of the last element so that the day you add a new element you don't have to change an existing line just to add a comma.
Quoted 16 lines
arch/csky/abiv1/mmap.c +1
-5+ };/* * We only need to do colour alignment if either the I or D @@ -61,11 +66,6 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,return addr; } - info.flags =0 ; - info.length = len; - info.low_limit = mm->mmap_base; - info.high_limit =TASK_SIZE ; info.align_mask = do_align ? (PAGE_MASK & (SHMLBA -1 )) :0 ; - info.align_offset = pgoff <<PAGE_SHIFT ;return vm_unmapped_area (&info); }
PATCH v3 6/12 parisc: Use initializer for struct vm_unmapped_area_info
In reply to Rick EdgecombeFuture changes will need to add a new member to struct vm_unmapped_area_info. This would cause trouble for any call site that doesn't initialize the struct. Currently every caller sets each member manually, so if new members are added they will be uninitialized and the core code parsing the struct will see garbage in the new member.
It could be possible to initialize the new member manually to 0 at each call site. This and a couple other options were discussed, and a working consensus (see links) was that in general the best way to accomplish this would be via static initialization with designated member initiators. Having some struct vm_unmapped_area_info instances not zero initialized will put those sites at risk of feeding garbage into vm_unmapped_area() if the convention is to zero initialize the struct and any new member addition misses a call site that initializes each member manually.
It could be possible to leave the code mostly untouched, and just change the line:
struct vm_unmapped_area_info info
to:
struct vm_unmapped_area_info info = {};
However, that would leave cleanup for the members that are manually set to zero, as it would no longer be required.
So to be reduce the chance of bugs via uninitialized members, instead simply continue the process to initialize the struct this way tree wide. This will zero any unspecified members. Move the member initializers to the struct declaration when they are known at that time. Leave the members out that were manually initialized to zero, as this would be redundant for designated initializers.
---
v3:
- Fixed spelling errors in log
- Be consistent about field vs member in log
Hi,
This patch was split and refactored out of a tree-wide change [0] to just zero-init each struct vm_unmapped_area_info. The overall goal of the series is to help shadow stack guard gaps. Currently, there is only one arch with shadow stacks, but two more are in progress. It is compile tested only.
There was further discussion that this method of initializing the structs while nice in some ways has a greater risk of introducing bugs in some of the more complicated callers. Since this version was reviewed my arch maintainers already, leave it as was already acknowledged.
Thanks,
Rick
[0] https://lore.kernel.org/lkml/[email protected]/
--- arch/parisc/kernel/sys_parisc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/parisc/kernel/sys_parisc.c +3 -3
@@ -104,7 +104,9 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,struct vm_area_struct *vma, *prev;unsigned long filp_pgoff;int do_color_align; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = { + .length = len + };if (unlikely (len >TASK_SIZE ))return -ENOMEM ;
@@ -139,7 +141,6 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,return addr; } - info.length = len; info.align_mask = do_color_align ? (PAGE_MASK & (SHM_COLOUR -1 )) :0 ; info.align_offset =shared_align_offset (filp_pgoff, pgoff);
@@ -160,7 +161,6 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,*/ } - info.flags =0 ; info.low_limit = mm->mmap_base; info.high_limit =mmap_upper_limit (NULL );return vm_unmapped_area (&info);
Signature · 1 line
2.34.1
Re: [PATCH v3 06/12] parisc: Use initializer for struct vm_unmapped_area_info
Le 12/03/2024 à 23:28, Rick Edgecombe a écrit :
Quoted 24 lines
Future changes will need to add a new member to struct
vm_unmapped_area_info. This would cause trouble for any call site that doesn't initialize the struct. Currently every caller sets each member manually, so if new members are added they will be uninitialized and the core code parsing the struct will see garbage in the new member.It could be possible to initialize the new member manually to 0 at each call site. This and a couple other options were discussed, and a working consensus (see links) was that in general the best way to accomplish this would be via static initialization with designated member initiators. Having some struct
vm_unmapped_area_infoinstances not zero initialized will put those sites at risk of feeding garbage intovm_unmapped_area()if the convention is to zero initialize the struct and any new member addition misses a call site that initializes each member manually.It could be possible to leave the code mostly untouched, and just change the line:
structvm_unmapped_area_infoinfo
to:
structvm_unmapped_area_infoinfo = {};However, that would leave cleanup for the members that are manually set to zero, as it would no longer be required.
Quoted 8 lines
So to be reduce the chance of bugs via uninitialized members, instead simply continue the process to initialize the struct this way tree wide. This will zero any unspecified members. Move the member initializers to the struct declaration when they are known at that time. Leave the members out that were manually initialized to zero, as this would be redundant for designated initializers.
Signed-off-by Rick Edgecombe <[email protected]>
Quoted 64 lines
Acked-by Helge Deller <[email protected]>Cc "James E.J. Bottomley" <[email protected]>Cc Helge Deller <[email protected]>---
v3:
- Fixed spelling errors in log
- Be consistent about field vs member in logHi,
This patch was split and refactored out of a tree-wide change [0] to just zero-init each struct
vm_unmapped_area_info. The overall goal of the series is to help shadow stack guard gaps. Currently, there is only one arch with shadow stacks, but two more are in progress. It is compile tested only.There was further discussion that this method of initializing the structs while nice in some ways has a greater risk of introducing bugs in some of the more complicated callers. Since this version was reviewed my arch maintainers already, leave it as was already acknowledged.
Thanks,
Rick
[0] https://lore.kernel.org/lkml/[email protected]/
--- arch/parisc/kernel/sys_parisc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)arch/parisc/kernel/sys_parisc.c +3
-3@@ -104,7 +104,9 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,struct vm_area_struct *vma, *prev;unsigned long filp_pgoff;int do_color_align; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = { + .length = len + };if (unlikely (len >TASK_SIZE ))return -ENOMEM ;@@ -139,7 +141,6 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,return addr; } - info.length = len; info.align_mask = do_color_align ? (PAGE_MASK & (SHM_COLOUR -1 )) :0 ; info.align_offset =shared_align_offset (filp_pgoff, pgoff);@@ -160,7 +161,6 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,*/ } - info.flags =0 ; info.low_limit = mm->mmap_base; info.high_limit =mmap_upper_limit (NULL );return vm_unmapped_area (&info);
PATCH v3 7/12 powerpc: Use initializer for struct vm_unmapped_area_info
In reply to Rick EdgecombeFuture changes will need to add a new member to struct vm_unmapped_area_info. This would cause trouble for any call site that doesn't initialize the struct. Currently every caller sets each member manually, so if new members are added they will be uninitialized and the core code parsing the struct will see garbage in the new member.
It could be possible to initialize the new member manually to 0 at each call site. This and a couple other options were discussed, and a working consensus (see links) was that in general the best way to accomplish this would be via static initialization with designated member initiators. Having some struct vm_unmapped_area_info instances not zero initialized will put those sites at risk of feeding garbage into vm_unmapped_area() if the convention is to zero initialize the struct and any new member addition misses a call site that initializes each member manually.
It could be possible to leave the code mostly untouched, and just change the line:
struct vm_unmapped_area_info info
to:
struct vm_unmapped_area_info info = {};
However, that would leave cleanup for the members that are manually set to zero, as it would no longer be required.
So to be reduce the chance of bugs via uninitialized members, instead simply continue the process to initialize the struct this way tree wide. This will zero any unspecified members. Move the member initializers to the struct declaration when they are known at that time. Leave the members out that were manually initialized to zero, as this would be redundant for designated initializers.
---
v3:
- Fixed spelling errors in log
- Be consistent about field vs member in log
Hi,
This patch was split and refactored out of a tree-wide change [0] to just zero-init each struct vm_unmapped_area_info. The overall goal of the series is to help shadow stack guard gaps. Currently, there is only one arch with shadow stacks, but two more are in progress. It is compile tested only.
There was further discussion that this method of initializing the structs while nice in some ways has a greater risk of introducing bugs in some of the more complicated callers. Since this version was reviewed my arch maintainers already, leave it as was already acknowledged.
Thanks,
Rick
[0] https://lore.kernel.org/lkml/[email protected]/
--- arch/powerpc/mm/book3s64/slice.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)
arch/powerpc/mm/book3s64/slice.c +12 -11
@@ -282,12 +282,12 @@ static unsigned long slice_find_area_bottomup(struct mm_struct *mm, {int pshift =max_t (int, mmu_psize_defs[psize].shift,PAGE_SHIFT );unsigned long found, next_end; -struct vm_unmapped_area_info info; - - info.flags =0 ; - info.length = len; - info.align_mask =PAGE_MASK & ((1ul << pshift) -1 ); - info.align_offset =0 ; +struct vm_unmapped_area_info info = { + .flags =0 , + .length = len, + .align_mask =PAGE_MASK & ((1ul << pshift) -1 ), + .align_offset =0 + };/* * Check till the allow max value for this mmap request */
@@ -326,13 +326,14 @@ static unsigned long slice_find_area_topdown(struct mm_struct *mm, {int pshift =max_t (int, mmu_psize_defs[psize].shift,PAGE_SHIFT );unsigned long found, prev; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = { + .flags =VM_UNMAPPED_AREA_TOPDOWN , + .length = len, + .align_mask =PAGE_MASK & ((1ul << pshift) -1 ), + .align_offset =0 + };unsigned long min_addr =max (PAGE_SIZE , mmap_min_addr); - info.flags =VM_UNMAPPED_AREA_TOPDOWN ; - info.length = len; - info.align_mask =PAGE_MASK & ((1ul << pshift) -1 ); - info.align_offset =0 ;/* * If we are trying to allocate above DEFAULT_MAP_WINDOW * Add the different to the mmap_base.
Signature · 1 line
2.34.1
Re: [PATCH v3 07/12] powerpc: Use initializer for struct vm_unmapped_area_info
Le 12/03/2024 à 23:28, Rick Edgecombe a écrit :
Quoted 24 lines
Future changes will need to add a new member to struct
vm_unmapped_area_info. This would cause trouble for any call site that doesn't initialize the struct. Currently every caller sets each member manually, so if new members are added they will be uninitialized and the core code parsing the struct will see garbage in the new member.It could be possible to initialize the new member manually to 0 at each call site. This and a couple other options were discussed, and a working consensus (see links) was that in general the best way to accomplish this would be via static initialization with designated member initiators. Having some struct
vm_unmapped_area_infoinstances not zero initialized will put those sites at risk of feeding garbage intovm_unmapped_area()if the convention is to zero initialize the struct and any new member addition misses a call site that initializes each member manually.It could be possible to leave the code mostly untouched, and just change the line:
structvm_unmapped_area_infoinfo
to:
structvm_unmapped_area_infoinfo = {};However, that would leave cleanup for the members that are manually set to zero, as it would no longer be required.
Quoted 6 lines
So to be reduce the chance of bugs via uninitialized members, instead simply continue the process to initialize the struct this way tree wide. This will zero any unspecified members. Move the member initializers to the struct declaration when they are known at that time. Leave the members out that were manually initialized to zero, as this would be redundant for designated initializers.
I understand from this text that, as agreed, this patch removes the pointless/redundant zero-init of individual members. But it is not what is done, see below ?
Quoted 44 lines
Signed-off-by Rick Edgecombe <[email protected]>Acked-by Michael Ellerman <[email protected]>Cc Michael Ellerman <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Christophe Leroy <[email protected]>Cc Aneesh Kumar K.V <[email protected]>Cc Naveen N. Rao <[email protected]>---
v3:
- Fixed spelling errors in log
- Be consistent about field vs member in logHi,
This patch was split and refactored out of a tree-wide change [0] to just zero-init each struct
vm_unmapped_area_info. The overall goal of the series is to help shadow stack guard gaps. Currently, there is only one arch with shadow stacks, but two more are in progress. It is compile tested only.There was further discussion that this method of initializing the structs while nice in some ways has a greater risk of introducing bugs in some of the more complicated callers. Since this version was reviewed my arch maintainers already, leave it as was already acknowledged.
Thanks,
Rick
[0] https://lore.kernel.org/lkml/[email protected]/
--- arch/powerpc/mm/book3s64/slice.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-)arch/powerpc/mm/book3s64/slice.c +0
-0@@ -282,12 +282,12 @@ static unsigned long slice_find_area_bottomup(struct mm_struct *mm, {
arch/powerpc/mm/book3s64/slice.c static unsigned long slice_find_area_bottomup(struct mm_struct *mm, · line 286
Quoted 10 lines
arch/powerpc/mm/book3s64/slice.c +2
-6int pshift =max_t (int, mmu_psize_defs[psize].shift,PAGE_SHIFT );unsigned long found, next_end; -struct vm_unmapped_area_info info; - - info.flags =0 ; - info.length = len; - info.align_mask =PAGE_MASK & ((1ul << pshift) -1 ); - info.align_offset =0 ; +struct vm_unmapped_area_info info = { + .flags =0 ,
Please remove zero-init as agreed and explained in the commit message
arch/powerpc/mm/book3s64/slice.c static unsigned long slice_find_area_bottomup(struct mm_struct *mm, · line 289
Quoted 3 lines
arch/powerpc/mm/book3s64/slice.c +3
-0+ .length = len, + .align_mask =PAGE_MASK & ((1ul << pshift) -1 ), + .align_offset = 0
Same here.
Quoted 5 lines
arch/powerpc/mm/book3s64/slice.c +1
-0+ };/* * Check till the allow max value for this mmap request */
arch/powerpc/mm/book3s64/slice.c static unsigned long slice_find_area_topdown(struct mm_struct *mm, · line 333
Quoted 9 lines
arch/powerpc/mm/book3s64/slice.c +5
-1@@ -326,13 +326,14 @@ static unsigned long slice_find_area_topdown(struct mm_struct *mm, {int pshift =max_t (int, mmu_psize_defs[psize].shift,PAGE_SHIFT );unsigned long found, prev; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = { + .flags =VM_UNMAPPED_AREA_TOPDOWN , + .length = len, + .align_mask =PAGE_MASK & ((1ul << pshift) -1 ), + .align_offset =0
Same here.
arch/powerpc/mm/book3s64/slice.c static unsigned long slice_find_area_topdown(struct mm_struct *mm, · line 339
Quoted 10 lines
arch/powerpc/mm/book3s64/slice.c +1
-4+ };unsigned long min_addr =max (PAGE_SIZE , mmap_min_addr); - info.flags =VM_UNMAPPED_AREA_TOPDOWN ; - info.length = len; - info.align_mask =PAGE_MASK & ((1ul << pshift) -1 ); - info.align_offset =0 ;/* * If we are trying to allocate above DEFAULT_MAP_WINDOW * Add the different to the mmap_base.
Christophe
Re: [PATCH v3 07/12] powerpc: Use initializer for struct vm_unmapped_area_info
Quoted 4 lines · Christophe Leroy
I understand from this text that, as agreed, this patch removes the pointless/redundant zero-init of individual members. But it is not what
is done, see below ?
Err, right. I think I decided to leave it because it was already acked and there wasn't enough discussion on the ack to be sure. I will update it.
Michael Ellerman
reviewerRe: [PATCH v3 07/12] powerpc: Use initializer for struct vm_unmapped_area_info
Quoted 9 lines · Rick P
Quoted 4 lines · Christophe Leroy
I understand from this text that, as agreed, this patch removes the pointless/redundant zero-init of individual members. But it is not what
is done, see below ?Err, right. I think I decided to leave it because it was already acked and there wasn't enough discussion on the ack to be sure. I will update it.
That's fine by me, you can keep my ack.
cheers
PATCH v3 8/12 treewide: Use initializer for struct vm_unmapped_area_info
In reply to Rick EdgecombeFuture changes will need to add a new member to struct vm_unmapped_area_info. This would cause trouble for any call site that doesn't initialize the struct. Currently every caller sets each member manually, so if new ones are added they will be uninitialized and the core code parsing the struct will see garbage in the new member.
It could be possible to initialize the new member manually to 0 at each call site. This and a couple other options were discussed. Having some struct vm_unmapped_area_info instances not zero initialized will put those sites at risk of feeding garbage into vm_unmapped_area(), if the convention is to zero initialize the struct and any new field addition missed a call site that initializes each field manually. So it is useful to do things similar across the kernel.
The consensus (see links) was that in general the best way to accomplish taking into account both code cleanliness and minimizing the chance of introducing bugs, was to do C99 static initialization. As in:
struct vm_unmapped_area_info info = {};
With this method of initialization, the whole struct will be zero initialized, and any statements setting fields to zero will be unneeded. The change should not leave cleanup at the call sides.
While iterating though the possible solutions a few archs kindly acked other variations that still zero initialized the struct. These sites have been modified in previous changes using the pattern acked by the respective arch.
So to be reduce the chance of bugs via uninitialized fields, perform a tree wide change using the consensus for the best general way to do this change. Use C99 static initializing to zero the struct and remove and statements that simply set members to zero.
---
Hi archs,
For some context, this is part of a larger series to improve shadow stack guard gaps. It involves plumbing a new field via
struct vm_unmapped_area_info. The first user is x86, but arm and riscv may likely use it as well. The change is compile tested only for non-x86.
Thanks,
Rick
--- arch/alpha/kernel/osf_sys.c | 5 +---- arch/arc/mm/mmap.c | 4 +--- arch/arm/mm/mmap.c | 5 ++--- arch/loongarch/mm/mmap.c | 3 +-- arch/mips/mm/mmap.c | 3 +-- arch/s390/mm/hugetlbpage.c | 7 ++----- arch/s390/mm/mmap.c | 11 ++++------- arch/sh/mm/mmap.c | 5 ++--- arch/sparc/kernel/sys_sparc_32.c | 3 +-- arch/sparc/kernel/sys_sparc_64.c | 5 ++--- arch/sparc/mm/hugetlbpage.c | 7 ++----- arch/x86/kernel/sys_x86_64.c | 7 ++----- arch/x86/mm/hugetlbpage.c | 7 ++----- fs/hugetlbfs/inode.c | 7 ++----- mm/mmap.c | 9 ++------- 15 files changed, 27 insertions(+), 61 deletions(-)
arch/alpha/kernel/osf_sys.c +1 -4
@@ -1218,14 +1218,11 @@ static unsigned long arch_get_unmapped_area_1(unsigned long addr, unsigned long len, unsigned long limit) { -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {}; - info.flags =0 ; info.length = len; info.low_limit = addr; info.high_limit = limit; - info.align_mask =0 ; - info.align_offset =0 ;return vm_unmapped_area (&info); }
arch/arc/mm/mmap.c +1 -3
@@ -27,7 +27,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};/* * We enforce the MAP_FIXED case.
@@ -51,11 +51,9 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,return addr; } - info.flags =0 ; info.length = len; info.low_limit = mm->mmap_base; info.high_limit =TASK_SIZE ; - info.align_mask =0 ; info.align_offset = pgoff <<PAGE_SHIFT ;return vm_unmapped_area (&info); }
arch/arm/mm/mmap.c +2 -3
@@ -34,7 +34,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,struct vm_area_struct *vma;int do_align =0 ;int aliasing =cache_is_vipt_aliasing (); -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};/* * We only need to do colour alignment if either the I or D
@@ -68,7 +68,6 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,return addr; } - info.flags =0 ; info.length = len; info.low_limit = mm->mmap_base; info.high_limit =TASK_SIZE ;
@@ -87,7 +86,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,unsigned long addr = addr0;int do_align =0 ;int aliasing =cache_is_vipt_aliasing (); -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};/* * We only need to do colour alignment if either the I or D
arch/loongarch/mm/mmap.c +1 -2
@@ -24,7 +24,7 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,struct vm_area_struct *vma;unsigned long addr = addr0;int do_color_align; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};if (unlikely (len >TASK_SIZE ))return -ENOMEM ;
@@ -82,7 +82,6 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,*/ } - info.flags =0 ; info.low_limit = mm->mmap_base; info.high_limit =TASK_SIZE ;return vm_unmapped_area (&info);
arch/mips/mm/mmap.c +1 -2
@@ -34,7 +34,7 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,struct vm_area_struct *vma;unsigned long addr = addr0;int do_color_align; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};if (unlikely (len >TASK_SIZE ))return -ENOMEM ;
@@ -92,7 +92,6 @@ static unsigned long arch_get_unmapped_area_common(struct file *filp,*/ } - info.flags =0 ; info.low_limit = mm->mmap_base; info.high_limit =TASK_SIZE ;return vm_unmapped_area (&info);
arch/s390/mm/hugetlbpage.c +2 -5
@@ -258,14 +258,12 @@ static unsigned long hugetlb_get_unmapped_area_bottomup(struct file *file, unsigned long pgoff, unsigned long flags) {struct hstate *h =hstate_file (file); -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {}; - info.flags =0 ; info.length = len; info.low_limit = current->mm->mmap_base; info.high_limit =TASK_SIZE ; info.align_mask =PAGE_MASK & ~huge_page_mask (h); - info.align_offset =0 ;return vm_unmapped_area (&info); }
@@ -274,7 +272,7 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, unsigned long pgoff, unsigned long flags) {struct hstate *h =hstate_file (file); -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};unsigned long addr; info.flags =VM_UNMAPPED_AREA_TOPDOWN ;
@@ -282,7 +280,6 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, info.low_limit =PAGE_SIZE ; info.high_limit = current->mm->mmap_base; info.align_mask =PAGE_MASK & ~huge_page_mask (h); - info.align_offset =0 ; addr =vm_unmapped_area (&info);/*
arch/s390/mm/mmap.c +4 -7
@@ -77,7 +77,7 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};if (len >TASK_SIZE - mmap_min_addr)return -ENOMEM ;
@@ -93,14 +93,12 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, goto check_asce_limit; } - info.flags =0 ; info.length = len; info.low_limit = mm->mmap_base; info.high_limit =TASK_SIZE ;if (filp || (flags &MAP_SHARED )) info.align_mask =MMAP_ALIGN_MASK <<PAGE_SHIFT ; -else - info.align_mask =0 ; + info.align_offset = pgoff <<PAGE_SHIFT ; addr =vm_unmapped_area (&info);if (offset_in_page (addr))
@@ -116,7 +114,7 @@ unsigned long arch_get_unmapped_area_topdown(struct file *filp, unsigned long ad {struct vm_area_struct *vma;struct mm_struct *mm = current->mm; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};/* requested length too big for entire address space */ if (len >TASK_SIZE - mmap_min_addr)
@@ -140,8 +138,7 @@ unsigned long arch_get_unmapped_area_topdown(struct file *filp, unsigned long ad info.high_limit = mm->mmap_base;if (filp || (flags &MAP_SHARED )) info.align_mask =MMAP_ALIGN_MASK <<PAGE_SHIFT ; -else - info.align_mask =0 ; + info.align_offset = pgoff <<PAGE_SHIFT ; addr =vm_unmapped_area (&info);
arch/sh/mm/mmap.c +2 -3
@@ -57,7 +57,7 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr,struct mm_struct *mm = current->mm;struct vm_area_struct *vma;int do_colour_align; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};if (flags &MAP_FIXED ) {/* We do not accept a shared mapping if it would violate
@@ -88,7 +88,6 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr,return addr; } - info.flags =0 ; info.length = len; info.low_limit =TASK_UNMAPPED_BASE ; info.high_limit =TASK_SIZE ;
@@ -106,7 +105,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,struct mm_struct *mm = current->mm;unsigned long addr = addr0;int do_colour_align; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};if (flags &MAP_FIXED ) {/* We do not accept a shared mapping if it would violate
arch/sparc/kernel/sys_sparc_32.c +1 -2
@@ -41,7 +41,7 @@ SYSCALL_DEFINE0(getpagesize)unsigned long arch_get_unmapped_area (struct file *filp,unsigned long addr,unsigned long len,unsigned long pgoff,unsigned long flags) { -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};if (flags &MAP_FIXED ) {/* We do not accept a shared mapping if it would violate
@@ -59,7 +59,6 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, unsiif (!addr) addr =TASK_UNMAPPED_BASE ; - info.flags =0 ; info.length = len; info.low_limit = addr; info.high_limit =TASK_SIZE ;
arch/sparc/kernel/sys_sparc_64.c +2 -3
@@ -93,7 +93,7 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, unsistruct vm_area_struct * vma;unsigned long task_size =TASK_SIZE ;int do_color_align; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};if (flags &MAP_FIXED ) {/* We do not accept a shared mapping if it would violate
@@ -126,7 +126,6 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, unsireturn addr; } - info.flags =0 ; info.length = len; info.low_limit =TASK_UNMAPPED_BASE ; info.high_limit =min (task_size,VA_EXCLUDE_START );
@@ -154,7 +153,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,unsigned long task_size =STACK_TOP32 ;unsigned long addr = addr0;int do_color_align; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};/* This should only ever run for 32-bit processes. */ BUG_ON (!test_thread_flag (TIF_32BIT ));
arch/sparc/mm/hugetlbpage.c +2 -5
@@ -31,17 +31,15 @@ static unsigned long hugetlb_get_unmapped_area_bottomup(struct file *filp, {struct hstate *h =hstate_file (filp);unsigned long task_size =TASK_SIZE ; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};if (test_thread_flag (TIF_32BIT )) task_size =STACK_TOP32 ; - info.flags =0 ; info.length = len; info.low_limit =TASK_UNMAPPED_BASE ; info.high_limit =min (task_size,VA_EXCLUDE_START ); info.align_mask =PAGE_MASK & ~huge_page_mask (h); - info.align_offset =0 ; addr =vm_unmapped_area (&info);if ((addr & ~PAGE_MASK ) && task_size >VA_EXCLUDE_END ) {
@@ -63,7 +61,7 @@ hugetlb_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,struct hstate *h =hstate_file (filp);struct mm_struct *mm = current->mm;unsigned long addr = addr0; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};/* This should only ever run for 32-bit processes. */ BUG_ON (!test_thread_flag (TIF_32BIT ));
@@ -73,7 +71,6 @@ hugetlb_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, info.low_limit =PAGE_SIZE ; info.high_limit = mm->mmap_base; info.align_mask =PAGE_MASK & ~huge_page_mask (h); - info.align_offset =0 ; addr =vm_unmapped_area (&info);/*
arch/x86/kernel/sys_x86_64.c +2 -5
@@ -125,7 +125,7 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};unsigned long begin, end;if (flags &MAP_FIXED )
@@ -144,11 +144,9 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr,return addr; } - info.flags =0 ; info.length = len; info.low_limit = begin; info.high_limit = end; - info.align_mask =0 ; info.align_offset = pgoff <<PAGE_SHIFT ;if (filp) { info.align_mask =get_align_mask ();
@@ -165,7 +163,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,struct vm_area_struct *vma;struct mm_struct *mm = current->mm;unsigned long addr = addr0; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};/* requested length too big for entire address space */ if (len >TASK_SIZE )
@@ -210,7 +208,6 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,if (addr >DEFAULT_MAP_WINDOW && !in_32bit_syscall ()) info.high_limit +=TASK_SIZE_MAX -DEFAULT_MAP_WINDOW ; - info.align_mask =0 ; info.align_offset = pgoff <<PAGE_SHIFT ;if (filp) { info.align_mask =get_align_mask ();
arch/x86/mm/hugetlbpage.c +2 -5
@@ -51,9 +51,8 @@ static unsigned long hugetlb_get_unmapped_area_bottomup(struct file *file, unsigned long pgoff, unsigned long flags) {struct hstate *h =hstate_file (file); -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {}; - info.flags =0 ; info.length = len; info.low_limit =get_mmap_base (1 );
@@ -65,7 +64,6 @@ static unsigned long hugetlb_get_unmapped_area_bottomup(struct file *file, task_size_32bit() : task_size_64bit(addr > DEFAULT_MAP_WINDOW); info.align_mask =PAGE_MASK & ~huge_page_mask (h); - info.align_offset =0 ;return vm_unmapped_area (&info); }
@@ -74,7 +72,7 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, unsigned long pgoff, unsigned long flags) {struct hstate *h =hstate_file (file); -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {}; info.flags =VM_UNMAPPED_AREA_TOPDOWN ; info.length = len;
@@ -89,7 +87,6 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, info.high_limit +=TASK_SIZE_MAX -DEFAULT_MAP_WINDOW ; info.align_mask =PAGE_MASK & ~huge_page_mask (h); - info.align_offset =0 ; addr =vm_unmapped_area (&info);/*
fs/hugetlbfs/inode.c +2 -5
@@ -176,14 +176,12 @@ hugetlb_get_unmapped_area_bottomup(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {}; - info.flags =0 ; info.length = len; info.low_limit = current->mm->mmap_base; info.high_limit =arch_get_mmap_end (addr, len, flags); info.align_mask =PAGE_MASK & ~huge_page_mask (h); - info.align_offset =0 ;return vm_unmapped_area (&info); }
@@ -192,14 +190,13 @@ hugetlb_get_unmapped_area_topdown(struct file *file, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {}; info.flags =VM_UNMAPPED_AREA_TOPDOWN ; info.length = len; info.low_limit =PAGE_SIZE ; info.high_limit =arch_get_mmap_base (addr, current->mm->mmap_base); info.align_mask =PAGE_MASK & ~huge_page_mask (h); - info.align_offset =0 ; addr =vm_unmapped_area (&info);/*
mm/mmap.c +2 -7
@@ -1707,7 +1707,7 @@ generic_get_unmapped_area(struct file *filp, unsigned long addr, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma, *prev; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};const unsigned long mmap_end =arch_get_mmap_end (addr, len, flags);if (len > mmap_end - mmap_min_addr)
@@ -1725,12 +1725,9 @@ generic_get_unmapped_area(struct file *filp, unsigned long addr,return addr; } - info.flags =0 ; info.length = len; info.low_limit = mm->mmap_base; info.high_limit = mmap_end; - info.align_mask =0 ; - info.align_offset =0 ;return vm_unmapped_area (&info); }
@@ -1755,7 +1752,7 @@ generic_get_unmapped_area_topdown(struct file *filp, unsigned long addr, {struct vm_area_struct *vma, *prev;struct mm_struct *mm = current->mm; -struct vm_unmapped_area_info info; +struct vm_unmapped_area_info info = {};const unsigned long mmap_end =arch_get_mmap_end (addr, len, flags);/* requested length too big for entire address space */
@@ -1779,8 +1776,6 @@ generic_get_unmapped_area_topdown(struct file *filp, unsigned long addr, info.length = len; info.low_limit =PAGE_SIZE ; info.high_limit =arch_get_mmap_base (addr, mm->mmap_base); - info.align_mask =0 ; - info.align_offset =0 ; addr =vm_unmapped_area (&info);/*
Signature · 1 line
2.34.1
Re: [PATCH v3 08/12] treewide: Use initializer for struct vm_unmapped_area_info
Quoted 6 lines · Rick Edgecombe
So to be reduce the chance of bugs via uninitialized fields, perform a tree wide change using the consensus for the best general way to do this change. Use C99 static initializing to zero the struct and remove and statements that simply set members to zero.
Signed-off-by Rick Edgecombe <[email protected]>
Thanks! This looks to do exactly what it describes. :)
Signature · 1 line
Kees Cook
Re: [PATCH v3 08/12] treewide: Use initializer for struct vm_unmapped_area_info
Quoted 4 lines · Kees Cook
Thanks! This looks to do exactly what it describes. :)
Reviewed-by Kees Cook <[email protected]>
Thanks!
PATCH v3 9/12 mm: Take placement mappings gap into account
In reply to Rick EdgecombeWhen memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.
The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.
For MAP_GROWSDOWN/VM_GROWSDOWN and MAP_GROWSUP/VM_GROWSUP this has not been a problem in practice because applications place these kinds of mappings very early, when there is not many mappings to find a space between. But for shadow stacks, they may be placed throughout the lifetime of the application.
Use the start_gap field to find a space that includes the guard gap for the new mapping. Take care to not interfere with the alignment.
---
v3:
- Spelling fix in comment
v2:
- Remove VM_UNMAPPED_START_GAP_SET and have struct vm_unmapped_area_info initialized with zeros (in another patch). (Kirill)
- Drop unrelated space change (Kirill)
- Add comment around interactions of alignment and start gap step (Kirill)
--- include/linux/mm.h | 1 + mm/mmap.c | 12 +++++++++--- 2 files changed, 10 insertions(+), 3 deletions(-)
include/linux/mm.h +1 -0
@@ -3418,6 +3418,7 @@ struct vm_unmapped_area_info {unsigned long high_limit;unsigned long align_mask;unsigned long align_offset; +unsigned long start_gap; };extern unsigned long vm_unmapped_area (struct vm_unmapped_area_info *info);
mm/mmap.c +9 -3
@@ -1582,7 +1582,7 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info)MA_STATE (mas, ¤t->mm->mm_mt,0 ,0 );/* Adjust search length to account for worst case alignment overhead */ - length = info->length + info->align_mask; + length = info->length + info->align_mask + info->start_gap;if (length < info->length)return -ENOMEM ;
@@ -1594,7 +1594,13 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info)if (mas_empty_area (&mas, low_limit, high_limit -1 , length))return -ENOMEM ; - gap = mas.index; +/* +* Adjust for the gap first so it doesn't interfere with the +* later alignment. The first step is the minimum needed to +* fulill the start gap, the next steps is the minimum to align +* that. It is the minimum needed to fulill both. +*/ + gap = mas.index + info->start_gap; gap += (info->align_offset - gap) & info->align_mask; tmp =mas_next (&mas,ULONG_MAX );if (tmp && (tmp->vm_flags &VM_STARTGAP_FLAGS )) {/* Avoid prev check if possible */
@@ -1633,7 +1639,7 @@ static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info)MA_STATE (mas, ¤t->mm->mm_mt,0 ,0 );/* Adjust search length to account for worst case alignment overhead */ - length = info->length + info->align_mask; + length = info->length + info->align_mask + info->start_gap;if (length < info->length)return -ENOMEM ;
Signature · 1 line
2.34.1
Re: [PATCH v3 09/12] mm: Take placement mappings gap into account
Le 12/03/2024 à 23:28, Rick Edgecombe a écrit :
Quoted 46 lines
When memory is being placed,
mmap()will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK,VM_GROWSUPandVM_GROWSDOWN). In order to ensure guard gaps between mappings,mmap()needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.The long standing behavior of
mmap()is to ensure 1, but not take any care around 2. So for example, if there is aPAGE_SIZEfree area, and ammap()with aPAGE_SIZEsize, and a type that has a guard gap is being placed,mmap()may place the shadow stack in thePAGE_SIZEfree area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.For MAP_GROWSDOWN/VM_GROWSDOWN and MAP_GROWSUP/VM_GROWSUP this has not been a problem in practice because applications place these kinds of mappings very early, when there is not many mappings to find a space between. But for shadow stacks, they may be placed throughout the lifetime of the application.
Use the
start_gapfield to find a space that includes the guard gap for the new mapping. Take care to not interfere with the alignment.Signed-off-by Rick Edgecombe <[email protected]>---
v3:
- Spelling fix in commentv2:
- RemoveVM_UNMAPPED_START_GAP_SETand have structvm_unmapped_area_infoinitialized with zeros (in another patch). (Kirill)
- Drop unrelated space change (Kirill)
- Add comment around interactions of alignment and start gap step (Kirill)--- include/linux/mm.h | 1 + mm/mmap.c | 12 +++++++++--- 2 files changed, 10 insertions(+), 3 deletions(-)include/linux/mm.h +0
-0
include/linux/mm.h struct vm_unmapped_area_info { · line 3421
Quoted 4 lines
include/linux/mm.h +1
-0@@ -3418,6 +3418,7 @@ struct vm_unmapped_area_info {unsigned long high_limit;unsigned long align_mask;unsigned long align_offset; +unsigned long start_gap;
Only a start_gap is needed ? No need of an end_gap ?
Quoted 40 lines
include/linux/mm.h +0
-0};extern unsigned long vm_unmapped_area (struct vm_unmapped_area_info *info);mm/mmap.c +9
-3@@ -1582,7 +1582,7 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info)MA_STATE (mas, ¤t->mm->mm_mt,0 ,0 );/* Adjust search length to account for worst case alignment overhead */ - length = info->length + info->align_mask; + length = info->length + info->align_mask + info->start_gap;if (length < info->length)return -ENOMEM ;@@ -1594,7 +1594,13 @@ static unsigned long unmapped_area(struct vm_unmapped_area_info *info)if (mas_empty_area (&mas, low_limit, high_limit -1 , length))return -ENOMEM ; - gap = mas.index; +/* +* Adjust for the gap first so it doesn't interfere with the +* later alignment. The first step is the minimum needed to +* fulill the start gap, the next steps is the minimum to align +* that. It is the minimum needed to fulill both. +*/ + gap = mas.index + info->start_gap; gap += (info->align_offset - gap) & info->align_mask; tmp =mas_next (&mas,ULONG_MAX );if (tmp && (tmp->vm_flags &VM_STARTGAP_FLAGS )) {/* Avoid prev check if possible */ @@ -1633,7 +1639,7 @@ static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info)MA_STATE (mas, ¤t->mm->mm_mt,0 ,0 );/* Adjust search length to account for worst case alignment overhead */ - length = info->length + info->align_mask; + length = info->length + info->align_mask + info->start_gap;if (length < info->length)return -ENOMEM ;
Re: [PATCH v3 09/12] mm: Take placement mappings gap into account
Quoted 2 lines · Christophe Leroy
Only a
start_gapis needed ? No need of anend_gap?
Yea, shadow stacks only have a start gap. If any end gap is needed, it will be much easier to add after all of this.
Christophe Leroy
Reviewed-byRe: [PATCH v3 09/12] mm: Take placement mappings gap into account
Le 13/03/2024 à 15:58, Edgecombe, Rick P a écrit :
Quoted 6 lines · Christophe Leroy
Quoted 2 lines
Only a
start_gapis needed ? No need of anend_gap?Yea, shadow stacks only have a start gap. If any end gap is needed, it will be much easier to add after all of this.
Ok
PATCH v3 10/12 x86/mm: Implement HAVE_ARCH_UNMAPPED_AREA_VMFLAGS
In reply to Rick EdgecombeWhen memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.
The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.
Add x86 arch implementations of arch_get_unmapped_area_vmflags/_topdown() so future changes can allow the guard gap of type of vma being placed to be taken into account. This will be used for shadow stack memory.
---
v3:
- Commit log grammar
v2:
- Remove unnecessary added extern
--- arch/x86/include/asm/pgtable_64.h | 1 + arch/x86/kernel/sys_x86_64.c | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-)
arch/x86/include/asm/pgtable_64.h +1 -0
@@ -244,6 +244,7 @@ extern void cleanup_highmap(void);#define HAVE_ARCH_UNMAPPED_AREA #define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN +#define HAVE_ARCH_UNMAPPED_AREA_VMFLAGS #define PAGE_AGP PAGE_KERNEL_NOCACHE#define HAVE_PAGE_AGP 1
arch/x86/kernel/sys_x86_64.c +20 -5
@@ -120,8 +120,8 @@ static void find_start_end(unsigned long addr, unsigned long flags, }unsigned long -arch_get_unmapped_area (struct file *filp,unsigned long addr, -unsigned long len,unsigned long pgoff,unsigned long flags) +arch_get_unmapped_area_vmflags (struct file *filp,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags,vm_flags_t vm_flags) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma;
@@ -156,9 +156,9 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, }unsigned long -arch_get_unmapped_area_topdown (struct file *filp,const unsigned long addr0, -const unsigned long len,const unsigned long pgoff, -const unsigned long flags) +arch_get_unmapped_area_topdown_vmflags (struct file *filp,unsigned long addr0, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t vm_flags) {struct vm_area_struct *vma;struct mm_struct *mm = current->mm;
@@ -227,3 +227,18 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,*/ return arch_get_unmapped_area (filp, addr0, len, pgoff, flags); } + +unsigned long +arch_get_unmapped_area (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff,unsigned long flags) +{ +return arch_get_unmapped_area_vmflags (filp, addr, len, pgoff, flags,0 ); +} + +unsigned long +arch_get_unmapped_area_topdown (struct file *filp,const unsigned long addr, +const unsigned long len,const unsigned long pgoff, +const unsigned long flags) +{ +return arch_get_unmapped_area_topdown_vmflags (filp, addr, len, pgoff, flags,0 ); +}
Signature · 1 line
2.34.1
Re: [PATCH v3 10/12] x86/mm: Implement HAVE_ARCH_UNMAPPED_AREA_VMFLAGS
Le 12/03/2024 à 23:28, Rick Edgecombe a écrit :
Quoted 81 lines
When memory is being placed,
mmap()will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK,VM_GROWSUPandVM_GROWSDOWN). In order to ensure guard gaps between mappings,mmap()needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.The long standing behavior of
mmap()is to ensure 1, but not take any care around 2. So for example, if there is aPAGE_SIZEfree area, and ammap()with aPAGE_SIZEsize, and a type that has a guard gap is being placed,mmap()may place the shadow stack in thePAGE_SIZEfree area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.Add x86 arch implementations of arch_get_unmapped_area_vmflags/_topdown() so future changes can allow the guard gap of type of vma being placed to be taken into account. This will be used for shadow stack memory.
Signed-off-by Rick Edgecombe <[email protected]>---
v3:
- Commit log grammarv2:
- Remove unnecessary added extern--- arch/x86/include/asm/pgtable_64.h | 1 + arch/x86/kernel/sys_x86_64.c | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-)arch/x86/include/asm/pgtable_64.h +1
-0@@ -244,6 +244,7 @@ extern void cleanup_highmap(void);#define HAVE_ARCH_UNMAPPED_AREA #define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN +#define HAVE_ARCH_UNMAPPED_AREA_VMFLAGS #define PAGE_AGP PAGE_KERNEL_NOCACHE#define HAVE_PAGE_AGP 1arch/x86/kernel/sys_x86_64.c +10
-5@@ -120,8 +120,8 @@ static void find_start_end(unsigned long addr, unsigned long flags, }unsigned long -arch_get_unmapped_area (struct file *filp,unsigned long addr, -unsigned long len,unsigned long pgoff,unsigned long flags) +arch_get_unmapped_area_vmflags (struct file *filp,unsigned long addr,unsigned long len, +unsigned long pgoff,unsigned long flags,vm_flags_t vm_flags) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma;@@ -156,9 +156,9 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, }unsigned long -arch_get_unmapped_area_topdown (struct file *filp,const unsigned long addr0, -const unsigned long len,const unsigned long pgoff, -const unsigned long flags) +arch_get_unmapped_area_topdown_vmflags (struct file *filp,unsigned long addr0, +unsigned long len,unsigned long pgoff, +unsigned long flags,vm_flags_t vm_flags) {struct vm_area_struct *vma;struct mm_struct *mm = current->mm;@@ -227,3 +227,18 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,*/ return arch_get_unmapped_area (filp, addr0, len, pgoff, flags); } + +unsigned long +arch_get_unmapped_area (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff,unsigned long flags) +{
arch/x86/kernel/sys_x86_64.c arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, · line 244
Quoted 10 lines
arch/x86/kernel/sys_x86_64.c +10
-0+return arch_get_unmapped_area_vmflags (filp, addr, len, pgoff, flags,0 ); +} + +unsigned long +arch_get_unmapped_area_topdown (struct file *filp,const unsigned long addr, +const unsigned long len,const unsigned long pgoff, +const unsigned long flags) +{ +return arch_get_unmapped_area_topdown_vmflags (filp, addr, len, pgoff, flags,0 ); +}
Wouldn't it be better to define those two as static inlines ?
Re: [PATCH v3 10/12] x86/mm: Implement HAVE_ARCH_UNMAPPED_AREA_VMFLAGS
Quoted 14 lines · Christophe Leroy
Quoted 14 lines
arch/x86/kernel/sys_x86_64.c +4
-0+ +unsigned long +arch_get_unmapped_area (struct file *filp,unsigned long addr, + unsigned long len, unsigned long pgoff, unsignedlong flags)
+{
+ returnarch_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, 0);arch/x86/kernel/sys_x86_64.c +4
-0+} + +unsigned long +arch_get_unmapped_area_topdown(struct file *filp, const unsignedlong addr,
+ const unsigned long len, const unsigned
Quoted 8 lines
Quoted 6 lines
long pgoff,
+ const unsigned long flags)
+{
+ returnarch_get_unmapped_area_topdown_vmflags(filp, addr, len, pgoff, flags, 0);
+}Wouldn't it be better to define those two as static inlines ?
Yes, I think so. It is generic functionality (though not needed until the next shadow stack feature), so doesn't need to be in arch/x86 either.
Thanks for your comments on the series and the RB tags.
Re: [PATCH v3 10/12] x86/mm: Implement HAVE_ARCH_UNMAPPED_AREA_VMFLAGS
Quoted 3 lines · Rick Edgecombe
Quoted 1 line
Wouldn't it be better to define those two as static inlines ?
Yes, I think so.
I gave this a try and it turned out not to fit into any header too well. I decided putting it into a generic header was not great since these are actually supposed to be arch implementations. Emphasizing that point, mmap.c actually defines these unless HAVE_ARCH_UNMAPPED_AREA is defined. So to make it work, mm.h would have to assume that if HAVE_ARCH_UNMAPPED_AREA and HAVE_ARCH_UNMAPPED_AREA_TOPDOWN are defined, the arch actually doesn't have an arch_unmmapped_area() and wants a static inline version of arch_get_unmapped_area(). It confuses the meaning of HAVE_ARCH_UNMAPPED_AREA a bit to mean the opposite in some cases. Adding a ARCH_WANTS_UNMAPPED_AREA_INLINE seemed excessive.
As for putting them in an arch/x86 header, I tried asm/mmu.h, but arch_get_unmapped_area_topdown/_vmflags() had to be forward declared. But then also vm_flags_t couldn't be pulled in properly because of a circular dependency. A few others hit weirdness or breakages.
So in the end, I decided to just leave it as a non-static inline in arch/x86. Unless there are any objections, I'm going to just let 0-day build test all the archs, and I'll post the series with the rest of the feedback in a few days.
PATCH v3 11/12 x86/mm: Care about shadow stack guard gap during placement
In reply to Rick EdgecombeWhen memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things:
1. That the new mapping isn’t placed in an any existing mappings guard gaps.
2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps.
The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA.
Now that the vm_flags is passed into the arch get_unmapped_area()'s, and vm_unmapped_area() is ready to consider it, have VM_SHADOW_STACK's get guard gap consideration for scenario 2.
--- arch/x86/kernel/sys_x86_64.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/kernel/sys_x86_64.c +10 -0
@@ -119,6 +119,14 @@ static void find_start_end(unsigned long addr, unsigned long flags, *end =task_size_64bit (addr >DEFAULT_MAP_WINDOW ); } +static inline unsigned long stack_guard_placement (vm_flags_t vm_flags) +{ +if (vm_flags &VM_SHADOW_STACK ) +return PAGE_SIZE ; + +return 0 ; +} +unsigned long arch_get_unmapped_area_vmflags (struct file *filp,unsigned long addr,unsigned long len,unsigned long pgoff,unsigned long flags,vm_flags_t vm_flags)
@@ -148,6 +156,7 @@ arch_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, unsigned l info.low_limit = begin; info.high_limit = end; info.align_offset = pgoff <<PAGE_SHIFT ; + info.start_gap =stack_guard_placement (vm_flags);if (filp) { info.align_mask =get_align_mask (); info.align_offset +=get_align_bits ();
@@ -197,6 +206,7 @@ arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr0, info.low_limit =PAGE_SIZE ; info.high_limit =get_mmap_base (0 ); + info.start_gap =stack_guard_placement (vm_flags);/* * If hint address is above DEFAULT_MAP_WINDOW, look for unmapped area
Signature · 1 line
2.34.1
PATCH v3 12/12 selftests/x86: Add placement guard gap test for shstk
In reply to Rick EdgecombeThe existing shadow stack test for guard gaps just checks that new mappings are not placed in an existing mapping's guard gap. Add one that checks that new mappings are not placed such that preexisting mappings are in the new mappings guard gap.
--- .../testing/selftests/x86/test_shadow_stack.c | 67 +++++++++++++++++-- 1 file changed, 63 insertions(+), 4 deletions(-)
tools/testing/selftests/x86/test_shadow_stack.c +63 -4
@@ -556,7 +556,7 @@ struct node {* looked at the shadow stack gaps. * 5. See if it landed in the gap. */ -int test_guard_gap (void ) +int test_guard_gap_other_gaps (void ) {void *free_area, *shstk, *test_map = (void *)0xFFFFFFFFFFFFFFFF ;struct node *head =NULL , *cur;
@@ -593,11 +593,64 @@ int test_guard_gap(void)if (shstk - test_map -PAGE_SIZE !=PAGE_SIZE )return 1 ; -printf ("[OK]\tGuard gap test\n" ); +printf ("[OK]\tGuard gap test, other mapping's gaps\n" );return 0 ; } +/* Tests respecting the guard gap of the mapping getting placed */ +int test_guard_gap_new_mappings_gaps (void ) +{ +void *free_area, *shstk_start, *test_map = (void *)0xFFFFFFFFFFFFFFFF ; +struct node *head =NULL , *cur; +int ret =0 ; + + free_area =mmap (0 ,PAGE_SIZE *4 ,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); +munmap (free_area,PAGE_SIZE *4 ); + +/* Test letting map_shadow_stack find a free space */ + shstk_start =mmap (free_area,PAGE_SIZE ,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); +if (shstk_start ==MAP_FAILED || shstk_start != free_area) +return 1 ; + +while (test_map > shstk_start) { + test_map = (void *)syscall (__NR_map_shadow_stack,0 ,PAGE_SIZE ,0 ); +if (test_map ==MAP_FAILED ) { +printf ("[INFO]\tmap_shadow_stack MAP_FAILED\n" ); + ret =1 ; +break ; + } + + cur =malloc (sizeof (*cur)); + cur->mapping = test_map; + + cur->next = head; + head = cur; + +if (test_map == free_area +PAGE_SIZE ) { +printf ("[INFO]\tNew mapping has other mapping in guard gap!\n" ); + ret =1 ; +break ; + } + } + +while (head) { + cur = head; + head = cur->next; +munmap (cur->mapping,PAGE_SIZE ); +free (cur); + } + +munmap (shstk_start,PAGE_SIZE ); + +if (!ret) +printf ("[OK]\tGuard gap test, placement mapping's gaps\n" ); + +return ret; +} +/* * Too complicated to pull it out of the 32 bit header, but also get the * 64 bit one needed above. Just define a copy here.
@@ -850,9 +903,15 @@ int main(int argc, char *argv[]) goto out; } -if (test_guard_gap ()) { +if (test_guard_gap_other_gaps ()) { ret =1 ; -printf ("[FAIL]\tGuard gap test\n" ); +printf ("[FAIL]\tGuard gap test, other mappings' gaps\n" ); + goto out; + } + +if (test_guard_gap_new_mappings_gaps ()) { + ret =1 ; +printf ("[FAIL]\tGuard gap test, placement mapping's gaps\n" ); goto out; }
Signature · 1 line
2.34.1