Skip to contentSkip to conversation

mremap: move_vma() fixes

PATCH Applied Dmitry Safonov 3 participants 16 messages Oct 13, 2020 – Jan 22, 2021

Subsystems (15)
lore
Share & tools
lore
#0 lore raw

cover mremap: move_vma() fixes

1 - seems to be historical issue on a rarely taken path 2,3 - fixes related to the new mremap() flag
5 - dax device/hugetlbfs possible issue

4,6 - refactoring

As those seems to be actual issues, sending this during the merge-window.

(Changes to architecture code are in the 6 patch, but Cc'ing maintainers on cover for the context, I hope it's fine).

Cc Alexander Viro <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Brian Geffon <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Dan Williams <[email protected]>
Cc Dave Jiang <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Kirill A. Shutemov <[email protected]>
Cc Mike Kravetz <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Russell King <[email protected]>
Cc Thomas Bogendoerfer <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Will Deacon <[email protected]>

Dmitry Safonov (6):
mm/mremap: Account memory on do_munmap() failure mm/mremap: For MREMAP_DONTUNMAP check security_vm_enough_memory_mm()
mremap: Don't allow MREMAP_DONTUNMAP on special_mappings and aio
vm_ops: Rename .split() callback to .may_split()
mremap: Check if it's possible to split original vma
mm: Forbid splitting special mappings

 arch/arm/kernel/vdso.c                    |  9 ----
 arch/arm64/kernel/vdso.c                  | 41 ++-----------------
 arch/mips/vdso/genvdso.c                  |  4 --
 arch/s390/kernel/vdso.c                   | 11 +----
 arch/x86/entry/vdso/vma.c                 | 17 --------
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  2 +-
 drivers/dax/device.c                      |  4 +-
 fs/aio.c                                  |  5 ++-
 include/linux/mm.h                        |  5 ++-
 ipc/shm.c                                 |  8 ++--
 mm/hugetlb.c                              |  2 +-
 mm/mmap.c                                 | 22 ++++++++--
 mm/mremap.c                               | 50 +++++++++++------------
 13 files changed, 63 insertions(+), 117 deletions(-)

base-commit: bbf5c979011a099af5dc76498918ed7df445635b

Signature · 1 line
2.28.0
#1 lore raw

PATCH 1/6 mm/mremap: Account memory on do_munmap() failure

move_vma() copies VMA without adding it to account, then unmaps old part of VMA. On failure it unmaps the new VMA. With hacks accounting in munmap is disabled as it's a copy of existing VMA.

Account the memory on munmap() failure which was previously copied into a new VMA.

Fixes commit e2ea83742133 ("[PATCH] mremap: move_vma fixes and cleanup")
---
 mm/mremap.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/mremap.c +2 -1
index 138abbae4f75..03d31a0d4c67 100644
@@ -450,7 +450,8 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 
 	if (do_munmap(mm, old_addr, old_len, uf_unmap) < 0) {
 		/* OOM: unable to split vma, just get accounts right */
-		vm_unacct_memory(excess >> PAGE_SHIFT);
+		if (vm_flags & VM_ACCOUNT)
+			vm_acct_memory(new_len >> PAGE_SHIFT);
 		excess = 0;
 	}
 
Signature · 1 line
2.28.0
#2 lore raw

PATCH 2/6 mm/mremap: For MREMAP_DONTUNMAP check security_vm_enough_memory_mm()

In reply to Dmitry Safonov

Currently memory is accounted post-mremap() with MREMAP_DONTUNMAP, which may break overcommit policy. So, check if there's enough memory before doing actual VMA copy.

Don't unset VM_ACCOUNT on MREMAP_DONTUNMAP. By semantics, such mremap() is actually a memory allocation. That also simplifies the error-path a little.

Also, as it's memory allocation on success don't reset hiwater_vm value.

Fixes commit e346b3813067 ("mm/mremap: add MREMAP_DONTUNMAP to mremap()")
---
 mm/mremap.c | 36 +++++++++++++-----------------------
 1 file changed, 13 insertions(+), 23 deletions(-)
mm/mremap.c +13 -23
index 03d31a0d4c67..c248f9a52125 100644
@@ -365,11 +365,19 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 	if (err)
 		return err;
 
+	if (unlikely(flags & MREMAP_DONTUNMAP && vm_flags & VM_ACCOUNT)) {
+		if (security_vm_enough_memory_mm(mm, new_len >> PAGE_SHIFT))
+			return -ENOMEM;
+	}
+
 	new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT);
 	new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff,
 			   &need_rmap_locks);
-	if (!new_vma)
+	if (!new_vma) {
+		if (unlikely(flags & MREMAP_DONTUNMAP && vm_flags & VM_ACCOUNT))
+			vm_unacct_memory(new_len >> PAGE_SHIFT);
 		return -ENOMEM;
+	}
 
 	moved_len = move_page_tables(vma, old_addr, new_vma, new_addr, old_len,
 				     need_rmap_locks);
@@ -398,7 +406,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 	}
 
 	/* Conceal VM_ACCOUNT so old reservation is not undone */
-	if (vm_flags & VM_ACCOUNT) {
+	if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) {
 		vma->vm_flags &= ~VM_ACCOUNT;
 		excess = vma->vm_end - vma->vm_start - old_len;
 		if (old_addr > vma->vm_start &&
@@ -423,34 +431,16 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 		untrack_pfn_moved(vma);
 
 	if (unlikely(!err && (flags & MREMAP_DONTUNMAP))) {
-		if (vm_flags & VM_ACCOUNT) {
-			/* Always put back VM_ACCOUNT since we won't unmap */
-			vma->vm_flags |= VM_ACCOUNT;
-
-			vm_acct_memory(new_len >> PAGE_SHIFT);
-		}
-
-		/*
-		 * VMAs can actually be merged back together in copy_vma
-		 * calling merge_vma. This can happen with anonymous vmas
-		 * which have not yet been faulted, so if we were to consider
-		 * this VMA split we'll end up adding VM_ACCOUNT on the
-		 * next VMA, which is completely unrelated if this VMA
-		 * was re-merged.
-		 */
-		if (split && new_vma == vma)
-			split = 0;
-
 		/* We always clear VM_LOCKED[ONFAULT] on the old vma */
 		vma->vm_flags &= VM_LOCKED_CLEAR_MASK;
 
 		/* Because we won't unmap we don't need to touch locked_vm */
-		goto out;
+		return new_addr;
 	}
 
 	if (do_munmap(mm, old_addr, old_len, uf_unmap) < 0) {
 		/* OOM: unable to split vma, just get accounts right */
-		if (vm_flags & VM_ACCOUNT)
+		if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP))
 			vm_acct_memory(new_len >> PAGE_SHIFT);
 		excess = 0;
 	}
@@ -459,7 +449,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 		mm->locked_vm += new_len >> PAGE_SHIFT;
 		*locked = true;
 	}
-out:
+
 	mm->hiwater_vm = hiwater_vm;
 
 	/* Restore VM_ACCOUNT if one or two pieces of vma left */
Signature · 1 line
2.28.0
#3 lore raw

Re: [PATCH 2/6] mm/mremap: For MREMAP_DONTUNMAP check security_vm_enough_memory_mm()

This looks good to me with a small comment.

mm/mremap.c static unsigned long move_vma(struct vm_area_struct *vma, · line 444

Quoted 5 lines
        if (do_munmap(mm, old_addr, old_len, uf_unmap) < 0) {
                /* OOM: unable to split vma, just get accounts right */
mm/mremap.c +1 -1
-               if (vm_flags & VM_ACCOUNT)
+               if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP))
                        vm_acct_memory(new_len >> PAGE_SHIFT);

Checking MREMAP_DONTUNMAP in the do_munmap path is unnecessary as MREMAP_DONTUNMAP will have already returned by this point.

#4 lore raw

Re: [PATCH 2/6] mm/mremap: For MREMAP_DONTUNMAP check security_vm_enough_memory_mm()

Quoted 10 lines · Brian Geffon

This looks good to me with a small comment.

mm/mremap.c static unsigned long move_vma(struct vm_area_struct *vma, · line 444

Quoted 5 lines
        if (do_munmap(mm, old_addr, old_len, uf_unmap) < 0) {
                /* OOM: unable to split vma, just get accounts right */
mm/mremap.c +1 -1
-               if (vm_flags & VM_ACCOUNT)
+               if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP))
                        vm_acct_memory(new_len >> PAGE_SHIFT);

Checking MREMAP_DONTUNMAP in the do_munmap path is unnecessary as MREMAP_DONTUNMAP will have already returned by this point.

In this code it is also used as err-path. In case move_page_tables() fails to move all page tables or .mremap() callback fails, the new VMA is unmapped.

IOW, MREMAP_DONTUNMAP returns under:
: if (unlikely(!err && (flags & MREMAP_DONTUNMAP))) {

Signature · 1 line
          Dima
#5 lore raw

Re: [PATCH 2/6] mm/mremap: For MREMAP_DONTUNMAP check security_vm_enough_memory_mm()

Ah, you're right. This individual patch looks good to me.

Brian

Quoted 22 lines · Dmitry Safonov
Quoted 10 lines · Brian Geffon

This looks good to me with a small comment.

mm/mremap.c static unsigned long move_vma(struct vm_area_struct *vma, · line 444

Quoted 5 lines
        if (do_munmap(mm, old_addr, old_len, uf_unmap) < 0) {
                /* OOM: unable to split vma, just get accounts right */
mm/mremap.c +1 -1
-               if (vm_flags & VM_ACCOUNT)
+               if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP))
                        vm_acct_memory(new_len >> PAGE_SHIFT);

Checking MREMAP_DONTUNMAP in the do_munmap path is unnecessary as MREMAP_DONTUNMAP will have already returned by this point.

In this code it is also used as err-path. In case move_page_tables() fails to move all page tables or .mremap() callback fails, the new VMA is unmapped.

IOW, MREMAP_DONTUNMAP returns under:
: if (unlikely(!err && (flags & MREMAP_DONTUNMAP))) {

--
Dima

#6 lore raw

PATCH 3/6 mremap: Don't allow MREMAP_DONTUNMAP on special_mappings and aio

In reply to Dmitry Safonov

As kernel expect to see only one of such mappings, any further operations on the VMA-copy may be unexpected by the kernel. Maybe it's being on the safe side, but there doesn't seem to be any expected use-case for this, so restrict it now.

Fixes commit e346b3813067 ("mm/mremap: add MREMAP_DONTUNMAP to mremap()")
---
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c | 2 +-
 fs/aio.c                                  | 5 ++++-
 include/linux/mm.h                        | 2 +-
 mm/mmap.c                                 | 6 +++++-
 mm/mremap.c                               | 2 +-
 5 files changed, 12 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/resctrl/pseudo_lock.c +1 -1
index 0daf2f1cf7a8..e916646adc69 100644
@@ -1458,7 +1458,7 @@ static int pseudo_lock_dev_release(struct inode *inode, struct file *filp)
 	return 0;
 }
 
-static int pseudo_lock_dev_mremap(struct vm_area_struct *area)
+static int pseudo_lock_dev_mremap(struct vm_area_struct *area, unsigned long flags)
 {
 	/* Not supported */
 	return -EINVAL;
fs/aio.c +4 -1
index d5ec30385566..3be3c0f77548 100644
@@ -324,13 +324,16 @@ static void aio_free_ring(struct kioctx *ctx)
 	}
 }
 
-static int aio_ring_mremap(struct vm_area_struct *vma)
+static int aio_ring_mremap(struct vm_area_struct *vma, unsigned long flags)
 {
 	struct file *file = vma->vm_file;
 	struct mm_struct *mm = vma->vm_mm;
 	struct kioctx_table *table;
 	int i, res = -EINVAL;
 
+	if (flags & MREMAP_DONTUNMAP)
+		return -EINVAL;
+
 	spin_lock(&mm->ioctx_lock);
 	rcu_read_lock();
 	table = rcu_dereference(mm->ioctx_table);
include/linux/mm.h +1 -1
index 16b799a0522c..fd51a4a1f722 100644
@@ -550,7 +550,7 @@ struct vm_operations_struct {
 	void (*open)(struct vm_area_struct * area);
 	void (*close)(struct vm_area_struct * area);
 	int (*split)(struct vm_area_struct * area, unsigned long addr);
-	int (*mremap)(struct vm_area_struct * area);
+	int (*mremap)(struct vm_area_struct *area, unsigned long flags);
 	vm_fault_t (*fault)(struct vm_fault *vmf);
 	vm_fault_t (*huge_fault)(struct vm_fault *vmf,
 			enum page_entry_size pe_size);
mm/mmap.c +5 -1
index bdd19f5b994e..50f853b0ec39 100644
@@ -3372,10 +3372,14 @@ static const char *special_mapping_name(struct vm_area_struct *vma)
 	return ((struct vm_special_mapping *)vma->vm_private_data)->name;
 }
 
-static int special_mapping_mremap(struct vm_area_struct *new_vma)
+static int special_mapping_mremap(struct vm_area_struct *new_vma,
+				  unsigned long flags)
 {
 	struct vm_special_mapping *sm = new_vma->vm_private_data;
 
+	if (flags & MREMAP_DONTUNMAP)
+		return -EINVAL;
+
 	if (WARN_ON_ONCE(current->mm != new_vma->vm_mm))
 		return -EFAULT;
 
mm/mremap.c +1 -1
index c248f9a52125..898e9818ba6d 100644
@@ -384,7 +384,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 	if (moved_len < old_len) {
 		err = -ENOMEM;
 	} else if (vma->vm_ops && vma->vm_ops->mremap) {
-		err = vma->vm_ops->mremap(new_vma);
+		err = vma->vm_ops->mremap(new_vma, flags);
 	}
 
 	if (unlikely(err)) {
Signature · 1 line
2.28.0
#7 lore raw

Re: [PATCH 3/6] mremap: Don't allow MREMAP_DONTUNMAP on special_mappings and aio

I don't think this situation can ever happen MREMAP_DONTUNMAP is already restricted to anonymous mappings (defined as not having vm_ops) and vma_to_resize checks that the mapping is anonymous before move_vma is called.

Quoted 100 lines · Dmitry Safonov

As kernel expect to see only one of such mappings, any further operations on the VMA-copy may be unexpected by the kernel. Maybe it's being on the safe side, but there doesn't seem to be any expected use-case for this, so restrict it now.

Fixes commit e346b3813067 ("mm/mremap: add MREMAP_DONTUNMAP to mremap()")
---
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c | 2 +-
 fs/aio.c                                  | 5 ++++-
 include/linux/mm.h                        | 2 +-
 mm/mmap.c                                 | 6 +++++-
 mm/mremap.c                               | 2 +-
 5 files changed, 12 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/resctrl/pseudo_lock.c +1 -1
index 0daf2f1cf7a8..e916646adc69 100644
@@ -1458,7 +1458,7 @@ static int pseudo_lock_dev_release(struct inode *inode, struct file *filp)
        return 0;
 }
 
-static int pseudo_lock_dev_mremap(struct vm_area_struct *area)
+static int pseudo_lock_dev_mremap(struct vm_area_struct *area, unsigned long flags)
 {
        /* Not supported */
        return -EINVAL;
fs/aio.c +4 -1
index d5ec30385566..3be3c0f77548 100644
@@ -324,13 +324,16 @@ static void aio_free_ring(struct kioctx *ctx)
        }
 }
 
-static int aio_ring_mremap(struct vm_area_struct *vma)
+static int aio_ring_mremap(struct vm_area_struct *vma, unsigned long flags)
 {
        struct file *file = vma->vm_file;
        struct mm_struct *mm = vma->vm_mm;
        struct kioctx_table *table;
        int i, res = -EINVAL;
 
+       if (flags & MREMAP_DONTUNMAP)
+               return -EINVAL;
+
        spin_lock(&mm->ioctx_lock);
        rcu_read_lock();
        table = rcu_dereference(mm->ioctx_table);
include/linux/mm.h +1 -1
index 16b799a0522c..fd51a4a1f722 100644
@@ -550,7 +550,7 @@ struct vm_operations_struct {
        void (*open)(struct vm_area_struct * area);
        void (*close)(struct vm_area_struct * area);
        int (*split)(struct vm_area_struct * area, unsigned long addr);
-       int (*mremap)(struct vm_area_struct * area);
+       int (*mremap)(struct vm_area_struct *area, unsigned long flags);
        vm_fault_t (*fault)(struct vm_fault *vmf);
        vm_fault_t (*huge_fault)(struct vm_fault *vmf,
                        enum page_entry_size pe_size);
mm/mmap.c +5 -1
index bdd19f5b994e..50f853b0ec39 100644
@@ -3372,10 +3372,14 @@ static const char *special_mapping_name(struct vm_area_struct *vma)
        return ((struct vm_special_mapping *)vma->vm_private_data)->name;
 }
 
-static int special_mapping_mremap(struct vm_area_struct *new_vma)
+static int special_mapping_mremap(struct vm_area_struct *new_vma,
+                                 unsigned long flags)
 {
        struct vm_special_mapping *sm = new_vma->vm_private_data;
 
+       if (flags & MREMAP_DONTUNMAP)
+               return -EINVAL;
+
        if (WARN_ON_ONCE(current->mm != new_vma->vm_mm))
                return -EFAULT;
 
mm/mremap.c +1 -2
index c248f9a52125..898e9818ba6d 100644
@@ -384,7 +384,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,
        if (moved_len < old_len) {
                err = -ENOMEM;
        } else if (vma->vm_ops && vma->vm_ops->mremap) {
-               err = vma->vm_ops->mremap(new_vma);
+               err = vma->vm_ops->mremap(new_vma, flags);
        }
 
        if (unlikely(err)) {
--

2.28.0

#8 lore raw

Re: [PATCH 3/6] mremap: Don't allow MREMAP_DONTUNMAP on special_mappings and aio

[I moved your reply to avoid top-posting]

Quoted 7 lines · Brian Geffon
Quoted 6 lines · Dmitry Safonov

As kernel expect to see only one of such mappings, any further operations on the VMA-copy may be unexpected by the kernel. Maybe it's being on the safe side, but there doesn't seem to be any expected use-case for this, so restrict it now.

Quoted 7 lines
Quoted 2 lines
Fixes commit e346b3813067 ("mm/mremap: add MREMAP_DONTUNMAP to mremap()")

I don't think this situation can ever happen MREMAP_DONTUNMAP is already restricted to anonymous mappings (defined as not having vm_ops) and vma_to_resize checks that the mapping is anonymous before move_vma is called.

I've looked again now, I think it is possible. One can call MREMAP_DONTUNMAP without MREMAP_FIXED and without resizing. So that the old VMA is copied at some free address.

The calltrace would be: mremap()=>move_vma()
[under if (flags & MREMAP_MAYMOVE)].

On the other side I agree with you that the fix could have been better if I realized the semantics that MREMAP_DONTUNMAP should only work with anonymous mappings.

Probably, a better fix would be to move
: if (flags & MREMAP_DONTUNMAP && (!vma_is_anonymous(vma) ||
: vma->vm_flags & VM_SHARED))
: return ERR_PTR(-EINVAL);

from vma_to_resize() into the mremap() syscall directly. What do you think?

Signature · 1 line
          Dima
#9 lore raw

Re: [PATCH 3/6] mremap: Don't allow MREMAP_DONTUNMAP on special_mappings and aio

Quoted 29 lines · Dmitry Safonov

[I moved your reply to avoid top-posting]

Quoted 7 lines · Brian Geffon
Quoted 6 lines · Dmitry Safonov

As kernel expect to see only one of such mappings, any further operations on the VMA-copy may be unexpected by the kernel. Maybe it's being on the safe side, but there doesn't seem to be any expected use-case for this, so restrict it now.

Quoted 7 lines
Quoted 2 lines
Fixes commit e346b3813067 ("mm/mremap: add MREMAP_DONTUNMAP to mremap()")

I don't think this situation can ever happen MREMAP_DONTUNMAP is already restricted to anonymous mappings (defined as not having vm_ops) and vma_to_resize checks that the mapping is anonymous before move_vma is called.

I've looked again now, I think it is possible. One can call MREMAP_DONTUNMAP without MREMAP_FIXED and without resizing. So that the old VMA is copied at some free address.

The calltrace would be: mremap()=>move_vma()
[under if (flags & MREMAP_MAYMOVE)].

On the other side I agree with you that the fix could have been better if I realized the semantics that MREMAP_DONTUNMAP should only work with anonymous mappings.

Quoted 7 lines

Probably, a better fix would be to move
: if (flags & MREMAP_DONTUNMAP && (!vma_is_anonymous(vma) ||
: vma->vm_flags & VM_SHARED))
: return ERR_PTR(-EINVAL);

from vma_to_resize() into the mremap() syscall directly. What do you think?

Ok, I've misread the code now, it checks vma_to_resize() before. I'll send a revert to this one.

Thanks for noticing,
Dima

#10 lore raw

PATCH 4/6 vm_ops: Rename .split() callback to .may_split()

In reply to Dmitry Safonov

Rename the callback to reflect that it's not called *on* or *after* split, but rather some time before the splitting to check if it's possible.

---
 drivers/dax/device.c | 4 ++--
 include/linux/mm.h   | 3 ++-
 ipc/shm.c            | 8 ++++----
 mm/hugetlb.c         | 2 +-
 mm/mmap.c            | 4 ++--
 5 files changed, 11 insertions(+), 10 deletions(-)
drivers/dax/device.c +2 -2
index 1e89513f3c59..223dd1d13d5c 100644
@@ -276,7 +276,7 @@ static vm_fault_t dev_dax_fault(struct vm_fault *vmf)
 	return dev_dax_huge_fault(vmf, PE_SIZE_PTE);
 }
 
-static int dev_dax_split(struct vm_area_struct *vma, unsigned long addr)
+static int dev_dax_may_split(struct vm_area_struct *vma, unsigned long addr)
 {
 	struct file *filp = vma->vm_file;
 	struct dev_dax *dev_dax = filp->private_data;
@@ -299,7 +299,7 @@ static unsigned long dev_dax_pagesize(struct vm_area_struct *vma)
 static const struct vm_operations_struct dax_vm_ops = {
 	.fault = dev_dax_fault,
 	.huge_fault = dev_dax_huge_fault,
-	.split = dev_dax_split,
+	.may_split = dev_dax_may_split,
 	.pagesize = dev_dax_pagesize,
 };
 
include/linux/mm.h +2 -1
index fd51a4a1f722..90887661ef44 100644
@@ -549,7 +549,8 @@ enum page_entry_size {
 struct vm_operations_struct {
 	void (*open)(struct vm_area_struct * area);
 	void (*close)(struct vm_area_struct * area);
-	int (*split)(struct vm_area_struct * area, unsigned long addr);
+	/* Called any time before splitting to check if it's allowed */
+	int (*may_split)(struct vm_area_struct *area, unsigned long addr);
 	int (*mremap)(struct vm_area_struct *area, unsigned long flags);
 	vm_fault_t (*fault)(struct vm_fault *vmf);
 	vm_fault_t (*huge_fault)(struct vm_fault *vmf,
ipc/shm.c +4 -4
index e25c7c6106bc..febd88daba8c 100644
@@ -434,13 +434,13 @@ static vm_fault_t shm_fault(struct vm_fault *vmf)
 	return sfd->vm_ops->fault(vmf);
 }
 
-static int shm_split(struct vm_area_struct *vma, unsigned long addr)
+static int shm_may_split(struct vm_area_struct *vma, unsigned long addr)
 {
 	struct file *file = vma->vm_file;
 	struct shm_file_data *sfd = shm_file_data(file);
 
-	if (sfd->vm_ops->split)
-		return sfd->vm_ops->split(vma, addr);
+	if (sfd->vm_ops->may_split)
+		return sfd->vm_ops->may_split(vma, addr);
 
 	return 0;
 }
@@ -582,7 +582,7 @@ static const struct vm_operations_struct shm_vm_ops = {
 	.open	= shm_open,	/* callback for a new vm-area open */
 	.close	= shm_close,	/* callback for when the vm-area is released */
 	.fault	= shm_fault,
-	.split	= shm_split,
+	.may_split = shm_may_split,
 	.pagesize = shm_pagesize,
 #if defined(CONFIG_NUMA)
 	.set_policy = shm_set_policy,
mm/hugetlb.c +1 -1
index 67fc6383995b..c4dc0e453be1 100644
@@ -3759,7 +3759,7 @@ const struct vm_operations_struct hugetlb_vm_ops = {
 	.fault = hugetlb_vm_op_fault,
 	.open = hugetlb_vm_op_open,
 	.close = hugetlb_vm_op_close,
-	.split = hugetlb_vm_op_split,
+	.may_split = hugetlb_vm_op_split,
 	.pagesize = hugetlb_vm_op_pagesize,
 };
 
mm/mmap.c +2 -2
index 50f853b0ec39..a62cb3ccafce 100644
@@ -2693,8 +2693,8 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
 	struct vm_area_struct *new;
 	int err;
 
-	if (vma->vm_ops && vma->vm_ops->split) {
-		err = vma->vm_ops->split(vma, addr);
+	if (vma->vm_ops && vma->vm_ops->may_split) {
+		err = vma->vm_ops->may_split(vma, addr);
 		if (err)
 			return err;
 	}
Signature · 1 line
2.28.0
#11 lore raw

PATCH 5/6 mremap: Check if it's possible to split original vma

In reply to Dmitry Safonov

If original VMA can't be split at the desired address, do_munmap() will fail and leave both new-copied VMA and old VMA. De-facto it's MREMAP_DONTUNMAP behaviour, which is unexpected.

Currently, it may fail such way for hugetlbfs and dax device mappings.

Minimize such unpleasant situations to OOM by checking .may_split() before attempting to create a VMA copy.

---
 mm/mremap.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
mm/mremap.c +10 -1
index 898e9818ba6d..3c4047c23673 100644
@@ -343,7 +343,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 	unsigned long excess = 0;
 	unsigned long hiwater_vm;
 	int split = 0;
-	int err;
+	int err = 0;
 	bool need_rmap_locks;
 
 	/*
@@ -353,6 +353,15 @@ static unsigned long move_vma(struct vm_area_struct *vma,
 	if (mm->map_count >= sysctl_max_map_count - 3)
 		return -ENOMEM;
 
+	if (vma->vm_ops && vma->vm_ops->may_split) {
+		if (vma->vm_start != old_addr)
+			err = vma->vm_ops->may_split(vma, old_addr);
+		if (!err && vma->vm_end != old_addr + old_len)
+			err = vma->vm_ops->may_split(vma, old_addr + old_len);
+		if (err)
+			return err;
+	}
+
 	/*
 	 * Advise KSM to break any KSM pages in the area to be moved:
 	 * it would be confusing if they were to turn up at the new
Signature · 1 line
2.28.0
#12 lore raw

PATCH 6/6 mm: Forbid splitting special mappings

In reply to Dmitry Safonov

Don't allow splitting of vm_special_mapping's.
It affects vdso/vvar areas. Uprobes have only one page in xol_area so they aren't affected.

Those restrictions were enforced by checks in .mremap() callbacks. Restrict resizing with generic .split() callback.

---
 arch/arm/kernel/vdso.c    |  9 ---------
 arch/arm64/kernel/vdso.c  | 41 +++------------------------------------
 arch/mips/vdso/genvdso.c  |  4 ----
 arch/s390/kernel/vdso.c   | 11 +----------
 arch/x86/entry/vdso/vma.c | 17 ----------------
 mm/mmap.c                 | 12 ++++++++++++
 6 files changed, 16 insertions(+), 78 deletions(-)
arch/arm/kernel/vdso.c +0 -9
index fddd08a6e063..3408269d19c7 100644
@@ -50,15 +50,6 @@ static const struct vm_special_mapping vdso_data_mapping = {
 static int vdso_mremap(const struct vm_special_mapping *sm,
 		struct vm_area_struct *new_vma)
 {
-	unsigned long new_size = new_vma->vm_end - new_vma->vm_start;
-	unsigned long vdso_size;
-
-	/* without VVAR page */
-	vdso_size = (vdso_total_pages - 1) << PAGE_SHIFT;
-
-	if (vdso_size != new_size)
-		return -EINVAL;
-
 	current->mm->context.vdso = new_vma->vm_start;
 
 	return 0;
arch/arm64/kernel/vdso.c +3 -38

81 lines not on this page · raw message

arch/mips/vdso/genvdso.c +0 -4
index abb06ae04b40..09e30eb4be86 100644
@@ -263,10 +263,6 @@ int main(int argc, char **argv)
 	fprintf(out_file, "	const struct vm_special_mapping *sm,\n");
 	fprintf(out_file, "	struct vm_area_struct *new_vma)\n");
 	fprintf(out_file, "{\n");
-	fprintf(out_file, "	unsigned long new_size =\n");
-	fprintf(out_file, "	new_vma->vm_end - new_vma->vm_start;\n");
-	fprintf(out_file, "	if (vdso_image.size != new_size)\n");
-	fprintf(out_file, "		return -EINVAL;\n");
 	fprintf(out_file, "	current->mm->context.vdso =\n");
 	fprintf(out_file, "	(void *)(new_vma->vm_start);\n");
 	fprintf(out_file, "	return 0;\n");
arch/s390/kernel/vdso.c +1 -10
index c4baefaa6e34..291ead792d64 100644
@@ -59,17 +59,8 @@ static vm_fault_t vdso_fault(const struct vm_special_mapping *sm,
 static int vdso_mremap(const struct vm_special_mapping *sm,
 		       struct vm_area_struct *vma)
 {
-	unsigned long vdso_pages;
-
-	vdso_pages = vdso64_pages;
-
-	if ((vdso_pages << PAGE_SHIFT) != vma->vm_end - vma->vm_start)
-		return -EINVAL;
-
-	if (WARN_ON_ONCE(current->mm != vma->vm_mm))
-		return -EFAULT;
-
 	current->mm->context.vdso_base = vma->vm_start;
+
 	return 0;
 }
 
arch/x86/entry/vdso/vma.c +0 -17
index 9185cb1d13b9..0c942b31825d 100644
@@ -89,30 +89,14 @@ static void vdso_fix_landing(const struct vdso_image *image,
 static int vdso_mremap(const struct vm_special_mapping *sm,
 		struct vm_area_struct *new_vma)
 {
-	unsigned long new_size = new_vma->vm_end - new_vma->vm_start;
 	const struct vdso_image *image = current->mm->context.vdso_image;
 
-	if (image->size != new_size)
-		return -EINVAL;
-
 	vdso_fix_landing(image, new_vma);
 	current->mm->context.vdso = (void __user *)new_vma->vm_start;
 
 	return 0;
 }
 
-static int vvar_mremap(const struct vm_special_mapping *sm,
-		struct vm_area_struct *new_vma)
-{
-	const struct vdso_image *image = new_vma->vm_mm->context.vdso_image;
-	unsigned long new_size = new_vma->vm_end - new_vma->vm_start;
-
-	if (new_size != -image->sym_vvar_start)
-		return -EINVAL;
-
-	return 0;
-}
-
 #ifdef CONFIG_TIME_NS
 static struct page *find_timens_vvar_page(struct vm_area_struct *vma)
 {
@@ -252,7 +236,6 @@ static const struct vm_special_mapping vdso_mapping = {
 static const struct vm_special_mapping vvar_mapping = {
 	.name = "[vvar]",
 	.fault = vvar_fault,
-	.mremap = vvar_mremap,
 };
 
 /*
mm/mmap.c +12 -0
index a62cb3ccafce..41100f6505ff 100644
@@ -3389,6 +3389,17 @@ static int special_mapping_mremap(struct vm_area_struct *new_vma,
 	return 0;
 }
 
+static int special_mapping_split(struct vm_area_struct *vma, unsigned long addr)
+{
+	/*
+	 * Forbid splitting special mappings - kernel has expectations over
+	 * the number of pages in mapping. Together with VM_DONTEXPAND
+	 * the size of vma should stay the same over the special mapping's
+	 * lifetime.
+	 */
+	return -EINVAL;
+}
+
 static const struct vm_operations_struct special_mapping_vmops = {
 	.close = special_mapping_close,
 	.fault = special_mapping_fault,
@@ -3396,6 +3407,7 @@ static const struct vm_operations_struct special_mapping_vmops = {
 	.name = special_mapping_name,
 	/* vDSO code relies that VVAR can't be accessed remotely */
 	.access = NULL,
+	.may_split = special_mapping_split,
 };
 
 static const struct vm_operations_struct legacy_special_mapping_vmops = {
Signature · 1 line
2.28.0
maintainer Acked-by on v1 6/6
#13 lore raw

Re: [PATCH 6/6] mm: Forbid splitting special mappings

Quoted 8 lines · Dmitry Safonov

Don't allow splitting of vm_special_mapping's.
It affects vdso/vvar areas. Uprobes have only one page in xol_area so they aren't affected.

Those restrictions were enforced by checks in .mremap() callbacks. Restrict resizing with generic .split() callback.

Quoted 8 lines
---
 arch/arm/kernel/vdso.c    |  9 ---------
 arch/arm64/kernel/vdso.c  | 41 +++------------------------------------
 arch/mips/vdso/genvdso.c  |  4 ----
 arch/s390/kernel/vdso.c   | 11 +----------
 arch/x86/entry/vdso/vma.c | 17 ----------------
 mm/mmap.c                 | 12 ++++++++++++
 6 files changed, 16 insertions(+), 78 deletions(-)

For arm64:

Will

maintainer
#14 lore raw

Re: [PATCH 6/6] mm: Forbid splitting special mappings

Quoted 18 lines · Will Deacon
Quoted 8 lines · Dmitry Safonov

Don't allow splitting of vm_special_mapping's.
It affects vdso/vvar areas. Uprobes have only one page in xol_area so they aren't affected.

Those restrictions were enforced by checks in .mremap() callbacks. Restrict resizing with generic .split() callback.

Quoted 8 lines
---
 arch/arm/kernel/vdso.c    |  9 ---------
 arch/arm64/kernel/vdso.c  | 41 +++------------------------------------
 arch/mips/vdso/genvdso.c  |  4 ----
 arch/s390/kernel/vdso.c   | 11 +----------
 arch/x86/entry/vdso/vma.c | 17 ----------------
 mm/mmap.c                 | 12 ++++++++++++
 6 files changed, 16 insertions(+), 78 deletions(-)
Quoted 3 lines

For arm64:

Wait -- this got merged ages ago! Why am I reading such old email?

Will

#15 lore raw

Re: [PATCH 6/6] mm: Forbid splitting special mappings

Quoted 19 lines · Will Deacon
Quoted 18 lines · Will Deacon
Quoted 16 lines · Dmitry Safonov

Don't allow splitting of vm_special_mapping's.
It affects vdso/vvar areas. Uprobes have only one page in xol_area so they aren't affected.

Those restrictions were enforced by checks in .mremap() callbacks. Restrict resizing with generic .split() callback.

---
 arch/arm/kernel/vdso.c    |  9 ---------
 arch/arm64/kernel/vdso.c  | 41 +++------------------------------------
 arch/mips/vdso/genvdso.c  |  4 ----
 arch/s390/kernel/vdso.c   | 11 +----------
 arch/x86/entry/vdso/vma.c | 17 ----------------
 mm/mmap.c                 | 12 ++++++++++++
 6 files changed, 16 insertions(+), 78 deletions(-)
Quoted 5 lines
Quoted 3 lines

For arm64:

Wait -- this got merged ages ago! Why am I reading such old email?

It's alright, thanks for looking at this again!

Signature · 1 line
          Dmitry