Skip to contentSkip to conversation

userfaultfd remap option

PATCH Superseded Suren Baghdasaryan 5 participants 28 messages Sep 14, 2023 – Sep 22, 2023

Subsystems (7)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Suren Baghdasaryan

author reviewer
#0 lore raw

cover userfaultfd remap option

This patch series introduces UFFDIO_REMAP feature to userfaultfd, which has long been implemented and maintained by Andrea in his local tree [1], but was not upstreamed due to lack of use cases where this approach would be better than allocating a new page and copying the contents.

UFFDIO_COPY performs ~20% better than UFFDIO_REMAP when the application needs pages to be allocated [2]. However, with UFFDIO_REMAP, if pages are available (in userspace) for recycling, as is usually the case in heap compaction algorithms, then we can avoid the page allocation and memcpy (done by UFFDIO_COPY). Also, since the pages are recycled in the userspace, we avoid the need to release (via madvise) the pages back to the kernel [3].
We see over 40% reduction (on a Google pixel 6 device) in the compacting thread’s completion time by using UFFDIO_REMAP vs. UFFDIO_COPY. This was measured using a benchmark that emulates a heap compaction implementation using userfaultfd (to allow concurrent accesses by application threads). More details of the usecase are explained in [3].

Furthermore, UFFDIO_REMAP enables remapping swapped-out pages without touching them within the same vma. Today, it can only be done by mremap, however it forces splitting the vma.

Main changes since Andrea's last version [1]:
1. Trivial translations from page to folio, mmap_sem to mmap_lock
2. Replace pmd_trans_unstable() with pte_offset_map_nolock() and handle its possible failure
3. Move pte mapping into remap_pages_pte to allow for retries when source page or anon_vma is contended. Since pte_offset_map_nolock() start RCU read section, we can't block anymore after mapping a pte, so have to unmap the ptesm do the locking and retry.
4. Add and use anon_vma_trylock_write() to avoid blocking while in RCU read section.
5. Accommodate changes in mmu_notifier_range_init() API, switch to mmu_notifier_invalidate_range_start_nonblock() to avoid blocking while in RCU read section.
6. Open-code now removed __swp_swapcount()
7. Replace pmd_read_atomic() with pmdp_get_lockless()
8. Add new selftest for UFFDIO_REMAP

[1] https://gitlab.com/aarcange/aa/-/commit/2aec7aea56b10438a3881a20a411aa4b1fc19e92
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/

Andrea Arcangeli (2):
userfaultfd: UFFDIO_REMAP: rmap preparation
userfaultfd: UFFDIO_REMAP uABI

Suren Baghdasaryan (1):
selftests/mm: add UFFDIO_REMAP ioctl test

 fs/userfaultfd.c                             |  49 ++
 include/linux/rmap.h                         |   5 +
 include/linux/userfaultfd_k.h                |  17 +
 include/uapi/linux/userfaultfd.h             |  22 +
 mm/huge_memory.c                             | 118 ++++
 mm/khugepaged.c                              |   3 +
 mm/rmap.c                                    |  13 +
 mm/userfaultfd.c                             | 586 +++++++++++++++++++
 tools/testing/selftests/mm/uffd-common.c     |  34 +-
 tools/testing/selftests/mm/uffd-common.h     |   1 +
 tools/testing/selftests/mm/uffd-unit-tests.c |  62 ++
 11 files changed, 908 insertions(+), 2 deletions(-)
Signature · 1 line
2.42.0.283.g2d96d420d3-goog
author reviewer
#1 lore raw

PATCH 1/3 userfaultfd: UFFDIO_REMAP: rmap preparation

From: Andrea Arcangeli <[email protected]>

As far as the rmap code is concerned, UFFDIO_REMAP only alters the page->mapping and page->index. It does it while holding the page lock. However folio_referenced() is doing rmap walks without taking the folio lock first, so folio_lock_anon_vma_read() must be updated to re-check that the folio->mapping didn't change after we obtained the anon_vma read lock.

UFFDIO_REMAP takes the anon_vma lock for writing before altering the folio->mapping, so if the folio->mapping is still the same after obtaining the anon_vma read lock (without the folio lock), the rmap walks can go ahead safely (and UFFDIO_REMAP will wait the rmap walk to complete before proceeding).

UFFDIO_REMAP serializes against itself with the folio lock.

All other places taking the anon_vma lock while holding the mmap_lock for writing, don't need to check if the folio->mapping has changed after taking the anon_vma lock, regardless of the folio lock, because UFFDIO_REMAP holds the mmap_lock for reading.

There's one constraint enforced to allow this simplification: the source pages passed to UFFDIO_REMAP must be mapped only in one vma, but this constraint is an acceptable tradeoff for UFFDIO_REMAP users.

The source addresses passed to UFFDIO_REMAP can be set as VM_DONTCOPY with MADV_DONTFORK to avoid any risk of the mapcount of the pages increasing if some thread of the process forks() before UFFDIO_REMAP run.

---
 mm/rmap.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
mm/rmap.c +13 -0
index ec7f8e6c9e48..c1ebbd23fa61 100644
@@ -542,6 +542,7 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio,
 	struct anon_vma *root_anon_vma;
 	unsigned long anon_mapping;
 
+repeat:
 	rcu_read_lock();
 	anon_mapping = (unsigned long)READ_ONCE(folio->mapping);
 	if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON)
@@ -586,6 +587,18 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio,
 	rcu_read_unlock();
 	anon_vma_lock_read(anon_vma);
 
+	/*
+	 * Check if UFFDIO_REMAP changed the anon_vma. This is needed
+	 * because we don't assume the folio was locked.
+	 */
+	if (unlikely((unsigned long) READ_ONCE(folio->mapping) !=
+		     anon_mapping)) {
+		anon_vma_unlock_read(anon_vma);
+		put_anon_vma(anon_vma);
+		anon_vma = NULL;
+		goto repeat;
+	}
+
 	if (atomic_dec_and_test(&anon_vma->refcount)) {
 		/*
 		 * Oops, we held the last refcount, release the lock
Signature · 1 line
2.42.0.283.g2d96d420d3-goog
#2 lore raw

Re: [PATCH 1/3] userfaultfd: UFFDIO_REMAP: rmap preparation

Quoted 4 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

As far as the rmap code is concerned, UFFDIO_REMAP only alters the page->mapping and page->index. It does it while holding the page

folio->mapping and folio->index. It does it while holding the folio

No other objections ...

author reviewer
#3 lore raw

Re: [PATCH 1/3] userfaultfd: UFFDIO_REMAP: rmap preparation

Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

As far as the rmap code is concerned, UFFDIO_REMAP only alters the page->mapping and page->index. It does it while holding the page

folio->mapping and folio->index. It does it while holding the folio

Ack.

Quoted 2 lines

No other objections ...

Thanks for the review!

author reviewer
#4 lore raw

PATCH 2/3 userfaultfd: UFFDIO_REMAP uABI

In reply to Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

---
 fs/userfaultfd.c                 |  49 +++
 include/linux/rmap.h             |   5 +
 include/linux/userfaultfd_k.h    |  17 +
 include/uapi/linux/userfaultfd.h |  22 ++
 mm/huge_memory.c                 | 118 +++++++
 mm/khugepaged.c                  |   3 +
 mm/userfaultfd.c                 | 586 +++++++++++++++++++++++++++++++
 7 files changed, 800 insertions(+)
fs/userfaultfd.c +49 -0
index 56eaae9dac1a..7bf64e7541c1 100644
@@ -2027,6 +2027,52 @@ static inline unsigned int uffd_ctx_features(__u64 user_features)
 	return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED;
 }
 
+static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
+			     unsigned long arg)
+{
+	__s64 ret;
+	struct uffdio_remap uffdio_remap;
+	struct uffdio_remap __user *user_uffdio_remap;
+	struct userfaultfd_wake_range range;
+
+	user_uffdio_remap = (struct uffdio_remap __user *) arg;
+
+	ret = -EFAULT;
+	if (copy_from_user(&uffdio_remap, user_uffdio_remap,
+			   /* don't copy "remap" last field */
+			   sizeof(uffdio_remap)-sizeof(__s64)))
+		goto out;
+
+	ret = validate_range(ctx->mm, uffdio_remap.dst, uffdio_remap.len);
+	if (ret)
+		goto out;
+	ret = validate_range(current->mm, uffdio_remap.src, uffdio_remap.len);
+	if (ret)
+		goto out;
+	ret = -EINVAL;
+	if (uffdio_remap.mode & ~(UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES|
+				  UFFDIO_REMAP_MODE_DONTWAKE))
+		goto out;
+
+	ret = remap_pages(ctx->mm, current->mm,
+			  uffdio_remap.dst, uffdio_remap.src,
+			  uffdio_remap.len, uffdio_remap.mode);
+	if (unlikely(put_user(ret, &user_uffdio_remap->remap)))
+		return -EFAULT;
+	if (ret < 0)
+		goto out;
+	/* len == 0 would wake all */
+	BUG_ON(!ret);
+	range.len = ret;
+	if (!(uffdio_remap.mode & UFFDIO_REMAP_MODE_DONTWAKE)) {
+		range.start = uffdio_remap.dst;
+		wake_userfault(ctx, &range);
+	}
+	ret = range.len == uffdio_remap.len ? 0 : -EAGAIN;
+out:
+	return ret;
+}
+
 /*
  * userland asks for a certain API version and we return which bits
  * and ioctl commands are implemented in this kernel for such API
@@ -2113,6 +2159,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,
 	case UFFDIO_ZEROPAGE:
 		ret = userfaultfd_zeropage(ctx, arg);
 		break;
+	case UFFDIO_REMAP:
+		ret = userfaultfd_remap(ctx, arg);
+		break;
 	case UFFDIO_WRITEPROTECT:
 		ret = userfaultfd_writeprotect(ctx, arg);
 		break;
include/linux/rmap.h +5 -0
index 51cc21ebb568..614c4b439907 100644
@@ -121,6 +121,11 @@ static inline void anon_vma_lock_write(struct anon_vma *anon_vma)
 	down_write(&anon_vma->root->rwsem);
 }
 
+static inline int anon_vma_trylock_write(struct anon_vma *anon_vma)
+{
+	return down_write_trylock(&anon_vma->root->rwsem);
+}
+
 static inline void anon_vma_unlock_write(struct anon_vma *anon_vma)
 {
 	up_write(&anon_vma->root->rwsem);
include/linux/userfaultfd_k.h +17 -0
index ac8c6854097c..2bc807dc390b 100644
@@ -93,6 +93,23 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm,
 extern long uffd_wp_range(struct vm_area_struct *vma,
 			  unsigned long start, unsigned long len, bool enable_wp);
 
+/* remap_pages */
+extern void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern ssize_t remap_pages(struct mm_struct *dst_mm,
+			   struct mm_struct *src_mm,
+			   unsigned long dst_start,
+			   unsigned long src_start,
+			   unsigned long len, __u64 flags);
+extern int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+				struct mm_struct *src_mm,
+				pmd_t *dst_pmd, pmd_t *src_pmd,
+				pmd_t dst_pmdval,
+				struct vm_area_struct *dst_vma,
+				struct vm_area_struct *src_vma,
+				unsigned long dst_addr,
+				unsigned long src_addr);
+
 /* mm helpers */
 static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma,
 					struct vm_userfaultfd_ctx vm_ctx)
include/uapi/linux/userfaultfd.h +22 -0
index 62151706c5a3..22d1c43e39f9 100644
@@ -49,6 +49,7 @@
 	((__u64)1 << _UFFDIO_WAKE |		\
 	 (__u64)1 << _UFFDIO_COPY |		\
 	 (__u64)1 << _UFFDIO_ZEROPAGE |		\
+	 (__u64)1 << _UFFDIO_REMAP |		\
 	 (__u64)1 << _UFFDIO_WRITEPROTECT |	\
 	 (__u64)1 << _UFFDIO_CONTINUE |		\
 	 (__u64)1 << _UFFDIO_POISON)
@@ -72,6 +73,7 @@
 #define _UFFDIO_WAKE			(0x02)
 #define _UFFDIO_COPY			(0x03)
 #define _UFFDIO_ZEROPAGE		(0x04)
+#define _UFFDIO_REMAP			(0x05)
 #define _UFFDIO_WRITEPROTECT		(0x06)
 #define _UFFDIO_CONTINUE		(0x07)
 #define _UFFDIO_POISON			(0x08)
@@ -91,6 +93,8 @@
 				      struct uffdio_copy)
 #define UFFDIO_ZEROPAGE		_IOWR(UFFDIO, _UFFDIO_ZEROPAGE,	\
 				      struct uffdio_zeropage)
+#define UFFDIO_REMAP		_IOWR(UFFDIO, _UFFDIO_REMAP,	\
+				      struct uffdio_remap)
 #define UFFDIO_WRITEPROTECT	_IOWR(UFFDIO, _UFFDIO_WRITEPROTECT, \
 				      struct uffdio_writeprotect)
 #define UFFDIO_CONTINUE		_IOWR(UFFDIO, _UFFDIO_CONTINUE,	\
@@ -340,6 +344,24 @@ struct uffdio_poison {
 	__s64 updated;
 };
 
+struct uffdio_remap {
+	__u64 dst;
+	__u64 src;
+	__u64 len;
+	/*
+	 * Especially if used to atomically remove memory from the
+	 * address space the wake on the dst range is not needed.
+	 */
+#define UFFDIO_REMAP_MODE_DONTWAKE		((__u64)1<<0)
+#define UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES	((__u64)1<<1)
+	__u64 mode;
+	/*
+	 * "remap" is written by the ioctl and must be at the end: the
+	 * copy_from_user will not read the last 8 bytes.
+	 */
+	__s64 remap;
+};
+
 /*
  * Flags for the userfaultfd(2) system call itself.
  */
mm/huge_memory.c +118 -0

124 lines not on this page · raw message

mm/khugepaged.c +3 -0
index 88433cc25d8a..af23248b3551 100644
@@ -1135,6 +1135,9 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
 	 * Prevent all access to pagetables with the exception of
 	 * gup_fast later handled by the ptep_clear_flush and the VM
 	 * handled by the anon_vma lock + PG_lock.
+	 *
+	 * UFFDIO_REMAP is prevented to race as well thanks to the
+	 * mmap_lock.
 	 */
 	mmap_write_lock(mm);
 	result = hugepage_vma_revalidate(mm, address, true, &vma, cc);
mm/userfaultfd.c +586 -0

589 lines not on this page · raw message

Signature · 1 line
2.42.0.283.g2d96d420d3-goog
#5 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 11 lines · Suren Baghdasaryan

+++ b/include/linux/userfaultfd_k.h

include/linux/userfaultfd_k.h +6 -0
@@ -93,6 +93,23 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm,
 extern long uffd_wp_range(struct vm_area_struct *vma,
 			  unsigned long start, unsigned long len, bool enable_wp);
 
+/* remap_pages */
+extern void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern ssize_t remap_pages(struct mm_struct *dst_mm,
+			   struct mm_struct *src_mm,
+			   unsigned long dst_start,

include/linux/userfaultfd_k.h extern int mwriteprotect_range(struct mm_struct *dst_mm, · line 111

Quoted 10 lines
include/linux/userfaultfd_k.h +10 -0
+			   unsigned long src_start,
+			   unsigned long len, __u64 flags);
+extern int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+				struct mm_struct *src_mm,
+				pmd_t *dst_pmd, pmd_t *src_pmd,
+				pmd_t dst_pmdval,
+				struct vm_area_struct *dst_vma,
+				struct vm_area_struct *src_vma,
+				unsigned long dst_addr,
+				unsigned long src_addr);

Drop the 'extern' markers from function declarations.

Quoted 20 lines
mm/huge_memory.c +20 -0
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+			 struct mm_struct *src_mm,
+			 pmd_t *dst_pmd, pmd_t *src_pmd,
+			 pmd_t dst_pmdval,
+			 struct vm_area_struct *dst_vma,
+			 struct vm_area_struct *src_vma,
+			 unsigned long dst_addr,
+			 unsigned long src_addr)
+{
+	pmd_t _dst_pmd, src_pmdval;
+	struct page *src_page;
+	struct anon_vma *src_anon_vma, *dst_anon_vma;
+	spinlock_t *src_ptl, *dst_ptl;
+	pgtable_t pgtable;
+	struct mmu_notifier_range range;
+
+	src_pmdval = *src_pmd;
+	src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+	BUG_ON(!pmd_trans_huge(src_pmdval));

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1974

Quoted 10 lines
mm/huge_memory.c +10 -0
+	BUG_ON(!pmd_none(dst_pmdval));
+	BUG_ON(!spin_is_locked(src_ptl));
+	mmap_assert_locked(src_mm);
+	mmap_assert_locked(dst_mm);
+	BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+	BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+	src_page = pmd_page(src_pmdval);
+	BUG_ON(!PageHead(src_page));
+	BUG_ON(!PageAnon(src_page));

Better to add a src_folio = page_folio(src_page);
and then folio_test_anon() here.

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1975

Quoted 1 line
mm/huge_memory.c +1 -0
+	if (unlikely(page_mapcount(src_page) != 1)) {

Brr, this is going to miss PTE mappings of this folio. I think you actually want folio_mapcount() instead, although it'd be more efficient to look at folio->_entire_mapcount == 1 and _nr_pages_mapped == 0. Not wure what a good name for that predicate would be.

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1980

Quoted 1 line
mm/huge_memory.c +1 -0
+	get_page(src_page);

folio_get()

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1988

Quoted 2 lines
mm/huge_memory.c +2 -0
+	/* block all concurrent rmap walks */
+	lock_page(src_page);

folio_lock()

Quoted 6 lines
mm/huge_memory.c +6 -0
+	/*
+	 * split_huge_page walks the anon_vma chain without the page
+	 * lock. Serialize against it with the anon_vma lock, the page
+	 * lock is not enough.
+	 */
+	src_anon_vma = folio_get_anon_vma(page_folio(src_page));

...

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (!src_anon_vma) {
+		unlock_page(src_page);

folio_unlock()

Review

Quoted 1 line
(quoted diff) +1 -0
+		put_page(src_page);

folio_put()

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2008

Quoted 3 lines
mm/huge_memory.c +3 -0
+	if (unlikely(!pmd_same(*src_pmd, src_pmdval) ||
+		     !pmd_same(*dst_pmd, dst_pmdval) ||
+		     page_mapcount(src_page) != 1)) {

similar folio_total_mapcount()

Quoted 5 lines
mm/huge_memory.c +5 -0
+		double_pt_unlock(src_ptl, dst_ptl);
+		anon_vma_unlock_write(src_anon_vma);
+		put_anon_vma(src_anon_vma);
+		unlock_page(src_page);
+		put_page(src_page);

...

Quoted 4 lines
mm/huge_memory.c +4 -0
+	BUG_ON(!PageHead(src_page));
+	BUG_ON(!PageAnon(src_page));
+	/* the PT lock is enough to keep the page pinned now */
+	put_page(src_page);

...

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2025

Quoted 3 lines
mm/huge_memory.c +3 -0
+	dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
+	WRITE_ONCE(src_page->mapping, (struct address_space *) dst_anon_vma);
+	WRITE_ONCE(src_page->index, linear_page_index(dst_vma, dst_addr));

Definitely want to use the folio here.

Quoted 2 lines
mm/huge_memory.c +2 -0
+	/* unblock rmap walks */
+	unlock_page(src_page);

...

Quoted 76 lines
mm/userfaultfd.c +76 -0
+static int remap_pages_pte(struct mm_struct *dst_mm,
+			   struct mm_struct *src_mm,
+			   pmd_t *dst_pmd,
+			   pmd_t *src_pmd,
+			   struct vm_area_struct *dst_vma,
+			   struct vm_area_struct *src_vma,
+			   unsigned long dst_addr,
+			   unsigned long src_addr,
+			   __u64 mode)
+{
+	swp_entry_t entry;
+	pte_t orig_src_pte, orig_dst_pte;
+	spinlock_t *src_ptl, *dst_ptl;
+	pte_t *src_pte = NULL;
+	pte_t *dst_pte = NULL;
+
+	struct folio *src_folio = NULL;
+	struct anon_vma *src_anon_vma = NULL;
+	struct anon_vma *dst_anon_vma;
+	struct mmu_notifier_range range;
+	int err = 0;
+
+retry:
+	dst_pte = pte_offset_map_nolock(dst_mm, dst_pmd, dst_addr, &dst_ptl);
+
+	/* If an huge pmd materialized from under us fail */
+	if (unlikely(!dst_pte)) {
+		err = -EFAULT;
+		goto out;
+	}
+
+	src_pte = pte_offset_map_nolock(src_mm, src_pmd, src_addr, &src_ptl);
+
+	/*
+	 * We held the mmap_lock for reading so MADV_DONTNEED
+	 * can zap transparent huge pages under us, or the
+	 * transparent huge page fault can establish new
+	 * transparent huge pages under us.
+	 */
+	if (unlikely(!src_pte)) {
+		err = -EFAULT;
+		goto out;
+	}
+
+	BUG_ON(pmd_none(*dst_pmd));
+	BUG_ON(pmd_none(*src_pmd));
+	BUG_ON(pmd_trans_huge(*dst_pmd));
+	BUG_ON(pmd_trans_huge(*src_pmd));
+
+	spin_lock(dst_ptl);
+	orig_dst_pte = *dst_pte;
+	spin_unlock(dst_ptl);
+	if (!pte_none(orig_dst_pte)) {
+		err = -EEXIST;
+		goto out;
+	}
+
+	spin_lock(src_ptl);
+	orig_src_pte = *src_pte;
+	spin_unlock(src_ptl);
+	if (pte_none(orig_src_pte)) {
+		if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES))
+			err = -ENOENT;
+		else /* nothing to do to remap a hole */
+			err = 0;
+		goto out;
+	}
+
+	if (pte_present(orig_src_pte)) {
+		if (!src_folio) {
+			struct folio *folio;
+
+			/*
+			 * Pin the page while holding the lock to be sure the
+			 * page isn't freed under us
+			 */

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 970

Quoted 10 lines
mm/userfaultfd.c +10 -0
+			spin_lock(src_ptl);
+			if (!pte_same(orig_src_pte, *src_pte)) {
+				spin_unlock(src_ptl);
+				err = -EAGAIN;
+				goto out;
+			}
+
+			folio = vm_normal_folio(src_vma, src_addr, orig_src_pte);
+			if (!folio || !folio_test_anon(folio) ||
+			    folio_estimated_sharers(folio) != 1) {

I wonder if we also want to fail if folio_test_large()? While we don't have large anon folios today, it seems to me that trying to migrate one of them like this is just not going to work.

#6 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 59 lines · Matthew Wilcox
Quoted 11 lines · Suren Baghdasaryan

+++ b/include/linux/userfaultfd_k.h

include/linux/userfaultfd_k.h +6 -0
@@ -93,6 +93,23 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm,
  extern long uffd_wp_range(struct vm_area_struct *vma,
  			  unsigned long start, unsigned long len, bool enable_wp);
 
+/* remap_pages */
+extern void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern ssize_t remap_pages(struct mm_struct *dst_mm,
+			   struct mm_struct *src_mm,
+			   unsigned long dst_start,

include/linux/userfaultfd_k.h extern int mwriteprotect_range(struct mm_struct *dst_mm, · line 111

Quoted 10 lines
include/linux/userfaultfd_k.h +10 -0
+			   unsigned long src_start,
+			   unsigned long len, __u64 flags);
+extern int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+				struct mm_struct *src_mm,
+				pmd_t *dst_pmd, pmd_t *src_pmd,
+				pmd_t dst_pmdval,
+				struct vm_area_struct *dst_vma,
+				struct vm_area_struct *src_vma,
+				unsigned long dst_addr,
+				unsigned long src_addr);

Drop the 'extern' markers from function declarations.

Quoted 20 lines
mm/huge_memory.c +20 -0
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+			 struct mm_struct *src_mm,
+			 pmd_t *dst_pmd, pmd_t *src_pmd,
+			 pmd_t dst_pmdval,
+			 struct vm_area_struct *dst_vma,
+			 struct vm_area_struct *src_vma,
+			 unsigned long dst_addr,
+			 unsigned long src_addr)
+{
+	pmd_t _dst_pmd, src_pmdval;
+	struct page *src_page;
+	struct anon_vma *src_anon_vma, *dst_anon_vma;
+	spinlock_t *src_ptl, *dst_ptl;
+	pgtable_t pgtable;
+	struct mmu_notifier_range range;
+
+	src_pmdval = *src_pmd;
+	src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+	BUG_ON(!pmd_trans_huge(src_pmdval));

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1974

Quoted 10 lines
mm/huge_memory.c +10 -0
+	BUG_ON(!pmd_none(dst_pmdval));
+	BUG_ON(!spin_is_locked(src_ptl));
+	mmap_assert_locked(src_mm);
+	mmap_assert_locked(dst_mm);
+	BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+	BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+	src_page = pmd_page(src_pmdval);
+	BUG_ON(!PageHead(src_page));
+	BUG_ON(!PageAnon(src_page));

Better to add a src_folio = page_folio(src_page);
and then folio_test_anon() here.

Quoted 6 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1975

Quoted 1 line
mm/huge_memory.c +1 -0
+	if (unlikely(page_mapcount(src_page) != 1)) {

Brr, this is going to miss PTE mappings of this folio. I think you actually want folio_mapcount() instead, although it'd be more efficient to look at folio->_entire_mapcount == 1 and _nr_pages_mapped == 0. Not wure what a good name for that predicate would be.

We have

* It only works on non shared anonymous pages because those can
* be relocated without generating non linear anon_vmas in the rmap
* code.
*
* It provides a zero copy mechanism to handle userspace page faults.
* The source vma pages should have mapcount == 1, which can be
* enforced by using madvise(MADV_DONTFORK) on src vma.

Use PageAnonExclusive(). As long as KSM is not involved and you don't use fork(), that flag should be good enough for that use case here.

[...]

Quoted 10 lines
Quoted 10 lines
mm/userfaultfd.c +10 -0
+			/*
+			 * Pin the page while holding the lock to be sure the
+			 * page isn't freed under us
+			 */
+			spin_lock(src_ptl);
+			if (!pte_same(orig_src_pte, *src_pte)) {
+				spin_unlock(src_ptl);
+				err = -EAGAIN;
+				goto out;
+			}
Quoted 8 lines

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 970

Quoted 4 lines
mm/userfaultfd.c +4 -0
+
+			folio = vm_normal_folio(src_vma, src_addr, orig_src_pte);
+			if (!folio || !folio_test_anon(folio) ||
+			    folio_estimated_sharers(folio) != 1) {

I wonder if we also want to fail if folio_test_large()? While we don't have large anon folios today, it seems to me that trying to migrate one of them like this is just not going to work.

Yes, refuse any PTE-mapped large folios.

Signature · 3 lines
Cheers,

David / dhildenb
#7 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 77 lines · David Hildenbrand
Quoted 59 lines · Matthew Wilcox
Quoted 11 lines · Suren Baghdasaryan

+++ b/include/linux/userfaultfd_k.h

include/linux/userfaultfd_k.h +6 -0
@@ -93,6 +93,23 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm,
   extern long uffd_wp_range(struct vm_area_struct *vma,
   			  unsigned long start, unsigned long len, bool enable_wp);
 
+/* remap_pages */
+extern void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern ssize_t remap_pages(struct mm_struct *dst_mm,
+			   struct mm_struct *src_mm,
+			   unsigned long dst_start,

include/linux/userfaultfd_k.h extern int mwriteprotect_range(struct mm_struct *dst_mm, · line 111

Quoted 10 lines
include/linux/userfaultfd_k.h +10 -0
+			   unsigned long src_start,
+			   unsigned long len, __u64 flags);
+extern int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+				struct mm_struct *src_mm,
+				pmd_t *dst_pmd, pmd_t *src_pmd,
+				pmd_t dst_pmdval,
+				struct vm_area_struct *dst_vma,
+				struct vm_area_struct *src_vma,
+				unsigned long dst_addr,
+				unsigned long src_addr);

Drop the 'extern' markers from function declarations.

Quoted 20 lines
mm/huge_memory.c +20 -0
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+			 struct mm_struct *src_mm,
+			 pmd_t *dst_pmd, pmd_t *src_pmd,
+			 pmd_t dst_pmdval,
+			 struct vm_area_struct *dst_vma,
+			 struct vm_area_struct *src_vma,
+			 unsigned long dst_addr,
+			 unsigned long src_addr)
+{
+	pmd_t _dst_pmd, src_pmdval;
+	struct page *src_page;
+	struct anon_vma *src_anon_vma, *dst_anon_vma;
+	spinlock_t *src_ptl, *dst_ptl;
+	pgtable_t pgtable;
+	struct mmu_notifier_range range;
+
+	src_pmdval = *src_pmd;
+	src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+	BUG_ON(!pmd_trans_huge(src_pmdval));

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1974

Quoted 10 lines
mm/huge_memory.c +10 -0
+	BUG_ON(!pmd_none(dst_pmdval));
+	BUG_ON(!spin_is_locked(src_ptl));
+	mmap_assert_locked(src_mm);
+	mmap_assert_locked(dst_mm);
+	BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+	BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+	src_page = pmd_page(src_pmdval);
+	BUG_ON(!PageHead(src_page));
+	BUG_ON(!PageAnon(src_page));

Better to add a src_folio = page_folio(src_page);
and then folio_test_anon() here.

Quoted 6 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1975

Quoted 1 line
mm/huge_memory.c +1 -0
+	if (unlikely(page_mapcount(src_page) != 1)) {

Brr, this is going to miss PTE mappings of this folio. I think you actually want folio_mapcount() instead, although it'd be more efficient to look at folio->_entire_mapcount == 1 and _nr_pages_mapped == 0. Not wure what a good name for that predicate would be.

We have

* It only works on non shared anonymous pages because those can
* be relocated without generating non linear anon_vmas in the rmap
* code.
*
* It provides a zero copy mechanism to handle userspace page faults.
* The source vma pages should have mapcount == 1, which can be
* enforced by using madvise(MADV_DONTFORK) on src vma.

Quoted 3 lines

Use PageAnonExclusive(). As long as KSM is not involved and you don't use fork(), that flag should be good enough for that use case here.

... and similarly don't do any of that swapcount stuff and only check if the swap pte is anon exclusive.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#8 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 79 lines · David Hildenbrand
Quoted 77 lines · David Hildenbrand
Quoted 59 lines · Matthew Wilcox
Quoted 11 lines · Suren Baghdasaryan

+++ b/include/linux/userfaultfd_k.h

include/linux/userfaultfd_k.h +6 -0
@@ -93,6 +93,23 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm,
   extern long uffd_wp_range(struct vm_area_struct *vma,
                      unsigned long start, unsigned long len, bool enable_wp);
 
+/* remap_pages */
+extern void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern ssize_t remap_pages(struct mm_struct *dst_mm,
+                      struct mm_struct *src_mm,
+                      unsigned long dst_start,

include/linux/userfaultfd_k.h extern int mwriteprotect_range(struct mm_struct *dst_mm, · line 111

Quoted 10 lines
include/linux/userfaultfd_k.h +10 -0
+                      unsigned long src_start,
+                      unsigned long len, __u64 flags);
+extern int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                           struct mm_struct *src_mm,
+                           pmd_t *dst_pmd, pmd_t *src_pmd,
+                           pmd_t dst_pmdval,
+                           struct vm_area_struct *dst_vma,
+                           struct vm_area_struct *src_vma,
+                           unsigned long dst_addr,
+                           unsigned long src_addr);

Drop the 'extern' markers from function declarations.

Quoted 20 lines
mm/huge_memory.c +20 -0
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                    struct mm_struct *src_mm,
+                    pmd_t *dst_pmd, pmd_t *src_pmd,
+                    pmd_t dst_pmdval,
+                    struct vm_area_struct *dst_vma,
+                    struct vm_area_struct *src_vma,
+                    unsigned long dst_addr,
+                    unsigned long src_addr)
+{
+   pmd_t _dst_pmd, src_pmdval;
+   struct page *src_page;
+   struct anon_vma *src_anon_vma, *dst_anon_vma;
+   spinlock_t *src_ptl, *dst_ptl;
+   pgtable_t pgtable;
+   struct mmu_notifier_range range;
+
+   src_pmdval = *src_pmd;
+   src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+   BUG_ON(!pmd_trans_huge(src_pmdval));

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1974

Quoted 10 lines
mm/huge_memory.c +10 -0
+   BUG_ON(!pmd_none(dst_pmdval));
+   BUG_ON(!spin_is_locked(src_ptl));
+   mmap_assert_locked(src_mm);
+   mmap_assert_locked(dst_mm);
+   BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+   BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+   src_page = pmd_page(src_pmdval);
+   BUG_ON(!PageHead(src_page));
+   BUG_ON(!PageAnon(src_page));

Better to add a src_folio = page_folio(src_page);
and then folio_test_anon() here.

Quoted 6 lines
Quoted 1 line

+ if (unlikely(page_mapcount(src_page) != 1)) {

Brr, this is going to miss PTE mappings of this folio. I think you actually want folio_mapcount() instead, although it'd be more efficient to look at folio->_entire_mapcount == 1 and _nr_pages_mapped == 0. Not wure what a good name for that predicate would be.

We have

* It only works on non shared anonymous pages because those can
* be relocated without generating non linear anon_vmas in the rmap
* code.
*
* It provides a zero copy mechanism to handle userspace page faults.
* The source vma pages should have mapcount == 1, which can be
* enforced by using madvise(MADV_DONTFORK) on src vma.

Quoted 5 lines
Quoted 3 lines

Use PageAnonExclusive(). As long as KSM is not involved and you don't use fork(), that flag should be good enough for that use case here.

... and similarly don't do any of that swapcount stuff and only check if the swap pte is anon exclusive.

I'm preparing v2 and this is the only part left for me to address but I'm not clear how. David, could you please clarify how I should be checking swap pte to be exclusive without swapcount?

Quoted 6 lines

--
Cheers,

David / dhildenb

#9 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 83 lines · Suren Baghdasaryan
Quoted 82 lines · David Hildenbrand
Quoted 80 lines · David Hildenbrand
Quoted 59 lines · Matthew Wilcox
Quoted 11 lines · Suren Baghdasaryan

+++ b/include/linux/userfaultfd_k.h

include/linux/userfaultfd_k.h +6 -0
@@ -93,6 +93,23 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm,
    extern long uffd_wp_range(struct vm_area_struct *vma,
                       unsigned long start, unsigned long len, bool enable_wp);
 
+/* remap_pages */
+extern void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern ssize_t remap_pages(struct mm_struct *dst_mm,
+                      struct mm_struct *src_mm,
+                      unsigned long dst_start,

include/linux/userfaultfd_k.h extern int mwriteprotect_range(struct mm_struct *dst_mm, · line 111

Quoted 10 lines
include/linux/userfaultfd_k.h +10 -0
+                      unsigned long src_start,
+                      unsigned long len, __u64 flags);
+extern int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                           struct mm_struct *src_mm,
+                           pmd_t *dst_pmd, pmd_t *src_pmd,
+                           pmd_t dst_pmdval,
+                           struct vm_area_struct *dst_vma,
+                           struct vm_area_struct *src_vma,
+                           unsigned long dst_addr,
+                           unsigned long src_addr);

Drop the 'extern' markers from function declarations.

Quoted 20 lines
mm/huge_memory.c +20 -0
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                    struct mm_struct *src_mm,
+                    pmd_t *dst_pmd, pmd_t *src_pmd,
+                    pmd_t dst_pmdval,
+                    struct vm_area_struct *dst_vma,
+                    struct vm_area_struct *src_vma,
+                    unsigned long dst_addr,
+                    unsigned long src_addr)
+{
+   pmd_t _dst_pmd, src_pmdval;
+   struct page *src_page;
+   struct anon_vma *src_anon_vma, *dst_anon_vma;
+   spinlock_t *src_ptl, *dst_ptl;
+   pgtable_t pgtable;
+   struct mmu_notifier_range range;
+
+   src_pmdval = *src_pmd;
+   src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+   BUG_ON(!pmd_trans_huge(src_pmdval));

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1974

Quoted 10 lines
mm/huge_memory.c +10 -0
+   BUG_ON(!pmd_none(dst_pmdval));
+   BUG_ON(!spin_is_locked(src_ptl));
+   mmap_assert_locked(src_mm);
+   mmap_assert_locked(dst_mm);
+   BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+   BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+   src_page = pmd_page(src_pmdval);
+   BUG_ON(!PageHead(src_page));
+   BUG_ON(!PageAnon(src_page));

Better to add a src_folio = page_folio(src_page);
and then folio_test_anon() here.

Quoted 6 lines
Quoted 1 line

+ if (unlikely(page_mapcount(src_page) != 1)) {

Brr, this is going to miss PTE mappings of this folio. I think you actually want folio_mapcount() instead, although it'd be more efficient to look at folio->_entire_mapcount == 1 and _nr_pages_mapped == 0. Not wure what a good name for that predicate would be.

We have

* It only works on non shared anonymous pages because those can
* be relocated without generating non linear anon_vmas in the rmap
* code.
*
* It provides a zero copy mechanism to handle userspace page faults.
* The source vma pages should have mapcount == 1, which can be
* enforced by using madvise(MADV_DONTFORK) on src vma.

Use PageAnonExclusive(). As long as KSM is not involved and you don't use fork(), that flag should be good enough for that use case here.

Quoted 6 lines
Quoted 2 lines

... and similarly don't do any of that swapcount stuff and only check if the swap pte is anon exclusive.

I'm preparing v2 and this is the only part left for me to address but I'm not clear how. David, could you please clarify how I should be checking swap pte to be exclusive without swapcount?

If you have a real swp pte (not a non-swap pte like migration entries) you should be able to just use pte_swp_exclusive().

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#10 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 88 lines · David Hildenbrand
Quoted 86 lines · Suren Baghdasaryan
Quoted 84 lines · David Hildenbrand
Quoted 77 lines · David Hildenbrand
Quoted 59 lines · Matthew Wilcox
Quoted 11 lines · Suren Baghdasaryan

+++ b/include/linux/userfaultfd_k.h

include/linux/userfaultfd_k.h +6 -0
@@ -93,6 +93,23 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm,
    extern long uffd_wp_range(struct vm_area_struct *vma,
                       unsigned long start, unsigned long len, bool enable_wp);
 
+/* remap_pages */
+extern void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern ssize_t remap_pages(struct mm_struct *dst_mm,
+                      struct mm_struct *src_mm,
+                      unsigned long dst_start,

include/linux/userfaultfd_k.h extern int mwriteprotect_range(struct mm_struct *dst_mm, · line 111

Quoted 10 lines
include/linux/userfaultfd_k.h +10 -0
+                      unsigned long src_start,
+                      unsigned long len, __u64 flags);
+extern int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                           struct mm_struct *src_mm,
+                           pmd_t *dst_pmd, pmd_t *src_pmd,
+                           pmd_t dst_pmdval,
+                           struct vm_area_struct *dst_vma,
+                           struct vm_area_struct *src_vma,
+                           unsigned long dst_addr,
+                           unsigned long src_addr);

Drop the 'extern' markers from function declarations.

Quoted 20 lines
mm/huge_memory.c +20 -0
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                    struct mm_struct *src_mm,
+                    pmd_t *dst_pmd, pmd_t *src_pmd,
+                    pmd_t dst_pmdval,
+                    struct vm_area_struct *dst_vma,
+                    struct vm_area_struct *src_vma,
+                    unsigned long dst_addr,
+                    unsigned long src_addr)
+{
+   pmd_t _dst_pmd, src_pmdval;
+   struct page *src_page;
+   struct anon_vma *src_anon_vma, *dst_anon_vma;
+   spinlock_t *src_ptl, *dst_ptl;
+   pgtable_t pgtable;
+   struct mmu_notifier_range range;
+
+   src_pmdval = *src_pmd;
+   src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+   BUG_ON(!pmd_trans_huge(src_pmdval));

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1974

Quoted 10 lines
mm/huge_memory.c +10 -0
+   BUG_ON(!pmd_none(dst_pmdval));
+   BUG_ON(!spin_is_locked(src_ptl));
+   mmap_assert_locked(src_mm);
+   mmap_assert_locked(dst_mm);
+   BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+   BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+   src_page = pmd_page(src_pmdval);
+   BUG_ON(!PageHead(src_page));
+   BUG_ON(!PageAnon(src_page));

Better to add a src_folio = page_folio(src_page);
and then folio_test_anon() here.

Quoted 6 lines
Quoted 1 line

+ if (unlikely(page_mapcount(src_page) != 1)) {

Brr, this is going to miss PTE mappings of this folio. I think you actually want folio_mapcount() instead, although it'd be more efficient to look at folio->_entire_mapcount == 1 and _nr_pages_mapped == 0. Not wure what a good name for that predicate would be.

We have

* It only works on non shared anonymous pages because those can
* be relocated without generating non linear anon_vmas in the rmap
* code.
*
* It provides a zero copy mechanism to handle userspace page faults.
* The source vma pages should have mapcount == 1, which can be
* enforced by using madvise(MADV_DONTFORK) on src vma.

Quoted 3 lines

Use PageAnonExclusive(). As long as KSM is not involved and you don't use fork(), that flag should be good enough for that use case here.

... and similarly don't do any of that swapcount stuff and only check if the swap pte is anon exclusive.

Quoted 6 lines
Quoted 3 lines

I'm preparing v2 and this is the only part left for me to address but I'm not clear how. David, could you please clarify how I should be checking swap pte to be exclusive without swapcount?

If you have a real swp pte (not a non-swap pte like migration entries) you should be able to just use pte_swp_exclusive().

Got it. Thanks!

Quoted 6 lines

--
Cheers,

David / dhildenb

#11 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

In reply to Suren Baghdasaryan
Quoted 6 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

Sorry to say, but these functions are unacceptably long. Please find ways to structure the code in a better way.

Signature · 3 lines
Cheers,

David / dhildenb
author reviewer
#12 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 11 lines · David Hildenbrand
Quoted 6 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

Sorry to say, but these functions are unacceptably long. Please find ways to structure the code in a better way.

Thanks for the comments, guys! I'll address them and will try to break the functions into smaller pieces.

Quoted 6 lines

--
Cheers,

David / dhildenb

reviewer
#13 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

In reply to Suren Baghdasaryan
Quoted 44 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

---
 fs/userfaultfd.c                 |  49 +++
 include/linux/rmap.h             |   5 +
 include/linux/userfaultfd_k.h    |  17 +
 include/uapi/linux/userfaultfd.h |  22 ++
 mm/huge_memory.c                 | 118 +++++++
 mm/khugepaged.c                  |   3 +
 mm/userfaultfd.c                 | 586 +++++++++++++++++++++++++++++++
 7 files changed, 800 insertions(+)
fs/userfaultfd.c +16 -0
index 56eaae9dac1a..7bf64e7541c1 100644
@@ -2027,6 +2027,52 @@ static inline unsigned int uffd_ctx_features(__u64 user_features)
        return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED;
 }
 
+static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
+                            unsigned long arg)
+{
+       __s64 ret;
+       struct uffdio_remap uffdio_remap;
+       struct uffdio_remap __user *user_uffdio_remap;
+       struct userfaultfd_wake_range range;
+
+       user_uffdio_remap = (struct uffdio_remap __user *) arg;
+
+       ret = -EFAULT;
+       if (copy_from_user(&uffdio_remap, user_uffdio_remap,
+                          /* don't copy "remap" last field */
+                          sizeof(uffdio_remap)-sizeof(__s64)))
+               goto out;
+

fs/userfaultfd.c static inline unsigned int uffd_ctx_features(__u64 user_features) · line 2055

Quoted 10 lines
fs/userfaultfd.c +10 -0
+       ret = validate_range(ctx->mm, uffdio_remap.dst, uffdio_remap.len);
+       if (ret)
+               goto out;
+       ret = validate_range(current->mm, uffdio_remap.src, uffdio_remap.len);
+       if (ret)
+               goto out;
+       ret = -EINVAL;
+       if (uffdio_remap.mode & ~(UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES|
+                                 UFFDIO_REMAP_MODE_DONTWAKE))
+               goto out;

Do you not need mmget_not_zero(ctx->mm) to make sure the MM can't be concurrently torn down while remap_pages() is running, similar to what the other userfaultfd ioctl handlers do?

Quoted 18 lines
fs/userfaultfd.c +18 -0
+       ret = remap_pages(ctx->mm, current->mm,
+                         uffdio_remap.dst, uffdio_remap.src,
+                         uffdio_remap.len, uffdio_remap.mode);
+       if (unlikely(put_user(ret, &user_uffdio_remap->remap)))
+               return -EFAULT;
+       if (ret < 0)
+               goto out;
+       /* len == 0 would wake all */
+       BUG_ON(!ret);
+       range.len = ret;
+       if (!(uffdio_remap.mode & UFFDIO_REMAP_MODE_DONTWAKE)) {
+               range.start = uffdio_remap.dst;
+               wake_userfault(ctx, &range);
+       }
+       ret = range.len == uffdio_remap.len ? 0 : -EAGAIN;
+out:
+       return ret;
+}

[...]

Quoted 29 lines
mm/huge_memory.c +21 -0
index 064fbd90822b..c7a9880a1f6a 100644
@@ -1932,6 +1932,124 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
        return ret;
 }
 
+#ifdef CONFIG_USERFAULTFD
+/*
+ * The PT lock for src_pmd and the mmap_lock for reading are held by
+ * the caller, but it must return after releasing the
+ * page_table_lock. We're guaranteed the src_pmd is a pmd_trans_huge
+ * until the PT lock of the src_pmd is released. Just move the page
+ * from src_pmd to dst_pmd if possible. Return zero if succeeded in
+ * moving the page, -EAGAIN if it needs to be repeated by the caller,
+ * or other errors in case of failure.
+ */
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                        struct mm_struct *src_mm,
+                        pmd_t *dst_pmd, pmd_t *src_pmd,
+                        pmd_t dst_pmdval,
+                        struct vm_area_struct *dst_vma,
+                        struct vm_area_struct *src_vma,
+                        unsigned long dst_addr,
+                        unsigned long src_addr)
+{
+       pmd_t _dst_pmd, src_pmdval;
+       struct page *src_page;

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1965

Quoted 10 lines
mm/huge_memory.c +10 -0
+       struct anon_vma *src_anon_vma, *dst_anon_vma;
+       spinlock_t *src_ptl, *dst_ptl;
+       pgtable_t pgtable;
+       struct mmu_notifier_range range;
+
+       src_pmdval = *src_pmd;
+       src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+       BUG_ON(!pmd_trans_huge(src_pmdval));
+       BUG_ON(!pmd_none(dst_pmdval));

Why can we assert that pmd_none(dst_pmdval) is true here? Can we not have concurrent faults (or userfaultfd operations) populating that PMD?

Quoted 54 lines
mm/huge_memory.c +54 -0
+       BUG_ON(!spin_is_locked(src_ptl));
+       mmap_assert_locked(src_mm);
+       mmap_assert_locked(dst_mm);
+       BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+       BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+       src_page = pmd_page(src_pmdval);
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       if (unlikely(page_mapcount(src_page) != 1)) {
+               spin_unlock(src_ptl);
+               return -EBUSY;
+       }
+
+       get_page(src_page);
+       spin_unlock(src_ptl);
+
+       mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr,
+                               src_addr + HPAGE_PMD_SIZE);
+       mmu_notifier_invalidate_range_start(&range);
+
+       /* block all concurrent rmap walks */
+       lock_page(src_page);
+
+       /*
+        * split_huge_page walks the anon_vma chain without the page
+        * lock. Serialize against it with the anon_vma lock, the page
+        * lock is not enough.
+        */
+       src_anon_vma = folio_get_anon_vma(page_folio(src_page));
+       if (!src_anon_vma) {
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+       anon_vma_lock_write(src_anon_vma);
+
+       dst_ptl = pmd_lockptr(dst_mm, dst_pmd);
+       double_pt_lock(src_ptl, dst_ptl);
+       if (unlikely(!pmd_same(*src_pmd, src_pmdval) ||
+                    !pmd_same(*dst_pmd, dst_pmdval) ||
+                    page_mapcount(src_page) != 1)) {
+               double_pt_unlock(src_ptl, dst_ptl);
+               anon_vma_unlock_write(src_anon_vma);
+               put_anon_vma(src_anon_vma);
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2029

Quoted 10 lines
mm/huge_memory.c +10 -0
+       /* the PT lock is enough to keep the page pinned now */
+       put_page(src_page);
+
+       dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
+       WRITE_ONCE(src_page->mapping, (struct address_space *) dst_anon_vma);
+       WRITE_ONCE(src_page->index, linear_page_index(dst_vma, dst_addr));
+
+       if (!pmd_same(pmdp_huge_clear_flush(src_vma, src_addr, src_pmd),
+                     src_pmdval))
+               BUG_ON(1);

I'm not sure we can assert that the PMDs are exactly equal; the CPU might have changed the A/D bits under us?

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2035

Quoted 6 lines
mm/huge_memory.c +6 -0
+       _dst_pmd = mk_huge_pmd(src_page, dst_vma->vm_page_prot);
+       _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
+       set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd);
+
+       pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd);
+       pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);

Are we allowed to move page tables between mm_structs on all architectures? The first example I found that looks a bit dodgy, looking through various architectures' pte_alloc_one(), is s390's page_table_alloc() which looks like page tables are tied to per-MM lists sometimes.
If that's not allowed, we might have to allocate a new deposit table and free the old one or something like that.

Quoted 20 lines
mm/huge_memory.c +17 -0
+       if (dst_mm != src_mm) {
+               add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+               add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR);
+       }
+       double_pt_unlock(src_ptl, dst_ptl);
+
+       anon_vma_unlock_write(src_anon_vma);
+       put_anon_vma(src_anon_vma);
+
+       /* unblock rmap walks */
+       unlock_page(src_page);
+
+       mmu_notifier_invalidate_range_end(&range);
+       return 0;
+}
+#endif /* CONFIG_USERFAULTFD */
+
 /*
  * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise.
  *

[...]

Quoted 4 lines
mm/userfaultfd.c +0 -0
index 96d9eae5c7cc..0cca60dfa8f8 100644

[...]

Quoted 4 lines
mm/userfaultfd.c +4 -0
+ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+                   unsigned long dst_start, unsigned long src_start,
+                   unsigned long len, __u64 mode)
+{

[...]

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1260

Quoted 4 lines
mm/userfaultfd.c +4 -0
+
+       if (pgprot_val(src_vma->vm_page_prot) !=
+           pgprot_val(dst_vma->vm_page_prot))
+               goto out;

Does this check intentionally allow moving pages from a PROT_READ|PROT_WRITE anonymous private VMA into a PROT_READ anonymous private VMA (on architectures like x86 and arm64 where CoW memory has the same protection flags as read-only memory), but forbid moving them from a PROT_READ|PROT_EXEC VMA into a PROT_READ VMA? I think this check needs at least a comment to explain what's going on here.

Quoted 19 lines
mm/userfaultfd.c +19 -0
+       /* only allow remapping if both are mlocked or both aren't */
+       if ((src_vma->vm_flags & VM_LOCKED) ^ (dst_vma->vm_flags & VM_LOCKED))
+               goto out;
+
+       /*
+        * Be strict and only allow remap_pages if either the src or
+        * dst range is registered in the userfaultfd to prevent
+        * userland errors going unnoticed. As far as the VM
+        * consistency is concerned, it would be perfectly safe to
+        * remove this check, but there's no useful usage for
+        * remap_pages ouside of userfaultfd registered ranges. This
+        * is after all why it is an ioctl belonging to the
+        * userfaultfd and not a syscall.
+        *
+        * Allow both vmas to be registered in the userfaultfd, just
+        * in case somebody finds a way to make such a case useful.
+        * Normally only one of the two vmas would be registered in
+        * the userfaultfd.
+        */

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1290

Quoted 10 lines
mm/userfaultfd.c +10 -0
+       if (!dst_vma->vm_userfaultfd_ctx.ctx &&
+           !src_vma->vm_userfaultfd_ctx.ctx)
+               goto out;
+
+       /*
+        * FIXME: only allow remapping across anonymous vmas,
+        * tmpfs should be added.
+        */
+       if (src_vma->vm_ops || dst_vma->vm_ops)
+               goto out;

I don't think it's okay to check for anonymous VMAs by checking ->vm_ops. There are some weird drivers whose ->mmap helpers don't set ->vm_ops and instead just shove all the necessary PTEs into the VMA right on ->mmap, so I think they end up with ->vm_ops==NULL. For example, kcov_mmap() looks that way. I'm not sure how common this is.

Though, uuuuuh, I guess if that's true, the existing vma_is_anonymous() is broken, since that also just checks ->vm_ops? I'm not sure what the consequences of that would be... Either way, vma_is_anonymous() might be the better way to check for anonymous VMAs here, and someone should figure out whether vma_is_anonymous() needs to be fixed.

Quoted 6 lines
mm/userfaultfd.c +6 -0
+       /*
+        * Ensure the dst_vma has a anon_vma or this page
+        * would get a NULL anon_vma when moved in the
+        * dst_vma.
+        */
+       err = -ENOMEM;

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1307

Quoted 10 lines
mm/userfaultfd.c +10 -0
+       if (unlikely(anon_vma_prepare(dst_vma)))
+               goto out;
+
+       for (src_addr = src_start, dst_addr = dst_start;
+            src_addr < src_start + len;) {
+               spinlock_t *ptl;
+               pmd_t dst_pmdval;
+
+               BUG_ON(dst_addr >= dst_start + len);
+               src_pmd = mm_find_pmd(src_mm, src_addr);

(this would blow up pretty badly if we could have transparent huge PUD in the region but I think that's limited to file VMAs so it's fine as it currently is)

Quoted 16 lines
mm/userfaultfd.c +16 -0
+               if (unlikely(!src_pmd)) {
+                       if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES)) {
+                               err = -ENOENT;
+                               break;
+                       }
+                       src_pmd = mm_alloc_pmd(src_mm, src_addr);
+                       if (unlikely(!src_pmd)) {
+                               err = -ENOMEM;
+                               break;
+                       }
+               }
+               dst_pmd = mm_alloc_pmd(dst_mm, dst_addr);
+               if (unlikely(!dst_pmd)) {
+                       err = -ENOMEM;
+                       break;
+               }

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1333

Quoted 10 lines
mm/userfaultfd.c +10 -0
+
+               dst_pmdval = pmdp_get_lockless(dst_pmd);
+               /*
+                * If the dst_pmd is mapped as THP don't
+                * override it and just be strict.
+                */
+               if (unlikely(pmd_trans_huge(dst_pmdval))) {
+                       err = -EEXIST;
+                       break;
+               }

This check is racy because the dst_pmd can still change at this point, from previously pointing to a zeroed PMD to now pointing to a hugepage, right? And we rely on remap_pages_pte() and remap_pages_huge_pmd() to recheck for that?
If yes, maybe add a comment noting this and explaining why we want this check.

Quoted 5 lines
mm/userfaultfd.c +5 -0
+               ptl = pmd_trans_huge_lock(src_pmd, src_vma);
+               if (ptl) {
+                       /*
+                        * Check if we can move the pmd without
+                        * splitting it. First check the address

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1348

Quoted 10 lines
mm/userfaultfd.c +10 -0
+                        * alignment to be the same in src/dst.  These
+                        * checks don't actually need the PT lock but
+                        * it's good to do it here to optimize this
+                        * block away at build time if
+                        * CONFIG_TRANSPARENT_HUGEPAGE is not set.
+                        */
+                       if (thp_aligned == -1)
+                               thp_aligned = ((src_addr & ~HPAGE_PMD_MASK) ==
+                                              (dst_addr & ~HPAGE_PMD_MASK));
+                       if (!thp_aligned || (src_addr & ~HPAGE_PMD_MASK) ||

This seems overly complicated, the only case when you can move a huge PMD is if both addresses are hugepage-aligned and you have enough length for one hugepage:

(src_addr & ~HPAGE_PMD_MASK) == 0 && (dst_addr & ~HPAGE_PMD_MASK) == 0 && (src_start + len - src_addr >= HPAGE_PMD_SIZE).

Quoted 37 lines
mm/userfaultfd.c +37 -0
+                           !pmd_none(dst_pmdval) ||
+                           src_start + len - src_addr < HPAGE_PMD_SIZE) {
+                               spin_unlock(ptl);
+                               /* Fall through */
+                               split_huge_pmd(src_vma, src_pmd, src_addr);
+                       } else {
+                               err = remap_pages_huge_pmd(dst_mm,
+                                                          src_mm,
+                                                          dst_pmd,
+                                                          src_pmd,
+                                                          dst_pmdval,
+                                                          dst_vma,
+                                                          src_vma,
+                                                          dst_addr,
+                                                          src_addr);
+                               cond_resched();
+
+                               if (!err) {
+                                       dst_addr += HPAGE_PMD_SIZE;
+                                       src_addr += HPAGE_PMD_SIZE;
+                                       moved += HPAGE_PMD_SIZE;
+                               }
+
+                               if ((!err || err == -EAGAIN) &&
+                                   fatal_signal_pending(current))
+                                       err = -EINTR;
+
+                               if (err && err != -EAGAIN)
+                                       break;
+
+                               continue;
+                       }
+               }
+
+               if (pmd_none(*src_pmd)) {
+                       if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES)) {
+                               err = -ENOENT;

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1395

Quoted 10 lines
mm/userfaultfd.c +10 -0
+                               break;
+                       }
+                       if (unlikely(__pte_alloc(src_mm, src_pmd))) {
+                               err = -ENOMEM;
+                               break;
+                       }
+               }
+
+               if (unlikely(pmd_none(dst_pmdval)) &&
+                   unlikely(__pte_alloc(dst_mm, dst_pmd))) {

Maybe just use pte_alloc() here?

Quoted 25 lines
mm/userfaultfd.c +25 -0
+                       err = -ENOMEM;
+                       break;
+               }
+
+               err = remap_pages_pte(dst_mm, src_mm,
+                                     dst_pmd, src_pmd,
+                                     dst_vma, src_vma,
+                                     dst_addr, src_addr,
+                                     mode);
+
+               cond_resched();
+
+               if (!err) {
+                       dst_addr += PAGE_SIZE;
+                       src_addr += PAGE_SIZE;
+                       moved += PAGE_SIZE;
+               }
+
+               if ((!err || err == -EAGAIN) &&
+                   fatal_signal_pending(current))
+                       err = -EINTR;
+
+               if (err && err != -EAGAIN)
+                       break;
+       }

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1430

Quoted 10 lines
mm/userfaultfd.c +10 -0
+
+out:
+       mmap_read_unlock(dst_mm);
+       if (dst_mm != src_mm)
+               mmap_read_unlock(src_mm);
+       BUG_ON(moved < 0);
+       BUG_ON(err > 0);
+       BUG_ON(!moved && !err);
+       return moved ? moved : err;
+}

Maybe you could try whether this function would look simpler with a shape roughly like:

for (src_addr = ...; src_addr < ...;) {
unsigned long step_size;

  if (hugepage case) {
    if (have to split) {
      split it;
      continue;
    }
    step_size = HPAGE_PMD_SIZE;
    ...
  } else {
    ... 4k case ...
    step_size = PAGE_SIZE;
  }
  ...
  cond_resched();
  if (!err) {
    dst_addr += step_size;
    src_addr += step_size;
    moved += step_size;

}
...
}

author reviewer
#14 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 479 lines · Jann Horn
Quoted 44 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

---
 fs/userfaultfd.c                 |  49 +++
 include/linux/rmap.h             |   5 +
 include/linux/userfaultfd_k.h    |  17 +
 include/uapi/linux/userfaultfd.h |  22 ++
 mm/huge_memory.c                 | 118 +++++++
 mm/khugepaged.c                  |   3 +
 mm/userfaultfd.c                 | 586 +++++++++++++++++++++++++++++++
 7 files changed, 800 insertions(+)
fs/userfaultfd.c +16 -0
index 56eaae9dac1a..7bf64e7541c1 100644
@@ -2027,6 +2027,52 @@ static inline unsigned int uffd_ctx_features(__u64 user_features)
        return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED;
 }
 
+static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
+                            unsigned long arg)
+{
+       __s64 ret;
+       struct uffdio_remap uffdio_remap;
+       struct uffdio_remap __user *user_uffdio_remap;
+       struct userfaultfd_wake_range range;
+
+       user_uffdio_remap = (struct uffdio_remap __user *) arg;
+
+       ret = -EFAULT;
+       if (copy_from_user(&uffdio_remap, user_uffdio_remap,
+                          /* don't copy "remap" last field */
+                          sizeof(uffdio_remap)-sizeof(__s64)))
+               goto out;
+

fs/userfaultfd.c static inline unsigned int uffd_ctx_features(__u64 user_features) · line 2055

Quoted 10 lines
fs/userfaultfd.c +10 -0
+       ret = validate_range(ctx->mm, uffdio_remap.dst, uffdio_remap.len);
+       if (ret)
+               goto out;
+       ret = validate_range(current->mm, uffdio_remap.src, uffdio_remap.len);
+       if (ret)
+               goto out;
+       ret = -EINVAL;
+       if (uffdio_remap.mode & ~(UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES|
+                                 UFFDIO_REMAP_MODE_DONTWAKE))
+               goto out;

Do you not need mmget_not_zero(ctx->mm) to make sure the MM can't be concurrently torn down while remap_pages() is running, similar to what the other userfaultfd ioctl handlers do?

Quoted 18 lines
fs/userfaultfd.c +18 -0
+       ret = remap_pages(ctx->mm, current->mm,
+                         uffdio_remap.dst, uffdio_remap.src,
+                         uffdio_remap.len, uffdio_remap.mode);
+       if (unlikely(put_user(ret, &user_uffdio_remap->remap)))
+               return -EFAULT;
+       if (ret < 0)
+               goto out;
+       /* len == 0 would wake all */
+       BUG_ON(!ret);
+       range.len = ret;
+       if (!(uffdio_remap.mode & UFFDIO_REMAP_MODE_DONTWAKE)) {
+               range.start = uffdio_remap.dst;
+               wake_userfault(ctx, &range);
+       }
+       ret = range.len == uffdio_remap.len ? 0 : -EAGAIN;
+out:
+       return ret;
+}

[...]

Quoted 29 lines
mm/huge_memory.c +21 -0
index 064fbd90822b..c7a9880a1f6a 100644
@@ -1932,6 +1932,124 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
        return ret;
 }
 
+#ifdef CONFIG_USERFAULTFD
+/*
+ * The PT lock for src_pmd and the mmap_lock for reading are held by
+ * the caller, but it must return after releasing the
+ * page_table_lock. We're guaranteed the src_pmd is a pmd_trans_huge
+ * until the PT lock of the src_pmd is released. Just move the page
+ * from src_pmd to dst_pmd if possible. Return zero if succeeded in
+ * moving the page, -EAGAIN if it needs to be repeated by the caller,
+ * or other errors in case of failure.
+ */
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                        struct mm_struct *src_mm,
+                        pmd_t *dst_pmd, pmd_t *src_pmd,
+                        pmd_t dst_pmdval,
+                        struct vm_area_struct *dst_vma,
+                        struct vm_area_struct *src_vma,
+                        unsigned long dst_addr,
+                        unsigned long src_addr)
+{
+       pmd_t _dst_pmd, src_pmdval;
+       struct page *src_page;

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1965

Quoted 10 lines
mm/huge_memory.c +10 -0
+       struct anon_vma *src_anon_vma, *dst_anon_vma;
+       spinlock_t *src_ptl, *dst_ptl;
+       pgtable_t pgtable;
+       struct mmu_notifier_range range;
+
+       src_pmdval = *src_pmd;
+       src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+       BUG_ON(!pmd_trans_huge(src_pmdval));
+       BUG_ON(!pmd_none(dst_pmdval));

Why can we assert that pmd_none(dst_pmdval) is true here? Can we not have concurrent faults (or userfaultfd operations) populating that PMD?

Quoted 54 lines
mm/huge_memory.c +54 -0
+       BUG_ON(!spin_is_locked(src_ptl));
+       mmap_assert_locked(src_mm);
+       mmap_assert_locked(dst_mm);
+       BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+       BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+       src_page = pmd_page(src_pmdval);
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       if (unlikely(page_mapcount(src_page) != 1)) {
+               spin_unlock(src_ptl);
+               return -EBUSY;
+       }
+
+       get_page(src_page);
+       spin_unlock(src_ptl);
+
+       mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr,
+                               src_addr + HPAGE_PMD_SIZE);
+       mmu_notifier_invalidate_range_start(&range);
+
+       /* block all concurrent rmap walks */
+       lock_page(src_page);
+
+       /*
+        * split_huge_page walks the anon_vma chain without the page
+        * lock. Serialize against it with the anon_vma lock, the page
+        * lock is not enough.
+        */
+       src_anon_vma = folio_get_anon_vma(page_folio(src_page));
+       if (!src_anon_vma) {
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+       anon_vma_lock_write(src_anon_vma);
+
+       dst_ptl = pmd_lockptr(dst_mm, dst_pmd);
+       double_pt_lock(src_ptl, dst_ptl);
+       if (unlikely(!pmd_same(*src_pmd, src_pmdval) ||
+                    !pmd_same(*dst_pmd, dst_pmdval) ||
+                    page_mapcount(src_page) != 1)) {
+               double_pt_unlock(src_ptl, dst_ptl);
+               anon_vma_unlock_write(src_anon_vma);
+               put_anon_vma(src_anon_vma);
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2029

Quoted 10 lines
mm/huge_memory.c +10 -0
+       /* the PT lock is enough to keep the page pinned now */
+       put_page(src_page);
+
+       dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
+       WRITE_ONCE(src_page->mapping, (struct address_space *) dst_anon_vma);
+       WRITE_ONCE(src_page->index, linear_page_index(dst_vma, dst_addr));
+
+       if (!pmd_same(pmdp_huge_clear_flush(src_vma, src_addr, src_pmd),
+                     src_pmdval))
+               BUG_ON(1);

I'm not sure we can assert that the PMDs are exactly equal; the CPU might have changed the A/D bits under us?

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2035

Quoted 6 lines
mm/huge_memory.c +6 -0
+       _dst_pmd = mk_huge_pmd(src_page, dst_vma->vm_page_prot);
+       _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
+       set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd);
+
+       pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd);
+       pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);

Are we allowed to move page tables between mm_structs on all architectures? The first example I found that looks a bit dodgy, looking through various architectures' pte_alloc_one(), is s390's page_table_alloc() which looks like page tables are tied to per-MM lists sometimes.
If that's not allowed, we might have to allocate a new deposit table and free the old one or something like that.

Quoted 20 lines
mm/huge_memory.c +17 -0
+       if (dst_mm != src_mm) {
+               add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+               add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR);
+       }
+       double_pt_unlock(src_ptl, dst_ptl);
+
+       anon_vma_unlock_write(src_anon_vma);
+       put_anon_vma(src_anon_vma);
+
+       /* unblock rmap walks */
+       unlock_page(src_page);
+
+       mmu_notifier_invalidate_range_end(&range);
+       return 0;
+}
+#endif /* CONFIG_USERFAULTFD */
+
 /*
  * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise.
  *

[...]

Quoted 4 lines
mm/userfaultfd.c +0 -0
index 96d9eae5c7cc..0cca60dfa8f8 100644

[...]

Quoted 4 lines
mm/userfaultfd.c +4 -0
+ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+                   unsigned long dst_start, unsigned long src_start,
+                   unsigned long len, __u64 mode)
+{

[...]

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1260

Quoted 4 lines
mm/userfaultfd.c +4 -0
+
+       if (pgprot_val(src_vma->vm_page_prot) !=
+           pgprot_val(dst_vma->vm_page_prot))
+               goto out;

Does this check intentionally allow moving pages from a PROT_READ|PROT_WRITE anonymous private VMA into a PROT_READ anonymous private VMA (on architectures like x86 and arm64 where CoW memory has the same protection flags as read-only memory), but forbid moving them from a PROT_READ|PROT_EXEC VMA into a PROT_READ VMA? I think this check needs at least a comment to explain what's going on here.

Quoted 19 lines
mm/userfaultfd.c +19 -0
+       /* only allow remapping if both are mlocked or both aren't */
+       if ((src_vma->vm_flags & VM_LOCKED) ^ (dst_vma->vm_flags & VM_LOCKED))
+               goto out;
+
+       /*
+        * Be strict and only allow remap_pages if either the src or
+        * dst range is registered in the userfaultfd to prevent
+        * userland errors going unnoticed. As far as the VM
+        * consistency is concerned, it would be perfectly safe to
+        * remove this check, but there's no useful usage for
+        * remap_pages ouside of userfaultfd registered ranges. This
+        * is after all why it is an ioctl belonging to the
+        * userfaultfd and not a syscall.
+        *
+        * Allow both vmas to be registered in the userfaultfd, just
+        * in case somebody finds a way to make such a case useful.
+        * Normally only one of the two vmas would be registered in
+        * the userfaultfd.
+        */

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1290

Quoted 10 lines
mm/userfaultfd.c +10 -0
+       if (!dst_vma->vm_userfaultfd_ctx.ctx &&
+           !src_vma->vm_userfaultfd_ctx.ctx)
+               goto out;
+
+       /*
+        * FIXME: only allow remapping across anonymous vmas,
+        * tmpfs should be added.
+        */
+       if (src_vma->vm_ops || dst_vma->vm_ops)
+               goto out;

I don't think it's okay to check for anonymous VMAs by checking ->vm_ops. There are some weird drivers whose ->mmap helpers don't set ->vm_ops and instead just shove all the necessary PTEs into the VMA right on ->mmap, so I think they end up with ->vm_ops==NULL. For example, kcov_mmap() looks that way. I'm not sure how common this is.

Though, uuuuuh, I guess if that's true, the existing vma_is_anonymous() is broken, since that also just checks ->vm_ops? I'm not sure what the consequences of that would be... Either way, vma_is_anonymous() might be the better way to check for anonymous VMAs here, and someone should figure out whether vma_is_anonymous() needs to be fixed.

Quoted 6 lines
mm/userfaultfd.c +6 -0
+       /*
+        * Ensure the dst_vma has a anon_vma or this page
+        * would get a NULL anon_vma when moved in the
+        * dst_vma.
+        */
+       err = -ENOMEM;

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1307

Quoted 10 lines
mm/userfaultfd.c +10 -0
+       if (unlikely(anon_vma_prepare(dst_vma)))
+               goto out;
+
+       for (src_addr = src_start, dst_addr = dst_start;
+            src_addr < src_start + len;) {
+               spinlock_t *ptl;
+               pmd_t dst_pmdval;
+
+               BUG_ON(dst_addr >= dst_start + len);
+               src_pmd = mm_find_pmd(src_mm, src_addr);

(this would blow up pretty badly if we could have transparent huge PUD in the region but I think that's limited to file VMAs so it's fine as it currently is)

Quoted 16 lines
mm/userfaultfd.c +16 -0
+               if (unlikely(!src_pmd)) {
+                       if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES)) {
+                               err = -ENOENT;
+                               break;
+                       }
+                       src_pmd = mm_alloc_pmd(src_mm, src_addr);
+                       if (unlikely(!src_pmd)) {
+                               err = -ENOMEM;
+                               break;
+                       }
+               }
+               dst_pmd = mm_alloc_pmd(dst_mm, dst_addr);
+               if (unlikely(!dst_pmd)) {
+                       err = -ENOMEM;
+                       break;
+               }

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1333

Quoted 10 lines
mm/userfaultfd.c +10 -0
+
+               dst_pmdval = pmdp_get_lockless(dst_pmd);
+               /*
+                * If the dst_pmd is mapped as THP don't
+                * override it and just be strict.
+                */
+               if (unlikely(pmd_trans_huge(dst_pmdval))) {
+                       err = -EEXIST;
+                       break;
+               }

This check is racy because the dst_pmd can still change at this point, from previously pointing to a zeroed PMD to now pointing to a hugepage, right? And we rely on remap_pages_pte() and remap_pages_huge_pmd() to recheck for that?
If yes, maybe add a comment noting this and explaining why we want this check.

Quoted 5 lines
mm/userfaultfd.c +5 -0
+               ptl = pmd_trans_huge_lock(src_pmd, src_vma);
+               if (ptl) {
+                       /*
+                        * Check if we can move the pmd without
+                        * splitting it. First check the address

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1348

Quoted 10 lines
mm/userfaultfd.c +10 -0
+                        * alignment to be the same in src/dst.  These
+                        * checks don't actually need the PT lock but
+                        * it's good to do it here to optimize this
+                        * block away at build time if
+                        * CONFIG_TRANSPARENT_HUGEPAGE is not set.
+                        */
+                       if (thp_aligned == -1)
+                               thp_aligned = ((src_addr & ~HPAGE_PMD_MASK) ==
+                                              (dst_addr & ~HPAGE_PMD_MASK));
+                       if (!thp_aligned || (src_addr & ~HPAGE_PMD_MASK) ||

This seems overly complicated, the only case when you can move a huge PMD is if both addresses are hugepage-aligned and you have enough length for one hugepage:

(src_addr & ~HPAGE_PMD_MASK) == 0 && (dst_addr & ~HPAGE_PMD_MASK) == 0 && (src_start + len - src_addr >= HPAGE_PMD_SIZE).

Quoted 37 lines
mm/userfaultfd.c +37 -0
+                           !pmd_none(dst_pmdval) ||
+                           src_start + len - src_addr < HPAGE_PMD_SIZE) {
+                               spin_unlock(ptl);
+                               /* Fall through */
+                               split_huge_pmd(src_vma, src_pmd, src_addr);
+                       } else {
+                               err = remap_pages_huge_pmd(dst_mm,
+                                                          src_mm,
+                                                          dst_pmd,
+                                                          src_pmd,
+                                                          dst_pmdval,
+                                                          dst_vma,
+                                                          src_vma,
+                                                          dst_addr,
+                                                          src_addr);
+                               cond_resched();
+
+                               if (!err) {
+                                       dst_addr += HPAGE_PMD_SIZE;
+                                       src_addr += HPAGE_PMD_SIZE;
+                                       moved += HPAGE_PMD_SIZE;
+                               }
+
+                               if ((!err || err == -EAGAIN) &&
+                                   fatal_signal_pending(current))
+                                       err = -EINTR;
+
+                               if (err && err != -EAGAIN)
+                                       break;
+
+                               continue;
+                       }
+               }
+
+               if (pmd_none(*src_pmd)) {
+                       if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES)) {
+                               err = -ENOENT;

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1395

Quoted 10 lines
mm/userfaultfd.c +10 -0
+                               break;
+                       }
+                       if (unlikely(__pte_alloc(src_mm, src_pmd))) {
+                               err = -ENOMEM;
+                               break;
+                       }
+               }
+
+               if (unlikely(pmd_none(dst_pmdval)) &&
+                   unlikely(__pte_alloc(dst_mm, dst_pmd))) {

Maybe just use pte_alloc() here?

Quoted 25 lines
mm/userfaultfd.c +25 -0
+                       err = -ENOMEM;
+                       break;
+               }
+
+               err = remap_pages_pte(dst_mm, src_mm,
+                                     dst_pmd, src_pmd,
+                                     dst_vma, src_vma,
+                                     dst_addr, src_addr,
+                                     mode);
+
+               cond_resched();
+
+               if (!err) {
+                       dst_addr += PAGE_SIZE;
+                       src_addr += PAGE_SIZE;
+                       moved += PAGE_SIZE;
+               }
+
+               if ((!err || err == -EAGAIN) &&
+                   fatal_signal_pending(current))
+                       err = -EINTR;
+
+               if (err && err != -EAGAIN)
+                       break;
+       }

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1430

Quoted 10 lines
mm/userfaultfd.c +10 -0
+
+out:
+       mmap_read_unlock(dst_mm);
+       if (dst_mm != src_mm)
+               mmap_read_unlock(src_mm);
+       BUG_ON(moved < 0);
+       BUG_ON(err > 0);
+       BUG_ON(!moved && !err);
+       return moved ? moved : err;
+}

Maybe you could try whether this function would look simpler with a shape roughly like:

for (src_addr = ...; src_addr < ...;) {
unsigned long step_size;

  if (hugepage case) {
    if (have to split) {
      split it;
      continue;
    }
    step_size = HPAGE_PMD_SIZE;
    ...
  } else {
    ... 4k case ...
    step_size = PAGE_SIZE;
  }
  ...
  cond_resched();
  if (!err) {
    dst_addr += step_size;
    src_addr += step_size;
    moved += step_size;
Quoted 3 lines

}
...
}

I'll need some time to gather the answers to all your questions and will reply once I have them ready.
Thanks for reviewing, Jann!

author reviewer
#15 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

In reply to Jann Horn
Quoted 48 lines · Jann Horn
Quoted 46 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

---
 fs/userfaultfd.c                 |  49 +++
 include/linux/rmap.h             |   5 +
 include/linux/userfaultfd_k.h    |  17 +
 include/uapi/linux/userfaultfd.h |  22 ++
 mm/huge_memory.c                 | 118 +++++++
 mm/khugepaged.c                  |   3 +
 mm/userfaultfd.c                 | 586 +++++++++++++++++++++++++++++++
 7 files changed, 800 insertions(+)
fs/userfaultfd.c +18 -0
index 56eaae9dac1a..7bf64e7541c1 100644
@@ -2027,6 +2027,52 @@ static inline unsigned int uffd_ctx_features(__u64 user_features)
        return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED;
 }
 
+static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
+                            unsigned long arg)
+{
+       __s64 ret;
+       struct uffdio_remap uffdio_remap;
+       struct uffdio_remap __user *user_uffdio_remap;
+       struct userfaultfd_wake_range range;
+
+       user_uffdio_remap = (struct uffdio_remap __user *) arg;
+
+       ret = -EFAULT;
+       if (copy_from_user(&uffdio_remap, user_uffdio_remap,
+                          /* don't copy "remap" last field */
+                          sizeof(uffdio_remap)-sizeof(__s64)))
+               goto out;
+
+       ret = validate_range(ctx->mm, uffdio_remap.dst, uffdio_remap.len);
+       if (ret)
Quoted 9 lines

fs/userfaultfd.c static inline unsigned int uffd_ctx_features(__u64 user_features) · line 2055

Quoted 8 lines
fs/userfaultfd.c +8 -0
+               goto out;
+       ret = validate_range(current->mm, uffdio_remap.src, uffdio_remap.len);
+       if (ret)
+               goto out;
+       ret = -EINVAL;
+       if (uffdio_remap.mode & ~(UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES|
+                                 UFFDIO_REMAP_MODE_DONTWAKE))
+               goto out;

Sorry for the delay with the answers...

Quoted 3 lines

Do you not need mmget_not_zero(ctx->mm) to make sure the MM can't be concurrently torn down while remap_pages() is running, similar to what the other userfaultfd ioctl handlers do?

Yes, I do need that. Will add.

Quoted 55 lines
Quoted 19 lines
fs/userfaultfd.c +18 -0
+       ret = remap_pages(ctx->mm, current->mm,
+                         uffdio_remap.dst, uffdio_remap.src,
+                         uffdio_remap.len, uffdio_remap.mode);
+       if (unlikely(put_user(ret, &user_uffdio_remap->remap)))
+               return -EFAULT;
+       if (ret < 0)
+               goto out;
+       /* len == 0 would wake all */
+       BUG_ON(!ret);
+       range.len = ret;
+       if (!(uffdio_remap.mode & UFFDIO_REMAP_MODE_DONTWAKE)) {
+               range.start = uffdio_remap.dst;
+               wake_userfault(ctx, &range);
+       }
+       ret = range.len == uffdio_remap.len ? 0 : -EAGAIN;
+out:
+       return ret;
+}

[...]

Quoted 35 lines
mm/huge_memory.c +27 -0
index 064fbd90822b..c7a9880a1f6a 100644
@@ -1932,6 +1932,124 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
        return ret;
 }
 
+#ifdef CONFIG_USERFAULTFD
+/*
+ * The PT lock for src_pmd and the mmap_lock for reading are held by
+ * the caller, but it must return after releasing the
+ * page_table_lock. We're guaranteed the src_pmd is a pmd_trans_huge
+ * until the PT lock of the src_pmd is released. Just move the page
+ * from src_pmd to dst_pmd if possible. Return zero if succeeded in
+ * moving the page, -EAGAIN if it needs to be repeated by the caller,
+ * or other errors in case of failure.
+ */
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                        struct mm_struct *src_mm,
+                        pmd_t *dst_pmd, pmd_t *src_pmd,
+                        pmd_t dst_pmdval,
+                        struct vm_area_struct *dst_vma,
+                        struct vm_area_struct *src_vma,
+                        unsigned long dst_addr,
+                        unsigned long src_addr)
+{
+       pmd_t _dst_pmd, src_pmdval;
+       struct page *src_page;
+       struct anon_vma *src_anon_vma, *dst_anon_vma;
+       spinlock_t *src_ptl, *dst_ptl;
+       pgtable_t pgtable;
+       struct mmu_notifier_range range;
+
+       src_pmdval = *src_pmd;
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1965

Quoted 4 lines
mm/huge_memory.c +4 -0
+       src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+       BUG_ON(!pmd_trans_huge(src_pmdval));
+       BUG_ON(!pmd_none(dst_pmdval));

Why can we assert that pmd_none(dst_pmdval) is true here? Can we not have concurrent faults (or userfaultfd operations) populating that PMD?

IIUC dst_pmdval is a copy of the value from dst_pmd, so that local copy should not change even if some concurrent operation changes dst_pmd. We can assert that it's pmd_none because we checked for that before calling remap_pages_huge_pmd. Later on we check if dst_pmd changed from under us (see pmd_same(*dst_pmd, dst_pmdval) check) and retry if that happened.

Quoted 60 lines
Quoted 60 lines
mm/huge_memory.c +59 -0
+       BUG_ON(!spin_is_locked(src_ptl));
+       mmap_assert_locked(src_mm);
+       mmap_assert_locked(dst_mm);
+       BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+       BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+       src_page = pmd_page(src_pmdval);
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       if (unlikely(page_mapcount(src_page) != 1)) {
+               spin_unlock(src_ptl);
+               return -EBUSY;
+       }
+
+       get_page(src_page);
+       spin_unlock(src_ptl);
+
+       mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr,
+                               src_addr + HPAGE_PMD_SIZE);
+       mmu_notifier_invalidate_range_start(&range);
+
+       /* block all concurrent rmap walks */
+       lock_page(src_page);
+
+       /*
+        * split_huge_page walks the anon_vma chain without the page
+        * lock. Serialize against it with the anon_vma lock, the page
+        * lock is not enough.
+        */
+       src_anon_vma = folio_get_anon_vma(page_folio(src_page));
+       if (!src_anon_vma) {
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+       anon_vma_lock_write(src_anon_vma);
+
+       dst_ptl = pmd_lockptr(dst_mm, dst_pmd);
+       double_pt_lock(src_ptl, dst_ptl);
+       if (unlikely(!pmd_same(*src_pmd, src_pmdval) ||
+                    !pmd_same(*dst_pmd, dst_pmdval) ||
+                    page_mapcount(src_page) != 1)) {
+               double_pt_unlock(src_ptl, dst_ptl);
+               anon_vma_unlock_write(src_anon_vma);
+               put_anon_vma(src_anon_vma);
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       /* the PT lock is enough to keep the page pinned now */
+       put_page(src_page);
+
+       dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
+       WRITE_ONCE(src_page->mapping, (struct address_space *) dst_anon_vma);
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2029

Quoted 5 lines
mm/huge_memory.c +5 -0
+       WRITE_ONCE(src_page->index, linear_page_index(dst_vma, dst_addr));
+
+       if (!pmd_same(pmdp_huge_clear_flush(src_vma, src_addr, src_pmd),
+                     src_pmdval))
+               BUG_ON(1);

I'm not sure we can assert that the PMDs are exactly equal; the CPU might have changed the A/D bits under us?

Yes. I wonder if I can simply remove the BUG_ON here like this:

src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);

Technically we don't use src_pmdval after this but for the possible future use that would keep things correct. If A/D bits changed from under us we will still copy correct values into dst_pmd.

Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2035

Quoted 7 lines
mm/huge_memory.c +6 -0
+       _dst_pmd = mk_huge_pmd(src_page, dst_vma->vm_page_prot);
+       _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
+       set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd);
+
+       pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd);
+       pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
Quoted 7 lines

Are we allowed to move page tables between mm_structs on all architectures? The first example I found that looks a bit dodgy, looking through various architectures' pte_alloc_one(), is s390's page_table_alloc() which looks like page tables are tied to per-MM lists sometimes.
If that's not allowed, we might have to allocate a new deposit table and free the old one or something like that.

Hmm. Yeah, looks like in the case of !CONFIG_PGSTE the table can be linked to mm->context.pgtable_list, so can't be moved to another mm. I guess I'll have to keep a pgtable allocated, ready to be deposited and free the old one. Maybe it's worth having an arch-specific function indicating whether moving a pgtable between MMs is supported? Or do it separately as an optimization. WDYT?

Quoted 37 lines
Quoted 21 lines
mm/huge_memory.c +17 -0
+       if (dst_mm != src_mm) {
+               add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+               add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR);
+       }
+       double_pt_unlock(src_ptl, dst_ptl);
+
+       anon_vma_unlock_write(src_anon_vma);
+       put_anon_vma(src_anon_vma);
+
+       /* unblock rmap walks */
+       unlock_page(src_page);
+
+       mmu_notifier_invalidate_range_end(&range);
+       return 0;
+}
+#endif /* CONFIG_USERFAULTFD */
+
 /*
  * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise.
  *

[...]

Quoted 4 lines
mm/userfaultfd.c +0 -0
index 96d9eae5c7cc..0cca60dfa8f8 100644

[...]

Quoted 4 lines
mm/userfaultfd.c +4 -0
+ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+                   unsigned long dst_start, unsigned long src_start,
+                   unsigned long len, __u64 mode)
+{

[...]

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1260

Quoted 4 lines
mm/userfaultfd.c +4 -0
+
+       if (pgprot_val(src_vma->vm_page_prot) !=
+           pgprot_val(dst_vma->vm_page_prot))
+               goto out;
Quoted 6 lines

Does this check intentionally allow moving pages from a PROT_READ|PROT_WRITE anonymous private VMA into a PROT_READ anonymous private VMA (on architectures like x86 and arm64 where CoW memory has the same protection flags as read-only memory), but forbid moving them from a PROT_READ|PROT_EXEC VMA into a PROT_READ VMA? I think this check needs at least a comment to explain what's going on here.

The check is simply to ensure the VMAs have the same access permissions to prevent page copies that should have different permissions. The fact that x86 and arm64 have the same protection bits for R/O and COW memory is a "side-effect" IMHO. I'm not sure what comment would be good here but I'm open to suggestions.

Quoted 37 lines
Quoted 20 lines
mm/userfaultfd.c +19 -0
+       /* only allow remapping if both are mlocked or both aren't */
+       if ((src_vma->vm_flags & VM_LOCKED) ^ (dst_vma->vm_flags & VM_LOCKED))
+               goto out;
+
+       /*
+        * Be strict and only allow remap_pages if either the src or
+        * dst range is registered in the userfaultfd to prevent
+        * userland errors going unnoticed. As far as the VM
+        * consistency is concerned, it would be perfectly safe to
+        * remove this check, but there's no useful usage for
+        * remap_pages ouside of userfaultfd registered ranges. This
+        * is after all why it is an ioctl belonging to the
+        * userfaultfd and not a syscall.
+        *
+        * Allow both vmas to be registered in the userfaultfd, just
+        * in case somebody finds a way to make such a case useful.
+        * Normally only one of the two vmas would be registered in
+        * the userfaultfd.
+        */

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1290

Quoted 10 lines
mm/userfaultfd.c +10 -0
+       if (!dst_vma->vm_userfaultfd_ctx.ctx &&
+           !src_vma->vm_userfaultfd_ctx.ctx)
+               goto out;
+
+       /*
+        * FIXME: only allow remapping across anonymous vmas,
+        * tmpfs should be added.
+        */
+       if (src_vma->vm_ops || dst_vma->vm_ops)
+               goto out;

I don't think it's okay to check for anonymous VMAs by checking ->vm_ops. There are some weird drivers whose ->mmap helpers don't set ->vm_ops and instead just shove all the necessary PTEs into the VMA right on ->mmap, so I think they end up with ->vm_ops==NULL. For example, kcov_mmap() looks that way. I'm not sure how common this is.

Quoted 6 lines

Though, uuuuuh, I guess if that's true, the existing vma_is_anonymous() is broken, since that also just checks ->vm_ops? I'm not sure what the consequences of that would be... Either way, vma_is_anonymous() might be the better way to check for anonymous VMAs here, and someone should figure out whether vma_is_anonymous() needs to be fixed.

Absolutely, vma_is_anonymous() should have been used here. Will fix it.

Quoted 13 lines
Quoted 13 lines
mm/userfaultfd.c +12 -0
+       /*
+        * Ensure the dst_vma has a anon_vma or this page
+        * would get a NULL anon_vma when moved in the
+        * dst_vma.
+        */
+       err = -ENOMEM;
+       if (unlikely(anon_vma_prepare(dst_vma)))
+               goto out;
+
+       for (src_addr = src_start, dst_addr = dst_start;
+            src_addr < src_start + len;) {
+               spinlock_t *ptl;
Quoted 8 lines

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1307

Quoted 4 lines
mm/userfaultfd.c +4 -0
+               pmd_t dst_pmdval;
+
+               BUG_ON(dst_addr >= dst_start + len);
+               src_pmd = mm_find_pmd(src_mm, src_addr);

(this would blow up pretty badly if we could have transparent huge PUD in the region but I think that's limited to file VMAs so it's fine as it currently is)

Should I add a comment here as a warning if in the future we decide to implement support for file-backed pages?

Quoted 28 lines
Quoted 17 lines
mm/userfaultfd.c +16 -0
+               if (unlikely(!src_pmd)) {
+                       if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES)) {
+                               err = -ENOENT;
+                               break;
+                       }
+                       src_pmd = mm_alloc_pmd(src_mm, src_addr);
+                       if (unlikely(!src_pmd)) {
+                               err = -ENOMEM;
+                               break;
+                       }
+               }
+               dst_pmd = mm_alloc_pmd(dst_mm, dst_addr);
+               if (unlikely(!dst_pmd)) {
+                       err = -ENOMEM;
+                       break;
+               }

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1333

Quoted 10 lines
mm/userfaultfd.c +10 -0
+
+               dst_pmdval = pmdp_get_lockless(dst_pmd);
+               /*
+                * If the dst_pmd is mapped as THP don't
+                * override it and just be strict.
+                */
+               if (unlikely(pmd_trans_huge(dst_pmdval))) {
+                       err = -EEXIST;
+                       break;
+               }
Quoted 5 lines

This check is racy because the dst_pmd can still change at this point, from previously pointing to a zeroed PMD to now pointing to a hugepage, right? And we rely on remap_pages_pte() and remap_pages_huge_pmd() to recheck for that?
If yes, maybe add a comment noting this and explaining why we want this check.

Yes, you are right. remap_pages_huge_pmd() does check pmd_same(*dst_pmd, dst_pmdval) after doing double_pt_lock(), so it will retry and exit on this check if dst_pmd got mapped as THP from under us. remap_pages_pte() OTOH simply does BUG_ON(pmd_trans_huge(*dst_pmd)), which is wrong. Instead it should also check if PMD value changed from under us and retry in such a case. I'll fix that and will add a comment here about this racy check and the retry logic.

Quoted 17 lines
Quoted 6 lines
mm/userfaultfd.c +5 -0
+               ptl = pmd_trans_huge_lock(src_pmd, src_vma);
+               if (ptl) {
+                       /*
+                        * Check if we can move the pmd without
+                        * splitting it. First check the address

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1348

Quoted 10 lines
mm/userfaultfd.c +10 -0
+                        * alignment to be the same in src/dst.  These
+                        * checks don't actually need the PT lock but
+                        * it's good to do it here to optimize this
+                        * block away at build time if
+                        * CONFIG_TRANSPARENT_HUGEPAGE is not set.
+                        */
+                       if (thp_aligned == -1)
+                               thp_aligned = ((src_addr & ~HPAGE_PMD_MASK) ==
+                                              (dst_addr & ~HPAGE_PMD_MASK));
+                       if (!thp_aligned || (src_addr & ~HPAGE_PMD_MASK) ||
Quoted 6 lines

This seems overly complicated, the only case when you can move a huge PMD is if both addresses are hugepage-aligned and you have enough length for one hugepage:

(src_addr & ~HPAGE_PMD_MASK) == 0 && (dst_addr & ~HPAGE_PMD_MASK) == 0 && (src_start + len - src_addr >= HPAGE_PMD_SIZE).

Ack.

Quoted 42 lines
Quoted 42 lines
mm/userfaultfd.c +41 -0
+                           !pmd_none(dst_pmdval) ||
+                           src_start + len - src_addr < HPAGE_PMD_SIZE) {
+                               spin_unlock(ptl);
+                               /* Fall through */
+                               split_huge_pmd(src_vma, src_pmd, src_addr);
+                       } else {
+                               err = remap_pages_huge_pmd(dst_mm,
+                                                          src_mm,
+                                                          dst_pmd,
+                                                          src_pmd,
+                                                          dst_pmdval,
+                                                          dst_vma,
+                                                          src_vma,
+                                                          dst_addr,
+                                                          src_addr);
+                               cond_resched();
+
+                               if (!err) {
+                                       dst_addr += HPAGE_PMD_SIZE;
+                                       src_addr += HPAGE_PMD_SIZE;
+                                       moved += HPAGE_PMD_SIZE;
+                               }
+
+                               if ((!err || err == -EAGAIN) &&
+                                   fatal_signal_pending(current))
+                                       err = -EINTR;
+
+                               if (err && err != -EAGAIN)
+                                       break;
+
+                               continue;
+                       }
+               }
+
+               if (pmd_none(*src_pmd)) {
+                       if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES)) {
+                               err = -ENOENT;
+                               break;
+                       }
+                       if (unlikely(__pte_alloc(src_mm, src_pmd))) {
+                               err = -ENOMEM;
Quoted 8 lines

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1395

Quoted 6 lines
mm/userfaultfd.c +6 -0
+                               break;
+                       }
+               }
+
+               if (unlikely(pmd_none(dst_pmdval)) &&
+                   unlikely(__pte_alloc(dst_mm, dst_pmd))) {

Maybe just use pte_alloc() here?

Ack.

Quoted 53 lines
Quoted 26 lines
mm/userfaultfd.c +25 -0
+                       err = -ENOMEM;
+                       break;
+               }
+
+               err = remap_pages_pte(dst_mm, src_mm,
+                                     dst_pmd, src_pmd,
+                                     dst_vma, src_vma,
+                                     dst_addr, src_addr,
+                                     mode);
+
+               cond_resched();
+
+               if (!err) {
+                       dst_addr += PAGE_SIZE;
+                       src_addr += PAGE_SIZE;
+                       moved += PAGE_SIZE;
+               }
+
+               if ((!err || err == -EAGAIN) &&
+                   fatal_signal_pending(current))
+                       err = -EINTR;
+
+               if (err && err != -EAGAIN)
+                       break;
+       }

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1430

Quoted 10 lines
mm/userfaultfd.c +10 -0
+
+out:
+       mmap_read_unlock(dst_mm);
+       if (dst_mm != src_mm)
+               mmap_read_unlock(src_mm);
+       BUG_ON(moved < 0);
+       BUG_ON(err > 0);
+       BUG_ON(!moved && !err);
+       return moved ? moved : err;
+}

Maybe you could try whether this function would look simpler with a shape roughly like:

for (src_addr = ...; src_addr < ...;) {
unsigned long step_size;

  if (hugepage case) {
    if (have to split) {
      split it;
      continue;
    }
    step_size = HPAGE_PMD_SIZE;
    ...
  } else {
    ... 4k case ...
    step_size = PAGE_SIZE;
Quoted 8 lines
  }
  ...
  cond_resched();
  if (!err) {
    dst_addr += step_size;
    src_addr += step_size;
    moved += step_size;
  }

Yeah, that looks simpler. I'll try that, thanks!

Quoted 2 lines

...
}

author reviewer
#16 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 395 lines · Suren Baghdasaryan
Quoted 48 lines · Jann Horn
Quoted 46 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

---
 fs/userfaultfd.c                 |  49 +++
 include/linux/rmap.h             |   5 +
 include/linux/userfaultfd_k.h    |  17 +
 include/uapi/linux/userfaultfd.h |  22 ++
 mm/huge_memory.c                 | 118 +++++++
 mm/khugepaged.c                  |   3 +
 mm/userfaultfd.c                 | 586 +++++++++++++++++++++++++++++++
 7 files changed, 800 insertions(+)
fs/userfaultfd.c +18 -0
index 56eaae9dac1a..7bf64e7541c1 100644
@@ -2027,6 +2027,52 @@ static inline unsigned int uffd_ctx_features(__u64 user_features)
        return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED;
 }
 
+static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
+                            unsigned long arg)
+{
+       __s64 ret;
+       struct uffdio_remap uffdio_remap;
+       struct uffdio_remap __user *user_uffdio_remap;
+       struct userfaultfd_wake_range range;
+
+       user_uffdio_remap = (struct uffdio_remap __user *) arg;
+
+       ret = -EFAULT;
+       if (copy_from_user(&uffdio_remap, user_uffdio_remap,
+                          /* don't copy "remap" last field */
+                          sizeof(uffdio_remap)-sizeof(__s64)))
+               goto out;
+
+       ret = validate_range(ctx->mm, uffdio_remap.dst, uffdio_remap.len);
+       if (ret)
Quoted 9 lines

fs/userfaultfd.c static inline unsigned int uffd_ctx_features(__u64 user_features) · line 2055

Quoted 8 lines
fs/userfaultfd.c +8 -0
+               goto out;
+       ret = validate_range(current->mm, uffdio_remap.src, uffdio_remap.len);
+       if (ret)
+               goto out;
+       ret = -EINVAL;
+       if (uffdio_remap.mode & ~(UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES|
+                                 UFFDIO_REMAP_MODE_DONTWAKE))
+               goto out;

Sorry for the delay with the answers...

Quoted 3 lines

Do you not need mmget_not_zero(ctx->mm) to make sure the MM can't be concurrently torn down while remap_pages() is running, similar to what the other userfaultfd ioctl handlers do?

Yes, I do need that. Will add.

Quoted 55 lines
Quoted 19 lines
fs/userfaultfd.c +18 -0
+       ret = remap_pages(ctx->mm, current->mm,
+                         uffdio_remap.dst, uffdio_remap.src,
+                         uffdio_remap.len, uffdio_remap.mode);
+       if (unlikely(put_user(ret, &user_uffdio_remap->remap)))
+               return -EFAULT;
+       if (ret < 0)
+               goto out;
+       /* len == 0 would wake all */
+       BUG_ON(!ret);
+       range.len = ret;
+       if (!(uffdio_remap.mode & UFFDIO_REMAP_MODE_DONTWAKE)) {
+               range.start = uffdio_remap.dst;
+               wake_userfault(ctx, &range);
+       }
+       ret = range.len == uffdio_remap.len ? 0 : -EAGAIN;
+out:
+       return ret;
+}

[...]

Quoted 35 lines
mm/huge_memory.c +27 -0
index 064fbd90822b..c7a9880a1f6a 100644
@@ -1932,6 +1932,124 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
        return ret;
 }
 
+#ifdef CONFIG_USERFAULTFD
+/*
+ * The PT lock for src_pmd and the mmap_lock for reading are held by
+ * the caller, but it must return after releasing the
+ * page_table_lock. We're guaranteed the src_pmd is a pmd_trans_huge
+ * until the PT lock of the src_pmd is released. Just move the page
+ * from src_pmd to dst_pmd if possible. Return zero if succeeded in
+ * moving the page, -EAGAIN if it needs to be repeated by the caller,
+ * or other errors in case of failure.
+ */
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                        struct mm_struct *src_mm,
+                        pmd_t *dst_pmd, pmd_t *src_pmd,
+                        pmd_t dst_pmdval,
+                        struct vm_area_struct *dst_vma,
+                        struct vm_area_struct *src_vma,
+                        unsigned long dst_addr,
+                        unsigned long src_addr)
+{
+       pmd_t _dst_pmd, src_pmdval;
+       struct page *src_page;
+       struct anon_vma *src_anon_vma, *dst_anon_vma;
+       spinlock_t *src_ptl, *dst_ptl;
+       pgtable_t pgtable;
+       struct mmu_notifier_range range;
+
+       src_pmdval = *src_pmd;
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1965

Quoted 4 lines
mm/huge_memory.c +4 -0
+       src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+       BUG_ON(!pmd_trans_huge(src_pmdval));
+       BUG_ON(!pmd_none(dst_pmdval));

Why can we assert that pmd_none(dst_pmdval) is true here? Can we not have concurrent faults (or userfaultfd operations) populating that PMD?

IIUC dst_pmdval is a copy of the value from dst_pmd, so that local copy should not change even if some concurrent operation changes dst_pmd. We can assert that it's pmd_none because we checked for that before calling remap_pages_huge_pmd. Later on we check if dst_pmd changed from under us (see pmd_same(*dst_pmd, dst_pmdval) check) and retry if that happened.

Quoted 60 lines
Quoted 60 lines
mm/huge_memory.c +59 -0
+       BUG_ON(!spin_is_locked(src_ptl));
+       mmap_assert_locked(src_mm);
+       mmap_assert_locked(dst_mm);
+       BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+       BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+       src_page = pmd_page(src_pmdval);
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       if (unlikely(page_mapcount(src_page) != 1)) {
+               spin_unlock(src_ptl);
+               return -EBUSY;
+       }
+
+       get_page(src_page);
+       spin_unlock(src_ptl);
+
+       mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr,
+                               src_addr + HPAGE_PMD_SIZE);
+       mmu_notifier_invalidate_range_start(&range);
+
+       /* block all concurrent rmap walks */
+       lock_page(src_page);
+
+       /*
+        * split_huge_page walks the anon_vma chain without the page
+        * lock. Serialize against it with the anon_vma lock, the page
+        * lock is not enough.
+        */
+       src_anon_vma = folio_get_anon_vma(page_folio(src_page));
+       if (!src_anon_vma) {
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+       anon_vma_lock_write(src_anon_vma);
+
+       dst_ptl = pmd_lockptr(dst_mm, dst_pmd);
+       double_pt_lock(src_ptl, dst_ptl);
+       if (unlikely(!pmd_same(*src_pmd, src_pmdval) ||
+                    !pmd_same(*dst_pmd, dst_pmdval) ||
+                    page_mapcount(src_page) != 1)) {
+               double_pt_unlock(src_ptl, dst_ptl);
+               anon_vma_unlock_write(src_anon_vma);
+               put_anon_vma(src_anon_vma);
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       /* the PT lock is enough to keep the page pinned now */
+       put_page(src_page);
+
+       dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
+       WRITE_ONCE(src_page->mapping, (struct address_space *) dst_anon_vma);
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2029

Quoted 5 lines
mm/huge_memory.c +5 -0
+       WRITE_ONCE(src_page->index, linear_page_index(dst_vma, dst_addr));
+
+       if (!pmd_same(pmdp_huge_clear_flush(src_vma, src_addr, src_pmd),
+                     src_pmdval))
+               BUG_ON(1);

I'm not sure we can assert that the PMDs are exactly equal; the CPU might have changed the A/D bits under us?

Yes. I wonder if I can simply remove the BUG_ON here like this:

src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);

Technically we don't use src_pmdval after this but for the possible future use that would keep things correct. If A/D bits changed from under us we will still copy correct values into dst_pmd.

Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2035

Quoted 7 lines
mm/huge_memory.c +6 -0
+       _dst_pmd = mk_huge_pmd(src_page, dst_vma->vm_page_prot);
+       _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
+       set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd);
+
+       pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd);
+       pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
Quoted 7 lines

Are we allowed to move page tables between mm_structs on all architectures? The first example I found that looks a bit dodgy, looking through various architectures' pte_alloc_one(), is s390's page_table_alloc() which looks like page tables are tied to per-MM lists sometimes.
If that's not allowed, we might have to allocate a new deposit table and free the old one or something like that.

Hmm. Yeah, looks like in the case of !CONFIG_PGSTE the table can be linked to mm->context.pgtable_list, so can't be moved to another mm. I guess I'll have to keep a pgtable allocated, ready to be deposited and free the old one. Maybe it's worth having an arch-specific function indicating whether moving a pgtable between MMs is supported? Or do it separately as an optimization. WDYT?

Quoted 37 lines
Quoted 21 lines
mm/huge_memory.c +17 -0
+       if (dst_mm != src_mm) {
+               add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+               add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR);
+       }
+       double_pt_unlock(src_ptl, dst_ptl);
+
+       anon_vma_unlock_write(src_anon_vma);
+       put_anon_vma(src_anon_vma);
+
+       /* unblock rmap walks */
+       unlock_page(src_page);
+
+       mmu_notifier_invalidate_range_end(&range);
+       return 0;
+}
+#endif /* CONFIG_USERFAULTFD */
+
 /*
  * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise.
  *

[...]

Quoted 4 lines
mm/userfaultfd.c +0 -0
index 96d9eae5c7cc..0cca60dfa8f8 100644

[...]

Quoted 4 lines
mm/userfaultfd.c +4 -0
+ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+                   unsigned long dst_start, unsigned long src_start,
+                   unsigned long len, __u64 mode)
+{

[...]

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1260

Quoted 4 lines
mm/userfaultfd.c +4 -0
+
+       if (pgprot_val(src_vma->vm_page_prot) !=
+           pgprot_val(dst_vma->vm_page_prot))
+               goto out;
Quoted 6 lines

Does this check intentionally allow moving pages from a PROT_READ|PROT_WRITE anonymous private VMA into a PROT_READ anonymous private VMA (on architectures like x86 and arm64 where CoW memory has the same protection flags as read-only memory), but forbid moving them from a PROT_READ|PROT_EXEC VMA into a PROT_READ VMA? I think this check needs at least a comment to explain what's going on here.

The check is simply to ensure the VMAs have the same access permissions to prevent page copies that should have different permissions. The fact that x86 and arm64 have the same protection bits for R/O and COW memory is a "side-effect" IMHO. I'm not sure what comment would be good here but I'm open to suggestions.

Quoted 37 lines
Quoted 20 lines
mm/userfaultfd.c +19 -0
+       /* only allow remapping if both are mlocked or both aren't */
+       if ((src_vma->vm_flags & VM_LOCKED) ^ (dst_vma->vm_flags & VM_LOCKED))
+               goto out;
+
+       /*
+        * Be strict and only allow remap_pages if either the src or
+        * dst range is registered in the userfaultfd to prevent
+        * userland errors going unnoticed. As far as the VM
+        * consistency is concerned, it would be perfectly safe to
+        * remove this check, but there's no useful usage for
+        * remap_pages ouside of userfaultfd registered ranges. This
+        * is after all why it is an ioctl belonging to the
+        * userfaultfd and not a syscall.
+        *
+        * Allow both vmas to be registered in the userfaultfd, just
+        * in case somebody finds a way to make such a case useful.
+        * Normally only one of the two vmas would be registered in
+        * the userfaultfd.
+        */

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1290

Quoted 10 lines
mm/userfaultfd.c +10 -0
+       if (!dst_vma->vm_userfaultfd_ctx.ctx &&
+           !src_vma->vm_userfaultfd_ctx.ctx)
+               goto out;
+
+       /*
+        * FIXME: only allow remapping across anonymous vmas,
+        * tmpfs should be added.
+        */
+       if (src_vma->vm_ops || dst_vma->vm_ops)
+               goto out;

I don't think it's okay to check for anonymous VMAs by checking ->vm_ops. There are some weird drivers whose ->mmap helpers don't set ->vm_ops and instead just shove all the necessary PTEs into the VMA right on ->mmap, so I think they end up with ->vm_ops==NULL. For example, kcov_mmap() looks that way. I'm not sure how common this is.

Quoted 6 lines

Though, uuuuuh, I guess if that's true, the existing vma_is_anonymous() is broken, since that also just checks ->vm_ops? I'm not sure what the consequences of that would be... Either way, vma_is_anonymous() might be the better way to check for anonymous VMAs here, and someone should figure out whether vma_is_anonymous() needs to be fixed.

Absolutely, vma_is_anonymous() should have been used here. Will fix it.

Quoted 13 lines
Quoted 13 lines
mm/userfaultfd.c +12 -0
+       /*
+        * Ensure the dst_vma has a anon_vma or this page
+        * would get a NULL anon_vma when moved in the
+        * dst_vma.
+        */
+       err = -ENOMEM;
+       if (unlikely(anon_vma_prepare(dst_vma)))
+               goto out;
+
+       for (src_addr = src_start, dst_addr = dst_start;
+            src_addr < src_start + len;) {
+               spinlock_t *ptl;
Quoted 8 lines

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1307

Quoted 4 lines
mm/userfaultfd.c +4 -0
+               pmd_t dst_pmdval;
+
+               BUG_ON(dst_addr >= dst_start + len);
+               src_pmd = mm_find_pmd(src_mm, src_addr);

(this would blow up pretty badly if we could have transparent huge PUD in the region but I think that's limited to file VMAs so it's fine as it currently is)

Should I add a comment here as a warning if in the future we decide to implement support for file-backed pages?

Quoted 28 lines
Quoted 17 lines
mm/userfaultfd.c +16 -0
+               if (unlikely(!src_pmd)) {
+                       if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES)) {
+                               err = -ENOENT;
+                               break;
+                       }
+                       src_pmd = mm_alloc_pmd(src_mm, src_addr);
+                       if (unlikely(!src_pmd)) {
+                               err = -ENOMEM;
+                               break;
+                       }
+               }
+               dst_pmd = mm_alloc_pmd(dst_mm, dst_addr);
+               if (unlikely(!dst_pmd)) {
+                       err = -ENOMEM;
+                       break;
+               }

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1333

Quoted 10 lines
mm/userfaultfd.c +10 -0
+
+               dst_pmdval = pmdp_get_lockless(dst_pmd);
+               /*
+                * If the dst_pmd is mapped as THP don't
+                * override it and just be strict.
+                */
+               if (unlikely(pmd_trans_huge(dst_pmdval))) {
+                       err = -EEXIST;
+                       break;
+               }
Quoted 5 lines

This check is racy because the dst_pmd can still change at this point, from previously pointing to a zeroed PMD to now pointing to a hugepage, right? And we rely on remap_pages_pte() and remap_pages_huge_pmd() to recheck for that?
If yes, maybe add a comment noting this and explaining why we want this check.

Quoted 8 lines

Yes, you are right. remap_pages_huge_pmd() does check pmd_same(*dst_pmd, dst_pmdval) after doing double_pt_lock(), so it will retry and exit on this check if dst_pmd got mapped as THP from under us. remap_pages_pte() OTOH simply does BUG_ON(pmd_trans_huge(*dst_pmd)), which is wrong. Instead it should also check if PMD value changed from under us and retry in such a case. I'll fix that and will add a comment here about this racy check and the retry logic.

Actually, remap_pages_pte() also indirectly handles the case when dst_pmd changes from under us with this code:

dst_pte = pte_offset_map_nolock(dst_mm, dst_pmd, dst_addr, &dst_ptl); /* If an huge pmd materialized from under us fail */

if (unlikely(!dst_pte)) {
        err = -EFAULT;
        goto out;
}

so, it would fail if someone concurrently mapped a THP at that location. Seems like this race is handled, so the only thing left here is to add a comment clarifying this.

Quoted 146 lines
Quoted 18 lines
Quoted 7 lines
mm/userfaultfd.c +5 -0
+               ptl = pmd_trans_huge_lock(src_pmd, src_vma);
+               if (ptl) {
+                       /*
+                        * Check if we can move the pmd without
+                        * splitting it. First check the address

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1348

Quoted 10 lines
mm/userfaultfd.c +10 -0
+                        * alignment to be the same in src/dst.  These
+                        * checks don't actually need the PT lock but
+                        * it's good to do it here to optimize this
+                        * block away at build time if
+                        * CONFIG_TRANSPARENT_HUGEPAGE is not set.
+                        */
+                       if (thp_aligned == -1)
+                               thp_aligned = ((src_addr & ~HPAGE_PMD_MASK) ==
+                                              (dst_addr & ~HPAGE_PMD_MASK));
+                       if (!thp_aligned || (src_addr & ~HPAGE_PMD_MASK) ||
Quoted 6 lines

This seems overly complicated, the only case when you can move a huge PMD is if both addresses are hugepage-aligned and you have enough length for one hugepage:

(src_addr & ~HPAGE_PMD_MASK) == 0 && (dst_addr & ~HPAGE_PMD_MASK) == 0 && (src_start + len - src_addr >= HPAGE_PMD_SIZE).

Ack.

Quoted 42 lines
Quoted 42 lines
mm/userfaultfd.c +41 -0
+                           !pmd_none(dst_pmdval) ||
+                           src_start + len - src_addr < HPAGE_PMD_SIZE) {
+                               spin_unlock(ptl);
+                               /* Fall through */
+                               split_huge_pmd(src_vma, src_pmd, src_addr);
+                       } else {
+                               err = remap_pages_huge_pmd(dst_mm,
+                                                          src_mm,
+                                                          dst_pmd,
+                                                          src_pmd,
+                                                          dst_pmdval,
+                                                          dst_vma,
+                                                          src_vma,
+                                                          dst_addr,
+                                                          src_addr);
+                               cond_resched();
+
+                               if (!err) {
+                                       dst_addr += HPAGE_PMD_SIZE;
+                                       src_addr += HPAGE_PMD_SIZE;
+                                       moved += HPAGE_PMD_SIZE;
+                               }
+
+                               if ((!err || err == -EAGAIN) &&
+                                   fatal_signal_pending(current))
+                                       err = -EINTR;
+
+                               if (err && err != -EAGAIN)
+                                       break;
+
+                               continue;
+                       }
+               }
+
+               if (pmd_none(*src_pmd)) {
+                       if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES)) {
+                               err = -ENOENT;
+                               break;
+                       }
+                       if (unlikely(__pte_alloc(src_mm, src_pmd))) {
+                               err = -ENOMEM;
Quoted 8 lines

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1395

Quoted 6 lines
mm/userfaultfd.c +6 -0
+                               break;
+                       }
+               }
+
+               if (unlikely(pmd_none(dst_pmdval)) &&
+                   unlikely(__pte_alloc(dst_mm, dst_pmd))) {

Maybe just use pte_alloc() here?

Ack.

Quoted 53 lines
Quoted 26 lines
mm/userfaultfd.c +25 -0
+                       err = -ENOMEM;
+                       break;
+               }
+
+               err = remap_pages_pte(dst_mm, src_mm,
+                                     dst_pmd, src_pmd,
+                                     dst_vma, src_vma,
+                                     dst_addr, src_addr,
+                                     mode);
+
+               cond_resched();
+
+               if (!err) {
+                       dst_addr += PAGE_SIZE;
+                       src_addr += PAGE_SIZE;
+                       moved += PAGE_SIZE;
+               }
+
+               if ((!err || err == -EAGAIN) &&
+                   fatal_signal_pending(current))
+                       err = -EINTR;
+
+               if (err && err != -EAGAIN)
+                       break;
+       }

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1430

Quoted 10 lines
mm/userfaultfd.c +10 -0
+
+out:
+       mmap_read_unlock(dst_mm);
+       if (dst_mm != src_mm)
+               mmap_read_unlock(src_mm);
+       BUG_ON(moved < 0);
+       BUG_ON(err > 0);
+       BUG_ON(!moved && !err);
+       return moved ? moved : err;
+}

Maybe you could try whether this function would look simpler with a shape roughly like:

for (src_addr = ...; src_addr < ...;) {
unsigned long step_size;

  if (hugepage case) {
    if (have to split) {
      split it;
      continue;
    }
    step_size = HPAGE_PMD_SIZE;
    ...
  } else {
    ... 4k case ...
    step_size = PAGE_SIZE;
Quoted 8 lines
  }
  ...
  cond_resched();
  if (!err) {
    dst_addr += step_size;
    src_addr += step_size;
    moved += step_size;
  }

Yeah, that looks simpler. I'll try that, thanks!

Quoted 2 lines

...
}

reviewer
#17 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

In reply to Suren Baghdasaryan
Quoted 8 lines · Suren Baghdasaryan
Quoted 7 lines · Jann Horn
Quoted 6 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

[...]

Quoted 1 line
Quoted 1 line
Quoted 1 line
mm/huge_memory.c +0 -0

[...]

Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
mm/huge_memory.c +17 -0
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                        struct mm_struct *src_mm,
+                        pmd_t *dst_pmd, pmd_t *src_pmd,
+                        pmd_t dst_pmdval,
+                        struct vm_area_struct *dst_vma,
+                        struct vm_area_struct *src_vma,
+                        unsigned long dst_addr,
+                        unsigned long src_addr)
+{
+       pmd_t _dst_pmd, src_pmdval;
+       struct page *src_page;
+       struct anon_vma *src_anon_vma, *dst_anon_vma;
+       spinlock_t *src_ptl, *dst_ptl;
+       pgtable_t pgtable;
+       struct mmu_notifier_range range;
+
+       src_pmdval = *src_pmd;
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1965

Quoted 4 lines
mm/huge_memory.c +4 -0
+       src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+       BUG_ON(!pmd_trans_huge(src_pmdval));
+       BUG_ON(!pmd_none(dst_pmdval));

Why can we assert that pmd_none(dst_pmdval) is true here? Can we not have concurrent faults (or userfaultfd operations) populating that PMD?

Quoted 6 lines

IIUC dst_pmdval is a copy of the value from dst_pmd, so that local copy should not change even if some concurrent operation changes dst_pmd. We can assert that it's pmd_none because we checked for that before calling remap_pages_huge_pmd. Later on we check if dst_pmd changed from under us (see pmd_same(*dst_pmd, dst_pmdval) check) and retry if that happened.

Oh, right, I don't know what I was thinking when I typed that.

But now I wonder about the check directly above that: What does this code do for swap PMDs? It looks like that might splat on the BUG_ON(!pmd_trans_huge(src_pmdval)). All we've checked on the path to here is that the virtual memory area is aligned, that the destination PMD is empty, and that pmd_trans_huge_lock() succeeded; but pmd_trans_huge_lock() explicitly permits swap PMDs (which is the swapped-out version of transhuge PMDs):

static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
struct vm_area_struct *vma)

{
        if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd))
                return __pmd_trans_huge_lock(pmd, vma);
        else
                return NULL;
}
Quoted 69 lines
Quoted 60 lines
Quoted 60 lines
mm/huge_memory.c +59 -0
+       BUG_ON(!spin_is_locked(src_ptl));
+       mmap_assert_locked(src_mm);
+       mmap_assert_locked(dst_mm);
+       BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+       BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+       src_page = pmd_page(src_pmdval);
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       if (unlikely(page_mapcount(src_page) != 1)) {
+               spin_unlock(src_ptl);
+               return -EBUSY;
+       }
+
+       get_page(src_page);
+       spin_unlock(src_ptl);
+
+       mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr,
+                               src_addr + HPAGE_PMD_SIZE);
+       mmu_notifier_invalidate_range_start(&range);
+
+       /* block all concurrent rmap walks */
+       lock_page(src_page);
+
+       /*
+        * split_huge_page walks the anon_vma chain without the page
+        * lock. Serialize against it with the anon_vma lock, the page
+        * lock is not enough.
+        */
+       src_anon_vma = folio_get_anon_vma(page_folio(src_page));
+       if (!src_anon_vma) {
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+       anon_vma_lock_write(src_anon_vma);
+
+       dst_ptl = pmd_lockptr(dst_mm, dst_pmd);
+       double_pt_lock(src_ptl, dst_ptl);
+       if (unlikely(!pmd_same(*src_pmd, src_pmdval) ||
+                    !pmd_same(*dst_pmd, dst_pmdval) ||
+                    page_mapcount(src_page) != 1)) {
+               double_pt_unlock(src_ptl, dst_ptl);
+               anon_vma_unlock_write(src_anon_vma);
+               put_anon_vma(src_anon_vma);
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       /* the PT lock is enough to keep the page pinned now */
+       put_page(src_page);
+
+       dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
+       WRITE_ONCE(src_page->mapping, (struct address_space *) dst_anon_vma);
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2029

Quoted 5 lines
mm/huge_memory.c +5 -0
+       WRITE_ONCE(src_page->index, linear_page_index(dst_vma, dst_addr));
+
+       if (!pmd_same(pmdp_huge_clear_flush(src_vma, src_addr, src_pmd),
+                     src_pmdval))
+               BUG_ON(1);

I'm not sure we can assert that the PMDs are exactly equal; the CPU might have changed the A/D bits under us?

Quoted 7 lines

Yes. I wonder if I can simply remove the BUG_ON here like this:

src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);

Technically we don't use src_pmdval after this but for the possible future use that would keep things correct. If A/D bits changed from under us we will still copy correct values into dst_pmd.

And when we set up the dst_pmd, we always mark it as dirty and accessed... so I guess that's fine.

Quoted 15 lines
Quoted 7 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2035

Quoted 6 lines
mm/huge_memory.c +6 -0
+       _dst_pmd = mk_huge_pmd(src_page, dst_vma->vm_page_prot);
+       _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
+       set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd);
+
+       pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd);
+       pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
Quoted 7 lines

Are we allowed to move page tables between mm_structs on all architectures? The first example I found that looks a bit dodgy, looking through various architectures' pte_alloc_one(), is s390's page_table_alloc() which looks like page tables are tied to per-MM lists sometimes.
If that's not allowed, we might have to allocate a new deposit table and free the old one or something like that.

Quoted 6 lines

Hmm. Yeah, looks like in the case of !CONFIG_PGSTE the table can be linked to mm->context.pgtable_list, so can't be moved to another mm. I guess I'll have to keep a pgtable allocated, ready to be deposited and free the old one. Maybe it's worth having an arch-specific function indicating whether moving a pgtable between MMs is supported? Or do it separately as an optimization. WDYT?

Hm, dunno. I guess you could have architectures opt in with some config flag similar to how flags like ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH are wired up - define it in init/Kconfig, select it in the architectures that support it, and then gate the fast version on that with #ifdef?

Quoted 43 lines
Quoted 36 lines
Quoted 20 lines
mm/huge_memory.c +17 -0
+       if (dst_mm != src_mm) {
+               add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+               add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR);
+       }
+       double_pt_unlock(src_ptl, dst_ptl);
+
+       anon_vma_unlock_write(src_anon_vma);
+       put_anon_vma(src_anon_vma);
+
+       /* unblock rmap walks */
+       unlock_page(src_page);
+
+       mmu_notifier_invalidate_range_end(&range);
+       return 0;
+}
+#endif /* CONFIG_USERFAULTFD */
+
 /*
  * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise.
  *

[...]

Quoted 4 lines
mm/userfaultfd.c +0 -0
index 96d9eae5c7cc..0cca60dfa8f8 100644

[...]

Quoted 4 lines
mm/userfaultfd.c +4 -0
+ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+                   unsigned long dst_start, unsigned long src_start,
+                   unsigned long len, __u64 mode)
+{

[...]

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1260

Quoted 4 lines
mm/userfaultfd.c +4 -0
+
+       if (pgprot_val(src_vma->vm_page_prot) !=
+           pgprot_val(dst_vma->vm_page_prot))
+               goto out;
Quoted 6 lines

Does this check intentionally allow moving pages from a PROT_READ|PROT_WRITE anonymous private VMA into a PROT_READ anonymous private VMA (on architectures like x86 and arm64 where CoW memory has the same protection flags as read-only memory), but forbid moving them from a PROT_READ|PROT_EXEC VMA into a PROT_READ VMA? I think this check needs at least a comment to explain what's going on here.

Quoted 5 lines

The check is simply to ensure the VMAs have the same access permissions to prevent page copies that should have different permissions. The fact that x86 and arm64 have the same protection bits for R/O and COW memory is a "side-effect" IMHO. I'm not sure what comment would be good here but I'm open to suggestions.

I'm not sure if you can do a meaningful security check on the ->vm_page_prot. I also don't think it matters for anonymous VMAs. I guess if you want to keep this check but make this behavior more consistent, you could put another check in front of this that rejects VMAs where vm_flags like VM_READ, VM_WRITE, VM_SHARED or VM_EXEC are different?

[...]

Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
mm/userfaultfd.c +12 -0
+       /*
+        * Ensure the dst_vma has a anon_vma or this page
+        * would get a NULL anon_vma when moved in the
+        * dst_vma.
+        */
+       err = -ENOMEM;
+       if (unlikely(anon_vma_prepare(dst_vma)))
+               goto out;
+
+       for (src_addr = src_start, dst_addr = dst_start;
+            src_addr < src_start + len;) {
+               spinlock_t *ptl;
Quoted 8 lines

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1307

Quoted 4 lines
mm/userfaultfd.c +4 -0
+               pmd_t dst_pmdval;
+
+               BUG_ON(dst_addr >= dst_start + len);
+               src_pmd = mm_find_pmd(src_mm, src_addr);

(this would blow up pretty badly if we could have transparent huge PUD in the region but I think that's limited to file VMAs so it's fine as it currently is)

Quoted 2 lines

Should I add a comment here as a warning if in the future we decide to implement support for file-backed pages?

Hm, yeah, I guess that might be a good idea.

author reviewer
#18 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 56 lines · Jann Horn
Quoted 8 lines · Suren Baghdasaryan
Quoted 7 lines · Jann Horn
Quoted 6 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

[...]

Quoted 1 line
Quoted 1 line
Quoted 1 line
mm/huge_memory.c +0 -0

[...]

Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
mm/huge_memory.c +17 -0
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                        struct mm_struct *src_mm,
+                        pmd_t *dst_pmd, pmd_t *src_pmd,
+                        pmd_t dst_pmdval,
+                        struct vm_area_struct *dst_vma,
+                        struct vm_area_struct *src_vma,
+                        unsigned long dst_addr,
+                        unsigned long src_addr)
+{
+       pmd_t _dst_pmd, src_pmdval;
+       struct page *src_page;
+       struct anon_vma *src_anon_vma, *dst_anon_vma;
+       spinlock_t *src_ptl, *dst_ptl;
+       pgtable_t pgtable;
+       struct mmu_notifier_range range;
+
+       src_pmdval = *src_pmd;
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1965

Quoted 4 lines
mm/huge_memory.c +4 -0
+       src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+       BUG_ON(!pmd_trans_huge(src_pmdval));
+       BUG_ON(!pmd_none(dst_pmdval));

Why can we assert that pmd_none(dst_pmdval) is true here? Can we not have concurrent faults (or userfaultfd operations) populating that PMD?

Quoted 6 lines

IIUC dst_pmdval is a copy of the value from dst_pmd, so that local copy should not change even if some concurrent operation changes dst_pmd. We can assert that it's pmd_none because we checked for that before calling remap_pages_huge_pmd. Later on we check if dst_pmd changed from under us (see pmd_same(*dst_pmd, dst_pmdval) check) and retry if that happened.

Oh, right, I don't know what I was thinking when I typed that.

But now I wonder about the check directly above that: What does this code do for swap PMDs? It looks like that might splat on the BUG_ON(!pmd_trans_huge(src_pmdval)). All we've checked on the path to here is that the virtual memory area is aligned, that the destination PMD is empty, and that pmd_trans_huge_lock() succeeded; but pmd_trans_huge_lock() explicitly permits swap PMDs (which is the swapped-out version of transhuge PMDs):

Quoted 8 lines

static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
struct vm_area_struct *vma)

{
        if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd))
                return __pmd_trans_huge_lock(pmd, vma);
        else
                return NULL;
}

Yeah... Ok, I think I'm missing a check for pmd_trans_huge(*src_pmd) after we lock it with pmd_trans_huge_lock(src_pmd, src_vma). And we can remove the above BUG_ON(). Would that address your concern?

Quoted 72 lines
Quoted 72 lines
Quoted 61 lines
Quoted 61 lines
mm/huge_memory.c +59 -0
+       BUG_ON(!spin_is_locked(src_ptl));
+       mmap_assert_locked(src_mm);
+       mmap_assert_locked(dst_mm);
+       BUG_ON(src_addr & ~HPAGE_PMD_MASK);
+       BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
+
+       src_page = pmd_page(src_pmdval);
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       if (unlikely(page_mapcount(src_page) != 1)) {
+               spin_unlock(src_ptl);
+               return -EBUSY;
+       }
+
+       get_page(src_page);
+       spin_unlock(src_ptl);
+
+       mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr,
+                               src_addr + HPAGE_PMD_SIZE);
+       mmu_notifier_invalidate_range_start(&range);
+
+       /* block all concurrent rmap walks */
+       lock_page(src_page);
+
+       /*
+        * split_huge_page walks the anon_vma chain without the page
+        * lock. Serialize against it with the anon_vma lock, the page
+        * lock is not enough.
+        */
+       src_anon_vma = folio_get_anon_vma(page_folio(src_page));
+       if (!src_anon_vma) {
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+       anon_vma_lock_write(src_anon_vma);
+
+       dst_ptl = pmd_lockptr(dst_mm, dst_pmd);
+       double_pt_lock(src_ptl, dst_ptl);
+       if (unlikely(!pmd_same(*src_pmd, src_pmdval) ||
+                    !pmd_same(*dst_pmd, dst_pmdval) ||
+                    page_mapcount(src_page) != 1)) {
+               double_pt_unlock(src_ptl, dst_ptl);
+               anon_vma_unlock_write(src_anon_vma);
+               put_anon_vma(src_anon_vma);
+               unlock_page(src_page);
+               put_page(src_page);
+               mmu_notifier_invalidate_range_end(&range);
+               return -EAGAIN;
+       }
+
+       BUG_ON(!PageHead(src_page));
+       BUG_ON(!PageAnon(src_page));
+       /* the PT lock is enough to keep the page pinned now */
+       put_page(src_page);
+
+       dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
+       WRITE_ONCE(src_page->mapping, (struct address_space *) dst_anon_vma);
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2029

Quoted 5 lines
mm/huge_memory.c +5 -0
+       WRITE_ONCE(src_page->index, linear_page_index(dst_vma, dst_addr));
+
+       if (!pmd_same(pmdp_huge_clear_flush(src_vma, src_addr, src_pmd),
+                     src_pmdval))
+               BUG_ON(1);

I'm not sure we can assert that the PMDs are exactly equal; the CPU might have changed the A/D bits under us?

Yes. I wonder if I can simply remove the BUG_ON here like this:

Quoted 8 lines
Quoted 5 lines

src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);

Technically we don't use src_pmdval after this but for the possible future use that would keep things correct. If A/D bits changed from under us we will still copy correct values into dst_pmd.

And when we set up the dst_pmd, we always mark it as dirty and accessed... so I guess that's fine.

Ack.

Quoted 23 lines
Quoted 16 lines
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 2035

Quoted 7 lines
mm/huge_memory.c +6 -0
+       _dst_pmd = mk_huge_pmd(src_page, dst_vma->vm_page_prot);
+       _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
+       set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd);
+
+       pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd);
+       pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
Quoted 7 lines

Are we allowed to move page tables between mm_structs on all architectures? The first example I found that looks a bit dodgy, looking through various architectures' pte_alloc_one(), is s390's page_table_alloc() which looks like page tables are tied to per-MM lists sometimes.
If that's not allowed, we might have to allocate a new deposit table and free the old one or something like that.

Quoted 6 lines

Hmm. Yeah, looks like in the case of !CONFIG_PGSTE the table can be linked to mm->context.pgtable_list, so can't be moved to another mm. I guess I'll have to keep a pgtable allocated, ready to be deposited and free the old one. Maybe it's worth having an arch-specific function indicating whether moving a pgtable between MMs is supported? Or do it separately as an optimization. WDYT?

Quoted 5 lines

Hm, dunno. I guess you could have architectures opt in with some config flag similar to how flags like ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH are wired up - define it in init/Kconfig, select it in the architectures that support it, and then gate the fast version on that with #ifdef?

Yeah, that sounds good to me. I can implement an unoptimized common path first and then introduce this optimization in the follow-up patches.

Quoted 50 lines
Quoted 44 lines
Quoted 37 lines
Quoted 21 lines
mm/huge_memory.c +17 -0
+       if (dst_mm != src_mm) {
+               add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+               add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR);
+       }
+       double_pt_unlock(src_ptl, dst_ptl);
+
+       anon_vma_unlock_write(src_anon_vma);
+       put_anon_vma(src_anon_vma);
+
+       /* unblock rmap walks */
+       unlock_page(src_page);
+
+       mmu_notifier_invalidate_range_end(&range);
+       return 0;
+}
+#endif /* CONFIG_USERFAULTFD */
+
 /*
  * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise.
  *

[...]

Quoted 4 lines
mm/userfaultfd.c +0 -0
index 96d9eae5c7cc..0cca60dfa8f8 100644

[...]

Quoted 4 lines
mm/userfaultfd.c +4 -0
+ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+                   unsigned long dst_start, unsigned long src_start,
+                   unsigned long len, __u64 mode)
+{

[...]

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1260

Quoted 4 lines
mm/userfaultfd.c +4 -0
+
+       if (pgprot_val(src_vma->vm_page_prot) !=
+           pgprot_val(dst_vma->vm_page_prot))
+               goto out;
Quoted 6 lines

Does this check intentionally allow moving pages from a PROT_READ|PROT_WRITE anonymous private VMA into a PROT_READ anonymous private VMA (on architectures like x86 and arm64 where CoW memory has the same protection flags as read-only memory), but forbid moving them from a PROT_READ|PROT_EXEC VMA into a PROT_READ VMA? I think this check needs at least a comment to explain what's going on here.

Quoted 5 lines

The check is simply to ensure the VMAs have the same access permissions to prevent page copies that should have different permissions. The fact that x86 and arm64 have the same protection bits for R/O and COW memory is a "side-effect" IMHO. I'm not sure what comment would be good here but I'm open to suggestions.

Quoted 6 lines

I'm not sure if you can do a meaningful security check on the ->vm_page_prot. I also don't think it matters for anonymous VMAs. I guess if you want to keep this check but make this behavior more consistent, you could put another check in front of this that rejects VMAs where vm_flags like VM_READ, VM_WRITE, VM_SHARED or VM_EXEC are different?

Ok, I'll post that in the next version and we can decide if that's enough.

Quoted 23 lines

[...]

Quoted 21 lines
Quoted 20 lines
Quoted 6 lines
mm/userfaultfd.c +6 -0
+       /*
+        * Ensure the dst_vma has a anon_vma or this page
+        * would get a NULL anon_vma when moved in the
+        * dst_vma.
+        */
+       err = -ENOMEM;

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1307

Quoted 10 lines
mm/userfaultfd.c +10 -0
+       if (unlikely(anon_vma_prepare(dst_vma)))
+               goto out;
+
+       for (src_addr = src_start, dst_addr = dst_start;
+            src_addr < src_start + len;) {
+               spinlock_t *ptl;
+               pmd_t dst_pmdval;
+
+               BUG_ON(dst_addr >= dst_start + len);
+               src_pmd = mm_find_pmd(src_mm, src_addr);

(this would blow up pretty badly if we could have transparent huge PUD in the region but I think that's limited to file VMAs so it's fine as it currently is)

Quoted 4 lines
Quoted 2 lines

Should I add a comment here as a warning if in the future we decide to implement support for file-backed pages?

Hm, yeah, I guess that might be a good idea.

Ack.

Thanks for the feedback!

reviewer
#19 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 65 lines · Suren Baghdasaryan
Quoted 55 lines · Jann Horn
Quoted 8 lines · Suren Baghdasaryan
Quoted 7 lines · Jann Horn
Quoted 6 lines · Suren Baghdasaryan

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

[...]

Quoted 1 line
Quoted 1 line
Quoted 1 line
mm/huge_memory.c +0 -0

[...]

Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
mm/huge_memory.c +17 -0
+int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+                        struct mm_struct *src_mm,
+                        pmd_t *dst_pmd, pmd_t *src_pmd,
+                        pmd_t dst_pmdval,
+                        struct vm_area_struct *dst_vma,
+                        struct vm_area_struct *src_vma,
+                        unsigned long dst_addr,
+                        unsigned long src_addr)
+{
+       pmd_t _dst_pmd, src_pmdval;
+       struct page *src_page;
+       struct anon_vma *src_anon_vma, *dst_anon_vma;
+       spinlock_t *src_ptl, *dst_ptl;
+       pgtable_t pgtable;
+       struct mmu_notifier_range range;
+
+       src_pmdval = *src_pmd;
Quoted 8 lines

mm/huge_memory.c int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · line 1965

Quoted 4 lines
mm/huge_memory.c +4 -0
+       src_ptl = pmd_lockptr(src_mm, src_pmd);
+
+       BUG_ON(!pmd_trans_huge(src_pmdval));
+       BUG_ON(!pmd_none(dst_pmdval));

Why can we assert that pmd_none(dst_pmdval) is true here? Can we not have concurrent faults (or userfaultfd operations) populating that PMD?

Quoted 6 lines

IIUC dst_pmdval is a copy of the value from dst_pmd, so that local copy should not change even if some concurrent operation changes dst_pmd. We can assert that it's pmd_none because we checked for that before calling remap_pages_huge_pmd. Later on we check if dst_pmd changed from under us (see pmd_same(*dst_pmd, dst_pmdval) check) and retry if that happened.

Oh, right, I don't know what I was thinking when I typed that.

But now I wonder about the check directly above that: What does this code do for swap PMDs? It looks like that might splat on the BUG_ON(!pmd_trans_huge(src_pmdval)). All we've checked on the path to here is that the virtual memory area is aligned, that the destination PMD is empty, and that pmd_trans_huge_lock() succeeded; but pmd_trans_huge_lock() explicitly permits swap PMDs (which is the swapped-out version of transhuge PMDs):

Quoted 8 lines

static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
struct vm_area_struct *vma)

{
        if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd))
                return __pmd_trans_huge_lock(pmd, vma);
        else
                return NULL;
}
Quoted 3 lines

Yeah... Ok, I think I'm missing a check for pmd_trans_huge(*src_pmd) after we lock it with pmd_trans_huge_lock(src_pmd, src_vma). And we can remove the above BUG_ON(). Would that address your concern?

Sounds good. It'll end up splitting huge swap entries but I guess the extra code for moving huge swap entries might not be worth it.

reviewer
#20 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

In reply to Jann Horn
Quoted 6 lines · Jann Horn

Though, uuuuuh, I guess if that's true, the existing vma_is_anonymous() is broken, since that also just checks ->vm_ops? I'm not sure what the consequences of that would be... Either way, vma_is_anonymous() might be the better way to check for anonymous VMAs here, and someone should figure out whether vma_is_anonymous() needs to be fixed.

(Not really relevant to the rest of the thread, but just as a
sidenote: Turns out this is not an issue; since commit bfd40eaff5ab ("mm: fix vma_is_anonymous() false-positives"), VMAs where the ->mmap handler does not set an operations pointer end up with a dummy operations pointer.)

#21 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

In reply to Suren Baghdasaryan

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1037

Quoted 6 lines · Suren Baghdasaryan
mm/userfaultfd.c +3 -0
+ 	if (!pte_same(ptep_clear_flush(src_vma, src_addr, src_pte),
+       	orig_src_pte))
+ 		BUG_ON(1);

Just a minor detail regarding these few lines:

Besides the less-than-ideal use of BUG_ON() here, I think that this code assumes that the PTE cannot change at this point. However, as the PTE was still mapped at this point, I think the access and dirty bits can be set.

tl;dr: this appears to be triggerable by userspace.

[ as for the performance of this code, the lack of batching would mean
that for multithreaded applications where more than a single page is remapped, performance would suffer ]

author reviewer
#22 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 8 lines · Nadav Amit

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1037

Quoted 7 lines · Suren Baghdasaryan
mm/userfaultfd.c +3 -0
+     if (!pte_same(ptep_clear_flush(src_vma, src_addr, src_pte),
+             orig_src_pte))
+             BUG_ON(1);
Quoted 5 lines

Just a minor detail regarding these few lines:

Besides the less-than-ideal use of BUG_ON() here, I think that this code assumes that the PTE cannot change at this point. However, as the PTE was still mapped at this point, I think the access and dirty bits can be set.

At this point we are holding PTLs for both PTEs (see double_pt_lock()). Can a PTE be modified from under us in this situation?

Quoted 6 lines

tl;dr: this appears to be triggerable by userspace.

[ as for the performance of this code, the lack of batching would mean
that for multithreaded applications where more than a single page is remapped, performance would suffer ]

Thanks for the note! I'll see if it's possible to implement some batching mechanism here.
Thanks,
Suren.

#23 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 15 lines · Suren Baghdasaryan
Quoted 10 lines · Nadav Amit

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1037

Quoted 7 lines · Suren Baghdasaryan
mm/userfaultfd.c +3 -0
+     if (!pte_same(ptep_clear_flush(src_vma, src_addr, src_pte),
+             orig_src_pte))
+             BUG_ON(1);

Just a minor detail regarding these few lines:

Quoted 7 lines
Quoted 3 lines

Besides the less-than-ideal use of BUG_ON() here, I think that this code assumes that the PTE cannot change at this point. However, as the PTE was still mapped at this point, I think the access and dirty bits can be set.

At this point we are holding PTLs for both PTEs (see double_pt_lock()). Can a PTE be modified from under us in this situation?

PTEs has several parts: access-control bits (e.g., writable), physical frame number, software-only bits and log-bits. The log-bits, which are “access” and “dirty” on x86, track whether the PTE has ever been used for translation or write correspondingly.

Without getting into all the subtleties (e.g., “access" can be set speculatively even if no actual access take place), as long as the PTE is present, it might be used for access (and write if it is writable) by other cores. The page-table locks are irrelevant here, because the PTE is not updated by software, but it is updated by the CPU itself during the page-walk/write.

author reviewer
#24 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 29 lines · Nadav Amit
Quoted 16 lines · Suren Baghdasaryan
Quoted 10 lines · Nadav Amit

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1037

Quoted 7 lines · Suren Baghdasaryan
mm/userfaultfd.c +3 -0
+     if (!pte_same(ptep_clear_flush(src_vma, src_addr, src_pte),
+             orig_src_pte))
+             BUG_ON(1);

Just a minor detail regarding these few lines:

Quoted 7 lines
Quoted 3 lines

Besides the less-than-ideal use of BUG_ON() here, I think that this code assumes that the PTE cannot change at this point. However, as the PTE was still mapped at this point, I think the access and dirty bits can be set.

At this point we are holding PTLs for both PTEs (see double_pt_lock()). Can a PTE be modified from under us in this situation?

PTEs has several parts: access-control bits (e.g., writable), physical frame number, software-only bits and log-bits. The log-bits, which are “access” and “dirty” on x86, track whether the PTE has ever been used for translation or write correspondingly.

Quoted 6 lines

Without getting into all the subtleties (e.g., “access" can be set speculatively even if no actual access take place), as long as the PTE is present, it might be used for access (and write if it is writable) by other cores. The page-table locks are irrelevant here, because the PTE is not updated by software, but it is updated by the CPU itself during the page-walk/write.

Ah, I see. I believe Jann also pointed this out in one of his comments and I didn't realize that. Thanks for the note! I'll see how I can rework this.

Quoted 4 lines

--
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

reviewer
#25 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

In reply to Suren Baghdasaryan
Quoted 124 lines · Suren Baghdasaryan
mm/userfaultfd.c +124 -0
+/*
+ * The mmap_lock for reading is held by the caller. Just move the page
+ * from src_pmd to dst_pmd if possible, and return true if succeeded
+ * in moving the page.
+ */
+static int remap_pages_pte(struct mm_struct *dst_mm,
+                          struct mm_struct *src_mm,
+                          pmd_t *dst_pmd,
+                          pmd_t *src_pmd,
+                          struct vm_area_struct *dst_vma,
+                          struct vm_area_struct *src_vma,
+                          unsigned long dst_addr,
+                          unsigned long src_addr,
+                          __u64 mode)
+{
+       swp_entry_t entry;
+       pte_t orig_src_pte, orig_dst_pte;
+       spinlock_t *src_ptl, *dst_ptl;
+       pte_t *src_pte = NULL;
+       pte_t *dst_pte = NULL;
+
+       struct folio *src_folio = NULL;
+       struct anon_vma *src_anon_vma = NULL;
+       struct anon_vma *dst_anon_vma;
+       struct mmu_notifier_range range;
+       int err = 0;
+
+retry:
+       dst_pte = pte_offset_map_nolock(dst_mm, dst_pmd, dst_addr, &dst_ptl);
+
+       /* If an huge pmd materialized from under us fail */
+       if (unlikely(!dst_pte)) {
+               err = -EFAULT;
+               goto out;
+       }
+
+       src_pte = pte_offset_map_nolock(src_mm, src_pmd, src_addr, &src_ptl);
+
+       /*
+        * We held the mmap_lock for reading so MADV_DONTNEED
+        * can zap transparent huge pages under us, or the
+        * transparent huge page fault can establish new
+        * transparent huge pages under us.
+        */
+       if (unlikely(!src_pte)) {
+               err = -EFAULT;
+               goto out;
+       }
+
+       BUG_ON(pmd_none(*dst_pmd));
+       BUG_ON(pmd_none(*src_pmd));
+       BUG_ON(pmd_trans_huge(*dst_pmd));
+       BUG_ON(pmd_trans_huge(*src_pmd));
+
+       spin_lock(dst_ptl);
+       orig_dst_pte = *dst_pte;
+       spin_unlock(dst_ptl);
+       if (!pte_none(orig_dst_pte)) {
+               err = -EEXIST;
+               goto out;
+       }
+
+       spin_lock(src_ptl);
+       orig_src_pte = *src_pte;
+       spin_unlock(src_ptl);
+       if (pte_none(orig_src_pte)) {
+               if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES))
+                       err = -ENOENT;
+               else /* nothing to do to remap a hole */
+                       err = 0;
+               goto out;
+       }
+
+       if (pte_present(orig_src_pte)) {
+               if (!src_folio) {
+                       struct folio *folio;
+
+                       /*
+                        * Pin the page while holding the lock to be sure the
+                        * page isn't freed under us
+                        */
+                       spin_lock(src_ptl);
+                       if (!pte_same(orig_src_pte, *src_pte)) {
+                               spin_unlock(src_ptl);
+                               err = -EAGAIN;
+                               goto out;
+                       }
+
+                       folio = vm_normal_folio(src_vma, src_addr, orig_src_pte);
+                       if (!folio || !folio_test_anon(folio) ||
+                           folio_estimated_sharers(folio) != 1) {
+                               spin_unlock(src_ptl);
+                               err = -EBUSY;
+                               goto out;
+                       }
+
+                       src_folio = folio;
+                       folio_get(src_folio);
+                       spin_unlock(src_ptl);
+
+                       /* try to block all concurrent rmap walks */
+                       if (!folio_trylock(src_folio)) {
+                               pte_unmap(&orig_src_pte);
+                               pte_unmap(&orig_dst_pte);
+                               src_pte = dst_pte = NULL;
+                               folio_lock(src_folio);
+                               goto retry;
+                       }
+               }
+
+               if (!src_anon_vma) {
+                       /*
+                        * folio_referenced walks the anon_vma chain
+                        * without the folio lock. Serialize against it with
+                        * the anon_vma lock, the folio lock is not enough.
+                        */
+                       src_anon_vma = folio_get_anon_vma(src_folio);
+                       if (!src_anon_vma) {
+                               /* page was unmapped from under us */
+                               err = -EAGAIN;
+                               goto out;
+                       }
+                       if (!anon_vma_trylock_write(src_anon_vma)) {
+                               pte_unmap(&orig_src_pte);

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1013

Quoted 10 lines
mm/userfaultfd.c +10 -0
+                               pte_unmap(&orig_dst_pte);
+                               src_pte = dst_pte = NULL;
+                               anon_vma_lock_write(src_anon_vma);
+                               goto retry;
+                       }
+               }
+
+               mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm,
+                                       src_addr, src_addr + PAGE_SIZE);
+               mmu_notifier_invalidate_range_start_nonblock(&range);

I'm pretty sure you're not allowed to use mmu_notifier_invalidate_range_start_nonblock(). Use mmu_notifier_invalidate_range_start() at a point where it's fine to block; perhaps all the way up in remap_pages() around the whole loop or something like that. mmu_notifier_invalidate_range_start_nonblock() is special magic sauce for OOM reaping (with deceptively inviting naming and no documentation that explains that this is evil hazmat code), and if you chase down what various users of MMU notifiers do when you just hand them a random notification where mmu_notifier_range_blockable() is false, you end up in fun paths like in KVM's kvm_mmu_notifier_invalidate_range_start(), which calls gfn_to_pfn_cache_invalidate_start(), which does this:

    /*
     * If the OOM reaper is active, then all vCPUs should have
     * been stopped already, so perform the request without
     * KVM_REQUEST_WAIT and be sad if any needed to be IPI'd.
     */
    if (!may_block)
       req &= ~KVM_REQUEST_WAIT;

    called = kvm_make_vcpus_request_mask(kvm, req, vcpu_bitmap);

    WARN_ON_ONCE(called && !may_block);

Which means if you hit this codepath from a place like userfaultfd where the process is probably still running (and not being OOM reaped in a context where it's guaranteed to have been killed and stopped), you could probably get KVM into a situation where it needs to wait for vCPUs to acknowledge that they've processed a message before it's safe to continue, but it can't wait because we're in nonsleepable context, and then it just continues without waiting and presumably the KVM TLB gets out of sync or something?

Quoted 122 lines
mm/userfaultfd.c +122 -0
+
+               double_pt_lock(dst_ptl, src_ptl);
+
+               if (!pte_same(*src_pte, orig_src_pte) ||
+                   !pte_same(*dst_pte, orig_dst_pte) ||
+                   folio_estimated_sharers(src_folio) != 1) {
+                       double_pt_unlock(dst_ptl, src_ptl);
+                       err = -EAGAIN;
+                       goto out;
+               }
+
+               BUG_ON(!folio_test_anon(src_folio));
+               /* the PT lock is enough to keep the page pinned now */
+               folio_put(src_folio);
+
+               dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
+               WRITE_ONCE(src_folio->mapping,
+                          (struct address_space *) dst_anon_vma);
+               WRITE_ONCE(src_folio->index, linear_page_index(dst_vma,
+                                                             dst_addr));
+
+               if (!pte_same(ptep_clear_flush(src_vma, src_addr, src_pte),
+                             orig_src_pte))
+                       BUG_ON(1);
+
+               orig_dst_pte = mk_pte(&src_folio->page, dst_vma->vm_page_prot);
+               orig_dst_pte = maybe_mkwrite(pte_mkdirty(orig_dst_pte),
+                                            dst_vma);
+
+               set_pte_at(dst_mm, dst_addr, dst_pte, orig_dst_pte);
+
+               if (dst_mm != src_mm) {
+                       inc_mm_counter(dst_mm, MM_ANONPAGES);
+                       dec_mm_counter(src_mm, MM_ANONPAGES);
+               }
+
+               double_pt_unlock(dst_ptl, src_ptl);
+
+               anon_vma_unlock_write(src_anon_vma);
+               mmu_notifier_invalidate_range_end(&range);
+               put_anon_vma(src_anon_vma);
+               src_anon_vma = NULL;
+
+               /* unblock rmap walks */
+               folio_unlock(src_folio);
+               src_folio = NULL;
+
+       } else {
+               struct swap_info_struct *si;
+               int swap_count;
+
+               entry = pte_to_swp_entry(orig_src_pte);
+               if (non_swap_entry(entry)) {
+                       if (is_migration_entry(entry)) {
+                               pte_unmap(&orig_src_pte);
+                               pte_unmap(&orig_dst_pte);
+                               src_pte = dst_pte = NULL;
+                               migration_entry_wait(src_mm, src_pmd,
+                                                    src_addr);
+                               err = -EAGAIN;
+                       } else
+                               err = -EFAULT;
+                       goto out;
+               }
+
+               /*
+                * COUNT_CONTINUE to be returned is fine here, no need
+                * of follow all swap continuation to check against
+                * number 1.
+                */
+               si = get_swap_device(entry);
+               if (!si) {
+                       err = -EBUSY;
+                       goto out;
+               }
+
+               swap_count = swap_swapcount(si, entry);
+               put_swap_device(si);
+               if (swap_count != 1) {
+                       err = -EBUSY;
+                       goto out;
+               }
+
+               double_pt_lock(dst_ptl, src_ptl);
+
+               if (!pte_same(*src_pte, orig_src_pte) ||
+                   !pte_same(*dst_pte, orig_dst_pte) ||
+                   swp_swapcount(entry) != 1) {
+                       double_pt_unlock(dst_ptl, src_ptl);
+                       err = -EAGAIN;
+                       goto out;
+               }
+
+               if (pte_val(ptep_get_and_clear(src_mm, src_addr, src_pte)) !=
+                   pte_val(orig_src_pte))
+                       BUG_ON(1);
+               set_pte_at(dst_mm, dst_addr, dst_pte, orig_src_pte);
+
+               if (dst_mm != src_mm) {
+                       inc_mm_counter(dst_mm, MM_ANONPAGES);
+                       dec_mm_counter(src_mm, MM_ANONPAGES);
+               }
+
+               double_pt_unlock(dst_ptl, src_ptl);
+       }
+
+out:
+       if (src_anon_vma) {
+               anon_vma_unlock_write(src_anon_vma);
+               put_anon_vma(src_anon_vma);
+       }
+       if (src_folio) {
+               folio_unlock(src_folio);
+               folio_put(src_folio);
+       }
+       if (dst_pte)
+               pte_unmap(dst_pte);
+       if (src_pte)
+               pte_unmap(src_pte);
+
+       return err;
+}
author reviewer
#26 lore raw

Re: [PATCH 2/3] userfaultfd: UFFDIO_REMAP uABI

Quoted 162 lines · Jann Horn
Quoted 124 lines · Suren Baghdasaryan
mm/userfaultfd.c +124 -0
+/*
+ * The mmap_lock for reading is held by the caller. Just move the page
+ * from src_pmd to dst_pmd if possible, and return true if succeeded
+ * in moving the page.
+ */
+static int remap_pages_pte(struct mm_struct *dst_mm,
+                          struct mm_struct *src_mm,
+                          pmd_t *dst_pmd,
+                          pmd_t *src_pmd,
+                          struct vm_area_struct *dst_vma,
+                          struct vm_area_struct *src_vma,
+                          unsigned long dst_addr,
+                          unsigned long src_addr,
+                          __u64 mode)
+{
+       swp_entry_t entry;
+       pte_t orig_src_pte, orig_dst_pte;
+       spinlock_t *src_ptl, *dst_ptl;
+       pte_t *src_pte = NULL;
+       pte_t *dst_pte = NULL;
+
+       struct folio *src_folio = NULL;
+       struct anon_vma *src_anon_vma = NULL;
+       struct anon_vma *dst_anon_vma;
+       struct mmu_notifier_range range;
+       int err = 0;
+
+retry:
+       dst_pte = pte_offset_map_nolock(dst_mm, dst_pmd, dst_addr, &dst_ptl);
+
+       /* If an huge pmd materialized from under us fail */
+       if (unlikely(!dst_pte)) {
+               err = -EFAULT;
+               goto out;
+       }
+
+       src_pte = pte_offset_map_nolock(src_mm, src_pmd, src_addr, &src_ptl);
+
+       /*
+        * We held the mmap_lock for reading so MADV_DONTNEED
+        * can zap transparent huge pages under us, or the
+        * transparent huge page fault can establish new
+        * transparent huge pages under us.
+        */
+       if (unlikely(!src_pte)) {
+               err = -EFAULT;
+               goto out;
+       }
+
+       BUG_ON(pmd_none(*dst_pmd));
+       BUG_ON(pmd_none(*src_pmd));
+       BUG_ON(pmd_trans_huge(*dst_pmd));
+       BUG_ON(pmd_trans_huge(*src_pmd));
+
+       spin_lock(dst_ptl);
+       orig_dst_pte = *dst_pte;
+       spin_unlock(dst_ptl);
+       if (!pte_none(orig_dst_pte)) {
+               err = -EEXIST;
+               goto out;
+       }
+
+       spin_lock(src_ptl);
+       orig_src_pte = *src_pte;
+       spin_unlock(src_ptl);
+       if (pte_none(orig_src_pte)) {
+               if (!(mode & UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES))
+                       err = -ENOENT;
+               else /* nothing to do to remap a hole */
+                       err = 0;
+               goto out;
+       }
+
+       if (pte_present(orig_src_pte)) {
+               if (!src_folio) {
+                       struct folio *folio;
+
+                       /*
+                        * Pin the page while holding the lock to be sure the
+                        * page isn't freed under us
+                        */
+                       spin_lock(src_ptl);
+                       if (!pte_same(orig_src_pte, *src_pte)) {
+                               spin_unlock(src_ptl);
+                               err = -EAGAIN;
+                               goto out;
+                       }
+
+                       folio = vm_normal_folio(src_vma, src_addr, orig_src_pte);
+                       if (!folio || !folio_test_anon(folio) ||
+                           folio_estimated_sharers(folio) != 1) {
+                               spin_unlock(src_ptl);
+                               err = -EBUSY;
+                               goto out;
+                       }
+
+                       src_folio = folio;
+                       folio_get(src_folio);
+                       spin_unlock(src_ptl);
+
+                       /* try to block all concurrent rmap walks */
+                       if (!folio_trylock(src_folio)) {
+                               pte_unmap(&orig_src_pte);
+                               pte_unmap(&orig_dst_pte);
+                               src_pte = dst_pte = NULL;
+                               folio_lock(src_folio);
+                               goto retry;
+                       }
+               }
+
+               if (!src_anon_vma) {
+                       /*
+                        * folio_referenced walks the anon_vma chain
+                        * without the folio lock. Serialize against it with
+                        * the anon_vma lock, the folio lock is not enough.
+                        */
+                       src_anon_vma = folio_get_anon_vma(src_folio);
+                       if (!src_anon_vma) {
+                               /* page was unmapped from under us */
+                               err = -EAGAIN;
+                               goto out;
+                       }
+                       if (!anon_vma_trylock_write(src_anon_vma)) {
+                               pte_unmap(&orig_src_pte);

mm/userfaultfd.c int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, · line 1013

Quoted 10 lines
mm/userfaultfd.c +10 -0
+                               pte_unmap(&orig_dst_pte);
+                               src_pte = dst_pte = NULL;
+                               anon_vma_lock_write(src_anon_vma);
+                               goto retry;
+                       }
+               }
+
+               mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm,
+                                       src_addr, src_addr + PAGE_SIZE);
+               mmu_notifier_invalidate_range_start_nonblock(&range);

I'm pretty sure you're not allowed to use mmu_notifier_invalidate_range_start_nonblock(). Use mmu_notifier_invalidate_range_start() at a point where it's fine to block; perhaps all the way up in remap_pages() around the whole loop or something like that. mmu_notifier_invalidate_range_start_nonblock() is special magic sauce for OOM reaping (with deceptively inviting naming and no documentation that explains that this is evil hazmat code), and if you chase down what various users of MMU notifiers do when you just hand them a random notification where mmu_notifier_range_blockable() is false, you end up in fun paths like in KVM's kvm_mmu_notifier_invalidate_range_start(), which calls gfn_to_pfn_cache_invalidate_start(), which does this:

    /*
     * If the OOM reaper is active, then all vCPUs should have
     * been stopped already, so perform the request without
     * KVM_REQUEST_WAIT and be sad if any needed to be IPI'd.
     */
    if (!may_block)
       req &= ~KVM_REQUEST_WAIT;

    called = kvm_make_vcpus_request_mask(kvm, req, vcpu_bitmap);

    WARN_ON_ONCE(called && !may_block);
Quoted 8 lines

Which means if you hit this codepath from a place like userfaultfd where the process is probably still running (and not being OOM reaped in a context where it's guaranteed to have been killed and stopped), you could probably get KVM into a situation where it needs to wait for vCPUs to acknowledge that they've processed a message before it's safe to continue, but it can't wait because we're in nonsleepable context, and then it just continues without waiting and presumably the KVM TLB gets out of sync or something?

Yeah, I was sceptical about this function too but it was very tempting :) Thanks for the warning, I'll move mmu_notifier_invalidate_range_start() to before we map the ptes because that's where we start the RCU read section.

Quoted 125 lines
Quoted 125 lines
mm/userfaultfd.c +122 -0
+
+               double_pt_lock(dst_ptl, src_ptl);
+
+               if (!pte_same(*src_pte, orig_src_pte) ||
+                   !pte_same(*dst_pte, orig_dst_pte) ||
+                   folio_estimated_sharers(src_folio) != 1) {
+                       double_pt_unlock(dst_ptl, src_ptl);
+                       err = -EAGAIN;
+                       goto out;
+               }
+
+               BUG_ON(!folio_test_anon(src_folio));
+               /* the PT lock is enough to keep the page pinned now */
+               folio_put(src_folio);
+
+               dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
+               WRITE_ONCE(src_folio->mapping,
+                          (struct address_space *) dst_anon_vma);
+               WRITE_ONCE(src_folio->index, linear_page_index(dst_vma,
+                                                             dst_addr));
+
+               if (!pte_same(ptep_clear_flush(src_vma, src_addr, src_pte),
+                             orig_src_pte))
+                       BUG_ON(1);
+
+               orig_dst_pte = mk_pte(&src_folio->page, dst_vma->vm_page_prot);
+               orig_dst_pte = maybe_mkwrite(pte_mkdirty(orig_dst_pte),
+                                            dst_vma);
+
+               set_pte_at(dst_mm, dst_addr, dst_pte, orig_dst_pte);
+
+               if (dst_mm != src_mm) {
+                       inc_mm_counter(dst_mm, MM_ANONPAGES);
+                       dec_mm_counter(src_mm, MM_ANONPAGES);
+               }
+
+               double_pt_unlock(dst_ptl, src_ptl);
+
+               anon_vma_unlock_write(src_anon_vma);
+               mmu_notifier_invalidate_range_end(&range);
+               put_anon_vma(src_anon_vma);
+               src_anon_vma = NULL;
+
+               /* unblock rmap walks */
+               folio_unlock(src_folio);
+               src_folio = NULL;
+
+       } else {
+               struct swap_info_struct *si;
+               int swap_count;
+
+               entry = pte_to_swp_entry(orig_src_pte);
+               if (non_swap_entry(entry)) {
+                       if (is_migration_entry(entry)) {
+                               pte_unmap(&orig_src_pte);
+                               pte_unmap(&orig_dst_pte);
+                               src_pte = dst_pte = NULL;
+                               migration_entry_wait(src_mm, src_pmd,
+                                                    src_addr);
+                               err = -EAGAIN;
+                       } else
+                               err = -EFAULT;
+                       goto out;
+               }
+
+               /*
+                * COUNT_CONTINUE to be returned is fine here, no need
+                * of follow all swap continuation to check against
+                * number 1.
+                */
+               si = get_swap_device(entry);
+               if (!si) {
+                       err = -EBUSY;
+                       goto out;
+               }
+
+               swap_count = swap_swapcount(si, entry);
+               put_swap_device(si);
+               if (swap_count != 1) {
+                       err = -EBUSY;
+                       goto out;
+               }
+
+               double_pt_lock(dst_ptl, src_ptl);
+
+               if (!pte_same(*src_pte, orig_src_pte) ||
+                   !pte_same(*dst_pte, orig_dst_pte) ||
+                   swp_swapcount(entry) != 1) {
+                       double_pt_unlock(dst_ptl, src_ptl);
+                       err = -EAGAIN;
+                       goto out;
+               }
+
+               if (pte_val(ptep_get_and_clear(src_mm, src_addr, src_pte)) !=
+                   pte_val(orig_src_pte))
+                       BUG_ON(1);
+               set_pte_at(dst_mm, dst_addr, dst_pte, orig_src_pte);
+
+               if (dst_mm != src_mm) {
+                       inc_mm_counter(dst_mm, MM_ANONPAGES);
+                       dec_mm_counter(src_mm, MM_ANONPAGES);
+               }
+
+               double_pt_unlock(dst_ptl, src_ptl);
+       }
+
+out:
+       if (src_anon_vma) {
+               anon_vma_unlock_write(src_anon_vma);
+               put_anon_vma(src_anon_vma);
+       }
+       if (src_folio) {
+               folio_unlock(src_folio);
+               folio_put(src_folio);
+       }
+       if (dst_pte)
+               pte_unmap(dst_pte);
+       if (src_pte)
+               pte_unmap(src_pte);
+
+       return err;
+}
author reviewer
#27 lore raw

PATCH 3/3 selftests/mm: add UFFDIO_REMAP ioctl test

In reply to Suren Baghdasaryan

Add a test for new UFFDIO_REMAP ioctl which uses uffd to remaps source into destination buffer while checking the contents of both after remapping. After the operation the content of the destination buffer should match the original source buffer's content while the source buffer should be zeroed.

---
 tools/testing/selftests/mm/uffd-common.c     | 34 ++++++++++-
 tools/testing/selftests/mm/uffd-common.h     |  1 +
 tools/testing/selftests/mm/uffd-unit-tests.c | 62 ++++++++++++++++++++
 3 files changed, 95 insertions(+), 2 deletions(-)
tools/testing/selftests/mm/uffd-common.c +32 -2
index 02b89860e193..fb449abc46e2 100644
@@ -484,8 +484,14 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args)
 		offset = (char *)(unsigned long)msg->arg.pagefault.address - area_dst;
 		offset &= ~(page_size-1);
 
-		if (copy_page(uffd, offset, args->apply_wp))
-			args->missing_faults++;
+		/* UFFD_REMAP is supported for anon non-shared mappings. */
+		if (uffd_test_ops == &anon_uffd_test_ops && !map_shared) {
+			if (remap_page(uffd, offset))
+				args->missing_faults++;
+		} else {
+			if (copy_page(uffd, offset, args->apply_wp))
+				args->missing_faults++;
+		}
 	}
 }
 
@@ -620,6 +626,30 @@ int copy_page(int ufd, unsigned long offset, bool wp)
 	return __copy_page(ufd, offset, false, wp);
 }
 
+int remap_page(int ufd, unsigned long offset)
+{
+	struct uffdio_remap uffdio_remap;
+
+	if (offset >= nr_pages * page_size)
+		err("unexpected offset %lu\n", offset);
+	uffdio_remap.dst = (unsigned long) area_dst + offset;
+	uffdio_remap.src = (unsigned long) area_src + offset;
+	uffdio_remap.len = page_size;
+	uffdio_remap.mode = UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES;
+	uffdio_remap.remap = 0;
+	if (ioctl(ufd, UFFDIO_REMAP, &uffdio_remap)) {
+		/* real retval in uffdio_remap.remap */
+		if (uffdio_remap.remap != -EEXIST)
+			err("UFFDIO_REMAP error: %"PRId64,
+			    (int64_t)uffdio_remap.remap);
+		wake_range(ufd, uffdio_remap.dst, page_size);
+	} else if (uffdio_remap.remap != page_size) {
+		err("UFFDIO_REMAP error: %"PRId64, (int64_t)uffdio_remap.remap);
+	} else
+		return 1;
+	return 0;
+}
+
 int uffd_open_dev(unsigned int flags)
 {
 	int fd, uffd;
tools/testing/selftests/mm/uffd-common.h +1 -0
index 7c4fa964c3b0..2bbb15d1920c 100644
@@ -111,6 +111,7 @@ void wp_range(int ufd, __u64 start, __u64 len, bool wp);
 void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args);
 int __copy_page(int ufd, unsigned long offset, bool retry, bool wp);
 int copy_page(int ufd, unsigned long offset, bool wp);
+int remap_page(int ufd, unsigned long offset);
 void *uffd_poll_thread(void *arg);
 
 int uffd_open_dev(unsigned int flags);
tools/testing/selftests/mm/uffd-unit-tests.c +62 -0

80 lines not on this page · raw message

Signature · 1 line
2.42.0.283.g2d96d420d3-goog