Skip to contentSkip to conversation

RFC: userfaultfd remap

PATCH In review Blake Caldwell 2 participants 7 messages Jan 12, 2019 – Jan 24, 2019

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover RFC: userfaultfd remap

Hello,

Since userfaultfd remap functionality was first proposed by Andrea Arcangeli [1], a new use case has been demonstrated for removing pages from the userfaultfd registered region. FluidMem [2] is a system for expanding or limiting the resident size of a VM using a remote key-value store as backing storage instead of swap space. It runs on the hypervisor and uses userfaultfd to manage the memory regions malloc'd by qemu. Since FluidMem maintains a constant resident size using an LRU list, it must evict pages to the remote key-value store to make room for pages that were just faulted in. This requires UFFDIO_REMAP to remove pages from the uncooperative userspace page fault handler.

The VM shadow page tables must be kept in sync after a remapping, so mmu_notifier_invalidate_range_(start/end) calls are made as necessary.

FluiMem enables page fault latencies to a remote key-value store that are as fast as swap backed by DRAM (/dev/pmem0) and 77% faster than swap with a SSD drive. pmbench [3] was used to measure page fault latencies with a 4 GB working set size, within a VM using 1 GB DRAM (20% local):

FluidMem (RAMCloud): 24.87 microseconds
Swap (pmem DRAM): 26.34 microseconds
Swap (NVMe over Fabrics): 41.73 microseconds
Swap (SSD): 106.56 microseconds

For real applications FluidMem has an additional benefit of allowing unused kernel pages to be removed from DRAM and stored in backend storage, making room for additional application pages to be kept in local DRAM. The useful memory capacity for the VM is increased.

The main complexity of this code is found in rmap, where it overwrites the page->index when it moves the page to a different vma with different vma->vm_pgoff. Overwriting page->index requires the rmap change and it's only possible when the page_mapcount is 1.

Changes since [1]:
- Changed the direction supported by UFFDIO_REMAP to the OUT direction needed by FluidMem. The IN direction is not necessary, as UFFDIO_COPY should be used instead because it doesn't require a TLB flush.
- Code has been kept up-to-date by Andrea in branch userfault from [4].

[1] https://lkml.org/lkml/2015/3/5/576
[2] Caldwell, Blake, Youngbin Im, Sangtae Ha, Richard Han, and Eric Keller. "FluidMem: Memory as a Service for the Datacenter." arXiv preprint arXiv:1707.07780 (2017).
https://github.com/blakecaldwell/fluidmem
[3] Yang, Jisoo, and Julian Seymour. "Pmbench: A Micro-Benchmark for Profiling Paging Performance on a System with Low-Latency SSDs." Information Technology-New Generations. Springer, Cham, 2018. 627-633.
https://bitbucket.org/jisooy/pmbench/src
[4] https://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git

Andrea Arcangeli (3):
userfaultfd: UFFDIO_REMAP: rmap preparation
userfaultfd: UFFDIO_REMAP uABI
userfaultfd: UFFDIO_REMAP

Blake Caldwell (1):
userfaultfd: change the direction for UFFDIO_REMAP to out

 Documentation/admin-guide/mm/userfaultfd.rst |  10 +
 fs/userfaultfd.c                             |  49 +++
 include/linux/userfaultfd_k.h                |  17 +
 include/uapi/linux/userfaultfd.h             |  25 +-
 mm/huge_memory.c                             | 117 ++++++
 mm/khugepaged.c                              |   3 +
 mm/rmap.c                                    |  13 +
 mm/userfaultfd.c                             | 536 +++++++++++++++++++++++++++
 8 files changed, 769 insertions(+), 1 deletion(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/4 userfaultfd: UFFDIO_REMAP: rmap preparation

From: Andrea Arcangeli <[email protected]>

As far as the rmap code is concerned, UFFDIO_REMAP only alters the page->mapping and page->index. It does it while holding the page lock. However page_referenced() is doing rmap walks without taking the page lock first, so page_lock_anon_vma_read must be updated to re-check that the page->mapping didn't change after we obtained the anon_vma read lock.

UFFDIO_REMAP takes the anon_vma lock for writing before altering the page->mapping, so if the page->mapping is still the same after obtaining the anon_vma read lock (without the page lock), the rmap walks can go ahead safely (and UFFDIO_REMAP will wait the rmap walk to complete before proceeding).

UFFDIO_REMAP serializes against itself with the page lock.

All other places taking the anon_vma lock while holding the mmap_sem for writing, don't need to check if the page->mapping has changed after taking the anon_vma lock, regardless of the page lock, because UFFDIO_REMAP holds the mmap_sem for reading.

There's one constraint enforced to allow this simplification: the source pages passed to UFFDIO_REMAP must be mapped only in one vma, but this constraint is an acceptable tradeoff for UFFDIO_REMAP users.

The source addresses passed to UFFDIO_REMAP should be set as VM_DONTCOPY with MADV_DONTFORK to avoid any risk of the mapcount of the pages increasing if some thread of the process forks() before UFFDIO_REMAP run.

---
 mm/rmap.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
mm/rmap.c +13 -0
index 0454ecc2..d8f228d 100644
@@ -510,6 +510,7 @@ struct anon_vma *page_lock_anon_vma_read(struct page *page)
 	struct anon_vma *root_anon_vma;
 	unsigned long anon_mapping;
 
+repeat:
 	rcu_read_lock();
 	anon_mapping = (unsigned long)READ_ONCE(page->mapping);
 	if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON)
@@ -548,6 +549,18 @@ struct anon_vma *page_lock_anon_vma_read(struct page *page)
 	rcu_read_unlock();
 	anon_vma_lock_read(anon_vma);
 
+	/*
+	 * Check if UFFDIO_REMAP changed the anon_vma. This is needed
+	 * because we don't assume the page was locked.
+	 */
+	if (unlikely((unsigned long) READ_ONCE(page->mapping) !=
+		     anon_mapping)) {
+		anon_vma_unlock_read(anon_vma);
+		put_anon_vma(anon_vma);
+		anon_vma = NULL;
+		goto repeat;
+	}
+
 	if (atomic_dec_and_test(&anon_vma->refcount)) {
 		/*
 		 * Oops, we held the last refcount, release the lock
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH 2/4 userfaultfd: UFFDIO_REMAP uABI

In reply to Blake Caldwell

From: Andrea Arcangeli <[email protected]>

This implements the uABI of UFFDIO_REMAP.

Notably one mode bitflag is also forwarded (and in turn known) by the lowlevel remap_pages method.

---
 include/uapi/linux/userfaultfd.h | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
include/uapi/linux/userfaultfd.h +24 -1
index 48f1a7c..a0d6106 100644
@@ -34,7 +34,8 @@
 #define UFFD_API_RANGE_IOCTLS			\
 	((__u64)1 << _UFFDIO_WAKE |		\
 	 (__u64)1 << _UFFDIO_COPY |		\
-	 (__u64)1 << _UFFDIO_ZEROPAGE)
+	 (__u64)1 << _UFFDIO_ZEROPAGE |		\
+	 (__u64)1 << _UFFDIO_REMAP)
 #define UFFD_API_RANGE_IOCTLS_BASIC		\
 	((__u64)1 << _UFFDIO_WAKE |		\
 	 (__u64)1 << _UFFDIO_COPY)
@@ -52,6 +53,7 @@
 #define _UFFDIO_WAKE			(0x02)
 #define _UFFDIO_COPY			(0x03)
 #define _UFFDIO_ZEROPAGE		(0x04)
+#define _UFFDIO_REMAP			(0x05)
 #define _UFFDIO_API			(0x3F)
 
 /* userfaultfd ioctl ids */
@@ -68,6 +70,8 @@
 				      struct uffdio_copy)
 #define UFFDIO_ZEROPAGE		_IOWR(UFFDIO, _UFFDIO_ZEROPAGE,	\
 				      struct uffdio_zeropage)
+#define UFFDIO_REMAP		_IOWR(UFFDIO, _UFFDIO_REMAP,	\
+				      struct uffdio_remap)
 
 /* read() structure */
 struct uffd_msg {
@@ -231,4 +235,23 @@ struct uffdio_zeropage {
 	__s64 zeropage;
 };
 
+struct uffdio_remap {
+	__u64 dst;
+	__u64 src;
+	__u64 len;
+	/*
+	 * Especially if used to atomically remove memory from the
+	 * address space the wake on the dst range is not needed.
+	 */
+#define UFFDIO_REMAP_MODE_DONTWAKE		((__u64)1<<0)
+#define UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES	((__u64)1<<1)
+	__u64 mode;
+
+	/*
+	 * "remap" is written by the ioctl and must be at the end: the
+	 * copy_from_user will not read the last 8 bytes.
+	 */
+	__s64 remap;
+};
+
 #endif /* _LINUX_USERFAULTFD_H */
Signature · 1 line
1.8.3.1
#3 lore raw

PATCH 3/4 userfaultfd: UFFDIO_REMAP

In reply to Blake Caldwell

From: Andrea Arcangeli <[email protected]>

This remap ioctl allows to atomically move a page in or out of an userfaultfd address space. It's more expensive than "copy" (and of course more expensive than "zerofill") as it requires a TLB flush on the source range for each ioctl, which is an expensive operation on SMP. Especially if copying only a few pages at time, copying without TLB flush is faster.

---
 fs/userfaultfd.c              |  49 ++++
 include/linux/userfaultfd_k.h |  17 ++
 mm/huge_memory.c              | 117 +++++++++
 mm/khugepaged.c               |   3 +
 mm/userfaultfd.c              | 536 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 722 insertions(+)
fs/userfaultfd.c +49 -0
index 89800fc..cf68cdb 100644
@@ -1792,6 +1792,52 @@ static inline unsigned int uffd_ctx_features(__u64 user_features)
 	return (unsigned int)user_features;
 }
 
+static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
+			     unsigned long arg)
+{
+	__s64 ret;
+	struct uffdio_remap uffdio_remap;
+	struct uffdio_remap __user *user_uffdio_remap;
+	struct userfaultfd_wake_range range;
+
+	user_uffdio_remap = (struct uffdio_remap __user *) arg;
+
+	ret = -EFAULT;
+	if (copy_from_user(&uffdio_remap, user_uffdio_remap,
+			   /* don't copy "remap" last field */
+			   sizeof(uffdio_remap)-sizeof(__s64)))
+		goto out;
+
+	ret = validate_range(ctx->mm, uffdio_remap.dst, uffdio_remap.len);
+	if (ret)
+		goto out;
+	ret = validate_range(current->mm, uffdio_remap.src, uffdio_remap.len);
+	if (ret)
+		goto out;
+	ret = -EINVAL;
+	if (uffdio_remap.mode & ~(UFFDIO_REMAP_MODE_ALLOW_SRC_HOLES|
+				  UFFDIO_REMAP_MODE_DONTWAKE))
+		goto out;
+
+	ret = remap_pages(ctx->mm, current->mm,
+			  uffdio_remap.dst, uffdio_remap.src,
+			  uffdio_remap.len, uffdio_remap.mode);
+	if (unlikely(put_user(ret, &user_uffdio_remap->remap)))
+		return -EFAULT;
+	if (ret < 0)
+		goto out;
+	/* len == 0 would wake all */
+	BUG_ON(!ret);
+	range.len = ret;
+	if (!(uffdio_remap.mode & UFFDIO_REMAP_MODE_DONTWAKE)) {
+		range.start = uffdio_remap.dst;
+		wake_userfault(ctx, &range);
+	}
+	ret = range.len == uffdio_remap.len ? 0 : -EAGAIN;
+out:
+	return ret;
+}
+
 /*
  * userland asks for a certain API version and we return which bits
  * and ioctl commands are implemented in this kernel for such API
@@ -1861,6 +1907,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,
 	case UFFDIO_ZEROPAGE:
 		ret = userfaultfd_zeropage(ctx, arg);
 		break;
+	case UFFDIO_REMAP:
+		ret = userfaultfd_remap(ctx, arg);
+		break;
 	}
 	return ret;
 }
include/linux/userfaultfd_k.h +17 -0
index 37c9eba..56fb0e6 100644
@@ -38,6 +38,23 @@ extern ssize_t mfill_zeropage(struct mm_struct *dst_mm,
 			      unsigned long len,
 			      bool *mmap_changing);
 
+/* remap_pages */
+extern void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
+extern ssize_t remap_pages(struct mm_struct *dst_mm,
+			   struct mm_struct *src_mm,
+			   unsigned long dst_start,
+			   unsigned long src_start,
+			   unsigned long len, __u64 flags);
+extern int remap_pages_huge_pmd(struct mm_struct *dst_mm,
+				struct mm_struct *src_mm,
+				pmd_t *dst_pmd, pmd_t *src_pmd,
+				pmd_t dst_pmdval,
+				struct vm_area_struct *dst_vma,
+				struct vm_area_struct *src_vma,
+				unsigned long dst_addr,
+				unsigned long src_addr);
+
 /* mm helpers */
 static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma,
 					struct vm_userfaultfd_ctx vm_ctx)
mm/huge_memory.c +117 -0

123 lines not on this page · raw message

mm/khugepaged.c +3 -0
index 4f01733..c91a748 100644
@@ -1002,6 +1002,9 @@ static void collapse_huge_page(struct mm_struct *mm,
 	 * Prevent all access to pagetables with the exception of
 	 * gup_fast later handled by the ptep_clear_flush and the VM
 	 * handled by the anon_vma lock + PG_lock.
+	 *
+	 * UFFDIO_REMAP is prevented to race as well thanks to the
+	 * mmap_sem.
 	 */
 	down_write(&mm->mmap_sem);
 	result = hugepage_vma_revalidate(mm, address, &vma);
mm/userfaultfd.c +536 -0

539 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#4 lore raw

PATCH 4/4 userfaultfd: change the direction for UFFDIO_REMAP to out

In reply to Blake Caldwell

Moving a page out of a userfaultfd registered region and into a userland anonymous vma is needed by the use case of uncooperatively limiting the resident size of the userfaultfd region. Reverse the direction of the original userfaultfd_remap() to the out direction. Now after memory has been removed, subsequent accesses will generate uffdio page fault events.

---
 Documentation/admin-guide/mm/userfaultfd.rst | 10 ++++++++++
 fs/userfaultfd.c                             |  6 +++---
 2 files changed, 13 insertions(+), 3 deletions(-)
Documentation/admin-guide/mm/userfaultfd.rst +10 -0
index 5048cf6..714af49 100644
@@ -108,6 +108,16 @@ UFFDIO_COPY. They're atomic as in guaranteeing that nothing can see an
 half copied page since it'll keep userfaulting until the copy has
 finished.
 
+To move pages out of a userfault registered region and into a user vma
+the UFFDIO_REMAP ioctl can be used. This is only possible for the
+"OUT" direction. For the "IN" direction, UFFDIO_COPY is preferred
+since UFFDIO_REMAP requires a TLB flush on the source range at a
+greater penalty than copying the page. With
+UFFDIO_REGISTER_MODE_MISSING set, subsequent accesses to the same
+region will generate a page fault event. This allows non-cooperative
+removal of memory in a userfaultfd registered vma, effectively
+limiting the amount of resident memory in such a region.
+
 QEMU/KVM
 ========
 
fs/userfaultfd.c +3 -3
index cf68cdb..8099da2 100644
@@ -1808,10 +1808,10 @@ static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
 			   sizeof(uffdio_remap)-sizeof(__s64)))
 		goto out;
 
-	ret = validate_range(ctx->mm, uffdio_remap.dst, uffdio_remap.len);
+	ret = validate_range(current->mm, uffdio_remap.dst, uffdio_remap.len);
 	if (ret)
 		goto out;
-	ret = validate_range(current->mm, uffdio_remap.src, uffdio_remap.len);
+	ret = validate_range(ctx->mm, uffdio_remap.src, uffdio_remap.len);
 	if (ret)
 		goto out;
 	ret = -EINVAL;
@@ -1819,7 +1819,7 @@ static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
 				  UFFDIO_REMAP_MODE_DONTWAKE))
 		goto out;
 
-	ret = remap_pages(ctx->mm, current->mm,
+	ret = remap_pages(current->mm, ctx->mm,
 			  uffdio_remap.dst, uffdio_remap.src,
 			  uffdio_remap.len, uffdio_remap.mode);
 	if (unlikely(put_user(ret, &user_uffdio_remap->remap)))
Signature · 1 line
1.8.3.1
#5 lore raw

Re: [PATCH 4/4] userfaultfd: change the direction for UFFDIO_REMAP to out

Hi,

Quoted 5 lines · Blake Caldwell

Moving a page out of a userfaultfd registered region and into a userland anonymous vma is needed by the use case of uncooperatively limiting the resident size of the userfaultfd region. Reverse the direction of the original userfaultfd_remap() to the out direction. Now after memory has been removed, subsequent accesses will generate uffdio page fault events.

It took me a while but better late then never :)

Why did you keep this as a separate patch? If the primary use case for UFFDIO_REMAP to move pages out of userfaultfd region, why not make it so from the beginning?

Quoted 56 lines
---
 Documentation/admin-guide/mm/userfaultfd.rst | 10 ++++++++++
 fs/userfaultfd.c                             |  6 +++---
 2 files changed, 13 insertions(+), 3 deletions(-)
Documentation/admin-guide/mm/userfaultfd.rst +10 -0
index 5048cf6..714af49 100644
@@ -108,6 +108,16 @@ UFFDIO_COPY. They're atomic as in guaranteeing that nothing can see an
 half copied page since it'll keep userfaulting until the copy has
 finished.
 
+To move pages out of a userfault registered region and into a user vma
+the UFFDIO_REMAP ioctl can be used. This is only possible for the
+"OUT" direction. For the "IN" direction, UFFDIO_COPY is preferred
+since UFFDIO_REMAP requires a TLB flush on the source range at a
+greater penalty than copying the page. With
+UFFDIO_REGISTER_MODE_MISSING set, subsequent accesses to the same
+region will generate a page fault event. This allows non-cooperative
+removal of memory in a userfaultfd registered vma, effectively
+limiting the amount of resident memory in such a region.
+
 QEMU/KVM
 ========
 
fs/userfaultfd.c +3 -3
index cf68cdb..8099da2 100644
@@ -1808,10 +1808,10 @@ static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
 			   sizeof(uffdio_remap)-sizeof(__s64)))
 		goto out;
 
-	ret = validate_range(ctx->mm, uffdio_remap.dst, uffdio_remap.len);
+	ret = validate_range(current->mm, uffdio_remap.dst, uffdio_remap.len);
 	if (ret)
 		goto out;
-	ret = validate_range(current->mm, uffdio_remap.src, uffdio_remap.len);
+	ret = validate_range(ctx->mm, uffdio_remap.src, uffdio_remap.len);
 	if (ret)
 		goto out;
 	ret = -EINVAL;
@@ -1819,7 +1819,7 @@ static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
 				  UFFDIO_REMAP_MODE_DONTWAKE))
 		goto out;
 
-	ret = remap_pages(ctx->mm, current->mm,
+	ret = remap_pages(current->mm, ctx->mm,
 			  uffdio_remap.dst, uffdio_remap.src,
 			  uffdio_remap.len, uffdio_remap.mode);
 	if (unlikely(put_user(ret, &user_uffdio_remap->remap)))
Signature · 1 line
1.8.3.1
Signature · 2 lines
Sincerely yours,
Mike.
#6 lore raw

Re: [PATCH 4/4] userfaultfd: change the direction for UFFDIO_REMAP to out

Quoted 12 lines · Mike Rapoport

Hi,

Quoted 5 lines · Blake Caldwell

Moving a page out of a userfaultfd registered region and into a userland anonymous vma is needed by the use case of uncooperatively limiting the resident size of the userfaultfd region. Reverse the direction of the original userfaultfd_remap() to the out direction. Now after memory has been removed, subsequent accesses will generate uffdio page fault events.

Quoted 5 lines

It took me a while but better late then never :)

Why did you keep this as a separate patch? If the primary use case for UFFDIO_REMAP to move pages out of userfaultfd region, why not make it so from the beginning?

Only to show what has changed since this was last proposed, but yes, that change to fs/userfaultfd.c should be squashed with patch 3. The purpose of patch 4 will only be documenting UFFDIO_REMAP.

I will make those changes for the next revision. Thanks for looking this over.

Quoted 61 lines
Quoted 57 lines

---
Documentation/admin-guide/mm/userfaultfd.rst | 10 ++++++++++
fs/userfaultfd.c | 6 +++---

2 files changed, 13 insertions(+), 3 deletions(-)
Documentation/admin-guide/mm/userfaultfd.rst +0 -0
index 5048cf6..714af49 100644
@@ -108,6 +108,16 @@ UFFDIO_COPY. They're atomic as in guaranteeing that nothing can see an

half copied page since it'll keep userfaulting until the copy has finished.

Documentation/admin-guide/mm/userfaultfd.rst +10 -0
+To move pages out of a userfault registered region and into a user vma
+the UFFDIO_REMAP ioctl can be used. This is only possible for the
+"OUT" direction. For the "IN" direction, UFFDIO_COPY is preferred
+since UFFDIO_REMAP requires a TLB flush on the source range at a
+greater penalty than copying the page. With
+UFFDIO_REGISTER_MODE_MISSING set, subsequent accesses to the same
+region will generate a page fault event. This allows non-cooperative
+removal of memory in a userfaultfd registered vma, effectively
+limiting the amount of resident memory in such a region.
+

QEMU/KVM
========

fs/userfaultfd.c +3 -3
index cf68cdb..8099da2 100644
@@ -1808,10 +1808,10 @@ static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
 			   sizeof(uffdio_remap)-sizeof(__s64)))
 		goto out;
 
-	ret = validate_range(ctx->mm, uffdio_remap.dst, uffdio_remap.len);
+	ret = validate_range(current->mm, uffdio_remap.dst, uffdio_remap.len);
 	if (ret)
 		goto out;
-	ret = validate_range(current->mm, uffdio_remap.src, uffdio_remap.len);
+	ret = validate_range(ctx->mm, uffdio_remap.src, uffdio_remap.len);
 	if (ret)
 		goto out;
 	ret = -EINVAL;
@@ -1819,7 +1819,7 @@ static int userfaultfd_remap(struct userfaultfd_ctx *ctx,
 				  UFFDIO_REMAP_MODE_DONTWAKE))
 		goto out;
 
-	ret = remap_pages(ctx->mm, current->mm,
+	ret = remap_pages(current->mm, ctx->mm,
 			  uffdio_remap.dst, uffdio_remap.src,
 			  uffdio_remap.len, uffdio_remap.mode);
 	if (unlikely(put_user(ret, &user_uffdio_remap->remap)))
Signature · 1 line
1.8.3.1
Signature · 2 lines
Sincerely yours,
Mike.