Skip to contentSkip to conversation

userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c

PATCH Superseded Mike Rapoport 2 participants 7 messages May 1 – May 11

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Mike Rapoport

author maintainer
#0 lore raw

v2 cover userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c

From: "Mike Rapoport (Microsoft)" <[email protected]>

Hi,

These patches merge fs/userfaultfd.c into mm/userfaultfd.c and make functions used only inside mm/userfaultfd.c static.

v2 changes:
* add a fix for pre-existing bug found by sashiko as a separate patch

v1: https://lore.kernel.org/all/[email protected]

Mike Rapoport (Microsoft) (3):
userfaultfd: ensure mremap_userfaultfd_fail() releases mmap_changing
userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c
userfaultfd: make functions that are not used outside uffd static

 MAINTAINERS                   |    1 -
 fs/Makefile                   |    1 -
 fs/userfaultfd.c              | 2231 --------------------------------
 include/linux/userfaultfd_k.h |   36 -
 mm/userfaultfd.c              | 2239 ++++++++++++++++++++++++++++++++-
 5 files changed, 2227 insertions(+), 2281 deletions(-)
 delete mode 100644 fs/userfaultfd.c

base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731

Signature · 1 line
2.53.0
author maintainer
#1 lore raw

PATCH v2 1/3 userfaultfd: ensure mremap_userfaultfd_fail() releases mmap_changing

From: "Mike Rapoport (Microsoft)" <[email protected]>

Sashiko says:

mremap_userfaultfd_prep() increments ctx->mmap_changing to stall concurrent operations, but mremap_userfaultfd_fail() does not decrement it before dropping the context reference.

If an mremap operation fails, ctx->mmap_changing remains elevated. This will causes subsequent userfaultfd operations like a UFFDIO_COPY to fail with -EAGAIN.

Decrement ctx->mmap_changing in mremap_userfaultfd_fail().

Fixes df2cc96e7701 ("userfaultfd: prevent non-cooperative events vs mcopy_atomic races")
---
 fs/userfaultfd.c | 1 +
 1 file changed, 1 insertion(+)
fs/userfaultfd.c +1 -0
index 4b53dc4a3266..ef963a58f1a1 100644
@@ -786,6 +786,7 @@ void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx)
 	if (!ctx)
 		return;
 
+	atomic_dec(&ctx->mmap_changing);
 	userfaultfd_ctx_put(ctx);
 }
 
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH v2 1/3] userfaultfd: ensure mremap_userfaultfd_fail() releases mmap_changing

Quoted 4 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Sashiko says:

Quoted 8 lines

mremap_userfaultfd_prep() increments ctx->mmap_changing to stall concurrent operations, but mremap_userfaultfd_fail() does not decrement it before dropping the context reference.

If an mremap operation fails, ctx->mmap_changing remains elevated. This will causes subsequent userfaultfd operations like a UFFDIO_COPY to fail with -EAGAIN.

Sounds like we should CC stable?

Quoted 14 lines

Decrement ctx->mmap_changing in mremap_userfaultfd_fail().

Fixes df2cc96e7701 ("userfaultfd: prevent non-cooperative events vs mcopy_atomic races")
---
 fs/userfaultfd.c | 1 +
 1 file changed, 1 insertion(+)
fs/userfaultfd.c +0 -0
index 4b53dc4a3266..ef963a58f1a1 100644

fs/userfaultfd.c void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx) · line 789

Quoted 4 lines
fs/userfaultfd.c +1 -0
@@ -786,6 +786,7 @@ void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx)
 	if (!ctx)
 		return;
 
+	atomic_dec(&ctx->mmap_changing);

I'll note that other users have a

	VM_WARN_ON_ONCE(atomic_read(&ctx->mmap_changing) < 0);

In there. Likely we should do the same?

Signature · 3 lines
Cheers,

David
author maintainer
#3 lore raw

Re: [PATCH v2 1/3] userfaultfd: ensure mremap_userfaultfd_fail() releases mmap_changing

Quoted 9 lines · David Hildenbrand (Arm)
Quoted 8 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Sashiko says:

mremap_userfaultfd_prep() increments ctx->mmap_changing to stall concurrent operations, but mremap_userfaultfd_fail() does not decrement it before dropping the context reference.

Quoted 6 lines
Quoted 4 lines

If an mremap operation fails, ctx->mmap_changing remains elevated. This will causes subsequent userfaultfd operations like a UFFDIO_COPY to fail with -EAGAIN.

Sounds like we should CC stable?

Yes.

Quoted 17 lines
Quoted 17 lines

Decrement ctx->mmap_changing in mremap_userfaultfd_fail().

Fixes df2cc96e7701 ("userfaultfd: prevent non-cooperative events vs mcopy_atomic races")
---
 fs/userfaultfd.c | 1 +
 1 file changed, 1 insertion(+)
fs/userfaultfd.c +0 -0
index 4b53dc4a3266..ef963a58f1a1 100644
@@ -786,6 +786,7 @@ void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx)
 	if (!ctx)
 		return;
 
Quoted 7 lines

fs/userfaultfd.c void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx) · line 789

Quoted 1 line
fs/userfaultfd.c +1 -0
+	atomic_dec(&ctx->mmap_changing);

I'll note that other users have a

	VM_WARN_ON_ONCE(atomic_read(&ctx->mmap_changing) < 0);

In there. Likely we should do the same?

Yeah, we could.

Quoted 4 lines
Signature · 3 lines
Cheers,

David
Signature · 2 lines
Sincerely yours,
Mike.
#4 lore raw

Re: [PATCH v2 1/3] userfaultfd: ensure mremap_userfaultfd_fail() releases mmap_changing

Quoted 40 lines · Mike Rapoport
Quoted 9 lines · David Hildenbrand (Arm)
Quoted 8 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Sashiko says:

mremap_userfaultfd_prep() increments ctx->mmap_changing to stall concurrent operations, but mremap_userfaultfd_fail() does not decrement it before dropping the context reference.

Quoted 6 lines
Quoted 4 lines

If an mremap operation fails, ctx->mmap_changing remains elevated. This will causes subsequent userfaultfd operations like a UFFDIO_COPY to fail with -EAGAIN.

Sounds like we should CC stable?

Yes.

Quoted 21 lines
Quoted 14 lines

Decrement ctx->mmap_changing in mremap_userfaultfd_fail().

Fixes df2cc96e7701 ("userfaultfd: prevent non-cooperative events vs mcopy_atomic races")
---
 fs/userfaultfd.c | 1 +
 1 file changed, 1 insertion(+)
fs/userfaultfd.c +0 -0
index 4b53dc4a3266..ef963a58f1a1 100644

fs/userfaultfd.c void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx) · line 789

Quoted 4 lines
fs/userfaultfd.c +1 -0
@@ -786,6 +786,7 @@ void mremap_userfaultfd_fail(struct vm_userfaultfd_ctx *vm_ctx)
 	if (!ctx)
 		return;
 
+	atomic_dec(&ctx->mmap_changing);

I'll note that other users have a

Quoted 5 lines
Quoted 3 lines
	VM_WARN_ON_ONCE(atomic_read(&ctx->mmap_changing) < 0);

In there. Likely we should do the same?

Yeah, we could.

In any case

Signature · 3 lines
Cheers,

David
author maintainer
#5 lore raw

PATCH v2 2/3 userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Historically userfaultfd implementation has been split between fs/userfaultfd.c and mm/userfaultfd.c.

The mm/ part implemented memory management operations, while the fs/ part implemented file descriptor handling and called into the mm/ part for the actual memory management work.

This separation is quite artificial and fs/userfaultfd.c does not seem to belong to fs/ because it's only a user if vfs APIs and like for other users, for example, memfd and secretmem, the file descriptor handling could live in mm/ as well.

"Append" fs/userfaultfd.c to mm/userfaultfd and update fs/Makefile and MAINTAINERS accordingly.

No intended functional changes.

Assisted-by Copilot:claude-opus-4-6
---
 MAINTAINERS      |    1 -
 fs/Makefile      |    1 -
 fs/userfaultfd.c | 2232 ----------------------------------------------
 mm/userfaultfd.c | 2215 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 2215 insertions(+), 2234 deletions(-)
 delete mode 100644 fs/userfaultfd.c
MAINTAINERS +0 -1
index 2fb1c75afd16..afd3343a7336 100644
@@ -17071,7 +17071,6 @@ R:	Peter Xu <[email protected]>
 L:	[email protected]
 S:	Maintained
 F:	Documentation/admin-guide/mm/userfaultfd.rst
-F:	fs/userfaultfd.c
 F:	include/asm-generic/pgtable_uffd.h
 F:	include/linux/userfaultfd_k.h
 F:	include/uapi/linux/userfaultfd.h
fs/Makefile +0 -1
index ae1b07f9c6a0..89a8a9d207d1 100644
@@ -27,7 +27,6 @@ obj-y				+= anon_inodes.o
 obj-$(CONFIG_SIGNALFD)		+= signalfd.o
 obj-$(CONFIG_TIMERFD)		+= timerfd.o
 obj-$(CONFIG_EVENTFD)		+= eventfd.o
-obj-$(CONFIG_USERFAULTFD)	+= userfaultfd.o
 obj-$(CONFIG_AIO)               += aio.o
 obj-$(CONFIG_FS_DAX)		+= dax.o
 obj-$(CONFIG_FS_ENCRYPTION)	+= crypto/
fs/userfaultfd.c deleted +0 -2232

2,232 lines not on this page · raw message

mm/userfaultfd.c +2215 -0

754 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.53.0

Part of the diff is not shown here. View the raw message.

author maintainer
#6 lore raw

PATCH v2 3/3 userfaultfd: make functions that are not used outside uffd static

In reply to Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

After merging fs/userfaultfd.c into mm/userfaultfd.c, several functions that were previously shared between the two files are now only used within mm/userfaultfd.c.

Make them static and remove their declarations from include/linux/userfaultfd_k.h.

Assisted-by Copilot:claude-opus-4-6
---
 include/linux/userfaultfd_k.h | 36 -----------------------------------
 mm/userfaultfd.c              | 24 +++++++++++------------
 2 files changed, 12 insertions(+), 48 deletions(-)
include/linux/userfaultfd_k.h +0 -36
index d2920f98ab86..3ec8e1071673 100644
@@ -147,26 +147,12 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at
 /* Flags controlling behavior. These behavior changes are mode-independent. */
 #define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)
 
-extern ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
-				 unsigned long src_start, unsigned long len,
-				 uffd_flags_t flags);
-extern ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
-				     unsigned long dst_start,
-				     unsigned long len);
-extern ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long dst_start,
-				     unsigned long len, uffd_flags_t flags);
-extern ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
-				   unsigned long len, uffd_flags_t flags);
-extern int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
-			       unsigned long len, bool enable_wp);
 extern long uffd_wp_range(struct vm_area_struct *vma,
 			  unsigned long start, unsigned long len, bool enable_wp);
 
 /* move_pages */
 void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
 void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
-ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
-		   unsigned long src_start, unsigned long len, __u64 flags);
 int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval,
 			struct vm_area_struct *dst_vma,
 			struct vm_area_struct *src_vma,
@@ -239,9 +225,6 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
 	return vma->vm_flags & __VM_UFFD_FLAGS;
 }
 
-bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
-		       bool wp_async);
-
 static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
 {
 	struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx;
@@ -271,25 +254,6 @@ extern void userfaultfd_unmap_complete(struct mm_struct *mm,
 extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
 extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
 
-void userfaultfd_reset_ctx(struct vm_area_struct *vma);
-
-struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
-					     struct vm_area_struct *prev,
-					     struct vm_area_struct *vma,
-					     unsigned long start,
-					     unsigned long end);
-
-int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
-			       struct vm_area_struct *vma,
-			       vm_flags_t vm_flags,
-			       unsigned long start, unsigned long end,
-			       bool wp_async);
-
-void userfaultfd_release_new(struct userfaultfd_ctx *ctx);
-
-void userfaultfd_release_all(struct mm_struct *mm,
-			     struct userfaultfd_ctx *ctx);
-
 static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
 {
 	/* Only wr-protect mode uses pte markers */
mm/userfaultfd.c +12 -12
index 720690f058d7..a6ebf89ff657 100644
@@ -958,7 +958,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx,
 	return copied ? copied : err;
 }
 
-ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
+static ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 			  unsigned long src_start, unsigned long len,
 			  uffd_flags_t flags)
 {
@@ -966,7 +966,7 @@ ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 			    uffd_flags_set_mode(flags, MFILL_ATOMIC_COPY));
 }
 
-ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
+static ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
 			      unsigned long start,
 			      unsigned long len)
 {
@@ -974,7 +974,7 @@ ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
 			    uffd_flags_set_mode(0, MFILL_ATOMIC_ZEROPAGE));
 }
 
-ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start,
+static ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start,
 			      unsigned long len, uffd_flags_t flags)
 {
 
@@ -990,7 +990,7 @@ ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start,
 			    uffd_flags_set_mode(flags, MFILL_ATOMIC_CONTINUE));
 }
 
-ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
+static ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
 			    unsigned long len, uffd_flags_t flags)
 {
 	return mfill_atomic(ctx, start, 0, len,
@@ -1026,7 +1026,7 @@ long uffd_wp_range(struct vm_area_struct *dst_vma,
 	return ret;
 }
 
-int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
+static int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
 			unsigned long len, bool enable_wp)
 {
 	struct mm_struct *dst_mm = ctx->mm;
@@ -1856,7 +1856,7 @@ static void uffd_move_unlock(struct vm_area_struct *dst_vma,
  * in the regions or not, but preventing the risk of having to split
  * the hugepmd during the remap.
  */
-ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
+static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 		   unsigned long src_start, unsigned long len, __u64 mode)
 {
 	struct mm_struct *mm = ctx->mm;
@@ -2031,7 +2031,7 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 	return moved ? moved : err;
 }
 
-bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
+static bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
 		       bool wp_async)
 {
 	const struct vm_uffd_ops *ops = vma_uffd_ops(vma);
@@ -2088,12 +2088,12 @@ static void userfaultfd_set_ctx(struct vm_area_struct *vma,
 				 (vma->vm_flags & ~__VM_UFFD_FLAGS) | vm_flags);
 }
 
-void userfaultfd_reset_ctx(struct vm_area_struct *vma)
+static void userfaultfd_reset_ctx(struct vm_area_struct *vma)
 {
 	userfaultfd_set_ctx(vma, NULL, 0);
 }
 
-struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
+static struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
 					     struct vm_area_struct *prev,
 					     struct vm_area_struct *vma,
 					     unsigned long start,
@@ -2132,7 +2132,7 @@ struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
 }
 
 /* Assumes mmap write lock taken, and mm_struct pinned. */
-int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
+static int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
 			       struct vm_area_struct *vma,
 			       vm_flags_t vm_flags,
 			       unsigned long start, unsigned long end,
@@ -2196,7 +2196,7 @@ int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
 	return 0;
 }
 
-void userfaultfd_release_new(struct userfaultfd_ctx *ctx)
+static void userfaultfd_release_new(struct userfaultfd_ctx *ctx)
 {
 	struct mm_struct *mm = ctx->mm;
 	struct vm_area_struct *vma;
@@ -2211,7 +2211,7 @@ void userfaultfd_release_new(struct userfaultfd_ctx *ctx)
 	mmap_write_unlock(mm);
 }
 
-void userfaultfd_release_all(struct mm_struct *mm,
+static void userfaultfd_release_all(struct mm_struct *mm,
 			     struct userfaultfd_ctx *ctx)
 {
 	struct vm_area_struct *vma, *prev;
Signature · 1 line
2.53.0