Skip to contentSkip to conversation

mm/vma: remove __vma_check_mmap_hook()

PATCH Applied Lorenzo Stoakes 4 participants 6 messages Apr 13 – Apr 15

Subsystems (3)
lore
Share & tools
lore

Lorenzo Stoakes

author reviewer
#0 lore raw

PATCH 1/1 mm/vma: remove __vma_check_mmap_hook()

Commit c50ca15dd496 ("mm: add vm_ops->mapped hook") introduced __vma_check_mmap_hook() in order to assert that a driver doesn't incorrectly implement both an f_op->mmap() and a vm_ops->mapped hook, the latter of which would not ultimately get invoked.

However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an mmap_prepare hook via the compatibility layer.

Thus the nested mmap_prepare() invocation might result in a VMA which has vm_ops->mapped set with an overlaying mmap() hook, causing the __vma_check_mmap_hook() to fail in vfs_mmap(), wrongly failing the operation.

This patch resolves this by simply removing the check, as we can't be certain that an mmap() hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.

---

Andrew -

c50ca15dd496 is in mm-stable, so thought best to do as fix-patch? Will leave a small bisection hazard (unfortunately) so putting this as close as possible to the patch it fixes would be ideal.

Thanks!

 include/linux/fs.h |  9 +--------
 mm/util.c          | 10 ----------
 2 files changed, 1 insertion(+), 18 deletions(-)
include/linux/fs.h +1 -8
index 0bdccfa70b44..f3ca9b841892 100644
@@ -2062,20 +2062,13 @@ void compat_set_desc_from_vma(struct vm_area_desc *desc, const struct file *file
 			      const struct vm_area_struct *vma);
 int __compat_vma_mmap(struct vm_area_desc *desc, struct vm_area_struct *vma);
 int compat_vma_mmap(struct file *file, struct vm_area_struct *vma);
-int __vma_check_mmap_hook(struct vm_area_struct *vma);
 
 static inline int vfs_mmap(struct file *file, struct vm_area_struct *vma)
 {
-	int err;
-
 	if (file->f_op->mmap_prepare)
 		return compat_vma_mmap(file, vma);
 
-	err = file->f_op->mmap(file, vma);
-	if (err)
-		return err;
-
-	return __vma_check_mmap_hook(vma);
+	return file->f_op->mmap(file, vma);
 }
 
 static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc)
mm/util.c +0 -10
index f063fd4de1e8..232c3930a662 100644
@@ -1281,16 +1281,6 @@ int compat_vma_mmap(struct file *file, struct vm_area_struct *vma)
 }
 EXPORT_SYMBOL(compat_vma_mmap);
 
-int __vma_check_mmap_hook(struct vm_area_struct *vma)
-{
-	/* vm_ops->mapped is not valid if mmap() is specified. */
-	if (vma->vm_ops && WARN_ON_ONCE(vma->vm_ops->mapped))
-		return -EINVAL;
-
-	return 0;
-}
-EXPORT_SYMBOL(__vma_check_mmap_hook);
-
 static void set_ps_flags(struct page_snapshot *ps, const struct folio *folio,
 			 const struct page *page)
 {

--
2.53.0

reviewer Acked-by on v1 1/1
#1 lore raw
Quoted 14 lines · Lorenzo Stoakes

Commit c50ca15dd496 ("mm: add vm_ops->mapped hook") introduced __vma_check_mmap_hook() in order to assert that a driver doesn't incorrectly implement both an f_op->mmap() and a vm_ops->mapped hook, the latter of which would not ultimately get invoked.

However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an mmap_prepare hook via the compatibility layer.

Thus the nested mmap_prepare() invocation might result in a VMA which has vm_ops->mapped set with an overlaying mmap() hook, causing the __vma_check_mmap_hook() to fail in vfs_mmap(), wrongly failing the operation.

Quoted 8 lines

This patch resolves this by simply removing the check, as we can't be certain that an mmap() hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.

Quoted 63 lines

---

Andrew -

c50ca15dd496 is in mm-stable, so thought best to do as fix-patch? Will leave a small bisection hazard (unfortunately) so putting this as close as possible to the patch it fixes would be ideal.

Thanks!

 include/linux/fs.h |  9 +--------
 mm/util.c          | 10 ----------
 2 files changed, 1 insertion(+), 18 deletions(-)
include/linux/fs.h +1 -8
index 0bdccfa70b44..f3ca9b841892 100644
@@ -2062,20 +2062,13 @@ void compat_set_desc_from_vma(struct vm_area_desc *desc, const struct file *file
 			      const struct vm_area_struct *vma);
 int __compat_vma_mmap(struct vm_area_desc *desc, struct vm_area_struct *vma);
 int compat_vma_mmap(struct file *file, struct vm_area_struct *vma);
-int __vma_check_mmap_hook(struct vm_area_struct *vma);
 
 static inline int vfs_mmap(struct file *file, struct vm_area_struct *vma)
 {
-	int err;
-
 	if (file->f_op->mmap_prepare)
 		return compat_vma_mmap(file, vma);
 
-	err = file->f_op->mmap(file, vma);
-	if (err)
-		return err;
-
-	return __vma_check_mmap_hook(vma);
+	return file->f_op->mmap(file, vma);
 }
 
 static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc)
mm/util.c +0 -11
index f063fd4de1e8..232c3930a662 100644
@@ -1281,16 +1281,6 @@ int compat_vma_mmap(struct file *file, struct vm_area_struct *vma)
 }
 EXPORT_SYMBOL(compat_vma_mmap);
 
-int __vma_check_mmap_hook(struct vm_area_struct *vma)
-{
-	/* vm_ops->mapped is not valid if mmap() is specified. */
-	if (vma->vm_ops && WARN_ON_ONCE(vma->vm_ops->mapped))
-		return -EINVAL;
-
-	return 0;
-}
-EXPORT_SYMBOL(__vma_check_mmap_hook);
-
 static void set_ps_flags(struct page_snapshot *ps, const struct folio *folio,
 			 const struct page *page)
 {
--

2.53.0

Tested-by on v1 1/1
#2 lore raw
In reply to Lorenzo Stoakes
Quoted 14 lines · Lorenzo Stoakes

Commit c50ca15dd496 ("mm: add vm_ops->mapped hook") introduced __vma_check_mmap_hook() in order to assert that a driver doesn't incorrectly implement both an f_op->mmap() and a vm_ops->mapped hook, the latter of which would not ultimately get invoked.

However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an mmap_prepare hook via the compatibility layer.

Thus the nested mmap_prepare() invocation might result in a VMA which has vm_ops->mapped set with an overlaying mmap() hook, causing the __vma_check_mmap_hook() to fail in vfs_mmap(), wrongly failing the operation.

Quoted 8 lines

This patch resolves this by simply removing the check, as we can't be certain that an mmap() hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.

Lorenzo, thank you for the swift fix. I applied this patch on top of the next-20260410 kernel, and confirmed the failure disappeared. Good.

author reviewer
#3 lore raw
Quoted 24 lines · Shinichiro Kawasaki
Quoted 14 lines · Lorenzo Stoakes

Commit c50ca15dd496 ("mm: add vm_ops->mapped hook") introduced __vma_check_mmap_hook() in order to assert that a driver doesn't incorrectly implement both an f_op->mmap() and a vm_ops->mapped hook, the latter of which would not ultimately get invoked.

However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an mmap_prepare hook via the compatibility layer.

Thus the nested mmap_prepare() invocation might result in a VMA which has vm_ops->mapped set with an overlaying mmap() hook, causing the __vma_check_mmap_hook() to fail in vfs_mmap(), wrongly failing the operation.

Quoted 8 lines

This patch resolves this by simply removing the check, as we can't be certain that an mmap() hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.

Quoted 4 lines

Lorenzo, thank you for the swift fix. I applied this patch on top of the next-20260410 kernel, and confirmed the failure disappeared. Good.

Thanks! :)

Cheers, Lorenzo

maintainer
#4 lore raw
In reply to Lorenzo Stoakes
Quoted 9 lines · On Mon, 13 Apr 2026 11:57:13 +0100 Lorenzo Stoakes <[email protected]> wrote:

Commit c50ca15dd496 ("mm: add vm_ops->mapped hook") introduced __vma_check_mmap_hook() in order to assert that a driver doesn't incorrectly implement both an f_op->mmap() and a vm_ops->mapped hook, the latter of which would not ultimately get invoked.

However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an mmap_prepare hook via the compatibility layer.

Quoted 8 lines

Thus the nested mmap_prepare() invocation might result in a VMA which has vm_ops->mapped set with an overlaying mmap() hook, causing the __vma_check_mmap_hook() to fail in vfs_mmap(), wrongly failing the operation.

This patch resolves this by simply removing the check, as we can't be certain that an mmap() hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.

Cool, thanks.

Quoted 1 line
Fixes c50ca15dd496 ("mm: add vm_ops->mapped hook")

That's presently in flight mm.git->Linus, So I'll include this fix in next week's second batch of MM updates into Linus.

Quoted 3 lines

c50ca15dd496 is in mm-stable, so thought best to do as fix-patch? Will leave a small bisection hazard (unfortunately) so putting this as close as possible to the patch it fixes would be ideal.

Yep, I did that.

author reviewer
#5 lore raw
Quoted 20 lines · Andrew Morton
Quoted 9 lines · On Mon, 13 Apr 2026 11:57:13 +0100 Lorenzo Stoakes <[email protected]> wrote:

Commit c50ca15dd496 ("mm: add vm_ops->mapped hook") introduced __vma_check_mmap_hook() in order to assert that a driver doesn't incorrectly implement both an f_op->mmap() and a vm_ops->mapped hook, the latter of which would not ultimately get invoked.

However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an mmap_prepare hook via the compatibility layer.

Quoted 8 lines

Thus the nested mmap_prepare() invocation might result in a VMA which has vm_ops->mapped set with an overlaying mmap() hook, causing the __vma_check_mmap_hook() to fail in vfs_mmap(), wrongly failing the operation.

This patch resolves this by simply removing the check, as we can't be certain that an mmap() hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.

Quoted 6 lines

Cool, thanks.

Quoted 1 line
Fixes c50ca15dd496 ("mm: add vm_ops->mapped hook")

That's presently in flight mm.git->Linus, So I'll include this fix in next week's second batch of MM updates into Linus.

Ack thanks!

Quoted 7 lines
Quoted 5 lines

c50ca15dd496 is in mm-stable, so thought best to do as fix-patch? Will leave a small bisection hazard (unfortunately) so putting this as close as possible to the patch it fixes would be ideal.

Yep, I did that.

Thanks!

Quoted 2 lines

Cheers, Lorenzo