mm/vma: remove __vma_check_mmap_hook()
Subsystems (3)
Lorenzo Stoakes
reviewerPATCH 1/1 mm/vma: remove __vma_check_mmap_hook()
Commit c50ca15dd496 ("mm: add vm_ops->mapped hook") introduced __vma_check_mmap_hook() in order to assert that a driver doesn't incorrectly implement both an f_op->mmap() and a vm_ops->mapped hook, the latter of which would not ultimately get invoked.
However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an mmap_prepare hook via the compatibility layer.
Thus the nested mmap_prepare() invocation might result in a VMA which has vm_ops->mapped set with an overlaying mmap() hook, causing the __vma_check_mmap_hook() to fail in vfs_mmap(), wrongly failing the operation.
This patch resolves this by simply removing the check, as we can't be certain that an mmap() hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.
---
Andrew -
c50ca15dd496 is in mm-stable, so thought best to do as fix-patch? Will leave a small bisection hazard (unfortunately) so putting this as close as possible to the patch it fixes would be ideal.
Thanks!
include/linux/fs.h | 9 +-------- mm/util.c | 10 ---------- 2 files changed, 1 insertion(+), 18 deletions(-)
include/linux/fs.h +1 -8
@@ -2062,20 +2062,13 @@ void compat_set_desc_from_vma(struct vm_area_desc *desc, const struct file *file const struct vm_area_struct *vma);int __compat_vma_mmap (struct vm_area_desc *desc,struct vm_area_struct *vma);int compat_vma_mmap (struct file *file,struct vm_area_struct *vma); -int __vma_check_mmap_hook (struct vm_area_struct *vma);static inline int vfs_mmap (struct file *file,struct vm_area_struct *vma) { -int err; -if (file->f_op->mmap_prepare)return compat_vma_mmap (file, vma); - err = file->f_op->mmap (file, vma); -if (err) -return err; - -return __vma_check_mmap_hook (vma); +return file->f_op->mmap (file, vma); }static inline int vfs_mmap_prepare (struct file *file,struct vm_area_desc *desc)
mm/util.c +0 -10
@@ -1281,16 +1281,6 @@ int compat_vma_mmap(struct file *file, struct vm_area_struct *vma) }EXPORT_SYMBOL (compat_vma_mmap); -int __vma_check_mmap_hook (struct vm_area_struct *vma) -{ -/* vm_ops->mapped is not valid if mmap() is specified. */ -if (vma->vm_ops &&WARN_ON_ONCE (vma->vm_ops->mapped)) -return -EINVAL ; - -return 0 ; -} -EXPORT_SYMBOL (__vma_check_mmap_hook); -static void set_ps_flags (struct page_snapshot *ps,const struct folio *folio,const struct page *page) {
--
2.53.0
Quoted 14 lines · Lorenzo Stoakes
Commit
c50ca15dd496("mm: addvm_ops->mappedhook") introduced__vma_check_mmap_hook()in order to assert that a driver doesn't incorrectly implement both anf_op->mmap()and avm_ops->mappedhook, the latter of which would not ultimately get invoked.However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an
mmap_preparehook via the compatibility layer.Thus the nested
mmap_prepare()invocation might result in a VMA which hasvm_ops->mappedset with an overlayingmmap()hook, causing the__vma_check_mmap_hook()to fail invfs_mmap(), wrongly failing the operation.
Quoted 8 lines
This patch resolves this by simply removing the check, as we can't be certain that an
mmap()hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.Fixes c50ca15dd496 ("mm: add vm_ops->mapped hook")Reported-by Shinichiro Kawasaki <[email protected]>Signed-off-by Lorenzo Stoakes <[email protected]>
Quoted 63 lines
---
Andrew -
c50ca15dd496is in mm-stable, so thought best to do as fix-patch? Will leave a small bisection hazard (unfortunately) so putting this as close as possible to the patch it fixes would be ideal.Thanks!
include/linux/fs.h | 9 +-------- mm/util.c | 10 ---------- 2 files changed, 1 insertion(+), 18 deletions(-)include/linux/fs.h +1
-8@@ -2062,20 +2062,13 @@ void compat_set_desc_from_vma(struct vm_area_desc *desc, const struct file *file const struct vm_area_struct *vma);int __compat_vma_mmap (struct vm_area_desc *desc,struct vm_area_struct *vma);int compat_vma_mmap (struct file *file,struct vm_area_struct *vma); -int __vma_check_mmap_hook (struct vm_area_struct *vma);static inline int vfs_mmap (struct file *file,struct vm_area_struct *vma) { -int err; -if (file->f_op->mmap_prepare)return compat_vma_mmap (file, vma); - err = file->f_op->mmap (file, vma); -if (err) -return err; - -return __vma_check_mmap_hook (vma); +return file->f_op->mmap (file, vma); }static inline int vfs_mmap_prepare (struct file *file,struct vm_area_desc *desc)mm/util.c +0
-11@@ -1281,16 +1281,6 @@ int compat_vma_mmap(struct file *file, struct vm_area_struct *vma) }EXPORT_SYMBOL (compat_vma_mmap); -int __vma_check_mmap_hook (struct vm_area_struct *vma) -{ -/* vm_ops->mapped is not valid if mmap() is specified. */ -if (vma->vm_ops &&WARN_ON_ONCE (vma->vm_ops->mapped)) -return -EINVAL ; - -return 0 ; -} -EXPORT_SYMBOL (__vma_check_mmap_hook); -static void set_ps_flags (struct page_snapshot *ps,const struct folio *folio,const struct page *page) { --2.53.0
Quoted 14 lines · Lorenzo Stoakes
Commit
c50ca15dd496("mm: addvm_ops->mappedhook") introduced__vma_check_mmap_hook()in order to assert that a driver doesn't incorrectly implement both anf_op->mmap()and avm_ops->mappedhook, the latter of which would not ultimately get invoked.However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an
mmap_preparehook via the compatibility layer.Thus the nested
mmap_prepare()invocation might result in a VMA which hasvm_ops->mappedset with an overlayingmmap()hook, causing the__vma_check_mmap_hook()to fail invfs_mmap(), wrongly failing the operation.
Quoted 8 lines
This patch resolves this by simply removing the check, as we can't be certain that an
mmap()hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.Fixes c50ca15dd496 ("mm: add vm_ops->mapped hook")Reported-by Shinichiro Kawasaki <[email protected]>Signed-off-by Lorenzo Stoakes <[email protected]>
Lorenzo, thank you for the swift fix. I applied this patch on top of the next-20260410 kernel, and confirmed the failure disappeared. Good.
Lorenzo Stoakes
reviewerQuoted 24 lines · Shinichiro Kawasaki
Quoted 14 lines · Lorenzo Stoakes
Commit
c50ca15dd496("mm: addvm_ops->mappedhook") introduced__vma_check_mmap_hook()in order to assert that a driver doesn't incorrectly implement both anf_op->mmap()and avm_ops->mappedhook, the latter of which would not ultimately get invoked.However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an
mmap_preparehook via the compatibility layer.Thus the nested
mmap_prepare()invocation might result in a VMA which hasvm_ops->mappedset with an overlayingmmap()hook, causing the__vma_check_mmap_hook()to fail invfs_mmap(), wrongly failing the operation.Quoted 8 lines
This patch resolves this by simply removing the check, as we can't be certain that an
mmap()hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.Fixes c50ca15dd496 ("mm: add vm_ops->mapped hook")Reported-by Shinichiro Kawasaki <[email protected]>Signed-off-by Lorenzo Stoakes <[email protected]>
Quoted 4 lines
Lorenzo, thank you for the swift fix. I applied this patch on top of the next-20260410 kernel, and confirmed the failure disappeared. Good.
Tested-by Shinichiro Kawasaki <[email protected]>
Thanks! :)
Cheers, Lorenzo
Andrew Morton
maintainerQuoted 9 lines · On Mon, 13 Apr 2026 11:57:13 +0100 Lorenzo Stoakes <[email protected]> wrote:
Commit
c50ca15dd496("mm: addvm_ops->mappedhook") introduced__vma_check_mmap_hook()in order to assert that a driver doesn't incorrectly implement both anf_op->mmap()and avm_ops->mappedhook, the latter of which would not ultimately get invoked.However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an
mmap_preparehook via the compatibility layer.
Quoted 8 lines
Thus the nested
mmap_prepare()invocation might result in a VMA which hasvm_ops->mappedset with an overlayingmmap()hook, causing the__vma_check_mmap_hook()to fail invfs_mmap(), wrongly failing the operation.This patch resolves this by simply removing the check, as we can't be certain that an
mmap()hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.
Cool, thanks.
Quoted 1 line
Fixes c50ca15dd496 ("mm: add vm_ops->mapped hook")
That's presently in flight mm.git->Linus, So I'll include this fix in next week's second batch of MM updates into Linus.
Quoted 3 lines
c50ca15dd496is in mm-stable, so thought best to do as fix-patch? Will leave a small bisection hazard (unfortunately) so putting this as close as possible to the patch it fixes would be ideal.
Yep, I did that.
Lorenzo Stoakes
reviewerQuoted 20 lines · Andrew Morton
Quoted 9 lines · On Mon, 13 Apr 2026 11:57:13 +0100 Lorenzo Stoakes <[email protected]> wrote:
Commit
c50ca15dd496("mm: addvm_ops->mappedhook") introduced__vma_check_mmap_hook()in order to assert that a driver doesn't incorrectly implement both anf_op->mmap()and avm_ops->mappedhook, the latter of which would not ultimately get invoked.However, this did not correctly account for stacked drivers (or drivers that otherwise use the compatibility layer) which might recursively call an
mmap_preparehook via the compatibility layer.Quoted 8 lines
Thus the nested
mmap_prepare()invocation might result in a VMA which hasvm_ops->mappedset with an overlayingmmap()hook, causing the__vma_check_mmap_hook()to fail invfs_mmap(), wrongly failing the operation.This patch resolves this by simply removing the check, as we can't be certain that an
mmap()hook doesn't at some point invoke the compatibility layer, and it's not worth trying to track it.
Quoted 6 lines
Cool, thanks.
Quoted 1 line
Fixes c50ca15dd496 ("mm: add vm_ops->mapped hook")That's presently in flight
mm.git->Linus, So I'll include this fix in next week's second batch of MM updates into Linus.
Ack thanks!
Quoted 7 lines
Quoted 5 lines
c50ca15dd496is in mm-stable, so thought best to do as fix-patch? Will leave a small bisection hazard (unfortunately) so putting this as close as possible to the patch it fixes would be ideal.Yep, I did that.
Thanks!
Quoted 2 lines
Cheers, Lorenzo