eliminate mmap() retry merge, add .mmap_prepare hook
Subsystems (5)
cover eliminate mmap() retry merge, add .mmap_prepare hook
During the mmap() of a file-backed mapping, we invoke the underlying driver file's mmap() callback in order to perform driver/file system initialisation of the underlying VMA.
This has been a source of issues in the past, including a significant security concern relating to unwinding of error state discovered by Jann Horn, as fixed in commit 5de195060b2e ("mm: resolve faulty mmap_region() error path behaviour") which performed the recent, significant, rework of mmap() as a whole.
However, we have had a fly in the ointment remain - drivers have a great deal of freedom in the .mmap() hook to manipulate VMA state (as well as page table state).
This can be problematic, as we can no longer reason sensibly about VMA state once the call is complete (the ability to do - anything - here does rather interfere with that).
In addition, callers may choose to do odd or unusual things which might interfere with subsequent steps in the mmap() process, and it may do so and then raise an error, requiring very careful unwinding of state about which we can make no assumptions.
Rather than providing such an open-ended interface, this series provides an alternative, far more restrictive one - we expose a whitelist of fields which can be adjusted by the driver, along with immutable state upon which the driver can make such decisions:
struct vm_area_desc {
/* Immutable state. */
struct mm_struct *mm;
unsigned long start;
unsigned long end;
/* Mutable fields. Populated with initial state. */
pgoff_t pgoff;
struct file *file;
vm_flags_t vm_flags;
pgprot_t page_prot;
/* Write-only fields. */
const struct vm_operations_struct *vm_ops;
void *private_data;
};The mmap logic then updates the state used to either merge with a VMA or establish a new VMA based upon this logic.
This is achieved via new file hook .mmap_prepare(), which is, importantly, invoked very early on in the mmap() process.
If an error arises, we can very simply abort the operation with very little unwinding of state required.
The existing logic contains another, related, peccadillo - since the .mmap() callback might do anything, it may also cause a previously unmergeable VMA to become mergeable with adjacent VMAs.
Right now the logic will retry a merge like this only if the driver changes VMA flags, and changes them in such a way that a merge might succeed (that is, the flags are not 'special', that is do not contain any of the flags specified in VM_SPECIAL).
This has also been the source of a great deal of pain - it's hard to reason about an .mmap() callback that might do - anything - but it's also hard to reason about setting up a VMA and writing to the maple tree, only to do it again utilising a great deal of shared state.
Since .mmap_prepare() sets fields before the first merge is even attempted, the use of this callback obviates the need for this retry merge logic.
A driver may only specify .mmap_prepare() or the deprecated .mmap() callback. In future we may add futher callbacks beyond .mmap_prepare() to faciliate all use cass as we convert drivers.
In researching this change, I examined every .mmap() callback, and discovered only a very few that set VMA state in such a way that a. the VMA flags changed and b. this would be mergeable.
In the majority of cases, it turns out that drivers are mapping kernel memory and thus ultimately set VM_PFNMAP, VM_MIXEDMAP, or other unmergeable VM_SPECIAL flags.
Of those that remain I identified a number of cases which are only applicable in DAX, setting the VM_HUGEPAGE flag:
* dax_mmap()
* erofs_file_mmap()
* ext4_file_mmap()
* xfs_file_mmap()
For this remerge to not occur and to impact users, each of these cases would require a user to mmap() files using DAX, in parts, immediately adjacent to one another.
This is a very unlikely usecase and so it does not appear to be worthwhile to adjust this functionality accordingly.
We can, however, very quickly do so if needed by simply adding an .mmap_prepare() callback to these as required.
There are two further non-DAX cases I idenitfied:
* orangefs_file_mmap() - Clears VM_RAND_READ if set, replacing with VM_SEQ_READ.
* usb_stream_hwdep_mmap() - Sets VM_DONTDUMP.
Both of these cases again seem very unlikely to be mmap()'d immediately adjacent to one another in a fashion that would result in a merge.
Finally, we are left with a viable case:
* secretmem_mmap() - Set VM_LOCKED, VM_DONTDUMP.
This is viable enough that the mm selftests trigger the logic as a matter of course. Therefore, this series replace the .secretmem_mmap() hook with .secret_mmap_prepare().
v1:
* Seems generally supported, so un-RFC :)
* Propagated tag, thanks Mike!
RFC v2:
* Renamed .mmap_proto() to .mmap_prepare() as per David.
* Made .mmap_prepare(), .mmap() mutually exclusive.
* Updated call_mmap() to bail out if .mmap_prepare() callback present as per Jann.
* Renamed vma_proto to vm_area_desc as per Mike.
* Added accidentally missing page_prot assignment/read from vm_area_desc.
* Renamed vm_area_desc->flags to vm_flags as per Liam.
* Added [__]call_mmap_prepare() for consistency with call_mmap().
* Added file_has_xxx_hook() helpers.
* Renamed file_has_valid_mmap() to file_has_valid_mmap_hooks() and check that the hook is mutually exclusive.
https://lore.kernel.org/all/[email protected]/
RFC v1:
https://lore.kernel.org/all/[email protected]/
Lorenzo Stoakes (3):
mm: introduce new .mmap_prepare() file callback
mm: secretmem: convert to .mmap_prepare() hook
mm/vma: remove mmap() retry merge
include/linux/fs.h | 38 +++++++++++++++ include/linux/mm_types.h | 24 ++++++++++ mm/memory.c | 3 +- mm/mmap.c | 2 +- mm/secretmem.c | 14 +++--- mm/vma.c | 82 ++++++++++++++++++++++++++------ tools/testing/vma/vma_internal.h | 79 ++++++++++++++++++++++++++++-- 7 files changed, 214 insertions(+), 28 deletions(-)
--
2.49.0
PATCH 1/3 mm: introduce new .mmap_prepare() file callback
Provide a means by which drivers can specify which fields of those permitted to be changed should be altered to prior to mmap()'ing a range (which may either result from a merge or from mapping an entirely new VMA).
Doing so is substantially safer than the existing .mmap() calback which provides unrestricted access to the part-constructed VMA and permits drivers and file systems to do 'creative' things which makes it hard to reason about the state of the VMA after the function returns.
The existing .mmap() callback's freedom has caused a great deal of issues, especially in error handling, as unwinding the mmap() state has proven to be non-trivial and caused significant issues in the past, for instance those addressed in commit 5de195060b2e ("mm: resolve faulty mmap_region() error path behaviour").
It also necessitates a second attempt at merge once the .mmap() callback has completed, which has caused issues in the past, is awkward, adds overhead and is difficult to reason about.
The .mmap_prepare() callback eliminates this requirement, as we can update fields prior to even attempting the first merge. It is safer, as we heavily restrict what can actually be modified, and being invoked very early in the mmap() process, error handling can be performed safely with very little unwinding of state required.
The .mmap_prepare() and deprecated .mmap() callbacks are mutually exclusive, so we permit only one to be invoked at a time.
Update vma userland test stubs to account for changes.
--- include/linux/fs.h | 38 +++++++++++++++ include/linux/mm_types.h | 24 ++++++++++ mm/memory.c | 3 +- mm/mmap.c | 2 +- mm/vma.c | 70 +++++++++++++++++++++++++++- tools/testing/vma/vma_internal.h | 79 ++++++++++++++++++++++++++++++-- 6 files changed, 208 insertions(+), 8 deletions(-)
include/linux/fs.h +38 -0
@@ -2169,6 +2169,7 @@ struct file_operations {int (*uring_cmd)(struct io_uring_cmd *ioucmd,unsigned int issue_flags);int (*uring_cmd_iopoll)(struct io_uring_cmd *,struct io_comp_batch *,unsigned int poll_flags); +int (*mmap_prepare)(struct vm_area_desc *); } __randomize_layout;/* Supports async buffered reads */
@@ -2238,11 +2239,48 @@ struct inode_operations {struct offset_ctx *(*get_offset_ctx)(struct inode *inode); }____cacheline_aligned ; +static inline bool file_has_deprecated_mmap_hook (struct file *file) +{ +return file->f_op->mmap; +} + +static inline bool file_has_mmap_prepare_hook (struct file *file) +{ +return file->f_op->mmap_prepare; +} + +/* Did the driver provide valid mmap hook configuration? */ +static inline bool file_has_valid_mmap_hooks (struct file *file) +{ +bool has_mmap =file_has_deprecated_mmap_hook (file); +bool has_mmap_prepare =file_has_mmap_prepare_hook (file); + +/* Hooks are mutually exclusive. */ +if (has_mmap && has_mmap_prepare) +return false; + +/* But at least one must be specified. */ +if (!has_mmap && !has_mmap_prepare) +return false; + +return true; +} +static inline int call_mmap (struct file *file,struct vm_area_struct *vma) { +/* If the driver specifies .mmap_prepare() this call is invalid. */ +if (file_has_mmap_prepare_hook (file)) +return -EINVAL ; +return file->f_op->mmap (file, vma); } +static inline int __call_mmap_prepare (struct file *file, +struct vm_area_desc *desc) +{ +return file->f_op->mmap_prepare (desc); +} +extern ssize_t vfs_read (struct file *,char __user *,size_t ,loff_t *);extern ssize_t vfs_write (struct file *,const char __user *,size_t ,loff_t *); extern ssize_t vfs_copy_file_range(struct file *, loff_t , struct file *,
include/linux/mm_types.h +24 -0
@@ -763,6 +763,30 @@ struct vma_numab_state {int prev_scan_seq; }; +/* +* Describes a VMA that is about to be mmap()'ed. Drivers may choose to +* manipulate mutable fields which will cause those fields to be updated in the +* resultant VMA. +* +* Helper functions are not required for manipulating any field. +*/ +struct vm_area_desc { +/* Immutable state. */ +struct mm_struct *mm; +unsigned long start; +unsigned long end; + +/* Mutable fields. Populated with initial state. */ +pgoff_t pgoff; +struct file *file; +vm_flags_t vm_flags; +pgprot_t page_prot; + +/* Write-only fields. */ +const struct vm_operations_struct *vm_ops; +void *private_data; +}; +/* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory
mm/memory.c +2 -1
@@ -527,10 +527,11 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr,dump_page (page,"bad pte" );pr_alert ("addr:%px vm_flags:%08lx anon_vma:%px mapping:%px index:%lx\n" , (void *)addr, vma->vm_flags, vma->anon_vma, mapping, index); -pr_alert ("file:%pD fault:%ps mmap:%ps read_folio:%ps\n" , +pr_alert ("file:%pD fault:%ps mmap:%ps mmap_prepare: %ps read_folio:%ps\n" , vma->vm_file, vma->vm_ops ? vma->vm_ops->fault :NULL , vma->vm_file ? vma->vm_file->f_op->mmap :NULL , + vma->vm_file ? vma->vm_file->f_op->mmap_prepare :NULL , mapping ? mapping->a_ops->read_folio :NULL );dump_stack ();add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE );
mm/mmap.c +1 -1
@@ -475,7 +475,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr, vm_flags &= ~VM_MAYEXEC ; } -if (!file->f_op->mmap) +if (!file_has_valid_mmap_hooks (file))return -ENODEV ;if (vm_flags & (VM_GROWSDOWN |VM_GROWSUP ))return -EINVAL ;
mm/vma.c +69 -1
116 lines not on this page · raw message
tools/testing/vma/vma_internal.h +74 -5
97 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH 1/3] mm: introduce new .mmap_prepare() file callback
Quoted 26 lines · Lorenzo Stoakes
Provide a means by which drivers can specify which fields of those permitted to be changed should be altered to prior to mmap()'ing a range (which may either result from a merge or from mapping an entirely new VMA).
Doing so is substantially safer than the existing .mmap() calback which provides unrestricted access to the part-constructed VMA and permits drivers and file systems to do 'creative' things which makes it hard to reason about the state of the VMA after the function returns.
The existing .mmap() callback's freedom has caused a great deal of issues, especially in error handling, as unwinding the
mmap()state has proven to be non-trivial and caused significant issues in the past, for instance those addressed in commit5de195060b2e("mm: resolve faultymmap_region()error path behaviour").It also necessitates a second attempt at merge once the .mmap() callback has completed, which has caused issues in the past, is awkward, adds overhead and is difficult to reason about.
The .mmap_prepare() callback eliminates this requirement, as we can update fields prior to even attempting the first merge. It is safer, as we heavily restrict what can actually be modified, and being invoked very early in the
mmap()process, error handling can be performed safely with very little unwinding of state required.
Quoted 5 lines
The .mmap_prepare() and deprecated .mmap() callbacks are mutually exclusive, so we permit only one to be invoked at a time.
Update vma userland test stubs to account for changes.
In general, looks very good to me.
Some comments, especially regarding suboptimal code duplciation with the stubs. (unless I am missing fine details :) )
Quoted 25 lines
Signed-off-by Lorenzo Stoakes <[email protected]>--- include/linux/fs.h | 38 +++++++++++++++ include/linux/mm_types.h | 24 ++++++++++ mm/memory.c | 3 +- mm/mmap.c | 2 +- mm/vma.c | 70 +++++++++++++++++++++++++++- tools/testing/vma/vma_internal.h | 79 ++++++++++++++++++++++++++++++-- 6 files changed, 208 insertions(+), 8 deletions(-)include/linux/fs.h +1
-0@@ -2169,6 +2169,7 @@ struct file_operations {int (*uring_cmd)(struct io_uring_cmd *ioucmd,unsigned int issue_flags);int (*uring_cmd_iopoll)(struct io_uring_cmd *,struct io_comp_batch *,unsigned int poll_flags); +int (*mmap_prepare)(struct vm_area_desc *); } __randomize_layout;/* Supports async buffered reads */ @@ -2238,11 +2239,48 @@ struct inode_operations {struct offset_ctx *(*get_offset_ctx)(struct inode *inode); }____cacheline_aligned ;
include/linux/fs.h struct inode_operations { · line 2250
Quoted 10 lines
include/linux/fs.h +9
-0+static inline bool file_has_deprecated_mmap_hook (struct file *file) +{ +return file->f_op->mmap; +} + +static inline bool file_has_mmap_prepare_hook (struct file *file) +{ +return file->f_op->mmap_prepare; +}
I am usually not a fan of such dummy helper functions .. I mean, how far do we go?
file_has_f_op()
file_is_non_null()
...
Or is this required for some stubbing regarding vma tests? But even the stubs below confuse me a bit, because they do exactly the same thing :(
:)
include/linux/fs.h struct inode_operations { · line 2259
Quoted 9 lines
include/linux/fs.h +9
-0+ +/* Did the driver provide valid mmap hook configuration? */ +static inline bool file_has_valid_mmap_hooks (struct file *file) +{ +bool has_mmap =file_has_deprecated_mmap_hook (file); +bool has_mmap_prepare =file_has_mmap_prepare_hook (file); + +/* Hooks are mutually exclusive. */ +if (has_mmap && has_mmap_prepare)
Should this be WARN_ON_ONCE() ?
include/linux/fs.h struct inode_operations { · line 2266
Quoted 7 lines
include/linux/fs.h +7
-0+return false; + +/* But at least one must be specified. */ +if (!has_mmap && !has_mmap_prepare) +return false; + +return true;
return has_mmap || has_mmap_prepare;
And I think you can drop the comment about "at least one" with that, should be quite clear from that simplified version.
include/linux/fs.h struct inode_operations { · line 2272
Quoted 6 lines
include/linux/fs.h +4
-0+} +static inline int call_mmap (struct file *file,struct vm_area_struct *vma) { +/* If the driver specifies .mmap_prepare() this call is invalid. */ +if (file_has_mmap_prepare_hook (file))
Should this be WARN_ON_ONCE() ?
Quoted 11 lines
include/linux/fs.h +8
-0+return -EINVAL ; +return file->f_op->mmap (file, vma); } +static inline int __call_mmap_prepare (struct file *file, +struct vm_area_desc *desc) +{ +return file->f_op->mmap_prepare (desc); +} +
[...]
Quoted 49 lines
struct file { struct address_space *f_mapping;include/linux/fs.h +28
-5+const struct file_operations *f_op; };#define VMA_LOCK_OFFSET 0x40000000@@ -1125,11 +1157,6 @@ static inline void vm_flags_clear(struct vm_area_struct *vma, vma->__vm_flags &= ~flags; } -static inline int call_mmap (struct file *,struct vm_area_struct *) -{ -return 0 ; -} -static inline int shmem_zero_setup (struct vm_area_struct *) {return 0 ;@@ -1405,4 +1432,46 @@ static inline void free_anon_vma_name(struct vm_area_struct *vma) (void )vma; } +static inline bool file_has_deprecated_mmap_hook (struct file *file) +{ +return file->f_op->mmap; +} + +static inline bool file_has_mmap_prepare_hook (struct file *file) +{ +return file->f_op->mmap_prepare; +} +> +/* Did the driver provide valid mmap hook configuration? */ +static inline bool file_has_valid_mmap_hooks(struct file *file) +{ +bool has_mmap =file_has_deprecated_mmap_hook (file); +bool has_mmap_prepare =file_has_mmap_prepare_hook (file); + +/* Hooks are mutually exclusive. */ +if (has_mmap && has_mmap_prepare) +return false; +> + /* But at least one must be specified. */ + if (!has_mmap && !has_mmap_prepare) +return false; + + return true;> +} + +static inline int call_mmap (struct file *file,struct vm_area_struct *vma) +{ +/* If the driver specifies .mmap_prepare() this call is invalid. */
include/linux/fs.h static inline void free_anon_vma_name(struct vm_area_struct *vma) · line 1471
Quoted 10 lines
include/linux/fs.h +10
-0+if (file_has_mmap_prepare_hook (file)) + return -EINVAL;> + + return file->f_op->mmap(file, vma); +} + +static inline int __call_mmap_prepare (struct file *file, +struct vm_area_desc *desc) +{ +return file->f_op->mmap_prepare (desc); +}
Hm, is there a way avoid a copy of the exact same code from fs.h, and essentially test the implementation in fs.h (-> more coverage by using less duplciated stubs?).
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 1/3] mm: introduce new .mmap_prepare() file callback
Quoted 27 lines · David Hildenbrand
Quoted 26 lines · Lorenzo Stoakes
Provide a means by which drivers can specify which fields of those permitted to be changed should be altered to prior to mmap()'ing a range (which may either result from a merge or from mapping an entirely new VMA).
Doing so is substantially safer than the existing .mmap() calback which provides unrestricted access to the part-constructed VMA and permits drivers and file systems to do 'creative' things which makes it hard to reason about the state of the VMA after the function returns.
The existing .mmap() callback's freedom has caused a great deal of issues, especially in error handling, as unwinding the
mmap()state has proven to be non-trivial and caused significant issues in the past, for instance those addressed in commit5de195060b2e("mm: resolve faultymmap_region()error path behaviour").It also necessitates a second attempt at merge once the .mmap() callback has completed, which has caused issues in the past, is awkward, adds overhead and is difficult to reason about.
The .mmap_prepare() callback eliminates this requirement, as we can update fields prior to even attempting the first merge. It is safer, as we heavily restrict what can actually be modified, and being invoked very early in the
mmap()process, error handling can be performed safely with very little unwinding of state required.
Quoted 7 lines
Quoted 5 lines
The .mmap_prepare() and deprecated .mmap() callbacks are mutually exclusive, so we permit only one to be invoked at a time.
Update vma userland test stubs to account for changes.
In general, looks very good to me.
Thanks!
Quoted 3 lines
Some comments, especially regarding suboptimal code duplciation with the stubs. (unless I am missing fine details :) )
Responding inline... :)
Quoted 40 lines
Quoted 24 lines
Signed-off-by Lorenzo Stoakes <[email protected]>--- include/linux/fs.h | 38 +++++++++++++++ include/linux/mm_types.h | 24 ++++++++++ mm/memory.c | 3 +- mm/mmap.c | 2 +- mm/vma.c | 70 +++++++++++++++++++++++++++- tools/testing/vma/vma_internal.h | 79 ++++++++++++++++++++++++++++++-- 6 files changed, 208 insertions(+), 8 deletions(-)include/linux/fs.h +1
-0@@ -2169,6 +2169,7 @@ struct file_operations {int (*uring_cmd)(struct io_uring_cmd *ioucmd,unsigned int issue_flags);int (*uring_cmd_iopoll)(struct io_uring_cmd *,struct io_comp_batch *,unsigned int poll_flags); +int (*mmap_prepare)(struct vm_area_desc *); } __randomize_layout;/* Supports async buffered reads */ @@ -2238,11 +2239,48 @@ struct inode_operations {struct offset_ctx *(*get_offset_ctx)(struct inode *inode);include/linux/fs.h struct inode_operations { · line 2249
Quoted 10 lines
include/linux/fs.h +9
-0}____cacheline_aligned ; +static inline bool file_has_deprecated_mmap_hook (struct file *file) +{ +return file->f_op->mmap; +} + +static inline bool file_has_mmap_prepare_hook (struct file *file) +{ +return file->f_op->mmap_prepare; +}I am usually not a fan of such dummy helper functions .. I mean, how far do we go?
file_has_f_op()
Quoted 8 lines
file_is_non_null()...
Or is this required for some stubbing regarding vma tests? But even the stubs below confuse me a bit, because they do exactly the same thing :(
:)
Yeah I know... it was more for clarity, but I take your point, this is possibly not really adding much.
In the first version this had a file && file->... which made a lot more sense for this. But then I fixed things up such that e.g. file_has_valid_mmap_hooks() assumed file != NULL.
So, will drop these on respin.
Quoted 12 lines
include/linux/fs.h struct inode_operations { · line 2259
Quoted 10 lines
include/linux/fs.h +9
-0+ +/* Did the driver provide valid mmap hook configuration? */ +static inline bool file_has_valid_mmap_hooks (struct file *file) +{ +bool has_mmap =file_has_deprecated_mmap_hook (file); +bool has_mmap_prepare =file_has_mmap_prepare_hook (file); + +/* Hooks are mutually exclusive. */ +if (has_mmap && has_mmap_prepare)Should this be
WARN_ON_ONCE()?
Ack you're right, will update!
Quoted 11 lines
include/linux/fs.h struct inode_operations { · line 2266
Quoted 8 lines
include/linux/fs.h +7
-0+return false; + +/* But at least one must be specified. */ +if (!has_mmap && !has_mmap_prepare) +return false; + +return true;return
has_mmap||has_mmap_prepare;
Quoted 2 lines
And I think you can drop the comment about "at least one" with that, should be quite clear from that simplified version.
Ack, will change.
Quoted 9 lines
include/linux/fs.h struct inode_operations { · line 2272
Quoted 7 lines
include/linux/fs.h +4
-0+} +static inline int call_mmap (struct file *file,struct vm_area_struct *vma) { +/* If the driver specifies .mmap_prepare() this call is invalid. */ +if (file_has_mmap_prepare_hook (file))Should this be
WARN_ON_ONCE()?
Ack, will fix!
Quoted 66 lines
Quoted 11 lines
include/linux/fs.h +8
-0+return -EINVAL ; +return file->f_op->mmap (file, vma); } +static inline int __call_mmap_prepare (struct file *file, +struct vm_area_desc *desc) +{ +return file->f_op->mmap_prepare (desc); +} +[...]
Quoted 52 lines
struct file { struct address_space *f_mapping;include/linux/fs.h +34
-5+const struct file_operations *f_op; };#define VMA_LOCK_OFFSET 0x40000000@@ -1125,11 +1157,6 @@ static inline void vm_flags_clear(struct vm_area_struct *vma, vma->__vm_flags &= ~flags; } -static inline int call_mmap (struct file *,struct vm_area_struct *) -{ -return 0 ; -} -static inline int shmem_zero_setup (struct vm_area_struct *) {return 0 ;@@ -1405,4 +1432,46 @@ static inline void free_anon_vma_name(struct vm_area_struct *vma) (void )vma; } +static inline bool file_has_deprecated_mmap_hook (struct file *file) +{ +return file->f_op->mmap; +} + +static inline bool file_has_mmap_prepare_hook (struct file *file) +{ +return file->f_op->mmap_prepare; +} +> +/* Did the driver provide valid mmap hook configuration? */ +static inline bool file_has_valid_mmap_hooks(struct file *file) +{ +bool has_mmap =file_has_deprecated_mmap_hook (file); +bool has_mmap_prepare =file_has_mmap_prepare_hook (file); + +/* Hooks are mutually exclusive. */ +if (has_mmap && has_mmap_prepare) +return false; +> + /* But at least one must be specified. */ + if (!has_mmap && !has_mmap_prepare) +return false; + + return true;> +} + +static inline int call_mmap (struct file *file,struct vm_area_struct *vma) +{ +/* If the driver specifies .mmap_prepare() this call is invalid. */ +if (file_has_mmap_prepare_hook (file)) + return -EINVAL;> + + return file->f_op->mmap(file, vma); +} + +static inline int __call_mmap_prepare(struct file *file,
Quoted 8 lines
include/linux/fs.h struct inode_operations { · line 2282
Quoted 4 lines
include/linux/fs.h +4
-0+ struct vm_area_desc *desc) +{ +return file->f_op->mmap_prepare (desc); +}Hm, is there a way avoid a copy of the exact same code from
fs.h, and essentially test the implementation infs.h(-> more coverage by using less duplciated stubs?).
Not really, this kind of copying is sadly part of it because we're intentionally isolating vma.c from everything else, and if we try to bring in other headers they import yet others and etc. etc. it becomes a combinatorial explosion potentially.
We might be able to address with the tools/include stuff, but I think this is one to be addressed at a later time in some cleanup code there.
I am keen to avoid this kind of thing as obviously things can get out of sync (the VMA stuff tries _very hard_ to minimise this and only provide stubs that truly are stubbed out.
I will add to todo to improve this situation!
Quoted 6 lines
--
Cheers,David / dhildenb
Thanks for review! :) will respin a v2 shortly.
Re: [PATCH 1/3] mm: introduce new .mmap_prepare() file callback
Quoted 12 lines
Quoted 11 lines
include/linux/fs.h struct inode_operations { · line 2282
Quoted 7 lines
include/linux/fs.h +6
-0+ +static inline int __call_mmap_prepare (struct file *file, +struct vm_area_desc *desc) +{ +return file->f_op->mmap_prepare (desc); +}Hm, is there a way avoid a copy of the exact same code from
fs.h, and essentially test the implementation infs.h(-> more coverage by using less duplciated stubs?).
Quoted 4 lines
Not really, this kind of copying is sadly part of it because we're intentionally isolating
vma.cfrom everything else, and if we try to bring in other headers they import yet others and etc. etc. it becomes a combinatorial explosion potentially.
I guess what would work is inlining __call_mmap_prepare() -- again, rather simple wrapper ... and having file_has_valid_mmap_hooks() + call_mmap() reside in vma.c. Hm.
As an alternative, we'd really need some separate header that does not allow for any other includes, and is essentially only included in the other header files.
Duplicating functions in such a way that they can easily go out of sync and are not getting tested is really suboptimal. :(
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 1/3] mm: introduce new .mmap_prepare() file callback
Quoted 21 lines · David Hildenbrand
Quoted 12 lines
Quoted 11 lines
include/linux/fs.h struct inode_operations { · line 2282
Quoted 7 lines
include/linux/fs.h +6
-0+ +static inline int __call_mmap_prepare (struct file *file, +struct vm_area_desc *desc) +{ +return file->f_op->mmap_prepare (desc); +}Hm, is there a way avoid a copy of the exact same code from
fs.h, and essentially test the implementation infs.h(-> more coverage by using less duplciated stubs?).Quoted 4 lines
Not really, this kind of copying is sadly part of it because we're intentionally isolating
vma.cfrom everything else, and if we try to bring in other headers they import yet others and etc. etc. it becomes a combinatorial explosion potentially.I guess what would work is inlining
__call_mmap_prepare()-- again, rather simple wrapper ... and havingfile_has_valid_mmap_hooks()+call_mmap()reside invma.c. Hm.
Quoted 6 lines
As an alternative, we'd really need some separate header that does not allow for any other includes, and is essentially only included in the other header files.
Duplicating functions in such a way that they can easily go out of sync and are not getting tested is really suboptimal. :(
This is a problem that already exists, if minimised. Perfect is the enemy of good - if we had make these tests existence depend on being able to isolate _everything_ they'd never happen :)
But I will definitely try to improve the situation, as I couldn't agree more about de-syncing and it's a concern I share with you.
I think we have a bit of a mess of header files anyway like this, random helpers put in random places etc.
It doesn't help that a random driver/shm reference call_mmap()...
Anyway, this is somehwat out of scope for this series, as we already have a number of instances like this and this is just symptomatic of an existing problem rather than introducing it.
I think one thing to do might be to have a separate header which is explicitly for functions like these to at least absolutely highlight this case.
The VMA tests need restructuring anyway, so it can be part of a bigger project to do some work cleaning up there.
todo++; :>)
Quoted 6 lines
--
Cheers,David / dhildenb
Re: [PATCH 1/3] mm: introduce new .mmap_prepare() file callback
Quoted 33 lines · Lorenzo Stoakes
Quoted 21 lines · David Hildenbrand
Quoted 12 lines
Quoted 11 lines
include/linux/fs.h struct inode_operations { · line 2282
Quoted 7 lines
include/linux/fs.h +6
-0+ +static inline int __call_mmap_prepare (struct file *file, +struct vm_area_desc *desc) +{ +return file->f_op->mmap_prepare (desc); +}Hm, is there a way avoid a copy of the exact same code from
fs.h, and essentially test the implementation infs.h(-> more coverage by using less duplciated stubs?).Quoted 4 lines
Not really, this kind of copying is sadly part of it because we're intentionally isolating
vma.cfrom everything else, and if we try to bring in other headers they import yet others and etc. etc. it becomes a combinatorial explosion potentially.I guess what would work is inlining
__call_mmap_prepare()-- again, rather simple wrapper ... and havingfile_has_valid_mmap_hooks()+call_mmap()reside invma.c. Hm.Quoted 6 lines
As an alternative, we'd really need some separate header that does not allow for any other includes, and is essentially only included in the other header files.
Duplicating functions in such a way that they can easily go out of sync and are not getting tested is really suboptimal. :(
This is a problem that already exists, if minimised. Perfect is the enemy of good - if we had make these tests existence depend on being able to isolate _everything_ they'd never happen :)
Quoted 7 lines
But I will definitely try to improve the situation, as I couldn't agree more about de-syncing and it's a concern I share with you.
I think we have a bit of a mess of header files anyway like this, random helpers put in random places etc.
It doesn't help that a random driver/shm reference
call_mmap()...
Yes ...
Quoted 7 lines
Anyway, this is somehwat out of scope for this series, as we already have a number of instances like this and this is just symptomatic of an existing problem rather than introducing it.
I think one thing to do might be to have a separate header which is explicitly for functions like these to at least absolutely highlight this case.
Yes, and then just include it in the relevant header files.
Quoted 3 lines
The VMA tests need restructuring anyway, so it can be part of a bigger project to do some work cleaning up there.
Cool!
Signature · 3 lines
Cheers, David / dhildenb
PATCH 2/3 mm: secretmem: convert to .mmap_prepare() hook
In reply to Lorenzo StoakesSecretmem has a simple .mmap() hook which is easily converted to the new .mmap_prepare() callback.
Importantly, it's a rare instance of an driver that manipulates a VMA which is mergeable (that is, not a VM_SPECIAL mapping) while also adjusting VMA flags which may adjust mergeability, meaning the retry merge logic might impact whether or not the VMA is merged.
By using .mmap_prepare() there's no longer any need to retry the merge later as we can simply set the correct flags from the start.
This change therefore allows us to remove the retry merge logic in a subsequent commit.
--- mm/secretmem.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
mm/secretmem.c +7 -7
@@ -120,18 +120,18 @@ static int secretmem_release(struct inode *inode, struct file *file)return 0 ; } -static int secretmem_mmap (struct file *file,struct vm_area_struct *vma) +static int secretmem_mmap_prepare (struct vm_area_desc *desc) { -unsigned long len = vma->vm_end - vma->vm_start; +unsigned long len = desc->end - desc->start; -if ((vma->vm_flags & (VM_SHARED |VM_MAYSHARE )) ==0 ) +if ((desc->vm_flags & (VM_SHARED |VM_MAYSHARE )) ==0 )return -EINVAL ; -if (!mlock_future_ok (vma->vm_mm, vma->vm_flags |VM_LOCKED , len)) +if (!mlock_future_ok (desc->mm, desc->vm_flags |VM_LOCKED , len))return -EAGAIN ; -vm_flags_set (vma,VM_LOCKED |VM_DONTDUMP ); - vma->vm_ops = &secretmem_vm_ops; + desc->vm_flags |=VM_LOCKED |VM_DONTDUMP ; + desc->vm_ops = &secretmem_vm_ops;return 0 ; }
@@ -143,7 +143,7 @@ bool vma_is_secretmem(struct vm_area_struct *vma)static const struct file_operations secretmem_fops = { .release = secretmem_release, - .mmap = secretmem_mmap, + .mmap_prepare = secretmem_mmap_prepare, }; static int secretmem_migrate_folio(struct address_space *mapping,
Signature · 1 line
2.49.0
Re: [PATCH 2/3] mm: secretmem: convert to .mmap_prepare() hook
Quoted 25 lines · Lorenzo Stoakes
Secretmem has a simple .mmap() hook which is easily converted to the new .mmap_prepare() callback.
Importantly, it's a rare instance of an driver that manipulates a VMA which is mergeable (that is, not a
VM_SPECIALmapping) while also adjusting VMA flags which may adjust mergeability, meaning the retry merge logic might impact whether or not the VMA is merged.By using .mmap_prepare() there's no longer any need to retry the merge later as we can simply set the correct flags from the start.
This change therefore allows us to remove the retry merge logic in a subsequent commit.
Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/secretmem.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)mm/secretmem.c +0
-0
mm/secretmem.c static int secretmem_release(struct inode *inode, struct file *file) · line 125
Quoted 8 lines
mm/secretmem.c +2
-2@@ -120,18 +120,18 @@ static int secretmem_release(struct inode *inode, struct file *file)return 0 ; } -static int secretmem_mmap (struct file *file,struct vm_area_struct *vma) +static int secretmem_mmap_prepare (struct vm_area_desc *desc) { -unsigned long len = vma->vm_end - vma->vm_start; +unsigned long len = desc->end - desc->start;
I'd have marked that const while touching it.
mm/secretmem.c static int secretmem_release(struct inode *inode, struct file *file) · line 134
Quoted 13 lines
mm/secretmem.c +4
-4-if ((vma->vm_flags & (VM_SHARED |VM_MAYSHARE )) ==0 ) +if ((desc->vm_flags & (VM_SHARED |VM_MAYSHARE )) ==0 )return -EINVAL ; -if (!mlock_future_ok (vma->vm_mm, vma->vm_flags |VM_LOCKED , len)) +if (!mlock_future_ok (desc->mm, desc->vm_flags |VM_LOCKED , len))return -EAGAIN ; -vm_flags_set (vma,VM_LOCKED |VM_DONTDUMP ); - vma->vm_ops = &secretmem_vm_ops; + desc->vm_flags |=VM_LOCKED |VM_DONTDUMP ; + desc->vm_ops = &secretmem_vm_ops;
Yeah, that looks much better.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 2/3] mm: secretmem: convert to .mmap_prepare() hook
Quoted 21 lines · David Hildenbrand
Quoted 20 lines · Lorenzo Stoakes
Secretmem has a simple .mmap() hook which is easily converted to the new .mmap_prepare() callback.
Importantly, it's a rare instance of an driver that manipulates a VMA which is mergeable (that is, not a
VM_SPECIALmapping) while also adjusting VMA flags which may adjust mergeability, meaning the retry merge logic might impact whether or not the VMA is merged.By using .mmap_prepare() there's no longer any need to retry the merge later as we can simply set the correct flags from the start.
This change therefore allows us to remove the retry merge logic in a subsequent commit.
Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/secretmem.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
Quoted 14 lines
mm/secretmem.c static int secretmem_release(struct inode *inode, struct file *file) · line 124
Quoted 12 lines
mm/secretmem.c +2
-2@@ -120,18 +120,18 @@ static int secretmem_release(struct inode *inode, struct file *file)return 0 ; } -static int secretmem_mmap (struct file *file,struct vm_area_struct *vma) +static int secretmem_mmap_prepare (struct vm_area_desc *desc) { -unsigned long len = vma->vm_end - vma->vm_start; +unsigned long len = desc->end - desc->start;I'd have marked that const while touching it.
Will fix up on respin.
Quoted 14 lines
mm/secretmem.c static int secretmem_release(struct inode *inode, struct file *file) · line 134
Quoted 11 lines
mm/secretmem.c +4
-4-if ((vma->vm_flags & (VM_SHARED |VM_MAYSHARE )) ==0 ) +if ((desc->vm_flags & (VM_SHARED |VM_MAYSHARE )) ==0 )return -EINVAL ; -if (!mlock_future_ok (vma->vm_mm, vma->vm_flags |VM_LOCKED , len)) +if (!mlock_future_ok (desc->mm, desc->vm_flags |VM_LOCKED , len))return -EAGAIN ; -vm_flags_set (vma,VM_LOCKED |VM_DONTDUMP ); - vma->vm_ops = &secretmem_vm_ops; + desc->vm_flags |=VM_LOCKED |VM_DONTDUMP ; + desc->vm_ops = &secretmem_vm_ops;Yeah, that looks much better.
Quoted 1 line
Thanks!
Quoted 6 lines
--
Cheers,David / dhildenb
PATCH 3/3 mm/vma: remove mmap() retry merge
In reply to Lorenzo StoakesWe have now introduced a mechanism that obviates the need for a reattempted merge via the mmap_prepare() file hook, so eliminate this functionality altogether.
The retry merge logic has been the cause of a great deal of complexity in the past and required a great deal of careful manoeuvring of code to ensure its continued and correct functionality.
It has also recently been involved in an issue surrounding maple tree state, which again points to its problematic nature.
We make it much easier to reason about mmap() logic by eliminating this and simply writing a VMA once. This also opens the doors to future optimisation and improvement in the mmap() logic.
For any device or file system which encounters unwanted VMA fragmentation as a result of this change (that is, having not implemented .mmap_prepare hooks), the issue is easily resolvable by doing so.
--- mm/vma.c | 14 -------------- 1 file changed, 14 deletions(-)
mm/vma.c +0 -14
@@ -24,7 +24,6 @@ struct mmap_state {void *vm_private_data;unsigned long charged; -bool retry_merge;struct vm_area_struct *prev;struct vm_area_struct *next;
@@ -2417,8 +2416,6 @@ static int __mmap_new_file_vma(struct mmap_state *map, !(map->flags &VM_MAYWRITE ) && (vma->vm_flags & VM_MAYWRITE)); -/* If the flags change (and are mergeable), let's retry later. */ - map->retry_merge = vma->vm_flags != map->flags && !(vma->vm_flags &VM_SPECIAL ); map->flags = vma->vm_flags;return 0 ;
@@ -2624,17 +2621,6 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr,if (have_mmap_prepare)set_vma_user_defined_fields (vma, &map); -/* If flags changed, we might be able to merge, so try again. */ -if (map.retry_merge) { -struct vm_area_struct *merged; -VMG_MMAP_STATE (vmg, &map, vma); - -vma_iter_config (map.vmi, map.addr, map.end); - merged =vma_merge_existing_range (&vmg); -if (merged) - vma = merged; - } -__mmap_complete (&map, vma);return addr;
Signature · 1 line
2.49.0
Re: [PATCH 3/3] mm/vma: remove mmap() retry merge
Quoted 15 lines · Lorenzo Stoakes
We have now introduced a mechanism that obviates the need for a reattempted merge via the
mmap_prepare()file hook, so eliminate this functionality altogether.The retry merge logic has been the cause of a great deal of complexity in the past and required a great deal of careful manoeuvring of code to ensure its continued and correct functionality.
It has also recently been involved in an issue surrounding maple tree state, which again points to its problematic nature.
We make it much easier to reason about
mmap()logic by eliminating this and simply writing a VMA once. This also opens the doors to future optimisation and improvement in themmap()logic.
Quoted 5 lines
For any device or file system which encounters unwanted VMA fragmentation as a result of this change (that is, having not implemented .mmap_prepare hooks), the issue is easily resolvable by doing so.
Signed-off-by Lorenzo Stoakes <[email protected]>
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH 3/3] mm/vma: remove mmap() retry merge
Quoted 16 lines · David Hildenbrand
Quoted 15 lines · Lorenzo Stoakes
We have now introduced a mechanism that obviates the need for a reattempted merge via the
mmap_prepare()file hook, so eliminate this functionality altogether.The retry merge logic has been the cause of a great deal of complexity in the past and required a great deal of careful manoeuvring of code to ensure its continued and correct functionality.
It has also recently been involved in an issue surrounding maple tree state, which again points to its problematic nature.
We make it much easier to reason about
mmap()logic by eliminating this and simply writing a VMA once. This also opens the doors to future optimisation and improvement in themmap()logic.
Quoted 7 lines
Quoted 5 lines
For any device or file system which encounters unwanted VMA fragmentation as a result of this change (that is, having not implemented .mmap_prepare hooks), the issue is easily resolvable by doing so.
Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by David Hildenbrand <[email protected]>
Thanks!
Quoted 6 lines
--
Cheers,David / dhildenb