mm: make MAP_PRIVATE-/dev/zero mappings truly anonymous
Subsystems (10)
Lorenzo Stoakes (ARM)
maintainerv2 cover mm: make MAP_PRIVATE-/dev/zero mappings truly anonymous
Historically anonymous memory was obtained in linux by MAP_PRIVATE-mapping /dev/zero.
The canonical way of doing these now is mmap() specifying MAP_PRIVATE | MAP_ANON, but we must continue to support the legacy means of obtaining these mappings.
As-is these mappings are an unusual edge-case - they satisfy vma_is_anonymous() but have non-NULL vma->vm_file, and their page offset is the offset into the /dev/zero file.
Commit 93c0c8dc87f6 ("mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios") causes all other anonymous folios to be tracked by their anon index (vma->vm_start >> PAGE_SHIFT at the point of first fault), leaving MAP_PRIVATE-/dev/zero as the outlier.
This series remedies the situation by making MAP_PRIVATE-/dev/zero mappings truly anonymous with !vma->vm_file and correct anonymous page offset.
It starts by bringing the memory character driver into mm/ - this file implements /dev/zero, /dev/mem among other things and is already (as clearly indicated by its name) within the remit of memory management.
By doing this, the file_is_dev_zero() function can be provided, internal to mm, which allows for positive identification of these mappings.
Using this, first prevent any other mappings from mapping memory anonymously, then make these mappings truly anonymous and eliminate all code in the kernel that previously had to account for these strange beasts.
Finally, it adds userland VMA tests to assert the behaviour and selftests to assert expected merge behaviour.
v2:
* Added tags (thanks everybody!)
* Reworked comment about mappings not setting themselves anon as per David.
* Fixed up test typo as per David.
* Fixed up test close() as per David.
v1:
https://lore.kernel.org/r/[email protected]
---
Lorenzo Stoakes (ARM) (6):
mm: move drivers/char/mem.c to mm/char-mem.c
mm: implement file_is_dev_zero() to uniquely identify /dev/zero mm/vma: only permit MAP_PRIVATE /dev/zero to be mapped anonymous mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous tools/testing/vma: add test to assert MAP_PRIVATE-/dev/zero is anon tools/testing/selftests/mm: add MAP_PRIVATE-/dev/zero merge tests
MAINTAINERS | 4 +- drivers/char/Makefile | 2 +- include/linux/mm.h | 10 +-- include/linux/pagemap.h | 3 +- mm/Makefile | 3 +- drivers/char/mem.c => mm/char-mem.c | 21 +++-- mm/internal.h | 20 +++-- mm/shmem.c | 2 +- mm/vma.c | 39 +++++++-- mm/vma.h | 3 - tools/testing/selftests/mm/merge.c | 95 ++++++++++++++++++++++ .../selftests/proc/proc-self-map-files-001.c | 2 +- .../selftests/proc/proc-self-map-files-002.c | 2 +- tools/testing/vma/include/dup.h | 10 ++- tools/testing/vma/shared.c | 9 ++ tools/testing/vma/tests/mmap.c | 37 +++++++++ 16 files changed, 220 insertions(+), 42 deletions(-)
---
base-commit: e3b5239afe1b8f0194db7436b17c33e94c1988c4
change-id: 20260902-map-private-dev-zero-ba36d76a2fc8
Best regards,
Signature · 1 line
Lorenzo Stoakes (ARM) <[email protected]>
Lorenzo Stoakes (ARM)
maintainerPATCH v2 1/6 mm: move drivers/char/mem.c to mm/char-mem.c
The memory character driver implements several mm-specific features and is always compiled into the kernel, so move it to mm/ where it belongs.
Among other things the driver implements /dev/mem which provides raw access to physical memory, and /dev/zero which either allows mapping of a shmem region (if mapped with MAP_SHARED) or, uniquely, anonymous memory (if mapped MAP_PRIVATE).
This change lays the foundations to allow MAP_PRIVATE-/dev/zero to be mapped precisely the same as anonymous memory is mapped as currently it is an edge case within mm.
Also update a couple of comments that reference 'drivers/char/mem.c' to reference 'mm/char-mem.c'.
--- MAINTAINERS | 4 ++-- drivers/char/Makefile | 2 +- mm/Makefile | 3 ++- drivers/char/mem.c => mm/char-mem.c | 2 +- mm/shmem.c | 2 +- 5 files changed, 7 insertions(+), 6 deletions(-)
MAINTAINERS +2 -2
@@ -17272,7 +17272,7 @@ F: Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers F: Documentation/ABI/testing/sysfs-kernel-mm-numa F: Documentation/admin-guide/mm/ F: Documentation/mm/ -F: drivers/char/mem.c +F: mm/char-mem.c F: include/linux/cma.h F: include/linux/dmapool.h F: include/linux/ioremap.h
@@ -17477,7 +17477,7 @@ L: [email protected] S: Maintained W: http://www.linux-mm.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm -F: drivers/char/mem.c +F: mm/char-mem.c F: include/trace/events/mmap.h F: fs/proc/task_mmu.c F: fs/proc/task_nommu.c
drivers/char/Makefile +1 -1
@@ -3,7 +3,7 @@ # Makefile for the kernel character device drivers. # -obj-y += mem.o random.o +obj-y += random.o obj-$(CONFIG_TTY_PRINTK) += ttyprintk.o obj-y += misc.o obj-$(CONFIG_TEST_MISC_MINOR) += misc_minor_kunit.o
mm/Makefile +2 -1
@@ -55,7 +55,8 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ mm_init.o percpu.o slab_common.o \ compaction.o show_mem.o \ interval_tree.o list_lru.o workingset.o \ - debug.o gup.o mmap_lock.o vma_init.o $(mmu-y) + debug.o gup.o mmap_lock.o vma_init.o char-mem.o \ + $(mmu-y) # Give 'page_alloc' its own module-parameter namespace page-alloc-y := page_alloc.o
mm/char-mem.c renamed from drivers/char/mem.c +1 -1
@@ -1,6 +1,6 @@// SPDX-License-Identifier: GPL-2.0 /* -* linux/drivers/char/mem.c +* mm/char-mem.c * * Copyright (C) 1991, 1992 Linus Torvalds *
mm/shmem.c +1 -1
@@ -2981,7 +2981,7 @@ unsigned long shmem_get_unmapped_area(struct file *file, sb =file_inode (file)->i_sb; }else {/* -* Called directly from mm/mmap.c, or drivers/char/mem.c +* Called directly from mm/mmap.c, or mm/char-mem.c * for "/dev/zero", to create a shared anonymous object. */ if (IS_ERR (shm_mnt))
Signature · 1 line
2.55.0
Lorenzo Stoakes (ARM)
maintainerPATCH v2 2/6 mm: implement file_is_dev_zero() to uniquely identify /dev/zero
In reply to Lorenzo Stoakes (ARM)To lay the foundation for a future change that converts MAP_PRIVATE-/dev/zero mappings to be truly anonymous, add the ability to uniquely identify these mappings.
With the memory character device now part of mm/ this is trivially achievable through a file_is_dev_zero() predicate that simply tests that the file operation hooks are zero_fops.
Also update userland VMA tests to expose file_is_dev_zero() and provide a stub zero_fops for testing.
--- mm/char-mem.c | 13 +++++++++++++ mm/internal.h | 3 +++ tools/testing/vma/include/dup.h | 7 +++++++ tools/testing/vma/shared.c | 9 +++++++++ 4 files changed, 32 insertions(+)
mm/char-mem.c +13 -0
@@ -31,6 +31,8 @@#include <linux/uaccess.h> #include <linux/security.h> +#include "internal.h" +#define DEVMEM_MINOR 1#define DEVPORT_MINOR 4
@@ -707,6 +709,17 @@ static const struct memdev {#endif }; +/** +* file_is_dev_zero() - is the specified @file associated with the /dev/zero +* driver? +* @file: File to test. +* Returns: true if it is, false otherwise. +*/ +bool file_is_dev_zero (const struct file *file) +{ +return file && file->f_op == &zero_fops; +} +static int memory_open (struct inode *inode,struct file *filp) {int minor;
mm/internal.h +3 -0
@@ -1638,4 +1638,7 @@ static inline bool can_spin_trylock(void) return true; } +/* char-mem.c */ +bool file_is_dev_zero (const struct file *file); +#endif /* __MM_INTERNAL_H */
tools/testing/vma/include/dup.h +7 -0
@@ -1641,3 +1641,10 @@ static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,return pgoff; } + +extern const struct file_operations zero_fops; + +static inline bool file_is_dev_zero (const struct file *file) +{ +return file && file->f_op == &zero_fops; +}
tools/testing/vma/shared.c +9 -0
@@ -12,6 +12,15 @@ const struct vm_operations_struct vma_dummy_vm_ops;struct anon_vma dummy_anon_vma;struct task_struct __current; +static int mmap_zero_prepare (struct vm_area_desc *desc) +{ +return 0 ; +} + +const struct file_operations zero_fops = { + .mmap_prepare = mmap_zero_prepare, +}; +struct vm_area_struct *alloc_vma (struct mm_struct *mm,unsigned long start,unsigned long end,pgoff_t pgoff,vma_flags_t vma_flags)
Signature · 1 line
2.55.0
Lorenzo Stoakes (ARM)
maintainerPATCH v2 3/6 mm/vma: only permit MAP_PRIVATE /dev/zero to be mapped anonymous
In reply to Lorenzo Stoakes (ARM)In order to use mmap_prepare() with MAP_PRIVATE mappings of /dev/zero without the success_hook hack we explicitly permitted mmap_prepare handlers to set NULL vm_ops.
However this is dangerous and we really only want to allow this for MAP_PRIVATE-mapped /dev/zero.
Therefore use the newly introduced file_is_dev_zero() to uniquely identify MAP_PRIVATE-/dev/zero mappings and only permit this behaviour for them.
Then, remove all ability for mmap_prepare or mmap hooks to set a VMA anonymous and update mmap_zero_prepare() to leave it to the core mmap code to do so.
Note that this disallows nested MAP_PRIVATE-mappings of /dev/zero regions. Doing this would be broken in any case.
We therefore do not need to update the mmap_prepare() compatibility layer to reflect these changes, as the mmap hook check suffices to disallow this behaviour.
Now we're setting vma->vm_ops to NULL for an mmap_prepare-initialised MAP_PRIVATE-/dev/zero mapping, we have to avoid a subtle issue when updating user-defined fields via set_vma_user_defined_fields().
The default for vma->vm_ops for all mmap_prepare-initialised mappings is vma_dummy_vm_ops, so map->vm_ops will be set to this and setting vma->vm_ops to this will render the VMA mistakenly non-anon.
In general, we should never be setting user-defined fields for an anonymous VMA, so explicitly check for this to avoid doing so for the one case where a mapping can be both mmap_prepare and anonymous.
In the case of legacy ->mmap hooks some drivers may set vma->vm_ops NULL believing this is the equivalent of setting no VMA operations. Therefore update mmap_file() to correct this by setting dummy VMA operations if this occurs.
An example of this is drm_gem_shmem_mmap() which deliberately clears vma->vm_ops before handing the VMA to dma-buf. Cases such as this will be updated when they are converted to mmap_prepare.
Also, in order to avoid a single commit bisection hazard, add a temporary workaround to set the VMA anonymous only after vma->vm_file is assigned in __mmap_new_file_vma().
This is because vma_set_range() calls vma_set_pgoff() and assert_sane_pgoff() in turn, prior to the vma->vm_file being assigned. If we set the VMA anonymous early then this assert will fail.
This is removed in the subsequent commit.
--- mm/char-mem.c | 6 +----- mm/internal.h | 17 ++++++++++------- mm/vma.c | 33 +++++++++++++++++++++++++-------- 3 files changed, 36 insertions(+), 20 deletions(-)
mm/char-mem.c +1 -5
@@ -508,11 +508,7 @@ static int mmap_zero_prepare(struct vm_area_desc *desc)if (vma_desc_test (desc,VMA_SHARED_BIT ))return shmem_zero_setup_desc (desc); -/* -* This is a highly unique situation where we mark a MAP_PRIVATE mapping -* of /dev/zero anonymous, despite it not being. -*/ -vma_desc_set_anonymous (desc); +/* MAP_PRIVATE semantics are taken care of for us by core mm. */ return 0 ; }
mm/internal.h +10 -7
@@ -226,15 +226,18 @@ static inline int mmap_file(struct file *file, struct vm_area_struct *vma) {int err =vfs_mmap (file, vma); -if (likely (!err)) -return 0 ; -/* -* OK, we tried to call the file hook for mmap(), but an error -* arose. The mapping is in an inconsistent state and we must not invoke -* any further hooks on it. +* Either we tried to call the file hook for mmap() and an error arose +* or a driver set vma->vm_ops = NULL intending there to be no VMA +* operations. +* +* In the former case the VMA is in an inconsistent state and we mustn't +* invoke any further hooks on it, in the latter case the hook actually +* wanted no further hooks to be invoked, so fix both by setting dummy +* VMA ops. */ - vma->vm_ops = &vma_dummy_vm_ops; +if (unlikely (err || !vma->vm_ops)) + vma->vm_ops = &vma_dummy_vm_ops;return err; }
mm/vma.c +25 -8
@@ -2621,6 +2621,19 @@ static int __mmap_new_file_vma(struct mmap_state *map,return 0 ; } +static bool map_is_private (const struct mmap_state *map) +{ +return !vma_flags_test (&map->vma_flags,VMA_SHARED_BIT ); +} + +static bool map_is_anon (const struct mmap_state *map) +{ +if (!map_is_private (map)) +return false; + +return !map->file ||file_is_dev_zero (map->file); +} +/* * __mmap_new_vma() - Allocate a new VMA for the region, as merging was not * possible.
@@ -2634,8 +2647,7 @@ static int __mmap_new_file_vma(struct mmap_state *map,static int __mmap_new_vma (struct mmap_state *map,struct vm_area_struct **vmap,struct mmap_action *action) { -const bool is_anon = !map->file && - !vma_flags_test (&map->vma_flags,VMA_SHARED_BIT ); +const bool is_anon =map_is_anon (map);struct vma_iterator *vmi = map->vmi;int error =0 ;struct vm_area_struct *vma;
@@ -2651,7 +2663,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap,vma_iter_config (vmi, map->addr, map->end); -if (is_anon) +if (is_anon && !map->file)vma_set_anonymous (vma);vma_set_range (vma, map->addr, map->end, map->pgoff, map->anon_pgoff);
@@ -2669,6 +2681,10 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap,else if (!is_anon) error =shmem_zero_setup (vma); +/* Temporary MAP_PRIVATE-/dev/zero workaround. */ +if (is_anon && map->file) +vma_set_anonymous (vma); +if (error) goto free_iter_vma;
@@ -2777,6 +2793,10 @@ static int call_mmap_prepare(struct mmap_state *map,if (err)return err; +/* Hooks cannot mark themselves anonymous. */ +if (!desc->vm_ops) +return -EINVAL ; + err =call_action_prepare (map, desc);if (err)return err;
@@ -2799,10 +2819,7 @@ static int call_mmap_prepare(struct mmap_state *map,static void set_vma_user_defined_fields (struct vm_area_struct *vma,struct mmap_state *map) { -if (map->vm_ops) - vma->vm_ops = map->vm_ops; -else /* Only /dev/zero should do this. */ -vma_set_anonymous (vma); + vma->vm_ops = map->vm_ops; vma->vm_private_data = map->vm_private_data; }
@@ -2882,7 +2899,7 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr, allocated_new = true; } -if (have_mmap_prepare) +if (have_mmap_prepare && !map_is_anon (&map))set_vma_user_defined_fields (vma, &map);__mmap_complete (&map, vma);
Signature · 1 line
2.55.0
Re: [PATCH v2 3/6] mm/vma: only permit MAP_PRIVATE /dev/zero to be mapped anonymous
mm/vma.c static int call_mmap_prepare(struct mmap_state *map, · line 2796
Quoted 13 lines
mm/vma.c +5
-0+/* Temporary MAP_PRIVATE-/dev/zero workaround. */ +if (is_anon && map->file) +vma_set_anonymous (vma); +if (error) goto free_iter_vma;@@ -2777,6 +2793,10 @@ static int call_mmap_prepare(struct mmap_state *map,if (err)return err; +/* Hooks cannot mark themselves anonymous. */
Wasn't this supposed to be reworked? :)
In any case,
Signature · 3 lines
Cheers, David
Lorenzo Stoakes (ARM)
maintainerRe: [PATCH v2 3/6] mm/vma: only permit MAP_PRIVATE /dev/zero to be mapped anonymous
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 8 lines
mm/vma.c +4
-0+/* Temporary MAP_PRIVATE-/dev/zero workaround. */ +if (is_anon && map->file) +vma_set_anonymous (vma); +if (error) goto free_iter_vma;
Quoted 7 lines
mm/vma.c static int call_mmap_prepare(struct mmap_state *map, · line 2796
Quoted 5 lines
mm/vma.c +1
-0@@ -2777,6 +2793,10 @@ static int call_mmap_prepare(struct mmap_state *map,if (err)return err; +/* Hooks cannot mark themselves anonymous. */ Wasn't this supposed to be reworked? :)
It is, you yourself pointed out 4/6 makes this irrelevant, so I changed it there :)
Quoted 9 lines
In any case,
--
Cheers,David
--
Cheers, Lorenzo
David Hildenbrand (Arm)
maintainerRe: [PATCH v2 3/6] mm/vma: only permit MAP_PRIVATE /dev/zero to be mapped anonymous
Quoted 13 lines · Lorenzo Stoakes (ARM)
Quoted 12 lines · David Hildenbrand (Arm)
Quoted 12 lines
mm/vma.c +4
-0+/* Temporary MAP_PRIVATE-/dev/zero workaround. */ +if (is_anon && map->file) +vma_set_anonymous (vma); +if (error) goto free_iter_vma;@@ -2777,6 +2793,10 @@ static int call_mmap_prepare(struct mmap_state *map,if (err)return err;
Quoted 6 lines
Quoted 3 lines
mm/vma.c static int call_mmap_prepare(struct mmap_state *map, · line 2796
Quoted 1 line
mm/vma.c +1
-0+/* Hooks cannot mark themselves anonymous. */ Wasn't this supposed to be reworked? :)
It is, you yourself pointed out 4/6 makes this irrelevant, so I changed it there :)
Ah, thought you'd adjust that right away. Works for me, thanks.
Signature · 3 lines
Cheers, David
Lorenzo Stoakes (ARM)
maintainerPATCH v2 4/6 mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous
In reply to Lorenzo Stoakes (ARM)When mapping /dev/zero with MAP_PRIVATE, one ends up with strange VMAs originating from Linux's distant past.
These have vma->vm_file set but NULL vma->vm_ops, meaning they satisfy vma_is_anonymous() but otherwise resemble a file-backed VMA.
The introduction of anonymous page offsets and their subsequent use as indexes for MAP_PRIVATE-file-backed mappings mean the rmap does the right thing with these but we are left with inconsistencies.
The vma_start_pgoff(vma) == vma_start_anon_pgoff(vma) invariant is true for all other anonymous VMAs, but not these.
These VMAs are also observable as files in /proc/<pid>/[maps, smaps, map_files] but otherwise behave like anonymous mappings.
Therefore let's make these VMAs actually anonymous at mapping time which will activate the anonymous code path for mappings.
This means we no longer have to account for this discrepancy anywhere and no longer have to think about these at all.
This is user-observable, as MAP_PRIVATE-/dev/zero will no longer appear in procfs as a file-backed mapping, but the impact of this change should be low as likely nobody is relying upon this.
However in any case, in using MAP_PRIVATE-/dev/zero they are explicitly asking anonymous memory, so no longer seeing these as file mappings is in fact correct.
A previous commit gave us file_is_dev_zero() to positively identify these mappings, so we expressly only do so for these alone.
Update assert_sane_pgoff(), the comment for vma_start_pgoff() and linear_anon_page_index() to reflect the change.
We make this change in call_mmap_prepare() alone as /dev/zero has been converted to an mmap_prepare hook and we do not permit nested MAP_PRIVATE mapping of /dev/zero.
We also remove the now defunct vma_desc_set_anonymous() and eliminate the temporary bisection hazard fix from the previous commit.
Also update the VMA userland tests to reflect the change.
Finally, update the procfs self tests proc-self-map-files-001 and proc-self-map-files-002 which both intend to map an arbitrary file MAP_PRIVATE then assert procfs state, but happen to choose /dev/zero.
Fix them by updating these to /proc/self/exe which is guaranteed to be present if procfs is mounted.
--- include/linux/mm.h | 10 ++------ include/linux/pagemap.h | 3 +-- mm/vma.c | 28 ++++++++++++++-------- mm/vma.h | 3 --- .../selftests/proc/proc-self-map-files-001.c | 2 +- .../selftests/proc/proc-self-map-files-002.c | 2 +- tools/testing/vma/include/dup.h | 3 +-- 7 files changed, 24 insertions(+), 27 deletions(-)
include/linux/mm.h +2 -8
@@ -1539,11 +1539,6 @@ static inline void vma_set_anonymous(struct vm_area_struct *vma) vma->vm_ops =NULL ; } -static inline void vma_desc_set_anonymous (struct vm_area_desc *desc) -{ - desc->vm_ops =NULL ; -} -static inline bool vma_is_anonymous (const struct vm_area_struct *vma) {return !vma->vm_ops;
@@ -4405,9 +4400,8 @@ static inline unsigned long vma_pages(const struct vm_area_struct *vma)* If @vma is a MAP_PRIVATE file-backed mapping, then this returns the * page offset within the file. * -* Edge cases: nommu does not abide by these, MAP_PRIVATE-/dev/zero satisfies -* vma_is_anonymous() but has file-backed page offset, and MAP_PRIVATE-pfnmap -* regions have their page offset set to the first PFN in the range. +* Edge cases: nommu does not abide by these and CoW MAP_PRIVATE-pfnmap regions +* have their page offset set to the first PFN in the range. * * Returns: The page offset of the start of @vma. */
include/linux/pagemap.h +1 -2
@@ -1128,8 +1128,7 @@ static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,const pgoff_t pgoff =__linear_anon_page_index (vma, address);VM_WARN_ON_ONCE (!vma_is_cow_mapping (vma)); -/* Account for MAP_PRIVATE-/dev/zero which is only semi-anonymous. */ -if (vma_is_anonymous (vma) && !vma->vm_file) +if (vma_is_anonymous (vma))VM_WARN_ON_ONCE (pgoff !=linear_page_index (vma, address));return pgoff;
mm/vma.c +18 -10
@@ -2621,6 +2621,13 @@ static int __mmap_new_file_vma(struct mmap_state *map,return 0 ; } +static void map_set_anon (struct mmap_state *map) +{ + map->file =NULL ; + map->vm_ops =NULL ; + map->pgoff = map->addr >>PAGE_SHIFT ; +} +static bool map_is_private (const struct mmap_state *map) {return !vma_flags_test (&map->vma_flags,VMA_SHARED_BIT );
@@ -2628,10 +2635,7 @@ static bool map_is_private(const struct mmap_state *map)static bool map_is_anon (const struct mmap_state *map) { -if (!map_is_private (map)) -return false; - -return !map->file ||file_is_dev_zero (map->file); +return map_is_private (map) && !map->file; }/*
@@ -2663,7 +2667,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap,vma_iter_config (vmi, map->addr, map->end); -if (is_anon && !map->file) +if (is_anon)vma_set_anonymous (vma);vma_set_range (vma, map->addr, map->end, map->pgoff, map->anon_pgoff);
@@ -2681,10 +2685,6 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap,else if (!is_anon) error =shmem_zero_setup (vma); -/* Temporary MAP_PRIVATE-/dev/zero workaround. */ -if (is_anon && map->file) -vma_set_anonymous (vma); -if (error) goto free_iter_vma;
@@ -2793,7 +2793,7 @@ static int call_mmap_prepare(struct mmap_state *map,if (err)return err; -/* Hooks cannot mark themselves anonymous. */ +/* It's invalid for mmap_preprare hooks to clear vm_ops. */ if (!desc->vm_ops)return -EINVAL ;
@@ -2813,6 +2813,14 @@ static int call_mmap_prepare(struct mmap_state *map, map->vm_ops = desc->vm_ops; map->vm_private_data = desc->private_data; +/* +* MAP_PRIVATE-/dev/zero mappings are an ancient way of getting +* anonymous mappings. Rather than allowing these mappings to be odd +* outliers, simply make them truly anonymous. +*/ +if (map_is_private (map) &&file_is_dev_zero (map->file)) +map_set_anon (map); +return 0 ; }
mm/vma.h +0 -3
@@ -267,9 +267,6 @@ static inline void assert_sane_pgoff(struct vm_area_struct *vma, pgoff_t pgoff)*/ if (!vma_is_anonymous (vma))return ; -/* MAP_PRIVATE-/dev/zero is anon, non-NULL vm_file, but has file pgoff. */ -if (vma->vm_file) -return ;/* If faulted in, could have been remapped. */ if (vma->anon_vma)return ;
tools/testing/selftests/proc/proc-self-map-files-001.c +1 -1
@@ -51,7 +51,7 @@ int main(void)int fd;unsigned long a, b; - fd =open ("/dev/zero" ,O_RDONLY ); + fd =open ("/proc/self/exe" ,O_RDONLY );if (fd ==-1 )return 1 ;
tools/testing/selftests/proc/proc-self-map-files-002.c +1 -1
@@ -57,7 +57,7 @@ int main(void)int fd;unsigned long a, b; - fd =open ("/dev/zero" ,O_RDONLY ); + fd =open ("/proc/self/exe" ,O_RDONLY );if (fd ==-1 )return 1 ;
tools/testing/vma/include/dup.h +1 -2
@@ -1635,8 +1635,7 @@ static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,const pgoff_t pgoff =__linear_anon_page_index (vma, address);VM_WARN_ON_ONCE (!vma_is_cow_mapping (vma)); -/* Account for MAP_PRIVATE-/dev/zero which is only semi-anonymous. */ -if (vma_is_anonymous (vma) && !vma->vm_file) +if (vma_is_anonymous (vma))VM_WARN_ON_ONCE (pgoff !=linear_page_index (vma, address));return pgoff;
Signature · 1 line
2.55.0
Re: [PATCH v2 4/6] mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous
Quoted 49 lines · Lorenzo Stoakes (ARM)
When mapping
/dev/zerowithMAP_PRIVATE, one ends up with strange VMAs originating from Linux's distant past.These have
vma->vm_fileset but NULLvma->vm_ops, meaning they satisfyvma_is_anonymous()but otherwise resemble a file-backed VMA.The introduction of anonymous page offsets and their subsequent use as indexes for MAP_PRIVATE-file-backed mappings mean the rmap does the right thing with these but we are left with inconsistencies.
The
vma_start_pgoff(vma)==vma_start_anon_pgoff(vma)invariant is true for all other anonymous VMAs, but not these.These VMAs are also observable as files in /proc/<pid>/[maps, smaps,
map_files] but otherwise behave like anonymous mappings.Therefore let's make these VMAs actually anonymous at mapping time which will activate the anonymous code path for mappings.
This means we no longer have to account for this discrepancy anywhere and no longer have to think about these at all.
This is user-observable, as MAP_PRIVATE-/dev/zero will no longer appear in procfs as a file-backed mapping, but the impact of this change should be low as likely nobody is relying upon this.
However in any case, in using MAP_PRIVATE-/dev/zero they are explicitly asking anonymous memory, so no longer seeing these as file mappings is in fact correct.
A previous commit gave us
file_is_dev_zero()to positively identify these mappings, so we expressly only do so for these alone.Update
assert_sane_pgoff(), the comment forvma_start_pgoff()andlinear_anon_page_index()to reflect the change.We make this change in
call_mmap_prepare()alone as/dev/zerohas been converted to anmmap_preparehook and we do not permit nestedMAP_PRIVATEmapping of/dev/zero.We also remove the now defunct
vma_desc_set_anonymous()and eliminate the temporary bisection hazard fix from the previous commit.Also update the VMA userland tests to reflect the change.
Finally, update the procfs self tests proc-self-map-files-001 and proc-self-map-files-002 which both intend to map an arbitrary file
MAP_PRIVATEthen assert procfs state, but happen to choose/dev/zero.
Quoted 5 lines
Fix them by updating these to
/proc/self/exewhich is guaranteed to be present if procfs is mounted.Signed-off-by Lorenzo Stoakes (ARM) <[email protected]>---
Signature · 3 lines
Cheers, David
Lorenzo Stoakes (ARM)
maintainerPATCH v2 5/6 tools/testing/vma: add test to assert MAP_PRIVATE-/dev/zero is anon
In reply to Lorenzo Stoakes (ARM)Now we've made MAP_PRIVATE-mapped /dev/zero mappings truly anonymous, add a VMA userland test to assert that this is the case and everything is as we would expect for an anonymous mapping.
--- tools/testing/vma/tests/mmap.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
tools/testing/vma/tests/mmap.c +37 -0
@@ -45,7 +45,44 @@ static bool test_mmap_region_basic(void)return true; } +static bool test_pure_anon_dev_zero (void ) +{ +const vma_flags_t vma_flags =mk_vma_flags (VMA_READ_BIT ,VMA_WRITE_BIT , +VMA_MAYREAD_BIT ,VMA_MAYWRITE_BIT ); +struct file file = { + .f_op = &zero_fops, + }; +struct mm_struct mm = {}; +struct vm_area_struct *vma; +unsigned long addr; +VMA_ITERATOR (vmi, &mm,0 ); + + current->mm = &mm; + +/* +* Map a MAP_PRIVATE-/dev/zero mapping at address 0x300000 with a page +* offset of 0x10, which we expect to be reset to the anonymous page +* offset. +*/ + addr =__mmap_region (&file,0x300000 ,0x3000 , vma_flags,0x10 ,NULL ); +ASSERT_EQ (addr,0x300000 ); + +/* Assert that it truly is an anonymous mapping. */ + vma =vma_lookup (&mm, addr); +ASSERT_NE (vma,NULL ); +ASSERT_TRUE (vma_is_anonymous (vma)); +ASSERT_EQ (vma->vm_file,NULL ); +ASSERT_EQ (vma->vm_private_data,NULL ); +/* Expect anonymous page offsets. */ +ASSERT_EQ (vma->vm_pgoff,0x300 ); +ASSERT_EQ (vma_start_anon_pgoff (vma),0x300 ); + +cleanup_mm (&mm, &vmi); +return true; +} +static void run_mmap_tests (int *num_tests,int *num_fail) {TEST (mmap_region_basic); +TEST (pure_anon_dev_zero); }
Signature · 1 line
2.55.0
Lorenzo Stoakes (ARM)
maintainerPATCH v2 6/6 tools/testing/selftests/mm: add MAP_PRIVATE-/dev/zero merge tests
In reply to Lorenzo Stoakes (ARM)Assert that MAP_PRIVATE-mapped /dev/zero mappings behave like they are anonymous.
Test both unfaulted and faulted/unfaulted merges with page offset 0 which would not merge if the mappings were treated as if they were file-backed.
With the recent change that makes them behave as pure anonymous mappings, the merges should succeed as their page offsets are equal to their anonymous page offsets.
--- tools/testing/selftests/mm/merge.c | 95 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)
tools/testing/selftests/mm/merge.c +95 -0
@@ -1362,6 +1362,101 @@ TEST_F(merge, anon_and_page_offset_mismatch_memfd)ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +5 * page_size); } +TEST_F(merge, merge_map_private_dev_zero_unfaulted) +{ +struct procmap_fd *procmap = &self->procmap; +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +char *ptr, *ptr2; +int fd_zero; + +if (access ("/dev/zero" ,F_OK )) +SKIP (return,"No /dev/zero." ); + fd_zero =open ("/dev/zero" ,O_RDWR ); +ASSERT_NE (fd_zero,-1 ); + +/* +* Map two MAP_PRIVATE-/dev/zero VMAs next to one another with offset 0 +* each. +* +* With these being made truly anonymous upon mapping, they will +* merge. If they were file-backed VMAs the page offsets would prevent +* the merge: +* +* |-----||------| |-------------| +* | ptr || ptr2 | -> | ptr | +* |-----||------| |-------------| +*/ + ptr =mmap (carveout,5 * page_size,PROT_READ |PROT_WRITE , +MAP_FIXED |MAP_PRIVATE , fd_zero,0 ); + ptr2 =mmap (&carveout[5 * page_size],5 * page_size, +PROT_READ |PROT_WRITE ,MAP_FIXED |MAP_PRIVATE , fd_zero,0 ); +close (fd_zero); +ASSERT_NE (ptr,MAP_FAILED ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* Assert that they merged. */ +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +10 * page_size); +} + +TEST_F(merge, merge_map_private_dev_zero_faulted_unfaulted) +{ +struct procmap_fd *procmap = &self->procmap; +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +char *ptr, *ptr2; +int fd_zero; + +if (access ("/dev/zero" ,F_OK )) +SKIP (return,"No /dev/zero." ); + fd_zero =open ("/dev/zero" ,O_RDWR ); +ASSERT_NE (fd_zero,-1 ); + +/* +* Map a MAP_PRIVATE mapping of /dev/zero with page offset 0, then fault +* it in: +* +* |-------------------------------| +* | faulted | +* |-------------------------------| +*/ + ptr =mmap (carveout,15 * page_size,PROT_READ |PROT_WRITE , +MAP_FIXED |MAP_PRIVATE , fd_zero,0 ); +ASSERT_NE (ptr,MAP_FAILED ); +memset (ptr,'x' ,15 * page_size); + +/* +* Unmap the middle: +* +* |---------| |---------| +* | faulted | | faulted | +* |---------| |---------| +*/ +ASSERT_EQ (munmap (&ptr[5 * page_size],5 * page_size),0 ); + +/* +* Map in a new unfaulted mapping in the middle with page offset 0 - +* this should merge and would not if it were treated as a file rather +* than pure anon: +* +* |---------|-----------|---------| +* | faulted | unfaulted | faulted | +* |---------|-----------|---------| +*/ + ptr2 =mmap (&carveout[5 * page_size],5 * page_size, +PROT_READ |PROT_WRITE ,MAP_FIXED |MAP_PRIVATE , + fd_zero,0 ); +close (fd_zero); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* Assert that they merged. */ +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); +} + TEST_F(merge_with_fork, mremap_faulted_to_unfaulted_prev) {struct procmap_fd *procmap = &self->procmap;
Signature · 1 line
2.55.0
Re: [PATCH v2 6/6] tools/testing/selftests/mm: add MAP_PRIVATE-/dev/zero merge tests
Quoted 6 lines · Lorenzo Stoakes (ARM)
Assert that MAP_PRIVATE-mapped
/dev/zeromappings behave like they are anonymous.Test both unfaulted and faulted/unfaulted merges with page offset 0 which would not merge if the mappings were treated as if they were file-backed.
Quoted 6 lines
With the recent change that makes them behave as pure anonymous mappings, the merges should succeed as their page offsets are equal to their anonymous page offsets.
Signed-off-by Lorenzo Stoakes (ARM) <[email protected]>---
Signature · 3 lines
Cheers, David
Andrew Morton
maintainerQuoted 26 lines · On Tue, 08 Sep 2026 12:23:37 +0100 "Lorenzo Stoakes (ARM)" <[email protected]>…
Historically anonymous memory was obtained in linux by MAP_PRIVATE-mapping
/dev/zero.The canonical way of doing these now is
mmap()specifyingMAP_PRIVATE|MAP_ANON, but we must continue to support the legacy means of obtaining these mappings.As-is these mappings are an unusual edge-case - they satisfy
vma_is_anonymous()but have non-NULLvma->vm_file, and their page offset is the offset into the/dev/zerofile.Commit
93c0c8dc87f6("mm/rmap: use anon pgoff to trackMAP_PRIVATEfile-backed anon folios") causes all other anonymous folios to be tracked by their anon index (vma->vm_start>>PAGE_SHIFTat the point of first fault), leaving MAP_PRIVATE-/dev/zero as the outlier.This series remedies the situation by making MAP_PRIVATE-/dev/zero mappings truly anonymous with !vma->vm_file and correct anonymous page offset.
It starts by bringing the memory character driver into
mm/- this file implements/dev/zero,/dev/memamong other things and is already (as clearly indicated by its name) within the remit of memory management.By doing this, the
file_is_dev_zero()function can be provided, internal to mm, which allows for positive identification of these mappings.
Quoted 6 lines
Using this, first prevent any other mappings from mapping memory anonymously, then make these mappings truly anonymous and eliminate all code in the kernel that previously had to account for these strange beasts.
Finally, it adds userland VMA tests to assert the behaviour and selftests to assert expected merge behaviour.
Thanks, I've updated mm-unstable to this version.
Quoted 5 lines
v2:
* Added tags (thanks everybody!)
* Reworked comment about mappings not setting themselves anon as per David.
* Fixed up test typo as per David.
* Fixed up testclose()as per David.
Here's how v2 altered mm.git:
mm/vma.c renamed from mm/vma.c~b +1 -1
@@ -2801,7 +2801,7 @@ static int call_mmap_prepare(struct mmapif (err)return err; -/* Hooks cannot mark themselves anonymous. */ +/* It's invalid for mmap_preprare hooks to clear vm_ops. */ if (!desc->vm_ops)return -EINVAL ;
tools/testing/selftests/mm/merge.c renamed from tools/testing/selftests/mm/merge.c~b +4 -13
@@ -1381,7 +1381,7 @@ TEST_F(merge, merge_map_private_dev_zero* * With these being made truly anonymous upon mapping, they will * merge. If they were file-backed VMAs the page offsets would prevent -* merge: +* the merge: * * |-----||------| |-------------| * | ptr || ptr2 | -> | ptr |
@@ -1389,17 +1389,11 @@ TEST_F(merge, merge_map_private_dev_zero*/ ptr =mmap (carveout,5 * page_size,PROT_READ |PROT_WRITE ,MAP_FIXED |MAP_PRIVATE , fd_zero,0 ); -if (ptr ==MAP_FAILED ) { -close (fd_zero); -ASSERT_TRUE (false); - } ptr2 =mmap (&carveout[5 * page_size],5 * page_size,PROT_READ |PROT_WRITE ,MAP_FIXED |MAP_PRIVATE , fd_zero,0 ); -if (ptr2 ==MAP_FAILED ) { -close (fd_zero); -ASSERT_TRUE (false); - }close (fd_zero); +ASSERT_NE (ptr,MAP_FAILED ); +ASSERT_NE (ptr2,MAP_FAILED );/* Assert that they merged. */ ASSERT_TRUE (find_vma_procmap (procmap, ptr));
@@ -1430,10 +1424,7 @@ TEST_F(merge, merge_map_private_dev_zero*/ ptr =mmap (carveout,15 * page_size,PROT_READ |PROT_WRITE ,MAP_FIXED |MAP_PRIVATE , fd_zero,0 ); -if (ptr ==MAP_FAILED ) { -close (fd_zero); -ASSERT_TRUE (false); - } +ASSERT_NE (ptr,MAP_FAILED );memset (ptr,'x' ,15 * page_size);/*
_