mm: improve map count checks
Subsystems (3)
Lorenzo Stoakes (Oracle)
maintainercover mm: improve map count checks
Firstly, in mremap(), it appears that our map count checks have been overly conservative - there is simply no reason to require that we have headroom of 4 mappings prior to moving the VMA, we only need headroom of 2 VMAs since commit 659ace584e7a ("mmap: don't return ENOMEM when mapcount is temporarily exceeded in munmap()").
Likely the original headroom of 4 mappings was a mistake, and 3 was actually intended.
Next, we access sysctl_max_map_count in a number of places without being all that careful about how we do so.
We introduces a simple helper that READ_ONCE()'s the field (get_sysctl_max_map_count()) to ensure that the field is accessed correctly. The WRITE_ONCE() side is already handled by the sysctl procfs code in proc_int_conv().
We also move this field to internal.h as there's no reason for anybody else to access it outside of mm. Unfortunately we have to maintain the extern variable, as mmap.c implements the procfs code.
Finally, we are accessing current->mm->map_count without holding the mmap write lock, which is also not correct, so this series ensures the lock is head before we access it.
We also abstract the check to a helper function, and add ASCII diagrams to explain why we're doing what we're doing.
Lorenzo Stoakes (Oracle) (3):
mm/mremap: correct invalid map count check
mm: abstract reading sysctl_max_map_count, and READ_ONCE() mm/mremap: check map count under mmap write lock and abstract
include/linux/mm.h | 2 - mm/internal.h | 6 ++ mm/mmap.c | 2 +- mm/mremap.c | 98 ++++++++++++++++++++++++------ mm/nommu.c | 2 +- mm/vma.c | 6 +- tools/testing/vma/include/custom.h | 3 - tools/testing/vma/include/dup.h | 9 +++ tools/testing/vma/main.c | 2 + 9 files changed, 100 insertions(+), 30 deletions(-)
--
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH 1/3 mm/mremap: correct invalid map count check
We currently check to see, if on moving a VMA when doing mremap(), if it might violate the sys.vm.max_map_count limit.
This was introduced in the mists of time prior to 2.6.12.
At this point in time, as now, the move_vma() operation would copy the VMA (+1 mapping if not merged), then potentially split the source VMA upon unmap.
Prior to commit 659ace584e7a ("mmap: don't return ENOMEM when mapcount is temporarily exceeded in munmap()"), a VMA split would check whether mm->map_count >= sysctl_max_map_count prior to a split before it ran.
On unmap of the source VMA, if we are moving a partial VMA, we might split the VMA twice.
This would mean, on invocation of split_vma() (as was), we'd check whether mm->map_count >= sysctl_max_map_count with a map count elevated by one, then again with a map count elevated by two, ending up with a map count elevated by three.
At this point we'd reduce the map count on unmap.
At the start of move_vma(), there was a check that has remained throughout mremap()'s history of mm->map_count >= sysctl_max_map_count - 3 (which implies mm->mmap_count + 4 > sysctl_max_map_count - that is, we must have headroom for 4 additional mappings).
After mm->map_count is elevated by 3, it is decremented by one once the unmap completes. The mmap write lock is held, so nothing else will observe mm->map_count > sysctl_max_map_count.
It appears this check was always incorrect - it should have either be one of 'mm->map_count > sysctl_max_map_count - 3' or 'mm->map_count >= sysctl_max_map_count - 2'.
After commit 659ace584e7a ("mmap: don't return ENOMEM when mapcount is temporarily exceeded in munmap()"), the map count check on split is eliminated in the newly introduced __split_vma(), which the unmap path uses, and has that path check whether mm->map_count >= sysctl_max_map_count.
This is valid since, net, an unmap can only cause an increase in map count of 1 (split both sides, unmap middle).
Since we only copy a VMA and (if MREMAP_DONTUNMAP is not set) unmap afterwards, the maximum number of additional mappings that will actually be subject to any check will be 2.
Therefore, update the check to assert this corrected value. Additionally, update the check introduced by commit ea2c3f6f5545 ("mm,mremap: bail out earlier in mremap_to under map pressure") to account for this.
While we're here, clean up the comment prior to that.
--- mm/mremap.c | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-)
mm/mremap.c +12 -16
@@ -1041,10 +1041,11 @@ static unsigned long prep_move_vma(struct vma_remap_struct *vrm)vm_flags_t dummy = vma->vm_flags;/* -* We'd prefer to avoid failure later on in do_munmap: -* which may split one vma into three before unmapping. +* We'd prefer to avoid failure later on in do_munmap: we copy a VMA, +* which may not merge, then (if MREMAP_DONTUNMAP is not set) unmap the +* source, which may split, causing a net increase of 2 mappings. */ -if (current->mm->map_count >= sysctl_max_map_count -3 ) +if (current->mm->map_count +2 > sysctl_max_map_count)return -ENOMEM ;if (vma->vm_ops && vma->vm_ops->may_split) {
@@ -1804,20 +1805,15 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm)return -EINVAL ;/* -* move_vma() need us to stay 4 maps below the threshold, otherwise -* it will bail out at the very beginning. -* That is a problem if we have already unmapped the regions here -* (new_addr, and old_addr), because userspace will not know the -* state of the vma's after it gets -ENOMEM. -* So, to avoid such scenario we can pre-compute if the whole -* operation has high chances to success map-wise. -* Worst-scenario case is when both vma's (new_addr and old_addr) get -* split in 3 before unmapping it. -* That means 2 more maps (1 for each) to the ones we already hold. -* Check whether current map count plus 2 still leads us to 4 maps below -* the threshold, otherwise return -ENOMEM here to be more safe. +* We may unmap twice before invoking move_vma(), that is if new_len < +* old_len (shrinking), and in the MREMAP_FIXED case, unmapping part of +* a VMA located at the destination. +* +* In the worst case, both unmappings will cause splits, resulting in a +* net increased map count of 2. In move_vma() we check for headroom of +* 2 additional mappings, so check early to avoid bailing out then. */ -if ((current->mm->map_count +2 ) >= sysctl_max_map_count -3 ) +if (current->mm->map_count +4 > sysctl_max_map_count)return -ENOMEM ;return 0 ;
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH 2/3 mm: abstract reading sysctl_max_map_count, and READ_ONCE()
In reply to Lorenzo Stoakes (Oracle)Concurrent reads and writes of sysctl_max_map_count are possible, so we should READ_ONCE() and WRITE_ONCE().
The sysctl procfs logic already enforces WRITE_ONCE(), so abstract the read side with get_sysctl_max_map_count().
While we're here, also move the field to mm/internal.h and add the getter there since only mm interacts with it, there's no need for anybody else to have access.
Finally, update the VMA userland tests to reflect the change.
--- include/linux/mm.h | 2 -- mm/internal.h | 6 ++++++ mm/mmap.c | 2 +- mm/mremap.c | 4 ++-- mm/nommu.c | 2 +- mm/vma.c | 6 +++--- tools/testing/vma/include/custom.h | 3 --- tools/testing/vma/include/dup.h | 9 +++++++++ tools/testing/vma/main.c | 2 ++ 9 files changed, 24 insertions(+), 12 deletions(-)
include/linux/mm.h +0 -2
@@ -207,8 +207,6 @@ static inline void __mm_zero_struct_page(struct page *page)#define MAPCOUNT_ELF_CORE_MARGIN (5)#define DEFAULT_MAX_MAP_COUNT (USHRT_MAX - MAPCOUNT_ELF_CORE_MARGIN) -extern int sysctl_max_map_count; -extern unsigned long sysctl_user_reserve_kbytes;extern unsigned long sysctl_admin_reserve_kbytes;
mm/internal.h +6 -0
@@ -1848,4 +1848,10 @@ static inline int pmdp_test_and_clear_young_notify(struct vm_area_struct *vma,#endif /* CONFIG_MMU_NOTIFIER */ +extern int sysctl_max_map_count; +static inline int get_sysctl_max_map_count (void ) +{ +return READ_ONCE (sysctl_max_map_count); +} +#endif /* __MM_INTERNAL_H */
mm/mmap.c +1 -1
@@ -375,7 +375,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr,return -EOVERFLOW ;/* Too many mappings? */ -if (mm->map_count > sysctl_max_map_count) +if (mm->map_count >get_sysctl_max_map_count ())return -ENOMEM ;/*
mm/mremap.c +2 -2
@@ -1045,7 +1045,7 @@ static unsigned long prep_move_vma(struct vma_remap_struct *vrm)* which may not merge, then (if MREMAP_DONTUNMAP is not set) unmap the * source, which may split, causing a net increase of 2 mappings. */ -if (current->mm->map_count +2 > sysctl_max_map_count) +if (current->mm->map_count +2 >get_sysctl_max_map_count ())return -ENOMEM ;if (vma->vm_ops && vma->vm_ops->may_split) {
@@ -1813,7 +1813,7 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm)* net increased map count of 2. In move_vma() we check for headroom of * 2 additional mappings, so check early to avoid bailing out then. */ -if (current->mm->map_count +4 > sysctl_max_map_count) +if (current->mm->map_count +4 >get_sysctl_max_map_count ())return -ENOMEM ;return 0 ;
mm/nommu.c +1 -1
@@ -1317,7 +1317,7 @@ static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,return -ENOMEM ; mm = vma->vm_mm; -if (mm->map_count >= sysctl_max_map_count) +if (mm->map_count >=get_sysctl_max_map_count ())return -ENOMEM ; region =kmem_cache_alloc (vm_region_jar,GFP_KERNEL );
mm/vma.c +3 -3
@@ -590,7 +590,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,static int split_vma (struct vma_iterator *vmi,struct vm_area_struct *vma,unsigned long addr,int new_below) { -if (vma->vm_mm->map_count >= sysctl_max_map_count) +if (vma->vm_mm->map_count >=get_sysctl_max_map_count ())return -ENOMEM ;return __split_vma (vmi, vma, addr, new_below);
@@ -1394,7 +1394,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms,* its limit temporarily, to help free resources as expected. */ if (vms->end < vms->vma->vm_end && - vms->vma->vm_mm->map_count >= sysctl_max_map_count) { + vms->vma->vm_mm->map_count >=get_sysctl_max_map_count ()) { error = -ENOMEM ; goto map_count_exceeded; }
@@ -2870,7 +2870,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,if (!may_expand_vm (mm, vm_flags, len >>PAGE_SHIFT ))return -ENOMEM ; -if (mm->map_count > sysctl_max_map_count) +if (mm->map_count >get_sysctl_max_map_count ())return -ENOMEM ;if (security_vm_enough_memory_mm (mm, len >>PAGE_SHIFT ))
tools/testing/vma/include/custom.h +0 -3
@@ -21,9 +21,6 @@ extern unsigned long dac_mmap_min_addr;#define VM_BUG_ON (_expr) (BUG_ON(_expr))#define VM_BUG_ON_VMA (_expr, _vma) (BUG_ON(_expr)) -/* We hardcode this for now. */ -#define sysctl_max_map_count 0x1000000UL -#define TASK_SIZE ((1ul << 47)-PAGE_SIZE)/*
tools/testing/vma/include/dup.h +9 -0
@@ -419,6 +419,9 @@ struct vma_iterator {#define EMPTY_VMA_FLAGS ((vma_flags_t){ }) +#define MAPCOUNT_ELF_CORE_MARGIN (5) +#define DEFAULT_MAX_MAP_COUNT (USHRT_MAX - MAPCOUNT_ELF_CORE_MARGIN) +/* What action should be taken after an .mmap_prepare call is complete? */ enum mmap_action_type {MMAP_NOTHING ,/* Mapping is complete, no further action. */
@@ -1342,3 +1345,9 @@ static inline void vma_set_file(struct vm_area_struct *vma, struct file *file)swap (vma->vm_file, file);fput (file); } + +extern int sysctl_max_map_count; +static inline int get_sysctl_max_map_count (void ) +{ +return READ_ONCE (sysctl_max_map_count); +}
tools/testing/vma/main.c +2 -0
@@ -14,6 +14,8 @@#include "tests/mmap.c" #include "tests/vma.c" +int sysctl_max_map_count__read_mostly =DEFAULT_MAX_MAP_COUNT ; +/* Helper functions which utilise static kernel functions. */ struct vm_area_struct *merge_existing (struct vma_merge_struct *vmg)
Signature · 1 line
2.53.0
Lorenzo Stoakes (Oracle)
maintainerPATCH 3/3 mm/mremap: check map count under mmap write lock and abstract
In reply to Lorenzo Stoakes (Oracle)We are checking the mmap count in check_mremap_params(), prior to obtaining an mmap write lock, which means that accesses to current->mm->map_count might race with this field being updated.
Resolve this by only checking this field after the mmap write lock is held.
Additionally, abstract this check into a helper function with extensive ASCII documentation of what's going on.
--- mm/mremap.c | 88 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 75 insertions(+), 13 deletions(-)
mm/mremap.c +75 -13
@@ -1028,6 +1028,75 @@ static void vrm_stat_account(struct vma_remap_struct *vrm, mm->locked_vm += pages; } +static bool __check_map_count_against_split (struct mm_struct *mm, +bool before_unmaps) +{ +const int sys_map_count =get_sysctl_max_map_count (); +int map_count = mm->map_count; + +mmap_assert_write_locked (mm); + +/* +* At the point of shrinking the VMA, if new_len < old_len, we unmap +* thusly in the worst case: +* +* old_addr+old_len old_addr+old_len +* |---------------.----.---------| |---------------| |---------| +* | . . | -> | +1 | -1 | +1 | +* |---------------.----.---------| |---------------| |---------| +* old_addr+new_len old_addr+new_len +* +* At the point of removing the portion of an existing VMA to make space +* for the moved VMA if MREMAP_FIXED, we unmap thusly in the worst case: +* +* new_addr new_addr+new_len new_addr new_addr+new_len +* |----.---------------.---------| |----| |---------| +* | . . | -> | +1 | -1 | +1 | +* |----.---------------.---------| |----| |---------| +* +* Therefore, before we consider the move anything, we have to account +* for 2 additional VMAs possibly being created upon these unmappings. +*/ +if (before_unmaps) + map_count +=2 ; + +/* +* At the point of MOVING the VMA: +* +* We start by copying a VMA, which creates an additional VMA if no +* merge occurs, then if not MREMAP_DONTUNMAP, we unmap the source VMA. +* In the worst case we might then observe: +* +* new_addr new_addr+new_len new_addr new_addr+new_len +* |----| |---------| |----|---------------|---------| +* | | | | -> | | +1 | | +* |----| |---------| |----|---------------|---------| +* +* old_addr old_addr+old_len old_addr old_addr+old_len +* |----.---------------.---------| |----| |---------| +* | . . | -> | +1 | -1 | +1 | +* |----.---------------.---------| |----| |---------| +* +* Therefore we must check to ensure we have headroom of 2 additional +* VMAs. +*/ +return map_count +2 <= sys_map_count; +} + +/* Do we violate the map count limit if we split VMAs when moving the VMA? */ +static bool check_map_count_against_split (void ) +{ +return __check_map_count_against_split (current->mm, +/*before_unmaps=*/ false); +} + +/* Do we violate the map count limit if we split VMAs prior to early unmaps? */ +static bool check_map_count_against_split_early (void ) +{ +return __check_map_count_against_split (current->mm, +/*before_unmaps=*/ true); +} +/* * Perform checks before attempting to write a VMA prior to it being * moved.
@@ -1045,7 +1114,7 @@ static unsigned long prep_move_vma(struct vma_remap_struct *vrm)* which may not merge, then (if MREMAP_DONTUNMAP is not set) unmap the * source, which may split, causing a net increase of 2 mappings. */ -if (current->mm->map_count +2 >get_sysctl_max_map_count ()) +if (!check_map_count_against_split ())return -ENOMEM ;if (vma->vm_ops && vma->vm_ops->may_split) {
@@ -1804,18 +1873,6 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm)if (vrm_overlaps (vrm))return -EINVAL ; -/* -* We may unmap twice before invoking move_vma(), that is if new_len < -* old_len (shrinking), and in the MREMAP_FIXED case, unmapping part of -* a VMA located at the destination. -* -* In the worst case, both unmappings will cause splits, resulting in a -* net increased map count of 2. In move_vma() we check for headroom of -* 2 additional mappings, so check early to avoid bailing out then. -*/ -if (current->mm->map_count +4 >get_sysctl_max_map_count ()) -return -ENOMEM ; -return 0 ; }
@@ -1925,6 +1982,11 @@ static unsigned long do_mremap(struct vma_remap_struct *vrm)return -EINTR ; vrm->mmap_locked = true; +if (!check_map_count_against_split_early ()) { +mmap_write_unlock (mm); +return -ENOMEM ; + } +if (vrm_move_only (vrm)) { res =remap_move (vrm); }else {
Signature · 1 line
2.53.0
Andrew Morton
maintainerQuoted 21 lines · On Wed, 11 Mar 2026 17:24:35 +0000 "Lorenzo Stoakes (Oracle)" <[email protected]>…
Firstly, in
mremap(), it appears that our map count checks have been overly conservative - there is simply no reason to require that we have headroom of 4 mappings prior to moving the VMA, we only need headroom of 2 VMAs since commit659ace584e7a("mmap: don't return ENOMEM when mapcount is temporarily exceeded inmunmap()").Likely the original headroom of 4 mappings was a mistake, and 3 was actually intended.
Next, we access
sysctl_max_map_countin a number of places without being all that careful about how we do so.We introduces a simple helper that READ_ONCE()'s the field (
get_sysctl_max_map_count()) to ensure that the field is accessed correctly. TheWRITE_ONCE()side is already handled by the sysctl procfs code inproc_int_conv().We also move this field to
internal.has there's no reason for anybody else to access it outside of mm. Unfortunately we have to maintain the extern variable, asmmap.cimplements the procfs code.
Quoted 6 lines
Finally, we are accessing
current->mm->map_countwithout holding the mmap write lock, which is also not correct, so this series ensures the lock is head before we access it.We also abstract the check to a helper function, and add ASCII diagrams to explain why we're doing what we're doing.
This little series has no reviews, if anyone is wanting to boost our R-b stats.
Or not. I plan to upstream it unless someone stops me.
Re: [PATCH 1/3] mm/mremap: correct invalid map count check
In reply to Lorenzo Stoakes (Oracle)Quoted 49 lines · Lorenzo Stoakes (Oracle)
We currently check to see, if on moving a VMA when doing
mremap(), if it might violate the sys.vm.max_map_count limit.This was introduced in the mists of time prior to 2.6.12.
At this point in time, as now, the
move_vma()operation would copy the VMA (+1 mapping if not merged), then potentially split the source VMA upon unmap.Prior to commit
659ace584e7a("mmap: don't return ENOMEM when mapcount is temporarily exceeded inmunmap()"), a VMA split would check whethermm->map_count>=sysctl_max_map_countprior to a split before it ran.On unmap of the source VMA, if we are moving a partial VMA, we might split the VMA twice.
This would mean, on invocation of
split_vma()(as was), we'd check whethermm->map_count>=sysctl_max_map_countwith a map count elevated by one, then again with a map count elevated by two, ending up with a map count elevated by three.At this point we'd reduce the map count on unmap.
At the start of
move_vma(), there was a check that has remained throughout mremap()'s history ofmm->map_count>=sysctl_max_map_count- 3 (which impliesmm->mmap_count+ 4 >sysctl_max_map_count- that is, we must have headroom for 4 additional mappings).After
mm->map_countis elevated by 3, it is decremented by one once the unmap completes. The mmap write lock is held, so nothing else will observemm->map_count>sysctl_max_map_count.It appears this check was always incorrect - it should have either be one of '
mm->map_count>sysctl_max_map_count- 3' or 'mm->map_count>=sysctl_max_map_count- 2'.After commit
659ace584e7a("mmap: don't return ENOMEM when mapcount is temporarily exceeded inmunmap()"), the map count check on split is eliminated in the newly introduced__split_vma(), which the unmap path uses, and has that path check whethermm->map_count>=sysctl_max_map_count.This is valid since, net, an unmap can only cause an increase in map count of 1 (split both sides, unmap middle).
Since we only copy a VMA and (if
MREMAP_DONTUNMAPis not set) unmap afterwards, the maximum number of additional mappings that will actually be subject to any check will be 2.
Quoted 7 lines
Therefore, update the check to assert this corrected value. Additionally, update the check introduced by commit
ea2c3f6f5545("mm,mremap: bail out earlier inmremap_tounder map pressure") to account for this.While we're here, clean up the comment prior to that.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Signature · 1 line
Pedro
Re: [PATCH 2/3] mm: abstract reading sysctl_max_map_count, and READ_ONCE()
In reply to Lorenzo Stoakes (Oracle)Quoted 6 lines · Lorenzo Stoakes (Oracle)
Concurrent reads and writes of
sysctl_max_map_countare possible, so we shouldREAD_ONCE()andWRITE_ONCE().The sysctl procfs logic already enforces
WRITE_ONCE(), so abstract the read side withget_sysctl_max_map_count().
Quoted 7 lines
While we're here, also move the field to
mm/internal.hand add the getter there since only mm interacts with it, there's no need for anybody else to have access.Finally, update the VMA userland tests to reflect the change.
Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Signature · 1 line
Pedro
Re: [PATCH 3/3] mm/mremap: check map count under mmap write lock and abstract
In reply to Lorenzo Stoakes (Oracle)Quoted 4 lines · Lorenzo Stoakes (Oracle)
We are checking the mmap count in
check_mremap_params(), prior to obtaining an mmap write lock, which means that accesses tocurrent->mm->map_countmight race with this field being updated.
Quoted 8 lines
Resolve this by only checking this field after the mmap write lock is held.
Additionally, abstract this check into a helper function with extensive ASCII documentation of what's going on.
Reported-by Jianzhou Zhao <[email protected]>Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>
Shouldn't this have a Fixes: and go to stable?
Quoted 33 lines
--- mm/mremap.c | 88 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 75 insertions(+), 13 deletions(-)mm/mremap.c +21
-0@@ -1028,6 +1028,75 @@ static void vrm_stat_account(struct vma_remap_struct *vrm, mm->locked_vm += pages; } +static bool __check_map_count_against_split (struct mm_struct *mm, +bool before_unmaps) +{ +const int sys_map_count =get_sysctl_max_map_count (); +int map_count = mm->map_count; + +mmap_assert_write_locked (mm); + +/* +* At the point of shrinking the VMA, if new_len < old_len, we unmap +* thusly in the worst case: +* +* old_addr+old_len old_addr+old_len +* |---------------.----.---------| |---------------| |---------| +* | . . | -> | +1 | -1 | +1 | +* |---------------.----.---------| |---------------| |---------| +* old_addr+new_len old_addr+new_len +* +* At the point of removing the portion of an existing VMA to make space +* for the moved VMA if MREMAP_FIXED, we unmap thusly in the worst case: +*
mm/mremap.c static void vrm_stat_account(struct vma_remap_struct *vrm, · line 1061
Quoted 10 lines
mm/mremap.c +10
-0+* new_addr new_addr+new_len new_addr new_addr+new_len +* |----.---------------.---------| |----| |---------| +* | . . | -> | +1 | -1 | +1 | +* |----.---------------.---------| |----| |---------| +* +* Therefore, before we consider the move anything, we have to account +* for 2 additional VMAs possibly being created upon these unmappings. +*/ +if (before_unmaps) + map_count +=2 ;
oooh, shiny shiny diagrams.
Signature · 1 line
Pedro
Pedro Falcato
reviewerQuoted 27 lines · Andrew Morton
Quoted 25 lines · On Wed, 11 Mar 2026 17:24:35 +0000 "Lorenzo Stoakes (Oracle)" <[email protected]>…
Firstly, in
mremap(), it appears that our map count checks have been overly conservative - there is simply no reason to require that we have headroom of 4 mappings prior to moving the VMA, we only need headroom of 2 VMAs since commit659ace584e7a("mmap: don't return ENOMEM when mapcount is temporarily exceeded inmunmap()").Likely the original headroom of 4 mappings was a mistake, and 3 was actually intended.
Next, we access
sysctl_max_map_countin a number of places without being all that careful about how we do so.We introduces a simple helper that READ_ONCE()'s the field (
get_sysctl_max_map_count()) to ensure that the field is accessed correctly. TheWRITE_ONCE()side is already handled by the sysctl procfs code inproc_int_conv().We also move this field to
internal.has there's no reason for anybody else to access it outside of mm. Unfortunately we have to maintain the extern variable, asmmap.cimplements the procfs code.Finally, we are accessing
current->mm->map_countwithout holding the mmap write lock, which is also not correct, so this series ensures the lock is head before we access it.
Quoted 5 lines
Quoted 2 lines
We also abstract the check to a helper function, and add ASCII diagrams to explain why we're doing what we're doing.
This little series has no reviews, if anyone is wanting to boost our R-b stats.
Thanks for the heads up! This was too stale for too long :)
Signature · 1 line
Pedro
Lorenzo Stoakes (Oracle)
maintainerRe: [PATCH 3/3] mm/mremap: check map count under mmap write lock and abstract
In reply to Pedro FalcatoQuoted 7 lines · Pedro Falcato
Quoted 6 lines · Lorenzo Stoakes (Oracle)
We are checking the mmap count in
check_mremap_params(), prior to obtaining an mmap write lock, which means that accesses tocurrent->mm->map_countmight race with this field being updated.Resolve this by only checking this field after the mmap write lock is held.
Quoted 8 lines
Quoted 6 lines
Additionally, abstract this check into a helper function with extensive ASCII documentation of what's going on.
Reported-by Jianzhou Zhao <[email protected]>Signed-off-by Lorenzo Stoakes (Oracle) <[email protected]>Reviewed-by Pedro Falcato <[email protected]>
Thanks!
Quoted 2 lines
Shouldn't this have a Fixes: and go to stable?
Nah this is really hard to hit and doing the check is just making it so mremap can bail out early, so it's not worth it.
Quoted 38 lines
Quoted 38 lines
--- mm/mremap.c | 88 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 75 insertions(+), 13 deletions(-)mm/mremap.c +25
-0@@ -1028,6 +1028,75 @@ static void vrm_stat_account(struct vma_remap_struct *vrm, mm->locked_vm += pages; } +static bool __check_map_count_against_split (struct mm_struct *mm, +bool before_unmaps) +{ +const int sys_map_count =get_sysctl_max_map_count (); +int map_count = mm->map_count; + +mmap_assert_write_locked (mm); + +/* +* At the point of shrinking the VMA, if new_len < old_len, we unmap +* thusly in the worst case: +* +* old_addr+old_len old_addr+old_len +* |---------------.----.---------| |---------------| |---------| +* | . . | -> | +1 | -1 | +1 | +* |---------------.----.---------| |---------------| |---------| +* old_addr+new_len old_addr+new_len +* +* At the point of removing the portion of an existing VMA to make space +* for the moved VMA if MREMAP_FIXED, we unmap thusly in the worst case: +* +* new_addr new_addr+new_len new_addr new_addr+new_len +* |----.---------------.---------| |----| |---------| +* | . . | -> | +1 | -1 | +1 | +* |----.---------------.---------| |----| |---------|
Quoted 8 lines
mm/mremap.c static void vrm_stat_account(struct vma_remap_struct *vrm, · line 1061
Quoted 6 lines
mm/mremap.c +6
-0+* +* Therefore, before we consider the move anything, we have to account +* for 2 additional VMAs possibly being created upon these unmappings. +*/ +if (before_unmaps) + map_count +=2 ;oooh, shiny shiny diagrams.
:)
Quoted 3 lines
--
Pedro
Cheers, Lorenzo