mm: nommu: add sysctl_max_map_count() check for do_mmap()
Subsystems (2)
PATCH 1/1 mm: nommu: add sysctl_max_map_count() check for do_mmap()
The sysctl variable vm.max_map_count (sysctl_max_map_count) is not expose under !MMU case but used it wit the default vaule DEFAULT_MAX_MAP_COUNT as a limit of count. This is currently used when a vma entry is split into two chunks (split_vma()) but not used when allocated (do_mmap()). As a result, even if users request a large number of allocate memory, it will keep allocating until OOM happens.
This commit introduces a check at the begining of do_mmap in nommu.c to prevent this situation.
This is detected with a LTP (Linux Test Project) test, which linked below.
Link:
https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/syscalls/munmap/munmap04.c
--- mm/nommu.c | 3 +++ 1 file changed, 3 insertions(+)
mm/nommu.c +3 -0
@@ -1035,6 +1035,9 @@ unsigned long do_mmap(struct file *file,if (ret <0 )return ret; +if (current->mm->map_count >=get_sysctl_max_map_count ()) +return -ENOMEM ; +/* we ignore the address hint */ addr =0 ; len =PAGE_ALIGN (len);
Signature · 1 line
2.43.0
Andrew Morton
maintainerQuoted 3 lines · On Thu, 2 Jul 2026 10:28:30 +0900 Hajime Tazaki <[email protected]> wrote:
The sysctl variable vm.max_map_count (
sysctl_max_map_count) is not expose under !MMU case but used it wit the default vauleDEFAULT_MAX_MAP_COUNTas a limit of count.
This sounds like an oversight. I see no reason why nommu users cannot alter max_map_count.
Quoted 13 lines
This is currently used when
a vma entry is split into two chunks (split_vma()) but not used when allocated (do_mmap()). As a result, even if users request a large number of allocate memory, it will keep allocating until OOM happens.This commit introduces a check at the begining of
do_mmapinnommu.cto prevent this situation....
mm/nommu.c unsigned long do_mmap(struct file *file, · line 1043
Quoted 9 lines
mm/nommu.c +3
-0@@ -1035,6 +1035,9 @@ unsigned long do_mmap(struct file *file,if (ret <0 )return ret; +if (current->mm->map_count >=get_sysctl_max_map_count ()) +return -ENOMEM ; +/* we ignore the address hint */ addr =0 ; len =PAGE_ALIGN (len);
Seems sensible, thanks. That's what the !NOMMU code does.
Andrew Morton
maintainerQuoted 7 lines · On Thu, 2 Jul 2026 10:28:30 +0900 Hajime Tazaki <[email protected]> wrote:
The sysctl variable vm.max_map_count (
sysctl_max_map_count) is not expose under !MMU case but used it wit the default vauleDEFAULT_MAX_MAP_COUNTas a limit of count. This is currently used when a vma entry is split into two chunks (split_vma()) but not used when allocated (do_mmap()). As a result, even if users request a large number of allocate memory, it will keep allocating until OOM happens.
Quoted 5 lines
This commit introduces a check at the begining of
do_mmapinnommu.cto prevent this situation.This is detected with a LTP (Linux Test Project) test, which linked below.
AI review flagged a couple of issues in the current code, one quite serious:
Quoted 13 lines · Andrew Morton
Quoted 10 lines · On Thu, 2 Jul 2026 10:28:30 +0900 Hajime Tazaki <[email protected]> wrote:
The sysctl variable vm.max_map_count (
sysctl_max_map_count) is not expose under !MMU case but used it wit the default vauleDEFAULT_MAX_MAP_COUNTas a limit of count. This is currently used when a vma entry is split into two chunks (split_vma()) but not used when allocated (do_mmap()). As a result, even if users request a large number of allocate memory, it will keep allocating until OOM happens.This commit introduces a check at the begining of
do_mmapinnommu.cto prevent this situation.
Quoted 7 lines
Quoted 2 lines
This is detected with a LTP (Linux Test Project) test, which linked below.
AI review flagged a couple of issues in the current code, one quite serious:
I will follow up with another patch for the issue detected here.
btw, I found (and sashiko also detected) several typos for this particular patch. Can I send v2 patch with the additional fix, or do you prefer that I'll only send the additional patch ?
anyway, thanks for your time to look at this.
-- Hajime
Andrew Morton
maintainerQuoted 19 lines · On Mon, 06 Jul 2026 09:19:02 +0900 Hajime Tazaki <[email protected]> wrote:
Quoted 16 lines · Andrew Morton
Quoted 12 lines · On Thu, 2 Jul 2026 10:28:30 +0900 Hajime Tazaki <[email protected]> wrote:
The sysctl variable vm.max_map_count (
sysctl_max_map_count) is not expose under !MMU case but used it wit the default vauleDEFAULT_MAX_MAP_COUNTas a limit of count. This is currently used when a vma entry is split into two chunks (split_vma()) but not used when allocated (do_mmap()). As a result, even if users request a large number of allocate memory, it will keep allocating until OOM happens.This commit introduces a check at the begining of
do_mmapinnommu.cto prevent this situation.This is detected with a LTP (Linux Test Project) test, which linked below.
Quoted 6 lines
Quoted 4 lines
AI review flagged a couple of issues in the current code, one quite serious:
I will follow up with another patch for the issue detected here.
Great, thanks.
Quoted 3 lines
btw, I found (and sashiko also detected) several typos for this particular patch. Can I send v2 patch with the additional fix, or do you prefer that I'll only send the additional patch ?
Ah. Well. Confession. I actually quietly fed your changelog through Gemini to fix up a few things, resulting in
: The sysctl variable vm.max_map_count (sysctl_max_map_count) is not exposed : under !MMU configurations, but its default value (DEFAULT_MAX_MAP_COUNT) : is still used as a allocation limit. Currently, this limit is enforced : when a VMA entry is split into two chunks (split_vma()), but it is not : checked during initial allocation (do_mmap()). As a result, if a user : requests a large number of memory allocations, the system will continue : allocating until it hits an Out-Of-Memory (OOM) condition.
:
: This commit introduces a check at the beginning of do_mmap() in nommu.c to : prevent this situation.
:
: This issue was detected using the Linux Test Project (LTP) test linked : below.
I did this "quietly" because it feels a bit rude telling people that their English isn't great ;)
Lots of non-English speakers appear to be using LLMs on their changelogs nowadays and as long as they double-check the result, I think it's working very well!
Quoted 12 lines · Andrew Morton
Quoted 2 lines
I will follow up with another patch for the issue detected here.
Great, thanks.
Quoted 3 lines
btw, I found (and sashiko also detected) several typos for this particular patch. Can I send v2 patch with the additional fix, or do you prefer that I'll only send the additional patch ?
Ah. Well. Confession. I actually quietly fed your changelog through Gemini to fix up a few things, resulting in
Quoted 13 lines
: The sysctl variable vm.max_map_count (
sysctl_max_map_count) is not exposed : under !MMU configurations, but its default value (DEFAULT_MAX_MAP_COUNT) : is still used as a allocation limit. Currently, this limit is enforced : when a VMA entry is split into two chunks (split_vma()), but it is not : checked during initial allocation (do_mmap()). As a result, if a user : requests a large number of memory allocations, the system will continue : allocating until it hits an Out-Of-Memory (OOM) condition.
:
: This commit introduces a check at the beginning ofdo_mmap()innommu.cto : prevent this situation.
:
: This issue was detected using the Linux Test Project (LTP) test linked : below.
thanks, this is perfect.
Quoted 6 lines
I did this "quietly" because it feels a bit rude telling people that their English isn't great ;)
Lots of non-English speakers appear to be using LLMs on their changelogs nowadays and as long as they double-check the result, I think it's working very well!
thanks for handling this and a kind way to fix my mistakes. yes, the AI era.
I will send a follow up patch for the additional issue.
-- Hajime