Skip to contentSkip to conversation

MAP_FIXED leaving holes on failure is very difficult to recover from and can even be a security hole

DISCUSSION Demi Marie Obenour 3 participants 4 messages Nov 5 – Nov 6 Memory management

lore
Share & tools
lore
#0 lore raw

Right now, a failed MAP_FIXED mmap() can leave a hole in the address space. Another mmap() can then enter this hole. This was fixed in the past, but the fix was undone in
4f87153e82c4 ("mm: change failure of MAP_FIXED to restoring the gap on failure").

This is really nasty, and can create serious problems for userspace. A simple example is MAP_FIXED over a part of a KVM memslot to expose memory to a guest. If that leaves a hole, the result is catastrophic:
Subsequent mmap()s (perhaps by malloc()) can get put into that address range. KVM will give the guest access to whatever happens to be put there, and one must assume the guest can turn that into a VM escape.

Other situations don't result in immediate disaster, but are still really hard to recover from. One can use MAP_FIXED_NOREPLACE to refill the hole if nobody else has filled it, but if someone has, there's no hope of making the original mapping (that the MAP_FIXED had clobbered) contiguous again. One must either figure out how to handle a non-contiguous mapping, rebuild the mapping elsewhere, or crash.

In my case, I can work around the problem by using a single memslot per file descriptor, but I suspect there are other use-cases that aren't so lucky. Yet others may simply not have realized this was possible; there's even a Stack Overflow answer [1] that claims that the map is replaced atomically. I only found out about it from Claude Opus 5.5. I asked it to write a test suite, and it pointed out that a failed MAP_FIXED mmap() can leave a hole. If I hadn't asked, I would have shipped a security vulnerability.

Should the behavior be changed again, so that mmap() with MAP_FIXED is atomic? That would provide a UAPI that actually makes sense and can be used without horribly ugly workarounds.

[1]: https://stackoverflow.com/a/28576674

Signature · 2 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
#1 lore raw

[ please check kernel mails in MAINTAINERS - my and Liam's at least were
incorrect, which makes it easy to miss. Fixed those up. ]

Thanks for the report.

In general linux system calls do not guarantee compound operation atomicity, and any assumption based on that is incorrect. If you see an error - the operation failed, perhaps partially.

Further, a failed MAP_FIXED leaving gaps was the case for most of the history of linux. Avoiding that is infeasible.

I advise you to review the NOTES section
https://man7.org/linux/man-pages/man2/mmap.2.html on MAP_FIXED - your suggested use of it is not recommended.

--
Cheers, Lorenzo

#2 lore raw
Quoted 2 lines · Lorenzo Stoakes (ARM)

[ please check kernel mails in MAINTAINERS - my and Liam's at least were
incorrect, which makes it easy to miss. Fixed those up. ]

I thought I did that, but obviously I didn't do it properly.

Quoted 11 lines

Thanks for the report.

In general linux system calls do not guarantee compound operation atomicity, and any assumption based on that is incorrect. If you see an error - the operation failed, perhaps partially.

Further, a failed MAP_FIXED leaving gaps was the case for most of the> history of linux. Avoiding that is infeasible.

I advise you to review the NOTES section
https://man7.org/linux/man-pages/man2/mmap.2.html on MAP_FIXED - your suggested use of it is not recommended.

Does that mean that MAP_FIXED should be deprecated? It seems that the safe use-cases can all use MAP_FIXED_NOREPLACE, and that the use-cases that cannot be served by MAP_FIXED_NOREPLACE cannot be served by current Linux.

Specifically, if a MAP_FIXED mmap() fails, the mapping might either still be there (in which case one _must_ call munmap() to cleanup) or it might not be there (in which case one _must not_ call munmap()). There's no race-free way to tell which is the case. This means that MAP_FIXED is unusable in multithreaded programs, unless one is okay with leaking address space.

Are there cases, such as anonymous mappings, that cannot leave holes? I believe there are some programs that depend on this to handle SIGBUS faults from truncated mapped files. If even that can leave a hole, the only thread-safe alternative would be longjmp() out of the SIGBUS handler.

An alternative behavior would be to unmap the pages, but keep the address space reserved, as if the mapping was a PROT_NONE anonymous one. That would avoid the "it may or may not have been freed" situation, as it would be safe to retry the mapping or call munmap().

Signature · 2 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
#3 lore raw
Quoted 11 lines · Demi Marie Obenour
Quoted 2 lines · Lorenzo Stoakes (ARM)

[ please check kernel mails in MAINTAINERS - my and Liam's at least were
incorrect, which makes it easy to miss. Fixed those up. ]

I thought I did that, but obviously I didn't do it properly.

Quoted 5 lines

Thanks for the report.

In general linux system calls do not guarantee compound operation atomicity, and any assumption based on that is incorrect. If you see an error - the operation failed, perhaps partially.

I'd think we only cared about not exposing the temporary state between unmap old and map new, right?

Quoted 11 lines
Quoted 5 lines

Further, a failed MAP_FIXED leaving gaps was the case for most of the> history of linux. Avoiding that is infeasible.

I advise you to review the NOTES section
https://man7.org/linux/man-pages/man2/mmap.2.html on MAP_FIXED - your suggested use of it is not recommended.

Does that mean that MAP_FIXED should be deprecated? It seems that the safe use-cases can all use MAP_FIXED_NOREPLACE, and that the use-cases that cannot be served by MAP_FIXED_NOREPLACE cannot be served by current Linux.

Quoted 6 lines

Specifically, if a MAP_FIXED mmap() fails, the mapping might either still be there (in which case one _must_ call munmap() to cleanup) or it might not be there (in which case one _must not_ call munmap()). There's no race-free way to tell which is the case. This means that MAP_FIXED is unusable in multithreaded programs, unless one is okay with leaking address space.

You said "This was fixed in the past, but the fix was undone in 4f87153e82c4". Any idea when exactly it was "fixed", whether the change was done consciously (or some refactoring side-effect) and if somebody started depending on it?
4f87153e82c4 is from 6.12 (2 years old by now) and nobody complained so far...

Quoted 1 line

Are there cases, such as anonymous mappings, that cannot leave holes?

(I don't have an immediate answer. I'd guess that if e.g. some kernel allocation can fail, this can happen)

Quoted 4 lines

I believe there are some programs that depend on this to handle SIGBUS faults from truncated mapped files. If even that can leave a hole, the only thread-safe alternative would be longjmp() out of the SIGBUS handler.

A concrete example would help I guess.

Quoted 4 lines

An alternative behavior would be to unmap the pages, but keep the address space reserved, as if the mapping was a PROT_NONE anonymous one. That would avoid the "it may or may not have been freed" situation, as it would be safe to retry the mapping or call munmap().

It would probably have to be a new flag as that's a different semantic from what we have now. And the use case for adding it would have to be strong IMHO.