mmap: is default non-populating behavior stable?

DISCUSSION Eugene V. Lyubimkin 6 participants 11 messages Nov 3, 2008 – Nov 5, 2008 Memory management

Share & tools

Eugene V. Lyubimkin

author

2008-11-03 21:59 UTC #0 lore raw

Hello kernel hackers!

The current implementation of mmap() in kernel is very convenient.
It allows to mmap(fd) very big amount of memory having small file as back-end. So one can mmap() 100 MiB on empty file, use first 10 KiB of memory, munmap() and have only 10 KiB of file at the end. And while working with memory, file will automatically be grown by read/write memory requests.

Question is: can user-space application rely on this behavior (I failed to find any documentation about this)?

TIA and please CC me in replies.

Signature · 1 line

Eugene V. Lyubimkin aka JackYF

Peter Zijlstra

2008-11-03 22:41 UTC #1 lore raw

Quoted 8 lines · Eugene V. Lyubimkin

Hello kernel hackers!
The current implementation of mmap() in kernel is very convenient.
It allows to mmap(fd) very big amount of memory having small file as back-end. So one can mmap() 100 MiB on empty file, use first 10 KiB of memory, munmap() and have only 10 KiB of file at the end. And while working with memory, file will automatically be grown by read/write memory requests.

Quoted 4 lines

Question is: can user-space application rely on this behavior (I failed to find any documentation about this)?
TIA and please CC me in replies.

mmap() writes past the end of the file should not grow the file if I understand things write, but produce a sigbus (after the first page size alignment).

The exact interaction of mmap() and truncate() I'm not exactly clear on.

The safe way to do things is to first create your file of at least the size you mmap, using truncate. This will create a sparse file, and will on any sane filesystem not take more space than its meta data.

Thereafter you can fill it with writes to the mmap.

Rik van Riel

2008-11-03 22:49 UTC #2 lore raw

Quoted 9 lines · Peter Zijlstra

Quoted 8 lines · Eugene V. Lyubimkin
Hello kernel hackers!
The current implementation of mmap() in kernel is very convenient.
It allows to mmap(fd) very big amount of memory having small file as back-end. So one can mmap() 100 MiB on empty file, use first 10 KiB of memory, munmap() and have only 10 KiB of file at the end. And while working with memory, file will automatically be grown by read/write memory requests.

Quoted 8 lines

Quoted 4 lines
Question is: can user-space application rely on this behavior (I failed to find any documentation about this)?
TIA and please CC me in replies.
mmap() writes past the end of the file should not grow the file if I understand things write, but produce a sigbus (after the first page size alignment).

Indeed, faulting beyond the end of file returns a SIGBUS, see these lines in mm/filemap.c:filemap_fault():

        size = (i_size_read(inode) + PAGE_CACHE_SIZE - 1) >> 
PAGE_CACHE_SHIFT;
        if (vmf->pgoff >= size)
                return VM_FAULT_SIGBUS;

Quoted 1 line

The exact interaction of mmap() and truncate() I'm not exactly clear on.

Truncate will reduce the size of the mmaps on the file to match the new file size, so processes accessing beyond the end of file will get a segmentation fault (SIGSEGV).

Quoted 5 lines

The safe way to do things is to first create your file of at least the size you mmap, using truncate. This will create a sparse file, and will on any sane filesystem not take more space than its meta data.
Thereafter you can fill it with writes to the mmap.

Agreed.

Signature · 7 lines

All Rights Reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Chris Friesen

2008-11-04 15:56 UTC #3 lore raw

Quoted 7 lines · Rik van Riel

Quoted 1 line · Peter Zijlstra
The exact interaction of mmap() and truncate() I'm not exactly clear on.
Truncate will reduce the size of the mmaps on the file to match the new file size, so processes accessing beyond the end of file will get a segmentation fault (SIGSEGV).

I suspect Peter was talking about using truncate() to set the initial file size, effectively increasing rather than reducing it.

Chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Peter Zijlstra

2008-11-04 16:06 UTC #4 lore raw

Quoted 11 lines · Chris Friesen

Quoted 7 lines · Rik van Riel
Quoted 1 line · Peter Zijlstra
The exact interaction of mmap() and truncate() I'm not exactly clear on.
Truncate will reduce the size of the mmaps on the file to match the new file size, so processes accessing beyond the end of file will get a segmentation fault (SIGSEGV).
I suspect Peter was talking about using truncate() to set the initial file size, effectively increasing rather than reducing it.

I was thinking of truncate() on an already mmap()'ed region, either increasing or decreasing the size so that part of the mmap becomes (in)valid.

I'm not sure how POSIX speaks of this.

I think Linux does the expected thing.

Alan Cox

2008-11-04 16:28 UTC #5 lore raw

Quoted 13 lines · On Tue, 04 Nov 2008 17:07:00 +0100 Peter Zijlstra <[email protected]> wrote:

Quoted 11 lines · Chris Friesen
Quoted 7 lines · Rik van Riel
Quoted 1 line · Peter Zijlstra
The exact interaction of mmap() and truncate() I'm not exactly clear on.
Truncate will reduce the size of the mmaps on the file to match the new file size, so processes accessing beyond the end of file will get a segmentation fault (SIGSEGV).
I suspect Peter was talking about using truncate() to set the initial file size, effectively increasing rather than reducing it.

Quoted 7 lines

I was thinking of truncate() on an already mmap()'ed region, either increasing or decreasing the size so that part of the mmap becomes (in)valid.
I'm not sure how POSIX speaks of this.
I think Linux does the expected thing.

I believe our behaviour is correct for mmap/mumap/truncate and it certainly used to be and was tested.

At the point you do anything involving mremap (which is non posix) our behaviour becomes rather bizarre.

Alan

Eugene V. Lyubimkin

author

2008-11-04 16:47 UTC #6 lore raw

Quoted 5 lines · Alan Cox

Quoted 3 lines · On Tue, 04 Nov 2008 17:07:00 +0100 Peter Zijlstra <[email protected]> wrote:
[snip]
I'm not sure how POSIX speaks of this.

Quoted 7 lines

Quoted 1 line
I think Linux does the expected thing.
I believe our behaviour is correct for mmap/mumap/truncate and it certainly used to be and was tested.
At the point you do anything involving mremap (which is non posix) our behaviour becomes rather bizarre.

Thanks to all for answers. I have made the conclusion that doing "open() new file, truncate(<big size>), mmap(<the same big size>), write/read some memory pages" should not populate other, untouched by write/read pages (until MAP_POPULATE given), right?

Signature · 1 line

Eugene V. Lyubimkin aka JackYF

Hugh Dickins

2008-11-05 16:42 UTC #7 lore raw

Quoted 4 lines · Eugene V. Lyubimkin

Quoted 3 lines · Alan Cox
I believe our behaviour is correct for mmap/mumap/truncate and it certainly used to be and was tested.

Agreed.

Quoted 3 lines

Quoted 3 lines
At the point you do anything involving mremap (which is non posix) our behaviour becomes rather bizarre.

Certainly mremap is non-POSIX, but I can't think of any way in which it would interfere with Eugene's assumptions about population.

(Every year or so we do wonder whether to change an extending mremap of a MAP_SHARED|MAP_ANONYMOUS object to extend the object itself instead of just SIGBUSing on the extension: but I've so far remained conservative about that, and Eugene appears to be thinking of more ordinary files.)

Quoted 5 lines

Thanks to all for answers. I have made the conclusion that doing "open() new file, truncate(<big size>), mmap(<the same big size>), write/read some memory pages" should not populate other, untouched by write/read pages (until MAP_POPULATE given), right?

That is a reasonable description of how the kernel tries and will always try to handle it, approximately; but I don't think you can rely upon it absolutely.

For a start, it depends on the filesystem: I believe that vfat, for example, does not support the concept of sparse files (files with holes in), so its truncate(<big size>) will allocate the whole of that big size initially.

I'm not sure what you mean by "populate": in mm, as in MAP_POPULATE, we're thinking of prefaulting pages into the user address space; but you're probably thinking of whether the blocks are allocated on disk?

Prefaulting hole pages into the user address space may imply allocating blocks on disk, or it may not: likely to depend on filesystem again.

Quoted 1 line

From time to time we toy with prefaulting adjacent pages when a fault

occurs (though IIRC tests have proved disappointing in the past): we'd like to keep that option open, but it would go against your guidelines above to some extent.

Hugh

Alan Cox

2008-11-05 16:54 UTC #8 lore raw

Quoted 4 lines

(Every year or so we do wonder whether to change an extending mremap of a MAP_SHARED|MAP_ANONYMOUS object to extend the object itself instead of just SIGBUSing on the extension: but I've so far remained conservative about that, and Eugene appears to be thinking of more ordinary files.)

Try an mremap of a VM_GROWS* mapping and all the other things of this nature. I would say our current behaviour is not what might be expected by users. The extending an object case is just one example of weird behaviour.

Alan

Eugene V. Lyubimkin

author

2008-11-05 17:46 UTC #9 lore raw

In reply to Hugh Dickins

Quoted 4 lines · Hugh Dickins

Quoted 4 lines
Thanks to all for answers. I have made the conclusion that doing "open() new file, truncate(<big size>), mmap(<the same big size>), write/read some memory pages" should not populate other, untouched by write/read pages (until MAP_POPULATE given), right?

[snip]

Quoted 4 lines

For a start, it depends on the filesystem: I believe that vfat, for example, does not support the concept of sparse files (files with holes in), so its truncate(<big size>) will allocate the whole of that big size initially.

For my case vfat is not an option fortunately.

Quoted 3 lines

I'm not sure what you mean by "populate": in mm, as in MAP_POPULATE, we're thinking of prefaulting pages into the user address space; but you're probably thinking of whether the blocks are allocated on disk?

Yes.

Quoted 4 lines

Quoted 1 line
From time to time we toy with prefaulting adjacent pages when a fault
occurs (though IIRC tests have proved disappointing in the past): we'd like to keep that option open, but it would go against your guidelines above to some extent.

It depends how is "adjacent" would count :) If several pages, probably not. If millions or similar, that would be a problem. It's very convenient to use such "open+truncate+mmap+write/read" behavior to make self-growing-on-demand cache in memory with disk as back-end without remaps.

Thanks for descriptive answer.

Signature · 1 line

Eugene V. Lyubimkin aka JackYF

Hugh Dickins

2008-11-05 23:31 UTC #10 lore raw

Quoted 8 lines · Eugene V. Lyubimkin

Quoted 4 lines · Hugh Dickins
Quoted 1 line
From time to time we toy with prefaulting adjacent pages when a fault
occurs (though IIRC tests have proved disappointing in the past): we'd like to keep that option open, but it would go against your guidelines above to some extent.
It depends how is "adjacent" would count :) If several pages, probably not. If millions or similar, that would be a problem.

That's fine, you'll be safe: you can be sure that it would never be in the kernel's interest to prefault more than "several" extra pages.

Well, bearing in mind that famous "640K enough for all" remark, let's not say "never"; but it won't prefault millions until memory is so abundant and I/O so fast that you'd be happy with it prefaulting millions yourself.

Quoted 3 lines

It's very convenient to use such
"open+truncate+mmap+write/read" behavior to make self-growing-on-demand cache in memory with disk as back-end without remaps.

Yes. Though one thing to beware of is running out of disk space:
whereas a write system call should be good at reporting -ENOSPC, the filesystem may not be able to handle running out of disk space when writing back dirty mmaped pages - it may quietly lose the data.

Hugh