Memory-Mapping with LFS
Who is the memory mapping expert? :)
What are the current file size limits for memory mapping via glibc's mmap() function on linux:
- for a native 32-Bit System not using LFS?
- for a native 32-Bit System using LFS?
- for a native 64-Bit System?
(linux-kernel >2.6, of course)
It would be nice if someone could tell me what I have to consider if I want to use memory mapping for files. I'm currently a little bit confused about it (information overflow :)). Personal opinions about speed (maybe increase or decrease for large files) are also welcome.
--------------------------
The glibc documentation says:
"Since mmapped pages can be stored back to their file when physical memory is low, it is possible to mmap files orders of magnitude larger than both the physical memory and swap space. The only limit is address space. The theoretical limit is 4GB on a 32-bit machine - however, the actual limit will be smaller since some areas will be reserved for other purposes. If the LFS interface is used the file size on 32-bit systems is not limited to 2GB (offsets are signed which reduces the addressable area of 4GB by half); the full 64-bit are available."
- I doubt that the full 64-Bit (something within Exabyte) are available in practical use. Right or wrong?
I've also found an old kernel-list e-mail from 2004 that says:
"There is a limit per process in the kernel vm that prevent from mmapping more than 512GB of data."
- Is this still true for the current kernel?
An example:
Let's presume the following case. I have an 8 GB file, 1 GB physical memory and I want to use memory mapping for that file using LFS on a 32-Bit machine.
- Is it possible?
If yes, let's presume that I have 2 or more pointers, that are frequently pointing to completely different places and switch the data they are pointing to.
- How is it managed (by the kernel)? Through the pages, that are mentioned in the glibc documentation above? Are these page operations really faster than normal random file access (lseek etc)?
Quoted 2 lines · Andreas Baer
What are the current file size limits for memory mapping via glibc's
mmap()function on linux:
Its 2^wordsize - areas used for other purposes. 1-3GB on i386. A couple of petabytes(?) on ia64.
Quoted 2 lines
- I doubt that the full 64-Bit (something within Exabyte) are available in practical use. Right or wrong?
You can likely map a 512 M section starting at 6GB into your address space.
Quoted 5 lines
I've also found an old kernel-list e-mail from 2004 that says:
"There is a limit per process in the kernel vm that prevent from mmapping more than 512GB of data."- Is this still true for the current kernel?
That depends on the architecture. Certainly no problem on Itanium.
Quoted 4 lines
Let's presume the following case. I have an 8 GB file, 1 GB physical memory and I want to use memory mapping for that file using LFS on a 32-Bit machine.
- Is it possible?
Yes. I am not sure why one would have to use LFS. This should work on any filesystem.
Quoted 7 lines
If yes, let's presume that I have 2 or more pointers, that are frequently pointing to completely different places and switch the data they are pointing to.
- How is it managed (by the kernel)? Through the pages, that are mentioned in the glibc documentation above? Are these page operations really faster than normal random file access (lseek etc)?
Kernel gets a fault if the page pointed to is not present and reads it in from the file. And yes its faster because I/O is aligned and the page does not need to be copied from kernel to user space.