wrong madvise(MADV_DONTNEED) semantic
Hi,
There is something wrong with the current madvise(MADV_DONTNEED) implementation. Both the manpage and the source code says that MADV_DONTNEED means that the application does not care about the data, so it might be thrown away by the kernel. But that's not what posix says:
http://www.opengroup.org/onlinepubs/009695399/functions/posix_madvise.html
It says that "The posix_madvise() function shall have no effect on the semantics of access to memory in the specified range". I.e. the data that was recorded shall be saved!
The current linux implementation of MADV_DONTNEED is rather an implementation of solaris' MADV_FREE, see its manpage:
http://docs.sun.com/app/docs/doc/816-5168/6mbb3hrde?a=view
Hence the current madvise_dontneed() implementation could be renamed into madvise_free() and the appropriate MADV_FREE case be added, while a new implementation of madvise_dontneed() _needs_ be written. It may for instance go through the range so as to zap clean pages (since it is safe), and set dirty pages as being least recently used so that they will be considered as good candidates for eviction.
(And the manpage should get corrected too)
Regards,
Samuel Thibault
PATCH 1/1 Hotfix for Re: wrong madvise(MADV_DONTNEED) semantic
Quoted 17 lines · Samuel Thibault
There is something wrong with the current
madvise(MADV_DONTNEED)implementation. Both the manpage and the source code says thatMADV_DONTNEEDmeans that the application does not care about the data, so it might be thrown away by the kernel. But that's not what posix says:http://www.opengroup.org/onlinepubs/009695399/functions/posix_madvise.html
It says that "The
posix_madvise()function shall have no effect on the semantics of access to memory in the specified range". I.e. the data that was recorded shall be saved!The current linux implementation of
MADV_DONTNEEDis rather an implementation of solaris'MADV_FREE, see its manpage:
http://docs.sun.com/app/docs/doc/816-5168/6mbb3hrde?a=view
Quoted 6 lines
Hence the current
madvise_dontneed()implementation could be renamed intomadvise_free()and the appropriateMADV_FREEcase be added, while a new implementation ofmadvise_dontneed()_needs_ be written. It may for instance go through the range so as to zap clean pages (since it is safe), and set dirty pages as being least recently used so that they will be considered as good candidates for eviction.
Agreed. Unless someone else comes up with something better soon, you can use below patch.
Akpm, would you merge this?
Jörn
Signature · 24 lines
The story so far: In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move. -- Douglas Adams Signed-off-by: Jörn Engel <[email protected]> --- mm/madvise.c | 4 ++++ 1 files changed, 4 insertions(+) --- linux-2.6.12-rc4logfs/mm/madvise.c~madvice_bug 2005-03-04 11:40:50.000000000 +0100 +++ linux-2.6.12-rc4logfs/mm/madvise.c 2005-06-28 16:34:54.000000000 +0200 @@ -97,6 +97,10 @@ static long madvise_willneed(struct vm_a static long madvise_dontneed(struct vm_area_struct * vma, unsigned long start, unsigned long end) { + return 0; + /* Really ugly bugfix - madvice may never change anything + * beyond performance. Dropping dirty pages is right out. + */ if ((vma->vm_flags & VM_LOCKED) || is_vm_hugetlb_page(vma)) return -EINVAL;
Quoted 4 lines · Samuel Thibault
There is something wrong with the current
madvise(MADV_DONTNEED)implementation. Both the manpage and the source code says thatMADV_DONTNEEDmeans that the application does not care about the data, so it might be thrown away by the kernel.
I agree that the Linux madvise manpage is unclear and should be fixed.
If your interpretation of the problem is correct, then it should be trivial to write a test program demonstrating the problem. Did you write the simple test program and run it?
I did, and the results make me think that the Linux implementation does conform to the POSIX spec you quote. I did not observe any data loss.
So it's just a documentation issue.
Besides, if you read the documentation closely, it does not say what you think it says.
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap) or zero-fill-on-demand pages for mappings without an underlying file.
You seem to think that "reloading ... from the underlying mapped file" means that changes are lost, but that's not implied.
Also, the parenthetical near the top of the manpage "(except in the case of MADV_DONTNEED)" is, AFAICS, wrong. MADV_DONTNEED does not affect semantics. It looks to me like someone "improved" madvise.2 without actually understanding what they were talking about.
Below is the test program I used.
-andy
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <sys/stat.h>
typedef unsigned int u32;
void die(char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
vfprintf(stderr, fmt, ap);
va_end(ap);
exit(1);
}int check_cookie(char *file, u32 cookie)
{
u32 buf;
int fd;
if((fd = open(file, O_RDONLY, 0)) == -1)
die("%s: %s\n", file, strerror(errno));
if(read(fd, &buf, sizeof(buf)) == -1)
die("read: %s\n", strerror(errno));
close(fd);
return buf == cookie;
}int dotest(char *file, u32 cookie, int do_msync, int do_madvise)
{
void *p;
int fd, len = 16 * 1024, prot = PROT_READ|PROT_WRITE;
if((fd = open(file, O_RDWR|O_CREAT, 0666)) == -1)
die("%s: %s\n", file, strerror(errno));
if(ftruncate(fd, len) == -1)
die("ftruncate: %s\n", strerror(errno));
if(write(fd, "", 1) == -1)
die("write: %s\n", strerror(errno));
if((p = mmap(0, len, prot, MAP_SHARED, fd, 0)) == MAP_FAILED)
die("mmap: %s\n", strerror(errno));
*(u32 *)p = cookie;
if(do_msync)
if(msync(p, len, MS_SYNC) == -1)
die("msync: %s\n", strerror(errno));
if(do_madvise)
if(madvise(p, len, MADV_DONTNEED) == -1)
die("madvise: %s\n", strerror(errno));
if(close(fd) == -1)
die("close: %s\n", strerror(errno));
printf("c = %08x msync: %s madvise: %s %s\n",
cookie, do_msync ? "YES" : " NO", do_madvise ? "YES" : " NO",
check_cookie(file, cookie) ? "ok" : "FAILED");
}int main(int argc, char **argv)
{
if(argc != 2) die("usage: %s file\n", argv[0]);
lrand48();
dotest(argv[1], lrand48(), 0, 0);
dotest(argv[1], lrand48(), 1, 0);
dotest(argv[1], lrand48(), 0, 1);
dotest(argv[1], lrand48(), 1, 1);
}Quoted 13 lines · Andy Isaacson
Besides, if you read the documentation closely, it does not say what you think it says.
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap) or zero-fill-on-demand pages for mappings without an underlying file.
Quoted 2 lines
You seem to think that "reloading ... from the underlying mapped file" means that changes are lost, but that's not implied.
This wording _does_ imply that changes are lost if the file is mapped writable and not mysnc'ed or if the memory mapping is anonymous.
In the former, changes are dropped and the file is reread from the stale on-disk copy. In the latter case, the data is dropped and the pages are zero-filled on access.
Robert Love
Quoted 14 lines · Robert Love
Quoted 13 lines · Andy Isaacson
Besides, if you read the documentation closely, it does not say what you think it says.
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap) or zero-fill-on-demand pages for mappings without an underlying file.
Quoted 4 lines
Quoted 2 lines
You seem to think that "reloading ... from the underlying mapped file" means that changes are lost, but that's not implied.
This wording _does_ imply that changes are lost
I contest your interpretation of the manpage; while it could be read the way you suggest, I claim that because Linux mmap is inherently coherent (as opposed to, for example, AIX 4.1 mmap) then the "underlying file" already contains the updated contents, and ergo msync is not required for correct MAP_SHARED semantics on Linux, and the manpage as it stands is (misleading, but) both accurate to the 2.6.11 implementation and compliant with the POSIX description posted earlier.
Quoted 1 line
if the file is mapped writable and not mysnc'ed
This is the case that my posted example code exercises, and I did not see any problems. Is there some additional circumstance that is necessary to cause it to break? (I tested on 2.6.11-rc5 or something close to that.)
Quoted 4 lines
or if the memory mapping is anonymous.
In the latter case, the data is dropped and the pages are zero-filled on access.
Yes, MAP_ANONYMOUS is a more interesting case. Somebody else will have to write the testcase for that...
I think the correct docs fix is to simply delete the misleading parts of madvise.2 so that it reads
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.)
and remove the erroneous parenthetical in the first paragraph.
... unless, of course, someone can actually demonstrate a case where madvise results in differing semantics...
-andy
Quoted 3 lines · Andy Isaacson, le Tue 28 Jun 2005 11:16:20 -0700, a écrit :
If your interpretation of the problem is correct, then it should be trivial to write a test program demonstrating the problem. Did you write the simple test program and run it?
I indeed didn't, trusting both the man page, the source code comments, and my knowledge of zap_page_range().
Quoted 10 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap) or zero-fill-on-demand pages for mappings without an underlying file.
Quoted 2 lines
You seem to think that "reloading ... from the underlying mapped file" means that changes are lost, but that's not implied.
I didn't say anything precise. What mostly feared me was the "zero-fill-on-demand pages for mappings without an underlying file."
Quoted 1 line
Below is the test program I used.
It does indeed work, but this is no proof. It your testcase it does indeed work, since the page still remains in the page cache (it's a shared mapping of the file). But now try this one. It uses private mappings, and fails as expected, getting 0 in the ANONYMOUS case, and the original file value in the file mapping case.
Regards,
Samuel Thibault
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <unistd.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <sys/stat.h>
typedef unsigned int u32;
void die(char *fmt, ...)
{
va_list ap;
va_start(ap, fmt);
vfprintf(stderr, fmt, ap);
va_end(ap);
exit(1);
}int check_cookie(char *file, u32 cookie)
{
u32 buf;
int fd;
if((fd = open(file, O_RDONLY, 0)) == -1)
die("%s: %s\n", file, strerror(errno));
if(read(fd, &buf, sizeof(buf)) == -1)
die("read: %s\n", strerror(errno));
close(fd);
return buf == cookie;
}int dotest(char *file, u32 cookie, int do_anonymous, int do_msync, int do_madvise)
{
void *p;
int fd, len = 16 * 1024, prot = PROT_READ|PROT_WRITE;
u32 newcookie;
if (!do_anonymous) {
if((fd = open(file, O_RDWR|O_CREAT, 0666)) == -1)
die("%s: %s\n", file, strerror(errno));
if(ftruncate(fd, len) == -1)
die("ftruncate: %s\n", strerror(errno));
if(write(fd, "", 1) == -1)
die("write: %s\n", strerror(errno));
}
if((p = mmap(0, len, prot, (do_anonymous ? MAP_ANONYMOUS : 0) | MAP_PRIVATE, do_anonymous ? -1 : fd, 0)) == MAP_FAILED)
die("mmap: %s\n", strerror(errno));
*(u32 *)p = cookie;
if(do_msync)
if(msync(p, len, MS_SYNC) == -1)
die("msync: %s\n", strerror(errno));
if(do_madvise)
if(madvise(p, len, MADV_DONTNEED) == -1)
die("madvise: %s\n", strerror(errno));
newcookie = *(u32 *)p;
printf("c = %08x msync: %s madvise: %s %s\n",
cookie, do_msync ? "YES" : " NO", do_madvise ? "YES" : " NO",
newcookie == cookie ? "ok" : "FAILED");
}int main(int argc, char **argv)
{
if(argc != 2) die("usage: %s file\n", argv[0]);
lrand48();
dotest(argv[1], lrand48(), 0, 0, 0);
dotest(argv[1], lrand48(), 0, 1, 0);
dotest(argv[1], lrand48(), 0, 0, 1);
dotest(argv[1], lrand48(), 0, 1, 1);
dotest(argv[1], lrand48(), 1, 0, 0);
dotest(argv[1], lrand48(), 1, 1, 0);
dotest(argv[1], lrand48(), 1, 0, 1);
dotest(argv[1], lrand48(), 1, 1, 1);
}Quoted 7 lines · Andy Isaacson
I contest your interpretation of the manpage; while it could be read the way you suggest, I claim that because Linux mmap is inherently coherent (as opposed to, for example, AIX 4.1 mmap) then the "underlying file" already contains the updated contents, and ergo msync is not required for correct
MAP_SHAREDsemantics on Linux, and the manpage as it stands is (misleading, but) both accurate to the 2.6.11 implementation and compliant with the POSIX description posted earlier.
Well, there is nothing guaranteeing (either in the Linux implementation or the code) that the in-memory changes are synced back to disk. You have to call msyc().
Quoted 6 lines
Quoted 1 line
if the file is mapped writable and not mysnc'ed
This is the case that my posted example code exercises, and I did not see any problems. Is there some additional circumstance that is necessary to cause it to break? (I tested on 2.6.11-rc5 or something close to that.)
Yah, I am not--at all--talking about actual behavior. Just that the wording definitely says, its kind of been a common belief, that MADV_DONTNEED literally ditches your data. If you need it, don't call MADV_DONTNEED.
Quoted 7 lines
Quoted 4 lines
or if the memory mapping is anonymous.
In the latter case, the data is dropped and the pages are zero-filled on access.
Yes,
MAP_ANONYMOUSis a more interesting case. Somebody else will have to write the testcase for that...
It would have to discard the pages, losing the data, unless it caused swapout (let's hope not).
Quoted 8 lines
I think the correct docs fix is to simply delete the misleading parts of madvise.2 so that it reads
MADV_DONTNEED(quoted diff) +0
-0Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.)
Quoted 4 lines
and remove the erroneous parenthetical in the first paragraph.
.. unless, of course, someone can actually demonstrate a case where madvise results in differing semantics...
Nod.
I think we need to resolve the differences between the man pages, comments, expected user behavior, kernel implementation, POSIX standard, and what other OS's do. Figure out what to do, then unify everything.
Robert Love
Quoted 3 lines · Robert Love, le Tue 28 Jun 2005 15:23:43 -0400, a écrit :
I think we need to resolve the differences between the man pages, comments, expected user behavior, kernel implementation, POSIX standard, and what other OS's do. Figure out what to do, then unify everything.
Well, posix says data should be kept. The solaris man page for madvise, since it proposes a MADV_FREE case too do agree with posix.
I've tested the program (thanks Andy) on solaris 5.8, it does work fine. On OSF1 it failed on the anonymous case.
Some people may want their data to be kept, so the safe side is to keep data safe, i.e. both kernel and manpage be corrected, but leave a bug notice in the manpage about previous kernels.
Regards,
Samuel
Quoted 6 lines · Jörn Engel
Plus, common sense agrees with posix. An implementation of madvice that doesn't do anything should be perfectly sane and functionally equivalent to any other implementation. Only differences should be in performance.
Dropping dirty pages is a clear functional difference.
I like the idea (I think someone suggested this early on) of renaming the current MADV_DONTNEED to MADV_FREE and then adding a correct MADV_DONTNEED.
And, as I said, the man page needs clarification.
Robert Love
Quoted 4 lines · Robert Love
I like the idea (I think someone suggested this early on) of renaming the current
MADV_DONTNEEDtoMADV_FREEand then adding a correctMADV_DONTNEED.
Imo, that's still a crime against common sense. Madvice should give the kernel some advice about which data to keep or not to keep in memory, hence the name. It should *not* tell the kernel to corrupt data, which currently appears to be the case.
If the application knows 100% that it is the _only_ possible user of this data and will never again use it, dropping dirty pages might be a sane option. Effectively that translates to anonymous memory only. In all other cases, dirty pages should be written back.
Quoted 1 line
And, as I said, the man page needs clarification.
Definitely.
Jörn
Signature · 2 lines
Eighty percent of success is showing up. -- Woody Allen
Quoted 3 lines · Jörn Engel, le Tue 28 Jun 2005 22:17:04 +0200, a écrit :
If the application knows 100% that it is the _only_ possible user of this data and will never again use it, dropping dirty pages might be a sane option. Effectively that translates to anonymous memory only.
And private file mappings?
Regards,
Samue
Quoted 6 lines · Samuel Thibault
Quoted 3 lines · Jörn Engel, le Tue 28 Jun 2005 22:17:04 +0200, a écrit :
If the application knows 100% that it is the _only_ possible user of this data and will never again use it, dropping dirty pages might be a sane option. Effectively that translates to anonymous memory only.
And private file mappings?
As in inode->i_nlink == 0? Yes, if you can prove that only this one thread still has it open. How to deal with multithreaded processes? I don't know and would default to "write it back" again.
Besides, writing the dirty pages to backing store can only hurt performance, never correctness. The data could already be synced at the time of the madvice call anyway.
Jörn
Signature · 2 lines
This above all: to thine own self be true. -- Shakespeare
Quoted 6 lines · Samuel Thibault
Quoted 3 lines · Jörn Engel, le Tue 28 Jun 2005 22:17:04 +0200, a écrit :
If the application knows 100% that it is the _only_ possible user of this data and will never again use it, dropping dirty pages might be a sane option. Effectively that translates to anonymous memory only.
And private file mappings?
So long as the mapping exists, the data should not disappear. So a MAP_PRIVATE mapping should behave just like a MAP_SHARED mapping, and both need to be fixed to not lose data due to madvise(MADV_DONTNEED). (I agree with Joern, MADV_FREE seems like an ill-advised extension.)
If this means some swap needs to be allocated, well, so be it.
-andy
Quoted 7 lines · Samuel Thibault
Quoted 3 lines · Robert Love, le Tue 28 Jun 2005 15:23:43 -0400, a écrit :
I think we need to resolve the differences between the man pages, comments, expected user behavior, kernel implementation, POSIX standard, and what other OS's do. Figure out what to do, then unify everything.
Well, posix says data should be kept. The solaris man page for madvise, since it proposes a
MADV_FREEcase too do agree with posix.
Plus, common sense agrees with posix. An implementation of madvice that doesn't do anything should be perfectly sane and functionally equivalent to any other implementation. Only differences should be in performance.
Dropping dirty pages is a clear functional difference.
Quoted 6 lines
I've tested the program (thanks Andy) on solaris 5.8, it does work fine. On OSF1 it failed on the anonymous case.
Some people may want their data to be kept, so the safe side is to keep data safe, i.e. both kernel and manpage be corrected, but leave a bug notice in the manpage about previous kernels.
Sounds sane.
Jörn
Signature · 3 lines
There's nothing better for promoting creativity in a medium than making an audience feel "Hmm I could do better than that!" -- Douglas Adams in a slashdot interview
Re: wrong madvise(MADV DONTNEED) semantic
In reply to Robert LoveQuoted 16 lines · Andy Isaacson
Quoted 15 lines
Besides, if you read the documentation closely, it does not say what you
think it says.
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap) or zero-fill-on-demand pages for mappings without an underlying file.
Quoted 5 lines
Quoted 2 lines
You seem to think that "reloading ... from the underlying mapped file" means that changes are lost, but that's not implied.
This wording _does_ imply that changes are lost if the file is mapped writable and not mysnc'ed or if the memory mapping is anonymous.
A little late into this thread, but...
Indeed it does imply that, because that was what I understood when (IIRC) I wrote that text in the man page.
Quoted 3 lines
In the former, changes are dropped and the file is reread from the stale on-disk copy. In the latter case, the data is dropped and the pages are zero-filled on access.
Yes.
Cheers,
Michael
Signature · 2 lines
5 GB Mailbox, 50 FreeSMS http://www.gmx.net/de/go/promail +++ GMX - die erste Adresse für Mail, Message, More +++
Re: wrong madvise(MADV DONTNEED) semantic
In reply to Robert LoveQuoted 45 lines
Quoted 8 lines
I contest your interpretation of the manpage; while it could be read the way you suggest, I claim that because Linux mmap is inherently coherent (as opposed to, for example, AIX 4.1 mmap) then the "underlying
file" already contains the updated contents, and ergo msync is not required for correctMAP_SHAREDsemantics on Linux, and the manpage as it stands is (misleading, but) both accurate to the 2.6.11 implementation and compliant with the POSIX description posted earlier.Well, there is nothing guaranteeing (either in the Linux implementation or the code) that the in-memory changes are synced back to disk. You have to call
msyc().Quoted 6 lines
Quoted 1 line
if the file is mapped writable and not mysnc'ed
This is the case that my posted example code exercises, and I did not see any problems. Is there some additional circumstance that is necessary to cause it to break? (I tested on 2.6.11-rc5 or something close to that.)
Yah, I am not--at all--talking about actual behavior. Just that the wording definitely says, its kind of been a common belief, that
MADV_DONTNEEDliterally ditches your data. If you need it, don't callMADV_DONTNEED.Quoted 7 lines
Quoted 4 lines
or if the memory mapping is anonymous.
In the latter case, the data is dropped and the pages are zero-filled on access.
Yes,
MAP_ANONYMOUSis a more interesting case. Somebody else will have to write the testcase for that...It would have to discard the pages, losing the data, unless it caused swapout (let's hope not).
Quoted 9 lines
I think the correct docs fix is to simply delete the misleading parts of
madvise.2 so that it reads
MADV_DONTNEED(quoted diff) +0
-0Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.)
Quoted 6 lines
Quoted 4 lines
and remove the erroneous parenthetical in the first paragraph.
.. unless, of course, someone can actually demonstrate a case where madvise results in differing semantics...
Nod.
Well, as far as I last knew Linux MADV_DONTNEED is indeed destructive, and not POSIX.1-2001 conformant.
Quoted 3 lines
I think we need to resolve the differences between the man pages, comments, expected user behavior, kernel implementation, POSIX standard, and what other OS's do. Figure out what to do, then unify everything.
First of all, I'm always happy to receive man page patches at [email protected].
My understanding of Linux MADV_DONTNEED is that it differs from POSIX and from *some* other Unix implementations.
For Linux:
For a MAP_PRIVATE region, the mapped pages are explicitly discarded, so that modifications to the pages are lost. The next access results in a page fault re-initializing either with the contents of the underlying mapped file, or, in the case of an anonymous mapping, with 0. (A while ago, I verified this behaviour using a test program.)
For a MAP_SHARED region, the kernel may discard modified pages, depending on the architecture. (I seem to recall getting this information about architecture variation from an email conversation with Jamie Lokier; I haven't verified it. On x86 at least, I haven't seen an pages discarded for MADV_DONTNEED on a MAP_SHARED region.)
On some some other Unix implementations, MADV_DONTNEED simply informs the kernel that the specified pages can be swapped out if necessary. (i.e., MADV_DONTNEED on those implementations is non-destructive.)
I realise that the current Linux madvise(2) manual page doesn't go into quite the above level of detail. If someone wants to confirm or improve the above, I'm happy to update the man page.
Cheers,
Michael
Signature · 2 lines
Weitersagen: GMX DSL-Flatrates mit Tempo-Garantie! Ab 4,99 Euro/Monat: http://www.gmx.net/de/go/dsl
Re: wrong madvise(MADV DONTNEED) semantic
Quoted 6 lines · Michael Kerrisk
For a
MAP_SHAREDregion, the kernel may discard modified pages, depending on the architecture. (I seem to recall getting this information about architecture variation from an email conversation with Jamie Lokier; I haven't verified it. On x86 at least, I haven't seen an pages discarded forMADV_DONTNEEDon aMAP_SHAREDregion.)
The discard in MAP_SHARED case isn't explicit. It's due to CPU cache incoherency, so the data (or just some of the data) is lost simply by not syncing properly between userspace and the page cache.
It wouldn't be apparent on fully-coherent architectures like x86.
That was when I last looked at the code (during the 2.5 time frame). I didn't test it, only examined the code, but quite thoroughly.
-- Jamie
Quoted 12 lines · Jörn Engel
Quoted 3 lines · Robert Love
I like the idea (I think someone suggested this early on) of renaming the current
MADV_DONTNEEDtoMADV_FREEand then adding a correctMADV_DONTNEED.Imo, that's still a crime against common sense. Madvice should give the kernel some advice about which data to keep or not to keep in memory, hence the name. It should *not* tell the kernel to corrupt data, which currently appears to be the case.
Quoted 4 lines
If the application knows 100% that it is the _only_ possible user of this data and will never again use it, dropping dirty pages might be a sane option. Effectively that translates to anonymous memory only. In all other cases, dirty pages should be written back.
There is also the case of shmget/shmat memory segments. Some applications will use these in order to map a very large amount of memory and then madvise(MADV_DONTNEED) in order to play nice with the rest of the system should memory pressure / system load / etc require it. Obviously if other tasks have these segments mapped, the pages cannot be discarded. If a task is the sole "mapper" of the region and doesn't need that memory (ever again) it would be good to avoid the i/o overhead of swapping it out and just discarding it. Perhaps MADV_DONTNEED isn't the right place for this, but there is demand for this behavior.
--Darren Hart
Quoted 9 lines
Quoted 3 lines
And, as I said, the man page needs clarification.
Definitely.
Jörn