mm: madvise: Ignore repeated MADV_DONTNEED hints
Subsystems (8)
Mel Gorman
reviewerRFC PATCH 1/1 mm: madvise: Ignore repeated MADV_DONTNEED hints
glibc malloc changed behaviour in glibc 2.10 to have per-thread arenas instead of creating new areans if the existing ones were contended. The decision appears to have been made so the allocator scales better but the downside is that madvise(MADV_DONTNEED) is now called for these per-thread areans during free. This tears down pages that would have previously remained. There is nothing wrong with this decision from a functional point of view but any threaded application that frequently allocates/frees the same-sized region is going to incur the full teardown and refault costs.
This is extremely obvious in the ebizzy benchmark. At its core, threads are frequently freeing and allocating buffers of the same size. It is much faster on distributions with older versions of glibc. Profiles showed that a large amount of system CPU time was spent on tearing down and refaulting pages.
This patch identifies when a thread is frequently calling MADV_DONTNEED on the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.
ebizzy Overall Throughput
3.19.0-rc6 3.19.0-rc6
vanilla madvise-v1r1
Hmean Rsec-1 12619.93 ( 0.00%) 34807.02 (175.81%)
Hmean Rsec-3 33434.19 ( 0.00%) 100733.77 (201.29%)
Hmean Rsec-5 45796.68 ( 0.00%) 134257.34 (193.16%)
Hmean Rsec-7 53146.93 ( 0.00%) 145512.85 (173.79%)
Hmean Rsec-12 55132.87 ( 0.00%) 145560.86 (164.02%)
Hmean Rsec-18 54846.52 ( 0.00%) 145120.79 (164.59%)
Hmean Rsec-24 54368.95 ( 0.00%) 142733.89 (162.53%)
Hmean Rsec-30 54388.86 ( 0.00%) 141424.09 (160.02%)
Hmean Rsec-32 54047.11 ( 0.00%) 139151.76 (157.46%)And the system CPU usage was also much reduced
3.19.0-rc6 3.19.0-rc6
vanilla madvise-v1r1
User 2647.19 8347.26
System 5742.90 42.42
Elapsed 1350.60 1350.65It's even more ridiculous on a 4 socket machine
ebizzy Overall Throughput
3.19.0-rc6 3.19.0-rc6
vanilla madvise-v1r1
Hmean Rsec-1 5354.37 ( 0.00%) 12838.61 (139.78%)
Hmean Rsec-4 10338.41 ( 0.00%) 50514.52 (388.61%)
Hmean Rsec-7 7766.33 ( 0.00%) 88555.30 (1040.25%)
Hmean Rsec-12 7188.40 ( 0.00%) 154180.78 (2044.86%)
Hmean Rsec-21 7001.82 ( 0.00%) 266555.51 (3706.95%)
Hmean Rsec-30 8975.08 ( 0.00%) 314369.88 (3402.70%)
Hmean Rsec-48 12136.53 ( 0.00%) 358525.74 (2854.10%)
Hmean Rsec-79 12607.37 ( 0.00%) 341646.49 (2609.89%)
Hmean Rsec-110 12563.37 ( 0.00%) 338058.65 (2590.83%)
Hmean Rsec-141 11701.85 ( 0.00%) 331255.78 (2730.80%)
Hmean Rsec-172 10987.39 ( 0.00%) 312003.62 (2739.65%)
Hmean Rsec-192 12050.46 ( 0.00%) 296401.88 (2359.67%) 3.19.0-rc6 3.19.0-rc6
vanilla madvise-v1r1User 4136.44 53506.65
System 50262.68 906.49
Elapsed 1802.07 1801.99
Note in both cases that the elapsed time is similar because the benchmark is configured to run for a fixed duration.
MADV_FREE would have a lower cost if the underlying allocator used it but there is no guarantee that allocators will use it. Arguably the kernel has no business preventing an application developer shooting themselves in a foot but this is a case where it's relatively easy to detect the bad behaviour and avoid it.
--- fs/exec.c | 4 ++++ include/linux/sched.h | 5 +++++ kernel/fork.c | 5 +++++ mm/madvise.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 70 insertions(+)
fs/exec.c +4 -0
@@ -1551,6 +1551,10 @@ static int do_execveat_common(int fd, struct filename *filename, current->in_execve =0 ;acct_update_integrals (current);task_numa_free (current); +if (current->madvise_state) { +kfree (current->madvise_state); + current->madvise_state =NULL ; + }free_bprm (bprm);kfree (pathbuf);putname (filename);
include/linux/sched.h +5 -0
@@ -1271,6 +1271,9 @@ enum perf_event_task_context { perf_nr_task_contexts, }; +/* mm/madvise.c */ +struct madvise_state_info ; +struct task_struct {volatile long state;/* -1 unrunnable, 0 runnable, >0 stopped */ void *stack;
@@ -1637,6 +1640,8 @@ struct task_struct {struct page_frag task_frag; +struct madvise_state_info *madvise_state; +#ifdef CONFIG_TASK_DELAY_ACCT struct task_delay_info *delays;#endif
kernel/fork.c +5 -0
@@ -246,6 +246,11 @@ void __put_task_struct(struct task_struct *tsk)delayacct_tsk_free (tsk);put_signal_struct (tsk->signal); +if (current->madvise_state) { +kfree (current->madvise_state); + current->madvise_state =NULL ; + } +if (!profile_handoff_task (tsk))free_task (tsk); }
mm/madvise.c +56 -0
74 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerQuoted 8 lines · On Mon, 2 Feb 2015 16:55:25 +0000 Mel Gorman <[email protected]> wrote:
glibc malloc changed behaviour in glibc 2.10 to have per-thread arenas instead of creating new areans if the existing ones were contended. The decision appears to have been made so the allocator scales better but the downside is that
madvise(MADV_DONTNEED)is now called for these per-thread areans during free. This tears down pages that would have previously remained. There is nothing wrong with this decision from a functional point of view but any threaded application that frequently allocates/frees the same-sized region is going to incur the full teardown and refault costs.
MADV_DONTNEED has been there for many years. How could this problem not have been noticed during glibc 2.10 development/testing? Is there some more recent kernel change which is triggering this?
Quoted 2 lines
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint.
That's pretty nasty-looking :(
And presumably there are all sorts of behaviours which will still trigger the problem but which will avoid the start/end equality test in ignore_madvise_hint()?
Really, this is a glibc problem and only a glibc problem. MADV_DONTNEED is unavoidably expensive and glibc is calling MADV_DONTNEED for a region which it *does* need. Is there something preventing this from being addressed within glibc?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 11 lines · Andrew Morton
Quoted 8 lines · On Mon, 2 Feb 2015 16:55:25 +0000 Mel Gorman <[email protected]> wrote:
glibc malloc changed behaviour in glibc 2.10 to have per-thread arenas instead of creating new areans if the existing ones were contended. The decision appears to have been made so the allocator scales better but the downside is that
madvise(MADV_DONTNEED)is now called for these per-thread areans during free. This tears down pages that would have previously remained. There is nothing wrong with this decision from a functional point of view but any threaded application that frequently allocates/frees the same-sized region is going to incur the full teardown and refault costs.
Quoted 2 lines
MADV_DONTNEEDhas been there for many years. How could this problem not have been noticed during glibc 2.10 development/testing?
I do not know. I only spotted it due to switching distributions. Looping allocations and frees of the same sizes is considered inefficient and it might have been dismissed on those grounds. It's probably less noticeable when it only affects threaded applications.
Quoted 3 lines
Is there
some more recent kernel change which is triggering this?
Not that I'm aware of.
Quoted 5 lines
Quoted 2 lines
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint.That's pretty nasty-looking :(
Yep, it is but we're very limited in terms of what we can do within the kernel here.
Quoted 4 lines
And presumably there are all sorts of behaviours which will still trigger the problem but which will avoid the start/end equality test in
ignore_madvise_hint()?
Yes. I would expect that a simple pattern of multiple allocs followed by multiple frees in a loop would also trigger it.
Quoted 3 lines
Really, this is a glibc problem and only a glibc problem.
MADV_DONTNEEDis unavoidably expensive and glibc is callingMADV_DONTNEEDfor a region which it *does* need.
To be fair to glibc, it calls it on a region it *thinks* it doesn't need only to reuse it immediately afterwards because of how the benchmark is implemented.
Quoted 2 lines
Is there something
preventing this from being addressed within glibc?
I doubt it other than I expect they'll punt it back and blame either the application for being stupid or the kernel for being slow.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerQuoted 5 lines · On Mon, 2 Feb 2015 22:18:24 +0000 Mel Gorman <[email protected]> wrote:
Quoted 2 lines
Is there something
preventing this from being addressed within glibc?I doubt it other than I expect they'll punt it back and blame either the application for being stupid or the kernel for being slow.
*Is* the application being stupid? What is it actually doing? Something like
pthread_routine()
{
p = malloc(X);
do_some(work);
free(p);
return;
}?
If so, that doesn't seem stupid?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Mon, 2 Feb 2015 22:18:24 +0000 Mel Gorman <[email protected]> wrote:
Quoted 2 lines
Is there something
preventing this from being addressed within glibc?I doubt it other than I expect they'll punt it back and blame either the application for being stupid or the kernel for being slow.
Quoted 8 lines
*Is* the application being stupid? What is it actually doing? Something like
pthread_routine(){ p = malloc(X); do_some(work); free(p);
Ebizzy adds a time based loop in there. But yeah, pretty much a standard pthread model.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 9 lines · Andrew Morton
Quoted 5 lines · On Mon, 2 Feb 2015 22:18:24 +0000 Mel Gorman <[email protected]> wrote:
Quoted 2 lines
Is there something
preventing this from being addressed within glibc?I doubt it other than I expect they'll punt it back and blame either the application for being stupid or the kernel for being slow.
*Is* the application being stupid? What is it actually doing?
Only a little. There is little simulated think time between the allocation and the subsequent free. It means the cost of alloc/free dominates where in "real" applications they would either be reusing buffers if they were constantly needed or the think time would mask the cost of the free.
Quoted 10 lines
Something like
pthread_routine(){ p = malloc(X); do_some(work); free(p); return; }
Pretty much. There is a search_mem() function that
alloc(copy_size)
memcpy
search
free(copy)
A real application might try and avoid the copy or reuse buffers if they encountered this particular problem.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 51 lines · Mel Gorman
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Mon, 2 Feb 2015 16:55:25 +0000 Mel Gorman <[email protected]> wrote:
glibc malloc changed behaviour in glibc 2.10 to have per-thread arenas instead of creating new areans if the existing ones were contended. The decision appears to have been made so the allocator scales better but the downside is that
madvise(MADV_DONTNEED)is now called for these per-thread areans during free. This tears down pages that would have previously remained. There is nothing wrong with this decision from a functional point of view but any threaded application that frequently allocates/frees the same-sized region is going to incur the full teardown and refault costs.Quoted 2 lines
MADV_DONTNEEDhas been there for many years. How could this problem not have been noticed during glibc 2.10 development/testing?I do not know. I only spotted it due to switching distributions. Looping allocations and frees of the same sizes is considered inefficient and it might have been dismissed on those grounds. It's probably less noticeable when it only affects threaded applications.
Quoted 3 lines
Is there
some more recent kernel change which is triggering this?Not that I'm aware of.
Quoted 5 lines
Quoted 2 lines
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint.That's pretty nasty-looking :(
Yep, it is but we're very limited in terms of what we can do within the kernel here.
Quoted 4 lines
And presumably there are all sorts of behaviours which will still trigger the problem but which will avoid the start/end equality test in
ignore_madvise_hint()?Yes. I would expect that a simple pattern of multiple allocs followed by multiple frees in a loop would also trigger it.
Quoted 3 lines
Really, this is a glibc problem and only a glibc problem.
MADV_DONTNEEDis unavoidably expensive and glibc is callingMADV_DONTNEEDfor a region which it *does* need.To be fair to glibc, it calls it on a region it *thinks* it doesn't need only to reuse it immediately afterwards because of how the benchmark is implemented.
Quoted 5 lines
Quoted 2 lines
Is there something
preventing this from being addressed within glibc?I doubt it other than I expect they'll punt it back and blame either the application for being stupid or the kernel for being slow.
This sounds like something that could benefit from Minchan's MADV_FREE, instead of MADV_DONTNEED.
If non page aligned malloc/free does not depend on pages being zeroed, I suspect an MADV_DONTNEED resulting from a malloc/free loop also does not depend on it.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.
The manpage, at least, claims that we zero-fill after MADV_DONTNEED is called:
Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.
So if we have anything depending on the behavior that it's _always_ zero-filled after an MADV_DONTNEED, this will break it.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
[CC linux-api, man pages]
Quoted 8 lines · Dave Hansen
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.The manpage, at least, claims that we zero-fill after
MADV_DONTNEEDis called:
Quoted 7 lines
Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.So if we have anything depending on the behavior that it's _always_ zero-filled after an
MADV_DONTNEED, this will break it.
OK, so that's a third person (including me) who understood it as a zero-fill guarantee. I think the man page should be clarified (if it's indeed not guaranteed), or we have a bug.
The implementation actually skips MADV_DONTNEED for VM_LOCKED|VM_HUGETLB|VM_PFNMAP vma's.
I'm not sure about VM_PFNMAP, these are probably special enough. For mlock, one could expect that mlocking and MADV_DONTNEED would be in some opposition, but it's not documented in the manpage AFAIK. Neither is the hugetlb case, which could be really unexpected by the user.
Next, what the man page says about guarantees:
"The kernel is free to ignore the advice."
- that would suggest that nothing is guaranteed
"This call does not influence the semantics of the application (except in the case of MADV_DONTNEED)"
- that depends if the reader understands it as "does influence by MADV_DONTNEED" or "may influence by MADV_DONTNEED"
- btw, isn't MADV_DONTFORK another exception that does influence the semantics? And since it's mentioned as a workaround for some hardware, is it OK to ignore this advice?
And the part you already cited:
"Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill on-demand pages for mappings without an underlying file."
- The word "will result" did sound as a guarantee at least to me. So here it could be changed to "may result (unless the advice is ignored)"?
And if we agree that there is indeed no guarantee, what's the actual semantic difference from MADV_FREE? I guess none? So there's only a possible perfomance difference?
Vlastimil
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Quoted 19 lines · Vlastimil Babka
[CC linux-api, man pages]
Quoted 8 lines · Dave Hansen
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.The manpage, at least, claims that we zero-fill after
MADV_DONTNEEDis called:Quoted 7 lines
Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.So if we have anything depending on the behavior that it's _always_ zero-filled after an
MADV_DONTNEED, this will break it.
Quoted 6 lines
OK, so that's a third person (including me) who understood it as a zero-fill guarantee. I think the man page should be clarified (if it's indeed not guaranteed), or we have a bug.
The implementation actually skips
MADV_DONTNEEDfor VM_LOCKED|VM_HUGETLB|VM_PFNMAP vma's.
It doesn't skip. It fails with -EINVAL. Or I miss something.
Quoted 2 lines
- The word "will result" did sound as a guarantee at least to me. So here it could be changed to "may result (unless the advice is ignored)"?
It's too late to fix documentation. Applications already depends on the beheviour.
Signature · 1 line
Kirill A. Shutemov
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Quoted 20 lines · Kirill A. Shutemov
Quoted 19 lines · Vlastimil Babka
[CC linux-api, man pages]
Quoted 15 lines · Dave Hansen
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.The manpage, at least, claims that we zero-fill after
MADV_DONTNEEDis called:Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.So if we have anything depending on the behavior that it's _always_ zero-filled after an
MADV_DONTNEED, this will break it.
Quoted 8 lines
Quoted 6 lines
OK, so that's a third person (including me) who understood it as a zero-fill guarantee. I think the man page should be clarified (if it's indeed not guaranteed), or we have a bug.
The implementation actually skips
MADV_DONTNEEDfor VM_LOCKED|VM_HUGETLB|VM_PFNMAP vma's.It doesn't skip. It fails with
-EINVAL. Or I miss something.
No, I missed that. Thanks for pointing out. The manpage also explains EINVAL in this case:
* The application is attempting to release locked or shared pages (with MADV_DONTNEED).
- that covers mlocking ok, not sure if the rest fits the "shared pages" case though. I dont see any check for other kinds of shared pages in the code.
Quoted 5 lines
Quoted 2 lines
- The word "will result" did sound as a guarantee at least to me. So here it could be changed to "may result (unless the advice is ignored)"?
It's too late to fix documentation. Applications already depends on the beheviour.
Right, so as long as they check for EINVAL, it should be safe. It appears that jemalloc does.
I still wouldnt be sure just by reading the man page that the clearing is guaranteed whenever I dont get an error return value, though,
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Hello Vlastimil
Thanks for CCing me into this thread.
Quoted 28 lines · Vlastimil Babka
Quoted 27 lines · Kirill A. Shutemov
Quoted 25 lines · Vlastimil Babka
[CC linux-api, man pages]
Quoted 8 lines · Dave Hansen
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.The manpage, at least, claims that we zero-fill after
MADV_DONTNEEDis called:Quoted 7 lines
Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.So if we have anything depending on the behavior that it's _always_ zero-filled after an
MADV_DONTNEED, this will break it.OK, so that's a third person (including me) who understood it as a zero-fill guarantee. I think the man page should be clarified (if it's indeed not guaranteed), or we have a bug.
The implementation actually skips
MADV_DONTNEEDfor VM_LOCKED|VM_HUGETLB|VM_PFNMAP vma's.
Quoted 7 lines
Quoted 1 line
It doesn't skip. It fails with
-EINVAL. Or I miss something.No, I missed that. Thanks for pointing out. The manpage also explains EINVAL in this case:
* The application is attempting to release locked or shared pages (with
MADV_DONTNEED).
Yes, there is that. But the page could be more explicit when discussing MADV_DONTNEED in the main text. I've done that.
Quoted 2 lines
- that covers mlocking ok, not sure if the rest fits the "shared pages" case though. I dont see any check for other kinds of shared pages in the code.
Agreed. "shared" here seems confused. I've removed it. And I've added mention of "Huge TLB pages" for this error.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
- The word "will result" did sound as a guarantee at least to me. So here it could be changed to "may result (unless the advice is ignored)"?
It's too late to fix documentation. Applications already depends on the beheviour.
Right, so as long as they check for EINVAL, it should be safe. It appears that jemalloc does.
So, first a brief question: in the cases where the call does not error out, are we agreed that in the current implementation, MADV_DONTNEED will always result in zero-filled pages when the region is faulted back in (when we consider pages that are not backed by a file)?
Quoted 2 lines
I still wouldnt be sure just by reading the man page that the clearing is guaranteed whenever I dont get an error return value, though,
I'm not quite sure what you want here. I mean: if there's an error, then the DONTNEED action didn't occur, right? Therefore, there won't be zero-filled pages. But, for what it's worth, I added "If the operation succeeds" at the start of that sentence beginning "Subsequent accesses...".
Now, some history, explaining why the page is a bit of a mess, and for that matter why I could really use more help on it from MM folk (especially in the form of actual patches [1], rather than notes about deficiencies in the documentation), because:
***I simply cannot keep up with all of the details***.
Once upon a time (Linux 2.4), there was madvise() with just 5 flags:
MADV_NORMAL
MADV_RANDOM
MADV_SEQUENTIAL
MADV_WILLNEED
MADV_DONTNEED
And already a dozen years ago, *I* added the text about MADV_DONTNEED. Back then, I believe it was true. I'm not sure if it's still true now, but I assume for the moment that it is, and await feedback. And the text saying that the call does not affect the semantics of memory access dates back even further (and was then true, MADV_DONTNEED aside).
Those 5 flags have analogs in POSIX's posix_madvise() (albeit, there is a semantic mismatch between the destructive MADV_DONTNEED and POSIX's nondestructive POSIX_MADV_DONTNEED). They also appear on most other implementations.
Since the original implementation, numerous pieces of cruft^W^W^W excellent new flags have been overloaded into this one system call. Some of those certainly violated the "does not change the semantics of the application" statement, but, sadly, the kernel developers who implemented MADV_REMOVE or MADV_DONTFORK did not think to send a patch to the man page for those new flags, one that might have noted that the semantics of the application are changed by such flags. Equally sadly, I did overlook to scan the bigger page when *I* added documentation of these flags to those pages, otherwise I might have caught that detail.
So, just to repeat, I could really use more help on it from MM folk in the form of actual patches to the man page.
Thanks,
Michael
[1] https://www.kernel.org/doc/man-pages/patches.html
Signature · 9 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Quoted 3 lines · Michael Kerrisk (man-pages)
Hello Vlastimil
Thanks for CCing me into this thread.
NP
Quoted 12 lines · Vlastimil Babka
Quoted 10 lines · Kirill A. Shutemov
Quoted 3 lines · Vlastimil Babka
It doesn't skip. It fails with
-EINVAL. Or I miss something.No, I missed that. Thanks for pointing out. The manpage also explains EINVAL in this case:
* The application is attempting to release locked or shared pages (with
MADV_DONTNEED).
Quoted 9 lines
Yes, there is that. But the page could be more explicit when discussing
MADV_DONTNEEDin the main text. I've done that.Quoted 2 lines
- that covers mlocking ok, not sure if the rest fits the "shared pages" case though. I dont see any check for other kinds of shared pages in the code.
Agreed. "shared" here seems confused. I've removed it. And I've added mention of "Huge TLB pages" for this error.
Thanks.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
- The word "will result" did sound as a guarantee at least to me. So here it could be changed to "may result (unless the advice is ignored)"?
It's too late to fix documentation. Applications already depends on the beheviour.
Quoted 7 lines
Quoted 2 lines
Right, so as long as they check for EINVAL, it should be safe. It appears that jemalloc does.
So, first a brief question: in the cases where the call does not error out, are we agreed that in the current implementation,
MADV_DONTNEEDwill always result in zero-filled pages when the region is faulted back in (when we consider pages that are not backed by a file)?
I'd agree at this point.
Also we should probably mention anonymously shared pages (shmem). I think they behave the same as file here.
Quoted 4 lines
Quoted 2 lines
I still wouldnt be sure just by reading the man page that the clearing is guaranteed whenever I dont get an error return value, though,
I'm not quite sure what you want here. I mean: if there's an error,
I was just reiterating that the guarantee is not clear from if you consider all the statements in the man page.
Quoted 4 lines
then the DONTNEED action didn't occur, right? Therefore, there won't be zero-filled pages. But, for what it's worth, I added "If the operation succeeds" at the start of that sentence beginning "Subsequent accesses...".
Yes, that should clarify it. Thanks!
Quoted 6 lines
Now, some history, explaining why the page is a bit of a mess, and for that matter why I could really use more help on it from MM folk (especially in the form of actual patches [1], rather than notes about deficiencies in the documentation), because:
***I simply cannot keep up with all of the details***.
I see, and expected it would be like this. I would just send patch if the situation was clear, but here we should agree first, and I thought you should be involved from the beginning.
Quoted 30 lines
Once upon a time (Linux 2.4), there was
madvise()with just 5 flags:
MADV_NORMAL
MADV_RANDOM
MADV_SEQUENTIAL
MADV_WILLNEED
MADV_DONTNEEDAnd already a dozen years ago, *I* added the text about
MADV_DONTNEED. Back then, I believe it was true. I'm not sure if it's still true now, but I assume for the moment that it is, and await feedback. And the text saying that the call does not affect the semantics of memory access dates back even further (and was then true,MADV_DONTNEEDaside).Those 5 flags have analogs in POSIX's
posix_madvise()(albeit, there is a semantic mismatch between the destructiveMADV_DONTNEEDand POSIX's nondestructivePOSIX_MADV_DONTNEED). They also appear on most other implementations.Since the original implementation, numerous pieces of cruft^W^W^W excellent new flags have been overloaded into this one system call. Some of those certainly violated the "does not change the semantics of the application" statement, but, sadly, the kernel developers who implemented
MADV_REMOVEorMADV_DONTFORKdid not think to send a patch to the man page for those new flags, one that might have noted that the semantics of the application are changed by such flags. Equally sadly, I did overlook to scan the bigger page when *I* added documentation of these flags to those pages, otherwise I might have caught that detail.
Quoted 2 lines
So, just to repeat, I could really use more help on it from MM folk in the form of actual patches to the man page.
Thanks for the background. I'll try to remember to check for man-pages part when I review some api changing patch.
Quoted 6 lines
Thanks,
Michael
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Hello Vlastimil,
Quoted 21 lines · Vlastimil Babka
Quoted 20 lines · Michael Kerrisk (man-pages)
Quoted 8 lines · Vlastimil Babka
Quoted 4 lines · Kirill A. Shutemov
On Tue, Feb 03, 2015 at 09:19:15AM +0100, Vlastimil Babka wrote:
It doesn't skip. It fails with
-EINVAL. Or I miss something.Quoted 6 lines
No, I missed that. Thanks for pointing out. The manpage also explains EINVAL in
this case:* The application is attempting to release locked or shared pages (with
MADV_DONTNEED).Yes, there is that. But the page could be more explicit when discussing
MADV_DONTNEEDin the main text. I've done that.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
- that covers mlocking ok, not sure if the rest fits the "shared pages" case
though. I dont see any check for other kinds of shared pages in the code.Agreed. "shared" here seems confused. I've removed it. And I've added mention of "Huge TLB pages" for this error.
Thanks.
I also added those cases for MADV_REMOVE, BTW.
Quoted 12 lines
Quoted 12 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
- The word "will result" did sound as a guarantee at least to me. So here it
could be changed to "may result (unless the advice is ignored)"?It's too late to fix documentation. Applications already depends on the beheviour.
Right, so as long as they check for EINVAL, it should be safe. It appears that
jemalloc does.
Quoted 8 lines
Quoted 5 lines
So, first a brief question: in the cases where the call does not error out,
are we agreed that in the current implementation,MADV_DONTNEEDwill always result in zero-filled pages when the region is faulted back in (when we consider pages that are not backed by a file)?I'd agree at this point.
Thanks for the confirmation.
Quoted 2 lines
Also we should probably mention anonymously shared pages (shmem). I think they behave the same as file here.
You mean tmpfs here, right? (I don't keep all of the synonyms straight.)
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I still wouldnt be sure just by reading the man page that the clearing is guaranteed whenever I dont get an error return value, though,
I'm not quite sure what you want here. I mean: if there's an error,
I was just reiterating that the guarantee is not clear from if you consider all the statements in the man page.
Quoted 6 lines
Quoted 4 lines
then the DONTNEED action didn't occur, right? Therefore, there won't be zero-filled pages. But, for what it's worth, I added "If the operation succeeds" at the start of that sentence beginning "Subsequent accesses...".
Yes, that should clarify it. Thanks!
Okay.
Quoted 10 lines
Quoted 6 lines
Now, some history, explaining why the page is a bit of a mess, and for that matter why I could really use more help on it from MM folk (especially in the form of actual patches [1], rather than notes about deficiencies in the documentation), because:
***I simply cannot keep up with all of the details***.
I see, and expected it would be like this. I would just send patch if the situation was clear, but here we should agree first, and I thought you should be involved from the beginning.
Sorry -- I should have made it clearer, this statement was not targeted at you personally, or even necessarily at this particular thread. It was a general comment, that came up sharply to me as I looked at how much cruft there is in the madvise() page.
Quoted 30 lines
Quoted 30 lines
Once upon a time (Linux 2.4), there was
madvise()with just 5 flags:
MADV_NORMAL
MADV_RANDOM
MADV_SEQUENTIAL
MADV_WILLNEED
MADV_DONTNEEDAnd already a dozen years ago, *I* added the text about
MADV_DONTNEED. Back then, I believe it was true. I'm not sure if it's still true now, but I assume for the moment that it is, and await feedback. And the text saying that the call does not affect the semantics of memory access dates back even further (and was then true,MADV_DONTNEEDaside).Those 5 flags have analogs in POSIX's
posix_madvise()(albeit, there is a semantic mismatch between the destructiveMADV_DONTNEEDand POSIX's nondestructivePOSIX_MADV_DONTNEED). They also appear on most other implementations.Since the original implementation, numerous pieces of cruft^W^W^W excellent new flags have been overloaded into this one system call. Some of those certainly violated the "does not change the semantics of the application" statement, but, sadly, the kernel developers who implemented
MADV_REMOVEorMADV_DONTFORKdid not think to send a patch to the man page for those new flags, one that might have noted that the semantics of the application are changed by such flags. Equally sadly, I did overlook to scan the bigger page when *I* added documentation of these flags to those pages, otherwise I might have caught that detail.
Quoted 5 lines
Quoted 2 lines
So, just to repeat, I could really use more help on it from MM folk in the form of actual patches to the man page.
Thanks for the background. I'll try to remember to check for man-pages part when I review some api changing patch.
That would be great.
Thanks,
Michael
Signature · 7 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/ -- To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Quoted 7 lines · Michael Kerrisk (man-pages)
Hello Vlastimil,
Quoted 4 lines · Vlastimil Babka
Quoted 4 lines
Quoted 3 lines
- that covers mlocking ok, not sure if the rest fits the "shared pages" case
though. I dont see any check for other kinds of shared pages in the code.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Agreed. "shared" here seems confused. I've removed it. And I've added mention of "Huge TLB pages" for this error.
Thanks.
I also added those cases for
MADV_REMOVE, BTW.
Right. There's also the following for MADV_REMOVE that needs updating:
"Currently, only shmfs/tmpfs supports this; other filesystems return with the error ENOSYS."
- it's not just shmem/tmpfs anymore. It should be best to refer to fallocate(2) option FALLOC_FL_PUNCH_HOLE which seems to be (more) up to date.
- AFAICS it doesn't return ENOSYS but EOPNOTSUPP. Also neither error code is listed in the ERRORS section.
Quoted 19 lines
Quoted 19 lines
Quoted 17 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
- The word "will result" did sound as a guarantee at least to me. So here it
could be changed to "may result (unless the advice is ignored)"?It's too late to fix documentation. Applications already depends on the beheviour.
Right, so as long as they check for EINVAL, it should be safe. It appears that
jemalloc does.So, first a brief question: in the cases where the call does not error out,
are we agreed that in the current implementation,MADV_DONTNEEDwill always result in zero-filled pages when the region is faulted back in (when we consider pages that are not backed by a file)?
Quoted 8 lines
Quoted 1 line
I'd agree at this point.
Thanks for the confirmation.
Quoted 2 lines
Also we should probably mention anonymously shared pages (shmem). I think they behave the same as file here.
You mean tmpfs here, right? (I don't keep all of the synonyms straight.)
shmem is tmpfs (that by itself would fit under "files" just fine), but also sys V segments created by shmget(2) and also mappings created by mmap with MAP_SHARED | MAP_ANONYMOUS. I'm not sure if there's a single manpage to refer to the full list.
Thanks,
Vlastimil
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Quoted 27 lines · Vlastimil Babka
Quoted 11 lines · Michael Kerrisk (man-pages)
Hello Vlastimil,
Quoted 7 lines · Vlastimil Babka
Quoted 7 lines
Quoted 5 lines
- that covers mlocking ok, not sure if the rest fits the "shared pages" case
though. I dont see any check for other kinds of shared pages in the code.Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Agreed. "shared" here seems confused. I've removed it. And I've added mention of "Huge TLB pages" for this error.
Thanks.
I also added those cases for
MADV_REMOVE, BTW.Right. There's also the following for
MADV_REMOVEthat needs updating:"Currently, only shmfs/tmpfs supports this; other filesystems return with the error ENOSYS."
Quoted 6 lines
- it's not just shmem/tmpfs anymore. It should be best to refer to fallocate(2) option
FALLOC_FL_PUNCH_HOLEwhich seems to be (more) up to date.- AFAICS it doesn't return ENOSYS but EOPNOTSUPP. Also neither error code is listed in the ERRORS section.
Yup, I recently added that as well, based on a patch from Jan Chaloupka.
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
Quoted 3 lines
- The word "will result" did sound as a guarantee at least to me. So here it
could be changed to "may result (unless the advice is ignored)"?It's too late to fix documentation. Applications already depends on the
beheviour.Right, so as long as they check for EINVAL, it should be safe. It appears
that
jemalloc does.Quoted 7 lines
Quoted 5 lines
So, first a brief question: in the cases where the call does not error out,
are we agreed that in the current implementation,MADV_DONTNEEDwill always result in zero-filled pages when the region is faulted back in (when we consider pages that are not backed by a file)?I'd agree at this point.
Thanks for the confirmation.
Quoted 2 lines
Also we should probably mention anonymously shared pages (shmem). I think they behave the same as file here.
Quoted 6 lines
Quoted 1 line
You mean tmpfs here, right? (I don't keep all of the synonyms straight.)
shmem is tmpfs (that by itself would fit under "files" just fine), but also sys V segments created by shmget(2) and also mappings created by mmap with
MAP_SHARED|MAP_ANONYMOUS. I'm not sure if there's a single manpage to refer to the full list.
So, how about this text:
After a successful MADV_DONTNEED operation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either reloading of the memory contents from the underlying mapped file (for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Thanks,
Michael
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Hello,
Quoted 72 lines · Michael Kerrisk (man-pages)
Quoted 27 lines · Vlastimil Babka
Quoted 11 lines · Michael Kerrisk (man-pages)
Hello Vlastimil,
Quoted 7 lines · Vlastimil Babka
Quoted 7 lines
Quoted 5 lines
- that covers mlocking ok, not sure if the rest fits the "shared pages" case
though. I dont see any check for other kinds of shared pages in the code.Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Agreed. "shared" here seems confused. I've removed it. And I've added mention of "Huge TLB pages" for this error.
Thanks.
I also added those cases for
MADV_REMOVE, BTW.Right. There's also the following for
MADV_REMOVEthat needs updating:"Currently, only shmfs/tmpfs supports this; other filesystems return with the error ENOSYS."
Quoted 6 lines
- it's not just shmem/tmpfs anymore. It should be best to refer to fallocate(2) option
FALLOC_FL_PUNCH_HOLEwhich seems to be (more) up to date.- AFAICS it doesn't return ENOSYS but EOPNOTSUPP. Also neither error code is listed in the ERRORS section.
Yup, I recently added that as well, based on a patch from Jan Chaloupka.
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
Quoted 3 lines
- The word "will result" did sound as a guarantee at least to me. So here it
could be changed to "may result (unless the advice is ignored)"?It's too late to fix documentation. Applications already depends on the
beheviour.Right, so as long as they check for EINVAL, it should be safe. It appears
that
jemalloc does.Quoted 7 lines
Quoted 5 lines
So, first a brief question: in the cases where the call does not error out,
are we agreed that in the current implementation,MADV_DONTNEEDwill always result in zero-filled pages when the region is faulted back in (when we consider pages that are not backed by a file)?I'd agree at this point.
Thanks for the confirmation.
Quoted 2 lines
Also we should probably mention anonymously shared pages (shmem). I think they behave the same as file here.
Quoted 6 lines
Quoted 1 line
You mean tmpfs here, right? (I don't keep all of the synonyms straight.)
shmem is tmpfs (that by itself would fit under "files" just fine), but also sys V segments created by shmget(2) and also mappings created by mmap with
MAP_SHARED|MAP_ANONYMOUS. I'm not sure if there's a single manpage to refer to the full list.So, how about this text:
Quoted 9 lines
After a successful
MADV_DONTNEEDoperation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either reloading of the memory contents from the underlying mapped file (for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Hmm, I'd like to clarify.
Whether it was intention or not, some of userspace developers thought about that syscall drop pages instantly if was no-error return so that they will see more free pages(ie, rss for the process will be decreased) with keeping the VMA. Can we rely on it?
And we should make error section, too.
"locked" covers mlock(2) and you said you will add hugetlb. Then, VM_PFNMAP? In that case, it fails. How can we say about VM_PFNMAP? special mapping for some drivers?
One more thing, "The kernel is free to ignore the advice". It conflicts "This call does not influence the semantics of the application (except in the case of MADV_DONTNEED)" so
is it okay we can believe "The kernel is free to ingmore the advise except MADV_DONTNEED"?
Thanks.
Signature · 8 lines
Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Quoted 85 lines · Minchan Kim
Hello,
Quoted 72 lines · Michael Kerrisk (man-pages)
Quoted 27 lines · Vlastimil Babka
Quoted 11 lines · Michael Kerrisk (man-pages)
Hello Vlastimil,
Quoted 7 lines · Vlastimil Babka
Quoted 7 lines
Quoted 5 lines
- that covers mlocking ok, not sure if the rest fits the "shared pages" case
though. I dont see any check for other kinds of shared pages in the code.Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Agreed. "shared" here seems confused. I've removed it. And I've added mention of "Huge TLB pages" for this error.
Thanks.
I also added those cases for
MADV_REMOVE, BTW.Right. There's also the following for
MADV_REMOVEthat needs updating:"Currently, only shmfs/tmpfs supports this; other filesystems return with the error ENOSYS."
Quoted 6 lines
- it's not just shmem/tmpfs anymore. It should be best to refer to fallocate(2) option
FALLOC_FL_PUNCH_HOLEwhich seems to be (more) up to date.- AFAICS it doesn't return ENOSYS but EOPNOTSUPP. Also neither error code is listed in the ERRORS section.
Yup, I recently added that as well, based on a patch from Jan Chaloupka.
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
Quoted 3 lines
- The word "will result" did sound as a guarantee at least to me. So here it
could be changed to "may result (unless the advice is ignored)"?It's too late to fix documentation. Applications already depends on the
beheviour.Right, so as long as they check for EINVAL, it should be safe. It appears
that
jemalloc does.Quoted 7 lines
Quoted 5 lines
So, first a brief question: in the cases where the call does not error out,
are we agreed that in the current implementation,MADV_DONTNEEDwill always result in zero-filled pages when the region is faulted back in (when we consider pages that are not backed by a file)?I'd agree at this point.
Thanks for the confirmation.
Quoted 2 lines
Also we should probably mention anonymously shared pages (shmem). I think they behave the same as file here.
Quoted 6 lines
Quoted 1 line
You mean tmpfs here, right? (I don't keep all of the synonyms straight.)
shmem is tmpfs (that by itself would fit under "files" just fine), but also sys V segments created by shmget(2) and also mappings created by mmap with
MAP_SHARED|MAP_ANONYMOUS. I'm not sure if there's a single manpage to refer to the full list.So, how about this text:
Quoted 9 lines
After a successful
MADV_DONTNEEDoperation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either reloading of the memory contents from the underlying mapped file (for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Quoted 6 lines
Hmm, I'd like to clarify.
Whether it was intention or not, some of userspace developers thought about that syscall drop pages instantly if was no-error return so that they will see more free
pages(ie, rss for the process will be decreased)with keeping the VMA. Can we rely on it?
I do not know. Michael?
Quoted 4 lines
And we should make error section, too.
"locked" covers mlock(2) and you said you will add hugetlb. Then,VM_PFNMAP? In that case, it fails. How can we say aboutVM_PFNMAP? special mapping for some drivers?
I'm open for offers on what to add.
Quoted 5 lines
One more thing, "The kernel is free to ignore the advice". It conflicts "This call does not influence the semantics of the application (except in the case of
MADV_DONTNEED)" so
is it okay we can believe "The kernel is free to ingmore the advise exceptMADV_DONTNEED"?
I decided to just drop the sentence
The kernel is free to ignore the advice.
It creates misunderstandings, and does not really add information.
Cheers,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Hello, Michael
Quoted 86 lines · Michael Kerrisk (man-pages)
Quoted 85 lines · Minchan Kim
Hello,
Quoted 81 lines · Michael Kerrisk (man-pages)
Quoted 27 lines · Vlastimil Babka
Quoted 11 lines · Michael Kerrisk (man-pages)
Hello Vlastimil,
Quoted 7 lines · Vlastimil Babka
Quoted 7 lines
Quoted 5 lines
- that covers mlocking ok, not sure if the rest fits the "shared pages" case
though. I dont see any check for other kinds of shared pages in the code.Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Agreed. "shared" here seems confused. I've removed it. And I've added mention of "Huge TLB pages" for this error.
Thanks.
I also added those cases for
MADV_REMOVE, BTW.Right. There's also the following for
MADV_REMOVEthat needs updating:"Currently, only shmfs/tmpfs supports this; other filesystems return with the error ENOSYS."
Quoted 6 lines
- it's not just shmem/tmpfs anymore. It should be best to refer to fallocate(2) option
FALLOC_FL_PUNCH_HOLEwhich seems to be (more) up to date.- AFAICS it doesn't return ENOSYS but EOPNOTSUPP. Also neither error code is listed in the ERRORS section.
Yup, I recently added that as well, based on a patch from Jan Chaloupka.
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
Quoted 3 lines
- The word "will result" did sound as a guarantee at least to me. So here it
could be changed to "may result (unless the advice is ignored)"?It's too late to fix documentation. Applications already depends on the
beheviour.Right, so as long as they check for EINVAL, it should be safe. It appears
that
jemalloc does.Quoted 7 lines
Quoted 5 lines
So, first a brief question: in the cases where the call does not error out,
are we agreed that in the current implementation,MADV_DONTNEEDwill always result in zero-filled pages when the region is faulted back in (when we consider pages that are not backed by a file)?I'd agree at this point.
Thanks for the confirmation.
Quoted 2 lines
Also we should probably mention anonymously shared pages (shmem). I think they behave the same as file here.
Quoted 6 lines
Quoted 1 line
You mean tmpfs here, right? (I don't keep all of the synonyms straight.)
shmem is tmpfs (that by itself would fit under "files" just fine), but also sys V segments created by shmget(2) and also mappings created by mmap with
MAP_SHARED|MAP_ANONYMOUS. I'm not sure if there's a single manpage to refer to the full list.So, how about this text:
After a successful
MADV_DONTNEEDoperation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either reloading of the memory contents from the underlying mapped file (for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Quoted 8 lines
Quoted 6 lines
Hmm, I'd like to clarify.
Whether it was intention or not, some of userspace developers thought about that syscall drop pages instantly if was no-error return so that they will see more free
pages(ie, rss for the process will be decreased)with keeping the VMA. Can we rely on it?I do not know. Michael?
It's important to identify difference between MADV_DONTNEED and MADV_FREE so it would be better to clear out in this chance.
Quoted 7 lines
Quoted 5 lines
And we should make error section, too.
"locked" covers mlock(2) and you said you will add hugetlb. Then,VM_PFNMAP? In that case, it fails. How can we say aboutVM_PFNMAP? special mapping for some drivers?I'm open for offers on what to add.
I suggests from quote "LWN" http://lwn.net/Articles/162860/
"*special mapping* which is not made up of "normal" pages. It is usually created by device drivers which map special memory areas into user space"
Quoted 7 lines
Quoted 6 lines
One more thing, "The kernel is free to ignore the advice". It conflicts "This call does not influence the semantics of the application (except in the case of
MADV_DONTNEED)" so
is it okay we can believe "The kernel is free to ingmore the advise exceptMADV_DONTNEED"?
Quoted 5 lines
I decided to just drop the sentence
The kernel is free to ignore the advice.
It creates misunderstandings, and does not really add information.
Sounds good.
Quoted 9 lines
Cheers,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Signature · 2 lines
Kind regards, Minchan Kim
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Hello Minchan
Quoted 9 lines · Minchan Kim
Hello, Michael
Quoted 6 lines · Michael Kerrisk (man-pages)
Quoted 5 lines · Minchan Kim
Hello,
Quoted 2 lines · Michael Kerrisk (man-pages)
Quoted 1 line · Vlastimil Babka
On 02/04/2015 03:00 PM, Michael Kerrisk (man-pages) wrote:
[...]
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
And we should make error section, too.
"locked" covers mlock(2) and you said you will add hugetlb. Then,VM_PFNMAP? In that case, it fails. How can we say aboutVM_PFNMAP? special mapping for some drivers?I'm open for offers on what to add.
I suggests from quote "LWN" http://lwn.net/Articles/162860/
"*special mapping* which is not made up of "normal" pages. It is usually created by device drivers which map special memory areas into user space"
Thanks. I've added mention of VM_PFNMAP in the discussion of both MADV_DONTNEED and MADV_REMOVE, and noted that both of those operations will give an error when applied to VM_PFNMAP pages.
Cheers,
Michael
Signature · 9 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
In reply to Michael Kerrisk (man-pages)On Wed 04-02-15 20:24:27, Michael Kerrisk wrote:
[...]
Quoted 7 lines
So, how about this text:
After a successful
MADV_DONTNEEDoperation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either reloading of the memory contents from the underlying mapped file
"
result in either providing the up-to-date contents of the underlying mapped file
"
Would be more precise IMO because reload might be interpreted as a major fault which is not necessarily the case (see below).
Quoted 4 lines
(for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Yes, this wording is better because many users are not aware of MAP_ANON|MAP_SHARED being file backed in fact and mmap man page doesn't mention that.
I am just wondering whether it makes sense to mention that MADV_DONTNEED for shared mappings might be surprising and not freeing the backing pages thus not really freeing memory until there is a memory pressure. But maybe this is too implementation specific for a man page. What about the following wording on top of yours?
"
Please note that the MADV_DONTNEED hint on shared mappings might not lead to immediate freeing of pages in the range. The kernel is free to delay this until an appropriate moment. RSS of the calling process will be reduced however.
"
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Hi Michael
Quoted 10 lines · Michal Hocko
On Wed 04-02-15 20:24:27, Michael Kerrisk wrote:
[...]Quoted 7 lines
So, how about this text:
After a successful
MADV_DONTNEEDoperation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either reloading of the memory contents from the underlying mapped file
Quoted 4 lines
"
result in either providing the up-to-date contents of the underlying mapped file
"
Thanks! I did something like that. See below.
Quoted 11 lines
Would be more precise IMO because reload might be interpreted as a major fault which is not necessarily the case (see below).
Quoted 4 lines
(for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Yes, this wording is better because many users are not aware of MAP_ANON|MAP_SHARED being file backed in fact and mmap man page doesn't mention that.
(Michal, would you have a text to propose to add to the mmap(2) page? Maybe it would be useful to add something there.)
Quoted 12 lines
I am just wondering whether it makes sense to mention that
MADV_DONTNEEDfor shared mappings might be surprising and not freeing the backing pages thus not really freeing memory until there is a memory pressure. But maybe this is too implementation specific for a man page. What about the following wording on top of yours?
"
Please note that theMADV_DONTNEEDhint on shared mappings might not lead to immediate freeing of pages in the range. The kernel is free to delay this until an appropriate moment. RSS of the calling process will be reduced however.
"
Thanks! I added this, but dropped in the word "immediately" in the last sentence, since I assume that was implied. So now we have:
After a successful MADV_DONTNEED operation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either repopulating the mem‐ ory contents from the up-to-date contents of the under‐ lying mapped file (for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Note that, when applied to shared mappings, MADV_DONT‐ NEED might not lead to immediate freeing of the pages in
the range. The kernel is free to delay freeing the
pages until an appropriate moment. The resident set
size (RSS) of the calling process will be immediately reduced however.
The current draft of the page can be found in a branch,
http://git.kernel.org/cgit/docs/man-pages/man-pages.git/log/?h=draft_madvise
Thanks,
Michael
Signature · 9 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
On Fri 06-02-15 16:57:50, Michael Kerrisk wrote:
[...]
Quoted 6 lines
Quoted 3 lines
Yes, this wording is better because many users are not aware of MAP_ANON|MAP_SHARED being file backed in fact and mmap man page doesn't mention that.
(Michal, would you have a text to propose to add to the mmap(2) page? Maybe it would be useful to add something there.)
I am half way on vacation, but I can cook a patch after I am back after week.
Quoted 25 lines
Quoted 11 lines
I am just wondering whether it makes sense to mention that
MADV_DONTNEEDfor shared mappings might be surprising and not freeing the backing pages thus not really freeing memory until there is a memory pressure. But maybe this is too implementation specific for a man page. What about the following wording on top of yours?
"
Please note that theMADV_DONTNEEDhint on shared mappings might not lead to immediate freeing of pages in the range. The kernel is free to delay this until an appropriate moment. RSS of the calling process will be reduced however.
"Thanks! I added this, but dropped in the word "immediately" in the last sentence, since I assume that was implied. So now we have:
After a successful
MADV_DONTNEEDoperation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either repopulating the mem‐ ory contents from the up-to-date contents of the under‐ lying mapped file (for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Quoted 6 lines
Note that, when applied to shared mappings, MADV_DONT‐ NEED might not lead to immediate freeing of the pages in
the range. The kernel is free to delay freeing the
pages until an appropriate moment. The resident set
size (RSS) of the calling process will be immediately reduced however.
This sounds good to me and it is definitely much better than the current state. Thanks!
Quoted 13 lines
The current draft of the page can be found in a branch,
http://git.kernel.org/cgit/docs/man-pages/man-pages.git/log/?h=draft_madviseThanks,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Signature · 2 lines
Michal Hocko SUSE Labs
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
In reply to Michael Kerrisk (man-pages)Quoted 61 lines · Michael Kerrisk (man-pages)
Hi Michael
Quoted 10 lines · Michal Hocko
On Wed 04-02-15 20:24:27, Michael Kerrisk wrote:
[...]Quoted 7 lines
So, how about this text:
After a successful
MADV_DONTNEEDoperation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either reloading of the memory contents from the underlying mapped fileQuoted 4 lines
"
result in either providing the up-to-date contents of the underlying mapped file
"Thanks! I did something like that. See below.
Quoted 11 lines
Would be more precise IMO because reload might be interpreted as a major fault which is not necessarily the case (see below).
Quoted 4 lines
(for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Yes, this wording is better because many users are not aware of MAP_ANON|MAP_SHARED being file backed in fact and mmap man page doesn't mention that.
(Michal, would you have a text to propose to add to the mmap(2) page? Maybe it would be useful to add something there.)
Quoted 12 lines
I am just wondering whether it makes sense to mention that
MADV_DONTNEEDfor shared mappings might be surprising and not freeing the backing pages thus not really freeing memory until there is a memory pressure. But maybe this is too implementation specific for a man page. What about the following wording on top of yours?
"
Please note that theMADV_DONTNEEDhint on shared mappings might not lead to immediate freeing of pages in the range. The kernel is free to delay this until an appropriate moment. RSS of the calling process will be reduced however.
"Thanks! I added this, but dropped in the word "immediately" in the last sentence, since I assume that was implied. So now we have:
After a successful
MADV_DONTNEEDoperation, the seman‐ tics of memory access in the specified region are
changed: subsequent accesses of pages in the range will succeed, but will result in either repopulating the mem‐ ory contents from the up-to-date contents of the under‐ lying mapped file (for shared file mappings, shared anonymous mappings, and shmem-based techniques such as System V shared memory segments) or zero-fill-on-demand pages for anonymous private mappings.
Quoted 6 lines
Note that, when applied to shared mappings, MADV_DONT‐ NEED might not lead to immediate freeing of the pages in
the range. The kernel is free to delay freeing the
pages until an appropriate moment. The resident set
size (RSS) of the calling process will be immediately reduced however.
Looks good. So, I can parse it that anonymous private mappings will lead to immediate freeing of the pages in the range so it's clearly different with MADV_FREE.
Quoted 14 lines
The current draft of the page can be found in a branch,
http://git.kernel.org/cgit/docs/man-pages/man-pages.git/log/?h=draft_madviseThanks,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Signature · 6 lines
Kind regards, Minchan Kim -- To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
In reply to Vlastimil BabkaQuoted 42 lines · Vlastimil Babka
Quoted 20 lines · Kirill A. Shutemov
Quoted 19 lines · Vlastimil Babka
[CC linux-api, man pages]
Quoted 15 lines · Dave Hansen
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.The manpage, at least, claims that we zero-fill after
MADV_DONTNEEDis called:Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.So if we have anything depending on the behavior that it's _always_ zero-filled after an
MADV_DONTNEED, this will break it.Quoted 8 lines
Quoted 6 lines
OK, so that's a third person (including me) who understood it as a zero-fill guarantee. I think the man page should be clarified (if it's indeed not guaranteed), or we have a bug.
The implementation actually skips
MADV_DONTNEEDfor VM_LOCKED|VM_HUGETLB|VM_PFNMAP vma's.It doesn't skip. It fails with
-EINVAL. Or I miss something.No, I missed that. Thanks for pointing out. The manpage also explains EINVAL in this case:
* The application is attempting to release locked or shared pages (with
MADV_DONTNEED).- that covers mlocking ok, not sure if the rest fits the "shared pages" case though. I dont see any check for other kinds of shared pages in the code.
Quoted 3 lines
Quoted 2 lines
- The word "will result" did sound as a guarantee at least to me. So here it could be changed to "may result (unless the advice is ignored)"?
Quoted 9 lines
Quoted 2 lines
It's too late to fix documentation. Applications already depends on the beheviour.
Right, so as long as they check for EINVAL, it should be safe. It appears that jemalloc does.
I still wouldnt be sure just by reading the man page that the clearing is guaranteed whenever I dont get an error return value, though,
IMHO,
Man page said
"MADV_DONTNEED: Subsequent accesses of pages in this range will succeed,
but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file."
Heap by allocated by malloc(3) is anonymous page so it's a mapping withtout an underlying file so userspace can expect zero-fill.
Man page said
"EINVAL: The application is attempting to release locked or shared pages (with MADV_DONTNEED)"
So, user can expect the call on area by allocated by malloc(3) if he doesn't call mlock will always be successful.
Man page said
"madivse: This call does not influence the semantics of the application (except in the case of MADV_DONTNEED)"
So, we shouldn't break MADV_DONTNEED's semantic which free pages instantly. It's a long time semantic and it was one of arguable issues on MADV_FREE Rik had tried long time ago to replace MADV_DONTNEED with MADV_FREE.
Signature · 2 lines
Kind regards, Minchan Kim
Mel Gorman
reviewerRe: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
In reply to Vlastimil BabkaQuoted 19 lines · Vlastimil Babka
[CC linux-api, man pages]
Quoted 8 lines · Dave Hansen
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.The manpage, at least, claims that we zero-fill after
MADV_DONTNEEDis called:Quoted 7 lines
Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.So if we have anything depending on the behavior that it's _always_ zero-filled after an
MADV_DONTNEED, this will break it.
Quoted 7 lines
OK, so that's a third person (including me) who understood it as a zero-fill guarantee. I think the man page should be clarified (if it's indeed not guaranteed), or we have a bug.
The implementation actually skips
MADV_DONTNEEDfor VM_LOCKED|VM_HUGETLB|VM_PFNMAP vma's.
This was the first reason why I did not consider the zero-filling to be a guarantee. That said, at this point I'm also not considering pushing this patch towards the kernel. I agree that this is a glibc bug so I've dropped a line to some glibc people to see what they think the approach should be.
Quoted 5 lines
I'm not sure about
VM_PFNMAP, these are probably special enough. For mlock, one could expect that mlocking andMADV_DONTNEEDwould be in some opposition, but it's not documented in the manpage AFAIK. Neither is the hugetlb case, which could be really unexpected by the user.
The equivalent posix page also lacks details on how exactly this flag should behave. hugetlb is sortof special in that it's always backed by a ram-based file where the contents can be refaulted. It gets hairy when the mapping has been created to look anonymous but is not anonymous really. The semantics of hugetlb have always been fuzzy.
Quoted 6 lines
Next, what the man page says about guarantees:
"The kernel is free to ignore the advice."
- that would suggest that nothing is guaranteed
Yep, another reason why I did not clear the page when ignoring the hint.
Quoted 10 lines
"This call does not influence the semantics of the application (except in the case of
MADV_DONTNEED)"- that depends if the reader understands it as "does influence by
MADV_DONTNEED" or "may influence byMADV_DONTNEED"- btw, isn't
MADV_DONTFORKanother exception that does influence the semantics? And since it's mentioned as a workaround for some hardware, is it OK to ignore this advice?
MADV_DONTFORK is also a Linux-specific extention. It happens to be one that if it gets ignored then the application will be very surprised.
Quoted 9 lines
And the part you already cited:
"Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill on-demand pages for mappings without an underlying file."
- The word "will result" did sound as a guarantee at least to me. So here it could be changed to "may result (unless the advice is ignored)"?
The wording should be "may result" as there are circumstances where it gets ignored even without this prototype patch.
Quoted 4 lines
And if we agree that there is indeed no guarantee, what's the actual semantic difference from
MADV_FREE? I guess none? So there's only a possible perfomance difference?
Timing. MADV_DONTNEED if it has an effect is immediate, is a heavier operations and RSS is reduced. MADV_FREE only has an impact in the future if there is memory pressure.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Quoted 1 line · Mel Gorman
On Tue, Feb 03, 2015 at 09:19:15AM +0100, Vlastimil Babka wrote:
[...]
Quoted 8 lines
Quoted 4 lines
And if we agree that there is indeed no guarantee, what's the actual semantic difference from
MADV_FREE? I guess none? So there's only a possible perfomance difference?Timing.
MADV_DONTNEEDif it has an effect is immediate, is a heavier operations and RSS is reduced.MADV_FREEonly has an impact in the future if there is memory pressure.
JFTR. the man page for MADV_FREE has been proposed already
(https://lkml.org/lkml/2014/12/5/63 should be the last version AFAIR). I do not see it in the man-pages git tree but the patch was not in time for 3.19 so I guess it will only appear in 3.20.
Signature · 6 lines
Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: MADV_DONTNEED semantics? Was: [RFC PATCH] mm: madvise: Ignore repeated MADV_DONTNEED hints
Quoted 8 lines · Michal Hocko
Quoted 1 line · Mel Gorman
On Tue, Feb 03, 2015 at 09:19:15AM +0100, Vlastimil Babka wrote:
[...]
Quoted 5 lines
Quoted 4 lines
And if we agree that there is indeed no guarantee, what's the actual semantic difference from
MADV_FREE? I guess none? So there's only a possible perfomance difference?
Quoted 9 lines
Quoted 3 lines
Timing.
MADV_DONTNEEDif it has an effect is immediate, is a heavier operations and RSS is reduced.MADV_FREEonly has an impact in the future if there is memory pressure.JFTR. the man page for
MADV_FREEhas been proposed already
(https://lkml.org/lkml/2014/12/5/63 should be the last version AFAIR). I do not see it in the man-pages git tree but the patch was not in time for 3.19 so I guess it will only appear in 3.20.
Yikes! That patch was buried in the bottom of a locked filing cabinet in a disused lavatory. I unfortunately don't read every thread that comes my way, especially if it doesn't look like a man-pages patch (i.e., falls in the middle of an LKML thread that starts on another topic, and doesn't see linux-man@). I'll respond to that patch soon. (There are some problems that mean I could not accept it, AFAICT.)
Thanks,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
Mel Gorman
reviewerQuoted 8 lines · Dave Hansen
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.The manpage, at least, claims that we zero-fill after
MADV_DONTNEEDis called:
It also claims that the kernel is free to ignore the advice.
Quoted 7 lines
Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.So if we have anything depending on the behavior that it's _always_ zero-filled after an
MADV_DONTNEED, this will break it.
True. I'd be surprised if any application depended on that but to be safe, an ignored hint could clear the pages. It would still be cheaper than a full teardown and refault.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 17 lines · Mel Gorman
Quoted 8 lines · Dave Hansen
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.The manpage, at least, claims that we zero-fill after
MADV_DONTNEEDis called:It also claims that the kernel is free to ignore the advice.
Quoted 5 lines
Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.
Quoted 4 lines
Quoted 2 lines
So if we have anything depending on the behavior that it's _always_ zero-filled after an
MADV_DONTNEED, this will break it.True. I'd be surprised if any application depended on that
IIUC, jemalloc depends on this[1].
[1] https://github.com/jemalloc/jemalloc/blob/dev/src/chunk_mmap.c#L117
Signature · 7 lines
Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Mel Gorman
reviewerQuoted 18 lines · Kirill A. Shutemov
Quoted 17 lines · Mel Gorman
Quoted 8 lines · Dave Hansen
Quoted 3 lines · Mel Gorman
This patch identifies when a thread is frequently calling
MADV_DONTNEEDon the same region of memory and starts ignoring the hint. On an 8-core single-socket machine this was the impact on ebizzy using glibc 2.19.The manpage, at least, claims that we zero-fill after
MADV_DONTNEEDis called:It also claims that the kernel is free to ignore the advice.
Quoted 5 lines
Quoted 4 lines
MADV_DONTNEED
Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file.
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
So if we have anything depending on the behavior that it's _always_ zero-filled after an
MADV_DONTNEED, this will break it.True. I'd be surprised if any application depended on that
IIUC, jemalloc depends on this[1].
[1] https://github.com/jemalloc/jemalloc/blob/dev/src/chunk_mmap.c#L117
Hope they never back regions with hugetlb then or fall apart if the process called mlockall
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>