thread stacks and strict vm overcommit accounting
Hello,
This question is relevent to 2.6.20.
I noticed that if the RSS for the stack size is say, 8MB, running a single-threaded process doesn't incur an increase of 8MB to Committed_AS (/proc/meminfo).
However, on multi-threaded apps linked with pthread (on Debian Etch with 2.6.20 vanilla x86_64), every thread will incur the the specified maximum stack size RSS (assuming that you use the default attr). In other words, it appears that vm accounting works differently in that case.
Is this the intended behaviour?
Signature · 3 lines
Dan Aloni XIV LTD, http://www.xivstorage.com da-x (at) monatomic.org, dan (at) xiv.co.il
Quoted 9 lines · On Tue, 13 Mar 2007 18:33:20 +0200 Dan Aloni <[email protected]> wrote:
Hello,
This question is relevent to 2.6.20.
I noticed that if the RSS for the stack size is say, 8MB, running a single-threaded process doesn't incur an increase of 8MB to
Committed_AS(/proc/meminfo).
Quoted 7 lines
However, on multi-threaded apps linked with pthread (on Debian Etch with 2.6.20 vanilla
x86_64), every thread will incur the the specified maximum stack size RSS (assuming that you use the default attr). In other words, it appears that vm accounting works differently in that case.Is this the intended behaviour?
That sounds like a bug to me.
Quoted 5 lines · Andrew Morton
Quoted 5 lines · On Tue, 13 Mar 2007 18:33:20 +0200 Dan Aloni <[email protected]> wrote:
This question is relevent to 2.6.20.
I noticed that if the RSS for the stack size is say, 8MB, running
I think you meant to say RLIMIT_STACK rather than RSS, didn't you, Dan?
Quoted 2 lines
Quoted 2 lines
a single-threaded process doesn't incur an increase of 8MB to
Committed_AS(/proc/meminfo).
Stack RSS should certainly be included in Committed_AS, but RLIMIT_STACK merely limits how big the stack vma may grow to:
at any moment the stack vma is probably very much smaller, and only its current size is accounted in Committed_AS.
Quoted 6 lines
Quoted 6 lines
However, on multi-threaded apps linked with pthread (on Debian Etch with 2.6.20 vanilla
x86_64), every thread will incur the the specified maximum stack size RSS (assuming that you use the default attr). In other words, it appears that vm accounting works differently in that case.
I'm guessing that the pthread stacks are mmap'ed as greatest extents (probably because that's the easiest way to keep them apart), rather than as small MAP_GROWSDOWN areas to be expanded later on fault. If so, then those would indeed account the maximum in Committed_AS.
Quoted 4 lines
Quoted 2 lines
Is this the intended behaviour?
That sounds like a bug to me.
I'm suspecting it's an oddity rather than a bug.
Hugh
Quoted 3 lines · Hugh Dickins
I'm guessing that the pthread stacks are mmap'ed as greatest extents (probably because that's the easiest way to keep them apart), rather than as small
MAP_GROWSDOWNareas to be expanded later on fault.
Please all, forget about MAP_GROWSDOWN. It's useless. If thread stacks are not completely mapped (address space allocation, memory allocation is not needed) it means subsequent unrelated mmaps can fall into the address space which is meant to be used for the stack, hence preventing the stack from growing.
libpthread uses an mmap for the complete stack size all the time and this of course is accounted for in the kernel.
Quoted 4 lines
Stack RSS should certainly be included in
Committed_AS, butRLIMIT_STACKmerely limits how big the stack vma may grow to:
at any moment the stack vma is probably very much smaller, and only its current size is accounted inCommitted_AS.
With a typical size as a fuzz factor preaccounted in later kernels.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Is this the intended behaviour?
That sounds like a bug to me.
I'm suspecting it's an oddity rather than a bug.
It is intended behaviour.
Quoted 6 lines · On Thu, 15 Mar 2007 23:33:43 +0000 Alan Cox <[email protected]> wrote:
Quoted 4 lines
Stack RSS should certainly be included in
Committed_AS, butRLIMIT_STACKmerely limits how big the stack vma may grow to:
at any moment the stack vma is probably very much smaller, and only its current size is accounted inCommitted_AS.With a typical size as a fuzz factor preaccounted in later kernels.
Where's that done?
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Is this the intended behaviour?
That sounds like a bug to me.
I'm suspecting it's an oddity rather than a bug.
It is intended behaviour.
Each instance of
main()
{
sleep(100);
}appears to increase Committed_AS by around 200kb. But we've committed to providing it with 8MB for stack.
How come this is correct?
Quoted 11 lines · Andrew Morton
Quoted 6 lines · On Thu, 15 Mar 2007 23:33:43 +0000 Alan Cox <[email protected]> wrote:
Quoted 4 lines
Stack RSS should certainly be included in
Committed_AS, butRLIMIT_STACKmerely limits how big the stack vma may grow to:
at any moment the stack vma is probably very much smaller, and only its current size is accounted inCommitted_AS.With a typical size as a fuzz factor preaccounted in later kernels.
Where's that done?
I don't know what Alan is referring to there.
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Is this the intended behaviour?
That sounds like a bug to me.
I'm suspecting it's an oddity rather than a bug.
It is intended behaviour.
Intended in the way the different stacks are implemented, but odd enough for us to wonder at the difference.
Quoted 4 lines
Each instance of
main()
Quoted 8 lines
{ sleep(100); }appears to increase
Committed_ASby around 200kb. But we've committed to providing it with 8MB for stack.How come this is correct?
We've no more committed to providing each instance with 8MB of stack, than we've committed to providing each instance with RLIMIT_AS of address space. The rlimits are limits, not commitments, surely?
Hugh
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
With a typical size as a fuzz factor preaccounted in later kernels.
Where's that done?
I don't know what Alan is referring to there.
fs/exec.c - we add 20 pages to the stack vma size initially.
Quoted 3 lines
We've no more committed to providing each instance with 8MB of stack, than we've committed to providing each instance with
RLIMIT_ASof address space. The rlimits are limits, not commitments, surely?
Yes, its just that the C programming language is utterly and mindbogglingly broken when it comes to resource exhaustion for the stack.
Alan
Quoted 8 lines · Hugh Dickins
Quoted 4 lines
appears to increase
Committed_ASby around 200kb. But we've committed to providing it with 8MB for stack.How come this is correct?
We've no more committed to providing each instance with 8MB of stack, than we've committed to providing each instance with
RLIMIT_ASof address space. The rlimits are limits, not commitments, surely?
RLIMIT_STACK only applies to the initial thread, POSIX threads have just stack size attribute, not maximum thread stack size attribute. If you set it explicitly with pthread_attr_setstacksize, then libpthread will honor whatever thread stack size you want, otherwise it just uses some default thread stack size. This happens to be in NPTL derived from RLIMIT_STACK value, but very well could be something else.
Jakub
Quoted 12 lines · Andrew Morton
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Is this the intended behaviour?
That sounds like a bug to me.
I'm suspecting it's an oddity rather than a bug.
It is intended behaviour.
Each instance of
main()
Quoted 8 lines
{ sleep(100); }appears to increase
Committed_ASby around 200kb. But we've committed to providing it with 8MB for stack.How come this is correct?
Perhaps it makes a lot of sense if you regard stack growth at the same sense that you regard heap growth by the means of brk().
Just by the fact that the stack is limited on default and RLIMIT_DATA is unlimited, doesn't mean the we need to account for the maximum stack size.
Perhaps for embedded systems where you want to have overcommit_memory=2 overcommit_ratio=100 and no swap (for design constraints), just to make sure that allocations fail *always before* OOM gets triggered (and therefore OOM never gets triggered, thankfully), it would have been useful to look at Commited_AS to realize how much the system is close to the maximum memory utilization potential.
Learning about this 'oddity' in Commited_AS, I'd guess it would be better for me not to rely on it for measurements and perhaps tweak smaller values of RSS_STACK for processes on that embedded system.
Signature · 3 lines
Dan Aloni XIV LTD, http://www.xivstorage.com da-x (at) monatomic.org, dan (at) xiv.co.il
Quoted 15 lines · On Thu, 15 Mar 2007 11:06:21 -0800 Andrew Morton <[email protected]>…
Quoted 15 lines · On Tue, 13 Mar 2007 18:33:20 +0200 Dan Aloni <[email protected]> wrote:
Hello,
This question is relevent to 2.6.20.
I noticed that if the RSS for the stack size is say, 8MB, running a single-threaded process doesn't incur an increase of 8MB to
Committed_AS(/proc/meminfo).However, on multi-threaded apps linked with pthread (on Debian Etch with 2.6.20 vanilla
x86_64), every thread will incur the the specified maximum stack size RSS (assuming that you use the default attr). In other words, it appears that vm accounting works differently in that case.
Quoted 3 lines
Quoted 1 line
Is this the intended behaviour?
That sounds like a bug to me.
AFAIK, "main" thread's stack is marked as VM_GROWS?? and its size can be changed dynamically. "other" threads' stack are alloced by mmap (or malloc maybe) and it never grows. This is difference between multi-thread and single thread.
So, you should be carefull to the size of stack when you use multi-threaded apps and vm_overcommit_ratio at the same time. Because MAP_NORESERVE is accounted if sysctl_overcommit_memory == OVERCOMMIT_NEVER, a program like java will fail to create a new thread sometimes.
I have no good idea to fix this difference, sorry.
-Kame