Detailed Stack Information Patch [0/3]
Hi,
this is a patch which give you a better overview of the userland application stack usage, especially for embedded linux.
Currently you are only able to dump the main process/thread stack usage which is showed in proc/pid/status by the "VmStk" Value. But you get no information about the consumed stack memory of the the threads.
For some reasons it becomes important to know how much memory is consumed by each thread:
- Get out of virtual memory by creating a lot of threads (f.e. the developer did assign each of them the default size)
- Misuse the thread stack for big temporary data buffers
- Thread stack overruns
So this patch gives the developer an important tool to figure out if there is one of this issues.
The patch is splitted in three parts.
Part 1 :
--------
fs/exec.c | 4 ++++ fs/proc/array.c | 22 ++++++++++++++++++++++ fs/proc/task_mmu.c | 12 ++++++++++++ include/linux/sched.h | 3 +++ init/Kconfig | 12 ++++++++++++ kernel/fork.c | 5 +++++ 6 files changed, 58 insertions(+)
This is an enhancement in the /proc/<pid>/tasks/<tid>/maps and smaps which marks the mapping where the thread stack pointer reside with "[thread stack]".
Also there is a new entry "stack usage" in proc/pid/status, which will you give the current stack usage.
This feature will be enabled the which "enable /proc/<pid> stack monitoring" under "General setup-->"
Part 2:
-------
fs/proc/Makefile | 1 fs/proc/stackmon.c | 254 +++++++++++++++++++++++++++++++++++++++++++++++++++++ init/Kconfig | 10 ++ 3 files changed, 265 insertions(+)
This enable a new /proc/stackmon file entry. A cat /proc/stackmon will produce a output like:
bytes pages maxpages vm_start vm_end processid threadid name
436 1 1 afdbf000-afdd4000 pid: 409 tid: 409 syslogd
1168 1 1 afd12000-afd27000 pid: 411 tid: 411 sh
516 1 1 afe6c000-afe81000 pid: 412 tid: 412 getty
4580 2 2 af918000-af92d000 pid: 419 tid: 419 catThe first value is the current effektive stack usage in bytes.
The second value is the current real stack usage in pages. This means how many pages are really occupied by the stack.
The thrird value is the maximum real stack usage in pages. This value will be determinate by the difference of the highest used page in the mapping and the page of stack start address.
The fourth value is the start- and end- address of the mapping where the stack currently reside.
The fifth value process id.
The sixth value is thread id.
And the seventh and last entry is the name of the process.
This feature will be enabled the which "enable /proc/<pid> stack monitoring" under "General setup-->".
Part 3:
-------
init/Kconfig | 13 ++ mm/Makefile | 1 mm/stackdbg.c | 319 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 332 insertions(+)
There is also an additional stack monitor which can be enabled by boot time or by the /sys filesystem. Which this you are able to detect if a application use more stack than a given value. If the application exceeds this value, it will receive a SIGTRAP signal.
If there is a debugger attached at this time, there is the ability to examinate the stack usage. Otherwise the application will be terminated. In both cases a kernel log entry "pid:%d (%s) tid:%d stack size %lu exceeds max stack size." will be written.
There are following entries under /sys/kernel/stackmon to control the monitor.
mode:
Setting this to an value not equal zero will start the stack monitoring
thread. Default is 0. Setting it to zero will stop the kernel thread.
stacksize:
This value is the stack size in kb which triggers a SIGTRAP to the
application, if the stack usage is equal or more. Default is 256 kb.
ticks:
Number of ticks between the monitoring invocation. A higher value will
give a less change to trigger a stack over usage, but will also result in a less CPU usage. Default is 1.
All this parameters can also set at boot time with the kernel parameter "stackmon=<stacksize>:<mode>:<ticks>".
The stack monitor can also be compiled as a module.
This patch is against 2.6.29. The patch is cpu independent, so it should work on all linux supported architectures, it was tested under x86 and powerpc. There is no dependency to a library: glibc, uclibc and all other should work.
I hope you like it and want ask for inclusion into the kernel or linux-next? Please give it a try.
If you have ideas how to do things in a better way, please let me know.
Have a nice day,
Stefani
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Stefani Seibold
- Get out of virtual memory by creating a lot of threads (f.e. the developer did assign each of them the default size)
The application just fails then? I don't think that needs a new monitoring tool.
Quoted 1 line
- Misuse the thread stack for big temporary data buffers
That would be better checked for at compile time
(except for alloca, but that is quite rare)
Quoted 1 line
- Thread stack overruns
Your method would be racy at best to determine this because you don't keep track of the worst case, only the current case.
So e.g. if you monitoring app checks once per second the stack could overflow between your monitoring intervals, but already have bounced back before the checker comes in.
gcc has support to generate stack overflow checking code, that would be more reliable. Alternatively you could keep track of consumption in the VMA that has the stack, but that can't handle very large jumps (like f() { char x[1<<30]; } ) The later can only be handled well by the compiler.
-Andi
Signature · 7 lines
[email protected] -- Speaking for myself only. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Andi,
Am Dienstag, den 31.03.2009, 17:49 +0200 schrieb Andi Kleen:
Quoted 8 lines
Quoted 3 lines · Stefani Seibold
- Get out of virtual memory by creating a lot of threads (f.e. the developer did assign each of them the default size)
The application just fails then? I don't think that needs a new monitoring tool.
First, this patch is not only a monitoring tool. Only the last part 3/3 is the monitoring tool.
Patch 1/3 enhance the the proc/<pid>/task/<tid>/maps by the marking the thread stack.
Patch 2/3 gives you an overview of the current process/thread stack usage with the /proc/stackmon entry.
Quoted 4 lines
Quoted 1 line
- Misuse the thread stack for big temporary data buffers
That would be better checked for at compile time
(except for alloca, but that is quite rare)
Fine but it did not work for functions like:
void foo(int n)
{
char buf[n*1024];}
This is valid with gcc.
Quoted 10 lines
Quoted 2 lines
- Thread stack overruns
Your method would be racy at best to determine this because you don't keep track of the worst case, only the current case.
So e.g. if you monitoring app checks once per second the stack could overflow between your monitoring intervals, but already have bounced back before the checker comes in.
The Monitor is part 3/3. And you are right it is not a complete rock solid solution. But it works in many cases and thats is what counts.
Quoted 4 lines
Alternatively you could keep
track of consumption in the VMA that has the stack, but that can't handle very large jumps (likef(){ char x[1<<30]; } ) The later can only be handled well by the compiler.
Thats is exactly what i am doing, i walk through the pages of the thread stack mapped memory and keep track of the highest access page. So i have the high water mark of the used stack.
The patches are not intrusive, especially part 1.
Quoted 1 line
Stefani
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Stefani Seibold
Am Dienstag, den 31.03.2009, 17:49 +0200 schrieb Andi Kleen:
Quoted 6 lines
Quoted 1 line · Stefani Seibold
- Misuse the thread stack for big temporary data buffers
That would be better checked for at compile time
(except for alloca, but that is quite rare)Fine but it did not work for functions like:
Quoted 7 lines
void foo(int n)
{ char buf[n*1024];}
This is valid with gcc.
Good call. checkstack should look for those as well. It is certainly possible to detect statically and warn about:
10: 29 c4 sub %eax,%esp
Runaway recursions are a different matter, though. The code I once had to detect them depends on an old version of smatch, which in turn depends on gcc 3.1. And even assuming this was in a reasonable shape, I still don't know what to do about it. The kernel has thousands of recursions and trying to work out how deep each one may stack is a never-ending project.
JA?rn
Signature · 9 lines
A quarrel is quickly settled when deserted by one party; there is no battle unless there be two. -- Seneca -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 12 lines · Stefani Seibold
Hi Andi,
Am Dienstag, den 31.03.2009, 17:49 +0200 schrieb Andi Kleen:
Quoted 8 lines
Quoted 3 lines · Stefani Seibold
- Get out of virtual memory by creating a lot of threads (f.e. the developer did assign each of them the default size)
The application just fails then? I don't think that needs a new monitoring tool.
Quoted 5 lines
First, this patch is not only a monitoring tool. Only the last part 3/3 is the monitoring tool.
Patch 1/3 enhance the the proc/<pid>/task/<tid>/maps by the marking the thread stack.
Well some implementation of it. There are certainly runtimes that switch stacks. For example what happens when someone uses sigaltstack()?
You'll probably need some more sanity checks, otherwise a monitor will occasionally just report crap in legitimate cases. Unfortunately it's difficult to distingush the legitimate cases from the bugs.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
- Misuse the thread stack for big temporary data buffers
That would be better checked for at compile time
(except for alloca, but that is quite rare)Fine but it did not work for functions like:
That's the alloca() case, but you can disable both with the right options. There's still the "recursive function" case.
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
- Thread stack overruns
Your method would be racy at best to determine this because you don't keep track of the worst case, only the current case.
Quoted 7 lines
Quoted 4 lines
So e.g. if you monitoring app checks once per second the stack could overflow between your monitoring intervals, but already have bounced back before the checker comes in.
The Monitor is part 3/3. And you are right it is not a complete rock solid solution. But it works in many cases and thats is what counts.
For stack overflow one would think a rock solid solution is needed? After all you'll crash if you miss a case.
Quoted 7 lines
Quoted 3 lines
track of consumption in the VMA that has the stack, but that can't handle very large jumps (like
f(){ char x[1<<30]; } ) The later can only be handled well by the compiler.Thats is exactly what i am doing, i walk through the pages of the thread stack mapped memory and keep track of the highest access page. So i have the high water mark of the used stack.
Ok. Of course it doesn't work for really large allocations. there used to be special patches around to add a special gap to catch those, but it's problematic with the tight address space on 32bit and also again not fully bullet proof. On 64bit the solution is typical to just use a large stack.
Quoted 1 line
The patches are not intrusive, especially part 1.
To be honest it seems too much like a special case hack to me to include by default. It could be probably done with a systemtap script in the same way, but I would really recommend to just build with gcc's stack overflow checker while testing together with static checking.
-Andi
Signature · 7 lines
[email protected] -- Speaking for myself only. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Andi,
stop complaining about the monitor. This is only an additional functionality.
The main purpose are part 1 and 2.
Quoted 4 lines
Well some implementation of it. There are certainly runtimes that switch stacks. For example what happens when someone uses
sigaltstack()?
What should happen with sigaltstack? This is complete independent from the process and thread stack. So it works.
Quoted 3 lines
That's the
alloca()case, but you can disable both with the right options. There's still the "recursive function" case.
And no idea ;-)
Quoted 7 lines
Quoted 3 lines
The Monitor is part 3/3. And you are right it is not a complete rock solid solution. But it works in many cases and thats is what counts.
For stack overflow one would think a rock solid solution is needed? After all you'll crash if you miss a case.
Again, the monitor is the only a part of the patch and i know that this is a issue.
The first two patches will also work without the monitor and if you don't like the monitor, no problem. It is a CONFIG_... parameter.
Quoted 6 lines
To be honest it seems too much like a special case hack to me to include by default. It could be probably done with a systemtap script in the same way, but I would really recommend to just build with gcc's stack overflow checker while testing together with static checking.
Thanks for the hack - I am not sure if you really had a look at my first posting nor had a look into my code.
We discus about complete different things. You have from user land no possibility to figure out where is the thread stack locate nor what was the highest used thread stack address.
That is a simple debug information which can provide very easily which the first two patches.
Stefani
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>