4KSTACKS + DEBUG_STACKOVERFLOW harmful
Noticed today that the combination of 4KSTACKS and DEBUG_STACKOVERFLOW config options is a bit deadly.
DEBUG_STACKOVERFLOW warns in do_IRQ if we're within THREAD_SIZE/8 of the end of useable stack space, or 512 bytes on a 4k stack.
If we are, then it goes down the dump_stack path, which uses most, if not all, of the remaining stack, thereby turning a well-intentioned warning into a full-blown catastrophe.
The callchain from the warning looks something like this, with stack usage shown as found on my x86 box:
4 dump_stack
4 show_trace
8 show_trace_log_lvl
4 dump_trace
print_context_stack
12 print_trace_address
print_symbol
232 __print_symbol
164 sprint_symbol
20 printk
___
448
448 bytes to tell us that we're within 512 bytes (or less) of certain doom... and I think there's call overhead on top of that?
The large stack usage in those 2 functions is due to big char arrays, of size KSYM_NAME_LEN (128 bytes) and KSYM_SYMBOL_LEN (223 bytes).
IOW, the stack warning effectively reduces useful stack left in our itty bitty 4k stacks by over 10%.
Any suggestions for ways around this? The warning is somewhat helpful, and I guess the obvious option is to lighten up the dump_stack path, but it's still effectively reducing precious available stack space by some amount.
With CONFIG_DEBUG_STACK_USAGE, we could print at oops time: "oh, and by the way, you blew your stack" if there is no zeroed stack space left, as a post-mortem. Even without that option, I think we could still check whether the *current* %esp at oops time has gone too far? But if we blew the stack, returned, and *then* oops, I think it'd be hard to know without the DEBUG_STACK_USAGE option that we ran out of room.
-Eric
Quoted 10 lines · Eric Sandeen
Noticed today that the combination of 4KSTACKS and
DEBUG_STACKOVERFLOWconfig options is a bit deadly.
DEBUG_STACKOVERFLOWwarns indo_IRQif we're within THREAD_SIZE/8 of the end of useable stack space, or 512 bytes on a 4k stack.If we are, then it goes down the
dump_stackpath, which uses most, if not all, of the remaining stack, thereby turning a well-intentioned warning into a full-blown catastrophe.
...
Quoted 7 lines
448 bytes to tell us that we're within 512 bytes (or less) of certain doom... and I think there's call overhead on top of that?
The large stack usage in those 2 functions is due to big char arrays, of size
KSYM_NAME_LEN(128 bytes) andKSYM_SYMBOL_LEN(223 bytes).
Quoted 8 lines
IOW, the stack warning effectively reduces useful stack left in our itty bitty 4k stacks by over 10%.
Any suggestions for ways around this? The warning is somewhat helpful, and I guess the obvious option is to lighten up the
dump_stackpath, but it's still effectively reducing precious available stack space by some amount.
A first step could be to allocate those two char arrays with kmalloc() instead of on the stack, but then I guess that dump_stack() gets called from places where we may not really want to be calling kmalloc(). I guess we could allocate the buffers earlier (like at boot time) and store pointers somewhere where dump stack can get to them later when it needs them.
Quoted 7 lines
With
CONFIG_DEBUG_STACK_USAGE, we could print at oops time: "oh, and by the way, you blew your stack" if there is no zeroed stack space left, as a post-mortem. Even without that option, I think we could still check whether the *current* %esp at oops time has gone too far? But if we blew the stack, returned, and *then* oops, I think it'd be hard to know without theDEBUG_STACK_USAGEoption that we ran out of room.
We could also simply have it warn at a higher limit, like 1024 bytes instead of 512. But I guess then we would get too many false positives and make it less useful.
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Quoted 11 lines · Jesper Juhl
Quoted 5 lines
Any suggestions for ways around this? The warning is somewhat helpful, and I guess the obvious option is to lighten up the
dump_stackpath, but it's still effectively reducing precious available stack space by some amount.A first step could be to allocate those two char arrays with
kmalloc()instead of on the stack, but then I guess thatdump_stack()gets called from places where we may not really want to be callingkmalloc(). I guess we could allocate the buffers earlier (like at boot time) and store pointers somewhere where dump stack can get to them later when it needs them.
Yep, I thought about something like that... and I assume you'd need a bit of locking around them too.
Quoted 11 lines
Quoted 7 lines
With
CONFIG_DEBUG_STACK_USAGE, we could print at oops time: "oh, and by the way, you blew your stack" if there is no zeroed stack space left, as a post-mortem. Even without that option, I think we could still check whether the *current* %esp at oops time has gone too far? But if we blew the stack, returned, and *then* oops, I think it'd be hard to know without theDEBUG_STACK_USAGEoption that we ran out of room.We could also simply have it warn at a higher limit, like 1024 bytes instead of 512. But I guess then we would get too many false positives and make it less useful.
Yes, but if you happen to warn deeper anyway, just because you got "lucky" with IRQ timing, you'll still explode. Regardless of where the threshold is, there's still a risk of starting the warning deeper than that. Whatever stack the warning takes effectively reduces the useable stack size.
-Eric
Quoted 4 lines · Eric Sandeen
Quoted 1 line · Jesper Juhl
A first step could be to allocate those two char arrays with
kmalloc()instead of on the stack, but then I guess thatdump_stack()gets called from places where we may not really want to be callingkmalloc(). I guess we could allocate the buffers earlier (like at boot time) and store pointers somewhere where dump stack can get to them later when it needs them.Yep, I thought about something like that... and I assume you'd need a bit of locking around them too.
How about turning off preemption and using a per-CPU buffer? Alternatively you could turn off IRQs, poke a per-CPU value to clue in any incoming NMIs, and switch to a separate stack. I suppose if you wanted it to work with all of 16 bytes of stack left on both thread and IRQ stacks, you could have separate per-CPU NMI stacks; the stack-dump would be poking a special per-CPU value and sending ourselves an NMI.
There are probably a half dozen other variants on ways to run screaming to the CPU saying "It hurts mommy!" and get a new stack in which we can play for a while.
Cheers,
Kyle Moffett
Quoted 5 lines · Eric Sandeen
Noticed today that the combination of 4KSTACKS and
DEBUG_STACKOVERFLOWconfig options is a bit deadly.
DEBUG_STACKOVERFLOWwarns indo_IRQif we're within THREAD_SIZE/8 of the end of useable stack space, or 512 bytes on a 4k stack.
...
Quoted 5 lines
The large stack usage in those 2 functions is due to big char arrays, of size
KSYM_NAME_LEN(128 bytes) andKSYM_SYMBOL_LEN(223 bytes).IOW, the stack warning effectively reduces useful stack left in our itty bitty 4k stacks by over 10%.
KSYM_NAME_LEN = 128 sounds stupid. The name which is wider than 80 chars?? Kernel shouldn't have names that long.
Say, 50 chars ought to be enough.
--
vda
Quoted 4 lines · Denys Vlasenko
KSYM_NAME_LEN= 128 sounds stupid. The name which is wider than 80 chars?? Kernel shouldn't have names that long.
Say, 50 chars ought to be enough.
Might be an enforcement problem, unless someone also writes a check_name_len.pl or so.
Jörn
Signature · 3 lines
The rabbit runs faster than the fox, because the rabbit is rinning for his life while the fox is only running for his dinner. -- Aesop
Quoted 8 lines · Jörn Engel
Quoted 4 lines · Denys Vlasenko
KSYM_NAME_LEN= 128 sounds stupid. The name which is wider than 80 chars?? Kernel shouldn't have names that long.
Say, 50 chars ought to be enough.Might be an enforcement problem, unless someone also writes a check_name_len.pl or so.
It's trivial to do it in scripts/mksysmap (script which generates System.map)
Currently it simply does
$NM -n $1 | grep -v '\( [aUw] \)\|\(__crc_\)\|\( \$[adt]\)' > $2
--
vda
Quoted 26 lines · Eric Sandeen
Noticed today that the combination of 4KSTACKS and
DEBUG_STACKOVERFLOWconfig options is a bit deadly.
DEBUG_STACKOVERFLOWwarns indo_IRQif we're within THREAD_SIZE/8 of the end of useable stack space, or 512 bytes on a 4k stack.If we are, then it goes down the
dump_stackpath, which uses most, if not all, of the remaining stack, thereby turning a well-intentioned warning into a full-blown catastrophe.The callchain from the warning looks something like this, with stack usage shown as found on my x86 box:
4
dump_stack
4show_trace
8show_trace_log_lvl
4dump_trace
print_context_stack
12print_trace_address
print_symbol
232__print_symbol
164sprint_symbol
20 printk
___
448
Quoted 8 lines
448 bytes to tell us that we're within 512 bytes (or less) of certain doom... and I think there's call overhead on top of that?
The large stack usage in those 2 functions is due to big char arrays, of size
KSYM_NAME_LEN(128 bytes) andKSYM_SYMBOL_LEN(223 bytes).IOW, the stack warning effectively reduces useful stack left in our itty bitty 4k stacks by over 10%.
Hi Eric,
Did you happen to get a patch together that reduces the stack usage of dump_stack?
Also, what did you use to print your (above) indented callchain stack usage of dump_stack?
I'd like to be able to audit the worst case stack usage of _all_ call chains that originate from a given thread. This would effectively be like DEBUG_STACK_USAGE except with finer grained (per call-chain) statistics. One crude way of doing this is to dump_stack() whenever a task's call-chain is the new "winner" as the biggest stack hog.
To do this safely it would seem to me that a leaner dump_stack() is needed...
Lastly, would it be reasonable to utilize systemtap to implement what I described above? I'm actually looking to debug 4KSTACKS as unobtrusively as possible so as to not alter the underlying kernel (in this case it happens to be a RHEL5 kernel but this could apply to any kernel).
please advise, thanks.
Quoted 4 lines · Mike Snitzer
Hi Eric,
Did you happen to get a patch together that reduces the stack usage of
dump_stack?
Nope... but Andi Kleen sent a patch to put the warning on the irq stack rather than the main process stack, so it avoids the original problem with DEBUG_STACKOVERFLOW at least.
Quoted 2 lines
Also, what did you use to print your (above) indented callchain stack usage of
dump_stack?
That was just hand-edited... :)
Quoted 5 lines
I'd like to be able to audit the worst case stack usage of _all_ call chains that originate from a given thread. This would effectively be like
DEBUG_STACK_USAGEexcept with finer grained (per call-chain) statistics. One crude way of doing this is todump_stack()whenever a task's call-chain is the new "winner" as the biggest stack hog.
When will you test for the new winner?
Quoted 1 line
To do this safely it would seem to me that a leaner
dump_stack()is needed...
It depends, I guess; if you have 8k stacks it'd probably fit ok in almost all cases, I think.
Quoted 5 lines
Lastly, would it be reasonable to utilize systemtap to implement what I described above? I'm actually looking to debug 4KSTACKS as unobtrusively as possible so as to not alter the underlying kernel (in this case it happens to be a RHEL5 kernel but this could apply to any kernel).
I don't actually know if systemtap can do what you want (not saying it can't; just saying I don't know...)
-Eric