Skip to contentSkip to conversation

4KSTACKS + DEBUG_STACKOVERFLOW harmful

DISCUSSION Eric Sandeen 6 participants 9 messages Aug 29, 2007 – May 28, 2008

lore
Share & tools
lore
#0 lore raw

Noticed today that the combination of 4KSTACKS and DEBUG_STACKOVERFLOW config options is a bit deadly.

DEBUG_STACKOVERFLOW warns in do_IRQ if we're within THREAD_SIZE/8 of the end of useable stack space, or 512 bytes on a 4k stack.

If we are, then it goes down the dump_stack path, which uses most, if not all, of the remaining stack, thereby turning a well-intentioned warning into a full-blown catastrophe.

The callchain from the warning looks something like this, with stack usage shown as found on my x86 box:

4 dump_stack
4 show_trace
8 show_trace_log_lvl
4 dump_trace
print_context_stack
12 print_trace_address
print_symbol
232 __print_symbol
164 sprint_symbol
20 printk
___
448

448 bytes to tell us that we're within 512 bytes (or less) of certain doom... and I think there's call overhead on top of that?

The large stack usage in those 2 functions is due to big char arrays, of size KSYM_NAME_LEN (128 bytes) and KSYM_SYMBOL_LEN (223 bytes).

IOW, the stack warning effectively reduces useful stack left in our itty bitty 4k stacks by over 10%.

Any suggestions for ways around this? The warning is somewhat helpful, and I guess the obvious option is to lighten up the dump_stack path, but it's still effectively reducing precious available stack space by some amount.

With CONFIG_DEBUG_STACK_USAGE, we could print at oops time: "oh, and by the way, you blew your stack" if there is no zeroed stack space left, as a post-mortem. Even without that option, I think we could still check whether the *current* %esp at oops time has gone too far? But if we blew the stack, returned, and *then* oops, I think it'd be hard to know without the DEBUG_STACK_USAGE option that we ran out of room.

-Eric

#1 lore raw
Quoted 10 lines · Eric Sandeen

Noticed today that the combination of 4KSTACKS and DEBUG_STACKOVERFLOW config options is a bit deadly.

DEBUG_STACKOVERFLOW warns in do_IRQ if we're within THREAD_SIZE/8 of the end of useable stack space, or 512 bytes on a 4k stack.

If we are, then it goes down the dump_stack path, which uses most, if not all, of the remaining stack, thereby turning a well-intentioned warning into a full-blown catastrophe.

...

Quoted 7 lines

448 bytes to tell us that we're within 512 bytes (or less) of certain doom... and I think there's call overhead on top of that?

The large stack usage in those 2 functions is due to big char arrays, of size KSYM_NAME_LEN (128 bytes) and KSYM_SYMBOL_LEN (223 bytes).

Quoted 8 lines

IOW, the stack warning effectively reduces useful stack left in our itty bitty 4k stacks by over 10%.

Any suggestions for ways around this? The warning is somewhat helpful, and I guess the obvious option is to lighten up the dump_stack path, but it's still effectively reducing precious available stack space by some amount.

A first step could be to allocate those two char arrays with kmalloc() instead of on the stack, but then I guess that dump_stack() gets called from places where we may not really want to be calling kmalloc(). I guess we could allocate the buffers earlier (like at boot time) and store pointers somewhere where dump stack can get to them later when it needs them.

Quoted 7 lines

With CONFIG_DEBUG_STACK_USAGE, we could print at oops time: "oh, and by the way, you blew your stack" if there is no zeroed stack space left, as a post-mortem. Even without that option, I think we could still check whether the *current* %esp at oops time has gone too far? But if we blew the stack, returned, and *then* oops, I think it'd be hard to know without the DEBUG_STACK_USAGE option that we ran out of room.

We could also simply have it warn at a higher limit, like 1024 bytes instead of 512. But I guess then we would get too many false positives and make it less useful.

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#2 lore raw
Quoted 11 lines · Jesper Juhl
Quoted 5 lines

Any suggestions for ways around this? The warning is somewhat helpful, and I guess the obvious option is to lighten up the dump_stack path, but it's still effectively reducing precious available stack space by some amount.

A first step could be to allocate those two char arrays with kmalloc() instead of on the stack, but then I guess that dump_stack() gets called from places where we may not really want to be calling kmalloc(). I guess we could allocate the buffers earlier (like at boot time) and store pointers somewhere where dump stack can get to them later when it needs them.

Yep, I thought about something like that... and I assume you'd need a bit of locking around them too.

Quoted 11 lines
Quoted 7 lines

With CONFIG_DEBUG_STACK_USAGE, we could print at oops time: "oh, and by the way, you blew your stack" if there is no zeroed stack space left, as a post-mortem. Even without that option, I think we could still check whether the *current* %esp at oops time has gone too far? But if we blew the stack, returned, and *then* oops, I think it'd be hard to know without the DEBUG_STACK_USAGE option that we ran out of room.

We could also simply have it warn at a higher limit, like 1024 bytes instead of 512. But I guess then we would get too many false positives and make it less useful.

Yes, but if you happen to warn deeper anyway, just because you got "lucky" with IRQ timing, you'll still explode. Regardless of where the threshold is, there's still a risk of starting the warning deeper than that. Whatever stack the warning takes effectively reduces the useable stack size.

-Eric

#3 lore raw
Quoted 4 lines · Eric Sandeen
Quoted 1 line · Jesper Juhl

A first step could be to allocate those two char arrays with kmalloc() instead of on the stack, but then I guess that dump_stack() gets called from places where we may not really want to be calling kmalloc(). I guess we could allocate the buffers earlier (like at boot time) and store pointers somewhere where dump stack can get to them later when it needs them.

Yep, I thought about something like that... and I assume you'd need a bit of locking around them too.

How about turning off preemption and using a per-CPU buffer? Alternatively you could turn off IRQs, poke a per-CPU value to clue in any incoming NMIs, and switch to a separate stack. I suppose if you wanted it to work with all of 16 bytes of stack left on both thread and IRQ stacks, you could have separate per-CPU NMI stacks; the stack-dump would be poking a special per-CPU value and sending ourselves an NMI.

There are probably a half dozen other variants on ways to run screaming to the CPU saying "It hurts mommy!" and get a new stack in which we can play for a while.

Cheers,
Kyle Moffett

#4 lore raw
In reply to Eric Sandeen
Quoted 5 lines · Eric Sandeen

Noticed today that the combination of 4KSTACKS and DEBUG_STACKOVERFLOW config options is a bit deadly.

DEBUG_STACKOVERFLOW warns in do_IRQ if we're within THREAD_SIZE/8 of the end of useable stack space, or 512 bytes on a 4k stack.

...

Quoted 5 lines

The large stack usage in those 2 functions is due to big char arrays, of size KSYM_NAME_LEN (128 bytes) and KSYM_SYMBOL_LEN (223 bytes).

IOW, the stack warning effectively reduces useful stack left in our itty bitty 4k stacks by over 10%.

KSYM_NAME_LEN = 128 sounds stupid. The name which is wider than 80 chars?? Kernel shouldn't have names that long.
Say, 50 chars ought to be enough.
--
vda

#5 lore raw
Quoted 4 lines · Denys Vlasenko

KSYM_NAME_LEN = 128 sounds stupid. The name which is wider than 80 chars?? Kernel shouldn't have names that long.
Say, 50 chars ought to be enough.

Might be an enforcement problem, unless someone also writes a check_name_len.pl or so.

Jörn

Signature · 3 lines
The rabbit runs faster than the fox, because the rabbit is rinning for
his life while the fox is only running for his dinner.
-- Aesop
#6 lore raw
Quoted 8 lines · Jörn Engel
Quoted 4 lines · Denys Vlasenko

KSYM_NAME_LEN = 128 sounds stupid. The name which is wider than 80 chars?? Kernel shouldn't have names that long.
Say, 50 chars ought to be enough.

Might be an enforcement problem, unless someone also writes a check_name_len.pl or so.

It's trivial to do it in scripts/mksysmap (script which generates System.map)

Currently it simply does

$NM -n $1 | grep -v '\( [aUw] \)\|\(__crc_\)\|\( \$[adt]\)' > $2

--
vda

#7 lore raw
In reply to Eric Sandeen
Quoted 26 lines · Eric Sandeen

Noticed today that the combination of 4KSTACKS and DEBUG_STACKOVERFLOW config options is a bit deadly.

DEBUG_STACKOVERFLOW warns in do_IRQ if we're within THREAD_SIZE/8 of the end of useable stack space, or 512 bytes on a 4k stack.

If we are, then it goes down the dump_stack path, which uses most, if not all, of the remaining stack, thereby turning a well-intentioned warning into a full-blown catastrophe.

The callchain from the warning looks something like this, with stack usage shown as found on my x86 box:

4 dump_stack
4 show_trace
8 show_trace_log_lvl
4 dump_trace
print_context_stack
12 print_trace_address
print_symbol
232 __print_symbol
164 sprint_symbol
20 printk
___
448

Quoted 8 lines

448 bytes to tell us that we're within 512 bytes (or less) of certain doom... and I think there's call overhead on top of that?

The large stack usage in those 2 functions is due to big char arrays, of size KSYM_NAME_LEN (128 bytes) and KSYM_SYMBOL_LEN (223 bytes).

IOW, the stack warning effectively reduces useful stack left in our itty bitty 4k stacks by over 10%.

Hi Eric,

Did you happen to get a patch together that reduces the stack usage of dump_stack?

Also, what did you use to print your (above) indented callchain stack usage of dump_stack?

I'd like to be able to audit the worst case stack usage of _all_ call chains that originate from a given thread. This would effectively be like DEBUG_STACK_USAGE except with finer grained (per call-chain) statistics. One crude way of doing this is to dump_stack() whenever a task's call-chain is the new "winner" as the biggest stack hog.

To do this safely it would seem to me that a leaner dump_stack() is needed...

Lastly, would it be reasonable to utilize systemtap to implement what I described above? I'm actually looking to debug 4KSTACKS as unobtrusively as possible so as to not alter the underlying kernel (in this case it happens to be a RHEL5 kernel but this could apply to any kernel).

please advise, thanks.

#8 lore raw
Quoted 4 lines · Mike Snitzer

Hi Eric,

Did you happen to get a patch together that reduces the stack usage of dump_stack?

Nope... but Andi Kleen sent a patch to put the warning on the irq stack rather than the main process stack, so it avoids the original problem with DEBUG_STACKOVERFLOW at least.

Quoted 2 lines

Also, what did you use to print your (above) indented callchain stack usage of dump_stack?

That was just hand-edited... :)

Quoted 5 lines

I'd like to be able to audit the worst case stack usage of _all_ call chains that originate from a given thread. This would effectively be like DEBUG_STACK_USAGE except with finer grained (per call-chain) statistics. One crude way of doing this is to dump_stack() whenever a task's call-chain is the new "winner" as the biggest stack hog.

When will you test for the new winner?

Quoted 1 line

To do this safely it would seem to me that a leaner dump_stack() is needed...

It depends, I guess; if you have 8k stacks it'd probably fit ok in almost all cases, I think.

Quoted 5 lines

Lastly, would it be reasonable to utilize systemtap to implement what I described above? I'm actually looking to debug 4KSTACKS as unobtrusively as possible so as to not alter the underlying kernel (in this case it happens to be a RHEL5 kernel but this could apply to any kernel).

I don't actually know if systemtap can do what you want (not saying it can't; just saying I don't know...)

-Eric