linux-2.6 x86_64 kgdb issue
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 44 lines · Amit S. Kale
Quoted 9 lines · Vladimir A. Barinov
Quoted 8 lines · Amit S. Kale
Looking at this again:
Call Trace: {kgdb_notify} {notifier_call_chain} {do_stack_segment} {stack_segment} {io_outb} {kgdb_mem2hex}Quoted 8 lines
Quoted 5 lines
Why is
io_outbbeing called fromkgdb_mem2hex.kgdb_mem2hexrefers to data directly and not throughio_outb.Perhaps it's got something to do with iommu feature. Have you used "iommu" on kernel command line?
I have iommu switched on, but disabling this lead to the same
dump_stackresult.This is confusing. Could you shed some light on this.
Quoted 11 lines
But I've used earlier version of
kgdb_8250.candio_outb()was a callback similar tokgdb_oiwrite()in kgdb-2.6.16.tar.bz2. I merged thekgdb_8250.cfrom the kgdb-2.6.16.tar.bz2 and got thedump_stack:{notifier_call_chain} {do_stack_segment} {stack_segment} {kgdb_mem2hex} {kgdb_mem2hex}Quoted 3 lines
Also I've used 2.6.10 with this stack exception behavior. In 2.6.16 patched with kgdb-2.6.16.tar.bz2 the behavior is so that the target reboots after multiple steps and a "continue" command in the end.
Now it looks like we have a stack overflow. It would result in a stack exception. Stack overflow usually results in a complete breakdown of a kernel since there is no stack to handle the stack exception itself. Processors upto pentium used float their buses, which would be detected by the surrounding hardware and cause a reset. I am not sure whether modern processors and/or hardware does that.
Quoted 1 line
Quoted 5 lines
Quoted 2 lines
Just want to note that in 2.6.10 kernel the stack exception doesn't occur if
CONFIG_64BITinlinux/kernel/kgdb.cis not defined.
CONFIG_64BITprobably requires more stack. That's why you see a stack exception.
I added some debug info to the thread and got stack overflows; it was trivial to double the size of the stack. I was saving a back trace of the stack during at each preemption point (Ex: spinlock) to allow me to see the context of the active holders of spinlocks. I configured it with CONFIG_DEBUG_PREEMPT_AUDIT and enabled large stacks in:
------------------------------------------------------------------------
include/asm-i386/thread_info.h:
------------------------------------------------------------------------
#ifdef CONFIG_DEBUG_PREEMPT_AUDIT
#define THREAD_SIZE (8192 * 2)
#else
#ifdef CONFIG_4KSTACKS
#define THREAD_SIZE (4096)
#else
#define THREAD_SIZE (8192)
#endif
#endif
-------------------------------------------------------------------------
All you really need to do is change THREAD_SIZE from
(8192) to (8192 * 2). I didn't have any problems in i386.
Quoted 5 lines
Unfortunately
x86_64architecture doesn't provide any stack overflow debugging mechanism. Perhaps you can implement a little code inkgdb_handle_exception, which checks whether we are beyond 7168 bytes of stack usage on entry. If we are, declare a panic indicating a possible stack overflow later.
I was getting stack overflows on the SPARC architecture when compiling the kernel -O1 for kgdb/kdbx debugging. I allocated a hot physical page for each CPU as it was brought on line and then mapped it on the fly when we got a stack overflow. I then pushed out the register window that caused the trap, and then continued with the normal panic path.
Perhaps we should add a kgdb config menu option and #define CONFIG_16KSTACKS to double the stack size so the kernel can be debugged with more context available. I'm currently using -O0 for the networking stack and -O1 for the rest of the kernel. Sounds like it would be helpful now for AMD64 targets.
-piet
Quoted 53 lines
-Amit
Quoted 39 lines
Vladimir
Quoted 36 lines
-Amit
Quoted 33 lines · Vladimir A. Barinov
Hi All,
I'm working with em64t dual xeon board and have a problems with kgdb when SMP is on.
During step by step debugging I've got the error message and gdb server lost connection
to the target (gdb log is attached)Putting simple
printk()anddump_stack()into thekgdb_notify():..... if (cmd == DIE_TRAP) { printk("DIE_TRAP, args->str=%s, kgdb_may_fault=%d\n",args->str,kgdb_may_fault); dump_stack();}
......I've got trace:
DIE_TRAP, args->str=stack segment, kgdb_may_fault=1Call Trace: {kgdb_notify} {notifier_call_chain} {do_stack_segment} {stack_segment} {io_outb} {kgdb_mem2hex}The stack exception occurs always at the same step during debugging in
kgdb_mem2hex().
I've attached patch that fixes this issue. Could you please review, is this patch appropriate
to the problem?Vladimir
-------------------------------------------------------
All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
_______________________________________________
Kgdb-bugreport mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
Signature · 2 lines
--- [email protected]
Quoted 6 lines
Perhaps we should add a kgdb config menu option and #define
CONFIG_16KSTACKSto double the stack size so the kernel can be debugged with more context available. I'm currently using -O0 for the networking stack and -O1 for the rest of the kernel. Sounds like it would be helpful now for AMD64 targets.
You only got stack overflows when working with kgdb right?
Sounds like a kgdb problem to me then that can and should be probably fixed. e.g. afaik kgdb isn't reentrant anyways so it could just use static buffers.
I would suggest against adding any new config options for this - it would conflict with the great goal of having loadable debuggers that work on any kernels.
-Andi
Quoted 7 lines · Andi Kleen
Quoted 5 lines
Perhaps we should add a kgdb config menu option and #define
CONFIG_16KSTACKSto double the stack size so the kernel can be debugged with more context available. I'm currently using -O0 for the networking stack and -O1 for the rest of the kernel. Sounds like it would be helpful now for AMD64 targets.You only got stack overflows when working with kgdb right?
Yes, I was using kgdb to look at the stack audits I stored in the thread structure.
Quoted 3 lines
Sounds like a kgdb problem to me then that can and should be probably fixed. e.g. afaik kgdb isn't reentrant anyways so it could just use static buffers.
On Solaris the problem was that the kernel stack was larger because tail optimization was disabled with optimization disabled. I'm not having a kgdb problem with i386, it seems that Vladimir is/was and Amit suspected it being due to the AMD64 requiring largers stacks. Seems plausible to me. I thought you might have thoughts on that.
Quoted 4 lines
I would suggest against adding any new config options for this - it would conflict with the great goal of having loadable debuggers that work on any kernels.
What's the conflict with larger kernel stacks and a loadable (kgdb) module? Like Andy Morton I prefer to avoid using modules when using kgdb; so I wouldn't have run into a problem.
I was suggesting larger stack space for the kernel, not kgdb. I agree this case might be one where kgdb has caused the kernel to trip over the edge. I don't feel comfortable running on a kernel that running that close to running out of stack space. Maybe that's just a personal preference; I'm paranoid I guess. I like having rock solid systems and wacking the stack isn't always detected. On SunOS we had a REDZONE but last I check Linux didn't; has one been added? If it hasn't it might be good to keep in mind having a CPU specific physical page available when we grow into the REDZONE. Looked to me like the stack grows right into the thread structure; might make a nice exploit for a linux root kit.
Having loadable debuggers seems a bit high hopes, as 'we' haven't even release linux with kgdb built into the Linux src yet.
-piet
Quoted 2 lines
-Andi
Signature · 2 lines
--- [email protected]
Quoted 11 lines · Piet Delaney
Quoted 7 lines · Andi Kleen
Quoted 5 lines
Perhaps we should add a kgdb config menu option and #define
CONFIG_16KSTACKSto double the stack size so the kernel can be debugged with more context available. I'm currently using -O0 for the networking stack and -O1 for the rest of the kernel. Sounds like it would be helpful now for AMD64 targets.You only got stack overflows when working with kgdb right?
Yes, I was using kgdb to look at the stack audits I stored in the thread structure.
Again this likely points to kgdb using too much stack.
Quoted 8 lines
Quoted 2 lines
Sounds like a kgdb problem to me then that can and should be probably fixed. e.g. afaik kgdb isn't reentrant anyways so it could just use static buffers.
On Solaris the problem was that the kernel stack was larger because tail optimization was disabled with optimization disabled. I'm not having a kgdb problem with i386, it seems that Vladimir is/was and Amit suspected it being due to the AMD64 requiring largers stacks. Seems plausible to me. I thought you might have thoughts on that.
Stack usage in Linux isn't that tight that it should make a difference. If something needs too much stack we just fix it.
Quoted 7 lines
Quoted 4 lines
I would suggest against adding any new config options for this - it would conflict with the great goal of having loadable debuggers that work on any kernels.
What's the conflict with larger kernel stacks and a loadable (kgdb) module?
We'll not increase the stacks by default but the debugger should work anyways.
Quoted 7 lines
I was suggesting larger stack space for the kernel, not kgdb. I agree this case might be one where kgdb has caused the kernel to trip over the edge. I don't feel comfortable running on a kernel that running that close to running out of stack space. Maybe that's just a personal preference; I'm paranoid I guess. I like having rock solid systems and wacking the stack isn't always detected. On SunOS we had a REDZONE but last I check Linux didn't; has one been added?
Interrupts can check for too much stack space used.
Quoted 5 lines
If it hasn't it might
be good to keep in mind having a CPU specific physical page available when we grow into the REDZONE. Looked to me like the stack grows right into the thread structure; might make a nice exploit for a linux root kit.
If you have a stack overflow you usually have other problems than that.
Quoted 2 lines
Having loadable debuggers seems a bit high hopes, as 'we' haven't even release linux with kgdb built into the Linux src yet.
Yes because you if modular works you don't need to build it in.
Modular was working at some point on x86-64 for kdb and the original 2.6 version of kgdb was nearly there too.
-Andi
Quoted 8 lines · Andi Kleen
Quoted 7 lines · Piet Delaney
Quoted 6 lines · Andi Kleen
Quoted 5 lines
Perhaps we should add a kgdb config menu option and #define
CONFIG_16KSTACKSto double the stack size so the kernel can be debugged with more context available. I'm currently using -O0 for the networking stack and -O1 for the rest of the kernel. Sounds like it would be helpful now for AMD64 targets.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
You only got stack overflows when working with kgdb right?
Yes, I was using kgdb to look at the stack audits I stored in the thread structure.
Again this likely points to kgdb using too much stack.
My bet is that in this case I was storing a LOT of data in the thread structure, so the space left for the stack was massively reduced.
I was using a debugging tool for taking snapshots of the top 16 PC's on the stack when you take a spinlock; for each premption level. It's thread specific data, so the thread structure seemed like a reasonable place to store it.
Quoted 19 lines
Quoted 10 lines
Quoted 4 lines
Sounds like a kgdb problem to me then that can and should be probably fixed. e.g. afaik kgdb isn't reentrant anyways so it could just use static buffers.
On Solaris the problem was that the kernel stack was larger because tail optimization was disabled with optimization disabled. I'm not having a kgdb problem with i386, it seems that Vladimir is/was and Amit suspected it being due to the AMD64 requiring largers stacks. Seems plausible to me. I thought you might have thoughts on that.
Stack usage in Linux isn't that tight that it should make a difference. If something needs too much stack we just fix it.
Quoted 5 lines
Quoted 4 lines
I would suggest against adding any new config options for this - it would conflict with the great goal of having loadable debuggers that work on any kernels.
Quoted 5 lines
Quoted 2 lines
What's the conflict with larger kernel stacks and a loadable (kgdb) module?
We'll not increase the stacks by default but the debugger should work anyways.
Sure but the debugger environment must tolerate larger stacks. A developer may prefer to use a larger stack, like in my case of storing debug info in the thread structure. The interrupt stack checks can easily miss nicking the thread structure, so increasing the stack size for experimentation SHOULD always work.
Quoted 10 lines
Quoted 8 lines
I was suggesting larger stack space for the kernel, not kgdb. I agree this case might be one where kgdb has caused the kernel to trip over the edge. I don't feel comfortable running on a kernel that running that close to running out of stack space. Maybe that's just a personal preference; I'm paranoid I guess. I like having rock solid systems and wacking the stack isn't always detected. On SunOS we had a REDZONE but last I check Linux didn't; has one been added?
Interrupts can check for too much stack space used.
But this can miss a minor abuse. The interrupt check is a quick and simple hack but I wonder if it's really optimal for commercial implementations.
Quoted 8 lines
Quoted 6 lines
If it hasn't it might
be good to keep in mind having a CPU specific physical page available when we grow into the REDZONE. Looked to me like the stack grows right into the thread structure; might make a nice exploit for a linux root kit.If you have a stack overflow you usually have other problems than that.
Yep, like viewing the stack with kgdb. With Solaris all of the task use the same virtual address space for the stack, so mapping in a physical page was easy. Linux stacks are mapped linear 1:1 to physical pages, so it's not easy.
Quoted 5 lines
Quoted 3 lines
Having loadable debuggers seems a bit high hopes, as 'we' haven't even release linux with kgdb built into the Linux src yet.
Yes because you if modular works you don't need to build it in.
I think all modules should be ABLE to be built in. 2.6 has made good progress in that regard. I'd prefer to see kgdb also ABLE to be linked in or a module; just like most modules are doing now. I don't see an advantage to the Linux community to REQUIRING the kgdb module to NOT being ABLE to be linked in. Using modules often requires more work to set up and get working right. It has it's advantages, I'd prefer to leave it up to the developer to choose. Sometimes you unload a module and pre-existing callbacks can mess up things in unexpected way when the size of data structures change. I've used kgdb on modules but for stuff like the TCP stack I really don't want to mess with that. I'd prefer we make it easier for users to get set up for kernel debugging. Just config kgdb, avoid using kernel modules, install and run. Having kgdb in the kernel source tree doesn't preclude it's use as a module. It does require it to be under the GNU license. Unfortunately the HPUX kgdb patch wasn't under the GNU license; it had some nice features, like kgdb over TCP with ssh encryption. An that was 10 years ago!
Quoted 3 lines
Modular was working at some point on x86-64 for kdb and the original 2.6 version of kgdb was nearly there too.
Yea, I saw that, too bad we can't seem to get a version of kgdb into the kernel that can be update with the rest of the kernel. As I recall NetBSD, FreeBSD, and OpenBSD have always done it that way. 'We' almost got George Anzinger's kgdb patch into Linux; it was working great in the mm series. Seems it should all be configurable with #ifdef CONFIG_KGDB just like other kernel features. If you want to support other debuggers than gdb I suppose common trap code could use a common CONFIG_* token.
-piet
Quoted 3 lines
-Andi
Signature · 2 lines
--- [email protected]
Quoted 4 lines
My bet is that in this case I was storing a LOT of data in the thread structure, so the space left for the stack was massively reduced.
Ok so it was your bug. Don't do that.
Quoted 1 line
Sure but the debugger environment must tolerate larger stacks.
No, Linux doesn't tolerate larger stacks.
Quoted 3 lines
But this can miss a minor abuse. The interrupt check is a quick and simple hack but I wonder if it's really optimal for commercial implementations.
In practice if you overwrite thread_info you crash eventually and it's noticed. If you write below thread_info but keep ti intact then the redzone would likely not catch it either.
I don't think an additional red zone would improve overflow detection in a significant way.
Quoted 1 line
I think all modules should be ABLE to be built in.
If you have a working module it can be easily built in too. Just hacks that don't work with modules are bad.
-Andi
On Wed, May 31, 2006 at 09:13:53AM +0200, Andi Kleen wrote:
[snip]
Quoted 4 lines
Yes because you if modular works you don't need to build it in.
Modular was working at some point on x86-64 for kdb and the original 2.6 version of kgdb was nearly there too.
FWIW, the only change the current version of kgdb makes that would prevent it from being totally modular is the debugger_active check in __might_sleep().
Signature · 1 line
Tom Rini
Quoted 4 lines · Tom Rini
On Wed, May 31, 2006 at 09:13:53AM +0200, Andi Kleen wrote:
[snip]
Quoted 8 lines
Quoted 4 lines
Yes because you if modular works you don't need to build it in.
Modular was working at some point on x86-64 for kdb and the original 2.6 version of kgdb was nearly there too.
FWIW, the only change the current version of kgdb makes that would prevent it from being totally modular is the
debugger_activecheck in
Can you post the patch and a description?
-Andi
Quoted 7 lines · Andi Kleen
Quoted 6 lines · Tom Rini
On Wed, May 31, 2006 at 09:13:53AM +0200, Andi Kleen wrote:
[snip]
Quoted 2 lines
Yes because you if modular works you don't need to build it in.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Modular was working at some point on x86-64 for kdb and the original 2.6 version of kgdb was nearly there too.
FWIW, the only change the current version of kgdb makes that would prevent it from being totally modular is the
debugger_activecheck inCan you post the patch and a description?
The change is a simple if (atomic_read(&debugger_active)) return right at the start. And I'm embarrased to say the change predates me on the project so I'm not 100% sure on the lineage and it might be totally bogus now.
Signature · 1 line
Tom Rini
Quoted 14 lines · Tom Rini
Quoted 13 lines · Andi Kleen
Quoted 11 lines · Tom Rini
On Wed, May 31, 2006 at 09:13:53AM +0200, Andi Kleen wrote:
[snip]
Quoted 4 lines
Yes because you if modular works you don't need to build it in.
Modular was working at some point on x86-64 for kdb and the original 2.6 version of kgdb was nearly there too.
FWIW, the only change the current version of kgdb makes that would prevent it from being totally modular is the
debugger_activecheck in
Quoted 6 lines
Quoted 1 line
Can you post the patch and a description?
The change is a simple if (
atomic_read(&debugger_active)) return right at the start. And I'm embarrased to say the change predates me on the project so I'm not 100% sure on the lineage and it might be totally bogus now.
And why do you need it? Where does the debugger call might sleep?
-Andi
Quoted 7 lines · Andi Kleen
Quoted 6 lines · Tom Rini
On Wed, May 31, 2006 at 09:13:53AM +0200, Andi Kleen wrote:
[snip]
Quoted 2 lines
Yes because you if modular works you don't need to build it in.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Modular was working at some point on x86-64 for kdb and the original 2.6 version of kgdb was nearly there too.
FWIW, the only change the current version of kgdb makes that would prevent it from being totally modular is the
debugger_activecheck inCan you post the patch and a description?
It's maintained at SourceForge:
http://sourceforge.net/projects/kgdb
Patches can be downloaded from:
http://sourceforge.net/project/showfiles.php?group_id=5073
I suspect that you will likely want to add yourself to the kgdb-bugreport mailing list.
https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
Mailing list is acting up; I've been having trouble posting due to a mailserver problem; I contacted the SourceForge folks about it.
Andrew Morton has recently been taking snapshots and including them in his mm series. It would be nice to get in "in order" and ready for being merged up into the linux tree. When I applied the 2.6.13 patch I think I noticed some '#ifdef KGDB' code missing. I'd like to see the patch totally disabled by not configuring KGDB. The patch instructions are also a bit misleading, at least for 2.6.13; I had to apply all of the patches in the series to avoid a lot a patch rejects.
-piet
Quoted 2 lines
-Andi
Signature · 2 lines
--- [email protected]
Quoted 3 lines · Piet Delaney
tried patching the latest stable kennel, 2.6.16.9 I thought, to 2.6.17.rc2 but got conflicts.
2.6.16.x is a branch, based on 2.6.16.
2.6.17-rcX is on the trunk. The published diffs are against 2.6.16.
So you should have patched 2.6.16.
Quoted 4 lines
A few years ago Linus said he wanted to integrated the kgdb patch but then stopped when he found our there were two patches. Now there only seems to be one. Do you know why it's not being integrated into the std linux kernel?
Nobody is doing the work to get the patches in order and merged up. It'd require a month or two moderately intense effort.
Quoted 2 lines · Piet Delaney
Mailing list is acting up; I've been having trouble posting due to a mailserver problem; I contacted the SourceForge folks about it.
Perhaps my problem was likely not having a SourceForge account; I added one last evening an it looks like my posting showed up on the mailing list.
http://sourceforge.net/account/newuser_emailverify.php
-piet