Old bug with 'gdb/dbxread.c' and screwed up MIPS symbolic debugging...

I have attempted to do as much research and testing as I possibly can before posting this. I remember reading a thread associated with this problem in the past, but things are still not working properly. I have also taken the liberty of CC'ing the binutils and linux-mips list just in case.

GOAL
----
To use my KGDB stub with GDB and/or Insight to debug my embedded MIPS kernel over the serial link utilizing symbolic debugging.

TOOLCHAIN
---------
binutils-2.11.92.0.10 (stock)
gcc-3.0.2 (stock)
glibc-2.2.3 (minor patches to ld-scripts and a small fixes for ipc/shm) gdb-10312001 (fresh out of cvs this morning w/patch applied, see bottom)

CONFIGURATION LINES FOR TOOLS
-----------------------------
../binutils-2.11.92.0.10/configure --prefix=/opt/toolchains/mips --target=mipsel-linux

AR=mipsel-linux-ar RANLIB=mipsel-linux-ranlib ../gcc-3.0.2/configure --prefix=/opt/toolchains/mips --target=mipsel-linux i686-pc-linux-gnu --with-newlib --disable-shared --enable-languages=c

BUILD_CC=gcc CC=mipsel-linux-gcc AR=mipsel-linux-ar AS=mipsel-linux-as RANLIB=mipsel-linux-ranlib ../glibc-2.2.3/configure --prefix=/opt/toolchains/mips/mipsel-linux mipsel-linux --build=i686-pc-linux-gnu --enable-add-ons --with-elf --disable-profile --with-headers=/opt/toolchains/mips/mipsel-linux/include --mandir=/opt/toolchains/mips/man --infodir=/opt/toolchains/mips/info

AR=mipsel-linux-ar RANLIB=mipsel-linux-ranlib ../gcc-3.0.2/configure --prefix=/opt/toolchains/mips --target=mipsel-linux i686-pc-linux-gnu --with-gxx-include-dir=/opt/toolchains/mips/mipsel-linux/include --mandir=/opt/toolchains/mips/man --infodir=/opt/toolchains/mips/info --enable-languages=c,c++ --enable-threads

../gdb-10312001/configure --prefix=/opt/toolchains/mips --target=mips-linux-elf

KERNEL
------
Last 2.4.5 release from oss.sgi.com CVS

TYPICAl KERNEL COMPILE LINE
---------------------------
mipsel-linux-gcc -I /opt/mips/settop/include/asm/gcc -D__KERNEL__ -I/opt/mips/settop/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing -g -G 0 -mno-abicalls -fno-pic -mcpu=r5000 -mips2
-Wa,--trap -pipe -I .. -I /opt/mips/settop/fs -funsigned-char -c -o
xfs_griostubs.o xfs_griostubs.c
Assembler messages:
Warning: The -mcpu option is deprecated. Please use -march and -mtune instead.
Warning: The -march option is incompatible to -mipsN and therefore ignored.

PROBLEM REPORT
--------------
I am using a NEC MIPS VR5432 in little endian and 32-bit mode. If I run 'mipsel-linux-objdump -d vmlinux', I get addresses starting with 0x8000XXXX. With older toolchains I remember getting 0xffffffff8000XXXX. My kernel boots fine and patiently waits for GDB to connect. If I use GDB stock from CVS without applying any patches, the following output occurs:

This GDB was configured as "--host=i686-pc-linux-gnu --target=mips-linux-elf"... (gdb) target remote /dev/ttyS1
Remote debugging using /dev/ttyS1
0x80012c88 in breakinst () at 1879
1879 sock_unregister(PF_PACKET);
(gdb) bt

#0  0x80012c88 in breakinst () at af_packet.c:1879
#1  0x8020aabc in brcm_irq_setup () at irq.c:421
#2  0x8020aaf0 in init_IRQ () at irq.c:434
#3  0x801fc83c in start_kernel () at init/main.c:524
#4  0x801fd6f8 in init_arch (argc=160, argv=0xb3000000, envp=0x2, 
    prom_vec=0xbf) at setup.c:425

(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap. 0x80012c88 in breakinst () at af_packet.c:1879
1879 sock_unregister(PF_PACKET);
(gdb) bt
#0 0x80012c88 in breakinst () at af_packet.c:1879
#1 0x8001a554 in sys_create_module (name_user=0x10001dc8 "brcmdrv",

    size=713264) at module.c:305

(gdb) c
Continuing.

Which is clearly wrong. 'breakinst' is clearly not in that file, but all the other symbolics in the backtrace are correct. Then if I go to insert a module, 'breakinst' again is decoded at the wrong place, but it gets 'sys_create_module' module symbol decoded right. I will point out that 'breakinst' is defined in 'arch/mips/kernel/gdb-stub.c' and FWIW, looks like:

        __asm__ __volatile__("
                        .globl  breakinst
                        .set    noreorder
                        nop
breakinst:              break
                        nop
                        .set    reorder
        ");

"SOLUTION"
----------
On August 15, H.J. Lu applied a patch to 'gdb/dbxread.c' shown here:

diff -urN -x CVS work/gdb/dbxread.c gdb-5.0-08162001/gdb/dbxread.c --- work/gdb/dbxread.c Tue Oct 30 16:33:33 2001
+++ gdb-5.0-08162001/gdb/dbxread.c Wed Aug 15 00:02:28 2001
@@ -951,7 +951,10 @@

        (intern).n_type = bfd_h_get_8 (abfd, (extern)->e_type);            \
        (intern).n_strx = bfd_h_get_32 (abfd, (extern)->e_strx);           \
        (intern).n_desc = bfd_h_get_16 (abfd, (extern)->e_desc);           \
   -    (intern).n_value = bfd_h_get_32 (abfd, (extern)->e_value);         \

+ if (bfd_get_sign_extend_vma
(abfd)) \
+ (intern).n_value = bfd_h_get_signed_32 (abfd,
(extern)->e_value); \
+ else \

   +      (intern).n_value = bfd_h_get_32 (abfd, (extern)->e_value);       \
      }
 
    /* Invariant: The symbol pointed to by symbuf_idx is the first one

If I back out this change, my debug output is "correct", but I no longer have the nice line numbers and files decoded for me:

(gdb) target remote /dev/ttyS1
Remote debugging using /dev/ttyS1
0x80012c88 in breakinst ()
(gdb) bt

#0  0x80012c88 in breakinst ()
#1  0x8020aabc in brcm_irq_setup ()
#2  0x8020aaf0 in init_IRQ ()
#3  0x801fc83c in start_kernel ()
#4  0x801fd6f8 in init_arch ()

(gdb) c
Continuing.

Also, if I attempt to back out this patch from the latest insight CVS code, it has not affect. Insight would still decode 'breakinst' at 'af_packet.c'.

CONCLUSION
----------
Conclusion? Uhh, things don't work. I greatly appreciate input from people on this issue and hope I have supplied enough detail. I don't want to start digging into the gdb source too deep without hearing other people's opinions. Thanks.

-Steve

 Steven J. Hill - Embedded SW Engineer

Well, the change in objdump output is purely cosmetic. For 32bit object formats we just truncate the display now.

Does this happen for any other symbol than breakinst? Breakinst is effectively a function with no debugging info. That case historically causes us problems, so we probably missed another need for sign extension.

If you back it out, I believe we just give up in confusion :) This is int the reading of stabs info. breakinst has no stabs info, so it's getting its line info somewhere else.

Please point me at a copy of your kernel binary with debug info, and I'll look into it.

Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

As an aside, is there an option to turn this truncation off? The vr5432 as and ld will still pass around 64 bit addresses.

It might - assembler debugging ...

Yes, you want to look for a version of breakinst that isn't sign extended. Since pulling the above patch helped it won't be in .stabs so the symbol table?

Andrew

It shouldn't happen in that case, I think. The vr5432 is configured as a mips64 target, isn't it?

I don't think it does, at least...

It's not that breakinst isn't sign extended. This is very much like the address ranges issue that came up with -ffunction-sections or linkonce sections.

I'm writing this as I debug. Excuse the flow of consciousness (or skip down to the end!).

Here's our bug:
(top-gdb) p/x *b
$21 = {startaddr = 0x34, endaddr = 0xffffffff80215314, function = 0x0,
superblock = 0x0, gcc_compile_flag = 0x2, nsyms = 0x6, sym = {0x8755bbc}}

The startaddr is obviously wrong. This is the first symtab listed for kernel. So where does that startaddr come from?

(By the way, our debugging of long longs is abysmal. I already filed a PR about this I think. It makes tracking 64bit CORE_ADDRs a real pain; they're printed with the upper half garbage.)

The answer is that the startaddr comes from the psymtab. During psymbol reading:

Hardware watchpoint 24: *$139

Old value = 18446744069414584372
New value = 52

at:

630 && (CUR_SYMBOL_VALUE
631 != ANOFFSET (objfile->section_offsets,
632 SECT_OFF_TEXT (objfile))))))
633 {
634 TEXTLOW (pst) = CUR_SYMBOL_VALUE;
635 textlow_not_set = 0;
636 }
637 #endif /* DBXREAD_ONLY */
638 add_psymbol_to_list (namestring, p - namestring,
639 VAR_NAMESPACE, LOC_BLOCK,

Here's the offending stabs entry:
317176 FUN 0 1870 0000000000000034 1689303 packet_exit:f(0,20)

i.e. it has value 0x34 (52). That's bad.

Now, there's two things wrong here. One of them is the bad value. I think that I've already seen this problem, and that it has something to do with the way stabs are and used to be emitted. packet_exit is presumably in the .text.exit section, which is presumably the problem. Before linking, the stab looked like:

2971 FUN 0 1870 0000000000000000 159366 packet_exit:f(0,20)

and had a relocation:
0000000000008b58 R_MIPS_32 .text.exit
unless I miss my guess.

So it looks like binutils did not relocate the stabs for .text.exit properly.

Why? It's pretty simple; there was nothing to relocate it to. From the kernel's linker script:

  /* Sections to be discarded */
  /DISCARD/ :
  {
        *(.text.exit)
        *(.data.exit)
        *(.exitcall.exit)
  }

So instead of the subtle error we get in objfiles containing multiple sections, which we'll still need to deal with for the kernel for .text.init, we have a completely bogus result.

We need to discard the stabs records for discarded symbols. Of course, we're just reading the psymtab in when we get here. We don't have symbols yet. We could do this by a second pass after reading, instead of the hack with textlow, but that's gross.

This makes it impossible to debug at least MIPS kernels with stabs and gdb, so I very much want to fix it. I'm not sure how this works on x86, but I'd guess it had to do with differences in relocation types. Anyone have an example handy?

Meanwhile, Steven, as a quick hack - try removing the /DISCARD/ bit from your linker script and relinking. What happens? Does everything else seem to work?

Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

The problem is worse than stabs. If a function is marked __exit _and_ some code in another section refers to that function then :-

* ld resolves the reference as offset xxx from the start of section .text.exit which is expected to get a decent start address.
* Section .text.exit is discarded, giving it a zero start address.
* The function call ends up as a branch to _address_ xxx.

This is a silent bug on many architectures, it only bites when the __exit function is called, usually on a rarely tested error path. On architectures that use PC relative branches (such as IA64), the linker may not be able to fit a PC relative branch from the high kernel address to the low (and incorrect) xxx address into an instruction so it gets an error during link. Section .data.exit is even worse, most references to data are via full word pointers and the bug is silent again.

ld should probably discard sections and all symbols in those sections before resolving external references.

Daniel Jacobowitz wrote:

[snip]

Success! Yes, leaving those sections in allows for the debugger to work properly. Here is output from a fresh gdb checked out of cvs this morning:

GNU gdb 2001-11-01-cvs
Copyright 2001 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu
--target=mips-linux-elf"...
(gdb) target remote /dev/ttyS1
Remote debugging using /dev/ttyS1
0x80012c88 in breakinst () at gdb-stub.c:907
907 __asm__ __volatile__("
(gdb) bt

   #0  0x80012c88 in breakinst () at gdb-stub.c:907
   #1  0x8020aabc in brcm_irq_setup () at irq.c:421
   #2  0x8020aaf0 in init_IRQ () at irq.c:434
   #3  0x801fc83c in start_kernel () at init/main.c:524
   #4  0x801fd6f8 in init_arch (argc=160, argv=0xb3000000, envp=0x2, 
       prom_vec=0xbf) at setup.c:425

(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap. 0x80012c88 in breakinst () at gdb-stub.c:907
907 __asm__ __volatile__("
(gdb) bt
#0 0x80012c88 in breakinst () at gdb-stub.c:907
#1 0x8001a554 in sys_create_module (name_user=0x10001dc8 "brcmdrv",

       size=713264) at module.c:305

(gdb) c
Continuing.

So, it would seem according to you and Keith, we have a linker bug that is specific to MIPS and other architectures that do not use PC relative branches that needs to be resolved?

I also tried a fresh checkout of the insight debugger from cvs this morning and it works great too. Thanks for the interim solution. I would like to be copied on remaining discussions including the design of the fix/patch. Thank you everyone.

-Steve

 Steven J. Hill - Embedded SW Engineer

... I wonder why you get a second SIGTRAP. Never happens to me. Quirk of your hardware?

No, I think that what Keith was saying described a worse side effect of the removed sections. I've no idea why this doesn't manifest on other platforms.

So can we work around this in GDB, or can we get the .stabs removed? Does ld edit the .stabs? I'll enquire over on that list.

Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

My bad. I have breakpoint instructions set inside 'kernel/module.c' so that I can single step through module insertions. I should have added a 'bt' output in gdb so you could see that. Everything is working great.

-Steve

 Steven J. Hill - Embedded SW Engineer