thread debugging with target gdb

Hi,
I am using buildroot 2012.08 , configured for ARM cortex A9, crosstools and glibc-2.11.
The glibc has been built with 2.6.38.8 headers, my running kernel is 3.2.21, in qemu for versatile express.

The issue is that thread debugging does not work, first because /lib/libthread_db.so.1 was not pushed on the 'target' directory I 'solved' this by adding it:

CTNG_LIBS_LIB := ld*.so libc.so libcrypt.so libdl.so libgcc_s.so
libm.so \
libnsl.so libpthread.so libresolv.so librt.so
libutil.so \
libthread_db.so

... in crosstools-ng.mk

but is is maybe not the right way.

I have activated some debug log in gdb, and I am falling into those issues.

Reading symbols from /root/ihm...done.
(gdb) r
Starting program: /root/ihm
Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1. td_ta_new failed: application not linked with libthread thread_db_load_search returning 0
Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1. td_ta_new failed: application not linked with libthread Trying host libthread_db library: /lib/libthread_db.so.1. td_ta_new failed: application not linked with libthread thread_db_load_search returning 0
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Error: Can't open display:
[Inferior 1 (process 612) exited with code 01]

Thus the "td_ta_new" symbol is found but the call fails

Any clues or hints ?
Regards
Thierry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120919/517f2ca4/attachment.html>

It should probably be:

ifeq ($(BR2_PACKAGE_GDB_SERVER),y)
CTNG_LIBS_LIB += libthread_db.so
endif

But indeed, it looks like this is missing in crosstools-ng.mk.

Yann?

Are you running full gdb on the target? You may have more luck with
gdbserver on the target and output/host/usr/bin/arm-*-gdb on the host. Don't forget to do 'set sysroot .../output/staging' in gdb.

[snip]

Regards,
Arnout

Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Actually I am not currently -debugging- a multithreaded process, but building and configuring a platform for other developers that will require full gdb.
?
On the web, I have found some contradictory information about the libpthread and libthread_db stripping. And moreover:

On my host (Ubuntu) libthread_db is stripped and libpthread is not In my Buildroot, libthread_db is not stripped, and libpthread is !!

By reading at the sources of GDB, it does not seem that it uses debug information of libthread_db, but only performs dlopen on it, dlsym, and calls "td_ta_new" ?
I have also seeked into the nptl-db sources, leading to the conclusion that the symbol containing the NPTL version was not found in libpthread, and it does not get any further.

I really wonder if the crosstools logic for stripping these libraries is in the right way. I will try to do the opposite, as soon as I get more familiar with the crosstools build.

Of course, if you have any ideas, that would save me a lot of time

Trying gdbserver is my next step, but I feel pessimistic / my conclusions above.

Cheers
Thierry

-----Original message-----
From: Arnout Vandecappelle?<[email protected]>
Sent: Wed 19-Sep-2012 23:04
To: Thierry Bultel <[email protected]>
Cc: buildroot at busybox.net
Subject: Re: [Buildroot] thread debugging with target gdb

It should probably be:

ifeq ($(BR2_PACKAGE_GDB_SERVER),y)
CTNG_LIBS_LIB += libthread_db.so
endif

But indeed, it looks like this is missing in crosstools-ng.mk.

Yann?

Are you running full gdb on the target? You may have more luck with
gdbserver on the target and output/host/usr/bin/arm-*-gdb on the host. Don't forget to do 'set sysroot .../output/staging' in gdb.

[snip]

Regards,
Arnout

Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be <http://www.mind.be> 
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle <http://www.linkedin.com/in/arnoutvandecappelle> 
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120919/d8b8774b/attachment.html>

Most BR developers use gdbserver only, so that's better supported than full gdb on
the target.

Regards,
Arnout

Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

I finnally solved my problem, and threads debugging now works fine.

The fix consists in 2 things

1) adding libpthreads_db.so on target
Just like you said, this can be achieved in croostools-ng.mk like this:

ifeq ($(BR2_PACKAGE_GDB_SERVER),y)
CTNG_LIBS_LIB += libthread_db.so
endif

But I would also add:

ifeq ($(BR2_PACKAGE_GDB),y)
CTNG_LIBS_LIB += libthread_db.so
endif

since full gdb definitively needs it. Whatever ORing mecanism should maybe be done, not to add the lib twice, as options are not exclusive.

2) Not stripping libpthread.so

In the top Makefile:

ifeq ($(BR2_PACKAGE_GDB),y)
BR2_STRIP_EXCLUDE_FILES += libpthread*.so*
endif

Thanks for your help, is there a chance that those fixes get integrated in the mainline ?

Cheers
Thierry

-----Original message-----
From: Arnout Vandecappelle?<[email protected]>
Sent: Wed 19-Sep-2012 23:30
To: Thierry Bultel <[email protected]>
Cc: buildroot at busybox.net
Subject: Re: [Buildroot] thread debugging with target gdb

Most BR developers use gdbserver only, so that's better supported than full gdb on
the target.

Regards,
Arnout

Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be <http://www.mind.be> 
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle <http://www.linkedin.com/in/arnoutvandecappelle> 
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F ifeq ($(BR2_PACKAGE_GDB),y)

ifeq ($(BR2_PACKAGE_GDB),y)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120920/bb972315/attachment.html>

Dear Thierry Bultel,

Yes, sure. Could you send a proper patch? This way we can keep attribution, and credit you for the investigation of this problem.

Thanks!

Thomas

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

The libthread_db was not copied on target
GDB needs libpthread to be not stripped

diff -Naur buildroot-2012.08.orig/Makefile buildroot-2012.08/Makefile
@@ -416,6 +416,10 @@
  
  $(TARGET_DIR): $(BUILD_DIR)/.root
  
+ifeq ($(BR2_PACKAGE_GDB),y)
+BR2_STRIP_EXCLUDE_FILES += libpthread*.so*
+endif
+
  STRIP_FIND_CMD = find $(TARGET_DIR)
  ifneq (,$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS)))
  STRIP_FIND_CMD += \( $(call finddirclauses,$(TARGET_DIR),$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS))) \) -prune -o

diff -Naur buildroot-2012.08.orig/toolchain/toolchain-crosstool-ng/crosstool-ng.mk buildroot-2012.08/toolchain/toolchain-crosstool-ng/crosstool-ng.mk
@@ -41,6 +41,18 @@
  CTNG_LIBS_LIB := ld*.so libc.so libcrypt.so libdl.so libgcc_s.so libm.so    \
                   libnsl.so libpthread.so libresolv.so librt.so libutil.so
  
+ifeq ($(BR2_PACKAGE_GDB_SERVER),y)
+CTNG_LIBS_LIB_THREAD_DB_NEEDED := y
+endif
+
+ifeq ($(BR2_PACKAGE_GDB),y)
+CTNG_LIBS_LIB_THREAD_DB_NEEDED := y
+endif
+
+ifeq ($(CTNG_LIBS_LIB_THREAD_DB_NEEDED),y)
+CTNG_LIBS_LIB += libthread_db.so
+endif
+
  #--------------
  # The libc-specific system libraries (in /lib)
  # Note: it may be needed to tweak the NSS libs in the glibc and eglibc cases...