Skip to contentSkip to conversation

GDB Enhancements for device threaded core support

PATCH Superseded blloyd 5 participants 8 messages Oct 31, 2013 – Nov 3, 2013

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

blloyd

author
#0 lore raw

cover GDB Enhancements for device threaded core support

core file generation can create a broken core file when threading is used in the application. This change provides updates to the gdb package so multi-threaded applications debugging from failures works properly from end devices that include gdb.

The following changes since commit 523f2a9ea970713fb775bc48f84b67420b1106a0:

cross-canadian: Improve comment (2013-10-30 18:01:34 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib blloyd/gdb_assist
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=blloyd/gdb_assist

blloyd (1):
Enhancements to allow core file debugging to work properly from
devices

 meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
 meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
 2 files changed, 15 insertions(+), 6 deletions(-)
Signature · 1 line
1.8.1.4
author
#1 lore raw

PATCH 1/1 Enhancements to allow core file debugging to work properly from devices

gdb includes a script to make generating core files trivial. Add a package to allow inclusion of this script in images. GDB also requires symbols that are usually stripped from the threads library to properly generate a core dump. Therefore, ensure when gdb is installed the necessary debug file is also present so gdb is fully functional.

---
 meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
 meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
 2 files changed, 15 insertions(+), 6 deletions(-)
meta/recipes-devtools/gdb/gdb-common.inc +7 -5
index 6954954..02eef85 100644
@@ -61,8 +61,10 @@ do_install_append() {
 	rm -rf ${D}${datadir}/locale
 }
 
-RRECOMMENDS_gdb_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db "
-RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db "
-
+# Unfortunately, for gdb multi-threading to function reliably it must have the eglibc debug symbols
+# which are a part of the eglibc-dev package.  So we have added a dependency on that here.
+RRECOMMENDS_gdb_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RDEPENDS_gcore_append = " gdb "
meta/recipes-devtools/gdb/gdb.inc +8 -1
index 3321a24..9bf0e86 100644
@@ -10,5 +10,12 @@ SRC_URI += "file://kill_arm_map_symbols.patch \
 #export CFLAGS_append=" -L${STAGING_LIBDIR}"
 
 # cross-canadian must not see this
-PACKAGES =+ "gdbserver"
+PACKAGES =+ "gdbserver gcore"
 FILES_gdbserver = "${bindir}/gdbserver"
+FILES_gcore = "${bindir}/gcore"
+
+do_install_append() {
+  # When trying to find what went wrong with a program, a core dump can be very helpful
+  # Go ahead and make it easier to get one when we have gdb on the system
+  install -m 0755 ${S}/gdb/gdb_gcore.sh ${D}${bindir}/gcore
+}
Signature · 1 line
1.8.1.4
#2 lore raw

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

Quoted 18 lines · blloyd

gdb includes a script to make generating core files trivial. Add a package to allow inclusion of this script in images. GDB also requires symbols that are usually stripped from the threads library to properly generate a core dump. Therefore, ensure when gdb is installed the necessary debug file is also present so gdb is fully functional.

---
  meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
  meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
  2 files changed, 15 insertions(+), 6 deletions(-)
meta/recipes-devtools/gdb/gdb-common.inc +0 -2
index 6954954..02eef85 100644
@@ -61,8 +61,10 @@ do_install_append() {
  	rm -rf ${D}${datadir}/locale
  }
 
-RRECOMMENDS_gdb_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db "

meta/recipes-devtools/gdb/gdb-common.inc do_install_append() { · line 70

Quoted 10 lines
meta/recipes-devtools/gdb/gdb-common.inc +7 -3
-RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db "
-
+# Unfortunately, for gdb multi-threading to function reliably it must have the eglibc debug symbols
+# which are a part of the eglibc-dev package.  So we have added a dependency on that here.
+RRECOMMENDS_gdb_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RDEPENDS_gcore_append = " gdb "

The debug symbols should be in the -dbg package, not the -dev package. The -dev package should simply be symlinks and special development only components, while the -dbg is the split symbols.

With the eglibc-dbg installed onto the gdb environment, it should be possible to have all necessary symbols to do whatever processing is required. If this doesn't work, it likely needs to be investigated. (If something in -dev is required, likely there is another bug somewhere else that needs to be fixed in the packaging of eglibc.)

Do you have steps on using this code (a test command perhaps?). I've recently been asked to investigate some issues with debugging when using libpthread, and I suspect the problem may be over zealous stripping of the libpthread.so on the target. But I still haven't finished my evaluation yet.

--Mark

Quoted 19 lines
meta/recipes-devtools/gdb/gdb.inc +8 -1
index 3321a24..9bf0e86 100644
@@ -10,5 +10,12 @@ SRC_URI += "file://kill_arm_map_symbols.patch \
  #export CFLAGS_append=" -L${STAGING_LIBDIR}"
 
  # cross-canadian must not see this
-PACKAGES =+ "gdbserver"
+PACKAGES =+ "gdbserver gcore"
  FILES_gdbserver = "${bindir}/gdbserver"
+FILES_gcore = "${bindir}/gcore"
+
+do_install_append() {
+  # When trying to find what went wrong with a program, a core dump can be very helpful
+  # Go ahead and make it easier to get one when we have gdb on the system
+  install -m 0755 ${S}/gdb/gdb_gcore.sh ${D}${bindir}/gcore
+}
#3 lore raw

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

The patch adds gcore when gcore package is added to the install packages list for an image. That gives a simple core generation wrapper around gcc, which might simplify your testing.
It appears -dev includes -dbg, and the file I want is in -dbg, so I can submit a new patch limiting to only -dbg just fine. The -dev for glib-thread-db isn't very big so I didn't catch the extra files not really needed. I have used this in an image that was still 50mbish in size. Since before I made the patch, I made a hybrid system by manually placing the one debug symbols file onto the disk, and once the symbols file was in place cores generated became readable, and before that the cores were unreadable, I have strong evidence only the debug symbols are needed.

gcore PID
is the syntax to create a core dump using the helper script. You can also use gdb directly to debug or generate a core dump. If the program is multi-threaded, without the above patch a core loaded into a system that has full -dev config (either an SDK or larger system image for development) could not properly read the core dump. With the above patch or adding the debug symbol file manually, the core could be read by the debugger. Since I am building a i686 based system, I am able to run my images on a PC in a sysroot, which made my testing easier. I also tried via a standard end system type disk on a device and a development environment loaded on the device, and got the same results.

Brian

Quoted 72 lines · Mark Hatle
Quoted 18 lines · blloyd

gdb includes a script to make generating core files trivial. Add a package to allow inclusion of this script in images. GDB also requires symbols that are usually stripped from the threads library to properly generate a core dump. Therefore, ensure when gdb is installed the necessary debug file is also present so gdb is fully functional.

---
  meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
  meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
  2 files changed, 15 insertions(+), 6 deletions(-)
meta/recipes-devtools/gdb/gdb-common.inc +0 -2
index 6954954..02eef85 100644
@@ -61,8 +61,10 @@ do_install_append() {
  	rm -rf ${D}${datadir}/locale
  }
 
-RRECOMMENDS_gdb_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db "

meta/recipes-devtools/gdb/gdb-common.inc do_install_append() { · line 70

Quoted 10 lines
meta/recipes-devtools/gdb/gdb-common.inc +7 -3
-RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db "
-
+# Unfortunately, for gdb multi-threading to function reliably it must have the eglibc debug symbols
+# which are a part of the eglibc-dev package.  So we have added a dependency on that here.
+RRECOMMENDS_gdb_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RDEPENDS_gcore_append = " gdb "

The debug symbols should be in the -dbg package, not the -dev package. The -dev package should simply be symlinks and special development only components, while the -dbg is the split symbols.

With the eglibc-dbg installed onto the gdb environment, it should be possible to have all necessary symbols to do whatever processing is required. If this doesn't work, it likely needs to be investigated. (If something in -dev is required, likely there is another bug somewhere else that needs to be fixed in the packaging of eglibc.)

Do you have steps on using this code (a test command perhaps?). I've recently been asked to investigate some issues with debugging when using libpthread, and I suspect the problem may be over zealous stripping of the libpthread.so on the target. But I still haven't finished my evaluation yet.

--Mark

Quoted 9 lines
meta/recipes-devtools/gdb/gdb.inc +0 -0
index 3321a24..9bf0e86 100644
@@ -10,5 +10,12 @@ SRC_URI += "file://kill_arm_map_symbols.patch \
  #export CFLAGS_append=" -L${STAGING_LIBDIR}"
 
  # cross-canadian must not see this

meta/recipes-devtools/gdb/gdb.inc SRC_URI += "file://kill_arm_map_symbols.patch \ · line 21

Quoted 10 lines
meta/recipes-devtools/gdb/gdb.inc +8 -1
-PACKAGES =+ "gdbserver"
+PACKAGES =+ "gdbserver gcore"
  FILES_gdbserver = "${bindir}/gdbserver"
+FILES_gcore = "${bindir}/gcore"
+
+do_install_append() {
+  # When trying to find what went wrong with a program, a core dump can be very helpful
+  # Go ahead and make it easier to get one when we have gdb on the system
+  install -m 0755 ${S}/gdb/gdb_gcore.sh ${D}${bindir}/gcore
+}

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#4 lore raw

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

In reply to blloyd
Quoted 1 line · blloyd

+FILES_gcore = "${bindir}/gcore"

Why can't this script just be added to the gdb package?

Ross

#5 lore raw

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

Quoted 5 lines · Ross
Quoted 1 line · On 31 October 2013 06:54, blloyd <[email protected] <javascript:;>> wrote:

+FILES_gcore = "${bindir}/gcore"

Why can't this script just be added to the gdb package?

actually if we have it as a separate package then all those new rrcommends baggage can be added to gcore package instead of gdb proper

Quoted 7 lines

Ross
_______________________________________________
Openembedded-core mailing list [email protected] <javascript:;>
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#6 lore raw

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

Thank you for the interest.

Personally, I wouldn't mind just adding gcore to the base package, as it isn't very large. However, it really is a support script that isn't necessary for gdb or gdb-server. It does make testing the other part of the patch easier, though. But as it isn't strictly necessary, I opted to let people opt in or out of having it.
I am hoping in the future there will be a way to view available packages so end users can browse a list and add things without having to have pre-knowledge that it exists to go looking for it.

I do not think the -dbg symbols should be a requirement of the gcore package. It wasn't while using gcore that I found the missing dependency, but actually trying to analyze crash dumps created with gdb installed on the device and a segfaulting multi-threaded program. All core files produced would have no more than one usable thread's data in it, and it never seemed to be from the crashing thread. OS level crash dumps and gdb debugging of the application both were not properly functional until those symbols were placed on the device, after which both worked. Therefore, it seems appropriate to place the dependency against the gdb package as it fixes a defect with the program installed by that package. While I haven't been using gdb-server on the device, the research I did pointed to that program also needing those symbols, so the dependency was added there too.

As I have never used a system that didn't have gdb installed, I am not sure whether core dumps can be generated without the debugger installed. If they can, perhaps the eglibc package itself should be made NO_STRIP or else eglibc made to recommend it's own debug symbols. I don't really like the no strip idea, as stripped binaries are smaller and thus faster to load and these symbols are not needed very often, so why force them to be loaded all the time, but it is an option.

As far as testing this, I could make a small multi-threaded application that intentionally dereferences a null pointer in one of 4 threads spawned. That can show the broken nature of gdb core generation without the thread library symbols. If this is needed, I will put one together and just reply here that it can be pulled from the poky-contrib repository. I found and resolved the problem while working on a project my company is not yet ready to release.

However, if you just want to see that gdb doesn't handle threads properly, you can just pick a multi-threaded program, do a core dump (the gcore script makes this trivial), and then load the core file in a debugger on the development system. Even though the development system has the symbols, that core won't be readable except for one thread.

I will do another patch fixing the commit message and incorporating improvements we identify during this discussion.

Brian

Quoted 22 lines · Khem Raj

On Friday, November 1, 2013, Burton, Ross wrote:

        On 31 October 2013 06:54, blloyd <[email protected]>
        wrote:
        > +FILES_gcore = "${bindir}/gcore"
        
        Why can't this script just be added to the gdb package?

actually if we have it as a separate package then all those new rrcommends baggage can be added to gcore package instead of gdb proper

Ross
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#7 lore raw

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

In reply to blloyd
Quoted 18 lines · blloyd

gdb includes a script to make generating core files trivial. Add a package to allow inclusion of this script in images. GDB also requires symbols that are usually stripped from the threads library to properly generate a core dump. Therefore, ensure when gdb is installed the necessary debug file is also present so gdb is fully functional.

---
 meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
 meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
 2 files changed, 15 insertions(+), 6 deletions(-)
meta/recipes-devtools/gdb/gdb-common.inc +0 -2
index 6954954..02eef85 100644
@@ -61,8 +61,10 @@ do_install_append() {
 	rm -rf ${D}${datadir}/locale
 }
 
-RRECOMMENDS_gdb_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db "

meta/recipes-devtools/gdb/gdb-common.inc do_install_append() { · line 70

Quoted 10 lines
meta/recipes-devtools/gdb/gdb-common.inc +7 -3
-RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db "
-
+# Unfortunately, for gdb multi-threading to function reliably it must have the eglibc debug symbols
+# which are a part of the eglibc-dev package.  So we have added a dependency on that here.
+RRECOMMENDS_gdb_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RDEPENDS_gcore_append = " gdb "

Aren't they part of the -dbg package, not -dev?

You also need to put "gdb: " at the start of the commit message summary as per the commit message guidelines. It may be worth splitting this change into two commits too.

Cheers,

Richard