Skip to contentSkip to conversation

oeqa/selftest/minidebuginfo: Create selftest for minidebuginfo

PATCH In review Nathan Rossi 2 participants 5 messages Nov 3, 2022 – Nov 8, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 oeqa/selftest/minidebuginfo: Create selftest for minidebuginfo

From: Nathan Rossi <[email protected]>

Add a new selftest to validate minidebuginfo support. This selftest builds a complete target image with PACKAGE_MINIDEBUGINFO enabled. ELFs included in the image are expected to have minidebuginfo included in the resulting executables and shared libraries, the self test validates this by unpacking the image and checking for the associated ".gnu_debugdata" section on busybox and libc ELFs.

---
 meta/lib/oeqa/selftest/cases/minidebuginfo.py | 43 +++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/minidebuginfo.py
meta/lib/oeqa/selftest/cases/minidebuginfo.py new file +43 -0
new file mode 100644
index 0000000000..7947c3803c
@@ -0,0 +1,43 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+import os
+import subprocess
+import tempfile
+import shutil
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var, runCmd
+
+
+class Minidebuginfo(OESelftestTestCase):
+    def test_minidebuginfo(self):
+        target_sys = get_bb_var("TARGET_SYS")
+        binutils = "binutils-cross-{}".format(get_bb_var("TARGET_ARCH"))
+
+        self.write_config("""
+PACKAGE_MINIDEBUGINFO = "1"
+IMAGE_FSTYPES = "tar.bz2"
+""")
+        bitbake("core-image-minimal {}:do_addto_recipe_sysroot".format(binutils))
+
+        deploy_dir = get_bb_var("DEPLOY_DIR_IMAGE")
+        native_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", binutils)
+        readelf = get_bb_var("READELF", "core-image-minimal")
+
+        # confirm that executables and shared libraries contain an ELF section
+        # ".gnu_debugdata" which stores minidebuginfo.
+        with tempfile.TemporaryDirectory(prefix = "unpackfs-") as unpackedfs:
+            filename = os.path.join(deploy_dir, "core-image-minimal-{}.tar.bz2".format(self.td["MACHINE"]))
+            shutil.unpack_archive(filename, unpackedfs)
+
+            r = runCmd([readelf, "-W", "-S", os.path.join(unpackedfs, "bin", "busybox")],
+                    native_sysroot = native_sysroot, target_sys = target_sys)
+            self.assertIn(".gnu_debugdata", r.output)
+
+            r = runCmd([readelf, "-W", "-S", os.path.join(unpackedfs, "lib", "libc.so.6")],
+                    native_sysroot = native_sysroot, target_sys = target_sys)
+            self.assertIn(".gnu_debugdata", r.output)
+

---
2.37.2

#1 lore raw

PATCH 2/3 glibc-locale: Do not INHIBIT_DEFAULT_DEPS

From: Nathan Rossi <[email protected]>

The glibc-locale recipe already partially depends on the base depends in order to satisfy the do_package dependency on binutils. However since commit d6ffd683bf6 NM has defaulted to gcc-nm, meaning do_package depends on gcc (for minidebuginfo).

Whilst the do_package task could handle having the dependencies explicitly defined (either in glibc-locale or in package.bbclass), setting these would require some amount of conditional dependency configuration (cross/crosssdk/etc.). Since both binutils and gcc are already dependencies of virtual/libc (although compilerlibs is not), having glibc-locale not inhibit the default depends simplifies the handling of this situation for both glibc-locale and package.bbclass.

---
 meta/recipes-core/glibc/glibc-locale.inc | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)
meta/recipes-core/glibc/glibc-locale.inc +3 -8
index 7c14abfe99..7f70b3ca4f 100644
@@ -5,14 +5,9 @@ SUMMARY = "Locale data from glibc"
 BPN = "glibc"
 LOCALEBASEPN = "${MLPREFIX}glibc"
 
-# glibc-collateral.inc inhibits all default deps, but do_package needs objcopy
-# ERROR: objcopy failed with exit code 127 (cmd was 'i586-webos-linux-objcopy' --only-keep-debug 'glibc-locale/2.17-r0/package/usr/lib/gconv/IBM1166.so' 'glibc-locale/2.17-r0/package/usr/lib/gconv/.debug/IBM1166.so')
-# ERROR: Function failed: split_and_strip_files
-BINUTILSDEP = "virtual/${MLPREFIX}${TARGET_PREFIX}binutils:do_populate_sysroot"
-BINUTILSDEP:class-nativesdk = "virtual/${TARGET_PREFIX}binutils-crosssdk:do_populate_sysroot"
-do_package[depends] += "${BINUTILSDEP}"
-
-DEPENDS += "virtual/libc"
+# Do not inhibit default deps, do_package requires binutils/gcc for
+# objcopy/gcc-nm and glibc-locale depends on virtual/libc directly.
+INHIBIT_DEFAULT_DEPS = ""
 
 # Binary locales are generated at build time if ENABLE_BINARY_LOCALE_GENERATION
 # is set. The idea is to avoid running localedef on the target (at first boot)

---
2.37.2

#2 lore raw

PATCH 3/3 package: Fix handling of minidebuginfo with newer binutils

In reply to Nathan Rossi

From: Nathan Rossi <[email protected]>

Newer versions of binutils (2.38+) have changed how the "--only-keep-debug" of objcopy behaves when stripping non-debug sections from an ELF.

https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=68f543154e92ab0f5d6c569e0fa143f5e8bd2d80

This change causes associated sections to be correctly marked as NOBITS with the section contents removed from the output. The side effect is that this causes issues with objcopy's ability to perform symbol and relocation stripping (-S/--strip-all) on the debug split ELF, such that with some object files (e.g. kernel modules) objcopy fails to strip symbols/relocations with an error like the following:

.../.debug/nls_cp950.ko[.rodata]: file truncated

Because of this it is now problematic to generate minidebuginfo for these types of ELF objects. However it is not typically useful to inject minidebuginfo into these types of ELFs, and other distributions (e.g. Fedora, referring to find-debuginfo.sh of debugedit) only insert minidebuginfo into executables and shared libraries.

This change causes the minidebuginfo injection to only apply to EXEC/DYN type ELFs, which limits the injection to executables and shared libraires.

Additionally this change fixes the parsing of the sections from the "readelf -W -S" output which was not accounting for the section index column having leading spaces for single digit index values e.g. "[ 1]".

---
 meta/classes-global/package.bbclass | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
meta/classes-global/package.bbclass +18 -3
index 2d985d8aff..7a0a428b30 100644
@@ -490,16 +490,31 @@ def inject_minidebuginfo(file, dvar, dv, d):
         bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfo injection'.format(file))
         return
 
+    # minidebuginfo does not make sense to apply to ELF objects other than
+    # executables and shared libraries, skip applying the minidebuginfo
+    # generation for objects like kernel modules.
+    for line in subprocess.check_output([readelf, '-h', debugfile], universal_newlines=True).splitlines():
+        if not line.strip().startswith("Type:"):
+            continue
+        elftype = line.split(":")[1].strip()
+        if not any(elftype.startswith(i) for i in ["EXEC", "DYN"]):
+            bb.debug(1, 'ELF file {} is not executable/shared, skipping minidebuginfo injection'.format(file))
+            return
+        break
+
     # Find non-allocated PROGBITS, NOTE, and NOBITS sections in the debuginfo.
     # We will exclude all of these from minidebuginfo to save space.
     remove_section_names = []
     for line in subprocess.check_output([readelf, '-W', '-S', debugfile], universal_newlines=True).splitlines():
-        fields = line.split()
-        if len(fields) < 8:
+        # strip the leading "  [ 1]" section index to allow splitting on space
+        if ']' not in line:
+            continue
+        fields = line[line.index(']') + 1:].split()
+        if len(fields) < 7:
             continue
         name = fields[0]
         type = fields[1]
-        flags = fields[7]
+        flags = fields[6]
         # .debug_ sections will be removed by objcopy -S so no need to explicitly remove them
         if name.startswith('.debug_'):
             continue

---
2.37.2

#3 lore raw

Re: [OE-core] [PATCH 3/3] package: Fix handling of minidebuginfo with newer binutils

Hello,

This series causes the following failure on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4354/steps/14/logs/stdio

2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add_bindir: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add_machine: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_alternatives: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_basic: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_binary: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_extlayer: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_func: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_glob: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_todir_glob: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_invalid: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_orig: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_patch: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_postinstall: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_renamed: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_script: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_subdir: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_todir: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_wildcard: ERROR
Quoted 87 lines · Nathan Rossi

From: Nathan Rossi <[email protected]>

Newer versions of binutils (2.38+) have changed how the "--only-keep-debug" of objcopy behaves when stripping non-debug sections from an ELF.

https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=68f543154e92ab0f5d6c569e0fa143f5e8bd2d80

This change causes associated sections to be correctly marked as NOBITS with the section contents removed from the output. The side effect is that this causes issues with objcopy's ability to perform symbol and relocation stripping (-S/--strip-all) on the debug split ELF, such that with some object files (e.g. kernel modules) objcopy fails to strip symbols/relocations with an error like the following:

.../.debug/nls_cp950.ko[.rodata]: file truncated

Because of this it is now problematic to generate minidebuginfo for these types of ELF objects. However it is not typically useful to inject minidebuginfo into these types of ELFs, and other distributions (e.g. Fedora, referring to find-debuginfo.sh of debugedit) only insert minidebuginfo into executables and shared libraries.

This change causes the minidebuginfo injection to only apply to EXEC/DYN type ELFs, which limits the injection to executables and shared libraires.

Additionally this change fixes the parsing of the sections from the "readelf -W -S" output which was not accounting for the section index column having leading spaces for single digit index values e.g. "[ 1]".

---
 meta/classes-global/package.bbclass | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
meta/classes-global/package.bbclass +18 -4
index 2d985d8aff..7a0a428b30 100644
@@ -490,16 +490,31 @@ def inject_minidebuginfo(file, dvar, dv, d):
         bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfo injection'.format(file))
         return
 
+    # minidebuginfo does not make sense to apply to ELF objects other than
+    # executables and shared libraries, skip applying the minidebuginfo
+    # generation for objects like kernel modules.
+    for line in subprocess.check_output([readelf, '-h', debugfile], universal_newlines=True).splitlines():
+        if not line.strip().startswith("Type:"):
+            continue
+        elftype = line.split(":")[1].strip()
+        if not any(elftype.startswith(i) for i in ["EXEC", "DYN"]):
+            bb.debug(1, 'ELF file {} is not executable/shared, skipping minidebuginfo injection'.format(file))
+            return
+        break
+
     # Find non-allocated PROGBITS, NOTE, and NOBITS sections in the debuginfo.
     # We will exclude all of these from minidebuginfo to save space.
     remove_section_names = []
     for line in subprocess.check_output([readelf, '-W', '-S', debugfile], universal_newlines=True).splitlines():
-        fields = line.split()
-        if len(fields) < 8:
+        # strip the leading "  [ 1]" section index to allow splitting on space
+        if ']' not in line:
+            continue
+        fields = line[line.index(']') + 1:].split()
+        if len(fields) < 7:
             continue
         name = fields[0]
         type = fields[1]
-        flags = fields[7]
+        flags = fields[6]
         # .debug_ sections will be removed by objcopy -S so no need to explicitly remove them
         if name.startswith('.debug_'):
             continue
---

2.37.2

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#172617): https://lists.openembedded.org/g/openembedded-core/message/172617
Mute This Topic: https://lists.openembedded.org/mt/94752153/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

Re: [OE-core] [PATCH 3/3] package: Fix handling of minidebuginfo with newer binutils

Quoted 6 lines · Alexandre Belloni

Hello,

This series causes the following failure on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4354/steps/14/logs/stdio

I looked into these failures, the SRC_URI error appears to be an issue with the lic_checksum test itself, and it presents without this series. I have posted a patch to resolve it
https://lists.openembedded.org/g/openembedded-core/message/172945.

The other errors around "test.bb" bash hash changes do not appear to be errors from the recipetools bitbake execution (note the "2 ERROR messages"). I suspect they are just captured during the fetch selftest cases in a common stderr buffer? I was unable to reproduce them locally when running fetch/recipetool.

Regards,
Nathan

Quoted 115 lines
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add_bindir: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add_machine: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_alternatives: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_basic: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_binary: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_extlayer: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_func: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_glob: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_todir_glob: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_invalid: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_orig: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_patch: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_postinstall: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_renamed: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_script: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_subdir: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_todir: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_wildcard: ERROR
Quoted 79 lines · Nathan Rossi

From: Nathan Rossi <[email protected]>

Newer versions of binutils (2.38+) have changed how the "--only-keep-debug" of objcopy behaves when stripping non-debug sections from an ELF.

https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=68f543154e92ab0f5d6c569e0fa143f5e8bd2d80

This change causes associated sections to be correctly marked as NOBITS with the section contents removed from the output. The side effect is that this causes issues with objcopy's ability to perform symbol and relocation stripping (-S/--strip-all) on the debug split ELF, such that with some object files (e.g. kernel modules) objcopy fails to strip symbols/relocations with an error like the following:

.../.debug/nls_cp950.ko[.rodata]: file truncated

Because of this it is now problematic to generate minidebuginfo for these types of ELF objects. However it is not typically useful to inject minidebuginfo into these types of ELFs, and other distributions (e.g. Fedora, referring to find-debuginfo.sh of debugedit) only insert minidebuginfo into executables and shared libraries.

This change causes the minidebuginfo injection to only apply to EXEC/DYN type ELFs, which limits the injection to executables and shared libraires.

Additionally this change fixes the parsing of the sections from the "readelf -W -S" output which was not accounting for the section index column having leading spaces for single digit index values e.g. "[ 1]".

---
 meta/classes-global/package.bbclass | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
meta/classes-global/package.bbclass +18 -4
index 2d985d8aff..7a0a428b30 100644
@@ -490,16 +490,31 @@ def inject_minidebuginfo(file, dvar, dv, d):
         bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfo injection'.format(file))
         return
 
+    # minidebuginfo does not make sense to apply to ELF objects other than
+    # executables and shared libraries, skip applying the minidebuginfo
+    # generation for objects like kernel modules.
+    for line in subprocess.check_output([readelf, '-h', debugfile], universal_newlines=True).splitlines():
+        if not line.strip().startswith("Type:"):
+            continue
+        elftype = line.split(":")[1].strip()
+        if not any(elftype.startswith(i) for i in ["EXEC", "DYN"]):
+            bb.debug(1, 'ELF file {} is not executable/shared, skipping minidebuginfo injection'.format(file))
+            return
+        break
+
     # Find non-allocated PROGBITS, NOTE, and NOBITS sections in the debuginfo.
     # We will exclude all of these from minidebuginfo to save space.
     remove_section_names = []
     for line in subprocess.check_output([readelf, '-W', '-S', debugfile], universal_newlines=True).splitlines():
-        fields = line.split()
-        if len(fields) < 8:
+        # strip the leading "  [ 1]" section index to allow splitting on space
+        if ']' not in line:
+            continue
+        fields = line[line.index(']') + 1:].split()
+        if len(fields) < 7:
             continue
         name = fields[0]
         type = fields[1]
-        flags = fields[7]
+        flags = fields[6]
         # .debug_ sections will be removed by objcopy -S so no need to explicitly remove them
         if name.startswith('.debug_'):
             continue
---

2.37.2

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#172617): https://lists.openembedded.org/g/openembedded-core/message/172617
Mute This Topic: https://lists.openembedded.org/mt/94752153/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com