package.bbclass: inject "minidebuginfo" into packaged binaries
PATCH 1/1 package.bbclass: inject "minidebuginfo" into packaged binaries
"Mini debuginfo" is a special section in ELF executables containing minimal compressed debuginfo for non-exported symbols:
https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html
It lets debugging tools produce better stack traces, including local function names, without incurring the space overhead of full debuginfo. The feature was originally developed for Fedora:
https://fedoraproject.org/wiki/Features/MiniDebugInfo
but nowadays it is widely supported in the ecosystem, including in gdb and elfutils (and therefore also in tools which use elfutils, such as systemd-coredump).
This patch adds an optional extra step in package.bbclass to inject minidebuginfo while stripping and splitting out debuginfo. It can be enabled by setting PACKAGE_MINIDEBUGINFO=1. In my testing, this increases the size of resulting binaries by roughly 5%.
The code for producing and re-injecting the minidebuginfo is my own Python implementation but corresponds directly to the shell implementation that RPM uses for doing the same:
--- meta/classes/package.bbclass | 84 ++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+)
meta/classes/package.bbclass +84 -0
@@ -245,6 +245,8 @@ python () { deps = "" for dep in (d.getVar('PACKAGE_DEPENDS') or "").split(): deps += " %s:do_populate_sysroot" % dep + if d.getVar('PACKAGE_MINIDEBUGINFO') == '1': + deps += ' xz-native:do_populate_sysroot' d.appendVarFlag('do_package', 'depends', deps) # shlibs requires any DEPENDS to have already packaged for the *.list files
@@ -459,6 +461,83 @@ def splitstaticdebuginfo(file, dvar, debugstaticdir, debugstaticlibdir, debugsta return (file, sources) +def inject_minidebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d): + # Extract just the symbols from debuginfo into minidebuginfo, + # compress it with xz and inject it back into the binary in a .gnu_debugdata section. + # https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html + + import subprocess + + readelf = d.getVar('READELF') + nm = d.getVar('NM') + objcopy = d.getVar('OBJCOPY') + + minidebuginfodir = d.expand('${WORKDIR}/minidebuginfo') + + src = file[len(dvar):] + dest = debuglibdir + os.path.dirname(src) + debugdir + "/" + os.path.basename(src) + debugappend + debugfile = dvar + dest + minidebugfile = minidebuginfodir + src + '.minidebug' + bb.utils.mkdirhier(os.path.dirname(minidebugfile)) + + # If we didn't produce debuginfo for any reason, we can't produce minidebuginfo either + # so skip it. + if not os.path.exists(debugfile): + bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfo injection'.format(file)) + return + + # Find non-allocated PROGBITS, NOTE, and NOBITS sections in the debuginfo. + # We will exclude all of these from minidebuginfo to save space. + remove_section_names = [] + for line in subprocess.check_output([readelf, '-W', '-S', debugfile], universal_newlines=True).splitlines(): + fields = line.split() + if len(fields) < 8: + continue + name = fields[0] + type = fields[1] + flags = fields[7] + # .debug_ sections will be removed by objcopy -S so no need to explicitly remove them + if name.startswith('.debug_'): + continue + if 'A' not in flags and type in ['PROGBITS', 'NOTE', 'NOBITS']: + remove_section_names.append(name) + + # List dynamic symbols in the binary. We can exclude these from minidebuginfo + # because they are always present in the binary. + dynsyms = set() + for line in subprocess.check_output([nm, '-D', file, '--format=posix', '--defined-only'], universal_newlines=True).splitlines(): + dynsyms.add(line.split()[0]) + + # Find all function symbols from debuginfo which aren't in the dynamic symbols table. + # These are the ones we want to keep in minidebuginfo. + keep_symbols_file = minidebugfile + '.symlist' + found_any_symbols = False + with open(keep_symbols_file, 'w') as f: + for line in subprocess.check_output([nm, debugfile, '--format=sysv', '--defined-only'], universal_newlines=True).splitlines(): + fields = line.split('|') + if len(fields) < 7: + continue + name = fields[0].strip() + type = fields[3].strip() + if type == 'FUNC' and name not in dynsyms: + f.write('{}\n'.format(name)) + found_any_symbols = True + + if not found_any_symbols: + bb.debug(1, 'ELF file {} contains no symbols, skipping minidebuginfo injection'.format(file)) + return + + bb.utils.remove(minidebugfile) + bb.utils.remove(minidebugfile + '.xz') + + subprocess.check_call([objcopy, '-S'] + + ['--remove-section={}'.format(s) for s in remove_section_names] + + ['--keep-symbols={}'.format(keep_symbols_file), debugfile, minidebugfile]) + + subprocess.check_call(['xz', '--keep', minidebugfile]) + + subprocess.check_call([objcopy, '--add-section', '.gnu_debugdata={}.xz'.format(minidebugfile), file]) + def copydebugsources(debugsrcdir, sources, d): # The debug src information written out to sourcefile is further processed # and copied to the destination here.
@@ -1185,6 +1264,11 @@ python split_and_strip_files () { oe.utils.multiprocess_launch(oe.package.runstrip, sfiles, d) + # Build "minidebuginfo" and reinject it back into the stripped binaries + if d.getVar('PACKAGE_MINIDEBUGINFO') == '1': + oe.utils.multiprocess_launch(inject_minidebuginfo, list(elffiles), d, + extraargs=(dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d)) + # # End of strip #
Signature · 1 line
2.21.1
Thanks! I'd propose this is actually enabled by default, with a possibility to disable for those situations where 5% extra is critical.
Alex
Quoted 168 lines · Dan Callaghan
"Mini debuginfo" is a special section in ELF executables containing minimal compressed debuginfo for non-exported symbols:
https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html
It lets debugging tools produce better stack traces, including local function names, without incurring the space overhead of full debuginfo. The feature was originally developed for Fedora:
https://fedoraproject.org/wiki/Features/MiniDebugInfo
but nowadays it is widely supported in the ecosystem, including in gdb and elfutils (and therefore also in tools which use elfutils, such as systemd-coredump).
This patch adds an optional extra step in package.bbclass to inject minidebuginfo while stripping and splitting out debuginfo. It can be enabled by setting PACKAGE_MINIDEBUGINFO=1. In my testing, this increases the size of resulting binaries by roughly 5%.
The code for producing and re-injecting the minidebuginfo is my own Python implementation but corresponds directly to the shell implementation that RPM uses for doing the same:
Signed-off-by Dan Callaghan <[email protected]>--- meta/classes/package.bbclass | 84 ++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+)meta/classes/package.bbclass +2
-0@@ -245,6 +245,8 @@ python () { deps = "" for dep in (d.getVar('PACKAGE_DEPENDS') or "").split(): deps += " %s:do_populate_sysroot" % dep + if d.getVar('PACKAGE_MINIDEBUGINFO') == '1': + deps += ' xz-native:do_populate_sysroot' d.appendVarFlag('do_package', 'depends', deps) # shlibs requires any DEPENDS to have already packaged for the*.list files
meta/classes/package.bbclass +0
-0@@ -459,6 +461,83 @@ def splitstaticdebuginfo(file, dvar, debugstaticdir,debugstaticlibdir, debugsta
return (file, sources)
+def
inject_minidebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d):meta/classes/package.bbclass +2
-0+ # Extract just the symbols from debuginfo into minidebuginfo, + # compress it with xz and inject it back into the binary in a.gnu_debugdata section.
meta/classes/package.bbclass +12
-0+ # https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html + + import subprocess + + readelf = d.getVar('READELF') + nm = d.getVar('NM') + objcopy = d.getVar('OBJCOPY') + + minidebuginfodir = d.expand('${WORKDIR}/minidebuginfo') + + src = file[len(dvar):] + dest = debuglibdir + os.path.dirname(src) + debugdir + "/" +
os.path.basename(src)+ debugappendmeta/classes/package.bbclass +5
-0+ debugfile = dvar + dest + minidebugfile = minidebuginfodir + src + '.minidebug' + bb.utils.mkdirhier(os.path.dirname(minidebugfile)) + + # If we didn't produce debuginfo for any reason, we can't produceminidebuginfo either
meta/classes/package.bbclass +3
-0+ # so skip it. + if not os.path.exists(debugfile): + bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfoinjection'.format(file))
meta/classes/package.bbclass +3
-0+ return + + # Find non-allocated PROGBITS, NOTE, and NOBITS sections in thedebuginfo.
meta/classes/package.bbclass +3
-0+ # We will exclude all of these from minidebuginfo to save space. + remove_section_names = [] + for line in subprocess.check_output([readelf, '-W', '-S', debugfile],universal_newlines=True).splitlines():
meta/classes/package.bbclass +7
-0+ fields = line.split() + if len(fields) < 8: + continue + name = fields[0] + type = fields[1] + flags = fields[7] + # .debug_ sections will be removed by objcopy -S so no need toexplicitly remove them
meta/classes/package.bbclass +6
-0+ if name.startswith('.debug_'): + continue + if 'A' not in flags and type in ['PROGBITS', 'NOTE', 'NOBITS']: + remove_section_names.append(name) + + # List dynamic symbols in the binary. We can exclude these fromminidebuginfo
meta/classes/package.bbclass +3
-0+ # because they are always present in the binary. + dynsyms = set() + for line in subprocess.check_output([nm, '-D', file,'--format=posix', '--defined-only'], universal_newlines=True).splitlines():
meta/classes/package.bbclass +3
-0+ dynsyms.add(line.split()[0]) + + # Find all function symbols from debuginfo which aren't in thedynamic symbols table.
meta/classes/package.bbclass +5
-0+ # These are the ones we want to keep in minidebuginfo. + keep_symbols_file = minidebugfile + '.symlist' + found_any_symbols = False + with open(keep_symbols_file, 'w') as f: + for line in subprocess.check_output([nm, debugfile,'--format=sysv', '--defined-only'], universal_newlines=True).splitlines():
meta/classes/package.bbclass +11
-0+ fields = line.split('|') + if len(fields) < 7: + continue + name = fields[0].strip() + type = fields[3].strip() + if type == 'FUNC' and name not in dynsyms: + f.write('{}\n'.format(name)) + found_any_symbols = True + + if not found_any_symbols: + bb.debug(1, 'ELF file {} contains no symbols, skippingminidebuginfo injection'.format(file))
meta/classes/package.bbclass +7
-0+ return + + bb.utils.remove(minidebugfile) + bb.utils.remove(minidebugfile + '.xz') + + subprocess.check_call([objcopy, '-S'] + + ['--remove-section={}'.format(s) for s in
remove_section_names] +
+ ['--keep-symbols={}'.format(keep_symbols_file),
debugfile, minidebugfile])meta/classes/package.bbclass +4
-0+ + subprocess.check_call(['xz', '--keep', minidebugfile]) + + subprocess.check_call([objcopy, '--add-section','.gnu_debugdata={}.xz'.format(minidebugfile), file])
+
defcopydebugsources(debugsrcdir, sources, d):
# The debug src information written out to sourcefile is further
processed# and copied to the destination here.meta/classes/package.bbclass +1
-0@@ -1185,6 +1264,11 @@ python split_and_strip_files () { oe.utils.multiprocess_launch(oe.package.runstrip, sfiles, d) + # Build "minidebuginfo" and reinject it back into the strippedbinaries
meta/classes/package.bbclass +2
-0+ if d.getVar('PACKAGE_MINIDEBUGINFO') == '1': + oe.utils.multiprocess_launch(inject_minidebuginfo,list(elffiles), d,
+ extraargs=(dvar, debugdir,
debuglibdir, debugappend, debugsrcdir, d))
+# # End of strip #--
2.21.1
Quoted 3 lines · Alexander Kanavin
Thanks! I'd propose this is actually enabled by default, with a possibility to disable for those situations where 5% extra is critical.
It should be a distro choice to enable it, there are
Other solutions like breakpad which does something similar and already deployed on top of OE if you are proposing that poky should enable it by default that’s a separate issue and perhaps would be ok
Quoted 174 lines
Alex
Quoted 168 lines · Dan Callaghan
"Mini debuginfo" is a special section in ELF executables containing minimal compressed debuginfo for non-exported symbols:
https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html
It lets debugging tools produce better stack traces, including local function names, without incurring the space overhead of full debuginfo. The feature was originally developed for Fedora:
https://fedoraproject.org/wiki/Features/MiniDebugInfo
but nowadays it is widely supported in the ecosystem, including in gdb and elfutils (and therefore also in tools which use elfutils, such as systemd-coredump).
This patch adds an optional extra step in package.bbclass to inject minidebuginfo while stripping and splitting out debuginfo. It can be enabled by setting PACKAGE_MINIDEBUGINFO=1. In my testing, this increases the size of resulting binaries by roughly 5%.
The code for producing and re-injecting the minidebuginfo is my own Python implementation but corresponds directly to the shell implementation that RPM uses for doing the same:
Signed-off-by Dan Callaghan <[email protected]>--- meta/classes/package.bbclass | 84 ++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+)meta/classes/package.bbclass +2
-0@@ -245,6 +245,8 @@ python () { deps = "" for dep in (d.getVar('PACKAGE_DEPENDS') or "").split(): deps += " %s:do_populate_sysroot" % dep + if d.getVar('PACKAGE_MINIDEBUGINFO') == '1': + deps += ' xz-native:do_populate_sysroot' d.appendVarFlag('do_package', 'depends', deps) # shlibs requires any DEPENDS to have already packaged for the*.list files
meta/classes/package.bbclass +0
-0@@ -459,6 +461,83 @@ def splitstaticdebuginfo(file, dvar, debugstaticdir,debugstaticlibdir, debugsta
return (file, sources)
+def
inject_minidebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d):meta/classes/package.bbclass +2
-0+ # Extract just the symbols from debuginfo into minidebuginfo, + # compress it with xz and inject it back into the binary in a.gnu_debugdata section.
meta/classes/package.bbclass +12
-0+ # https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html + + import subprocess + + readelf = d.getVar('READELF') + nm = d.getVar('NM') + objcopy = d.getVar('OBJCOPY') + + minidebuginfodir = d.expand('${WORKDIR}/minidebuginfo') + + src = file[len(dvar):] + dest = debuglibdir + os.path.dirname(src) + debugdir + "/" +
os.path.basename(src)+ debugappendmeta/classes/package.bbclass +5
-0+ debugfile = dvar + dest + minidebugfile = minidebuginfodir + src + '.minidebug' + bb.utils.mkdirhier(os.path.dirname(minidebugfile)) + + # If we didn't produce debuginfo for any reason, we can't produceminidebuginfo either
meta/classes/package.bbclass +3
-0+ # so skip it. + if not os.path.exists(debugfile): + bb.debug(1, 'ELF file {} has no debuginfo, skippingminidebuginfo injection'.format(file))
meta/classes/package.bbclass +3
-0+ return + + # Find non-allocated PROGBITS, NOTE, and NOBITS sections in thedebuginfo.
meta/classes/package.bbclass +3
-0+ # We will exclude all of these from minidebuginfo to save space. + remove_section_names = [] + for line in subprocess.check_output([readelf, '-W', '-S',debugfile], universal_newlines=True).splitlines():
meta/classes/package.bbclass +7
-0+ fields = line.split() + if len(fields) < 8: + continue + name = fields[0] + type = fields[1] + flags = fields[7] + # .debug_ sections will be removed by objcopy -S so no need toexplicitly remove them
meta/classes/package.bbclass +6
-0+ if name.startswith('.debug_'): + continue + if 'A' not in flags and type in ['PROGBITS', 'NOTE', 'NOBITS']: + remove_section_names.append(name) + + # List dynamic symbols in the binary. We can exclude these fromminidebuginfo
meta/classes/package.bbclass +3
-0+ # because they are always present in the binary. + dynsyms = set() + for line in subprocess.check_output([nm, '-D', file,'--format=posix', '--defined-only'], universal_newlines=True).splitlines():
meta/classes/package.bbclass +3
-0+ dynsyms.add(line.split()[0]) + + # Find all function symbols from debuginfo which aren't in thedynamic symbols table.
meta/classes/package.bbclass +5
-0+ # These are the ones we want to keep in minidebuginfo. + keep_symbols_file = minidebugfile + '.symlist' + found_any_symbols = False + with open(keep_symbols_file, 'w') as f: + for line in subprocess.check_output([nm, debugfile,'--format=sysv', '--defined-only'], universal_newlines=True).splitlines():
meta/classes/package.bbclass +11
-0+ fields = line.split('|') + if len(fields) < 7: + continue + name = fields[0].strip() + type = fields[3].strip() + if type == 'FUNC' and name not in dynsyms: + f.write('{}\n'.format(name)) + found_any_symbols = True + + if not found_any_symbols: + bb.debug(1, 'ELF file {} contains no symbols, skippingminidebuginfo injection'.format(file))
meta/classes/package.bbclass +7
-0+ return + + bb.utils.remove(minidebugfile) + bb.utils.remove(minidebugfile + '.xz') + + subprocess.check_call([objcopy, '-S'] + + ['--remove-section={}'.format(s) for s in
remove_section_names] +
+
['--keep-symbols={}'.format(keep_symbols_file), debugfile, minidebugfile])meta/classes/package.bbclass +4
-0+ + subprocess.check_call(['xz', '--keep', minidebugfile]) + + subprocess.check_call([objcopy, '--add-section','.gnu_debugdata={}.xz'.format(minidebugfile), file])
+
defcopydebugsources(debugsrcdir, sources, d):
# The debug src information written out to sourcefile is further
processed# and copied to the destination here.meta/classes/package.bbclass +1
-0@@ -1185,6 +1264,11 @@ python split_and_strip_files () { oe.utils.multiprocess_launch(oe.package.runstrip, sfiles, d) + # Build "minidebuginfo" and reinject it back into the strippedbinaries
meta/classes/package.bbclass +2
-0+ if d.getVar('PACKAGE_MINIDEBUGINFO') == '1': + oe.utils.multiprocess_launch(inject_minidebuginfo,list(elffiles), d,
+ extraargs=(dvar, debugdir,
debuglibdir, debugappend, debugsrcdir, d))
+# # End of strip #--
2.21.1