package.py: warn if target is not a valid ELF file in dwarfsrcfiles
PATCH 1/1 package.py: warn if target is not a valid ELF file in dwarfsrcfiles
From: "mark.yang" <[email protected]>
When using the Clang toolchain with LTO, intermediate .o files are generated as LLVM bitcode files instead of ELF files. While .so libraries are ultimately ELF, .a files are not. In this case, dwarfsrcfiles always fails.
Adding the -ffat-lto-objects option allows the Clang toolchain to produce .o files that are recognized as ELF, but this is not always a suitable solution as it causes issues when used in conjunction with -fsanitize=cfi.
Therefore, we need to handle cases where dwarfsrcfiles encounters a file that is not a valid ELF. Change the behavior from a fatal error to a warning to notify the user.
--- meta/lib/oe/package.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
meta/lib/oe/package.py +6 -2
@@ -782,8 +782,12 @@ def source_info(file, d, fatal=True): if retval != 0 and retval != 255: msg = "dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else "") if fatal: - bb.fatal(msg) - bb.note(msg) + if "not a valid ELF file" in output: + bb.warn(msg) + else: + bb.fatal(msg) + else: + bb.note(msg) debugsources = parse_debugsources_from_dwarfsrcfiles_output(output)
Can you please provide a way to reproduce and observe the issue? I'm somewhat confused as to what the original issue is: is dwarfsrcfiles called on an .a file? That is not an ELF file regardless of whether clang-with-lto is enabled, so why does it work in one case but not in the other?
Alex
Quoted 48 lines · mark.yang via lists.openembedded.org
From: "mark.yang" <[email protected]>
When using the Clang toolchain with LTO, intermediate .o files are generated as LLVM bitcode files instead of ELF files. While .so libraries are ultimately ELF, .a files are not. In this case, dwarfsrcfiles always fails.
Adding the -ffat-lto-objects option allows the Clang toolchain to produce .o files that are recognized as ELF, but this is not always a suitable solution as it causes issues when used in conjunction with -fsanitize=cfi.
Therefore, we need to handle cases where dwarfsrcfiles encounters a file that is not a valid ELF. Change the behavior from a fatal error to a warning to notify the user.
Signed-off-by mark.yang <[email protected]>--- meta/lib/oe/package.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)meta/lib/oe/package.py +6
-3@@ -782,8 +782,12 @@ def source_info(file, d, fatal=True): if retval != 0 and retval != 255: msg = "dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else "") if fatal: - bb.fatal(msg) - bb.note(msg) + if "not a valid ELF file" in output: + bb.warn(msg) + else: + bb.fatal(msg) + else: + bb.note(msg) debugsources = parse_debugsources_from_dwarfsrcfiles_output(output) -=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.
View/Reply Online (#228547): https://lists.openembedded.org/g/openembedded-core/message/228547
Mute This Topic: https://lists.openembedded.org/mt/116947501/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
My error report is as follows.
https://errors.yoctoproject.org/Errors/Details/892988/
I am using only the openembedded-core layer, with nodistro. local.conf
Quoted 13 lines
PREFERRED_TOOLCHAIN_TARGET= "clang"
DISTRO_FEATURES:append = " ld-is-lld"
require conf/distro/include/lto.inc
DISTRO_FEATURES:append = " lto"
# The settings below were taken from meta-clang. LTO:toolchain-clang:class-target = "${@bb.utils.contains('DISTRO_FEATURES', 'thin-lto', '-flto=thin', '-flto -fuse-ld=lld', d)}"
LTO:toolchain-clang:class-nativesdk = "${@bb.utils.contains('DISTRO_FEATURES', 'thin-lto', '-flto=thin', '-flto -fuse-ld=lld', d)}"
dwarfsrcfiles checks .a files during the package process.
Whether using LTO or not, DWARF information is extracted by iterating through .o files inside .a files. However, they must be ELF objects.
To summarize, it is as follows.
The problem is that when using clang + LTO, the .o files are not ELF but LLVM IR bitcode.
Quoted 62 lines · Alexander Kanavin
Can you please provide a way to reproduce and observe the issue? I'm somewhat confused as to what the original issue is: is dwarfsrcfiles called on an .a file? That is not an ELF file regardless of whether clang-with-lto is enabled, so why does it work in one case but not in the other?
Alex
Quoted 49 lines · mark.yang via lists.openembedded.org
From: "mark.yang" <[email protected]>
When using the Clang toolchain with LTO, intermediate .o files are generated as
LLVM bitcode files instead of ELF files. While .so libraries are ultimately
ELF, .a files are not. In this case, dwarfsrcfiles always fails.Adding the -ffat-lto-objects option allows the Clang toolchain to produce .o
files that are recognized as ELF, but this is not always a suitable solution
as it causes issues when used in conjunction with -fsanitize=cfi.Therefore, we need to handle cases where dwarfsrcfiles encounters a file that
is not a valid ELF. Change the behavior from a fatal error to a warning tonotify the user.
Signed-off-by mark.yang <[email protected]>---
meta/lib/oe/package.py| 8 ++++++--1 file changed, 6 insertions(+), 2 deletions(-)meta/lib/oe/package.py +0
-0@@ -782,8 +782,12 @@ def source_info(file, d, fatal=True):if retval != 0 and retval != 255:
msg = "dwarfsrcfiles failed with exit code %s (cmd was %s)%s" % (retval, cmd, ":\n%s" % output if output else "")
if fatal:
-bb.fatal(msg)
-bb.note(msg)
+ if "not a valid ELF file" in output:
+bb.warn(msg)
+ else:
+bb.fatal(msg)
+ else:
+bb.note(msg)debugsources =
parse_debugsources_from_dwarfsrcfiles_output(output)
Quoted 16 lines · mark.yang via lists.openembedded.org
My error report is as follows.
https://errors.yoctoproject.org/Errors/Details/892988/I am using only the openembedded-core layer, with nodistro. local.conf
PREFERRED_TOOLCHAIN_TARGET= "clang"
DISTRO_FEATURES:append = " ld-is-lld"
require conf/distro/include/lto.inc
DISTRO_FEATURES:append = " lto"
# The settings below were taken from meta-clang.
LTO:toolchain-clang:class-target = "${@bb.utils.contains('DISTRO_FEATURES', 'thin-lto', '-flto=thin', '-flto -fuse-ld=lld', d)}" LTO:toolchain-clang:class-nativesdk = "${@bb.utils.contains('DISTRO_FEATURES', 'thin-lto', '-flto=thin', '-flto -fuse-ld=lld', d)}"
Quoted 6 lines
dwarfsrcfiles checks .a files during the package process.
Whether using LTO or not, DWARF information is extracted by iterating through .o files inside .a files. However, they must be ELF objects.To summarize, it is as follows.
The problem is that when using clang + LTO, the .o files are not ELF but LLVM IR bitcode.
Thanks for the reproduction instructions and the error message.
The fix is somewhat problematic:
- it relies on the particular console output from dwarfsrcfiles which may change over time (maybe subtly), and then the code around it will quietly cease to work;
- it obscures other possible issues that would be exposed by the same error from dwarfsrcfiles: there can be many reasons for something not being a valid elf file, and they should remain fatal errors;
- conversely for this particular use case it still emits warnings, even though there's nothing the user can do about it - should it rather say nothing if .a has only llvm-ir files?
Is it possible to rather guard the call to dwarfsrcfiles with a set of
conditions: inspecting .a file, clang as toolchain and lto in distro_features, and skip the call if all conditions are true?
Alex
Thank you for reviewing this, Alex.
I have incorporated it into
https://lists.openembedded.org/g/openembedded-core/topic/patch_package_py_skip/116968059
and uploaded a new patch.