bpftool fails on 32-bit hosts when -Wa,--compress-debug-sections is used
Hi bpf@!
Tl;DR:
BPF programs built with clang's -g -Wa,--compress-debug-sections fail when processed by bpftool on section alignment checks as:
$ cat src/core/bpf/socket_bind/bpf.c
struct socket_bind_map_t { int value; };
struct socket_bind_map_t sd_bind_allow __attribute__((section(".maps"), used)); $ clang-16 -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g -Wa,--compress-debug-sections
$ bpftool gen object bpf.o bpf.unstripped.o
libbpf: ELF section #9 has inconsistent alignment addr=8 != d=4 in bpf.unstripped.o
Error: failed to link 'bpf.unstripped.o': Invalid argument (22)This happens only when bpftool is a 32-bit ELF binary. 64-bit bpftool seems to work as expected.
More words:
Here is my understanding of the failure. Without the -Wa,--compress-debug-sections option clang generates byte-aligned uncompressed .debug sections:
$ clang -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g
$ readelf -SW bpf.unstripped.o
There are 19 section headers, starting at offset 0x530: Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
[ 0] NULL 0000000000000000 000000 000000 00 0 0 0
[ 1] .strtab STRTAB 0000000000000000 000471 0000b8 00 0 0 1
[ 2] .text PROGBITS 0000000000000000 000040 000000 00 AX 0 0 4
[ 3] .maps PROGBITS 0000000000000000 000040 000004 00 WA 0 0 4
[ 4] .debug_abbrev PROGBITS 0000000000000000 000044 00004c 00 0 0 1
[ 5] .debug_info PROGBITS 0000000000000000 000090 00003d 00 0 0 1
[ 6] .rel.debug_info REL 0000000000000000 000380 000040 10 I 18 5 8
[ 7] .debug_str_offsets PROGBITS 0000000000000000 0000cd 000024 00 0 0 1
[ 8] .rel.debug_str_offsets REL 0000000000000000 0003c0 000070 10 I 18 7 8
[ 9] .debug_str PROGBITS 0000000000000000 0000f1 000080 01 MS 0 0 1
[10] .debug_addr PROGBITS 0000000000000000 000171 000010 00 0 0 1
[11] .rel.debug_addr REL 0000000000000000 000430 000010 10 I 18 10 8
[12] .BTF PROGBITS 0000000000000000 000184 000099 00 0 0 4
[13] .rel.BTF REL 0000000000000000 000440 000010 10 I 18 12 8
[14] .debug_line PROGBITS 0000000000000000 00021d 000043 00 0 0 1
[15] .rel.debug_line REL 0000000000000000 000450 000020 10 I 18 14 8
[16] .debug_line_str PROGBITS 0000000000000000 000260 000041 01 MS 0 0 1
[17] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 000470 000001 00 E 18 0 1
[18] .symtab SYMTAB 0000000000000000 0002a8 0000d8 18 1 8 8
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
D (mbind), p (processor specific)Note how .debug_str has alignment=1 here. Loading of such a section has no problems on x86_64 or i686.
But when we add -Wa,--compress-debug-sections some debug sections become aligned 8 bytes (usual ELF64) alignment:
$ clang -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g -Wa,--compress-debug-sections
$ LANG=C readelf -SW bpf.unstripped.o
There are 19 section headers, starting at offset 0x530: Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
[ 0] NULL 0000000000000000 000000 000000 00 0 0 0
[ 1] .strtab STRTAB 0000000000000000 000471 0000b8 00 0 0 1
[ 2] .text PROGBITS 0000000000000000 000040 000000 00 AX 0 0 4
[ 3] .maps PROGBITS 0000000000000000 000040 000004 00 WA 0 0 4
[ 4] .debug_abbrev PROGBITS 0000000000000000 000044 00004c 00 0 0 1
[ 5] .debug_info PROGBITS 0000000000000000 000090 00003d 00 0 0 1
[ 6] .rel.debug_info REL 0000000000000000 000380 000040 10 I 18 5 8
[ 7] .debug_str_offsets PROGBITS 0000000000000000 0000cd 000024 00 0 0 1
[ 8] .rel.debug_str_offsets REL 0000000000000000 0003c0 000070 10 I 18 7 8
[ 9] .debug_str PROGBITS 0000000000000000 0000f1 000086 01 MSC 0 0 8
[10] .debug_addr PROGBITS 0000000000000000 000177 000010 00 0 0 1
[11] .rel.debug_addr REL 0000000000000000 000430 000010 10 I 18 10 8
[12] .BTF PROGBITS 0000000000000000 000188 000099 00 0 0 4
[13] .rel.BTF REL 0000000000000000 000440 000010 10 I 18 12 8
[14] .debug_line PROGBITS 0000000000000000 000221 000043 00 0 0 1
[15] .rel.debug_line REL 0000000000000000 000450 000020 10 I 18 14 8
[16] .debug_line_str PROGBITS 0000000000000000 000264 000041 01 MS 0 0 1
[17] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 000470 000001 00 E 18 0 1
[18] .symtab SYMTAB 0000000000000000 0002a8 0000d8 18 1 8 8
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
D (mbind), p (processor specific)Note how .debug_str alignment changed from 1-byte to 8-byte alignment.
AFAIU 8-byte alignment of compressed section for ELF64 files is expected:
compressed sections all have a small header that consistes of a few 64-bit values (compression type, uncompressed size, uncompressed data alignment).
Thus the binary loads as expected by x86_64 bpftool and fails when loaded by i686 bpftool.
Should bpftool work in this scenario? Or compressed sections are not supported on 32-bit hosts?
It feels like debugging sections with strings should be easily decompressable on any host type.
Thanks!
Signature · 1 line
Sergei
Quoted 8 lines · Sergei Trofimovich
Hi bpf@!
Tl;DR:
BPF programs built with
clang's-g -Wa,--compress-debug-sectionsfail when processed bybpftoolon section alignment checks as:
Quoted 7 lines
$ cat src/core/bpf/socket_bind/bpf.c struct socket_bind_map_t { int value; }; struct socket_bind_map_t sd_bind_allow __attribute__((section(".maps"), used));$ clang-16 -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g -Wa,--compress-debug-sections $ bpftool gen object bpf.o bpf.unstripped.o libbpf: ELF section #9 has inconsistent alignment addr=8 != d=4 in bpf.unstripped.o
So we are talking about this check in libbpf's linker:
if (sec->shdr->sh_addralign != sec->data->d_align)
And it's confusing that it would be ok to have actual data alignment that doesn't match declared alignment in section header? Is that expected in ELF in general?
Having said that, I think we can mitigate all this easily by ignoring DWARF sections. Libbpf linker drops all that while linking, see is_ignored_sec() check, but when doing a sanity check we still validate everything for DWARF sections.
So perhaps an easy way out is just to ignore DWARF sections in sanity checks as well?
Quoted 101 lines
Error: failed to link 'bpf.unstripped.o': Invalid argument (22)
This happens only when
bpftoolis a 32-bit ELF binary. 64-bitbpftoolseems to work as expected.More words:
Here is my understanding of the failure. Without the
-Wa,--compress-debug-sectionsoptionclanggenerates byte-aligned uncompressed.debugsections:$ clang -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g $ readelf -SW bpf.unstripped.o There are 19 section headers, starting at offset 0x530:Section Headers: [Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 [ 1] .strtab STRTAB 0000000000000000 000471 0000b8 00 0 0 1 [ 2] .text PROGBITS 0000000000000000 000040 000000 00 AX 0 0 4 [ 3] .maps PROGBITS 0000000000000000 000040 000004 00 WA 0 0 4 [ 4] .debug_abbrev PROGBITS 0000000000000000 000044 00004c 00 0 0 1 [ 5] .debug_info PROGBITS 0000000000000000 000090 00003d 00 0 0 1 [ 6] .rel.debug_info REL 0000000000000000 000380 000040 10 I 18 5 8 [ 7] .debug_str_offsets PROGBITS 0000000000000000 0000cd 000024 00 0 0 1 [ 8] .rel.debug_str_offsets REL 0000000000000000 0003c0 000070 10 I 18 7 8 [ 9] .debug_str PROGBITS 0000000000000000 0000f1 000080 01 MS 0 0 1 [10] .debug_addr PROGBITS 0000000000000000 000171 000010 00 0 0 1 [11] .rel.debug_addr REL 0000000000000000 000430 000010 10 I 18 10 8 [12] .BTF PROGBITS 0000000000000000 000184 000099 00 0 0 4 [13] .rel.BTF REL 0000000000000000 000440 000010 10 I 18 12 8 [14] .debug_line PROGBITS 0000000000000000 00021d 000043 00 0 0 1 [15] .rel.debug_line REL 0000000000000000 000450 000020 10 I 18 14 8 [16] .debug_line_str PROGBITS 0000000000000000 000260 000041 01 MS 0 0 1 [17] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 000470 000001 00 E 18 0 1 [18] .symtab SYMTAB 0000000000000000 0002a8 0000d8 18 1 8 8 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), D (mbind), p (processor specific)Note how
.debug_strhasalignment=1here. Loading of such a section has no problems onx86_64or i686.But when we add
-Wa,--compress-debug-sectionssome debug sections become aligned 8 bytes (usualELF64) alignment:$ clang -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g -Wa,--compress-debug-sections $ LANG=C readelf -SW bpf.unstripped.o There are 19 section headers, starting at offset 0x530:Section Headers: [Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 [ 1] .strtab STRTAB 0000000000000000 000471 0000b8 00 0 0 1 [ 2] .text PROGBITS 0000000000000000 000040 000000 00 AX 0 0 4 [ 3] .maps PROGBITS 0000000000000000 000040 000004 00 WA 0 0 4 [ 4] .debug_abbrev PROGBITS 0000000000000000 000044 00004c 00 0 0 1 [ 5] .debug_info PROGBITS 0000000000000000 000090 00003d 00 0 0 1 [ 6] .rel.debug_info REL 0000000000000000 000380 000040 10 I 18 5 8 [ 7] .debug_str_offsets PROGBITS 0000000000000000 0000cd 000024 00 0 0 1 [ 8] .rel.debug_str_offsets REL 0000000000000000 0003c0 000070 10 I 18 7 8 [ 9] .debug_str PROGBITS 0000000000000000 0000f1 000086 01 MSC 0 0 8 [10] .debug_addr PROGBITS 0000000000000000 000177 000010 00 0 0 1 [11] .rel.debug_addr REL 0000000000000000 000430 000010 10 I 18 10 8 [12] .BTF PROGBITS 0000000000000000 000188 000099 00 0 0 4 [13] .rel.BTF REL 0000000000000000 000440 000010 10 I 18 12 8 [14] .debug_line PROGBITS 0000000000000000 000221 000043 00 0 0 1 [15] .rel.debug_line REL 0000000000000000 000450 000020 10 I 18 14 8 [16] .debug_line_str PROGBITS 0000000000000000 000264 000041 01 MS 0 0 1 [17] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 000470 000001 00 E 18 0 1 [18] .symtab SYMTAB 0000000000000000 0002a8 0000d8 18 1 8 8 Key to Flags: W (write), A (alloc), X (execute), M (merge), S (strings), I (info), L (link order), O (extra OS processing required), G (group), T (TLS), C (compressed), x (unknown), o (OS specific), E (exclude), D (mbind), p (processor specific)Note how
.debug_stralignment changed from 1-byte to 8-byte alignment.AFAIU 8-byte alignment of compressed section for
ELF64files is expected:
compressed sections all have a small header that consistes of a few 64-bit values (compression type, uncompressed size, uncompressed data alignment).Thus the binary loads as expected by
x86_64bpftooland fails when loaded byi686bpftool.Should
bpftoolwork in this scenario? Or compressed sections are not supported on 32-bit hosts?It feels like debugging sections with strings should be easily decompressable on any host type.
Thanks!
--
Sergei
PATCH 1/1 libbpf: skip DWARF sections in linker sanity check
clang can generate (with -g -Wa,--compress-debug-sections) 4-byte aligned DWARF sections that declare themselves to be 8-byte aligned in the section header. Since DWARF sections are dropped during linking anyway, just skip running the sanity checks on them.
--- tools/lib/bpf/linker.c | 3 +++ 1 file changed, 3 insertions(+)
tools/lib/bpf/linker.c +3 -0
@@ -719,6 +719,9 @@ static int linker_sanity_check_elf(struct src_obj *obj)return -EINVAL ; } +if (is_dwarf_sec_name (sec->sec_name)) +continue ; +if (sec->shdr->sh_addralign && !is_pow_of_2 (sec->shdr->sh_addralign)) { pr_warn("ELF section #%zu alignment %llu is non pow-of-2 alignment in %s\n", sec->sec_idx, (long long unsigned)sec->shdr->sh_addralign,
base-commit: 733763285acfe8dffd6e39ad2ed3d1222b32a901
Signature · 1 line
2.42.0
Re: [PATCH] libbpf: skip DWARF sections in linker sanity check
Quoted 9 lines · Alyssa Ross
clang can generate (with -g -Wa,--compress-debug-sections) 4-byte aligned DWARF sections that declare themselves to be 8-byte aligned in the section header. Since DWARF sections are dropped during linking anyway, just skip running the sanity checks on them.
Reported-by Sergei Trofimovich <[email protected]>Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Alyssa Ross <[email protected]>
Looks reasonable to skip, thus:
Re: [PATCH] libbpf: skip DWARF sections in linker sanity check
In reply to Alyssa RossThis patch was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Hello:
This patch was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Quoted 11 lines · On Tue, 19 Dec 2023 12:03:24 +0100 you wrote:
clang can generate (with -g -Wa,--compress-debug-sections) 4-byte aligned DWARF sections that declare themselves to be 8-byte aligned in the section header. Since DWARF sections are dropped during linking anyway, just skip running the sanity checks on them.
Reported-by Sergei Trofimovich <[email protected]>Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Alyssa Ross <[email protected]>[...]
Here is the summary with links:
- libbpf: skip DWARF sections in linker sanity check
https://git.kernel.org/bpf/bpf-next/c/a4897b87775c
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html