Skip to contentSkip to conversation

bpftool fails on 32-bit hosts when -Wa,--compress-debug-sections is used

PATCH Applied Sergei Trofimovich 5 participants 5 messages Dec 11, 2023 – Dec 20, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi bpf@!

Tl;DR:

BPF programs built with clang's -g -Wa,--compress-debug-sections fail when processed by bpftool on section alignment checks as:

    $ cat src/core/bpf/socket_bind/bpf.c
    struct socket_bind_map_t { int value; };
    struct socket_bind_map_t sd_bind_allow __attribute__((section(".maps"), used));
    $ clang-16 -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g -Wa,--compress-debug-sections
    $ bpftool gen object bpf.o bpf.unstripped.o
    libbpf: ELF section #9 has inconsistent alignment addr=8 != d=4 in bpf.unstripped.o
    Error: failed to link 'bpf.unstripped.o': Invalid argument (22)

This happens only when bpftool is a 32-bit ELF binary. 64-bit bpftool seems to work as expected.

More words:

Here is my understanding of the failure. Without the -Wa,--compress-debug-sections option clang generates byte-aligned uncompressed .debug sections:

    $ clang -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g
    $ readelf -SW bpf.unstripped.o
    There are 19 section headers, starting at offset 0x530:
    Section Headers:
      [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
      [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
      [ 1] .strtab           STRTAB          0000000000000000 000471 0000b8 00      0   0  1
      [ 2] .text             PROGBITS        0000000000000000 000040 000000 00  AX  0   0  4
      [ 3] .maps             PROGBITS        0000000000000000 000040 000004 00  WA  0   0  4
      [ 4] .debug_abbrev     PROGBITS        0000000000000000 000044 00004c 00      0   0  1
      [ 5] .debug_info       PROGBITS        0000000000000000 000090 00003d 00      0   0  1
      [ 6] .rel.debug_info   REL             0000000000000000 000380 000040 10   I 18   5  8
      [ 7] .debug_str_offsets PROGBITS        0000000000000000 0000cd 000024 00      0   0  1
      [ 8] .rel.debug_str_offsets REL             0000000000000000 0003c0 000070 10   I 18   7  8
      [ 9] .debug_str        PROGBITS        0000000000000000 0000f1 000080 01  MS  0   0  1
      [10] .debug_addr       PROGBITS        0000000000000000 000171 000010 00      0   0  1
      [11] .rel.debug_addr   REL             0000000000000000 000430 000010 10   I 18  10  8
      [12] .BTF              PROGBITS        0000000000000000 000184 000099 00      0   0  4
      [13] .rel.BTF          REL             0000000000000000 000440 000010 10   I 18  12  8
      [14] .debug_line       PROGBITS        0000000000000000 00021d 000043 00      0   0  1
      [15] .rel.debug_line   REL             0000000000000000 000450 000020 10   I 18  14  8
      [16] .debug_line_str   PROGBITS        0000000000000000 000260 000041 01  MS  0   0  1
      [17] .llvm_addrsig     LOOS+0xfff4c03  0000000000000000 000470 000001 00   E 18   0  1
      [18] .symtab           SYMTAB          0000000000000000 0002a8 0000d8 18      1   8  8
    Key to Flags:
      W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
      L (link order), O (extra OS processing required), G (group), T (TLS),
      C (compressed), x (unknown), o (OS specific), E (exclude),
      D (mbind), p (processor specific)

Note how .debug_str has alignment=1 here. Loading of such a section has no problems on x86_64 or i686.

But when we add -Wa,--compress-debug-sections some debug sections become aligned 8 bytes (usual ELF64) alignment:

    $ clang -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g -Wa,--compress-debug-sections
    $ LANG=C readelf -SW bpf.unstripped.o
    There are 19 section headers, starting at offset 0x530:
    Section Headers:
      [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
      [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
      [ 1] .strtab           STRTAB          0000000000000000 000471 0000b8 00      0   0  1
      [ 2] .text             PROGBITS        0000000000000000 000040 000000 00  AX  0   0  4
      [ 3] .maps             PROGBITS        0000000000000000 000040 000004 00  WA  0   0  4
      [ 4] .debug_abbrev     PROGBITS        0000000000000000 000044 00004c 00      0   0  1
      [ 5] .debug_info       PROGBITS        0000000000000000 000090 00003d 00      0   0  1
      [ 6] .rel.debug_info   REL             0000000000000000 000380 000040 10   I 18   5  8
      [ 7] .debug_str_offsets PROGBITS        0000000000000000 0000cd 000024 00      0   0  1
      [ 8] .rel.debug_str_offsets REL             0000000000000000 0003c0 000070 10   I 18   7  8
      [ 9] .debug_str        PROGBITS        0000000000000000 0000f1 000086 01 MSC  0   0  8
      [10] .debug_addr       PROGBITS        0000000000000000 000177 000010 00      0   0  1
      [11] .rel.debug_addr   REL             0000000000000000 000430 000010 10   I 18  10  8
      [12] .BTF              PROGBITS        0000000000000000 000188 000099 00      0   0  4
      [13] .rel.BTF          REL             0000000000000000 000440 000010 10   I 18  12  8
      [14] .debug_line       PROGBITS        0000000000000000 000221 000043 00      0   0  1
      [15] .rel.debug_line   REL             0000000000000000 000450 000020 10   I 18  14  8
      [16] .debug_line_str   PROGBITS        0000000000000000 000264 000041 01  MS  0   0  1
      [17] .llvm_addrsig     LOOS+0xfff4c03  0000000000000000 000470 000001 00   E 18   0  1
      [18] .symtab           SYMTAB          0000000000000000 0002a8 0000d8 18      1   8  8
    Key to Flags:
      W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
      L (link order), O (extra OS processing required), G (group), T (TLS),
      C (compressed), x (unknown), o (OS specific), E (exclude),
      D (mbind), p (processor specific)

Note how .debug_str alignment changed from 1-byte to 8-byte alignment.

AFAIU 8-byte alignment of compressed section for ELF64 files is expected:
compressed sections all have a small header that consistes of a few 64-bit values (compression type, uncompressed size, uncompressed data alignment).

Thus the binary loads as expected by x86_64 bpftool and fails when loaded by i686 bpftool.

Should bpftool work in this scenario? Or compressed sections are not supported on 32-bit hosts?

It feels like debugging sections with strings should be easily decompressable on any host type.

Thanks!

Signature · 1 line
  Sergei
#1 lore raw
Quoted 8 lines · Sergei Trofimovich

Hi bpf@!

Tl;DR:

BPF programs built with clang's -g -Wa,--compress-debug-sections fail when processed by bpftool on section alignment checks as:

Quoted 7 lines
    $ cat src/core/bpf/socket_bind/bpf.c
    struct socket_bind_map_t { int value; };
    struct socket_bind_map_t sd_bind_allow __attribute__((section(".maps"), used));
    $ clang-16 -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g -Wa,--compress-debug-sections
    $ bpftool gen object bpf.o bpf.unstripped.o
    libbpf: ELF section #9 has inconsistent alignment addr=8 != d=4 in bpf.unstripped.o

So we are talking about this check in libbpf's linker:

if (sec->shdr->sh_addralign != sec->data->d_align)

And it's confusing that it would be ok to have actual data alignment that doesn't match declared alignment in section header? Is that expected in ELF in general?

Having said that, I think we can mitigate all this easily by ignoring DWARF sections. Libbpf linker drops all that while linking, see is_ignored_sec() check, but when doing a sanity check we still validate everything for DWARF sections.

So perhaps an easy way out is just to ignore DWARF sections in sanity checks as well?

Quoted 101 lines

Error: failed to link 'bpf.unstripped.o': Invalid argument (22)

This happens only when bpftool is a 32-bit ELF binary. 64-bit bpftool seems to work as expected.

More words:

Here is my understanding of the failure. Without the -Wa,--compress-debug-sections option clang generates byte-aligned uncompressed .debug sections:

    $ clang -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g
    $ readelf -SW bpf.unstripped.o
    There are 19 section headers, starting at offset 0x530:
    Section Headers:
      [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
      [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
      [ 1] .strtab           STRTAB          0000000000000000 000471 0000b8 00      0   0  1
      [ 2] .text             PROGBITS        0000000000000000 000040 000000 00  AX  0   0  4
      [ 3] .maps             PROGBITS        0000000000000000 000040 000004 00  WA  0   0  4
      [ 4] .debug_abbrev     PROGBITS        0000000000000000 000044 00004c 00      0   0  1
      [ 5] .debug_info       PROGBITS        0000000000000000 000090 00003d 00      0   0  1
      [ 6] .rel.debug_info   REL             0000000000000000 000380 000040 10   I 18   5  8
      [ 7] .debug_str_offsets PROGBITS        0000000000000000 0000cd 000024 00      0   0  1
      [ 8] .rel.debug_str_offsets REL             0000000000000000 0003c0 000070 10   I 18   7  8
      [ 9] .debug_str        PROGBITS        0000000000000000 0000f1 000080 01  MS  0   0  1
      [10] .debug_addr       PROGBITS        0000000000000000 000171 000010 00      0   0  1
      [11] .rel.debug_addr   REL             0000000000000000 000430 000010 10   I 18  10  8
      [12] .BTF              PROGBITS        0000000000000000 000184 000099 00      0   0  4
      [13] .rel.BTF          REL             0000000000000000 000440 000010 10   I 18  12  8
      [14] .debug_line       PROGBITS        0000000000000000 00021d 000043 00      0   0  1
      [15] .rel.debug_line   REL             0000000000000000 000450 000020 10   I 18  14  8
      [16] .debug_line_str   PROGBITS        0000000000000000 000260 000041 01  MS  0   0  1
      [17] .llvm_addrsig     LOOS+0xfff4c03  0000000000000000 000470 000001 00   E 18   0  1
      [18] .symtab           SYMTAB          0000000000000000 0002a8 0000d8 18      1   8  8
    Key to Flags:
      W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
      L (link order), O (extra OS processing required), G (group), T (TLS),
      C (compressed), x (unknown), o (OS specific), E (exclude),
      D (mbind), p (processor specific)

Note how .debug_str has alignment=1 here. Loading of such a section has no problems on x86_64 or i686.

But when we add -Wa,--compress-debug-sections some debug sections become aligned 8 bytes (usual ELF64) alignment:

    $ clang -target bpf -fno-stack-protector -c src/core/bpf/socket_bind/bpf.c -o bpf.unstripped.o -g -Wa,--compress-debug-sections
    $ LANG=C readelf -SW bpf.unstripped.o
    There are 19 section headers, starting at offset 0x530:
    Section Headers:
      [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
      [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
      [ 1] .strtab           STRTAB          0000000000000000 000471 0000b8 00      0   0  1
      [ 2] .text             PROGBITS        0000000000000000 000040 000000 00  AX  0   0  4
      [ 3] .maps             PROGBITS        0000000000000000 000040 000004 00  WA  0   0  4
      [ 4] .debug_abbrev     PROGBITS        0000000000000000 000044 00004c 00      0   0  1
      [ 5] .debug_info       PROGBITS        0000000000000000 000090 00003d 00      0   0  1
      [ 6] .rel.debug_info   REL             0000000000000000 000380 000040 10   I 18   5  8
      [ 7] .debug_str_offsets PROGBITS        0000000000000000 0000cd 000024 00      0   0  1
      [ 8] .rel.debug_str_offsets REL             0000000000000000 0003c0 000070 10   I 18   7  8
      [ 9] .debug_str        PROGBITS        0000000000000000 0000f1 000086 01 MSC  0   0  8
      [10] .debug_addr       PROGBITS        0000000000000000 000177 000010 00      0   0  1
      [11] .rel.debug_addr   REL             0000000000000000 000430 000010 10   I 18  10  8
      [12] .BTF              PROGBITS        0000000000000000 000188 000099 00      0   0  4
      [13] .rel.BTF          REL             0000000000000000 000440 000010 10   I 18  12  8
      [14] .debug_line       PROGBITS        0000000000000000 000221 000043 00      0   0  1
      [15] .rel.debug_line   REL             0000000000000000 000450 000020 10   I 18  14  8
      [16] .debug_line_str   PROGBITS        0000000000000000 000264 000041 01  MS  0   0  1
      [17] .llvm_addrsig     LOOS+0xfff4c03  0000000000000000 000470 000001 00   E 18   0  1
      [18] .symtab           SYMTAB          0000000000000000 0002a8 0000d8 18      1   8  8
    Key to Flags:
      W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
      L (link order), O (extra OS processing required), G (group), T (TLS),
      C (compressed), x (unknown), o (OS specific), E (exclude),
      D (mbind), p (processor specific)

Note how .debug_str alignment changed from 1-byte to 8-byte alignment.

AFAIU 8-byte alignment of compressed section for ELF64 files is expected:
compressed sections all have a small header that consistes of a few 64-bit values (compression type, uncompressed size, uncompressed data alignment).

Thus the binary loads as expected by x86_64 bpftool and fails when loaded by i686 bpftool.

Should bpftool work in this scenario? Or compressed sections are not supported on 32-bit hosts?

It feels like debugging sections with strings should be easily decompressable on any host type.

Thanks!

--

Sergei

#2 lore raw

PATCH 1/1 libbpf: skip DWARF sections in linker sanity check

clang can generate (with -g -Wa,--compress-debug-sections) 4-byte aligned DWARF sections that declare themselves to be 8-byte aligned in the section header. Since DWARF sections are dropped during linking anyway, just skip running the sanity checks on them.

---
 tools/lib/bpf/linker.c | 3 +++
 1 file changed, 3 insertions(+)
tools/lib/bpf/linker.c +3 -0
index 52a2901e8bd0..16bca56002ab 100644
@@ -719,6 +719,9 @@ static int linker_sanity_check_elf(struct src_obj *obj)
 			return -EINVAL;
 		}
 
+		if (is_dwarf_sec_name(sec->sec_name))
+			continue;
+
 		if (sec->shdr->sh_addralign && !is_pow_of_2(sec->shdr->sh_addralign)) {
 			pr_warn("ELF section #%zu alignment %llu is non pow-of-2 alignment in %s\n",
 				sec->sec_idx, (long long unsigned)sec->shdr->sh_addralign,

base-commit: 733763285acfe8dffd6e39ad2ed3d1222b32a901

Signature · 1 line
2.42.0
maintainer Acked-by on v1 1/1
#3 lore raw

Re: [PATCH] libbpf: skip DWARF sections in linker sanity check

Quoted 9 lines · Alyssa Ross

clang can generate (with -g -Wa,--compress-debug-sections) 4-byte aligned DWARF sections that declare themselves to be 8-byte aligned in the section header. Since DWARF sections are dropped during linking anyway, just skip running the sanity checks on them.

Looks reasonable to skip, thus:

#4 lore raw

Re: [PATCH] libbpf: skip DWARF sections in linker sanity check

In reply to Alyssa Ross
This patch was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:

Hello:

This patch was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:

Quoted 11 lines · On Tue, 19 Dec 2023 12:03:24 +0100 you wrote:

clang can generate (with -g -Wa,--compress-debug-sections) 4-byte aligned DWARF sections that declare themselves to be 8-byte aligned in the section header. Since DWARF sections are dropped during linking anyway, just skip running the sanity checks on them.

[...]

Here is the summary with links:
- libbpf: skip DWARF sections in linker sanity check

    https://git.kernel.org/bpf/bpf-next/c/a4897b87775c

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html