Problem with BTF generation on mips64el

PATCH Superseded Tony Ambardar 10 participants 40 messages May 31, 2024 – Jun 26, 2024

Subsystems (3)

Stable branch BPF Compiler attributes

lore

Share & tools

lore

Tony Ambardar

author

2024-05-31 01:30 UTC #0 lore raw

Hello,

For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.

Some relate to resolve_btfids:

Quoted 10 lines

LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr

Quoted 8 lines

WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32

These do not appear to be #ifdef-related and have similar past reports [1].

I also see many pahole failures during BTF encoding of modules, such as:

Quoted 21 lines

  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
LD [M] lib/test_bpf.ko
BTF [M] lib/test_bpf.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
LD [M] lib/crc-ccitt.ko
BTF [M] lib/crc-ccitt.ko
die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
die__process_unit: tag not supported 0x11 (compile_unit)!
die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
LD [M] lib/libcrc32c.ko
BTF [M] lib/libcrc32c.ko
die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
die__process_unit: tag not supported 0x11 (compile_unit)!
die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!

Quoted 8 lines

LD [M] lib/ts_kmp.ko
BTF [M] lib/ts_kmp.ko
LD [M] lib/ts_bm.ko
BTF [M] lib/ts_bm.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!

I have seen reports of various similar "die__" messages on the dwarves list and repo, with the hint of an elfutils connection [2] but nothing conclusive.

Details of the git commit and build environment are as follows:

Quoted 29 lines

$ git log -1 --oneline  bpf/master
9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
test_sk_storage_tracing.c

$ lsb_release -a
Description:    Ubuntu 22.04.4 LTS

$ cat gcc-compile.txt
ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6

$ mips64el-linux-gnuabi64-gcc --version
mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0

$ mips64el-linux-gnuabi64-ld --version
GNU ld (GNU Binutils for Ubuntu) 2.38

$ pahole --version
v1.26

$ ldd $(which pahole)
        linux-vdso.so.1 (0x00007fff16f3f000)
        libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
        libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)

Quoted 6 lines

$ dpkg -s elfutils
Package: elfutils
...
Version: 0.186-1build1
Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)

For reference, I also attached the full .config and build log from the above.

I should add this is not only a problem with the latest bpf/master but also appears to affect the 6.6.x LTS kernel, which I tested while building a mips64el OpenWrt distro image. That build environment employs the latest gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.

Not only do I see similar warnings from resolve_btfids and pahole, but while running the distro image I encounter module loading failures that suggest ELF corruption in some module .ko files, based on the following:

Quoted 5 lines

root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
...
init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
...

Quoted 7 lines

$ man init_module
...
The following errors may additionally occur for init_module():
...
     EINVAL param_values is invalid, or some part of the ELF image in
     module_image contains inconsistencies.
...

I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.

Thanks,
Tony Ambardar

Link https://lore.kernel.org/all/[email protected]/ [1] open here

Link https://github.com/acmel/dwarves/issues/45 [2]

Hengqi Chen

2024-05-31 02:18 UTC #1 lore raw

Hi Tony,

Quoted 28 lines · Tony Ambardar

Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
Some relate to resolve_btfids:
Quoted 10 lines
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
Quoted 8 lines
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32

Quoted 11 lines

These do not appear to be #ifdef-related and have similar past reports [1].
I also see many pahole failures during BTF encoding of modules, such as:
Quoted 7 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!

The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.

Quoted 90 lines

Quoted 14 lines
LD [M] lib/test_bpf.ko
BTF [M] lib/test_bpf.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
LD [M] lib/crc-ccitt.ko
BTF [M] lib/crc-ccitt.ko
die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
die__process_unit: tag not supported 0x11 (compile_unit)!
die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
LD [M] lib/libcrc32c.ko
BTF [M] lib/libcrc32c.ko
die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
die__process_unit: tag not supported 0x11 (compile_unit)!
die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
Quoted 8 lines
LD [M] lib/ts_kmp.ko
BTF [M] lib/ts_kmp.ko
LD [M] lib/ts_bm.ko
BTF [M] lib/ts_bm.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
I have seen reports of various similar "die__" messages on the dwarves list and repo, with the hint of an elfutils connection [2] but nothing conclusive.
Details of the git commit and build environment are as follows:
Quoted 29 lines
$ git log -1 --oneline  bpf/master
9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
test_sk_storage_tracing.c
$ lsb_release -a
Description:    Ubuntu 22.04.4 LTS
$ cat gcc-compile.txt
ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
$ mips64el-linux-gnuabi64-gcc --version
mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
$ mips64el-linux-gnuabi64-ld --version
GNU ld (GNU Binutils for Ubuntu) 2.38
$ pahole --version
v1.26
$ ldd $(which pahole)
        linux-vdso.so.1 (0x00007fff16f3f000)
        libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
        libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
Quoted 6 lines
$ dpkg -s elfutils
Package: elfutils
...
Version: 0.186-1build1
Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)
For reference, I also attached the full .config and build log from the above.
I should add this is not only a problem with the latest bpf/master but also appears to affect the 6.6.x LTS kernel, which I tested while building a mips64el OpenWrt distro image. That build environment employs the latest gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.
Not only do I see similar warnings from resolve_btfids and pahole, but while running the distro image I encounter module loading failures that suggest ELF corruption in some module .ko files, based on the following:
Quoted 5 lines
root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
...
init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
...
Quoted 7 lines
$ man init_module
...
The following errors may additionally occur for init_module():
...
     EINVAL param_values is invalid, or some part of the ELF image in
     module_image contains inconsistencies.
...

Quoted 8 lines

I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]

Cheers,
Hengqi

Jiri Olsa

2024-05-31 08:13 UTC #2 lore raw

Quoted 32 lines · Hengqi Chen

Hi Tony,
Quoted 29 lines · Tony Ambardar
Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
Some relate to resolve_btfids:
Quoted 10 lines
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
Quoted 8 lines
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32
These do not appear to be #ifdef-related and have similar past reports [1].

I can reproduce the warning just for bpf_session_cookie, which has fix in progress:
https://lore.kernel.org/bpf/[email protected]/T/#t

Quoted 11 lines

Quoted 10 lines
I also see many pahole failures during BTF encoding of modules, such as:
Quoted 7 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!

Quoted 4 lines

The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.

hi,
+1, could you also check the pahole version you used?

jirka

Quoted 103 lines

Quoted 91 lines
Quoted 15 lines
LD [M] lib/test_bpf.ko
BTF [M] lib/test_bpf.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
LD [M] lib/crc-ccitt.ko
BTF [M] lib/crc-ccitt.ko
die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
die__process_unit: tag not supported 0x11 (compile_unit)!
die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
LD [M] lib/libcrc32c.ko
BTF [M] lib/libcrc32c.ko
die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
die__process_unit: tag not supported 0x11 (compile_unit)!
die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
Quoted 8 lines
LD [M] lib/ts_kmp.ko
BTF [M] lib/ts_kmp.ko
LD [M] lib/ts_bm.ko
BTF [M] lib/ts_bm.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
I have seen reports of various similar "die__" messages on the dwarves list and repo, with the hint of an elfutils connection [2] but nothing conclusive.
Details of the git commit and build environment are as follows:
Quoted 29 lines
$ git log -1 --oneline  bpf/master
9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
test_sk_storage_tracing.c
$ lsb_release -a
Description:    Ubuntu 22.04.4 LTS
$ cat gcc-compile.txt
ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
$ mips64el-linux-gnuabi64-gcc --version
mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
$ mips64el-linux-gnuabi64-ld --version
GNU ld (GNU Binutils for Ubuntu) 2.38
$ pahole --version
v1.26
$ ldd $(which pahole)
        linux-vdso.so.1 (0x00007fff16f3f000)
        libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
        libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
Quoted 6 lines
$ dpkg -s elfutils
Package: elfutils
...
Version: 0.186-1build1
Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)
For reference, I also attached the full .config and build log from the above.
I should add this is not only a problem with the latest bpf/master but also appears to affect the 6.6.x LTS kernel, which I tested while building a mips64el OpenWrt distro image. That build environment employs the latest gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.
Not only do I see similar warnings from resolve_btfids and pahole, but while running the distro image I encounter module loading failures that suggest ELF corruption in some module .ko files, based on the following:
Quoted 5 lines
root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
...
init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
...
Quoted 7 lines
$ man init_module
...
The following errors may additionally occur for init_module():
...
     EINVAL param_values is invalid, or some part of the ELF image in
     module_image contains inconsistencies.
...
Quoted 8 lines
I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]
Cheers,
Hengqi

Tony Ambardar

author

2024-05-31 11:36 UTC #3 lore raw

Hi Jiri,

Quoted 32 lines · Jiri Olsa

Quoted 31 lines · Hengqi Chen
Hi Tony,
Quoted 28 lines · Tony Ambardar
Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
Some relate to resolve_btfids:
Quoted 18 lines
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32

Quoted 5 lines

Quoted 1 line
Quoted 1 line
These do not appear to be #ifdef-related and have similar past reports [1].
I can reproduce the warning just for bpf_session_cookie, which has fix in progress:
https://lore.kernel.org/bpf/[email protected]/T/#t

I gather there are different root causes for these warnings. From the link, the issue with bpf_session_cookie seems related to conditional compilation, which have come up before on the mailing list IIRC.

In comparison, consider my above warning for bpf_key_put, which is defined in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is enabled in my config and so not the issue. I can in fact see the global text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.

So I suspect the warnings might come from linker or optimization problems, or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?

Quoted 12 lines

Quoted 12 lines
Quoted 11 lines
I also see many pahole failures during BTF encoding of modules, such as:
Quoted 7 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!

Quoted 7 lines

Quoted 4 lines
The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.
hi,
+1, could you also check the pahole version you used?

No luck I'm afraid with using the latest elfutils as suggested.

I used pahole v1.26 in my original testing for this bug report, as noted below (might have been buried a bit).

Quoted 20 lines

jirka

SNIP

Quoted 15 lines

Details of the git commit and build environment are as follows:

Quoted 13 lines

$ git log -1 --oneline  bpf/master
9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
test_sk_storage_tracing.c

$ lsb_release -a
Description:    Ubuntu 22.04.4 LTS

$ cat gcc-compile.txt
ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6

$ mips64el-linux-gnuabi64-gcc --version
mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0

Quoted 19 lines

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
$ mips64el-linux-gnuabi64-ld --version
GNU ld (GNU Binutils for Ubuntu) 2.38
$ pahole --version
v1.26
SNIP
Quoted 11 lines
Quoted 8 lines
I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]
Cheers,
Hengqi

Thanks,
Tony

Jiri Olsa

2024-05-31 14:21 UTC #4 lore raw

Quoted 41 lines · Tony Ambardar

Hi Jiri,
Quoted 32 lines · Jiri Olsa
Quoted 31 lines · Hengqi Chen
Hi Tony,
Quoted 28 lines · Tony Ambardar
Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
Some relate to resolve_btfids:
Quoted 18 lines
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32
Quoted 5 lines
Quoted 1 line
Quoted 1 line
These do not appear to be #ifdef-related and have similar past reports [1].
I can reproduce the warning just for bpf_session_cookie, which has fix in progress:
https://lore.kernel.org/bpf/[email protected]/T/#t

Quoted 8 lines

I gather there are different root causes for these warnings. From the link, the issue with bpf_session_cookie seems related to conditional compilation, which have come up before on the mailing list IIRC.
In comparison, consider my above warning for bpf_key_put, which is defined in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is enabled in my config and so not the issue. I can in fact see the global text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.

yes, that would be a problem, the resolve_btfids goes through the BTF lists and needs to resolve the function in vmlinux, when it's not there, it will output the warning above

Quoted 3 lines

So I suspect the warnings might come from linker or optimization problems, or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?

I can reproduce now resolve_btfids warnings:

LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_is_return
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages

when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder the __bpf_kfunc flags do not work properly on this cross compile chain.. and linker just optimize it away?

#define __bpf_kfunc __used noinline

thanks,
jirka

Quoted 68 lines

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
I also see many pahole failures during BTF encoding of modules, such as:
Quoted 7 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
Quoted 7 lines
Quoted 4 lines
The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.
hi,
+1, could you also check the pahole version you used?
No luck I'm afraid with using the latest elfutils as suggested.
I used pahole v1.26 in my original testing for this bug report, as noted below (might have been buried a bit).
Quoted 20 lines
jirka
SNIP
Quoted 15 lines
Quoted 15 lines
Details of the git commit and build environment are as follows:
Quoted 13 lines
$ git log -1 --oneline  bpf/master
9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
test_sk_storage_tracing.c
$ lsb_release -a
Description:    Ubuntu 22.04.4 LTS
$ cat gcc-compile.txt
ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
$ mips64el-linux-gnuabi64-gcc --version
mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
$ mips64el-linux-gnuabi64-ld --version
GNU ld (GNU Binutils for Ubuntu) 2.38
$ pahole --version
v1.26
SNIP
Quoted 11 lines
Quoted 8 lines
I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]
Cheers,
Hengqi
Thanks,
Tony

Tony Ambardar

author

2024-06-03 09:02 UTC #5 lore raw

Quoted 56 lines · Jiri Olsa

Quoted 41 lines · Tony Ambardar
Hi Jiri,
Quoted 32 lines · Jiri Olsa
Quoted 31 lines · Hengqi Chen
Hi Tony,
Quoted 28 lines · Tony Ambardar
Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
Some relate to resolve_btfids:
Quoted 18 lines
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32
Quoted 5 lines
Quoted 1 line
Quoted 1 line
These do not appear to be #ifdef-related and have similar past reports [1].
I can reproduce the warning just for bpf_session_cookie, which has fix in progress:
https://lore.kernel.org/bpf/[email protected]/T/#t
Quoted 8 lines
I gather there are different root causes for these warnings. From the link, the issue with bpf_session_cookie seems related to conditional compilation, which have come up before on the mailing list IIRC.
In comparison, consider my above warning for bpf_key_put, which is defined in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is enabled in my config and so not the issue. I can in fact see the global text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.
yes, that would be a problem, the resolve_btfids goes through the BTF lists and needs to resolve the function in vmlinux, when it's not there, it will output the warning above
Quoted 1 line

Quoted 5 lines

Quoted 2 lines
So I suspect the warnings might come from linker or optimization problems, or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?
I can reproduce now resolve_btfids warnings:

That's great to hear! Did you also use mips64el? And were you able to see the various "die__proc:" errors from pahole during BTF generation?

Quoted 17 lines

LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_is_return
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages

Quoted 3 lines

when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder the __bpf_kfunc flags do not work properly on this cross compile chain.. and linker just optimize it away?

Yes, I verified the problem occurs during link-time garbage cleaning, when LTO is enabled through e.g. LD_DEAD_CODE_DATA_ELIMINATION=y.

Quoted 3 lines

#define __bpf_kfunc __used noinline

After researching, I believe this isn't a mips problem specifically, rather the __bpf_kfunc tag only partly works. It appears the __used macro works at the compiler and IR-level but is ignored by the linker. I have a working fix that also handles the linker level, so I'll post the patches shortly.

Unfortunately, that still leaves the pahole errors generating BTF, which are serious as they seem to break module loading.

Quoted 2 lines

thanks,
jirka

Cheers,
Tony

Quoted 69 lines

Quoted 69 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
I also see many pahole failures during BTF encoding of modules, such as:
Quoted 7 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
Quoted 7 lines
Quoted 4 lines
The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.
hi,
+1, could you also check the pahole version you used?
No luck I'm afraid with using the latest elfutils as suggested.
I used pahole v1.26 in my original testing for this bug report, as noted below (might have been buried a bit).
Quoted 20 lines
jirka
SNIP
Quoted 15 lines
Quoted 15 lines
Details of the git commit and build environment are as follows:
Quoted 13 lines
$ git log -1 --oneline  bpf/master
9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
test_sk_storage_tracing.c
$ lsb_release -a
Description:    Ubuntu 22.04.4 LTS
$ cat gcc-compile.txt
ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
$ mips64el-linux-gnuabi64-gcc --version
mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
$ mips64el-linux-gnuabi64-ld --version
GNU ld (GNU Binutils for Ubuntu) 2.38
$ pahole --version
v1.26
SNIP
Quoted 11 lines
Quoted 8 lines
I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]
Cheers,
Hengqi
Thanks,
Tony

Jiri Olsa

2024-06-03 09:19 UTC #6 lore raw

Quoted 57 lines · Tony Ambardar

Quoted 56 lines · Jiri Olsa
Quoted 41 lines · Tony Ambardar
Hi Jiri,
Quoted 32 lines · Jiri Olsa
Quoted 31 lines · Hengqi Chen
Hi Tony,
Quoted 28 lines · Tony Ambardar
Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
Some relate to resolve_btfids:
Quoted 18 lines
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32
Quoted 5 lines
Quoted 1 line
Quoted 1 line
These do not appear to be #ifdef-related and have similar past reports [1].
I can reproduce the warning just for bpf_session_cookie, which has fix in progress:
https://lore.kernel.org/bpf/[email protected]/T/#t
Quoted 8 lines
I gather there are different root causes for these warnings. From the link, the issue with bpf_session_cookie seems related to conditional compilation, which have come up before on the mailing list IIRC.
In comparison, consider my above warning for bpf_key_put, which is defined in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is enabled in my config and so not the issue. I can in fact see the global text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.
yes, that would be a problem, the resolve_btfids goes through the BTF lists and needs to resolve the function in vmlinux, when it's not there, it will output the warning above
Quoted 1 line

Quoted 7 lines

Quoted 5 lines
Quoted 2 lines
So I suspect the warnings might come from linker or optimization problems, or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?
I can reproduce now resolve_btfids warnings:
That's great to hear! Did you also use mips64el? And were you able to see

nope, just mips64, couldn't find mips64el on fedora

Quoted 1 line

the various "die__proc:" errors from pahole during BTF generation?

yep, I can see them as well

Quoted 26 lines

Quoted 18 lines
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_is_return
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
Quoted 3 lines
when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder the __bpf_kfunc flags do not work properly on this cross compile chain.. and linker just optimize it away?
Yes, I verified the problem occurs during link-time garbage cleaning, when LTO is enabled through e.g. LD_DEAD_CODE_DATA_ELIMINATION=y.
Quoted 1 line

Quoted 7 lines

Quoted 2 lines
#define __bpf_kfunc __used noinline
After researching, I believe this isn't a mips problem specifically, rather the __bpf_kfunc tag only partly works. It appears the __used macro works at the compiler and IR-level but is ignored by the linker. I have a working fix that also handles the linker level, so I'll post the patches shortly.

great, I was wondering how to fix that

thanks,
jirka

Quoted 78 lines

Unfortunately, that still leaves the pahole errors generating BTF, which are serious as they seem to break module loading.
Quoted 2 lines
thanks,
jirka
Cheers,
Tony
Quoted 69 lines
Quoted 69 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
I also see many pahole failures during BTF encoding of modules, such as:
Quoted 7 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
Quoted 7 lines
Quoted 4 lines
The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.
hi,
+1, could you also check the pahole version you used?
No luck I'm afraid with using the latest elfutils as suggested.
I used pahole v1.26 in my original testing for this bug report, as noted below (might have been buried a bit).
Quoted 20 lines
jirka
SNIP
Quoted 15 lines
Quoted 15 lines
Details of the git commit and build environment are as follows:
Quoted 13 lines
$ git log -1 --oneline  bpf/master
9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
test_sk_storage_tracing.c
$ lsb_release -a
Description:    Ubuntu 22.04.4 LTS
$ cat gcc-compile.txt
ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
$ mips64el-linux-gnuabi64-gcc --version
mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
$ mips64el-linux-gnuabi64-ld --version
GNU ld (GNU Binutils for Ubuntu) 2.38
$ pahole --version
v1.26
SNIP
Quoted 11 lines
Quoted 8 lines
I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]
Cheers,
Hengqi
Thanks,
Tony

Tony Ambardar

author

2024-06-03 12:16 UTC #7 lore raw

cover bpf: Fix linker optimization removing kfuncs

In reply to Tony Ambardar

This patch series fixes unwanted stripping of kernel kfuncs during linker optimization, as indicated by build warnings from resolve_btfids e.g. "WARN: resolve_btfids: unresolved symbol ...". This can happen because the __bpf_kfunc macro annotating kfunc declarations is ignored during linking.

Patch 1 adds support for the compiler attribute "__retain__", used to avoid linker garbage cleanup. Patch 2 then updates __bpf_kfunc to use this attribute when LTO builds are enabled.

Tony Ambardar (2):
Compiler Attributes: Add __retain macro
bpf: Harden __bpf_kfunc tag against linker kfunc removal

 include/linux/btf.h                 |  2 +-
 include/linux/compiler_attributes.h | 14 ++++++++++++++
 2 files changed, 15 insertions(+), 1 deletion(-)

Signature · 1 line

2.34.1

Tony Ambardar

author

2024-06-03 12:16 UTC #8 lore raw

PATCH 1/2 Compiler Attributes: Add __retain macro

Some code includes the __used macro to prevent functions and data from being optimized out. This macro implements __attribute__((__used__)), which operates at the compiler and IR-level, and so still allows a linker to remove objects intended to be kept.

Compilers supporting __attribute__((__retain__)) can address this gap by setting the flag SHF_GNU_RETAIN on the section of a function/variable, indicating to the linker the object should be retained. This attribute is available since gcc 11, clang 13, and binutils 2.36.

Provide a __retain macro implementing __attribute__((__retain__)), whose first user will be the '__bpf_kfunc' tag.

Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here

Cc [email protected] # v6.6+

Signed-off-by Tony Ambardar <[email protected]>

---
 include/linux/compiler_attributes.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

include/linux/compiler_attributes.h +14 -0

index 32284cd26d52..1c22e1a734dc 100644
@@ -326,6 +326,20 @@
  */
 #define __pure                          __attribute__((__pure__))
 
+/*
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
 /*
  *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-section-function-attribute
  *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Variable-Attributes.html#index-section-variable-attribute

Signature · 1 line

2.34.1

Miguel Ojeda

2024-06-03 13:57 UTC #9 lore raw

Re: [PATCH bpf v1 1/2] Compiler Attributes: Add __retain macro

include/linux/compiler_attributes.h line 337

Quoted 4 lines · Tony Ambardar

include/linux/compiler_attributes.h +3 -0

+#if __has_attribute(__retain__) && \
+       (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+        defined(CONFIG_LTO_CLANG))

Since this attribute depends on the configuration, please move it to compiler_types.h instead.

Cheers,
Miguel

Tony Ambardar

author

2024-06-04 02:38 UTC #10 lore raw

Re: [PATCH bpf v1 1/2] Compiler Attributes: Add __retain macro

Quoted 8 lines · Miguel Ojeda

include/linux/compiler_attributes.h line 337
Quoted 4 lines · Tony Ambardar
include/linux/compiler_attributes.h +3 -0
+#if __has_attribute(__retain__) && \
+       (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+        defined(CONFIG_LTO_CLANG))
Since this attribute depends on the configuration, please move it to compiler_types.h instead.

Noted and thanks for clarifying; I'll change in v2.

Cheers,
Tony

Quoted 3 lines

Cheers,
Miguel

Tony Ambardar

author

2024-06-03 12:16 UTC #11 lore raw

PATCH 2/2 bpf: Harden __bpf_kfunc tag against linker kfunc removal

In reply to Tony Ambardar

BPF kfuncs are often not directly referenced and may be inadvertently removed by optimization steps during kernel builds, thus the __bpf_kfunc tag mitigates against this removal by including the __used macro. However, this macro alone does not prevent removal during linking, and may still yield build warnings (e.g. on mips64el):

LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32

Update the __bpf_kfunc tag to better guard against linker optimization by including the new __retain compiler macro, which fixes the warnings above.

Verify the __retain macro with readelf by checking object flags for 'R':

  $ readelf -Wa kernel/trace/bpf_trace.o
  Section Headers:
    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
  ...
    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
  ...
  Key to Flags:
  ...
    R (retain), D (mbind), p (processor specific)

Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here

Reported-by kernel test robot <[email protected]>

Closes https://lore.kernel.org/r/[email protected]/ open here

Fixes 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")

Cc [email protected] # v6.6+

Signed-off-by Tony Ambardar <[email protected]>

---
 include/linux/btf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

include/linux/btf.h +1 -1

index f9e56fd12a9f..7c3e40c3295e 100644
@@ -82,7 +82,7 @@
  * as to avoid issues such as the compiler inlining or eliding either a static
  * kfunc, or a global kfunc in an LTO build.
  */
-#define __bpf_kfunc __used noinline
+#define __bpf_kfunc __used __retain noinline
 
 #define __bpf_kfunc_start_defs()					       \
 	__diag_push();							       \

Signature · 1 line

2.34.1

Tony Ambardar

author

2024-06-04 05:23 UTC #12 lore raw

v2 cover bpf: Fix linker optimization removing kfuncs

In reply to Tony Ambardar

Patch 1 adds support for the compiler attribute "__retain__", used to avoid linker garbage cleanup. Patch 2 then updates __bpf_kfunc to use this attribute when LTO builds are enabled.

Build-tested locally against mips64el with gcc 13.3, and run through upstream kernel-patches/bpf CI.

Tony Ambardar (2):
compiler_types.h: Define __retain for __attribute__((__retain__))
bpf: Harden __bpf_kfunc tag against linker kfunc removal

 include/linux/btf.h            |  2 +-
 include/linux/compiler_types.h | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)

Signature · 5 lines

v2:
- move __retain macro to compiler_types.h (Miguel Ojeda)

-- 
2.34.1

Tony Ambardar

author

2024-06-04 05:23 UTC #13 lore raw

PATCH v2 1/2 compiler_types.h: Define __retain for __attribute__((__retain__))

Provide a __retain macro implementing __attribute__((__retain__)), whose first user will be the '__bpf_kfunc' tag.

Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here

Cc [email protected] # v6.6+

Signed-off-by Tony Ambardar <[email protected]>

---
 include/linux/compiler_types.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

include/linux/compiler_types.h +23 -0

index 93600de3800b..f14c275950b5 100644
@@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
 # define __preserve_most
 #endif
 
+/*
+ * Annotating a function/variable with __retain tells the compiler to place
+ * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
+ * instructs the linker to retain the object during garbage-cleanup or LTO
+ * phases.
+ *
+ * Note that the __used macro is also used to prevent functions or data
+ * being optimized out, but operates at the compiler/IR-level and may still
+ * allow unintended removal of objects during linking.
+ *
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
 /* Compiler specific macros. */
 #ifdef __clang__
 #include <linux/compiler-clang.h>

Signature · 1 line

2.34.1

Yonghong Song

reviewer

2024-06-05 05:55 UTC #14 lore raw

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

Quoted 36 lines · Tony Ambardar

Some code includes the __used macro to prevent functions and data from being optimized out. This macro implements __attribute__((__used__)), which operates at the compiler and IR-level, and so still allows a linker to remove objects intended to be kept.
Compilers supporting __attribute__((__retain__)) can address this gap by setting the flag SHF_GNU_RETAIN on the section of a function/variable, indicating to the linker the object should be retained. This attribute is available since gcc 11, clang 13, and binutils 2.36.
Provide a __retain macro implementing __attribute__((__retain__)), whose first user will be the '__bpf_kfunc' tag.
Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here
Cc [email protected] # v6.6+
Signed-off-by Tony Ambardar <[email protected]>
---
  include/linux/compiler_types.h | 23 +++++++++++++++++++++++
  1 file changed, 23 insertions(+)
include/linux/compiler_types.h +8 -0
index 93600de3800b..f14c275950b5 100644
@@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
  # define __preserve_most
  #endif
 
+/*
+ * Annotating a function/variable with __retain tells the compiler to place
+ * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
+ * instructs the linker to retain the object during garbage-cleanup or LTO
+ * phases.
+ *
+ * Note that the __used macro is also used to prevent functions or data
+ * being optimized out, but operates at the compiler/IR-level and may still

include/linux/compiler_types.h static inline void __chk_io_ptr(const volatile void __iomem *ptr) { } · line 163

Quoted 10 lines

include/linux/compiler_types.h +10 -0

+ * allow unintended removal of objects during linking.
+ *
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))

Could you explain why CONFIG_LTO_CLANG is added here? IIUC, the __used macro permits garbage collection at section level, so CLANG_LTO_CLANG without CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
shuold not change final section dynamics, right?

Quoted 8 lines

include/linux/compiler_types.h +5 -0

+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
  /* Compiler specific macros. */
  #ifdef __clang__
  #include <linux/compiler-clang.h>

Tony Ambardar

author

2024-06-10 22:56 UTC #15 lore raw

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

Quoted 45 lines · Yonghong Song

Quoted 43 lines · Tony Ambardar
Some code includes the __used macro to prevent functions and data from being optimized out. This macro implements __attribute__((__used__)), which operates at the compiler and IR-level, and so still allows a linker to remove objects intended to be kept.
Compilers supporting __attribute__((__retain__)) can address this gap by setting the flag SHF_GNU_RETAIN on the section of a function/variable, indicating to the linker the object should be retained. This attribute is available since gcc 11, clang 13, and binutils 2.36.
Provide a __retain macro implementing __attribute__((__retain__)), whose first user will be the '__bpf_kfunc' tag.
Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here
Cc [email protected] # v6.6+
Signed-off-by Tony Ambardar <[email protected]>
---
  include/linux/compiler_types.h | 23 +++++++++++++++++++++++
  1 file changed, 23 insertions(+)
include/linux/compiler_types.h +16 -0
index 93600de3800b..f14c275950b5 100644
@@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
  # define __preserve_most
  #endif
+/*
+ * Annotating a function/variable with __retain tells the compiler to place
+ * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
+ * instructs the linker to retain the object during garbage-cleanup or LTO
+ * phases.
+ *
+ * Note that the __used macro is also used to prevent functions or data
+ * being optimized out, but operates at the compiler/IR-level and may still
+ * allow unintended removal of objects during linking.
+ *
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
+#if __has_attribute(__retain__) && \

Quoted 8 lines

include/linux/compiler_types.h static inline void __chk_io_ptr(const volatile void __iomem *ptr) { } · line 163
Quoted 2 lines
include/linux/compiler_types.h +2 -0
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
Could you explain why CONFIG_LTO_CLANG is added here? IIUC, the __used macro permits garbage collection at section level, so CLANG_LTO_CLANG without CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
shuold not change final section dynamics, right?

Hi Yonghong,

I included the conditional guard to ensure consistent behaviour between __retain and other features forcing split sections. In particular, the same guard is used in vmlinux.lds.h to merge split sections where needed. For example, using __retain in llvm builds without CONFIG_LTO was failing CI tests on kernel-patches/bpf because the kernel didn't boot properly. And in further testing, the kernel had no issues loading BPF kfunc modules with such split sections, so I left the module (partial) linking scripts alone.

Maybe I misunderstand you question re: __used?

Thanks,
Tony

Quoted 9 lines

include/linux/compiler_types.h +5 -0

+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
  /* Compiler specific macros. */
  #ifdef __clang__
  #include <linux/compiler-clang.h>

Yonghong Song

reviewer

2024-06-14 18:47 UTC #16 lore raw

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

Quoted 54 lines · Tony Ambardar

Quoted 43 lines · Yonghong Song
Quoted 42 lines · Tony Ambardar
Some code includes the __used macro to prevent functions and data from being optimized out. This macro implements __attribute__((__used__)), which operates at the compiler and IR-level, and so still allows a linker to remove objects intended to be kept.
Compilers supporting __attribute__((__retain__)) can address this gap by setting the flag SHF_GNU_RETAIN on the section of a function/variable, indicating to the linker the object should be retained. This attribute is available since gcc 11, clang 13, and binutils 2.36.
Provide a __retain macro implementing __attribute__((__retain__)), whose first user will be the '__bpf_kfunc' tag.
Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here
Cc [email protected] # v6.6+
Signed-off-by Tony Ambardar <[email protected]>
---
   include/linux/compiler_types.h | 23 +++++++++++++++++++++++
   1 file changed, 23 insertions(+)
include/linux/compiler_types.h +15 -0
index 93600de3800b..f14c275950b5 100644
@@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
   # define __preserve_most
   #endif
+/*
+ * Annotating a function/variable with __retain tells the compiler to place
+ * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
+ * instructs the linker to retain the object during garbage-cleanup or LTO
+ * phases.
+ *
+ * Note that the __used macro is also used to prevent functions or data
+ * being optimized out, but operates at the compiler/IR-level and may still
+ * allow unintended removal of objects during linking.
+ *
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
Quoted 8 lines
include/linux/compiler_types.h static inline void __chk_io_ptr(const volatile void __iomem *ptr) { } · line 162
Quoted 3 lines
include/linux/compiler_types.h +3 -0
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
Could you explain why CONFIG_LTO_CLANG is added here? IIUC, the __used macro permits garbage collection at section level, so CLANG_LTO_CLANG without CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
shuold not change final section dynamics, right?
Hi Yonghong,

Quoted 7 lines

I included the conditional guard to ensure consistent behaviour between __retain and other features forcing split sections. In particular, the same guard is used in vmlinux.lds.h to merge split sections where needed. For example, using __retain in llvm builds without CONFIG_LTO was failing CI tests on kernel-patches/bpf because the kernel didn't boot properly. And in further testing, the kernel had no issues loading BPF kfunc modules with such split sections, so I left the module (partial) linking scripts alone.

I tried with both bpf and bpf-next tree and I cannot make CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y in .config file. The following are all occurances in Kconfig:

$ egrep -r HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/mips/Kconfig:      select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/powerpc/Kconfig:   select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if HAVE_OBJTOOL_MCOUNT && (!ARCH_USING_PATCHABLE_FUNCTION_ENTRY || (!CC_IS_GCC || GCC_VERSION >= 110100))
arch/riscv/Kconfig:     select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD
init/Kconfig:config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
init/Kconfig:   depends on HAVE_LD_DEAD_CODE_DATA_ELIMINATION

Are there some pending patches to enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION for x86?

I could foce CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with the following hack:

init/Kconfig +1 -1

index 72404c1f2157..adf8718e2f5b 100644
@@ -1402,7 +1402,7 @@ config CC_OPTIMIZE_FOR_SIZE
 endchoice
 
 config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
-       bool
+       def_bool y
        help
          This requires that the arch annotates or otherwise protects
          its external entry points from being discarded. Linker scripts

But with the above, I cannot boot the kernel.

Did I miss anything?

Quoted 13 lines

Maybe I misunderstand you question re: __used?
Thanks,
Tony
Quoted 8 lines
Quoted 8 lines
include/linux/compiler_types.h +5 -0
+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
   /* Compiler specific macros. */
   #ifdef __clang__
   #include <linux/compiler-clang.h>

Tony Ambardar

author

2024-06-15 06:57 UTC #17 lore raw

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

Quoted 64 lines · Yonghong Song

Quoted 54 lines · Tony Ambardar
Quoted 43 lines · Yonghong Song
Quoted 42 lines · Tony Ambardar
Some code includes the __used macro to prevent functions and data from being optimized out. This macro implements __attribute__((__used__)), which operates at the compiler and IR-level, and so still allows a linker to remove objects intended to be kept.
Compilers supporting __attribute__((__retain__)) can address this gap by setting the flag SHF_GNU_RETAIN on the section of a function/variable, indicating to the linker the object should be retained. This attribute is available since gcc 11, clang 13, and binutils 2.36.
Provide a __retain macro implementing __attribute__((__retain__)), whose first user will be the '__bpf_kfunc' tag.
Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here
Cc [email protected] # v6.6+
Signed-off-by Tony Ambardar <[email protected]>
---
   include/linux/compiler_types.h | 23 +++++++++++++++++++++++
   1 file changed, 23 insertions(+)
include/linux/compiler_types.h +15 -0
index 93600de3800b..f14c275950b5 100644
@@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
   # define __preserve_most
   #endif
+/*
+ * Annotating a function/variable with __retain tells the compiler to place
+ * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
+ * instructs the linker to retain the object during garbage-cleanup or LTO
+ * phases.
+ *
+ * Note that the __used macro is also used to prevent functions or data
+ * being optimized out, but operates at the compiler/IR-level and may still
+ * allow unintended removal of objects during linking.
+ *
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
Quoted 8 lines
include/linux/compiler_types.h static inline void __chk_io_ptr(const volatile void __iomem *ptr) { } · line 162
Quoted 3 lines
include/linux/compiler_types.h +3 -0
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
Could you explain why CONFIG_LTO_CLANG is added here? IIUC, the __used macro permits garbage collection at section level, so CLANG_LTO_CLANG without CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
shuold not change final section dynamics, right?
Hi Yonghong,
Quoted 7 lines
I included the conditional guard to ensure consistent behaviour between __retain and other features forcing split sections. In particular, the same guard is used in vmlinux.lds.h to merge split sections where needed. For example, using __retain in llvm builds without CONFIG_LTO was failing CI tests on kernel-patches/bpf because the kernel didn't boot properly. And in further testing, the kernel had no issues loading BPF kfunc modules with such split sections, so I left the module (partial) linking scripts alone.

Quoted 2 lines

I tried with both bpf and bpf-next tree and I cannot make CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y in .config file. The following are all occurances in Kconfig:

My understanding is one doesn't directly set HAVE_LD_DEAD_CODE_...; it's a per-arch capability flag which guards setting LD_DEAD_CODE_DATA_ELIMINATION but only targets "small systems" (i.e. embedded), so no surprise x86 isn't in the arch list below.

Quoted 10 lines

$ egrep -r HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/mips/Kconfig:      select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/powerpc/Kconfig:   select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if HAVE_OBJTOOL_MCOUNT && (!ARCH_USING_PATCHABLE_FUNCTION_ENTRY || (!CC_IS_GCC || GCC_VERSION >= 110100))
arch/riscv/Kconfig:     select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD
init/Kconfig:config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
init/Kconfig:   depends on HAVE_LD_DEAD_CODE_DATA_ELIMINATION

Are there some pending patches to enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION for x86?

I doubt it given the target arches above, but curious what's the need for x86 support? Only x86_32? My patches were motivated seeing resolve_btfids and pahole errors for a couple years on MIPS routers. I don't recall seeing the same for x86 builds, so my testing focussed more on preserving x86 builds rather than adding/testing the arch flag for x86.

Quoted 15 lines

I could foce CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with the following hack:

init/Kconfig +1 -1

index 72404c1f2157..adf8718e2f5b 100644
@@ -1402,7 +1402,7 @@ config CC_OPTIMIZE_FOR_SIZE
 endchoice
 config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
-       bool
+       def_bool y
        help
          This requires that the arch annotates or otherwise protects
          its external entry points from being discarded. Linker scripts

Quoted 1 line

But with the above, I cannot boot the kernel.

OK, interesting exercise. Setting HAVE_LD_DEAD_CODE_DATA_ELIMINATION shouldn't change anything itself so I suppose you are also setting LD_DEAD_CODE_DATA_ELIMINATION? From previous testing on kernel-patches/CI, first guess would be vmlinux linker script doing section merges unaware of some x86 quirk. Or x86-specific linker script unhappy with split sections.

Quoted 17 lines

Did I miss anything?
Quoted 13 lines
Maybe I misunderstand you question re: __used?
Thanks,
Tony
Quoted 8 lines
Quoted 8 lines
include/linux/compiler_types.h +5 -0
+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
   /* Compiler specific macros. */
   #ifdef __clang__
   #include <linux/compiler-clang.h>

Yonghong Song

reviewer

2024-06-17 03:27 UTC #18 lore raw

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

Quoted 63 lines · Tony Ambardar

Quoted 62 lines · Yonghong Song
Quoted 61 lines · Tony Ambardar
Quoted 43 lines · Yonghong Song
Quoted 42 lines · Tony Ambardar
Some code includes the __used macro to prevent functions and data from being optimized out. This macro implements __attribute__((__used__)), which operates at the compiler and IR-level, and so still allows a linker to remove objects intended to be kept.
Compilers supporting __attribute__((__retain__)) can address this gap by setting the flag SHF_GNU_RETAIN on the section of a function/variable, indicating to the linker the object should be retained. This attribute is available since gcc 11, clang 13, and binutils 2.36.
Provide a __retain macro implementing __attribute__((__retain__)), whose first user will be the '__bpf_kfunc' tag.
Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here
Cc [email protected] # v6.6+
Signed-off-by Tony Ambardar <[email protected]>
---
    include/linux/compiler_types.h | 23 +++++++++++++++++++++++
    1 file changed, 23 insertions(+)
include/linux/compiler_types.h +15 -0
index 93600de3800b..f14c275950b5 100644
@@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
    # define __preserve_most
    #endif
+/*
+ * Annotating a function/variable with __retain tells the compiler to place
+ * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
+ * instructs the linker to retain the object during garbage-cleanup or LTO
+ * phases.
+ *
+ * Note that the __used macro is also used to prevent functions or data
+ * being optimized out, but operates at the compiler/IR-level and may still
+ * allow unintended removal of objects during linking.
+ *
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
Quoted 8 lines
include/linux/compiler_types.h static inline void __chk_io_ptr(const volatile void __iomem *ptr) { } · line 162
Quoted 3 lines
include/linux/compiler_types.h +3 -0
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
Could you explain why CONFIG_LTO_CLANG is added here? IIUC, the __used macro permits garbage collection at section level, so CLANG_LTO_CLANG without CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
shuold not change final section dynamics, right?
Hi Yonghong,
I included the conditional guard to ensure consistent behaviour between __retain and other features forcing split sections. In particular, the same guard is used in vmlinux.lds.h to merge split sections where needed. For example, using __retain in llvm builds without CONFIG_LTO was failing CI tests on kernel-patches/bpf because the kernel didn't boot properly. And in further testing, the kernel had no issues loading BPF kfunc modules with such split sections, so I left the module (partial) linking scripts alone.

Quoted 6 lines

Quoted 2 lines
I tried with both bpf and bpf-next tree and I cannot make CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y in .config file. The following are all occurances in Kconfig:
My understanding is one doesn't directly set HAVE_LD_DEAD_CODE_...; it's a per-arch capability flag which guards setting LD_DEAD_CODE_DATA_ELIMINATION but only targets "small systems" (i.e. embedded), so no surprise x86 isn't in the arch list below.

I see. Yes, mips should support it but not x86. No wonder why I cannot reproduce.

Quoted 29 lines

Quoted 10 lines
$ egrep -r HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/mips/Kconfig:      select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/powerpc/Kconfig:   select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if HAVE_OBJTOOL_MCOUNT && (!ARCH_USING_PATCHABLE_FUNCTION_ENTRY || (!CC_IS_GCC || GCC_VERSION >= 110100))
arch/riscv/Kconfig:     select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD
init/Kconfig:config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
init/Kconfig:   depends on HAVE_LD_DEAD_CODE_DATA_ELIMINATION
Are there some pending patches to enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION for x86?
I doubt it given the target arches above, but curious what's the need for x86 support? Only x86_32? My patches were motivated seeing resolve_btfids and pahole errors for a couple years on MIPS routers. I don't recall seeing the same for x86 builds, so my testing focussed more on preserving x86 builds rather than adding/testing the arch flag for x86.
Quoted 14 lines
I could foce CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with the following hack:
init/Kconfig +1 -1
index 72404c1f2157..adf8718e2f5b 100644
@@ -1402,7 +1402,7 @@ config CC_OPTIMIZE_FOR_SIZE
  endchoice
  config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
-       bool
+       def_bool y
         help
           This requires that the arch annotates or otherwise protects
           its external entry points from being discarded. Linker scripts
 

Quoted 6 lines

Quoted 1 line
But with the above, I cannot boot the kernel.
OK, interesting exercise. Setting HAVE_LD_DEAD_CODE_DATA_ELIMINATION shouldn't change anything itself so I suppose you are also setting LD_DEAD_CODE_DATA_ELIMINATION? From previous testing on kernel-patches/CI, first guess would be vmlinux linker script doing section merges unaware of some x86 quirk. Or x86-specific linker script unhappy with split sections.

I guess x86 needs additional change to make HAVE_LD_DEAD_CODE_DATA_ELIMINATION work. I still curious about why CONFIG_LTO_CLANG is necessary.

In asm-generic/vmlinux.lds.h,

/*
* LD_DEAD_CODE_DATA_ELIMINATION option enables -fdata-sections, which
* generates .data.identifier sections, which need to be pulled in with
* .data. We don't want to pull in .data..other sections, which Linux
* has defined. Same for text and bss.
*
* With LTO_CLANG, the linker also splits sections by default, so we need
* these macros to combine the sections during the final link.
*
* RODATA_MAIN is not used because existing code already defines .rodata.x
* sections to be brought in with rodata.
*/
#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG)
#define TEXT_MAIN .text .text.[0-9a-zA-Z_]*
#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..L* .data..compoundliteral* .data.$__unnamed_* .data.$L* #define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]*
#define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* .rodata..L*
#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral*
#define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]*
#else
#define TEXT_MAIN .text
#define DATA_MAIN .data
#define SDATA_MAIN .sdata
#define RODATA_MAIN .rodata
#define BSS_MAIN .bss
#define SBSS_MAIN .sbss
#endif

If CONFIG_LTO_CLANG is defined and CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is not defined, it is not clear whether __used functions will get eliminated or not. I tried with thinlto with a simple example on x86 with some unused function marked with __used, and that function survived in the final binary.

But your patch won't hurt, so I am okay with it.

Quoted 16 lines

Quoted 16 lines
Did I miss anything?
Quoted 12 lines
Maybe I misunderstand you question re: __used?
Thanks,
Tony
Quoted 8 lines
Quoted 8 lines
include/linux/compiler_types.h +5 -0
+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
    /* Compiler specific macros. */
    #ifdef __clang__
    #include <linux/compiler-clang.h>

Tony Ambardar

author

2024-06-04 05:23 UTC #19 lore raw

PATCH v2 2/2 bpf: Harden __bpf_kfunc tag against linker kfunc removal

In reply to Tony Ambardar

Update the __bpf_kfunc tag to better guard against linker optimization by including the new __retain compiler macro, which fixes the warnings above.

Verify the __retain macro with readelf by checking object flags for 'R':

  $ readelf -Wa kernel/trace/bpf_trace.o
  Section Headers:
    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
  ...
    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
  ...
  Key to Flags:
  ...
    R (retain), D (mbind), p (processor specific)

Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here

Reported-by kernel test robot <[email protected]>

Closes https://lore.kernel.org/r/[email protected]/ open here

Fixes 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")

Cc [email protected] # v6.6+

Signed-off-by Tony Ambardar <[email protected]>

---
 include/linux/btf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

include/linux/btf.h +1 -1

index f9e56fd12a9f..7c3e40c3295e 100644
@@ -82,7 +82,7 @@
  * as to avoid issues such as the compiler inlining or eliding either a static
  * kfunc, or a global kfunc in an LTO build.
  */
-#define __bpf_kfunc __used noinline
+#define __bpf_kfunc __used __retain noinline
 
 #define __bpf_kfunc_start_defs()					       \
 	__diag_push();							       \

Signature · 1 line

2.34.1

Jiri Olsa

Tested-by Reviewed-by on v2 2/2

2024-06-04 07:56 UTC #20 lore raw

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

Quoted 39 lines · Tony Ambardar

BPF kfuncs are often not directly referenced and may be inadvertently removed by optimization steps during kernel builds, thus the __bpf_kfunc tag mitigates against this removal by including the __used macro. However, this macro alone does not prevent removal during linking, and may still yield build warnings (e.g. on mips64el):
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32
Update the __bpf_kfunc tag to better guard against linker optimization by including the new __retain compiler macro, which fixes the warnings above.
Verify the __retain macro with readelf by checking object flags for 'R':
  $ readelf -Wa kernel/trace/bpf_trace.o
  Section Headers:
    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
  ...
    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
  ...
  Key to Flags:
  ...
    R (retain), D (mbind), p (processor specific)

Quoted 6 lines

Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here
Reported-by kernel test robot <[email protected]>
Closes https://lore.kernel.org/r/[email protected]/ open here
Fixes 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
Cc [email protected] # v6.6+
Signed-off-by Tony Ambardar <[email protected]>

tested on mips64 cross build and the warnings are gone and related functions are in the vmlinux

patchset looks good to me

Tested-by Jiri Olsa <[email protected]>

Reviewed-by Jiri Olsa <[email protected]>

thanks,
jirka

Quoted 20 lines

---
 include/linux/btf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

include/linux/btf.h +1 -1

index f9e56fd12a9f..7c3e40c3295e 100644
@@ -82,7 +82,7 @@
  * as to avoid issues such as the compiler inlining or eliding either a static
  * kfunc, or a global kfunc in an LTO build.
  */
-#define __bpf_kfunc __used noinline
+#define __bpf_kfunc __used __retain noinline
 
 #define __bpf_kfunc_start_defs()					       \
 	__diag_push();							       \

Signature · 1 line

2.34.1

Geert Uytterhoeven

2024-06-25 10:46 UTC #21 lore raw

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

In reply to Tony Ambardar

Hi Tony,

Quoted 39 lines · Tony Ambardar

BPF kfuncs are often not directly referenced and may be inadvertently removed by optimization steps during kernel builds, thus the __bpf_kfunc tag mitigates against this removal by including the __used macro. However, this macro alone does not prevent removal during linking, and may still yield build warnings (e.g. on mips64el):
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32
Update the __bpf_kfunc tag to better guard against linker optimization by including the new __retain compiler macro, which fixes the warnings above.
Verify the __retain macro with readelf by checking object flags for 'R':
 $ readelf -Wa kernel/trace/bpf_trace.o
 Section Headers:
   [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
 ...
   [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
 ...
 Key to Flags:
 ...
   R (retain), D (mbind), p (processor specific)

Quoted 6 lines

Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here
Reported-by kernel test robot <[email protected]>
Closes https://lore.kernel.org/r/[email protected]/ open here
Fixes 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
Cc [email protected] # v6.6+
Signed-off-by Tony Ambardar <[email protected]>

Thanks for your patch, which is now commit 7bdcedd5c8fb88e7 ("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in v6.10-rc5.

This is causing build failures on ARM with
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:

    net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11859 | {
          | ^
    net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11872 | {
          | ^
    net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11885 | {
          | ^
    net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11906 | {
          | ^
    net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    12092 | {
          | ^
    net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
      713 | {
          | ^
    net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
      736 | {
          | ^
    net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
      769 | {
          | ^
    [...]

My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).

include/linux/btf.h line 88

Quoted 11 lines

include/linux/btf.h +1 -1

@@ -82,7 +82,7 @@
 * as to avoid issues such as the compiler inlining or eliding either a static
 * kfunc, or a global kfunc in an LTO build.
 */
-#define __bpf_kfunc __used noinline
+#define __bpf_kfunc __used __retain noinline
 
 #define __bpf_kfunc_start_defs()					       \
 	__diag_push();							       \

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Jiri Olsa

2024-06-26 09:52 UTC #22 lore raw

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

Quoted 82 lines · Geert Uytterhoeven

Hi Tony,
Quoted 39 lines · Tony Ambardar
BPF kfuncs are often not directly referenced and may be inadvertently removed by optimization steps during kernel builds, thus the __bpf_kfunc tag mitigates against this removal by including the __used macro. However, this macro alone does not prevent removal during linking, and may still yield build warnings (e.g. on mips64el):
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32
Update the __bpf_kfunc tag to better guard against linker optimization by including the new __retain compiler macro, which fixes the warnings above.
Verify the __retain macro with readelf by checking object flags for 'R':
 $ readelf -Wa kernel/trace/bpf_trace.o
 Section Headers:
   [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
 ...
   [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
 ...
 Key to Flags:
 ...
   R (retain), D (mbind), p (processor specific)
Quoted 6 lines
Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here
Reported-by kernel test robot <[email protected]>
Closes https://lore.kernel.org/r/[email protected]/ open here
Fixes 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
Cc [email protected] # v6.6+
Signed-off-by Tony Ambardar <[email protected]>
Thanks for your patch, which is now commit 7bdcedd5c8fb88e7 ("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in v6.10-rc5.
This is causing build failures on ARM with
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:
    net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11859 | {
          | ^
    net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11872 | {
          | ^
    net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11885 | {
          | ^
    net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11906 | {
          | ^
    net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    12092 | {
          | ^
    net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
      713 | {
          | ^
    net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
      736 | {
          | ^
    net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
      769 | {
          | ^
    [...]

Quoted 1 line

My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).

hum, so it'd mean __has_attribute(__retain__) returns true while gcc still ignores the retain attribute.. like in this bug which seems similar:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99587
but not sure how it got fixed.. any chance you can upgrade gcc and retest?

jirka

Quoted 23 lines

include/linux/btf.h line 88
Quoted 12 lines
include/linux/btf.h +1 -1
@@ -82,7 +82,7 @@
 * as to avoid issues such as the compiler inlining or eliding either a static
 * kfunc, or a global kfunc in an LTO build.
 */
-#define __bpf_kfunc __used noinline
+#define __bpf_kfunc __used __retain noinline
 
 #define __bpf_kfunc_start_defs()					       \
 	__diag_push();							       \
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Geert Uytterhoeven

2024-06-26 11:40 UTC #23 lore raw

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

Hi Jiri,

Quoted 81 lines · Jiri Olsa

Quoted 80 lines · Geert Uytterhoeven
Quoted 39 lines · Tony Ambardar
BPF kfuncs are often not directly referenced and may be inadvertently removed by optimization steps during kernel builds, thus the __bpf_kfunc tag mitigates against this removal by including the __used macro. However, this macro alone does not prevent removal during linking, and may still yield build warnings (e.g. on mips64el):
LD vmlinux
BTFIDS vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
NM System.map
SORTTAB vmlinux
OBJCOPY vmlinux.32
Update the __bpf_kfunc tag to better guard against linker optimization by including the new __retain compiler macro, which fixes the warnings above.
Verify the __retain macro with readelf by checking object flags for 'R':
 $ readelf -Wa kernel/trace/bpf_trace.o
 Section Headers:
   [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
 ...
   [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
 ...
 Key to Flags:
 ...
   R (retain), D (mbind), p (processor specific)
Quoted 6 lines
Link https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/ open here
Reported-by kernel test robot <[email protected]>
Closes https://lore.kernel.org/r/[email protected]/ open here
Fixes 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
Cc [email protected] # v6.6+
Signed-off-by Tony Ambardar <[email protected]>
Thanks for your patch, which is now commit 7bdcedd5c8fb88e7 ("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in v6.10-rc5.
This is causing build failures on ARM with
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:
    net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11859 | {
          | ^
    net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11872 | {
          | ^
    net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11885 | {
          | ^
    net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    11906 | {
          | ^
    net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
    12092 | {
          | ^
    net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
      713 | {
          | ^
    net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
      736 | {
          | ^
    net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
      769 | {
          | ^
    [...]

Quoted 6 lines

Quoted 1 line
My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).
hum, so it'd mean __has_attribute(__retain__) returns true while gcc still ignores the retain attribute.. like in this bug which seems similar:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99587
but not sure how it got fixed.. any chance you can upgrade gcc and retest?

Indeed, __has_attribute(__retain__) returns true, while the attribute is not supported.

My test program:

cat > /tmp/a.c <<EOF
#if __has_attribute(__retain__)
#warning __retain__ OK
#else
#warning No __retain__
#endif

int x __attribute__((__retain__));
EOF

$ arm-linux-gnueabihf-gcc-11 -c /tmp/a.c # gcc version 11.4.0 (Ubuntu
11.4.0-1ubuntu1~22.04))

/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]

    2 | #warning __retain__ OK
      |  ^~~~~~~
/tmp/a.c:7:1: warning: ‘retain’ attribute ignored [-Wattributes]
    7 | int x __attribute__((__retain__));
      | ^~~

Oops

$ arm-linux-gnueabihf-gcc-12 -c /tmp/a.c # gcc version 12.3.0 (Ubuntu
12.3.0-1ubuntu1~22.04)
/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~

Fixed

It works fine with the native gcc-11:

$ gcc-11 -c /tmp/a.c # gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~

I gave it a try on all installed gcc-11 compilers.

/usr/bin/aarch64-linux-gnu-gcc-11
/usr/bin/alpha-linux-gnu-gcc-11
/usr/bin/arm-linux-gnueabi-gcc-11
/usr/bin/arm-linux-gnueabihf-gcc-11
/usr/bin/hppa64-linux-gnu-gcc-11
/usr/bin/hppa-linux-gnu-gcc-11
/usr/bin/m68k-linux-gnu-gcc-11
/usr/bin/powerpc64le-linux-gnu-gcc-11
/usr/bin/powerpc64-linux-gnu-gcc-11
/usr/bin/powerpc-linux-gnu-gcc-11
/usr/bin/riscv64-linux-gnu-gcc-11
/usr/bin/s390x-linux-gnu-gcc-11
/usr/bin/sh4-linux-gnu-gcc-11
/usr/bin/sparc64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnux32-gcc-11

All of them failed (incl. x32), except for the native x86_64-linux-gnu-gcc-11.

It works fine with all installed gcc-12 compilers (arm-linux-gnueabihf-gcc-12, m68k-linux-gnu-gcc-12, x86_64-linux-gnu-gcc-12).

With gcc-9, the absence of __retain__ is detected correctly.

Gr{oetje,eeting}s,

Geert

Signature · 5 lines

Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

[email protected]

CI Applied

2024-06-14 17:20 UTC #24 lore raw

Re: [PATCH bpf v2 0/2] bpf: Fix linker optimization removing kfuncs

In reply to Tony Ambardar

This series was applied to bpf/bpf.git (master) by Daniel Borkmann <[email protected]>:

Hello:

This series was applied to bpf/bpf.git (master)
by Daniel Borkmann <[email protected]>:

Quoted 10 lines · On Mon, 3 Jun 2024 22:23:14 -0700 you wrote:

This patch series fixes unwanted stripping of kernel kfuncs during linker optimization, as indicated by build warnings from resolve_btfids e.g. "WARN: resolve_btfids: unresolved symbol ...". This can happen because the __bpf_kfunc macro annotating kfunc declarations is ignored during linking.
Patch 1 adds support for the compiler attribute "__retain__", used to avoid linker garbage cleanup. Patch 2 then updates __bpf_kfunc to use this attribute when LTO builds are enabled.
[...]

Here is the summary with links:
- [bpf,v2,1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

    https://git.kernel.org/bpf/bpf/c/0a5d3258d7c9
  - [bpf,v2,2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal
    https://git.kernel.org/bpf/bpf/c/7bdcedd5c8fb

You are awesome, thank you!

Signature · 2 lines

Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Tony Ambardar

author

2024-05-31 10:49 UTC #25 lore raw

In reply to Hengqi Chen

Hello Hengqi,

Quoted 20 lines · Hengqi Chen

Hi Tony,
Quoted 7 lines · Tony Ambardar
Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
SNIP
Quoted 8 lines
Quoted 8 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!

Quoted 4 lines

The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.

Good idea. I tried rebuilding elfutils from the latest upstream commit:
https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508

I then linked this elfutils with pahole built from the latest pahole/next:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993

I also confirmed resolve_btfids links to the new elfutils, and then rebuilt the kernel with the same config. Unfortunately, the warnings/errors from resolve_btfids and pahole still occur.

Quoted 5 lines

SNIP
Quoted 4 lines
I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.

Quoted 8 lines

Quoted 5 lines
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]
Cheers,
Hengqi

Thanks for the suggestion,
Tony

Arnaldo Carvalho de Melo

2024-05-31 16:06 UTC #26 lore raw

Quoted 22 lines · Tony Ambardar

Hello Hengqi,
Quoted 19 lines · Hengqi Chen
Hi Tony,
Quoted 7 lines · Tony Ambardar
Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
SNIP
Quoted 8 lines
Quoted 8 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!

Can you check the kernel CONFIG_ variables related to DEBUG information and post them here? I have this on fedora:

[acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
[acme@nine linux]$

If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the fedora config.

- Arnaldo

Quoted 31 lines

Quoted 4 lines
The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.
Good idea. I tried rebuilding elfutils from the latest upstream commit:
https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
I then linked this elfutils with pahole built from the latest pahole/next:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
I also confirmed resolve_btfids links to the new elfutils, and then rebuilt the kernel with the same config. Unfortunately, the warnings/errors from resolve_btfids and pahole still occur.
Quoted 5 lines
SNIP
Quoted 4 lines
I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.
Quoted 8 lines
Quoted 5 lines
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]
Cheers,
Hengqi
Thanks for the suggestion,
Tony

Tony Ambardar

author

2024-05-31 21:46 UTC #27 lore raw

Quoted 39 lines · Arnaldo Carvalho de Melo

Quoted 22 lines · Tony Ambardar
Hello Hengqi,
Quoted 19 lines · Hengqi Chen
Hi Tony,
Quoted 7 lines · Tony Ambardar
Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
SNIP
Quoted 8 lines
Quoted 8 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
Can you check the kernel CONFIG_ variables related to DEBUG information and post them here? I have this on fedora:
[acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
[acme@nine linux]$

Quoted 6 lines

If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the fedora config.
- Arnaldo

OK, I have those 3 settings as expected on bpf/master branch:

kodidev:~/linux$ grep CONFIG_DEBUG_INFO .config
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_NONE is not set
# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
CONFIG_DEBUG_INFO_DWARF4=y
# CONFIG_DEBUG_INFO_DWARF5 is not set
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_INFO_COMPRESSED_NONE=y
# CONFIG_DEBUG_INFO_COMPRESSED_ZLIB is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y

I also attached the full .config in my original mail if you think other config options might be suspect.

Is there anything else I can check? What do you make of the many pahole "die__process:" warnings? Could it be related to your comment here:
https://github.com/acmel/dwarves/issues/45#issuecomment-1974004606

Thanks,
Tony

Quoted 31 lines

Quoted 31 lines
Quoted 4 lines
The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.
Good idea. I tried rebuilding elfutils from the latest upstream commit:
https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
I then linked this elfutils with pahole built from the latest pahole/next:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
I also confirmed resolve_btfids links to the new elfutils, and then rebuilt the kernel with the same config. Unfortunately, the warnings/errors from resolve_btfids and pahole still occur.
Quoted 5 lines
SNIP
Quoted 4 lines
I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.
Quoted 8 lines
Quoted 5 lines
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]
Cheers,
Hengqi
Thanks for the suggestion,
Tony

Tony Ambardar

author

2024-06-03 11:20 UTC #28 lore raw

In reply to Arnaldo Carvalho de Melo

Quoted 39 lines · Arnaldo Carvalho de Melo

Quoted 22 lines · Tony Ambardar
Hello Hengqi,
Quoted 19 lines · Hengqi Chen
Hi Tony,
Quoted 7 lines · Tony Ambardar
Hello,
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
SNIP
Quoted 8 lines
Quoted 8 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
Can you check the kernel CONFIG_ variables related to DEBUG information and post them here? I have this on fedora:
[acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
[acme@nine linux]$

Quoted 3 lines

If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the fedora config.

One more useful observation: the pahole BTF generation problems appear only for mips64. If I build the same config for mips32be I see none of those "die__process: errors" from pahole and the processed modules load properly.

I also tried running pahole under gdb to look further, but lack knowledge of pahole and libdw internals, so couldn't narrow things down to a pahole, elfutils or compiler-output problem.

To help troubleshoot further, I packaged my base vmlinux, some module .ko files showing problems, and a reproducer script. I've uploaded the tarball here:

https://www.dropbox.com/scl/fi/3ce22fi2q861wqvbq9mwy/repro_die__process.tar.xz?rlkey=ev494phabmfl5qe55xrn1jh3t&st=vrp5mxhh&dl=0

Thanks,
Tony

Quoted 34 lines

- Arnaldo
Quoted 31 lines
Quoted 4 lines
The issue seems to be related to elfutils. Have you tried build from the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the reloc_simple_type hook.
Good idea. I tried rebuilding elfutils from the latest upstream commit:
https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
I then linked this elfutils with pahole built from the latest pahole/next:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
I also confirmed resolve_btfids links to the new elfutils, and then rebuilt the kernel with the same config. Unfortunately, the warnings/errors from resolve_btfids and pahole still occur.
Quoted 5 lines
SNIP
Quoted 4 lines
I'd be grateful if some of the BTF/pahole experts could please review this issue and share next steps or other details I might provide.
Quoted 8 lines
Quoted 5 lines
Thanks,
Tony Ambardar
Link https://lore.kernel.org/all/[email protected]/ [1] open here
Link https://github.com/acmel/dwarves/issues/45 [2]
Cheers,
Hengqi
Thanks for the suggestion,
Tony

Arnaldo Carvalho de Melo

2024-06-03 14:56 UTC #29 lore raw

Quoted 45 lines · Tony Ambardar

Quoted 32 lines · Arnaldo Carvalho de Melo
Quoted 8 lines · Tony Ambardar
Quoted 6 lines · Hengqi Chen
Quoted 3 lines · Tony Ambardar
For some time now I'm seeing multiple issues during BTF generation while building recent kernels targeting mips64el, and would appreciate some help to understand and fix the problems.
SNIP
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
  CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
  CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
  CC [M]  net/psample/psample.mod.o
  LD [M]  crypto/cmac.ko
BTF [M] crypto/cmac.ko
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
Can you check the kernel CONFIG_ variables related to DEBUG information and post them here? I have this on fedora:
[acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
[acme@nine linux]$
Quoted 3 lines
If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the fedora config.
One more useful observation: the pahole BTF generation problems appear only for mips64. If I build the same config for mips32be I see none of those "die__process: errors" from pahole and the processed modules load properly.
I also tried running pahole under gdb to look further, but lack knowledge of pahole and libdw internals, so couldn't narrow things down to a pahole, elfutils or compiler-output problem.

Quoted 5 lines

To help troubleshoot further, I packaged my base vmlinux, some module .ko files showing problems, and a reproducer script. I've uploaded the tarball here:
https://www.dropbox.com/scl/fi/3ce22fi2q861wqvbq9mwy/repro_die__process.tar.xz?rlkey=ev494phabmfl5qe55xrn1jh3t&st=vrp5mxhh&dl=0

So, with that vmlinux, on a Fedora 39 x86_64 notebook:

⬢[acme@toolbox repro_die__process]$ readelf -wi vmlinux | head -45 Contents of the .debug_info section:

Compilation Unit @ offset 0:
Length: 0x3f (32-bit)
Version: 4
Abbrev Offset: 0
Pointer Size: 8
<0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)

    <c>   DW_AT_stmt_list   : 0
    <10>   DW_AT_ranges      : 0
    <14>   DW_AT_name        : (indirect string, offset: 0): arch/mips/kernel/head.S
    <18>   DW_AT_comp_dir    : (indirect string, offset: 0x18): /home/kodidev/linux
    <1c>   DW_AT_producer    : (indirect string, offset: 0x2c): GNU AS 2.42
    <20>   DW_AT_language    : 32769	(MIPS assembler)
 <1><22>: Abbrev Number: 2 (DW_TAG_subprogram)
    <23>   DW_AT_name        : (indirect string, offset: 0x38): kernel_entry
    <27>   DW_AT_external    : 1
    <27>   DW_AT_type        : <0x41>
    <28>   DW_AT_low_pc      : 0xffffffff80b0ce68
    <30>   DW_AT_high_pc     : 172
 <1><32>: Abbrev Number: 2 (DW_TAG_subprogram)
    <33>   DW_AT_name        : (indirect string, offset: 0x45): smp_bootstrap
    <37>   DW_AT_external    : 1
    <37>   DW_AT_type        : <0x41>
    <38>   DW_AT_low_pc      : 0xffffffff80b0cf14
    <40>   DW_AT_high_pc     : 44
 <1><41>: Abbrev Number: 3 (DW_TAG_unspecified_type)
 <1><42>: Abbrev Number: 0
  Compilation Unit @ offset 0x43:
   Length:        0x22ed9 (32-bit)
   Version:       4
   Abbrev Offset: 0x26
   Pointer Size:  8
 <0><4e>: Abbrev Number: 192 (DW_TAG_compile_unit)
    <50>   DW_AT_producer    : (indirect string, offset: 0x8147): GNU C11 13.3.0 -G 0 -mel -mno-check-zero-division -mabi=64 -mno-abicalls -msoft-float -march=mips64r2 -msym32 -mllsc -mips64r2 -mno-shared -g -gdwarf-4 -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-strict-aliasing -fno-pic -ffreestanding -fstack-check=no -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector -fno-stack-clash-protection -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack -ffunction-sections -fdata-sections
    <54>   DW_AT_language    : 12	(ANSI C99)
    <55>   DW_AT_name        : (indirect string, offset: 0xce66): init/main.c
    <59>   DW_AT_comp_dir    : (indirect string, offset: 0x18): /home/kodidev/linux
    <5d>   DW_AT_ranges      : 0x1350
    <61>   DW_AT_low_pc      : 0
    <69>   DW_AT_stmt_list   : 0x80
 <1><6d>: Abbrev Number: 91 (DW_TAG_base_type)
    <6e>   DW_AT_byte_size   : 8
    <6f>   DW_AT_encoding    : 7	(unsigned)
    <70>   DW_AT_name        : (indirect string, offset: 0xb1c7): long unsigned int

⬢[acme@toolbox repro_die__process]$

⬢[acme@toolbox repro_die__process]$ time pahole -F dwarf vmlinux | wc -l 84377

real 0m10.992s
user 0m9.526s
sys 0m1.445s
⬢[acme@toolbox repro_die__process]$

No warnings, 84377 lines of types printed from DWARF that file, which is comparable to the BTF in it:

⬢[acme@toolbox repro_die__process]$ time pahole -F btf vmlinux | wc -l 83994

real 0m0.121s
user 0m0.095s
sys 0m0.031s
⬢[acme@toolbox repro_die__process]$

There are a few diffs from types generated from DWARF to those generated from BTF (unexpected ones, that is) as noticed by btfdiff (found in the pahole git repo), that its just some print ordering issue:

⬢[acme@toolbox repro_die__process]$ btfdiff vmlinux

tmp/btfdiff.btf.u762WM renamed from tmp/btfdiff.dwarf.oX71hp +45 ~~-45~~

@@ -10027,6 +10027,24 @@ struct bio_map_data {
 };
 struct bio_post_read_ctx {
 	struct bio *               bio;                                                  /*     0     8 */
+	struct work_struct {
+		/* typedef atomic_long_t -> atomic64_t */ struct {
+			/* typedef s64 -> __s64 */ long long int counter;                /*     8     8 */
+		} data; /*     8     8 */
+		struct list_head {
+			struct list_head * next;                                         /*    16     8 */
+			struct list_head * prev;                                         /*    24     8 */
+		} entry; /*    16    16 */
+		/* typedef work_func_t */ void               (*func)(struct work_struct *); /*    32     8 */
+	} work; /*     8    32 */
+	unsigned int               cur_step;                                             /*    40     4 */
+	unsigned int               enabled_steps;                                        /*    44     4 */
+
+	/* size: 48, cachelines: 1, members: 4 */
+	/* last cacheline: 48 bytes */
+};
+struct bio_post_read_ctx {
+	struct bio *               bio;                                                  /*     0     8 */
 	struct f2fs_sb_info *      sbi;                                                  /*     8     8 */
 	struct work_struct {
 		/* typedef atomic_long_t -> atomic64_t */ struct {

@@ -10049,24 +10067,6 @@ struct bio_post_read_ctx {
 	/* sum members: 57, holes: 1, sum holes: 3 */
 	/* padding: 4 */
 };
-struct bio_post_read_ctx {
-	struct bio *               bio;                                                  /*     0     8 */
-	struct work_struct {
-		/* typedef atomic_long_t -> atomic64_t */ struct {
-			/* typedef s64 -> __s64 */ long long int counter;                /*     8     8 */
-		} data; /*     8     8 */
-		struct list_head {
-			struct list_head * next;                                         /*    16     8 */
-			struct list_head * prev;                                         /*    24     8 */
-		} entry; /*    16    16 */
-		/* typedef work_func_t */ void               (*func)(struct work_struct *); /*    32     8 */
-	} work; /*     8    32 */
-	unsigned int               cur_step;                                             /*    40     4 */
-	unsigned int               enabled_steps;                                        /*    44     4 */
-
-	/* size: 48, cachelines: 1, members: 4 */
-	/* last cacheline: 48 bytes */
-};
 struct bio_set {
 	struct kmem_cache *        bio_slab;                                             /*     0     8 */
 	unsigned int               front_pad;                                            /*     8     4 */

@@ -43044,6 +43044,17 @@ struct dma_block {
 	/* last cacheline: 16 bytes */
 };
 struct dma_chan {
+	int                        lock;                                                 /*     0     4 */
+
+	/* XXX 4 bytes hole, try to pack */
+
+	const char  *              device_id;                                            /*     8     8 */
+
+	/* size: 16, cachelines: 1, members: 2 */
+	/* sum members: 12, holes: 1, sum holes: 4 */
+	/* last cacheline: 16 bytes */
+};
+struct dma_chan {
 	struct dma_device *        device;                                               /*     0     8 */
 	struct device *            slave;                                                /*     8     8 */
 	/* typedef dma_cookie_t -> s32 -> __s32 */ int                        cookie;    /*    16     4 */

@@ -43071,17 +43082,6 @@ struct dma_chan {
 	/* sum members: 108, holes: 1, sum holes: 4 */
 	/* last cacheline: 48 bytes */
 };
-struct dma_chan {
-	int                        lock;                                                 /*     0     4 */
-
-	/* XXX 4 bytes hole, try to pack */
-
-	const char  *              device_id;                                            /*     8     8 */
-
-	/* size: 16, cachelines: 1, members: 2 */
-	/* sum members: 12, holes: 1, sum holes: 4 */
-	/* last cacheline: 16 bytes */
-};
 struct dma_chan_dev {
 	struct dma_chan *          chan;                                                 /*     0     8 */
 	struct device {

@@ -125786,12 +125786,11 @@ struct perf_aux_event {
 		/* typedef __u16 */ short unsigned int misc;                             /*     4     2 */
 		/* typedef __u16 */ short unsigned int size;                             /*     6     2 */
 	} header; /*     0     8 */
-	/* typedef u64 -> __u64 */ long long unsigned int     offset;                    /*     8     8 */
-	/* typedef u64 -> __u64 */ long long unsigned int     size;                      /*    16     8 */
-	/* typedef u64 -> __u64 */ long long unsigned int     flags;                     /*    24     8 */
+	/* typedef u32 -> __u32 */ unsigned int               pid;                       /*     8     4 */
+	/* typedef u32 -> __u32 */ unsigned int               tid;                       /*    12     4 */
 
-	/* size: 32, cachelines: 1, members: 4 */
-	/* last cacheline: 32 bytes */
+	/* size: 16, cachelines: 1, members: 3 */
+	/* last cacheline: 16 bytes */
 };
 struct perf_aux_event {
 	struct perf_event_header {

@@ -125799,11 +125798,12 @@ struct perf_aux_event {
 		/* typedef __u16 */ short unsigned int misc;                             /*     4     2 */
 		/* typedef __u16 */ short unsigned int size;                             /*     6     2 */
 	} header; /*     0     8 */
-	/* typedef u32 -> __u32 */ unsigned int               pid;                       /*     8     4 */
-	/* typedef u32 -> __u32 */ unsigned int               tid;                       /*    12     4 */
+	/* typedef u64 -> __u64 */ long long unsigned int     offset;                    /*     8     8 */
+	/* typedef u64 -> __u64 */ long long unsigned int     size;                      /*    16     8 */
+	/* typedef u64 -> __u64 */ long long unsigned int     flags;                     /*    24     8 */
 
-	/* size: 16, cachelines: 1, members: 3 */
-	/* last cacheline: 16 bytes */
+	/* size: 32, cachelines: 1, members: 4 */
+	/* last cacheline: 32 bytes */
 };
 struct perf_bpf_event {
 	struct bpf_prog *          prog;                                                 /*     0     8 */

@@ -163037,10 +163037,11 @@ struct syscall_tp_t {
 
 	/* XXX 4 bytes hole, try to pack */
 
-	long unsigned int          args[6];                                              /*    16    48 */
+	long unsigned int          ret;                                                  /*    16     8 */
 
-	/* size: 64, cachelines: 1, members: 3 */
-	/* sum members: 60, holes: 1, sum holes: 4 */
+	/* size: 24, cachelines: 1, members: 3 */
+	/* sum members: 20, holes: 1, sum holes: 4 */
+	/* last cacheline: 24 bytes */
 };
 struct syscall_tp_t {
 	struct trace_entry {

@@ -163053,11 +163054,10 @@ struct syscall_tp_t {
 
 	/* XXX 4 bytes hole, try to pack */
 
-	long unsigned int          ret;                                                  /*    16     8 */
+	long unsigned int          args[6];                                              /*    16    48 */
 
-	/* size: 24, cachelines: 1, members: 3 */
-	/* sum members: 20, holes: 1, sum holes: 4 */
-	/* last cacheline: 24 bytes */
+	/* size: 64, cachelines: 1, members: 3 */
+	/* sum members: 60, holes: 1, sum holes: 4 */
 };
 struct syscall_trace_enter {
 	struct trace_entry {

⬢[acme@toolbox repro_die__process]$

But if we do it manually and without expanding types as btfdiff does:

⬢[acme@toolbox repro_die__process]$ pahole -F btf -C dma_chan vmlinux struct dma_chan {

	struct dma_device *        device;               /*     0     8 */
	struct device *            slave;                /*     8     8 */
	dma_cookie_t               cookie;               /*    16     4 */
	dma_cookie_t               completed_cookie;     /*    20     4 */
	int                        chan_id;              /*    24     4 */

/* XXX 4 bytes hole, try to pack */

struct dma_chan_dev * dev; /* 32 8 */
const char * name; /* 40 8 */
char * dbg_client_name; /* 48 8 */
struct list_head device_node; /* 56 16 */
/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */
struct dma_chan_percpu * local; /* 72 8 */
int client_count; /* 80 4 */
int table_count; /* 84 4 */
struct dma_router * router; /* 88 8 */
void * route_data; /* 96 8 */
void * private; /* 104 8 */

/* size: 112, cachelines: 2, members: 15 */ /* sum members: 108, holes: 1, sum holes: 4 */ /* last cacheline: 48 bytes */
};

⬢[acme@toolbox repro_die__process]$ pahole -F dwarf -C dma_chan vmlinux

struct dma_chan {
	int                        lock;                 /*     0     4 */

	/* XXX 4 bytes hole, try to pack */

	const char  *              device_id;            /*     8     8 */

	/* size: 16, cachelines: 1, members: 2 */
	/* sum members: 12, holes: 1, sum holes: 4 */
	/* last cacheline: 16 bytes */
};

⬢[acme@toolbox repro_die__process]$

⬢[acme@toolbox perf-tools]$ git grep 'struct dma_chan {' arch/mips/include/asm/mach-au1x00/au1000_dma.h:struct dma_chan { include/linux/dmaengine.h:struct dma_chan { kernel/dma.c:struct dma_chan {
⬢[acme@toolbox perf-tools]$

So we have this 'struct dma_chan' in the common kernel in include/linux/dmaengine.h

struct dma_chan {
        struct dma_device *device;
        struct device *slave;
        dma_cookie_t cookie;
        dma_cookie_t completed_cookie;

        /* sysfs */
        int chan_id;
        struct dma_chan_dev *dev;
        const char *name;       
#ifdef CONFIG_DEBUG_FS
        char *dbg_client_name;

#endif

        struct list_head device_node;
        struct dma_chan_percpu __percpu *local;
        int client_count;       
        int table_count;

        /* DMA router */
        struct dma_router *router;
        void *route_data;

        void *private;
};

And also this one in kernel/dma.c

struct dma_chan {
        int  lock;
        const char *device_id;
};

If we don't specify the name but grep for it:

⬢[acme@toolbox repro_die__process]$ pahole -F btf vmlinux | grep 'struct dma_chan {' -A5 struct dma_chan {

	struct dma_device *        device;               /*     0     8 */
	struct device *            slave;                /*     8     8 */
	dma_cookie_t               cookie;               /*    16     4 */
	dma_cookie_t               completed_cookie;     /*    20     4 */
	int                        chan_id;              /*    24     4 */

--
struct dma_chan {
int lock; /* 0 4 */

/* XXX 4 bytes hole, try to pack */

const char * device_id; /* 8 8 */
⬢[acme@toolbox repro_die__process]$ pahole -F dwarf vmlinux | grep 'struct dma_chan {' -A5 struct dma_chan {
int lock; /* 0 4 */

/* XXX 4 bytes hole, try to pack */

const char * device_id; /* 8 8 */

--
struct dma_chan {

	struct dma_device *        device;               /*     0     8 */
	struct device *            slave;                /*     8     8 */
	dma_cookie_t               cookie;               /*    16     4 */
	dma_cookie_t               completed_cookie;     /*    20     4 */
	int                        chan_id;              /*    24     4 */

⬢[acme@toolbox repro_die__process]$

So the vmlinux seems to have its BTF section successfully generated, lemme check the .ko files.

- Arnaldo

Arnaldo Carvalho de Melo

2024-06-03 17:40 UTC #30 lore raw

elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

Quoted 2 lines · Arnaldo Carvalho de Melo

So the vmlinux seems to have its BTF section successfully generated, lemme check the .ko files.

⬢[acme@toolbox repro_die__process]$ pahole -F dwarf netdevsim.ko | wc -l
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)! 11448
⬢[acme@toolbox repro_die__process]$ pahole -F btf netdevsim.ko | wc -l
libbpf: Invalid BTF string section
pahole: file 'netdevsim.ko' has no btf type information.
0
⬢[acme@toolbox repro_die__process]$

We manage to load a lot of DWARF types but then we hit that warning, no BTF encoded.

Humm,

⬢[acme@toolbox repro_die__process]$ eu-readelf -winfo netdevsim.ko | wc -l 0
⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | wc -l 779837
⬢[acme@toolbox repro_die__process]$

It seems this is some problem with elfutils...

After I added:

⬢[acme@toolbox pahole]$ git diff

dwarf_loader.c +2 -2

index f59477b44dfea026..b832c93cc2194eaf 100644
@@ -2847,8 +2847,8 @@ static int die__process(Dwarf_Die *die, struct cu *cu, struct conf_load *conf)
        }
 
        if (tag != DW_TAG_compile_unit && tag != DW_TAG_type_unit) {
-               fprintf(stderr, "%s: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got %s (0x%x)!\n",
-                       __FUNCTION__, dwarf_tag_name(tag), tag);
+               fprintf(stderr, "%s: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got %s (0x%x) @ %llx!\n",
+                       __FUNCTION__, dwarf_tag_name(tag), tag, (unsigned long long)dwarf_dieoffset(die));
                return -EINVAL;
        }

⬢[acme@toolbox pahole]$

We get:

⬢[acme@toolbox repro_die__process]$ pahole -F dwarf netdevsim.ko > /dev/null
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd) @ 529de! ⬢[acme@toolbox repro_die__process]$

And if we look at binutils' readeld DWARF dump we get:

⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | grep '<529de>' -B5 -A8
Compilation Unit @ offset 0x529d3:
Length: 0x2132e (32-bit)
Version: 4
Abbrev Offset: 0x1d0e
Pointer Size: 8
<0><529de>: Abbrev Number: 108 (DW_TAG_compile_unit)

    <529df>   DW_AT_producer    : (indirect string, offset: 0x41389): GNU C11 13.3.0 -G 0 -mel -mno-check-zero-division -mabi=64 -mno-abicalls -msoft-float -march=mips64r2 -msym32 -mlong-calls -mllsc -mips64r2 -mno-shared -g -gdwarf-4 -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-strict-aliasing -fno-pic -ffreestanding -fstack-check=no -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector -fno-stack-clash-protection -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack
    <529e3>   DW_AT_language    : 12	(ANSI C99)
    <529e4>   DW_AT_name        : (indirect string, offset: 0x44ce0): drivers/net/netdevsim/ethtool.c
    <529e8>   DW_AT_comp_dir    : (indirect string, offset: 0x3f407): /home/kodidev/linux
    <529ec>   DW_AT_low_pc      : 0x5270
    <529f4>   DW_AT_high_pc     : 0x65c
    <529fc>   DW_AT_stmt_list   : 0x709d

<1><52a00>: Abbrev Number: 57 (DW_TAG_base_type)
⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | grep '<529de>' -B5 -A8

And at that <529de> die it _is_ a DW_TAG_compile_unit, and we managed to process two other DW_TAG_compile_units up to that point:

⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | grep -w DW_TAG_compile_unit
<0><b>: Abbrev Number: 188 (DW_TAG_compile_unit)
<0><26c16>: Abbrev Number: 188 (DW_TAG_compile_unit)
<0><529de>: Abbrev Number: 108 (DW_TAG_compile_unit)
<0><73d10>: Abbrev Number: 204 (DW_TAG_compile_unit)
<0><a43a8>: Abbrev Number: 160 (DW_TAG_compile_unit)
<0><c7e66>: Abbrev Number: 110 (DW_TAG_compile_unit)
<0><e8f21>: Abbrev Number: 158 (DW_TAG_compile_unit)
<0><10c525>: Abbrev Number: 115 (DW_TAG_compile_unit)
<0><12da49>: Abbrev Number: 167 (DW_TAG_compile_unit)
<0><1546b9>: Abbrev Number: 140 (DW_TAG_compile_unit)
<0><17656a>: Abbrev Number: 1 (DW_TAG_compile_unit)
⬢[acme@toolbox repro_die__process]$

And eu-readelf also doesn't work for that mips vmlinux:

⬢[acme@toolbox repro_die__process]$ file vmlinux
vmlinux: ELF 64-bit LSB executable, MIPS, MIPS64 rel2 version 1 (SYSV), statically linked, BuildID[sha1]=19c8dafc6aa33a2e63b87487dfb10bf215bdf3a3, with debug_info, not stripped ⬢[acme@toolbox repro_die__process]$ eu-readelf -winfo vmlinux
⬢[acme@toolbox repro_die__process]$ eu-readelf -winfo+ vmlinux
⬢[acme@toolbox repro_die__process]$ readelf -wi vmlinux | head -30
Contents of the .debug_info section:

Compilation Unit @ offset 0:
Length: 0x3f (32-bit)
Version: 4
Abbrev Offset: 0
Pointer Size: 8
<0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)

    <c>   DW_AT_stmt_list   : 0
    <10>   DW_AT_ranges      : 0
    <14>   DW_AT_name        : (indirect string, offset: 0): arch/mips/kernel/head.S
    <18>   DW_AT_comp_dir    : (indirect string, offset: 0x18): /home/kodidev/linux
    <1c>   DW_AT_producer    : (indirect string, offset: 0x2c): GNU AS 2.42
    <20>   DW_AT_language    : 32769	(MIPS assembler)
 <1><22>: Abbrev Number: 2 (DW_TAG_subprogram)
    <23>   DW_AT_name        : (indirect string, offset: 0x38): kernel_entry
    <27>   DW_AT_external    : 1
    <27>   DW_AT_type        : <0x41>
    <28>   DW_AT_low_pc      : 0xffffffff80b0ce68
    <30>   DW_AT_high_pc     : 172
 <1><32>: Abbrev Number: 2 (DW_TAG_subprogram)
    <33>   DW_AT_name        : (indirect string, offset: 0x45): smp_bootstrap
    <37>   DW_AT_external    : 1
    <37>   DW_AT_type        : <0x41>
    <38>   DW_AT_low_pc      : 0xffffffff80b0cf14
    <40>   DW_AT_high_pc     : 44

<1><41>: Abbrev Number: 3 (DW_TAG_unspecified_type) <1><42>: Abbrev Number: 0
Compilation Unit @ offset 0x43:
Length: 0x22ed9 (32-bit)
⬢[acme@toolbox repro_die__process]$

Couldn't find a way to ask eu-readelf for more verbose output, where we could perhaps get some clue as to why it produces nothing while binutils readelf manages to grok it, Mark, do you know some other way to ask eu-readelf to produce more debug output?

I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state that then made eu-readelf to not be able to process it while pahole, that uses eltuils' libraries, was able to process the first two CUs for a kernel module and all the CUs for the vmlinux file :-\

Mark, the whole thread is available at:

https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u

Now tryint to look at elfutils source to see if I can figure out something there..

- Arnaldo

⬢[acme@toolbox repro_die__process]$ ltrace eu-readelf -winfo netdevsim.ko <SNIP>

strlen(".gdb_index")                                                                 = 10
strncmp(".mdebug.abi64", ".gdb_index", 10)                                           = 6
elf_nextscn(0x55fddd480470, 0x55fddd4820e8, 10, 103)                                 = 0x55fddd4821b8
gelf_getshdr(0x55fddd4821b8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_strptr(0x55fddd480470, 54, 349, 0x55fddd4831f8)                                  = 0x7f0970925879
strlen(".note.GNU-stack")                                                            = 15
strncmp(".note.GNU-stack", ".debug_abbrev", 13)                                      = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_addr")                                                                = 11
strncmp(".note.GNU-stack", ".debug_addr", 11)                                        = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_aranges")                                                             = 14
strncmp(".note.GNU-stack", ".debug_aranges", 14)                                     = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_frame")                                                               = 12
strncmp(".note.GNU-stack", ".debug_frame", 12)                                       = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_info")                                                                = 11
strncmp(".note.GNU-stack", ".debug_info", 11)                                        = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_types")                                                               = 12
strncmp(".note.GNU-stack", ".debug_types", 12)                                       = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_line")                                                                = 11
strncmp(".note.GNU-stack", ".debug_line", 11)                                        = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_loc")                                                                 = 10
strncmp(".note.GNU-stack", ".debug_loc", 10)                                         = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_loclists")                                                            = 15
strncmp(".note.GNU-stack", ".debug_loclists", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_pubnames")                                                            = 15
strncmp(".note.GNU-stack", ".debug_pubnames", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_str")                                                                 = 10
strncmp(".note.GNU-stack", ".debug_str", 10)                                         = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_line_str")                                                            = 15
strncmp(".note.GNU-stack", ".debug_line_str", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_str_offsets")                                                         = 18
strncmp(".note.GNU-stack", ".debug_str_offsets", 18)                                 = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_macinfo")                                                             = 14
strncmp(".note.GNU-stack", ".debug_macinfo", 14)                                     = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_macro")                                                               = 12
strncmp(".note.GNU-stack", ".debug_macro", 12)                                       = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_ranges")                                                              = 13
strncmp(".note.GNU-stack", ".debug_ranges", 13)                                      = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_rnglists")                                                            = 15
strncmp(".note.GNU-stack", ".debug_rnglists", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".eh_frame")                                                                  = 9
strncmp(".note.GNU-stack", ".eh_frame", 9)                                           = 9
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".eh_frame_hdr")                                                              = 13
strncmp(".note.GNU-stack", ".eh_frame_hdr", 13)                                      = 9
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".gcc_except_table")                                                          = 17
strncmp(".note.GNU-stack", ".gcc_except_table", 17)                                  = 7
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".gdb_index")                                                                 = 10
strncmp(".note.GNU-stack", ".gdb_index", 10)                                         = 7
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
elf_nextscn(0x55fddd480470, 0x55fddd4821b8, 14, 103)                                 = 0x55fddd482288
gelf_getshdr(0x55fddd482288, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482288, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482358
gelf_getshdr(0x55fddd482358, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482358, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482428
gelf_getshdr(0x55fddd482428, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482428, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd4824f8
gelf_getshdr(0x55fddd4824f8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd4824f8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd4825c8
gelf_getshdr(0x55fddd4825c8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd4825c8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482698
gelf_getshdr(0x55fddd482698, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482698, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482768
gelf_getshdr(0x55fddd482768, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482768, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482838
gelf_getshdr(0x55fddd482838, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482838, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482908
gelf_getshdr(0x55fddd482908, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482908, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd4829d8
gelf_getshdr(0x55fddd4829d8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd4829d8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482aa8
gelf_getshdr(0x55fddd482aa8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482aa8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482b78
gelf_getshdr(0x55fddd482b78, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482b78, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482c48
gelf_getshdr(0x55fddd482c48, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482c48, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482d18
gelf_getshdr(0x55fddd482d18, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482d18, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482de8
gelf_getshdr(0x55fddd482de8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482de8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482eb8
gelf_getshdr(0x55fddd482eb8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_strptr(0x55fddd480470, 54, 513, 0x55fddd4831f8)                                  = 0x7f097092591d
strlen(".BTF")                                                                       = 4
strncmp(".BTF", ".debug_abbrev", 13)                                                 = -34
strlen(".debug_addr")                                                                = 11
strncmp(".BTF", ".debug_addr", 11)                                                   = -34
strlen(".debug_aranges")                                                             = 14
strncmp(".BTF", ".debug_aranges", 14)                                                = -34
strlen(".debug_frame")                                                               = 12
strncmp(".BTF", ".debug_frame", 12)                                                  = -34
strlen(".debug_info")                                                                = 11
strncmp(".BTF", ".debug_info", 11)                                                   = -34
strlen(".debug_types")                                                               = 12
strncmp(".BTF", ".debug_types", 12)                                                  = -34
strlen(".debug_line")                                                                = 11
strncmp(".BTF", ".debug_line", 11)                                                   = -34
strlen(".debug_loc")                                                                 = 10
strncmp(".BTF", ".debug_loc", 10)                                                    = -34
strlen(".debug_loclists")                                                            = 15
strncmp(".BTF", ".debug_loclists", 15)                                               = -34
strlen(".debug_pubnames")                                                            = 15
strncmp(".BTF", ".debug_pubnames", 15)                                               = -34
strlen(".debug_str")                                                                 = 10
strncmp(".BTF", ".debug_str", 10)                                                    = -34
strlen(".debug_line_str")                                                            = 15
strncmp(".BTF", ".debug_line_str", 15)                                               = -34
strlen(".debug_str_offsets")                                                         = 18
strncmp(".BTF", ".debug_str_offsets", 18)                                            = -34
strlen(".debug_macinfo")                                                             = 14
strncmp(".BTF", ".debug_macinfo", 14)                                                = -34
strlen(".debug_macro")                                                               = 12
strncmp(".BTF", ".debug_macro", 12)                                                  = -34
strlen(".debug_ranges")                                                              = 13
strncmp(".BTF", ".debug_ranges", 13)                                                 = -34
strlen(".debug_rnglists")                                                            = 15
strncmp(".BTF", ".debug_rnglists", 15)                                               = -34
strlen(".eh_frame")                                                                  = 9
strncmp(".BTF", ".eh_frame", 9)                                                      = -35
strlen(".eh_frame_hdr")                                                              = 13
strncmp(".BTF", ".eh_frame_hdr", 13)                                                 = -35
strlen(".gcc_except_table")                                                          = 17
strncmp(".BTF", ".gcc_except_table", 17)                                             = -37
strlen(".gdb_index")                                                                 = 10
strncmp(".BTF", ".gdb_index", 10)                                                    = -37
elf_nextscn(0x55fddd480470, 0x55fddd482eb8, 10, 103)                                 = 0x55fddd482f88
gelf_getshdr(0x55fddd482f88, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482f88, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd483058
gelf_getshdr(0x55fddd483058, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd483058, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd483128
gelf_getshdr(0x55fddd483128, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd483128, 0x55fddd480470, 0x55fddd4831f8)          = 0
dwfl_end(0, 165, 0x55fddd480538, 0x55fddd4831f8)                                     = 0
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(0x55fddd47eef0)                                                                 = <void>
<... dwfl_getmodules resumed> )                                                      = 0
dwfl_end(0x55fddd47ea90, 0x55fddd47eee0, 0x55fddd47e, 1)                             = 6
close(3)                                                                             = 0
__cxa_finalize(0x55fddbe44780, 10, 0, 0x7fff00974ee0)                                = 1

+++ exited (status 0) +++
⬢[acme@toolbox repro_die__process]$

Mark Wielaard

2024-06-03 19:27 UTC #31 lore raw

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

Hi,

Quoted 9 lines · Arnaldo Carvalho de Melo

Couldn't find a way to ask eu-readelf for more verbose output, where we could perhaps get some clue as to why it produces nothing while binutils readelf manages to grok it, Mark, do you know some other way to ask eu-readelf to produce more debug output?
I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state that then made eu-readelf to not be able to process it while pahole, that uses eltuils' libraries, was able to process the first two CUs for a kernel module and all the CUs for the vmlinux file :-\

I haven't looked at the vmlinux file. But for the .ko file the issue is that the elfutils MIPS backend isn't complete. Specifically MIPS relocations aren't recognized (and so cannot be applied). There are some pending patches which try to fix that:

https://patchwork.sourceware.org/project/elfutils/list/?series=31601

Cheers,

Mark

Tony Ambardar

author

2024-06-04 03:47 UTC #32 lore raw

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

Hi Mark,

Quoted 13 lines · Mark Wielaard

Quoted 12 lines · Arnaldo Carvalho de Melo
Couldn't find a way to ask eu-readelf for more verbose output, where we could perhaps get some clue as to why it produces nothing while binutils readelf manages to grok it, Mark, do you know some other way to ask eu-readelf to produce more debug output?
I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state that then made eu-readelf to not be able to process it while pahole, that uses eltuils' libraries, was able to process the first two CUs for a kernel module and all the CUs for the vmlinux file :-\
Mark, the whole thread is available at:

Quoted 8 lines

Quoted 1 line
https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
I haven't looked at the vmlinux file. But for the .ko file the issue is that the elfutils MIPS backend isn't complete. Specifically MIPS relocations aren't recognized (and so cannot be applied). There are some pending patches which try to fix that:
https://patchwork.sourceware.org/project/elfutils/list/?series=31601

Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend work for MIPS, and I locally rebuilt elfutils and then pahole from their respective next/main branches. For elfutils, main (935ee131cf7c) includes

e259f126 Support Mips architecture
f2acb069 stack: Fix stack unwind failure on mips
db33cb0c backends: Add register_info, return_value_location, core_note mips

which partially applies the patchwork series but leaves out the support for readelf, strip, and elflint.

I believe this means the vmlinux and .ko files I shared are OK, or is there more backend work needed for MIPS?

The bits missing in eu-readelf would explain the blank output both Arnaldo and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the patchwork readelf patch locally but ran into merge conflicts.

CCing Ying Huang for any more insight.

Thanks,
Tony

Ying Huang

2024-06-04 08:29 UTC #33 lore raw

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

Hi Tony,

Yes, I also found that merge will have some conflicts now.

And you can modify the key code as follows to fix the issue about "eu-readelf -w" could not show info.

But this can not fix other display error in the result of the "eu-readelf -w" which need modify the way to get symbol index and type.

Quoted 22 lines

libelf/libelfP.h +1 -0

index bdd2cc6a..6565ee02 100644
@@ -620,4 +620,5 @@ extern void __libelf_reset_rawdata (Elf_Scn *scn, void *buf, size_t size,
 #define ELF64_MIPS_R_TYPE1(i)          ((i) & 0xff)
 #define ELF64_MIPS_R_TYPE2(i)           (((i) >> 8) & 0xff)
 #define ELF64_MIPS_R_TYPE3(i)           (((i) >> 16) & 0xff)
+#define is_debug_section_type(type) (type == SHT_PROGBITS || type == SHT_MIPS_DWARF)
 #endif  /* libelfP.h */ > diff --git a/src/readelf.c b/src/readelf.c

index 0e931184..e88cf67c 100644

src/readelf.c > @@ -12043,7 +12139,7 @@ print_debug (Dwfl_Module *dwflmod, Ebl *ebl, GElf_Ehdr *ehdr) renamed from src/readelf.c +1 -1

 	  GElf_Shdr shdr_mem;
 	  GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_mem);
 
-	  if (shdr != NULL && shdr->sh_type == SHT_PROGBITS)
+	  if (shdr != NULL && is_debug_section_type(shdr->sh_type))
 	    {
 	      const char *name = elf_strptr (ebl->elf, shstrndx,
 					     shdr->sh_name);

src/readelf.c > @@ -12043,7 +12139,7 @@ print_debug (Dwfl_Module *dwflmod, Ebl *ebl, GElf_Ehdr *ehdr) print_debug (Dwfl_Module *dwflmod, Ebl *ebl, GElf_Ehdr *ehdr) · line 12175

Quoted 8 lines

src/readelf.c > @@ -12043,7 +12139,7 @@ print_debug (Dwfl_Module *dwflmod, Ebl *ebl, GElf_Ehdr *ehdr) renamed from src/readelf.c +1 -1

@@ -12073,7 +12169,7 @@ print_debug (Dwfl_Module *dwflmod, Ebl *ebl, GElf_Ehdr *ehdr)
       GElf_Shdr shdr_mem;
       GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_mem);
 
-      if (shdr != NULL && shdr->sh_type == SHT_PROGBITS)
+      if (shdr != NULL && is_debug_section_type(shdr->sh_type))
 	{
 	  static const struct
 	  {

Thanks,

Ying

Quoted 45 lines · Tony Ambardar

Hi Mark,
Quoted 13 lines · Mark Wielaard
Quoted 12 lines · Arnaldo Carvalho de Melo
Couldn't find a way to ask eu-readelf for more verbose output, where we could perhaps get some clue as to why it produces nothing while binutils readelf manages to grok it, Mark, do you know some other way to ask eu-readelf to produce more debug output?
I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state that then made eu-readelf to not be able to process it while pahole, that uses eltuils' libraries, was able to process the first two CUs for a kernel module and all the CUs for the vmlinux file :-\
Mark, the whole thread is available at:
Quoted 7 lines
Quoted 1 line
https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
I haven't looked at the vmlinux file. But for the .ko file the issue is that the elfutils MIPS backend isn't complete. Specifically MIPS relocations aren't recognized (and so cannot be applied). There are some pending patches which try to fix that:
https://patchwork.sourceware.org/project/elfutils/list/?series=31601
Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend work for MIPS, and I locally rebuilt elfutils and then pahole from their respective next/main branches. For elfutils, main (935ee131cf7c) includes
e259f126 Support Mips architecture
f2acb069 stack: Fix stack unwind failure on mips
db33cb0c backends: Add register_info, return_value_location, core_note mips
which partially applies the patchwork series but leaves out the support for readelf, strip, and elflint.
I believe this means the vmlinux and .ko files I shared are OK, or is there more backend work needed for MIPS?
The bits missing in eu-readelf would explain the blank output both Arnaldo and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the patchwork readelf patch locally but ran into merge conflicts.
CCing Ying Huang for any more insight.
Thanks,
Tony

Tony Ambardar

author Tested-by

2024-06-11 06:36 UTC #34 lore raw

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

In reply to Tony Ambardar

Quoted 39 lines · Tony Ambardar

Hi Mark,
Quoted 13 lines · Mark Wielaard
Quoted 12 lines · Arnaldo Carvalho de Melo
Couldn't find a way to ask eu-readelf for more verbose output, where we could perhaps get some clue as to why it produces nothing while binutils readelf manages to grok it, Mark, do you know some other way to ask eu-readelf to produce more debug output?
I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state that then made eu-readelf to not be able to process it while pahole, that uses eltuils' libraries, was able to process the first two CUs for a kernel module and all the CUs for the vmlinux file :-\
Mark, the whole thread is available at:
Quoted 8 lines
Quoted 1 line
https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
I haven't looked at the vmlinux file. But for the .ko file the issue is that the elfutils MIPS backend isn't complete. Specifically MIPS relocations aren't recognized (and so cannot be applied). There are some pending patches which try to fix that:
https://patchwork.sourceware.org/project/elfutils/list/?series=31601
Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend work for MIPS, and I locally rebuilt elfutils and then pahole from their respective next/main branches. For elfutils, main (935ee131cf7c) includes
e259f126 Support Mips architecture
f2acb069 stack: Fix stack unwind failure on mips
db33cb0c backends: Add register_info, return_value_location, core_note mips
which partially applies the patchwork series but leaves out the support for readelf, strip, and elflint.
I believe this means the vmlinux and .ko files I shared are OK, or is there more backend work needed for MIPS?

Quoted 8 lines

The bits missing in eu-readelf would explain the blank output both Arnaldo and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the patchwork readelf patch locally but ran into merge conflicts.
CCing Ying Huang for any more insight.
Thanks,
Tony

Hello all,

A short update, starting with answering my own question.

No, apparently the above commits *do not* complete the backend work. Ying Huang submitted additional related patches since March 5: [1][2]

strip: Adapt src/strip -o -f on mips
readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
elflint: adapt src/elflint --gnu src/nm on mips
test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh

Despite the titles, these patches do include core backend changes for MIPS. I resolved the various merge conflicts [3], rebuilt elfutils, and retested kernel builds to now find:

- pahole is able to read DWARF[45] info and create .BTF for modules
- resolve_btfids can successfully patch .BTF_ids in modules
- kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)

Huzzah!

Ying:

Thank you for developing these MIPS patches. In your view, are the MIPS changes now complete, or do you plan further updates that might improve or impact parsing DWARF debug/reloc info in apps like pahole?

Mark:

Given that BTF usage on Linux/MIPS is basically broken without these patches, could I request some of your review time for them to be merged? If it's helpful, my branch [3] includes all patches with conflicts fixed, and I also successfully ran the elfutils self-tests (including MIPS from Ying). Please feel free to add for these patches:

Tested-by Tony Ambardar <[email protected]>

Arnaldo:

Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks! I reran your tests with the updated elfutils and latest pahole (pre-1.27), and then found:

- everything that worked before, still works
- your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
- we now see expected output from "eu-readelf -winfo netdevsim.ko"

Regarding pahole, DWARF parsing and BTF generation now works:
(with no more die__process: error messages seen)

    kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
    14504

but strangely pahole still doesn't read its own generated BTF:

    kodidev:~/linux$ pahole -F btf netdevsim.ko
    libbpf: Invalid BTF string section
    pahole: file 'netdevsim.ko' has no btf type information.

Poking inside a little further:

    kodidev:~/linux$ ltrace -S pahole -F btf netdevsim.ko
    [...]
    argp_parse(0x563d47da42a0, 4, 0x7ffd5e552698, 0 <unfinished ...>
    SYS_318(0x7fc385bf84d8, 8, 1, 0x7fc385ce9908)    = 8
    SYS_brk(0)                                       = 0x563d47e37000
    SYS_brk(0x563d47e58000)                          = 0x563d47e58000
    <... argp_parse resumed> )                       = 0
    dwarves__init(0x20000, 0, -4096, 213)            = 0
    dwarves__resolve_cacheline_size(0x563d47da40c0, 0, 24, 213) = 64
    cus__new(5, 0, 64, 213)                          = 0x563d47e372a0
    memset(0x563d47da44c0, ' ', 127)                 = 0x563d47da44c0
    strlen("/sys/kernel/btf/")                       = 16
    strncmp("netdevsim.ko", "/sys/kernel/btf/", 16)  = 63
    cus__load_files(0x563d47e372a0, 0x563d47da40c0, 0x7ffd5e5526b0,
    0x563d47da40c0 <unfinished ...>
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e552210, 4096) = 0
    SYS_read(3, "\177ELF\002\001\001", 4096)         = 4096
    SYS_close(3)                                     = 0
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_fcntl(3, 1, 0, 0x7fc3859de2f0)               = 1
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e5521f0, 4096) = 0
    SYS_pread(3, 0x7ffd5e5521f0, 64, 0)              = 64
    SYS_pread(3, 0x563d47e3e470, 3520, 0x4f60c8)     = 3520
    SYS_pread(3, 0x563d47e3f240, 525, 0x4f5eb8)      = 525
    SYS_pread(3, 0x563d47e3f460, 0x45dd, 0x2b1fb6)   = 0x45dd
    SYS_write(2, "libbpf: Invalid BTF string secti"..., 35libbpf: Invalid BTF
    string section
    ) = 35
    SYS_close(3)                                     = 0
    <... cus__load_files resumed> )                  = 0xffffffff
    access("netdevsim.ko", 4 <unfinished ...>
    SYS_access("netdevsim.ko", 04)                   = 0
    <... access resumed> )                           = 0
    fprintf(0x7fc385bf26a0, "pahole: file '%s' has no %s type"...,
    "netdevsim.ko", "btf" <unfinished ...>
    SYS_write(2, "pahole: file 'netdevsim.ko' has "..., 57pahole: file
    'netdevsim.ko' has no btf type information.
    ) = 57
    <... fprintf resumed> )                          = 57
    SYS_exit_group(1 <no return ...>
    +++ exited (status 1) +++

Could you help investigate this further? Maybe a libbpf issue? For the record, I also tried building pahole with embedded libbpf 1.4.3 without any change. (side note: please make pahole --version also cover libbpf)

Many thanks everyone for your help,
Tony

[1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
[2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
[3]:
https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Tony Ambardar

author

2024-06-11 07:51 UTC #35 lore raw

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

Quoted 17 lines · Tony Ambardar

On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
[snip]
Arnaldo:
Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks! I reran your tests with the updated elfutils and latest pahole (pre-1.27), and then found:
- everything that worked before, still works
- your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
- we now see expected output from "eu-readelf -winfo netdevsim.ko"
Regarding pahole, DWARF parsing and BTF generation now works:
(with no more die__process: error messages seen)

Quoted 8 lines

    kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
    14504

but strangely pahole still doesn't read its own generated BTF:

    kodidev:~/linux$ pahole -F btf netdevsim.ko
    libbpf: Invalid BTF string section
    pahole: file 'netdevsim.ko' has no btf type information.

After staring at this I realized that this is split BTF and we needed to issue "pahole -F btf --btf_base vmlinux netdevsim.ko", which does work. Unfortunately, the libbpf error message is a bit cryptic; perhaps a future update could mention "--btf_base"?

Quoted 10 lines

[snip]
Many thanks everyone for your help,
Tony
[1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
[2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
[3]:
https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Mark Wielaard

2024-06-11 13:17 UTC #36 lore raw

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

In reply to Tony Ambardar

Hi,

Adding elfutils-devel to CC to keep everyone up to date on the state of the patches.

Quoted 72 lines · Tony Ambardar

Quoted 34 lines · Tony Ambardar
Quoted 13 lines · Mark Wielaard
Quoted 12 lines · Arnaldo Carvalho de Melo
Couldn't find a way to ask eu-readelf for more verbose output, where we could perhaps get some clue as to why it produces nothing while binutils readelf manages to grok it, Mark, do you know some other way to ask eu-readelf to produce more debug output?
I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state that then made eu-readelf to not be able to process it while pahole, that uses eltuils' libraries, was able to process the first two CUs for a kernel module and all the CUs for the vmlinux file :-\
Mark, the whole thread is available at:
Quoted 8 lines
Quoted 1 line
https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
I haven't looked at the vmlinux file. But for the .ko file the issue is that the elfutils MIPS backend isn't complete. Specifically MIPS relocations aren't recognized (and so cannot be applied). There are some pending patches which try to fix that:
https://patchwork.sourceware.org/project/elfutils/list/?series=31601
Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend work for MIPS, and I locally rebuilt elfutils and then pahole from their respective next/main branches. For elfutils, main (935ee131cf7c) includes
e259f126 Support Mips architecture
f2acb069 stack: Fix stack unwind failure on mips
db33cb0c backends: Add register_info, return_value_location, core_note mips
which partially applies the patchwork series but leaves out the support for readelf, strip, and elflint.
Quoted 6 lines
I believe this means the vmlinux and .ko files I shared are OK, or is there more backend work needed for MIPS?
The bits missing in eu-readelf would explain the blank output both Arnaldo and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the patchwork readelf patch locally but ran into merge conflicts.
A short update, starting with answering my own question.
No, apparently the above commits *do not* complete the backend work. Ying Huang submitted additional related patches since March 5: [1][2]
strip: Adapt src/strip -o -f on mips
readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
elflint: adapt src/elflint --gnu src/nm on mips
test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
Despite the titles, these patches do include core backend changes for MIPS. I resolved the various merge conflicts [3], rebuilt elfutils, and retested kernel builds to now find:
- pahole is able to read DWARF[45] info and create .BTF for modules
- resolve_btfids can successfully patch .BTF_ids in modules
- kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
Huzzah!
Ying:
Thank you for developing these MIPS patches. In your view, are the MIPS changes now complete, or do you plan further updates that might improve or impact parsing DWARF debug/reloc info in apps like pahole?
Mark:

Quoted 7 lines

Given that BTF usage on Linux/MIPS is basically broken without these patches, could I request some of your review time for them to be merged? If it's helpful, my branch [3] includes all patches with conflicts fixed, and I also successfully ran the elfutils self-tests (including MIPS from Ying). Please feel free to add for these patches:
Tested-by Tony Ambardar <[email protected]>

Yes, I would very much like to integrate the rest of these patches. But I keep running out of time. The main issues were that, as you noticed, the patches mix backend and frontend tool changes a bit. I don't have access to a MIPS system to test them on. There are a couple of different MIPS abis (I believe all combinations of 32/64 bit and big/little endianness), but people have only tested on mips64le (maybe that is the only relevant one these days?) And finally the way MIPS represents relocations is slightly different than any other ELF architecture does. So we have to translate that somewhere to make the standards functions work. I have to convince myself that doing that in elf_getdata as the patches do is the right place.

Quoted 7 lines

Many thanks everyone for your help,
Tony
[1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
[2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
[3]:
https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Tony Ambardar

author

2024-06-12 00:18 UTC #37 lore raw

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

Quoted 86 lines · Mark Wielaard

Hi,
Adding elfutils-devel to CC to keep everyone up to date on the state of the patches.
Quoted 72 lines · Tony Ambardar
Quoted 34 lines · Tony Ambardar
Quoted 13 lines · Mark Wielaard
Quoted 12 lines · Arnaldo Carvalho de Melo
Couldn't find a way to ask eu-readelf for more verbose output, where we could perhaps get some clue as to why it produces nothing while binutils readelf manages to grok it, Mark, do you know some other way to ask eu-readelf to produce more debug output?
I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state that then made eu-readelf to not be able to process it while pahole, that uses eltuils' libraries, was able to process the first two CUs for a kernel module and all the CUs for the vmlinux file :-\
Mark, the whole thread is available at:
Quoted 8 lines
Quoted 1 line
https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
I haven't looked at the vmlinux file. But for the .ko file the issue is that the elfutils MIPS backend isn't complete. Specifically MIPS relocations aren't recognized (and so cannot be applied). There are some pending patches which try to fix that:
https://patchwork.sourceware.org/project/elfutils/list/?series=31601
Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend work for MIPS, and I locally rebuilt elfutils and then pahole from their respective next/main branches. For elfutils, main (935ee131cf7c) includes
e259f126 Support Mips architecture
f2acb069 stack: Fix stack unwind failure on mips
db33cb0c backends: Add register_info, return_value_location, core_note mips
which partially applies the patchwork series but leaves out the support for readelf, strip, and elflint.
Quoted 6 lines
I believe this means the vmlinux and .ko files I shared are OK, or is there more backend work needed for MIPS?
The bits missing in eu-readelf would explain the blank output both Arnaldo and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the patchwork readelf patch locally but ran into merge conflicts.
A short update, starting with answering my own question.
No, apparently the above commits *do not* complete the backend work. Ying Huang submitted additional related patches since March 5: [1][2]
strip: Adapt src/strip -o -f on mips
readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
elflint: adapt src/elflint --gnu src/nm on mips
test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
Despite the titles, these patches do include core backend changes for MIPS. I resolved the various merge conflicts [3], rebuilt elfutils, and retested kernel builds to now find:
- pahole is able to read DWARF[45] info and create .BTF for modules
- resolve_btfids can successfully patch .BTF_ids in modules
- kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
Huzzah!
Ying:
Thank you for developing these MIPS patches. In your view, are the MIPS changes now complete, or do you plan further updates that might improve or impact parsing DWARF debug/reloc info in apps like pahole?
Mark:
Quoted 7 lines
Given that BTF usage on Linux/MIPS is basically broken without these patches, could I request some of your review time for them to be merged? If it's helpful, my branch [3] includes all patches with conflicts fixed, and I also successfully ran the elfutils self-tests (including MIPS from Ying). Please feel free to add for these patches:
Tested-by Tony Ambardar <[email protected]>

Quoted 12 lines

Yes, I would very much like to integrate the rest of these patches. But I keep running out of time. The main issues were that, as you noticed, the patches mix backend and frontend tool changes a bit. I don't have access to a MIPS system to test them on. There are a couple of different MIPS abis (I believe all combinations of 32/64 bit and big/little endianness), but people have only tested on mips64le (maybe that is the only relevant one these days?) And finally the way MIPS represents relocations is slightly different than any other ELF architecture does. So we have to translate that somewhere to make the standards functions work. I have to convince myself that doing that in elf_getdata as the patches do is the right place.

Glad to hear there's strong interest. Not much I can add to the code structure discussion but I *can* confirm my testing included both mips64el and mips32be, specifically to improve word-size/endianness coverage. The former is supported by Debian, while the latter can still be found on many embedded devices like consumer routers. If you need additional testing for review/merge I can help with that (using OpenWrt/QEMU primarily).

Quoted 8 lines

Quoted 7 lines
Many thanks everyone for your help,
Tony
[1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
[2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
[3]:
https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Ying Huang

2024-06-12 03:43 UTC #38 lore raw

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

In reply to Mark Wielaard

Hi Mark,

Regarding the current questions, I have a few points that needed to be explained.

Quoted 81 lines · Mark Wielaard

Hi,
Adding elfutils-devel to CC to keep everyone up to date on the state of the patches.
Quoted 75 lines · Tony Ambardar
Quoted 32 lines · Tony Ambardar
Quoted 13 lines · Mark Wielaard
Quoted 12 lines · Arnaldo Carvalho de Melo
Couldn't find a way to ask eu-readelf for more verbose output, where we could perhaps get some clue as to why it produces nothing while binutils readelf manages to grok it, Mark, do you know some other way to ask eu-readelf to produce more debug output?
I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state that then made eu-readelf to not be able to process it while pahole, that uses eltuils' libraries, was able to process the first two CUs for a kernel module and all the CUs for the vmlinux file :-\
Mark, the whole thread is available at:
Quoted 7 lines
Quoted 1 line
https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
I haven't looked at the vmlinux file. But for the .ko file the issue is that the elfutils MIPS backend isn't complete. Specifically MIPS relocations aren't recognized (and so cannot be applied). There are some pending patches which try to fix that:
https://patchwork.sourceware.org/project/elfutils/list/?series=31601
Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend work for MIPS, and I locally rebuilt elfutils and then pahole from their respective next/main branches. For elfutils, main (935ee131cf7c) includes
e259f126 Support Mips architecture
f2acb069 stack: Fix stack unwind failure on mips
db33cb0c backends: Add register_info, return_value_location, core_note mips
which partially applies the patchwork series but leaves out the support for readelf, strip, and elflint.
Quoted 6 lines
I believe this means the vmlinux and .ko files I shared are OK, or is there more backend work needed for MIPS?
The bits missing in eu-readelf would explain the blank output both Arnaldo and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the patchwork readelf patch locally but ran into merge conflicts.
A short update, starting with answering my own question.
No, apparently the above commits *do not* complete the backend work. Ying Huang submitted additional related patches since March 5: [1][2]
strip: Adapt src/strip -o -f on mips
readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
elflint: adapt src/elflint --gnu src/nm on mips
test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
Despite the titles, these patches do include core backend changes for MIPS. I resolved the various merge conflicts [3], rebuilt elfutils, and retested kernel builds to now find:
- pahole is able to read DWARF[45] info and create .BTF for modules
- resolve_btfids can successfully patch .BTF_ids in modules
- kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
Huzzah!
Ying:
Thank you for developing these MIPS patches. In your view, are the MIPS changes now complete, or do you plan further updates that might improve or impact parsing DWARF debug/reloc info in apps like pahole?
Mark:
Given that BTF usage on Linux/MIPS is basically broken without these patches, could I request some of your review time for them to be merged? If it's helpful, my branch [3] includes all patches with conflicts fixed, and I also successfully ran the elfutils self-tests (including MIPS from Ying). Please feel free to add for these patches:

Quoted 4 lines

Quoted 1 line
Tested-by Tony Ambardar <[email protected]>
Yes, I would very much like to integrate the rest of these patches. But I keep running out of time. The main issues were that, as you noticed, the patches mix backend and frontend tool changes a bit.

The reason about the mixture and title is that only by fixing the acquisition of relocation information can 'strip' and 'readelf -w' work normally.

Now it seems really confusing, so I want to do some changes based on your opinions, change the titles or reorganize these patches to make them look more logical.

Quoted 5 lines

I don't have
access to a MIPS system to test them on. There are a couple of different MIPS abis (I believe all combinations of 32/64 bit and big/little endianness), but people have only tested on mips64le (maybe that is the only relevant one these days?)

I have tested other mips abi before, I will test it agagin and attach the results of 'make check' with and without patch.

And add a description of mips abi support in the commit message.

Quoted 5 lines

And finally the way MIPS
represents relocations is slightly different than any other ELF architecture does. So we have to translate that somewhere to make the standards functions work. I have to convince myself that doing that in elf_getdata as the patches do is the right place.

The controversial problem was the location of the code, the code was to change the original relocation info to ensure mips can obtain the correct index value and symble index.

Or we did not modify the original data, we modify the way to obtain the index value and symble index at funtion 'gelf_getrela' in file 'libelf/gelf_getrela.c','libelf/gelf_getrel.c','libelf/gelf_update_rela.c','libelf/gelf_update_rel.c'.

Where the function 'gelf_getrela' is called，modify the relocation info that has been obtained .

What do you think of this?

Thanks,

Ying

Quoted 8 lines

Quoted 8 lines
Many thanks everyone for your help,
Tony
[1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
[2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
[3]:
https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Ying Huang

2024-06-12 03:24 UTC #39 lore raw

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

In reply to Tony Ambardar

Hi Tony,

Thanks for your fix about the conflict and test about the whole patch.

Now, there is currently one test case that has not been added, which is

tests/run-backtrace-core-mips.sh.
Thanks,
Ying

Quoted 166 lines · Tony Ambardar

Quoted 37 lines · Tony Ambardar
Hi Mark,
Quoted 13 lines · Mark Wielaard
Quoted 12 lines · Arnaldo Carvalho de Melo
Couldn't find a way to ask eu-readelf for more verbose output, where we could perhaps get some clue as to why it produces nothing while binutils readelf manages to grok it, Mark, do you know some other way to ask eu-readelf to produce more debug output?
I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state that then made eu-readelf to not be able to process it while pahole, that uses eltuils' libraries, was able to process the first two CUs for a kernel module and all the CUs for the vmlinux file :-\
Mark, the whole thread is available at:
Quoted 7 lines
Quoted 1 line
https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
I haven't looked at the vmlinux file. But for the .ko file the issue is that the elfutils MIPS backend isn't complete. Specifically MIPS relocations aren't recognized (and so cannot be applied). There are some pending patches which try to fix that:
https://patchwork.sourceware.org/project/elfutils/list/?series=31601
Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend work for MIPS, and I locally rebuilt elfutils and then pahole from their respective next/main branches. For elfutils, main (935ee131cf7c) includes
e259f126 Support Mips architecture
f2acb069 stack: Fix stack unwind failure on mips
db33cb0c backends: Add register_info, return_value_location, core_note mips
which partially applies the patchwork series but leaves out the support for readelf, strip, and elflint.
I believe this means the vmlinux and .ko files I shared are OK, or is there more backend work needed for MIPS?
Quoted 8 lines
The bits missing in eu-readelf would explain the blank output both Arnaldo and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the patchwork readelf patch locally but ran into merge conflicts.
CCing Ying Huang for any more insight.
Thanks,
Tony
Hello all,
A short update, starting with answering my own question.
No, apparently the above commits *do not* complete the backend work. Ying Huang submitted additional related patches since March 5: [1][2]
strip: Adapt src/strip -o -f on mips
readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
elflint: adapt src/elflint --gnu src/nm on mips
test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
Despite the titles, these patches do include core backend changes for MIPS. I resolved the various merge conflicts [3], rebuilt elfutils, and retested kernel builds to now find:
- pahole is able to read DWARF[45] info and create .BTF for modules
- resolve_btfids can successfully patch .BTF_ids in modules
- kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
Huzzah!
Ying:
Thank you for developing these MIPS patches. In your view, are the MIPS changes now complete, or do you plan further updates that might improve or impact parsing DWARF debug/reloc info in apps like pahole?
Mark:
Given that BTF usage on Linux/MIPS is basically broken without these patches, could I request some of your review time for them to be merged? If it's helpful, my branch [3] includes all patches with conflicts fixed, and I also successfully ran the elfutils self-tests (including MIPS from Ying). Please feel free to add for these patches:
Tested-by Tony Ambardar <[email protected]>
Arnaldo:
Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks! I reran your tests with the updated elfutils and latest pahole (pre-1.27), and then found:
- everything that worked before, still works
- your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
- we now see expected output from "eu-readelf -winfo netdevsim.ko"
Regarding pahole, DWARF parsing and BTF generation now works:
(with no more die__process: error messages seen)
    kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
    14504
but strangely pahole still doesn't read its own generated BTF:
    kodidev:~/linux$ pahole -F btf netdevsim.ko
    libbpf: Invalid BTF string section
    pahole: file 'netdevsim.ko' has no btf type information.
Poking inside a little further:
    kodidev:~/linux$ ltrace -S pahole -F btf netdevsim.ko
    [...]
    argp_parse(0x563d47da42a0, 4, 0x7ffd5e552698, 0 <unfinished ...>
    SYS_318(0x7fc385bf84d8, 8, 1, 0x7fc385ce9908)    = 8
    SYS_brk(0)                                       = 0x563d47e37000
    SYS_brk(0x563d47e58000)                          = 0x563d47e58000
    <... argp_parse resumed> )                       = 0
    dwarves__init(0x20000, 0, -4096, 213)            = 0
    dwarves__resolve_cacheline_size(0x563d47da40c0, 0, 24, 213) = 64
    cus__new(5, 0, 64, 213)                          = 0x563d47e372a0
    memset(0x563d47da44c0, ' ', 127)                 = 0x563d47da44c0
    strlen("/sys/kernel/btf/")                       = 16
    strncmp("netdevsim.ko", "/sys/kernel/btf/", 16)  = 63
    cus__load_files(0x563d47e372a0, 0x563d47da40c0, 0x7ffd5e5526b0,
    0x563d47da40c0 <unfinished ...>
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e552210, 4096) = 0
    SYS_read(3, "\177ELF\002\001\001", 4096)         = 4096
    SYS_close(3)                                     = 0
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_fcntl(3, 1, 0, 0x7fc3859de2f0)               = 1
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e5521f0, 4096) = 0
    SYS_pread(3, 0x7ffd5e5521f0, 64, 0)              = 64
    SYS_pread(3, 0x563d47e3e470, 3520, 0x4f60c8)     = 3520
    SYS_pread(3, 0x563d47e3f240, 525, 0x4f5eb8)      = 525
    SYS_pread(3, 0x563d47e3f460, 0x45dd, 0x2b1fb6)   = 0x45dd
    SYS_write(2, "libbpf: Invalid BTF string secti"..., 35libbpf: Invalid BTF
    string section
    ) = 35
    SYS_close(3)                                     = 0
    <... cus__load_files resumed> )                  = 0xffffffff
    access("netdevsim.ko", 4 <unfinished ...>
    SYS_access("netdevsim.ko", 04)                   = 0
    <... access resumed> )                           = 0
    fprintf(0x7fc385bf26a0, "pahole: file '%s' has no %s type"...,
    "netdevsim.ko", "btf" <unfinished ...>
    SYS_write(2, "pahole: file 'netdevsim.ko' has "..., 57pahole: file
    'netdevsim.ko' has no btf type information.
    ) = 57
    <... fprintf resumed> )                          = 57
    SYS_exit_group(1 <no return ...>
    +++ exited (status 1) +++
Could you help investigate this further? Maybe a libbpf issue? For the record, I also tried building pahole with embedded libbpf 1.4.3 without any change. (side note: please make pahole --version also cover libbpf)
Many thanks everyone for your help,
Tony
[1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
[2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
[3]:
https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/