bpf: Question about odd BPF verifier behaviour
Hello!
Whilst in the midst of testing a v5.19 to v6.1 kernel upgrade, we happened to notice that one of our sleepable LSM based eBPF programs was failing to load on the newer v6.1 kernel. Using the below trivial eBPF program as our reproducer:
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
char LICENSE[] SEC("license") = "Dual BSD/GPL";
SEC("lsm.s/bprm_committed_creds")
int BPF_PROG(dbg, struct linux_binprm *bprm)
{
char buf[64] = {0};
bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
return 0;
}The verifier emits the following error message when attempting to load the above eBPF program:
-- BEGIN PROG LOAD LOG --
reg type unsupported for arg#0 function dbg#5
0: R1=ctx(off=0,imm=0) R10=fp0
; int BPF_PROG(dbg, struct linux_binprm *bprm)
0: (79) r1 = *(u64 *)(r1 +0)
func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 137293 type STRUCT 'linux_binprm' 1: R1_w=ptr_linux_binprm(off=0,imm=0)
1: (b7) r2 = 0 ; R2_w=0
; char buf[64] = {0};
[...]
; bpf_ima_file_hash(bprm->file, buf, 64);
10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
12: (07) r2 += -64 ; R2_w=fp-64
; bpf_ima_file_hash(bprm->file, buf, 64);
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1 R1 is of type file but file is expected
processed 15 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0 -- END PROG LOAD LOG --
What particularly strikes out at me is the following 2 lines returned in the error message:
cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1 R1 is of type file but file is expected
In this particular case, the above message suggested to me that there's likely multiple struct file definitions that exist within the kernel's BTF and that the verifier is possibly getting confused about which one it should be using, or perhaps some of the struct file definitions included in the kernel's BTF actually differ and hence when performing the btf_struct_ids_match() check in check_reg_type()
[0] the verifier fails with this error message? Could this potentially be a problem with the toolchain (Currently, using latest pahole/LLVM built from source)?
Additionally, I also noticed that when we walk the BTF struct defintions via btf_struct_walk() from btf_struct_ids_match(), the size passed to btf_struct_walk() is explicitly set to 1. Yet, msize used throughout btf_struct_walk() can certainly be > 1 when evaluating a struct defintions members and hence why we're also tripping over this condition [1] in btf_struct_walk(). Don't completely understaed this code yet, so I don't know whether this is actually a problem or not.
Keen to here what your thoughts are on this one.
[0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/bpf/verifier.c#n6278
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/bpf/btf.c#n6237
/M
Quoted 71 lines · Matt Bobrowski
Hello!
Whilst in the midst of testing a v5.19 to v6.1 kernel upgrade, we happened to notice that one of our sleepable LSM based eBPF programs was failing to load on the newer v6.1 kernel. Using the below trivial eBPF program as our reproducer:
#include "
vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>char LICENSE[]
SEC("license")= "Dual BSD/GPL";
SEC("lsm.s/bprm_committed_creds")
intBPF_PROG(dbg, struct linux_binprm *bprm){ char buf[64] = {0}; bpf_ima_file_hash(bprm->file, buf, sizeof(buf)); return 0; }The verifier emits the following error message when attempting to load the above eBPF program:
-- BEGIN PROG LOAD LOG --
reg type unsupported for arg#0 function dbg#5
0: R1=ctx(off=0,imm=0) R10=fp0
; intBPF_PROG(dbg, struct linux_binprm *bprm)
0: (79) r1 = *(u64 *)(r1 +0)
func 'bpf_lsm_bprm_committed_creds' arg0 hasbtf_id137293 type STRUCT 'linux_binprm' 1: R1_w=ptr_linux_binprm(off=0,imm=0)
1: (b7) r2 = 0 ; R2_w=0
; char buf[64] = {0};
[...]
;bpf_ima_file_hash(bprm->file, buf, 64);
10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
12: (07) r2 += -64 ; R2_w=fp-64
;bpf_ima_file_hash(bprm->file, buf, 64);
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in structllist_nodewith off 0 size 1 R1 is of type file but file is expected
processed 15 insns (limit 1000000)max_states_per_insn0total_states0peak_states0mark_read0 -- END PROG LOAD LOG --What particularly strikes out at me is the following 2 lines returned in the error message:
cannot access ptr member next with moff 0 in struct
llist_nodewith off 0 size 1 R1 is of type file but file is expectedIn this particular case, the above message suggested to me that there's likely multiple struct file definitions that exist within the kernel's BTF and that the verifier is possibly getting confused about which one it should be using, or perhaps some of the struct file definitions included in the kernel's BTF actually differ and hence when performing the
btf_struct_ids_match()check incheck_reg_type()
[0] the verifier fails with this error message? Could this potentially be a problem with the toolchain (Currently, using latest pahole/LLVM built from source)?Additionally, I also noticed that when we walk the BTF struct defintions via
btf_struct_walk()frombtf_struct_ids_match(), the size passed tobtf_struct_walk()is explicitly set to 1. Yet, msize used throughoutbtf_struct_walk()can certainly be > 1 when evaluating a struct defintions members and hence why we're also tripping over this condition [1] inbtf_struct_walk(). Don't completely understaed this code yet, so I don't know whether this is actually a problem or not.
Quoted 1 line
Keen to here what your thoughts are on this one.
Note that I'm using the latest pahole [0] and LLVM [1] when building things here.
Andrii/Arnaldo, do you happen to have any pointers on debugging this BTF ID redundancy, which I suspect is what's going on here?
[0] https://git.kernel.org/pub/scm/devel/pahole/pahole.git
[1] https://github.com/llvm/llvm-project.git
/M
Eduard Zingerman
maintainerQuoted 47 lines · Matt Bobrowski
Hello!
Whilst in the midst of testing a v5.19 to v6.1 kernel upgrade, we happened to notice that one of our sleepable LSM based eBPF programs was failing to load on the newer v6.1 kernel. Using the below trivial eBPF program as our reproducer:
#include "
vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>char LICENSE[]
SEC("license")= "Dual BSD/GPL";
SEC("lsm.s/bprm_committed_creds")
intBPF_PROG(dbg, struct linux_binprm *bprm){ char buf[64] = {0}; bpf_ima_file_hash(bprm->file, buf, sizeof(buf)); return 0; }The verifier emits the following error message when attempting to load the above eBPF program:
-- BEGIN PROG LOAD LOG --
reg type unsupported for arg#0 function dbg#5
0: R1=ctx(off=0,imm=0) R10=fp0
; intBPF_PROG(dbg, struct linux_binprm *bprm)
0: (79) r1 = *(u64 *)(r1 +0)
func 'bpf_lsm_bprm_committed_creds' arg0 hasbtf_id137293 type STRUCT 'linux_binprm' 1: R1_w=ptr_linux_binprm(off=0,imm=0)
1: (b7) r2 = 0 ; R2_w=0
; char buf[64] = {0};
[...]
;bpf_ima_file_hash(bprm->file, buf, 64);
10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
12: (07) r2 += -64 ; R2_w=fp-64
;bpf_ima_file_hash(bprm->file, buf, 64);
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in structllist_nodewith off 0 size 1 R1 is of type file but file is expected
processed 15 insns (limit 1000000)max_states_per_insn0total_states0peak_states0mark_read0 -- END PROG LOAD LOG --
Quoted 5 lines
What particularly strikes out at me is the following 2 lines returned in the error message:
cannot access ptr member next with moff 0 in struct
llist_nodewith off 0 size 1 R1 is of type file but file is expected
Hi Matt,
I tried your program as a ./test_progs test using v6.1 kernel and don't see any error messages:
VERIFIER LOG:
=============
func#0 @0
reg type unsupported for arg#0 function dbg#5
0: R1=ctx(off=0,imm=0) R10=fp0
; int BPF_PROG(dbg, struct linux_binprm *bprm)
0: (79) r1 = *(u64 *)(r1 +0)
func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 3061 type STRUCT 'linux_binprm' 1: R1_w=ptr_linux_binprm(off=0,imm=0)
1: (b7) r2 = 0 ; R2_w=0
; char buf[64] = {0};
2: (7b) *(u64 *)(r10 -8) = r2last_idx 2 first_idx 0
regs=4 stack=0 before 1: (b7) r2 = 0
3: R2_w=P0 R10=fp0 fp-8_w=00000000
3: (7b) *(u64 *)(r10 -16) = r2 ; R2_w=P0 R10=fp0 fp-16_w=00000000 4: (7b) *(u64 *)(r10 -24) = r2 ; R2_w=P0 R10=fp0 fp-24_w=00000000 5: (7b) *(u64 *)(r10 -32) = r2 ; R2_w=P0 R10=fp0 fp-32_w=00000000 6: (7b) *(u64 *)(r10 -40) = r2 ; R2_w=P0 R10=fp0 fp-40_w=00000000 7: (7b) *(u64 *)(r10 -48) = r2 ; R2_w=P0 R10=fp0 fp-48_w=00000000 8: (7b) *(u64 *)(r10 -56) = r2 ; R2_w=P0 R10=fp0 fp-56_w=00000000 9: (7b) *(u64 *)(r10 -64) = r2 ; R2_w=P0 R10=fp0 fp-64_w=00000000
; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
12: (07) r2 += -64 ; R2_w=fp-64
; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
13: (b4) w3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193last_idx 14 first_idx 0
regs=8 stack=0 before 13: (b4) w3 = 64
15: R0_w=scalar() fp-8_w=mmmmmmmm fp-16_w=mmmmmmmm fp-24_w=mmmmmmmm fp-32_w=mmmmmmmm fp-40_w=mmmmmmmm fp-48_w=mmmmmmmm fp-56_w=mmmmmmmm fp-64_w=mmmmmmmm ; int BPF_PROG(dbg, struct linux_binprm *bprm)
15: (b4) w0 = 0 ; R0_w=0
16: (95) exit
I use the following revision: 830b3c68c1fb "Linux 6.1". (also works with current bpf-next master).
Could you please provide some details on how you compile/load the program?
Thanks,
Eduard
[...]
Hey Eduard!
Quoted 96 lines · Eduard Zingerman
Quoted 47 lines · Matt Bobrowski
Hello!
Whilst in the midst of testing a v5.19 to v6.1 kernel upgrade, we happened to notice that one of our sleepable LSM based eBPF programs was failing to load on the newer v6.1 kernel. Using the below trivial eBPF program as our reproducer:
#include "
vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>char LICENSE[]
SEC("license")= "Dual BSD/GPL";
SEC("lsm.s/bprm_committed_creds")
intBPF_PROG(dbg, struct linux_binprm *bprm){ char buf[64] = {0}; bpf_ima_file_hash(bprm->file, buf, sizeof(buf)); return 0; }The verifier emits the following error message when attempting to load the above eBPF program:
-- BEGIN PROG LOAD LOG --
reg type unsupported for arg#0 function dbg#5
0: R1=ctx(off=0,imm=0) R10=fp0
; intBPF_PROG(dbg, struct linux_binprm *bprm)
0: (79) r1 = *(u64 *)(r1 +0)
func 'bpf_lsm_bprm_committed_creds' arg0 hasbtf_id137293 type STRUCT 'linux_binprm' 1: R1_w=ptr_linux_binprm(off=0,imm=0)
1: (b7) r2 = 0 ; R2_w=0
; char buf[64] = {0};
[...]
;bpf_ima_file_hash(bprm->file, buf, 64);
10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
12: (07) r2 += -64 ; R2_w=fp-64
;bpf_ima_file_hash(bprm->file, buf, 64);
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in structllist_nodewith off 0 size 1 R1 is of type file but file is expected
processed 15 insns (limit 1000000)max_states_per_insn0total_states0peak_states0mark_read0 -- END PROG LOAD LOG --Quoted 5 lines
What particularly strikes out at me is the following 2 lines returned in the error message:
cannot access ptr member next with moff 0 in struct
llist_nodewith off 0 size 1 R1 is of type file but file is expectedHi Matt,
I tried your program as a ./test_progs test using v6.1 kernel and don't see any error messages:
VERIFIER LOG:
=============
func#0 @0
reg type unsupported for arg#0 function dbg#5
0: R1=ctx(off=0,imm=0) R10=fp0
; intBPF_PROG(dbg, struct linux_binprm *bprm)
0: (79) r1 = *(u64 *)(r1 +0)
func 'bpf_lsm_bprm_committed_creds' arg0 hasbtf_id3061 type STRUCT 'linux_binprm' 1: R1_w=ptr_linux_binprm(off=0,imm=0)
1: (b7) r2 = 0 ; R2_w=0
; char buf[64] = {0};
2: (7b) *(u64 *)(r10 -8) = r2last_idx2first_idx0
regs=4 stack=0 before 1: (b7) r2 = 0
3: R2_w=P0 R10=fp0 fp-8_w=000000003: (7b) *(u64 *)(r10 -16) = r2 ; R2_w=P0 R10=fp0 fp-16_w=00000000 4: (7b) *(u64 *)(r10 -24) = r2 ; R2_w=P0 R10=fp0 fp-24_w=00000000 5: (7b) *(u64 *)(r10 -32) = r2 ; R2_w=P0 R10=fp0 fp-32_w=00000000 6: (7b) *(u64 *)(r10 -40) = r2 ; R2_w=P0 R10=fp0 fp-40_w=00000000 7: (7b) *(u64 *)(r10 -48) = r2 ; R2_w=P0 R10=fp0 fp-48_w=00000000 8: (7b) *(u64 *)(r10 -56) = r2 ; R2_w=P0 R10=fp0 fp-56_w=00000000 9: (7b) *(u64 *)(r10 -64) = r2 ; R2_w=P0 R10=fp0 fp-64_w=00000000;
bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
12: (07) r2 += -64 ; R2_w=fp-64
;bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
13: (b4) w3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193last_idx14first_idx0
regs=8 stack=0 before 13: (b4) w3 = 64
15: R0_w=scalar() fp-8_w=mmmmmmmm fp-16_w=mmmmmmmm fp-24_w=mmmmmmmm fp-32_w=mmmmmmmm fp-40_w=mmmmmmmm fp-48_w=mmmmmmmm fp-56_w=mmmmmmmm fp-64_w=mmmmmmmm ; intBPF_PROG(dbg, struct linux_binprm *bprm)
15: (b4) w0 = 0 ; R0_w=0
16: (95) exit
Quoted 4 lines
I use the following revision:
830b3c68c1fb"Linux 6.1". (also works with current bpf-next master).Could you please provide some details on how you compile/load the program?
Firstly, thanks for taking a peek at this! Secondly, I do apologies, I should've provided some more detailed on how I'm reproducing this in my initial email. Below you can find a transcript of how I'm conducting my tests:
The source OS which things (kernel and BPF reproducer program) are being built on:
$ cat /etc/os-release PRETTY_NAME="Debian GNU/Linux rodete" NAME="Debian GNU/Linux" VERSION_ID="rodete" VERSION="12 (rodete)" VERSION_CODENAME=rodete ID=debian
Building latest LLVM and Pahole from source:
$ sudo apt install -y cmake
$ cmake --version cmake version 3.25.1
$ git clone https://github.com/llvm/llvm-project.git && cd llvm-project && \ mkdir build && \ cd build && \ cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_BUILD_TYPE=Release -G "Unix Makefiles" ../llvm && \ make -j $(nproc) && \ sudo make install
$ clang --version clang version 17.0.0 (https://github.com/llvm/llvm-project.git bc85cf1687435f28fb01b1aa5303317e6118490c) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /usr/local/bin
$ sudo apt install -y libdwarf-dev libdw-dev
$ git clone git://git.kernel.org/pub/scm/devel/pahole/pahole.git && \ cd pahole && \ mkdir build && \ cd build && \ cmake -DCMAKE_INSTALL_PREFIX=/usr -D__LIB=lib .. && \ make -j $(nproc) && \ sudo make install
$ pahole --version v1.25
Building a test kernel:
$ git clone https://github.com/torvalds/linux.git && cd linux
$ make defconfig && make kvm_guest.config
$ scripts/config \ -e BPF \ -e BPF_SYSCALL \ -e BPF_LSM \ -e BPF_JIT \ -e BPF_EVENTS \ -e DEBUG_INFO \ -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \ -e DEBUG_INFO_BTF \ -e DEBUG_INFO_BTF_MODULES \ -e PAHOLE_HAS_SPLIT_BTF \ -e FTRACE \ -e DYNAMIC_FTRACE \ -e FUNCTION_TRACER
$ make olddefconfig
$ make -j`nproc`
Building the BPF reproducer program:
$ git clone https://github.com/libbpf/libbpf-bootstrap.git
# Both libbpf and bpftool should be the latest versions here.
$ git submodule update --init --recursive
$ cd examples/c
$ cat > fentry.bpf.c<<EOF #include "vmlinux.h" #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h>
char LICENSE[] SEC("license") = "Dual BSD/GPL";
SEC("lsm.s/bprm_committed_creds")
int BPF_PROG(dbg, struct linux_binprm *bprm)
{
char buf[64] = {0};
bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
return 0;
}
EOF
$ make -j`nproc` fentry
At this point, I basically launch the built kernel using QEMU and push the built 'fentry' BPF program to the VM and run it. At that point, I face the BPF verifier issue.
LMK whether you need any more information.
/M
Eduard Zingerman
maintainerOn Thu, 2023-02-23 at 09:37 +0000, Matt Bobrowski wrote:
[...]
Quoted 3 lines
LMK whether you need any more information.
/M
Hi Matt,
Unfortunately I can't reproduce the issue. Here are the versions of the tools/repos that I use:
- kernel (tried both):
- https://github.com/torvalds/linux.git
a5c95ca18a98 ("Merge tag 'drm-next-2023-02-23' of git://anongit.freedesktop.org/drm/drm")
- https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git
830b3c68c1fb ("Linux 6.1")- config (tried both):
- one obtained using your instructions
- my small debug config for executing BPF tests ([1])
- LLVM:
https://github.com/llvm/llvm-project.git
bc85cf168743 ("[TextAPI] Add support for TBDv5 Files to nm & tapi-diff")
- pahole:
[email protected]:acmel/dwarves.git
ef68019 ("pahole: Update man page for options also")
- libbpf-bootstrap (just followed your instructions):
https://github.com/libbpf/libbpf-bootstrap
db4f7ad ("cmake: Fix btf header missing in legacy kernel env.")
- gcc (from my distro):
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
- cat /etc/os-release
NAME="Linux Mint"
VERSION="21.1 (Vera)"
ID=linuxmint
ID_LIKE="ubuntu debian"
PRETTY_NAME="Linux Mint 21.1"
VERSION_ID="21.1"
VERSION_CODENAME=vera
UBUNTU_CODENAME=jammy
Could you please copy-paste output of the fentry application, I'd like to see the log output of the libbpf while it processes relocations, e.g. here is what it prints for me:
# /home/eddy/work/libbpf-bootstrap/examples/c/fentry
libbpf: loading object 'fentry_bpf' from buffer
libbpf: elf: section(3) lsm.s/bprm_committed_creds, size 136, link 0, flags 6, type=1
libbpf: sec 'lsm.s/bprm_committed_creds': found program 'dbg' at insn offset 0 (0 bytes), code size 17 insns (136 bytes)
libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1
libbpf: license of fentry_bpf is Dual BSD/GPL
libbpf: elf: section(5) .BTF, size 5114, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 188, link 0, flags 0, type=1
libbpf: elf: section(10) .symtab, size 96, link 1, flags 0, type=2
libbpf: looking for externs among 4 symbols...
libbpf: collected 0 externs total
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: sec 'lsm.s/bprm_committed_creds': found 1 CO-RE relocations
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [7241] struct linux_binprm in [vmlinux]
libbpf: prog 'dbg': relo #0: <byte_off> [6] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #0 <byte_off> [7241] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: patched insn #10 (LDX/ST/STX) off 64 -> 64
Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` to see output of the BPF programs.Also, could you please compile veristat tool as below:
cd ${kernel}/tools/testing/selftests/bpf
make -j16 veristat
And post the output of the following command (from within QEMU):
./veristat -l7 -v ${path-to-libbpf-bootstrap-within-vm}/examples/c/.output/fentry.bpf.oIt should produce the verification log as an output.
The reason I'm asking is that your verification log looks kinda strange:
Quoted 6 lines
;
bpf_ima_file_hash(bprm->file, buf, 64);
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in structllist_nodewith off 0 size 1 R1 is of type file but file is expected
processed 15 insns (limit 1000000)max_states_per_insn0total_states0peak_states0mark_read0
I don't understand why it mentions struct llist_node here and don't have such messages in my log ([2]).
Thanks,
Eduard
[1] My config for BPF testing
https://gist.github.com/eddyz87/aca79692d7bf57cfdd01b283b4304fd8 [2] Veristat verification log https://gist.github.com/eddyz87/49b211740bf99c426a37a3555b4542a3
Eduard Zingerman
maintainerOn Thu, 2023-02-23 at 14:42 +0200, Eduard Zingerman wrote:
[...]
Quoted 3 lines
- pahole:
[email protected]:acmel/dwarves.git
ef68019 ("pahole: Update man page for options also")
[...]
Quoted 2 lines
- gcc (from my distro):
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
For the sake of completeness I tried pahole master instead of next:
431df45 ("btfdiff: Exclude Rust CUs since those are not
yet being converted to BTF on the Linux kernel")
And gcc 12 for kernel build (looks like its the gcc version for Debian 12). The example still works.
¯\_(ツ)_/¯
Quoted 54 lines · Eduard Zingerman
On Thu, 2023-02-23 at 09:37 +0000, Matt Bobrowski wrote:
[...]Quoted 3 lines
LMK whether you need any more information.
/M
Hi Matt,
Unfortunately I can't reproduce the issue. Here are the versions of the
tools/reposthat I use:- kernel (tried both):
- https://github.com/torvalds/linux.gita5c95ca18a98 ("Merge tag 'drm-next-2023-02-23' of git://anongit.freedesktop.org/drm/drm") - https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git 830b3c68c1fb ("Linux 6.1")- config (tried both):
- one obtained using your instructions
- my small debug config for executing BPF tests ([1])
- LLVM:
https://github.com/llvm/llvm-project.git
bc85cf168743("[TextAPI] Add support for TBDv5 Files to nm & tapi-diff")
- pahole:
[email protected]:acmel/dwarves.git
ef68019 ("pahole: Update man page for options also")
- libbpf-bootstrap (just followed your instructions):
https://github.com/libbpf/libbpf-bootstrap
db4f7ad ("cmake: Fix btf header missing in legacy kernel env.")
- gcc (from my distro):
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
- cat/etc/os-release
NAME="Linux Mint"
VERSION="21.1 (Vera)"
ID=linuxmint
ID_LIKE="ubuntu debian"
PRETTY_NAME="Linux Mint 21.1"
VERSION_ID="21.1"
VERSION_CODENAME=vera
UBUNTU_CODENAME=jammyCould you please copy-paste output of the
fentryapplication, I'd like to see the log output of the libbpf while it processes relocations, e.g. here is what it prints for me:# /home/eddy/work/libbpf-bootstrap/examples/c/fentry libbpf: loading object 'fentry_bpf' from buffer libbpf: elf: section(3) lsm.s/bprm_committed_creds, size 136, link 0, flags 6, type=1 libbpf: sec 'lsm.s/bprm_committed_creds': found program 'dbg' at insn offset 0 (0 bytes), code size 17 insns (136 bytes) libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1 libbpf: license of fentry_bpf is Dual BSD/GPL libbpf: elf: section(5) .BTF, size 5114, link 0, flags 0, type=1 libbpf: elf: section(7) .BTF.ext, size 188, link 0, flags 0, type=1 libbpf: elf: section(10) .symtab, size 96, link 1, flags 0, type=2 libbpf: looking for externs among 4 symbols...
Quoted 8 lines
libbpf: collected 0 externs total
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: sec 'lsm.s/bprm_committed_creds': found 1 CO-RE relocations
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [7241] struct linux_binprm in [vmlinux]
libbpf: prog 'dbg': relo #0: <byte_off> [6] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #0 <byte_off> [7241] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: patched insn #10 (LDX/ST/STX) off 64 -> 64
Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` to see output of the BPF programs.Sure, here it is:
# ./fentry
libbpf: loading object 'fentry_bpf' from buffer
libbpf: elf: section(3) lsm.s/bprm_committed_creds, size 136, link 0, flags 6, type=1
libbpf: sec 'lsm.s/bprm_committed_creds': found program 'dbg' at insn offset 0 (0 bytes), code size 17 insns (136 bytes)
libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1
libbpf: license of fentry_bpf is Dual BSD/GPL
libbpf: elf: section(5) .BTF, size 5149, link 0, flags 0, type=1
libbpf: elf: section(7) .BTF.ext, size 188, link 0, flags 0, type=1
libbpf: elf: section(10) .symtab, size 96, link 1, flags 0, type=2
libbpf: looking for externs among 4 symbols...
libbpf: collected 0 externs total
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: sec 'lsm.s/bprm_committed_creds': found 1 CO-RE relocations
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [3971] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [9214] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [36310] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [36973] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [122219] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [151720] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [163602] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [175117] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [176645] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [179130] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [189263] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [237046] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [240978] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [264207] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [268773] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [276058] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [295158] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [306160] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [347067] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [349932] struct linux_binprm in [vmlinux]
libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [380629] struct linux_binprm in [vmlinux]
libbpf: prog 'dbg': relo #0: <byte_off> [6] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #0 <byte_off> [3971] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #1 <byte_off> [9214] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #2 <byte_off> [36310] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #3 <byte_off> [36973] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #4 <byte_off> [122219] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #5 <byte_off> [151720] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #6 <byte_off> [163602] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #7 <byte_off> [175117] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #8 <byte_off> [176645] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #9 <byte_off> [179130] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #10 <byte_off> [189263] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #11 <byte_off> [237046] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #12 <byte_off> [240978] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #13 <byte_off> [264207] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #14 <byte_off> [268773] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #15 <byte_off> [276058] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #16 <byte_off> [295158] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #17 <byte_off> [306160] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #18 <byte_off> [347067] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #19 <byte_off> [349932] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: matching candidate #20 <byte_off> [380629] struct linux_binprm.file (0:11 @ offset 64)
libbpf: prog 'dbg': relo #0: patched insn #10 (LDX/ST/STX) off 64 -> 64
libbpf: prog 'dbg': BPF program load failed: Permission denied
libbpf: prog 'dbg': -- BEGIN PROG LOAD LOG --
reg type unsupported for arg#0 function dbg#5
0: R1=ctx(off=0,imm=0) R10=fp0
; int BPF_PROG(dbg, struct linux_binprm *bprm)
0: (79) r1 = *(u64 *)(r1 +0)
func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 176645 type STRUCT 'linux_binprm'
1: R1_w=trusted_ptr_linux_binprm(off=0,imm=0)
1: (b7) r2 = 0 ; R2_w=0
; char buf[64] = {0};
2: (7b) *(u64 *)(r10 -8) = r2 ; R2_w=0 R10=fp0 fp-8_w=00000000
3: (7b) *(u64 *)(r10 -16) = r2 ; R2_w=0 R10=fp0 fp-16_w=00000000
4: (7b) *(u64 *)(r10 -24) = r2 ; R2_w=0 R10=fp0 fp-24_w=00000000
5: (7b) *(u64 *)(r10 -32) = r2 ; R2_w=0 R10=fp0 fp-32_w=00000000
6: (7b) *(u64 *)(r10 -40) = r2 ; R2_w=0 R10=fp0 fp-40_w=00000000
7: (7b) *(u64 *)(r10 -48) = r2 ; R2_w=0 R10=fp0 fp-48_w=00000000
8: (7b) *(u64 *)(r10 -56) = r2 ; R2_w=0 R10=fp0 fp-56_w=00000000
9: (7b) *(u64 *)(r10 -64) = r2 ; R2_w=0 R10=fp0 fp-64_w=00000000
; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
12: (07) r2 += -64 ; R2_w=fp-64
; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1
R1 is of type file but file is expected
processed 15 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'dbg': failed to load: -13
libbpf: failed to load object 'fentry_bpf'
libbpf: failed to load BPF skeleton 'fentry_bpf': -13
Failed to open BPF skeletonIt looks like there are a lot more relocations attempted by libbpf, but I suspect that's a result of their being multiple definitions of that same struct within the running kernel's BTF?
Quoted 20 lines
Also, could you please compile
veristattool as below:cd ${kernel}/tools/testing/selftests/bpf
make -j16 veristatAnd post the output of the following command (from within QEMU):
./veristat -l7 -v ${path-to-libbpf-bootstrap-within-vm}/examples/c/.output/fentry.bpf.oIt should produce the verification log as an output.
The reason I'm asking is that your verification log looks kinda strange:
Quoted 6 lines
;
bpf_ima_file_hash(bprm->file, buf, 64);
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in structllist_nodewith off 0 size 1 R1 is of type file but file is expected
processed 15 insns (limit 1000000)max_states_per_insn0total_states0peak_states0mark_read0
Quoted 2 lines
I don't understand why it mentions
struct llist_nodehere and don't have such messages in my log ([2]).
Yes, I also found this strange and couldn't find a valid explanation for it either. Looking at the BPF verifier code in the kernel, we hit this case when performing the struct member walk in btf_struct_walk().
Here is the output from the veristat utility:
./veristat -l7 -v ./fentry.bpf.o
Processing 'fentry.bpf.o'...
libbpf: prog 'dbg': BPF program load failed: Permission denied
libbpf: prog 'dbg': failed to load: -13
libbpf: failed to load object './fentry.bpf.o'
PROCESSING ./fentry.bpf.o/dbg, DURATION US: 4903, VERDICT: failure, VERIFIER LOG:
func#0 @0
reg type unsupported for arg#0 function dbg#5
0: R1=ctx(off=0,imm=0) R10=fp0
; int BPF_PROG(dbg, struct linux_binprm *bprm)
0: (79) r1 = *(u64 *)(r1 +0)
func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 176645 type STRUCT 'linux_binprm' 1: R1_w=trusted_ptr_linux_binprm(off=0,imm=0)
1: (b7) r2 = 0 ; R2_w=0
; char buf[64] = {0};
2: (7b) *(u64 *)(r10 -8) = r2last_idx 2 first_idx 0
regs=4 stack=0 before 1: (b7) r2 = 0
3: R2_w=0 R10=fp0 fp-8_w=00000000
3: (7b) *(u64 *)(r10 -16) = r2last_idx 3 first_idx 0
regs=4 stack=0 before 2: (7b) *(u64 *)(r10 -8) = r2
regs=4 stack=0 before 1: (b7) r2 = 0
4: R2_w=0 R10=fp0 fp-16_w=00000000
4: (7b) *(u64 *)(r10 -24) = r2last_idx 4 first_idx 0
regs=4 stack=0 before 3: (7b) *(u64 *)(r10 -16) = r2
regs=4 stack=0 before 2: (7b) *(u64 *)(r10 -8) = r2
regs=4 stack=0 before 1: (b7) r2 = 0
5: R2_w=0 R10=fp0 fp-24_w=00000000
5: (7b) *(u64 *)(r10 -32) = r2last_idx 5 first_idx 0
regs=4 stack=0 before 4: (7b) *(u64 *)(r10 -24) = r2
regs=4 stack=0 before 3: (7b) *(u64 *)(r10 -16) = r2
regs=4 stack=0 before 2: (7b) *(u64 *)(r10 -8) = r2
regs=4 stack=0 before 1: (b7) r2 = 0
6: R2_w=0 R10=fp0 fp-32_w=00000000
6: (7b) *(u64 *)(r10 -40) = r2last_idx 6 first_idx 0
regs=4 stack=0 before 5: (7b) *(u64 *)(r10 -32) = r2
regs=4 stack=0 before 4: (7b) *(u64 *)(r10 -24) = r2
regs=4 stack=0 before 3: (7b) *(u64 *)(r10 -16) = r2
regs=4 stack=0 before 2: (7b) *(u64 *)(r10 -8) = r2
regs=4 stack=0 before 1: (b7) r2 = 0
7: R2_w=0 R10=fp0 fp-40_w=00000000
7: (7b) *(u64 *)(r10 -48) = r2last_idx 7 first_idx 0
regs=4 stack=0 before 6: (7b) *(u64 *)(r10 -40) = r2
regs=4 stack=0 before 5: (7b) *(u64 *)(r10 -32) = r2
regs=4 stack=0 before 4: (7b) *(u64 *)(r10 -24) = r2
regs=4 stack=0 before 3: (7b) *(u64 *)(r10 -16) = r2
regs=4 stack=0 before 2: (7b) *(u64 *)(r10 -8) = r2
regs=4 stack=0 before 1: (b7) r2 = 0
8: R2_w=0 R10=fp0 fp-48_w=00000000
8: (7b) *(u64 *)(r10 -56) = r2last_idx 8 first_idx 0
regs=4 stack=0 before 7: (7b) *(u64 *)(r10 -48) = r2
regs=4 stack=0 before 6: (7b) *(u64 *)(r10 -40) = r2
regs=4 stack=0 before 5: (7b) *(u64 *)(r10 -32) = r2
regs=4 stack=0 before 4: (7b) *(u64 *)(r10 -24) = r2
regs=4 stack=0 before 3: (7b) *(u64 *)(r10 -16) = r2
regs=4 stack=0 before 2: (7b) *(u64 *)(r10 -8) = r2
regs=4 stack=0 before 1: (b7) r2 = 0
9: R2_w=0 R10=fp0 fp-56_w=00000000
9: (7b) *(u64 *)(r10 -64) = r2last_idx 9 first_idx 0
regs=4 stack=0 before 8: (7b) *(u64 *)(r10 -56) = r2
regs=4 stack=0 before 7: (7b) *(u64 *)(r10 -48) = r2
regs=4 stack=0 before 6: (7b) *(u64 *)(r10 -40) = r2
regs=4 stack=0 before 5: (7b) *(u64 *)(r10 -32) = r2
regs=4 stack=0 before 4: (7b) *(u64 *)(r10 -24) = r2
regs=4 stack=0 before 3: (7b) *(u64 *)(r10 -16) = r2
regs=4 stack=0 before 2: (7b) *(u64 *)(r10 -8) = r2
regs=4 stack=0 before 1: (b7) r2 = 0
10: R2_w=0 R10=fp0 fp-64_w=00000000
; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0)
11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0
;
12: (07) r2 += -64 ; R2_w=fp-64
; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1 R1 is of type file but file is expected
verification time 4903 usec
stack depth 64
processed 15 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
File Program Verdict Duration (us) Insns States Peak states ------------ ------- ------- ------------- ----- ------ ----------- fentry.bpf.o dbg failure 4903 15 0 0 ------------ ------- ------- ------------- ----- ------ -----------
Done. Processed 1 files, 0 programs. Skipped 1 files, 0 programs.
Quoted 4 lines
[1] My config for BPF testing
https://gist.github.com/eddyz87/aca79692d7bf57cfdd01b283b4304fd8 [2] Veristat verification log https://gist.github.com/eddyz87/49b211740bf99c426a37a3555b4542a3
/M
Eduard Zingerman
maintainerOn Fri, 2023-02-24 at 05:31 +0000, Matt Bobrowski wrote:
[...]
Quoted 118 lines
Quoted 15 lines
Could you please copy-paste output of the
fentryapplication, I'd like to see the log output of the libbpf while it processes relocations, e.g. here is what it prints for me:# /home/eddy/work/libbpf-bootstrap/examples/c/fentry libbpf: loading object 'fentry_bpf' from buffer libbpf: elf: section(3) lsm.s/bprm_committed_creds, size 136, link 0, flags 6, type=1 libbpf: sec 'lsm.s/bprm_committed_creds': found program 'dbg' at insn offset 0 (0 bytes), code size 17 insns (136 bytes) libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1 libbpf: license of fentry_bpf is Dual BSD/GPL libbpf: elf: section(5) .BTF, size 5114, link 0, flags 0, type=1 libbpf: elf: section(7) .BTF.ext, size 188, link 0, flags 0, type=1 libbpf: elf: section(10) .symtab, size 96, link 1, flags 0, type=2 libbpf: looking for externs among 4 symbols...Quoted 8 lines
libbpf: collected 0 externs total libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0 libbpf: sec 'lsm.s/bprm_committed_creds': found 1 CO-RE relocations libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [7241] struct linux_binprm in [vmlinux] libbpf: prog 'dbg': relo #0: <byte_off> [6] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #0 <byte_off> [7241] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: patched insn #10 (LDX/ST/STX) off 64 -> 64 Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` to see output of the BPF programs.Sure, here it is:
# ./fentry libbpf: loading object 'fentry_bpf' from buffer libbpf: elf: section(3) lsm.s/bprm_committed_creds, size 136, link 0, flags 6, type=1 libbpf: sec 'lsm.s/bprm_committed_creds': found program 'dbg' at insn offset 0 (0 bytes), code size 17 insns (136 bytes) libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1 libbpf: license of fentry_bpf is Dual BSD/GPL libbpf: elf: section(5) .BTF, size 5149, link 0, flags 0, type=1 libbpf: elf: section(7) .BTF.ext, size 188, link 0, flags 0, type=1 libbpf: elf: section(10) .symtab, size 96, link 1, flags 0, type=2 libbpf: looking for externs among 4 symbols... libbpf: collected 0 externs total libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0 libbpf: sec 'lsm.s/bprm_committed_creds': found 1 CO-RE relocations libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [3971] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [9214] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [36310] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [36973] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [122219] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [151720] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [163602] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [175117] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [176645] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [179130] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [189263] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [237046] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [240978] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [264207] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [268773] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [276058] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [295158] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [306160] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [347067] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [349932] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [380629] struct linux_binprm in [vmlinux] libbpf: prog 'dbg': relo #0: <byte_off> [6] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #0 <byte_off> [3971] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #1 <byte_off> [9214] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #2 <byte_off> [36310] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #3 <byte_off> [36973] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #4 <byte_off> [122219] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #5 <byte_off> [151720] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #6 <byte_off> [163602] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #7 <byte_off> [175117] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #8 <byte_off> [176645] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #9 <byte_off> [179130] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #10 <byte_off> [189263] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #11 <byte_off> [237046] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #12 <byte_off> [240978] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #13 <byte_off> [264207] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #14 <byte_off> [268773] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #15 <byte_off> [276058] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #16 <byte_off> [295158] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #17 <byte_off> [306160] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #18 <byte_off> [347067] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #19 <byte_off> [349932] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #20 <byte_off> [380629] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: patched insn #10 (LDX/ST/STX) off 64 -> 64 libbpf: prog 'dbg': BPF program load failed: Permission denied libbpf: prog 'dbg': -- BEGIN PROG LOAD LOG -- reg type unsupported for arg#0 function dbg#5 0: R1=ctx(off=0,imm=0) R10=fp0 ; int BPF_PROG(dbg, struct linux_binprm *bprm) 0: (79) r1 = *(u64 *)(r1 +0) func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 176645 type STRUCT 'linux_binprm' 1: R1_w=trusted_ptr_linux_binprm(off=0,imm=0) 1: (b7) r2 = 0 ; R2_w=0 ; char buf[64] = {0}; 2: (7b) *(u64 *)(r10 -8) = r2 ; R2_w=0 R10=fp0 fp-8_w=00000000 3: (7b) *(u64 *)(r10 -16) = r2 ; R2_w=0 R10=fp0 fp-16_w=00000000 4: (7b) *(u64 *)(r10 -24) = r2 ; R2_w=0 R10=fp0 fp-24_w=00000000 5: (7b) *(u64 *)(r10 -32) = r2 ; R2_w=0 R10=fp0 fp-32_w=00000000 6: (7b) *(u64 *)(r10 -40) = r2 ; R2_w=0 R10=fp0 fp-40_w=00000000 7: (7b) *(u64 *)(r10 -48) = r2 ; R2_w=0 R10=fp0 fp-48_w=00000000 8: (7b) *(u64 *)(r10 -56) = r2 ; R2_w=0 R10=fp0 fp-56_w=00000000 9: (7b) *(u64 *)(r10 -64) = r2 ; R2_w=0 R10=fp0 fp-64_w=00000000 ; bpf_ima_file_hash(bprm->file, buf, sizeof(buf)); 10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0) 11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0 ; 12: (07) r2 += -64 ; R2_w=fp-64 ; bpf_ima_file_hash(bprm->file, buf, sizeof(buf)); 13: (b7) r3 = 64 ; R3_w=64 14: (85) call bpf_ima_file_hash#193 cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1 R1 is of type file but file is expected processed 15 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0 -- END PROG LOAD LOG -- libbpf: prog 'dbg': failed to load: -13 libbpf: failed to load object 'fentry_bpf' libbpf: failed to load BPF skeleton 'fentry_bpf': -13 Failed to open BPF skeleton
Quoted 3 lines
It looks like there are a lot more relocations attempted by libbpf, but I suspect that's a result of their being multiple definitions of that same struct within the running kernel's BTF?
This shouldn't really be the case, as pahole de-duplicates BTF definitions when BTF is added to vmlinux.
One scenario I can think of is when linux_binprm data structure comes from multiple modules but not from vmlinux itself. However, the log would be a bit different in such case:
libbpf: CO-RE relocating [107] struct bpf_testmod_struct_arg_2: found target candidate [90383] struct bpf_testmod_struct_arg_2 in [bpf_testmod]
libbpf: CO-RE relocating [107] struct bpf_testmod_struct_arg_2: found target candidate [90353] struct bpf_testmod_struct_arg_2 in [bpf_testmod1]
Note in [bpf_testmod] and in [bpf_testmod1] which are my test modules. In your log it says in [vmlinux].
Which suggests that there are multiple _conflicting_ definitions of linux_binprm in your vmlinux and these definitions could not be de-duplicated. Could you please run the following command inside QEMU and share the output?
bpftool btf dump file /sys/kernel/btf/vmlinux | grep "'linux_binprm'" -A 30
Or outside the VM:
bpftool btf dump file {kernel}/vmlinux | grep "'linux_binprm'" -A 30
Also, could you please share full .config?
Do you use any non-standard compilation flags?
Quoted 21 lines
Quoted 21 lines
Also, could you please compile
veristattool as below:cd ${kernel}/tools/testing/selftests/bpf
make -j16 veristatAnd post the output of the following command (from within QEMU):
./veristat -l7 -v ${path-to-libbpf-bootstrap-within-vm}/examples/c/.output/fentry.bpf.oIt should produce the verification log as an output.
The reason I'm asking is that your verification log looks kinda strange:
Quoted 6 lines
;
bpf_ima_file_hash(bprm->file, buf, 64);
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in structllist_nodewith off 0 size 1 R1 is of type file but file is expected
processed 15 insns (limit 1000000)max_states_per_insn0total_states0peak_states0mark_read0
Quoted 6 lines
Quoted 2 lines
I don't understand why it mentions
struct llist_nodehere and don't have such messages in my log ([2]).Yes, I also found this strange and couldn't find a valid explanation for it either. Looking at the BPF verifier code in the kernel, we hit this case when performing the struct member walk in
btf_struct_walk().
To be honest, it looks like something is off with BTF ids and llist_node gets randomly picked, but that's a speculation w/o hard evidence.
Thanks,
Eduard
Sorry Eduard, I replied late last night although the email bounced due to exceeding the mail char limit. Let's try attaching a compressed variant of the requested files, which includes the compiled kernel's BTF and the kernel's config.
Quoted 134 lines · Eduard Zingerman
On Fri, 2023-02-24 at 05:31 +0000, Matt Bobrowski wrote:
[...]Quoted 118 lines
Quoted 15 lines
Could you please copy-paste output of the
fentryapplication, I'd like to see the log output of the libbpf while it processes relocations, e.g. here is what it prints for me:# /home/eddy/work/libbpf-bootstrap/examples/c/fentry libbpf: loading object 'fentry_bpf' from buffer libbpf: elf: section(3) lsm.s/bprm_committed_creds, size 136, link 0, flags 6, type=1 libbpf: sec 'lsm.s/bprm_committed_creds': found program 'dbg' at insn offset 0 (0 bytes), code size 17 insns (136 bytes) libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1 libbpf: license of fentry_bpf is Dual BSD/GPL libbpf: elf: section(5) .BTF, size 5114, link 0, flags 0, type=1 libbpf: elf: section(7) .BTF.ext, size 188, link 0, flags 0, type=1 libbpf: elf: section(10) .symtab, size 96, link 1, flags 0, type=2 libbpf: looking for externs among 4 symbols...Quoted 8 lines
libbpf: collected 0 externs total libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0 libbpf: sec 'lsm.s/bprm_committed_creds': found 1 CO-RE relocations libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [7241] struct linux_binprm in [vmlinux] libbpf: prog 'dbg': relo #0: <byte_off> [6] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #0 <byte_off> [7241] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: patched insn #10 (LDX/ST/STX) off 64 -> 64 Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` to see output of the BPF programs.Sure, here it is:
# ./fentry libbpf: loading object 'fentry_bpf' from buffer libbpf: elf: section(3) lsm.s/bprm_committed_creds, size 136, link 0, flags 6, type=1 libbpf: sec 'lsm.s/bprm_committed_creds': found program 'dbg' at insn offset 0 (0 bytes), code size 17 insns (136 bytes) libbpf: elf: section(4) license, size 13, link 0, flags 3, type=1 libbpf: license of fentry_bpf is Dual BSD/GPL libbpf: elf: section(5) .BTF, size 5149, link 0, flags 0, type=1 libbpf: elf: section(7) .BTF.ext, size 188, link 0, flags 0, type=1 libbpf: elf: section(10) .symtab, size 96, link 1, flags 0, type=2 libbpf: looking for externs among 4 symbols... libbpf: collected 0 externs total libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0 libbpf: sec 'lsm.s/bprm_committed_creds': found 1 CO-RE relocations libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [3971] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [9214] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [36310] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [36973] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [122219] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [151720] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [163602] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [175117] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [176645] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [179130] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [189263] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [237046] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [240978] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [264207] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [268773] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [276058] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [295158] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [306160] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [347067] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [349932] struct linux_binprm in [vmlinux] libbpf: CO-RE relocating [6] struct linux_binprm: found target candidate [380629] struct linux_binprm in [vmlinux] libbpf: prog 'dbg': relo #0: <byte_off> [6] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #0 <byte_off> [3971] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #1 <byte_off> [9214] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #2 <byte_off> [36310] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #3 <byte_off> [36973] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #4 <byte_off> [122219] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #5 <byte_off> [151720] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #6 <byte_off> [163602] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #7 <byte_off> [175117] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #8 <byte_off> [176645] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #9 <byte_off> [179130] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #10 <byte_off> [189263] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #11 <byte_off> [237046] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #12 <byte_off> [240978] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #13 <byte_off> [264207] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #14 <byte_off> [268773] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #15 <byte_off> [276058] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #16 <byte_off> [295158] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #17 <byte_off> [306160] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #18 <byte_off> [347067] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #19 <byte_off> [349932] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: matching candidate #20 <byte_off> [380629] struct linux_binprm.file (0:11 @ offset 64) libbpf: prog 'dbg': relo #0: patched insn #10 (LDX/ST/STX) off 64 -> 64 libbpf: prog 'dbg': BPF program load failed: Permission denied libbpf: prog 'dbg': -- BEGIN PROG LOAD LOG -- reg type unsupported for arg#0 function dbg#5 0: R1=ctx(off=0,imm=0) R10=fp0 ; int BPF_PROG(dbg, struct linux_binprm *bprm) 0: (79) r1 = *(u64 *)(r1 +0) func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 176645 type STRUCT 'linux_binprm' 1: R1_w=trusted_ptr_linux_binprm(off=0,imm=0) 1: (b7) r2 = 0 ; R2_w=0 ; char buf[64] = {0}; 2: (7b) *(u64 *)(r10 -8) = r2 ; R2_w=0 R10=fp0 fp-8_w=00000000 3: (7b) *(u64 *)(r10 -16) = r2 ; R2_w=0 R10=fp0 fp-16_w=00000000 4: (7b) *(u64 *)(r10 -24) = r2 ; R2_w=0 R10=fp0 fp-24_w=00000000 5: (7b) *(u64 *)(r10 -32) = r2 ; R2_w=0 R10=fp0 fp-32_w=00000000 6: (7b) *(u64 *)(r10 -40) = r2 ; R2_w=0 R10=fp0 fp-40_w=00000000 7: (7b) *(u64 *)(r10 -48) = r2 ; R2_w=0 R10=fp0 fp-48_w=00000000 8: (7b) *(u64 *)(r10 -56) = r2 ; R2_w=0 R10=fp0 fp-56_w=00000000 9: (7b) *(u64 *)(r10 -64) = r2 ; R2_w=0 R10=fp0 fp-64_w=00000000 ; bpf_ima_file_hash(bprm->file, buf, sizeof(buf)); 10: (79) r1 = *(u64 *)(r1 +64) ; R1_w=ptr_file(off=0,imm=0) 11: (bf) r2 = r10 ; R2_w=fp0 R10=fp0 ; 12: (07) r2 += -64 ; R2_w=fp-64 ; bpf_ima_file_hash(bprm->file, buf, sizeof(buf)); 13: (b7) r3 = 64 ; R3_w=64 14: (85) call bpf_ima_file_hash#193 cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1 R1 is of type file but file is expected processed 15 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0 -- END PROG LOAD LOG -- libbpf: prog 'dbg': failed to load: -13 libbpf: failed to load object 'fentry_bpf' libbpf: failed to load BPF skeleton 'fentry_bpf': -13 Failed to open BPF skeletonQuoted 3 lines
It looks like there are a lot more relocations attempted by libbpf, but I suspect that's a result of their being multiple definitions of that same struct within the running kernel's BTF?
This shouldn't really be the case, as pahole de-duplicates BTF definitions when BTF is added to vmlinux.
One scenario I can think of is when
linux_binprmdata structure comes from multiple modules but not fromvmlinuxitself. However, the log would be a bit different in such case:libbpf: CO-RE relocating [107] struct
bpf_testmod_struct_arg_2: found target candidate [90383] structbpf_testmod_struct_arg_2in [bpf_testmod]
libbpf: CO-RE relocating [107] structbpf_testmod_struct_arg_2: found target candidate [90353] structbpf_testmod_struct_arg_2in [bpf_testmod1]
Quoted 7 lines
Note
in [bpf_testmod]andin [bpf_testmod1]which are my test modules. In your log it saysin [vmlinux].Which suggests that there are multiple _conflicting_ definitions of
linux_binprmin yourvmlinuxand these definitions could not be de-duplicated. Could you please run the following command inside QEMU and share the output?
This sounds about right, as I'm also seeing the following during the compilation phase of the kernel:
WARN: multiple IDs found for 'linux_binprm': 3971, 380629 - using 3971
WARN: multiple IDs found for 'task_struct': 214, 384262 - using 214
WARN: multiple IDs found for 'file': 454, 384289 - using 454
WARN: multiple IDs found for 'vm_area_struct': 459, 384292 - using 459
WARN: multiple IDs found for 'seq_file': 1069, 384326 - using 1069
WARN: multiple IDs found for 'inode': 711, 384439 - using 711
WARN: multiple IDs found for 'path': 743, 384468 - using 743
WARN: multiple IDs found for 'cgroup': 765, 384477 - using 765
I don't quite see understand why this would just start being a problem now though? Perhaps this could be an issue with pahole? But then again, you've also managed to build pahole from source and not run into such issues...
Quoted 7 lines
bpftool btf dump file
/sys/kernel/btf/vmlinux| grep "'linux_binprm'" -A 30Or outside the VM:
bpftool btf dump file {kernel}/vmlinux | grep "'
linux_binprm'" -A 30Also, could you please share full
.config?
Sure, the output from both can be found within the attached archive.
Quoted 1 line
Do you use any non-standard compilation flags?
Not for the kernel, nor the BPF program.
Would it help if I also provided the raw compiled binary of the BPF program?
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Also, could you please compile
veristattool as below:cd ${kernel}/tools/testing/selftests/bpf
make -j16 veristatAnd post the output of the following command (from within QEMU):
./veristat -l7 -v ${path-to-libbpf-bootstrap-within-vm}/examples/c/.output/fentry.bpf.oIt should produce the verification log as an output.
The reason I'm asking is that your verification log looks kinda strange:
Quoted 6 lines
;
bpf_ima_file_hash(bprm->file, buf, 64);
13: (b7) r3 = 64 ; R3_w=64
14: (85) call bpf_ima_file_hash#193
cannot access ptr member next with moff 0 in structllist_nodewith off 0 size 1 R1 is of type file but file is expected
processed 15 insns (limit 1000000)max_states_per_insn0total_states0peak_states0mark_read0I don't understand why it mentions
struct llist_nodehere and don't have such messages in my log ([2]).
Quoted 6 lines
Quoted 3 lines
Yes, I also found this strange and couldn't find a valid explanation for it either. Looking at the BPF verifier code in the kernel, we hit this case when performing the struct member walk in
btf_struct_walk().To be honest, it looks like something is off with BTF ids and
llist_nodegets randomly picked, but that's a speculation w/o hard evidence.
Yes, I agree, but I'm not 100% sure why this is happening either. Through some rough debugging, I've literally observed the BPF verifier passing 2 completely different BTF IDs to btf_struct_ids_match(), but I also don't know whether this is expected or not? :)
/M
Eduard Zingerman
maintainerQuoted 4 lines · Matt Bobrowski
Sorry Eduard, I replied late last night although the email bounced due to exceeding the mail char limit. Let's try attaching a compressed variant of the requested files, which includes the compiled kernel's BTF and the kernel's config.
Hi Matt,
I tried using your config but still can't reproduce the issue. Will try to do it using debian 12 chroot tomorrow or on Monday.
Thanks,
Eduard
Eduard Zingerman
maintainerQuoted 10 lines · Eduard Zingerman
Quoted 4 lines · Matt Bobrowski
Sorry Eduard, I replied late last night although the email bounced due to exceeding the mail char limit. Let's try attaching a compressed variant of the requested files, which includes the compiled kernel's BTF and the kernel's config.
Hi Matt,
I tried using your config but still can't reproduce the issue. Will try to do it using debian 12 chroot tomorrow or on Monday.
Hi Matt,
Short update:
I've reproduced the issue with multiple STRUCT 'linux_binprm' BTF IDs in Debian testing chroot, thank you for providing all details. Attaching the instructions in the end of the email.
Need some time to analyze pahole behavior.
Thanks,
Eduard
--
host root:
mkdir bookworm
sudo debootstrap testing bookworm/ http://deb.debian.org/debian/
sudo mount -t proc proc bookworm/proc/
sudo mount -t sysfs sys bookworm/sys/
sudo chroot bookworm/ /bin/bash
bookworm root:
apt install python3 bc build-essential git \
cmake libdwarf-dev libdw-dev flex bison \
kmod cpio libncurses5-dev libelf-dev libssl-dev
adduser eddy
sudo eddy
bookworm user:
cd ~
git clone git://git.kernel.org/pub/scm/devel/pahole/pahole.git
mkdir build && \
cd build && \
cmake -D__LIB=lib .. && \
make -j $(nproc)
export PATH=/home/eddy/pahole/build/:$PATH
cd ~
git clone https://github.com/torvalds/linux.git && cd linux
make defconfig && make kvm_guest.config
scripts/config \
-e BPF \
-e BPF_SYSCALL \
-e BPF_LSM \
-e BPF_JIT \
-e BPF_EVENTS \
-e DEBUG_INFO \
-e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
-e DEBUG_INFO_BTF \
-e DEBUG_INFO_BTF_MODULES \
-e PAHOLE_HAS_SPLIT_BTF \
-e FTRACE \
-e DYNAMIC_FTRACE \
-e FUNCTION_TRACER
make olddefconfig
# Multiple warnings in the end:
# WARN: multiple IDs found for 'task_struct': 176, 23383 - using 176
# WARN: multiple IDs found for 'file': 638, 23422 - using 638
# ... make -C ./tools/bpf/bpftool
./tools/bpf/bpftool/bpftool btf dump file ./vmlinux | \
grep -c linux_binprm
# The output is 19 for meRepository versions:
- kernel:
f3a2439f20d9 ("Merge tag 'rproc-v6.3' of git://git.kernel.org/pub/scm/linux/kernel/git/remoteproc/linux") - pahole:
431df45 ("btfdiff: Exclude Rust CUs since those are not yet being converted to BTF on the Linux kernel")
Quoted 13 lines · Eduard Zingerman
Quoted 10 lines · Eduard Zingerman
Quoted 4 lines · Matt Bobrowski
Sorry Eduard, I replied late last night although the email bounced due to exceeding the mail char limit. Let's try attaching a compressed variant of the requested files, which includes the compiled kernel's BTF and the kernel's config.
Hi Matt,
I tried using your config but still can't reproduce the issue. Will try to do it using debian 12 chroot tomorrow or on Monday.
Quoted 8 lines
Hi Matt,
Short update:
I've reproduced the issue with multiple STRUCT 'linux_binprm' BTF IDs in Debian testing chroot, thank you for providing all details. Attaching the instructions in the end of the email.
Need some time to analyze pahole behavior.
Try using [0] to pinpoint what actually is different between any two linux_binprm definitions. I've hacked up this "tool" last time I had to pinpoint where two BTF types diverge, maybe it will save you a bit of time as well. I'd like to put this functionality into btfdump ([1]), but I didn't get to it yet, unfortunately.
[0] https://github.com/libbpf/libbpf-bootstrap/tree/btfdiff-hack
[1] https://github.com/anakryiko/btfdump
Quoted 12 lines
Thanks,
Eduard--
host root:
mkdir bookworm
sudo debootstrap testing bookworm/ http://deb.debian.org/debian/
sudo mount -t proc proc bookworm/proc/
sudo mount -t sysfs sys bookworm/sys/
sudo chroot bookworm//bin/bash
[...]
Quoted 24 lines · Andrii Nakryiko
Quoted 13 lines · Eduard Zingerman
Quoted 10 lines · Eduard Zingerman
Quoted 4 lines · Matt Bobrowski
Sorry Eduard, I replied late last night although the email bounced due to exceeding the mail char limit. Let's try attaching a compressed variant of the requested files, which includes the compiled kernel's BTF and the kernel's config.
Hi Matt,
I tried using your config but still can't reproduce the issue. Will try to do it using debian 12 chroot tomorrow or on Monday.
Quoted 8 lines
Hi Matt,
Short update:
I've reproduced the issue with multiple STRUCT 'linux_binprm' BTF IDs in Debian testing chroot, thank you for providing all details. Attaching the instructions in the end of the email.
Need some time to analyze pahole behavior.
Quoted 6 lines
Try using [0] to pinpoint what actually is different between any two
linux_binprmdefinitions. I've hacked up this "tool" last time I had to pinpoint where two BTF types diverge, maybe it will save you a bit of time as well. I'd like to put this functionality into btfdump ([1]), but I didn't get to it yet, unfortunately.
It's not just linux_binprm but a bunch of other structs (quite a lot of them that seem to diverge)
[...]
WARN: multiple IDs found for 'sock_common': 4400, 53212 - using 4400
WARN: multiple IDs found for 'request_sock': 4458, 53257 - using 4458
WARN: multiple IDs found for 'task_struct': 265, 55830 - using 265
WARN: multiple IDs found for 'file': 474, 55854 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 55857 - using 480
WARN: multiple IDs found for 'seq_file': 670, 55891 - using 670
WARN: multiple IDs found for 'sock': 3623, 55918 - using 3623
WARN: multiple IDs found for 'sk_buff': 3095, 56014 - using 3095
WARN: multiple IDs found for 'fib6_info': 8129, 56069 - using 8129
WARN: multiple IDs found for 'inode': 854, 56092 - using 854
WARN: multiple IDs found for 'path': 886, 56121 - using 886
WARN: multiple IDs found for 'cgroup': 913, 56130 - using 913
WARN: multiple IDs found for 'xdp_buff': 3838, 56558 - using 3838
WARN: multiple IDs found for 'socket': 3848, 56563 - using 3848
WARN: multiple IDs found for 'sock_common': 4400, 56890 - using 4400
WARN: multiple IDs found for 'request_sock': 4458, 56938 - using 4458
WARN: multiple IDs found for 'task_struct': 265, 58141 - using 265
WARN: multiple IDs found for 'inode': 854, 58153 - using 854
WARN: multiple IDs found for 'path': 886, 58182 - using 886
WARN: multiple IDs found for 'file': 474, 58207 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 58210 - using 480
WARN: multiple IDs found for 'seq_file': 670, 58389 - using 670
WARN: multiple IDs found for 'task_struct': 265, 58862 - using 265
WARN: multiple IDs found for 'file': 474, 58889 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 58892 - using 480
WARN: multiple IDs found for 'inode': 854, 58954 - using 854
WARN: multiple IDs found for 'path': 886, 58983 - using 886
WARN: multiple IDs found for 'cgroup': 913, 58992 - using 913
WARN: multiple IDs found for 'seq_file': 670, 59157 - using 670
WARN: multiple IDs found for 'sock': 3623, 59309 - using 3623
WARN: multiple IDs found for 'sk_buff': 3095, 59312 - using 3095
WARN: multiple IDs found for 'xdp_buff': 3838, 59699 - using 3838
WARN: multiple IDs found for 'socket': 3848, 59704 - using 3848
WARN: multiple IDs found for 'sock_common': 4400, 60021 - using 4400
WARN: multiple IDs found for 'request_sock': 4458, 60066 - using 4458
WARN: multiple IDs found for 'file': 474, 60890 - using 474
WARN: multiple IDs found for 'task_struct': 265, 60900 - using 265
WARN: multiple IDs found for 'vm_area_struct': 480, 60932 - using 480
WARN: multiple IDs found for 'inode': 854, 60962 - using 854
WARN: multiple IDs found for 'path': 886, 60991 - using 886
WARN: multiple IDs found for 'cgroup': 913, 61000 - using 913
WARN: multiple IDs found for 'seq_file': 670, 61174 - using 670
WARN: multiple IDs found for 'task_struct': 265, 62225 - using 265
WARN: multiple IDs found for 'file': 474, 62256 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 62259 - using 480
WARN: multiple IDs found for 'seq_file': 670, 62289 - using 670
WARN: multiple IDs found for 'inode': 854, 62573 - using 854
WARN: multiple IDs found for 'path': 886, 62602 - using 886 [...]
BTF .btf.vmlinux.bin.o
seems to take 10+ minutes. Something seems off here, it seems like the deduplication is failing for a lot of types which causes pahole to run for so long.
My pahole is at:
431df45 ("btfdiff: Exclude Rust CUs since those are not yet being converted to BTF on the Linux kernel")
Clang at:
9e5bfa1ae30b ("[AArch64] Add some tests for multiple uses of extended vector extracts. NFC")
and kernel at:
68bfd65fb98d ("Merge branch 'move SYS() macro to test_progs.h and run mptcp in a dedicated netns'")
and I am using the same config as vmtest.sh.
I was able to "fix" this with using clang 16.x and the following hacky pahole:
030e3b4 - core: Add DW_TAG_unspecified_type to tag__is_tag_type() set (HEAD) (2023-02-26 Arnaldo Carvalho de Melo)
f20515b - dwarves: support DW_TAG_atomic_type (2023-02-26 David Lamparter) de24234 - pahole: Prep 1.24 (tag: v1.24) (2022-08-22 Arnaldo Carvalho de Melo) d6c9528 - dwarf_loader: Encode char type as signed (2022-08-10 Yonghong Song)
and the the following patch on top:
btf_encoder.c +0 -0
@@ -939,6 +939,8 @@ static int btf_encoder__encode_tag(struct
btf_encoder *encoder, struct tag *tag,
return btf_encoder__add_enum_type(encoder, tag, conf_load);
case DW_TAG_subroutine_type:
return btf_encoder__add_func_proto(encoder,
tag__ftype(tag), type_id_off);
+ case DW_TAG_unspecified_type:
+ return 0;
default:
fprintf(stderr, "Unsupported DW_TAG_%s(0x%x)\n",
dwarf_tag_name(tag->tag), tag->tag);dwarves.h +0 -2
@@ -544,10 +544,8 @@ static inline int tag__is_tag_type(const struct tag *tag) tag->tag == DW_TAG_reference_type || tag->tag == DW_TAG_restrict_type || tag->tag == DW_TAG_subroutine_type || - tag->tag == DW_TAG_unspecified_type || tag->tag == DW_TAG_volatile_type || tag->tag == DW_TAG_atomic_type || - tag->tag == DW_TAG_unspecified_type || tag->tag == DW_TAG_LLVM_annotation; }
Now this is not a clean build and still has errors like:
tag__recode_dwarf_type: couldn't find 0xe82f542 type for 0xe82f4d2 (subprogram)!tag__recode_dwarf_type: couldn't find 0xe82f542 type for 0xe82f4e2 (subprogram)!
Quoted 18 lines
[0] https://github.com/libbpf/libbpf-bootstrap/tree/btfdiff-hack
[1] https://github.com/anakryiko/btfdumpQuoted 12 lines
Thanks,
Eduard--
host root:
mkdir bookworm
sudo debootstrap testing bookworm/ http://deb.debian.org/debian/
sudo mount -t proc proc bookworm/proc/
sudo mount -t sysfs sys bookworm/sys/
sudo chroot bookworm//bin/bash[...]
Quoted 34 lines · KP Singh
Quoted 24 lines · Andrii Nakryiko
Quoted 13 lines · Eduard Zingerman
Quoted 10 lines · Eduard Zingerman
Quoted 4 lines · Matt Bobrowski
Sorry Eduard, I replied late last night although the email bounced due to exceeding the mail char limit. Let's try attaching a compressed variant of the requested files, which includes the compiled kernel's BTF and the kernel's config.
Hi Matt,
I tried using your config but still can't reproduce the issue. Will try to do it using debian 12 chroot tomorrow or on Monday.
Quoted 8 lines
Hi Matt,
Short update:
I've reproduced the issue with multiple STRUCT 'linux_binprm' BTF IDs in Debian testing chroot, thank you for providing all details. Attaching the instructions in the end of the email.
Need some time to analyze pahole behavior.Quoted 6 lines
Try using [0] to pinpoint what actually is different between any two
linux_binprmdefinitions. I've hacked up this "tool" last time I had to pinpoint where two BTF types diverge, maybe it will save you a bit of time as well. I'd like to put this functionality into btfdump ([1]), but I didn't get to it yet, unfortunately.
Quoted 2 lines
It's not just
linux_binprmbut a bunch of other structs (quite a lot of them that seem to diverge)
It also does not seem to matter if I build the kernel with gcc or clang, my suspicion went to an underlying dependency (libdw / elfutils) I tried recompiling these from HEAD and downgrading the package version too, but to no avail.
Quoted 141 lines
[...]
WARN: multiple IDs found for 'sock_common': 4400, 53212 - using 4400
WARN: multiple IDs found for 'request_sock': 4458, 53257 - using 4458
WARN: multiple IDs found for 'task_struct': 265, 55830 - using 265
WARN: multiple IDs found for 'file': 474, 55854 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 55857 - using 480
WARN: multiple IDs found for 'seq_file': 670, 55891 - using 670
WARN: multiple IDs found for 'sock': 3623, 55918 - using 3623
WARN: multiple IDs found for 'sk_buff': 3095, 56014 - using 3095
WARN: multiple IDs found for 'fib6_info': 8129, 56069 - using 8129
WARN: multiple IDs found for 'inode': 854, 56092 - using 854
WARN: multiple IDs found for 'path': 886, 56121 - using 886
WARN: multiple IDs found for 'cgroup': 913, 56130 - using 913
WARN: multiple IDs found for 'xdp_buff': 3838, 56558 - using 3838
WARN: multiple IDs found for 'socket': 3848, 56563 - using 3848
WARN: multiple IDs found for 'sock_common': 4400, 56890 - using 4400
WARN: multiple IDs found for 'request_sock': 4458, 56938 - using 4458
WARN: multiple IDs found for 'task_struct': 265, 58141 - using 265
WARN: multiple IDs found for 'inode': 854, 58153 - using 854
WARN: multiple IDs found for 'path': 886, 58182 - using 886
WARN: multiple IDs found for 'file': 474, 58207 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 58210 - using 480
WARN: multiple IDs found for 'seq_file': 670, 58389 - using 670
WARN: multiple IDs found for 'task_struct': 265, 58862 - using 265
WARN: multiple IDs found for 'file': 474, 58889 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 58892 - using 480
WARN: multiple IDs found for 'inode': 854, 58954 - using 854
WARN: multiple IDs found for 'path': 886, 58983 - using 886
WARN: multiple IDs found for 'cgroup': 913, 58992 - using 913
WARN: multiple IDs found for 'seq_file': 670, 59157 - using 670
WARN: multiple IDs found for 'sock': 3623, 59309 - using 3623
WARN: multiple IDs found for 'sk_buff': 3095, 59312 - using 3095
WARN: multiple IDs found for 'xdp_buff': 3838, 59699 - using 3838
WARN: multiple IDs found for 'socket': 3848, 59704 - using 3848
WARN: multiple IDs found for 'sock_common': 4400, 60021 - using 4400
WARN: multiple IDs found for 'request_sock': 4458, 60066 - using 4458
WARN: multiple IDs found for 'file': 474, 60890 - using 474
WARN: multiple IDs found for 'task_struct': 265, 60900 - using 265
WARN: multiple IDs found for 'vm_area_struct': 480, 60932 - using 480
WARN: multiple IDs found for 'inode': 854, 60962 - using 854
WARN: multiple IDs found for 'path': 886, 60991 - using 886
WARN: multiple IDs found for 'cgroup': 913, 61000 - using 913
WARN: multiple IDs found for 'seq_file': 670, 61174 - using 670
WARN: multiple IDs found for 'task_struct': 265, 62225 - using 265
WARN: multiple IDs found for 'file': 474, 62256 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 62259 - using 480
WARN: multiple IDs found for 'seq_file': 670, 62289 - using 670
WARN: multiple IDs found for 'inode': 854, 62573 - using 854
WARN: multiple IDs found for 'path': 886, 62602 - using 886 [...]BTF .btf.vmlinux.bin.o
seems to take 10+ minutes. Something seems off here, it seems like the deduplication is failing for a lot of types which causes pahole to run for so long.
My pahole is at:
431df45 ("btfdiff: Exclude Rust CUs since those are not yet being converted to BTF on the Linux kernel")
Clang at:
9e5bfa1ae30b("[AArch64] Add some tests for multiple uses of extended vector extracts. NFC")and kernel at:
68bfd65fb98d("Merge branch 'moveSYS()macro totest_progs.hand run mptcp in a dedicated netns'")and I am using the same config as
vmtest.sh.I was able to "fix" this with using clang 16.x and the following hacky pahole:
030e3b4 - core: Add
DW_TAG_unspecified_typetotag__is_tag_type()set (HEAD) (2023-02-26 Arnaldo Carvalho de Melo)
f20515b - dwarves: supportDW_TAG_atomic_type(2023-02-26 David Lamparter) de24234 - pahole: Prep 1.24 (tag: v1.24) (2022-08-22 Arnaldo Carvalho de Melo) d6c9528 -dwarf_loader: Encode char type as signed (2022-08-10 Yonghong Song)and the the following patch on top:
btf_encoder.c +2
-0@@ -939,6 +939,8 @@ static int btf_encoder__encode_tag(struct btf_encoder *encoder, struct tag *tag, return btf_encoder__add_enum_type(encoder, tag, conf_load);case DW_TAG_subroutine_type:return btf_encoder__add_func_proto (encoder,tag__ftype (tag), type_id_off); +case DW_TAG_unspecified_type: +return 0 ;default :fprintf(stderr, "Unsupported DW_TAG_%s(0x%x)\n", dwarf_tag_name(tag->tag), tag->tag);dwarves.h +0
-2@@ -544,10 +544,8 @@ static inline int tag__is_tag_type(const struct tag *tag) tag->tag == DW_TAG_reference_type || tag->tag == DW_TAG_restrict_type || tag->tag == DW_TAG_subroutine_type || - tag->tag == DW_TAG_unspecified_type || tag->tag == DW_TAG_volatile_type || tag->tag == DW_TAG_atomic_type || - tag->tag == DW_TAG_unspecified_type || tag->tag == DW_TAG_LLVM_annotation; }Now this is not a clean build and still has errors like:
tag__recode_dwarf_type: couldn't find0xe82f542type for0xe82f4d2(subprogram)!tag__recode_dwarf_type: couldn't find0xe82f542type for0xe82f4e2(subprogram)!Quoted 18 lines
[0] https://github.com/libbpf/libbpf-bootstrap/tree/btfdiff-hack
[1] https://github.com/anakryiko/btfdumpQuoted 12 lines
Thanks,
Eduard--
host root:
mkdir bookworm
sudo debootstrap testing bookworm/ http://deb.debian.org/debian/
sudo mount -t proc proc bookworm/proc/
sudo mount -t sysfs sys bookworm/sys/
sudo chroot bookworm//bin/bash[...]
Quoted 34 lines · KP Singh
Quoted 24 lines · Andrii Nakryiko
Quoted 13 lines · Eduard Zingerman
Quoted 10 lines · Eduard Zingerman
Quoted 4 lines · Matt Bobrowski
Sorry Eduard, I replied late last night although the email bounced due to exceeding the mail char limit. Let's try attaching a compressed variant of the requested files, which includes the compiled kernel's BTF and the kernel's config.
Hi Matt,
I tried using your config but still can't reproduce the issue. Will try to do it using debian 12 chroot tomorrow or on Monday.
Quoted 8 lines
Hi Matt,
Short update:
I've reproduced the issue with multiple STRUCT 'linux_binprm' BTF IDs in Debian testing chroot, thank you for providing all details. Attaching the instructions in the end of the email.
Need some time to analyze pahole behavior.Quoted 6 lines
Try using [0] to pinpoint what actually is different between any two
linux_binprmdefinitions. I've hacked up this "tool" last time I had to pinpoint where two BTF types diverge, maybe it will save you a bit of time as well. I'd like to put this functionality into btfdump ([1]), but I didn't get to it yet, unfortunately.
Quoted 2 lines
It's not just
linux_binprmbut a bunch of other structs (quite a lot of them that seem to diverge)
yes, task_struct is usually a thing that suffers from such duplications, as it forms a huge graph of types. Which is where that btfdiff "tool" comes handly, it recursively compares side-by-side two types that are supposed to be equal (but are not), by ignoring BTF type IDs and trying to pin point actual differences (like STRUCT vs FWD, or extra field, or whatever). It just needs two starting type IDs.
Quoted 8 lines
[...]
WARN: multiple IDs found for 'sock_common': 4400, 53212 - using 4400
WARN: multiple IDs found for 'request_sock': 4458, 53257 - using 4458
WARN: multiple IDs found for 'task_struct': 265, 55830 - using 265
WARN: multiple IDs found for 'file': 474, 55854 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 55857 - using 480
WARN: multiple IDs found for 'seq_file': 670, 55891 - using 670
[...]
Quoted 11 lines
I was able to "fix" this with using clang 16.x and the following hacky pahole:
030e3b4 - core: Add
DW_TAG_unspecified_typetotag__is_tag_type()set (HEAD) (2023-02-26 Arnaldo Carvalho de Melo)
f20515b - dwarves: supportDW_TAG_atomic_type(2023-02-26 David Lamparter) de24234 - pahole: Prep 1.24 (tag: v1.24) (2022-08-22 Arnaldo Carvalho de Melo) d6c9528 -dwarf_loader: Encode char type as signed (2022-08-10 Yonghong Song)and the the following patch on top:
I'd start with understanding what BTF and DWARF differences are causing the issue before trying to come up with the fix. For that we don't even need config or repro steps, it should be enough to share vmlinux with BTF and DWARF, and start from there.
But I'm sure Eduard is on top of this already (especially that he can repro the issue now).
Eduard Zingerman
maintainerQuoted 38 lines · Andrii Nakryiko
Quoted 34 lines · KP Singh
Quoted 24 lines · Andrii Nakryiko
Quoted 13 lines · Eduard Zingerman
Quoted 10 lines · Eduard Zingerman
Quoted 4 lines · Matt Bobrowski
Sorry Eduard, I replied late last night although the email bounced due to exceeding the mail char limit. Let's try attaching a compressed variant of the requested files, which includes the compiled kernel's BTF and the kernel's config.
Hi Matt,
I tried using your config but still can't reproduce the issue. Will try to do it using debian 12 chroot tomorrow or on Monday.
Quoted 8 lines
Hi Matt,
Short update:
I've reproduced the issue with multiple STRUCT 'linux_binprm' BTF IDs in Debian testing chroot, thank you for providing all details. Attaching the instructions in the end of the email.
Need some time to analyze pahole behavior.Quoted 6 lines
Try using [0] to pinpoint what actually is different between any two
linux_binprmdefinitions. I've hacked up this "tool" last time I had to pinpoint where two BTF types diverge, maybe it will save you a bit of time as well. I'd like to put this functionality into btfdump ([1]), but I didn't get to it yet, unfortunately.Quoted 2 lines
It's not just
linux_binprmbut a bunch of other structs (quite a lot of them that seem to diverge)
Quoted 7 lines
yes,
task_structis usually a thing that suffers from such duplications, as it forms a huge graph of types. Which is where that btfdiff "tool" comes handly, it recursively compares side-by-side two types that are supposed to be equal (but are not), by ignoring BTF type IDs and trying to pin point actual differences (like STRUCT vs FWD, or extra field, or whatever). It just needs two starting type IDs.
Thank you for links to these tools!
Quoted 22 lines
Quoted 9 lines
[...]
WARN: multiple IDs found for 'sock_common': 4400, 53212 - using 4400
WARN: multiple IDs found for 'request_sock': 4458, 53257 - using 4458
WARN: multiple IDs found for 'task_struct': 265, 55830 - using 265
WARN: multiple IDs found for 'file': 474, 55854 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 55857 - using 480
WARN: multiple IDs found for 'seq_file': 670, 55891 - using 670[...]
Quoted 9 lines
I was able to "fix" this with using clang 16.x and the following hacky pahole:
030e3b4 - core: Add
DW_TAG_unspecified_typetotag__is_tag_type()set (HEAD) (2023-02-26 Arnaldo Carvalho de Melo)
f20515b - dwarves: supportDW_TAG_atomic_type(2023-02-26 David Lamparter) de24234 - pahole: Prep 1.24 (tag: v1.24) (2022-08-22 Arnaldo Carvalho de Melo) d6c9528 -dwarf_loader: Encode char type as signed (2022-08-10 Yonghong Song)
Quoted 8 lines
Quoted 2 lines
and the the following patch on top:
I'd start with understanding what BTF and DWARF differences are causing the issue before trying to come up with the fix. For that we don't even need config or repro steps, it should be enough to share vmlinux with BTF and DWARF, and start from there.
Yes, I suspect that there is some kind of unanticipated anomaly for some DWARF encoding for some kind of objects, just need to find the root for the diverging type hierarchies.
Quoted 2 lines
But I'm sure Eduard is on top of this already (especially that he can repro the issue now).
I'm working on it, nothing to report yet, but I'm working on it.
Quoted 81 lines · Eduard Zingerman
Quoted 38 lines · Andrii Nakryiko
Quoted 34 lines · KP Singh
Quoted 24 lines · Andrii Nakryiko
Quoted 13 lines · Eduard Zingerman
Quoted 10 lines · Eduard Zingerman
Quoted 4 lines · Matt Bobrowski
Sorry Eduard, I replied late last night although the email bounced due to exceeding the mail char limit. Let's try attaching a compressed variant of the requested files, which includes the compiled kernel's BTF and the kernel's config.
Hi Matt,
I tried using your config but still can't reproduce the issue. Will try to do it using debian 12 chroot tomorrow or on Monday.
Quoted 8 lines
Hi Matt,
Short update:
I've reproduced the issue with multiple STRUCT 'linux_binprm' BTF IDs in Debian testing chroot, thank you for providing all details. Attaching the instructions in the end of the email.
Need some time to analyze pahole behavior.Quoted 6 lines
Try using [0] to pinpoint what actually is different between any two
linux_binprmdefinitions. I've hacked up this "tool" last time I had to pinpoint where two BTF types diverge, maybe it will save you a bit of time as well. I'd like to put this functionality into btfdump ([1]), but I didn't get to it yet, unfortunately.Quoted 2 lines
It's not just
linux_binprmbut a bunch of other structs (quite a lot of them that seem to diverge)Quoted 7 lines
yes,
task_structis usually a thing that suffers from such duplications, as it forms a huge graph of types. Which is where that btfdiff "tool" comes handly, it recursively compares side-by-side two types that are supposed to be equal (but are not), by ignoring BTF type IDs and trying to pin point actual differences (like STRUCT vs FWD, or extra field, or whatever). It just needs two starting type IDs.Thank you for links to these tools!
Quoted 22 lines
Quoted 9 lines
[...]
WARN: multiple IDs found for 'sock_common': 4400, 53212 - using 4400
WARN: multiple IDs found for 'request_sock': 4458, 53257 - using 4458
WARN: multiple IDs found for 'task_struct': 265, 55830 - using 265
WARN: multiple IDs found for 'file': 474, 55854 - using 474
WARN: multiple IDs found for 'vm_area_struct': 480, 55857 - using 480
WARN: multiple IDs found for 'seq_file': 670, 55891 - using 670[...]
Quoted 9 lines
I was able to "fix" this with using clang 16.x and the following hacky pahole:
030e3b4 - core: Add
DW_TAG_unspecified_typetotag__is_tag_type()set (HEAD) (2023-02-26 Arnaldo Carvalho de Melo)
f20515b - dwarves: supportDW_TAG_atomic_type(2023-02-26 David Lamparter) de24234 - pahole: Prep 1.24 (tag: v1.24) (2022-08-22 Arnaldo Carvalho de Melo) d6c9528 -dwarf_loader: Encode char type as signed (2022-08-10 Yonghong Song)Quoted 8 lines
Quoted 2 lines
and the the following patch on top:
I'd start with understanding what BTF and DWARF differences are causing the issue before trying to come up with the fix. For that we don't even need config or repro steps, it should be enough to share vmlinux with BTF and DWARF, and start from there.
Quoted 9 lines
Yes, I suspect that there is some kind of unanticipated anomaly for some DWARF encoding for some kind of objects, just need to find the root for the diverging type hierarchies.
Quoted 2 lines
But I'm sure Eduard is on top of this already (especially that he can repro the issue now).
I'm working on it, nothing to report yet, but I'm working on it.
Thanks, please keep us posted!
Eduard Zingerman
maintainerOn Mon, 2023-02-27 at 11:31 -0800, Andrii Nakryiko wrote:
[...]
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
I'd start with understanding what BTF and DWARF differences are causing the issue before trying to come up with the fix. For that we don't even need config or repro steps, it should be enough to share vmlinux with BTF and DWARF, and start from there.
Yes, I suspect that there is some kind of unanticipated anomaly for some DWARF encoding for some kind of objects, just need to find the root for the diverging type hierarchies.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
But I'm sure Eduard is on top of this already (especially that he can repro the issue now).
I'm working on it, nothing to report yet, but I'm working on it.
Thanks, please keep us posted!
It is interesting how everything is interconnected. The patch for pahole below happens to help. I prepared it last week while working on new DWARF encoding scheme for btf_type_tag.
I still need to track down which "unspecified_type" entries caused the issue in this particular case. Will post an update tomorrow.
Meanwhile, Matt, KP, could you please verify the patch on your side? It is for the "next" branch of pahole.
---
From 09fac63ca08e25aea499f827283b07cc87a7daab Mon Sep 17 00:00:00 2001
From: Eduard Zingerman <[email protected]>
Date: Tue, 21 Feb 2023 19:23:00 +0200
Subject: [PATCH] dwarf_loader: Fix for BTF id drift caused by adding
unspecified types
Recent changes to handle unspecified types (see [1]) cause BTF ID drift.
Specifically, the intent of commits [2], [3] and [4] is to render references to unspecified types as void type.
However, as a consequence:
- in die__process_unit() call to cu__add_tag() allocates small_id for unspecified type tags and adds these tags to cu->types_table;
- btf_encoder__encode_tag() skips generation of BTF entries for DW_TAG_unspecified_type tags.
Such logic causes ID drift if unspecified type is not the last type processed for compilation unit. small_id of each type following unspecified type in the cu->types_table would have its BTF id off by -1. Thus renders references established on recode phase invalid.
This commit reverts unspecified_type id/tag tracking, instead:
- small_id for unspecified type tags is set to 0, thus reference to unspecified type tag would render BTF id of a void on recode phase;
- unspecified type tags are not added to cu->types_table.
[1] https://lore.kernel.org/all/Y0R7uu3s%[email protected]/
[2] bcc648a10cbc ("btf_encoder: Encode DW_TAG_unspecified_type returning routines as void")
[3] cffe5e1f75e1 ("core: Record if a CU has a DW_TAG_unspecified_type")
[4] 75e0fe28bb02 ("core: Add DW_TAG_unspecified_type to tag__is_tag_type() set")
--- btf_encoder.c | 8 -------- dwarf_loader.c | 25 +++++++++++++++++++------ dwarves.h | 8 -------- 3 files changed, 19 insertions(+), 22 deletions(-)
btf_encoder.c +0 -8
@@ -69,7 +69,6 @@ struct btf_encoder {const char *filename;struct elf_symtab *symtab;uint32_t type_id_off; -uint32_t unspecified_type;int saved_func_cnt;bool has_index_type, need_index_type,
@@ -635,11 +634,6 @@ static int32_t btf_encoder__tag_type(struct btf_encoder *encoder, uint32_t tag_tif (tag_type ==0 )return 0 ; -if (encoder->unspecified_type && tag_type == encoder->unspecified_type) { -// No provision for encoding this, turn it into void. -return 0 ; - } -return encoder->type_id_off + tag_type; }
@@ -1746,8 +1740,6 @@ int btf_encoder__encode_cu(struct btf_encoder *encoder, struct cu *cu, struct co encoder->cu = cu; encoder->type_id_off =btf__type_cnt (encoder->btf) -1 ; -if (encoder->cu->unspecified_type.tag) - encoder->unspecified_type = encoder->cu->unspecified_type.type;if (!encoder->has_index_type) {/* cu__find_base_type_by_name() takes "type_id_t *id" */
dwarf_loader.c +19 -6
@@ -2155,8 +2155,7 @@ static struct tag *__die__process_tag(Dwarf_Die *die, struct cu *cu,case DW_TAG_atomic_type: tag =die__create_new_tag (die, cu);break ;case DW_TAG_unspecified_type: - cu->unspecified_type.tag = - tag =die__create_new_tag (die, cu);break ; + tag =die__create_new_tag (die, cu);break ;case DW_TAG_pointer_type: tag =die__create_new_pointer_tag (die, cu, conf);break ;case DW_TAG_ptr_to_member_type:
@@ -2219,13 +2218,27 @@ static int die__process_unit(Dwarf_Die *die, struct cu *cu, struct conf_load *cocontinue ; } -uint32_t id; -cu__add_tag (cu, tag, &id); +uint32_t id =0 ; +/* There is no BTF representation for unspecified types. +* Currently we want such types to be represented as `void` +* (and thus skip BTF encoding). +* +* As BTF encoding is skipped, such types must not be added to type table, +* otherwise an ID for a type would be allocated and we would be forced +* to put something in BTF at this ID. +* Thus avoid `cu__add_tag()` call for such types. +* +* On the other hand, there might be references to this type from other +* tags, so `dwarf_cu__find_tag_by_ref()` must return something. +* Thus call `cu__hash()` for such types. +* +* Note, that small_id of zero would be assigned to unspecified type entry. +*/ +if (tag->tag != DW_TAG_unspecified_type) +cu__add_tag (cu, tag, &id);cu__hash (cu, tag);struct dwarf_tag *dtag = tag->priv; dtag->small_id = id; -if (tag->tag == DW_TAG_unspecified_type) - cu->unspecified_type.type = id; }while (dwarf_siblingof (die, die) ==0 );return 0 ;
dwarves.h +0 -8
@@ -236,10 +236,6 @@ struct debug_fmt_ops {#define ARCH_MAX_REGISTER_PARAMS 8 -/* -* unspecified_type: If this CU has a DW_TAG_unspecified_type, as BTF doesn't have a representation for this -* and thus we need to check functions returning this to convert it to void. -*/ struct cu {struct list_head node;struct list_head tags;
@@ -248,10 +244,6 @@ struct cu {struct ptr_table functions_table;struct ptr_table tags_table;struct rb_root functions; -struct { -struct tag *tag; -uint32_t type; - } unspecified_type;char *name;char *filename;void *priv;
Signature · 1 line
2.39.1
KP Singh
Tested-byQuoted 59 lines · Eduard Zingerman
On Mon, 2023-02-27 at 11:31 -0800, Andrii Nakryiko wrote:
[...]Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
I'd start with understanding what BTF and DWARF differences are causing the issue before trying to come up with the fix. For that we don't even need config or repro steps, it should be enough to share vmlinux with BTF and DWARF, and start from there.
Yes, I suspect that there is some kind of unanticipated anomaly for some DWARF encoding for some kind of objects, just need to find the root for the diverging type hierarchies.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
But I'm sure Eduard is on top of this already (especially that he can repro the issue now).
I'm working on it, nothing to report yet, but I'm working on it.
Thanks, please keep us posted!
It is interesting how everything is interconnected. The patch for pahole below happens to help. I prepared it last week while working on new DWARF encoding scheme for
btf_type_tag.I still need to track down which "
unspecified_type" entries caused the issue in this particular case. Will post an update tomorrow.Meanwhile, Matt, KP, could you please verify the patch on your side? It is for the "next" branch of pahole.
---
From
09fac63ca08e25aea499f827283b07cc87a7daabMon Sep 17 00:00:00 2001
From: Eduard Zingerman <[email protected]>
Date: Tue, 21 Feb 2023 19:23:00 +0200
Subject: [PATCH]dwarf_loader: Fix for BTF id drift caused by adding
unspecified typesRecent changes to handle unspecified types (see [1]) cause BTF ID drift.
Specifically, the intent of commits [2], [3] and [4] is to render references to unspecified types as void type.
However, as a consequence:
- indie__process_unit()call tocu__add_tag()allocatessmall_idfor unspecified type tags and adds these tags tocu->types_table;
-btf_encoder__encode_tag()skips generation of BTF entries forDW_TAG_unspecified_typetags.Such logic causes ID drift if unspecified type is not the last type processed for compilation unit.
small_idof each type following unspecified type in thecu->types_tablewould have its BTF id off by -1. Thus renders references established on recode phase invalid.This commit reverts
unspecified_typeid/tag tracking, instead:
-small_idfor unspecified type tags is set to 0, thus reference to unspecified type tag would render BTF id of avoidon recode phase;
- unspecified type tags are not added tocu->types_table.
Quoted 7 lines
[1] https://lore.kernel.org/all/Y0R7uu3s%[email protected]/
[2]bcc648a10cbc("btf_encoder: EncodeDW_TAG_unspecified_typereturning routines as void")
[3]cffe5e1f75e1("core: Record if a CU has aDW_TAG_unspecified_type")
[4]75e0fe28bb02("core: AddDW_TAG_unspecified_typetotag__is_tag_type()set")Fixes bcc648a10cbc ("btf_encoder: Encode DW_TAG_unspecified_type returning routines as void")Signed-off-by Eduard Zingerman <[email protected]>
Thank you so much Eduard, this worked:
* No duplicate BTF ID warnings
* No 15 minute BTF ID generation
* Matt's reproducer loads successfully.
I had a sneaky suspicion that it was these unspecified types, which is why my hacky patch which got unspecified types out of the way got things to *mostly* work.
Eduard Zingerman
maintainerQuoted 73 lines · KP Singh
Quoted 59 lines · Eduard Zingerman
On Mon, 2023-02-27 at 11:31 -0800, Andrii Nakryiko wrote:
[...]Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
I'd start with understanding what BTF and DWARF differences are causing the issue before trying to come up with the fix. For that we don't even need config or repro steps, it should be enough to share vmlinux with BTF and DWARF, and start from there.
Yes, I suspect that there is some kind of unanticipated anomaly for some DWARF encoding for some kind of objects, just need to find the root for the diverging type hierarchies.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
But I'm sure Eduard is on top of this already (especially that he can repro the issue now).
I'm working on it, nothing to report yet, but I'm working on it.
Thanks, please keep us posted!
It is interesting how everything is interconnected. The patch for pahole below happens to help. I prepared it last week while working on new DWARF encoding scheme for
btf_type_tag.I still need to track down which "
unspecified_type" entries caused the issue in this particular case. Will post an update tomorrow.Meanwhile, Matt, KP, could you please verify the patch on your side? It is for the "next" branch of pahole.
---
From
09fac63ca08e25aea499f827283b07cc87a7daabMon Sep 17 00:00:00 2001
From: Eduard Zingerman <[email protected]>
Date: Tue, 21 Feb 2023 19:23:00 +0200
Subject: [PATCH]dwarf_loader: Fix for BTF id drift caused by adding
unspecified typesRecent changes to handle unspecified types (see [1]) cause BTF ID drift.
Specifically, the intent of commits [2], [3] and [4] is to render references to unspecified types as void type.
However, as a consequence:
- indie__process_unit()call tocu__add_tag()allocatessmall_idfor unspecified type tags and adds these tags tocu->types_table;
-btf_encoder__encode_tag()skips generation of BTF entries forDW_TAG_unspecified_typetags.Such logic causes ID drift if unspecified type is not the last type processed for compilation unit.
small_idof each type following unspecified type in thecu->types_tablewould have its BTF id off by -1. Thus renders references established on recode phase invalid.This commit reverts
unspecified_typeid/tag tracking, instead:
-small_idfor unspecified type tags is set to 0, thus reference to unspecified type tag would render BTF id of avoidon recode phase;
- unspecified type tags are not added tocu->types_table.Quoted 7 lines
[1] https://lore.kernel.org/all/Y0R7uu3s%[email protected]/
[2]bcc648a10cbc("btf_encoder: EncodeDW_TAG_unspecified_typereturning routines as void")
[3]cffe5e1f75e1("core: Record if a CU has aDW_TAG_unspecified_type")
[4]75e0fe28bb02("core: AddDW_TAG_unspecified_typetotag__is_tag_type()set")Fixes bcc648a10cbc ("btf_encoder: Encode DW_TAG_unspecified_type returning routines as void")Signed-off-by Eduard Zingerman <[email protected]>Tested-by KP Singh <[email protected]>Reported-by Matt Bobrowski <[email protected]>Thank you so much Eduard, this worked:
Quoted 7 lines
* No duplicate BTF ID warnings
* No 15 minute BTF ID generation
* Matt's reproducer loads successfully.I had a sneaky suspicion that it was these unspecified types, which is why my hacky patch which got unspecified types out of the way got things to *mostly* work.
Hi KP,
Thanks a lot for testing!
I found the root cause for the bug (took me longer than I would like to admit...). Using the patch below the reproducer from Matt works as expected and warnings are gone.
Still, I think that my patch from yesterday is a more general approach, as it correctly handles unspecified types that occur in non-tail position, so I'll post that one.
Thanks,
Eduard
---
From daa53248e8a5087edbceaffe1fad51f9eb06e922 Mon Sep 17 00:00:00 2001
From: Eduard Zingerman <[email protected]>
Date: Tue, 28 Feb 2023 19:44:22 +0200
Subject: [PATCH] btf_encoder: reset encoder->unspecified_type for each CU
The field encoder->unspecified_type is set but not reset by function btf_encoder__encode_cu() when processed cu has unspecified type. The following sequence of events might occur when BTF encoding is requested:
- CU with unspecified type is processed:
- unspecified type id is 42
- encoder->unspecified_type is set to 42
- CU without unspecified type is processed next using the same encoder object:
- some struct foo has id 42 in this CU
- the references to struct foo are set 0 by function
`btf_encoder__tag_type()`.
This commit sets encoder->unspecified_type to 0 when CU does not have unspecified type.
This issue was reported in thread [1].
See also [2].
[1] https://lore.kernel.org/bpf/Y%[email protected]/
[2] https://lore.kernel.org/all/Y0R7uu3s%[email protected]/
--- btf_encoder.c | 2 ++ 1 file changed, 2 insertions(+)
btf_encoder.c +2 -0
@@ -1748,6 +1748,8 @@ int btf_encoder__encode_cu(struct btf_encoder *encoder, struct cu *cu, struct co encoder->type_id_off =btf__type_cnt (encoder->btf) -1 ;if (encoder->cu->unspecified_type.tag) encoder->unspecified_type = encoder->cu->unspecified_type.type; +else + encoder->unspecified_type =0 ;if (!encoder->has_index_type) {/* cu__find_base_type_by_name() takes "type_id_t *id" */
Signature · 1 line
2.39.1
Quoted 87 lines · Eduard Zingerman
Quoted 73 lines · KP Singh
Quoted 59 lines · Eduard Zingerman
On Mon, 2023-02-27 at 11:31 -0800, Andrii Nakryiko wrote:
[...]Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
I'd start with understanding what BTF and DWARF differences are causing the issue before trying to come up with the fix. For that we don't even need config or repro steps, it should be enough to share vmlinux with BTF and DWARF, and start from there.
Yes, I suspect that there is some kind of unanticipated anomaly for some DWARF encoding for some kind of objects, just need to find the root for the diverging type hierarchies.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
But I'm sure Eduard is on top of this already (especially that he can repro the issue now).
I'm working on it, nothing to report yet, but I'm working on it.
Thanks, please keep us posted!
It is interesting how everything is interconnected. The patch for pahole below happens to help. I prepared it last week while working on new DWARF encoding scheme for
btf_type_tag.I still need to track down which "
unspecified_type" entries caused the issue in this particular case. Will post an update tomorrow.Meanwhile, Matt, KP, could you please verify the patch on your side? It is for the "next" branch of pahole.
---
From
09fac63ca08e25aea499f827283b07cc87a7daabMon Sep 17 00:00:00 2001
From: Eduard Zingerman <[email protected]>
Date: Tue, 21 Feb 2023 19:23:00 +0200
Subject: [PATCH]dwarf_loader: Fix for BTF id drift caused by adding
unspecified typesRecent changes to handle unspecified types (see [1]) cause BTF ID drift.
Specifically, the intent of commits [2], [3] and [4] is to render references to unspecified types as void type.
However, as a consequence:
- indie__process_unit()call tocu__add_tag()allocatessmall_idfor unspecified type tags and adds these tags tocu->types_table;
-btf_encoder__encode_tag()skips generation of BTF entries forDW_TAG_unspecified_typetags.Such logic causes ID drift if unspecified type is not the last type processed for compilation unit.
small_idof each type following unspecified type in thecu->types_tablewould have its BTF id off by -1. Thus renders references established on recode phase invalid.This commit reverts
unspecified_typeid/tag tracking, instead:
-small_idfor unspecified type tags is set to 0, thus reference to unspecified type tag would render BTF id of avoidon recode phase;
- unspecified type tags are not added tocu->types_table.Quoted 7 lines
[1] https://lore.kernel.org/all/Y0R7uu3s%[email protected]/
[2]bcc648a10cbc("btf_encoder: EncodeDW_TAG_unspecified_typereturning routines as void")
[3]cffe5e1f75e1("core: Record if a CU has aDW_TAG_unspecified_type")
[4]75e0fe28bb02("core: AddDW_TAG_unspecified_typetotag__is_tag_type()set")Fixes bcc648a10cbc ("btf_encoder: Encode DW_TAG_unspecified_type returning routines as void")Signed-off-by Eduard Zingerman <[email protected]>Tested-by KP Singh <[email protected]>Reported-by Matt Bobrowski <[email protected]>Thank you so much Eduard, this worked:
Quoted 7 lines
* No duplicate BTF ID warnings
* No 15 minute BTF ID generation
* Matt's reproducer loads successfully.I had a sneaky suspicion that it was these unspecified types, which is why my hacky patch which got unspecified types out of the way got things to *mostly* work.
Hi KP,
Thanks a lot for testing!
Quoted 7 lines
I found the root cause for the bug (took me longer than I would like to admit...). Using the patch below the reproducer from Matt works as expected and warnings are gone.
Still, I think that my patch from yesterday is a more general approach, as it correctly handles unspecified types that occur in non-tail position, so I'll post that one.
I agree, please send it to dwarves@ as a proper patch. Thank you for digging into this and fixing it quickly!
Quoted 54 lines
Thanks,
Eduard---
From
daa53248e8a5087edbceaffe1fad51f9eb06e922Mon Sep 17 00:00:00 2001
From: Eduard Zingerman <[email protected]>
Date: Tue, 28 Feb 2023 19:44:22 +0200
Subject: [PATCH]btf_encoder: resetencoder->unspecified_typefor each CUThe field
encoder->unspecified_typeis set but not reset by functionbtf_encoder__encode_cu()when processedcuhas unspecified type. The following sequence of events might occur when BTF encoding is requested:
- CU with unspecified type is processed:
- unspecified type id is 42
-encoder->unspecified_typeis set to 42
- CU without unspecified type is processed next using the sameencoderobject:
- somestruct foohas id 42 in this CU
- the references tostruct fooare set 0 by function`btf_encoder__tag_type()`.This commit sets
encoder->unspecified_typeto 0 when CU does not have unspecified type.This issue was reported in thread [1].
See also [2].
[1] https://lore.kernel.org/bpf/Y%[email protected]/
[2] https://lore.kernel.org/all/Y0R7uu3s%[email protected]/Fixes 52b25808e44a ("btf_encoder: Store type_id_off, unspecified type in encoder")Reported-by Matt Bobrowski <[email protected]>Signed-off-by Eduard Zingerman <[email protected]>--- btf_encoder.c | 2 ++ 1 file changed, 2 insertions(+)btf_encoder.c +2
-1@@ -1748,6 +1748,8 @@ int btf_encoder__encode_cu(struct btf_encoder *encoder, struct cu *cu, struct co encoder->type_id_off =btf__type_cnt (encoder->btf) -1 ;if (encoder->cu->unspecified_type.tag) encoder->unspecified_type = encoder->cu->unspecified_type.type; +else + encoder->unspecified_type =0 ;if (!encoder->has_index_type) {/* cu__find_base_type_by_name() takes "type_id_t *id" */ --2.39.1