selftests/bpf: DENYLIST.aarch64: Remove fexit_sleep
Subsystems (3)
PATCH 1/1 selftests/bpf: DENYLIST.aarch64: Remove fexit_sleep
fexit_sleep test runs successfully now on the CI so remove it from the deny list.
--- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 - 1 file changed, 1 deletion(-)
tools/testing/selftests/bpf/DENYLIST.aarch64 +0 -1
@@ -1,6 +1,5 @@ bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 -fexit_sleep # The test never returns. The remaining tests cannot start. kprobe_multi_bench_attach # needs CONFIG_FPROBE kprobe_multi_test # needs CONFIG_FPROBE module_attach # prog 'kprobe_multi': failed to auto-attach: -95
Signature · 1 line
2.40.1
Daniel Borkmann
maintainerQuoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.
Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Quoted 17 lines
Signed-off-by Puranjay Mohan <[email protected]>--- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 - 1 file changed, 1 deletion(-)tools/testing/selftests/bpf/DENYLIST.aarch64 +0
-1@@ -1,6 +1,5 @@ bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 -fexit_sleep # The test never returns. The remaining tests cannot start. kprobe_multi_bench_attach # needs CONFIG_FPROBE kprobe_multi_test # needs CONFIG_FPROBE module_attach # prog 'kprobe_multi': failed to auto-attach: -95
Quoted 6 lines · Daniel Borkmann
Quoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Actually, I never saw this test failing on my local setup and yesterday I tried running it on the CI where it passed as well. So, I assumed that this would be fixed by some commit. I am not sure which exact commit might have fixed this.
Manu, Martin
When this was added to the deny list was this failing every time and did you have some reproducer for this. If there is a reproducer, I can try fixing it but when ran normally this test never fails for me.
Thanks,
Puranjay
Quoted 15 lines · Puranjay Mohan
Quoted 6 lines · Daniel Borkmann
Quoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Actually, I never saw this test failing on my local setup and yesterday I tried running it on the CI where it passed as well. So, I assumed that this would be fixed by some commit. I am not sure which exact commit might have fixed this.
Quoted 6 lines
Manu, Martin
When this was added to the deny list was this failing every time and did you have some reproducer for this. If there is a reproducer, I can try fixing it but when ran normally this test never fails for me.
I think this never worked until
https://lore.kernel.org/lkml/[email protected]/
was merged, FTrace direct calls was blocking tracing programs on ARM, since then it has always worked.
- KP
Quoted 2 lines
Thanks,
Puranjay
Daniel Borkmann
maintainerQuoted 18 lines · KP Singh
Quoted 17 lines · Puranjay Mohan
Quoted 6 lines · Daniel Borkmann
Quoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Actually, I never saw this test failing on my local setup and yesterday I tried running it on the CI where it passed as well. So, I assumed that this would be fixed by some commit. I am not sure which exact commit might have fixed this.
Manu, Martin
Quoted 8 lines
Quoted 3 lines
When this was added to the deny list was this failing every time and did you have some reproducer for this. If there is a reproducer, I can try fixing it but when ran normally this test never fails for me.
I think this never worked until
https://lore.kernel.org/lkml/[email protected]/
was merged, FTrace direct calls was blocking tracing programs on ARM, since then it has always worked.
Awesome, thanks! I'll add this to the commit desc then when applying.
Quoted 24 lines · Daniel Borkmann
Quoted 22 lines · KP Singh
Quoted 20 lines · Puranjay Mohan
Quoted 6 lines · Daniel Borkmann
Quoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Actually, I never saw this test failing on my local setup and yesterday I tried running it on the CI where it passed as well. So, I assumed that this would be fixed by some commit. I am not sure which exact commit might have fixed this.
Manu, Martin
When this was added to the deny list was this failing every time and did you have some reproducer for this. If there is a reproducer, I can try fixing it but when ran normally this test never fails for me.
Quoted 6 lines
Quoted 4 lines
I think this never worked until
https://lore.kernel.org/lkml/[email protected]/
was merged, FTrace direct calls was blocking tracing programs on ARM, since then it has always worked.Awesome, thanks! I'll add this to the commit desc then when applying.
I originally removed that test from the denylist:
https://lore.kernel.org/lkml/[email protected]/
But then it was re-introduced by Martin here:
https://lore.kernel.org/bpf/CABRcYmJZ2uUQ4S9rqm+H0N9otjDBv5v45tGjRGKfX2+GZ9gxbw@mail.gmail.com/T/#mc65a794a852bb8b6850cc98be09a90cdc8c76c06
I moved on to work on different things and never investigated it. I don't know what would have broken it and what would have fixed it. I don't remember having observed this "test never returns" problem myself.
Quoted 23 lines · Daniel Borkmann
Quoted 22 lines · KP Singh
Quoted 20 lines · Puranjay Mohan
Quoted 6 lines · Daniel Borkmann
Quoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Actually, I never saw this test failing on my local setup and yesterday I tried running it on the CI where it passed as well. So, I assumed that this would be fixed by some commit. I am not sure which exact commit might have fixed this.
Manu, Martin
When this was added to the deny list was this failing every time and did you have some reproducer for this. If there is a reproducer, I can try fixing it but when ran normally this test never fails for me.
Quoted 6 lines
Quoted 4 lines
I think this never worked until
https://lore.kernel.org/lkml/[email protected]/
was merged, FTrace direct calls was blocking tracing programs on ARM, since then it has always worked.Awesome, thanks! I'll add this to the commit desc then when applying.
The commit that added this to the deny list said:31f4f810d533 ("selftests/bpf: Add fexit_sleep to DENYLIST.aarch64")
``` It is reported that the fexit_sleep never returns in aarch64. The remaining tests cannot start. ```
So, if the lack of Ftrace direct calls would be the reason then the failure would be due to fexit programs not being supported on arm64.
But this says that the selftest never returns therefore is not related to ftrace direct call support but another bug?
Thanks,
Puranjay
Daniel Borkmann
maintainerQuoted 40 lines · Puranjay Mohan
Quoted 23 lines · Daniel Borkmann
Quoted 22 lines · KP Singh
Quoted 20 lines · Puranjay Mohan
Quoted 6 lines · Daniel Borkmann
Quoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Actually, I never saw this test failing on my local setup and yesterday I tried running it on the CI where it passed as well. So, I assumed that this would be fixed by some commit. I am not sure which exact commit might have fixed this.
Manu, Martin
When this was added to the deny list was this failing every time and did you have some reproducer for this. If there is a reproducer, I can try fixing it but when ran normally this test never fails for me.
Quoted 6 lines
Quoted 4 lines
I think this never worked until
https://lore.kernel.org/lkml/[email protected]/
was merged, FTrace direct calls was blocking tracing programs on ARM, since then it has always worked.Awesome, thanks! I'll add this to the commit desc then when applying.
The commit that added this to the deny list said:
31f4f810d533("selftests/bpf: Addfexit_sleepto DENYLIST.aarch64")``` It is reported that the fexit_sleep never returns in aarch64. The remaining tests cannot start. ```
Quoted 5 lines
So, if the lack of Ftrace direct calls would be the reason then the failure would be due to fexit programs not being supported on arm64.
But this says that the selftest never returns therefore is not related to ftrace direct call support but another bug?
Fwiw, at least it is passing in the BPF CI now.
https://github.com/kernel-patches/bpf/actions/runs/9841781347/job/27169610006
Quoted 34 lines · Daniel Borkmann
Quoted 32 lines · Puranjay Mohan
Quoted 29 lines · Daniel Borkmann
Quoted 18 lines · KP Singh
Quoted 17 lines · Puranjay Mohan
Quoted 6 lines · Daniel Borkmann
Quoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Actually, I never saw this test failing on my local setup and yesterday I tried running it on the CI where it passed as well. So, I assumed that this would be fixed by some commit. I am not sure which exact commit might have fixed this.
Manu, Martin
Quoted 8 lines
Quoted 3 lines
When this was added to the deny list was this failing every time and did you have some reproducer for this. If there is a reproducer, I can try fixing it but when ran normally this test never fails for me.
I think this never worked until
https://lore.kernel.org/lkml/[email protected]/
was merged, FTrace direct calls was blocking tracing programs on ARM, since then it has always worked.Awesome, thanks! I'll add this to the commit desc then when applying.
Quoted 7 lines
Quoted 7 lines
The commit that added this to the deny list said:
31f4f810d533("selftests/bpf: Addfexit_sleepto DENYLIST.aarch64")``` It is reported that the fexit_sleep never returns in aarch64. The remaining tests cannot start. ```
It may also have something to do with sleepable programs. But I think it's generally in the category of "BPF tracing was catching up with ARM", it has now.
- KP
Quoted 10 lines
Quoted 6 lines
So, if the lack of Ftrace direct calls would be the reason then the failure would be due to fexit programs not being supported on arm64.
But this says that the selftest never returns therefore is not related to ftrace direct call support but another bug?
Fwiw, at least it is passing in the BPF CI now.
https://github.com/kernel-patches/bpf/actions/runs/9841781347/job/27169610006
Daniel Borkmann
maintainerQuoted 34 lines · KP Singh
Quoted 33 lines · Daniel Borkmann
Quoted 32 lines · Puranjay Mohan
Quoted 21 lines · Daniel Borkmann
Quoted 20 lines · KP Singh
Quoted 18 lines · Puranjay Mohan
Quoted 6 lines · Daniel Borkmann
Quoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Actually, I never saw this test failing on my local setup and yesterday I tried running it on the CI where it passed as well. So, I assumed that this would be fixed by some commit. I am not sure which exact commit might have fixed this.
Manu, Martin
When this was added to the deny list was this failing every time and did you have some reproducer for this. If there is a reproducer, I can try fixing it but when ran normally this test never fails for me.
Quoted 6 lines
Quoted 4 lines
I think this never worked until
https://lore.kernel.org/lkml/[email protected]/
was merged, FTrace direct calls was blocking tracing programs on ARM, since then it has always worked.Awesome, thanks! I'll add this to the commit desc then when applying.
The commit that added this to the deny list said:
31f4f810d533("selftests/bpf: Addfexit_sleepto DENYLIST.aarch64")
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
``` It is reported that the fexit_sleep never returns in aarch64. The remaining tests cannot start. ```It may also have something to do with sleepable programs. But I think it's generally in the category of "BPF tracing was catching up with ARM", it has now.
Hm, the latest run actually hangs in fexit_sleep (which is the test right after fexit_bpf2bpf). So looks like this was too early. It seems some CI runs pass on arm64 but others fail:
https://github.com/kernel-patches/bpf/actions/runs/9859826851/job/27224868398 (fail)
https://github.com/kernel-patches/bpf/actions/runs/9859837213/job/27224955045 (pass)
Puranjay, do you have a chance to look into this again?
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
So, if the lack of Ftrace direct calls would be the reason then the failure would be due to fexit programs not being supported on arm64.
But this says that the selftest never returns therefore is not related to ftrace direct call support but another bug?
Fwiw, at least it is passing in the BPF CI now.
https://github.com/kernel-patches/bpf/actions/runs/9841781347/job/27169610006
________________________________________
From: Daniel Borkmann <[email protected]>
Sent: Tuesday, July 9, 2024 10:44 AM
To: KP Singh <[email protected]>
Cc: Puranjay Mohan <[email protected]>; Andrii Nakryiko <[email protected]>; Eduard Zingerman <[email protected]>; Mykola Lysenko <[email protected]>; Alexei Starovoitov <[email protected]>; Martin KaFai Lau <[email protected]>; Song Liu <[email protected]>; Yonghong Song <[email protected]>; John Fastabend <[email protected]>; Stanislav Fomichev <[email protected]>; Hao Luo <[email protected]>; Jiri Olsa <[email protected]>; Shuah Khan <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; Manu Bretelle <[email protected]>; Florent Revest <[email protected]> Subject: Re: [PATCH bpf] selftests/bpf: DENYLIST.aarch64: Remove fexit_sleep
Quoted 34 lines · KP Singh
Quoted 33 lines · Daniel Borkmann
Quoted 32 lines · Puranjay Mohan
Quoted 21 lines · Daniel Borkmann
Quoted 20 lines · KP Singh
Quoted 18 lines · Puranjay Mohan
Quoted 6 lines · Daniel Borkmann
Quoted 2 lines · Puranjay Mohan
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Do you happen to know which commit fixed it? If yes, might be nice to have it documented in the commit message.
Actually, I never saw this test failing on my local setup and yesterday I tried running it on the CI where it passed as well. So, I assumed that this would be fixed by some commit. I am not sure which exact commit might have fixed this.
Manu, Martin
When this was added to the deny list was this failing every time and did you have some reproducer for this. If there is a reproducer, I can try fixing it but when ran normally this test never fails for me.
Quoted 6 lines
Quoted 4 lines
I think this never worked until
https://lore.kernel.org/lkml/[email protected]/
was merged, FTrace direct calls was blocking tracing programs on ARM, since then it has always worked.Awesome, thanks! I'll add this to the commit desc then when applying.
The commit that added this to the deny list said:
31f4f810d533("selftests/bpf: Addfexit_sleepto DENYLIST.aarch64")
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
``` It is reported that the fexit_sleep never returns in aarch64. The remaining tests cannot start. ```It may also have something to do with sleepable programs. But I think it's generally in the category of "BPF tracing was catching up with ARM", it has now.
Hm, the latest run actually hangs in fexit_sleep (which is the test right after fexit_bpf2bpf). So looks like this was too early. It seems some CI runs pass on arm64 but others fail:
https://github.com/kernel-patches/bpf/actions/runs/9859826851/job/27224868398 (fail)
https://github.com/kernel-patches/bpf/actions/runs/9859837213/job/27224955045 (pass)
Puranjay, do you have a chance to look into this again?
Probably unrelated... but when I tried to reproduce this using qemu in full emulation mode [0], I am getting a kernel crash for fexit_sleep, but also for fexit_bpf2bpf, fentry_fexit
stacktraces look like (for fentry_fexit)
root@(none):/mnt/vmtest/selftests/bpf# ./test_progs -v -t fentry_fexit bpf_testmod.ko is already unloaded. Loading bpf_testmod.ko... Successfully loaded bpf_testmod.ko. test_fentry_fexit:PASS:fentry_skel_load 0 nsec test_fentry_fexit:PASS:fexit_skel_load 0 nsec
test_fentry_fexit:PASS:fentry_attach 0 nsec
test_fentry_fexit:PASS:fexit_attach 0 nsec
Unable to handle kernel paging request at virtual address ffff0000c2a80e68
Mem abort info:
ESR = 0x0000000096000004
EC = 0x25: DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC = 0x04: level 0 translation fault
Data abort info:
ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
CM = 0, WnR = 0, TnD = 0, TagAccess = 0
GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
swapper pgtable: 4k pages, 52-bit VAs, pgdp=0000000041b4a000 [ffff0000c2a80e68] pgd=1000000042f28003, p4d=0000000000000000
Internal error: Oops: 0000000096000004 [#1] SMP Modules linked in: bpf_testmod(OE) [last unloaded: bpf_testmod(OE)] CPU: 0 PID: 97 Comm: test_progs Tainted: G OE 6.10.0-rc6-gb0eedd920017-dirty #67 Hardware name: linux,dummy-virt (DT) pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--) pc : __bpf_tramp_enter+0x58/0x190 lr : __bpf_tramp_enter+0xd8/0x190
sp : ffff800084afbc10
x29: ffff800084afbc10 x28: fff00000c28c2e80 x27: 0000000000000000
x26: 0000000000000000 x25: 0000000000000050 x24: 0000000000000000
x23: 000000000000000a x22: fff00000c28c2e80 x21: 0000ffffed100070
x20: ffff800082032938 x19: ffff0000c2a80c00 x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffffed100070
x14: 0000000000000000 x13: ffff800082032938 x12: 0000000000000000
x11: 0000000000020007 x10: 0000000000000007 x9 : 00000000ffffffff x8 : 0000000000004008 x7 : ffff80008218fa78 x6 : 0000000000000000 x5 : 0000000000000001 x4 : 0000000086db7919 x3 : 0000000095481a34 x2 : 0000000000000001 x1 : fff00000c28c2e80 x0 : 0000000000000001
Call trace: __bpf_tramp_enter+0x58/0x190 bpf_trampoline_6442499844+0x44/0x158 bpf_fentry_test1+0x8/0x10 bpf_prog_test_run_tracing+0x190/0x328 __sys_bpf+0x844/0x2148 __arm64_sys_bpf+0x2c/0x48 invoke_syscall+0x4c/0x118 el0_svc_common.constprop.0+0x48/0xf0 do_el0_svc+0x24/0x38 el0_svc+0x4c/0x120 el0t_64_sync_handler+0xc0/0xc8 el0t_64_sync+0x190/0x198 Code: 52800001 97f9f3df 942a3be8 35000400 (f9413660) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops: Fatal exception
SMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x00,00000006,8c13bd78,576676af
Memory Limit: none
For "fexit_sleep" and "fexit_bpf2bpf" respectively:
$ ( cd 9859826851 && vmtest -k kbuild-output/arch/arm64/boot/Image.gz -r ../aarch64-rootfs -a aarch64 '/bin/mount bpffs /sys/fs/bpf -t bpf && ip link set lo up && cd /mnt/vmtest/selftests/bpf/ && ./test_progs -v -t fexit_sleep' ) => Image.gz ===> Booting ===> Setting up VM ===> Running command root@(none):/# bpf_testmod: loading out-of-tree module taints kernel. bpf_testmod: module verification failed: signature and/or required key missing - tainting kernel Unable to handle kernel paging request at virtual address ffff0000c19c2668 Mem abort info: ESR = 0x0000000096000004 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x04: level 0 translation fault Data abort info: ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 CM = 0, WnR = 0, TnD = 0, TagAccess = 0 GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 swapper pgtable: 4k pages, 52-bit VAs, pgdp=0000000041b4a000 [ffff0000c19c2668] pgd=1000000042f28003, p4d=0000000000000000 Internal error: Oops: 0000000096000004 [#1] SMP Modules linked in: bpf_testmod(OE) CPU: 1 PID: 91 Comm: test_progs Tainted: G OE 6.10.0-rc6-gb0eedd920017-dirty #67 Hardware name: linux,dummy-virt (DT) pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--) pc : __bpf_tramp_enter+0x58/0x190 lr : __bpf_tramp_enter+0xd8/0x190 sp : ffff800084c4bda0 x29: ffff800084c4bda0 x28: fff00000c274ae80 x27: 0000000000000000 x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 x23: 0000000060001000 x22: 0000ffffa36b7a54 x21: 00000000ffffffff x20: ffff800082032938 x19: ffff0000c19c2400 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: ffff800082032938 x12: 0000000000000000 x11: 0000000000020007 x10: 0000000000000007 x9 : 00000000ffffffff x8 : 0000000000004008 x7 : ffff80008218fa78 x6 : 0000000000000000 x5 : 0000000000000001 x4 : 0000000086db7919 x3 : 0000000095481a34 x2 : 0000000000000001 x1 : fff00000c274ae80 x0 : 0000000000000001 Call trace: __bpf_tramp_enter+0x58/0x190 bpf_trampoline_6442487232+0x44/0x158 __arm64_sys_nanosleep+0x8/0xf0 invoke_syscall+0x4c/0x118 el0_svc_common.constprop.0+0x48/0xf0 do_el0_svc+0x24/0x38 el0_svc+0x4c/0x120 el0t_64_sync_handler+0xc0/0xc8 el0t_64_sync+0x190/0x198 Code: 52800001 97f9f3df 942a3be8 35000400 (f9413660) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops: Fatal exception SMP: stopping secondary CPUs Kernel Offset: disabled CPU features: 0x00,00000006,8c13bd78,576676af Memory Limit: none Failed to run command
Caused by:
0: Failed to QGA guest-exec-status
1: error running guest_exec_status
2: Broken pipe (os error 32)
3: Broken pipe (os error 32)
[11:46:14] chantra@devvm17937:scratchpad $
[11:47:56] chantra@devvm17937:scratchpad $
[11:47:57] chantra@devvm17937:scratchpad $ ( cd 9859826851 && vmtest -k kbuild-output/arch/arm64/boot/Image.gz -r ../aarch64-rootfs -a aarch64 '/bin/mount bpffs /sys/fs/bpf -t bpf && ip link set lo up && cd /mnt/vmtest/selftests/bpf/ && ./test_progs -v -t fexit_bpf2bpf' ) => Image.gz
===> Booting
===> Setting up VM
===> Running command
root@(none):/# bpf_testmod: loading out-of-tree module taints kernel. bpf_testmod: module verification failed: signature and/or required key missing - tainting kernel Unable to handle kernel paging request at virtual address ffff0000c278de68 Mem abort info: ESR = 0x0000000096000004 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x04: level 0 translation fault Data abort info: ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 CM = 0, WnR = 0, TnD = 0, TagAccess = 0 GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 swapper pgtable: 4k pages, 52-bit VAs, pgdp=0000000041b4a000 [ffff0000c278de68] pgd=1000000042f28003, p4d=0000000000000000 Internal error: Oops: 0000000096000004 [#1] SMP Modules linked in: bpf_testmod(OE) CPU: 1 PID: 87 Comm: test_progs Tainted: G OE 6.10.0-rc6-gb0eedd920017-dirty #67 Hardware name: linux,dummy-virt (DT) pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--) pc : __bpf_tramp_enter+0x58/0x190 lr : __bpf_tramp_enter+0xd8/0x190 sp : ffff800084c4ba90 x29: ffff800084c4ba90 x28: ffff800080a32d10 x27: ffff800080a32d80 x26: ffff8000813e0ad8 x25: ffff800084c4bce4 x24: ffff800082fbd048 x23: 0000000000000001 x22: fff00000c2732e80 x21: fff00000c18a3200 x20: ffff800082032938 x19: ffff0000c278dc00 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000aaaabcc22aa0 x14: 0000000000000000 x13: ffff800082032938 x12: 0000000000000000 x11: 0000000000000000 x10: 000000000ac0d5af x9 : 000000000ac0d5af x8 : 00000000a4d7a457 x7 : ffff80008218fa78 x6 : 0000000000000000 x5 : 0000000000000002 x4 : 0000000006fa0785 x3 : 0000000081d7cd4c x2 : 0000000000000202 x1 : fff00000c2732e80 x0 : 0000000000000001 Call trace: __bpf_tramp_enter+0x58/0x190 bpf_trampoline_34359738386+0x44/0xf8 bpf_prog_3b052b77318ab7c4_test_pkt_md_access+0x8/0x118 bpf_test_run+0x200/0x3a0 bpf_prog_test_run_skb+0x328/0x6d8 __sys_bpf+0x844/0x2148 __arm64_sys_bpf+0x2c/0x48 invoke_syscall+0x4c/0x118 el0_svc_common.constprop.0+0x48/0xf0 do_el0_svc+0x24/0x38 el0_svc+0x4c/0x120 el0t_64_sync_handler+0xc0/0xc8 el0t_64_sync+0x190/0x198 Code: 52800001 97f9f3df 942a3be8 35000400 (f9413660) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops: Fatal exception in interrupt SMP: stopping secondary CPUs Kernel Offset: disabled CPU features: 0x00,00000006,8c13bd78,576676af Memory Limit: none Failed to run command
Caused by:
0: Failed to QGA guest-exec-status
1: error running guest_exec_status
2: Broken pipe (os error 32)
3: Broken pipe (os error 32)
[0] https://chantra.github.io/bpfcitools/bpfci-troubleshooting.html
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
So, if the lack of Ftrace direct calls would be the reason then the failure would be due to fexit programs not being supported on arm64.
But this says that the selftest never returns therefore is not related to ftrace direct call support but another bug?
Fwiw, at least it is passing in the BPF CI now.
https://github.com/kernel-patches/bpf/actions/runs/9841781347/job/27169610006
[SNIP]
Quoted 204 lines
Hm, the latest run actually hangs in
fexit_sleep(which is the test right afterfexit_bpf2bpf). So looks like this was too early. It seems some CI runs pass on arm64 but others fail:https://github.com/kernel-patches/bpf/actions/runs/9859826851/job/27224868398 (fail)
https://github.com/kernel-patches/bpf/actions/runs/9859837213/job/27224955045 (pass)Puranjay, do you have a chance to look into this again?
Probably unrelated... but when I tried to reproduce this using qemu in full emulation mode [0], I am getting a kernel crash for
fexit_sleep, but also forfexit_bpf2bpf,fentry_fexitstacktraces look like (for
fentry_fexit)root@(none):/mnt/vmtest/selftests/bpf# ./test_progs -v -t fentry_fexit bpf_testmod.ko is already unloaded. Loading bpf_testmod.ko... Successfully loaded bpf_testmod.ko. test_fentry_fexit:PASS:fentry_skel_load 0 nsec test_fentry_fexit:PASS:fexit_skel_load 0 nsectest_fentry_fexit:PASS:fentry_attach 0 nsec
test_fentry_fexit:PASS:fexit_attach 0 nsecUnable to handle kernel paging request at virtual address ffff0000c2a80e68Mem abort info:
ESR =0x0000000096000004
EC =0x25: DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
FSC =0x04: level 0 translation fault
Data abort info:
ISV = 0, ISS =0x00000004, ISS2 =0x00000000
CM = 0, WnR = 0, TnD = 0, TagAccess = 0
GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
swapper pgtable: 4k pages, 52-bit VAs, pgdp=0000000041b4a000 [ffff0000c2a80e68] pgd=1000000042f28003, p4d=0000000000000000Internal error: Oops: 0000000096000004 [#1] SMP Modules linked in: bpf_testmod(OE) [last unloaded: bpf_testmod(OE)] CPU: 0 PID: 97 Comm: test_progs Tainted: G OE 6.10.0-rc6-gb0eedd920017-dirty #67 Hardware name: linux,dummy-virt (DT) pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--) pc : __bpf_tramp_enter+0x58/0x190 lr : __bpf_tramp_enter+0xd8/0x190sp :
ffff800084afbc10
x29:ffff800084afbc10x28:fff00000c28c2e80x27: 0000000000000000
x26: 0000000000000000 x25: 0000000000000050 x24: 0000000000000000
x23:000000000000000ax22:fff00000c28c2e80x21:0000ffffed100070
x20:ffff800082032938x19:ffff0000c2a80c00x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000000 x15:0000ffffed100070
x14: 0000000000000000 x13:ffff800082032938x12: 0000000000000000
x11: 0000000000020007 x10: 0000000000000007 x9 :00000000ffffffffx8 : 0000000000004008 x7 :ffff80008218fa78x6 : 0000000000000000 x5 : 0000000000000001 x4 :0000000086db7919x3 :0000000095481a34x2 : 0000000000000001 x1 :fff00000c28c2e80x0 : 0000000000000001Call trace: __bpf_tramp_enter+0x58/0x190 bpf_trampoline_6442499844+0x44/0x158 bpf_fentry_test1+0x8/0x10 bpf_prog_test_run_tracing+0x190/0x328 __sys_bpf+0x844/0x2148 __arm64_sys_bpf+0x2c/0x48 invoke_syscall+0x4c/0x118 el0_svc_common.constprop.0+0x48/0xf0 do_el0_svc+0x24/0x38 el0_svc+0x4c/0x120 el0t_64_sync_handler+0xc0/0xc8 el0t_64_sync+0x190/0x198 Code: 52800001 97f9f3df 942a3be8 35000400 (f9413660) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops: Fatal exceptionSMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x00,00000006,8c13bd78,576676af
Memory Limit: noneFor "
fexit_sleep" and "fexit_bpf2bpf" respectively:$ ( cd 9859826851 && vmtest -k kbuild-output/arch/arm64/boot/Image.gz -r ../aarch64-rootfs -a aarch64 '/bin/mount bpffs /sys/fs/bpf -t bpf && ip link set lo up && cd /mnt/vmtest/selftests/bpf/ && ./test_progs -v -t fexit_sleep' ) => Image.gz ===> Booting ===> Setting up VM ===> Running command root@(none):/# bpf_testmod: loading out-of-tree module taints kernel. bpf_testmod: module verification failed: signature and/or required key missing - tainting kernel Unable to handle kernel paging request at virtual address ffff0000c19c2668 Mem abort info: ESR = 0x0000000096000004 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x04: level 0 translation fault Data abort info: ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 CM = 0, WnR = 0, TnD = 0, TagAccess = 0 GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 swapper pgtable: 4k pages, 52-bit VAs, pgdp=0000000041b4a000 [ffff0000c19c2668] pgd=1000000042f28003, p4d=0000000000000000 Internal error: Oops: 0000000096000004 [#1] SMP Modules linked in: bpf_testmod(OE) CPU: 1 PID: 91 Comm: test_progs Tainted: G OE 6.10.0-rc6-gb0eedd920017-dirty #67 Hardware name: linux,dummy-virt (DT) pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--) pc : __bpf_tramp_enter+0x58/0x190 lr : __bpf_tramp_enter+0xd8/0x190 sp : ffff800084c4bda0 x29: ffff800084c4bda0 x28: fff00000c274ae80 x27: 0000000000000000 x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 x23: 0000000060001000 x22: 0000ffffa36b7a54 x21: 00000000ffffffff x20: ffff800082032938 x19: ffff0000c19c2400 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: ffff800082032938 x12: 0000000000000000 x11: 0000000000020007 x10: 0000000000000007 x9 : 00000000ffffffff x8 : 0000000000004008 x7 : ffff80008218fa78 x6 : 0000000000000000 x5 : 0000000000000001 x4 : 0000000086db7919 x3 : 0000000095481a34 x2 : 0000000000000001 x1 : fff00000c274ae80 x0 : 0000000000000001 Call trace: __bpf_tramp_enter+0x58/0x190 bpf_trampoline_6442487232+0x44/0x158 __arm64_sys_nanosleep+0x8/0xf0 invoke_syscall+0x4c/0x118 el0_svc_common.constprop.0+0x48/0xf0 do_el0_svc+0x24/0x38 el0_svc+0x4c/0x120 el0t_64_sync_handler+0xc0/0xc8 el0t_64_sync+0x190/0x198 Code: 52800001 97f9f3df 942a3be8 35000400 (f9413660) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops: Fatal exception SMP: stopping secondary CPUs Kernel Offset: disabled CPU features: 0x00,00000006,8c13bd78,576676af Memory Limit: none Failed to run commandCaused by:
0: Failed to QGA guest-exec-status
1: error runningguest_exec_status
2: Broken pipe (os error 32)
3: Broken pipe (os error 32)
[11:46:14] chantra@devvm17937:scratchpad $
[11:47:56] chantra@devvm17937:scratchpad $
[11:47:57] chantra@devvm17937:scratchpad $ ( cd 9859826851 && vmtest -k kbuild-output/arch/arm64/boot/Image.gz -r ../aarch64-rootfs -a aarch64 '/bin/mountbpffs/sys/fs/bpf-t bpf && ip link set lo up && cd/mnt/vmtest/selftests/bpf/&& ./test_progs -v -tfexit_bpf2bpf' ) => Image.gz
===> Booting
===> Setting up VM
===> Running commandroot@(none):/# bpf_testmod: loading out-of-tree module taints kernel. bpf_testmod: module verification failed: signature and/or required key missing - tainting kernel Unable to handle kernel paging request at virtual address ffff0000c278de68 Mem abort info: ESR = 0x0000000096000004 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x04: level 0 translation fault Data abort info: ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 CM = 0, WnR = 0, TnD = 0, TagAccess = 0 GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 swapper pgtable: 4k pages, 52-bit VAs, pgdp=0000000041b4a000 [ffff0000c278de68] pgd=1000000042f28003, p4d=0000000000000000 Internal error: Oops: 0000000096000004 [#1] SMP Modules linked in: bpf_testmod(OE) CPU: 1 PID: 87 Comm: test_progs Tainted: G OE 6.10.0-rc6-gb0eedd920017-dirty #67 Hardware name: linux,dummy-virt (DT) pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--) pc : __bpf_tramp_enter+0x58/0x190 lr : __bpf_tramp_enter+0xd8/0x190 sp : ffff800084c4ba90 x29: ffff800084c4ba90 x28: ffff800080a32d10 x27: ffff800080a32d80 x26: ffff8000813e0ad8 x25: ffff800084c4bce4 x24: ffff800082fbd048 x23: 0000000000000001 x22: fff00000c2732e80 x21: fff00000c18a3200 x20: ffff800082032938 x19: ffff0000c278dc00 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000aaaabcc22aa0 x14: 0000000000000000 x13: ffff800082032938 x12: 0000000000000000 x11: 0000000000000000 x10: 000000000ac0d5af x9 : 000000000ac0d5af x8 : 00000000a4d7a457 x7 : ffff80008218fa78 x6 : 0000000000000000 x5 : 0000000000000002 x4 : 0000000006fa0785 x3 : 0000000081d7cd4c x2 : 0000000000000202 x1 : fff00000c2732e80 x0 : 0000000000000001 Call trace: __bpf_tramp_enter+0x58/0x190 bpf_trampoline_34359738386+0x44/0xf8 bpf_prog_3b052b77318ab7c4_test_pkt_md_access+0x8/0x118 bpf_test_run+0x200/0x3a0 bpf_prog_test_run_skb+0x328/0x6d8 __sys_bpf+0x844/0x2148 __arm64_sys_bpf+0x2c/0x48 invoke_syscall+0x4c/0x118 el0_svc_common.constprop.0+0x48/0xf0 do_el0_svc+0x24/0x38 el0_svc+0x4c/0x120 el0t_64_sync_handler+0xc0/0xc8 el0t_64_sync+0x190/0x198 Code: 52800001 97f9f3df 942a3be8 35000400 (f9413660) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops: Fatal exception in interrupt SMP: stopping secondary CPUs Kernel Offset: disabled CPU features: 0x00,00000006,8c13bd78,576676af Memory Limit: none Failed to run command
Quoted 8 lines
Caused by:
0: Failed to QGA guest-exec-status
1: error runningguest_exec_status
2: Broken pipe (os error 32)
3: Broken pipe (os error 32)[0] https://chantra.github.io/bpfcitools/bpfci-troubleshooting.html
Thanks for sharing the logs,
I will try to reproduce this and find the root cause.
Thanks,
Puranjay
Hi,
I was able find the root cause of this bug and will send a fix soon!
Quoted 1 line
Unable to handle kernel paging request at virtual address ffff0000c2a80e68
We are running this test on Qemu with '-cpu max', this means 52-bit virtual addresses are being used.
The trampolines generation code has the following two lines:
emit_addr_mov_i64(A64_R(0), (const u64)im, ctx); emit_call((const u64)__bpf_tramp_enter, ctx);
here the address of struct bpf_tramp_image is moved to R0 and passed as an argument to __bpf_tramp_enter().
emit_addr_mov_i64() assumes that the address passed to it is in the vmalloc space and uses at most 48 bits. It sets all the remaining bits to 1.
but struct bpf_tramp_image is allocated using kzalloc() and when 52-bit VAs are used, its address is not guaranteed to be 48-bit, therefore we see this bug, where 0xfff[0]0000c2a80e68 is converted to 0xfff[f]0000c2a80e68 when the trampoline is generated.
The fix would be use emit_a64_mov_i64() for moving this address into R0.
Thanks,
Puranjay
Daniel Borkmann
maintainerQuoted 21 lines · Puranjay Mohan
Hi,
I was able find the root cause of this bug and will send a fix soon!Quoted 1 line
Unable to handle kernel paging request at virtual address ffff0000c2a80e68We are running this test on Qemu with '-cpu max', this means 52-bit virtual addresses are being used.
The trampolines generation code has the following two lines:
emit_addr_mov_i64(A64_R(0), (const u64)im, ctx); emit_call((const u64)__bpf_tramp_enter, ctx);here the address of struct
bpf_tramp_imageis moved to R0 and passed as an argument to__bpf_tramp_enter().
emit_addr_mov_i64()assumes that the address passed to it is in the vmalloc space and uses at most 48 bits. It sets all the remaining bits to 1.
Quoted 6 lines
but struct
bpf_tramp_imageis allocated usingkzalloc()and when 52-bit VAs are used, its address is not guaranteed to be 48-bit, therefore we see this bug, where 0xfff[0]0000c2a80e68 is converted to 0xfff[f]0000c2a80e68 when the trampoline is generated.The fix would be use
emit_a64_mov_i64()for moving this address into R0.
Excellent find!
Daniel Borkmann
maintainerHi Puranjay,
On 7/11/24 4:00 PM, Puranjay Mohan wrote:
[...]
Quoted 19 lines
I was able find the root cause of this bug and will send a fix soon!
Quoted 1 line
Unable to handle kernel paging request at virtual address ffff0000c2a80e68We are running this test on Qemu with '-cpu max', this means 52-bit virtual addresses are being used.
The trampolines generation code has the following two lines:
emit_addr_mov_i64(A64_R(0), (const u64)im, ctx); emit_call((const u64)__bpf_tramp_enter, ctx);here the address of struct
bpf_tramp_imageis moved to R0 and passed as an argument to__bpf_tramp_enter().
emit_addr_mov_i64()assumes that the address passed to it is in the vmalloc space and uses at most 48 bits. It sets all the remaining bits to 1.
Quoted 6 lines
but struct
bpf_tramp_imageis allocated usingkzalloc()and when 52-bit VAs are used, its address is not guaranteed to be 48-bit, therefore we see this bug, where 0xfff[0]0000c2a80e68 is converted to 0xfff[f]0000c2a80e68 when the trampoline is generated.The fix would be use
emit_a64_mov_i64()for moving this address into R0.
It looks like there is still an issue left. A recent CI run on bpf-next is still hitting the same on arm64:
Base:
https://github.com/kernel-patches/bpf/commits/series/870746%3D%3Ebpf-next/
CI:
https://github.com/kernel-patches/bpf/actions/runs/9905842936/job/27366435436
[...]
#89/11 fexit_bpf2bpf/func_replace_global_func:OK #89/12 fexit_bpf2bpf/fentry_to_cgroup_bpf:OK #89/13 fexit_bpf2bpf/func_replace_progmap:OK #89 fexit_bpf2bpf:OK
Error: The operation was canceled.
Thanks,
Daniel
Quoted 40 lines · Daniel Borkmann
Hi Puranjay,
On 7/11/24 4:00 PM, Puranjay Mohan wrote:
[...]Quoted 19 lines
I was able find the root cause of this bug and will send a fix soon!
Quoted 1 line
Unable to handle kernel paging request at virtual address ffff0000c2a80e68We are running this test on Qemu with '-cpu max', this means 52-bit virtual addresses are being used.
The trampolines generation code has the following two lines:
emit_addr_mov_i64(A64_R(0), (const u64)im, ctx); emit_call((const u64)__bpf_tramp_enter, ctx);here the address of struct
bpf_tramp_imageis moved to R0 and passed as an argument to__bpf_tramp_enter().
emit_addr_mov_i64()assumes that the address passed to it is in the vmalloc space and uses at most 48 bits. It sets all the remaining bits to 1.Quoted 6 lines
but struct
bpf_tramp_imageis allocated usingkzalloc()and when 52-bit VAs are used, its address is not guaranteed to be 48-bit, therefore we see this bug, where 0xfff[0]0000c2a80e68 is converted to 0xfff[f]0000c2a80e68 when the trampoline is generated.The fix would be use
emit_a64_mov_i64()for moving this address into R0.It looks like there is still an issue left. A recent CI run on bpf-next is still hitting the same on arm64:
Base:
https://github.com/kernel-patches/bpf/commits/series/870746%3D%3Ebpf-next/CI:
Quoted 8 lines
https://github.com/kernel-patches/bpf/actions/runs/9905842936/job/27366435436 [...] #89/11 fexit_bpf2bpf/func_replace_global_func:OK #89/12 fexit_bpf2bpf/fentry_to_cgroup_bpf:OK #89/13 fexit_bpf2bpf/func_replace_progmap:OK #89 fexit_bpf2bpf:OK Error: The operation was canceled.
Let's denylist that test again for now?
Daniel Borkmann
maintainerQuoted 40 lines · Alexei Starovoitov
Quoted 39 lines · Daniel Borkmann
On 7/11/24 4:00 PM, Puranjay Mohan wrote:
[...]Quoted 19 lines
I was able find the root cause of this bug and will send a fix soon!
Quoted 1 line
Unable to handle kernel paging request at virtual address ffff0000c2a80e68We are running this test on Qemu with '-cpu max', this means 52-bit virtual addresses are being used.
The trampolines generation code has the following two lines:
emit_addr_mov_i64(A64_R(0), (const u64)im, ctx); emit_call((const u64)__bpf_tramp_enter, ctx);here the address of struct
bpf_tramp_imageis moved to R0 and passed as an argument to__bpf_tramp_enter().
emit_addr_mov_i64()assumes that the address passed to it is in the vmalloc space and uses at most 48 bits. It sets all the remaining bits to 1.Quoted 6 lines
but struct
bpf_tramp_imageis allocated usingkzalloc()and when 52-bit VAs are used, its address is not guaranteed to be 48-bit, therefore we see this bug, where 0xfff[0]0000c2a80e68 is converted to 0xfff[f]0000c2a80e68 when the trampoline is generated.The fix would be use
emit_a64_mov_i64()for moving this address into R0.It looks like there is still an issue left. A recent CI run on bpf-next is still hitting the same on arm64:
Base:
https://github.com/kernel-patches/bpf/commits/series/870746%3D%3Ebpf-next/CI:
https://github.com/kernel-patches/bpf/actions/runs/9905842936/job/27366435436
Quoted 8 lines
Quoted 6 lines
[...] #89/11 fexit_bpf2bpf/func_replace_global_func:OK #89/12 fexit_bpf2bpf/fentry_to_cgroup_bpf:OK #89/13 fexit_bpf2bpf/func_replace_progmap:OK #89 fexit_bpf2bpf:OK Error: The operation was canceled.Let's denylist that test again for now?
Agree, done/pushed now.
Hi Daniel, Manu
I was able to reproduce this issue on KVM and found the root cause for this hang! The other issue that we fixed is unrelated to this hang and doesn't occur on self hosted github runners as they use 48-bit VAs.
The userspace test code has:
#define STACK_SIZE (1024 * 1024)
static char child_stack[STACK_SIZE];
cpid = clone(do_sleep, child_stack + STACK_SIZE, CLONE_FILES | SIGCHLD, fexit_skel);arm64 requires the stack pointer to be 16 byte aligned otherwise SPAlignmentFault occurs, this appears as Bus error in the userspace.
The stack provided to the clone system call is not guaranteed to be aligned properly in this selftest.
The test hangs on the following line:
while (READ_ONCE(fexit_skel->bss->fentry_cnt) != 2);
Because the child process is killed due to SPAlignmentFault, the fentry_cnt remains at 0!
Reading the man page of clone system call, the correct way to allocate stack for this call is using mmap like this:
stack = mmap(NULL, STACK_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_STACK, -1, 0);
This fixes the issue, I will send a patch to use this and once again remove this test from DENYLIST and I hope this time it fixes it for good.
Quoted 9 lines
It looks like there is still an issue left. A recent CI run on bpf-next is still hitting the same on arm64:
Base:
https://github.com/kernel-patches/bpf/commits/series/870746%3D%3Ebpf-next/CI:
Quoted 8 lines
https://github.com/kernel-patches/bpf/actions/runs/9905842936/job/27366435436 [...] #89/11 fexit_bpf2bpf/func_replace_global_func:OK #89/12 fexit_bpf2bpf/fentry_to_cgroup_bpf:OK #89/13 fexit_bpf2bpf/func_replace_progmap:OK #89 fexit_bpf2bpf:OK Error: The operation was canceled.
Thanks,
Puranjay
Quoted 26 lines · Puranjay Mohan
Hi Daniel, Manu
I was able to reproduce this issue on KVM and found the root cause for this hang! The other issue that we fixed is unrelated to this hang and doesn't occur on self hosted github runners as they use 48-bit VAs.The userspace test code has:
#define STACK_SIZE (1024 * 1024) static char child_stack[STACK_SIZE]; cpid = clone(do_sleep, child_stack + STACK_SIZE, CLONE_FILES | SIGCHLD, fexit_skel);arm64 requires the stack pointer to be 16 byte aligned otherwise SPAlignmentFault occurs, this appears as Bus error in the userspace.
The stack provided to the clone system call is not guaranteed to be aligned properly in this selftest.
The test hangs on the following line:
while (READ_ONCE(fexit_skel->bss->fentry_cnt) != 2);Because the child process is killed due to SPAlignmentFault, the
fentry_cntremains at 0!
Quoted 7 lines
Reading the man page of clone system call, the correct way to allocate stack for this call is using mmap like this:
stack =
mmap(NULL, STACK_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_STACK, -1, 0);This fixes the issue, I will send a patch to use this and once again remove this test from DENYLIST and I hope this time it fixes it for good.
Wow. Great find. Good to know. prog_tests/ns_current_pid_tgid.c has the same issue probably.
Quoted 30 lines · Alexei Starovoitov
Quoted 29 lines · Puranjay Mohan
Hi Daniel, Manu
I was able to reproduce this issue on KVM and found the root cause for this hang! The other issue that we fixed is unrelated to this hang and doesn't occur on self hosted github runners as they use 48-bit VAs.The userspace test code has:
#define STACK_SIZE (1024 * 1024) static char child_stack[STACK_SIZE]; cpid = clone(do_sleep, child_stack + STACK_SIZE, CLONE_FILES | SIGCHLD, fexit_skel);arm64 requires the stack pointer to be 16 byte aligned otherwise SPAlignmentFault occurs, this appears as Bus error in the userspace.
The stack provided to the clone system call is not guaranteed to be aligned properly in this selftest.
The test hangs on the following line:
while (READ_ONCE(fexit_skel->bss->fentry_cnt) != 2);Because the child process is killed due to SPAlignmentFault, the
fentry_cntremains at 0!Reading the man page of clone system call, the correct way to allocate stack for this call is using mmap like this:
Quoted 7 lines
Quoted 4 lines
stack =
mmap(NULL, STACK_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_STACK, -1, 0);This fixes the issue, I will send a patch to use this and once again remove this test from DENYLIST and I hope this time it fixes it for good.
Wow. Great find. Good to know.
prog_tests/ns_current_pid_tgid.chas the same issue probably.
Yes, I checked that test as well using gdb and fortunately it gets a 16 byte aligned stack pointer, but this is just luck, so I will send a patch to fix that test as well.
Thanks,
Puranjay
________________________________________
From: Puranjay Mohan
Sent: Thursday, July 11, 2024 7:00 AM
To: Manu Bretelle; Daniel Borkmann; KP Singh
Cc: Andrii Nakryiko; Eduard Zingerman; Mykola Lysenko; Alexei Starovoitov; Martin KaFai Lau; Song Liu; Yonghong Song; John Fastabend; Stanislav Fomichev; Hao Luo; Jiri Olsa; Shuah Khan; [email protected]; [email protected]; [email protected]; Florent Revest Subject: Re: [PATCH bpf] selftests/bpf: DENYLIST.aarch64: Remove fexit_sleep
Hi,
I was able find the root cause of this bug and will send a fix soon!
Quoted 1 line
Unable to handle kernel paging request at virtual address ffff0000c2a80e68
I was able to confirm the fix using the artifacts from https://github.com/kernel-patches/bpf/actions/runs/9905842936
Thanks
Manu
Hi Manu,
Quoted 4 lines
I was able to confirm the fix using the artifacts from https://github.com/kernel-patches/bpf/actions/runs/9905842936
Thanks
Thanks for testing the fix.
This bug has been resolved now but the test still hangs sometimes. Unfortunately, I am not able to reproduce this hang
using vmtest. Can you extract some logs from the CI somehow?? If it is hanging in the kernel there should be some
soft lockup or RCU lockup related messages.
I was talking about this with Kumar and we think that this test is hanging in the userspace in the following loop:
while (READ_ONCE(fexit_skel->bss->fentry_cnt) != 2);
Could it be that fentry_cnt is > 2 somehow before we reach this?? This is only a random guess though.
Thanks,
Puranjay
Quoted 9 lines · Puranjay Mohan
Thanks for testing the fix.
This bug has been resolved now but the test still hangs sometimes. Unfortunately, I am not able to reproduce this hang using vmtest.
I have not able to reproduce the original error either. I will try to reproduce on the actual CI host next week (which are native arm64 host compared to my local setup which uses full emulation).
Quoted 3 lines
Can you extract some logs from the CI somehow?? If it is hanging in the kernel there should be some
soft lockup or RCU lockup related messages.
I think once we execute the test, vmtest does not track the console logs anymore, so we won’t see those. That should be fixable, but for now, I won’t be able to get more logs than you get from the UI currently.
Quoted 9 lines
I was talking about this with Kumar and we think that this test is hanging in the userspace in the following loop:
while (
READ_ONCE(fexit_skel->bss->fentry_cnt)!= 2);Could it be that
fentry_cntis > 2 somehow before we reach this?? This is only a random guess though.
Manu
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Hello:
This patch was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Quoted 7 lines · On Fri, 5 Jul 2024 14:50:09 +0000 you wrote:
fexit_sleeptest runs successfully now on the CI so remove it from the deny list.Signed-off-by Puranjay Mohan <[email protected]>--- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 - 1 file changed, 1 deletion(-)
Here is the summary with links:
- [bpf] selftests/bpf: DENYLIST.aarch64: Remove fexit_sleep
https://git.kernel.org/bpf/bpf-next/c/90dc946059b7
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html