s390x: selftests/bpf are failing on CI
Ihor Solodrai
reviewerHi Ilya,
After recent merges from upstream, CI started failing both on bpf and bpf-next trees. Yonghong Song and Song Liu submitted a couple of fixes that are already applied to bpf tree, but there are still failures on s390x.
https://github.com/kernel-patches/bpf/actions/runs/14163772245
Could you please investigate?
A snippet from the log:
#11 atomics:OK
trace_kprobe: Could not probe notrace function __s390x_sys_bpf
trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep
trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep
trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep
trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep
trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep test_attach_probe:PASS:skel_open 0 nsec
test_attach_probe:PASS:skel_load 0 nsec
test_attach_probe:PASS:check_bss 0 nsec test_attach_probe_manual:PASS:skel_kprobe_manual_open_and_load 0 nsec test_attach_probe_manual:PASS:uprobe_offset 0 nsec
libbpf: prog 'handle_kprobe': failed to create kprobe '__s390x_sys_nanosleep+0x0' perf event: -EINVAL test_attach_probe_manual:FAIL:attach_kprobe unexpected error: -22
#12/1 attach_probe/manual-default:FAIL
test_attach_probe_manual:PASS:skel_kprobe_manual_open_and_load 0 nsec test_attach_probe_manual:PASS:uprobe_offset 0 nsec
libbpf: failed to add legacy kprobe event for '__s390x_sys_nanosleep+0x0': -EINVAL
libbpf: prog 'handle_kprobe': failed to create kprobe '__s390x_sys_nanosleep+0x0' perf event: -EINVAL test_attach_probe_manual:FAIL:attach_kprobe unexpected error: -22
#12/2 attach_probe/manual-legacy:FAIL
test_attach_probe_manual:PASS:skel_kprobe_manual_open_and_load 0 nsec test_attach_probe_manual:PASS:uprobe_offset 0 nsec
libbpf: prog 'handle_kprobe': failed to create kprobe '__s390x_sys_nanosleep+0x0' perf event: -EINVAL test_attach_probe_manual:FAIL:attach_kprobe unexpected error: -22
#12/3 attach_probe/manual-perf:FAIL
test_attach_probe_manual:PASS:skel_kprobe_manual_open_and_load 0 nsec test_attach_probe_manual:PASS:uprobe_offset 0 nsec
libbpf: prog 'handle_kprobe': failed to create kprobe '__s390x_sys_nanosleep+0x0' perf event: -EINVAL test_attach_probe_manual:FAIL:attach_kprobe unexpected error: -22
#12/4 attach_probe/manual-link:FAIL
libbpf: prog 'handle_kprobe_auto': failed to create kprobe '__se_sys_nanosleep+0x0' perf event: -ENOENT test_attach_probe_auto:FAIL:attach_kprobe_auto unexpected error: -2
libbpf: prog 'handle_kretprobe_auto': failed to create kretprobe '__se_sys_nanosleep+0x0' perf event: -ENOENT test_attach_probe_auto:FAIL:attach_kretprobe_auto unexpected error: -2 test_attach_probe_auto:PASS:auto-attach should fail for old-style name 0 nsec test_attach_probe_auto:PASS:attach_uretprobe_byname 0 nsec test_attach_probe_auto:FAIL:check_kprobe_auto_res unexpected check_kprobe_auto_res: actual 0 != expected 11 test_attach_probe_auto:FAIL:check_kretprobe_auto_res unexpected check_kretprobe_auto_res: actual 0 != expected 22 test_attach_probe_auto:PASS:check_uretprobe_byname_res 0 nsec
#12/5 attach_probe/auto:FAIL
Ihor Solodrai
reviewerQuoted 31 lines · iii
Quoted 7 lines · Ihor Solodrai
Hi Ilya,
After recent merges from upstream, CI started failing both on bpf and bpf-next trees. Yonghong Song and Song Liu submitted a couple of fixes that are already applied to bpf tree, but there are still failures on s390x.
Quoted 5 lines
https://github.com/kernel-patches/bpf/actions/runs/14163772245
Could you please investigate?
[...]
Hi Ihor,
Thanks for the heads up.
I tried this manually with the kernel commit07be1f644ff9and the tests are passing:# ./test_progs -t attach_probe#12/1 attach_probe/manual-default:OK #12/2 attach_probe/manual-legacy:OK #12/3 attach_probe/manual-perf:OK #12/4 attach_probe/manual-link:OK #12/5 attach_probe/auto:OK #12/6 attach_probe/kprobe-sleepable:OK #12/7 attach_probe/uprobe-lib:OK #12/8 attach_probe/uprobe-sleepable:OK #12/9 attach_probe/uprobe-ref_ctr:OK #12 attach_probe:OK
Quoted 4 lines
Summary: 1/9 PASSED, 0 SKIPPED, 0 FAILED
So this must be a config issue.
I'm not sure what is causing__s390x_sys_nanosleepto be notrace, but this doesn't look normal.
Hi, thank you for taking a look. Do they succeed if you run prior tests too? I remember situations when a test would succeed when run independently, but fail when running after other tests.
Quoted 6 lines
I also see that the newer test runs are green:
https://github.com/kernel-patches/bpf/actions/runs/14182247375
Do you know if something changed in the meantime?
Yeah, I just disabled the tests failing on s390x for now to unblock the CI. This is temporary.
A little off-topic: it looks like ebpf runners are offline again, could be due to recent github runner version bump.
Quoted 3 lines
Best regards,
Ilya
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 18 lines · iii
Quoted 17 lines · iii
Quoted 16 lines · Ihor Solodrai
Quoted 15 lines · iii
Quoted 14 lines · Ihor Solodrai
Hi Ilya,
After recent merges from upstream, CI started failing both on bpf and
bpf-next trees. Yonghong Song and Song Liu submitted a couple of fixes
that are already applied to bpf tree, but there are still failures on
s390x.https://github.com/kernel-patches/bpf/actions/runs/14163772245
Could you please investigate?
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[...]
I could not reproduce this on a s390x machine, so I downloaded the CI artifacts and noticed that
/sys/kernel/debug/tracing/available_filter_functionsis empty. Turns out this is caused by cross-compilation.
I have let our kernel team (on Cc:) know and they will investigate this further.
There must be an endianness issue in the sorttable tool. Reverting just the commit fa1518875286 ("s390: Sort mcount locations at build time") fixes the issue. I'll take a look.
Ihor Solodrai
reviewerQuoted 28 lines · Vasily Gorbik
Quoted 18 lines · iii
Quoted 17 lines · iii
Quoted 16 lines · Ihor Solodrai
Quoted 15 lines · iii
Quoted 14 lines · Ihor Solodrai
Hi Ilya,
After recent merges from upstream, CI started failing both on bpf and
bpf-next trees. Yonghong Song and Song Liu submitted a couple of fixes
that are already applied to bpf tree, but there are still failures on
s390x.https://github.com/kernel-patches/bpf/actions/runs/14163772245
Could you please investigate?
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
[...]
I could not reproduce this on a s390x machine, so I downloaded the CI artifacts and noticed that
/sys/kernel/debug/tracing/available_filter_functionsis empty. Turns out this is caused by cross-compilation.
I have let our kernel team (on Cc:) know and they will investigate this further.
Quoted 3 lines
There must be an endianness issue in the sorttable tool. Reverting just the commit
fa1518875286("s390: Sort mcount locations at build time") fixes the issue. I'll take a look.
Hi Vasily.
I can confirm that reverting fa1518875286 fixes the selftests failures. I added a revert commit as a CI-specific patch for now.
Please keep me in cc.
Thank you for the help!
Ihor Solodrai
reviewerQuoted 5 lines · iii
Quoted 3 lines · Ihor Solodrai
[...]
A little off-topic: it looks like ebpf runners are offline again, could be due to recent github runner version bump.
Quoted 8 lines
The new docker image looks malformed:
$ docker run -it --rm ghcr.io/kernel-patches/runner:main-noble-s390x exec /entrypoint.sh: exec format error$ docker run -it --rm --entrypoint=/bin/sh ghcr.io/kernel-patches/runner:main-noble-s390x # cat /entrypoint.sh 404: Not Found
Please make sure you're using an up-to-date image.
Digest: sha256:946298c96c51af666f8747be3afcf9a8e59483c40cf790cd94939abf6ec22969
A malformed image was pushed on 2025-03-20, then reverted and later updated correctly.
If you're using the same ansible setup that we do, restarting the systemd service controlling the runners should help.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Best regards,
Ilya