running bpftool build tests in CI
Hi,
as part of the cleanup/automation effort in the BPF selfests directory, I am now taking a look at test_bpftool_build.sh, which ensures that the different supported ways of building bpftool work correctly. As this script does not really exercize anything at runtime but rather at build time, I'd like to propose to introduce a dedicated step in the CI automation that already builds and run selftests. I have opened two PRs in kernel-patches vmtest ([0]) and libbpf/ci ([1]), hoping I am not confusing which code should go where between the different repositories kept in sync with each other. I have also opened a dummy PR on kernel-patches/bpf ([2], not to be merged) that shows how this test would look like in CI.
This really is a RFC, as not all tests from test_bpftool_build.sh are being executed: I suspect those based on .config to be currently broken, but that can be handled as a second step, depending on the chosen direction.
Any comment welcome !
Alexis
[0] https://github.com/kernel-patches/vmtest/pull/517
[1] https://github.com/libbpf/ci/pull/236
[2] https://github.com/kernel-patches/bpf/pull/13368
Signature · 3 lines
Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Ihor Solodrai
reviewerQuoted 13 lines · Alexis Lothoré
Hi,
as part of the cleanup/automation effort in the BPF selfests directory, I am now taking a look attest_bpftool_build.sh, which ensures that the different supported ways of building bpftool work correctly. As this script does not really exercize anything at runtime but rather at build time, I'd like to propose to introduce a dedicated step in the CI automation that already builds and run selftests. I have opened two PRs in kernel-patches vmtest ([0]) and libbpf/ci ([1]), hoping I am not confusing which code should go where between the different repositories kept in sync with each other. I have also opened a dummy PR on kernel-patches/bpf ([2], not to be merged) that shows how this test would look like in CI.
Quoted 6 lines
This really is a RFC, as not all tests from
test_bpftool_build.share being executed: I suspect those based on .config to be currently broken, but that can be handled as a second step, depending on the chosen direction.Any comment welcome !
Hi Alexis, thank you for working on this.
I am a little confused about what are we trying to test here and why. Maybe you could explain.
Some variants of bpftool build are CI-exercised regularly:
* selftests/bpf directly depend on bpftool, it's built and is used as part of the main "test_progs*" suite
* bpftool has a standalone github mirror with it's own CI pipeline:
https://github.com/libbpf/bpftool
Looking at test_bpftool_build.sh (written 6y ago btw), it seems to be more of a Makefile infra test than bpftool test specifically.
Given how complicated some in-tree tools/selftests/ Makefiles are [1], running infra tests like this would be nice. But then maybe we should develop them beyond test_bpftool_build.sh?
I briefly skimmed over the CI PRs you've submitted, and one thing I would say is that we don't want a "bpftool build" test to block the kernel build when it fails. It either needs to be a separate job, or a step that is allowed to fail.
Vineet recently did a similar thing, where the gcc-bpf selftests build was interleaved with the kernel build and I nacked it with the same justification [2].
I'm thinking the whole "kernel build" workflow should be refactored into either separate selftests build jobs, or some post processing that looks at what artifacts where built successfully and reports failures. This will provide a little more friendly UI/UX. You're welcome to look into that if you're interested. But that's not directly relevant to the changes you're proposing.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://github.com/kernel-patches/vmtest/pull/502#discussion_r3724931742
Hi Ihor,
thanks for the feedback
Quoted 19 lines · Ihor Solodrai
Quoted 18 lines · Alexis Lothoré
Hi,
as part of the cleanup/automation effort in the BPF selfests directory, I am now taking a look attest_bpftool_build.sh, which ensures that the different supported ways of building bpftool work correctly. As this script does not really exercize anything at runtime but rather at build time, I'd like to propose to introduce a dedicated step in the CI automation that already builds and run selftests. I have opened two PRs in kernel-patches vmtest ([0]) and libbpf/ci ([1]), hoping I am not confusing which code should go where between the different repositories kept in sync with each other. I have also opened a dummy PR on kernel-patches/bpf ([2], not to be merged) that shows how this test would look like in CI.This really is a RFC, as not all tests from
test_bpftool_build.share being executed: I suspect those based on .config to be currently broken, but that can be handled as a second step, depending on the chosen direction.
Quoted 6 lines
Quoted 1 line
Any comment welcome !
Hi Alexis, thank you for working on this.
I am a little confused about what are we trying to test here and why. Maybe you could explain.
Sure, I may have been a bit light on this. The test_bpftool_build.sh located in tools/testing/selftests/bpf focuses on the various ways of building bpftool:
- through kbuild (eg: make tools/bpf)
- by changing make execution dir (eg: make -C tools/bpf/bpftool)
- by running the tools/ main makefile (eg: cd tools && make bpf)
- by running the bpftool main makefile (eg: cd tools/bpf/bpftool && make)
This listing is also cross-tested with output path configuration, testing both O=<output_dir> and OUTPUT=<output_dir>.
Will be included in the relevant commits if this RFC goes further in this direction.
Quoted 13 lines
Some variants of bpftool build are CI-exercised regularly:
* selftests/bpf directly depend on bpftool, it's built and is used as part of the main "test_progs*" suite
* bpftool has a standalone github mirror with it's own CI pipeline:
https://github.com/libbpf/bpftoolLooking at
test_bpftool_build.sh(written 6y ago btw), it seems to be more of aMakefileinfra test than bpftool test specifically.Given how complicated some in-tree
tools/selftests/Makefiles are [1], running infra tests like this would be nice. But then maybe we should develop them beyondtest_bpftool_build.sh?
Quoted 8 lines
I briefly skimmed over the CI PRs you've submitted, and one thing I would say is that we don't want a "bpftool build" test to block the kernel build when it fails. It either needs to be a separate job, or a step that is allowed to fail.
Vineet recently did a similar thing, where the gcc-bpf selftests build was interleaved with the kernel build and I nacked it with the same justification [2].
Understood, thanks for the pointers. I see that Vineet came up with a new revision that eventually got merged, moving gcc-bpf as a dedicated job in kernel-build-test.yml instead of kernel-build.yml (and so, a failure on gcc-bpf does not mark kernel build as failed) I guess I can replicate this (also, making sure to add this run_tests toggle, maybe ?). However, I feel like it would not make much sense to have a build and a test part, with artifacts going from the former to the latter, as the build step actually _is_ the test. Also, the only needed artifact to run the build is the kernel source tree (we don't need any vmlinux or .config, at least for now)
Would it be ok if I try it this way ?
Quoted 6 lines
I'm thinking the whole "kernel build" workflow should be refactored into either separate selftests build jobs, or some post processing that looks at what artifacts where built successfully and reports failures. This will provide a little more friendly UI/UX. You're welcome to look into that if you're interested. But that's not directly relevant to the changes you're proposing.
Trying to rephrase it to make sure I get your point, the goal would be to have green/red checks specifically on selftests builds, rather than a global green/red check on the macro job "build kernel and selftests" ? Or does it go further than that ?
If I get it correctly, yes I'd be glad to try and help on that, as a separate task.
Alexis
Quoted 11 lines
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://github.com/kernel-patches/vmtest/pull/502#discussion_r3724931742
Signature · 3 lines
Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Ihor Solodrai
reviewerQuoted 39 lines · Alexis Lothoré
Hi Ihor,
thanks for the feedbackQuoted 19 lines · Ihor Solodrai
Quoted 18 lines · Alexis Lothoré
Hi,
as part of the cleanup/automation effort in the BPF selfests directory, I am now taking a look attest_bpftool_build.sh, which ensures that the different supported ways of building bpftool work correctly. As this script does not really exercize anything at runtime but rather at build time, I'd like to propose to introduce a dedicated step in the CI automation that already builds and run selftests. I have opened two PRs in kernel-patches vmtest ([0]) and libbpf/ci ([1]), hoping I am not confusing which code should go where between the different repositories kept in sync with each other. I have also opened a dummy PR on kernel-patches/bpf ([2], not to be merged) that shows how this test would look like in CI.This really is a RFC, as not all tests from
test_bpftool_build.share being executed: I suspect those based on .config to be currently broken, but that can be handled as a second step, depending on the chosen direction.Quoted 6 lines
Quoted 1 line
Any comment welcome !
Hi Alexis, thank you for working on this.
I am a little confused about what are we trying to test here and why. Maybe you could explain.
Sure, I may have been a bit light on this. The
test_bpftool_build.shlocated intools/testing/selftests/bpffocuses on the various ways of building bpftool:
- through kbuild (eg: maketools/bpf)
- by changing make execution dir (eg: make -Ctools/bpf/bpftool)
- by running thetools/main makefile (eg: cd tools && make bpf)
- by running the bpftool main makefile (eg: cdtools/bpf/bpftool&& make)
Quoted 2 lines
This listing is also cross-tested with output path configuration, testing both O=<output_dir> and OUTPUT=<output_dir>.
Right. That's what I meant by "it's testing makefile infra". It's checking whether any of the listed ways for initiating the build is broken.
I agree that adding tests of this kind to CI is a good idea. My point is it's worth expanding the coverage beyond the bpftool.
There was a patchset with fixes for out-of-tree build of selftests recently, for example [1]. The tests would help to catch this earlier.
[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 37 lines
Will be included in the relevant commits if this RFC goes further in this direction.
Quoted 13 lines
Some variants of bpftool build are CI-exercised regularly:
* selftests/bpf directly depend on bpftool, it's built and is used as part of the main "test_progs*" suite
* bpftool has a standalone github mirror with it's own CI pipeline:
https://github.com/libbpf/bpftoolLooking at
test_bpftool_build.sh(written 6y ago btw), it seems to be more of aMakefileinfra test than bpftool test specifically.Given how complicated some in-tree
tools/selftests/Makefiles are [1], running infra tests like this would be nice. But then maybe we should develop them beyondtest_bpftool_build.sh?Quoted 8 lines
I briefly skimmed over the CI PRs you've submitted, and one thing I would say is that we don't want a "bpftool build" test to block the kernel build when it fails. It either needs to be a separate job, or a step that is allowed to fail.
Vineet recently did a similar thing, where the gcc-bpf selftests build was interleaved with the kernel build and I nacked it with the same justification [2].
Understood, thanks for the pointers. I see that Vineet came up with a new revision that eventually got merged, moving gcc-bpf as a dedicated job in kernel-build-test.yml instead of kernel-build.yml (and so, a failure on gcc-bpf does not mark kernel build as failed) I guess I can replicate this (also, making sure to add this
run_teststoggle, maybe ?). However, I feel like it would not make much sense to have a build and a test part, with artifacts going from the former to the latter, as the build step actually _is_ the test. Also, the only needed artifact to run the build is the kernel source tree (we don't need any vmlinux or .config, at least for now)
Quoted 1 line
Would it be ok if I try it this way ?
I think if the "bpftool build" test doesn't depend on the kernel build at all, it can be implemented as a new workflow independent of everything. You don't even need new code in libbpf/ci for that, just a new .yaml in kernel-patches/vmtest/.github/workflows/, and maybe a new shell script. And it can probably run on github-provided ubuntu runners. That would be a right way to implement it, given the current state of things IMO.
Quoted 8 lines
Quoted 7 lines
I'm thinking the whole "kernel build" workflow should be refactored into either separate selftests build jobs, or some post processing that looks at what artifacts where built successfully and reports failures. This will provide a little more friendly UI/UX. You're welcome to look into that if you're interested. But that's not directly relevant to the changes you're proposing.
Quoted 7 lines
Trying to rephrase it to make sure I get your point, the goal would be to have green/red checks specifically on selftests builds, rather than a global green/red check on the macro job "build kernel and selftests" ? Or does it go further than that ?
If I get it correctly, yes I'd be glad to try and help on that, as a separate task.
Ok, let me expand on this. Thinking out loud below.
As you correctly noted, currently we have a kernel build + selftests build clumped together in a single job. I don't know if this was intentional design when it was set up, but it's what's there.
The advantage of "build many things in one job" compared to "one job - one artifact" extreme is in the cost of transition between the jobs on the performance side (artifacts upload/download, potentially new runner cold start etc.), and in the complexity of the independent jobs (you have to set up prereqs every time, for example).
The disadvantage is in the coupling of the dependencies. For example, currently if selftests/bpf build fails, veristat jobs don't run, because they depend on the combined build job. But they actually *can* run if the kernel build succeeded, but selftests build failed.
An argument can be made that if your CI run is red, you're going to dig into the logs anyways, so this is all bikeshedding. But I think a clear "kernel build failed" or "bpftool build failed" signal saves a bit of time for everyone.
An idea I have on how we could have the cake and eat it too - avoid the cost while keeping the clear signal - is to keep independent build steps in the combined job as allowed to fail, so the job always succeeds. And then have a separate check for "this artifact is missing
- that means the build failed - here is the log", either in a dependent job or as a lightweight "check" job.
It's not free though: some cost will remain in the complexity of the combined job, and the checks.
Validating this idea requires github .yaml hackery (not a fun activity in general) and experimentation. It looks doable to me, but there might be problems with it I haven't thought about.
Quoted 18 lines
Alexis
Quoted 3 lines · Ihor Solodrai
Quoted 2 lines · Alexis Lothoré
Hi Ihor,
thanks for the feedback
[...]
Quoted 16 lines
Quoted 11 lines
Sure, I may have been a bit light on this. The
test_bpftool_build.shlocated intools/testing/selftests/bpffocuses on the various ways of building bpftool:
- through kbuild (eg: maketools/bpf)
- by changing make execution dir (eg: make -Ctools/bpf/bpftool)
- by running thetools/main makefile (eg: cd tools && make bpf)
- by running the bpftool main makefile (eg: cdtools/bpf/bpftool&& make)This listing is also cross-tested with output path configuration, testing both O=<output_dir> and OUTPUT=<output_dir>.
Right. That's what I meant by "it's testing makefile infra". It's checking whether any of the listed ways for initiating the build is broken.
Quoted 8 lines
I agree that adding tests of this kind to CI is a good idea. My point is it's worth expanding the coverage beyond the bpftool.
There was a patchset with fixes for out-of-tree build of selftests recently, for example [1]. The tests would help to catch this earlier.
Ah, ok, thanks for the clarification, now I get how it would help to have the same kind of thing for selftests build (even though I am not clear about the exact list of supported cases aside from the one used in CI, but that can be sorted out later).
[...]
Quoted 13 lines
Quoted 11 lines
Understood, thanks for the pointers. I see that Vineet came up with a new revision that eventually got merged, moving gcc-bpf as a dedicated job in kernel-build-test.yml instead of kernel-build.yml (and so, a failure on gcc-bpf does not mark kernel build as failed) I guess I can replicate this (also, making sure to add this
run_teststoggle, maybe ?). However, I feel like it would not make much sense to have a build and a test part, with artifacts going from the former to the latter, as the build step actually _is_ the test. Also, the only needed artifact to run the build is the kernel source tree (we don't need any vmlinux or .config, at least for now)Quoted 1 line
Would it be ok if I try it this way ?
Quoted 7 lines
I think if the "bpftool build" test doesn't depend on the kernel build at all, it can be implemented as a new workflow independent of everything. You don't even need new code in libbpf/ci for that, just a new .yaml in kernel-patches/vmtest/.github/workflows/, and maybe a new shell script. And it can probably run on github-provided ubuntu runners. That would be a right way to implement it, given the current state of things IMO.
ACK, I'll try to apply all this decoupling from the existing workflows then.
Quoted 37 lines
Quoted 7 lines
Quoted 6 lines
I'm thinking the whole "kernel build" workflow should be refactored into either separate selftests build jobs, or some post processing that looks at what artifacts where built successfully and reports failures. This will provide a little more friendly UI/UX. You're welcome to look into that if you're interested. But that's not directly relevant to the changes you're proposing.
Quoted 7 lines
Trying to rephrase it to make sure I get your point, the goal would be to have green/red checks specifically on selftests builds, rather than a global green/red check on the macro job "build kernel and selftests" ? Or does it go further than that ?
If I get it correctly, yes I'd be glad to try and help on that, as a separate task.
Ok, let me expand on this. Thinking out loud below.
As you correctly noted, currently we have a kernel build + selftests build clumped together in a single job. I don't know if this was intentional design when it was set up, but it's what's there.
The advantage of "build many things in one job" compared to "one job - one artifact" extreme is in the cost of transition between the jobs on the performance side (artifacts upload/download, potentially new runner cold start etc.), and in the complexity of the independent jobs (you have to set up prereqs every time, for example).
The disadvantage is in the coupling of the dependencies. For example, currently if selftests/bpf build fails, veristat jobs don't run, because they depend on the combined build job. But they actually *can* run if the kernel build succeeded, but selftests build failed.
An argument can be made that if your CI run is red, you're going to dig into the logs anyways, so this is all bikeshedding. But I think a clear "kernel build failed" or "bpftool build failed" signal saves a bit of time for everyone.
Quoted 6 lines
An idea I have on how we could have the cake and eat it too - avoid the cost while keeping the clear signal - is to keep independent build steps in the combined job as allowed to fail, so the job always succeeds. And then have a separate check for "this artifact is missing
- that means the build failed - here is the log", either in a dependent job or as a lightweight "check" job.
Thanks, this clarifies a lot the direction to aim for. I now grasp the expected benefit from this potential rework.
Quoted 6 lines
It's not free though: some cost will remain in the complexity of the combined job, and the checks.
Validating this idea requires github .yaml hackery (not a fun activity in general) and experimentation. It looks doable to me, but there might be problems with it I haven't thought about.
So as stated before, unless someone picks it up before I do so, I can take a look at this and try to tweak those workflows to reach the improvements mentioned above, aiming to get clearer signals about what is failing, _while_ trying to keep complexity and CI execution time reasonable. I am definitely not an expert in github automation, so that will possibly lead to a few back and forths and experiments, but that will be an opportunity to learn :) I also have to state that I am still expected to make progress on the selftests conversion/cleanup though, but I can put this item right below in my todo list, as this is pretty related.
Signature · 3 lines
Alexis Lothoré, Bootlin Embedded Linux and Kernel engineering https://bootlin.com