Packaging bpftool and libbpf: GitHub or kernel?
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
Does bpftool work on older kernel?
Our current approach is that we (openSUSE/SLES) essentially have two version of libbpf: a public shared library that uses GitHub mirror as source, which the general userspace sees and links to; and a private static library built from kernel source used by bpftool, perf, resolve_btfids, selftests, etc. A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they took similar approach.
This approach means that the version of bpftool and libbpf are _not_ always in sync[1], which I read may causes problem since libbpf and bpftool depends on specific version of each other[2].
Using the GitHub mirror of bpftool to package both libbpf and bpftool would kept their version in sync, and was suggested[3]. Although the same could be said if we switch back to packaging libbpf from kernel source, an additional appeal for using GitHub mirrors is that it decouples bpftool from kernel, making it easily upgradable and with a clearer changelog (the latter is quite important for enterprise users) like libbpf.
The main concern with using GitHub mirror is that bpftool may be updated far beyond the version that comes with the runtime kernel. AFAIK bpftool should work on older kernel since CO-RE is used for built-in BPF iterators and the underlying libbpf work on older kernel itself. Nonetheless, it would be nice to get a confirmation from the maintainers.
Are there any other downsides to switching to GitHub mirror for bpftool?
A side note: if we want all userspace visible libbpf to have a coherent version, perf needs to use the shared libbpf library as well (either autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to backport patches to kernel source to keep up with userspace package (libbpf) changes could be a pain.
Shung-Hsi
1: In practice kernel & bpftool version often falls behind libbpf in
snapshot distro (e.g. openSUSE Leap, Ubutnu LTS), and the other way
around on rolling distro (e.g. openSUSE Tumbleweed, Arch).
2: https://lore.kernel.org/bpf/CAADnVQK-arrrNrgtu48_f--WCwR5ki2KGaX=mN2qmW_AcRyb=w@mail.gmail.com/
3: https://lore.kernel.org/bpf/[email protected]/
4: https://src.fedoraproject.org/rpms/kernel-tools/blob/82960989c918f81fcc6742a6d765afbec5fa4f74/f/kernel-tools.spec#_248
Quoted 9 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
An off-topic, yet somewhat related question that I also tried to figure out is "why the GitHub mirror for libbpf and bpftool exist at the first place?". It is a non-trivial amount of work for the maintainers after all.
For libbpf, the main uses case for GitHub seem to be for it to be used as submodule for other projects (e.g. pahole[1]), and that alone seem to suffice.
For bpftool the reason seems to be less clear[2]. From what I can tell right now its mainly use for CI (this applies to libbpf as well), which is definitely useful.
But I wonder whether packaging one of the motives to create the mirrors initially? Can't seem to find anything in this regard.
1: https://github.com/acmel/dwarves/tree/master/lib
2: https://lore.kernel.org/bpf/CAEf4BzZ+0XpH_zJ0P78vjzmFAH3kGZ21w3-LcSEG=B=+ZQWJ=w@mail.gmail.com/
Quoted 1 line
[snip]
Quoted 12 lines · Shung-Hsi Yu
Quoted 9 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
Quoted 6 lines
An off-topic, yet somewhat related question that I also tried to figure out is "why the GitHub mirror for libbpf and bpftool exist at the first place?". It is a non-trivial amount of work for the maintainers after all.
For libbpf, the main uses case for GitHub seem to be for it to be used as submodule for other projects (e.g. pahole[1]), and that alone seem to suffice.
Then it should be enough for bpftool, too :) The bcc repository uses it as a submodule, for example.
The work is non-trivial, but when compared to libbpf, I managed to preserve most of the Makefile from the kernel tree and all of the C code, and bpftool also gets patches less often.
Quoted 4 lines
For bpftool the reason seems to be less clear[2]. From what I can tell right now its mainly use for CI (this applies to libbpf as well), which is definitely useful.
Yes. At the moment, the CI present on bpftool's mirror is more limited than libbpf's. But it allows me to test some compilation variants:
regular builds, static builds, cross-compiling (to some extent). Some additional checks that would make little sense to have in the kernel repo, too. It's mostly for checking that none of these build configurations break when I sync from the kernel, and helped me find and fix several issues in the build system on the mirror.
This CI on the mirror doesn't cover bpftool's features, but these should be tested in the BPF CI itself, so we can catch regressions before patches are merged. There are some tests already, not many, I'd like to improve that someday. Anyway.
Quoted 7 lines
But I wonder whether packaging one of the motives to create the mirrors initially? Can't seem to find anything in this regard.
1: https://github.com/acmel/dwarves/tree/master/lib
2: https://lore.kernel.org/bpf/CAEf4BzZ+0XpH_zJ0P78vjzmFAH3kGZ21w3-LcSEG=B=+ZQWJ=w@mail.gmail.com/
It seems like you haven't come across this one?:
https://lore.kernel.org/bpf/[email protected]/t/
Yes, easing packaging was one of the motivations for the mirror. As mentioned in my other answer, I've not taken the time to reach out to package maintainers yet, so this hasn't really materialised at this point.
CI, indeed, was another motivation.
Submodules, or simply making it easier to hack with bpftool's code, was yet another thing. Microsoft folks intend to make bpftool compatible with eBPF for Windows. It's quite simpler to work on that from a repo which is mostly uncoupled from the Linux tree.
Perhaps the most important was to make it easier to just download, build and use bpftool, for all users who need to get the latest version, or to patch it, or to create static builds, or to cross-compile, or whatever reason might cause you to compile it from the sources. For all those cases, getting the mirror is faster and requires less space than getting the kernel repo. This makes a nice difference when periodically rebuilding images in automated workflows, for example.
Does this answer your questions?
Quentin
Hello,
Quoted 12 lines · Quentin Monnet
Quoted 11 lines · Shung-Hsi Yu
Quoted 9 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
...
Quoted 7 lines
Quoted 6 lines
But I wonder whether packaging one of the motives to create the mirrors initially? Can't seem to find anything in this regard.
1: https://github.com/acmel/dwarves/tree/master/lib
2: https://lore.kernel.org/bpf/CAEf4BzZ+0XpH_zJ0P78vjzmFAH3kGZ21w3-LcSEG=B=+ZQWJ=w@mail.gmail.com/
Quoted 7 lines
It seems like you haven't come across this one?:
https://lore.kernel.org/bpf/[email protected]/t/Yes, easing packaging was one of the motivations for the mirror. As mentioned in my other answer, I've not taken the time to reach out to package maintainers yet, so this hasn't really materialised at this point.
For me as a package maintainer submodules are a major pain. They always need special handling, break down, get out of sync between different projects using them, etc.
Somehow in the past it was possible to build and install development versions of libraries during development of tools using them, and release both when a feature was finished.
Arguably using submodules for development may work for some people. For most it would cause having the wrong submodule code when switching branches, stray submodule changes in patches, etc.
Using submodules as a general method of distributing dependencies is hell.
The usual methods of downloading and releasing the source code don't work, the submodules have to be specifically bundled.
If the dependency in question has a bug each tool author has to be coaxed to update to a fixed version and re-release, or each tool patched separately.
Over time API bitrots and is given up, each tool requiring specific and different release or git SHA of the dependency. We can see that happening a lot in ecosystems where vendoring is the norm.
Thanks
Michal
Quoted 16 lines · Quentin Monnet
Quoted 15 lines · Shung-Hsi Yu
Quoted 9 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
An off-topic, yet somewhat related question that I also tried to figure out is "why the GitHub mirror for libbpf and bpftool exist at the first place?". It is a non-trivial amount of work for the maintainers after all.
Quoted 5 lines
Quoted 2 lines
For libbpf, the main uses case for GitHub seem to be for it to be used as submodule for other projects (e.g. pahole[1]), and that alone seem to suffice.
Then it should be enough for bpftool, too :) The bcc repository uses it as a submodule, for example.
Hmm... bcc having bpftool as a submodule is a news to me. bcc having libbpf as a submodule is quite straight forward, but I didn't know bcc depends on bpftool as well.
Quoted 27 lines
The work is non-trivial, but when compared to libbpf, I managed to preserve most of the
Makefilefrom the kernel tree and all of the C code, and bpftool also gets patches less often.Quoted 4 lines
For bpftool the reason seems to be less clear[2]. From what I can tell right now its mainly use for CI (this applies to libbpf as well), which is definitely useful.
Yes. At the moment, the CI present on bpftool's mirror is more limited than libbpf's. But it allows me to test some compilation variants:
regular builds, static builds, cross-compiling (to some extent). Some additional checks that would make little sense to have in the kernel repo, too. It's mostly for checking that none of these build configurations break when I sync from the kernel, and helped me find and fix several issues in the build system on the mirror.This CI on the mirror doesn't cover bpftool's features, but these should be tested in the BPF CI itself, so we can catch regressions before patches are merged. There are some tests already, not many, I'd like to improve that someday. Anyway.
Quoted 5 lines
But I wonder whether packaging one of the motives to create the mirrors initially? Can't seem to find anything in this regard.
Quoted 5 lines
Quoted 2 lines
1: https://github.com/acmel/dwarves/tree/master/lib
2: https://lore.kernel.org/bpf/CAEf4BzZ+0XpH_zJ0P78vjzmFAH3kGZ21w3-LcSEG=B=+ZQWJ=w@mail.gmail.com/It seems like you haven't come across this one?:
https://lore.kernel.org/bpf/[email protected]/t/
That has all I wanted to know and more, not sure how I was able to missed that, my Xapian searching skill really needs more work :)
Quoted 12 lines
Yes, easing packaging was one of the motivations for the mirror. As mentioned in my other answer, I've not taken the time to reach out to package maintainers yet, so this hasn't really materialised at this point.
CI, indeed, was another motivation.
Submodules, or simply making it easier to hack with bpftool's code, was yet another thing. Microsoft folks intend to make bpftool compatible with eBPF for Windows. It's quite simpler to work on that from a repo which is mostly uncoupled from the Linux tree.
Quoted 8 lines
Perhaps the most important was to make it easier to just download, build and use bpftool, for all users who need to get the latest version, or to patch it, or to create static builds, or to cross-compile, or whatever reason might cause you to compile it from the sources. For all those cases, getting the mirror is faster and requires less space than getting the kernel repo. This makes a nice difference when periodically rebuilding images in automated workflows, for example.
While for distro packaging the net benefit is somewhat unclear (after all that the point of this thread), I very much agree with the above, this does give users much greater control over the bpftool they can choose to use.
Quoted 1 line
Does this answer your questions?
Yes, thanks!
Quoted 1 line
Quentin
Quoted 5 lines · Shung-Hsi Yu
A side note: if we want all userspace visible libbpf to have a coherent version, perf needs to use the shared libbpf library as well (either autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to backport patches to kernel source to keep up with userspace package (libbpf) changes could be a pain.
So basically, this here is the reason we're building libbpf from the kernel tree for the RHEL package: If we use the github version we'd need to juggle two different versions of libbpf, one for the in-kernel-tree users (perf as you mention, but also the BPF selftests), and one for the userspace packages. Also, having libbpf in the kernel tree means we can just backport patches to it along with the BPF-related kernel patches (we do quite extensive BPF backports for each RHEL version). Finally, building from the kernel tree means we can use the existing kernel-related procedures for any out of order hotfixes (since AFAIK none of the github repositories have any concept of stable branches that receive fixes).
YMMV of course, but figured I'd share our reasoning. To be clear, building from the kernel tree is not without its own pain points (mostly related to how the build scripts are structured for our kernel builds). We've discussed moving to the github version of libbpf multiple times, but every time we've concluded that it would be more, not less, painful than having the kernel tree be the single source of truth.
-Toke
Thanks for sharing! I though I'd expands on what you said to draw a clearer picture of the challenges.
Quoted 7 lines · Toke Høiland-Jørgensen
Quoted 5 lines · Shung-Hsi Yu
A side note: if we want all userspace visible libbpf to have a coherent version, perf needs to use the shared libbpf library as well (either autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to backport patches to kernel source to keep up with userspace package (libbpf) changes could be a pain.
Here some more context for completeness. Kernel source changes are published at a much slower pace than userspace. When an application in the kernel source (e.g. perf) depends on the userspace library, it's kind of like trying to catchup a car on a bike, which is doable, as evident by the plethora of userspace libraries perf already depends on. While I don't having experience maintaining perf, judging by tools/perf/Makefile.config that does not seem like an easy feat.
For perf to use libbpf in kernel would mean that it's just depending on something that moves at the same pace.
That said, maybe perf won't need additional backport to keep up with libbpf as long as we keep it within that same major version (and disable deprecation warning)? @Andrii
Now that We've got pass libbpf 1.0 it seems like a good time to reconsider.
Quoted 8 lines
So basically, this here is the reason we're building libbpf from the kernel tree for the RHEL package: If we use the github version we'd need to juggle two different versions of libbpf, one for the in-kernel-tree users (perf as you mention, but also the BPF selftests), and one for the userspace packages. Also, having libbpf in the kernel tree means we can just backport patches to it along with the BPF-related kernel patches (we do quite extensive BPF backports for each RHEL version).
Quoted 4 lines
Finally, building from the kernel tree means we can use the existing kernel-related procedures for any out of order hotfixes (since AFAIK none of the github repositories have any concept of stable branches that receive fixes).
+1
Got something similar in place as well and being able to stick with existing procedure is appealing.
Quoted 6 lines
YMMV of course, but figured I'd share our reasoning. To be clear, building from the kernel tree is not without its own pain points (mostly related to how the build scripts are structured for our kernel builds). We've discussed moving to the github version of libbpf multiple times, but every time we've concluded that it would be more, not less, painful than having the kernel tree be the single source of truth.
We package maintainer are certainly quite hard to please :)
Just having an individual package easy to work with is not enough, we want it to be easier for most packages before jumping on the bandwagon, which is why this email ended up talking about perf despite it started as a discussion on packaging libbpf and bpftool.
I suppose the mileage depends on the build system & scripts in use and how much backporting is done; the more kernel backporting (along with more established processes in place), the more painful it'd be to move to the GitHub version. My gut feeling is that SLES do less backporting compared to RHEL when it comes to BPF, and that probably placed us closer to the middle ground.
Thanks,
Shung-Hsi
Quoted 2 lines
-Toke
Quoted 21 lines · Shung-Hsi Yu
Thanks for sharing! I though I'd expands on what you said to draw a clearer picture of the challenges.
Quoted 7 lines · Toke Høiland-Jørgensen
Quoted 5 lines · Shung-Hsi Yu
A side note: if we want all userspace visible libbpf to have a coherent version, perf needs to use the shared libbpf library as well (either autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to backport patches to kernel source to keep up with userspace package (libbpf) changes could be a pain.
Here some more context for completeness. Kernel source changes are published at a much slower pace than userspace. When an application in the kernel source (e.g. perf) depends on the userspace library, it's kind of like trying to catchup a car on a bike, which is doable, as evident by the plethora of userspace libraries perf already depends on. While I don't having experience maintaining perf, judging by
tools/perf/Makefile.configthat does not seem like an easy feat.
Quoted 8 lines
For perf to use libbpf in kernel would mean that it's just depending on something that moves at the same pace.
That said, maybe perf won't need additional backport to keep up with libbpf as long as we keep it within that same major version (and disable deprecation warning)? @Andrii
Now that We've got pass libbpf 1.0 it seems like a good time to reconsider.
I'm not sure what the proposal is, but I'll delegate to Arnaldo.
Quoted 33 lines
Quoted 9 lines
So basically, this here is the reason we're building libbpf from the kernel tree for the RHEL package: If we use the github version we'd need to juggle two different versions of libbpf, one for the in-kernel-tree users (perf as you mention, but also the BPF selftests), and one for the userspace packages. Also, having libbpf in the kernel tree means we can just backport patches to it along with the BPF-related kernel patches (we do quite extensive BPF backports for each RHEL version).
Quoted 4 lines
Finally, building from the kernel tree means we can use the existing kernel-related procedures for any out of order hotfixes (since AFAIK none of the github repositories have any concept of stable branches that receive fixes).
+1
Got something similar in place as well and being able to stick with existing procedure is appealing.
Quoted 6 lines
YMMV of course, but figured I'd share our reasoning. To be clear, building from the kernel tree is not without its own pain points (mostly related to how the build scripts are structured for our kernel builds). We've discussed moving to the github version of libbpf multiple times, but every time we've concluded that it would be more, not less, painful than having the kernel tree be the single source of truth.
We package maintainer are certainly quite hard to please :)
Just having an individual package easy to work with is not enough, we want it to be easier for most packages before jumping on the bandwagon, which is why this email ended up talking about perf despite it started as a discussion on packaging libbpf and bpftool.
Quoted 6 lines
I suppose the mileage depends on the build system & scripts in use and how much backporting is done; the more kernel backporting (along with more established processes in place), the more painful it'd be to move to the GitHub version. My gut feeling is that SLES do less backporting compared to RHEL when it comes to BPF, and that probably placed us closer to the middle ground.
Even though libbpf source is developed in kernel repo, it's not tied to specific kernel. So any kernel backports have no relevance to libbpf itself. It's yet another reason to switch to Github mirror. Github is merging libbpf-related fixes from both bpf and bpf-next trees during sync, and is meant to always be the latest and best version with all fixes included.
I won't claim anything for perf, maybe Arnaldo can clarify, but I suspect that perf is also meant to be relatively independent from specific kernel and work on wide variety of kernels.
As for stable branches. For libbpf, we don't have it because we didn't need it yet. We did have bug fix patch releases that seem to be working out fine, though.
Quoted 6 lines
Thanks,
Shung-HsiQuoted 2 lines
-Toke
Quoted 3 lines · Andrii Nakryiko
I won't claim anything for perf, maybe Arnaldo can clarify, but I suspect that perf is also meant to be relatively independent from specific kernel and work on wide variety of kernels.
Supposedly it is though I've never been able to convince myself to decouple (and ship the latest perf) for SLE. One reason is documented userspace functionality that has no backing kernel support which for an Enterprise distro opens us up to bug reports.
For a while Vince was maintaining a page documenting breakage in the perf_event ABI but last I checked it ended at V4.0. Not sure if he just stopped updating, or that's where breakage ended :)
I know how Fedora maintains perf but I keep meaning to look at how RHEL maintains it. As Michal said our code base is V5.14 for SLE15* and due to the frequent code-refactoring in perf, backporting perf userspace fixes for SLE is a real chore unless we agressively maintain forward porting ("forklifting").
Tony
Signature · 2 lines
Tony Jones SUSE Kernel Performance Team
Quoted 4 lines · On Wed, 19 Apr 2023, [email protected] wrote:
For a while Vince was maintaining a page documenting breakage in the
perf_eventABI but last I checked it ended at V4.0. Not sure if he just stopped updating, or that's where breakage ended :)
I actually still try to keep an eye on things. I originally tracked things because we had to work around a lot of those breakages in PAPI.
There have been a few breakages since then, mostly in rdpmc support and also some weird things on ARM64. I stopped documenting things as thoroughly in part because despite the common wisdom I don't think the kernel devs care much about ABI breaks at least with the perf interface.
Vince
Em Wed, Apr 19, 2023 at 12:42:39PM -0700, Andrii Nakryiko escreveu:
Quoted 14 lines · Shung-Hsi Yu
Quoted 13 lines
Thanks for sharing! I though I'd expands on what you said to draw a clearer picture of the challenges.
Quoted 7 lines · Toke Høiland-Jørgensen
Quoted 5 lines · Shung-Hsi Yu
A side note: if we want all userspace visible libbpf to have a coherent version, perf needs to use the shared libbpf library as well (either autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to backport patches to kernel source to keep up with userspace package (libbpf) changes could be a pain.
Quoted 4 lines
Quoted 4 lines
Here some more context for completeness. Kernel source changes are published at a much slower pace than userspace. When an application in the kernel source (e.g. perf) depends on the userspace library, it's kind of like trying to catchup a car on a bike, which is doable, as evident by the
That is why perf is continuously (at least) build tested against lots of distros, see the last test output:
[perfbuilder@five ~]$ export BUILD_TARBALL=http://192.168.86.10/perf/perf-6.3.0-rc1.tar.xz [perfbuilder@five ~]$ time dm
1 151.45 almalinux:8 : Ok gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16) , clang version 14.0.6 (Red Hat 14.0.6-1.module_el8.7.0+3277+b822483f) 2 150.54 almalinux:9 : Ok gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2) , clang version 14.0.6 (Red Hat 14.0.6-4.el9_1) 3 159.04 alpine:3.15 : Ok gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027 , Alpine clang version 12.0.1 4 153.81 alpine:3.16 : Ok gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219 , Alpine clang version 13.0.1 5 137.88 alpine:3.17 : Ok gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924 , Alpine clang version 15.0.7 6 139.32 alpine:edge : Ok gcc (Alpine 12.2.1_git20220924-r9) 12.2.1 20220924 , Alpine clang version 16.0.0 7 109.51 alt:p9 : Ok x86_64-alt-linux-gcc (GCC) 8.4.1 20200305 (ALT p9 8.4.1-alt0.p9.1) , clang version 10.0.0 8 104.99 alt:p10 : Ok x86_64-alt-linux-gcc (GCC) 10.3.1 20210703 (ALT Sisyphus 10.3.1-alt2) , clang version 11.0.1 9 85.35 alt:sisyphus : Ok x86_64-alt-linux-gcc (GCC) 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt2) , ALT Linux Team clang version 13.0.1 10 111.22 amazonlinux:2 : Ok gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15) , clang version 11.1.0 (Amazon Linux 2 11.1.0-1.amzn2.0.2) 11 127.05 amazonlinux:2023 : Ok gcc (GCC) 11.3.1 20221121 (Red Hat 11.3.1-4) , clang version 15.0.6 (Amazon Linux 15.0.6-3.amzn2023.0.2) 12 126.56 amazonlinux:devel : Ok gcc (GCC) 11.3.1 20221121 (Red Hat 11.3.1-4) , clang version 15.0.6 (Amazon Linux 15.0.6-3.amzn2023.0.2) 13 154.76 archlinux:base : Ok gcc (GCC) 12.2.0 , clang version 14.0.6 14 135.39 centos:stream : Ok gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18) , clang version 15.0.7 (Red Hat 15.0.7-1.module_el8.8.0+1258+af79b238) 15 40.01 clearlinux:latest : Ok gcc (Clear Linux OS for Intel Architecture) 12.2.1 20230412 releases/gcc-12.2.0-699-g43ab94d20e 16 95.89 debian:10 : Ok gcc (Debian 8.3.0-6) 8.3.0 , Debian clang version 11.0.1-2~deb10u1 17 121.95 debian:11 : Ok gcc (Debian 10.2.1-6) 10.2.1 20210110 , Debian clang version 11.0.1-2 18 136.49 debian:experimental : Ok gcc (Debian 12.2.0-14) 12.2.0 , Debian clang version 14.0.6 19 29.78 debian:experimental-x-arm64 : Ok aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0 20 22.96 debian:experimental-x-mips : Ok mips-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0 21 3.37 debian:experimental-x-mips64 : FAIL gcc version 10.2.1 20210110 (Debian 10.2.1-6) 22 11.29 debian:experimental-x-mipsel : FAIL gcc version 12.2.0 (Debian 12.2.0-14) 23 33.28 fedora:26 : Ok gcc (GCC) 7.3.1 20180130 (Red Hat 7.3.1-2) 24 33.29 fedora:27 : Ok gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-6) 25 29.77 fedora:28 : Ok gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2) 26 32.27 fedora:29 : Ok gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2) 27 34.18 fedora:30 : Ok gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2) 28 139.98 fedora:31 : Ok gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2) , clang version 9.0.1 (Fedora 9.0.1-4.fc31) 29 119.93 fedora:32 : Ok gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1) , clang version 10.0.1 (Fedora 10.0.1-3.fc32) 30 123.87 fedora:33 : Ok gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1) , clang version 11.0.0 (Fedora 11.0.0-3.fc33) 31 188.77 fedora:34 : Ok gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2) , clang version 12.0.1 (Fedora 12.0.1-1.fc34) 32 22.44 fedora:34-x-ARC-glibc : Ok arc-linux-gcc (ARC HS GNU/Linux glibc toolchain 2019.03-rc1) 8.3.1 20190225 33 20.33 fedora:34-x-ARC-uClibc : Ok arc-linux-gcc (ARCv2 ISA Linux uClibc toolchain 2019.03-rc1) 8.3.1 20190225 34 172.51 fedora:35 : Ok gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-3) , clang version 13.0.1 (Fedora 13.0.1-1.fc35) 35 138.29 fedora:36 : Ok gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4) , clang version 14.0.5 (Fedora 14.0.5-2.fc36) 36 139.99 fedora:37 : Ok gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4) , clang version 15.0.7 (Fedora 15.0.7-1.fc37) 37 144.73 fedora:38 : Ok gcc (GCC) 13.0.1 20230401 (Red Hat 13.0.1-0) , clang version 16.0.0 (Fedora 16.0.0-2.fc38) 38 191.37 fedora:39 : Ok gcc (GCC) 13.0.1 20230404 (Red Hat 13.0.1-0) , clang version 16.0.1 (Fedora 16.0.1-1.fc39) 39 187.13 fedora:rawhide : Ok gcc (GCC) 13.0.1 20230404 (Red Hat 13.0.1-0) , clang version 16.0.1 (Fedora 16.0.1-1.fc39) 40 13.08 gentoo:stage3 : FAIL gcc version 12.2.1 20230121 (Gentoo 12.2.1_p20230121-r1 p10) 41 152.25 manjaro:base : Ok gcc (GCC) 12.2.1 20230201 , clang version 15.0.7 42 148.30 opensuse:15.4 : Ok gcc (SUSE Linux) 7.5.0 , clang version 13.0.1 43 152.82 opensuse:15.5 : Ok gcc (SUSE Linux) 7.5.0 , clang version 15.0.7 44 187.75 opensuse:tumbleweed : Ok gcc (SUSE Linux) 12.2.1 20221020 [revision 0aaef83351473e8f4eb774f8f999bbe87a4866d7] , clang version 15.0.6 45 133.67 oraclelinux:8 : Ok gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16.0.2) , clang version 14.0.6 (Red Hat 14.0.6-1.0.1.module+el8.7.0+20823+214a699d) 46 132.26 oraclelinux:9 : Ok gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2.1.0.2) , clang version 14.0.6 (Red Hat 14.0.6-4.0.1.el9_1) 47 150.02 rockylinux:8 : Ok gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16) , clang version 14.0.6 (Red Hat 14.0.6-1.module+el8.7.0+1080+d88dc670) 48 133.78 rockylinux:9 : Ok gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2) , clang version 14.0.6 (Red Hat 14.0.6-4.el9_1) 49 28.97 ubuntu:18.04 : Ok gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 50 25.06 ubuntu:18.04-x-arm : Ok arm-linux-gnueabihf-gcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0 51 25.06 ubuntu:18.04-x-arm64 : Ok aarch64-linux-gnu-gcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0 52 20.54 ubuntu:18.04-x-m68k : Ok m68k-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 53 24.56 ubuntu:18.04-x-powerpc : Ok powerpc-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 54 26.07 ubuntu:18.04-x-powerpc64 : Ok powerpc64-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 55 26.46 ubuntu:18.04-x-powerpc64el : Ok powerpc64le-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 56 113.72 ubuntu:18.04-x-riscv64 : Ok riscv64-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 57 22.55 ubuntu:18.04-x-s390 : Ok s390x-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 58 24.35 ubuntu:18.04-x-sh4 : Ok sh4-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 59 22.54 ubuntu:18.04-x-sparc64 : Ok sparc64-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 60 31.87 ubuntu:20.04 : Ok gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 61 179.64 ubuntu:22.04 : Ok gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 , Ubuntu clang version 14.0.0-1ubuntu1 62 1.26 ubuntu:22.04-x-riscv64 : FAIL gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04) 63 87.25 ubuntu:22.10 : Ok gcc (Ubuntu 12.2.0-3ubuntu1) 12.2.0 , Ubuntu clang version 15.0.7 64 86.96 ubuntu:23.04 : Ok gcc (Ubuntu 12.2.0-17ubuntu1) 12.2.0 , Ubuntu clang version 15.0.7
BUILD_TARBALL_HEAD=f8b04f975d2c3d7c8e8cb53155744c20a41813ac 65 6011.52
real 101m23.391s
user 0m51.216s
sys 0m40.407s
[perfbuilder@five ~]$
Quoted 3 lines
Quoted 3 lines
plethora of userspace libraries perf already depends on. While I don't having experience maintaining perf, judging by
tools/perf/Makefile.configthat does not seem like an easy feat.
Not that bad having the tests we have in place :-)
Quoted 2 lines
Quoted 2 lines
For perf to use libbpf in kernel would mean that it's just depending on something that moves at the same pace.
More tests to check build both with the in-kernel libbpf and with the one in the distro:
⬢[acme@toolbox perf-tools-next]$ grep LIBBPF_DYNAMIC tools/perf/tests/make make_libbpf_dynamic := LIBBPF_DYNAMIC=1
⬢[acme@toolbox perf-tools-next]$
Quoted 4 lines
Quoted 4 lines
That said, maybe perf won't need additional backport to keep up with libbpf as long as we keep it within that same major version (and disable deprecation warning)? @Andrii
Quoted 8 lines
Quoted 1 line
Now that We've got pass libbpf 1.0 it seems like a good time to reconsider.
I'm not sure what the proposal is, but I'll delegate to Arnaldo.
Quoted 4 lines
Quoted 4 lines
So basically, this here is the reason we're building libbpf from the kernel tree for the RHEL package: If we use the github version we'd need to juggle two different versions of libbpf, one for the in-kernel-tree users (perf as you mention, but also the BPF selftests), and one for the
Have you tried with LIBBPF_DYNAMIC=1?
Quoted 56 lines
Quoted 28 lines
Quoted 8 lines
userspace packages. Also, having libbpf in the kernel tree means we can just backport patches to it along with the BPF-related kernel patches (we do quite extensive BPF backports for each RHEL version).
Finally, building from the kernel tree means we can use the existing kernel-related procedures for any out of order hotfixes (since AFAIK none of the github repositories have any concept of stable branches that receive fixes).
+1
Got something similar in place as well and being able to stick with existing procedure is appealing.
Quoted 6 lines
YMMV of course, but figured I'd share our reasoning. To be clear, building from the kernel tree is not without its own pain points (mostly related to how the build scripts are structured for our kernel builds). We've discussed moving to the github version of libbpf multiple times, but every time we've concluded that it would be more, not less, painful than having the kernel tree be the single source of truth.
We package maintainer are certainly quite hard to please :)
Just having an individual package easy to work with is not enough, we want it to be easier for most packages before jumping on the bandwagon, which is why this email ended up talking about perf despite it started as a discussion on packaging libbpf and bpftool.
Quoted 6 lines
I suppose the mileage depends on the build system & scripts in use and how much backporting is done; the more kernel backporting (along with more established processes in place), the more painful it'd be to move to the GitHub version. My gut feeling is that SLES do less backporting compared to RHEL when it comes to BPF, and that probably placed us closer to the middle ground.
Even though libbpf source is developed in kernel repo, it's not tied to specific kernel. So any kernel backports have no relevance to libbpf itself. It's yet another reason to switch to Github mirror. Github is merging libbpf-related fixes from both bpf and bpf-next trees during sync, and is meant to always be the latest and best version with all fixes included.
I won't claim anything for perf, maybe Arnaldo can clarify, but I suspect that perf is also meant to be relatively independent from specific kernel and work on wide variety of kernels.
As for stable branches. For libbpf, we don't have it because we didn't need it yet. We did have bug fix patch releases that seem to be working out fine, though.
Quoted 6 lines
Thanks,
Shung-HsiQuoted 2 lines
-Toke
Signature · 1 line
- Arnaldo
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Quoted 1 line
Does bpftool work on older kernel?
It should, although it's not perfect. Most features from current bpftool should work as expected on older kernels. However, if I remember correctly you would have trouble loading programs on pre-BTF kernels, because bpftool relies on libbpf >= 1.0 and only accepts map definitions with BTF info, and attempts to create these maps with BTF, which fails and blocks the load process.
But we're trying to keep backward-compatibility, so if we're only talking of kernels recent enough to support BTF, then I'd expect bpftool to work. If this is not the case, please report on this list.
Quoted 7 lines
Our current approach is that we (openSUSE/SLES) essentially have two version of libbpf: a public shared library that uses GitHub mirror as source, which the general userspace sees and links to; and a private static library built from kernel source used by bpftool, perf,
resolve_btfids, selftests, etc. A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they took similar approach.
I would like them to reconsider this choice eventually. Sounds like for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to have a real bpftool package instead of having to install linux-tools-common + linux-tools-generic, or to have distros in general (Ubuntu/Debian at least) stop compiling out the JIT disassembler, although this is not strictly related to the location of the sources. I've not found the time to reach out to package maintainers yet.
Quoted 4 lines
This approach means that the version of bpftool and libbpf are _not_ always in sync[1], which I read may causes problem since libbpf and bpftool depends on specific version of each other[2].
Whatever source you use, I would strongly recommend finding a way to keep both in sync. Libbpf has stabilised its API when reaching 1.0, but bpftool taps into some of the internals of the library. Features or new definitions are usually added at the same time to libbpf and bpftool, and if you get a mismatch between the two, you're taking risks to get build issues.
Quoted 7 lines
Using the GitHub mirror of bpftool to package both libbpf and bpftool would kept their version in sync, and was suggested[3]. Although the same could be said if we switch back to packaging libbpf from kernel source, an additional appeal for using GitHub mirrors is that it decouples bpftool from kernel, making it easily upgradable and with a clearer changelog (the latter is quite important for enterprise users) like libbpf.
Happy to read these changelogs I write are useful to someone :). Yes, this is my point.
Quoted 6 lines
The main concern with using GitHub mirror is that bpftool may be updated far beyond the version that comes with the runtime kernel. AFAIK bpftool should work on older kernel since CO-RE is used for built-in BPF iterators and the underlying libbpf work on older kernel itself. Nonetheless, it would be nice to get a confirmation from the maintainers.
As explained above - Mostly, it should work. Otherwise, we can look into fixing it.
As a side note, I'm open to suggestions/contributions to make life easier for packaging for the mirror. For example, Mahé and I recently added GitHub workflows to ship statically-built binaries for amd64 and arm64 on releases, as well as tarballs with both bpftool+libbpf sources. If there's something else to make packaging easier, I'm happy to talk about it.
Quoted 2 lines
Are there any other downsides to switching to GitHub mirror for bpftool?
Nothing else comes to mind, but again I'm not packaging for distros. See Toke's message.
I hope this helps,
Quentin
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
Quoted 9 lines
Quoted 1 line
Does bpftool work on older kernel?
It should, although it's not perfect. Most features from current bpftool should work as expected on older kernels. However, if I remember correctly you would have trouble loading programs on pre-BTF kernels, because bpftool relies on libbpf >= 1.0 and only accepts map definitions with BTF info, and attempts to create these maps with BTF, which fails and blocks the load process.
Quoted 3 lines
But we're trying to keep backward-compatibility, so if we're only talking of kernels recent enough to support BTF, then I'd expect bpftool to work. If this is not the case, please report on this list.
It won't build:
https://lore.kernel.org/bpf/[email protected]/
Quoted 21 lines
Quoted 6 lines
Our current approach is that we (openSUSE/SLES) essentially have two version of libbpf: a public shared library that uses GitHub mirror as source, which the general userspace sees and links to; and a private static library built from kernel source used by bpftool, perf,
resolve_btfids, selftests, etc. A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they took similar approach.I would like them to reconsider this choice eventually. Sounds like for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to have a real bpftool package instead of having to install linux-tools-common + linux-tools-generic, or to have distros in general (Ubuntu/Debian at least) stop compiling out the JIT disassembler, although this is not strictly related to the location of the sources. I've not found the time to reach out to package maintainers yet.
Quoted 4 lines
This approach means that the version of bpftool and libbpf are _not_ always in sync[1], which I read may causes problem since libbpf and bpftool depends on specific version of each other[2].
Quoted 6 lines
Whatever source you use, I would strongly recommend finding a way to keep both in sync. Libbpf has stabilised its API when reaching 1.0, but bpftool taps into some of the internals of the library. Features or new definitions are usually added at the same time to libbpf and bpftool, and if you get a mismatch between the two, you're taking risks to get build issues.
In other words no API exists.
Quoted 9 lines
Quoted 6 lines
Using the GitHub mirror of bpftool to package both libbpf and bpftool would kept their version in sync, and was suggested[3]. Although the same could be said if we switch back to packaging libbpf from kernel source, an additional appeal for using GitHub mirrors is that it decouples bpftool from kernel, making it easily upgradable and with a clearer changelog (the latter is quite important for enterprise users) like libbpf.
Happy to read these changelogs I write are useful to someone :). Yes, this is my point.
Yes, publishing the changelog in a usable way relieves others of the need to write thier own, usually with less understanding of the changes in question. That's generally the idea of opensource - not endlessly repeating what has already been done :)
Quoted 9 lines
Quoted 5 lines
The main concern with using GitHub mirror is that bpftool may be updated far beyond the version that comes with the runtime kernel. AFAIK bpftool should work on older kernel since CO-RE is used for built-in BPF iterators and the underlying libbpf work on older kernel itself. Nonetheless, it would be nice to get a confirmation from the maintainers.
As explained above - Mostly, it should work. Otherwise, we can look into fixing it.
Quoted 6 lines
As a side note, I'm open to suggestions/contributions to make life easier for packaging for the mirror. For example, Mahé and I recently added GitHub workflows to ship statically-built binaries for amd64 and arm64 on releases, as well as tarballs with both bpftool+libbpf sources. If there's something else to make packaging easier, I'm happy to talk about it.
Make it possible to build with system-installed libbpf. If it's released it should have versioned dependency on a libbpf release, and libbpf from that version on should be good enough to build it.
I tried copying those 'private' headers into a separate directory, and link against static libbpf, and it seems to work. Of course, having an actual API would be much better.
Thanks
Michal
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 21 lines
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 7 lines
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Does bpftool work on older kernel?
It should, although it's not perfect. Most features from current bpftool should work as expected on older kernels. However, if I remember correctly you would have trouble loading programs on pre-BTF kernels, because bpftool relies on libbpf >= 1.0 and only accepts map definitions with BTF info, and attempts to create these maps with BTF, which fails and blocks the load process.
Quoted 6 lines
Quoted 3 lines
But we're trying to keep backward-compatibility, so if we're only talking of kernels recent enough to support BTF, then I'd expect bpftool to work. If this is not the case, please report on this list.
It won't build:
https://lore.kernel.org/bpf/[email protected]/
True in this case, and this is something that needs to get fixed. Thanks for reopening that thread! Are you building bpftool on kernels older than 5.15? (genuine curiosity)
Quoted 22 lines
Quoted 22 lines
Quoted 7 lines
Our current approach is that we (openSUSE/SLES) essentially have two version of libbpf: a public shared library that uses GitHub mirror as source, which the general userspace sees and links to; and a private static library built from kernel source used by bpftool, perf,
resolve_btfids, selftests, etc. A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they took similar approach.I would like them to reconsider this choice eventually. Sounds like for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to have a real bpftool package instead of having to install linux-tools-common + linux-tools-generic, or to have distros in general (Ubuntu/Debian at least) stop compiling out the JIT disassembler, although this is not strictly related to the location of the sources. I've not found the time to reach out to package maintainers yet.
Quoted 4 lines
This approach means that the version of bpftool and libbpf are _not_ always in sync[1], which I read may causes problem since libbpf and bpftool depends on specific version of each other[2].
Quoted 8 lines
Quoted 6 lines
Whatever source you use, I would strongly recommend finding a way to keep both in sync. Libbpf has stabilised its API when reaching 1.0, but bpftool taps into some of the internals of the library. Features or new definitions are usually added at the same time to libbpf and bpftool, and if you get a mismatch between the two, you're taking risks to get build issues.
In other words no API exists.
Of course it does. Libbpf exposes a specific set of functions to user applications.
But correct, from bpftool's perspective, there are a few locations where we accept to derogate and to access to the internals directly, making it more dependent on a specific version, or commit, of libbpf, and blurring the notion of API.
This special relationship is nothing new though, and it has been discussed before. It derives from both tools being developed in the same repository, and bpftool being so tightly linked to libbpf - it has been qualified of command-line interface for libbpf in the past. Bpftool's version number itself is aligned on libbpf's. (As a side note, bpftool used to pull libbpf's headers directly from libbpf's dir instead of installing them locally, which facilitated this mix-in for public/internal headers in the first place.)
I know you advocated making the required functions part of the API, given that some users (such as bpftool) need them. These functions are not exposed, by choice. They are not judged relevant to generic user space application (I'm sure libbpf's maintainers are opened to discussion if use cases come up). Some of the internals we get from libbpf are also mostly to avoid re-implementing things, such as netlink attributes processing, or implementing hash maps. These have nothing to do in libbpf's API.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Using the GitHub mirror of bpftool to package both libbpf and bpftool would kept their version in sync, and was suggested[3]. Although the same could be said if we switch back to packaging libbpf from kernel source, an additional appeal for using GitHub mirrors is that it decouples bpftool from kernel, making it easily upgradable and with a clearer changelog (the latter is quite important for enterprise users) like libbpf.
Quoted 7 lines
Quoted 2 lines
Happy to read these changelogs I write are useful to someone :). Yes, this is my point.
Yes, publishing the changelog in a usable way relieves others of the need to write thier own, usually with less understanding of the changes in question. That's generally the idea of opensource - not endlessly repeating what has already been done :)
Not discussing that - My point is that I've received little feedback on the mirror so far, so it's hard for me to figure out who's using it or whether anyone has been reading the changelogs.
Quoted 17 lines
Quoted 10 lines
Quoted 6 lines
The main concern with using GitHub mirror is that bpftool may be updated far beyond the version that comes with the runtime kernel. AFAIK bpftool should work on older kernel since CO-RE is used for built-in BPF iterators and the underlying libbpf work on older kernel itself. Nonetheless, it would be nice to get a confirmation from the maintainers.
As explained above - Mostly, it should work. Otherwise, we can look into fixing it.
Quoted 6 lines
As a side note, I'm open to suggestions/contributions to make life easier for packaging for the mirror. For example, Mahé and I recently added GitHub workflows to ship statically-built binaries for amd64 and arm64 on releases, as well as tarballs with both bpftool+libbpf sources. If there's something else to make packaging easier, I'm happy to talk about it.
Quoted 7 lines
Make it possible to build with system-installed libbpf. If it's released it should have versioned dependency on a libbpf release, and libbpf from that version on should be good enough to build it.
I tried copying those 'private' headers into a separate directory, and link against static libbpf, and it seems to work. Of course, having an actual API would be much better.
Just as you said yourself, the missing stability is in the way. I don't see this happening as long as bpftool is using libbpf's internals. I do expect builds to work most of the time by copying the headers as you did, but as soon as something changes and it no longer does, everyone will start filing issues on GitHub instead of using the version that works, and I don't want that.
As for decoupling from the internal: making the functions part of the API is not an option. One option could be to move this code into further dependencies shared between libbpf and bpftool - although I guess libbpf developers will have little appetite for that. We could also duplicate the necessary code in bpftool, which doesn't sound optimal, but might be one solution. Other options have been discussed before, such as moving bpftool into libbpf's directory/mirror and shipping both together, but there was no consensus at the time, and I don't expect libbpf to ship with bpftool any time soon.
Quoted 28 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 27 lines
Hello,
Quoted 23 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Quoted 8 lines
Quoted 1 line
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 1 line
Does bpftool work on older kernel?
It should, although it's not perfect. Most features from current bpftool should work as expected on older kernels. However, if I remember correctly you would have trouble loading programs on pre-BTF kernels, because bpftool relies on libbpf >= 1.0 and only accepts map definitions with BTF info, and attempts to create these maps with BTF, which fails and blocks the load process.
But we're trying to keep backward-compatibility, so if we're only talking of kernels recent enough to support BTF, then I'd expect bpftool to work. If this is not the case, please report on this list.
Quoted 6 lines
Quoted 2 lines
It won't build:
https://lore.kernel.org/bpf/[email protected]/True in this case, and this is something that needs to get fixed. Thanks for reopening that thread! Are you building bpftool on kernels older than 5.15? (genuine curiosity)
Yes, 5.14 and 5.3. I would not be able to notice this particular breakage otherwise.
Quoted 38 lines
Quoted 21 lines
Quoted 21 lines
Quoted 6 lines
Our current approach is that we (openSUSE/SLES) essentially have two version of libbpf: a public shared library that uses GitHub mirror as source, which the general userspace sees and links to; and a private static library built from kernel source used by bpftool, perf,
resolve_btfids, selftests, etc. A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they took similar approach.I would like them to reconsider this choice eventually. Sounds like for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to have a real bpftool package instead of having to install linux-tools-common + linux-tools-generic, or to have distros in general (Ubuntu/Debian at least) stop compiling out the JIT disassembler, although this is not strictly related to the location of the sources. I've not found the time to reach out to package maintainers yet.
Quoted 4 lines
This approach means that the version of bpftool and libbpf are _not_ always in sync[1], which I read may causes problem since libbpf and bpftool depends on specific version of each other[2].
Quoted 8 lines
Quoted 6 lines
Whatever source you use, I would strongly recommend finding a way to keep both in sync. Libbpf has stabilised its API when reaching 1.0, but bpftool taps into some of the internals of the library. Features or new definitions are usually added at the same time to libbpf and bpftool, and if you get a mismatch between the two, you're taking risks to get build issues.
In other words no API exists.
Of course it does. Libbpf exposes a specific set of functions to user applications.
But correct, from bpftool's perspective, there are a few locations where we accept to derogate and to access to the internals directly, making it more dependent on a specific version, or commit, of libbpf, and blurring the notion of API.
Quoted 4 lines
This special relationship is nothing new though, and it has been discussed before. It derives from both tools being developed in the same repository, and bpftool being so tightly linked to libbpf - it has been qualified of command-line interface for libbpf in the past. Bpftool's
If bpftool is a commandline interface for libbpf maybe the best choice is to just dump both into the same repository, and provide make targets for building one, the other, and both.
Quoted 5 lines
version number itself is aligned on libbpf's. (As a side note, bpftool used to pull libbpf's headers directly from libbpf's dir instead of installing them locally, which facilitated this mix-in for public/internal headers in the first place.)
Quoted 8 lines
I know you advocated making the required functions part of the API, given that some users (such as bpftool) need them. These functions are not exposed, by choice. They are not judged relevant to generic user space application (I'm sure libbpf's maintainers are opened to discussion if use cases come up). Some of the internals we get from libbpf are also mostly to avoid re-implementing things, such as netlink attributes processing, or implementing hash maps. These have nothing to do in libbpf's API.
And we do not have microframeworks for implementing reusable hashmaps or netlink parsers. I am sure that bpftool is not the first nor last tool that needs a hashmap or parse netlink but the ecosystem for small single-purpose C libraries never really took off.
Quoted 24 lines
Quoted 16 lines
Quoted 9 lines
Quoted 5 lines
The main concern with using GitHub mirror is that bpftool may be updated far beyond the version that comes with the runtime kernel. AFAIK bpftool should work on older kernel since CO-RE is used for built-in BPF iterators and the underlying libbpf work on older kernel itself. Nonetheless, it would be nice to get a confirmation from the maintainers.
As explained above - Mostly, it should work. Otherwise, we can look into fixing it.
Quoted 6 lines
As a side note, I'm open to suggestions/contributions to make life easier for packaging for the mirror. For example, Mahé and I recently added GitHub workflows to ship statically-built binaries for amd64 and arm64 on releases, as well as tarballs with both bpftool+libbpf sources. If there's something else to make packaging easier, I'm happy to talk about it.
Quoted 7 lines
Make it possible to build with system-installed libbpf. If it's released it should have versioned dependency on a libbpf release, and libbpf from that version on should be good enough to build it.
I tried copying those 'private' headers into a separate directory, and link against static libbpf, and it seems to work. Of course, having an actual API would be much better.
Quoted 6 lines
Just as you said yourself, the missing stability is in the way. I don't see this happening as long as bpftool is using libbpf's internals. I do expect builds to work most of the time by copying the headers as you did, but as soon as something changes and it no longer does, everyone will start filing issues on GitHub instead of using the version that works, and I don't want that.
So these are not really separate projects, they should ship together.
Quoted 9 lines
As for decoupling from the internal: making the functions part of the API is not an option. One option could be to move this code into further dependencies shared between libbpf and bpftool - although I guess libbpf developers will have little appetite for that. We could also duplicate the necessary code in bpftool, which doesn't sound optimal, but might be one solution. Other options have been discussed before, such as moving bpftool into libbpf's directory/mirror and shipping both together, but there was no consensus at the time, and I don't expect libbpf to ship with bpftool any time soon.
Why is that? Is there any speciffic reason for the sepaaration if bpftool needs specific libbpf anyway?
Thanks
Michal
Quoted 39 lines · Michal Suchánek
Quoted 28 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 27 lines
Hello,
Quoted 23 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Quoted 8 lines
Quoted 1 line
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 4 lines
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
git submodule update --init --recursive didn't work?
Quoted 4 lines
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quentin mentioned that he's packaging (or will package) libbpf sources as part of bpftool release on Github. I've been this for other libbpf-using tools as well, and it works pretty well (at least for Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
[0] https://github.com/libbpf/veristat/releases/tag/v0.1
Quoted 62 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 2 lines
Does bpftool work on older kernel?
It should, although it's not perfect. Most features from current bpftool should work as expected on older kernels. However, if I remember correctly you would have trouble loading programs on pre-BTF kernels, because bpftool relies on libbpf >= 1.0 and only accepts map definitions with BTF info, and attempts to create these maps with BTF, which fails and blocks the load process.
But we're trying to keep backward-compatibility, so if we're only talking of kernels recent enough to support BTF, then I'd expect bpftool to work. If this is not the case, please report on this list.
Quoted 6 lines
Quoted 2 lines
It won't build:
https://lore.kernel.org/bpf/[email protected]/True in this case, and this is something that needs to get fixed. Thanks for reopening that thread! Are you building bpftool on kernels older than 5.15? (genuine curiosity)
Yes, 5.14 and 5.3. I would not be able to notice this particular breakage otherwise.
Quoted 38 lines
Quoted 21 lines
Quoted 21 lines
Quoted 6 lines
Our current approach is that we (openSUSE/SLES) essentially have two version of libbpf: a public shared library that uses GitHub mirror as source, which the general userspace sees and links to; and a private static library built from kernel source used by bpftool, perf,
resolve_btfids, selftests, etc. A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they took similar approach.I would like them to reconsider this choice eventually. Sounds like for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to have a real bpftool package instead of having to install linux-tools-common + linux-tools-generic, or to have distros in general (Ubuntu/Debian at least) stop compiling out the JIT disassembler, although this is not strictly related to the location of the sources. I've not found the time to reach out to package maintainers yet.
Quoted 4 lines
This approach means that the version of bpftool and libbpf are _not_ always in sync[1], which I read may causes problem since libbpf and bpftool depends on specific version of each other[2].
Quoted 8 lines
Quoted 6 lines
Whatever source you use, I would strongly recommend finding a way to keep both in sync. Libbpf has stabilised its API when reaching 1.0, but bpftool taps into some of the internals of the library. Features or new definitions are usually added at the same time to libbpf and bpftool, and if you get a mismatch between the two, you're taking risks to get build issues.
In other words no API exists.
Of course it does. Libbpf exposes a specific set of functions to user applications.
But correct, from bpftool's perspective, there are a few locations where we accept to derogate and to access to the internals directly, making it more dependent on a specific version, or commit, of libbpf, and blurring the notion of API.
Quoted 8 lines
Quoted 4 lines
This special relationship is nothing new though, and it has been discussed before. It derives from both tools being developed in the same repository, and bpftool being so tightly linked to libbpf - it has been qualified of command-line interface for libbpf in the past. Bpftool's
If bpftool is a commandline interface for libbpf maybe the best choice is to just dump both into the same repository, and provide make targets for building one, the other, and both.
bpftool is way more than just an interface to libbpf, so it doesn't make sense to bundle them in one repo. Yes, we take advantage of them being developed together to give it more and better features, which otherwise we probably just wouldn't add to either libbpf or bpftool (like BTFgen, for example).
This whole submodule vs shared library has been discussed so many times. We are open to making your life easier as a packager where possible, if you can be specific about the issues that are hurting you. But this relationship between bpftool and libbpf and use of internal APIs was a conscious decision and it has its big benefits, which definitely outweigh git submodule inconvenience. We are not changing it, it's pretty fundamental for bpftool.
Quoted 69 lines
Quoted 6 lines
version number itself is aligned on libbpf's. (As a side note, bpftool used to pull libbpf's headers directly from libbpf's dir instead of installing them locally, which facilitated this mix-in for public/internal headers in the first place.)
Quoted 8 lines
I know you advocated making the required functions part of the API, given that some users (such as bpftool) need them. These functions are not exposed, by choice. They are not judged relevant to generic user space application (I'm sure libbpf's maintainers are opened to discussion if use cases come up). Some of the internals we get from libbpf are also mostly to avoid re-implementing things, such as netlink attributes processing, or implementing hash maps. These have nothing to do in libbpf's API.
And we do not have microframeworks for implementing reusable hashmaps or netlink parsers. I am sure that bpftool is not the first nor last tool that needs a hashmap or parse netlink but the ecosystem for small single-purpose C libraries never really took off.
Quoted 24 lines
Quoted 16 lines
Quoted 9 lines
Quoted 5 lines
The main concern with using GitHub mirror is that bpftool may be updated far beyond the version that comes with the runtime kernel. AFAIK bpftool should work on older kernel since CO-RE is used for built-in BPF iterators and the underlying libbpf work on older kernel itself. Nonetheless, it would be nice to get a confirmation from the maintainers.
As explained above - Mostly, it should work. Otherwise, we can look into fixing it.
Quoted 6 lines
As a side note, I'm open to suggestions/contributions to make life easier for packaging for the mirror. For example, Mahé and I recently added GitHub workflows to ship statically-built binaries for amd64 and arm64 on releases, as well as tarballs with both bpftool+libbpf sources. If there's something else to make packaging easier, I'm happy to talk about it.
Quoted 7 lines
Make it possible to build with system-installed libbpf. If it's released it should have versioned dependency on a libbpf release, and libbpf from that version on should be good enough to build it.
I tried copying those 'private' headers into a separate directory, and link against static libbpf, and it seems to work. Of course, having an actual API would be much better.
Quoted 6 lines
Just as you said yourself, the missing stability is in the way. I don't see this happening as long as bpftool is using libbpf's internals. I do expect builds to work most of the time by copying the headers as you did, but as soon as something changes and it no longer does, everyone will start filing issues on GitHub instead of using the version that works, and I don't want that.
So these are not really separate projects, they should ship together.
Quoted 9 lines
As for decoupling from the internal: making the functions part of the API is not an option. One option could be to move this code into further dependencies shared between libbpf and bpftool - although I guess libbpf developers will have little appetite for that. We could also duplicate the necessary code in bpftool, which doesn't sound optimal, but might be one solution. Other options have been discussed before, such as moving bpftool into libbpf's directory/mirror and shipping both together, but there was no consensus at the time, and I don't expect libbpf to ship with bpftool any time soon.
Why is that? Is there any speciffic reason for the sepaaration if bpftool needs specific libbpf anyway?
Thanks
Michal
Quoted 40 lines · Andrii Nakryiko
Quoted 39 lines · Michal Suchánek
Quoted 36 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 21 lines
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 7 lines
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 6 lines
Quoted 4 lines
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
git submodule update --init --recursive didn't work?
That's one part of the manual fixup.
The other part is after each git operation that could possibly cause the submodules to go out of sync, basically any operation that changes the checked-out commit.
Of course, you can make some shell aliases that append whatever submodule chicanery to whatever git command you might issue, and tell everyone else to do that, and then it will work in that one shell, and not in any other shell nor any tool that invokes git directly.
Quoted 17 lines
Quoted 3 lines
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quentin mentioned that he's packaging (or will package) libbpf sources as part of bpftool release on Github. I've been this for other libbpf-using tools as well, and it works pretty well (at least for Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
Quoted 1 line
So how many copies of libbpf do I need for having a CO-RE toolchain?
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
When there is a bug in libbpf how many places need to be patched to fix it?
Thanks
Michal
Quoted 51 lines · Michal Suchánek
Quoted 40 lines · Andrii Nakryiko
Quoted 39 lines · Michal Suchánek
Quoted 36 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 21 lines
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 7 lines
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 6 lines
Quoted 4 lines
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
git submodule update --init --recursive didn't work?
That's one part of the manual fixup.
Quoted 8 lines
The other part is after each git operation that could possibly cause the submodules to go out of sync, basically any operation that changes the checked-out commit.
Of course, you can make some shell aliases that append whatever submodule chicanery to whatever git command you might issue, and tell everyone else to do that, and then it will work in that one shell, and not in any other shell nor any tool that invokes git directly.
Are we discussing a *standard* Git submodule feature and argue that because it might be cumbersome or unfamiliar to some engineers that projects should avoid using Git submodules?
For one, I don't have any special aliases for dealing with Git submodules and it works fine. If I jump between branches or tags which update Git submodule reference, I do above git submodule update --init --recursive explicitly if I see that Git status shows out-of-sync Git submodule state. If I want to update a Git submodule, I update the submodule's Git repo, and then git add it in the repo that uses this submodule. I haven't run into any other issues with this.
Having said that, I do realize that some build systems have more troubles with Git submodules (this was a complaint from Fedora packagers), and I empathize with this, which is why I do the archiving of submodule sources when cutting releases. If you'd prefer some other way of dealing with this, please let's have a constructive discussion about that. Suggesting projects to stop using Git submodules isn't that, IMO.
Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quentin mentioned that he's packaging (or will package) libbpf sources as part of bpftool release on Github. I've been this for other libbpf-using tools as well, and it works pretty well (at least for Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
Quoted 3 lines
Quoted 1 line
So how many copies of libbpf do I need for having a CO-RE toolchain?
What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop, etc are tools. The fact they are using statically linked libbpf through Git submodule is irrelevant to end users. You need one libbpf in the system (for those who link dynamically against libbpf), the rest are just tools.
Quoted 3 lines
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
That's up to tools, not libbpf. You are over pivoting on libbpf here. There is one view of the kernel, it depends on what features the kernel supports. If the tool requires some specific functionality of libbpf, it will update its Git submodule reference to get a version of libbpf that provides that feature. That's the point, an application/tool is in control of what kind of features it gets from libbpf.
Quoted 3 lines
When there is a bug in libbpf how many places need to be patched to fix it?
That's up to tools, again. If the bug is affecting them, they should cut a new version of their *tool*, using a patched version of libbpf from Github. If it doesn't affect them, then it doesn't matter *to them*.
Quoted 4 lines
Thanks
Michal
Quoted 56 lines · Andrii Nakryiko
Quoted 55 lines · Michal Suchánek
Quoted 40 lines · Andrii Nakryiko
Quoted 39 lines · Michal Suchánek
Quoted 36 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 21 lines
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 7 lines
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 6 lines
Quoted 4 lines
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
git submodule update --init --recursive didn't work?
That's one part of the manual fixup.
The other part is after each git operation that could possibly cause the submodules to go out of sync, basically any operation that changes the checked-out commit.
Quoted 8 lines
Quoted 4 lines
Of course, you can make some shell aliases that append whatever submodule chicanery to whatever git command you might issue, and tell everyone else to do that, and then it will work in that one shell, and not in any other shell nor any tool that invokes git directly.
Are we discussing a *standard* Git submodule feature and argue that because it might be cumbersome or unfamiliar to some engineers that projects should avoid using Git submodules?
As far as I am aware they are unfamiliar to *most* engineers, and for good reasons.
Quoted 8 lines
For one, I don't have any special aliases for dealing with Git submodules and it works fine. If I jump between branches or tags which update Git submodule reference, I do above
git submodule update --init --recursiveexplicitly if I see that Git status shows out-of-sync Git submodule state. If I want to update a Git submodule, I update the submodule's Git repo, and then git add it in the repo that uses this submodule. I haven't run into any other issues with this.
You know, git could just handle submodules automagically. As you say, it's not rocket science. For historical reasons it does not.
With that working with submodules is cumbersome, and it's additional thing that can break down that the engineer needs to be constantly aware of increasing the mental overhead of working with such projects.
It may not be much of a problem for people who work with such projects daily but not everyone does. Those who don't need to do the mental switch whenever submodules are encountered, and are prone to getting issues when they forget that they have to go that extra mile for this specific project.
Quoted 40 lines
Quoted 17 lines
Quoted 17 lines
Quoted 3 lines
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quentin mentioned that he's packaging (or will package) libbpf sources as part of bpftool release on Github. I've been this for other libbpf-using tools as well, and it works pretty well (at least for Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
Quoted 3 lines
Quoted 1 line
So how many copies of libbpf do I need for having a CO-RE toolchain?
What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop, etc are tools. The fact they are using statically linked libbpf through Git submodule is irrelevant to end users. You need one libbpf in the system (for those who link dynamically against libbpf), the rest are just tools.
Quoted 3 lines
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
That's up to tools, not libbpf. You are over pivoting on libbpf here. There is one view of the kernel, it depends on what features the kernel supports. If the tool requires some specific functionality of libbpf, it will update its Git submodule reference to get a version of libbpf that provides that feature. That's the point, an application/tool is in control of what kind of features it gets from libbpf.
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
When there is a bug in libbpf how many places need to be patched to fix it?
That's up to tools, again. If the bug is affecting them, they should cut a new version of their *tool*, using a patched version of libbpf from Github. If it doesn't affect them, then it doesn't matter *to them*.
I don't share your optimism about this happening in the real world.
For one the issue that the github tarballs do not contain the submodule and thus cannot be built was raised nearly two months ago, and while a test snapshot that does include the submodule is released, a release does not exist yet.
For people to make use of the repository without a release cut they need to replicate that submodule support - that is add support for submodules in their development tooling. Otherwise you personally cutting a release becomes a single point of failure.
Because there is no API it's not really advisable to just apply patches on top of the last release either. Applying patches may cause the main project and the submodule to go out of sync, the submodule would not get updated by applying a patch to the main project, and the other way around.
Suppose a severe security bug that requires patching libbpf is found. Now there is a number of tools that are each tied to one specific version of libbpf, and cannot be upgraded to up-to-date fixed version because that would break them. I would hope that never happens. Nonetheless, libbpf is used to generate code, and if the code is generated wrong worst case it can have severe security implications.
Thanks
Michal
Quoted 84 lines · Michal Suchánek
Quoted 55 lines · Andrii Nakryiko
Quoted 54 lines · Michal Suchánek
Quoted 39 lines · Andrii Nakryiko
Quoted 38 lines · Michal Suchánek
Quoted 36 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 21 lines
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 7 lines
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 6 lines
Quoted 4 lines
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
git submodule update --init --recursive didn't work?
That's one part of the manual fixup.
The other part is after each git operation that could possibly cause the submodules to go out of sync, basically any operation that changes the checked-out commit.
Quoted 8 lines
Quoted 4 lines
Of course, you can make some shell aliases that append whatever submodule chicanery to whatever git command you might issue, and tell everyone else to do that, and then it will work in that one shell, and not in any other shell nor any tool that invokes git directly.
Are we discussing a *standard* Git submodule feature and argue that because it might be cumbersome or unfamiliar to some engineers that projects should avoid using Git submodules?
As far as I am aware they are unfamiliar to *most* engineers, and for good reasons.
Quoted 8 lines
For one, I don't have any special aliases for dealing with Git submodules and it works fine. If I jump between branches or tags which update Git submodule reference, I do above
git submodule update --init --recursiveexplicitly if I see that Git status shows out-of-sync Git submodule state. If I want to update a Git submodule, I update the submodule's Git repo, and then git add it in the repo that uses this submodule. I haven't run into any other issues with this.You know, git could just handle submodules automagically. As you say, it's not rocket science. For historical reasons it does not.
With that working with submodules is cumbersome, and it's additional thing that can break down that the engineer needs to be constantly aware of increasing the mental overhead of working with such projects.
Quoted 5 lines
It may not be much of a problem for people who work with such projects daily but not everyone does. Those who don't need to do the mental switch whenever submodules are encountered, and are prone to getting issues when they forget that they have to go that extra mile for this specific project.
For me it's less about having to go through the extra loop. It's that submodules would require git to be installed, network access, which all adds extra moving parts compared to a tarball...
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quentin mentioned that he's packaging (or will package) libbpf sources as part of bpftool release on Github. I've been this for other libbpf-using tools as well, and it works pretty well (at least for Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
and having libbpf included in bpftool release means the complain above no longer holds. Though I have yet test build the mirror version of libbpf and bpftool like Michal has done.
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 9 lines
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
So how many copies of libbpf do I need for having a CO-RE toolchain?
What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop, etc are tools. The fact they are using statically linked libbpf through Git submodule is irrelevant to end users. You need one libbpf in the system (for those who link dynamically against libbpf), the rest are just tools.
Quoted 3 lines
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
Quoted 7 lines
Quoted 7 lines
That's up to tools, not libbpf. You are over pivoting on libbpf here. There is one view of the kernel, it depends on what features the kernel supports. If the tool requires some specific functionality of libbpf, it will update its Git submodule reference to get a version of libbpf that provides that feature. That's the point, an application/tool is in control of what kind of features it gets from libbpf.
Since libbpf has a stable API & ABI, is it theoretically possible for bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
What I'd like to do it build libbpf and bpftool out of bpftool GitHub mirror's release tarball (w/ submodule included, which exists now for snapshot). For the rest of the tool that does not depends on libbpf private function, have them dynamically link to the libbpf built from bpftool's source, just like how libelf is dynamically linked.
I'm not saying that those tools should not have libbpf as submodule; as submodule do look useful. But for packaging I really would like to have the option of choosing the exact version of libbpf being used.
Quoted 36 lines
Quoted 7 lines
Quoted 2 lines
When there is a bug in libbpf how many places need to be patched to fix it?
That's up to tools, again. If the bug is affecting them, they should cut a new version of their *tool*, using a patched version of libbpf from Github. If it doesn't affect them, then it doesn't matter *to them*.
I don't share your optimism about this happening in the real world.
For one the issue that the github tarballs do not contain the submodule and thus cannot be built was raised nearly two months ago, and while a test snapshot that does include the submodule is released, a release does not exist yet.
For people to make use of the repository without a release cut they need to replicate that submodule support - that is add support for submodules in their development tooling. Otherwise you personally cutting a release becomes a single point of failure.
Because there is no API it's not really advisable to just apply patches on top of the last release either. Applying patches may cause the main project and the submodule to go out of sync, the submodule would not get updated by applying a patch to the main project, and the other way around.
Suppose a severe security bug that requires patching libbpf is found. Now there is a number of tools that are each tied to one specific version of libbpf, and cannot be upgraded to up-to-date fixed version because that would break them. I would hope that never happens. Nonetheless, libbpf is used to generate code, and if the code is generated wrong worst case it can have severe security implications.
Thanks
Michal
Quoted 100 lines · Shung-Hsi Yu
Quoted 84 lines · Michal Suchánek
Quoted 55 lines · Andrii Nakryiko
Quoted 54 lines · Michal Suchánek
Quoted 39 lines · Andrii Nakryiko
Quoted 38 lines · Michal Suchánek
Quoted 36 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 21 lines
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 7 lines
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 6 lines
Quoted 4 lines
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
git submodule update --init --recursive didn't work?
That's one part of the manual fixup.
The other part is after each git operation that could possibly cause the submodules to go out of sync, basically any operation that changes the checked-out commit.
Quoted 8 lines
Quoted 4 lines
Of course, you can make some shell aliases that append whatever submodule chicanery to whatever git command you might issue, and tell everyone else to do that, and then it will work in that one shell, and not in any other shell nor any tool that invokes git directly.
Are we discussing a *standard* Git submodule feature and argue that because it might be cumbersome or unfamiliar to some engineers that projects should avoid using Git submodules?
As far as I am aware they are unfamiliar to *most* engineers, and for good reasons.
Quoted 8 lines
For one, I don't have any special aliases for dealing with Git submodules and it works fine. If I jump between branches or tags which update Git submodule reference, I do above
git submodule update --init --recursiveexplicitly if I see that Git status shows out-of-sync Git submodule state. If I want to update a Git submodule, I update the submodule's Git repo, and then git add it in the repo that uses this submodule. I haven't run into any other issues with this.You know, git could just handle submodules automagically. As you say, it's not rocket science. For historical reasons it does not.
With that working with submodules is cumbersome, and it's additional thing that can break down that the engineer needs to be constantly aware of increasing the mental overhead of working with such projects.
Quoted 5 lines
It may not be much of a problem for people who work with such projects daily but not everyone does. Those who don't need to do the mental switch whenever submodules are encountered, and are prone to getting issues when they forget that they have to go that extra mile for this specific project.
For me it's less about having to go through the extra loop. It's that submodules would require git to be installed, network access, which all adds extra moving parts compared to a tarball...
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quentin mentioned that he's packaging (or will package) libbpf sources as part of bpftool release on Github. I've been this for other libbpf-using tools as well, and it works pretty well (at least for Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
and having libbpf included in bpftool release means the complain above no longer holds. Though I have yet test build the mirror version of libbpf and bpftool like Michal has done.
Great. This seems to work well for other tools that use libbpf through submodule (anakryiko/retsnoop and libbpf/veristat on Github)
Quoted 31 lines
Quoted 23 lines
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
Quoted 3 lines
Quoted 1 line
So how many copies of libbpf do I need for having a CO-RE toolchain?
What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop, etc are tools. The fact they are using statically linked libbpf through Git submodule is irrelevant to end users. You need one libbpf in the system (for those who link dynamically against libbpf), the rest are just tools.
Quoted 3 lines
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
Quoted 7 lines
Quoted 7 lines
That's up to tools, not libbpf. You are over pivoting on libbpf here. There is one view of the kernel, it depends on what features the kernel supports. If the tool requires some specific functionality of libbpf, it will update its Git submodule reference to get a version of libbpf that provides that feature. That's the point, an application/tool is in control of what kind of features it gets from libbpf.
Quoted 2 lines
Since libbpf has a stable API & ABI, is it theoretically possible for bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
No, because libbpf is not just a set of APIs. Newer libbpf versions support more BPF-side features, more kernel features, etc, etc. Libbpf is not a typical user-space library, it is a BPF loader, and even if user-visible API doesn't change, libbpf's support for various BPF-side features is extended. Which is important for tools like bpftool, retsnoop, veristat which rely on loading and working with BPF object files.
Quoted 6 lines
What I'd like to do it build libbpf and bpftool out of bpftool GitHub mirror's release tarball (w/ submodule included, which exists now for snapshot). For the rest of the tool that does not depends on libbpf private function, have them dynamically link to the libbpf built from bpftool's source, just like how libelf is dynamically linked.
Please don't do it, let applications control which libbpf versions they are using. It's not just about user space APIs, I can't emphasize this enough. Don't think you know better than developers of respective applications, don't try to dictate how those applications should be organized and developed.
One good example is iproute2, which chose to link (or not) with libbpf dynamically. Now users periodically report various issues where their BPF object files are not loaded, and it often comes down to unexpected version of libbpf (or lack of libbpf support altogether) which which iproute2 was built/deployed. This is just putting a burden on iproute2 users, and accidentally libbpf maintainers, for no good reason.
Quoted 4 lines
I'm not saying that those tools should not have libbpf as submodule; as submodule do look useful. But for packaging I really would like to have the option of choosing the exact version of libbpf being used.
The exact version of libbpf used by bpftool, retsnoop, veristat, etc *is not relevant* to you as a packager. If you want happy users, use *exact* version of libbpf from submodule to build them, with which application was developed, tested, and advertised supported BPF features. There is no reuse to be done here, they all can be on different (and sometimes not yet released) libbpf version. For good reasons, which are outside of your control as a packager.
Quoted 37 lines
Quoted 37 lines
Quoted 8 lines
Quoted 3 lines
When there is a bug in libbpf how many places need to be patched to fix it?
That's up to tools, again. If the bug is affecting them, they should cut a new version of their *tool*, using a patched version of libbpf from Github. If it doesn't affect them, then it doesn't matter *to them*.
I don't share your optimism about this happening in the real world.
For one the issue that the github tarballs do not contain the submodule and thus cannot be built was raised nearly two months ago, and while a test snapshot that does include the submodule is released, a release does not exist yet.
For people to make use of the repository without a release cut they need to replicate that submodule support - that is add support for submodules in their development tooling. Otherwise you personally cutting a release becomes a single point of failure.
Because there is no API it's not really advisable to just apply patches on top of the last release either. Applying patches may cause the main project and the submodule to go out of sync, the submodule would not get updated by applying a patch to the main project, and the other way around.
Suppose a severe security bug that requires patching libbpf is found. Now there is a number of tools that are each tied to one specific version of libbpf, and cannot be upgraded to up-to-date fixed version because that would break them. I would hope that never happens. Nonetheless, libbpf is used to generate code, and if the code is generated wrong worst case it can have severe security implications.
Thanks
Michal
Quoted 147 lines · Andrii Nakryiko
Quoted 100 lines · Shung-Hsi Yu
Quoted 84 lines · Michal Suchánek
Quoted 55 lines · Andrii Nakryiko
Quoted 54 lines · Michal Suchánek
Quoted 39 lines · Andrii Nakryiko
Quoted 38 lines · Michal Suchánek
Quoted 36 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 21 lines
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 7 lines
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 6 lines
Quoted 4 lines
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
git submodule update --init --recursive didn't work?
That's one part of the manual fixup.
The other part is after each git operation that could possibly cause the submodules to go out of sync, basically any operation that changes the checked-out commit.
Quoted 8 lines
Quoted 4 lines
Of course, you can make some shell aliases that append whatever submodule chicanery to whatever git command you might issue, and tell everyone else to do that, and then it will work in that one shell, and not in any other shell nor any tool that invokes git directly.
Are we discussing a *standard* Git submodule feature and argue that because it might be cumbersome or unfamiliar to some engineers that projects should avoid using Git submodules?
As far as I am aware they are unfamiliar to *most* engineers, and for good reasons.
Quoted 8 lines
For one, I don't have any special aliases for dealing with Git submodules and it works fine. If I jump between branches or tags which update Git submodule reference, I do above
git submodule update --init --recursiveexplicitly if I see that Git status shows out-of-sync Git submodule state. If I want to update a Git submodule, I update the submodule's Git repo, and then git add it in the repo that uses this submodule. I haven't run into any other issues with this.You know, git could just handle submodules automagically. As you say, it's not rocket science. For historical reasons it does not.
With that working with submodules is cumbersome, and it's additional thing that can break down that the engineer needs to be constantly aware of increasing the mental overhead of working with such projects.
Quoted 5 lines
It may not be much of a problem for people who work with such projects daily but not everyone does. Those who don't need to do the mental switch whenever submodules are encountered, and are prone to getting issues when they forget that they have to go that extra mile for this specific project.
For me it's less about having to go through the extra loop. It's that submodules would require git to be installed, network access, which all adds extra moving parts compared to a tarball...
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quentin mentioned that he's packaging (or will package) libbpf sources as part of bpftool release on Github. I've been this for other libbpf-using tools as well, and it works pretty well (at least for Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
and having libbpf included in bpftool release means the complain above no longer holds. Though I have yet test build the mirror version of libbpf and bpftool like Michal has done.
Great. This seems to work well for other tools that use libbpf through submodule (anakryiko/retsnoop and libbpf/veristat on Github)
Quoted 31 lines
Quoted 23 lines
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
Quoted 3 lines
Quoted 1 line
So how many copies of libbpf do I need for having a CO-RE toolchain?
What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop, etc are tools. The fact they are using statically linked libbpf through Git submodule is irrelevant to end users. You need one libbpf in the system (for those who link dynamically against libbpf), the rest are just tools.
Quoted 3 lines
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
Quoted 7 lines
Quoted 7 lines
That's up to tools, not libbpf. You are over pivoting on libbpf here. There is one view of the kernel, it depends on what features the kernel supports. If the tool requires some specific functionality of libbpf, it will update its Git submodule reference to get a version of libbpf that provides that feature. That's the point, an application/tool is in control of what kind of features it gets from libbpf.
Quoted 2 lines
Since libbpf has a stable API & ABI, is it theoretically possible for bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
Quoted 7 lines
No, because libbpf is not just a set of APIs. Newer libbpf versions support more BPF-side features, more kernel features, etc, etc. Libbpf is not a typical user-space library, it is a BPF loader, and even if user-visible API doesn't change, libbpf's support for various BPF-side features is extended. Which is important for tools like bpftool, retsnoop, veristat which rely on loading and working with BPF object files.
The converse of this is also true: if your system is upgraded to a new kernel version with new BPF features, the libbpf version should follow it, and all applications linked against it will automatically take advantage of any bugfixes regardless without having to wait for each application to be updated.
Libbpf is really no different from any other library here, and I really don't get why you keep insisting it's "special"...
Quoted 11 lines
Quoted 5 lines
What I'd like to do it build libbpf and bpftool out of bpftool GitHub mirror's release tarball (w/ submodule included, which exists now for snapshot). For the rest of the tool that does not depends on libbpf private function, have them dynamically link to the libbpf built from bpftool's source, just like how libelf is dynamically linked.
Please don't do it, let applications control which libbpf versions they are using. It's not just about user space APIs, I can't emphasize this enough. Don't think you know better than developers of respective applications, don't try to dictate how those applications should be organized and developed.
A well-behaved application will detect which features are available in the system version of the libraries they use, and if something is missing that it needs, either work around it or refuse to build. We do this with libbpf in xdp-tools and the only issues we've had with it has been the changing API in pre-1.0 libbpf...
Quoted 6 lines
One good example is iproute2, which chose to link (or not) with libbpf dynamically. Now users periodically report various issues where their BPF object files are not loaded, and it often comes down to unexpected version of libbpf (or lack of libbpf support altogether) which which iproute2 was built/deployed. This is just putting a burden on iproute2 users, and accidentally libbpf maintainers, for no good reason.
How would this have been any different if iproute2 was statically linked against libbpf?
Quoted 11 lines
Quoted 3 lines
I'm not saying that those tools should not have libbpf as submodule; as submodule do look useful. But for packaging I really would like to have the option of choosing the exact version of libbpf being used.
The exact version of libbpf used by bpftool, retsnoop, veristat, etc *is not relevant* to you as a packager. If you want happy users, use *exact* version of libbpf from submodule to build them, with which application was developed, tested, and advertised supported BPF features. There is no reuse to be done here, they all can be on different (and sometimes not yet released) libbpf version. For good reasons, which are outside of your control as a packager.
This is... just not how distributions work. As a user I trust my distribution to provide me with a coherent system where critical system libraries are maintained and receive timely updates. And I absolutely trust the distribution more to do this over application developers who just vendor in some version as a submodule and leave it there until they need a new feature...
-Toke
Quoted 158 lines · Toke Høiland-Jørgensen
Quoted 147 lines · Andrii Nakryiko
Quoted 100 lines · Shung-Hsi Yu
Quoted 84 lines · Michal Suchánek
Quoted 55 lines · Andrii Nakryiko
Quoted 54 lines · Michal Suchánek
Quoted 39 lines · Andrii Nakryiko
Quoted 38 lines · Michal Suchánek
Quoted 36 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 21 lines
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 7 lines
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 6 lines
Quoted 4 lines
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
git submodule update --init --recursive didn't work?
That's one part of the manual fixup.
The other part is after each git operation that could possibly cause the submodules to go out of sync, basically any operation that changes the checked-out commit.
Quoted 8 lines
Quoted 4 lines
Of course, you can make some shell aliases that append whatever submodule chicanery to whatever git command you might issue, and tell everyone else to do that, and then it will work in that one shell, and not in any other shell nor any tool that invokes git directly.
Are we discussing a *standard* Git submodule feature and argue that because it might be cumbersome or unfamiliar to some engineers that projects should avoid using Git submodules?
As far as I am aware they are unfamiliar to *most* engineers, and for good reasons.
Quoted 8 lines
For one, I don't have any special aliases for dealing with Git submodules and it works fine. If I jump between branches or tags which update Git submodule reference, I do above
git submodule update --init --recursiveexplicitly if I see that Git status shows out-of-sync Git submodule state. If I want to update a Git submodule, I update the submodule's Git repo, and then git add it in the repo that uses this submodule. I haven't run into any other issues with this.You know, git could just handle submodules automagically. As you say, it's not rocket science. For historical reasons it does not.
With that working with submodules is cumbersome, and it's additional thing that can break down that the engineer needs to be constantly aware of increasing the mental overhead of working with such projects.
Quoted 5 lines
It may not be much of a problem for people who work with such projects daily but not everyone does. Those who don't need to do the mental switch whenever submodules are encountered, and are prone to getting issues when they forget that they have to go that extra mile for this specific project.
For me it's less about having to go through the extra loop. It's that submodules would require git to be installed, network access, which all adds extra moving parts compared to a tarball...
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quentin mentioned that he's packaging (or will package) libbpf sources as part of bpftool release on Github. I've been this for other libbpf-using tools as well, and it works pretty well (at least for Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
and having libbpf included in bpftool release means the complain above no longer holds. Though I have yet test build the mirror version of libbpf and bpftool like Michal has done.
Great. This seems to work well for other tools that use libbpf through submodule (anakryiko/retsnoop and libbpf/veristat on Github)
Quoted 31 lines
Quoted 23 lines
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
Quoted 3 lines
Quoted 1 line
So how many copies of libbpf do I need for having a CO-RE toolchain?
What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop, etc are tools. The fact they are using statically linked libbpf through Git submodule is irrelevant to end users. You need one libbpf in the system (for those who link dynamically against libbpf), the rest are just tools.
Quoted 3 lines
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
Quoted 7 lines
Quoted 7 lines
That's up to tools, not libbpf. You are over pivoting on libbpf here. There is one view of the kernel, it depends on what features the kernel supports. If the tool requires some specific functionality of libbpf, it will update its Git submodule reference to get a version of libbpf that provides that feature. That's the point, an application/tool is in control of what kind of features it gets from libbpf.
Quoted 2 lines
Since libbpf has a stable API & ABI, is it theoretically possible for bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
Quoted 7 lines
No, because libbpf is not just a set of APIs. Newer libbpf versions support more BPF-side features, more kernel features, etc, etc. Libbpf is not a typical user-space library, it is a BPF loader, and even if user-visible API doesn't change, libbpf's support for various BPF-side features is extended. Which is important for tools like bpftool, retsnoop, veristat which rely on loading and working with BPF object files.
Quoted 5 lines
The converse of this is also true: if your system is upgraded to a new kernel version with new BPF features, the libbpf version should follow it, and all applications linked against it will automatically take advantage of any bugfixes regardless without having to wait for each application to be updated.
No, if my application was not developed to take advantage of a new kernel feature, newer libbpf will do nothing for me. If my application wants to support that feature, I'll update my application and correspondingly update libbpf embedded in it. If my application is affected by some bug fix, I'll update libbpf even faster than distros will get to it.
I've heard all such arguments over the last few years. They are not convincing and my own practical experience shows irrelevance of the above argument.
Quoted 3 lines
Libbpf is really no different from any other library here, and I really don't get why you keep insisting it's "special"...
It's special in the sense that it provides two sets of APIs -- for user-space (typical libraries) and BPF object files. Besides that, for BPF-side it's not even a set of APIs (headers, helpers, etc), it also provides some set of functionality that can improve or be extended over time. E.g., libbpf used to not support non-inlined BPF subprograms, and then it started supporting them. In terms of API/ABI -- nothing changed. Yet the change is very important.
Now, I build a tool that is using libbpf and some BPF functionality, e.g., retsnoop. Libbpf just got SEC("ksyscall") support. Retsnoop wants to take advantage of it. I just go and use SEC("ksyscall") programs in .bpf.c files that are embedded inside retsnoop. I don't have to *and don't want to* do feature detection of whether a particular libbpf version that happens to be installed/packaged on the system supports this version. I *know* it does, because I control it, through a submodule. That's what I care about.
Whether some distro insists on libbpf being shared across any libbpf-using application or not is none of my concern. Libbpf is an implementation detail of my application (retsnoop), it's not for the packager to decide how I develop and structure my tool.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
What I'd like to do it build libbpf and bpftool out of bpftool GitHub mirror's release tarball (w/ submodule included, which exists now for snapshot). For the rest of the tool that does not depends on libbpf private function, have them dynamically link to the libbpf built from bpftool's source, just like how libelf is dynamically linked.
Quoted 7 lines
Quoted 5 lines
Please don't do it, let applications control which libbpf versions they are using. It's not just about user space APIs, I can't emphasize this enough. Don't think you know better than developers of respective applications, don't try to dictate how those applications should be organized and developed.
A well-behaved application will detect which features are available in
No, a well-behaved application will provide a reliable functionality without necessarily paying maintenance and development cost of a maze of #ifdef-ery just to satisfy arbitrary distro requirements of linking with some shared library ("because security").
Quoted 12 lines
the system version of the libraries they use, and if something is missing that it needs, either work around it or refuse to build. We do this with libbpf in xdp-tools and the only issues we've had with it has been the changing API in pre-1.0 libbpf...
Quoted 6 lines
One good example is iproute2, which chose to link (or not) with libbpf dynamically. Now users periodically report various issues where their BPF object files are not loaded, and it often comes down to unexpected version of libbpf (or lack of libbpf support altogether) which which iproute2 was built/deployed. This is just putting a burden on iproute2 users, and accidentally libbpf maintainers, for no good reason.
Quoted 2 lines
How would this have been any different if iproute2 was statically linked against libbpf?
iproute2 version would determine what BPF features are supported, and it would be consistent across distros and end user systems, regardless of what libbpf shared library happens to be packaged and installed. And users would know that starting from version X iproute2 is libbpf-1.0+ compatible in what sort of BPF object file features are supported by iproute2 when loading BPF programs.
Quoted 13 lines
Quoted 5 lines
Quoted 4 lines
I'm not saying that those tools should not have libbpf as submodule; as submodule do look useful. But for packaging I really would like to have the option of choosing the exact version of libbpf being used.
Quoted 7 lines
The exact version of libbpf used by bpftool, retsnoop, veristat, etc *is not relevant* to you as a packager. If you want happy users, use *exact* version of libbpf from submodule to build them, with which application was developed, tested, and advertised supported BPF features. There is no reuse to be done here, they all can be on different (and sometimes not yet released) libbpf version. For good reasons, which are outside of your control as a packager.
Quoted 6 lines
This is... just not how distributions work. As a user I trust my distribution to provide me with a coherent system where critical system libraries are maintained and receive timely updates. And I absolutely trust the distribution more to do this over application developers who just vendor in some version as a submodule and leave it there until they need a new feature...
Ok.
Quoted 3 lines
-Toke
Quoted 47 lines · Andrii Nakryiko
Quoted 41 lines
Quoted 33 lines
Quoted 30 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 9 lines
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
So how many copies of libbpf do I need for having a CO-RE toolchain?
What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop, etc are tools. The fact they are using statically linked libbpf through Git submodule is irrelevant to end users. You need one libbpf in the system (for those who link dynamically against libbpf), the rest are just tools.
Quoted 3 lines
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
Quoted 7 lines
Quoted 7 lines
That's up to tools, not libbpf. You are over pivoting on libbpf here. There is one view of the kernel, it depends on what features the kernel supports. If the tool requires some specific functionality of libbpf, it will update its Git submodule reference to get a version of libbpf that provides that feature. That's the point, an application/tool is in control of what kind of features it gets from libbpf.
Quoted 2 lines
Since libbpf has a stable API & ABI, is it theoretically possible for bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
Quoted 7 lines
No, because libbpf is not just a set of APIs. Newer libbpf versions support more BPF-side features, more kernel features, etc, etc. Libbpf is not a typical user-space library, it is a BPF loader, and even if user-visible API doesn't change, libbpf's support for various BPF-side features is extended. Which is important for tools like bpftool, retsnoop, veristat which rely on loading and working with BPF object files.
Quoted 5 lines
The converse of this is also true: if your system is upgraded to a new kernel version with new BPF features, the libbpf version should follow it, and all applications linked against it will automatically take advantage of any bugfixes regardless without having to wait for each application to be updated.
Quoted 6 lines
No, if my application was not developed to take advantage of a new kernel feature, newer libbpf will do nothing for me. If my application wants to support that feature, I'll update my application and correspondingly update libbpf embedded in it. If my application is affected by some bug fix, I'll update libbpf even faster than distros will get to it.
You may do that, but you're also someone who is following the development of libbpf closely and pay attention to when bugs appear. Not all applications developers have the same vigilance for all the libraries they rely on. Which is the reason distros generally take on the responsibility of ensuring their users receive timely library updates.
Quoted 3 lines
I've heard all such arguments over the last few years. They are not convincing and my own practical experience shows irrelevance of the above argument.
I don't doubt your personal experience, I'm just objecting to you dismissing other points of view just because you haven't experienced them yourself.
Quoted 10 lines
Quoted 2 lines
Libbpf is really no different from any other library here, and I really don't get why you keep insisting it's "special"...
It's special in the sense that it provides two sets of APIs -- for user-space (typical libraries) and BPF object files. Besides that, for BPF-side it's not even a set of APIs (headers, helpers, etc), it also provides some set of functionality that can improve or be extended over time. E.g., libbpf used to not support non-inlined BPF subprograms, and then it started supporting them. In terms of API/ABI -- nothing changed. Yet the change is very important.
Lots of libraries do that. File format libraries support new format features without changing their API, networking libraries support new protocol features, etc. So again, libbpf is not special in this respect.
Quoted 8 lines
Now, I build a tool that is using libbpf and some BPF functionality, e.g., retsnoop. Libbpf just got
SEC("ksyscall")support. Retsnoop wants to take advantage of it. I just go and useSEC("ksyscall")programs in.bpf.cfiles that are embedded inside retsnoop. I don't have to *and don't want to* do feature detection of whether a particular libbpf version that happens to be installed/packaged on the system supports this version. I *know* it does, because I control it, through a submodule. That's what I care about.
Right, so just require a minimum version of the library where the API you want to use is available. That is pretty standard and distros deal with this all the time. This is not an argument for static linking or vendoring...
Quoted 4 lines
Whether some distro insists on libbpf being shared across any libbpf-using application or not is none of my concern. Libbpf is an implementation detail of my application (retsnoop), it's not for the packager to decide how I develop and structure my tool.
Right, well, you don't *have* to be cooperative with the wider ecosystem, of course. Just as packagers don't have to follow your recommendations if they have good reasons not to. I believe we've had this discussion before, and I don't think we're going to agree this time around either, so let's not waste any more virtual ink on rehashing it :)
-Toke
Quoted 57 lines · Toke Høiland-Jørgensen
Quoted 47 lines · Andrii Nakryiko
Quoted 41 lines
Quoted 33 lines
Quoted 30 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 9 lines
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
So how many copies of libbpf do I need for having a CO-RE toolchain?
What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop, etc are tools. The fact they are using statically linked libbpf through Git submodule is irrelevant to end users. You need one libbpf in the system (for those who link dynamically against libbpf), the rest are just tools.
Quoted 3 lines
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
Quoted 7 lines
Quoted 7 lines
That's up to tools, not libbpf. You are over pivoting on libbpf here. There is one view of the kernel, it depends on what features the kernel supports. If the tool requires some specific functionality of libbpf, it will update its Git submodule reference to get a version of libbpf that provides that feature. That's the point, an application/tool is in control of what kind of features it gets from libbpf.
Quoted 2 lines
Since libbpf has a stable API & ABI, is it theoretically possible for bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
Quoted 7 lines
No, because libbpf is not just a set of APIs. Newer libbpf versions support more BPF-side features, more kernel features, etc, etc. Libbpf is not a typical user-space library, it is a BPF loader, and even if user-visible API doesn't change, libbpf's support for various BPF-side features is extended. Which is important for tools like bpftool, retsnoop, veristat which rely on loading and working with BPF object files.
Quoted 5 lines
The converse of this is also true: if your system is upgraded to a new kernel version with new BPF features, the libbpf version should follow it, and all applications linked against it will automatically take advantage of any bugfixes regardless without having to wait for each application to be updated.
Quoted 6 lines
No, if my application was not developed to take advantage of a new kernel feature, newer libbpf will do nothing for me. If my application wants to support that feature, I'll update my application and correspondingly update libbpf embedded in it. If my application is affected by some bug fix, I'll update libbpf even faster than distros will get to it.
Quoted 6 lines
You may do that, but you're also someone who is following the development of libbpf closely and pay attention to when bugs appear. Not all applications developers have the same vigilance for all the libraries they rely on. Which is the reason distros generally take on the responsibility of ensuring their users receive timely library updates.
The discussion was about bpftool, veristat, and retsnoop. "You may do that" applies to all of them. I'm not forcing anyone else to follow the same approach (e.g., I'm not forcing perf to vendor libbpf, for example), I'm just opposing someone else forcing us (bpftool, veristat, retsnoop) to not vendor libbpf.
Quoted 8 lines
Quoted 4 lines
I've heard all such arguments over the last few years. They are not convincing and my own practical experience shows irrelevance of the above argument.
I don't doubt your personal experience, I'm just objecting to you dismissing other points of view just because you haven't experienced them yourself.
I acknowledged the security argument. And disagreeing and dismissing are two big differences. So yes, I don't think the security argument outweighs all the downsides of not controlling the exact libbpf version my application relies on. And specifically for libbpf-related use cases, libbpf-using applications are running under root. They are not supposed to accept untrusted ELF files. So even if there is some bug leading to crashes or some malfunction, I don't buy the "emergency, we need to update all libbpf-using apps ASAP" argument, sorry.
Quoted 12 lines
Quoted 11 lines
Quoted 3 lines
Libbpf is really no different from any other library here, and I really don't get why you keep insisting it's "special"...
It's special in the sense that it provides two sets of APIs -- for user-space (typical libraries) and BPF object files. Besides that, for BPF-side it's not even a set of APIs (headers, helpers, etc), it also provides some set of functionality that can improve or be extended over time. E.g., libbpf used to not support non-inlined BPF subprograms, and then it started supporting them. In terms of API/ABI -- nothing changed. Yet the change is very important.
Quoted 4 lines
Lots of libraries do that. File format libraries support new format features without changing their API, networking libraries support new protocol features, etc. So again, libbpf is not special in this respect.
I didn't mean to start a "which library is the most special" contest. The original point was about libbpf 1.0 stability of API, and that was my objection, because it's not just about available APIs and their stability. And libbpf is not used for the sake of using libbpf, it is used with (usually embedded) BPF object file(s) that application is expected to work, including any new SEC() support, global variables, etc, etc.
Quoted 10 lines
Quoted 9 lines
Now, I build a tool that is using libbpf and some BPF functionality, e.g., retsnoop. Libbpf just got
SEC("ksyscall")support. Retsnoop wants to take advantage of it. I just go and useSEC("ksyscall")programs in.bpf.cfiles that are embedded inside retsnoop. I don't have to *and don't want to* do feature detection of whether a particular libbpf version that happens to be installed/packaged on the system supports this version. I *know* it does, because I control it, through a submodule. That's what I care about.
Quoted 4 lines
Right, so just require a minimum version of the library where the API you want to use is available. That is pretty standard and distros deal with this all the time. This is not an argument for static linking or vendoring...
bpftool can't do that due to use of internal APIs. For others (retsnoop, veristat), I don't want to be restricted by the released libbpf version, or by expecting that the target system has new enough libbpf installed.
Quoted 11 lines
Quoted 5 lines
Whether some distro insists on libbpf being shared across any libbpf-using application or not is none of my concern. Libbpf is an implementation detail of my application (retsnoop), it's not for the packager to decide how I develop and structure my tool.
Right, well, you don't *have* to be cooperative with the wider ecosystem, of course. Just as packagers don't have to follow your recommendations if they have good reasons not to. I believe we've had this discussion before, and I don't think we're going to agree this time around either, so let's not waste any more virtual ink on rehashing it :)
Exactly, so I'm not sure why we are even having this conversation all over again. I agree on not wasting virtual ink anymore. I'm not forcing anyone to follow my advice, I expect others to not force me to follow theirs.
Quoted 3 lines
-Toke
Quoted 8 lines · Andrii Nakryiko
Quoted 6 lines · Toke Høiland-Jørgensen
[snip]
Right, well, you don't *have* to be cooperative with the wider ecosystem, of course. Just as packagers don't have to follow your recommendations if they have good reasons not to. I believe we've had this discussion before, and I don't think we're going to agree this time around either, so let's not waste any more virtual ink on rehashing it :)
Quoted 4 lines
Exactly, so I'm not sure why we are even having this conversation all over again. I agree on not wasting virtual ink anymore. I'm not forcing anyone to follow my advice, I expect others to not force me to follow theirs.
Thanks for still going through the reasoning.
I don't have anything to add to the discussion, so instead here's an attempt to summarize the thread thus far, reading between the lines here and there to keep it terse but complete; feel free to point out where I misunderstood.
# Packaging bpftool and libbpf
- bpftool and libbpf version should be kept in sync
- interdependency is by design
- bpftool uses private functionality of libbpf
- bpftool generated file is tie to specific libbpf (?)
- the GitHub mirror is the recommended source
- benefits of using the GitHub mirror includes
- ease of upgrade
- maintainer crafted changelog
- downsides of using the GitHub mirror has to do with kernel backporting
- git submodule requires extra work for distros to package
- offsetted if the source of submodules are released along
- bpftool releases will (have a file that) includes submodules' source along going forward
- bpftool and libbpf both should work on earlier kernel (if not it's a bug)
# Other
- motivations for GitHub mirror
- ease of distribution, packaging, build
- CI, to be used as submodule, Window support, etc.
- libbpf interface stability
- stable API and ABI (within major version)
- BPF object format is not considered stable
- libbpf is not opinionated in how it's used as a library, either
- statically or dynamically linked
- a tagged release or a random commit
- on statically linking libbpf
- reasoning
- full control of implementation detail, decouples from distro package
- against
- difficulty in applying fixes
Shung-Hsi
Quoted 3 lines
Quoted 3 lines
-Toke
Quoted 22 lines · Shung-Hsi Yu
Quoted 8 lines · Andrii Nakryiko
Quoted 6 lines · Toke Høiland-Jørgensen
[snip]
Right, well, you don't *have* to be cooperative with the wider ecosystem, of course. Just as packagers don't have to follow your recommendations if they have good reasons not to. I believe we've had this discussion before, and I don't think we're going to agree this time around either, so let's not waste any more virtual ink on rehashing it :)Quoted 4 lines
Exactly, so I'm not sure why we are even having this conversation all over again. I agree on not wasting virtual ink anymore. I'm not forcing anyone to follow my advice, I expect others to not force me to follow theirs.
Thanks for still going through the reasoning.
I don't have anything to add to the discussion, so instead here's an attempt to summarize the thread thus far, reading between the lines here and there to keep it terse but complete; feel free to point out where I misunderstood.
Quoted 6 lines
# Packaging bpftool and libbpf
- bpftool and libbpf version should be kept in sync
- interdependency is by design
- bpftool uses private functionality of libbpf
- bpftool generated file is tie to specific libbpf (?)
this bullet point is not true, generated BPF skeleton or statically linked BPF object file should work across wide range of libbpf versions
Quoted 6 lines
- the GitHub mirror is the recommended source
- benefits of using the GitHub mirror includes
- ease of upgrade
- maintainer crafted changelog
I'd also say consistency between all distros (assuming everyone use Github mirror). Because release X means exactly the same set of commits.
Quoted 14 lines
- downsides of using the GitHub mirror has to do with kernel backporting
- git submodule requires extra work for distros to package
- offsetted if the source of submodules are released along- bpftool releases will (have a file that) includes submodules' source along going forward
- bpftool and libbpf both should work on earlier kernel (if not it's a bug)
# Other
Quoted 7 lines
- motivations for GitHub mirror
- ease of distribution, packaging, build
- CI, to be used as submodule, Window support, etc.- libbpf interface stability
- stable API and ABI (within major version)
- BPF object format is not considered stable
BPF object file format is stable, but not frozen, it can keep evolving
Quoted 11 lines
- libbpf is not opinionated in how it's used as a library, either
- statically or dynamically linked
- a tagged release or a random commit- on statically linking libbpf
- reasoning
- full control of implementation detail, decouples from distro package
- against
- difficulty in applying fixes
Looks good to me apart from things I pointed out above. Thanks for summarizing!
Quoted 6 lines
Shung-Hsi
Quoted 3 lines
Quoted 3 lines
-Toke
Quoted 86 lines · Michal Suchánek
Quoted 56 lines · Andrii Nakryiko
Quoted 55 lines · Michal Suchánek
Quoted 40 lines · Andrii Nakryiko
Quoted 39 lines · Michal Suchánek
Quoted 36 lines · Quentin Monnet
2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <[email protected]>
Quoted 21 lines
Hello,
Quoted 18 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
Quoted 7 lines
Quoted 5 lines
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Things get only ever more complex when submodules are involved.
I understand the generic pain points from your other email. But could you be more specific for the case of bpftool? It's not like we're shipping all lib dependencies as submodules. Sync-ups are specifically aligned to the same commit used to sync the libbpf mirror, so that it's pretty much as if we had the right version of the library shipped in the repository - only, it's one --recurse-submodules away.
Quoted 6 lines
Quoted 4 lines
It's so in every project that uses submodules. Except git does not recurse into submodules by default, you have to fix it up by hand. Forges don't support submodules so you will not get the submodule when downloading the project archive, and won't see it the the project tree.
git submodule update --init --recursive didn't work?
That's one part of the manual fixup.
The other part is after each git operation that could possibly cause the submodules to go out of sync, basically any operation that changes the checked-out commit.
Quoted 8 lines
Quoted 4 lines
Of course, you can make some shell aliases that append whatever submodule chicanery to whatever git command you might issue, and tell everyone else to do that, and then it will work in that one shell, and not in any other shell nor any tool that invokes git directly.
Are we discussing a *standard* Git submodule feature and argue that because it might be cumbersome or unfamiliar to some engineers that projects should avoid using Git submodules?
As far as I am aware they are unfamiliar to *most* engineers, and for good reasons.
Quoted 8 lines
For one, I don't have any special aliases for dealing with Git submodules and it works fine. If I jump between branches or tags which update Git submodule reference, I do above
git submodule update --init --recursiveexplicitly if I see that Git status shows out-of-sync Git submodule state. If I want to update a Git submodule, I update the submodule's Git repo, and then git add it in the repo that uses this submodule. I haven't run into any other issues with this.You know, git could just handle submodules automagically. As you say, it's not rocket science. For historical reasons it does not.
With that working with submodules is cumbersome, and it's additional thing that can break down that the engineer needs to be constantly aware of increasing the mental overhead of working with such projects.
Quoted 5 lines
It may not be much of a problem for people who work with such projects daily but not everyone does. Those who don't need to do the mental switch whenever submodules are encountered, and are prone to getting issues when they forget that they have to go that extra mile for this specific project.
I agree with you that Git submodules are not the most straightforward feature usability-wise. Where I disagree is that this is enough reason to not use Git submodules. They do provide a lot of value.
Quoted 49 lines
Quoted 41 lines
Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
After previous experience with submodules I did not even try, I just patched the makefile to use system libbpf before attempting anything else.
Quentin mentioned that he's packaging (or will package) libbpf sources as part of bpftool release on Github. I've been this for other libbpf-using tools as well, and it works pretty well (at least for Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
By switching up actual libbpf used to compile with bpftool, you are potentially introducing subtle problems that your users will be quite unhappy about, if they run into them. Let's work together to make it easier for you to package bpftool properly. We can't switch bpftool to reliably use system-wide libbpf (either static or shared, doesn't matter) because of dependency on internal functionality.
Quoted 3 lines
Quoted 1 line
So how many copies of libbpf do I need for having a CO-RE toolchain?
What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop, etc are tools. The fact they are using statically linked libbpf through Git submodule is irrelevant to end users. You need one libbpf in the system (for those who link dynamically against libbpf), the rest are just tools.
Quoted 3 lines
Will different tools have different view of the kernel because they each use different private copy of libbpf with different features?
That's up to tools, not libbpf. You are over pivoting on libbpf here. There is one view of the kernel, it depends on what features the kernel supports. If the tool requires some specific functionality of libbpf, it will update its Git submodule reference to get a version of libbpf that provides that feature. That's the point, an application/tool is in control of what kind of features it gets from libbpf.
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
When there is a bug in libbpf how many places need to be patched to fix it?
That's up to tools, again. If the bug is affecting them, they should cut a new version of their *tool*, using a patched version of libbpf from Github. If it doesn't affect them, then it doesn't matter *to them*.
Quoted 6 lines
I don't share your optimism about this happening in the real world.
For one the issue that the github tarballs do not contain the submodule and thus cannot be built was raised nearly two months ago, and while a test snapshot that does include the submodule is released, a release does not exist yet.
Because no one raised this issue earlier (for bpftool). Fedora packagers raised it for retsnoop, so retsnoop has it. That's how development works, one can't anticipate all the possible issues, they need to be reported and worked on.
Quoted 5 lines
For people to make use of the repository without a release cut they need to replicate that submodule support - that is add support for submodules in their development tooling. Otherwise you personally cutting a release becomes a single point of failure.
I hope distros won't be packaging an unreleased version, though?
Quoted 6 lines
Because there is no API it's not really advisable to just apply patches on top of the last release either. Applying patches may cause the main project and the submodule to go out of sync, the submodule would not get updated by applying a patch to the main project, and the other way around.
I'm not sure where we are going with this overall discussion at this point, tbh.
Quoted 8 lines
Suppose a severe security bug that requires patching libbpf is found. Now there is a number of tools that are each tied to one specific version of libbpf, and cannot be upgraded to up-to-date fixed version because that would break them. I would hope that never happens. Nonetheless, libbpf is used to generate code, and if the code is generated wrong worst case it can have severe security implications.
Yes, I hear this argument from packagers all the time. Yet, somehow it's been fine for the last few years. Please realize that there are many reasons why we do want submodules and static linking of libbpf, and accept that projects do decide to stick to submodules. It might not be perfect, but the benefits of such an approach outweigh the hypothetical issues you brought up.
This has all been discussed multiple times, as I said, I don't think any of us added anything new to previous discussions. So if you are interested, please work with us to improve your life as a packager, but also accept the way this project is set up on Github.
Quoted 3 lines
Thanks
Michal
Quoted 3 lines · Quentin Monnet
My point is that I've received little feedback on the mirror
so far, so it's hard for me to figure out who's using it or whether anyone has been reading the changelogs.
FWIW, the ebpf-for-windows project uses both bpftool and libbpf as git submodules.
Both of them are, today, very Linux-centric and not usable directly cross-platform (something I'd like to see change in the future as I've previously discussed but not had enough cycles to drive much).
So today the libbpf mirror is used as a submodule just to get the header files, where ebpf-for-windows currently uses its own C implementation of the same prototypes, to map to Windows ioctls instead of Linux syscalls.
And today bpftool is used via a github fork of the mirror, where various ifdefs and Windows-specific alternatives are in the fork. The intent is to clean it up and upstream it, but I just haven't gotten to that yet. But ebpf-for-windows does not currently support BTF so it is like an older Linux kernel in that sense.
So we haven't updated either to commits that rely on libbpf >= 1.0 since the APIs we support were removed.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Does bpftool work on older kernel?
It should, although it's not perfect. Most features from current bpftool should work as expected on older kernels. However, if I remember correctly you would have trouble loading programs on pre-BTF kernels, because bpftool relies on libbpf >= 1.0 and only accepts map definitions with BTF info, and attempts to create these maps with BTF, which fails and blocks the load process.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
But we're trying to keep backward-compatibility, so if we're only talking of kernels recent enough to support BTF, then I'd expect bpftool to work. If this is not the case, please report on this list.
Yeah, unless/until we support BTF in the future, we cannot use the current HEAD of the libbpf or bpftool mirrors, only older commits we sync to.
In that sense, the removal of pre-1.0 APIs made the mirrors more Linux-only and less cross-platform.
Dave
Quoted 25 lines · Quentin Monnet
Hi Shung-Hsi,
Quoted 10 lines · Shung-Hsi Yu
Hi,
I'm considering switch to bpftool's mirror on GitHub for packaging (instead of using the source found in kernel), but realize that it should goes hand-in-hand with how libbpf is packaged, which eventually leads these questions:
What is the suggested approach for packaging bpftool and libbpf? Which source is preferred, GitHub or kernel?
As you can see from the previous discussions, the suggested approach would be to package from the GitHub mirror, with libbpf and bpftool in sync.
My main argument for the mirror is that it keeps things simpler, and there's no need to deal with the rest of the kernel sources for these packages. Download from the mirrors, build, ship. But then I have limited experience at packaging for distros, and I can understand Toke's point of view, too. So ultimately, the call is yours.
Quoted 8 lines
Quoted 1 line
Does bpftool work on older kernel?
It should, although it's not perfect. Most features from current bpftool should work as expected on older kernels. However, if I remember correctly you would have trouble loading programs on pre-BTF kernels, because bpftool relies on libbpf >= 1.0 and only accepts map definitions with BTF info, and attempts to create these maps with BTF, which fails and blocks the load process.
Not really. Libbpf won't upload BTF if the kernel doesn't support it and it's not required for correct BPF map (e.g., local storage) or BPF program functioning.
Quoted 55 lines
But we're trying to keep backward-compatibility, so if we're only talking of kernels recent enough to support BTF, then I'd expect bpftool to work. If this is not the case, please report on this list.
Quoted 7 lines
Our current approach is that we (openSUSE/SLES) essentially have two version of libbpf: a public shared library that uses GitHub mirror as source, which the general userspace sees and links to; and a private static library built from kernel source used by bpftool, perf,
resolve_btfids, selftests, etc. A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they took similar approach.I would like them to reconsider this choice eventually. Sounds like for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to have a real bpftool package instead of having to install linux-tools-common + linux-tools-generic, or to have distros in general (Ubuntu/Debian at least) stop compiling out the JIT disassembler, although this is not strictly related to the location of the sources. I've not found the time to reach out to package maintainers yet.
Quoted 4 lines
This approach means that the version of bpftool and libbpf are _not_ always in sync[1], which I read may causes problem since libbpf and bpftool depends on specific version of each other[2].
Whatever source you use, I would strongly recommend finding a way to keep both in sync. Libbpf has stabilised its API when reaching 1.0, but bpftool taps into some of the internals of the library. Features or new definitions are usually added at the same time to libbpf and bpftool, and if you get a mismatch between the two, you're taking risks to get build issues.
Quoted 7 lines
Using the GitHub mirror of bpftool to package both libbpf and bpftool would kept their version in sync, and was suggested[3]. Although the same could be said if we switch back to packaging libbpf from kernel source, an additional appeal for using GitHub mirrors is that it decouples bpftool from kernel, making it easily upgradable and with a clearer changelog (the latter is quite important for enterprise users) like libbpf.
Happy to read these changelogs I write are useful to someone :). Yes, this is my point.
Quoted 6 lines
The main concern with using GitHub mirror is that bpftool may be updated far beyond the version that comes with the runtime kernel. AFAIK bpftool should work on older kernel since CO-RE is used for built-in BPF iterators and the underlying libbpf work on older kernel itself. Nonetheless, it would be nice to get a confirmation from the maintainers.
As explained above - Mostly, it should work. Otherwise, we can look into fixing it.
Quoted 6 lines
As a side note, I'm open to suggestions/contributions to make life easier for packaging for the mirror. For example, Mahé and I recently added GitHub workflows to ship statically-built binaries for amd64 and arm64 on releases, as well as tarballs with both bpftool+libbpf sources. If there's something else to make packaging easier, I'm happy to talk about it.
Please contribute this for veristat ([0]) and retsnoop ([1]). I've been packaging submodule sources for them using a simple script ([2]), but if this can be automated, that would be awesome. Thanks!
[0] https://github.com/libbpf/veristat/
[1] https://github.com/anakryiko/retsnoop/
[2] https://github.com/anakryiko/retsnoop/blob/master/scripts/archive-srcs-full.sh
Quoted 5 lines
Quoted 3 lines
Are there any other downsides to switching to GitHub mirror for bpftool?
Nothing else comes to mind, but again I'm not packaging for distros.
+1.
Libbpf, bpftool, and other libbpf-dependent tools are meant to be backwards compatible with old kernels as much as possible. Bugs do happen, but those should be reported and fixed. Otherwise if some newer feature is requiring a newer kernel, we always try to develop it in a way that will fail gracefully. So there is no reason to not switch to Github-based mirrors, for both libbpf and bpftool.
Quoted 4 lines
See Toke's message.
I hope this helps,
Quentin