Establishing /usr/lib/bpf as a convention for eBPF bytecode files?

Hi everyone

As you have no doubt noticed, we have started thinking about how to package eBPF-related applications in distributions. As a part of this, I've been thinking about what to recommend for applications that ship pre-compiled BPF byte-code files.

The obvious place to place those would be somewhere in the system $LIBDIR (i.e., /usr/lib or /usr/lib64, depending on the distro). But since BPF byte code is its own binary format, different from regular executables, I think having a separate path to put those under makes sense. So I'm proposing to establish a convention that pre-compiled BPF programs be installed into /usr/lib{,64}/bpf.

This would let users discover which BPF programs are shipped on their system, and it could be used to discover which package loaded a particular BPF program, by walking the directory to find the file a loaded program came from. It would not work for dynamically-generated bytecode, of course, but I think at least some applications will end up shipping pre-compiled bytecode files (we're doing that for xdp-tools, for instance).

As I said, this would be a convention. We're already using it for xdp-tools[0], so my plan is to use that as the "first mover", try to get distributions to establish the path as a part of their filesystem layout, and then just try to encourage packages to use it. Hopefully it will catch on.

Does anyone have any objections to this? Do you think it is a complete waste of time, or is it worth giving it a shot? :)

Quoted 22 lines · Toke Høiland-Jørgensen

Hi everyone

As you have no doubt noticed, we have started thinking about how to package eBPF-related applications in distributions. As a part of this, I've been thinking about what to recommend for applications that ship pre-compiled BPF byte-code files.

The obvious place to place those would be somewhere in the system $LIBDIR (i.e., /usr/lib or /usr/lib64, depending on the distro). But since BPF byte code is its own binary format, different from regular executables, I think having a separate path to put those under makes sense. So I'm proposing to establish a convention that pre-compiled BPF programs be installed into /usr/lib{,64}/bpf.

This would let users discover which BPF programs are shipped on their system, and it could be used to discover which package loaded a particular BPF program, by walking the directory to find the file a loaded program came from. It would not work for dynamically-generated bytecode, of course, but I think at least some applications will end up shipping pre-compiled bytecode files (we're doing that for xdp-tools, for instance).

Quoted 8 lines

As I said, this would be a convention. We're already using it for xdp-tools[0], so my plan is to use that as the "first mover", try to get distributions to establish the path as a part of their filesystem layout, and then just try to encourage packages to use it. Hopefully it will catch on.

Does anyone have any objections to this? Do you think it is a complete waste of time, or is it worth giving it a shot? :)

What will be the name of file/directory ?
What is going to be the mechanism to clean it up?
What will be stored in there? Just .o files ?

libbcc stores original C and rewritten C in /var/tmp/bcc/bpf_prog_TAG/ It was useful for debugging. Since TAG is used as directory name reloading the same bcc script creates the same dir and /var/tmp periodically gets cleaned by reboot.

Installing bpf .o into common location feels useful. Not sure though how you can convince folks to follow such convention.

That was the main problem with libbcc. Not everything is using that lib. So teaching folks who debug bpf in production to look into /var/tmp/bcc wasn't enough. 'bpftool p s' is still the main mechanism. Some C++ services embed bpf .o as part of x86 binary and that binary is installed. They wouldn't want to split bpf .o into separate file since it will complicate dependency management, risk conflicts, etc. Just food for thought.

Quoted 41 lines · On Mon, 9 Dec 2019 17:40:19 -0800 Alexei Starovoitov…

Quoted 22 lines · Toke Høiland-Jørgensen

Hi everyone

As you have no doubt noticed, we have started thinking about how to package eBPF-related applications in distributions. As a part of this, I've been thinking about what to recommend for applications that ship pre-compiled BPF byte-code files.

The obvious place to place those would be somewhere in the system $LIBDIR (i.e., /usr/lib or /usr/lib64, depending on the distro). But since BPF byte code is its own binary format, different from regular executables, I think having a separate path to put those under makes sense. So I'm proposing to establish a convention that pre-compiled BPF programs be installed into /usr/lib{,64}/bpf.

This would let users discover which BPF programs are shipped on their system, and it could be used to discover which package loaded a particular BPF program, by walking the directory to find the file a loaded program came from. It would not work for dynamically-generated bytecode, of course, but I think at least some applications will end up shipping pre-compiled bytecode files (we're doing that for xdp-tools, for instance).

Quoted 8 lines

As I said, this would be a convention. We're already using it for xdp-tools[0], so my plan is to use that as the "first mover", try to get distributions to establish the path as a part of their filesystem layout, and then just try to encourage packages to use it. Hopefully it will catch on.

Does anyone have any objections to this? Do you think it is a complete waste of time, or is it worth giving it a shot? :)

What will be the name of file/directory ?
What is going to be the mechanism to clean it up?
What will be stored in there? Just .o files ?

libbcc stores original C and rewritten C in /var/tmp/bcc/bpf_prog_TAG/ It was useful for debugging. Since TAG is used as directory name reloading the same bcc script creates the same dir and /var/tmp periodically gets cleaned by reboot.

Quoted 2 lines

Installing bpf .o into common location feels useful. Not sure though how you can convince folks to follow such convention.

I imagine the files under /usr/lib{,64}/bpf/ will be pre-compiled binaries (fairly static file). These will be delivered together with the distro RPM file. The distro will detect/enforce that two packages cannot use the same name for bpf .o files.

I see three different types of BPF-object files, which belong in different places (suggestion in parentheses):

1. Pre-compiled binaries via RPM. (/usr/lib? [1])
2. Application "startup" compiled "cached" BPF-object (/var/cache? [2]).
3. Runtime dynamic compiled BPF-objects short lived (/run? [3])

You can follow the links below, to see if match descriptions in the Filesystem Hierarchy Standard[4].

I think that filetype 1 + 2 makes sense to store in files. For filetype 3 (the highly dynamic runtime re-compiled files) I'm not sure it makes sense to store those in any central place. Applications could use /run/application-name/, but it will be a pain to deal with filename-clashes. As Alexei brings up cleanup; /run/ is cleared at the beginning of the boot process[3].

For fileytpe 2, I suggest /var/cache/bpf/, but with an additional application name as a subdir, this is to avoid name clashes (which then becomes the applications responsibility with in its own dir).