BPF updates

This eventually leads us to this patch, which implements a minimal eBPF file system. The idea is a bit similar, but to the point that these inodes reside at one or multiple mount points. A directory hierarchy can be tailored to a specific application use-case from the

various subsystem users and maps/progs pinned inside it. Two new eBPF commands (BPF_PIN_FD, BPF_NEW_FD) have been added to the syscall in order to create one or multiple special inodes from an existing file descriptor that points to a map/program (we call it eBPF fd pinning), or to create a new file descriptor from an existing special inode. BPF_PIN_FD requires CAP_SYS_ADMIN capabilities, whereas BPF_NEW_FD can also be done unpriviledged when having appropriate permissions to the path.

Quoted 4 lines · Daniel Borkmann

This eventually leads us to this patch, which implements a minimal eBPF file system. The idea is a bit similar, but to the point that these inodes reside at one or multiple mount points. A directory hierarchy can be tailored to a specific application use-case from the

Quoted 8 lines

various subsystem users and maps/progs pinned inside it. Two new eBPF commands (BPF_PIN_FD, BPF_NEW_FD) have been added to the syscall in order to create one or multiple special inodes from an existing file descriptor that points to a map/program (we call it eBPF fd pinning), or to create a new file descriptor from an existing special inode. BPF_PIN_FD requires CAP_SYS_ADMIN capabilities, whereas BPF_NEW_FD can also be done unpriviledged when having appropriate permissions to the path.

In my opinion this is very un-unixiy, I have to say at least.

Namespaces at some point dealt with the same problem, they nowadays use bind mounts of /proc/$$/ns/* to some place in the file hierarchy to keep the namespace alive. This at least allows someone to build up its own hierarchy with normal unix tools and not hidden inside a C-program. For filedescriptors we already have /proc/$$/fd/* but it seems that doesn't work out of the box nowadays.

I don't know in terms of how many objects bpf should be able to handle and if such a bind-mount based solution would work, I guess not.

In my opinion I still favor a user space approach. Subsystems which use ebpf in a way that no user space program needs to be running to control them would need to export the fds by itself. E.g. something like sysfs/kobject for tc? The hierarchy would then be in control of the subsystem which could also create a proper naming hierarchy or maybe even use an already given one. Do most other eBPF users really need to persist file descriptors somewhere without user space control and pick them up later?

Quoted 14 lines · Hannes Frederic Sowa

Quoted 4 lines · Daniel Borkmann

This eventually leads us to this patch, which implements a minimal eBPF file system. The idea is a bit similar, but to the point that these inodes reside at one or multiple mount points. A directory hierarchy can be tailored to a specific application use-case from the

Quoted 8 lines

various subsystem users and maps/progs pinned inside it. Two new eBPF commands (BPF_PIN_FD, BPF_NEW_FD) have been added to the syscall in order to create one or multiple special inodes from an existing file descriptor that points to a map/program (we call it eBPF fd pinning), or to create a new file descriptor from an existing special inode. BPF_PIN_FD requires CAP_SYS_ADMIN capabilities, whereas BPF_NEW_FD can also be done unpriviledged when having appropriate permissions to the path.

Quoted 8 lines

In my opinion this is very un-unixiy, I have to say at least.

Namespaces at some point dealt with the same problem, they nowadays use bind mounts of /proc/$$/ns/* to some place in the file hierarchy to keep the namespace alive. This at least allows someone to build up its own hierarchy with normal unix tools and not hidden inside a C-program. For filedescriptors we already have /proc/$$/fd/* but it seems that doesn't work out of the box nowadays.

Yes, that doesn't work out of the box, but I also don't know how usable that would really be. The idea is roughly rather similar to the paths passed to bind(2)/connect(2) on Unix domain sockets, as mentioned. You have a map/prog resource that you stick to a special inode so that you can retrieve it at a later point in time from the same or different processes through a new fd pointing to the resource from user side, so that the bpf(2) syscall can be performed upon it.

With Unix tools, you could still create/remove a hierarchy or unlink those that have maps/progs. You are correct that tools that don't implement bpf(2) currently cannot access the content behind it, since bpf(2) manages access to the data itself. I did like the 2nd idea though, mentioned in the commit log, but don't know how flexible we are in terms of adding S_IFBPF to the UAPI.

Quoted 11 lines

I don't know in terms of how many objects bpf should be able to handle and if such a bind-mount based solution would work, I guess not.

In my opinion I still favor a user space approach. Subsystems which use ebpf in a way that no user space program needs to be running to control them would need to export the fds by itself. E.g. something like sysfs/kobject for tc? The hierarchy would then be in control of the subsystem which could also create a proper naming hierarchy or maybe even use an already given one. Do most other eBPF users really need to persist file descriptors somewhere without user space control and pick them up later?

I was thinking about a strict predefined hierarchy dictated by the kernel as well, but was then considering a more flexible approach that could be tailored freely to various use cases. A predefined hierarchy would most likely need to be resolved per subsystem and it's not really easy to map this properly. F.e. if the kernel would try to provide unique ids (as opposed to have a name or annotation member through the syscall), it could end up being quite cryptic. If we let the users choose names, I'm not sure if a single hierarchy level would be enough. Then, additionally you have facilities like tail calls that eBPF programs could do.

In such cases, one could even craft relationships where a (strict auto generated) tree representation would not be sufficient (f.e. recirculation
up to a certain depth). The tail called programs could be changed atomically during runtime, etc. The other issue related to a per subsystem
representation is that bpf(2) is the central management interface for

creating/accessing maps/progs, and each subsystem then has its own little interface to "install" them internally (f.e. via netlink, setsockopt(2), etc). That means, with tail calls, only the 'root' programs are installed there and further transactions would be needed in order to make individual
subsystems aware, so they could potentially generate some hierarchy; don't
know, it seems rather complex.

Quoted 32 lines · Daniel Borkmann

Quoted 14 lines · Hannes Frederic Sowa

Quoted 4 lines · Daniel Borkmann

This eventually leads us to this patch, which implements a minimal eBPF file system. The idea is a bit similar, but to the point that these inodes reside at one or multiple mount points. A directory hierarchy can be tailored to a specific application use-case from the

Quoted 8 lines

various subsystem users and maps/progs pinned inside it. Two new eBPF commands (BPF_PIN_FD, BPF_NEW_FD) have been added to the syscall in order to create one or multiple special inodes from an existing file descriptor that points to a map/program (we call it eBPF fd pinning), or to create a new file descriptor from an existing special inode. BPF_PIN_FD requires CAP_SYS_ADMIN capabilities, whereas BPF_NEW_FD can also be done unpriviledged when having appropriate permissions to the path.

Quoted 8 lines

In my opinion this is very un-unixiy, I have to say at least.

Namespaces at some point dealt with the same problem, they nowadays use bind mounts of /proc/$$/ns/* to some place in the file hierarchy to keep the namespace alive. This at least allows someone to build up its own hierarchy with normal unix tools and not hidden inside a C-program. For filedescriptors we already have /proc/$$/fd/* but it seems that doesn't work out of the box nowadays.

Yes, that doesn't work out of the box, but I also don't know how usable that would really be. The idea is roughly rather similar to the paths passed to bind(2)/connect(2) on Unix domain sockets, as mentioned. You have a map/prog resource that you stick to a special inode so that you can retrieve it at a later point in time from the same or different processes through a new fd pointing to the resource from user side, so that the bpf(2) syscall can be performed upon it.

Quoted 6 lines

With Unix tools, you could still create/remove a hierarchy or unlink those that have maps/progs. You are correct that tools that don't implement bpf(2) currently cannot access the content behind it, since bpf(2) manages access to the data itself. I did like the 2nd idea though, mentioned in the commit log, but don't know how flexible we are in terms of adding S_IFBPF to the UAPI.

I don't think it should be a problem. You referred to POSIX Standard in your other mail but I can't see any reason why not to establish a new file mode. Anyway, FreeBSD (e.g. whiteouts) and Solaris (e.g. Doors, Event Ports) are just examples of new modes being added.

mknod /bpf/map/1 m 1 1

:)

Yes, maybe I think this is a better solution architectural instead of constructing a new filesystem.

Quoted 12 lines

Quoted 11 lines

I don't know in terms of how many objects bpf should be able to handle and if such a bind-mount based solution would work, I guess not.

In my opinion I still favor a user space approach. Subsystems which use ebpf in a way that no user space program needs to be running to control them would need to export the fds by itself. E.g. something like sysfs/kobject for tc? The hierarchy would then be in control of the subsystem which could also create a proper naming hierarchy or maybe even use an already given one. Do most other eBPF users really need to persist file descriptors somewhere without user space control and pick them up later?

Quoted 9 lines

I was thinking about a strict predefined hierarchy dictated by the kernel as well, but was then considering a more flexible approach that could be tailored freely to various use cases. A predefined hierarchy would most likely need to be resolved per subsystem and it's not really easy to map this properly. F.e. if the kernel would try to provide unique ids (as opposed to have a name or annotation member through the syscall), it could end up being quite cryptic. If we let the users choose names, I'm not sure if a single hierarchy level would be enough. Then, additionally you have facilities like tail calls that eBPF programs could do.

I don't think that most subsystems need to expose those file descriptors. Seccomp probably will have a supervisor process running and per aggregation will also have a user space process running keeping the fd alive. So it is all about tc/sched.

And I am not sure if tc will really needs a filesystem to handle all this. The simplest approach is to just keep a name <-> fd mapping somewhere in the net/sched/ subsystem and use this for all tc users.

I see that tail calls makes this all very difficult to show which entity uses which ebpf entity in some way, as it looks like n:m relationships.

Quoted 7 lines

In such cases, one could even craft relationships where a (strict auto generated) tree representation would not be sufficient (f.e. recirculation
up to a certain depth). The tail called programs could be changed atomically during runtime, etc. The other issue related to a per subsystem
representation is that bpf(2) is the central management interface for

Quoted 8 lines

creating/accessing maps/progs, and each subsystem then has its own little interface to "install" them internally (f.e. via netlink, setsockopt(2), etc). That means, with tail calls, only the 'root' programs are installed there and further transactions would be needed in order to make individual
subsystems aware, so they could potentially generate some hierarchy; don't
know, it seems rather complex.

I understand, this is really not suitable to represent in its entirety in sysfs or any kind of hierarchical structure right now. Either we limit it somewhat (Alexei will certainly intervene here) or one of your filesystem approaches will win.

But I still wonder why people are so against user space dependencies?

Another idea that I discussed with Daniel just to have it publicly
available: a userspace helper would be called for every ebpf entity change so it could mirror or keep track ebpf handles in user space. You can think along the lines of kernel/core_pattern. This would probably also depend on non-anon-inode usage of ebpf fds.

Quoted 12 lines

Quoted 2 lines

but don't know how flexible we are in
terms of adding S_IFBPF to the UAPI.

I don't think it should be a problem. You referred to POSIX Standard in your other mail but I can't see any reason why not to establish a new file mode. Anyway, FreeBSD (e.g. whiteouts) and Solaris (e.g. Doors, Event Ports) are just examples of new modes being added.

mknod /bpf/map/1 m 1 1

:)

Quoted 2 lines

Yes, maybe I think this is a better solution architectural instead of constructing a new filesystem.

Yeah, also 'man 2 stat' lists a couple of others used by various systems.

The pro's of this approach would be that no new file system would be needed and the special inode could be placed on top of any 'regular' file system that would support special files. I do like that as well.

I'm wondering whether this would prevent us in future from opening access to shell tools etc on that special file, but probably one could provide a default set of file ops via init_special_inode() that could be overloaded by the underlying fs if required.

Namespaces at some point dealt with the same problem, they nowadays use bind mounts of/proc/$$/ns/* to some place in the file hierarchy to keep the namespace alive. This at least allows someone to build up its own hierarchy with normal unix tools and not hidden inside a C-program. For filedescriptors we already have/proc/$$/fd/* but it seems that doesn't work out of the box nowadays.

I don't know in terms of how many objects bpf should be able to handle and if such a bind-mount based solution would work, I guess not.

In my opinion I still favor a user space approach.

Subsystems which use
ebpf in a way that no user space program needs to be running to control them would need to export the fds by itself. E.g. something like sysfs/kobject for tc? The hierarchy would then be in control of the subsystem which could also create a proper naming hierarchy or maybe even use an already given one. Do most other eBPF users really need to persist file descriptors somewhere without user space control and pick them up later?

Quoted 6 lines · Hannes Frederic Sowa

Namespaces at some point dealt with the same problem, they nowadays use bind mounts of/proc/$$/ns/* to some place in the file hierarchy to keep the namespace alive. This at least allows someone to build up its own hierarchy with normal unix tools and not hidden inside a C-program. For filedescriptors we already have/proc/$$/fd/* but it seems that doesn't work out of the box nowadays.

bind mounting of /proc/../fd was initially proposed by Andy and we've looked at it thoroughly, but after discussion with Eric it became apparent that it doesn't fit here. At the end we need shell tools to access maps.

Also I think you missed the hierarchy in this patch set _is_ built with normal 'mkdir' and files are removed with 'rm'.

The only thing that C does is BPF_PIN_FD of fd that was received from bpf syscall. That's way cleaner api than doing bind mount from C program.

We've considered letting open() of the file return bpf specific anon-inode, but decided to reserve that for other more natural file operations. Therefore BPF_NEW_FD is needed.

Quoted 2 lines

I don't know in terms of how many objects bpf should be able to handle and if such a bind-mount based solution would work, I guess not.

We definitely missed you at the last plumbers where it was discussed :)

Quoted 1 line

In my opinion I still favor a user space approach.

that's not acceptable for tracing use cases. No daemons allowed.

Quoted 8 lines

Subsystems which use
ebpf in a way that no user space program needs to be running to control them would need to export the fds by itself. E.g. something like sysfs/kobject for tc? The hierarchy would then be in control of the subsystem which could also create a proper naming hierarchy or maybe even use an already given one. Do most other eBPF users really need to persist file descriptors somewhere without user space control and pick them up later?

I think it's way cleaner to have one way of solving it (like this patch does) instead of asking every subsystem to solve it differently. We've also looked at sysfs and it's ugly when it comes to removing, since the user cannot use normal 'rm'.

Hi Alexei,

Quoted 8 lines · Alexei Starovoitov

Quoted 6 lines · Hannes Frederic Sowa

Namespaces at some point dealt with the same problem, they nowadays use bind mounts of/proc/$$/ns/* to some place in the file hierarchy to keep the namespace alive. This at least allows someone to build up its own hierarchy with normal unix tools and not hidden inside a C-program. For filedescriptors we already have/proc/$$/fd/* but it seems that doesn't work out of the box nowadays.

Quoted 4 lines

bind mounting of /proc/../fd was initially proposed by Andy and we've looked at it thoroughly, but after discussion with Eric it became apparent that it doesn't fit here. At the end we need shell tools to access maps.

Oh yes, I want shell tools for this very much! Maybe even that things like strings, grep etc. work. :)

Quoted 2 lines

Also I think you missed the hierarchy in this patch set _is_ built with normal 'mkdir' and files are removed with 'rm'.

I did not miss that, I am just concerned that if the kernel does not enforce such a hierarchy automatically it won't really happen.

Oh, tracing does not allow daemons. Why? I can only imagine embedded users, no?

Quoted 2 lines · Hannes Frederic Sowa

Oh, tracing does not allow daemons. Why? I can only imagine embedded users, no?

yes and for networking: restartability and HA.
cannot really do that with fuse/daemons.

This eventually leads us to this patch, which implements a minimal eBPF file system. The idea is a bit similar, but to the point that these inodes reside at one or multiple mount points. A directory hierarchy can be tailored to a specific application use-case from the

various subsystem users and maps/progs pinned inside it. Two new eBPF commands (BPF_PIN_FD, BPF_NEW_FD) have been added to the syscall in order to create one or multiple special inodes from an existing file descriptor that points to a map/program (we call it eBPF fd pinning), or to create a new file descriptor from an existing special inode. BPF_PIN_FD requires CAP_SYS_ADMIN capabilities, whereas BPF_NEW_FD can also be done unpriviledged when having appropriate permissions to the path.

Another question:
Should multiple mount of the filesystem result in an empty fs (a new instance) or in one were one can see other ebpf-fs entities? I think Daniel wanted to already use the mountpoint as some kind of hierarchy delimiter. I would have used directories for that and multiple mounts would then have resulted in the same content of the filesystem. IMHO this would remove some ambiguity but then the question arises how this is handled in a namespaced environment. Was there some specific reason to do so?

Quoted 9 lines · Hannes Frederic Sowa

Another question:
Should multiple mount of the filesystem result in an empty fs (a new instance) or in one were one can see other ebpf-fs entities? I think Daniel wanted to already use the mountpoint as some kind of hierarchy delimiter. I would have used directories for that and multiple mounts would then have resulted in the same content of the filesystem. IMHO this would remove some ambiguity but then the question arises how this is handled in a namespaced environment. Was there some specific reason to do so?

That's an interesting question!
I think all mounts should be independent.
I can see tracing using one and networking using another one with different hierarchies suitable for their own use cases. What's an advantage to have the same content everywhere? Feels harder to manage, since different users would need to coordinate.

Quoted 11 lines · Alexei Starovoitov

Quoted 9 lines · Hannes Frederic Sowa

Another question:
Should multiple mount of the filesystem result in an empty fs (a new instance) or in one were one can see other ebpf-fs entities? I think Daniel wanted to already use the mountpoint as some kind of hierarchy delimiter. I would have used directories for that and multiple mounts would then have resulted in the same content of the filesystem. IMHO this would remove some ambiguity but then the question arises how this is handled in a namespaced environment. Was there some specific reason to do so?

Quoted 7 lines

That's an interesting question!
I think all mounts should be independent.
I can see tracing using one and networking using another one with different hierarchies suitable for their own use cases. What's an advantage to have the same content everywhere? Feels harder to manage, since different users would need to coordinate.

I initially had it as a mount_single() file system, where I was thinking to have an entry under /sys/fs/bpf/, so all subsystems would work on top of that mount point, but for the same reasons above I lifted that restriction.

Quoted 20 lines · Daniel Borkmann

Quoted 11 lines · Alexei Starovoitov

Quoted 9 lines · Hannes Frederic Sowa

Another question:
Should multiple mount of the filesystem result in an empty fs (a new instance) or in one were one can see other ebpf-fs entities? I think Daniel wanted to already use the mountpoint as some kind of hierarchy delimiter. I would have used directories for that and multiple mounts would then have resulted in the same content of the filesystem. IMHO this would remove some ambiguity but then the question arises how this is handled in a namespaced environment. Was there some specific reason to do so?

Quoted 7 lines

That's an interesting question!
I think all mounts should be independent.
I can see tracing using one and networking using another one with different hierarchies suitable for their own use cases. What's an advantage to have the same content everywhere? Feels harder to manage, since different users would need to coordinate.

Quoted 3 lines

I initially had it as a mount_single() file system, where I was thinking to have an entry under /sys/fs/bpf/, so all subsystems would work on top of that mount point, but for the same reasons above I lifted that restriction.

I am missing something.

When I suggested using a filesystem it was my thought there would be exactly one superblock per map, and the map would be specified at mount time. You clearly are not implementing that.

A filesystem per map makes sense as you have a key-value store with one file per key.

The idea is that something resembling your bpf_pin_fd function would be the mount system call for the filesystem.

The the keys in the map could be read by "ls /mountpoint/". Key values could be inspected with "cat /mountpoint/key".

On 10/16/15 11:41 AM, Eric W. Biederman wrote:

Quoted 5 lines

I am missing something.

When I suggested using a filesystem it was my thought there would be exactly one superblock per map, and the map would be specified at mount time. You clearly are not implementing that.

I don't think it's practical to have sb per map, since that would mean sb per prog and that won't scale.

Also map today is an fd that belongs to a process. I cannot see an api from C program to do 'mount of FD' that wouldn't look like ugly hack.

Quoted 8 lines

A filesystem per map makes sense as you have a key-value store with one file per key.

The idea is that something resembling your bpf_pin_fd function would be the mount system call for the filesystem.

The the keys in the map could be read by "ls /mountpoint/". Key values could be inspected with "cat /mountpoint/key".

yes. that is still the goal for follow up patches, but contained within given bpffs. Something bpf_pin_fd-like command for bpf syscall would create files for keys in a map and allow 'cat' via open/read. Such api would be much cleaner from C app point of view. Potentially we can allow mount of a file created via BPF_PIN_FD that will expand into keys/values.
All of that are our future plans.
There, actually, the main contention point is 'how to represent keys and values'. whether key is hex representation or we need some pretty-printers via format string or via schema? etc, etc. We tried few ideas of representing keys in our fuse implementations, but don't have an agreement yet.

Quoted 1 line · Alexei Starovoitov

On 10/16/15 11:41 AM, Eric W. Biederman wrote:

[...]

Quoted 8 lines

Quoted 5 lines

I am missing something.

When I suggested using a filesystem it was my thought there would be exactly one superblock per map, and the map would be specified at mount time. You clearly are not implementing that.

I don't think it's practical to have sb per map, since that would mean sb per prog and that won't scale.

What do you mean won't scale? You want to have a name per map/prog so the basic complexity appears the same. Is there some crucial interaction between the persistent dodads you are placing on a filesystem that I am missing?

Given the fact you don't normally need any persistence without a program I am puzzled why "scaling" is an issue of any kind. This is for a comparitively rare case if I am not mistaken.

Quoted 3 lines

Also map today is an fd that belongs to a process. I cannot see an api from C program to do 'mount of FD' that wouldn't look like ugly hack.

mount -t bpffs ... -o fd=1234

That is not all convoluted or hacky. Especially compared to some of the alternatives I am seeing.

It is no problem at all to wrap something like that in a nice function call that has the exact same complexity of use as any of the other options that are being explored to give something that starts out as a filedescriptor a name.

Quoted 9 lines

Quoted 8 lines

A filesystem per map makes sense as you have a key-value store with one file per key.

The idea is that something resembling your bpf_pin_fd function would be the mount system call for the filesystem.

The the keys in the map could be read by "ls /mountpoint/". Key values could be inspected with "cat /mountpoint/key".

Quoted 12 lines

yes. that is still the goal for follow up patches, but contained within given bpffs. Something bpf_pin_fd-like command for bpf syscall would create files for keys in a map and allow 'cat' via open/read. Such api would be much cleaner from C app point of view. Potentially we can allow mount of a file created via BPF_PIN_FD that will expand into keys/values.
All of that are our future plans.
There, actually, the main contention point is 'how to represent keys and values'. whether key is hex representation or we need some pretty-printers via format string or via schema? etc, etc. We tried few ideas of representing keys in our fuse implementations, but don't have an agreement yet.

My gut feel would be to keep it simple and use the same representation you use in your existing system calls. Certainly ordinary filenames are keys of arbitrary binary data that can included everything except a '\0' byte. That they are human readable is a nice convention, but not at all fundamental to what they are.

We can argue about api for 2nd, whether it's mount with fd=1234 string or else, but for the first mount style doesn't make sense.

Quoted 2 lines · Alexei Starovoitov

We can argue about api for 2nd, whether it's mount with fd=1234 string or else, but for the first mount style doesn't make sense.

Why does mount not make sense? It is exactly what you are looking for so why does it not make sense?

Quoted 5 lines · Eric W. Biederman

Quoted 2 lines · Alexei Starovoitov

We can argue about api for 2nd, whether it's mount with fd=1234 string or else, but for the first mount style doesn't make sense.

Quoted 2 lines

Why does mount not make sense? It is exactly what you are looking for so why does it not make sense?

hmm, how do you get a new fd back after mounting it?
Note, open cannot be overloaded, so we end up with BPF_NEW_FD anyway, but now it's more convoluted and empty mounts are everywhere.

Anyway, another idea I've been brainstorming with Hannes today a bit is about the following:

We register two major numbers, one for eBPF maps (X), one for eBPF progs (Y). A user can either via cmdline call something like ... mknod /dev/bpf/maps/map_pkts c X Z to create a special character device, or alternatively out of an application through mknod(2) syscall (f.e. tc when setting up maps/progs internally from the obj file for a classifer).

Then, we still have 2 eBPF commands for bpf(2) syscall to add, say (for example) BPF_BIND_DEV and BPF_FETCH_DEV. The application that created a map (or prog) already has the map fd and after mknod(2) it can open(2) the special file to get the special file fd. Then it can call something like bpf(BPF_BIND_DEV, &attr, sizeof(attr))) where attr looks like:

   union bpf_attr attr = {
     .bpf_fd    = bpf_fd,
     .dev_fd    = dev_fd,
   };

The bpf(2) syscall can check whether dev_fd belongs to an eBPF special file and it can then copy over file->private_data from the bpf_fd to the dev_fd's underlying file, where the private_data, as we know, from the bpf_fd already points to a proper bpf_map/bpf_prog structure. The map/prog would then get ref'ed and lives onwards in the char device's lifetime. No special hashtable, gc, etc needed. The char device has fops that we can define by ourself, and unlinking would drop the ref from its private_data.

Now to the other part: BPF_FETCH_DEV would work similar. The application opens the device, and fills bpf_attr as follows again:

   union bpf_attr attr = {
     .bpf_fd    = 0,
     .dev_fd    = dev_fd,
   };

This would allow us to look up the map/prog from the dev_fd's file-> private_data, and installs a new fd via bpf_{map,prog}_new_fd() that is returned from bpf(2) for bpf-related access. The remaining fops from the char device could still be reserved for possibilities like debugging in future.

Now in future (2nd step), could either be to use Eric's idea and then do something like mount -t bpffs ... -o /dev/bpf/maps/map_pkts to dump attributes or other properties to some location for inspection from such a special file, or we could use kobjects for that attached to the device if the fops from the cdev should not be sufficient.

So closing the loop to the special files where there were concerns:

This won't forbid to have a future shell-style access possibility, and it would also not end up as a nightmare on what you mentioned with the S_ISSOCK-like bit in the other email.

The pinning mechanism would not require an extra file system to be mounted somewhere, and yet the user can define himself an arbitrary hierarchy where he puts the special files as this facility already exists. An approach like this looks overall cleaner to me, and most likely be realizable in fewer lines of code as well.

Thoughts?

that indeed sounds cleaner, less lines of code, no fs, etc, but I don't see how it will work yet.

On 10/18/2015 04:20 AM, Alexei Starovoitov wrote:
...

Quoted 2 lines

that indeed sounds cleaner, less lines of code, no fs, etc, but I don't see how it will work yet.

I'll have some code ready very soon to show the concept. Will post it here tonight, stay tuned. ;)

Okay, I have pushed some rough working proof of concept here:

https://git.breakpoint.cc/cgit/dborkman/net-next.git/log/?h=ebpf-fds-final5

So the idea eventually had to be slightly modified after giving this further
thoughts and is the following:

We have 3 commands (BPF_DEV_CREATE, BPF_DEV_DESTROY, BPF_DEV_CONNECT), and related to that a bpf_attr extension with only a single __u32 fd member in it.

The nice thing about it is that you can create/unlink as many as you want, but
when you remove the real device from an application via bpf_dev_destroy(fd),
then all links disappear with it. Just like in the case of a normal device driver.

On device creation, the kernel will return the minor number via bpf(2), so you
can access the file easily, f.e. /dev/bpf/bpf_map<minor> resp. /dev/bpf/bpf_prog<minor>,
and then move on with mknod(2) or symlink(2) from there if wished.

Quoted 11 lines · Daniel Borkmann

Okay, I have pushed some rough working proof of concept here:

https://git.breakpoint.cc/cgit/dborkman/net-next.git/log/?h=ebpf-fds-final5

So the idea eventually had to be slightly modified after giving this further
thoughts and is the following:

We have 3 commands (BPF_DEV_CREATE, BPF_DEV_DESTROY, BPF_DEV_CONNECT), and related to that a bpf_attr extension with only a single __u32 fd member in it.

...

Quoted 6 lines

The nice thing about it is that you can create/unlink as many as you want, but
when you remove the real device from an application via bpf_dev_destroy(fd),
then all links disappear with it. Just like in the case of a normal device driver.

interesting idea!
What happens if user app creates a dev via bpf_dev_create(), exits and then admin does rm of that dev ?
Looks like map/prog will leak ?
So the only proper way to delete such cdevs is via bpf_dev_destroy ?

Quoted 6 lines

On device creation, the kernel will return the minor number via bpf(2), so you
can access the file easily, f.e. /dev/bpf/bpf_map<minor> resp. /dev/bpf/bpf_prog<minor>,
and then move on with mknod(2) or symlink(2) from there if wished.

what if admin mknod in that dir with some arbitrary minor ?

mknod will succeed, but it won't hold anything?

looks like bpf_dev_connect will handle it gracefully.
So these cdevs should only be created and destroyed via bpf syscall and only sensible operations on them is open() to get fd and pass to bpf_dev_connect and symlink. Anything else admin should be careful not to do. Right?

Hi,

Quoted 20 lines · Alexei Starovoitov

Quoted 11 lines · Daniel Borkmann

Okay, I have pushed some rough working proof of concept here:

https://git.breakpoint.cc/cgit/dborkman/net-next.git/log/?h=ebpf-fds-final5

So the idea eventually had to be slightly modified after giving this further
thoughts and is the following:

We have 3 commands (BPF_DEV_CREATE, BPF_DEV_DESTROY, BPF_DEV_CONNECT), and related to that a bpf_attr extension with only a single __u32 fd member in it.

...

Quoted 6 lines

The nice thing about it is that you can create/unlink as many as you want, but
when you remove the real device from an application via bpf_dev_destroy(fd),
then all links disappear with it. Just like in the case of a normal device driver.

Quoted 5 lines

interesting idea!
What happens if user app creates a dev via bpf_dev_create(), exits and then admin does rm of that dev ?
Looks like map/prog will leak ?
So the only proper way to delete such cdevs is via bpf_dev_destroy ?

The mknod is not the holder but rather the kobject which should be represented in sysfs will be. So you can still get the map major:minor by looking up the /dev file in the correspdonding sysfs directory or I think we should provide a 'unbind' file, which will drop the kobject if the user writes a '1' to it.

Quoted 7 lines

Quoted 5 lines

On device creation, the kernel will return the minor number via bpf(2), so you
can access the file easily, f.e. /dev/bpf/bpf_map<minor> resp. /dev/bpf/bpf_prog<minor>,
and then move on with mknod(2) or symlink(2) from there if wished.

what if admin mknod in that dir with some arbitrary minor ?

Basically, -EIO. :)

Quoted 1 line

mknod will succeed, but it won't hold anything?

That is right now true for basically all mknod operations, which udev creates.

Quoted 5 lines

looks like bpf_dev_connect will handle it gracefully.
So these cdevs should only be created and destroyed via bpf syscall and only sensible operations on them is open() to get fd and pass to bpf_dev_connect and symlink. Anything else admin should be careful not to do. Right?

Besides maybe some statistics and other stuff in sysfs directory, no, that is all.

Bye,
Hannes

Quoted 27 lines · Hannes Frederic Sowa

Quoted 25 lines · Alexei Starovoitov

Quoted 11 lines · Daniel Borkmann

Okay, I have pushed some rough working proof of concept here:

https://git.breakpoint.cc/cgit/dborkman/net-next.git/log/?h=ebpf-fds-final5

So the idea eventually had to be slightly modified after giving this further
thoughts and is the following:

We have 3 commands (BPF_DEV_CREATE, BPF_DEV_DESTROY, BPF_DEV_CONNECT), and related to that a bpf_attr extension with only a single __u32 fd member in it.

...

Quoted 6 lines

The nice thing about it is that you can create/unlink as many as you want, but
when you remove the real device from an application via bpf_dev_destroy(fd),
then all links disappear with it. Just like in the case of a normal device driver.

interesting idea!
What happens if user app creates a dev via bpf_dev_create(), exits and then admin does rm of that dev ?
Looks like map/prog will leak ?
So the only proper way to delete such cdevs is via bpf_dev_destroy ?

Quoted 5 lines

The mknod is not the holder but rather the kobject which should be represented in sysfs will be. So you can still get the map major:minor by looking up the /dev file in the correspdonding sysfs directory or I think we should provide a 'unbind' file, which will drop the kobject if the user writes a '1' to it.

I agree, this could still be done.

Quoted 9 lines

Quoted 7 lines

Quoted 5 lines

On device creation, the kernel will return the minor number via bpf(2), so you
can access the file easily, f.e. /dev/bpf/bpf_map<minor> resp. /dev/bpf/bpf_prog<minor>,
and then move on with mknod(2) or symlink(2) from there if wished.

what if admin mknod in that dir with some arbitrary minor ?

Basically, -EIO. :)

Quoted 16 lines

Quoted 1 line

mknod will succeed, but it won't hold anything?

That is right now true for basically all mknod operations, which udev creates.

Quoted 5 lines

looks like bpf_dev_connect will handle it gracefully.
So these cdevs should only be created and destroyed via bpf syscall and only sensible operations on them is open() to get fd and pass to bpf_dev_connect and symlink. Anything else admin should be careful not to do. Right?

Besides maybe some statistics and other stuff in sysfs directory, no, that is all.

Bye,
Hannes

Quoted 7 lines

Quoted 5 lines

The mknod is not the holder but rather the kobject which should be represented in sysfs will be. So you can still get the map major:minor by looking up the /dev file in the correspdonding sysfs directory or I think we should provide a 'unbind' file, which will drop the kobject if the user writes a '1' to it.

I agree, this could still be done.

As Hannes said, under /sys/class/bpf/ an admin can see all held nodes, so visibility is there for free at all times. The device management (creation/ deletion) itself and the mknod's pointing to it are simply decoupled.

This whole approach looks sound to me, also integrates nicely into the existing Linux facilities, and works on top of every fs supporting special files. Much cleaner than an extra file-system that would be required by a syscall in order to make the syscall work.

Quoted 7 lines · Daniel Borkmann

Quoted 7 lines

Quoted 5 lines

The mknod is not the holder but rather the kobject which should be represented in sysfs will be. So you can still get the map major:minor by looking up the /dev file in the correspdonding sysfs directory or I think we should provide a 'unbind' file, which will drop the kobject if the user writes a '1' to it.

I agree, this could still be done.

imo doing 'rm' is way cleaner then dealing with 'unbind' file.

Quoted 8 lines

As Hannes said, under /sys/class/bpf/ an admin can see all held nodes, so visibility is there for free at all times. The device management (creation/ deletion) itself and the mknod's pointing to it are simply decoupled.

This whole approach looks sound to me, also integrates nicely into the existing Linux facilities, and works on top of every fs supporting special files. Much cleaner than an extra file-system that would be required by a syscall in order to make the syscall work.

thanks for the explanations. I think I got a complete picture now on how such cdev will be used and I don't like it.
There is nothing in linux or any unix that creates thousands of cdevs on the fly, but here user apps will create/destroy them back and forth and they would need to do it quickly. Whole sysfs/kobj baggage is

completely unnecessary here. The kernel will consume more memory for no real reason other than cdev are used to keep prog/maps around.

imo fs is cleaner and we can tailor it to be similar to cdev style.

An eBPF program or map loading/destruction is *not* by any means to be considered fast-path. We currently hold a global mutex during loading. So, how can that be considered fast-path? Similarly, socket creation/ destruction is also not fast-path, etc. Do you expect that applications would create/destroy these devices within milliseconds? I'd argue that something would be seriously wrong with that application, then. Such persistent maps are to be considered rather mid-long living objects in the system. The fast-path surely is the data-path of them.

Quoted 2 lines

completely unnecessary here. The kernel will consume more memory for no real reason other than cdev are used to keep prog/maps around.

I don't consider this a big issue, and well worth the trade-off. You'll have an infrastructure that integrates *nicely* into the *existing* kernel model *and* tooling with the proposed patch. This is a HUGE plus. The UAPI of this is simple and minimal. And to me, these are in-fact special files, not regular ones.

Quoted 1 line

imo fs is cleaner and we can tailor it to be similar to cdev style.

Really, IMHO I think this is over-designed, and much much more hacky. We design a whole new file system that works *exactly* like cdevs, takes likely more than twice the code and complexity to realize but just to save a few bytes ...? I don't understand that.

Quoted 8 lines · Daniel Borkmann

An eBPF program or map loading/destruction is *not* by any means to be considered fast-path. We currently hold a global mutex during loading. So, how can that be considered fast-path? Similarly, socket creation/ destruction is also not fast-path, etc. Do you expect that applications would create/destroy these devices within milliseconds? I'd argue that something would be seriously wrong with that application, then. Such persistent maps are to be considered rather mid-long living objects in the system. The fast-path surely is the data-path of them.

consider seccomp that loads several programs for every tab, then container use case where we're loading two for each as well. Who knows what use cases will come up in the future.
It's obviously not a fast path that is being hit million times a second, but why add overhead when it's unnecessary?

Quoted 8 lines

Quoted 2 lines

completely unnecessary here. The kernel will consume more memory for no real reason other than cdev are used to keep prog/maps around.

I don't consider this a big issue, and well worth the trade-off. You'll have an infrastructure that integrates *nicely* into the *existing* kernel model *and* tooling with the proposed patch. This is a HUGE plus. The UAPI of this is simple and minimal. And to me, these are in-fact special files, not regular ones.

Seriously? Syscall to create/destory cdevs is a nice api?

Not by any means. We can argue in circles, but it doesn't fit. Using cdev to hold maps is a hack.
Telling kernel that fake device is created only to hold a map? This fake device doesn't have any of the device properties. Look at fops->open. Surely it's a smart trick, but it's not behaving like device.

uapi for fs adds only two commands, whereas cdev needs three.

Quoted 6 lines

Quoted 1 line

imo fs is cleaner and we can tailor it to be similar to cdev style.

Really, IMHO I think this is over-designed, and much much more hacky. We design a whole new file system that works *exactly* like cdevs, takes likely more than twice the code and complexity to realize but just to save a few bytes ...? I don't understand that.

Let's argue with facts. Your fs patch 758 lines vs cdev 483. In fs the cost is single alloc_inode(), whereas in cdev the struct cdev and mutex will add to all maps and progs (even when they're not going to be pinned), plus kobj allocation, plus gigantic struct device, plus sysfs inodes, etc. Way more than few bytes of difference.

where do you see 'over-design' in fs? It's a straightforward code and most of it is boilerplate like all other fs.
Kill rmdir/mkdir ops, term bit and options and the fs diff will shrink by another 200+ lines.

Hi,

Quoted 10 lines · Alexei Starovoitov

Quoted 8 lines · Daniel Borkmann

An eBPF program or map loading/destruction is *not* by any means to be considered fast-path. We currently hold a global mutex during loading. So, how can that be considered fast-path? Similarly, socket creation/ destruction is also not fast-path, etc. Do you expect that applications would create/destroy these devices within milliseconds? I'd argue that something would be seriously wrong with that application, then. Such persistent maps are to be considered rather mid-long living objects in the system. The fast-path surely is the data-path of them.

Quoted 5 lines

consider seccomp that loads several programs for every tab, then container use case where we're loading two for each as well. Who knows what use cases will come up in the future.
It's obviously not a fast path that is being hit million times a second, but why add overhead when it's unnecessary?

But the browser using seccomp does not need persistent maps at all. It

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

completely unnecessary here. The kernel will consume more memory for no real reason other than cdev are used to keep prog/maps around.

I don't consider this a big issue, and well worth the trade-off. You'll have an infrastructure that integrates *nicely* into the *existing* kernel model *and* tooling with the proposed patch. This is a HUGE plus. The UAPI of this is simple and minimal. And to me, these are in-fact special files, not regular ones.

Quoted 1 line

Seriously? Syscall to create/destory cdevs is a nice api?

It is pretty normal to do. They don't use syscall but ioctl on a special pseudo device node (lvm2/device-mapper, tun, etc.). Same thing, very overloaded syscall. Also I don't see a reason to not make the creation possible via sysfs, which would be even nicer but not orthogonal to the current creation of maps, which is nowadays set in stone by uapi.

sysfs already has infrastructure to ensure supportability and debugability. Dependency graphs can be created to see which programs depend on which bpf_progs. kobjects are singeltons in sysfs representation. If you have multiple ebpf filesystems, even maybe referencing the same hashtable with gigabytes of data multiple times, there needs to be some way to help administrators to check resource usage, statistics, tweak and tune the rhashtable. All this needs to be handled as well in the future. It doesn't really fit the filesystem model, but representing a kobject seems to be a good fit to me.

Policy can be done by user space with help of udev. Selinux policies can easily being extended to allow specific domains access. Namespace "passthrough" is defined for devices already.

Quoted 4 lines

where do you see 'over-design' in fs? It's a straightforward code and most of it is boilerplate like all other fs.
Kill rmdir/mkdir ops, term bit and options and the fs diff will shrink by another 200+ lines.

I don't think that only a filesystem will do it in the foreseeable future. You want to have tools like lsof reporting which map and which program has references to each other. Those file nodes will need more metadata attached in future anyway, so currently just comparing lines of codes seems not to be a good way for arguing.

Quoted 19 lines · Hannes Frederic Sowa

Hi,

Quoted 15 lines · Alexei Starovoitov

Quoted 8 lines · Daniel Borkmann

An eBPF program or map loading/destruction is *not* by any means to be considered fast-path. We currently hold a global mutex during loading. So, how can that be considered fast-path? Similarly, socket creation/ destruction is also not fast-path, etc. Do you expect that applications would create/destroy these devices within milliseconds? I'd argue that something would be seriously wrong with that application, then. Such persistent maps are to be considered rather mid-long living objects in the system. The fast-path surely is the data-path of them.

consider seccomp that loads several programs for every tab, then container use case where we're loading two for each as well. Who knows what use cases will come up in the future.
It's obviously not a fast path that is being hit million times a second, but why add overhead when it's unnecessary?

Quoted 1 line

But the browser using seccomp does not need persistent maps at all. It

today it doesn't, but if it was a light weight feature, it could have been used much more broadly.
Currently we're using user agent for networking and it's a pain.

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

completely unnecessary here. The kernel will consume more memory for no real reason other than cdev are used to keep prog/maps around.

I don't consider this a big issue, and well worth the trade-off. You'll have an infrastructure that integrates *nicely* into the *existing* kernel model *and* tooling with the proposed patch. This is a HUGE plus. The UAPI of this is simple and minimal. And to me, these are in-fact special files, not regular ones.

Quoted 7 lines

Quoted 1 line

Seriously? Syscall to create/destory cdevs is a nice api?

It is pretty normal to do. They don't use syscall but ioctl on a special pseudo device node (lvm2/device-mapper, tun, etc.). Same thing, very overloaded syscall. Also I don't see a reason to not make the creation possible via sysfs, which would be even nicer but not orthogonal to the current creation of maps, which is nowadays set in stone by uapi.

Isn't it your point going against cdev? ioctls on devs are overloaded, whereas here we have clean bpf syscall with no extra baggage. Going old-school to device model goes against that clean syscall approach.

Quoted 9 lines

sysfs already has infrastructure to ensure supportability and debugability. Dependency graphs can be created to see which programs depend on which bpf_progs. kobjects are singeltons in sysfs representation. If you have multiple ebpf filesystems, even maybe referencing the same hashtable with gigabytes of data multiple times, there needs to be some way to help administrators to check resource usage, statistics, tweak and tune the rhashtable. All this needs to be handled as well in the future. It doesn't really fit the filesystem model, but representing a kobject seems to be a good fit to me.

quite the opposite. The way cdev patch is done now, there is no way to extend it to create a hierarchy without breaking users, whereas fs style with initial Daniel's patch can be extended. Doing 'resource stats' via sysfs requires bpf to add to sysfs, which is not this cdev approach.

Quoted 3 lines

Policy can be done by user space with help of udev. Selinux policies can easily being extended to allow specific domains access. Namespace "passthrough" is defined for devices already.

that's an interesting point, but isn't it better done with fs? you can go much more fine-grained with directory permissions in fs. With different users/namespaces having their own hierarchies and mounts whereas cdev dumps everything into one spot.

Quoted 10 lines

Quoted 4 lines

where do you see 'over-design' in fs? It's a straightforward code and most of it is boilerplate like all other fs.
Kill rmdir/mkdir ops, term bit and options and the fs diff will shrink by another 200+ lines.

I don't think that only a filesystem will do it in the foreseeable future. You want to have tools like lsof reporting which map and which program has references to each other. Those file nodes will need more metadata attached in future anyway, so currently just comparing lines of codes seems not to be a good way for arguing.

my point was that both have roughly the same number, but the lines of code will grow in either approach and when cdev starts as a hack I can only see more hacks added in the future, whereas fs gives us full flexibility to do any type of file access and user visible representation.
Also I don't buy the point of reinventing sysfs. bpffs is not doing sysfs. I don't want to see _every_ bpf object in sysfs. It's way too much overhead. Classic doesn't have sysfs and everyone have been using it just fine.

bpffs is solving the need to 'pin_fd' when user process exits. That's it. Let's solve that and not go into any of this sysfs stuff.

I doubt it will stay a lightweight feature as it should not be in the responsibility of user space to provide those debug facilities.

The bpf syscall is still used to create the pseudo nodes. If they should be persistent they just get registered in the sysfs class hierarchy.

Quoted 2 lines

Doing 'resource stats' via sysfs requires bpf to add to sysfs, which is not this cdev approach.

This is not yet part of the patch, but I think this would be added. Daniel?

I don't think there are broad differences. But in case a namespaces uses huge number of maps with tons of data, the admin in the initial namespace might want to debug that without searching all mountpoints and find dependencies between processes etc. IMHO sysfs approach can be better extended here.

Quoted 4 lines

Also I don't buy the point of reinventing sysfs. bpffs is not doing sysfs. I don't want to see _every_ bpf object in sysfs. It's way too much overhead. Classic doesn't have sysfs and everyone have been using it just fine.

But classic bpf does not have persistence for maps and data. ;) There is a 1:1 relationship between socket and bpf_prog for example.

But how can the filesystem be extended in terms of tunables and information? File attributes? Wouldn't it need the same infrastructure otherwise as sysfs? Some third-party lookup filesystem or ioctl? This char dev approach also pins maps and progs while giving more policy in hand of central user space programs we are currently using (udev, systemd, whatever, etc.).

Quoted 3 lines · Hannes Frederic Sowa

I doubt it will stay a lightweight feature as it should not be in the responsibility of user space to provide those debug facilities.

It feels we're talking past each other.
I want to solve 'persistent map' problem.
debugging of maps/progs, hierarchy, etc are all nice to have, but different issues.

In case of persistent maps I imagine unprivileged process would want to use it eventually as well, so this requirement already kills cdev approach for me, since I don't think we ever let unprivileged apps create cdev with syscall.

Quoted 2 lines

The bpf syscall is still used to create the pseudo nodes. If they should be persistent they just get registered in the sysfs class hierarchy.

nope. they should not. sysfs is debugging/tunning facility. There is absolutely no need for bpf to plug into sysfs.

Quoted 5 lines

Quoted 2 lines

Doing 'resource stats' via sysfs requires bpf to add to sysfs, which is not this cdev approach.

This is not yet part of the patch, but I think this would be added. Daniel?

please don't. I'm strongly against adding unnecessary bloat.

Quoted 5 lines

I don't think there are broad differences. But in case a namespaces uses huge number of maps with tons of data, the admin in the initial namespace might want to debug that without searching all mountpoints and find dependencies between processes etc. IMHO sysfs approach can be better extended here.

sure, then we can force all bpffs to have the same hierarchy and mounted in /sys/kernel/bpf location. That would be the same.

It feels you're pushing for cdev only because of that potential debugging need. Did you actually face that need? I didn't and don't like to add 'nice to have' feature until real need comes.

Quoted 9 lines · Alexei Starovoitov

Quoted 3 lines · Hannes Frederic Sowa

I doubt it will stay a lightweight feature as it should not be in the responsibility of user space to provide those debug facilities.

It feels we're talking past each other.
I want to solve 'persistent map' problem.
debugging of maps/progs, hierarchy, etc are all nice to have, but different issues.

Ok, so you are saying that this file system will have *only* regular files that are to be considered nodes for eBPF maps and progs. Nothing else will ever be added to it? Yet, eBPF map and prog nodes are *not* *regular files* to me, this seems odd.

As soon as you are starting to add additional folders that contain files dumping additional meta data, etc, you basically end up on a bit of similar concept to sysfs, no?

Quoted 4 lines

In case of persistent maps I imagine unprivileged process would want to use it eventually as well, so this requirement already kills cdev approach for me, since I don't think we ever let unprivileged apps create cdev with syscall.

Hmm, I see. So far the discussion was only about having this for privileged users (also in this fs patch). F.e. privileged system daemons could setup and distribute progs/maps to consumers, etc (f.e. seccomp and tc case).

When we start lifting this, eBPF maps by its own will become a real kernel IPC facility for unprivileged Linux applications (independently whether they are connected to an actual eBPF program). Those kernel IPC facilities that are anchored in the file system like named pipes and Unix domain sockets
are indicated as such as special files, no?

Quoted 2 lines

sure, then we can force all bpffs to have the same hierarchy and mounted in /sys/kernel/bpf location. That would be the same.

That would imply to have a mount_single() file system (like f.e. tracefs and
securityfs), right?

So you'd loose having various mounts in different namespaces. And if you allow various mount points, how would that /facilitate/ to an admin to identify all eBPF objects/resources currently present in the system?

Or to an application developer finding possible mount points for his own application so that bpf(2) syscall to create these nodes succeeds? Would you make mounting also unprivileged? What if various distros have these mount points at different locations? What should unprivileged applications do to know that they can use these locations for themselves?

Also, since they are only regular files, one can only try and find these objects based on their naming schemes, which seems to get a bit odd in case this file system also carries (perhaps future?) other regular files that are not eBPF map and program-special.

Quoted 3 lines

It feels you're pushing for cdev only because of that potential debugging need. Did you actually face that need? I didn't and don't like to add 'nice to have' feature until real need comes.

I think this discussion arose, because the question of how flexible we are in future to extend this facility. Nothing more.

Quoted 16 lines · Daniel Borkmann

Quoted 9 lines · Alexei Starovoitov

Quoted 3 lines · Hannes Frederic Sowa

I doubt it will stay a lightweight feature as it should not be in the responsibility of user space to provide those debug facilities.

It feels we're talking past each other.
I want to solve 'persistent map' problem.
debugging of maps/progs, hierarchy, etc are all nice to have, but different issues.

Ok, so you are saying that this file system will have *only* regular files that are to be considered nodes for eBPF maps and progs. Nothing else will ever be added to it? Yet, eBPF map and prog nodes are *not* *regular files* to me, this seems odd.

Quoted 3 lines

As soon as you are starting to add additional folders that contain files dumping additional meta data, etc, you basically end up on a bit of similar concept to sysfs, no?

as we discussed in this thread and earlier during plumbers I think it would be good to expose key/values somehow in this fs. 'how' is a big question.

But regardless which path we take, sysfs is too rigid. For the sake of argument say we do every key as a new file in bpffs. It's not very scalable, but comparing to sysfs it's better (resource wise).

If we decide to add bpf syscall command to expose map details we can provide pretty printer to it in a form of printk-like string or via some schema passed to syscall, so that keys(file names) can look properly in bpffs.
The above and other ideas all possible in bpffs, but not possible in sysfs.

Quoted 8 lines

Quoted 4 lines

In case of persistent maps I imagine unprivileged process would want to use it eventually as well, so this requirement already kills cdev approach for me, since I don't think we ever let unprivileged apps create cdev with syscall.

Hmm, I see. So far the discussion was only about having this for privileged users (also in this fs patch). F.e. privileged system daemons could setup and distribute progs/maps to consumers, etc (f.e. seccomp and tc case).

It completely makes sense to restrict it to admin today, but design should not prevent relaxing it in the future.

Quoted 6 lines

When we start lifting this, eBPF maps by its own will become a real kernel IPC facility for unprivileged Linux applications (independently whether they are connected to an actual eBPF program). Those kernel IPC facilities that are anchored in the file system like named pipes and Unix domain sockets
are indicated as such as special files, no?

not everything in unix is a model that should be followed. af_unix with name[0]!=0 is a bad api that wasn't thought through. Thankfully Linux improved it with abstract names that don't use special files.
bpf maps obviously is not an IPC (either pinned or not).

Quoted 6 lines

Quoted 2 lines

sure, then we can force all bpffs to have the same hierarchy and mounted in /sys/kernel/bpf location. That would be the same.

That would imply to have a mount_single() file system (like f.e. tracefs and
securityfs), right?

Probably. I'm not sure whether it should be single fs or we allow multiple mount points. There are pro and con for both.

Quoted 3 lines

So you'd loose having various mounts in different namespaces. And if you allow various mount points, how would that /facilitate/ to an admin to identify all eBPF objects/resources currently present in the system?

if it's single mount point there are no issues, but would be nice to separate users and namespaces somehow.

Quoted 5 lines

Or to an application developer finding possible mount points for his own application so that bpf(2) syscall to create these nodes succeeds? Would you make mounting also unprivileged? What if various distros have these mount points at different locations? What should unprivileged applications do to know that they can use these locations for themselves?

mounting is root only of course and having standard location answers all of these questions.

Quoted 4 lines

Also, since they are only regular files, one can only try and find these objects based on their naming schemes, which seems to get a bit odd in case this file system also carries (perhaps future?) other regular files that are not eBPF map and program-special.

not sure what you meant. If names are given by kernel, there are no problem finding them. If by user, we'd file attributes like the way you did with xattr.

Quoted 6 lines

Quoted 3 lines

It feels you're pushing for cdev only because of that potential debugging need. Did you actually face that need? I didn't and don't like to add 'nice to have' feature until real need comes.

I think this discussion arose, because the question of how flexible we are in future to extend this facility. Nothing more.

Exactly and cdev style pushes us into the corner of traditional cdev with ioctl which I don't think is flexible enough.

Quoted 3 lines

as we discussed in this thread and earlier during plumbers I think it would be good to expose key/values somehow in this fs. 'how' is a big question.

Yes, it is a big question, and probably best left to the domain-specific application itself, which can already dump the map nowadays via bpf(2) syscall. You can add bindings to various languages to make it available elsewhere as well.

Or, you have a user space 'bpf' tool that can connect to any map that is being exposed with whatever model, and have modular pretty printers in user space somewhere located as shared objects, they could get auto-loaded in the background. Maps could get an annotation attached as an attribute during creation that is being exposed somewhere, so it can be mapped to a pretty printer shared object. This would better be solved in user space entirely, in my opinion, why should the kernel add complexity for this when this is so much user-space application specific anyway?

As we all agreed, looking into key/values via shell is a rare event and not needed most of the times. It comes with it's own problems (f.e. think of dumping a possible rhashtable map with key/values as files). But even iff we'd want to stick this into files by all means, fusefs can do this specific job entirely in user space _plus_ fetching these shared objects for pretty printers etc, all we need for this is to add this annotation/ mapping attribute somewhere to bpf_maps and that's all it takes.

This question is no doubt independant of the fd pinning mechanism, but as I said, I don't think sticking this into the kernel is a good idea. Why would that be the kernel's job?

In the other email, you are mentioning fdinfo. fdinfo can be done for any map/prog already today by just adding the right .show_fdinfo() callback to bpf_map_fops and bpf_prog_fops, so we let the anon-inodes that we already use today to do this job for free and such debugging info can be inspected through procfs already. This is common practice, f.e. look at timerfd, signalfd and others.

Quoted 4 lines

But regardless which path we take, sysfs is too rigid. For the sake of argument say we do every key as a new file in bpffs. It's not very scalable, but comparing to sysfs it's better (resource wise).

I doubt this is scaleable at all, no matter if its sysfs or a own custom fs. How should that work. You have a map with possibly thousands or millions
of entries. Are these files to be generated on the fly like in procfs as soon as you enter that directory? Or as a one-time snapshot (but then the user mights want to create various snapshots)? There might be new map elements as building blocks in the future such as pipes, ring buffers etc. How are they being dumped as files?

Quoted 5 lines

not everything in unix is a model that should be followed. af_unix with name[0]!=0 is a bad api that wasn't thought through. Thankfully Linux improved it with abstract names that don't use special files.
bpf maps obviously is not an IPC (either pinned or not).

So, if this pinning facility is unprivileged and available for *all* applications, then applications can in-fact use eBPF maps (w/o any other aides such as Unix domain sockets to transfer fds) among themselves to exchange state via bpf(2) syscall. It doesn't need a corresponding program.

Okay, sure, but then having a mount_single() and separating users and namespaces is still not being resolved, as you've noticed.

So, if you distribute the names through the kernel and dictate a strict hierarchy, then we'll end up with a similar model that cdevs resolve.

Quoted 26 lines · Daniel Borkmann

Quoted 3 lines

as we discussed in this thread and earlier during plumbers I think it would be good to expose key/values somehow in this fs. 'how' is a big question.

Yes, it is a big question, and probably best left to the domain-specific application itself, which can already dump the map nowadays via bpf(2) syscall. You can add bindings to various languages to make it available elsewhere as well.

Or, you have a user space 'bpf' tool that can connect to any map that is being exposed with whatever model, and have modular pretty printers in user space somewhere located as shared objects, they could get auto-loaded in the background. Maps could get an annotation attached as an attribute during creation that is being exposed somewhere, so it can be mapped to a pretty printer shared object. This would better be solved in user space entirely, in my opinion, why should the kernel add complexity for this when this is so much user-space application specific anyway?

As we all agreed, looking into key/values via shell is a rare event and not needed most of the times. It comes with it's own problems (f.e. think of dumping a possible rhashtable map with key/values as files). But even iff we'd want to stick this into files by all means, fusefs can do this specific job entirely in user space _plus_ fetching these shared objects for pretty printers etc, all we need for this is to add this annotation/ mapping attribute somewhere to bpf_maps and that's all it takes.

Quoted 3 lines

This question is no doubt independant of the fd pinning mechanism, but as I said, I don't think sticking this into the kernel is a good idea. Why would that be the kernel's job?

agree with all of the concerns above. I said it would be good for kernel to expose key/values and I still think it would be a useful feature. Regardless whether kernel does it or not in the future, the point was 'IF we want kernel to do it then bpf FS is the right way'.

Quoted 6 lines

In the other email, you are mentioning fdinfo. fdinfo can be done for any map/prog already today by just adding the right .show_fdinfo() callback to bpf_map_fops and bpf_prog_fops, so we let the anon-inodes that we already use today to do this job for free and such debugging info can be inspected through procfs already. This is common practice, f.e. look at timerfd, signalfd and others.

I know. That's exactly what I proposed, but again the point was that fdinfo of regular FDs should match in style to pinned FDs, 'cat /sys/kernel/bpf/.../map5' should be similar to 'cat /proc/.../fdinfo/5'
and 'cat /sys/kernel/bpf...' you can only cleanly do with bpffs.

Quoted 5 lines

Quoted 4 lines

But regardless which path we take, sysfs is too rigid. For the sake of argument say we do every key as a new file in bpffs. It's not very scalable, but comparing to sysfs it's better (resource wise).

Quoted 8 lines

I doubt this is scaleable at all, no matter if its sysfs or a own custom fs. How should that work. You have a map with possibly thousands or millions
of entries. Are these files to be generated on the fly like in procfs as soon as you enter that directory? Or as a one-time snapshot (but then the user mights want to create various snapshots)? There might be new map elements as building blocks in the future such as pipes, ring buffers etc. How are they being dumped as files?

you're arguing that keys as files are not scalable. sure. See what I said above "it's not very scalable"
The point is that fs approach is more flexible comparing to cdev.

Quoted 11 lines

Quoted 5 lines

not everything in unix is a model that should be followed. af_unix with name[0]!=0 is a bad api that wasn't thought through. Thankfully Linux improved it with abstract names that don't use special files.
bpf maps obviously is not an IPC (either pinned or not).

So, if this pinning facility is unprivileged and available for *all* applications, then applications can in-fact use eBPF maps (w/o any other aides such as Unix domain sockets to transfer fds) among themselves to exchange state via bpf(2) syscall. It doesn't need a corresponding program.

Obviously I know that, but it doesn't make it an IPC.
Just because two processes can talk to each other via normal tcpip it doesn't make tcpip an IPC mechanism.
The point is "just because two processes can communicate with each other via X (bpf maps) we are not going to optimize (or make architectural decisions in X) just for this use case". It's a job of generic IPC and we have enough of them already.

Quoted 2 lines

Okay, sure, but then having a mount_single() and separating users and namespaces is still not being resolved, as you've noticed.

yes and that's what I proposed to do:
Tweaking this FS patch to do mount_single() and define directory structure is the best way forward.

Quoted 2 lines

So, if you distribute the names through the kernel and dictate a strict hierarchy, then we'll end up with a similar model that cdevs resolve.

yes. exactly.
but comparing to cdev, it will be:
- cheaper for kernel to keep (memory wise)
- faster to pin FDs
- do normal 'rm' to destroy
- possible to extend to unprivileged users
- possible to add fdinfo (same output for pinned and normal fd)
- possible to expose key/value

I'm puzzled how you can keep arguing in favor of cdev when it's obviously deficient comparing to fs and fs has no disadvantages. Looks like we can only resolve it over beer.
How about we setup a public hangout ? Today or tomorrow?

Just FYI: Using a device for this kind of interface is pretty much a non-starter as that quickly gets you into situations where things do not work in containers. If someone gets a version of device namespaces past GregKH it might be up for discussion to use character devices.

On 10/20/2015 08:56 PM, Eric W. Biederman wrote:
...

Quoted 5 lines

Just FYI: Using a device for this kind of interface is pretty much a non-starter as that quickly gets you into situations where things do not work in containers. If someone gets a version of device namespaces past GregKH it might be up for discussion to use character devices.

Okay, you are referring to this discussion here:

http://thread.gmane.org/gmane.linux.kernel.containers/26760

What had been mentioned earlier in this thread was to have a namespace pass-through facility enforced by device cgroups we have in the kernel, which is one out of various means used to enforce policy today by deployment systems such as docker, for example. But more below.

I think this all depends on the kind of expectations we have, where all this is going. In the original proposal, it was agreed to have the operation that creates a node as 'capable(CAP_SYS_ADMIN)'-only (in the way like most of the rest of eBPF is restricted), and based on the use case we distribute such objects to unprivileged applications. But I understand that it seems the trend lately to lift eBPF restrictions at some point anyway, and thus the CAP_SYS_ADMIN is suddenly irrelevant again. Fair enough.

Don't get me wrong, I really don't mind if it will be some version of this fs patch or whatever architecture else we find consensus on, I think this discussion is merely trying to evaluate/discuss on what seems to be a good fit, also in terms of future requirements and integration.

So far, during this discussion, it was proposed to modify the file system to a single-mount one and to stick this under /sys/kernel/bpf/. This will not have "real" namespace support either, but it was proposed to have a following structure:

/sys/kernel/bpf/username/<optional_dirs_mkdir_by_user>/progX

So, the file system will have kind of a user home-directory for each user to isolate through permissions, if I understood correctly.

If we really want to go this route, then I think there are no big stones in the way for the other model either. It should look roughly drafted like the below.

Together with device cgroups for containers, it would allow scenarios where you can have:

* eBPF (map/prog) device pass-through so a map/prog could even be shared out from the initial namespace into individual ones/all (one could possibly extend such maps as read-only for these consumers).
* eBPF device creation for unprivileged users with permissions being set accordingly (as in fs case).
* Since cgroup controller can also do wildcards on major/minors, we could make that further fine-grained.
* eBPF device creation can also be enforced by the cgroup controller to be entirely disallowed for a specific container.

(An admin can determine the dynamically created major f.e. under /proc/devices.)

Quoted 6 lines

So far, during this discussion, it was proposed to modify the file system

Quoted 5 lines

to a single-mount one and to stick this under/sys/kernel/bpf/. This will not have "real" namespace support either, but it was proposed to have a following structure:

/sys/kernel/bpf/username/<optional_dirs_mkdir_by_user>/progX

This would probably work as you would typically map the ebpf map using -v like this to give a stable path:

docker run -v /sys/kernel/bpf/foo/maps/progX:/map proX

Quoted 12 lines

Quoted 12 lines

Together with device cgroups for containers, it would allow scenarios where you can have:

* eBPF (map/prog) device pass-through so a map/prog could even be shared out from the initial namespace into individual ones/all (one could possibly extend such maps as read-only for these consumers).
* eBPF device creation for unprivileged users with permissions being set accordingly (as in fs case).
* Since cgroup controller can also do wildcards on major/minors, we could make that further fine-grained.
* eBPF device creation can also be enforced by the cgroup controller to be entirely disallowed for a specific container.

I've read the discussion passively and my take away is that, frankly, I think the differences are somewhat minor. Both architectures can scale to what we need. Both will do the job. I'm slightly worried about exposing uAPI as a FS, I think that didn't work too well for sysfs. It's pretty much a define the format once and never touch it again kind of deal.

all users) When you have to hack drivers/base/core.c to get there it should have been a warning sign that something is wrong with this cdev approach.

Quoted 6 lines

I've read the discussion passively and my take away is that, frankly, I think the differences are somewhat minor. Both architectures can scale to what we need. Both will do the job. I'm slightly worried about exposing uAPI as a FS, I think that didn't work too well for sysfs. It's pretty much a define the format once and never touch it again kind of deal.

It's even worse in cdev style since it piggy backs on sysfs.

On 10/20/2015 08:56 PM, Eric W. Biederman wrote:
...

Quoted 5 lines

Just FYI: Using a device for this kind of interface is pretty much a non-starter as that quickly gets you into situations where things do not work in containers. If someone gets a version of device namespaces past GregKH it might be up for discussion to use character devices.

Okay, you are referring to this discussion here:

http://thread.gmane.org/gmane.linux.kernel.containers/26760

Quoted 10 lines · Daniel Borkmann

On 10/20/2015 08:56 PM, Eric W. Biederman wrote:
...

Quoted 5 lines

Just FYI: Using a device for this kind of interface is pretty much a non-starter as that quickly gets you into situations where things do not work in containers. If someone gets a version of device namespaces past GregKH it might be up for discussion to use character devices.

Okay, you are referring to this discussion here:

Quoted 1 line

   http://thread.gmane.org/gmane.linux.kernel.containers/26760

That is a piece of it. It is an old old discussion (which generally has been handled poorly). For the forseeable future device namespaces have a firm NACK by GregKH. Which means that dynamic character device based interfaces do not work in containers. Which means if you are not talking about physical hardware, character devices are a poor fit.

Quoted 11 lines · Daniel Borkmann

Quoted 9 lines · Alexei Starovoitov

Quoted 3 lines · Hannes Frederic Sowa

I doubt it will stay a lightweight feature as it should not be in the responsibility of user space to provide those debug facilities.

It feels we're talking past each other.
I want to solve 'persistent map' problem.
debugging of maps/progs, hierarchy, etc are all nice to have, but different issues.

Quoted 8 lines

Ok, so you are saying that this file system will have *only* regular files that are to be considered nodes for eBPF maps and progs. Nothing else will ever be added to it? Yet, eBPF map and prog nodes are *not* *regular files* to me, this seems odd.

As soon as you are starting to add additional folders that contain files dumping additional meta data, etc, you basically end up on a bit of similar concept to sysfs, no?

as we discussed in this thread and earlier during plumbers I think it would be good to expose key/values somehow in this fs. 'how' is a big question.
But regardless which path we take, sysfs is too rigid. For the sake of argument say we do every key as a new file in bpffs. It's not very scalable, but comparing to sysfs it's better (resource wise).
If we decide to add bpf syscall command to expose map details we can provide pretty printer to it in a form of printk-like string or via some schema passed to syscall, so that keys(file names) can look properly in bpffs.
The above and other ideas all possible in bpffs, but not possible in sysfs.

Quoted 8 lines

Quoted 4 lines

In case of persistent maps I imagine unprivileged process would want to use it eventually as well, so this requirement already kills cdev approach for me, since I don't think we ever let unprivileged apps create cdev with syscall.

Hmm, I see. So far the discussion was only about having this for privileged users (also in this fs patch). F.e. privileged system daemons could setup and distribute progs/maps to consumers, etc (f.e. seccomp and tc case).

It completely makes sense to restrict it to admin today, but design should not prevent relaxing it in the future.

Quoted 6 lines

When we start lifting this, eBPF maps by its own will become a real kernel IPC facility for unprivileged Linux applications (independently whether they are connected to an actual eBPF program). Those kernel IPC facilities that are anchored in the file system like named pipes and Unix domain sockets
are indicated as such as special files, no?

not everything in unix is a model that should be followed. af_unix with name[0]!=0 is a bad api that wasn't thought through. Thankfully Linux improved it with abstract names that don't use special files.
bpf maps obviously is not an IPC (either pinned or not).

Quoted 6 lines

Quoted 2 lines

sure, then we can force all bpffs to have the same hierarchy and mounted in /sys/kernel/bpf location. That would be the same.

That would imply to have a mount_single() file system (like f.e. tracefs and
securityfs), right?

Probably. I'm not sure whether it should be single fs or we allow multiple mount points. There are pro and con for both.

Quoted 3 lines

So you'd loose having various mounts in different namespaces. And if you allow various mount points, how would that /facilitate/ to an admin to identify all eBPF objects/resources currently present in the system?

if it's single mount point there are no issues, but would be nice to separate users and namespaces somehow.

Quoted 5 lines

Or to an application developer finding possible mount points for his own application so that bpf(2) syscall to create these nodes succeeds? Would you make mounting also unprivileged? What if various distros have these mount points at different locations? What should unprivileged applications do to know that they can use these locations for themselves?

mounting is root only of course and having standard location answers all of these questions.

Quoted 4 lines

Also, since they are only regular files, one can only try and find these objects based on their naming schemes, which seems to get a bit odd in case this file system also carries (perhaps future?) other regular files that are not eBPF map and program-special.

not sure what you meant. If names are given by kernel, there are no problem finding them. If by user, we'd file attributes like the way you did with xattr.

Quoted 4 lines

Quoted 3 lines

It feels you're pushing for cdev only because of that potential debugging need. Did you actually face that need? I didn't and don't like to add 'nice to have' feature until real need comes.

Quoted 2 lines

I think this discussion arose, because the question of how flexible we are in future to extend this facility. Nothing more.

Exactly and cdev style pushes us into the corner of traditional cdev with ioctl which I don't think is flexible enough.

Quoted 3 lines · Hannes Frederic Sowa

I doubt it will stay a lightweight feature as it should not be in the responsibility of user space to provide those debug facilities.

It feels we're talking past each other.
I want to solve 'persistent map' problem.
debugging of maps/progs, hierarchy, etc are all nice to have, but different issues.

In case of persistent maps I imagine unprivileged process would want to use it eventually as well, so this requirement already kills cdev approach for me, since I don't think we ever let unprivileged apps create cdev with syscall.

Quoted 2 lines

The bpf syscall is still used to create the pseudo nodes. If they should be persistent they just get registered in the sysfs class hierarchy.

nope. they should not. sysfs is debugging/tunning facility. There is absolutely no need for bpf to plug into sysfs.

Quoted 5 lines

Quoted 2 lines

Doing 'resource stats' via sysfs requires bpf to add to sysfs, which is not this cdev approach.

This is not yet part of the patch, but I think this would be added. Daniel?

please don't. I'm strongly against adding unnecessary bloat.

Quoted 5 lines

I don't think there are broad differences. But in case a namespaces uses huge number of maps with tons of data, the admin in the initial namespace might want to debug that without searching all mountpoints and find dependencies between processes etc. IMHO sysfs approach can be better extended here.

sure, then we can force all bpffs to have the same hierarchy and mounted in /sys/kernel/bpf location. That would be the same.

It feels you're pushing for cdev only because of that potential debugging need. Did you actually face that need? I didn't and don't like to add 'nice to have' feature until real need comes.

Quoted 7 lines

Quoted 4 lines

Also I don't buy the point of reinventing sysfs. bpffs is not doing sysfs. I don't want to see _every_ bpf object in sysfs. It's way too much overhead. Classic doesn't have sysfs and everyone have been using it just fine.

But classic bpf does not have persistence for maps and data. ;) There is a 1:1 relationship between socket and bpf_prog for example.

single task in seccomp can have a chain of bpf progs, so hierarchy is already there.

Quoted 6 lines

But how can the filesystem be extended in terms of tunables and information? File attributes? Wouldn't it need the same infrastructure otherwise as sysfs? Some third-party lookup filesystem or ioctl? This char dev approach also pins maps and progs while giving more policy in hand of central user space programs we are currently using (udev, systemd, whatever, etc.).

tunables for bpf maps? There are no such things today.
I think you're implying that we can add rhashtable type of map, so admin can tune thresholds ? Ouch. I think if we add it, its parameters will be specified by the user that is creating the map only. There will be no tunables exposed to sysfs and there should be no way of creating maps via sysfs.

I bet commercial software will make use of this ebpf framework, too. And the kernel always helped me and gave me a way to see what is going on, debug which part of my operating system universe interacts with which other part. Merely dropping file descriptors with data attached to them in an filesystem seems not to fulfill my need at all. I would love to see where resources are referenced and why, like I am nowadays.

Quoted 3 lines

Quoted 3 lines

It feels you're pushing for cdev only because of that potential debugging need. Did you actually face that need? I didn't and don't like to add 'nice to have' feature until real need comes.

Given that we want to monitor the load of a hashmap for graphing purposes. Or liberate some hashmaps from its restriction on number of keys and make upper bounds configurable by admins who know the dimensions of their systems and not some software deep down buried in the bpf syscall where I might not have access to source code. In tc force e.g. hashmaps to do garbage collection because we cannot be sure that under DoS attacks user space clean up gets scheduled early enough if ebpf adds flows to hashtables. I do see need to expand and implement some kind of policy in the future.

Quoted 2 lines

Quoted 2 lines

single task in seccomp can have a chain of bpf progs, so hierarchy is already there.

And it would be great to inspect them.

I am fine with creating maps only by bpf syscall. But to hide configuration details or at least not be really able to query them easily seems odd to me. If we go with the ebpffs how could those attributes be added?

Quoted 6 lines · Hannes Frederic Sowa

I bet commercial software will make use of this ebpf framework, too. And the kernel always helped me and gave me a way to see what is going on, debug which part of my operating system universe interacts with which other part. Merely dropping file descriptors with data attached to them in an filesystem seems not to fulfill my need at all. I would love to see where resources are referenced and why, like I am nowadays.

agree. common fs with hierarchy will give this visibility in one place.

Quoted 12 lines

Quoted 3 lines

Quoted 3 lines

It feels you're pushing for cdev only because of that potential debugging need. Did you actually face that need? I didn't and don't like to add 'nice to have' feature until real need comes.

Given that we want to monitor the load of a hashmap for graphing purposes. Or liberate some hashmaps from its restriction on number of keys and make upper bounds configurable by admins who know the dimensions of their systems and not some software deep down buried in the bpf syscall where I might not have access to source code. In tc force e.g. hashmaps to do garbage collection because we cannot be sure that under DoS attacks user space clean up gets scheduled early enough if ebpf adds flows to hashtables. I do see need to expand and implement some kind of policy in the future.

disagree here. admin should not interfere with map parameters. What you proposing above sounds very very dangerous.
Admins to configure GC of maps? What do you think the programs will do with such sophisticated maps? What kind of networking app you have in mind? Anyway that's a bit off-topic. I'm very curious though.

Quoted 3 lines

Quoted 2 lines

Quoted 2 lines

single task in seccomp can have a chain of bpf progs, so hierarchy is already there.

And it would be great to inspect them.

again let's not mix criu and lsof-like requirements with 'pin fd'. For visibility of normal maps we can add fdinfo and lsof can pick it up without any fs or any cdevs.

Quoted 4 lines

I am fine with creating maps only by bpf syscall. But to hide configuration details or at least not be really able to query them easily seems odd to me. If we go with the ebpffs how could those attributes be added?

I'm not advocating to hide details. Most of the time maps will not be pinned, so fdinfo seems the easiest way to show things like key_size, value_size, max_entries, type.

Even if we decide to do it some other way, it's not related to 'pin fd' discussion, since debugging/visibility is nice to have for all bpf objects. Note that walking of key/value without pretty-printers provided by the app is meaningless for admin, so only things like 'how much memory this map is using' are useful.

May be we should try to draft the hierarchy of this common fs. How about:
/sys/kernel/bpf/username/optional_dirs_mkdir_by_user/progX and 'cat' of it will print the same as fdinfo for normal maps, so admin can see what maps were pinned by user and its cost.

Inside 'fdinfo' output we can provide pointers to which progs are using which maps as

# cat /sys/kernel/bpf/.../mapX
key_size: 4
used_by: /proc/xxx/fd/5
# cat /sys/kernel/bpf/.../progY
type: socket
using: /proc/xxx/fd/6
using: /sys/kernel/bpf/.../mapZ
and similar for cat /proc/xxx/fdinfo/6
but showing hierarchy as directories is non starter, since
it's no a tree.

All of these would be nice, but doesn't have to be implemented along with 'pin fd' feature.

<off-topic>
Just a pretty obvious idea is accurate sampling of flows. </off-topic>

fdinfo tells me where my position in a file is and which locks the file have?

So far, if someone wants to delve into the details of a map my approach would be to take the file descriptor and make it persistence. I have to think about that some more.

Yes, absolutely and I am absolutely against pretty printing key values in kernel domain.

So cat-ing them will produce text output with some details about the map? This is what I wanted to avoid. The concept with symlinks and small files seems much cleaner and nicer to me. Also you cannot add writable attributes to this filesystem or you overload stuff heavily?

It is not a tree but a graph, sure, that's why sysfs allows to break the cyclic dependencies and create symlinks (see holders/ directories). ;)

And if you implement the same set of features IMHO you basically re-implement sysfs. In the beginning we just expose the basic maps and there won't be any features in sysfs, but it will be cheap to have read/write flags on maps etc. etc. (I don't know what people will come up with, yet.). In my opinion those are clearly attributes of a map and should be defined and managed alongside with their holders.

The pinfd feature will provide the future infrastructure alongside to make this usable, so I think it is worth spending time to think about it.

Quoted 25 lines · Eric W. Biederman

Quoted 20 lines · Daniel Borkmann

Quoted 11 lines · Alexei Starovoitov

Quoted 9 lines · Hannes Frederic Sowa

Another question:
Should multiple mount of the filesystem result in an empty fs (a new instance) or in one were one can see other ebpf-fs entities? I think Daniel wanted to already use the mountpoint as some kind of hierarchy delimiter. I would have used directories for that and multiple mounts would then have resulted in the same content of the filesystem. IMHO this would remove some ambiguity but then the question arises how this is handled in a namespaced environment. Was there some specific reason to do so?

Quoted 7 lines

That's an interesting question!
I think all mounts should be independent.
I can see tracing using one and networking using another one with different hierarchies suitable for their own use cases. What's an advantage to have the same content everywhere? Feels harder to manage, since different users would need to coordinate.

Quoted 3 lines

I initially had it as a mount_single() file system, where I was thinking to have an entry under /sys/fs/bpf/, so all subsystems would work on top of that mount point, but for the same reasons above I lifted that restriction.

Quoted 5 lines

I am missing something.

When I suggested using a filesystem it was my thought there would be exactly one superblock per map, and the map would be specified at mount time. You clearly are not implementing that.

I don't think it's practical to have sb per map, since that would mean sb per prog and that won't scale.
Also map today is an fd that belongs to a process. I cannot see an api from C program to do 'mount of FD' that wouldn't look like ugly hack.

Quoted 8 lines

A filesystem per map makes sense as you have a key-value store with one file per key.

The idea is that something resembling your bpf_pin_fd function would be the mount system call for the filesystem.

The the keys in the map could be read by "ls /mountpoint/". Key values could be inspected with "cat /mountpoint/key".

yes. that is still the goal for follow up patches, but contained within given bpffs. Something bpf_pin_fd-like command for bpf syscall would create files for keys in a map and allow 'cat' via open/read. Such api would be much cleaner from C app point of view. Potentially we can allow mount of a file created via BPF_PIN_FD that will expand into keys/values.

There, actually, the main contention point is 'how to represent keys and values'. whether key is hex representation or we need some pretty-printers via format string or via schema? etc, etc. We tried few ideas of representing keys in our fuse implementations, but don't have an agreement yet.

This set adds support for persistent maps/progs. Please see individual patches for further details.

A man-page update to bpf(2) will be sent afterwards, also a iproute2 patch for support in tc.

Thanks!