Edited draft of bpf(2) man page for review/enhancement

Hello Alexei,

I took the draft 3 of the bpf(2) man page that you sent back in March and did some substantial editing to clarify the language and add a few technical details. Could you please check the revised version below, to ensure I did not inject any errors.

I also added a number of FIXMEs for pieces of the page that need further work. Could you take a look at these and let me know your thoughts, please.

.TH BPF 2 2015-03-10 "Linux" "Linux Programmer's Manual" .SH NAME
bpf - perform a command on an extended BPF map or program .SH SYNOPSIS
.nf
.B #include <linux/bpf.h>
.sp
.BI "int bpf(int cmd, union bpf_attr *attr, unsigned int size);

.SH DESCRIPTION
The
.BR bpf ()
system call performs a range of operations related to extended Berkeley Packet Filters.
Extended BPF (or eBPF) is similar to
the original BPF (or classic BPF) used to filter network packets. For both BPF and eBPF programs,
the kernel statically analyzes the programs before loading them, in order to ensure that they cannot harm the running system. .P
eBPF extends classic BPF in multiple ways including the ability to call in-kernel helper functions (via the
.B BPF_CALL
opcode extension provided by eBPF)
and access shared data structures such as BPF maps.

The programs can be written in a restricted C that is compiled into .\" FIXME In the next line, what is "a restricted C"? Where does
.\" one get further information about it?

eBPF bytecode and executed on the in-kernel virtual machine or just-in-time compiled into native code.
.SS Extended BPF Design/Architecture
.P
.\" FIXME In the following line, what does "different data types" mean?
.\" Are the values in a map not just blobs?

BPF maps are a generic data structure for storage of different data types. A user process can create multiple maps (with key/value-pairs being opaque bytes of data) and access them via file descriptors. BPF programs can access maps from inside the kernel in parallel. It's up to the user process and BPF program to decide what they store inside maps.
.P
BPF programs are similar to kernel modules.
They are loaded by the user
process and automatically unloaded when the process exits.

Each BPF program is a set of instructions that is safe to run until its completion.
The in-kernel BPF verifier statically determines that the program terminates and is safe to execute.
.\" FIXME In the following sentence, what does "takes hold" mean?

During verification, the program takes hold of maps that it intends to use, so selected maps cannot be removed until the program is unloaded.

BPF programs can be attached to different events.
.\" FIXME: In the next sentence , "packets" are not "events". What .\" do you really mean to say here? ("the arrival of a network packet"?)

These events can be packets, tracing
events, and other types that may be added in the future.

A new event triggers execution of the BPF program, which
may store information about the event in the maps.
Beyond storing data, BPF programs may call into in-kernel helper functions.

The same program can be attached to multiple events and different programs can access the same map:
.\" FIXME Can maps be shared between processes? (E.g., what happens
.\" when fork() is called?)

Quoted 10 lines · Michael Kerrisk (man-pages)

Hello Alexei,

I took the draft 3 of the bpf(2) man page that you sent back in March and did some substantial editing to clarify the language and add a few technical details. Could you please check the revised version below, to ensure I did not inject any errors.

I also added a number of FIXMEs for pieces of the page that need further work. Could you take a look at these and let me know your thoughts, please.

That's great, thanks! Minor comments:

...

Quoted 9 lines

.TH BPF 2 2015-03-10 "Linux" "Linux Programmer's Manual" .SH NAME
bpf - perform a command on an extended BPF map or program .SH SYNOPSIS
.nf
.B #include <linux/bpf.h>
.sp
.BI "int bpf(int cmd, union bpf_attr *attr, unsigned int size);

Quoted 16 lines

.SH DESCRIPTION
The
.BR bpf ()
system call performs a range of operations related to extended Berkeley Packet Filters.
Extended BPF (or eBPF) is similar to
the original BPF (or classic BPF) used to filter network packets. For both BPF and eBPF programs,
the kernel statically analyzes the programs before loading them, in order to ensure that they cannot harm the running system. .P
eBPF extends classic BPF in multiple ways including the ability to call in-kernel helper functions (via the
.B BPF_CALL
opcode extension provided by eBPF)
and access shared data structures such as BPF maps.

I would perhaps emphasize that maps can be shared among in-kernel eBPF programs, but also between kernel and user space.

Quoted 3 lines

The programs can be written in a restricted C that is compiled into .\" FIXME In the next line, what is "a restricted C"? Where does
.\" one get further information about it?

So far only from the kernel samples directory and for tc classifier and action, from the tc man page and/or examples/bpf/ in the tc git tree.

Quoted 6 lines

eBPF bytecode and executed on the in-kernel virtual machine or just-in-time compiled into native code.
.SS Extended BPF Design/Architecture
.P
.\" FIXME In the following line, what does "different data types" mean?
.\" Are the values in a map not just blobs?

Sort of, currently, these blobs can have different sizes of keys and values (you can even have structs as keys). For the map itself they are treated as blob internally. However, recently, bpf tail call got added where you can lookup another program from an array map and call into it. Here, that particular type of map can only have entries of type of eBPF program fd. I think, if needed, adding a paragraph to the tail call could be done as follow-up after we have an initial man page in the tree included.

Quoted 10 lines

BPF maps are a generic data structure for storage of different data types. A user process can create multiple maps (with key/value-pairs being opaque bytes of data) and access them via file descriptors. BPF programs can access maps from inside the kernel in parallel. It's up to the user process and BPF program to decide what they store inside maps.
.P
BPF programs are similar to kernel modules.
They are loaded by the user
process and automatically unloaded when the process exits.

Generally that's true. Btw, in 4.1 kernel, tc(8) also got support for eBPF classifier and actions, and here it's slightly different: in tc, we load the programs, maps etc, and push down the eBPF program fd in order to let the kernel hold reference on the program itself.

Thus, there, the program fd that the application owns is gone when the application terminates, but the eBPF program itself still lives on inside the kernel. But perhaps it's already too much detail to mention here ...

Quoted 5 lines

Each BPF program is a set of instructions that is safe to run until its completion.
The in-kernel BPF verifier statically determines that the program terminates and is safe to execute.
.\" FIXME In the following sentence, what does "takes hold" mean?

Takes a reference. Meaning, that maps cannot disappear under us while the eBPF program that is using them in the kernel is still alive.

Quoted 6 lines

During verification, the program takes hold of maps that it intends to use, so selected maps cannot be removed until the program is unloaded.

BPF programs can be attached to different events.
.\" FIXME: In the next sentence , "packets" are not "events". What .\" do you really mean to say here? ("the arrival of a network packet"?)

Socket filters is meant here: setsockopt(sock, SOL_SOCKET, SO_ATTACH_BPF, ...);

Quoted 2 lines

These events can be packets, tracing
events, and other types that may be added in the future.

There's already one more type worth mentioning: tc classifier and actions (tc/tc_bpf.{c,h}).

Quoted 3 lines

A new event triggers execution of the BPF program, which
may store information about the event in the maps.
Beyond storing data, BPF programs may call into in-kernel helper functions.

I would mention that these in-kernel helpers are a fixed set of functions provided by the kernel (linux/bpf.h: BPF_FUNC_*), so you cannot call into arbitrary ones.

Quoted 4 lines

The same program can be attached to multiple events and different programs can access the same map:
.\" FIXME Can maps be shared between processes? (E.g., what happens
.\" when fork() is called?)

Yes, they can. Map file descriptors can also be transferred via socket passing (SCM_RIGHTS). tc is doing that, i.e. since tc terminates itself after configuring the kernel, it can pass the map fds via Unix domain socket to an BPF agent that will be the new owner. That can f.e. be a shell, so this is very well possible.

Hi Daniel,

Sorry for the long delay in following up....

Quoted 41 lines · Michael Kerrisk (man-pages)

Quoted 10 lines

Hello Alexei,

I took the draft 3 of the bpf(2) man page that you sent back in March and did some substantial editing to clarify the language and add a few technical details. Could you please check the revised version below, to ensure I did not inject any errors.

I also added a number of FIXMEs for pieces of the page that need further work. Could you take a look at these and let me know your thoughts, please.

That's great, thanks! Minor comments:

...

Quoted 25 lines

.TH BPF 2 2015-03-10 "Linux" "Linux Programmer's Manual" .SH NAME
bpf - perform a command on an extended BPF map or program .SH SYNOPSIS
.nf
.B #include <linux/bpf.h>
.sp
.BI "int bpf(int cmd, union bpf_attr *attr, unsigned int size);

.SH DESCRIPTION
The
.BR bpf ()
system call performs a range of operations related to extended Berkeley Packet Filters.
Extended BPF (or eBPF) is similar to
the original BPF (or classic BPF) used to filter network packets. For both BPF and eBPF programs,
the kernel statically analyzes the programs before loading them, in order to ensure that they cannot harm the running system. .P
eBPF extends classic BPF in multiple ways including the ability to call in-kernel helper functions (via the
.B BPF_CALL
opcode extension provided by eBPF)
and access shared data structures such as BPF maps.

Quoted 2 lines

I would perhaps emphasize that maps can be shared among in-kernel eBPF programs, but also between kernel and user space.

This is covered later in the page, under the "BPF maps" subheading. Maybe you missed that? (Or did you think it doesn't suffice?)

Quoted 4 lines

Quoted 3 lines

The programs can be written in a restricted C that is compiled into .\" FIXME In the next line, what is "a restricted C"? Where does
.\" one get further information about it?

Quoted 3 lines

So far only from the kernel samples directory and for tc classifier and action, from the tc man page and/or examples/bpf/ in the tc git tree.

So, given that we are several weeks down the track, and things may have changed, I'll re-ask the questions ;-) :

* Is this restricted C documented anywhere?

* Is the procedure for compiling this restricted C documented anywhere? (Yes, it's LLVM, but are the suitable pipelines/options documented somewhere?)

Quoted 7 lines

Quoted 6 lines

eBPF bytecode and executed on the in-kernel virtual machine or just-in-time compiled into native code.
.SS Extended BPF Design/Architecture
.P
.\" FIXME In the following line, what does "different data types" mean?
.\" Are the values in a map not just blobs?

Quoted 8 lines

Sort of, currently, these blobs can have different sizes of keys and values (you can even have structs as keys). For the map itself they are treated as blob internally. However, recently, bpf tail call got added where you can lookup another program from an array map and call into it. Here, that particular type of map can only have entries of type of eBPF program fd. I think, if needed, adding a paragraph to the tail call could be done as follow-up after we have an initial man page in the tree included.

Okay -- I've added a FIXME placeholder for this, so we can revisit.

Quoted 16 lines

Quoted 10 lines

BPF maps are a generic data structure for storage of different data types. A user process can create multiple maps (with key/value-pairs being opaque bytes of data) and access them via file descriptors. BPF programs can access maps from inside the kernel in parallel. It's up to the user process and BPF program to decide what they store inside maps.
.P
BPF programs are similar to kernel modules.
They are loaded by the user
process and automatically unloaded when the process exits.

Generally that's true. Btw, in 4.1 kernel, tc(8) also got support for eBPF classifier and actions, and here it's slightly different: in tc, we load the programs, maps etc, and push down the eBPF program fd in order to let the kernel hold reference on the program itself.

Quoted 4 lines

Thus, there, the program fd that the application owns is gone when the application terminates, but the eBPF program itself still lives on inside the kernel. But perhaps it's already too much detail to mention here ...

Well, it should be documented somewhere....

Quoted 8 lines

Quoted 5 lines

Each BPF program is a set of instructions that is safe to run until its completion.
The in-kernel BPF verifier statically determines that the program terminates and is safe to execute.
.\" FIXME In the following sentence, what does "takes hold" mean?

Takes a reference. Meaning, that maps cannot disappear under us while the eBPF program that is using them in the kernel is still alive.

So, I changed this to:

[[
During verification, the kernel increments reference counts for each of the maps that the eBPF program uses,
so that the selected maps cannot be removed until the program is unloaded. ]]

Okay?

I'll send out a new draft soon, but in the meantime hopefully you or Alexei might have a chance to answer some open questions (see my other mail to Alexei, which will be sent soon), so I can further edit the page before sending it out.

Quoted 3 lines · Michael Kerrisk (man-pages)

Hi Daniel,

Sorry for the long delay in following up....

No worries, eBPF is quite some material. ;)

Quoted 42 lines · Daniel Borkmann

Quoted 41 lines · Michael Kerrisk (man-pages)

Quoted 10 lines

Hello Alexei,

I took the draft 3 of the bpf(2) man page that you sent back in March and did some substantial editing to clarify the language and add a few technical details. Could you please check the revised version below, to ensure I did not inject any errors.

I also added a number of FIXMEs for pieces of the page that need further work. Could you take a look at these and let me know your thoughts, please.

That's great, thanks! Minor comments:

...

Quoted 25 lines

.TH BPF 2 2015-03-10 "Linux" "Linux Programmer's Manual" .SH NAME
bpf - perform a command on an extended BPF map or program .SH SYNOPSIS
.nf
.B #include <linux/bpf.h>
.sp
.BI "int bpf(int cmd, union bpf_attr *attr, unsigned int size);

.SH DESCRIPTION
The
.BR bpf ()
system call performs a range of operations related to extended Berkeley Packet Filters.
Extended BPF (or eBPF) is similar to
the original BPF (or classic BPF) used to filter network packets. For both BPF and eBPF programs,
the kernel statically analyzes the programs before loading them, in order to ensure that they cannot harm the running system. .P
eBPF extends classic BPF in multiple ways including the ability to call in-kernel helper functions (via the
.B BPF_CALL
opcode extension provided by eBPF)
and access shared data structures such as BPF maps.

Quoted 5 lines

Quoted 2 lines

I would perhaps emphasize that maps can be shared among in-kernel eBPF programs, but also between kernel and user space.

This is covered later in the page, under the "BPF maps" subheading. Maybe you missed that? (Or did you think it doesn't suffice?)

Okay, I presume you mean:

Maps are a generic data structure for storage of different types and sharing data between the kernel and user-space programs.

Maybe, to emphasize both options a bit (not sure if it's better in my words, though):

Maps are a generic data structure for storage of different types and allow for sharing data among eBPF kernel programs, but also between kernel and user-space applications.

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

The programs can be written in a restricted C that is compiled into .\" FIXME In the next line, what is "a restricted C"? Where does
.\" one get further information about it?

Quoted 8 lines

Quoted 3 lines

So far only from the kernel samples directory and for tc classifier and action, from the tc man page and/or examples/bpf/ in the tc git tree.

So, given that we are several weeks down the track, and things may have changed, I'll re-ask the questions ;-) :

* Is this restricted C documented anywhere?

Not (yet) that I'm aware of. We were thinking that short-mid term to polish the stuff that resides in the kernel documentation, that is, Documentation/networking/filter.txt, to get it in a better shape, which I presume, would also include a documentation on the restricted C. So far, examples are provided in the tc-bpf man page (see link below).

The set of available helper functions callable from eBPF resides under (enum bpf_func_id):
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/bpf.h

Quoted 21 lines

Quoted 20 lines

Quoted 10 lines

BPF maps are a generic data structure for storage of different data types. A user process can create multiple maps (with key/value-pairs being opaque bytes of data) and access them via file descriptors. BPF programs can access maps from inside the kernel in parallel. It's up to the user process and BPF program to decide what they store inside maps.
.P
BPF programs are similar to kernel modules.
They are loaded by the user
process and automatically unloaded when the process exits.

Generally that's true. Btw, in 4.1 kernel, tc(8) also got support for eBPF classifier and actions, and here it's slightly different: in tc, we load the programs, maps etc, and push down the eBPF program fd in order to let the kernel hold reference on the program itself.

Thus, there, the program fd that the application owns is gone when the application terminates, but the eBPF program itself still lives on inside the kernel. But perhaps it's already too much detail to mention here ...

Quoted 1 line

Well, it should be documented somewhere....

Yep, fwiw some time ago I've hacked together a man page for tc:

https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/commit/?id=cbdd1e6921d21815e35d2a96526cfbad5ac98e09

Quoted 4 lines

I'll send out a new draft soon, but in the meantime hopefully you or Alexei might have a chance to answer some open questions (see my other mail to Alexei, which will be sent soon), so I can further edit the page before sending it out.

Later on, we should also add a paragraph on eBPF tail calls, but one step at a time.

Hello Alexei,

I took the draft 3 of the bpf(2) man page that you sent back in March and did some substantial editing to clarify the language and add a few technical details. Could you please check the revised version below, to ensure I did not inject any errors.

I also added a number of FIXMEs for pieces of the page that need further work. Could you take a look at these and let me know your thoughts, please.

.SH DESCRIPTION
The
.BR bpf ()
system call performs a range of operations related to extended Berkeley Packet Filters.
Extended BPF (or eBPF) is similar to
the original BPF (or classic BPF) used to filter network packets. For both BPF and eBPF programs,

.\" FIXME In the next line, what is "a restricted C"? Where does
.\" one get further information about it?

.\" FIXME In the following sentence, what does "takes hold" mean? During verification, the program takes hold of maps that it intends to use, so selected maps cannot be removed until the program is unloaded.

BPF programs can be attached to different events.
.\" FIXME: In the next sentence , "packets" are not "events". What .\" do you really mean to say here? ("the arrival of a network packet"?) These events can be packets, tracing
events, and other types that may be added in the future.

.\" FIXME Can maps be shared between processes? (E.g., what happens
.\" when fork() is called?)

struct { /* Used by BPF_PROG_LOAD */
__u32 prog_type;
__u32 insn_cnt;
__aligned_u64 insns; /* 'const struct bpf_insn *' */
__aligned_u64 license; /* 'const char *' */
__u32 log_level; /* verbosity level of verifier */
__u32 log_size; /* size of user buffer */
__aligned_u64 log_buf; /* user supplied 'char *'
buffer */
};

bpf_create_map(enum bpf_map_type map_type, int key_size,
int value_size, int max_entries)
{
union bpf_attr attr = {
.map_type = map_type,
.key_size = key_size,
.value_size = value_size,
.max_entries = max_entries
};

return bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
}

Currently, two
.I map_type
are supported:

.in +4n
.nf
enum bpf_map_type {
BPF_MAP_TYPE_UNSPEC,
BPF_MAP_TYPE_HASH,
BPF_MAP_TYPE_ARRAY,

.\" FIXME Explain the purpose of BPF_MAP_TYPE_UNSPEC

.\" FIXME We need an explanation of BPF_MAP_TYPE_HASH here
.\" FIXME We need an explanation of BPF_MAP_TYPE_ARRAY here .\" FIXME We need an explanation of why one might choose HASH versus ARRAY

.\" FIXME Here, I think we need some statement about what 'value' must
.\" point to. Presumable, it must be a buffer at least as large as
.\" the map's 'value_size' attribute?

bpf_get_next_key(int fd, void *key, void *next_key)

{
     union bpf_attr attr = {
         .map_fd = fd,
         .key = ptr_to_u64(key),
         .next_key = ptr_to_u64(next_key),
     };

     return bpf(BPF_MAP_GET_NEXT_KEY, &attr, sizeof(attr));
}

.fi
.in

.\" FIXME Need to explain the return value on success here.

When the user-space program that created a map exits, all maps will be deleted automatically.
.\" FIXME What are the semantics when a file descriptor is duplicated
.\" (dup() etc.)? (I.e., when is a map deallocated automatically?)

enum bpf_prog_type {
BPF_PROG_TYPE_UNSPEC,
.\" FIXME Explain the purpose of BPF_PROG_TYPE_UNSPEC
BPF_PROG_TYPE_SOCKET_FILTER, BPF_PROG_TYPE_SCHED_CLS,
.\" FIXME BPF_PROG_TYPE_SCHED_CLS appears not to exist?

.\" FIXME The next sentence fragment is incomplete and
.I bpf_context
is a pointer to a
.IR "struct sk_buff" .
Programs cannot access fields of
.I sk_buff
directly.

More program types may be added in the future. .\" FIXME The following sentence is grammatically broken.
.\" What should it say?
Like
.B BPF_PROG_TYPE_KPROBE
and
.I bpf_context
for it may be defined as a pointer to a
.IR "struct pt_regs" .

is a license string, which must be GPL compatible to call helper functions .\" FIXME Maybe we should list the GPL compatible strings that can be
.\" specified?

Maps are accessible from BPF programs and are used to exchange data between BPF programs and between BPF programs and user-space programs. Programs process various events (like kprobe, packets) and
store their data into maps.
User-space programs fetch data from the maps.
.\" FIXME We need some elaboration here... What does the next sentence mean? Either the same or a different map may be used by user space
as a configuration space to alter program behavior on the fly.

.SH EXAMPLES
.\" FIXME It would be nice if this was a complete working example

         BPF_MOV64_IMM(BPF_REG_1, 1),                /* r1 = 1 */
         BPF_XADD(BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0),
.\" FIXME What does 'lock' in the line below mean?
                                 /* lock *(u64 *) r0 += r1 */

BPF_MOV64_IMM(BPF_REG_0, 0), /* r0 = 0 */
BPF_EXIT_INSN(), /* return r0 */
};

prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, prog,
.\" FIXME The next line looks wrong. Should it not be .\"
.\" sizeof(prog) / sizeof(struct bpf_insn) ?

                             sizeof(prog), "GPL");

Quoted 6 lines · Michael Kerrisk (man-pages)

Hello Alexei,

I took the draft 3 of the bpf(2) man page that you sent back in March and did some substantial editing to clarify the language and add a few technical details. Could you please check the revised version below, to ensure I did not inject any errors.

Great!

Quoted 3 lines

I also added a number of FIXMEs for pieces of the page that need further work. Could you take a look at these and let me know your thoughts, please.

Thanks to Daniel for answering some of them :)

Other answers and comments:

Quoted 8 lines

.SH DESCRIPTION
The
.BR bpf ()
system call performs a range of operations related to extended Berkeley Packet Filters.
Extended BPF (or eBPF) is similar to
the original BPF (or classic BPF) used to filter network packets. For both BPF and eBPF programs,

btw, for tc man page we've adopted eBPF and cBPF abbreviations. Since just BPF was confusing especially in tc context that supports both. This man page mainly talking about eBPF, but I think it would help to call 'classic BPF' as 'cBPF' as well.
tc-bpf manpage:
https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/tree/man/man8/tc-bpf.8?h=net-next

Quoted 2 lines

.\" FIXME In the next line, what is "a restricted C"? Where does
.\" one get further information about it?

As Daniel said there is no spec for this C. It's a normal C where things like loops, global variables, vararg, floating point, struct passing and bunch of other things are not supported.

Quoted 3 lines

.\" FIXME In the following sentence, what does "takes hold" mean? During verification, the program takes hold of maps that it intends to use, so selected maps cannot be removed until the program is unloaded.

if 'takes hold' is confusing, just remove that sentence. Its purpose was to describe that map will be still alive while some program is using it. Traditional refcnt approach.

Quoted 5 lines

BPF programs can be attached to different events.
.\" FIXME: In the next sentence , "packets" are not "events". What .\" do you really mean to say here? ("the arrival of a network packet"?) These events can be packets, tracing
events, and other types that may be added in the future.

yes. For eBPF programs attached to sockets 'event == arrival of a packet on that socket'.

For eBPF attached to tc classifier 'event == classification request by qdisc'.

Quoted 2 lines

.\" FIXME Can maps be shared between processes? (E.g., what happens
.\" when fork() is called?)

yes. Would be good to mention ability to pass prog and map FDs via scm_rights.

Quoted 10 lines

struct { /* Used by BPF_PROG_LOAD */
__u32 prog_type;
__u32 insn_cnt;
__aligned_u64 insns; /* 'const struct bpf_insn *' */
__aligned_u64 license; /* 'const char *' */
__u32 log_level; /* verbosity level of verifier */
__u32 log_size; /* size of user buffer */
__aligned_u64 log_buf; /* user supplied 'char *'
buffer */
};

this part is already obsolete. The following field was added:
__u32 kern_version; /* checked when prog_type=kprobe */
see commit 2541517c32be2.

Quoted 10 lines

bpf_create_map(enum bpf_map_type map_type, int key_size,
int value_size, int max_entries)
{
union bpf_attr attr = {
.map_type = map_type,
.key_size = key_size,
.value_size = value_size,
.max_entries = max_entries
};

Quoted 2 lines

     return bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
}

btw, this simple wrappers are currently in samples/bpf/ and in iproute2/tc/tc_bpf.c. We're trying to consolidate them into generic libbpf library in tools/lib/

Quoted 4 lines

Currently, two
.I map_type
are supported:

Quoted 6 lines

.in +4n
.nf
enum bpf_map_type {
BPF_MAP_TYPE_UNSPEC,
BPF_MAP_TYPE_HASH,
BPF_MAP_TYPE_ARRAY,

in the net-next tree we have BPF_MAP_TYPE_PROG_ARRAY as well.

Quoted 1 line

.\" FIXME Explain the purpose of BPF_MAP_TYPE_UNSPEC

only to reserve zero as invalid type. both for programs and maps.

Quoted 3 lines

.\" FIXME We need an explanation of BPF_MAP_TYPE_HASH here
.\" FIXME We need an explanation of BPF_MAP_TYPE_ARRAY here .\" FIXME We need an explanation of why one might choose HASH versus ARRAY

type implies underlying implementation.
commit 0f8e4bd8a1fc8c4 describes hash map.
commit 28fbcfa08d8ed7c describes array map.
Briefly:
hash - hashtable where map_update_elem() replaces elements in atomic way. Elements allocated on the fly.
array - all elements pre-allocated, zero initialized and cannot be delete. Fastest possible lookup (just like arrays in C)

Quoted 3 lines

.\" FIXME Here, I think we need some statement about what 'value' must
.\" point to. Presumable, it must be a buffer at least as large as
.\" the map's 'value_size' attribute?

yes. that's correct. Though buffer > value_size doesn't make sense.

Quoted 13 lines

bpf_get_next_key(int fd, void *key, void *next_key)

{
     union bpf_attr attr = {
         .map_fd = fd,
         .key = ptr_to_u64(key),
         .next_key = ptr_to_u64(next_key),
     };

     return bpf(BPF_MAP_GET_NEXT_KEY, &attr, sizeof(attr));
}

.fi
.in

Quoted 1 line

.\" FIXME Need to explain the return value on success here.

hmm, you mean something like:
'If key is found, returns zero and sets next_key to the next element'?

Quoted 4 lines

When the user-space program that created a map exits, all maps will be deleted automatically.
.\" FIXME What are the semantics when a file descriptor is duplicated
.\" (dup() etc.)? (I.e., when is a map deallocated automatically?)

just like other pseudo files. The kernel side will be freed when refcnt reaches zero, so dup or scm_rights passing will keep map/program alive.

Quoted 6 lines

enum bpf_prog_type {
BPF_PROG_TYPE_UNSPEC,
.\" FIXME Explain the purpose of BPF_PROG_TYPE_UNSPEC
BPF_PROG_TYPE_SOCKET_FILTER, BPF_PROG_TYPE_SCHED_CLS,
.\" FIXME BPF_PROG_TYPE_SCHED_CLS appears not to exist?

The current list is:
enum bpf_prog_type {
BPF_PROG_TYPE_UNSPEC,
BPF_PROG_TYPE_SOCKET_FILTER,
BPF_PROG_TYPE_KPROBE,
BPF_PROG_TYPE_SCHED_CLS,
BPF_PROG_TYPE_SCHED_ACT,
};

Quoted 8 lines

.\" FIXME The next sentence fragment is incomplete and
.I bpf_context
is a pointer to a
.IR "struct sk_buff" .
Programs cannot access fields of
.I sk_buff
directly.

Actually now in case of SOCKET_FILTER, SCHED_CLS, SCHED_ACT the program can now access skb fields.
See 'struct __sk_buff' and commit 9bac3d6d548e5

Quoted 9 lines

More program types may be added in the future. .\" FIXME The following sentence is grammatically broken.
.\" What should it say?
Like
.B BPF_PROG_TYPE_KPROBE
and
.I bpf_context
for it may be defined as a pointer to a
.IR "struct pt_regs" .

Sorry. I meant to say that 'struct bpf_context * == struct pt_regs *' in case of TYPE_KPROBE.

Quoted 3 lines

is a license string, which must be GPL compatible to call helper functions .\" FIXME Maybe we should list the GPL compatible strings that can be
.\" specified?

probably not. At least I wouldn't.

Quoted 8 lines

Maps are accessible from BPF programs and are used to exchange data between BPF programs and between BPF programs and user-space programs. Programs process various events (like kprobe, packets) and
store their data into maps.
User-space programs fetch data from the maps.
.\" FIXME We need some elaboration here... What does the next sentence mean? Either the same or a different map may be used by user space
as a configuration space to alter program behavior on the fly.

Sorry. I tried to emphasize that user space can use maps as a way to pass information to the program.
For example, typical tracing program aggregates information about kprobe events into a map, then user space reads it. The information can go the other way too.
User space can populate a map with some data and depending on that data, the program will do different things.

Quoted 2 lines

.SH EXAMPLES
.\" FIXME It would be nice if this was a complete working example

open_raw_sock() takes too many lines not related to bpf. The examples in samples/bpf/*_kern.c are all functional :) May be a reference will work instead?

Quoted 4 lines

         BPF_MOV64_IMM(BPF_REG_1, 1),                /* r1 = 1 */
         BPF_XADD(BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0),
.\" FIXME What does 'lock' in the line below mean?
                                 /* lock *(u64 *) r0 += r1 */

it means that it's 'lock xadd' equivalent.

Quoted 4 lines

         BPF_MOV64_IMM(BPF_REG_0, 0),                /* r0 = 0 */
         BPF_EXIT_INSN(),                            /* return r0 */
     };

Quoted 5 lines

prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, prog,
.\" FIXME The next line looks wrong. Should it not be .\"
.\" sizeof(prog) / sizeof(struct bpf_insn) ?

                             sizeof(prog), "GPL");

sizeof(prog) is correct for this particular sample code. btw, this full example is in samples/bpf/sock_example.c

Quoted 3 lines

As Daniel said there is no spec for this C. It's a normal C where things like loops, global variables, vararg, floating point, struct passing and bunch of other things are not supported.

I assume we're talking about the LLVM front-end, right?

Am I correct that these kernel source files are examples of this restricted C:

samples/bpf/tcbpf1_kern.c
samples/bpf/tracex2_kern.c
samples/bpf/tracex4_kern.c
samples/bpf/tracex1_kern.c
samples/bpf/tracex3_kern.c
samples/bpf/sockex1_kern.c
samples/bpf/sockex2_kern.c

And samples/bpf/Makefile shows the necessary LLVM incantation to produce the BPF binaries, right?

Anyway, I added the following text in NOTES:

eBPF objects (maps and programs) can be shared between pro‐
cesses. For example, after fork(2), the child inherits file
descriptors referring to the same eBPF objects. In addition,
file descriptors referring to eBPF objects can be transferred over UNIX domain sockets. File descriptors referring to eBPF objects can be duplicated in the usual way, using dup(2) and similar calls. An eBPF object is deallocated only after all file descriptors referring to the object have been closed.

Is the above all correct?

This makes me curious: why was the BPF functionality not designed as a *set* of system calls (as per these wrappers), rather than the existing multiplexed call?

[[
If
.I key
is found, the operation returns zero and sets the
.I next_key
pointer to the key of the next element.
]]

right?

Quoted 6 lines

Quoted 4 lines

          BPF_MOV64_IMM(BPF_REG_1, 1),                /* r1 = 1 */
          BPF_XADD(BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0),
.\" FIXME What does 'lock' in the line below mean?
                                  /* lock *(u64 *) r0 += r1 */

it means that it's 'lock xadd' equivalent.

Sorry -- you've assumed I'm cleverer than I am... :-} I'm not wiser after that comment. What is 'lock xadd'?

If you might have a chance to look at my questions above and let me know your thoughts, then I could further edit the page before sending out the next draft.

Quoted 5 lines · Michael Kerrisk (man-pages)

Quoted 3 lines

As Daniel said there is no spec for this C. It's a normal C where things like loops, global variables, vararg, floating point, struct passing and bunch of other things are not supported.

I assume we're talking about the LLVM front-end, right?

yes. clang.
There is a bpf backend for gcc, but it's bit rotting now.

Quoted 9 lines

Am I correct that these kernel source files are examples of this restricted C:

samples/bpf/tcbpf1_kern.c
samples/bpf/tracex2_kern.c
samples/bpf/tracex4_kern.c
samples/bpf/tracex1_kern.c
samples/bpf/tracex3_kern.c
samples/bpf/sockex1_kern.c
samples/bpf/sockex2_kern.c

yes.

Quoted 2 lines

And samples/bpf/Makefile shows the necessary LLVM incantation to produce the BPF binaries, right?

yes.
Now with llvm 3.7 coming out soon it's even simpler. Just:
clang -O2 -target bpf -c file.c

Quoted 11 lines

Anyway, I added the following text in NOTES:

eBPF objects (maps and programs) can be shared between pro‐
cesses. For example, after fork(2), the child inherits file
descriptors referring to the same eBPF objects. In addition,
file descriptors referring to eBPF objects can be transferred over UNIX domain sockets. File descriptors referring to eBPF objects can be duplicated in the usual way, using dup(2) and similar calls. An eBPF object is deallocated only after all file descriptors referring to the object have been closed.

Quoted 1 line

Is the above all correct?

yes. all correct.

Quoted 3 lines

This makes me curious: why was the BPF functionality not designed as a *set* of system calls (as per these wrappers), rather than the existing multiplexed call?

because new commands are much easier to add to existing syscall instead of adding new syscall for every new command.

Quoted 8 lines

[[
If
.I key
is found, the operation returns zero and sets the
.I next_key
pointer to the key of the next element.
]]

Quoted 1 line

right?

yes.

Quoted 9 lines

Quoted 6 lines

Quoted 4 lines

          BPF_MOV64_IMM(BPF_REG_1, 1),                /* r1 = 1 */
          BPF_XADD(BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0),
.\" FIXME What does 'lock' in the line below mean?
                                  /* lock *(u64 *) r0 += r1 */

it means that it's 'lock xadd' equivalent.

Sorry -- you've assumed I'm cleverer than I am... :-} I'm not wiser after that comment. What is 'lock xadd'?

I meant that it is == atomic64_add

Quoted 3 lines

If you might have a chance to look at my questions above and let me know your thoughts, then I could further edit the page before sending out the next draft.

I think would be great to get some form of the man page out and work on it incrementally. Quite a few folks have asked for it.