Documenting eBPF - extended Berkeley Packet Filter
Subsystems (4)
cover Documenting eBPF - extended Berkeley Packet Filter
Question: What kernel tree should this go into???
If going through Jonathan Corbet, will it appear sooner here???
https://www.kernel.org/doc/html/latest/
If it will not appear sooner that way, then it's likely best to keep it in sync with the tree that takes eBPF code changes.
This marks the beginning of user-facing developer documentation for using eBPF (extended Berkeley Packet Filter) as part of the kernel Documentation/ tree.
This documentation is also available here[1], as an intermidiate quick way of prototyping and releasing the documentation. The autoriative and official version of the documentation is what gets included in the kernel tree. The docs at [2] will get updated based on what gets accepted after the standard peer-review kernel process.
[1] http://prototype-kernel.readthedocs.io/en/latest/bpf/index.html
[2] https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/Documentation
Thanks to the following people, who have already reviewed and fixed earlier versions of this documentation on the IOvisor mailing-list:
Alexander Alemayhu <[email protected]>
Alexei Starovoitov <[email protected]>
Daniel Borkmann <[email protected]>
Quentin Monnet <[email protected]>
---
Jesper Dangaard Brouer (4):
doc/bpf: start eBPF documentation tree bpf/ doc/bpf: document interacting with eBPF maps doc/bpf: describes the different types of eBPF maps available doc/bpf: describe BCC the BPF Compiler Collection
Documentation/bpf/bcc_tool_chain.rst | 37 +++++ Documentation/bpf/ebpf_maps.rst | 256 +++++++++++++++++++++++++++++++++ Documentation/bpf/ebpf_maps_types.rst | 119 +++++++++++++++ Documentation/bpf/index.rst | 68 +++++++++ Documentation/index.rst | 1 5 files changed, 481 insertions(+) create mode 100644 Documentation/bpf/bcc_tool_chain.rst create mode 100644 Documentation/bpf/ebpf_maps.rst create mode 100644 Documentation/bpf/ebpf_maps_types.rst create mode 100644 Documentation/bpf/index.rst
--
PATCH 1/4 doc/bpf: start eBPF documentation tree bpf/
The learning curve for eBPF programs is hard. The purpose of this documentation (subtree) is to make it easier for developers to get started using and writing eBPF programs.
Including bpf/index under section User-oriented documentation.
Thanks to Quentin Monnet <[email protected]> for improving this document with areas eBFP is used in and early review.
--- Documentation/bpf/index.rst | 66 +++++++++++++++++++++++++++++++++++++++++++ Documentation/index.rst | 1 + 2 files changed, 67 insertions(+) create mode 100644 Documentation/bpf/index.rst
Documentation/bpf/index.rst new file +66 -0
@@ -0,0 +1,66 @@ +====================================== +eBPF - extended Berkeley Packet Filter +====================================== + +Introduction +============ + +The Berkeley Packet Filter (BPF) started (`article 1992`_) as a +special-purpose virtual machine (register based filter evaluator) for +filtering network packets, best known for its use in tcpdump. It is +documented in the kernel tree, in the first part of: +`Documentation/networking/filter.txt`_ + +The extended BPF (eBPF) variant has become a universal in-kernel +virtual machine, that has hooks all over the kernel. The eBPF +instruction set is quite different, see description in section "BPF +kernel internals" of `Documentation/networking/filter.txt`_ or look at +this `presentation by Alexei`_. + +Areas using eBPF: + * XDP - eXpress Data Path + * `Traffic control`_ + * Sockets + * Firewalling (``xt_bpf`` module) + * Tracing + * Tracepoints + * kprobe (dynamic tracing of a kernel function call) + * cgroups + +Documentation +============= + +The primary user documentation for extended BPF is in the man-page for +the `bpf(2)`_ syscall. + +This documentation is focused on the kernel tree's `samples/bpf/`_ and +`tools/lib/bpf/`_. It is worth mentioning that other projects exist, +like BCC_, that has a slightly different user-facing +syntax, but is interfacing with the same kernel facilities as those +covered by this documentation. + +.. toctree:: + :maxdepth: 1 + +.. links: + +.. _article 1992: http://www.tcpdump.org/papers/bpf-usenix93.pdf + +.. _bpf(2): http://man7.org/linux/man-pages/man2/bpf.2.html + +.. _Documentation/networking/filter.txt: + https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/filter.txt + +.. _presentation by Alexei: + http://www.slideshare.net/AlexeiStarovoitov/bpf-inkernel-virtual-machine + +.. _samples/bpf/: + https://github.com/torvalds/linux/blob/master/samples/bpf/ + +.. _tools/lib/bpf/: + https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/tools/lib/bpf/ + +.. _Traffic control: http://man7.org/linux/man-pages/man8/tc-bpf.8.html + +.. _BCC: https://github.com/iovisor/bcc +
Documentation/index.rst +1 -0
@@ -23,6 +23,7 @@ trying to get it to work optimally on a given system. :maxdepth: 2 admin-guide/index + bpf/index Introduction to kernel development ----------------------------------
PATCH 2/4 doc/bpf: document interacting with eBPF maps
In reply to Jesper Dangaard BrouerDescribe what eBPF maps are, how to create them, and how to interact with them.
Thanks to Quentin Monnet <[email protected]> for improving this document by fixing many typos and early review.
--- Documentation/bpf/ebpf_maps.rst | 255 +++++++++++++++++++++++++++++++++++++++ Documentation/bpf/index.rst | 2 2 files changed, 257 insertions(+) create mode 100644 Documentation/bpf/ebpf_maps.rst
Documentation/bpf/ebpf_maps.rst new file +255 -0
@@ -0,0 +1,255 @@ +========= +eBPF maps +========= + +This document describes what eBPF maps are, how you create them +(`Creating a map`_), and how to interact with them (`Interacting with +maps`_). + +Using eBPF maps is a method to keep state between invocations of the +eBPF program, and allows sharing data between eBPF kernel programs, +and also between kernel and user-space applications. + +Basically a key/value store with arbitrary structure (from man-page +`bpf(2)`_): + + eBPF maps are a generic data structure for storage of different data + types. Data types are generally treated as binary blobs, so a user + just specifies the size of the key and the size of the value at + map-creation time. In other words, a key/value for a given map can + have an arbitrary structure. + +The map handles are file descriptors, and multiple maps can be created +and accessed by multiple programs (from man-page `bpf(2)`_): + + A user process can create multiple maps (with key/value-pairs being + opaque bytes of data) and access them via file descriptors. + Different eBPF programs can access the same maps in parallel. It's + up to the user process and eBPF program to decide what they store + inside maps. + +.. _`Creating a map`: + +Creating a map +============== + +A map is created based on a request from userspace, via the `bpf`_ +syscall (specifically `bpf_cmd`_ BPF_MAP_CREATE), which returns a new +file descriptor that refers to the map. On error, -1 is returned and +errno is set to EINVAL, EPERM, or ENOMEM. These are the struct +``bpf_attr`` setup arguments to use when creating a map via the +syscall: + +.. code-block:: c + + bpf(BPF_MAP_CREATE, &bpf_attr, sizeof(bpf_attr)); + +Notice how this kernel ABI is extensible, as more struct arguments can +easily be added later as the sizeof(bpf_attr) is passed along to the +syscall. This also implies that API users must clear/zero +sizeof(bpf_attr), as compiler can size-align the struct differently, +to avoid garbage data to be interpreted as parameters by future +kernels. + +The following configuration attributes are needed when creating the map: + +.. code-block:: c + + union bpf_attr { + struct { /* anonymous struct used by BPF_MAP_CREATE command */ + __u32 map_type; /* one of enum bpf_map_type */ + __u32 key_size; /* size of key in bytes */ + __u32 value_size; /* size of value in bytes */ + __u32 max_entries; /* max number of entries in a map */ + __u32 map_flags; /* prealloc or not */ + }; + } + +.. _bpf_cmd: http://lxr.free-electrons.com/ident?i=bpf_cmd + + +Kernel sample/bpf ELF convention +-------------------------------- + +For programs under samples/bpf/, defining a map have been integrated +with ELF binary generated by LLVM. This is purely one example of a +userspace convention and not part of the kernel ABI. It still invokes +the bpf syscall. + +Map definitions are done by defining a ``struct bpf_map_def`` with an +elf section __attribute__ ``SEC("maps")``, in the xxx_kern.c file. +The maps file descriptor is available in the userspace xxx_user.c +file, via global array variable ``map_fd[]``, and the array map index +corresponds to the order the maps sections were defined in elf file of +xxx_kern.c file. Behind the scenes it is the ``load_bpf_file()`` call +(from `samples/bpf/bpf_load`_) that takes care of parsing ELF file +compiled by LLVM, pickup 'maps' section and creates maps via the bpf +syscall. + +.. code-block:: c + + struct bpf_map_def { + unsigned int type; + unsigned int key_size; + unsigned int value_size; + unsigned int max_entries; + unsigned int map_flags; + }; + + struct bpf_map_def SEC("maps") my_map = { + .type = BPF_MAP_TYPE_XXX, + .key_size = sizeof(u32), + .value_size = sizeof(u64), + .max_entries = 42, + .map_flags = 0 + }; + +.. section links + +.. _samples/bpf/bpf_load: + https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/samples/bpf/bpf_load.c + +Qdisc Traffic Control convention +-------------------------------- + +It is worth mentioning, that qdisc TC (Traffic Control), also use ELF +files for defining the maps, but it uses another layout. See man-page +`tc-bpf(8)`_ and `tc bpf examples`_ in iproute2.git_ tree. + +.. _iproute2.git: + https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/about/ + +.. _tc bpf examples: + https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/tree/examples/bpf + +.. _tc-bpf(8): http://man7.org/linux/man-pages/man8/tc-bpf.8.html + + +Interacting with maps +===================== + +Interacting with eBPF maps happens through some **lookup/update/delete** +primitives. + +When writing eBFP programs using load helpers and libraries from +samples/bpf/ and tools/lib/bpf/. Common function name API have been +created that hides the details of how kernel vs. userspace access +these primitives (which is quite different). + +The common function names (parameters and return values differs): + +.. code-block:: c + + void bpf_map_lookup_elem(map, void *key. ...); + void bpf_map_update_elem(map, void *key, ..., __u64 flags); + void bpf_map_delete_elem(map, void *key); + +The ``flags`` argument in ``bpf_map_update_elem()`` allows to define +semantics on whether the element exists: + +.. code-block:: c + + /* File: include/uapi/linux/bpf.h */ + /* flags for BPF_MAP_UPDATE_ELEM command */ + #define BPF_ANY 0 /* create new element or update existing */ + #define BPF_NOEXIST 1 /* create new element only if it didn't exist */ + #define BPF_EXIST 2 /* only update existing element */ + +Userspace +--------- +The userspace API map helpers are defined in `tools/lib/bpf/bpf.h`_ +and looks like this: + +.. code-block:: c + + /* Userspace helpers */ + int bpf_map_lookup_elem(int fd, void *key, void *value); + int bpf_map_update_elem(int fd, void *key, void *value, __u64 flags); + int bpf_map_delete_elem(int fd, void *key); + /* Only userspace: */ + int bpf_map_get_next_key(int fd, void *key, void *next_key); + + +Interacting with an eBPF map from **userspace**, happens through the +`bpf`_ syscall and a file descriptor. See how the map handle ``int +fd`` is a file descriptor . On success, zero is returned, on +failures -1 is returned and errno is set. + +Wrappers for the bpf syscall is implemented in `tools/lib/bpf/bpf.c`_, +and ends up calling functions in `kernel/bpf/syscall.c`_, like +`map_lookup_elem`_. + +.. code-block:: c + + /* Corresponding syscall bpf commands from userspace */ + enum bpf_cmd { + [...] + BPF_MAP_LOOKUP_ELEM, + BPF_MAP_UPDATE_ELEM, + BPF_MAP_DELETE_ELEM, + BPF_MAP_GET_NEXT_KEY, + [...] + }; + +Notice how ``void *key`` and ``void *value`` are passed as a void +pointers. Given the memory seperation between kernel and userspace, +this is a copy of the value. Kernel primitives like +``copy_from_user()`` and ``copy_to_user()`` are used, e.g. see +`map_lookup_elem`_, which also kmalloc+kfree memory for a short +period. + +From userspace, there is no function call to atomically increment or +decrement the value 'in-place'. The bpf_map_update_elem() call will +overwrite the existing value, with a copy of the value supplied. +Depending on the map type, the overwrite will happen in an atomic way, +e.g. using locking mechanisms specific to the map type. + +.. section links + +.. _tools/lib/bpf/bpf.h: + https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/tools/lib/bpf/bpf.h + +.. _tools/lib/bpf/bpf.c: + https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/tools/lib/bpf/bpf.c + +.. _map_lookup_elem: http://lxr.free-electrons.com/ident?i=map_lookup_elem + +.. _kernel/bpf/syscall.c: + https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/kernel/bpf/syscall.c + + +Kernel-side eBPF program +------------------------ + +The API mapping for eBPF programs on the kernel-side is fairly hard to +follow. It related to `samples/bpf/bpf_helpers.h`_ and maps into +`kernel/bpf/helpers.c`_ via macros. + +.. code-block:: c + + /* eBPF program helpers */ + void *bpf_map_lookup_elem(void *map, void *key); + int bpf_map_update_elem(void *map, void *key, void *value, unsigned long long flags); + int bpf_map_delete_elem(void *map, void *key); + +The eBPF-program running kernel-side interacts more directly with the +map data structures. For example the call ``bpf_map_lookup_elem()`` +returns a direct pointer to the 'value' memory-element inside the +kernel (while userspace gets a copy). This allows the eBPF-program to +atomically increment or decrement the value 'in-place', by using +appropiate compiler primitives like ``__sync_fetch_and_add()``, which +is understood by LLVM when generating eBPF instructions. + +.. section links + +.. _samples/bpf/bpf_helpers.h: + https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/samples/bpf/bpf_helpers.h + +.. _kernel/bpf/helpers.c: + https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/kernel/bpf/helpers.c + +.. links + +.. _bpf(2): http://man7.org/linux/man-pages/man2/bpf.2.html + +.. _bpf: http://man7.org/linux/man-pages/man2/bpf.2.html
Documentation/bpf/index.rst +2 -0
@@ -42,6 +42,8 @@ covered by this documentation. .. toctree:: :maxdepth: 1 + ebpf_maps + .. links: .. _article 1992: http://www.tcpdump.org/papers/bpf-usenix93.pdf
PATCH 3/4 doc/bpf: describes the different types of eBPF maps available
In reply to Jesper Dangaard BrouerThe purpose is to help choose the right map type based on the individual use-case.
To start with, only BPF_MAP_TYPE_ARRAY is described. It is the plan that all types should have descriptions here.
--- Documentation/bpf/ebpf_maps.rst | 3 + Documentation/bpf/ebpf_maps_types.rst | 119 +++++++++++++++++++++++++++++++++ Documentation/bpf/index.rst | 1 3 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 Documentation/bpf/ebpf_maps_types.rst
Documentation/bpf/ebpf_maps.rst +2 -1
@@ -4,7 +4,8 @@ eBPF maps This document describes what eBPF maps are, how you create them (`Creating a map`_), and how to interact with them (`Interacting with -maps`_). +maps`_). The different map types available are described here: +:doc:`ebpf_maps_types`. Using eBPF maps is a method to keep state between invocations of the eBPF program, and allows sharing data between eBPF kernel programs,
Documentation/bpf/ebpf_maps_types.rst new file +119 -0
119 lines not on this page · raw message
Documentation/bpf/index.rst +1 -0
@@ -43,6 +43,7 @@ covered by this documentation. :maxdepth: 1 ebpf_maps + ebpf_maps_types .. links:
PATCH 4/4 doc/bpf: describe BCC the BPF Compiler Collection
In reply to Jesper Dangaard BrouerIt is worth mentioning BCC (BPF Compiler Collection) in-order to direct developers into that community.
--- Documentation/bpf/bcc_tool_chain.rst | 37 ++++++++++++++++++++++++++++++++++ Documentation/bpf/index.rst | 5 ++--- 2 files changed, 39 insertions(+), 3 deletions(-) create mode 100644 Documentation/bpf/bcc_tool_chain.rst
Documentation/bpf/bcc_tool_chain.rst new file +37 -0
@@ -0,0 +1,37 @@ +============================= +BCC (BPF Compiler Collection) +============================= + +BCC is a toolkit to make eBPF programs easier to write, with +front-ends in Python and Lua. BCC requires LLVM and clang (in version +3.7.1 or newer) to be available on target, because BCC programs do +runtime compilation of the restricted-C code into eBPF instructions. + +BCC includes several useful tools_ and examples_, developed by +recognized performance analyst `Brendan Gregg`_ and covered with a +tutorial_ and slides_. + +.. _tools: + https://github.com/iovisor/bcc/tree/master/tools + +.. _examples: + https://github.com/iovisor/bcc/tree/master/examples + +.. _`Brendan Gregg`: http://www.brendangregg.com/ + +.. _tutorial: + https://github.com/iovisor/bcc/blob/master/docs/tutorial.md + +.. _slides: + http://www.slideshare.net/brendangregg/linux-bpf-superpowers/43/ + +The project maintains an overview of `eBPF supported kernels`_ and +what versions got which specific features. There is also a `BCC +Reference Guide`_. + +.. _eBPF supported kernels: + https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md + +.. _BCC Reference Guide: + https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md +
Documentation/bpf/index.rst +2 -3
@@ -35,7 +35,7 @@ the `bpf(2)`_ syscall. This documentation is focused on the kernel tree's `samples/bpf/`_ and `tools/lib/bpf/`_. It is worth mentioning that other projects exist, -like BCC_, that has a slightly different user-facing +like :doc:`bcc_tool_chain`, that has a slightly different user-facing syntax, but is interfacing with the same kernel facilities as those covered by this documentation.
@@ -44,6 +44,7 @@ covered by this documentation. ebpf_maps ebpf_maps_types + bcc_tool_chain .. links:
@@ -65,5 +66,3 @@ covered by this documentation. .. _Traffic control: http://man7.org/linux/man-pages/man8/tc-bpf.8.html -.. _BCC: https://github.com/iovisor/bcc -
Jonathan Corbet
maintainerQuoted 6 lines · On Tue, 07 Feb 2017 15:30:11 +0100 Jesper Dangaard Brouer <[email protected]>…
Question: What kernel tree should this go into???
If going through Jonathan Corbet, will it appear sooner here???
https://www.kernel.org/doc/html/latest/
If it will not appear sooner that way, then it's likely best to keep it in sync with the tree that takes eBPF code changes.
I've developed a fairly strong preference for carrying patches touching index.rst; otherwise I spend a lot of time explaining merge conflicts to Linus.
If the consensus is that this is ready to go, I expect I can squeeze it in for 4.11. I'm not too worried about regressions...:)
I haven't actually built it yet, but from a first look it seems like an awfully good start. The one thing that comes to mind is that I'm likely to want to move it at some point. I'd really like to start a separate book for user-space developer documentation, and this would certainly belong there. That book doesn't exist yet, though, so I can't quite blame you, hard as I might try, for not putting this document there.
Thanks,
jon
Quoted 7 lines · On Tue, 7 Feb 2017 08:37:17 -0700 Jonathan Corbet <[email protected]> wrote:
Quoted 4 lines · On Tue, 07 Feb 2017 15:30:11 +0100 Jesper Dangaard Brouer <[email protected]>…
Question: What kernel tree should this go into???
If going through Jonathan Corbet, will it appear sooner here???
https://www.kernel.org/doc/html/latest/
What about this question? Or let me ask in another way, what tree is
https://www.kernel.org/doc/html/latest/ based on?
Quoted 10 lines
Quoted 2 lines
If it will not appear sooner that way, then it's likely best to keep it in sync with the tree that takes eBPF code changes.
I've developed a fairly strong preference for carrying patches touching
index.rst; otherwise I spend a lot of time explaining merge conflicts to Linus.If the consensus is that this is ready to go, I expect I can squeeze it in for 4.11. I'm not too worried about regressions...:)
Quoted 6 lines
I haven't actually built it yet, but from a first look it seems like an awfully good start. The one thing that comes to mind is that I'm likely to want to move it at some point. I'd really like to start a separate book for user-space developer documentation, and this would certainly belong there. That book doesn't exist yet, though, so I can't quite blame you, hard as I might try, for not putting this document there.
Yes, I was also wondering hard where to put it... and a book for user-space developer documentation would likely be the right place, but it was not there, as you mention ;-)
I'm fine with moving it later under another "book". Linking to it as
HTML would still be the same right? (https://www.kernel.org/doc/html/latest/bpf/index.html)
And is the Documentation/bpf/ directory the correct place?
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Jonathan Corbet
maintainerQuoted 7 lines · On Tue, 7 Feb 2017 17:09:08 +0100 Jesper Dangaard Brouer <[email protected]>…
Quoted 4 lines
Quoted 4 lines
Question: What kernel tree should this go into???
If going through Jonathan Corbet, will it appear sooner here???
https://www.kernel.org/doc/html/latest/What about this question? Or let me ask in another way, what tree is
https://www.kernel.org/doc/html/latest/ based on?
I believe it's generated from the current -rc. If this stuff goes into 4.11, it should show up there next week.
Quoted 7 lines
Yes, I was also wondering hard where to put it... and a book for user-space developer documentation would likely be the right place, but it was not there, as you mention ;-)
I'm fine with moving it later under another "book". Linking to it as
HTML would still be the same right? (https://www.kernel.org/doc/html/latest/bpf/index.html)
And is theDocumentation/bpf/directory the correct place?
Moving it would change the URL, of course. If we want to avoid that, we should try to come up with the proper placement from the outset. And we would want to move it; I really want to clean up the mess that is the top-level directory.
How about if it goes into Documentation/userspace-guide/bpf ? The intermediate directory could just be empty for now, I'll put the book structure into place later on. Then the URL for the BPF guide itself wouldn't change.
jon
Quoted 27 lines · On Tue, 7 Feb 2017 09:46:08 -0700 Jonathan Corbet <[email protected]> wrote:
Quoted 7 lines · On Tue, 7 Feb 2017 17:09:08 +0100 Jesper Dangaard Brouer <[email protected]>…
Quoted 4 lines
Quoted 4 lines
Question: What kernel tree should this go into???
If going through Jonathan Corbet, will it appear sooner here???
https://www.kernel.org/doc/html/latest/What about this question? Or let me ask in another way, what tree is
https://www.kernel.org/doc/html/latest/ based on?I believe it's generated from the current -rc. If this stuff goes into 4.11, it should show up there next week.
Quoted 7 lines
Yes, I was also wondering hard where to put it... and a book for user-space developer documentation would likely be the right place, but it was not there, as you mention ;-)
I'm fine with moving it later under another "book". Linking to it as
HTML would still be the same right? (https://www.kernel.org/doc/html/latest/bpf/index.html)
And is theDocumentation/bpf/directory the correct place?Moving it would change the URL, of course. If we want to avoid that, we should try to come up with the proper placement from the outset. And we would want to move it; I really want to clean up the mess that is the top-level directory.
Quoted 4 lines
How about if it goes into
Documentation/userspace-guide/bpf? The intermediate directory could just be empty for now, I'll put the book structure into place later on. Then the URL for the BPF guide itself wouldn't change.
I sounds like Daniel (see other email) have bigger plans for what Documentation/BPF/ should contain. E.g. consolidating Documentation/networking/filter.txt which covers the cBPF/eBPF internals. If that is the case (and I like the idea), then it goes beyond a "userspace-guide". And perhaps "BPF" is a "book" of its own?
And it seems Daniel is proposing capital-letters BPF for the directory name "Documentation/BPF/"? Any opinions on that? (I'm neutral)
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Jonathan Corbet
maintainerQuoted 5 lines · On Tue, 7 Feb 2017 21:51:49 +0100 Jesper Dangaard Brouer <[email protected]>…
I sounds like Daniel (see other email) have bigger plans for what
Documentation/BPF/should contain. E.g. consolidatingDocumentation/networking/filter.txtwhich covers the cBPF/eBPF internals. If that is the case (and I like the idea), then it goes beyond a "userspace-guide". And perhaps "BPF" is a "book" of its own?
One of the real problems with the kernel's documentation is that there is really almost no thought given to who the audience is. We have docs for kernel developers, for system admins, for user-space developers, etc., and it's all mixed up into one big jumble.
An objective of mine in launching into this whole project is to try to fix that, so that people can readily find the documentation they need. So I don't think a single top-level directory, with a mix of user-space API info and "internals", is the right direction to go. The internals docs should, IMO, go elsewhere, probably in the core-api manual.
See what I'm getting at here?
Quoted 2 lines
And it seems Daniel is proposing capital-letters BPF for the directory name "
Documentation/BPF/"? Any opinions on that? (I'm neutral)
I think we should paint it green; otherwise I'm not too concerned about this particular point...:)
Thanks,
jon
Quoted 14 lines · On Tue, 7 Feb 2017 14:23:23 -0700 Jonathan Corbet <[email protected]> wrote:
Quoted 5 lines · On Tue, 7 Feb 2017 21:51:49 +0100 Jesper Dangaard Brouer <[email protected]>…
I sounds like Daniel (see other email) have bigger plans for what
Documentation/BPF/should contain. E.g. consolidatingDocumentation/networking/filter.txtwhich covers the cBPF/eBPF internals. If that is the case (and I like the idea), then it goes beyond a "userspace-guide". And perhaps "BPF" is a "book" of its own?One of the real problems with the kernel's documentation is that there is really almost no thought given to who the audience is. We have docs for kernel developers, for system admins, for user-space developers, etc., and it's all mixed up into one big jumble.
Quoted 7 lines
An objective of mine in launching into this whole project is to try to fix that, so that people can readily find the documentation they need. So I don't think a single top-level directory, with a mix of user-space API info and "internals", is the right direction to go. The internals docs should, IMO, go elsewhere, probably in the core-api manual.
See what I'm getting at here?
First I was reluctant (as it would be "easier" just to cramp every eBPF thing into one directory). Thinking more about, I agree with you, and I like your vision. Focus on the target audience and avoid mixing different target audience in the same document/book is the way forward.
My audience and objective is helping developers getting started using eBPF, not core-developers on eBPF (like Daniel). I do see that, if we start mixing in too much "internals" then we loose sense of the original target audience, and then they "exit" as they get "lost" in details that does not concern them.
Separating BPF docs into different directories (or "books") will make us think about the target audience.
I would like to propose directory structure:
Documentation/user-guide/bpf/
Documentation/core-api/bpf/
Quoted 5 lines
Quoted 2 lines
And it seems Daniel is proposing capital-letters BPF for the directory name "
Documentation/BPF/"? Any opinions on that? (I'm neutral)I think we should paint it green; otherwise I'm not too concerned about this particular point...:)
True, bikeshedding...
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Daniel Borkmann
maintainerHi Jesper,
Quoted 6 lines · Jesper Dangaard Brouer
Question: What kernel tree should this go into???
If going through Jonathan Corbet, will it appear sooner here???
https://www.kernel.org/doc/html/latest/
If it will not appear sooner that way, then it's likely best to keep it in sync with the tree that takes eBPF code changes.
For initial parts, I don't have a preference (Jonathan has though, so seems fine via docs tree then). If at some /later/ point in time features come in along with doc updates (similar to test case updates), probably best to route them via net-next.
Quoted 9 lines
This marks the beginning of user-facing developer documentation for using eBPF (extended Berkeley Packet Filter) as part of the kernel
Documentation/tree.This documentation is also available here[1], as an intermidiate quick way of prototyping and releasing the documentation. The autoriative and official version of the documentation is what gets included in the kernel tree. The docs at [2] will get updated based on what gets accepted after the standard peer-review kernel process.
Thanks for your effort of writing a doc. Some high-level comments on the set from my PoV first.
I think it's definitely the right direction to move everything BPF related to Documentation/BPF/. Right now, there are a lot of different places with different kind of documentation, f.e.:
* Documentation/networking/filter.txt
Covers some cBPF/eBPF internals, tooling, etc; mostly technical, historically the central spot for BPF documentation. "filter" in filter.txt is long obsolete name, but looks like various sites, talks, blogs, etc still link to it. (At best, we should keep the file saying that the doc moved to Documentation/BPF/.)
* bpf(2) man page
Has a good start, but right now is heavily behind the current user facing kernel code.
* include/uapi/linux/bpf.h
Mostly relevant for helper function API description.
* netdev conference slides/proceedings
Also contain mostly technical details on eBPF.
* https://github.com/iovisor/bpf-docs
Non-exhaustive collection of various talks from different confs.
* https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/
Even bigger and more complete list of documentation material.
* Various lwn articles ;), blog posts (f.e. from Brendan), etc.
Now, challenge is to bring the relevant parts together and logically separated into Documentation/BPF/ and bpf(2) man page. I think everything user API relevant would help most if it updates bpf(2) man page. That can be explanation of different map types, interaction with maps, quirks, etc.
Eventually also helper functions. Right now they're all documented in include/uapi/linux/bpf.h and that's okay as it ships along with the kernel code, so they're in sync. Eventually, there should be some more elaborate description of them, perhaps with tiny examples, in bpf(2) as well, since it's part of the uapi and stable (helpers themselves at least).
The Documentation/networking/filter.txt would need to be reworked a bit and split into pieces for Documentation/BPF/, so we keep that as a central place for the technical parts. Documentation/RCU/ is doing a great job at that, and I would like to see Documentation/BPF/ being as helpful for developers here. Part of that would be to add missing pieces from the various available sources mentioned above or elsewhere, so people can get a deeper understanding on internals beyond reading just man page for the user API description; explaining concepts, etc. Part of that can also be on the ELF loader bits, etc, no doubt.
On a quick glance on the remaining patches, it seems a mixture of quotes from the bpf(2) man page, filter.txt along with sample code explanation and lots of external links, part of them to https://git.kernel.org/cgit/linux/...,
http://lxr.free-electrons.com, http://man7.org, http://www.slideshare.net/,
https://github.com/torvalds/linux/..., etc. I would suggest that instead we should make Documentation/BPF/ eventually as self-contained as possible and the central place for the internals where others link to instead. It might take a bit, but I think definitely worth it.
Regarding patches, could you split the parts that should go to bpf(2) off and submit them to man page tree for inclusion? I think that would be incredibly helpful. And the remaining internals to start a Documentation/BPF/ directory, even if initially just small (but will grow over time of course)?
Thanks,
Daniel
Quoted 34 lines
[1] http://prototype-kernel.readthedocs.io/en/latest/bpf/index.html
[2] https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/DocumentationThanks to the following people, who have already reviewed and fixed earlier versions of this documentation on the IOvisor mailing-list:
Alexander Alemayhu <[email protected]>
Alexei Starovoitov <[email protected]>
Daniel Borkmann <[email protected]>
Quentin Monnet <[email protected]>---
Jesper Dangaard Brouer (4):
doc/bpf: start eBPF documentation tree bpf/ doc/bpf: document interacting with eBPF maps doc/bpf: describes the different types of eBPF maps available doc/bpf: describe BCC the BPF Compiler CollectionDocumentation/bpf/bcc_tool_chain.rst | 37 +++++ Documentation/bpf/ebpf_maps.rst | 256 +++++++++++++++++++++++++++++++++ Documentation/bpf/ebpf_maps_types.rst | 119 +++++++++++++++ Documentation/bpf/index.rst | 68 +++++++++ Documentation/index.rst | 1 5 files changed, 481 insertions(+)create mode 100644
Documentation/bpf/bcc_tool_chain.rstcreate mode 100644Documentation/bpf/ebpf_maps.rstcreate mode 100644Documentation/bpf/ebpf_maps_types.rstcreate mode 100644Documentation/bpf/index.rst--
Quoted 25 lines · On Tue, 07 Feb 2017 17:43:38 +0100 Daniel Borkmann <[email protected]> wrote:
Hi Jesper,
Quoted 6 lines · Jesper Dangaard Brouer
Question: What kernel tree should this go into???
If going through Jonathan Corbet, will it appear sooner here???
https://www.kernel.org/doc/html/latest/
If it will not appear sooner that way, then it's likely best to keep it in sync with the tree that takes eBPF code changes.For initial parts, I don't have a preference (Jonathan has though, so seems fine via docs tree then). If at some /later/ point in time features come in along with doc updates (similar to test case updates), probably best to route them via net-next.
Quoted 9 lines
This marks the beginning of user-facing developer documentation for using eBPF (extended Berkeley Packet Filter) as part of the kernel
Documentation/tree.This documentation is also available here[1], as an intermidiate quick way of prototyping and releasing the documentation. The autoriative and official version of the documentation is what gets included in the kernel tree. The docs at [2] will get updated based on what gets accepted after the standard peer-review kernel process.
Quoted 6 lines
Thanks for your effort of writing a doc. Some high-level comments on the set from my PoV first.
I think it's definitely the right direction to move everything BPF related to
Documentation/BPF/. Right now, there are a lot of different places with different kind of documentation, f.e.:
Agree that we need some in-kernel place to centralize bpf related documentation, as it is too scattered at the moment.
Quoted 6 lines
*
Documentation/networking/filter.txt
Covers some cBPF/eBPF internals, tooling, etc; mostly technical, historically the central spot for BPF documentation. "filter" in filter.txt is long obsolete name, but looks like various sites, talks, blogs, etc still link to it. (At best, we should keep the file saying that the doc moved toDocumentation/BPF/.)
Agree.
Quoted 3 lines
* bpf(2) man page
Has a good start, but right now is heavily behind the current user facing kernel code.
Yes, the man-page have proven to get out-of-sync. This is one of the reasons I prefer this in-kernel-tree documentation, as documentation can follow the patchset submission, instead of being something developers need to submit _after_ patches are accepted.
Quoted 12 lines
*
include/uapi/linux/bpf.h
Mostly relevant for helper function API description.* netdev conference slides/proceedings
Also contain mostly technical details on eBPF.* https://github.com/iovisor/bpf-docs
Non-exhaustive collection of various talks from different confs.* https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/
Even bigger and more complete list of documentation material.
Quoted 6 lines
* Various lwn articles ;), blog posts (f.e. from Brendan), etc.
Now, challenge is to bring the relevant parts together and logically separated into
Documentation/BPF/and bpf(2) man page. I think everything user API relevant would help most if it updates bpf(2) man page. That can be explanation of different map types, interaction with maps, quirks, etc.
Sorry, but I disagree. The man-page bpf(2) should only describe the bpf syscall. Details on map types should be documented in this documentation. Why, because this allow us to enforce documentation of a new map type is included together with the code submission (else it will never get documented).
Quoted 6 lines
Eventually also helper functions. Right now they're all documented in
include/uapi/linux/bpf.hand that's okay as it ships along with the kernel code, so they're in sync. Eventually, there should be some more elaborate description of them, perhaps with tiny examples, in bpf(2) as well, since it's part of the uapi and stable (helpers themselves at least).
IMHO descriptions for bpf helpers function does not belong in the man-page for the bpf(2) syscall. The bpf helpers are something that gets used by the ebpf program code which runs kernel side. (And again the same argument about introducing new will not get updated in the man page).
Quoted 10 lines
The
Documentation/networking/filter.txtwould need to be reworked a bit and split into pieces forDocumentation/BPF/, so we keep that as a central place for the technical parts.Documentation/RCU/is doing a great job at that, and I would like to seeDocumentation/BPF/being as helpful for developers here. Part of that would be to add missing pieces from the various available sources mentioned above or elsewhere, so people can get a deeper understanding on internals beyond reading just man page for the user API description; explaining concepts, etc. Part of that can also be on the ELF loader bits, etc, no doubt.
Agree
Quoted 10 lines
On a quick glance on the remaining patches, it seems a mixture of quotes from the bpf(2) man page, filter.txt along with sample code explanation and lots of external links, part of them to
https://git.kernel.org/cgit/linux/..., http://lxr.free-electrons.com,
http://man7.org, http://www.slideshare.net/,
https://github.com/torvalds/linux/..., etc. I would suggest that instead we should makeDocumentation/BPF/eventually as self-contained as possible and the central place for the internals where others link to instead. It might take a bit, but I think definitely worth it.
In time it can be more self-contained, but as of this writing, as you mention yourself, bpf doc resources are fragmented all over, thus the obviously doc reflect that with many links.
Quoted 5 lines
Regarding patches, could you split the parts that should go to bpf(2) off and submit them to man page tree for inclusion? I think that would be incredibly helpful. And the remaining internals to start a
Documentation/BPF/directory, even if initially just small (but will grow over time of course)?
As noted, I don't believe in populating the man page further is right direction, sorry, and I've already argued why.
Quoted 34 lines
Quoted 34 lines
[1] http://prototype-kernel.readthedocs.io/en/latest/bpf/index.html
[2]
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/DocumentationThanks to the following people, who have already reviewed and fixed earlier versions of this documentation on the IOvisor mailing-list:
Alexander Alemayhu <[email protected]>
Alexei Starovoitov <[email protected]>
Daniel Borkmann <[email protected]>
Quentin Monnet <[email protected]>---
Jesper Dangaard Brouer (4):
doc/bpf: start eBPF documentation tree bpf/ doc/bpf: document interacting with eBPF maps doc/bpf: describes the different types of eBPF maps available doc/bpf: describe BCC the BPF Compiler CollectionDocumentation/bpf/bcc_tool_chain.rst | 37 +++++ Documentation/bpf/ebpf_maps.rst | 256 +++++++++++++++++++++++++++++++++ Documentation/bpf/ebpf_maps_types.rst | 119 +++++++++++++++ Documentation/bpf/index.rst | 68 +++++++++ Documentation/index.rst | 1 5 files changed, 481insertions(+) create mode 100644
Documentation/bpf/bcc_tool_chain.rstcreate mode 100644Documentation/bpf/ebpf_maps.rstcreate mode 100644Documentation/bpf/ebpf_maps_types.rstcreate mode 100644Documentation/bpf/index.rst
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Daniel Borkmann
maintainerQuoted 76 lines · Jesper Dangaard Brouer
Quoted 25 lines · On Tue, 07 Feb 2017 17:43:38 +0100 Daniel Borkmann <[email protected]> wrote:
Hi Jesper,
Quoted 6 lines · Jesper Dangaard Brouer
Question: What kernel tree should this go into???
If going through Jonathan Corbet, will it appear sooner here???
https://www.kernel.org/doc/html/latest/
If it will not appear sooner that way, then it's likely best to keep it in sync with the tree that takes eBPF code changes.For initial parts, I don't have a preference (Jonathan has though, so seems fine via docs tree then). If at some /later/ point in time features come in along with doc updates (similar to test case updates), probably best to route them via net-next.
Quoted 9 lines
This marks the beginning of user-facing developer documentation for using eBPF (extended Berkeley Packet Filter) as part of the kernel
Documentation/tree.This documentation is also available here[1], as an intermidiate quick way of prototyping and releasing the documentation. The autoriative and official version of the documentation is what gets included in the kernel tree. The docs at [2] will get updated based on what gets accepted after the standard peer-review kernel process.
Quoted 6 lines
Thanks for your effort of writing a doc. Some high-level comments on the set from my PoV first.
I think it's definitely the right direction to move everything BPF related to
Documentation/BPF/. Right now, there are a lot of different places with different kind of documentation, f.e.:Agree that we need some in-kernel place to centralize bpf related documentation, as it is too scattered at the moment.
Quoted 6 lines
*
Documentation/networking/filter.txt
Covers some cBPF/eBPF internals, tooling, etc; mostly technical, historically the central spot for BPF documentation. "filter" in filter.txt is long obsolete name, but looks like various sites, talks, blogs, etc still link to it. (At best, we should keep the file saying that the doc moved toDocumentation/BPF/.)Agree.
Quoted 3 lines
* bpf(2) man page
Has a good start, but right now is heavily behind the current user facing kernel code.Yes, the man-page have proven to get out-of-sync. This is one of the reasons I prefer this in-kernel-tree documentation, as documentation can follow the patchset submission, instead of being something developers need to submit _after_ patches are accepted.
Quoted 12 lines
*
include/uapi/linux/bpf.h
Mostly relevant for helper function API description.* netdev conference slides/proceedings Also contain mostly technical details on eBPF.
* https://github.com/iovisor/bpf-docs
Non-exhaustive collection of various talks from different confs.* https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/
Even bigger and more complete list of documentation material.Quoted 6 lines
* Various lwn articles ;), blog posts (f.e. from Brendan), etc.
Now, challenge is to bring the relevant parts together and logically separated into
Documentation/BPF/and bpf(2) man page. I think everything user API relevant would help most if it updates bpf(2) man page. That can be explanation of different map types, interaction with maps, quirks, etc.
Quoted 5 lines
Sorry, but I disagree. The man-page bpf(2) should only describe the bpf syscall. Details on map types should be documented in this documentation. Why, because this allow us to enforce documentation of a new map type is included together with the code submission (else it will never get documented).
But essential part of the syscall is to create new maps, interact with them, etc so it's definitely relevant for the man page. The man page has a couple of FIXMEs in its source that Michael Kerrisk added in the course of reviewing and editing patches that were submitted; to give one example related to this:
[...]
Currently, the following values are supported for .IR map_type :
.in +4n
.nf
enum bpf_map_type {
BPF_MAP_TYPE_UNSPEC, /* Reserve 0 as invalid map type */
BPF_MAP_TYPE_HASH,
BPF_MAP_TYPE_ARRAY,
BPF_MAP_TYPE_PROG_ARRAY,
};
.fi
.in .I map_type
selects one of the available map implementations in the kernel.
.\" FIXME We need an explanation of why one might choose each of .\" these map implementations
[...]
Thus, I think it makes sense to address these points there instead of duplicating as a programmer guide in the kernel tree. Maps are not added as frequently as helper calls, and neither require writing long novels to document, so we could require submitters to send a patch there. When it comes to deeper kernel internals that would not be appropriate in man pages project, but need to be documented nevertheless (perhaps a guide on what is necessary to implement new map types), they could then go to the kernel documentation.
What I'm trying to say is that it makes sense to logically defragment these bits that are relevant for man-page (since user/prog developer specific) and for kernel doc tree (rather kernel developer specific), so there's less duplication of efforts eventually.
Quoted 7 lines
Quoted 6 lines
Eventually also helper functions. Right now they're all documented in
include/uapi/linux/bpf.hand that's okay as it ships along with the kernel code, so they're in sync. Eventually, there should be some more elaborate description of them, perhaps with tiny examples, in bpf(2) as well, since it's part of the uapi and stable (helpers themselves at least).
Quoted 5 lines
IMHO descriptions for bpf helpers function does not belong in the man-page for the bpf(2) syscall. The bpf helpers are something that gets used by the ebpf program code which runs kernel side. (And again the same argument about introducing new will not get updated in the man page).
But also here, the program runs in kernel space, sure, but people writing programs are not necessarily the same that write kernel code f.e. on netdev/lkml, or have a kernel source tree at hand. For helper functions, the documentation is in-sync and part of include/uapi/linux/bpf.h which gets shipped as kernel header anyway from distro side.
Man-pages get shipped as well, so they could contain additional infos related to helpers, f.e. what helpers are tied to which program type, etc; again, illustration from bpf.2 source:
[...]
.BR BPF_PROG_TYPE_KPROBE " (since Linux 4.1) .\" commit 2541517c32be2531e0da59dfd7efc1ce844644f5 [To be documented]
.\" FIXME Document this program type
.\" Describe allowed helper functions for this program type
.\" Describe bpf_context for this program type
.\"
.\" FIXME We need text here to describe 'kern_version' [...]
Quoted 73 lines
Quoted 10 lines
The
Documentation/networking/filter.txtwould need to be reworked a bit and split into pieces forDocumentation/BPF/, so we keep that as a central place for the technical parts.Documentation/RCU/is doing a great job at that, and I would like to seeDocumentation/BPF/being as helpful for developers here. Part of that would be to add missing pieces from the various available sources mentioned above or elsewhere, so people can get a deeper understanding on internals beyond reading just man page for the user API description; explaining concepts, etc. Part of that can also be on the ELF loader bits, etc, no doubt.Agree
Quoted 10 lines
On a quick glance on the remaining patches, it seems a mixture of quotes from the bpf(2) man page, filter.txt along with sample code explanation and lots of external links, part of them to
https://git.kernel.org/cgit/linux/..., http://lxr.free-electrons.com,
http://man7.org, http://www.slideshare.net/,
https://github.com/torvalds/linux/..., etc. I would suggest that instead we should makeDocumentation/BPF/eventually as self-contained as possible and the central place for the internals where others link to instead. It might take a bit, but I think definitely worth it.In time it can be more self-contained, but as of this writing, as you mention yourself, bpf doc resources are fragmented all over, thus the obviously doc reflect that with many links.
Quoted 5 lines
Regarding patches, could you split the parts that should go to bpf(2) off and submit them to man page tree for inclusion? I think that would be incredibly helpful. And the remaining internals to start a
Documentation/BPF/directory, even if initially just small (but will grow over time of course)?As noted, I don't believe in populating the man page further is right direction, sorry, and I've already argued why.
Quoted 34 lines
Quoted 34 lines
[1] http://prototype-kernel.readthedocs.io/en/latest/bpf/index.html
[2]
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/DocumentationThanks to the following people, who have already reviewed and fixed earlier versions of this documentation on the IOvisor mailing-list:
Alexander Alemayhu <[email protected]>
Alexei Starovoitov <[email protected]>
Daniel Borkmann <[email protected]>
Quentin Monnet <[email protected]>---
Jesper Dangaard Brouer (4):
doc/bpf: start eBPF documentation tree bpf/ doc/bpf: document interacting with eBPF maps doc/bpf: describes the different types of eBPF maps available doc/bpf: describe BCC the BPF Compiler CollectionDocumentation/bpf/bcc_tool_chain.rst | 37 +++++ Documentation/bpf/ebpf_maps.rst | 256 +++++++++++++++++++++++++++++++++ Documentation/bpf/ebpf_maps_types.rst | 119 +++++++++++++++ Documentation/bpf/index.rst | 68 +++++++++ Documentation/index.rst | 1 5 files changed, 481insertions(+) create mode 100644
Documentation/bpf/bcc_tool_chain.rstcreate mode 100644Documentation/bpf/ebpf_maps.rstcreate mode 100644Documentation/bpf/ebpf_maps_types.rstcreate mode 100644Documentation/bpf/index.rst