Document BPF_MAP_TYPE_ARRAY
Subsystems (2)
v7 cover Document BPF_MAP_TYPE_ARRAY
Add documentation for BPF_MAP_TYPE_ARRAY and BPF_MAP_TYPE_PERCPU_ARRAY variant, including kernel version introduced, usage and examples.
v6->v7:
- Remove 2^32 reference and reword paragraph
reported by Jiri Olsa and Daniel Borkmann
v5->v6:
- Rework sample code into individual snippets
- Grammar mods suggested by Bagas Sanjaja
v4->v5:
- Use formatting consistent with *_TYPE_HASH docs
- Dropped cgroup doc patch from this set
- Fix grammar and typos reported by Bagas Sanjaya
- Fix typo and version reported by Donald Hunter
- Update examples to be libbpf v1 compatible
v3->v4:
- fix doctest failure due to missing newline
v2->v3:
- wrap text to 80 chars and add newline at end of file
v1->v2:
- point to selftests for functional examples
- update examples to follow kernel style
- add docs for BPF_F_MMAPABLE
Dave Tucker (1):
bpf, docs: document BPF_MAP_TYPE_ARRAY
Documentation/bpf/map_array.rst | 232 ++++++++++++++++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 Documentation/bpf/map_array.rst
Signature · 1 line
2.35.1
PATCH v7 1/1 bpf, docs: document BPF_MAP_TYPE_ARRAY
From: Dave Tucker <[email protected]>
Add documentation for the BPF_MAP_TYPE_ARRAY including kernel version introduced, usage and examples. Also document BPF_MAP_TYPE_PERCPU_ARRAY which is similar.
--- Documentation/bpf/map_array.rst | 232 ++++++++++++++++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 Documentation/bpf/map_array.rst
Documentation/bpf/map_array.rst new file +232 -0
@@ -0,0 +1,232 @@ +.. SPDX-License-Identifier: GPL-2.0-only +.. Copyright (C) 2022 Red Hat, Inc. + +================================================ +BPF_MAP_TYPE_ARRAY and BPF_MAP_TYPE_PERCPU_ARRAY +================================================ + +.. note:: + - ``BPF_MAP_TYPE_ARRAY`` was introduced in kernel version 3.19 + - ``BPF_MAP_TYPE_PERCPU_ARRAY`` was introduced in version 4.6 + +``BPF_MAP_TYPE_ARRAY`` and ``BPF_MAP_TYPE_PERCPU_ARRAY`` provide generic array +storage. The key type is an unsigned 32-bit integer (4 bytes) and the map is +of constant size. The size of the array is defined in ``max_entries`` at +creation time. All array elements are pre-allocated and zero initialized when +created. ``BPF_MAP_TYPE_PERCPU_ARRAY`` uses a different memory region for each +CPU whereas ``BPF_MAP_TYPE_ARRAY`` uses the same memory region. The value +stored can be of any size, however, small values will be rounded up to 8 +bytes. + +Since kernel 5.5, memory mapping may be enabled for ``BPF_MAP_TYPE_ARRAY`` by +setting the flag ``BPF_F_MMAPABLE``. The map definition is page-aligned and +starts on the first page. Sufficient page-sized and page-aligned blocks of +memory are allocated to store all array values, starting on the second page, +which in some cases will result in over-allocation of memory. The benefit of +using this is increased performance and ease of use since userspace programs +would not be required to use helper functions to access and mutate data. + +Usage +===== + +.. c:function:: + void *bpf_map_lookup_elem(struct bpf_map *map, const void *key) + +Array elements can be retrieved using the ``bpf_map_lookup_elem()`` helper. +This helper returns a pointer into the array element, so to avoid data races +with userspace reading the value, the user must use primitives like +``__sync_fetch_and_add()`` when updating the value in-place. Access from +userspace uses the libbpf API of the same name. + +.. c:function:: + long bpf_map_update_elem(struct bpf_map *map, const void *key, const void *value, u64 flags) + +Array elements can also be added using the ``bpf_map_update_elem()`` helper or +libbpf API. + +``bpf_map_update_elem()`` returns 0 on success, or negative error in case of +failure. + +Since the array is of constant size, ``bpf_map_delete_elem()`` is not supported. +To clear an array element, you may use ``bpf_map_update_elem()`` to insert a +zero value to that index. + +Per CPU Array +------------- + +Values stored in ``BPF_MAP_TYPE_ARRAY`` can be accessed by multiple programs +across different CPUs. To restrict storage to a single CPU, you may use a +``BPF_MAP_TYPE_PERCPU_ARRAY``. + +When using a ``BPF_MAP_TYPE_PERCPU_ARRAY`` the ``bpf_map_update_elem()`` and +``bpf_map_lookup_elem()`` helpers automatically access the hash slot for the +current CPU. + +.. c:function:: + void *bpf_map_lookup_percpu_elem(struct bpf_map *map, const void *key, u32 cpu) + +The ``bpf_map_lookup_percpu_elem()`` helper can be used to lookup the array +value for a specific CPU. Returns value on success , or ``NULL`` if no entry was +found or ``cpu`` is invalid. + +Concurrency +----------- + +Since kernel version 5.1, the BPF infrastructure provides ``struct bpf_spin_lock`` +to synchronize access. + +Examples +======== + +Please see the ``tools/testing/selftests/bpf`` directory for functional +examples. The code samples below demonstrate API usage. + +Kernel BPF +---------- + +This snippet shows how to declare an array in a BPF program. + +.. code-block:: c + + struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 256); + } my_map SEC(".maps"); + + +This example BPF program shows how to access an array element. + +.. code-block:: c + + int bpf_prog(struct __sk_buff *skb) + { + int index = load_byte(skb, + ETH_HLEN + offsetof(struct iphdr, protocol)); + long *value; + + if (skb->pkt_type != PACKET_OUTGOING) + return 0; + + value = bpf_map_lookup_elem(&my_map, &index); + if (value) + __sync_fetch_and_add(value, skb->len); + + return 0; + } + +Userspace +--------- + +BPF_MAP_TYPE_ARRAY +~~~~~~~~~~~~~~~~~~ + +This snippet shows how to create an array, using ``bpf_map_create_opts`` to +set flags. + +.. code-block:: c + + #include <bpf/libbpf.h> + #include <bpf/bpf.h> + + int create_array() { + int fd; + LIBBPF_OPTS(bpf_map_create_opts, opts, .map_flags = BPF_F_MMAPABLE); + fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, + "example_array", /* name */ + sizeof(__u32), /* key size */ + sizeof(long), /* value size */ + 256, /* max entries */ + &opts); /* create opts */ + return fd; + } + +This snippet shows how to initialize the elements of an array. + +.. code-block:: c + + int initialize_array(int fd) { + __u32 i; + long value; + int ret; + + for (i = 0; i < 256; i++) { + value = i; + ret = bpf_map_update_elem(fd, &i, &value, BPF_ANY); + if (ret < 0) + return ret; + } + + return ret; + } + +This snippet shows how to retrieve an element value from an array. + +.. code-block:: c + + int lookup(int fd) { + __u32 index = 42; + long value; + int ret = bpf_map_lookup_elem(fd, &index, &value); + if (ret < 0) + return ret; + + /* use value here */ + assert(value == 42); + + return ret; + } + +BPF_MAP_TYPE_PERCPU_ARRAY +~~~~~~~~~~~~~~~~~~~~~~~~~ + +This snippet shows how to initialize the elements of a per CPU array. + +.. code-block:: c + + int initialize_array(int fd) { + int ncpus = libbpf_num_possible_cpus(); + long values[ncpus]; + __u32 i, j; + int ret; + + for (i = 0; i < 256 ; i++) { + for (j = 0; j < ncpus; j++) + values[j] = i; + ret = bpf_map_update_elem(fd, &i, &values, BPF_ANY); + if (ret < 0) + return ret; + } + + return ret; + } + +This snippet shows how to access the per CPU elements of an array value. + +.. code-block:: c + + int lookup(int fd) { + int ncpus = libbpf_num_possible_cpus(); + __u32 index = 42, j; + long values[ncpus]; + int ret = bpf_map_lookup_elem(fd, &index, &values); + if (ret < 0) + return ret; + + for (j = 0; j < ncpus; j++) { + /* Use per CPU value here */ + assert(values[j] == 42); + } + + return ret; + } + +Semantics +========= + +As shown in the example above, when accessing a ``BPF_MAP_TYPE_PERCPU_ARRAY`` +in userspace, each value is an array with ``ncpus`` elements. + +When calling ``bpf_map_update_elem()`` the flag ``BPF_NOEXIST`` can not be used +for these maps.
Signature · 1 line
2.35.1
Re: [PATCH bpf-next v7 1/1] bpf, docs: document BPF_MAP_TYPE_ARRAY
Quoted 29 lines · Donald Hunter
From: Dave Tucker <[email protected]>
Add documentation for the
BPF_MAP_TYPE_ARRAYincluding kernel version introduced, usage and examples. Also documentBPF_MAP_TYPE_PERCPU_ARRAYwhich is similar.Signed-off-by Dave Tucker <[email protected]>Signed-off-by Donald Hunter <[email protected]>--- Documentation/bpf/map_array.rst | 232 ++++++++++++++++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 Documentation/bpf/map_array.rstDocumentation/bpf/map_array.rst new file +9
-0@@ -0,0 +1,232 @@ +.. SPDX-License-Identifier: GPL-2.0-only +.. Copyright (C) 2022 Red Hat, Inc. + +================================================ +BPF_MAP_TYPE_ARRAY and BPF_MAP_TYPE_PERCPU_ARRAY +================================================ + +.. note:: + - ``BPF_MAP_TYPE_ARRAY`` was introduced in kernel version 3.19
Documentation/bpf/map_array.rst line 19
Quoted 10 lines
Documentation/bpf/map_array.rst new file +10
-0+ - ``BPF_MAP_TYPE_PERCPU_ARRAY`` was introduced in version 4.6 + +``BPF_MAP_TYPE_ARRAY`` and ``BPF_MAP_TYPE_PERCPU_ARRAY`` provide generic array +storage. The key type is an unsigned 32-bit integer (4 bytes) and the map is +of constant size. The size of the array is defined in ``max_entries`` at +creation time. All array elements are pre-allocated and zero initialized when +created. ``BPF_MAP_TYPE_PERCPU_ARRAY`` uses a different memory region for each +CPU whereas ``BPF_MAP_TYPE_ARRAY`` uses the same memory region. The value +stored can be of any size, however, small values will be rounded up to 8 +bytes.
Actually all values are rounded up to 8.
Maybe we should say that all array elements are aligned to 8 instead of values are rounded?
Because values_size=4 stays as 4 from bpf prog pov.
The progs cannot access bytes 5,6,7,8 though that memory is consumed.
Quoted 34 lines
Documentation/bpf/map_array.rst new file +34
-0+ +Since kernel 5.5, memory mapping may be enabled for ``BPF_MAP_TYPE_ARRAY`` by +setting the flag ``BPF_F_MMAPABLE``. The map definition is page-aligned and +starts on the first page. Sufficient page-sized and page-aligned blocks of +memory are allocated to store all array values, starting on the second page, +which in some cases will result in over-allocation of memory. The benefit of +using this is increased performance and ease of use since userspace programs +would not be required to use helper functions to access and mutate data. + +Usage +===== + +.. c:function:: + void *bpf_map_lookup_elem(struct bpf_map *map, const void *key) + +Array elements can be retrieved using the ``bpf_map_lookup_elem()`` helper. +This helper returns a pointer into the array element, so to avoid data races +with userspace reading the value, the user must use primitives like +``__sync_fetch_and_add()`` when updating the value in-place. Access from +userspace uses the libbpf API of the same name. + +.. c:function:: + long bpf_map_update_elem(struct bpf_map *map, const void *key, const void *value, u64 flags) + +Array elements can also be added using the ``bpf_map_update_elem()`` helper or +libbpf API. + +``bpf_map_update_elem()`` returns 0 on success, or negative error in case of +failure. + +Since the array is of constant size, ``bpf_map_delete_elem()`` is not supported. +To clear an array element, you may use ``bpf_map_update_elem()`` to insert a +zero value to that index. +
Documentation/bpf/map_array.rst line 63
Quoted 10 lines
Documentation/bpf/map_array.rst new file +10
-0+Per CPU Array +------------- + +Values stored in ``BPF_MAP_TYPE_ARRAY`` can be accessed by multiple programs +across different CPUs. To restrict storage to a single CPU, you may use a +``BPF_MAP_TYPE_PERCPU_ARRAY``. + +When using a ``BPF_MAP_TYPE_PERCPU_ARRAY`` the ``bpf_map_update_elem()`` and +``bpf_map_lookup_elem()`` helpers automatically access the hash slot for the +current CPU.
hash slot?
the copy paste went wrong? :)
Quoted 34 lines
Documentation/bpf/map_array.rst new file +34
-0+ +.. c:function:: + void *bpf_map_lookup_percpu_elem(struct bpf_map *map, const void *key, u32 cpu) + +The ``bpf_map_lookup_percpu_elem()`` helper can be used to lookup the array +value for a specific CPU. Returns value on success , or ``NULL`` if no entry was +found or ``cpu`` is invalid. + +Concurrency +----------- + +Since kernel version 5.1, the BPF infrastructure provides ``struct bpf_spin_lock`` +to synchronize access. + +Examples +======== + +Please see the ``tools/testing/selftests/bpf`` directory for functional +examples. The code samples below demonstrate API usage. + +Kernel BPF +---------- + +This snippet shows how to declare an array in a BPF program. + +.. code-block:: c + + struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, u32); + __type(value, long); + __uint(max_entries, 256); + } my_map SEC(".maps"); +
Documentation/bpf/map_array.rst line 107
Quoted 10 lines
Documentation/bpf/map_array.rst new file +10
-0+ +This example BPF program shows how to access an array element. + +.. code-block:: c + + int bpf_prog(struct __sk_buff *skb) + { + int index = load_byte(skb, + ETH_HLEN + offsetof(struct iphdr, protocol)); + long *value;
Please avoid using deprecated instructions like load_byte in examples.
Quoted 18 lines
Documentation/bpf/map_array.rst new file +18
-0+ + if (skb->pkt_type != PACKET_OUTGOING) + return 0; + + value = bpf_map_lookup_elem(&my_map, &index); + if (value) + __sync_fetch_and_add(value, skb->len); + + return 0; + } + +Userspace +--------- + +BPF_MAP_TYPE_ARRAY +~~~~~~~~~~~~~~~~~~ + +This snippet shows how to create an array, using ``bpf_map_create_opts`` to
Documentation/bpf/map_array.rst line 135
Quoted 10 lines
Documentation/bpf/map_array.rst new file +10
-0+set flags. + +.. code-block:: c + + #include <bpf/libbpf.h> + #include <bpf/bpf.h> + + int create_array() { + int fd; + LIBBPF_OPTS(bpf_map_create_opts, opts, .map_flags = BPF_F_MMAPABLE);
Add empty line pls.
Quoted 26 lines
Documentation/bpf/map_array.rst new file +26
-0+ fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, + "example_array", /* name */ + sizeof(__u32), /* key size */ + sizeof(long), /* value size */ + 256, /* max entries */ + &opts); /* create opts */ + return fd; + } + +This snippet shows how to initialize the elements of an array. + +.. code-block:: c + + int initialize_array(int fd) { + __u32 i; + long value; + int ret; + + for (i = 0; i < 256; i++) { + value = i; + ret = bpf_map_update_elem(fd, &i, &value, BPF_ANY); + if (ret < 0) + return ret; + } + + return ret;
Documentation/bpf/map_array.rst line 171
Quoted 10 lines
Documentation/bpf/map_array.rst new file +10
-0+ } + +This snippet shows how to retrieve an element value from an array. + +.. code-block:: c + + int lookup(int fd) { + __u32 index = 42; + long value; + int ret = bpf_map_lookup_elem(fd, &index, &value);
Empty line pls.
Or better yet do 'int ret;'
and ret = bpf_map... on a separate line.
Quoted 32 lines
Documentation/bpf/map_array.rst new file +32
-0+ if (ret < 0) + return ret; + + /* use value here */ + assert(value == 42); + + return ret; + } + +BPF_MAP_TYPE_PERCPU_ARRAY +~~~~~~~~~~~~~~~~~~~~~~~~~ + +This snippet shows how to initialize the elements of a per CPU array. + +.. code-block:: c + + int initialize_array(int fd) { + int ncpus = libbpf_num_possible_cpus(); + long values[ncpus]; + __u32 i, j; + int ret; + + for (i = 0; i < 256 ; i++) { + for (j = 0; j < ncpus; j++) + values[j] = i; + ret = bpf_map_update_elem(fd, &i, &values, BPF_ANY); + if (ret < 0) + return ret; + } + + return ret; + }
Documentation/bpf/map_array.rst line 213
Quoted 10 lines
Documentation/bpf/map_array.rst new file +10
-0+ +This snippet shows how to access the per CPU elements of an array value. + +.. code-block:: c + + int lookup(int fd) { + int ncpus = libbpf_num_possible_cpus(); + __u32 index = 42, j; + long values[ncpus]; + int ret = bpf_map_lookup_elem(fd, &index, &values);
same here.
Quoted 22 lines
Documentation/bpf/map_array.rst new file +19
-1+ if (ret < 0) + return ret; + + for (j = 0; j < ncpus; j++) { + /* Use per CPU value here */ + assert(values[j] == 42); + } + + return ret; + } + +Semantics +========= + +As shown in the example above, when accessing a ``BPF_MAP_TYPE_PERCPU_ARRAY`` +in userspace, each value is an array with ``ncpus`` elements. + +When calling ``bpf_map_update_elem()`` the flag ``BPF_NOEXIST`` can not be used +for these maps. --2.35.1
Re: [PATCH bpf-next v7 1/1] bpf, docs: document BPF_MAP_TYPE_ARRAY
Quoted 7 lines · Alexei Starovoitov
On Fri, Oct 7, 2022 at 9:39 AM Donald Hunter <[email protected]> wrote:
Actually all values are rounded up to 8.
Maybe we should say that all array elements are aligned to 8 instead of values are rounded?
Because values_size=4 stays as 4 from bpf prog pov.
The progs cannot access bytes 5,6,7,8 though that memory is consumed.
Agreed, I will reword to mention alignment instead of rounding.
Quoted 6 lines
Documentation/bpf/map_array.rst line 63
Quoted 3 lines
Documentation/bpf/map_array.rst new file +3
-0+When using a ``BPF_MAP_TYPE_PERCPU_ARRAY`` the ``bpf_map_update_elem()`` and +``bpf_map_lookup_elem()`` helpers automatically access the hash slot for the +current CPU.hash slot?
the copy paste went wrong? :)
Good catch.
Quoted 7 lines
Documentation/bpf/map_array.rst line 107
Quoted 5 lines
Documentation/bpf/map_array.rst new file +5
-0+ int bpf_prog(struct __sk_buff *skb) + { + int index = load_byte(skb, + ETH_HLEN + offsetof(struct iphdr, protocol)); + long *value;Please avoid using deprecated instructions like
load_bytein examples.
Will rewrite to use bpf_skb_load_bytes.
Quoted 5 lines
Documentation/bpf/map_array.rst line 135
Quoted 3 lines
Documentation/bpf/map_array.rst new file +3
-0+ int create_array() { + int fd; + LIBBPF_OPTS(bpf_map_create_opts, opts, .map_flags = BPF_F_MMAPABLE);Add empty line pls.
Ack.
Quoted 8 lines
Documentation/bpf/map_array.rst line 171
Quoted 4 lines
Documentation/bpf/map_array.rst new file +4
-0+ int lookup(int fd) { + __u32 index = 42; + long value; + int ret = bpf_map_lookup_elem(fd, &index, &value);Empty line pls.
Or better yet do 'int ret;'
and ret =bpf_map... on a separate line.
Ack.
Quoted 7 lines
Documentation/bpf/map_array.rst line 213
Quoted 5 lines
Documentation/bpf/map_array.rst new file +5
-0+ int lookup(int fd) { + int ncpus = libbpf_num_possible_cpus(); + __u32 index = 42, j; + long values[ncpus]; + int ret = bpf_map_lookup_elem(fd, &index, &values);same here.
Ack. Thansk for the review!