bpf: cgroup local storage
Subsystems (8)
cover bpf: cgroup local storage
This patchset implements cgroup local storage for bpf programs. The main idea is to provide a fast accessible memory for storing various per-cgroup data, e.g. number of transmitted packets.
Cgroup local storage looks as a special type of map for userspace, and is accessible using generic bpf maps API for reading and updating of the data. The (cgroup inode id, attachment type) pair is used as a map key.
A user can't create new entries or destroy existing entries; it happens automatically when a user attaches/detaches a bpf program to a cgroup.
Quoted 1 line
From a bpf program's point of view, cgroup storage is accessible
without lookup using the special get_local_storage() helper function. It takes a map fd as an argument. It always returns a valid pointer to the corresponding memory area.
To implement such a lookup-free access a pointer to the cgroup storage is saved for an attachment of a bpf program to a cgroup, if required by the program. Before running the program, it's saved in a special global per-cpu variable, which is accessible from the get_local_storage() helper.
This patchset implement only cgroup local storage, however the API is intentionally made extensible to support other local storage types
further: e.g. thread local storage, socket local storage, etc.
Patch (1) adds an ability to charge bpf maps for consuming memory dynamically.
Patch (2) introduces cgroup storage maps.
Patch (3) implements a mechanism to pass cgroup storage pointer to a bpf program.
Patch (4) implements allocation/releasing of cgroup local storage on attaching/detaching of a bpf program to/from a cgroup. Patch (5) extends bpf_prog_array to store cgroup storage pointers. Patch (6) introduces BPF_PTR_TO_MAP_VALUE, required to skip non-necessary NULL-check in bpf programs.
Patch (7) disables creation of maps of cgroup storage maps. Patch (8) introduces the get_local_storage() helper. Patch (9) syncs bpf.h to tools/.
Patch (10) adds cgroup storage maps support to bpftool. Patch (11) adds support for testing programs which are using cgroup storage without actually attaching them to cgroups. Patches (12), (13) and (14) are adding necessary tests.
Roman Gushchin (14):
bpf: add ability to charge bpf maps memory dynamically
bpf: introduce cgroup storage maps
bpf: pass a pointer to a cgroup storage using pcpu variable
bpf: allocate cgroup storage entries on attaching bpf programs
bpf: extend bpf_prog_array to store pointers to the cgroup storage bpf/verifier: introduce BPF_PTR_TO_MAP_VALUE
bpf: don't allow create maps of cgroup local storages
bpf: introduce the bpf_get_local_storage() helper function
bpf: sync bpf.h to tools/
bpftool: add support for CGROUP_STORAGE maps bpf/test_run: support cgroup local storage selftests/bpf: add verifier cgroup storage tests selftests/bpf: add a cgroup storage test
samples/bpf: extend test_cgrp2_attach2 test to use cgroup storage
include/linux/bpf-cgroup.h | 53 ++++ include/linux/bpf.h | 25 +- include/linux/bpf_types.h | 3 + include/uapi/linux/bpf.h | 19 +- kernel/bpf/Makefile | 1 + kernel/bpf/cgroup.c | 54 +++- kernel/bpf/core.c | 76 ++--- kernel/bpf/helpers.c | 20 ++ kernel/bpf/local_storage.c | 369 ++++++++++++++++++++++ kernel/bpf/map_in_map.c | 3 +- kernel/bpf/syscall.c | 53 +++- kernel/bpf/verifier.c | 38 ++- net/bpf/test_run.c | 13 +- net/core/filter.c | 23 +- samples/bpf/test_cgrp2_attach2.c | 27 +- tools/bpf/bpftool/map.c | 1 + tools/include/uapi/linux/bpf.h | 9 +- tools/testing/selftests/bpf/Makefile | 4 +- tools/testing/selftests/bpf/bpf_helpers.h | 2 + tools/testing/selftests/bpf/test_cgroup_storage.c | 130 ++++++++ tools/testing/selftests/bpf/test_verifier.c | 123 +++++++- 21 files changed, 965 insertions(+), 81 deletions(-) create mode 100644 kernel/bpf/local_storage.c create mode 100644 tools/testing/selftests/bpf/test_cgroup_storage.c
Signature · 1 line
2.14.4
PATCH 1/14 bpf: add ability to charge bpf maps memory dynamically
This commits extends existing bpf maps memory charging API to support dynamic charging/uncharging.
This is required to account memory used by maps,
if all entries are created dynamically after
the map initialization.
--- include/linux/bpf.h | 2 ++ kernel/bpf/syscall.c | 53 +++++++++++++++++++++++++++++++++++++--------------- 2 files changed, 40 insertions(+), 15 deletions(-)
include/linux/bpf.h +2 -0
@@ -434,6 +434,8 @@ struct bpf_map * __must_check bpf_map_inc(struct bpf_map *map, bool uref);void bpf_map_put_with_uref (struct bpf_map *map);void bpf_map_put (struct bpf_map *map);int bpf_map_precharge_memlock (u32 pages); +int bpf_map_charge_memlock (struct bpf_map *map,u32 pages); +void bpf_map_uncharge_memlock (struct bpf_map *map,u32 pages);void *bpf_map_area_alloc (size_t size,int numa_node);void bpf_map_area_free (void *base);void bpf_map_init_from_attr (struct bpf_map *map,union bpf_attr *attr);
kernel/bpf/syscall.c +38 -15
91 lines not on this page · raw message
Signature · 1 line
2.14.4
PATCH 2/14 bpf: introduce cgroup storage maps
In reply to Roman GushchinThis commit introduces BPF_MAP_TYPE_CGROUP_STORAGE maps:
a special type of maps which are implementing the cgroup storage.
Quoted 1 line
From the userspace point of view it's almost a generic
hash map with the (cgroup inode id, attachment type) pair used as a key.
The only difference is that some operations are restricted:
1) a user can't create new entries,
2) a user can't remove existing entries.
The lookup from userspace is o(log(n)).
--- include/linux/bpf-cgroup.h | 38 +++++ include/linux/bpf.h | 1 + include/linux/bpf_types.h | 3 + include/uapi/linux/bpf.h | 6 + kernel/bpf/Makefile | 1 + kernel/bpf/local_storage.c | 367 +++++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 12 ++ 7 files changed, 428 insertions(+) create mode 100644 kernel/bpf/local_storage.c
include/linux/bpf-cgroup.h +38 -0
@@ -3,19 +3,39 @@#define _BPF_CGROUP_H#include <linux/jump_label.h> +#include <linux/rbtree.h> #include <uapi/linux/bpf.h> struct sock ;struct sockaddr ;struct cgroup ;struct sk_buff ; +struct bpf_map ; +struct bpf_prog ;struct bpf_sock_ops_kern ; +struct bpf_cgroup_storage ; #ifdef CONFIG_CGROUP_BPF extern struct static_key_false cgroup_bpf_enabled_key; #define cgroup_bpf_enabled static_branch_unlikely(&cgroup_bpf_enabled_key) +struct bpf_cgroup_storage_map; + +struct bpf_storage_buffer { + struct rcu_head rcu; + char data[0]; +}; + +struct bpf_cgroup_storage { + struct bpf_storage_buffer *buf; + struct bpf_cgroup_storage_map *map; + struct bpf_cgroup_storage_key key; + struct list_head list; + struct rb_node node; + struct rcu_head rcu; +}; + struct bpf_prog_list { struct list_head node; struct bpf_prog *prog;
@@ -76,6 +96,15 @@ int __cgroup_bpf_run_filter_sock_ops(struct sock *sk,int __cgroup_bpf_check_dev_permission (short dev_type,u32 major,u32 minor,short access,enum bpf_attach_type type); +struct bpf_cgroup_storage *bpf_cgroup_storage_alloc (struct bpf_prog *prog); +void bpf_cgroup_storage_free (struct bpf_cgroup_storage *storage); +void bpf_cgroup_storage_link (struct bpf_cgroup_storage *storage, +struct cgroup *cgroup, +enum bpf_attach_type type); +void bpf_cgroup_storage_unlink (struct bpf_cgroup_storage *storage); +int bpf_cgroup_storage_assign (struct bpf_prog *prog,struct bpf_map *map); +void bpf_cgroup_storage_release (struct bpf_prog *prog,struct bpf_map *map); +/* Wrappers for __cgroup_bpf_run_filter_skb() guarded by cgroup_bpf_enabled. */ #define BPF_CGROUP_RUN_PROG_INET_INGRESS (sk, skb) \ ({ \
@@ -194,6 +223,15 @@ struct cgroup_bpf {};static inline void cgroup_bpf_put (struct cgroup *cgrp) {}static inline int cgroup_bpf_inherit (struct cgroup *cgrp) {return 0 ; } +static inline int bpf_cgroup_storage_assign (struct bpf_prog *prog, +struct bpf_map *map) {return 0 ; } +static inline void bpf_cgroup_storage_release (struct bpf_prog *prog, +struct bpf_map *map) {} +static inline struct bpf_cgroup_storage *bpf_cgroup_storage_alloc ( +struct bpf_prog *prog) {return 0 ; } +static inline void bpf_cgroup_storage_free ( +struct bpf_cgroup_storage *storage) {} +#define cgroup_bpf_enabled (0)#define BPF_CGROUP_PRE_CONNECT_ENABLED (sk) (0)#define BPF_CGROUP_RUN_PROG_INET_INGRESS (sk,skb) ({ 0; })
include/linux/bpf.h +1 -0
@@ -281,6 +281,7 @@ struct bpf_prog_aux {struct bpf_prog *prog;struct user_struct *user;u64 load_time;/* ns since boottime */ +struct bpf_map *cgroup_storage;char name[BPF_OBJ_NAME_LEN ];#ifdef CONFIG_SECURITY void *security;
include/linux/bpf_types.h +3 -0
@@ -37,6 +37,9 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_PERF_EVENT_ARRAY, perf_event_array_map_ops)#ifdef CONFIG_CGROUPS BPF_MAP_TYPE(BPF_MAP_TYPE_CGROUP_ARRAY, cgroup_array_map_ops)#endif +#ifdef CONFIG_CGROUP_BPF +BPF_MAP_TYPE(BPF_MAP_TYPE_CGROUP_STORAGE, cgroup_storage_map_ops) +#endif BPF_MAP_TYPE(BPF_MAP_TYPE_HASH, htab_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_PERCPU_HASH, htab_percpu_map_ops)BPF_MAP_TYPE (BPF_MAP_TYPE_LRU_HASH , htab_lru_map_ops)
include/uapi/linux/bpf.h +6 -0
@@ -75,6 +75,11 @@ struct bpf_lpm_trie_key {__u8 data[0 ];/* Arbitrary size */ }; +struct bpf_cgroup_storage_key { +__u64 cgroup_inode_id;/* cgroup inode id */ +__u32 attach_type;/* program attach type */ +}; +/* BPF syscall commands, see bpf(2) man-page for details. */ enum bpf_cmd {BPF_MAP_CREATE ,
@@ -120,6 +125,7 @@ enum bpf_map_type {BPF_MAP_TYPE_CPUMAP ,BPF_MAP_TYPE_XSKMAP ,BPF_MAP_TYPE_SOCKHASH , + BPF_MAP_TYPE_CGROUP_STORAGE, }; enum bpf_prog_type {
kernel/bpf/Makefile +1 -0
@@ -3,6 +3,7 @@ obj-y := core.o obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o +obj-$(CONFIG_BPF_SYSCALL) += local_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o obj-$(CONFIG_BPF_SYSCALL) += btf.o ifeq ($(CONFIG_NET),y)
kernel/bpf/local_storage.c new file +367 -0
367 lines not on this page · raw message
kernel/bpf/verifier.c +12 -0
@@ -5140,6 +5140,14 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env)return -E2BIG ; } +if (map->map_type ==BPF_MAP_TYPE_CGROUP_STORAGE && +bpf_cgroup_storage_assign (env->prog, map)) { +verbose (env, +"only one cgroup storage is allowed\n" ); +fdput (f); +return -EBUSY ; + } +/* hold the map. If the program is rejected by verifier, * the map will be released by release_maps() or it * will be used by the valid program until it's unloaded
@@ -5148,6 +5156,10 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env) map =bpf_map_inc (map, false);if (IS_ERR (map)) {fdput (f); +if (map->map_type == +BPF_MAP_TYPE_CGROUP_STORAGE ) +bpf_cgroup_storage_release (env->prog, + map);return PTR_ERR (map); } env->used_maps[env->used_map_cnt++] = map;
Signature · 1 line
2.14.4
PATCH 3/14 bpf: pass a pointer to a cgroup storage using pcpu variable
In reply to Roman GushchinThis commit introduces the bpf_cgroup_storage_set() helper, which will be used to pass a pointer to a cgroup storage to the bpf helper.
--- include/linux/bpf-cgroup.h | 14 ++++++++++++++ kernel/bpf/local_storage.c | 2 ++ 2 files changed, 16 insertions(+)
include/linux/bpf-cgroup.h +14 -0
@@ -20,6 +20,8 @@ struct bpf_cgroup_storage;extern struct static_key_false cgroup_bpf_enabled_key;#define cgroup_bpf_enabled static_branch_unlikely(&cgroup_bpf_enabled_key) +DECLARE_PER_CPU(void*, bpf_cgroup_storage); +struct bpf_cgroup_storage_map ;struct bpf_storage_buffer {
@@ -96,6 +98,17 @@ int __cgroup_bpf_run_filter_sock_ops(struct sock *sk,int __cgroup_bpf_check_dev_permission (short dev_type,u32 major,u32 minor,short access,enum bpf_attach_type type); +static inline void bpf_cgroup_storage_set (struct bpf_cgroup_storage *storage) +{ +struct bpf_storage_buffer *buf; + +if (!storage) +return ; + + buf =rcu_dereference (storage->buf); +this_cpu_write (bpf_cgroup_storage, &buf->data[0 ]); +} +struct bpf_cgroup_storage *bpf_cgroup_storage_alloc (struct bpf_prog *prog);void bpf_cgroup_storage_free (struct bpf_cgroup_storage *storage); void bpf_cgroup_storage_link(struct bpf_cgroup_storage *storage,
@@ -223,6 +236,7 @@ struct cgroup_bpf {};static inline void cgroup_bpf_put (struct cgroup *cgrp) {}static inline int cgroup_bpf_inherit (struct cgroup *cgrp) {return 0 ; } +static inline void bpf_cgroup_storage_set (struct bpf_cgroup_storage *storage) {}static inline int bpf_cgroup_storage_assign (struct bpf_prog *prog,struct bpf_map *map) {return 0 ; } static inline void bpf_cgroup_storage_release(struct bpf_prog *prog,
kernel/bpf/local_storage.c +2 -0
@@ -7,6 +7,8 @@#include <linux/rbtree.h> #include <linux/slab.h> +DEFINE_PER_CPU(void*, bpf_cgroup_storage); +#ifdef CONFIG_CGROUP_BPF struct bpf_cgroup_storage_map {
Signature · 1 line
2.14.4
PATCH 4/14 bpf: allocate cgroup storage entries on attaching bpf programs
In reply to Roman GushchinIf a bpf program is using cgroup local storage, allocate
a bpf_cgroup_storage structure automatically on attaching the program to a cgroup and save the pointer into the corresponding bpf_prog_list entry.
Analogically, release the cgroup local storage on detaching of the bpf program.
--- include/linux/bpf-cgroup.h | 1 + kernel/bpf/cgroup.c | 28 ++++++++++++++++++++++++++-- 2 files changed, 27 insertions(+), 2 deletions(-)
include/linux/bpf-cgroup.h +1 -0
@@ -41,6 +41,7 @@ struct bpf_cgroup_storage {struct bpf_prog_list {struct list_head node;struct bpf_prog *prog; +struct bpf_cgroup_storage *storage; };struct bpf_prog_array ;
kernel/bpf/cgroup.c +26 -2
@@ -34,6 +34,8 @@ void cgroup_bpf_put(struct cgroup *cgrp)list_for_each_entry_safe (pl, tmp, progs, node) {list_del (&pl->node);bpf_prog_put (pl->prog); +bpf_cgroup_storage_unlink (pl->storage); +bpf_cgroup_storage_free (pl->storage);kfree (pl);static_branch_dec (&cgroup_bpf_enabled_key); }
@@ -189,6 +191,7 @@ int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog, {struct list_head *progs = &cgrp->bpf.progs[type];struct bpf_prog *old_prog =NULL ; +struct bpf_cgroup_storage *storage, *old_storage =NULL ;struct cgroup_subsys_state *css;struct bpf_prog_list *pl;bool pl_was_allocated;
@@ -211,6 +214,10 @@ int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog,if (prog_list_length (progs) >=BPF_CGROUP_MAX_PROGS )return -E2BIG ; + storage =bpf_cgroup_storage_alloc (prog); +if (IS_ERR (storage)) +return -ENOMEM ; +if (flags &BPF_F_ALLOW_MULTI ) {list_for_each_entry (pl, progs, node)if (pl->prog == prog)
@@ -218,24 +225,33 @@ int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog,return -EINVAL ; pl =kmalloc (sizeof (*pl),GFP_KERNEL ); -if (!pl) +if (!pl) { +bpf_cgroup_storage_free (storage);return -ENOMEM ; + } + pl_was_allocated = true; pl->prog = prog; + pl->storage = storage;list_add_tail (&pl->node, progs); }else {if (list_empty (progs)) { pl =kmalloc (sizeof (*pl),GFP_KERNEL ); -if (!pl) +if (!pl) { +bpf_cgroup_storage_free (storage);return -ENOMEM ; + } pl_was_allocated = true;list_add_tail (&pl->node, progs); }else { pl =list_first_entry (progs,typeof (*pl), node); old_prog = pl->prog; + old_storage = pl->storage; +bpf_cgroup_storage_unlink (old_storage); pl_was_allocated = false; } pl->prog = prog; + pl->storage = storage; } cgrp->bpf.flags[type] = flags;
@@ -258,10 +274,13 @@ int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog, }static_branch_inc (&cgroup_bpf_enabled_key); +if (old_storage) +bpf_cgroup_storage_free (old_storage);if (old_prog) {bpf_prog_put (old_prog);static_branch_dec (&cgroup_bpf_enabled_key); } +bpf_cgroup_storage_link (storage, cgrp, type);return 0 ; cleanup:
@@ -277,6 +296,9 @@ int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog,/* and cleanup the prog list */ pl->prog = old_prog; +bpf_cgroup_storage_free (pl->storage); + pl->storage = old_storage; +bpf_cgroup_storage_link (old_storage, cgrp, type);if (pl_was_allocated) {list_del (&pl->node);kfree (pl);
@@ -357,6 +379,8 @@ int __cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,/* now can actually delete it from this cgroup list */ list_del (&pl->node); +bpf_cgroup_storage_unlink (pl->storage); +bpf_cgroup_storage_free (pl->storage);kfree (pl);if (list_empty (progs))/* last program was detached, reset flags to zero */
Signature · 1 line
2.14.4
PATCH 5/14 bpf: extend bpf_prog_array to store pointers to the cgroup storage
In reply to Roman GushchinThis patch converts bpf_prog_array from an array of prog pointers to the array of struct bpf_prog_array_item elements.
This allows to save a cgroup storage pointer for each bpf program efficiently attached to a cgroup.
--- include/linux/bpf.h | 19 +++++++++----- kernel/bpf/cgroup.c | 24 ++++++++++------- kernel/bpf/core.c | 76 +++++++++++++++++++++++++++-------------------------- 3 files changed, 66 insertions(+), 53 deletions(-)
include/linux/bpf.h +13 -6
@@ -348,9 +348,14 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,* The 'struct bpf_prog_array *' should only be replaced with xchg() * since other cpus are walking the array of pointers in parallel. */ +struct bpf_prog_array_item { +struct bpf_prog *prog; +struct bpf_cgroup_storage *cgroup_storage; +}; +struct bpf_prog_array {struct rcu_head rcu; -struct bpf_prog *progs[0 ]; +struct bpf_prog_array_item items[0 ]; };struct bpf_prog_array __rcu *bpf_prog_array_alloc (u32 prog_cnt,gfp_t flags);
@@ -371,7 +376,8 @@ int bpf_prog_array_copy(struct bpf_prog_array __rcu *old_array,#define __BPF_PROG_RUN_ARRAY (array, ctx, func, check_non_null) \ ({ \ - struct bpf_prog **_prog, *__prog; \ + struct bpf_prog_array_item *_item; \ + struct bpf_prog *_prog; \ struct bpf_prog_array *_array; \ u32 _ret = 1; \ preempt_disable(); \
@@ -379,10 +385,11 @@ int bpf_prog_array_copy(struct bpf_prog_array __rcu *old_array, _array =rcu_dereference (array); \if (unlikely (check_non_null && !_array))\ goto _out; \ - _prog = _array->progs; \ -while ((__prog =READ_ONCE (*_prog))) { \ - _ret &=func (__prog, ctx); \ - _prog++; \ + _item = &_array->items[0 ]; \ +while ((_prog =READ_ONCE (_item->prog))) { \ +bpf_cgroup_storage_set (_item->cgroup_storage); \ + _ret &=func (_prog, ctx); \ + _item++; \ } \ _out: \rcu_read_unlock (); \
kernel/bpf/cgroup.c +14 -10
@@ -117,16 +117,20 @@ static int compute_effective_progs(struct cgroup *cgrp, cnt =0 ; p = cgrp;do { -if (cnt ==0 || (p->bpf.flags[type] &BPF_F_ALLOW_MULTI )) -list_for_each_entry (pl, - &p->bpf.progs[type], node) { -if (!pl->prog) -continue ; -rcu_dereference_protected (progs,1 )-> - progs[cnt++] = pl->prog; - } - p =cgroup_parent (p); - }while (p); +if (cnt >0 && !(p->bpf.flags[type] &BPF_F_ALLOW_MULTI )) +continue ; + +list_for_each_entry (pl, &p->bpf.progs[type], node) { +if (!pl->prog) +continue ; + +rcu_dereference_protected (progs,1 )-> + items[cnt].prog = pl->prog; +rcu_dereference_protected (progs,1 )-> + items[cnt].cgroup_storage = pl->storage; + cnt++; + } + }while ((p =cgroup_parent (p))); *array = progs;return 0 ;
kernel/bpf/core.c +39 -37
163 lines not on this page · raw message
Signature · 1 line
2.14.4
kbuild test robot
build botRe: [PATCH bpf-net 05/14] bpf: extend bpf_prog_array to store pointers to the cgroup storage
Thank you for the patch! Perhaps something to improve:
Hi Roman,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on bpf-next/master]
[also build test WARNING on v4.18-rc2]
[cannot apply to next-20180628]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Roman-Gushchin/bpf-cgroup-local-storage/20180629-035104 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master reproduce: # apt-get install sparse make ARCH=x86_64 allmodconfig make C=1 CF=-D__CHECK_ENDIAN__
sparse warnings: (new ones prefixed by >>)
include/linux/filter.h:632:16: sparse: expression using sizeof(void) kernel/bpf/core.c:603:16: sparse: expression using sizeof(void)
kernel/bpf/core.c:1572:31: sparse: incorrect type in return expression (different address spaces) @@ expected struct bpf_prog_array [noderef] <asn:4>* @@ got sn:4>* @@
kernel/bpf/core.c:1572:31: expected struct bpf_prog_array [noderef] <asn:4>*
kernel/bpf/core.c:1572:31: got void *
kernel/bpf/core.c:1577:17: sparse: incorrect type in return expression (different address spaces) @@ expected struct bpf_prog_array [noderef] <asn:4>* @@ got rray [noderef] <asn:4>* @@
kernel/bpf/core.c:1577:17: expected struct bpf_prog_array [noderef] <asn:4>*
kernel/bpf/core.c:1577:17: got struct bpf_prog_array *<noident>
kernel/bpf/core.c:1585:9: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct callback_head *head @@ got struct callback_hstruct callback_head *head @@
kernel/bpf/core.c:1585:9: expected struct callback_head *head
kernel/bpf/core.c:1585:9: got struct callback_head [noderef] <asn:4>*<noident>
>> kernel/bpf/core.c:1610:16: sparse: incompatible types in comparison expression (different address spaces) include/linux/slab.h:631:13: sparse: undefined identifier '__builtin_mul_overflow' >> kernel/bpf/core.c:1659:44: sparse: incorrect type in initializer (different address spaces) @@ expected struct bpf_prog_array_item *item @@ got struct bpf_prog_astruct bpf_prog_array_item *item @@ >> kernel/bpf/core.c:1683:26: sparse: incorrect type in assignment (different address spaces) @@ expected struct bpf_prog_array_item *existing @@ got struct bpf_prog_astruct bpf_prog_array_item *existing @@ kernel/bpf/core.c:1711:15: sparse: incorrect type in assignment (different address spaces) @@ expected struct bpf_prog_array *array @@ got struct bpf_prog_astruct bpf_prog_array *array @@ >> kernel/bpf/core.c:1717:26: sparse: incorrect type in assignment (different address spaces) @@ expected struct bpf_prog_array_item *[assigned] existing @@ got structstruct bpf_prog_array_item *[assigned] existing @@ >> kernel/bpf/core.c:1748:41: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct bpf_prog_array *[addressable] array @@ got strstruct bpf_prog_array *[addressable] array @@
include/trace/events/xdp.h:28:1: sparse: Using plain integer as NULL pointer include/trace/events/xdp.h:53:1: sparse: Using plain integer as NULL pointer include/trace/events/xdp.h:111:1: sparse: Using plain integer as NULL pointer include/trace/events/xdp.h:126:1: sparse: Using plain integer as NULL pointer include/trace/events/xdp.h:162:1: sparse: Using plain integer as NULL pointer include/trace/events/xdp.h:197:1: sparse: Using plain integer as NULL pointer include/trace/events/xdp.h:232:1: sparse: Using plain integer as NULL pointer kernel/bpf/core.c:950:18: sparse: Initializer entry defined twice include/linux/slab.h:631:13: sparse: call with no type!
vim +1610 kernel/bpf/core.c
1568
1569 struct bpf_prog_array __rcu *bpf_prog_array_alloc(u32 prog_cnt, gfp_t flags)
1570 {
1571 if (prog_cnt)
> 1572 return kzalloc(sizeof(struct bpf_prog_array) +
1573 sizeof(struct bpf_prog_array_item) *
1574 (prog_cnt + 1),
1575 flags);
1576
> 1577 return &empty_prog_array.hdr;
1578 }
1579
1580 void bpf_prog_array_free(struct bpf_prog_array __rcu *progs)
1581 {
1582 if (!progs ||
1583 progs == (struct bpf_prog_array __rcu *)&empty_prog_array.hdr)
1584 return;
> 1585 kfree_rcu(progs, rcu);
1586 }
1587
1588 int bpf_prog_array_length(struct bpf_prog_array __rcu *array)
1589 {
1590 struct bpf_prog_array_item *item;
1591 u32 cnt = 0;
1592
1593 rcu_read_lock();
1594 item = rcu_dereference(array)->items;
1595 for (; item->prog; item++)
1596 if (item->prog != &dummy_bpf_prog.prog)
1597 cnt++;
1598 rcu_read_unlock();
1599 return cnt;
1600 }
1601
1602
1603 static bool bpf_prog_array_copy_core(struct bpf_prog_array *array,
1604 u32 *prog_ids,
1605 u32 request_cnt)
1606 {
1607 struct bpf_prog_array_item *item;
1608 int i = 0;
1609
> 1610 item = rcu_dereference(array)->items;
1611 for (; item->prog; item++) {
1612 if (item->prog == &dummy_bpf_prog.prog)
1613 continue;
1614 prog_ids[i] = item->prog->aux->id;
1615 if (++i == request_cnt) {
1616 item++;
1617 break;
1618 }
1619 }
1620
1621 return !!(item->prog);
1622 }
1623
1624 int bpf_prog_array_copy_to_user(struct bpf_prog_array __rcu *array,
1625 __u32 __user *prog_ids, u32 cnt)
1626 {
1627 unsigned long err = 0;
1628 bool nospc;
1629 u32 *ids;
1630
1631 /* users of this function are doing:
1632 * cnt = bpf_prog_array_length();
1633 * if (cnt > 0)
1634 * bpf_prog_array_copy_to_user(..., cnt);
1635 * so below kcalloc doesn't need extra cnt > 0 check, but
1636 * bpf_prog_array_length() releases rcu lock and
1637 * prog array could have been swapped with empty or larger array,
1638 * so always copy 'cnt' prog_ids to the user.
1639 * In a rare race the user will see zero prog_ids
1640 */
1641 ids = kcalloc(cnt, sizeof(u32), GFP_USER | __GFP_NOWARN);
1642 if (!ids)
1643 return -ENOMEM;
1644 rcu_read_lock();
1645 nospc = bpf_prog_array_copy_core(array, ids, cnt);
1646 rcu_read_unlock();
1647 err = copy_to_user(prog_ids, ids, cnt * sizeof(u32));
1648 kfree(ids);
1649 if (err)
1650 return -EFAULT;
1651 if (nospc)
1652 return -ENOSPC;
1653 return 0;
1654 }
1655
1656 void bpf_prog_array_delete_safe(struct bpf_prog_array __rcu *array,
1657 struct bpf_prog *old_prog)
1658 {
> 1659 struct bpf_prog_array_item *item = array->items;
1660
1661 for (; item->prog; item++)
1662 if (item->prog == old_prog) {
1663 WRITE_ONCE(item->prog, &dummy_bpf_prog.prog);
1664 break;
1665 }
1666 }
1667
1668 int bpf_prog_array_copy(struct bpf_prog_array __rcu *old_array,
1669 struct bpf_prog *exclude_prog,
1670 struct bpf_prog *include_prog,
1671 struct bpf_prog_array **new_array)
1672 {
1673 int new_prog_cnt, carry_prog_cnt = 0;
1674 struct bpf_prog_array_item *existing;
1675 struct bpf_prog_array *array;
1676 bool found_exclude = false;
1677 int new_prog_idx = 0;
1678
1679 /* Figure out how many existing progs we need to carry over to
1680 * the new array.
1681 */
1682 if (old_array) {
> 1683 existing = old_array->items;
1684 for (; existing->prog; existing++) {
1685 if (existing->prog == exclude_prog) {
1686 found_exclude = true;
1687 continue;
1688 }
1689 if (existing->prog != &dummy_bpf_prog.prog)
1690 carry_prog_cnt++;
1691 if (existing->prog == include_prog)
1692 return -EEXIST;
1693 }
1694 }
1695
1696 if (exclude_prog && !found_exclude)
1697 return -ENOENT;
1698
1699 /* How many progs (not NULL) will be in the new array? */
1700 new_prog_cnt = carry_prog_cnt;
1701 if (include_prog)
1702 new_prog_cnt += 1;
1703
1704 /* Do we have any prog (not NULL) in the new array? */
1705 if (!new_prog_cnt) {
1706 *new_array = NULL;
1707 return 0;
1708 }
1709
1710 /* +1 as the end of prog_array is marked with NULL */
1711 array = bpf_prog_array_alloc(new_prog_cnt + 1, GFP_KERNEL);
1712 if (!array)
1713 return -ENOMEM;
1714
1715 /* Fill in the new prog array */
1716 if (carry_prog_cnt) {
> 1717 existing = old_array->items;
1718 for (; existing->prog; existing++)
1719 if (existing->prog != exclude_prog &&
1720 existing->prog != &dummy_bpf_prog.prog) {
1721 array->items[new_prog_idx++].prog =
1722 existing->prog;
1723 }
1724 }
1725 if (include_prog)
1726 array->items[new_prog_idx++].prog = include_prog;
1727 array->items[new_prog_idx].prog = NULL;
1728 *new_array = array;
1729 return 0;
1730 }
1731
1732 int bpf_prog_array_copy_info(struct bpf_prog_array __rcu *array,
1733 u32 *prog_ids, u32 request_cnt,
1734 u32 *prog_cnt)
1735 {
1736 u32 cnt = 0;
1737
1738 if (array)
1739 cnt = bpf_prog_array_length(array);
1740
1741 *prog_cnt = cnt;
1742
1743 /* return early if user requested only program count or nothing to copy */
1744 if (!request_cnt || !cnt)
1745 return 0;
1746
1747 /* this function is called under trace/bpf_trace.c: bpf_event_mutex */
> 1748 return bpf_prog_array_copy_core(array, prog_ids, request_cnt) ? -ENOSPC 1749 : 0;
1750 }
1751
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH 6/14 bpf/verifier: introduce BPF_PTR_TO_MAP_VALUE
In reply to Roman GushchinBPF_MAP_TYPE_CGROUP_STORAGE maps are special in a way that the access from the bpf program side is lookup-free. That means the result is guaranteed to be a valid pointer to the cgroup storage; no NULL-check is required.
This patch introduces BPF_PTR_TO_MAP_VALUE return type, which is required to cause the verifier accept programs, which are not checking the map value pointer for being NULL.
--- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-)
include/linux/bpf.h +1 -0
@@ -154,6 +154,7 @@ enum bpf_arg_type {enum bpf_return_type {RET_INTEGER ,/* function returns integer */ RET_VOID ,/* function doesn't return anything */ +RET_PTR_TO_MAP_VALUE ,/* returns a pointer to map elem value */ RET_PTR_TO_MAP_VALUE_OR_NULL ,/* returns a pointer to map elem value or NULL */ };
kernel/bpf/verifier.c +6 -2
@@ -2545,8 +2545,12 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insnmark_reg_unknown (env, regs,BPF_REG_0 ); }else if (fn->ret_type ==RET_VOID ) { regs[BPF_REG_0 ].type =NOT_INIT ; - }else if (fn->ret_type ==RET_PTR_TO_MAP_VALUE_OR_NULL ) { - regs[BPF_REG_0 ].type =PTR_TO_MAP_VALUE_OR_NULL ; + }else if (fn->ret_type ==RET_PTR_TO_MAP_VALUE_OR_NULL || + fn->ret_type ==RET_PTR_TO_MAP_VALUE ) { +if (fn->ret_type ==RET_PTR_TO_MAP_VALUE ) + regs[BPF_REG_0 ].type =PTR_TO_MAP_VALUE ; +else + regs[BPF_REG_0 ].type =PTR_TO_MAP_VALUE_OR_NULL ;/* There is no offset yet applied, variable or fixed */ mark_reg_known_zero (env, regs,BPF_REG_0 ); regs[BPF_REG_0 ].off =0 ;
Signature · 1 line
2.14.4
PATCH 7/14 bpf: don't allow create maps of cgroup local storages
In reply to Roman GushchinAs there is one-to-one relation between a bpf program and cgroup local storage map, there is no sense in creating a map of cgroup local storage maps.
Forbid it explicitly to avoid possible side effects.
--- kernel/bpf/map_in_map.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/bpf/map_in_map.c +2 -1
@@ -23,7 +23,8 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)* is a runtime binding. Doing static check alone * in the verifier is not enough. */ -if (inner_map->map_type ==BPF_MAP_TYPE_PROG_ARRAY ) { +if (inner_map->map_type ==BPF_MAP_TYPE_PROG_ARRAY || + inner_map->map_type ==BPF_MAP_TYPE_CGROUP_STORAGE ) {fdput (f);return ERR_PTR (-ENOTSUPP ); }
Signature · 1 line
2.14.4
PATCH 8/14 bpf: introduce the bpf_get_local_storage() helper function
In reply to Roman GushchinThe bpf_get_local_storage() helper function is used to get a pointer to the bpf local storage from a bpf program.
It takes a pointer to a storage map and flags as arguments. Right now it accepts only cgroup storage maps, and flags argument has to be 0. Further it can be extended to support other types of local storage: e.g. thread local storage etc.
--- include/linux/bpf.h | 2 ++ include/uapi/linux/bpf.h | 13 ++++++++++++- kernel/bpf/cgroup.c | 2 ++ kernel/bpf/helpers.c | 20 ++++++++++++++++++++ kernel/bpf/verifier.c | 18 ++++++++++++++++++ net/core/filter.c | 23 ++++++++++++++++++++++- 6 files changed, 76 insertions(+), 2 deletions(-)
include/linux/bpf.h +2 -0
@@ -771,6 +771,8 @@ extern const struct bpf_func_proto bpf_sock_map_update_proto;extern const struct bpf_func_proto bpf_sock_hash_update_proto;extern const struct bpf_func_proto bpf_get_current_cgroup_id_proto; +extern const struct bpf_func_proto bpf_get_local_storage_proto; +/* Shared helpers among cBPF and eBPF. */ void bpf_user_rnd_init_once (void );u64 bpf_user_rnd_u32 (u64 r1,u64 r2,u64 r3,u64 r4,u64 r5);
include/uapi/linux/bpf.h +12 -1
@@ -2081,6 +2081,16 @@ union bpf_attr {* Return * A 64-bit integer containing the current cgroup id based * on the cgroup within which the current task is running. +* +* void* get_local_storage(void *map, u64 flags) +* Description +* Get the pointer to the local storage area. +* The type and the size of the local storage is defined +* by the *map* argument. +* The *flags* meaning is specific for each map type, +* and has to be 0 for cgroup local storage. +* Return +* Pointer to the local storage area. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -2163,7 +2173,8 @@ union bpf_attr {FN (rc_repeat), \FN (rc_keydown), \FN (skb_cgroup_id), \ -FN (get_current_cgroup_id), +FN (get_current_cgroup_id), \ +FN (get_local_storage),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call
kernel/bpf/cgroup.c +2 -0
@@ -629,6 +629,8 @@ cgroup_dev_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_map_delete_elem_proto;case BPF_FUNC_get_current_uid_gid:return &bpf_get_current_uid_gid_proto; +case BPF_FUNC_get_local_storage: +return &bpf_get_local_storage_proto;case BPF_FUNC_trace_printk:if (capable (CAP_SYS_ADMIN ))return bpf_get_trace_printk_proto ();
kernel/bpf/helpers.c +20 -0
@@ -193,4 +193,24 @@ const struct bpf_func_proto bpf_get_current_cgroup_id_proto = { .gpl_only = false, .ret_type =RET_INTEGER , }; + +DECLARE_PER_CPU(void*, bpf_cgroup_storage); + +BPF_CALL_2(bpf_get_local_storage, struct bpf_map *, map, u64, flags) +{ +/* map and flags arguments are not used now, +* but provide an ability to extend the API +* for other types of local storages. +* verifier checks that their values are correct. +*/ +return (u64 )this_cpu_read (bpf_cgroup_storage); +} + +const struct bpf_func_proto bpf_get_local_storage_proto = { + .func = bpf_get_local_storage, + .gpl_only = false, + .ret_type =RET_PTR_TO_MAP_VALUE , + .arg1_type =ARG_CONST_MAP_PTR , + .arg2_type =ARG_ANYTHING , +};#endif
kernel/bpf/verifier.c +18 -0
@@ -2127,6 +2127,10 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, func_id != BPF_FUNC_current_task_under_cgroup) goto error;break ; +case BPF_MAP_TYPE_CGROUP_STORAGE : +if (func_id != BPF_FUNC_get_local_storage) + goto error; +break ;/* devmap returns a pointer to a live net_device ifindex that we cannot * allow to be modified from bpf side. So do not allow lookup elements * for now.
@@ -2209,6 +2213,10 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,if (map->map_type !=BPF_MAP_TYPE_SOCKHASH ) goto error;break ; +case BPF_FUNC_get_local_storage: +if (map->map_type !=BPF_MAP_TYPE_CGROUP_STORAGE ) + goto error; +break ;default :break ; }
@@ -2533,6 +2541,16 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn } regs =cur_regs (env); + +/* check that flags argument in get_local_storage(map, flags) is 0, +* this is required because get_local_storage() can't return an error. +*/ +if (func_id == BPF_FUNC_get_local_storage && + !tnum_equals_const (regs[BPF_REG_2 ].var_off,0 )) { +verbose (env,"get_local_storage() doesn't support non-zero flags\n" ); +return -EINVAL ; + } +/* reset caller saved regs */ for (i =0 ; i <CALLER_SAVED_REGS ; i++) {mark_reg_not_init (env, regs, caller_saved[i]);
net/core/filter.c +22 -1
@@ -4652,6 +4652,8 @@ sock_filter_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)*/ case BPF_FUNC_get_current_uid_gid:return &bpf_get_current_uid_gid_proto; +case BPF_FUNC_get_local_storage: +return &bpf_get_local_storage_proto;default :return bpf_base_func_proto (func_id); }
@@ -4674,6 +4676,8 @@ sock_addr_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)default :return NULL ; } +case BPF_FUNC_get_local_storage: +return &bpf_get_local_storage_proto;default :return bpf_base_func_proto (func_id); }
@@ -4696,6 +4700,17 @@ sk_filter_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) } } +static const struct bpf_func_proto * +cg_skb_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog) +{ +switch (func_id) { +case BPF_FUNC_get_local_storage: +return &bpf_get_local_storage_proto; +default : +return sk_filter_func_proto (func_id, prog); + } +} +static const struct bpf_func_proto *tc_cls_act_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog) {
@@ -4816,6 +4831,8 @@ sock_ops_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_sock_map_update_proto;case BPF_FUNC_sock_hash_update:return &bpf_sock_hash_update_proto; +case BPF_FUNC_get_local_storage: +return &bpf_get_local_storage_proto;default :return bpf_base_func_proto (func_id); }
@@ -4835,6 +4852,8 @@ sk_msg_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_msg_cork_bytes_proto;case BPF_FUNC_msg_pull_data:return &bpf_msg_pull_data_proto; +case BPF_FUNC_get_local_storage: +return &bpf_get_local_storage_proto;default :return bpf_base_func_proto (func_id); }
@@ -4862,6 +4881,8 @@ sk_skb_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_sk_redirect_map_proto;case BPF_FUNC_sk_redirect_hash:return &bpf_sk_redirect_hash_proto; +case BPF_FUNC_get_local_storage: +return &bpf_get_local_storage_proto;default :return bpf_base_func_proto (func_id); }
@@ -6664,7 +6685,7 @@ const struct bpf_prog_ops xdp_prog_ops = { };const struct bpf_verifier_ops cg_skb_verifier_ops = { - .get_func_proto = sk_filter_func_proto, + .get_func_proto = cg_skb_func_proto, .is_valid_access = sk_filter_is_valid_access, .convert_ctx_access = bpf_convert_ctx_access, };
Signature · 1 line
2.14.4
kbuild test robot
build botRe: [PATCH bpf-net 08/14] bpf: introduce the bpf_get_local_storage() helper function
Thank you for the patch! Yet something to improve:
Hi Roman,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on bpf-next/master]
[also build test ERROR on v4.18-rc2]
[cannot apply to next-20180628]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Roman-Gushchin/bpf-cgroup-local-storage/20180629-035104 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master config: um-x86_64_defconfig (attached as .config) compiler: gcc-7 (Debian 7.3.0-16) 7.3.0 reproduce: # save the attached .config to linux build tree make ARCH=um SUBARCH=x86_64
All errors (new ones prefixed by >>):
net/core/filter.o: In function `cg_skb_func_proto':
filter.c:(.text+0x58a7): undefined reference to bpf_get_local_storage_proto' net/core/filter.o: In function sock_filter_func_proto':
filter.c:(.text+0x5b3d): undefined reference to bpf_get_local_storage_proto' net/core/filter.o: In function sock_ops_func_proto':
filter.c:(.text+0x5b9d): undefined reference to bpf_get_local_storage_proto' net/core/filter.o: In function sk_skb_func_proto':
filter.c:(.text+0x5c60): undefined reference to bpf_get_local_storage_proto' net/core/filter.o: In function sk_msg_func_proto':
filter.c:(.text+0x5cc3): undefined reference to bpf_get_local_storage_proto' net/core/filter.o:filter.c:(.text+0x5ee6): more undefined references to bpf_get_local_storage_proto' follow
>> collect2: error: ld returned 1 exit status
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH 9/14 bpf: sync bpf.h to tools/
In reply to Roman GushchinSync cgroup storage related changes:
1) new BPF_MAP_TYPE_CGROUP_STORAGE map type
2) struct bpf_cgroup_sotrage_key definition
3) get_local_storage() helper
--- tools/include/uapi/linux/bpf.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
tools/include/uapi/linux/bpf.h +8 -1
@@ -75,6 +75,11 @@ struct bpf_lpm_trie_key {__u8 data[0 ];/* Arbitrary size */ }; +struct bpf_cgroup_storage_key { +__u64 cgroup_inode_id;/* cgroup inode id */ +__u32 attach_type;/* program attach type */ +}; +/* BPF syscall commands, see bpf(2) man-page for details. */ enum bpf_cmd {BPF_MAP_CREATE ,
@@ -120,6 +125,7 @@ enum bpf_map_type {BPF_MAP_TYPE_CPUMAP ,BPF_MAP_TYPE_XSKMAP ,BPF_MAP_TYPE_SOCKHASH , + BPF_MAP_TYPE_CGROUP_STORAGE, }; enum bpf_prog_type {
@@ -2157,7 +2163,8 @@ union bpf_attr {FN (rc_repeat), \FN (rc_keydown), \FN (skb_cgroup_id), \ -FN (get_current_cgroup_id), +FN (get_current_cgroup_id), \ +FN (get_local_storage),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call
Signature · 1 line
2.14.4
PATCH 10/14 bpftool: add support for CGROUP_STORAGE maps
In reply to Roman GushchinAdd BPF_MAP_TYPE_CGROUP_STORAGE maps to the list of maps types which bpftool recognizes.
--- tools/bpf/bpftool/map.c | 1 + 1 file changed, 1 insertion(+)
tools/bpf/bpftool/map.c +1 -0
@@ -67,6 +67,7 @@ static const char * const map_type_name[] = { [BPF_MAP_TYPE_SOCKMAP] = "sockmap", [BPF_MAP_TYPE_CPUMAP] = "cpumap", [BPF_MAP_TYPE_SOCKHASH] = "sockhash", + [BPF_MAP_TYPE_CGROUP_STORAGE] = "cgroup_storage", };static bool map_is_per_cpu (__u32 type)
Signature · 1 line
2.14.4
PATCH 11/14 bpf/test_run: support cgroup local storage
In reply to Roman GushchinAllocate a temporary cgroup storage to use for bpf program test runs.
Because the test program is not actually attached to a cgroup, the storage is allocated manually just for the execution of the bpf program.
If the program is executed multiple times, the storage is not zeroed on each run, emulating multiple runs of the program, attached to a real cgroup.
--- net/bpf/test_run.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
net/bpf/test_run.c +11 -2
@@ -11,12 +11,14 @@#include <linux/filter.h> #include <linux/sched/signal.h> -static __always_inline u32 bpf_test_run_one(struct bpf_prog *prog, void *ctx) +static __always_inline u32 bpf_test_run_one(struct bpf_prog *prog, void *ctx, +struct bpf_cgroup_storage *storage) {u32 ret;preempt_disable ();rcu_read_lock (); +bpf_cgroup_storage_set (storage); ret =BPF_PROG_RUN (prog, ctx);rcu_read_unlock ();preempt_enable ();
@@ -26,14 +28,19 @@ static __always_inline u32 bpf_test_run_one(struct bpf_prog *prog, void *ctx)static u32 bpf_test_run (struct bpf_prog *prog,void *ctx,u32 repeat,u32 *time) { +struct bpf_cgroup_storage *storage =NULL ;u64 time_start, time_spent =0 ;u32 ret =0 , i; + storage =bpf_cgroup_storage_alloc (prog); +if (IS_ERR (storage)) +return PTR_ERR (storage); +if (!repeat) repeat =1 ; time_start =ktime_get_ns ();for (i =0 ; i < repeat; i++) { - ret =bpf_test_run_one (prog, ctx); + ret =bpf_test_run_one (prog, ctx, storage);if (need_resched ()) {if (signal_pending (current))break ;
@@ -46,6 +53,8 @@ static u32 bpf_test_run(struct bpf_prog *prog, void *ctx, u32 repeat, u32 *time)do_div (time_spent, repeat); *time = time_spent >U32_MAX ?U32_MAX : (u32 )time_spent; +bpf_cgroup_storage_free (storage); +return ret; }
Signature · 1 line
2.14.4
PATCH 12/14 selftests/bpf: add verifier cgroup storage tests
In reply to Roman GushchinAdd the following verifier tests to cover the cgroup storage functionality:
1) valid access to the cgroup storage
2) invalid access: use regular hashmap instead of cgroup storage map
3) invalid access: use invalid map fd
4) invalid access: try access memory after the cgroup storage
5) invalid access: try access memory before the cgroup storage
6) invalid access: call get_local_storage() with non-zero flags
For tests 2)-6) check returned error strings.
Expected output:
$ ./test_verifier #0/u add+sub+mul OK #0/p add+sub+mul OK #1/u DIV32 by 0, zero check 1 OK ... #280/p valid cgroup storage access OK #281/p invalid cgroup storage access 1 OK #282/p invalid cgroup storage access 2 OK #283/p invalid per-cgroup storage access 3 OK #284/p invalid cgroup storage access 4 OK #285/p invalid cgroup storage access 5 OK ... #649/p pass modified ctx pointer to helper, 2 OK #650/p pass modified ctx pointer to helper, 3 OK Summary: 901 PASSED, 0 SKIPPED, 0 FAILED
--- tools/testing/selftests/bpf/bpf_helpers.h | 2 + tools/testing/selftests/bpf/test_verifier.c | 123 +++++++++++++++++++++++++++- 2 files changed, 124 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/bpf_helpers.h +2 -0
@@ -133,6 +133,8 @@ static int (*bpf_rc_keydown)(void *ctx, unsigned int protocol, (void *) BPF_FUNC_rc_keydown;static unsigned long long (*bpf_get_current_cgroup_id)(void ) = (void *) BPF_FUNC_get_current_cgroup_id; +static void *(*bpf_get_local_storage)(void *map,unsigned long long flags) = + (void *) BPF_FUNC_get_local_storage;/* llvm builtin functions that eBPF C program may use to * emit BPF_LD_ABS and BPF_LD_IND instructions
tools/testing/selftests/bpf/test_verifier.c +122 -1
159 lines not on this page · raw message
Signature · 1 line
2.14.4
PATCH 13/14 selftests/bpf: add a cgroup storage test
In reply to Roman GushchinImplement a test to cover the cgroup storage functionality. The test implements a bpf program which drops every second packet by using the cgroup storage as a persistent storage.
The test also use the userspace API to check the data in the cgroup storage, alter it, and check that the loaded and attached bpf program sees the update.
Expected output:
$ ./test_cgroup_storage test_cgroup_storage:PASS
--- tools/testing/selftests/bpf/Makefile | 4 +- tools/testing/selftests/bpf/test_cgroup_storage.c | 130 ++++++++++++++++++++++ 2 files changed, 133 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/test_cgroup_storage.c
tools/testing/selftests/bpf/Makefile +3 -1
@@ -22,7 +22,8 @@ $(TEST_CUSTOM_PROGS): $(OUTPUT)/%: %.c # Order correspond to 'make run_tests' order TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test_progs \ test_align test_verifier_log test_dev_cgroup test_tcpbpf_user \ - test_sock test_btf test_sockmap test_lirc_mode2_user get_cgroup_id_user + test_sock test_btf test_sockmap test_lirc_mode2_user get_cgroup_id_user \ + test_cgroup_storage TEST_GEN_FILES = test_pkt_access.o test_xdp.o test_l4lb.o test_tcp_estats.o test_obj_id.o \ test_pkt_md_access.o test_xdp_redirect.o test_xdp_meta.o sockmap_parse_prog.o \
@@ -63,6 +64,7 @@ $(OUTPUT)/test_sock_addr: cgroup_helpers.c $(OUTPUT)/test_sockmap: cgroup_helpers.c $(OUTPUT)/test_progs: trace_helpers.c $(OUTPUT)/get_cgroup_id_user: cgroup_helpers.c +$(OUTPUT)/test_cgroup_storage: cgroup_helpers.c .PHONY: force
tools/testing/selftests/bpf/test_cgroup_storage.c new file +130 -0
130 lines not on this page · raw message
Signature · 1 line
2.14.4
PATCH 14/14 samples/bpf: extend test_cgrp2_attach2 test to use cgroup storage
In reply to Roman GushchinThe test_cgrp2_attach test covers bpf cgroup attachment code well, so let's re-use it for testing allocation/releasing of cgroup storage.
The extension is pretty straightforward: the bpf program will use the cgroup storage to save the number of transmitted bytes.
Expected output:
$ ./test_cgrp2_attach2 Attached DROP prog. This ping in cgroup /foo should fail... ping: sendmsg: Operation not permitted Attached DROP prog. This ping in cgroup /foo/bar should fail... ping: sendmsg: Operation not permitted Attached PASS prog. This ping in cgroup /foo/bar should pass... Detached PASS from /foo/bar while DROP is attached to /foo. This ping in cgroup /foo/bar should fail... ping: sendmsg: Operation not permitted Attached PASS from /foo/bar and detached DROP from /foo. This ping in cgroup /foo/bar should pass... ### override:PASS ### multi:PASS
--- samples/bpf/test_cgrp2_attach2.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-)
samples/bpf/test_cgrp2_attach2.c +26 -1
@@ -8,7 +8,8 @@* information. The number of invocations of the program, which maps * to the number of packets received, is stored to key 0. Key 1 is * incremented on each iteration by the number of bytes stored in -* the skb. +* the skb. The program also stores the number of received bytes +* in the cgroup storage. * * - Attaches the new program to a cgroup using BPF_PROG_ATTACH *
@@ -21,12 +22,15 @@#include <stdio.h> #include <stdlib.h> #include <assert.h> +#include <sys/resource.h> +#include <sys/time.h> #include <unistd.h> #include <linux/bpf.h> #include <bpf/bpf.h> #include "bpf_insn.h" +#include "bpf_rlimit.h" #include "cgroup_helpers.h" #define FOO "/foo"
@@ -205,6 +209,8 @@ static int map_fd = -1;static int prog_load_cnt (int verdict,int val) { +int cgroup_storage_fd; +if (map_fd <0 ) map_fd =bpf_create_map (BPF_MAP_TYPE_ARRAY ,4 ,8 ,1 ,0 );if (map_fd <0 ) {
@@ -212,6 +218,13 @@ static int prog_load_cnt(int verdict, int val)return -1 ; } + cgroup_storage_fd =bpf_create_map (BPF_MAP_TYPE_CGROUP_STORAGE , +sizeof (struct bpf_cgroup_storage_key ),8 ,0 ,0 ); +if (cgroup_storage_fd <0 ) { +printf ("failed to create map '%s'\n" ,strerror (errno)); +return -1 ; + } +struct bpf_insn prog[] = {BPF_MOV32_IMM (BPF_REG_0 ,0 ),BPF_STX_MEM (BPF_W ,BPF_REG_10 ,BPF_REG_0 ,-4 ),/* *(u32 *)(fp - 4) = r0 */
@@ -222,6 +235,11 @@ static int prog_load_cnt(int verdict, int val)BPF_JMP_IMM (BPF_JEQ ,BPF_REG_0 ,0 ,2 ),BPF_MOV64_IMM (BPF_REG_1 , val),/* r1 = 1 */ BPF_RAW_INSN (BPF_STX |BPF_XADD |BPF_DW ,BPF_REG_0 ,BPF_REG_1 ,0 ,0 ),/* xadd r0 += r1 */ +BPF_LD_MAP_FD (BPF_REG_1 , cgroup_storage_fd), +BPF_MOV64_IMM (BPF_REG_2 ,0 ), +BPF_RAW_INSN (BPF_JMP |BPF_CALL ,0 ,0 ,0 , BPF_FUNC_get_local_storage), +BPF_MOV64_IMM (BPF_REG_1 , val), +BPF_RAW_INSN (BPF_STX |BPF_XADD |BPF_W ,BPF_REG_0 ,BPF_REG_1 ,0 ,0 ),BPF_MOV64_IMM (BPF_REG_0 , verdict),/* r0 = verdict */ BPF_EXIT_INSN(), };
@@ -237,6 +255,7 @@ static int prog_load_cnt(int verdict, int val)printf ("Output from verifier:\n%s\n-------\n" , bpf_log_buf);return 0 ; } +close (cgroup_storage_fd);return ret; }
@@ -414,6 +433,12 @@ static int test_multiprog(void)int main (int argc,char **argv) {int rc =0 ; +struct rlimit r = {1024 *1024 ,RLIM_INFINITY }; + +if (setrlimit (RLIMIT_MEMLOCK , &r)) { +log_err ("Setrlimit(RLIMIT_MEMLOCK) failed" ); +return 1 ; + } rc =test_foo_bar ();if (rc)
Signature · 1 line
2.14.4
Quoted 3 lines · Roman Gushchin
This patchset implements cgroup local storage for bpf programs. The main idea is to provide a fast accessible memory for storing various per-cgroup data, e.g. number of transmitted packets.
Just noticed a typo in the subject: "bpf-net" :)
Will resend the patchset.
Sorry for confusion.
Thanks,
Roman