Add support for memory-mapping BPF array maps
Subsystems (21)
v5 cover Add support for memory-mapping BPF array maps
This patch set adds ability to memory-map BPF array maps (single- and multi-element). The primary use case is memory-mapping BPF array maps, created to back global data variables, created by libbpf implicitly. This allows for much better usability, along with avoiding syscalls to read or update data completely.
Due to memory-mapping requirements, BPF array map that is supposed to be memory-mapped, has to be created with special BPF_F_MMAPABLE attribute, which triggers slightly different memory allocation strategy internally. See patch 1 for details.
Libbpf is extended to detect kernel support for this flag, and if supported, will specify it for all global data maps automatically.
Patch #1 refactors bpf_map_inc() and converts bpf_map's refcnt to atomic64_t to make refcounting never fail. Patch #2 does similar refactoring for bpf_prog_add()/bpf_prog_inc().
v4->v5:
- change bpf_prog's refcnt to atomic64_t (Daniel);
v3->v4:
- add mmap's open() callback to fix refcounting (Johannes);
- switch to remap_vmalloc_pages() instead of custom fault handler (Johannes);
- converted bpf_map's refcnt/usercnt into atomic64_t;
- provide default bpf_map_default_vmops handling open/close properly;
v2->v3:
- change allocation strategy to avoid extra pointer dereference (Jakub);
v1->v2:
- fix map lookup code generation for BPF_F_MMAPABLE case;
- prevent BPF_F_MMAPABLE flag for all but plain array map type;
- centralize ref-counting in generic bpf_map_mmap();
- don't use uref counting (Alexei);
- use vfree() directly;
- print flags with %x (Song);
- extend tests to verify bpf_map_{lookup,update}_elem() logic as well.
Andrii Nakryiko (5):
bpf: switch bpf_map ref counter to atomic64_t so bpf_map_inc() never
fails
bpf: convert bpf_prog refcnt to atomic64_t
bpf: add mmap() support for BPF_MAP_TYPE_ARRAY
libbpf: make global data internal arrays mmap()-able, if possible selftests/bpf: add BPF_TYPE_MAP_ARRAY mmap() tests
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 9 +- .../net/ethernet/cavium/thunder/nicvf_main.c | 9 +- .../net/ethernet/freescale/dpaa2/dpaa2-eth.c | 7 +- .../net/ethernet/mellanox/mlx4/en_netdev.c | 24 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 18 +- .../net/ethernet/netronome/nfp/bpf/offload.c | 4 +- drivers/net/ethernet/qlogic/qede/qede_main.c | 8 +- drivers/net/virtio_net.c | 7 +- include/linux/bpf.h | 34 +-- include/linux/vmalloc.h | 1 + include/uapi/linux/bpf.h | 3 + kernel/bpf/arraymap.c | 58 ++++- kernel/bpf/inode.c | 7 +- kernel/bpf/map_in_map.c | 2 +- kernel/bpf/syscall.c | 174 ++++++++++---- kernel/bpf/verifier.c | 6 +- kernel/bpf/xskmap.c | 6 +- kernel/events/core.c | 7 +- mm/vmalloc.c | 20 ++ net/core/bpf_sk_storage.c | 2 +- tools/include/uapi/linux/bpf.h | 3 + tools/lib/bpf/libbpf.c | 32 ++- .../selftests/bpf/prog_tests/core_reloc.c | 45 ++-- tools/testing/selftests/bpf/prog_tests/mmap.c | 220 ++++++++++++++++++ tools/testing/selftests/bpf/progs/test_mmap.c | 45 ++++ 25 files changed, 576 insertions(+), 175 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/mmap.c create mode 100644 tools/testing/selftests/bpf/progs/test_mmap.c
Signature · 1 line
2.17.1
PATCH v5 1/5 bpf: switch bpf_map ref counter to atomic64_t so bpf_map_inc() never fails
92117d8443bc ("bpf: fix refcnt overflow") turned refcounting of bpf_map into potentially failing operation, when refcount reaches BPF_MAX_REFCNT limit (32k). Due to using 32-bit counter, it's possible in practice to overflow refcounter and make it wrap around to 0, causing erroneous map free, while there are still references to it, causing use-after-free problems.
But having a failing refcounting operations are problematic in some cases. One example is mmap() interface. After establishing initial memory-mapping, user is allowed to arbitrarily map/remap/unmap parts of mapped memory, arbitrarily splitting it into multiple non-contiguous regions. All this happening without any control from the users of mmap subsystem. Rather mmap subsystem sends notifications to original creator of memory mapping through open/close callbacks, which are optionally specified during initial memory mapping creation. These callbacks are used to maintain accurate refcount for bpf_map (see next patch in this series). The problem is that open() callback is not supposed to fail, because memory-mapped resource is set up and properly referenced. This is posing a problem for using memory-mapping with BPF maps.
One solution to this is to maintain separate refcount for just memory-mappings and do single bpf_map_inc/bpf_map_put when it goes from/to zero, respectively. There are similar use cases in current work on tcp-bpf, necessitating extra counter as well. This seems like a rather unfortunate and ugly solution that doesn't scale well to various new use cases.
Another approach to solve this is to use non-failing refcount_t type, which uses 32-bit counter internally, but, once reaching overflow state at UINT_MAX, stays there. This utlimately causes memory leak, but prevents use after free.
But given refcounting is not the most performance-critical operation with BPF maps (it's not used from running BPF program code), we can also just switch to 64-bit counter that can't overflow in practice, potentially disadvantaging 32-bit platforms a tiny bit. This simplifies semantics and allows above described scenarios to not worry about failing refcount increment operation.
In terms of struct bpf_map size, we are still good and use the same amount of space:
BEFORE (3 cache lines, 8 bytes of padding at the end):
struct bpf_map {
const struct bpf_map_ops * ops __attribute__((__aligned__(64))); /* 0 8 */
struct bpf_map * inner_map_meta; /* 8 8 */
void * security; /* 16 8 */
enum bpf_map_type map_type; /* 24 4 */
u32 key_size; /* 28 4 */
u32 value_size; /* 32 4 */
u32 max_entries; /* 36 4 */
u32 map_flags; /* 40 4 */
int spin_lock_off; /* 44 4 */
u32 id; /* 48 4 */
int numa_node; /* 52 4 */
u32 btf_key_type_id; /* 56 4 */
u32 btf_value_type_id; /* 60 4 */
/* --- cacheline 1 boundary (64 bytes) --- */
struct btf * btf; /* 64 8 */
struct bpf_map_memory memory; /* 72 16 */
bool unpriv_array; /* 88 1 */
bool frozen; /* 89 1 */
/* XXX 38 bytes hole, try to pack */
/* --- cacheline 2 boundary (128 bytes) --- */
atomic_t refcnt __attribute__((__aligned__(64))); /* 128 4 */
atomic_t usercnt; /* 132 4 */
struct work_struct work; /* 136 32 */
char name[16]; /* 168 16 */
/* size: 192, cachelines: 3, members: 21 */
/* sum members: 146, holes: 1, sum holes: 38 */
/* padding: 8 */
/* forced alignments: 2, forced holes: 1, sum forced holes: 38 */
} __attribute__((__aligned__(64)));
AFTER (same 3 cache lines, no extra padding now):
struct bpf_map {
const struct bpf_map_ops * ops __attribute__((__aligned__(64))); /* 0 8 */
struct bpf_map * inner_map_meta; /* 8 8 */
void * security; /* 16 8 */
enum bpf_map_type map_type; /* 24 4 */
u32 key_size; /* 28 4 */
u32 value_size; /* 32 4 */
u32 max_entries; /* 36 4 */
u32 map_flags; /* 40 4 */
int spin_lock_off; /* 44 4 */
u32 id; /* 48 4 */
int numa_node; /* 52 4 */
u32 btf_key_type_id; /* 56 4 */
u32 btf_value_type_id; /* 60 4 */
/* --- cacheline 1 boundary (64 bytes) --- */
struct btf * btf; /* 64 8 */
struct bpf_map_memory memory; /* 72 16 */
bool unpriv_array; /* 88 1 */
bool frozen; /* 89 1 */
/* XXX 38 bytes hole, try to pack */
/* --- cacheline 2 boundary (128 bytes) --- */
atomic64_t refcnt __attribute__((__aligned__(64))); /* 128 8 */
atomic64_t usercnt; /* 136 8 */
struct work_struct work; /* 144 32 */
char name[16]; /* 176 16 */
/* size: 192, cachelines: 3, members: 21 */
/* sum members: 154, holes: 1, sum holes: 38 */
/* forced alignments: 2, forced holes: 1, sum forced holes: 38 */
} __attribute__((__aligned__(64)));
This patch, while modifying all users of bpf_map_inc, also cleans up its interface to match bpf_map_put with separate operations for bpf_map_inc and bpf_map_inc_with_uref (to match bpf_map_put and bpf_map_put_with_uref, respectively). Also, given there are no users of bpf_map_inc_not_zero specifying uref=true, remove uref flag and default to uref=false internally.
--- .../net/ethernet/netronome/nfp/bpf/offload.c | 4 +- include/linux/bpf.h | 10 ++-- kernel/bpf/inode.c | 2 +- kernel/bpf/map_in_map.c | 2 +- kernel/bpf/syscall.c | 51 ++++++++----------- kernel/bpf/verifier.c | 6 +-- kernel/bpf/xskmap.c | 6 +-- net/core/bpf_sk_storage.c | 2 +- 8 files changed, 34 insertions(+), 49 deletions(-)
drivers/net/ethernet/netronome/nfp/bpf/offload.c +1 -3
@@ -46,9 +46,7 @@ nfp_map_ptr_record(struct nfp_app_bpf *bpf, struct nfp_prog *nfp_prog,/* Grab a single ref to the map for our record. The prog destroy ndo * happens after free_used_maps(). */ - map =bpf_map_inc (map, false); -if (IS_ERR (map)) -return PTR_ERR (map); +bpf_map_inc (map); record =kmalloc (sizeof (*record),GFP_KERNEL );if (!record) {
include/linux/bpf.h +5 -5
@@ -103,8 +103,8 @@ struct bpf_map {/* The 3rd and 4th cacheline with misc members to avoid false sharing * particularly with refcounting. */ -atomic_t refcnt____cacheline_aligned ; -atomic_t usercnt; +atomic64_t refcnt____cacheline_aligned ; +atomic64_t usercnt;struct work_struct work;char name[BPF_OBJ_NAME_LEN ]; };
@@ -783,9 +783,9 @@ void bpf_map_free_id(struct bpf_map *map, bool do_idr_lock);struct bpf_map *bpf_map_get_with_uref (u32 ufd);struct bpf_map *__bpf_map_get (struct fd f); -struct bpf_map *__must_check bpf_map_inc (struct bpf_map *map,bool uref); -struct bpf_map *__must_check bpf_map_inc_not_zero (struct bpf_map *map, -bool uref); +void bpf_map_inc (struct bpf_map *map); +void bpf_map_inc_with_uref (struct bpf_map *map); +struct bpf_map *__must_check bpf_map_inc_not_zero (struct bpf_map *map);void bpf_map_put_with_uref (struct bpf_map *map);void bpf_map_put (struct bpf_map *map);int bpf_map_charge_memlock (struct bpf_map *map,u32 pages);
kernel/bpf/inode.c +1 -1
@@ -34,7 +34,7 @@ static void *bpf_any_get(void *raw, enum bpf_type type) raw =bpf_prog_inc (raw);break ;case BPF_TYPE_MAP : - raw =bpf_map_inc (raw, true); +bpf_map_inc_with_uref (raw);break ;default :WARN_ON_ONCE (1 );
kernel/bpf/map_in_map.c +1 -1
@@ -98,7 +98,7 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map,return inner_map;if (bpf_map_meta_equal (map->inner_map_meta, inner_map)) - inner_map =bpf_map_inc (inner_map, false); +bpf_map_inc (inner_map);else inner_map =ERR_PTR (-EINVAL );
kernel/bpf/syscall.c +22 -29
109 lines not on this page · raw message
kernel/bpf/verifier.c +1 -5
@@ -8179,11 +8179,7 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env)* will be used by the valid program until it's unloaded * and all maps are released in free_used_maps() */ - map =bpf_map_inc (map, false); -if (IS_ERR (map)) { -fdput (f); -return PTR_ERR (map); - } +bpf_map_inc (map); aux->map_index = env->used_map_cnt; env->used_maps[env->used_map_cnt++] = map;
kernel/bpf/xskmap.c +2 -4
@@ -11,10 +11,8 @@int xsk_map_inc (struct xsk_map *map) { -struct bpf_map *m = &map->map; - - m =bpf_map_inc (m, false); -return PTR_ERR_OR_ZERO (m); +bpf_map_inc (&map->map); +return 0 ; }void xsk_map_put (struct xsk_map *map)
net/core/bpf_sk_storage.c +1 -1
@@ -798,7 +798,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk)* Try to grab map refcnt to make sure that it's still * alive and prevent concurrent removal. */ - map =bpf_map_inc_not_zero (&smap->map, false); + map =bpf_map_inc_not_zero (&smap->map);if (IS_ERR (map))continue ;
Signature · 1 line
2.17.1
PATCH v5 2/5 bpf: convert bpf_prog refcnt to atomic64_t
In reply to Andrii NakryikoSimilarly to bpf_map's refcnt/usercnt, convert bpf_prog's refcnt to atomic64 and remove artificial 32k limit. This allows to make bpf_prog's refcounting non-failing, simplifying logic of users of bpf_prog_add/bpf_prog_inc.
Validated compilation by running allyesconfig kernel build.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 9 ++---- .../net/ethernet/cavium/thunder/nicvf_main.c | 9 ++---- .../net/ethernet/freescale/dpaa2/dpaa2-eth.c | 7 ++--- .../net/ethernet/mellanox/mlx4/en_netdev.c | 24 ++++----------- .../net/ethernet/mellanox/mlx5/core/en_main.c | 18 ++++------- drivers/net/ethernet/qlogic/qede/qede_main.c | 8 ++--- drivers/net/virtio_net.c | 7 ++--- include/linux/bpf.h | 13 ++++---- kernel/bpf/inode.c | 5 ++-- kernel/bpf/syscall.c | 30 ++++++------------- kernel/events/core.c | 7 ++--- 11 files changed, 40 insertions(+), 97 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +2 -7
@@ -3171,13 +3171,8 @@ static int bnxt_init_one_rx_ring(struct bnxt *bp, int ring_nr)bnxt_init_rxbd_pages (ring, type);if (BNXT_RX_PAGE_MODE (bp) && bp->xdp_prog) { - rxr->xdp_prog =bpf_prog_add (bp->xdp_prog,1 ); -if (IS_ERR (rxr->xdp_prog)) { -int rc =PTR_ERR (rxr->xdp_prog); - - rxr->xdp_prog =NULL ; -return rc; - } +bpf_prog_add (bp->xdp_prog,1 ); + rxr->xdp_prog = bp->xdp_prog; } prod = rxr->rx_prod;for (i =0 ; i < bp->rx_ring_size; i++) {
drivers/net/ethernet/cavium/thunder/nicvf_main.c +2 -7
@@ -1876,13 +1876,8 @@ static int nicvf_xdp_setup(struct nicvf *nic, struct bpf_prog *prog)if (nic->xdp_prog) {/* Attach BPF program */ - nic->xdp_prog =bpf_prog_add (nic->xdp_prog, nic->rx_queues -1 ); -if (!IS_ERR (nic->xdp_prog)) { - bpf_attached = true; - }else { - ret =PTR_ERR (nic->xdp_prog); - nic->xdp_prog =NULL ; - } +bpf_prog_add (nic->xdp_prog, nic->rx_queues -1 ); + bpf_attached = true; }/* Calculate Tx queues needed for XDP and network stack */
drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c +2 -5
@@ -1807,11 +1807,8 @@ static int setup_xdp(struct net_device *dev, struct bpf_prog *prog)if (prog && !xdp_mtu_valid (priv, dev->mtu))return -EINVAL ; -if (prog) { - prog =bpf_prog_add (prog, priv->num_channels); -if (IS_ERR (prog)) -return PTR_ERR (prog); - } +if (prog) +bpf_prog_add (prog, priv->num_channels); up =netif_running (dev); need_update = (!!priv->xdp_prog != !!prog);
drivers/net/ethernet/mellanox/mlx4/en_netdev.c +6 -18
@@ -2286,11 +2286,7 @@ int mlx4_en_try_alloc_resources(struct mlx4_en_priv *priv, lockdep_is_held(&priv->mdev->state_lock));if (xdp_prog && carry_xdp_prog) { - xdp_prog =bpf_prog_add (xdp_prog, tmp->rx_ring_num); -if (IS_ERR (xdp_prog)) { -mlx4_en_free_resources (tmp); -return PTR_ERR (xdp_prog); - } +bpf_prog_add (xdp_prog, tmp->rx_ring_num);for (i =0 ; i < tmp->rx_ring_num; i++)rcu_assign_pointer (tmp->rx_ring[i]->xdp_prog, xdp_prog);
@@ -2782,11 +2778,9 @@ static int mlx4_xdp_set(struct net_device *dev, struct bpf_prog *prog)* program for a new one. */ if (priv->tx_ring_num[TX_XDP ] == xdp_ring_num) { -if (prog) { - prog =bpf_prog_add (prog, priv->rx_ring_num -1 ); -if (IS_ERR (prog)) -return PTR_ERR (prog); - } +if (prog) +bpf_prog_add (prog, priv->rx_ring_num -1 ); +mutex_lock (&mdev->state_lock);for (i =0 ; i < priv->rx_ring_num; i++) { old_prog = rcu_dereference_protected(
@@ -2807,13 +2801,8 @@ static int mlx4_xdp_set(struct net_device *dev, struct bpf_prog *prog)if (!tmp)return -ENOMEM ; -if (prog) { - prog =bpf_prog_add (prog, priv->rx_ring_num -1 ); -if (IS_ERR (prog)) { - err =PTR_ERR (prog); - goto out; - } - } +if (prog) +bpf_prog_add (prog, priv->rx_ring_num -1 );mutex_lock (&mdev->state_lock);memcpy (&new_prof, priv->prof,sizeof (struct mlx4_en_port_profile ));
@@ -2862,7 +2851,6 @@ static int mlx4_xdp_set(struct net_device *dev, struct bpf_prog *prog) unlock_out:mutex_unlock (&mdev->state_lock); -out:kfree (tmp);return err; }
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +5 -13
@@ -408,12 +408,9 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, rq->stats = &c->priv->channel_stats[c->ix].rq;INIT_WORK (&rq->recover_work, mlx5e_rq_err_cqe_work); - rq->xdp_prog = params->xdp_prog ?bpf_prog_inc (params->xdp_prog) :NULL ; -if (IS_ERR (rq->xdp_prog)) { - err =PTR_ERR (rq->xdp_prog); - rq->xdp_prog =NULL ; - goto err_rq_wq_destroy; - } +if (params->xdp_prog) +bpf_prog_inc (params->xdp_prog); + rq->xdp_prog = params->xdp_prog; rq_xdp_ix = rq->ix;if (xsk)
@@ -4406,16 +4403,11 @@ static int mlx5e_xdp_set(struct net_device *netdev, struct bpf_prog *prog)/* no need for full reset when exchanging programs */ reset = (!priv->channels.params.xdp_prog || !prog); -if (was_opened && !reset) { +if (was_opened && !reset)/* num_channels is invariant here, so we can take the * batched reference right upfront. */ - prog =bpf_prog_add (prog, priv->channels.num); -if (IS_ERR (prog)) { - err =PTR_ERR (prog); - goto unlock; - } - } +bpf_prog_add (prog, priv->channels.num);if (was_opened && reset) {struct mlx5e_channels new_channels = {};
drivers/net/ethernet/qlogic/qede/qede_main.c +2 -6
@@ -2107,12 +2107,8 @@ static int qede_start_queues(struct qede_dev *edev, bool clear_stats)if (rc) goto out; - fp->rxq->xdp_prog =bpf_prog_add (edev->xdp_prog,1 ); -if (IS_ERR (fp->rxq->xdp_prog)) { - rc =PTR_ERR (fp->rxq->xdp_prog); - fp->rxq->xdp_prog =NULL ; - goto out; - } +bpf_prog_add (edev->xdp_prog,1 ); + fp->rxq->xdp_prog = edev->xdp_prog; }if (fp->type &QEDE_FASTPATH_TX ) {
drivers/net/virtio_net.c +2 -5
@@ -2445,11 +2445,8 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog,if (!prog && !old_prog)return 0 ; -if (prog) { - prog =bpf_prog_add (prog, vi->max_queue_pairs -1 ); -if (IS_ERR (prog)) -return PTR_ERR (prog); - } +if (prog) +bpf_prog_add (prog, vi->max_queue_pairs -1 );/* Make sure NAPI is not using any XDP TX queues for RX. */ if (netif_running (dev)) {
include/linux/bpf.h +5 -8
@@ -485,7 +485,7 @@ struct bpf_func_info_aux { };struct bpf_prog_aux { -atomic_t refcnt; +atomic64_t refcnt;u32 used_map_cnt;u32 max_ctx_offset;u32 max_pkt_offset;
@@ -770,9 +770,9 @@ extern const struct bpf_verifier_ops xdp_analyzer_ops;struct bpf_prog *bpf_prog_get (u32 ufd);struct bpf_prog *bpf_prog_get_type_dev (u32 ufd,enum bpf_prog_type type,bool attach_drv); -struct bpf_prog *__must_check bpf_prog_add (struct bpf_prog *prog,int i); +void bpf_prog_add (struct bpf_prog *prog,int i);void bpf_prog_sub (struct bpf_prog *prog,int i); -struct bpf_prog *__must_check bpf_prog_inc (struct bpf_prog *prog); +void bpf_prog_inc (struct bpf_prog *prog);struct bpf_prog *__must_check bpf_prog_inc_not_zero (struct bpf_prog *prog);void bpf_prog_put (struct bpf_prog *prog);int __bpf_prog_charge (struct user_struct *user,u32 pages);
@@ -912,10 +912,8 @@ static inline struct bpf_prog *bpf_prog_get_type_dev(u32 ufd,return ERR_PTR (-EOPNOTSUPP ); } -static inline struct bpf_prog *__must_check bpf_prog_add (struct bpf_prog *prog, -int i) +static inline void bpf_prog_add (struct bpf_prog *prog,int i) { -return ERR_PTR (-EOPNOTSUPP ); }static inline void bpf_prog_sub (struct bpf_prog *prog,int i)
@@ -926,9 +924,8 @@ static inline void bpf_prog_put(struct bpf_prog *prog) { } -static inline struct bpf_prog *__must_check bpf_prog_inc (struct bpf_prog *prog) +static inline void bpf_prog_inc (struct bpf_prog *prog) { -return ERR_PTR (-EOPNOTSUPP ); } static inline struct bpf_prog *__must_check
kernel/bpf/inode.c +3 -2
@@ -31,7 +31,7 @@ static void *bpf_any_get(void *raw, enum bpf_type type) {switch (type) {case BPF_TYPE_PROG : - raw =bpf_prog_inc (raw); +bpf_prog_inc (raw);break ;case BPF_TYPE_MAP :bpf_map_inc_with_uref (raw);
@@ -534,7 +534,8 @@ static struct bpf_prog *__get_prog_inode(struct inode *inode, enum bpf_prog_typeif (!bpf_prog_get_ok (prog, &type, false))return ERR_PTR (-EINVAL ); -return bpf_prog_inc (prog); +bpf_prog_inc (prog); +return prog; }struct bpf_prog *bpf_prog_get_type_path (const char *name,enum bpf_prog_type type)
kernel/bpf/syscall.c +9 -21
@@ -1339,7 +1339,7 @@ static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred)static void __bpf_prog_put (struct bpf_prog *prog,bool do_idr_lock) { -if (atomic_dec_and_test (&prog->aux->refcnt)) { +if (atomic64_dec_and_test (&prog->aux->refcnt)) {perf_event_bpf_event (prog,PERF_BPF_EVENT_PROG_UNLOAD ,0 );/* bpf_prog_free_id() must be called first */ bpf_prog_free_id (prog, do_idr_lock);
@@ -1445,16 +1445,9 @@ static struct bpf_prog *____bpf_prog_get(struct fd f)return f.file->private_data; } -/* prog's refcnt limit */ -#define BPF_MAX_REFCNT 32768 - -struct bpf_prog *bpf_prog_add (struct bpf_prog *prog,int i) +void bpf_prog_add (struct bpf_prog *prog,int i) { -if (atomic_add_return (i, &prog->aux->refcnt) >BPF_MAX_REFCNT ) { -atomic_sub (i, &prog->aux->refcnt); -return ERR_PTR (-EBUSY ); - } -return prog; +atomic64_add (i, &prog->aux->refcnt); }EXPORT_SYMBOL_GPL (bpf_prog_add);
@@ -1465,13 +1458,13 @@ void bpf_prog_sub(struct bpf_prog *prog, int i)* path holds a reference to the program, thus atomic_sub() can * be safely used in such cases! */ -WARN_ON (atomic_sub_return (i, &prog->aux->refcnt) ==0 ); +WARN_ON (atomic64_sub_return (i, &prog->aux->refcnt) ==0 ); }EXPORT_SYMBOL_GPL (bpf_prog_sub); -struct bpf_prog *bpf_prog_inc (struct bpf_prog *prog) +void bpf_prog_inc (struct bpf_prog *prog) { -return bpf_prog_add (prog,1 ); +atomic64_inc (&prog->aux->refcnt); }EXPORT_SYMBOL_GPL (bpf_prog_inc);
@@ -1480,12 +1473,7 @@ struct bpf_prog *bpf_prog_inc_not_zero(struct bpf_prog *prog) {int refold; - refold =atomic_fetch_add_unless (&prog->aux->refcnt,1 ,0 ); - -if (refold >=BPF_MAX_REFCNT ) { -__bpf_prog_put (prog, false); -return ERR_PTR (-EBUSY ); - } + refold =atomic64_fetch_add_unless (&prog->aux->refcnt,1 ,0 );if (!refold)return ERR_PTR (-ENOENT );
@@ -1523,7 +1511,7 @@ static struct bpf_prog *__bpf_prog_get(u32 ufd, enum bpf_prog_type *attach_type, goto out; } - prog =bpf_prog_inc (prog); +bpf_prog_inc (prog); out:fdput (f);return prog;
@@ -1714,7 +1702,7 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) prog->orig_prog =NULL ; prog->jited =0 ; -atomic_set (&prog->aux->refcnt,1 ); +atomic64_set (&prog->aux->refcnt,1 ); prog->gpl_compatible = is_gpl ?1 :0 ;if (bpf_prog_is_dev_bound (prog->aux)) {
kernel/events/core.c +2 -5
@@ -10477,12 +10477,9 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu, context = parent_event->overflow_handler_context;#if defined(CONFIG_BPF_SYSCALL ) && defined(CONFIG_EVENT_TRACING )if (overflow_handler == bpf_overflow_handler) { -struct bpf_prog *prog =bpf_prog_inc (parent_event->prog); +struct bpf_prog *prog = parent_event->prog; -if (IS_ERR (prog)) { - err =PTR_ERR (prog); - goto err_ns; - } +bpf_prog_inc (prog); event->prog = prog; event->orig_overflow_handler = parent_event->orig_overflow_handler;
Signature · 1 line
2.17.1
PATCH v5 3/5 bpf: add mmap() support for BPF_MAP_TYPE_ARRAY
In reply to Andrii NakryikoAdd ability to memory-map contents of BPF array map. This is extremely useful for working with BPF global data from userspace programs. It allows to avoid typical bpf_map_{lookup,update}_elem operations, improving both performance and usability.
There had to be special considerations for map freezing, to avoid having writable memory view into a frozen map. To solve this issue, map freezing and mmap-ing is happening under mutex now:
- if map is already frozen, no writable mapping is allowed;
- if map has writable memory mappings active (accounted in map->writecnt),
map freezing will keep failing with -EBUSY;
- once number of writable memory mappings drops to zero, map freezing can be performed again.
Only non-per-CPU plain arrays are supported right now. Maps with spinlocks can't be memory mapped either.
For BPF_F_MMAPABLE array, memory allocation has to be done through vmalloc() to be mmap()'able. We also need to make sure that array data memory is page-sized and page-aligned, so we over-allocate memory in such a way that struct bpf_array is at the end of a single page of memory with array->value being aligned with the start of the second page. On deallocation we need to accomodate this memory arrangement to free vmalloc()'ed memory correctly.
One important consideration regarding how memory-mapping subsystem functions. Memory-mapping subsystem provides few optional callbacks, among them open() and close(). close() is called for each memory region that is unmapped, so that users can decrease their reference counters and free up resources, if necessary. open() is *almost* symmetrical: it's called for each memory region that is being mapped, **except** the very first one. So bpf_map_mmap does initial refcnt bump, while open() will do any extra ones after that. Thus number of close() calls is equal to number of open() calls plus one more.
--- include/linux/bpf.h | 11 ++-- include/linux/vmalloc.h | 1 + include/uapi/linux/bpf.h | 3 ++ kernel/bpf/arraymap.c | 58 +++++++++++++++++--- kernel/bpf/syscall.c | 99 ++++++++++++++++++++++++++++++++-- mm/vmalloc.c | 20 +++++++ tools/include/uapi/linux/bpf.h | 3 ++ 7 files changed, 183 insertions(+), 12 deletions(-)
include/linux/bpf.h +8 -3
@@ -12,6 +12,7 @@#include <linux/err.h> #include <linux/rbtree_latch.h> #include <linux/numa.h> +#include <linux/mm_types.h> #include <linux/wait.h> #include <linux/u64_stats_sync.h> #include <linux/refcount.h>
@@ -68,6 +69,7 @@ struct bpf_map_ops { u64 *imm, u32 off);int (*map_direct_value_meta)(const struct bpf_map *map,u64 imm,u32 *off); +int (*map_mmap)(struct bpf_map *map,struct vm_area_struct *vma); };struct bpf_map_memory {
@@ -96,9 +98,10 @@ struct bpf_map {u32 btf_value_type_id;struct btf *btf;struct bpf_map_memory memory; +char name[BPF_OBJ_NAME_LEN ];bool unpriv_array; -bool frozen;/* write-once */ -/* 48 bytes hole */ +bool frozen;/* write-once; write-protected by freeze_mutex */ +/* 22 bytes hole */ /* The 3rd and 4th cacheline with misc members to avoid false sharing * particularly with refcounting.
@@ -106,7 +109,8 @@ struct bpf_map {atomic64_t refcnt____cacheline_aligned ;atomic64_t usercnt;struct work_struct work; -char name[BPF_OBJ_NAME_LEN ]; +struct mutex freeze_mutex; +u64 writecnt;/* writable mmap cnt; protected by freeze_mutex */ };static inline bool map_value_has_spin_lock (const struct bpf_map *map)
@@ -795,6 +799,7 @@ void bpf_map_charge_finish(struct bpf_map_memory *mem);void bpf_map_charge_move (struct bpf_map_memory *dst,struct bpf_map_memory *src);void *bpf_map_area_alloc (size_t size,int numa_node); +void *bpf_map_area_mmapable_alloc (size_t size,int numa_node);void bpf_map_area_free (void *base);void bpf_map_init_from_attr (struct bpf_map *map,union bpf_attr *attr);
include/linux/vmalloc.h +1 -0
@@ -93,6 +93,7 @@ extern void *vzalloc(unsigned long size);extern void *vmalloc_user (unsigned long size);extern void *vmalloc_node (unsigned long size,int node);extern void *vzalloc_node (unsigned long size,int node); +extern void *vmalloc_user_node_flags (unsigned long size,int node,gfp_t flags);extern void *vmalloc_exec (unsigned long size);extern void *vmalloc_32 (unsigned long size);extern void *vmalloc_32_user (unsigned long size);
include/uapi/linux/bpf.h +3 -0
@@ -348,6 +348,9 @@ enum bpf_attach_type {/* Clone map from listener for newly accepted socket */ #define BPF_F_CLONE (1U << 9) +/* Enable memory-mapping BPF map */ +#define BPF_F_MMAPABLE (1U << 10) +/* flags for BPF_PROG_QUERY */ #define BPF_F_QUERY_EFFECTIVE (1U << 0)
kernel/bpf/arraymap.c +52 -6
107 lines not on this page · raw message
kernel/bpf/syscall.c +96 -3
151 lines not on this page · raw message
mm/vmalloc.c +20 -0
@@ -2671,6 +2671,26 @@ void *vzalloc_node(unsigned long size, int node) }EXPORT_SYMBOL (vzalloc_node); +/** +* vmalloc_user_node_flags - allocate memory for userspace on a specific node +* @size: allocation size +* @node: numa node +* @flags: flags for the page level allocator +* +* The resulting memory area is zeroed so it can be mapped to userspace +* without leaking data. +* +* Return: pointer to the allocated memory or %NULL on error +*/ +void *vmalloc_user_node_flags (unsigned long size,int node,gfp_t flags) +{ +return __vmalloc_node_range (size,SHMLBA ,VMALLOC_START ,VMALLOC_END , + flags | __GFP_ZERO,PAGE_KERNEL , +VM_USERMAP , node, +__builtin_return_address (0 )); +} +EXPORT_SYMBOL (vmalloc_user_node_flags); +/** * vmalloc_exec - allocate virtually contiguous, executable memory * @size: allocation size
tools/include/uapi/linux/bpf.h +3 -0
@@ -348,6 +348,9 @@ enum bpf_attach_type {/* Clone map from listener for newly accepted socket */ #define BPF_F_CLONE (1U << 9) +/* Enable memory-mapping BPF map */ +#define BPF_F_MMAPABLE (1U << 10) +/* flags for BPF_PROG_QUERY */ #define BPF_F_QUERY_EFFECTIVE (1U << 0)
Signature · 1 line
2.17.1
PATCH v5 4/5 libbpf: make global data internal arrays mmap()-able, if possible
In reply to Andrii NakryikoAdd detection of BPF_F_MMAPABLE flag support for arrays and add it as an extra flag to internal global data maps, if supported by kernel. This allows users to memory-map global data and use it without BPF map operations, greatly simplifying user experience.
--- tools/lib/bpf/libbpf.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-)
tools/lib/bpf/libbpf.c +30 -2
@@ -142,6 +142,8 @@ struct bpf_capabilities { __u32 btf_func:1;/* BTF_KIND_VAR and BTF_KIND_DATASEC support */ __u32 btf_datasec:1; +/* BPF_F_MMAPABLE is supported for arrays */ + __u32 array_mmap:1; };/*
@@ -857,8 +859,6 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,pr_warn ("failed to alloc map name\n" );return -ENOMEM ; } -pr_debug ("map '%s' (global data): at sec_idx %d, offset %zu.\n" , - map_name, map->sec_idx, map->sec_offset); def = &map->def; def->type =BPF_MAP_TYPE_ARRAY ;
@@ -866,6 +866,12 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, def->value_size = data->d_size; def->max_entries =1 ; def->map_flags = type ==LIBBPF_MAP_RODATA ?BPF_F_RDONLY_PROG :0 ; +if (obj->caps.array_mmap) + def->map_flags |=BPF_F_MMAPABLE ; + +pr_debug ("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n" , + map_name, map->sec_idx, map->sec_offset, def->map_flags); +if (data_buff) { *data_buff =malloc (data->d_size);if (!*data_buff) {
@@ -2161,6 +2167,27 @@ static int bpf_object__probe_btf_datasec(struct bpf_object *obj)return 0 ; } +static int bpf_object__probe_array_mmap (struct bpf_object *obj) +{ +struct bpf_create_map_attr attr = { + .map_type =BPF_MAP_TYPE_ARRAY , + .map_flags =BPF_F_MMAPABLE , + .key_size =sizeof (int ), + .value_size =sizeof (int ), + .max_entries =1 , + }; +int fd; + + fd =bpf_create_map_xattr (&attr); +if (fd >=0 ) { + obj->caps.array_mmap =1 ; +close (fd); +return 1 ; + } + +return 0 ; +} +static int bpf_object__probe_caps (struct bpf_object *obj) {
@@ -2169,6 +2196,7 @@ bpf_object__probe_caps(struct bpf_object *obj) bpf_object__probe_global_data, bpf_object__probe_btf_func, bpf_object__probe_btf_datasec, + bpf_object__probe_array_mmap, };int i, ret;
Signature · 1 line
2.17.1
PATCH v5 5/5 selftests/bpf: add BPF_TYPE_MAP_ARRAY mmap() tests
In reply to Andrii NakryikoAdd selftests validating mmap()-ing BPF array maps: both single-element and multi-element ones. Check that plain bpf_map_update_elem() and bpf_map_lookup_elem() work correctly with memory-mapped array. Also convert CO-RE relocation tests to use memory-mapped views of global data.
--- .../selftests/bpf/prog_tests/core_reloc.c | 45 ++-- tools/testing/selftests/bpf/prog_tests/mmap.c | 220 ++++++++++++++++++ tools/testing/selftests/bpf/progs/test_mmap.c | 45 ++++ 3 files changed, 292 insertions(+), 18 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/mmap.c create mode 100644 tools/testing/selftests/bpf/progs/test_mmap.c
tools/testing/selftests/bpf/prog_tests/core_reloc.c +27 -18
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0 #include <test_progs.h> #include "progs/core_reloc_types.h" +#include <sys/mman.h> #define STRUCT_TO_CHAR_PTR (struct_name) (const char *)&(struct struct_name)
@@ -453,8 +454,15 @@ struct data {char out[256 ]; }; +static size_t roundup_page (size_t sz) +{ +long page_size =sysconf (_SC_PAGE_SIZE); +return (sz + page_size -1 ) / page_size * page_size; +} +void test_core_reloc (void ) { +const size_t mmap_sz =roundup_page (sizeof (struct data ));struct bpf_object_load_attr load_attr = {};struct core_reloc_test_case *test_case;const char *tp_name, *probe_name;
@@ -463,8 +471,8 @@ void test_core_reloc(void)struct bpf_map *data_map;struct bpf_program *prog;struct bpf_object *obj; -const int zero =0 ; -struct data data; +struct data *data; +void *mmap_data =NULL ;for (i =0 ; i <ARRAY_SIZE (test_cases); i++) { test_case = &test_cases[i];
@@ -476,8 +484,7 @@ void test_core_reloc(void) ); obj =bpf_object__open_file (test_case->bpf_obj_file, &opts); -if (CHECK (IS_ERR_OR_NULL (obj),"obj_open" , -"failed to open '%s': %ld\n" , +if (CHECK (IS_ERR (obj),"obj_open" ,"failed to open '%s': %ld\n" , test_case->bpf_obj_file,PTR_ERR (obj)))continue ;
@@ -519,24 +526,22 @@ void test_core_reloc(void)if (CHECK (!data_map,"find_data_map" ,"data map not found\n" )) goto cleanup; -memset (&data,0 ,sizeof (data)); -memcpy (data.in, test_case->input, test_case->input_len); - - err =bpf_map_update_elem (bpf_map__fd (data_map), - &zero, &data,0 ); -if (CHECK (err,"update_data_map" , -"failed to update .data map: %d\n" , err)) + mmap_data =mmap (NULL , mmap_sz,PROT_READ |PROT_WRITE , +MAP_SHARED ,bpf_map__fd (data_map),0 ); +if (CHECK (mmap_data ==MAP_FAILED ,"mmap" , +".bss mmap failed: %d" , errno)) { + mmap_data =NULL ; goto cleanup; + } + data = mmap_data; + +memset (mmap_data,0 ,sizeof (*data)); +memcpy (data->in, test_case->input, test_case->input_len);/* trigger test run */ usleep (1 ); - err =bpf_map_lookup_elem (bpf_map__fd (data_map), &zero, &data); -if (CHECK (err,"get_result" , -"failed to get output data: %d\n" , err)) - goto cleanup; - - equal =memcmp (data.out, test_case->output, + equal =memcmp (data->out, test_case->output, test_case->output_len) ==0 ;if (CHECK (!equal,"check_result" ,"input/output data don't match\n" )) {
@@ -548,12 +553,16 @@ void test_core_reloc(void) }for (j =0 ; j < test_case->output_len; j++) {printf ("output byte #%d: EXP 0x%02hhx GOT 0x%02hhx\n" , - j, test_case->output[j], data.out[j]); + j, test_case->output[j], data->out[j]); } goto cleanup; } cleanup: +if (mmap_data) { +CHECK_FAIL (munmap (mmap_data, mmap_sz)); + mmap_data =NULL ; + }if (!IS_ERR_OR_NULL (link)) {bpf_link__destroy (link); link =NULL ;
tools/testing/selftests/bpf/prog_tests/mmap.c new file +220 -0
220 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_mmap.c new file +45 -0
45 lines not on this page · raw message
Signature · 1 line
2.17.1